Extending XMHF to support 64-bit Intel and modern operating systems

a.img is the first 1 MiB of a MBR disk (containing MBR header, grub, etc.). By “magic” I mean that it is not directly available from files that can be downloaded online.

I have updated the docs with some QEMU commands I usually use.

Hi Amit, I see that you have approved the PR. Since I do not have merge permissions to the uberspark/uberxmhf repo, could you please merge it if it looks good to you? Thanks.

Sorry, I must have clicked approved by mistake. Am still reviewing it but hope to finish by end of this week. Thanks and sorry for the delay, it is a large patchset…

No worries. Please take your time.

Ok I made a pass on all the folders and files within the PR (quite a few of them!). They are looking good for the most part. I am guessing most of the xmhf-64/hypapps, xmhf-64/xmhf/src/libbaremetal and xmhf-64/third-party are all brought forward from the legacy 32-bit xmhf code? Are there any specific addition and/or changes you made in the aforementioned folders?

Also I would suggest making the following changes:

  1. docs/pc-intel-x86_64/introduction.rst : Change Some bugs found in XMHF are fixed in XMHF-64 to XMHF-64 also adds various other logic refinements and fixes to XMHF

  2. CHANGELOG.md: Bug FixesLogic Fixes

  3. docs/pc-intel-x86_64/supported-os.rst : You are using * to mean two different things in the table (XMHF 32/64 bit arch and DRT support) and it is confusing at a first glance. I would suggest using two different symbols * and # perhaps to distinguish the footnotes.

I am diving into the testing phase now. My goal is to be able to successfully build xmhf-64 based on the instructions you have on Debian 11 and then run it within QEMU. Will post back my progress as I go through it. Meanwhile, please let me know if there are any special pointers/cases I need to be aware of (that is not in the current documentation of the PR).

Thanks again for your patience!

Ok build was successful with a fresh install of Debian 11 within Hyper-V. I had to install aptitude as it was not installed by default. Can you please add sudo apt install aptitude to the list of packages to install on Ubuntu/Debian within docs/pc-intel-x86_64/build.rst?

I am moving on to install Debian 11 64-bit within QEMU and executing the generated hypervisor-x86-amd64.bin.gz. Stay tuned…

Also add a note within docs/pc-intel-x86_64/build.rst to say that if you re-build with a change in ./configure parameters (e.g., say you build with --enable-dmap and then build with --disable-dmap) you need to issue make distclean before the new ./configure command. If not the build fails.

Also the --enable-debug-qemu command does not seem to be described within docs/pc-intel-x86_64/build.rst

Some changes are made in hypapps/trustvisor to run in x64. Most of other directories are brought from legacy xmhf directory.

Changed

I have changed aptitude to apt-get. They are basically the same, though personally I am used to aptitude. apt-get is usually installed by default in Debian.

Added

Added

Thanks for addressing my previous comments @lxylxy123456 !

Ok, so I followed instructions within docs/pc-intel-x86_64/build.rst and built the hypervisor using the following commands on a Debian-11 64-bit host running under Hyper-V with 2 cores and nested VMX enabled (/proc/cpuinfo shows vmx). GCC version is Debian 10.2.1-6.

./autogen.sh
./configure --with-approot=hypapps/helloworld --with-target-subarch=amd64 --enable-debug-symbols --enable-debug-qemu --with-amd64-max-phys-addr=0x200000000 --disable-debug-serial --enable-debug-vga --disable-dmap --disable-drt

I then followed the instructions in docs/pc-intel-x86_64/installing.rst and installed QEMU, created the guest OS disk, installed Debian 11 64-bit and executed the guest (without XMHF) using the following command:

qemu-system-x86_64 -m 1024 --drive media=disk,file=testguest.qcow2 --enable-kvm -serial stdio -cpu Haswell,vmx=yes

The guest OS booted up just fine and I was able to access the internet etc. within the guest OS inside QEMU.

Next, I installed the hypervisor init and runtime files as specified within docs/pc-intel-x86_64/installing.rst and updated the GRUB entries as specified. I had to mount the guest qcow2 disk for this. Then booted up the guest OS again with the aforementioned QEMU command. The guest loads fine but its quite slow (even with KVM acceleration) in graphics mode. I guess this is expected with the nested virtualization overheads, but I think we need to make a note of this in the documentation and perhaps state that a text mode boot-up might be more productive within QEMU.

Also a few things to note at runtime with QEMU:

  1. serial debug messages are not being reflected in the host console; they are only appearing in the guest VM (since --enable-vga is used).
  2. I tried with -serial file:mylog.log but no cigar there
  3. sometimes the guest VM barfs during startup with a unhandled exception 3 within XMHF. This is not always reproducible but does happen on and off. I have attached a screen-shot to this effect which shows the guest VM output.

A few points that need to make it to the documentation:

  1. I needed to install the following packages for QEMU to work within the host:

    sudo apt-get update
    sudo apt install qemu-kvm libvirt-clients libvirt-daemon-system bridge-utils virtinst libvirt-daemon
    

    This needs to be mentioned within docs/pc-intel-x86_64/installing.rst

  2. The default QEMU networking setup does not allow communication between the host and the guest (uses SLIRP). So the only way to add the hypervisor init and runtime binaries to the guest /boot partition is by mounting the QEMU guest disk. The following worked for me for mounting:

    modprobe nbd max_part=8
    qemu-nbd --connect=/dev/nbd0 <name of guest disk.qcow2>
    fdisk /dev/nbd0 -l
    mount /dev/nbd0p1 <mount path on host>
    

    The corresponding steps to unmount once the hypervisor images have been copied over:

    umount <mount path on host>
    qemu-nbd --disconnect /dev/nbd0
    rmmod nbd
    

    This needs to be mentioned within docs/pc-intel-x86_64/installing.rst

  3. The update GRUB commands within docs/pc-intel-x86_64/installing.rst before the multiple disks section needs to have a sudo prefix to work

I am going to go ahead and test DRT and DMAP support within your PR. I see that you mention you have tested it on a HP machine. Can you provide me with the details of the platform/model so I can see if I can find one that I have which is the closest match?

Also you mention Windows 10 support. How do you install XMHF in such a case? Will it use the multiple disks installation method? i.e., have Windows 10 as is on the disk and boot via a USB disk with XMHF on it?

I am a little bit confused about your setup. Are you running something like the following?

Intel processor
  Windows
    Hyper-V
      Debian (for compiling)
        KVM & QEMU
          XMHF
            Debian (for testing)

If that is the case, it is normal for “Debian (for testing)” to be slow, since it is nested in a few layers.

On my end, my setup is

Intel processor
  Fedora or Debian (for compiling)
    KVM & QEMU
      XMHF
        Debian (for testing)

It is possible to compile and test on the same Debian machine. Then you can avoid copying files around multiple physical / virtual OSes. That is something like:

(Intel processor) or (Hyper-V under Windows)
  Fedora or Debian (just for running KVM)
    KVM & QEMU
      Debian (for compiling)
      XMHF
        The same Debian (for testing)

For example, my Debian virtual machine’s GRUB menu looks like:

Debian GNU/Linux                      
Advanced options for Debian GNU/Linux 
XMHF-i386                             
XMHF-amd64                            

When I want to compile XMHF, I can go to Debian GNU/Linux. After compiling and installing, I just need to reboot and select XMHF-amd64.

I am not sure why you are using --enable-debug-vga. If you use --enable-debug-serial --disable-debug-vga, the QEMU serial port should work perfectly.

In my setup, I see this problem sometimes when I am using GDB to debug. This problem may be explained if you are running the host Debian under Hyper-V. I have to say that KVM is not 100% reliable.

I think this is out of scope of XMHF’s documentation. These commands are used to install QEMU, which is a pre-requisite for running XMHF in QEMU.

Ideally setting up XMHF in QEMU and XMHF in real hardware are the same. That is, no documents should be needed if the user knows how to use QEMU well. I think it is better to add a separate .rst file addressing hints and setup about QEMU (hopefully in a future PR).

I did not address it because I think it is also out of scope. I use the following to forward host’s SSH_PORT to guest’s port 22, so I can ssh the guest from the host. This option can be found from QEMU’s wiki IIRC.

"-device" "e1000,netdev=net0" "-netdev" "user,id=net0,hostfwd=tcp::${SSH_PORT}-:22"

Currently I am aware of 3 ways to install XMHF for a QEMU guest.

  1. Compile on QEMU guest, run on QEMU guest. This is likely the slowest, but avoids setting up network or qemu-nbd
  2. Compile on host, copy the binaries to guest using qemu-nbd, then run on guest (what you are doing here). This is faster because when compiling, the QEMU guest does not need to boot. It is still a little bit slow because umount sometimes has a delay. Also, this requires sudo on the host.
  3. Compile on host, create a small disk using debugfs and mkfs.ext4. The small disk is used to boot XMHF, then XMHF will load Debian from another disk. This method is how I currently run CI. It is faster than qemu-nbd and does not require sudo. However, setting it up can be tricky

Added

@superymk knows better about the machine model. I know that it is EliteBook 2540p, with CPU information: Intel(R) Core(TM) i5 CPU M 540 @ 2.53GHz stepping 05

There are two ways to set it up.

  1. I use this way on a physical HP 2540P. Simply install Debian and Windows on the same disk. Then install XMHF on Debian. This way should also work on QEMU, though the .qcow2 file will become very large, after installing 2 OSes.
  2. I use this way in QEMU.
    1. First install Debian on debian.qcow2.
    2. Then install Windows on windows.qcow2.
    3. Find a way to boot Windows from Debian’s GRUB. There are 2 ways (choose one)
      a. Then in one QEMU machine, boot debian.qcow2, but mount windows.qcow2 as a second disk. Then run update-grub. At this time GRUB should detect Windows and create a boot entry.
      b. Add a GRUB menu entry to /etc/grub.d I obtained from a previous run, see below
    4. Use the Debian GRUB to boot XMHF and Windows
#!/bin/sh
exec tail -n +3 $0
menuentry 'Windows' --class windows --class os $menuentry_id_option 'windows' {
	insmod part_msdos
	insmod ntfs
	set root='hd1,msdos1'
	parttool ${root} hidden-
	drivemap -s (hd0) ${root}
	chainloader +1
}

The GRUB menu should look like

Debian GNU/Linux                     
Advanced options for Debian GNU/Linux
Windows                              
XMHF-i386                            
XMHF-amd64                           

The multiple disks installation method should work on physical machines. I recently realized that it does not work on KVM. This is likely a bug in KVM (I see KVM_GET_PIT2 failed: Input/output error). See uberxmhf/bug_077 at notes · lxylxy123456/uberxmhf · GitHub for details.

Update on Windows testing: as I mentioned earlier, multiple disks installation on KVM has a bug when booting Windows. I confirmed that the bug is in KVM and reported it in https://bugzilla.kernel.org/show_bug.cgi?id=216046.

However, I found a workaround. QEMU uses SeaBIOS by default. If the BIOS is re-compiled with “System Management Mode (SMM)” set to off, the problem will disappear. The BIOS I compiled is in https://github.com/lxylxy123456/uberxmhf/blob/notes/bug_031/bios.bin. In QEMU command line add -bios bios.bin.

Multiple disks installation can be generated using a Python script in the xmhf64 branch in my repository. If you clone https://github.com/lxylxy123456/uberxmhf/tree/xmhf64/, then you can use the script.

python3 /path/to/lxylxy123456/uberxmhf/branch/xmhf64/./tools/ci/grub.py \
	--subarch "amd64" \
	--xmhf-bin "/path/to/your/build/directory/" \
	--work-dir "/path/to/output/" \
	--boot-dir "/path/to/lxylxy123456/uberxmhf/branch/xmhf64/./tools/ci/boot/"

The script will take /path/to/your/build/directory/init-x86-amd64.bin and /path/to/your/build/directory/hypervisor-x86-amd64.bin.gz. The script will generate /path/to/output/grub/c.img. This c.img can be used in the QEMU argument: -drive media=disk,file=c.img,index=0.

(Hopefully I will add the above to the documentation in a future PR)

Please let me know if you need other helps in testing. If needed, I also have QEMU images for all OSes I tested uploaded on Google Drive (Windows XP - 10, Debian 11, Fedora 35).

Thanks @lxylxy123456!

I am currently traveling but will check out your instructions once am back end of next week.

1 Like

Hi @amitvasudevan, is there any updates about testing the new xmhf-64? Please let me know if you have any questions.

Hi @lxylxy123456 ,

Sorry for the delay in my response. Between the summer travel and my Haswell box biting the dust, I have not had luck in continuing testing the PR uninterrupted. However, I now have a new Haswell box and performed a couple of tests with results below:

System: Core I7-4770 (Haswell) , 4GB ram
OS: Windows 10: Build 19014.1826 64-bit and Debian 11 64-bit
GRUB 2 on MBR and /dev/sda1, Windows on /dev/sda2 and Debian on /dev/sda5
System is configured for multi-booting Debian and Windows and works fine without XMHF installed.

With XMHF installed, I have the following results:

  1. ./configure --with-approot=hypapps/helloworld --with-target-subarch=amd64 --enable-debug-symbols --with-amd64-max-phys-addr=0x200000000 --enable-debug-serial=0x3f8 --disable-dmap --disable-drt

    Both Windows 10 and Debian 11 boot and work correctly with the above configuration. I am able to interact with GRUB 2 after XMHF is loaded and boot the desired OS. All applications and peripherals work as normal.

  2. ./configure --with-approot=hypapps/helloworld --with-target-subarch=amd64 --enable-debug-symbols --with-amd64-max-phys-addr=0x200000000 --enable-debug-serial=0x3f8 --enable-dmap --enable-drt

    I am able to boot into Debian 11 successfully, but the moment I try interacting with
    peripherals such as disk and/or network and sometimes the keyboard, there is a seg-fault and eventually kernel panics. I don’t see any new serial messages in the debug console, beyond the last message during bootup:

    CPU(0x05): WRMSR (MTRR) 0x000002ff 0x0000000000000c00 (old = 
    0x0000000000000000)  
    CPU(0x05): Update EPT memory types due to MTRR 
    Found in xcph table
    Found in xcph table  
    Found in xcph table
    Found in xcph table
    Found in xcph table
    Found in xcph table
    Found in xcph table
    Found in xcph table
    Found in xcph table
    Found in xcph table
    

    I am unable to try to boot into Windows 10 because the GRUB 2 menu does not recognize any key stroke after XMHF loads and presents the GRUB 2 menu the second time. My system is configured to boot into Debian 11 by default, so it always picks up Debian without recognizing any key-stroke after XMHF is loaded

It looks like enabling DRT and DMAP is somehow affecting the way peripherals are being handled. However I did not spot anything out of the ordinary on the serial logs during boot-up. The DRT happens successfully and XMHF loads successfully and even boots into the OS as I have mentioned above. Any thoughts? I am happy to run additional tests on my testbed if you want.

Thanks!

Hi @amitvasudevan ,

No worries, thanks for the update.

Unfortunately, given the serial output I am not sure what causes the bug. I will discuss with @superymk to see whether he has any ideas. Things may become tricky if our hardware cannot reproduce the bug.

In the mean time, could you please try only enabling one of DRT or DMAP?

  • --enable-drt --disable-dmap
  • --disable-drt --enable-dmap

Also, for the Linux seg-fault and kernel panic, does the error message from Linux look related to this bug? If convenient, can you also attach the error message from Linux?

Thank you,
Eric Li

I had a quick discussion with @superymk , and here are the results:

  1. @superymk said that he has access to a Haswell computer that may be able to reproduce this bug. But due to other deadlines, we do not have time to test on this computer now. Expecting to test a month later.
  2. I created the PR in early May. After that, a lot of changes happened in my branch, including some DMAP bug fixes by @superymk. Currently I don’t want to update the PR because your code review will be gone. I am hoping to create a new PR for these updates in the future. Could you please also try the latest version of my repo (branch xmhf64)? The building process is almost the same. The only difference is that XMHF code is in the repo root directory, not xmhf-64/.
  3. @superymk said that he tested XMHF previously on an HP 840 G1 Elitebook, Haswell, with DMAP, without DRT, a private hypapp, Ubuntu 20.04.3 x64. This configuration works well.