The documentation for installing the micro-hypervisor mentions ensuring that the GRUB entry is
outside the AUTOMAGIC KERNEL LIST
And then removing lines from /boot/grub/menu.lst
Is it possible to get an example of what this grub entry should look like? I’m confused what entries should remain, and what the entry for the default OS should look like?
After following the steps to downgrade GRUB, after a reboot I end up in a grub commandline (grub>)
Also, I am confused if I have not completely downgraded, as the header states
GNU GRUB version 2.02~beta2-36ubuntu3.28
Minimal BASH-like line editing is supported. For the first word, TAB lists possible command completions. anywhere else TAB lists possible device or file completions.
grub> _
From this command line, I can set the linux kernel requirements and boot into linux. However, I’m missing how to boot the hypervisor (it seems like my updates to /boot/grub/menu.lst are not occurring/being referenced after I downgraded GRUB).
set root=(hd0,1)
linux /vmlinux-4.4.236+ root=/dev/<host>-vg/root
initrd /initrd-4.4.236+
boot
I re-ran the steps to downgrade grub
sudo apt-get purge grub-pc grub-common
sudo apt-get install grub
sudo update-grub
grub-install /dev/sda
Now, I am getting a different behavior. I go to a screen that shows Ubuntu 16.04. However, I am then falling into a BusyBox v1.22.1 (Ubuntu 1:1.22.0-15ubuntu1.4) build-in shell (ash). (initramfs)
I assume that this means that the hypervisor is loading successfully? But for some reason the filesystem for the kernel is not being mounted?
Do I need to modify the GRUB entry for the Default OS?
When I run init I get the following alert (after it gives up waiting for root device).
ALERT! UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx does not exist. Dropping to a shell!
Where cat /proc/cmdline results in:
root=UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx ro splash
By changing the root=UUID=... to specify the device (i.e. root=/dev/...) I am able to successfully boot into linux. Is there a test function I can perform to verify I have successfully booted the hypervisor (i.e. is there a test hypervisor call I can make)?
Great @Cap! Would you mind updating the docs with your escapade just so we have it for others? Thanks!
I just realized that I was silently failing to mount the hypervisor. When I run my grub entry for the hypervisor I get Error 17: Cannot mount selected partition
Update: when I modify my grub entry, changing the rootnoverify to uuid to match the automagic entries, I get a screen printout with:
Boot from (hd0,0) ext2 xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
, bss=0x0 [Multiboot-module @ 0x1a200000, 0x200 bytes]
[Multiboot-module @ 0x1a201000, 0xce40 bytes]
Starting up ...
However, I does not make any progress from here.
Can you please post your serial debug console output? Please see the docs to setup serial debugging. Thanks!
@amitvasudevan, Sorry for the delay in posting this.
I think I have a configuration error in the BIOS. Here is the end of the output log:
ERR: VMXON disabled by feature control MSR (5)
FATAL ERROR: TXT not supported
I forgot that TXT is often disabled by default in the BIOS.
Ahh…I think the docs mention to turn it on? If not, would be great to add a note to that effect. Thanks @Cap!
The docs mention TXT (and TPM) needs to be enabled. I had skipped past this, and assumed they were enabled by default.
Now I see the grub menu pop up a second time. And can boot from there into the guest OS.
I still seem to be having issues with my grub menu.lst After booting the micro-hypervisor, grub is invoked a second time. However, I seem to experience errors when I attempt to make any keyboard inputs (change grub menu items). To work around this, I attempted to use the savedefault options discussed in the docs. However, I’m not sure that I have things configured properly.
I am attaching my serial log: new_start.log (87.6 KB)
and my grub menu.lst file: grub_menu.lst (5.6 KB)
Looking at the log it seems like the micro-hypervisor is being loaded recursively. Which version of GRUB are you using? Did you remember to downgrade to GRUB 1 per the documentation?
Also, I would suggest removing everything from your menu.lst except for timeout, Default OS and uberXMHF. Remove the savedefault lines and set timeout to 20. Then follow the procedure below:
restart the machine and when GRUB menu pops up for the first time, manually select uberXMHF and let the micro-hypervisor boot
the micro-hypervisor will load with serial messages appearing on your debug console.
At some point, when the micro-hypervisor finishes loading up it will start the guest and you will find the GRUB menu pop up again. This time manually select Default OS and let it boot. Keep watching for serial messages from the micro-hypervisor, you should see some E820 related messages and then a few accesses to CRx registers before the guest OS begins to boot.
Post the new serial log and menu.lst and we can go from there…
Thanks!
Ahh ok. I think this is a good step forward. At least we have the micro-hypervisor booting up and loading the guest. It seems like there is some unsupported RDMSR exception caused likely due to some virtualization kernel module.
You may need to boot into the vanilla OS (without the micro-hypervisor) and ensure kvm-intel.ko is not loaded (e.g., via module blacklisting). Also can you please post your Linux kernel command line parameters?
@amitvasudevan,
Thank you! I do have kvm kernel modules loaded (kvm.ko and kvm_intel.ko). I have blacklisted those, but am still haulting when attempting to boot the OS.
kernel command line parameters (from /proc/cmdline): cmdline.txt (27 Bytes)
How/where did you blacklist them? Can you give me the output of ls /etc/modprobe.d/?
Lets try adding the following to your kernel command line parameters (in addition to what you already have):
text nomodeset debug earlyprintk=serial,ttyS0,115200,keep loglevel=8 memblock=debug nmi_watchdog=0
Can you give this a whirl?
I added blacklist kvm to /etc/modprobe.d/blacklist.conf
my /etc/modprobe.d has:
amd64-microcode-blacklist.conf
blacklist.conf
blacklist-rare-network.conf
fbdev-blacklist.conf
mlx4.conf
blacklist-ath_pci.conf
blacklist-firewire.conf
blacklist-watchdog.conf
intel-microcode-blacklist.conf
blacklist-framebuffer.conf
dkms.conf
iwlwifi.conf
Ok add the following lines to /etc/modprobe.d/blacklist.conf (in addition to changing the command line as mentioned above):
blacklist kvm
blacklist kvm_intel
blacklist intel_rapl
Then boot the micro-hypervisor again and see what happens…
@amitvasudevan, thank you! adding blacklist intel_rapl allowed for a successful boot of the guest OS!
Great! Yeah the rapl driver is known to access (undocumented) power MSRs which is currently not supported by the micro-hypervisor.
I will go ahead and incorporate some of this into the documentation within the PR.
Thanks for testing this @Cap! Am glad it worked out!