Building Hypervisor/Linux OS Kernels

Cap · May 26, 2020, 4:45pm

Hello,

I am trying to build the x86_32 hypervisor. I have a couple of questions as I work through this process.

It appears that I need to install uberspark in order to verify the hypervisor? Can I install the hypervisor without this verification (i.e. to get a quick start?)
The documentation lists ubuntu 16.04 and linux 4.4.x as supported OSes. Does the subbranch matter (i.e. ubuntu 16.04.6 vs 16.04.1)?
Additionally, there is a requirement that the OS have CONFIG_X86_PAE=n this requires rebuilding the kernel. Does it matter where the source files come from? Should we use apt-get source or pull from a repo (e.g., kernel.org)

amitvasudevan · May 26, 2020, 5:14pm

You will need to install uberSpark to verify and build uberXMHF. You can however, skip the verification part and directly build the binary if you wish to.

It should not.

Use apt-get source. For example: apt-get source linux-image-$(uname -r) will grab the kernel sources for your installed system.

Also check to make sure you have all the required build dependencies installed before you attempt to build your kernel. For Ubuntu, this is described at the following URL:

https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel

Cap · May 27, 2020, 5:13pm

I believe that I have built a 4.4.x kernel without PAE (when I fgrep "CONFIG_X86_PAE" /boot/config-$(uname-r) it returns CONFIG_X86_PAE is not set.

For future users, is there a location where I should save the linux-*.deb files?

amitvasudevan · May 27, 2020, 7:12pm

Hey @Cap,

Perfect, yes it looks like your kernel build went off without any problems. I am thinking we should create a new repository for kernel .deb sources and binaries within github/uberspark, say github/uberspark/uberxmhf-linux-kernels.git. We could then host pre-built linux kernel images, sources and headers for specific kernel arch and versions we support leveraging git lfs. Your thoughts?

Cap · May 28, 2020, 1:12pm

I think it makes sense to have a separate storage location for these large files (400+ MB). As they are not required, but could be beneficial to have easy access to.

I currently have 5 deb files:

linux-firmware-image-4.4.219_i386
linux-headers-4.4.219_i386
linux-image-4.4.219_i386
linux-image-4.4.219-dbg_i386
linux-libc-dev_4.4.219_i386

Cap · October 27, 2020, 8:21pm

Here are my notes about building a 4.4.x kernel without PAE.

I started with the default Ubuntu 16.04 iso ( 32-bit PC (i386) server install image).
I installed the kernel build dependencies apt-get install libncurses5-dev flex bison openssl libssl-dev dkms libelf-dev libpci-dev libiberty-dev autoconf
I downloaded the source git clone git://kernel.ubuntu.com/ubuntu/ubuntu-xenial.git
I modified the .config to disable PAE (which set by default).
4a. Edited using make menuconfig. Under Processor type and features —>
4b. Changing the entry for High Memory Support (from 64G to off), which then allows removing PAE Support.
Copied a debian script cp debian/scripts/retpoline-extract-one scripts/ubuntu-retpoline-extract-one
build the kernel make -j4
kernel modules make modules_install
install make install
This default creates a very large initrd.img file, which causes an out of memory error upon boot. To reduce this, remove unnecessary kernel modules find /lib/modules/4.4.{version} -name *.ko -exec strip --strip-unneeded {} +
Update initrd.img: update-initramfs -c -k 4.4.{version}
Upon reboot, the new kernel will be loaded by grub (as it has a higher version than from the iso).

amitvasudevan · October 27, 2020, 8:26pm

Great stuff @Cap. Very Useful!

Were you also able to generate the .deb files as you mentioned in this post before? It would be good to have generated .deb available to have easy access to as you mentioned.

Cap · October 28, 2020, 12:18am

@amitvasudevan,
I do not have all of the *.deb files mentioned in the prior post. However, I did generate new linux-headers-4.4.236+_i386.deb and linux-image-4.4.236+_i386.deb.

Is the plan still to create a new repo for these?

amitvasudevan · October 28, 2020, 12:38am

Is the firmware, libc and dbg required? I do remember firmware being used for DTBs at least on the Pi3, but am not sure if there is something analogous on x86. I have not used the libc and dbg versions before.

Yes, I will create the repo and post back with the organization I have in my mind

amitvasudevan · October 28, 2020, 12:42am

This URL: https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel

also talks about a linux-headers-*_all.deb in addition to the linux-header-*_i386.deb. Do you know the difference between the two and if we need both? I just want to make sure that we are generating the correct set of .debs that someone could install via:

sudo dpkg -i linux*.deb
sudo reboot

Thanks!

amitvasudevan · October 28, 2020, 1:08am

Ok. I have created the following repository to house the Linux kernels for überXMHF:

https://github.com/uberspark/uberxmhf-linux-kernels.git

I am thinking of the following organization for now:

/README.md --> topl-level readme; please edit this to add a meaningful description
  ubuntu/README.rst --> ubuntu distribution readme; describes how to build from sources and instructions to install
     12.04.LTS/
        x86_32/ 
           v4.4.x --> will have the .deb files and gzipped tarball of sources used to build the tarball and the config file
     16.04.LTS/
        x86_32/  
           v4.4.x --> same as above
   raspbian/README.rst --> raspbian distribution readme; describes how to build from sources and instructions to install
      jessie-lite/ 
         armv8_32/
           v4.4.x --> same as above

Let me know your thoughts. Also what are the sizes of these .debs and source tarball approx.? Just want to make sure we stay within github limits.

Thanks @Cap. This is very useful!

Cap · October 28, 2020, 5:46am

This structure makes sense. I made an initial PR based upon this, please feel free and comment.

As I didn’t modify the source, I just left a pointer to the git repo for the source files
(in the README.rst).

As for the size of the .deb files, the linux-headers are ~8MB, and the linux-image is ~45 MB.

amitvasudevan · October 31, 2020, 4:32pm

5 posts were split to a new topic: überXMHF (x86) GRUB boot-up configuration

Cap · November 2, 2020, 10:53pm

As a note, likely need to strip the kernel after installing new kernel (otherwise the initrd.img-<version> might be too large).

This can be performed by using the following steps (reference Stack Exchange Post ):

cd /lib/modules/<version>
sudo find . -name *.ko -exec strip --strip-unneeded {} +
sudo update-initramfs -c -k <version>

amitvasudevan · November 2, 2020, 10:55pm

@Cap. Can you update the PR with these changes? Thanks!

amitvasudevan · November 12, 2020, 4:12pm

I have merged the PRs onto master within uberspark/uberxmhf-linux-kernels.git

Closing topic thread. Thanks @Cap

Cap · November 19, 2020, 5:09pm

So on x86, I think that one of the kernel options should be changed. Currently, the default memory split is selected (3/1, user/kernel). However, due to highmem being disabled, this results in little physical memory being available (<800M, shown when running free, despite 8G being installed). Changing this configuration option to 1/3, user/kernel shows more physical memory (>2.6G).

amitvasudevan · November 19, 2020, 5:10pm

Ahh. Do you want to add a PR to this effect?

Thanks @Cap!

Cap · November 23, 2020, 8:04pm

PR submitted, setting the kernel/user split to be 2/2. This seems to limit the available memory (as reported by free) to 2G; however, when I used a 3/1 split I experienced errors running user programs (e.g., for JVM to initialize it required providing memory limits smaller than the defaults). This split of 2/2 seems to provide a trade-off of available memory and usable for many user applications.