Building x86 uberapp

Is there a good example uberapp on the x86 uberXMHF to use as a baseline for building an x86 uberapp (similar to uhcalltest on the rpi3-uberXMHF)?

Linked PR(s):


There are a few uapps like approvexec, sstep etc. That should hopefully get you started. But perhaps you can add a port of the uhcalltest to x86 if you can. I believe its just swapping out the libuhcall with x86 implementation which is vmcall instruction as shown in the aforementioned uapps.


OK. I’ll look into this.

Is it worth making this type of library for x86? I guess that would entail also adding the corresponding kernel module. I’ll start first without this and just directly include the vmcall instruction (and the va2pa, and mlock calls).

As I start looking at __vmcall() there appear to be different versions (e.g., rgapp-steptrace takes 3 arguments, and rgapp-syscallog takes 4 arguments).

It seems to me that in general, eax is the ID for the hypercall, ebx often seems to be 0, and edx is the physical address of the memory buffer being sent to the hypervisor (this holds for all but rgapp-syscallog).

A related question. It is not clear to me where to integrate the functionality into the core hypervisor? Or what is the parallel to the uapp-* in the rapsberry pi 3 version (and its listing in core/ghcall.c)?

Hey @cap,

My responses inline:

You are correct in this observation. The unified __vmcall() interface should take in 4 parameters and its up to the üApp to choose what it wants to pass through those general purpose registers.

Yes. keep eax as hypercall ID. ebx is the high order 32-bit of the physical address and edx is the low-order 32-bit of the physical address. ecx can be anything based on the üApp and hypercall.

The entire hypercall orchestration begins within the xcihub_icptvmcall function located in uxmhf/xmhf-uobjs/xc_ihub/xcihub_icptvmcall.c. This in turn calls the function xc_hcbinvoke_helper within uxmhf/xmhf-uobjs/xc_ihub/xcihub_hcbinvoke.c that is responsible for cycling through all the registered üApp hypercall handlers based on the static registration variable _xcihub_hypapp_info_table which is defined within uxmhf/xmhf-uobjs/include/xc.h.

Execution then flows to slab_main within the corresponding üApp extension (e.g., xh_aprvexec) which then dispatches control to the locally defined _hcb_hypercall function that processes the hypercall.

Yes, you can start this without making the library. But it would be beneficial to have the library eventually so that other üApps can benefit from it.


@amitvasudevan, thank you! This is very helpful!

I have made an initial version, but when I attempt to execute it, I am getting an error when I attempt to make the hypercall: Illegal instruction (core dumped)

Can you give some pointers on how to begin debugging this error. As a note, I get a similar error when trying to run existing rgapps.

Are you sure the micro-hypervisor is running underneath? Hook up a serial cable per the debugging documentation to check and see if you see boot-up messages.

@amitvasudevan, I am receiving debug messages over the serial port (from the boot up). However, I do not see any debug messages when I attempt to run my test uberapp (I experience similar results from legacy ones as well).

Hmm…what interface are you using to debug print? Also, perhaps the guest OS is loading up a serial driver that is messing with the baud-rate etc. Can you try, re-initializing the UART from within your üapp before doing a debug print?

The default serial port (0x3f8) ttyS0

My uberapp is crashing when I attempt to make the __vmcall(), at the asm volatile (.

Do any of the existing üapps exhibit a similar behavior, e.g., aprvexec?

yes, all the existing ones generate the same error at the same instruction (beginning of asm volatile in __vmcall())

Is there a gcc version dependency?

I’ve been working with gcc 5.4.0

Can you post the error message?

Hmm, this could be the issue. You need to be using compcert v3.1. I am assuming you are using the latest version of uberXMHF x86 (based on uberSpark) correct?

The only output generated when I attempt to run any of the uberapps is:
Illegal instruction (core dumped)
Then it falls back into the shell.
When I run it in gdb, I get the following additional details:

Program received signal SIGILL, Illegal instruction.
0x0804a1ed in __vmcall (edx=0, ebx=0, eax=10) at rgapp-uhcalltest.c:35
35           asm volatile (

Perhaps I am confused about how the guest/test function needs to be compiled. So I have just used the associated Makefile within the rgapp-* folder, and built these locally.

I am using the develop branch of uberXMHF x86 (and building the core micro-hypervisor with uberSpark).

Ok that is the right approach.

My bad, I was confused about your previous questions. You can build the files in the rgapp folder with any compiler. So the version of gcc you are using should be fine.

Are you sure the micro-hypervisor is running? Can you post the micro-hypervisor boot-up serial log?


Please find the boot-up serial log. x86_startup.txt (24.4 KB)

Looking at the log it seems like the micro-hypervisor is restarting during startup due to some failure. Is that the full log? The last line I am seeing is: gp_s5_setupcpustate[6]: Reloaded segment registers

that is the full log.

Is hardware virtualization enabled in the BIOS and how many cores are enabled? Can you ensure hyper-threading is enabled? Also please boot into the vanilla OS (without hypervisor) and post the output of cat /proc/cpuinfo once you have enabled hyperthreading.