Hypercalls in kernel modules

uberXMHF uobject Collection Rich-Guest OS
1

I am using uxmhf-rpi3 and attempting to make a hypercall from a linux kernel module. For userspace applications, I would normally link to libuhcall.a (as is done in the rgapp examples).

However, when I attempt to add this library into my Makefile for the kernel module, I receive a warning that the kernel module needs unknown symbol uhcall where uhcall is defined in libuhcall.a.

Can a hypercall be made from kernel space using libuhcall? Does it need to be linked differently for a kernel module?

2

Alternatively, when I include libuhcall.a in the source files for the kernel module, I get a different linking error that uhcall.o uses VFP register arguments while the kernel module does not. This causes the linking to fail.

3

Hey @Cap,

You cannot used libuhcall for kernel-mode drivers. You will have to invoke the HVC instruction directly. This logic is embedded within the example uhcallkmod example in the rgapps folder – take a look. We should probably make a kernel mode library at some point…

4

OK. For the VA to PA mapping, can I use virt_to_phys() (from asm/io.h)? Or is it not safe to assume the kernel has kmalloc’d the memory where of uhcall_buff in the kernel module?

5

@Cap, you would need to use the following kernel functions to accomplish buffer allocation before using the HVC instruction via something like uhcallkmod_hvc:

#include <linux/mm.h>
#include <linux/highmem.h>

struct page *k_page;

k_page = alloc_page(GFP_KERNEL | __GFP_ZERO);

At this point you can get the virtual address of this page via:

page_address(k_page);

You can then typecast this VA to whatever data structure you want to operate on within the kernel.

Then when you are ready to issue the hypercall (via the HVC instruction) pass the physical address of the buffer via:

page_to_phys(k_page);

If you are using uhcallkmod_hvc as your function to issue the HVC instruction, then the buffer parameter to that function should be the physical address as you obtained above using page_to_phys

Don’t forget to use __free_page(k_page) once you are done with using the page buffer.

Let me know if you run into any issues.

Coming to think of it we should make a libkhcall (the kernel mode dual of libuhcall) that has similar functionality. But we can get to that once you have this working.

Hope this helps!

6

@amitvasudevan, thanks!

To make sure I am understanding the changes between a user-space hypercall and a kernel-space hypercall.

In user-space, I would create/place the data buffer for the hypercall in a global variable:
__attribute__((aligned(4096))) __attribute__((section(".data"))) uhcalltest_param_t uhctp;

Whereas, in kernel-space I should instead use alloc_page()?

Would the libkhcall have a va2pa(), or would is assume a pa is being passed (as uhcallkmod_hvc() currently does?

7

Yes, or you can do a mmap to get a 4K aligned buffer dynamically in user mode.

Yes

You can use page_to_phys as described above which is essentially a va2pa implementation in kernel mode.

8

What is being passed to uhcallkmod_hvc(u32 uhcall_function, void *uhcall_buffer, u32 uhcall_buffer_len) for the pa (uhcall_buffer)? It seems that in uhcall.c the pa is determined for the data struct being passed to the hypercall. Then this value is cast as a pointer (uhcallp.uhcall_buffer=(void *)(uint32_t)uhcall_buffer_paddr) and given to uhcallkmod_hvc()?

9

@Cap,

What is being passed to uhcallkmod_hvc(u32 uhcall_function, void *uhcall_buffer, u32 uhcall_buffer_len) for the pa ( uhcall_buffer )?

In user mode this would be accomplished via va2pa of a page-aligned data variable. In kernel mode, you need to use page_to_phys on a page-aligned kernel data variable (which alloc_page grabs for you).

10

As a note. #include <linux/mm.h> requires a few additional dependencies to define constants.
I also needed the following:
#include <linux/kernel.h> for MAX_SIZE
#include <linux/numa.h> for NUMA_NO_NODE
#include <linux/gfp.h> for __GFP_ZERO

11

I created a kernel library based upon this. Please see PR: https://github.com/uberspark/uberxmhf/pull/7

12

Hey @Cap,

Sorry for the delay in my response. Been swamped at work. So can you just summarize what this library functionality provides and how you envision this to be used in general?

Thanks!

13

This is for the uberxmhf-rpi3.

The library provides an interface for making hypercalls from kernel modules. It is similar to the existing user space library (libuhcall), except it is design to be integrated into kernel modules.

It takes the hypercall inputs, copys them to a new kernel page, sends this to the hypervisor, then returns the new values to the input hypercall memory location (and frees the kernel page).

14

Ahh nice :). Do you mind updating the current docs to explain the usage of libuhcall and libkhcall? I have noted this in the PR as well. Thanks for the contribution!

15

I added a note to that effect.

16

Hey @Cap,

Can you revise the PR so that the new documentation you added to docs/rpi3-cortex_a53-armv8_32\build.rst actually pops one level of indentation?

he libraries blurb appears as a sub bullet of the outer “Note” bullet, but should be at the same level as the “Note” bullet.

Also the inline literal uhcall() should have spacing between the end of literal `` and the word for following it. This gives currently gives a warning during the documentation build and the resulting documentation becomes a little weird at that point of rendering.

You can check with:

make clean
make docs_html

within the docs/ folder.

Thanks!

17

PR merged and linked below. Closing thread. Thanks@Cap

18