überXMHF (rpi3-armv8_32): Convert into a üobject collection with a single üobject

This task will convert the existing rpi-armv8_32 version of the überXMHF microhypervisor into a single üobject collection.

Linked PRs and merges:

Are there staging bridges to support building uobjects on rpi3?
If not what software tools are needed for the cc, ld, and as bridges? Looking at the Makefile for uberxmhf-rpi3, I am assuming:

  • arm-linux-gnueabihf-{gcc, as, ar, ld, objcopy}

Looking at the file structure for the bridges, where there is a top-level for the container architecture and a sub-level for the uberpsark supported architecgture what is the desired setup (i.e. amd64/armv7 or armv7/armv7)?

@Cap, we need to add staging bridges to build on rpi3.

I have created a separate task for this and am linking it below:

https://forums.uberspark.org/t/uberspark-add-uberspark-compiler-assembler-linker-bridges-for-gcc-gnu-as-and-gnu-ld-amd64-armv8-32/114/2

I have assigned this to you. Let me know if you have any questions on that task thread. Once we finish the aforementioned task, we can bump back here to move on with this one…

Thanks!

Looking at the creation of an uberspark manifest for uberxmhf-rpi3, I have a couple questions.

  1. How is the namespace identified. I’m currently assuming uberspark/uobjcoll/generic/uberxmhf-rpi3
  2. Would the platform be rpi3?
  3. For cpu, this was generic in the bridges. Should it be specific to the Raspberry Pi 3 cpu (cortex A53)?
  4. Is a hpl different from any required?
  5. Is a sentinel other than call required?
  6. We are planning on placing this into a single uobj. Would this templar be: uberspark/uobjcoll/generic/uberxmhf-rpi3/uxmhf-rpi3?
  7. With a public method of "uxmhf-rpi3":["call"]?

Hey @Cap,

My answers to your questions are as below:

  1. How is the namespace identified. I’m currently assuming uberspark/uobjcoll/generic/uberxmhf-rpi3

Every uobjcoll namespace is required to begin with uberspark/uobjcoll/
What comes after the aforementioned prefix is something that is still up for discussion. I am thinking we should have at least two top-level folders generic (to house platform independent uobjcolls) and platform (to house platform dependent uobjcolls). Within platform we can currently start with another folder rpi3 which is Raspberry Pi 3. Of course one can envision having other platforms such as pc, rpi4 etc.

So circling back to your question. I would say use the uberspark/uobjcoll/platform/rpi3/uxmhf namespace for now.

  1. Would the platform be rpi3 ?

Yes, and arch will be armv8_32

  1. For cpu, this was generic in the bridges. Should it be specific to the Raspberry Pi 3 cpu (cortex A53)?

cpu can be generic for now since we don’t really make use of cortex-a53 specific extensions within the micro-hypervisor source yet.

  1. Is a hpl different from any required?

Yes. Let us make hpl as hyp to designate hyp mode privileges that are required for this collection.

  1. Is a sentinel other than call required?

Nope. we will stick with the call sentinel for this uobjcollection for now.

  1. We are planning on placing this into a single uobj. Would this templar be: uberspark/uobjcoll/generic/uberxmhf-rpi3/uxmhf-rpi3 ?

Going with the previous recommendation, it would be:
uberspark/uobjcoll/platform/rpi3/uxmhf/main

  1. With a public method of "uxmhf-rpi3":["call"] ?

Would be public method: “main” : [“call”]

I would recommend as a first step to put all the sources, including library sources that the micro-hypervisor relies upon within main uobj and all the headers within main/include folder when you are attempting to build this initial uobjcoll.

We can then pare off the supporting libraries as uobjrtl (run-time libraries) when the time comes.

Also, I am thinking it might be a good idea to propagate some of the high-level namespace, platform, cpu, arch selection criterion into the documentation as you go through with this excercise.

Thanks!

@amitvasudevan, Thanks!

I will try and keep notes and track potential feedback into the docs.

As I start moving the source files into the uobj directory. Do I keep the existing structure (i.e. grouping inside directories such as libxmhfc), or do I simply place all the files in the main directory and all the headers in main/include?

When there are multiple files in the sources of the uobj manifest, how are the individual files separated (i.e. ''file1.c'', ''file2.c'', ... or ''file1.c, file2.c, ...")?

If there are directories within (e.g., main/include) are these referenced by name or by path (i.e. arm8-32.h or include/arm8-32.h)?

I will try and keep notes and track potential feedback into the docs.

Thanks @Cap

As I start moving the source files into the uobj directory. Do I keep the existing structure (i.e. grouping inside directories such as libxmhfc), or do I simply place all the files in the main directory and all the headers in main/include?

You can try to group them for sure. e.g., current uxmhf-rpi3/common/* --> main/common/*, uxmhf-rpi3/core/* --> main/core/*

Leave out uxmhf-rpi3/bootstrap since that will be replaced by a loader.

When there are multiple files in the sources of the uobj manifest, how are the individual files separated (i.e. ''file1.c'', ''file2.c'', ... or ''file1.c, file2.c, ..." )?

“c-files” : [ “core/xxx.c”, “common/yyy.c”, “libxmhfc/zzz.c”]

See: https://docs.uberspark.org/nextgen-toolkit/reference/manifest.html#uberspark-uobj-manifest-node

If there are directories within (e.g., main/include ) are these referenced by name or by path (i.e. arm8-32.h or include/arm8-32.h )?

All files that go into main/include should be listed within the manifest in the h-files node. For example:

“h-files” : [ “include/arm8-32.h”, “include/types.h”, etc. ]

The headers will then be accessible through the: uberspark/uobjcoll/platform/rpi3/uxmhf/include namespace within the uobject c sources.

So for example,
#include <arm8-32.h> within the code should be changed to
#include <uberspark/uobjcoll/platform/rpi3/uxmhf/include/arm8-32.h>

Finally, you will need to include the top-level uberspark.h header file within each source file:

#include <uberspark/include/uberspark.h>

Feel free to let me know if you need any further clarifications. Thanks for your help with this!

Thanks!

Now, looking at refactoring the Makefile.

uxmhf-rpi3 utilizes multiple Makefiles (some generating libraries). How do I account for this in the uberspark build framework?

Is there documentation detailing what the uberspark framework is performing when running uberspark uobjcoll build?

I would recommend as a first step to put all the sources, including library sources that the micro-hypervisor relies upon within main uobj and all the headers within main/include folder when you are attempting to build this initial uobjcoll

@Cap, you will need to move all the sources and headers referenced by all the Makefiles into the respective uobjcoll source/include folder as I had mentioned before. After that exercise, you should no longer need the original Makefiles. uberspark uobjcoll build should read the manifest and build the uobjcoll binary with all the sources specified.

OK, cool.

I’m getting an error when I run uberspark uobjcoll build, stating that a file it is looking for does not exist (uberspark/staging/rpi3/uberspark/sentinels/cpu/armv8_32/generic/any/interuobjcoll/call/uberspark.json)

I copied the uberspark.json(s) from uberspark/staging/rpi3/uberspark/sentinels/cpu/x86_32/generic/any/* and was able to make additional progress. Is this something that uberspark should have automatically done (or did I miss a preparatory step)?

However, now I am receiving an internal uberspark error. After created uobj collection uobjs public methods hashtable and association list with addresses
I get:
uberspark: internal error, uncaught exception: Not_found
And make reports an Error 125

We need to add call sentinels for the armv8_32 architecture, so this was expected :slight_smile:

However, now I am receiving an internal uberspark error. After created uobj collection uobjs public methods hashtable and association list with addresses
I get:
uberspark: internal error, uncaught exception: Not_found
And make reports an Error 125

Hmmm, not sure about this one. Can you maybe share your uberspark/uberspark.git and uberspark/uberxmhf.git fork branches so I can take a look and attempt to make a fix?

Thanks!

Sure thing. I have added you as a collaborator on my forks.

the uberxmhf branch is feature-uberobjcol, and the uberspark is feature-rpi3sentinels.

Heya @Cap,

the uberxmhf branch is feature-uberobjcol ,

Am able to clone your fork and switch to the branch, but don’t seem to see the uobjcoll folder within uxmhf-rpi3/. Am I missing something?

Thanks!

Nevermind @Cap. A git fetch -a worked. Thanks!

@Cap, I have pushed some changes to both branches. Please pull and give it a whirl on your side. Thanks!

@amitvasudevan, Thank you!

As a note, the header locations needed to be slightly modified. From #include <uberspark/uobjcoll/platform/rpi3/uxmhf/include/arm8-32.h> to #include <uberspark/uobjcoll/platform/rpi3/uxmhf/main/include/arm8-32.h> where the main directory needed to be added.

Also, I am currently getting an error of a conflict in types for <int/uint>_fast8_t between the stdint.h in uberxmhf-rpi3 and arm-linux-gnueabihf

Can you try adding -nostdinc to the “params” JSON node of the compiler bridge manifest?

e.g., “params” : [ “-nostdinc” ]

I receive an error. I think that uberspark.h includes stdint.h, and others that are not found when run with the -nostdinc param.

When I edited libxmhfc/stdint.h to change the typedef from int32_t to int8_t, I was able to get it to make progress. However, now I am getting a conflicting type error for the c functions (e.g., memset)

When I edited libxmhfc/stdint.h to change the typedef from int32_t to int8_t , I was able to get it to make progress

We should not be doing this.

I think that uberspark.h includes stdint.h

I think it might be time for us to take on the task of creating a uobjrtl for the C runtime library within the main uberspark framework :slight_smile:

Let us branch this off as a separate task.

Would you mind sending a PR in for the rpi3-sentinels branch of uberspark/uberspark.git repo first? I would like to merge that changeset on develop before beginning the uobjrtl task.

Thanks!