Booting with grub

pa4wdh · July 3, 2021, 5:23pm

Hi All,

During my experiments with u-boot i noticed that it is able to boot an EFI binary, which made me wonder if i could actually use u-boot to start grub. I usually like grub because it allows me to choose between different kernel versions and configurations.

I started my experiment with plain grub 2.06. It compiles fine for RISC-V. Actual installation in /boot with grub-install requires the use for the --force parameter because it can’t find the efibootmgr, which is ok because it’s not an actual UEFI platform.

I interrupted u-boot in it’s boot sequence and was able to start grub with these commands (nvme 0:3 is my /boot):

pci enum
nvme scan
load nvme 0:3 $kernel_addr_r /EFI/gentoo/grubriscv64.efi
bootefi $kernel_addr_r

After that grub showed it’s usual prompt. It is even able to access my SD card and my SSD. Now here’s the catch: This version doesn’t support the “linux” command (required to load the linux kernel) on RISC-V

Then i found this thread on the grub-devel maillinglist:
https://lists.gnu.org/archive/html/grub-devel/2021-03/msg00343.html
These patches implement just the linux command for RISC-V. I added the patches to my grub and it’s partially succesful.

If i just type the regular linux command including all the parameters listed in extlinux.conf the kernel actually boots. Unfortunately a lot of hardware doesn’t work (including the SSD, the SD card does work), i think because of a missing devicetree. Grub has support for devicetrees, but if i load a devicetree i only get this message:

EFI stub: ERROR: /chosen/boot-hartid missing or invalid!

It seems that u-boot adds this to the devicetree and grub doesn’t. As far as i know the boot hart is selected at random during boot, is it possible to set it to a fixed value so i can add it as a fixed entry in the devicetree?

pa4wdh · July 3, 2021, 6:23pm

Patching grub to simply add the entry to the devicetree was actually simpler than i expected

This is the patch:

diff -Naur grub-2.06/grub-core/loader/riscv64/linux.c grub-2.06-mod/grub-core/loader/riscv64/linux.c
--- grub-2.06/grub-core/loader/riscv64/linux.c	2021-07-03 19:45:08.797409969 +0200
+++ grub-2.06-mod/grub-core/loader/riscv64/linux.c	2021-07-03 19:43:54.673239976 +0200
@@ -48,6 +48,13 @@
 static grub_addr_t initrd_start;
 static grub_addr_t initrd_end;
 
+#define get_hartid(__v)                        \
+{                                      \
+  __asm__ __volatile__ ("mv %0, tp"    \
+                       : "=r" (__v) :  \
+                       : "memory");    \
+}
+
 grub_err_t
 grub_arch_efi_linux_check_image (struct linux_arch_kernel_header * lh)
 {
@@ -68,6 +75,7 @@
 finalize_params_linux (void)
 {
   int node, retval;
+  int hartid;
 
   void *fdt;
 
@@ -83,6 +91,9 @@
   if (node < 1)
     goto failure;
 
+  get_hartid(hartid);
+  grub_fdt_set_prop32 (fdt, node, "boot-hartid", hartid);
+
   /* Set initrd info */
   if (initrd_start && initrd_end > initrd_start)
     {

After that the kernel booted … and immediately oops’ed when the initscripts tried to mount eftvarsfs. After i removed that from the initscrips it worked . My system is now running after being booted via grub.

Now the next question is how to instruct u-boot to execute the required commands to start grub automatically, does anyone have a clue? I tried to add it to extlinux.conf with it’s own label but that doesn’t work.

pa4wdh · July 4, 2021, 10:51am

It seems that whatever u-boot does at boot time is hardcoded in include/config_distro_bootcmd.h

If first tries to boot from all partitions with extlinux.conf, and only if that fails it tries a EFI boot. So i’d have to remove extlinux.conf from both the SD card and the SSD before it would try that. I made a small patch to change that order to: first try EFI boot, then try with extlinux.conf. And it works . My unmatched now automatically boots to grub which presents me with the usual menu over the serial console.

In case anyone is interested, this is the patch:

diff -Naur u-boot-2021.01_rc5/include/config_distro_bootcmd.h u-boot-2021.01_rc5-mod/include/config_distro_bootcmd.h                                            
--- u-boot-2021.01_rc5/include/config_distro_bootcmd.h  2021-07-04 11:32:47.9783
21442 +0200                                                                     
+++ u-boot-2021.01_rc5-mod/include/config_distro_bootcmd.h      2021-07-04 11:34
:05.930534226 +0200                                                             
@@ -462,11 +462,11 @@                                                           
        "scan_dev_for_boot="                                              \     
                "echo Scanning ${devtype} "                               \     
                                "${devnum}:${distro_bootpart}...; "       \     
+               SCAN_DEV_FOR_EFI                                          \     
                "for prefix in ${boot_prefixes}; do "                     \     
                        "run scan_dev_for_extlinux; "                     \     
                        "run scan_dev_for_scripts; "                      \     
                "done;"                                                   \     
-               SCAN_DEV_FOR_EFI                                          \     
                "\0"                                                      \     
        \                                                                       
        "scan_dev_for_boot_part="                                         \

xypron · July 6, 2021, 7:20am

Hi,

I am the maintainer of the UEFI sub-system in U-Boot.

When OpenSBI calls U-Boot it transfers the boot hart ID in register a0 to U-Boot. This value is stored in gd->arch.boot_hart.

When the bootefi command is executed it fixes up the device-tree and sets the /chose/boot-hartid property in the device-tree.

The current patch series for GRUB to enable booting RISC-V booting into Linux is available here: [PATCH v2 0/7] Add LoadFile2 and riscv Linux loader. There is a pending Linux patch needed: [PATCH 1/1] RISC-V: load initrd wherever it fits into memory - Heinrich Schuchardt

Best regards

Heinrich

X512 · July 6, 2021, 12:19pm

I found that U-Boot UEFI provide invalid memory map that reports PMP-protected machine-mode memory as EfiBootServicesData that is available to general use after exiting boot services according to UEFI specification. Attempt to use PMP-protected memory in supervisor and user mode will cause CPU exception. I am not sure what exact memory type should be used, probably EfiReservedMemoryType, maybe that address range should be excluded from UEFI memory map.

FDT reserved memory entry:

node('reserved-memory')
    prop('#address-cells'): 2 (len 4)
    prop('#size-cells'): 2 (len 4)
    prop('ranges'):
       (len 0)
    node('mmode_resv0@80000000')
      prop('reg'): (0x80000000, 0x00040000) (len 16)

UEFI memory map:

  phys: 0x80000000, virt: 0x80000000, size: 0x40000, bootServicesData, attrs: 0x8
  phys: 0x80040000, virt: 0x80040000, size: 0x7eb6000, conventionalMemory, attrs: 0x8
  phys: 0x87ef6000, virt: 0x87ef6000, size: 0x14000, ACPIReclaimMemory, attrs: 0x8
  phys: 0x87f0a000, virt: 0x87f0a000, size: 0x741cb000, conventionalMemory, attrs: 0x8
  phys: 0xfc0d5000, virt: 0xfc0d5000, size: 0x25e9000, loaderData, attrs: 0x8
  phys: 0xfe6be000, virt: 0xfe6be000, size: 0x59000, loaderCode, attrs: 0x8
  phys: 0xfe717000, virt: 0xfe717000, size: 0x7000, reservedMemoryType, attrs: 0x8
  phys: 0xfe71e000, virt: 0xfe71e000, size: 0x1000, bootServicesData, attrs: 0x8
  phys: 0xfe71f000, virt: 0xfe71f000, size: 0x1000, runtimeServicesData, attrs: 0x8000000000000008
  phys: 0xfe720000, virt: 0xfe720000, size: 0x2000, bootServicesData, attrs: 0x8
  phys: 0xfe722000, virt: 0xfe722000, size: 0x1000, reservedMemoryType, attrs: 0x8
  phys: 0xfe723000, virt: 0xfe723000, size: 0x3000, runtimeServicesData, attrs: 0x8000000000000008
  phys: 0xfe726000, virt: 0xfe726000, size: 0x1000, bootServicesData, attrs: 0x8
  phys: 0xfe727000, virt: 0xfe727000, size: 0x4000, runtimeServicesData, attrs: 0x8000000000000008
  phys: 0xfe72b000, virt: 0xfe72b000, size: 0x1000, reservedMemoryType, attrs: 0x8
  phys: 0xfe72c000, virt: 0xfe72c000, size: 0x1000, bootServicesData, attrs: 0x8
  phys: 0xfe72d000, virt: 0xfe72d000, size: 0x1000, reservedMemoryType, attrs: 0x8
  phys: 0xfe72e000, virt: 0xfe72e000, size: 0x2000, bootServicesData, attrs: 0x8
  phys: 0xfe730000, virt: 0xfe730000, size: 0x2000, reservedMemoryType, attrs: 0x8
  phys: 0xfe732000, virt: 0xfe732000, size: 0x1000, bootServicesData, attrs: 0x8
  phys: 0xfe733000, virt: 0xfe733000, size: 0x1000, reservedMemoryType, attrs: 0x8
  phys: 0xfe734000, virt: 0xfe734000, size: 0x1000, bootServicesData, attrs: 0x8
  phys: 0xfe735000, virt: 0xfe735000, size: 0x1000, reservedMemoryType, attrs: 0x8
  phys: 0xfe736000, virt: 0xfe736000, size: 0x1000, bootServicesData, attrs: 0x8
  phys: 0xfe737000, virt: 0xfe737000, size: 0x1000, reservedMemoryType, attrs: 0x8
  phys: 0xfe738000, virt: 0xfe738000, size: 0x2000, bootServicesData, attrs: 0x8
  phys: 0xfe73a000, virt: 0xfe73a000, size: 0x1000, reservedMemoryType, attrs: 0x8
  phys: 0xfe73b000, virt: 0xfe73b000, size: 0x2000, bootServicesData, attrs: 0x8
  phys: 0xfe73d000, virt: 0xfe73d000, size: 0x1826000, loaderData, attrs: 0x8
  phys: 0xfff63000, virt: 0xfff63000, size: 0x1000, runtimeServicesCode, attrs: 0x8000000000000008
  phys: 0xfff64000, virt: 0xfff64000, size: 0x9c000, loaderData, attrs: 0x8
  phys: 0x100000000, virt: 0x100000000, size: 0x380000000, bootServicesData, attrs: 0x8

Where I can report bug?

xypron · July 6, 2021, 12:46pm

@x512:

Please, have a look at devicetree-specification/chapter3-devicenodes.rst at master · devicetree-org/devicetree-specification · GitHub :

" Reserved regions with the no-map property must be listed in the memory map with type EfiReservedMemoryType . All other reserved regions must be listed with type EfiBootServicesData ."

U-Boot is doing exactly this.

It is Linux (or Haiku OS) that will know that it cannot use reserved-memory unrestrictedly. If you want to avoid speculative access, you should use the no-map property in the device-tree.

Best regards

Heinrich

X512 · July 6, 2021, 12:51pm

But it is listed as EfiBootServicesData:

phys: 0x80000000, virt: 0x80000000, size: 0x40000, bootServicesData, attrs: 0x8

Currently there are no way to identify what memory blocks with EfiBootServicesData can be used by OS without reading FDT. FDT is not mentioned in UEFI specification. Provided memory map is taken from physical HiFive Unmatched board.

UEFI spec says:

Table 30. Memory Type Usage after ExitBootServices()
...
EfiBootServicesData - Memory available for general use.

Most of free RAM is also listed as EfiBootServicesData so ignoring it is a bad idea:

phys: 0x100000000, virt: 0x100000000, size: 0x380000000, bootServicesData, attrs: 0x8

X512 · July 6, 2021, 1:07pm

There are no no-map property in FDT DTB table provided by UEFI (GUID: {B1B621D5-F19C-41A5-830B-D9152C69AAE0}). I use standard SD card provided with board without any modifications. UEFI boot loader is located at USB flash drive.

  node('reserved-memory')
    prop('#address-cells'): 2 (len 4)
    prop('#size-cells'): 2 (len 4)
    prop('ranges'):
       (len 0)
    node('mmode_resv0@80000000')
      prop('reg'): (0x80000000, 0x00040000) (len 16)

pa4wdh · July 8, 2021, 5:50pm

Thanks for joining @xypron.

I am aware there is a more recent patchset to add RISC-V support for the linux command to grub. From the replies to that patchset i got the impression it doesn’t work yet, so that why i’m still using the old patches.
If i’m not mistaken the new patchset also doesn’t set the /chosen/boot-hartid property in the devicetree.

xypron · July 9, 2021, 12:06pm

/chosen/boot-hartid is set by U-Boot not by GRUB.

arch/riscv/lib/fdt_fixup.c:161: err = fdt_setprop_u32(blob, chosen_offset, “boot-hartid”,
arch/riscv/lib/fdt_fixup.c-162- gd->arch.boot_hart);

pa4wdh · July 11, 2021, 10:32am

I know it’s set by u-boot, i’ve seen the code
My point is that grub also has the ability to load a devicetree, and i think that means it should also add the boot-hartid in case it’s not present.

xypron · July 12, 2021, 7:35am

Updating the boot hart is not the only dtb fixup that is needed. I have implemented in U-Boot a protocol that could be called by GRUB and have further created GRUB patches for it. See

Please, mail Daniel Kieper and the GRUB-devel mailing list to endorse the suggestion.

Best regards

Heinrich

pa4wdh · August 11, 2021, 6:38am

Sorry for my very late reply, i finally got to testing this …

I’m now testing with u-boot 2021.07 (with sifive patches) and grub 2.06 with the [PATCH v2 0/7] Add LoadFile2 and riscv Linux loader and [PATCH 0/2] efi: device tree fix-up patches applied, trying to load kernels 5.12.11 and 5.13.4. It still fails with the message:

EFI stub: ERROR: /chosen/boot-hartid missing or invalid!

I have checked u-boot’s .config file and CONFIG_EFI_DT_FIXUP is set to y.
The grub commands i’m using to load the kernel are:

 devicetree (hd0,3)/gentoo-5.13.4.dtb
 linux (hd0,3)/gentoo-5.13.4.gz root=/dev/nvme0n1p5 rootwait console=ttySIF,115200 earlycon

Do you have a suggestion on how to troubleshoot this?

tekkamanninja · August 24, 2021, 9:45am

this patch works, I have successfully boot linux by GRUB on Starlight Board

tekkamanninja · August 24, 2021, 9:48am

BTW, the boot flow is auto, but only problem is that GRUB does NOT support vmlinuz (Image.gz zip kernel), only support vmlinux(Image)

the next step should be compressed kernel support

tekkamanninja · August 24, 2021, 9:56am

I have fixed this problem by your patch :

you can try my repo : GitHub - tekkamanninja/grub at riscv_devel_Nikita_V2

bkeys · August 24, 2021, 7:08pm

Do you have some instructions on how to get this flashed onto an SD card for the unmatched? or even better, a ready made image?

pa4wdh · August 25, 2021, 10:01am

Today i got some time to dive into this, it seems i was pretty close. The code to update the FDT was only reached when using an initrd, and since i don’t use an initrd it didn’t work for me
I created a small patch to move it outside the initrd-condition and my system is not booted with this grub version
This is the patch i made, it is to be applied after applying the patch series i mentioned earlier:

--- grub-2.06/grub-core/loader/efi/linux.c	2021-08-25 11:30:12.186661512 +0200
+++ grub-2.06-mod/grub-core/loader/efi/linux.c	2021-08-25 11:25:50.337761208 +0200
@@ -95,13 +95,14 @@
 
   void *fdt;
 
+  fdt = grub_fdt_load (GRUB_EFI_LINUX_FDT_EXTRA_SPACE);
+
+  if (!fdt)
+	goto failure;
+
   /* Set initrd info */
   if (initrd_start && initrd_end > initrd_start)
     {
-      fdt = grub_fdt_load (GRUB_EFI_LINUX_FDT_EXTRA_SPACE);
-
-      if (!fdt)
-	goto failure;
 
       node = grub_fdt_find_subnode (fdt, 0, "chosen");
       if (node < 0)

And to answer my own question on troubleshooting: On the grub commandline you can set the variable debug to get debugging info. In my case i simply used set debug=all.

@bkeys after compiling grub you can just use grub-install to get the bits and pieces in the right place. Getting u-boot to actually load it may require some fiddling.

pa4wdh · August 25, 2021, 5:10pm

@tekkamanninja i just noticed the version number in your screenshot which seems a bit odd to me. Grub has released 2.06 a few weeks ago, where did you get the sources?

schwab · August 26, 2021, 10:04am

Index of /repositories/home:/Andreas_Schwab:/riscv:/unmatched/images has images that use grub for booting.

Topic		Replies	Views
Stuck loading device tree HiFive Unmatched	6	3156	August 1, 2021
Boots from ssd kernel does not mount ssd root HiFive Unmatched	4	1097	April 3, 2024
A plea to desktop-focused OS distro creators on booting HiFive Unmatched	17	2523	August 23, 2021
Booting, a plea to SiFive on the HiFive Unmatched HiFive Unmatched	1	2920	January 12, 2021
Booting Linux from UEFI on QEMU for U540 board HiFive Unleashed	9	3081	July 25, 2023

Booting with grub

Related topics