I’m one of the people working on Haiku running on the SiFive Unmatched! (X512 being the lead)
I have a short plea to everyone working on various operating system distributions for the SiFive Unmatched. Given the slow transfer speeds of the bit-banged SD Card on the SiFive Unmatched, and the general complexity of writing the u-boot loader to the SD Card… I have the following proposal / best practice:
Encourage users to create a “firmware SD card”
What I mean by this, is don’t place your operating systems on the SD card. As an example, Haiku creates such a “firmware sd card” via the script here:
Release generic OS images, started via a RISCV64 EFI bootloader.
The model above means the SiFive unmatched can have a “firmware SD card” you don’t need to worry about, and all distros only need to release a generic “EFI-started rv64gc” image (much like how x86 works) that can be booted via USB flash. That image can even have an installer to write to the NVMe under a standard EFI model and function just like an x86 desktop.
u-boot compiled for the unmatched, providing EFI bios services, will pass a pre-configured FDT image for the Unmatched into EFI via a hook.
This helps to sidestep the whole complex “customized arm board image” model that has plagued the arm world for years.
Good point. Really unsure. I think the unmatched does have an on-board flash, but it’s not the default? u-boot is compiled for the Unmatched with OpenSBI embedded in it.
I actually like to fact that u-boot and OpenSBI are on a “removable” medium (SDcard, but NVMe would be fine too). Reason for that is that if you somehow screw up you can just connect it to a different computer correct the problem and get it booting again in no time. If this was all stored in a (fixed) flash on the board itself this would simply be impossible without reaching to your soldering iron . Of course you could add something to solve this problem, but that would only add extra complexity.
You have to flip switches to boot from flash. So if flash gets messed up, then you flip the switches to boot from sdcard, boot a rescue image that fixes flash, and then flip the switches back. And then since the sdcard rescue image probably doesn’t have the flash image you want, you then have to try to write flash again. And hope that a novice can follow and understand these instructions. We need more stable boot looders before this will be practical. Or a different board design with multiple banks of flash so the old version is still there if the new one doesn’t work.
But otherwise, yes, sdcard is more convenient for developers, and this is an eval/dev board, so that it why SiFive did it this way. Real computers should boot from flash into EFI.
I think u-boot needs some more work before it will run from flash on unmatched. Maybe a missing driver? I don’t know the details.
I know u-boot does 100% run from SD card, but yeah… really unsure about the onboard flash. What do the Unmatched boards ship with in the onboard flash?
Easy to update firmware is nice, but I worry that data on SD card may be lost if PC power was not turn on for a long time. Flash memory may need updating to avoid data loss.
Wouldn’t an on-board flash chip suffer from the same problem? Or is flash in sdcards such low quality that this applies to sdcards only?
When it comes to on-board flash i think PC-Engines has a nice solution for their APU boards. They have on board flash for BIOS, and in case that for some reason doesn’t work there is a module you can place on a special connector on the board. This disables the on-board flash and contains a flash chip with a known-good BIOS. You can boot the system with this BIOS, remove the module at runtime, and re-flash the on-board BIOS.
I haven’t heard of problems with people losing data on sdcards due to lack of refresh. A web search suggests that the data can live up to 10 years, but depending on the quality of the card and your luck, might only last a few years. So I would not recommend using an sdcard for long term storage, but if you are writing a new image every couple of months, or even once a year, you should not run into this problem.
The main reason this isn’t done currently is that the Linux ecosystem still hasn’t sorted out its EFI situation yet. I think there is EFI stub support in the kernel these days, but no support in GRUB so no proper bootloader, hence why extlinux is still being used. One day it’ll catch up to FreeBSD (and Haiku now if it’s been merged?) but until then they’ll keep training users to use non-PC-like methods.
FWIW the on-board flash on the Unmatched doesn’t need some kind of closed procedure. It’s just a flash chip connected via SPI that you can mount as a block device under your OS of choice (mx25l driver on FreeBSD that’s built by default as a module you can load, or whatever the equivalent is on Linux), with the flashing method being identical to how you’d put the firmware on an SD card instead, they’re treated the same (were U-Boot to support reading the DIP switches and changing which SPI device it loads from you could just dd a firmware-only SD card to the on-board flash). The only additional source of inconvenience is that you can write to the SD card from a different machine.
It does already, when I was bringing up FreeBSD I initially booted off of USB (with a firmware-only SD card) and it worked just fine. I continue to do the same with NVMe instead of USB, no OS in sight on my SD card. There are just two minor issues that you can encounter:
The default bootcmd scans NVMe before USB, and scanning NVMe has the side-effect of enumerating the PCI bus. bootcmd_usb0 only runs usb start, it does not also do anything to cause pci enum to run, so you cannot just run it on its own currently, and only works during the auto-boot sequence because bootcmd_nvme0 runs earlier.
USB is scanned lazily rather than eagerly. This means if you’re booting of NVMe then usb start has not been run, and so an EFI application (likely your bootloader) cannot see any block devices corresponding to any USB devices you have, only the NVMe drive (and in theory other PCIe-attached block devices). Though in practice your OS’s bootloader is going to live on the same device as your root filesystem. (The only reason I noticed this is I had my rootfs on NVMe but was using a USB drive to hold development kernels that I would copy from my Mac and load manually in FreeBSD’s bootloader, so would have to remember to run usb start manually in U-Boot.)
Your patch does indeed address those two quirks, but it doesn’t do what you say it does since that already works in the common case (unless that got broken in the process of upstreaming the out-of-tree patches to U-Boot; I’ve only ever used the version that came with the board as it’s sufficient for my purposes for now and have been waiting on things to stabilise upstream).
Boot from a EFI bootloader on a USB drive, EFI applications can’t see the USB block device in EFI hook services. It seems pretty simple. It could be a bug in EFI (bios services not getting populated with USB devices), but this definitely fixes it in my testing, and matches some other popular u-boot target boards:
[kallisti5@eris u-boot]$ grep -R “CONFIG_PREBOOT” configs/ | grep usb
configs/cm_fx6_defconfig:CONFIG_PREBOOT=“usb start;sf probe”
configs/mx6cuboxi_defconfig:CONFIG_PREBOOT=“if hdmidet; then usb start; setenv stdin serial,usbkbd; setenv stdout serial,vidconsole; setenv stderr serial,vidconsole; else setenv stdin serial; setenv stdout serial; setenv stderr serial; fi;”
configs/rpi_4_32b_defconfig:CONFIG_PREBOOT=“pci enum; usb start;”
configs/rpi_4_defconfig:CONFIG_PREBOOT=“pci enum; usb start;”
configs/rpi_arm64_defconfig:CONFIG_PREBOOT=“pci enum; usb start;”
configs/tbs2910_defconfig:CONFIG_PREBOOT=“echo PCI:; pci enum; pci 1; usb start; if hdmidet; then run set_con_hdmi; else run set_con_serial; fi”
configs/teres_i_defconfig:CONFIG_PREBOOT=“setenv usb_pgood_delay 2000; usb start”
configs/sifive_unmatched_defconfig:CONFIG_PREBOOT=“pci enum; usb start;”
All available busses should be available to u-boot loaders.
USB is scanned lazily rather than eagerly.
In testing, USB isn’t scanned at all when block devices are requested within u-boot’s EFI bios services.
Anything u-boot outside of u-boot master doesn’t matter
That doesn’t make sense, if it loaded your bootloader from a USB drive then it clearly found the USB drive, and pci enum; usb start is both necessary and sufficient for that to happen so adding that as a preboot command should have zero effect if you’re booting from USB. AFAICT the line you added never featured in the earlier out-of-tree patch series, so either you’re doing something wrong or upstream U-Boot is missing something else.
When I tested with u-boot binary from Release 2021.06.00 · sifive/freedom-u-sdk · GitHub, automatic USB boot caused no USB boot devices enumerated by EFI boot loader. Maybe some bug in u-boot EFI implementation. Currently “pci enum ; usb start ; run bootcmd_usb0” command is used for USB boot.
. Of course you could add something to solve this problem, but that would only add extra complexity.