Reporting Success, FreeBSD on my Unmatched

[zBeeble]

I’m here to report success. I have my HiFive Unmatched booted with FreeBSD-current. The FreeBSD page for the HiFive is here: riscv/HiFiveUnmatched - FreeBSD Wiki

Until there is an image to do this, this was my process (I did this on an AMD threadripper, but, in theory, you should be able to do it on any FreeBSD host):

0. This proceedure should work for a USB drive or an NVMe drive (I got a USB to NVMe enclosure so I could hook the NVMe drive to my workstation)
1. I checked out the git FreeBSD source on my FreeBSD machine.
   git clone https://github.com/jrtc27/freebsd-src/tree/unmatched
2. I made sure my FreeBSD machine had qemu installed (so that I could cross compile stuff)
3. I ran “MAKEOBJDIRPREFIX=~/FreeBSD/obj/ make -j16 TARGET_ARCH=riscv64 buildworld”
4. I ran “MAKEOBJDIRPREFIX=~/FreeBSD/obj/ make -j16 TARGET_ARCH=riscv64 buildkernel”
5. I partitioned my nvme with GPT: 100 meg EFI, 64G swap, Rest ZFS.
   [2:24:24]dgilbert@ump:~/devel/rogue> gpart show

=>        34  3907029101  nda0  GPT  (1.8T)
          34           6        - free -  (3.0K)
          40      204800     1  efi  (100M)
      204840   134217728     2  freebsd-swap  (64G)
   134422568  3772606560     3  freebsd-zfs  (1.8T)
  3907029128           7        - free -  (3.5K)

6. I created and mounted my zfs tree under /mnt. make sure to set bootfs=your boot fs

zpool create -R /mnt zump da0p3
zfs set mountpoint=none zump
zfs create zump/ROOT
zfs create -o mountpoint=/ zump/ROOT/default
zpool set bootfs=zump/ROOT/default
zfs create zump/usr
zfs create -o mountpoint=/usr/local zump/usr/local
zfs create -o mountpoint=/home zump/home
zfs create -o mountpoint=/usr/obj zump/usr/obj
zfs create -o mountpoint=/usr/src zump/usr/src
zfs create -o mountpoint=/usr/ports zump/usr/ports
zfs create zump/var
zfs create -o mountpoint=/var/log zump/var/log
zfs create -o mountpoint=/var/crash zump/var/crash
zfs create -o mountpoint=/var/audit zump/var/audit
zfs create -o mountpoint=/var/cache zump/var/cache
zfs create -o mountpoint=/var/mail zump/var/mail
zfs create -o mountpoint=/var/tmp zump/var/tmp

7. I ran “MAKEOBJDIRPREFIX=~/FreeBSD/obj/ make -j16 TARGET_ARCH=riscv64 installworld DESTDIR=/mnt”
8. I ran “MAKEOBJDIRPREFIX=~/FreeBSD/obj/ make -j16 TARGET_ARCH=riscv64 installkernel DESTDIR=/mnt”
9. I ran “MAKEOBJDIRPREFIX=~/FreeBSD/obj/ make -j16 TARGET_ARCH=riscv64 distribution DESTDIR=/mnt”
10. (on edit, because I forgot … I actually even forgot this when I was doing it) edit /mnt/boot/loader.conf to contain the line "zfs_enable=“YES” … if you don’t do that you’ll need to know how to manually load it with the loader, boot single-user, rw mount root, and then edit it… so just do it here.
11. export the zfs /mnt

cp /mnt/boot/loader.efi /tmp
zfs export zump

12. newfs_msdosfs /dev/da0p1 the 100meg partition.
13. mount it and put what was /mnt/boot/loader.efi as /mnt/EFI/BOOT/BOOTRISCV64.EFI on the msdoss partition.

mount_msdosfs /dev/da0p1 /mnt
mkdir -p /mnt/EFI/BOOT
cp /tmp/loader.efi /mnt/EFI/BOOT/BOOTRISC64.EFI
umount /mnt

14. attach this media and boot.

… A few cool things to note here: you’ve just compiled a complete operating system from source and installed it onto a new architecture from scratch. Congrats if this is your first real bootstrap!

[tinkerbug]

Impressive - definitely need to give this a go.
zfs is a pretty resource hungry file system AFAIK. How does this perform on the unmatched?

[pa4wdh]

Congrats on getting FreeBSD going. As a Gentoo user I understand the value of compiling your complete OS from source

[zBeeble]

Even with compression=on on ZFS, I don’t ever find ZFS to be the limiting factor. ZFS is more memory intensive than processor intensive and the choice to put 16G of memory on this board means that ZFS has lots of room for being as busy as the board can be.

As an example I can launch several ports builds — which kick off -j4 by default (they detect the number of CPUs)… and the filesystem or “system” CPU percentage never rises into the “it matters” zone. ZFS in this situation often caches as much as a couple of gig of data, but so far, as hard as I hit it, it hasn’t even swapped.

If you want to compare speeds, my best benchmark so far is a little C++ game I’m working on. It heavily uses boost, among other things. It takes 5 CPU seconds to compile on my threadripper 2950X. It takes 93 CPU seconds on the unmatched. all-in-all not a bad result for the unmatched.

But don’t be afraid of ZFS … runs like a champ.

[tinkerbug]

I’m not afraid of ZFS - still supporting some rather large ZFS implementations (multi PB) - just surprised to find it running on Risc-V True, 16GB will help a lot with this. Have you tried to run some throughput stats?

Will have to get anther nvme card to build this - definitely on my list now.
Thanks

[zBeeble]

Um… so it’s a 2T SN550 that I have in there now. I’ve got another 500G SN550 on the way. By no means does it max the card. FreeBSD’s dev/null transfer speed is in the 200 to 250 megabyte/s range.

That said, ZFS handily gets into the 75-ish megabyte/s range. More performance testing later, tho. For me ZFS is just required for the management it gives me. UFS isn’t any faster on this platform.

64 bit is ZFS’ jam.

[tinkerbug]

Managed to follow your steps to create the nvme.
I used a VM with a freshly installed FreeBSD 13.0 image as my build-host

The only hick-up was the git clone. for some reason I needed to use this instead
git clone https://github.com/jrtc27/freebsd-src.git

From other posts I assume that we still need an SD Card for u-boot etc. Is this the case for this install also? if not then I went wrong somewhere as I only have some garbage on the serial console

Thanks

[jimw (Jim Wilson)]

Yes, you still need the sdcard to boot. The zsbl in rom transfers control to the fsbl on the sdcard.

[zBeeble]

Thanks for the replication. I have install media available now… but we’re still chasing a bug or two. Anyone wanting to jump on can ask me for ALPHA quality install images.

With the most recent kernel patch (a little VM issue) … it should all be self-hosted.

[tinkerbug]

I’m still missing something - but it feels like there’s progress on my end

• updated u-boot to 2021.07 and loaded u-boot-spl.bin and u-boot.itb to partitions 1 and 2
• loaded boot1.efi, loader*.efi and loader.conf into /boot of partition 3 SD Card
  – copied BOOTRISC64.EFI into EFI/BOOT (on SDCard as well as part 1 on the nvme)
  the system starts booting and ultimately drops into an efi shell.
  efi-show displays
  freebsd BS,RS LoaderPath = /efi\boot\bootriscv64.efi
  freebsd BS,RS LoaderDev = /VenHW (UUID)/SD(0)/SD(0)/HD(3,GPT,UUID

is LoaderDev pointing to partition 3 of the SD card? if so, how can I point it to the NVMe?

Thanks

[zBeeble]

My install isn’t using boot1.efi. Only loader.efi renamed as /EFI/BOOT/BOOTRISC64.EFI. So the “EFI” partition only contains the one file. loader.efi willl look for a partition of type freebsd-ufs or freebsd-zfs and load things like /boot/loader.conf from it.

Technically, even on wintel, boot1.efi isn’t required.

[tinkerbug]

thanks for your feedback

I recreated the SD Card - 3 partitions. 1 and 2 for u-boot and 3 flagged as ESP with a fat16

root@ws14l:/mnt# sgdisk -p /dev/mmcblk0
Number Start (sector) End (sector) Size Code Name
1 34 2081 1024.0 KiB FFFF u-boot-spl
2 2082 10273 4.0 MiB FFFF opensbi-uboot
3 16384 221183 100.0 MiB EF00 EFI

root@ws14l:/mnt# find .
.
./EFI
./EFI/BOOT
./EFI/BOOT/BOOTRISC64.EFI

NVMe looks like this:
root@ws14l:~# sgdisk -p /dev/sda
Disk /dev/sda: 976773168 sectors, 465.8 GiB
Number Start (sector) End (sector) Size Code Name
1 2048 206847 100.0 MiB EF00
2 206848 134424575 64.0 GiB A502
3 134424576 976773119 401.7 GiB A504

partition 1 looks like this
root@ws14l:/mnt# find .
.
./EFI
./EFI/BOOT
./EFI/BOOT/BOOTRISC64.EFI

u-boot complains that there is no system image. If I cancel autoboot and attempt it manually I get
EFI Boot Manager: cannot load any image

mmc part shows the 3 partitions of the SDCard

I’m obviously missing something but can’t figure out what. Any hints?

Thanks

[zBeeble]

I never put anything on the SDCard. My NVMe looks like this:

[2:6:86]root@ump:~> gpart show nda0
=>        34  3907029101  nda0  GPT  (1.8T)
          34           6        - free -  (3.0K)
          40      204800     1  efi  (100M)
      204840   134217728     2  freebsd-swap  (64G)
   134422568  3772606560     3  freebsd-zfs  (1.8T)
  3907029128           7        - free -  (3.5K)

You can see here that they are the same program.

[2:11:91]root@ump:~> ll -lR /d/EFI/EFI/BOOT/BOOTRISCV64.EFI
-rwxr-xr-x  1 root  wheel  1423660 Jul  4 13:20 /d/EFI/EFI/BOOT/BOOTRISCV64.EFI
[2:12:92]root@ump:~> ll /boot/loader.efi
-r-xr-xr-x  2 root  wheel  1423660 Jul  4 17:14 /boot/loader.efi

Now… This is important:

[2:13:93]root@ump:~> zpool get bootfs zump
NAME  PROPERTY  VALUE              SOURCE
zump  bootfs    zump/ROOT/default  local

… that says that the boot fs is zump/ROOT/default on the zfs pool. If you look at my mounted running system, you see:

[2:14:94]root@ump:~> df -h
Filesystem                   Size    Used   Avail Capacity  Mounted on
zump/ROOT/default            1.7T    2.5G    1.7T     0%    /
devfs                        1.0K    1.0K      0B   100%    /dev
tmpfs                        3.0G     34M    3.0G     1%    /tmp
zump/usr/local               1.7T    4.0G    1.7T     0%    /usr/local
zump/home                    1.7T     96K    1.7T     0%    /home
zump/var/crash               1.7T    7.7G    1.7T     0%    /var/crash
zump/usr/ports               1.7T    2.4G    1.7T     0%    /usr/ports
zump/var/log                 1.7T    544K    1.7T     0%    /var/log
zump/usr/obj                 1.7T    3.3G    1.7T     0%    /usr/obj
zump/var/audit               1.7T     96K    1.7T     0%    /var/audit
zump/var/tmp                 1.7T     96K    1.7T     0%    /var/tmp
zump/var/mail                1.7T    144K    1.7T     0%    /var/mail
zump/usr/src                 1.7T    3.0G    1.7T     0%    /usr/src
zump/var/cache               1.7T    104K    1.7T     0%    /var/cache
zump/home/dgilbert           1.7T    4.7M    1.7T     0%    /home/dgilbert
zump/var/cache/ccache        1.7T    1.0G    1.7T     0%    /var/cache/ccache
vr:/usr/ports/distfiles      1.2T    339G    916G    27%    /usr/ports/distfiles
vr:/home                     6.7T    726K    6.7T     0%    /d/vr/home
vr:/home/dgilbert            6.9T    215G    6.7T     3%    /d/vr/home/dgilbert
vr:/home/dgilbert/FreeBSD    6.7T     29G    6.7T     0%    /d/vr/home/dgilbert/FreeBSD
vr:/vr1/tmp                  9.6T    2.9T    6.7T    30%    /d/vr/tmp
/dev/nda0p1                  100M    1.4M     99M     1%    /d/EFI

… if you notice that usr and var are not mount points, congrats. This is part and parcel of how boot environments work. zump/BOOT/default contains all of what would be var and usr (except subdirectories that mount on their own)… zump/BOOT/usr and zump/BOOT/var are there… but canmount=false.

[jrtc27 (Jessica Clarke)]

Various points here:

1. You have the wrong file name; it should be BOOTRISCV64.EFI, not BOOTRISC64.EFI

2. There is no reason to put an ESP on the SD card. Using the SD card more than you need to will just slow things down (NVMe is not blazing fast but it sure does beat an SD card, plus NVMe is one of the first devices for U-Boot to probe, and the SD card the last, so you save a few seconds off the U-Boot boot time by not having to let it go all the way through the list), and it’s also just a pain to juggle multiple disks that’s totally unnecessary, just have the ESP on your NVMe drive and be done with it.

3. boot1.efi is deprecated as it serves little purpose other than to complicate the boot process.

4. … but even if you were using boot1.efi, it expects to load loader.efi from the root filesystem, not the ESP.

5. Having the ESP and root filesystem on different disks can actually cause weird issues. U-Boot probes block devices lazily, so depending on the boot order you can end up booting your ESP and not having the block device with the root filesystem partition on it available. I have run into this when copying development kernels back and forth on a USB stick, as my ESP (and root filesystem) live on the NVMe drive, so U-Boot doesn’t even bother running usb start, meaning the FreeBSD boot loader cannot see any USB drives and thus I cannot manually load a kernel from it (the workaround for this is to drop to a U-Boot prompt and manually run pci enum and usb start before letting it boot). Moreover, loader’s heuristics are to prefer a root freebsd-ufs or freebsd-zfs partition on the same disk as the ESP, so if you grow a more complex setup your system could suddenly start trying to load a kernel from and mount as root the wrong partition.

Thus, the official way to do things is as zBeeble said in the original post: put the ESP on the same disk as your root partition, and use loader.efi as EFI/BOOT/BOOTRISCV64.EFI (no other files needed) in the ESP.

[tinkerbug]

@zBeeble

zpool set bootfs=zump/ROOT/default

This is at least one area where things went wrong for me
I believe you need to specify the pool name at the end?

I destroyed the pool and restarted from scratch - unfortunately, still no luck
zfs list |grep zump
zump 2.51G 385G 96K none
zump/ROOT 2.51G 385G 96K none
zump/ROOT/default 2.51G 385G 2.51G /mnt
zump/home 96K 385G 96K /mnt/home
zump/usr 480K 385G 96K none
zump/usr/local 96K 385G 96K /mnt/usr/local
zump/usr/obj 96K 385G 96K /mnt/usr/obj
zump/usr/ports 96K 385G 96K /mnt/usr/ports
zump/usr/src 96K 385G 96K /mnt/usr/src
zump/var 676K 385G 96K none
zump/var/audit 96K 385G 96K /mnt/var/audit
zump/var/cache 96K 385G 96K /mnt/var/cache
zump/var/crash 100K 385G 100K /mnt/var/crash
zump/var/log 96K 385G 96K /mnt/var/log
zump/var/mail 96K 385G 96K /mnt/var/mail
zump/var/tmp 96K 385G 96K /mnt/var/tmp

root@freebsd-ca1:~ # zpool get bootfs zump
NAME PROPERTY VALUE SOURCE
zump bootfs zump/ROOT/default local
root@freebsd-ca1:~ #

root@freebsd-ca1:~ # mount -t msdos /dev/da1p1 /mnt
root@freebsd-ca1:~ # cd /mnt
root@freebsd-ca1:/mnt # find .
.
./EFI
./EFI/BOOT
./EFI/BOOT/BOOTRISC64.EFI

root@freebsd-ca1:~ # ls -latr /mnt/boot/loader.efi
-r-xr-xr-x 2 root wheel 1423292 Jul 11 19:30 /mnt/boot/loader.efi

root@freebsd-ca1:/mnt # ls -latr /mnt/EFI/BOOT/BOOTRISC64.EFI
-rwxr-xr-x 1 root wheel 1423292 Jul 11 19:02 /mnt/EFI/BOOT/BOOTRISC64.EFI

I removed the 3rd partition from the SDCard - only the two u-boot partitions left.

This is the console log
U-Boot SPL 2021.07-00427-g490101a5e5 (Jul 11 2021 - 13:23:09 -0400)
Trying to boot from MMC1

U-Boot 2021.07-00427-g490101a5e5 (Jul 11 2021 - 13:23:09 -0400)

CPU: rv64imafdc
Model: SiFive HiFive Unmatched A00
DRAM: 16 GiB
MMC: spi@10050000:mmc@0: 0
Loading Environment from nowhere… OK
EEPROM: SiFive PCB EEPROM format v1
Product ID: 0002 (HiFive Unmatched)
PCB revision: 3
BOM revision: B
BOM variant: 0
Serial number: SF105SZ212200289
Ethernet MAC address: 70:b3:d5:92:f7:d2
CRC: 327bd10e
In: serial@10010000
Out: serial@10010000
Err: serial@10010000
Model: SiFive HiFive Unmatched A00
Net: eth0: ethernet@10090000
Hit any key to stop autoboot: 0
PCIE-0: Link up (Gen1-x8, Bus0)

Device 0: Vendor: 0xc0a9 Rev: P2CR031 Prod: 2053E4E38ED7
Type: Hard Disk
Capacity: 476940.0 MB = 465.7 GB (976773168 x 512)
… is now current device
Scanning nvme 0:1…
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Scanning disk mmc@0.blk…
** Unrecognized filesystem type **
** Unrecognized filesystem type **
Scanning disk nvme#0.blk#1…
** Unrecognized filesystem type **
** Unrecognized filesystem type **
Found 7 disks
** Unable to read file ubootefi.var **
Failed to load EFI variables
BootOrder not defined
EFI boot manager: Cannot load any image
starting USB…
Bus xhci_pci: Register 4000840 NbrPorts 4
Starting the controller
USB XHCI 1.00
scanning bus xhci_pci for devices… 4 USB Device(s) found
scanning usb for storage devices… 0 Storage Device(s) found

Device 0: unknown device
switch to partitions #0, OK
mmc0 is current device
** Unrecognized filesystem type **
ethernet@10090000: PHY present at 0
ethernet@10090000: Starting autonegotiation…
ethernet@10090000: Autonegotiation complete
ethernet@10090000: link up, 1000Mbps full-duplex (lpa: 0x2800)
BOOTP broadcast 1
BOOTP broadcast 2
DHCP client bound to address 192.168.1.116 (256 ms)
*** ERROR: serverip' not set Cannot autoload with TFTPGET missing environment variable: pxeuuid missing environment variable: bootfile Retrieving file: pxelinux.cfg/01-70-b3-d5-92-f7-d2 ethernet@10090000: PHY present at 0 ethernet@10090000: Starting autonegotiation... ethernet@10090000: Autonegotiation complete ethernet@10090000: link up, 1000Mbps full-duplex (lpa: 0x2800) *** ERROR: serverip’ not set
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/C0A80174
ethernet@10090000: PHY present at 0
ethernet@10090000: Starting autonegotiation…

the nvme disk is detected (Device 0)

minor detail

@zBeeble

zfs export zump

this should probably be zpool export zump??

[jrtc27 (Jessica Clarke)]

You still have that persistent typo in your filename; it’s BOOTRISCV64.EFI, not BOOTRISC64.EFI.

[tinkerbug]

apologies - completely missed this typo. now it works and FreeBSD is booting!

I also removed everything but the two u-boot partitions from the SDCard.

Thanks for your patience and help.

[tinkerbug]

Thanks - this makes it much clearer.

[zBeeble]

sorry for the typos. You also put zpool set bootfs zump/ROOT/default zump … where it needs to be zpool set bootfs=zump/ROOT/default zump (the = sign).

Anyways… sorry for the typos. Install media should be available soon.

My unmatched finished self-hosting today… :)!

[tinkerbug]

The system dumped me into mountroot> during my first attempt.
I did double check that loader.conf is set as described in the initial procedure

@zBeeble

(on edit, because I forgot … I actually even forgot this when I was doing it) edit /mnt/boot/loader.conf to contain the line "zfs_enable=“YES” … if you don’t do that you’ll need to know how to manually load it with the loader, boot single-user, rw mount root, and then edit it… so just do it here.

A bit of internet research gave me this hint:
“Check that in /boot/loader.conf you have zfs_load=“YES” and in /etc/rc.conf zfs_enable=“YES””
changing it around like this makes my setup boot without issues.

Can you please verify/confirm?

Thanks