Reporting Success, FreeBSD on my Unmatched

HiFive Unmatched
22

Yeah… sorry about that. I wasn’t checking my work like it was a post on my blog… I was just quickly typing it into the window. _load is for the loader.conf and _enable is for rc.conf.

23

Since ZFS is playing nicely with this setup I’ve been tempted to throw a real HBA into the system to see if that would work.
Tried with 2 different generations of LSI/Avago adapters but both fail with similar errors:

mps0: <Avago Technologies (LSI) SAS2308> at device 0.0 on pci7
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps0: Failed to setup interrupts
mps0: IOC Facts based allocation failed with error 6, exit
device_attach: mps0 attach returned 6

Assume that “Failed to setup interrupts” is at the heart of this not working. Any idea? or is this something hat hasn’t been implemented yet?
Just curious.

Thanks

24

I wouldn’t think SAS AIC’s would have gotten much coverage / testing for this ISA since that typically is more server oriented hardware.

I myself am thinking of trying a thunderbolt AIC in my system to see if it will play nice with my GPU thru bifurcation.

25

I assume the HBA is trying to use MSIs rather than legacy interrupts, which are all that are currently supported (I believe that’s also true of the Linux driver too).

26

Found this: mps - there’s also an equivalent for
mpr
This seems to confirm that MSI or MSIX is required… bummer

Unfortunately, disabling both MSI and MSIX doesn’t seem to do the trick…

Is this a missing Kernel bit or is this missing somewhere deeper?

27

Just a quick update - the SAS Controller is working under linux with kernel 5.12.14

ubuntu@ubuntu:~$ uname -a
Linux ubuntu 5.12.14 #1 SMP Thu Jul 29 13:24:37 EDT 2021 riscv64 riscv64 riscv64 GNU/Linux

Adapter is detected as a PCI device
ubuntu@ubuntu:~$ lspci
…
07:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)

ubuntu@ubuntu:~$ dmesg |grep scsi
[ 8.663142] scsi host0: Fusion MPT SAS Host
[ 112.764821] scsi 0:0:0:0: Direct-Access ATA SAMSUNG HD103UJ 1112 PQ: 0 ANSI: 6
[ 112.764860] scsi 0:0:0:0: SATA: handle(0x0009), sas_addr(0x4433221103000000), phy(3), device_name(0x0000000000000000)
[ 112.764870] scsi 0:0:0:0: enclosure logical id (0x500605b00badc1d0), slot(3)
[ 112.764979] scsi 0:0:0:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 112.764991] scsi 0:0:0:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)

Access to the drive seems to work as well
ubuntu@ubuntu:~/Sifive$ sudo hdparm -Tt /dev/sda

/dev/sda:
Timing cached reads: 372 MB in 2.01 seconds = 185.06 MB/sec
Timing buffered disk reads: 296 MB in 3.01 seconds = 98.49 MB/sec

ubuntu@ubuntu:/mnt$ sudo dd if=/dev/zero of=/mnt/test.out bs=8k count=100k
…
838860800 bytes (839 MB, 800 MiB) copied, 7.09733 s, 118 MB/s

I know, not a real performance test but I wanted to verify that the drive can be written and read in a sensible fashion.

28

My guess is the legacy interrupt code for those drivers is just incomplete or has bit-rotted, HBA’s are generally going to be used in systems where MSI support is present.

29

For anyone who stumbles across this thread in future: I’ve just committed the remaining patches from my tree to FreeBSD’s main branch so my fork is no longer needed.

2 Likes
30

Thanks Jessica
Are there any plans to implement MSI interrupts for this architecture?
Besides HBAs, NICs could also benefit from this. (Intel 520s etc.)

31

Yes, it’s on my list. The xHCI driver for USB would also make use of it if supported. The Synopsys IP used has a built-in MSI controller and the interrupt line used for it is in the device tree, FreeBSD just doesn’t yet support it because all of the PCIe controller drivers it has that are for variants of that IP instead use their own external MSI controllers (that provide more advanced functionality), so this is the first time FreeBSD has needed to support it.

32

Cool - Thank you.

33

I can’t get this to work.

I am very new to FreeBSD. Do I perform actions as root or regular user?

I did this:

  1. $ cd ~
  2. $ git clone GitHub - jrtc27/freebsd-src: FreeBSD src tree (read-only mirror)
  3. $ cd freebsd-src
  4. $ MAKEOBJDIRPREFIX=~/FreeBSD/obj/ make -j4 TARGET_ARCH=riscv64 buildworld
  5. $ MAKEOBJDIRPREFIX=~/FreeBSD/obj/ make -j4 TARGET_ARCH=riscv64 buildkernel
  6. Created EFI- and freebsd-ufs-partitions with gpart on my NVMe-drive
  7. Constructed file systems with newfs_msdos (only FAT16 was allowed due to size of partition? – I alloted 512MB for EFI partition) and newfs, respectively

  8. mount /dev/da0p2 /mnt

  9. mkdir /mnt/boot

  10. mount_msdosfs /dev/da0p1 /mnt/boot

  11. $ MAKEOBJDIRPREFIX=~/FreeBSD/obj/ make -j4 TARGET_ARCH=riscv64 installworld DESTDIR=/mnt
  12. Now make fails with a permissions problem (I should probably have known).
  13. If I try as root, I get: “MAKEOBJDIRPREFIX=~/FreeBSD/obj/: unknown command”

I decided to try a different strategy: Run everything as root…

  1. $ su -

  2. cd ~ (now I am in /root directory of course)

  3. git clone GitHub - jrtc27/freebsd-src: FreeBSD src tree (read-only mirror)

  4. cd freebsd-src

Since for some reason I can’t use “MAKEOBJDIRPREFIX=” as root, I just use:
5) # make -j4 TARGET_ARCH=riscv64 buildworld
6) # make -j4 TARGET_ARCH=riscv64 buildkernel
7) # mount /dev/da0p2 /mnt
8) # mkdir /mnt/boot
9) # mount_msdosfs /dev/da0p1 /mnt/boot
I guess by now my “objects” are in /usr/obj/ directory, so I just run
10) # make -j4 TARGET_ARCH=riscv64 installworld DESTDIR=/mnt

This time it eventually fails with:
"install: link /mnt/boot/loader_lua.efi → /mnt/boot/loader.efi: Operation not supported

make[2]: stopped in /root/freebsd-src
*** [installworld] Error code 2

make[1]: stopped in /root/freebsd-src
1 error

make[1]: stopped in /root/freebsd-src

make: stopped in /root/freebsd-src"

Maybe it’s just no so simple to replace ZFS from the instructions given above with UFS?
And again: Which commands are run as root? Which commands are run as regular user?

34

Cryo:

You don’t want your EFI partition mounted at /mnt/boot when trying to installworld. It’s trying to create a link in an msdos file system which doesn’t support links. Once you’ve done installworld, installkernel, and make distribution, you can umount /mnt and then work on setting up the efi partition separately.

I am running using ufs and it works fine. (I’m running on a USB thumb drive.)

Also, as Jessica mentions above, all the Unmatched support for FreeBSD is now in the official FreeBSD source and you should get the source from there.

35

Thanks for responding, skibo. What you say makes sense, now that I think about it.
I must have missed Jessica’s post regarding full support for the Unmatched board in the official sources, but I will definitely test that first thing tomorrow. I guess it will be FreeBSD 14.0 current then…

36

I just tried the latest snapshot from freebsd.org and it just works! I grabbed FreeBSD-14.0-CURRENT-riscv-riscv64-20210812-d20e9e02db3-248636-memstick.img.xz, the Aug 12th snapshot, and wrote it to a USB thumb drive and it boots to the installer. I then managed to install FreeBSD to my SSD. Amazing!

37

Yes - I tried that to. And it is amazing. However for some reason I can’t connect with cgem0. I get no DHCP offer.

38

Also: “shutdown -p now” or “init 0” doesn’t work as expected (I have to hold physical power button for a couple of seconds to completely power off machine). Same goes for reboot. Still the work done to provide installer image for this architecture is amazing!

39

I would like to build the U-boot files to flash onto an SD-card, but in /usr/ports/sysutils/ I can only find specific port for the Unleashed board (u-boot-sifive-fu540). How would one go about using the instructions given on HiFive Unmatched — Das U-Boot unknown version documentation ? Using make CROSS_COMPILE=riscv64-unknown-freebsd- PLATFORM=generic doesn’t work for building openSBI…

EDIT:
It probably should have been: “gmake CROSS_COMPILE=riscv64-none-elf- PLATFORM=generic” …
Trying /usr/ports/sysutils/u-boot-sifive-fu540/ with “make MODEL=sifive_fu740 BOARD_CONFIG=sifive_unmatched_defconfig” doesn’t produce working files (u-boot-spl.bin & u-boot.itb) either…

EDIT 2:
It sort of works now. I used a combination of pkg and ports:

# pkg install opensbi
# cd /usr/ports/sysutils/u-boot-sifive-fu540/
# make MODEL=sifive-fu740 BOARD_CONFIG=sifive_unmatched_defconfig -j4
# make install
# sgdisk --clear --set-alignment=2 \
        --new=1:34:2081 --change-name=1:loader1 --typecode=1:5B193300-FC78-40CD-8002-E86C45580B47 \
        --new=2:2082:10273 --change-name=2:loader2 --typecode=2:2E54B353-1271-4842-806F-E436D6AF6985 \
        /dev/da0
# dd if=/usr/local/share/u-boot/u-boot-sifive-fu740/u-boot-spl.bin of=/dev/da0p1 conv=sync
# dd if=/usr/local/share/u-boot/u-boot-sifive-fu740/u-boot.itb of=/dev/da0p2 conv=sync

Network still doesn’t work. Powering off machine doesn’t work. Rebooting doesn’t work. Also sometimes chassis fan never starts (probably something wrong with the board)…

40

poweroff (which should be equivalent to shutdown -p now) definitely works for me, though it does look like it’s hung as it’s not instant (maybe 10-20s from memory after the final kernel message?). Lack of reboot support is due to a firmware deficiency (also affects Linux). For networking, are you using a gigabit connection or not? The driver doesn’t currently support the SiFive-specific quirk for non-gigabit ethernet (why they had to do it differently from everyone else I don’t know). Chassis fans are done entirely by firmware, so not FreeBSD’s fault.

41

FYI that’s because root’s shell is csh which has different syntax for things like environment variables.