Booting from Flash

I managed to get my Unmatched to boot from flash. It was a matter of enabling the SPI flash drivers in the SPL, and adding an SPI option to spl_boot_device(). Here are the diffs I applied to U-Boot:

diff --git a/arch/riscv/dts/hifive-unmatched-a00-u-boot.dtsi b/arch/riscv/dts/hifive-unmatched-a00-u-boot.dtsi
index c5475aa149..5e499f7562 100644
--- a/arch/riscv/dts/hifive-unmatched-a00-u-boot.dtsi
+++ b/arch/riscv/dts/hifive-unmatched-a00-u-boot.dtsi
@@ -30,6 +30,13 @@
 	clocks = <&rtcclk>;
 };
 
+&qspi0 {
+	u-boot,dm-spl;
+	flash@0 {
+		u-boot,dm-spl;
+	};
+};
+
 &spi0 {
 	mmc@0 {
 		u-boot,dm-spl;
diff --git a/board/sifive/unmatched/spl.c b/board/sifive/unmatched/spl.c
index d5663274cd..7c0beedc08 100644
--- a/board/sifive/unmatched/spl.c
+++ b/board/sifive/unmatched/spl.c
@@ -22,6 +22,7 @@
 #define GEM_PHY_RESET	SIFIVE_GENERIC_GPIO_NR(0, 12)
 
 #define MODE_SELECT_REG		0x1000
+#define MODE_SELECT_SPI		0x6
 #define MODE_SELECT_SD		0xb
 #define MODE_SELECT_MASK	GENMASK(3, 0)
 
@@ -123,6 +124,8 @@ u32 spl_boot_device(void)
 	u32 boot_device = mode_select & MODE_SELECT_MASK;
 
 	switch (boot_device) {
+	case MODE_SELECT_SPI:
+		return BOOT_DEVICE_SPI;
 	case MODE_SELECT_SD:
 		return BOOT_DEVICE_MMC1;
 	default:
diff --git a/configs/sifive_unmatched_defconfig b/configs/sifive_unmatched_defconfig
index 38b7acd536..a8652c615e 100644
--- a/configs/sifive_unmatched_defconfig
+++ b/configs/sifive_unmatched_defconfig
@@ -6,6 +6,7 @@ CONFIG_SPL_DM_SPI=y
 CONFIG_DEFAULT_DEVICE_TREE="hifive-unmatched-a00"
 CONFIG_SPL_MMC_SUPPORT=y
 CONFIG_SPL=y
+CONFIG_SPL_SPI_FLASH_SUPPORT=y
 CONFIG_SPL_SPI_SUPPORT=y
 CONFIG_TARGET_SIFIVE_UNMATCHED=y
 CONFIG_ARCH_RV64I=y
@@ -19,7 +20,11 @@ CONFIG_DISPLAY_CPUINFO=y
 CONFIG_DISPLAY_BOARDINFO=y
 CONFIG_DISPLAY_BOARDINFO_LATE=y
 CONFIG_SPL_SEPARATE_BSS=y
+CONFIG_SPL_MTD_SUPPORT=y
+CONFIG_SPL_DM_SPI_FLASH=y
 CONFIG_SPL_DM_RESET=y
+CONFIG_SPL_SPI_LOAD=y
+CONFIG_SYS_SPI_U_BOOT_OFFS=0x104400
 CONFIG_SPL_YMODEM_SUPPORT=y
 CONFIG_CMD_EEPROM=y
 CONFIG_CMD_MEMINFO=y
@@ -29,6 +34,8 @@ CONFIG_CMD_PCI=y
 CONFIG_CMD_USB=y
 CONFIG_SYS_RELOC_GD_ENV_ADDR=y
 CONFIG_SPL_CLK=y
+CONFIG_SF_DEFAULT_MODE=0
+CONFIG_SPI_FLASH_ISSI=y
 CONFIG_E1000=y
 CONFIG_NVME=y
 CONFIG_PCI=y

Information on building U-boot is here: HiFive Unmatched — Das U-Boot unknown version documentation

Here is the script I used to create a disk image to be written to the flash:

dd if=/dev/zero of=FLASH.img bs=512 count=12288

sgdisk -g --clear -a 1 \
  --new=1:34:2081         --change-name=1:spl --typecode=1:5B193300-FC78-40CD-8002-E86C45580B47 \
  --new=2:2082:10273      --change-name=2:uboot  --typecode=2:2E54B353-1271-4842-806F-E436D6AF6985 \
  FLASH.img

dd if=../u-boot/spl/u-boot-spl.bin of=FLASH.img bs=512 seek=34 conv=sync,notrunc
dd if=../u-boot/u-boot.itb  of=FLASH.img bs=512 seek=2082 conv=sync,notrunc


I wrote the image to flash while running FreeBSD using “dd if=FLASH.img of=/dev/flash/spi0 bs=4096 conv=sync”. I’m not sure what the flash device is on Linux.

Flip the boot DIP switches to 6 and yank the SD card out.

6 Likes

On unleashed running linux, flash is /dev/mtd0 and can be written with the flashcp program. It should be the same on unmatched. dd perhaps works also.

Works good!

U-Boot SPL 2021.07-dirty (Aug 23 2021 - 22:10:37 -0700)
Trying to boot from SPI        <------------


U-Boot 2021.07-dirty (Aug 23 2021 - 22:10:37 -0700)

CPU:   rv64imafdc
Model: SiFive HiFive Unmatched A00
DRAM:  16 GiB

Here’s an Ubuntu kernel that has /dev/mtd0 enabled.

https://www.w6rz.net/linux-image-5.14.0-rc7_5.14.0-rc7-1_riscv64.deb
https://www.w6rz.net/linux-headers-5.14.0-rc7_5.14.0-rc7-1_riscv64.deb
https://www.w6rz.net/linux-libc-dev_5.14.0-rc7-1_riscv64.deb

And here are the u-boot 2021.07 files that boot from flash (with @skibo patches and SiFive patches).

https://www.w6rz.net/u-boot-flash-1200mhz.itb
https://www.w6rz.net/u-boot-spl-flash-1200mhz.bin

https://www.w6rz.net/u-boot-flash-1400mhz.itb
https://www.w6rz.net/u-boot-spl-flash-1400mhz.bin

https://www.w6rz.net/u-boot-flash-1500mhz.itb
https://www.w6rz.net/u-boot-spl-flash-1500mhz.bin

I used flashcp to write the flash.

sudo apt-get install mtd-utils

sudo flashcp -v flash.img /dev/mtd0

1 Like

Making the SPI readable will allow us to store the environment and UEFI variables there.
Can the boot ROM boot from SPI? Then we could put OpenSBI and U-Boot or maybe in future EDK II there and get rid of the SD-card.

Cf. Boot without SD Card?

Yes, I’m booting just from SPI. The SD card slot is empty. After booting the Ubuntu kernel linked above:

ubuntu@riscv64:~$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
mtdblock0     31:0    0    32M  0 disk 
nvme0n1      259:0    0 465.8G  0 disk 
├─nvme0n1p1  259:1    0 465.6G  0 part /
├─nvme0n1p12 259:2    0     4M  0 part 
├─nvme0n1p13 259:3    0     1M  0 part 
├─nvme0n1p14 259:4    0     4M  0 part 
└─nvme0n1p15 259:5    0   106M  0 part /boot/efi
ubuntu@riscv64:~$

Unfortunately, /var/log/syslog and dmesg get spammed every 4 seconds with:

[Thu Aug 26 06:03:54 2021] mmc_spi spi1.0: no support for card's volts
[Thu Aug 26 06:03:54 2021] mmc0: error -22 whilst initialising SDIO card

But it works fine otherwise.

Since the patches are pretty simple, I’d say they could be upstreamed soon.

How does the boot ROM identify that it should boot from SPI? Where do SPL+OpenSBI and main U-Boot have to be stored? Is there a way to unbrick the board if the SPI image is bad?

The position of the boot select switches determines whether you boot from flash or sdcard. The first post in this thread mentions setting the boot select switches to 6.

If you break flash, you just flip the switches back to boot from sdcard and then use an sdcard image to fix flash.

Flash is just another disk, so it can hold a disk image just like the sdcard does.

I see the same problem with the syslog spamming regarding mmc if there is no SD card inserted. Interestingly it goes away after inserting one - even if it is removed again. I wonder if it is related with the voltage warning thingy…? That could probably be disabled by removing the voltage-ranges line from the DT as explained in this (possibly outdated) blog post: July | 2014 | Vonger's Blog
If so that DT should be fixed “officially” I guess (what’s upstream for these)?
Or there is a proper quirks setting for the controller? AFAICT the driver is built-in into the Ubuntu kernel… that makes experimenting with that too much of a hassle for me right now.

It looks like the correct solution is to use the card detect signal. There’s a patch for the device tree here.

https://patchwork.kernel.org/project/linux-riscv/patch/20210616074645.429578-2-bmeng.cn@gmail.com/

I’m not sure if that’s all that’s needed. I haven’t tried it yet. I do have an Ubuntu kernel with MMC disabled.

https://www.w6rz.net/linux-image-5.14.0_5.14.0-1_riscv64.deb
https://www.w6rz.net/linux-headers-5.14.0_5.14.0-1_riscv64.deb
https://www.w6rz.net/linux-libc-dev_5.14.0-1_riscv64.deb

I have tested it and yes that’s all that is needed. One can simply replace the existing dtb in /lib/firmware/5.11.0-1017-generic/device-tree/sifive/ or whatever (but beware it will be overwritten by kernel updates). To ease recompilation of a decompiled version, here is the change with expanded values: gpios = <0x0b 0x0f 0x01>;

Writing the image with dd takes quite a while. If you install flashrom you can improve on that a lot. It also is a good idea to create the images with all ones because that reduces the wear on the flash (that’s the tr part below). I use these bits to create and flash the image instead of what skibo posted above:

tr '\0' '\377' < /dev/zero | dd of=FLASH.img iflag=fullblock bs=1M count=32
/sbin/sgdisk -g --clear -a 1 \
  --new=1:34:2081         --change-name=1:spl --typecode=1:5B193300-FC78-40CD-8002-E86C45580B47 \
  --new=2:2082:10273      --change-name=2:uboot  --typecode=2:2E54B353-1271-4842-806F-E436D6AF6985 \
  FLASH.img

dd if=u-boot/spl/u-boot-spl.bin of=FLASH.img bs=512 seek=34 conv=sync,notrunc
dd if=u-boot/u-boot.itb         of=FLASH.img bs=512 seek=2082 conv=sync,notrunc

scp FLASH.img root@unmatched:

ssh root@unmatched "flashrom -p linux_mtd --noverify -w FLASH.img && poweroff"

Note that I create a 32MB image to match the flash size (a requirement of flashrom) and that the last two steps operate via SSH on the host unmatched. Replace that with the IP of your board if necessary.

1 Like

As mentioned before you can dd to SPI-NOR Flash directly under linux (done that before) as mtdblock is available.

In general (quick look) the patch above looks OK. Basically the code should be the same as on Unleashed (copy & paste). I am not sure you need all the CONFIG_* here. For example CONFIG_SYS_SPI_U_BOOT_OFFS should be part of DTSI. From Unleashed:

 20     config {                                                                    
 21         u-boot,spl-payload-offset = <0x105000>; /* loader2 @1044KB */           
 22     };  

I highly suggest to send it out to U-Boot development mailing list, just double check that SPI-NOR layout is matching Unleashed.

There are a few thigns missing in arch/riscv/cpu/fu740/Kconfig, seems to be:

 41 if ENV_IS_IN_SPI_FLASH                                                          
 42                                                                                 
 43 config ENV_OFFSET                                                               
 44     default 0x505000                                                            
 45                                                                                 
 46 config ENV_SIZE                                                                 
 47     default 0x20000                                                             
 48                                                                                 
 49 config ENV_SECT_SIZE                                                            
 50     default 0x10000                                                             
 51                                                                                 
 52 endif # ENV_IS_IN_SPI_FLASH

See email in U-Boot archive (2020, Jul 15): [PATCH v5 3/6] env: Enable SPI flash env for SiFive FU540

SPI flash device on HiFive Unleashed has 32MiB Size.

This patch adds SPI flash environment after U-Boot proper
partition with a size of 128KiB.

SPI flash partition layout(32MiB):
    0 - 34      : reserved for GPT header
   35 - 39      : unused
   40 - 2087    : loader1 (SPL, FSBL)
 2088 - 10279   : loader2 (U-Boot proper, U-Boot)
10280 - 10535   : environment
10536 - 65494   : rootfs
65528 - 65536   : distro script

Note: the loader1 must start from 40th sector even though
there are 6 free sectors prior since 40th sector is nearest
flash sector boundary. 

I had tried to arrange the flash partitions so that they are aligned to 4K boundaries to make it easier to update the SPL and U-boot partitions but I could not get the SPL to boot unless I set up the partitions exactly as I did (sectors 34 and 2082). I never went back to figure out why. Also, I should’ve added a dos or EFI partition to store the environment or have a boot.scr.

EDIT: Ha. I figured it out. When I moved the partitions to align them, I think I forgot to modify the seek values in the dd commands so I corrupted the image. I managed to get it to work.

@skibo Did you create a patch for upstream U-Boot yet?

It would be great if somebody from SiFive could update https://github.com/sifive/freedom-u-sdk#readme to show the boot select settings for booting from SPI.

Not yet. When I get some time, I want to incorporate @davidlt 's suggestions.

I’ve finally had some time to come back to this. This new patch is simplified a bit and supports storing the environment in flash. It is mostly the same as the July 15, 2020 commit to add SPI support to Unleashed but without support for putting the distro in flash.

diff --git a/arch/riscv/cpu/fu740/Kconfig b/arch/riscv/cpu/fu740/Kconfig
index 049a0a0584..3e0c1fddc8 100644
--- a/arch/riscv/cpu/fu740/Kconfig
+++ b/arch/riscv/cpu/fu740/Kconfig
@@ -40,3 +40,16 @@ config SIFIVE_FU740
 	imply DM_I2C
 	imply SYS_I2C_OCORES
 	imply SPL_I2C
+
+if ENV_IS_IN_SPI_FLASH
+
+config ENV_OFFSET
+	default 0x505000
+
+config ENV_SIZE
+	default 0x20000
+
+config ENV_SECT_SIZE
+	default 0x10000
+
+endif # ENV_IS_IN_SPI_FLASH
diff --git a/arch/riscv/dts/hifive-unmatched-a00-u-boot.dtsi b/arch/riscv/dts/hifive-unmatched-a00-u-boot.dtsi
index c5475aa149..1ee8ab1868 100644
--- a/arch/riscv/dts/hifive-unmatched-a00-u-boot.dtsi
+++ b/arch/riscv/dts/hifive-unmatched-a00-u-boot.dtsi
@@ -16,6 +16,10 @@
 		u-boot,dm-spl;
 	};
 
+	config {
+		u-boot,spl-payload-offset = <0x105000>; /* loader2 @1044KB */
+	};
+
 	hfclk {
 		u-boot,dm-spl;
 	};
@@ -30,6 +34,13 @@
 	clocks = <&rtcclk>;
 };
 
+&qspi0 {
+	u-boot,dm-spl;
+	flash@0 {
+		u-boot,dm-spl;
+	};
+};
+
 &spi0 {
 	mmc@0 {
 		u-boot,dm-spl;
diff --git a/board/sifive/unmatched/Kconfig b/board/sifive/unmatched/Kconfig
index fb2c1fbb58..fe213fd504 100644
--- a/board/sifive/unmatched/Kconfig
+++ b/board/sifive/unmatched/Kconfig
@@ -26,6 +26,7 @@ config SPL_OPENSBI_LOAD_ADDR
 config BOARD_SPECIFIC_OPTIONS # dummy
 	def_bool y
 	select SIFIVE_FU740
+	select ENV_IS_IN_SPI_FLASH
 	select SUPPORT_SPL
 	select RESET_SIFIVE
 	select BINMAN
diff --git a/board/sifive/unmatched/spl.c b/board/sifive/unmatched/spl.c
index d5663274cd..7c0beedc08 100644
--- a/board/sifive/unmatched/spl.c
+++ b/board/sifive/unmatched/spl.c
@@ -22,6 +22,7 @@
 #define GEM_PHY_RESET	SIFIVE_GENERIC_GPIO_NR(0, 12)
 
 #define MODE_SELECT_REG		0x1000
+#define MODE_SELECT_SPI		0x6
 #define MODE_SELECT_SD		0xb
 #define MODE_SELECT_MASK	GENMASK(3, 0)
 
@@ -123,6 +124,8 @@ u32 spl_boot_device(void)
 	u32 boot_device = mode_select & MODE_SELECT_MASK;
 
 	switch (boot_device) {
+	case MODE_SELECT_SPI:
+		return BOOT_DEVICE_SPI;
 	case MODE_SELECT_SD:
 		return BOOT_DEVICE_MMC1;
 	default:
diff --git a/configs/sifive_unmatched_defconfig b/configs/sifive_unmatched_defconfig
index 1dde98e0ae..28deb1f481 100644
--- a/configs/sifive_unmatched_defconfig
+++ b/configs/sifive_unmatched_defconfig
@@ -6,6 +6,7 @@ CONFIG_SPL_DM_SPI=y
 CONFIG_DEFAULT_DEVICE_TREE="hifive-unmatched-a00"
 CONFIG_SPL_MMC_SUPPORT=y
 CONFIG_SPL=y
+CONFIG_SPL_SPI_FLASH_SUPPORT=y
 CONFIG_SPL_SPI_SUPPORT=y
 CONFIG_AHCI=y
 CONFIG_TARGET_SIFIVE_UNMATCHED=y
@@ -20,7 +21,9 @@ CONFIG_DISPLAY_CPUINFO=y
 CONFIG_DISPLAY_BOARDINFO=y
 CONFIG_DISPLAY_BOARDINFO_LATE=y
 CONFIG_SPL_SEPARATE_BSS=y
+CONFIG_SPL_DM_SPI_FLASH=y
 CONFIG_SPL_DM_RESET=y
+CONFIG_SPL_SPI_LOAD=y
 CONFIG_SPL_YMODEM_SUPPORT=y
 CONFIG_CMD_EEPROM=y
 CONFIG_CMD_MEMINFO=y
@@ -28,10 +31,13 @@ CONFIG_CMD_PWM=y
 CONFIG_CMD_GPT_RENAME=y
 CONFIG_CMD_PCI=y
 CONFIG_CMD_USB=y
+CONFIG_USE_ENV_SPI_BUS=y
+CONFIG_ENV_SPI_BUS=1
 CONFIG_SYS_RELOC_GD_ENV_ADDR=y
 CONFIG_SCSI_AHCI=y
 CONFIG_AHCI_PCI=y
 CONFIG_SPL_CLK=y
+CONFIG_SPI_FLASH_ISSI=y
 CONFIG_E1000=y
 CONFIG_NVME=y
 CONFIG_PCI=y

I am now using partitions the same as the Unleashed which are aligned to make them easier to update. I have used the following script to create the correct partitions in flash and copy the FSBL (u-boot-spl.bin) and u-boot into their partitions:

#!/bin/sh

sgdisk -g --clear -a 1 \
  --new=1:40:2087         --change-name=1:spl --typecode=1:5B193300-FC78-40CD-8002-E86C45580B47 \
  --new=2:2088:10279      --change-name=2:uboot  --typecode=2:2E54B353-1271-4842-806F-E436D6AF6985 \
  --new=3:10280:10535	  --change-name=3:env   --typecode=3:0FC63DAF-8483-4772-8E79-3D69D8477DE4 \
  /dev/mtdblock0

partprobe

dd if=u-boot-spl.bin of=/dev/mtdblock0 bs=4096 seek=5 conv=sync
dd if=u-boot.itb  of=/dev/mtdblock0 bs=4096 seek=261 conv=sync

2 Likes

I also added these patches to 1. enable fallback booting from SD (useful for testing and if Flash contains errors) and 2. a configuration file for genimage to automatically build a flash image.
This can then be flashed in linux using flash_erase /dev/mtd0 0 0 && dd if=spi-nor.img of=/dev/mtd0 bs=4K status=progress.
Note that these offsets are still for the first path of skibo (u-boot,spl-payload-offset = <0x104400>;)

diff --git a/board/sifive/unmatched/spl.c b/board/sifive/unmatched/spl.c
index d5663274cd..c6aa3c15af 100644
--- a/board/sifive/unmatched/spl.c
+++ b/board/sifive/unmatched/spl.c
@@ -22,6 +22,7 @@
 #define GEM_PHY_RESET  SIFIVE_GENERIC_GPIO_NR(0, 12)
 
 #define MODE_SELECT_REG                0x1000
+#define MODE_SELECT_SPI                0x6
 #define MODE_SELECT_SD         0xb
 #define MODE_SELECT_MASK       GENMASK(3, 0)
 
@@ -123,6 +124,8 @@ u32 spl_boot_device(void)
        u32 boot_device = mode_select & MODE_SELECT_MASK;
 
        switch (boot_device) {
+       case MODE_SELECT_SPI:
+               return BOOT_DEVICE_SPI;
        case MODE_SELECT_SD:
                return BOOT_DEVICE_MMC1;
        default:
@@ -132,6 +135,15 @@ u32 spl_boot_device(void)
        }
 }
 
+// overriding spl.c
+void board_boot_order(u32 *spl_boot_list)
+{
+       spl_boot_list[0] = spl_boot_device();
+       // enable fallback to booting from SD-Card
+       spl_boot_list[1] = BOOT_DEVICE_MMC1;
+}
diff --git a/board/sifive/unmatched/genimage_spi-nor.cfg b/board/sifive/unmatched/genimage_spi-nor.cfg
new file mode 100644
index 0000000000..8e7e688101
--- /dev/null
+++ b/board/sifive/unmatched/genimage_spi-nor.cfg
@@ -0,0 +1,25 @@
+image spi-nor.img {
+       size = 32M
+
+       hdimage {
+               gpt = true
+       }
+
+       partition u-boot-spl {
+               image = "u-boot-spl.bin"
+               offset = 34K
+               partition-type-uuid = 5B193300-FC78-40CD-8002-E86C45580B47
+       }
+
+    partition u-boot {
+        image = "u-boot.itb"
+        offset = 2082K
+        partition-type-uuid = 2E54B353-1271-4842-806F-E436D6AF6985
+    }
+
+    partition env {
+        offset = 5140K
+        size = 512K
+        partition-type-uuid = 0FC63DAF-8483-4772-8E79-3D69D8477DE4
+    }
+}

Booting from flash patches have been committed to U-Boot. Nice.

1 Like

Since U-boot support has been integrated, is there now a simpler process for enabling booting from SPI flash with a recent SiFive software release? Ideally, without needing to rebuild anything?

I recently acquired a used board with a broken MicroSD card latch, so I’m looking for a way to boot the machine without have to hold the MicroSD card in place.

Thanks!

I think it can be done, at least with Ubuntu 22.04. First, download the latest u-boot .deb file.

http://launchpadlibrarian.net/593560393/u-boot-sifive_2022.01+dfsg-2ubuntu2_riscv64.deb

Then extract the .deb with dpkg-deb.

dpkg-deb -x u-boot-sifive_2022.01+dfsg-2ubuntu2_riscv64.deb /tmp

The files will be in /tmp/usr/lib/u-boot/sifive_unmatched/

Load the MTD block driver.

sudo modprobe mtdblock

And then write the flash.

sudo sgdisk -g --clear -a 1 \
  --new=1:40:2087         --change-name=1:spl --typecode=1:5B193300-FC78-40CD-8002-E86C45580B47 \
  --new=2:2088:10279      --change-name=2:uboot  --typecode=2:2E54B353-1271-4842-806F-E436D6AF6985 \
  --new=3:10280:10535	  --change-name=3:env   --typecode=3:0FC63DAF-8483-4772-8E79-3D69D8477DE4 \
  /dev/mtdblock0

sudo partprobe

sudo dd if=u-boot-spl.bin of=/dev/mtdblock0 bs=4096 seek=5 conv=sync
sudo dd if=u-boot.itb  of=/dev/mtdblock0 bs=4096 seek=261 conv=sync

Flip the boot DIP switches to 6 and reboot.

1 Like