Microsemi expansion board support/issue

I got my Microsemi PCIe root complex expansion board recently. I am, however, having issues with getting Linux to report the existence of the root complex and the PCIe switch and AHCI controller built into the board.
The first thing I noticed was that the kernel my board came with (my HiFive unleashed came bundled with the expansion board) predated the git commits adding support for the expansion board. So I enabled the Microsemi PCIe host device in the SDK’s linux kernel and built the SDK.
However that still did not work. So I took a quick look at the Microsemi driver. It looks like it requires entries in the DTB for the Microsemi root complex (which sound right since afaik chiplink is just a memory interface and provides no dev enumeration & configuration functionality). From the soc docs it sounds like the DTB is embedded in the FSBL. Is the source available for that? Or even just the source for the DTB, Or a mechanism to update the DTB?

P.S.
If you are going to be building kernels to distribute to others, it would be nice if you ensure that the CONFIG_IKCONFIG and CONFIG_IKCONFIG_PROC, kernel config options are set so that people can quickly and easily determine what functionality the currently running kernel has, as well as build the same kernel.

The microsemi device tree info can be found here


cbe684f0d6bdb3b7f0a432

There is a way to upgrade the FSBL, which is documented in the HiFive Unleashed Getting Started Guide. This requires flipping a few switches on the board and putting in an SDCard that has updated FSBL. Unfortunately, we haven’t released an updated FSBL that has the Microsemi device tree info in it.

We are in the process of making FSBL sources available. Hopefully they will be available soon.

1 Like

First of all, Thanks for the repo link. That was the dts I was looking for.

However I did want to point out that the initial commit of the dts file in that repo seems to imply that the HiFive unleashed shipped with a DTS including support for the Microsemi root complex, but I just cross-compiled dtc and use it to generate a dts from /sys/firmware/devicetree/base and while I see the xilinx root complex in there, I don’t see the microsemi one, so the commit seems a bit miss leading.

Also, If I understand correctly, barring reversing the fsbl (or implementing kexec support for riscv, lol), the expansion board can’t be used until either the fsbl source is released or an fsbl update is published?

I’m a compiler guy, not a linux kernel guy, so I don’t know the details of what is going on here. But I have the same problem, I bought a board and it isn’t usable, so I’ve been asking questions. I have a pointer to the microsemi driver and a pointer to the microsemi device tree entry, but I don’t really know what to do with them yet. I do know that a guy on the freenode #riscv IRC managed to get his board working, but I don’t know what he did to make it work. He is a linux kernel developer though, so he apparently found a way to work around the FSBL problem. Maybe you can build the device tree into the kernel instead of using the FSBL one?

Anyways, yes, there is a problem here. We need a little more documentation and infrastructure to help people who bought one of these Microsemi expansion boards figure out how to use them.

I just realized that I should be able to modify the bbl to have an embedded dtb (since it’s the bbl that hands it to the kernel). So I’m working on modifying the bbl. it should be as simple as linking in the binary, and switching out where the dtb/fdt pointer points (the init asm from the bbl passes the address of the dtb to the first bbl C function).

Size note: kexec (at least on arm… and it’s unimplemented on riscv so it’s kinda undefined atm) allows you to pass in a dtb to the new kernel you are going to load. While I have done a fair amount of kernel dev, I dont feel up to the task if implemented kexec for riscv :stuck_out_tongue:.

Probably part of the problem is there are only (according to crowdsupply) 23 of us with the expansion board.

Ok I got it work (in that devices show up in lspci). I have not tried using any devices. I don’t have a m.2 hd or a sata HD (that I’m willing to sacrifice) on hand right now.
I was able to get it to work by patching the bbl to optionally accept a dts to compile and embedd. However I did have to make some modification to the dts in the repo linked above. I imagine most of the things I had to change would normally be populated by the FSBL.

If people are interesting I could post patches (after I clean them up a bit).

1 Like

Definitely, please do. :slight_smile:

Looks like tarballs are not allowed, so copy&paste it is. just FYI in the interest of time. I did not filter out all the changes I made to the linux kernel config that were not related to the microsemi root complext. However the should not cause any harm

README:

Disclaimer. This is a rough patch set, not very well tested. I obviously do not accept responsiblity if using this damages any of your hardware.

This package should contain these three files:
riscv-device-tree-doc-0001-add-compile-time-optional-dts-override.patch
riscv-pk-0001-add-compile-time-optional-dts-override.patch
sdk-0001-add-compile-time-optional-dts-override.patch

each patch should apply to one git repo. 
The sdk-* patch applies to the main SiFive freedom-u-sdk repo.
The riscv-pk-* patch applies to the riscv-pk submodule of the freedom-u-sdk repo.
the riscv-device-tree-doc-* should be applies the repo it's name starts with (it's avaible on github with that name, and a link to it is in the forum thread this archive was first posted on).

I strongly recommend you take a quick glance at the patches before applying them. They do have comments explaining a few of the things in them. Of espeical notice is the 'local-mac-address' in the dts, you need to set that to your boards current mac.
After you have applied the patches there is one thing you need to do, and one optional but strongly recommended step:
    first you need to cd into the riscv-pk dir and run "autoreconf" (you will need the corresponding autotools pkgs installed from your distro). Failure to do this will prevent the changes from properly taking effect.
    optionally (it doesn't matter if you'd applied your mac changes yet): I recommend you either build the sdk (if you have not done so already), or run "rm -rf work/bbl.* work/riscv-pk work/linux" in the root of your sdk (to force rebuild of applical submodules), and then run the make again. Afterwords you should install the resulting bbl.bin to you sdcard and let you hifive unleased boot up. Before the linux kernel starts, it should spit out the dts that the FSBL handed off to the bbl (ok there are a few changes that the bbl made to it, but thats not a huge issue). You may want to compare the dts patch against the values in that dts (make sure the values I changed result in matches in your origional dts).

Finally there are two make to build with the dts override. you can either export "dts_source" in your shell and point the enviorment varial to the localtion of the patched dts in the riscv-device-tree-doc repo, or you can uncomment the dts_source variable in the Makefile and point it to the right location. If your sdk repo is not clean, simply run an "rm -rf work/bbl.* work/riscv-pk work/linux". Then filnally run the make and install the resulting bbl.
Good luck

sdk-0001-add-compile-time-optional-dts-override.patch

diff --git a/Makefile b/Makefile
index 39d8b45..df7251c 100644
--- a/Makefile
+++ b/Makefile
@@ -28,6 +28,11 @@ linux_srcdir := $(srcdir)/linux
 linux_wrkdir := $(wrkdir)/linux
 linux_defconfig := $(confdir)/linux_defconfig
 
+#dts_source := /data/src/riscv-device-tree-doc/examples/sifive-hifive_unleashed-microsemi.dts
+# You can either specify  the dts_source as an enviorment varial,
+# or uncomment this line and point it at the file
+# if you do neither, than bbl will NOT be built with a dts override
+#dts_source := /path/to/dts/file
 vmlinux := $(linux_wrkdir)/vmlinux
 vmlinux_stripped := $(linux_wrkdir)/vmlinux-stripped
 
@@ -53,6 +58,14 @@ rootfs := $(wrkdir)/rootfs.bin
 
 target := riscv64-unknown-linux-gnu
 
+# Comment out this line to disable device tree printing
+CONFIG_PRINT_DTB=--enable-print-device-tree
+ifneq ($(dts_source),)
+
+CONFIG_USE_DTS=--with-dts=$(dts_source)
+
+endif
+
 .PHONY: all
 all: $(hex)
 	@echo
@@ -154,6 +167,8 @@ $(bbl): $(pk_srcdir) $(vmlinux_stripped)
 	cd $(pk_wrkdir) && $</configure \
 		--host=$(target) \
 		--with-payload=$(vmlinux_stripped) \
+		$(CONFIG_USE_DTS) \
+		$(CONFIG_PRINT_DTB) \
 		--enable-logo \
 		--with-logo=$(abspath conf/sifive_logo.txt)
 	CFLAGS="-mabi=$(ABI) -march=$(ISA)" $(MAKE) -C $(pk_wrkdir)
diff --git a/conf/linux_defconfig b/conf/linux_defconfig
index cd87340..8c2b7eb 100644
--- a/conf/linux_defconfig
+++ b/conf/linux_defconfig
@@ -2,22 +2,29 @@ CONFIG_SMP=y
 # CONFIG_COMPACTION is not set
 CONFIG_HZ_100=y
 CONFIG_PCI=y
+CONFIG_PCI_DMA_32=y
+CONFIG_PCIEPORTBUS=y
 CONFIG_PCI_DEBUG=y
-CONFIG_PCIE_XILINX=y
+CONFIG_PCIE_MICROSEMI=y
+CONFIG_PCI_SW_SWITCHTEC=y
 CONFIG_CROSS_COMPILE="riscv64-unknown-linux-gnu-"
 CONFIG_DEFAULT_HOSTNAME="ucbvax"
 # CONFIG_CROSS_MEMORY_ATTACH is not set
+CONFIG_IKCONFIG=y
+CONFIG_IKCONFIG_PROC=y
 CONFIG_NAMESPACES=y
 CONFIG_BLK_DEV_INITRD=y
 # CONFIG_RD_BZIP2 is not set
 # CONFIG_RD_LZMA is not set
-# CONFIG_RD_XZ is not set
 # CONFIG_RD_LZO is not set
 # CONFIG_RD_LZ4 is not set
 CONFIG_EMBEDDED=y
+CONFIG_MODULES=y
+CONFIG_MODULE_UNLOAD=y
 # CONFIG_BLK_DEV_BSG is not set
 CONFIG_PARTITION_ADVANCED=y
 # CONFIG_IOSCHED_DEADLINE is not set
+CONFIG_BINFMT_MISC=y
 CONFIG_NET=y
 CONFIG_PACKET=y
 CONFIG_UNIX=y
@@ -27,6 +34,7 @@ CONFIG_INET=y
 # CONFIG_INET_XFRM_MODE_BEET is not set
 # CONFIG_INET_DIAG is not set
 # CONFIG_IPV6 is not set
+# CONFIG_UEVENT_HELPER is not set
 CONFIG_DEVTMPFS=y
 CONFIG_DEVTMPFS_MOUNT=y
 # CONFIG_FIRMWARE_IN_KERNEL is not set
@@ -34,19 +42,24 @@ CONFIG_MTD=y
 CONFIG_MTD_BLOCK_RO=y
 CONFIG_MTD_M25P80=y
 CONFIG_MTD_SPI_NOR=y
-CONFIG_VIRTIO_BLK=y
+CONFIG_BLK_DEV_LOOP=y
 CONFIG_EEPROM_AT24=y
 CONFIG_BLK_DEV_SD=y
 CONFIG_ATA=y
 CONFIG_SATA_AHCI=y
 CONFIG_SATA_SIL24=y
+CONFIG_SATA_MV=y
 CONFIG_NETDEVICES=y
-CONFIG_VIRTIO_NET=y
 CONFIG_MACB=y
+CONFIG_E1000E=y
+CONFIG_R8169=y
 CONFIG_MICROSEMI_PHY=y
+CONFIG_INPUT_MOUSEDEV=m
+CONFIG_INPUT_JOYDEV=m
+CONFIG_INPUT_EVDEV=m
 # CONFIG_INPUT_KEYBOARD is not set
 # CONFIG_INPUT_MOUSE is not set
-# CONFIG_VT is not set
+# CONFIG_LEGACY_PTYS is not set
 CONFIG_SERIAL_8250=y
 CONFIG_SERIAL_8250_CONSOLE=y
 # CONFIG_SERIAL_8250_PCI is not set
@@ -54,10 +67,11 @@ CONFIG_SERIAL_OF_PLATFORM=y
 CONFIG_SERIAL_SIFIVE=y
 CONFIG_SERIAL_SIFIVE_CONSOLE=y
 # CONFIG_HVC_RISCV_SBI is not set
-CONFIG_VIRTIO_CONSOLE=y
 # CONFIG_HW_RANDOM is not set
 CONFIG_I2C=y
 CONFIG_I2C_OCORES=y
+CONFIG_I2C_SLAVE=y
+CONFIG_I2C_SLAVE_EEPROM=m
 CONFIG_SPI=y
 CONFIG_SPI_SIFIVE=y
 CONFIG_GPIOLIB=y
@@ -65,10 +79,34 @@ CONFIG_GPIO_SYSFS=y
 CONFIG_GPIO_SIFIVE=y
 CONFIG_POWER_RESET=y
 CONFIG_POWER_RESET_GPIO_RESTART=y
+CONFIG_VGA_ARB_MAX_GPUS=4
+CONFIG_DRM=m
+CONFIG_DRM_RADEON=m
+CONFIG_DRM_AMDGPU=m
+CONFIG_FRAMEBUFFER_CONSOLE=y
+CONFIG_LOGO=y
+CONFIG_UHID=m
 CONFIG_USB=y
 CONFIG_USB_XHCI_HCD=y
 CONFIG_USB_EHCI_HCD=y
 CONFIG_USB_STORAGE=y
+CONFIG_USB_GADGET=y
+CONFIG_USB_NET2272=y
+CONFIG_USB_NET2280=y
+CONFIG_USB_CONFIGFS=y
+CONFIG_USB_CONFIGFS_SERIAL=y
+CONFIG_USB_CONFIGFS_ACM=y
+CONFIG_USB_CONFIGFS_OBEX=y
+CONFIG_USB_CONFIGFS_NCM=y
+CONFIG_USB_CONFIGFS_ECM=y
+CONFIG_USB_CONFIGFS_ECM_SUBSET=y
+CONFIG_USB_CONFIGFS_RNDIS=y
+CONFIG_USB_CONFIGFS_EEM=y
+CONFIG_USB_CONFIGFS_MASS_STORAGE=y
+CONFIG_USB_CONFIGFS_F_LB_SS=y
+CONFIG_USB_CONFIGFS_F_FS=y
+CONFIG_USB_CONFIGFS_F_HID=y
+CONFIG_USB_CONFIGFS_F_PRINTER=y
 CONFIG_MMC=y
 CONFIG_MMC_SPI=y
 CONFIG_NEW_LEDS=y
@@ -85,15 +123,22 @@ CONFIG_CLK_GEMGXL_MGMT=y
 CONFIG_PWM=y
 CONFIG_PWM_SIFIVE=y
 CONFIG_NVMEM_SIFIVE_U500_OTP=y
-CONFIG_EXT3_FS=y
+CONFIG_EXT4_FS=y
+CONFIG_FUSE_FS=m
+CONFIG_CUSE=m
+CONFIG_OVERLAY_FS=m
+CONFIG_MSDOS_FS=m
+CONFIG_VFAT_FS=m
 # CONFIG_PROC_PAGE_MONITOR is not set
 CONFIG_TMPFS=y
 # CONFIG_MISC_FILESYSTEMS is not set
 # CONFIG_NETWORK_FILESYSTEMS is not set
 CONFIG_CMDLINE_BOOL=y
-CONFIG_CMDLINE="earlyprintk"
+CONFIG_CMDLINE="earlyprintk console=ttySI0"
+CONFIG_CMDLINE_OVERRIDE=y
 CONFIG_PRINTK_TIME=y
 CONFIG_STRIP_ASM_SYMS=y
 CONFIG_DEBUG_SECTION_MISMATCH=y
 CONFIG_STACKTRACE=y
+CONFIG_CRYPTO_ECHAINIV=y
 # CONFIG_CRYPTO_HW is not set

riscv-device-tree-doc-0001-add-compile-time-optional-dts-override.patch

diff --git a/examples/sifive-hifive_unleashed-microsemi.dts b/examples/sifive-hifive_unleashed-microsemi.dts
index ddad3e2..126a9de 100644
--- a/examples/sifive-hifive_unleashed-microsemi.dts
+++ b/examples/sifive-hifive_unleashed-microsemi.dts
@@ -15,7 +15,7 @@
 	};
 
 	firmware {
-		sifive,fsbl = "YYYY-MM-DD";
+		sifive,fsbl = "2018-03-20";
 	};
 
 	L3: cpus {
@@ -152,7 +152,7 @@
 	};
 	L36: memory@80000000 {
 		device_type = "memory";
-		reg = <0x0 0x80000000 0x1f 0x80000000>;
+		reg = <0x0 0x80000000 0x02 0x0>;
 	};
 	L2: soc {
 		#address-cells = <2>;
@@ -166,7 +166,7 @@
 			clock-output-names = "xtal";
 		};
 		prci: prci@10000000 {
-			compatible = "sifive,ux00prci0";
+			compatible = "sifive,aloeprci0";
 			reg = <0x0 0x10000000 0x0 0x1000>;
 			reg-names = "control";
 			clocks = <&refclk>;
@@ -258,7 +258,9 @@
 			interrupts = <53>;
 			reg = <0x0 0x10090000 0x0 0x2000>;
 			reg-names = "control";
-
+            /* Modify to match you mac, normally the FSBL would do this for you
+             * however by overriding the dts in the bbl we are losing that
+             * functionality*/
 			local-mac-address = [00 00 00 00 00 00];
 			phy-mode = "gmii";
 			clock-names = "pclk", "hclk", "tx_clk";
@@ -385,7 +387,7 @@
 			reg-names = "mem";
 		};
 		L37: memory-controller@100b0000 {
-			compatible = "sifive,ux00ddr0";
+			compatible = "sifive,aloeddr0";
 			interrupt-parent = <&L4>;
 			interrupts = <31>;
 			reg = <0x0 0x100b0000 0x0 0x4000>;

riscv-pk-0001-add-compile-time-optional-dts-override.patch

diff --git a/Makefile.in b/Makefile.in
index 0268629..4d4b7d9 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -78,6 +78,13 @@ VPATH := $(addprefix $(src_dir)/, $(sprojs_enabled))
 # Programs and flags 
 #-------------------------------------------------------------------------
 
+DTS_SOURCE = @DTS_SOURCE@
+ifneq ($(DTS_SOURCE),)
+
+MACHINE_DTS_ASM_SRC=dtbpayload.S
+DTS_USE_CFLAGS=-DDTB_BIN=\"emb.dtb\"
+endif
+
 # C++ compiler
 #  - CPPFLAGS : flags for the preprocessor (eg. -I,-D)
 #  - CXXFLAGS : flags for C++ compiler (eg. -Wall,-g,-O3)
@@ -85,10 +92,13 @@ VPATH := $(addprefix $(src_dir)/, $(sprojs_enabled))
 CC            := @CC@
 READELF       := @READELF@
 OBJCOPY       := @OBJCOPY@
-CFLAGS        := @CFLAGS@ $(CFLAGS) -DBBL_PAYLOAD=\"bbl_payload\" -DBBL_LOGO_FILE=\"bbl_logo_file\"
+CFLAGS        := @CFLAGS@ $(CFLAGS) $(DTS_USE_CFLAGS) -DBBL_PAYLOAD=\"bbl_payload\" -DBBL_LOGO_FILE=\"bbl_logo_file\"
 BBL_PAYLOAD   := @BBL_PAYLOAD@
 COMPILE       := $(CC) -MMD -MP $(CFLAGS) \
                  $(sprojs_include)
+
+
+
 # Linker
 #  - LDFLAGS : Flags for the linker (eg. -L)
 #  - LIBS    : Library flags (eg. -l)
diff --git a/machine/dtbpayload.S b/machine/dtbpayload.S
new file mode 100644
index 0000000..f352c9a
--- /dev/null
+++ b/machine/dtbpayload.S
@@ -0,0 +1,6 @@
+
+  .section ".data"
+  .globl _dtb_start, _dtb_end
+_dtb_start:
+  .incbin DTB_BIN
+_dtb_end:
diff --git a/machine/machine.ac b/machine/machine.ac
index 65acf04..d59e2b0 100644
--- a/machine/machine.ac
+++ b/machine/machine.ac
@@ -1,4 +1,7 @@
 AC_ARG_ENABLE([fp-emulation], AS_HELP_STRING([--disable-fp-emulation], [Disable floating-point emulation]))
 AS_IF([test "x$enable_fp_emulation" != "xno"], [
   AC_DEFINE([PK_ENABLE_FP_EMULATION],,[Define if floating-point emulation is enabled])
+
+AC_ARG_WITH([payload], AS_HELP_STRING([--with-dts], [Set an embedded dtb ]),
+  [AC_SUBST([DTS_SOURCE], $with_dts, [dts for bbl's embedded dtb])], )
 ])
diff --git a/machine/machine.mk.in b/machine/machine.mk.in
index d6829eb..a711a6a 100644
--- a/machine/machine.mk.in
+++ b/machine/machine.mk.in
@@ -34,4 +34,12 @@ machine_c_srcs = \
 
 machine_asm_srcs = \
   mentry.S \
+  $(MACHINE_DTS_ASM_SRC) \
   fp_asm.S \
+
+
+dtbpayload.o: emb.dtb
+
+
+emb.dtb: $(DTS_SOURCE)
+	dtc -I dts -O dtb -o emb.dtb $(DTS_SOURCE)
diff --git a/machine/minit.c b/machine/minit.c
index cd909f3..bb96618 100644
--- a/machine/minit.c
+++ b/machine/minit.c
@@ -137,13 +137,27 @@ static void wake_harts()
       *OTHER_HLS(hart)->ipi = 1; // wakeup the hart
 }
 
+#ifdef DTB_BIN
+extern uint32_t _dtb_start;
+extern uint32_t _dtb_end;
+#endif
+
 void init_first_hart(uintptr_t hartid, uintptr_t dtb)
 {
+    uintptr_t dtb_save = dtb;
+#if defined(DTB_BIN)
+    dtb = (uintptr_t)&_dtb_start;
+#endif
   // Confirm console as early as possible
   query_uart(dtb);
   query_uart16550(dtb);
   query_htif(dtb);
   printm("bbl loader\r\n");
+  printm("dtb_save: %0x\r\n", dtb_save);
+  printm("dtb:      %0x\r\n", dtb);
+#if defined(DTB_BIN)
+  printm("dtb_end:  %0x\r\n", &_dtb_end);
+#endif
 
   hart_init();
   hls_init(0); // this might get called again from parse_config_string

EDIT: In the even the FSBL either has source released or a FSBL binary update is release with support for the microsemi expansion board, use that instead of these patches

This seems like an issue we should solve, so that these sorts of hacks are not required. I am going to work on preparing an updated Unleashed firmware image and blessed gateware images for vc707+microsemi (built from freedom) that work together. I’ll post again here when the matching versions are up on our website.

FYI, This solution will require reprogramming the FPGA, which I hope everyone here is comfortable doing.

@dullfire: That’s a really cool approach.

I made it to work by adding microsemi dt entry in bbl dynamically.
I have verified with a display attached via RADEON 650 GPU.

In case anybody wants to try the dynamic approach
Here are the patches:


Note: You still the Linux config changes & microsemi driver in Linux.

1 Like

@atish Thanks.
After taking a quick look at your repo, patching in the dt entry does seem like a better approach as far as usability by others. It’s also much less error prone (no need for board specific changes)

@terpstra
That would be great. We need DT changes in future as well to support different kernel features (to name a few . timer interrupt or cpu topology). How difficult is to update the firmware image everytime?

Otherwise we have to make the necessary changes in bbl either with libfdt support (my patch) or static linking approach as suggested by @dullfire.

Yeah but it links entire libfdt into bbl. So it increases bbl size as well.
Thus, I am not sure if that is going to be accepted.

@atish I think those sort of DTS changes will be best done by modifying the open-source version of the FSBL as soon as it’s ready. The reason I am working on a patch for this is because you need a matching hardware change to be able to distinguish between the FPGA boards attached via ChipLink. Software alone can’t solve this problem, unless you are willing to program a HiFive Unleashed to only work with one expansion board (the current situation).

Curious what the latest is on this?

@AndrewBack Since I haven’t seen anything from @terpstra about an updated firmware release, it looks like we are waiting on SiFive to release the FSBL code and/or an updated firnware image.

If you do not want to wait for the official release, you have two choices. If you are comfortable tinkering with the device tree, you can use my patches posted above to make the Microsemi expansion board work. I have test it on a PLX USB2232 (a usb gadget device) and was able to get it to communicate with another system.

However if you want something simpler, @atish posted a link to a git fork that he has that will patch in just the needed device tree nodes for you. I have not personally tested it, but to me it looks like the only down sides would be pulling in libfdt and the corresponding size increase.

Sorry, believe it or not I’ve been working on this issue pretty much the entire time since I posted. I am still trying to prepare an image that is suitable for release.

4 Likes

Hi! Did somebody try to update a firmware in the FPGA? I tried to do that through JTAG interface but it failed with “Signal Integrity Failure” and I do not know what to do :frowning: I was never work with JTAG before. Could somebody help me with that?

@terpstra just thought I’d check in and see what the status on this was?

1 Like

I’m also interested in hearing the status on this @terpstra