We want to host one or two HiFive Unleashed boards at a datacenter. For this, the requirement is that they are rack-mounted in a 19" rack. (We don’t use the Microsemi expansion board).
So I have a rough idea about how to go about this, but with some problems that I need help with.
I’m going to get an empty 1U or 2U 19" case off ebay.
I will use an RPi 3 or similar as an NBD server / power / serial server inside the case. This will be connected via USB ports to the board to provide power to the barrel plug (from USB) and serial port.
The problem is remote reset (of the SiFive board). We have found that they hang under the very heavy build loads we give them (which is likely a kernel issue, rather than hardware). It requires resetting the boards manually using the switch every few days. I want to make this work remotely. However this is where it gets adventurous - I could desolder the switch and wire it to a relay, controlled by GPIO pins on the RPi. But desoldering a switch on a $1000 board is somewhat risky. Is there an easier way to provide remote reset?
There will be 2+ ethernet cables coming out of the back, but that’s likely OK. Fitting a network switch inside the case is I guess also possible, but it starts to get cramped.
There is a theory that the flash bug might have been fixed when the flash driver was rewritten in 4.16. They have been trying to reproduce with 4.19 and not succeeding. However, it is possible that they aren’t driving the boards hard enough to reproduce.
My board only reboots when I’m testing a new kernel. It has been up for weeks at a time when I wasn’t trying kernel patches. I do lots of make -j4 builds and checks. I’m using NBD for root and running without swap at 1GHz. If you are using flash for swap, that might be causing the problem.
TBH whether or not it’s a kernel or hardware bug isn’t material here. We will undoubtedly want to hard reset the board at some point, and when it’s installed in the data center across the other side of London that won’t be a simple matter unless we can do it remotely.
David noticed that the reset and power button lines are exposed on the 96boards header, so it looks like we can wire something to pull one of those to ground. At least no desoldering involved …
Regarding powering your boards from USB ports: I’m not sure what you have in mind, but two observations:
The barrel plugs on the HiFiveU boards expect 12Vdc, but USB ports only supply 5Vdc, unless special step-up converters are used.
The power supply that we ship with the HiFiveU boards is rated to supply up to 24W. Most USB ports can only deliver 0.5W without special negotiation. I’m sure the HiFiveU board doesn’t use 24W on average, but I suspect it does draw more than 0.5W, particularly when the board is first switched on.
Some more general testbed comments:
We’ve got several HiFiveU boards in a rack here. We use an Ethernet-controlled power switch to power them on and off. I think this is preferable to using the warm reset pushbutton, if only because a hard power cycle with long enough off-duration is guaranteed to put the board into a known state: particularly in the face of any reset-related hardware bugs. We already suspect we have at least one of these bugs. Plus, there is no soldering needed.
Here are several AC or DC power control systems that I have tested and that are known to work. This is not an endorsement; I do not have an economic relationship with any of these companies (beyond having worked with or purchased their products):
In our testbed, we use 2U rack shelves and are currently placing three HiFiveU boards per rack shelf. We attach the HiFiveU boards to ventilation slots on the rack shelves at three points per board, to prevent the boards from moving around in the rack. We use M3 standoffs/spacers and two 1/8"-1" fender washers per spacer to keep everything together. There are probably better ways to do it but this has been working well so far.
If a SiFive blog post on the topic would be useful, please let us know.
[Edited to fix USB power supply limits - h/t to Bruce for spotting]
Yes definitely a blog about this would be great! Sounds right that both ethernet power control and not using USB for power would be good. What do you use as a file server / serial port server (if anything)?
I’m working on a similar problem. I’m using the expansion board, but the management controller I’m designing (using a HiFive 1) is going to connect to the main board and not the expansion. Remote power control and hard/soft reset are two of the main issues I’m planning on tackling. There are some RESET and POWER lines attached to the 2x20 low speed connector, and I’ve just got my boards today so I have not done much work on them. They may offer a solution. Otherwise, I’m probably going to add either a MOSFET or relay to control power into the barrel connector instead. I’ll be making my designs open source under a very permissive license (MIT or Apache 2 most likely) so others can benefit. I’ll be sure to post any findings I come across.
Regarding what we use as file servers / serial port servers: as Jim wrote, we tend to use NUCs, but only because they were easily available to us. If you have flexibility in this area, would suggest considering a machine with multiple PCIe-connected Ethernet interfaces, remote manageability, and more storage options.
Yes indeed, I’m using a NUC as NBD server at the moment. Would like to move to something cheaper, but an RPi isn’t suitable (I realized) since it has poor SATA support. The UDOO x86 looks interesting.