It would be great to see the source for the HiFive1B bootloader similar to what was posted for the HiFive1 double-tap bootloader discussed in this thread Bootloader restore.
If there are expectations that the HiFive1B bootloader is making with respect to what is running on the ESP32, I would like to see that.
Honestly, I was confused seeing ESP32 spew coming from the UART that is connected to the FE310 on the HiFive1B, but it must be the 1B bootloader mirroring the AT cmd/resp with ESP32. Seeing how the bootloader is setting up and making that communication with the ESP32 would go a long way in clearing up what is going on on the FE310 prior to any program that starts execution at 0x20010000