As @Almindor notes above, It seems to be something wrong with clock setting, indeed the clock path diagram in Figure 3 of Chapter 6 of the FE310-G002 Manual is drawn with a few significant mistakes, thus giving very confusing observations when programming and/or changing configurations of the clock path.
See Understanding the PRCI clock path or the description and examples in the GitHub repo at Demonstrating PRCI.
Otherwise @edsut, there’s not much need to worry about bricking the FE310 SoC when removing the doubletap bootloader or replacing any of the code in FLASH at 0x2000 0000 upwards. See discussion at