Hi, I’m trying to get an ethernet driver working in my operating system (seL4) on the P550 and have a question about cache coherency with the DMA and the application cores. TRM seems to imply that there is coherency, but I see that (a) linux has enabled the RISCV_DMA_NON_COHERENT config for this device and (b) the uboot ethernet driver seems to access descriptors and buffers via the non-coherent address range in the memory map (ie adds an offset to the address that reaches into the 0xc0_0000_0000 region that seems to mirror the DDR area). I have tried just using DDR for buffers and descriptors in my driver (w/o any cache ops) and the DMA is mostly non-functional.
Related question, is the SRAM at 0x59000000 cacheable memory and should the ethernet DMA be able to access it reliably? (it seems to work for my driver most of the time if i use that area for buffers and descriptors - but am still seeing some issues) .
Out of curiosity, where do you see the RISCV_DMA_NON_COHERENT option in Linux? I’m looking at the defconfig for the P550 in SiFive’s fork of Linux but don’t see that option. I would have also assumed that DMA is cache coherent.
EDIT: nevermind, I see it once I build Linux as RISCV_DMA_NONCOHERENT.
The EIC7700X TRM doesn’t say anything about the Zicbom extension. It looks like RISCV_DMA_NONCOHERENT is selected indirectly by ERRATA_THEAD being in the defconfig. Not sure what that’s about.
NON_STANDARD_CACHE_OPS is also set. There is platform specific code for this part in drivers/soc/sifive/sifive_cache.c that has a flush function (using a register in the cache controller, @0x201_0200 I think). But I don’t see where the ethernet driver actually uses these.
AFAIK, the coherency of that peripheral depends on how it’s connected to the core complex. The EIC7700 TRM says the following:
If a Front Port access targets the Memory Port, a coherency manager is responsible for maintaining coherency with the L1, pL2, and L3 caches.
Thus, for a peripheral to be cache coherent, it needs to use front port for DMA, and the coherence should be guaranteed by the SoC. For reference, this is something Starfive shows for JH7100 (Sifive U74MC):
Therefore, Starfive claims GMAC on JH7110 is cache coherent (through front port), unlike it’s previous Gen JH7100.
In EIC7700’s case, I see kernel Kconfig is patched with
select ARCH_DMA_DEFAULT_COHERENT if !SOC_SIFIVE_EIC7700
And there’s no dma-coherent attribute on the device-tree to override the default “noncoherent” (surprisingly there’s a dma-coherent attribute on the PCIe controller that’s commented out). Although ESWIN doesn’t say it clearly in TRM, I highly suspect EIC7700 is in the same scenario as JH7100, where GMAC is NOT connected to front port, and for DMA, you need either cache invalidation (flush) or access it from CPU side through sys port @c0_0000_0000 (uncached). Both methods are hooked into Linux DMA framework through RISCV_DMA_NONCOHERENT. I think technically you should be able to use SRAM, as it’s mapped uncached through peripheral port from CPU perspective, but I have not tried it and I can’t say for sure. ESWIN should update the TRM to include a similar diagram as Starfive does and clearly states the cache coherency for peripheral devices on a case-by-case basis.
Thanks! that makes sense. Where do you see that the sys port @c0_0000_0000 is supported in Linux DMA framework. I tried to search for that address and did not see it anywhere (shouldn’t it be in the DTS somewhere?) u-boot just hard codes a #define for a SYSPORT_OFFSET in the driver.
It turns out my TX issue seems to be a PHY setup/physical layer problem. if I insert a LAN probe between the P550 and my switch, the problem goes away and pings are 100% reliable. So I would say that using the SRAM is a viable option, at least for my case.
I slightly misunderstood RISCV_DMA_NONCOHERENT. The access to memory as uncached through sys port is done here:
And convert_pfn_from_mem_to_sys_port is:
The relevant offset is defined here:
It’s really a hack in the common code path, and should be cleaned up. It’s definitely not ready for upstreaming. I know Samuel has a cleaned-up version of this, and we are yet to see if upstream will accept it.
On related note, the convert_pfn_from_mem_to_sys_port is actually buggy, as it wrongly converts MMIO below DIE0 DDR base address to sys port. I have a patch for it here:
