Freeze after setting SATP register

I am writing EFI boot loader and I get freeze after setting SATP register and before sfence.vma instruction. Freeze occurs only on real hardware (HiFive Unmatched), it works fine on QEMU and TinyEMU emulators. I use 64 bit mode and sv39 paging with 3 level page tables (no large pages). Paging is disabled during construction of page table. I tried to reduce mapping to minimum with only identity mappings but it still not working.

Linux from SD card boots and works fine. EFI loader is located on USB 3 flash drive.

Any ideas what I am doing wrong? I can add additional debug output if needed.

UART log:

  node('reserved-memory')
    prop('#address-cells'): 2 (len 4)
    prop('#size-cells'): 2 (len 4)
    prop('ranges'):
       (len 0)
    node('mmode_resv0@80000000')
      prop('reg'): (0x80000000, 0x00040000) (len 16)

...

UART:
  kind: sifive
  regs: 0x10010000, 0x1000
  irq: 39
  clock: 0
kernel:
  text: 0xffffffc002010000, 0x17e000
  data: 0xffffffc00218e000, 0x59000
  entry: 0xffffffc0020935ae
Kernel stack at 0xffffffc0025dc000
System provided memory map:
  phys: 0x80000000, virt: 0x80000000, size: 0x40000, bootServicesData, attrs: 0x8
  phys: 0x80040000, virt: 0x80040000, size: 0x7eb6000, conventionalMemory, attrs: 0x8
  phys: 0x87ef6000, virt: 0x87ef6000, size: 0x14000, ACPIReclaimMemory, attrs: 0x8
  phys: 0x87f0a000, virt: 0x87f0a000, size: 0x741cb000, conventionalMemory, attrs: 0x8
  phys: 0xfc0d5000, virt: 0xfc0d5000, size: 0x25e9000, loaderData, attrs: 0x8
  phys: 0xfe6be000, virt: 0xfe6be000, size: 0x59000, loaderCode, attrs: 0x8
  phys: 0xfe717000, virt: 0xfe717000, size: 0x7000, reservedMemoryType, attrs: 0x8
  phys: 0xfe71e000, virt: 0xfe71e000, size: 0x1000, bootServicesData, attrs: 0x8
  phys: 0xfe71f000, virt: 0xfe71f000, size: 0x1000, runtimeServicesData, attrs: 0x8000000000000008
  phys: 0xfe720000, virt: 0xfe720000, size: 0x2000, bootServicesData, attrs: 0x8
  phys: 0xfe722000, virt: 0xfe722000, size: 0x1000, reservedMemoryType, attrs: 0x8
  phys: 0xfe723000, virt: 0xfe723000, size: 0x3000, runtimeServicesData, attrs: 0x8000000000000008
  phys: 0xfe726000, virt: 0xfe726000, size: 0x1000, bootServicesData, attrs: 0x8
  phys: 0xfe727000, virt: 0xfe727000, size: 0x4000, runtimeServicesData, attrs: 0x8000000000000008
  phys: 0xfe72b000, virt: 0xfe72b000, size: 0x1000, reservedMemoryType, attrs: 0x8
  phys: 0xfe72c000, virt: 0xfe72c000, size: 0x1000, bootServicesData, attrs: 0x8
  phys: 0xfe72d000, virt: 0xfe72d000, size: 0x1000, reservedMemoryType, attrs: 0x8
  phys: 0xfe72e000, virt: 0xfe72e000, size: 0x2000, bootServicesData, attrs: 0x8
  phys: 0xfe730000, virt: 0xfe730000, size: 0x2000, reservedMemoryType, attrs: 0x8
  phys: 0xfe732000, virt: 0xfe732000, size: 0x1000, bootServicesData, attrs: 0x8
  phys: 0xfe733000, virt: 0xfe733000, size: 0x1000, reservedMemoryType, attrs: 0x8
  phys: 0xfe734000, virt: 0xfe734000, size: 0x1000, bootServicesData, attrs: 0x8
  phys: 0xfe735000, virt: 0xfe735000, size: 0x1000, reservedMemoryType, attrs: 0x8
  phys: 0xfe736000, virt: 0xfe736000, size: 0x1000, bootServicesData, attrs: 0x8
  phys: 0xfe737000, virt: 0xfe737000, size: 0x1000, reservedMemoryType, attrs: 0x8
  phys: 0xfe738000, virt: 0xfe738000, size: 0x2000, bootServicesData, attrs: 0x8
  phys: 0xfe73a000, virt: 0xfe73a000, size: 0x1000, reservedMemoryType, attrs: 0x8
  phys: 0xfe73b000, virt: 0xfe73b000, size: 0x2000, bootServicesData, attrs: 0x8
  phys: 0xfe73d000, virt: 0xfe73d000, size: 0x1826000, loaderData, attrs: 0x8
  phys: 0xfff63000, virt: 0xfff63000, size: 0x1000, runtimeServicesCode, attrs: 0x8000000000000008
  phys: 0xfff64000, virt: 0xfff64000, size: 0x9c000, loaderData, attrs: 0x8
  phys: 0x100000000, virt: 0x100000000, size: 0x380000000, bootServicesData, attrs: 0x8
physMemRange: 0x80040000, 0x7ffc0000
Boot loader:
MapRange(0xfc0d5000 - 0xfe6bdfff, 0xfc0d5000 - 0xfe6bdfff, 0x25e9000)
MapRange(0xfe6be000 - 0xfe716fff, 0xfe6be000 - 0xfe716fff, 0x59000)
MapRange(0xfe73d000 - 0xfff62fff, 0xfe73d000 - 0xfff62fff, 0x1826000)
MapRange(0xfff64000 - 0xffffffff, 0xfff64000 - 0xffffffff, 0x9c000)
Boot loader stack
  SP: 0xff73b1e8
  gStackBase: 0xff73b418
EFI runtime services:
MapRange(0xfe71f000 - 0xfe71ffff, 0xfe71f000 - 0xfe71ffff, 0x1000)
MapRange(0xfe723000 - 0xfe725fff, 0xfe723000 - 0xfe725fff, 0x3000)
MapRange(0xfe727000 - 0xfe72afff, 0xfe727000 - 0xfe72afff, 0x4000)
MapRange(0xfff63000 - 0xfff63fff, 0xfff63000 - 0xfff63fff, 0x1000)
Devices:
MapRange(0x10010000 - 0x10010fff, 0x10010000 - 0x10010fff, 0x1000)
PageTable:
  0x10010000 - 0x10010fff: 0x10010000 - 0x10010fff, 0x1000, {valid, read, write}
  0xfc0d5000 - 0xfe716fff: 0xfc0d5000 - 0xfe716fff, 0x2642000, {valid, read, write, exec}
  0xfe71f000 - 0xfe71ffff: 0xfe71f000 - 0xfe71ffff, 0x1000, {valid, read, write, exec}
  0xfe723000 - 0xfe725fff: 0xfe723000 - 0xfe725fff, 0x3000, {valid, read, write, exec}
  0xfe727000 - 0xfe72afff: 0xfe727000 - 0xfe72afff, 0x4000, {valid, read, write, exec}
  0xfe73d000 - 0xffffffff: 0xfe73d000 - 0xffffffff, 0x18c3000, {valid, read, write, exec}
SATP: 0x80000000000fc0d4
Calling ExitBootServices. So long, EFI!
Switched to legacy serial output
&arch_start_kernel: 0x00000000fe6be66e
&gKernelArgs: 0x00000000fe70e048
sstatus: 0x8000000200006000 (ie: {}, pie: {}, spp: u, fs: dirty, xs: off, sum: 0, mxr: 0, uxl: 2, sd: 1)
sie: {}
sip: {}
sscratch: 0x0
[PRE] SetSatp()

Mappings are displayed in virtual address, physical address, size, flags order. “PageTable:” is a dump of page table that will be loaded to SATP register.

2 Likes

I solved problem. It was caused by missing hardware support of setting accessed and dirty PTE flags. If flags are not set access to page always trigger page fault. Kernel is supposed to set flags in page fault handler. Problem in boot loader was solved by setting accessed and dirty PTE flags during initial page table construction.

QEMU and TinyEMU set flags automatically.

Freedom U740-C000 Manual says:

It is important to note the U74 does not automatically set the accessed ( A ) and dirty ( D ) bits in a
Sv39 Page Table Entry (PTE). Instead, the U74 MMU will raise a page fault exception for a read
to a page with PTE.A=0 or a write to a page with PTE.D=0 .

3 Likes