Kernel panic - not syncing: Fatal exception in interrupt

[imbabu (Babu)]

Hello All,

I’m facing kernel panic in Linux Kernel 5.10.41 Version. I attached the log below.

I install the below packages in Unmatched board.

apt-get install ufw iptables docker-riscv64 systemd-container

after im stating the docker services. by using below commands

$ mkdir /etc/ld.so.conf.d
$ echo /usr/local/lib > /etc/ld.so.conf.d/libseccomp-riscv64.conf
$ ldconfig

Docker

$ systemctl daemon-reload
$ systemctl enable containerd
$ systemctl start containerd
$ systemctl start docker
$ systemctl enable docker

kernel panic log

root@Exaleap-Unmatched-Linux-1:~# [ 173.306589] Bridge firewalling registered

[ 174.468119] Initializing XFRM netlink socket

[ 175.168418] Unable to handle kernel paging request at virtual address 0000005f81b2087a

[ 175.175579] Oops [#1]

[ 175.177820] Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat ip_tables br_netfilter overlay nls_cp437 fuse

[ 175.194762] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G W 5.10.41 #1

[ 175.202142] epc: ffffffdf81b2087a ra : ffffffe000b415d4 sp : ffffffe07ffe3a80

[ 175.209263] gp : ffffffe0018089b8 tp : ffffffe07ffcab00 t0 : 0000000000000000

[ 175.216472] t1 : 0000000000000000 t2 : 00000000160000e0 s0 : ffffffe07ffe3ad0

[ 175.223682] s1 : 0000000000000002 a0 : ffffffe085d74320 a1 : ffffffe083aec100

[ 175.230891] a2 : ffffffe07ffe3ad8 a3 : 00000000000249f0 a4 : ffffffdf81b2087a

[ 175.238099] a5 : ffffffe083de8fa0 a6 : 0000000000000000 a7 : 0000000000000000

[ 175.245309] s2 : 0000000000000001 s3 : ffffffe083de8f80 s4 : ffffffe083aec100

[ 175.252518] s5 : 0000000000000001 s6 : 0000000000000003 s7 : ffffffe07ffe3ad8

[ 175.259728] s8 : ffffffe083dcc830 s9 : ffffffe083dcc800 s10: ffffffe000bb5340

[ 175.266936] s11: ffffffe083dcc8a8 t3 : 00000000160000e0 t4 : 0000000000020000

[ 175.274145] t5 : 0000000000000000 t6 : 0000000000000000

[ 175.279445] status: 0000000200000120 badaddr: 0000005f81b2087a cause: 000000000000000c

[ 175.287397] —[ end trace a6c6602989b9c91b ]—

[ 175.291972] Kernel panic - not syncing: Fatal exception in interrupt

[ 175.298298] SMP: stopping secondary CPUs

[ 175.302222] —[ end Kernel panic - not syncing: Fatal exception in interrupt ]–

same kernel panic is came once we are doing voice recognization. attached the screen shot

I’m unable to find the root cause for the kernel panic.
even i tried in the Linux kernel 5.10.19 and 5.10.99 versions also in that kernel also im facing same issue.

i observe that same issue posted in git lab @giantswarm (Giant Swarm · GitHub) Kernel panics in 5.10.37 version

[kernel] Unable to handle kernel paging request at virtual address

i obeserved that after enabling below configurations, im getting kernel panic, may be if im wrong please correct me.

+CONFIG_SLUB_MEMCG_SYSFS_ON=y
+CONFIG_NF_CONNTRACK=m
+CONFIG_NF_CONNTRACK_PROCFS=y
+CONFIG_NF_CONNTRACK_EVENTS=y
+CONFIG_NF_CT_PROTO_DCCP=y
+CONFIG_NF_CT_PROTO_SCTP=y
+CONFIG_NF_CT_PROTO_UDPLITE=y
+CONFIG_NF_NAT=m
+CONFIG_NF_NAT_MASQUERADE=y
+CONFIG_NETFILTER_XTABLES=m
+CONFIG_NETFILTER_XT_NAT=m
+CONFIG_NF_DEFRAG_IPV4=m
+CONFIG_IP_NF_IPTABLES=m
+CONFIG_IP_NF_NAT=m
+CONFIG_IP_NF_TARGET_MASQUERADE=m
+CONFIG_NF_DEFRAG_IPV6=m
+CONFIG_NF_CONNTRACK_BRIDGE=m

In This kernel 5.10.41 dont have kexec & kernel crash dump support for riscv. they given supporting from 5.13 kernel onwards.
what is the main cause for **> end Kernel panic - not syncing: Fatal exception in interrupt**
is related to interrupt drivers issue, is there any bug in the interrupts or any issue with network drivers.

how can i debug the oops and kernel panic issue. Please help me to fix this issue.

Thanks
Babu P

[imbabu (Babu)]

Above the meta-sifive 2021.02 branches, I added the above network configurations.

I enabled stack_tracer & dependencies configurations in my image, I hope it gives call trace/backtrace stack memory information when the kernel crashes but I did get that information when the kernel panic occurs

Externally I triggered the stock memory address
By using the following commands.

root@Unmatched-Linux-1:~# sysctl kernel.stack_tracer_enabled

kernel.stack_tracer_enabled = 0

root@Unmatched-Linux-1:~# echo 1 > /proc/sys/kernel/stack_tracer_enabled

root@Unmatched-Linux-1:~# sysctl kernel.stack_tracer_enabled

kernel.stack_tracer_enabled = 1

root@Unmatched-Linux-1:~# echo 1 > /proc/sys/kernel/stack_tracer_enabled

root@Unmatched-Linux-1:~# WARNING: CPU: 0 PID: 13 at arch/riscv/kernel/patch.c:59

patch_insn_write+0x7e/0xce
[ 222.159556] Modules linked in: cfg80211 overlay nls_cp437 fuse
[ 222.165380] CPU: 0 PID: 13 Comm: migration/0 Tainted: G W 5.10.41 #1
[ 222.173022] epc: ffffffe0002046c6 ra : ffffffe0002046c4 sp : ffffffe08008bbe0
[ 222.180143] gp : ffffffe001a3f458 tp : ffffffe080080000 t0 : ffffffe08008bb68
[ 222.187353] t1 : 0000000000000001 t2 : ffffffe08008bbe0 s0 : ffffffe08008bc10
[ 222.194562] s1 : 0000000000000ed0 a0 : 0000000000000000 a1 : ffffffffffffffff
[ 222.201771] a2 : 0000000000000001 a3 : 0000000000000000 a4 : 0000000000000001
[ 222.208980] a5 : ffffffe1f7d1a550 a6 : e34d8ab4a7a73200 a7 : 0000000000000000
[ 222.216190] s2 : ffffffe000241ec8 s3 : 0000000000000008 s4 : ffffffe08008bc38
[ 222.223399] s5 : 0000000000000a00 s6 : ffffffe001375b00 s7 : ffffffe07fe03020
[ 222.230607] s8 : 0000000000000001 s9 : 0000000000000002 s10: 0000000000000000
[ 222.237817] s11: 0000000000000002 t3 : c48cc3ae089ba000 t4 : 00000000003d6643
[ 222.245026] t5 : ffffffe001a3f3a0 t6 : ffffffe001001a94
[ 222.250325] status: 0000000200000100 badaddr: 0000000000000004 cause: 0000000000000003
[ 222.258228] —[ end trace 5be7cf159ce057fe ]—
[ 222.262852] ------------[ cut here ]------------
[ 222.267441] WARNING: CPU: 0 PID: 13 at arch/riscv/kernel/patch.c:59 patch_insn_write+0x7e/0xce
[ 222.276033] Modules linked in: cfg80211 overlay nls_cp437 fuse
[ 222.281856] CPU: 0 PID: 13 Comm: migration/0 Tainted: G W 5.10.41 #1
[ 222.289499] epc: ffffffe0002046c6 ra : ffffffe0002046c4 sp : ffffffe08008bbe0
[ 222.296621] gp : ffffffe001a3f458 tp : ffffffe080080000 t0 : ffffffe08008bb68
[ 222.303830] t1 : 0000000000000001 t2 : ffffffe08008bbe0 s0 : ffffffe08008bc10
[ 222.311039] s1 : 0000000000000eea a0 : 0000000000000000 a1 : ffffffffffffffff
[ 222.318248] a2 : 0000000000000001 a3 : 0000000000000000 a4 : 0000000000000001
[ 222.325458] a5 : ffffffe1f7d1a550 a6 : e34d8ab4a7a73200 a7 : 0000000000000000
[ 222.332667] s2 : ffffffe000241ee2 s3 : 0000000000000008 s4 : ffffffe08008bc38
[ 222.339876] s5 : 0000000000000a00 s6 : ffffffe001375b00 s7 : ffffffe07fe03020
[ 222.347085] s8 : 0000000000000001 s9 : 0000000000000002 s10: 0000000000000000
[ 222.354294] s11: 0000000000000002 t3 : c48cc3ae089ba000 t4 : 00000000003d6643
[ 222.361503] t5 : ffffffe001a3f3a0 t6 : ffffffe001001a94
[ 222.366802] status: 0000000200000100 badaddr: 0000000000000004 cause: 0000000000000003
[ 222.374705] —[ end trace 5be7cf159ce057ff ]—
[ 222.379329] ------------[ cut here ]------------
[ 222.383918] WARNING: CPU: 0 PID: 13 at arch/riscv/kernel/patch.c:59 patch_insn_write+0x7e/0xce
[ 222.392510] Modules linked in: cfg80211 overlay nls_cp437 fuse
[ 222.398334] CPU: 0 PID: 13 Comm: migration/0 Tainted: G W 5.10.41 #1
[ 222.405976] epc: ffffffe0002046c6 ra : ffffffe0002046c4 sp : ffffffe08008bbe0
[ 222.413097] gp : ffffffe001a3f458 tp : ffffffe080080000 t0 : ffffffe08008bb68
[ 222.420307] t1 : 0000000000000001 t2 : ffffffe08008bbe0 s0 : ffffffe08008bc10
[ 222.427516] s1 : 0000000000000f08 a0 : 0000000000000000 a1 : ffffffffffffffff
[ 222.434725] a2 : 0000000000000001 a3 : 0000000000000000 a4 : 0000000000000001
[ 222.441934] a5 : ffffffe1f7d1a550 a6 : e34d8ab4a7a73200 a7 : 0000000000000000
[ 222.449144] s2 : ffffffe000241f00 s3 : 0000000000000008 s4 : ffffffe08008bc38
[ 222.456353] s5 : 0000000000000a00 s6 : ffffffe001375b00 s7 : ffffffe07fe03020
[ 222.463562] s8 : 0000000000000001 s9 : 0000000000000002 s10: 0000000000000000
[ 222.470771] s11: 0000000000000002 t3 : c48cc3ae089ba000 t4 : 00000000003d6643
[ 222.477980] t5 : ffffffe001a3f3a0 t6 : ffffffe001001a94
[ 222.483279] status: 0000000200000100 badaddr: 0000000000000004 cause: 0000000000000003
[ 222.491182] —[ end trace 5be7cf159ce05800 ]—
[ 222.495806] ------------[ cut here ]------------
[ 222.500395] WARNING: CPU: 0 PID: 13 at arch/riscv/kernel/patch.c:59 patch_insn_write+0x7e/0xce
[ 222.508987] Modules linked in: cfg80211 overlay nls_cp437 fuse
[ 222.514811] CPU: 0 PID: 13 Comm: migration/0 Tainted: G W 5.10.41 #1
[ 222.522453] epc: ffffffe0002046c6 ra : ffffffe0002046c4 sp : ffffffe08008bbe0
[ 222.529575] gp : ffffffe001a3f458 tp : ffffffe080080000 t0 : ffffffe08008bb68
[ 222.536784] t1 : 0000000000000001 t2 : ffffffe08008bbe0 s0 : ffffffe08008bc10
[ 222.543993] s1 : 0000000000000f2c a0 : 0000000000000000 a1 : ffffffffffffffff
[ 222.551202] a2 : 0000000000000001 a3 : 0000000000000000 a4 : 0000000000000001
[ 222.558411] a5 : ffffffe1f7d1a550 a6 : e34d8ab4a7a73200 a7 : 0000000000000000
[ 222.565621] s2 : ffffffe000241f24 s3 : 0000000000000008 s4 : ffffffe08008bc38
[ 222.572829] s5 : 0000000000000a00 s6 : ffffffe001375b00 s7 : ffffffe07fe03020
[ 222.580039] s8 : 0000000000000001 s9 : 0000000000000002 s10: 0000000000000000
[ 222.587248] s11: 0000000000000002 t3 : c48cc3ae089ba000 t4 : 00000000003d6643
[ 222.594457] t5 : ffffffe001a3f3a0 t6 : ffffffe001001a94
[ 222.599756] status: 0000000200000100 badaddr: 0000000000000004 cause: 0000000000000003
[ 222.607659] —[ end trace 5be7cf159ce05801 ]—
[ 222.612284] ------------[ cut here ]-----------

It only gives stack sp addresses. I need a full address in memory with functions(calltrace/backtrace). which function running in process during kernel panic.

Currently, Riscv has support or not.

Please clarify my doubts I have seen that many people are already experiencing kernel panic issues. How do you solve this problem from your side?

Please help me to resolve this kernel panic issue.
Looking forward to your valuable answer.

[Jim (Jim Sughrue)]

Hi Babu,

We’ve discussed this a little bit internally. The general consensus is that it might be good to file a ticket with Ubuntu if you haven’t done this already since it seems to be related to Ubuntu release and not a SiFive software release.

The other thought would be to debug the issue within the kernel using the information from the panic.

I’m not aware of any network or network interrupt issues that may be the source of the issue.

[imbabu (Babu)]

Hi Jim Sughrue,

Thanks for your valuable reply.
Still i didn’t raise ticket in ubuntu. I will do right now.

Thanks
Babu

[imbabu (Babu)]

Hello Jim Sughrue,

We are not using Ubuntu instead, we are using yocto build system to create images using meta-sifive 2021.02 braches ( Linux kernel 5.10.41). If

If we raise the ticket in the Ubuntu forum. I do not think it’s useful for us.

NOTE: With the same configuration, The fu540 image works well without kernel panic issues.
Only in fu740 facing kernel panic problem.

as my latest distro image after enabling below configuration getting kernel panic issue in fu740 distro image.

CONFIG:

+CONFIG_BLK_RQ_ALLOC_TIME=y
+CONFIG_BLK_WBT=y
+CONFIG_BLK_CGROUP_IOLATENCY=y
+CONFIG_BLK_CGROUP_IOCOST=y
+CONFIG_BLK_WBT_MQ=y
+CONFIG_NETFILTER_NETLINK=m
+CONFIG_NETFILTER_NETLINK_ACCT=m

Thanks
Babu P

[Jim (Jim Sughrue)]

Hi Babu,

It looked like you were using Ubuntu on Unmatched:

@imbabu

I install the below packages in Unmatched board.

apt-get install ufw iptables docker-riscv64 systemd-container

If the issue doesn’t occur in Ubuntu then please disregard my comment about filing an Ubuntu ticket.

[imbabu (Babu)]

Hello Jim Sughrue,

In my case, I was using an apt-get server. You may be confused about this.
we use Yocto to create custom distro images.
we are creating the base (with apt & minimal packages) image. If any additional packages are required. we will install via apt-get.
We maintain our own apt server.

Thanks
Babu P.