Hello DPDK mentors,

We are facing an issue, KNI core stuck after 4-5 days of traffic. Can
anybody guide us on this?

We have set isolcpu, rcu_nocbs in grub for given cores.

Aug 22 01:38:29 kernel: INFO: rcu_sched self-detected stall on CPU { 2}
(t=60001 jiffies g=426362199 c=426362198 q=150932)

Aug 22 01:38:29 svc kernel: Task dump for CPU 2:

Aug 22 01:38:29 svc kernel: kni_veth3_3     R  running task        0
4915      2 0x00080088

Aug 22 01:38:29 svc kernel: Call Trace:



Aug 22 01:38:29 svc kernel: <IRQ>  [<ffffffffae8da318>]
sched_show_task+0xa8/0x110

Aug 22 01:38:29svc kernel: [<ffffffffae8de159>] dump_cpu_task+0x39/0x70

Aug 22 01:38:29 svc kernel: [<ffffffffae958480>]
rcu_dump_cpu_stacks+0x90/0xd0

Aug 22 01:38:29 fantastic kernel: [<ffffffffae95bb42>]
rcu_check_callbacks+0x442/0x730

Aug 22 01:38:29 svc kernel: [<ffffffffae910370>] ?
tick_sched_do_timer+0x50/0x50

Aug 22 01:38:29 vc kernel: [<ffffffffae8af186>]
update_process_times+0x46/0x80

Aug 22 01:38:29 svc kernel: [<ffffffffae9100e0>] tick_sched_handle+0x30/0x70

Aug 22 01:38:29 svc kernel: [<ffffffffae9103a9>] tick_sched_timer+0x39/0x80

Aug 22 01:38:29 svc kernel: [<ffffffffae8ca26e>]
__hrtimer_run_queues+0x10e/0x270

Aug 22 01:38:29 svc kernel: [<ffffffffae8ca7cf>]
hrtimer_interrupt+0xaf/0x1d0

Aug 22 01:38:29 svc kernel: [<ffffffffae85cdfb>]
local_apic_timer_interrupt+0x3b/0x60

Aug 22 01:38:29 vc kernel: [<ffffffffaef98a83>]
smp_apic_timer_interrupt+0x43/0x60

Aug 22 01:38:29 vc kernel: [<ffffffffaef94fba>]
apic_timer_interrupt+0x16a/0x170

Aug 22 01:38:29 -svc kernel: [<ffffffffae91776d>] ?
native_queued_spin_lock_slowpath+0x1d/0x200

Aug 22 01:38:29 -svc kernel: [<ffffffffaef7afe5>]
queued_spin_lock_slowpath+0xb/0xf

Aug 22 01:38:29 svc kernel: [<ffffffffaef89733>] _raw_spin_lock_bh+0x33/0x40

Aug 22 01:38:29 svc kernel: [<ffffffffc0901ba6>]
__nfqnl_enqueue_packet+0x56/0x160 [nfnetlink_queue]

Aug 22 01:38:29 svc kernel: [<ffffffffc09028af>]
nfqnl_enqueue_packet+0x24f/0x310 [nfnetlink_queue]

Aug 22 01:38:29 -svc kernel: [<ffffffffaee95cb6>] nf_queue+0x186/0x1e0

Aug 22 01:38:29 svc kernel: [<ffffffffaee94afc>] nf_hook_slow+0x6c/0x110

Aug 22 01:38:29 vc kernel: [<ffffffffaee9f559>] ip_rcv+0x339/0x420

Aug 22 01:38:29 svc kernel: [<ffffffffaee9ea80>] ?
inet_del_offload+0x40/0x40

Aug 22 01:38:29-svc kernel: [<ffffffffaee54829>]
__netif_receive_skb_core+0x729/0xa10

Aug 22 01:38:29 svc kernel: [<ffffffffaef9682a>] ?
reschedule_interrupt+0x16a/0x170

Aug 22 01:38:29 -svc kernel: [<ffffffffaee54b28>]
__netif_receive_skb+0x18/0x60

Aug 22 01:38:29 svc kernel: [<ffffffffaee55afe>] process_backlog+0xae/0x180

Aug 22 01:38:29 svc kernel: [<ffffffffaee551cf>] net_rx_action+0x26f/0x390

Aug 22 01:38:29 svc kernel: [<ffffffffae8a4b95>] __do_softirq+0xf5/0x280

Aug 22 01:38:29 -svc kernel: [<ffffffffaef974ec>] call_softirq+0x1c/0x30

Aug 22 01:38:29 svc kernel: <EOI>  [<ffffffffae82f715>] do_softirq+0x65/0xa0

Aug 22 01:38:29-svc kernel: [<ffffffffaee53263>] netif_rx_ni+0x33/0x70

Aug 22 01:38:29 svc kernel: [<ffffffffc07548da>]
kni_net_rx_normal+0x21a/0x330 [rte_kni]

Aug 22 01:38:29 svc kernel: [<ffffffffc0755f10>] ?
kni_thread_single+0xa0/0xa0 [rte_kni]

Aug 22 01:38:29 svc kernel: [<ffffffffc0755aa5>] kni_net_rx+0x15/0x20
[rte_kni]

Aug 22 01:38:29-svc kernel: [<ffffffffc0755f40>]
kni_thread_multiple+0x30/0x60 [rte_kni]

Aug 22 01:38:29 svc kernel: [<ffffffffae8c5e71>] kthread+0xd1/0xe0

Aug 22 01:38:29 -svc kernel: [<ffffffffae8c5da0>] ?
insert_kthread_work+0x40/0x40

Aug 22 01:38:29 svc kernel: [<ffffffffaef93ddd>]
ret_from_fork_nospec_begin+0x7/0x21

Aug 22 01:38:29 svc kernel: [<ffffffffae8c5da0>] ?
insert_kthread_work+0x40/0x40





Thanks,

Chetan