Hello DPDK mentors,

We are facing an issue, KNI core stuck after 4-5 days of traffic. Can anybody guide us on this?

We have set isolcpu, rcu_nocbs in grub for given cores.

Aug 22 01:38:29 kernel: INFO: rcu_sched self-detected stall on CPU { 2}  (t=60001 jiffies g=426362199 c=426362198 q=150932)

Aug 22 01:38:29 svc kernel: Task dump for CPU 2:

Aug 22 01:38:29 svc kernel: kni_veth3_3     R  running task        0  4915      2 0x00080088

Aug 22 01:38:29 svc kernel: Call Trace:

 

Aug 22 01:38:29 svc kernel: <IRQ>  [<ffffffffae8da318>] sched_show_task+0xa8/0x110

Aug 22 01:38:29svc kernel: [<ffffffffae8de159>] dump_cpu_task+0x39/0x70

Aug 22 01:38:29 svc kernel: [<ffffffffae958480>] rcu_dump_cpu_stacks+0x90/0xd0

Aug 22 01:38:29 fantastic kernel: [<ffffffffae95bb42>] rcu_check_callbacks+0x442/0x730

Aug 22 01:38:29 svc kernel: [<ffffffffae910370>] ? tick_sched_do_timer+0x50/0x50

Aug 22 01:38:29 vc kernel: [<ffffffffae8af186>] update_process_times+0x46/0x80

Aug 22 01:38:29 svc kernel: [<ffffffffae9100e0>] tick_sched_handle+0x30/0x70

Aug 22 01:38:29 svc kernel: [<ffffffffae9103a9>] tick_sched_timer+0x39/0x80

Aug 22 01:38:29 svc kernel: [<ffffffffae8ca26e>] __hrtimer_run_queues+0x10e/0x270

Aug 22 01:38:29 svc kernel: [<ffffffffae8ca7cf>] hrtimer_interrupt+0xaf/0x1d0

Aug 22 01:38:29 svc kernel: [<ffffffffae85cdfb>] local_apic_timer_interrupt+0x3b/0x60

Aug 22 01:38:29 vc kernel: [<ffffffffaef98a83>] smp_apic_timer_interrupt+0x43/0x60

Aug 22 01:38:29 vc kernel: [<ffffffffaef94fba>] apic_timer_interrupt+0x16a/0x170

Aug 22 01:38:29 -svc kernel: [<ffffffffae91776d>] ? native_queued_spin_lock_slowpath+0x1d/0x200

Aug 22 01:38:29 -svc kernel: [<ffffffffaef7afe5>] queued_spin_lock_slowpath+0xb/0xf

Aug 22 01:38:29 svc kernel: [<ffffffffaef89733>] _raw_spin_lock_bh+0x33/0x40

Aug 22 01:38:29 svc kernel: [<ffffffffc0901ba6>] __nfqnl_enqueue_packet+0x56/0x160 [nfnetlink_queue]

Aug 22 01:38:29 svc kernel: [<ffffffffc09028af>] nfqnl_enqueue_packet+0x24f/0x310 [nfnetlink_queue]

Aug 22 01:38:29 -svc kernel: [<ffffffffaee95cb6>] nf_queue+0x186/0x1e0

Aug 22 01:38:29 svc kernel: [<ffffffffaee94afc>] nf_hook_slow+0x6c/0x110

Aug 22 01:38:29 vc kernel: [<ffffffffaee9f559>] ip_rcv+0x339/0x420

Aug 22 01:38:29 svc kernel: [<ffffffffaee9ea80>] ? inet_del_offload+0x40/0x40

Aug 22 01:38:29-svc kernel: [<ffffffffaee54829>] __netif_receive_skb_core+0x729/0xa10

Aug 22 01:38:29 svc kernel: [<ffffffffaef9682a>] ? reschedule_interrupt+0x16a/0x170

Aug 22 01:38:29 -svc kernel: [<ffffffffaee54b28>] __netif_receive_skb+0x18/0x60

Aug 22 01:38:29 svc kernel: [<ffffffffaee55afe>] process_backlog+0xae/0x180

Aug 22 01:38:29 svc kernel: [<ffffffffaee551cf>] net_rx_action+0x26f/0x390

Aug 22 01:38:29 svc kernel: [<ffffffffae8a4b95>] __do_softirq+0xf5/0x280

Aug 22 01:38:29 -svc kernel: [<ffffffffaef974ec>] call_softirq+0x1c/0x30

Aug 22 01:38:29 svc kernel: <EOI>  [<ffffffffae82f715>] do_softirq+0x65/0xa0

Aug 22 01:38:29-svc kernel: [<ffffffffaee53263>] netif_rx_ni+0x33/0x70

Aug 22 01:38:29 svc kernel: [<ffffffffc07548da>] kni_net_rx_normal+0x21a/0x330 [rte_kni]

Aug 22 01:38:29 svc kernel: [<ffffffffc0755f10>] ? kni_thread_single+0xa0/0xa0 [rte_kni]

Aug 22 01:38:29 svc kernel: [<ffffffffc0755aa5>] kni_net_rx+0x15/0x20 [rte_kni]

Aug 22 01:38:29-svc kernel: [<ffffffffc0755f40>] kni_thread_multiple+0x30/0x60 [rte_kni]

Aug 22 01:38:29 svc kernel: [<ffffffffae8c5e71>] kthread+0xd1/0xe0

Aug 22 01:38:29 -svc kernel: [<ffffffffae8c5da0>] ? insert_kthread_work+0x40/0x40

Aug 22 01:38:29 svc kernel: [<ffffffffaef93ddd>] ret_from_fork_nospec_begin+0x7/0x21

Aug 22 01:38:29 svc kernel: [<ffffffffae8c5da0>] ? insert_kthread_work+0x40/0x40

 

 

Thanks,

Chetan

 

Disclaimer: © 2023 Truminds Software Systems. This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful.