* DPDK KNI problem [not found] <PN3P287MB0113FC136FCAF602D48280F49F1FA@PN3P287MB0113.INDP287.PROD.OUTLOOK.COM> @ 2023-08-22 6:56 ` chetan bhasin 2023-08-22 8:04 ` chetan bhasin 0 siblings, 1 reply; 2+ messages in thread From: chetan bhasin @ 2023-08-22 6:56 UTC (permalink / raw) To: dev [-- Attachment #1: Type: text/plain, Size: 4236 bytes --] Hello DPDK mentors, We are facing an issue, KNI core stuck after 4-5 days of traffic. Can anybody guide us on this? We have set isolcpu, rcu_nocbs in grub for given cores. Aug 22 01:38:29 kernel: INFO: rcu_sched self-detected stall on CPU { 2} (t=60001 jiffies g=426362199 c=426362198 q=150932) Aug 22 01:38:29 svc kernel: Task dump for CPU 2: Aug 22 01:38:29 svc kernel: kni_veth3_3 R running task 0 4915 2 0x00080088 Aug 22 01:38:29 svc kernel: Call Trace: Aug 22 01:38:29 svc kernel: <IRQ> [<ffffffffae8da318>] sched_show_task+0xa8/0x110 Aug 22 01:38:29svc kernel: [<ffffffffae8de159>] dump_cpu_task+0x39/0x70 Aug 22 01:38:29 svc kernel: [<ffffffffae958480>] rcu_dump_cpu_stacks+0x90/0xd0 Aug 22 01:38:29 fantastic kernel: [<ffffffffae95bb42>] rcu_check_callbacks+0x442/0x730 Aug 22 01:38:29 svc kernel: [<ffffffffae910370>] ? tick_sched_do_timer+0x50/0x50 Aug 22 01:38:29 vc kernel: [<ffffffffae8af186>] update_process_times+0x46/0x80 Aug 22 01:38:29 svc kernel: [<ffffffffae9100e0>] tick_sched_handle+0x30/0x70 Aug 22 01:38:29 svc kernel: [<ffffffffae9103a9>] tick_sched_timer+0x39/0x80 Aug 22 01:38:29 svc kernel: [<ffffffffae8ca26e>] __hrtimer_run_queues+0x10e/0x270 Aug 22 01:38:29 svc kernel: [<ffffffffae8ca7cf>] hrtimer_interrupt+0xaf/0x1d0 Aug 22 01:38:29 svc kernel: [<ffffffffae85cdfb>] local_apic_timer_interrupt+0x3b/0x60 Aug 22 01:38:29 vc kernel: [<ffffffffaef98a83>] smp_apic_timer_interrupt+0x43/0x60 Aug 22 01:38:29 vc kernel: [<ffffffffaef94fba>] apic_timer_interrupt+0x16a/0x170 Aug 22 01:38:29 -svc kernel: [<ffffffffae91776d>] ? native_queued_spin_lock_slowpath+0x1d/0x200 Aug 22 01:38:29 -svc kernel: [<ffffffffaef7afe5>] queued_spin_lock_slowpath+0xb/0xf Aug 22 01:38:29 svc kernel: [<ffffffffaef89733>] _raw_spin_lock_bh+0x33/0x40 Aug 22 01:38:29 svc kernel: [<ffffffffc0901ba6>] __nfqnl_enqueue_packet+0x56/0x160 [nfnetlink_queue] Aug 22 01:38:29 svc kernel: [<ffffffffc09028af>] nfqnl_enqueue_packet+0x24f/0x310 [nfnetlink_queue] Aug 22 01:38:29 -svc kernel: [<ffffffffaee95cb6>] nf_queue+0x186/0x1e0 Aug 22 01:38:29 svc kernel: [<ffffffffaee94afc>] nf_hook_slow+0x6c/0x110 Aug 22 01:38:29 vc kernel: [<ffffffffaee9f559>] ip_rcv+0x339/0x420 Aug 22 01:38:29 svc kernel: [<ffffffffaee9ea80>] ? inet_del_offload+0x40/0x40 Aug 22 01:38:29-svc kernel: [<ffffffffaee54829>] __netif_receive_skb_core+0x729/0xa10 Aug 22 01:38:29 svc kernel: [<ffffffffaef9682a>] ? reschedule_interrupt+0x16a/0x170 Aug 22 01:38:29 -svc kernel: [<ffffffffaee54b28>] __netif_receive_skb+0x18/0x60 Aug 22 01:38:29 svc kernel: [<ffffffffaee55afe>] process_backlog+0xae/0x180 Aug 22 01:38:29 svc kernel: [<ffffffffaee551cf>] net_rx_action+0x26f/0x390 Aug 22 01:38:29 svc kernel: [<ffffffffae8a4b95>] __do_softirq+0xf5/0x280 Aug 22 01:38:29 -svc kernel: [<ffffffffaef974ec>] call_softirq+0x1c/0x30 Aug 22 01:38:29 svc kernel: <EOI> [<ffffffffae82f715>] do_softirq+0x65/0xa0 Aug 22 01:38:29-svc kernel: [<ffffffffaee53263>] netif_rx_ni+0x33/0x70 Aug 22 01:38:29 svc kernel: [<ffffffffc07548da>] kni_net_rx_normal+0x21a/0x330 [rte_kni] Aug 22 01:38:29 svc kernel: [<ffffffffc0755f10>] ? kni_thread_single+0xa0/0xa0 [rte_kni] Aug 22 01:38:29 svc kernel: [<ffffffffc0755aa5>] kni_net_rx+0x15/0x20 [rte_kni] Aug 22 01:38:29-svc kernel: [<ffffffffc0755f40>] kni_thread_multiple+0x30/0x60 [rte_kni] Aug 22 01:38:29 svc kernel: [<ffffffffae8c5e71>] kthread+0xd1/0xe0 Aug 22 01:38:29 -svc kernel: [<ffffffffae8c5da0>] ? insert_kthread_work+0x40/0x40 Aug 22 01:38:29 svc kernel: [<ffffffffaef93ddd>] ret_from_fork_nospec_begin+0x7/0x21 Aug 22 01:38:29 svc kernel: [<ffffffffae8c5da0>] ? insert_kthread_work+0x40/0x40 Thanks, Chetan *Disclaimer*: © 2023 Truminds Software Systems. This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. [-- Attachment #2: Type: text/html, Size: 6822 bytes --] ^ permalink raw reply [flat|nested] 2+ messages in thread
* DPDK KNI problem 2023-08-22 6:56 ` DPDK KNI problem chetan bhasin @ 2023-08-22 8:04 ` chetan bhasin 0 siblings, 0 replies; 2+ messages in thread From: chetan bhasin @ 2023-08-22 8:04 UTC (permalink / raw) To: dev [-- Attachment #1: Type: text/plain, Size: 3692 bytes --] Hello DPDK mentors, We are facing an issue, KNI core stuck after 4-5 days of traffic. Can anybody guide us on this? We have set isolcpu, rcu_nocbs in grub for given cores. Aug 22 01:38:29 kernel: INFO: rcu_sched self-detected stall on CPU { 2} (t=60001 jiffies g=426362199 c=426362198 q=150932) Aug 22 01:38:29 svc kernel: Task dump for CPU 2: Aug 22 01:38:29 svc kernel: kni_veth3_3 R running task 0 4915 2 0x00080088 Aug 22 01:38:29 svc kernel: Call Trace: Aug 22 01:38:29 svc kernel: <IRQ> [<ffffffffae8da318>] sched_show_task+0xa8/0x110 Aug 22 01:38:29svc kernel: [<ffffffffae8de159>] dump_cpu_task+0x39/0x70 Aug 22 01:38:29 svc kernel: [<ffffffffae958480>] rcu_dump_cpu_stacks+0x90/0xd0 Aug 22 01:38:29 fantastic kernel: [<ffffffffae95bb42>] rcu_check_callbacks+0x442/0x730 Aug 22 01:38:29 svc kernel: [<ffffffffae910370>] ? tick_sched_do_timer+0x50/0x50 Aug 22 01:38:29 vc kernel: [<ffffffffae8af186>] update_process_times+0x46/0x80 Aug 22 01:38:29 svc kernel: [<ffffffffae9100e0>] tick_sched_handle+0x30/0x70 Aug 22 01:38:29 svc kernel: [<ffffffffae9103a9>] tick_sched_timer+0x39/0x80 Aug 22 01:38:29 svc kernel: [<ffffffffae8ca26e>] __hrtimer_run_queues+0x10e/0x270 Aug 22 01:38:29 svc kernel: [<ffffffffae8ca7cf>] hrtimer_interrupt+0xaf/0x1d0 Aug 22 01:38:29 svc kernel: [<ffffffffae85cdfb>] local_apic_timer_interrupt+0x3b/0x60 Aug 22 01:38:29 vc kernel: [<ffffffffaef98a83>] smp_apic_timer_interrupt+0x43/0x60 Aug 22 01:38:29 vc kernel: [<ffffffffaef94fba>] apic_timer_interrupt+0x16a/0x170 Aug 22 01:38:29 -svc kernel: [<ffffffffae91776d>] ? native_queued_spin_lock_slowpath+0x1d/0x200 Aug 22 01:38:29 -svc kernel: [<ffffffffaef7afe5>] queued_spin_lock_slowpath+0xb/0xf Aug 22 01:38:29 svc kernel: [<ffffffffaef89733>] _raw_spin_lock_bh+0x33/0x40 Aug 22 01:38:29 svc kernel: [<ffffffffc0901ba6>] __nfqnl_enqueue_packet+0x56/0x160 [nfnetlink_queue] Aug 22 01:38:29 svc kernel: [<ffffffffc09028af>] nfqnl_enqueue_packet+0x24f/0x310 [nfnetlink_queue] Aug 22 01:38:29 -svc kernel: [<ffffffffaee95cb6>] nf_queue+0x186/0x1e0 Aug 22 01:38:29 svc kernel: [<ffffffffaee94afc>] nf_hook_slow+0x6c/0x110 Aug 22 01:38:29 vc kernel: [<ffffffffaee9f559>] ip_rcv+0x339/0x420 Aug 22 01:38:29 svc kernel: [<ffffffffaee9ea80>] ? inet_del_offload+0x40/0x40 Aug 22 01:38:29-svc kernel: [<ffffffffaee54829>] __netif_receive_skb_core+0x729/0xa10 Aug 22 01:38:29 svc kernel: [<ffffffffaef9682a>] ? reschedule_interrupt+0x16a/0x170 Aug 22 01:38:29 -svc kernel: [<ffffffffaee54b28>] __netif_receive_skb+0x18/0x60 Aug 22 01:38:29 svc kernel: [<ffffffffaee55afe>] process_backlog+0xae/0x180 Aug 22 01:38:29 svc kernel: [<ffffffffaee551cf>] net_rx_action+0x26f/0x390 Aug 22 01:38:29 svc kernel: [<ffffffffae8a4b95>] __do_softirq+0xf5/0x280 Aug 22 01:38:29 -svc kernel: [<ffffffffaef974ec>] call_softirq+0x1c/0x30 Aug 22 01:38:29 svc kernel: <EOI> [<ffffffffae82f715>] do_softirq+0x65/0xa0 Aug 22 01:38:29-svc kernel: [<ffffffffaee53263>] netif_rx_ni+0x33/0x70 Aug 22 01:38:29 svc kernel: [<ffffffffc07548da>] kni_net_rx_normal+0x21a/0x330 [rte_kni] Aug 22 01:38:29 svc kernel: [<ffffffffc0755f10>] ? kni_thread_single+0xa0/0xa0 [rte_kni] Aug 22 01:38:29 svc kernel: [<ffffffffc0755aa5>] kni_net_rx+0x15/0x20 [rte_kni] Aug 22 01:38:29-svc kernel: [<ffffffffc0755f40>] kni_thread_multiple+0x30/0x60 [rte_kni] Aug 22 01:38:29 svc kernel: [<ffffffffae8c5e71>] kthread+0xd1/0xe0 Aug 22 01:38:29 -svc kernel: [<ffffffffae8c5da0>] ? insert_kthread_work+0x40/0x40 Aug 22 01:38:29 svc kernel: [<ffffffffaef93ddd>] ret_from_fork_nospec_begin+0x7/0x21 Aug 22 01:38:29 svc kernel: [<ffffffffae8c5da0>] ? insert_kthread_work+0x40/0x40 Thanks, Chetan [-- Attachment #2: Type: text/html, Size: 6420 bytes --] ^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2023-08-22 8:05 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <PN3P287MB0113FC136FCAF602D48280F49F1FA@PN3P287MB0113.INDP287.PROD.OUTLOOK.COM>
2023-08-22 6:56 ` DPDK KNI problem chetan bhasin
2023-08-22 8:04 ` chetan bhasin
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).