* DPDK KNI problem
[not found] <PN3P287MB0113FC136FCAF602D48280F49F1FA@PN3P287MB0113.INDP287.PROD.OUTLOOK.COM>
@ 2023-08-22 6:56 ` chetan bhasin
2023-08-22 8:04 ` chetan bhasin
0 siblings, 1 reply; 2+ messages in thread
From: chetan bhasin @ 2023-08-22 6:56 UTC (permalink / raw)
To: dev
[-- Attachment #1: Type: text/plain, Size: 4236 bytes --]
Hello DPDK mentors,
We are facing an issue, KNI core stuck after 4-5 days of traffic. Can
anybody guide us on this?
We have set isolcpu, rcu_nocbs in grub for given cores.
Aug 22 01:38:29 kernel: INFO: rcu_sched self-detected stall on CPU { 2}
(t=60001 jiffies g=426362199 c=426362198 q=150932)
Aug 22 01:38:29 svc kernel: Task dump for CPU 2:
Aug 22 01:38:29 svc kernel: kni_veth3_3 R running task 0
4915 2 0x00080088
Aug 22 01:38:29 svc kernel: Call Trace:
Aug 22 01:38:29 svc kernel: <IRQ> [<ffffffffae8da318>]
sched_show_task+0xa8/0x110
Aug 22 01:38:29svc kernel: [<ffffffffae8de159>] dump_cpu_task+0x39/0x70
Aug 22 01:38:29 svc kernel: [<ffffffffae958480>]
rcu_dump_cpu_stacks+0x90/0xd0
Aug 22 01:38:29 fantastic kernel: [<ffffffffae95bb42>]
rcu_check_callbacks+0x442/0x730
Aug 22 01:38:29 svc kernel: [<ffffffffae910370>] ?
tick_sched_do_timer+0x50/0x50
Aug 22 01:38:29 vc kernel: [<ffffffffae8af186>]
update_process_times+0x46/0x80
Aug 22 01:38:29 svc kernel: [<ffffffffae9100e0>] tick_sched_handle+0x30/0x70
Aug 22 01:38:29 svc kernel: [<ffffffffae9103a9>] tick_sched_timer+0x39/0x80
Aug 22 01:38:29 svc kernel: [<ffffffffae8ca26e>]
__hrtimer_run_queues+0x10e/0x270
Aug 22 01:38:29 svc kernel: [<ffffffffae8ca7cf>]
hrtimer_interrupt+0xaf/0x1d0
Aug 22 01:38:29 svc kernel: [<ffffffffae85cdfb>]
local_apic_timer_interrupt+0x3b/0x60
Aug 22 01:38:29 vc kernel: [<ffffffffaef98a83>]
smp_apic_timer_interrupt+0x43/0x60
Aug 22 01:38:29 vc kernel: [<ffffffffaef94fba>]
apic_timer_interrupt+0x16a/0x170
Aug 22 01:38:29 -svc kernel: [<ffffffffae91776d>] ?
native_queued_spin_lock_slowpath+0x1d/0x200
Aug 22 01:38:29 -svc kernel: [<ffffffffaef7afe5>]
queued_spin_lock_slowpath+0xb/0xf
Aug 22 01:38:29 svc kernel: [<ffffffffaef89733>] _raw_spin_lock_bh+0x33/0x40
Aug 22 01:38:29 svc kernel: [<ffffffffc0901ba6>]
__nfqnl_enqueue_packet+0x56/0x160 [nfnetlink_queue]
Aug 22 01:38:29 svc kernel: [<ffffffffc09028af>]
nfqnl_enqueue_packet+0x24f/0x310 [nfnetlink_queue]
Aug 22 01:38:29 -svc kernel: [<ffffffffaee95cb6>] nf_queue+0x186/0x1e0
Aug 22 01:38:29 svc kernel: [<ffffffffaee94afc>] nf_hook_slow+0x6c/0x110
Aug 22 01:38:29 vc kernel: [<ffffffffaee9f559>] ip_rcv+0x339/0x420
Aug 22 01:38:29 svc kernel: [<ffffffffaee9ea80>] ?
inet_del_offload+0x40/0x40
Aug 22 01:38:29-svc kernel: [<ffffffffaee54829>]
__netif_receive_skb_core+0x729/0xa10
Aug 22 01:38:29 svc kernel: [<ffffffffaef9682a>] ?
reschedule_interrupt+0x16a/0x170
Aug 22 01:38:29 -svc kernel: [<ffffffffaee54b28>]
__netif_receive_skb+0x18/0x60
Aug 22 01:38:29 svc kernel: [<ffffffffaee55afe>] process_backlog+0xae/0x180
Aug 22 01:38:29 svc kernel: [<ffffffffaee551cf>] net_rx_action+0x26f/0x390
Aug 22 01:38:29 svc kernel: [<ffffffffae8a4b95>] __do_softirq+0xf5/0x280
Aug 22 01:38:29 -svc kernel: [<ffffffffaef974ec>] call_softirq+0x1c/0x30
Aug 22 01:38:29 svc kernel: <EOI> [<ffffffffae82f715>] do_softirq+0x65/0xa0
Aug 22 01:38:29-svc kernel: [<ffffffffaee53263>] netif_rx_ni+0x33/0x70
Aug 22 01:38:29 svc kernel: [<ffffffffc07548da>]
kni_net_rx_normal+0x21a/0x330 [rte_kni]
Aug 22 01:38:29 svc kernel: [<ffffffffc0755f10>] ?
kni_thread_single+0xa0/0xa0 [rte_kni]
Aug 22 01:38:29 svc kernel: [<ffffffffc0755aa5>] kni_net_rx+0x15/0x20
[rte_kni]
Aug 22 01:38:29-svc kernel: [<ffffffffc0755f40>]
kni_thread_multiple+0x30/0x60 [rte_kni]
Aug 22 01:38:29 svc kernel: [<ffffffffae8c5e71>] kthread+0xd1/0xe0
Aug 22 01:38:29 -svc kernel: [<ffffffffae8c5da0>] ?
insert_kthread_work+0x40/0x40
Aug 22 01:38:29 svc kernel: [<ffffffffaef93ddd>]
ret_from_fork_nospec_begin+0x7/0x21
Aug 22 01:38:29 svc kernel: [<ffffffffae8c5da0>] ?
insert_kthread_work+0x40/0x40
Thanks,
Chetan
*Disclaimer*: © 2023 Truminds Software Systems. This e-mail contains
PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the
addressee(s). If you are not the intended recipient, please notify the
sender by e-mail and delete the original message. Further, you are not to
copy, disclose, or distribute this e-mail or its contents to any other
person and any such actions are unlawful.
[-- Attachment #2: Type: text/html, Size: 6822 bytes --]
^ permalink raw reply [flat|nested] 2+ messages in thread
* DPDK KNI problem
2023-08-22 6:56 ` DPDK KNI problem chetan bhasin
@ 2023-08-22 8:04 ` chetan bhasin
0 siblings, 0 replies; 2+ messages in thread
From: chetan bhasin @ 2023-08-22 8:04 UTC (permalink / raw)
To: dev
[-- Attachment #1: Type: text/plain, Size: 3692 bytes --]
Hello DPDK mentors,
We are facing an issue, KNI core stuck after 4-5 days of traffic. Can
anybody guide us on this?
We have set isolcpu, rcu_nocbs in grub for given cores.
Aug 22 01:38:29 kernel: INFO: rcu_sched self-detected stall on CPU { 2}
(t=60001 jiffies g=426362199 c=426362198 q=150932)
Aug 22 01:38:29 svc kernel: Task dump for CPU 2:
Aug 22 01:38:29 svc kernel: kni_veth3_3 R running task 0
4915 2 0x00080088
Aug 22 01:38:29 svc kernel: Call Trace:
Aug 22 01:38:29 svc kernel: <IRQ> [<ffffffffae8da318>]
sched_show_task+0xa8/0x110
Aug 22 01:38:29svc kernel: [<ffffffffae8de159>] dump_cpu_task+0x39/0x70
Aug 22 01:38:29 svc kernel: [<ffffffffae958480>]
rcu_dump_cpu_stacks+0x90/0xd0
Aug 22 01:38:29 fantastic kernel: [<ffffffffae95bb42>]
rcu_check_callbacks+0x442/0x730
Aug 22 01:38:29 svc kernel: [<ffffffffae910370>] ?
tick_sched_do_timer+0x50/0x50
Aug 22 01:38:29 vc kernel: [<ffffffffae8af186>]
update_process_times+0x46/0x80
Aug 22 01:38:29 svc kernel: [<ffffffffae9100e0>] tick_sched_handle+0x30/0x70
Aug 22 01:38:29 svc kernel: [<ffffffffae9103a9>] tick_sched_timer+0x39/0x80
Aug 22 01:38:29 svc kernel: [<ffffffffae8ca26e>]
__hrtimer_run_queues+0x10e/0x270
Aug 22 01:38:29 svc kernel: [<ffffffffae8ca7cf>]
hrtimer_interrupt+0xaf/0x1d0
Aug 22 01:38:29 svc kernel: [<ffffffffae85cdfb>]
local_apic_timer_interrupt+0x3b/0x60
Aug 22 01:38:29 vc kernel: [<ffffffffaef98a83>]
smp_apic_timer_interrupt+0x43/0x60
Aug 22 01:38:29 vc kernel: [<ffffffffaef94fba>]
apic_timer_interrupt+0x16a/0x170
Aug 22 01:38:29 -svc kernel: [<ffffffffae91776d>] ?
native_queued_spin_lock_slowpath+0x1d/0x200
Aug 22 01:38:29 -svc kernel: [<ffffffffaef7afe5>]
queued_spin_lock_slowpath+0xb/0xf
Aug 22 01:38:29 svc kernel: [<ffffffffaef89733>] _raw_spin_lock_bh+0x33/0x40
Aug 22 01:38:29 svc kernel: [<ffffffffc0901ba6>]
__nfqnl_enqueue_packet+0x56/0x160 [nfnetlink_queue]
Aug 22 01:38:29 svc kernel: [<ffffffffc09028af>]
nfqnl_enqueue_packet+0x24f/0x310 [nfnetlink_queue]
Aug 22 01:38:29 -svc kernel: [<ffffffffaee95cb6>] nf_queue+0x186/0x1e0
Aug 22 01:38:29 svc kernel: [<ffffffffaee94afc>] nf_hook_slow+0x6c/0x110
Aug 22 01:38:29 vc kernel: [<ffffffffaee9f559>] ip_rcv+0x339/0x420
Aug 22 01:38:29 svc kernel: [<ffffffffaee9ea80>] ?
inet_del_offload+0x40/0x40
Aug 22 01:38:29-svc kernel: [<ffffffffaee54829>]
__netif_receive_skb_core+0x729/0xa10
Aug 22 01:38:29 svc kernel: [<ffffffffaef9682a>] ?
reschedule_interrupt+0x16a/0x170
Aug 22 01:38:29 -svc kernel: [<ffffffffaee54b28>]
__netif_receive_skb+0x18/0x60
Aug 22 01:38:29 svc kernel: [<ffffffffaee55afe>] process_backlog+0xae/0x180
Aug 22 01:38:29 svc kernel: [<ffffffffaee551cf>] net_rx_action+0x26f/0x390
Aug 22 01:38:29 svc kernel: [<ffffffffae8a4b95>] __do_softirq+0xf5/0x280
Aug 22 01:38:29 -svc kernel: [<ffffffffaef974ec>] call_softirq+0x1c/0x30
Aug 22 01:38:29 svc kernel: <EOI> [<ffffffffae82f715>] do_softirq+0x65/0xa0
Aug 22 01:38:29-svc kernel: [<ffffffffaee53263>] netif_rx_ni+0x33/0x70
Aug 22 01:38:29 svc kernel: [<ffffffffc07548da>]
kni_net_rx_normal+0x21a/0x330 [rte_kni]
Aug 22 01:38:29 svc kernel: [<ffffffffc0755f10>] ?
kni_thread_single+0xa0/0xa0 [rte_kni]
Aug 22 01:38:29 svc kernel: [<ffffffffc0755aa5>] kni_net_rx+0x15/0x20
[rte_kni]
Aug 22 01:38:29-svc kernel: [<ffffffffc0755f40>]
kni_thread_multiple+0x30/0x60 [rte_kni]
Aug 22 01:38:29 svc kernel: [<ffffffffae8c5e71>] kthread+0xd1/0xe0
Aug 22 01:38:29 -svc kernel: [<ffffffffae8c5da0>] ?
insert_kthread_work+0x40/0x40
Aug 22 01:38:29 svc kernel: [<ffffffffaef93ddd>]
ret_from_fork_nospec_begin+0x7/0x21
Aug 22 01:38:29 svc kernel: [<ffffffffae8c5da0>] ?
insert_kthread_work+0x40/0x40
Thanks,
Chetan
[-- Attachment #2: Type: text/html, Size: 6420 bytes --]
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2023-08-22 8:05 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <PN3P287MB0113FC136FCAF602D48280F49F1FA@PN3P287MB0113.INDP287.PROD.OUTLOOK.COM>
2023-08-22 6:56 ` DPDK KNI problem chetan bhasin
2023-08-22 8:04 ` chetan bhasin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).