From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 29185430CC; Tue, 22 Aug 2023 08:56:47 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id F099C4021D; Tue, 22 Aug 2023 08:56:46 +0200 (CEST) Received: from mail-ed1-f44.google.com (mail-ed1-f44.google.com [209.85.208.44]) by mails.dpdk.org (Postfix) with ESMTP id 5375040041 for ; Tue, 22 Aug 2023 08:56:46 +0200 (CEST) Received: by mail-ed1-f44.google.com with SMTP id 4fb4d7f45d1cf-52a06f5f556so2424841a12.2 for ; Mon, 21 Aug 2023 23:56:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692687405; x=1693292205; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=vyiqaBGqNkZcP8kL5dEImTCrGARqIeqW/iyoVXN+JIQ=; b=hzyWcDLMZmHCoZ8dCnizqGZBqFdjrIYMWsr/kNNHkqXcsFI8z4+CPXfrolmyl6DoB/ R7lmnLJ6IaR3uotZ6KnIINP0FvCTU1+BRR053V+chqh0eLLyV3VpVkfajlmlu0oZfChq I5J1qZtPEmmKATcYZijitEH20tO1nPrY+RunT1ielnsTFlBItX7GX2vrtF7/wmCFJfk7 TUc3QvM/R9GKWu1tyqry0UQpSapCBzd1xebTb6EOQnYHNYHLL42EJD7cZILputn2He4k kYCP2LyxT8vjhpxzJrXmZotWQz3RmNSAv5b3vqDyMvrqRVB3EKnyMfKOXmfbqNRDGbSL 7IUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692687405; x=1693292205; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=vyiqaBGqNkZcP8kL5dEImTCrGARqIeqW/iyoVXN+JIQ=; b=ELFKO8kufa5snU7TgMC6XG5V+F4d/P0EemBK22wOrXlq7gTibQTArCO7IUNcgSLnRF TnUOMHc4sdpi9jh1vjol1yrzqippN2r211PaeU1upIAcAkm4kHKHeJtZBNn6PuEuyt3P 8rjr1fl0Vdj1K45VN/Pt3NOV/+bpn5Q3ZoFWCyXoKBqj3o3/GH9M5sn/KqKnVAwTDcSE RlAy1NU7AmK3pcMqWL2GMrwB1WNaqxOc0ZOS5kUSF19M44pSabJ0i5IVEB1Jku7CMTrb Rw3yHoWKcYuqCLnbFBhLX5sK+i5BAIig4EhOSS4Hd88aEX3qWZWsWRNrmOlOTofqlqC3 xHKw== X-Gm-Message-State: AOJu0YziQtSq4Y27KdO6Amxfur/z5ljlv0Ey5Zdksd2JTz4qdDbnfg4I IZskDxfRUpxfFQ4Qc64MaEA0F7pvcS2Y7ZlUibL5+Dj3 X-Google-Smtp-Source: AGHT+IEpWiHyJIU7CBdCt95YBYUFSoNULQoT27xQ9kSnxwA2Xq2wLEiV9Vp1GdhNbNqkk/vjy5ie08BWRF+BpO/h7Zg= X-Received: by 2002:a05:6402:b35:b0:523:1a09:4e2b with SMTP id bo21-20020a0564020b3500b005231a094e2bmr5677205edb.40.1692687405199; Mon, 21 Aug 2023 23:56:45 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: chetan bhasin Date: Tue, 22 Aug 2023 12:26:30 +0530 Message-ID: Subject: DPDK KNI problem To: dev@dpdk.org Content-Type: multipart/alternative; boundary="00000000000080b7da06037d7fb6" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --00000000000080b7da06037d7fb6 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello DPDK mentors, We are facing an issue, KNI core stuck after 4-5 days of traffic. Can anybody guide us on this? We have set isolcpu, rcu_nocbs in grub for given cores. Aug 22 01:38:29 kernel: INFO: rcu_sched self-detected stall on CPU { 2} (t=3D60001 jiffies g=3D426362199 c=3D426362198 q=3D150932) Aug 22 01:38:29 svc kernel: Task dump for CPU 2: Aug 22 01:38:29 svc kernel: kni_veth3_3 R running task 0 4915 2 0x00080088 Aug 22 01:38:29 svc kernel: Call Trace: Aug 22 01:38:29 svc kernel: [] sched_show_task+0xa8/0x110 Aug 22 01:38:29svc kernel: [] dump_cpu_task+0x39/0x70 Aug 22 01:38:29 svc kernel: [] rcu_dump_cpu_stacks+0x90/0xd0 Aug 22 01:38:29 fantastic kernel: [] rcu_check_callbacks+0x442/0x730 Aug 22 01:38:29 svc kernel: [] ? tick_sched_do_timer+0x50/0x50 Aug 22 01:38:29 vc kernel: [] update_process_times+0x46/0x80 Aug 22 01:38:29 svc kernel: [] tick_sched_handle+0x30/0x7= 0 Aug 22 01:38:29 svc kernel: [] tick_sched_timer+0x39/0x80 Aug 22 01:38:29 svc kernel: [] __hrtimer_run_queues+0x10e/0x270 Aug 22 01:38:29 svc kernel: [] hrtimer_interrupt+0xaf/0x1d0 Aug 22 01:38:29 svc kernel: [] local_apic_timer_interrupt+0x3b/0x60 Aug 22 01:38:29 vc kernel: [] smp_apic_timer_interrupt+0x43/0x60 Aug 22 01:38:29 vc kernel: [] apic_timer_interrupt+0x16a/0x170 Aug 22 01:38:29 -svc kernel: [] ? native_queued_spin_lock_slowpath+0x1d/0x200 Aug 22 01:38:29 -svc kernel: [] queued_spin_lock_slowpath+0xb/0xf Aug 22 01:38:29 svc kernel: [] _raw_spin_lock_bh+0x33/0x4= 0 Aug 22 01:38:29 svc kernel: [] __nfqnl_enqueue_packet+0x56/0x160 [nfnetlink_queue] Aug 22 01:38:29 svc kernel: [] nfqnl_enqueue_packet+0x24f/0x310 [nfnetlink_queue] Aug 22 01:38:29 -svc kernel: [] nf_queue+0x186/0x1e0 Aug 22 01:38:29 svc kernel: [] nf_hook_slow+0x6c/0x110 Aug 22 01:38:29 vc kernel: [] ip_rcv+0x339/0x420 Aug 22 01:38:29 svc kernel: [] ? inet_del_offload+0x40/0x40 Aug 22 01:38:29-svc kernel: [] __netif_receive_skb_core+0x729/0xa10 Aug 22 01:38:29 svc kernel: [] ? reschedule_interrupt+0x16a/0x170 Aug 22 01:38:29 -svc kernel: [] __netif_receive_skb+0x18/0x60 Aug 22 01:38:29 svc kernel: [] process_backlog+0xae/0x180 Aug 22 01:38:29 svc kernel: [] net_rx_action+0x26f/0x390 Aug 22 01:38:29 svc kernel: [] __do_softirq+0xf5/0x280 Aug 22 01:38:29 -svc kernel: [] call_softirq+0x1c/0x30 Aug 22 01:38:29 svc kernel: [] do_softirq+0x65/0xa= 0 Aug 22 01:38:29-svc kernel: [] netif_rx_ni+0x33/0x70 Aug 22 01:38:29 svc kernel: [] kni_net_rx_normal+0x21a/0x330 [rte_kni] Aug 22 01:38:29 svc kernel: [] ? kni_thread_single+0xa0/0xa0 [rte_kni] Aug 22 01:38:29 svc kernel: [] kni_net_rx+0x15/0x20 [rte_kni] Aug 22 01:38:29-svc kernel: [] kni_thread_multiple+0x30/0x60 [rte_kni] Aug 22 01:38:29 svc kernel: [] kthread+0xd1/0xe0 Aug 22 01:38:29 -svc kernel: [] ? insert_kthread_work+0x40/0x40 Aug 22 01:38:29 svc kernel: [] ret_from_fork_nospec_begin+0x7/0x21 Aug 22 01:38:29 svc kernel: [] ? insert_kthread_work+0x40/0x40 Thanks, Chetan *Disclaimer*: =C2=A9 2023 Truminds Software Systems. This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. --00000000000080b7da06037d7fb6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello DPDK mentors,

We are facing an i= ssue, KNI core stuck after 4-5 days of traffic. Can anybody guide us on thi= s?

We have set isolcpu, rcu_nocbs in grub for given cores.<= /div>

Aug 22 01:38:29 kernel: INFO: rcu_sched self-detecte= d stall on CPU { 2}=C2=A0 (t=3D60001 jiffies g=3D426362199 c=3D426362198 q= =3D150932)

Aug 22 01:38:29 svc kernel: Task dump for CPU 2:<= /u>

Aug 22 01:38:29 svc kernel: kni_veth3_3=C2=A0=C2=A0= =C2=A0=C2=A0 R=C2=A0 running task=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 0=C2=A0 4915=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2 0x00080088

Aug 22 01:38:29 svc kernel: Call Trace:

=C2=A0

Aug 22 01:38:29 svc kernel: <IRQ>=C2=A0 [<f= fffffffae8da318>] sched_show_task+0xa8/0x110

Aug 22 01:38:29svc kernel: [<ffffffffae8de159>= ] dump_cpu_task+0x39/0x70

Aug 22 01:38:29 svc kernel: [<ffffffffae958480>= ;] rcu_dump_cpu_stacks+0x90/0xd0

Aug 22 01:38:29 fantastic kernel: [<ffffffffae95b= b42>] rcu_check_callbacks+0x442/0x730

Aug 22 01:38:29 svc kernel: [<ffffffffae910370>= ;] ? tick_sched_do_timer+0x50/0x50

Aug 22 01:38:29 vc kernel: [<ffffffffae8af186>= ] update_process_times+0x46/0x80

Aug 22 01:38:29 svc kernel: [<ffffffffae9100e0>= ;] tick_sched_handle+0x30/0x70

Aug 22 01:38:29 svc kernel: [<ffffffffae9103a9>= ;] tick_sched_timer+0x39/0x80

Aug 22 01:38:29 svc kernel: [<ffffffffae8ca26e>= ;] __hrtimer_run_queues+0x10e/0x270

Aug 22 01:38:29 svc kernel: [<ffffffffae8ca7cf>= ;] hrtimer_interrupt+0xaf/0x1d0

Aug 22 01:38:29 svc kernel: [<ffffffffae85cdfb>= ;] local_apic_timer_interrupt+0x3b/0x60

Aug 22 01:38:29 vc kernel: [<ffffffffaef98a83>= ] smp_apic_timer_interrupt+0x43/0x60

Aug 22 01:38:29 vc kernel: [<ffffffffaef94fba>= ] apic_timer_interrupt+0x16a/0x170

Aug 22 01:38:29 -svc kernel: [<ffffffffae91776d&g= t;] ? native_queued_spin_lock_slowpath+0x1d/0x200

Aug 22 01:38:29 -svc kernel: [<ffffffffaef7afe5&g= t;] queued_spin_lock_slowpath+0xb/0xf

Aug 22 01:38:29 svc kernel: [<ffffffffaef89733>= ;] _raw_spin_lock_bh+0x33/0x40

Aug 22 01:38:29 svc kernel: [<ffffffffc0901ba6>= ;] __nfqnl_enqueue_packet+0x56/0x160 [nfnetlink_queue]

Aug 22 01:38:29 svc kernel: [<ffffffffc09028af>= ;] nfqnl_enqueue_packet+0x24f/0x310 [nfnetlink_queue]

Aug 22 01:38:29 -svc kernel: [<ffffffffaee95cb6&g= t;] nf_queue+0x186/0x1e0

Aug 22 01:38:29 svc kernel: [<ffffffffaee94afc>= ;] nf_hook_slow+0x6c/0x110

Aug 22 01:38:29 vc kernel: [<ffffffffaee9f559>= ] ip_rcv+0x339/0x420

Aug 22 01:38:29 svc kernel: [<ffffffffaee9ea80>= ;] ? inet_del_offload+0x40/0x40

Aug 22 01:38:29-svc kernel: [<ffffffffaee54829>= ;] __netif_receive_skb_core+0x729/0xa10

Aug 22 01:38:29 svc kernel: [<ffffffffaef9682a>= ;] ? reschedule_interrupt+0x16a/0x170

Aug 22 01:38:29 -svc kernel: [<ffffffffaee54b28&g= t;] __netif_receive_skb+0x18/0x60

Aug 22 01:38:29 svc kernel: [<ffffffffaee55afe>= ;] process_backlog+0xae/0x180

Aug 22 01:38:29 svc kernel: [<ffffffffaee551cf>= ;] net_rx_action+0x26f/0x390

Aug 22 01:38:29 svc kernel: [<ffffffffae8a4b95>= ;] __do_softirq+0xf5/0x280

Aug 22 01:38:29 -svc kernel: [<ffffffffaef974ec&g= t;] call_softirq+0x1c/0x30

Aug 22 01:38:29 svc kernel: <EOI>=C2=A0 [<f= fffffffae82f715>] do_softirq+0x65/0xa0

Aug 22 01:38:29-svc kernel: [<ffffffffaee53263>= ;] netif_rx_ni+0x33/0x70

Aug 22 01:38:29 svc kernel: [<ffffffffc07548da>= ;] kni_net_rx_normal+0x21a/0x330 [rte_kni]

Aug 22 01:38:29 svc kernel: [<ffffffffc0755f10>= ;] ? kni_thread_single+0xa0/0xa0 [rte_kni]

Aug 22 01:38:29 svc kernel: [<ffffffffc0755aa5>= ;] kni_net_rx+0x15/0x20 [rte_kni]

Aug 22 01:38:29-svc kernel: [<ffffffffc0755f40>= ;] kni_thread_multiple+0x30/0x60 [rte_kni]

Aug 22 01:38:29 svc kernel: [<ffffffffae8c5e71>= ;] kthread+0xd1/0xe0

Aug 22 01:38:29 -svc kernel: [<ffffffffae8c5da0&g= t;] ? insert_kthread_work+0x40/0x40

Aug 22 01:38:29 svc kernel: [<ffffffffaef93ddd>= ;] ret_from_fork_nospec_begin+0x7/0x21

Aug 22 01:38:29 svc kernel: [<ffffffffae8c5da0>= ;] ? insert_kthread_work+0x40/0x40

=C2=A0

=C2=A0

Thanks,

Chetan

=C2=A0

Disclaimer: =C2=A9 2023 Trumind= s Software Systems. This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORM= ATION intended solely for the use of the addressee(s). If you are not the i= ntended recipient, please notify the sender by e-mail and delete the original message. Furthe= r, you are not to copy, disclose, or distribute this e-mail or its contents= to any other person and any such actions are unlawful.

--00000000000080b7da06037d7fb6--