From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AD71F430CE; Tue, 22 Aug 2023 10:05:13 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 93E5642B8B; Tue, 22 Aug 2023 10:05:13 +0200 (CEST) Received: from mail-lf1-f46.google.com (mail-lf1-f46.google.com [209.85.167.46]) by mails.dpdk.org (Postfix) with ESMTP id 8606242D0C for ; Tue, 22 Aug 2023 10:05:12 +0200 (CEST) Received: by mail-lf1-f46.google.com with SMTP id 2adb3069b0e04-4ff882397ecso6303353e87.3 for ; Tue, 22 Aug 2023 01:05:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692691511; x=1693296311; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=jJMMDKR9OiFVOYYhBaWZXxoCu/IDyYpOY0e6HQ/nfBw=; b=AMWXBNfXYMUmJqcbusuiK705Bdjl3LUstMhjBQwxngya9pAB/6ynsbhanYxjHa8n3q wd8VYGQR/KjIn7twOsAqan5xK7TJegfe0zcWo0Ix6LgLM5Z3FE/1t3JdGJKple3BOWL4 VbKRXhjTGGfvquqbDWxe4+SScvQs8KXxF62WOFVpXkhSS4Q9cTiInqnSzdAih8a8m9XJ PDNQwQzoAsNWpFMBEWX1nRhqWrZfjc0eeL3k9dSj5b4chPCvVUXvc1DAFZHOJFU/sy9F JAn8YWBQCzjmHW/QbbPNOSIvW4xH5YGRIaQnpLQgC8KQi+5ZJS+qVU0Q2+0bTvmFCdzu m6hQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692691511; x=1693296311; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jJMMDKR9OiFVOYYhBaWZXxoCu/IDyYpOY0e6HQ/nfBw=; b=Mi+KbLJtY5+WxzYtLM63WmZv3zFZQa0PmVw2V42wZaGhz1c6m5PmGU9sje3clRyTA3 no9n8Blpce65jzqbFFf3dJEXYmHu/TaMQuKcHgwRv9ixzhkdYn43vl+B/+Ya+PLL6AYt KyOjXMN/McaUILCN2D6lVJq2WguxgdGOLLe/XUPkY6f32D+RGiftI+AAypqWytgv0TV3 KvVX87dsyICmoCC3Rvkj6X/LyIWh+6r+DHWg7OEKGoRNOMM0t+h1Ap+3nIMHGFLzpQft 2keqBzhwYhtHh2lwZhsg5gIjLnuSHRXDBgfUpyBfnv6nItxEra3S3bqzv8SiMdEAXtsE 5VEg== X-Gm-Message-State: AOJu0YyE2K9am7oLCLsEQOXiNnGioxmCsbl1vPe1b+HqNxy+a3g2CfG4 6PXb7+PtzZulZzmMYFhZ5ensj4N2DkaVi325mfdQOt+2 X-Google-Smtp-Source: AGHT+IET1cOe9kNK6dgAkQCdRKfusnZCVNs9rxtmV4PRkYMd/RjKGMFWfZfbR7H/zsycWh203mH6eaLUiCkUMzTFuK4= X-Received: by 2002:a19:9159:0:b0:4fb:cab9:ddf with SMTP id y25-20020a199159000000b004fbcab90ddfmr6219118lfj.57.1692691511135; Tue, 22 Aug 2023 01:05:11 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: chetan bhasin Date: Tue, 22 Aug 2023 13:34:56 +0530 Message-ID: Subject: DPDK KNI problem To: dev@dpdk.org Content-Type: multipart/alternative; boundary="0000000000003c534406037e7432" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --0000000000003c534406037e7432 Content-Type: text/plain; charset="UTF-8" Hello DPDK mentors, We are facing an issue, KNI core stuck after 4-5 days of traffic. Can anybody guide us on this? We have set isolcpu, rcu_nocbs in grub for given cores. Aug 22 01:38:29 kernel: INFO: rcu_sched self-detected stall on CPU { 2} (t=60001 jiffies g=426362199 c=426362198 q=150932) Aug 22 01:38:29 svc kernel: Task dump for CPU 2: Aug 22 01:38:29 svc kernel: kni_veth3_3 R running task 0 4915 2 0x00080088 Aug 22 01:38:29 svc kernel: Call Trace: Aug 22 01:38:29 svc kernel: [] sched_show_task+0xa8/0x110 Aug 22 01:38:29svc kernel: [] dump_cpu_task+0x39/0x70 Aug 22 01:38:29 svc kernel: [] rcu_dump_cpu_stacks+0x90/0xd0 Aug 22 01:38:29 fantastic kernel: [] rcu_check_callbacks+0x442/0x730 Aug 22 01:38:29 svc kernel: [] ? tick_sched_do_timer+0x50/0x50 Aug 22 01:38:29 vc kernel: [] update_process_times+0x46/0x80 Aug 22 01:38:29 svc kernel: [] tick_sched_handle+0x30/0x70 Aug 22 01:38:29 svc kernel: [] tick_sched_timer+0x39/0x80 Aug 22 01:38:29 svc kernel: [] __hrtimer_run_queues+0x10e/0x270 Aug 22 01:38:29 svc kernel: [] hrtimer_interrupt+0xaf/0x1d0 Aug 22 01:38:29 svc kernel: [] local_apic_timer_interrupt+0x3b/0x60 Aug 22 01:38:29 vc kernel: [] smp_apic_timer_interrupt+0x43/0x60 Aug 22 01:38:29 vc kernel: [] apic_timer_interrupt+0x16a/0x170 Aug 22 01:38:29 -svc kernel: [] ? native_queued_spin_lock_slowpath+0x1d/0x200 Aug 22 01:38:29 -svc kernel: [] queued_spin_lock_slowpath+0xb/0xf Aug 22 01:38:29 svc kernel: [] _raw_spin_lock_bh+0x33/0x40 Aug 22 01:38:29 svc kernel: [] __nfqnl_enqueue_packet+0x56/0x160 [nfnetlink_queue] Aug 22 01:38:29 svc kernel: [] nfqnl_enqueue_packet+0x24f/0x310 [nfnetlink_queue] Aug 22 01:38:29 -svc kernel: [] nf_queue+0x186/0x1e0 Aug 22 01:38:29 svc kernel: [] nf_hook_slow+0x6c/0x110 Aug 22 01:38:29 vc kernel: [] ip_rcv+0x339/0x420 Aug 22 01:38:29 svc kernel: [] ? inet_del_offload+0x40/0x40 Aug 22 01:38:29-svc kernel: [] __netif_receive_skb_core+0x729/0xa10 Aug 22 01:38:29 svc kernel: [] ? reschedule_interrupt+0x16a/0x170 Aug 22 01:38:29 -svc kernel: [] __netif_receive_skb+0x18/0x60 Aug 22 01:38:29 svc kernel: [] process_backlog+0xae/0x180 Aug 22 01:38:29 svc kernel: [] net_rx_action+0x26f/0x390 Aug 22 01:38:29 svc kernel: [] __do_softirq+0xf5/0x280 Aug 22 01:38:29 -svc kernel: [] call_softirq+0x1c/0x30 Aug 22 01:38:29 svc kernel: [] do_softirq+0x65/0xa0 Aug 22 01:38:29-svc kernel: [] netif_rx_ni+0x33/0x70 Aug 22 01:38:29 svc kernel: [] kni_net_rx_normal+0x21a/0x330 [rte_kni] Aug 22 01:38:29 svc kernel: [] ? kni_thread_single+0xa0/0xa0 [rte_kni] Aug 22 01:38:29 svc kernel: [] kni_net_rx+0x15/0x20 [rte_kni] Aug 22 01:38:29-svc kernel: [] kni_thread_multiple+0x30/0x60 [rte_kni] Aug 22 01:38:29 svc kernel: [] kthread+0xd1/0xe0 Aug 22 01:38:29 -svc kernel: [] ? insert_kthread_work+0x40/0x40 Aug 22 01:38:29 svc kernel: [] ret_from_fork_nospec_begin+0x7/0x21 Aug 22 01:38:29 svc kernel: [] ? insert_kthread_work+0x40/0x40 Thanks, Chetan --0000000000003c534406037e7432 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


Hello DPDK mentors,=

We are facing an issue, KNI core stuck after 4-5 days of t= raffic. Can anybody guide us on this?

We have set isolcpu, = rcu_nocbs in grub for given cores.

Aug 22 01:38:29 kernel: INFO: rcu_sched self-detecte= d stall on CPU { 2}=C2=A0 (t=3D60001 jiffies g=3D426362199 c=3D426362198 q= =3D150932)

Aug 22 01:38:29 svc kernel: Task dump for CPU 2:<= /u>

Aug 22 01:38:29 svc kernel: kni_veth3_3=C2=A0=C2=A0= =C2=A0=C2=A0 R=C2=A0 running task=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 0=C2=A0 4915=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2 0x00080088

Aug 22 01:38:29 svc kernel: Call Trace:

=C2=A0

Aug 22 01:38:29 svc kernel: <IRQ>=C2=A0 [<f= fffffffae8da318>] sched_show_task+0xa8/0x110

Aug 22 01:38:29svc kernel: [<ffffffffae8de159>= ] dump_cpu_task+0x39/0x70

Aug 22 01:38:29 svc kernel: [<ffffffffae958480>= ;] rcu_dump_cpu_stacks+0x90/0xd0

Aug 22 01:38:29 fantastic kernel: [<ffffffffae95b= b42>] rcu_check_callbacks+0x442/0x730

Aug 22 01:38:29 svc kernel: [<ffffffffae910370>= ;] ? tick_sched_do_timer+0x50/0x50

Aug 22 01:38:29 vc kernel: [<ffffffffae8af186>= ] update_process_times+0x46/0x80

Aug 22 01:38:29 svc kernel: [<ffffffffae9100e0>= ;] tick_sched_handle+0x30/0x70

Aug 22 01:38:29 svc kernel: [<ffffffffae9103a9>= ;] tick_sched_timer+0x39/0x80

Aug 22 01:38:29 svc kernel: [<ffffffffae8ca26e>= ;] __hrtimer_run_queues+0x10e/0x270

Aug 22 01:38:29 svc kernel: [<ffffffffae8ca7cf>= ;] hrtimer_interrupt+0xaf/0x1d0

Aug 22 01:38:29 svc kernel: [<ffffffffae85cdfb>= ;] local_apic_timer_interrupt+0x3b/0x60

Aug 22 01:38:29 vc kernel: [<ffffffffaef98a83>= ] smp_apic_timer_interrupt+0x43/0x60

Aug 22 01:38:29 vc kernel: [<ffffffffaef94fba>= ] apic_timer_interrupt+0x16a/0x170

Aug 22 01:38:29 -svc kernel: [<ffffffffae91776d&g= t;] ? native_queued_spin_lock_slowpath+0x1d/0x200

Aug 22 01:38:29 -svc kernel: [<ffffffffaef7afe5&g= t;] queued_spin_lock_slowpath+0xb/0xf

Aug 22 01:38:29 svc kernel: [<ffffffffaef89733>= ;] _raw_spin_lock_bh+0x33/0x40

Aug 22 01:38:29 svc kernel: [<ffffffffc0901ba6>= ;] __nfqnl_enqueue_packet+0x56/0x160 [nfnetlink_queue]

Aug 22 01:38:29 svc kernel: [<ffffffffc09028af>= ;] nfqnl_enqueue_packet+0x24f/0x310 [nfnetlink_queue]

Aug 22 01:38:29 -svc kernel: [<ffffffffaee95cb6&g= t;] nf_queue+0x186/0x1e0

Aug 22 01:38:29 svc kernel: [<ffffffffaee94afc>= ;] nf_hook_slow+0x6c/0x110

Aug 22 01:38:29 vc kernel: [<ffffffffaee9f559>= ] ip_rcv+0x339/0x420

Aug 22 01:38:29 svc kernel: [<ffffffffaee9ea80>= ;] ? inet_del_offload+0x40/0x40

Aug 22 01:38:29-svc kernel: [<ffffffffaee54829>= ;] __netif_receive_skb_core+0x729/0xa10

Aug 22 01:38:29 svc kernel: [<ffffffffaef9682a>= ;] ? reschedule_interrupt+0x16a/0x170

Aug 22 01:38:29 -svc kernel: [<ffffffffaee54b28&g= t;] __netif_receive_skb+0x18/0x60

Aug 22 01:38:29 svc kernel: [<ffffffffaee55afe>= ;] process_backlog+0xae/0x180

Aug 22 01:38:29 svc kernel: [<ffffffffaee551cf>= ;] net_rx_action+0x26f/0x390

Aug 22 01:38:29 svc kernel: [<ffffffffae8a4b95>= ;] __do_softirq+0xf5/0x280

Aug 22 01:38:29 -svc kernel: [<ffffffffaef974ec&g= t;] call_softirq+0x1c/0x30

Aug 22 01:38:29 svc kernel: <EOI>=C2=A0 [<f= fffffffae82f715>] do_softirq+0x65/0xa0

Aug 22 01:38:29-svc kernel: [<ffffffffaee53263>= ;] netif_rx_ni+0x33/0x70

Aug 22 01:38:29 svc kernel: [<ffffffffc07548da>= ;] kni_net_rx_normal+0x21a/0x330 [rte_kni]

Aug 22 01:38:29 svc kernel: [<ffffffffc0755f10>= ;] ? kni_thread_single+0xa0/0xa0 [rte_kni]

Aug 22 01:38:29 svc kernel: [<ffffffffc0755aa5>= ;] kni_net_rx+0x15/0x20 [rte_kni]

Aug 22 01:38:29-svc kernel: [<ffffffffc0755f40>= ;] kni_thread_multiple+0x30/0x60 [rte_kni]

Aug 22 01:38:29 svc kernel: [<ffffffffae8c5e71>= ;] kthread+0xd1/0xe0

Aug 22 01:38:29 -svc kernel: [<ffffffffae8c5da0&g= t;] ? insert_kthread_work+0x40/0x40

Aug 22 01:38:29 svc kernel: [<ffffffffaef93ddd>= ;] ret_from_fork_nospec_begin+0x7/0x21

Aug 22 01:38:29 svc kernel: [<ffffffffae8c5da0>= ;] ? insert_kthread_work+0x40/0x40

=C2=A0

=C2=A0

Thanks,

Chetan



--0000000000003c534406037e7432--