From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9619142A14 for ; Fri, 28 Apr 2023 14:36:56 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 862324113C; Fri, 28 Apr 2023 14:36:54 +0200 (CEST) Received: from mail-yw1-f181.google.com (mail-yw1-f181.google.com [209.85.128.181]) by mails.dpdk.org (Postfix) with ESMTP id 3406140A87 for ; Fri, 28 Apr 2023 14:36:53 +0200 (CEST) Received: by mail-yw1-f181.google.com with SMTP id 00721157ae682-54f6f0dae19so79282957b3.0 for ; Fri, 28 Apr 2023 05:36:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1682685412; x=1685277412; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=TA/9IjXasYaLrcy32bf5e4Q1cKDHV5ABZKo+u9FE0Ec=; b=lBfBT6SnrjECtik+v9TWHqhOa88a3X1GokCWbfN0QQWQOiilfG4is/61Q2U1ZgkJPK W17T1i91ZEJepRMdgOEyxmT42XbINeX53/Dg+Ibv0CjH8VE4zWRHakri54eFKAb7ZO4u MT6TS7vcxFWdR+eY1+Jte6HN7jvau2IsT23/OKiajz8K1C3mACoT535AZ3NDBFysa6k9 XjfDzaTaBEYndl86TQ9e7DSZrodKTkpGWpBB+NeRfHAj0K1LAe6soTFUqa0QhpUZLhpe rv7wlyZbiUopk1jhE61NmlfRagc2X03RP5sy4NbPzgSYUMC4FJl9Vu6XwHLD2xr6Dj6r /sUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682685412; x=1685277412; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=TA/9IjXasYaLrcy32bf5e4Q1cKDHV5ABZKo+u9FE0Ec=; b=WvTEoh51XDy8yarYeb7asMPrAemcVYjjLWLYPdxyipPoQP4bZ5NKv/JgjkoOjo9CXU PL3VrY9rcqoZ8/B3wVd+AUc0Y28iAgHa0ch2lW52xt58u2oYdBX0CdlQ+RMyGMYIhPAL TzRC2rwzMMIqnUTrf0O+c97Jf9no6OVYuAyJYTw4SpVZnYRdQSq/Y21G/AaQdUJXJqZY fyBR7MYeY29+6h8BkhAIlTEcJnKPdtvSCn6BApBNfr9DKgMywhGs07yisSXk/A3Z4QiV 3D4Nwp6zIGGSZnWX2Eyv9QsiMsIAlbn3W4uSPq6ZIwTt4JURw17EK5jkSc1LMG+xuwPs 7zNA== X-Gm-Message-State: AC+VfDz1EtyBcaQTTV8zDoz/9w1ebJ4Ujpgtnj06OiZem/RKM2Fdo1sk Fe3/Kil25UANWIqBnXBmhCLV5VZXbnbA9Jd0zHxOCGaD6mg= X-Google-Smtp-Source: ACHHUZ66JHn5M+OtgbNnlumPOU2WpZUGixUAF588b22WY2HzBLH7raOESF9HHLc5JrhlEV+Vy0oSs3CwvM5Lrv4b0k8= X-Received: by 2002:a0d:d88b:0:b0:54f:32a5:efde with SMTP id a133-20020a0dd88b000000b0054f32a5efdemr3716155ywe.10.1682685412149; Fri, 28 Apr 2023 05:36:52 -0700 (PDT) MIME-Version: 1.0 From: Hari Haran Date: Fri, 28 Apr 2023 18:06:40 +0530 Message-ID: Subject: DPDK Event Dev Queue Enqueue failure observed To: users@dpdk.org Content-Type: multipart/alternative; boundary="000000000000428c9905fa64ba0c" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --000000000000428c9905fa64ba0c Content-Type: text/plain; charset="UTF-8" In Data Application single Process used, not multi process case. During posting packets from one thread to another thread(both threads are in different cores), we have used DPDK Event Dev Queue. Observed Event Dev Queue Enqueue failure. Cross checked the EvtDevQ stats, it showed packets are in Inflight mode, not posted to Queue it seems and enqueue failed and not recovered afterwards. Also schedule calls happened multiple times. Note: Receiver thread is not in tight loop Event Enqueue function: rte_event_enqueue_burst() Schedule call function: rte_service_run_iter_on_app_lcore() How can I fix it? Any help is appreciated. Port 0 and Queue 0 Used in this case. Dev id: 0 Required Logs information below: *EAL Arguments used:* ArgCnt 0 Value Appuserplane ArgCnt 1 Value -l 93,38,94,39 ArgCnt 2 Value -n 2 ArgCnt 3 Value --main-lcore=93 *EAL prints during Init:* EAL: Detected 112 lcore(s) EAL: Detected 2 NUMA nodes EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: No available hugepages reported in hugepages-2048kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: No legacy callbacks, legacy socket not created *Event Dev Stats:* Dev=0 Port=1EventDev todo-fix-name: ports 4, qids 3 rx 32768 drop 0 tx 4096 sched calls: 628945658 sched cq/qid call: 628964843 sched no IQ enq: 628926401 sched no CQ enq: 628942982 inflight 32768, credits: 0 Port 0 rx 0 drop 0 tx 4096 inflight 4096 Max New: 32768 Avg cycles PP: 0 Credits: 0 Receive burst distribution: 0:100% 1-4:0.00% 5-8:0.00% 9-12:0.00% rx ring used: 0 free: 4096 cq ring used: 0 free: 128 Port 1 rx 0 drop 0 tx 0 inflight 0 Max New: 32768 Avg cycles PP: 0 Credits: 0 Receive burst distribution: 0:100% rx ring used: 0 free: 4096 cq ring used: 0 free: 128 Port 2 rx 32768 drop 0 tx 0 inflight 0 Max New: 32768 Avg cycles PP: 0 Credits: 0 Receive burst distribution: 0:-nan% rx ring used: 0 free: 4096 cq ring used: 0 free: 128 Port 3 (SingleCons) rx 0 drop 0 tx 0 inflight 0 Max New: 32768 Avg cycles PP: 0 Credits: 0 Receive burst distribution: 0:-nan% rx ring used: 0 free: 4096 cq ring used: 0 free: 128 Queue 0 (Atomic) rx 32768 drop 0 tx 4096 Per Port Stats: Port 0: Pkts: 4096 Flows: 1 Port 1: Pkts: 0 Flows: 0 Port 2: Pkts: 0 Flows: 0 Port 3: Pkts: 0 Flows: 0 iq 0: Used 28672 Queue 1 (Atomic) rx 0 drop 0 tx 0 Per Port Stats: Port 0: Pkts: 0 Flows: 0 Port 1: Pkts: 0 Flows: 0 Port 2: Pkts: 0 Flows: 0 Port 3: Pkts: 0 Flows: 0 -- iqs empty -- Queue 2 (Directed) rx 0 drop 0 tx 0 Per Port Stats: Port 0: Pkts: 0 Flows: 0 Port 1: Pkts: 0 Flows: 0 Port 2: Pkts: 0 Flows: 0 Port 3: Pkts: 0 Flows: 0 -- iqs empty -- Regards, Hariharan --000000000000428c9905fa64ba0c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
In Data Application single Process used, not multi process= case.

During posting packets from one thread to another thread(bot= h threads are in different cores), we have used DPDK Event Dev Queue.
Observed Event Dev Queue Enqueue failure. Cross checked the EvtDevQ stats= , it showed packets are in Inflight mode, not posted to Queue it seems and = enqueue failed and not recovered afterwards.
Also schedule calls happene= d multiple times.

Note: Receiver thread is not in tight loop

= Event Enqueue function: rte_event_enqueue_burst()

Schedule call func= tion: rte_service_run_iter_on_app_lcore()

How can I fix it? Any help= is appreciated.

Port 0 and Queue 0 Used in this case.
Dev id: 0=

Required Logs information below:

EAL Arguments use= d:

ArgCnt 0 Value Appuserplane
ArgCnt 1 Value -l 93,38,94,39<= br>ArgCnt 2 Value -n 2
ArgCnt 3 Value --main-lcore=3D93

EAL pr= ints during Init:

EAL: Detected 112 lcore(s)
EAL: Detected 2 = NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process so= cket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing V= FIO support...
EAL: VFIO support initialized
EAL: No legacy callbacks= , legacy socket not created


Event Dev S= tats:

Dev=3D0 Port=3D1EventDev todo-fix-name: ports 4, qids 3=C2=A0 =C2=A0 =C2=A0rx =C2=A0 32768
=C2=A0 =C2=A0 =C2=A0drop 0
=C2= =A0 =C2=A0 =C2=A0tx =C2=A0 4096
=C2=A0 =C2=A0 =C2=A0sched calls: 6289456= 58
=C2=A0 =C2=A0 =C2=A0sched cq/qid call: 628964843
=C2=A0 =C2=A0 =C2= =A0sched no IQ enq: 628926401
=C2=A0 =C2=A0 =C2=A0sched no CQ enq: 62894= 2982
=C2=A0 =C2=A0 =C2=A0inflight 32768, credits: 0
=C2=A0 =C2=A0Port= 0
=C2=A0 =C2=A0 =C2=A0rx =C2=A0 0 =C2=A0drop 0 =C2=A0tx =C2=A0 4096 =C2= =A0 =C2=A0inflight 4096
=C2=A0 =C2=A0 =C2=A0Max New: 32768 =C2=A0Avg cy= cles PP: 0 =C2=A0 =C2=A0Credits: 0
=C2=A0 =C2=A0 =C2=A0Receive burst dis= tribution:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00:100% 1-4:0.00% 5-8:0.00% = 9-12:0.00%
=C2=A0 =C2=A0 =C2=A0 rx ring used: =C2=A0 =C2=A00 free: 4096 =
=C2=A0 =C2=A0 =C2=A0 cq ring used: =C2=A0 =C2=A00 free: =C2=A0128
= =C2=A0 =C2=A0Port 1
=C2=A0 =C2=A0 =C2=A0rx =C2=A0 0 =C2=A0drop 0 =C2=A0t= x =C2=A0 0 =C2=A0 inflight 0
=C2=A0 =C2=A0 =C2=A0Max New: 32768 =C2=A0A= vg cycles PP: 0 =C2=A0 =C2=A0Credits: 0
=C2=A0 =C2=A0 =C2=A0Receive burs= t distribution:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00:100%
=C2=A0 =C2= =A0 =C2=A0 rx ring used: =C2=A0 =C2=A00 free: 4096
=C2=A0 =C2=A0 =C2=A0= cq ring used: =C2=A0 =C2=A00 free: =C2=A0128
=C2=A0 =C2=A0Port 2
= =C2=A0 =C2=A0 =C2=A0rx =C2=A0 32768 =C2=A0drop 0 =C2=A0tx =C2=A0 0 =C2=A0 i= nflight 0
=C2=A0 =C2=A0 =C2=A0Max New: 32768 =C2=A0Avg cycles PP: 0 =C2= =A0 =C2=A0Credits: 0
=C2=A0 =C2=A0 =C2=A0Receive burst distribution:
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00:-nan%
=C2=A0 =C2=A0 =C2=A0 rx ring u= sed: =C2=A0 =C2=A00 free: 4096
=C2=A0 =C2=A0 =C2=A0 cq ring used: =C2= =A0 =C2=A00 free: =C2=A0128
=C2=A0 =C2=A0Port 3 =C2=A0(SingleCons)
= =C2=A0 =C2=A0 =C2=A0rx =C2=A0 0 =C2=A0drop 0 =C2=A0tx =C2=A0 0 =C2=A0 infli= ght 0
=C2=A0 =C2=A0 =C2=A0Max New: 32768 =C2=A0Avg cycles PP: 0 =C2=A0 = =C2=A0Credits: 0
=C2=A0 =C2=A0 =C2=A0Receive burst distribution:
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00:-nan%
=C2=A0 =C2=A0 =C2=A0 rx ring used= : =C2=A0 =C2=A00 free: 4096
=C2=A0 =C2=A0 =C2=A0 cq ring used: =C2=A0 = =C2=A00 free: =C2=A0128
=C2=A0 =C2=A0Queue 0 (Atomic)
=C2=A0 =C2=A0 = =C2=A0rx =C2=A0 32768 =C2=A0drop 0 =C2=A0tx =C2=A0 4096
=C2=A0 =C2=A0 = =C2=A0Per Port Stats:
=C2=A0 =C2=A0 =C2=A0 =C2=A0Port 0: Pkts: 4096 =C2= =A0 =C2=A0Flows: 1
=C2=A0 =C2=A0 =C2=A0 =C2=A0Port 1: Pkts: 0 =C2=A0 Flo= ws: 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0Port 2: Pkts: 0 =C2=A0 Flows: 0
=C2= =A0 =C2=A0 =C2=A0 =C2=A0Port 3: Pkts: 0 =C2=A0 Flows: 0
=C2=A0 =C2=A0 = =C2=A0 iq 0: Used 28672
=C2=A0 =C2=A0Queue 1 (Atomic)
=C2=A0 =C2=A0 = =C2=A0rx =C2=A0 0 =C2=A0drop 0 =C2=A0tx =C2=A0 0
=C2=A0 =C2=A0 =C2=A0Per= Port Stats:
=C2=A0 =C2=A0 =C2=A0 =C2=A0Port 0: Pkts: 0 =C2=A0 Flows: 0<= br>=C2=A0 =C2=A0 =C2=A0 =C2=A0Port 1: Pkts: 0 =C2=A0 Flows: 0
=C2=A0 =C2= =A0 =C2=A0 =C2=A0Port 2: Pkts: 0 =C2=A0 Flows: 0
=C2=A0 =C2=A0 =C2=A0 = =C2=A0Port 3: Pkts: 0 =C2=A0 Flows: 0
=C2=A0 =C2=A0 =C2=A0-- iqs empty -= -
=C2=A0 =C2=A0Queue 2 (Directed)
=C2=A0 =C2=A0 =C2=A0rx =C2=A0 0 =C2= =A0drop 0 =C2=A0tx =C2=A0 0
=C2=A0 =C2=A0 =C2=A0Per Port Stats:
=C2= =A0 =C2=A0 =C2=A0 =C2=A0Port 0: Pkts: 0 =C2=A0 Flows: 0
=C2=A0 =C2=A0 = =C2=A0 =C2=A0Port 1: Pkts: 0 =C2=A0 Flows: 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0= Port 2: Pkts: 0 =C2=A0 Flows: 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0Port 3: Pkts:= 0 =C2=A0 Flows: 0
=C2=A0 =C2=A0 =C2=A0-- iqs empty --
Regards,
Hariharan


--000000000000428c9905fa64ba0c--