From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0895842A97 for ; Mon, 8 May 2023 16:29:17 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C4E27410ED; Mon, 8 May 2023 16:29:16 +0200 (CEST) Received: from mail-yb1-f174.google.com (mail-yb1-f174.google.com [209.85.219.174]) by mails.dpdk.org (Postfix) with ESMTP id 7F03140685 for ; Mon, 8 May 2023 16:29:15 +0200 (CEST) Received: by mail-yb1-f174.google.com with SMTP id 3f1490d57ef6-b9d9dad9edfso6278710276.2 for ; Mon, 08 May 2023 07:29:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683556154; x=1686148154; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=OqEzCY+TqCf/fLD/P2LD9SjuO8CpnawcNa4Lw4ELIm0=; b=QW4FLG6Pv6yZZ/9tb8KpJkNFOP8SbYGkn0ahd6ycUiZiEQcGY+pIjfujH8dDiK1GIN LjJ853gp4pe6KhD8GRaKuYMHYH+ecUq2l8wtUL2sgzwyXBiRTXfX4zQLrIMxwG1PH/Ur thQTstYuyqJanmNI19IeKNlfo50B5Y2YYtsQS0hKw9hTiwzqGy7EeCGTN866KKYMp3x+ DU0cCYfZXMlYjXo3uadsdf38B3TtOh6kG168F8D3ejdI4E8dqq0OA6N9WvLgj4xc307O KHcK/4hYpBkmfXPBypH7hqvpjj7FqANvlM6xAo1HGtu8Itud0D+2d0Dbv2A8xFf1uwls y6MA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683556154; x=1686148154; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=OqEzCY+TqCf/fLD/P2LD9SjuO8CpnawcNa4Lw4ELIm0=; b=S9C1pV5Al80qeTkcpc6svK9VlO81OoZ3otttoIAxZ30Nx5kROpufPFugE/Q/aQMAkB /GphVcUZWVfpOqSmX1GDaSsz334kaz3agUQKm3E+S+0qbY8Fs9oOeKDUPh8BdqYjdrmf UoGAWbNSqzvVomQuzaD+xhLJ4AJMGQamoPokxHPnauUPw8yGv9pIMvnG3nVrWQPaHf2f pMSEgt5ZP2l5mZ0wnZMXDbDZ2/FYWPtUchzvMbqtc1JciotC4fZqnQvHsOg8tAnOeWFO Abq48XT/xHbV2j0W86se34OxghfRfKJEQblB+wtgjKy4iKJ5qFiPwE9d8TPZC+mv7rH+ +Flg== X-Gm-Message-State: AC+VfDy8TfIZKt60DelVDpKm+djzU+m8lYvG+G8qBhMsj2FJHxS/n3A8 55PnfKFhUmUOlKaSPjSfOIkamVUCmMCyNP0Gt3zhSZKAyDE= X-Google-Smtp-Source: ACHHUZ5HfVu4EWad9V20ke3e4xmRLo8Bf0VrbUnVvPkjN6ccP0WXnLpFh/qOTfxkM9R4zJuKtfNK1VeAL43Hu0+W0Bk= X-Received: by 2002:a25:c89:0:b0:b8f:384b:4f83 with SMTP id 131-20020a250c89000000b00b8f384b4f83mr9982476ybm.33.1683556154552; Mon, 08 May 2023 07:29:14 -0700 (PDT) MIME-Version: 1.0 From: Hari Haran Date: Mon, 8 May 2023 19:59:02 +0530 Message-ID: Subject: EventDev Queue Hang after process restart To: users@dpdk.org Content-Type: multipart/alternative; boundary="0000000000008d35d905fb2f76b0" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --0000000000008d35d905fb2f76b0 Content-Type: text/plain; charset="UTF-8" In Data Application single Process used, not multi process case. During posting packets from one thread to another thread(both threads are in different cores), we have used DPDK Event Dev Queue. After process restart, observed Event Dev Queue Enqueue failure while enqueue the packets rte_event_enqueue_burst() . Cross checked the EvtDevQ stats, it showed packets are in Inflight mode, not posted to Queue it seems and enqueue failed and not recovered afterwards. Also schedule calls happened multiple times. Note: Receiver thread is not in tight loop Event Enqueue function: rte_event_enqueue_burst() - This return enqueue failure Schedule call function: rte_service_run_iter_on_app_lcore() How can I fix it? Any help is appreciated. DevId 0 Tx Port 2 RX port 0 Queue 0 Used in this case. Required Logs information below: *EAL Arguments used:* ArgCnt 0 Value Appuserplane ArgCnt 1 Value -l 93,38,94,39 ArgCnt 2 Value -n 2 ArgCnt 3 Value --main-lcore=93 *EAL prints during Init:* EAL: Detected 112 lcore(s) EAL: Detected 2 NUMA nodes EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: No available hugepages reported in hugepages-2048kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: No legacy callbacks, legacy socket not created *Event Dev Stats:* Dev=0 Port=1EventDev todo-fix-name: ports 4, qids 3 rx 32768 drop 0 tx 4096 sched calls: 628945658 sched cq/qid call: 628964843 sched no IQ enq: 628926401 sched no CQ enq: 628942982 inflight 32768, credits: 0 Port 0 rx 0 drop 0 tx 4096 inflight 4096 Max New: 32768 Avg cycles PP: 0 Credits: 0 Receive burst distribution: 0:100% 1-4:0.00% 5-8:0.00% 9-12:0.00% rx ring used: 0 free: 4096 cq ring used: 0 free: 128 Port 1 rx 0 drop 0 tx 0 inflight 0 Max New: 32768 Avg cycles PP: 0 Credits: 0 Receive burst distribution: 0:100% rx ring used: 0 free: 4096 cq ring used: 0 free: 128 Port 2 rx 32768 drop 0 tx 0 inflight 0 Max New: 32768 Avg cycles PP: 0 Credits: 0 Receive burst distribution: 0:-nan% rx ring used: 0 free: 4096 cq ring used: 0 free: 128 Port 3 (SingleCons) rx 0 drop 0 tx 0 inflight 0 Max New: 32768 Avg cycles PP: 0 Credits: 0 Receive burst distribution: 0:-nan% rx ring used: 0 free: 4096 cq ring used: 0 free: 128 Queue 0 (Atomic) rx 32768 drop 0 tx 4096 Per Port Stats: Port 0: Pkts: 4096 Flows: 1 Port 1: Pkts: 0 Flows: 0 Port 2: Pkts: 0 Flows: 0 Port 3: Pkts: 0 Flows: 0 iq 0: Used 28672 Queue 1 (Atomic) rx 0 drop 0 tx 0 Per Port Stats: Port 0: Pkts: 0 Flows: 0 Port 1: Pkts: 0 Flows: 0 Port 2: Pkts: 0 Flows: 0 Port 3: Pkts: 0 Flows: 0 -- iqs empty -- Queue 2 (Directed) rx 0 drop 0 tx 0 Per Port Stats: Port 0: Pkts: 0 Flows: 0 Port 1: Pkts: 0 Flows: 0 Port 2: Pkts: 0 Flows: 0 Port 3: Pkts: 0 Flows: 0 -- iqs empty -- --0000000000008d35d905fb2f76b0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
In Data Application single Process used, not multi process= case.

During posting packets from one thread to another thread(both= threads are in different cores), we have used DPDK Event Dev Queue.
After process restart, observed Event Dev Queue Enqueue failure while enqu= eue the packets=C2=A0 rte_event_enqueue_burst() . Cross checked the EvtDevQ stats, it showed pack= ets are in Inflight mode, not posted to Queue it seems and enqueue failed a= nd not recovered afterwards.

Also schedule calls happened multiple = times.

Note: Receiver thread is not in tight loop

Event Enque= ue function: rte_event_enqueue_burst() - This return enqueue failure
=C2= =A0
Schedule call function: rte_service_run_iter_on_app_lcore()

H= ow can I fix it? Any help is appreciated.

DevId 0
Tx Port 2
RX port 0=C2=A0
Queue 0 Used in this case.

Required= Logs information below:

EAL Argumen= ts used:

ArgCnt 0 Value Appuserplane
ArgCnt 1 Value -l 93,38,= 94,39
ArgCnt 2 Value -n 2
ArgCnt 3 Value --main-lcore=3D93

= EAL prints during Init:

EAL: Detected 112 lcore(s)
EAL: Detec= ted 2 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-proc= ess socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA&#= 39;
EAL: No available hugepages reported in hugepages-2048kB
EAL: Pro= bing VFIO support...
EAL: VFIO support initialized
EAL: No legacy cal= lbacks, legacy socket not created


Event= Dev Stats:

Dev=3D0 Port=3D1EventDev todo-fix-name: ports 4, qid= s 3
=C2=A0 =C2=A0 =C2=A0rx =C2=A0 32768
=C2=A0 =C2=A0 =C2=A0drop 0=C2=A0 =C2=A0 =C2=A0tx =C2=A0 4096
=C2=A0 =C2=A0 =C2=A0sched calls: 628= 945658
=C2=A0 =C2=A0 =C2=A0sched cq/qid call: 628964843
=C2=A0 =C2=A0= =C2=A0sched no IQ enq: 628926401
=C2=A0 =C2=A0 =C2=A0sched no CQ enq: 6= 28942982
=C2=A0 =C2=A0 =C2=A0inflight 32768, credits: 0
=C2=A0 =C2=A0= Port 0
=C2=A0 =C2=A0 =C2=A0rx =C2=A0 0 =C2=A0drop 0 =C2=A0tx =C2=A0 4096= =C2=A0 =C2=A0inflight 4096
=C2=A0 =C2=A0 =C2=A0Max New: 32768 =C2=A0Avg= cycles PP: 0 =C2=A0 =C2=A0Credits: 0
=C2=A0 =C2=A0 =C2=A0Receive burst = distribution:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00:100% 1-4:0.00% 5-8:0.0= 0% 9-12:0.00%
=C2=A0 =C2=A0 =C2=A0 rx ring used: =C2=A0 =C2=A00 free: 40= 96
=C2=A0 =C2=A0 =C2=A0 cq ring used: =C2=A0 =C2=A00 free: =C2=A0128
= =C2=A0 =C2=A0Port 1
=C2=A0 =C2=A0 =C2=A0rx =C2=A0 0 =C2=A0drop 0 =C2=A0t= x =C2=A0 0 =C2=A0 inflight 0
=C2=A0 =C2=A0 =C2=A0Max New: 32768 =C2=A0Av= g cycles PP: 0 =C2=A0 =C2=A0Credits: 0
=C2=A0 =C2=A0 =C2=A0Receive burst= distribution:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00:100%
=C2=A0 =C2=A0= =C2=A0 rx ring used: =C2=A0 =C2=A00 free: 4096
=C2=A0 =C2=A0 =C2=A0 cq = ring used: =C2=A0 =C2=A00 free: =C2=A0128
=C2=A0 =C2=A0Port 2
=C2=A0 = =C2=A0 =C2=A0rx =C2=A0 32768 =C2=A0drop 0 =C2=A0tx =C2=A0 0 =C2=A0 inflight= 0
=C2=A0 =C2=A0 =C2=A0Max New: 32768 =C2=A0Avg cycles PP: 0 =C2=A0 =C2= =A0Credits: 0
=C2=A0 =C2=A0 =C2=A0Receive burst distribution:
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A00:-nan%
=C2=A0 =C2=A0 =C2=A0 rx ring used: = =C2=A0 =C2=A00 free: 4096
=C2=A0 =C2=A0 =C2=A0 cq ring used: =C2=A0 =C2= =A00 free: =C2=A0128
=C2=A0 =C2=A0Port 3 =C2=A0(SingleCons)
=C2=A0 = =C2=A0 =C2=A0rx =C2=A0 0 =C2=A0drop 0 =C2=A0tx =C2=A0 0 =C2=A0 inflight 0=C2=A0 =C2=A0 =C2=A0Max New: 32768 =C2=A0Avg cycles PP: 0 =C2=A0 =C2=A0Cr= edits: 0
=C2=A0 =C2=A0 =C2=A0Receive burst distribution:
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A00:-nan%
=C2=A0 =C2=A0 =C2=A0 rx ring used: =C2= =A0 =C2=A00 free: 4096
=C2=A0 =C2=A0 =C2=A0 cq ring used: =C2=A0 =C2=A00= free: =C2=A0128
=C2=A0 =C2=A0Queue 0 (Atomic)
=C2=A0 =C2=A0 =C2=A0rx= =C2=A0 32768 =C2=A0drop 0 =C2=A0tx =C2=A0 4096
=C2=A0 =C2=A0 =C2=A0Per = Port Stats:
=C2=A0 =C2=A0 =C2=A0 =C2=A0Port 0: Pkts: 4096 =C2=A0 =C2=A0F= lows: 1
=C2=A0 =C2=A0 =C2=A0 =C2=A0Port 1: Pkts: 0 =C2=A0 Flows: 0
= =C2=A0 =C2=A0 =C2=A0 =C2=A0Port 2: Pkts: 0 =C2=A0 Flows: 0
=C2=A0 =C2=A0= =C2=A0 =C2=A0Port 3: Pkts: 0 =C2=A0 Flows: 0
=C2=A0 =C2=A0 =C2=A0 iq 0:= Used 28672
=C2=A0 =C2=A0Queue 1 (Atomic)
=C2=A0 =C2=A0 =C2=A0rx =C2= =A0 0 =C2=A0drop 0 =C2=A0tx =C2=A0 0
=C2=A0 =C2=A0 =C2=A0Per Port Stats:=
=C2=A0 =C2=A0 =C2=A0 =C2=A0Port 0: Pkts: 0 =C2=A0 Flows: 0
=C2=A0 = =C2=A0 =C2=A0 =C2=A0Port 1: Pkts: 0 =C2=A0 Flows: 0
=C2=A0 =C2=A0 =C2=A0= =C2=A0Port 2: Pkts: 0 =C2=A0 Flows: 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0Port 3= : Pkts: 0 =C2=A0 Flows: 0
=C2=A0 =C2=A0 =C2=A0-- iqs empty --
=C2=A0 = =C2=A0Queue 2 (Directed)
=C2=A0 =C2=A0 =C2=A0rx =C2=A0 0 =C2=A0drop 0 = =C2=A0tx =C2=A0 0
=C2=A0 =C2=A0 =C2=A0Per Port Stats:
=C2=A0 =C2=A0 = =C2=A0 =C2=A0Port 0: Pkts: 0 =C2=A0 Flows: 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0= Port 1: Pkts: 0 =C2=A0 Flows: 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0Port 2: Pkts:= 0 =C2=A0 Flows: 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0Port 3: Pkts: 0 =C2=A0 Flo= ws: 0
=C2=A0 =C2=A0 =C2=A0-- iqs empty --
--0000000000008d35d905fb2f76b0--