From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 786D3A0032 for ; Tue, 15 Mar 2022 20:29:09 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 01BFC40395; Tue, 15 Mar 2022 20:29:09 +0100 (CET) Received: from mail-vk1-f169.google.com (mail-vk1-f169.google.com [209.85.221.169]) by mails.dpdk.org (Postfix) with ESMTP id D4F874014F for ; Tue, 15 Mar 2022 20:29:07 +0100 (CET) Received: by mail-vk1-f169.google.com with SMTP id m84so136726vke.1 for ; Tue, 15 Mar 2022 12:29:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=cr2tvs7+79cyLYT8k6Ppf6nr+L3Z/duv/LrxGPuluo4=; b=o+CtBTZZKXEBsLHnYvmGvvV1+Kp6fro2jrhNtPeX10sRpJX+dgH0PGL+xVkeXO9wzU /QAlJfGGlOLZVqs9C4YHpx0Ei2MQ7RgwOMYcKQnj422sOYTQyhT6Y7J4TvsH11uXfWpL TD0DnPBdnjgSDcdJZSvA0YvCO7jbzEW558ZPNHCMwdyrrP48VHbcYobUBwp765p/X3dt WQxPPdarrw9iHcFgvuYX/uS7lzecOkeww8vVDIDlB1Q/1RAP5qa0R1R4AkOpvxkwO8FB K/0/gZhQzwr9xQwaG4sY3H3NwHn9fn8ovKRx685dNM0ZQfxEIdt3DRUFuYGWDanE9zge x6+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=cr2tvs7+79cyLYT8k6Ppf6nr+L3Z/duv/LrxGPuluo4=; b=LQeYLDd3Pm9O3/YSshx/IbbCmMwqmrlsO/7x5Gg4oYkBrdkRtVyIxGJ7zboiGU/ZOA YUBXBOYxaLZy0U5lOXB3blAMUTzpgVPbzxAyePiASjWVhv1qLi90NH9vaFCdUZ3sc/NZ r3bDgmGtii8ui8ajlKfZrE0yWJowxkdDv7wKdGEJx5CtOCaWYxUHO9oZx1BG0Eo8aAsA r2Q2+8nt/Ntrz9CxmC/+mmKSxLL/xTf1ohA3CYqTOHvfk+hv9RvSib3xS0Jiaw5jk4So gWEFMkouwk4gm8aGse66TJfY9O3ROtariM24/APvPo3LObwzb2Zo+wweK3H7h45r2lj1 hVXQ== X-Gm-Message-State: AOAM533nExCf0fXQOnGhhGJxYwaz+/u280c+lpw4KDReuydTwK0VoqXL 9+K/yRhJVbA/naLOfm5rR4SMVx4TRgw0RYZmZvgY44olEDk= X-Google-Smtp-Source: ABdhPJyblp1tDGuJGQRCzzEGh8kHHjkXf2GK36eTlpmVhc832gJ97i0iSHFTpT7KJY2s6af99ma2PGuWRjZd1wi6Ce4= X-Received: by 2002:a1f:9895:0:b0:331:7132:72cf with SMTP id a143-20020a1f9895000000b00331713272cfmr12377617vke.7.1647372546847; Tue, 15 Mar 2022 12:29:06 -0700 (PDT) MIME-Version: 1.0 From: fwefew 4t4tg <7532yahoo@gmail.com> Date: Tue, 15 Mar 2022 15:28:55 -0400 Message-ID: Subject: (AWS ENA NIC) queued v. out packets reported by any of the DPDK stats functions To: users@dpdk.org Content-Type: multipart/alternative; boundary="00000000000077b99a05da46cf6b" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --00000000000077b99a05da46cf6b Content-Type: text/plain; charset="UTF-8" On DPDK head commit (917229) I have a simple program creating two TXQs each sending 15000 IPV4 UDP packets. As far as I can see there are no errors anymore. I have looked. I burst TX them in blocks of 15; rte_eth_tx_burst always returns 15. The lcores are launched with rte_eal_mp_remote_launch, and main() blocks until rte_eal_mp_wait_lcore() returns. However, at any time after rte_eal_mp_wait_lcore returns showing the number of out packets as per rte_eth_stats_get or xstats per rte_eth_xstats_get I see two things: - the reported number of queued packets per TXQ is always 15000 - the number of out packets or "good packets" is close to 15000*2 but never 30000 It appears like the TXQs still have work in-queue even after the lcore's thread returns and rte_eal_mp_wait_lcore returns. I've tried to rte_eth_tx_done_cleanup and/or rte_eth_dev_tx_queue_stop before I get the stats. That doesn't seem to help. The TXQ config is default except for checksum offloads: 000000.336378246 DEBUG reinvent_dpdk_initaws.cpp:1341 TXQ conf: {"tx_thresh": { "pthresh": 0, "hthresh": 0, "wthresh": 0 }, "tx_rs_thresh": 0, "tx_free_thresh": 0, "tx_deferred_start": 0, "tx_offloads": 00008006, } Is this expected behavior? I've look through code; I haven't seen clear signs of a TXQ oriented lcore flushed or waiting for the output queue to be actually written onto the wire before exiting. lcoreId: 01, txqIndex: 1, packetsQueued: 15000, sleeps: 0, totalSleepTimeNs: 0 lcoreId: 00, txqIndex: 0, packetsQueued: 15000, sleeps: 0, totalSleepTimeNs: 0 000000.351345192 INFO reinvent_dpdk_udp_integration_test.cpp:585 waiting for DPDK worker threads to stop in packets : 0 out packets : 29000 in bytes : 0 out bytes : 1566000 missed pkts : 0 in err pkts : 0 out err pkts: 0 rx allc errs: 0 rx_good_packets : 0 tx_good_packets : 29952 rx_good_bytes : 0 tx_good_bytes : 1617408 rx_missed_errors : 0 rx_errors : 0 tx_errors : 0 rx_mbuf_allocation_errors : 0 tx_q0_packets : 15000 tx_q0_bytes : 810000 tx_q1_packets : 15000 tx_q1_bytes : 810000 wd_expired : 0 dev_start : 1 dev_stop : 0 tx_drops : 0 bw_in_allowance_exceeded : 0 bw_out_allowance_exceeded : 0 pps_allowance_exceeded : 0 conntrack_allowance_exceeded : 0 linklocal_allowance_exceeded : 0 tx_q0_cnt : 15000 tx_q1_cnt : 15000 tx_q0_bytes : 810000 tx_q1_bytes : 810000 tx_q0_prepare_ctx_err : 0 tx_q1_prepare_ctx_err : 0 tx_q0_tx_poll : 54669 tx_q1_tx_poll : 54316 tx_q0_doorbells : 2311 tx_q1_doorbells : 2321 tx_q0_bad_req_id : 0 tx_q1_bad_req_id : 0 tx_q0_available_desc : 15 tx_q1_available_desc : 7 tx_q0_missed_tx : 0 tx_q1_missed_tx : 0 ena_tx_queue_release(): Tx queue 0:0 released ena_tx_queue_release(): Tx queue 0:1 released --00000000000077b99a05da46cf6b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On DPDK head commit (917229) I have a simple program creat= ing two TXQs each sending 15000 IPV4 UDP packets.=C2=A0

= As far as I can see there are no errors anymore. I have looked. I burst TX = them in blocks of 15; rte_eth_tx_burst always returns 15.

The lcores= are launched with rte_eal_mp_remote_launch, and main() blocks until=C2=A0r= te_eal_mp_wait_lcore() returns.=C2=A0

However, at any time after rte= _eal_mp_wait_lcore returns showing the number of out packets as per=C2=A0rt= e_eth_stats_get or xstats per=C2=A0rte_eth_xstats_get I see two things:
=
- the reported number of queued packets per TXQ is always 15000
- the number of out packets or "good packets" is close to 1500= 0*2 but never 30000

It appears like the TXQs still= have work in-queue even after the lcore's thread returns and rte_eal_m= p_wait_lcore returns. I've tried to rte_eth_tx_done_cleanup and/or rte_= eth_dev_tx_queue_stop before I get the stats. That doesn't seem to help= .

The TXQ config is default except for checksum=C2= =A0offloads:

000000.336378246 DEBUG reinvent_dpdk_initaws.cpp:1341 T= XQ conf: {"tx_thresh": { "pthresh": 0, "hthresh&qu= ot;: 0, "wthresh": 0 }, "tx_rs_thresh": 0, "tx_fre= e_thresh": 0, "tx_deferred_start": 0, "tx_offloads"= ;: 00008006, }

Is this expected behavior? I've look t= hrough code; I haven't seen clear signs of a TXQ oriented lcore flushed= or waiting for the output queue to be actually written onto the wire befor= e exiting.

= lcoreId: 01, txqIndex: 1, packetsQueued: 15000, sleeps: 0, totalSleepTimeNs= : 0

= lcoreId: 00, txqIndex: 0, packetsQueued: 15000, sleeps: 0, totalSleepTimeNs= : 0

= 000000.351345192 INFO=C2=A0 reinvent_dpdk_udp_integration_test.cpp:585 waiting for DPDK worker thre= ads to stop

= in=C2=A0 packets : 0

= out packets : 29000

= in=C2=A0 bytes =C2=A0 : 0

= out bytes =C2=A0 : 15660= 00

= missed pkts : 0

= in err pkts : 0

= out err pkts: 0

= rx allc errs: 0

= rx_good_packets =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 0

= tx_good_packets =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 29952

= rx_good_bytes =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 0

= tx_good_bytes =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 1617408

= rx_missed_errors=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 0

= rx_errors =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 0

= tx_errors =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 0

= rx_mbuf_allocation_errors =C2= =A0 =C2=A0 =C2=A0 : 0

= tx_q0_packets =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 15000

= tx_q0_bytes =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 810000=

= tx_q1_packets =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 15000

= tx_q1_bytes =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 810000=

= wd_expired=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 0

= dev_start =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 1

= dev_stop=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 0

= tx_drops=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 0

= bw_in_allowance_exceeded=C2=A0 = =C2=A0 =C2=A0 =C2=A0 : 0

= bw_out_allowance_exceeded =C2= =A0 =C2=A0 =C2=A0 : 0

= pps_allowance_exceeded=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 0

= conntrack_allowance_exceeded=C2= =A0 =C2=A0 : 0

= linklocal_allowance_exceeded=C2= =A0 =C2=A0 : 0

= tx_q0_cnt =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 15000

= tx_q1_cnt =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 15000

= tx_q0_bytes =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 810000=

= tx_q1_bytes =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 810000=

= tx_q0_prepare_ctx_err =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 0

= tx_q1_prepare_ctx_err =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 0

= tx_q0_tx_poll =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 54669

= tx_q1_tx_poll =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 54316

= tx_q0_doorbells =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 2311

= tx_q1_doorbells =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 2321

= tx_q0_bad_req_id=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 0

= tx_q1_bad_req_id=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 0

= tx_q0_available_desc=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 15

= tx_q1_available_desc=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 7

= tx_q0_missed_tx =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 0

= tx_q1_missed_tx =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 0

= ena_tx_queue_release(): Tx queue 0:0 released

= ena_tx_queue_release(): Tx queue 0:1 released

--00000000000077b99a05da46cf6b--