* (AWS ENA NIC) queued v. out packets reported by any of the DPDK stats functions
@ 2022-03-15 19:28 fwefew 4t4tg
2022-03-15 20:15 ` Stephen Hemminger
0 siblings, 1 reply; 2+ messages in thread
From: fwefew 4t4tg @ 2022-03-15 19:28 UTC (permalink / raw)
To: users
[-- Attachment #1: Type: text/plain, Size: 3322 bytes --]
On DPDK head commit (917229) I have a simple program creating two TXQs each
sending 15000 IPV4 UDP packets.
As far as I can see there are no errors anymore. I have looked. I burst TX
them in blocks of 15; rte_eth_tx_burst always returns 15.
The lcores are launched with rte_eal_mp_remote_launch, and main() blocks
until rte_eal_mp_wait_lcore() returns.
However, at any time after rte_eal_mp_wait_lcore returns showing the number
of out packets as per rte_eth_stats_get or xstats per rte_eth_xstats_get I
see two things:
- the reported number of queued packets per TXQ is always 15000
- the number of out packets or "good packets" is close to 15000*2 but never
30000
It appears like the TXQs still have work in-queue even after the lcore's
thread returns and rte_eal_mp_wait_lcore returns. I've tried to
rte_eth_tx_done_cleanup and/or rte_eth_dev_tx_queue_stop before I get the
stats. That doesn't seem to help.
The TXQ config is default except for checksum offloads:
000000.336378246 DEBUG reinvent_dpdk_initaws.cpp:1341 TXQ conf:
{"tx_thresh": { "pthresh": 0, "hthresh": 0, "wthresh": 0 }, "tx_rs_thresh":
0, "tx_free_thresh": 0, "tx_deferred_start": 0, "tx_offloads": 00008006, }
Is this expected behavior? I've look through code; I haven't seen clear
signs of a TXQ oriented lcore flushed or waiting for the output queue to be
actually written onto the wire before exiting.
lcoreId: 01, txqIndex: 1, packetsQueued: 15000, sleeps: 0,
totalSleepTimeNs: 0
lcoreId: 00, txqIndex: 0, packetsQueued: 15000, sleeps: 0,
totalSleepTimeNs: 0
000000.351345192 INFO reinvent_dpdk_udp_integration_test.cpp:585 waiting
for DPDK worker threads to stop
in packets : 0
out packets : 29000
in bytes : 0
out bytes : 1566000
missed pkts : 0
in err pkts : 0
out err pkts: 0
rx allc errs: 0
rx_good_packets : 0
tx_good_packets : 29952
rx_good_bytes : 0
tx_good_bytes : 1617408
rx_missed_errors : 0
rx_errors : 0
tx_errors : 0
rx_mbuf_allocation_errors : 0
tx_q0_packets : 15000
tx_q0_bytes : 810000
tx_q1_packets : 15000
tx_q1_bytes : 810000
wd_expired : 0
dev_start : 1
dev_stop : 0
tx_drops : 0
bw_in_allowance_exceeded : 0
bw_out_allowance_exceeded : 0
pps_allowance_exceeded : 0
conntrack_allowance_exceeded : 0
linklocal_allowance_exceeded : 0
tx_q0_cnt : 15000
tx_q1_cnt : 15000
tx_q0_bytes : 810000
tx_q1_bytes : 810000
tx_q0_prepare_ctx_err : 0
tx_q1_prepare_ctx_err : 0
tx_q0_tx_poll : 54669
tx_q1_tx_poll : 54316
tx_q0_doorbells : 2311
tx_q1_doorbells : 2321
tx_q0_bad_req_id : 0
tx_q1_bad_req_id : 0
tx_q0_available_desc : 15
tx_q1_available_desc : 7
tx_q0_missed_tx : 0
tx_q1_missed_tx : 0
ena_tx_queue_release(): Tx queue 0:0 released
ena_tx_queue_release(): Tx queue 0:1 released
[-- Attachment #2: Type: text/html, Size: 21143 bytes --]
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: (AWS ENA NIC) queued v. out packets reported by any of the DPDK stats functions
2022-03-15 19:28 (AWS ENA NIC) queued v. out packets reported by any of the DPDK stats functions fwefew 4t4tg
@ 2022-03-15 20:15 ` Stephen Hemminger
0 siblings, 0 replies; 2+ messages in thread
From: Stephen Hemminger @ 2022-03-15 20:15 UTC (permalink / raw)
To: fwefew 4t4tg; +Cc: users
On Tue, 15 Mar 2022 15:28:55 -0400
fwefew 4t4tg <7532yahoo@gmail.com> wrote:
> On DPDK head commit (917229) I have a simple program creating two TXQs each
> sending 15000 IPV4 UDP packets.
>
> As far as I can see there are no errors anymore. I have looked. I burst TX
> them in blocks of 15; rte_eth_tx_burst always returns 15.
>
> The lcores are launched with rte_eal_mp_remote_launch, and main() blocks
> until rte_eal_mp_wait_lcore() returns.
>
> However, at any time after rte_eal_mp_wait_lcore returns showing the number
> of out packets as per rte_eth_stats_get or xstats per rte_eth_xstats_get I
> see two things:
>
> - the reported number of queued packets per TXQ is always 15000
> - the number of out packets or "good packets" is close to 15000*2 but never
> 30000
>
> It appears like the TXQs still have work in-queue even after the lcore's
> thread returns and rte_eal_mp_wait_lcore returns. I've tried to
> rte_eth_tx_done_cleanup and/or rte_eth_dev_tx_queue_stop before I get the
> stats. That doesn't seem to help.
>
> The TXQ config is default except for checksum offloads:
>
> 000000.336378246 DEBUG reinvent_dpdk_initaws.cpp:1341 TXQ conf:
> {"tx_thresh": { "pthresh": 0, "hthresh": 0, "wthresh": 0 }, "tx_rs_thresh":
> 0, "tx_free_thresh": 0, "tx_deferred_start": 0, "tx_offloads": 00008006, }
>
> Is this expected behavior? I've look through code; I haven't seen clear
> signs of a TXQ oriented lcore flushed or waiting for the output queue to be
> actually written onto the wire before exiting.
>
No part of lcore_wait is related directly to TxQ.
The TxQ runs asynchronously. If you have to check if packet was sent, the
you would need to use rte_eth_tx_descriptor_status().
But not all hardware supports this API; AWS ENA does not.
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2022-03-15 20:15 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-03-15 19:28 (AWS ENA NIC) queued v. out packets reported by any of the DPDK stats functions fwefew 4t4tg
2022-03-15 20:15 ` Stephen Hemminger
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).