On DPDK head commit (917229) I have a simple program creating two TXQs each
sending 15000 IPV4 UDP packets.

As far as I can see there are no errors anymore. I have looked. I burst TX
them in blocks of 15; rte_eth_tx_burst always returns 15.

The lcores are launched with rte_eal_mp_remote_launch, and main() blocks
until rte_eal_mp_wait_lcore() returns.

However, at any time after rte_eal_mp_wait_lcore returns showing the number
of out packets as per rte_eth_stats_get or xstats per rte_eth_xstats_get I
see two things:

- the reported number of queued packets per TXQ is always 15000
- the number of out packets or "good packets" is close to 15000*2 but never
30000

It appears like the TXQs still have work in-queue even after the lcore's
thread returns and rte_eal_mp_wait_lcore returns. I've tried to
rte_eth_tx_done_cleanup and/or rte_eth_dev_tx_queue_stop before I get the
stats. That doesn't seem to help.

The TXQ config is default except for checksum offloads:

000000.336378246 DEBUG reinvent_dpdk_initaws.cpp:1341 TXQ conf:
{"tx_thresh": { "pthresh": 0, "hthresh": 0, "wthresh": 0 }, "tx_rs_thresh":
0, "tx_free_thresh": 0, "tx_deferred_start": 0, "tx_offloads": 00008006, }

Is this expected behavior? I've look through code; I haven't seen clear
signs of a TXQ oriented lcore flushed or waiting for the output queue to be
actually written onto the wire before exiting.

lcoreId: 01, txqIndex: 1, packetsQueued: 15000, sleeps: 0,
totalSleepTimeNs: 0

lcoreId: 00, txqIndex: 0, packetsQueued: 15000, sleeps: 0,
totalSleepTimeNs: 0

000000.351345192 INFO  reinvent_dpdk_udp_integration_test.cpp:585 waiting
for DPDK worker threads to stop

in  packets : 0

out packets : 29000

in  bytes   : 0

out bytes   : 1566000

missed pkts : 0

in err pkts : 0

out err pkts: 0

rx allc errs: 0

rx_good_packets                 : 0

tx_good_packets                 : 29952

rx_good_bytes                   : 0

tx_good_bytes                   : 1617408

rx_missed_errors                : 0

rx_errors                       : 0

tx_errors                       : 0

rx_mbuf_allocation_errors       : 0

tx_q0_packets                   : 15000

tx_q0_bytes                     : 810000

tx_q1_packets                   : 15000

tx_q1_bytes                     : 810000

wd_expired                      : 0

dev_start                       : 1

dev_stop                        : 0

tx_drops                        : 0

bw_in_allowance_exceeded        : 0

bw_out_allowance_exceeded       : 0

pps_allowance_exceeded          : 0

conntrack_allowance_exceeded    : 0

linklocal_allowance_exceeded    : 0

tx_q0_cnt                       : 15000

tx_q1_cnt                       : 15000

tx_q0_bytes                     : 810000

tx_q1_bytes                     : 810000

tx_q0_prepare_ctx_err           : 0

tx_q1_prepare_ctx_err           : 0

tx_q0_tx_poll                   : 54669

tx_q1_tx_poll                   : 54316

tx_q0_doorbells                 : 2311

tx_q1_doorbells                 : 2321

tx_q0_bad_req_id                : 0

tx_q1_bad_req_id                : 0

tx_q0_available_desc            : 15

tx_q1_available_desc            : 7

tx_q0_missed_tx                 : 0

tx_q1_missed_tx                 : 0

ena_tx_queue_release(): Tx queue 0:0 released

ena_tx_queue_release(): Tx queue 0:1 released