It looks like this was caused by not having enough TX descriptors, we were only specifying 64, increasing to 1024 fixes the problem.

From: Birtles, Alan
Sent: 08 February 2022 09:34
To: users@dpdk.org
Subject: Connectx-6 LX scheduled sending only sending 25 packets

We are trying to use send scheduling on a Connectx-6 LX. If we set no timestamps on the packet buffers and manually send each packet at approximately the right time everything works. However if we set timestamps in the buffers then the first 25 packets are sent and the packets are received at the expected times but all subsequent calls to rte_eth_tx_burst return 0. If its relevant we are sending a single packet in each burst with timestamps 125 us apart.
We've tried setting the timestamps to low values and the packets are transmitted correctly and as expected the tx_pp_timestamp_past_errors value is incremented. We also set high values and this worked too with tx_pp_timestamp_future_errors incrementing.
Any ideas where to start debugging this? I couldn't see any API which would give an error code for rte_eth_tx_burst failing.
We're using DPDK 21.08 with the 5.4.3.1 mellanox driver on ubuntu 20.04.