rte_eth_tx_burst return 0 after some time

rte_eth_tx_burst return 0 after some time

[Alan Arondel]

Hello Everyone,

I'm trying to add an export module to my application. I use DPDK
version 20.11, a 82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb
and the vfio-pci driver.
port configuration is the following:

port_conf->txmode.mq_mode = ETH_MQ_TX_NONE;
port_conf->txmode.offloads = DEV_TX_OFFLOAD_MULTI_SEGS;

I use only 1 queue on the card and the configuration is the following :
    nb_desc : 4096
    rs_threshold : 32
    free_threshold : 32

I have an issue where at some point during the lifetime of the
application, the rte_eth_tx_burst return only 0 and nothing can be
done once it is stuck after several seconds of the beginning of
transmission.

I have checked couple of things :
- i still get burst of packets to send
- i still have lot of mbuf in my mempool
- i use t-rex to generate traffic to forward and one of them have lot
of small packet and i can support up to 2,8Mpps (roughly 7,8Gbps)
sadly with another traffic i can't go up more than 200Kpps at best
(roughly 2Gbps if not less)

once the application is stuck, I tried to stop the traffic and restart
it with a very low bandwidth like 40Mbps but it changes nothing

here is a code snippet of what i'm trying to do :

nb_rx = rte_ring_sc_dequeue_burst(od_h->rx_rings[i], (void **) pkts_burst,

OUTPUT_DPDK_BURST_SIZE, NULL);

for (j = 0; j < nb_rx; j++) {

  if (!rte_pktmbuf_is_contiguous(pkts_burst[j])) {
      stats->jumbo_frame ++; /* i got some jumbo frame in both traffic
which i want to track, mbuf size is 1512 + RTE_PKTMBUF_HEADROOM */
   }

   ret = rte_eth_tx_burst(port_id, 0, &pkts_burst[j], 1); /* i know
this is weird but i have another issue with the order of the packets
when sending more than 1 packet even if the mbuf inside pkts_burst are
ordered correctly */
        if (unlikely(! ret)) {
              /* packet not sent, just drop it */
              rte_pktmbuf_free(pkts_burst[j]);
              stats->freed_mbuf ++;

               int flushed = rte_eth_tx_done_cleanup(port_id, 0, 0);
/* this is my last test since i saw we could flush the mbuf stuck in
the cache of the driver */
               if (flushed >= 0) {
                    stats->mbuf_flushed += flushed; /* i can see some
of them but not moving once the application do not send anymore
packets */
                } else {
                   stats->mbuf_flushed = flushed;
                }
          } else {
                stats->nb_sent ++;
                 sent = 1;
          }
     }
  i ++;
}

As far as I understand, the logic of the code seems to be ok since it
works on another type of traffic.
However i have no clue why it is stuck at some point. If you got some
insight to dig further on this i'm all ears.
Also, I would like to understand the free_threshold. As far as i
understand, mbuf will "stay" in the driver into there is not enough
descriptor available regarding the threshold hence the mbuf return to
the mempool. Is that right ?

Thank you for any help i will have on this.

Regards,
Alan.

Re: rte_eth_tx_burst return 0 after some time

[Stephen Hemminger]

On Mon, 9 Sep 2024 15:25:23 +0200
Alan Arondel <arondel.alan@gmail.com> wrote:

> Hello Everyone,
> 
> I'm trying to add an export module to my application. I use DPDK
> version 20.11, a 82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb
> and the vfio-pci driver.
> port configuration is the following:
> 
> port_conf->txmode.mq_mode = ETH_MQ_TX_NONE;
> port_conf->txmode.offloads = DEV_TX_OFFLOAD_MULTI_SEGS;
> 
> I use only 1 queue on the card and the configuration is the following :
>     nb_desc : 4096
>     rs_threshold : 32
>     free_threshold : 32
> 
> I have an issue where at some point during the lifetime of the
> application, the rte_eth_tx_burst return only 0 and nothing can be
> done once it is stuck after several seconds of the beginning of
> transmission.

Looks like the driver you are passing packets is unable to send
packets. Most drivers have an internal transmit ring, so either
the hardware is not consuming the packets from the ring or the
packets have been sent but the transmit ring cleanup logic has
not been called.

In most cases, problems are in your application. Best to start
with making sure simple applications like testpmd and l3fwd
work with similar configurations.