rte_eth_tx_burst() always returns 0 in tight loop

rte_eth_tx_burst() always returns 0 in tight loop

[Antonio Di Bacco]

I'm trying to send packets continuously in a  tight loop with a burst
size of 8 and packets are 9600 bytes long.
If I don't insert a delay after the rte_eth_tx_burst it always returns 0.

What's the explanation of this behaviour ?

Best regards,
Antonio.

Re: rte_eth_tx_burst() always returns 0 in tight loop

[Gábor LENCSE]

Dear Antonio,

According to my experience, the rte_eth_tx_burst() function reports the 
packets as "sent" (by a non-zero return value), when they are still in 
the transmit buffer.

(If you are interested in the details, you can see them in Section 3.6.5 
of this paper: http://www.hit.bme.hu/~lencse/publications/e104-b_2_128.pdf )

Therefore, I think that the return value of 0 may mean that 
rte_eth_tx_burst() can't even commit itself for the future delivery of 
the packets. I could only guess why. E.g. all its resources have been 
exhausted.

Best regards,

Gábor


7/3/2022 5:57 PM keltezéssel, Antonio Di Bacco írta:
> I'm trying to send packets continuously in a  tight loop with a burst
> size of 8 and packets are 9600 bytes long.
> If I don't insert a delay after the rte_eth_tx_burst it always returns 0.
>
> What's the explanation of this behaviour ?
>
> Best regards,
> Antonio.

Re: rte_eth_tx_burst() always returns 0 in tight loop

[Antonio Di Bacco]

I wonder why calling eth_dev_tx_burst in a tight loop doesn't allow to
write the packets into the transmit buffer. Only solution I found is
to include a small delay after the tx_burst that is less than the
estimated serialization time of the packet in order to be able to
saturate the ethernet line.

Anyway I wonder if this is the right approach.

Thx,
Antonio.

On Sun, Jul 3, 2022 at 10:19 PM Gábor LENCSE <lencse@hit.bme.hu> wrote:
>
> Dear Antonio,
>
> According to my experience, the rte_eth_tx_burst() function reports the
> packets as "sent" (by a non-zero return value), when they are still in
> the transmit buffer.
>
> (If you are interested in the details, you can see them in Section 3.6.5
> of this paper: http://www.hit.bme.hu/~lencse/publications/e104-b_2_128.pdf )
>
> Therefore, I think that the return value of 0 may mean that
> rte_eth_tx_burst() can't even commit itself for the future delivery of
> the packets. I could only guess why. E.g. all its resources have been
> exhausted.
>
> Best regards,
>
> Gábor
>
>
> 7/3/2022 5:57 PM keltezéssel, Antonio Di Bacco írta:
> > I'm trying to send packets continuously in a  tight loop with a burst
> > size of 8 and packets are 9600 bytes long.
> > If I don't insert a delay after the rte_eth_tx_burst it always returns 0.
> >
> > What's the explanation of this behaviour ?
> >
> > Best regards,
> > Antonio.
>

Re: rte_eth_tx_burst() always returns 0 in tight loop

[Stephen Hemminger]

On Wed, 6 Jul 2022 09:21:28 +0200
Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:

> I wonder why calling eth_dev_tx_burst in a tight loop doesn't allow to
> write the packets into the transmit buffer. Only solution I found is
> to include a small delay after the tx_burst that is less than the
> estimated serialization time of the packet in order to be able to
> saturate the ethernet line.
> 
> Anyway I wonder if this is the right approach.
> 
> Thx,
> Antonio.
> 
> On Sun, Jul 3, 2022 at 10:19 PM Gábor LENCSE <lencse@hit.bme.hu> wrote:
> >
> > Dear Antonio,
> >
> > According to my experience, the rte_eth_tx_burst() function reports the
> > packets as "sent" (by a non-zero return value), when they are still in
> > the transmit buffer.
> >
> > (If you are interested in the details, you can see them in Section 3.6.5
> > of this paper: http://www.hit.bme.hu/~lencse/publications/e104-b_2_128.pdf )
> >
> > Therefore, I think that the return value of 0 may mean that
> > rte_eth_tx_burst() can't even commit itself for the future delivery of
> > the packets. I could only guess why. E.g. all its resources have been
> > exhausted.
> >
> > Best regards,
> >
> > Gábor
> >
> >
> > 7/3/2022 5:57 PM keltezéssel, Antonio Di Bacco írta:  
> > > I'm trying to send packets continuously in a  tight loop with a burst
> > > size of 8 and packets are 9600 bytes long.
> > > If I don't insert a delay after the rte_eth_tx_burst it always returns 0.
> > >
> > > What's the explanation of this behaviour ?
> > >
> > > Best regards,
> > > Antonio.  
> >  

Which driver? How did you set the tx_free threshold.
The driver will need to cleanup already transmitted packets.

Re: rte_eth_tx_burst() always returns 0 in tight loop

[Antonio Di Bacco]

I have an E810-C card with  iavf-4.4.2.1  ice-1.8.8 drivers.

My tx_free_threshold is 8 together with tx_rs_thresh.

I have a tight loop sending BURSTs of 8 packets, each packet is 9014
bytes long (8 packets take 6 usecs to be serialized).
If I put the rte_delay_us_block to 7 then everything works fine, every
cycle 8 packets are transmitted.
If I lower  the rte_delay_us_block to 1 usec, then I observe that the
first FOR cycle is ok, nb_tx is 8 as expected, then, the second cycle
prints a 7 while all subsequent cycles print Z (zero packets sent). I
know that 1 usec delay is too small and I expect that no packets are
transmitted for some cycles but I don't understand why I get an nb_tx
set to 0 forever after the first two cycles.

for (;;)
{
        rte_spinlock_lock(&spinlock_conf[src_port]) ;
        const uint16_t nb_tx = rte_eth_tx_burst(src_port, 0, tx_bufs,
BURST_SIZE);
        rte_spinlock_unlock(&spinlock_conf[src_port]);

        rte_delay_us_block(7);  // tested with 1

        if (nb_tx == 0)
            printf("Z");
        else if (nb_tx < BURST_SIZE)
            printf("nb_tx %d\n", nb_tx);

        tx += nb_tx;

        if (unlikely(nb_tx < BURST_SIZE)) {
            uint16_t buf;

            for (buf = nb_tx; buf < BURST_SIZE; buf++)
                rte_pktmbuf_free(tx_bufs[buf]);
        }
}

On Wed, Jul 6, 2022 at 5:00 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Wed, 6 Jul 2022 09:21:28 +0200
> Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:
>
> > I wonder why calling eth_dev_tx_burst in a tight loop doesn't allow to
> > write the packets into the transmit buffer. Only solution I found is
> > to include a small delay after the tx_burst that is less than the
> > estimated serialization time of the packet in order to be able to
> > saturate the ethernet line.
> >
> > Anyway I wonder if this is the right approach.
> >
> > Thx,
> > Antonio.
> >
> > On Sun, Jul 3, 2022 at 10:19 PM Gábor LENCSE <lencse@hit.bme.hu> wrote:
> > >
> > > Dear Antonio,
> > >
> > > According to my experience, the rte_eth_tx_burst() function reports the
> > > packets as "sent" (by a non-zero return value), when they are still in
> > > the transmit buffer.
> > >
> > > (If you are interested in the details, you can see them in Section 3.6.5
> > > of this paper: http://www.hit.bme.hu/~lencse/publications/e104-b_2_128.pdf )
> > >
> > > Therefore, I think that the return value of 0 may mean that
> > > rte_eth_tx_burst() can't even commit itself for the future delivery of
> > > the packets. I could only guess why. E.g. all its resources have been
> > > exhausted.
> > >
> > > Best regards,
> > >
> > > Gábor
> > >
> > >
> > > 7/3/2022 5:57 PM keltezéssel, Antonio Di Bacco írta:
> > > > I'm trying to send packets continuously in a  tight loop with a burst
> > > > size of 8 and packets are 9600 bytes long.
> > > > If I don't insert a delay after the rte_eth_tx_burst it always returns 0.
> > > >
> > > > What's the explanation of this behaviour ?
> > > >
> > > > Best regards,
> > > > Antonio.
> > >
>
> Which driver? How did you set the tx_free threshold.
> The driver will need to cleanup already transmitted packets.

Re: rte_eth_tx_burst() always returns 0 in tight loop

[Antonio Di Bacco]

Is there any way to check if a TX queue is full before transmitting
using the rte_eth_rx_burst() or should I rely on the return value of
rte_eth_tx_burst()?

On Thu, Jul 7, 2022 at 4:30 PM Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:
>
> I have an E810-C card with  iavf-4.4.2.1  ice-1.8.8 drivers.
>
> My tx_free_threshold is 8 together with tx_rs_thresh.
>
> I have a tight loop sending BURSTs of 8 packets, each packet is 9014
> bytes long (8 packets take 6 usecs to be serialized).
> If I put the rte_delay_us_block to 7 then everything works fine, every
> cycle 8 packets are transmitted.
> If I lower  the rte_delay_us_block to 1 usec, then I observe that the
> first FOR cycle is ok, nb_tx is 8 as expected, then, the second cycle
> prints a 7 while all subsequent cycles print Z (zero packets sent). I
> know that 1 usec delay is too small and I expect that no packets are
> transmitted for some cycles but I don't understand why I get an nb_tx
> set to 0 forever after the first two cycles.
>
> for (;;)
> {
>         rte_spinlock_lock(&spinlock_conf[src_port]) ;
>         const uint16_t nb_tx = rte_eth_tx_burst(src_port, 0, tx_bufs,
> BURST_SIZE);
>         rte_spinlock_unlock(&spinlock_conf[src_port]);
>
>         rte_delay_us_block(7);  // tested with 1
>
>         if (nb_tx == 0)
>             printf("Z");
>         else if (nb_tx < BURST_SIZE)
>             printf("nb_tx %d\n", nb_tx);
>
>         tx += nb_tx;
>
>         if (unlikely(nb_tx < BURST_SIZE)) {
>             uint16_t buf;
>
>             for (buf = nb_tx; buf < BURST_SIZE; buf++)
>                 rte_pktmbuf_free(tx_bufs[buf]);
>         }
> }
>
> On Wed, Jul 6, 2022 at 5:00 PM Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> >
> > On Wed, 6 Jul 2022 09:21:28 +0200
> > Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:
> >
> > > I wonder why calling eth_dev_tx_burst in a tight loop doesn't allow to
> > > write the packets into the transmit buffer. Only solution I found is
> > > to include a small delay after the tx_burst that is less than the
> > > estimated serialization time of the packet in order to be able to
> > > saturate the ethernet line.
> > >
> > > Anyway I wonder if this is the right approach.
> > >
> > > Thx,
> > > Antonio.
> > >
> > > On Sun, Jul 3, 2022 at 10:19 PM Gábor LENCSE <lencse@hit.bme.hu> wrote:
> > > >
> > > > Dear Antonio,
> > > >
> > > > According to my experience, the rte_eth_tx_burst() function reports the
> > > > packets as "sent" (by a non-zero return value), when they are still in
> > > > the transmit buffer.
> > > >
> > > > (If you are interested in the details, you can see them in Section 3.6.5
> > > > of this paper: http://www.hit.bme.hu/~lencse/publications/e104-b_2_128.pdf )
> > > >
> > > > Therefore, I think that the return value of 0 may mean that
> > > > rte_eth_tx_burst() can't even commit itself for the future delivery of
> > > > the packets. I could only guess why. E.g. all its resources have been
> > > > exhausted.
> > > >
> > > > Best regards,
> > > >
> > > > Gábor
> > > >
> > > >
> > > > 7/3/2022 5:57 PM keltezéssel, Antonio Di Bacco írta:
> > > > > I'm trying to send packets continuously in a  tight loop with a burst
> > > > > size of 8 and packets are 9600 bytes long.
> > > > > If I don't insert a delay after the rte_eth_tx_burst it always returns 0.
> > > > >
> > > > > What's the explanation of this behaviour ?
> > > > >
> > > > > Best regards,
> > > > > Antonio.
> > > >
> >
> > Which driver? How did you set the tx_free threshold.
> > The driver will need to cleanup already transmitted packets.

Re: rte_eth_tx_burst() always returns 0 in tight loop

[Stephen Hemminger]

On Mon, 26 Sep 2022 17:36:06 +0200
Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:

> Is there any way to check if a TX queue is full before transmitting
> using the rte_eth_rx_burst() or should I rely on the return value of
> rte_eth_tx_burst()?

There is an API that might help rte_eth_tx_descriptor_status
but only some drivers support it. And it requires some detailed knowledge
about which descriptor is being used.

Re: rte_eth_tx_burst() always returns 0 in tight loop

[Gábor LENCSE]

Dear Antonio,

I have not met your problem during the implementation of siitperf, 
because I always had to send the frames at a given frame rate, that is, 
I used the appropriate timing.

Could you check, how early the failure begins? I mean that does already 
the second or third call fail? Or it happens only after a few hundred 
calls?

IMHO, the latter is natural, and it means that you have exhausted some 
resources.

Best regards,

Gábor

7/6/2022 9:21 AM keltezéssel, Antonio Di Bacco írta:
> I wonder why calling eth_dev_tx_burst in a tight loop doesn't allow to
> write the packets into the transmit buffer. Only solution I found is
> to include a small delay after the tx_burst that is less than the
> estimated serialization time of the packet in order to be able to
> saturate the ethernet line.
>
> Anyway I wonder if this is the right approach.
>
> Thx,
> Antonio.
>
> On Sun, Jul 3, 2022 at 10:19 PM Gábor LENCSE <lencse@hit.bme.hu> wrote:
>> Dear Antonio,
>>
>> According to my experience, the rte_eth_tx_burst() function reports the
>> packets as "sent" (by a non-zero return value), when they are still in
>> the transmit buffer.
>>
>> (If you are interested in the details, you can see them in Section 3.6.5
>> of this paper: http://www.hit.bme.hu/~lencse/publications/e104-b_2_128.pdf )
>>
>> Therefore, I think that the return value of 0 may mean that
>> rte_eth_tx_burst() can't even commit itself for the future delivery of
>> the packets. I could only guess why. E.g. all its resources have been
>> exhausted.
>>
>> Best regards,
>>
>> Gábor
>>
>>
>> 7/3/2022 5:57 PM keltezéssel, Antonio Di Bacco írta:
>>> I'm trying to send packets continuously in a  tight loop with a burst
>>> size of 8 and packets are 9600 bytes long.
>>> If I don't insert a delay after the rte_eth_tx_burst it always returns 0.
>>>
>>> What's the explanation of this behaviour ?
>>>
>>> Best regards,
>>> Antonio.