Corrupted payload when sending many individual mbufs in loop

Corrupted payload when sending many individual mbufs in loop

[Alan Beadle]

Hello,

I am working on an application that involves fragmenting a large
buffer into many smaller packets/mbufs and sending them with DPDK. The
fragment packets for each data buffer are currently sent in a loop
without any throttling (I plan to add some sort of rate limiting
later). I am not using any of the special DPDK features for IP
fragmention. I am simply preparing and sending mbufs for each
fragment.

The problem I am having is that when I need to send large buffers,
which involve many packets being sent in a tight loop, sometimes part
of the payload is corrupted when received on the other machine. This
is rare and only happens above a certain data rate. I would like to
have a reliable way to detect (or even better, prevent) this
corruption.

Out of paranoia I placed an rte_compiler_barrier() and then an
rte_mb() (a memory barrier) between the code that prepares the mbuf
and the actual tx_burst(). It did not help and I think it should be
unnecessary.

I found some information about the function rte_mbuf_check() and have
started using it to check every received packet on the other end. It
fails to detect this corruption.

Any advice would be appreciated.
-Alan

Re: Corrupted payload when sending many individual mbufs in loop

[Stephen Hemminger]

On Tue, 4 Feb 2025 17:36:31 -0500
Alan Beadle <ab.beadle@gmail.com> wrote:

> Hello,
> 
> I am working on an application that involves fragmenting a large
> buffer into many smaller packets/mbufs and sending them with DPDK. The
> fragment packets for each data buffer are currently sent in a loop
> without any throttling (I plan to add some sort of rate limiting
> later). I am not using any of the special DPDK features for IP
> fragmention. I am simply preparing and sending mbufs for each
> fragment.
> 
> The problem I am having is that when I need to send large buffers,
> which involve many packets being sent in a tight loop, sometimes part
> of the payload is corrupted when received on the other machine. This
> is rare and only happens above a certain data rate. I would like to
> have a reliable way to detect (or even better, prevent) this
> corruption.
> 
> Out of paranoia I placed an rte_compiler_barrier() and then an
> rte_mb() (a memory barrier) between the code that prepares the mbuf
> and the actual tx_burst(). It did not help and I think it should be
> unnecessary.
> 
> I found some information about the function rte_mbuf_check() and have
> started using it to check every received packet on the other end. It
> fails to detect this corruption.
> 
> Any advice would be appreciated.
> -Alan

Are you using refcounting of the mbuf. Are you reusing the data?
DPDK has no callback or mechanism to indicate when data has actually been
sent; ie the mbuf may still be in use for some amount of time while sitting
in the device transmit queue and being DMA'd. Fragmenting can be complex
to do.

Re: Corrupted payload when sending many individual mbufs in loop

[Alan Beadle]

> Are you using refcounting of the mbuf. Are you reusing the data?
> DPDK has no callback or mechanism to indicate when data has actually been
> sent; ie the mbuf may still be in use for some amount of time while sitting
> in the device transmit queue and being DMA'd. Fragmenting can be complex
> to do.

Unlike my earlier threads, in this case I am not saving any mbufs nor
am I doing anything at all with the refcount. For retries I am
regenerating the lost packet from scratch each time. The problem is
that at high tx rates the payload is getting corrupted and I have no
idea why.

Re: Corrupted payload when sending many individual mbufs in loop

[Alan Beadle]

Please disregard. I found my problem and it was not due to DPDK.
Thanks.