[dpdk-users] dpdk packet loss,delay, retransmission

[dpdk-users] dpdk packet loss,delay, retransmission

[Anupama Laxmi]

I see delay in TCP packet transfer while transferring huge files using
SSH/SCP. The delay is not seen with file sizes up to 35 MB. Beyond 35 MB
there is delay due to TCP out of order /retransmission and packet loss. For
file size 762835531 earlier it took 8 seconds for scp transfer now it takes
~~ 4 mins after DPDK upgrade. I suspected Small RX and TX queue size on
NICs which are connected to DPDK is causing packet drops.

   1. Increased TX queue and RX queue size as follows.
   RTE_TEST_RX_DESC_DEFAULT 128 RTE_TEST_TX_DESC_DEFAULT 512

to

RTE_TEST_RX_DESC_DEFAULT 4096 RTE_TEST_TX_DESC_DEFAULT 4096

Saw little improvement. The time taken for scp reduced to 2mins and 30
seconds for file of size 762835531.

   1. Tried to rate limit the TX rate using: rte_eth_set_queue_rate_limit
   API

Not seeing much improvement.

Please suggest ways to configure the mbuf,queue size to optimize the
performance and reduce the delay and drop in packet transfer.

Re: [dpdk-users] dpdk packet loss,delay, retransmission

[Stephen Hemminger]

On Sun, 21 Jul 2019 15:11:34 +0530
Anupama Laxmi <anupamalaxmi4@gmail.com> wrote:

> I see delay in TCP packet transfer while transferring huge files using
> SSH/SCP. The delay is not seen with file sizes up to 35 MB. Beyond 35 MB
> there is delay due to TCP out of order /retransmission and packet loss. For
> file size 762835531 earlier it took 8 seconds for scp transfer now it takes
> ~~ 4 mins after DPDK upgrade. I suspected Small RX and TX queue size on
> NICs which are connected to DPDK is causing packet drops.
> 
>    1. Increased TX queue and RX queue size as follows.
>    RTE_TEST_RX_DESC_DEFAULT 128 RTE_TEST_TX_DESC_DEFAULT 512
> 
> to
> 
> RTE_TEST_RX_DESC_DEFAULT 4096 RTE_TEST_TX_DESC_DEFAULT 4096
> 
> Saw little improvement. The time taken for scp reduced to 2mins and 30
> seconds for file of size 762835531.
> 
>    1. Tried to rate limit the TX rate using: rte_eth_set_queue_rate_limit
>    API
> 
> Not seeing much improvement.
> 
> Please suggest ways to configure the mbuf,queue size to optimize the
> performance and reduce the delay and drop in packet transfer.

Welcome to the world of buffer bloat.

The root cause is that large receive and transmit queues give worse performance
in the real world.  The optimum queue size is actually smaller than you think.
The DPDK examples are all badly tuned for real life, they encourage bufferbloat.
The values are to allow UDP  packet benchmarks to be fast.

What is your packet size and data rate?
You should aim for < 1ms of buffering.

DPDK could do better by implementing something like Byte Queue Limits in
Linux and/or CoDel. But this requires a software queueing layer on top of
the hardware.