[dpdk-users] DPDK Distributor app issues

[dpdk-users] DPDK Distributor app issues

[George Yaneff]

I'm playing with the distributor example application from dpdk 19.05.0 I have three machines connected sequentially with an UTP Ethernet cable. The
  middle is Ubuntu 18.04.2 LTS with a eigth core Intel I7 processor and 24 GB ram.  The eth device used is Intel I350 with four ports.  
  The strange behavior I'm observing is when I ping the second node from the first node (traffic passing through the distributor)  there are always one or two packets lost.
  When I use
l2fwd or
basicfwd applications there is no packet loss.  Anyone knows if this is normal or if it is not - what I'm doing wrong?  
  Another thing I noticed is that the ping replies come to the first node with around one second of delay. (1024ms on most packets)  Again there is no latency with the basicfwd (packets arriving in 0.101 ms there).  Is there any way to minimize this delay in distributor application also?  
  George 

Re: [dpdk-users] DPDK Distributor app issues

[Van Haaren, Harry]

> -----Original Message-----
> From: users [mailto:users-bounces@dpdk.org] On Behalf Of George Yaneff
> Sent: Monday, June 10, 2019 3:40 PM
> To: users@dpdk.org
> Subject: [dpdk-users] DPDK Distributor app issues
> 
> I'm playing with the distributor example application from dpdk 19.05.0

Ah cool - Not sure what your goal is, but if you're looking for the
more future-proof load-balancing / distribution APIs in DPDK the I
recommend to look at the Eventdev library, as this has been added to
DPDK for more "pipelined" applications and load-balancing within those
pipeline stages.

If you know your use case matches distributor library, and are aware of
the existence of Eventdev libs, please ignore the above :)

> I
> have three machines connected sequentially with an UTP Ethernet cable. The
>   middle is Ubuntu 18.04.2 LTS with a eigth core Intel I7 processor and 24
> GB ram.  The eth device used is Intel I350 with four ports.
>
>   The strange behavior I'm observing is when I ping the second node from the
> first node (traffic passing through the distributor)  there are always one
> or two packets lost.
>   When I use
> l2fwd or
> basicfwd applications there is no packet loss.  Anyone knows if this is
> normal or if it is not - what I'm doing wrong?

This seems strange - although I'm not very familiar with Distributor sample app.

Perhaps add some statistics counters around the "rte_pktmbuf_free()" calls in
the sample app, to identify if the sample-app is dropping packets in SW, or if
the root cause is really platform configuration?


>   Another thing I noticed is that the ping replies come to the first node
> with around one second of delay. (1024ms on most packets)  Again there is no
> latency with the basicfwd (packets arriving in 0.101 ms there).  Is there
> any way to minimize this delay in distributor application also?

The sample application adds artificial "work" per packet to simulate a real
world distributing workload:
http://git.dpdk.org/dpdk/tree/examples/distributor/main.c#n574

Per packet, 100 cycles of work is performed, so eg:
(burst-size # of packets * 100 cycles) @ CPU freq + RX, TX and Distributor
work will be the total delay along the wire.


>   George

Hope that helps, -Harry

Re: [dpdk-users] DPDK Distributor app issues

[George Yaneff]

Harry, thank you for the answer.
 I will look into this event dev lib, but I hoped my time with the distributor lib will not be wasted. Actually  it pretty much fits into my use case.
  
  About the packet loss I suspect that they are always the first one or two packets, I'm still debugging and trying to  figure out what really happens.  
  About the delay - unfortunately the cycles in the worker are not the cause for them. When I comment these lines  there is no effect on the delay. Also these is some buffering happening in the tx_core but the reason is not there also.  I suspect some buffering happens internally in the distributor itself.  I understand that the throughput is traded for bigger latency, but hoped there will be a way to tune this.  
  Regards  George  
  
  
  







 >-------- Оригинално писмо --------

 >От: "Van Haaren, Harry" harry.van.haaren@intel.com

 >Относно: RE: [dpdk-users] DPDK Distributor app issues

 >До: George Yaneff  , "users@dpdk.org"  

 >Изпратено на: 10.06.2019 19:13


> -----Original Message-----

> From: users [mailto:users-bounces@dpdk.org] On Behalf Of George Yaneff

> Sent: Monday, June 10, 2019 3:40 PM

> To:  users@dpdk.org 

> Subject: [dpdk-users] DPDK Distributor app issues

> 

> I'm playing with the distributor example application from dpdk 19.05.0



Ah cool - Not sure what your goal is, but if you're looking for the

more future-proof load-balancing / distribution APIs in DPDK the I

recommend to look at the Eventdev library, as this has been added to

DPDK for more "pipelined" applications and load-balancing within those

pipeline stages.



If you know your use case matches distributor library, and are aware of

the existence of Eventdev libs, please ignore the above :)



> I

> have three machines connected sequentially with an UTP Ethernet cable. The

>   middle is Ubuntu 18.04.2 LTS with a eigth core Intel I7 processor and 24

> GB ram.  The eth device used is Intel I350 with four ports.

>

>   The strange behavior I'm observing is when I ping the second node from the

> first node (traffic passing through the distributor)  there are always one

> or two packets lost.

>   When I use

> l2fwd or

> basicfwd applications there is no packet loss.  Anyone knows if this is

> normal or if it is not - what I'm doing wrong?



This seems strange - although I'm not very familiar with Distributor sample app.



Perhaps add some statistics counters around the "rte_pktmbuf_free()" calls in

the sample app, to identify if the sample-app is dropping packets in SW, or if

the root cause is really platform configuration?





>   Another thing I noticed is that the ping replies come to the first node

> with around one second of delay. (1024ms on most packets)  Again there is no

> latency with the basicfwd (packets arriving in 0.101 ms there).  Is there

> any way to minimize this delay in distributor application also?



The sample application adds artificial "work" per packet to simulate a real

world distributing workload:

 http://git.dpdk.org/dpdk/tree/examples/distributor/main.c#n574 



Per packet, 100 cycles of work is performed, so eg:

(burst-size # of packets * 100 cycles) @ CPU freq + RX, TX and Distributor

work will be the total delay along the wire.





>   George



Hope that helps, -Harry