[dpdk-users] DPDK weird forwarding loss

[dpdk-users] DPDK weird forwarding loss

[Zhihan Jiang]

Hello,

My hardware is HP Proliant DL360 Gen9 with Intel 82599 NIC, 40 cores.
Traffic generator is IXIA Optixia XM2, using IXNetwork.

I use CentOS 6.5 on the host machine, creating 15 SR-IOVs (virtual
functions) on each of the 4 ports and attach them to the guest machine
(CentOS 6.7) I created.

Other settings: intel_iommu=on iommu=pt / blacklist ixgbevf on the host
machine / pause frame off / pin all the ports to the same NUMA node &
socket / VM uses CPU on the same NUMA node & socket.

I attached the vfs on the guest machines to the igb_uio and tried to run
l3fwd. Also sending 64 byte packets from IXIA to do the forwarding job. The
expected throughput should be 14.88 Mpps but I am experiencing 6% loss. I
also tried lower rate like 1~11 Mpps and various frame size (128/512/1518),
but there is always ~0.5%- 1% packet loss

The command line for l3fwd is:
./build/l3fwd -c6 -n4 -w [whitelist devices] --socket-mem=1024,0 -- -p3
--config "(0,0,1), (1,0,2)"

To look into the issue I tried to add the statistic interface on l3fwd. I
found that for 64 byte packet, when <11 Mpps I have no forwarding loss,
which means all the packets received by DPDK are forwarded back to IXIA,
also means all the loss occurs between IXIA and the DPDK. For
128/256/512/1518 byte packet, no forwarding loss shows on the statistics.
(also 0 rx loss/ 0 rx error from rte_eth_stats_get function)

After that I tried following tests but none of them worked:
1. change burst size from 32 to 16, 64, 128, 512.
2. change the rx descriptor size from 128 to 512.
3. use DPDK KNI to create virtual NIC and try to use ethtool to listen to
the incoming packet. Still the same percentage of packet loss (~0.5%)
occured.
4. also tried testpmd but still, the same percentage of packet loss.

I am wondering what is causing this weird transmitting loss between IXIA
and DPDK. It looks like the packet loss occurs in the NIC or before DPDK
can even poll it. Does anyone have any suggestion of checking why this loss
occurs and how I can fix it?

P.S. I have read the threads about the similar issue but they don't seems
to help.
http://dpdk.info/ml/archives/dev/2014-January/001098.html
http://dpdk.org/ml/archives/dev/2015-September/023480.html
http://dpdk.org/ml/archives/dev/2013-May/000123.html

Thanks so much for any help.
Jack

Re: [dpdk-users] DPDK weird forwarding loss

[Kyle Larose]

Hi Jack,

On Mon, Nov 30, 2015 at 8:42 PM, Zhihan Jiang <jackcharm233@gmail.com> wrote:
> Hello,
>

> Other settings: intel_iommu=on iommu=pt / blacklist ixgbevf on the host
> machine / pause frame off / pin all the ports to the same NUMA node &
> socket / VM uses CPU on the same NUMA node & socket.
>

> but there is always ~0.5%- 1% packet loss
>
> The command line for l3fwd is:
> ./build/l3fwd -c6 -n4 -w [whitelist devices] --socket-mem=1024,0 -- -p3
> --config "(0,0,1), (1,0,2)"
>


Try isolating the CPUs on your guest and host so that the forwarding
application cannot be preempted by something else.

See the 'isolcpus' kernel boot parameter.

http://www.linuxtopia.org/online_books/linux_kernel/kernel_configuration/re46.html

In my experience, drops that happen at low rates are caused by the
polling thread not being scheduled to receive packets often enough.
This can either be because it was preempted, which isolcpus will fix,
or because it is sleeping. IIRC l3fwd doesn't sleep, which leaves the
first case the only possibility.

Hope that helps,

Kyle

Re: [dpdk-users] DPDK weird forwarding loss

[Zhihan Jiang]

Hey Kyle,

Thank you for the help. I have tried to add isolcpus=20-27 (which are the 8
cores i pinned to the VM in host) into the /etc/grub.conf (and I also tried
to add isolcpus into guest machine VM) but seems like it doesn't work. I
also tried to disable SMT (hyper-threading) but that just made the result
worse.

I did some performance test and I will list here just in case. To clarify,
TX loss means the packet that dropped before reaching DPDK, FWD loss means
the packet that dropped by DPDK l3fwd app.

64byte packet

MPPS           Loss Rate        TX Loss        FWD Loss
14.88             6.810%              0.018%         6.791%
14.00             1.029%              0.003%         1.000%
13.00             0.273%              0.003%         0.270%
12.00             0.211%              0.211%         0.000%
11.00             0.216%              0.216%         0.000%
10.00             0.240%              0.240%         0.000%
09.00             0.250%              0.250%         0.000%
08.00             0.248%              0.248%         0.000%

The trend can pretty much be told by this stats but it really bugs me why
TX Loss is decreasing while PPS increases. I was wondering if it's bug with
IXIA but I tested it today the packet it sent is OK.

Thank for any help.
Jack



On Wed, Dec 2, 2015 at 2:55 AM, Kyle Larose <eomereadig@gmail.com> wrote:

> Hi Jack,
>
> On Mon, Nov 30, 2015 at 8:42 PM, Zhihan Jiang <jackcharm233@gmail.com>
> wrote:
> > Hello,
> >
>
> > Other settings: intel_iommu=on iommu=pt / blacklist ixgbevf on the host
> > machine / pause frame off / pin all the ports to the same NUMA node &
> > socket / VM uses CPU on the same NUMA node & socket.
> >
>
> > but there is always ~0.5%- 1% packet loss
> >
> > The command line for l3fwd is:
> > ./build/l3fwd -c6 -n4 -w [whitelist devices] --socket-mem=1024,0 -- -p3
> > --config "(0,0,1), (1,0,2)"
> >
>
>
> Try isolating the CPUs on your guest and host so that the forwarding
> application cannot be preempted by something else.
>
> See the 'isolcpus' kernel boot parameter.
>
>
> http://www.linuxtopia.org/online_books/linux_kernel/kernel_configuration/re46.html
>
> In my experience, drops that happen at low rates are caused by the
> polling thread not being scheduled to receive packets often enough.
> This can either be because it was preempted, which isolcpus will fix,
> or because it is sleeping. IIRC l3fwd doesn't sleep, which leaves the
> first case the only possibility.
>
> Hope that helps,
>
> Kyle
>