Re: [dpdk-dev] DPDK Performance issue with l2fwd

[Alex Markuze <alex@weka.io>]

Hi Zachary,
Your issue may be with the PCI-e 3, with 16 lanes Each slot is limited to
128Gb/s[3].
Now, AFAIK[1] the CPU is connected to the  I/O with a single PCI-E slot.

Several thoughts that may help you:

1. You can figure out the max b/w by running netsurf over the kernel
interfaces (w/o DPDK). Each CPU can handle the Netperf and the Completion
interrupts with grace (packets of 64K and all offloads on) for 10Gb nics.
With more then 10 Nics I would disable the IRQ balancer and make sure
interrupts are spread evenly by setting the  IRQ affinity manually [2].
As long as you have a physical core(NO hyperthreading) per NIC port you can
figure out the MAX B/W you can get with all the nics.

2. You can try using (If available to you , obviously) 40Gb and 56Gb Nics
(Mellanox), In this case for each Netperf flow you will need to separate
each Netperf Stream and the interrupts to different Cores to Reach wire
speed as long as both cores are on the same NUMA node(lscpu).

Hope this helps.

[1]
http://komposter.com.ua/documents/PCI_Express_Base_Specification_Revision_3.0.pdf
[2]
http://h50146.www5.hp.com/products/software/oe/linux/mainstream/support/whitepaper/pdfs/4AA4-9294ENW.pdf
[3]http://en.wikipedia.org/wiki/PCI_Express#PCI_Express_3.x


On Thu, Jul 10, 2014 at 11:07 AM, <Zachary.Jen@cas-well.com> wrote:

> Hey Guys,
>
> Recently, I have used l2fwd to test 160G (82599 10G * 16 ports), but I
> got a strange pheromone in my test.
>
> When I used 12 ports to test the performance of l2fwd, it can work fine
> and achieve 120G.
> But it got abnormal when I using over than 12 port. Part of ports seems
> something wrong and no any Tx/Rx.
> Has anyone know about this?
>
> My testing Environment.
> 1. E5-2658 v2 (10 cores) * 2
>
> http://ark.intel.com/zh-tw/products/76160/Intel-Xeon-Processor-E5-2658-v2-25M-Cache-2_40-GHz
> 2. one core handle one port. (In order to get best performance.)
> 3. No any QPI crossing  issue.
> 4. l2fwd parameters
>      4.1 -c 0xF0FF -- -P 0xF00FF  => 120G get!
>      4.2 -c 0xFF0FF -- -P 0xFF0FF => Failed! Only first 10 ports can
> work well.
>      4.3 -c 0x3F3FF -- -P 0x3F3FF => Failed! Only first 10 ports can
> work well.
>
> BTW, I have tried lots of parameter sets and if I set the ports number
> over than 12 ports, it only first 10 ports got work.
> Else, everything got well.
>
> Can anyone help me to solve the issue? Or DPDK only can set less equal
> than 12 ports?
> Or DPDK max throughput is 120G?
>
> 本信件可能包含瑞祺電通機密資訊，非指定之收件者，請勿使用或揭露本信件內容，並請銷毀此信件。 This email may contain
> confidential information. Please do not use or disclose it in any way and
> delete it if you are not the intended recipient.
>