* [dpdk-dev] Performances are not scale with multiple ports
@ 2013-05-28 2:26 Shinae Woo
2013-05-28 2:30 ` Naoto MATSUMOTO
2013-05-28 9:22 ` Thomas Monjalon
0 siblings, 2 replies; 8+ messages in thread
From: Shinae Woo @ 2013-05-28 2:26 UTC (permalink / raw)
To: dev
[-- Attachment #1: Type: text/plain, Size: 4461 bytes --]
Hello, all.
I play the dpdk-1.2.3r1 with examples.
But I can not achieve the line-rate packet receive performance,
and the performance is not scale with multiple ports.
For example, in example l2fwd, I have tested two cases with 2 ports, and 4
ports,
using belowed command line each
./build/l2fwd -cf -n3 -- -p3
./build/l2fwd -cf -n3 -- -pf
But both cases, the aggregated performance are not scale.
== experiments environments ==
- Two Intel 82599 NICs (total 4 ports)
- Intel Xeon X5690 @ 3.47GHz * 2 (total 12 cores)
- 1024 * 2MB hugepages
- Linux 2.6.38-15-server
- Each ports receiving 10Gbps of traffic of 64 bytes packets, 14.88Mpps.
*1. Packet forwarding performance*
In 2 ports case, receive performance is 13Mpps,
In 4 ports case, not 26Mbps, only 16.8Mpps.
Port statistics ====================================
Statistics for port 0 ------------------------------
Packets sent: 4292256
Packets received: 6517396
Packets dropped: 2224776
Statistics for port 1 ------------------------------
Packets sent: 4291840
Packets received: 6517044
Packets dropped: 2225556
Aggregate statistics ===============================
Total packets sent: 8584128
Total packets received: 13034472
Total packets dropped: 4450332
====================================================
Port statistics ====================================
Statistics for port 0 ------------------------------
Packets sent: 1784064
Packets received: 2632700
Packets dropped: 848128
Statistics for port 1 ------------------------------
Packets sent: 1784104
Packets received: 2632196
Packets dropped: 848596
Statistics for port 2 ------------------------------
Packets sent: 3587616
Packets received: 5816344
Packets dropped: 2200176
Statistics for port 3 ------------------------------
Packets sent: 3587712
Packets received: 5787848
Packets dropped: 2228684
Aggregate statistics ===============================
Total packets sent: 10743560
Total packets received: 16869152
Total packets dropped: 6125608
====================================================
*2. Packet receiving performance*
I fix the codes for only receiving packets (not forwarding),
the performance is still not scalable as each 13.3Mpps, 18Mpps.
Port statistics ====================================
Statistics for port 0 ------------------------------
Packets sent: 0
Packets received: 6678860
Packets dropped: 0
Statistics for port 1 ------------------------------
Packets sent: 0
Packets received: 6646120
Packets dropped: 0
Aggregate statistics ===============================
Total packets sent: 0
Total packets received: 13325012
Total packets dropped: 0
====================================================
Port statistics ====================================
Statistics for port 0 ------------------------------
Packets sent: 0
Packets received: 3129624
Packets dropped: 0
Statistics for port 1 ------------------------------
Packets sent: 0
Packets received: 3131292
Packets dropped: 0
Statistics for port 2 ------------------------------
Packets sent: 0
Packets received: 6260908
Packets dropped: 0
Statistics for port 3 ------------------------------
Packets sent: 0
Packets received: 6238764
Packets dropped: 0
Aggregate statistics ===============================
Total packets sent: 0
Total packets received: 18760640
Total packets dropped: 0
====================================================
The question is that
1. How I can achieve each port receiving full 14.88Mpps ?
What might be the bottleneck in current environment?
2. Why the performance using multiple ports is not scale?
I guess doubling ports shows the doubling the receiving performance,
but it shows not. I am curious about what is limiting the packet
receivng performance.
Thanks,
Shinae
[-- Attachment #2: Type: text/html, Size: 5153 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dpdk-dev] Performances are not scale with multiple ports
2013-05-28 2:26 [dpdk-dev] Performances are not scale with multiple ports Shinae Woo
@ 2013-05-28 2:30 ` Naoto MATSUMOTO
2013-05-28 3:05 ` Shinae Woo
2013-05-28 9:22 ` Thomas Monjalon
1 sibling, 1 reply; 8+ messages in thread
From: Naoto MATSUMOTO @ 2013-05-28 2:30 UTC (permalink / raw)
To: Shinae Woo; +Cc: dev
FYI: Disruptive IP Networking with Intel DPDK on Linux
http://slidesha.re/SeVFZo
On Tue, 28 May 2013 11:26:30 +0900
Shinae Woo <shinae2012@gmail.com> wrote:
> Hello, all.
>
> I play the dpdk-1.2.3r1 with examples.
>
> But I can not achieve the line-rate packet receive performance,
> and the performance is not scale with multiple ports.
>
> For example, in example l2fwd, I have tested two cases with 2 ports, and 4
> ports,
> using belowed command line each
>
> ./build/l2fwd -cf -n3 -- -p3
> ./build/l2fwd -cf -n3 -- -pf
>
> But both cases, the aggregated performance are not scale.
>
> == experiments environments ==
> - Two Intel 82599 NICs (total 4 ports)
> - Intel Xeon X5690 @ 3.47GHz * 2 (total 12 cores)
> - 1024 * 2MB hugepages
> - Linux 2.6.38-15-server
> - Each ports receiving 10Gbps of traffic of 64 bytes packets, 14.88Mpps.
>
> *1. Packet forwarding performance*
>
> In 2 ports case, receive performance is 13Mpps,
> In 4 ports case, not 26Mbps, only 16.8Mpps.
>
> Port statistics ====================================
> Statistics for port 0 ------------------------------
> Packets sent: 4292256
> Packets received: 6517396
> Packets dropped: 2224776
> Statistics for port 1 ------------------------------
> Packets sent: 4291840
> Packets received: 6517044
> Packets dropped: 2225556
> Aggregate statistics ===============================
> Total packets sent: 8584128
> Total packets received: 13034472
> Total packets dropped: 4450332
> ====================================================
>
> Port statistics ====================================
> Statistics for port 0 ------------------------------
> Packets sent: 1784064
> Packets received: 2632700
> Packets dropped: 848128
> Statistics for port 1 ------------------------------
> Packets sent: 1784104
> Packets received: 2632196
> Packets dropped: 848596
> Statistics for port 2 ------------------------------
> Packets sent: 3587616
> Packets received: 5816344
> Packets dropped: 2200176
> Statistics for port 3 ------------------------------
> Packets sent: 3587712
> Packets received: 5787848
> Packets dropped: 2228684
> Aggregate statistics ===============================
> Total packets sent: 10743560
> Total packets received: 16869152
> Total packets dropped: 6125608
> ====================================================
>
> *2. Packet receiving performance*
> I fix the codes for only receiving packets (not forwarding),
> the performance is still not scalable as each 13.3Mpps, 18Mpps.
>
> Port statistics ====================================
> Statistics for port 0 ------------------------------
> Packets sent: 0
> Packets received: 6678860
> Packets dropped: 0
> Statistics for port 1 ------------------------------
> Packets sent: 0
> Packets received: 6646120
> Packets dropped: 0
> Aggregate statistics ===============================
> Total packets sent: 0
> Total packets received: 13325012
> Total packets dropped: 0
> ====================================================
>
> Port statistics ====================================
> Statistics for port 0 ------------------------------
> Packets sent: 0
> Packets received: 3129624
> Packets dropped: 0
> Statistics for port 1 ------------------------------
> Packets sent: 0
> Packets received: 3131292
> Packets dropped: 0
> Statistics for port 2 ------------------------------
> Packets sent: 0
> Packets received: 6260908
> Packets dropped: 0
> Statistics for port 3 ------------------------------
> Packets sent: 0
> Packets received: 6238764
> Packets dropped: 0
> Aggregate statistics ===============================
> Total packets sent: 0
> Total packets received: 18760640
> Total packets dropped: 0
> ====================================================
>
> The question is that
> 1. How I can achieve each port receiving full 14.88Mpps ?
> What might be the bottleneck in current environment?
> 2. Why the performance using multiple ports is not scale?
> I guess doubling ports shows the doubling the receiving performance,
> but it shows not. I am curious about what is limiting the packet
> receivng performance.
>
> Thanks,
> Shinae
--
SAKURA Internet Inc. / Senior Researcher
Naoto MATSUMOTO <n-matsumoto@sakura.ad.jp>
SAKURA Internet Research Center <http://research.sakura.ad.jp/>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dpdk-dev] Performances are not scale with multiple ports
2013-05-28 2:30 ` Naoto MATSUMOTO
@ 2013-05-28 3:05 ` Shinae Woo
2013-05-28 3:15 ` Emre Eraltan
2013-05-29 3:09 ` Naoto MATSUMOTO
0 siblings, 2 replies; 8+ messages in thread
From: Shinae Woo @ 2013-05-28 3:05 UTC (permalink / raw)
To: Naoto MATSUMOTO; +Cc: dev
[-- Attachment #1: Type: text/plain, Size: 6074 bytes --]
Thanks for sharing Naoto.
So in your experiments, the forwarding performance still does not reach the
line rate.
Your perf record shows that the cpu spend most of the time in polling for
receiving packets,
and no other heavy operation.
Even though the application polling packets in its best,
the forwarder miss some packets from elsewhere from the application-side.
The dpdk document shows that 160Mpps forwarding performance in 2 sockets,
but I can only reach the 13 Mpps in 2 ports.
Even doubling the number of ports to 4 ports, the performance is still less
than 17Mpps.
I want to know where is the bottleneck lies in my environments, or
how I can reprocuce the same performance as the dpdk published.
Thank you,
Shinae
On Tue, May 28, 2013 at 11:30 AM, Naoto MATSUMOTO
<n-matsumoto@sakura.ad.jp>wrote:
>
> FYI: Disruptive IP Networking with Intel DPDK on Linux
> http://slidesha.re/SeVFZo
>
>
> On Tue, 28 May 2013 11:26:30 +0900
> Shinae Woo <shinae2012@gmail.com> wrote:
>
> > Hello, all.
> >
> > I play the dpdk-1.2.3r1 with examples.
> >
> > But I can not achieve the line-rate packet receive performance,
> > and the performance is not scale with multiple ports.
> >
> > For example, in example l2fwd, I have tested two cases with 2 ports, and
> 4
> > ports,
> > using belowed command line each
> >
> > ./build/l2fwd -cf -n3 -- -p3
> > ./build/l2fwd -cf -n3 -- -pf
> >
> > But both cases, the aggregated performance are not scale.
> >
> > == experiments environments ==
> > - Two Intel 82599 NICs (total 4 ports)
> > - Intel Xeon X5690 @ 3.47GHz * 2 (total 12 cores)
> > - 1024 * 2MB hugepages
> > - Linux 2.6.38-15-server
> > - Each ports receiving 10Gbps of traffic of 64 bytes packets, 14.88Mpps.
> >
> > *1. Packet forwarding performance*
> >
> > In 2 ports case, receive performance is 13Mpps,
> > In 4 ports case, not 26Mbps, only 16.8Mpps.
> >
> > Port statistics ====================================
> > Statistics for port 0 ------------------------------
> > Packets sent: 4292256
> > Packets received: 6517396
> > Packets dropped: 2224776
> > Statistics for port 1 ------------------------------
> > Packets sent: 4291840
> > Packets received: 6517044
> > Packets dropped: 2225556
> > Aggregate statistics ===============================
> > Total packets sent: 8584128
> > Total packets received: 13034472
> > Total packets dropped: 4450332
> > ====================================================
> >
> > Port statistics ====================================
> > Statistics for port 0 ------------------------------
> > Packets sent: 1784064
> > Packets received: 2632700
> > Packets dropped: 848128
> > Statistics for port 1 ------------------------------
> > Packets sent: 1784104
> > Packets received: 2632196
> > Packets dropped: 848596
> > Statistics for port 2 ------------------------------
> > Packets sent: 3587616
> > Packets received: 5816344
> > Packets dropped: 2200176
> > Statistics for port 3 ------------------------------
> > Packets sent: 3587712
> > Packets received: 5787848
> > Packets dropped: 2228684
> > Aggregate statistics ===============================
> > Total packets sent: 10743560
> > Total packets received: 16869152
> > Total packets dropped: 6125608
> > ====================================================
> >
> > *2. Packet receiving performance*
> > I fix the codes for only receiving packets (not forwarding),
> > the performance is still not scalable as each 13.3Mpps, 18Mpps.
> >
> > Port statistics ====================================
> > Statistics for port 0 ------------------------------
> > Packets sent: 0
> > Packets received: 6678860
> > Packets dropped: 0
> > Statistics for port 1 ------------------------------
> > Packets sent: 0
> > Packets received: 6646120
> > Packets dropped: 0
> > Aggregate statistics ===============================
> > Total packets sent: 0
> > Total packets received: 13325012
> > Total packets dropped: 0
> > ====================================================
> >
> > Port statistics ====================================
> > Statistics for port 0 ------------------------------
> > Packets sent: 0
> > Packets received: 3129624
> > Packets dropped: 0
> > Statistics for port 1 ------------------------------
> > Packets sent: 0
> > Packets received: 3131292
> > Packets dropped: 0
> > Statistics for port 2 ------------------------------
> > Packets sent: 0
> > Packets received: 6260908
> > Packets dropped: 0
> > Statistics for port 3 ------------------------------
> > Packets sent: 0
> > Packets received: 6238764
> > Packets dropped: 0
> > Aggregate statistics ===============================
> > Total packets sent: 0
> > Total packets received: 18760640
> > Total packets dropped: 0
> > ====================================================
> >
> > The question is that
> > 1. How I can achieve each port receiving full 14.88Mpps ?
> > What might be the bottleneck in current environment?
> > 2. Why the performance using multiple ports is not scale?
> > I guess doubling ports shows the doubling the receiving performance,
> > but it shows not. I am curious about what is limiting the packet
> > receivng performance.
> >
> > Thanks,
> > Shinae
>
> --
> SAKURA Internet Inc. / Senior Researcher
> Naoto MATSUMOTO <n-matsumoto@sakura.ad.jp>
> SAKURA Internet Research Center <http://research.sakura.ad.jp/>
>
>
[-- Attachment #2: Type: text/html, Size: 7711 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dpdk-dev] Performances are not scale with multiple ports
2013-05-28 3:05 ` Shinae Woo
@ 2013-05-28 3:15 ` Emre Eraltan
2013-05-28 3:29 ` Stephen Hemminger
2013-05-29 3:09 ` Naoto MATSUMOTO
1 sibling, 1 reply; 8+ messages in thread
From: Emre Eraltan @ 2013-05-28 3:15 UTC (permalink / raw)
To: Shinae Woo; +Cc: dev
[-- Attachment #1: Type: text/plain, Size: 7375 bytes --]
Hello Shinae,
Did you try to use the testpmd tool with multiple queues per port? It
gives you more flexibility compared to l2fwd app.
You need to trigger the RSS feature of the NIC by sending different
streams (just by changing the destination port for instance or any
information in the 5-tuple). This will load balance your packets among
several cores so that you can probe multiple queues with different
cores. Otherwise, you will use only one core (or thread if HT is
enabled) per port for the RX side.
Best Regards,
Emre
--
Emre ERALTAN
6WIND Field Application Engineer
On 27/05/2013 20:05, Shinae Woo wrote:
> Thanks for sharing Naoto.
>
> So in your experiments, the forwarding performance still does not
> reach the line rate.
>
> Your perf record shows that the cpu spend most of the time in polling
> for receiving packets,
> and no other heavy operation.
> Even though the application polling packets in its best,
> the forwarder miss some packets from elsewhere from the application-side.
>
> The dpdk document shows that 160Mpps forwarding performance in 2 sockets,
> but I can only reach the 13 Mpps in 2 ports.
> Even doubling the number of ports to 4 ports, the performance is still
> less than 17Mpps.
>
> I want to know where is the bottleneck lies in my environments, or
> how I can reprocuce the same performance as the dpdk published.
>
> Thank you,
> Shinae
>
>
>
> On Tue, May 28, 2013 at 11:30 AM, Naoto MATSUMOTO
> <n-matsumoto@sakura.ad.jp <mailto:n-matsumoto@sakura.ad.jp>> wrote:
>
>
> FYI: Disruptive IP Networking with Intel DPDK on Linux
> http://slidesha.re/SeVFZo
>
>
> On Tue, 28 May 2013 11:26:30 +0900
> Shinae Woo <shinae2012@gmail.com <mailto:shinae2012@gmail.com>> wrote:
>
> > Hello, all.
> >
> > I play the dpdk-1.2.3r1 with examples.
> >
> > But I can not achieve the line-rate packet receive performance,
> > and the performance is not scale with multiple ports.
> >
> > For example, in example l2fwd, I have tested two cases with 2
> ports, and 4
> > ports,
> > using belowed command line each
> >
> > ./build/l2fwd -cf -n3 -- -p3
> > ./build/l2fwd -cf -n3 -- -pf
> >
> > But both cases, the aggregated performance are not scale.
> >
> > == experiments environments ==
> > - Two Intel 82599 NICs (total 4 ports)
> > - Intel Xeon X5690 @ 3.47GHz * 2 (total 12 cores)
> > - 1024 * 2MB hugepages
> > - Linux 2.6.38-15-server
> > - Each ports receiving 10Gbps of traffic of 64 bytes packets,
> 14.88Mpps.
> >
> > *1. Packet forwarding performance*
> >
> > In 2 ports case, receive performance is 13Mpps,
> > In 4 ports case, not 26Mbps, only 16.8Mpps.
> >
> > Port statistics ====================================
> > Statistics for port 0 ------------------------------
> > Packets sent: 4292256
> > Packets received: 6517396
> > Packets dropped: 2224776
> > Statistics for port 1 ------------------------------
> > Packets sent: 4291840
> > Packets received: 6517044
> > Packets dropped: 2225556
> > Aggregate statistics ===============================
> > Total packets sent: 8584128
> > Total packets received: 13034472
> > Total packets dropped: 4450332
> > ====================================================
> >
> > Port statistics ====================================
> > Statistics for port 0 ------------------------------
> > Packets sent: 1784064
> > Packets received: 2632700
> > Packets dropped: 848128
> > Statistics for port 1 ------------------------------
> > Packets sent: 1784104
> > Packets received: 2632196
> > Packets dropped: 848596
> > Statistics for port 2 ------------------------------
> > Packets sent: 3587616
> > Packets received: 5816344
> > Packets dropped: 2200176
> > Statistics for port 3 ------------------------------
> > Packets sent: 3587712
> > Packets received: 5787848
> > Packets dropped: 2228684
> > Aggregate statistics ===============================
> > Total packets sent: 10743560
> > Total packets received: 16869152
> > Total packets dropped: 6125608
> > ====================================================
> >
> > *2. Packet receiving performance*
> > I fix the codes for only receiving packets (not forwarding),
> > the performance is still not scalable as each 13.3Mpps, 18Mpps.
> >
> > Port statistics ====================================
> > Statistics for port 0 ------------------------------
> > Packets sent: 0
> > Packets received: 6678860
> > Packets dropped: 0
> > Statistics for port 1 ------------------------------
> > Packets sent: 0
> > Packets received: 6646120
> > Packets dropped: 0
> > Aggregate statistics ===============================
> > Total packets sent: 0
> > Total packets received: 13325012
> > Total packets dropped: 0
> > ====================================================
> >
> > Port statistics ====================================
> > Statistics for port 0 ------------------------------
> > Packets sent: 0
> > Packets received: 3129624
> > Packets dropped: 0
> > Statistics for port 1 ------------------------------
> > Packets sent: 0
> > Packets received: 3131292
> > Packets dropped: 0
> > Statistics for port 2 ------------------------------
> > Packets sent: 0
> > Packets received: 6260908
> > Packets dropped: 0
> > Statistics for port 3 ------------------------------
> > Packets sent: 0
> > Packets received: 6238764
> > Packets dropped: 0
> > Aggregate statistics ===============================
> > Total packets sent: 0
> > Total packets received: 18760640
> > Total packets dropped: 0
> > ====================================================
> >
> > The question is that
> > 1. How I can achieve each port receiving full 14.88Mpps ?
> > What might be the bottleneck in current environment?
> > 2. Why the performance using multiple ports is not scale?
> > I guess doubling ports shows the doubling the receiving
> performance,
> > but it shows not. I am curious about what is limiting the packet
> > receivng performance.
> >
> > Thanks,
> > Shinae
>
> --
> SAKURA Internet Inc. / Senior Researcher
> Naoto MATSUMOTO <n-matsumoto@sakura.ad.jp
> <mailto:n-matsumoto@sakura.ad.jp>>
> SAKURA Internet Research Center <http://research.sakura.ad.jp/>
>
>
[-- Attachment #2: Type: text/html, Size: 14024 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dpdk-dev] Performances are not scale with multiple ports
2013-05-28 3:15 ` Emre Eraltan
@ 2013-05-28 3:29 ` Stephen Hemminger
2013-05-28 4:00 ` Shinae Woo
0 siblings, 1 reply; 8+ messages in thread
From: Stephen Hemminger @ 2013-05-28 3:29 UTC (permalink / raw)
To: Emre Eraltan; +Cc: dev
On Mon, 27 May 2013 20:15:23 -0700
Emre Eraltan <emre.eraltan@6wind.com> wrote:
> Hello Shinae,
>
> Did you try to use the testpmd tool with multiple queues per port? It
> gives you more flexibility compared to l2fwd app.
>
> You need to trigger the RSS feature of the NIC by sending different
> streams (just by changing the destination port for instance or any
> information in the 5-tuple). This will load balance your packets among
> several cores so that you can probe multiple queues with different
> cores. Otherwise, you will use only one core (or thread if HT is
> enabled) per port for the RX side.
>
> Best Regards,
> Emre
>
I wrote a partial baked version of pktgen for DPDK. Let me see if I can
get ok to release and clean it up.
Windriver has one as well, and they use it in their demo clases.
Not sure why it never got included.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dpdk-dev] Performances are not scale with multiple ports
2013-05-28 3:29 ` Stephen Hemminger
@ 2013-05-28 4:00 ` Shinae Woo
0 siblings, 0 replies; 8+ messages in thread
From: Shinae Woo @ 2013-05-28 4:00 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: dev
[-- Attachment #1: Type: text/plain, Size: 1563 bytes --]
Actually, I made my own packet reception application using DPDK
to utilize multiple RSS which map each core (in my case, 12 queues for 12
cores).
But it shows that multiple RSS shows, the reception performance is rather
decreasing as below.
# queue
1 2 3 4 5 6 Bandwidth (Gbps) 8.95 8.90 8.89 8.57 7.95 7.62 #queue 7 8 9
10 11 12 Bandwidth (Gbps) 7.18 7.13 7.12 7.04 7.58 6.80
I will try with testpmd to load balance again.
I use the custom packet generator module over Packet I/O,
but I'm looking forward to the pktgen for DPDK of Stephen's, too.
Thanks,
Shinae
On Tue, May 28, 2013 at 12:29 PM, Stephen Hemminger <
stephen@networkplumber.org> wrote:
> On Mon, 27 May 2013 20:15:23 -0700
> Emre Eraltan <emre.eraltan@6wind.com> wrote:
>
> > Hello Shinae,
> >
> > Did you try to use the testpmd tool with multiple queues per port? It
> > gives you more flexibility compared to l2fwd app.
> >
> > You need to trigger the RSS feature of the NIC by sending different
> > streams (just by changing the destination port for instance or any
> > information in the 5-tuple). This will load balance your packets among
> > several cores so that you can probe multiple queues with different
> > cores. Otherwise, you will use only one core (or thread if HT is
> > enabled) per port for the RX side.
> >
> > Best Regards,
> > Emre
> >
>
> I wrote a partial baked version of pktgen for DPDK. Let me see if I can
> get ok to release and clean it up.
>
> Windriver has one as well, and they use it in their demo clases.
> Not sure why it never got included.
>
>
[-- Attachment #2: Type: text/html, Size: 4037 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dpdk-dev] Performances are not scale with multiple ports
2013-05-28 2:26 [dpdk-dev] Performances are not scale with multiple ports Shinae Woo
2013-05-28 2:30 ` Naoto MATSUMOTO
@ 2013-05-28 9:22 ` Thomas Monjalon
1 sibling, 0 replies; 8+ messages in thread
From: Thomas Monjalon @ 2013-05-28 9:22 UTC (permalink / raw)
To: Shinae Woo; +Cc: dev
Hello,
28/05/2013 04:26, Shinae Woo :
> ./build/l2fwd -cf -n3 -- -p3
> ./build/l2fwd -cf -n3 -- -pf
>
> But both cases, the aggregated performance are not scale.
[..]
> The question is that
> 1. How I can achieve each port receiving full 14.88Mpps ?
> What might be the bottleneck in current environment?
> 2. Why the performance using multiple ports is not scale?
> I guess doubling ports shows the doubling the receiving performance,
> but it shows not. I am curious about what is limiting the packet
> receivng performance.
In order to scale with line-rate performance, you probably have to adjust the
configuration to your hardware. Several parameters should be considered:
- PCIe bus (is it fast enough ?)
- NUMA sockets (copy between NUMA nodes could slow forwarding)
- CPU threads (classic SMP is better than hyperthreading)
To make it short, you could check your PCI bus and your CPU configuration.
For the latter, /proc/cpuinfo should provide enough information with fields
"physical id" and "core id".
--
Thomas
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dpdk-dev] Performances are not scale with multiple ports
2013-05-28 3:05 ` Shinae Woo
2013-05-28 3:15 ` Emre Eraltan
@ 2013-05-29 3:09 ` Naoto MATSUMOTO
1 sibling, 0 replies; 8+ messages in thread
From: Naoto MATSUMOTO @ 2013-05-29 3:09 UTC (permalink / raw)
To: Shinae Woo; +Cc: dev
Hi shiane and all
Emre and Stephen, thanks your suggest for us.
We are not testing multi-port traffic gen using NETMAP though DPDK l3fw,
but we understood some packet recive-side without NETMAP Linux.
Please try it using stephen's pktgen if you have more traffic gen/recive
server resources or Using IXIA.
best regards,
--
Naoto
On Tue, 28 May 2013 12:05:50 +0900
Shinae Woo <shinae2012@gmail.com> wrote:
> Thanks for sharing Naoto.
>
> So in your experiments, the forwarding performance still does not reach the
> line rate.
>
> Your perf record shows that the cpu spend most of the time in polling for
> receiving packets,
> and no other heavy operation.
> Even though the application polling packets in its best,
> the forwarder miss some packets from elsewhere from the application-side.
>
> The dpdk document shows that 160Mpps forwarding performance in 2 sockets,
> but I can only reach the 13 Mpps in 2 ports.
> Even doubling the number of ports to 4 ports, the performance is still less
> than 17Mpps.
>
> I want to know where is the bottleneck lies in my environments, or
> how I can reprocuce the same performance as the dpdk published.
>
> Thank you,
> Shinae
>
>
>
> On Tue, May 28, 2013 at 11:30 AM, Naoto MATSUMOTO
> <n-matsumoto@sakura.ad.jp>wrote:
>
> >
> > FYI: Disruptive IP Networking with Intel DPDK on Linux
> > http://slidesha.re/SeVFZo
> >
> >
> > On Tue, 28 May 2013 11:26:30 +0900
> > Shinae Woo <shinae2012@gmail.com> wrote:
> >
> > > Hello, all.
> > >
> > > I play the dpdk-1.2.3r1 with examples.
> > >
> > > But I can not achieve the line-rate packet receive performance,
> > > and the performance is not scale with multiple ports.
> > >
> > > For example, in example l2fwd, I have tested two cases with 2 ports, and
> > 4
> > > ports,
> > > using belowed command line each
> > >
> > > ./build/l2fwd -cf -n3 -- -p3
> > > ./build/l2fwd -cf -n3 -- -pf
> > >
> > > But both cases, the aggregated performance are not scale.
> > >
> > > == experiments environments ==
> > > - Two Intel 82599 NICs (total 4 ports)
> > > - Intel Xeon X5690 @ 3.47GHz * 2 (total 12 cores)
> > > - 1024 * 2MB hugepages
> > > - Linux 2.6.38-15-server
> > > - Each ports receiving 10Gbps of traffic of 64 bytes packets, 14.88Mpps.
> > >
> > > *1. Packet forwarding performance*
> > >
> > > In 2 ports case, receive performance is 13Mpps,
> > > In 4 ports case, not 26Mbps, only 16.8Mpps.
> > >
> > > Port statistics ====================================
> > > Statistics for port 0 ------------------------------
> > > Packets sent: 4292256
> > > Packets received: 6517396
> > > Packets dropped: 2224776
> > > Statistics for port 1 ------------------------------
> > > Packets sent: 4291840
> > > Packets received: 6517044
> > > Packets dropped: 2225556
> > > Aggregate statistics ===============================
> > > Total packets sent: 8584128
> > > Total packets received: 13034472
> > > Total packets dropped: 4450332
> > > ====================================================
> > >
> > > Port statistics ====================================
> > > Statistics for port 0 ------------------------------
> > > Packets sent: 1784064
> > > Packets received: 2632700
> > > Packets dropped: 848128
> > > Statistics for port 1 ------------------------------
> > > Packets sent: 1784104
> > > Packets received: 2632196
> > > Packets dropped: 848596
> > > Statistics for port 2 ------------------------------
> > > Packets sent: 3587616
> > > Packets received: 5816344
> > > Packets dropped: 2200176
> > > Statistics for port 3 ------------------------------
> > > Packets sent: 3587712
> > > Packets received: 5787848
> > > Packets dropped: 2228684
> > > Aggregate statistics ===============================
> > > Total packets sent: 10743560
> > > Total packets received: 16869152
> > > Total packets dropped: 6125608
> > > ====================================================
> > >
> > > *2. Packet receiving performance*
> > > I fix the codes for only receiving packets (not forwarding),
> > > the performance is still not scalable as each 13.3Mpps, 18Mpps.
> > >
> > > Port statistics ====================================
> > > Statistics for port 0 ------------------------------
> > > Packets sent: 0
> > > Packets received: 6678860
> > > Packets dropped: 0
> > > Statistics for port 1 ------------------------------
> > > Packets sent: 0
> > > Packets received: 6646120
> > > Packets dropped: 0
> > > Aggregate statistics ===============================
> > > Total packets sent: 0
> > > Total packets received: 13325012
> > > Total packets dropped: 0
> > > ====================================================
> > >
> > > Port statistics ====================================
> > > Statistics for port 0 ------------------------------
> > > Packets sent: 0
> > > Packets received: 3129624
> > > Packets dropped: 0
> > > Statistics for port 1 ------------------------------
> > > Packets sent: 0
> > > Packets received: 3131292
> > > Packets dropped: 0
> > > Statistics for port 2 ------------------------------
> > > Packets sent: 0
> > > Packets received: 6260908
> > > Packets dropped: 0
> > > Statistics for port 3 ------------------------------
> > > Packets sent: 0
> > > Packets received: 6238764
> > > Packets dropped: 0
> > > Aggregate statistics ===============================
> > > Total packets sent: 0
> > > Total packets received: 18760640
> > > Total packets dropped: 0
> > > ====================================================
> > >
> > > The question is that
> > > 1. How I can achieve each port receiving full 14.88Mpps ?
> > > What might be the bottleneck in current environment?
> > > 2. Why the performance using multiple ports is not scale?
> > > I guess doubling ports shows the doubling the receiving performance,
> > > but it shows not. I am curious about what is limiting the packet
> > > receivng performance.
> > >
> > > Thanks,
> > > Shinae
> >
> > --
> > SAKURA Internet Inc. / Senior Researcher
> > Naoto MATSUMOTO <n-matsumoto@sakura.ad.jp>
> > SAKURA Internet Research Center <http://research.sakura.ad.jp/>
> >
> >
--
SAKURA Internet Inc. / Senior Researcher
Naoto MATSUMOTO <n-matsumoto@sakura.ad.jp>
SAKURA Internet Research Center <http://research.sakura.ad.jp/>
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2013-05-29 3:09 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-05-28 2:26 [dpdk-dev] Performances are not scale with multiple ports Shinae Woo
2013-05-28 2:30 ` Naoto MATSUMOTO
2013-05-28 3:05 ` Shinae Woo
2013-05-28 3:15 ` Emre Eraltan
2013-05-28 3:29 ` Stephen Hemminger
2013-05-28 4:00 ` Shinae Woo
2013-05-29 3:09 ` Naoto MATSUMOTO
2013-05-28 9:22 ` Thomas Monjalon
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).