From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f45.google.com (mail-pa0-f45.google.com [209.85.220.45]) by dpdk.org (Postfix) with ESMTP id EB5F830E for ; Tue, 28 May 2013 05:05:44 +0200 (CEST) Received: by mail-pa0-f45.google.com with SMTP id tj12so2443736pac.32 for ; Mon, 27 May 2013 20:05:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=kUGtUIICq+j0eLcQKWv6NuCs3fWUFy6Y380OHLjUxs4=; b=wYOU4Erekq/F+lkcN5loWQSUzhWr3QbAEx1+/wKXu/RLRmY/+MSVyReTdqZWtz/jlO JHUzVbPK0hSA9zdmAYcajwk3hJoPuaOYdNK2jLCtGucJ6X64tXqC6B2nFSV0ICnjb5Dg QxKZQrblCayM77pvR5jaze0W4PmTiU+ZbO+IDe3qSxHY421AaaKdpttSAftTPPzBBa3v pBJCEFfDT4BybjgzVU7MK5LWK9K0XWU6RSXJQfMxQsNhk7TuIuHSFO2qGtHPOZE3LaVu Vvfw/5KZgH8WVKN9OQ3YPbYvkAw8yFfw2jggaER16vxgLdIUzuSHctJaMWJZKen0L2DO onYg== MIME-Version: 1.0 X-Received: by 10.67.13.199 with SMTP id fa7mr32870154pad.185.1369710350110; Mon, 27 May 2013 20:05:50 -0700 (PDT) Received: by 10.66.155.170 with HTTP; Mon, 27 May 2013 20:05:50 -0700 (PDT) In-Reply-To: <20130528113005.5E6D.C42C3789@sakura.ad.jp> References: <20130528113005.5E6D.C42C3789@sakura.ad.jp> Date: Tue, 28 May 2013 12:05:50 +0900 Message-ID: From: Shinae Woo To: Naoto MATSUMOTO Content-Type: multipart/alternative; boundary=e89a8ffbaf09dd12d304ddbe8aa7 Cc: dev@dpdk.org Subject: Re: [dpdk-dev] Performances are not scale with multiple ports X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 03:05:45 -0000 --e89a8ffbaf09dd12d304ddbe8aa7 Content-Type: text/plain; charset=ISO-8859-1 Thanks for sharing Naoto. So in your experiments, the forwarding performance still does not reach the line rate. Your perf record shows that the cpu spend most of the time in polling for receiving packets, and no other heavy operation. Even though the application polling packets in its best, the forwarder miss some packets from elsewhere from the application-side. The dpdk document shows that 160Mpps forwarding performance in 2 sockets, but I can only reach the 13 Mpps in 2 ports. Even doubling the number of ports to 4 ports, the performance is still less than 17Mpps. I want to know where is the bottleneck lies in my environments, or how I can reprocuce the same performance as the dpdk published. Thank you, Shinae On Tue, May 28, 2013 at 11:30 AM, Naoto MATSUMOTO wrote: > > FYI: Disruptive IP Networking with Intel DPDK on Linux > http://slidesha.re/SeVFZo > > > On Tue, 28 May 2013 11:26:30 +0900 > Shinae Woo wrote: > > > Hello, all. > > > > I play the dpdk-1.2.3r1 with examples. > > > > But I can not achieve the line-rate packet receive performance, > > and the performance is not scale with multiple ports. > > > > For example, in example l2fwd, I have tested two cases with 2 ports, and > 4 > > ports, > > using belowed command line each > > > > ./build/l2fwd -cf -n3 -- -p3 > > ./build/l2fwd -cf -n3 -- -pf > > > > But both cases, the aggregated performance are not scale. > > > > == experiments environments == > > - Two Intel 82599 NICs (total 4 ports) > > - Intel Xeon X5690 @ 3.47GHz * 2 (total 12 cores) > > - 1024 * 2MB hugepages > > - Linux 2.6.38-15-server > > - Each ports receiving 10Gbps of traffic of 64 bytes packets, 14.88Mpps. > > > > *1. Packet forwarding performance* > > > > In 2 ports case, receive performance is 13Mpps, > > In 4 ports case, not 26Mbps, only 16.8Mpps. > > > > Port statistics ==================================== > > Statistics for port 0 ------------------------------ > > Packets sent: 4292256 > > Packets received: 6517396 > > Packets dropped: 2224776 > > Statistics for port 1 ------------------------------ > > Packets sent: 4291840 > > Packets received: 6517044 > > Packets dropped: 2225556 > > Aggregate statistics =============================== > > Total packets sent: 8584128 > > Total packets received: 13034472 > > Total packets dropped: 4450332 > > ==================================================== > > > > Port statistics ==================================== > > Statistics for port 0 ------------------------------ > > Packets sent: 1784064 > > Packets received: 2632700 > > Packets dropped: 848128 > > Statistics for port 1 ------------------------------ > > Packets sent: 1784104 > > Packets received: 2632196 > > Packets dropped: 848596 > > Statistics for port 2 ------------------------------ > > Packets sent: 3587616 > > Packets received: 5816344 > > Packets dropped: 2200176 > > Statistics for port 3 ------------------------------ > > Packets sent: 3587712 > > Packets received: 5787848 > > Packets dropped: 2228684 > > Aggregate statistics =============================== > > Total packets sent: 10743560 > > Total packets received: 16869152 > > Total packets dropped: 6125608 > > ==================================================== > > > > *2. Packet receiving performance* > > I fix the codes for only receiving packets (not forwarding), > > the performance is still not scalable as each 13.3Mpps, 18Mpps. > > > > Port statistics ==================================== > > Statistics for port 0 ------------------------------ > > Packets sent: 0 > > Packets received: 6678860 > > Packets dropped: 0 > > Statistics for port 1 ------------------------------ > > Packets sent: 0 > > Packets received: 6646120 > > Packets dropped: 0 > > Aggregate statistics =============================== > > Total packets sent: 0 > > Total packets received: 13325012 > > Total packets dropped: 0 > > ==================================================== > > > > Port statistics ==================================== > > Statistics for port 0 ------------------------------ > > Packets sent: 0 > > Packets received: 3129624 > > Packets dropped: 0 > > Statistics for port 1 ------------------------------ > > Packets sent: 0 > > Packets received: 3131292 > > Packets dropped: 0 > > Statistics for port 2 ------------------------------ > > Packets sent: 0 > > Packets received: 6260908 > > Packets dropped: 0 > > Statistics for port 3 ------------------------------ > > Packets sent: 0 > > Packets received: 6238764 > > Packets dropped: 0 > > Aggregate statistics =============================== > > Total packets sent: 0 > > Total packets received: 18760640 > > Total packets dropped: 0 > > ==================================================== > > > > The question is that > > 1. How I can achieve each port receiving full 14.88Mpps ? > > What might be the bottleneck in current environment? > > 2. Why the performance using multiple ports is not scale? > > I guess doubling ports shows the doubling the receiving performance, > > but it shows not. I am curious about what is limiting the packet > > receivng performance. > > > > Thanks, > > Shinae > > -- > SAKURA Internet Inc. / Senior Researcher > Naoto MATSUMOTO > SAKURA Internet Research Center > > --e89a8ffbaf09dd12d304ddbe8aa7 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Thanks for sharing Naoto.

So i= n your experiments, the forwarding performance still does not reach the lin= e rate.

Your perf record shows that the cpu spend most of the = time in polling for receiving packets,
and no other heavy operation.
Even though the appl= ication polling packets in its best,
the forwarder miss some packets fro= m elsewhere from the application-side.

The dpdk document = shows that 160Mpps forwarding performance in 2 sockets,
but I can only reach the 13 Mpps in 2 ports.
Even doubling t= he number of ports to 4 ports, the performance is still less than 17Mpps.
I want to know where is the bottleneck lies in my environm= ents, or
how I can reprocuce the same performance as the dpdk published.<= br>
Thank you,
Shinae

<= /div>


On Tue, = May 28, 2013 at 11:30 AM, Naoto MATSUMOTO <n-matsumoto@sakura.ad.jp= > wrote:

FYI: Disruptive IP Networking with Intel DPDK on Linux
http://slidesha.re/= SeVFZo


On Tue, 28 May 2013 11:26:30 +0900
Shinae Woo <shinae2012@gmail.com= > wrote:

> Hello, all.
>
> I play the dpdk-1.2.3r1 with examples.
>
> But I can not achieve the line-rate packet receive performance,
> and the performance is not scale with multiple ports.
>
> For example, in example l2fwd, I have tested two cases with 2 ports, a= nd 4
> ports,
> using belowed command line each
>
> ./build/l2fwd -cf -n3 -- -p3
> ./build/l2fwd -cf -n3 -- -pf
>
> But both cases, the aggregated performance are not scale.
>
> =3D=3D experiments environments =3D=3D
> - Two Intel 82599 NICs (total 4 ports)
> - Intel Xeon X5690 =A0@ 3.47GHz * 2 (total 12 cores)
> - 1024 * 2MB hugepages
> - Linux 2.6.38-15-server
> - Each ports receiving 10Gbps of traffic of 64 bytes packets, 14.88Mpp= s.
>
> *1. Packet forwarding performance*
>
> In 2 ports case, =A0receive performance is 13Mpps,
> In 4 ports case, =A0not 26Mbps, only 16.8Mpps.
>
> Port statistics =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Statistics for port 0 ------------------------------
> Packets sent: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A04292256
> Packets received: =A0 =A0 =A0 =A0 =A0 =A0 =A06517396
> Packets dropped: =A0 =A0 =A0 =A0 =A0 =A0 =A0 2224776
> Statistics for port 1 ------------------------------
> Packets sent: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A04291840
> Packets received: =A0 =A0 =A0 =A0 =A0 =A0 =A06517044
> Packets dropped: =A0 =A0 =A0 =A0 =A0 =A0 =A0 2225556
> Aggregate statistics =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Total packets sent: =A0 =A0 =A0 =A0 =A0 =A08584128
> Total packets received: =A0 =A0 =A0 13034472
> Total packets dropped: =A0 =A0 =A0 =A0 4450332
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D
>
> Port statistics =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Statistics for port 0 ------------------------------
> Packets sent: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A01784064
> Packets received: =A0 =A0 =A0 =A0 =A0 =A0 =A02632700
> Packets dropped: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0848128
> Statistics for port 1 ------------------------------
> Packets sent: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A01784104
> Packets received: =A0 =A0 =A0 =A0 =A0 =A0 =A02632196
> Packets dropped: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0848596
> Statistics for port 2 ------------------------------
> Packets sent: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A03587616
> Packets received: =A0 =A0 =A0 =A0 =A0 =A0 =A05816344
> Packets dropped: =A0 =A0 =A0 =A0 =A0 =A0 =A0 2200176
> Statistics for port 3 ------------------------------
> Packets sent: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A03587712
> Packets received: =A0 =A0 =A0 =A0 =A0 =A0 =A05787848
> Packets dropped: =A0 =A0 =A0 =A0 =A0 =A0 =A0 2228684
> Aggregate statistics =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Total packets sent: =A0 =A0 =A0 =A0 =A0 10743560
> Total packets received: =A0 =A0 =A0 16869152
> Total packets dropped: =A0 =A0 =A0 =A0 6125608
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D
>
> *2. Packet receiving performance*
> I fix the codes for only recei= ving packets (not forwarding),
> the performance is still not scalable as each 13.3Mpps, 18Mpps.
>
> Port statistics =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Statistics for port 0 ------------------------------
> Packets sent: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00
> Packets received: =A0 =A0 =A0 =A0 =A0 =A0 =A06678860
> Packets dropped: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
> Statistics for port 1 ------------------------------
> Packets sent: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00
> Packets received: =A0 =A0 =A0 =A0 =A0 =A0 =A06646120
> Packets dropped: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
> Aggregate statistics =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Total packets sent: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00
> Total packets received: =A0 =A0 =A0 13325012
> Total packets dropped: =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D
>
> Port statistics =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Statistics for port 0 ------------------------------
> Packets sent: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00
> Packets received: =A0 =A0 =A0 =A0 =A0 =A0 =A03129624
> Packets dropped: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
> Statistics for port 1 ------------------------------
> Packets sent: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00
> Packets received: =A0 =A0 =A0 =A0 =A0 =A0 =A03131292
> Packets dropped: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
> Statistics for port 2 ------------------------------
> Packets sent: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00
> Packets received: =A0 =A0 =A0 =A0 =A0 =A0 =A06260908
> Packets dropped: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
> Statistics for port 3 ------------------------------
> Packets sent: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00
> Packets received: =A0 =A0 =A0 =A0 =A0 =A0 =A06238764
> Packets dropped: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
> Aggregate statistics =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Total packets sent: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00
> Total packets received: =A0 =A0 =A0 18760640
> Total packets dropped: =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D
>
> The question is that
> 1. How I can achieve each port receiving full 14.88Mpps ?
> =A0 =A0 What might be the bottleneck in current environment?
> 2. Why the performance using multiple ports is not scale?
> =A0 =A0 I guess doubling ports shows the doubling the receiving perfor= mance,
> =A0 =A0 but it shows not. I am curious about what is limiting the pack= et
> receivng performance.
>
> Thanks,
> Shinae

--
SAKURA Internet Inc. / Senior Researcher
Naoto MATSUMOTO <n-matsumoto= @sakura.ad.jp>
SAKURA Internet Research Center <http://research.sakura.ad.jp/>


--e89a8ffbaf09dd12d304ddbe8aa7--