From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id 6B1121F3 for ; Sun, 22 Sep 2013 19:41:05 +0200 (CEST) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga101.jf.intel.com with ESMTP; 22 Sep 2013 10:41:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.90,957,1371106800"; d="scan'208";a="399390725" Received: from orsmsx105.amr.corp.intel.com ([10.22.225.132]) by fmsmga001.fm.intel.com with ESMTP; 22 Sep 2013 10:41:43 -0700 Received: from orsmsx102.amr.corp.intel.com ([169.254.1.143]) by ORSMSX105.amr.corp.intel.com ([169.254.4.17]) with mapi id 14.03.0123.003; Sun, 22 Sep 2013 10:41:43 -0700 From: "Venkatesan, Venky" To: Chris Pappas , "dev@dpdk.org" Thread-Topic: [dpdk-dev] Question regarding throughput number with DPDK l2fwd with Wind River System's pktgen Thread-Index: AQHOt6CP+d9qTbuX6UOS/ygVikeOoZnSAtCA Date: Sun, 22 Sep 2013 17:41:43 +0000 Message-ID: <1FD9B82B8BF2CF418D9A1000154491D973F4DF62@ORSMSX102.amr.corp.intel.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.22.254.139] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] Question regarding throughput number with DPDK l2fwd with Wind River System's pktgen X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Sep 2013 17:41:05 -0000 Chris,=20 The numbers you are getting are correct. :) Practically speaking, most motherboards pin out between 4 and 5 x8 slots to= every CPU socket. At PCI-E Gen 2 speeds (5 GT/s), each slot is capable of = carrying 20 Gb/s of traffic (limited to ~16 Gb/s of 64B packets). I would = have expected the 64-byte traffic capacity to be a bit higher than 80 Gb/s= , but either way the numbers you are achieving are well within the capabili= ty of the system if you are careful about pinning cores to ports, which you= seem to be doing. QPI is not a limiter either for the amount of traffic yo= u are generating currently.=20 Regards, -Venky -----Original Message----- From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Chris Pappas Sent: Sunday, September 22, 2013 7:32 AM To: dev@dpdk.org Subject: [dpdk-dev] Question regarding throughput number with DPDK l2fwd wi= th Wind River System's pktgen Hi, We have a question about the performance numbers we are getting measured th= rough the pktgen application provided by Wind River Systems. The current se= tup is the following: We have two machines, each equipped with 6 dual-port 10 GbE NICs (with a to= tal of 12 ports). Machine 0 runs DPDK L2FWD code, and Machine 1 runs Wind R= iver System's pktgen. L2FWD is modified to forward the incoming packets to = other statically assigned output port. Our machines have two Intel Xeon E5-2600 CPUs connected via QPI, and has tw= o riser slots each having three 10Gbps NICs. Two NICS in riser slot 1 (NIC0 and NIC1) is connected to CPU 1 via PCIe Gen3, while the remaining NIC2 is connected to CPU2 also via PCIe Gen3. In riser slot 2, all NICs (NI= Cs 3,4, and 5) are connected to CPU2 via PCIe Gen3. We were careful to assi= gn the NIC ports to cores of CPU sockets that have direct physical connecti= on to achieve max performance. With this setup, we are getting 120 Gbps throughput measured by pktgen with= packet size 1500 Bytes. For 64 Byte packets, we are getting around 80 Gbps= . Do these performance numbers make sense? We are reading related papers in t= his domain, and seems like our numbers are unusually high. We did our theor= etical calculation and find that it should theoretically be possible becaus= e it does not hit the PCIe bandwidth or our machine, nor does it exceed QPI= bandwidth when packets are forwarded over the NUMA node. Can you share you= r thoughts / experience with this? Thank you, Chris