From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 6306868C3 for ; Wed, 17 Sep 2014 06:12:45 +0200 (CEST) Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP; 16 Sep 2014 21:12:10 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.04,538,1406617200"; d="scan'208";a="603960982" Received: from fmsmsx104.amr.corp.intel.com ([10.18.124.202]) by orsmga002.jf.intel.com with ESMTP; 16 Sep 2014 21:18:19 -0700 Received: from fmsmsx101.amr.corp.intel.com (10.18.124.199) by fmsmsx104.amr.corp.intel.com (10.18.124.202) with Microsoft SMTP Server (TLS) id 14.3.195.1; Tue, 16 Sep 2014 21:18:19 -0700 Received: from shsmsx152.ccr.corp.intel.com (10.239.6.52) by fmsmsx101.amr.corp.intel.com (10.18.124.199) with Microsoft SMTP Server (TLS) id 14.3.195.1; Tue, 16 Sep 2014 21:18:19 -0700 Received: from shsmsx104.ccr.corp.intel.com ([169.254.5.230]) by SHSMSX152.ccr.corp.intel.com ([169.254.6.190]) with mapi id 14.03.0195.001; Wed, 17 Sep 2014 12:18:17 +0800 From: "Zhang, Helin" To: "dev@dpdk.org" Thread-Topic: i40e: Steps and required configurations of how to achieve the best performance! Thread-Index: Ac/SLgkTpOKQDD2QRa+/Ir9xVuotwQ== Date: Wed, 17 Sep 2014 04:18:16 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: [dpdk-dev] i40e: Steps and required configurations of how to achieve the best performance! X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Sep 2014 04:12:46 -0000 Hi all As a lot of special configurations are needed for achieving the best perfor= mance on DPDK, and we are asked by a lot of guys here, I'd like to share wi= th all of you about the steps and required configurations of how to achieve= the best performance of i40e. I am trying to list all and what I am using = and have done to get the best performance on i40e, though something might s= till be missed. So, supplements are welcome! Please do not ask me the real performance numbers, as I am not the official= way to publish the real numbers! 1. Hardware Platform: -- Intel(r) Haswell(r) server -- Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz -- Big enough memory, e.g. 32G, deployed on different memory channels -- Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ / Intel Corp= oration Ethernet Controller X710 for 10GbE SFP+ -- Make sure it is B0 hardware -- Update the firmware to at least 4.2.4 or newer, as they may have impac= t on performance -- Make sure inserting the NICs to the correct PCIe Gen3 slot, as PCIe Gen2= cannot provide enough bandwidth 2. Software Platform: -- Fedora 20 with updating kernel to 3.15.10, this is what I am using for t= esting. -- GCC 4.8.2 -- Kernle 3.15.10 -- DPDK 1.7.0 or later version 3. BIOS Configurations: -- Enhanced Intel Speedstep: Disabled -- Processor C3: Disabled -- Processor C6: Disabled -- Hyper-Threading: Enabled -- Intel VT-d: Disable -- MLC Streamer: Enabled -- MLC Spatial Prefetcher: Enabled -- DCU Data Prefetcher: Enabled -- DCU Instruction Prefetcher: Enabled -- Direct Cache Access (DCA): Enabled -- CPU Power and Performance Policy: Performance -- Memory Power Optimization: Performance -- Memory RAS and Performance Configuratioin -> NUMA Optimized: Enabled -- *Extended Tag: Enabled Note that 'Extended Tag' might not be seen in some BIOS, see 'compile setti= ngs' for doing that at runtime. 4. Grub Configurations: -- Set huge pages, e.g. 'default_hugepagesz=3D1G hugepagesz=3D1G hugepages= =3D8' -- Isolate cpu cores to be used for rx/tx from Linux scheduling, e.g. ' iso= lcpus=3D2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17' 5. Compile Settings: -- Change below configuration items in config files CONFIG_RTE_PCI_CONFIG=3Dy CONFIG_RTE_PCI_EXTENDED_TAG=3D"on" CONFIG_RTE_LIBRTE_I40E_16BYTE_RX_DESC=3Dy 6. Application Command Line Parameters: -- For 40G ports, make sure to use two ports on two different cards, as it = cannot provide 80G bps on a single PCI gen3 slot. -- Make sure the lcores to be used are on the CPU socket which the NIC PCI = slot directly connected to. -- Run tools/ cpu_layout.py to get the lcore/socket topology. -- use 'lspci -q | grep Eth' to check the PCI address of the NIC ports -- e.g. for PCI address of 8x:00.x, it should use lcores on socket 1, oth= erwise, use lcores on socket 0 -- Make sure to use 2, 4 or more queues for l3fwd or testpmd to get better = performance, 4 queues might be enough -- e.g. run l3fwd on two ports on two different cards, with using two queue= s and two lcores per port ./l3fwd -c 0x3c0000 -n 4 -w 82:00.0 -w 85:00.0 -- -p 0x3 --config '(0,0,1= 8),(0,1,19),(1,0,20),(1,1,21)' -- e.g. run testpmd on two ports on two different cards, with using two que= ues and two lcores per port ./testpmd -c 0x3fc0001 -n 4 -w 82:00.0 -w 85:00.0 -- -i --coremask=3D0x3c= 0000 --nb-cores=3D4 --burst=3D32 --rxfreet=3D64 --txfreet=3D64 --txqflags= =3D0xf03 --mbcache=3D256 --rxq=3D2 --txq=3D2 --rss-ip 7. Stream Configurations on Packet Generator: -- Create a stream, e.g. on IXIA -- Set the Ethernet II type to 0x0800 -- Set the protocols to IPv4 -- Do not set any layer protocols, as I use IP rss -- Set correct destination IP address according to l3fwd-lpm/l3fwd-exact_= match routing table. -- Set the source IP to random, this is very important to make sure the p= acket will be received in multiple queues. -- Note that the desc MAC equals or not equals to the NIC port MAC may resu= lt in different performance numbers. -- Note that promiscuous mode is enabled or not may result in different per= formance numbers. Regards, Helin