From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by dpdk.org (Postfix) with ESMTP id B825868E8 for ; Wed, 12 Feb 2014 12:48:04 +0100 (CET) Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga101.fm.intel.com with ESMTP; 12 Feb 2014 03:49:27 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.95,832,1384329600"; d="scan'208";a="480206148" Received: from irsmsx103.ger.corp.intel.com ([163.33.3.157]) by fmsmga002.fm.intel.com with ESMTP; 12 Feb 2014 03:49:26 -0800 Received: from irsmsx151.ger.corp.intel.com (163.33.192.59) by IRSMSX103.ger.corp.intel.com (163.33.3.157) with Microsoft SMTP Server (TLS) id 14.3.123.3; Wed, 12 Feb 2014 11:49:24 +0000 Received: from irsmsx103.ger.corp.intel.com ([169.254.3.84]) by IRSMSX151.ger.corp.intel.com ([169.254.4.192]) with mapi id 14.03.0123.003; Wed, 12 Feb 2014 11:49:23 +0000 From: "Richardson, Bruce" To: Prashant Upadhyaya , "dev@dpdk.org" Thread-Topic: NUMA CPU Sockets and DPDK Thread-Index: Ac8n47elgPfPBMgcTQqNl7LBk7WTrwAA2p2g Date: Wed, 12 Feb 2014 11:49:23 +0000 Message-ID: <59AF69C657FD0841A61C55336867B5B01A99B166@IRSMSX103.ger.corp.intel.com> References: In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [163.33.239.181] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] NUMA CPU Sockets and DPDK X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Feb 2014 11:48:05 -0000 >=20 > What has been your experience of using DPDK based app's in NUMA mode > with multiple sockets where some cores are present on one socket and > other cores on some other socket. >=20 > I am migrating my application from one intel machine with 8 cores, all in > one socket to a 32 core machine where 16 cores are in one socket and 16 > other cores in the second socket. > My core 0 does all initialization for mbuf's, nic ports, queues etc. and = uses > SOCKET_ID_ANY for socket related parameters. It is recommended that you decide ahead of time on what cores on what numa = socket different parts of your application are going to run, and then set u= p your objects in memory appropriately. SOCKET_ID_ANY should only be used t= o allocate items that are not for use in the data-path and for which you th= erefore don't care about access time. Any objects for rings or mempools sho= uld be created by specifying the correct socket to allocate the memory on. = If you are working using two sockets, in some cases you may want to duplica= te your data structures, for example, use two memory pools - one on each so= cket - instead of one, so that all data access is local. >=20 > The usecase works, but I think I am running into performance issues on th= e > 32 core machine. > The lscpu output on my 32 core machine shows the following - NUMA > node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30 > NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31 > I am using core 1 to lift all the data from a single queue of an 82599EB = port > and I see that the cpu utilization for this core 1 is way too high even f= or > lifting traffic of 1 Gbps with packet size of 650 bytes. How are you measuring the cpu utilization, because when using the Intel DPD= K in most cases your cpu utilization will always be 100% as you are constan= tly polling? Therefore actual cpu headroom can be hard to judge at times. Another thing to consider is the numa nodes to which your NICs are connecte= d. You can check using the rte_eth_dev_socket_id() what numa socket your NI= C is connected to - assuming a modern platform where the PCI connects strai= ght to the CPUs. Whatever numa node that is connected to, you want to run t= he code for polling the NIC RX queues on that numa node, and do all packet = transmission using cores on that NUMA node. =20 >=20 > In general, does one need to be careful in working with multiple sockets = and > so forth, any comments would be helpful. =20 In general, yes, you need to be a bit more careful, but the basic rules as = outlined above should give you a good start.