From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id B81ABB58D for ; Mon, 16 Feb 2015 02:55:23 +0100 (CET) Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP; 15 Feb 2015 17:47:51 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.09,584,1418112000"; d="scan'208";a="455064716" Received: from pgsmsx103.gar.corp.intel.com ([10.221.44.82]) by FMSMGA003.fm.intel.com with ESMTP; 15 Feb 2015 17:40:18 -0800 Received: from shsmsx103.ccr.corp.intel.com (10.239.4.69) by PGSMSX103.gar.corp.intel.com (10.221.44.82) with Microsoft SMTP Server (TLS) id 14.3.195.1; Mon, 16 Feb 2015 09:55:10 +0800 Received: from shsmsx102.ccr.corp.intel.com ([169.254.2.62]) by SHSMSX103.ccr.corp.intel.com ([169.254.4.197]) with mapi id 14.03.0195.001; Mon, 16 Feb 2015 09:55:09 +0800 From: "Liang, Cunming" To: Neil Horman Thread-Topic: [dpdk-dev] [PATCH v6 12/19] malloc: fix the issue of SOCKET_ID_ANY Thread-Index: AQHQRy3bkgrqC7N8BUG8tbfwqJ1xqpzuWICAgAKFUACAAF+xAIABPW1g Date: Mon, 16 Feb 2015 01:55:09 +0000 Message-ID: References: <1423728996-3004-1-git-send-email-cunming.liang@intel.com> <1423791501-1555-1-git-send-email-cunming.liang@intel.com> <1423791501-1555-13-git-send-email-cunming.liang@intel.com> <20150213175708.GB17402@neilslaptop.think-freely.org> <20150215140917.GA6302@neilslaptop.think-freely.org> In-Reply-To: <20150215140917.GA6302@neilslaptop.think-freely.org> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] [PATCH v6 12/19] malloc: fix the issue of SOCKET_ID_ANY X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Feb 2015 01:55:24 -0000 Hi, > -----Original Message----- > From: Neil Horman [mailto:nhorman@tuxdriver.com] > Sent: Sunday, February 15, 2015 10:09 PM > To: Liang, Cunming > Cc: dev@dpdk.org > Subject: Re: [dpdk-dev] [PATCH v6 12/19] malloc: fix the issue of SOCKET_= ID_ANY >=20 > On Sun, Feb 15, 2015 at 12:43:03AM +0000, Liang, Cunming wrote: > > Hi, > > > > > -----Original Message----- > > > From: Neil Horman [mailto:nhorman@tuxdriver.com] > > > Sent: Saturday, February 14, 2015 1:57 AM > > > To: Liang, Cunming > > > Cc: dev@dpdk.org > > > Subject: Re: [dpdk-dev] [PATCH v6 12/19] malloc: fix the issue of > SOCKET_ID_ANY > > > > > > On Fri, Feb 13, 2015 at 09:38:14AM +0800, Cunming Liang wrote: > > > > Add check for rte_socket_id(), avoid get unexpected return like (-1= ). > > > > > > > > Signed-off-by: Cunming Liang > > > > --- > > > > lib/librte_malloc/malloc_heap.h | 7 ++++++- > > > > 1 file changed, 6 insertions(+), 1 deletion(-) > > > > > > > > diff --git a/lib/librte_malloc/malloc_heap.h > b/lib/librte_malloc/malloc_heap.h > > > > index b4aec45..a47136d 100644 > > > > --- a/lib/librte_malloc/malloc_heap.h > > > > +++ b/lib/librte_malloc/malloc_heap.h > > > > @@ -44,7 +44,12 @@ extern "C" { > > > > static inline unsigned > > > > malloc_get_numa_socket(void) > > > > { > > > > - return rte_socket_id(); > > > > + unsigned socket_id =3D rte_socket_id(); > > > > + > > > > + if (socket_id =3D=3D (unsigned)SOCKET_ID_ANY) > > > > + return 0; > > > > + > > > > + return socket_id; > > > Why is -1 unexpected? Isn't it reasonable to assume that some memory= is > > > equidistant from all cpu numa nodes? > > [LCM] One piece of memory will be whole allocated from one specific NUM= A > node. But won't be like some part from one and the other part from anothe= r. > > If no specific NUMA node assigned(SOCKET_ID_ANY/-1), it firstly asks fo= r the > current NUMA node where current core belongs to. > > 'malloc_get_numa_socket()' is called on that time. When the time 1:1 > thread/core mapping is assumed and the default value is 0, it always will= return a > none (-1) value. > > Now rte_socket_id() may return -1 in the case the pthread runs on multi= -cores > which are not belongs to one NUMA node, or in the case _socket_id is not = yet > assigned and the default value is (-1). So if current _socket_id is -1, t= hen just pick > up the first node as the candidate. Probably I shall add more comments fo= r this. > > > > Ok, but doesn't that provide an abnormal bias for node 0? I was thinking= it > might be better to be honest with the application so that it can choose a= node > according to its own policy. [LCM] Personally I like the idea grant application to make the decision. Either add a simple default configure or defines the more flexible policy o= f SOCKET_ID_ANY like 1) use the assigned default socket_id; 2) use current socket_id, if fail go= to 1); 3) (weight)round robin across the malloc_heaps; 4) use current socket_id, i= f fail goto 3); and etc. But on another side, the well-tuned application are usually NUMA friendly. = Instead of using SOCKET_ID_ANY, it most often assigned the expected socket_= id. Except getting the real current valid socket_id, The policy won't help on t= he affinity but mainly helps on the memory utilization. I guess the worry comes from the case, after lots of memory allocation happ= ens on socket 0, a new memzone_reserve fails when it definitely has to do i= t on socket 0 as well. In this case, either changes the default NUMA node or balance the allocatio= n won't solve the problem, but respite it happening. It's because the explicit assignment allocation (memzone_reserve, malloc wi= th a specified socket_id) may not average balanced. In reverse, if reserving all necessary memzone first, even malloc fails on = default socket, it will try to get allocation from other NUMA node. I think it's out of the scope of this patch series. On current moment, usin= g the simplest way taking node 0 as default socket_id is not bad. For more, we can post on separate patch and involved more on the discussion= . Thanks. >=20 > Neil >=20 > > > Neil > > > > > > > } > > > > > > > > void * > > > > -- > > > > 1.8.1.4 > > > > > > > > > >