From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E622FA0C50; Fri, 16 Jul 2021 16:25:04 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6F7BA4067B; Fri, 16 Jul 2021 16:25:04 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 5D8F240151 for ; Fri, 16 Jul 2021 16:25:02 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10046"; a="210549390" X-IronPort-AV: E=Sophos;i="5.84,245,1620716400"; d="scan'208";a="210549390" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jul 2021 07:25:01 -0700 X-IronPort-AV: E=Sophos;i="5.84,245,1620716400"; d="scan'208";a="413989170" Received: from bricha3-mobl.ger.corp.intel.com ([10.252.26.216]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-SHA; 16 Jul 2021 07:24:58 -0700 Date: Fri, 16 Jul 2021 15:24:55 +0100 From: Bruce Richardson To: Juraj =?utf-8?Q?Linke=C5=A1?= Cc: David Christensen , "thomas@monjalon.net" , "david.marchand@redhat.com" , "Honnappa.Nagarahalli@arm.com" , "Ruifeng.Wang@arm.com" , "ferruh.yigit@intel.com" , "jerinjacobk@gmail.com" , "dev@dpdk.org" Message-ID: References: <1618827522-31828-1-git-send-email-juraj.linkes@pantheon.tech> <1624964105-6525-1-git-send-email-juraj.linkes@pantheon.tech> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Subject: Re: [dpdk-dev] [PATCH v4] build: optional NUMA and cpu counts detection X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Fri, Jul 16, 2021 at 01:53:18PM +0000, Juraj Linkeš wrote: > > > > -----Original Message----- > > From: David Christensen > > Sent: Tuesday, July 6, 2021 8:11 PM > > To: Bruce Richardson ; Juraj Linkeš > > > > Cc: thomas@monjalon.net; david.marchand@redhat.com; > > Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com; > > ferruh.yigit@intel.com; jerinjacobk@gmail.com; dev@dpdk.org > > Subject: Re: [dpdk-dev] [PATCH v4] build: optional NUMA and cpu counts > > detection > > > > > > > > On 7/6/21 2:08 AM, Bruce Richardson wrote: > > > On Tue, Jul 06, 2021 at 08:56:37AM +0000, Juraj Linkeš wrote: > > >> > > >> > > >>> -----Original Message----- > > >>> From: Bruce Richardson > > >>> Sent: Tuesday, June 29, 2021 1:29 PM > > >>> To: Juraj Linkeš > > >>> Cc: thomas@monjalon.net; david.marchand@redhat.com; > > >>> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com; > > >>> ferruh.yigit@intel.com; jerinjacobk@gmail.com; dev@dpdk.org > > >>> Subject: Re: [PATCH v4] build: optional NUMA and cpu counts > > >>> detection > > >>> > > >>> On Tue, Jun 29, 2021 at 12:55:05PM +0200, Juraj Linkeš wrote: > > >>>> Add an option to automatically discover the host's numa and cpu > > >>>> counts and use those values for a non cross-build. > > >>>> Give users the option to override the per-arch default values or > > >>>> values from cross files by specifying them on the command line with > > >>>> -Dmax_lcores and -Dmax_numa_nodes. > > >>>> > > >>>> Signed-off-by: Juraj Linkeš > > >>>> Reviewed-by: Honnappa Nagarahalli > > >>>> --- > > >>> Two very minor suggestions inline below. > > >>> > > >>> Acked-by: Bruce Richardson > > >>> > > >>>> > > >>> > > >>>> +max_lcores = get_option('max_lcores') if max_lcores == 'auto' > > >>> > > >>> Rather than "auto", would "detect" be a clearer name for this option value? > > >>> > > >>> > > >>>> +option('max_lcores', type: 'string', value: 'default', description: > > >>>> + 'Set maximum number of cores/threads supported by EAL. The > > >>>> +default is different per-arch. Set to auto to detect the number of > > >>>> +cores on the > > >>> build machine.') option('max_numa_nodes', type: 'string', value: > > >>> 'default', > > >>> description: > > >>>> + 'Set highest NUMA node supported by EAL. The default is > > >>>> +different per-arch. Set to auto to detect the highest numa node on > > >>>> +the build machine.') > > >>> > > >>> I'd put the explicit values of "default" and "auto"(or "detect") in > > >>> quotes "" to make clear they are literal values. > > >>> > > >> > > >> Thanks, Bruce, I'll change it. I have one extra question now that I'm looking > > at the patch: > > >> What does subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False) return > > exactly? Is the the number of NUMA nodes (looks like it) or the highest NUMA > > node on the system (the highest number of all NUMA nodes)? I'm asking > > because of how NUMA works on P9: > > >> NUMA node0 CPU(s): 0-63 > > >> NUMA node8 CPU(s): 64-127 > > >> NUMA node252 CPU(s): > > >> NUMA node253 CPU(s): > > >> NUMA node254 CPU(s): > > >> NUMA node255 CPU(s): > > >> > > >> Here we need not just two NUMA nodes, but at least 9 (0-8). Linux and > > Windows should return the highest NUMA, not sure about FreeBSD. Or maybe > > we should return the highest NUMA on which there are actual CPUs? > > > > > > I'm not sure, and I think to be really sure we'd need it tested on a > > > P9 system. The help text for the sysctl node says "Number of physical > > > memory domains available", which would imply 2 in the case above. > > > [However, we also would need to find out how BSD numbers the domains, > > > too, as it's possible an OS could just call them 0 and 1, rather than > > > 0 and 8 if it wanted to.] > > > > > > In short, we'd need to test to be sure. Is FreeBSD on P9 a supported > > > config, and if so can the P9 maintainer perhaps help out with testing? > > > > Results of the v4 patch on an IBM AC922 P9 system with Linux: > > > > Can you get results from FreeBSD as well? > > > $ python3 get-numa-count.py > > 8 > > NUMA node0 CPU(s): 0-63 > > NUMA node8 CPU(s): 64-127 > > Is this the right number for your case, i.e. are you able to use both numa nodes when RTE_MAX_NUMA_NODES=8? > > Or maybe this is a question for Bruce or Thomas - what do we need to set in RTE_MAX_NUMA_NODES to be able to use all numa nodes on the system? The highest numa node number or that + 1? In linux, with 4 numa nodes, there will be node0-node3 under /sys/devices/system/node - do we need to set RTE_MAX_NUMA_NODES to 3 or 4? > 4, I believe.