From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 997C0A04B7; Wed, 14 Oct 2020 11:23:33 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0F7B01DD8C; Wed, 14 Oct 2020 11:22:29 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id 93FBA1DD06 for ; Wed, 14 Oct 2020 11:22:26 +0200 (CEST) IronPort-SDR: hTcZ2a7590QmMglWd0HWi4d3nfsQIOIpE6a44tiRRsNzRJgZtb8yIchl5Z7laFVyDjDXv4SgJA YTNOQH3xagWg== X-IronPort-AV: E=McAfee;i="6000,8403,9773"; a="145382969" X-IronPort-AV: E=Sophos;i="5.77,374,1596524400"; d="scan'208";a="145382969" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Oct 2020 02:22:24 -0700 IronPort-SDR: piMHAheqGtr22/Byn7puKpOWWdcsJ3eH+rmInWOJfrndacMYGy9iM0Y3W76IBqH62uXOvYO1NV OoDPE3Fcm3Kw== X-IronPort-AV: E=Sophos;i="5.77,374,1596524400"; d="scan'208";a="463816278" Received: from bricha3-mobl.ger.corp.intel.com ([10.254.145.91]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-SHA; 14 Oct 2020 02:22:19 -0700 Date: Wed, 14 Oct 2020 10:22:12 +0100 From: Bruce Richardson To: Juraj =?utf-8?Q?Linke=C5=A1?= Cc: "Ruifeng.Wang@arm.com" , "Honnappa.Nagarahalli@arm.com" , "Phil.Yang@arm.com" , "vcchunga@amazon.com" , "Dharmik.Thakkar@arm.com" , "jerinjacobk@gmail.com" , "hemant.agrawal@nxp.com" , "dev@dpdk.org" Message-ID: <20201014092212.GB1513@bricha3-MOBL.ger.corp.intel.com> References: <1600867161-15673-1-git-send-email-juraj.linkes@pantheon.tech> <1602600882-695-1-git-send-email-juraj.linkes@pantheon.tech> <1602600882-695-4-git-send-email-juraj.linkes@pantheon.tech> <20201013154757.GC1496@bricha3-MOBL.ger.corp.intel.com> <42b8d77b7d764e0789fb74ef2f9401d4@pantheon.tech> <20201014081610.GA1513@bricha3-MOBL.ger.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Subject: Re: [dpdk-dev] [RFC PATCH v2 3/6] build: automatic NUMA and cpu counts detection X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Wed, Oct 14, 2020 at 08:52:12AM +0000, Juraj Linkeš wrote: > > > > -----Original Message----- > > From: Bruce Richardson > > Sent: Wednesday, October 14, 2020 10:16 AM > > To: Juraj Linkeš > > Cc: Ruifeng.Wang@arm.com; Honnappa.Nagarahalli@arm.com; > > Phil.Yang@arm.com; vcchunga@amazon.com; Dharmik.Thakkar@arm.com; > > jerinjacobk@gmail.com; hemant.agrawal@nxp.com; dev@dpdk.org > > Subject: Re: [RFC PATCH v2 3/6] build: automatic NUMA and cpu counts > > detection > > > > On Wed, Oct 14, 2020 at 06:04:52AM +0000, Juraj Linkeš wrote: > > > > > > > > > > -----Original Message----- > > > > From: Bruce Richardson > > > > Sent: Tuesday, October 13, 2020 5:48 PM > > > > To: Juraj Linkeš > > > > Cc: Ruifeng.Wang@arm.com; Honnappa.Nagarahalli@arm.com; > > > > Phil.Yang@arm.com; vcchunga@amazon.com; Dharmik.Thakkar@arm.com; > > > > jerinjacobk@gmail.com; hemant.agrawal@nxp.com; dev@dpdk.org > > > > Subject: Re: [RFC PATCH v2 3/6] build: automatic NUMA and cpu counts > > > > detection > > > > > > > > On Tue, Oct 13, 2020 at 04:54:39PM +0200, Juraj Linkeš wrote: > > > > > The build machine's number of cpus and numa nodes vary, resulting > > > > > in mismatched counts of RTE_MAX_LCORE and RTE_MAX_NUMA_NODES > > for > > > > many > > > > > builds. Automatically discover the host's numa and cpu counts to > > > > > remove this mismatch for native builds. Use current defaults for cross > > builds. > > > > > Leave users the option to override both if the specify a non-zero > > > > > amount on the command line. > > > > > > > > > > Signed-off-by: Juraj Linkeš > > > > > --- > > > > > buildtools/get_cpu_count.py | 7 +++++++ > > > > > buildtools/get_numa_count.py | 17 +++++++++++++++++ > > > > > buildtools/meson.build | 2 ++ > > > > > config/meson.build | 20 ++++++++++++++++++-- > > > > > meson_options.txt | 8 ++++---- > > > > > 5 files changed, 48 insertions(+), 6 deletions(-) create mode > > > > > 100644 buildtools/get_cpu_count.py create mode 100644 > > > > > buildtools/get_numa_count.py > > > > > > > > > > diff --git a/buildtools/get_cpu_count.py > > > > > b/buildtools/get_cpu_count.py new file mode 100644 index > > > > > 000000000..386f85f8b > > > > > --- /dev/null > > > > > +++ b/buildtools/get_cpu_count.py > > > > > @@ -0,0 +1,7 @@ > > > > > +#!/usr/bin/python3 > > > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020 > > > > > +PANTHEON.tech s.r.o. > > > > > + > > > > > +import os > > > > > + > > > > > +print(os.cpu_count()) > > > > > diff --git a/buildtools/get_numa_count.py > > > > > b/buildtools/get_numa_count.py new file mode 100644 index > > > > > 000000000..f2ad35532 > > > > > --- /dev/null > > > > > +++ b/buildtools/get_numa_count.py > > > > > @@ -0,0 +1,17 @@ > > > > > +#!/usr/bin/python3 > > > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020 > > > > > +PANTHEON.tech s.r.o. > > > > > + > > > > > +import ctypes > > > > > +import glob > > > > > +import os > > > > > + > > > > > +if os.name == 'posix': > > > > > + print(len(glob.glob('/sys/devices/system/node/node*'))) > > > > > > > > FreeBSD also reports it's os.name as posix, but doesn't have a > > > > /sys/devices path to query. > > > > > > > > > > Do you know how do I get numa count on FreeBDS? I don't want to use the > > numa library if we can to avoid unneccesary dependencies. Do we need to cover > > some other cases other than FreeBDS, other Linux and Windows? > > > > > > > To get numa count on freebsd, you can query it using "sysctl vm.ndomains" > > command. However, the bigger issue here is that you need something different > > other than os.name to distinguish between linux and freebsd. > > > > That should only be a matter of finding the right Python api - https://docs.python.org/3/library/sys.html?highlight=platform#sys.platform should be it. > Yep. Though thinking about it afterwards too, checking between bsd and linux could also be as simple as using the /sys path if it exists, and trying to use sysctl otherwise. /Bruce