Re: [dpdk-dev] [PATCH v2] build: optional NUMA and cpu counts detection

["Juraj Linkeš" <juraj.linkes@pantheon.tech>]

I didn't notice this e-mail, since it was sent just to dev@dpdk.org and not to me explicitly, which left it unnoticed by my filters, unfortunately.

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of David Christensen
> Sent: Wednesday, March 31, 2021 9:07 PM
> To: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v2] build: optional NUMA and cpu counts
> detection
> 
> 
> 
> On 3/31/21 2:06 AM, Juraj Linkeš wrote:
> > Add an option to automatically discover the host's numa and cpu counts
> > and use those values for a non cross-build.
> > Give users the option to override the per-arch default values or
> > values from cross files by specifying them on the command line with
> > -Dmax_lcores and -Dmax_numa_nodes.
> >
> > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> > ---
> >   MAINTAINERS                  |  2 ++
> >   buildtools/get-cpu-count.py  |  7 ++++++
> >   buildtools/get-numa-count.py | 22 +++++++++++++++++
> >   buildtools/meson.build       |  2 ++
> >   config/meson.build           | 47 ++++++++++++++++++++++++++++++++++--
> >   config/x86/meson.build       |  2 ++
> >   meson_options.txt            |  8 +++---
> >   7 files changed, 84 insertions(+), 6 deletions(-)
> >   create mode 100644 buildtools/get-cpu-count.py
> >   create mode 100644 buildtools/get-numa-count.py
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS index 0ec5588540..7270f33cf5
> > 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -99,6 +99,8 @@ F: meson_options.txt
> >   F: config/
> >   F: buildtools/chkincs/
> >   F: buildtools/call-sphinx-build.py
> > +F: buildtools/get-cpu-count.py
> > +F: buildtools/get-numa-count.py
> >   F: buildtools/list-dir-globs.py
> >   F: buildtools/pkg-config/
> >   F: buildtools/symlink-drivers-solibs.sh
> > diff --git a/buildtools/get-cpu-count.py b/buildtools/get-cpu-count.py
> > new file mode 100644 index 0000000000..317b32088f
> > --- /dev/null
> > +++ b/buildtools/get-cpu-count.py
> > @@ -0,0 +1,7 @@
> > +#!/usr/bin/env python3
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2021
> > +PANTHEON.tech s.r.o.
> > +
> > +import os
> > +
> > +print(os.cpu_count())
> 
> Returns the expected value on a P9 system.
> 
> > diff --git a/buildtools/get-numa-count.py
> > b/buildtools/get-numa-count.py new file mode 100644 index
> > 0000000000..77ef2b9f24
> > --- /dev/null
> > +++ b/buildtools/get-numa-count.py
> > @@ -0,0 +1,22 @@
> > +#!/usr/bin/env python3
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2021
> > +PANTHEON.tech s.r.o.
> > +
> > +import ctypes
> > +import glob
> > +import os
> > +import subprocess
> > +
> > +if os.name == 'posix':
> > +    if os.path.isdir('/sys/devices/system/node'):
> > +        print(len(glob.glob('/sys/devices/system/node/node*')))
> > +    else:
> > +        subprocess.run(['sysctl', '-n', 'vm.ndomains'])
> > +
> > +elif os.name == 'nt':
> > +    libkernel32 = ctypes.windll.kernel32
> > +
> > +    count = ctypes.c_ulong()
> > +
> > +    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
> > +    print(count.value + 1)
> 
> Does not return the expected value on my P9 system (NUMA nodes are not
> contiguous).  Got 6, expect to see at least 8 otherwise I can't use lcores on
> NUMA node 8.
> 

I guess the problem here is that we're not really setting the number of NUMA nodes that are usable, but the highest NUMA that is usable?

In that case, the proper config in should be 256 so that even node255 is usable, is that right?

> $ python3 ./get-numa-count.py
> 6
> $ lscpu
> Architecture:        ppc64le
> Byte Order:          Little Endian
> CPU(s):              128
> On-line CPU(s) list: 0-127
> Thread(s) per core:  4
> Core(s) per socket:  16
> Socket(s):           2
> NUMA node(s):        6
> Model:               2.3 (pvr 004e 1203)
> Model name:          POWER9, altivec supported
> CPU max MHz:         3800.0000
> CPU min MHz:         2300.0000
> L1d cache:           32K
> L1i cache:           32K
> L2 cache:            512K
> L3 cache:            10240K
> NUMA node0 CPU(s):   0-63
> NUMA node8 CPU(s):   64-127
> NUMA node252 CPU(s):
> NUMA node253 CPU(s):
> NUMA node254 CPU(s):
> NUMA node255 CPU(s):
> 
> See attached ls-node.txt.gz for full directory structure on a P9 system.
> 
> Dave