From: Bruce Richardson <bruce.richardson@intel.com>
To: "Juraj Linkeš" <juraj.linkes@pantheon.tech>
Cc: David Christensen <drc@linux.vnet.ibm.com>,
"thomas@monjalon.net" <thomas@monjalon.net>,
"david.marchand@redhat.com" <david.marchand@redhat.com>,
"Honnappa.Nagarahalli@arm.com" <Honnappa.Nagarahalli@arm.com>,
"Ruifeng.Wang@arm.com" <Ruifeng.Wang@arm.com>,
"ferruh.yigit@intel.com" <ferruh.yigit@intel.com>,
"jerinjacobk@gmail.com" <jerinjacobk@gmail.com>,
"dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH v4] build: optional NUMA and cpu counts detection
Date: Fri, 16 Jul 2021 15:24:55 +0100 [thread overview]
Message-ID: <YPGWt+P5T9jfnNO+@bricha3-MOBL.ger.corp.intel.com> (raw)
In-Reply-To: <fa796fd941394ab9bef5130971a48d5b@pantheon.tech>
On Fri, Jul 16, 2021 at 01:53:18PM +0000, Juraj Linkeš wrote:
>
>
> > -----Original Message-----
> > From: David Christensen <drc@linux.vnet.ibm.com>
> > Sent: Tuesday, July 6, 2021 8:11 PM
> > To: Bruce Richardson <bruce.richardson@intel.com>; Juraj Linkeš
> > <juraj.linkes@pantheon.tech>
> > Cc: thomas@monjalon.net; david.marchand@redhat.com;
> > Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> > ferruh.yigit@intel.com; jerinjacobk@gmail.com; dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH v4] build: optional NUMA and cpu counts
> > detection
> >
> >
> >
> > On 7/6/21 2:08 AM, Bruce Richardson wrote:
> > > On Tue, Jul 06, 2021 at 08:56:37AM +0000, Juraj Linkeš wrote:
> > >>
> > >>
> > >>> -----Original Message-----
> > >>> From: Bruce Richardson <bruce.richardson@intel.com>
> > >>> Sent: Tuesday, June 29, 2021 1:29 PM
> > >>> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > >>> Cc: thomas@monjalon.net; david.marchand@redhat.com;
> > >>> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> > >>> ferruh.yigit@intel.com; jerinjacobk@gmail.com; dev@dpdk.org
> > >>> Subject: Re: [PATCH v4] build: optional NUMA and cpu counts
> > >>> detection
> > >>>
> > >>> On Tue, Jun 29, 2021 at 12:55:05PM +0200, Juraj Linkeš wrote:
> > >>>> Add an option to automatically discover the host's numa and cpu
> > >>>> counts and use those values for a non cross-build.
> > >>>> Give users the option to override the per-arch default values or
> > >>>> values from cross files by specifying them on the command line with
> > >>>> -Dmax_lcores and -Dmax_numa_nodes.
> > >>>>
> > >>>> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > >>>> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> > >>>> ---
> > >>> Two very minor suggestions inline below.
> > >>>
> > >>> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> > >>>
> > >>>>
> > >>> <snip>
> > >>>> +max_lcores = get_option('max_lcores') if max_lcores == 'auto'
> > >>>
> > >>> Rather than "auto", would "detect" be a clearer name for this option value?
> > >>>
> > >>> <snip>
> > >>>> +option('max_lcores', type: 'string', value: 'default', description:
> > >>>> + 'Set maximum number of cores/threads supported by EAL. The
> > >>>> +default is different per-arch. Set to auto to detect the number of
> > >>>> +cores on the
> > >>> build machine.') option('max_numa_nodes', type: 'string', value:
> > >>> 'default',
> > >>> description:
> > >>>> + 'Set highest NUMA node supported by EAL. The default is
> > >>>> +different per-arch. Set to auto to detect the highest numa node on
> > >>>> +the build machine.')
> > >>>
> > >>> I'd put the explicit values of "default" and "auto"(or "detect") in
> > >>> quotes "" to make clear they are literal values.
> > >>>
> > >>
> > >> Thanks, Bruce, I'll change it. I have one extra question now that I'm looking
> > at the patch:
> > >> What does subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False) return
> > exactly? Is the the number of NUMA nodes (looks like it) or the highest NUMA
> > node on the system (the highest number of all NUMA nodes)? I'm asking
> > because of how NUMA works on P9:
> > >> NUMA node0 CPU(s): 0-63
> > >> NUMA node8 CPU(s): 64-127
> > >> NUMA node252 CPU(s):
> > >> NUMA node253 CPU(s):
> > >> NUMA node254 CPU(s):
> > >> NUMA node255 CPU(s):
> > >>
> > >> Here we need not just two NUMA nodes, but at least 9 (0-8). Linux and
> > Windows should return the highest NUMA, not sure about FreeBSD. Or maybe
> > we should return the highest NUMA on which there are actual CPUs?
> > >
> > > I'm not sure, and I think to be really sure we'd need it tested on a
> > > P9 system. The help text for the sysctl node says "Number of physical
> > > memory domains available", which would imply 2 in the case above.
> > > [However, we also would need to find out how BSD numbers the domains,
> > > too, as it's possible an OS could just call them 0 and 1, rather than
> > > 0 and 8 if it wanted to.]
> > >
> > > In short, we'd need to test to be sure. Is FreeBSD on P9 a supported
> > > config, and if so can the P9 maintainer perhaps help out with testing?
> >
> > Results of the v4 patch on an IBM AC922 P9 system with Linux:
> >
>
> Can you get results from FreeBSD as well?
>
> > $ python3 get-numa-count.py
> > 8
> > NUMA node0 CPU(s): 0-63
> > NUMA node8 CPU(s): 64-127
> <snip>
> Is this the right number for your case, i.e. are you able to use both numa nodes when RTE_MAX_NUMA_NODES=8?
>
> Or maybe this is a question for Bruce or Thomas - what do we need to set in RTE_MAX_NUMA_NODES to be able to use all numa nodes on the system? The highest numa node number or that + 1? In linux, with 4 numa nodes, there will be node0-node3 under /sys/devices/system/node - do we need to set RTE_MAX_NUMA_NODES to 3 or 4?
>
4, I believe.
next prev parent reply other threads:[~2021-07-16 14:25 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-20 12:55 [dpdk-dev] [PATCH v1 1/1] " Juraj Linkeš
2020-12-23 11:32 ` Juraj Linkeš
2021-03-31 9:06 ` [dpdk-dev] [PATCH v2] " Juraj Linkeš
2021-03-31 19:07 ` David Christensen
2021-04-16 12:43 ` Juraj Linkeš
2021-04-19 10:18 ` [dpdk-dev] [PATCH v3] " Juraj Linkeš
2021-04-28 19:33 ` David Christensen
2021-06-29 10:55 ` [dpdk-dev] [PATCH v4] " Juraj Linkeš
2021-06-29 11:28 ` Bruce Richardson
2021-07-06 8:56 ` Juraj Linkeš
2021-07-06 9:08 ` Bruce Richardson
2021-07-06 18:10 ` David Christensen
2021-07-16 13:53 ` Juraj Linkeš
2021-07-16 14:24 ` Bruce Richardson [this message]
2021-07-20 20:49 ` David Christensen
2021-07-21 12:41 ` Juraj Linkeš
2021-06-29 18:00 ` Stephen Hemminger
2021-07-21 13:04 ` [dpdk-dev] [PATCH v5] " Juraj Linkeš
2021-08-02 12:44 ` Juraj Linkeš
2021-08-02 23:29 ` David Christensen
2021-08-03 10:21 ` Juraj Linkeš
2021-08-31 0:54 ` Piotr Kubaj
2021-08-31 7:54 ` Juraj Linkeš
2021-08-31 8:02 ` Bruce Richardson
2021-09-09 7:20 ` Juraj Linkeš
2021-09-09 8:01 ` Bruce Richardson
2021-09-09 8:08 ` Thomas Monjalon
2021-09-09 8:42 ` Juraj Linkeš
2021-09-09 16:44 ` David Christensen
2021-08-31 12:32 ` Piotr Kubaj
2021-08-17 10:45 ` [dpdk-dev] [PATCH v6] " Juraj Linkeš
2021-09-16 7:46 ` Thomas Monjalon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YPGWt+P5T9jfnNO+@bricha3-MOBL.ger.corp.intel.com \
--to=bruce.richardson@intel.com \
--cc=Honnappa.Nagarahalli@arm.com \
--cc=Ruifeng.Wang@arm.com \
--cc=david.marchand@redhat.com \
--cc=dev@dpdk.org \
--cc=drc@linux.vnet.ibm.com \
--cc=ferruh.yigit@intel.com \
--cc=jerinjacobk@gmail.com \
--cc=juraj.linkes@pantheon.tech \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).