DPDK patches and discussions
 help / color / mirror / Atom feed
From: David Christensen <drc@linux.vnet.ibm.com>
To: Juraj Linkeš <juraj.linkes@pantheon.tech>,
	Bruce Richardson <bruce.richardson@intel.com>
Cc: "thomas@monjalon.net" <thomas@monjalon.net>,
	"david.marchand@redhat.com" <david.marchand@redhat.com>,
	"Honnappa.Nagarahalli@arm.com" <Honnappa.Nagarahalli@arm.com>,
	"Ruifeng.Wang@arm.com" <Ruifeng.Wang@arm.com>,
	"ferruh.yigit@intel.com" <ferruh.yigit@intel.com>,
	"jerinjacobk@gmail.com" <jerinjacobk@gmail.com>,
	"dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH v4] build: optional NUMA and cpu counts detection
Date: Tue, 20 Jul 2021 13:49:31 -0700
Message-ID: <ce143540-3237-9ed3-ce2e-552590b42470@linux.vnet.ibm.com> (raw)
In-Reply-To: <fa796fd941394ab9bef5130971a48d5b@pantheon.tech>



On 7/16/21 6:53 AM, Juraj Linkeš wrote:
> 
> 
>> -----Original Message-----
>> From: David Christensen <drc@linux.vnet.ibm.com>
>> Sent: Tuesday, July 6, 2021 8:11 PM
>> To: Bruce Richardson <bruce.richardson@intel.com>; Juraj Linkeš
>> <juraj.linkes@pantheon.tech>
>> Cc: thomas@monjalon.net; david.marchand@redhat.com;
>> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
>> ferruh.yigit@intel.com; jerinjacobk@gmail.com; dev@dpdk.org
>> Subject: Re: [dpdk-dev] [PATCH v4] build: optional NUMA and cpu counts
>> detection
>>
>>
>>
>> On 7/6/21 2:08 AM, Bruce Richardson wrote:
>>> On Tue, Jul 06, 2021 at 08:56:37AM +0000, Juraj Linkeš wrote:
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Bruce Richardson <bruce.richardson@intel.com>
>>>>> Sent: Tuesday, June 29, 2021 1:29 PM
>>>>> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
>>>>> Cc: thomas@monjalon.net; david.marchand@redhat.com;
>>>>> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
>>>>> ferruh.yigit@intel.com; jerinjacobk@gmail.com; dev@dpdk.org
>>>>> Subject: Re: [PATCH v4] build: optional NUMA and cpu counts
>>>>> detection
>>>>>
>>>>> On Tue, Jun 29, 2021 at 12:55:05PM +0200, Juraj Linkeš wrote:
>>>>>> Add an option to automatically discover the host's numa and cpu
>>>>>> counts and use those values for a non cross-build.
>>>>>> Give users the option to override the per-arch default values or
>>>>>> values from cross files by specifying them on the command line with
>>>>>> -Dmax_lcores and -Dmax_numa_nodes.
>>>>>>
>>>>>> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>>>>>> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
>>>>>> ---
>>>>> Two very minor suggestions inline below.
>>>>>
>>>>> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
>>>>>
>>>>>>
>>>>> <snip>
>>>>>> +max_lcores = get_option('max_lcores') if max_lcores == 'auto'
>>>>>
>>>>> Rather than "auto", would "detect" be a clearer name for this option value?
>>>>>
>>>>> <snip>
>>>>>> +option('max_lcores', type: 'string', value: 'default', description:
>>>>>> +       'Set maximum number of cores/threads supported by EAL. The
>>>>>> +default is different per-arch. Set to auto to detect the number of
>>>>>> +cores on the
>>>>> build machine.') option('max_numa_nodes', type: 'string', value:
>>>>> 'default',
>>>>> description:
>>>>>> +       'Set highest NUMA node supported by EAL. The default is
>>>>>> +different per-arch. Set to auto to detect the highest numa node on
>>>>>> +the build machine.')
>>>>>
>>>>> I'd put the explicit values of "default" and "auto"(or "detect") in
>>>>> quotes "" to make clear they are literal values.
>>>>>
>>>>
>>>> Thanks, Bruce, I'll change it. I have one extra question now that I'm looking
>> at the patch:
>>>> What does subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False) return
>> exactly? Is the the number of NUMA nodes (looks like it) or the highest NUMA
>> node on the system (the highest number of all NUMA nodes)? I'm asking
>> because of how NUMA works on P9:
>>>> NUMA node0 CPU(s):   0-63
>>>> NUMA node8 CPU(s):   64-127
>>>> NUMA node252 CPU(s):
>>>> NUMA node253 CPU(s):
>>>> NUMA node254 CPU(s):
>>>> NUMA node255 CPU(s):
>>>>
>>>> Here we need not just two NUMA nodes, but at least 9 (0-8). Linux and
>> Windows should return the highest NUMA, not sure about FreeBSD. Or maybe
>> we should return the highest NUMA on which there are actual CPUs?
>>>
>>> I'm not sure, and I think to be really sure we'd need it tested on a
>>> P9 system. The help text for the sysctl node says "Number of physical
>>> memory domains available", which would imply 2 in the case above.
>>> [However, we also would need to find out how BSD numbers the domains,
>>> too, as it's possible an OS could just call them 0 and 1, rather than
>>> 0 and 8 if it wanted to.]
>>>
>>> In short, we'd need to test to be sure. Is FreeBSD on P9 a supported
>>> config, and if so can the P9 maintainer perhaps help out with testing?
>>
>> Results of the v4 patch on an IBM AC922 P9 system with Linux:
>>
> 
> Can you get results from FreeBSD as well?

I can't help with FreeBSD here, I only have Linux on systems within IBM.

> 
>> $ python3 get-numa-count.py
>> 8
>> NUMA node0 CPU(s):   0-63
>> NUMA node8 CPU(s):   64-127
> <snip>
> Is this the right number for your case, i.e. are you able to use both numa nodes when RTE_MAX_NUMA_NODES=8?

node8 above is the ninth NUMA node so I'd need to use 
RTE_MAX_NUMA_NODES=9 as a minimum so that any arrays using that value 
are adequately sized.

Dave

  parent reply	other threads:[~2021-07-20 20:49 UTC|newest]

Thread overview: 32+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-20 12:55 [dpdk-dev] [PATCH v1 1/1] " Juraj Linkeš
2020-12-23 11:32 ` Juraj Linkeš
2021-03-31  9:06 ` [dpdk-dev] [PATCH v2] " Juraj Linkeš
2021-03-31 19:07   ` David Christensen
2021-04-16 12:43     ` Juraj Linkeš
2021-04-19 10:18   ` [dpdk-dev] [PATCH v3] " Juraj Linkeš
2021-04-28 19:33     ` David Christensen
2021-06-29 10:55     ` [dpdk-dev] [PATCH v4] " Juraj Linkeš
2021-06-29 11:28       ` Bruce Richardson
2021-07-06  8:56         ` Juraj Linkeš
2021-07-06  9:08           ` Bruce Richardson
2021-07-06 18:10             ` David Christensen
2021-07-16 13:53               ` Juraj Linkeš
2021-07-16 14:24                 ` Bruce Richardson
2021-07-20 20:49                 ` David Christensen [this message]
2021-07-21 12:41                   ` Juraj Linkeš
2021-06-29 18:00       ` Stephen Hemminger
2021-07-21 13:04       ` [dpdk-dev] [PATCH v5] " Juraj Linkeš
2021-08-02 12:44         ` Juraj Linkeš
2021-08-02 23:29           ` David Christensen
2021-08-03 10:21             ` Juraj Linkeš
2021-08-31  0:54               ` Piotr Kubaj
2021-08-31  7:54                 ` Juraj Linkeš
2021-08-31  8:02                   ` Bruce Richardson
2021-09-09  7:20                     ` Juraj Linkeš
2021-09-09  8:01                       ` Bruce Richardson
2021-09-09  8:08                         ` Thomas Monjalon
2021-09-09  8:42                           ` Juraj Linkeš
2021-09-09 16:44                         ` David Christensen
2021-08-31 12:32                   ` Piotr Kubaj
2021-08-17 10:45         ` [dpdk-dev] [PATCH v6] " Juraj Linkeš
2021-09-16  7:46           ` Thomas Monjalon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ce143540-3237-9ed3-ce2e-552590b42470@linux.vnet.ibm.com \
    --to=drc@linux.vnet.ibm.com \
    --cc=Honnappa.Nagarahalli@arm.com \
    --cc=Ruifeng.Wang@arm.com \
    --cc=bruce.richardson@intel.com \
    --cc=david.marchand@redhat.com \
    --cc=dev@dpdk.org \
    --cc=ferruh.yigit@intel.com \
    --cc=jerinjacobk@gmail.com \
    --cc=juraj.linkes@pantheon.tech \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK patches and discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ https://inbox.dpdk.org/dev \
		dev@dpdk.org
	public-inbox-index dev

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git