From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id 39875A0C4C;
Tue, 20 Jul 2021 22:49:44 +0200 (CEST)
Received: from [217.70.189.124] (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id 247794068B;
Tue, 20 Jul 2021 22:49:44 +0200 (CEST)
Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com
[148.163.156.1]) by mails.dpdk.org (Postfix) with ESMTP id EEF9F40689
for <dev@dpdk.org>; Tue, 20 Jul 2021 22:49:42 +0200 (CEST)
Received: from pps.filterd (m0098393.ppops.net [127.0.0.1])
by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id
16KKYSvS141841; Tue, 20 Jul 2021 16:49:37 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com;
h=subject : to : cc :
references : from : message-id : date : mime-version : in-reply-to :
content-type : content-transfer-encoding; s=pp1;
bh=zvcc7rYdbpsnu/Bc4J6oTbD2z0sNuyiJb4vhV7rlZ44=;
b=gkQA5jrpUYPPU+PxfdAutvfNKzugF+FfUDdPa89ysjBf8IM4oD/yWqnjh04hC12SP9/U
mNaYIFjIUtHRUmiFXlVtOjFBMcGYFSPmd/67ucnaC8TilE7yRIdu4B8mG2UYeLBaRQhE
RVrFPCU25BM1P+dbmWATuqUpKK8pJ+RuUvqosz9uqedoMEyyMOl0PLQuUVaOTz/7ZlKf
hP9xHc4HcjWJvque0eCWN7YZx+m42TXW3jMSF2BIqzrV6MyDkRRXAQ7qKUXjSmCl60xu
0FiAEOK5ElRtPsy0y5At3YQ5Wnvl8Oc3Vowbea7gD0yFiOQXa+We6Oh06B7ne/491+WC ng==
Received: from pps.reinject (localhost [127.0.0.1])
by mx0a-001b2d01.pphosted.com with ESMTP id 39wy71x3m4-1
(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT);
Tue, 20 Jul 2021 16:49:37 -0400
Received: from m0098393.ppops.net (m0098393.ppops.net [127.0.0.1])
by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 16KKa9KP003257;
Tue, 20 Jul 2021 16:49:37 -0400
Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com
[169.63.214.131])
by mx0a-001b2d01.pphosted.com with ESMTP id 39wy71x3kq-1
(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT);
Tue, 20 Jul 2021 16:49:36 -0400
Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1])
by ppma01dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 16KKWSXJ032504;
Tue, 20 Jul 2021 20:49:35 GMT
Received: from b03cxnp07028.gho.boulder.ibm.com
(b03cxnp07028.gho.boulder.ibm.com [9.17.130.15])
by ppma01dal.us.ibm.com with ESMTP id 39upucm2vd-1
(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT);
Tue, 20 Jul 2021 20:49:35 +0000
Received: from b03ledav001.gho.boulder.ibm.com
(b03ledav001.gho.boulder.ibm.com [9.17.130.232])
by b03cxnp07028.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id
16KKnY5N45023690
(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);
Tue, 20 Jul 2021 20:49:34 GMT
Received: from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1])
by IMSVA (Postfix) with ESMTP id 91C5F6E054;
Tue, 20 Jul 2021 20:49:34 +0000 (GMT)
Received: from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1])
by IMSVA (Postfix) with ESMTP id E9EE16E059;
Tue, 20 Jul 2021 20:49:32 +0000 (GMT)
Received: from Davids-MBP.randomparity.org (unknown [9.211.129.91])
by b03ledav001.gho.boulder.ibm.com (Postfix) with ESMTP;
Tue, 20 Jul 2021 20:49:32 +0000 (GMT)
To: =?UTF-8?Q?Juraj_Linke=c5=a1?= <juraj.linkes@pantheon.tech>,
Bruce Richardson <bruce.richardson@intel.com>
Cc: "thomas@monjalon.net" <thomas@monjalon.net>,
"david.marchand@redhat.com" <david.marchand@redhat.com>,
"Honnappa.Nagarahalli@arm.com" <Honnappa.Nagarahalli@arm.com>,
"Ruifeng.Wang@arm.com" <Ruifeng.Wang@arm.com>,
"ferruh.yigit@intel.com" <ferruh.yigit@intel.com>,
"jerinjacobk@gmail.com" <jerinjacobk@gmail.com>, "dev@dpdk.org"
<dev@dpdk.org>
References: <1618827522-31828-1-git-send-email-juraj.linkes@pantheon.tech>
<1624964105-6525-1-git-send-email-juraj.linkes@pantheon.tech>
<YNsD6FLhwUkJ3ltd@bricha3-MOBL.ger.corp.intel.com>
<d3db6ed877a24873ba3fe6528a578a9c@pantheon.tech>
<YOQdlbSEe2CmTPWO@bricha3-MOBL.ger.corp.intel.com>
<c7bd1f52-134a-1726-488c-114d9fe8871c@linux.vnet.ibm.com>
<fa796fd941394ab9bef5130971a48d5b@pantheon.tech>
From: David Christensen <drc@linux.vnet.ibm.com>
Message-ID: <ce143540-3237-9ed3-ce2e-552590b42470@linux.vnet.ibm.com>
Date: Tue, 20 Jul 2021 13:49:31 -0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0)
Gecko/20100101 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: <fa796fd941394ab9bef5130971a48d5b@pantheon.tech>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-TM-AS-GCONF: 00
X-Proofpoint-GUID: MENOoeJyUYx2A4ThNp0lHsHwUOpX4GMf
X-Proofpoint-ORIG-GUID: mTYD-lS9QlJ3sx3MkjBXW3z41cGxIsKu
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790
definitions=2021-07-20_13:2021-07-19,
2021-07-20 signatures=0
X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0
priorityscore=1501
lowpriorityscore=0 clxscore=1015 suspectscore=0 mlxlogscore=999
spamscore=0 mlxscore=0 malwarescore=0 adultscore=0 phishscore=0
impostorscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx
scancount=1 engine=8.12.0-2104190000 definitions=main-2107200130
Subject: Re: [dpdk-dev] [PATCH v4] build: optional NUMA and cpu counts
detection
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>
On 7/16/21 6:53 AM, Juraj Linkeš wrote:
>
>
>> -----Original Message-----
>> From: David Christensen <drc@linux.vnet.ibm.com>
>> Sent: Tuesday, July 6, 2021 8:11 PM
>> To: Bruce Richardson <bruce.richardson@intel.com>; Juraj Linkeš
>> <juraj.linkes@pantheon.tech>
>> Cc: thomas@monjalon.net; david.marchand@redhat.com;
>> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
>> ferruh.yigit@intel.com; jerinjacobk@gmail.com; dev@dpdk.org
>> Subject: Re: [dpdk-dev] [PATCH v4] build: optional NUMA and cpu counts
>> detection
>>
>>
>>
>> On 7/6/21 2:08 AM, Bruce Richardson wrote:
>>> On Tue, Jul 06, 2021 at 08:56:37AM +0000, Juraj Linkeš wrote:
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Bruce Richardson <bruce.richardson@intel.com>
>>>>> Sent: Tuesday, June 29, 2021 1:29 PM
>>>>> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
>>>>> Cc: thomas@monjalon.net; david.marchand@redhat.com;
>>>>> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
>>>>> ferruh.yigit@intel.com; jerinjacobk@gmail.com; dev@dpdk.org
>>>>> Subject: Re: [PATCH v4] build: optional NUMA and cpu counts
>>>>> detection
>>>>>
>>>>> On Tue, Jun 29, 2021 at 12:55:05PM +0200, Juraj Linkeš wrote:
>>>>>> Add an option to automatically discover the host's numa and cpu
>>>>>> counts and use those values for a non cross-build.
>>>>>> Give users the option to override the per-arch default values or
>>>>>> values from cross files by specifying them on the command line with
>>>>>> -Dmax_lcores and -Dmax_numa_nodes.
>>>>>>
>>>>>> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>>>>>> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
>>>>>> ---
>>>>> Two very minor suggestions inline below.
>>>>>
>>>>> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
>>>>>
>>>>>>
>>>>> <snip>
>>>>>> +max_lcores = get_option('max_lcores') if max_lcores == 'auto'
>>>>>
>>>>> Rather than "auto", would "detect" be a clearer name for this option value?
>>>>>
>>>>> <snip>
>>>>>> +option('max_lcores', type: 'string', value: 'default', description:
>>>>>> + 'Set maximum number of cores/threads supported by EAL. The
>>>>>> +default is different per-arch. Set to auto to detect the number of
>>>>>> +cores on the
>>>>> build machine.') option('max_numa_nodes', type: 'string', value:
>>>>> 'default',
>>>>> description:
>>>>>> + 'Set highest NUMA node supported by EAL. The default is
>>>>>> +different per-arch. Set to auto to detect the highest numa node on
>>>>>> +the build machine.')
>>>>>
>>>>> I'd put the explicit values of "default" and "auto"(or "detect") in
>>>>> quotes "" to make clear they are literal values.
>>>>>
>>>>
>>>> Thanks, Bruce, I'll change it. I have one extra question now that I'm looking
>> at the patch:
>>>> What does subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False) return
>> exactly? Is the the number of NUMA nodes (looks like it) or the highest NUMA
>> node on the system (the highest number of all NUMA nodes)? I'm asking
>> because of how NUMA works on P9:
>>>> NUMA node0 CPU(s): 0-63
>>>> NUMA node8 CPU(s): 64-127
>>>> NUMA node252 CPU(s):
>>>> NUMA node253 CPU(s):
>>>> NUMA node254 CPU(s):
>>>> NUMA node255 CPU(s):
>>>>
>>>> Here we need not just two NUMA nodes, but at least 9 (0-8). Linux and
>> Windows should return the highest NUMA, not sure about FreeBSD. Or maybe
>> we should return the highest NUMA on which there are actual CPUs?
>>>
>>> I'm not sure, and I think to be really sure we'd need it tested on a
>>> P9 system. The help text for the sysctl node says "Number of physical
>>> memory domains available", which would imply 2 in the case above.
>>> [However, we also would need to find out how BSD numbers the domains,
>>> too, as it's possible an OS could just call them 0 and 1, rather than
>>> 0 and 8 if it wanted to.]
>>>
>>> In short, we'd need to test to be sure. Is FreeBSD on P9 a supported
>>> config, and if so can the P9 maintainer perhaps help out with testing?
>>
>> Results of the v4 patch on an IBM AC922 P9 system with Linux:
>>
>
> Can you get results from FreeBSD as well?
I can't help with FreeBSD here, I only have Linux on systems within IBM.
>
>> $ python3 get-numa-count.py
>> 8
>> NUMA node0 CPU(s): 0-63
>> NUMA node8 CPU(s): 64-127
> <snip>
> Is this the right number for your case, i.e. are you able to use both numa nodes when RTE_MAX_NUMA_NODES=8?
node8 above is the ninth NUMA node so I'd need to use
RTE_MAX_NUMA_NODES=9 as a minimum so that any arrays using that value
are adequately sized.
Dave