From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E0A83A0C48; Tue, 6 Jul 2021 20:10:52 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 613844128B; Tue, 6 Jul 2021 20:10:52 +0200 (CEST) Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by mails.dpdk.org (Postfix) with ESMTP id F0EB74120E for ; Tue, 6 Jul 2021 20:10:50 +0200 (CEST) Received: from pps.filterd (m0098414.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 166I4nJW007361; Tue, 6 Jul 2021 14:10:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=subject : to : cc : references : from : message-id : date : mime-version : in-reply-to : content-type : content-transfer-encoding; s=pp1; bh=s/TI+PMZF3BZkRMQswaV3ldH+62GfG9J90BEdHOtmaY=; b=m2LXObld0/M0aZa1Yw2c2OHRBW0oIRb809XxpIk0tfeioVW5EcV1SdF0Rd3b8DYMD6xR 6exGGZzoKgZCtLLXesIki38AUkkbtfs8/IExXQQauj/E6XQMaTuWpmh1nYa/F7zoFKaz 9DG3DDOqLXNW6IXUg3FEG1Gpxxg+GOxWCVJeRdJc02/1JO6c3mEXiXQ59yThxr+ww/4Q +O6/txZQe5gCy5pgUzbI0rCl26yELvgqGe8dYdHZVaGahqBBD40lnxrTuyxqCVQArg7W DUVG1Eatt0nmUwCWh/AZOenVtpOH729lGSIzaDYS0lYZftzWekmrm3VdlIrlSZhmhZjv LA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 39m8xt5d6r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 06 Jul 2021 14:10:47 -0400 Received: from m0098414.ppops.net (m0098414.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 166I5Q9r009771; Tue, 6 Jul 2021 14:10:47 -0400 Received: from ppma04dal.us.ibm.com (7a.29.35a9.ip4.static.sl-reverse.com [169.53.41.122]) by mx0b-001b2d01.pphosted.com with ESMTP id 39m8xt5d6c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 06 Jul 2021 14:10:47 -0400 Received: from pps.filterd (ppma04dal.us.ibm.com [127.0.0.1]) by ppma04dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 166HvqG1009399; Tue, 6 Jul 2021 18:10:46 GMT Received: from b01cxnp22033.gho.pok.ibm.com (b01cxnp22033.gho.pok.ibm.com [9.57.198.23]) by ppma04dal.us.ibm.com with ESMTP id 39jfhc6u0c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 06 Jul 2021 18:10:46 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp22033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 166IAjqL10944798 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 6 Jul 2021 18:10:45 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A90FB112063; Tue, 6 Jul 2021 18:10:45 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id BD60B112062; Tue, 6 Jul 2021 18:10:44 +0000 (GMT) Received: from Davids-MBP.randomparity.org (unknown [9.163.29.254]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 6 Jul 2021 18:10:44 +0000 (GMT) To: Bruce Richardson , =?UTF-8?Q?Juraj_Linke=c5=a1?= Cc: "thomas@monjalon.net" , "david.marchand@redhat.com" , "Honnappa.Nagarahalli@arm.com" , "Ruifeng.Wang@arm.com" , "ferruh.yigit@intel.com" , "jerinjacobk@gmail.com" , "dev@dpdk.org" References: <1618827522-31828-1-git-send-email-juraj.linkes@pantheon.tech> <1624964105-6525-1-git-send-email-juraj.linkes@pantheon.tech> From: David Christensen Message-ID: Date: Tue, 6 Jul 2021 11:10:44 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: t3PBpeEagIltbLtki2qNIY-iUjW-JrhM X-Proofpoint-GUID: EWTQGdbyOxrdLlLen5868w6lAgPOjk6e X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-07-06_09:2021-07-06, 2021-07-06 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1011 adultscore=0 malwarescore=0 phishscore=0 priorityscore=1501 spamscore=0 impostorscore=0 mlxlogscore=999 suspectscore=0 lowpriorityscore=0 mlxscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107060085 Subject: Re: [dpdk-dev] [PATCH v4] build: optional NUMA and cpu counts detection X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 7/6/21 2:08 AM, Bruce Richardson wrote: > On Tue, Jul 06, 2021 at 08:56:37AM +0000, Juraj Linkeš wrote: >> >> >>> -----Original Message----- >>> From: Bruce Richardson >>> Sent: Tuesday, June 29, 2021 1:29 PM >>> To: Juraj Linkeš >>> Cc: thomas@monjalon.net; david.marchand@redhat.com; >>> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com; >>> ferruh.yigit@intel.com; jerinjacobk@gmail.com; dev@dpdk.org >>> Subject: Re: [PATCH v4] build: optional NUMA and cpu counts detection >>> >>> On Tue, Jun 29, 2021 at 12:55:05PM +0200, Juraj Linkeš wrote: >>>> Add an option to automatically discover the host's numa and cpu counts >>>> and use those values for a non cross-build. >>>> Give users the option to override the per-arch default values or >>>> values from cross files by specifying them on the command line with >>>> -Dmax_lcores and -Dmax_numa_nodes. >>>> >>>> Signed-off-by: Juraj Linkeš >>>> Reviewed-by: Honnappa Nagarahalli >>>> --- >>> Two very minor suggestions inline below. >>> >>> Acked-by: Bruce Richardson >>> >>>> >>> >>>> +max_lcores = get_option('max_lcores') if max_lcores == 'auto' >>> >>> Rather than "auto", would "detect" be a clearer name for this option value? >>> >>> >>>> +option('max_lcores', type: 'string', value: 'default', description: >>>> + 'Set maximum number of cores/threads supported by EAL. The >>>> +default is different per-arch. Set to auto to detect the number of cores on the >>> build machine.') option('max_numa_nodes', type: 'string', value: 'default', >>> description: >>>> + 'Set highest NUMA node supported by EAL. The default is >>>> +different per-arch. Set to auto to detect the highest numa node on >>>> +the build machine.') >>> >>> I'd put the explicit values of "default" and "auto"(or "detect") in quotes "" to >>> make clear they are literal values. >>> >> >> Thanks, Bruce, I'll change it. I have one extra question now that I'm looking at the patch: >> What does subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False) return exactly? Is the the number of NUMA nodes (looks like it) or the highest NUMA node on the system (the highest number of all NUMA nodes)? I'm asking because of how NUMA works on P9: >> NUMA node0 CPU(s): 0-63 >> NUMA node8 CPU(s): 64-127 >> NUMA node252 CPU(s): >> NUMA node253 CPU(s): >> NUMA node254 CPU(s): >> NUMA node255 CPU(s): >> >> Here we need not just two NUMA nodes, but at least 9 (0-8). Linux and Windows should return the highest NUMA, not sure about FreeBSD. Or maybe we should return the highest NUMA on which there are actual CPUs? > > I'm not sure, and I think to be really sure we'd need it tested on a P9 > system. The help text for the sysctl node says "Number of physical memory > domains available", which would imply 2 in the case above. [However, we also > would need to find out how BSD numbers the domains, too, as it's possible > an OS could just call them 0 and 1, rather than 0 and 8 if it wanted to.] > > In short, we'd need to test to be sure. Is FreeBSD on P9 a supported > config, and if so can the P9 maintainer perhaps help out with testing? Results of the v4 patch on an IBM AC922 P9 system with Linux: $ python3 get-cpu-count.py 128 $ python3 get-numa-count.py 8 $ lscpu Architecture: ppc64le Byte Order: Little Endian CPU(s): 128 On-line CPU(s) list: 0-127 Thread(s) per core: 4 Core(s) per socket: 16 Socket(s): 2 NUMA node(s): 6 Model: 2.3 (pvr 004e 1203) Model name: POWER9, altivec supported CPU max MHz: 3800.0000 CPU min MHz: 2300.0000 L1d cache: 32K L1i cache: 32K L2 cache: 512K L3 cache: 10240K NUMA node0 CPU(s): 0-63 NUMA node8 CPU(s): 64-127 NUMA node252 CPU(s): NUMA node253 CPU(s): NUMA node254 CPU(s): NUMA node255 CPU(s): OVS has a problem with requiring contiguous NUMA nodes that we've submitted patches to fix, so we need to ensure that's not a problem in DPDK. I've been doing things like "--socket-mem=2048,0,0,0,0,0,0,0,2048" so far to manage the memory-to-NUMA mapping which has been working fine, so it depends on how vm.ndomains will be. Dave