From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8D725A04B7; Tue, 13 Oct 2020 16:55:38 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 6DD481DCC4; Tue, 13 Oct 2020 16:54:58 +0200 (CEST) Received: from lb.pantheon.sk (lb.pantheon.sk [46.229.239.20]) by dpdk.org (Postfix) with ESMTP id 4669A1DBAA for ; Tue, 13 Oct 2020 16:54:54 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by lb.pantheon.sk (Postfix) with ESMTP id 1596BAB04C; Tue, 13 Oct 2020 16:54:53 +0200 (CEST) X-Virus-Scanned: amavisd-new at siecit.sk Received: from lb.pantheon.sk ([127.0.0.1]) by localhost (lb.pantheon.sk [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sONB6uuchAj3; Tue, 13 Oct 2020 16:54:51 +0200 (CEST) Received: from service-node1.lab.pantheon.local (unknown [46.229.239.141]) by lb.pantheon.sk (Postfix) with ESMTP id B51AFB6B43; Tue, 13 Oct 2020 16:54:47 +0200 (CEST) From: =?UTF-8?q?Juraj=20Linke=C5=A1?= To: bruce.richardson@intel.com, Ruifeng.Wang@arm.com, Honnappa.Nagarahalli@arm.com, Phil.Yang@arm.com, vcchunga@amazon.com, Dharmik.Thakkar@arm.com, jerinjacobk@gmail.com, hemant.agrawal@nxp.com Cc: dev@dpdk.org, =?UTF-8?q?Juraj=20Linke=C5=A1?= Date: Tue, 13 Oct 2020 16:54:39 +0200 Message-Id: <1602600882-695-4-git-send-email-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1602600882-695-1-git-send-email-juraj.linkes@pantheon.tech> References: <1600867161-15673-1-git-send-email-juraj.linkes@pantheon.tech> <1602600882-695-1-git-send-email-juraj.linkes@pantheon.tech> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [RFC PATCH v2 3/6] build: automatic NUMA and cpu counts detection X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The build machine's number of cpus and numa nodes vary, resulting in mismatched counts of RTE_MAX_LCORE and RTE_MAX_NUMA_NODES for many builds. Automatically discover the host's numa and cpu counts to remove this mismatch for native builds. Use current defaults for cross builds. Leave users the option to override both if the specify a non-zero amount on the command line. Signed-off-by: Juraj Linkeš --- buildtools/get_cpu_count.py | 7 +++++++ buildtools/get_numa_count.py | 17 +++++++++++++++++ buildtools/meson.build | 2 ++ config/meson.build | 20 ++++++++++++++++++-- meson_options.txt | 8 ++++---- 5 files changed, 48 insertions(+), 6 deletions(-) create mode 100644 buildtools/get_cpu_count.py create mode 100644 buildtools/get_numa_count.py diff --git a/buildtools/get_cpu_count.py b/buildtools/get_cpu_count.py new file mode 100644 index 000000000..386f85f8b --- /dev/null +++ b/buildtools/get_cpu_count.py @@ -0,0 +1,7 @@ +#!/usr/bin/python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2020 PANTHEON.tech s.r.o. + +import os + +print(os.cpu_count()) diff --git a/buildtools/get_numa_count.py b/buildtools/get_numa_count.py new file mode 100644 index 000000000..f2ad35532 --- /dev/null +++ b/buildtools/get_numa_count.py @@ -0,0 +1,17 @@ +#!/usr/bin/python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2020 PANTHEON.tech s.r.o. + +import ctypes +import glob +import os + +if os.name == 'posix': + print(len(glob.glob('/sys/devices/system/node/node*'))) +elif os.name == 'nt': + libkernel32 = ctypes.windll.kernel32 + + count = ctypes.c_ulong() + + libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count)) + print(count.value + 1) diff --git a/buildtools/meson.build b/buildtools/meson.build index 04808dabc..925e733b1 100644 --- a/buildtools/meson.build +++ b/buildtools/meson.build @@ -17,3 +17,5 @@ else endif map_to_win_cmd = py3 + files('map_to_win.py') sphinx_wrapper = py3 + files('call-sphinx-build.py') +get_cpu_count_cmd = py3 + files('get_cpu_count.py') +get_numa_count_cmd = py3 + files('get_numa_count.py') diff --git a/config/meson.build b/config/meson.build index 4bd65d98e..84c31c8e7 100644 --- a/config/meson.build +++ b/config/meson.build @@ -226,8 +226,24 @@ foreach arg: warning_flags endforeach # set other values pulled from the build options -dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores')) -dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes')) +max_lcores = get_option('max_lcores') +if max_lcores == 0 + if meson.is_cross_build() + max_lcores = 4 + else + max_lcores = run_command(get_cpu_count_cmd).stdout() + endif +endif +dpdk_conf.set('RTE_MAX_LCORE', max_lcores) +max_numa_nodes = get_option('max_numa_nodes') +if max_numa_nodes == 0 + if meson.is_cross_build() + max_numa_nodes = 128 + else + max_numa_nodes = run_command(get_numa_count_cmd).stdout() + endif +endif +dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes) dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports')) dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet')) dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp')) diff --git a/meson_options.txt b/meson_options.txt index 9bf18ab6b..60a949fca 100644 --- a/meson_options.txt +++ b/meson_options.txt @@ -26,10 +26,10 @@ option('machine', type: 'string', value: 'native', description: 'set the target machine type') option('max_ethports', type: 'integer', value: 32, description: 'maximum number of Ethernet devices') -option('max_lcores', type: 'integer', value: 128, - description: 'maximum number of cores/threads supported by EAL') -option('max_numa_nodes', type: 'integer', value: 4, - description: 'maximum number of NUMA nodes supported by EAL') +option('max_lcores', type: 'integer', value: 0, + description: 'maximum number of cores/threads supported by EAL. Value 0 means the number of cpus on the host will be used') +option('max_numa_nodes', type: 'integer', value: 0, + description: 'maximum number of NUMA nodes supported by EAL. Value 0 means the number of numa nodes on the host will be used') option('enable_trace_fp', type: 'boolean', value: false, description: 'enable fast path trace points.') option('tests', type: 'boolean', value: true, -- 2.20.1