[dpdk-dev] [RFC PATCH 0/2] Arm build options rework

[dpdk-dev] [RFC PATCH 0/2] Arm build options rework

[Juraj Linkeš]

The current way of specifying Arm configuration options is insufficient
since we can't identify the SoC we're building for from the MIDR
information.

Instead use user input - the machine meson option to specify the SoC, e.g.:
meson soc_test -Dmachine=thunderx
meson soc_test --cross-file config/arm/thunderx/arm64_thunderx_linux_gcc

We also want to be able to disable which drivers (and possibly
libraries) are built without user input. This is useful when building:
1. on an SoC that is slow and we want to build only what is necessary
without the user having to check which libraries they have installed
2. a cross build on a fast aarch64 machine but with target SoC which
differs in capabilities or libraries.

Juraj Linkeš (2):
  build: rework Arm build options
  build: disable drivers from file

 config/arm/arm64_armv8_linux_gcc              |  28 ----
 .../arm/{ => armada}/arm64_armada_linux_gcc   |   2 +-
 config/arm/armada/meson.build                 |  31 ++++
 config/arm/default/arm64_armv8_linux_gcc      |  23 +++
 config/arm/default/meson.build                |   8 +
 config/arm/meson.build                        | 139 +++++++++++-------
 .../{ => thunderx}/arm64_thunderx_linux_gcc   |   2 +-
 config/arm/thunderx/meson.build               |  18 +++
 .../thunderxt88/arm64_thunderxt88_linux_gcc   |  16 ++
 config/arm/thunderxt88/meson.build            |  18 +++
 config/meson.build                            |   9 +-
 drivers/meson.build                           |  13 +-
 12 files changed, 218 insertions(+), 89 deletions(-)
 delete mode 100644 config/arm/arm64_armv8_linux_gcc
 rename config/arm/{ => armada}/arm64_armada_linux_gcc (93%)
 create mode 100644 config/arm/armada/meson.build
 create mode 100644 config/arm/default/arm64_armv8_linux_gcc
 create mode 100644 config/arm/default/meson.build
 rename config/arm/{ => thunderx}/arm64_thunderx_linux_gcc (92%)
 create mode 100644 config/arm/thunderx/meson.build
 create mode 100644 config/arm/thunderxt88/arm64_thunderxt88_linux_gcc
 create mode 100644 config/arm/thunderxt88/meson.build

-- 
2.20.1

[dpdk-dev] [RFC PATCH 1/2] build: rework Arm build options

[Juraj Linkeš]

The current system can identify only the implementer and part number
of the arm device we're targeting, which is enough to identify the
target CPU. However, even the full MIDR information is not enough to
identify the SoC we're targeting.

Expand the "machine" meson variable to allow specifying the target arm
SoC. The SoC identification implies the CPU, so drop the current MIDR
based CPU identification in favor of user input, which is the only way
to identify the target SoC. This implies that native builds are not
possible. Instead do a default (non-optimized, but executable on any
aarch64 device) build.

Also use the machine custom property in cross files for cross builds.

An example of where the current system is insufficient is the cortex-a72
CPU. These cores are used in a variety of SoC from Broadcomm, NXP,
Huawei and others. The MIDR information would only identify the core,
but there's not way to identify the SoC to fine tune the build to it.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/arm64_armv8_linux_gcc              |  28 ----
 config/arm/default/arm64_armv8_linux_gcc      |  23 +++
 config/arm/default/meson.build                |   8 +
 config/arm/meson.build                        | 139 +++++++++++-------
 .../{ => thunderx}/arm64_thunderx_linux_gcc   |   2 +-
 config/arm/thunderx/meson.build               |  18 +++
 .../thunderxt88/arm64_thunderxt88_linux_gcc   |  16 ++
 config/arm/thunderxt88/meson.build            |  18 +++
 config/meson.build                            |   9 +-
 9 files changed, 177 insertions(+), 84 deletions(-)
 delete mode 100644 config/arm/arm64_armv8_linux_gcc
 create mode 100644 config/arm/default/arm64_armv8_linux_gcc
 create mode 100644 config/arm/default/meson.build
 rename config/arm/{ => thunderx}/arm64_thunderx_linux_gcc (92%)
 create mode 100644 config/arm/thunderx/meson.build
 create mode 100644 config/arm/thunderxt88/arm64_thunderxt88_linux_gcc
 create mode 100644 config/arm/thunderxt88/meson.build

diff --git a/config/arm/arm64_armv8_linux_gcc b/config/arm/arm64_armv8_linux_gcc
deleted file mode 100644
index 88f0ff9da..000000000
--- a/config/arm/arm64_armv8_linux_gcc
+++ /dev/null
@@ -1,28 +0,0 @@
-[binaries]
-c = 'aarch64-linux-gnu-gcc'
-cpp = 'aarch64-linux-gnu-cpp'
-ar = 'aarch64-linux-gnu-gcc-ar'
-strip = 'aarch64-linux-gnu-strip'
-pkgconfig = 'aarch64-linux-gnu-pkg-config'
-pcap-config = ''
-
-[host_machine]
-system = 'linux'
-cpu_family = 'aarch64'
-cpu = 'armv8-a'
-endian = 'little'
-
-[properties]
-implementor_id = 'generic'
-
-# Valid options for Arm's implementor_pn:
-# 'default': valid for all armv8-a architectures (default value)
-# '0xd03':   cortex-a53
-# '0xd04':   cortex-a35
-# '0xd05':   cortex-a55
-# '0xd07':   cortex-a57
-# '0xd08':   cortex-a72
-# '0xd09':   cortex-a73
-# '0xd0a':   cortex-a75
-# '0xd0b':   cortex-a76
-implementor_pn = 'default'
diff --git a/config/arm/default/arm64_armv8_linux_gcc b/config/arm/default/arm64_armv8_linux_gcc
new file mode 100644
index 000000000..56efd7a05
--- /dev/null
+++ b/config/arm/default/arm64_armv8_linux_gcc
@@ -0,0 +1,23 @@
+[binaries]
+c = 'aarch64-linux-gnu-gcc'
+cpp = 'aarch64-linux-gnu-cpp'
+ar = 'aarch64-linux-gnu-gcc-ar'
+strip = 'aarch64-linux-gnu-strip'
+pkgconfig = 'aarch64-linux-gnu-pkg-config'
+pcap-config = ''
+
+[host_machine]
+system = 'linux'
+cpu_family = 'aarch64'
+cpu = 'armv8-a'
+endian = 'little'
+
+[properties]
+# Valid values for the machine property are the soc names identified by
+# directories in config/arm
+# Possible suffixes for socs:
+# -mn-kc
+# where m is the number of numa nodes and k is the number of lcores
+# e.g. for thunderx soc the possible values are "thunderx" or "thunderx-1n-48c"
+# the suffixes themselves need to be supported in config/arm/<soc_dir>/meson.build
+machine = 'default'
diff --git a/config/arm/default/meson.build b/config/arm/default/meson.build
new file mode 100644
index 000000000..5714d0084
--- /dev/null
+++ b/config/arm/default/meson.build
@@ -0,0 +1,8 @@
+flags = [
+	['RTE_MACHINE', '"armv8a"'],
+	['RTE_MAX_LCORE', 256],
+	['RTE_USE_C11_MEM_MODEL', true],
+	['RTE_CACHE_LINE_SIZE', 128]
+]
+
+machine_args = ['-march=armv8-a+crc']
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 8728051d5..05b790f40 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -2,11 +2,11 @@
 # Copyright(c) 2017 Intel Corporation.
 # Copyright(c) 2017 Cavium, Inc
 
-# for checking defines we need to use the correct compiler flags
-march_opt = '-march=@0@'.format(machine)
-
-arm_force_native_march = false
-arm_force_default_march = (machine == 'default')
+if machine == 'native'
+	# arm doesn't support a native build - user input of soc is required
+	# default to the portable build instead of native
+	machine = 'default'
+endif
 
 flags_common_default = [
 	# Accelarate rte_memcpy. Be sure to run unit test (memcpy_perf_autotest)
@@ -131,6 +131,7 @@ impl_dpaa = ['NXP DPAA', flags_dpaa, machine_args_generic]
 dpdk_conf.set('RTE_FORCE_INTRINSICS', 1)
 
 if not dpdk_conf.get('RTE_ARCH_64')
+	# armv7 build
 	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
 	dpdk_conf.set('RTE_ARCH_ARM', 1)
 	dpdk_conf.set('RTE_ARCH_ARMv7', 1)
@@ -138,41 +139,9 @@ if not dpdk_conf.get('RTE_ARCH_64')
 	# mk/machine/armv7a/rte.vars.mk sets it too
 	machine_args += '-mfpu=neon'
 else
+	# aarch64 build
 	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
 	dpdk_conf.set('RTE_ARCH_ARM64', 1)
-
-	machine = []
-	cmd_generic = ['generic', '', '', 'default', '']
-	cmd_output = cmd_generic # Set generic by default
-	machine_args = [] # Clear previous machine args
-	if arm_force_default_march and not meson.is_cross_build()
-		machine = impl_generic
-		impl_pn = 'default'
-	elif not meson.is_cross_build()
-		# The script returns ['Implementer', 'Variant', 'Architecture',
-		# 'Primary Part number', 'Revision']
-		detect_vendor = find_program(join_paths(
-				meson.current_source_dir(), 'armv8_machine.py'))
-		cmd = run_command(detect_vendor.path())
-		if cmd.returncode() == 0
-			cmd_output = cmd.stdout().to_lower().strip().split(' ')
-		endif
-		# Set to generic if variable is not found
-		machine = get_variable('impl_' + cmd_output[0], ['generic'])
-		if machine[0] == 'generic'
-			machine = impl_generic
-			cmd_output = cmd_generic
-		endif
-		impl_pn = cmd_output[3]
-		if arm_force_native_march == true
-			impl_pn = 'native'
-		endif
-	else
-		impl_id = meson.get_cross_property('implementor_id', 'generic')
-		impl_pn = meson.get_cross_property('implementor_pn', 'default')
-		machine = get_variable('impl_' + impl_id)
-	endif
-
 	# Apply Common Defaults. These settings may be overwritten by machine
 	# settings later.
 	foreach flag: flags_common_default
@@ -181,30 +150,89 @@ else
 		endif
 	endforeach
 
-	message('Implementer : ' + machine[0])
-	foreach flag: machine[1]
+	soc = machine.split('-')[0]
+	# TODO make sure the subdir exists
+	subdir(soc)
+
+	if soc != 'default'
+		# combine common and soc specific flags
+		# TODO check that soc is in the dict
+		flags = flags['common'] + flags[soc]
+	endif
+
+	foreach flag: flags
 		if flag.length() > 0
 			dpdk_conf.set(flag[0], flag[1])
 		endif
 	endforeach
 
-	foreach marg: machine[2]
-		if marg[0] == impl_pn
-			foreach flag: marg[1]
-				if cc.has_argument(flag)
-					machine_args += flag
-				endif
-			endforeach
-			# Apply any extra machine specific flags.
-			foreach flag: marg.get(2, flags_default_extra)
-				if flag.length() > 0
-					dpdk_conf.set(flag[0], flag[1])
-				endif
-			endforeach
+	verified_machine_args = []
+
+	foreach flag: machine_args
+		if cc.has_argument(flag)
+			verified_machine_args += flag
+		else
+			message('Unsupported machine flag "@0@", ignoring'.format(flag))
 		endif
 	endforeach
+	machine_args = verified_machine_args
+
+#	machine = []
+#	cmd_generic = ['generic', '', '', 'default', '']
+#	cmd_output = cmd_generic # Set generic by default
+#	machine_args = [] # Clear previous machine args
+#	if arm_force_default_march and not meson.is_cross_build()
+#		machine = impl_generic
+#		impl_pn = 'default'
+#	elif not meson.is_cross_build()
+#		# The script returns ['Implementer', 'Variant', 'Architecture',
+#		# 'Primary Part number', 'Revision']
+#		detect_vendor = find_program(join_paths(
+#				meson.current_source_dir(), 'armv8_machine.py'))
+#		cmd = run_command(detect_vendor.path())
+#		if cmd.returncode() == 0
+#			cmd_output = cmd.stdout().to_lower().strip().split(' ')
+#		endif
+#		# Set to generic if variable is not found
+#		machine = get_variable('impl_' + cmd_output[0], ['generic'])
+#		if machine[0] == 'generic'
+#			machine = impl_generic
+#			cmd_output = cmd_generic
+#		endif
+#		impl_pn = cmd_output[3]
+#		if arm_force_native_march == true
+#			impl_pn = 'native'
+#		endif
+#	else
+#		impl_id = meson.get_cross_property('implementor_id', 'generic')
+#		impl_pn = meson.get_cross_property('implementor_pn', 'default')
+#		machine = get_variable('impl_' + impl_id)
+#	endif
+#
+#	message('Implementer : ' + machine[0])
+#	foreach flag: machine[1]
+#		if flag.length() > 0
+#			dpdk_conf.set(flag[0], flag[1])
+#		endif
+#	endforeach
+#
+#	foreach marg: machine[2]
+#		if marg[0] == impl_pn
+#			foreach flag: marg[1]
+#				if cc.has_argument(flag)
+#					machine_args += flag
+#				endif
+#			endforeach
+#			# Apply any extra machine specific flags.
+#			foreach flag: marg.get(2, flags_default_extra)
+#				if flag.length() > 0
+#					dpdk_conf.set(flag[0], flag[1])
+#				endif
+#			endforeach
+#		endif
+#	endforeach
 endif
-message(machine_args)
+message('Using machine args: @0@'.format(machine_args))
 
 if (cc.get_define('__ARM_NEON', args: machine_args) != '' or
     cc.get_define('__aarch64__', args: machine_args) != '')
@@ -225,3 +253,6 @@ if cc.get_define('__ARM_FEATURE_CRYPTO', args: machine_args) != ''
 	compile_time_cpuflags += ['RTE_CPUFLAG_AES', 'RTE_CPUFLAG_PMULL',
 	'RTE_CPUFLAG_SHA1', 'RTE_CPUFLAG_SHA2']
 endif
+
+message('dpdk conf options after arm stuff: @0@'.format(dpdk_conf))
+
diff --git a/config/arm/arm64_thunderx_linux_gcc b/config/arm/thunderx/arm64_thunderx_linux_gcc
similarity index 92%
rename from config/arm/arm64_thunderx_linux_gcc
rename to config/arm/thunderx/arm64_thunderx_linux_gcc
index 6572ab615..e53a86e8f 100644
--- a/config/arm/arm64_thunderx_linux_gcc
+++ b/config/arm/thunderx/arm64_thunderx_linux_gcc
@@ -13,4 +13,4 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x43'
+machine = 'thunderx'
diff --git a/config/arm/thunderx/meson.build b/config/arm/thunderx/meson.build
new file mode 100644
index 000000000..873d79a64
--- /dev/null
+++ b/config/arm/thunderx/meson.build
@@ -0,0 +1,18 @@
+flags = {
+	'common': [
+		['RTE_CACHE_LINE_SIZE', 128],
+		['RTE_MAX_VFIO_GROUPS', 128],
+		['RTE_MACHINE', '"thunderx"'],
+		['RTE_USE_C11_MEM_MODEL', false]
+	],
+	'thunderx': [
+		['RTE_MAX_NUMA_NODES', 2],
+		['RTE_MAX_LCORE', 96]
+	],
+	'thunderx-1n-48c': [
+		['RTE_MAX_NUMA_NODES', 1],
+		['RTE_MAX_LCORE', 48]
+	]
+}
+
+machine_args = ['-march=armv8-a+crc+crypto','-mcpu=thunderx']
diff --git a/config/arm/thunderxt88/arm64_thunderxt88_linux_gcc b/config/arm/thunderxt88/arm64_thunderxt88_linux_gcc
new file mode 100644
index 000000000..9a622786c
--- /dev/null
+++ b/config/arm/thunderxt88/arm64_thunderxt88_linux_gcc
@@ -0,0 +1,16 @@
+[binaries]
+c = 'aarch64-linux-gnu-gcc'
+cpp = 'aarch64-linux-gnu-cpp'
+ar = 'aarch64-linux-gnu-gcc-ar'
+strip = 'aarch64-linux-gnu-strip'
+pkgconfig = 'aarch64-linux-gnu-pkg-config'
+pcap-config = ''
+
+[host_machine]
+system = 'linux'
+cpu_family = 'aarch64'
+cpu = 'armv8-a'
+endian = 'little'
+
+[properties]
+machine = 'thunderxt88'
diff --git a/config/arm/thunderxt88/meson.build b/config/arm/thunderxt88/meson.build
new file mode 100644
index 000000000..2aa5d2cdd
--- /dev/null
+++ b/config/arm/thunderxt88/meson.build
@@ -0,0 +1,18 @@
+flags = {
+	'common': [
+		['RTE_CACHE_LINE_SIZE', 128],
+		['RTE_MAX_VFIO_GROUPS', 128],
+		['RTE_MACHINE', '"thunderx"'],
+		['RTE_USE_C11_MEM_MODEL', false]
+	],
+	'thunderxt88': [
+		['RTE_MAX_NUMA_NODES', 2],
+		['RTE_MAX_LCORE', 96]
+	],
+	'thunderxt88-1n-48c': [
+		['RTE_MAX_NUMA_NODES', 1],
+		['RTE_MAX_LCORE', 48]
+	]
+}
+
+machine_args = ['-mcpu=thunderxt88']
diff --git a/config/meson.build b/config/meson.build
index 69f2aeb60..177cbd49c 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -63,7 +63,14 @@ meson.add_install_script('../buildtools/symlink-drivers-solibs.sh',
 
 # set the machine type and cflags for it
 if meson.is_cross_build()
-	machine = host_machine.cpu()
+	if not host_machine.cpu_family().startswith('aarch')
+		# don't change the machine config for aarch32/64 builds
+		# that config is set in the cross file to identify the soc
+		# we're building for
+		machine = host_machine.cpu()
+	else
+		machine = meson.get_cross_property('machine', 'default')
+	endif
 else
 	machine = get_option('machine')
 endif
-- 
2.20.1

Re: [dpdk-dev] [RFC PATCH 1/2] build: rework Arm build options

[Honnappa Nagarahalli]

Thanks Juraj for the quick RFC. Few comments inline.

<snip>

> 
> The current system can identify only the implementer and part number of
> the arm device we're targeting, which is enough to identify the target CPU.
> However, even the full MIDR information is not enough to identify the SoC
> we're targeting.
> 
> Expand the "machine" meson variable to allow specifying the target arm SoC.
> The SoC identification implies the CPU, so drop the current MIDR based CPU
> identification in favor of user input, which is the only way to identify the
> target SoC. This implies that native builds are not possible. Instead do a
> default (non-optimized, but executable on any
> aarch64 device) build.
> 
> Also use the machine custom property in cross files for cross builds.
> 
> An example of where the current system is insufficient is the cortex-a72 CPU.
> These cores are used in a variety of SoC from Broadcomm, NXP, Huawei and
> others. The MIDR information would only identify the core, but there's not
> way to identify the SoC to fine tune the build to it.
> 
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
>  config/arm/arm64_armv8_linux_gcc              |  28 ----
>  config/arm/default/arm64_armv8_linux_gcc      |  23 +++
>  config/arm/default/meson.build                |   8 +
>  config/arm/meson.build                        | 139 +++++++++++-------
>  .../{ => thunderx}/arm64_thunderx_linux_gcc   |   2 +-
>  config/arm/thunderx/meson.build               |  18 +++
>  .../thunderxt88/arm64_thunderxt88_linux_gcc   |  16 ++
>  config/arm/thunderxt88/meson.build            |  18 +++
>  config/meson.build                            |   9 +-
>  9 files changed, 177 insertions(+), 84 deletions(-)  delete mode 100644
> config/arm/arm64_armv8_linux_gcc  create mode 100644
> config/arm/default/arm64_armv8_linux_gcc
>  create mode 100644 config/arm/default/meson.build  rename
> config/arm/{ => thunderx}/arm64_thunderx_linux_gcc (92%)  create mode
> 100644 config/arm/thunderx/meson.build  create mode 100644
> config/arm/thunderxt88/arm64_thunderxt88_linux_gcc
>  create mode 100644 config/arm/thunderxt88/meson.build
> 
> diff --git a/config/arm/arm64_armv8_linux_gcc
> b/config/arm/arm64_armv8_linux_gcc
> deleted file mode 100644
> index 88f0ff9da..000000000
> --- a/config/arm/arm64_armv8_linux_gcc
> +++ /dev/null
> @@ -1,28 +0,0 @@
> -[binaries]
> -c = 'aarch64-linux-gnu-gcc'
> -cpp = 'aarch64-linux-gnu-cpp'
> -ar = 'aarch64-linux-gnu-gcc-ar'
> -strip = 'aarch64-linux-gnu-strip'
> -pkgconfig = 'aarch64-linux-gnu-pkg-config'
> -pcap-config = ''
> -
> -[host_machine]
> -system = 'linux'
> -cpu_family = 'aarch64'
> -cpu = 'armv8-a'
> -endian = 'little'
> -
> -[properties]
> -implementor_id = 'generic'
> -
> -# Valid options for Arm's implementor_pn:
> -# 'default': valid for all armv8-a architectures (default value)
> -# '0xd03':   cortex-a53
> -# '0xd04':   cortex-a35
> -# '0xd05':   cortex-a55
> -# '0xd07':   cortex-a57
> -# '0xd08':   cortex-a72
> -# '0xd09':   cortex-a73
> -# '0xd0a':   cortex-a75
> -# '0xd0b':   cortex-a76
> -implementor_pn = 'default'
> diff --git a/config/arm/default/arm64_armv8_linux_gcc
> b/config/arm/default/arm64_armv8_linux_gcc
> new file mode 100644
> index 000000000..56efd7a05
> --- /dev/null
> +++ b/config/arm/default/arm64_armv8_linux_gcc
> @@ -0,0 +1,23 @@
> +[binaries]
> +c = 'aarch64-linux-gnu-gcc'
> +cpp = 'aarch64-linux-gnu-cpp'
> +ar = 'aarch64-linux-gnu-gcc-ar'
> +strip = 'aarch64-linux-gnu-strip'
> +pkgconfig = 'aarch64-linux-gnu-pkg-config'
> +pcap-config = ''
> +
> +[host_machine]
> +system = 'linux'
> +cpu_family = 'aarch64'
> +cpu = 'armv8-a'
> +endian = 'little'
> +
> +[properties]
> +# Valid values for the machine property are the soc names identified by
> +# directories in config/arm # Possible suffixes for socs:
> +# -mn-kc
> +# where m is the number of numa nodes and k is the number of lcores #
> +e.g. for thunderx soc the possible values are "thunderx" or "thunderx-1n-
> 48c"
Is it required to talk about the suffixes? I think the statement "Valid values for the machine property are the soc names identified by directories in config/arm" should be sufficient.

> +# the suffixes themselves need to be supported in
> +config/arm/<soc_dir>/meson.build machine = 'default'
IMO, "default" does not convey much information. Is it possible to change it without a deprecation notice? "generic" or "armv8" conveys more information.
Any opinions from others?

> diff --git a/config/arm/default/meson.build
> b/config/arm/default/meson.build new file mode 100644 index
> 000000000..5714d0084
> --- /dev/null
> +++ b/config/arm/default/meson.build
> @@ -0,0 +1,8 @@
> +flags = [
> +	['RTE_MACHINE', '"armv8a"'],
> +	['RTE_MAX_LCORE', 256],
> +	['RTE_USE_C11_MEM_MODEL', true],
> +	['RTE_CACHE_LINE_SIZE', 128]
> +]
> +
> +machine_args = ['-march=armv8-a+crc']
> diff --git a/config/arm/meson.build b/config/arm/meson.build index
> 8728051d5..05b790f40 100644
> --- a/config/arm/meson.build
> +++ b/config/arm/meson.build
> @@ -2,11 +2,11 @@
>  # Copyright(c) 2017 Intel Corporation.
>  # Copyright(c) 2017 Cavium, Inc
> 
> -# for checking defines we need to use the correct compiler flags -march_opt
> = '-march=@0@'.format(machine)
> -
> -arm_force_native_march = false
> -arm_force_default_march = (machine == 'default')
> +if machine == 'native'
> +	# arm doesn't support a native build - user input of soc is required
> +	# default to the portable build instead of native
> +	machine = 'default'
Do we need a message here to the user?

> +endif
> 
>  flags_common_default = [
>  	# Accelarate rte_memcpy. Be sure to run unit test
> (memcpy_perf_autotest) @@ -131,6 +131,7 @@ impl_dpaa = ['NXP DPAA',
> flags_dpaa, machine_args_generic]  dpdk_conf.set('RTE_FORCE_INTRINSICS',
> 1)
> 
>  if not dpdk_conf.get('RTE_ARCH_64')
> +	# armv7 build
>  	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
>  	dpdk_conf.set('RTE_ARCH_ARM', 1)
>  	dpdk_conf.set('RTE_ARCH_ARMv7', 1)
> @@ -138,41 +139,9 @@ if not dpdk_conf.get('RTE_ARCH_64')
>  	# mk/machine/armv7a/rte.vars.mk sets it too
>  	machine_args += '-mfpu=neon'
>  else
> +	# aarch64 build
>  	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
>  	dpdk_conf.set('RTE_ARCH_ARM64', 1)
> -
> -	machine = []
> -	cmd_generic = ['generic', '', '', 'default', '']
> -	cmd_output = cmd_generic # Set generic by default
> -	machine_args = [] # Clear previous machine args
> -	if arm_force_default_march and not meson.is_cross_build()
> -		machine = impl_generic
> -		impl_pn = 'default'
> -	elif not meson.is_cross_build()
> -		# The script returns ['Implementer', 'Variant', 'Architecture',
> -		# 'Primary Part number', 'Revision']
> -		detect_vendor = find_program(join_paths(
> -				meson.current_source_dir(),
> 'armv8_machine.py'))
> -		cmd = run_command(detect_vendor.path())
> -		if cmd.returncode() == 0
> -			cmd_output = cmd.stdout().to_lower().strip().split(' ')
> -		endif
> -		# Set to generic if variable is not found
> -		machine = get_variable('impl_' + cmd_output[0], ['generic'])
> -		if machine[0] == 'generic'
> -			machine = impl_generic
> -			cmd_output = cmd_generic
> -		endif
> -		impl_pn = cmd_output[3]
> -		if arm_force_native_march == true
> -			impl_pn = 'native'
> -		endif
> -	else
> -		impl_id = meson.get_cross_property('implementor_id',
> 'generic')
> -		impl_pn = meson.get_cross_property('implementor_pn',
> 'default')
> -		machine = get_variable('impl_' + impl_id)
> -	endif
> -
>  	# Apply Common Defaults. These settings may be overwritten by
> machine
>  	# settings later.
>  	foreach flag: flags_common_default
> @@ -181,30 +150,89 @@ else
>  		endif
>  	endforeach
> 
> -	message('Implementer : ' + machine[0])
> -	foreach flag: machine[1]
> +	soc = machine.split('-')[0]
> +	# TODO make sure the subdir exists
> +	subdir(soc)
> +
> +	if soc != 'default'
> +		# combine common and soc specific flags
> +		# TODO check that soc is in the dict
> +		flags = flags['common'] + flags[soc]
> +	endif
> +
> +	foreach flag: flags
>  		if flag.length() > 0
>  			dpdk_conf.set(flag[0], flag[1])
>  		endif
>  	endforeach
> 
> -	foreach marg: machine[2]
> -		if marg[0] == impl_pn
> -			foreach flag: marg[1]
> -				if cc.has_argument(flag)
> -					machine_args += flag
> -				endif
> -			endforeach
> -			# Apply any extra machine specific flags.
> -			foreach flag: marg.get(2, flags_default_extra)
> -				if flag.length() > 0
> -					dpdk_conf.set(flag[0], flag[1])
> -				endif
> -			endforeach
> +	verified_machine_args = []
> +
> +	foreach flag: machine_args
> +		if cc.has_argument(flag)
> +			verified_machine_args += flag
> +		else
> +			message('Unsupported machine flag "@0@",
> ignoring'.format(flag))
>  		endif
>  	endforeach
> +	machine_args = verified_machine_args
> +
> +#	machine = []
> +#	cmd_generic = ['generic', '', '', 'default', '']
> +#	cmd_output = cmd_generic # Set generic by default
> +#	machine_args = [] # Clear previous machine args
> +#	if arm_force_default_march and not meson.is_cross_build()
> +#		machine = impl_generic
> +#		impl_pn = 'default'
> +#	elif not meson.is_cross_build()
Isn't this for native build? Do we still need this?

> +#		# The script returns ['Implementer', 'Variant', 'Architecture',
> +#		# 'Primary Part number', 'Revision']
> +#		detect_vendor = find_program(join_paths(
> +#				meson.current_source_dir(),
> 'armv8_machine.py'))
> +#		cmd = run_command(detect_vendor.path())
> +#		if cmd.returncode() == 0
> +#			cmd_output = cmd.stdout().to_lower().strip().split(' ')
> +#		endif
> +#		# Set to generic if variable is not found
> +#		machine = get_variable('impl_' + cmd_output[0], ['generic'])
> +#		if machine[0] == 'generic'
> +#			machine = impl_generic
> +#			cmd_output = cmd_generic
> +#		endif
> +#		impl_pn = cmd_output[3]
> +#		if arm_force_native_march == true
> +#			impl_pn = 'native'
> +#		endif
> +#	else
> +#		impl_id = meson.get_cross_property('implementor_id',
> 'generic')
> +#		impl_pn = meson.get_cross_property('implementor_pn',
> 'default')
> +#		machine = get_variable('impl_' + impl_id)
> +#	endif
> +#
> +#	message('Implementer : ' + machine[0])
> +#	foreach flag: machine[1]
> +#		if flag.length() > 0
> +#			dpdk_conf.set(flag[0], flag[1])
> +#		endif
> +#	endforeach
> +#
> +#	foreach marg: machine[2]
> +#		if marg[0] == impl_pn
> +#			foreach flag: marg[1]
> +#				if cc.has_argument(flag)
> +#					machine_args += flag
> +#				endif
> +#			endforeach
> +#			# Apply any extra machine specific flags.
> +#			foreach flag: marg.get(2, flags_default_extra)
> +#				if flag.length() > 0
> +#					dpdk_conf.set(flag[0], flag[1])
> +#				endif
> +#			endforeach
> +#		endif
> +#	endforeach
>  endif
> -message(machine_args)
> +message('Using machine args: @0@'.format(machine_args))
> 
>  if (cc.get_define('__ARM_NEON', args: machine_args) != '' or
>      cc.get_define('__aarch64__', args: machine_args) != '') @@ -225,3 +253,6
> @@ if cc.get_define('__ARM_FEATURE_CRYPTO', args: machine_args) != ''
>  	compile_time_cpuflags += ['RTE_CPUFLAG_AES',
> 'RTE_CPUFLAG_PMULL',
>  	'RTE_CPUFLAG_SHA1', 'RTE_CPUFLAG_SHA2']  endif
> +
> +message('dpdk conf options after arm stuff: @0@'.format(dpdk_conf))
> +
> diff --git a/config/arm/arm64_thunderx_linux_gcc
> b/config/arm/thunderx/arm64_thunderx_linux_gcc
> similarity index 92%
> rename from config/arm/arm64_thunderx_linux_gcc
> rename to config/arm/thunderx/arm64_thunderx_linux_gcc
> index 6572ab615..e53a86e8f 100644
> --- a/config/arm/arm64_thunderx_linux_gcc
> +++ b/config/arm/thunderx/arm64_thunderx_linux_gcc
> @@ -13,4 +13,4 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = '0x43'
> +machine = 'thunderx'
> diff --git a/config/arm/thunderx/meson.build
> b/config/arm/thunderx/meson.build new file mode 100644 index
> 000000000..873d79a64
> --- /dev/null
> +++ b/config/arm/thunderx/meson.build
> @@ -0,0 +1,18 @@
> +flags = {
> +	'common': [
> +		['RTE_CACHE_LINE_SIZE', 128],
> +		['RTE_MAX_VFIO_GROUPS', 128],
> +		['RTE_MACHINE', '"thunderx"'],
> +		['RTE_USE_C11_MEM_MODEL', false]
> +	],
> +	'thunderx': [
> +		['RTE_MAX_NUMA_NODES', 2],
> +		['RTE_MAX_LCORE', 96]
> +	],
> +	'thunderx-1n-48c': [
> +		['RTE_MAX_NUMA_NODES', 1],
> +		['RTE_MAX_LCORE', 48]
> +	]
> +}
> +
> +machine_args = ['-march=armv8-a+crc+crypto','-mcpu=thunderx']
> diff --git a/config/arm/thunderxt88/arm64_thunderxt88_linux_gcc
> b/config/arm/thunderxt88/arm64_thunderxt88_linux_gcc
> new file mode 100644
> index 000000000..9a622786c
> --- /dev/null
> +++ b/config/arm/thunderxt88/arm64_thunderxt88_linux_gcc
> @@ -0,0 +1,16 @@
> +[binaries]
> +c = 'aarch64-linux-gnu-gcc'
> +cpp = 'aarch64-linux-gnu-cpp'
> +ar = 'aarch64-linux-gnu-gcc-ar'
> +strip = 'aarch64-linux-gnu-strip'
> +pkgconfig = 'aarch64-linux-gnu-pkg-config'
> +pcap-config = ''
> +
> +[host_machine]
> +system = 'linux'
> +cpu_family = 'aarch64'
> +cpu = 'armv8-a'
> +endian = 'little'
> +
> +[properties]
> +machine = 'thunderxt88'
> diff --git a/config/arm/thunderxt88/meson.build
> b/config/arm/thunderxt88/meson.build
> new file mode 100644
> index 000000000..2aa5d2cdd
> --- /dev/null
> +++ b/config/arm/thunderxt88/meson.build
> @@ -0,0 +1,18 @@
> +flags = {
> +	'common': [
> +		['RTE_CACHE_LINE_SIZE', 128],
> +		['RTE_MAX_VFIO_GROUPS', 128],
> +		['RTE_MACHINE', '"thunderx"'],
> +		['RTE_USE_C11_MEM_MODEL', false]
> +	],
> +	'thunderxt88': [
> +		['RTE_MAX_NUMA_NODES', 2],
> +		['RTE_MAX_LCORE', 96]
> +	],
> +	'thunderxt88-1n-48c': [
> +		['RTE_MAX_NUMA_NODES', 1],
> +		['RTE_MAX_LCORE', 48]
> +	]
> +}
> +
> +machine_args = ['-mcpu=thunderxt88']
> diff --git a/config/meson.build b/config/meson.build index
> 69f2aeb60..177cbd49c 100644
> --- a/config/meson.build
> +++ b/config/meson.build
> @@ -63,7 +63,14 @@ meson.add_install_script('../buildtools/symlink-
> drivers-solibs.sh',
> 
>  # set the machine type and cflags for it  if meson.is_cross_build()
> -	machine = host_machine.cpu()
> +	if not host_machine.cpu_family().startswith('aarch')
> +		# don't change the machine config for aarch32/64 builds
> +		# that config is set in the cross file to identify the soc
> +		# we're building for
> +		machine = host_machine.cpu()
> +	else
> +		machine = meson.get_cross_property('machine', 'default')
> +	endif
>  else
>  	machine = get_option('machine')
>  endif
> --
> 2.20.1

Re: [dpdk-dev] [RFC PATCH 1/2] build: rework Arm build options

[Juraj Linkeš]

> -----Original Message-----
> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Sent: Thursday, September 24, 2020 6:19 AM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>; bruce.richardson@intel.com;
> Ruifeng Wang <Ruifeng.Wang@arm.com>; Phil Yang <Phil.Yang@arm.com>;
> vcchunga@amazon.com; Dharmik Thakkar <Dharmik.Thakkar@arm.com>;
> jerinjacobk@gmail.com; hemant.agrawal@nxp.com
> Cc: dev@dpdk.org; nd <nd@arm.com>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>
> Subject: RE: [RFC PATCH 1/2] build: rework Arm build options
> 
> Thanks Juraj for the quick RFC. Few comments inline.
> 
> <snip>
> 
> >
> > The current system can identify only the implementer and part number
> > of the arm device we're targeting, which is enough to identify the target CPU.
> > However, even the full MIDR information is not enough to identify the
> > SoC we're targeting.
> >
> > Expand the "machine" meson variable to allow specifying the target arm SoC.
> > The SoC identification implies the CPU, so drop the current MIDR based
> > CPU identification in favor of user input, which is the only way to
> > identify the target SoC. This implies that native builds are not
> > possible. Instead do a default (non-optimized, but executable on any
> > aarch64 device) build.
> >
> > Also use the machine custom property in cross files for cross builds.
> >
> > An example of where the current system is insufficient is the cortex-a72 CPU.
> > These cores are used in a variety of SoC from Broadcomm, NXP, Huawei
> > and others. The MIDR information would only identify the core, but
> > there's not way to identify the SoC to fine tune the build to it.
> >
> > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > ---
> >  config/arm/arm64_armv8_linux_gcc              |  28 ----
> >  config/arm/default/arm64_armv8_linux_gcc      |  23 +++
> >  config/arm/default/meson.build                |   8 +
> >  config/arm/meson.build                        | 139 +++++++++++-------
> >  .../{ => thunderx}/arm64_thunderx_linux_gcc   |   2 +-
> >  config/arm/thunderx/meson.build               |  18 +++
> >  .../thunderxt88/arm64_thunderxt88_linux_gcc   |  16 ++
> >  config/arm/thunderxt88/meson.build            |  18 +++
> >  config/meson.build                            |   9 +-
> >  9 files changed, 177 insertions(+), 84 deletions(-)  delete mode
> > 100644 config/arm/arm64_armv8_linux_gcc  create mode 100644
> > config/arm/default/arm64_armv8_linux_gcc
> >  create mode 100644 config/arm/default/meson.build  rename
> > config/arm/{ => thunderx}/arm64_thunderx_linux_gcc (92%)  create mode
> > 100644 config/arm/thunderx/meson.build  create mode 100644
> > config/arm/thunderxt88/arm64_thunderxt88_linux_gcc
> >  create mode 100644 config/arm/thunderxt88/meson.build
> >
> > diff --git a/config/arm/arm64_armv8_linux_gcc
> > b/config/arm/arm64_armv8_linux_gcc
> > deleted file mode 100644
> > index 88f0ff9da..000000000
> > --- a/config/arm/arm64_armv8_linux_gcc
> > +++ /dev/null
> > @@ -1,28 +0,0 @@
> > -[binaries]
> > -c = 'aarch64-linux-gnu-gcc'
> > -cpp = 'aarch64-linux-gnu-cpp'
> > -ar = 'aarch64-linux-gnu-gcc-ar'
> > -strip = 'aarch64-linux-gnu-strip'
> > -pkgconfig = 'aarch64-linux-gnu-pkg-config'
> > -pcap-config = ''
> > -
> > -[host_machine]
> > -system = 'linux'
> > -cpu_family = 'aarch64'
> > -cpu = 'armv8-a'
> > -endian = 'little'
> > -
> > -[properties]
> > -implementor_id = 'generic'
> > -
> > -# Valid options for Arm's implementor_pn:
> > -# 'default': valid for all armv8-a architectures (default value)
> > -# '0xd03':   cortex-a53
> > -# '0xd04':   cortex-a35
> > -# '0xd05':   cortex-a55
> > -# '0xd07':   cortex-a57
> > -# '0xd08':   cortex-a72
> > -# '0xd09':   cortex-a73
> > -# '0xd0a':   cortex-a75
> > -# '0xd0b':   cortex-a76
> > -implementor_pn = 'default'
> > diff --git a/config/arm/default/arm64_armv8_linux_gcc
> > b/config/arm/default/arm64_armv8_linux_gcc
> > new file mode 100644
> > index 000000000..56efd7a05
> > --- /dev/null
> > +++ b/config/arm/default/arm64_armv8_linux_gcc
> > @@ -0,0 +1,23 @@
> > +[binaries]
> > +c = 'aarch64-linux-gnu-gcc'
> > +cpp = 'aarch64-linux-gnu-cpp'
> > +ar = 'aarch64-linux-gnu-gcc-ar'
> > +strip = 'aarch64-linux-gnu-strip'
> > +pkgconfig = 'aarch64-linux-gnu-pkg-config'
> > +pcap-config = ''
> > +
> > +[host_machine]
> > +system = 'linux'
> > +cpu_family = 'aarch64'
> > +cpu = 'armv8-a'
> > +endian = 'little'
> > +
> > +[properties]
> > +# Valid values for the machine property are the soc names identified
> > +by # directories in config/arm # Possible suffixes for socs:
> > +# -mn-kc
> > +# where m is the number of numa nodes and k is the number of lcores #
> > +e.g. for thunderx soc the possible values are "thunderx" or
> > +"thunderx-1n-
> > 48c"
> Is it required to talk about the suffixes? I think the statement "Valid values for
> the machine property are the soc names identified by directories in config/arm"
> should be sufficient.
> 

Yes, this is not the right place for this, so a shorted explanation is enough.

> > +# the suffixes themselves need to be supported in
> > +config/arm/<soc_dir>/meson.build machine = 'default'
> IMO, "default" does not convey much information. Is it possible to change it
> without a deprecation notice? "generic" or "armv8" conveys more information.
> Any opinions from others?
> 

Maybe we can have multiple labels mean the same thing - default, generic, armv8 or even generic-armv8 (that's the most descriptive name IMO) could all mean the generic arm build executable on all aarch64 SoCs. That many names is probably confusing, but sticking to two (default and one more) seems feasible.

What does everyone else think?

> > diff --git a/config/arm/default/meson.build
> > b/config/arm/default/meson.build new file mode 100644 index
> > 000000000..5714d0084
> > --- /dev/null
> > +++ b/config/arm/default/meson.build
> > @@ -0,0 +1,8 @@
> > +flags = [
> > +	['RTE_MACHINE', '"armv8a"'],
> > +	['RTE_MAX_LCORE', 256],
> > +	['RTE_USE_C11_MEM_MODEL', true],
> > +	['RTE_CACHE_LINE_SIZE', 128]
> > +]
> > +
> > +machine_args = ['-march=armv8-a+crc']
> > diff --git a/config/arm/meson.build b/config/arm/meson.build index
> > 8728051d5..05b790f40 100644
> > --- a/config/arm/meson.build
> > +++ b/config/arm/meson.build
> > @@ -2,11 +2,11 @@
> >  # Copyright(c) 2017 Intel Corporation.
> >  # Copyright(c) 2017 Cavium, Inc
> >
> > -# for checking defines we need to use the correct compiler flags
> > -march_opt = '-march=@0@'.format(machine)
> > -
> > -arm_force_native_march = false
> > -arm_force_default_march = (machine == 'default')
> > +if machine == 'native'
> > +	# arm doesn't support a native build - user input of soc is required
> > +	# default to the portable build instead of native
> > +	machine = 'default'
> Do we need a message here to the user?
> 

Better to have one, good catch.

> > +endif
> >
> >  flags_common_default = [
> >  	# Accelarate rte_memcpy. Be sure to run unit test
> > (memcpy_perf_autotest) @@ -131,6 +131,7 @@ impl_dpaa = ['NXP DPAA',
> > flags_dpaa, machine_args_generic]
> > dpdk_conf.set('RTE_FORCE_INTRINSICS',
> > 1)
> >
> >  if not dpdk_conf.get('RTE_ARCH_64')
> > +	# armv7 build
> >  	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
> >  	dpdk_conf.set('RTE_ARCH_ARM', 1)
> >  	dpdk_conf.set('RTE_ARCH_ARMv7', 1)
> > @@ -138,41 +139,9 @@ if not dpdk_conf.get('RTE_ARCH_64')
> >  	# mk/machine/armv7a/rte.vars.mk sets it too
> >  	machine_args += '-mfpu=neon'
> >  else
> > +	# aarch64 build
> >  	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
> >  	dpdk_conf.set('RTE_ARCH_ARM64', 1)
> > -
> > -	machine = []
> > -	cmd_generic = ['generic', '', '', 'default', '']
> > -	cmd_output = cmd_generic # Set generic by default
> > -	machine_args = [] # Clear previous machine args
> > -	if arm_force_default_march and not meson.is_cross_build()
> > -		machine = impl_generic
> > -		impl_pn = 'default'
> > -	elif not meson.is_cross_build()
> > -		# The script returns ['Implementer', 'Variant', 'Architecture',
> > -		# 'Primary Part number', 'Revision']
> > -		detect_vendor = find_program(join_paths(
> > -				meson.current_source_dir(),
> > 'armv8_machine.py'))
> > -		cmd = run_command(detect_vendor.path())
> > -		if cmd.returncode() == 0
> > -			cmd_output = cmd.stdout().to_lower().strip().split(' ')
> > -		endif
> > -		# Set to generic if variable is not found
> > -		machine = get_variable('impl_' + cmd_output[0], ['generic'])
> > -		if machine[0] == 'generic'
> > -			machine = impl_generic
> > -			cmd_output = cmd_generic
> > -		endif
> > -		impl_pn = cmd_output[3]
> > -		if arm_force_native_march == true
> > -			impl_pn = 'native'
> > -		endif
> > -	else
> > -		impl_id = meson.get_cross_property('implementor_id',
> > 'generic')
> > -		impl_pn = meson.get_cross_property('implementor_pn',
> > 'default')
> > -		machine = get_variable('impl_' + impl_id)
> > -	endif
> > -
> >  	# Apply Common Defaults. These settings may be overwritten by
> > machine
> >  	# settings later.
> >  	foreach flag: flags_common_default
> > @@ -181,30 +150,89 @@ else
> >  		endif
> >  	endforeach
> >
> > -	message('Implementer : ' + machine[0])
> > -	foreach flag: machine[1]
> > +	soc = machine.split('-')[0]
> > +	# TODO make sure the subdir exists
> > +	subdir(soc)
> > +
> > +	if soc != 'default'
> > +		# combine common and soc specific flags
> > +		# TODO check that soc is in the dict
> > +		flags = flags['common'] + flags[soc]
> > +	endif
> > +
> > +	foreach flag: flags
> >  		if flag.length() > 0
> >  			dpdk_conf.set(flag[0], flag[1])
> >  		endif
> >  	endforeach
> >
> > -	foreach marg: machine[2]
> > -		if marg[0] == impl_pn
> > -			foreach flag: marg[1]
> > -				if cc.has_argument(flag)
> > -					machine_args += flag
> > -				endif
> > -			endforeach
> > -			# Apply any extra machine specific flags.
> > -			foreach flag: marg.get(2, flags_default_extra)
> > -				if flag.length() > 0
> > -					dpdk_conf.set(flag[0], flag[1])
> > -				endif
> > -			endforeach
> > +	verified_machine_args = []
> > +
> > +	foreach flag: machine_args
> > +		if cc.has_argument(flag)
> > +			verified_machine_args += flag
> > +		else
> > +			message('Unsupported machine flag "@0@",
> > ignoring'.format(flag))
> >  		endif
> >  	endforeach
> > +	machine_args = verified_machine_args
> > +
> > +#	machine = []
> > +#	cmd_generic = ['generic', '', '', 'default', '']
> > +#	cmd_output = cmd_generic # Set generic by default
> > +#	machine_args = [] # Clear previous machine args
> > +#	if arm_force_default_march and not meson.is_cross_build()
> > +#		machine = impl_generic
> > +#		impl_pn = 'default'
> > +#	elif not meson.is_cross_build()
> Isn't this for native build? Do we still need this?
> 

Yes, I should've removed this instead of commenting it out.

> > +#		# The script returns ['Implementer', 'Variant', 'Architecture',
> > +#		# 'Primary Part number', 'Revision']
> > +#		detect_vendor = find_program(join_paths(
> > +#				meson.current_source_dir(),
> > 'armv8_machine.py'))
> > +#		cmd = run_command(detect_vendor.path())
> > +#		if cmd.returncode() == 0
> > +#			cmd_output = cmd.stdout().to_lower().strip().split(' ')
> > +#		endif
> > +#		# Set to generic if variable is not found
> > +#		machine = get_variable('impl_' + cmd_output[0], ['generic'])
> > +#		if machine[0] == 'generic'
> > +#			machine = impl_generic
> > +#			cmd_output = cmd_generic
> > +#		endif
> > +#		impl_pn = cmd_output[3]
> > +#		if arm_force_native_march == true
> > +#			impl_pn = 'native'
> > +#		endif
> > +#	else
> > +#		impl_id = meson.get_cross_property('implementor_id',
> > 'generic')
> > +#		impl_pn = meson.get_cross_property('implementor_pn',
> > 'default')
> > +#		machine = get_variable('impl_' + impl_id)
> > +#	endif
> > +#
> > +#	message('Implementer : ' + machine[0])
> > +#	foreach flag: machine[1]
> > +#		if flag.length() > 0
> > +#			dpdk_conf.set(flag[0], flag[1])
> > +#		endif
> > +#	endforeach
> > +#
> > +#	foreach marg: machine[2]
> > +#		if marg[0] == impl_pn
> > +#			foreach flag: marg[1]
> > +#				if cc.has_argument(flag)
> > +#					machine_args += flag
> > +#				endif
> > +#			endforeach
> > +#			# Apply any extra machine specific flags.
> > +#			foreach flag: marg.get(2, flags_default_extra)
> > +#				if flag.length() > 0
> > +#					dpdk_conf.set(flag[0], flag[1])
> > +#				endif
> > +#			endforeach
> > +#		endif
> > +#	endforeach
> >  endif
> > -message(machine_args)
> > +message('Using machine args: @0@'.format(machine_args))
> >
> >  if (cc.get_define('__ARM_NEON', args: machine_args) != '' or
> >      cc.get_define('__aarch64__', args: machine_args) != '') @@ -225,3
> > +253,6 @@ if cc.get_define('__ARM_FEATURE_CRYPTO', args: machine_args)
> != ''
> >  	compile_time_cpuflags += ['RTE_CPUFLAG_AES',
> 'RTE_CPUFLAG_PMULL',
> >  	'RTE_CPUFLAG_SHA1', 'RTE_CPUFLAG_SHA2']  endif
> > +
> > +message('dpdk conf options after arm stuff: @0@'.format(dpdk_conf))
> > +
> > diff --git a/config/arm/arm64_thunderx_linux_gcc
> > b/config/arm/thunderx/arm64_thunderx_linux_gcc
> > similarity index 92%
> > rename from config/arm/arm64_thunderx_linux_gcc
> > rename to config/arm/thunderx/arm64_thunderx_linux_gcc
> > index 6572ab615..e53a86e8f 100644
> > --- a/config/arm/arm64_thunderx_linux_gcc
> > +++ b/config/arm/thunderx/arm64_thunderx_linux_gcc
> > @@ -13,4 +13,4 @@ cpu = 'armv8-a'
> >  endian = 'little'
> >
> >  [properties]
> > -implementor_id = '0x43'
> > +machine = 'thunderx'
> > diff --git a/config/arm/thunderx/meson.build
> > b/config/arm/thunderx/meson.build new file mode 100644 index
> > 000000000..873d79a64
> > --- /dev/null
> > +++ b/config/arm/thunderx/meson.build
> > @@ -0,0 +1,18 @@
> > +flags = {
> > +	'common': [
> > +		['RTE_CACHE_LINE_SIZE', 128],
> > +		['RTE_MAX_VFIO_GROUPS', 128],
> > +		['RTE_MACHINE', '"thunderx"'],
> > +		['RTE_USE_C11_MEM_MODEL', false]
> > +	],
> > +	'thunderx': [
> > +		['RTE_MAX_NUMA_NODES', 2],
> > +		['RTE_MAX_LCORE', 96]
> > +	],
> > +	'thunderx-1n-48c': [
> > +		['RTE_MAX_NUMA_NODES', 1],
> > +		['RTE_MAX_LCORE', 48]
> > +	]
> > +}
> > +
> > +machine_args = ['-march=armv8-a+crc+crypto','-mcpu=thunderx']
> > diff --git a/config/arm/thunderxt88/arm64_thunderxt88_linux_gcc
> > b/config/arm/thunderxt88/arm64_thunderxt88_linux_gcc
> > new file mode 100644
> > index 000000000..9a622786c
> > --- /dev/null
> > +++ b/config/arm/thunderxt88/arm64_thunderxt88_linux_gcc
> > @@ -0,0 +1,16 @@
> > +[binaries]
> > +c = 'aarch64-linux-gnu-gcc'
> > +cpp = 'aarch64-linux-gnu-cpp'
> > +ar = 'aarch64-linux-gnu-gcc-ar'
> > +strip = 'aarch64-linux-gnu-strip'
> > +pkgconfig = 'aarch64-linux-gnu-pkg-config'
> > +pcap-config = ''
> > +
> > +[host_machine]
> > +system = 'linux'
> > +cpu_family = 'aarch64'
> > +cpu = 'armv8-a'
> > +endian = 'little'
> > +
> > +[properties]
> > +machine = 'thunderxt88'
> > diff --git a/config/arm/thunderxt88/meson.build
> > b/config/arm/thunderxt88/meson.build
> > new file mode 100644
> > index 000000000..2aa5d2cdd
> > --- /dev/null
> > +++ b/config/arm/thunderxt88/meson.build
> > @@ -0,0 +1,18 @@
> > +flags = {
> > +	'common': [
> > +		['RTE_CACHE_LINE_SIZE', 128],
> > +		['RTE_MAX_VFIO_GROUPS', 128],
> > +		['RTE_MACHINE', '"thunderx"'],
> > +		['RTE_USE_C11_MEM_MODEL', false]
> > +	],
> > +	'thunderxt88': [
> > +		['RTE_MAX_NUMA_NODES', 2],
> > +		['RTE_MAX_LCORE', 96]
> > +	],
> > +	'thunderxt88-1n-48c': [
> > +		['RTE_MAX_NUMA_NODES', 1],
> > +		['RTE_MAX_LCORE', 48]
> > +	]
> > +}
> > +
> > +machine_args = ['-mcpu=thunderxt88']
> > diff --git a/config/meson.build b/config/meson.build index
> > 69f2aeb60..177cbd49c 100644
> > --- a/config/meson.build
> > +++ b/config/meson.build
> > @@ -63,7 +63,14 @@ meson.add_install_script('../buildtools/symlink-
> > drivers-solibs.sh',
> >
> >  # set the machine type and cflags for it  if meson.is_cross_build()
> > -	machine = host_machine.cpu()
> > +	if not host_machine.cpu_family().startswith('aarch')
> > +		# don't change the machine config for aarch32/64 builds
> > +		# that config is set in the cross file to identify the soc
> > +		# we're building for
> > +		machine = host_machine.cpu()
> > +	else
> > +		machine = meson.get_cross_property('machine', 'default')
> > +	endif
> >  else
> >  	machine = get_option('machine')
> >  endif
> > --
> > 2.20.1

Re: [dpdk-dev] [RFC PATCH 1/2] build: rework Arm build options

[Honnappa Nagarahalli]

<snip>

> >
> > >
> > > The current system can identify only the implementer and part number
> > > of the arm device we're targeting, which is enough to identify the target
> CPU.
> > > However, even the full MIDR information is not enough to identify
> > > the SoC we're targeting.
> > >
> > > Expand the "machine" meson variable to allow specifying the target arm
> SoC.
> > > The SoC identification implies the CPU, so drop the current MIDR
> > > based CPU identification in favor of user input, which is the only
> > > way to identify the target SoC. This implies that native builds are
> > > not possible. Instead do a default (non-optimized, but executable on
> > > any
> > > aarch64 device) build.
> > >
> > > Also use the machine custom property in cross files for cross builds.
> > >
> > > An example of where the current system is insufficient is the cortex-a72
> CPU.
> > > These cores are used in a variety of SoC from Broadcomm, NXP, Huawei
> > > and others. The MIDR information would only identify the core, but
> > > there's not way to identify the SoC to fine tune the build to it.
> > >
> > > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > ---
> > >  config/arm/arm64_armv8_linux_gcc              |  28 ----
> > >  config/arm/default/arm64_armv8_linux_gcc      |  23 +++
> > >  config/arm/default/meson.build                |   8 +
> > >  config/arm/meson.build                        | 139 +++++++++++-------
> > >  .../{ => thunderx}/arm64_thunderx_linux_gcc   |   2 +-
> > >  config/arm/thunderx/meson.build               |  18 +++
> > >  .../thunderxt88/arm64_thunderxt88_linux_gcc   |  16 ++
> > >  config/arm/thunderxt88/meson.build            |  18 +++
> > >  config/meson.build                            |   9 +-
> > >  9 files changed, 177 insertions(+), 84 deletions(-)  delete mode
> > > 100644 config/arm/arm64_armv8_linux_gcc  create mode 100644
> > > config/arm/default/arm64_armv8_linux_gcc
> > >  create mode 100644 config/arm/default/meson.build  rename
> > > config/arm/{ => thunderx}/arm64_thunderx_linux_gcc (92%)  create
> > > mode
> > > 100644 config/arm/thunderx/meson.build  create mode 100644
> > > config/arm/thunderxt88/arm64_thunderxt88_linux_gcc
> > >  create mode 100644 config/arm/thunderxt88/meson.build
> > >
> > > diff --git a/config/arm/arm64_armv8_linux_gcc
> > > b/config/arm/arm64_armv8_linux_gcc
> > > deleted file mode 100644
> > > index 88f0ff9da..000000000
> > > --- a/config/arm/arm64_armv8_linux_gcc
> > > +++ /dev/null
> > > @@ -1,28 +0,0 @@
> > > -[binaries]
> > > -c = 'aarch64-linux-gnu-gcc'
> > > -cpp = 'aarch64-linux-gnu-cpp'
> > > -ar = 'aarch64-linux-gnu-gcc-ar'
> > > -strip = 'aarch64-linux-gnu-strip'
> > > -pkgconfig = 'aarch64-linux-gnu-pkg-config'
> > > -pcap-config = ''
> > > -
> > > -[host_machine]
> > > -system = 'linux'
> > > -cpu_family = 'aarch64'
> > > -cpu = 'armv8-a'
> > > -endian = 'little'
> > > -
> > > -[properties]
> > > -implementor_id = 'generic'
> > > -
> > > -# Valid options for Arm's implementor_pn:
> > > -# 'default': valid for all armv8-a architectures (default value)
> > > -# '0xd03':   cortex-a53
> > > -# '0xd04':   cortex-a35
> > > -# '0xd05':   cortex-a55
> > > -# '0xd07':   cortex-a57
> > > -# '0xd08':   cortex-a72
> > > -# '0xd09':   cortex-a73
> > > -# '0xd0a':   cortex-a75
> > > -# '0xd0b':   cortex-a76
> > > -implementor_pn = 'default'
> > > diff --git a/config/arm/default/arm64_armv8_linux_gcc
> > > b/config/arm/default/arm64_armv8_linux_gcc
> > > new file mode 100644
> > > index 000000000..56efd7a05
> > > --- /dev/null
> > > +++ b/config/arm/default/arm64_armv8_linux_gcc
> > > @@ -0,0 +1,23 @@
> > > +[binaries]
> > > +c = 'aarch64-linux-gnu-gcc'
> > > +cpp = 'aarch64-linux-gnu-cpp'
> > > +ar = 'aarch64-linux-gnu-gcc-ar'
> > > +strip = 'aarch64-linux-gnu-strip'
> > > +pkgconfig = 'aarch64-linux-gnu-pkg-config'
> > > +pcap-config = ''
> > > +
> > > +[host_machine]
> > > +system = 'linux'
> > > +cpu_family = 'aarch64'
> > > +cpu = 'armv8-a'
> > > +endian = 'little'
> > > +
> > > +[properties]
> > > +# Valid values for the machine property are the soc names
> > > +identified by # directories in config/arm # Possible suffixes for socs:
> > > +# -mn-kc
> > > +# where m is the number of numa nodes and k is the number of lcores
> > > +# e.g. for thunderx soc the possible values are "thunderx" or
> > > +"thunderx-1n-
> > > 48c"
> > Is it required to talk about the suffixes? I think the statement
> > "Valid values for the machine property are the soc names identified by
> directories in config/arm"
> > should be sufficient.
> >
> 
> Yes, this is not the right place for this, so a shorted explanation is enough.
> 
> > > +# the suffixes themselves need to be supported in
> > > +config/arm/<soc_dir>/meson.build machine = 'default'
> > IMO, "default" does not convey much information. Is it possible to
> > change it without a deprecation notice? "generic" or "armv8" conveys more
> information.
> > Any opinions from others?
> >
> 
> Maybe we can have multiple labels mean the same thing - default, generic,
> armv8 or even generic-armv8 (that's the most descriptive name IMO) could
> all mean the generic arm build executable on all aarch64 SoCs. That many
> names is probably confusing, but sticking to two (default and one more)
> seems feasible.
> 
> What does everyone else think?
'armv8' makes lot more sense to me and it conveys what the build does. We can keep 'default' to be backward compatible, we can skip others.

> 
> > > diff --git a/config/arm/default/meson.build
> > > b/config/arm/default/meson.build new file mode 100644 index
> > > 000000000..5714d0084
> > > --- /dev/null
> > > +++ b/config/arm/default/meson.build
> > > @@ -0,0 +1,8 @@
> > > +flags = [
> > > +	['RTE_MACHINE', '"armv8a"'],
> > > +	['RTE_MAX_LCORE', 256],
> > > +	['RTE_USE_C11_MEM_MODEL', true],
> > > +	['RTE_CACHE_LINE_SIZE', 128]
> > > +]
> > > +
> > > +machine_args = ['-march=armv8-a+crc']
> > > diff --git a/config/arm/meson.build b/config/arm/meson.build index
> > > 8728051d5..05b790f40 100644
> > > --- a/config/arm/meson.build
> > > +++ b/config/arm/meson.build
> > > @@ -2,11 +2,11 @@
> > >  # Copyright(c) 2017 Intel Corporation.
> > >  # Copyright(c) 2017 Cavium, Inc
> > >
> > > -# for checking defines we need to use the correct compiler flags
> > > -march_opt = '-march=@0@'.format(machine)
> > > -
> > > -arm_force_native_march = false
> > > -arm_force_default_march = (machine == 'default')
> > > +if machine == 'native'
> > > +	# arm doesn't support a native build - user input of soc is required
> > > +	# default to the portable build instead of native
> > > +	machine = 'default'
> > Do we need a message here to the user?
> >
> 
> Better to have one, good catch.
> 
> > > +endif
> > >
> > >  flags_common_default = [
> > >  	# Accelarate rte_memcpy. Be sure to run unit test
> > > (memcpy_perf_autotest) @@ -131,6 +131,7 @@ impl_dpaa = ['NXP
> DPAA',
> > > flags_dpaa, machine_args_generic]
> > > dpdk_conf.set('RTE_FORCE_INTRINSICS',
> > > 1)
> > >
> > >  if not dpdk_conf.get('RTE_ARCH_64')
> > > +	# armv7 build
> > >  	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
> > >  	dpdk_conf.set('RTE_ARCH_ARM', 1)
> > >  	dpdk_conf.set('RTE_ARCH_ARMv7', 1) @@ -138,41 +139,9 @@ if not
> > > dpdk_conf.get('RTE_ARCH_64')
> > >  	# mk/machine/armv7a/rte.vars.mk sets it too
> > >  	machine_args += '-mfpu=neon'
> > >  else
> > > +	# aarch64 build
> > >  	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
> > >  	dpdk_conf.set('RTE_ARCH_ARM64', 1)
> > > -
> > > -	machine = []
> > > -	cmd_generic = ['generic', '', '', 'default', '']
> > > -	cmd_output = cmd_generic # Set generic by default
> > > -	machine_args = [] # Clear previous machine args
> > > -	if arm_force_default_march and not meson.is_cross_build()
> > > -		machine = impl_generic
> > > -		impl_pn = 'default'
> > > -	elif not meson.is_cross_build()
> > > -		# The script returns ['Implementer', 'Variant', 'Architecture',
> > > -		# 'Primary Part number', 'Revision']
> > > -		detect_vendor = find_program(join_paths(
> > > -				meson.current_source_dir(),
> > > 'armv8_machine.py'))
> > > -		cmd = run_command(detect_vendor.path())
> > > -		if cmd.returncode() == 0
> > > -			cmd_output = cmd.stdout().to_lower().strip().split(' ')
> > > -		endif
> > > -		# Set to generic if variable is not found
> > > -		machine = get_variable('impl_' + cmd_output[0], ['generic'])
> > > -		if machine[0] == 'generic'
> > > -			machine = impl_generic
> > > -			cmd_output = cmd_generic
> > > -		endif
> > > -		impl_pn = cmd_output[3]
> > > -		if arm_force_native_march == true
> > > -			impl_pn = 'native'
> > > -		endif
> > > -	else
> > > -		impl_id = meson.get_cross_property('implementor_id',
> > > 'generic')
> > > -		impl_pn = meson.get_cross_property('implementor_pn',
> > > 'default')
> > > -		machine = get_variable('impl_' + impl_id)
> > > -	endif
> > > -
> > >  	# Apply Common Defaults. These settings may be overwritten by
> > > machine
> > >  	# settings later.
> > >  	foreach flag: flags_common_default @@ -181,30 +150,89 @@ else
> > >  		endif
> > >  	endforeach
> > >
> > > -	message('Implementer : ' + machine[0])
> > > -	foreach flag: machine[1]
> > > +	soc = machine.split('-')[0]
> > > +	# TODO make sure the subdir exists
> > > +	subdir(soc)
> > > +
> > > +	if soc != 'default'
> > > +		# combine common and soc specific flags
> > > +		# TODO check that soc is in the dict
> > > +		flags = flags['common'] + flags[soc]
> > > +	endif
> > > +
> > > +	foreach flag: flags
> > >  		if flag.length() > 0
> > >  			dpdk_conf.set(flag[0], flag[1])
> > >  		endif
> > >  	endforeach
> > >
> > > -	foreach marg: machine[2]
> > > -		if marg[0] == impl_pn
> > > -			foreach flag: marg[1]
> > > -				if cc.has_argument(flag)
> > > -					machine_args += flag
> > > -				endif
> > > -			endforeach
> > > -			# Apply any extra machine specific flags.
> > > -			foreach flag: marg.get(2, flags_default_extra)
> > > -				if flag.length() > 0
> > > -					dpdk_conf.set(flag[0], flag[1])
> > > -				endif
> > > -			endforeach
> > > +	verified_machine_args = []
> > > +
> > > +	foreach flag: machine_args
> > > +		if cc.has_argument(flag)
> > > +			verified_machine_args += flag
> > > +		else
> > > +			message('Unsupported machine flag "@0@",
> > > ignoring'.format(flag))
> > >  		endif
> > >  	endforeach
> > > +	machine_args = verified_machine_args
> > > +
> > > +#	machine = []
> > > +#	cmd_generic = ['generic', '', '', 'default', '']
> > > +#	cmd_output = cmd_generic # Set generic by default
> > > +#	machine_args = [] # Clear previous machine args
> > > +#	if arm_force_default_march and not meson.is_cross_build()
> > > +#		machine = impl_generic
> > > +#		impl_pn = 'default'
> > > +#	elif not meson.is_cross_build()
> > Isn't this for native build? Do we still need this?
> >
> 
> Yes, I should've removed this instead of commenting it out.
> 
> > > +#		# The script returns ['Implementer', 'Variant', 'Architecture',
> > > +#		# 'Primary Part number', 'Revision']
> > > +#		detect_vendor = find_program(join_paths(
> > > +#				meson.current_source_dir(),
> > > 'armv8_machine.py'))
> > > +#		cmd = run_command(detect_vendor.path())
> > > +#		if cmd.returncode() == 0
> > > +#			cmd_output = cmd.stdout().to_lower().strip().split(' ')
> > > +#		endif
> > > +#		# Set to generic if variable is not found
> > > +#		machine = get_variable('impl_' + cmd_output[0], ['generic'])
> > > +#		if machine[0] == 'generic'
> > > +#			machine = impl_generic
> > > +#			cmd_output = cmd_generic
> > > +#		endif
> > > +#		impl_pn = cmd_output[3]
> > > +#		if arm_force_native_march == true
> > > +#			impl_pn = 'native'
> > > +#		endif
> > > +#	else
> > > +#		impl_id = meson.get_cross_property('implementor_id',
> > > 'generic')
> > > +#		impl_pn = meson.get_cross_property('implementor_pn',
> > > 'default')
> > > +#		machine = get_variable('impl_' + impl_id)
> > > +#	endif
> > > +#
> > > +#	message('Implementer : ' + machine[0])
> > > +#	foreach flag: machine[1]
> > > +#		if flag.length() > 0
> > > +#			dpdk_conf.set(flag[0], flag[1])
> > > +#		endif
> > > +#	endforeach
> > > +#
> > > +#	foreach marg: machine[2]
> > > +#		if marg[0] == impl_pn
> > > +#			foreach flag: marg[1]
> > > +#				if cc.has_argument(flag)
> > > +#					machine_args += flag
> > > +#				endif
> > > +#			endforeach
> > > +#			# Apply any extra machine specific flags.
> > > +#			foreach flag: marg.get(2, flags_default_extra)
> > > +#				if flag.length() > 0
> > > +#					dpdk_conf.set(flag[0], flag[1])
> > > +#				endif
> > > +#			endforeach
> > > +#		endif
> > > +#	endforeach
> > >  endif
> > > -message(machine_args)
> > > +message('Using machine args: @0@'.format(machine_args))
> > >
> > >  if (cc.get_define('__ARM_NEON', args: machine_args) != '' or
> > >      cc.get_define('__aarch64__', args: machine_args) != '') @@
> > > -225,3
> > > +253,6 @@ if cc.get_define('__ARM_FEATURE_CRYPTO', args:
> > > +machine_args)
> > != ''
> > >  	compile_time_cpuflags += ['RTE_CPUFLAG_AES',
> > 'RTE_CPUFLAG_PMULL',
> > >  	'RTE_CPUFLAG_SHA1', 'RTE_CPUFLAG_SHA2']  endif
> > > +
> > > +message('dpdk conf options after arm stuff: @0@'.format(dpdk_conf))
> > > +
> > > diff --git a/config/arm/arm64_thunderx_linux_gcc
> > > b/config/arm/thunderx/arm64_thunderx_linux_gcc
> > > similarity index 92%
> > > rename from config/arm/arm64_thunderx_linux_gcc
> > > rename to config/arm/thunderx/arm64_thunderx_linux_gcc
> > > index 6572ab615..e53a86e8f 100644
> > > --- a/config/arm/arm64_thunderx_linux_gcc
> > > +++ b/config/arm/thunderx/arm64_thunderx_linux_gcc
> > > @@ -13,4 +13,4 @@ cpu = 'armv8-a'
> > >  endian = 'little'
> > >
> > >  [properties]
> > > -implementor_id = '0x43'
> > > +machine = 'thunderx'
> > > diff --git a/config/arm/thunderx/meson.build
> > > b/config/arm/thunderx/meson.build new file mode 100644 index
> > > 000000000..873d79a64
> > > --- /dev/null
> > > +++ b/config/arm/thunderx/meson.build
> > > @@ -0,0 +1,18 @@
> > > +flags = {
> > > +	'common': [
> > > +		['RTE_CACHE_LINE_SIZE', 128],
> > > +		['RTE_MAX_VFIO_GROUPS', 128],
> > > +		['RTE_MACHINE', '"thunderx"'],
> > > +		['RTE_USE_C11_MEM_MODEL', false]
> > > +	],
> > > +	'thunderx': [
> > > +		['RTE_MAX_NUMA_NODES', 2],
> > > +		['RTE_MAX_LCORE', 96]
> > > +	],
> > > +	'thunderx-1n-48c': [
> > > +		['RTE_MAX_NUMA_NODES', 1],
> > > +		['RTE_MAX_LCORE', 48]
> > > +	]
> > > +}
> > > +
> > > +machine_args = ['-march=armv8-a+crc+crypto','-mcpu=thunderx']
> > > diff --git a/config/arm/thunderxt88/arm64_thunderxt88_linux_gcc
> > > b/config/arm/thunderxt88/arm64_thunderxt88_linux_gcc
> > > new file mode 100644
> > > index 000000000..9a622786c
> > > --- /dev/null
> > > +++ b/config/arm/thunderxt88/arm64_thunderxt88_linux_gcc
> > > @@ -0,0 +1,16 @@
> > > +[binaries]
> > > +c = 'aarch64-linux-gnu-gcc'
> > > +cpp = 'aarch64-linux-gnu-cpp'
> > > +ar = 'aarch64-linux-gnu-gcc-ar'
> > > +strip = 'aarch64-linux-gnu-strip'
> > > +pkgconfig = 'aarch64-linux-gnu-pkg-config'
> > > +pcap-config = ''
> > > +
> > > +[host_machine]
> > > +system = 'linux'
> > > +cpu_family = 'aarch64'
> > > +cpu = 'armv8-a'
> > > +endian = 'little'
> > > +
> > > +[properties]
> > > +machine = 'thunderxt88'
> > > diff --git a/config/arm/thunderxt88/meson.build
> > > b/config/arm/thunderxt88/meson.build
> > > new file mode 100644
> > > index 000000000..2aa5d2cdd
> > > --- /dev/null
> > > +++ b/config/arm/thunderxt88/meson.build
> > > @@ -0,0 +1,18 @@
> > > +flags = {
> > > +	'common': [
> > > +		['RTE_CACHE_LINE_SIZE', 128],
> > > +		['RTE_MAX_VFIO_GROUPS', 128],
> > > +		['RTE_MACHINE', '"thunderx"'],
> > > +		['RTE_USE_C11_MEM_MODEL', false]
> > > +	],
> > > +	'thunderxt88': [
> > > +		['RTE_MAX_NUMA_NODES', 2],
> > > +		['RTE_MAX_LCORE', 96]
> > > +	],
> > > +	'thunderxt88-1n-48c': [
> > > +		['RTE_MAX_NUMA_NODES', 1],
> > > +		['RTE_MAX_LCORE', 48]
> > > +	]
> > > +}
> > > +
> > > +machine_args = ['-mcpu=thunderxt88']
> > > diff --git a/config/meson.build b/config/meson.build index
> > > 69f2aeb60..177cbd49c 100644
> > > --- a/config/meson.build
> > > +++ b/config/meson.build
> > > @@ -63,7 +63,14 @@ meson.add_install_script('../buildtools/symlink-
> > > drivers-solibs.sh',
> > >
> > >  # set the machine type and cflags for it  if meson.is_cross_build()
> > > -	machine = host_machine.cpu()
> > > +	if not host_machine.cpu_family().startswith('aarch')
> > > +		# don't change the machine config for aarch32/64 builds
> > > +		# that config is set in the cross file to identify the soc
> > > +		# we're building for
> > > +		machine = host_machine.cpu()
> > > +	else
> > > +		machine = meson.get_cross_property('machine', 'default')
> > > +	endif
> > >  else
> > >  	machine = get_option('machine')
> > >  endif
> > > --
> > > 2.20.1

Re: [dpdk-dev] [RFC PATCH 1/2] build: rework Arm build options

[Ali Saidi]

<snip>

> > > >
> > > > The current system can identify only the implementer and part number
> > > > of the arm device we're targeting, which is enough to identify the target
> > CPU.
> > > > However, even the full MIDR information is not enough to identify
> > > > the SoC we're targeting.
> > > >
> > > > Expand the "machine" meson variable to allow specifying the target arm
> > SoC.
> > > > The SoC identification implies the CPU, so drop the current MIDR
> > > > based CPU identification in favor of user input, which is the only
> > > > way to identify the target SoC. This implies that native builds are
> > > > not possible. Instead do a default (non-optimized, but executable on
> > > > any
> > > > aarch64 device) build.
I don't think this is really what we want. The MIDR identifies the architecture
features that are supported and we should continue to use that for native
builds.
 
The difference between SoCs really comes down to the RAM and core count, which
are also detectable or could be specified seperately and defaulted to the
detected values. 
 
<snip>
 
> > 
> > Maybe we can have multiple labels mean the same thing - default, generic,
> > armv8 or even generic-armv8 (that's the most descriptive name IMO) could
> > all mean the generic arm build executable on all aarch64 SoCs. That many
> > names is probably confusing, but sticking to two (default and one more)
> > seems feasible.
> > 
> > What does everyone else think?
> 'armv8' makes lot more sense to me and it conveys what the build does. We can keep 'default' to be backward compatible, we can skip others.

native build below should just be that. Detect the MIDR, use that for the feature
flags and use the existing number of CPUs (or a user-specified number). I like 
generic-armv8 personally beacuse specifies a generic build that won't be 
optimized. 

Re: [dpdk-dev] [RFC PATCH 1/2] build: rework Arm build options

[Honnappa Nagarahalli]

<snip>
> 
> > > > >
> > > > > The current system can identify only the implementer and part
> > > > > number of the arm device we're targeting, which is enough to
> > > > > identify the target
> > > CPU.
> > > > > However, even the full MIDR information is not enough to
> > > > > identify the SoC we're targeting.
> > > > >
> > > > > Expand the "machine" meson variable to allow specifying the
> > > > > target arm
> > > SoC.
> > > > > The SoC identification implies the CPU, so drop the current MIDR
> > > > > based CPU identification in favor of user input, which is the
> > > > > only way to identify the target SoC. This implies that native
> > > > > builds are not possible. Instead do a default (non-optimized,
> > > > > but executable on any
> > > > > aarch64 device) build.
> I don't think this is really what we want. The MIDR identifies the architecture
> features that are supported and we should continue to use that for native
> builds.
> 
> The difference between SoCs really comes down to the RAM and core count,
> which are also detectable or could be specified seperately and defaulted to
> the detected values.
Hi Ali, thank you for the feedback. The differences between SoCs can be more than RAM/core count (ex: NUMA details) and addresses the need to compile only some modules of DPDK.
But, having said that we will keep the native build also. It will continue to function the way it does today. 

> 
> <snip>
> 
> > >
> > > Maybe we can have multiple labels mean the same thing - default,
> > > generic,
> > > armv8 or even generic-armv8 (that's the most descriptive name IMO)
> > > could all mean the generic arm build executable on all aarch64 SoCs.
> > > That many names is probably confusing, but sticking to two (default
> > > and one more) seems feasible.
> > >
> > > What does everyone else think?
> > 'armv8' makes lot more sense to me and it conveys what the build does.
> We can keep 'default' to be backward compatible, we can skip others.
> 
> native build below should just be that. Detect the MIDR, use that for the
> feature flags and use the existing number of CPUs (or a user-specified
> number). I like
> generic-armv8 personally beacuse specifies a generic build that won't be
> optimized.

[dpdk-dev] [RFC PATCH 2/2] build: disable drivers from file

[Juraj Linkeš]

Add the capability to specify which drivers will NOT be build in a file.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 .../arm/{ => armada}/arm64_armada_linux_gcc   |  2 +-
 config/arm/armada/meson.build                 | 31 +++++++++++++++++++
 drivers/meson.build                           | 13 +++++---
 3 files changed, 41 insertions(+), 5 deletions(-)
 rename config/arm/{ => armada}/arm64_armada_linux_gcc (93%)
 create mode 100644 config/arm/armada/meson.build

diff --git a/config/arm/arm64_armada_linux_gcc b/config/arm/armada/arm64_armada_linux_gcc
similarity index 93%
rename from config/arm/arm64_armada_linux_gcc
rename to config/arm/armada/arm64_armada_linux_gcc
index fa40c0398..8b4b5c8ad 100644
--- a/config/arm/arm64_armada_linux_gcc
+++ b/config/arm/armada/arm64_armada_linux_gcc
@@ -14,4 +14,4 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x56'
+machine = 'armada'
diff --git a/config/arm/armada/meson.build b/config/arm/armada/meson.build
new file mode 100644
index 000000000..22fdce5fe
--- /dev/null
+++ b/config/arm/armada/meson.build
@@ -0,0 +1,31 @@
+flags = {
+	'common': [
+		['RTE_CACHE_LINE_SIZE', 64],
+		['RTE_MACHINE', '"armv8a"'],
+		['CONFIG_RTE_LIBRTE_DPAA_BUS=n', false],
+		['CONFIG_RTE_LIBRTE_COMMON_DPAAX=n', false],
+		['CONFIG_RTE_LIBRTE_FSLMC_BUS=n', false],
+		['CONFIG_RTE_LIBRTE_DPAA2_MEMPOOL=n', false],
+		['CONFIG_RTE_LIBRTE_DPAA2_PMD=n', false],
+		['CONFIG_RTE_LIBRTE_DPAA_BUS=n', false],
+		['CONFIG_RTE_LIBRTE_DPAA_MEMPOOL=n', false],
+		['CONFIG_RTE_LIBRTE_DPAA_PMD=n', false],
+		['CONFIG_RTE_LIBRTE_PMD_DPAA_EVENTDEV=n', false],
+		['CONFIG_RTE_LIBRTE_PMD_DPAA_SEC=n', false],
+		['CONFIG_RTE_LIBRTE_PMD_CAAM_JR=n', false],
+		['CONFIG_RTE_LIBRTE_PMD_DPAA2_EVENTDEV=n', false],
+		['CONFIG_RTE_LIBRTE_PMD_DPAA2_SEC=n', false],
+		['CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV=n', false],
+		['CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n', false],
+		['CONFIG_RTE_LIBRTE_PFE_PMD=n', false],
+		['CONFIG_RTE_LIBRTE_ENETC_PMD=n', false],
+		['CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n', false],
+		['CONFIG_RTE_LIBRTE_VHOST_NUMA=n', false]
+	],
+	'armada': [
+		['RTE_MAX_NUMA_NODES', 1],
+		['RTE_MAX_LCORE', 16]
+	]
+}
+
+machine_args = ['-march=armv8-a+crc']
diff --git a/drivers/meson.build b/drivers/meson.build
index 5f9526557..c5deb6136 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -76,9 +76,17 @@ foreach subpath:subdirs
 		ext_deps = []
 		pkgconfig_extra_libs = []
 
+		if fmt_name == ''
+			fmt_name = name
+		endif
+		conf_flag = config_flag_fmt.format(fmt_name.to_upper())
+
 		if disabled_drivers.contains(drv_path)
 			build = false
 			reason = 'Explicitly disabled via build config'
+		elif dpdk_conf.has(conf_flag) and not dpdk_conf.get(conf_flag)
+			build = false
+			reason = 'Disabled via configuration file'
 		else
 			# pull in driver directory which should update all the local variables
 			subdir(drv_path)
@@ -113,10 +121,7 @@ foreach subpath:subdirs
 		else
 			enabled_drivers += name
 
-			if fmt_name == ''
-				fmt_name = name
-			endif
-			dpdk_conf.set(config_flag_fmt.format(fmt_name.to_upper()),1)
+			dpdk_conf.set(conf_flag, 1)
 			lib_name = driver_name_fmt.format(fmt_name)
 
 			dpdk_extra_ldflags += pkgconfig_extra_libs
-- 
2.20.1

Re: [dpdk-dev] [RFC PATCH 2/2] build: disable drivers from file

[Bruce Richardson]

On Wed, Sep 23, 2020 at 03:19:21PM +0200, Juraj Linkeš wrote:
> Add the capability to specify which drivers will NOT be build in a file.
> 
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
>  .../arm/{ => armada}/arm64_armada_linux_gcc   |  2 +-
>  config/arm/armada/meson.build                 | 31 +++++++++++++++++++
>  drivers/meson.build                           | 13 +++++---
>  3 files changed, 41 insertions(+), 5 deletions(-)
>  rename config/arm/{ => armada}/arm64_armada_linux_gcc (93%)
>  create mode 100644 config/arm/armada/meson.build
> 
> diff --git a/config/arm/arm64_armada_linux_gcc b/config/arm/armada/arm64_armada_linux_gcc
> similarity index 93%
> rename from config/arm/arm64_armada_linux_gcc
> rename to config/arm/armada/arm64_armada_linux_gcc
> index fa40c0398..8b4b5c8ad 100644
> --- a/config/arm/arm64_armada_linux_gcc
> +++ b/config/arm/armada/arm64_armada_linux_gcc
> @@ -14,4 +14,4 @@ cpu = 'armv8-a'
>  endian = 'little'
>  
>  [properties]
> -implementor_id = '0x56'
> +machine = 'armada'
> diff --git a/config/arm/armada/meson.build b/config/arm/armada/meson.build
> new file mode 100644
> index 000000000..22fdce5fe
> --- /dev/null
> +++ b/config/arm/armada/meson.build
> @@ -0,0 +1,31 @@
> +flags = {
> +	'common': [
> +		['RTE_CACHE_LINE_SIZE', 64],
> +		['RTE_MACHINE', '"armv8a"'],
> +		['CONFIG_RTE_LIBRTE_DPAA_BUS=n', false],
> +		['CONFIG_RTE_LIBRTE_COMMON_DPAAX=n', false],
> +		['CONFIG_RTE_LIBRTE_FSLMC_BUS=n', false],
> +		['CONFIG_RTE_LIBRTE_DPAA2_MEMPOOL=n', false],
> +		['CONFIG_RTE_LIBRTE_DPAA2_PMD=n', false],
> +		['CONFIG_RTE_LIBRTE_DPAA_BUS=n', false],
> +		['CONFIG_RTE_LIBRTE_DPAA_MEMPOOL=n', false],
> +		['CONFIG_RTE_LIBRTE_DPAA_PMD=n', false],
> +		['CONFIG_RTE_LIBRTE_PMD_DPAA_EVENTDEV=n', false],
> +		['CONFIG_RTE_LIBRTE_PMD_DPAA_SEC=n', false],
> +		['CONFIG_RTE_LIBRTE_PMD_CAAM_JR=n', false],
> +		['CONFIG_RTE_LIBRTE_PMD_DPAA2_EVENTDEV=n', false],
> +		['CONFIG_RTE_LIBRTE_PMD_DPAA2_SEC=n', false],
> +		['CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV=n', false],
> +		['CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n', false],
> +		['CONFIG_RTE_LIBRTE_PFE_PMD=n', false],
> +		['CONFIG_RTE_LIBRTE_ENETC_PMD=n', false],
> +		['CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n', false],
> +		['CONFIG_RTE_LIBRTE_VHOST_NUMA=n', false]
> +	],

Rather than setting a list of flags like this make-style, why not have a
disabled-drivers setting, and add that to any user-provided ones? It would
save making any changes to the drivers/meson.build file, and therefore be
a lot cleaner. It would also allow using wildcards etc. to disable whole
classes of drivers

Re: [dpdk-dev] [RFC PATCH 2/2] build: disable drivers from file

[Juraj Linkeš]

> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Wednesday, September 23, 2020 3:59 PM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Cc: Ruifeng.Wang@arm.com; Honnappa.Nagarahalli@arm.com;
> Phil.Yang@arm.com; vcchunga@amazon.com; Dharmik.Thakkar@arm.com;
> jerinjacobk@gmail.com; hemant.agrawal@nxp.com; dev@dpdk.org
> Subject: Re: [RFC PATCH 2/2] build: disable drivers from file
> 
> On Wed, Sep 23, 2020 at 03:19:21PM +0200, Juraj Linkeš wrote:
> > Add the capability to specify which drivers will NOT be build in a file.
> >
> > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > ---
> >  .../arm/{ => armada}/arm64_armada_linux_gcc   |  2 +-
> >  config/arm/armada/meson.build                 | 31 +++++++++++++++++++
> >  drivers/meson.build                           | 13 +++++---
> >  3 files changed, 41 insertions(+), 5 deletions(-)  rename
> > config/arm/{ => armada}/arm64_armada_linux_gcc (93%)  create mode
> > 100644 config/arm/armada/meson.build
> >
> > diff --git a/config/arm/arm64_armada_linux_gcc
> > b/config/arm/armada/arm64_armada_linux_gcc
> > similarity index 93%
> > rename from config/arm/arm64_armada_linux_gcc rename to
> > config/arm/armada/arm64_armada_linux_gcc
> > index fa40c0398..8b4b5c8ad 100644
> > --- a/config/arm/arm64_armada_linux_gcc
> > +++ b/config/arm/armada/arm64_armada_linux_gcc
> > @@ -14,4 +14,4 @@ cpu = 'armv8-a'
> >  endian = 'little'
> >
> >  [properties]
> > -implementor_id = '0x56'
> > +machine = 'armada'
> > diff --git a/config/arm/armada/meson.build
> > b/config/arm/armada/meson.build new file mode 100644 index
> > 000000000..22fdce5fe
> > --- /dev/null
> > +++ b/config/arm/armada/meson.build
> > @@ -0,0 +1,31 @@
> > +flags = {
> > +	'common': [
> > +		['RTE_CACHE_LINE_SIZE', 64],
> > +		['RTE_MACHINE', '"armv8a"'],
> > +		['CONFIG_RTE_LIBRTE_DPAA_BUS=n', false],
> > +		['CONFIG_RTE_LIBRTE_COMMON_DPAAX=n', false],
> > +		['CONFIG_RTE_LIBRTE_FSLMC_BUS=n', false],
> > +		['CONFIG_RTE_LIBRTE_DPAA2_MEMPOOL=n', false],
> > +		['CONFIG_RTE_LIBRTE_DPAA2_PMD=n', false],
> > +		['CONFIG_RTE_LIBRTE_DPAA_BUS=n', false],
> > +		['CONFIG_RTE_LIBRTE_DPAA_MEMPOOL=n', false],
> > +		['CONFIG_RTE_LIBRTE_DPAA_PMD=n', false],
> > +		['CONFIG_RTE_LIBRTE_PMD_DPAA_EVENTDEV=n', false],
> > +		['CONFIG_RTE_LIBRTE_PMD_DPAA_SEC=n', false],
> > +		['CONFIG_RTE_LIBRTE_PMD_CAAM_JR=n', false],
> > +		['CONFIG_RTE_LIBRTE_PMD_DPAA2_EVENTDEV=n', false],
> > +		['CONFIG_RTE_LIBRTE_PMD_DPAA2_SEC=n', false],
> > +		['CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV=n',
> false],
> > +		['CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n',
> false],
> > +		['CONFIG_RTE_LIBRTE_PFE_PMD=n', false],
> > +		['CONFIG_RTE_LIBRTE_ENETC_PMD=n', false],
> > +		['CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n', false],
> > +		['CONFIG_RTE_LIBRTE_VHOST_NUMA=n', false]
> > +	],
> 
> Rather than setting a list of flags like this make-style, why not have a disabled-
> drivers setting, and add that to any user-provided ones? It would save making
> any changes to the drivers/meson.build file, and therefore be a lot cleaner. It
> would also allow using wildcards etc. to disable whole classes of drivers
> 

I didn't think of this, thanks for the suggestion, that's much better.

Re: [dpdk-dev] [RFC PATCH 0/2] Arm build options rework

[Honnappa Nagarahalli]

Adding Ali

> -----Original Message-----
> From: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Sent: Wednesday, September 23, 2020 8:19 AM
> To: bruce.richardson@intel.com; Ruifeng Wang <Ruifeng.Wang@arm.com>;
> Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Phil Yang
> <Phil.Yang@arm.com>; vcchunga@amazon.com; Dharmik Thakkar
> <Dharmik.Thakkar@arm.com>; jerinjacobk@gmail.com;
> hemant.agrawal@nxp.com
> Cc: dev@dpdk.org; Juraj Linkeš <juraj.linkes@pantheon.tech>
> Subject: [RFC PATCH 0/2] Arm build options rework
> 
> The current way of specifying Arm configuration options is insufficient since
> we can't identify the SoC we're building for from the MIDR information.
> 
> Instead use user input - the machine meson option to specify the SoC, e.g.:
> meson soc_test -Dmachine=thunderx
> meson soc_test --cross-file
> config/arm/thunderx/arm64_thunderx_linux_gcc
> 
> We also want to be able to disable which drivers (and possibly
> libraries) are built without user input. This is useful when building:
> 1. on an SoC that is slow and we want to build only what is necessary without
> the user having to check which libraries they have installed 2. a cross build on
> a fast aarch64 machine but with target SoC which differs in capabilities or
> libraries.
> 
> Juraj Linkeš (2):
>   build: rework Arm build options
>   build: disable drivers from file
> 
>  config/arm/arm64_armv8_linux_gcc              |  28 ----
>  .../arm/{ => armada}/arm64_armada_linux_gcc   |   2 +-
>  config/arm/armada/meson.build                 |  31 ++++
>  config/arm/default/arm64_armv8_linux_gcc      |  23 +++
>  config/arm/default/meson.build                |   8 +
>  config/arm/meson.build                        | 139 +++++++++++-------
>  .../{ => thunderx}/arm64_thunderx_linux_gcc   |   2 +-
>  config/arm/thunderx/meson.build               |  18 +++
>  .../thunderxt88/arm64_thunderxt88_linux_gcc   |  16 ++
>  config/arm/thunderxt88/meson.build            |  18 +++
>  config/meson.build                            |   9 +-
>  drivers/meson.build                           |  13 +-
>  12 files changed, 218 insertions(+), 89 deletions(-)  delete mode 100644
> config/arm/arm64_armv8_linux_gcc  rename config/arm/{ =>
> armada}/arm64_armada_linux_gcc (93%)  create mode 100644
> config/arm/armada/meson.build  create mode 100644
> config/arm/default/arm64_armv8_linux_gcc
>  create mode 100644 config/arm/default/meson.build  rename
> config/arm/{ => thunderx}/arm64_thunderx_linux_gcc (92%)  create mode
> 100644 config/arm/thunderx/meson.build  create mode 100644
> config/arm/thunderxt88/arm64_thunderxt88_linux_gcc
>  create mode 100644 config/arm/thunderxt88/meson.build
> 
> --
> 2.20.1

[dpdk-dev] [RFC PATCH v2 0/6] Arm build options rework

[Juraj Linkeš]

The current way of specifying Arm configuration options is insufficient
since we can't identify the SoC we're building for from the MIDR
information.

Add a way to discover cpu count a numa node count and use that instead
of statically defined values. This augments the current native builds.
A bonus is that users may now use -Dmax_lcores and -Dmax_numa_nodes to
configure those from command line, which didn't work before for Arm
builds.

For cross builds, the cpu count and numa node count are specified in
cross files.

We also want to be able to disable which drivers (and possibly
libraries) are built without user input. This is useful when building:
1. on an SoC that is slow and we want to build only what is necessary
  without the user having to check which libraries they have installed
2. a cross build on a fast aarch64 machine but with target SoC which
  differs in capabilities or libraries.
This is achieved by specifying the drivers in cross files. Only libnuma
can be now disabled.

v2:
Major rework of the whole series.

Juraj Linkeš (6):
  build: rename default Arm build to generic-armv8
  build: refactor Arm build
  build: automatic NUMA and cpu counts detection
  build: move core and NUMA counts to cross files
  build: disable Arm drivers
  build: update Arm builds with makefile flags

 buildtools/get_cpu_count.py          |   7 +
 buildtools/get_numa_count.py         |  17 ++
 buildtools/meson.build               |   2 +
 config/arm/arm64_armada_linux_gcc    |   6 +-
 config/arm/arm64_armv8_linux_gcc     |  17 +-
 config/arm/arm64_bluefield_linux_gcc |   6 +-
 config/arm/arm64_dpaa_linux_gcc      |   5 +-
 config/arm/arm64_emag_linux_gcc      |   4 +-
 config/arm/arm64_n1sdp_linux_gcc     |   7 +-
 config/arm/arm64_octeontx2_linux_gcc |   7 +-
 config/arm/arm64_stingray_linux_gcc  |   6 +-
 config/arm/arm64_thunderx2_linux_gcc |   6 +-
 config/arm/arm64_thunderx_linux_gcc  |   4 +-
 config/arm/meson.build               | 293 +++++++++++++++------------
 config/meson.build                   |  23 ++-
 drivers/meson.build                  |   6 +-
 meson.build                          |   1 +
 meson_options.txt                    |   8 +-
 18 files changed, 267 insertions(+), 158 deletions(-)
 create mode 100644 buildtools/get_cpu_count.py
 create mode 100644 buildtools/get_numa_count.py

-- 
2.20.1

[dpdk-dev] [RFC PATCH v2 1/6] build: rename default Arm build to generic-armv8

[Juraj Linkeš]

The current machine='default' build name is not descriptive. The actual
default build is machine='native'. Add an alternative string which does
the same build and better describes what we're building:
machine='generic-armv8'. Leave machine='default' for backwards
compatibility.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/meson.build | 31 ++++++++++++++++---------------
 config/meson.build     |  3 ---
 2 files changed, 16 insertions(+), 18 deletions(-)

diff --git a/config/arm/meson.build b/config/arm/meson.build
index 42c0c34a5..c371f983e 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -1,12 +1,13 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation.
 # Copyright(c) 2017 Cavium, Inc
+# Copyright(c) 2020 PANTHEON.tech s.r.o.
 
 # for checking defines we need to use the correct compiler flags
 march_opt = '-march=@0@'.format(machine)
 
 arm_force_native_march = false
-arm_force_default_march = (machine == 'default')
+arm_force_generic_march = (machine == 'default' or machine == 'generic-armv8')
 
 flags_common_default = [
 	# Accelarate rte_memcpy. Be sure to run unit test (memcpy_perf_autotest)
@@ -87,7 +88,7 @@ flags_octeontx2_extra = [
 	['RTE_EAL_IGB_UIO', false],
 	['RTE_USE_C11_MEM_MODEL', true]]
 
-machine_args_generic = [
+machine_args_default = [
 	['default', ['-march=armv8-a+crc']],
 	['native', ['-march=native']],
 	['0xd03', ['-mcpu=cortex-a53']],
@@ -113,20 +114,20 @@ machine_args_emag = [
 	['native', ['-march=native']]]
 
 ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page G7-5321)
-impl_generic = ['Generic armv8', flags_generic, machine_args_generic]
-impl_0x41 = ['Arm', flags_arm, machine_args_generic]
-impl_0x42 = ['Broadcom', flags_generic, machine_args_generic]
+impl_generic = ['Generic armv8', flags_generic, machine_args_default]
+impl_0x41 = ['Arm', flags_arm, machine_args_default]
+impl_0x42 = ['Broadcom', flags_generic, machine_args_default]
 impl_0x43 = ['Cavium', flags_cavium, machine_args_cavium]
-impl_0x44 = ['DEC', flags_generic, machine_args_generic]
-impl_0x49 = ['Infineon', flags_generic, machine_args_generic]
-impl_0x4d = ['Motorola', flags_generic, machine_args_generic]
-impl_0x4e = ['NVIDIA', flags_generic, machine_args_generic]
+impl_0x44 = ['DEC', flags_generic, machine_args_default]
+impl_0x49 = ['Infineon', flags_generic, machine_args_default]
+impl_0x4d = ['Motorola', flags_generic, machine_args_default]
+impl_0x4e = ['NVIDIA', flags_generic, machine_args_default]
 impl_0x50 = ['Ampere Computing', flags_emag, machine_args_emag]
-impl_0x51 = ['Qualcomm', flags_generic, machine_args_generic]
-impl_0x53 = ['Samsung', flags_generic, machine_args_generic]
-impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_generic]
-impl_0x69 = ['Intel', flags_generic, machine_args_generic]
-impl_dpaa = ['NXP DPAA', flags_dpaa, machine_args_generic]
+impl_0x51 = ['Qualcomm', flags_generic, machine_args_default]
+impl_0x53 = ['Samsung', flags_generic, machine_args_default]
+impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_default]
+impl_0x69 = ['Intel', flags_generic, machine_args_default]
+impl_dpaa = ['NXP DPAA', flags_dpaa, machine_args_default]
 
 dpdk_conf.set('RTE_FORCE_INTRINSICS', 1)
 
@@ -145,7 +146,7 @@ else
 	cmd_generic = ['generic', '', '', 'default', '']
 	cmd_output = cmd_generic # Set generic by default
 	machine_args = [] # Clear previous machine args
-	if arm_force_default_march and not meson.is_cross_build()
+	if arm_force_generic_march and not meson.is_cross_build()
 		machine = impl_generic
 		impl_pn = 'default'
 	elif not meson.is_cross_build()
diff --git a/config/meson.build b/config/meson.build
index 9fb903c9b..4bd65d98e 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -80,9 +80,6 @@ if machine == 'default'
 		machine = 'corei7'
 	elif host_machine.cpu_family().startswith('arm')
 		machine = 'armv7-a'
-	elif host_machine.cpu_family().startswith('aarch')
-		# arm64 manages defaults in config/arm/meson.build
-		machine = 'default'
 	elif host_machine.cpu_family().startswith('ppc')
 		machine = 'power8'
 	endif
-- 
2.20.1

[dpdk-dev] [RFC PATCH v2 2/6] build: refactor Arm build

[Juraj Linkeš]

* Rename variables to have names that better describe what the variables
store
* Remove unused or superfluous variables
* Change a list to dictionary where key lookup is needed
* Add informatory comments in the code
* Minor code restructure and reformatting

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/arm64_armada_linux_gcc    |   2 +-
 config/arm/arm64_armv8_linux_gcc     |   8 +-
 config/arm/arm64_bluefield_linux_gcc |   4 +-
 config/arm/arm64_dpaa_linux_gcc      |   2 +-
 config/arm/arm64_emag_linux_gcc      |   2 +-
 config/arm/arm64_n1sdp_linux_gcc     |   4 +-
 config/arm/arm64_octeontx2_linux_gcc |   4 +-
 config/arm/arm64_stingray_linux_gcc  |   4 +-
 config/arm/arm64_thunderx2_linux_gcc |   4 +-
 config/arm/arm64_thunderx_linux_gcc  |   2 +-
 config/arm/meson.build               | 246 +++++++++++++++------------
 11 files changed, 153 insertions(+), 129 deletions(-)

diff --git a/config/arm/arm64_armada_linux_gcc b/config/arm/arm64_armada_linux_gcc
index fa40c0398..52c5f4476 100644
--- a/config/arm/arm64_armada_linux_gcc
+++ b/config/arm/arm64_armada_linux_gcc
@@ -14,4 +14,4 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x56'
+implementer_id = '0x56'
diff --git a/config/arm/arm64_armv8_linux_gcc b/config/arm/arm64_armv8_linux_gcc
index 88f0ff9da..13ee8b223 100644
--- a/config/arm/arm64_armv8_linux_gcc
+++ b/config/arm/arm64_armv8_linux_gcc
@@ -13,10 +13,10 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = 'generic'
+implementer_id = 'generic'
 
-# Valid options for Arm's implementor_pn:
-# 'default': valid for all armv8-a architectures (default value)
+# Valid options for Arm's part_number:
+# 'generic': valid for all armv8-a architectures (default value)
 # '0xd03':   cortex-a53
 # '0xd04':   cortex-a35
 # '0xd05':   cortex-a55
@@ -25,4 +25,4 @@ implementor_id = 'generic'
 # '0xd09':   cortex-a73
 # '0xd0a':   cortex-a75
 # '0xd0b':   cortex-a76
-implementor_pn = 'default'
+part_number = 'generic'
diff --git a/config/arm/arm64_bluefield_linux_gcc b/config/arm/arm64_bluefield_linux_gcc
index 86797d23c..b79389d85 100644
--- a/config/arm/arm64_bluefield_linux_gcc
+++ b/config/arm/arm64_bluefield_linux_gcc
@@ -13,5 +13,5 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x41'
-implementor_pn = '0xd08'
+implementer_id = '0x41'
+part_number = '0xd08'
diff --git a/config/arm/arm64_dpaa_linux_gcc b/config/arm/arm64_dpaa_linux_gcc
index 1a4682154..573ae7e42 100644
--- a/config/arm/arm64_dpaa_linux_gcc
+++ b/config/arm/arm64_dpaa_linux_gcc
@@ -14,4 +14,4 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = 'dpaa'
+implementer_id = 'dpaa'
diff --git a/config/arm/arm64_emag_linux_gcc b/config/arm/arm64_emag_linux_gcc
index 8edcd3e97..24f3d533e 100644
--- a/config/arm/arm64_emag_linux_gcc
+++ b/config/arm/arm64_emag_linux_gcc
@@ -13,4 +13,4 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x50'
+implementer_id = '0x50'
diff --git a/config/arm/arm64_n1sdp_linux_gcc b/config/arm/arm64_n1sdp_linux_gcc
index 022e06303..6fb3f02ea 100644
--- a/config/arm/arm64_n1sdp_linux_gcc
+++ b/config/arm/arm64_n1sdp_linux_gcc
@@ -13,5 +13,5 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x41'
-implementor_pn = '0xd0c'
+implementer_id = '0x41'
+part_number = '0xd0c'
diff --git a/config/arm/arm64_octeontx2_linux_gcc b/config/arm/arm64_octeontx2_linux_gcc
index 365bd7cbd..ac1042806 100644
--- a/config/arm/arm64_octeontx2_linux_gcc
+++ b/config/arm/arm64_octeontx2_linux_gcc
@@ -13,5 +13,5 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x43'
-implementor_pn = '0xb2'
+implementer_id = '0x43'
+part_number = '0xb2'
diff --git a/config/arm/arm64_stingray_linux_gcc b/config/arm/arm64_stingray_linux_gcc
index 86797d23c..b79389d85 100644
--- a/config/arm/arm64_stingray_linux_gcc
+++ b/config/arm/arm64_stingray_linux_gcc
@@ -13,5 +13,5 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x41'
-implementor_pn = '0xd08'
+implementer_id = '0x41'
+part_number = '0xd08'
diff --git a/config/arm/arm64_thunderx2_linux_gcc b/config/arm/arm64_thunderx2_linux_gcc
index 2b41acc61..dd257745e 100644
--- a/config/arm/arm64_thunderx2_linux_gcc
+++ b/config/arm/arm64_thunderx2_linux_gcc
@@ -13,5 +13,5 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x43'
-implementor_pn = '0xaf'
+implementer_id = '0x43'
+part_number = '0xaf'
diff --git a/config/arm/arm64_thunderx_linux_gcc b/config/arm/arm64_thunderx_linux_gcc
index 6572ab615..670764437 100644
--- a/config/arm/arm64_thunderx_linux_gcc
+++ b/config/arm/arm64_thunderx_linux_gcc
@@ -13,4 +13,4 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x43'
+implementer_id = '0x43'
diff --git a/config/arm/meson.build b/config/arm/meson.build
index c371f983e..646438f09 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -3,12 +3,12 @@
 # Copyright(c) 2017 Cavium, Inc
 # Copyright(c) 2020 PANTHEON.tech s.r.o.
 
-# for checking defines we need to use the correct compiler flags
-march_opt = '-march=@0@'.format(machine)
-
+# set arm_force_native_march if you want to use machine args below
+# instead of discovered values; only works when doing an actual native build
 arm_force_native_march = false
-arm_force_generic_march = (machine == 'default' or machine == 'generic-armv8')
+native_machine_args = ['-march=native', '-mtune=native']
 
+# common flags to all aarch64 builds, with lowest priority
 flags_common_default = [
 	# Accelarate rte_memcpy. Be sure to run unit test (memcpy_perf_autotest)
 	# to determine the best threshold in code. Refer to notes in source file
@@ -16,8 +16,9 @@ flags_common_default = [
 	['RTE_ARCH_ARM64_MEMCPY', false],
 	#	['RTE_ARM64_MEMCPY_ALIGNED_THRESHOLD', 2048],
 	#	['RTE_ARM64_MEMCPY_UNALIGNED_THRESHOLD', 512],
-	# Leave below RTE_ARM64_MEMCPY_xxx options commented out, unless there're
-	# strong reasons.
+
+	# Leave below RTE_ARM64_MEMCPY_xxx options commented out,
+	# unless there are strong reasons.
 	#	['RTE_ARM64_MEMCPY_SKIP_GCC_VER_CHECK', false],
 	#	['RTE_ARM64_MEMCPY_ALIGN_MASK', 0xF],
 	#	['RTE_ARM64_MEMCPY_STRICT_ALIGN', false],
@@ -28,110 +29,124 @@ flags_common_default = [
 
 	['RTE_SCHED_VECTOR', false],
 	['RTE_ARM_USE_WFE', false],
+	['RTE_CACHE_LINE_SIZE', 128],
+	['RTE_ARCH_ARM64', true]
 ]
 
+# implementer specific aarch64 flags, with middle priority
+# (will overwrite common flags)
 flags_generic = [
 	['RTE_MACHINE', '"armv8a"'],
 	['RTE_MAX_LCORE', 256],
 	['RTE_USE_C11_MEM_MODEL', true],
-	['RTE_CACHE_LINE_SIZE', 128]]
+	['RTE_CACHE_LINE_SIZE', 128]
+]
 flags_arm = [
 	['RTE_MACHINE', '"armv8a"'],
 	['RTE_MAX_LCORE', 16],
 	['RTE_USE_C11_MEM_MODEL', true],
-	['RTE_CACHE_LINE_SIZE', 64]]
+	['RTE_CACHE_LINE_SIZE', 64]
+]
 flags_cavium = [
 	['RTE_CACHE_LINE_SIZE', 128],
 	['RTE_MAX_NUMA_NODES', 2],
 	['RTE_MAX_LCORE', 96],
-	['RTE_MAX_VFIO_GROUPS', 128]]
+	['RTE_MAX_VFIO_GROUPS', 128]
+]
 flags_dpaa = [
 	['RTE_MACHINE', '"dpaa"'],
 	['RTE_USE_C11_MEM_MODEL', true],
 	['RTE_CACHE_LINE_SIZE', 64],
 	['RTE_MAX_NUMA_NODES', 1],
 	['RTE_MAX_LCORE', 16],
-	['RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', false]]
+	['RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', false]
+]
 flags_emag = [
 	['RTE_MACHINE', '"emag"'],
-	['RTE_CACHE_LINE_SIZE', 64],
 	['RTE_MAX_NUMA_NODES', 1],
-	['RTE_MAX_LCORE', 32]]
+	['RTE_MAX_LCORE', 32],
+	['RTE_CACHE_LINE_SIZE', 64]
+]
 flags_armada = [
 	['RTE_MACHINE', '"armv8a"'],
-	['RTE_CACHE_LINE_SIZE', 64],
 	['RTE_MAX_NUMA_NODES', 1],
-	['RTE_MAX_LCORE', 16]]
+	['RTE_MAX_LCORE', 16],
+	['RTE_CACHE_LINE_SIZE', 64]
+]
 
-flags_default_extra = []
+# part number specific aarch64 flags, with highest priority
+# (will overwrite both common and implementer specific flags)
 flags_n1sdp_extra = [
 	['RTE_MACHINE', '"n1sdp"'],
 	['RTE_MAX_NUMA_NODES', 1],
 	['RTE_MAX_LCORE', 4],
 	['RTE_EAL_NUMA_AWARE_HUGEPAGES', false],
-	['RTE_LIBRTE_VHOST_NUMA', false]]
+	['RTE_LIBRTE_VHOST_NUMA', false]
+]
 flags_thunderx_extra = [
 	['RTE_MACHINE', '"thunderx"'],
-	['RTE_USE_C11_MEM_MODEL', false]]
+	['RTE_USE_C11_MEM_MODEL', false]
+]
 flags_thunderx2_extra = [
 	['RTE_MACHINE', '"thunderx2"'],
 	['RTE_CACHE_LINE_SIZE', 64],
 	['RTE_MAX_NUMA_NODES', 2],
 	['RTE_MAX_LCORE', 256],
 	['RTE_ARM_FEATURE_ATOMICS', true],
-	['RTE_USE_C11_MEM_MODEL', true]]
+	['RTE_USE_C11_MEM_MODEL', true]
+]
 flags_octeontx2_extra = [
 	['RTE_MACHINE', '"octeontx2"'],
 	['RTE_MAX_NUMA_NODES', 1],
 	['RTE_MAX_LCORE', 36],
 	['RTE_ARM_FEATURE_ATOMICS', true],
 	['RTE_EAL_IGB_UIO', false],
-	['RTE_USE_C11_MEM_MODEL', true]]
-
-machine_args_default = [
-	['default', ['-march=armv8-a+crc']],
-	['native', ['-march=native']],
-	['0xd03', ['-mcpu=cortex-a53']],
-	['0xd04', ['-mcpu=cortex-a35']],
-	['0xd07', ['-mcpu=cortex-a57']],
-	['0xd08', ['-mcpu=cortex-a72']],
-	['0xd09', ['-mcpu=cortex-a73']],
-	['0xd0a', ['-mcpu=cortex-a75']],
-	['0xd0b', ['-mcpu=cortex-a76']],
-	['0xd0c', ['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'], flags_n1sdp_extra]]
-
-machine_args_cavium = [
-	['default', ['-march=armv8-a+crc+crypto','-mcpu=thunderx']],
-	['native', ['-march=native']],
-	['0xa1', ['-mcpu=thunderxt88'], flags_thunderx_extra],
-	['0xa2', ['-mcpu=thunderxt81'], flags_thunderx_extra],
-	['0xa3', ['-mcpu=thunderxt83'], flags_thunderx_extra],
-	['0xaf', ['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'], flags_thunderx2_extra],
-	['0xb2', ['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'], flags_octeontx2_extra]]
-
-machine_args_emag = [
-	['default', ['-march=armv8-a+crc+crypto', '-mtune=emag']],
-	['native', ['-march=native']]]
+	['RTE_USE_C11_MEM_MODEL', true]
+]
+# arm config (implementer 0x41) is the default config
+pn_config_default = {
+	'generic': [['-march=armv8-a+crc']],
+	'0xd03': [['-mcpu=cortex-a53']],
+	'0xd04': [['-mcpu=cortex-a35']],
+	'0xd07': [['-mcpu=cortex-a57']],
+	'0xd08': [['-mcpu=cortex-a72']],
+	'0xd09': [['-mcpu=cortex-a73']],
+	'0xd0a': [['-mcpu=cortex-a75']],
+	'0xd0b': [['-mcpu=cortex-a76']],
+	'0xd0c': [['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'], flags_n1sdp_extra]
+}
+pn_config_cavium = {
+	'generic': [['-march=armv8-a+crc+crypto', '-mcpu=thunderx']],
+	'0xa1': [['-mcpu=thunderxt88'], flags_thunderx_extra],
+	'0xa2': [['-mcpu=thunderxt81'], flags_thunderx_extra],
+	'0xa3': [['-mcpu=thunderxt83'], flags_thunderx_extra],
+	'0xaf': [['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'], flags_thunderx2_extra],
+	'0xb2': [['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'], flags_octeontx2_extra],
+}
+pn_config_emag = {
+	'generic': [['-march=armv8-a+crc+crypto', '-mtune=emag']]
+}
 
 ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page G7-5321)
-impl_generic = ['Generic armv8', flags_generic, machine_args_default]
-impl_0x41 = ['Arm', flags_arm, machine_args_default]
-impl_0x42 = ['Broadcom', flags_generic, machine_args_default]
-impl_0x43 = ['Cavium', flags_cavium, machine_args_cavium]
-impl_0x44 = ['DEC', flags_generic, machine_args_default]
-impl_0x49 = ['Infineon', flags_generic, machine_args_default]
-impl_0x4d = ['Motorola', flags_generic, machine_args_default]
-impl_0x4e = ['NVIDIA', flags_generic, machine_args_default]
-impl_0x50 = ['Ampere Computing', flags_emag, machine_args_emag]
-impl_0x51 = ['Qualcomm', flags_generic, machine_args_default]
-impl_0x53 = ['Samsung', flags_generic, machine_args_default]
-impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_default]
-impl_0x69 = ['Intel', flags_generic, machine_args_default]
-impl_dpaa = ['NXP DPAA', flags_dpaa, machine_args_default]
+impl_generic = ['Generic armv8', flags_generic, pn_config_default]
+impl_0x41 = ['Arm', flags_arm, pn_config_default]
+impl_0x42 = ['Broadcom', flags_generic, pn_config_default]
+impl_0x43 = ['Cavium', flags_cavium, pn_config_cavium]
+impl_0x44 = ['DEC', flags_generic, pn_config_default]
+impl_0x49 = ['Infineon', flags_generic, pn_config_default]
+impl_0x4d = ['Motorola', flags_generic, pn_config_default]
+impl_0x4e = ['NVIDIA', flags_generic, pn_config_default]
+impl_0x50 = ['Ampere Computing', flags_emag, pn_config_emag]
+impl_0x51 = ['Qualcomm', flags_generic, pn_config_default]
+impl_0x53 = ['Samsung', flags_generic, pn_config_default]
+impl_0x56 = ['Marvell ARMADA', flags_armada, pn_config_default]
+impl_0x69 = ['Intel', flags_generic, pn_config_default]
+impl_dpaa = ['NXP DPAA', flags_dpaa, pn_config_default]
 
 dpdk_conf.set('RTE_FORCE_INTRINSICS', 1)
 
 if not dpdk_conf.get('RTE_ARCH_64')
+	# armv7 build
 	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
 	dpdk_conf.set('RTE_ARCH_ARM', 1)
 	dpdk_conf.set('RTE_ARCH_ARMv7', 1)
@@ -139,73 +154,82 @@ if not dpdk_conf.get('RTE_ARCH_64')
 	# mk/machine/armv7a/rte.vars.mk sets it too
 	machine_args += '-mfpu=neon'
 else
-	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
-	dpdk_conf.set('RTE_ARCH_ARM64', 1)
+	# aarch64 build
+	if not meson.is_cross_build()
+		if machine == 'default' or machine == 'generic-arvm8'
+			# default build
+			impl_config = impl_generic
+			part_number = 'generic'
+		else
+			# native build
+			# The script returns ['Implementer', 'Variant', 'Architecture',
+			# 'Primary Part number', 'Revision']
+			detect_vendor = find_program(join_paths(
+					meson.current_source_dir(), 'armv8_machine.py'))
+			cmd = run_command(detect_vendor.path())
+			if cmd.returncode() == 0
+				cmd_output = cmd.stdout().to_lower().strip().split(' ')
+			endif
+			if arm_force_native_march == true
+				part_number = 'native'
+			else
+				part_number = cmd_output[3]
+			endif
+			# Set to generic implementer if implementer is not found
+			impl_config = get_variable('impl_' + cmd_output[0], 'impl_generic')
+		endif
+	else
+		# cross build
+		impl_id = meson.get_cross_property('implementer_id', 'generic')
+		part_number = meson.get_cross_property('part_number', 'generic')
+		impl_config = get_variable('impl_' + impl_id)
+	endif
 
-	machine = []
-	cmd_generic = ['generic', '', '', 'default', '']
-	cmd_output = cmd_generic # Set generic by default
-	machine_args = [] # Clear previous machine args
-	if arm_force_generic_march and not meson.is_cross_build()
-		machine = impl_generic
-		impl_pn = 'default'
+	message('Arm implementer: ' + impl_config[0])
+	message('Arm part number: ' + part_number)
+
+	implementer_flags = impl_config[1]
+	part_number_config = impl_config[2]
+
+	if part_number_config.has_key(part_number)
+		# use the specified part_number machine args if found
+		part_number_config = part_number_config[part_number]
+	elif part_number == 'native'
+		# use native machine args
+		part_number_config = [[native_machine_args]]
 	elif not meson.is_cross_build()
-		# The script returns ['Implementer', 'Variant', 'Architecture',
-		# 'Primary Part number', 'Revision']
-		detect_vendor = find_program(join_paths(
-				meson.current_source_dir(), 'armv8_machine.py'))
-		cmd = run_command(detect_vendor.path())
-		if cmd.returncode() == 0
-			cmd_output = cmd.stdout().to_lower().strip().split(' ')
-		endif
-		# Set to generic if variable is not found
-		machine = get_variable('impl_' + cmd_output[0], ['generic'])
-		if machine[0] == 'generic'
-			machine = impl_generic
-			cmd_output = cmd_generic
-		endif
-		impl_pn = cmd_output[3]
-		if arm_force_native_march == true
-			impl_pn = 'native'
-		endif
+		# default to generic machine args if part_number is not found
+		# and not forcing native machine args
+		# but don't default in cross-builds; if part_number is specified
+		# incorrectly in a cross-file, it needs to be fixed there
+		part_number_config = part_number_config['generic']
 	else
-		impl_id = meson.get_cross_property('implementor_id', 'generic')
-		impl_pn = meson.get_cross_property('implementor_pn', 'default')
-		machine = get_variable('impl_' + impl_id)
+		# cross build and part number is not in part_number_config
+		error('Cross build part number 0@0 not found.'.format(part_number))
 	endif
 
-	# Apply Common Defaults. These settings may be overwritten by machine
-	# settings later.
-	foreach flag: flags_common_default
-		if flag.length() > 0
-			dpdk_conf.set(flag[0], flag[1])
+	dpdk_flags = flags_common_default + implementer_flags
+
+	if part_number_config.length() > 1
+		dpdk_flags += part_number_config[1]
+	endif
+
+	machine_args = [] # Clear previous machine args
+	foreach flag: part_number_config[0]
+		if cc.has_argument(flag)
+			machine_args += flag
 		endif
 	endforeach
 
-	message('Implementer : ' + machine[0])
-	foreach flag: machine[1]
+	foreach flag: dpdk_flags
 		if flag.length() > 0
 			dpdk_conf.set(flag[0], flag[1])
 		endif
 	endforeach
 
-	foreach marg: machine[2]
-		if marg[0] == impl_pn
-			foreach flag: marg[1]
-				if cc.has_argument(flag)
-					machine_args += flag
-				endif
-			endforeach
-			# Apply any extra machine specific flags.
-			foreach flag: marg.get(2, flags_default_extra)
-				if flag.length() > 0
-					dpdk_conf.set(flag[0], flag[1])
-				endif
-			endforeach
-		endif
-	endforeach
+
 endif
-message(machine_args)
+message('Using machine args: @0@'.format(machine_args))
 
 if (cc.get_define('__ARM_NEON', args: machine_args) != '' or
     cc.get_define('__aarch64__', args: machine_args) != '')
-- 
2.20.1

[dpdk-dev] [RFC PATCH v2 3/6] build: automatic NUMA and cpu counts detection

[Juraj Linkeš]

The build machine's number of cpus and numa nodes vary, resulting in
mismatched counts of RTE_MAX_LCORE and RTE_MAX_NUMA_NODES for many
builds. Automatically discover the host's numa and cpu counts to remove
this mismatch for native builds. Use current defaults for cross builds.
Leave users the option to override both if the specify a non-zero
amount on the command line.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 buildtools/get_cpu_count.py  |  7 +++++++
 buildtools/get_numa_count.py | 17 +++++++++++++++++
 buildtools/meson.build       |  2 ++
 config/meson.build           | 20 ++++++++++++++++++--
 meson_options.txt            |  8 ++++----
 5 files changed, 48 insertions(+), 6 deletions(-)
 create mode 100644 buildtools/get_cpu_count.py
 create mode 100644 buildtools/get_numa_count.py

diff --git a/buildtools/get_cpu_count.py b/buildtools/get_cpu_count.py
new file mode 100644
index 000000000..386f85f8b
--- /dev/null
+++ b/buildtools/get_cpu_count.py
@@ -0,0 +1,7 @@
+#!/usr/bin/python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2020 PANTHEON.tech s.r.o.
+
+import os
+
+print(os.cpu_count())
diff --git a/buildtools/get_numa_count.py b/buildtools/get_numa_count.py
new file mode 100644
index 000000000..f2ad35532
--- /dev/null
+++ b/buildtools/get_numa_count.py
@@ -0,0 +1,17 @@
+#!/usr/bin/python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2020 PANTHEON.tech s.r.o.
+
+import ctypes
+import glob
+import os
+
+if os.name == 'posix':
+    print(len(glob.glob('/sys/devices/system/node/node*')))
+elif os.name == 'nt':
+    libkernel32 = ctypes.windll.kernel32
+
+    count = ctypes.c_ulong()
+
+    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
+    print(count.value + 1)
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 04808dabc..925e733b1 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -17,3 +17,5 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+get_cpu_count_cmd = py3 + files('get_cpu_count.py')
+get_numa_count_cmd = py3 + files('get_numa_count.py')
diff --git a/config/meson.build b/config/meson.build
index 4bd65d98e..84c31c8e7 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -226,8 +226,24 @@ foreach arg: warning_flags
 endforeach
 
 # set other values pulled from the build options
-dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
-dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
+max_lcores = get_option('max_lcores')
+if max_lcores == 0
+	if meson.is_cross_build()
+		max_lcores = 4
+	else
+		max_lcores = run_command(get_cpu_count_cmd).stdout()
+	endif
+endif
+dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
+max_numa_nodes = get_option('max_numa_nodes')
+if max_numa_nodes == 0
+	if meson.is_cross_build()
+		max_numa_nodes = 128
+	else
+		max_numa_nodes = run_command(get_numa_count_cmd).stdout()
+	endif
+endif
+dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
 dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
 dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
 dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
diff --git a/meson_options.txt b/meson_options.txt
index 9bf18ab6b..60a949fca 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -26,10 +26,10 @@ option('machine', type: 'string', value: 'native',
 	description: 'set the target machine type')
 option('max_ethports', type: 'integer', value: 32,
 	description: 'maximum number of Ethernet devices')
-option('max_lcores', type: 'integer', value: 128,
-	description: 'maximum number of cores/threads supported by EAL')
-option('max_numa_nodes', type: 'integer', value: 4,
-	description: 'maximum number of NUMA nodes supported by EAL')
+option('max_lcores', type: 'integer', value: 0,
+	description: 'maximum number of cores/threads supported by EAL. Value 0 means the number of cpus on the host will be used')
+option('max_numa_nodes', type: 'integer', value: 0,
+	description: 'maximum number of NUMA nodes supported by EAL. Value 0 means the number of numa nodes on the host will be used')
 option('enable_trace_fp', type: 'boolean', value: false,
 	description: 'enable fast path trace points.')
 option('tests', type: 'boolean', value: true,
-- 
2.20.1

Re: [dpdk-dev] [RFC PATCH v2 3/6] build: automatic NUMA and cpu counts detection

[Bruce Richardson]

On Tue, Oct 13, 2020 at 04:54:39PM +0200, Juraj Linkeš wrote:
> The build machine's number of cpus and numa nodes vary, resulting in
> mismatched counts of RTE_MAX_LCORE and RTE_MAX_NUMA_NODES for many
> builds. Automatically discover the host's numa and cpu counts to remove
> this mismatch for native builds. Use current defaults for cross builds.
> Leave users the option to override both if the specify a non-zero
> amount on the command line.
> 
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
>  buildtools/get_cpu_count.py  |  7 +++++++
>  buildtools/get_numa_count.py | 17 +++++++++++++++++
>  buildtools/meson.build       |  2 ++
>  config/meson.build           | 20 ++++++++++++++++++--
>  meson_options.txt            |  8 ++++----
>  5 files changed, 48 insertions(+), 6 deletions(-)
>  create mode 100644 buildtools/get_cpu_count.py
>  create mode 100644 buildtools/get_numa_count.py
> 
> diff --git a/buildtools/get_cpu_count.py b/buildtools/get_cpu_count.py
> new file mode 100644
> index 000000000..386f85f8b
> --- /dev/null
> +++ b/buildtools/get_cpu_count.py
> @@ -0,0 +1,7 @@
> +#!/usr/bin/python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2020 PANTHEON.tech s.r.o.
> +
> +import os
> +
> +print(os.cpu_count())
> diff --git a/buildtools/get_numa_count.py b/buildtools/get_numa_count.py
> new file mode 100644
> index 000000000..f2ad35532
> --- /dev/null
> +++ b/buildtools/get_numa_count.py
> @@ -0,0 +1,17 @@
> +#!/usr/bin/python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2020 PANTHEON.tech s.r.o.
> +
> +import ctypes
> +import glob
> +import os
> +
> +if os.name == 'posix':
> +    print(len(glob.glob('/sys/devices/system/node/node*')))

FreeBSD also reports it's os.name as posix, but doesn't have a /sys/devices
path to query.

> +elif os.name == 'nt':
> +    libkernel32 = ctypes.windll.kernel32
> +
> +    count = ctypes.c_ulong()
> +
> +    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
> +    print(count.value + 1)
> diff --git a/buildtools/meson.build b/buildtools/meson.build
> index 04808dabc..925e733b1 100644
> --- a/buildtools/meson.build
> +++ b/buildtools/meson.build
> @@ -17,3 +17,5 @@ else
>  endif
>  map_to_win_cmd = py3 + files('map_to_win.py')
>  sphinx_wrapper = py3 + files('call-sphinx-build.py')
> +get_cpu_count_cmd = py3 + files('get_cpu_count.py')
> +get_numa_count_cmd = py3 + files('get_numa_count.py')
> diff --git a/config/meson.build b/config/meson.build
> index 4bd65d98e..84c31c8e7 100644
> --- a/config/meson.build
> +++ b/config/meson.build
> @@ -226,8 +226,24 @@ foreach arg: warning_flags
>  endforeach
>  
>  # set other values pulled from the build options
> -dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
> -dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
> +max_lcores = get_option('max_lcores')
> +if max_lcores == 0
> +	if meson.is_cross_build()
> +		max_lcores = 4

This seems a rather low number to default to. I would suspect in this case
you might be better just to error out, i.e. require that the max_lcores
value is given when cross-compiling.

> +	else
> +		max_lcores = run_command(get_cpu_count_cmd).stdout()
> +	endif
> +endif
> +dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> +max_numa_nodes = get_option('max_numa_nodes')
> +if max_numa_nodes == 0
> +	if meson.is_cross_build()
> +		max_numa_nodes = 128

Same here, I think it's better to force the user to specify a value.
BTW: did you unintentionally switch the max cores and max numa nodes values
in this patch?

> +	else
> +		max_numa_nodes = run_command(get_numa_count_cmd).stdout()
> +	endif
> +endif
> +dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
>  dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
>  dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
>  dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
> diff --git a/meson_options.txt b/meson_options.txt
> index 9bf18ab6b..60a949fca 100644
> --- a/meson_options.txt
> +++ b/meson_options.txt
> @@ -26,10 +26,10 @@ option('machine', type: 'string', value: 'native',
>  	description: 'set the target machine type')
>  option('max_ethports', type: 'integer', value: 32,
>  	description: 'maximum number of Ethernet devices')
> -option('max_lcores', type: 'integer', value: 128,
> -	description: 'maximum number of cores/threads supported by EAL')
> -option('max_numa_nodes', type: 'integer', value: 4,
> -	description: 'maximum number of NUMA nodes supported by EAL')
> +option('max_lcores', type: 'integer', value: 0,
> +	description: 'maximum number of cores/threads supported by EAL. Value 0 means the number of cpus on the host will be used')
> +option('max_numa_nodes', type: 'integer', value: 0,
> +	description: 'maximum number of NUMA nodes supported by EAL. Value 0 means the number of numa nodes on the host will be used')
>  option('enable_trace_fp', type: 'boolean', value: false,
>  	description: 'enable fast path trace points.')
>  option('tests', type: 'boolean', value: true,
> -- 
> 2.20.1
> 

Re: [dpdk-dev] [RFC PATCH v2 3/6] build: automatic NUMA and cpu counts detection

[Juraj Linkeš]

> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Tuesday, October 13, 2020 5:48 PM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Cc: Ruifeng.Wang@arm.com; Honnappa.Nagarahalli@arm.com;
> Phil.Yang@arm.com; vcchunga@amazon.com; Dharmik.Thakkar@arm.com;
> jerinjacobk@gmail.com; hemant.agrawal@nxp.com; dev@dpdk.org
> Subject: Re: [RFC PATCH v2 3/6] build: automatic NUMA and cpu counts
> detection
> 
> On Tue, Oct 13, 2020 at 04:54:39PM +0200, Juraj Linkeš wrote:
> > The build machine's number of cpus and numa nodes vary, resulting in
> > mismatched counts of RTE_MAX_LCORE and RTE_MAX_NUMA_NODES for
> many
> > builds. Automatically discover the host's numa and cpu counts to
> > remove this mismatch for native builds. Use current defaults for cross builds.
> > Leave users the option to override both if the specify a non-zero
> > amount on the command line.
> >
> > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > ---
> >  buildtools/get_cpu_count.py  |  7 +++++++
> > buildtools/get_numa_count.py | 17 +++++++++++++++++
> >  buildtools/meson.build       |  2 ++
> >  config/meson.build           | 20 ++++++++++++++++++--
> >  meson_options.txt            |  8 ++++----
> >  5 files changed, 48 insertions(+), 6 deletions(-)  create mode 100644
> > buildtools/get_cpu_count.py  create mode 100644
> > buildtools/get_numa_count.py
> >
> > diff --git a/buildtools/get_cpu_count.py b/buildtools/get_cpu_count.py
> > new file mode 100644 index 000000000..386f85f8b
> > --- /dev/null
> > +++ b/buildtools/get_cpu_count.py
> > @@ -0,0 +1,7 @@
> > +#!/usr/bin/python3
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > +PANTHEON.tech s.r.o.
> > +
> > +import os
> > +
> > +print(os.cpu_count())
> > diff --git a/buildtools/get_numa_count.py
> > b/buildtools/get_numa_count.py new file mode 100644 index
> > 000000000..f2ad35532
> > --- /dev/null
> > +++ b/buildtools/get_numa_count.py
> > @@ -0,0 +1,17 @@
> > +#!/usr/bin/python3
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > +PANTHEON.tech s.r.o.
> > +
> > +import ctypes
> > +import glob
> > +import os
> > +
> > +if os.name == 'posix':
> > +    print(len(glob.glob('/sys/devices/system/node/node*')))
> 
> FreeBSD also reports it's os.name as posix, but doesn't have a /sys/devices path
> to query.
> 

Do you know how do I get numa count on FreeBDS? I don't want to use the numa library if we can to avoid unneccesary dependencies. Do we need to cover some other cases other than FreeBDS, other Linux and Windows?

> > +elif os.name == 'nt':
> > +    libkernel32 = ctypes.windll.kernel32
> > +
> > +    count = ctypes.c_ulong()
> > +
> > +    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
> > +    print(count.value + 1)
> > diff --git a/buildtools/meson.build b/buildtools/meson.build index
> > 04808dabc..925e733b1 100644
> > --- a/buildtools/meson.build
> > +++ b/buildtools/meson.build
> > @@ -17,3 +17,5 @@ else
> >  endif
> >  map_to_win_cmd = py3 + files('map_to_win.py')  sphinx_wrapper = py3 +
> > files('call-sphinx-build.py')
> > +get_cpu_count_cmd = py3 + files('get_cpu_count.py')
> > +get_numa_count_cmd = py3 + files('get_numa_count.py')
> > diff --git a/config/meson.build b/config/meson.build index
> > 4bd65d98e..84c31c8e7 100644
> > --- a/config/meson.build
> > +++ b/config/meson.build
> > @@ -226,8 +226,24 @@ foreach arg: warning_flags  endforeach
> >
> >  # set other values pulled from the build options
> > -dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
> > -dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
> > +max_lcores = get_option('max_lcores') if max_lcores == 0
> > +	if meson.is_cross_build()
> > +		max_lcores = 4
> 
> This seems a rather low number to default to. I would suspect in this case you
> might be better just to error out, i.e. require that the max_lcores value is given
> when cross-compiling.
> 
> > +	else
> > +		max_lcores = run_command(get_cpu_count_cmd).stdout()
> > +	endif
> > +endif
> > +dpdk_conf.set('RTE_MAX_LCORE', max_lcores) max_numa_nodes =
> > +get_option('max_numa_nodes') if max_numa_nodes == 0
> > +	if meson.is_cross_build()
> > +		max_numa_nodes = 128
> 
> Same here, I think it's better to force the user to specify a value.
> BTW: did you unintentionally switch the max cores and max numa nodes values
> in this patch?
> 

Yes, the values are switched. Forcing the user to specify these might be tricky - we want to specify cpu and numa counts in cross files and this check happens before the values from cross files are used (in another patch I'm asking about having defaults in config/arm/meson.build and if we don't have defaults we won't want them there as well. The remaning question would be what get priority - command line options or cross file options. I'm leaning towards command line options). I'll think about switching the order.

> > +	else
> > +		max_numa_nodes =
> run_command(get_numa_count_cmd).stdout()
> > +	endif
> > +endif
> > +dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
> >  dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
> > dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
> > dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
> > diff --git a/meson_options.txt b/meson_options.txt index
> > 9bf18ab6b..60a949fca 100644
> > --- a/meson_options.txt
> > +++ b/meson_options.txt
> > @@ -26,10 +26,10 @@ option('machine', type: 'string', value: 'native',
> >  	description: 'set the target machine type')  option('max_ethports',
> > type: 'integer', value: 32,
> >  	description: 'maximum number of Ethernet devices')
> > -option('max_lcores', type: 'integer', value: 128,
> > -	description: 'maximum number of cores/threads supported by EAL')
> > -option('max_numa_nodes', type: 'integer', value: 4,
> > -	description: 'maximum number of NUMA nodes supported by EAL')
> > +option('max_lcores', type: 'integer', value: 0,
> > +	description: 'maximum number of cores/threads supported by EAL.
> > +Value 0 means the number of cpus on the host will be used')
> option('max_numa_nodes', type: 'integer', value: 0,
> > +	description: 'maximum number of NUMA nodes supported by EAL. Value
> 0
> > +means the number of numa nodes on the host will be used')
> >  option('enable_trace_fp', type: 'boolean', value: false,
> >  	description: 'enable fast path trace points.')  option('tests',
> > type: 'boolean', value: true,
> > --
> > 2.20.1
> >

Re: [dpdk-dev] [RFC PATCH v2 3/6] build: automatic NUMA and cpu counts detection

[Bruce Richardson]

On Wed, Oct 14, 2020 at 06:04:52AM +0000, Juraj Linkeš wrote:
> 
> 
> > -----Original Message-----
> > From: Bruce Richardson <bruce.richardson@intel.com>
> > Sent: Tuesday, October 13, 2020 5:48 PM
> > To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > Cc: Ruifeng.Wang@arm.com; Honnappa.Nagarahalli@arm.com;
> > Phil.Yang@arm.com; vcchunga@amazon.com; Dharmik.Thakkar@arm.com;
> > jerinjacobk@gmail.com; hemant.agrawal@nxp.com; dev@dpdk.org
> > Subject: Re: [RFC PATCH v2 3/6] build: automatic NUMA and cpu counts
> > detection
> > 
> > On Tue, Oct 13, 2020 at 04:54:39PM +0200, Juraj Linkeš wrote:
> > > The build machine's number of cpus and numa nodes vary, resulting in
> > > mismatched counts of RTE_MAX_LCORE and RTE_MAX_NUMA_NODES for
> > many
> > > builds. Automatically discover the host's numa and cpu counts to
> > > remove this mismatch for native builds. Use current defaults for cross builds.
> > > Leave users the option to override both if the specify a non-zero
> > > amount on the command line.
> > >
> > > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > ---
> > >  buildtools/get_cpu_count.py  |  7 +++++++
> > > buildtools/get_numa_count.py | 17 +++++++++++++++++
> > >  buildtools/meson.build       |  2 ++
> > >  config/meson.build           | 20 ++++++++++++++++++--
> > >  meson_options.txt            |  8 ++++----
> > >  5 files changed, 48 insertions(+), 6 deletions(-)  create mode 100644
> > > buildtools/get_cpu_count.py  create mode 100644
> > > buildtools/get_numa_count.py
> > >
> > > diff --git a/buildtools/get_cpu_count.py b/buildtools/get_cpu_count.py
> > > new file mode 100644 index 000000000..386f85f8b
> > > --- /dev/null
> > > +++ b/buildtools/get_cpu_count.py
> > > @@ -0,0 +1,7 @@
> > > +#!/usr/bin/python3
> > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > > +PANTHEON.tech s.r.o.
> > > +
> > > +import os
> > > +
> > > +print(os.cpu_count())
> > > diff --git a/buildtools/get_numa_count.py
> > > b/buildtools/get_numa_count.py new file mode 100644 index
> > > 000000000..f2ad35532
> > > --- /dev/null
> > > +++ b/buildtools/get_numa_count.py
> > > @@ -0,0 +1,17 @@
> > > +#!/usr/bin/python3
> > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > > +PANTHEON.tech s.r.o.
> > > +
> > > +import ctypes
> > > +import glob
> > > +import os
> > > +
> > > +if os.name == 'posix':
> > > +    print(len(glob.glob('/sys/devices/system/node/node*')))
> > 
> > FreeBSD also reports it's os.name as posix, but doesn't have a /sys/devices path
> > to query.
> > 
> 
> Do you know how do I get numa count on FreeBDS? I don't want to use the numa library if we can to avoid unneccesary dependencies. Do we need to cover some other cases other than FreeBDS, other Linux and Windows?
> 

To get numa count on freebsd, you can query it using "sysctl vm.ndomains"
command. However, the bigger issue here is that you need something
different other than os.name to distinguish between linux and freebsd.

Regards,
/Bruce

Re: [dpdk-dev] [RFC PATCH v2 3/6] build: automatic NUMA and cpu counts detection

[Juraj Linkeš]

> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Wednesday, October 14, 2020 10:16 AM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Cc: Ruifeng.Wang@arm.com; Honnappa.Nagarahalli@arm.com;
> Phil.Yang@arm.com; vcchunga@amazon.com; Dharmik.Thakkar@arm.com;
> jerinjacobk@gmail.com; hemant.agrawal@nxp.com; dev@dpdk.org
> Subject: Re: [RFC PATCH v2 3/6] build: automatic NUMA and cpu counts
> detection
> 
> On Wed, Oct 14, 2020 at 06:04:52AM +0000, Juraj Linkeš wrote:
> >
> >
> > > -----Original Message-----
> > > From: Bruce Richardson <bruce.richardson@intel.com>
> > > Sent: Tuesday, October 13, 2020 5:48 PM
> > > To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > Cc: Ruifeng.Wang@arm.com; Honnappa.Nagarahalli@arm.com;
> > > Phil.Yang@arm.com; vcchunga@amazon.com; Dharmik.Thakkar@arm.com;
> > > jerinjacobk@gmail.com; hemant.agrawal@nxp.com; dev@dpdk.org
> > > Subject: Re: [RFC PATCH v2 3/6] build: automatic NUMA and cpu counts
> > > detection
> > >
> > > On Tue, Oct 13, 2020 at 04:54:39PM +0200, Juraj Linkeš wrote:
> > > > The build machine's number of cpus and numa nodes vary, resulting
> > > > in mismatched counts of RTE_MAX_LCORE and RTE_MAX_NUMA_NODES
> for
> > > many
> > > > builds. Automatically discover the host's numa and cpu counts to
> > > > remove this mismatch for native builds. Use current defaults for cross
> builds.
> > > > Leave users the option to override both if the specify a non-zero
> > > > amount on the command line.
> > > >
> > > > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > > ---
> > > >  buildtools/get_cpu_count.py  |  7 +++++++
> > > > buildtools/get_numa_count.py | 17 +++++++++++++++++
> > > >  buildtools/meson.build       |  2 ++
> > > >  config/meson.build           | 20 ++++++++++++++++++--
> > > >  meson_options.txt            |  8 ++++----
> > > >  5 files changed, 48 insertions(+), 6 deletions(-)  create mode
> > > > 100644 buildtools/get_cpu_count.py  create mode 100644
> > > > buildtools/get_numa_count.py
> > > >
> > > > diff --git a/buildtools/get_cpu_count.py
> > > > b/buildtools/get_cpu_count.py new file mode 100644 index
> > > > 000000000..386f85f8b
> > > > --- /dev/null
> > > > +++ b/buildtools/get_cpu_count.py
> > > > @@ -0,0 +1,7 @@
> > > > +#!/usr/bin/python3
> > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > > > +PANTHEON.tech s.r.o.
> > > > +
> > > > +import os
> > > > +
> > > > +print(os.cpu_count())
> > > > diff --git a/buildtools/get_numa_count.py
> > > > b/buildtools/get_numa_count.py new file mode 100644 index
> > > > 000000000..f2ad35532
> > > > --- /dev/null
> > > > +++ b/buildtools/get_numa_count.py
> > > > @@ -0,0 +1,17 @@
> > > > +#!/usr/bin/python3
> > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > > > +PANTHEON.tech s.r.o.
> > > > +
> > > > +import ctypes
> > > > +import glob
> > > > +import os
> > > > +
> > > > +if os.name == 'posix':
> > > > +    print(len(glob.glob('/sys/devices/system/node/node*')))
> > >
> > > FreeBSD also reports it's os.name as posix, but doesn't have a
> > > /sys/devices path to query.
> > >
> >
> > Do you know how do I get numa count on FreeBDS? I don't want to use the
> numa library if we can to avoid unneccesary dependencies. Do we need to cover
> some other cases other than FreeBDS, other Linux and Windows?
> >
> 
> To get numa count on freebsd, you can query it using "sysctl vm.ndomains"
> command. However, the bigger issue here is that you need something different
> other than os.name to distinguish between linux and freebsd.
> 

That should only be a matter of finding the right Python api - https://docs.python.org/3/library/sys.html?highlight=platform#sys.platform should be it.

> Regards,
> /Bruce

Re: [dpdk-dev] [RFC PATCH v2 3/6] build: automatic NUMA and cpu counts detection

[Bruce Richardson]

On Wed, Oct 14, 2020 at 08:52:12AM +0000, Juraj Linkeš wrote:
> 
> 
> > -----Original Message-----
> > From: Bruce Richardson <bruce.richardson@intel.com>
> > Sent: Wednesday, October 14, 2020 10:16 AM
> > To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > Cc: Ruifeng.Wang@arm.com; Honnappa.Nagarahalli@arm.com;
> > Phil.Yang@arm.com; vcchunga@amazon.com; Dharmik.Thakkar@arm.com;
> > jerinjacobk@gmail.com; hemant.agrawal@nxp.com; dev@dpdk.org
> > Subject: Re: [RFC PATCH v2 3/6] build: automatic NUMA and cpu counts
> > detection
> > 
> > On Wed, Oct 14, 2020 at 06:04:52AM +0000, Juraj Linkeš wrote:
> > >
> > >
> > > > -----Original Message-----
> > > > From: Bruce Richardson <bruce.richardson@intel.com>
> > > > Sent: Tuesday, October 13, 2020 5:48 PM
> > > > To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > > Cc: Ruifeng.Wang@arm.com; Honnappa.Nagarahalli@arm.com;
> > > > Phil.Yang@arm.com; vcchunga@amazon.com; Dharmik.Thakkar@arm.com;
> > > > jerinjacobk@gmail.com; hemant.agrawal@nxp.com; dev@dpdk.org
> > > > Subject: Re: [RFC PATCH v2 3/6] build: automatic NUMA and cpu counts
> > > > detection
> > > >
> > > > On Tue, Oct 13, 2020 at 04:54:39PM +0200, Juraj Linkeš wrote:
> > > > > The build machine's number of cpus and numa nodes vary, resulting
> > > > > in mismatched counts of RTE_MAX_LCORE and RTE_MAX_NUMA_NODES
> > for
> > > > many
> > > > > builds. Automatically discover the host's numa and cpu counts to
> > > > > remove this mismatch for native builds. Use current defaults for cross
> > builds.
> > > > > Leave users the option to override both if the specify a non-zero
> > > > > amount on the command line.
> > > > >
> > > > > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > > > ---
> > > > >  buildtools/get_cpu_count.py  |  7 +++++++
> > > > > buildtools/get_numa_count.py | 17 +++++++++++++++++
> > > > >  buildtools/meson.build       |  2 ++
> > > > >  config/meson.build           | 20 ++++++++++++++++++--
> > > > >  meson_options.txt            |  8 ++++----
> > > > >  5 files changed, 48 insertions(+), 6 deletions(-)  create mode
> > > > > 100644 buildtools/get_cpu_count.py  create mode 100644
> > > > > buildtools/get_numa_count.py
> > > > >
> > > > > diff --git a/buildtools/get_cpu_count.py
> > > > > b/buildtools/get_cpu_count.py new file mode 100644 index
> > > > > 000000000..386f85f8b
> > > > > --- /dev/null
> > > > > +++ b/buildtools/get_cpu_count.py
> > > > > @@ -0,0 +1,7 @@
> > > > > +#!/usr/bin/python3
> > > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > > > > +PANTHEON.tech s.r.o.
> > > > > +
> > > > > +import os
> > > > > +
> > > > > +print(os.cpu_count())
> > > > > diff --git a/buildtools/get_numa_count.py
> > > > > b/buildtools/get_numa_count.py new file mode 100644 index
> > > > > 000000000..f2ad35532
> > > > > --- /dev/null
> > > > > +++ b/buildtools/get_numa_count.py
> > > > > @@ -0,0 +1,17 @@
> > > > > +#!/usr/bin/python3
> > > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > > > > +PANTHEON.tech s.r.o.
> > > > > +
> > > > > +import ctypes
> > > > > +import glob
> > > > > +import os
> > > > > +
> > > > > +if os.name == 'posix':
> > > > > +    print(len(glob.glob('/sys/devices/system/node/node*')))
> > > >
> > > > FreeBSD also reports it's os.name as posix, but doesn't have a
> > > > /sys/devices path to query.
> > > >
> > >
> > > Do you know how do I get numa count on FreeBDS? I don't want to use the
> > numa library if we can to avoid unneccesary dependencies. Do we need to cover
> > some other cases other than FreeBDS, other Linux and Windows?
> > >
> > 
> > To get numa count on freebsd, you can query it using "sysctl vm.ndomains"
> > command. However, the bigger issue here is that you need something different
> > other than os.name to distinguish between linux and freebsd.
> > 
> 
> That should only be a matter of finding the right Python api - https://docs.python.org/3/library/sys.html?highlight=platform#sys.platform should be it.
>
Yep. Though thinking about it afterwards too, checking between bsd and
linux could also be as simple as using the /sys path if it exists, and
trying to use sysctl otherwise.

/Bruce 

[dpdk-dev] [RFC PATCH v2 4/6] build: move core and NUMA counts to cross files

[Juraj Linkeš]

The values of RTE_MAX_NUMA_NODES and RTE_MAX_LCORE are going to vary for
different build machines, even for the same Implementer ID and Part
Number combinations. Move the fixed values to cross files and use
automatic discovery/values from cmdline options for native build. Use
predefined values for generic-armv8 build, since that build is not tied
to the build machine.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/arm64_armada_linux_gcc    |  2 ++
 config/arm/arm64_armv8_linux_gcc     |  5 ++++
 config/arm/arm64_bluefield_linux_gcc |  1 +
 config/arm/arm64_dpaa_linux_gcc      |  2 ++
 config/arm/arm64_emag_linux_gcc      |  2 ++
 config/arm/arm64_n1sdp_linux_gcc     |  2 ++
 config/arm/arm64_octeontx2_linux_gcc |  2 ++
 config/arm/arm64_stingray_linux_gcc  |  1 +
 config/arm/arm64_thunderx2_linux_gcc |  2 ++
 config/arm/arm64_thunderx_linux_gcc  |  2 ++
 config/arm/meson.build               | 36 +++++++++++++++-------------
 11 files changed, 41 insertions(+), 16 deletions(-)

diff --git a/config/arm/arm64_armada_linux_gcc b/config/arm/arm64_armada_linux_gcc
index 52c5f4476..dca2b911e 100644
--- a/config/arm/arm64_armada_linux_gcc
+++ b/config/arm/arm64_armada_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 
 [properties]
 implementer_id = '0x56'
+max_numa_nodes = 1
+max_lcore = 16
diff --git a/config/arm/arm64_armv8_linux_gcc b/config/arm/arm64_armv8_linux_gcc
index 13ee8b223..ac12fa62d 100644
--- a/config/arm/arm64_armv8_linux_gcc
+++ b/config/arm/arm64_armv8_linux_gcc
@@ -26,3 +26,8 @@ implementer_id = 'generic'
 # '0xd0a':   cortex-a75
 # '0xd0b':   cortex-a76
 part_number = 'generic'
+
+# Supported extra configuration
+# max_numa_nodes = n  # will set RTE_MAX_NUMA_NODES
+# max_lcore = n       # will set RTE_MAX_LCORE
+max_lcore = 256
diff --git a/config/arm/arm64_bluefield_linux_gcc b/config/arm/arm64_bluefield_linux_gcc
index b79389d85..40265ac34 100644
--- a/config/arm/arm64_bluefield_linux_gcc
+++ b/config/arm/arm64_bluefield_linux_gcc
@@ -15,3 +15,4 @@ endian = 'little'
 [properties]
 implementer_id = '0x41'
 part_number = '0xd08'
+max_lcore = 16
diff --git a/config/arm/arm64_dpaa_linux_gcc b/config/arm/arm64_dpaa_linux_gcc
index 573ae7e42..c5f738006 100644
--- a/config/arm/arm64_dpaa_linux_gcc
+++ b/config/arm/arm64_dpaa_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 
 [properties]
 implementer_id = 'dpaa'
+max_numa_nodes = 1
+max_lcore = 16
diff --git a/config/arm/arm64_emag_linux_gcc b/config/arm/arm64_emag_linux_gcc
index 24f3d533e..a2c10133f 100644
--- a/config/arm/arm64_emag_linux_gcc
+++ b/config/arm/arm64_emag_linux_gcc
@@ -14,3 +14,5 @@ endian = 'little'
 
 [properties]
 implementer_id = '0x50'
+max_numa_nodes = 1
+max_lcore = 32
diff --git a/config/arm/arm64_n1sdp_linux_gcc b/config/arm/arm64_n1sdp_linux_gcc
index 6fb3f02ea..05fad95e8 100644
--- a/config/arm/arm64_n1sdp_linux_gcc
+++ b/config/arm/arm64_n1sdp_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 [properties]
 implementer_id = '0x41'
 part_number = '0xd0c'
+max_numa_nodes = 1
+max_lcore = 4
diff --git a/config/arm/arm64_octeontx2_linux_gcc b/config/arm/arm64_octeontx2_linux_gcc
index ac1042806..2ed9fd03d 100644
--- a/config/arm/arm64_octeontx2_linux_gcc
+++ b/config/arm/arm64_octeontx2_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 [properties]
 implementer_id = '0x43'
 part_number = '0xb2'
+max_numa_nodes = 1
+max_lcore = 36
diff --git a/config/arm/arm64_stingray_linux_gcc b/config/arm/arm64_stingray_linux_gcc
index b79389d85..40265ac34 100644
--- a/config/arm/arm64_stingray_linux_gcc
+++ b/config/arm/arm64_stingray_linux_gcc
@@ -15,3 +15,4 @@ endian = 'little'
 [properties]
 implementer_id = '0x41'
 part_number = '0xd08'
+max_lcore = 16
diff --git a/config/arm/arm64_thunderx2_linux_gcc b/config/arm/arm64_thunderx2_linux_gcc
index dd257745e..d538a59e1 100644
--- a/config/arm/arm64_thunderx2_linux_gcc
+++ b/config/arm/arm64_thunderx2_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 [properties]
 implementer_id = '0x43'
 part_number = '0xaf'
+max_numa_nodes = 2
+max_lcore = 256
diff --git a/config/arm/arm64_thunderx_linux_gcc b/config/arm/arm64_thunderx_linux_gcc
index 670764437..656d103e3 100644
--- a/config/arm/arm64_thunderx_linux_gcc
+++ b/config/arm/arm64_thunderx_linux_gcc
@@ -14,3 +14,5 @@ endian = 'little'
 
 [properties]
 implementer_id = '0x43'
+max_numa_nodes = 1
+max_lcore = 96
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 646438f09..f525403f6 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -33,44 +33,40 @@ flags_common_default = [
 	['RTE_ARCH_ARM64', true]
 ]
 
+# add these to defaults when machine='generic-armv8'
+flags_force_generic = [
+	['RTE_MAX_NUMA_NODES', 4],
+	['RTE_MAX_LCORE', 256]
+]
+
 # implementer specific aarch64 flags, with middle priority
 # (will overwrite common flags)
 flags_generic = [
 	['RTE_MACHINE', '"armv8a"'],
-	['RTE_MAX_LCORE', 256],
 	['RTE_USE_C11_MEM_MODEL', true],
 	['RTE_CACHE_LINE_SIZE', 128]
 ]
 flags_arm = [
 	['RTE_MACHINE', '"armv8a"'],
-	['RTE_MAX_LCORE', 16],
 	['RTE_USE_C11_MEM_MODEL', true],
 	['RTE_CACHE_LINE_SIZE', 64]
 ]
 flags_cavium = [
 	['RTE_CACHE_LINE_SIZE', 128],
-	['RTE_MAX_NUMA_NODES', 2],
-	['RTE_MAX_LCORE', 96],
 	['RTE_MAX_VFIO_GROUPS', 128]
 ]
 flags_dpaa = [
 	['RTE_MACHINE', '"dpaa"'],
 	['RTE_USE_C11_MEM_MODEL', true],
 	['RTE_CACHE_LINE_SIZE', 64],
-	['RTE_MAX_NUMA_NODES', 1],
-	['RTE_MAX_LCORE', 16],
 	['RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', false]
 ]
 flags_emag = [
 	['RTE_MACHINE', '"emag"'],
-	['RTE_MAX_NUMA_NODES', 1],
-	['RTE_MAX_LCORE', 32],
 	['RTE_CACHE_LINE_SIZE', 64]
 ]
 flags_armada = [
 	['RTE_MACHINE', '"armv8a"'],
-	['RTE_MAX_NUMA_NODES', 1],
-	['RTE_MAX_LCORE', 16],
 	['RTE_CACHE_LINE_SIZE', 64]
 ]
 
@@ -78,8 +74,6 @@ flags_armada = [
 # (will overwrite both common and implementer specific flags)
 flags_n1sdp_extra = [
 	['RTE_MACHINE', '"n1sdp"'],
-	['RTE_MAX_NUMA_NODES', 1],
-	['RTE_MAX_LCORE', 4],
 	['RTE_EAL_NUMA_AWARE_HUGEPAGES', false],
 	['RTE_LIBRTE_VHOST_NUMA', false]
 ]
@@ -90,15 +84,11 @@ flags_thunderx_extra = [
 flags_thunderx2_extra = [
 	['RTE_MACHINE', '"thunderx2"'],
 	['RTE_CACHE_LINE_SIZE', 64],
-	['RTE_MAX_NUMA_NODES', 2],
-	['RTE_MAX_LCORE', 256],
 	['RTE_ARM_FEATURE_ATOMICS', true],
 	['RTE_USE_C11_MEM_MODEL', true]
 ]
 flags_octeontx2_extra = [
 	['RTE_MACHINE', '"octeontx2"'],
-	['RTE_MAX_NUMA_NODES', 1],
-	['RTE_MAX_LCORE', 36],
 	['RTE_ARM_FEATURE_ATOMICS', true],
 	['RTE_EAL_IGB_UIO', false],
 	['RTE_USE_C11_MEM_MODEL', true]
@@ -160,6 +150,7 @@ else
 			# default build
 			impl_config = impl_generic
 			part_number = 'generic'
+			flags_common_default += flags_force_generic
 		else
 			# native build
 			# The script returns ['Implementer', 'Variant', 'Architecture',
@@ -214,6 +205,19 @@ else
 		dpdk_flags += part_number_config[1]
 	endif
 
+	# apply cross-specific options
+	if meson.is_cross_build()
+		# configure RTE_MAX_NUMA_NODES and RTE_MAX_LCORE from cross file
+		# RFC: do we want defaults for these two? or should we require
+		# that all cross file define them?
+		dpdk_flags += [
+			['RTE_MAX_NUMA_NODES', meson.get_cross_property('max_numa_nodes', 1)]
+		]
+		dpdk_flags += [
+			['RTE_MAX_LCORE', meson.get_cross_property('max_lcore', 16)]
+		]
+	endif
+
 	machine_args = [] # Clear previous machine args
 	foreach flag: part_number_config[0]
 		if cc.has_argument(flag)
-- 
2.20.1

[dpdk-dev] [RFC PATCH v2 5/6] build: disable Arm drivers

[Juraj Linkeš]

A few options that disabled drivers in the old makefiles were improperly
ported to the meson build system. Fix this by adding a to the list of
disabled drivers, similarly how the command line option works.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/meson.build | 9 +++++----
 drivers/meson.build    | 2 +-
 meson.build            | 1 +
 3 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/config/arm/meson.build b/config/arm/meson.build
index f525403f6..3a1e13f64 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -8,6 +8,11 @@
 arm_force_native_march = false
 native_machine_args = ['-march=native', '-mtune=native']
 
+# RFC: do we want these drivers to be disabled by default on arm?
+# The previous way to disable these drivers never worked
+# RTE_LIBRTE_AVP_PMD and the rest would be overwritten to true if the necessary deps were found
+disabled_drivers += ['net/avp', 'net/fm10k', 'net/sfc']
+
 # common flags to all aarch64 builds, with lowest priority
 flags_common_default = [
 	# Accelarate rte_memcpy. Be sure to run unit test (memcpy_perf_autotest)
@@ -23,10 +28,6 @@ flags_common_default = [
 	#	['RTE_ARM64_MEMCPY_ALIGN_MASK', 0xF],
 	#	['RTE_ARM64_MEMCPY_STRICT_ALIGN', false],
 
-	['RTE_LIBRTE_FM10K_PMD', false],
-	['RTE_LIBRTE_SFC_EFX_PMD', false],
-	['RTE_LIBRTE_AVP_PMD', false],
-
 	['RTE_SCHED_VECTOR', false],
 	['RTE_ARM_USE_WFE', false],
 	['RTE_CACHE_LINE_SIZE', 128],
diff --git a/drivers/meson.build b/drivers/meson.build
index 5f9526557..f9250f3bb 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -17,7 +17,7 @@ subdirs = [
 	'baseband', # depends on common and bus.
 ]
 
-disabled_drivers = run_command(list_dir_globs, get_option('disable_drivers'),
+disabled_drivers += run_command(list_dir_globs, get_option('disable_drivers'),
 		).stdout().split()
 
 default_cflags = machine_args
diff --git a/meson.build b/meson.build
index 61d9a4f5f..8dadd70dc 100644
--- a/meson.build
+++ b/meson.build
@@ -21,6 +21,7 @@ dpdk_drivers = []
 dpdk_extra_ldflags = []
 dpdk_libs_disabled = []
 dpdk_drvs_disabled = []
+disabled_drivers = []
 abi_version_file = files('ABI_VERSION')
 
 if host_machine.cpu_family().startswith('x86')
-- 
2.20.1

Re: [dpdk-dev] [RFC PATCH v2 5/6] build: disable Arm drivers

[Bruce Richardson]

On Tue, Oct 13, 2020 at 04:54:41PM +0200, Juraj Linkeš wrote:
> A few options that disabled drivers in the old makefiles were improperly
> ported to the meson build system. Fix this by adding a to the list of
> disabled drivers, similarly how the command line option works.
> 
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
>  config/arm/meson.build | 9 +++++----
>  drivers/meson.build    | 2 +-
>  meson.build            | 1 +
>  3 files changed, 7 insertions(+), 5 deletions(-)
> 
This seems a better approach to disabling the drivers.

Acked-by: Bruce Richardson <bruce.richardson@intel.com>

[dpdk-dev] [RFC PATCH v2 6/6] build: update Arm builds with makefile flags

[Juraj Linkeš]

Not all flags were moved properly from the old Makefile build system.
Disable corresponding drivers and libnuma in cross files and remove
deprecated config.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/arm64_armada_linux_gcc    | 2 ++
 config/arm/arm64_armv8_linux_gcc     | 8 ++++++--
 config/arm/arm64_bluefield_linux_gcc | 1 +
 config/arm/arm64_dpaa_linux_gcc      | 1 +
 config/arm/arm64_n1sdp_linux_gcc     | 1 +
 config/arm/arm64_octeontx2_linux_gcc | 1 +
 config/arm/arm64_stingray_linux_gcc  | 1 +
 config/arm/meson.build               | 5 ++++-
 drivers/meson.build                  | 4 ++++
 9 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/config/arm/arm64_armada_linux_gcc b/config/arm/arm64_armada_linux_gcc
index dca2b911e..8fa856fed 100644
--- a/config/arm/arm64_armada_linux_gcc
+++ b/config/arm/arm64_armada_linux_gcc
@@ -17,3 +17,5 @@ endian = 'little'
 implementer_id = '0x56'
 max_numa_nodes = 1
 max_lcore = 16
+has_libnuma = 0
+disabled_drivers = ['bus/dpaa', 'bus/fslmc', 'common/dpaax']
diff --git a/config/arm/arm64_armv8_linux_gcc b/config/arm/arm64_armv8_linux_gcc
index ac12fa62d..797152b9e 100644
--- a/config/arm/arm64_armv8_linux_gcc
+++ b/config/arm/arm64_armv8_linux_gcc
@@ -28,6 +28,10 @@ implementer_id = 'generic'
 part_number = 'generic'
 
 # Supported extra configuration
-# max_numa_nodes = n  # will set RTE_MAX_NUMA_NODES
-# max_lcore = n       # will set RTE_MAX_LCORE
+# max_numa_nodes = n   # will set RTE_MAX_NUMA_NODES
+# max_lcore = n        # will set RTE_MAX_LCORE
 max_lcore = 256
+
+# has_libnuma = 0      # set to 0 if the SoC is not a NUMA system
+# disabled_drivers += ['bus/dpaa', 'crypto']
+    # add to the set of disabled libraries
diff --git a/config/arm/arm64_bluefield_linux_gcc b/config/arm/arm64_bluefield_linux_gcc
index 40265ac34..e25067b7e 100644
--- a/config/arm/arm64_bluefield_linux_gcc
+++ b/config/arm/arm64_bluefield_linux_gcc
@@ -16,3 +16,4 @@ endian = 'little'
 implementer_id = '0x41'
 part_number = '0xd08'
 max_lcore = 16
+has_libnuma = 0
diff --git a/config/arm/arm64_dpaa_linux_gcc b/config/arm/arm64_dpaa_linux_gcc
index c5f738006..120e90ed9 100644
--- a/config/arm/arm64_dpaa_linux_gcc
+++ b/config/arm/arm64_dpaa_linux_gcc
@@ -17,3 +17,4 @@ endian = 'little'
 implementer_id = 'dpaa'
 max_numa_nodes = 1
 max_lcore = 16
+has_libnuma = 0
diff --git a/config/arm/arm64_n1sdp_linux_gcc b/config/arm/arm64_n1sdp_linux_gcc
index 05fad95e8..0d249dffa 100644
--- a/config/arm/arm64_n1sdp_linux_gcc
+++ b/config/arm/arm64_n1sdp_linux_gcc
@@ -17,3 +17,4 @@ implementer_id = '0x41'
 part_number = '0xd0c'
 max_numa_nodes = 1
 max_lcore = 4
+has_libnuma = 0
diff --git a/config/arm/arm64_octeontx2_linux_gcc b/config/arm/arm64_octeontx2_linux_gcc
index 2ed9fd03d..56495d655 100644
--- a/config/arm/arm64_octeontx2_linux_gcc
+++ b/config/arm/arm64_octeontx2_linux_gcc
@@ -17,3 +17,4 @@ implementer_id = '0x43'
 part_number = '0xb2'
 max_numa_nodes = 1
 max_lcore = 36
+has_libnuma = 0
diff --git a/config/arm/arm64_stingray_linux_gcc b/config/arm/arm64_stingray_linux_gcc
index 40265ac34..e25067b7e 100644
--- a/config/arm/arm64_stingray_linux_gcc
+++ b/config/arm/arm64_stingray_linux_gcc
@@ -16,3 +16,4 @@ endian = 'little'
 implementer_id = '0x41'
 part_number = '0xd08'
 max_lcore = 16
+has_libnuma = 0
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 3a1e13f64..c1067dbb5 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -91,7 +91,6 @@ flags_thunderx2_extra = [
 flags_octeontx2_extra = [
 	['RTE_MACHINE', '"octeontx2"'],
 	['RTE_ARM_FEATURE_ATOMICS', true],
-	['RTE_EAL_IGB_UIO', false],
 	['RTE_USE_C11_MEM_MODEL', true]
 ]
 # arm config (implementer 0x41) is the default config
@@ -217,6 +216,10 @@ else
 		dpdk_flags += [
 			['RTE_MAX_LCORE', meson.get_cross_property('max_lcore', 16)]
 		]
+		cross_has_libnuma = meson.get_cross_property('has_libnuma', '')
+		if cross_has_libnuma != ''
+			has_libnuma = cross_has_libnuma
+		endif
 	endif
 
 	machine_args = [] # Clear previous machine args
diff --git a/drivers/meson.build b/drivers/meson.build
index f9250f3bb..b14d62fe4 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -20,6 +20,10 @@ subdirs = [
 disabled_drivers += run_command(list_dir_globs, get_option('disable_drivers'),
 		).stdout().split()
 
+if meson.is_cross_build()
+	disabled_drivers += meson.get_cross_property('disabled_drivers', [])
+endif
+
 default_cflags = machine_args
 default_cflags += ['-DALLOW_EXPERIMENTAL_API']
 default_cflags += ['-DALLOW_INTERNAL_API']
-- 
2.20.1

[dpdk-dev] [RFC PATCH v3 0/6] Arm build options rework

[Juraj Linkeš]

The current way of specifying Arm configuration options is insufficient
since we can't identify the SoC we're building for from the MIDR
information.


Add a way to discover cpu count a numa node count and use that instead
of statically defined values. This augments the current native builds.
A bonus is that users may now use -Dmax_lcores and -Dmax_numa_nodes to
configure those from command line, which didn't work before for Arm
builds.

For cross builds, the cpu count and numa node count are specified in
cross files.

We also want to be able to disable which drivers (and possibly
libraries) are built without user input. This is useful when building:
1. on an SoC that is slow and we want to build only what is necessary
  without the user having to check which libraries they have installed
2. a cross build on a fast aarch64 machine but with target SoC which
  differs in capabilities or libraries.
This is achieved by specifying the drivers in cross files. Only libnuma
can be now disabled.

v2:
Major rework of the whole series.

v3:
Added numa and core count defaults for x86 default build.
Removed numa and core count defaults. Now requiring defaults to be
specified in a cross file or on the cmdline.
Added FreeBDS support for numa count discovery.

Juraj Linkeš (6):
  build: rename default Arm build to generic-armv8
  build: refactor Arm build
  build: automatic NUMA and cpu counts detection
  build: move core and NUMA counts to cross files
  build: disable Arm drivers
  build: update Arm builds with makefile flags

 buildtools/get_cpu_count.py          |   7 +
 buildtools/get_numa_count.py         |  22 ++
 buildtools/meson.build               |   2 +
 config/arm/arm64_armada_linux_gcc    |   6 +-
 config/arm/arm64_armv8_linux_gcc     |  18 +-
 config/arm/arm64_bluefield_linux_gcc |   7 +-
 config/arm/arm64_dpaa_linux_gcc      |   5 +-
 config/arm/arm64_emag_linux_gcc      |   4 +-
 config/arm/arm64_n1sdp_linux_gcc     |   7 +-
 config/arm/arm64_octeontx2_linux_gcc |   7 +-
 config/arm/arm64_stingray_linux_gcc  |   7 +-
 config/arm/arm64_thunderx2_linux_gcc |   6 +-
 config/arm/arm64_thunderx_linux_gcc  |   4 +-
 config/arm/meson.build               | 297 +++++++++++++++------------
 config/meson.build                   |  45 +++-
 drivers/meson.build                  |   6 +-
 meson.build                          |   1 +
 meson_options.txt                    |   8 +-
 18 files changed, 302 insertions(+), 157 deletions(-)
 create mode 100644 buildtools/get_cpu_count.py
 create mode 100644 buildtools/get_numa_count.py

-- 
2.20.1

[dpdk-dev] [RFC PATCH v3 1/6] build: rename default Arm build to generic-armv8

[Juraj Linkeš]

The current machine='default' build name is not descriptive. The actual
default build is machine='native'. Add an alternative string which does
the same build and better describes what we're building:
machine='generic-armv8'. Leave machine='default' for backwards
compatibility.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/meson.build | 31 ++++++++++++++++---------------
 config/meson.build     |  3 ---
 2 files changed, 16 insertions(+), 18 deletions(-)

diff --git a/config/arm/meson.build b/config/arm/meson.build
index b49203fa8..c88e34a2b 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -1,12 +1,13 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation.
 # Copyright(c) 2017 Cavium, Inc
+# Copyright(c) 2020 PANTHEON.tech s.r.o.
 
 # for checking defines we need to use the correct compiler flags
 march_opt = '-march=@0@'.format(machine)
 
 arm_force_native_march = false
-arm_force_default_march = (machine == 'default')
+arm_force_generic_march = (machine == 'default' or machine == 'generic-armv8')
 
 flags_common_default = [
 	# Accelarate rte_memcpy. Be sure to run unit test (memcpy_perf_autotest)
@@ -87,7 +88,7 @@ flags_octeontx2_extra = [
 	['RTE_EAL_IGB_UIO', false],
 	['RTE_USE_C11_MEM_MODEL', true]]
 
-machine_args_generic = [
+machine_args_default = [
 	['default', ['-march=armv8-a+crc', '-moutline-atomics']],
 	['native', ['-march=native']],
 	['0xd03', ['-mcpu=cortex-a53']],
@@ -113,20 +114,20 @@ machine_args_emag = [
 	['native', ['-march=native']]]
 
 ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page G7-5321)
-impl_generic = ['Generic armv8', flags_generic, machine_args_generic]
-impl_0x41 = ['Arm', flags_arm, machine_args_generic]
-impl_0x42 = ['Broadcom', flags_generic, machine_args_generic]
+impl_generic = ['Generic armv8', flags_generic, machine_args_default]
+impl_0x41 = ['Arm', flags_arm, machine_args_default]
+impl_0x42 = ['Broadcom', flags_generic, machine_args_default]
 impl_0x43 = ['Cavium', flags_cavium, machine_args_cavium]
-impl_0x44 = ['DEC', flags_generic, machine_args_generic]
-impl_0x49 = ['Infineon', flags_generic, machine_args_generic]
-impl_0x4d = ['Motorola', flags_generic, machine_args_generic]
-impl_0x4e = ['NVIDIA', flags_generic, machine_args_generic]
+impl_0x44 = ['DEC', flags_generic, machine_args_default]
+impl_0x49 = ['Infineon', flags_generic, machine_args_default]
+impl_0x4d = ['Motorola', flags_generic, machine_args_default]
+impl_0x4e = ['NVIDIA', flags_generic, machine_args_default]
 impl_0x50 = ['Ampere Computing', flags_emag, machine_args_emag]
-impl_0x51 = ['Qualcomm', flags_generic, machine_args_generic]
-impl_0x53 = ['Samsung', flags_generic, machine_args_generic]
-impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_generic]
-impl_0x69 = ['Intel', flags_generic, machine_args_generic]
-impl_dpaa = ['NXP DPAA', flags_dpaa, machine_args_generic]
+impl_0x51 = ['Qualcomm', flags_generic, machine_args_default]
+impl_0x53 = ['Samsung', flags_generic, machine_args_default]
+impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_default]
+impl_0x69 = ['Intel', flags_generic, machine_args_default]
+impl_dpaa = ['NXP DPAA', flags_dpaa, machine_args_default]
 
 dpdk_conf.set('RTE_ARCH_ARM', 1)
 dpdk_conf.set('RTE_FORCE_INTRINSICS', 1)
@@ -145,7 +146,7 @@ else
 	cmd_generic = ['generic', '', '', 'default', '']
 	cmd_output = cmd_generic # Set generic by default
 	machine_args = [] # Clear previous machine args
-	if arm_force_default_march and not meson.is_cross_build()
+	if arm_force_generic_march and not meson.is_cross_build()
 		machine = impl_generic
 		impl_pn = 'default'
 	elif not meson.is_cross_build()
diff --git a/config/meson.build b/config/meson.build
index 258b01d06..a57c8ae9e 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -80,9 +80,6 @@ if machine == 'default'
 		machine = 'corei7'
 	elif host_machine.cpu_family().startswith('arm')
 		machine = 'armv7-a'
-	elif host_machine.cpu_family().startswith('aarch')
-		# arm64 manages defaults in config/arm/meson.build
-		machine = 'default'
 	elif host_machine.cpu_family().startswith('ppc')
 		machine = 'power8'
 	endif
-- 
2.20.1

Re: [dpdk-dev] [RFC PATCH v3 1/6] build: rename default Arm build to generic-armv8

[Bruce Richardson]

On Wed, Oct 21, 2020 at 01:37:36PM +0200, Juraj Linkeš wrote:
> The current machine='default' build name is not descriptive. The actual
> default build is machine='native'. Add an alternative string which does
> the same build and better describes what we're building:
> machine='generic-armv8'. Leave machine='default' for backwards
> compatibility.
> 
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> ---
> config/arm/meson.build | 31 ++++++++++++++++---------------
> config/meson.build     |  3 --- 2 files changed, 16 insertions(+), 18
> deletions(-)
> 
> diff --git a/config/arm/meson.build b/config/arm/meson.build index
> b49203fa8..c88e34a2b 100644 --- a/config/arm/meson.build +++
> b/config/arm/meson.build @@ -1,12 +1,13 @@ # SPDX-License-Identifier:
> BSD-3-Clause # Copyright(c) 2017 Intel Corporation.  # Copyright(c) 2017
> Cavium, Inc +# Copyright(c) 2020 PANTHEON.tech s.r.o.
>  
>  # for checking defines we need to use the correct compiler flags
>  march_opt = '-march=@0@'.format(machine)
>  
>  arm_force_native_march = false -arm_force_default_march = (machine ==
>  'default') +arm_force_generic_march = (machine == 'default' or machine
>  == 'generic-armv8')
>  

Rather than having this just for arm, how about adding "generic" as a
synonym for "default" at the top level, since it probably better describes
what "default" does for all architectures.

Re: [dpdk-dev] [RFC PATCH v3 1/6] build: rename default Arm build to generic-armv8

[Juraj Linkeš]

> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Wednesday, October 21, 2020 1:53 PM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Cc: Ruifeng.Wang@arm.com; Honnappa.Nagarahalli@arm.com;
> Phil.Yang@arm.com; vcchunga@amazon.com; Dharmik.Thakkar@arm.com;
> jerinjacobk@gmail.com; hemant.agrawal@nxp.com; dev@dpdk.org
> Subject: Re: [RFC PATCH v3 1/6] build: rename default Arm build to generic-
> armv8
> 
> On Wed, Oct 21, 2020 at 01:37:36PM +0200, Juraj Linkeš wrote:
> > The current machine='default' build name is not descriptive. The
> > actual default build is machine='native'. Add an alternative string
> > which does the same build and better describes what we're building:
> > machine='generic-armv8'. Leave machine='default' for backwards
> > compatibility.
> >
> > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> ---
> > config/arm/meson.build | 31 ++++++++++++++++---------------
> > config/meson.build     |  3 --- 2 files changed, 16 insertions(+), 18
> > deletions(-)
> >
> > diff --git a/config/arm/meson.build b/config/arm/meson.build index
> > b49203fa8..c88e34a2b 100644 --- a/config/arm/meson.build +++
> > b/config/arm/meson.build @@ -1,12 +1,13 @@ # SPDX-License-Identifier:
> > BSD-3-Clause # Copyright(c) 2017 Intel Corporation.  # Copyright(c)
> > 2017 Cavium, Inc +# Copyright(c) 2020 PANTHEON.tech s.r.o.
> >
> >  # for checking defines we need to use the correct compiler flags
> > march_opt = '-march=@0@'.format(machine)
> >
> >  arm_force_native_march = false -arm_force_default_march = (machine ==
> >  'default') +arm_force_generic_march = (machine == 'default' or
> > machine  == 'generic-armv8')
> >
> 
> Rather than having this just for arm, how about adding "generic" as a synonym
> for "default" at the top level, since it probably better describes what "default"
> does for all architectures.

Okay, makes sense. I'll do that in the next version.

[dpdk-dev] [RFC PATCH v3 2/6] build: refactor Arm build

[Juraj Linkeš]

* Rename variables to have names that better describe what the variables
store
* Remove unused or superfluous variables
* Change a list to dictionary where key lookup is needed
* Add informatory comments in the code
* Minor code restructure and reformatting

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/arm64_armada_linux_gcc    |   2 +-
 config/arm/arm64_armv8_linux_gcc     |   8 +-
 config/arm/arm64_bluefield_linux_gcc |   4 +-
 config/arm/arm64_dpaa_linux_gcc      |   2 +-
 config/arm/arm64_emag_linux_gcc      |   2 +-
 config/arm/arm64_n1sdp_linux_gcc     |   4 +-
 config/arm/arm64_octeontx2_linux_gcc |   4 +-
 config/arm/arm64_stingray_linux_gcc  |   4 +-
 config/arm/arm64_thunderx2_linux_gcc |   4 +-
 config/arm/arm64_thunderx_linux_gcc  |   2 +-
 config/arm/meson.build               | 247 +++++++++++++++------------
 11 files changed, 153 insertions(+), 130 deletions(-)

diff --git a/config/arm/arm64_armada_linux_gcc b/config/arm/arm64_armada_linux_gcc
index fa40c0398..52c5f4476 100644
--- a/config/arm/arm64_armada_linux_gcc
+++ b/config/arm/arm64_armada_linux_gcc
@@ -14,4 +14,4 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x56'
+implementer_id = '0x56'
diff --git a/config/arm/arm64_armv8_linux_gcc b/config/arm/arm64_armv8_linux_gcc
index 88f0ff9da..13ee8b223 100644
--- a/config/arm/arm64_armv8_linux_gcc
+++ b/config/arm/arm64_armv8_linux_gcc
@@ -13,10 +13,10 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = 'generic'
+implementer_id = 'generic'
 
-# Valid options for Arm's implementor_pn:
-# 'default': valid for all armv8-a architectures (default value)
+# Valid options for Arm's part_number:
+# 'generic': valid for all armv8-a architectures (default value)
 # '0xd03':   cortex-a53
 # '0xd04':   cortex-a35
 # '0xd05':   cortex-a55
@@ -25,4 +25,4 @@ implementor_id = 'generic'
 # '0xd09':   cortex-a73
 # '0xd0a':   cortex-a75
 # '0xd0b':   cortex-a76
-implementor_pn = 'default'
+part_number = 'generic'
diff --git a/config/arm/arm64_bluefield_linux_gcc b/config/arm/arm64_bluefield_linux_gcc
index 86797d23c..b79389d85 100644
--- a/config/arm/arm64_bluefield_linux_gcc
+++ b/config/arm/arm64_bluefield_linux_gcc
@@ -13,5 +13,5 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x41'
-implementor_pn = '0xd08'
+implementer_id = '0x41'
+part_number = '0xd08'
diff --git a/config/arm/arm64_dpaa_linux_gcc b/config/arm/arm64_dpaa_linux_gcc
index 1a4682154..573ae7e42 100644
--- a/config/arm/arm64_dpaa_linux_gcc
+++ b/config/arm/arm64_dpaa_linux_gcc
@@ -14,4 +14,4 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = 'dpaa'
+implementer_id = 'dpaa'
diff --git a/config/arm/arm64_emag_linux_gcc b/config/arm/arm64_emag_linux_gcc
index 8edcd3e97..24f3d533e 100644
--- a/config/arm/arm64_emag_linux_gcc
+++ b/config/arm/arm64_emag_linux_gcc
@@ -13,4 +13,4 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x50'
+implementer_id = '0x50'
diff --git a/config/arm/arm64_n1sdp_linux_gcc b/config/arm/arm64_n1sdp_linux_gcc
index 022e06303..6fb3f02ea 100644
--- a/config/arm/arm64_n1sdp_linux_gcc
+++ b/config/arm/arm64_n1sdp_linux_gcc
@@ -13,5 +13,5 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x41'
-implementor_pn = '0xd0c'
+implementer_id = '0x41'
+part_number = '0xd0c'
diff --git a/config/arm/arm64_octeontx2_linux_gcc b/config/arm/arm64_octeontx2_linux_gcc
index 365bd7cbd..ac1042806 100644
--- a/config/arm/arm64_octeontx2_linux_gcc
+++ b/config/arm/arm64_octeontx2_linux_gcc
@@ -13,5 +13,5 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x43'
-implementor_pn = '0xb2'
+implementer_id = '0x43'
+part_number = '0xb2'
diff --git a/config/arm/arm64_stingray_linux_gcc b/config/arm/arm64_stingray_linux_gcc
index 86797d23c..b79389d85 100644
--- a/config/arm/arm64_stingray_linux_gcc
+++ b/config/arm/arm64_stingray_linux_gcc
@@ -13,5 +13,5 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x41'
-implementor_pn = '0xd08'
+implementer_id = '0x41'
+part_number = '0xd08'
diff --git a/config/arm/arm64_thunderx2_linux_gcc b/config/arm/arm64_thunderx2_linux_gcc
index 2b41acc61..dd257745e 100644
--- a/config/arm/arm64_thunderx2_linux_gcc
+++ b/config/arm/arm64_thunderx2_linux_gcc
@@ -13,5 +13,5 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x43'
-implementor_pn = '0xaf'
+implementer_id = '0x43'
+part_number = '0xaf'
diff --git a/config/arm/arm64_thunderx_linux_gcc b/config/arm/arm64_thunderx_linux_gcc
index 6572ab615..670764437 100644
--- a/config/arm/arm64_thunderx_linux_gcc
+++ b/config/arm/arm64_thunderx_linux_gcc
@@ -13,4 +13,4 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x43'
+implementer_id = '0x43'
diff --git a/config/arm/meson.build b/config/arm/meson.build
index c88e34a2b..66b08bfb5 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -3,12 +3,12 @@
 # Copyright(c) 2017 Cavium, Inc
 # Copyright(c) 2020 PANTHEON.tech s.r.o.
 
-# for checking defines we need to use the correct compiler flags
-march_opt = '-march=@0@'.format(machine)
-
+# set arm_force_native_march if you want to use machine args below
+# instead of discovered values; only works when doing an actual native build
 arm_force_native_march = false
-arm_force_generic_march = (machine == 'default' or machine == 'generic-armv8')
+native_machine_args = ['-march=native', '-mtune=native']
 
+# common flags to all aarch64 builds, with lowest priority
 flags_common_default = [
 	# Accelarate rte_memcpy. Be sure to run unit test (memcpy_perf_autotest)
 	# to determine the best threshold in code. Refer to notes in source file
@@ -16,8 +16,9 @@ flags_common_default = [
 	['RTE_ARCH_ARM64_MEMCPY', false],
 	#	['RTE_ARM64_MEMCPY_ALIGNED_THRESHOLD', 2048],
 	#	['RTE_ARM64_MEMCPY_UNALIGNED_THRESHOLD', 512],
-	# Leave below RTE_ARM64_MEMCPY_xxx options commented out, unless there're
-	# strong reasons.
+
+	# Leave below RTE_ARM64_MEMCPY_xxx options commented out,
+	# unless there are strong reasons.
 	#	['RTE_ARM64_MEMCPY_SKIP_GCC_VER_CHECK', false],
 	#	['RTE_ARM64_MEMCPY_ALIGN_MASK', 0xF],
 	#	['RTE_ARM64_MEMCPY_STRICT_ALIGN', false],
@@ -28,184 +29,206 @@ flags_common_default = [
 
 	['RTE_SCHED_VECTOR', false],
 	['RTE_ARM_USE_WFE', false],
+	['RTE_CACHE_LINE_SIZE', 128],
+	['RTE_ARCH_ARM64', true]
 ]
 
+# implementer specific aarch64 flags, with middle priority
+# (will overwrite common flags)
 flags_generic = [
 	['RTE_MACHINE', '"armv8a"'],
 	['RTE_MAX_LCORE', 256],
 	['RTE_USE_C11_MEM_MODEL', true],
-	['RTE_CACHE_LINE_SIZE', 128]]
+	['RTE_CACHE_LINE_SIZE', 128]
+]
 flags_arm = [
 	['RTE_MACHINE', '"armv8a"'],
 	['RTE_MAX_LCORE', 16],
 	['RTE_USE_C11_MEM_MODEL', true],
-	['RTE_CACHE_LINE_SIZE', 64]]
+	['RTE_CACHE_LINE_SIZE', 64]
+]
 flags_cavium = [
 	['RTE_CACHE_LINE_SIZE', 128],
 	['RTE_MAX_NUMA_NODES', 2],
 	['RTE_MAX_LCORE', 96],
-	['RTE_MAX_VFIO_GROUPS', 128]]
+	['RTE_MAX_VFIO_GROUPS', 128]
+]
 flags_dpaa = [
 	['RTE_MACHINE', '"dpaa"'],
 	['RTE_USE_C11_MEM_MODEL', true],
 	['RTE_CACHE_LINE_SIZE', 64],
 	['RTE_MAX_NUMA_NODES', 1],
 	['RTE_MAX_LCORE', 16],
-	['RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', false]]
+	['RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', false]
+]
 flags_emag = [
 	['RTE_MACHINE', '"emag"'],
-	['RTE_CACHE_LINE_SIZE', 64],
 	['RTE_MAX_NUMA_NODES', 1],
-	['RTE_MAX_LCORE', 32]]
+	['RTE_MAX_LCORE', 32],
+	['RTE_CACHE_LINE_SIZE', 64]
+]
 flags_armada = [
 	['RTE_MACHINE', '"armv8a"'],
-	['RTE_CACHE_LINE_SIZE', 64],
 	['RTE_MAX_NUMA_NODES', 1],
-	['RTE_MAX_LCORE', 16]]
+	['RTE_MAX_LCORE', 16],
+	['RTE_CACHE_LINE_SIZE', 64]
+]
 
-flags_default_extra = []
+# part number specific aarch64 flags, with highest priority
+# (will overwrite both common and implementer specific flags)
 flags_n1sdp_extra = [
 	['RTE_MACHINE', '"n1sdp"'],
 	['RTE_MAX_NUMA_NODES', 1],
 	['RTE_MAX_LCORE', 4],
 	['RTE_EAL_NUMA_AWARE_HUGEPAGES', false],
-	['RTE_LIBRTE_VHOST_NUMA', false]]
+	['RTE_LIBRTE_VHOST_NUMA', false]
+]
 flags_thunderx_extra = [
 	['RTE_MACHINE', '"thunderx"'],
-	['RTE_USE_C11_MEM_MODEL', false]]
+	['RTE_USE_C11_MEM_MODEL', false]
+]
 flags_thunderx2_extra = [
 	['RTE_MACHINE', '"thunderx2"'],
 	['RTE_CACHE_LINE_SIZE', 64],
 	['RTE_MAX_NUMA_NODES', 2],
 	['RTE_MAX_LCORE', 256],
 	['RTE_ARM_FEATURE_ATOMICS', true],
-	['RTE_USE_C11_MEM_MODEL', true]]
+	['RTE_USE_C11_MEM_MODEL', true]
+]
 flags_octeontx2_extra = [
 	['RTE_MACHINE', '"octeontx2"'],
 	['RTE_MAX_NUMA_NODES', 1],
 	['RTE_MAX_LCORE', 36],
 	['RTE_ARM_FEATURE_ATOMICS', true],
 	['RTE_EAL_IGB_UIO', false],
-	['RTE_USE_C11_MEM_MODEL', true]]
-
-machine_args_default = [
-	['default', ['-march=armv8-a+crc', '-moutline-atomics']],
-	['native', ['-march=native']],
-	['0xd03', ['-mcpu=cortex-a53']],
-	['0xd04', ['-mcpu=cortex-a35']],
-	['0xd07', ['-mcpu=cortex-a57']],
-	['0xd08', ['-mcpu=cortex-a72']],
-	['0xd09', ['-mcpu=cortex-a73']],
-	['0xd0a', ['-mcpu=cortex-a75']],
-	['0xd0b', ['-mcpu=cortex-a76']],
-	['0xd0c', ['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'], flags_n1sdp_extra]]
-
-machine_args_cavium = [
-	['default', ['-march=armv8-a+crc+crypto','-mcpu=thunderx']],
-	['native', ['-march=native']],
-	['0xa1', ['-mcpu=thunderxt88'], flags_thunderx_extra],
-	['0xa2', ['-mcpu=thunderxt81'], flags_thunderx_extra],
-	['0xa3', ['-mcpu=thunderxt83'], flags_thunderx_extra],
-	['0xaf', ['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'], flags_thunderx2_extra],
-	['0xb2', ['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'], flags_octeontx2_extra]]
-
-machine_args_emag = [
-	['default', ['-march=armv8-a+crc+crypto', '-mtune=emag']],
-	['native', ['-march=native']]]
+	['RTE_USE_C11_MEM_MODEL', true]
+]
+# arm config (implementer 0x41) is the default config
+pn_config_default = {
+	'generic': [['-march=armv8-a+crc', '-moutline-atomics']],
+	'0xd03': [['-mcpu=cortex-a53']],
+	'0xd04': [['-mcpu=cortex-a35']],
+	'0xd07': [['-mcpu=cortex-a57']],
+	'0xd08': [['-mcpu=cortex-a72']],
+	'0xd09': [['-mcpu=cortex-a73']],
+	'0xd0a': [['-mcpu=cortex-a75']],
+	'0xd0b': [['-mcpu=cortex-a76']],
+	'0xd0c': [['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'], flags_n1sdp_extra]
+}
+pn_config_cavium = {
+	'generic': [['-march=armv8-a+crc+crypto', '-mcpu=thunderx']],
+	'0xa1': [['-mcpu=thunderxt88'], flags_thunderx_extra],
+	'0xa2': [['-mcpu=thunderxt81'], flags_thunderx_extra],
+	'0xa3': [['-mcpu=thunderxt83'], flags_thunderx_extra],
+	'0xaf': [['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'], flags_thunderx2_extra],
+	'0xb2': [['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'], flags_octeontx2_extra],
+}
+pn_config_emag = {
+	'generic': [['-march=armv8-a+crc+crypto', '-mtune=emag']]
+}
 
 ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page G7-5321)
-impl_generic = ['Generic armv8', flags_generic, machine_args_default]
-impl_0x41 = ['Arm', flags_arm, machine_args_default]
-impl_0x42 = ['Broadcom', flags_generic, machine_args_default]
-impl_0x43 = ['Cavium', flags_cavium, machine_args_cavium]
-impl_0x44 = ['DEC', flags_generic, machine_args_default]
-impl_0x49 = ['Infineon', flags_generic, machine_args_default]
-impl_0x4d = ['Motorola', flags_generic, machine_args_default]
-impl_0x4e = ['NVIDIA', flags_generic, machine_args_default]
-impl_0x50 = ['Ampere Computing', flags_emag, machine_args_emag]
-impl_0x51 = ['Qualcomm', flags_generic, machine_args_default]
-impl_0x53 = ['Samsung', flags_generic, machine_args_default]
-impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_default]
-impl_0x69 = ['Intel', flags_generic, machine_args_default]
-impl_dpaa = ['NXP DPAA', flags_dpaa, machine_args_default]
+impl_generic = ['Generic armv8', flags_generic, pn_config_default]
+impl_0x41 = ['Arm', flags_arm, pn_config_default]
+impl_0x42 = ['Broadcom', flags_generic, pn_config_default]
+impl_0x43 = ['Cavium', flags_cavium, pn_config_cavium]
+impl_0x44 = ['DEC', flags_generic, pn_config_default]
+impl_0x49 = ['Infineon', flags_generic, pn_config_default]
+impl_0x4d = ['Motorola', flags_generic, pn_config_default]
+impl_0x4e = ['NVIDIA', flags_generic, pn_config_default]
+impl_0x50 = ['Ampere Computing', flags_emag, pn_config_emag]
+impl_0x51 = ['Qualcomm', flags_generic, pn_config_default]
+impl_0x53 = ['Samsung', flags_generic, pn_config_default]
+impl_0x56 = ['Marvell ARMADA', flags_armada, pn_config_default]
+impl_0x69 = ['Intel', flags_generic, pn_config_default]
+impl_dpaa = ['NXP DPAA', flags_dpaa, pn_config_default]
 
 dpdk_conf.set('RTE_ARCH_ARM', 1)
 dpdk_conf.set('RTE_FORCE_INTRINSICS', 1)
 
 if dpdk_conf.get('RTE_ARCH_32')
+	# armv7 build
 	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
 	dpdk_conf.set('RTE_ARCH_ARMv7', 1)
 	# the minimum architecture supported, armv7-a, needs the following,
 	# mk/machine/armv7a/rte.vars.mk sets it too
 	machine_args += '-mfpu=neon'
 else
-	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
-	dpdk_conf.set('RTE_ARCH_ARM64', 1)
+	# aarch64 build
+	if not meson.is_cross_build()
+		if machine == 'default' or machine == 'generic-armv8'
+			# default build
+			impl_config = impl_generic
+			part_number = 'generic'
+		else
+			# native build
+			# The script returns ['Implementer', 'Variant', 'Architecture',
+			# 'Primary Part number', 'Revision']
+			detect_vendor = find_program(join_paths(
+					meson.current_source_dir(), 'armv8_machine.py'))
+			cmd = run_command(detect_vendor.path())
+			if cmd.returncode() == 0
+				cmd_output = cmd.stdout().to_lower().strip().split(' ')
+			endif
+			if arm_force_native_march == true
+				part_number = 'native'
+			else
+				part_number = cmd_output[3]
+			endif
+			# Set to generic implementer if implementer is not found
+			impl_config = get_variable('impl_' + cmd_output[0], 'impl_generic')
+		endif
+	else
+		# cross build
+		impl_id = meson.get_cross_property('implementer_id', 'generic')
+		part_number = meson.get_cross_property('part_number', 'generic')
+		impl_config = get_variable('impl_' + impl_id)
+	endif
 
-	machine = []
-	cmd_generic = ['generic', '', '', 'default', '']
-	cmd_output = cmd_generic # Set generic by default
-	machine_args = [] # Clear previous machine args
-	if arm_force_generic_march and not meson.is_cross_build()
-		machine = impl_generic
-		impl_pn = 'default'
+	message('Arm implementer: ' + impl_config[0])
+	message('Arm part number: ' + part_number)
+
+	implementer_flags = impl_config[1]
+	part_number_config = impl_config[2]
+
+	if part_number_config.has_key(part_number)
+		# use the specified part_number machine args if found
+		part_number_config = part_number_config[part_number]
+	elif part_number == 'native'
+		# use native machine args
+		part_number_config = [[native_machine_args]]
 	elif not meson.is_cross_build()
-		# The script returns ['Implementer', 'Variant', 'Architecture',
-		# 'Primary Part number', 'Revision']
-		detect_vendor = find_program(join_paths(
-				meson.current_source_dir(), 'armv8_machine.py'))
-		cmd = run_command(detect_vendor.path())
-		if cmd.returncode() == 0
-			cmd_output = cmd.stdout().to_lower().strip().split(' ')
-		endif
-		# Set to generic if variable is not found
-		machine = get_variable('impl_' + cmd_output[0], ['generic'])
-		if machine[0] == 'generic'
-			machine = impl_generic
-			cmd_output = cmd_generic
-		endif
-		impl_pn = cmd_output[3]
-		if arm_force_native_march == true
-			impl_pn = 'native'
-		endif
+		# default to generic machine args if part_number is not found
+		# and not forcing native machine args
+		# but don't default in cross-builds; if part_number is specified
+		# incorrectly in a cross-file, it needs to be fixed there
+		part_number_config = part_number_config['generic']
 	else
-		impl_id = meson.get_cross_property('implementor_id', 'generic')
-		impl_pn = meson.get_cross_property('implementor_pn', 'default')
-		machine = get_variable('impl_' + impl_id)
+		# cross build and part number is not in part_number_config
+		error('Cross build part number 0@0 not found.'.format(part_number))
 	endif
 
-	# Apply Common Defaults. These settings may be overwritten by machine
-	# settings later.
-	foreach flag: flags_common_default
-		if flag.length() > 0
-			dpdk_conf.set(flag[0], flag[1])
+	dpdk_flags = flags_common_default + implementer_flags
+
+	if part_number_config.length() > 1
+		dpdk_flags += part_number_config[1]
+	endif
+
+	machine_args = [] # Clear previous machine args
+	foreach flag: part_number_config[0]
+		if cc.has_argument(flag)
+			machine_args += flag
 		endif
 	endforeach
 
-	message('Implementer : ' + machine[0])
-	foreach flag: machine[1]
+	foreach flag: dpdk_flags
 		if flag.length() > 0
 			dpdk_conf.set(flag[0], flag[1])
 		endif
 	endforeach
-
-	foreach marg: machine[2]
-		if marg[0] == impl_pn
-			foreach flag: marg[1]
-				if cc.has_argument(flag)
-					machine_args += flag
-				endif
-			endforeach
-			# Apply any extra machine specific flags.
-			foreach flag: marg.get(2, flags_default_extra)
-				if flag.length() > 0
-					dpdk_conf.set(flag[0], flag[1])
-				endif
-			endforeach
-		endif
-	endforeach
 endif
-message(machine_args)
+
+message('Using machine args: @0@'.format(machine_args))
 
 if (cc.get_define('__ARM_NEON', args: machine_args) != '' or
     cc.get_define('__aarch64__', args: machine_args) != '')
-- 
2.20.1

[dpdk-dev] [RFC PATCH v3 3/6] build: automatic NUMA and cpu counts detection

[Juraj Linkeš]

The build machine's number of cpus and numa nodes vary, resulting in
mismatched counts of RTE_MAX_LCORE and RTE_MAX_NUMA_NODES for many
builds. Automatically discover the host's numa and cpu counts to remove
this mismatch for native builds. Use current defaults for default builds.
Force the users to specify the counts for cross build in cross files or
on the command line.
Give users the option to override the discovery or values from cross
files by specifying them on the command line with -Dmax_lcores and
-Dmax_numa_nodes.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 buildtools/get_cpu_count.py  |  7 ++++++
 buildtools/get_numa_count.py | 22 +++++++++++++++++++
 buildtools/meson.build       |  2 ++
 config/meson.build           | 42 ++++++++++++++++++++++++++++++++++--
 meson_options.txt            |  8 +++----
 5 files changed, 75 insertions(+), 6 deletions(-)
 create mode 100644 buildtools/get_cpu_count.py
 create mode 100644 buildtools/get_numa_count.py

diff --git a/buildtools/get_cpu_count.py b/buildtools/get_cpu_count.py
new file mode 100644
index 000000000..386f85f8b
--- /dev/null
+++ b/buildtools/get_cpu_count.py
@@ -0,0 +1,7 @@
+#!/usr/bin/python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2020 PANTHEON.tech s.r.o.
+
+import os
+
+print(os.cpu_count())
diff --git a/buildtools/get_numa_count.py b/buildtools/get_numa_count.py
new file mode 100644
index 000000000..f0c49973a
--- /dev/null
+++ b/buildtools/get_numa_count.py
@@ -0,0 +1,22 @@
+#!/usr/bin/python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2020 PANTHEON.tech s.r.o.
+
+import ctypes
+import glob
+import os
+import subprocess
+
+if os.name == 'posix':
+    if os.path.isdir('/sys/devices/system/node'):
+        print(len(glob.glob('/sys/devices/system/node/node*')))
+    else:
+        print(subprocess.run(['sysctl', 'vm.ndomains'], capture_output=True).stdout)
+
+elif os.name == 'nt':
+    libkernel32 = ctypes.windll.kernel32
+
+    count = ctypes.c_ulong()
+
+    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
+    print(count.value + 1)
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 04808dabc..925e733b1 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -17,3 +17,5 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+get_cpu_count_cmd = py3 + files('get_cpu_count.py')
+get_numa_count_cmd = py3 + files('get_numa_count.py')
diff --git a/config/meson.build b/config/meson.build
index a57c8ae9e..c4477f977 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -74,7 +74,11 @@ endif
 # still being able to support the CPU features required for DPDK.
 # This can be bumped up by the DPDK project, but it can never be an
 # invariant like 'native'
+max_lcores = get_option('max_lcores')
+max_numa_nodes = get_option('max_numa_nodes')
 if machine == 'default'
+	max_numa_nodes = 4
+	max_lcores = 128
 	if host_machine.cpu_family().startswith('x86')
 		# matches the old pre-meson build systems default
 		machine = 'corei7'
@@ -83,6 +87,22 @@ if machine == 'default'
 	elif host_machine.cpu_family().startswith('ppc')
 		machine = 'power8'
 	endif
+elif not meson.is_cross_build()
+	# find host core count and numa node count for native builds
+	if max_lcores == 0
+		max_lcores = run_command(get_cpu_count_cmd).stdout().to_int()
+		min_lcores = 2
+		if max_lcores < min_lcores
+			message('Found less than @0@ cores, building for @0@ cores'.format(min_lcores))
+			max_lcores = min_lcores
+		else
+			message('Found @0@ cores'.format(max_lcores))
+		endif
+	endif
+	if max_numa_nodes == 0
+		max_numa_nodes = run_command(get_numa_count_cmd).stdout().to_int()
+		message('Found @0@ numa nodes'.format(max_numa_nodes))
+	endif
 endif
 
 dpdk_conf.set('RTE_MACHINE', machine)
@@ -227,8 +247,10 @@ foreach arg: warning_flags
 endforeach
 
 # set other values pulled from the build options
-dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
-dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
+if not meson.is_cross_build()
+	dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
+	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
+endif
 dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
 dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
 dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
@@ -247,6 +269,22 @@ compile_time_cpuflags = []
 subdir(arch_subdir)
 dpdk_conf.set('RTE_COMPILE_TIME_CPUFLAGS', ','.join(compile_time_cpuflags))
 
+# check that cpu and numa count is set in cross builds
+if meson.is_cross_build()
+    	if max_lcores > 0
+		# specified on the cmdline
+		dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
+	elif not dpdk_conf.has('RTE_MAX_LCORE')
+		error('Number of cores for cross build not specified in @0@ subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
+	endif
+	if max_numa_nodes > 0
+		# specified on the cmdline
+		dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
+	elif not dpdk_conf.has('RTE_MAX_NUMA_NODES')
+		error('Number of numa nodes for cross build not specified in @0@ subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
+	endif
+endif
+
 # set the install path for the drivers
 dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
 
diff --git a/meson_options.txt b/meson_options.txt
index 9bf18ab6b..01b0c45c3 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -26,10 +26,10 @@ option('machine', type: 'string', value: 'native',
 	description: 'set the target machine type')
 option('max_ethports', type: 'integer', value: 32,
 	description: 'maximum number of Ethernet devices')
-option('max_lcores', type: 'integer', value: 128,
-	description: 'maximum number of cores/threads supported by EAL')
-option('max_numa_nodes', type: 'integer', value: 4,
-	description: 'maximum number of NUMA nodes supported by EAL')
+option('max_lcores', type: 'integer', value: 0,
+	description: 'maximum number of cores/threads supported by EAL. Value 0 means the number of cpus on the host will be used. For cross build, set to non-zero to overwrite the cross-file value.')
+option('max_numa_nodes', type: 'integer', value: 0,
+	description: 'maximum number of NUMA nodes supported by EAL. Value 0 means the number of numa nodes on the host will be used. For cross build, set to non-zero to overwrite the cross-file value.')
 option('enable_trace_fp', type: 'boolean', value: false,
 	description: 'enable fast path trace points.')
 option('tests', type: 'boolean', value: true,
-- 
2.20.1

Re: [dpdk-dev] [RFC PATCH v3 3/6] build: automatic NUMA and cpu counts detection

[Bruce Richardson]

On Wed, Oct 21, 2020 at 01:37:38PM +0200, Juraj Linkeš wrote:
> The build machine's number of cpus and numa nodes vary, resulting in
> mismatched counts of RTE_MAX_LCORE and RTE_MAX_NUMA_NODES for many
> builds. Automatically discover the host's numa and cpu counts to remove
> this mismatch for native builds. Use current defaults for default builds.
> Force the users to specify the counts for cross build in cross files or
> on the command line.
> Give users the option to override the discovery or values from cross
> files by specifying them on the command line with -Dmax_lcores and
> -Dmax_numa_nodes.
> 
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
>  buildtools/get_cpu_count.py  |  7 ++++++
>  buildtools/get_numa_count.py | 22 +++++++++++++++++++
>  buildtools/meson.build       |  2 ++
>  config/meson.build           | 42 ++++++++++++++++++++++++++++++++++--
>  meson_options.txt            |  8 +++----
>  5 files changed, 75 insertions(+), 6 deletions(-)
>  create mode 100644 buildtools/get_cpu_count.py
>  create mode 100644 buildtools/get_numa_count.py
> 
> diff --git a/buildtools/get_cpu_count.py b/buildtools/get_cpu_count.py
> new file mode 100644
> index 000000000..386f85f8b
> --- /dev/null
> +++ b/buildtools/get_cpu_count.py
> @@ -0,0 +1,7 @@
> +#!/usr/bin/python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2020 PANTHEON.tech s.r.o.
> +
> +import os
> +
> +print(os.cpu_count())
> diff --git a/buildtools/get_numa_count.py b/buildtools/get_numa_count.py
> new file mode 100644
> index 000000000..f0c49973a
> --- /dev/null
> +++ b/buildtools/get_numa_count.py
> @@ -0,0 +1,22 @@
> +#!/usr/bin/python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2020 PANTHEON.tech s.r.o.
> +
> +import ctypes
> +import glob
> +import os
> +import subprocess
> +
> +if os.name == 'posix':
> +    if os.path.isdir('/sys/devices/system/node'):
> +        print(len(glob.glob('/sys/devices/system/node/node*')))
> +    else:
> +        print(subprocess.run(['sysctl', 'vm.ndomains'], capture_output=True).stdout)
> +
> +elif os.name == 'nt':
> +    libkernel32 = ctypes.windll.kernel32
> +
> +    count = ctypes.c_ulong()
> +
> +    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
> +    print(count.value + 1)
> diff --git a/buildtools/meson.build b/buildtools/meson.build
> index 04808dabc..925e733b1 100644
> --- a/buildtools/meson.build
> +++ b/buildtools/meson.build
> @@ -17,3 +17,5 @@ else
>  endif
>  map_to_win_cmd = py3 + files('map_to_win.py')
>  sphinx_wrapper = py3 + files('call-sphinx-build.py')
> +get_cpu_count_cmd = py3 + files('get_cpu_count.py')
> +get_numa_count_cmd = py3 + files('get_numa_count.py')
> diff --git a/config/meson.build b/config/meson.build
> index a57c8ae9e..c4477f977 100644
> --- a/config/meson.build
> +++ b/config/meson.build
> @@ -74,7 +74,11 @@ endif
>  # still being able to support the CPU features required for DPDK.
>  # This can be bumped up by the DPDK project, but it can never be an
>  # invariant like 'native'
> +max_lcores = get_option('max_lcores')
> +max_numa_nodes = get_option('max_numa_nodes')
>  if machine == 'default'
> +	max_numa_nodes = 4
> +	max_lcores = 128

This doesn't seem right, since you are overriding the user-specified values
with hard-coded ones.

>  	if host_machine.cpu_family().startswith('x86')
>  		# matches the old pre-meson build systems default
>  		machine = 'corei7'
> @@ -83,6 +87,22 @@ if machine == 'default'
>  	elif host_machine.cpu_family().startswith('ppc')
>  		machine = 'power8'
>  	endif
> +elif not meson.is_cross_build()
> +	# find host core count and numa node count for native builds
> +	if max_lcores == 0
> +		max_lcores = run_command(get_cpu_count_cmd).stdout().to_int()
> +		min_lcores = 2
> +		if max_lcores < min_lcores
> +			message('Found less than @0@ cores, building for @0@ cores'.format(min_lcores))
> +			max_lcores = min_lcores
> +		else
> +			message('Found @0@ cores'.format(max_lcores))
> +		endif
> +	endif
> +	if max_numa_nodes == 0
> +		max_numa_nodes = run_command(get_numa_count_cmd).stdout().to_int()
> +		message('Found @0@ numa nodes'.format(max_numa_nodes))
> +	endif
>  endif
>  
>  dpdk_conf.set('RTE_MACHINE', machine)
> @@ -227,8 +247,10 @@ foreach arg: warning_flags
>  endforeach
>  
>  # set other values pulled from the build options
> -dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
> -dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
> +if not meson.is_cross_build()
> +	dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> +	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
> +endif

Rather than conditionally setting the value here, you should move the
checks below up above this to simplify things.

>  dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
>  dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
>  dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
> @@ -247,6 +269,22 @@ compile_time_cpuflags = []
>  subdir(arch_subdir)
>  dpdk_conf.set('RTE_COMPILE_TIME_CPUFLAGS', ','.join(compile_time_cpuflags))
>  
> +# check that cpu and numa count is set in cross builds
> +if meson.is_cross_build()
> +    	if max_lcores > 0
> +		# specified on the cmdline
> +		dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> +	elif not dpdk_conf.has('RTE_MAX_LCORE')
> +		error('Number of cores for cross build not specified in @0@ subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
> +	endif
> +	if max_numa_nodes > 0
> +		# specified on the cmdline
> +		dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
> +	elif not dpdk_conf.has('RTE_MAX_NUMA_NODES')
> +		error('Number of numa nodes for cross build not specified in @0@ subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
> +	endif
> +endif
> +
>  # set the install path for the drivers
>  dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
>  
> diff --git a/meson_options.txt b/meson_options.txt
> index 9bf18ab6b..01b0c45c3 100644
> --- a/meson_options.txt
> +++ b/meson_options.txt
> @@ -26,10 +26,10 @@ option('machine', type: 'string', value: 'native',
>  	description: 'set the target machine type')
>  option('max_ethports', type: 'integer', value: 32,
>  	description: 'maximum number of Ethernet devices')
> -option('max_lcores', type: 'integer', value: 128,
> -	description: 'maximum number of cores/threads supported by EAL')
> -option('max_numa_nodes', type: 'integer', value: 4,
> -	description: 'maximum number of NUMA nodes supported by EAL')
> +option('max_lcores', type: 'integer', value: 0,
> +	description: 'maximum number of cores/threads supported by EAL. Value 0 means the number of cpus on the host will be used. For cross build, set to non-zero to overwrite the cross-file value.')
> +option('max_numa_nodes', type: 'integer', value: 0,
> +	description: 'maximum number of NUMA nodes supported by EAL. Value 0 means the number of numa nodes on the host will be used. For cross build, set to non-zero to overwrite the cross-file value.')

I don't like this change, because it very much assumes for
non-cross-compiles that people will be running DPDK on the system they
build it on. That's a very, very big assumption! I'm ok with having zero as
a "detect" option, and having the values overridden from cross-files, but
not with detection as the default out-of-the-box option! Lots of users may
pull builds from a CI based on VMs with just a few cores, for instance.

>  option('enable_trace_fp', type: 'boolean', value: false,
>  	description: 'enable fast path trace points.')
>  option('tests', type: 'boolean', value: true,
> -- 
> 2.20.1
> 

Re: [dpdk-dev] [RFC PATCH v3 3/6] build: automatic NUMA and cpu counts detection

[Juraj Linkeš]

> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Wednesday, October 21, 2020 2:02 PM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Cc: Ruifeng.Wang@arm.com; Honnappa.Nagarahalli@arm.com;
> Phil.Yang@arm.com; vcchunga@amazon.com; Dharmik.Thakkar@arm.com;
> jerinjacobk@gmail.com; hemant.agrawal@nxp.com; dev@dpdk.org
> Subject: Re: [RFC PATCH v3 3/6] build: automatic NUMA and cpu counts
> detection
> 
> On Wed, Oct 21, 2020 at 01:37:38PM +0200, Juraj Linkeš wrote:
> > The build machine's number of cpus and numa nodes vary, resulting in
> > mismatched counts of RTE_MAX_LCORE and RTE_MAX_NUMA_NODES for
> many
> > builds. Automatically discover the host's numa and cpu counts to
> > remove this mismatch for native builds. Use current defaults for default builds.
> > Force the users to specify the counts for cross build in cross files
> > or on the command line.
> > Give users the option to override the discovery or values from cross
> > files by specifying them on the command line with -Dmax_lcores and
> > -Dmax_numa_nodes.
> >
> > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > ---
> >  buildtools/get_cpu_count.py  |  7 ++++++
> > buildtools/get_numa_count.py | 22 +++++++++++++++++++
> >  buildtools/meson.build       |  2 ++
> >  config/meson.build           | 42 ++++++++++++++++++++++++++++++++++--
> >  meson_options.txt            |  8 +++----
> >  5 files changed, 75 insertions(+), 6 deletions(-)  create mode 100644
> > buildtools/get_cpu_count.py  create mode 100644
> > buildtools/get_numa_count.py
> >
> > diff --git a/buildtools/get_cpu_count.py b/buildtools/get_cpu_count.py
> > new file mode 100644 index 000000000..386f85f8b
> > --- /dev/null
> > +++ b/buildtools/get_cpu_count.py
> > @@ -0,0 +1,7 @@
> > +#!/usr/bin/python3
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > +PANTHEON.tech s.r.o.
> > +
> > +import os
> > +
> > +print(os.cpu_count())
> > diff --git a/buildtools/get_numa_count.py
> > b/buildtools/get_numa_count.py new file mode 100644 index
> > 000000000..f0c49973a
> > --- /dev/null
> > +++ b/buildtools/get_numa_count.py
> > @@ -0,0 +1,22 @@
> > +#!/usr/bin/python3
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > +PANTHEON.tech s.r.o.
> > +
> > +import ctypes
> > +import glob
> > +import os
> > +import subprocess
> > +
> > +if os.name == 'posix':
> > +    if os.path.isdir('/sys/devices/system/node'):
> > +        print(len(glob.glob('/sys/devices/system/node/node*')))
> > +    else:
> > +        print(subprocess.run(['sysctl', 'vm.ndomains'],
> > +capture_output=True).stdout)
> > +
> > +elif os.name == 'nt':
> > +    libkernel32 = ctypes.windll.kernel32
> > +
> > +    count = ctypes.c_ulong()
> > +
> > +    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
> > +    print(count.value + 1)
> > diff --git a/buildtools/meson.build b/buildtools/meson.build index
> > 04808dabc..925e733b1 100644
> > --- a/buildtools/meson.build
> > +++ b/buildtools/meson.build
> > @@ -17,3 +17,5 @@ else
> >  endif
> >  map_to_win_cmd = py3 + files('map_to_win.py')  sphinx_wrapper = py3 +
> > files('call-sphinx-build.py')
> > +get_cpu_count_cmd = py3 + files('get_cpu_count.py')
> > +get_numa_count_cmd = py3 + files('get_numa_count.py')
> > diff --git a/config/meson.build b/config/meson.build index
> > a57c8ae9e..c4477f977 100644
> > --- a/config/meson.build
> > +++ b/config/meson.build
> > @@ -74,7 +74,11 @@ endif
> >  # still being able to support the CPU features required for DPDK.
> >  # This can be bumped up by the DPDK project, but it can never be an
> > # invariant like 'native'
> > +max_lcores = get_option('max_lcores') max_numa_nodes =
> > +get_option('max_numa_nodes')
> >  if machine == 'default'
> > +	max_numa_nodes = 4
> > +	max_lcores = 128
> 
> This doesn't seem right, since you are overriding the user-specified values with
> hard-coded ones.
> 

I understand we're using the default build/generic to build portalbe dpdk distro packages, meaning the settings for those packages should always be the same, no? If not, what should the default/generic build be? And when would someone do a default/generic build with their values? It wouldn't be a default/generic anymore, right?

> >  	if host_machine.cpu_family().startswith('x86')
> >  		# matches the old pre-meson build systems default
> >  		machine = 'corei7'
> > @@ -83,6 +87,22 @@ if machine == 'default'
> >  	elif host_machine.cpu_family().startswith('ppc')
> >  		machine = 'power8'
> >  	endif
> > +elif not meson.is_cross_build()
> > +	# find host core count and numa node count for native builds
> > +	if max_lcores == 0
> > +		max_lcores =
> run_command(get_cpu_count_cmd).stdout().to_int()
> > +		min_lcores = 2
> > +		if max_lcores < min_lcores
> > +			message('Found less than @0@ cores, building for
> @0@ cores'.format(min_lcores))
> > +			max_lcores = min_lcores
> > +		else
> > +			message('Found @0@ cores'.format(max_lcores))
> > +		endif
> > +	endif
> > +	if max_numa_nodes == 0
> > +		max_numa_nodes =
> run_command(get_numa_count_cmd).stdout().to_int()
> > +		message('Found @0@ numa nodes'.format(max_numa_nodes))
> > +	endif
> >  endif
> >
> >  dpdk_conf.set('RTE_MACHINE', machine) @@ -227,8 +247,10 @@ foreach
> > arg: warning_flags  endforeach
> >
> >  # set other values pulled from the build options
> > -dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
> > -dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
> > +if not meson.is_cross_build()
> > +	dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> > +	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes) endif
> 
> Rather than conditionally setting the value here, you should move the checks
> below up above this to simplify things.
> 

Do you mean the cross build checks? Those have to be after subdir(arch_subdir) so that we can override the values from cross files (as the commit message says). 

> >  dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
> > dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
> > dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp')) @@
> > -247,6 +269,22 @@ compile_time_cpuflags = []
> >  subdir(arch_subdir)
> >  dpdk_conf.set('RTE_COMPILE_TIME_CPUFLAGS',
> > ','.join(compile_time_cpuflags))
> >
> > +# check that cpu and numa count is set in cross builds if
> > +meson.is_cross_build()
> > +    	if max_lcores > 0
> > +		# specified on the cmdline
> > +		dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> > +	elif not dpdk_conf.has('RTE_MAX_LCORE')
> > +		error('Number of cores for cross build not specified in @0@
> subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
> > +	endif
> > +	if max_numa_nodes > 0
> > +		# specified on the cmdline
> > +		dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
> > +	elif not dpdk_conf.has('RTE_MAX_NUMA_NODES')
> > +		error('Number of numa nodes for cross build not specified in
> @0@ subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
> > +	endif
> > +endif
> > +
> >  # set the install path for the drivers
> > dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
> >
> > diff --git a/meson_options.txt b/meson_options.txt index
> > 9bf18ab6b..01b0c45c3 100644
> > --- a/meson_options.txt
> > +++ b/meson_options.txt
> > @@ -26,10 +26,10 @@ option('machine', type: 'string', value: 'native',
> >  	description: 'set the target machine type')  option('max_ethports',
> > type: 'integer', value: 32,
> >  	description: 'maximum number of Ethernet devices')
> > -option('max_lcores', type: 'integer', value: 128,
> > -	description: 'maximum number of cores/threads supported by EAL')
> > -option('max_numa_nodes', type: 'integer', value: 4,
> > -	description: 'maximum number of NUMA nodes supported by EAL')
> > +option('max_lcores', type: 'integer', value: 0,
> > +	description: 'maximum number of cores/threads supported by EAL.
> > +Value 0 means the number of cpus on the host will be used. For cross build,
> set to non-zero to overwrite the cross-file value.') option('max_numa_nodes',
> type: 'integer', value: 0,
> > +	description: 'maximum number of NUMA nodes supported by EAL. Value
> 0
> > +means the number of numa nodes on the host will be used. For cross
> > +build, set to non-zero to overwrite the cross-file value.')
> 
> I don't like this change, because it very much assumes for non-cross-compiles
> that people will be running DPDK on the system they build it on. That's a very,
> very big assumption!

I'll be using definitions from https://mesonbuild.com/Cross-compilation.html.
I understand cross compilation to be building for a diffent host machine than the build machine (which is aligned with pretty much every definition I found). I understand this to be true not only for builds between architectures, but also within an architecture (e.g. x86_64 build machine building for x86_64 host machine).
So yes, when someone does a native build, it stands to reason they want to use it on the build machine. If they wanted to use it elsewhere, they would cross compile.
Another thing is the current build philosophy is to detect as much as possible (not having statically defined configuration, as you mentioned in the past). Detecting the number of cores and numa nodes fits this perfectly.
And yet another thing is that the assumption seems to be already present in the build system - it already detects a lot things, some of which may not be satisfied on machines other than the build machine. I may be wrong about this.

> I'm ok with having zero as a "detect" option, and having
> the values overridden from cross-files, but not with detection as the default out-
> of-the-box option! Lots of users may pull builds from a CI based on VMs with
> just a few cores, for instance.

If not having the automatic detection is a concern because of users using CI builds, then we (if it's from our CI) can change what we're building in CI - the default/generic build seems like a good fit because it's supposed to work on a variety of systems. Expecting that native build from random VMs would work anywhere doesn't seen very realistic - it's been build for that VM environment (because it's a native build).

Here's my understanding on which the current version is based:
1. Since we want to get away from having statically defined configuration, numa and core count discovery is exactly what we should have in the build system. Since discorery is currently the default for lib/drivers, it stands to reason it should be default for everything else, if possible.
2. Native build should produce binaries matching the build machine as well as possible.
3. Default/generic build should produce binaries executable on a range of systems (ideally all systems of a given architecture).
4. Other builds, that is non-native builds, are cross-compilation, since we're building for host machine other that the build machine.

What I haven't taken into account is users using CI builds. That could be remedied by modifying the CI a bit while being consistent with what native/default/generic/cross builds are (or should be). And in any case, if we're not interested in testing the exact CI environment (which we aren't, since we don't want to use 2 cores with 1 numa), we really shouldn't be doing native builds there.

I'm interested in hearing where my thinking deviates from yours.

> 
> >  option('enable_trace_fp', type: 'boolean', value: false,
> >  	description: 'enable fast path trace points.')  option('tests',
> > type: 'boolean', value: true,
> > --
> > 2.20.1
> >

Re: [dpdk-dev] [RFC PATCH v3 3/6] build: automatic NUMA and cpu counts detection

[Bruce Richardson]

On Wed, Oct 21, 2020 at 01:01:41PM +0000, Juraj Linkeš wrote:
> 
> 
> > -----Original Message-----
> > From: Bruce Richardson <bruce.richardson@intel.com>
> > Sent: Wednesday, October 21, 2020 2:02 PM
> > To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > Cc: Ruifeng.Wang@arm.com; Honnappa.Nagarahalli@arm.com;
> > Phil.Yang@arm.com; vcchunga@amazon.com; Dharmik.Thakkar@arm.com;
> > jerinjacobk@gmail.com; hemant.agrawal@nxp.com; dev@dpdk.org
> > Subject: Re: [RFC PATCH v3 3/6] build: automatic NUMA and cpu counts
> > detection
> > 
> > On Wed, Oct 21, 2020 at 01:37:38PM +0200, Juraj Linkeš wrote:
> > > The build machine's number of cpus and numa nodes vary, resulting in
> > > mismatched counts of RTE_MAX_LCORE and RTE_MAX_NUMA_NODES for
> > many
> > > builds. Automatically discover the host's numa and cpu counts to
> > > remove this mismatch for native builds. Use current defaults for default builds.
> > > Force the users to specify the counts for cross build in cross files
> > > or on the command line.
> > > Give users the option to override the discovery or values from cross
> > > files by specifying them on the command line with -Dmax_lcores and
> > > -Dmax_numa_nodes.
> > >
> > > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > ---
> > >  buildtools/get_cpu_count.py  |  7 ++++++
> > > buildtools/get_numa_count.py | 22 +++++++++++++++++++
> > >  buildtools/meson.build       |  2 ++
> > >  config/meson.build           | 42 ++++++++++++++++++++++++++++++++++--
> > >  meson_options.txt            |  8 +++----
> > >  5 files changed, 75 insertions(+), 6 deletions(-)  create mode 100644
> > > buildtools/get_cpu_count.py  create mode 100644
> > > buildtools/get_numa_count.py
> > >
> > > diff --git a/buildtools/get_cpu_count.py b/buildtools/get_cpu_count.py
> > > new file mode 100644 index 000000000..386f85f8b
> > > --- /dev/null
> > > +++ b/buildtools/get_cpu_count.py
> > > @@ -0,0 +1,7 @@
> > > +#!/usr/bin/python3
> > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > > +PANTHEON.tech s.r.o.
> > > +
> > > +import os
> > > +
> > > +print(os.cpu_count())
> > > diff --git a/buildtools/get_numa_count.py
> > > b/buildtools/get_numa_count.py new file mode 100644 index
> > > 000000000..f0c49973a
> > > --- /dev/null
> > > +++ b/buildtools/get_numa_count.py
> > > @@ -0,0 +1,22 @@
> > > +#!/usr/bin/python3
> > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > > +PANTHEON.tech s.r.o.
> > > +
> > > +import ctypes
> > > +import glob
> > > +import os
> > > +import subprocess
> > > +
> > > +if os.name == 'posix':
> > > +    if os.path.isdir('/sys/devices/system/node'):
> > > +        print(len(glob.glob('/sys/devices/system/node/node*')))
> > > +    else:
> > > +        print(subprocess.run(['sysctl', 'vm.ndomains'],
> > > +capture_output=True).stdout)
> > > +
> > > +elif os.name == 'nt':
> > > +    libkernel32 = ctypes.windll.kernel32
> > > +
> > > +    count = ctypes.c_ulong()
> > > +
> > > +    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
> > > +    print(count.value + 1)
> > > diff --git a/buildtools/meson.build b/buildtools/meson.build index
> > > 04808dabc..925e733b1 100644
> > > --- a/buildtools/meson.build
> > > +++ b/buildtools/meson.build
> > > @@ -17,3 +17,5 @@ else
> > >  endif
> > >  map_to_win_cmd = py3 + files('map_to_win.py')  sphinx_wrapper = py3 +
> > > files('call-sphinx-build.py')
> > > +get_cpu_count_cmd = py3 + files('get_cpu_count.py')
> > > +get_numa_count_cmd = py3 + files('get_numa_count.py')
> > > diff --git a/config/meson.build b/config/meson.build index
> > > a57c8ae9e..c4477f977 100644
> > > --- a/config/meson.build
> > > +++ b/config/meson.build
> > > @@ -74,7 +74,11 @@ endif
> > >  # still being able to support the CPU features required for DPDK.
> > >  # This can be bumped up by the DPDK project, but it can never be an
> > > # invariant like 'native'
> > > +max_lcores = get_option('max_lcores') max_numa_nodes =
> > > +get_option('max_numa_nodes')
> > >  if machine == 'default'
> > > +	max_numa_nodes = 4
> > > +	max_lcores = 128
> > 
> > This doesn't seem right, since you are overriding the user-specified values with
> > hard-coded ones.
> > 
> 
> I understand we're using the default build/generic to build portalbe dpdk distro packages, meaning the settings for those packages should always be the same, no? If not, what should the default/generic build be? And when would someone do a default/generic build with their values? It wouldn't be a default/generic anymore, right?
> 
> > >  	if host_machine.cpu_family().startswith('x86')
> > >  		# matches the old pre-meson build systems default
> > >  		machine = 'corei7'
> > > @@ -83,6 +87,22 @@ if machine == 'default'
> > >  	elif host_machine.cpu_family().startswith('ppc')
> > >  		machine = 'power8'
> > >  	endif
> > > +elif not meson.is_cross_build()
> > > +	# find host core count and numa node count for native builds
> > > +	if max_lcores == 0
> > > +		max_lcores =
> > run_command(get_cpu_count_cmd).stdout().to_int()
> > > +		min_lcores = 2
> > > +		if max_lcores < min_lcores
> > > +			message('Found less than @0@ cores, building for
> > @0@ cores'.format(min_lcores))
> > > +			max_lcores = min_lcores
> > > +		else
> > > +			message('Found @0@ cores'.format(max_lcores))
> > > +		endif
> > > +	endif
> > > +	if max_numa_nodes == 0
> > > +		max_numa_nodes =
> > run_command(get_numa_count_cmd).stdout().to_int()
> > > +		message('Found @0@ numa nodes'.format(max_numa_nodes))
> > > +	endif
> > >  endif
> > >
> > >  dpdk_conf.set('RTE_MACHINE', machine) @@ -227,8 +247,10 @@ foreach
> > > arg: warning_flags  endforeach
> > >
> > >  # set other values pulled from the build options
> > > -dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
> > > -dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
> > > +if not meson.is_cross_build()
> > > +	dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> > > +	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes) endif
> > 
> > Rather than conditionally setting the value here, you should move the checks
> > below up above this to simplify things.
> > 
> 
> Do you mean the cross build checks? Those have to be after subdir(arch_subdir) so that we can override the values from cross files (as the commit message says). 
> 
> > >  dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
> > > dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
> > > dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp')) @@
> > > -247,6 +269,22 @@ compile_time_cpuflags = []
> > >  subdir(arch_subdir)
> > >  dpdk_conf.set('RTE_COMPILE_TIME_CPUFLAGS',
> > > ','.join(compile_time_cpuflags))
> > >
> > > +# check that cpu and numa count is set in cross builds if
> > > +meson.is_cross_build()
> > > +    	if max_lcores > 0
> > > +		# specified on the cmdline
> > > +		dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> > > +	elif not dpdk_conf.has('RTE_MAX_LCORE')
> > > +		error('Number of cores for cross build not specified in @0@
> > subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
> > > +	endif
> > > +	if max_numa_nodes > 0
> > > +		# specified on the cmdline
> > > +		dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
> > > +	elif not dpdk_conf.has('RTE_MAX_NUMA_NODES')
> > > +		error('Number of numa nodes for cross build not specified in
> > @0@ subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
> > > +	endif
> > > +endif
> > > +
> > >  # set the install path for the drivers
> > > dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
> > >
> > > diff --git a/meson_options.txt b/meson_options.txt index
> > > 9bf18ab6b..01b0c45c3 100644
> > > --- a/meson_options.txt
> > > +++ b/meson_options.txt
> > > @@ -26,10 +26,10 @@ option('machine', type: 'string', value: 'native',
> > >  	description: 'set the target machine type')  option('max_ethports',
> > > type: 'integer', value: 32,
> > >  	description: 'maximum number of Ethernet devices')
> > > -option('max_lcores', type: 'integer', value: 128,
> > > -	description: 'maximum number of cores/threads supported by EAL')
> > > -option('max_numa_nodes', type: 'integer', value: 4,
> > > -	description: 'maximum number of NUMA nodes supported by EAL')
> > > +option('max_lcores', type: 'integer', value: 0,
> > > +	description: 'maximum number of cores/threads supported by EAL.
> > > +Value 0 means the number of cpus on the host will be used. For cross build,
> > set to non-zero to overwrite the cross-file value.') option('max_numa_nodes',
> > type: 'integer', value: 0,
> > > +	description: 'maximum number of NUMA nodes supported by EAL. Value
> > 0
> > > +means the number of numa nodes on the host will be used. For cross
> > > +build, set to non-zero to overwrite the cross-file value.')
> > 
> > I don't like this change, because it very much assumes for non-cross-compiles
> > that people will be running DPDK on the system they build it on. That's a very,
> > very big assumption!
> 
> I'll be using definitions from https://mesonbuild.com/Cross-compilation.html.
> I understand cross compilation to be building for a diffent host machine than the build machine (which is aligned with pretty much every definition I found). I understand this to be true not only for builds between architectures, but also within an architecture (e.g. x86_64 build machine building for x86_64 host machine).
> So yes, when someone does a native build, it stands to reason they want to use it on the build machine. If they wanted to use it elsewhere, they would cross compile.
> Another thing is the current build philosophy is to detect as much as possible (not having statically defined configuration, as you mentioned in the past). Detecting the number of cores and numa nodes fits this perfectly.
> And yet another thing is that the assumption seems to be already present in the build system - it already detects a lot things, some of which may not be satisfied on machines other than the build machine. I may be wrong about this.
> 
> > I'm ok with having zero as a "detect" option, and having
> > the values overridden from cross-files, but not with detection as the default out-
> > of-the-box option! Lots of users may pull builds from a CI based on VMs with
> > just a few cores, for instance.
> 
> If not having the automatic detection is a concern because of users using CI builds, then we (if it's from our CI) can change what we're building in CI - the default/generic build seems like a good fit because it's supposed to work on a variety of systems. Expecting that native build from random VMs would work anywhere doesn't seen very realistic - it's been build for that VM environment (because it's a native build).
> 
> Here's my understanding on which the current version is based:
> 1. Since we want to get away from having statically defined configuration, numa and core count discovery is exactly what we should have in the build system. Since discorery is currently the default for lib/drivers, it stands to reason it should be default for everything else, if possible.
> 2. Native build should produce binaries matching the build machine as well as possible.
> 3. Default/generic build should produce binaries executable on a range of systems (ideally all systems of a given architecture).
> 4. Other builds, that is non-native builds, are cross-compilation, since we're building for host machine other that the build machine.
> 
> What I haven't taken into account is users using CI builds. That could be remedied by modifying the CI a bit while being consistent with what native/default/generic/cross builds are (or should be). And in any case, if we're not interested in testing the exact CI environment (which we aren't, since we don't want to use 2 cores with 1 numa), we really shouldn't be doing native builds there.
> 
> I'm interested in hearing where my thinking deviates from yours.
> 

There are a number of points in which we differ, I think.

Firstly, the use of "native" and "default/generic" for the "machine"
parameter refers only to the instruction-set level from the compiler, and
should not affect any other settings, since all settings are independent.
Therefore, setting "machine" to "native" does not mean that we should
detect cores and numa nodes, and similarly setting it to "default" does not
mean that we should ignore the settings for these values and pick our own
chosen default values.

Secondly, the use of cross-compilation only applies when you are compiling
for a different architecture or environment (e.g. OS) to what you are
building on. Building on a 4-core x86 machine to run on a dual-socket,
32-core x86 machine is not cross-compiling, but still needs to work by
default. Something like building a 32-bit binary on a 64-bit OS is in most
cases not done by cross-compilation either, but is rather outside the scope
of the discussion, except as a reference point to show the scope of
differences which can be accomodated as "native builds".

In terms of dynamic configuration for things like cores and numa nodes, the
ideal end state here is not to have them detected at build time on the host
system, but instead to have them detected at runtime and sized dynamically.
In the absense of that, these values should be set to reasonable defaults
so that when a user compiles up a binary without settings these explicitly
it should run on 95%+ of systems of that type.

This is my understanding of the issues, anyway. :-)

Regards,
/Bruce

Re: [dpdk-dev] [RFC PATCH v3 3/6] build: automatic NUMA and cpu counts detection

[Bruce Richardson]

On Wed, Oct 21, 2020 at 03:13:19PM +0100, Bruce Richardson wrote:
> On Wed, Oct 21, 2020 at 01:01:41PM +0000, Juraj Linkeš wrote:
> > 
> > 
> > > -----Original Message-----
> > > From: Bruce Richardson <bruce.richardson@intel.com>
> > > Sent: Wednesday, October 21, 2020 2:02 PM
> > > To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > Cc: Ruifeng.Wang@arm.com; Honnappa.Nagarahalli@arm.com;
> > > Phil.Yang@arm.com; vcchunga@amazon.com; Dharmik.Thakkar@arm.com;
> > > jerinjacobk@gmail.com; hemant.agrawal@nxp.com; dev@dpdk.org
> > > Subject: Re: [RFC PATCH v3 3/6] build: automatic NUMA and cpu counts
> > > detection
> > > 
> > > On Wed, Oct 21, 2020 at 01:37:38PM +0200, Juraj Linkeš wrote:
> > > > The build machine's number of cpus and numa nodes vary, resulting in
> > > > mismatched counts of RTE_MAX_LCORE and RTE_MAX_NUMA_NODES for
> > > many
> > > > builds. Automatically discover the host's numa and cpu counts to
> > > > remove this mismatch for native builds. Use current defaults for default builds.
> > > > Force the users to specify the counts for cross build in cross files
> > > > or on the command line.
> > > > Give users the option to override the discovery or values from cross
> > > > files by specifying them on the command line with -Dmax_lcores and
> > > > -Dmax_numa_nodes.
> > > >
> > > > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > > ---
> > > >  buildtools/get_cpu_count.py  |  7 ++++++
> > > > buildtools/get_numa_count.py | 22 +++++++++++++++++++
> > > >  buildtools/meson.build       |  2 ++
> > > >  config/meson.build           | 42 ++++++++++++++++++++++++++++++++++--
> > > >  meson_options.txt            |  8 +++----
> > > >  5 files changed, 75 insertions(+), 6 deletions(-)  create mode 100644
> > > > buildtools/get_cpu_count.py  create mode 100644
> > > > buildtools/get_numa_count.py
> > > >
> > > > diff --git a/buildtools/get_cpu_count.py b/buildtools/get_cpu_count.py
> > > > new file mode 100644 index 000000000..386f85f8b
> > > > --- /dev/null
> > > > +++ b/buildtools/get_cpu_count.py
> > > > @@ -0,0 +1,7 @@
> > > > +#!/usr/bin/python3
> > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > > > +PANTHEON.tech s.r.o.
> > > > +
> > > > +import os
> > > > +
> > > > +print(os.cpu_count())
> > > > diff --git a/buildtools/get_numa_count.py
> > > > b/buildtools/get_numa_count.py new file mode 100644 index
> > > > 000000000..f0c49973a
> > > > --- /dev/null
> > > > +++ b/buildtools/get_numa_count.py
> > > > @@ -0,0 +1,22 @@
> > > > +#!/usr/bin/python3
> > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > > > +PANTHEON.tech s.r.o.
> > > > +
> > > > +import ctypes
> > > > +import glob
> > > > +import os
> > > > +import subprocess
> > > > +
> > > > +if os.name == 'posix':
> > > > +    if os.path.isdir('/sys/devices/system/node'):
> > > > +        print(len(glob.glob('/sys/devices/system/node/node*')))
> > > > +    else:
> > > > +        print(subprocess.run(['sysctl', 'vm.ndomains'],
> > > > +capture_output=True).stdout)
> > > > +
> > > > +elif os.name == 'nt':
> > > > +    libkernel32 = ctypes.windll.kernel32
> > > > +
> > > > +    count = ctypes.c_ulong()
> > > > +
> > > > +    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
> > > > +    print(count.value + 1)
> > > > diff --git a/buildtools/meson.build b/buildtools/meson.build index
> > > > 04808dabc..925e733b1 100644
> > > > --- a/buildtools/meson.build
> > > > +++ b/buildtools/meson.build
> > > > @@ -17,3 +17,5 @@ else
> > > >  endif
> > > >  map_to_win_cmd = py3 + files('map_to_win.py')  sphinx_wrapper = py3 +
> > > > files('call-sphinx-build.py')
> > > > +get_cpu_count_cmd = py3 + files('get_cpu_count.py')
> > > > +get_numa_count_cmd = py3 + files('get_numa_count.py')
> > > > diff --git a/config/meson.build b/config/meson.build index
> > > > a57c8ae9e..c4477f977 100644
> > > > --- a/config/meson.build
> > > > +++ b/config/meson.build
> > > > @@ -74,7 +74,11 @@ endif
> > > >  # still being able to support the CPU features required for DPDK.
> > > >  # This can be bumped up by the DPDK project, but it can never be an
> > > > # invariant like 'native'
> > > > +max_lcores = get_option('max_lcores') max_numa_nodes =
> > > > +get_option('max_numa_nodes')
> > > >  if machine == 'default'
> > > > +	max_numa_nodes = 4
> > > > +	max_lcores = 128
> > > 
> > > This doesn't seem right, since you are overriding the user-specified values with
> > > hard-coded ones.
> > > 
> > 
> > I understand we're using the default build/generic to build portalbe dpdk distro packages, meaning the settings for those packages should always be the same, no? If not, what should the default/generic build be? And when would someone do a default/generic build with their values? It wouldn't be a default/generic anymore, right?
> > 
> > > >  	if host_machine.cpu_family().startswith('x86')
> > > >  		# matches the old pre-meson build systems default
> > > >  		machine = 'corei7'
> > > > @@ -83,6 +87,22 @@ if machine == 'default'
> > > >  	elif host_machine.cpu_family().startswith('ppc')
> > > >  		machine = 'power8'
> > > >  	endif
> > > > +elif not meson.is_cross_build()
> > > > +	# find host core count and numa node count for native builds
> > > > +	if max_lcores == 0
> > > > +		max_lcores =
> > > run_command(get_cpu_count_cmd).stdout().to_int()
> > > > +		min_lcores = 2
> > > > +		if max_lcores < min_lcores
> > > > +			message('Found less than @0@ cores, building for
> > > @0@ cores'.format(min_lcores))
> > > > +			max_lcores = min_lcores
> > > > +		else
> > > > +			message('Found @0@ cores'.format(max_lcores))
> > > > +		endif
> > > > +	endif
> > > > +	if max_numa_nodes == 0
> > > > +		max_numa_nodes =
> > > run_command(get_numa_count_cmd).stdout().to_int()
> > > > +		message('Found @0@ numa nodes'.format(max_numa_nodes))
> > > > +	endif
> > > >  endif
> > > >
> > > >  dpdk_conf.set('RTE_MACHINE', machine) @@ -227,8 +247,10 @@ foreach
> > > > arg: warning_flags  endforeach
> > > >
> > > >  # set other values pulled from the build options
> > > > -dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
> > > > -dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
> > > > +if not meson.is_cross_build()
> > > > +	dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> > > > +	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes) endif
> > > 
> > > Rather than conditionally setting the value here, you should move the checks
> > > below up above this to simplify things.
> > > 
> > 
> > Do you mean the cross build checks? Those have to be after subdir(arch_subdir) so that we can override the values from cross files (as the commit message says). 
> > 
> > > >  dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
> > > > dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
> > > > dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp')) @@
> > > > -247,6 +269,22 @@ compile_time_cpuflags = []
> > > >  subdir(arch_subdir)
> > > >  dpdk_conf.set('RTE_COMPILE_TIME_CPUFLAGS',
> > > > ','.join(compile_time_cpuflags))
> > > >
> > > > +# check that cpu and numa count is set in cross builds if
> > > > +meson.is_cross_build()
> > > > +    	if max_lcores > 0
> > > > +		# specified on the cmdline
> > > > +		dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> > > > +	elif not dpdk_conf.has('RTE_MAX_LCORE')
> > > > +		error('Number of cores for cross build not specified in @0@
> > > subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
> > > > +	endif
> > > > +	if max_numa_nodes > 0
> > > > +		# specified on the cmdline
> > > > +		dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
> > > > +	elif not dpdk_conf.has('RTE_MAX_NUMA_NODES')
> > > > +		error('Number of numa nodes for cross build not specified in
> > > @0@ subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
> > > > +	endif
> > > > +endif
> > > > +
> > > >  # set the install path for the drivers
> > > > dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
> > > >
> > > > diff --git a/meson_options.txt b/meson_options.txt index
> > > > 9bf18ab6b..01b0c45c3 100644
> > > > --- a/meson_options.txt
> > > > +++ b/meson_options.txt
> > > > @@ -26,10 +26,10 @@ option('machine', type: 'string', value: 'native',
> > > >  	description: 'set the target machine type')  option('max_ethports',
> > > > type: 'integer', value: 32,
> > > >  	description: 'maximum number of Ethernet devices')
> > > > -option('max_lcores', type: 'integer', value: 128,
> > > > -	description: 'maximum number of cores/threads supported by EAL')
> > > > -option('max_numa_nodes', type: 'integer', value: 4,
> > > > -	description: 'maximum number of NUMA nodes supported by EAL')
> > > > +option('max_lcores', type: 'integer', value: 0,
> > > > +	description: 'maximum number of cores/threads supported by EAL.
> > > > +Value 0 means the number of cpus on the host will be used. For cross build,
> > > set to non-zero to overwrite the cross-file value.') option('max_numa_nodes',
> > > type: 'integer', value: 0,
> > > > +	description: 'maximum number of NUMA nodes supported by EAL. Value
> > > 0
> > > > +means the number of numa nodes on the host will be used. For cross
> > > > +build, set to non-zero to overwrite the cross-file value.')
> > > 
> > > I don't like this change, because it very much assumes for non-cross-compiles
> > > that people will be running DPDK on the system they build it on. That's a very,
> > > very big assumption!
> > 
> > I'll be using definitions from https://mesonbuild.com/Cross-compilation.html.
> > I understand cross compilation to be building for a diffent host machine than the build machine (which is aligned with pretty much every definition I found). I understand this to be true not only for builds between architectures, but also within an architecture (e.g. x86_64 build machine building for x86_64 host machine).
> > So yes, when someone does a native build, it stands to reason they want to use it on the build machine. If they wanted to use it elsewhere, they would cross compile.
> > Another thing is the current build philosophy is to detect as much as possible (not having statically defined configuration, as you mentioned in the past). Detecting the number of cores and numa nodes fits this perfectly.
> > And yet another thing is that the assumption seems to be already present in the build system - it already detects a lot things, some of which may not be satisfied on machines other than the build machine. I may be wrong about this.
> > 
> > > I'm ok with having zero as a "detect" option, and having
> > > the values overridden from cross-files, but not with detection as the default out-
> > > of-the-box option! Lots of users may pull builds from a CI based on VMs with
> > > just a few cores, for instance.
> > 
> > If not having the automatic detection is a concern because of users using CI builds, then we (if it's from our CI) can change what we're building in CI - the default/generic build seems like a good fit because it's supposed to work on a variety of systems. Expecting that native build from random VMs would work anywhere doesn't seen very realistic - it's been build for that VM environment (because it's a native build).
> > 
> > Here's my understanding on which the current version is based:
> > 1. Since we want to get away from having statically defined configuration, numa and core count discovery is exactly what we should have in the build system. Since discorery is currently the default for lib/drivers, it stands to reason it should be default for everything else, if possible.
> > 2. Native build should produce binaries matching the build machine as well as possible.
> > 3. Default/generic build should produce binaries executable on a range of systems (ideally all systems of a given architecture).
> > 4. Other builds, that is non-native builds, are cross-compilation, since we're building for host machine other that the build machine.
> > 
> > What I haven't taken into account is users using CI builds. That could be remedied by modifying the CI a bit while being consistent with what native/default/generic/cross builds are (or should be). And in any case, if we're not interested in testing the exact CI environment (which we aren't, since we don't want to use 2 cores with 1 numa), we really shouldn't be doing native builds there.
> > 
> > I'm interested in hearing where my thinking deviates from yours.
> > 
> 
> There are a number of points in which we differ, I think.
> 
> Firstly, the use of "native" and "default/generic" for the "machine"
> parameter refers only to the instruction-set level from the compiler, and
> should not affect any other settings, since all settings are independent.
> Therefore, setting "machine" to "native" does not mean that we should
> detect cores and numa nodes, and similarly setting it to "default" does not
> mean that we should ignore the settings for these values and pick our own
> chosen default values.
> 
> Secondly, the use of cross-compilation only applies when you are compiling
> for a different architecture or environment (e.g. OS) to what you are
> building on. Building on a 4-core x86 machine to run on a dual-socket,
> 32-core x86 machine is not cross-compiling, but still needs to work by
> default. Something like building a 32-bit binary on a 64-bit OS is in most
> cases not done by cross-compilation either, but is rather outside the scope
> of the discussion, except as a reference point to show the scope of
> differences which can be accomodated as "native builds".
> 
> In terms of dynamic configuration for things like cores and numa nodes, the
> ideal end state here is not to have them detected at build time on the host
> system, but instead to have them detected at runtime and sized dynamically.
> In the absense of that, these values should be set to reasonable defaults
> so that when a user compiles up a binary without settings these explicitly
> it should run on 95%+ of systems of that type.
> 
> This is my understanding of the issues, anyway. :-)
> 
What could possibly work is to set the defaults for these to "0" as done in
your patch, but thereafter have the resulting defaults set
per-architecture, rather than globally. That would allow x86 to tune things
more for native-style builds, while in all cases allowing user to override.

/Bruce

Re: [dpdk-dev] [RFC PATCH v3 3/6] build: automatic NUMA and cpu counts detection

[Juraj Linkeš]

> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Wednesday, October 21, 2020 4:28 PM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Cc: Ruifeng.Wang@arm.com; Honnappa.Nagarahalli@arm.com;
> Phil.Yang@arm.com; vcchunga@amazon.com; Dharmik.Thakkar@arm.com;
> jerinjacobk@gmail.com; hemant.agrawal@nxp.com; dev@dpdk.org
> Subject: Re: [dpdk-dev] [RFC PATCH v3 3/6] build: automatic NUMA and cpu
> counts detection
> 
> On Wed, Oct 21, 2020 at 03:13:19PM +0100, Bruce Richardson wrote:
> > On Wed, Oct 21, 2020 at 01:01:41PM +0000, Juraj Linkeš wrote:
> > >
> > >
> > > > -----Original Message-----
> > > > From: Bruce Richardson <bruce.richardson@intel.com>
> > > > Sent: Wednesday, October 21, 2020 2:02 PM
> > > > To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > > Cc: Ruifeng.Wang@arm.com; Honnappa.Nagarahalli@arm.com;
> > > > Phil.Yang@arm.com; vcchunga@amazon.com;
> Dharmik.Thakkar@arm.com;
> > > > jerinjacobk@gmail.com; hemant.agrawal@nxp.com; dev@dpdk.org
> > > > Subject: Re: [RFC PATCH v3 3/6] build: automatic NUMA and cpu
> > > > counts detection
> > > >
> > > > On Wed, Oct 21, 2020 at 01:37:38PM +0200, Juraj Linkeš wrote:
> > > > > The build machine's number of cpus and numa nodes vary,
> > > > > resulting in mismatched counts of RTE_MAX_LCORE and
> > > > > RTE_MAX_NUMA_NODES for
> > > > many
> > > > > builds. Automatically discover the host's numa and cpu counts to
> > > > > remove this mismatch for native builds. Use current defaults for default
> builds.
> > > > > Force the users to specify the counts for cross build in cross
> > > > > files or on the command line.
> > > > > Give users the option to override the discovery or values from
> > > > > cross files by specifying them on the command line with
> > > > > -Dmax_lcores and -Dmax_numa_nodes.
> > > > >
> > > > > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > > > ---
> > > > >  buildtools/get_cpu_count.py  |  7 ++++++
> > > > > buildtools/get_numa_count.py | 22 +++++++++++++++++++
> > > > >  buildtools/meson.build       |  2 ++
> > > > >  config/meson.build           | 42 ++++++++++++++++++++++++++++++++++-
> -
> > > > >  meson_options.txt            |  8 +++----
> > > > >  5 files changed, 75 insertions(+), 6 deletions(-)  create mode
> > > > > 100644 buildtools/get_cpu_count.py  create mode 100644
> > > > > buildtools/get_numa_count.py
> > > > >
> > > > > diff --git a/buildtools/get_cpu_count.py
> > > > > b/buildtools/get_cpu_count.py new file mode 100644 index
> > > > > 000000000..386f85f8b
> > > > > --- /dev/null
> > > > > +++ b/buildtools/get_cpu_count.py
> > > > > @@ -0,0 +1,7 @@
> > > > > +#!/usr/bin/python3
> > > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > > > > +PANTHEON.tech s.r.o.
> > > > > +
> > > > > +import os
> > > > > +
> > > > > +print(os.cpu_count())
> > > > > diff --git a/buildtools/get_numa_count.py
> > > > > b/buildtools/get_numa_count.py new file mode 100644 index
> > > > > 000000000..f0c49973a
> > > > > --- /dev/null
> > > > > +++ b/buildtools/get_numa_count.py
> > > > > @@ -0,0 +1,22 @@
> > > > > +#!/usr/bin/python3
> > > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > > > > +PANTHEON.tech s.r.o.
> > > > > +
> > > > > +import ctypes
> > > > > +import glob
> > > > > +import os
> > > > > +import subprocess
> > > > > +
> > > > > +if os.name == 'posix':
> > > > > +    if os.path.isdir('/sys/devices/system/node'):
> > > > > +        print(len(glob.glob('/sys/devices/system/node/node*')))
> > > > > +    else:
> > > > > +        print(subprocess.run(['sysctl', 'vm.ndomains'],
> > > > > +capture_output=True).stdout)
> > > > > +
> > > > > +elif os.name == 'nt':
> > > > > +    libkernel32 = ctypes.windll.kernel32
> > > > > +
> > > > > +    count = ctypes.c_ulong()
> > > > > +
> > > > > +    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
> > > > > +    print(count.value + 1)
> > > > > diff --git a/buildtools/meson.build b/buildtools/meson.build
> > > > > index
> > > > > 04808dabc..925e733b1 100644
> > > > > --- a/buildtools/meson.build
> > > > > +++ b/buildtools/meson.build
> > > > > @@ -17,3 +17,5 @@ else
> > > > >  endif
> > > > >  map_to_win_cmd = py3 + files('map_to_win.py')  sphinx_wrapper =
> > > > > py3 +
> > > > > files('call-sphinx-build.py')
> > > > > +get_cpu_count_cmd = py3 + files('get_cpu_count.py')
> > > > > +get_numa_count_cmd = py3 + files('get_numa_count.py')
> > > > > diff --git a/config/meson.build b/config/meson.build index
> > > > > a57c8ae9e..c4477f977 100644
> > > > > --- a/config/meson.build
> > > > > +++ b/config/meson.build
> > > > > @@ -74,7 +74,11 @@ endif
> > > > >  # still being able to support the CPU features required for DPDK.
> > > > >  # This can be bumped up by the DPDK project, but it can never
> > > > > be an # invariant like 'native'
> > > > > +max_lcores = get_option('max_lcores') max_numa_nodes =
> > > > > +get_option('max_numa_nodes')
> > > > >  if machine == 'default'
> > > > > +	max_numa_nodes = 4
> > > > > +	max_lcores = 128
> > > >
> > > > This doesn't seem right, since you are overriding the
> > > > user-specified values with hard-coded ones.
> > > >
> > >
> > > I understand we're using the default build/generic to build portalbe dpdk
> distro packages, meaning the settings for those packages should always be the
> same, no? If not, what should the default/generic build be? And when would
> someone do a default/generic build with their values? It wouldn't be a
> default/generic anymore, right?
> > >
> > > > >  	if host_machine.cpu_family().startswith('x86')
> > > > >  		# matches the old pre-meson build systems default
> > > > >  		machine = 'corei7'
> > > > > @@ -83,6 +87,22 @@ if machine == 'default'
> > > > >  	elif host_machine.cpu_family().startswith('ppc')
> > > > >  		machine = 'power8'
> > > > >  	endif
> > > > > +elif not meson.is_cross_build()
> > > > > +	# find host core count and numa node count for native builds
> > > > > +	if max_lcores == 0
> > > > > +		max_lcores =
> > > > run_command(get_cpu_count_cmd).stdout().to_int()
> > > > > +		min_lcores = 2
> > > > > +		if max_lcores < min_lcores
> > > > > +			message('Found less than @0@ cores, building
> for
> > > > @0@ cores'.format(min_lcores))
> > > > > +			max_lcores = min_lcores
> > > > > +		else
> > > > > +			message('Found @0@
> cores'.format(max_lcores))
> > > > > +		endif
> > > > > +	endif
> > > > > +	if max_numa_nodes == 0
> > > > > +		max_numa_nodes =
> > > > run_command(get_numa_count_cmd).stdout().to_int()
> > > > > +		message('Found @0@ numa
> nodes'.format(max_numa_nodes))
> > > > > +	endif
> > > > >  endif
> > > > >
> > > > >  dpdk_conf.set('RTE_MACHINE', machine) @@ -227,8 +247,10 @@
> > > > > foreach
> > > > > arg: warning_flags  endforeach
> > > > >
> > > > >  # set other values pulled from the build options
> > > > > -dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
> > > > > -dpdk_conf.set('RTE_MAX_NUMA_NODES',
> > > > > get_option('max_numa_nodes'))
> > > > > +if not meson.is_cross_build()
> > > > > +	dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> > > > > +	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
> endif
> > > >
> > > > Rather than conditionally setting the value here, you should move
> > > > the checks below up above this to simplify things.
> > > >
> > >
> > > Do you mean the cross build checks? Those have to be after
> subdir(arch_subdir) so that we can override the values from cross files (as the
> commit message says).
> > >
> > > > >  dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
> > > > > dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
> > > > > dpdk_conf.set('RTE_ENABLE_TRACE_FP',
> > > > > get_option('enable_trace_fp')) @@
> > > > > -247,6 +269,22 @@ compile_time_cpuflags = []
> > > > >  subdir(arch_subdir)
> > > > >  dpdk_conf.set('RTE_COMPILE_TIME_CPUFLAGS',
> > > > > ','.join(compile_time_cpuflags))
> > > > >
> > > > > +# check that cpu and numa count is set in cross builds if
> > > > > +meson.is_cross_build()
> > > > > +    	if max_lcores > 0
> > > > > +		# specified on the cmdline
> > > > > +		dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> > > > > +	elif not dpdk_conf.has('RTE_MAX_LCORE')
> > > > > +		error('Number of cores for cross build not specified in
> @0@
> > > > subdir (e.g. in a cross-file) nor on the
> > > > cmdline'.format(arch_subdir))
> > > > > +	endif
> > > > > +	if max_numa_nodes > 0
> > > > > +		# specified on the cmdline
> > > > > +		dpdk_conf.set('RTE_MAX_NUMA_NODES',
> max_numa_nodes)
> > > > > +	elif not dpdk_conf.has('RTE_MAX_NUMA_NODES')
> > > > > +		error('Number of numa nodes for cross build not
> specified in
> > > > @0@ subdir (e.g. in a cross-file) nor on the
> > > > cmdline'.format(arch_subdir))
> > > > > +	endif
> > > > > +endif
> > > > > +
> > > > >  # set the install path for the drivers
> > > > > dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
> > > > >
> > > > > diff --git a/meson_options.txt b/meson_options.txt index
> > > > > 9bf18ab6b..01b0c45c3 100644
> > > > > --- a/meson_options.txt
> > > > > +++ b/meson_options.txt
> > > > > @@ -26,10 +26,10 @@ option('machine', type: 'string', value: 'native',
> > > > >  	description: 'set the target machine type')
> > > > > option('max_ethports',
> > > > > type: 'integer', value: 32,
> > > > >  	description: 'maximum number of Ethernet devices')
> > > > > -option('max_lcores', type: 'integer', value: 128,
> > > > > -	description: 'maximum number of cores/threads supported by
> EAL')
> > > > > -option('max_numa_nodes', type: 'integer', value: 4,
> > > > > -	description: 'maximum number of NUMA nodes supported by
> EAL')
> > > > > +option('max_lcores', type: 'integer', value: 0,
> > > > > +	description: 'maximum number of cores/threads supported by
> EAL.
> > > > > +Value 0 means the number of cpus on the host will be used. For
> > > > > +cross build,
> > > > set to non-zero to overwrite the cross-file value.')
> > > > option('max_numa_nodes',
> > > > type: 'integer', value: 0,
> > > > > +	description: 'maximum number of NUMA nodes supported by
> EAL.
> > > > > +Value
> > > > 0
> > > > > +means the number of numa nodes on the host will be used. For
> > > > > +cross build, set to non-zero to overwrite the cross-file
> > > > > +value.')
> > > >
> > > > I don't like this change, because it very much assumes for
> > > > non-cross-compiles that people will be running DPDK on the system
> > > > they build it on. That's a very, very big assumption!
> > >
> > > I'll be using definitions from https://mesonbuild.com/Cross-
> compilation.html.
> > > I understand cross compilation to be building for a diffent host machine than
> the build machine (which is aligned with pretty much every definition I found). I
> understand this to be true not only for builds between architectures, but also
> within an architecture (e.g. x86_64 build machine building for x86_64 host
> machine).
> > > So yes, when someone does a native build, it stands to reason they want to
> use it on the build machine. If they wanted to use it elsewhere, they would cross
> compile.
> > > Another thing is the current build philosophy is to detect as much as possible
> (not having statically defined configuration, as you mentioned in the past).
> Detecting the number of cores and numa nodes fits this perfectly.
> > > And yet another thing is that the assumption seems to be already present in
> the build system - it already detects a lot things, some of which may not be
> satisfied on machines other than the build machine. I may be wrong about this.
> > >
> > > > I'm ok with having zero as a "detect" option, and having the
> > > > values overridden from cross-files, but not with detection as the
> > > > default out- of-the-box option! Lots of users may pull builds from
> > > > a CI based on VMs with just a few cores, for instance.
> > >
> > > If not having the automatic detection is a concern because of users using CI
> builds, then we (if it's from our CI) can change what we're building in CI - the
> default/generic build seems like a good fit because it's supposed to work on a
> variety of systems. Expecting that native build from random VMs would work
> anywhere doesn't seen very realistic - it's been build for that VM environment
> (because it's a native build).
> > >
> > > Here's my understanding on which the current version is based:
> > > 1. Since we want to get away from having statically defined configuration,
> numa and core count discovery is exactly what we should have in the build
> system. Since discorery is currently the default for lib/drivers, it stands to reason
> it should be default for everything else, if possible.
> > > 2. Native build should produce binaries matching the build machine as well as
> possible.
> > > 3. Default/generic build should produce binaries executable on a range of
> systems (ideally all systems of a given architecture).
> > > 4. Other builds, that is non-native builds, are cross-compilation, since we're
> building for host machine other that the build machine.
> > >
> > > What I haven't taken into account is users using CI builds. That could be
> remedied by modifying the CI a bit while being consistent with what
> native/default/generic/cross builds are (or should be). And in any case, if we're
> not interested in testing the exact CI environment (which we aren't, since we
> don't want to use 2 cores with 1 numa), we really shouldn't be doing native
> builds there.
> > >
> > > I'm interested in hearing where my thinking deviates from yours.
> > >
> >
> > There are a number of points in which we differ, I think.
> >
> > Firstly, the use of "native" and "default/generic" for the "machine"
> > parameter refers only to the instruction-set level from the compiler,
> > and should not affect any other settings, since all settings are independent.

It seems that what you're saying may be true from the x86 point of view, but is not true from arm point of view and thus not true in general.
It's not true that all setting are independent for arm build. When building with machine=native we're discovering what machine we're building on and setting some settings (other than numa and core count) are based on that. For machine=generic we also have a set of values that build the generic binaries you describe below. More on this in that section.

> > Therefore, setting "machine" to "native" does not mean that we should
> > detect cores and numa nodes, and similarly setting it to "default"
> > does not mean that we should ignore the settings for these values and
> > pick our own chosen default values.
> >

I don't think this follows (if all settings are independent, then setting machine to native or anything else implies nothing for other settings). But I think it does follow from the what you write next.

However, since machine="default/generic" does not specify a type of build, but rather just one configuration option (or a small subset for arm builds), removing the ability to overwrite the default values when machine="default/generic" is not warranted.

> > Secondly, the use of cross-compilation only applies when you are
> > compiling for a different architecture or environment (e.g. OS) to
> > what you are building on. Building on a 4-core x86 machine to run on a
> > dual-socket, 32-core x86 machine is not cross-compiling, but still
> > needs to work by default.

We (the arm community, or at least a part of it) frequently refer to aarch64 -> aarch64 cross-compilation when we're building for a specific SoC (usually small/slow) on a fast server grade aarch64 machine. This is probably be because we're conflating what most people consider cross-compilation to be (building for a different arch or OS) with what meson cross compilation is (using a cross-file). Arm SoCs vary greatly and we want to capture that variation in cross files and use the cross files even on the same architecture (or rather, we want users to be able to). From now on when talking about cross compilation I'll mean meson cross compilation (i.e. using a cross file).

> > Something like building a 32-bit binary on a
> > 64-bit OS is in most cases not done by cross-compilation either, but
> > is rather outside the scope of the discussion, except as a reference
> > point to show the scope of differences which can be accomodated as "native
> builds".
> >
> > In terms of dynamic configuration for things like cores and numa
> > nodes, the ideal end state here is not to have them detected at build
> > time on the host system, but instead to have them detected at runtime and
> sized dynamically.

This would simplify everything, from building to users not having to worry about one more thing - would this be hard/time consuming to implement?

> > In the absense of that, these values should be set to reasonable
> > defaults so that when a user compiles up a binary without settings
> > these explicitly it should run on 95%+ of systems of that type.
> >

This, coupled with the paragraph about cross-compilation imply that we shouldn't default to discovering the values and I think that's reasonable. Having the ability to specify these on the command line, using an optional discovery mechanism and specifying these in a cross file should cover our (arm) needs.

> > This is my understanding of the issues, anyway. :-)
> >
> What could possibly work is to set the defaults for these to "0" as done in your
> patch, but thereafter have the resulting defaults set per-architecture, rather
> than globally. That would allow x86 to tune things more for native-style builds,
> while in all cases allowing user to override.
> 
> /Bruce

I'll change it so that per-arch (and for arm, per processor type) defaults are the defaults that will be used (when set to 0). I'd like to give users the option to use the discovery mechanism - I think we'll have to use another value with specific meaning and we can't use positive integers so I guess the best we can do is use -1 for this.

Juraj

Re: [dpdk-dev] [RFC PATCH v3 3/6] build: automatic NUMA and cpu counts detection

[Bruce Richardson]

On Fri, Oct 23, 2020 at 10:07:17AM +0000, Juraj Linkeš wrote:
> 
> 
> > -----Original Message-----
> > From: Bruce Richardson <bruce.richardson@intel.com>
> > Sent: Wednesday, October 21, 2020 4:28 PM
> > To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > Cc: Ruifeng.Wang@arm.com; Honnappa.Nagarahalli@arm.com;
> > Phil.Yang@arm.com; vcchunga@amazon.com; Dharmik.Thakkar@arm.com;
> > jerinjacobk@gmail.com; hemant.agrawal@nxp.com; dev@dpdk.org
> > Subject: Re: [dpdk-dev] [RFC PATCH v3 3/6] build: automatic NUMA and cpu
> > counts detection
> > 
<snip>
> > What could possibly work is to set the defaults for these to "0" as done in your
> > patch, but thereafter have the resulting defaults set per-architecture, rather
> > than globally. That would allow x86 to tune things more for native-style builds,
> > while in all cases allowing user to override.
> > 
> > /Bruce
> 
> I'll change it so that per-arch (and for arm, per processor type) defaults are the defaults that will be used (when set to 0). I'd like to give users the option to use the discovery mechanism - I think we'll have to use another value with specific meaning and we can't use positive integers so I guess the best we can do is use -1 for this.
> 
Well, if you want, if the defaults are set per-architecture, is you could
use detection for the default values for the ARM builds if the build-type
is set to "native" and the max_cores etc. are set to zero. For x86 that is
almost certainly not what we want, so we'd use hard-coded values for those
but no reason other architectures all need to do the same.

/Bruce

Re: [dpdk-dev] [RFC PATCH v3 3/6] build: automatic NUMA and cpu counts detection

[Honnappa Nagarahalli]

<snip>

> > >
> > > On Wed, Oct 21, 2020 at 01:37:38PM +0200, Juraj Linkeš wrote:
> > > > The build machine's number of cpus and numa nodes vary, resulting
> > > > in mismatched counts of RTE_MAX_LCORE and
> RTE_MAX_NUMA_NODES for
> > > many
> > > > builds. Automatically discover the host's numa and cpu counts to
> > > > remove this mismatch for native builds. Use current defaults for default
> builds.
> > > > Force the users to specify the counts for cross build in cross
> > > > files or on the command line.
> > > > Give users the option to override the discovery or values from
> > > > cross files by specifying them on the command line with
> > > > -Dmax_lcores and -Dmax_numa_nodes.
> > > >
> > > > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > > ---
> > > >  buildtools/get_cpu_count.py  |  7 ++++++
> > > > buildtools/get_numa_count.py | 22 +++++++++++++++++++
> > > >  buildtools/meson.build       |  2 ++
> > > >  config/meson.build           | 42
> ++++++++++++++++++++++++++++++++++--
> > > >  meson_options.txt            |  8 +++----
> > > >  5 files changed, 75 insertions(+), 6 deletions(-)  create mode
> > > > 100644 buildtools/get_cpu_count.py  create mode 100644
> > > > buildtools/get_numa_count.py
> > > >
> > > > diff --git a/buildtools/get_cpu_count.py
> > > > b/buildtools/get_cpu_count.py new file mode 100644 index
> > > > 000000000..386f85f8b
> > > > --- /dev/null
> > > > +++ b/buildtools/get_cpu_count.py
> > > > @@ -0,0 +1,7 @@
> > > > +#!/usr/bin/python3
> > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > > > +PANTHEON.tech s.r.o.
> > > > +
> > > > +import os
> > > > +
> > > > +print(os.cpu_count())
> > > > diff --git a/buildtools/get_numa_count.py
> > > > b/buildtools/get_numa_count.py new file mode 100644 index
> > > > 000000000..f0c49973a
> > > > --- /dev/null
> > > > +++ b/buildtools/get_numa_count.py
> > > > @@ -0,0 +1,22 @@
> > > > +#!/usr/bin/python3
> > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > > > +PANTHEON.tech s.r.o.
> > > > +
> > > > +import ctypes
> > > > +import glob
> > > > +import os
> > > > +import subprocess
> > > > +
> > > > +if os.name == 'posix':
> > > > +    if os.path.isdir('/sys/devices/system/node'):
> > > > +        print(len(glob.glob('/sys/devices/system/node/node*')))
> > > > +    else:
> > > > +        print(subprocess.run(['sysctl', 'vm.ndomains'],
> > > > +capture_output=True).stdout)
> > > > +
> > > > +elif os.name == 'nt':
> > > > +    libkernel32 = ctypes.windll.kernel32
> > > > +
> > > > +    count = ctypes.c_ulong()
> > > > +
> > > > +    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
> > > > +    print(count.value + 1)
> > > > diff --git a/buildtools/meson.build b/buildtools/meson.build index
> > > > 04808dabc..925e733b1 100644
> > > > --- a/buildtools/meson.build
> > > > +++ b/buildtools/meson.build
> > > > @@ -17,3 +17,5 @@ else
> > > >  endif
> > > >  map_to_win_cmd = py3 + files('map_to_win.py')  sphinx_wrapper =
> > > > py3 +
> > > > files('call-sphinx-build.py')
> > > > +get_cpu_count_cmd = py3 + files('get_cpu_count.py')
> > > > +get_numa_count_cmd = py3 + files('get_numa_count.py')
> > > > diff --git a/config/meson.build b/config/meson.build index
> > > > a57c8ae9e..c4477f977 100644
> > > > --- a/config/meson.build
> > > > +++ b/config/meson.build
> > > > @@ -74,7 +74,11 @@ endif
> > > >  # still being able to support the CPU features required for DPDK.
> > > >  # This can be bumped up by the DPDK project, but it can never be
> > > > an # invariant like 'native'
> > > > +max_lcores = get_option('max_lcores') max_numa_nodes =
> > > > +get_option('max_numa_nodes')
> > > >  if machine == 'default'
> > > > +	max_numa_nodes = 4
> > > > +	max_lcores = 128
> > >
> > > This doesn't seem right, since you are overriding the user-specified
> > > values with hard-coded ones.
> > >
> >
> > I understand we're using the default build/generic to build portalbe dpdk
> distro packages, meaning the settings for those packages should always be
> the same, no? If not, what should the default/generic build be? And when
> would someone do a default/generic build with their values? It wouldn't be a
> default/generic anymore, right?
> >
> > > >  	if host_machine.cpu_family().startswith('x86')
> > > >  		# matches the old pre-meson build systems default
> > > >  		machine = 'corei7'
> > > > @@ -83,6 +87,22 @@ if machine == 'default'
> > > >  	elif host_machine.cpu_family().startswith('ppc')
> > > >  		machine = 'power8'
> > > >  	endif
> > > > +elif not meson.is_cross_build()
> > > > +	# find host core count and numa node count for native builds
> > > > +	if max_lcores == 0
> > > > +		max_lcores =
> > > run_command(get_cpu_count_cmd).stdout().to_int()
> > > > +		min_lcores = 2
> > > > +		if max_lcores < min_lcores
> > > > +			message('Found less than @0@ cores, building for
> > > @0@ cores'.format(min_lcores))
> > > > +			max_lcores = min_lcores
> > > > +		else
> > > > +			message('Found @0@ cores'.format(max_lcores))
> > > > +		endif
> > > > +	endif
> > > > +	if max_numa_nodes == 0
> > > > +		max_numa_nodes =
> > > run_command(get_numa_count_cmd).stdout().to_int()
> > > > +		message('Found @0@ numa
> nodes'.format(max_numa_nodes))
> > > > +	endif
> > > >  endif
> > > >
> > > >  dpdk_conf.set('RTE_MACHINE', machine) @@ -227,8 +247,10 @@
> > > > foreach
> > > > arg: warning_flags  endforeach
> > > >
> > > >  # set other values pulled from the build options
> > > > -dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
> > > > -dpdk_conf.set('RTE_MAX_NUMA_NODES',
> get_option('max_numa_nodes'))
> > > > +if not meson.is_cross_build()
> > > > +	dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> > > > +	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
> endif
> > >
> > > Rather than conditionally setting the value here, you should move
> > > the checks below up above this to simplify things.
> > >
> >
> > Do you mean the cross build checks? Those have to be after
> subdir(arch_subdir) so that we can override the values from cross files (as
> the commit message says).
> >
> > > >  dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
> > > > dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
> > > > dpdk_conf.set('RTE_ENABLE_TRACE_FP',
> > > > get_option('enable_trace_fp')) @@
> > > > -247,6 +269,22 @@ compile_time_cpuflags = []
> > > >  subdir(arch_subdir)
> > > >  dpdk_conf.set('RTE_COMPILE_TIME_CPUFLAGS',
> > > > ','.join(compile_time_cpuflags))
> > > >
> > > > +# check that cpu and numa count is set in cross builds if
> > > > +meson.is_cross_build()
> > > > +    	if max_lcores > 0
> > > > +		# specified on the cmdline
> > > > +		dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> > > > +	elif not dpdk_conf.has('RTE_MAX_LCORE')
> > > > +		error('Number of cores for cross build not specified in @0@
> > > subdir (e.g. in a cross-file) nor on the
> > > cmdline'.format(arch_subdir))
> > > > +	endif
> > > > +	if max_numa_nodes > 0
> > > > +		# specified on the cmdline
> > > > +		dpdk_conf.set('RTE_MAX_NUMA_NODES',
> max_numa_nodes)
> > > > +	elif not dpdk_conf.has('RTE_MAX_NUMA_NODES')
> > > > +		error('Number of numa nodes for cross build not specified in
> > > @0@ subdir (e.g. in a cross-file) nor on the
> > > cmdline'.format(arch_subdir))
> > > > +	endif
> > > > +endif
> > > > +
> > > >  # set the install path for the drivers
> > > > dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
> > > >
> > > > diff --git a/meson_options.txt b/meson_options.txt index
> > > > 9bf18ab6b..01b0c45c3 100644
> > > > --- a/meson_options.txt
> > > > +++ b/meson_options.txt
> > > > @@ -26,10 +26,10 @@ option('machine', type: 'string', value: 'native',
> > > >  	description: 'set the target machine type')
> > > > option('max_ethports',
> > > > type: 'integer', value: 32,
> > > >  	description: 'maximum number of Ethernet devices')
> > > > -option('max_lcores', type: 'integer', value: 128,
> > > > -	description: 'maximum number of cores/threads supported by EAL')
> > > > -option('max_numa_nodes', type: 'integer', value: 4,
> > > > -	description: 'maximum number of NUMA nodes supported by EAL')
> > > > +option('max_lcores', type: 'integer', value: 0,
> > > > +	description: 'maximum number of cores/threads supported by EAL.
> > > > +Value 0 means the number of cpus on the host will be used. For
> > > > +cross build,
> > > set to non-zero to overwrite the cross-file value.')
> > > option('max_numa_nodes',
> > > type: 'integer', value: 0,
> > > > +	description: 'maximum number of NUMA nodes supported by EAL.
> > > > +Value
> > > 0
> > > > +means the number of numa nodes on the host will be used. For
> > > > +cross build, set to non-zero to overwrite the cross-file value.')
> > >
> > > I don't like this change, because it very much assumes for
> > > non-cross-compiles that people will be running DPDK on the system
> > > they build it on. That's a very, very big assumption!
> >
> > I'll be using definitions from https://mesonbuild.com/Cross-
> compilation.html.
> > I understand cross compilation to be building for a diffent host machine
> than the build machine (which is aligned with pretty much every definition I
> found). I understand this to be true not only for builds between
> architectures, but also within an architecture (e.g. x86_64 build machine
> building for x86_64 host machine).
> > So yes, when someone does a native build, it stands to reason they want to
> use it on the build machine. If they wanted to use it elsewhere, they would
> cross compile.
> > Another thing is the current build philosophy is to detect as much as
> possible (not having statically defined configuration, as you mentioned in the
> past). Detecting the number of cores and numa nodes fits this perfectly.
> > And yet another thing is that the assumption seems to be already present
> in the build system - it already detects a lot things, some of which may not be
> satisfied on machines other than the build machine. I may be wrong about
> this.
> >
> > > I'm ok with having zero as a "detect" option, and having the values
> > > overridden from cross-files, but not with detection as the default
> > > out- of-the-box option! Lots of users may pull builds from a CI
> > > based on VMs with just a few cores, for instance.
> >
> > If not having the automatic detection is a concern because of users using CI
> builds, then we (if it's from our CI) can change what we're building in CI - the
> default/generic build seems like a good fit because it's supposed to work on
> a variety of systems. Expecting that native build from random VMs would
> work anywhere doesn't seen very realistic - it's been build for that VM
> environment (because it's a native build).
> >
> > Here's my understanding on which the current version is based:
> > 1. Since we want to get away from having statically defined configuration,
> numa and core count discovery is exactly what we should have in the build
> system. Since discorery is currently the default for lib/drivers, it stands to
> reason it should be default for everything else, if possible.
> > 2. Native build should produce binaries matching the build machine as well
> as possible.
> > 3. Default/generic build should produce binaries executable on a range of
> systems (ideally all systems of a given architecture).
> > 4. Other builds, that is non-native builds, are cross-compilation, since we're
> building for host machine other that the build machine.
> >
> > What I haven't taken into account is users using CI builds. That could be
> remedied by modifying the CI a bit while being consistent with what
> native/default/generic/cross builds are (or should be). And in any case, if
> we're not interested in testing the exact CI environment (which we aren't,
> since we don't want to use 2 cores with 1 numa), we really shouldn't be doing
> native builds there.
> >
> > I'm interested in hearing where my thinking deviates from yours.
> >
> 
> There are a number of points in which we differ, I think.
> 
> Firstly, the use of "native" and "default/generic" for the "machine"
> parameter refers only to the instruction-set level from the compiler, and
> should not affect any other settings, since all settings are independent.
> Therefore, setting "machine" to "native" does not mean that we should
> detect cores and numa nodes, and similarly setting it to "default" does not
> mean that we should ignore the settings for these values and pick our own
> chosen default values.
Apologies to go to an older discussion.
I am trying to understand the definitions/expectations for 'native' and 'generic' builds.
As you say, instruction-set level is definitely one parameter. But, I think other DPDK specific parameters should also be considered.
For ex: RTE_MAX_LCORE should have a particular value for 'generic' build in all the supported architectures. The value could be different for each architecture, but it is fixed for the 'generic' build for a given architecture. Otherwise, the 'generic' build might not run on all the machines of that architecture.

Similarly, for 'native' build, is there any reason not to include other DPDK parameters as part of the definition? IMO, 'native' should refer to the entire build machine, not just the ISA. i.e. build on the target machine.

> 
> Secondly, the use of cross-compilation only applies when you are compiling
> for a different architecture or environment (e.g. OS) to what you are building
> on. Building on a 4-core x86 machine to run on a dual-socket, 32-core x86
> machine is not cross-compiling, but still needs to work by default. Something
> like building a 32-bit binary on a 64-bit OS is in most cases not done by cross-
> compilation either, but is rather outside the scope of the discussion, except
> as a reference point to show the scope of differences which can be
> accomodated as "native builds".
> 
> In terms of dynamic configuration for things like cores and numa nodes, the
> ideal end state here is not to have them detected at build time on the host
> system, but instead to have them detected at runtime and sized dynamically.
> In the absense of that, these values should be set to reasonable defaults so
> that when a user compiles up a binary without settings these explicitly it
> should run on 95%+ of systems of that type.
For the 'generic' build (which I also read it as DPDK binaries packaged with distros), this should be 100%.

> 
> This is my understanding of the issues, anyway. :-)
> 
> Regards,
> /Bruce

Re: [dpdk-dev] [RFC PATCH v3 3/6] build: automatic NUMA and cpu counts detection

[Bruce Richardson]

On Thu, Oct 29, 2020 at 04:31:33AM +0000, Honnappa Nagarahalli wrote:
> <snip>
> 
> > > >
> > > > On Wed, Oct 21, 2020 at 01:37:38PM +0200, Juraj Linkeš wrote:
> > > > > The build machine's number of cpus and numa nodes vary, resulting
> > > > > in mismatched counts of RTE_MAX_LCORE and
> > RTE_MAX_NUMA_NODES for
> > > > many
> > > > > builds. Automatically discover the host's numa and cpu counts to
> > > > > remove this mismatch for native builds. Use current defaults for default
> > builds.
> > > > > Force the users to specify the counts for cross build in cross
> > > > > files or on the command line.
> > > > > Give users the option to override the discovery or values from
> > > > > cross files by specifying them on the command line with
> > > > > -Dmax_lcores and -Dmax_numa_nodes.
> > > > >
> > > > > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > > > ---
> > > > >  buildtools/get_cpu_count.py  |  7 ++++++
> > > > > buildtools/get_numa_count.py | 22 +++++++++++++++++++
> > > > >  buildtools/meson.build       |  2 ++
> > > > >  config/meson.build           | 42
> > ++++++++++++++++++++++++++++++++++--
> > > > >  meson_options.txt            |  8 +++----
> > > > >  5 files changed, 75 insertions(+), 6 deletions(-)  create mode
> > > > > 100644 buildtools/get_cpu_count.py  create mode 100644
> > > > > buildtools/get_numa_count.py
> > > > >
> > > > > diff --git a/buildtools/get_cpu_count.py
> > > > > b/buildtools/get_cpu_count.py new file mode 100644 index
> > > > > 000000000..386f85f8b
> > > > > --- /dev/null
> > > > > +++ b/buildtools/get_cpu_count.py
> > > > > @@ -0,0 +1,7 @@
> > > > > +#!/usr/bin/python3
> > > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > > > > +PANTHEON.tech s.r.o.
> > > > > +
> > > > > +import os
> > > > > +
> > > > > +print(os.cpu_count())
> > > > > diff --git a/buildtools/get_numa_count.py
> > > > > b/buildtools/get_numa_count.py new file mode 100644 index
> > > > > 000000000..f0c49973a
> > > > > --- /dev/null
> > > > > +++ b/buildtools/get_numa_count.py
> > > > > @@ -0,0 +1,22 @@
> > > > > +#!/usr/bin/python3
> > > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > > > > +PANTHEON.tech s.r.o.
> > > > > +
> > > > > +import ctypes
> > > > > +import glob
> > > > > +import os
> > > > > +import subprocess
> > > > > +
> > > > > +if os.name == 'posix':
> > > > > +    if os.path.isdir('/sys/devices/system/node'):
> > > > > +        print(len(glob.glob('/sys/devices/system/node/node*')))
> > > > > +    else:
> > > > > +        print(subprocess.run(['sysctl', 'vm.ndomains'],
> > > > > +capture_output=True).stdout)
> > > > > +
> > > > > +elif os.name == 'nt':
> > > > > +    libkernel32 = ctypes.windll.kernel32
> > > > > +
> > > > > +    count = ctypes.c_ulong()
> > > > > +
> > > > > +    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
> > > > > +    print(count.value + 1)
> > > > > diff --git a/buildtools/meson.build b/buildtools/meson.build index
> > > > > 04808dabc..925e733b1 100644
> > > > > --- a/buildtools/meson.build
> > > > > +++ b/buildtools/meson.build
> > > > > @@ -17,3 +17,5 @@ else
> > > > >  endif
> > > > >  map_to_win_cmd = py3 + files('map_to_win.py')  sphinx_wrapper =
> > > > > py3 +
> > > > > files('call-sphinx-build.py')
> > > > > +get_cpu_count_cmd = py3 + files('get_cpu_count.py')
> > > > > +get_numa_count_cmd = py3 + files('get_numa_count.py')
> > > > > diff --git a/config/meson.build b/config/meson.build index
> > > > > a57c8ae9e..c4477f977 100644
> > > > > --- a/config/meson.build
> > > > > +++ b/config/meson.build
> > > > > @@ -74,7 +74,11 @@ endif
> > > > >  # still being able to support the CPU features required for DPDK.
> > > > >  # This can be bumped up by the DPDK project, but it can never be
> > > > > an # invariant like 'native'
> > > > > +max_lcores = get_option('max_lcores') max_numa_nodes =
> > > > > +get_option('max_numa_nodes')
> > > > >  if machine == 'default'
> > > > > +	max_numa_nodes = 4
> > > > > +	max_lcores = 128
> > > >
> > > > This doesn't seem right, since you are overriding the user-specified
> > > > values with hard-coded ones.
> > > >
> > >
> > > I understand we're using the default build/generic to build portalbe dpdk
> > distro packages, meaning the settings for those packages should always be
> > the same, no? If not, what should the default/generic build be? And when
> > would someone do a default/generic build with their values? It wouldn't be a
> > default/generic anymore, right?
> > >
> > > > >  	if host_machine.cpu_family().startswith('x86')
> > > > >  		# matches the old pre-meson build systems default
> > > > >  		machine = 'corei7'
> > > > > @@ -83,6 +87,22 @@ if machine == 'default'
> > > > >  	elif host_machine.cpu_family().startswith('ppc')
> > > > >  		machine = 'power8'
> > > > >  	endif
> > > > > +elif not meson.is_cross_build()
> > > > > +	# find host core count and numa node count for native builds
> > > > > +	if max_lcores == 0
> > > > > +		max_lcores =
> > > > run_command(get_cpu_count_cmd).stdout().to_int()
> > > > > +		min_lcores = 2
> > > > > +		if max_lcores < min_lcores
> > > > > +			message('Found less than @0@ cores, building for
> > > > @0@ cores'.format(min_lcores))
> > > > > +			max_lcores = min_lcores
> > > > > +		else
> > > > > +			message('Found @0@ cores'.format(max_lcores))
> > > > > +		endif
> > > > > +	endif
> > > > > +	if max_numa_nodes == 0
> > > > > +		max_numa_nodes =
> > > > run_command(get_numa_count_cmd).stdout().to_int()
> > > > > +		message('Found @0@ numa
> > nodes'.format(max_numa_nodes))
> > > > > +	endif
> > > > >  endif
> > > > >
> > > > >  dpdk_conf.set('RTE_MACHINE', machine) @@ -227,8 +247,10 @@
> > > > > foreach
> > > > > arg: warning_flags  endforeach
> > > > >
> > > > >  # set other values pulled from the build options
> > > > > -dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
> > > > > -dpdk_conf.set('RTE_MAX_NUMA_NODES',
> > get_option('max_numa_nodes'))
> > > > > +if not meson.is_cross_build()
> > > > > +	dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> > > > > +	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
> > endif
> > > >
> > > > Rather than conditionally setting the value here, you should move
> > > > the checks below up above this to simplify things.
> > > >
> > >
> > > Do you mean the cross build checks? Those have to be after
> > subdir(arch_subdir) so that we can override the values from cross files (as
> > the commit message says).
> > >
> > > > >  dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
> > > > > dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
> > > > > dpdk_conf.set('RTE_ENABLE_TRACE_FP',
> > > > > get_option('enable_trace_fp')) @@
> > > > > -247,6 +269,22 @@ compile_time_cpuflags = []
> > > > >  subdir(arch_subdir)
> > > > >  dpdk_conf.set('RTE_COMPILE_TIME_CPUFLAGS',
> > > > > ','.join(compile_time_cpuflags))
> > > > >
> > > > > +# check that cpu and numa count is set in cross builds if
> > > > > +meson.is_cross_build()
> > > > > +    	if max_lcores > 0
> > > > > +		# specified on the cmdline
> > > > > +		dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> > > > > +	elif not dpdk_conf.has('RTE_MAX_LCORE')
> > > > > +		error('Number of cores for cross build not specified in @0@
> > > > subdir (e.g. in a cross-file) nor on the
> > > > cmdline'.format(arch_subdir))
> > > > > +	endif
> > > > > +	if max_numa_nodes > 0
> > > > > +		# specified on the cmdline
> > > > > +		dpdk_conf.set('RTE_MAX_NUMA_NODES',
> > max_numa_nodes)
> > > > > +	elif not dpdk_conf.has('RTE_MAX_NUMA_NODES')
> > > > > +		error('Number of numa nodes for cross build not specified in
> > > > @0@ subdir (e.g. in a cross-file) nor on the
> > > > cmdline'.format(arch_subdir))
> > > > > +	endif
> > > > > +endif
> > > > > +
> > > > >  # set the install path for the drivers
> > > > > dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
> > > > >
> > > > > diff --git a/meson_options.txt b/meson_options.txt index
> > > > > 9bf18ab6b..01b0c45c3 100644
> > > > > --- a/meson_options.txt
> > > > > +++ b/meson_options.txt
> > > > > @@ -26,10 +26,10 @@ option('machine', type: 'string', value: 'native',
> > > > >  	description: 'set the target machine type')
> > > > > option('max_ethports',
> > > > > type: 'integer', value: 32,
> > > > >  	description: 'maximum number of Ethernet devices')
> > > > > -option('max_lcores', type: 'integer', value: 128,
> > > > > -	description: 'maximum number of cores/threads supported by EAL')
> > > > > -option('max_numa_nodes', type: 'integer', value: 4,
> > > > > -	description: 'maximum number of NUMA nodes supported by EAL')
> > > > > +option('max_lcores', type: 'integer', value: 0,
> > > > > +	description: 'maximum number of cores/threads supported by EAL.
> > > > > +Value 0 means the number of cpus on the host will be used. For
> > > > > +cross build,
> > > > set to non-zero to overwrite the cross-file value.')
> > > > option('max_numa_nodes',
> > > > type: 'integer', value: 0,
> > > > > +	description: 'maximum number of NUMA nodes supported by EAL.
> > > > > +Value
> > > > 0
> > > > > +means the number of numa nodes on the host will be used. For
> > > > > +cross build, set to non-zero to overwrite the cross-file value.')
> > > >
> > > > I don't like this change, because it very much assumes for
> > > > non-cross-compiles that people will be running DPDK on the system
> > > > they build it on. That's a very, very big assumption!
> > >
> > > I'll be using definitions from https://mesonbuild.com/Cross-
> > compilation.html.
> > > I understand cross compilation to be building for a diffent host machine
> > than the build machine (which is aligned with pretty much every definition I
> > found). I understand this to be true not only for builds between
> > architectures, but also within an architecture (e.g. x86_64 build machine
> > building for x86_64 host machine).
> > > So yes, when someone does a native build, it stands to reason they want to
> > use it on the build machine. If they wanted to use it elsewhere, they would
> > cross compile.
> > > Another thing is the current build philosophy is to detect as much as
> > possible (not having statically defined configuration, as you mentioned in the
> > past). Detecting the number of cores and numa nodes fits this perfectly.
> > > And yet another thing is that the assumption seems to be already present
> > in the build system - it already detects a lot things, some of which may not be
> > satisfied on machines other than the build machine. I may be wrong about
> > this.
> > >
> > > > I'm ok with having zero as a "detect" option, and having the values
> > > > overridden from cross-files, but not with detection as the default
> > > > out- of-the-box option! Lots of users may pull builds from a CI
> > > > based on VMs with just a few cores, for instance.
> > >
> > > If not having the automatic detection is a concern because of users using CI
> > builds, then we (if it's from our CI) can change what we're building in CI - the
> > default/generic build seems like a good fit because it's supposed to work on
> > a variety of systems. Expecting that native build from random VMs would
> > work anywhere doesn't seen very realistic - it's been build for that VM
> > environment (because it's a native build).
> > >
> > > Here's my understanding on which the current version is based:
> > > 1. Since we want to get away from having statically defined configuration,
> > numa and core count discovery is exactly what we should have in the build
> > system. Since discorery is currently the default for lib/drivers, it stands to
> > reason it should be default for everything else, if possible.
> > > 2. Native build should produce binaries matching the build machine as well
> > as possible.
> > > 3. Default/generic build should produce binaries executable on a range of
> > systems (ideally all systems of a given architecture).
> > > 4. Other builds, that is non-native builds, are cross-compilation, since we're
> > building for host machine other that the build machine.
> > >
> > > What I haven't taken into account is users using CI builds. That could be
> > remedied by modifying the CI a bit while being consistent with what
> > native/default/generic/cross builds are (or should be). And in any case, if
> > we're not interested in testing the exact CI environment (which we aren't,
> > since we don't want to use 2 cores with 1 numa), we really shouldn't be doing
> > native builds there.
> > >
> > > I'm interested in hearing where my thinking deviates from yours.
> > >
> > 
> > There are a number of points in which we differ, I think.
> > 
> > Firstly, the use of "native" and "default/generic" for the "machine"
> > parameter refers only to the instruction-set level from the compiler, and
> > should not affect any other settings, since all settings are independent.
> > Therefore, setting "machine" to "native" does not mean that we should
> > detect cores and numa nodes, and similarly setting it to "default" does not
> > mean that we should ignore the settings for these values and pick our own
> > chosen default values.
> Apologies to go to an older discussion.
> I am trying to understand the definitions/expectations for 'native' and 'generic' builds.
> As you say, instruction-set level is definitely one parameter.

Part of the confusion arises from the fact that originally that was the
only parameter set by this - and on x86 it still is. Perhaps this parameter
needs to be renamed to "isa-level" or "architecture-flag" or similar to
reflect its meaning. This would then allow a new "machine" setting, which
can be considered separately. The question then is how much that helps with
the main issue under discussion, that of cores and numa node values.

> But, I think other DPDK specific parameters should also be considered.
> For ex: RTE_MAX_LCORE should have a particular value for 'generic' build in all the supported architectures. The value could be different for each architecture, but it is fixed for the 'generic' build for a given architecture. Otherwise, the 'generic' build might not run on all the machines of that architecture.
> 
> Similarly, for 'native' build, is there any reason not to include other DPDK parameters as part of the definition? IMO, 'native' should refer to the entire build machine, not just the ISA. i.e. build on the target machine.
> 

While I understand the idea here, it is somewhat complicated by the fact
that the meson options given in "meson_options.txt" cannot be set by meson
code, which means that when we change the machine flag to "native" we can
only use or ignore the user-provided lcores and numa nodes setting - we
have no way to change them and reflect those changes back to the user. :-(
This leads to the situation in the discussion in this thread, where we
start needing all sorts of magic values to indicate use of machine-type
defaults or detected defaults.

Regards,
/Bruce

Re: [dpdk-dev] [RFC PATCH v3 3/6] build: automatic NUMA and cpu counts detection

[Honnappa Nagarahalli]

<snip>

> >
> > > > >
> > > > > On Wed, Oct 21, 2020 at 01:37:38PM +0200, Juraj Linkeš wrote:
> > > > > > The build machine's number of cpus and numa nodes vary,
> > > > > > resulting in mismatched counts of RTE_MAX_LCORE and
> > > RTE_MAX_NUMA_NODES for
> > > > > many
> > > > > > builds. Automatically discover the host's numa and cpu counts
> > > > > > to remove this mismatch for native builds. Use current
> > > > > > defaults for default
> > > builds.
> > > > > > Force the users to specify the counts for cross build in cross
> > > > > > files or on the command line.
> > > > > > Give users the option to override the discovery or values from
> > > > > > cross files by specifying them on the command line with
> > > > > > -Dmax_lcores and -Dmax_numa_nodes.
> > > > > >
> > > > > > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > > > > ---
> > > > > >  buildtools/get_cpu_count.py  |  7 ++++++
> > > > > > buildtools/get_numa_count.py | 22 +++++++++++++++++++
> > > > > >  buildtools/meson.build       |  2 ++
> > > > > >  config/meson.build           | 42
> > > ++++++++++++++++++++++++++++++++++--
> > > > > >  meson_options.txt            |  8 +++----
> > > > > >  5 files changed, 75 insertions(+), 6 deletions(-)  create
> > > > > > mode
> > > > > > 100644 buildtools/get_cpu_count.py  create mode 100644
> > > > > > buildtools/get_numa_count.py
> > > > > >
> > > > > > diff --git a/buildtools/get_cpu_count.py
> > > > > > b/buildtools/get_cpu_count.py new file mode 100644 index
> > > > > > 000000000..386f85f8b
> > > > > > --- /dev/null
> > > > > > +++ b/buildtools/get_cpu_count.py
> > > > > > @@ -0,0 +1,7 @@
> > > > > > +#!/usr/bin/python3
> > > > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > > > > > +PANTHEON.tech s.r.o.
> > > > > > +
> > > > > > +import os
> > > > > > +
> > > > > > +print(os.cpu_count())
> > > > > > diff --git a/buildtools/get_numa_count.py
> > > > > > b/buildtools/get_numa_count.py new file mode 100644 index
> > > > > > 000000000..f0c49973a
> > > > > > --- /dev/null
> > > > > > +++ b/buildtools/get_numa_count.py
> > > > > > @@ -0,0 +1,22 @@
> > > > > > +#!/usr/bin/python3
> > > > > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > > > > > +PANTHEON.tech s.r.o.
> > > > > > +
> > > > > > +import ctypes
> > > > > > +import glob
> > > > > > +import os
> > > > > > +import subprocess
> > > > > > +
> > > > > > +if os.name == 'posix':
> > > > > > +    if os.path.isdir('/sys/devices/system/node'):
> > > > > > +        print(len(glob.glob('/sys/devices/system/node/node*')))
> > > > > > +    else:
> > > > > > +        print(subprocess.run(['sysctl', 'vm.ndomains'],
> > > > > > +capture_output=True).stdout)
> > > > > > +
> > > > > > +elif os.name == 'nt':
> > > > > > +    libkernel32 = ctypes.windll.kernel32
> > > > > > +
> > > > > > +    count = ctypes.c_ulong()
> > > > > > +
> > > > > > +
> libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
> > > > > > +    print(count.value + 1)
> > > > > > diff --git a/buildtools/meson.build b/buildtools/meson.build
> > > > > > index
> > > > > > 04808dabc..925e733b1 100644
> > > > > > --- a/buildtools/meson.build
> > > > > > +++ b/buildtools/meson.build
> > > > > > @@ -17,3 +17,5 @@ else
> > > > > >  endif
> > > > > >  map_to_win_cmd = py3 + files('map_to_win.py')  sphinx_wrapper
> > > > > > =
> > > > > > py3 +
> > > > > > files('call-sphinx-build.py')
> > > > > > +get_cpu_count_cmd = py3 + files('get_cpu_count.py')
> > > > > > +get_numa_count_cmd = py3 + files('get_numa_count.py')
> > > > > > diff --git a/config/meson.build b/config/meson.build index
> > > > > > a57c8ae9e..c4477f977 100644
> > > > > > --- a/config/meson.build
> > > > > > +++ b/config/meson.build
> > > > > > @@ -74,7 +74,11 @@ endif
> > > > > >  # still being able to support the CPU features required for DPDK.
> > > > > >  # This can be bumped up by the DPDK project, but it can never
> > > > > > be an # invariant like 'native'
> > > > > > +max_lcores = get_option('max_lcores') max_numa_nodes =
> > > > > > +get_option('max_numa_nodes')
> > > > > >  if machine == 'default'
> > > > > > +	max_numa_nodes = 4
> > > > > > +	max_lcores = 128
> > > > >
> > > > > This doesn't seem right, since you are overriding the
> > > > > user-specified values with hard-coded ones.
> > > > >
> > > >
> > > > I understand we're using the default build/generic to build
> > > > portalbe dpdk
> > > distro packages, meaning the settings for those packages should
> > > always be the same, no? If not, what should the default/generic
> > > build be? And when would someone do a default/generic build with
> > > their values? It wouldn't be a default/generic anymore, right?
> > > >
> > > > > >  	if host_machine.cpu_family().startswith('x86')
> > > > > >  		# matches the old pre-meson build systems default
> > > > > >  		machine = 'corei7'
> > > > > > @@ -83,6 +87,22 @@ if machine == 'default'
> > > > > >  	elif host_machine.cpu_family().startswith('ppc')
> > > > > >  		machine = 'power8'
> > > > > >  	endif
> > > > > > +elif not meson.is_cross_build()
> > > > > > +	# find host core count and numa node count for native builds
> > > > > > +	if max_lcores == 0
> > > > > > +		max_lcores =
> > > > > run_command(get_cpu_count_cmd).stdout().to_int()
> > > > > > +		min_lcores = 2
> > > > > > +		if max_lcores < min_lcores
> > > > > > +			message('Found less than @0@ cores,
> building for
> > > > > @0@ cores'.format(min_lcores))
> > > > > > +			max_lcores = min_lcores
> > > > > > +		else
> > > > > > +			message('Found @0@
> cores'.format(max_lcores))
> > > > > > +		endif
> > > > > > +	endif
> > > > > > +	if max_numa_nodes == 0
> > > > > > +		max_numa_nodes =
> > > > > run_command(get_numa_count_cmd).stdout().to_int()
> > > > > > +		message('Found @0@ numa
> > > nodes'.format(max_numa_nodes))
> > > > > > +	endif
> > > > > >  endif
> > > > > >
> > > > > >  dpdk_conf.set('RTE_MACHINE', machine) @@ -227,8 +247,10 @@
> > > > > > foreach
> > > > > > arg: warning_flags  endforeach
> > > > > >
> > > > > >  # set other values pulled from the build options
> > > > > > -dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
> > > > > > -dpdk_conf.set('RTE_MAX_NUMA_NODES',
> > > get_option('max_numa_nodes'))
> > > > > > +if not meson.is_cross_build()
> > > > > > +	dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> > > > > > +	dpdk_conf.set('RTE_MAX_NUMA_NODES',
> max_numa_nodes)
> > > endif
> > > > >
> > > > > Rather than conditionally setting the value here, you should
> > > > > move the checks below up above this to simplify things.
> > > > >
> > > >
> > > > Do you mean the cross build checks? Those have to be after
> > > subdir(arch_subdir) so that we can override the values from cross
> > > files (as the commit message says).
> > > >
> > > > > >  dpdk_conf.set('RTE_MAX_ETHPORTS',
> get_option('max_ethports'))
> > > > > > dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
> > > > > > dpdk_conf.set('RTE_ENABLE_TRACE_FP',
> > > > > > get_option('enable_trace_fp')) @@
> > > > > > -247,6 +269,22 @@ compile_time_cpuflags = []
> > > > > >  subdir(arch_subdir)
> > > > > >  dpdk_conf.set('RTE_COMPILE_TIME_CPUFLAGS',
> > > > > > ','.join(compile_time_cpuflags))
> > > > > >
> > > > > > +# check that cpu and numa count is set in cross builds if
> > > > > > +meson.is_cross_build()
> > > > > > +    	if max_lcores > 0
> > > > > > +		# specified on the cmdline
> > > > > > +		dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> > > > > > +	elif not dpdk_conf.has('RTE_MAX_LCORE')
> > > > > > +		error('Number of cores for cross build not specified in
> @0@
> > > > > subdir (e.g. in a cross-file) nor on the
> > > > > cmdline'.format(arch_subdir))
> > > > > > +	endif
> > > > > > +	if max_numa_nodes > 0
> > > > > > +		# specified on the cmdline
> > > > > > +		dpdk_conf.set('RTE_MAX_NUMA_NODES',
> > > max_numa_nodes)
> > > > > > +	elif not dpdk_conf.has('RTE_MAX_NUMA_NODES')
> > > > > > +		error('Number of numa nodes for cross build not
> specified
> > > > > > +in
> > > > > @0@ subdir (e.g. in a cross-file) nor on the
> > > > > cmdline'.format(arch_subdir))
> > > > > > +	endif
> > > > > > +endif
> > > > > > +
> > > > > >  # set the install path for the drivers
> > > > > > dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
> > > > > >
> > > > > > diff --git a/meson_options.txt b/meson_options.txt index
> > > > > > 9bf18ab6b..01b0c45c3 100644
> > > > > > --- a/meson_options.txt
> > > > > > +++ b/meson_options.txt
> > > > > > @@ -26,10 +26,10 @@ option('machine', type: 'string', value:
> 'native',
> > > > > >  	description: 'set the target machine type')
> > > > > > option('max_ethports',
> > > > > > type: 'integer', value: 32,
> > > > > >  	description: 'maximum number of Ethernet devices')
> > > > > > -option('max_lcores', type: 'integer', value: 128,
> > > > > > -	description: 'maximum number of cores/threads supported
> by EAL')
> > > > > > -option('max_numa_nodes', type: 'integer', value: 4,
> > > > > > -	description: 'maximum number of NUMA nodes supported
> by EAL')
> > > > > > +option('max_lcores', type: 'integer', value: 0,
> > > > > > +	description: 'maximum number of cores/threads supported
> by EAL.
> > > > > > +Value 0 means the number of cpus on the host will be used.
> > > > > > +For cross build,
> > > > > set to non-zero to overwrite the cross-file value.')
> > > > > option('max_numa_nodes',
> > > > > type: 'integer', value: 0,
> > > > > > +	description: 'maximum number of NUMA nodes supported
> by EAL.
> > > > > > +Value
> > > > > 0
> > > > > > +means the number of numa nodes on the host will be used. For
> > > > > > +cross build, set to non-zero to overwrite the cross-file
> > > > > > +value.')
> > > > >
> > > > > I don't like this change, because it very much assumes for
> > > > > non-cross-compiles that people will be running DPDK on the
> > > > > system they build it on. That's a very, very big assumption!
> > > >
> > > > I'll be using definitions from https://mesonbuild.com/Cross-
> > > compilation.html.
> > > > I understand cross compilation to be building for a diffent host
> > > > machine
> > > than the build machine (which is aligned with pretty much every
> > > definition I found). I understand this to be true not only for
> > > builds between architectures, but also within an architecture (e.g.
> > > x86_64 build machine building for x86_64 host machine).
> > > > So yes, when someone does a native build, it stands to reason they
> > > > want to
> > > use it on the build machine. If they wanted to use it elsewhere,
> > > they would cross compile.
> > > > Another thing is the current build philosophy is to detect as much
> > > > as
> > > possible (not having statically defined configuration, as you
> > > mentioned in the past). Detecting the number of cores and numa nodes
> fits this perfectly.
> > > > And yet another thing is that the assumption seems to be already
> > > > present
> > > in the build system - it already detects a lot things, some of which
> > > may not be satisfied on machines other than the build machine. I may
> > > be wrong about this.
> > > >
> > > > > I'm ok with having zero as a "detect" option, and having the
> > > > > values overridden from cross-files, but not with detection as
> > > > > the default
> > > > > out- of-the-box option! Lots of users may pull builds from a CI
> > > > > based on VMs with just a few cores, for instance.
> > > >
> > > > If not having the automatic detection is a concern because of
> > > > users using CI
> > > builds, then we (if it's from our CI) can change what we're building
> > > in CI - the default/generic build seems like a good fit because it's
> > > supposed to work on a variety of systems. Expecting that native
> > > build from random VMs would work anywhere doesn't seen very
> > > realistic - it's been build for that VM environment (because it's a native
> build).
> > > >
> > > > Here's my understanding on which the current version is based:
> > > > 1. Since we want to get away from having statically defined
> > > > configuration,
> > > numa and core count discovery is exactly what we should have in the
> > > build system. Since discorery is currently the default for
> > > lib/drivers, it stands to reason it should be default for everything else, if
> possible.
> > > > 2. Native build should produce binaries matching the build machine
> > > > as well
> > > as possible.
> > > > 3. Default/generic build should produce binaries executable on a
> > > > range of
> > > systems (ideally all systems of a given architecture).
> > > > 4. Other builds, that is non-native builds, are cross-compilation,
> > > > since we're
> > > building for host machine other that the build machine.
> > > >
> > > > What I haven't taken into account is users using CI builds. That
> > > > could be
> > > remedied by modifying the CI a bit while being consistent with what
> > > native/default/generic/cross builds are (or should be). And in any
> > > case, if we're not interested in testing the exact CI environment
> > > (which we aren't, since we don't want to use 2 cores with 1 numa),
> > > we really shouldn't be doing native builds there.
> > > >
> > > > I'm interested in hearing where my thinking deviates from yours.
> > > >
> > >
> > > There are a number of points in which we differ, I think.
> > >
> > > Firstly, the use of "native" and "default/generic" for the "machine"
> > > parameter refers only to the instruction-set level from the
> > > compiler, and should not affect any other settings, since all settings are
> independent.
> > > Therefore, setting "machine" to "native" does not mean that we
> > > should detect cores and numa nodes, and similarly setting it to
> > > "default" does not mean that we should ignore the settings for these
> > > values and pick our own chosen default values.
> > Apologies to go to an older discussion.
> > I am trying to understand the definitions/expectations for 'native' and
> 'generic' builds.
> > As you say, instruction-set level is definitely one parameter.
> 
> Part of the confusion arises from the fact that originally that was the only
> parameter set by this - and on x86 it still is. Perhaps this parameter needs to
Just wondering, for x86, what does it mean if we set the max_num_cores and max_numa_nodes based on dynamic detection for 'native' build?
ISA still remains the same as before. But, the build might not work on machines with higher number of cores and numa nodes.
At the same time, the build also might not work on a machine with a different ISA. The users need to be aware that the target machine has the same ISA and same number of cores/numa nodes as the target machine.

> be renamed to "isa-level" or "architecture-flag" or similar to reflect its
> meaning. This would then allow a new "machine" setting, which can be
> considered separately. The question then is how much that helps with the
> main issue under discussion, that of cores and numa node values.
If we rename it, we will have backward compatibility issue (i.e. 'native' build on x86 will have different meaning and whoever wants the original meaning, have to change to using this new name). Not sure about the complexity in meson scripts.


> 
> > But, I think other DPDK specific parameters should also be considered.
> > For ex: RTE_MAX_LCORE should have a particular value for 'generic' build in
> all the supported architectures. The value could be different for each
> architecture, but it is fixed for the 'generic' build for a given architecture.
> Otherwise, the 'generic' build might not run on all the machines of that
> architecture.
> >
> > Similarly, for 'native' build, is there any reason not to include other DPDK
> parameters as part of the definition? IMO, 'native' should refer to the entire
> build machine, not just the ISA. i.e. build on the target machine.
> >
> 
> While I understand the idea here, it is somewhat complicated by the fact that
> the meson options given in "meson_options.txt" cannot be set by meson
> code, which means that when we change the machine flag to "native" we
> can only use or ignore the user-provided lcores and numa nodes setting - we
> have no way to change them and reflect those changes back to the user. :-(
> This leads to the situation in the discussion in this thread, where we start
> needing all sorts of magic values to indicate use of machine-type defaults or
> detected defaults.
I am wondering why we need to take the max_num_cores and max_numa_nodes from the user? This option was not provided in the make build system. I ask this question because for 'generic' this has to be a static/known configuration. For cross builds, this info can come (or derived) from the cross build file.
Was it supposed to be used in conjunction with 'native' build?

> 
> Regards,
> /Bruce

Re: [dpdk-dev] [RFC PATCH v3 3/6] build: automatic NUMA and cpu counts detection

[Bruce Richardson]

On Mon, Nov 02, 2020 at 07:01:44PM +0000, Honnappa Nagarahalli wrote:
> <snip>
> 
> > >
> > Part of the confusion arises from the fact that originally that was the only
> > parameter set by this - and on x86 it still is. Perhaps this parameter needs to
> Just wondering, for x86, what does it mean if we set the max_num_cores and max_numa_nodes based on dynamic detection for 'native' build?
> ISA still remains the same as before. But, the build might not work on machines with higher number of cores and numa nodes.
> At the same time, the build also might not work on a machine with a different ISA. The users need to be aware that the target machine has the same ISA and same number of cores/numa nodes as the target machine.
> 
Yes, that is a fair summary.

> > be renamed to "isa-level" or "architecture-flag" or similar to reflect its
> > meaning. This would then allow a new "machine" setting, which can be
> > considered separately. The question then is how much that helps with the
> > main issue under discussion, that of cores and numa node values.
> If we rename it, we will have backward compatibility issue (i.e. 'native' build on x86 will have different meaning and whoever wants the original meaning, have to change to using this new name). Not sure about the complexity in meson scripts.
> 

Yep, it was just a thought to see if it could help in this situation.

> 
> > 
> > > But, I think other DPDK specific parameters should also be considered.
> > > For ex: RTE_MAX_LCORE should have a particular value for 'generic' build in
> > all the supported architectures. The value could be different for each
> > architecture, but it is fixed for the 'generic' build for a given architecture.
> > Otherwise, the 'generic' build might not run on all the machines of that
> > architecture.
> > >
> > > Similarly, for 'native' build, is there any reason not to include other DPDK
> > parameters as part of the definition? IMO, 'native' should refer to the entire
> > build machine, not just the ISA. i.e. build on the target machine.
> > >
> > 
> > While I understand the idea here, it is somewhat complicated by the fact that
> > the meson options given in "meson_options.txt" cannot be set by meson
> > code, which means that when we change the machine flag to "native" we
> > can only use or ignore the user-provided lcores and numa nodes setting - we
> > have no way to change them and reflect those changes back to the user. :-(
> > This leads to the situation in the discussion in this thread, where we start
> > needing all sorts of magic values to indicate use of machine-type defaults or
> > detected defaults.
> I am wondering why we need to take the max_num_cores and max_numa_nodes from the user? This option was not provided in the make build system. I ask this question because for 'generic' this has to be a static/known configuration. For cross builds, this info can come (or derived) from the cross build file.
> Was it supposed to be used in conjunction with 'native' build?
> 

Well, it was configurable in the build config files same as all other DPDK
build settings with make. When working first on meson, I felt it was a
setting the user might be likely to want to tune, which is why I put it
into the meson_options.txt and nobody suggested otherwise on review [which
is the reason why many of the current options are the way they are :-)].

From my side, I have a couple of unknowns:
1. How big a difference in terms of memory use etc. of DPDK does it make by
   having really big values for these core/numa counts? If there is not much
   difference, then there is indeed little value in having them configurable
   at all, and we should just use big defaults and be done with it.
2. If there is a noticable difference in these settings, how many users are
   going to want to actually go to the trouble of tweaking these?
3. How big an effort is it to switch to having these settings made entirely
   dynamic at runtime? Doing so would naturally make the need for these
   settings completely go away.

With all that said, I'd be ok with a number of solutions. I'm ok to have
these dropped as meson options and just have them specified in other ways,
e.g. cross-file, or from meson.build files. [For x86, I'd tend towards
having them defined in rte_config.h inside x86-specific ifdefs].
Alternatively, I'm also happy enough with the proposed scheme here of
allowing user override, with platform defaults using "0"-value and
detection using "-1".

Regards,
/Bruce

Re: [dpdk-dev] [RFC PATCH v3 3/6] build: automatic NUMA and cpu counts detection

[Juraj Linkeš]

> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Tuesday, November 3, 2020 10:45 AM
> To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Ruifeng Wang
> <Ruifeng.Wang@arm.com>; Phil Yang <Phil.Yang@arm.com>;
> vcchunga@amazon.com; Dharmik Thakkar <Dharmik.Thakkar@arm.com>;
> jerinjacobk@gmail.com; hemant.agrawal@nxp.com; dev@dpdk.org; nd
> <nd@arm.com>
> Subject: Re: [RFC PATCH v3 3/6] build: automatic NUMA and cpu counts
> detection
> 
> On Mon, Nov 02, 2020 at 07:01:44PM +0000, Honnappa Nagarahalli wrote:
> > <snip>
> >
> > > >
> > > Part of the confusion arises from the fact that originally that was
> > > the only parameter set by this - and on x86 it still is. Perhaps
> > > this parameter needs to
> > Just wondering, for x86, what does it mean if we set the max_num_cores and
> max_numa_nodes based on dynamic detection for 'native' build?
> > ISA still remains the same as before. But, the build might not work on
> machines with higher number of cores and numa nodes.
> > At the same time, the build also might not work on a machine with a different
> ISA. The users need to be aware that the target machine has the same ISA and
> same number of cores/numa nodes as the target machine.
> >
> Yes, that is a fair summary.
> 

There's also additional confusion in what the build should produce with no user input. You mentioned that builds from CI are used outside of CI and when a user compiles up a binary without settings these [cpu and numa counts] explicitly it should run on 95%+ of systems of that type (why not 100%?). These requirements/expectations suggest that generic settings should be the default (machine=generic so that CI builds are usable on machines with different ISA and then using generic defaults for other settings (we're talking mainly about cpu and numa counts, but it should apply to everything - there are more settings that need generic defaults for Arm)).
If we change the default to generic, then setting machine=native would only set the machine args and not change the behavior of other vars (i.e. use defaults for the build machine).

> > > be renamed to "isa-level" or "architecture-flag" or similar to
> > > reflect its meaning. This would then allow a new "machine" setting,
> > > which can be considered separately. The question then is how much
> > > that helps with the main issue under discussion, that of cores and numa node
> values.
> > If we rename it, we will have backward compatibility issue (i.e. 'native' build on
> x86 will have different meaning and whoever wants the original meaning, have
> to change to using this new name). Not sure about the complexity in meson
> scripts.
> >
> 
> Yep, it was just a thought to see if it could help in this situation.
> 

I also don't think renaming or repurposing the machine meson option is the way to go, not only because of backwards compatibility but also because the option sets RTE_MACHINE and machine args, which lines up with the name nicely. As a side note, I grepped RTE_MACHINE and I don't see it being used anywhere - do we need that config?
But this doesn't work for Arm, since setting just machine args doesn't make sense - setting that also necessitates setting of some other dpdk config as well.

What could help is new meson option. I'm working on adding an option where the user could specify the set of target configuration, or soc configuration. I could add the configuration for generic build as one of the available sets and then we'd be able to use that variable instead of machine for Arm builds.

> >
> > >
> > > > But, I think other DPDK specific parameters should also be considered.
> > > > For ex: RTE_MAX_LCORE should have a particular value for 'generic'
> > > > build in
> > > all the supported architectures. The value could be different for
> > > each architecture, but it is fixed for the 'generic' build for a given
> architecture.
> > > Otherwise, the 'generic' build might not run on all the machines of
> > > that architecture.
> > > >
> > > > Similarly, for 'native' build, is there any reason not to include
> > > > other DPDK
> > > parameters as part of the definition? IMO, 'native' should refer to
> > > the entire build machine, not just the ISA. i.e. build on the target machine.
> > > >
> > >
> > > While I understand the idea here, it is somewhat complicated by the
> > > fact that the meson options given in "meson_options.txt" cannot be
> > > set by meson code, which means that when we change the machine flag
> > > to "native" we can only use or ignore the user-provided lcores and
> > > numa nodes setting - we have no way to change them and reflect those
> > > changes back to the user. :-( This leads to the situation in the
> > > discussion in this thread, where we start needing all sorts of magic
> > > values to indicate use of machine-type defaults or detected defaults.
> > I am wondering why we need to take the max_num_cores and
> max_numa_nodes from the user? This option was not provided in the make build
> system. I ask this question because for 'generic' this has to be a static/known
> configuration. For cross builds, this info can come (or derived) from the cross
> build file.
> > Was it supposed to be used in conjunction with 'native' build?
> >
> 
> Well, it was configurable in the build config files same as all other DPDK build
> settings with make. When working first on meson, I felt it was a setting the user
> might be likely to want to tune, which is why I put it into the meson_options.txt
> and nobody suggested otherwise on review [which is the reason why many of
> the current options are the way they are :-)].
> 
> From my side, I have a couple of unknowns:
> 1. How big a difference in terms of memory use etc. of DPDK does it make by
>    having really big values for these core/numa counts? If there is not much
>    difference, then there is indeed little value in having them configurable
>    at all, and we should just use big defaults and be done with it.
> 2. If there is a noticable difference in these settings, how many users are
>    going to want to actually go to the trouble of tweaking these?
> 3. How big an effort is it to switch to having these settings made entirely
>    dynamic at runtime? Doing so would naturally make the need for these
>    settings completely go away.
> 
> With all that said, I'd be ok with a number of solutions. I'm ok to have these
> dropped as meson options and just have them specified in other ways, e.g.
> cross-file, or from meson.build files. [For x86, I'd tend towards having them
> defined in rte_config.h inside x86-specific ifdefs].
> Alternatively, I'm also happy enough with the proposed scheme here of allowing
> user override, with platform defaults using "0"-value and detection using "-1".
> 
> Regards,
> /Bruce

Re: [dpdk-dev] [RFC PATCH v3 3/6] build: automatic NUMA and cpu counts detection

[Honnappa Nagarahalli]

<snip>

> >
> > > >
> > > Part of the confusion arises from the fact that originally that was
> > > the only parameter set by this - and on x86 it still is. Perhaps
> > > this parameter needs to
> > Just wondering, for x86, what does it mean if we set the max_num_cores
> and max_numa_nodes based on dynamic detection for 'native' build?
> > ISA still remains the same as before. But, the build might not work on
> machines with higher number of cores and numa nodes.
> > At the same time, the build also might not work on a machine with a
> different ISA. The users need to be aware that the target machine has the
> same ISA and same number of cores/numa nodes as the target machine.
> >
> Yes, that is a fair summary.
> 
> > > be renamed to "isa-level" or "architecture-flag" or similar to
> > > reflect its meaning. This would then allow a new "machine" setting,
> > > which can be considered separately. The question then is how much
> > > that helps with the main issue under discussion, that of cores and numa
> node values.
> > If we rename it, we will have backward compatibility issue (i.e. 'native' build
> on x86 will have different meaning and whoever wants the original meaning,
> have to change to using this new name). Not sure about the complexity in
> meson scripts.
> >
> 
> Yep, it was just a thought to see if it could help in this situation.
> 
> >
> > >
> > > > But, I think other DPDK specific parameters should also be considered.
> > > > For ex: RTE_MAX_LCORE should have a particular value for 'generic'
> > > > build in
> > > all the supported architectures. The value could be different for
> > > each architecture, but it is fixed for the 'generic' build for a given
> architecture.
> > > Otherwise, the 'generic' build might not run on all the machines of
> > > that architecture.
> > > >
> > > > Similarly, for 'native' build, is there any reason not to include
> > > > other DPDK
> > > parameters as part of the definition? IMO, 'native' should refer to
> > > the entire build machine, not just the ISA. i.e. build on the target machine.
> > > >
> > >
> > > While I understand the idea here, it is somewhat complicated by the
> > > fact that the meson options given in "meson_options.txt" cannot be
> > > set by meson code, which means that when we change the machine flag
> > > to "native" we can only use or ignore the user-provided lcores and
> > > numa nodes setting - we have no way to change them and reflect those
> > > changes back to the user. :-( This leads to the situation in the
> > > discussion in this thread, where we start needing all sorts of magic
> > > values to indicate use of machine-type defaults or detected defaults.
> > I am wondering why we need to take the max_num_cores and
> max_numa_nodes from the user? This option was not provided in the make
> build system. I ask this question because for 'generic' this has to be a
> static/known configuration. For cross builds, this info can come (or derived)
> from the cross build file.
> > Was it supposed to be used in conjunction with 'native' build?
> >
> 
> Well, it was configurable in the build config files same as all other DPDK build
> settings with make. When working first on meson, I felt it was a setting the
> user might be likely to want to tune, which is why I put it into the
> meson_options.txt and nobody suggested otherwise on review [which is the
> reason why many of the current options are the way they are :-)].
> 
> From my side, I have a couple of unknowns:
> 1. How big a difference in terms of memory use etc. of DPDK does it make by
>    having really big values for these core/numa counts? If there is not much
>    difference, then there is indeed little value in having them configurable
>    at all, and we should just use big defaults and be done with it.
> 2. If there is a noticable difference in these settings, how many users are
>    going to want to actually go to the trouble of tweaking these?
> 3. How big an effort is it to switch to having these settings made entirely
>    dynamic at runtime? Doing so would naturally make the need for these
>    settings completely go away.
A lot of the usage seems to be in the test code. Are these constants exposed to the users?

> 
> With all that said, I'd be ok with a number of solutions. I'm ok to have these
> dropped as meson options and just have them specified in other ways, e.g.
> cross-file, or from meson.build files. [For x86, I'd tend towards having them
> defined in rte_config.h inside x86-specific ifdefs].
> Alternatively, I'm also happy enough with the proposed scheme here of
> allowing user override, with platform defaults using "0"-value and detection
> using "-1".
Ok, we will stick to the methods in this patch.

> 
> Regards,
> /Bruce

[dpdk-dev] [RFC PATCH v3 4/6] build: move core and NUMA counts to cross files

[Juraj Linkeš]

The values of RTE_MAX_NUMA_NODES and RTE_MAX_LCORE are going to vary for
different build machines, even for the same Implementer ID and Part
Number combinations. Move the fixed values to cross files and use
automatic discovery/values from cmdline options for native build. Use
predefined values for generic-armv8 build, since that build is not tied
to the build machine.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/arm64_armada_linux_gcc    |  2 ++
 config/arm/arm64_armv8_linux_gcc     |  6 ++++
 config/arm/arm64_bluefield_linux_gcc |  2 ++
 config/arm/arm64_dpaa_linux_gcc      |  2 ++
 config/arm/arm64_emag_linux_gcc      |  2 ++
 config/arm/arm64_n1sdp_linux_gcc     |  2 ++
 config/arm/arm64_octeontx2_linux_gcc |  2 ++
 config/arm/arm64_stingray_linux_gcc  |  2 ++
 config/arm/arm64_thunderx2_linux_gcc |  2 ++
 config/arm/arm64_thunderx_linux_gcc  |  2 ++
 config/arm/meson.build               | 43 +++++++++++++++++-----------
 11 files changed, 51 insertions(+), 16 deletions(-)

diff --git a/config/arm/arm64_armada_linux_gcc b/config/arm/arm64_armada_linux_gcc
index 52c5f4476..36100c594 100644
--- a/config/arm/arm64_armada_linux_gcc
+++ b/config/arm/arm64_armada_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 
 [properties]
 implementer_id = '0x56'
+max_numa_nodes = 1
+max_lcores = 16
diff --git a/config/arm/arm64_armv8_linux_gcc b/config/arm/arm64_armv8_linux_gcc
index 13ee8b223..779333199 100644
--- a/config/arm/arm64_armv8_linux_gcc
+++ b/config/arm/arm64_armv8_linux_gcc
@@ -26,3 +26,9 @@ implementer_id = 'generic'
 # '0xd0a':   cortex-a75
 # '0xd0b':   cortex-a76
 part_number = 'generic'
+
+# Supported extra configuration
+# max_numa_nodes = n  # will set RTE_MAX_NUMA_NODES
+# max_lcores = n      # will set RTE_MAX_LCORE
+max_lcores = 256
+max_numa_nodes = 4
diff --git a/config/arm/arm64_bluefield_linux_gcc b/config/arm/arm64_bluefield_linux_gcc
index b79389d85..7ff6fd309 100644
--- a/config/arm/arm64_bluefield_linux_gcc
+++ b/config/arm/arm64_bluefield_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 [properties]
 implementer_id = '0x41'
 part_number = '0xd08'
+max_numa_nodes = 1
+max_lcores = 16
diff --git a/config/arm/arm64_dpaa_linux_gcc b/config/arm/arm64_dpaa_linux_gcc
index 573ae7e42..adae86f57 100644
--- a/config/arm/arm64_dpaa_linux_gcc
+++ b/config/arm/arm64_dpaa_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 
 [properties]
 implementer_id = 'dpaa'
+max_numa_nodes = 1
+max_lcores = 16
diff --git a/config/arm/arm64_emag_linux_gcc b/config/arm/arm64_emag_linux_gcc
index 24f3d533e..3cccca901 100644
--- a/config/arm/arm64_emag_linux_gcc
+++ b/config/arm/arm64_emag_linux_gcc
@@ -14,3 +14,5 @@ endian = 'little'
 
 [properties]
 implementer_id = '0x50'
+max_numa_nodes = 1
+max_lcores = 32
diff --git a/config/arm/arm64_n1sdp_linux_gcc b/config/arm/arm64_n1sdp_linux_gcc
index 6fb3f02ea..3a8b46812 100644
--- a/config/arm/arm64_n1sdp_linux_gcc
+++ b/config/arm/arm64_n1sdp_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 [properties]
 implementer_id = '0x41'
 part_number = '0xd0c'
+max_numa_nodes = 1
+max_lcores = 4
diff --git a/config/arm/arm64_octeontx2_linux_gcc b/config/arm/arm64_octeontx2_linux_gcc
index ac1042806..2baf95a2a 100644
--- a/config/arm/arm64_octeontx2_linux_gcc
+++ b/config/arm/arm64_octeontx2_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 [properties]
 implementer_id = '0x43'
 part_number = '0xb2'
+max_numa_nodes = 1
+max_lcores = 36
diff --git a/config/arm/arm64_stingray_linux_gcc b/config/arm/arm64_stingray_linux_gcc
index b79389d85..7ff6fd309 100644
--- a/config/arm/arm64_stingray_linux_gcc
+++ b/config/arm/arm64_stingray_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 [properties]
 implementer_id = '0x41'
 part_number = '0xd08'
+max_numa_nodes = 1
+max_lcores = 16
diff --git a/config/arm/arm64_thunderx2_linux_gcc b/config/arm/arm64_thunderx2_linux_gcc
index dd257745e..07440fe5f 100644
--- a/config/arm/arm64_thunderx2_linux_gcc
+++ b/config/arm/arm64_thunderx2_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 [properties]
 implementer_id = '0x43'
 part_number = '0xaf'
+max_numa_nodes = 2
+max_lcores = 256
diff --git a/config/arm/arm64_thunderx_linux_gcc b/config/arm/arm64_thunderx_linux_gcc
index 670764437..16ba78fd6 100644
--- a/config/arm/arm64_thunderx_linux_gcc
+++ b/config/arm/arm64_thunderx_linux_gcc
@@ -14,3 +14,5 @@ endian = 'little'
 
 [properties]
 implementer_id = '0x43'
+max_numa_nodes = 1
+max_lcores = 96
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 66b08bfb5..5605e807b 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -33,44 +33,40 @@ flags_common_default = [
 	['RTE_ARCH_ARM64', true]
 ]
 
+# add these to defaults when machine='generic-armv8'
+flags_force_generic = [
+	['RTE_MAX_NUMA_NODES', 4],
+	['RTE_MAX_LCORE', 256]
+]
+
 # implementer specific aarch64 flags, with middle priority
 # (will overwrite common flags)
 flags_generic = [
 	['RTE_MACHINE', '"armv8a"'],
-	['RTE_MAX_LCORE', 256],
 	['RTE_USE_C11_MEM_MODEL', true],
 	['RTE_CACHE_LINE_SIZE', 128]
 ]
 flags_arm = [
 	['RTE_MACHINE', '"armv8a"'],
-	['RTE_MAX_LCORE', 16],
 	['RTE_USE_C11_MEM_MODEL', true],
 	['RTE_CACHE_LINE_SIZE', 64]
 ]
 flags_cavium = [
 	['RTE_CACHE_LINE_SIZE', 128],
-	['RTE_MAX_NUMA_NODES', 2],
-	['RTE_MAX_LCORE', 96],
 	['RTE_MAX_VFIO_GROUPS', 128]
 ]
 flags_dpaa = [
 	['RTE_MACHINE', '"dpaa"'],
 	['RTE_USE_C11_MEM_MODEL', true],
 	['RTE_CACHE_LINE_SIZE', 64],
-	['RTE_MAX_NUMA_NODES', 1],
-	['RTE_MAX_LCORE', 16],
 	['RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', false]
 ]
 flags_emag = [
 	['RTE_MACHINE', '"emag"'],
-	['RTE_MAX_NUMA_NODES', 1],
-	['RTE_MAX_LCORE', 32],
 	['RTE_CACHE_LINE_SIZE', 64]
 ]
 flags_armada = [
 	['RTE_MACHINE', '"armv8a"'],
-	['RTE_MAX_NUMA_NODES', 1],
-	['RTE_MAX_LCORE', 16],
 	['RTE_CACHE_LINE_SIZE', 64]
 ]
 
@@ -78,8 +74,6 @@ flags_armada = [
 # (will overwrite both common and implementer specific flags)
 flags_n1sdp_extra = [
 	['RTE_MACHINE', '"n1sdp"'],
-	['RTE_MAX_NUMA_NODES', 1],
-	['RTE_MAX_LCORE', 4],
 	['RTE_EAL_NUMA_AWARE_HUGEPAGES', false],
 	['RTE_LIBRTE_VHOST_NUMA', false]
 ]
@@ -90,15 +84,11 @@ flags_thunderx_extra = [
 flags_thunderx2_extra = [
 	['RTE_MACHINE', '"thunderx2"'],
 	['RTE_CACHE_LINE_SIZE', 64],
-	['RTE_MAX_NUMA_NODES', 2],
-	['RTE_MAX_LCORE', 256],
 	['RTE_ARM_FEATURE_ATOMICS', true],
 	['RTE_USE_C11_MEM_MODEL', true]
 ]
 flags_octeontx2_extra = [
 	['RTE_MACHINE', '"octeontx2"'],
-	['RTE_MAX_NUMA_NODES', 1],
-	['RTE_MAX_LCORE', 36],
 	['RTE_ARM_FEATURE_ATOMICS', true],
 	['RTE_EAL_IGB_UIO', false],
 	['RTE_USE_C11_MEM_MODEL', true]
@@ -160,6 +150,7 @@ else
 			# default build
 			impl_config = impl_generic
 			part_number = 'generic'
+			flags_common_default += flags_force_generic
 		else
 			# native build
 			# The script returns ['Implementer', 'Variant', 'Architecture',
@@ -214,6 +205,26 @@ else
 		dpdk_flags += part_number_config[1]
 	endif
 
+	# apply cross-specific options
+	if meson.is_cross_build()
+		# configure RTE_MAX_NUMA_NODES and RTE_MAX_LCORE from cross file
+		# RFC: do we want defaults for these two? or should we require
+		# that all cross file define them?
+		cross_max_numa_nodes = meson.get_cross_property('max_numa_nodes', 0)
+		if cross_max_numa_nodes != 0
+			dpdk_flags += [
+				['RTE_MAX_NUMA_NODES', cross_max_numa_nodes]
+			]
+		endif
+		cross_max_lcores = meson.get_cross_property('max_lcores', 0)
+		if cross_max_lcores != 0
+			message('Setting RTE_MAX_LCORE from cross file')
+			dpdk_flags += [
+				['RTE_MAX_LCORE', cross_max_lcores]
+			]
+		endif
+	endif
+
 	machine_args = [] # Clear previous machine args
 	foreach flag: part_number_config[0]
 		if cc.has_argument(flag)
-- 
2.20.1

[dpdk-dev] [RFC PATCH v3 5/6] build: disable Arm drivers

[Juraj Linkeš]

A few options that disabled drivers in the old makefiles were improperly
ported to the meson build system. Fix this by adding a to the list of
disabled drivers, similarly how the command line option works.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
 config/arm/meson.build | 9 +++++----
 drivers/meson.build    | 2 +-
 meson.build            | 1 +
 3 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/config/arm/meson.build b/config/arm/meson.build
index 5605e807b..87a208c97 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -8,6 +8,11 @@
 arm_force_native_march = false
 native_machine_args = ['-march=native', '-mtune=native']
 
+# RFC: do we want these drivers to be disabled by default on arm?
+# The previous way to disable these drivers never worked
+# RTE_NET_AVP and the rest would be overwritten to true if the necessary deps were found
+disabled_drivers += ['net/avp', 'net/fm10k', 'net/sfc']
+
 # common flags to all aarch64 builds, with lowest priority
 flags_common_default = [
 	# Accelarate rte_memcpy. Be sure to run unit test (memcpy_perf_autotest)
@@ -23,10 +28,6 @@ flags_common_default = [
 	#	['RTE_ARM64_MEMCPY_ALIGN_MASK', 0xF],
 	#	['RTE_ARM64_MEMCPY_STRICT_ALIGN', false],
 
-	['RTE_NET_FM10K', false],
-	['RTE_NET_SFC_EFX', false],
-	['RTE_NET_AVP', false],
-
 	['RTE_SCHED_VECTOR', false],
 	['RTE_ARM_USE_WFE', false],
 	['RTE_CACHE_LINE_SIZE', 128],
diff --git a/drivers/meson.build b/drivers/meson.build
index a5a6fed06..a41ff909e 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -18,7 +18,7 @@ subdirs = [
 	'baseband', # depends on common and bus.
 ]
 
-disabled_drivers = run_command(list_dir_globs, get_option('disable_drivers'),
+disabled_drivers += run_command(list_dir_globs, get_option('disable_drivers'),
 		).stdout().split()
 
 default_cflags = machine_args
diff --git a/meson.build b/meson.build
index 61d9a4f5f..8dadd70dc 100644
--- a/meson.build
+++ b/meson.build
@@ -21,6 +21,7 @@ dpdk_drivers = []
 dpdk_extra_ldflags = []
 dpdk_libs_disabled = []
 dpdk_drvs_disabled = []
+disabled_drivers = []
 abi_version_file = files('ABI_VERSION')
 
 if host_machine.cpu_family().startswith('x86')
-- 
2.20.1

[dpdk-dev] [RFC PATCH v3 6/6] build: update Arm builds with makefile flags

[Juraj Linkeš]

Not all flags were moved properly from the old Makefile build system.
Disable corresponding drivers and libnuma in cross files and remove
deprecated config.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/arm64_armada_linux_gcc    | 2 ++
 config/arm/arm64_armv8_linux_gcc     | 8 ++++++--
 config/arm/arm64_bluefield_linux_gcc | 1 +
 config/arm/arm64_dpaa_linux_gcc      | 1 +
 config/arm/arm64_n1sdp_linux_gcc     | 1 +
 config/arm/arm64_octeontx2_linux_gcc | 1 +
 config/arm/arm64_stingray_linux_gcc  | 1 +
 config/arm/meson.build               | 5 ++++-
 drivers/meson.build                  | 4 ++++
 9 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/config/arm/arm64_armada_linux_gcc b/config/arm/arm64_armada_linux_gcc
index 36100c594..e33f1e742 100644
--- a/config/arm/arm64_armada_linux_gcc
+++ b/config/arm/arm64_armada_linux_gcc
@@ -17,3 +17,5 @@ endian = 'little'
 implementer_id = '0x56'
 max_numa_nodes = 1
 max_lcores = 16
+has_libnuma = 0
+disabled_drivers = ['bus/dpaa', 'bus/fslmc', 'common/dpaax']
diff --git a/config/arm/arm64_armv8_linux_gcc b/config/arm/arm64_armv8_linux_gcc
index 779333199..c6010e303 100644
--- a/config/arm/arm64_armv8_linux_gcc
+++ b/config/arm/arm64_armv8_linux_gcc
@@ -28,7 +28,11 @@ implementer_id = 'generic'
 part_number = 'generic'
 
 # Supported extra configuration
-# max_numa_nodes = n  # will set RTE_MAX_NUMA_NODES
-# max_lcores = n      # will set RTE_MAX_LCORE
+# max_numa_nodes = n   # will set RTE_MAX_NUMA_NODES
+# max_lcores = n        # will set RTE_MAX_LCORE
 max_lcores = 256
 max_numa_nodes = 4
+
+# has_libnuma = 0      # set to 0 if the SoC is not a NUMA system
+# disabled_drivers += ['bus/dpaa', 'crypto']
+    # add to the set of disabled libraries
diff --git a/config/arm/arm64_bluefield_linux_gcc b/config/arm/arm64_bluefield_linux_gcc
index 7ff6fd309..e154ac4fd 100644
--- a/config/arm/arm64_bluefield_linux_gcc
+++ b/config/arm/arm64_bluefield_linux_gcc
@@ -17,3 +17,4 @@ implementer_id = '0x41'
 part_number = '0xd08'
 max_numa_nodes = 1
 max_lcores = 16
+has_libnuma = 0
diff --git a/config/arm/arm64_dpaa_linux_gcc b/config/arm/arm64_dpaa_linux_gcc
index adae86f57..78883a2ec 100644
--- a/config/arm/arm64_dpaa_linux_gcc
+++ b/config/arm/arm64_dpaa_linux_gcc
@@ -17,3 +17,4 @@ endian = 'little'
 implementer_id = 'dpaa'
 max_numa_nodes = 1
 max_lcores = 16
+has_libnuma = 0
diff --git a/config/arm/arm64_n1sdp_linux_gcc b/config/arm/arm64_n1sdp_linux_gcc
index 3a8b46812..4493227f3 100644
--- a/config/arm/arm64_n1sdp_linux_gcc
+++ b/config/arm/arm64_n1sdp_linux_gcc
@@ -17,3 +17,4 @@ implementer_id = '0x41'
 part_number = '0xd0c'
 max_numa_nodes = 1
 max_lcores = 4
+has_libnuma = 0
diff --git a/config/arm/arm64_octeontx2_linux_gcc b/config/arm/arm64_octeontx2_linux_gcc
index 2baf95a2a..ff9e5c403 100644
--- a/config/arm/arm64_octeontx2_linux_gcc
+++ b/config/arm/arm64_octeontx2_linux_gcc
@@ -17,3 +17,4 @@ implementer_id = '0x43'
 part_number = '0xb2'
 max_numa_nodes = 1
 max_lcores = 36
+has_libnuma = 0
diff --git a/config/arm/arm64_stingray_linux_gcc b/config/arm/arm64_stingray_linux_gcc
index 7ff6fd309..e154ac4fd 100644
--- a/config/arm/arm64_stingray_linux_gcc
+++ b/config/arm/arm64_stingray_linux_gcc
@@ -17,3 +17,4 @@ implementer_id = '0x41'
 part_number = '0xd08'
 max_numa_nodes = 1
 max_lcores = 16
+has_libnuma = 0
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 87a208c97..0eae3c3b7 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -91,7 +91,6 @@ flags_thunderx2_extra = [
 flags_octeontx2_extra = [
 	['RTE_MACHINE', '"octeontx2"'],
 	['RTE_ARM_FEATURE_ATOMICS', true],
-	['RTE_EAL_IGB_UIO', false],
 	['RTE_USE_C11_MEM_MODEL', true]
 ]
 # arm config (implementer 0x41) is the default config
@@ -224,6 +223,10 @@ else
 				['RTE_MAX_LCORE', cross_max_lcores]
 			]
 		endif
+		cross_has_libnuma = meson.get_cross_property('has_libnuma', '')
+		if cross_has_libnuma != ''
+			has_libnuma = cross_has_libnuma
+		endif
 	endif
 
 	machine_args = [] # Clear previous machine args
diff --git a/drivers/meson.build b/drivers/meson.build
index a41ff909e..4bdc3d5c9 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -21,6 +21,10 @@ subdirs = [
 disabled_drivers += run_command(list_dir_globs, get_option('disable_drivers'),
 		).stdout().split()
 
+if meson.is_cross_build()
+	disabled_drivers += meson.get_cross_property('disabled_drivers', [])
+endif
+
 default_cflags = machine_args
 default_cflags += ['-DALLOW_EXPERIMENTAL_API']
 default_cflags += ['-DALLOW_INTERNAL_API']
-- 
2.20.1

[dpdk-dev] [PATCH v4 0/6] Arm build options rework

[Juraj Linkeš]

The current way of specifying Arm configuration options is insufficient
since we can't identify the SoC we're building for from the MIDR
information. For example, we can't distinguish between N1SDP, Graviton2
or Ampere Altra.

Add a way to specify the cpu count and numa node count for cross builds.

We also want to be able to disable which drivers (and possibly
libraries) are built without user input. This is useful when building:
1. on an SoC that is slow and we want to build only what is necessary
  without the user having to check which libraries they have installed
2. a cross build on a fast aarch64 machine but with target SoC which
  differs in capabilities or libraries.
This is achieved by specifying the drivers in cross files.

Among libraries, only libnuma can be now disabled.

Also add an optional way to discover cpu count a numa node count. Fix
-Dmax_lcores and -Dmax_numa_nodes for arm builds.

v2:
Major rework of the whole series.

v3:
Added numa and core count defaults for x86 default build.
Removed numa and core count defaults. Now requiring defaults to be
specified in a cross file or on the cmdline.
Added FreeBDS support for numa count discovery.

v4:
Make automatic numa and cpu counts discovery optional.

Juraj Linkeš (6):
  build: alias default build as generic
  build: refactor Arm build
  build: optional NUMA and cpu counts detection
  build: add core and NUMA counts to cross files
  build: disable Arm drivers
  build: update Arm builds with makefile flags

 buildtools/get_cpu_count.py          |   7 +
 buildtools/get_numa_count.py         |  22 +++
 buildtools/meson.build               |   2 +
 config/arm/arm64_armada_linux_gcc    |   6 +-
 config/arm/arm64_armv8_linux_gcc     |  18 +-
 config/arm/arm64_bluefield_linux_gcc |   7 +-
 config/arm/arm64_dpaa_linux_gcc      |   5 +-
 config/arm/arm64_emag_linux_gcc      |   4 +-
 config/arm/arm64_n1sdp_linux_gcc     |   7 +-
 config/arm/arm64_octeontx2_linux_gcc |   7 +-
 config/arm/arm64_stingray_linux_gcc  |   7 +-
 config/arm/arm64_thunderx2_linux_gcc |   6 +-
 config/arm/arm64_thunderx_linux_gcc  |   4 +-
 config/arm/meson.build               | 278 ++++++++++++++++-----------
 config/meson.build                   |  54 +++++-
 drivers/meson.build                  |   6 +-
 meson.build                          |   1 +
 meson_options.txt                    |   8 +-
 18 files changed, 306 insertions(+), 143 deletions(-)
 create mode 100644 buildtools/get_cpu_count.py
 create mode 100644 buildtools/get_numa_count.py

-- 
2.20.1

[dpdk-dev] [PATCH v4 1/6] build: alias default build as generic

[Juraj Linkeš]

The current machine='default' build name is not descriptive. The actual
default build is machine='native'. Add an alternative string which does
the same build and better describes what we're building:
machine='generic'. Leave machine='default' for backwards compatibility.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/meson.build | 31 ++++++++++++++++---------------
 config/meson.build     |  6 +++---
 2 files changed, 19 insertions(+), 18 deletions(-)

diff --git a/config/arm/meson.build b/config/arm/meson.build
index b49203fa8..491842cad 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -1,12 +1,13 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation.
 # Copyright(c) 2017 Cavium, Inc
+# Copyright(c) 2020 PANTHEON.tech s.r.o.
 
 # for checking defines we need to use the correct compiler flags
 march_opt = '-march=@0@'.format(machine)
 
 arm_force_native_march = false
-arm_force_default_march = (machine == 'default')
+arm_force_generic_march = (machine == 'generic')
 
 flags_common_default = [
 	# Accelarate rte_memcpy. Be sure to run unit test (memcpy_perf_autotest)
@@ -87,7 +88,7 @@ flags_octeontx2_extra = [
 	['RTE_EAL_IGB_UIO', false],
 	['RTE_USE_C11_MEM_MODEL', true]]
 
-machine_args_generic = [
+machine_args_default = [
 	['default', ['-march=armv8-a+crc', '-moutline-atomics']],
 	['native', ['-march=native']],
 	['0xd03', ['-mcpu=cortex-a53']],
@@ -113,20 +114,20 @@ machine_args_emag = [
 	['native', ['-march=native']]]
 
 ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page G7-5321)
-impl_generic = ['Generic armv8', flags_generic, machine_args_generic]
-impl_0x41 = ['Arm', flags_arm, machine_args_generic]
-impl_0x42 = ['Broadcom', flags_generic, machine_args_generic]
+impl_generic = ['Generic armv8', flags_generic, machine_args_default]
+impl_0x41 = ['Arm', flags_arm, machine_args_default]
+impl_0x42 = ['Broadcom', flags_generic, machine_args_default]
 impl_0x43 = ['Cavium', flags_cavium, machine_args_cavium]
-impl_0x44 = ['DEC', flags_generic, machine_args_generic]
-impl_0x49 = ['Infineon', flags_generic, machine_args_generic]
-impl_0x4d = ['Motorola', flags_generic, machine_args_generic]
-impl_0x4e = ['NVIDIA', flags_generic, machine_args_generic]
+impl_0x44 = ['DEC', flags_generic, machine_args_default]
+impl_0x49 = ['Infineon', flags_generic, machine_args_default]
+impl_0x4d = ['Motorola', flags_generic, machine_args_default]
+impl_0x4e = ['NVIDIA', flags_generic, machine_args_default]
 impl_0x50 = ['Ampere Computing', flags_emag, machine_args_emag]
-impl_0x51 = ['Qualcomm', flags_generic, machine_args_generic]
-impl_0x53 = ['Samsung', flags_generic, machine_args_generic]
-impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_generic]
-impl_0x69 = ['Intel', flags_generic, machine_args_generic]
-impl_dpaa = ['NXP DPAA', flags_dpaa, machine_args_generic]
+impl_0x51 = ['Qualcomm', flags_generic, machine_args_default]
+impl_0x53 = ['Samsung', flags_generic, machine_args_default]
+impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_default]
+impl_0x69 = ['Intel', flags_generic, machine_args_default]
+impl_dpaa = ['NXP DPAA', flags_dpaa, machine_args_default]
 
 dpdk_conf.set('RTE_ARCH_ARM', 1)
 dpdk_conf.set('RTE_FORCE_INTRINSICS', 1)
@@ -145,7 +146,7 @@ else
 	cmd_generic = ['generic', '', '', 'default', '']
 	cmd_output = cmd_generic # Set generic by default
 	machine_args = [] # Clear previous machine args
-	if arm_force_default_march and not meson.is_cross_build()
+	if arm_force_generic_march and not meson.is_cross_build()
 		machine = impl_generic
 		impl_pn = 'default'
 	elif not meson.is_cross_build()
diff --git a/config/meson.build b/config/meson.build
index 258b01d06..918ca2dfb 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -68,13 +68,13 @@ else
 	machine = get_option('machine')
 endif
 
-# machine type 'default' is special, it defaults to the per arch agreed common
+# machine type 'generic' is special, it defaults to the per arch agreed common
 # minimal baseline needed for DPDK.
 # That might not be the most optimized, but the most portable version while
 # still being able to support the CPU features required for DPDK.
 # This can be bumped up by the DPDK project, but it can never be an
 # invariant like 'native'
-if machine == 'default'
+if machine == 'default' or machine == 'generic'
 	if host_machine.cpu_family().startswith('x86')
 		# matches the old pre-meson build systems default
 		machine = 'corei7'
@@ -82,7 +82,7 @@ if machine == 'default'
 		machine = 'armv7-a'
 	elif host_machine.cpu_family().startswith('aarch')
 		# arm64 manages defaults in config/arm/meson.build
-		machine = 'default'
+		machine = 'generic'
 	elif host_machine.cpu_family().startswith('ppc')
 		machine = 'power8'
 	endif
-- 
2.20.1

Re: [dpdk-dev] [PATCH v4 1/6] build: alias default build as generic

[Honnappa Nagarahalli]

<snip>

> 
> The current machine='default' build name is not descriptive. The actual
> default build is machine='native'. Add an alternative string which does the
> same build and better describes what we're building:
> machine='generic'. Leave machine='default' for backwards compatibility.
> 
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
>  config/arm/meson.build | 31 ++++++++++++++++---------------
>  config/meson.build     |  6 +++---
>  2 files changed, 19 insertions(+), 18 deletions(-)
> 
> diff --git a/config/arm/meson.build b/config/arm/meson.build index
> b49203fa8..491842cad 100644
> --- a/config/arm/meson.build
> +++ b/config/arm/meson.build
> @@ -1,12 +1,13 @@
>  # SPDX-License-Identifier: BSD-3-Clause  # Copyright(c) 2017 Intel
> Corporation.
>  # Copyright(c) 2017 Cavium, Inc
> +# Copyright(c) 2020 PANTHEON.tech s.r.o.
> 
>  # for checking defines we need to use the correct compiler flags  march_opt
> = '-march=@0@'.format(machine)
> 
>  arm_force_native_march = false
> -arm_force_default_march = (machine == 'default')
> +arm_force_generic_march = (machine == 'generic')
> 
>  flags_common_default = [
>  	# Accelarate rte_memcpy. Be sure to run unit test
> (memcpy_perf_autotest) @@ -87,7 +88,7 @@ flags_octeontx2_extra = [
>  	['RTE_EAL_IGB_UIO', false],
>  	['RTE_USE_C11_MEM_MODEL', true]]
> 
> -machine_args_generic = [
> +machine_args_default = [
Is this change required? 'machine_args_generic' aligns much better with 'machine == generic' (change in the file config/meson.build below)

>  	['default', ['-march=armv8-a+crc', '-moutline-atomics']],
>  	['native', ['-march=native']],
>  	['0xd03', ['-mcpu=cortex-a53']],
> @@ -113,20 +114,20 @@ machine_args_emag = [
>  	['native', ['-march=native']]]
> 
>  ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page G7-5321)
> -impl_generic = ['Generic armv8', flags_generic, machine_args_generic]
> -impl_0x41 = ['Arm', flags_arm, machine_args_generic]
> -impl_0x42 = ['Broadcom', flags_generic, machine_args_generic]
> +impl_generic = ['Generic armv8', flags_generic, machine_args_default]
> +impl_0x41 = ['Arm', flags_arm, machine_args_default]
> +impl_0x42 = ['Broadcom', flags_generic, machine_args_default]
>  impl_0x43 = ['Cavium', flags_cavium, machine_args_cavium]
> -impl_0x44 = ['DEC', flags_generic, machine_args_generic]
> -impl_0x49 = ['Infineon', flags_generic, machine_args_generic] -impl_0x4d =
> ['Motorola', flags_generic, machine_args_generic] -impl_0x4e = ['NVIDIA',
> flags_generic, machine_args_generic]
> +impl_0x44 = ['DEC', flags_generic, machine_args_default]
> +impl_0x49 = ['Infineon', flags_generic, machine_args_default] impl_0x4d
> += ['Motorola', flags_generic, machine_args_default] impl_0x4e =
> +['NVIDIA', flags_generic, machine_args_default]
>  impl_0x50 = ['Ampere Computing', flags_emag, machine_args_emag]
> -impl_0x51 = ['Qualcomm', flags_generic, machine_args_generic]
> -impl_0x53 = ['Samsung', flags_generic, machine_args_generic]
> -impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_generic]
> -impl_0x69 = ['Intel', flags_generic, machine_args_generic] -impl_dpaa =
> ['NXP DPAA', flags_dpaa, machine_args_generic]
> +impl_0x51 = ['Qualcomm', flags_generic, machine_args_default]
> +impl_0x53 = ['Samsung', flags_generic, machine_args_default]
> +impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_default]
> +impl_0x69 = ['Intel', flags_generic, machine_args_default] impl_dpaa =
> +['NXP DPAA', flags_dpaa, machine_args_default]
> 
>  dpdk_conf.set('RTE_ARCH_ARM', 1)
>  dpdk_conf.set('RTE_FORCE_INTRINSICS', 1) @@ -145,7 +146,7 @@ else
>  	cmd_generic = ['generic', '', '', 'default', '']
>  	cmd_output = cmd_generic # Set generic by default
>  	machine_args = [] # Clear previous machine args
> -	if arm_force_default_march and not meson.is_cross_build()
> +	if arm_force_generic_march and not meson.is_cross_build()
>  		machine = impl_generic
>  		impl_pn = 'default'
>  	elif not meson.is_cross_build()
> diff --git a/config/meson.build b/config/meson.build index
> 258b01d06..918ca2dfb 100644
> --- a/config/meson.build
> +++ b/config/meson.build
> @@ -68,13 +68,13 @@ else
>  	machine = get_option('machine')
>  endif
> 
> -# machine type 'default' is special, it defaults to the per arch agreed
> common
> +# machine type 'generic' is special, it defaults to the per arch agreed
> +common
>  # minimal baseline needed for DPDK.
>  # That might not be the most optimized, but the most portable version while
> # still being able to support the CPU features required for DPDK.
>  # This can be bumped up by the DPDK project, but it can never be an  #
> invariant like 'native'
Nit, may be it is worth mentioning that 'default' is allowed for backward compatibility?

> -if machine == 'default'
> +if machine == 'default' or machine == 'generic'
>  	if host_machine.cpu_family().startswith('x86')
>  		# matches the old pre-meson build systems default
>  		machine = 'corei7'
> @@ -82,7 +82,7 @@ if machine == 'default'
>  		machine = 'armv7-a'
>  	elif host_machine.cpu_family().startswith('aarch')
>  		# arm64 manages defaults in config/arm/meson.build
> -		machine = 'default'
> +		machine = 'generic'
>  	elif host_machine.cpu_family().startswith('ppc')
>  		machine = 'power8'
>  	endif
> --
> 2.20.1

Re: [dpdk-dev] [PATCH v4 1/6] build: alias default build as generic

[Juraj Linkeš]

> -----Original Message-----
> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Sent: Tuesday, October 27, 2020 4:54 AM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>; bruce.richardson@intel.com;
> Ruifeng Wang <Ruifeng.Wang@arm.com>; Phil Yang <Phil.Yang@arm.com>;
> vcchunga@amazon.com; Dharmik Thakkar <Dharmik.Thakkar@arm.com>;
> jerinjacobk@gmail.com; hemant.agrawal@nxp.com
> Cc: dev@dpdk.org; nd <nd@arm.com>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>
> Subject: RE: [PATCH v4 1/6] build: alias default build as generic
> 
> <snip>
> 
> >
> > The current machine='default' build name is not descriptive. The
> > actual default build is machine='native'. Add an alternative string
> > which does the same build and better describes what we're building:
> > machine='generic'. Leave machine='default' for backwards compatibility.
> >
> > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > ---
> >  config/arm/meson.build | 31 ++++++++++++++++---------------
> >  config/meson.build     |  6 +++---
> >  2 files changed, 19 insertions(+), 18 deletions(-)
> >
> > diff --git a/config/arm/meson.build b/config/arm/meson.build index
> > b49203fa8..491842cad 100644
> > --- a/config/arm/meson.build
> > +++ b/config/arm/meson.build
> > @@ -1,12 +1,13 @@
> >  # SPDX-License-Identifier: BSD-3-Clause  # Copyright(c) 2017 Intel
> > Corporation.
> >  # Copyright(c) 2017 Cavium, Inc
> > +# Copyright(c) 2020 PANTHEON.tech s.r.o.
> >
> >  # for checking defines we need to use the correct compiler flags
> > march_opt = '-march=@0@'.format(machine)
> >
> >  arm_force_native_march = false
> > -arm_force_default_march = (machine == 'default')
> > +arm_force_generic_march = (machine == 'generic')
> >
> >  flags_common_default = [
> >  	# Accelarate rte_memcpy. Be sure to run unit test
> > (memcpy_perf_autotest) @@ -87,7 +88,7 @@ flags_octeontx2_extra = [
> >  	['RTE_EAL_IGB_UIO', false],
> >  	['RTE_USE_C11_MEM_MODEL', true]]
> >
> > -machine_args_generic = [
> > +machine_args_default = [
> Is this change required? 'machine_args_generic' aligns much better with
> 'machine == generic' (change in the file config/meson.build below)
> 

I thought a lot about the naming and I went back and forth on this before eventually settling on default instead of generic. My reasons are because machine_args_default doesn't just hold information about the build we're doing when machine='generic', but also non-generic information (and thus generic doesn't that well) and it is also used like you would use a default configuration - for implementers which don't have specific configuration.

In a later patch I renamed the "default" configuration of these machine_args to "generic" which is where the actual generic part of the configuration lies. Maybe I could've done that in this patch. Or maybe move this change to that patch - this seems better, since this change is not really needed for this patch and is more of a refactor.

> >  	['default', ['-march=armv8-a+crc', '-moutline-atomics']],
> >  	['native', ['-march=native']],
> >  	['0xd03', ['-mcpu=cortex-a53']],
> > @@ -113,20 +114,20 @@ machine_args_emag = [
> >  	['native', ['-march=native']]]
> >
> >  ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page
> > G7-5321) -impl_generic = ['Generic armv8', flags_generic,
> > machine_args_generic]
> > -impl_0x41 = ['Arm', flags_arm, machine_args_generic]
> > -impl_0x42 = ['Broadcom', flags_generic, machine_args_generic]
> > +impl_generic = ['Generic armv8', flags_generic, machine_args_default]
> > +impl_0x41 = ['Arm', flags_arm, machine_args_default]
> > +impl_0x42 = ['Broadcom', flags_generic, machine_args_default]
> >  impl_0x43 = ['Cavium', flags_cavium, machine_args_cavium]
> > -impl_0x44 = ['DEC', flags_generic, machine_args_generic]
> > -impl_0x49 = ['Infineon', flags_generic, machine_args_generic]
> > -impl_0x4d = ['Motorola', flags_generic, machine_args_generic]
> > -impl_0x4e = ['NVIDIA', flags_generic, machine_args_generic]
> > +impl_0x44 = ['DEC', flags_generic, machine_args_default]
> > +impl_0x49 = ['Infineon', flags_generic, machine_args_default]
> > +impl_0x4d = ['Motorola', flags_generic, machine_args_default]
> > +impl_0x4e = ['NVIDIA', flags_generic, machine_args_default]
> >  impl_0x50 = ['Ampere Computing', flags_emag, machine_args_emag]
> > -impl_0x51 = ['Qualcomm', flags_generic, machine_args_generic]
> > -impl_0x53 = ['Samsung', flags_generic, machine_args_generic]
> > -impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_generic]
> > -impl_0x69 = ['Intel', flags_generic, machine_args_generic] -impl_dpaa
> > = ['NXP DPAA', flags_dpaa, machine_args_generic]
> > +impl_0x51 = ['Qualcomm', flags_generic, machine_args_default]
> > +impl_0x53 = ['Samsung', flags_generic, machine_args_default]
> > +impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_default]
> > +impl_0x69 = ['Intel', flags_generic, machine_args_default] impl_dpaa
> > += ['NXP DPAA', flags_dpaa, machine_args_default]
> >
> >  dpdk_conf.set('RTE_ARCH_ARM', 1)
> >  dpdk_conf.set('RTE_FORCE_INTRINSICS', 1) @@ -145,7 +146,7 @@ else
> >  	cmd_generic = ['generic', '', '', 'default', '']
> >  	cmd_output = cmd_generic # Set generic by default
> >  	machine_args = [] # Clear previous machine args
> > -	if arm_force_default_march and not meson.is_cross_build()
> > +	if arm_force_generic_march and not meson.is_cross_build()
> >  		machine = impl_generic
> >  		impl_pn = 'default'
> >  	elif not meson.is_cross_build()
> > diff --git a/config/meson.build b/config/meson.build index
> > 258b01d06..918ca2dfb 100644
> > --- a/config/meson.build
> > +++ b/config/meson.build
> > @@ -68,13 +68,13 @@ else
> >  	machine = get_option('machine')
> >  endif
> >
> > -# machine type 'default' is special, it defaults to the per arch
> > agreed common
> > +# machine type 'generic' is special, it defaults to the per arch
> > +agreed common
> >  # minimal baseline needed for DPDK.
> >  # That might not be the most optimized, but the most portable version
> > while # still being able to support the CPU features required for DPDK.
> >  # This can be bumped up by the DPDK project, but it can never be an
> > # invariant like 'native'
> Nit, may be it is worth mentioning that 'default' is allowed for backward
> compatibility?
> 

Ok, I'll add that.

> > -if machine == 'default'
> > +if machine == 'default' or machine == 'generic'
> >  	if host_machine.cpu_family().startswith('x86')
> >  		# matches the old pre-meson build systems default
> >  		machine = 'corei7'
> > @@ -82,7 +82,7 @@ if machine == 'default'
> >  		machine = 'armv7-a'
> >  	elif host_machine.cpu_family().startswith('aarch')
> >  		# arm64 manages defaults in config/arm/meson.build
> > -		machine = 'default'
> > +		machine = 'generic'
> >  	elif host_machine.cpu_family().startswith('ppc')
> >  		machine = 'power8'
> >  	endif
> > --
> > 2.20.1

[dpdk-dev] [PATCH v4 2/6] build: refactor Arm build

[Juraj Linkeš]

* Rename variables to have names that better describe what the variables
store
* Remove unused or superfluous variables
* Change a list to dictionary where key lookup is needed
* Add informatory comments in the code
* Minor code restructure and reformatting

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/arm64_armada_linux_gcc    |   2 +-
 config/arm/arm64_armv8_linux_gcc     |   8 +-
 config/arm/arm64_bluefield_linux_gcc |   4 +-
 config/arm/arm64_dpaa_linux_gcc      |   2 +-
 config/arm/arm64_emag_linux_gcc      |   2 +-
 config/arm/arm64_n1sdp_linux_gcc     |   4 +-
 config/arm/arm64_octeontx2_linux_gcc |   4 +-
 config/arm/arm64_stingray_linux_gcc  |   4 +-
 config/arm/arm64_thunderx2_linux_gcc |   4 +-
 config/arm/arm64_thunderx_linux_gcc  |   2 +-
 config/arm/meson.build               | 247 +++++++++++++++------------
 11 files changed, 153 insertions(+), 130 deletions(-)

diff --git a/config/arm/arm64_armada_linux_gcc b/config/arm/arm64_armada_linux_gcc
index fa40c0398..52c5f4476 100644
--- a/config/arm/arm64_armada_linux_gcc
+++ b/config/arm/arm64_armada_linux_gcc
@@ -14,4 +14,4 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x56'
+implementer_id = '0x56'
diff --git a/config/arm/arm64_armv8_linux_gcc b/config/arm/arm64_armv8_linux_gcc
index 88f0ff9da..13ee8b223 100644
--- a/config/arm/arm64_armv8_linux_gcc
+++ b/config/arm/arm64_armv8_linux_gcc
@@ -13,10 +13,10 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = 'generic'
+implementer_id = 'generic'
 
-# Valid options for Arm's implementor_pn:
-# 'default': valid for all armv8-a architectures (default value)
+# Valid options for Arm's part_number:
+# 'generic': valid for all armv8-a architectures (default value)
 # '0xd03':   cortex-a53
 # '0xd04':   cortex-a35
 # '0xd05':   cortex-a55
@@ -25,4 +25,4 @@ implementor_id = 'generic'
 # '0xd09':   cortex-a73
 # '0xd0a':   cortex-a75
 # '0xd0b':   cortex-a76
-implementor_pn = 'default'
+part_number = 'generic'
diff --git a/config/arm/arm64_bluefield_linux_gcc b/config/arm/arm64_bluefield_linux_gcc
index 86797d23c..b79389d85 100644
--- a/config/arm/arm64_bluefield_linux_gcc
+++ b/config/arm/arm64_bluefield_linux_gcc
@@ -13,5 +13,5 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x41'
-implementor_pn = '0xd08'
+implementer_id = '0x41'
+part_number = '0xd08'
diff --git a/config/arm/arm64_dpaa_linux_gcc b/config/arm/arm64_dpaa_linux_gcc
index 1a4682154..573ae7e42 100644
--- a/config/arm/arm64_dpaa_linux_gcc
+++ b/config/arm/arm64_dpaa_linux_gcc
@@ -14,4 +14,4 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = 'dpaa'
+implementer_id = 'dpaa'
diff --git a/config/arm/arm64_emag_linux_gcc b/config/arm/arm64_emag_linux_gcc
index 8edcd3e97..24f3d533e 100644
--- a/config/arm/arm64_emag_linux_gcc
+++ b/config/arm/arm64_emag_linux_gcc
@@ -13,4 +13,4 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x50'
+implementer_id = '0x50'
diff --git a/config/arm/arm64_n1sdp_linux_gcc b/config/arm/arm64_n1sdp_linux_gcc
index 022e06303..6fb3f02ea 100644
--- a/config/arm/arm64_n1sdp_linux_gcc
+++ b/config/arm/arm64_n1sdp_linux_gcc
@@ -13,5 +13,5 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x41'
-implementor_pn = '0xd0c'
+implementer_id = '0x41'
+part_number = '0xd0c'
diff --git a/config/arm/arm64_octeontx2_linux_gcc b/config/arm/arm64_octeontx2_linux_gcc
index 365bd7cbd..ac1042806 100644
--- a/config/arm/arm64_octeontx2_linux_gcc
+++ b/config/arm/arm64_octeontx2_linux_gcc
@@ -13,5 +13,5 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x43'
-implementor_pn = '0xb2'
+implementer_id = '0x43'
+part_number = '0xb2'
diff --git a/config/arm/arm64_stingray_linux_gcc b/config/arm/arm64_stingray_linux_gcc
index 86797d23c..b79389d85 100644
--- a/config/arm/arm64_stingray_linux_gcc
+++ b/config/arm/arm64_stingray_linux_gcc
@@ -13,5 +13,5 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x41'
-implementor_pn = '0xd08'
+implementer_id = '0x41'
+part_number = '0xd08'
diff --git a/config/arm/arm64_thunderx2_linux_gcc b/config/arm/arm64_thunderx2_linux_gcc
index 2b41acc61..dd257745e 100644
--- a/config/arm/arm64_thunderx2_linux_gcc
+++ b/config/arm/arm64_thunderx2_linux_gcc
@@ -13,5 +13,5 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x43'
-implementor_pn = '0xaf'
+implementer_id = '0x43'
+part_number = '0xaf'
diff --git a/config/arm/arm64_thunderx_linux_gcc b/config/arm/arm64_thunderx_linux_gcc
index 6572ab615..670764437 100644
--- a/config/arm/arm64_thunderx_linux_gcc
+++ b/config/arm/arm64_thunderx_linux_gcc
@@ -13,4 +13,4 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x43'
+implementer_id = '0x43'
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 491842cad..6c31ab167 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -3,12 +3,12 @@
 # Copyright(c) 2017 Cavium, Inc
 # Copyright(c) 2020 PANTHEON.tech s.r.o.
 
-# for checking defines we need to use the correct compiler flags
-march_opt = '-march=@0@'.format(machine)
-
+# set arm_force_native_march if you want to use machine args below
+# instead of discovered values; only works when doing an actual native build
 arm_force_native_march = false
-arm_force_generic_march = (machine == 'generic')
+native_machine_args = ['-march=native', '-mtune=native']
 
+# common flags to all aarch64 builds, with lowest priority
 flags_common_default = [
 	# Accelarate rte_memcpy. Be sure to run unit test (memcpy_perf_autotest)
 	# to determine the best threshold in code. Refer to notes in source file
@@ -16,8 +16,9 @@ flags_common_default = [
 	['RTE_ARCH_ARM64_MEMCPY', false],
 	#	['RTE_ARM64_MEMCPY_ALIGNED_THRESHOLD', 2048],
 	#	['RTE_ARM64_MEMCPY_UNALIGNED_THRESHOLD', 512],
-	# Leave below RTE_ARM64_MEMCPY_xxx options commented out, unless there're
-	# strong reasons.
+
+	# Leave below RTE_ARM64_MEMCPY_xxx options commented out,
+	# unless there are strong reasons.
 	#	['RTE_ARM64_MEMCPY_SKIP_GCC_VER_CHECK', false],
 	#	['RTE_ARM64_MEMCPY_ALIGN_MASK', 0xF],
 	#	['RTE_ARM64_MEMCPY_STRICT_ALIGN', false],
@@ -28,184 +29,206 @@ flags_common_default = [
 
 	['RTE_SCHED_VECTOR', false],
 	['RTE_ARM_USE_WFE', false],
+	['RTE_CACHE_LINE_SIZE', 128],
+	['RTE_ARCH_ARM64', true]
 ]
 
+# implementer specific aarch64 flags, with middle priority
+# (will overwrite common flags)
 flags_generic = [
 	['RTE_MACHINE', '"armv8a"'],
 	['RTE_MAX_LCORE', 256],
 	['RTE_USE_C11_MEM_MODEL', true],
-	['RTE_CACHE_LINE_SIZE', 128]]
+	['RTE_CACHE_LINE_SIZE', 128]
+]
 flags_arm = [
 	['RTE_MACHINE', '"armv8a"'],
 	['RTE_MAX_LCORE', 16],
 	['RTE_USE_C11_MEM_MODEL', true],
-	['RTE_CACHE_LINE_SIZE', 64]]
+	['RTE_CACHE_LINE_SIZE', 64]
+]
 flags_cavium = [
 	['RTE_CACHE_LINE_SIZE', 128],
 	['RTE_MAX_NUMA_NODES', 2],
 	['RTE_MAX_LCORE', 96],
-	['RTE_MAX_VFIO_GROUPS', 128]]
+	['RTE_MAX_VFIO_GROUPS', 128]
+]
 flags_dpaa = [
 	['RTE_MACHINE', '"dpaa"'],
 	['RTE_USE_C11_MEM_MODEL', true],
 	['RTE_CACHE_LINE_SIZE', 64],
 	['RTE_MAX_NUMA_NODES', 1],
 	['RTE_MAX_LCORE', 16],
-	['RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', false]]
+	['RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', false]
+]
 flags_emag = [
 	['RTE_MACHINE', '"emag"'],
-	['RTE_CACHE_LINE_SIZE', 64],
 	['RTE_MAX_NUMA_NODES', 1],
-	['RTE_MAX_LCORE', 32]]
+	['RTE_MAX_LCORE', 32],
+	['RTE_CACHE_LINE_SIZE', 64]
+]
 flags_armada = [
 	['RTE_MACHINE', '"armv8a"'],
-	['RTE_CACHE_LINE_SIZE', 64],
 	['RTE_MAX_NUMA_NODES', 1],
-	['RTE_MAX_LCORE', 16]]
+	['RTE_MAX_LCORE', 16],
+	['RTE_CACHE_LINE_SIZE', 64]
+]
 
-flags_default_extra = []
+# part number specific aarch64 flags, with highest priority
+# (will overwrite both common and implementer specific flags)
 flags_n1sdp_extra = [
 	['RTE_MACHINE', '"n1sdp"'],
 	['RTE_MAX_NUMA_NODES', 1],
 	['RTE_MAX_LCORE', 4],
 	['RTE_EAL_NUMA_AWARE_HUGEPAGES', false],
-	['RTE_LIBRTE_VHOST_NUMA', false]]
+	['RTE_LIBRTE_VHOST_NUMA', false]
+]
 flags_thunderx_extra = [
 	['RTE_MACHINE', '"thunderx"'],
-	['RTE_USE_C11_MEM_MODEL', false]]
+	['RTE_USE_C11_MEM_MODEL', false]
+]
 flags_thunderx2_extra = [
 	['RTE_MACHINE', '"thunderx2"'],
 	['RTE_CACHE_LINE_SIZE', 64],
 	['RTE_MAX_NUMA_NODES', 2],
 	['RTE_MAX_LCORE', 256],
 	['RTE_ARM_FEATURE_ATOMICS', true],
-	['RTE_USE_C11_MEM_MODEL', true]]
+	['RTE_USE_C11_MEM_MODEL', true]
+]
 flags_octeontx2_extra = [
 	['RTE_MACHINE', '"octeontx2"'],
 	['RTE_MAX_NUMA_NODES', 1],
 	['RTE_MAX_LCORE', 36],
 	['RTE_ARM_FEATURE_ATOMICS', true],
 	['RTE_EAL_IGB_UIO', false],
-	['RTE_USE_C11_MEM_MODEL', true]]
-
-machine_args_default = [
-	['default', ['-march=armv8-a+crc', '-moutline-atomics']],
-	['native', ['-march=native']],
-	['0xd03', ['-mcpu=cortex-a53']],
-	['0xd04', ['-mcpu=cortex-a35']],
-	['0xd07', ['-mcpu=cortex-a57']],
-	['0xd08', ['-mcpu=cortex-a72']],
-	['0xd09', ['-mcpu=cortex-a73']],
-	['0xd0a', ['-mcpu=cortex-a75']],
-	['0xd0b', ['-mcpu=cortex-a76']],
-	['0xd0c', ['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'], flags_n1sdp_extra]]
-
-machine_args_cavium = [
-	['default', ['-march=armv8-a+crc+crypto','-mcpu=thunderx']],
-	['native', ['-march=native']],
-	['0xa1', ['-mcpu=thunderxt88'], flags_thunderx_extra],
-	['0xa2', ['-mcpu=thunderxt81'], flags_thunderx_extra],
-	['0xa3', ['-mcpu=thunderxt83'], flags_thunderx_extra],
-	['0xaf', ['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'], flags_thunderx2_extra],
-	['0xb2', ['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'], flags_octeontx2_extra]]
-
-machine_args_emag = [
-	['default', ['-march=armv8-a+crc+crypto', '-mtune=emag']],
-	['native', ['-march=native']]]
+	['RTE_USE_C11_MEM_MODEL', true]
+]
+# arm config (implementer 0x41) is the default config
+pn_config_default = {
+	'generic': [['-march=armv8-a+crc', '-moutline-atomics']],
+	'0xd03': [['-mcpu=cortex-a53']],
+	'0xd04': [['-mcpu=cortex-a35']],
+	'0xd07': [['-mcpu=cortex-a57']],
+	'0xd08': [['-mcpu=cortex-a72']],
+	'0xd09': [['-mcpu=cortex-a73']],
+	'0xd0a': [['-mcpu=cortex-a75']],
+	'0xd0b': [['-mcpu=cortex-a76']],
+	'0xd0c': [['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'], flags_n1sdp_extra]
+}
+pn_config_cavium = {
+	'generic': [['-march=armv8-a+crc+crypto', '-mcpu=thunderx']],
+	'0xa1': [['-mcpu=thunderxt88'], flags_thunderx_extra],
+	'0xa2': [['-mcpu=thunderxt81'], flags_thunderx_extra],
+	'0xa3': [['-mcpu=thunderxt83'], flags_thunderx_extra],
+	'0xaf': [['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'], flags_thunderx2_extra],
+	'0xb2': [['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'], flags_octeontx2_extra],
+}
+pn_config_emag = {
+	'generic': [['-march=armv8-a+crc+crypto', '-mtune=emag']]
+}
 
 ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page G7-5321)
-impl_generic = ['Generic armv8', flags_generic, machine_args_default]
-impl_0x41 = ['Arm', flags_arm, machine_args_default]
-impl_0x42 = ['Broadcom', flags_generic, machine_args_default]
-impl_0x43 = ['Cavium', flags_cavium, machine_args_cavium]
-impl_0x44 = ['DEC', flags_generic, machine_args_default]
-impl_0x49 = ['Infineon', flags_generic, machine_args_default]
-impl_0x4d = ['Motorola', flags_generic, machine_args_default]
-impl_0x4e = ['NVIDIA', flags_generic, machine_args_default]
-impl_0x50 = ['Ampere Computing', flags_emag, machine_args_emag]
-impl_0x51 = ['Qualcomm', flags_generic, machine_args_default]
-impl_0x53 = ['Samsung', flags_generic, machine_args_default]
-impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_default]
-impl_0x69 = ['Intel', flags_generic, machine_args_default]
-impl_dpaa = ['NXP DPAA', flags_dpaa, machine_args_default]
+impl_generic = ['Generic armv8', flags_generic, pn_config_default]
+impl_0x41 = ['Arm', flags_arm, pn_config_default]
+impl_0x42 = ['Broadcom', flags_generic, pn_config_default]
+impl_0x43 = ['Cavium', flags_cavium, pn_config_cavium]
+impl_0x44 = ['DEC', flags_generic, pn_config_default]
+impl_0x49 = ['Infineon', flags_generic, pn_config_default]
+impl_0x4d = ['Motorola', flags_generic, pn_config_default]
+impl_0x4e = ['NVIDIA', flags_generic, pn_config_default]
+impl_0x50 = ['Ampere Computing', flags_emag, pn_config_emag]
+impl_0x51 = ['Qualcomm', flags_generic, pn_config_default]
+impl_0x53 = ['Samsung', flags_generic, pn_config_default]
+impl_0x56 = ['Marvell ARMADA', flags_armada, pn_config_default]
+impl_0x69 = ['Intel', flags_generic, pn_config_default]
+impl_dpaa = ['NXP DPAA', flags_dpaa, pn_config_default]
 
 dpdk_conf.set('RTE_ARCH_ARM', 1)
 dpdk_conf.set('RTE_FORCE_INTRINSICS', 1)
 
 if dpdk_conf.get('RTE_ARCH_32')
+	# armv7 build
 	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
 	dpdk_conf.set('RTE_ARCH_ARMv7', 1)
 	# the minimum architecture supported, armv7-a, needs the following,
 	# mk/machine/armv7a/rte.vars.mk sets it too
 	machine_args += '-mfpu=neon'
 else
-	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
-	dpdk_conf.set('RTE_ARCH_ARM64', 1)
+	# aarch64 build
+	if not meson.is_cross_build()
+		if machine == 'generic'
+			# default build
+			impl_config = impl_generic
+			part_number = 'generic'
+		else
+			# native build
+			# The script returns ['Implementer', 'Variant', 'Architecture',
+			# 'Primary Part number', 'Revision']
+			detect_vendor = find_program(join_paths(
+					meson.current_source_dir(), 'armv8_machine.py'))
+			cmd = run_command(detect_vendor.path())
+			if cmd.returncode() == 0
+				cmd_output = cmd.stdout().to_lower().strip().split(' ')
+			endif
+			if arm_force_native_march == true
+				part_number = 'native'
+			else
+				part_number = cmd_output[3]
+			endif
+			# Set to generic implementer if implementer is not found
+			impl_config = get_variable('impl_' + cmd_output[0], 'impl_generic')
+		endif
+	else
+		# cross build
+		impl_id = meson.get_cross_property('implementer_id', 'generic')
+		part_number = meson.get_cross_property('part_number', 'generic')
+		impl_config = get_variable('impl_' + impl_id)
+	endif
 
-	machine = []
-	cmd_generic = ['generic', '', '', 'default', '']
-	cmd_output = cmd_generic # Set generic by default
-	machine_args = [] # Clear previous machine args
-	if arm_force_generic_march and not meson.is_cross_build()
-		machine = impl_generic
-		impl_pn = 'default'
+	message('Arm implementer: ' + impl_config[0])
+	message('Arm part number: ' + part_number)
+
+	implementer_flags = impl_config[1]
+	part_number_config = impl_config[2]
+
+	if part_number_config.has_key(part_number)
+		# use the specified part_number machine args if found
+		part_number_config = part_number_config[part_number]
+	elif part_number == 'native'
+		# use native machine args
+		part_number_config = [[native_machine_args]]
 	elif not meson.is_cross_build()
-		# The script returns ['Implementer', 'Variant', 'Architecture',
-		# 'Primary Part number', 'Revision']
-		detect_vendor = find_program(join_paths(
-				meson.current_source_dir(), 'armv8_machine.py'))
-		cmd = run_command(detect_vendor.path())
-		if cmd.returncode() == 0
-			cmd_output = cmd.stdout().to_lower().strip().split(' ')
-		endif
-		# Set to generic if variable is not found
-		machine = get_variable('impl_' + cmd_output[0], ['generic'])
-		if machine[0] == 'generic'
-			machine = impl_generic
-			cmd_output = cmd_generic
-		endif
-		impl_pn = cmd_output[3]
-		if arm_force_native_march == true
-			impl_pn = 'native'
-		endif
+		# default to generic machine args if part_number is not found
+		# and not forcing native machine args
+		# but don't default in cross-builds; if part_number is specified
+		# incorrectly in a cross-file, it needs to be fixed there
+		part_number_config = part_number_config['generic']
 	else
-		impl_id = meson.get_cross_property('implementor_id', 'generic')
-		impl_pn = meson.get_cross_property('implementor_pn', 'default')
-		machine = get_variable('impl_' + impl_id)
+		# cross build and part number is not in part_number_config
+		error('Cross build part number 0@0 not found.'.format(part_number))
 	endif
 
-	# Apply Common Defaults. These settings may be overwritten by machine
-	# settings later.
-	foreach flag: flags_common_default
-		if flag.length() > 0
-			dpdk_conf.set(flag[0], flag[1])
+	dpdk_flags = flags_common_default + implementer_flags
+
+	if part_number_config.length() > 1
+		dpdk_flags += part_number_config[1]
+	endif
+
+	machine_args = [] # Clear previous machine args
+	foreach flag: part_number_config[0]
+		if cc.has_argument(flag)
+			machine_args += flag
 		endif
 	endforeach
 
-	message('Implementer : ' + machine[0])
-	foreach flag: machine[1]
+	foreach flag: dpdk_flags
 		if flag.length() > 0
 			dpdk_conf.set(flag[0], flag[1])
 		endif
 	endforeach
-
-	foreach marg: machine[2]
-		if marg[0] == impl_pn
-			foreach flag: marg[1]
-				if cc.has_argument(flag)
-					machine_args += flag
-				endif
-			endforeach
-			# Apply any extra machine specific flags.
-			foreach flag: marg.get(2, flags_default_extra)
-				if flag.length() > 0
-					dpdk_conf.set(flag[0], flag[1])
-				endif
-			endforeach
-		endif
-	endforeach
 endif
-message(machine_args)
+
+message('Using machine args: @0@'.format(machine_args))
 
 if (cc.get_define('__ARM_NEON', args: machine_args) != '' or
     cc.get_define('__aarch64__', args: machine_args) != '')
-- 
2.20.1

Re: [dpdk-dev] [PATCH v4 2/6] build: refactor Arm build

[Honnappa Nagarahalli]

<snip>
I think we need to split this further. Few suggestions below.

> 
> * Rename variables to have names that better describe what the variables
> store
This should be a separate commit.

> * Remove unused or superfluous variables
Same here

> * Change a list to dictionary where key lookup is needed
Same here

> * Add informatory comments in the code

> * Minor code restructure and reformatting
Same for this

> 
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
>  config/arm/arm64_armada_linux_gcc    |   2 +-
>  config/arm/arm64_armv8_linux_gcc     |   8 +-
>  config/arm/arm64_bluefield_linux_gcc |   4 +-
>  config/arm/arm64_dpaa_linux_gcc      |   2 +-
>  config/arm/arm64_emag_linux_gcc      |   2 +-
>  config/arm/arm64_n1sdp_linux_gcc     |   4 +-
>  config/arm/arm64_octeontx2_linux_gcc |   4 +-
>  config/arm/arm64_stingray_linux_gcc  |   4 +-
>  config/arm/arm64_thunderx2_linux_gcc |   4 +-
>  config/arm/arm64_thunderx_linux_gcc  |   2 +-
>  config/arm/meson.build               | 247 +++++++++++++++------------
>  11 files changed, 153 insertions(+), 130 deletions(-)
> 
> diff --git a/config/arm/arm64_armada_linux_gcc
> b/config/arm/arm64_armada_linux_gcc
> index fa40c0398..52c5f4476 100644
> --- a/config/arm/arm64_armada_linux_gcc
> +++ b/config/arm/arm64_armada_linux_gcc
> @@ -14,4 +14,4 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = '0x56'
> +implementer_id = '0x56'
Implementor and implementer mean the same. Looked at Arm specs, they use 'implementer'. So, I am fine.

> diff --git a/config/arm/arm64_armv8_linux_gcc
> b/config/arm/arm64_armv8_linux_gcc
> index 88f0ff9da..13ee8b223 100644
> --- a/config/arm/arm64_armv8_linux_gcc
> +++ b/config/arm/arm64_armv8_linux_gcc
> @@ -13,10 +13,10 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = 'generic'
> +implementer_id = 'generic'
> 
> -# Valid options for Arm's implementor_pn:
> -# 'default': valid for all armv8-a architectures (default value)
> +# Valid options for Arm's part_number:
> +# 'generic': valid for all armv8-a architectures (default value)
>  # '0xd03':   cortex-a53
>  # '0xd04':   cortex-a35
>  # '0xd05':   cortex-a55
> @@ -25,4 +25,4 @@ implementor_id = 'generic'
>  # '0xd09':   cortex-a73
>  # '0xd0a':   cortex-a75
>  # '0xd0b':   cortex-a76
> -implementor_pn = 'default'
> +part_number = 'generic'
Same here, Arm specs refer to this as 'PartNumber'. So, this should be fine.
I like 'generic' for part_number here.

> diff --git a/config/arm/arm64_bluefield_linux_gcc
> b/config/arm/arm64_bluefield_linux_gcc
> index 86797d23c..b79389d85 100644
> --- a/config/arm/arm64_bluefield_linux_gcc
> +++ b/config/arm/arm64_bluefield_linux_gcc
> @@ -13,5 +13,5 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = '0x41'
> -implementor_pn = '0xd08'
> +implementer_id = '0x41'
> +part_number = '0xd08'
> diff --git a/config/arm/arm64_dpaa_linux_gcc
> b/config/arm/arm64_dpaa_linux_gcc index 1a4682154..573ae7e42 100644
> --- a/config/arm/arm64_dpaa_linux_gcc
> +++ b/config/arm/arm64_dpaa_linux_gcc
> @@ -14,4 +14,4 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = 'dpaa'
> +implementer_id = 'dpaa'
> diff --git a/config/arm/arm64_emag_linux_gcc
> b/config/arm/arm64_emag_linux_gcc index 8edcd3e97..24f3d533e 100644
> --- a/config/arm/arm64_emag_linux_gcc
> +++ b/config/arm/arm64_emag_linux_gcc
> @@ -13,4 +13,4 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = '0x50'
> +implementer_id = '0x50'
> diff --git a/config/arm/arm64_n1sdp_linux_gcc
> b/config/arm/arm64_n1sdp_linux_gcc
> index 022e06303..6fb3f02ea 100644
> --- a/config/arm/arm64_n1sdp_linux_gcc
> +++ b/config/arm/arm64_n1sdp_linux_gcc
> @@ -13,5 +13,5 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = '0x41'
> -implementor_pn = '0xd0c'
> +implementer_id = '0x41'
> +part_number = '0xd0c'
> diff --git a/config/arm/arm64_octeontx2_linux_gcc
> b/config/arm/arm64_octeontx2_linux_gcc
> index 365bd7cbd..ac1042806 100644
> --- a/config/arm/arm64_octeontx2_linux_gcc
> +++ b/config/arm/arm64_octeontx2_linux_gcc
> @@ -13,5 +13,5 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = '0x43'
> -implementor_pn = '0xb2'
> +implementer_id = '0x43'
> +part_number = '0xb2'
> diff --git a/config/arm/arm64_stingray_linux_gcc
> b/config/arm/arm64_stingray_linux_gcc
> index 86797d23c..b79389d85 100644
> --- a/config/arm/arm64_stingray_linux_gcc
> +++ b/config/arm/arm64_stingray_linux_gcc
> @@ -13,5 +13,5 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = '0x41'
> -implementor_pn = '0xd08'
> +implementer_id = '0x41'
> +part_number = '0xd08'
> diff --git a/config/arm/arm64_thunderx2_linux_gcc
> b/config/arm/arm64_thunderx2_linux_gcc
> index 2b41acc61..dd257745e 100644
> --- a/config/arm/arm64_thunderx2_linux_gcc
> +++ b/config/arm/arm64_thunderx2_linux_gcc
> @@ -13,5 +13,5 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = '0x43'
> -implementor_pn = '0xaf'
> +implementer_id = '0x43'
> +part_number = '0xaf'
> diff --git a/config/arm/arm64_thunderx_linux_gcc
> b/config/arm/arm64_thunderx_linux_gcc
> index 6572ab615..670764437 100644
> --- a/config/arm/arm64_thunderx_linux_gcc
> +++ b/config/arm/arm64_thunderx_linux_gcc
> @@ -13,4 +13,4 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = '0x43'
> +implementer_id = '0x43'
> diff --git a/config/arm/meson.build b/config/arm/meson.build index
> 491842cad..6c31ab167 100644
> --- a/config/arm/meson.build
> +++ b/config/arm/meson.build
> @@ -3,12 +3,12 @@
>  # Copyright(c) 2017 Cavium, Inc
>  # Copyright(c) 2020 PANTHEON.tech s.r.o.
> 
> -# for checking defines we need to use the correct compiler flags -march_opt
> = '-march=@0@'.format(machine)
> -
> +# set arm_force_native_march if you want to use machine args below #
> +instead of discovered values; only works when doing an actual native
> +build
>  arm_force_native_march = false
> -arm_force_generic_march = (machine == 'generic')
> +native_machine_args = ['-march=native', '-mtune=native']
> 
> +# common flags to all aarch64 builds, with lowest priority
>  flags_common_default = [
>  	# Accelarate rte_memcpy. Be sure to run unit test
> (memcpy_perf_autotest)
>  	# to determine the best threshold in code. Refer to notes in source
> file @@ -16,8 +16,9 @@ flags_common_default = [
>  	['RTE_ARCH_ARM64_MEMCPY', false],
>  	#	['RTE_ARM64_MEMCPY_ALIGNED_THRESHOLD', 2048],
>  	#	['RTE_ARM64_MEMCPY_UNALIGNED_THRESHOLD', 512],
> -	# Leave below RTE_ARM64_MEMCPY_xxx options commented out,
> unless there're
> -	# strong reasons.
> +
> +	# Leave below RTE_ARM64_MEMCPY_xxx options commented out,
> +	# unless there are strong reasons.
>  	#	['RTE_ARM64_MEMCPY_SKIP_GCC_VER_CHECK', false],
>  	#	['RTE_ARM64_MEMCPY_ALIGN_MASK', 0xF],
>  	#	['RTE_ARM64_MEMCPY_STRICT_ALIGN', false],
> @@ -28,184 +29,206 @@ flags_common_default = [
> 
>  	['RTE_SCHED_VECTOR', false],
>  	['RTE_ARM_USE_WFE', false],
> +	['RTE_CACHE_LINE_SIZE', 128],
> +	['RTE_ARCH_ARM64', true]
>  ]
> 
> +# implementer specific aarch64 flags, with middle priority # (will
> +overwrite common flags)
>  flags_generic = [
>  	['RTE_MACHINE', '"armv8a"'],
>  	['RTE_MAX_LCORE', 256],
>  	['RTE_USE_C11_MEM_MODEL', true],
> -	['RTE_CACHE_LINE_SIZE', 128]]
> +	['RTE_CACHE_LINE_SIZE', 128]
> +]
Any particular reason for this change? (and similar changes below)

>  flags_arm = [
>  	['RTE_MACHINE', '"armv8a"'],
>  	['RTE_MAX_LCORE', 16],
>  	['RTE_USE_C11_MEM_MODEL', true],
> -	['RTE_CACHE_LINE_SIZE', 64]]
> +	['RTE_CACHE_LINE_SIZE', 64]
> +]
>  flags_cavium = [
>  	['RTE_CACHE_LINE_SIZE', 128],
>  	['RTE_MAX_NUMA_NODES', 2],
>  	['RTE_MAX_LCORE', 96],
> -	['RTE_MAX_VFIO_GROUPS', 128]]
> +	['RTE_MAX_VFIO_GROUPS', 128]
> +]
>  flags_dpaa = [
>  	['RTE_MACHINE', '"dpaa"'],
>  	['RTE_USE_C11_MEM_MODEL', true],
>  	['RTE_CACHE_LINE_SIZE', 64],
>  	['RTE_MAX_NUMA_NODES', 1],
>  	['RTE_MAX_LCORE', 16],
> -	['RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', false]]
> +	['RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', false] ]
This is not needed

>  flags_emag = [
>  	['RTE_MACHINE', '"emag"'],
> -	['RTE_CACHE_LINE_SIZE', 64],
>  	['RTE_MAX_NUMA_NODES', 1],
> -	['RTE_MAX_LCORE', 32]]
> +	['RTE_MAX_LCORE', 32],
> +	['RTE_CACHE_LINE_SIZE', 64]
> +]
>  flags_armada = [
>  	['RTE_MACHINE', '"armv8a"'],
> -	['RTE_CACHE_LINE_SIZE', 64],
>  	['RTE_MAX_NUMA_NODES', 1],
> -	['RTE_MAX_LCORE', 16]]
> +	['RTE_MAX_LCORE', 16],
> +	['RTE_CACHE_LINE_SIZE', 64]
> +]
Any reason for this change?

> 
> -flags_default_extra = []
> +# part number specific aarch64 flags, with highest priority # (will
> +overwrite both common and implementer specific flags)
>  flags_n1sdp_extra = [
>  	['RTE_MACHINE', '"n1sdp"'],
>  	['RTE_MAX_NUMA_NODES', 1],
>  	['RTE_MAX_LCORE', 4],
>  	['RTE_EAL_NUMA_AWARE_HUGEPAGES', false],
> -	['RTE_LIBRTE_VHOST_NUMA', false]]
> +	['RTE_LIBRTE_VHOST_NUMA', false]
> +]
>  flags_thunderx_extra = [
>  	['RTE_MACHINE', '"thunderx"'],
> -	['RTE_USE_C11_MEM_MODEL', false]]
> +	['RTE_USE_C11_MEM_MODEL', false]
> +]
>  flags_thunderx2_extra = [
>  	['RTE_MACHINE', '"thunderx2"'],
>  	['RTE_CACHE_LINE_SIZE', 64],
>  	['RTE_MAX_NUMA_NODES', 2],
>  	['RTE_MAX_LCORE', 256],
>  	['RTE_ARM_FEATURE_ATOMICS', true],
> -	['RTE_USE_C11_MEM_MODEL', true]]
> +	['RTE_USE_C11_MEM_MODEL', true]
> +]
>  flags_octeontx2_extra = [
>  	['RTE_MACHINE', '"octeontx2"'],
>  	['RTE_MAX_NUMA_NODES', 1],
>  	['RTE_MAX_LCORE', 36],
>  	['RTE_ARM_FEATURE_ATOMICS', true],
>  	['RTE_EAL_IGB_UIO', false],
> -	['RTE_USE_C11_MEM_MODEL', true]]
> -
> -machine_args_default = [
> -	['default', ['-march=armv8-a+crc', '-moutline-atomics']],
> -	['native', ['-march=native']],
> -	['0xd03', ['-mcpu=cortex-a53']],
> -	['0xd04', ['-mcpu=cortex-a35']],
> -	['0xd07', ['-mcpu=cortex-a57']],
> -	['0xd08', ['-mcpu=cortex-a72']],
> -	['0xd09', ['-mcpu=cortex-a73']],
> -	['0xd0a', ['-mcpu=cortex-a75']],
> -	['0xd0b', ['-mcpu=cortex-a76']],
> -	['0xd0c', ['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'],
> flags_n1sdp_extra]]
> -
> -machine_args_cavium = [
> -	['default', ['-march=armv8-a+crc+crypto','-mcpu=thunderx']],
> -	['native', ['-march=native']],
> -	['0xa1', ['-mcpu=thunderxt88'], flags_thunderx_extra],
> -	['0xa2', ['-mcpu=thunderxt81'], flags_thunderx_extra],
> -	['0xa3', ['-mcpu=thunderxt83'], flags_thunderx_extra],
> -	['0xaf', ['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'],
> flags_thunderx2_extra],
> -	['0xb2', ['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'],
> flags_octeontx2_extra]]
> -
> -machine_args_emag = [
> -	['default', ['-march=armv8-a+crc+crypto', '-mtune=emag']],
> -	['native', ['-march=native']]]
> +	['RTE_USE_C11_MEM_MODEL', true]
> +]
> +# arm config (implementer 0x41) is the default config pn_config_default
'pn' here for 'part_number' is not consistent.

> += {
> +	'generic': [['-march=armv8-a+crc', '-moutline-atomics']],
> +	'0xd03': [['-mcpu=cortex-a53']],
> +	'0xd04': [['-mcpu=cortex-a35']],
> +	'0xd07': [['-mcpu=cortex-a57']],
> +	'0xd08': [['-mcpu=cortex-a72']],
> +	'0xd09': [['-mcpu=cortex-a73']],
> +	'0xd0a': [['-mcpu=cortex-a75']],
> +	'0xd0b': [['-mcpu=cortex-a76']],
> +	'0xd0c': [['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'],
> +flags_n1sdp_extra] } pn_config_cavium = {
> +	'generic': [['-march=armv8-a+crc+crypto', '-mcpu=thunderx']],
> +	'0xa1': [['-mcpu=thunderxt88'], flags_thunderx_extra],
> +	'0xa2': [['-mcpu=thunderxt81'], flags_thunderx_extra],
> +	'0xa3': [['-mcpu=thunderxt83'], flags_thunderx_extra],
> +	'0xaf': [['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'],
> flags_thunderx2_extra],
> +	'0xb2': [['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'],
> +flags_octeontx2_extra], } pn_config_emag = {
> +	'generic': [['-march=armv8-a+crc+crypto', '-mtune=emag']] }
> 
>  ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page G7-5321)
> -impl_generic = ['Generic armv8', flags_generic, machine_args_default]
> -impl_0x41 = ['Arm', flags_arm, machine_args_default]
> -impl_0x42 = ['Broadcom', flags_generic, machine_args_default]
> -impl_0x43 = ['Cavium', flags_cavium, machine_args_cavium]
> -impl_0x44 = ['DEC', flags_generic, machine_args_default]
> -impl_0x49 = ['Infineon', flags_generic, machine_args_default] -impl_0x4d =
> ['Motorola', flags_generic, machine_args_default] -impl_0x4e = ['NVIDIA',
> flags_generic, machine_args_default]
> -impl_0x50 = ['Ampere Computing', flags_emag, machine_args_emag]
> -impl_0x51 = ['Qualcomm', flags_generic, machine_args_default]
> -impl_0x53 = ['Samsung', flags_generic, machine_args_default]
> -impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_default]
> -impl_0x69 = ['Intel', flags_generic, machine_args_default] -impl_dpaa =
> ['NXP DPAA', flags_dpaa, machine_args_default]
> +impl_generic = ['Generic armv8', flags_generic, pn_config_default]
> +impl_0x41 = ['Arm', flags_arm, pn_config_default]
> +impl_0x42 = ['Broadcom', flags_generic, pn_config_default]
> +impl_0x43 = ['Cavium', flags_cavium, pn_config_cavium]
> +impl_0x44 = ['DEC', flags_generic, pn_config_default]
> +impl_0x49 = ['Infineon', flags_generic, pn_config_default] impl_0x4d =
> +['Motorola', flags_generic, pn_config_default] impl_0x4e = ['NVIDIA',
> +flags_generic, pn_config_default]
> +impl_0x50 = ['Ampere Computing', flags_emag, pn_config_emag]
> +impl_0x51 = ['Qualcomm', flags_generic, pn_config_default]
> +impl_0x53 = ['Samsung', flags_generic, pn_config_default]
> +impl_0x56 = ['Marvell ARMADA', flags_armada, pn_config_default]
> +impl_0x69 = ['Intel', flags_generic, pn_config_default] impl_dpaa =
> +['NXP DPAA', flags_dpaa, pn_config_default]
> 
>  dpdk_conf.set('RTE_ARCH_ARM', 1)
>  dpdk_conf.set('RTE_FORCE_INTRINSICS', 1)
> 
>  if dpdk_conf.get('RTE_ARCH_32')
> +	# armv7 build
>  	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
>  	dpdk_conf.set('RTE_ARCH_ARMv7', 1)
>  	# the minimum architecture supported, armv7-a, needs the
> following,
>  	# mk/machine/armv7a/rte.vars.mk sets it too
>  	machine_args += '-mfpu=neon'
>  else
> -	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
> -	dpdk_conf.set('RTE_ARCH_ARM64', 1)
> +	# aarch64 build
> +	if not meson.is_cross_build()
> +		if machine == 'generic'
> +			# default build
> +			impl_config = impl_generic
> +			part_number = 'generic'
> +		else
> +			# native build
> +			# The script returns ['Implementer', 'Variant',
> 'Architecture',
> +			# 'Primary Part number', 'Revision']
> +			detect_vendor = find_program(join_paths(
> +					meson.current_source_dir(),
> 'armv8_machine.py'))
> +			cmd = run_command(detect_vendor.path())
> +			if cmd.returncode() == 0
> +				cmd_output =
> cmd.stdout().to_lower().strip().split(' ')
> +			endif
> +			if arm_force_native_march == true
> +				part_number = 'native'
> +			else
> +				part_number = cmd_output[3]
> +			endif
> +			# Set to generic implementer if implementer is not
> found
> +			impl_config = get_variable('impl_' + cmd_output[0],
> 'impl_generic')
> +		endif
> +	else
> +		# cross build
> +		impl_id = meson.get_cross_property('implementer_id',
> 'generic')
> +		part_number = meson.get_cross_property('part_number',
> 'generic')
> +		impl_config = get_variable('impl_' + impl_id)
> +	endif
> 
> -	machine = []
> -	cmd_generic = ['generic', '', '', 'default', '']
> -	cmd_output = cmd_generic # Set generic by default
> -	machine_args = [] # Clear previous machine args
> -	if arm_force_generic_march and not meson.is_cross_build()
> -		machine = impl_generic
> -		impl_pn = 'default'
> +	message('Arm implementer: ' + impl_config[0])
> +	message('Arm part number: ' + part_number)
> +
> +	implementer_flags = impl_config[1]
> +	part_number_config = impl_config[2]
> +
> +	if part_number_config.has_key(part_number)
> +		# use the specified part_number machine args if found
> +		part_number_config = part_number_config[part_number]
> +	elif part_number == 'native'
> +		# use native machine args
> +		part_number_config = [[native_machine_args]]
>  	elif not meson.is_cross_build()
> -		# The script returns ['Implementer', 'Variant', 'Architecture',
> -		# 'Primary Part number', 'Revision']
> -		detect_vendor = find_program(join_paths(
> -				meson.current_source_dir(),
> 'armv8_machine.py'))
> -		cmd = run_command(detect_vendor.path())
> -		if cmd.returncode() == 0
> -			cmd_output = cmd.stdout().to_lower().strip().split('
> ')
> -		endif
> -		# Set to generic if variable is not found
> -		machine = get_variable('impl_' + cmd_output[0], ['generic'])
> -		if machine[0] == 'generic'
> -			machine = impl_generic
> -			cmd_output = cmd_generic
> -		endif
> -		impl_pn = cmd_output[3]
> -		if arm_force_native_march == true
> -			impl_pn = 'native'
> -		endif
> +		# default to generic machine args if part_number is not found
> +		# and not forcing native machine args
> +		# but don't default in cross-builds; if part_number is specified
> +		# incorrectly in a cross-file, it needs to be fixed there
> +		part_number_config = part_number_config['generic']
>  	else
> -		impl_id = meson.get_cross_property('implementor_id',
> 'generic')
> -		impl_pn = meson.get_cross_property('implementor_pn',
> 'default')
> -		machine = get_variable('impl_' + impl_id)
> +		# cross build and part number is not in part_number_config
> +		error('Cross build part number 0@0 not
> found.'.format(part_number))
>  	endif
> 
> -	# Apply Common Defaults. These settings may be overwritten by
> machine
> -	# settings later.
> -	foreach flag: flags_common_default
> -		if flag.length() > 0
> -			dpdk_conf.set(flag[0], flag[1])
> +	dpdk_flags = flags_common_default + implementer_flags
> +
> +	if part_number_config.length() > 1
> +		dpdk_flags += part_number_config[1]
> +	endif
> +
> +	machine_args = [] # Clear previous machine args
> +	foreach flag: part_number_config[0]
> +		if cc.has_argument(flag)
> +			machine_args += flag
>  		endif
>  	endforeach
> 
> -	message('Implementer : ' + machine[0])
> -	foreach flag: machine[1]
> +	foreach flag: dpdk_flags
>  		if flag.length() > 0
>  			dpdk_conf.set(flag[0], flag[1])
>  		endif
>  	endforeach
> -
> -	foreach marg: machine[2]
> -		if marg[0] == impl_pn
> -			foreach flag: marg[1]
> -				if cc.has_argument(flag)
> -					machine_args += flag
> -				endif
> -			endforeach
> -			# Apply any extra machine specific flags.
> -			foreach flag: marg.get(2, flags_default_extra)
> -				if flag.length() > 0
> -					dpdk_conf.set(flag[0], flag[1])
> -				endif
> -			endforeach
> -		endif
> -	endforeach
>  endif
> -message(machine_args)
> +
> +message('Using machine args: @0@'.format(machine_args))
> 
>  if (cc.get_define('__ARM_NEON', args: machine_args) != '' or
>      cc.get_define('__aarch64__', args: machine_args) != '')
> --
> 2.20.1

Re: [dpdk-dev] [PATCH v4 2/6] build: refactor Arm build

[Juraj Linkeš]

> -----Original Message-----
> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Sent: Tuesday, October 27, 2020 5:56 AM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>; bruce.richardson@intel.com;
> Ruifeng Wang <Ruifeng.Wang@arm.com>; Phil Yang <Phil.Yang@arm.com>;
> vcchunga@amazon.com; Dharmik Thakkar <Dharmik.Thakkar@arm.com>;
> jerinjacobk@gmail.com; hemant.agrawal@nxp.com
> Cc: dev@dpdk.org; nd <nd@arm.com>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>
> Subject: RE: [PATCH v4 2/6] build: refactor Arm build
> 
> <snip>
> I think we need to split this further. Few suggestions below.
> 
> >
> > * Rename variables to have names that better describe what the
> > variables store
> This should be a separate commit.
> 
> > * Remove unused or superfluous variables
> Same here
> 
> > * Change a list to dictionary where key lookup is needed
> Same here
> 
> > * Add informatory comments in the code
> 
> > * Minor code restructure and reformatting
> Same for this
> 

Ok, hopefully I'll be able to separate these cleanly.

> >
> > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > ---
> >  config/arm/arm64_armada_linux_gcc    |   2 +-
> >  config/arm/arm64_armv8_linux_gcc     |   8 +-
> >  config/arm/arm64_bluefield_linux_gcc |   4 +-
> >  config/arm/arm64_dpaa_linux_gcc      |   2 +-
> >  config/arm/arm64_emag_linux_gcc      |   2 +-
> >  config/arm/arm64_n1sdp_linux_gcc     |   4 +-
> >  config/arm/arm64_octeontx2_linux_gcc |   4 +-
> >  config/arm/arm64_stingray_linux_gcc  |   4 +-
> >  config/arm/arm64_thunderx2_linux_gcc |   4 +-
> >  config/arm/arm64_thunderx_linux_gcc  |   2 +-
> >  config/arm/meson.build               | 247 +++++++++++++++------------
> >  11 files changed, 153 insertions(+), 130 deletions(-)
> >
> > diff --git a/config/arm/arm64_armada_linux_gcc
> > b/config/arm/arm64_armada_linux_gcc
> > index fa40c0398..52c5f4476 100644
> > --- a/config/arm/arm64_armada_linux_gcc
> > +++ b/config/arm/arm64_armada_linux_gcc
> > @@ -14,4 +14,4 @@ cpu = 'armv8-a'
> >  endian = 'little'
> >
> >  [properties]
> > -implementor_id = '0x56'
> > +implementer_id = '0x56'
> Implementor and implementer mean the same. Looked at Arm specs, they use
> 'implementer'. So, I am fine.
> 

That's where I got this and some of the other variable name changes from.

> > diff --git a/config/arm/arm64_armv8_linux_gcc
> > b/config/arm/arm64_armv8_linux_gcc
> > index 88f0ff9da..13ee8b223 100644
> > --- a/config/arm/arm64_armv8_linux_gcc
> > +++ b/config/arm/arm64_armv8_linux_gcc
> > @@ -13,10 +13,10 @@ cpu = 'armv8-a'
> >  endian = 'little'
> >
> >  [properties]
> > -implementor_id = 'generic'
> > +implementer_id = 'generic'
> >
> > -# Valid options for Arm's implementor_pn:
> > -# 'default': valid for all armv8-a architectures (default value)
> > +# Valid options for Arm's part_number:
> > +# 'generic': valid for all armv8-a architectures (default value)
> >  # '0xd03':   cortex-a53
> >  # '0xd04':   cortex-a35
> >  # '0xd05':   cortex-a55
> > @@ -25,4 +25,4 @@ implementor_id = 'generic'
> >  # '0xd09':   cortex-a73
> >  # '0xd0a':   cortex-a75
> >  # '0xd0b':   cortex-a76
> > -implementor_pn = 'default'
> > +part_number = 'generic'
> Same here, Arm specs refer to this as 'PartNumber'. So, this should be fine.
> I like 'generic' for part_number here.
> 
> > diff --git a/config/arm/arm64_bluefield_linux_gcc
> > b/config/arm/arm64_bluefield_linux_gcc
> > index 86797d23c..b79389d85 100644
> > --- a/config/arm/arm64_bluefield_linux_gcc
> > +++ b/config/arm/arm64_bluefield_linux_gcc
> > @@ -13,5 +13,5 @@ cpu = 'armv8-a'
> >  endian = 'little'
> >
> >  [properties]
> > -implementor_id = '0x41'
> > -implementor_pn = '0xd08'
> > +implementer_id = '0x41'
> > +part_number = '0xd08'
> > diff --git a/config/arm/arm64_dpaa_linux_gcc
> > b/config/arm/arm64_dpaa_linux_gcc index 1a4682154..573ae7e42 100644
> > --- a/config/arm/arm64_dpaa_linux_gcc
> > +++ b/config/arm/arm64_dpaa_linux_gcc
> > @@ -14,4 +14,4 @@ cpu = 'armv8-a'
> >  endian = 'little'
> >
> >  [properties]
> > -implementor_id = 'dpaa'
> > +implementer_id = 'dpaa'
> > diff --git a/config/arm/arm64_emag_linux_gcc
> > b/config/arm/arm64_emag_linux_gcc index 8edcd3e97..24f3d533e 100644
> > --- a/config/arm/arm64_emag_linux_gcc
> > +++ b/config/arm/arm64_emag_linux_gcc
> > @@ -13,4 +13,4 @@ cpu = 'armv8-a'
> >  endian = 'little'
> >
> >  [properties]
> > -implementor_id = '0x50'
> > +implementer_id = '0x50'
> > diff --git a/config/arm/arm64_n1sdp_linux_gcc
> > b/config/arm/arm64_n1sdp_linux_gcc
> > index 022e06303..6fb3f02ea 100644
> > --- a/config/arm/arm64_n1sdp_linux_gcc
> > +++ b/config/arm/arm64_n1sdp_linux_gcc
> > @@ -13,5 +13,5 @@ cpu = 'armv8-a'
> >  endian = 'little'
> >
> >  [properties]
> > -implementor_id = '0x41'
> > -implementor_pn = '0xd0c'
> > +implementer_id = '0x41'
> > +part_number = '0xd0c'
> > diff --git a/config/arm/arm64_octeontx2_linux_gcc
> > b/config/arm/arm64_octeontx2_linux_gcc
> > index 365bd7cbd..ac1042806 100644
> > --- a/config/arm/arm64_octeontx2_linux_gcc
> > +++ b/config/arm/arm64_octeontx2_linux_gcc
> > @@ -13,5 +13,5 @@ cpu = 'armv8-a'
> >  endian = 'little'
> >
> >  [properties]
> > -implementor_id = '0x43'
> > -implementor_pn = '0xb2'
> > +implementer_id = '0x43'
> > +part_number = '0xb2'
> > diff --git a/config/arm/arm64_stingray_linux_gcc
> > b/config/arm/arm64_stingray_linux_gcc
> > index 86797d23c..b79389d85 100644
> > --- a/config/arm/arm64_stingray_linux_gcc
> > +++ b/config/arm/arm64_stingray_linux_gcc
> > @@ -13,5 +13,5 @@ cpu = 'armv8-a'
> >  endian = 'little'
> >
> >  [properties]
> > -implementor_id = '0x41'
> > -implementor_pn = '0xd08'
> > +implementer_id = '0x41'
> > +part_number = '0xd08'
> > diff --git a/config/arm/arm64_thunderx2_linux_gcc
> > b/config/arm/arm64_thunderx2_linux_gcc
> > index 2b41acc61..dd257745e 100644
> > --- a/config/arm/arm64_thunderx2_linux_gcc
> > +++ b/config/arm/arm64_thunderx2_linux_gcc
> > @@ -13,5 +13,5 @@ cpu = 'armv8-a'
> >  endian = 'little'
> >
> >  [properties]
> > -implementor_id = '0x43'
> > -implementor_pn = '0xaf'
> > +implementer_id = '0x43'
> > +part_number = '0xaf'
> > diff --git a/config/arm/arm64_thunderx_linux_gcc
> > b/config/arm/arm64_thunderx_linux_gcc
> > index 6572ab615..670764437 100644
> > --- a/config/arm/arm64_thunderx_linux_gcc
> > +++ b/config/arm/arm64_thunderx_linux_gcc
> > @@ -13,4 +13,4 @@ cpu = 'armv8-a'
> >  endian = 'little'
> >
> >  [properties]
> > -implementor_id = '0x43'
> > +implementer_id = '0x43'
> > diff --git a/config/arm/meson.build b/config/arm/meson.build index
> > 491842cad..6c31ab167 100644
> > --- a/config/arm/meson.build
> > +++ b/config/arm/meson.build
> > @@ -3,12 +3,12 @@
> >  # Copyright(c) 2017 Cavium, Inc
> >  # Copyright(c) 2020 PANTHEON.tech s.r.o.
> >
> > -# for checking defines we need to use the correct compiler flags
> > -march_opt = '-march=@0@'.format(machine)
> > -
> > +# set arm_force_native_march if you want to use machine args below #
> > +instead of discovered values; only works when doing an actual native
> > +build
> >  arm_force_native_march = false
> > -arm_force_generic_march = (machine == 'generic')
> > +native_machine_args = ['-march=native', '-mtune=native']
> >
> > +# common flags to all aarch64 builds, with lowest priority
> >  flags_common_default = [
> >  	# Accelarate rte_memcpy. Be sure to run unit test
> > (memcpy_perf_autotest)
> >  	# to determine the best threshold in code. Refer to notes in source
> > file @@ -16,8 +16,9 @@ flags_common_default = [
> >  	['RTE_ARCH_ARM64_MEMCPY', false],
> >  	#	['RTE_ARM64_MEMCPY_ALIGNED_THRESHOLD', 2048],
> >  	#	['RTE_ARM64_MEMCPY_UNALIGNED_THRESHOLD', 512],
> > -	# Leave below RTE_ARM64_MEMCPY_xxx options commented out,
> > unless there're
> > -	# strong reasons.
> > +
> > +	# Leave below RTE_ARM64_MEMCPY_xxx options commented out,
> > +	# unless there are strong reasons.
> >  	#	['RTE_ARM64_MEMCPY_SKIP_GCC_VER_CHECK', false],
> >  	#	['RTE_ARM64_MEMCPY_ALIGN_MASK', 0xF],
> >  	#	['RTE_ARM64_MEMCPY_STRICT_ALIGN', false],
> > @@ -28,184 +29,206 @@ flags_common_default = [
> >
> >  	['RTE_SCHED_VECTOR', false],
> >  	['RTE_ARM_USE_WFE', false],
> > +	['RTE_CACHE_LINE_SIZE', 128],
> > +	['RTE_ARCH_ARM64', true]
> >  ]
> >
> > +# implementer specific aarch64 flags, with middle priority # (will
> > +overwrite common flags)
> >  flags_generic = [
> >  	['RTE_MACHINE', '"armv8a"'],
> >  	['RTE_MAX_LCORE', 256],
> >  	['RTE_USE_C11_MEM_MODEL', true],
> > -	['RTE_CACHE_LINE_SIZE', 128]]
> > +	['RTE_CACHE_LINE_SIZE', 128]
> > +]
> Any particular reason for this change? (and similar changes below)
> 

The first bracket is split from the second bracket so I did the same for the last two brackets. It makes it more apparent which brackets are paired, it's more consistent and also in line with how flags_common_default is formatted.

> >  flags_arm = [
> >  	['RTE_MACHINE', '"armv8a"'],
> >  	['RTE_MAX_LCORE', 16],
> >  	['RTE_USE_C11_MEM_MODEL', true],
> > -	['RTE_CACHE_LINE_SIZE', 64]]
> > +	['RTE_CACHE_LINE_SIZE', 64]
> > +]
> >  flags_cavium = [
> >  	['RTE_CACHE_LINE_SIZE', 128],
> >  	['RTE_MAX_NUMA_NODES', 2],
> >  	['RTE_MAX_LCORE', 96],
> > -	['RTE_MAX_VFIO_GROUPS', 128]]
> > +	['RTE_MAX_VFIO_GROUPS', 128]
> > +]
> >  flags_dpaa = [
> >  	['RTE_MACHINE', '"dpaa"'],
> >  	['RTE_USE_C11_MEM_MODEL', true],
> >  	['RTE_CACHE_LINE_SIZE', 64],
> >  	['RTE_MAX_NUMA_NODES', 1],
> >  	['RTE_MAX_LCORE', 16],
> > -	['RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', false]]
> > +	['RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', false] ]
> This is not needed
> 

Do you mean the space? It should be a line break. I'll check the exact characters, but I see this as adding a space in my local patch. Or do you mean the config option? It's set to true by default in config/meson.build and according to [1] it should be disabled.

[1] http://git.dpdk.org/dpdk/tree/config/defconfig_arm64-dpaa-linuxapp-gcc?h=v20.08

> >  flags_emag = [
> >  	['RTE_MACHINE', '"emag"'],
> > -	['RTE_CACHE_LINE_SIZE', 64],
> >  	['RTE_MAX_NUMA_NODES', 1],
> > -	['RTE_MAX_LCORE', 32]]
> > +	['RTE_MAX_LCORE', 32],
> > +	['RTE_CACHE_LINE_SIZE', 64]
> > +]
> >  flags_armada = [
> >  	['RTE_MACHINE', '"armv8a"'],
> > -	['RTE_CACHE_LINE_SIZE', 64],
> >  	['RTE_MAX_NUMA_NODES', 1],
> > -	['RTE_MAX_LCORE', 16]]
> > +	['RTE_MAX_LCORE', 16],
> > +	['RTE_CACHE_LINE_SIZE', 64]
> > +]
> Any reason for this change?
> 

The default (from flags_common_default) is 128 and I found here [0] that it should be set to 64 so I added it here. Should this also be in a separate patch (apart from those 4 already mention above)?

[0] http://git.dpdk.org/dpdk/tree/config/defconfig_arm64-armada-linuxapp-gcc?h=v20.08

> >
> > -flags_default_extra = []
> > +# part number specific aarch64 flags, with highest priority # (will
> > +overwrite both common and implementer specific flags)
> >  flags_n1sdp_extra = [
> >  	['RTE_MACHINE', '"n1sdp"'],
> >  	['RTE_MAX_NUMA_NODES', 1],
> >  	['RTE_MAX_LCORE', 4],
> >  	['RTE_EAL_NUMA_AWARE_HUGEPAGES', false],
> > -	['RTE_LIBRTE_VHOST_NUMA', false]]
> > +	['RTE_LIBRTE_VHOST_NUMA', false]
> > +]
> >  flags_thunderx_extra = [
> >  	['RTE_MACHINE', '"thunderx"'],
> > -	['RTE_USE_C11_MEM_MODEL', false]]
> > +	['RTE_USE_C11_MEM_MODEL', false]
> > +]
> >  flags_thunderx2_extra = [
> >  	['RTE_MACHINE', '"thunderx2"'],
> >  	['RTE_CACHE_LINE_SIZE', 64],
> >  	['RTE_MAX_NUMA_NODES', 2],
> >  	['RTE_MAX_LCORE', 256],
> >  	['RTE_ARM_FEATURE_ATOMICS', true],
> > -	['RTE_USE_C11_MEM_MODEL', true]]
> > +	['RTE_USE_C11_MEM_MODEL', true]
> > +]
> >  flags_octeontx2_extra = [
> >  	['RTE_MACHINE', '"octeontx2"'],
> >  	['RTE_MAX_NUMA_NODES', 1],
> >  	['RTE_MAX_LCORE', 36],
> >  	['RTE_ARM_FEATURE_ATOMICS', true],
> >  	['RTE_EAL_IGB_UIO', false],
> > -	['RTE_USE_C11_MEM_MODEL', true]]
> > -
> > -machine_args_default = [
> > -	['default', ['-march=armv8-a+crc', '-moutline-atomics']],
> > -	['native', ['-march=native']],
> > -	['0xd03', ['-mcpu=cortex-a53']],
> > -	['0xd04', ['-mcpu=cortex-a35']],
> > -	['0xd07', ['-mcpu=cortex-a57']],
> > -	['0xd08', ['-mcpu=cortex-a72']],
> > -	['0xd09', ['-mcpu=cortex-a73']],
> > -	['0xd0a', ['-mcpu=cortex-a75']],
> > -	['0xd0b', ['-mcpu=cortex-a76']],
> > -	['0xd0c', ['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'],
> > flags_n1sdp_extra]]
> > -
> > -machine_args_cavium = [
> > -	['default', ['-march=armv8-a+crc+crypto','-mcpu=thunderx']],
> > -	['native', ['-march=native']],
> > -	['0xa1', ['-mcpu=thunderxt88'], flags_thunderx_extra],
> > -	['0xa2', ['-mcpu=thunderxt81'], flags_thunderx_extra],
> > -	['0xa3', ['-mcpu=thunderxt83'], flags_thunderx_extra],
> > -	['0xaf', ['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'],
> > flags_thunderx2_extra],
> > -	['0xb2', ['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'],
> > flags_octeontx2_extra]]
> > -
> > -machine_args_emag = [
> > -	['default', ['-march=armv8-a+crc+crypto', '-mtune=emag']],
> > -	['native', ['-march=native']]]
> > +	['RTE_USE_C11_MEM_MODEL', true]
> > +]
> > +# arm config (implementer 0x41) is the default config
> > +pn_config_default
> 'pn' here for 'part_number' is not consistent.
> 

Ok, I can rename it to part_number_config_default. Same for the other two pn variables.

> > += {
> > +	'generic': [['-march=armv8-a+crc', '-moutline-atomics']],
> > +	'0xd03': [['-mcpu=cortex-a53']],
> > +	'0xd04': [['-mcpu=cortex-a35']],
> > +	'0xd07': [['-mcpu=cortex-a57']],
> > +	'0xd08': [['-mcpu=cortex-a72']],
> > +	'0xd09': [['-mcpu=cortex-a73']],
> > +	'0xd0a': [['-mcpu=cortex-a75']],
> > +	'0xd0b': [['-mcpu=cortex-a76']],
> > +	'0xd0c': [['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'],
> > +flags_n1sdp_extra] } pn_config_cavium = {
> > +	'generic': [['-march=armv8-a+crc+crypto', '-mcpu=thunderx']],
> > +	'0xa1': [['-mcpu=thunderxt88'], flags_thunderx_extra],
> > +	'0xa2': [['-mcpu=thunderxt81'], flags_thunderx_extra],
> > +	'0xa3': [['-mcpu=thunderxt83'], flags_thunderx_extra],
> > +	'0xaf': [['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'],
> > flags_thunderx2_extra],
> > +	'0xb2': [['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'],
> > +flags_octeontx2_extra], } pn_config_emag = {
> > +	'generic': [['-march=armv8-a+crc+crypto', '-mtune=emag']] }
> >
> >  ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page
> > G7-5321) -impl_generic = ['Generic armv8', flags_generic,
> > machine_args_default]
> > -impl_0x41 = ['Arm', flags_arm, machine_args_default]
> > -impl_0x42 = ['Broadcom', flags_generic, machine_args_default]
> > -impl_0x43 = ['Cavium', flags_cavium, machine_args_cavium]
> > -impl_0x44 = ['DEC', flags_generic, machine_args_default]
> > -impl_0x49 = ['Infineon', flags_generic, machine_args_default]
> > -impl_0x4d = ['Motorola', flags_generic, machine_args_default]
> > -impl_0x4e = ['NVIDIA', flags_generic, machine_args_default]
> > -impl_0x50 = ['Ampere Computing', flags_emag, machine_args_emag]
> > -impl_0x51 = ['Qualcomm', flags_generic, machine_args_default]
> > -impl_0x53 = ['Samsung', flags_generic, machine_args_default]
> > -impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_default]
> > -impl_0x69 = ['Intel', flags_generic, machine_args_default] -impl_dpaa
> > = ['NXP DPAA', flags_dpaa, machine_args_default]
> > +impl_generic = ['Generic armv8', flags_generic, pn_config_default]
> > +impl_0x41 = ['Arm', flags_arm, pn_config_default]
> > +impl_0x42 = ['Broadcom', flags_generic, pn_config_default]
> > +impl_0x43 = ['Cavium', flags_cavium, pn_config_cavium]
> > +impl_0x44 = ['DEC', flags_generic, pn_config_default]
> > +impl_0x49 = ['Infineon', flags_generic, pn_config_default] impl_0x4d
> > += ['Motorola', flags_generic, pn_config_default] impl_0x4e =
> > +['NVIDIA', flags_generic, pn_config_default]
> > +impl_0x50 = ['Ampere Computing', flags_emag, pn_config_emag]
> > +impl_0x51 = ['Qualcomm', flags_generic, pn_config_default]
> > +impl_0x53 = ['Samsung', flags_generic, pn_config_default]
> > +impl_0x56 = ['Marvell ARMADA', flags_armada, pn_config_default]
> > +impl_0x69 = ['Intel', flags_generic, pn_config_default] impl_dpaa =
> > +['NXP DPAA', flags_dpaa, pn_config_default]
> >
> >  dpdk_conf.set('RTE_ARCH_ARM', 1)
> >  dpdk_conf.set('RTE_FORCE_INTRINSICS', 1)
> >
> >  if dpdk_conf.get('RTE_ARCH_32')
> > +	# armv7 build
> >  	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
> >  	dpdk_conf.set('RTE_ARCH_ARMv7', 1)
> >  	# the minimum architecture supported, armv7-a, needs the following,
> >  	# mk/machine/armv7a/rte.vars.mk sets it too
> >  	machine_args += '-mfpu=neon'
> >  else
> > -	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
> > -	dpdk_conf.set('RTE_ARCH_ARM64', 1)
> > +	# aarch64 build
> > +	if not meson.is_cross_build()
> > +		if machine == 'generic'
> > +			# default build
> > +			impl_config = impl_generic
> > +			part_number = 'generic'
> > +		else
> > +			# native build
> > +			# The script returns ['Implementer', 'Variant',
> > 'Architecture',
> > +			# 'Primary Part number', 'Revision']
> > +			detect_vendor = find_program(join_paths(
> > +					meson.current_source_dir(),
> > 'armv8_machine.py'))
> > +			cmd = run_command(detect_vendor.path())
> > +			if cmd.returncode() == 0
> > +				cmd_output =
> > cmd.stdout().to_lower().strip().split(' ')
> > +			endif
> > +			if arm_force_native_march == true
> > +				part_number = 'native'
> > +			else
> > +				part_number = cmd_output[3]
> > +			endif
> > +			# Set to generic implementer if implementer is not
> > found
> > +			impl_config = get_variable('impl_' + cmd_output[0],
> > 'impl_generic')
> > +		endif
> > +	else
> > +		# cross build
> > +		impl_id = meson.get_cross_property('implementer_id',
> > 'generic')
> > +		part_number = meson.get_cross_property('part_number',
> > 'generic')
> > +		impl_config = get_variable('impl_' + impl_id)
> > +	endif
> >
> > -	machine = []
> > -	cmd_generic = ['generic', '', '', 'default', '']
> > -	cmd_output = cmd_generic # Set generic by default
> > -	machine_args = [] # Clear previous machine args
> > -	if arm_force_generic_march and not meson.is_cross_build()
> > -		machine = impl_generic
> > -		impl_pn = 'default'
> > +	message('Arm implementer: ' + impl_config[0])
> > +	message('Arm part number: ' + part_number)
> > +
> > +	implementer_flags = impl_config[1]
> > +	part_number_config = impl_config[2]
> > +
> > +	if part_number_config.has_key(part_number)
> > +		# use the specified part_number machine args if found
> > +		part_number_config = part_number_config[part_number]
> > +	elif part_number == 'native'
> > +		# use native machine args
> > +		part_number_config = [[native_machine_args]]
> >  	elif not meson.is_cross_build()
> > -		# The script returns ['Implementer', 'Variant', 'Architecture',
> > -		# 'Primary Part number', 'Revision']
> > -		detect_vendor = find_program(join_paths(
> > -				meson.current_source_dir(),
> > 'armv8_machine.py'))
> > -		cmd = run_command(detect_vendor.path())
> > -		if cmd.returncode() == 0
> > -			cmd_output = cmd.stdout().to_lower().strip().split('
> > ')
> > -		endif
> > -		# Set to generic if variable is not found
> > -		machine = get_variable('impl_' + cmd_output[0], ['generic'])
> > -		if machine[0] == 'generic'
> > -			machine = impl_generic
> > -			cmd_output = cmd_generic
> > -		endif
> > -		impl_pn = cmd_output[3]
> > -		if arm_force_native_march == true
> > -			impl_pn = 'native'
> > -		endif
> > +		# default to generic machine args if part_number is not found
> > +		# and not forcing native machine args
> > +		# but don't default in cross-builds; if part_number is specified
> > +		# incorrectly in a cross-file, it needs to be fixed there
> > +		part_number_config = part_number_config['generic']
> >  	else
> > -		impl_id = meson.get_cross_property('implementor_id',
> > 'generic')
> > -		impl_pn = meson.get_cross_property('implementor_pn',
> > 'default')
> > -		machine = get_variable('impl_' + impl_id)
> > +		# cross build and part number is not in part_number_config
> > +		error('Cross build part number 0@0 not
> > found.'.format(part_number))
> >  	endif
> >
> > -	# Apply Common Defaults. These settings may be overwritten by
> > machine
> > -	# settings later.
> > -	foreach flag: flags_common_default
> > -		if flag.length() > 0
> > -			dpdk_conf.set(flag[0], flag[1])
> > +	dpdk_flags = flags_common_default + implementer_flags
> > +
> > +	if part_number_config.length() > 1
> > +		dpdk_flags += part_number_config[1]
> > +	endif
> > +
> > +	machine_args = [] # Clear previous machine args
> > +	foreach flag: part_number_config[0]
> > +		if cc.has_argument(flag)
> > +			machine_args += flag
> >  		endif
> >  	endforeach
> >
> > -	message('Implementer : ' + machine[0])
> > -	foreach flag: machine[1]
> > +	foreach flag: dpdk_flags
> >  		if flag.length() > 0
> >  			dpdk_conf.set(flag[0], flag[1])
> >  		endif
> >  	endforeach
> > -
> > -	foreach marg: machine[2]
> > -		if marg[0] == impl_pn
> > -			foreach flag: marg[1]
> > -				if cc.has_argument(flag)
> > -					machine_args += flag
> > -				endif
> > -			endforeach
> > -			# Apply any extra machine specific flags.
> > -			foreach flag: marg.get(2, flags_default_extra)
> > -				if flag.length() > 0
> > -					dpdk_conf.set(flag[0], flag[1])
> > -				endif
> > -			endforeach
> > -		endif
> > -	endforeach
> >  endif
> > -message(machine_args)
> > +
> > +message('Using machine args: @0@'.format(machine_args))
> >
> >  if (cc.get_define('__ARM_NEON', args: machine_args) != '' or
> >      cc.get_define('__aarch64__', args: machine_args) != '')
> > --
> > 2.20.1

Re: [dpdk-dev] [PATCH v4 2/6] build: refactor Arm build

[Juraj Linkeš]

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Juraj Linkeš
> Sent: Tuesday, October 27, 2020 9:11 AM
> To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>;
> bruce.richardson@intel.com; Ruifeng Wang <Ruifeng.Wang@arm.com>; Phil
> Yang <Phil.Yang@arm.com>; vcchunga@amazon.com; Dharmik Thakkar
> <Dharmik.Thakkar@arm.com>; jerinjacobk@gmail.com;
> hemant.agrawal@nxp.com
> Cc: dev@dpdk.org; nd <nd@arm.com>; nd <nd@arm.com>
> Subject: Re: [dpdk-dev] [PATCH v4 2/6] build: refactor Arm build
> 
> 
> 
> > -----Original Message-----
> > From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> > Sent: Tuesday, October 27, 2020 5:56 AM
> > To: Juraj Linkeš <juraj.linkes@pantheon.tech>;
> > bruce.richardson@intel.com; Ruifeng Wang <Ruifeng.Wang@arm.com>; Phil
> > Yang <Phil.Yang@arm.com>; vcchunga@amazon.com; Dharmik Thakkar
> > <Dharmik.Thakkar@arm.com>; jerinjacobk@gmail.com;
> > hemant.agrawal@nxp.com
> > Cc: dev@dpdk.org; nd <nd@arm.com>; Honnappa Nagarahalli
> > <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>
> > Subject: RE: [PATCH v4 2/6] build: refactor Arm build
> >
> > <snip>
> > I think we need to split this further. Few suggestions below.
> >
> > >
> > > * Rename variables to have names that better describe what the
> > > variables store
> > This should be a separate commit.
> >
> > > * Remove unused or superfluous variables
> > Same here
> >
> > > * Change a list to dictionary where key lookup is needed
> > Same here
> >
> > > * Add informatory comments in the code
> >
> > > * Minor code restructure and reformatting
> > Same for this
> >
> 
> Ok, hopefully I'll be able to separate these cleanly.
> 
> > >
> > > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > ---
> > >  config/arm/arm64_armada_linux_gcc    |   2 +-
> > >  config/arm/arm64_armv8_linux_gcc     |   8 +-
> > >  config/arm/arm64_bluefield_linux_gcc |   4 +-
> > >  config/arm/arm64_dpaa_linux_gcc      |   2 +-
> > >  config/arm/arm64_emag_linux_gcc      |   2 +-
> > >  config/arm/arm64_n1sdp_linux_gcc     |   4 +-
> > >  config/arm/arm64_octeontx2_linux_gcc |   4 +-
> > >  config/arm/arm64_stingray_linux_gcc  |   4 +-
> > >  config/arm/arm64_thunderx2_linux_gcc |   4 +-
> > >  config/arm/arm64_thunderx_linux_gcc  |   2 +-
> > >  config/arm/meson.build               | 247 +++++++++++++++------------
> > >  11 files changed, 153 insertions(+), 130 deletions(-)
> > >
> > > diff --git a/config/arm/arm64_armada_linux_gcc
> > > b/config/arm/arm64_armada_linux_gcc
> > > index fa40c0398..52c5f4476 100644
> > > --- a/config/arm/arm64_armada_linux_gcc
> > > +++ b/config/arm/arm64_armada_linux_gcc
> > > @@ -14,4 +14,4 @@ cpu = 'armv8-a'
> > >  endian = 'little'
> > >
> > >  [properties]
> > > -implementor_id = '0x56'
> > > +implementer_id = '0x56'
> > Implementor and implementer mean the same. Looked at Arm specs, they
> > use 'implementer'. So, I am fine.
> >
> 
> That's where I got this and some of the other variable name changes from.
> 
> > > diff --git a/config/arm/arm64_armv8_linux_gcc
> > > b/config/arm/arm64_armv8_linux_gcc
> > > index 88f0ff9da..13ee8b223 100644
> > > --- a/config/arm/arm64_armv8_linux_gcc
> > > +++ b/config/arm/arm64_armv8_linux_gcc
> > > @@ -13,10 +13,10 @@ cpu = 'armv8-a'
> > >  endian = 'little'
> > >
> > >  [properties]
> > > -implementor_id = 'generic'
> > > +implementer_id = 'generic'
> > >
> > > -# Valid options for Arm's implementor_pn:
> > > -# 'default': valid for all armv8-a architectures (default value)
> > > +# Valid options for Arm's part_number:
> > > +# 'generic': valid for all armv8-a architectures (default value)
> > >  # '0xd03':   cortex-a53
> > >  # '0xd04':   cortex-a35
> > >  # '0xd05':   cortex-a55
> > > @@ -25,4 +25,4 @@ implementor_id = 'generic'
> > >  # '0xd09':   cortex-a73
> > >  # '0xd0a':   cortex-a75
> > >  # '0xd0b':   cortex-a76
> > > -implementor_pn = 'default'
> > > +part_number = 'generic'
> > Same here, Arm specs refer to this as 'PartNumber'. So, this should be fine.
> > I like 'generic' for part_number here.
> >
> > > diff --git a/config/arm/arm64_bluefield_linux_gcc
> > > b/config/arm/arm64_bluefield_linux_gcc
> > > index 86797d23c..b79389d85 100644
> > > --- a/config/arm/arm64_bluefield_linux_gcc
> > > +++ b/config/arm/arm64_bluefield_linux_gcc
> > > @@ -13,5 +13,5 @@ cpu = 'armv8-a'
> > >  endian = 'little'
> > >
> > >  [properties]
> > > -implementor_id = '0x41'
> > > -implementor_pn = '0xd08'
> > > +implementer_id = '0x41'
> > > +part_number = '0xd08'
> > > diff --git a/config/arm/arm64_dpaa_linux_gcc
> > > b/config/arm/arm64_dpaa_linux_gcc index 1a4682154..573ae7e42 100644
> > > --- a/config/arm/arm64_dpaa_linux_gcc
> > > +++ b/config/arm/arm64_dpaa_linux_gcc
> > > @@ -14,4 +14,4 @@ cpu = 'armv8-a'
> > >  endian = 'little'
> > >
> > >  [properties]
> > > -implementor_id = 'dpaa'
> > > +implementer_id = 'dpaa'
> > > diff --git a/config/arm/arm64_emag_linux_gcc
> > > b/config/arm/arm64_emag_linux_gcc index 8edcd3e97..24f3d533e 100644
> > > --- a/config/arm/arm64_emag_linux_gcc
> > > +++ b/config/arm/arm64_emag_linux_gcc
> > > @@ -13,4 +13,4 @@ cpu = 'armv8-a'
> > >  endian = 'little'
> > >
> > >  [properties]
> > > -implementor_id = '0x50'
> > > +implementer_id = '0x50'
> > > diff --git a/config/arm/arm64_n1sdp_linux_gcc
> > > b/config/arm/arm64_n1sdp_linux_gcc
> > > index 022e06303..6fb3f02ea 100644
> > > --- a/config/arm/arm64_n1sdp_linux_gcc
> > > +++ b/config/arm/arm64_n1sdp_linux_gcc
> > > @@ -13,5 +13,5 @@ cpu = 'armv8-a'
> > >  endian = 'little'
> > >
> > >  [properties]
> > > -implementor_id = '0x41'
> > > -implementor_pn = '0xd0c'
> > > +implementer_id = '0x41'
> > > +part_number = '0xd0c'
> > > diff --git a/config/arm/arm64_octeontx2_linux_gcc
> > > b/config/arm/arm64_octeontx2_linux_gcc
> > > index 365bd7cbd..ac1042806 100644
> > > --- a/config/arm/arm64_octeontx2_linux_gcc
> > > +++ b/config/arm/arm64_octeontx2_linux_gcc
> > > @@ -13,5 +13,5 @@ cpu = 'armv8-a'
> > >  endian = 'little'
> > >
> > >  [properties]
> > > -implementor_id = '0x43'
> > > -implementor_pn = '0xb2'
> > > +implementer_id = '0x43'
> > > +part_number = '0xb2'
> > > diff --git a/config/arm/arm64_stingray_linux_gcc
> > > b/config/arm/arm64_stingray_linux_gcc
> > > index 86797d23c..b79389d85 100644
> > > --- a/config/arm/arm64_stingray_linux_gcc
> > > +++ b/config/arm/arm64_stingray_linux_gcc
> > > @@ -13,5 +13,5 @@ cpu = 'armv8-a'
> > >  endian = 'little'
> > >
> > >  [properties]
> > > -implementor_id = '0x41'
> > > -implementor_pn = '0xd08'
> > > +implementer_id = '0x41'
> > > +part_number = '0xd08'
> > > diff --git a/config/arm/arm64_thunderx2_linux_gcc
> > > b/config/arm/arm64_thunderx2_linux_gcc
> > > index 2b41acc61..dd257745e 100644
> > > --- a/config/arm/arm64_thunderx2_linux_gcc
> > > +++ b/config/arm/arm64_thunderx2_linux_gcc
> > > @@ -13,5 +13,5 @@ cpu = 'armv8-a'
> > >  endian = 'little'
> > >
> > >  [properties]
> > > -implementor_id = '0x43'
> > > -implementor_pn = '0xaf'
> > > +implementer_id = '0x43'
> > > +part_number = '0xaf'
> > > diff --git a/config/arm/arm64_thunderx_linux_gcc
> > > b/config/arm/arm64_thunderx_linux_gcc
> > > index 6572ab615..670764437 100644
> > > --- a/config/arm/arm64_thunderx_linux_gcc
> > > +++ b/config/arm/arm64_thunderx_linux_gcc
> > > @@ -13,4 +13,4 @@ cpu = 'armv8-a'
> > >  endian = 'little'
> > >
> > >  [properties]
> > > -implementor_id = '0x43'
> > > +implementer_id = '0x43'
> > > diff --git a/config/arm/meson.build b/config/arm/meson.build index
> > > 491842cad..6c31ab167 100644
> > > --- a/config/arm/meson.build
> > > +++ b/config/arm/meson.build
> > > @@ -3,12 +3,12 @@
> > >  # Copyright(c) 2017 Cavium, Inc
> > >  # Copyright(c) 2020 PANTHEON.tech s.r.o.
> > >
> > > -# for checking defines we need to use the correct compiler flags
> > > -march_opt = '-march=@0@'.format(machine)
> > > -
> > > +# set arm_force_native_march if you want to use machine args below
> > > +# instead of discovered values; only works when doing an actual
> > > +native build
> > >  arm_force_native_march = false
> > > -arm_force_generic_march = (machine == 'generic')
> > > +native_machine_args = ['-march=native', '-mtune=native']
> > >
> > > +# common flags to all aarch64 builds, with lowest priority
> > >  flags_common_default = [
> > >  	# Accelarate rte_memcpy. Be sure to run unit test
> > > (memcpy_perf_autotest)
> > >  	# to determine the best threshold in code. Refer to notes in
> > > source file @@ -16,8 +16,9 @@ flags_common_default = [
> > >  	['RTE_ARCH_ARM64_MEMCPY', false],
> > >  	#	['RTE_ARM64_MEMCPY_ALIGNED_THRESHOLD', 2048],
> > >  	#	['RTE_ARM64_MEMCPY_UNALIGNED_THRESHOLD', 512],
> > > -	# Leave below RTE_ARM64_MEMCPY_xxx options commented out,
> > > unless there're
> > > -	# strong reasons.
> > > +
> > > +	# Leave below RTE_ARM64_MEMCPY_xxx options commented out,
> > > +	# unless there are strong reasons.
> > >  	#	['RTE_ARM64_MEMCPY_SKIP_GCC_VER_CHECK', false],
> > >  	#	['RTE_ARM64_MEMCPY_ALIGN_MASK', 0xF],
> > >  	#	['RTE_ARM64_MEMCPY_STRICT_ALIGN', false],
> > > @@ -28,184 +29,206 @@ flags_common_default = [
> > >
> > >  	['RTE_SCHED_VECTOR', false],
> > >  	['RTE_ARM_USE_WFE', false],
> > > +	['RTE_CACHE_LINE_SIZE', 128],
> > > +	['RTE_ARCH_ARM64', true]
> > >  ]
> > >
> > > +# implementer specific aarch64 flags, with middle priority # (will
> > > +overwrite common flags)
> > >  flags_generic = [
> > >  	['RTE_MACHINE', '"armv8a"'],
> > >  	['RTE_MAX_LCORE', 256],
> > >  	['RTE_USE_C11_MEM_MODEL', true],
> > > -	['RTE_CACHE_LINE_SIZE', 128]]
> > > +	['RTE_CACHE_LINE_SIZE', 128]
> > > +]
> > Any particular reason for this change? (and similar changes below)
> >
> 
> The first bracket is split from the second bracket so I did the same for the last
> two brackets. It makes it more apparent which brackets are paired, it's more
> consistent and also in line with how flags_common_default is formatted.
> 
> > >  flags_arm = [
> > >  	['RTE_MACHINE', '"armv8a"'],
> > >  	['RTE_MAX_LCORE', 16],
> > >  	['RTE_USE_C11_MEM_MODEL', true],
> > > -	['RTE_CACHE_LINE_SIZE', 64]]
> > > +	['RTE_CACHE_LINE_SIZE', 64]
> > > +]
> > >  flags_cavium = [
> > >  	['RTE_CACHE_LINE_SIZE', 128],
> > >  	['RTE_MAX_NUMA_NODES', 2],
> > >  	['RTE_MAX_LCORE', 96],
> > > -	['RTE_MAX_VFIO_GROUPS', 128]]
> > > +	['RTE_MAX_VFIO_GROUPS', 128]
> > > +]
> > >  flags_dpaa = [
> > >  	['RTE_MACHINE', '"dpaa"'],
> > >  	['RTE_USE_C11_MEM_MODEL', true],
> > >  	['RTE_CACHE_LINE_SIZE', 64],
> > >  	['RTE_MAX_NUMA_NODES', 1],
> > >  	['RTE_MAX_LCORE', 16],
> > > -	['RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', false]]
> > > +	['RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', false] ]
> > This is not needed
> >
> 
> Do you mean the space? It should be a line break. I'll check the exact characters,
> but I see this as adding a space in my local patch. Or do you mean the config
> option? It's set to true by default in config/meson.build and according to [1] it
> should be disabled.
> 
> [1] http://git.dpdk.org/dpdk/tree/config/defconfig_arm64-dpaa-linuxapp-
> gcc?h=v20.08
> 
> > >  flags_emag = [
> > >  	['RTE_MACHINE', '"emag"'],
> > > -	['RTE_CACHE_LINE_SIZE', 64],
> > >  	['RTE_MAX_NUMA_NODES', 1],
> > > -	['RTE_MAX_LCORE', 32]]
> > > +	['RTE_MAX_LCORE', 32],
> > > +	['RTE_CACHE_LINE_SIZE', 64]
> > > +]
> > >  flags_armada = [
> > >  	['RTE_MACHINE', '"armv8a"'],
> > > -	['RTE_CACHE_LINE_SIZE', 64],
> > >  	['RTE_MAX_NUMA_NODES', 1],
> > > -	['RTE_MAX_LCORE', 16]]
> > > +	['RTE_MAX_LCORE', 16],
> > > +	['RTE_CACHE_LINE_SIZE', 64]
> > > +]
> > Any reason for this change?
> >
> 
> The default (from flags_common_default) is 128 and I found here [0] that it
> should be set to 64 so I added it here. Should this also be in a separate patch
> (apart from those 4 already mention above)?
> 
> [0] http://git.dpdk.org/dpdk/tree/config/defconfig_arm64-armada-linuxapp-
> gcc?h=v20.08
> 

Sorry, I overlooked that I just moved the config around. I don't remember why. I could see that I wanted to make the order consistent across the different flag group, but that is not the case even with these changes. I'll make it consistent when I split the patch.

> > >
> > > -flags_default_extra = []
> > > +# part number specific aarch64 flags, with highest priority # (will
> > > +overwrite both common and implementer specific flags)
> > >  flags_n1sdp_extra = [
> > >  	['RTE_MACHINE', '"n1sdp"'],
> > >  	['RTE_MAX_NUMA_NODES', 1],
> > >  	['RTE_MAX_LCORE', 4],
> > >  	['RTE_EAL_NUMA_AWARE_HUGEPAGES', false],
> > > -	['RTE_LIBRTE_VHOST_NUMA', false]]
> > > +	['RTE_LIBRTE_VHOST_NUMA', false]
> > > +]
> > >  flags_thunderx_extra = [
> > >  	['RTE_MACHINE', '"thunderx"'],
> > > -	['RTE_USE_C11_MEM_MODEL', false]]
> > > +	['RTE_USE_C11_MEM_MODEL', false]
> > > +]
> > >  flags_thunderx2_extra = [
> > >  	['RTE_MACHINE', '"thunderx2"'],
> > >  	['RTE_CACHE_LINE_SIZE', 64],
> > >  	['RTE_MAX_NUMA_NODES', 2],
> > >  	['RTE_MAX_LCORE', 256],
> > >  	['RTE_ARM_FEATURE_ATOMICS', true],
> > > -	['RTE_USE_C11_MEM_MODEL', true]]
> > > +	['RTE_USE_C11_MEM_MODEL', true]
> > > +]
> > >  flags_octeontx2_extra = [
> > >  	['RTE_MACHINE', '"octeontx2"'],
> > >  	['RTE_MAX_NUMA_NODES', 1],
> > >  	['RTE_MAX_LCORE', 36],
> > >  	['RTE_ARM_FEATURE_ATOMICS', true],
> > >  	['RTE_EAL_IGB_UIO', false],
> > > -	['RTE_USE_C11_MEM_MODEL', true]]
> > > -
> > > -machine_args_default = [
> > > -	['default', ['-march=armv8-a+crc', '-moutline-atomics']],
> > > -	['native', ['-march=native']],
> > > -	['0xd03', ['-mcpu=cortex-a53']],
> > > -	['0xd04', ['-mcpu=cortex-a35']],
> > > -	['0xd07', ['-mcpu=cortex-a57']],
> > > -	['0xd08', ['-mcpu=cortex-a72']],
> > > -	['0xd09', ['-mcpu=cortex-a73']],
> > > -	['0xd0a', ['-mcpu=cortex-a75']],
> > > -	['0xd0b', ['-mcpu=cortex-a76']],
> > > -	['0xd0c', ['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'],
> > > flags_n1sdp_extra]]
> > > -
> > > -machine_args_cavium = [
> > > -	['default', ['-march=armv8-a+crc+crypto','-mcpu=thunderx']],
> > > -	['native', ['-march=native']],
> > > -	['0xa1', ['-mcpu=thunderxt88'], flags_thunderx_extra],
> > > -	['0xa2', ['-mcpu=thunderxt81'], flags_thunderx_extra],
> > > -	['0xa3', ['-mcpu=thunderxt83'], flags_thunderx_extra],
> > > -	['0xaf', ['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'],
> > > flags_thunderx2_extra],
> > > -	['0xb2', ['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'],
> > > flags_octeontx2_extra]]
> > > -
> > > -machine_args_emag = [
> > > -	['default', ['-march=armv8-a+crc+crypto', '-mtune=emag']],
> > > -	['native', ['-march=native']]]
> > > +	['RTE_USE_C11_MEM_MODEL', true]
> > > +]
> > > +# arm config (implementer 0x41) is the default config
> > > +pn_config_default
> > 'pn' here for 'part_number' is not consistent.
> >
> 
> Ok, I can rename it to part_number_config_default. Same for the other two pn
> variables.
> 
> > > += {
> > > +	'generic': [['-march=armv8-a+crc', '-moutline-atomics']],
> > > +	'0xd03': [['-mcpu=cortex-a53']],
> > > +	'0xd04': [['-mcpu=cortex-a35']],
> > > +	'0xd07': [['-mcpu=cortex-a57']],
> > > +	'0xd08': [['-mcpu=cortex-a72']],
> > > +	'0xd09': [['-mcpu=cortex-a73']],
> > > +	'0xd0a': [['-mcpu=cortex-a75']],
> > > +	'0xd0b': [['-mcpu=cortex-a76']],
> > > +	'0xd0c': [['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'],
> > > +flags_n1sdp_extra] } pn_config_cavium = {
> > > +	'generic': [['-march=armv8-a+crc+crypto', '-mcpu=thunderx']],
> > > +	'0xa1': [['-mcpu=thunderxt88'], flags_thunderx_extra],
> > > +	'0xa2': [['-mcpu=thunderxt81'], flags_thunderx_extra],
> > > +	'0xa3': [['-mcpu=thunderxt83'], flags_thunderx_extra],
> > > +	'0xaf': [['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'],
> > > flags_thunderx2_extra],
> > > +	'0xb2': [['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'],
> > > +flags_octeontx2_extra], } pn_config_emag = {
> > > +	'generic': [['-march=armv8-a+crc+crypto', '-mtune=emag']] }
> > >
> > >  ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page
> > > G7-5321) -impl_generic = ['Generic armv8', flags_generic,
> > > machine_args_default]
> > > -impl_0x41 = ['Arm', flags_arm, machine_args_default]
> > > -impl_0x42 = ['Broadcom', flags_generic, machine_args_default]
> > > -impl_0x43 = ['Cavium', flags_cavium, machine_args_cavium]
> > > -impl_0x44 = ['DEC', flags_generic, machine_args_default]
> > > -impl_0x49 = ['Infineon', flags_generic, machine_args_default]
> > > -impl_0x4d = ['Motorola', flags_generic, machine_args_default]
> > > -impl_0x4e = ['NVIDIA', flags_generic, machine_args_default]
> > > -impl_0x50 = ['Ampere Computing', flags_emag, machine_args_emag]
> > > -impl_0x51 = ['Qualcomm', flags_generic, machine_args_default]
> > > -impl_0x53 = ['Samsung', flags_generic, machine_args_default]
> > > -impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_default]
> > > -impl_0x69 = ['Intel', flags_generic, machine_args_default]
> > > -impl_dpaa = ['NXP DPAA', flags_dpaa, machine_args_default]
> > > +impl_generic = ['Generic armv8', flags_generic, pn_config_default]
> > > +impl_0x41 = ['Arm', flags_arm, pn_config_default]
> > > +impl_0x42 = ['Broadcom', flags_generic, pn_config_default]
> > > +impl_0x43 = ['Cavium', flags_cavium, pn_config_cavium]
> > > +impl_0x44 = ['DEC', flags_generic, pn_config_default]
> > > +impl_0x49 = ['Infineon', flags_generic, pn_config_default]
> > > +impl_0x4d = ['Motorola', flags_generic, pn_config_default]
> > > +impl_0x4e = ['NVIDIA', flags_generic, pn_config_default]
> > > +impl_0x50 = ['Ampere Computing', flags_emag, pn_config_emag]
> > > +impl_0x51 = ['Qualcomm', flags_generic, pn_config_default]
> > > +impl_0x53 = ['Samsung', flags_generic, pn_config_default]
> > > +impl_0x56 = ['Marvell ARMADA', flags_armada, pn_config_default]
> > > +impl_0x69 = ['Intel', flags_generic, pn_config_default] impl_dpaa =
> > > +['NXP DPAA', flags_dpaa, pn_config_default]
> > >
> > >  dpdk_conf.set('RTE_ARCH_ARM', 1)
> > >  dpdk_conf.set('RTE_FORCE_INTRINSICS', 1)
> > >
> > >  if dpdk_conf.get('RTE_ARCH_32')
> > > +	# armv7 build
> > >  	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
> > >  	dpdk_conf.set('RTE_ARCH_ARMv7', 1)
> > >  	# the minimum architecture supported, armv7-a, needs the following,
> > >  	# mk/machine/armv7a/rte.vars.mk sets it too
> > >  	machine_args += '-mfpu=neon'
> > >  else
> > > -	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
> > > -	dpdk_conf.set('RTE_ARCH_ARM64', 1)
> > > +	# aarch64 build
> > > +	if not meson.is_cross_build()
> > > +		if machine == 'generic'
> > > +			# default build
> > > +			impl_config = impl_generic
> > > +			part_number = 'generic'
> > > +		else
> > > +			# native build
> > > +			# The script returns ['Implementer', 'Variant',
> > > 'Architecture',
> > > +			# 'Primary Part number', 'Revision']
> > > +			detect_vendor = find_program(join_paths(
> > > +					meson.current_source_dir(),
> > > 'armv8_machine.py'))
> > > +			cmd = run_command(detect_vendor.path())
> > > +			if cmd.returncode() == 0
> > > +				cmd_output =
> > > cmd.stdout().to_lower().strip().split(' ')
> > > +			endif
> > > +			if arm_force_native_march == true
> > > +				part_number = 'native'
> > > +			else
> > > +				part_number = cmd_output[3]
> > > +			endif
> > > +			# Set to generic implementer if implementer is not
> > > found
> > > +			impl_config = get_variable('impl_' + cmd_output[0],
> > > 'impl_generic')
> > > +		endif
> > > +	else
> > > +		# cross build
> > > +		impl_id = meson.get_cross_property('implementer_id',
> > > 'generic')
> > > +		part_number = meson.get_cross_property('part_number',
> > > 'generic')
> > > +		impl_config = get_variable('impl_' + impl_id)
> > > +	endif
> > >
> > > -	machine = []
> > > -	cmd_generic = ['generic', '', '', 'default', '']
> > > -	cmd_output = cmd_generic # Set generic by default
> > > -	machine_args = [] # Clear previous machine args
> > > -	if arm_force_generic_march and not meson.is_cross_build()
> > > -		machine = impl_generic
> > > -		impl_pn = 'default'
> > > +	message('Arm implementer: ' + impl_config[0])
> > > +	message('Arm part number: ' + part_number)
> > > +
> > > +	implementer_flags = impl_config[1]
> > > +	part_number_config = impl_config[2]
> > > +
> > > +	if part_number_config.has_key(part_number)
> > > +		# use the specified part_number machine args if found
> > > +		part_number_config = part_number_config[part_number]
> > > +	elif part_number == 'native'
> > > +		# use native machine args
> > > +		part_number_config = [[native_machine_args]]
> > >  	elif not meson.is_cross_build()
> > > -		# The script returns ['Implementer', 'Variant', 'Architecture',
> > > -		# 'Primary Part number', 'Revision']
> > > -		detect_vendor = find_program(join_paths(
> > > -				meson.current_source_dir(),
> > > 'armv8_machine.py'))
> > > -		cmd = run_command(detect_vendor.path())
> > > -		if cmd.returncode() == 0
> > > -			cmd_output = cmd.stdout().to_lower().strip().split('
> > > ')
> > > -		endif
> > > -		# Set to generic if variable is not found
> > > -		machine = get_variable('impl_' + cmd_output[0], ['generic'])
> > > -		if machine[0] == 'generic'
> > > -			machine = impl_generic
> > > -			cmd_output = cmd_generic
> > > -		endif
> > > -		impl_pn = cmd_output[3]
> > > -		if arm_force_native_march == true
> > > -			impl_pn = 'native'
> > > -		endif
> > > +		# default to generic machine args if part_number is not found
> > > +		# and not forcing native machine args
> > > +		# but don't default in cross-builds; if part_number is specified
> > > +		# incorrectly in a cross-file, it needs to be fixed there
> > > +		part_number_config = part_number_config['generic']
> > >  	else
> > > -		impl_id = meson.get_cross_property('implementor_id',
> > > 'generic')
> > > -		impl_pn = meson.get_cross_property('implementor_pn',
> > > 'default')
> > > -		machine = get_variable('impl_' + impl_id)
> > > +		# cross build and part number is not in part_number_config
> > > +		error('Cross build part number 0@0 not
> > > found.'.format(part_number))
> > >  	endif
> > >
> > > -	# Apply Common Defaults. These settings may be overwritten by
> > > machine
> > > -	# settings later.
> > > -	foreach flag: flags_common_default
> > > -		if flag.length() > 0
> > > -			dpdk_conf.set(flag[0], flag[1])
> > > +	dpdk_flags = flags_common_default + implementer_flags
> > > +
> > > +	if part_number_config.length() > 1
> > > +		dpdk_flags += part_number_config[1]
> > > +	endif
> > > +
> > > +	machine_args = [] # Clear previous machine args
> > > +	foreach flag: part_number_config[0]
> > > +		if cc.has_argument(flag)
> > > +			machine_args += flag
> > >  		endif
> > >  	endforeach
> > >
> > > -	message('Implementer : ' + machine[0])
> > > -	foreach flag: machine[1]
> > > +	foreach flag: dpdk_flags
> > >  		if flag.length() > 0
> > >  			dpdk_conf.set(flag[0], flag[1])
> > >  		endif
> > >  	endforeach
> > > -
> > > -	foreach marg: machine[2]
> > > -		if marg[0] == impl_pn
> > > -			foreach flag: marg[1]
> > > -				if cc.has_argument(flag)
> > > -					machine_args += flag
> > > -				endif
> > > -			endforeach
> > > -			# Apply any extra machine specific flags.
> > > -			foreach flag: marg.get(2, flags_default_extra)
> > > -				if flag.length() > 0
> > > -					dpdk_conf.set(flag[0], flag[1])
> > > -				endif
> > > -			endforeach
> > > -		endif
> > > -	endforeach
> > >  endif
> > > -message(machine_args)
> > > +
> > > +message('Using machine args: @0@'.format(machine_args))
> > >
> > >  if (cc.get_define('__ARM_NEON', args: machine_args) != '' or
> > >      cc.get_define('__aarch64__', args: machine_args) != '')
> > > --
> > > 2.20.1

Re: [dpdk-dev] [PATCH v4 2/6] build: refactor Arm build

[Honnappa Nagarahalli]

<snip>

> >
> > > >  flags_emag = [
> > > >  	['RTE_MACHINE', '"emag"'],
> > > > -	['RTE_CACHE_LINE_SIZE', 64],
> > > >  	['RTE_MAX_NUMA_NODES', 1],
> > > > -	['RTE_MAX_LCORE', 32]]
> > > > +	['RTE_MAX_LCORE', 32],
> > > > +	['RTE_CACHE_LINE_SIZE', 64]
> > > > +]
> > > >  flags_armada = [
> > > >  	['RTE_MACHINE', '"armv8a"'],
> > > > -	['RTE_CACHE_LINE_SIZE', 64],
> > > >  	['RTE_MAX_NUMA_NODES', 1],
> > > > -	['RTE_MAX_LCORE', 16]]
> > > > +	['RTE_MAX_LCORE', 16],
> > > > +	['RTE_CACHE_LINE_SIZE', 64]
> > > > +]
> > > Any reason for this change?
> > >
> >
> > The default (from flags_common_default) is 128 and I found here [0]
> > that it should be set to 64 so I added it here. Should this also be in
> > a separate patch (apart from those 4 already mention above)?
> >
> > [0]
> > http://git.dpdk.org/dpdk/tree/config/defconfig_arm64-armada-linuxapp-
> > gcc?h=v20.08
> >
> 
> Sorry, I overlooked that I just moved the config around. I don't remember
> why. I could see that I wanted to make the order consistent across the
> different flag group, but that is not the case even with these changes. I'll
> make it consistent when I split the patch.
Ok

<snip>

Re: [dpdk-dev] [PATCH v4 2/6] build: refactor Arm build

[Honnappa Nagarahalli]

<snip>

> > I think we need to split this further. Few suggestions below.
> >
> > >
> > > * Rename variables to have names that better describe what the
> > > variables store
> > This should be a separate commit.
> >
> > > * Remove unused or superfluous variables
> > Same here
> >
> > > * Change a list to dictionary where key lookup is needed
> > Same here
> >
> > > * Add informatory comments in the code
> >
> > > * Minor code restructure and reformatting
> > Same for this
> >
> 
> Ok, hopefully I'll be able to separate these cleanly.
> 
> > >
> > > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > ---
> > >  config/arm/arm64_armada_linux_gcc    |   2 +-
> > >  config/arm/arm64_armv8_linux_gcc     |   8 +-
> > >  config/arm/arm64_bluefield_linux_gcc |   4 +-
> > >  config/arm/arm64_dpaa_linux_gcc      |   2 +-
> > >  config/arm/arm64_emag_linux_gcc      |   2 +-
> > >  config/arm/arm64_n1sdp_linux_gcc     |   4 +-
> > >  config/arm/arm64_octeontx2_linux_gcc |   4 +-
> > >  config/arm/arm64_stingray_linux_gcc  |   4 +-
> > >  config/arm/arm64_thunderx2_linux_gcc |   4 +-
> > >  config/arm/arm64_thunderx_linux_gcc  |   2 +-
> > >  config/arm/meson.build               | 247 +++++++++++++++------------
> > >  11 files changed, 153 insertions(+), 130 deletions(-)
> > >
> > > diff --git a/config/arm/arm64_armada_linux_gcc
> > > b/config/arm/arm64_armada_linux_gcc
> > > index fa40c0398..52c5f4476 100644
> > > --- a/config/arm/arm64_armada_linux_gcc
> > > +++ b/config/arm/arm64_armada_linux_gcc
> > > @@ -14,4 +14,4 @@ cpu = 'armv8-a'
> > >  endian = 'little'
> > >
> > >  [properties]
> > > -implementor_id = '0x56'
> > > +implementer_id = '0x56'
> > Implementor and implementer mean the same. Looked at Arm specs, they
> > use 'implementer'. So, I am fine.
> >
> 
> That's where I got this and some of the other variable name changes from.
> 
> > > diff --git a/config/arm/arm64_armv8_linux_gcc
> > > b/config/arm/arm64_armv8_linux_gcc
> > > index 88f0ff9da..13ee8b223 100644
> > > --- a/config/arm/arm64_armv8_linux_gcc
> > > +++ b/config/arm/arm64_armv8_linux_gcc
> > > @@ -13,10 +13,10 @@ cpu = 'armv8-a'
> > >  endian = 'little'
> > >
> > >  [properties]
> > > -implementor_id = 'generic'
> > > +implementer_id = 'generic'
> > >
> > > -# Valid options for Arm's implementor_pn:
> > > -# 'default': valid for all armv8-a architectures (default value)
> > > +# Valid options for Arm's part_number:
> > > +# 'generic': valid for all armv8-a architectures (default value)
> > >  # '0xd03':   cortex-a53
> > >  # '0xd04':   cortex-a35
> > >  # '0xd05':   cortex-a55
> > > @@ -25,4 +25,4 @@ implementor_id = 'generic'
> > >  # '0xd09':   cortex-a73
> > >  # '0xd0a':   cortex-a75
> > >  # '0xd0b':   cortex-a76
> > > -implementor_pn = 'default'
> > > +part_number = 'generic'
> > Same here, Arm specs refer to this as 'PartNumber'. So, this should be fine.
> > I like 'generic' for part_number here.
> >
> > > diff --git a/config/arm/arm64_bluefield_linux_gcc
> > > b/config/arm/arm64_bluefield_linux_gcc
> > > index 86797d23c..b79389d85 100644
> > > --- a/config/arm/arm64_bluefield_linux_gcc
> > > +++ b/config/arm/arm64_bluefield_linux_gcc
> > > @@ -13,5 +13,5 @@ cpu = 'armv8-a'
> > >  endian = 'little'
> > >

[...]

> > > diff --git a/config/arm/meson.build b/config/arm/meson.build index
> > > 491842cad..6c31ab167 100644
> > > --- a/config/arm/meson.build
> > > +++ b/config/arm/meson.build
> > > @@ -3,12 +3,12 @@
> > >  # Copyright(c) 2017 Cavium, Inc
> > >  # Copyright(c) 2020 PANTHEON.tech s.r.o.
> > >
> > > -# for checking defines we need to use the correct compiler flags
> > > -march_opt = '-march=@0@'.format(machine)
> > > -

[...]

> > >
> > > +# implementer specific aarch64 flags, with middle priority # (will
> > > +overwrite common flags)
> > >  flags_generic = [
> > >  	['RTE_MACHINE', '"armv8a"'],
> > >  	['RTE_MAX_LCORE', 256],
> > >  	['RTE_USE_C11_MEM_MODEL', true],
> > > -	['RTE_CACHE_LINE_SIZE', 128]]
> > > +	['RTE_CACHE_LINE_SIZE', 128]
> > > +]
> > Any particular reason for this change? (and similar changes below)
> >
> 
> The first bracket is split from the second bracket so I did the same for the last
> two brackets. It makes it more apparent which brackets are paired, it's more
> consistent and also in line with how flags_common_default is formatted.
Ok

> 

[...]

> > >  flags_dpaa = [
> > >  	['RTE_MACHINE', '"dpaa"'],
> > >  	['RTE_USE_C11_MEM_MODEL', true],
> > >  	['RTE_CACHE_LINE_SIZE', 64],
> > >  	['RTE_MAX_NUMA_NODES', 1],
> > >  	['RTE_MAX_LCORE', 16],
> > > -	['RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', false]]
> > > +	['RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', false] ]
> > This is not needed
> >
> 
> Do you mean the space? It should be a line break. I'll check the exact
I meant the space. I guess it needs to be on the next line.

> characters, but I see this as adding a space in my local patch. Or do you mean
> the config option? It's set to true by default in config/meson.build and
> according to [1] it should be disabled.
> 
> [1] http://git.dpdk.org/dpdk/tree/config/defconfig_arm64-dpaa-linuxapp-
> gcc?h=v20.08
> 
> > >  flags_emag = [
> > >  	['RTE_MACHINE', '"emag"'],
> > > -	['RTE_CACHE_LINE_SIZE', 64],
> > >  	['RTE_MAX_NUMA_NODES', 1],
> > > -	['RTE_MAX_LCORE', 32]]
> > > +	['RTE_MAX_LCORE', 32],
> > > +	['RTE_CACHE_LINE_SIZE', 64]
> > > +]
> > >  flags_armada = [
> > >  	['RTE_MACHINE', '"armv8a"'],
> > > -	['RTE_CACHE_LINE_SIZE', 64],
> > >  	['RTE_MAX_NUMA_NODES', 1],
> > > -	['RTE_MAX_LCORE', 16]]
> > > +	['RTE_MAX_LCORE', 16],
> > > +	['RTE_CACHE_LINE_SIZE', 64]
> > > +]
> > Any reason for this change?
> >
> 
> The default (from flags_common_default) is 128 and I found here [0] that it
> should be set to 64 so I added it here. Should this also be in a separate patch
> (apart from those 4 already mention above)?
The RTE_CACHE_LINE_SIZE is already set to 64 for 'flags_armada'. It looks like you have moved the line down.

> 
> [0] http://git.dpdk.org/dpdk/tree/config/defconfig_arm64-armada-
> linuxapp-gcc?h=v20.08
> 
> > >
> > > -flags_default_extra = []
> > > +# part number specific aarch64 flags, with highest priority # (will
> > > +overwrite both common and implementer specific flags)
> > >  flags_n1sdp_extra = [
> > >  	['RTE_MACHINE', '"n1sdp"'],
> > >  	['RTE_MAX_NUMA_NODES', 1],
> > >  	['RTE_MAX_LCORE', 4],
> > >  	['RTE_EAL_NUMA_AWARE_HUGEPAGES', false],
> > > -	['RTE_LIBRTE_VHOST_NUMA', false]]
> > > +	['RTE_LIBRTE_VHOST_NUMA', false]
> > > +]

<snip>

> > > -
> > > -machine_args_emag = [
> > > -	['default', ['-march=armv8-a+crc+crypto', '-mtune=emag']],
> > > -	['native', ['-march=native']]]
> > > +	['RTE_USE_C11_MEM_MODEL', true]
> > > +]
> > > +# arm config (implementer 0x41) is the default config
> > > +pn_config_default
> > 'pn' here for 'part_number' is not consistent.
> >
> 
> Ok, I can rename it to part_number_config_default. Same for the other two
> pn variables.
Ok
> 

<snip>

> > > --
> > > 2.20.1

Re: [dpdk-dev] [PATCH v4 2/6] build: refactor Arm build

[Honnappa Nagarahalli]

<snip>

> 
> * Rename variables to have names that better describe what the variables
> store
> * Remove unused or superfluous variables
> * Change a list to dictionary where key lookup is needed
> * Add informatory comments in the code
> * Minor code restructure and reformatting
> 
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
>  config/arm/arm64_armada_linux_gcc    |   2 +-
>  config/arm/arm64_armv8_linux_gcc     |   8 +-
>  config/arm/arm64_bluefield_linux_gcc |   4 +-
>  config/arm/arm64_dpaa_linux_gcc      |   2 +-
>  config/arm/arm64_emag_linux_gcc      |   2 +-
>  config/arm/arm64_n1sdp_linux_gcc     |   4 +-
>  config/arm/arm64_octeontx2_linux_gcc |   4 +-
>  config/arm/arm64_stingray_linux_gcc  |   4 +-
>  config/arm/arm64_thunderx2_linux_gcc |   4 +-
>  config/arm/arm64_thunderx_linux_gcc  |   2 +-
>  config/arm/meson.build               | 247 +++++++++++++++------------
>  11 files changed, 153 insertions(+), 130 deletions(-)
> 

<snip>

> diff --git a/config/arm/meson.build b/config/arm/meson.build index
> 491842cad..6c31ab167 100644
> --- a/config/arm/meson.build
> +++ b/config/arm/meson.build
> @@ -3,12 +3,12 @@
>  # Copyright(c) 2017 Cavium, Inc
>  # Copyright(c) 2020 PANTHEON.tech s.r.o.
> 
> -# for checking defines we need to use the correct compiler flags -march_opt
> = '-march=@0@'.format(machine)
> -
> +# set arm_force_native_march if you want to use machine args below #
> +instead of discovered values; only works when doing an actual native
> +build
>  arm_force_native_march = false
> -arm_force_generic_march = (machine == 'generic')
> +native_machine_args = ['-march=native', '-mtune=native']
> 

[...]

> -
> -machine_args_default = [
> -	['default', ['-march=armv8-a+crc', '-moutline-atomics']],
> -	['native', ['-march=native']],
> -	['0xd03', ['-mcpu=cortex-a53']],
> -	['0xd04', ['-mcpu=cortex-a35']],
> -	['0xd07', ['-mcpu=cortex-a57']],
> -	['0xd08', ['-mcpu=cortex-a72']],
> -	['0xd09', ['-mcpu=cortex-a73']],
> -	['0xd0a', ['-mcpu=cortex-a75']],
> -	['0xd0b', ['-mcpu=cortex-a76']],
> -	['0xd0c', ['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'],
> flags_n1sdp_extra]]
> -
> -machine_args_cavium = [
> -	['default', ['-march=armv8-a+crc+crypto','-mcpu=thunderx']],
> -	['native', ['-march=native']],
> -	['0xa1', ['-mcpu=thunderxt88'], flags_thunderx_extra],
> -	['0xa2', ['-mcpu=thunderxt81'], flags_thunderx_extra],
> -	['0xa3', ['-mcpu=thunderxt83'], flags_thunderx_extra],
> -	['0xaf', ['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'],
> flags_thunderx2_extra],
> -	['0xb2', ['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'],
> flags_octeontx2_extra]]
> -
> -machine_args_emag = [
> -	['default', ['-march=armv8-a+crc+crypto', '-mtune=emag']],
> -	['native', ['-march=native']]]
> +	['RTE_USE_C11_MEM_MODEL', true]
> +]
> +# arm config (implementer 0x41) is the default config pn_config_default
What does it mean by 'default' here? I am somewhat confused between 'default' and 'generic'. We should look to remove 'default' as much as possible and stick with 'generic'.

> += {
> +	'generic': [['-march=armv8-a+crc', '-moutline-atomics']],
I like that we have taken out 'native' from this list. Would it be possible to take out 'generic' from this and others below. This is because the binary built with 'generic' build should run on any Arm platform. There is no dependency on any underlying platform.

> +	'0xd03': [['-mcpu=cortex-a53']],
> +	'0xd04': [['-mcpu=cortex-a35']],
> +	'0xd07': [['-mcpu=cortex-a57']],
> +	'0xd08': [['-mcpu=cortex-a72']],
> +	'0xd09': [['-mcpu=cortex-a73']],
> +	'0xd0a': [['-mcpu=cortex-a75']],
> +	'0xd0b': [['-mcpu=cortex-a76']],
> +	'0xd0c': [['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'],
> +flags_n1sdp_extra]
'flags_n1sdp_extra' does not fit here. For the part number '0xd0c' there could be multiple SoCs (N1SDP is one of them). So, if the SoC is not known, but if we know that the CPU is N1, then we should build a N1-Generic build.
So, from my perspective, there are 3 kinds of binaries:
1) generic - best portability -  (possibly) least optimized for a given platform
2) Core-Generic (for ex: N1-generic) - Portable on all N1 based SoCs only - Optimized for N1 cores
3) SoC specific - (possibly) Not portable - Most optimized for the SoC

 } pn_config_cavium = {
> +	'generic': [['-march=armv8-a+crc+crypto', '-mcpu=thunderx']],
'generic' does not make sense here.

> +	'0xa1': [['-mcpu=thunderxt88'], flags_thunderx_extra],
> +	'0xa2': [['-mcpu=thunderxt81'], flags_thunderx_extra],
> +	'0xa3': [['-mcpu=thunderxt83'], flags_thunderx_extra],
> +	'0xaf': [['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'],
> flags_thunderx2_extra],
> +	'0xb2': [['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'],
> +flags_octeontx2_extra], } pn_config_emag = {
> +	'generic': [['-march=armv8-a+crc+crypto', '-mtune=emag']] }
Same here.
I understand that this is coming from the existing code. But, I think we should try to set it right.

> 
>  ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page G7-5321)
Nit, Would be good to remove the reference to the doc

> -impl_generic = ['Generic armv8', flags_generic, machine_args_default]
> -impl_0x41 = ['Arm', flags_arm, machine_args_default]
> -impl_0x42 = ['Broadcom', flags_generic, machine_args_default]
> -impl_0x43 = ['Cavium', flags_cavium, machine_args_cavium]
> -impl_0x44 = ['DEC', flags_generic, machine_args_default]
> -impl_0x49 = ['Infineon', flags_generic, machine_args_default] -impl_0x4d =
> ['Motorola', flags_generic, machine_args_default] -impl_0x4e = ['NVIDIA',
> flags_generic, machine_args_default]
> -impl_0x50 = ['Ampere Computing', flags_emag, machine_args_emag]
> -impl_0x51 = ['Qualcomm', flags_generic, machine_args_default]
> -impl_0x53 = ['Samsung', flags_generic, machine_args_default]
> -impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_default]
> -impl_0x69 = ['Intel', flags_generic, machine_args_default] -impl_dpaa =
> ['NXP DPAA', flags_dpaa, machine_args_default]
> +impl_generic = ['Generic armv8', flags_generic, pn_config_default]
> +impl_0x41 = ['Arm', flags_arm, pn_config_default]
> +impl_0x42 = ['Broadcom', flags_generic, pn_config_default]
> +impl_0x43 = ['Cavium', flags_cavium, pn_config_cavium]
> +impl_0x44 = ['DEC', flags_generic, pn_config_default]
> +impl_0x49 = ['Infineon', flags_generic, pn_config_default] impl_0x4d =
> +['Motorola', flags_generic, pn_config_default] impl_0x4e = ['NVIDIA',
> +flags_generic, pn_config_default]
> +impl_0x50 = ['Ampere Computing', flags_emag, pn_config_emag]
> +impl_0x51 = ['Qualcomm', flags_generic, pn_config_default]
> +impl_0x53 = ['Samsung', flags_generic, pn_config_default]
> +impl_0x56 = ['Marvell ARMADA', flags_armada, pn_config_default]
> +impl_0x69 = ['Intel', flags_generic, pn_config_default] impl_dpaa =
> +['NXP DPAA', flags_dpaa, pn_config_default]
> 
>  dpdk_conf.set('RTE_ARCH_ARM', 1)
>  dpdk_conf.set('RTE_FORCE_INTRINSICS', 1)
> 
>  if dpdk_conf.get('RTE_ARCH_32')
> +	# armv7 build
>  	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
>  	dpdk_conf.set('RTE_ARCH_ARMv7', 1)
>  	# the minimum architecture supported, armv7-a, needs the
> following,
>  	# mk/machine/armv7a/rte.vars.mk sets it too
>  	machine_args += '-mfpu=neon'
>  else
> -	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
> -	dpdk_conf.set('RTE_ARCH_ARM64', 1)
> +	# aarch64 build
> +	if not meson.is_cross_build()
> +		if machine == 'generic'
> +			# default build
                                              ^^^^^^^^^^^ Generic build?

> +			impl_config = impl_generic
> +			part_number = 'generic'
> +		else
> +			# native build
> +			# The script returns ['Implementer', 'Variant',
> 'Architecture',
> +			# 'Primary Part number', 'Revision']
> +			detect_vendor = find_program(join_paths(
> +					meson.current_source_dir(),
> 'armv8_machine.py'))
> +			cmd = run_command(detect_vendor.path())
> +			if cmd.returncode() == 0
> +				cmd_output =
> cmd.stdout().to_lower().strip().split(' ')
> +			endif
> +			if arm_force_native_march == true
> +				part_number = 'native'
> +			else
> +				part_number = cmd_output[3]
> +			endif
> +			# Set to generic implementer if implementer is not
> found
> +			impl_config = get_variable('impl_' + cmd_output[0],
> 'impl_generic')
> +		endif
> +	else
> +		# cross build
> +		impl_id = meson.get_cross_property('implementer_id',
> 'generic')
> +		part_number = meson.get_cross_property('part_number',
> 'generic')
> +		impl_config = get_variable('impl_' + impl_id)
> +	endif
> 
> -	machine = []
> -	cmd_generic = ['generic', '', '', 'default', '']
> -	cmd_output = cmd_generic # Set generic by default
> -	machine_args = [] # Clear previous machine args
> -	if arm_force_generic_march and not meson.is_cross_build()
> -		machine = impl_generic
> -		impl_pn = 'default'
> +	message('Arm implementer: ' + impl_config[0])
> +	message('Arm part number: ' + part_number)
> +
> +	implementer_flags = impl_config[1]
> +	part_number_config = impl_config[2]
> +
> +	if part_number_config.has_key(part_number)
> +		# use the specified part_number machine args if found
> +		part_number_config = part_number_config[part_number]
> +	elif part_number == 'native'
> +		# use native machine args
> +		part_number_config = [[native_machine_args]]
>  	elif not meson.is_cross_build()
> -		# The script returns ['Implementer', 'Variant', 'Architecture',
> -		# 'Primary Part number', 'Revision']
> -		detect_vendor = find_program(join_paths(
> -				meson.current_source_dir(),
> 'armv8_machine.py'))
> -		cmd = run_command(detect_vendor.path())
> -		if cmd.returncode() == 0
> -			cmd_output = cmd.stdout().to_lower().strip().split('
> ')
> -		endif
> -		# Set to generic if variable is not found
> -		machine = get_variable('impl_' + cmd_output[0], ['generic'])
> -		if machine[0] == 'generic'
> -			machine = impl_generic
> -			cmd_output = cmd_generic
> -		endif
> -		impl_pn = cmd_output[3]
> -		if arm_force_native_march == true
> -			impl_pn = 'native'
> -		endif
> +		# default to generic machine args if part_number is not found
> +		# and not forcing native machine args
> +		# but don't default in cross-builds; if part_number is specified
> +		# incorrectly in a cross-file, it needs to be fixed there
> +		part_number_config = part_number_config['generic']
>  	else
> -		impl_id = meson.get_cross_property('implementor_id',
> 'generic')
> -		impl_pn = meson.get_cross_property('implementor_pn',
> 'default')
> -		machine = get_variable('impl_' + impl_id)
> +		# cross build and part number is not in part_number_config
> +		error('Cross build part number 0@0 not
> found.'.format(part_number))
>  	endif
> 
> -	# Apply Common Defaults. These settings may be overwritten by
> machine
> -	# settings later.
> -	foreach flag: flags_common_default
> -		if flag.length() > 0
> -			dpdk_conf.set(flag[0], flag[1])
> +	dpdk_flags = flags_common_default + implementer_flags
> +
> +	if part_number_config.length() > 1
> +		dpdk_flags += part_number_config[1]
> +	endif
> +
> +	machine_args = [] # Clear previous machine args
> +	foreach flag: part_number_config[0]
> +		if cc.has_argument(flag)
> +			machine_args += flag
>  		endif
>  	endforeach
> 
> -	message('Implementer : ' + machine[0])
> -	foreach flag: machine[1]
> +	foreach flag: dpdk_flags
>  		if flag.length() > 0
>  			dpdk_conf.set(flag[0], flag[1])
>  		endif
>  	endforeach
> -
> -	foreach marg: machine[2]
> -		if marg[0] == impl_pn
> -			foreach flag: marg[1]
> -				if cc.has_argument(flag)
> -					machine_args += flag
> -				endif
> -			endforeach
> -			# Apply any extra machine specific flags.
> -			foreach flag: marg.get(2, flags_default_extra)
> -				if flag.length() > 0
> -					dpdk_conf.set(flag[0], flag[1])
> -				endif
> -			endforeach
> -		endif
> -	endforeach
>  endif
> -message(machine_args)
> +
> +message('Using machine args: @0@'.format(machine_args))
> 
>  if (cc.get_define('__ARM_NEON', args: machine_args) != '' or
>      cc.get_define('__aarch64__', args: machine_args) != '')
> --
> 2.20.1

Re: [dpdk-dev] [PATCH v4 2/6] build: refactor Arm build

[Juraj Linkeš]

> -----Original Message-----
> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Sent: Wednesday, October 28, 2020 5:59 PM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>; bruce.richardson@intel.com;
> Ruifeng Wang <Ruifeng.Wang@arm.com>; Phil Yang <Phil.Yang@arm.com>;
> vcchunga@amazon.com; Dharmik Thakkar <Dharmik.Thakkar@arm.com>;
> jerinjacobk@gmail.com; hemant.agrawal@nxp.com
> Cc: dev@dpdk.org; nd <nd@arm.com>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>
> Subject: RE: [PATCH v4 2/6] build: refactor Arm build
> 
> <snip>
> 
> >
> > * Rename variables to have names that better describe what the
> > variables store
> > * Remove unused or superfluous variables
> > * Change a list to dictionary where key lookup is needed
> > * Add informatory comments in the code
> > * Minor code restructure and reformatting
> >
> > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > ---
> >  config/arm/arm64_armada_linux_gcc    |   2 +-
> >  config/arm/arm64_armv8_linux_gcc     |   8 +-
> >  config/arm/arm64_bluefield_linux_gcc |   4 +-
> >  config/arm/arm64_dpaa_linux_gcc      |   2 +-
> >  config/arm/arm64_emag_linux_gcc      |   2 +-
> >  config/arm/arm64_n1sdp_linux_gcc     |   4 +-
> >  config/arm/arm64_octeontx2_linux_gcc |   4 +-
> >  config/arm/arm64_stingray_linux_gcc  |   4 +-
> >  config/arm/arm64_thunderx2_linux_gcc |   4 +-
> >  config/arm/arm64_thunderx_linux_gcc  |   2 +-
> >  config/arm/meson.build               | 247 +++++++++++++++------------
> >  11 files changed, 153 insertions(+), 130 deletions(-)
> >
> 
> <snip>
> 
> > diff --git a/config/arm/meson.build b/config/arm/meson.build index
> > 491842cad..6c31ab167 100644
> > --- a/config/arm/meson.build
> > +++ b/config/arm/meson.build
> > @@ -3,12 +3,12 @@
> >  # Copyright(c) 2017 Cavium, Inc
> >  # Copyright(c) 2020 PANTHEON.tech s.r.o.
> >
> > -# for checking defines we need to use the correct compiler flags
> > -march_opt = '-march=@0@'.format(machine)
> > -
> > +# set arm_force_native_march if you want to use machine args below #
> > +instead of discovered values; only works when doing an actual native
> > +build
> >  arm_force_native_march = false
> > -arm_force_generic_march = (machine == 'generic')
> > +native_machine_args = ['-march=native', '-mtune=native']
> >
> 
> [...]
> 
> > -
> > -machine_args_default = [
> > -	['default', ['-march=armv8-a+crc', '-moutline-atomics']],
> > -	['native', ['-march=native']],
> > -	['0xd03', ['-mcpu=cortex-a53']],
> > -	['0xd04', ['-mcpu=cortex-a35']],
> > -	['0xd07', ['-mcpu=cortex-a57']],
> > -	['0xd08', ['-mcpu=cortex-a72']],
> > -	['0xd09', ['-mcpu=cortex-a73']],
> > -	['0xd0a', ['-mcpu=cortex-a75']],
> > -	['0xd0b', ['-mcpu=cortex-a76']],
> > -	['0xd0c', ['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'],
> > flags_n1sdp_extra]]
> > -
> > -machine_args_cavium = [
> > -	['default', ['-march=armv8-a+crc+crypto','-mcpu=thunderx']],
> > -	['native', ['-march=native']],
> > -	['0xa1', ['-mcpu=thunderxt88'], flags_thunderx_extra],
> > -	['0xa2', ['-mcpu=thunderxt81'], flags_thunderx_extra],
> > -	['0xa3', ['-mcpu=thunderxt83'], flags_thunderx_extra],
> > -	['0xaf', ['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'],
> > flags_thunderx2_extra],
> > -	['0xb2', ['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'],
> > flags_octeontx2_extra]]
> > -
> > -machine_args_emag = [
> > -	['default', ['-march=armv8-a+crc+crypto', '-mtune=emag']],
> > -	['native', ['-march=native']]]
> > +	['RTE_USE_C11_MEM_MODEL', true]
> > +]
> > +# arm config (implementer 0x41) is the default config
> > +pn_config_default
> What does it mean by 'default' here? I am somewhat confused between 'default'
> and 'generic'. We should look to remove 'default' as much as possible and stick
> with 'generic'.
> 

This default means what default means, no special meaning, that is if there isn't a more specific configuration, default to this one. It's possible that generic is better, but now that I think about it, using something else than default or generic might be the best to avoid confusion. Possibly just part_number_arm, which will make it in line with the other var names.

> > += {
> > +	'generic': [['-march=armv8-a+crc', '-moutline-atomics']],
> I like that we have taken out 'native' from this list. Would it be possible to take
> out 'generic' from this and others below. This is because the binary built with
> 'generic' build should run on any Arm platform. There is no dependency on any
> underlying platform.
> 

This actually means generic part for the implementer, not generic for everything. I understand this is here to produce binaries that would run on everything from that impelemeter (in line of what you mention below, this would be implementer-generic configuration, a fourth category). In my patchset, it's also a fallback when building for an unknown part number from the implementer.

Since this is not generic for everything, only for the implementer, we're lacking the true common default machine args for everything.

> > +	'0xd03': [['-mcpu=cortex-a53']],
> > +	'0xd04': [['-mcpu=cortex-a35']],
> > +	'0xd07': [['-mcpu=cortex-a57']],
> > +	'0xd08': [['-mcpu=cortex-a72']],
> > +	'0xd09': [['-mcpu=cortex-a73']],
> > +	'0xd0a': [['-mcpu=cortex-a75']],
> > +	'0xd0b': [['-mcpu=cortex-a76']],
> > +	'0xd0c': [['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'],
> > +flags_n1sdp_extra]
> 'flags_n1sdp_extra' does not fit here. For the part number '0xd0c' there could be
> multiple SoCs (N1SDP is one of them). So, if the SoC is not known, but if we
> know that the CPU is N1, then we should build a N1-Generic build.

This should be core-generic configuration, I can rename it to flags_n1generic_extra.

> So, from my perspective, there are 3 kinds of binaries:
> 1) generic - best portability -  (possibly) least optimized for a given platform
> 2) Core-Generic (for ex: N1-generic) - Portable on all N1 based SoCs only -
> Optimized for N1 cores
> 3) SoC specific - (possibly) Not portable - Most optimized for the SoC
> 

The fourth category I mentioned above, implementer-generic, is used in how the arm configuration is currently processed:
1) default configuration
Added to/overwritten by
2) implementer configuration (one of which is the configuration for generic build)
Added to/overwritten by
3) part number (or core-generic) configuration
Added to/overwritten by
4) SoC configuration (what we're adding now)

It's not organized as cleanly - the machine args contain both 2) and 3). Depending on what we want to do with the 'generic' part number I'll adjust the config organization.

>  } pn_config_cavium = {
> > +	'generic': [['-march=armv8-a+crc+crypto', '-mcpu=thunderx']],
> 'generic' does not make sense here.
> 
> > +	'0xa1': [['-mcpu=thunderxt88'], flags_thunderx_extra],
> > +	'0xa2': [['-mcpu=thunderxt81'], flags_thunderx_extra],
> > +	'0xa3': [['-mcpu=thunderxt83'], flags_thunderx_extra],
> > +	'0xaf': [['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'],
> > flags_thunderx2_extra],
> > +	'0xb2': [['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'],
> > +flags_octeontx2_extra], } pn_config_emag = {
> > +	'generic': [['-march=armv8-a+crc+crypto', '-mtune=emag']] }
> Same here.
> I understand that this is coming from the existing code. But, I think we should try
> to set it right.
> 

The generic config for these makes sense if a fourth category, implementer-generic makes sense.
For example, for part_number_config_emag this means: for implementer _0x50 (Ampere Computing) use the generic machine args if there's nothing more specific (which there isn't - no other part number).

> >
> >  ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page
> > G7-5321)
> Nit, Would be good to remove the reference to the doc
> 

Is that because the docs mentioned here may not be the most recent? It was useful to me in understanding where the implementer/part number values come from.

> > -impl_generic = ['Generic armv8', flags_generic, machine_args_default]
> > -impl_0x41 = ['Arm', flags_arm, machine_args_default]
> > -impl_0x42 = ['Broadcom', flags_generic, machine_args_default]
> > -impl_0x43 = ['Cavium', flags_cavium, machine_args_cavium]
> > -impl_0x44 = ['DEC', flags_generic, machine_args_default]
> > -impl_0x49 = ['Infineon', flags_generic, machine_args_default]
> > -impl_0x4d = ['Motorola', flags_generic, machine_args_default]
> > -impl_0x4e = ['NVIDIA', flags_generic, machine_args_default]
> > -impl_0x50 = ['Ampere Computing', flags_emag, machine_args_emag]
> > -impl_0x51 = ['Qualcomm', flags_generic, machine_args_default]
> > -impl_0x53 = ['Samsung', flags_generic, machine_args_default]
> > -impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_default]
> > -impl_0x69 = ['Intel', flags_generic, machine_args_default] -impl_dpaa
> > = ['NXP DPAA', flags_dpaa, machine_args_default]
> > +impl_generic = ['Generic armv8', flags_generic, pn_config_default]
> > +impl_0x41 = ['Arm', flags_arm, pn_config_default]
> > +impl_0x42 = ['Broadcom', flags_generic, pn_config_default]
> > +impl_0x43 = ['Cavium', flags_cavium, pn_config_cavium]
> > +impl_0x44 = ['DEC', flags_generic, pn_config_default]
> > +impl_0x49 = ['Infineon', flags_generic, pn_config_default] impl_0x4d
> > += ['Motorola', flags_generic, pn_config_default] impl_0x4e =
> > +['NVIDIA', flags_generic, pn_config_default]
> > +impl_0x50 = ['Ampere Computing', flags_emag, pn_config_emag]
> > +impl_0x51 = ['Qualcomm', flags_generic, pn_config_default]
> > +impl_0x53 = ['Samsung', flags_generic, pn_config_default]
> > +impl_0x56 = ['Marvell ARMADA', flags_armada, pn_config_default]
> > +impl_0x69 = ['Intel', flags_generic, pn_config_default] impl_dpaa =
> > +['NXP DPAA', flags_dpaa, pn_config_default]
> >
> >  dpdk_conf.set('RTE_ARCH_ARM', 1)
> >  dpdk_conf.set('RTE_FORCE_INTRINSICS', 1)
> >
> >  if dpdk_conf.get('RTE_ARCH_32')
> > +	# armv7 build
> >  	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
> >  	dpdk_conf.set('RTE_ARCH_ARMv7', 1)
> >  	# the minimum architecture supported, armv7-a, needs the following,
> >  	# mk/machine/armv7a/rte.vars.mk sets it too
> >  	machine_args += '-mfpu=neon'
> >  else
> > -	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
> > -	dpdk_conf.set('RTE_ARCH_ARM64', 1)
> > +	# aarch64 build
> > +	if not meson.is_cross_build()
> > +		if machine == 'generic'
> > +			# default build
>                                               ^^^^^^^^^^^ Generic build?
> 

Already addressed in the next version.

> > +			impl_config = impl_generic
> > +			part_number = 'generic'
> > +		else
> > +			# native build
> > +			# The script returns ['Implementer', 'Variant',
> > 'Architecture',
> > +			# 'Primary Part number', 'Revision']
> > +			detect_vendor = find_program(join_paths(
> > +					meson.current_source_dir(),
> > 'armv8_machine.py'))
> > +			cmd = run_command(detect_vendor.path())
> > +			if cmd.returncode() == 0
> > +				cmd_output =
> > cmd.stdout().to_lower().strip().split(' ')
> > +			endif
> > +			if arm_force_native_march == true
> > +				part_number = 'native'
> > +			else
> > +				part_number = cmd_output[3]
> > +			endif
> > +			# Set to generic implementer if implementer is not
> > found
> > +			impl_config = get_variable('impl_' + cmd_output[0],
> > 'impl_generic')
> > +		endif
> > +	else
> > +		# cross build
> > +		impl_id = meson.get_cross_property('implementer_id',
> > 'generic')
> > +		part_number = meson.get_cross_property('part_number',
> > 'generic')
> > +		impl_config = get_variable('impl_' + impl_id)
> > +	endif
> >
> > -	machine = []
> > -	cmd_generic = ['generic', '', '', 'default', '']
> > -	cmd_output = cmd_generic # Set generic by default
> > -	machine_args = [] # Clear previous machine args
> > -	if arm_force_generic_march and not meson.is_cross_build()
> > -		machine = impl_generic
> > -		impl_pn = 'default'
> > +	message('Arm implementer: ' + impl_config[0])
> > +	message('Arm part number: ' + part_number)
> > +
> > +	implementer_flags = impl_config[1]
> > +	part_number_config = impl_config[2]
> > +
> > +	if part_number_config.has_key(part_number)
> > +		# use the specified part_number machine args if found
> > +		part_number_config = part_number_config[part_number]
> > +	elif part_number == 'native'
> > +		# use native machine args
> > +		part_number_config = [[native_machine_args]]
> >  	elif not meson.is_cross_build()
> > -		# The script returns ['Implementer', 'Variant', 'Architecture',
> > -		# 'Primary Part number', 'Revision']
> > -		detect_vendor = find_program(join_paths(
> > -				meson.current_source_dir(),
> > 'armv8_machine.py'))
> > -		cmd = run_command(detect_vendor.path())
> > -		if cmd.returncode() == 0
> > -			cmd_output = cmd.stdout().to_lower().strip().split('
> > ')
> > -		endif
> > -		# Set to generic if variable is not found
> > -		machine = get_variable('impl_' + cmd_output[0], ['generic'])
> > -		if machine[0] == 'generic'
> > -			machine = impl_generic
> > -			cmd_output = cmd_generic
> > -		endif
> > -		impl_pn = cmd_output[3]
> > -		if arm_force_native_march == true
> > -			impl_pn = 'native'
> > -		endif
> > +		# default to generic machine args if part_number is not found
> > +		# and not forcing native machine args
> > +		# but don't default in cross-builds; if part_number is specified
> > +		# incorrectly in a cross-file, it needs to be fixed there
> > +		part_number_config = part_number_config['generic']
> >  	else
> > -		impl_id = meson.get_cross_property('implementor_id',
> > 'generic')
> > -		impl_pn = meson.get_cross_property('implementor_pn',
> > 'default')
> > -		machine = get_variable('impl_' + impl_id)
> > +		# cross build and part number is not in part_number_config
> > +		error('Cross build part number 0@0 not
> > found.'.format(part_number))
> >  	endif
> >
> > -	# Apply Common Defaults. These settings may be overwritten by
> > machine
> > -	# settings later.
> > -	foreach flag: flags_common_default
> > -		if flag.length() > 0
> > -			dpdk_conf.set(flag[0], flag[1])
> > +	dpdk_flags = flags_common_default + implementer_flags
> > +
> > +	if part_number_config.length() > 1
> > +		dpdk_flags += part_number_config[1]
> > +	endif
> > +
> > +	machine_args = [] # Clear previous machine args
> > +	foreach flag: part_number_config[0]
> > +		if cc.has_argument(flag)
> > +			machine_args += flag
> >  		endif
> >  	endforeach
> >
> > -	message('Implementer : ' + machine[0])
> > -	foreach flag: machine[1]
> > +	foreach flag: dpdk_flags
> >  		if flag.length() > 0
> >  			dpdk_conf.set(flag[0], flag[1])
> >  		endif
> >  	endforeach
> > -
> > -	foreach marg: machine[2]
> > -		if marg[0] == impl_pn
> > -			foreach flag: marg[1]
> > -				if cc.has_argument(flag)
> > -					machine_args += flag
> > -				endif
> > -			endforeach
> > -			# Apply any extra machine specific flags.
> > -			foreach flag: marg.get(2, flags_default_extra)
> > -				if flag.length() > 0
> > -					dpdk_conf.set(flag[0], flag[1])
> > -				endif
> > -			endforeach
> > -		endif
> > -	endforeach
> >  endif
> > -message(machine_args)
> > +
> > +message('Using machine args: @0@'.format(machine_args))
> >
> >  if (cc.get_define('__ARM_NEON', args: machine_args) != '' or
> >      cc.get_define('__aarch64__', args: machine_args) != '')
> > --
> > 2.20.1

Re: [dpdk-dev] [PATCH v4 2/6] build: refactor Arm build

[Honnappa Nagarahalli]

> > <snip>
> >
> > > diff --git a/config/arm/meson.build b/config/arm/meson.build index
> > > 491842cad..6c31ab167 100644
> > > --- a/config/arm/meson.build
> > > +++ b/config/arm/meson.build
> > > @@ -3,12 +3,12 @@
> > >  # Copyright(c) 2017 Cavium, Inc
> > >  # Copyright(c) 2020 PANTHEON.tech s.r.o.
> > >
> > > -# for checking defines we need to use the correct compiler flags
> > > -march_opt = '-march=@0@'.format(machine)
> > > -
> > > +# set arm_force_native_march if you want to use machine args below
> > > +# instead of discovered values; only works when doing an actual
> > > +native build
> > >  arm_force_native_march = false
> > > -arm_force_generic_march = (machine == 'generic')
> > > +native_machine_args = ['-march=native', '-mtune=native']
> > >
> >
> > [...]
> >
> > > -
> > > -machine_args_default = [
> > > -	['default', ['-march=armv8-a+crc', '-moutline-atomics']],
> > > -	['native', ['-march=native']],
> > > -	['0xd03', ['-mcpu=cortex-a53']],
> > > -	['0xd04', ['-mcpu=cortex-a35']],
> > > -	['0xd07', ['-mcpu=cortex-a57']],
> > > -	['0xd08', ['-mcpu=cortex-a72']],
> > > -	['0xd09', ['-mcpu=cortex-a73']],
> > > -	['0xd0a', ['-mcpu=cortex-a75']],
> > > -	['0xd0b', ['-mcpu=cortex-a76']],
> > > -	['0xd0c', ['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'],
> > > flags_n1sdp_extra]]
> > > -
> > > -machine_args_cavium = [
> > > -	['default', ['-march=armv8-a+crc+crypto','-mcpu=thunderx']],
> > > -	['native', ['-march=native']],
> > > -	['0xa1', ['-mcpu=thunderxt88'], flags_thunderx_extra],
> > > -	['0xa2', ['-mcpu=thunderxt81'], flags_thunderx_extra],
> > > -	['0xa3', ['-mcpu=thunderxt83'], flags_thunderx_extra],
> > > -	['0xaf', ['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'],
> > > flags_thunderx2_extra],
> > > -	['0xb2', ['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'],
> > > flags_octeontx2_extra]]
> > > -
> > > -machine_args_emag = [
> > > -	['default', ['-march=armv8-a+crc+crypto', '-mtune=emag']],
> > > -	['native', ['-march=native']]]
> > > +	['RTE_USE_C11_MEM_MODEL', true]
> > > +]
> > > +# arm config (implementer 0x41) is the default config
> > > +pn_config_default
> > What does it mean by 'default' here? I am somewhat confused between
> 'default'
> > and 'generic'. We should look to remove 'default' as much as possible
> > and stick with 'generic'.
> >
> 
> This default means what default means, no special meaning, that is if there isn't
> a more specific configuration, default to this one. It's possible that generic is
> better, but now that I think about it, using something else than default or
> generic might be the best to avoid confusion. Possibly just part_number_arm,
> which will make it in line with the other var names.
Agree, better to call it 'part_number_arm'.

> 
> > > += {
> > > +	'generic': [['-march=armv8-a+crc', '-moutline-atomics']],
> > I like that we have taken out 'native' from this list. Would it be
> > possible to take out 'generic' from this and others below. This is
> > because the binary built with 'generic' build should run on any Arm
> > platform. There is no dependency on any underlying platform.
> >
> 
> This actually means generic part for the implementer, not generic for
> everything. I understand this is here to produce binaries that would run on
> everything from that impelemeter (in line of what you mention below, this
> would be implementer-generic configuration, a fourth category). In my
> patchset, it's also a fallback when building for an unknown part number from
> the implementer.
We do not need implementer-generic binaries/build. However, we will have some parameters that are common across all the part numbers from that implementer (probably we should not call it as 'implementer-generic' to avoid confusion. May be 'implementer-common-flags' or 'implementer-flags-extra'). Those parameters can be used for every part number.

If we know the implementer, we will have the part number as well (this is part of the Arm architecture definition). Unknown part number should result in an error message.

> 
> Since this is not generic for everything, only for the implementer, we're lacking
> the true common default machine args for everything.
> 
> > > +	'0xd03': [['-mcpu=cortex-a53']],
> > > +	'0xd04': [['-mcpu=cortex-a35']],
> > > +	'0xd07': [['-mcpu=cortex-a57']],
> > > +	'0xd08': [['-mcpu=cortex-a72']],
> > > +	'0xd09': [['-mcpu=cortex-a73']],
> > > +	'0xd0a': [['-mcpu=cortex-a75']],
> > > +	'0xd0b': [['-mcpu=cortex-a76']],
> > > +	'0xd0c': [['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'],
> > > +flags_n1sdp_extra]
> > 'flags_n1sdp_extra' does not fit here. For the part number '0xd0c'
> > there could be multiple SoCs (N1SDP is one of them). So, if the SoC is
> > not known, but if we know that the CPU is N1, then we should build a N1-
> Generic build.
> 
> This should be core-generic configuration, I can rename it to
> flags_n1generic_extra.
Agree.

> 
> > So, from my perspective, there are 3 kinds of binaries:
> > 1) generic - best portability -  (possibly) least optimized for a
> > given platform
> > 2) Core-Generic (for ex: N1-generic) - Portable on all N1 based SoCs
> > only - Optimized for N1 cores
> > 3) SoC specific - (possibly) Not portable - Most optimized for the SoC
> >
> 
> The fourth category I mentioned above, implementer-generic, is used in how
We should not have this category.

> the arm configuration is currently processed:
> 1) default configuration
> Added to/overwritten by
> 2) implementer configuration (one of which is the configuration for generic
> build) Added to/overwritten by
This should be just parameters that are common across all the part numbers of that implementer (for ex: RTE_CACHE_LINE_SIZE = 128), not a build.

> 3) part number (or core-generic) configuration Added to/overwritten by
> 4) SoC configuration (what we're adding now)
> 
> It's not organized as cleanly - the machine args contain both 2) and 3).
> Depending on what we want to do with the 'generic' part number I'll adjust the
> config organization.
> 
> >  } pn_config_cavium = {
> > > +	'generic': [['-march=armv8-a+crc+crypto', '-mcpu=thunderx']],
> > 'generic' does not make sense here.
> >
> > > +	'0xa1': [['-mcpu=thunderxt88'], flags_thunderx_extra],
> > > +	'0xa2': [['-mcpu=thunderxt81'], flags_thunderx_extra],
> > > +	'0xa3': [['-mcpu=thunderxt83'], flags_thunderx_extra],
> > > +	'0xaf': [['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'],
> > > flags_thunderx2_extra],
> > > +	'0xb2': [['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'],
> > > +flags_octeontx2_extra], } pn_config_emag = {
> > > +	'generic': [['-march=armv8-a+crc+crypto', '-mtune=emag']] }
> > Same here.
> > I understand that this is coming from the existing code. But, I think
> > we should try to set it right.
> >
> 
> The generic config for these makes sense if a fourth category, implementer-
> generic makes sense.
> For example, for part_number_config_emag this means: for implementer
> _0x50 (Ampere Computing) use the generic machine args if there's nothing
> more specific (which there isn't - no other part number).
There should be a part number for this. Not sure why it is not present here. I will find out the info.

> 
> > >
> > >  ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page
> > > G7-5321)
> > Nit, Would be good to remove the reference to the doc
> >
> 
> Is that because the docs mentioned here may not be the most recent? It was
> useful to me in understanding where the implementer/part number values
> come from.
Yes, mainly because the spec gets updated. Instead you could say "Refer to MIDR in Arm Architecture Reference Manual")

> 
> > > -impl_generic = ['Generic armv8', flags_generic,
> > > machine_args_default]
> > > -impl_0x41 = ['Arm', flags_arm, machine_args_default]
> > > -impl_0x42 = ['Broadcom', flags_generic, machine_args_default]
> > > -impl_0x43 = ['Cavium', flags_cavium, machine_args_cavium]
> > > -impl_0x44 = ['DEC', flags_generic, machine_args_default]
> > > -impl_0x49 = ['Infineon', flags_generic, machine_args_default]
> > > -impl_0x4d = ['Motorola', flags_generic, machine_args_default]
> > > -impl_0x4e = ['NVIDIA', flags_generic, machine_args_default]
> > > -impl_0x50 = ['Ampere Computing', flags_emag, machine_args_emag]
> > > -impl_0x51 = ['Qualcomm', flags_generic, machine_args_default]
> > > -impl_0x53 = ['Samsung', flags_generic, machine_args_default]
> > > -impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_default]
> > > -impl_0x69 = ['Intel', flags_generic, machine_args_default]
> > > -impl_dpaa = ['NXP DPAA', flags_dpaa, machine_args_default]
> > > +impl_generic = ['Generic armv8', flags_generic, pn_config_default]
> > > +impl_0x41 = ['Arm', flags_arm, pn_config_default]
> > > +impl_0x42 = ['Broadcom', flags_generic, pn_config_default]
> > > +impl_0x43 = ['Cavium', flags_cavium, pn_config_cavium]
> > > +impl_0x44 = ['DEC', flags_generic, pn_config_default]
> > > +impl_0x49 = ['Infineon', flags_generic, pn_config_default]
> > > +impl_0x4d = ['Motorola', flags_generic, pn_config_default]
> > > +impl_0x4e = ['NVIDIA', flags_generic, pn_config_default]
> > > +impl_0x50 = ['Ampere Computing', flags_emag, pn_config_emag]
> > > +impl_0x51 = ['Qualcomm', flags_generic, pn_config_default]
> > > +impl_0x53 = ['Samsung', flags_generic, pn_config_default]
> > > +impl_0x56 = ['Marvell ARMADA', flags_armada, pn_config_default]
> > > +impl_0x69 = ['Intel', flags_generic, pn_config_default] impl_dpaa =
> > > +['NXP DPAA', flags_dpaa, pn_config_default]
> > >
> > >  dpdk_conf.set('RTE_ARCH_ARM', 1)
> > >  dpdk_conf.set('RTE_FORCE_INTRINSICS', 1)
> > >
> > >  if dpdk_conf.get('RTE_ARCH_32')
> > > +	# armv7 build
> > >  	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
> > >  	dpdk_conf.set('RTE_ARCH_ARMv7', 1)
> > >  	# the minimum architecture supported, armv7-a, needs the following,
> > >  	# mk/machine/armv7a/rte.vars.mk sets it too
> > >  	machine_args += '-mfpu=neon'
> > >  else
> > > -	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
> > > -	dpdk_conf.set('RTE_ARCH_ARM64', 1)
> > > +	# aarch64 build
> > > +	if not meson.is_cross_build()
> > > +		if machine == 'generic'
> > > +			# default build
> >                                               ^^^^^^^^^^^ Generic build?
> >
> 
> Already addressed in the next version.
> 
> > > +			impl_config = impl_generic
> > > +			part_number = 'generic'
> > > +		else
> > > +			# native build
> > > +			# The script returns ['Implementer', 'Variant',
> > > 'Architecture',
> > > +			# 'Primary Part number', 'Revision']
> > > +			detect_vendor = find_program(join_paths(
> > > +					meson.current_source_dir(),
> > > 'armv8_machine.py'))
> > > +			cmd = run_command(detect_vendor.path())
> > > +			if cmd.returncode() == 0
> > > +				cmd_output =
> > > cmd.stdout().to_lower().strip().split(' ')
> > > +			endif
> > > +			if arm_force_native_march == true
> > > +				part_number = 'native'
> > > +			else
> > > +				part_number = cmd_output[3]
> > > +			endif
> > > +			# Set to generic implementer if implementer is not
> > > found
> > > +			impl_config = get_variable('impl_' + cmd_output[0],
> > > 'impl_generic')
> > > +		endif
> > > +	else
> > > +		# cross build
> > > +		impl_id = meson.get_cross_property('implementer_id',
> > > 'generic')
> > > +		part_number = meson.get_cross_property('part_number',
> > > 'generic')
> > > +		impl_config = get_variable('impl_' + impl_id)
> > > +	endif
> > >
> > > -	machine = []
> > > -	cmd_generic = ['generic', '', '', 'default', '']
> > > -	cmd_output = cmd_generic # Set generic by default
> > > -	machine_args = [] # Clear previous machine args
> > > -	if arm_force_generic_march and not meson.is_cross_build()
> > > -		machine = impl_generic
> > > -		impl_pn = 'default'
> > > +	message('Arm implementer: ' + impl_config[0])
> > > +	message('Arm part number: ' + part_number)
> > > +
> > > +	implementer_flags = impl_config[1]
> > > +	part_number_config = impl_config[2]
> > > +
> > > +	if part_number_config.has_key(part_number)
> > > +		# use the specified part_number machine args if found
> > > +		part_number_config = part_number_config[part_number]
> > > +	elif part_number == 'native'
> > > +		# use native machine args
> > > +		part_number_config = [[native_machine_args]]
> > >  	elif not meson.is_cross_build()
> > > -		# The script returns ['Implementer', 'Variant', 'Architecture',
> > > -		# 'Primary Part number', 'Revision']
> > > -		detect_vendor = find_program(join_paths(
> > > -				meson.current_source_dir(),
> > > 'armv8_machine.py'))
> > > -		cmd = run_command(detect_vendor.path())
> > > -		if cmd.returncode() == 0
> > > -			cmd_output = cmd.stdout().to_lower().strip().split('
> > > ')
> > > -		endif
> > > -		# Set to generic if variable is not found
> > > -		machine = get_variable('impl_' + cmd_output[0], ['generic'])
> > > -		if machine[0] == 'generic'
> > > -			machine = impl_generic
> > > -			cmd_output = cmd_generic
> > > -		endif
> > > -		impl_pn = cmd_output[3]
> > > -		if arm_force_native_march == true
> > > -			impl_pn = 'native'
> > > -		endif
> > > +		# default to generic machine args if part_number is not found
> > > +		# and not forcing native machine args
> > > +		# but don't default in cross-builds; if part_number is specified
> > > +		# incorrectly in a cross-file, it needs to be fixed there
> > > +		part_number_config = part_number_config['generic']
> > >  	else
> > > -		impl_id = meson.get_cross_property('implementor_id',
> > > 'generic')
> > > -		impl_pn = meson.get_cross_property('implementor_pn',
> > > 'default')
> > > -		machine = get_variable('impl_' + impl_id)
> > > +		# cross build and part number is not in part_number_config
> > > +		error('Cross build part number 0@0 not
> > > found.'.format(part_number))
> > >  	endif
> > >
> > > -	# Apply Common Defaults. These settings may be overwritten by
> > > machine
> > > -	# settings later.
> > > -	foreach flag: flags_common_default
> > > -		if flag.length() > 0
> > > -			dpdk_conf.set(flag[0], flag[1])
> > > +	dpdk_flags = flags_common_default + implementer_flags
> > > +
> > > +	if part_number_config.length() > 1
> > > +		dpdk_flags += part_number_config[1]
> > > +	endif
> > > +
> > > +	machine_args = [] # Clear previous machine args
> > > +	foreach flag: part_number_config[0]
> > > +		if cc.has_argument(flag)
> > > +			machine_args += flag
> > >  		endif
> > >  	endforeach
> > >
> > > -	message('Implementer : ' + machine[0])
> > > -	foreach flag: machine[1]
> > > +	foreach flag: dpdk_flags
> > >  		if flag.length() > 0
> > >  			dpdk_conf.set(flag[0], flag[1])
> > >  		endif
> > >  	endforeach
> > > -
> > > -	foreach marg: machine[2]
> > > -		if marg[0] == impl_pn
> > > -			foreach flag: marg[1]
> > > -				if cc.has_argument(flag)
> > > -					machine_args += flag
> > > -				endif
> > > -			endforeach
> > > -			# Apply any extra machine specific flags.
> > > -			foreach flag: marg.get(2, flags_default_extra)
> > > -				if flag.length() > 0
> > > -					dpdk_conf.set(flag[0], flag[1])
> > > -				endif
> > > -			endforeach
> > > -		endif
> > > -	endforeach
> > >  endif
> > > -message(machine_args)
> > > +
> > > +message('Using machine args: @0@'.format(machine_args))
> > >
> > >  if (cc.get_define('__ARM_NEON', args: machine_args) != '' or
> > >      cc.get_define('__aarch64__', args: machine_args) != '')
> > > --
> > > 2.20.1

Re: [dpdk-dev] [PATCH v4 2/6] build: refactor Arm build

[Juraj Linkeš]

> -----Original Message-----
> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Sent: Thursday, October 29, 2020 9:54 PM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>; bruce.richardson@intel.com;
> Ruifeng Wang <Ruifeng.Wang@arm.com>; Phil Yang <Phil.Yang@arm.com>;
> vcchunga@amazon.com; Dharmik Thakkar <Dharmik.Thakkar@arm.com>;
> jerinjacobk@gmail.com; hemant.agrawal@nxp.com
> Cc: dev@dpdk.org; nd <nd@arm.com>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>
> Subject: RE: [PATCH v4 2/6] build: refactor Arm build
> 
> > > <snip>
> > >
> > > > diff --git a/config/arm/meson.build b/config/arm/meson.build index
> > > > 491842cad..6c31ab167 100644
> > > > --- a/config/arm/meson.build
> > > > +++ b/config/arm/meson.build
> > > > @@ -3,12 +3,12 @@
> > > >  # Copyright(c) 2017 Cavium, Inc
> > > >  # Copyright(c) 2020 PANTHEON.tech s.r.o.
> > > >
> > > > -# for checking defines we need to use the correct compiler flags
> > > > -march_opt = '-march=@0@'.format(machine)
> > > > -
> > > > +# set arm_force_native_march if you want to use machine args
> > > > +below # instead of discovered values; only works when doing an
> > > > +actual native build
> > > >  arm_force_native_march = false
> > > > -arm_force_generic_march = (machine == 'generic')
> > > > +native_machine_args = ['-march=native', '-mtune=native']
> > > >
> > >
> > > [...]
> > >
> > > > -
> > > > -machine_args_default = [
> > > > -	['default', ['-march=armv8-a+crc', '-moutline-atomics']],
> > > > -	['native', ['-march=native']],
> > > > -	['0xd03', ['-mcpu=cortex-a53']],
> > > > -	['0xd04', ['-mcpu=cortex-a35']],
> > > > -	['0xd07', ['-mcpu=cortex-a57']],
> > > > -	['0xd08', ['-mcpu=cortex-a72']],
> > > > -	['0xd09', ['-mcpu=cortex-a73']],
> > > > -	['0xd0a', ['-mcpu=cortex-a75']],
> > > > -	['0xd0b', ['-mcpu=cortex-a76']],
> > > > -	['0xd0c', ['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'],
> > > > flags_n1sdp_extra]]
> > > > -
> > > > -machine_args_cavium = [
> > > > -	['default', ['-march=armv8-a+crc+crypto','-mcpu=thunderx']],
> > > > -	['native', ['-march=native']],
> > > > -	['0xa1', ['-mcpu=thunderxt88'], flags_thunderx_extra],
> > > > -	['0xa2', ['-mcpu=thunderxt81'], flags_thunderx_extra],
> > > > -	['0xa3', ['-mcpu=thunderxt83'], flags_thunderx_extra],
> > > > -	['0xaf', ['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'],
> > > > flags_thunderx2_extra],
> > > > -	['0xb2', ['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'],
> > > > flags_octeontx2_extra]]
> > > > -
> > > > -machine_args_emag = [
> > > > -	['default', ['-march=armv8-a+crc+crypto', '-mtune=emag']],
> > > > -	['native', ['-march=native']]]
> > > > +	['RTE_USE_C11_MEM_MODEL', true]
> > > > +]
> > > > +# arm config (implementer 0x41) is the default config
> > > > +pn_config_default
> > > What does it mean by 'default' here? I am somewhat confused between
> > 'default'
> > > and 'generic'. We should look to remove 'default' as much as
> > > possible and stick with 'generic'.
> > >
> >
> > This default means what default means, no special meaning, that is if
> > there isn't a more specific configuration, default to this one. It's
> > possible that generic is better, but now that I think about it, using
> > something else than default or generic might be the best to avoid
> > confusion. Possibly just part_number_arm, which will make it in line with the
> other var names.
> Agree, better to call it 'part_number_arm'.
> 
> >
> > > > += {
> > > > +	'generic': [['-march=armv8-a+crc', '-moutline-atomics']],
> > > I like that we have taken out 'native' from this list. Would it be
> > > possible to take out 'generic' from this and others below. This is
> > > because the binary built with 'generic' build should run on any Arm
> > > platform. There is no dependency on any underlying platform.
> > >
> >
> > This actually means generic part for the implementer, not generic for
> > everything. I understand this is here to produce binaries that would
> > run on everything from that impelemeter (in line of what you mention
> > below, this would be implementer-generic configuration, a fourth
> > category). In my patchset, it's also a fallback when building for an
> > unknown part number from the implementer.
> We do not need implementer-generic binaries/build. However, we will have
> some parameters that are common across all the part numbers from that
> implementer (probably we should not call it as 'implementer-generic' to avoid
> confusion. May be 'implementer-common-flags' or 'implementer-flags-extra').
> Those parameters can be used for every part number.

These are currently named flags_<implementer> such as flags_arm and flags_cavium. We could rename them to implementer_flags_<implementer>.

> 
> If we know the implementer, we will have the part number as well (this is part of
> the Arm architecture definition). Unknown part number should result in an error
> message.
> 

Yes, we'll always have both. But there are still a couple of corner cases with native builds, which should always work:
1. We don't support the implementer (i.e. there's no variable with name implementer_<implementer_id>). In this case, what do default to? The generic build, which is the current behavior?
2. We support the implementer, but we don't support the part number. In this case, we can just use native machine args (-march=native) and use only implementer specific flags.

For generic, soc-specifc and cross builds it makes sense to error with uknown implementer/part number, since we wouldn't know what to build otherwise.

> >
> > Since this is not generic for everything, only for the implementer,
> > we're lacking the true common default machine args for everything.
> >
> > > > +	'0xd03': [['-mcpu=cortex-a53']],
> > > > +	'0xd04': [['-mcpu=cortex-a35']],
> > > > +	'0xd07': [['-mcpu=cortex-a57']],
> > > > +	'0xd08': [['-mcpu=cortex-a72']],
> > > > +	'0xd09': [['-mcpu=cortex-a73']],
> > > > +	'0xd0a': [['-mcpu=cortex-a75']],
> > > > +	'0xd0b': [['-mcpu=cortex-a76']],
> > > > +	'0xd0c': [['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'],
> > > > +flags_n1sdp_extra]
> > > 'flags_n1sdp_extra' does not fit here. For the part number '0xd0c'
> > > there could be multiple SoCs (N1SDP is one of them). So, if the SoC
> > > is not known, but if we know that the CPU is N1, then we should
> > > build a N1-
> > Generic build.
> >
> > This should be core-generic configuration, I can rename it to
> > flags_n1generic_extra.
> Agree.
> 
> >
> > > So, from my perspective, there are 3 kinds of binaries:
> > > 1) generic - best portability -  (possibly) least optimized for a
> > > given platform
> > > 2) Core-Generic (for ex: N1-generic) - Portable on all N1 based SoCs
> > > only - Optimized for N1 cores
> > > 3) SoC specific - (possibly) Not portable - Most optimized for the
> > > SoC
> > >
> >
> > The fourth category I mentioned above, implementer-generic, is used in
> > how
> We should not have this category.
> 
> > the arm configuration is currently processed:
> > 1) default configuration
> > Added to/overwritten by
> > 2) implementer configuration (one of which is the configuration for
> > generic
> > build) Added to/overwritten by
> This should be just parameters that are common across all the part numbers of
> that implementer (for ex: RTE_CACHE_LINE_SIZE = 128), not a build.
> 

It's not a build, just how the configation is organized and processed, as in we apply the common default flags, then implementer flags, then part number flags and then soc flags, overwriting flags in all of the steps.

> > 3) part number (or core-generic) configuration Added to/overwritten by
> > 4) SoC configuration (what we're adding now)
> >
> > It's not organized as cleanly - the machine args contain both 2) and 3).
> > Depending on what we want to do with the 'generic' part number I'll
> > adjust the config organization.
> >
> > >  } pn_config_cavium = {
> > > > +	'generic': [['-march=armv8-a+crc+crypto', '-mcpu=thunderx']],
> > > 'generic' does not make sense here.
> > >
> > > > +	'0xa1': [['-mcpu=thunderxt88'], flags_thunderx_extra],
> > > > +	'0xa2': [['-mcpu=thunderxt81'], flags_thunderx_extra],
> > > > +	'0xa3': [['-mcpu=thunderxt83'], flags_thunderx_extra],
> > > > +	'0xaf': [['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'],
> > > > flags_thunderx2_extra],
> > > > +	'0xb2': [['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'],
> > > > +flags_octeontx2_extra], } pn_config_emag = {
> > > > +	'generic': [['-march=armv8-a+crc+crypto', '-mtune=emag']] }
> > > Same here.
> > > I understand that this is coming from the existing code. But, I
> > > think we should try to set it right.
> > >
> >
> > The generic config for these makes sense if a fourth category,
> > implementer- generic makes sense.
> > For example, for part_number_config_emag this means: for implementer
> > _0x50 (Ampere Computing) use the generic machine args if there's
> > nothing more specific (which there isn't - no other part number).
> There should be a part number for this. Not sure why it is not present here. I will
> find out the info.
> 
> >
> > > >
> > > >  ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page
> > > > G7-5321)
> > > Nit, Would be good to remove the reference to the doc
> > >
> >
> > Is that because the docs mentioned here may not be the most recent? It
> > was useful to me in understanding where the implementer/part number
> > values come from.
> Yes, mainly because the spec gets updated. Instead you could say "Refer to
> MIDR in Arm Architecture Reference Manual")
> 

Ok, I'll make this change.

> >
> > > > -impl_generic = ['Generic armv8', flags_generic,
> > > > machine_args_default]
> > > > -impl_0x41 = ['Arm', flags_arm, machine_args_default]
> > > > -impl_0x42 = ['Broadcom', flags_generic, machine_args_default]
> > > > -impl_0x43 = ['Cavium', flags_cavium, machine_args_cavium]
> > > > -impl_0x44 = ['DEC', flags_generic, machine_args_default]
> > > > -impl_0x49 = ['Infineon', flags_generic, machine_args_default]
> > > > -impl_0x4d = ['Motorola', flags_generic, machine_args_default]
> > > > -impl_0x4e = ['NVIDIA', flags_generic, machine_args_default]
> > > > -impl_0x50 = ['Ampere Computing', flags_emag, machine_args_emag]
> > > > -impl_0x51 = ['Qualcomm', flags_generic, machine_args_default]
> > > > -impl_0x53 = ['Samsung', flags_generic, machine_args_default]
> > > > -impl_0x56 = ['Marvell ARMADA', flags_armada,
> > > > machine_args_default]
> > > > -impl_0x69 = ['Intel', flags_generic, machine_args_default]
> > > > -impl_dpaa = ['NXP DPAA', flags_dpaa, machine_args_default]
> > > > +impl_generic = ['Generic armv8', flags_generic,
> > > > +pn_config_default]
> > > > +impl_0x41 = ['Arm', flags_arm, pn_config_default]
> > > > +impl_0x42 = ['Broadcom', flags_generic, pn_config_default]
> > > > +impl_0x43 = ['Cavium', flags_cavium, pn_config_cavium]
> > > > +impl_0x44 = ['DEC', flags_generic, pn_config_default]
> > > > +impl_0x49 = ['Infineon', flags_generic, pn_config_default]
> > > > +impl_0x4d = ['Motorola', flags_generic, pn_config_default]
> > > > +impl_0x4e = ['NVIDIA', flags_generic, pn_config_default]
> > > > +impl_0x50 = ['Ampere Computing', flags_emag, pn_config_emag]
> > > > +impl_0x51 = ['Qualcomm', flags_generic, pn_config_default]
> > > > +impl_0x53 = ['Samsung', flags_generic, pn_config_default]
> > > > +impl_0x56 = ['Marvell ARMADA', flags_armada, pn_config_default]
> > > > +impl_0x69 = ['Intel', flags_generic, pn_config_default] impl_dpaa
> > > > += ['NXP DPAA', flags_dpaa, pn_config_default]
> > > >
> > > >  dpdk_conf.set('RTE_ARCH_ARM', 1)
> > > >  dpdk_conf.set('RTE_FORCE_INTRINSICS', 1)
> > > >
> > > >  if dpdk_conf.get('RTE_ARCH_32')
> > > > +	# armv7 build
> > > >  	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
> > > >  	dpdk_conf.set('RTE_ARCH_ARMv7', 1)
> > > >  	# the minimum architecture supported, armv7-a, needs the following,
> > > >  	# mk/machine/armv7a/rte.vars.mk sets it too
> > > >  	machine_args += '-mfpu=neon'
> > > >  else
> > > > -	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
> > > > -	dpdk_conf.set('RTE_ARCH_ARM64', 1)
> > > > +	# aarch64 build
> > > > +	if not meson.is_cross_build()
> > > > +		if machine == 'generic'
> > > > +			# default build
> > >                                               ^^^^^^^^^^^ Generic build?
> > >
> >
> > Already addressed in the next version.
> >
> > > > +			impl_config = impl_generic
> > > > +			part_number = 'generic'
> > > > +		else
> > > > +			# native build
> > > > +			# The script returns ['Implementer', 'Variant',
> > > > 'Architecture',
> > > > +			# 'Primary Part number', 'Revision']
> > > > +			detect_vendor = find_program(join_paths(
> > > > +					meson.current_source_dir(),
> > > > 'armv8_machine.py'))
> > > > +			cmd = run_command(detect_vendor.path())
> > > > +			if cmd.returncode() == 0
> > > > +				cmd_output =
> > > > cmd.stdout().to_lower().strip().split(' ')
> > > > +			endif
> > > > +			if arm_force_native_march == true
> > > > +				part_number = 'native'
> > > > +			else
> > > > +				part_number = cmd_output[3]
> > > > +			endif
> > > > +			# Set to generic implementer if implementer is not
> > > > found
> > > > +			impl_config = get_variable('impl_' + cmd_output[0],
> > > > 'impl_generic')
> > > > +		endif
> > > > +	else
> > > > +		# cross build
> > > > +		impl_id = meson.get_cross_property('implementer_id',
> > > > 'generic')
> > > > +		part_number = meson.get_cross_property('part_number',
> > > > 'generic')
> > > > +		impl_config = get_variable('impl_' + impl_id)
> > > > +	endif
> > > >
> > > > -	machine = []
> > > > -	cmd_generic = ['generic', '', '', 'default', '']
> > > > -	cmd_output = cmd_generic # Set generic by default
> > > > -	machine_args = [] # Clear previous machine args
> > > > -	if arm_force_generic_march and not meson.is_cross_build()
> > > > -		machine = impl_generic
> > > > -		impl_pn = 'default'
> > > > +	message('Arm implementer: ' + impl_config[0])
> > > > +	message('Arm part number: ' + part_number)
> > > > +
> > > > +	implementer_flags = impl_config[1]
> > > > +	part_number_config = impl_config[2]
> > > > +
> > > > +	if part_number_config.has_key(part_number)
> > > > +		# use the specified part_number machine args if found
> > > > +		part_number_config = part_number_config[part_number]
> > > > +	elif part_number == 'native'
> > > > +		# use native machine args
> > > > +		part_number_config = [[native_machine_args]]
> > > >  	elif not meson.is_cross_build()
> > > > -		# The script returns ['Implementer', 'Variant', 'Architecture',
> > > > -		# 'Primary Part number', 'Revision']
> > > > -		detect_vendor = find_program(join_paths(
> > > > -				meson.current_source_dir(),
> > > > 'armv8_machine.py'))
> > > > -		cmd = run_command(detect_vendor.path())
> > > > -		if cmd.returncode() == 0
> > > > -			cmd_output = cmd.stdout().to_lower().strip().split('
> > > > ')
> > > > -		endif
> > > > -		# Set to generic if variable is not found
> > > > -		machine = get_variable('impl_' + cmd_output[0], ['generic'])
> > > > -		if machine[0] == 'generic'
> > > > -			machine = impl_generic
> > > > -			cmd_output = cmd_generic
> > > > -		endif
> > > > -		impl_pn = cmd_output[3]
> > > > -		if arm_force_native_march == true
> > > > -			impl_pn = 'native'
> > > > -		endif
> > > > +		# default to generic machine args if part_number is not found
> > > > +		# and not forcing native machine args
> > > > +		# but don't default in cross-builds; if part_number is specified
> > > > +		# incorrectly in a cross-file, it needs to be fixed there
> > > > +		part_number_config = part_number_config['generic']
> > > >  	else
> > > > -		impl_id = meson.get_cross_property('implementor_id',
> > > > 'generic')
> > > > -		impl_pn = meson.get_cross_property('implementor_pn',
> > > > 'default')
> > > > -		machine = get_variable('impl_' + impl_id)
> > > > +		# cross build and part number is not in part_number_config
> > > > +		error('Cross build part number 0@0 not
> > > > found.'.format(part_number))
> > > >  	endif
> > > >
> > > > -	# Apply Common Defaults. These settings may be overwritten by
> > > > machine
> > > > -	# settings later.
> > > > -	foreach flag: flags_common_default
> > > > -		if flag.length() > 0
> > > > -			dpdk_conf.set(flag[0], flag[1])
> > > > +	dpdk_flags = flags_common_default + implementer_flags
> > > > +
> > > > +	if part_number_config.length() > 1
> > > > +		dpdk_flags += part_number_config[1]
> > > > +	endif
> > > > +
> > > > +	machine_args = [] # Clear previous machine args
> > > > +	foreach flag: part_number_config[0]
> > > > +		if cc.has_argument(flag)
> > > > +			machine_args += flag
> > > >  		endif
> > > >  	endforeach
> > > >
> > > > -	message('Implementer : ' + machine[0])
> > > > -	foreach flag: machine[1]
> > > > +	foreach flag: dpdk_flags
> > > >  		if flag.length() > 0
> > > >  			dpdk_conf.set(flag[0], flag[1])
> > > >  		endif
> > > >  	endforeach
> > > > -
> > > > -	foreach marg: machine[2]
> > > > -		if marg[0] == impl_pn
> > > > -			foreach flag: marg[1]
> > > > -				if cc.has_argument(flag)
> > > > -					machine_args += flag
> > > > -				endif
> > > > -			endforeach
> > > > -			# Apply any extra machine specific flags.
> > > > -			foreach flag: marg.get(2, flags_default_extra)
> > > > -				if flag.length() > 0
> > > > -					dpdk_conf.set(flag[0], flag[1])
> > > > -				endif
> > > > -			endforeach
> > > > -		endif
> > > > -	endforeach
> > > >  endif
> > > > -message(machine_args)
> > > > +
> > > > +message('Using machine args: @0@'.format(machine_args))
> > > >
> > > >  if (cc.get_define('__ARM_NEON', args: machine_args) != '' or
> > > >      cc.get_define('__aarch64__', args: machine_args) != '')
> > > > --
> > > > 2.20.1

Re: [dpdk-dev] [PATCH v4 2/6] build: refactor Arm build

[Honnappa Nagarahalli]

<snip>

> > > >
> > > > > diff --git a/config/arm/meson.build b/config/arm/meson.build
> > > > > index
> > > > > 491842cad..6c31ab167 100644
> > > > > --- a/config/arm/meson.build
> > > > > +++ b/config/arm/meson.build
> > > > > @@ -3,12 +3,12 @@
> > > > >  # Copyright(c) 2017 Cavium, Inc  # Copyright(c) 2020
> > > > > PANTHEON.tech s.r.o.
> > > > >
> > > > > -# for checking defines we need to use the correct compiler
> > > > > flags -march_opt = '-march=@0@'.format(machine)
> > > > > -
> > > > > +# set arm_force_native_march if you want to use machine args
> > > > > +below # instead of discovered values; only works when doing an
> > > > > +actual native build
> > > > >  arm_force_native_march = false
> > > > > -arm_force_generic_march = (machine == 'generic')
> > > > > +native_machine_args = ['-march=native', '-mtune=native']
> > > > >
> > > >
> > > > [...]
> > > >
> > > > > -
> > > > > -machine_args_default = [
> > > > > -	['default', ['-march=armv8-a+crc', '-moutline-atomics']],
> > > > > -	['native', ['-march=native']],
> > > > > -	['0xd03', ['-mcpu=cortex-a53']],
> > > > > -	['0xd04', ['-mcpu=cortex-a35']],
> > > > > -	['0xd07', ['-mcpu=cortex-a57']],
> > > > > -	['0xd08', ['-mcpu=cortex-a72']],
> > > > > -	['0xd09', ['-mcpu=cortex-a73']],
> > > > > -	['0xd0a', ['-mcpu=cortex-a75']],
> > > > > -	['0xd0b', ['-mcpu=cortex-a76']],
> > > > > -	['0xd0c', ['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-
> n1'],
> > > > > flags_n1sdp_extra]]
> > > > > -
> > > > > -machine_args_cavium = [
> > > > > -	['default', ['-march=armv8-a+crc+crypto','-mcpu=thunderx']],
> > > > > -	['native', ['-march=native']],
> > > > > -	['0xa1', ['-mcpu=thunderxt88'], flags_thunderx_extra],
> > > > > -	['0xa2', ['-mcpu=thunderxt81'], flags_thunderx_extra],
> > > > > -	['0xa3', ['-mcpu=thunderxt83'], flags_thunderx_extra],
> > > > > -	['0xaf', ['-march=armv8.1-a+crc+crypto','-
> mcpu=thunderx2t99'],
> > > > > flags_thunderx2_extra],
> > > > > -	['0xb2', ['-march=armv8.2-a+crc+crypto+lse','-
> mcpu=octeontx2'],
> > > > > flags_octeontx2_extra]]
> > > > > -
> > > > > -machine_args_emag = [
> > > > > -	['default', ['-march=armv8-a+crc+crypto', '-mtune=emag']],
> > > > > -	['native', ['-march=native']]]
> > > > > +	['RTE_USE_C11_MEM_MODEL', true] ] # arm config
> (implementer
> > > > > +0x41) is the default config pn_config_default
> > > > What does it mean by 'default' here? I am somewhat confused
> > > > between
> > > 'default'
> > > > and 'generic'. We should look to remove 'default' as much as
> > > > possible and stick with 'generic'.
> > > >
> > >
> > > This default means what default means, no special meaning, that is
> > > if there isn't a more specific configuration, default to this one.
> > > It's possible that generic is better, but now that I think about it,
> > > using something else than default or generic might be the best to
> > > avoid confusion. Possibly just part_number_arm, which will make it
> > > in line with the
> > other var names.
> > Agree, better to call it 'part_number_arm'.
> >
> > >
> > > > > += {
> > > > > +	'generic': [['-march=armv8-a+crc', '-moutline-atomics']],
> > > > I like that we have taken out 'native' from this list. Would it be
> > > > possible to take out 'generic' from this and others below. This is
> > > > because the binary built with 'generic' build should run on any
> > > > Arm platform. There is no dependency on any underlying platform.
> > > >
> > >
> > > This actually means generic part for the implementer, not generic
> > > for everything. I understand this is here to produce binaries that
> > > would run on everything from that impelemeter (in line of what you
> > > mention below, this would be implementer-generic configuration, a
> > > fourth category). In my patchset, it's also a fallback when building
> > > for an unknown part number from the implementer.
> > We do not need implementer-generic binaries/build. However, we will
> > have some parameters that are common across all the part numbers from
> > that implementer (probably we should not call it as
> > 'implementer-generic' to avoid confusion. May be 'implementer-common-
> flags' or 'implementer-flags-extra').
> > Those parameters can be used for every part number.
> 
> These are currently named flags_<implementer> such as flags_arm and
> flags_cavium. We could rename them to
> implementer_flags_<implementer>.
Ok

> 
> >
> > If we know the implementer, we will have the part number as well (this
> > is part of the Arm architecture definition). Unknown part number
> > should result in an error message.
> >
> 
> Yes, we'll always have both. But there are still a couple of corner cases with
> native builds, which should always work:
I do not think these are corner cases. These are basically cases where the support is not added yet in the build system. We do not need to handle these specifically.
I also do not think the notion that the 'native' build should always work is correct (at least on Arm platforms). The reason I say that is because 'native', IMO, not only defines the ISA, but other parameters as well in DPDK.

> 1. We don't support the implementer (i.e. there's no variable with name
> implementer_<implementer_id>). In this case, what do default to? The
> generic build, which is the current behavior?
This should result in an error. The implementer should add support in meson.build

> 2. We support the implementer, but we don't support the part number. In
> this case, we can just use native machine args (-march=native) and use only
> implementer specific flags.
Same thing, should result in error. The implementer should add support in meson.build.

> 
> For generic, soc-specifc and cross builds it makes sense to error with uknown
> implementer/part number, since we wouldn't know what to build otherwise.
> 
> > >
> > > Since this is not generic for everything, only for the implementer,
> > > we're lacking the true common default machine args for everything.
> > >
> > > > > +	'0xd03': [['-mcpu=cortex-a53']],
> > > > > +	'0xd04': [['-mcpu=cortex-a35']],
> > > > > +	'0xd07': [['-mcpu=cortex-a57']],
> > > > > +	'0xd08': [['-mcpu=cortex-a72']],
> > > > > +	'0xd09': [['-mcpu=cortex-a73']],
> > > > > +	'0xd0a': [['-mcpu=cortex-a75']],
> > > > > +	'0xd0b': [['-mcpu=cortex-a76']],
> > > > > +	'0xd0c': [['-march=armv8.2-a+crc+crypto',
> > > > > +'-mcpu=neoverse-n1'], flags_n1sdp_extra]
> > > > 'flags_n1sdp_extra' does not fit here. For the part number '0xd0c'
> > > > there could be multiple SoCs (N1SDP is one of them). So, if the
> > > > SoC is not known, but if we know that the CPU is N1, then we
> > > > should build a N1-
> > > Generic build.
> > >
> > > This should be core-generic configuration, I can rename it to
> > > flags_n1generic_extra.
> > Agree.
> >
> > >
> > > > So, from my perspective, there are 3 kinds of binaries:
> > > > 1) generic - best portability -  (possibly) least optimized for a
> > > > given platform
> > > > 2) Core-Generic (for ex: N1-generic) - Portable on all N1 based
> > > > SoCs only - Optimized for N1 cores
> > > > 3) SoC specific - (possibly) Not portable - Most optimized for the
> > > > SoC
> > > >
> > >
> > > The fourth category I mentioned above, implementer-generic, is used
> > > in how
> > We should not have this category.
> >
> > > the arm configuration is currently processed:
> > > 1) default configuration
> > > Added to/overwritten by
> > > 2) implementer configuration (one of which is the configuration for
> > > generic
> > > build) Added to/overwritten by
> > This should be just parameters that are common across all the part
> > numbers of that implementer (for ex: RTE_CACHE_LINE_SIZE = 128), not a
> build.
> >
> 
> It's not a build, just how the configation is organized and processed, as in we
> apply the common default flags, then implementer flags, then part number
> flags and then soc flags, overwriting flags in all of the steps.
Ok

> 
> > > 3) part number (or core-generic) configuration Added to/overwritten
> > > by
> > > 4) SoC configuration (what we're adding now)
> > >
> > > It's not organized as cleanly - the machine args contain both 2) and 3).
> > > Depending on what we want to do with the 'generic' part number I'll
> > > adjust the config organization.
> > >
> > > >  } pn_config_cavium = {
> > > > > +	'generic': [['-march=armv8-a+crc+crypto', '-
> mcpu=thunderx']],
> > > > 'generic' does not make sense here.
> > > >
> > > > > +	'0xa1': [['-mcpu=thunderxt88'], flags_thunderx_extra],
> > > > > +	'0xa2': [['-mcpu=thunderxt81'], flags_thunderx_extra],
> > > > > +	'0xa3': [['-mcpu=thunderxt83'], flags_thunderx_extra],
> > > > > +	'0xaf': [['-march=armv8.1-a+crc+crypto','-
> mcpu=thunderx2t99'],
> > > > > flags_thunderx2_extra],
> > > > > +	'0xb2':
> > > > > +[['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'],
> > > > > +flags_octeontx2_extra], } pn_config_emag = {
> > > > > +	'generic': [['-march=armv8-a+crc+crypto', '-mtune=emag']] }
> > > > Same here.
> > > > I understand that this is coming from the existing code. But, I
> > > > think we should try to set it right.
> > > >
> > >
> > > The generic config for these makes sense if a fourth category,
> > > implementer- generic makes sense.
> > > For example, for part_number_config_emag this means: for
> implementer
> > > _0x50 (Ampere Computing) use the generic machine args if there's
> > > nothing more specific (which there isn't - no other part number).
> > There should be a part number for this. Not sure why it is not present
> > here. I will find out the info.
> >
> > >
> > > > >
> > > > >  ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page
> > > > > G7-5321)
> > > > Nit, Would be good to remove the reference to the doc
> > > >
> > >
> > > Is that because the docs mentioned here may not be the most recent?
> > > It was useful to me in understanding where the implementer/part
> > > number values come from.
> > Yes, mainly because the spec gets updated. Instead you could say
> > "Refer to MIDR in Arm Architecture Reference Manual")
> >
> 
> Ok, I'll make this change.
> 
> > >
> > > > > -impl_generic = ['Generic armv8', flags_generic,
> > > > > machine_args_default]
> > > > > -impl_0x41 = ['Arm', flags_arm, machine_args_default]
> > > > > -impl_0x42 = ['Broadcom', flags_generic, machine_args_default]
> > > > > -impl_0x43 = ['Cavium', flags_cavium, machine_args_cavium]
> > > > > -impl_0x44 = ['DEC', flags_generic, machine_args_default]
> > > > > -impl_0x49 = ['Infineon', flags_generic, machine_args_default]
> > > > > -impl_0x4d = ['Motorola', flags_generic, machine_args_default]
> > > > > -impl_0x4e = ['NVIDIA', flags_generic, machine_args_default]
> > > > > -impl_0x50 = ['Ampere Computing', flags_emag,
> machine_args_emag]
> > > > > -impl_0x51 = ['Qualcomm', flags_generic, machine_args_default]
> > > > > -impl_0x53 = ['Samsung', flags_generic, machine_args_default]
> > > > > -impl_0x56 = ['Marvell ARMADA', flags_armada,
> > > > > machine_args_default]
> > > > > -impl_0x69 = ['Intel', flags_generic, machine_args_default]
> > > > > -impl_dpaa = ['NXP DPAA', flags_dpaa, machine_args_default]
> > > > > +impl_generic = ['Generic armv8', flags_generic,
> > > > > +pn_config_default]
> > > > > +impl_0x41 = ['Arm', flags_arm, pn_config_default]
> > > > > +impl_0x42 = ['Broadcom', flags_generic, pn_config_default]
> > > > > +impl_0x43 = ['Cavium', flags_cavium, pn_config_cavium]
> > > > > +impl_0x44 = ['DEC', flags_generic, pn_config_default]
> > > > > +impl_0x49 = ['Infineon', flags_generic, pn_config_default]
> > > > > +impl_0x4d = ['Motorola', flags_generic, pn_config_default]
> > > > > +impl_0x4e = ['NVIDIA', flags_generic, pn_config_default]
> > > > > +impl_0x50 = ['Ampere Computing', flags_emag, pn_config_emag]
> > > > > +impl_0x51 = ['Qualcomm', flags_generic, pn_config_default]
> > > > > +impl_0x53 = ['Samsung', flags_generic, pn_config_default]
> > > > > +impl_0x56 = ['Marvell ARMADA', flags_armada, pn_config_default]
> > > > > +impl_0x69 = ['Intel', flags_generic, pn_config_default]
> > > > > +impl_dpaa = ['NXP DPAA', flags_dpaa, pn_config_default]
> > > > >
> > > > >  dpdk_conf.set('RTE_ARCH_ARM', 1)
> > > > > dpdk_conf.set('RTE_FORCE_INTRINSICS', 1)
> > > > >
> > > > >  if dpdk_conf.get('RTE_ARCH_32')
> > > > > +	# armv7 build
> > > > >  	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
> > > > >  	dpdk_conf.set('RTE_ARCH_ARMv7', 1)
> > > > >  	# the minimum architecture supported, armv7-a, needs the
> following,
> > > > >  	# mk/machine/armv7a/rte.vars.mk sets it too
> > > > >  	machine_args += '-mfpu=neon'
> > > > >  else
> > > > > -	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
> > > > > -	dpdk_conf.set('RTE_ARCH_ARM64', 1)
> > > > > +	# aarch64 build
> > > > > +	if not meson.is_cross_build()
> > > > > +		if machine == 'generic'
> > > > > +			# default build
> > > >                                               ^^^^^^^^^^^ Generic build?
> > > >
> > >
> > > Already addressed in the next version.
> > >
> > > > > +			impl_config = impl_generic
> > > > > +			part_number = 'generic'
> > > > > +		else
> > > > > +			# native build
> > > > > +			# The script returns ['Implementer', 'Variant',
> > > > > 'Architecture',
> > > > > +			# 'Primary Part number', 'Revision']
> > > > > +			detect_vendor = find_program(join_paths(
> > > > > +					meson.current_source_dir(),
> > > > > 'armv8_machine.py'))
> > > > > +			cmd = run_command(detect_vendor.path())
> > > > > +			if cmd.returncode() == 0
> > > > > +				cmd_output =
> > > > > cmd.stdout().to_lower().strip().split(' ')
> > > > > +			endif
> > > > > +			if arm_force_native_march == true
> > > > > +				part_number = 'native'
> > > > > +			else
> > > > > +				part_number = cmd_output[3]
> > > > > +			endif
> > > > > +			# Set to generic implementer if implementer
> is not
> > > > > found
> > > > > +			impl_config = get_variable('impl_' +
> cmd_output[0],
> > > > > 'impl_generic')
> > > > > +		endif
> > > > > +	else
> > > > > +		# cross build
> > > > > +		impl_id =
> meson.get_cross_property('implementer_id',
> > > > > 'generic')
> > > > > +		part_number =
> meson.get_cross_property('part_number',
> > > > > 'generic')
> > > > > +		impl_config = get_variable('impl_' + impl_id)
> > > > > +	endif
> > > > >
> > > > > -	machine = []
> > > > > -	cmd_generic = ['generic', '', '', 'default', '']
> > > > > -	cmd_output = cmd_generic # Set generic by default
> > > > > -	machine_args = [] # Clear previous machine args
> > > > > -	if arm_force_generic_march and not meson.is_cross_build()
> > > > > -		machine = impl_generic
> > > > > -		impl_pn = 'default'
> > > > > +	message('Arm implementer: ' + impl_config[0])
> > > > > +	message('Arm part number: ' + part_number)
> > > > > +
> > > > > +	implementer_flags = impl_config[1]
> > > > > +	part_number_config = impl_config[2]
> > > > > +
> > > > > +	if part_number_config.has_key(part_number)
> > > > > +		# use the specified part_number machine args if
> found
> > > > > +		part_number_config =
> part_number_config[part_number]
> > > > > +	elif part_number == 'native'
> > > > > +		# use native machine args
> > > > > +		part_number_config = [[native_machine_args]]
> > > > >  	elif not meson.is_cross_build()
> > > > > -		# The script returns ['Implementer', 'Variant',
> 'Architecture',
> > > > > -		# 'Primary Part number', 'Revision']
> > > > > -		detect_vendor = find_program(join_paths(
> > > > > -				meson.current_source_dir(),
> > > > > 'armv8_machine.py'))
> > > > > -		cmd = run_command(detect_vendor.path())
> > > > > -		if cmd.returncode() == 0
> > > > > -			cmd_output =
> cmd.stdout().to_lower().strip().split('
> > > > > ')
> > > > > -		endif
> > > > > -		# Set to generic if variable is not found
> > > > > -		machine = get_variable('impl_' + cmd_output[0],
> ['generic'])
> > > > > -		if machine[0] == 'generic'
> > > > > -			machine = impl_generic
> > > > > -			cmd_output = cmd_generic
> > > > > -		endif
> > > > > -		impl_pn = cmd_output[3]
> > > > > -		if arm_force_native_march == true
> > > > > -			impl_pn = 'native'
> > > > > -		endif
> > > > > +		# default to generic machine args if part_number is
> not found
> > > > > +		# and not forcing native machine args
> > > > > +		# but don't default in cross-builds; if part_number is
> specified
> > > > > +		# incorrectly in a cross-file, it needs to be fixed there
> > > > > +		part_number_config =
> part_number_config['generic']
> > > > >  	else
> > > > > -		impl_id =
> meson.get_cross_property('implementor_id',
> > > > > 'generic')
> > > > > -		impl_pn =
> meson.get_cross_property('implementor_pn',
> > > > > 'default')
> > > > > -		machine = get_variable('impl_' + impl_id)
> > > > > +		# cross build and part number is not in
> part_number_config
> > > > > +		error('Cross build part number 0@0 not
> > > > > found.'.format(part_number))
> > > > >  	endif
> > > > >
> > > > > -	# Apply Common Defaults. These settings may be
> overwritten by
> > > > > machine
> > > > > -	# settings later.
> > > > > -	foreach flag: flags_common_default
> > > > > -		if flag.length() > 0
> > > > > -			dpdk_conf.set(flag[0], flag[1])
> > > > > +	dpdk_flags = flags_common_default + implementer_flags
> > > > > +
> > > > > +	if part_number_config.length() > 1
> > > > > +		dpdk_flags += part_number_config[1]
> > > > > +	endif
> > > > > +
> > > > > +	machine_args = [] # Clear previous machine args
> > > > > +	foreach flag: part_number_config[0]
> > > > > +		if cc.has_argument(flag)
> > > > > +			machine_args += flag
> > > > >  		endif
> > > > >  	endforeach
> > > > >
> > > > > -	message('Implementer : ' + machine[0])
> > > > > -	foreach flag: machine[1]
> > > > > +	foreach flag: dpdk_flags
> > > > >  		if flag.length() > 0
> > > > >  			dpdk_conf.set(flag[0], flag[1])
> > > > >  		endif
> > > > >  	endforeach
> > > > > -
> > > > > -	foreach marg: machine[2]
> > > > > -		if marg[0] == impl_pn
> > > > > -			foreach flag: marg[1]
> > > > > -				if cc.has_argument(flag)
> > > > > -					machine_args += flag
> > > > > -				endif
> > > > > -			endforeach
> > > > > -			# Apply any extra machine specific flags.
> > > > > -			foreach flag: marg.get(2, flags_default_extra)
> > > > > -				if flag.length() > 0
> > > > > -					dpdk_conf.set(flag[0],
> flag[1])
> > > > > -				endif
> > > > > -			endforeach
> > > > > -		endif
> > > > > -	endforeach
> > > > >  endif
> > > > > -message(machine_args)
> > > > > +
> > > > > +message('Using machine args: @0@'.format(machine_args))
> > > > >
> > > > >  if (cc.get_define('__ARM_NEON', args: machine_args) != '' or
> > > > >      cc.get_define('__aarch64__', args: machine_args) != '')
> > > > > --
> > > > > 2.20.1

Re: [dpdk-dev] [PATCH v4 2/6] build: refactor Arm build

[Juraj Linkeš]

> -----Original Message-----
> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Sent: Monday, November 2, 2020 8:32 PM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>; bruce.richardson@intel.com;
> Ruifeng Wang <Ruifeng.Wang@arm.com>; Phil Yang <Phil.Yang@arm.com>;
> vcchunga@amazon.com; Dharmik Thakkar <Dharmik.Thakkar@arm.com>;
> jerinjacobk@gmail.com; hemant.agrawal@nxp.com
> Cc: dev@dpdk.org; nd <nd@arm.com>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>
> Subject: RE: [PATCH v4 2/6] build: refactor Arm build
> 
> <snip>
> 
> > > > >
> > > > > > diff --git a/config/arm/meson.build b/config/arm/meson.build
> > > > > > index
> > > > > > 491842cad..6c31ab167 100644
> > > > > > --- a/config/arm/meson.build
> > > > > > +++ b/config/arm/meson.build
> > > > > > @@ -3,12 +3,12 @@
> > > > > >  # Copyright(c) 2017 Cavium, Inc  # Copyright(c) 2020
> > > > > > PANTHEON.tech s.r.o.
> > > > > >
> > > > > > -# for checking defines we need to use the correct compiler
> > > > > > flags -march_opt = '-march=@0@'.format(machine)
> > > > > > -
> > > > > > +# set arm_force_native_march if you want to use machine args
> > > > > > +below # instead of discovered values; only works when doing
> > > > > > +an actual native build
> > > > > >  arm_force_native_march = false -arm_force_generic_march =
> > > > > > (machine == 'generic')
> > > > > > +native_machine_args = ['-march=native', '-mtune=native']
> > > > > >
> > > > >
> > > > > [...]
> > > > >
> > > > > > -
> > > > > > -machine_args_default = [
> > > > > > -	['default', ['-march=armv8-a+crc', '-moutline-atomics']],
> > > > > > -	['native', ['-march=native']],
> > > > > > -	['0xd03', ['-mcpu=cortex-a53']],
> > > > > > -	['0xd04', ['-mcpu=cortex-a35']],
> > > > > > -	['0xd07', ['-mcpu=cortex-a57']],
> > > > > > -	['0xd08', ['-mcpu=cortex-a72']],
> > > > > > -	['0xd09', ['-mcpu=cortex-a73']],
> > > > > > -	['0xd0a', ['-mcpu=cortex-a75']],
> > > > > > -	['0xd0b', ['-mcpu=cortex-a76']],
> > > > > > -	['0xd0c', ['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-
> > n1'],
> > > > > > flags_n1sdp_extra]]
> > > > > > -
> > > > > > -machine_args_cavium = [
> > > > > > -	['default', ['-march=armv8-a+crc+crypto','-mcpu=thunderx']],
> > > > > > -	['native', ['-march=native']],
> > > > > > -	['0xa1', ['-mcpu=thunderxt88'], flags_thunderx_extra],
> > > > > > -	['0xa2', ['-mcpu=thunderxt81'], flags_thunderx_extra],
> > > > > > -	['0xa3', ['-mcpu=thunderxt83'], flags_thunderx_extra],
> > > > > > -	['0xaf', ['-march=armv8.1-a+crc+crypto','-
> > mcpu=thunderx2t99'],
> > > > > > flags_thunderx2_extra],
> > > > > > -	['0xb2', ['-march=armv8.2-a+crc+crypto+lse','-
> > mcpu=octeontx2'],
> > > > > > flags_octeontx2_extra]]
> > > > > > -
> > > > > > -machine_args_emag = [
> > > > > > -	['default', ['-march=armv8-a+crc+crypto', '-mtune=emag']],
> > > > > > -	['native', ['-march=native']]]
> > > > > > +	['RTE_USE_C11_MEM_MODEL', true] ] # arm config
> > (implementer
> > > > > > +0x41) is the default config pn_config_default
> > > > > What does it mean by 'default' here? I am somewhat confused
> > > > > between
> > > > 'default'
> > > > > and 'generic'. We should look to remove 'default' as much as
> > > > > possible and stick with 'generic'.
> > > > >
> > > >
> > > > This default means what default means, no special meaning, that is
> > > > if there isn't a more specific configuration, default to this one.
> > > > It's possible that generic is better, but now that I think about
> > > > it, using something else than default or generic might be the best
> > > > to avoid confusion. Possibly just part_number_arm, which will make
> > > > it in line with the
> > > other var names.
> > > Agree, better to call it 'part_number_arm'.
> > >
> > > >
> > > > > > += {
> > > > > > +	'generic': [['-march=armv8-a+crc', '-moutline-atomics']],
> > > > > I like that we have taken out 'native' from this list. Would it
> > > > > be possible to take out 'generic' from this and others below.
> > > > > This is because the binary built with 'generic' build should run
> > > > > on any Arm platform. There is no dependency on any underlying
> platform.
> > > > >
> > > >
> > > > This actually means generic part for the implementer, not generic
> > > > for everything. I understand this is here to produce binaries that
> > > > would run on everything from that impelemeter (in line of what you
> > > > mention below, this would be implementer-generic configuration, a
> > > > fourth category). In my patchset, it's also a fallback when
> > > > building for an unknown part number from the implementer.
> > > We do not need implementer-generic binaries/build. However, we will
> > > have some parameters that are common across all the part numbers
> > > from that implementer (probably we should not call it as
> > > 'implementer-generic' to avoid confusion. May be
> > > 'implementer-common-
> > flags' or 'implementer-flags-extra').
> > > Those parameters can be used for every part number.
> >
> > These are currently named flags_<implementer> such as flags_arm and
> > flags_cavium. We could rename them to implementer_flags_<implementer>.
> Ok
> 
> >
> > >
> > > If we know the implementer, we will have the part number as well (this
> > > is part of the Arm architecture definition). Unknown part number
> > > should result in an error message.
> > >
> >
> > Yes, we'll always have both. But there are still a couple of corner cases with
> > native builds, which should always work:
> I do not think these are corner cases. These are basically cases where the
> support is not added yet in the build system. We do not need to handle these
> specifically.
> I also do not think the notion that the 'native' build should always work is correct
> (at least on Arm platforms). The reason I say that is because 'native', IMO, not
> only defines the ISA, but other parameters as well in DPDK.
> 

So our stance would basically be the native build doesn't work for unsupported implementers/part numbers and then direct the user to generic build. That seems reasonable.

> > 1. We don't support the implementer (i.e. there's no variable with name
> > implementer_<implementer_id>). In this case, what do default to? The
> > generic build, which is the current behavior?
> This should result in an error. The implementer should add support in
> meson.build
> 

Ok, then we need to change the current configuration. The implementers were added in bulk with the same configuration, so that can't really be considered supporting them in this way. I think we should keep implementers which have specific implementer flags or machine configuration defined for them and remove the rest. That would leave us with:
implementer_generic = ['Generic armv8', flags_implementer_generic, part_number_config_arm]
implementer_0x41 = ['Arm', flags_implementer_arm, part_number_config_arm]
implementer_0x43 = ['Cavium', flags_implementer_cavium, part_number_config_cavium]
implementer_0x50 = ['Ampere Computing', flags_implementer_emag, part_number_config_emag]
implementer_0x56 = ['Marvell ARMADA', flags_implementer_armada, part_number_config_arm]
implementer_dpaa = ['NXP DPAA', flags_implementer_dpaa, part_number_config_arm]

And we would remove these (everything with the same flags/part number config as the generic build):
implementer_0x42 = ['Broadcom', flags_implementer_generic, part_number_config_arm]
implementer_0x44 = ['DEC', flags_implementer_generic, part_number_config_arm]
implementer_0x49 = ['Infineon', flags_implementer_generic, part_number_config_arm]
implementer_0x4d = ['Motorola', flags_implementer_generic, part_number_config_arm]
implementer_0x4e = ['NVIDIA', flags_implementer_generic, part_number_config_arm]
implementer_0x51 = ['Qualcomm', flags_implementer_generic, part_number_config_arm]
implementer_0x53 = ['Samsung', flags_implementer_generic, part_number_config_arm]
implementer_0x69 = ['Intel', flags_implementer_generic, part_number_config_arm]

> > 2. We support the implementer, but we don't support the part number. In
> > this case, we can just use native machine args (-march=native) and use only
> > implementer specific flags.
> Same thing, should result in error. The implementer should add support in
> meson.build.
> 

There are two different scenarios, native and soc-specific/cross builds.
For native builds, we could just use the implementer flags with native machine args. This could be still viewed as implementer intending to support it for native build - by omitting it, they'd basically be saying there aren't any extra flags for the part number. There are hidden assumptions here and it results in data that's not clear, so It's probably best to enforce even for native builds, as that we remove any ambiguity.
For soc-specific/cross builds, we don't know which machine args to use, so that's an error.

I'll make the changes so that an unknown implementer/part number results in an error for all builds.

> >
> > For generic, soc-specifc and cross builds it makes sense to error with uknown
> > implementer/part number, since we wouldn't know what to build otherwise.
> >
> > > >
> > > > Since this is not generic for everything, only for the implementer,
> > > > we're lacking the true common default machine args for everything.
> > > >
> > > > > > +	'0xd03': [['-mcpu=cortex-a53']],
> > > > > > +	'0xd04': [['-mcpu=cortex-a35']],
> > > > > > +	'0xd07': [['-mcpu=cortex-a57']],
> > > > > > +	'0xd08': [['-mcpu=cortex-a72']],
> > > > > > +	'0xd09': [['-mcpu=cortex-a73']],
> > > > > > +	'0xd0a': [['-mcpu=cortex-a75']],
> > > > > > +	'0xd0b': [['-mcpu=cortex-a76']],
> > > > > > +	'0xd0c': [['-march=armv8.2-a+crc+crypto',
> > > > > > +'-mcpu=neoverse-n1'], flags_n1sdp_extra]
> > > > > 'flags_n1sdp_extra' does not fit here. For the part number '0xd0c'
> > > > > there could be multiple SoCs (N1SDP is one of them). So, if the
> > > > > SoC is not known, but if we know that the CPU is N1, then we
> > > > > should build a N1-
> > > > Generic build.
> > > >
> > > > This should be core-generic configuration, I can rename it to
> > > > flags_n1generic_extra.
> > > Agree.
> > >
> > > >
> > > > > So, from my perspective, there are 3 kinds of binaries:
> > > > > 1) generic - best portability -  (possibly) least optimized for a
> > > > > given platform
> > > > > 2) Core-Generic (for ex: N1-generic) - Portable on all N1 based
> > > > > SoCs only - Optimized for N1 cores
> > > > > 3) SoC specific - (possibly) Not portable - Most optimized for the
> > > > > SoC
> > > > >
> > > >
> > > > The fourth category I mentioned above, implementer-generic, is used
> > > > in how
> > > We should not have this category.
> > >
> > > > the arm configuration is currently processed:
> > > > 1) default configuration
> > > > Added to/overwritten by
> > > > 2) implementer configuration (one of which is the configuration for
> > > > generic
> > > > build) Added to/overwritten by
> > > This should be just parameters that are common across all the part
> > > numbers of that implementer (for ex: RTE_CACHE_LINE_SIZE = 128), not a
> > build.
> > >
> >
> > It's not a build, just how the configation is organized and processed, as in we
> > apply the common default flags, then implementer flags, then part number
> > flags and then soc flags, overwriting flags in all of the steps.
> Ok
> 
> >
> > > > 3) part number (or core-generic) configuration Added to/overwritten
> > > > by
> > > > 4) SoC configuration (what we're adding now)
> > > >
> > > > It's not organized as cleanly - the machine args contain both 2) and 3).
> > > > Depending on what we want to do with the 'generic' part number I'll
> > > > adjust the config organization.
> > > >
> > > > >  } pn_config_cavium = {
> > > > > > +	'generic': [['-march=armv8-a+crc+crypto', '-
> > mcpu=thunderx']],
> > > > > 'generic' does not make sense here.
> > > > >
> > > > > > +	'0xa1': [['-mcpu=thunderxt88'], flags_thunderx_extra],
> > > > > > +	'0xa2': [['-mcpu=thunderxt81'], flags_thunderx_extra],
> > > > > > +	'0xa3': [['-mcpu=thunderxt83'], flags_thunderx_extra],
> > > > > > +	'0xaf': [['-march=armv8.1-a+crc+crypto','-
> > mcpu=thunderx2t99'],
> > > > > > flags_thunderx2_extra],
> > > > > > +	'0xb2':
> > > > > > +[['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'],
> > > > > > +flags_octeontx2_extra], } pn_config_emag = {
> > > > > > +	'generic': [['-march=armv8-a+crc+crypto', '-mtune=emag']] }
> > > > > Same here.
> > > > > I understand that this is coming from the existing code. But, I
> > > > > think we should try to set it right.
> > > > >
> > > >
> > > > The generic config for these makes sense if a fourth category,
> > > > implementer- generic makes sense.
> > > > For example, for part_number_config_emag this means: for
> > implementer
> > > > _0x50 (Ampere Computing) use the generic machine args if there's
> > > > nothing more specific (which there isn't - no other part number).
> > > There should be a part number for this. Not sure why it is not present
> > > here. I will find out the info.
> > >
> > > >
> > > > > >
> > > > > >  ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page
> > > > > > G7-5321)
> > > > > Nit, Would be good to remove the reference to the doc
> > > > >
> > > >
> > > > Is that because the docs mentioned here may not be the most recent?
> > > > It was useful to me in understanding where the implementer/part
> > > > number values come from.
> > > Yes, mainly because the spec gets updated. Instead you could say
> > > "Refer to MIDR in Arm Architecture Reference Manual")
> > >
> >
> > Ok, I'll make this change.
> >
> > > >
> > > > > > -impl_generic = ['Generic armv8', flags_generic,
> > > > > > machine_args_default]
> > > > > > -impl_0x41 = ['Arm', flags_arm, machine_args_default]
> > > > > > -impl_0x42 = ['Broadcom', flags_generic, machine_args_default]
> > > > > > -impl_0x43 = ['Cavium', flags_cavium, machine_args_cavium]
> > > > > > -impl_0x44 = ['DEC', flags_generic, machine_args_default]
> > > > > > -impl_0x49 = ['Infineon', flags_generic, machine_args_default]
> > > > > > -impl_0x4d = ['Motorola', flags_generic, machine_args_default]
> > > > > > -impl_0x4e = ['NVIDIA', flags_generic, machine_args_default]
> > > > > > -impl_0x50 = ['Ampere Computing', flags_emag,
> > machine_args_emag]
> > > > > > -impl_0x51 = ['Qualcomm', flags_generic, machine_args_default]
> > > > > > -impl_0x53 = ['Samsung', flags_generic, machine_args_default]
> > > > > > -impl_0x56 = ['Marvell ARMADA', flags_armada,
> > > > > > machine_args_default]
> > > > > > -impl_0x69 = ['Intel', flags_generic, machine_args_default]
> > > > > > -impl_dpaa = ['NXP DPAA', flags_dpaa, machine_args_default]
> > > > > > +impl_generic = ['Generic armv8', flags_generic,
> > > > > > +pn_config_default]
> > > > > > +impl_0x41 = ['Arm', flags_arm, pn_config_default]
> > > > > > +impl_0x42 = ['Broadcom', flags_generic, pn_config_default]
> > > > > > +impl_0x43 = ['Cavium', flags_cavium, pn_config_cavium]
> > > > > > +impl_0x44 = ['DEC', flags_generic, pn_config_default]
> > > > > > +impl_0x49 = ['Infineon', flags_generic, pn_config_default]
> > > > > > +impl_0x4d = ['Motorola', flags_generic, pn_config_default]
> > > > > > +impl_0x4e = ['NVIDIA', flags_generic, pn_config_default]
> > > > > > +impl_0x50 = ['Ampere Computing', flags_emag, pn_config_emag]
> > > > > > +impl_0x51 = ['Qualcomm', flags_generic, pn_config_default]
> > > > > > +impl_0x53 = ['Samsung', flags_generic, pn_config_default]
> > > > > > +impl_0x56 = ['Marvell ARMADA', flags_armada, pn_config_default]
> > > > > > +impl_0x69 = ['Intel', flags_generic, pn_config_default]
> > > > > > +impl_dpaa = ['NXP DPAA', flags_dpaa, pn_config_default]
> > > > > >
> > > > > >  dpdk_conf.set('RTE_ARCH_ARM', 1)
> > > > > > dpdk_conf.set('RTE_FORCE_INTRINSICS', 1)
> > > > > >
> > > > > >  if dpdk_conf.get('RTE_ARCH_32')
> > > > > > +	# armv7 build
> > > > > >  	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
> > > > > >  	dpdk_conf.set('RTE_ARCH_ARMv7', 1)
> > > > > >  	# the minimum architecture supported, armv7-a, needs the
> > following,
> > > > > >  	# mk/machine/armv7a/rte.vars.mk sets it too
> > > > > >  	machine_args += '-mfpu=neon'
> > > > > >  else
> > > > > > -	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
> > > > > > -	dpdk_conf.set('RTE_ARCH_ARM64', 1)
> > > > > > +	# aarch64 build
> > > > > > +	if not meson.is_cross_build()
> > > > > > +		if machine == 'generic'
> > > > > > +			# default build
> > > > >                                               ^^^^^^^^^^^ Generic build?
> > > > >
> > > >
> > > > Already addressed in the next version.
> > > >
> > > > > > +			impl_config = impl_generic
> > > > > > +			part_number = 'generic'
> > > > > > +		else
> > > > > > +			# native build
> > > > > > +			# The script returns ['Implementer', 'Variant',
> > > > > > 'Architecture',
> > > > > > +			# 'Primary Part number', 'Revision']
> > > > > > +			detect_vendor = find_program(join_paths(
> > > > > > +					meson.current_source_dir(),
> > > > > > 'armv8_machine.py'))
> > > > > > +			cmd = run_command(detect_vendor.path())
> > > > > > +			if cmd.returncode() == 0
> > > > > > +				cmd_output =
> > > > > > cmd.stdout().to_lower().strip().split(' ')
> > > > > > +			endif
> > > > > > +			if arm_force_native_march == true
> > > > > > +				part_number = 'native'
> > > > > > +			else
> > > > > > +				part_number = cmd_output[3]
> > > > > > +			endif
> > > > > > +			# Set to generic implementer if implementer
> > is not
> > > > > > found
> > > > > > +			impl_config = get_variable('impl_' +
> > cmd_output[0],
> > > > > > 'impl_generic')
> > > > > > +		endif
> > > > > > +	else
> > > > > > +		# cross build
> > > > > > +		impl_id =
> > meson.get_cross_property('implementer_id',
> > > > > > 'generic')
> > > > > > +		part_number =
> > meson.get_cross_property('part_number',
> > > > > > 'generic')
> > > > > > +		impl_config = get_variable('impl_' + impl_id)
> > > > > > +	endif
> > > > > >
> > > > > > -	machine = []
> > > > > > -	cmd_generic = ['generic', '', '', 'default', '']
> > > > > > -	cmd_output = cmd_generic # Set generic by default
> > > > > > -	machine_args = [] # Clear previous machine args
> > > > > > -	if arm_force_generic_march and not meson.is_cross_build()
> > > > > > -		machine = impl_generic
> > > > > > -		impl_pn = 'default'
> > > > > > +	message('Arm implementer: ' + impl_config[0])
> > > > > > +	message('Arm part number: ' + part_number)
> > > > > > +
> > > > > > +	implementer_flags = impl_config[1]
> > > > > > +	part_number_config = impl_config[2]
> > > > > > +
> > > > > > +	if part_number_config.has_key(part_number)
> > > > > > +		# use the specified part_number machine args if
> > found
> > > > > > +		part_number_config =
> > part_number_config[part_number]
> > > > > > +	elif part_number == 'native'
> > > > > > +		# use native machine args
> > > > > > +		part_number_config = [[native_machine_args]]
> > > > > >  	elif not meson.is_cross_build()
> > > > > > -		# The script returns ['Implementer', 'Variant',
> > 'Architecture',
> > > > > > -		# 'Primary Part number', 'Revision']
> > > > > > -		detect_vendor = find_program(join_paths(
> > > > > > -				meson.current_source_dir(),
> > > > > > 'armv8_machine.py'))
> > > > > > -		cmd = run_command(detect_vendor.path())
> > > > > > -		if cmd.returncode() == 0
> > > > > > -			cmd_output =
> > cmd.stdout().to_lower().strip().split('
> > > > > > ')
> > > > > > -		endif
> > > > > > -		# Set to generic if variable is not found
> > > > > > -		machine = get_variable('impl_' + cmd_output[0],
> > ['generic'])
> > > > > > -		if machine[0] == 'generic'
> > > > > > -			machine = impl_generic
> > > > > > -			cmd_output = cmd_generic
> > > > > > -		endif
> > > > > > -		impl_pn = cmd_output[3]
> > > > > > -		if arm_force_native_march == true
> > > > > > -			impl_pn = 'native'
> > > > > > -		endif
> > > > > > +		# default to generic machine args if part_number is
> > not found
> > > > > > +		# and not forcing native machine args
> > > > > > +		# but don't default in cross-builds; if part_number is
> > specified
> > > > > > +		# incorrectly in a cross-file, it needs to be fixed there
> > > > > > +		part_number_config =
> > part_number_config['generic']
> > > > > >  	else
> > > > > > -		impl_id =
> > meson.get_cross_property('implementor_id',
> > > > > > 'generic')
> > > > > > -		impl_pn =
> > meson.get_cross_property('implementor_pn',
> > > > > > 'default')
> > > > > > -		machine = get_variable('impl_' + impl_id)
> > > > > > +		# cross build and part number is not in
> > part_number_config
> > > > > > +		error('Cross build part number 0@0 not
> > > > > > found.'.format(part_number))
> > > > > >  	endif
> > > > > >
> > > > > > -	# Apply Common Defaults. These settings may be
> > overwritten by
> > > > > > machine
> > > > > > -	# settings later.
> > > > > > -	foreach flag: flags_common_default
> > > > > > -		if flag.length() > 0
> > > > > > -			dpdk_conf.set(flag[0], flag[1])
> > > > > > +	dpdk_flags = flags_common_default + implementer_flags
> > > > > > +
> > > > > > +	if part_number_config.length() > 1
> > > > > > +		dpdk_flags += part_number_config[1]
> > > > > > +	endif
> > > > > > +
> > > > > > +	machine_args = [] # Clear previous machine args
> > > > > > +	foreach flag: part_number_config[0]
> > > > > > +		if cc.has_argument(flag)
> > > > > > +			machine_args += flag
> > > > > >  		endif
> > > > > >  	endforeach
> > > > > >
> > > > > > -	message('Implementer : ' + machine[0])
> > > > > > -	foreach flag: machine[1]
> > > > > > +	foreach flag: dpdk_flags
> > > > > >  		if flag.length() > 0
> > > > > >  			dpdk_conf.set(flag[0], flag[1])
> > > > > >  		endif
> > > > > >  	endforeach
> > > > > > -
> > > > > > -	foreach marg: machine[2]
> > > > > > -		if marg[0] == impl_pn
> > > > > > -			foreach flag: marg[1]
> > > > > > -				if cc.has_argument(flag)
> > > > > > -					machine_args += flag
> > > > > > -				endif
> > > > > > -			endforeach
> > > > > > -			# Apply any extra machine specific flags.
> > > > > > -			foreach flag: marg.get(2, flags_default_extra)
> > > > > > -				if flag.length() > 0
> > > > > > -					dpdk_conf.set(flag[0],
> > flag[1])
> > > > > > -				endif
> > > > > > -			endforeach
> > > > > > -		endif
> > > > > > -	endforeach
> > > > > >  endif
> > > > > > -message(machine_args)
> > > > > > +
> > > > > > +message('Using machine args: @0@'.format(machine_args))
> > > > > >
> > > > > >  if (cc.get_define('__ARM_NEON', args: machine_args) != '' or
> > > > > >      cc.get_define('__aarch64__', args: machine_args) != '')
> > > > > > --
> > > > > > 2.20.1

[dpdk-dev] [PATCH v4 3/6] build: optional NUMA and cpu counts detection

[Juraj Linkeš]

Add an option to automatically discover the host's numa and cpu counts
and use those values for a non cross-build.
Give users the option to override the per-arch default values or values
from cross files by specifying them on the command line with -Dmax_lcores
and -Dmax_numa_nodes.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 buildtools/get_cpu_count.py  |  7 ++++++
 buildtools/get_numa_count.py | 22 +++++++++++++++++
 buildtools/meson.build       |  2 ++
 config/meson.build           | 48 ++++++++++++++++++++++++++++++++++--
 meson_options.txt            |  8 +++---
 5 files changed, 81 insertions(+), 6 deletions(-)
 create mode 100644 buildtools/get_cpu_count.py
 create mode 100644 buildtools/get_numa_count.py

diff --git a/buildtools/get_cpu_count.py b/buildtools/get_cpu_count.py
new file mode 100644
index 000000000..386f85f8b
--- /dev/null
+++ b/buildtools/get_cpu_count.py
@@ -0,0 +1,7 @@
+#!/usr/bin/python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2020 PANTHEON.tech s.r.o.
+
+import os
+
+print(os.cpu_count())
diff --git a/buildtools/get_numa_count.py b/buildtools/get_numa_count.py
new file mode 100644
index 000000000..f0c49973a
--- /dev/null
+++ b/buildtools/get_numa_count.py
@@ -0,0 +1,22 @@
+#!/usr/bin/python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2020 PANTHEON.tech s.r.o.
+
+import ctypes
+import glob
+import os
+import subprocess
+
+if os.name == 'posix':
+    if os.path.isdir('/sys/devices/system/node'):
+        print(len(glob.glob('/sys/devices/system/node/node*')))
+    else:
+        print(subprocess.run(['sysctl', 'vm.ndomains'], capture_output=True).stdout)
+
+elif os.name == 'nt':
+    libkernel32 = ctypes.windll.kernel32
+
+    count = ctypes.c_ulong()
+
+    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
+    print(count.value + 1)
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 04808dabc..925e733b1 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -17,3 +17,5 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+get_cpu_count_cmd = py3 + files('get_cpu_count.py')
+get_numa_count_cmd = py3 + files('get_numa_count.py')
diff --git a/config/meson.build b/config/meson.build
index 918ca2dfb..5c6fdcc7d 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -230,8 +230,14 @@ foreach arg: warning_flags
 endforeach
 
 # set other values pulled from the build options
-dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
-dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
+if not meson.is_cross_build()
+	# set default values
+	# these defaults may be overwritten by meson.build in an arch subdir
+	# or later overwritten if a user specifies a value on the command line
+	# or the user chooses to use values discovered from the build machine
+	dpdk_conf.set('RTE_MAX_LCORE', 128)
+	dpdk_conf.set('RTE_MAX_NUMA_NODES', 4)
+endif
 dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
 dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
 dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
@@ -250,6 +256,44 @@ compile_time_cpuflags = []
 subdir(arch_subdir)
 dpdk_conf.set('RTE_COMPILE_TIME_CPUFLAGS', ','.join(compile_time_cpuflags))
 
+max_lcores = get_option('max_lcores')
+max_numa_nodes = get_option('max_numa_nodes')
+if max_lcores > 0
+	dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
+elif max_lcores == -1
+	if not meson.is_cross_build()
+		max_lcores = run_command(get_cpu_count_cmd).stdout().to_int()
+		min_lcores = 2
+		if max_lcores < min_lcores
+			message('Found less than @0@ cores, building for @0@ cores'.format(min_lcores))
+			max_lcores = min_lcores
+		else
+			message('Found @0@ cores'.format(max_lcores))
+		endif
+		dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
+	endif
+endif
+
+if max_numa_nodes > 0
+	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
+elif max_numa_nodes == -1
+	if not meson.is_cross_build()
+		max_numa_nodes = run_command(get_numa_count_cmd).stdout().to_int()
+		message('Found @0@ numa nodes'.format(max_numa_nodes))
+		dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
+	endif
+endif
+
+# check that cpu and numa count is set in cross builds
+if meson.is_cross_build()
+	if not dpdk_conf.has('RTE_MAX_LCORE')
+		error('Number of cores for cross build not specified in @0@ subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
+	endif
+	if not dpdk_conf.has('RTE_MAX_NUMA_NODES')
+		error('Number of numa nodes for cross build not specified in @0@ subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
+	endif
+endif
+
 # set the install path for the drivers
 dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
 
diff --git a/meson_options.txt b/meson_options.txt
index 9bf18ab6b..a78c21b0d 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -26,10 +26,10 @@ option('machine', type: 'string', value: 'native',
 	description: 'set the target machine type')
 option('max_ethports', type: 'integer', value: 32,
 	description: 'maximum number of Ethernet devices')
-option('max_lcores', type: 'integer', value: 128,
-	description: 'maximum number of cores/threads supported by EAL')
-option('max_numa_nodes', type: 'integer', value: 4,
-	description: 'maximum number of NUMA nodes supported by EAL')
+option('max_lcores', type: 'integer', value: 0,
+	description: 'maximum number of cores/threads supported by EAL. Set to positive integer to overwrite per-arch or cross-compilation defaults. Set to -1 to use number of cores on the build machine.')
+option('max_numa_nodes', type: 'integer', value: 0,
+	description: 'maximum number of NUMA nodes supported by EAL. Set to positive integer to overwrite per-arch or cross-compilation defaults. Set to -1 to use number of numa nodes on the build machine.')
 option('enable_trace_fp', type: 'boolean', value: false,
 	description: 'enable fast path trace points.')
 option('tests', type: 'boolean', value: true,
-- 
2.20.1

Re: [dpdk-dev] [PATCH v4 3/6] build: optional NUMA and cpu counts detection

[Bruce Richardson]

On Fri, Oct 23, 2020 at 04:48:05PM +0200, Juraj Linkeš wrote:
> Add an option to automatically discover the host's numa and cpu counts
> and use those values for a non cross-build.
> Give users the option to override the per-arch default values or values
> from cross files by specifying them on the command line with -Dmax_lcores
> and -Dmax_numa_nodes.
> 
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
>  buildtools/get_cpu_count.py  |  7 ++++++
>  buildtools/get_numa_count.py | 22 +++++++++++++++++
>  buildtools/meson.build       |  2 ++
>  config/meson.build           | 48 ++++++++++++++++++++++++++++++++++--
>  meson_options.txt            |  8 +++---
>  5 files changed, 81 insertions(+), 6 deletions(-)
>  create mode 100644 buildtools/get_cpu_count.py
>  create mode 100644 buildtools/get_numa_count.py
> 
> diff --git a/buildtools/get_cpu_count.py b/buildtools/get_cpu_count.py
> new file mode 100644
> index 000000000..386f85f8b
> --- /dev/null
> +++ b/buildtools/get_cpu_count.py
> @@ -0,0 +1,7 @@
> +#!/usr/bin/python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2020 PANTHEON.tech s.r.o.
> +
> +import os
> +
> +print(os.cpu_count())
> diff --git a/buildtools/get_numa_count.py b/buildtools/get_numa_count.py
> new file mode 100644
> index 000000000..f0c49973a
> --- /dev/null
> +++ b/buildtools/get_numa_count.py
> @@ -0,0 +1,22 @@
> +#!/usr/bin/python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2020 PANTHEON.tech s.r.o.
> +
> +import ctypes
> +import glob
> +import os
> +import subprocess
> +
> +if os.name == 'posix':
> +    if os.path.isdir('/sys/devices/system/node'):
> +        print(len(glob.glob('/sys/devices/system/node/node*')))
> +    else:
> +        print(subprocess.run(['sysctl', 'vm.ndomains'], capture_output=True).stdout)

I think you can shorten this, by just calling subprocess.run and not
capturing anything, in which case the stdout will be printed as normal.

	subprocess.run(['sysctl', 'vm.ndomains'])

> +
> +elif os.name == 'nt':
> +    libkernel32 = ctypes.windll.kernel32
> +
> +    count = ctypes.c_ulong()
> +
> +    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
> +    print(count.value + 1)
> diff --git a/buildtools/meson.build b/buildtools/meson.build
> index 04808dabc..925e733b1 100644
> --- a/buildtools/meson.build
> +++ b/buildtools/meson.build
> @@ -17,3 +17,5 @@ else
>  endif
>  map_to_win_cmd = py3 + files('map_to_win.py')
>  sphinx_wrapper = py3 + files('call-sphinx-build.py')
> +get_cpu_count_cmd = py3 + files('get_cpu_count.py')
> +get_numa_count_cmd = py3 + files('get_numa_count.py')
> diff --git a/config/meson.build b/config/meson.build
> index 918ca2dfb..5c6fdcc7d 100644
> --- a/config/meson.build
> +++ b/config/meson.build
> @@ -230,8 +230,14 @@ foreach arg: warning_flags
>  endforeach
>  
>  # set other values pulled from the build options
> -dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
> -dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
> +if not meson.is_cross_build()
> +	# set default values
> +	# these defaults may be overwritten by meson.build in an arch subdir
> +	# or later overwritten if a user specifies a value on the command line
> +	# or the user chooses to use values discovered from the build machine
> +	dpdk_conf.set('RTE_MAX_LCORE', 128)
> +	dpdk_conf.set('RTE_MAX_NUMA_NODES', 4)
> +endif

I think it might be better to remove this block, and instead set the values
below where you do the additional checks. It's confusing having these
settings managed in multiple places.

>  dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
>  dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
>  dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
> @@ -250,6 +256,44 @@ compile_time_cpuflags = []
>  subdir(arch_subdir)
>  dpdk_conf.set('RTE_COMPILE_TIME_CPUFLAGS', ','.join(compile_time_cpuflags))
>  
> +max_lcores = get_option('max_lcores')
> +max_numa_nodes = get_option('max_numa_nodes')
> +if max_lcores > 0
> +	dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> +elif max_lcores == -1
> +	if not meson.is_cross_build()
> +		max_lcores = run_command(get_cpu_count_cmd).stdout().to_int()
> +		min_lcores = 2
> +		if max_lcores < min_lcores
> +			message('Found less than @0@ cores, building for @0@ cores'.format(min_lcores))
> +			max_lcores = min_lcores
> +		else
> +			message('Found @0@ cores'.format(max_lcores))
> +		endif
> +		dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> +	endif
> +endif
> +
> +if max_numa_nodes > 0
> +	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
> +elif max_numa_nodes == -1
> +	if not meson.is_cross_build()
> +		max_numa_nodes = run_command(get_numa_count_cmd).stdout().to_int()
> +		message('Found @0@ numa nodes'.format(max_numa_nodes))
> +		dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
> +	endif
> +endif
> +
> +# check that cpu and numa count is set in cross builds
> +if meson.is_cross_build()
> +	if not dpdk_conf.has('RTE_MAX_LCORE')
> +		error('Number of cores for cross build not specified in @0@ subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
> +	endif
> +	if not dpdk_conf.has('RTE_MAX_NUMA_NODES')
> +		error('Number of numa nodes for cross build not specified in @0@ subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
> +	endif
> +endif
> +

I think we need some comments explaining all the logic here, and probably
some documentation updates.

>  # set the install path for the drivers
>  dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
>  
> diff --git a/meson_options.txt b/meson_options.txt
> index 9bf18ab6b..a78c21b0d 100644
> --- a/meson_options.txt
> +++ b/meson_options.txt
> @@ -26,10 +26,10 @@ option('machine', type: 'string', value: 'native',
>  	description: 'set the target machine type')
>  option('max_ethports', type: 'integer', value: 32,
>  	description: 'maximum number of Ethernet devices')
> -option('max_lcores', type: 'integer', value: 128,
> -	description: 'maximum number of cores/threads supported by EAL')
> -option('max_numa_nodes', type: 'integer', value: 4,
> -	description: 'maximum number of NUMA nodes supported by EAL')
> +option('max_lcores', type: 'integer', value: 0,
> +	description: 'maximum number of cores/threads supported by EAL. Set to positive integer to overwrite per-arch or cross-compilation defaults. Set to -1 to use number of cores on the build machine.')
> +option('max_numa_nodes', type: 'integer', value: 0,
> +	description: 'maximum number of NUMA nodes supported by EAL. Set to positive integer to overwrite per-arch or cross-compilation defaults. Set to -1 to use number of numa nodes on the build machine.')
>  option('enable_trace_fp', type: 'boolean', value: false,
>  	description: 'enable fast path trace points.')
>  option('tests', type: 'boolean', value: true,
> -- 
> 2.20.1
> 

Re: [dpdk-dev] [PATCH v4 3/6] build: optional NUMA and cpu counts detection

[Juraj Linkeš]

> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Tuesday, October 27, 2020 12:21 PM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Cc: Ruifeng.Wang@arm.com; Honnappa.Nagarahalli@arm.com;
> Phil.Yang@arm.com; vcchunga@amazon.com; Dharmik.Thakkar@arm.com;
> jerinjacobk@gmail.com; hemant.agrawal@nxp.com; dev@dpdk.org
> Subject: Re: [PATCH v4 3/6] build: optional NUMA and cpu counts detection
> 
> On Fri, Oct 23, 2020 at 04:48:05PM +0200, Juraj Linkeš wrote:
> > Add an option to automatically discover the host's numa and cpu counts
> > and use those values for a non cross-build.
> > Give users the option to override the per-arch default values or
> > values from cross files by specifying them on the command line with
> > -Dmax_lcores and -Dmax_numa_nodes.
> >
> > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > ---
> >  buildtools/get_cpu_count.py  |  7 ++++++
> > buildtools/get_numa_count.py | 22 +++++++++++++++++
> >  buildtools/meson.build       |  2 ++
> >  config/meson.build           | 48 ++++++++++++++++++++++++++++++++++--
> >  meson_options.txt            |  8 +++---
> >  5 files changed, 81 insertions(+), 6 deletions(-)  create mode 100644
> > buildtools/get_cpu_count.py  create mode 100644
> > buildtools/get_numa_count.py
> >
> > diff --git a/buildtools/get_cpu_count.py b/buildtools/get_cpu_count.py
> > new file mode 100644 index 000000000..386f85f8b
> > --- /dev/null
> > +++ b/buildtools/get_cpu_count.py
> > @@ -0,0 +1,7 @@
> > +#!/usr/bin/python3
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > +PANTHEON.tech s.r.o.
> > +
> > +import os
> > +
> > +print(os.cpu_count())
> > diff --git a/buildtools/get_numa_count.py
> > b/buildtools/get_numa_count.py new file mode 100644 index
> > 000000000..f0c49973a
> > --- /dev/null
> > +++ b/buildtools/get_numa_count.py
> > @@ -0,0 +1,22 @@
> > +#!/usr/bin/python3
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > +PANTHEON.tech s.r.o.
> > +
> > +import ctypes
> > +import glob
> > +import os
> > +import subprocess
> > +
> > +if os.name == 'posix':
> > +    if os.path.isdir('/sys/devices/system/node'):
> > +        print(len(glob.glob('/sys/devices/system/node/node*')))
> > +    else:
> > +        print(subprocess.run(['sysctl', 'vm.ndomains'],
> > +capture_output=True).stdout)
> 
> I think you can shorten this, by just calling subprocess.run and not capturing
> anything, in which case the stdout will be printed as normal.
> 
> 	subprocess.run(['sysctl', 'vm.ndomains'])
> 

This will also print out the resulting object (e.g. CompletedProcess(args=['ls', '-ls', '/sys/devices/system/node'], returncode=0)), but an assignment will take care of that. I'll make the change.

> > +
> > +elif os.name == 'nt':
> > +    libkernel32 = ctypes.windll.kernel32
> > +
> > +    count = ctypes.c_ulong()
> > +
> > +    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
> > +    print(count.value + 1)
> > diff --git a/buildtools/meson.build b/buildtools/meson.build index
> > 04808dabc..925e733b1 100644
> > --- a/buildtools/meson.build
> > +++ b/buildtools/meson.build
> > @@ -17,3 +17,5 @@ else
> >  endif
> >  map_to_win_cmd = py3 + files('map_to_win.py')  sphinx_wrapper = py3 +
> > files('call-sphinx-build.py')
> > +get_cpu_count_cmd = py3 + files('get_cpu_count.py')
> > +get_numa_count_cmd = py3 + files('get_numa_count.py')
> > diff --git a/config/meson.build b/config/meson.build index
> > 918ca2dfb..5c6fdcc7d 100644
> > --- a/config/meson.build
> > +++ b/config/meson.build
> > @@ -230,8 +230,14 @@ foreach arg: warning_flags  endforeach
> >
> >  # set other values pulled from the build options
> > -dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
> > -dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
> > +if not meson.is_cross_build()
> > +	# set default values
> > +	# these defaults may be overwritten by meson.build in an arch subdir
> > +	# or later overwritten if a user specifies a value on the command line
> > +	# or the user chooses to use values discovered from the build machine
> > +	dpdk_conf.set('RTE_MAX_LCORE', 128)
> > +	dpdk_conf.set('RTE_MAX_NUMA_NODES', 4) endif
> 
> I think it might be better to remove this block, and instead set the values below
> where you do the additional checks. It's confusing having these settings
> managed in multiple places.
> 

It needs to be set before subdir(arch_subdir) which is why it's here.

The order goes like this:
Global defaults (or x86 defaults)
arch_subdir defaults overwrite those (or non-x86 defaults)
cmdline options or optional discovery overwrite those

What about I move these to config/x86/meson.build? It looks like the place where per-arch defaults should be. Then we'd only have the second part after arch_subdir.

> >  dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
> > dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
> > dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp')) @@
> > -250,6 +256,44 @@ compile_time_cpuflags = []
> >  subdir(arch_subdir)
> >  dpdk_conf.set('RTE_COMPILE_TIME_CPUFLAGS',
> > ','.join(compile_time_cpuflags))
> >
> > +max_lcores = get_option('max_lcores') max_numa_nodes =
> > +get_option('max_numa_nodes') if max_lcores > 0
> > +	dpdk_conf.set('RTE_MAX_LCORE', max_lcores) elif max_lcores == -1
> > +	if not meson.is_cross_build()
> > +		max_lcores =
> run_command(get_cpu_count_cmd).stdout().to_int()
> > +		min_lcores = 2
> > +		if max_lcores < min_lcores
> > +			message('Found less than @0@ cores, building for
> @0@ cores'.format(min_lcores))
> > +			max_lcores = min_lcores
> > +		else
> > +			message('Found @0@ cores'.format(max_lcores))
> > +		endif
> > +		dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> > +	endif
> > +endif
> > +
> > +if max_numa_nodes > 0
> > +	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes) elif
> > +max_numa_nodes == -1
> > +	if not meson.is_cross_build()
> > +		max_numa_nodes =
> run_command(get_numa_count_cmd).stdout().to_int()
> > +		message('Found @0@ numa nodes'.format(max_numa_nodes))
> > +		dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
> > +	endif
> > +endif
> > +
> > +# check that cpu and numa count is set in cross builds if
> > +meson.is_cross_build()
> > +	if not dpdk_conf.has('RTE_MAX_LCORE')
> > +		error('Number of cores for cross build not specified in @0@
> subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
> > +	endif
> > +	if not dpdk_conf.has('RTE_MAX_NUMA_NODES')
> > +		error('Number of numa nodes for cross build not specified in
> @0@ subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
> > +	endif
> > +endif
> > +
> 
> I think we need some comments explaining all the logic here, and probably some
> documentation updates.
> 

Will add comments. Where is this documented?

> >  # set the install path for the drivers
> > dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
> >
> > diff --git a/meson_options.txt b/meson_options.txt index
> > 9bf18ab6b..a78c21b0d 100644
> > --- a/meson_options.txt
> > +++ b/meson_options.txt
> > @@ -26,10 +26,10 @@ option('machine', type: 'string', value: 'native',
> >  	description: 'set the target machine type')  option('max_ethports',
> > type: 'integer', value: 32,
> >  	description: 'maximum number of Ethernet devices')
> > -option('max_lcores', type: 'integer', value: 128,
> > -	description: 'maximum number of cores/threads supported by EAL')
> > -option('max_numa_nodes', type: 'integer', value: 4,
> > -	description: 'maximum number of NUMA nodes supported by EAL')
> > +option('max_lcores', type: 'integer', value: 0,
> > +	description: 'maximum number of cores/threads supported by EAL. Set
> > +to positive integer to overwrite per-arch or cross-compilation defaults. Set to
> -1 to use number of cores on the build machine.') option('max_numa_nodes',
> type: 'integer', value: 0,
> > +	description: 'maximum number of NUMA nodes supported by EAL. Set to
> > +positive integer to overwrite per-arch or cross-compilation defaults.
> > +Set to -1 to use number of numa nodes on the build machine.')
> >  option('enable_trace_fp', type: 'boolean', value: false,
> >  	description: 'enable fast path trace points.')  option('tests',
> > type: 'boolean', value: true,
> > --
> > 2.20.1
> >

Re: [dpdk-dev] [PATCH v4 3/6] build: optional NUMA and cpu counts detection

[Bruce Richardson]

On Tue, Oct 27, 2020 at 03:50:46PM +0000, Juraj Linkeš wrote:
> 
> 
> > -----Original Message-----
> > From: Bruce Richardson <bruce.richardson@intel.com>
> > Sent: Tuesday, October 27, 2020 12:21 PM
> > To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > Cc: Ruifeng.Wang@arm.com; Honnappa.Nagarahalli@arm.com;
> > Phil.Yang@arm.com; vcchunga@amazon.com; Dharmik.Thakkar@arm.com;
> > jerinjacobk@gmail.com; hemant.agrawal@nxp.com; dev@dpdk.org
> > Subject: Re: [PATCH v4 3/6] build: optional NUMA and cpu counts detection
> > 
> > On Fri, Oct 23, 2020 at 04:48:05PM +0200, Juraj Linkeš wrote:
> > > Add an option to automatically discover the host's numa and cpu counts
> > > and use those values for a non cross-build.
> > > Give users the option to override the per-arch default values or
> > > values from cross files by specifying them on the command line with
> > > -Dmax_lcores and -Dmax_numa_nodes.
> > >
> > > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > ---
> > >  buildtools/get_cpu_count.py  |  7 ++++++
> > > buildtools/get_numa_count.py | 22 +++++++++++++++++
> > >  buildtools/meson.build       |  2 ++
> > >  config/meson.build           | 48 ++++++++++++++++++++++++++++++++++--
> > >  meson_options.txt            |  8 +++---
> > >  5 files changed, 81 insertions(+), 6 deletions(-)  create mode 100644
> > > buildtools/get_cpu_count.py  create mode 100644
> > > buildtools/get_numa_count.py
> > >
> > > diff --git a/buildtools/get_cpu_count.py b/buildtools/get_cpu_count.py
> > > new file mode 100644 index 000000000..386f85f8b
> > > --- /dev/null
> > > +++ b/buildtools/get_cpu_count.py
> > > @@ -0,0 +1,7 @@
> > > +#!/usr/bin/python3
> > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > > +PANTHEON.tech s.r.o.
> > > +
> > > +import os
> > > +
> > > +print(os.cpu_count())
> > > diff --git a/buildtools/get_numa_count.py
> > > b/buildtools/get_numa_count.py new file mode 100644 index
> > > 000000000..f0c49973a
> > > --- /dev/null
> > > +++ b/buildtools/get_numa_count.py
> > > @@ -0,0 +1,22 @@
> > > +#!/usr/bin/python3
> > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > > +PANTHEON.tech s.r.o.
> > > +
> > > +import ctypes
> > > +import glob
> > > +import os
> > > +import subprocess
> > > +
> > > +if os.name == 'posix':
> > > +    if os.path.isdir('/sys/devices/system/node'):
> > > +        print(len(glob.glob('/sys/devices/system/node/node*')))
> > > +    else:
> > > +        print(subprocess.run(['sysctl', 'vm.ndomains'],
> > > +capture_output=True).stdout)
> > 
> > I think you can shorten this, by just calling subprocess.run and not capturing
> > anything, in which case the stdout will be printed as normal.
> > 
> > 	subprocess.run(['sysctl', 'vm.ndomains'])
> > 
> 
> This will also print out the resulting object (e.g. CompletedProcess(args=['ls', '-ls', '/sys/devices/system/node'], returncode=0)), but an assignment will take care of that. I'll make the change.
> 

Not unless you are running interactively in the python3 REPL.
For example:

$ cat test_meminfo.py
#! /usr/bin/env python3

from subprocess import run
run(['cat', '/proc/meminfo'])

$ python3 test_meminfo.py | tail -n 5
Hugepagesize:       2048 kB
Hugetlb:        17825792 kB
DirectMap4k:     1056788 kB
DirectMap2M:     9758720 kB
DirectMap1G:    88080384 kB

$

[dpdk-dev] [PATCH v4 4/6] build: add core and NUMA counts to cross files

[Juraj Linkeš]

Add support for setting core count and numa nodes in cross files. The
values specified in cross files will override the default values.
Also add missing default values.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/arm64_armada_linux_gcc    |  2 ++
 config/arm/arm64_armv8_linux_gcc     |  6 ++++++
 config/arm/arm64_bluefield_linux_gcc |  2 ++
 config/arm/arm64_dpaa_linux_gcc      |  2 ++
 config/arm/arm64_emag_linux_gcc      |  2 ++
 config/arm/arm64_n1sdp_linux_gcc     |  2 ++
 config/arm/arm64_octeontx2_linux_gcc |  2 ++
 config/arm/arm64_stingray_linux_gcc  |  2 ++
 config/arm/arm64_thunderx2_linux_gcc |  2 ++
 config/arm/arm64_thunderx_linux_gcc  |  2 ++
 config/arm/meson.build               | 20 ++++++++++++++++++++
 11 files changed, 44 insertions(+)

diff --git a/config/arm/arm64_armada_linux_gcc b/config/arm/arm64_armada_linux_gcc
index 52c5f4476..36100c594 100644
--- a/config/arm/arm64_armada_linux_gcc
+++ b/config/arm/arm64_armada_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 
 [properties]
 implementer_id = '0x56'
+max_numa_nodes = 1
+max_lcores = 16
diff --git a/config/arm/arm64_armv8_linux_gcc b/config/arm/arm64_armv8_linux_gcc
index 13ee8b223..779333199 100644
--- a/config/arm/arm64_armv8_linux_gcc
+++ b/config/arm/arm64_armv8_linux_gcc
@@ -26,3 +26,9 @@ implementer_id = 'generic'
 # '0xd0a':   cortex-a75
 # '0xd0b':   cortex-a76
 part_number = 'generic'
+
+# Supported extra configuration
+# max_numa_nodes = n  # will set RTE_MAX_NUMA_NODES
+# max_lcores = n      # will set RTE_MAX_LCORE
+max_lcores = 256
+max_numa_nodes = 4
diff --git a/config/arm/arm64_bluefield_linux_gcc b/config/arm/arm64_bluefield_linux_gcc
index b79389d85..7ff6fd309 100644
--- a/config/arm/arm64_bluefield_linux_gcc
+++ b/config/arm/arm64_bluefield_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 [properties]
 implementer_id = '0x41'
 part_number = '0xd08'
+max_numa_nodes = 1
+max_lcores = 16
diff --git a/config/arm/arm64_dpaa_linux_gcc b/config/arm/arm64_dpaa_linux_gcc
index 573ae7e42..adae86f57 100644
--- a/config/arm/arm64_dpaa_linux_gcc
+++ b/config/arm/arm64_dpaa_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 
 [properties]
 implementer_id = 'dpaa'
+max_numa_nodes = 1
+max_lcores = 16
diff --git a/config/arm/arm64_emag_linux_gcc b/config/arm/arm64_emag_linux_gcc
index 24f3d533e..3cccca901 100644
--- a/config/arm/arm64_emag_linux_gcc
+++ b/config/arm/arm64_emag_linux_gcc
@@ -14,3 +14,5 @@ endian = 'little'
 
 [properties]
 implementer_id = '0x50'
+max_numa_nodes = 1
+max_lcores = 32
diff --git a/config/arm/arm64_n1sdp_linux_gcc b/config/arm/arm64_n1sdp_linux_gcc
index 6fb3f02ea..3a8b46812 100644
--- a/config/arm/arm64_n1sdp_linux_gcc
+++ b/config/arm/arm64_n1sdp_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 [properties]
 implementer_id = '0x41'
 part_number = '0xd0c'
+max_numa_nodes = 1
+max_lcores = 4
diff --git a/config/arm/arm64_octeontx2_linux_gcc b/config/arm/arm64_octeontx2_linux_gcc
index ac1042806..2baf95a2a 100644
--- a/config/arm/arm64_octeontx2_linux_gcc
+++ b/config/arm/arm64_octeontx2_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 [properties]
 implementer_id = '0x43'
 part_number = '0xb2'
+max_numa_nodes = 1
+max_lcores = 36
diff --git a/config/arm/arm64_stingray_linux_gcc b/config/arm/arm64_stingray_linux_gcc
index b79389d85..7ff6fd309 100644
--- a/config/arm/arm64_stingray_linux_gcc
+++ b/config/arm/arm64_stingray_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 [properties]
 implementer_id = '0x41'
 part_number = '0xd08'
+max_numa_nodes = 1
+max_lcores = 16
diff --git a/config/arm/arm64_thunderx2_linux_gcc b/config/arm/arm64_thunderx2_linux_gcc
index dd257745e..07440fe5f 100644
--- a/config/arm/arm64_thunderx2_linux_gcc
+++ b/config/arm/arm64_thunderx2_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 [properties]
 implementer_id = '0x43'
 part_number = '0xaf'
+max_numa_nodes = 2
+max_lcores = 256
diff --git a/config/arm/arm64_thunderx_linux_gcc b/config/arm/arm64_thunderx_linux_gcc
index 670764437..16ba78fd6 100644
--- a/config/arm/arm64_thunderx_linux_gcc
+++ b/config/arm/arm64_thunderx_linux_gcc
@@ -14,3 +14,5 @@ endian = 'little'
 
 [properties]
 implementer_id = '0x43'
+max_numa_nodes = 1
+max_lcores = 96
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 6c31ab167..b5a62f89d 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -38,12 +38,14 @@ flags_common_default = [
 flags_generic = [
 	['RTE_MACHINE', '"armv8a"'],
 	['RTE_MAX_LCORE', 256],
+	['RTE_MAX_NUMA_NODES', 4],
 	['RTE_USE_C11_MEM_MODEL', true],
 	['RTE_CACHE_LINE_SIZE', 128]
 ]
 flags_arm = [
 	['RTE_MACHINE', '"armv8a"'],
 	['RTE_MAX_LCORE', 16],
+	['RTE_MAX_NUMA_NODES', 1],
 	['RTE_USE_C11_MEM_MODEL', true],
 	['RTE_CACHE_LINE_SIZE', 64]
 ]
@@ -214,6 +216,24 @@ else
 		dpdk_flags += part_number_config[1]
 	endif
 
+	# apply cross-specific options
+	if meson.is_cross_build()
+		# configure RTE_MAX_NUMA_NODES and RTE_MAX_LCORE from cross file
+		cross_max_numa_nodes = meson.get_cross_property('max_numa_nodes', 0)
+		if cross_max_numa_nodes != 0
+			dpdk_flags += [
+				['RTE_MAX_NUMA_NODES', cross_max_numa_nodes]
+			]
+		endif
+		cross_max_lcores = meson.get_cross_property('max_lcores', 0)
+		if cross_max_lcores != 0
+			message('Setting RTE_MAX_LCORE from cross file')
+			dpdk_flags += [
+				['RTE_MAX_LCORE', cross_max_lcores]
+			]
+		endif
+	endif
+
 	machine_args = [] # Clear previous machine args
 	foreach flag: part_number_config[0]
 		if cc.has_argument(flag)
-- 
2.20.1

[dpdk-dev] [PATCH v4 5/6] build: disable Arm drivers

[Juraj Linkeš]

A few options that disabled drivers in the old makefiles were improperly
ported to the meson build system. Fix this by adding a to the list of
disabled drivers, similarly how the command line option works.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
 config/arm/meson.build | 9 +++++----
 drivers/meson.build    | 2 +-
 meson.build            | 1 +
 3 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/config/arm/meson.build b/config/arm/meson.build
index b5a62f89d..5def9e744 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -8,6 +8,11 @@
 arm_force_native_march = false
 native_machine_args = ['-march=native', '-mtune=native']
 
+# RFC: do we want these drivers to be disabled by default on arm?
+# The previous way to disable these drivers never worked
+# RTE_NET_AVP and the rest would be overwritten to true if the necessary deps were found
+disabled_drivers += ['net/avp', 'net/fm10k', 'net/sfc']
+
 # common flags to all aarch64 builds, with lowest priority
 flags_common_default = [
 	# Accelarate rte_memcpy. Be sure to run unit test (memcpy_perf_autotest)
@@ -23,10 +28,6 @@ flags_common_default = [
 	#	['RTE_ARM64_MEMCPY_ALIGN_MASK', 0xF],
 	#	['RTE_ARM64_MEMCPY_STRICT_ALIGN', false],
 
-	['RTE_NET_FM10K', false],
-	['RTE_NET_SFC_EFX', false],
-	['RTE_NET_AVP', false],
-
 	['RTE_SCHED_VECTOR', false],
 	['RTE_ARM_USE_WFE', false],
 	['RTE_CACHE_LINE_SIZE', 128],
diff --git a/drivers/meson.build b/drivers/meson.build
index a5a6fed06..a41ff909e 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -18,7 +18,7 @@ subdirs = [
 	'baseband', # depends on common and bus.
 ]
 
-disabled_drivers = run_command(list_dir_globs, get_option('disable_drivers'),
+disabled_drivers += run_command(list_dir_globs, get_option('disable_drivers'),
 		).stdout().split()
 
 default_cflags = machine_args
diff --git a/meson.build b/meson.build
index 61d9a4f5f..8dadd70dc 100644
--- a/meson.build
+++ b/meson.build
@@ -21,6 +21,7 @@ dpdk_drivers = []
 dpdk_extra_ldflags = []
 dpdk_libs_disabled = []
 dpdk_drvs_disabled = []
+disabled_drivers = []
 abi_version_file = files('ABI_VERSION')
 
 if host_machine.cpu_family().startswith('x86')
-- 
2.20.1

[dpdk-dev] [PATCH v4 6/6] build: update Arm builds with makefile flags

[Juraj Linkeš]

Not all flags were moved properly from the old Makefile build system.
Disable corresponding drivers and libnuma in cross files and remove
deprecated config.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/arm64_armada_linux_gcc    | 2 ++
 config/arm/arm64_armv8_linux_gcc     | 8 ++++++--
 config/arm/arm64_bluefield_linux_gcc | 1 +
 config/arm/arm64_dpaa_linux_gcc      | 1 +
 config/arm/arm64_n1sdp_linux_gcc     | 1 +
 config/arm/arm64_octeontx2_linux_gcc | 1 +
 config/arm/arm64_stingray_linux_gcc  | 1 +
 config/arm/meson.build               | 5 ++++-
 drivers/meson.build                  | 4 ++++
 9 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/config/arm/arm64_armada_linux_gcc b/config/arm/arm64_armada_linux_gcc
index 36100c594..e33f1e742 100644
--- a/config/arm/arm64_armada_linux_gcc
+++ b/config/arm/arm64_armada_linux_gcc
@@ -17,3 +17,5 @@ endian = 'little'
 implementer_id = '0x56'
 max_numa_nodes = 1
 max_lcores = 16
+has_libnuma = 0
+disabled_drivers = ['bus/dpaa', 'bus/fslmc', 'common/dpaax']
diff --git a/config/arm/arm64_armv8_linux_gcc b/config/arm/arm64_armv8_linux_gcc
index 779333199..c6010e303 100644
--- a/config/arm/arm64_armv8_linux_gcc
+++ b/config/arm/arm64_armv8_linux_gcc
@@ -28,7 +28,11 @@ implementer_id = 'generic'
 part_number = 'generic'
 
 # Supported extra configuration
-# max_numa_nodes = n  # will set RTE_MAX_NUMA_NODES
-# max_lcores = n      # will set RTE_MAX_LCORE
+# max_numa_nodes = n   # will set RTE_MAX_NUMA_NODES
+# max_lcores = n        # will set RTE_MAX_LCORE
 max_lcores = 256
 max_numa_nodes = 4
+
+# has_libnuma = 0      # set to 0 if the SoC is not a NUMA system
+# disabled_drivers += ['bus/dpaa', 'crypto']
+    # add to the set of disabled libraries
diff --git a/config/arm/arm64_bluefield_linux_gcc b/config/arm/arm64_bluefield_linux_gcc
index 7ff6fd309..e154ac4fd 100644
--- a/config/arm/arm64_bluefield_linux_gcc
+++ b/config/arm/arm64_bluefield_linux_gcc
@@ -17,3 +17,4 @@ implementer_id = '0x41'
 part_number = '0xd08'
 max_numa_nodes = 1
 max_lcores = 16
+has_libnuma = 0
diff --git a/config/arm/arm64_dpaa_linux_gcc b/config/arm/arm64_dpaa_linux_gcc
index adae86f57..78883a2ec 100644
--- a/config/arm/arm64_dpaa_linux_gcc
+++ b/config/arm/arm64_dpaa_linux_gcc
@@ -17,3 +17,4 @@ endian = 'little'
 implementer_id = 'dpaa'
 max_numa_nodes = 1
 max_lcores = 16
+has_libnuma = 0
diff --git a/config/arm/arm64_n1sdp_linux_gcc b/config/arm/arm64_n1sdp_linux_gcc
index 3a8b46812..4493227f3 100644
--- a/config/arm/arm64_n1sdp_linux_gcc
+++ b/config/arm/arm64_n1sdp_linux_gcc
@@ -17,3 +17,4 @@ implementer_id = '0x41'
 part_number = '0xd0c'
 max_numa_nodes = 1
 max_lcores = 4
+has_libnuma = 0
diff --git a/config/arm/arm64_octeontx2_linux_gcc b/config/arm/arm64_octeontx2_linux_gcc
index 2baf95a2a..ff9e5c403 100644
--- a/config/arm/arm64_octeontx2_linux_gcc
+++ b/config/arm/arm64_octeontx2_linux_gcc
@@ -17,3 +17,4 @@ implementer_id = '0x43'
 part_number = '0xb2'
 max_numa_nodes = 1
 max_lcores = 36
+has_libnuma = 0
diff --git a/config/arm/arm64_stingray_linux_gcc b/config/arm/arm64_stingray_linux_gcc
index 7ff6fd309..e154ac4fd 100644
--- a/config/arm/arm64_stingray_linux_gcc
+++ b/config/arm/arm64_stingray_linux_gcc
@@ -17,3 +17,4 @@ implementer_id = '0x41'
 part_number = '0xd08'
 max_numa_nodes = 1
 max_lcores = 16
+has_libnuma = 0
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 5def9e744..fc6fdf3ba 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -103,7 +103,6 @@ flags_octeontx2_extra = [
 	['RTE_MAX_NUMA_NODES', 1],
 	['RTE_MAX_LCORE', 36],
 	['RTE_ARM_FEATURE_ATOMICS', true],
-	['RTE_EAL_IGB_UIO', false],
 	['RTE_USE_C11_MEM_MODEL', true]
 ]
 # arm config (implementer 0x41) is the default config
@@ -233,6 +232,10 @@ else
 				['RTE_MAX_LCORE', cross_max_lcores]
 			]
 		endif
+		cross_has_libnuma = meson.get_cross_property('has_libnuma', '')
+		if cross_has_libnuma != ''
+			has_libnuma = cross_has_libnuma
+		endif
 	endif
 
 	machine_args = [] # Clear previous machine args
diff --git a/drivers/meson.build b/drivers/meson.build
index a41ff909e..4bdc3d5c9 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -21,6 +21,10 @@ subdirs = [
 disabled_drivers += run_command(list_dir_globs, get_option('disable_drivers'),
 		).stdout().split()
 
+if meson.is_cross_build()
+	disabled_drivers += meson.get_cross_property('disabled_drivers', [])
+endif
+
 default_cflags = machine_args
 default_cflags += ['-DALLOW_EXPERIMENTAL_API']
 default_cflags += ['-DALLOW_INTERNAL_API']
-- 
2.20.1

Re: [dpdk-dev] [PATCH v4 0/6] Arm build options rework

[Honnappa Nagarahalli]

<snip>

> 
> The current way of specifying Arm configuration options is insufficient since
> we can't identify the SoC we're building for from the MIDR information. For
> example, we can't distinguish between N1SDP, Graviton2 or Ampere Altra.
> 
> Add a way to specify the cpu count and numa node count for cross builds.
> 
> We also want to be able to disable which drivers (and possibly
> libraries) are built without user input. This is useful when building:
> 1. on an SoC that is slow and we want to build only what is necessary
>   without the user having to check which libraries they have installed 2. a
> cross build on a fast aarch64 machine but with target SoC which
I assume cross building on x86 for a Arm target SoC does not need any additional changes.

>   differs in capabilities or libraries.
> This is achieved by specifying the drivers in cross files.
> 
> Among libraries, only libnuma can be now disabled.
> 
> Also add an optional way to discover cpu count a numa node count. Fix -
> Dmax_lcores and -Dmax_numa_nodes for arm builds.
> 
> v2:
> Major rework of the whole series.
> 
> v3:
> Added numa and core count defaults for x86 default build.
> Removed numa and core count defaults. Now requiring defaults to be
> specified in a cross file or on the cmdline.
> Added FreeBDS support for numa count discovery.
> 
> v4:
> Make automatic numa and cpu counts discovery optional.
> 
> Juraj Linkeš (6):
>   build: alias default build as generic
>   build: refactor Arm build
>   build: optional NUMA and cpu counts detection
>   build: add core and NUMA counts to cross files
>   build: disable Arm drivers
>   build: update Arm builds with makefile flags
> 
>  buildtools/get_cpu_count.py          |   7 +
>  buildtools/get_numa_count.py         |  22 +++
>  buildtools/meson.build               |   2 +
>  config/arm/arm64_armada_linux_gcc    |   6 +-
>  config/arm/arm64_armv8_linux_gcc     |  18 +-
>  config/arm/arm64_bluefield_linux_gcc |   7 +-
>  config/arm/arm64_dpaa_linux_gcc      |   5 +-
>  config/arm/arm64_emag_linux_gcc      |   4 +-
>  config/arm/arm64_n1sdp_linux_gcc     |   7 +-
>  config/arm/arm64_octeontx2_linux_gcc |   7 +-
>  config/arm/arm64_stingray_linux_gcc  |   7 +-
>  config/arm/arm64_thunderx2_linux_gcc |   6 +-
>  config/arm/arm64_thunderx_linux_gcc  |   4 +-
>  config/arm/meson.build               | 278 ++++++++++++++++-----------
>  config/meson.build                   |  54 +++++-
>  drivers/meson.build                  |   6 +-
>  meson.build                          |   1 +
>  meson_options.txt                    |   8 +-
>  18 files changed, 306 insertions(+), 143 deletions(-)  create mode 100644
> buildtools/get_cpu_count.py  create mode 100644
> buildtools/get_numa_count.py
> 
> --
> 2.20.1

Re: [dpdk-dev] [PATCH v4 0/6] Arm build options rework

[Juraj Linkeš]

> -----Original Message-----
> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Sent: Tuesday, October 27, 2020 4:54 AM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>; bruce.richardson@intel.com;
> Ruifeng Wang <Ruifeng.Wang@arm.com>; Phil Yang <Phil.Yang@arm.com>;
> vcchunga@amazon.com; Dharmik Thakkar <Dharmik.Thakkar@arm.com>;
> jerinjacobk@gmail.com; hemant.agrawal@nxp.com
> Cc: dev@dpdk.org; nd <nd@arm.com>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>
> Subject: RE: [PATCH v4 0/6] Arm build options rework
> 
> <snip>
> 
> >
> > The current way of specifying Arm configuration options is
> > insufficient since we can't identify the SoC we're building for from
> > the MIDR information. For example, we can't distinguish between N1SDP,
> Graviton2 or Ampere Altra.
> >
> > Add a way to specify the cpu count and numa node count for cross builds.
> >
> > We also want to be able to disable which drivers (and possibly
> > libraries) are built without user input. This is useful when building:
> > 1. on an SoC that is slow and we want to build only what is necessary
> >   without the user having to check which libraries they have installed
> > 2. a cross build on a fast aarch64 machine but with target SoC which
> I assume cross building on x86 for a Arm target SoC does not need any
> additional changes.
> 

Here I mean cross build to mean using a cross-file, so in this respect the cross build would be the same on x86 and Arm.

> >   differs in capabilities or libraries.
> > This is achieved by specifying the drivers in cross files.
> >
> > Among libraries, only libnuma can be now disabled.
> >
> > Also add an optional way to discover cpu count a numa node count. Fix
> > - Dmax_lcores and -Dmax_numa_nodes for arm builds.
> >
> > v2:
> > Major rework of the whole series.
> >
> > v3:
> > Added numa and core count defaults for x86 default build.
> > Removed numa and core count defaults. Now requiring defaults to be
> > specified in a cross file or on the cmdline.
> > Added FreeBDS support for numa count discovery.
> >
> > v4:
> > Make automatic numa and cpu counts discovery optional.
> >
> > Juraj Linkeš (6):
> >   build: alias default build as generic
> >   build: refactor Arm build
> >   build: optional NUMA and cpu counts detection
> >   build: add core and NUMA counts to cross files
> >   build: disable Arm drivers
> >   build: update Arm builds with makefile flags
> >
> >  buildtools/get_cpu_count.py          |   7 +
> >  buildtools/get_numa_count.py         |  22 +++
> >  buildtools/meson.build               |   2 +
> >  config/arm/arm64_armada_linux_gcc    |   6 +-
> >  config/arm/arm64_armv8_linux_gcc     |  18 +-
> >  config/arm/arm64_bluefield_linux_gcc |   7 +-
> >  config/arm/arm64_dpaa_linux_gcc      |   5 +-
> >  config/arm/arm64_emag_linux_gcc      |   4 +-
> >  config/arm/arm64_n1sdp_linux_gcc     |   7 +-
> >  config/arm/arm64_octeontx2_linux_gcc |   7 +-
> >  config/arm/arm64_stingray_linux_gcc  |   7 +-
> >  config/arm/arm64_thunderx2_linux_gcc |   6 +-
> >  config/arm/arm64_thunderx_linux_gcc  |   4 +-
> >  config/arm/meson.build               | 278 ++++++++++++++++-----------
> >  config/meson.build                   |  54 +++++-
> >  drivers/meson.build                  |   6 +-
> >  meson.build                          |   1 +
> >  meson_options.txt                    |   8 +-
> >  18 files changed, 306 insertions(+), 143 deletions(-)  create mode
> > 100644 buildtools/get_cpu_count.py  create mode 100644
> > buildtools/get_numa_count.py
> >
> > --
> > 2.20.1

[dpdk-dev] [PATCH v5 00/11] Arm build options rework

[Juraj Linkeš]

The current way of specifying Arm configuration options is insufficient
since we can't identify the SoC we're building for from the MIDR
information. For example, we can't distinguish between N1SDP, Graviton2
or Ampere Altra.

Add a way to specify the cpu count and numa node count for cross builds.

We also want to be able to disable which drivers (and possibly
libraries) are built without user input. This is useful when building:
1. on an SoC that is slow and we want to build only what is necessary
  without the user having to check which libraries they have installed
2. a cross build on a fast aarch64 machine but with target SoC which
  differs in capabilities or libraries.
This is achieved by specifying the drivers in cross files.

Among libraries, only libnuma can be now disabled.

Also add an optional way to discover cpu count a numa node count. Fix
-Dmax_lcores and -Dmax_numa_nodes for arm builds.

The current implementation adds/supports the following:
* x86 -> aarch64 cross build with added config options/disabled
  drivers/libs
* aarch64 -> aarch64 builds for a different SoCs must be done using
  cross-files
* max numa nodes and max lcore may be specified on the command line to
  overwrite the values for any (native or cross) build

v2:
Major rework of the whole series.

v3:
Added numa and core count defaults for x86 default build.
Removed numa and core count defaults. Now requiring defaults to be
specified in a cross file or on the cmdline.
Added FreeBDS support for numa count discovery.

v4:
Make automatic numa and cpu counts discovery optional.

v5:
Split the refactor patch into smaller patches.
Simplify buildtools/get_numa_count.py.
Add more explanation to cover letter.

Juraj Linkeš (11):
  build: alias default build as generic
  build: rename Arm build variables
  build: remove unused or superfluous variables
  build: Arm reformat, comments, move config
  build: simplify how Arm flags are processed
  build: use dict in Arm part number config
  build: Arm generic and native build setup
  build: optional NUMA and cpu counts detection
  build: add core and NUMA counts to cross files
  build: disable Arm drivers
  build: update Arm builds with makefile flags

 buildtools/get_cpu_count.py          |   7 +
 buildtools/get_numa_count.py         |  22 ++
 buildtools/meson.build               |   2 +
 config/arm/arm64_armada_linux_gcc    |   6 +-
 config/arm/arm64_armv8_linux_gcc     |  18 +-
 config/arm/arm64_bluefield_linux_gcc |   7 +-
 config/arm/arm64_dpaa_linux_gcc      |   5 +-
 config/arm/arm64_emag_linux_gcc      |   4 +-
 config/arm/arm64_n1sdp_linux_gcc     |   7 +-
 config/arm/arm64_octeontx2_linux_gcc |   7 +-
 config/arm/arm64_stingray_linux_gcc  |   7 +-
 config/arm/arm64_thunderx2_linux_gcc |   6 +-
 config/arm/arm64_thunderx_linux_gcc  |   4 +-
 config/arm/meson.build               | 302 ++++++++++++++++-----------
 config/meson.build                   |  57 ++++-
 config/x86/meson.build               |   2 +
 drivers/meson.build                  |   6 +-
 meson.build                          |   1 +
 meson_options.txt                    |   8 +-
 19 files changed, 324 insertions(+), 154 deletions(-)
 create mode 100644 buildtools/get_cpu_count.py
 create mode 100644 buildtools/get_numa_count.py

-- 
2.20.1

[dpdk-dev] [PATCH v5 01/11] build: alias default build as generic

[Juraj Linkeš]

The current machine='default' build name is not descriptive. The actual
default build is machine='native'. Add an alternative string which does
the same build and better describes what we're building:
machine='generic'. Leave machine='default' for backwards compatibility.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/meson.build | 5 +++--
 config/meson.build     | 9 +++++----
 2 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/config/arm/meson.build b/config/arm/meson.build
index b49203fa8..4fd32f0ca 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -1,12 +1,13 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation.
 # Copyright(c) 2017 Cavium, Inc
+# Copyright(c) 2020 PANTHEON.tech s.r.o.
 
 # for checking defines we need to use the correct compiler flags
 march_opt = '-march=@0@'.format(machine)
 
 arm_force_native_march = false
-arm_force_default_march = (machine == 'default')
+arm_force_generic_march = (machine == 'generic')
 
 flags_common_default = [
 	# Accelarate rte_memcpy. Be sure to run unit test (memcpy_perf_autotest)
@@ -145,7 +146,7 @@ else
 	cmd_generic = ['generic', '', '', 'default', '']
 	cmd_output = cmd_generic # Set generic by default
 	machine_args = [] # Clear previous machine args
-	if arm_force_default_march and not meson.is_cross_build()
+	if arm_force_generic_march and not meson.is_cross_build()
 		machine = impl_generic
 		impl_pn = 'default'
 	elif not meson.is_cross_build()
diff --git a/config/meson.build b/config/meson.build
index 258b01d06..c7f7aa6e2 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -68,13 +68,14 @@ else
 	machine = get_option('machine')
 endif
 
-# machine type 'default' is special, it defaults to the per arch agreed common
-# minimal baseline needed for DPDK.
+# machine type 'generic' is special, it defaults to the per arch agreed common
+# minimal baseline needed for DPDK. Machine type 'default' is also supported
+# with the same meaning for backwards compatibility.
 # That might not be the most optimized, but the most portable version while
 # still being able to support the CPU features required for DPDK.
 # This can be bumped up by the DPDK project, but it can never be an
 # invariant like 'native'
-if machine == 'default'
+if machine == 'default' or machine == 'generic'
 	if host_machine.cpu_family().startswith('x86')
 		# matches the old pre-meson build systems default
 		machine = 'corei7'
@@ -82,7 +83,7 @@ if machine == 'default'
 		machine = 'armv7-a'
 	elif host_machine.cpu_family().startswith('aarch')
 		# arm64 manages defaults in config/arm/meson.build
-		machine = 'default'
+		machine = 'generic'
 	elif host_machine.cpu_family().startswith('ppc')
 		machine = 'power8'
 	endif
-- 
2.20.1

Re: [dpdk-dev] [PATCH v5 01/11] build: alias default build as generic

[Bruce Richardson]

On Wed, Oct 28, 2020 at 03:03:55PM +0100, Juraj Linkeš wrote:
> The current machine='default' build name is not descriptive. The actual
> default build is machine='native'. Add an alternative string which does
> the same build and better describes what we're building:
> machine='generic'. Leave machine='default' for backwards compatibility.
> 
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---

Looks good generally. Only suggestion is that the comment for the "machine"
value in meson_options.txt needs to be expanded and should describe
"generic" and "native" options.

Did you check for any documentation updates that may be needed for this?

>  config/arm/meson.build | 5 +++--
>  config/meson.build     | 9 +++++----
>  2 files changed, 8 insertions(+), 6 deletions(-)
> 
> diff --git a/config/arm/meson.build b/config/arm/meson.build
> index b49203fa8..4fd32f0ca 100644
> --- a/config/arm/meson.build
> +++ b/config/arm/meson.build
> @@ -1,12 +1,13 @@
>  # SPDX-License-Identifier: BSD-3-Clause
>  # Copyright(c) 2017 Intel Corporation.
>  # Copyright(c) 2017 Cavium, Inc
> +# Copyright(c) 2020 PANTHEON.tech s.r.o.
>  
>  # for checking defines we need to use the correct compiler flags
>  march_opt = '-march=@0@'.format(machine)
>  
>  arm_force_native_march = false
> -arm_force_default_march = (machine == 'default')
> +arm_force_generic_march = (machine == 'generic')
>  
>  flags_common_default = [
>  	# Accelarate rte_memcpy. Be sure to run unit test (memcpy_perf_autotest)
> @@ -145,7 +146,7 @@ else
>  	cmd_generic = ['generic', '', '', 'default', '']
>  	cmd_output = cmd_generic # Set generic by default
>  	machine_args = [] # Clear previous machine args
> -	if arm_force_default_march and not meson.is_cross_build()
> +	if arm_force_generic_march and not meson.is_cross_build()
>  		machine = impl_generic
>  		impl_pn = 'default'
>  	elif not meson.is_cross_build()
> diff --git a/config/meson.build b/config/meson.build
> index 258b01d06..c7f7aa6e2 100644
> --- a/config/meson.build
> +++ b/config/meson.build
> @@ -68,13 +68,14 @@ else
>  	machine = get_option('machine')
>  endif
>  
> -# machine type 'default' is special, it defaults to the per arch agreed common
> -# minimal baseline needed for DPDK.
> +# machine type 'generic' is special, it defaults to the per arch agreed common
> +# minimal baseline needed for DPDK. Machine type 'default' is also supported
> +# with the same meaning for backwards compatibility.
>  # That might not be the most optimized, but the most portable version while
>  # still being able to support the CPU features required for DPDK.
>  # This can be bumped up by the DPDK project, but it can never be an
>  # invariant like 'native'
> -if machine == 'default'
> +if machine == 'default' or machine == 'generic'
>  	if host_machine.cpu_family().startswith('x86')
>  		# matches the old pre-meson build systems default
>  		machine = 'corei7'
> @@ -82,7 +83,7 @@ if machine == 'default'
>  		machine = 'armv7-a'
>  	elif host_machine.cpu_family().startswith('aarch')
>  		# arm64 manages defaults in config/arm/meson.build
> -		machine = 'default'
> +		machine = 'generic'
>  	elif host_machine.cpu_family().startswith('ppc')
>  		machine = 'power8'
>  	endif
> -- 
> 2.20.1
> 

[dpdk-dev] [PATCH v5 02/11] build: rename Arm build variables

[Juraj Linkeš]

Rename Arm build variables and values so that they better conform to Arm
specifications. Also rename generically sounding variable to names that
better capture what the variables hold.

Rename machine_args_generic to part_number_config_default since the
variable contains more than just the generic machine args and is
referenced multiple times with the intended use as a default value.
Rename the default machine args to generic machine args to reflect that.
The rest of the variables are self-explanatory.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/arm64_armada_linux_gcc    |  2 +-
 config/arm/arm64_armv8_linux_gcc     |  8 ++--
 config/arm/arm64_bluefield_linux_gcc |  4 +-
 config/arm/arm64_dpaa_linux_gcc      |  2 +-
 config/arm/arm64_emag_linux_gcc      |  2 +-
 config/arm/arm64_n1sdp_linux_gcc     |  4 +-
 config/arm/arm64_octeontx2_linux_gcc |  4 +-
 config/arm/arm64_stingray_linux_gcc  |  4 +-
 config/arm/arm64_thunderx2_linux_gcc |  4 +-
 config/arm/arm64_thunderx_linux_gcc  |  2 +-
 config/arm/meson.build               | 72 ++++++++++++++--------------
 11 files changed, 54 insertions(+), 54 deletions(-)

diff --git a/config/arm/arm64_armada_linux_gcc b/config/arm/arm64_armada_linux_gcc
index fa40c0398..52c5f4476 100644
--- a/config/arm/arm64_armada_linux_gcc
+++ b/config/arm/arm64_armada_linux_gcc
@@ -14,4 +14,4 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x56'
+implementer_id = '0x56'
diff --git a/config/arm/arm64_armv8_linux_gcc b/config/arm/arm64_armv8_linux_gcc
index 88f0ff9da..13ee8b223 100644
--- a/config/arm/arm64_armv8_linux_gcc
+++ b/config/arm/arm64_armv8_linux_gcc
@@ -13,10 +13,10 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = 'generic'
+implementer_id = 'generic'
 
-# Valid options for Arm's implementor_pn:
-# 'default': valid for all armv8-a architectures (default value)
+# Valid options for Arm's part_number:
+# 'generic': valid for all armv8-a architectures (default value)
 # '0xd03':   cortex-a53
 # '0xd04':   cortex-a35
 # '0xd05':   cortex-a55
@@ -25,4 +25,4 @@ implementor_id = 'generic'
 # '0xd09':   cortex-a73
 # '0xd0a':   cortex-a75
 # '0xd0b':   cortex-a76
-implementor_pn = 'default'
+part_number = 'generic'
diff --git a/config/arm/arm64_bluefield_linux_gcc b/config/arm/arm64_bluefield_linux_gcc
index 86797d23c..b79389d85 100644
--- a/config/arm/arm64_bluefield_linux_gcc
+++ b/config/arm/arm64_bluefield_linux_gcc
@@ -13,5 +13,5 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x41'
-implementor_pn = '0xd08'
+implementer_id = '0x41'
+part_number = '0xd08'
diff --git a/config/arm/arm64_dpaa_linux_gcc b/config/arm/arm64_dpaa_linux_gcc
index 1a4682154..573ae7e42 100644
--- a/config/arm/arm64_dpaa_linux_gcc
+++ b/config/arm/arm64_dpaa_linux_gcc
@@ -14,4 +14,4 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = 'dpaa'
+implementer_id = 'dpaa'
diff --git a/config/arm/arm64_emag_linux_gcc b/config/arm/arm64_emag_linux_gcc
index 8edcd3e97..24f3d533e 100644
--- a/config/arm/arm64_emag_linux_gcc
+++ b/config/arm/arm64_emag_linux_gcc
@@ -13,4 +13,4 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x50'
+implementer_id = '0x50'
diff --git a/config/arm/arm64_n1sdp_linux_gcc b/config/arm/arm64_n1sdp_linux_gcc
index 022e06303..6fb3f02ea 100644
--- a/config/arm/arm64_n1sdp_linux_gcc
+++ b/config/arm/arm64_n1sdp_linux_gcc
@@ -13,5 +13,5 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x41'
-implementor_pn = '0xd0c'
+implementer_id = '0x41'
+part_number = '0xd0c'
diff --git a/config/arm/arm64_octeontx2_linux_gcc b/config/arm/arm64_octeontx2_linux_gcc
index 365bd7cbd..ac1042806 100644
--- a/config/arm/arm64_octeontx2_linux_gcc
+++ b/config/arm/arm64_octeontx2_linux_gcc
@@ -13,5 +13,5 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x43'
-implementor_pn = '0xb2'
+implementer_id = '0x43'
+part_number = '0xb2'
diff --git a/config/arm/arm64_stingray_linux_gcc b/config/arm/arm64_stingray_linux_gcc
index 86797d23c..b79389d85 100644
--- a/config/arm/arm64_stingray_linux_gcc
+++ b/config/arm/arm64_stingray_linux_gcc
@@ -13,5 +13,5 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x41'
-implementor_pn = '0xd08'
+implementer_id = '0x41'
+part_number = '0xd08'
diff --git a/config/arm/arm64_thunderx2_linux_gcc b/config/arm/arm64_thunderx2_linux_gcc
index 2b41acc61..dd257745e 100644
--- a/config/arm/arm64_thunderx2_linux_gcc
+++ b/config/arm/arm64_thunderx2_linux_gcc
@@ -13,5 +13,5 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x43'
-implementor_pn = '0xaf'
+implementer_id = '0x43'
+part_number = '0xaf'
diff --git a/config/arm/arm64_thunderx_linux_gcc b/config/arm/arm64_thunderx_linux_gcc
index 6572ab615..670764437 100644
--- a/config/arm/arm64_thunderx_linux_gcc
+++ b/config/arm/arm64_thunderx_linux_gcc
@@ -13,4 +13,4 @@ cpu = 'armv8-a'
 endian = 'little'
 
 [properties]
-implementor_id = '0x43'
+implementer_id = '0x43'
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 4fd32f0ca..dacbb947b 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -88,8 +88,8 @@ flags_octeontx2_extra = [
 	['RTE_EAL_IGB_UIO', false],
 	['RTE_USE_C11_MEM_MODEL', true]]
 
-machine_args_generic = [
-	['default', ['-march=armv8-a+crc', '-moutline-atomics']],
+part_number_config_default = [
+	['generic', ['-march=armv8-a+crc', '-moutline-atomics']],
 	['native', ['-march=native']],
 	['0xd03', ['-mcpu=cortex-a53']],
 	['0xd04', ['-mcpu=cortex-a35']],
@@ -100,8 +100,8 @@ machine_args_generic = [
 	['0xd0b', ['-mcpu=cortex-a76']],
 	['0xd0c', ['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'], flags_n1sdp_extra]]
 
-machine_args_cavium = [
-	['default', ['-march=armv8-a+crc+crypto','-mcpu=thunderx']],
+part_number_config_cavium = [
+	['generic', ['-march=armv8-a+crc+crypto','-mcpu=thunderx']],
 	['native', ['-march=native']],
 	['0xa1', ['-mcpu=thunderxt88'], flags_thunderx_extra],
 	['0xa2', ['-mcpu=thunderxt81'], flags_thunderx_extra],
@@ -109,25 +109,25 @@ machine_args_cavium = [
 	['0xaf', ['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'], flags_thunderx2_extra],
 	['0xb2', ['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'], flags_octeontx2_extra]]
 
-machine_args_emag = [
-	['default', ['-march=armv8-a+crc+crypto', '-mtune=emag']],
+part_number_config_emag = [
+	['generic', ['-march=armv8-a+crc+crypto', '-mtune=emag']],
 	['native', ['-march=native']]]
 
 ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page G7-5321)
-impl_generic = ['Generic armv8', flags_generic, machine_args_generic]
-impl_0x41 = ['Arm', flags_arm, machine_args_generic]
-impl_0x42 = ['Broadcom', flags_generic, machine_args_generic]
-impl_0x43 = ['Cavium', flags_cavium, machine_args_cavium]
-impl_0x44 = ['DEC', flags_generic, machine_args_generic]
-impl_0x49 = ['Infineon', flags_generic, machine_args_generic]
-impl_0x4d = ['Motorola', flags_generic, machine_args_generic]
-impl_0x4e = ['NVIDIA', flags_generic, machine_args_generic]
-impl_0x50 = ['Ampere Computing', flags_emag, machine_args_emag]
-impl_0x51 = ['Qualcomm', flags_generic, machine_args_generic]
-impl_0x53 = ['Samsung', flags_generic, machine_args_generic]
-impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_generic]
-impl_0x69 = ['Intel', flags_generic, machine_args_generic]
-impl_dpaa = ['NXP DPAA', flags_dpaa, machine_args_generic]
+implementer_generic = ['Generic armv8', flags_generic, part_number_config_default]
+implementer_0x41 = ['Arm', flags_arm, part_number_config_default]
+implementer_0x42 = ['Broadcom', flags_generic, part_number_config_default]
+implementer_0x43 = ['Cavium', flags_cavium, part_number_config_cavium]
+implementer_0x44 = ['DEC', flags_generic, part_number_config_default]
+implementer_0x49 = ['Infineon', flags_generic, part_number_config_default]
+implementer_0x4d = ['Motorola', flags_generic, part_number_config_default]
+implementer_0x4e = ['NVIDIA', flags_generic, part_number_config_default]
+implementer_0x50 = ['Ampere Computing', flags_emag, part_number_config_emag]
+implementer_0x51 = ['Qualcomm', flags_generic, part_number_config_default]
+implementer_0x53 = ['Samsung', flags_generic, part_number_config_default]
+implementer_0x56 = ['Marvell ARMADA', flags_armada, part_number_config_default]
+implementer_0x69 = ['Intel', flags_generic, part_number_config_default]
+implementer_dpaa = ['NXP DPAA', flags_dpaa, part_number_config_default]
 
 dpdk_conf.set('RTE_ARCH_ARM', 1)
 dpdk_conf.set('RTE_FORCE_INTRINSICS', 1)
@@ -142,13 +142,13 @@ else
 	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
 	dpdk_conf.set('RTE_ARCH_ARM64', 1)
 
-	machine = []
-	cmd_generic = ['generic', '', '', 'default', '']
+	implementer_config = []
+	cmd_generic = ['generic', '', '', 'generic', '']
 	cmd_output = cmd_generic # Set generic by default
 	machine_args = [] # Clear previous machine args
 	if arm_force_generic_march and not meson.is_cross_build()
-		machine = impl_generic
-		impl_pn = 'default'
+		implementer_config = implementer_generic
+		part_number = 'generic'
 	elif not meson.is_cross_build()
 		# The script returns ['Implementer', 'Variant', 'Architecture',
 		# 'Primary Part number', 'Revision']
@@ -159,19 +159,19 @@ else
 			cmd_output = cmd.stdout().to_lower().strip().split(' ')
 		endif
 		# Set to generic if variable is not found
-		machine = get_variable('impl_' + cmd_output[0], ['generic'])
-		if machine[0] == 'generic'
-			machine = impl_generic
+		implementer_config = get_variable('implementer_' + cmd_output[0], ['generic'])
+		if implementer_config[0] == 'generic'
+			implementer_config = implementer_generic
 			cmd_output = cmd_generic
 		endif
-		impl_pn = cmd_output[3]
+		part_number = cmd_output[3]
 		if arm_force_native_march == true
-			impl_pn = 'native'
+			part_number = 'native'
 		endif
 	else
-		impl_id = meson.get_cross_property('implementor_id', 'generic')
-		impl_pn = meson.get_cross_property('implementor_pn', 'default')
-		machine = get_variable('impl_' + impl_id)
+		implementer_id = meson.get_cross_property('implementer_id', 'generic')
+		part_number = meson.get_cross_property('part_number', 'generic')
+		implementer_config = get_variable('implementer_' + implementer_id)
 	endif
 
 	# Apply Common Defaults. These settings may be overwritten by machine
@@ -182,15 +182,15 @@ else
 		endif
 	endforeach
 
-	message('Implementer : ' + machine[0])
-	foreach flag: machine[1]
+	message('Implementer : ' + implementer_config[0])
+	foreach flag: implementer_config[1]
 		if flag.length() > 0
 			dpdk_conf.set(flag[0], flag[1])
 		endif
 	endforeach
 
-	foreach marg: machine[2]
-		if marg[0] == impl_pn
+	foreach marg: implementer_config[2]
+		if marg[0] == part_number
 			foreach flag: marg[1]
 				if cc.has_argument(flag)
 					machine_args += flag
-- 
2.20.1

Re: [dpdk-dev] [PATCH v5 02/11] build: rename Arm build variables

[Ruifeng Wang]

> -----Original Message-----
> From: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Sent: Wednesday, October 28, 2020 10:04 PM
> To: bruce.richardson@intel.com; Ruifeng Wang <Ruifeng.Wang@arm.com>;
> Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Phil Yang
> <Phil.Yang@arm.com>; vcchunga@amazon.com; Dharmik Thakkar
> <Dharmik.Thakkar@arm.com>; jerinjacobk@gmail.com;
> hemant.agrawal@nxp.com
> Cc: dev@dpdk.org; Juraj Linkeš <juraj.linkes@pantheon.tech>
> Subject: [PATCH v5 02/11] build: rename Arm build variables
> 
> Rename Arm build variables and values so that they better conform to Arm
> specifications. Also rename generically sounding variable to names that
> better capture what the variables hold.
> 
> Rename machine_args_generic to part_number_config_default since the
> variable contains more than just the generic machine args and is referenced
> multiple times with the intended use as a default value.
> Rename the default machine args to generic machine args to reflect that.
> The rest of the variables are self-explanatory.
> 
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
>  config/arm/arm64_armada_linux_gcc    |  2 +-
>  config/arm/arm64_armv8_linux_gcc     |  8 ++--
>  config/arm/arm64_bluefield_linux_gcc |  4 +-
>  config/arm/arm64_dpaa_linux_gcc      |  2 +-
>  config/arm/arm64_emag_linux_gcc      |  2 +-
>  config/arm/arm64_n1sdp_linux_gcc     |  4 +-
>  config/arm/arm64_octeontx2_linux_gcc |  4 +-
> config/arm/arm64_stingray_linux_gcc  |  4 +-
> config/arm/arm64_thunderx2_linux_gcc |  4 +-
> config/arm/arm64_thunderx_linux_gcc  |  2 +-
>  config/arm/meson.build               | 72 ++++++++++++++--------------
>  11 files changed, 54 insertions(+), 54 deletions(-)
> 
> diff --git a/config/arm/arm64_armada_linux_gcc
> b/config/arm/arm64_armada_linux_gcc
> index fa40c0398..52c5f4476 100644
> --- a/config/arm/arm64_armada_linux_gcc
> +++ b/config/arm/arm64_armada_linux_gcc
> @@ -14,4 +14,4 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = '0x56'
> +implementer_id = '0x56'
> diff --git a/config/arm/arm64_armv8_linux_gcc
> b/config/arm/arm64_armv8_linux_gcc
> index 88f0ff9da..13ee8b223 100644
> --- a/config/arm/arm64_armv8_linux_gcc
> +++ b/config/arm/arm64_armv8_linux_gcc
> @@ -13,10 +13,10 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = 'generic'
> +implementer_id = 'generic'
> 
> -# Valid options for Arm's implementor_pn:
> -# 'default': valid for all armv8-a architectures (default value)
> +# Valid options for Arm's part_number:
> +# 'generic': valid for all armv8-a architectures (default value)
>  # '0xd03':   cortex-a53
>  # '0xd04':   cortex-a35
>  # '0xd05':   cortex-a55
> @@ -25,4 +25,4 @@ implementor_id = 'generic'
>  # '0xd09':   cortex-a73
>  # '0xd0a':   cortex-a75
>  # '0xd0b':   cortex-a76
> -implementor_pn = 'default'
> +part_number = 'generic'
> diff --git a/config/arm/arm64_bluefield_linux_gcc
> b/config/arm/arm64_bluefield_linux_gcc
> index 86797d23c..b79389d85 100644
> --- a/config/arm/arm64_bluefield_linux_gcc
> +++ b/config/arm/arm64_bluefield_linux_gcc
> @@ -13,5 +13,5 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = '0x41'
> -implementor_pn = '0xd08'
> +implementer_id = '0x41'
> +part_number = '0xd08'
> diff --git a/config/arm/arm64_dpaa_linux_gcc
> b/config/arm/arm64_dpaa_linux_gcc index 1a4682154..573ae7e42 100644
> --- a/config/arm/arm64_dpaa_linux_gcc
> +++ b/config/arm/arm64_dpaa_linux_gcc
> @@ -14,4 +14,4 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = 'dpaa'
> +implementer_id = 'dpaa'
> diff --git a/config/arm/arm64_emag_linux_gcc
> b/config/arm/arm64_emag_linux_gcc index 8edcd3e97..24f3d533e 100644
> --- a/config/arm/arm64_emag_linux_gcc
> +++ b/config/arm/arm64_emag_linux_gcc
> @@ -13,4 +13,4 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = '0x50'
> +implementer_id = '0x50'
> diff --git a/config/arm/arm64_n1sdp_linux_gcc
> b/config/arm/arm64_n1sdp_linux_gcc
> index 022e06303..6fb3f02ea 100644
> --- a/config/arm/arm64_n1sdp_linux_gcc
> +++ b/config/arm/arm64_n1sdp_linux_gcc
> @@ -13,5 +13,5 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = '0x41'
> -implementor_pn = '0xd0c'
> +implementer_id = '0x41'
> +part_number = '0xd0c'
> diff --git a/config/arm/arm64_octeontx2_linux_gcc
> b/config/arm/arm64_octeontx2_linux_gcc
> index 365bd7cbd..ac1042806 100644
> --- a/config/arm/arm64_octeontx2_linux_gcc
> +++ b/config/arm/arm64_octeontx2_linux_gcc
> @@ -13,5 +13,5 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = '0x43'
> -implementor_pn = '0xb2'
> +implementer_id = '0x43'
> +part_number = '0xb2'
> diff --git a/config/arm/arm64_stingray_linux_gcc
> b/config/arm/arm64_stingray_linux_gcc
> index 86797d23c..b79389d85 100644
> --- a/config/arm/arm64_stingray_linux_gcc
> +++ b/config/arm/arm64_stingray_linux_gcc
> @@ -13,5 +13,5 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = '0x41'
> -implementor_pn = '0xd08'
> +implementer_id = '0x41'
> +part_number = '0xd08'
> diff --git a/config/arm/arm64_thunderx2_linux_gcc
> b/config/arm/arm64_thunderx2_linux_gcc
> index 2b41acc61..dd257745e 100644
> --- a/config/arm/arm64_thunderx2_linux_gcc
> +++ b/config/arm/arm64_thunderx2_linux_gcc
> @@ -13,5 +13,5 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = '0x43'
> -implementor_pn = '0xaf'
> +implementer_id = '0x43'
> +part_number = '0xaf'
> diff --git a/config/arm/arm64_thunderx_linux_gcc
> b/config/arm/arm64_thunderx_linux_gcc
> index 6572ab615..670764437 100644
> --- a/config/arm/arm64_thunderx_linux_gcc
> +++ b/config/arm/arm64_thunderx_linux_gcc
> @@ -13,4 +13,4 @@ cpu = 'armv8-a'
>  endian = 'little'
> 
>  [properties]
> -implementor_id = '0x43'
> +implementer_id = '0x43'
> diff --git a/config/arm/meson.build b/config/arm/meson.build index
> 4fd32f0ca..dacbb947b 100644
> --- a/config/arm/meson.build
> +++ b/config/arm/meson.build
> @@ -88,8 +88,8 @@ flags_octeontx2_extra = [
>  	['RTE_EAL_IGB_UIO', false],
>  	['RTE_USE_C11_MEM_MODEL', true]]
> 
> -machine_args_generic = [
> -	['default', ['-march=armv8-a+crc', '-moutline-atomics']],
> +part_number_config_default = [
> +	['generic', ['-march=armv8-a+crc', '-moutline-atomics']],
>  	['native', ['-march=native']],
>  	['0xd03', ['-mcpu=cortex-a53']],
>  	['0xd04', ['-mcpu=cortex-a35']],
> @@ -100,8 +100,8 @@ machine_args_generic = [
>  	['0xd0b', ['-mcpu=cortex-a76']],
>  	['0xd0c', ['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'],
> flags_n1sdp_extra]]
> 
> -machine_args_cavium = [
> -	['default', ['-march=armv8-a+crc+crypto','-mcpu=thunderx']],
> +part_number_config_cavium = [
> +	['generic', ['-march=armv8-a+crc+crypto','-mcpu=thunderx']],
>  	['native', ['-march=native']],
>  	['0xa1', ['-mcpu=thunderxt88'], flags_thunderx_extra],
>  	['0xa2', ['-mcpu=thunderxt81'], flags_thunderx_extra], @@ -109,25
> +109,25 @@ machine_args_cavium = [
>  	['0xaf', ['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'],
> flags_thunderx2_extra],
>  	['0xb2', ['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'],
> flags_octeontx2_extra]]
> 
> -machine_args_emag = [
> -	['default', ['-march=armv8-a+crc+crypto', '-mtune=emag']],
> +part_number_config_emag = [
> +	['generic', ['-march=armv8-a+crc+crypto', '-mtune=emag']],
>  	['native', ['-march=native']]]
> 
>  ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page G7-5321)
> -impl_generic = ['Generic armv8', flags_generic, machine_args_generic]
> -impl_0x41 = ['Arm', flags_arm, machine_args_generic]
> -impl_0x42 = ['Broadcom', flags_generic, machine_args_generic]
> -impl_0x43 = ['Cavium', flags_cavium, machine_args_cavium]
> -impl_0x44 = ['DEC', flags_generic, machine_args_generic]
> -impl_0x49 = ['Infineon', flags_generic, machine_args_generic] -impl_0x4d =
> ['Motorola', flags_generic, machine_args_generic] -impl_0x4e = ['NVIDIA',
> flags_generic, machine_args_generic]
> -impl_0x50 = ['Ampere Computing', flags_emag, machine_args_emag]
> -impl_0x51 = ['Qualcomm', flags_generic, machine_args_generic]
> -impl_0x53 = ['Samsung', flags_generic, machine_args_generic]
> -impl_0x56 = ['Marvell ARMADA', flags_armada, machine_args_generic]
> -impl_0x69 = ['Intel', flags_generic, machine_args_generic] -impl_dpaa =
> ['NXP DPAA', flags_dpaa, machine_args_generic]
> +implementer_generic = ['Generic armv8', flags_generic,
> +part_number_config_default]
> +implementer_0x41 = ['Arm', flags_arm, part_number_config_default]
> +implementer_0x42 = ['Broadcom', flags_generic,
> +part_number_config_default]
> +implementer_0x43 = ['Cavium', flags_cavium, part_number_config_cavium]
> +implementer_0x44 = ['DEC', flags_generic, part_number_config_default]
> +implementer_0x49 = ['Infineon', flags_generic,
> +part_number_config_default] implementer_0x4d = ['Motorola',
> +flags_generic, part_number_config_default] implementer_0x4e =
> +['NVIDIA', flags_generic, part_number_config_default]
> +implementer_0x50 = ['Ampere Computing', flags_emag,
> +part_number_config_emag]
> +implementer_0x51 = ['Qualcomm', flags_generic,
> +part_number_config_default]
> +implementer_0x53 = ['Samsung', flags_generic,
> +part_number_config_default]
> +implementer_0x56 = ['Marvell ARMADA', flags_armada,
> +part_number_config_default]
> +implementer_0x69 = ['Intel', flags_generic, part_number_config_default]
> +implementer_dpaa = ['NXP DPAA', flags_dpaa,
> part_number_config_default]
> 
>  dpdk_conf.set('RTE_ARCH_ARM', 1)
>  dpdk_conf.set('RTE_FORCE_INTRINSICS', 1) @@ -142,13 +142,13 @@ else
>  	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
>  	dpdk_conf.set('RTE_ARCH_ARM64', 1)
> 
> -	machine = []
> -	cmd_generic = ['generic', '', '', 'default', '']
> +	implementer_config = []
> +	cmd_generic = ['generic', '', '', 'generic', '']
>  	cmd_output = cmd_generic # Set generic by default
>  	machine_args = [] # Clear previous machine args
>  	if arm_force_generic_march and not meson.is_cross_build()
> -		machine = impl_generic
> -		impl_pn = 'default'
> +		implementer_config = implementer_generic
> +		part_number = 'generic'
>  	elif not meson.is_cross_build()
>  		# The script returns ['Implementer', 'Variant', 'Architecture',
>  		# 'Primary Part number', 'Revision']
> @@ -159,19 +159,19 @@ else
>  			cmd_output = cmd.stdout().to_lower().strip().split(' ')
>  		endif
>  		# Set to generic if variable is not found
> -		machine = get_variable('impl_' + cmd_output[0], ['generic'])
> -		if machine[0] == 'generic'
> -			machine = impl_generic
> +		implementer_config = get_variable('implementer_' +
> cmd_output[0], ['generic'])
> +		if implementer_config[0] == 'generic'
> +			implementer_config = implementer_generic
>  			cmd_output = cmd_generic
>  		endif
> -		impl_pn = cmd_output[3]
> +		part_number = cmd_output[3]
>  		if arm_force_native_march == true
> -			impl_pn = 'native'
> +			part_number = 'native'
>  		endif
>  	else
> -		impl_id = meson.get_cross_property('implementor_id',
> 'generic')
> -		impl_pn = meson.get_cross_property('implementor_pn',
> 'default')
> -		machine = get_variable('impl_' + impl_id)
> +		implementer_id =
> meson.get_cross_property('implementer_id', 'generic')
> +		part_number = meson.get_cross_property('part_number',
> 'generic')
> +		implementer_config = get_variable('implementer_' +
> implementer_id)
>  	endif
> 
>  	# Apply Common Defaults. These settings may be overwritten by
> machine @@ -182,15 +182,15 @@ else
>  		endif
>  	endforeach
> 
> -	message('Implementer : ' + machine[0])
> -	foreach flag: machine[1]
> +	message('Implementer : ' + implementer_config[0])
> +	foreach flag: implementer_config[1]
>  		if flag.length() > 0
>  			dpdk_conf.set(flag[0], flag[1])
>  		endif
>  	endforeach
> 
> -	foreach marg: machine[2]
> -		if marg[0] == impl_pn
> +	foreach marg: implementer_config[2]
> +		if marg[0] == part_number
>  			foreach flag: marg[1]
>  				if cc.has_argument(flag)
>  					machine_args += flag
> --
> 2.20.1

Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>

[dpdk-dev] [PATCH v5 03/11] build: remove unused or superfluous variables

[Juraj Linkeš]

Remove variables that were either not used, referenced just once or not
needed.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/meson.build | 20 +++++++-------------
 1 file changed, 7 insertions(+), 13 deletions(-)

diff --git a/config/arm/meson.build b/config/arm/meson.build
index dacbb947b..bd2acfc21 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -3,11 +3,7 @@
 # Copyright(c) 2017 Cavium, Inc
 # Copyright(c) 2020 PANTHEON.tech s.r.o.
 
-# for checking defines we need to use the correct compiler flags
-march_opt = '-march=@0@'.format(machine)
-
 arm_force_native_march = false
-arm_force_generic_march = (machine == 'generic')
 
 flags_common_default = [
 	# Accelarate rte_memcpy. Be sure to run unit test (memcpy_perf_autotest)
@@ -63,7 +59,6 @@ flags_armada = [
 	['RTE_MAX_NUMA_NODES', 1],
 	['RTE_MAX_LCORE', 16]]
 
-flags_default_extra = []
 flags_n1sdp_extra = [
 	['RTE_MACHINE', '"n1sdp"'],
 	['RTE_MAX_NUMA_NODES', 1],
@@ -142,11 +137,9 @@ else
 	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
 	dpdk_conf.set('RTE_ARCH_ARM64', 1)
 
-	implementer_config = []
-	cmd_generic = ['generic', '', '', 'generic', '']
-	cmd_output = cmd_generic # Set generic by default
+	implementer_id = 'generic'
 	machine_args = [] # Clear previous machine args
-	if arm_force_generic_march and not meson.is_cross_build()
+	if machine == 'generic' and not meson.is_cross_build()
 		implementer_config = implementer_generic
 		part_number = 'generic'
 	elif not meson.is_cross_build()
@@ -157,14 +150,15 @@ else
 		cmd = run_command(detect_vendor.path())
 		if cmd.returncode() == 0
 			cmd_output = cmd.stdout().to_lower().strip().split(' ')
+			implementer_id = cmd_output[0]
+			part_number = cmd_output[3]
 		endif
 		# Set to generic if variable is not found
-		implementer_config = get_variable('implementer_' + cmd_output[0], ['generic'])
+		implementer_config = get_variable('implementer_' + implementer_id, ['generic'])
 		if implementer_config[0] == 'generic'
 			implementer_config = implementer_generic
-			cmd_output = cmd_generic
+			part_number = 'generic'
 		endif
-		part_number = cmd_output[3]
 		if arm_force_native_march == true
 			part_number = 'native'
 		endif
@@ -197,7 +191,7 @@ else
 				endif
 			endforeach
 			# Apply any extra machine specific flags.
-			foreach flag: marg.get(2, flags_default_extra)
+			foreach flag: marg.get(2, [])
 				if flag.length() > 0
 					dpdk_conf.set(flag[0], flag[1])
 				endif
-- 
2.20.1

Re: [dpdk-dev] [PATCH v5 03/11] build: remove unused or superfluous variables

[Ruifeng Wang]

> -----Original Message-----
> From: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Sent: Wednesday, October 28, 2020 10:04 PM
> To: bruce.richardson@intel.com; Ruifeng Wang <Ruifeng.Wang@arm.com>;
> Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Phil Yang
> <Phil.Yang@arm.com>; vcchunga@amazon.com; Dharmik Thakkar
> <Dharmik.Thakkar@arm.com>; jerinjacobk@gmail.com;
> hemant.agrawal@nxp.com
> Cc: dev@dpdk.org; Juraj Linkeš <juraj.linkes@pantheon.tech>
> Subject: [PATCH v5 03/11] build: remove unused or superfluous variables
> 
> Remove variables that were either not used, referenced just once or not
> needed.
> 
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
>  config/arm/meson.build | 20 +++++++-------------
>  1 file changed, 7 insertions(+), 13 deletions(-)
> 
> diff --git a/config/arm/meson.build b/config/arm/meson.build index
> dacbb947b..bd2acfc21 100644
> --- a/config/arm/meson.build
> +++ b/config/arm/meson.build
> @@ -3,11 +3,7 @@
>  # Copyright(c) 2017 Cavium, Inc
>  # Copyright(c) 2020 PANTHEON.tech s.r.o.
> 
> -# for checking defines we need to use the correct compiler flags -march_opt
> = '-march=@0@'.format(machine)
> -
>  arm_force_native_march = false
> -arm_force_generic_march = (machine == 'generic')
> 
>  flags_common_default = [
>  	# Accelarate rte_memcpy. Be sure to run unit test
> (memcpy_perf_autotest) @@ -63,7 +59,6 @@ flags_armada = [
>  	['RTE_MAX_NUMA_NODES', 1],
>  	['RTE_MAX_LCORE', 16]]
> 
> -flags_default_extra = []
>  flags_n1sdp_extra = [
>  	['RTE_MACHINE', '"n1sdp"'],
>  	['RTE_MAX_NUMA_NODES', 1],
> @@ -142,11 +137,9 @@ else
>  	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
>  	dpdk_conf.set('RTE_ARCH_ARM64', 1)
> 
> -	implementer_config = []
> -	cmd_generic = ['generic', '', '', 'generic', '']
> -	cmd_output = cmd_generic # Set generic by default
> +	implementer_id = 'generic'
>  	machine_args = [] # Clear previous machine args
> -	if arm_force_generic_march and not meson.is_cross_build()
> +	if machine == 'generic' and not meson.is_cross_build()
>  		implementer_config = implementer_generic
>  		part_number = 'generic'
>  	elif not meson.is_cross_build()
> @@ -157,14 +150,15 @@ else
>  		cmd = run_command(detect_vendor.path())
>  		if cmd.returncode() == 0
>  			cmd_output = cmd.stdout().to_lower().strip().split(' ')
> +			implementer_id = cmd_output[0]
> +			part_number = cmd_output[3]
>  		endif
>  		# Set to generic if variable is not found
> -		implementer_config = get_variable('implementer_' +
> cmd_output[0], ['generic'])
> +		implementer_config = get_variable('implementer_' +
> implementer_id,
> +['generic'])
>  		if implementer_config[0] == 'generic'
>  			implementer_config = implementer_generic
> -			cmd_output = cmd_generic
> +			part_number = 'generic'
>  		endif
> -		part_number = cmd_output[3]
>  		if arm_force_native_march == true
>  			part_number = 'native'
>  		endif
> @@ -197,7 +191,7 @@ else
>  				endif
>  			endforeach
>  			# Apply any extra machine specific flags.
> -			foreach flag: marg.get(2, flags_default_extra)
> +			foreach flag: marg.get(2, [])
>  				if flag.length() > 0
>  					dpdk_conf.set(flag[0], flag[1])
>  				endif
> --
> 2.20.1
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>

[dpdk-dev] [PATCH v5 04/11] build: Arm reformat, comments, move config

[Juraj Linkeš]

Change formatting so that it's more consistent and readable, add/modify
comments/stdout messages, move configuration options to more appropriate
places and make the order consistent according to these rules:
1. First list generic configuration options, then list options that may
   be overwritten. List SoC-specific options last.
2. For SoC-specific options, list number of cores before the number of
   NUMA nodes, to make it consistent with config/meson.build.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/meson.build | 84 +++++++++++++++++++++++++-----------------
 1 file changed, 51 insertions(+), 33 deletions(-)

diff --git a/config/arm/meson.build b/config/arm/meson.build
index bd2acfc21..c970c8803 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -5,6 +5,7 @@
 
 arm_force_native_march = false
 
+# common flags to all aarch64 builds, with lowest priority
 flags_common_default = [
 	# Accelarate rte_memcpy. Be sure to run unit test (memcpy_perf_autotest)
 	# to determine the best threshold in code. Refer to notes in source file
@@ -12,8 +13,8 @@ flags_common_default = [
 	['RTE_ARCH_ARM64_MEMCPY', false],
 	#	['RTE_ARM64_MEMCPY_ALIGNED_THRESHOLD', 2048],
 	#	['RTE_ARM64_MEMCPY_UNALIGNED_THRESHOLD', 512],
-	# Leave below RTE_ARM64_MEMCPY_xxx options commented out, unless there're
-	# strong reasons.
+	# Leave below RTE_ARM64_MEMCPY_xxx options commented out,
+	# unless there are strong reasons.
 	#	['RTE_ARM64_MEMCPY_SKIP_GCC_VER_CHECK', false],
 	#	['RTE_ARM64_MEMCPY_ALIGN_MASK', 0xF],
 	#	['RTE_ARM64_MEMCPY_STRICT_ALIGN', false],
@@ -24,65 +25,82 @@ flags_common_default = [
 
 	['RTE_SCHED_VECTOR', false],
 	['RTE_ARM_USE_WFE', false],
+	['RTE_ARCH_ARM64', true],
+	['RTE_CACHE_LINE_SIZE', 128]
 ]
 
+# implementer specific generic aarch64 flags, with middle priority
+# (will overwrite common flags)
 flags_generic = [
 	['RTE_MACHINE', '"armv8a"'],
-	['RTE_MAX_LCORE', 256],
 	['RTE_USE_C11_MEM_MODEL', true],
-	['RTE_CACHE_LINE_SIZE', 128]]
+	['RTE_CACHE_LINE_SIZE', 128],
+	['RTE_MAX_LCORE', 256]
+]
 flags_arm = [
 	['RTE_MACHINE', '"armv8a"'],
-	['RTE_MAX_LCORE', 16],
 	['RTE_USE_C11_MEM_MODEL', true],
-	['RTE_CACHE_LINE_SIZE', 64]]
+	['RTE_CACHE_LINE_SIZE', 64],
+	['RTE_MAX_LCORE', 16]
+]
 flags_cavium = [
+	['RTE_MAX_VFIO_GROUPS', 128],
 	['RTE_CACHE_LINE_SIZE', 128],
-	['RTE_MAX_NUMA_NODES', 2],
 	['RTE_MAX_LCORE', 96],
-	['RTE_MAX_VFIO_GROUPS', 128]]
+	['RTE_MAX_NUMA_NODES', 2]
+]
 flags_dpaa = [
 	['RTE_MACHINE', '"dpaa"'],
+	['RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', false],
 	['RTE_USE_C11_MEM_MODEL', true],
 	['RTE_CACHE_LINE_SIZE', 64],
-	['RTE_MAX_NUMA_NODES', 1],
 	['RTE_MAX_LCORE', 16],
-	['RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', false]]
+	['RTE_MAX_NUMA_NODES', 1]
+]
 flags_emag = [
 	['RTE_MACHINE', '"emag"'],
 	['RTE_CACHE_LINE_SIZE', 64],
-	['RTE_MAX_NUMA_NODES', 1],
-	['RTE_MAX_LCORE', 32]]
+	['RTE_MAX_LCORE', 32],
+	['RTE_MAX_NUMA_NODES', 1]
+]
 flags_armada = [
 	['RTE_MACHINE', '"armv8a"'],
 	['RTE_CACHE_LINE_SIZE', 64],
-	['RTE_MAX_NUMA_NODES', 1],
-	['RTE_MAX_LCORE', 16]]
+	['RTE_MAX_LCORE', 16],
+	['RTE_MAX_NUMA_NODES', 1]
+]
 
+# part number specific aarch64 flags, with highest priority
+# (will overwrite both common and implementer specific flags)
 flags_n1sdp_extra = [
 	['RTE_MACHINE', '"n1sdp"'],
-	['RTE_MAX_NUMA_NODES', 1],
-	['RTE_MAX_LCORE', 4],
 	['RTE_EAL_NUMA_AWARE_HUGEPAGES', false],
-	['RTE_LIBRTE_VHOST_NUMA', false]]
+	['RTE_LIBRTE_VHOST_NUMA', false],
+	['RTE_MAX_LCORE', 4],
+	['RTE_MAX_NUMA_NODES', 1]
+]
 flags_thunderx_extra = [
 	['RTE_MACHINE', '"thunderx"'],
-	['RTE_USE_C11_MEM_MODEL', false]]
+	['RTE_USE_C11_MEM_MODEL', false]
+]
 flags_thunderx2_extra = [
 	['RTE_MACHINE', '"thunderx2"'],
+	['RTE_ARM_FEATURE_ATOMICS', true],
+	['RTE_USE_C11_MEM_MODEL', true],
 	['RTE_CACHE_LINE_SIZE', 64],
-	['RTE_MAX_NUMA_NODES', 2],
 	['RTE_MAX_LCORE', 256],
-	['RTE_ARM_FEATURE_ATOMICS', true],
-	['RTE_USE_C11_MEM_MODEL', true]]
+	['RTE_MAX_NUMA_NODES', 2]
+]
 flags_octeontx2_extra = [
 	['RTE_MACHINE', '"octeontx2"'],
-	['RTE_MAX_NUMA_NODES', 1],
-	['RTE_MAX_LCORE', 36],
 	['RTE_ARM_FEATURE_ATOMICS', true],
+	['RTE_USE_C11_MEM_MODEL', true],
 	['RTE_EAL_IGB_UIO', false],
-	['RTE_USE_C11_MEM_MODEL', true]]
+	['RTE_MAX_LCORE', 36],
+	['RTE_MAX_NUMA_NODES', 1]
+]
 
+# arm config (implementer 0x41) is the default config
 part_number_config_default = [
 	['generic', ['-march=armv8-a+crc', '-moutline-atomics']],
 	['native', ['-march=native']],
@@ -93,8 +111,8 @@ part_number_config_default = [
 	['0xd09', ['-mcpu=cortex-a73']],
 	['0xd0a', ['-mcpu=cortex-a75']],
 	['0xd0b', ['-mcpu=cortex-a76']],
-	['0xd0c', ['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'], flags_n1sdp_extra]]
-
+	['0xd0c', ['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'], flags_n1sdp_extra]
+]
 part_number_config_cavium = [
 	['generic', ['-march=armv8-a+crc+crypto','-mcpu=thunderx']],
 	['native', ['-march=native']],
@@ -102,11 +120,12 @@ part_number_config_cavium = [
 	['0xa2', ['-mcpu=thunderxt81'], flags_thunderx_extra],
 	['0xa3', ['-mcpu=thunderxt83'], flags_thunderx_extra],
 	['0xaf', ['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'], flags_thunderx2_extra],
-	['0xb2', ['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'], flags_octeontx2_extra]]
-
+	['0xb2', ['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'], flags_octeontx2_extra]
+]
 part_number_config_emag = [
 	['generic', ['-march=armv8-a+crc+crypto', '-mtune=emag']],
-	['native', ['-march=native']]]
+	['native', ['-march=native']]
+]
 
 ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page G7-5321)
 implementer_generic = ['Generic armv8', flags_generic, part_number_config_default]
@@ -128,15 +147,14 @@ dpdk_conf.set('RTE_ARCH_ARM', 1)
 dpdk_conf.set('RTE_FORCE_INTRINSICS', 1)
 
 if dpdk_conf.get('RTE_ARCH_32')
+	# armv7 build
 	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
 	dpdk_conf.set('RTE_ARCH_ARMv7', 1)
 	# the minimum architecture supported, armv7-a, needs the following,
 	# mk/machine/armv7a/rte.vars.mk sets it too
 	machine_args += '-mfpu=neon'
 else
-	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
-	dpdk_conf.set('RTE_ARCH_ARM64', 1)
-
+	# aarch64 build
 	implementer_id = 'generic'
 	machine_args = [] # Clear previous machine args
 	if machine == 'generic' and not meson.is_cross_build()
@@ -199,7 +217,7 @@ else
 		endif
 	endforeach
 endif
-message(machine_args)
+message('Using machine args: @0@'.format(machine_args))
 
 if (cc.get_define('__ARM_NEON', args: machine_args) != '' or
     cc.get_define('__aarch64__', args: machine_args) != '')
-- 
2.20.1

Re: [dpdk-dev] [PATCH v5 04/11] build: Arm reformat, comments, move config

[Ruifeng Wang]

> -----Original Message-----
> From: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Sent: Wednesday, October 28, 2020 10:04 PM
> To: bruce.richardson@intel.com; Ruifeng Wang <Ruifeng.Wang@arm.com>;
> Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Phil Yang
> <Phil.Yang@arm.com>; vcchunga@amazon.com; Dharmik Thakkar
> <Dharmik.Thakkar@arm.com>; jerinjacobk@gmail.com;
> hemant.agrawal@nxp.com
> Cc: dev@dpdk.org; Juraj Linkeš <juraj.linkes@pantheon.tech>
> Subject: [PATCH v5 04/11] build: Arm reformat, comments, move config
> 
> Change formatting so that it's more consistent and readable, add/modify
> comments/stdout messages, move configuration options to more
> appropriate places and make the order consistent according to these rules:
> 1. First list generic configuration options, then list options that may
>    be overwritten. List SoC-specific options last.
> 2. For SoC-specific options, list number of cores before the number of
>    NUMA nodes, to make it consistent with config/meson.build.
> 
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
>  config/arm/meson.build | 84 +++++++++++++++++++++++++-----------------
>  1 file changed, 51 insertions(+), 33 deletions(-)
> 
> diff --git a/config/arm/meson.build b/config/arm/meson.build index
> bd2acfc21..c970c8803 100644
> --- a/config/arm/meson.build
> +++ b/config/arm/meson.build
> @@ -5,6 +5,7 @@
> 


<snip>

>  ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page G7-5321)
> implementer_generic = ['Generic armv8', flags_generic,
> part_number_config_default] @@ -128,15 +147,14 @@
> dpdk_conf.set('RTE_ARCH_ARM', 1)
> dpdk_conf.set('RTE_FORCE_INTRINSICS', 1)
> 
>  if dpdk_conf.get('RTE_ARCH_32')
> +	# armv7 build
>  	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
>  	dpdk_conf.set('RTE_ARCH_ARMv7', 1)
>  	# the minimum architecture supported, armv7-a, needs the following,
>  	# mk/machine/armv7a/rte.vars.mk sets it too

The reference to makefile can be removed.

>  	machine_args += '-mfpu=neon'
>  else
> -	dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
> -	dpdk_conf.set('RTE_ARCH_ARM64', 1)
> -
> +	# aarch64 build
>  	implementer_id = 'generic'
>  	machine_args = [] # Clear previous machine args
>  	if machine == 'generic' and not meson.is_cross_build() @@ -199,7
> +217,7 @@ else
>  		endif
>  	endforeach
>  endif
> -message(machine_args)
> +message('Using machine args: @0@'.format(machine_args))
> 
>  if (cc.get_define('__ARM_NEON', args: machine_args) != '' or
>      cc.get_define('__aarch64__', args: machine_args) != '')
> --
> 2.20.1

[dpdk-dev] [PATCH v5 05/11] build: simplify how Arm flags are processed

[Juraj Linkeš]

Set flags in one loop. Append flags to a list and use the list in the
loop.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/meson.build | 36 +++++++++++++++++-------------------
 1 file changed, 17 insertions(+), 19 deletions(-)

diff --git a/config/arm/meson.build b/config/arm/meson.build
index c970c8803..133a0d7fd 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -186,34 +186,32 @@ else
 		implementer_config = get_variable('implementer_' + implementer_id)
 	endif
 
-	# Apply Common Defaults. These settings may be overwritten by machine
-	# settings later.
-	foreach flag: flags_common_default
-		if flag.length() > 0
-			dpdk_conf.set(flag[0], flag[1])
-		endif
-	endforeach
+	message('Arm implementer: ' + implementer_config[0])
+	message('Arm part number: ' + part_number)
 
-	message('Implementer : ' + implementer_config[0])
-	foreach flag: implementer_config[1]
-		if flag.length() > 0
-			dpdk_conf.set(flag[0], flag[1])
-		endif
-	endforeach
+	# use default flags with implementer flags
+	dpdk_flags = flags_common_default + implementer_config[1]
 
+	machine_args = [] # Clear previous machine args
 	foreach marg: implementer_config[2]
 		if marg[0] == part_number
+			# apply supported machine args
 			foreach flag: marg[1]
 				if cc.has_argument(flag)
 					machine_args += flag
 				endif
 			endforeach
-			# Apply any extra machine specific flags.
-			foreach flag: marg.get(2, [])
-				if flag.length() > 0
-					dpdk_conf.set(flag[0], flag[1])
-				endif
-			endforeach
+			if marg.length() > 2
+				# add extra flags for the part
+				dpdk_flags += marg[2]
+			endif
+		endif
+	endforeach
+
+	# apply flags
+	foreach flag: dpdk_flags
+		if flag.length() > 0
+			dpdk_conf.set(flag[0], flag[1])
 		endif
 	endforeach
 endif
-- 
2.20.1

[dpdk-dev] [PATCH v5 06/11] build: use dict in Arm part number config

[Juraj Linkeš]

Use dictionary lookup instead of iterating over all elements in the
list. Fallback to generic part number if the discovered part number is
unknown.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/meson.build | 85 ++++++++++++++++++++++++------------------
 1 file changed, 48 insertions(+), 37 deletions(-)

diff --git a/config/arm/meson.build b/config/arm/meson.build
index 133a0d7fd..2f89cb2a8 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -101,31 +101,31 @@ flags_octeontx2_extra = [
 ]
 
 # arm config (implementer 0x41) is the default config
-part_number_config_default = [
-	['generic', ['-march=armv8-a+crc', '-moutline-atomics']],
-	['native', ['-march=native']],
-	['0xd03', ['-mcpu=cortex-a53']],
-	['0xd04', ['-mcpu=cortex-a35']],
-	['0xd07', ['-mcpu=cortex-a57']],
-	['0xd08', ['-mcpu=cortex-a72']],
-	['0xd09', ['-mcpu=cortex-a73']],
-	['0xd0a', ['-mcpu=cortex-a75']],
-	['0xd0b', ['-mcpu=cortex-a76']],
-	['0xd0c', ['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'], flags_n1sdp_extra]
-]
-part_number_config_cavium = [
-	['generic', ['-march=armv8-a+crc+crypto','-mcpu=thunderx']],
-	['native', ['-march=native']],
-	['0xa1', ['-mcpu=thunderxt88'], flags_thunderx_extra],
-	['0xa2', ['-mcpu=thunderxt81'], flags_thunderx_extra],
-	['0xa3', ['-mcpu=thunderxt83'], flags_thunderx_extra],
-	['0xaf', ['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'], flags_thunderx2_extra],
-	['0xb2', ['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'], flags_octeontx2_extra]
-]
-part_number_config_emag = [
-	['generic', ['-march=armv8-a+crc+crypto', '-mtune=emag']],
-	['native', ['-march=native']]
-]
+part_number_config_default = {
+	'generic': [['-march=armv8-a+crc', '-moutline-atomics']],
+	'native': [['-march=native']],
+	'0xd03': [['-mcpu=cortex-a53']],
+	'0xd04': [['-mcpu=cortex-a35']],
+	'0xd07': [['-mcpu=cortex-a57']],
+	'0xd08': [['-mcpu=cortex-a72']],
+	'0xd09': [['-mcpu=cortex-a73']],
+	'0xd0a': [['-mcpu=cortex-a75']],
+	'0xd0b': [['-mcpu=cortex-a76']],
+	'0xd0c': [['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'], flags_n1sdp_extra]
+}
+part_number_config_cavium = {
+	'generic': [['-march=armv8-a+crc+crypto', '-mcpu=thunderx']],
+	'native': [['-march=native']],
+	'0xa1': [['-mcpu=thunderxt88'], flags_thunderx_extra],
+	'0xa2': [['-mcpu=thunderxt81'], flags_thunderx_extra],
+	'0xa3': [['-mcpu=thunderxt83'], flags_thunderx_extra],
+	'0xaf': [['-march=armv8.1-a+crc+crypto','-mcpu=thunderx2t99'], flags_thunderx2_extra],
+	'0xb2': [['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'], flags_octeontx2_extra],
+}
+part_number_config_emag = {
+	'generic': [['-march=armv8-a+crc+crypto', '-mtune=emag']],
+	'native': [['-march=native']]
+}
 
 ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page G7-5321)
 implementer_generic = ['Generic armv8', flags_generic, part_number_config_default]
@@ -189,22 +189,33 @@ else
 	message('Arm implementer: ' + implementer_config[0])
 	message('Arm part number: ' + part_number)
 
+	part_number_config = implementer_config[2]
+	if part_number_config.has_key(part_number)
+		# use the specified part_number machine args if found
+		part_number_config = part_number_config[part_number]
+	elif not meson.is_cross_build()
+		# default to generic machine args if part_number is not found
+		# and not forcing native machine args
+		# but don't default in cross-builds; if part_number is specified
+		# incorrectly in a cross-file, it needs to be fixed there
+		part_number_config = part_number_config['generic']
+	else
+		# doing cross build and part number is not in part_number_config
+		error('Cross build part number 0@0 not found.'.format(part_number))
+	endif
+
 	# use default flags with implementer flags
 	dpdk_flags = flags_common_default + implementer_config[1]
+	if part_number_config.length() > 1
+		# add extra flags from the part number
+		dpdk_flags += part_number_config[1]
+	endif
 
+	# apply supported machine args
 	machine_args = [] # Clear previous machine args
-	foreach marg: implementer_config[2]
-		if marg[0] == part_number
-			# apply supported machine args
-			foreach flag: marg[1]
-				if cc.has_argument(flag)
-					machine_args += flag
-				endif
-			endforeach
-			if marg.length() > 2
-				# add extra flags for the part
-				dpdk_flags += marg[2]
-			endif
+	foreach flag: part_number_config[0]
+		if cc.has_argument(flag)
+			machine_args += flag
 		endif
 	endforeach
 
-- 
2.20.1

[dpdk-dev] [PATCH v5 07/11] build: Arm generic and native build setup

[Juraj Linkeš]

Simplify the code. Define native machine args in one place since they're
the same regardless of which Arm environment we're building on.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/meson.build | 56 +++++++++++++++++++++++-------------------
 1 file changed, 31 insertions(+), 25 deletions(-)

diff --git a/config/arm/meson.build b/config/arm/meson.build
index 2f89cb2a8..1866ad961 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -3,7 +3,10 @@
 # Copyright(c) 2017 Cavium, Inc
 # Copyright(c) 2020 PANTHEON.tech s.r.o.
 
+# set arm_force_native_march if you want to use machine args below
+# instead of discovered values in native builds
 arm_force_native_march = false
+native_machine_args = ['-march=native', '-mtune=native']
 
 # common flags to all aarch64 builds, with lowest priority
 flags_common_default = [
@@ -103,7 +106,6 @@ flags_octeontx2_extra = [
 # arm config (implementer 0x41) is the default config
 part_number_config_default = {
 	'generic': [['-march=armv8-a+crc', '-moutline-atomics']],
-	'native': [['-march=native']],
 	'0xd03': [['-mcpu=cortex-a53']],
 	'0xd04': [['-mcpu=cortex-a35']],
 	'0xd07': [['-mcpu=cortex-a57']],
@@ -115,7 +117,6 @@ part_number_config_default = {
 }
 part_number_config_cavium = {
 	'generic': [['-march=armv8-a+crc+crypto', '-mcpu=thunderx']],
-	'native': [['-march=native']],
 	'0xa1': [['-mcpu=thunderxt88'], flags_thunderx_extra],
 	'0xa2': [['-mcpu=thunderxt81'], flags_thunderx_extra],
 	'0xa3': [['-mcpu=thunderxt83'], flags_thunderx_extra],
@@ -123,8 +124,7 @@ part_number_config_cavium = {
 	'0xb2': [['-march=armv8.2-a+crc+crypto+lse','-mcpu=octeontx2'], flags_octeontx2_extra],
 }
 part_number_config_emag = {
-	'generic': [['-march=armv8-a+crc+crypto', '-mtune=emag']],
-	'native': [['-march=native']]
+	'generic': [['-march=armv8-a+crc+crypto', '-mtune=emag']]
 }
 
 ## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page G7-5321)
@@ -155,32 +155,35 @@ if dpdk_conf.get('RTE_ARCH_32')
 	machine_args += '-mfpu=neon'
 else
 	# aarch64 build
-	implementer_id = 'generic'
 	machine_args = [] # Clear previous machine args
-	if machine == 'generic' and not meson.is_cross_build()
-		implementer_config = implementer_generic
-		part_number = 'generic'
-	elif not meson.is_cross_build()
-		# The script returns ['Implementer', 'Variant', 'Architecture',
-		# 'Primary Part number', 'Revision']
-		detect_vendor = find_program(join_paths(
-				meson.current_source_dir(), 'armv8_machine.py'))
-		cmd = run_command(detect_vendor.path())
-		if cmd.returncode() == 0
-			cmd_output = cmd.stdout().to_lower().strip().split(' ')
-			implementer_id = cmd_output[0]
-			part_number = cmd_output[3]
-		endif
-		# Set to generic if variable is not found
-		implementer_config = get_variable('implementer_' + implementer_id, ['generic'])
-		if implementer_config[0] == 'generic'
+	if not meson.is_cross_build()
+		if machine == 'generic'
+			# generic native build
 			implementer_config = implementer_generic
 			part_number = 'generic'
-		endif
-		if arm_force_native_march == true
-			part_number = 'native'
+		else
+			# native build
+			# The script returns ['Implementer', 'Variant', 'Architecture',
+			# 'Primary Part number', 'Revision']
+			detect_vendor = find_program(join_paths(
+					meson.current_source_dir(), 'armv8_machine.py'))
+			cmd = run_command(detect_vendor.path())
+			if cmd.returncode() == 0
+				cmd_output = cmd.stdout().to_lower().strip().split(' ')
+				implementer_id = cmd_output[0]
+				part_number = cmd_output[3]
+			else
+				implementer_id = 'generic'
+				part_number = 'generic'
+			endif
+			# Set to generic if implementer is not found
+			implementer_config = get_variable('implementer_' + implementer_id, implementer_generic)
+			if arm_force_native_march == true
+				part_number = 'native'
+			endif
 		endif
 	else
+		# cross build
 		implementer_id = meson.get_cross_property('implementer_id', 'generic')
 		part_number = meson.get_cross_property('part_number', 'generic')
 		implementer_config = get_variable('implementer_' + implementer_id)
@@ -193,6 +196,9 @@ else
 	if part_number_config.has_key(part_number)
 		# use the specified part_number machine args if found
 		part_number_config = part_number_config[part_number]
+	elif part_number == 'native'
+		# use native machine args
+		part_number_config = [[native_machine_args]]
 	elif not meson.is_cross_build()
 		# default to generic machine args if part_number is not found
 		# and not forcing native machine args
-- 
2.20.1

[dpdk-dev] [PATCH v5 08/11] build: optional NUMA and cpu counts detection

[Juraj Linkeš]

Add an option to automatically discover the host's numa and cpu counts
and use those values for a non cross-build.
Give users the option to override the per-arch default values or values
from cross files by specifying them on the command line with -Dmax_lcores
and -Dmax_numa_nodes.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 buildtools/get_cpu_count.py  |  7 ++++++
 buildtools/get_numa_count.py | 22 +++++++++++++++++
 buildtools/meson.build       |  2 ++
 config/meson.build           | 48 ++++++++++++++++++++++++++++++++++--
 config/x86/meson.build       |  2 ++
 meson_options.txt            |  8 +++---
 6 files changed, 83 insertions(+), 6 deletions(-)
 create mode 100644 buildtools/get_cpu_count.py
 create mode 100644 buildtools/get_numa_count.py

diff --git a/buildtools/get_cpu_count.py b/buildtools/get_cpu_count.py
new file mode 100644
index 000000000..386f85f8b
--- /dev/null
+++ b/buildtools/get_cpu_count.py
@@ -0,0 +1,7 @@
+#!/usr/bin/python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2020 PANTHEON.tech s.r.o.
+
+import os
+
+print(os.cpu_count())
diff --git a/buildtools/get_numa_count.py b/buildtools/get_numa_count.py
new file mode 100644
index 000000000..e7f3de6b0
--- /dev/null
+++ b/buildtools/get_numa_count.py
@@ -0,0 +1,22 @@
+#!/usr/bin/python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2020 PANTHEON.tech s.r.o.
+
+import ctypes
+import glob
+import os
+import subprocess
+
+if os.name == 'posix':
+    if os.path.isdir('/sys/devices/system/node'):
+        print(len(glob.glob('/sys/devices/system/node/node*')))
+    else:
+        subprocess.run(['sysctl', 'vm.ndomains'])
+
+elif os.name == 'nt':
+    libkernel32 = ctypes.windll.kernel32
+
+    count = ctypes.c_ulong()
+
+    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
+    print(count.value + 1)
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 04808dabc..925e733b1 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -17,3 +17,5 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+get_cpu_count_cmd = py3 + files('get_cpu_count.py')
+get_numa_count_cmd = py3 + files('get_numa_count.py')
diff --git a/config/meson.build b/config/meson.build
index c7f7aa6e2..a51000ab2 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -231,8 +231,6 @@ foreach arg: warning_flags
 endforeach
 
 # set other values pulled from the build options
-dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
-dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
 dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
 dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
 dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
@@ -251,6 +249,52 @@ compile_time_cpuflags = []
 subdir(arch_subdir)
 dpdk_conf.set('RTE_COMPILE_TIME_CPUFLAGS', ','.join(compile_time_cpuflags))
 
+max_lcores = get_option('max_lcores')
+max_numa_nodes = get_option('max_numa_nodes')
+if max_lcores > 0
+	# Overwrite the default value from arch_subdir with user input
+	dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
+elif max_lcores == -1
+	# Overwrite the default value with discovered values
+	if not meson.is_cross_build()
+		# Discovery makes sense only for non-cross builds
+		max_lcores = run_command(get_cpu_count_cmd).stdout().to_int()
+		min_lcores = 2
+		# DPDK must be build for at least 2 cores
+		if max_lcores < min_lcores
+			message('Found less than @0@ cores, building for @0@ cores'.format(min_lcores))
+			max_lcores = min_lcores
+		else
+			message('Found @0@ cores'.format(max_lcores))
+		endif
+		dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
+	endif
+endif
+
+if max_numa_nodes > 0
+	# Overwrite the default value from arch_subdir with user input
+	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
+elif max_numa_nodes == -1
+	# Overwrite the default value with discovered values
+	if not meson.is_cross_build()
+		# Discovery makes sense only for non-cross builds
+		max_numa_nodes = run_command(get_numa_count_cmd).stdout().to_int()
+		message('Found @0@ numa nodes'.format(max_numa_nodes))
+		dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
+	endif
+endif
+
+# check that cpu and numa count is set in cross builds
+# and error out if it's not set
+if meson.is_cross_build()
+	if not dpdk_conf.has('RTE_MAX_LCORE')
+		error('Number of cores for cross build not specified in @0@ subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
+	endif
+	if not dpdk_conf.has('RTE_MAX_NUMA_NODES')
+		error('Number of numa nodes for cross build not specified in @0@ subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
+	endif
+endif
+
 # set the install path for the drivers
 dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
 
diff --git a/config/x86/meson.build b/config/x86/meson.build
index 31bfa63b1..4989d47f3 100644
--- a/config/x86/meson.build
+++ b/config/x86/meson.build
@@ -57,3 +57,5 @@ else
 endif
 
 dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
+dpdk_conf.set('RTE_MAX_LCORE', 128)
+dpdk_conf.set('RTE_MAX_NUMA_NODES', 4)
diff --git a/meson_options.txt b/meson_options.txt
index 9bf18ab6b..a78c21b0d 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -26,10 +26,10 @@ option('machine', type: 'string', value: 'native',
 	description: 'set the target machine type')
 option('max_ethports', type: 'integer', value: 32,
 	description: 'maximum number of Ethernet devices')
-option('max_lcores', type: 'integer', value: 128,
-	description: 'maximum number of cores/threads supported by EAL')
-option('max_numa_nodes', type: 'integer', value: 4,
-	description: 'maximum number of NUMA nodes supported by EAL')
+option('max_lcores', type: 'integer', value: 0,
+	description: 'maximum number of cores/threads supported by EAL. Set to positive integer to overwrite per-arch or cross-compilation defaults. Set to -1 to use number of cores on the build machine.')
+option('max_numa_nodes', type: 'integer', value: 0,
+	description: 'maximum number of NUMA nodes supported by EAL. Set to positive integer to overwrite per-arch or cross-compilation defaults. Set to -1 to use number of numa nodes on the build machine.')
 option('enable_trace_fp', type: 'boolean', value: false,
 	description: 'enable fast path trace points.')
 option('tests', type: 'boolean', value: true,
-- 
2.20.1

Re: [dpdk-dev] [PATCH v5 08/11] build: optional NUMA and cpu counts detection

[Bruce Richardson]

On Wed, Oct 28, 2020 at 03:04:02PM +0100, Juraj Linkeš wrote:
> Add an option to automatically discover the host's numa and cpu counts
> and use those values for a non cross-build.
> Give users the option to override the per-arch default values or values
> from cross files by specifying them on the command line with -Dmax_lcores
> and -Dmax_numa_nodes.
> 
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
>  buildtools/get_cpu_count.py  |  7 ++++++
>  buildtools/get_numa_count.py | 22 +++++++++++++++++
>  buildtools/meson.build       |  2 ++
>  config/meson.build           | 48 ++++++++++++++++++++++++++++++++++--
>  config/x86/meson.build       |  2 ++
>  meson_options.txt            |  8 +++---
>  6 files changed, 83 insertions(+), 6 deletions(-)
>  create mode 100644 buildtools/get_cpu_count.py
>  create mode 100644 buildtools/get_numa_count.py
> 
> diff --git a/buildtools/get_cpu_count.py b/buildtools/get_cpu_count.py
> new file mode 100644
> index 000000000..386f85f8b
> --- /dev/null
> +++ b/buildtools/get_cpu_count.py
> @@ -0,0 +1,7 @@
> +#!/usr/bin/python3

I think for DPDK python scripts we standardized on "/usr/bin/env python3"
[Someone please correct me if I'm wrong here!]

> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2020 PANTHEON.tech s.r.o.
> +
> +import os
> +
> +print(os.cpu_count())
> diff --git a/buildtools/get_numa_count.py b/buildtools/get_numa_count.py
> new file mode 100644
> index 000000000..e7f3de6b0
> --- /dev/null
> +++ b/buildtools/get_numa_count.py
> @@ -0,0 +1,22 @@
> +#!/usr/bin/python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2020 PANTHEON.tech s.r.o.
> +
> +import ctypes
> +import glob
> +import os
> +import subprocess
> +
> +if os.name == 'posix':
> +    if os.path.isdir('/sys/devices/system/node'):
> +        print(len(glob.glob('/sys/devices/system/node/node*')))
> +    else:
> +        subprocess.run(['sysctl', 'vm.ndomains'])
> +

I just realised that by default sysctl outputs the name of the element as
well as the value, which is not what we want. Using "-n" flag suppresses
the name, so please add that to the command. [Sorry for not spotting this
in previous versions].

> +elif os.name == 'nt':
> +    libkernel32 = ctypes.windll.kernel32
> +
> +    count = ctypes.c_ulong()
> +
> +    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
> +    print(count.value + 1)
> diff --git a/buildtools/meson.build b/buildtools/meson.build
> index 04808dabc..925e733b1 100644
> --- a/buildtools/meson.build
> +++ b/buildtools/meson.build
> @@ -17,3 +17,5 @@ else
>  endif
>  map_to_win_cmd = py3 + files('map_to_win.py')
>  sphinx_wrapper = py3 + files('call-sphinx-build.py')
> +get_cpu_count_cmd = py3 + files('get_cpu_count.py')
> +get_numa_count_cmd = py3 + files('get_numa_count.py')
> diff --git a/config/meson.build b/config/meson.build
> index c7f7aa6e2..a51000ab2 100644
> --- a/config/meson.build
> +++ b/config/meson.build
> @@ -231,8 +231,6 @@ foreach arg: warning_flags
>  endforeach
>  
>  # set other values pulled from the build options
> -dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
> -dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
>  dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
>  dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
>  dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
> @@ -251,6 +249,52 @@ compile_time_cpuflags = []
>  subdir(arch_subdir)
>  dpdk_conf.set('RTE_COMPILE_TIME_CPUFLAGS', ','.join(compile_time_cpuflags))
>  
> +max_lcores = get_option('max_lcores')
> +max_numa_nodes = get_option('max_numa_nodes')
> +if max_lcores > 0
> +	# Overwrite the default value from arch_subdir with user input
> +	dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> +elif max_lcores == -1
> +	# Overwrite the default value with discovered values
> +	if not meson.is_cross_build()

I'd suggest changing this to the inverse and erroring out, since I assume
that having -1 for cross-compilation should be an error?

	if meson.is_cross_build()
		error('....')
	endif

Thereafter you save a level of indentation for the rest of the block.

> +		# Discovery makes sense only for non-cross builds
> +		max_lcores = run_command(get_cpu_count_cmd).stdout().to_int()
> +		min_lcores = 2
> +		# DPDK must be build for at least 2 cores

Even 2 seems rather low. I'd personally look to set the minimum at 4.

> +		if max_lcores < min_lcores
> +			message('Found less than @0@ cores, building for @0@ cores'.format(min_lcores))
> +			max_lcores = min_lcores
> +		else
> +			message('Found @0@ cores'.format(max_lcores))
> +		endif
> +		dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> +	endif
> +endif
> +
> +if max_numa_nodes > 0
> +	# Overwrite the default value from arch_subdir with user input
> +	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
> +elif max_numa_nodes == -1
> +	# Overwrite the default value with discovered values
> +	if not meson.is_cross_build()
> +		# Discovery makes sense only for non-cross builds
> +		max_numa_nodes = run_command(get_numa_count_cmd).stdout().to_int()
> +		message('Found @0@ numa nodes'.format(max_numa_nodes))
> +		dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
> +	endif
> +endif
> +
> +# check that cpu and numa count is set in cross builds
> +# and error out if it's not set
> +if meson.is_cross_build()
> +	if not dpdk_conf.has('RTE_MAX_LCORE')
> +		error('Number of cores for cross build not specified in @0@ subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
> +	endif
> +	if not dpdk_conf.has('RTE_MAX_NUMA_NODES')
> +		error('Number of numa nodes for cross build not specified in @0@ subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
> +	endif
> +endif
> +

Not sure you need the is_cross_build bit, since if we don't have
RTE_MAX_LCORES set at all, then it's an error too. It just so happens that
the code paths to this point prevent that from ever happening.

Is the intention here that the subdir meson.build files will just literally
take the value from the cross-file, or are you doing other processing in
the ARM case, e.g. to compute it based on other info. If it's just the
former where it's always read from the cross-files, I'd have the reading
done in this file (above the other assignment work), since it would be
architecture agnostic.

>  # set the install path for the drivers
>  dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
>  
<snip>

Re: [dpdk-dev] [PATCH v5 08/11] build: optional NUMA and cpu counts detection

[Juraj Linkeš]

> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Wednesday, October 28, 2020 4:04 PM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Cc: Ruifeng.Wang@arm.com; Honnappa.Nagarahalli@arm.com;
> Phil.Yang@arm.com; vcchunga@amazon.com; Dharmik.Thakkar@arm.com;
> jerinjacobk@gmail.com; hemant.agrawal@nxp.com; dev@dpdk.org
> Subject: Re: [PATCH v5 08/11] build: optional NUMA and cpu counts detection
> 
> On Wed, Oct 28, 2020 at 03:04:02PM +0100, Juraj Linkeš wrote:
> > Add an option to automatically discover the host's numa and cpu counts
> > and use those values for a non cross-build.
> > Give users the option to override the per-arch default values or
> > values from cross files by specifying them on the command line with
> > -Dmax_lcores and -Dmax_numa_nodes.
> >
> > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > ---
> >  buildtools/get_cpu_count.py  |  7 ++++++
> > buildtools/get_numa_count.py | 22 +++++++++++++++++
> >  buildtools/meson.build       |  2 ++
> >  config/meson.build           | 48 ++++++++++++++++++++++++++++++++++--
> >  config/x86/meson.build       |  2 ++
> >  meson_options.txt            |  8 +++---
> >  6 files changed, 83 insertions(+), 6 deletions(-)  create mode 100644
> > buildtools/get_cpu_count.py  create mode 100644
> > buildtools/get_numa_count.py
> >
> > diff --git a/buildtools/get_cpu_count.py b/buildtools/get_cpu_count.py
> > new file mode 100644 index 000000000..386f85f8b
> > --- /dev/null
> > +++ b/buildtools/get_cpu_count.py
> > @@ -0,0 +1,7 @@
> > +#!/usr/bin/python3
> 
> I think for DPDK python scripts we standardized on "/usr/bin/env python3"
> [Someone please correct me if I'm wrong here!]
> 

The other scripts are using it, so I'll change it.

> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > +PANTHEON.tech s.r.o.
> > +
> > +import os
> > +
> > +print(os.cpu_count())
> > diff --git a/buildtools/get_numa_count.py
> > b/buildtools/get_numa_count.py new file mode 100644 index
> > 000000000..e7f3de6b0
> > --- /dev/null
> > +++ b/buildtools/get_numa_count.py
> > @@ -0,0 +1,22 @@
> > +#!/usr/bin/python3
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2020
> > +PANTHEON.tech s.r.o.
> > +
> > +import ctypes
> > +import glob
> > +import os
> > +import subprocess
> > +
> > +if os.name == 'posix':
> > +    if os.path.isdir('/sys/devices/system/node'):
> > +        print(len(glob.glob('/sys/devices/system/node/node*')))
> > +    else:
> > +        subprocess.run(['sysctl', 'vm.ndomains'])
> > +
> 
> I just realised that by default sysctl outputs the name of the element as well as
> the value, which is not what we want. Using "-n" flag suppresses the name, so
> please add that to the command. [Sorry for not spotting this in previous
> versions].
> 

Ok, I'll add this. I verified how the output looks and we shouldn't need to change anything after this.

> > +elif os.name == 'nt':
> > +    libkernel32 = ctypes.windll.kernel32
> > +
> > +    count = ctypes.c_ulong()
> > +
> > +    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
> > +    print(count.value + 1)
> > diff --git a/buildtools/meson.build b/buildtools/meson.build index
> > 04808dabc..925e733b1 100644
> > --- a/buildtools/meson.build
> > +++ b/buildtools/meson.build
> > @@ -17,3 +17,5 @@ else
> >  endif
> >  map_to_win_cmd = py3 + files('map_to_win.py')  sphinx_wrapper = py3 +
> > files('call-sphinx-build.py')
> > +get_cpu_count_cmd = py3 + files('get_cpu_count.py')
> > +get_numa_count_cmd = py3 + files('get_numa_count.py')
> > diff --git a/config/meson.build b/config/meson.build index
> > c7f7aa6e2..a51000ab2 100644
> > --- a/config/meson.build
> > +++ b/config/meson.build
> > @@ -231,8 +231,6 @@ foreach arg: warning_flags  endforeach
> >
> >  # set other values pulled from the build options
> > -dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
> > -dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
> > dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
> > dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
> > dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp')) @@
> > -251,6 +249,52 @@ compile_time_cpuflags = []
> >  subdir(arch_subdir)
> >  dpdk_conf.set('RTE_COMPILE_TIME_CPUFLAGS',
> > ','.join(compile_time_cpuflags))
> >
> > +max_lcores = get_option('max_lcores') max_numa_nodes =
> > +get_option('max_numa_nodes') if max_lcores > 0
> > +	# Overwrite the default value from arch_subdir with user input
> > +	dpdk_conf.set('RTE_MAX_LCORE', max_lcores) elif max_lcores == -1
> > +	# Overwrite the default value with discovered values
> > +	if not meson.is_cross_build()
> 
> I'd suggest changing this to the inverse and erroring out, since I assume that
> having -1 for cross-compilation should be an error?
> 
> 	if meson.is_cross_build()
> 		error('....')
> 	endif
> 
> Thereafter you save a level of indentation for the rest of the block.
> 

I thought of this in terms of cross-compilation taking precedence over discovered values, but erroring out makes sense.

> > +		# Discovery makes sense only for non-cross builds
> > +		max_lcores =
> run_command(get_cpu_count_cmd).stdout().to_int()
> > +		min_lcores = 2
> > +		# DPDK must be build for at least 2 cores
> 
> Even 2 seems rather low. I'd personally look to set the minimum at 4.
> 

This was informed by Honnappa's series: http://patches.dpdk.org/project/dpdk/list/?series=13041.
I believed he fixed the build so it works even with 2 cores. Setting it to lowest working number seems like the way to go to me.

> > +		if max_lcores < min_lcores
> > +			message('Found less than @0@ cores, building for
> @0@ cores'.format(min_lcores))
> > +			max_lcores = min_lcores
> > +		else
> > +			message('Found @0@ cores'.format(max_lcores))
> > +		endif
> > +		dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
> > +	endif
> > +endif
> > +
> > +if max_numa_nodes > 0
> > +	# Overwrite the default value from arch_subdir with user input
> > +	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes) elif
> > +max_numa_nodes == -1
> > +	# Overwrite the default value with discovered values
> > +	if not meson.is_cross_build()
> > +		# Discovery makes sense only for non-cross builds
> > +		max_numa_nodes =
> run_command(get_numa_count_cmd).stdout().to_int()
> > +		message('Found @0@ numa nodes'.format(max_numa_nodes))
> > +		dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
> > +	endif
> > +endif
> > +
> > +# check that cpu and numa count is set in cross builds # and error
> > +out if it's not set if meson.is_cross_build()
> > +	if not dpdk_conf.has('RTE_MAX_LCORE')
> > +		error('Number of cores for cross build not specified in @0@
> subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
> > +	endif
> > +	if not dpdk_conf.has('RTE_MAX_NUMA_NODES')
> > +		error('Number of numa nodes for cross build not specified in
> @0@ subdir (e.g. in a cross-file) nor on the cmdline'.format(arch_subdir))
> > +	endif
> > +endif
> > +
> 
> Not sure you need the is_cross_build bit, since if we don't have
> RTE_MAX_LCORES set at all, then it's an error too. It just so happens that the
> code paths to this point prevent that from ever happening.
> 

Ok, makes sense to do a generic check here.

> Is the intention here that the subdir meson.build files will just literally take the
> value from the cross-file, or are you doing other processing in the ARM case,
> e.g. to compute it based on other info. If it's just the former where it's always
> read from the cross-files, I'd have the reading done in this file (above the other
> assignment work), since it would be architecture agnostic.
> 

Good point, we're not doing anything special, so all configuration from cross files should be processed this way.

> >  # set the install path for the drivers
> > dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
> >
> <snip>

[dpdk-dev] [PATCH v5 09/11] build: add core and NUMA counts to cross files

[Juraj Linkeš]

Add support for setting core count and numa nodes in cross files. The
values specified in cross files will override the default values.
Also add missing default values.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 config/arm/arm64_armada_linux_gcc    |  2 ++
 config/arm/arm64_armv8_linux_gcc     |  6 ++++++
 config/arm/arm64_bluefield_linux_gcc |  2 ++
 config/arm/arm64_dpaa_linux_gcc      |  2 ++
 config/arm/arm64_emag_linux_gcc      |  2 ++
 config/arm/arm64_n1sdp_linux_gcc     |  2 ++
 config/arm/arm64_octeontx2_linux_gcc |  2 ++
 config/arm/arm64_stingray_linux_gcc  |  2 ++
 config/arm/arm64_thunderx2_linux_gcc |  2 ++
 config/arm/arm64_thunderx_linux_gcc  |  2 ++
 config/arm/meson.build               | 24 ++++++++++++++++++++++--
 11 files changed, 46 insertions(+), 2 deletions(-)

diff --git a/config/arm/arm64_armada_linux_gcc b/config/arm/arm64_armada_linux_gcc
index 52c5f4476..36100c594 100644
--- a/config/arm/arm64_armada_linux_gcc
+++ b/config/arm/arm64_armada_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 
 [properties]
 implementer_id = '0x56'
+max_numa_nodes = 1
+max_lcores = 16
diff --git a/config/arm/arm64_armv8_linux_gcc b/config/arm/arm64_armv8_linux_gcc
index 13ee8b223..779333199 100644
--- a/config/arm/arm64_armv8_linux_gcc
+++ b/config/arm/arm64_armv8_linux_gcc
@@ -26,3 +26,9 @@ implementer_id = 'generic'
 # '0xd0a':   cortex-a75
 # '0xd0b':   cortex-a76
 part_number = 'generic'
+
+# Supported extra configuration
+# max_numa_nodes = n  # will set RTE_MAX_NUMA_NODES
+# max_lcores = n      # will set RTE_MAX_LCORE
+max_lcores = 256
+max_numa_nodes = 4
diff --git a/config/arm/arm64_bluefield_linux_gcc b/config/arm/arm64_bluefield_linux_gcc
index b79389d85..7ff6fd309 100644
--- a/config/arm/arm64_bluefield_linux_gcc
+++ b/config/arm/arm64_bluefield_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 [properties]
 implementer_id = '0x41'
 part_number = '0xd08'
+max_numa_nodes = 1
+max_lcores = 16
diff --git a/config/arm/arm64_dpaa_linux_gcc b/config/arm/arm64_dpaa_linux_gcc
index 573ae7e42..adae86f57 100644
--- a/config/arm/arm64_dpaa_linux_gcc
+++ b/config/arm/arm64_dpaa_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 
 [properties]
 implementer_id = 'dpaa'
+max_numa_nodes = 1
+max_lcores = 16
diff --git a/config/arm/arm64_emag_linux_gcc b/config/arm/arm64_emag_linux_gcc
index 24f3d533e..3cccca901 100644
--- a/config/arm/arm64_emag_linux_gcc
+++ b/config/arm/arm64_emag_linux_gcc
@@ -14,3 +14,5 @@ endian = 'little'
 
 [properties]
 implementer_id = '0x50'
+max_numa_nodes = 1
+max_lcores = 32
diff --git a/config/arm/arm64_n1sdp_linux_gcc b/config/arm/arm64_n1sdp_linux_gcc
index 6fb3f02ea..3a8b46812 100644
--- a/config/arm/arm64_n1sdp_linux_gcc
+++ b/config/arm/arm64_n1sdp_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 [properties]
 implementer_id = '0x41'
 part_number = '0xd0c'
+max_numa_nodes = 1
+max_lcores = 4
diff --git a/config/arm/arm64_octeontx2_linux_gcc b/config/arm/arm64_octeontx2_linux_gcc
index ac1042806..2baf95a2a 100644
--- a/config/arm/arm64_octeontx2_linux_gcc
+++ b/config/arm/arm64_octeontx2_linux_gcc
@@ -15,3 +15,5 @@ endian = 'little'
 [properties]
 implementer_id = '0x43'
 part_number = '0xb2'
+max_numa_nodes = 1
+max_lcores = 36