[dpdk-dev] [PATCH 0/2] Make meson configurable.

[dpdk-dev] [PATCH 0/2] Make meson configurable.

[Ilya Maximets]

See the full description in patch #2.

Please, let me know if I'm missing some obvious way to disable
libs and drivers in meson.

Ilya Maximets (2):
  meson: don't check dependencies for tests if not required
  meson: make build configurable

 app/meson.build              |  5 +++++
 app/test/meson.build         | 38 ++++++++++++++++++------------------
 drivers/baseband/meson.build |  5 +++++
 drivers/bus/meson.build      |  6 ++++++
 drivers/common/meson.build   |  6 ++++++
 drivers/compress/meson.build |  5 +++++
 drivers/crypto/meson.build   |  5 +++++
 drivers/event/meson.build    |  6 ++++++
 drivers/mempool/meson.build  |  6 ++++++
 drivers/net/meson.build      |  6 ++++++
 drivers/raw/meson.build      |  6 ++++++
 lib/meson.build              |  5 +++++
 meson_options.txt            | 22 +++++++++++++++++++++
 13 files changed, 102 insertions(+), 19 deletions(-)

-- 
2.17.1

[dpdk-dev] [PATCH 1/2] meson: don't check dependencies for tests if not required

[Ilya Maximets]

Don't need to check dependencies if test apps will not be built anyway.

Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
---
 app/test/meson.build | 38 +++++++++++++++++++-------------------
 1 file changed, 19 insertions(+), 19 deletions(-)

diff --git a/app/test/meson.build b/app/test/meson.build
index 83391cef0..7a529b644 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -313,29 +313,29 @@ endif
 # specify -D_GNU_SOURCE unconditionally
 cflags += '-D_GNU_SOURCE'
 
-test_dep_objs = []
-if dpdk_conf.has('RTE_LIBRTE_COMPRESSDEV')
-	compress_test_dep = dependency('zlib', required: false)
-	if compress_test_dep.found()
-		test_dep_objs += compress_test_dep
-		test_sources += 'test_compressdev.c'
-		test_deps += 'compressdev'
-		fast_non_parallel_test_names += 'compressdev_autotest'
+if get_option('tests')
+	test_dep_objs = []
+	if dpdk_conf.has('RTE_LIBRTE_COMPRESSDEV')
+		compress_test_dep = dependency('zlib', required: false)
+		if compress_test_dep.found()
+			test_dep_objs += compress_test_dep
+			test_sources += 'test_compressdev.c'
+			test_deps += 'compressdev'
+			fast_non_parallel_test_names += 'compressdev_autotest'
+		endif
 	endif
-endif
 
-foreach d:test_deps
-	def_lib = get_option('default_library')
-	test_dep_objs += get_variable(def_lib + '_rte_' + d)
-endforeach
-test_dep_objs += cc.find_library('execinfo', required: false)
+	foreach d:test_deps
+		def_lib = get_option('default_library')
+		test_dep_objs += get_variable(def_lib + '_rte_' + d)
+	endforeach
+	test_dep_objs += cc.find_library('execinfo', required: false)
 
-link_libs = []
-if get_option('default_library') == 'static'
-	link_libs = dpdk_drivers
-endif
+	link_libs = []
+	if get_option('default_library') == 'static'
+		link_libs = dpdk_drivers
+	endif
 
-if get_option('tests')
 	dpdk_test = executable('dpdk-test',
 		test_sources,
 		link_whole: link_libs,
-- 
2.17.1

[dpdk-dev] [PATCH 2/2] meson: make build configurable

[Ilya Maximets]

The first thing many developers do before start building DPDK is
disabling all the not needed divers and libraries. This happens
just because more than a half of DPDK dirvers and libraries are not
needed for the particular reason. For example, you don't need
dpaa*, octeon*, various croypto devices, eventdev, etc. if you're
only want to build OVS for x86_64 with static linking.

By disabling everything you don't need, build speeds up literally 10x
times. This is important for CI systems. For example, TravisCI wastes
10 minutes for the default DPDK build just to check linking with OVS.

Another thing is the binary size. Number of DPDK libraries and,
as a result, size of resulted statically linked application decreases
significantly.

Important thing also that you're able to not install some dependencies
if you don't have them on a target platform. Just disable libs/drivers
that depends on it. Similar thing for the glibc version mismatch
between build and target platforms.

Also, I have to note that less code means less probability of
failures and less number of attack vectors.

This patch gives 'meson' the power of configurability that we
have with 'make'. Using new options it's possible to enable just
what you need and nothing more.

For example, following cmdline could be used to build almost minimal
set of DPDK libs and drivers to check OVS build:

  $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false \
                -Ddrivers_bus=pci,vdev          \
                -Ddrivers_mempool=ring          \
                -Ddrivers_net=null,virtio,ring  \
                -Ddrivers_crypto=virtio         \
                -Ddrivers_compress=none         \
                -Ddrivers_event=none            \
                -Ddrivers_baseband=none         \
                -Ddrivers_raw=none              \
                -Ddrivers_common=none           \
                -Dlibs=kvargs,eal,cmdline,ring,mempool,mbuf,net,meter,\
                       ethdev,pci,hash,cryptodev,pdump,vhost \
                -Dapps=none

Adding a few real net drivers will give configuration that can be used
in production environment.

Looks not very pretty, but this could be moved to a script.

Build details:

  Build targets in project: 57

  $ time ninja
  real    0m11,528s
  user    1m4,137s
  sys     0m4,935s

  $ du -sh ../dpdk_meson_install/
  3,5M    ../dpdk_meson_install/

To compare with what we have without these options:

  $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false
  Build targets in project: 434

  $ time ninja
  real    1m38,963s
  user    10m18,624s
  sys     0m45,478s

  $ du -sh ../dpdk_meson_install/
  27M     ../dpdk_meson_install/

10x speed up for the user time.
7.7 times size decrease.

This is probably not much user-friendly because it's not a Kconfig
and dependency tracking in meson is really poor, so it requires
usually few iterations to pick correct set of libraries to satisfy
all dependencies. However, it's not a big deal. Options intended
for a proficient users who knows what they need.

Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
---
 app/meson.build              |  5 +++++
 drivers/baseband/meson.build |  5 +++++
 drivers/bus/meson.build      |  6 ++++++
 drivers/common/meson.build   |  6 ++++++
 drivers/compress/meson.build |  5 +++++
 drivers/crypto/meson.build   |  5 +++++
 drivers/event/meson.build    |  6 ++++++
 drivers/mempool/meson.build  |  6 ++++++
 drivers/net/meson.build      |  6 ++++++
 drivers/raw/meson.build      |  6 ++++++
 lib/meson.build              |  5 +++++
 meson_options.txt            | 22 ++++++++++++++++++++++
 12 files changed, 83 insertions(+)

diff --git a/app/meson.build b/app/meson.build
index 2b9fdef74..48972954c 100644
--- a/app/meson.build
+++ b/app/meson.build
@@ -17,6 +17,11 @@ apps = [
 	'test-pipeline',
 	'test-pmd']
 
+enabled_apps = get_option('apps')
+if enabled_apps != 'all'
+	apps = (enabled_apps == 'none') ? [] : enabled_apps.split(',')
+endif
+
 # for BSD only
 lib_execinfo = cc.find_library('execinfo', required: false)
 
diff --git a/drivers/baseband/meson.build b/drivers/baseband/meson.build
index 52489df35..fabc80fc2 100644
--- a/drivers/baseband/meson.build
+++ b/drivers/baseband/meson.build
@@ -3,5 +3,10 @@
 
 drivers = ['null']
 
+enabled_drivers = get_option('drivers_baseband')
+if enabled_drivers != 'all'
+	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
+endif
+
 config_flag_fmt = 'RTE_LIBRTE_@0@_PMD'
 driver_name_fmt = 'rte_pmd_@0@'
diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
index 80de2d91d..d1400ff28 100644
--- a/drivers/bus/meson.build
+++ b/drivers/bus/meson.build
@@ -2,6 +2,12 @@
 # Copyright(c) 2017 Intel Corporation
 
 drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus']
+
+enabled_drivers = get_option('drivers_bus')
+if enabled_drivers != 'all'
+	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
+endif
+
 std_deps = ['eal']
 config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
 driver_name_fmt = 'rte_bus_@0@'
diff --git a/drivers/common/meson.build b/drivers/common/meson.build
index a50934108..311511365 100644
--- a/drivers/common/meson.build
+++ b/drivers/common/meson.build
@@ -3,5 +3,11 @@
 
 std_deps = ['eal']
 drivers = ['cpt', 'dpaax', 'mvep', 'octeontx', 'qat']
+
+enabled_drivers = get_option('drivers_common')
+if enabled_drivers != 'all'
+	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
+endif
+
 config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON'
 driver_name_fmt = 'rte_common_@0@'
diff --git a/drivers/compress/meson.build b/drivers/compress/meson.build
index 817ef3be4..5585ec537 100644
--- a/drivers/compress/meson.build
+++ b/drivers/compress/meson.build
@@ -3,6 +3,11 @@
 
 drivers = ['isal', 'octeontx', 'qat', 'zlib']
 
+enabled_drivers = get_option('drivers_compress')
+if enabled_drivers != 'all'
+	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
+endif
+
 std_deps = ['compressdev'] # compressdev pulls in all other needed deps
 config_flag_fmt = 'RTE_LIBRTE_@0@_PMD'
 driver_name_fmt = 'rte_pmd_@0@'
diff --git a/drivers/crypto/meson.build b/drivers/crypto/meson.build
index 83e78860e..7b1363f23 100644
--- a/drivers/crypto/meson.build
+++ b/drivers/crypto/meson.build
@@ -5,6 +5,11 @@ drivers = ['aesni_gcm', 'aesni_mb', 'caam_jr', 'ccp', 'dpaa_sec', 'dpaa2_sec',
 	'kasumi', 'mvsam', 'null', 'octeontx', 'openssl', 'qat', 'scheduler',
 	'snow3g', 'virtio', 'zuc']
 
+enabled_drivers = get_option('drivers_crypto')
+if enabled_drivers != 'all'
+	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
+endif
+
 std_deps = ['cryptodev'] # cryptodev pulls in all other needed deps
 config_flag_fmt = 'RTE_LIBRTE_@0@_PMD'
 driver_name_fmt = 'rte_pmd_@0@'
diff --git a/drivers/event/meson.build b/drivers/event/meson.build
index fb723f727..14e0c68ac 100644
--- a/drivers/event/meson.build
+++ b/drivers/event/meson.build
@@ -6,6 +6,12 @@ if not (toolchain == 'gcc' and cc.version().version_compare('<4.8.6') and
 	dpdk_conf.has('RTE_ARCH_ARM64'))
 	drivers += 'octeontx'
 endif
+
+enabled_drivers = get_option('drivers_event')
+if enabled_drivers != 'all'
+	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
+endif
+
 std_deps = ['eventdev', 'kvargs']
 config_flag_fmt = 'RTE_LIBRTE_@0@_EVENTDEV_PMD'
 driver_name_fmt = 'rte_pmd_@0@_event'
diff --git a/drivers/mempool/meson.build b/drivers/mempool/meson.build
index 4527d9806..cfacedd1b 100644
--- a/drivers/mempool/meson.build
+++ b/drivers/mempool/meson.build
@@ -2,6 +2,12 @@
 # Copyright(c) 2017 Intel Corporation
 
 drivers = ['bucket', 'dpaa', 'dpaa2', 'octeontx', 'ring', 'stack']
+
+enabled_drivers = get_option('drivers_mempool')
+if enabled_drivers != 'all'
+	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
+endif
+
 std_deps = ['mempool']
 config_flag_fmt = 'RTE_LIBRTE_@0@_MEMPOOL'
 driver_name_fmt = 'rte_mempool_@0@'
diff --git a/drivers/net/meson.build b/drivers/net/meson.build
index ed99896c3..901be5eac 100644
--- a/drivers/net/meson.build
+++ b/drivers/net/meson.build
@@ -42,6 +42,12 @@ drivers = ['af_packet',
 	'virtio',
 	'vmxnet3',
 ]
+
+enabled_drivers = get_option('drivers_net')
+if enabled_drivers != 'all'
+	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
+endif
+
 std_deps = ['ethdev', 'kvargs'] # 'ethdev' also pulls in mbuf, net, eal etc
 std_deps += ['bus_pci']         # very many PMDs depend on PCI, so make std
 std_deps += ['bus_vdev']        # same with vdev bus
diff --git a/drivers/raw/meson.build b/drivers/raw/meson.build
index a61cdccef..ca98e92f0 100644
--- a/drivers/raw/meson.build
+++ b/drivers/raw/meson.build
@@ -2,6 +2,12 @@
 # Copyright 2018 NXP
 
 drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev']
+
+enabled_drivers = get_option('drivers_raw')
+if enabled_drivers != 'all'
+	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
+endif
+
 std_deps = ['rawdev']
 config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV'
 driver_name_fmt = 'rte_pmd_@0@'
diff --git a/lib/meson.build b/lib/meson.build
index e067ce5ea..384867926 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -34,6 +34,11 @@ if is_windows
 	libraries = ['kvargs','eal'] # only supported libraries for windows
 endif
 
+enabled_libs = get_option('libs')
+if enabled_libs != 'all'
+	libraries = (enabled_libs == 'none') ? [] : enabled_libs.split(',')
+endif
+
 default_cflags = machine_args
 if cc.has_argument('-Wno-format-truncation')
 	default_cflags += '-Wno-format-truncation'
diff --git a/meson_options.txt b/meson_options.txt
index 16d9f92c6..5184917b0 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -2,6 +2,26 @@
 
 option('allow_invalid_socket_id', type: 'boolean', value: false,
 	description: 'allow out-of-range NUMA socket id\'s for platforms that don\'t report the value correctly')
+option('apps', type: 'string', value: 'all',
+	description: 'List of applications to build. Defaults to all')
+option('drivers_baseband', type: 'string', value: 'all',
+	description: 'List of baseband drivers to build. Defaults to all')
+option('drivers_bus', type: 'string', value: 'all',
+	description: 'List of bus drivers to build. Defaults to all')
+option('drivers_common', type: 'string', value: 'all',
+	description: 'List of common drivers to build. Defaults to all')
+option('drivers_compress', type: 'string', value: 'all',
+	description: 'List of compress drivers to build. Defaults to all')
+option('drivers_crypto', type: 'string', value: 'all',
+	description: 'List of crypto drivers to build. Defaults to all')
+option('drivers_event', type: 'string', value: 'all',
+	description: 'List of event drivers to build. Defaults to all')
+option('drivers_mempool', type: 'string', value: 'all',
+	description: 'List of mempool drivers to build. Defaults to all')
+option('drivers_net', type: 'string', value: 'all',
+	description: 'List of net drivers to build. Defaults to all')
+option('drivers_raw', type: 'string', value: 'all',
+	description: 'List of rawdev drivers to build. Defaults to all')
 option('drivers_install_subdir', type: 'string', value: 'dpdk/pmds-<VERSION>',
 	description: 'Subdirectory of libdir where to install PMDs. Defaults to using a versioned subdirectory.')
 option('enable_docs', type: 'boolean', value: false,
@@ -16,6 +36,8 @@ option('include_subdir_arch', type: 'string', value: '',
 	description: 'subdirectory where to install arch-dependent headers')
 option('kernel_dir', type: 'string', value: '',
 	description: 'path to the kernel for building kernel modules, they will be installed in $DEST_DIR/$kernel_dir/../extra/dpdk')
+option('libs', type: 'string', value: 'all',
+	description: 'List of libraries to build. Defaults to all')
 option('lib_musdk_dir', type: 'string', value: '',
 	description: 'path to the MUSDK library installation directory')
 option('machine', type: 'string', value: 'native',
-- 
2.17.1

Re: [dpdk-dev] [PATCH 2/2] meson: make build configurable

[Michael Santana Francisco]

On 5/29/19 12:39 PM, Ilya Maximets wrote:
> The first thing many developers do before start building DPDK is
> disabling all the not needed divers and libraries. This happens
> just because more than a half of DPDK dirvers and libraries are not
> needed for the particular reason. For example, you don't need
> dpaa*, octeon*, various croypto devices, eventdev, etc. if you're
> only want to build OVS for x86_64 with static linking.
>
> By disabling everything you don't need, build speeds up literally 10x
> times. This is important for CI systems. For example, TravisCI wastes
> 10 minutes for the default DPDK build just to check linking with OVS.
>
> Another thing is the binary size. Number of DPDK libraries and,
> as a result, size of resulted statically linked application decreases
> significantly.
>
> Important thing also that you're able to not install some dependencies
> if you don't have them on a target platform. Just disable libs/drivers
> that depends on it. Similar thing for the glibc version mismatch
> between build and target platforms.
>
> Also, I have to note that less code means less probability of
> failures and less number of attack vectors.
>
> This patch gives 'meson' the power of configurability that we
> have with 'make'. Using new options it's possible to enable just
> what you need and nothing more.
>
> For example, following cmdline could be used to build almost minimal
> set of DPDK libs and drivers to check OVS build:
>
>    $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false \
>                  -Ddrivers_bus=pci,vdev          \
>                  -Ddrivers_mempool=ring          \
>                  -Ddrivers_net=null,virtio,ring  \
>                  -Ddrivers_crypto=virtio         \
>                  -Ddrivers_compress=none         \
>                  -Ddrivers_event=none            \
>                  -Ddrivers_baseband=none         \
>                  -Ddrivers_raw=none              \
>                  -Ddrivers_common=none           \
>                  -Dlibs=kvargs,eal,cmdline,ring,mempool,mbuf,net,meter,\
>                         ethdev,pci,hash,cryptodev,pdump,vhost \
>                  -Dapps=none
>
> Adding a few real net drivers will give configuration that can be used
> in production environment.
>
> Looks not very pretty, but this could be moved to a script.
>
> Build details:
>
>    Build targets in project: 57
>
>    $ time ninja
>    real    0m11,528s
>    user    1m4,137s
>    sys     0m4,935s
>
>    $ du -sh ../dpdk_meson_install/
>    3,5M    ../dpdk_meson_install/
>
> To compare with what we have without these options:
>
>    $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false
>    Build targets in project: 434
>
>    $ time ninja
>    real    1m38,963s
>    user    10m18,624s
>    sys     0m45,478s
>
>    $ du -sh ../dpdk_meson_install/
>    27M     ../dpdk_meson_install/
>
> 10x speed up for the user time.
> 7.7 times size decrease.
>
> This is probably not much user-friendly because it's not a Kconfig
> and dependency tracking in meson is really poor, so it requires
> usually few iterations to pick correct set of libraries to satisfy
> all dependencies. However, it's not a big deal. Options intended
> for a proficient users who knows what they need.
>
> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
> ---
>   app/meson.build              |  5 +++++
>   drivers/baseband/meson.build |  5 +++++
>   drivers/bus/meson.build      |  6 ++++++
>   drivers/common/meson.build   |  6 ++++++
>   drivers/compress/meson.build |  5 +++++
>   drivers/crypto/meson.build   |  5 +++++
>   drivers/event/meson.build    |  6 ++++++
>   drivers/mempool/meson.build  |  6 ++++++
>   drivers/net/meson.build      |  6 ++++++
>   drivers/raw/meson.build      |  6 ++++++
>   lib/meson.build              |  5 +++++
>   meson_options.txt            | 22 ++++++++++++++++++++++
>   12 files changed, 83 insertions(+)
>
> diff --git a/app/meson.build b/app/meson.build
> index 2b9fdef74..48972954c 100644
> --- a/app/meson.build
> +++ b/app/meson.build
> @@ -17,6 +17,11 @@ apps = [
>   	'test-pipeline',
>   	'test-pmd']
>   
> +enabled_apps = get_option('apps')
> +if enabled_apps != 'all'
> +	apps = (enabled_apps == 'none') ? [] : enabled_apps.split(',')
> +endif
> +
>   # for BSD only
>   lib_execinfo = cc.find_library('execinfo', required: false)
>   
> diff --git a/drivers/baseband/meson.build b/drivers/baseband/meson.build
> index 52489df35..fabc80fc2 100644
> --- a/drivers/baseband/meson.build
> +++ b/drivers/baseband/meson.build
> @@ -3,5 +3,10 @@
>   
>   drivers = ['null']
>   
> +enabled_drivers = get_option('drivers_baseband')
> +if enabled_drivers != 'all'
> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
> +endif
> +
>   config_flag_fmt = 'RTE_LIBRTE_@0@_PMD'
>   driver_name_fmt = 'rte_pmd_@0@'
> diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
> index 80de2d91d..d1400ff28 100644
> --- a/drivers/bus/meson.build
> +++ b/drivers/bus/meson.build
> @@ -2,6 +2,12 @@
>   # Copyright(c) 2017 Intel Corporation
>   
>   drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus']
> +
> +enabled_drivers = get_option('drivers_bus')
> +if enabled_drivers != 'all'
> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
> +endif
> +
>   std_deps = ['eal']
>   config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
>   driver_name_fmt = 'rte_bus_@0@'
> diff --git a/drivers/common/meson.build b/drivers/common/meson.build
> index a50934108..311511365 100644
> --- a/drivers/common/meson.build
> +++ b/drivers/common/meson.build
> @@ -3,5 +3,11 @@
>   
>   std_deps = ['eal']
>   drivers = ['cpt', 'dpaax', 'mvep', 'octeontx', 'qat']
> +
> +enabled_drivers = get_option('drivers_common')
> +if enabled_drivers != 'all'
> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
> +endif
> +
>   config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON'
>   driver_name_fmt = 'rte_common_@0@'
> diff --git a/drivers/compress/meson.build b/drivers/compress/meson.build
> index 817ef3be4..5585ec537 100644
> --- a/drivers/compress/meson.build
> +++ b/drivers/compress/meson.build
> @@ -3,6 +3,11 @@
>   
>   drivers = ['isal', 'octeontx', 'qat', 'zlib']
>   
> +enabled_drivers = get_option('drivers_compress')
> +if enabled_drivers != 'all'
> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
> +endif
> +
>   std_deps = ['compressdev'] # compressdev pulls in all other needed deps
>   config_flag_fmt = 'RTE_LIBRTE_@0@_PMD'
>   driver_name_fmt = 'rte_pmd_@0@'
> diff --git a/drivers/crypto/meson.build b/drivers/crypto/meson.build
> index 83e78860e..7b1363f23 100644
> --- a/drivers/crypto/meson.build
> +++ b/drivers/crypto/meson.build
> @@ -5,6 +5,11 @@ drivers = ['aesni_gcm', 'aesni_mb', 'caam_jr', 'ccp', 'dpaa_sec', 'dpaa2_sec',
>   	'kasumi', 'mvsam', 'null', 'octeontx', 'openssl', 'qat', 'scheduler',
>   	'snow3g', 'virtio', 'zuc']
>   
> +enabled_drivers = get_option('drivers_crypto')
> +if enabled_drivers != 'all'
> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
> +endif
> +
>   std_deps = ['cryptodev'] # cryptodev pulls in all other needed deps
>   config_flag_fmt = 'RTE_LIBRTE_@0@_PMD'
>   driver_name_fmt = 'rte_pmd_@0@'
> diff --git a/drivers/event/meson.build b/drivers/event/meson.build
> index fb723f727..14e0c68ac 100644
> --- a/drivers/event/meson.build
> +++ b/drivers/event/meson.build
> @@ -6,6 +6,12 @@ if not (toolchain == 'gcc' and cc.version().version_compare('<4.8.6') and
>   	dpdk_conf.has('RTE_ARCH_ARM64'))
>   	drivers += 'octeontx'
>   endif
> +
> +enabled_drivers = get_option('drivers_event')
> +if enabled_drivers != 'all'
> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
> +endif
> +
>   std_deps = ['eventdev', 'kvargs']
>   config_flag_fmt = 'RTE_LIBRTE_@0@_EVENTDEV_PMD'
>   driver_name_fmt = 'rte_pmd_@0@_event'
> diff --git a/drivers/mempool/meson.build b/drivers/mempool/meson.build
> index 4527d9806..cfacedd1b 100644
> --- a/drivers/mempool/meson.build
> +++ b/drivers/mempool/meson.build
> @@ -2,6 +2,12 @@
>   # Copyright(c) 2017 Intel Corporation
>   
>   drivers = ['bucket', 'dpaa', 'dpaa2', 'octeontx', 'ring', 'stack']
> +
> +enabled_drivers = get_option('drivers_mempool')
> +if enabled_drivers != 'all'
> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
> +endif
> +
>   std_deps = ['mempool']
>   config_flag_fmt = 'RTE_LIBRTE_@0@_MEMPOOL'
>   driver_name_fmt = 'rte_mempool_@0@'
> diff --git a/drivers/net/meson.build b/drivers/net/meson.build
> index ed99896c3..901be5eac 100644
> --- a/drivers/net/meson.build
> +++ b/drivers/net/meson.build
> @@ -42,6 +42,12 @@ drivers = ['af_packet',
>   	'virtio',
>   	'vmxnet3',
>   ]
> +
> +enabled_drivers = get_option('drivers_net')
> +if enabled_drivers != 'all'
> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
> +endif
> +
>   std_deps = ['ethdev', 'kvargs'] # 'ethdev' also pulls in mbuf, net, eal etc
>   std_deps += ['bus_pci']         # very many PMDs depend on PCI, so make std
>   std_deps += ['bus_vdev']        # same with vdev bus
> diff --git a/drivers/raw/meson.build b/drivers/raw/meson.build
> index a61cdccef..ca98e92f0 100644
> --- a/drivers/raw/meson.build
> +++ b/drivers/raw/meson.build
> @@ -2,6 +2,12 @@
>   # Copyright 2018 NXP
>   
>   drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev']
> +
> +enabled_drivers = get_option('drivers_raw')
> +if enabled_drivers != 'all'
> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
> +endif
> +
>   std_deps = ['rawdev']
>   config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV'
>   driver_name_fmt = 'rte_pmd_@0@'
> diff --git a/lib/meson.build b/lib/meson.build
> index e067ce5ea..384867926 100644
> --- a/lib/meson.build
> +++ b/lib/meson.build
> @@ -34,6 +34,11 @@ if is_windows
>   	libraries = ['kvargs','eal'] # only supported libraries for windows
>   endif
>   
> +enabled_libs = get_option('libs')
> +if enabled_libs != 'all'
> +	libraries = (enabled_libs == 'none') ? [] : enabled_libs.split(',')
> +endif
> +
>   default_cflags = machine_args
>   if cc.has_argument('-Wno-format-truncation')
>   	default_cflags += '-Wno-format-truncation'
> diff --git a/meson_options.txt b/meson_options.txt
> index 16d9f92c6..5184917b0 100644
> --- a/meson_options.txt
> +++ b/meson_options.txt
> @@ -2,6 +2,26 @@
>   
>   option('allow_invalid_socket_id', type: 'boolean', value: false,
>   	description: 'allow out-of-range NUMA socket id\'s for platforms that don\'t report the value correctly')
> +option('apps', type: 'string', value: 'all',
> +	description: 'List of applications to build. Defaults to all')
> +option('drivers_baseband', type: 'string', value: 'all',
> +	description: 'List of baseband drivers to build. Defaults to all')
> +option('drivers_bus', type: 'string', value: 'all',
> +	description: 'List of bus drivers to build. Defaults to all')
> +option('drivers_common', type: 'string', value: 'all',
> +	description: 'List of common drivers to build. Defaults to all')
> +option('drivers_compress', type: 'string', value: 'all',
> +	description: 'List of compress drivers to build. Defaults to all')
> +option('drivers_crypto', type: 'string', value: 'all',
> +	description: 'List of crypto drivers to build. Defaults to all')
> +option('drivers_event', type: 'string', value: 'all',
> +	description: 'List of event drivers to build. Defaults to all')
> +option('drivers_mempool', type: 'string', value: 'all',
> +	description: 'List of mempool drivers to build. Defaults to all')
> +option('drivers_net', type: 'string', value: 'all',
> +	description: 'List of net drivers to build. Defaults to all')
> +option('drivers_raw', type: 'string', value: 'all',
> +	description: 'List of rawdev drivers to build. Defaults to all')
>   option('drivers_install_subdir', type: 'string', value: 'dpdk/pmds-<VERSION>',
>   	description: 'Subdirectory of libdir where to install PMDs. Defaults to using a versioned subdirectory.')
>   option('enable_docs', type: 'boolean', value: false,
> @@ -16,6 +36,8 @@ option('include_subdir_arch', type: 'string', value: '',
>   	description: 'subdirectory where to install arch-dependent headers')
>   option('kernel_dir', type: 'string', value: '',
>   	description: 'path to the kernel for building kernel modules, they will be installed in $DEST_DIR/$kernel_dir/../extra/dpdk')
> +option('libs', type: 'string', value: 'all',
> +	description: 'List of libraries to build. Defaults to all')
>   option('lib_musdk_dir', type: 'string', value: '',
>   	description: 'path to the MUSDK library installation directory')
>   option('machine', type: 'string', value: 'native',

I think this patch is the right step forward. This patch will speed up 
travis builds which is a big plus

Just one small comment, could you document/add some build examples to 
doc/build-sdk-meson.txt and other relevant doc files? Thanks

Acked-by: Michael Santana <msantana@redhat.com>

Re: [dpdk-dev] [PATCH 2/2] meson: make build configurable

[Luca Boccassi]

On Wed, 2019-05-29 at 19:39 +0300, Ilya Maximets wrote:
> The first thing many developers do before start building DPDK is
> disabling all the not needed divers and libraries. This happens
> just because more than a half of DPDK dirvers and libraries are not
> needed for the particular reason. For example, you don't need
> dpaa*, octeon*, various croypto devices, eventdev, etc. if you're
> only want to build OVS for x86_64 with static linking.
> 
> By disabling everything you don't need, build speeds up literally 10x
> times. This is important for CI systems. For example, TravisCI wastes
> 10 minutes for the default DPDK build just to check linking with OVS.
> 
> Another thing is the binary size. Number of DPDK libraries and,
> as a result, size of resulted statically linked application decreases
> significantly.
> 
> Important thing also that you're able to not install some
> dependencies
> if you don't have them on a target platform. Just disable
> libs/drivers
> that depends on it. Similar thing for the glibc version mismatch
> between build and target platforms.
> 
> Also, I have to note that less code means less probability of
> failures and less number of attack vectors.
> 
> This patch gives 'meson' the power of configurability that we
> have with 'make'. Using new options it's possible to enable just
> what you need and nothing more.
> 
> For example, following cmdline could be used to build almost minimal
> set of DPDK libs and drivers to check OVS build:
> 
>   $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false \
>                 -Ddrivers_bus=pci,vdev          \
>                 -Ddrivers_mempool=ring          \
>                 -Ddrivers_net=null,virtio,ring  \
>                 -Ddrivers_crypto=virtio         \
>                 -Ddrivers_compress=none         \
>                 -Ddrivers_event=none            \
>                 -Ddrivers_baseband=none         \
>                 -Ddrivers_raw=none              \
>                 -Ddrivers_common=none           \
>                
> -Dlibs=kvargs,eal,cmdline,ring,mempool,mbuf,net,meter,\
>                        ethdev,pci,hash,cryptodev,pdump,vhost \
>                 -Dapps=none
> 
> Adding a few real net drivers will give configuration that can be
> used
> in production environment.
> 
> Looks not very pretty, but this could be moved to a script.
> 
> Build details:
> 
>   Build targets in project: 57
> 
>   $ time ninja
>   real    0m11,528s
>   user    1m4,137s
>   sys     0m4,935s
> 
>   $ du -sh ../dpdk_meson_install/
>   3,5M    ../dpdk_meson_install/
> 
> To compare with what we have without these options:
> 
>   $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false
>   Build targets in project: 434
> 
>   $ time ninja
>   real    1m38,963s
>   user    10m18,624s
>   sys     0m45,478s
> 
>   $ du -sh ../dpdk_meson_install/
>   27M     ../dpdk_meson_install/
> 
> 10x speed up for the user time.
> 7.7 times size decrease.
> 
> This is probably not much user-friendly because it's not a Kconfig
> and dependency tracking in meson is really poor, so it requires
> usually few iterations to pick correct set of libraries to satisfy
> all dependencies. However, it's not a big deal. Options intended
> for a proficient users who knows what they need.

Hi,

We talked about this a few times in the past, and it was actually one
of the design goals to _avoid_ replicating the octopus-like config
system of the makefiles. That's because it makes the test matrix
insanely complicated, not to mention the harm to user friendliness,
among other things.

If someone doesn't want to use a PMD, they can just avoid installing it
- it's simple enough.

Sorry, but from me it's a very strong NACK.

-- 
Kind regards,
Luca Boccassi

Re: [dpdk-dev] [PATCH 2/2] meson: make build configurable

[Ilya Maximets]

On 29.05.2019 23:15, Michael Santana Francisco wrote:
> On 5/29/19 12:39 PM, Ilya Maximets wrote:
>> The first thing many developers do before start building DPDK is
>> disabling all the not needed divers and libraries. This happens
>> just because more than a half of DPDK dirvers and libraries are not
>> needed for the particular reason. For example, you don't need
>> dpaa*, octeon*, various croypto devices, eventdev, etc. if you're
>> only want to build OVS for x86_64 with static linking.
>>
>> By disabling everything you don't need, build speeds up literally 10x
>> times. This is important for CI systems. For example, TravisCI wastes
>> 10 minutes for the default DPDK build just to check linking with OVS.
>>
>> Another thing is the binary size. Number of DPDK libraries and,
>> as a result, size of resulted statically linked application decreases
>> significantly.
>>
>> Important thing also that you're able to not install some dependencies
>> if you don't have them on a target platform. Just disable libs/drivers
>> that depends on it. Similar thing for the glibc version mismatch
>> between build and target platforms.
>>
>> Also, I have to note that less code means less probability of
>> failures and less number of attack vectors.
>>
>> This patch gives 'meson' the power of configurability that we
>> have with 'make'. Using new options it's possible to enable just
>> what you need and nothing more.
>>
>> For example, following cmdline could be used to build almost minimal
>> set of DPDK libs and drivers to check OVS build:
>>
>>   $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false \
>>                 -Ddrivers_bus=pci,vdev          \
>>                 -Ddrivers_mempool=ring          \
>>                 -Ddrivers_net=null,virtio,ring  \
>>                 -Ddrivers_crypto=virtio         \
>>                 -Ddrivers_compress=none         \
>>                 -Ddrivers_event=none            \
>>                 -Ddrivers_baseband=none         \
>>                 -Ddrivers_raw=none              \
>>                 -Ddrivers_common=none           \
>>                 -Dlibs=kvargs,eal,cmdline,ring,mempool,mbuf,net,meter,\
>>                        ethdev,pci,hash,cryptodev,pdump,vhost \
>>                 -Dapps=none
>>
>> Adding a few real net drivers will give configuration that can be used
>> in production environment.
>>
>> Looks not very pretty, but this could be moved to a script.
>>
>> Build details:
>>
>>   Build targets in project: 57
>>
>>   $ time ninja
>>   real    0m11,528s
>>   user    1m4,137s
>>   sys     0m4,935s
>>
>>   $ du -sh ../dpdk_meson_install/
>>   3,5M    ../dpdk_meson_install/
>>
>> To compare with what we have without these options:
>>
>>   $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false
>>   Build targets in project: 434
>>
>>   $ time ninja
>>   real    1m38,963s
>>   user    10m18,624s
>>   sys     0m45,478s
>>
>>   $ du -sh ../dpdk_meson_install/
>>   27M     ../dpdk_meson_install/
>>
>> 10x speed up for the user time.
>> 7.7 times size decrease.
>>
>> This is probably not much user-friendly because it's not a Kconfig
>> and dependency tracking in meson is really poor, so it requires
>> usually few iterations to pick correct set of libraries to satisfy
>> all dependencies. However, it's not a big deal. Options intended
>> for a proficient users who knows what they need.
>>
>> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
>> ---
>>  app/meson.build              |  5 +++++
>>  drivers/baseband/meson.build |  5 +++++
>>  drivers/bus/meson.build      |  6 ++++++
>>  drivers/common/meson.build   |  6 ++++++
>>  drivers/compress/meson.build |  5 +++++
>>  drivers/crypto/meson.build   |  5 +++++
>>  drivers/event/meson.build    |  6 ++++++
>>  drivers/mempool/meson.build  |  6 ++++++
>>  drivers/net/meson.build      |  6 ++++++
>>  drivers/raw/meson.build      |  6 ++++++
>>  lib/meson.build              |  5 +++++
>>  meson_options.txt            | 22 ++++++++++++++++++++++
>>  12 files changed, 83 insertions(+)
>>
>> diff --git a/app/meson.build b/app/meson.build
>> index 2b9fdef74..48972954c 100644
>> --- a/app/meson.build
>> +++ b/app/meson.build
>> @@ -17,6 +17,11 @@ apps = [
>>  	'test-pipeline',
>>  	'test-pmd']
>>  
>> +enabled_apps = get_option('apps')
>> +if enabled_apps != 'all'
>> +	apps = (enabled_apps == 'none') ? [] : enabled_apps.split(',')
>> +endif
>> +
>>  # for BSD only
>>  lib_execinfo = cc.find_library('execinfo', required: false)
>>  
>> diff --git a/drivers/baseband/meson.build b/drivers/baseband/meson.build
>> index 52489df35..fabc80fc2 100644
>> --- a/drivers/baseband/meson.build
>> +++ b/drivers/baseband/meson.build
>> @@ -3,5 +3,10 @@
>>  
>>  drivers = ['null']
>>  
>> +enabled_drivers = get_option('drivers_baseband')
>> +if enabled_drivers != 'all'
>> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
>> +endif
>> +
>>  config_flag_fmt = 'RTE_LIBRTE_@0@_PMD'
>>  driver_name_fmt = 'rte_pmd_@0@'
>> diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
>> index 80de2d91d..d1400ff28 100644
>> --- a/drivers/bus/meson.build
>> +++ b/drivers/bus/meson.build
>> @@ -2,6 +2,12 @@
>>  # Copyright(c) 2017 Intel Corporation
>>  
>>  drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus']
>> +
>> +enabled_drivers = get_option('drivers_bus')
>> +if enabled_drivers != 'all'
>> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
>> +endif
>> +
>>  std_deps = ['eal']
>>  config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
>>  driver_name_fmt = 'rte_bus_@0@'
>> diff --git a/drivers/common/meson.build b/drivers/common/meson.build
>> index a50934108..311511365 100644
>> --- a/drivers/common/meson.build
>> +++ b/drivers/common/meson.build
>> @@ -3,5 +3,11 @@
>>  
>>  std_deps = ['eal']
>>  drivers = ['cpt', 'dpaax', 'mvep', 'octeontx', 'qat']
>> +
>> +enabled_drivers = get_option('drivers_common')
>> +if enabled_drivers != 'all'
>> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
>> +endif
>> +
>>  config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON'
>>  driver_name_fmt = 'rte_common_@0@'
>> diff --git a/drivers/compress/meson.build b/drivers/compress/meson.build
>> index 817ef3be4..5585ec537 100644
>> --- a/drivers/compress/meson.build
>> +++ b/drivers/compress/meson.build
>> @@ -3,6 +3,11 @@
>>  
>>  drivers = ['isal', 'octeontx', 'qat', 'zlib']
>>  
>> +enabled_drivers = get_option('drivers_compress')
>> +if enabled_drivers != 'all'
>> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
>> +endif
>> +
>>  std_deps = ['compressdev'] # compressdev pulls in all other needed deps
>>  config_flag_fmt = 'RTE_LIBRTE_@0@_PMD'
>>  driver_name_fmt = 'rte_pmd_@0@'
>> diff --git a/drivers/crypto/meson.build b/drivers/crypto/meson.build
>> index 83e78860e..7b1363f23 100644
>> --- a/drivers/crypto/meson.build
>> +++ b/drivers/crypto/meson.build
>> @@ -5,6 +5,11 @@ drivers = ['aesni_gcm', 'aesni_mb', 'caam_jr', 'ccp', 'dpaa_sec', 'dpaa2_sec',
>>  	'kasumi', 'mvsam', 'null', 'octeontx', 'openssl', 'qat', 'scheduler',
>>  	'snow3g', 'virtio', 'zuc']
>>  
>> +enabled_drivers = get_option('drivers_crypto')
>> +if enabled_drivers != 'all'
>> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
>> +endif
>> +
>>  std_deps = ['cryptodev'] # cryptodev pulls in all other needed deps
>>  config_flag_fmt = 'RTE_LIBRTE_@0@_PMD'
>>  driver_name_fmt = 'rte_pmd_@0@'
>> diff --git a/drivers/event/meson.build b/drivers/event/meson.build
>> index fb723f727..14e0c68ac 100644
>> --- a/drivers/event/meson.build
>> +++ b/drivers/event/meson.build
>> @@ -6,6 +6,12 @@ if not (toolchain == 'gcc' and cc.version().version_compare('<4.8.6') and
>>  	dpdk_conf.has('RTE_ARCH_ARM64'))
>>  	drivers += 'octeontx'
>>  endif
>> +
>> +enabled_drivers = get_option('drivers_event')
>> +if enabled_drivers != 'all'
>> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
>> +endif
>> +
>>  std_deps = ['eventdev', 'kvargs']
>>  config_flag_fmt = 'RTE_LIBRTE_@0@_EVENTDEV_PMD'
>>  driver_name_fmt = 'rte_pmd_@0@_event'
>> diff --git a/drivers/mempool/meson.build b/drivers/mempool/meson.build
>> index 4527d9806..cfacedd1b 100644
>> --- a/drivers/mempool/meson.build
>> +++ b/drivers/mempool/meson.build
>> @@ -2,6 +2,12 @@
>>  # Copyright(c) 2017 Intel Corporation
>>  
>>  drivers = ['bucket', 'dpaa', 'dpaa2', 'octeontx', 'ring', 'stack']
>> +
>> +enabled_drivers = get_option('drivers_mempool')
>> +if enabled_drivers != 'all'
>> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
>> +endif
>> +
>>  std_deps = ['mempool']
>>  config_flag_fmt = 'RTE_LIBRTE_@0@_MEMPOOL'
>>  driver_name_fmt = 'rte_mempool_@0@'
>> diff --git a/drivers/net/meson.build b/drivers/net/meson.build
>> index ed99896c3..901be5eac 100644
>> --- a/drivers/net/meson.build
>> +++ b/drivers/net/meson.build
>> @@ -42,6 +42,12 @@ drivers = ['af_packet',
>>  	'virtio',
>>  	'vmxnet3',
>>  ]
>> +
>> +enabled_drivers = get_option('drivers_net')
>> +if enabled_drivers != 'all'
>> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
>> +endif
>> +
>>  std_deps = ['ethdev', 'kvargs'] # 'ethdev' also pulls in mbuf, net, eal etc
>>  std_deps += ['bus_pci']         # very many PMDs depend on PCI, so make std
>>  std_deps += ['bus_vdev']        # same with vdev bus
>> diff --git a/drivers/raw/meson.build b/drivers/raw/meson.build
>> index a61cdccef..ca98e92f0 100644
>> --- a/drivers/raw/meson.build
>> +++ b/drivers/raw/meson.build
>> @@ -2,6 +2,12 @@
>>  # Copyright 2018 NXP
>>  
>>  drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev']
>> +
>> +enabled_drivers = get_option('drivers_raw')
>> +if enabled_drivers != 'all'
>> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
>> +endif
>> +
>>  std_deps = ['rawdev']
>>  config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV'
>>  driver_name_fmt = 'rte_pmd_@0@'
>> diff --git a/lib/meson.build b/lib/meson.build
>> index e067ce5ea..384867926 100644
>> --- a/lib/meson.build
>> +++ b/lib/meson.build
>> @@ -34,6 +34,11 @@ if is_windows
>>  	libraries = ['kvargs','eal'] # only supported libraries for windows
>>  endif
>>  
>> +enabled_libs = get_option('libs')
>> +if enabled_libs != 'all'
>> +	libraries = (enabled_libs == 'none') ? [] : enabled_libs.split(',')
>> +endif
>> +
>>  default_cflags = machine_args
>>  if cc.has_argument('-Wno-format-truncation')
>>  	default_cflags += '-Wno-format-truncation'
>> diff --git a/meson_options.txt b/meson_options.txt
>> index 16d9f92c6..5184917b0 100644
>> --- a/meson_options.txt
>> +++ b/meson_options.txt
>> @@ -2,6 +2,26 @@
>>  
>>  option('allow_invalid_socket_id', type: 'boolean', value: false,
>>  	description: 'allow out-of-range NUMA socket id\'s for platforms that don\'t report the value correctly')
>> +option('apps', type: 'string', value: 'all',
>> +	description: 'List of applications to build. Defaults to all')
>> +option('drivers_baseband', type: 'string', value: 'all',
>> +	description: 'List of baseband drivers to build. Defaults to all')
>> +option('drivers_bus', type: 'string', value: 'all',
>> +	description: 'List of bus drivers to build. Defaults to all')
>> +option('drivers_common', type: 'string', value: 'all',
>> +	description: 'List of common drivers to build. Defaults to all')
>> +option('drivers_compress', type: 'string', value: 'all',
>> +	description: 'List of compress drivers to build. Defaults to all')
>> +option('drivers_crypto', type: 'string', value: 'all',
>> +	description: 'List of crypto drivers to build. Defaults to all')
>> +option('drivers_event', type: 'string', value: 'all',
>> +	description: 'List of event drivers to build. Defaults to all')
>> +option('drivers_mempool', type: 'string', value: 'all',
>> +	description: 'List of mempool drivers to build. Defaults to all')
>> +option('drivers_net', type: 'string', value: 'all',
>> +	description: 'List of net drivers to build. Defaults to all')
>> +option('drivers_raw', type: 'string', value: 'all',
>> +	description: 'List of rawdev drivers to build. Defaults to all')
>>  option('drivers_install_subdir', type: 'string', value: 'dpdk/pmds-<VERSION>',
>>  	description: 'Subdirectory of libdir where to install PMDs. Defaults to using a versioned subdirectory.')
>>  option('enable_docs', type: 'boolean', value: false,
>> @@ -16,6 +36,8 @@ option('include_subdir_arch', type: 'string', value: '',
>>  	description: 'subdirectory where to install arch-dependent headers')
>>  option('kernel_dir', type: 'string', value: '',
>>  	description: 'path to the kernel for building kernel modules, they will be installed in $DEST_DIR/$kernel_dir/../extra/dpdk')
>> +option('libs', type: 'string', value: 'all',
>> +	description: 'List of libraries to build. Defaults to all')
>>  option('lib_musdk_dir', type: 'string', value: '',
>>  	description: 'path to the MUSDK library installation directory')
>>  option('machine', type: 'string', value: 'native',
> 
> I think this patch is the right step forward. This patch will speed up travis builds which is a big plus
> 
> Just one small comment, could you document/add some build examples to doc/build-sdk-meson.txt and other relevant doc files? Thanks
> 
> Acked-by: Michael Santana <msantana@redhat.com>
> 

Thanks. Will add some docs for the next revisions.

Best regards, Ilya Maximets.

Re: [dpdk-dev] [PATCH 2/2] meson: make build configurable

[Ilya Maximets]

On 29.05.2019 23:37, Luca Boccassi wrote:
> On Wed, 2019-05-29 at 19:39 +0300, Ilya Maximets wrote:
>> The first thing many developers do before start building DPDK is
>> disabling all the not needed divers and libraries. This happens
>> just because more than a half of DPDK dirvers and libraries are not
>> needed for the particular reason. For example, you don't need
>> dpaa*, octeon*, various croypto devices, eventdev, etc. if you're
>> only want to build OVS for x86_64 with static linking.
>>
>> By disabling everything you don't need, build speeds up literally 10x
>> times. This is important for CI systems. For example, TravisCI wastes
>> 10 minutes for the default DPDK build just to check linking with OVS.
>>
>> Another thing is the binary size. Number of DPDK libraries and,
>> as a result, size of resulted statically linked application decreases
>> significantly.
>>
>> Important thing also that you're able to not install some
>> dependencies
>> if you don't have them on a target platform. Just disable
>> libs/drivers
>> that depends on it. Similar thing for the glibc version mismatch
>> between build and target platforms.
>>
>> Also, I have to note that less code means less probability of
>> failures and less number of attack vectors.
>>
>> This patch gives 'meson' the power of configurability that we
>> have with 'make'. Using new options it's possible to enable just
>> what you need and nothing more.
>>
>> For example, following cmdline could be used to build almost minimal
>> set of DPDK libs and drivers to check OVS build:
>>
>>   $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false \
>>                 -Ddrivers_bus=pci,vdev          \
>>                 -Ddrivers_mempool=ring          \
>>                 -Ddrivers_net=null,virtio,ring  \
>>                 -Ddrivers_crypto=virtio         \
>>                 -Ddrivers_compress=none         \
>>                 -Ddrivers_event=none            \
>>                 -Ddrivers_baseband=none         \
>>                 -Ddrivers_raw=none              \
>>                 -Ddrivers_common=none           \
>>                
>> -Dlibs=kvargs,eal,cmdline,ring,mempool,mbuf,net,meter,\
>>                        ethdev,pci,hash,cryptodev,pdump,vhost \
>>                 -Dapps=none
>>
>> Adding a few real net drivers will give configuration that can be
>> used
>> in production environment.
>>
>> Looks not very pretty, but this could be moved to a script.
>>
>> Build details:
>>
>>   Build targets in project: 57
>>
>>   $ time ninja
>>   real    0m11,528s
>>   user    1m4,137s
>>   sys     0m4,935s
>>
>>   $ du -sh ../dpdk_meson_install/
>>   3,5M    ../dpdk_meson_install/
>>
>> To compare with what we have without these options:
>>
>>   $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false
>>   Build targets in project: 434
>>
>>   $ time ninja
>>   real    1m38,963s
>>   user    10m18,624s
>>   sys     0m45,478s
>>
>>   $ du -sh ../dpdk_meson_install/
>>   27M     ../dpdk_meson_install/
>>
>> 10x speed up for the user time.
>> 7.7 times size decrease.
>>
>> This is probably not much user-friendly because it's not a Kconfig
>> and dependency tracking in meson is really poor, so it requires
>> usually few iterations to pick correct set of libraries to satisfy
>> all dependencies. However, it's not a big deal. Options intended
>> for a proficient users who knows what they need.
> 
> Hi,
> 
> We talked about this a few times in the past, and it was actually one
> of the design goals to _avoid_ replicating the octopus-like config
> system of the makefiles. That's because it makes the test matrix
> insanely complicated, not to mention the harm to user friendliness,
> among other things.
> 
> If someone doesn't want to use a PMD, they can just avoid installing it
> - it's simple enough.

So how can I do this? I don't think 'ninja install' has such option.
Also, if you think that it is safe to skip some libs/drivers in installation
process, it must be safe to not build them at all. It's just a waste of
time and computational resources to build something known to be not used.
And if you're going to ship DPDK libraries separately in distros, you'll
have to test their different combinations anyway. If they're so independent
that you don't need to test them in various combinations, than your point
about test matrix is not valid.

> 
> Sorry, but from me it's a very strong NACK.

Sorry, but let me disagree with you. For me, meson configurability is the
essential thing to have in terms of deprecating the 'make' build system.
DPDK was and keeps being (in most cases) the library that users statically
linking to a single application built for particular platform and not using
for anything else. This means that user in most cases knows which parts
needed and which parts will never be used. Current meson build system
doesn't allow to disable anything forcing users to link with the whole bunch
of unused code.

One major case is that you have to have build environment equal to your
target platform in terms of availability of external libraries. So, if I
have some external library on build system, meson will build all the modules
it depends from and will link them to my application. As a result I'll not
be able to run my application on a target platform without installing
additional dependencies which is not acceptable. This patch will allow to
specifically disable all the libs that has unsatisfiable dependencies on
target. Without the patch it's required to manually remove resulted libs and
fix pkg-config and stuff before building apps. This is far less user-friendly
than options I proposed. And yes, I still have to waste time for building
libraries I'll remove right after.

While testing OVS on TravisCI, DPDK was built far more than 30K times
which is more than half of a year of a wasted computational resources
(if we'll count 10 minutes per build).
I think this time could be used more wisely.

Best regards, Ilya Maximets.

Re: [dpdk-dev] [PATCH 2/2] meson: make build configurable

[Bruce Richardson]

On Wed, May 29, 2019 at 09:37:20PM +0100, Luca Boccassi wrote:
> On Wed, 2019-05-29 at 19:39 +0300, Ilya Maximets wrote:
> > The first thing many developers do before start building DPDK is
> > disabling all the not needed divers and libraries. This happens
> > just because more than a half of DPDK dirvers and libraries are not
> > needed for the particular reason. For example, you don't need
> > dpaa*, octeon*, various croypto devices, eventdev, etc. if you're
> > only want to build OVS for x86_64 with static linking.
> > 
> > By disabling everything you don't need, build speeds up literally 10x
> > times. This is important for CI systems. For example, TravisCI wastes
> > 10 minutes for the default DPDK build just to check linking with OVS.
> > 
> > Another thing is the binary size. Number of DPDK libraries and,
> > as a result, size of resulted statically linked application decreases
> > significantly.
> > 
> > Important thing also that you're able to not install some
> > dependencies
> > if you don't have them on a target platform. Just disable
> > libs/drivers
> > that depends on it. Similar thing for the glibc version mismatch
> > between build and target platforms.
> > 
> > Also, I have to note that less code means less probability of
> > failures and less number of attack vectors.
> > 
> > This patch gives 'meson' the power of configurability that we
> > have with 'make'. Using new options it's possible to enable just
> > what you need and nothing more.
> > 
> > For example, following cmdline could be used to build almost minimal
> > set of DPDK libs and drivers to check OVS build:
> > 
> >   $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false \
> >                 -Ddrivers_bus=pci,vdev          \
> >                 -Ddrivers_mempool=ring          \
> >                 -Ddrivers_net=null,virtio,ring  \
> >                 -Ddrivers_crypto=virtio         \
> >                 -Ddrivers_compress=none         \
> >                 -Ddrivers_event=none            \
> >                 -Ddrivers_baseband=none         \
> >                 -Ddrivers_raw=none              \
> >                 -Ddrivers_common=none           \
> >                
> > -Dlibs=kvargs,eal,cmdline,ring,mempool,mbuf,net,meter,\
> >                        ethdev,pci,hash,cryptodev,pdump,vhost \
> >                 -Dapps=none
> > 
> > Adding a few real net drivers will give configuration that can be
> > used
> > in production environment.
> > 
> > Looks not very pretty, but this could be moved to a script.
> > 
> > Build details:
> > 
> >   Build targets in project: 57
> > 
> >   $ time ninja
> >   real    0m11,528s
> >   user    1m4,137s
> >   sys     0m4,935s
> > 
> >   $ du -sh ../dpdk_meson_install/
> >   3,5M    ../dpdk_meson_install/
> > 
> > To compare with what we have without these options:
> > 
> >   $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false
> >   Build targets in project: 434
> > 
> >   $ time ninja
> >   real    1m38,963s
> >   user    10m18,624s
> >   sys     0m45,478s
> > 
> >   $ du -sh ../dpdk_meson_install/
> >   27M     ../dpdk_meson_install/
> > 
> > 10x speed up for the user time.
> > 7.7 times size decrease.
> > 
> > This is probably not much user-friendly because it's not a Kconfig
> > and dependency tracking in meson is really poor, so it requires
> > usually few iterations to pick correct set of libraries to satisfy
> > all dependencies. However, it's not a big deal. Options intended
> > for a proficient users who knows what they need.
> 
> Hi,
> 
> We talked about this a few times in the past, and it was actually one
> of the design goals to _avoid_ replicating the octopus-like config
> system of the makefiles. That's because it makes the test matrix
> insanely complicated, not to mention the harm to user friendliness,
> among other things.
> 
> If someone doesn't want to use a PMD, they can just avoid installing it
> - it's simple enough.
> 
> Sorry, but from me it's a very strong NACK.
> 
I would agree with this position - tracking the dependencies of the
libraries etc. is a nightmare, and requires lots of ifdef'ery in the code
for handling cases where libraries don't exist.

However, I might be ok with limiting the drivers somewhat, since they don't
tend to depend on each other so much, though ideally I'd still prefer to
have one build of DPDK that has minimal configuration. If we say that we
can disable some drivers, though,  issue then becomes whether e.g. the bus
drivers could selectively be disabled, and the knock-on effects of that.
I'd hate to see the case where we end up having the meson.build files for
drivers becoming a massive list of conditional checks for a bunch of
internal dependencies. If someone is wanting to do a custom build of DPDK,
they can always patch out the subdirectories they don't want in the
meson.build files - but because of testing matrixes for such
configurations, I don't think its something we want to explicitly support.

/Bruce

Re: [dpdk-dev] [PATCH 2/2] meson: make build configurable

[Luca Boccassi]

On Thu, 2019-05-30 at 13:03 +0300, Ilya Maximets wrote:
> On 29.05.2019 23:37, Luca Boccassi wrote:
> > On Wed, 2019-05-29 at 19:39 +0300, Ilya Maximets wrote:
> > > The first thing many developers do before start building DPDK is
> > > disabling all the not needed divers and libraries. This happens
> > > just because more than a half of DPDK dirvers and libraries are
> > > not
> > > needed for the particular reason. For example, you don't need
> > > dpaa*, octeon*, various croypto devices, eventdev, etc. if you're
> > > only want to build OVS for x86_64 with static linking.
> > > 
> > > By disabling everything you don't need, build speeds up literally
> > > 10x
> > > times. This is important for CI systems. For example, TravisCI
> > > wastes
> > > 10 minutes for the default DPDK build just to check linking with
> > > OVS.
> > > 
> > > Another thing is the binary size. Number of DPDK libraries and,
> > > as a result, size of resulted statically linked application
> > > decreases
> > > significantly.
> > > 
> > > Important thing also that you're able to not install some
> > > dependencies
> > > if you don't have them on a target platform. Just disable
> > > libs/drivers
> > > that depends on it. Similar thing for the glibc version mismatch
> > > between build and target platforms.
> > > 
> > > Also, I have to note that less code means less probability of
> > > failures and less number of attack vectors.
> > > 
> > > This patch gives 'meson' the power of configurability that we
> > > have with 'make'. Using new options it's possible to enable just
> > > what you need and nothing more.
> > > 
> > > For example, following cmdline could be used to build almost
> > > minimal
> > > set of DPDK libs and drivers to check OVS build:
> > > 
> > >   $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false
> > > \
> > >                 -Ddrivers_bus=pci,vdev          \
> > >                 -Ddrivers_mempool=ring          \
> > >                 -Ddrivers_net=null,virtio,ring  \
> > >                 -Ddrivers_crypto=virtio         \
> > >                 -Ddrivers_compress=none         \
> > >                 -Ddrivers_event=none            \
> > >                 -Ddrivers_baseband=none         \
> > >                 -Ddrivers_raw=none              \
> > >                 -Ddrivers_common=none           \
> > >                
> > > -Dlibs=kvargs,eal,cmdline,ring,mempool,mbuf,net,meter,\
> > >                        ethdev,pci,hash,cryptodev,pdump,vhost \
> > >                 -Dapps=none
> > > 
> > > Adding a few real net drivers will give configuration that can be
> > > used
> > > in production environment.
> > > 
> > > Looks not very pretty, but this could be moved to a script.
> > > 
> > > Build details:
> > > 
> > >   Build targets in project: 57
> > > 
> > >   $ time ninja
> > >   real    0m11,528s
> > >   user    1m4,137s
> > >   sys     0m4,935s
> > > 
> > >   $ du -sh ../dpdk_meson_install/
> > >   3,5M    ../dpdk_meson_install/
> > > 
> > > To compare with what we have without these options:
> > > 
> > >   $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false
> > >   Build targets in project: 434
> > > 
> > >   $ time ninja
> > >   real    1m38,963s
> > >   user    10m18,624s
> > >   sys     0m45,478s
> > > 
> > >   $ du -sh ../dpdk_meson_install/
> > >   27M     ../dpdk_meson_install/
> > > 
> > > 10x speed up for the user time.
> > > 7.7 times size decrease.
> > > 
> > > This is probably not much user-friendly because it's not a
> > > Kconfig
> > > and dependency tracking in meson is really poor, so it requires
> > > usually few iterations to pick correct set of libraries to
> > > satisfy
> > > all dependencies. However, it's not a big deal. Options intended
> > > for a proficient users who knows what they need.
> > 
> > Hi,
> > 
> > We talked about this a few times in the past, and it was actually
> > one
> > of the design goals to _avoid_ replicating the octopus-like config
> > system of the makefiles. That's because it makes the test matrix
> > insanely complicated, not to mention the harm to user friendliness,
> > among other things.
> > 
> > If someone doesn't want to use a PMD, they can just avoid
> > installing it
> > - it's simple enough.
> 
> So how can I do this? I don't think 'ninja install' has such option.
> Also, if you think that it is safe to skip some libs/drivers in
> installation
> process, it must be safe to not build them at all. It's just a waste
> of
> time and computational resources to build something known to be not
> used.
> And if you're going to ship DPDK libraries separately in distros,
> you'll
> have to test their different combinations anyway. If they're so
> independent
> that you don't need to test them in various combinations, than your
> point
> about test matrix is not valid.

It can be done in the packaging step, or post-install if there's no
packaging. An operating system vendor is free to do its own test and
support plan, and decide to leave out some PMDs from it. Canonical does
something similar: in Debian/Ubuntu and derivatives we package PMDs
individually, and then Canonical groups them in 2 sets - a subset they
guarantee to support from their own resources, and the full set as
delivered by the community. But the point is that it's a step that they
decide to take, and pay the price for it in terms of time investment in
validating that particular combination, rather than the onus being on
our very limited and already stretched resources to validate all
combinations.

We can focus our resources into making sure the few combinations that
_must_ be supported, for example due to external dependencies, work
fine.

> > Sorry, but from me it's a very strong NACK.
> 
> Sorry, but let me disagree with you. For me, meson configurability is
> the
> essential thing to have in terms of deprecating the 'make' build
> system.
> DPDK was and keeps being (in most cases) the library that users
> statically
> linking to a single application built for particular platform and not
> using
> for anything else. This means that user in most cases knows which
> parts
> needed and which parts will never be used. Current meson build system
> doesn't allow to disable anything forcing users to link with the
> whole bunch
> of unused code.
> 
> One major case is that you have to have build environment equal to
> your
> target platform in terms of availability of external libraries. So,
> if I
> have some external library on build system, meson will build all the
> modules
> it depends from and will link them to my application. As a result
> I'll not
> be able to run my application on a target platform without installing
> additional dependencies which is not acceptable. This patch will
> allow to
> specifically disable all the libs that has unsatisfiable dependencies
> on
> target. Without the patch it's required to manually remove resulted
> libs and
> fix pkg-config and stuff before building apps. This is far less user-
> friendly
> than options I proposed. And yes, I still have to waste time for
> building
> libraries I'll remove right after.
> 
> While testing OVS on TravisCI, DPDK was built far more than 30K times
> which is more than half of a year of a wasted computational resources
> (if we'll count 10 minutes per build).
> I think this time could be used more wisely.
> 
> Best regards, Ilya Maximets.

But that's the thing: as it was discussed recently, we need to move
away from DPDK being by default a "special sauce" toolkit with millions
of customizations that is custom built and statically linked, like
busybox, and to a situation where it's just another set of system
libraries like any other, shipped by the operating system, like glibc -
see the threads about stable API/ABI and OS-driven delivery by Ray.
The status quo of an insanely granular build configuration that means
everyone is using something different from each other is a bug - not a
feature. Excessive _build time_ configuration exacerbates and
encourages this bug.

Sure, an initial build from scratch will take a couple of minutes more
- mildly annoying, but worth the price. Caching takes care of that
problem already pretty well. Also standardizing on radically fewer
build configurations means you _don't_ have to rebuild DPDK for third
party testing - you just consume the pre-built binaries from the OS of
choice, or the PPA or similar for backports and HEAD builds. That will
save even more time and resources in third-party build systems.

-- 
Kind regards,
Luca Boccassi

Re: [dpdk-dev] [PATCH 2/2] meson: make build configurable

[Ilya Maximets]

On 30.05.2019 13:22, Bruce Richardson wrote:
> On Wed, May 29, 2019 at 09:37:20PM +0100, Luca Boccassi wrote:
>> On Wed, 2019-05-29 at 19:39 +0300, Ilya Maximets wrote:
>>> The first thing many developers do before start building DPDK is
>>> disabling all the not needed divers and libraries. This happens
>>> just because more than a half of DPDK dirvers and libraries are not
>>> needed for the particular reason. For example, you don't need
>>> dpaa*, octeon*, various croypto devices, eventdev, etc. if you're
>>> only want to build OVS for x86_64 with static linking.
>>>
>>> By disabling everything you don't need, build speeds up literally 10x
>>> times. This is important for CI systems. For example, TravisCI wastes
>>> 10 minutes for the default DPDK build just to check linking with OVS.
>>>
>>> Another thing is the binary size. Number of DPDK libraries and,
>>> as a result, size of resulted statically linked application decreases
>>> significantly.
>>>
>>> Important thing also that you're able to not install some
>>> dependencies
>>> if you don't have them on a target platform. Just disable
>>> libs/drivers
>>> that depends on it. Similar thing for the glibc version mismatch
>>> between build and target platforms.
>>>
>>> Also, I have to note that less code means less probability of
>>> failures and less number of attack vectors.
>>>
>>> This patch gives 'meson' the power of configurability that we
>>> have with 'make'. Using new options it's possible to enable just
>>> what you need and nothing more.
>>>
>>> For example, following cmdline could be used to build almost minimal
>>> set of DPDK libs and drivers to check OVS build:
>>>
>>>   $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false \
>>>                 -Ddrivers_bus=pci,vdev          \
>>>                 -Ddrivers_mempool=ring          \
>>>                 -Ddrivers_net=null,virtio,ring  \
>>>                 -Ddrivers_crypto=virtio         \
>>>                 -Ddrivers_compress=none         \
>>>                 -Ddrivers_event=none            \
>>>                 -Ddrivers_baseband=none         \
>>>                 -Ddrivers_raw=none              \
>>>                 -Ddrivers_common=none           \
>>>                
>>> -Dlibs=kvargs,eal,cmdline,ring,mempool,mbuf,net,meter,\
>>>                        ethdev,pci,hash,cryptodev,pdump,vhost \
>>>                 -Dapps=none
>>>
>>> Adding a few real net drivers will give configuration that can be
>>> used
>>> in production environment.
>>>
>>> Looks not very pretty, but this could be moved to a script.
>>>
>>> Build details:
>>>
>>>   Build targets in project: 57
>>>
>>>   $ time ninja
>>>   real    0m11,528s
>>>   user    1m4,137s
>>>   sys     0m4,935s
>>>
>>>   $ du -sh ../dpdk_meson_install/
>>>   3,5M    ../dpdk_meson_install/
>>>
>>> To compare with what we have without these options:
>>>
>>>   $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false
>>>   Build targets in project: 434
>>>
>>>   $ time ninja
>>>   real    1m38,963s
>>>   user    10m18,624s
>>>   sys     0m45,478s
>>>
>>>   $ du -sh ../dpdk_meson_install/
>>>   27M     ../dpdk_meson_install/
>>>
>>> 10x speed up for the user time.
>>> 7.7 times size decrease.
>>>
>>> This is probably not much user-friendly because it's not a Kconfig
>>> and dependency tracking in meson is really poor, so it requires
>>> usually few iterations to pick correct set of libraries to satisfy
>>> all dependencies. However, it's not a big deal. Options intended
>>> for a proficient users who knows what they need.
>>
>> Hi,
>>
>> We talked about this a few times in the past, and it was actually one
>> of the design goals to _avoid_ replicating the octopus-like config
>> system of the makefiles. That's because it makes the test matrix
>> insanely complicated, not to mention the harm to user friendliness,
>> among other things.
>>
>> If someone doesn't want to use a PMD, they can just avoid installing it
>> - it's simple enough.
>>
>> Sorry, but from me it's a very strong NACK.
>>
> I would agree with this position - tracking the dependencies of the
> libraries etc. is a nightmare, and requires lots of ifdef'ery in the code
> for handling cases where libraries don't exist.
> 
> However, I might be ok with limiting the drivers somewhat, since they don't
> tend to depend on each other so much, though ideally I'd still prefer to
> have one build of DPDK that has minimal configuration. If we say that we
> can disable some drivers, though,  issue then becomes whether e.g. the bus
> drivers could selectively be disabled, and the knock-on effects of that.
> I'd hate to see the case where we end up having the meson.build files for
> drivers becoming a massive list of conditional checks for a bunch of
> internal dependencies. If someone is wanting to do a custom build of DPDK,
> they can always patch out the subdirectories they don't want in the
> meson.build files - but because of testing matrixes for such
> configurations, I don't think its something we want to explicitly support.

Thanks Bruce.
I see your point.

Please, take a look at the first patch of this series. It doesn't directly
connected to this one, but should speed up meson configuration a bit.

Best regards, Ilya Maximets.

Re: [dpdk-dev] [PATCH 1/2] meson: don't check dependencies for tests if not required

[Bruce Richardson]

On Wed, May 29, 2019 at 07:39:57PM +0300, Ilya Maximets wrote:
> Don't need to check dependencies if test apps will not be built anyway.
> 
> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
> ---
>  app/test/meson.build | 38 +++++++++++++++++++-------------------
>  1 file changed, 19 insertions(+), 19 deletions(-)
> 
Agree with the idea.

Would this work as a shorter alternative placed at the top of the file?

if not get_option('tests')
	subdir_done()
endif

/Bruce

Re: [dpdk-dev] [PATCH 2/2] meson: make build configurable

[Ilya Maximets]

On 30.05.2019 14:06, Luca Boccassi wrote:
> On Thu, 2019-05-30 at 13:03 +0300, Ilya Maximets wrote:
>> On 29.05.2019 23:37, Luca Boccassi wrote:
>>> On Wed, 2019-05-29 at 19:39 +0300, Ilya Maximets wrote:
>>>> The first thing many developers do before start building DPDK is
>>>> disabling all the not needed divers and libraries. This happens
>>>> just because more than a half of DPDK dirvers and libraries are
>>>> not
>>>> needed for the particular reason. For example, you don't need
>>>> dpaa*, octeon*, various croypto devices, eventdev, etc. if you're
>>>> only want to build OVS for x86_64 with static linking.
>>>>
>>>> By disabling everything you don't need, build speeds up literally
>>>> 10x
>>>> times. This is important for CI systems. For example, TravisCI
>>>> wastes
>>>> 10 minutes for the default DPDK build just to check linking with
>>>> OVS.
>>>>
>>>> Another thing is the binary size. Number of DPDK libraries and,
>>>> as a result, size of resulted statically linked application
>>>> decreases
>>>> significantly.
>>>>
>>>> Important thing also that you're able to not install some
>>>> dependencies
>>>> if you don't have them on a target platform. Just disable
>>>> libs/drivers
>>>> that depends on it. Similar thing for the glibc version mismatch
>>>> between build and target platforms.
>>>>
>>>> Also, I have to note that less code means less probability of
>>>> failures and less number of attack vectors.
>>>>
>>>> This patch gives 'meson' the power of configurability that we
>>>> have with 'make'. Using new options it's possible to enable just
>>>> what you need and nothing more.
>>>>
>>>> For example, following cmdline could be used to build almost
>>>> minimal
>>>> set of DPDK libs and drivers to check OVS build:
>>>>
>>>>   $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false
>>>> \
>>>>                 -Ddrivers_bus=pci,vdev          \
>>>>                 -Ddrivers_mempool=ring          \
>>>>                 -Ddrivers_net=null,virtio,ring  \
>>>>                 -Ddrivers_crypto=virtio         \
>>>>                 -Ddrivers_compress=none         \
>>>>                 -Ddrivers_event=none            \
>>>>                 -Ddrivers_baseband=none         \
>>>>                 -Ddrivers_raw=none              \
>>>>                 -Ddrivers_common=none           \
>>>>                
>>>> -Dlibs=kvargs,eal,cmdline,ring,mempool,mbuf,net,meter,\
>>>>                        ethdev,pci,hash,cryptodev,pdump,vhost \
>>>>                 -Dapps=none
>>>>
>>>> Adding a few real net drivers will give configuration that can be
>>>> used
>>>> in production environment.
>>>>
>>>> Looks not very pretty, but this could be moved to a script.
>>>>
>>>> Build details:
>>>>
>>>>   Build targets in project: 57
>>>>
>>>>   $ time ninja
>>>>   real    0m11,528s
>>>>   user    1m4,137s
>>>>   sys     0m4,935s
>>>>
>>>>   $ du -sh ../dpdk_meson_install/
>>>>   3,5M    ../dpdk_meson_install/
>>>>
>>>> To compare with what we have without these options:
>>>>
>>>>   $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false
>>>>   Build targets in project: 434
>>>>
>>>>   $ time ninja
>>>>   real    1m38,963s
>>>>   user    10m18,624s
>>>>   sys     0m45,478s
>>>>
>>>>   $ du -sh ../dpdk_meson_install/
>>>>   27M     ../dpdk_meson_install/
>>>>
>>>> 10x speed up for the user time.
>>>> 7.7 times size decrease.
>>>>
>>>> This is probably not much user-friendly because it's not a
>>>> Kconfig
>>>> and dependency tracking in meson is really poor, so it requires
>>>> usually few iterations to pick correct set of libraries to
>>>> satisfy
>>>> all dependencies. However, it's not a big deal. Options intended
>>>> for a proficient users who knows what they need.
>>>
>>> Hi,
>>>
>>> We talked about this a few times in the past, and it was actually
>>> one
>>> of the design goals to _avoid_ replicating the octopus-like config
>>> system of the makefiles. That's because it makes the test matrix
>>> insanely complicated, not to mention the harm to user friendliness,
>>> among other things.
>>>
>>> If someone doesn't want to use a PMD, they can just avoid
>>> installing it
>>> - it's simple enough.
>>
>> So how can I do this? I don't think 'ninja install' has such option.
>> Also, if you think that it is safe to skip some libs/drivers in
>> installation
>> process, it must be safe to not build them at all. It's just a waste
>> of
>> time and computational resources to build something known to be not
>> used.
>> And if you're going to ship DPDK libraries separately in distros,
>> you'll
>> have to test their different combinations anyway. If they're so
>> independent
>> that you don't need to test them in various combinations, than your
>> point
>> about test matrix is not valid.
> 
> It can be done in the packaging step, or post-install if there's no
> packaging. An operating system vendor is free to do its own test and
> support plan, and decide to leave out some PMDs from it.

This technically means doing this manually/write custom scripts.

> Canonical does
> something similar: in Debian/Ubuntu and derivatives we package PMDs
> individually, and then Canonical groups them in 2 sets - a subset they
> guarantee to support from their own resources, and the full set as
> delivered by the community. But the point is that it's a step that they
> decide to take, and pay the price for it in terms of time investment in
> validating that particular combination, rather than the onus being on
> our very limited and already stretched resources to validate all
> combinations.
> 
> We can focus our resources into making sure the few combinations that
> _must_ be supported, for example due to external dependencies, work
> fine.
> 
>>> Sorry, but from me it's a very strong NACK.
>>
>> Sorry, but let me disagree with you. For me, meson configurability is
>> the
>> essential thing to have in terms of deprecating the 'make' build
>> system.
>> DPDK was and keeps being (in most cases) the library that users
>> statically
>> linking to a single application built for particular platform and not
>> using
>> for anything else. This means that user in most cases knows which
>> parts
>> needed and which parts will never be used. Current meson build system
>> doesn't allow to disable anything forcing users to link with the
>> whole bunch
>> of unused code.
>>
>> One major case is that you have to have build environment equal to
>> your
>> target platform in terms of availability of external libraries. So,
>> if I
>> have some external library on build system, meson will build all the
>> modules
>> it depends from and will link them to my application. As a result
>> I'll not
>> be able to run my application on a target platform without installing
>> additional dependencies which is not acceptable. This patch will
>> allow to
>> specifically disable all the libs that has unsatisfiable dependencies
>> on
>> target. Without the patch it's required to manually remove resulted
>> libs and
>> fix pkg-config and stuff before building apps. This is far less user-
>> friendly
>> than options I proposed. And yes, I still have to waste time for
>> building
>> libraries I'll remove right after.
>>
>> While testing OVS on TravisCI, DPDK was built far more than 30K times
>> which is more than half of a year of a wasted computational resources
>> (if we'll count 10 minutes per build).
>> I think this time could be used more wisely.
>>
>> Best regards, Ilya Maximets.
> 
> But that's the thing: as it was discussed recently, we need to move
> away from DPDK being by default a "special sauce" toolkit with millions
> of customizations that is custom built and statically linked, like
> busybox, and to a situation where it's just another set of system
> libraries like any other, shipped by the operating system, like glibc -
> see the threads about stable API/ABI and OS-driven delivery by Ray.
> The status quo of an insanely granular build configuration that means
> everyone is using something different from each other is a bug - not a
> feature. Excessive _build time_ configuration exacerbates and
> encourages this bug.
> 
> Sure, an initial build from scratch will take a couple of minutes more
> - mildly annoying, but worth the price. Caching takes care of that
> problem already pretty well. Also standardizing on radically fewer
> build configurations means you _don't_ have to rebuild DPDK for third
> party testing - you just consume the pre-built binaries from the OS of
> choice, or the PPA or similar for backports and HEAD builds. That will
> save even more time and resources in third-party build systems.
> 

CI systems usually build everything from scratch. So caching doesn't help.
And I don't think that distros will have any pre-built binaries for the
old systems used in public CI systems. For example, TravisCI has 'Trusty'
as a default environment. At most you may use 'Xenial'. But I don't think
that DPDK 18.11 will ever be packaged for them. Also, distros mostly has
dynamic libraries in their packages not providing static ones. It might
be changed in the future, but still we'll not have them for 'xenial'.

So, OVS will stuck with slow building DPDK each time from scratch on Travis.
And developers will maintain local patches for meson for stripping extra
libraries or local build scripts that will package only what they need.

Best regards, Ilya Maximets.

Re: [dpdk-dev] [PATCH 1/2] meson: don't check dependencies for tests if not required

[Ilya Maximets]

On 30.05.2019 14:55, Bruce Richardson wrote:
> On Wed, May 29, 2019 at 07:39:57PM +0300, Ilya Maximets wrote:
>> Don't need to check dependencies if test apps will not be built anyway.
>>
>> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
>> ---
>>  app/test/meson.build | 38 +++++++++++++++++++-------------------
>>  1 file changed, 19 insertions(+), 19 deletions(-)
>>
> Agree with the idea.
> 
> Would this work as a shorter alternative placed at the top of the file?
> 
> if not get_option('tests')
> 	subdir_done()
> endif

This looks good to me.
However, the resulted patch will be much larger because we'll have to
shift most of it to the left. If it's OK, I'll prepare v2 with this change.
What do you think?

Best regards, Ilya Maximets.

Re: [dpdk-dev] [PATCH 1/2] meson: don't check dependencies for tests if not required

[Bruce Richardson]

On Thu, May 30, 2019 at 03:06:17PM +0300, Ilya Maximets wrote:
> On 30.05.2019 14:55, Bruce Richardson wrote:
> > On Wed, May 29, 2019 at 07:39:57PM +0300, Ilya Maximets wrote:
> >> Don't need to check dependencies if test apps will not be built anyway.
> >>
> >> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
> >> ---
> >>  app/test/meson.build | 38 +++++++++++++++++++-------------------
> >>  1 file changed, 19 insertions(+), 19 deletions(-)
> >>
> > Agree with the idea.
> > 
> > Would this work as a shorter alternative placed at the top of the file?
> > 
> > if not get_option('tests')
> > 	subdir_done()
> > endif
> 
> This looks good to me.
> However, the resulted patch will be much larger because we'll have to
> shift most of it to the left. If it's OK, I'll prepare v2 with this change.
> What do you think?
> 
Yes, there will be some left-shifting, but it should just be a single block
from lines 338-419, which is probably ok. The end result is better, I
think.

[dpdk-dev] [PATCH v2] meson: don't check dependencies for tests if not required

[Ilya Maximets]

Don't need to check dependencies if test apps will not be built anyway.

Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
---

Version 2:
  - 'get_option('tests')' check moved to the top.

 app/test/meson.build | 141 ++++++++++++++++++++++---------------------
 1 file changed, 72 insertions(+), 69 deletions(-)

diff --git a/app/test/meson.build b/app/test/meson.build
index 83391cef0..4de856f93 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -1,6 +1,10 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation
 
+if not get_option('tests')
+	subdir_done()
+endif
+
 test_sources = files('commands.c',
 	'packet_burst_generator.c',
 	'sample_packet_forward.c',
@@ -335,86 +339,85 @@ if get_option('default_library') == 'static'
 	link_libs = dpdk_drivers
 endif
 
-if get_option('tests')
-	dpdk_test = executable('dpdk-test',
-		test_sources,
-		link_whole: link_libs,
-		dependencies: test_dep_objs,
-		c_args: [cflags, '-DALLOW_EXPERIMENTAL_API'],
-		install_rpath: driver_install_path,
-		install: false)
+dpdk_test = executable('dpdk-test',
+	test_sources,
+	link_whole: link_libs,
+	dependencies: test_dep_objs,
+	c_args: [cflags, '-DALLOW_EXPERIMENTAL_API'],
+	install_rpath: driver_install_path,
+	install: false)
 
-	# some perf tests (eg: memcpy perf autotest)take very long
-	# to complete, so timeout to 10 minutes
-	timeout_seconds = 600
-	timeout_seconds_fast = 10
-
-	# Retrieve the number of CPU cores, defaulting to 4.
-	num_cores = '0-3'
-	if host_machine.system() == 'linux'
-		num_cores = run_command('cat',
-					'/sys/devices/system/cpu/present'
-				       ).stdout().strip()
-	elif host_machine.system() == 'freebsd'
-		snum_cores = run_command('/sbin/sysctl', '-n',
-					 'hw.ncpu').stdout().strip()
-		inum_cores = snum_cores.to_int() - 1
-                num_cores = '0-@0@'.format(inum_cores)
-	endif
+# some perf tests (eg: memcpy perf autotest)take very long
+# to complete, so timeout to 10 minutes
+timeout_seconds = 600
+timeout_seconds_fast = 10
 
-	num_cores_arg = '-l ' + num_cores
+# Retrieve the number of CPU cores, defaulting to 4.
+num_cores = '0-3'
+if host_machine.system() == 'linux'
+	num_cores = run_command('cat',
+				'/sys/devices/system/cpu/present'
+			       ).stdout().strip()
+elif host_machine.system() == 'freebsd'
+	snum_cores = run_command('/sbin/sysctl', '-n',
+				 'hw.ncpu').stdout().strip()
+	inum_cores = snum_cores.to_int() - 1
+        num_cores = '0-@0@'.format(inum_cores)
+endif
 
-	test_args = [num_cores_arg, '-n 4']
-	foreach arg : fast_parallel_test_names
-		if host_machine.system() == 'linux'
-			test(arg, dpdk_test,
-				  env : ['DPDK_TEST=' + arg],
-				  args : test_args +
-					 ['--file-prefix=@0@'.format(arg)],
-			timeout : timeout_seconds_fast,
-			suite : 'fast-tests')
-		else
-			test(arg, dpdk_test,
-				env : ['DPDK_TEST=' + arg],
-				args : test_args,
-			timeout : timeout_seconds_fast,
-			suite : 'fast-tests')
-		endif
-	endforeach
+num_cores_arg = '-l ' + num_cores
 
-	foreach arg : fast_non_parallel_test_names
+test_args = [num_cores_arg, '-n 4']
+foreach arg : fast_parallel_test_names
+	if host_machine.system() == 'linux'
+		test(arg, dpdk_test,
+			  env : ['DPDK_TEST=' + arg],
+			  args : test_args +
+				 ['--file-prefix=@0@'.format(arg)],
+		timeout : timeout_seconds_fast,
+		suite : 'fast-tests')
+	else
 		test(arg, dpdk_test,
 			env : ['DPDK_TEST=' + arg],
 			args : test_args,
-			timeout : timeout_seconds_fast,
-			is_parallel : false,
-			suite : 'fast-tests')
-	endforeach
+		timeout : timeout_seconds_fast,
+		suite : 'fast-tests')
+	endif
+endforeach
 
-	foreach arg : perf_test_names
-		test(arg, dpdk_test,
+foreach arg : fast_non_parallel_test_names
+	test(arg, dpdk_test,
+		env : ['DPDK_TEST=' + arg],
+		args : test_args,
+		timeout : timeout_seconds_fast,
+		is_parallel : false,
+		suite : 'fast-tests')
+endforeach
+
+foreach arg : perf_test_names
+	test(arg, dpdk_test,
+	env : ['DPDK_TEST=' + arg],
+	args : test_args,
+	timeout : timeout_seconds,
+	is_parallel : false,
+	suite : 'perf-tests')
+endforeach
+
+foreach arg : driver_test_names
+	test(arg, dpdk_test,
 		env : ['DPDK_TEST=' + arg],
 		args : test_args,
 		timeout : timeout_seconds,
 		is_parallel : false,
-		suite : 'perf-tests')
-	endforeach
+		suite : 'driver-tests')
+endforeach
 
-	foreach arg : driver_test_names
-		test(arg, dpdk_test,
-			env : ['DPDK_TEST=' + arg],
-			args : test_args,
-			timeout : timeout_seconds,
-			is_parallel : false,
-			suite : 'driver-tests')
-	endforeach
+foreach arg : dump_test_names
+	test(arg, dpdk_test,
+		env : ['DPDK_TEST=' + arg],
+		args : test_args,
+		timeout : timeout_seconds,
+		is_parallel : false,
+		suite : 'debug-tests')
+endforeach
 
-	foreach arg : dump_test_names
-		test(arg, dpdk_test,
-			env : ['DPDK_TEST=' + arg],
-			args : test_args,
-			timeout : timeout_seconds,
-			is_parallel : false,
-			suite : 'debug-tests')
-	endforeach
-endif
-- 
2.17.1

Re: [dpdk-dev] [PATCH v2] meson: don't check dependencies for tests if not required

[Bruce Richardson]

On Thu, May 30, 2019 at 03:38:36PM +0300, Ilya Maximets wrote:
> Don't need to check dependencies if test apps will not be built anyway.
> 
> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
> ---
> 
> Version 2:
>   - 'get_option('tests')' check moved to the top.
> 
>  app/test/meson.build | 141 ++++++++++++++++++++++---------------------
>  1 file changed, 72 insertions(+), 69 deletions(-)
> 
Acked-by: Bruce Richardson <bruce.richardson@intel.com>

Re: [dpdk-dev] [PATCH v2] meson: don't check dependencies for tests if not required

[Aaron Conole]

Ilya Maximets <i.maximets@samsung.com> writes:

> Don't need to check dependencies if test apps will not be built anyway.
>
> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
> ---
>
> Version 2:
>   - 'get_option('tests')' check moved to the top.
>
>  app/test/meson.build | 141 ++++++++++++++++++++++---------------------
>  1 file changed, 72 insertions(+), 69 deletions(-)
>

Acked-by: Aaron Conole <aconole@redhat.com>

Thanks for this, Ilya!

Re: [dpdk-dev] [PATCH 2/2] meson: make build configurable

[Luca Boccassi]

On Thu, 2019-05-30 at 14:59 +0300, Ilya Maximets wrote:
> On 30.05.2019 14:06, Luca Boccassi wrote:
> > On Thu, 2019-05-30 at 13:03 +0300, Ilya Maximets wrote:
> > > On 29.05.2019 23:37, Luca Boccassi wrote:
> > > > On Wed, 2019-05-29 at 19:39 +0300, Ilya Maximets wrote:
> > > > > The first thing many developers do before start building DPDK
> > > > > is
> > > > > disabling all the not needed divers and libraries. This
> > > > > happens
> > > > > just because more than a half of DPDK dirvers and libraries
> > > > > are
> > > > > not
> > > > > needed for the particular reason. For example, you don't need
> > > > > dpaa*, octeon*, various croypto devices, eventdev, etc. if
> > > > > you're
> > > > > only want to build OVS for x86_64 with static linking.
> > > > > 
> > > > > By disabling everything you don't need, build speeds up
> > > > > literally
> > > > > 10x
> > > > > times. This is important for CI systems. For example,
> > > > > TravisCI
> > > > > wastes
> > > > > 10 minutes for the default DPDK build just to check linking
> > > > > with
> > > > > OVS.
> > > > > 
> > > > > Another thing is the binary size. Number of DPDK libraries
> > > > > and,
> > > > > as a result, size of resulted statically linked application
> > > > > decreases
> > > > > significantly.
> > > > > 
> > > > > Important thing also that you're able to not install some
> > > > > dependencies
> > > > > if you don't have them on a target platform. Just disable
> > > > > libs/drivers
> > > > > that depends on it. Similar thing for the glibc version
> > > > > mismatch
> > > > > between build and target platforms.
> > > > > 
> > > > > Also, I have to note that less code means less probability of
> > > > > failures and less number of attack vectors.
> > > > > 
> > > > > This patch gives 'meson' the power of configurability that we
> > > > > have with 'make'. Using new options it's possible to enable
> > > > > just
> > > > > what you need and nothing more.
> > > > > 
> > > > > For example, following cmdline could be used to build almost
> > > > > minimal
> > > > > set of DPDK libs and drivers to check OVS build:
> > > > > 
> > > > >   $ meson build -Dexamples='' -Dtests=false
> > > > > -Denable_kmods=false
> > > > > \
> > > > >                 -Ddrivers_bus=pci,vdev          \
> > > > >                 -Ddrivers_mempool=ring          \
> > > > >                 -Ddrivers_net=null,virtio,ring  \
> > > > >                 -Ddrivers_crypto=virtio         \
> > > > >                 -Ddrivers_compress=none         \
> > > > >                 -Ddrivers_event=none            \
> > > > >                 -Ddrivers_baseband=none         \
> > > > >                 -Ddrivers_raw=none              \
> > > > >                 -Ddrivers_common=none           \
> > > > >                
> > > > > -Dlibs=kvargs,eal,cmdline,ring,mempool,mbuf,net,meter,\
> > > > >                        ethdev,pci,hash,cryptodev,pdump,vhost
> > > > > \
> > > > >                 -Dapps=none
> > > > > 
> > > > > Adding a few real net drivers will give configuration that
> > > > > can be
> > > > > used
> > > > > in production environment.
> > > > > 
> > > > > Looks not very pretty, but this could be moved to a script.
> > > > > 
> > > > > Build details:
> > > > > 
> > > > >   Build targets in project: 57
> > > > > 
> > > > >   $ time ninja
> > > > >   real    0m11,528s
> > > > >   user    1m4,137s
> > > > >   sys     0m4,935s
> > > > > 
> > > > >   $ du -sh ../dpdk_meson_install/
> > > > >   3,5M    ../dpdk_meson_install/
> > > > > 
> > > > > To compare with what we have without these options:
> > > > > 
> > > > >   $ meson build -Dexamples='' -Dtests=false
> > > > > -Denable_kmods=false
> > > > >   Build targets in project: 434
> > > > > 
> > > > >   $ time ninja
> > > > >   real    1m38,963s
> > > > >   user    10m18,624s
> > > > >   sys     0m45,478s
> > > > > 
> > > > >   $ du -sh ../dpdk_meson_install/
> > > > >   27M     ../dpdk_meson_install/
> > > > > 
> > > > > 10x speed up for the user time.
> > > > > 7.7 times size decrease.
> > > > > 
> > > > > This is probably not much user-friendly because it's not a
> > > > > Kconfig
> > > > > and dependency tracking in meson is really poor, so it
> > > > > requires
> > > > > usually few iterations to pick correct set of libraries to
> > > > > satisfy
> > > > > all dependencies. However, it's not a big deal. Options
> > > > > intended
> > > > > for a proficient users who knows what they need.
> > > > 
> > > > Hi,
> > > > 
> > > > We talked about this a few times in the past, and it was
> > > > actually
> > > > one
> > > > of the design goals to _avoid_ replicating the octopus-like
> > > > config
> > > > system of the makefiles. That's because it makes the test
> > > > matrix
> > > > insanely complicated, not to mention the harm to user
> > > > friendliness,
> > > > among other things.
> > > > 
> > > > If someone doesn't want to use a PMD, they can just avoid
> > > > installing it
> > > > - it's simple enough.
> > > 
> > > So how can I do this? I don't think 'ninja install' has such
> > > option.
> > > Also, if you think that it is safe to skip some libs/drivers in
> > > installation
> > > process, it must be safe to not build them at all. It's just a
> > > waste
> > > of
> > > time and computational resources to build something known to be
> > > not
> > > used.
> > > And if you're going to ship DPDK libraries separately in distros,
> > > you'll
> > > have to test their different combinations anyway. If they're so
> > > independent
> > > that you don't need to test them in various combinations, than
> > > your
> > > point
> > > about test matrix is not valid.
> > 
> > It can be done in the packaging step, or post-install if there's no
> > packaging. An operating system vendor is free to do its own test
> > and
> > support plan, and decide to leave out some PMDs from it.
> 
> This technically means doing this manually/write custom scripts.

It's a standard part of packaging software, both in RPM and DEB - not
too familiar with AUR, but I suspect it's not too different in that
regard.

> > Canonical does
> > something similar: in Debian/Ubuntu and derivatives we package PMDs
> > individually, and then Canonical groups them in 2 sets - a subset
> > they
> > guarantee to support from their own resources, and the full set as
> > delivered by the community. But the point is that it's a step that
> > they
> > decide to take, and pay the price for it in terms of time
> > investment in
> > validating that particular combination, rather than the onus being
> > on
> > our very limited and already stretched resources to validate all
> > combinations.
> > 
> > We can focus our resources into making sure the few combinations
> > that
> > _must_ be supported, for example due to external dependencies, work
> > fine.
> > 
> > > > Sorry, but from me it's a very strong NACK.
> > > 
> > > Sorry, but let me disagree with you. For me, meson
> > > configurability is
> > > the
> > > essential thing to have in terms of deprecating the 'make' build
> > > system.
> > > DPDK was and keeps being (in most cases) the library that users
> > > statically
> > > linking to a single application built for particular platform and
> > > not
> > > using
> > > for anything else. This means that user in most cases knows which
> > > parts
> > > needed and which parts will never be used. Current meson build
> > > system
> > > doesn't allow to disable anything forcing users to link with the
> > > whole bunch
> > > of unused code.
> > > 
> > > One major case is that you have to have build environment equal
> > > to
> > > your
> > > target platform in terms of availability of external libraries.
> > > So,
> > > if I
> > > have some external library on build system, meson will build all
> > > the
> > > modules
> > > it depends from and will link them to my application. As a result
> > > I'll not
> > > be able to run my application on a target platform without
> > > installing
> > > additional dependencies which is not acceptable. This patch will
> > > allow to
> > > specifically disable all the libs that has unsatisfiable
> > > dependencies
> > > on
> > > target. Without the patch it's required to manually remove
> > > resulted
> > > libs and
> > > fix pkg-config and stuff before building apps. This is far less
> > > user-
> > > friendly
> > > than options I proposed. And yes, I still have to waste time for
> > > building
> > > libraries I'll remove right after.
> > > 
> > > While testing OVS on TravisCI, DPDK was built far more than 30K
> > > times
> > > which is more than half of a year of a wasted computational
> > > resources
> > > (if we'll count 10 minutes per build).
> > > I think this time could be used more wisely.
> > > 
> > > Best regards, Ilya Maximets.
> > 
> > But that's the thing: as it was discussed recently, we need to move
> > away from DPDK being by default a "special sauce" toolkit with
> > millions
> > of customizations that is custom built and statically linked, like
> > busybox, and to a situation where it's just another set of system
> > libraries like any other, shipped by the operating system, like
> > glibc -
> > see the threads about stable API/ABI and OS-driven delivery by Ray.
> > The status quo of an insanely granular build configuration that
> > means
> > everyone is using something different from each other is a bug -
> > not a
> > feature. Excessive _build time_ configuration exacerbates and
> > encourages this bug.
> > 
> > Sure, an initial build from scratch will take a couple of minutes
> > more
> > - mildly annoying, but worth the price. Caching takes care of that
> > problem already pretty well. Also standardizing on radically fewer
> > build configurations means you _don't_ have to rebuild DPDK for
> > third
> > party testing - you just consume the pre-built binaries from the OS
> > of
> > choice, or the PPA or similar for backports and HEAD builds. That
> > will
> > save even more time and resources in third-party build systems.
> > 
> 
> CI systems usually build everything from scratch. So caching doesn't
> help.
> And I don't think that distros will have any pre-built binaries for
> the
> old systems used in public CI systems. For example, TravisCI has
> 'Trusty'
> as a default environment. At most you may use 'Xenial'. But I don't
> think
> that DPDK 18.11 will ever be packaged for them. Also, distros mostly
> has
> dynamic libraries in their packages not providing static ones. It
> might
> be changed in the future, but still we'll not have them for 'xenial'.
> 
> So, OVS will stuck with slow building DPDK each time from scratch on
> Travis.
> And developers will maintain local patches for meson for stripping
> extra
> libraries or local build scripts that will package only what they
> need.
> 
> Best regards, Ilya Maximets.

Well, I'm not sure most CI systems build everything from scratch, and
even if they do, ccache is supported natively on Travis and others - we
use it for DPDK's Travis config, for example. I've yet to see a Travis
configuration that rebuilds the kernel, glibc, compiler and every
single other dependency of a project from scratch every time, though :-
)
In fact, Travis has native support for APT repositories and even PPAs
to do the opposite, and it's quite popular as far as I know.

Making a PPA or a project on OBS available with repositories for older
LTS releases is not only possible, but we already do it today - it's
based mostly on request, so for example even though Xenial shipped with
DPDk 2.2, a 17.11 PPA is available and maintained already, and I'm sure
if we ask very nicely 18.11 can be made available too :-)

I'm not sure about the RPM world, but in Debian, Ubuntu and derivatives
we ship both static and dynamic libraries by default, and that includes
DPDK from day 1.

So I can assure you I simpathize with the idea of making CIs faster,
and there are many ways to achieve that, that don't require manually
hacking the dependencies. For example, I've set up the ZeroMQ CI so
that all dependencies from the project and backports for Xenial are
built on OBS just as they would be built by the distro, with no
compromises or local just-in-time "hacks" to speed up individual
builds, so that binary packages are available and installed by Travis
via APT on each build. That takes less than 30s regardless of the
number of packages (within reason), so whatever we add as a dependency
it's still the same.

I had done that even for Trusty and Pristine at the time:

https://build.opensuse.org/project/monitor/network:messaging:zeromq:git-stable?arch_x86_64=1&defaults=0&repo_xUbuntu_14_04=1&succeeded=1
https://build.opensuse.org/project/monitor/network:messaging:zeromq:git-stable?arch_x86_64=1&defaults=0&repo_xUbuntu_12_04=1&succeeded=1

Here's a CI job that uses the Trusty repository:

https://travis-ci.org/zeromq/czmq/jobs/538626364

-- 
Kind regards,
Luca Boccassi

Re: [dpdk-dev] [PATCH v2] meson: don't check dependencies for tests if not required

[Luca Boccassi]

On Thu, 2019-05-30 at 15:38 +0300, Ilya Maximets wrote:
> Don't need to check dependencies if test apps will not be built
> anyway.
> 
> Signed-off-by: Ilya Maximets <
> i.maximets@samsung.com
> >
> ---
> 
> Version 2:
>   - 'get_option('tests')' check moved to the top.
> 
>  app/test/meson.build | 141 ++++++++++++++++++++++-------------------
> --
>  1 file changed, 72 insertions(+), 69 deletions(-)

Acked-by: Luca Boccassi <bluca@debian.org>

-- 
Kind regards,
Luca Boccassi

Re: [dpdk-dev] [PATCH v2] meson: don't check dependencies for tests if not required

[Thomas Monjalon]

30/05/2019 14:48, Bruce Richardson:
> On Thu, May 30, 2019 at 03:38:36PM +0300, Ilya Maximets wrote:
> > Don't need to check dependencies if test apps will not be built anyway.
> > 
> > Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Aaron Conole <aconole@redhat.com>
> Acked-by: Luca Boccassi <bluca@debian.org>

Applied, thanks