Re: [dpdk-dev] [PATCH 2/2] meson: make build configurable

[Ilya Maximets <i.maximets@samsung.com>]

On 29.05.2019 23:15, Michael Santana Francisco wrote:
> On 5/29/19 12:39 PM, Ilya Maximets wrote:
>> The first thing many developers do before start building DPDK is
>> disabling all the not needed divers and libraries. This happens
>> just because more than a half of DPDK dirvers and libraries are not
>> needed for the particular reason. For example, you don't need
>> dpaa*, octeon*, various croypto devices, eventdev, etc. if you're
>> only want to build OVS for x86_64 with static linking.
>>
>> By disabling everything you don't need, build speeds up literally 10x
>> times. This is important for CI systems. For example, TravisCI wastes
>> 10 minutes for the default DPDK build just to check linking with OVS.
>>
>> Another thing is the binary size. Number of DPDK libraries and,
>> as a result, size of resulted statically linked application decreases
>> significantly.
>>
>> Important thing also that you're able to not install some dependencies
>> if you don't have them on a target platform. Just disable libs/drivers
>> that depends on it. Similar thing for the glibc version mismatch
>> between build and target platforms.
>>
>> Also, I have to note that less code means less probability of
>> failures and less number of attack vectors.
>>
>> This patch gives 'meson' the power of configurability that we
>> have with 'make'. Using new options it's possible to enable just
>> what you need and nothing more.
>>
>> For example, following cmdline could be used to build almost minimal
>> set of DPDK libs and drivers to check OVS build:
>>
>>   $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false \
>>                 -Ddrivers_bus=pci,vdev          \
>>                 -Ddrivers_mempool=ring          \
>>                 -Ddrivers_net=null,virtio,ring  \
>>                 -Ddrivers_crypto=virtio         \
>>                 -Ddrivers_compress=none         \
>>                 -Ddrivers_event=none            \
>>                 -Ddrivers_baseband=none         \
>>                 -Ddrivers_raw=none              \
>>                 -Ddrivers_common=none           \
>>                 -Dlibs=kvargs,eal,cmdline,ring,mempool,mbuf,net,meter,\
>>                        ethdev,pci,hash,cryptodev,pdump,vhost \
>>                 -Dapps=none
>>
>> Adding a few real net drivers will give configuration that can be used
>> in production environment.
>>
>> Looks not very pretty, but this could be moved to a script.
>>
>> Build details:
>>
>>   Build targets in project: 57
>>
>>   $ time ninja
>>   real    0m11,528s
>>   user    1m4,137s
>>   sys     0m4,935s
>>
>>   $ du -sh ../dpdk_meson_install/
>>   3,5M    ../dpdk_meson_install/
>>
>> To compare with what we have without these options:
>>
>>   $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false
>>   Build targets in project: 434
>>
>>   $ time ninja
>>   real    1m38,963s
>>   user    10m18,624s
>>   sys     0m45,478s
>>
>>   $ du -sh ../dpdk_meson_install/
>>   27M     ../dpdk_meson_install/
>>
>> 10x speed up for the user time.
>> 7.7 times size decrease.
>>
>> This is probably not much user-friendly because it's not a Kconfig
>> and dependency tracking in meson is really poor, so it requires
>> usually few iterations to pick correct set of libraries to satisfy
>> all dependencies. However, it's not a big deal. Options intended
>> for a proficient users who knows what they need.
>>
>> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
>> ---
>>  app/meson.build              |  5 +++++
>>  drivers/baseband/meson.build |  5 +++++
>>  drivers/bus/meson.build      |  6 ++++++
>>  drivers/common/meson.build   |  6 ++++++
>>  drivers/compress/meson.build |  5 +++++
>>  drivers/crypto/meson.build   |  5 +++++
>>  drivers/event/meson.build    |  6 ++++++
>>  drivers/mempool/meson.build  |  6 ++++++
>>  drivers/net/meson.build      |  6 ++++++
>>  drivers/raw/meson.build      |  6 ++++++
>>  lib/meson.build              |  5 +++++
>>  meson_options.txt            | 22 ++++++++++++++++++++++
>>  12 files changed, 83 insertions(+)
>>
>> diff --git a/app/meson.build b/app/meson.build
>> index 2b9fdef74..48972954c 100644
>> --- a/app/meson.build
>> +++ b/app/meson.build
>> @@ -17,6 +17,11 @@ apps = [
>>  	'test-pipeline',
>>  	'test-pmd']
>>  
>> +enabled_apps = get_option('apps')
>> +if enabled_apps != 'all'
>> +	apps = (enabled_apps == 'none') ? [] : enabled_apps.split(',')
>> +endif
>> +
>>  # for BSD only
>>  lib_execinfo = cc.find_library('execinfo', required: false)
>>  
>> diff --git a/drivers/baseband/meson.build b/drivers/baseband/meson.build
>> index 52489df35..fabc80fc2 100644
>> --- a/drivers/baseband/meson.build
>> +++ b/drivers/baseband/meson.build
>> @@ -3,5 +3,10 @@
>>  
>>  drivers = ['null']
>>  
>> +enabled_drivers = get_option('drivers_baseband')
>> +if enabled_drivers != 'all'
>> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
>> +endif
>> +
>>  config_flag_fmt = 'RTE_LIBRTE_@0@_PMD'
>>  driver_name_fmt = 'rte_pmd_@0@'
>> diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
>> index 80de2d91d..d1400ff28 100644
>> --- a/drivers/bus/meson.build
>> +++ b/drivers/bus/meson.build
>> @@ -2,6 +2,12 @@
>>  # Copyright(c) 2017 Intel Corporation
>>  
>>  drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus']
>> +
>> +enabled_drivers = get_option('drivers_bus')
>> +if enabled_drivers != 'all'
>> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
>> +endif
>> +
>>  std_deps = ['eal']
>>  config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
>>  driver_name_fmt = 'rte_bus_@0@'
>> diff --git a/drivers/common/meson.build b/drivers/common/meson.build
>> index a50934108..311511365 100644
>> --- a/drivers/common/meson.build
>> +++ b/drivers/common/meson.build
>> @@ -3,5 +3,11 @@
>>  
>>  std_deps = ['eal']
>>  drivers = ['cpt', 'dpaax', 'mvep', 'octeontx', 'qat']
>> +
>> +enabled_drivers = get_option('drivers_common')
>> +if enabled_drivers != 'all'
>> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
>> +endif
>> +
>>  config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON'
>>  driver_name_fmt = 'rte_common_@0@'
>> diff --git a/drivers/compress/meson.build b/drivers/compress/meson.build
>> index 817ef3be4..5585ec537 100644
>> --- a/drivers/compress/meson.build
>> +++ b/drivers/compress/meson.build
>> @@ -3,6 +3,11 @@
>>  
>>  drivers = ['isal', 'octeontx', 'qat', 'zlib']
>>  
>> +enabled_drivers = get_option('drivers_compress')
>> +if enabled_drivers != 'all'
>> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
>> +endif
>> +
>>  std_deps = ['compressdev'] # compressdev pulls in all other needed deps
>>  config_flag_fmt = 'RTE_LIBRTE_@0@_PMD'
>>  driver_name_fmt = 'rte_pmd_@0@'
>> diff --git a/drivers/crypto/meson.build b/drivers/crypto/meson.build
>> index 83e78860e..7b1363f23 100644
>> --- a/drivers/crypto/meson.build
>> +++ b/drivers/crypto/meson.build
>> @@ -5,6 +5,11 @@ drivers = ['aesni_gcm', 'aesni_mb', 'caam_jr', 'ccp', 'dpaa_sec', 'dpaa2_sec',
>>  	'kasumi', 'mvsam', 'null', 'octeontx', 'openssl', 'qat', 'scheduler',
>>  	'snow3g', 'virtio', 'zuc']
>>  
>> +enabled_drivers = get_option('drivers_crypto')
>> +if enabled_drivers != 'all'
>> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
>> +endif
>> +
>>  std_deps = ['cryptodev'] # cryptodev pulls in all other needed deps
>>  config_flag_fmt = 'RTE_LIBRTE_@0@_PMD'
>>  driver_name_fmt = 'rte_pmd_@0@'
>> diff --git a/drivers/event/meson.build b/drivers/event/meson.build
>> index fb723f727..14e0c68ac 100644
>> --- a/drivers/event/meson.build
>> +++ b/drivers/event/meson.build
>> @@ -6,6 +6,12 @@ if not (toolchain == 'gcc' and cc.version().version_compare('<4.8.6') and
>>  	dpdk_conf.has('RTE_ARCH_ARM64'))
>>  	drivers += 'octeontx'
>>  endif
>> +
>> +enabled_drivers = get_option('drivers_event')
>> +if enabled_drivers != 'all'
>> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
>> +endif
>> +
>>  std_deps = ['eventdev', 'kvargs']
>>  config_flag_fmt = 'RTE_LIBRTE_@0@_EVENTDEV_PMD'
>>  driver_name_fmt = 'rte_pmd_@0@_event'
>> diff --git a/drivers/mempool/meson.build b/drivers/mempool/meson.build
>> index 4527d9806..cfacedd1b 100644
>> --- a/drivers/mempool/meson.build
>> +++ b/drivers/mempool/meson.build
>> @@ -2,6 +2,12 @@
>>  # Copyright(c) 2017 Intel Corporation
>>  
>>  drivers = ['bucket', 'dpaa', 'dpaa2', 'octeontx', 'ring', 'stack']
>> +
>> +enabled_drivers = get_option('drivers_mempool')
>> +if enabled_drivers != 'all'
>> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
>> +endif
>> +
>>  std_deps = ['mempool']
>>  config_flag_fmt = 'RTE_LIBRTE_@0@_MEMPOOL'
>>  driver_name_fmt = 'rte_mempool_@0@'
>> diff --git a/drivers/net/meson.build b/drivers/net/meson.build
>> index ed99896c3..901be5eac 100644
>> --- a/drivers/net/meson.build
>> +++ b/drivers/net/meson.build
>> @@ -42,6 +42,12 @@ drivers = ['af_packet',
>>  	'virtio',
>>  	'vmxnet3',
>>  ]
>> +
>> +enabled_drivers = get_option('drivers_net')
>> +if enabled_drivers != 'all'
>> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
>> +endif
>> +
>>  std_deps = ['ethdev', 'kvargs'] # 'ethdev' also pulls in mbuf, net, eal etc
>>  std_deps += ['bus_pci']         # very many PMDs depend on PCI, so make std
>>  std_deps += ['bus_vdev']        # same with vdev bus
>> diff --git a/drivers/raw/meson.build b/drivers/raw/meson.build
>> index a61cdccef..ca98e92f0 100644
>> --- a/drivers/raw/meson.build
>> +++ b/drivers/raw/meson.build
>> @@ -2,6 +2,12 @@
>>  # Copyright 2018 NXP
>>  
>>  drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev']
>> +
>> +enabled_drivers = get_option('drivers_raw')
>> +if enabled_drivers != 'all'
>> +	drivers = (enabled_drivers == 'none') ? [] : enabled_drivers.split(',')
>> +endif
>> +
>>  std_deps = ['rawdev']
>>  config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV'
>>  driver_name_fmt = 'rte_pmd_@0@'
>> diff --git a/lib/meson.build b/lib/meson.build
>> index e067ce5ea..384867926 100644
>> --- a/lib/meson.build
>> +++ b/lib/meson.build
>> @@ -34,6 +34,11 @@ if is_windows
>>  	libraries = ['kvargs','eal'] # only supported libraries for windows
>>  endif
>>  
>> +enabled_libs = get_option('libs')
>> +if enabled_libs != 'all'
>> +	libraries = (enabled_libs == 'none') ? [] : enabled_libs.split(',')
>> +endif
>> +
>>  default_cflags = machine_args
>>  if cc.has_argument('-Wno-format-truncation')
>>  	default_cflags += '-Wno-format-truncation'
>> diff --git a/meson_options.txt b/meson_options.txt
>> index 16d9f92c6..5184917b0 100644
>> --- a/meson_options.txt
>> +++ b/meson_options.txt
>> @@ -2,6 +2,26 @@
>>  
>>  option('allow_invalid_socket_id', type: 'boolean', value: false,
>>  	description: 'allow out-of-range NUMA socket id\'s for platforms that don\'t report the value correctly')
>> +option('apps', type: 'string', value: 'all',
>> +	description: 'List of applications to build. Defaults to all')
>> +option('drivers_baseband', type: 'string', value: 'all',
>> +	description: 'List of baseband drivers to build. Defaults to all')
>> +option('drivers_bus', type: 'string', value: 'all',
>> +	description: 'List of bus drivers to build. Defaults to all')
>> +option('drivers_common', type: 'string', value: 'all',
>> +	description: 'List of common drivers to build. Defaults to all')
>> +option('drivers_compress', type: 'string', value: 'all',
>> +	description: 'List of compress drivers to build. Defaults to all')
>> +option('drivers_crypto', type: 'string', value: 'all',
>> +	description: 'List of crypto drivers to build. Defaults to all')
>> +option('drivers_event', type: 'string', value: 'all',
>> +	description: 'List of event drivers to build. Defaults to all')
>> +option('drivers_mempool', type: 'string', value: 'all',
>> +	description: 'List of mempool drivers to build. Defaults to all')
>> +option('drivers_net', type: 'string', value: 'all',
>> +	description: 'List of net drivers to build. Defaults to all')
>> +option('drivers_raw', type: 'string', value: 'all',
>> +	description: 'List of rawdev drivers to build. Defaults to all')
>>  option('drivers_install_subdir', type: 'string', value: 'dpdk/pmds-<VERSION>',
>>  	description: 'Subdirectory of libdir where to install PMDs. Defaults to using a versioned subdirectory.')
>>  option('enable_docs', type: 'boolean', value: false,
>> @@ -16,6 +36,8 @@ option('include_subdir_arch', type: 'string', value: '',
>>  	description: 'subdirectory where to install arch-dependent headers')
>>  option('kernel_dir', type: 'string', value: '',
>>  	description: 'path to the kernel for building kernel modules, they will be installed in $DEST_DIR/$kernel_dir/../extra/dpdk')
>> +option('libs', type: 'string', value: 'all',
>> +	description: 'List of libraries to build. Defaults to all')
>>  option('lib_musdk_dir', type: 'string', value: '',
>>  	description: 'path to the MUSDK library installation directory')
>>  option('machine', type: 'string', value: 'native',
> 
> I think this patch is the right step forward. This patch will speed up travis builds which is a big plus
> 
> Just one small comment, could you document/add some build examples to doc/build-sdk-meson.txt and other relevant doc files? Thanks
> 
> Acked-by: Michael Santana <msantana@redhat.com>
> 

Thanks. Will add some docs for the next revisions.

Best regards, Ilya Maximets.