Re: [dpdk-dev] [PATCH 2/2] meson: make build configurable

[Ilya Maximets <i.maximets@samsung.com>]

On 29.05.2019 23:37, Luca Boccassi wrote:
> On Wed, 2019-05-29 at 19:39 +0300, Ilya Maximets wrote:
>> The first thing many developers do before start building DPDK is
>> disabling all the not needed divers and libraries. This happens
>> just because more than a half of DPDK dirvers and libraries are not
>> needed for the particular reason. For example, you don't need
>> dpaa*, octeon*, various croypto devices, eventdev, etc. if you're
>> only want to build OVS for x86_64 with static linking.
>>
>> By disabling everything you don't need, build speeds up literally 10x
>> times. This is important for CI systems. For example, TravisCI wastes
>> 10 minutes for the default DPDK build just to check linking with OVS.
>>
>> Another thing is the binary size. Number of DPDK libraries and,
>> as a result, size of resulted statically linked application decreases
>> significantly.
>>
>> Important thing also that you're able to not install some
>> dependencies
>> if you don't have them on a target platform. Just disable
>> libs/drivers
>> that depends on it. Similar thing for the glibc version mismatch
>> between build and target platforms.
>>
>> Also, I have to note that less code means less probability of
>> failures and less number of attack vectors.
>>
>> This patch gives 'meson' the power of configurability that we
>> have with 'make'. Using new options it's possible to enable just
>> what you need and nothing more.
>>
>> For example, following cmdline could be used to build almost minimal
>> set of DPDK libs and drivers to check OVS build:
>>
>>   $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false \
>>                 -Ddrivers_bus=pci,vdev          \
>>                 -Ddrivers_mempool=ring          \
>>                 -Ddrivers_net=null,virtio,ring  \
>>                 -Ddrivers_crypto=virtio         \
>>                 -Ddrivers_compress=none         \
>>                 -Ddrivers_event=none            \
>>                 -Ddrivers_baseband=none         \
>>                 -Ddrivers_raw=none              \
>>                 -Ddrivers_common=none           \
>>                
>> -Dlibs=kvargs,eal,cmdline,ring,mempool,mbuf,net,meter,\
>>                        ethdev,pci,hash,cryptodev,pdump,vhost \
>>                 -Dapps=none
>>
>> Adding a few real net drivers will give configuration that can be
>> used
>> in production environment.
>>
>> Looks not very pretty, but this could be moved to a script.
>>
>> Build details:
>>
>>   Build targets in project: 57
>>
>>   $ time ninja
>>   real    0m11,528s
>>   user    1m4,137s
>>   sys     0m4,935s
>>
>>   $ du -sh ../dpdk_meson_install/
>>   3,5M    ../dpdk_meson_install/
>>
>> To compare with what we have without these options:
>>
>>   $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false
>>   Build targets in project: 434
>>
>>   $ time ninja
>>   real    1m38,963s
>>   user    10m18,624s
>>   sys     0m45,478s
>>
>>   $ du -sh ../dpdk_meson_install/
>>   27M     ../dpdk_meson_install/
>>
>> 10x speed up for the user time.
>> 7.7 times size decrease.
>>
>> This is probably not much user-friendly because it's not a Kconfig
>> and dependency tracking in meson is really poor, so it requires
>> usually few iterations to pick correct set of libraries to satisfy
>> all dependencies. However, it's not a big deal. Options intended
>> for a proficient users who knows what they need.
> 
> Hi,
> 
> We talked about this a few times in the past, and it was actually one
> of the design goals to _avoid_ replicating the octopus-like config
> system of the makefiles. That's because it makes the test matrix
> insanely complicated, not to mention the harm to user friendliness,
> among other things.
> 
> If someone doesn't want to use a PMD, they can just avoid installing it
> - it's simple enough.

So how can I do this? I don't think 'ninja install' has such option.
Also, if you think that it is safe to skip some libs/drivers in installation
process, it must be safe to not build them at all. It's just a waste of
time and computational resources to build something known to be not used.
And if you're going to ship DPDK libraries separately in distros, you'll
have to test their different combinations anyway. If they're so independent
that you don't need to test them in various combinations, than your point
about test matrix is not valid.

> 
> Sorry, but from me it's a very strong NACK.

Sorry, but let me disagree with you. For me, meson configurability is the
essential thing to have in terms of deprecating the 'make' build system.
DPDK was and keeps being (in most cases) the library that users statically
linking to a single application built for particular platform and not using
for anything else. This means that user in most cases knows which parts
needed and which parts will never be used. Current meson build system
doesn't allow to disable anything forcing users to link with the whole bunch
of unused code.

One major case is that you have to have build environment equal to your
target platform in terms of availability of external libraries. So, if I
have some external library on build system, meson will build all the modules
it depends from and will link them to my application. As a result I'll not
be able to run my application on a target platform without installing
additional dependencies which is not acceptable. This patch will allow to
specifically disable all the libs that has unsatisfiable dependencies on
target. Without the patch it's required to manually remove resulted libs and
fix pkg-config and stuff before building apps. This is far less user-friendly
than options I proposed. And yes, I still have to waste time for building
libraries I'll remove right after.

While testing OVS on TravisCI, DPDK was built far more than 30K times
which is more than half of a year of a wasted computational resources
(if we'll count 10 minutes per build).
I think this time could be used more wisely.

Best regards, Ilya Maximets.