Re: [dpdk-dev] [PATCH 2/2] meson: make build configurable

[Luca Boccassi <bluca@debian.org>]

On Thu, 2019-05-30 at 14:59 +0300, Ilya Maximets wrote:
> On 30.05.2019 14:06, Luca Boccassi wrote:
> > On Thu, 2019-05-30 at 13:03 +0300, Ilya Maximets wrote:
> > > On 29.05.2019 23:37, Luca Boccassi wrote:
> > > > On Wed, 2019-05-29 at 19:39 +0300, Ilya Maximets wrote:
> > > > > The first thing many developers do before start building DPDK
> > > > > is
> > > > > disabling all the not needed divers and libraries. This
> > > > > happens
> > > > > just because more than a half of DPDK dirvers and libraries
> > > > > are
> > > > > not
> > > > > needed for the particular reason. For example, you don't need
> > > > > dpaa*, octeon*, various croypto devices, eventdev, etc. if
> > > > > you're
> > > > > only want to build OVS for x86_64 with static linking.
> > > > > 
> > > > > By disabling everything you don't need, build speeds up
> > > > > literally
> > > > > 10x
> > > > > times. This is important for CI systems. For example,
> > > > > TravisCI
> > > > > wastes
> > > > > 10 minutes for the default DPDK build just to check linking
> > > > > with
> > > > > OVS.
> > > > > 
> > > > > Another thing is the binary size. Number of DPDK libraries
> > > > > and,
> > > > > as a result, size of resulted statically linked application
> > > > > decreases
> > > > > significantly.
> > > > > 
> > > > > Important thing also that you're able to not install some
> > > > > dependencies
> > > > > if you don't have them on a target platform. Just disable
> > > > > libs/drivers
> > > > > that depends on it. Similar thing for the glibc version
> > > > > mismatch
> > > > > between build and target platforms.
> > > > > 
> > > > > Also, I have to note that less code means less probability of
> > > > > failures and less number of attack vectors.
> > > > > 
> > > > > This patch gives 'meson' the power of configurability that we
> > > > > have with 'make'. Using new options it's possible to enable
> > > > > just
> > > > > what you need and nothing more.
> > > > > 
> > > > > For example, following cmdline could be used to build almost
> > > > > minimal
> > > > > set of DPDK libs and drivers to check OVS build:
> > > > > 
> > > > >   $ meson build -Dexamples='' -Dtests=false
> > > > > -Denable_kmods=false
> > > > > \
> > > > >                 -Ddrivers_bus=pci,vdev          \
> > > > >                 -Ddrivers_mempool=ring          \
> > > > >                 -Ddrivers_net=null,virtio,ring  \
> > > > >                 -Ddrivers_crypto=virtio         \
> > > > >                 -Ddrivers_compress=none         \
> > > > >                 -Ddrivers_event=none            \
> > > > >                 -Ddrivers_baseband=none         \
> > > > >                 -Ddrivers_raw=none              \
> > > > >                 -Ddrivers_common=none           \
> > > > >                
> > > > > -Dlibs=kvargs,eal,cmdline,ring,mempool,mbuf,net,meter,\
> > > > >                        ethdev,pci,hash,cryptodev,pdump,vhost
> > > > > \
> > > > >                 -Dapps=none
> > > > > 
> > > > > Adding a few real net drivers will give configuration that
> > > > > can be
> > > > > used
> > > > > in production environment.
> > > > > 
> > > > > Looks not very pretty, but this could be moved to a script.
> > > > > 
> > > > > Build details:
> > > > > 
> > > > >   Build targets in project: 57
> > > > > 
> > > > >   $ time ninja
> > > > >   real    0m11,528s
> > > > >   user    1m4,137s
> > > > >   sys     0m4,935s
> > > > > 
> > > > >   $ du -sh ../dpdk_meson_install/
> > > > >   3,5M    ../dpdk_meson_install/
> > > > > 
> > > > > To compare with what we have without these options:
> > > > > 
> > > > >   $ meson build -Dexamples='' -Dtests=false
> > > > > -Denable_kmods=false
> > > > >   Build targets in project: 434
> > > > > 
> > > > >   $ time ninja
> > > > >   real    1m38,963s
> > > > >   user    10m18,624s
> > > > >   sys     0m45,478s
> > > > > 
> > > > >   $ du -sh ../dpdk_meson_install/
> > > > >   27M     ../dpdk_meson_install/
> > > > > 
> > > > > 10x speed up for the user time.
> > > > > 7.7 times size decrease.
> > > > > 
> > > > > This is probably not much user-friendly because it's not a
> > > > > Kconfig
> > > > > and dependency tracking in meson is really poor, so it
> > > > > requires
> > > > > usually few iterations to pick correct set of libraries to
> > > > > satisfy
> > > > > all dependencies. However, it's not a big deal. Options
> > > > > intended
> > > > > for a proficient users who knows what they need.
> > > > 
> > > > Hi,
> > > > 
> > > > We talked about this a few times in the past, and it was
> > > > actually
> > > > one
> > > > of the design goals to _avoid_ replicating the octopus-like
> > > > config
> > > > system of the makefiles. That's because it makes the test
> > > > matrix
> > > > insanely complicated, not to mention the harm to user
> > > > friendliness,
> > > > among other things.
> > > > 
> > > > If someone doesn't want to use a PMD, they can just avoid
> > > > installing it
> > > > - it's simple enough.
> > > 
> > > So how can I do this? I don't think 'ninja install' has such
> > > option.
> > > Also, if you think that it is safe to skip some libs/drivers in
> > > installation
> > > process, it must be safe to not build them at all. It's just a
> > > waste
> > > of
> > > time and computational resources to build something known to be
> > > not
> > > used.
> > > And if you're going to ship DPDK libraries separately in distros,
> > > you'll
> > > have to test their different combinations anyway. If they're so
> > > independent
> > > that you don't need to test them in various combinations, than
> > > your
> > > point
> > > about test matrix is not valid.
> > 
> > It can be done in the packaging step, or post-install if there's no
> > packaging. An operating system vendor is free to do its own test
> > and
> > support plan, and decide to leave out some PMDs from it.
> 
> This technically means doing this manually/write custom scripts.

It's a standard part of packaging software, both in RPM and DEB - not
too familiar with AUR, but I suspect it's not too different in that
regard.

> > Canonical does
> > something similar: in Debian/Ubuntu and derivatives we package PMDs
> > individually, and then Canonical groups them in 2 sets - a subset
> > they
> > guarantee to support from their own resources, and the full set as
> > delivered by the community. But the point is that it's a step that
> > they
> > decide to take, and pay the price for it in terms of time
> > investment in
> > validating that particular combination, rather than the onus being
> > on
> > our very limited and already stretched resources to validate all
> > combinations.
> > 
> > We can focus our resources into making sure the few combinations
> > that
> > _must_ be supported, for example due to external dependencies, work
> > fine.
> > 
> > > > Sorry, but from me it's a very strong NACK.
> > > 
> > > Sorry, but let me disagree with you. For me, meson
> > > configurability is
> > > the
> > > essential thing to have in terms of deprecating the 'make' build
> > > system.
> > > DPDK was and keeps being (in most cases) the library that users
> > > statically
> > > linking to a single application built for particular platform and
> > > not
> > > using
> > > for anything else. This means that user in most cases knows which
> > > parts
> > > needed and which parts will never be used. Current meson build
> > > system
> > > doesn't allow to disable anything forcing users to link with the
> > > whole bunch
> > > of unused code.
> > > 
> > > One major case is that you have to have build environment equal
> > > to
> > > your
> > > target platform in terms of availability of external libraries.
> > > So,
> > > if I
> > > have some external library on build system, meson will build all
> > > the
> > > modules
> > > it depends from and will link them to my application. As a result
> > > I'll not
> > > be able to run my application on a target platform without
> > > installing
> > > additional dependencies which is not acceptable. This patch will
> > > allow to
> > > specifically disable all the libs that has unsatisfiable
> > > dependencies
> > > on
> > > target. Without the patch it's required to manually remove
> > > resulted
> > > libs and
> > > fix pkg-config and stuff before building apps. This is far less
> > > user-
> > > friendly
> > > than options I proposed. And yes, I still have to waste time for
> > > building
> > > libraries I'll remove right after.
> > > 
> > > While testing OVS on TravisCI, DPDK was built far more than 30K
> > > times
> > > which is more than half of a year of a wasted computational
> > > resources
> > > (if we'll count 10 minutes per build).
> > > I think this time could be used more wisely.
> > > 
> > > Best regards, Ilya Maximets.
> > 
> > But that's the thing: as it was discussed recently, we need to move
> > away from DPDK being by default a "special sauce" toolkit with
> > millions
> > of customizations that is custom built and statically linked, like
> > busybox, and to a situation where it's just another set of system
> > libraries like any other, shipped by the operating system, like
> > glibc -
> > see the threads about stable API/ABI and OS-driven delivery by Ray.
> > The status quo of an insanely granular build configuration that
> > means
> > everyone is using something different from each other is a bug -
> > not a
> > feature. Excessive _build time_ configuration exacerbates and
> > encourages this bug.
> > 
> > Sure, an initial build from scratch will take a couple of minutes
> > more
> > - mildly annoying, but worth the price. Caching takes care of that
> > problem already pretty well. Also standardizing on radically fewer
> > build configurations means you _don't_ have to rebuild DPDK for
> > third
> > party testing - you just consume the pre-built binaries from the OS
> > of
> > choice, or the PPA or similar for backports and HEAD builds. That
> > will
> > save even more time and resources in third-party build systems.
> > 
> 
> CI systems usually build everything from scratch. So caching doesn't
> help.
> And I don't think that distros will have any pre-built binaries for
> the
> old systems used in public CI systems. For example, TravisCI has
> 'Trusty'
> as a default environment. At most you may use 'Xenial'. But I don't
> think
> that DPDK 18.11 will ever be packaged for them. Also, distros mostly
> has
> dynamic libraries in their packages not providing static ones. It
> might
> be changed in the future, but still we'll not have them for 'xenial'.
> 
> So, OVS will stuck with slow building DPDK each time from scratch on
> Travis.
> And developers will maintain local patches for meson for stripping
> extra
> libraries or local build scripts that will package only what they
> need.
> 
> Best regards, Ilya Maximets.

Well, I'm not sure most CI systems build everything from scratch, and
even if they do, ccache is supported natively on Travis and others - we
use it for DPDK's Travis config, for example. I've yet to see a Travis
configuration that rebuilds the kernel, glibc, compiler and every
single other dependency of a project from scratch every time, though :-
)
In fact, Travis has native support for APT repositories and even PPAs
to do the opposite, and it's quite popular as far as I know.

Making a PPA or a project on OBS available with repositories for older
LTS releases is not only possible, but we already do it today - it's
based mostly on request, so for example even though Xenial shipped with
DPDk 2.2, a 17.11 PPA is available and maintained already, and I'm sure
if we ask very nicely 18.11 can be made available too :-)

I'm not sure about the RPM world, but in Debian, Ubuntu and derivatives
we ship both static and dynamic libraries by default, and that includes
DPDK from day 1.

So I can assure you I simpathize with the idea of making CIs faster,
and there are many ways to achieve that, that don't require manually
hacking the dependencies. For example, I've set up the ZeroMQ CI so
that all dependencies from the project and backports for Xenial are
built on OBS just as they would be built by the distro, with no
compromises or local just-in-time "hacks" to speed up individual
builds, so that binary packages are available and installed by Travis
via APT on each build. That takes less than 30s regardless of the
number of packages (within reason), so whatever we add as a dependency
it's still the same.

I had done that even for Trusty and Pristine at the time:

https://build.opensuse.org/project/monitor/network:messaging:zeromq:git-stable?arch_x86_64=1&defaults=0&repo_xUbuntu_14_04=1&succeeded=1
https://build.opensuse.org/project/monitor/network:messaging:zeromq:git-stable?arch_x86_64=1&defaults=0&repo_xUbuntu_12_04=1&succeeded=1

Here's a CI job that uses the Trusty repository:

https://travis-ci.org/zeromq/czmq/jobs/538626364

-- 
Kind regards,
Luca Boccassi