Re: [dpdk-dev] [RFC PATCH] replace DPDK config and build system

[Bruce Richardson <bruce.richardson@intel.com>]

On Wed, Jun 07, 2017 at 02:23:04PM +0100, Luca Boccassi wrote:
> On Wed, 2017-06-07 at 13:08 +0000, Van Haaren, Harry wrote:
> > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce
> > > Richardson
> > > Sent: Wednesday, June 7, 2017 11:48 AM
> > > To: dev@dpdk.org
> > > Cc: Richardson, Bruce <bruce.richardson@intel.com>
> > > Subject: [dpdk-dev] [RFC PATCH] replace DPDK config and build
> > > system
> > > 
> > > Hi all,
> > > 
> > > following on from the pressing need to add support in DPDK for
> > > detecting
> > > and managing external dependencies, I undertook to see what options
> > > we had.
> > > However, unrelated to this, over time, I have become increasingly
> > > frustrated by the complexity of the DPDK configuration and build
> > > system. As
> > > such, I feel that looking at some of the newer build tools that are
> > > out
> > > there might give us the additional functionality we want, along
> > > with other
> > > benefits. As such, I undertook a prototype using "meson"[1] for
> > > configuration, which uses "ninja" as a backend for doing the actual
> > > build.
> > > 
> > > With these tools we get the following benefits (not a complete
> > > list):
> > > * support for detecting dependencies on the system
> > > * support for detecting compiler features, including functions,
> > > defines
> > > * improved maintainability through a high-level language, which
> > > gives
> > >   decent error messages including line numbers (!!)
> > > * co-existence with the existing makefile system without making any
> > > changes
> > >   to it
> > > * faster builds using ninja - on my many-core system, the builds
> > > seem
> > >   significantly faster than with our existing system. Especially in
> > > the
> > >   nothing-has-changed case, builds with my prototype return
> > > instantly,
> > >   compared to taking a few seconds to recursively check each
> > > directory with
> > >   the current build system
> > > * the ability to switch to using a standard "ninja" + "ninja
> > > install" setup
> > > * the chance to rework our existing build-config files, and
> > > hopefully
> > >   pretty much remove them.
> > > * pkg-config support.
> > > * we get to move away from our bespoke build system
> > > * dependencies in each lib can be moved back to being tracked in
> > > the libs
> > >   files themselves, not up a level
> > > 
> > > 
> > > Of course, it's not a panacea, but having spent hours on the
> > > prototype thus
> > > far, I find working with meson and ninja far more user-friendly
> > > than
> > > working on our makefiles, and again the build speed is a really
> > > nice
> > > improvment too.
> > > 
> > > The prototype is incomplete, but it does build a reasonable number
> > > of our
> > > libraries, some unit tests, the i40e PMD and the testpmd binary,
> > > and I have
> > > successfully passed traffic using testpmd from the build. Some
> > > things are
> > > not fully correct, e.g. static builds aren't working right now, as
> > > I haven't
> > > correctly done all the dependency tracking, I think, and the cpu
> > > flag
> > > detection has issues. It also has only been tried on x86_64 linux,
> > > on a
> > > couple of systems, so YMMV. However, I feel it's a reasonable
> > > enough start
> > > point to show what we might be able to achieve.
> > > 
> > > Please take the prototype and test it out. I think it's a better
> > > alternative to trying to bolt on additional functionality to our
> > > existing
> > > config and build system.
> > 
> > 
> > Test drive done; here are my experiences / thoughts:
> > 
> > 1) Understanding the Meson build files is much easier for me than the
> > current build-files. I'll admit that bash scripting is not my forte,
> > with the caveat that if my basic bash scripting doesn't suffice,
> > others in the community are probably in a similar position.
> > 
> > 2) I see huge value in pkg-config integration - linking made easy.
> > From a developer usability POV, a project should provide a pkg-config 
> > file, and configure itself based on the pkg-config files available on
> > the system. 
> > 
> > 3) The speed of ninja is impressive - particularly in the small-
> > amounts-of-work case. Try it on your own machine if you don't believe
> > me :)
> > 
> > 4) "build variants" are super easy, meson build && meson
> > build_variant  will create two directories, with .o files contained
> > under it (same as one possible usage of the current system - just
> > calling out that that feature remains).
> > 
> > 5) Vim users, there is a "mesonic" plugin[1] which provides syntax
> > highlighting for Meson files, the Meson Options file, and Meson
> > integration to the Vim compiler hooks. Aka typing  :make   will cause
> > Ninja to be invoked and compile. There are more fancy features, check
> > the script page.
> > 
> > 6) Ninja uninstall will reverse a previous installation, and returns
> > a clean system. Nice for testing and integrating DPDK with
> > applications that expect DPDK to be installed system-wide.
> > 
> > 
> > Of course no build system is perfect - and I'd guess Meson also has a
> > few gritty details that are not ideal - but to me this test-drive has
> > shown promise. Opinions from non-developers may be of particular
> > interest..
> > 
> > @DPDK Packagers, do you have experience with Meson? Does it integrate
> > well with (your) existing infrastructure? Hope you don't mind me
> > adding you to CC :)
> 
> While I do not have personal experience with it, I know that recently
> the Debian/Ubuntu tools have gained support for meson+ninja (debhelper
> version 10.3), so it should not be a problem for myself and Christian.
> 
> In fact, I personally think it's a big and very welcome improvement, so
> thank you Bruce for this work!
> 
> I will try to find time soon to test this patch in the context of
> packaging the libraries.
>

Great, thanks for taking a look at it. Please note that this does
require meson 0.4 right now, because we do so much stuff with optimizing
for instruction sets we really need the new "has_define()" function that
comes only in meson 0.4. However, I anticipate that given the time it is
likely to take to get anything like this into DPDK, meson 0.4 should be
pretty widespread by then. :-)

/Bruce