Re: [dpdk-dev] [RFC PATCH] replace DPDK config and build system

[Bruce Richardson <bruce.richardson@intel.com>]

On Thu, Jun 22, 2017 at 01:14:54PM -0400, Neil Horman wrote:
> On Wed, Jun 07, 2017 at 11:47:42AM +0100, Bruce Richardson wrote:
> > Hi all,
> > 
> > following on from the pressing need to add support in DPDK for detecting
> > and managing external dependencies, I undertook to see what options we had.
> > However, unrelated to this, over time, I have become increasingly
> > frustrated by the complexity of the DPDK configuration and build system. As
> > such, I feel that looking at some of the newer build tools that are out
> > there might give us the additional functionality we want, along with other
> > benefits. As such, I undertook a prototype using "meson"[1] for
> > configuration, which uses "ninja" as a backend for doing the actual build.
> > 
> > With these tools we get the following benefits (not a complete list):
> > * support for detecting dependencies on the system
> > * support for detecting compiler features, including functions, defines
> > * improved maintainability through a high-level language, which gives
> >   decent error messages including line numbers (!!)
> > * co-existence with the existing makefile system without making any changes
> >   to it
> > * faster builds using ninja - on my many-core system, the builds seem
> >   significantly faster than with our existing system. Especially in the
> >   nothing-has-changed case, builds with my prototype return instantly,
> >   compared to taking a few seconds to recursively check each directory with
> >   the current build system
> > * the ability to switch to using a standard "ninja" + "ninja install" setup
> > * the chance to rework our existing build-config files, and hopefully
> >   pretty much remove them.
> > * pkg-config support.
> > * we get to move away from our bespoke build system
> > * dependencies in each lib can be moved back to being tracked in the libs
> >   files themselves, not up a level
> > 
> > 
> > Of course, it's not a panacea, but having spent hours on the prototype thus
> > far, I find working with meson and ninja far more user-friendly than
> > working on our makefiles, and again the build speed is a really nice
> > improvment too.
> > 
> > The prototype is incomplete, but it does build a reasonable number of our
> > libraries, some unit tests, the i40e PMD and the testpmd binary, and I have
> > successfully passed traffic using testpmd from the build. Some things are
> > not fully correct, e.g. static builds aren't working right now, as I haven't
> > correctly done all the dependency tracking, I think, and the cpu flag
> > detection has issues. It also has only been tried on x86_64 linux, on a
> > couple of systems, so YMMV. However, I feel it's a reasonable enough start
> > point to show what we might be able to achieve.
> > 
> > Please take the prototype and test it out. I think it's a better
> > alternative to trying to bolt on additional functionality to our existing
> > config and build system.
> > 
> > [1] http://mesonbuild.com/
> 
> A few notes:

Hi Neil,

thanks for taking a look at this. I agree with all your notes below, and
added some comments of my own to each point below. Overall, I fully
agree with your conclusion that it's too early to dump make - even the
best case plans envisaged for this, would not see make dropped for quite a
number of releases [if ever].

> 
> 1) It is quick for a native build, which is nice
> 
> 2) Its seemingly much more clear to understand than the current Makefile system
> (which is horribly opaque).  That said, its still not a particularly common
> build system.  I wonder if there wouldn't be better reception to just cleaning
> up the existing Makefiles, with something like autoconf.   That likely still
> won't be as fast as ninja, but it will be common and well understood, and still
> reasonably fast.

Sure, that is an option. I decided to look for alternatives mainly
because IIRC any mention of autotools in the past has met a very negative
reaction on-list. I could be wrong on this but I got the general
impression that many thought an autotools cure was better than the
disease! :-) Secondly, working with a non-makefile based system allows
old and new systems to co-exist for as long as we want them too, without
having to do a wholesale replacement, or go to great lengths to manage
the two within the one set of (make)files.

> 
> 3) The tools that are needed here, aren't universally available.  I can speak to
> RHEL specifically which doesn't have ninja currently, nor meson (they are
> available in EPEL, but that can't be used for official builds). Its not an
> insurmountable problem, but its something to consider when considering product
> reach, and the impact this will have on distributions.

I'll admit that meson is fairly new, but from what I read it is also
gaining quite a bit of acceptance in the gnome world at least, so it's
not really that obscure a system. Ninja, on the other hand, has been
around since 2011, and is widely used by many other tools, e.g. cmake,
gyp and meson, and projects like chromium, or llvm, so I'm a bit
surprised that is is not in RHEL. Are other packages compiled using
ninja in RHEL?

> 
> 4) I'm a bit lost as to how inheritance works in meson (or if it exists at all).
> All the examples you provide here seem to duplicate work.  For instance, several
> libraries independently check  if dpdk_conf.has('RTE_MACHINE_CPUFLAG_SSE4_1')
> and set proper cflags accordingly, rather than inheriting the flag from a higher
> level.  While that in and of itself isn't a singular problem, it could well be a
> problem at scale for more complex operations (see below)

Yep, good point. Things like that I would always initially blame on my
novice implementation as I was learning meson as I went along. If it is
really a problem, doing a full implementation should quickly show that
up, and we can go back to the drawing board.

> 
> 5) I know you said this was just an incomplete prototype, which is fine, but I
> noticed that the build it produces doesn't yield any pmd exported info (of the
> type you would get using dpdk-pmdinfo.py).  Thats because the building of that
> info requires a multi-stage build process, wherein .o files are post-processed
> to generate new .c files, which are then compiled and linked into the final .so
> or .a file.  While I think I see how you might use the gen_src and custom_target
> rules to make that happen, the (supposed) lack of inheritance noted above, could
> lead to the duplication of alot of fairly complex rules that would have to be
> distributed to every pmd in the system.  Thats something to consider.

Yep. I explicitly decided not to tackle the pmdinfo stuff for the
prototype, but I hadn't considered the implications for duplication in
the build files. I'll try and dig into this soon before we invest
serious work into this.

> 
> 6) Integration with other build systems.  We build some kernel modules in dpdk
> that likely need to remain as part of the make build system (they don't have to,
> I suppose, but it would seem like alot of effort to change that).  Is using
> something non-make going to fit there?

I was assuming we'd still use make for the kernel modules, but haven't
tried to see how that would work yet.

> 
> Just a few thoughts.  All in all the speed is nice, but I think theres alot to
> consider before dumping Make.  Not so much because Make is great, but because
> its common, and well understood, and integrated with other projects.
> 
Yep, at this point I'd rate it as "definitely worth looking at further",
rather than a definite commitment to replace what we have.
I'm aiming to start doing a post-prototype implementation in a staging
tree in the near future and we can see how that shapes up.

/Bruce