From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by dpdk.org (Postfix) with ESMTP id 95B202BB8 for ; Wed, 7 Jun 2017 15:27:18 +0200 (CEST) Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 07 Jun 2017 06:27:17 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.39,311,1493708400"; d="scan'208";a="95566300" Received: from bricha3-mobl3.ger.corp.intel.com ([10.237.221.28]) by orsmga004.jf.intel.com with SMTP; 07 Jun 2017 06:27:15 -0700 Received: by (sSMTP sendmail emulation); Wed, 07 Jun 2017 14:27:14 +0100 Date: Wed, 7 Jun 2017 14:27:14 +0100 From: Bruce Richardson To: Luca Boccassi Cc: "Van Haaren, Harry" , "dev@dpdk.org" , "christian.ehrhardt@canonical.com" Message-ID: <20170607132714.GB59372@bricha3-MOBL3.ger.corp.intel.com> References: <20170607104743.275149-1-bruce.richardson@intel.com> <1496841784.25214.6.camel@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1496841784.25214.6.camel@gmail.com> Organization: Intel Research and =?iso-8859-1?Q?De=ACvel?= =?iso-8859-1?Q?opment?= Ireland Ltd. User-Agent: Mutt/1.8.1 (2017-04-11) Subject: Re: [dpdk-dev] [RFC PATCH] replace DPDK config and build system X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jun 2017 13:27:19 -0000 On Wed, Jun 07, 2017 at 02:23:04PM +0100, Luca Boccassi wrote: > On Wed, 2017-06-07 at 13:08 +0000, Van Haaren, Harry wrote: > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce > > > Richardson > > > Sent: Wednesday, June 7, 2017 11:48 AM > > > To: dev@dpdk.org > > > Cc: Richardson, Bruce > > > Subject: [dpdk-dev] [RFC PATCH] replace DPDK config and build > > > system > > > > > > Hi all, > > > > > > following on from the pressing need to add support in DPDK for > > > detecting > > > and managing external dependencies, I undertook to see what options > > > we had. > > > However, unrelated to this, over time, I have become increasingly > > > frustrated by the complexity of the DPDK configuration and build > > > system. As > > > such, I feel that looking at some of the newer build tools that are > > > out > > > there might give us the additional functionality we want, along > > > with other > > > benefits. As such, I undertook a prototype using "meson"[1] for > > > configuration, which uses "ninja" as a backend for doing the actual > > > build. > > > > > > With these tools we get the following benefits (not a complete > > > list): > > > * support for detecting dependencies on the system > > > * support for detecting compiler features, including functions, > > > defines > > > * improved maintainability through a high-level language, which > > > gives > > >   decent error messages including line numbers (!!) > > > * co-existence with the existing makefile system without making any > > > changes > > >   to it > > > * faster builds using ninja - on my many-core system, the builds > > > seem > > >   significantly faster than with our existing system. Especially in > > > the > > >   nothing-has-changed case, builds with my prototype return > > > instantly, > > >   compared to taking a few seconds to recursively check each > > > directory with > > >   the current build system > > > * the ability to switch to using a standard "ninja" + "ninja > > > install" setup > > > * the chance to rework our existing build-config files, and > > > hopefully > > >   pretty much remove them. > > > * pkg-config support. > > > * we get to move away from our bespoke build system > > > * dependencies in each lib can be moved back to being tracked in > > > the libs > > >   files themselves, not up a level > > > > > > > > > Of course, it's not a panacea, but having spent hours on the > > > prototype thus > > > far, I find working with meson and ninja far more user-friendly > > > than > > > working on our makefiles, and again the build speed is a really > > > nice > > > improvment too. > > > > > > The prototype is incomplete, but it does build a reasonable number > > > of our > > > libraries, some unit tests, the i40e PMD and the testpmd binary, > > > and I have > > > successfully passed traffic using testpmd from the build. Some > > > things are > > > not fully correct, e.g. static builds aren't working right now, as > > > I haven't > > > correctly done all the dependency tracking, I think, and the cpu > > > flag > > > detection has issues. It also has only been tried on x86_64 linux, > > > on a > > > couple of systems, so YMMV. However, I feel it's a reasonable > > > enough start > > > point to show what we might be able to achieve. > > > > > > Please take the prototype and test it out. I think it's a better > > > alternative to trying to bolt on additional functionality to our > > > existing > > > config and build system. > > > > > > Test drive done; here are my experiences / thoughts: > > > > 1) Understanding the Meson build files is much easier for me than the > > current build-files. I'll admit that bash scripting is not my forte, > > with the caveat that if my basic bash scripting doesn't suffice, > > others in the community are probably in a similar position. > > > > 2) I see huge value in pkg-config integration - linking made easy. > > From a developer usability POV, a project should provide a pkg-config > > file, and configure itself based on the pkg-config files available on > > the system.  > > > > 3) The speed of ninja is impressive - particularly in the small- > > amounts-of-work case. Try it on your own machine if you don't believe > > me :) > > > > 4) "build variants" are super easy, meson build && meson > > build_variant  will create two directories, with .o files contained > > under it (same as one possible usage of the current system - just > > calling out that that feature remains). > > > > 5) Vim users, there is a "mesonic" plugin[1] which provides syntax > > highlighting for Meson files, the Meson Options file, and Meson > > integration to the Vim compiler hooks. Aka typing  :make   will cause > > Ninja to be invoked and compile. There are more fancy features, check > > the script page. > > > > 6) Ninja uninstall will reverse a previous installation, and returns > > a clean system. Nice for testing and integrating DPDK with > > applications that expect DPDK to be installed system-wide. > > > > > > Of course no build system is perfect - and I'd guess Meson also has a > > few gritty details that are not ideal - but to me this test-drive has > > shown promise. Opinions from non-developers may be of particular > > interest.. > > > > @DPDK Packagers, do you have experience with Meson? Does it integrate > > well with (your) existing infrastructure? Hope you don't mind me > > adding you to CC :) > > While I do not have personal experience with it, I know that recently > the Debian/Ubuntu tools have gained support for meson+ninja (debhelper > version 10.3), so it should not be a problem for myself and Christian. > > In fact, I personally think it's a big and very welcome improvement, so > thank you Bruce for this work! > > I will try to find time soon to test this patch in the context of > packaging the libraries. > Great, thanks for taking a look at it. Please note that this does require meson 0.4 right now, because we do so much stuff with optimizing for instruction sets we really need the new "has_define()" function that comes only in meson 0.4. However, I anticipate that given the time it is likely to take to get anything like this into DPDK, meson 0.4 should be pretty widespread by then. :-) /Bruce