From: Bruce Richardson <bruce.richardson@intel.com>
To: Neil Horman <nhorman@tuxdriver.com>
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] [dpdk-dev, 01/17] build: add initial infrastructure for meson & ninja builds
Date: Fri, 8 Sep 2017 14:55:08 +0100 [thread overview]
Message-ID: <20170908135508.GA37820@bricha3-MOBL3.ger.corp.intel.com> (raw)
In-Reply-To: <20170908115706.GA21804@hmswarspite.think-freely.org>
On Fri, Sep 08, 2017 at 07:57:06AM -0400, Neil Horman wrote:
> On Fri, Sep 08, 2017 at 09:50:26AM +0100, Bruce Richardson wrote:
> > On Thu, Sep 07, 2017 at 12:21:57PM -0400, Neil Horman wrote:
> > > On Fri, Sep 01, 2017 at 11:04:00AM +0100, Bruce Richardson wrote:
> > > > To build with meson and ninja, we need some initial infrastructure in
> > > > place. The build files for meson always need to be called "meson.build",
> > > > and options get placed in meson_options.txt
> > > >
> > > > This commit adds a top-level meson.build file, which sets up the global
> > > > variables for tracking drivers, libraries, etc., and then includes other
> > > > build files, before finishing by writing the global build configuration
> > > > header file and a DPDK pkgconfig file at the end, using some of those same
> > > > globals.
> > > >
> > > > >From the top level build file, the only include file thus far is for the
> > > > config folder, which does some other setup of global configuration
> > > > parameters, including pulling in architecture specific parameters from an
> > > > architectural subdirectory. A number of configuration build options are
> > > > provided for the project to tune a number of global variables which will be
> > > > used later e.g. max numa nodes, max cores, etc. These settings all make
> > > > their way to the global build config header "rte_build_config.h". There is
> > > > also a file "rte_config.h", which includes "rte_build_config.h", and this
> > > > file is meant to hold other build-time values which are present in our
> > > > current static build configuration but are not normally meant for
> > > > user-configuration. Ideally, over time, the values placed here should be
> > > > moved to the individual libraries or drivers which want those values.
> > > >
> > > > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > > > Reviewed-by: Harry van Haaren <harry.van.haaren@intel.com>
> > >
> > > I feel like I need to underscore my previous concern here. While I'm not
> > > opposed per-se to a new build system, I am very concerned about the burden that
> > > switching places on downstream consumers, in particular distributions (since I
> > > represent one of them). Moving to a new build system with new tools means those
> > > tools need to be packaged, tested and shipped, which is a significant work
> > > effort. While it might be a net gain long term, its something you need to keep
> > > in mind when making these changes.
> > >
> > Understood. If there is anything we/I can do to make this transition
> > easier, please flag it for consideration.
> >
> Thank you, I appreciate that.
>
> > > I know you've said that we will be keepting the existing build system,
> > > I just need to be sure everyone understands just how important that
> > > is.
> > >
> > What is your feeling here, in terms of timescale. After any new system
> > reaches feature parity, how long would you estimate that we would need
> > to support the existing makefile system before it would be safe to
> > deprecate it? Should we start a deprecation plan, or is it best just to
> > commit to support both until we get all - or almost all - downstream
> > consumers switched over? While I wouldn't push for deprecating the old
> > system any time soon, and I wouldn't consider maintaining the two
> > unduly burdensome, it's not something we want to do in the long term.
> >
> I was hoping to avoid putting a specific time frame on it, but its a fair
> question to ask. I feel like any particular timetable is somewhat arbitrary.
> Keith suggested a year, which is likely as good as any in my mind. To put a bit
> more detail behind it, a RHEL release cycle is anywhere from 6 to 18 months, so
> a year fits well. If we assume starting a few weeks back when you first
> proposed this change, that its going to be merged, that gives us time to package
> the build components, build the new package using them, get it through a qa
> cycle and fix anything that pops up as a result. That way, when the switch is
> made, it can be done with an immediate deprecation of the old build system with
> a level of confidence that some of the more esoteric build targets/configs will
> likely work.
>
> > > Though perhaps the time frame for keeping the current build system as priarmy is
> > > less concerning, as feature parity is even more critical. That is to say, the
> > > new build system must be able to produce the same configurations that the
> > > current build system does. Without it I don't think anyone will be able to use
> > > it consistently, and that will leave a great number of users in a very poor
> > > position. I think getting a little closer to parity with the current system is
> > > warranted. I'd suggest as a gating factor:
> > >
> > > 1) Building on all supported arches
> > > 2) Cross building on all supported arches
> > > 3) Proper identification of targeted machine (i.e. equivalent of the machine
> > > component of the current build system)
> > >
> > The question there is gating factor for what? Presumably not for merging
> > into the staging tree. But for merging into the main tree for releases?
> > I'd push back a little on that, as the new system does not interfere in
> > any way with the old, and by keeping it in a staging tree until it
> > reaches full feature parity will make the job considerably harder. For
> > example, it means that anyone submitting a new driver or library has to
> > submit the code and makefiles in one set and the meson patches in a
> > separate one for a separate build tree. It also makes it less likely
> > that people will try out the new system and find the issues with it, and
> > help fill in the gaps. While I can understand us not recommending the
> > new build system until it reaches feature parity, I think there are a
> > lot of benefits to be got by making it widely available, even if it's
> > incomplete.
>
> Yes, sorry, the implied "what" here is gating its introduction to mainline. I
> have no problem with this going into a development or staging branch/tree, only
> with it getting merged to mainline and becoming the primary build system today.
> I get that it makes reaching feature parity harder, but to not do so relegates
> anyone that hasn't had a chance to test the new build system to second class
> citizen status (or at least potentially does so). To be a bit more specific, I
> can see how energized you might be to get this in place now because you've
> tested it on a wide array on intel hardware, but I'm guessing if it went in
> today, people at IBM and Linaro would have to drop active developement to start
> switching their build environments over to the new system lest they get left out
> in the cold. I think its more about balacing where the hardship lies here.
>
> As I'm writing this, I wonder if a reasonable compromise couldn't involve the
> use of CI? That is to say, what if we integrated the build system now-ish, and
> stood up an official CI instance, that both:
> 1) built the dpdk in all style configuration, which mandated the use of the old
> build system (i.e. we implicitly mandate that the current build system stay
> working, and is not forgotten), and gate patch merges on that result.
>
> 2) add a test that any change to a meson file in mainline also include a change
> to the a Makefile
>
> I'm just spitballing here, but I'm looking for ways to enforce the continued use
> of the current build system above and beyond a verbal promise to do so. The
> idea is to ensure that it stays operational and primary to the development of
> dpdk until build system parity is reached.
>
Yes, that all makes sense, and I agree that the main issue is how to
ensure/enforce the fact that the old build system stays as the main way
to build DPDK, until such time as the new system reaches parity. A CI
system like you describe would be the ideal case, and we have some parts
of that, just not enough I think to satify our needs. On the other hand,
while holding the new system in a staging tree till it reaches parity
would definitely achieve that objective I think the cost of that is too
high.
Right now patchwork reports results of "Intel-compilation". I'm not
aware of any immediate plans to expand that - somebody please should if
there are. There is also the planned DPDK lab, but I'm not sure of the
schedule for that or if it will help in this instance.
For now, I'll prepare a slightly-updated V2 patchset, with a view to
getting work started on the build-next tree. While that is going on, we
can maybe continue to think about how to put in suitable guard-rails for
ensuring old system compatibility for when it's pulled to mainline.
Thanks again for raising your concerns about this. It's easy for me to
sometimes push too hard for something I'm enthusiastic about! :-) I do
want this migration done properly, so if you have the bandwidth I'd
appreciate any reviews or feedback you can give, as the work goes on.
Regards,
/Bruce
next prev parent reply other threads:[~2017-09-08 13:55 UTC|newest]
Thread overview: 107+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-09-01 10:03 [dpdk-dev] [PATCH 00/17] build DPDK libs and some drivers with meson/ninja Bruce Richardson
2017-09-01 10:04 ` [dpdk-dev] [PATCH 01/17] build: add initial infrastructure for meson & ninja builds Bruce Richardson
2017-09-04 13:36 ` Van Haaren, Harry
2017-09-04 13:51 ` Bruce Richardson
2017-09-06 16:18 ` Bruce Richardson
2017-09-07 13:52 ` Bruce Richardson
2017-09-07 16:21 ` [dpdk-dev] [dpdk-dev, " Neil Horman
2017-09-07 16:47 ` Wiles, Keith
2017-09-07 17:12 ` Neil Horman
2017-09-08 8:50 ` Bruce Richardson
2017-09-08 11:57 ` Neil Horman
2017-09-08 13:55 ` Bruce Richardson [this message]
2017-09-08 14:00 ` Bruce Richardson
2017-09-08 16:03 ` [dpdk-dev] [PATCH " Van Haaren, Harry
2017-09-08 16:13 ` Richardson, Bruce
2017-09-12 9:56 ` Bruce Richardson
2017-09-01 10:04 ` [dpdk-dev] [PATCH 02/17] eal: add eal library to meson build Bruce Richardson
2017-09-04 13:53 ` Van Haaren, Harry
2017-09-07 16:25 ` [dpdk-dev] [dpdk-dev, " Neil Horman
2017-09-08 8:51 ` Bruce Richardson
2017-09-01 10:04 ` [dpdk-dev] [PATCH 03/17] igb_uio: add igb_uio kmod " Bruce Richardson
2017-09-01 13:32 ` Luca Boccassi
2017-09-01 13:55 ` Bruce Richardson
2017-09-01 14:27 ` Luca Boccassi
2017-09-04 13:57 ` Van Haaren, Harry
2017-09-01 10:04 ` [dpdk-dev] [PATCH 04/17] build: add DPDK libraries to build Bruce Richardson
2017-09-04 14:08 ` Van Haaren, Harry
2017-09-04 14:55 ` Bruce Richardson
2017-09-01 10:04 ` [dpdk-dev] [PATCH 05/17] build: add buildtools to meson build Bruce Richardson
2017-09-04 14:24 ` Van Haaren, Harry
2017-09-01 10:04 ` [dpdk-dev] [PATCH 06/17] build: add infrastructure for building PMDs Bruce Richardson
2017-09-04 14:27 ` Van Haaren, Harry
2017-09-01 10:04 ` [dpdk-dev] [PATCH 07/17] drivers/mempool: add SW mempool drivers to meson build Bruce Richardson
2017-09-04 14:28 ` Van Haaren, Harry
2017-09-04 14:29 ` Van Haaren, Harry
2017-09-01 10:04 ` [dpdk-dev] [PATCH 08/17] drivers/crypto: add crypto drv class and null PMD to meson Bruce Richardson
2017-09-04 14:30 ` Van Haaren, Harry
2017-09-01 10:04 ` [dpdk-dev] [PATCH 09/17] crypto/openssl: add driver to meson build Bruce Richardson
2017-09-04 14:30 ` Van Haaren, Harry
2017-09-04 14:31 ` Van Haaren, Harry
2017-09-01 10:04 ` [dpdk-dev] [PATCH 10/17] crypto/qat: " Bruce Richardson
2017-09-04 14:32 ` Van Haaren, Harry
2017-09-01 10:04 ` [dpdk-dev] [PATCH 11/17] drivers/net: add net driver support " Bruce Richardson
2017-09-04 14:32 ` Van Haaren, Harry
2017-09-01 10:04 ` [dpdk-dev] [PATCH 12/17] drivers/net: add set of vdev PMDs to build Bruce Richardson
2017-09-04 14:34 ` Van Haaren, Harry
2017-09-01 10:04 ` [dpdk-dev] [PATCH 13/17] drivers/net: add drivers for Intel NICs to meson build Bruce Richardson
2017-09-01 10:04 ` [dpdk-dev] [PATCH 14/17] app/test-pmd: add test-pmd " Bruce Richardson
2017-09-04 15:01 ` Van Haaren, Harry
2017-09-01 10:04 ` [dpdk-dev] [PATCH 15/17] usertools: add usertools installation " Bruce Richardson
2017-09-04 15:00 ` Van Haaren, Harry
2017-09-01 10:04 ` [dpdk-dev] [PATCH 16/17] build: add option to version libs using DPDK version Bruce Richardson
2017-09-07 17:07 ` [dpdk-dev] [dpdk-dev, " Neil Horman
2017-09-08 8:58 ` Bruce Richardson
2017-09-01 10:04 ` [dpdk-dev] [PATCH 17/17] doc: add documentation on how to add new components to DPDK Bruce Richardson
2017-09-01 10:38 ` [dpdk-dev] [PATCH 00/17] build DPDK libs and some drivers with meson/ninja Bruce Richardson
2017-09-12 10:37 ` [dpdk-dev] [PATCH v2 " Bruce Richardson
2017-09-12 10:37 ` [dpdk-dev] [PATCH v2 01/17] build: add initial infrastructure for meson & ninja builds Bruce Richardson
2017-09-12 10:37 ` [dpdk-dev] [PATCH v2 02/17] eal: add eal library to meson build Bruce Richardson
2017-09-12 10:37 ` [dpdk-dev] [PATCH v2 03/17] igb_uio: add igb_uio kmod " Bruce Richardson
2017-09-12 10:37 ` [dpdk-dev] [PATCH v2 04/17] build: add DPDK libraries to build Bruce Richardson
2017-09-12 10:37 ` [dpdk-dev] [PATCH v2 05/17] build: add buildtools to meson build Bruce Richardson
2017-09-12 10:37 ` [dpdk-dev] [PATCH v2 06/17] build: add infrastructure for building PMDs Bruce Richardson
2017-09-12 10:37 ` [dpdk-dev] [PATCH v2 07/17] drivers/mempool: add SW mempool drivers to meson build Bruce Richardson
2017-09-12 10:38 ` [dpdk-dev] [PATCH v2 08/17] drivers/crypto: add crypto drv class and null PMD to meson Bruce Richardson
2017-09-12 10:38 ` [dpdk-dev] [PATCH v2 09/17] crypto/openssl: add driver to meson build Bruce Richardson
2017-09-12 10:38 ` [dpdk-dev] [PATCH v2 10/17] crypto/qat: " Bruce Richardson
2017-09-12 10:38 ` [dpdk-dev] [PATCH v2 11/17] drivers/net: add net driver support " Bruce Richardson
2017-09-12 10:38 ` [dpdk-dev] [PATCH v2 12/17] drivers/net: add set of vdev PMDs to build Bruce Richardson
2017-09-12 10:38 ` [dpdk-dev] [PATCH v2 13/17] drivers/net: add drivers for Intel NICs to meson build Bruce Richardson
2017-09-12 10:38 ` [dpdk-dev] [PATCH v2 14/17] app/test-pmd: add test-pmd " Bruce Richardson
2017-09-12 10:38 ` [dpdk-dev] [PATCH v2 15/17] usertools: add usertools installation " Bruce Richardson
2017-09-12 10:38 ` [dpdk-dev] [PATCH v2 16/17] build: add option to version libs using DPDK version Bruce Richardson
2017-09-13 11:32 ` Luca Boccassi
2017-09-13 13:11 ` Bruce Richardson
2017-09-13 17:02 ` Luca Boccassi
2017-09-12 10:38 ` [dpdk-dev] [PATCH v2 17/17] doc: add documentation on how to add new components to DPDK Bruce Richardson
2017-09-12 13:21 ` [dpdk-dev] [PATCH v2 00/17] build DPDK libs and some drivers with meson/ninja Wiles, Keith
2017-09-12 13:24 ` Bruce Richardson
2017-09-12 13:26 ` Wiles, Keith
2017-09-12 14:00 ` Wiles, Keith
2017-09-12 15:46 ` Bruce Richardson
2017-09-12 19:19 ` Neil Horman
2017-09-12 20:36 ` Wiles, Keith
2017-09-13 14:12 ` [dpdk-dev] [PATCH v3 " Bruce Richardson
2017-09-13 14:12 ` [dpdk-dev] [PATCH v3 01/17] build: add initial infrastructure for meson & ninja builds Bruce Richardson
2017-09-20 10:10 ` Timothy M. Redaelli
2017-09-20 10:24 ` Bruce Richardson
2017-09-20 10:33 ` Timothy M. Redaelli
2017-09-13 14:12 ` [dpdk-dev] [PATCH v3 02/17] eal: add eal library to meson build Bruce Richardson
2017-09-13 14:12 ` [dpdk-dev] [PATCH v3 03/17] igb_uio: add igb_uio kmod " Bruce Richardson
2017-09-13 14:12 ` [dpdk-dev] [PATCH v3 04/17] build: add DPDK libraries to build Bruce Richardson
2017-09-13 14:12 ` [dpdk-dev] [PATCH v3 05/17] build: add buildtools to meson build Bruce Richardson
2017-09-13 14:12 ` [dpdk-dev] [PATCH v3 06/17] build: add infrastructure for building PMDs Bruce Richardson
2017-09-13 14:12 ` [dpdk-dev] [PATCH v3 07/17] drivers/mempool: add SW mempool drivers to meson build Bruce Richardson
2017-09-13 14:12 ` [dpdk-dev] [PATCH v3 08/17] drivers/crypto: add crypto drv class and null PMD to meson Bruce Richardson
2017-09-13 14:12 ` [dpdk-dev] [PATCH v3 09/17] crypto/openssl: add driver to meson build Bruce Richardson
2017-09-13 14:12 ` [dpdk-dev] [PATCH v3 10/17] crypto/qat: " Bruce Richardson
2017-09-13 14:12 ` [dpdk-dev] [PATCH v3 11/17] drivers/net: add net driver support " Bruce Richardson
2017-09-13 14:12 ` [dpdk-dev] [PATCH v3 12/17] drivers/net: add set of vdev PMDs to build Bruce Richardson
2017-09-13 14:12 ` [dpdk-dev] [PATCH v3 13/17] drivers/net: add drivers for Intel NICs to meson build Bruce Richardson
2017-09-13 14:12 ` [dpdk-dev] [PATCH v3 14/17] app/test-pmd: add test-pmd " Bruce Richardson
2017-09-13 14:12 ` [dpdk-dev] [PATCH v3 15/17] usertools: add usertools installation " Bruce Richardson
2017-09-13 14:12 ` [dpdk-dev] [PATCH v3 16/17] build: add option to version libs using DPDK version Bruce Richardson
2017-09-13 14:12 ` [dpdk-dev] [PATCH v3 17/17] doc: add documentation on how to add new components to DPDK Bruce Richardson
2017-09-13 17:01 ` [dpdk-dev] [PATCH v3 00/17] build DPDK libs and some drivers with meson/ninja Luca Boccassi
2017-09-14 13:57 ` Bruce Richardson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170908135508.GA37820@bricha3-MOBL3.ger.corp.intel.com \
--to=bruce.richardson@intel.com \
--cc=dev@dpdk.org \
--cc=nhorman@tuxdriver.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).