From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by dpdk.org (Postfix) with ESMTP id 1A97D2BFE for ; Fri, 8 Sep 2017 16:00:29 +0200 (CEST) Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga105.fm.intel.com with ESMTP; 08 Sep 2017 07:00:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.42,361,1500966000"; d="scan'208";a="149673957" Received: from bricha3-mobl3.ger.corp.intel.com ([10.237.221.24]) by fmsmga006.fm.intel.com with SMTP; 08 Sep 2017 07:00:26 -0700 Received: by (sSMTP sendmail emulation); Fri, 08 Sep 2017 15:00:26 +0100 Date: Fri, 8 Sep 2017 15:00:25 +0100 From: Bruce Richardson To: Neil Horman , Thomas Monjalon , Xu Qian Q Cc: dev@dpdk.org Message-ID: <20170908140025.GB37820@bricha3-MOBL3.ger.corp.intel.com> References: <20170901100416.80264-2-bruce.richardson@intel.com> <20170907162157.GA18344@hmswarspite.think-freely.org> <20170908085026.GA35580@bricha3-MOBL3.ger.corp.intel.com> <20170908115706.GA21804@hmswarspite.think-freely.org> <20170908135508.GA37820@bricha3-MOBL3.ger.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20170908135508.GA37820@bricha3-MOBL3.ger.corp.intel.com> Organization: Intel Research and =?iso-8859-1?Q?De=ACvel?= =?iso-8859-1?Q?opment?= Ireland Ltd. User-Agent: Mutt/1.8.3 (2017-05-23) Subject: Re: [dpdk-dev] [dpdk-dev, 01/17] build: add initial infrastructure for meson & ninja builds X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Sep 2017 14:00:30 -0000 +Thomas and Qian from our validation team, in case they have any insight on build CI. On Fri, Sep 08, 2017 at 02:55:08PM +0100, Bruce Richardson wrote: > On Fri, Sep 08, 2017 at 07:57:06AM -0400, Neil Horman wrote: > > On Fri, Sep 08, 2017 at 09:50:26AM +0100, Bruce Richardson wrote: > > > On Thu, Sep 07, 2017 at 12:21:57PM -0400, Neil Horman wrote: > > > > On Fri, Sep 01, 2017 at 11:04:00AM +0100, Bruce Richardson wrote: > > > > > To build with meson and ninja, we need some initial infrastructure in > > > > > place. The build files for meson always need to be called "meson.build", > > > > > and options get placed in meson_options.txt > > > > > > > > > > This commit adds a top-level meson.build file, which sets up the global > > > > > variables for tracking drivers, libraries, etc., and then includes other > > > > > build files, before finishing by writing the global build configuration > > > > > header file and a DPDK pkgconfig file at the end, using some of those same > > > > > globals. > > > > > > > > > > >From the top level build file, the only include file thus far is for the > > > > > config folder, which does some other setup of global configuration > > > > > parameters, including pulling in architecture specific parameters from an > > > > > architectural subdirectory. A number of configuration build options are > > > > > provided for the project to tune a number of global variables which will be > > > > > used later e.g. max numa nodes, max cores, etc. These settings all make > > > > > their way to the global build config header "rte_build_config.h". There is > > > > > also a file "rte_config.h", which includes "rte_build_config.h", and this > > > > > file is meant to hold other build-time values which are present in our > > > > > current static build configuration but are not normally meant for > > > > > user-configuration. Ideally, over time, the values placed here should be > > > > > moved to the individual libraries or drivers which want those values. > > > > > > > > > > Signed-off-by: Bruce Richardson > > > > > Reviewed-by: Harry van Haaren > > > > > > > > I feel like I need to underscore my previous concern here. While I'm not > > > > opposed per-se to a new build system, I am very concerned about the burden that > > > > switching places on downstream consumers, in particular distributions (since I > > > > represent one of them). Moving to a new build system with new tools means those > > > > tools need to be packaged, tested and shipped, which is a significant work > > > > effort. While it might be a net gain long term, its something you need to keep > > > > in mind when making these changes. > > > > > > > Understood. If there is anything we/I can do to make this transition > > > easier, please flag it for consideration. > > > > > Thank you, I appreciate that. > > > > > > I know you've said that we will be keepting the existing build system, > > > > I just need to be sure everyone understands just how important that > > > > is. > > > > > > > What is your feeling here, in terms of timescale. After any new system > > > reaches feature parity, how long would you estimate that we would need > > > to support the existing makefile system before it would be safe to > > > deprecate it? Should we start a deprecation plan, or is it best just to > > > commit to support both until we get all - or almost all - downstream > > > consumers switched over? While I wouldn't push for deprecating the old > > > system any time soon, and I wouldn't consider maintaining the two > > > unduly burdensome, it's not something we want to do in the long term. > > > > > I was hoping to avoid putting a specific time frame on it, but its a fair > > question to ask. I feel like any particular timetable is somewhat arbitrary. > > Keith suggested a year, which is likely as good as any in my mind. To put a bit > > more detail behind it, a RHEL release cycle is anywhere from 6 to 18 months, so > > a year fits well. If we assume starting a few weeks back when you first > > proposed this change, that its going to be merged, that gives us time to package > > the build components, build the new package using them, get it through a qa > > cycle and fix anything that pops up as a result. That way, when the switch is > > made, it can be done with an immediate deprecation of the old build system with > > a level of confidence that some of the more esoteric build targets/configs will > > likely work. > > > > > > Though perhaps the time frame for keeping the current build system as priarmy is > > > > less concerning, as feature parity is even more critical. That is to say, the > > > > new build system must be able to produce the same configurations that the > > > > current build system does. Without it I don't think anyone will be able to use > > > > it consistently, and that will leave a great number of users in a very poor > > > > position. I think getting a little closer to parity with the current system is > > > > warranted. I'd suggest as a gating factor: > > > > > > > > 1) Building on all supported arches > > > > 2) Cross building on all supported arches > > > > 3) Proper identification of targeted machine (i.e. equivalent of the machine > > > > component of the current build system) > > > > > > > The question there is gating factor for what? Presumably not for merging > > > into the staging tree. But for merging into the main tree for releases? > > > I'd push back a little on that, as the new system does not interfere in > > > any way with the old, and by keeping it in a staging tree until it > > > reaches full feature parity will make the job considerably harder. For > > > example, it means that anyone submitting a new driver or library has to > > > submit the code and makefiles in one set and the meson patches in a > > > separate one for a separate build tree. It also makes it less likely > > > that people will try out the new system and find the issues with it, and > > > help fill in the gaps. While I can understand us not recommending the > > > new build system until it reaches feature parity, I think there are a > > > lot of benefits to be got by making it widely available, even if it's > > > incomplete. > > > > Yes, sorry, the implied "what" here is gating its introduction to mainline. I > > have no problem with this going into a development or staging branch/tree, only > > with it getting merged to mainline and becoming the primary build system today. > > I get that it makes reaching feature parity harder, but to not do so relegates > > anyone that hasn't had a chance to test the new build system to second class > > citizen status (or at least potentially does so). To be a bit more specific, I > > can see how energized you might be to get this in place now because you've > > tested it on a wide array on intel hardware, but I'm guessing if it went in > > today, people at IBM and Linaro would have to drop active developement to start > > switching their build environments over to the new system lest they get left out > > in the cold. I think its more about balacing where the hardship lies here. > > > > As I'm writing this, I wonder if a reasonable compromise couldn't involve the > > use of CI? That is to say, what if we integrated the build system now-ish, and > > stood up an official CI instance, that both: > > 1) built the dpdk in all style configuration, which mandated the use of the old > > build system (i.e. we implicitly mandate that the current build system stay > > working, and is not forgotten), and gate patch merges on that result. > > > > 2) add a test that any change to a meson file in mainline also include a change > > to the a Makefile > > > > I'm just spitballing here, but I'm looking for ways to enforce the continued use > > of the current build system above and beyond a verbal promise to do so. The > > idea is to ensure that it stays operational and primary to the development of > > dpdk until build system parity is reached. > > > > Yes, that all makes sense, and I agree that the main issue is how to > ensure/enforce the fact that the old build system stays as the main way > to build DPDK, until such time as the new system reaches parity. A CI > system like you describe would be the ideal case, and we have some parts > of that, just not enough I think to satify our needs. On the other hand, > while holding the new system in a staging tree till it reaches parity > would definitely achieve that objective I think the cost of that is too > high. > > Right now patchwork reports results of "Intel-compilation". I'm not > aware of any immediate plans to expand that - somebody please should if > there are. There is also the planned DPDK lab, but I'm not sure of the > schedule for that or if it will help in this instance. > > For now, I'll prepare a slightly-updated V2 patchset, with a view to > getting work started on the build-next tree. While that is going on, we > can maybe continue to think about how to put in suitable guard-rails for > ensuring old system compatibility for when it's pulled to mainline. > > Thanks again for raising your concerns about this. It's easy for me to > sometimes push too hard for something I'm enthusiastic about! :-) I do > want this migration done properly, so if you have the bandwidth I'd > appreciate any reviews or feedback you can give, as the work goes on. > > Regards, > /Bruce >