From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f170.google.com (mail-wi0-f170.google.com [209.85.212.170]) by dpdk.org (Postfix) with ESMTP id 2DB5F7F0C for ; Fri, 21 Nov 2014 14:13:06 +0100 (CET) Received: by mail-wi0-f170.google.com with SMTP id bs8so2699163wib.3 for ; Fri, 21 Nov 2014 05:23:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:organization :user-agent:in-reply-to:references:mime-version :content-transfer-encoding:content-type; bh=lQE4Mr/jC/BOjoL5NGNxTDcFX1x8IrCvYqTF6iI2c6U=; b=B6mR7bd2ciitWkdOyRPikgO2q/LSKTf9wJ9JL1bJVZsCCxeF5Knpzs/2nrKR0/IEF2 XdF5Dhp2j4Cagprid8Q8d6uIOYB/kncs8mtph3t9cg4eaaqvd5XGn8fevAr+I5ZnVoLe ZUEry7pF+47OXSObk/8JIyiTujMy87fUvmVuS1boeW0iCZMg68xX9q0wo7G3nvE6oZTH GfqZuVW+1tpTve2DE/b9d9K8SzbKslHq+lL61BtbeYoPCcELYsTDtAbjfXh1PJ70lJEm NsPFAeXFOAmheETNUI5BkCoSdkBKQRpca4navOjRuPzfYWY5e5DyEDt+/5fnC+dJy3ZQ kItQ== X-Gm-Message-State: ALoCoQnesFx8RnlSIZPNQC/e6CqeTalRNVZKyK/uA+ej6pxprXOomgI+4Lk8yXf5cLDlw35k8hHe X-Received: by 10.194.174.40 with SMTP id bp8mr7458927wjc.104.1416576220493; Fri, 21 Nov 2014 05:23:40 -0800 (PST) Received: from xps13.localnet ([109.190.92.136]) by mx.google.com with ESMTPSA id dx8sm8785745wib.2.2014.11.21.05.23.38 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Nov 2014 05:23:39 -0800 (PST) From: Thomas Monjalon To: Neil Horman Date: Fri, 21 Nov 2014 14:23:15 +0100 Message-ID: <2381044.cADuSdku07@xps13> Organization: 6WIND User-Agent: KMail/4.14.2 (Linux/3.17.2-1-ARCH; KDE/4.14.2; x86_64; ; ) In-Reply-To: <20141121010516.GA2599@localhost.localdomain> References: <5606183.AkpK27L7yz@xps13> <3636200.3Mr6UReKZ5@xps13> <20141121010516.GA2599@localhost.localdomain> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Cc: dev@dpdk.org Subject: Re: [dpdk-dev] versioning and maintenance X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2014 13:13:06 -0000 2014-11-20 20:05, Neil Horman: > On Thu, Nov 20, 2014 at 10:08:25PM +0100, Thomas Monjalon wrote: > > 2014-11-20 13:25, Neil Horman: > > > On Thu, Nov 20, 2014 at 06:09:10PM +0100, Thomas Monjalon wrote: > > > > 2014-11-19 10:13, Neil Horman: > > > > > On Wed, Nov 19, 2014 at 11:35:08AM +0000, Bruce Richardson wrote: > > > > > > On Wed, Nov 19, 2014 at 12:22:14PM +0100, Thomas Monjalon wrote: > > > > > > > Following the discussion we had with Neil during the conference call, > > > > > > > I suggest this plan, starting with the next release (2.0): > > > > > > > - add version numbers to libraries > > > > > > > - add version numbers to functions inside .map files > > > > > > > - create a git tree dedicated to maintenance and API compatibility > > > > > > > > > > > > > > It means these version numbers must be incremented when breaking the API. > > > > > > > Though the old code paths will be maintained and tested separately by volunteers. > > > > > > > A mailing list for maintenance purpose could be created if needed. > > > > > > > > > > > > > Hi Thomas, > > > > > > > > > > > > I really think that the versionning is best handled inside the main repository > > > > > > itself. Given that the proposed deprecation policy is over two releases i.e. an > > > > > > API is marked deprecated in release X and then removed in X+1, I don't see the > > > > > > maintaining of old code paths to be particularly onerous. > > > > > > > > > > > > /Bruce > > > > > > > > > > I agree with Bruce, even if it is on occasion an added workload, its not the > > > > > sort of thing that can or should be placed on an alternate developer. Backwards > > > > > compatibility is the sort of thing that has to be on the mind of the developer > > > > > when modifying an API, and on the mind of the reviewer when reviewing code. To > > > > > shunt that responsibility elsewhere invites the opportunity for backwards > > > > > compatibilty to be a second class citizen who's goal will never be reached, > > > > > because developers instituting ABI changes will never care about the > > > > > consequences, and anyone worrying about backwards compatibility will always be > > > > > playing catch up, possibly allowing ABI breaks to slip through. > > > > > > > > > > Neil > > > > > > > > Before taking a decision, we should detail every concern. > > > > > > > > 1/ > > > > Currently there are not a lot of API refactoring because DPDK is well tailored > > > > for x86 and Intel NICs. But we are seeing that new CPU and new NICs to support > > > > would require some adaptations. > > > > > > > Yes, you're absolutely right here. I had hoped that, during my presentation > > > that this would happen occasionaly, and that we would need to deal with it. > > > What I think you are implying here (correct me if I'm wrong), is that you would > > > advocate that we wait to introduce ABI versioning until after such refactoring > > > is, for lack of a better term "complete". The problem here is that, software > > > that is growing in user base is never "complete". What you are effectively > > > saying is that you want to wait until the API is in a state in which no (or > > > almost no) more changes are required, then fixate it. Thats quite simply never > > > going to happen. And if it does, it obviates the need for versioning at all. > > > > I agree Neil. This point is not about how long we should wait but how the > > overhead could be estimate for coming releases. > > > Well, I understand the desire, but I'm not sure how it can be accomplished. For > a given release, the overhead will be dependent on two factors: > > 1) The number off ABI changes in a given release > > 2) The extent of the ABI changes that were made. > > If we have a way to predict those, then we can estimate the overhead, but > without that information, you're kinda stuck. That said, if we all concur that > this is a necessecary effort to undertake, then the overhead is, not overly > important. Whats more important is providing enough time to alot enough time to > do the work for a given project. That is to say, when undertaking a large > refactoring, or other project that promises to make significant ABI changes, > that the developer needs to factor in time to design an implement backwards > compatibility. Put another way, if the developer does their job right, and > takes backwards compatibility seriously, the overhead to you as a maintainer is > nil. The onus to handle this extra effort needs to be on the developer. > > > > > 2/ > > > > I'm curious to know how you would handle a big change like the recent mbuf rework. > > > > Should we duplicate the structure and all the functions using mbuf? > > > > > > Several ways, what you suggest above is one way, although thats what I would > > > consider to be a pessimal case. Ideally such large changes are extreemely rare > > > (a search of the git history I think confirms this). Much more common are > > > small, limited changes to various API's for which providing multiple versions of > > > a function is a much more reasonable approach. > > > > > > In the event that we do decide to do a refactor that is so far reaching that we > > > simply don't feel like multi-versioning is feasible, the recourse is then to > > > deprecate the old API, publish that information on the deprecation schedule, > > > wait for a release, then replace it wholesale. When the API is released, we > > > bump the DSO version number. Note the versioning policy never guarantees that > > > backwards compatibility will always be available, nor does it stipulate that a > > > newer version of the API is available prior to removing the old one. The goal > > > here is to give distributors and application vendors advanced notice of ABI > > > breaking changes so that they can adapt appropriately before they are caught off > > > guard. If the new ABI can't be packaged alongside the old, then so be it, > > > downstream vendors will have to use the upstream git head to test and validate, > > > rather than a newer distribution release > > > > Seems reasonable. > > > > > Ideally though, that shouldn't happen, because it causes downstream headaches, > > > and we would really like to avoid that. Thats why I feel its so important to > > > keep this work in the main tree. If we segregate it to a separate location it > > > will make it all to easy for developers to ignore these needs and just assume we > > > constantly drop old ABI versions without providing backwards compatibility. > > > > > > > 3/ > > > > Should we add new fields at the end of its structure to avoid ABI breaking? > > > > > > > In the common case yes, this usually avoids ABI breakage, though it can't always > > > be relied upon (e.g. cases where structures are statically allocated by an > > > application). And then there are patches that attempt to reduce memory usage > > > and increase performance by re-arranging structures. In those cases we need to > > > do ABI versioning or announce/delay/release as noted above, though again, that > > > should really be avoided if possible. > > > > So there is no hope of having fields logically sorted. > > Not a major problem but we have to know it. And it should probably be > > documented if we choose this way. > > > Sure, though I'm not sure I agree with the statement above. Having fields > logically sorted seems like it should be a forgone conclusion in that the > developer should have laid those fields out in some semblance of order in the > first place. If a large data structure re-ordering is taking place such that > structure fields are getting rearranged, that in my mind is part of a large > refactoring for which the entire API that is affected by those data structures > must have a new version created to provide backward compatibility, or in the > extreeme case, we may need to preform a warn and deprecate/exchange operation as > noted previously, though again, that is a Nuclear option. Just to illustrate my thought: Let's imagine this struct { fish_name; fish_taste; vegetables_name; } When adding the new field "fish_cooking", we'll add it at the end to avoid ABI break. struct { fish_name; fish_taste; vegetables_name; fish_cooking; } So "fish_*" fields won't be grouped. It's mostly an esthetic/readability consequence. Now I'm hungry ;) > > > > 4/ > > > > Developers contribute because they need some changes. So when breaking > > > > an API, their application is already ready for the new version. > > > > I mean the author of such patch is probably not really motivated to keep ABI > > > > compability and duplicate the code path. > > > > > > > What? That doesn't make any sense. Its our job to enforce this requirement on > > > developers during the review cycle. If you don't feel like we can enforce > > > coding requirements on the project, we've already lost. I agree that an > > > application developer submitting a patch for DPDK might not care about ABI > > > compatibility because they've already modified their application, but they (and > > > we) need to recognize that there are more than just a handful of users of the > > > DPDK, some of whom don't participate in this community (i.e. are simply end > > > users). We need to make sure all users needs are met. Thats the entire point > > > of this patch series, to make DPDK available to a wider range of users. > > > > Exact. To make it simple, you care about end users and I have to care about > > developers motivation. But I perfectly understand the end users needs. > > I don't say we cannot enforce coding requirements. I just think it will be > > less pleasant. > > > I disagree with the assertion that you will loose developers becausee they don't > care about compatibility. You're developer base may change. This is no > different than any other requirement that you place on a developer. You make > all sorts of mandates regarding development (they can't break other older > supported cpu architecture, their code has to compile on all configurations, > etc). This is no different. > > > > > 5/ > > > > Intead of simply modifying an API function, it would appear as a whole new > > > > function with some differences compared to the old one. Such change is really > > > > not convenient to review. > > > > > > Um, yes, versioning is the process of creating an additional > > > function that closely resembles an older version of the same function, but with > > > different arguments and a newer version number. Thats what it is by defintion, > > > and yes, its additional work. All you're saying here is that, its extra work > > > and we shouldn't do it. I thought I made this clear on the call, its been done > > > in thousands of other libraries, but if you just don't want to do it, then you > > > should abandon distributions as a way to reach a larger community, but if you > > > want to see the DPDK reach a larger community, then this is something that has > > > to happen, hard or not. > > > > The goal of this discussion is to establish all the implications of this > > decision. We expose the facts. No conclusion. > > > You haven't exposed a fact, you've asserted an opinion. Theres is no notion of > something being convienient or inconvienient to review in any quantitative way. > If facts are your goal, you missed the mark here. Maybe you use a tool that I don't know. My main material for review is the patch. And I think it's simpler to check an one-line change than a duplicated code path. But instead of giving my opinion, I must expose what it looks like for a simple example: - void cook_fish() + void cook_fish(oil_bottle) { + use_oil(oil_bottle); start_fire(); put_fish(); wait(); stop_fire(); } vs - void cook_fish() + void __vsym cook_fish_v1() { start_fire(); put_fish(); wait(); stop_fire(); } + VERSION_SYMBOL(cook_fish, _v1, 1); + void cook_fish(oil_bottle) + { + use_oil(oil_bottle); + start_fire(); + put_fish(); + wait(); + stop_fire(); + } + BIND_DEFAULT_SYMBOL(cook_fish, 2); > > > > 6/ > > > > Testing ABI compatibility could be tricky. We would need a tool to check it's > > > > mostly OK. The good place for such a tool is in app/test. It was designed to be > > > > the unit tests of the API. > > > > > > That seems like a reasonable idea, but I'm not sure what the concern is. Are > > > you saying that you need to test every old version of the ABI? Thats fine. I > > > really don't think it has to be as stringent as the latest version testing, but > > > if you want to, it should be as easy as building the latest release of > > > the DPDK libraries, and the previous version of the test application. That will > > > force the previous version code paths to be used by the test app in the new > > > library and, if the test fully exercize the api, then you should get pretty good > > > coverage. > > > > Yes it will provide an unit test to developpers. > > > > > > 7/ > > > > This system would allow application developpers to upgrade DPDK to n+1 without > > > > rebuilding. But when upgrading to n+2, they should have adapted their > > > > application to comply with n+1 API (because n will be removed). > > > > > > Only assuming that the old ABI facet was deprecated at the same time the new ABI > > > was introduced. Theres nothing that says we have to do that, but I digress. > > > > > > > So this solution offers a delay between the upgrade decision and the > > > > app work. Note that they could prepare their application before upgrading. > > > > Anyway, an upgrade should be tested before doing it effectively. The behaviour > > > > of the application could change and require some adaptations. > > > > > > > Um, yes. Whats the concern here? > > > > I'm just trying to figure which workflows are eased by progressive ABI deprecation. > > > The workflow for end users, in that they are given an alert prior to a breaking > change, and the time to fix it, in a way that distributions can manage without > having to individually (as distributions) undertake that effort on their own, an > in a way that might one day provide for multi version compatibility. > > > > Downstream application developers need 2 > > > things: > > > > > > A) The ability to note that ABI changes are comming so that they can adapt to > > > the new version > > > > > > B) Time to do so > > > > > > The deprecation policy, if properly distributed by Distributions provide (A), > > > and the ABI versioning provides (B). I.e. they can get all the latest bug fixes > > > and enhancements while in parallel adapting to the comming new version. Note > > > ideally this will happen rarely, as having to constantly rebuild/adapt does not > > > sit will with application vendors who choose to go through distributions, but > > > we'll do the best we can. > > > > It's an interesting point. In a long-term distribution model like RHEL, do you > > plan to upgrade DPDK at each new release? > > > Given that you intermix hardware support with bug fixes and new features (which > granted is not uncommon), yes, I don't see any way to avoid doing so. We could > of course cherry pick bug fixes and non-ABI-breaking features, to preserve > compatibility, but doing so diverges from upstream quickly to the point that it > becomes extreemely difficult to maintain. As an example, the one project that > Red Hat does this on routinely is the kernel, and to do so employs a staff of > hundreds of engineers. No distribution wants to do that for every user space > library that they support. They/we are willing to do minor fixes in a given > release with the foreknoweldge that we can drop them when the next relase comes > out, but beyond that, the logistics just don't scale. > > > > > 8/ > > > > How to handle a change in the implementation of a PMD which severely impact > > > > the application? Example: an ol_flag was mis-used and the application has > > > > a workaround to interpret this flag but it's now incompatible with the fix. > > > > > > > We run into this sometimes in Fedora and RHEL, and doesn't require versioning. > > > The problem you describe is one in which something internal to the library that > > > an application has come to rely on. Fixing the bug isn't typically considered > > > within the purview of versioning, because you're not changing the ABI, you're > > > just correcting a bug in the PMD's behavior. Customers who ask for the behavior > > > to remain unchanged are asking for what's commonly referred to as "Bug for Bug > > > compatibility" and in those cases the application vendor needs to release a > > > corresponding fix. Developers can't be required to preserve buggy behavior. > > > > > > It should also be noted that in this case, ABI never changed. All the data > > > types/sizes/locations/etc have remained unchanged. Its just a bug in > > > interpretation of data passed accross the ABI. As such, theres nothing for ABI > > > versioning to do here. > > > > OK, that's what I thought. > > > > > > 9/ > > > > When we don't want to adapt an application, it means the development is > > > > finished and we don't care about the new features of the library. > > > > So I wonder if it wouldn't be more appropriate to provide stable releases > > > > with true maintenance to such users. I understood that is what Redhat provides > > > > to their customers. > > > > > > > No, thats incorrect, we frequently update packages to the latest upstream > > > version when at all possible. We are able to do this sepcifically because > > > upstream library releases provide ABI versioning, so that we can update with > > > confidence. If they don't do that, then yes, we are often restricted to > > > selecting a release and maintaining it for the duration of a major RHEL release, > > > which implies that security and feature updates are extreemely limited > > > > > > That said, if you wanted to do ongoing maintenence on each release, I suppsose > > > you could, in fact its somewhat simmilar to the -stable series that the kernel > > > uses, exept that the kernel enoys an extreemly stable user space ABI, and even > > > then the kernel -stable series doesn't take internal ABI changing patches, so > > > theres alot of divergence. You don't currently have that stable ABI interface, > > > and so I think you'll find that that doing this is way more work than just > > > supporting versioning. > > > > > > To illustrate, lets say you want to support maintenence releases the latest 3 > > > releases of the DPDK with patches. To do this, for every patch that is posted > > > to the dpdk that is a bug fix, you will have to apply it four times, one for > > > the git head, and again for each of the three releases that you are doing > > > maintenence on. the patch will of course apply cleanly to the git head, as > > > thats what the developer wrote it against, but the other three releases have > > > every opportunity to conflict with code introduced in the git head but that > > > couldn't be taken into the maintenece releases. Fixing those up is work that > > > you will either have to do, or request that the patch author do. And for this > > > work you will provide distibutions with about 2 years of ABI stability > > > (presuming an ~8 month release cycle), after which they are back to just living > > > with whatever they stabilized on until the next major relase (note a single RHEL > > > major release has a 10+ year life cycle). I would personally rather avoid that > > > work, and just do the ABI compatibility, as those patches are far fewer in > > > number, and it buys for the effort. > > > > Interesting point of view. > > Note that there is no plan to maintain stable version on dpdk.org. > > But if some volunteers want absolutely to do it (even after reading your comment), > > we cannot forbid it. > > > Certainly, and as I noted the kernel does that. But given the rate of change > that the DPDK undergoes, and the current size of the community, I don't think > anyone is going to step up to do that work. Thats really the underlying problem > here, you can solve this problem lots of ways if you have enough manpower, but > given the resources at hand, doing versioning in the master tree is really the > only viable solution. > > > > > Hope this discussion will bring a clear idea of what should be done with > > > > which implications. > > > > Thanks > > Thanks again I think my concerns are now well explained. It was important to expose clearly what a such ABI policy means. If nobody disagree with your approach, it should be accepted. Thanks -- Thomas