Re: [dpdk-dev] versioning and maintenance

[Neil Horman <nhorman@tuxdriver.com>]

On Thu, Nov 20, 2014 at 10:08:25PM +0100, Thomas Monjalon wrote:
> 2014-11-20 13:25, Neil Horman:
> > On Thu, Nov 20, 2014 at 06:09:10PM +0100, Thomas Monjalon wrote:
> > > 2014-11-19 10:13, Neil Horman:
> > > > On Wed, Nov 19, 2014 at 11:35:08AM +0000, Bruce Richardson wrote:
> > > > > On Wed, Nov 19, 2014 at 12:22:14PM +0100, Thomas Monjalon wrote:
> > > > > > Following the discussion we had with Neil during the conference call,
> > > > > > I suggest this plan, starting with the next release (2.0):
> > > > > > 	- add version numbers to libraries
> > > > > > 	- add version numbers to functions inside .map files
> > > > > > 	- create a git tree dedicated to maintenance and API compatibility
> > > > > > 
> > > > > > It means these version numbers must be incremented when breaking the API.
> > > > > > Though the old code paths will be maintained and tested separately by volunteers.
> > > > > > A mailing list for maintenance purpose could be created if needed.
> > > > > >
> > > > > Hi Thomas,
> > > > > 
> > > > > I really think that the versionning is best handled inside the main repository
> > > > > itself. Given that the proposed deprecation policy is over two releases i.e. an
> > > > > API is marked deprecated in release X and then removed in X+1, I don't see the
> > > > > maintaining of old code paths to be particularly onerous.
> > > > > 
> > > > > /Bruce
> > > > 
> > > > I agree with Bruce, even if it is on occasion an added workload, its not the
> > > > sort of thing that can or should be placed on an alternate developer.  Backwards
> > > > compatibility is the sort of thing that has to be on the mind of the developer
> > > > when modifying an API, and on the mind of the reviewer when reviewing code.  To
> > > > shunt that responsibility elsewhere invites the opportunity for backwards
> > > > compatibilty to be a second class citizen who's goal will never be reached,
> > > > because developers instituting ABI changes will never care about the
> > > > consequences, and anyone worrying about backwards compatibility will always be
> > > > playing catch up, possibly allowing ABI breaks to slip through.
> > > > 
> > > > Neil
> > >  
> > > Before taking a decision, we should detail every concern.
> > > 
> > > 1/
> > > Currently there are not a lot of API refactoring because DPDK is well tailored
> > > for x86 and Intel NICs. But we are seeing that new CPU and new NICs to support
> > > would require some adaptations.
> > > 
> > Yes, you're absolutely right here.  I had hoped that, during my presentation
> > that this would happen occasionaly, and that we would need to deal with it.
> > What I think you are implying here (correct me if I'm wrong), is that you would
> > advocate that we wait to introduce ABI versioning until after such refactoring
> > is, for lack of a better term "complete".  The problem here is that, software
> > that is growing in user base is never "complete".  What you are effectively
> > saying is that you want to wait until the API is in a state in which no (or
> > almost no) more changes are required, then fixate it.  Thats quite simply never
> > going to happen.  And if it does, it obviates the need for versioning at all.
> 
> I agree Neil. This point is not about how long we should wait but how the
> overhead could be estimate for coming releases.
> 
Well, I understand the desire, but I'm not sure how it can be accomplished.  For
a given release, the overhead will be dependent on two factors:

1) The number off ABI changes in a given release

2) The extent of the ABI changes that were made.

If we have a way to predict those, then we can estimate the overhead, but
without that information, you're kinda stuck.  That said, if we all concur that
this is a necessecary effort to undertake, then the overhead is, not overly
important.  Whats more important is providing enough time to alot enough time to
do the work for a given project.  That is to say, when undertaking a large
refactoring, or other project that promises to make significant ABI changes,
that the developer needs to factor in time to design an implement backwards
compatibility.  Put another way, if the developer does their job right, and
takes backwards compatibility seriously, the overhead to you as a maintainer is
nil.  The onus to handle this extra effort needs to be on the developer.

> > > 2/
> > > I'm curious to know how you would handle a big change like the recent mbuf rework.
> > > Should we duplicate the structure and all the functions using mbuf?
> > 
> > Several ways, what you suggest above is one way, although thats what I would
> > consider to be a pessimal case.  Ideally such large changes are extreemely rare
> > (a search of the git history I think confirms this).  Much more common are
> > small, limited changes to various API's for which providing multiple versions of
> > a function is a much more reasonable approach.
> > 
> > In the event that we do decide to do a refactor that is so far reaching that we
> > simply don't feel like multi-versioning is feasible, the recourse is then to
> > deprecate the old API, publish that information on the deprecation schedule,
> > wait for a release, then replace it wholesale.  When the API is released, we
> > bump the DSO version number.  Note the versioning policy never guarantees that
> > backwards compatibility will always be available, nor does it stipulate that a
> > newer version of the API is available prior to removing the old one. The goal
> > here is to give distributors and application vendors advanced notice of ABI
> > breaking changes so that they can adapt appropriately before they are caught off
> > guard.  If the new ABI can't be packaged alongside the old, then so be it,
> > downstream vendors will have to use the upstream git head to test and validate,
> > rather than a newer distribution release
> 
> Seems reasonable.
> 
> > Ideally though, that shouldn't happen, because it causes downstream headaches,
> > and we would really like to avoid that.  Thats why I feel its so important to
> > keep this work in the main tree.  If we segregate it to a separate location it
> > will make it all to easy for developers to ignore these needs and just assume we
> > constantly drop old ABI versions without providing backwards compatibility.
> > 
> > > 3/
> > > Should we add new fields at the end of its structure to avoid ABI breaking?
> > > 
> > In the common case yes, this usually avoids ABI breakage, though it can't always
> > be relied upon (e.g. cases where structures are statically allocated by an
> > application).  And then there are patches that attempt to reduce memory usage
> > and increase performance by re-arranging structures.  In those cases we need to
> > do ABI versioning or announce/delay/release as noted above, though again, that
> > should really be avoided if possible.
> 
> So there is no hope of having fields logically sorted.
> Not a major problem but we have to know it. And it should probably be
> documented if we choose this way.
> 
Sure, though I'm not sure I agree with the statement above.  Having fields
logically sorted seems like it should be a forgone  conclusion in that the
developer should have laid those fields out in some semblance of order in the
first place.  If a large data structure re-ordering is taking place such that
structure fields are getting rearranged, that in my mind is part of a large
refactoring for which the entire API that is affected by those data structures
must have a new version created to provide backward compatibility, or in the
extreeme case, we may need to preform a warn and deprecate/exchange operation as
noted previously, though again, that is a Nuclear option.

> > > 4/
> > > Developers contribute because they need some changes. So when breaking
> > > an API, their application is already ready for the new version.
> > > I mean the author of such patch is probably not really motivated to keep ABI
> > > compability and duplicate the code path.
> > > 
> > What?  That doesn't make any sense.  Its our job to enforce this requirement on
> > developers during the review cycle.  If you don't feel like we can enforce
> > coding requirements on the project, we've already lost.  I agree that an
> > application developer submitting a patch for DPDK might not care about ABI
> > compatibility because they've already modified their application, but they (and
> > we) need to recognize that there are more than just a handful of users of the
> > DPDK, some of whom don't participate in this community (i.e. are simply end
> > users).  We need to make sure all users needs are met.  Thats the entire point
> > of this patch series, to make DPDK available to a wider range of users.
> 
> Exact. To make it simple, you care about end users and I have to care about
> developers motivation. But I perfectly understand the end users needs.
> I don't say we cannot enforce coding requirements. I just think it will be
> less pleasant.
> 
I disagree with the assertion that you will loose developers becausee they don't
care about compatibility.  You're developer base may change.  This is no
different than any other requirement that you place on a developer.  You make
all sorts of mandates regarding development (they can't break other older
supported cpu architecture, their code has to compile on all configurations,
etc).  This is no different.

> > > 5/
> > > Intead of simply modifying an API function, it would appear as a whole new
> > > function with some differences compared to the old one. Such change is really
> > > not convenient to review.
> > 
> > Um, yes, versioning is the process of creating an additional
> > function that closely resembles an older version of the same function, but with
> > different arguments and a newer version number.  Thats what it is by defintion,
> > and yes, its additional work.  All you're saying here is that, its extra work
> > and we shouldn't do it.  I thought I made this clear on the call, its been done
> > in thousands of other libraries, but if you just don't want to do it, then you
> > should abandon distributions as a way to reach a larger community, but if you
> > want to see the DPDK reach a larger community, then this is something that has
> > to happen, hard or not.
> 
> The goal of this discussion is to establish all the implications of this
> decision. We expose the facts. No conclusion.
> 
You haven't exposed a fact, you've asserted an opinion.  Theres is no notion of
something being convienient or inconvienient to review in any quantitative way.
If facts are your goal, you missed the mark here.

> > > 6/
> > > Testing ABI compatibility could be tricky. We would need a tool to check it's
> > > mostly OK. The good place for such a tool is in app/test. It was designed to be
> > > the unit tests of the API.
> > 
> > That seems like a reasonable idea, but I'm not sure what the concern is.  Are
> > you saying that you need to test every old version of the ABI?  Thats fine.  I
> > really don't think it has to be as stringent as the latest version testing, but
> > if you want to, it should be as easy as building the latest release of
> > the DPDK libraries, and the previous version of the test application.  That will
> > force the previous version code paths to be used by the test app in the new
> > library and, if the test fully exercize the api, then you should get pretty good
> > coverage.
> 
> Yes it will provide an unit test to developpers.
> 
> > > 7/
> > > This system would allow application developpers to upgrade DPDK to n+1 without
> > > rebuilding. But when upgrading to n+2, they should have adapted their
> > > application to comply with n+1 API (because n will be removed).
> > 
> > Only assuming that the old ABI facet was deprecated at the same time the new ABI
> > was introduced.  Theres nothing that says we have to do that, but I digress.
> > 
> > > So this solution offers a delay between the upgrade decision and the
> > > app work. Note that they could prepare their application before upgrading.
> > > Anyway, an upgrade should be tested before doing it effectively. The behaviour
> > > of the application could change and require some adaptations.
> > > 
> > Um, yes.  Whats the concern here?
> 
> I'm just trying to figure which workflows are eased by progressive ABI deprecation.
> 
The workflow for end users, in that they are given an alert prior to a breaking
change, and the time to fix it, in a way that distributions can manage without
having to individually (as distributions) undertake that effort on their own, an
in a way that might one day provide for multi version compatibility.

> > Downstream application developers need 2
> > things:
> > 
> > A) The ability to note that ABI changes are comming so that they can adapt to
> > the new version
> > 
> > B) Time to do so
> > 
> > The deprecation policy, if properly distributed by Distributions provide (A),
> > and the ABI versioning provides (B).  I.e. they can get all the latest bug fixes
> > and enhancements while in parallel adapting to the comming new version. Note
> > ideally this will happen rarely, as having to constantly rebuild/adapt does not
> > sit will with application vendors who choose to go through distributions, but
> > we'll do the best we can.
> 
> It's an interesting point. In a long-term distribution model like RHEL, do you
> plan to upgrade DPDK at each new release?
> 
Given that you intermix hardware support with bug fixes and new features (which
granted is not uncommon), yes, I don't see any way to avoid doing so.  We could
of course cherry pick bug fixes and non-ABI-breaking features, to preserve
compatibility, but doing so diverges from upstream quickly to the point that it
becomes extreemely difficult to maintain.  As an example, the one project that
Red Hat does this on routinely is the kernel, and to do so employs a staff of
hundreds of engineers.  No distribution wants to do that for every user space
library that they support.  They/we are willing to do minor fixes in a given
release with the foreknoweldge that we can drop them when the next relase comes
out, but beyond that, the logistics just don't scale.

> > > 8/
> > > How to handle a change in the implementation of a PMD which severely impact
> > > the application? Example: an ol_flag was mis-used and the application has
> > > a workaround to interpret this flag but it's now incompatible with the fix.
> > > 
> > We run into this sometimes in Fedora and RHEL, and doesn't require versioning.
> > The problem you describe is one in which something internal to the library that
> > an application has come to rely on.  Fixing the bug isn't typically considered
> > within the purview of versioning, because you're not changing the ABI, you're
> > just correcting a bug in the PMD's behavior.  Customers who ask for the behavior
> > to remain unchanged are asking for what's commonly referred to as "Bug for Bug
> > compatibility" and in those cases the application vendor needs to release a
> > corresponding fix.  Developers can't be required to preserve buggy behavior.
> > 
> > It should also be noted that in this case, ABI never changed.  All the data
> > types/sizes/locations/etc have remained unchanged. Its just a bug in
> > interpretation of data passed accross the ABI. As such, theres nothing for ABI
> > versioning to do here.
> 
> OK, that's what I thought.
> 
> > > 9/
> > > When we don't want to adapt an application, it means the development is
> > > finished and we don't care about the new features of the library.
> > > So I wonder if it wouldn't be more appropriate to provide stable releases
> > > with true maintenance to such users. I understood that is what Redhat provides
> > > to their customers.
> > > 
> > No, thats incorrect, we frequently update packages to the latest upstream
> > version when at all possible.  We are able to do this sepcifically because
> > upstream library releases provide ABI versioning, so that we can update with
> > confidence.  If they don't do that, then yes, we are often restricted to
> > selecting a release and maintaining it for the duration of a major RHEL release,
> > which implies that security and feature updates are extreemely limited
> > 
> > That said, if you wanted to do ongoing maintenence on each release, I suppsose
> > you could, in fact its somewhat simmilar to the -stable series that the kernel
> > uses, exept that the kernel enoys an extreemly stable user space ABI, and even
> > then the kernel -stable series doesn't take internal ABI changing patches, so
> > theres alot of divergence.  You don't currently have that stable ABI interface,
> > and so I think you'll find that that doing this is way more work than just
> > supporting versioning.
> > 
> > To illustrate, lets say you want to support maintenence releases the latest 3
> > releases of the DPDK with patches.  To do this, for every patch that is posted
> > to the dpdk that is a bug fix, you will have to apply it four times, one for
> > the git head, and again for each of the three releases that you are doing
> > maintenence on.  the patch will of course apply cleanly to the git head, as
> > thats what the developer wrote it against, but the other three releases have
> > every opportunity to conflict with code introduced in the git head but that
> > couldn't be taken into the maintenece releases.  Fixing those up is work that
> > you will either have to do, or request that the patch author do.  And for this
> > work you will provide distibutions with about 2 years of ABI stability
> > (presuming an ~8 month release cycle), after which they are back to just living
> > with whatever they stabilized on until the next major relase (note a single RHEL
> > major release has a 10+ year life cycle).  I would personally rather avoid that
> > work, and just do the ABI compatibility, as those patches are far fewer in
> > number, and it buys for the effort.
> 
> Interesting point of view.
> Note that there is no plan to maintain stable version on dpdk.org.
> But if some volunteers want absolutely to do it (even after reading your comment),
> we cannot forbid it.
>
Certainly, and as I noted the kernel does that.  But given the rate of change
that the DPDK undergoes, and the current size of the community, I don't think
anyone is going to step up to do that work.  Thats really the underlying problem
here, you can solve this problem lots of ways if you have enough manpower, but
given the resources at hand, doing versioning in the master tree is really the
only viable solution.
 
> > > Hope this discussion will bring a clear idea of what should be done with
> > > which implications.
> > > Thanks
> Thanks again
>