Re: [dpdk-dev] versioning and maintenance

[Neil Horman <nhorman@tuxdriver.com>]

On Thu, Nov 20, 2014 at 06:09:10PM +0100, Thomas Monjalon wrote:
> Hi,
> 
> 2014-11-19 10:13, Neil Horman:
> > On Wed, Nov 19, 2014 at 11:35:08AM +0000, Bruce Richardson wrote:
> > > On Wed, Nov 19, 2014 at 12:22:14PM +0100, Thomas Monjalon wrote:
> > > > Following the discussion we had with Neil during the conference call,
> > > > I suggest this plan, starting with the next release (2.0):
> > > > 	- add version numbers to libraries
> > > > 	- add version numbers to functions inside .map files
> > > > 	- create a git tree dedicated to maintenance and API compatibility
> > > > 
> > > > It means these version numbers must be incremented when breaking the API.
> > > > Though the old code paths will be maintained and tested separately by volunteers.
> > > > A mailing list for maintenance purpose could be created if needed.
> > > >
> > > Hi Thomas,
> > > 
> > > I really think that the versionning is best handled inside the main repository
> > > itself. Given that the proposed deprecation policy is over two releases i.e. an
> > > API is marked deprecated in release X and then removed in X+1, I don't see the
> > > maintaining of old code paths to be particularly onerous.
> > > 
> > > /Bruce
> > 
> > I agree with Bruce, even if it is on occasion an added workload, its not the
> > sort of thing that can or should be placed on an alternate developer.  Backwards
> > compatibility is the sort of thing that has to be on the mind of the developer
> > when modifying an API, and on the mind of the reviewer when reviewing code.  To
> > shunt that responsibility elsewhere invites the opportunity for backwards
> > compatibilty to be a second class citizen who's goal will never be reached,
> > because developers instituting ABI changes will never care about the
> > consequences, and anyone worrying about backwards compatibility will always be
> > playing catch up, possibly allowing ABI breaks to slip through.
> > 
> > Neil
>  
> Before taking a decision, we should detail every concern.
> 
> 1/
> Currently there are not a lot of API refactoring because DPDK is well tailored
> for x86 and Intel NICs. But we are seeing that new CPU and new NICs to support
> would require some adaptations.
> 
Yes, you're absolutely right here.  I had hoped that, during my presentation
that this would happen occasionaly, and that we would need to deal with it.
What I think you are implying here (correct me if I'm wrong), is that you would
advocate that we wait to introduce ABI versioning until after such refactoring
is, for lack of a better term "complete".  The problem here is that, software
that is growing in user base is never "complete".  What you are effectively
saying is that you want to wait until the API is in a state in which no (or
almost no) more changes are required, then fixate it.  Thats quite simply never
going to happen.  And if it does, it obviates the need for versioning at all.

> 2/
> I'm curious to know how you would handle a big change like the recent mbuf rework.
> Should we duplicate the structure and all the functions using mbuf?
> 

Several ways, what you suggest above is one way, although thats what I would
consider to be a pessimal case.  Ideally such large changes are extreemely rare
(a search of the git history I think confirms this).  Much more common are
small, limited changes to various API's for which providing multiple versions of
a function is a much more reasonable approach.

In the event that we do decide to do a refactor that is so far reaching that we
simply don't feel like multi-versioning is feasible, the recourse is then to
deprecate the old API, publish that information on the deprecation schedule,
wait for a release, then replace it wholesale.  When the API is released, we
bump the DSO version number.  Note the versioning policy never guarantees that
backwards compatibility will always be available, nor does it stipulate that a
newer version of the API is available prior to removing the old one. The goal
here is to give distributors and application vendors advanced notice of ABI
breaking changes so that they can adapt appropriately before they are caught off
guard.  If the new ABI can't be packaged alongside the old, then so be it,
downstream vendors will have to use the upstream git head to test and validate,
rather than a newer distribution release

Ideally though, that shouldn't happen, because it causes downstream headaches,
and we would really like to avoid that.  Thats why I feel its so important to
keep this work in the main tree.  If we segregate it to a separate location it
will make it all to easy for developers to ignore these needs and just assume we
constantly drop old ABI versions without providing backwards compatibility.

> 3/
> Should we add new fields at the end of its structure to avoid ABI breaking?
> 
In the common case yes, this usually avoids ABI breakage, though it can't always
be relied upon (e.g. cases where structures are statically allocated by an
application).  And then there are patches that attempt to reduce memory usage
and increase performance by re-arranging structures.  In those cases we need to
do ABI versioning or announce/delay/release as noted above, though again, that
should really be avoided if possible.

> 4/
> Developers contribute because they need some changes. So when breaking
> an API, their application is already ready for the new version.
> I mean the author of such patch is probably not really motivated to keep ABI
> compability and duplicate the code path.
> 
What?  That doesn't make any sense.  Its our job to enforce this requirement on
developers during the review cycle.  If you don't feel like we can enforce
coding requirements on the project, we've already lost.  I agree that an
application developer submitting a patch for DPDK might not care about ABI
compatibility because they've already modified their application, but they (and
we) need to recognize that there are more than just a handful of users of the
DPDK, some of whom don't participate in this community (i.e. are simply end
users).  We need to make sure all users needs are met.  Thats the entire point
of this patch series, to make DPDK available to a wider range of users.

> 5/
> Intead of simply modifying an API function, it would appear as a whole new
> function with some differences compared to the old one. Such change is really
> not convenient to review.
> 

Um, yes, versioning is the process of creating an additional
function that closely resembles an older version of the same function, but with
different arguments and a newer version number.  Thats what it is by defintion,
and yes, its additional work.  All you're saying here is that, its extra work
and we shouldn't do it.  I thought I made this clear on the call, its been done
in thousands of other libraries, but if you just don't want to do it, then you
should abandon distributions as a way to reach a larger community, but if you
want to see the DPDK reach a larger community, then this is something that has
to happen, hard or not.

> 6/
> Testing ABI compatibility could be tricky. We would need a tool to check it's
> mostly OK. The good place for such a tool is in app/test. It was designed to be
> the unit tests of the API.
> 

That seems like a reasonable idea, but I'm not sure what the concern is.  Are
you saying that you need to test every old version of the ABI?  Thats fine.  I
really don't think it has to be as stringent as the latest version testing, but
if you want to, it should be as easy as building the latest release of
the DPDK libraries, and the previous version of the test application.  That will
force the previous version code paths to be used by the test app in the new
library and, if the test fully exercize the api, then you should get pretty good
coverage.

> 7/
> This system would allow application developpers to upgrade DPDK to n+1 without
> rebuilding. But when upgrading to n+2, they should have adapted their
> application to comply with n+1 API (because n will be removed).
Only assuming that the old ABI facet was deprecated at the same time the new ABI
was introduced.  Theres nothing that says we have to do that, but I digress.

> So this solution offers a delay between the upgrade decision and the
> app work. Note that they could prepare their application before upgrading.
> Anyway, an upgrade should be tested before doing it effectively. The behaviour
> of the application could change and require some adaptations.
> 
Um, yes.  Whats the concern here?  Downstream application developers need 2
things:

A) The ability to note that ABI changes are comming so that they can adapt to
the new version

B) Time to do so

The deprecation policy, if properly distributed by Distributions provide (A),
and the ABI versioning provides (B).  I.e. they can get all the latest bug fixes
and enhancements while in parallel adapting to the comming new version. Note
ideally this will happen rarely, as having to constantly rebuild/adapt does not
sit will with application vendors who choose to go through distributions, but
we'll do the best we can.

> 8/
> How to handle a change in the implementation of a PMD which severely impact
> the application? Example: an ol_flag was mis-used and the application has
> a workaround to interpret this flag but it's now incompatible with the fix.
> 
We run into this sometimes in Fedora and RHEL, and doesn't require versioning.
The problem you describe is one in which something internal to the library that
an application has come to rely on.  Fixing the bug isn't typically considered
within the purview of versioning, because you're not changing the ABI, you're
just correcting a bug in the PMD's behavior.  Customers who ask for the behavior
to remain unchanged are asking for what's commonly referred to as "Bug for Bug
compatibility" and in those cases the application vendor needs to release a
corresponding fix.  Developers can't be required to preserve buggy behavior.

It should also be noted that in this case, ABI never changed.  All the data
types/sizes/locations/etc have remained unchanged. Its just a bug in
interpretation of data passed accross the ABI. As such, theres nothing for ABI
versioning to do here.

> 9/
> When we don't want to adapt an application, it means the development is
> finished and we don't care about the new features of the library.
> So I wonder if it wouldn't be more appropriate to provide stable releases
> with true maintenance to such users. I understood that is what Redhat provides
> to their customers.
> 
No, thats incorrect, we frequently update packages to the latest upstream
version when at all possible.  We are able to do this sepcifically because
upstream library releases provide ABI versioning, so that we can update with
confidence.  If they don't do that, then yes, we are often restricted to
selecting a release and maintaining it for the duration of a major RHEL release,
which implies that security and feature updates are extreemely limited

That said, if you wanted to do ongoing maintenence on each release, I suppsose
you could, in fact its somewhat simmilar to the -stable series that the kernel
uses, exept that the kernel enoys an extreemly stable user space ABI, and even
then the kernel -stable series doesn't take internal ABI changing patches, so
theres alot of divergence.  You don't currently have that stable ABI interface,
and so I think you'll find that that doing this is way more work than just
supporting versioning.

To illustrate, lets say you want to support maintenence releases the latest 3
releases of the DPDK with patches.  To do this, for every patch that is posted
to the dpdk that is a bug fix, you will have to apply it four times, one for
the git head, and again for each of the three releases that you are doing
maintenence on.  the patch will of course apply cleanly to the git head, as
thats what the developer wrote it against, but the other three releases have
every opportunity to conflict with code introduced in the git head but that
couldn't be taken into the maintenece releases.  Fixing those up is work that
you will either have to do, or request that the patch author do.  And for this
work you will provide distibutions with about 2 years of ABI stability
(presuming an ~8 month release cycle), after which they are back to just living
with whatever they stabilized on until the next major relase (note a single RHEL
major release has a 10+ year life cycle).  I would personally rather avoid that
work, and just do the ABI compatibility, as those patches are far fewer in
number, and it buys for the effort.


> Hope this discussion will bring a clear idea of what should be done with
> which implications.
> Thanks
> -- 
> Thomas
>