Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability

[Luca Boccassi <bluca@debian.org>]

On Thu, 2019-04-04 at 11:54 +0100, Bruce Richardson wrote:
> On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote:
> > On 03-Apr-19 4:42 PM, Ray Kinsella wrote:
> > > Hi folks,
> > > 
> > > Recently I started a discussion with the DPDK Technical Board on
> > > DPDK
> > > ABI/API stability. This was born out informal feedback I had
> > > received
> > > from a number of users of DPDK about ABI churn. In turn this
> > > feedback
> > > then prompted an ABI analysis of DPDK using tools from abi-
> > > laboratory.
> > > 
> > > https://abi-laboratory.pro/index.php?view=timeline&l=dpdk
> > > 
> > > 
> > > I guess the short story is that DPDK ABI hasn't really settled
> > > down as
> > > the project has matured. If you take a look at the “Backward
> > > Compat.”
> > > column which measures ABI compatibility compared to the previous
> > > releases, you will see significant churn in the ABI over
> > > successive
> > > releases since v16.04.
> > > 
> > > Now compare DPDK to GStreamer as an example of a very mature
> > > project
> > > with a similar intent, a framework for building applications, and
> > > which
> > > enjoys a very stable API.
> > > 
> > > https://abi-laboratory.pro/index.php?view=timeline&l=gstreamer
> > > 
> > > 
> > > The DPDK ABI churn has the following affects for users:-
> > > 
> > > 1. The churn obliges users of DPDK to commit to a constant
> > > re-integration and re-validation effort for new versions of DPDK.
> > > This
> > > effort from their perspective may not add value to their
> > > consuming
> > > project, particular if they are only updating to "stay current".
> > > 2. The churn encourages users of DPDK to slip versions, putting
> > > off
> > > reintegration to later, building up technical debt and causing
> > > their
> > > projects to miss support for new hardware or features.
> > > 3. It makes DPDK different to almost every other system library
> > > and
> > > framework that an operating systems might ship. This makes DPDK
> > > trickier
> > > to dynamically link against, package and maintain for OS
> > > maintainers.
> > > 
> > > In order to address this issue, I have put together the minimal
> > > set of
> > > concrete proposals below for discussion at the Technical Board
> > > next
> > > Wednesday.
> > > 
> > > I wanted to share this, as these might not yet be the right
> > > proposals,
> > > however I am putting them out there for feedback to start the
> > > discussion.
> > > 
> > > Thanks,
> > > 
> > > Ray K
> > > 
> > > 
> > > Experimental API
> > > 1.	APIs designated as experimental are not considered part
> > > of the ABI
> > > and may change without warning at any time.
> > > 2.	APIs designated as experimental must be marked
> > > depreciated for a
> > > least one quarterly release before removal.
> > > 3.	APIs designated as experimental will no longer
> > > automatically graduate
> > > to core after one release, they may stay experimental until their
> > > author
> > > and the maintainer agree that graduation is appropriate.
> > > 
> > > Core API (non-experimental API)
> > > 4.	APIs designated as core must be depreciated for a least
> > > two years
> > > before removal, to facilitate the continued compatibility with
> > > LTS
> > > releases. A final removal notice will be published to the DPDK
> > > Mailing
> > > List, and if there are no strong objections only then an API may
> > > be
> > > removed.
> > > 5.	APIs designated as core may be changed as follows:-
> > > 5.a	The change proposer must demonstrated that the change
> > > has a
> > > supporting use case and could not be achieved in any other way.
> > > 5.b	ABI version compatibility must be retained, as
> > > described below.
> > 
> > Hi Ray,
> > 
> > My somewhat rambly 2 cents :)
> > 
> > While i think some solution has to be found for the situation, we
> > also have
> > to balance this against speed of development and new features
> > rollout.
> > 
> > For example, let's consider what i am intimately familiar with -
> > the memory
> > rework. I have made enormous efforts to ensure that pre-18.05 and
> > post-18.05
> > remain as ABI/API compatible as possible, but there were a couple
> > of API
> > calls that were removed, and there couldn't have been any
> > replacements
> > (these API's were exposing internal structures that shouldn't have
> > been
> > exposed in the first place), and 18.05 also broke the ABI
> > compatibility,
> > because there was no way to do it without it (shared internal
> > structures
> > needed to change in part to support multiprocess).
> > 
> > So, if i understand your proposal correctly, assuming a 2-year
> > waiting
> > period for the deprecation of core API's, you would essentially
> > still be
> > waiting for the memory rework to land for a year more. Moreover,
> > even
> > *after* it has landed, there was a continuous stream of
> > improvements and
> > bugfixes, some of which has broke ABI compatibility as well. Some
> > of them
> > were my fault (as in, i could've foreseen the need for those
> > changes, but
> > didn't), but others came as a result of people using these new
> > features in
> > the wild and reporting issues/problems/suggestions - i am but one
> > man, after
> > all. Plus, you know, there's only 24 hours in a day, and some stuff
> > takes
> > time to implement :)
> > 
> > Since this rework goes right at the heart of DPDK (arguably there
> > isn't a
> > more "core" API than memory!), there is no (sane) way in the
> > universe to 1)
> > keep backwards compatibility for this, or 2) keep two parallel
> > versions of
> > it. We also need to test all that, and, to be honest, one
> > validation cycle
> > for a release wouldn't be enough to figure out all of the kinks and
> > implications of such a case. It was really great that memory rework
> > has
> > landed in 18.05 and we had time to improve and prepare it for an
> > 18.11 LTS -
> > i think everyone can say that it's in much better shape in 18.11
> > than it was
> > in 18.05, but if we couldn't do an ABI break here or there, this
> > rate of
> > improvements would have slowed down significantly.
> > 
> > Now, i understand that this is probably a highly exceptional case,
> > but i'm
> > sure that maintainers of other parts of DPDK will have their own
> > examples of
> > similar things happening.
> > 
> > I have no idea what a proper solution would look like. Any
> > "splitting" of
> > the trees into "experimental" vs. "stable" will end up causing the
> > same
> > issue - people choose to use stable over experimental because,
> > well, it's
> > more stable, and new/experimental features don't get tested as much
> > because
> > no one runs the thing in the first place.
> > 
> > TL;DR we have to be careful not to constrain the pace of
> > development/bugfixing just for the sake of having a stable API/ABI
> > :)
> > 
> 
> Actually, I think we *do* need to constrain the pace of development
> for the
> sake of ABI stability. At this stage DPDK has been around for quite a
> number of years and so should be considered a fairly mature project -
> it
> should just start acting like it.
> 
> Now, in terms of features like the memory rework, that is indeed a
> case
> that there was no alternative other than a massive ABI break.
> However, for
> that rework there was a strong need for improvement in that area that
> we
> can make the case for an ABI break to support it - and it is of a
> scale
> that nothing other than an ABI change would do. For other areas and
> examples, I doubt there are many in the last couple of years that are
> of
> that scale.

Fully agree.

It's normal for new project, big and small, to start without a
stability promise as development ramps up, and then steer toward
stability as the user base grows. Sometimes the switch is made explicit
by crossing from a 0.x to a 1.x version numbering, sometimes it's not.
I don't think we crossed that boundary yet in this project, and I
believe that's the main question Ray was trying to raise: has the time
finally come for DPDK to do this phase shift?

Of course it comes with a price for all developers, and that's again
common.

> My thoughts on the matter are:
> 1. I think we really need to do work to start hiding more of our data
> structures - like what Stephen's latest RFC does. This hiding should
> reduce
> the scope for ABI breaks.

Yes, I'm a big fan of accessors and opaque structs.

> 2. Once done, I think we should commit to having an ABI break only in
> the
> rarest of circumstances, and only with very large justification. I
> want us
> to get to the point where DPDK releases can immediately be picked up
> by all
> linux distros and rolled out because they are ABI compatible.
> 
> I'm not sure I like the idea of planned ABI break releases - that
> strikes
> me as a plan where we end up with the same number of ABI breaks as
> before,
> just balled into one release.

I think that was intended as a compromise, especially as we move from
one model to the other, and more of a "if a breakage has to happen, it
must be in the X release" rather than "let's always break in the X
release" :-)

> Question for Kevin, Luca and others who look at distro-packaging: is
> it the
> case that each distro will only ship one version of DPDK, or is it
> possible
> that if we have ABI breaks, a distro will provide two copies of DPDK
> simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version?

We can ship multiple versions, although it's more work so there should
be a good reason to do it. At the moment in Debian and Ubuntu we don't,
and we tend to ship whatever the latest LTS version is at the distro
freeze milestone - for example Debian 10 which will be released soon
(TM) will have 18.11.0.

> So, in short, I'm generally in favour of a zero-tolerance approach
> for DPDK
> ABI breaks, and having ABI breaks as a major event reserved only for
> massive rework changes, such as major mbuf changes, or new memory
> layout or
> similar.
> 
> Regards,
> /Bruce
> 
-- 
Kind regards,
Luca Boccassi