DPDK patches and discussions
 help / color / mirror / Atom feed
From: Neil Horman <nhorman@tuxdriver.com>
To: Thomas Monjalon <thomas.monjalon@6wind.com>
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility
Date: Wed, 24 Sep 2014 14:19:40 -0400	[thread overview]
Message-ID: <20140924181940.GB4651@hmsreliant.think-freely.org> (raw)
In-Reply-To: <20140918191401.GP20389@hmsreliant.think-freely.org>

On Thu, Sep 18, 2014 at 03:14:01PM -0400, Neil Horman wrote:
> On Thu, Sep 18, 2014 at 08:23:36PM +0200, Thomas Monjalon wrote:
> > Hi Neil,
> > 
> > 2014-09-15 15:23, Neil Horman:
> > > The DPDK ABI develops and changes quickly, which makes it difficult for
> > > applications to keep up with the latest version of the library, especially when
> > > it (the DPDK) is built as a set of shared objects, as applications may be built
> > > against an older version of the library.
> > > 
> > > To mitigate this, this patch series introduces support for library and symbol
> > > versioning when the DPDK is built as a DSO.  Specifically, it does 4 things:
> > > 
> > > 1) Adds initial support for library versioning.  Each library now has a version
> > > map that explicitly calls out what symbols are exported to using applications,
> > > and assigns version(s) to them
> > > 
> > > 2) Adds support macros so that when libraries create incompatible ABI's,
> > > multiple versions may be supported so that applications linked against older
> > > DPDK releases can continue to function
> > > 
> > > 3) Adds library soname versioning suffixes so that when ABI's must be broken in
> > > a fashion that requires a rebuild of older applications, they will break at load
> > > time, rather than cause unexpected issues at run time.
> > > 
> > > 4) Adds documentation for ABI policy, and provides space to document deprecated
> > > ABI versions, so that applications might be warned of impending changes.
> > > 
> > > With these elements in place the DPDK has some support to allow for the extended
> > > maintenence of older API's while still allowing the freedom to develop new and
> > > improved API's.
> > > 
> > > Implementing this feature will require some additional effort on the part of
> > > developers and reviewers.  When reviewing patches, must be checked against
> > > existing exports to ensure that the function prototypes are not changing.  If
> > > they are, the versioning macros must be used, and the library export map should
> > > be updated to reflect the new version of the function.
> > > 
> > > When data structures change, if those structures are application accessible,
> > > apis that accept or return instances of those data structures should have new
> > > versions created so that users of the old data structure version might co-exist
> > > at the same time.
> > 
> > Thanks for your efforts.
> > But I feel this change has too many constraints for the current status of
> > the DPDK. It's probably too early to adopt such policy.
> > 
> I think you may be misunderstanding something.  What constraints do you beleive
> that this patch imposes?  Note it doesn't in any way prevent changes to the ABI
> of the DPDK, but rather gives us infrastructure to support multiple ABI
> revisions at the same time, so that applications built against DPDK shared
> libraries can continue to function properly at least for some time until we
> decide to deprecate that ABI level.
> This is all based on the versioning strategy outlined here:
> http://www.akkadia.org/drepper/dsohowto.pdf
> That may help clarify things for you.
> > By the way, this versioning doesn't cover structure changes.
> No, it doesn't.  No link-time mechanism does so.
> > How could it be managed?
> Thats a subject that is open to discussion, but my initial thinking is that we
> need to handle it on a case by case basis:
> * For minor updates, where allocation of a structure is done on the heap and new
> fields need to be added, appending them to the end of a structure and providing
> an initial value is sufficient.
> * For major changes, where fields need to be removed, or re-arranged, mostly
> likely the API surfaces which accept or return those structures as
> inputs/outputs will need to have new versions written to accept the new version
> of the structure, and internally we will have to support both formats for a time
> (according to the policy I documented, that is currently a single major
> release).  I.e. if you want to change struct foo, which is accepted as a
> parameter for the function bar(struct foo *ptr), then for a release we would
> need to create struct foo_v2 with the new format, map a new function foo_v2 to
> the exported foo@@DPDK_1.(X+1), and internally make the foo functions understand
> both the origional and v2 versions of the structure.  Then in DPDK release
> 1.X+2, we can remove the old version after posting a deprecation notice with
> version 1.(X+1)
> > Don't you think it would be more reliable if managed by packaging?
> Solving this with packaging defeats the purpose of having shared libraries at
> all.  While packaging each version of the dpdk separately is possible stopgap
> solution, in that it allows applications to link to differing versions of the
> library independently, but that negates any expectation of timely bugfixes for
> any given version of the DPDK.  That is to say, if you package things this way,
> and wind up with several parallel versions of the same package, and for any
> bugfix that comes out upstream, the packager then has the responsibility to
> adapt that fix to each package.  Thats an unscalable solution, and not something
> any packager is going to undertake willingly.  I did a hybrid version of this in
> fedora for exactly that reason.  I packaged the dpdk into its own directory, but
> have every intention of changing that directory every major release, so that
> application writers can clearly see when they need to stop updating the dpdk,
> lest their applications stop linking. I'm not going to have multiple dpdk
> packages to maintain in parallel, thats just too much work.
> > 
> > Thank you for opening this discussion with a constructive proposal. 
> > Let's check it later on once structures will be more stable since 
> > performance is the most critical target.
> If I'm being honest, I have to say thats a cop out answer.  We all know that
> structure stability isn't a priority for the DPDK, nor will it ever be in all
> likelyhood.  It will continue to evolve and grow as the hardware does.  And this
> patch set doesn't prevent that from happening.  All it does is provide some
> level of stability in the API for a period of time to let 3rd party application
> writers write and package applications with some allowance of time to keep up
> with upstream changes on their own schedule.
> I grant you that writing a good API for a shared library is difficult, but
> (and feel free to disagree with this), if we don't start enforcing policies that
> require good API design, its not going to happen on its own.  This patch set
> will highlight those API points which are difficult to maintain accross major
> releases, and force us to address and improve them.  To that end I've already
> begun talking to some of the individual library maintainers off list to address
> some of the API aspects that I have concerns about (exporting variable rather
> than accessor functions, structures that don't need to be visible to users,
> etc), and they've started reviewing them.  We can make this better, but we can't
> just say later, because theres no roadmap that lists structure stability as a
> line item.  As hardware improves, structures will change to operate more
> efficiently or support more features.  Without a hard plan, the initial goals of
> the DPDK (high performance networking) will relegate ABI to such a low priority
> that it will never be addressed. 
> To that end, can we discuss specifics?  Can you ennumerate direct points that
> you feel make this patch unworkable at this time?  I know you mentioned some
> above, and I think I addressed them (though please ask follow up questions if
> I've been unclear).  What other concerns do you have?
> Neil
Ping Thomas. I know you're busy, but I would like this to not fall off anyones
radar.  You alluded to concerns regarding what, for lack of a better term,
ABI/API lockin.  I had asked you to enuumerate/elaborate on specifics, but never
heard back.  Are there further specifics you wish to discuss, or are you
satisfied with the above answers?


  parent reply	other threads:[~2014-09-24 18:13 UTC|newest]

Thread overview: 42+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-09-15 19:23 Neil Horman
2014-09-15 19:23 ` [dpdk-dev] [PATCH 1/4] compat: Add infrastructure to support symbol versioning Neil Horman
2014-09-23 10:39   ` Sergio Gonzalez Monroy
2014-09-23 14:58     ` Neil Horman
2014-09-23 16:29       ` Sergio Gonzalez Monroy
2014-09-23 17:31         ` Neil Horman
2014-09-25 18:52   ` [dpdk-dev] [PATCH 1/4 v2] " Neil Horman
2014-09-26 14:16     ` Sergio Gonzalez Monroy
2014-09-26 15:16       ` Neil Horman
2014-09-26 15:33         ` Sergio Gonzalez Monroy
2014-09-26 16:22           ` Neil Horman
2014-09-26 19:19             ` Neil Horman
2014-09-29 15:44   ` [dpdk-dev] [PATCH 1/4 v3] " Neil Horman
2014-09-30  8:13     ` Sergio Gonzalez Monroy
2014-09-30 15:18   ` [dpdk-dev] [PATCH 1/4 v4] " Neil Horman
2014-10-01 10:15     ` Sergio Gonzalez Monroy
2014-10-01 10:38       ` Neil Horman
2014-10-01 11:28     ` Sergio Gonzalez Monroy
2014-09-15 19:23 ` [dpdk-dev] [PATCH 2/4] Provide initial versioning for all DPDK libraries Neil Horman
2014-09-19  9:45   ` Bruce Richardson
2014-09-19 10:22     ` Neil Horman
2014-10-01 11:25   ` Sergio Gonzalez Monroy
2014-10-01 14:43     ` Neil Horman
2014-09-15 19:23 ` [dpdk-dev] [PATCH 3/4] Add library version extenstion Neil Horman
2014-10-01 11:27   ` Sergio Gonzalez Monroy
2014-09-15 19:23 ` [dpdk-dev] [PATCH 4/4] docs: Add ABI documentation Neil Horman
2014-10-01 16:06   ` Sergio Gonzalez Monroy
2014-09-18 18:23 ` [dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility Thomas Monjalon
2014-09-18 19:14   ` Neil Horman
2014-09-19  8:57     ` Richardson, Bruce
2014-09-19 14:18     ` Venkatesan, Venky
2014-09-19 17:45       ` Neil Horman
2014-09-24 18:19     ` Neil Horman [this message]
2014-09-26 10:41       ` Thomas Monjalon
2014-09-26 14:45         ` Neil Horman
2014-09-26 22:02           ` Stephen Hemminger
2014-09-27  2:22             ` Neil Horman
2014-10-01 18:59           ` Neil Horman
2014-10-07 21:01             ` Neil Horman
2014-10-08 15:57               ` Thomas Monjalon
2014-10-08 18:46                 ` Butler, Siobhan A
2014-10-08 19:09                 ` Neil Horman

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20140924181940.GB4651@hmsreliant.think-freely.org \
    --to=nhorman@tuxdriver.com \
    --cc=dev@dpdk.org \
    --cc=thomas.monjalon@6wind.com \


* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).