From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp.tuxdriver.com (charlotte.tuxdriver.com [70.61.120.58]) by dpdk.org (Postfix) with ESMTP id 3F170137D for ; Wed, 24 Sep 2014 20:13:37 +0200 (CEST) Received: from hmsreliant.think-freely.org ([2001:470:8:a08:7aac:c0ff:fec2:933b] helo=localhost) by smtp.tuxdriver.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.63) (envelope-from ) id 1XWrAE-0003Nz-JS; Wed, 24 Sep 2014 14:19:49 -0400 Date: Wed, 24 Sep 2014 14:19:40 -0400 From: Neil Horman To: Thomas Monjalon Message-ID: <20140924181940.GB4651@hmsreliant.think-freely.org> References: <1410809031-19114-1-git-send-email-nhorman@tuxdriver.com> <2657938.GdCLzJqtVh@xps13> <20140918191401.GP20389@hmsreliant.think-freely.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140918191401.GP20389@hmsreliant.think-freely.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Score: -2.9 (--) X-Spam-Status: No Cc: dev@dpdk.org Subject: Re: [dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 18:13:37 -0000 On Thu, Sep 18, 2014 at 03:14:01PM -0400, Neil Horman wrote: > On Thu, Sep 18, 2014 at 08:23:36PM +0200, Thomas Monjalon wrote: > > Hi Neil, > > > > 2014-09-15 15:23, Neil Horman: > > > The DPDK ABI develops and changes quickly, which makes it difficult for > > > applications to keep up with the latest version of the library, especially when > > > it (the DPDK) is built as a set of shared objects, as applications may be built > > > against an older version of the library. > > > > > > To mitigate this, this patch series introduces support for library and symbol > > > versioning when the DPDK is built as a DSO. Specifically, it does 4 things: > > > > > > 1) Adds initial support for library versioning. Each library now has a version > > > map that explicitly calls out what symbols are exported to using applications, > > > and assigns version(s) to them > > > > > > 2) Adds support macros so that when libraries create incompatible ABI's, > > > multiple versions may be supported so that applications linked against older > > > DPDK releases can continue to function > > > > > > 3) Adds library soname versioning suffixes so that when ABI's must be broken in > > > a fashion that requires a rebuild of older applications, they will break at load > > > time, rather than cause unexpected issues at run time. > > > > > > 4) Adds documentation for ABI policy, and provides space to document deprecated > > > ABI versions, so that applications might be warned of impending changes. > > > > > > With these elements in place the DPDK has some support to allow for the extended > > > maintenence of older API's while still allowing the freedom to develop new and > > > improved API's. > > > > > > Implementing this feature will require some additional effort on the part of > > > developers and reviewers. When reviewing patches, must be checked against > > > existing exports to ensure that the function prototypes are not changing. If > > > they are, the versioning macros must be used, and the library export map should > > > be updated to reflect the new version of the function. > > > > > > When data structures change, if those structures are application accessible, > > > apis that accept or return instances of those data structures should have new > > > versions created so that users of the old data structure version might co-exist > > > at the same time. > > > > Thanks for your efforts. > > But I feel this change has too many constraints for the current status of > > the DPDK. It's probably too early to adopt such policy. > > > I think you may be misunderstanding something. What constraints do you beleive > that this patch imposes? Note it doesn't in any way prevent changes to the ABI > of the DPDK, but rather gives us infrastructure to support multiple ABI > revisions at the same time, so that applications built against DPDK shared > libraries can continue to function properly at least for some time until we > decide to deprecate that ABI level. > > This is all based on the versioning strategy outlined here: > http://www.akkadia.org/drepper/dsohowto.pdf > > That may help clarify things for you. > > > By the way, this versioning doesn't cover structure changes. > No, it doesn't. No link-time mechanism does so. > > > How could it be managed? > Thats a subject that is open to discussion, but my initial thinking is that we > need to handle it on a case by case basis: > > * For minor updates, where allocation of a structure is done on the heap and new > fields need to be added, appending them to the end of a structure and providing > an initial value is sufficient. > > * For major changes, where fields need to be removed, or re-arranged, mostly > likely the API surfaces which accept or return those structures as > inputs/outputs will need to have new versions written to accept the new version > of the structure, and internally we will have to support both formats for a time > (according to the policy I documented, that is currently a single major > release). I.e. if you want to change struct foo, which is accepted as a > parameter for the function bar(struct foo *ptr), then for a release we would > need to create struct foo_v2 with the new format, map a new function foo_v2 to > the exported foo@@DPDK_1.(X+1), and internally make the foo functions understand > both the origional and v2 versions of the structure. Then in DPDK release > 1.X+2, we can remove the old version after posting a deprecation notice with > version 1.(X+1) > > > Don't you think it would be more reliable if managed by packaging? > Solving this with packaging defeats the purpose of having shared libraries at > all. While packaging each version of the dpdk separately is possible stopgap > solution, in that it allows applications to link to differing versions of the > library independently, but that negates any expectation of timely bugfixes for > any given version of the DPDK. That is to say, if you package things this way, > and wind up with several parallel versions of the same package, and for any > bugfix that comes out upstream, the packager then has the responsibility to > adapt that fix to each package. Thats an unscalable solution, and not something > any packager is going to undertake willingly. I did a hybrid version of this in > fedora for exactly that reason. I packaged the dpdk into its own directory, but > have every intention of changing that directory every major release, so that > application writers can clearly see when they need to stop updating the dpdk, > lest their applications stop linking. I'm not going to have multiple dpdk > packages to maintain in parallel, thats just too much work. > > > > > Thank you for opening this discussion with a constructive proposal. > > Let's check it later on once structures will be more stable since > > performance is the most critical target. > If I'm being honest, I have to say thats a cop out answer. We all know that > structure stability isn't a priority for the DPDK, nor will it ever be in all > likelyhood. It will continue to evolve and grow as the hardware does. And this > patch set doesn't prevent that from happening. All it does is provide some > level of stability in the API for a period of time to let 3rd party application > writers write and package applications with some allowance of time to keep up > with upstream changes on their own schedule. > > I grant you that writing a good API for a shared library is difficult, but > (and feel free to disagree with this), if we don't start enforcing policies that > require good API design, its not going to happen on its own. This patch set > will highlight those API points which are difficult to maintain accross major > releases, and force us to address and improve them. To that end I've already > begun talking to some of the individual library maintainers off list to address > some of the API aspects that I have concerns about (exporting variable rather > than accessor functions, structures that don't need to be visible to users, > etc), and they've started reviewing them. We can make this better, but we can't > just say later, because theres no roadmap that lists structure stability as a > line item. As hardware improves, structures will change to operate more > efficiently or support more features. Without a hard plan, the initial goals of > the DPDK (high performance networking) will relegate ABI to such a low priority > that it will never be addressed. > > To that end, can we discuss specifics? Can you ennumerate direct points that > you feel make this patch unworkable at this time? I know you mentioned some > above, and I think I addressed them (though please ask follow up questions if > I've been unclear). What other concerns do you have? > > Neil > > Ping Thomas. I know you're busy, but I would like this to not fall off anyones radar. You alluded to concerns regarding what, for lack of a better term, ABI/API lockin. I had asked you to enuumerate/elaborate on specifics, but never heard back. Are there further specifics you wish to discuss, or are you satisfied with the above answers? Best Neil