DPDK patches and discussions
 help / color / mirror / Atom feed
From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
To: Ray Kinsella <mdr@ashroe.eu>,
	Bruce Richardson <bruce.richardson@intel.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>,
	Stephen Hemminger <stephen@networkplumber.org>,
	"Ananyev, Konstantin" <konstantin.ananyev@intel.com>,
	"thomas@monjalon.net" <thomas@monjalon.net>, nd <nd@arm.com>,
	nd <nd@arm.com>
Subject: Re: [dpdk-dev] ABI and inline functions
Date: Wed, 24 Apr 2019 05:15:51 +0000	[thread overview]
Message-ID: <VE1PR08MB5149CE7A9A3EB7C75F28DC08983C0@VE1PR08MB5149.eurprd08.prod.outlook.com> (raw)
Message-ID: <20190424051551.diuUkfbwAQ5gOjfDQUBnKiW7lHA5dQo5bTzJz1J8nXE@z> (raw)
In-Reply-To: <2ec4da50-d874-865a-6bcc-916ac676be39@ashroe.eu>

> On 18/04/2019 11:28, Bruce Richardson wrote:
> > On Thu, Apr 18, 2019 at 04:34:53AM +0000, Honnappa Nagarahalli wrote:
> >>>
> >>> On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli
> wrote:
> >>>> Hello,
> >>>> 	There was a conversation [1] in the context of RCU library. I
> >>>> thought it needs participation from broader audience. Summary for
> >>>> the context (please look at [1] for full details)
> >>>>
> >>>
> >>> Thanks for kicking off this discussion
> >>>
> >>>> 1) How do we provide ABI compatibility when the code base contains
> >>> inline functions? Unless we freeze development in these inline
> >>> functions and corresponding structures, it might not be possible to claim
> ABI compatibility.
> >>> Application has to be recompiled.
> >>>
> >>> I agree that in some cases the application "might" need to be
> >>> recompiled, but on the other hand I also think that there are many
> >>> cases where ABI function versioning can still be used to mitigate
> >>> things. For example, if we think of a couple of various scenarios:
> >>>
> >>> 1. If everything is inline and variables are allocated by app, e.g.
> >>> spinlock on stack, then there is no issue as everything is
> >>> application contained.
> >> If there is a bug fix which requires the structure to change, the application
> would need to recompile. I guess you are talking about a scenario when
> nothing changed in the inline function/variables.
> >>
> >
> > If the application wants the bugfix, then yes. However, if the app is
> > unaffected by the bug, then it should have the option of updating DPDK
> > without a recompile.
> 
> I would also imagine that should be an extremely rare case, that a bugfix
> would require a structure change ... perhaps for an alignment issues?
> 
> >
> >>>
> >>> 2. If the situation is as in #1, but the structures in question are
> >>> passed to non-inline DPDK functions. In this case, any changes to
> >>> the structures require those functions taking the structures to be
> >>> versioned for old and new structures
> >> I think this can get complicated on the inline function side. The application
> and the DPDK library will end up having different inline functions. The
> changed inline function needs to be aware of 2 structure formats or the inline
> function needs to be duplicated (one for each version of the structure). I
> guess these are the workarounds we have to do.
> >>
> > No, there is never any need for two versions of the inline functions,
> > only the newest version is needed. This is because in the case of a
> > newly compiled application only the newest version of the non-inline
> > functions is ever used. The other older versions are only used at
> > runtime for compatilibity with pre-compiled apps with the older inlines.
> >
> >>>
> >>> 3. If instead we have the case, like in rte_ring, where the data
> >>> structures are allocated using functions, but they are used via
> >>> inlines in the app. In this case the creation functions (and any
> >>> other function using the structures) need to be versioned so that
> >>> the application gets the expected structure back from the creation.
> >> The new structure members have to be added such that the previous
> >> layout is not affected. Either add at the end or use the gaps that
> >> are left because of cache line alignment.
> >>
> > Not necessarily. There is nothing that requires the older and newer
> > versions of a function to use the same structure. We can rename the
> > original structure when versioning the old function, and then create a
> > new structure with the original name for later recompiled code using
> > the newest ABIs.
> >
> >>>
> >>> It might be useful to think of what other scenarios we have and what
> >>> ones are likely to be problematic, especially those that can't be
> >>> solved by having multiple function versions.
> >>>
> >>>> 2) Every function that is not in the direct datapath (fastpath?)
> >>>> should not be inline. Exceptions or things like rx/tx burst, ring
> >>>> enqueue/dequeue, and packet alloc/free - Stephen
> >>>
> >>> Agree with this. Anything not data path should not be inline. The
> >>> next
> >> Yes, very clear on this.
> >>
> >>> question then is for data path items how to determine whether they
> >>> need to be inline or not. In general my rule-of-thumb:
> >>> * anything dealing with bursts of packets should not be inline
> >>> * anything what works with single packet at a time should be inline
> >>>
> >>> The one exception to this rule is cases where we have to consider
> >>> "empty polling" as a likely use-case. For example,
> >>> rte_ring_dequeue_burst works with bursts of packets, but there is a
> >>> valid application use case where a thread could be polling a set of
> >>> rings where only a small number are actually busy. Right now,
> >>> polling an empty ring only costs a few cycles, meaning that it's
> >>> neglible to have a few polls of empty rings before you get to a busy
> >>> one. Having that function not-inline will cause that cost to jump
> >>> significantly, so could cause problems. (This leads to the interesting
> scenario where ring enqueue is safe to un-inline, while dequeue is not).
> >> A good way to think about it would be - ratio of amount of work done in
> the function to cost of function call.
> >>
> >
> > Yes, I would tend to agree in general.  The other thing is the
> > frequency of calls, and - as already stated - whether it takes a burst
> > of not.  Because even if it's a trivial function that takes only 10
> > cycles and we want to uninline it, the cost may double; but if it
> > takes a burst of packets and is only used once/twice per burst the
> > cost per packet should still only be a fraction of a cycle.
> >
> >>>
> >>>> 3) Plus synchronization routines: spin/rwlock/barrier, etc. I think
> >>>> rcu should be one of such exceptions - it is just another
> >>>> synchronization mechanism after all (just a bit more
> >>>> sophisticated). - Konstantin
> >>>>
> >>> In general I believe the synchronisation primitives should be inline.
> >>> However, it does come down to cost too - if a function takes 300
> >>> cycles, do we really care if it takes 305 or 310 instead to make it not
> inline?
> >>> Hopefully most synchronisation primitives are faster than this so
> >>> this situation should not occur.
> >>>
> >>>> 2) and 3) can be good guidelines to what functions/APIs can be
> >>>> inline. But,
> >>> I believe this guideline exists today too. Are there any thoughts to
> >>> change this?
> >>>>
> >>>> Coming to 1), I think DPDK cannot provide ABI compatibility unless
> >>>> all the
> >>> inline functions are converted to normal functions and symbol
> >>> versioning is done for those (not bothering about performance).
> >>>>
> >>> I disagree. I think even in the case of #1, we should be able to
> >>> manage some changes without breaking ABI.
> >> I completely agree with you on trying to keep the ABI break surface small
> and doing due diligence when one is required. However, I think claiming 100%
> ABI compatibility all the time (or as frequently as other projects claim) might
> be tough. IMO, we need to look beyond this to solve the end user problem.
> May be packaging multiple LTS with distros when DPDK could not avoid
> breaking ABI compatibility?
> >>
> > Having multiple LTS's per distro would be nice, but it's putting a lot
> > more work on the distro folks, I think.
> 
> I completely disagree with this approach.
> Anytime I have seen this done, most frequently with interpreted languages,
> think multiple versions of Java and Python concurrently on a system, it has
> always been a usability nightmare.
> 
> 
> >
> >>>
> >>>> In this context, does it make sense to say that we will maintain
> >>>> API compatibility rather than saying ABI compatibility? This will
> >>>> also send the right message to the end users.
> >>>>
> >>> I would value ABI compatibility much higher than API compatibility.
> >>> If someone is recompiling the application anyway, making a couple of
> >>> small changes (large rework is obviously a different issue) to the
> >>> code should not be a massive issue, I hope. On the other hand, ABI
> >>> compatibility is needed to allow seamless update from one version to
> >>> another, and it's that ABI compatiblity that allows distro's to pick up our
> latest and greatest versions.
> >>>
> >> I think it is also important to setup the right expectations to DPDK users.
> i.e. DPDK will police itself better to provide ABI compatibility but occasionally
> it might not be possible.
> >>
> > The trouble here is that a DPDK release, as a product is either
> > backward compatible or not. Either a user can take it as a drop-in
> > replacement for the previous version or not. Users do not want to have
> > to go through a checklist for each app to see if the app uses only
> > "compatible" APIs or not. Same for the distro folks, they are not
> > going to take some libs from a release but not others because the ABI is
> broken for some but not others.
> 
> Agreed, a DPDK release is either backwards compatible or it isn't, and to
> Bruce's point if it isn't, it maters less if this is for a big API change or a small
> API change - the failure of the API stability guarantee is the significant piece.
> 
> The reality is that most other system libraries provide strong guarantees ... to
> date we have provided very little.
It would be good to look libraries that provide inline functions and ABI compatibility to better understand the methods used.

> 
> If we start saying "Yes, but except when", the exception case very quickly
> becomes the general case and then we are back to the beginning again.
> 
> >
> > Regards,
> > /Bruce
> >

  parent reply	other threads:[~2019-04-24  5:15 UTC|newest]

Thread overview: 40+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-04-17  5:12 Honnappa Nagarahalli
2019-04-17  5:12 ` Honnappa Nagarahalli
2019-04-17  8:36 ` Bruce Richardson
2019-04-17  8:36   ` Bruce Richardson
2019-04-17 16:52   ` Stephen Hemminger
2019-04-17 16:52     ` Stephen Hemminger
2019-04-17 17:46   ` Jerin Jacob Kollanukkaran
2019-04-17 17:46     ` Jerin Jacob Kollanukkaran
2019-04-17 18:51     ` Stephen Hemminger
2019-04-17 18:51       ` Stephen Hemminger
2019-04-18  5:56       ` [dpdk-dev] [EXT] " Jerin Jacob Kollanukkaran
2019-04-18  5:56         ` Jerin Jacob Kollanukkaran
2019-04-23 14:23         ` Ray Kinsella
2019-04-23 14:23           ` Ray Kinsella
2019-04-24 18:38           ` Jerin Jacob Kollanukkaran
2019-04-24 18:38             ` Jerin Jacob Kollanukkaran
2019-04-23 14:19       ` [dpdk-dev] " Ray Kinsella
2019-04-23 14:19         ` Ray Kinsella
2019-04-18  4:34   ` Honnappa Nagarahalli
2019-04-18  4:34     ` Honnappa Nagarahalli
2019-04-18 10:28     ` Bruce Richardson
2019-04-18 10:28       ` Bruce Richardson
2019-04-23 14:12       ` Ray Kinsella
2019-04-23 14:12         ` Ray Kinsella
2019-04-24  5:15         ` Honnappa Nagarahalli [this message]
2019-04-24  5:15           ` Honnappa Nagarahalli
2019-04-24 11:08         ` Burakov, Anatoly
2019-04-24 11:08           ` Burakov, Anatoly
2019-04-24 12:22           ` Ray Kinsella
2019-04-24 12:22             ` Ray Kinsella
2019-04-24 12:54             ` Burakov, Anatoly
2019-04-24 12:54               ` Burakov, Anatoly
2019-04-24 15:44               ` Stephen Hemminger
2019-04-24 15:44                 ` Stephen Hemminger
2019-04-30  8:52             ` Thomas Monjalon
2019-04-30  8:52               ` Thomas Monjalon
2019-04-24  5:08       ` Honnappa Nagarahalli
2019-04-24  5:08         ` Honnappa Nagarahalli
2019-04-24  8:49         ` Bruce Richardson
2019-04-24  8:49           ` Bruce Richardson

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=VE1PR08MB5149CE7A9A3EB7C75F28DC08983C0@VE1PR08MB5149.eurprd08.prod.outlook.com \
    --to=honnappa.nagarahalli@arm.com \
    --cc=bruce.richardson@intel.com \
    --cc=dev@dpdk.org \
    --cc=konstantin.ananyev@intel.com \
    --cc=mdr@ashroe.eu \
    --cc=nd@arm.com \
    --cc=stephen@networkplumber.org \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).