[dpdk-dev] ABI and inline functions

[dpdk-dev] ABI and inline functions

[Honnappa Nagarahalli]

Hello,
	There was a conversation [1] in the context of RCU library. I thought it needs participation from broader audience. Summary for the context (please look at [1] for full details)

1) How do we provide ABI compatibility when the code base contains inline functions? Unless we freeze development in these inline functions and corresponding structures, it might not be possible to claim ABI compatibility. Application has to be recompiled.
2) Every function that is not in the direct datapath (fastpath?) should not be inline. Exceptions or things like rx/tx burst, ring enqueue/dequeue, and packet alloc/free - Stephen
3) Plus synchronization routines: spin/rwlock/barrier, etc. I think rcu should be one of such exceptions - it is just another synchronization mechanism after all (just a bit more sophisticated). - Konstantin

2) and 3) can be good guidelines to what functions/APIs can be inline. But, I believe this guideline exists today too. Are there any thoughts to change this?

Coming to 1), I think DPDK cannot provide ABI compatibility unless all the inline functions are converted to normal functions and symbol versioning is done for those (not bothering about performance).

In this context, does it make sense to say that we will maintain API compatibility rather than saying ABI compatibility? This will also send the right message to the end users.


Thank you,
Honnappa

[1] http://mails.dpdk.org/archives/dev/2019-April/130151.html

[dpdk-dev] ABI and inline functions

[Honnappa Nagarahalli]

Hello,
	There was a conversation [1] in the context of RCU library. I thought it needs participation from broader audience. Summary for the context (please look at [1] for full details)

1) How do we provide ABI compatibility when the code base contains inline functions? Unless we freeze development in these inline functions and corresponding structures, it might not be possible to claim ABI compatibility. Application has to be recompiled.
2) Every function that is not in the direct datapath (fastpath?) should not be inline. Exceptions or things like rx/tx burst, ring enqueue/dequeue, and packet alloc/free - Stephen
3) Plus synchronization routines: spin/rwlock/barrier, etc. I think rcu should be one of such exceptions - it is just another synchronization mechanism after all (just a bit more sophisticated). - Konstantin

2) and 3) can be good guidelines to what functions/APIs can be inline. But, I believe this guideline exists today too. Are there any thoughts to change this?

Coming to 1), I think DPDK cannot provide ABI compatibility unless all the inline functions are converted to normal functions and symbol versioning is done for those (not bothering about performance).

In this context, does it make sense to say that we will maintain API compatibility rather than saying ABI compatibility? This will also send the right message to the end users.


Thank you,
Honnappa

[1] http://mails.dpdk.org/archives/dev/2019-April/130151.html

Re: [dpdk-dev] ABI and inline functions

[Bruce Richardson]

On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli wrote:
> Hello,
> 	There was a conversation [1] in the context of RCU library. I thought it needs participation from broader audience. Summary for the context (please look at [1] for full details)
>

Thanks for kicking off this discussion
 
> 1) How do we provide ABI compatibility when the code base contains inline functions? Unless we freeze development in these inline functions and corresponding structures, it might not be possible to claim ABI compatibility. Application has to be recompiled.

I agree that in some cases the application "might" need to be recompiled,
but on the other hand I also think that there are many cases where ABI
function versioning can still be used to mitigate things. For example, if
we think of a couple of various scenarios:

1. If everything is inline and variables are allocated by app, e.g.
spinlock on stack, then there is no issue as everything is application
contained.

2. If the situation is as in #1, but the structures in question are passed
to non-inline DPDK functions. In this case, any changes to the structures
require those functions taking the structures to be versioned for old and
new structures

3. If instead we have the case, like in rte_ring, where the data structures
are allocated using functions, but they are used via inlines in the app. In
this case the creation functions (and any other function using the
structures) need to be versioned so that the application gets the expected
structure back from the creation.

It might be useful to think of what other scenarios we have and what ones
are likely to be problematic, especially those that can't be solved by
having multiple function versions.

> 2) Every function that is not in the direct datapath (fastpath?) should not be inline. Exceptions or things like rx/tx burst, ring enqueue/dequeue, and packet alloc/free - Stephen

Agree with this. Anything not data path should not be inline. The next
question then is for data path items how to determine whether they need to
be inline or not. In general my rule-of-thumb:
* anything dealing with bursts of packets should not be inline
* anything what works with single packet at a time should be inline

The one exception to this rule is cases where we have to consider "empty
polling" as a likely use-case. For example, rte_ring_dequeue_burst works
with bursts of packets, but there is a valid application use case where a
thread could be polling a set of rings where only a small number are
actually busy. Right now, polling an empty ring only costs a few cycles,
meaning that it's neglible to have a few polls of empty rings before you
get to a busy one. Having that function not-inline will cause that cost to
jump significantly, so could cause problems. (This leads to the interesting
scenario where ring enqueue is safe to un-inline, while dequeue is not).

> 3) Plus synchronization routines: spin/rwlock/barrier, etc. I think rcu should be one of such exceptions - it is just another synchronization mechanism after all (just a bit more sophisticated). - Konstantin
> 
In general I believe the synchronisation primitives should be inline.
However, it does come down to cost too - if a function takes 300 cycles, do
we really care if it takes 305 or 310 instead to make it not inline?
Hopefully most synchronisation primitives are faster than this so this
situation should not occur.

> 2) and 3) can be good guidelines to what functions/APIs can be inline. But, I believe this guideline exists today too. Are there any thoughts to change this?
> 
> Coming to 1), I think DPDK cannot provide ABI compatibility unless all the inline functions are converted to normal functions and symbol versioning is done for those (not bothering about performance).
> 
I disagree. I think even in the case of #1, we should be able to manage
some changes without breaking ABI.

> In this context, does it make sense to say that we will maintain API
> compatibility rather than saying ABI compatibility? This will also send
> the right message to the end users.
> 
I would value ABI compatibility much higher than API compatibility. If
someone is recompiling the application anyway, making a couple of small
changes (large rework is obviously a different issue) to the code should
not be a massive issue, I hope. On the other hand, ABI compatibility is
needed to allow seamless update from one version to another, and it's that
ABI compatiblity that allows distro's to pick up our latest and greatest
versions. 

Personally, I'd be happy enough to allow API changes at any point without
deprecation notice, so long as function versioning is used to ensure ABI
compatibility is kept.

My 2c.
/Bruce

Re: [dpdk-dev] ABI and inline functions

[Bruce Richardson]

On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli wrote:
> Hello,
> 	There was a conversation [1] in the context of RCU library. I thought it needs participation from broader audience. Summary for the context (please look at [1] for full details)
>

Thanks for kicking off this discussion
 
> 1) How do we provide ABI compatibility when the code base contains inline functions? Unless we freeze development in these inline functions and corresponding structures, it might not be possible to claim ABI compatibility. Application has to be recompiled.

I agree that in some cases the application "might" need to be recompiled,
but on the other hand I also think that there are many cases where ABI
function versioning can still be used to mitigate things. For example, if
we think of a couple of various scenarios:

1. If everything is inline and variables are allocated by app, e.g.
spinlock on stack, then there is no issue as everything is application
contained.

2. If the situation is as in #1, but the structures in question are passed
to non-inline DPDK functions. In this case, any changes to the structures
require those functions taking the structures to be versioned for old and
new structures

3. If instead we have the case, like in rte_ring, where the data structures
are allocated using functions, but they are used via inlines in the app. In
this case the creation functions (and any other function using the
structures) need to be versioned so that the application gets the expected
structure back from the creation.

It might be useful to think of what other scenarios we have and what ones
are likely to be problematic, especially those that can't be solved by
having multiple function versions.

> 2) Every function that is not in the direct datapath (fastpath?) should not be inline. Exceptions or things like rx/tx burst, ring enqueue/dequeue, and packet alloc/free - Stephen

Agree with this. Anything not data path should not be inline. The next
question then is for data path items how to determine whether they need to
be inline or not. In general my rule-of-thumb:
* anything dealing with bursts of packets should not be inline
* anything what works with single packet at a time should be inline

The one exception to this rule is cases where we have to consider "empty
polling" as a likely use-case. For example, rte_ring_dequeue_burst works
with bursts of packets, but there is a valid application use case where a
thread could be polling a set of rings where only a small number are
actually busy. Right now, polling an empty ring only costs a few cycles,
meaning that it's neglible to have a few polls of empty rings before you
get to a busy one. Having that function not-inline will cause that cost to
jump significantly, so could cause problems. (This leads to the interesting
scenario where ring enqueue is safe to un-inline, while dequeue is not).

> 3) Plus synchronization routines: spin/rwlock/barrier, etc. I think rcu should be one of such exceptions - it is just another synchronization mechanism after all (just a bit more sophisticated). - Konstantin
> 
In general I believe the synchronisation primitives should be inline.
However, it does come down to cost too - if a function takes 300 cycles, do
we really care if it takes 305 or 310 instead to make it not inline?
Hopefully most synchronisation primitives are faster than this so this
situation should not occur.

> 2) and 3) can be good guidelines to what functions/APIs can be inline. But, I believe this guideline exists today too. Are there any thoughts to change this?
> 
> Coming to 1), I think DPDK cannot provide ABI compatibility unless all the inline functions are converted to normal functions and symbol versioning is done for those (not bothering about performance).
> 
I disagree. I think even in the case of #1, we should be able to manage
some changes without breaking ABI.

> In this context, does it make sense to say that we will maintain API
> compatibility rather than saying ABI compatibility? This will also send
> the right message to the end users.
> 
I would value ABI compatibility much higher than API compatibility. If
someone is recompiling the application anyway, making a couple of small
changes (large rework is obviously a different issue) to the code should
not be a massive issue, I hope. On the other hand, ABI compatibility is
needed to allow seamless update from one version to another, and it's that
ABI compatiblity that allows distro's to pick up our latest and greatest
versions. 

Personally, I'd be happy enough to allow API changes at any point without
deprecation notice, so long as function versioning is used to ensure ABI
compatibility is kept.

My 2c.
/Bruce

Re: [dpdk-dev] ABI and inline functions

[Stephen Hemminger]

On Wed, 17 Apr 2019 09:36:38 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:

> > 
> > Coming to 1), I think DPDK cannot provide ABI compatibility unless all the inline functions are converted to normal functions and symbol versioning is done for those (not bothering about performance).
> >   
> I disagree. I think even in the case of #1, we should be able to manage
> some changes without breaking ABI.
> 
> > In this context, does it make sense to say that we will maintain API
> > compatibility rather than saying ABI compatibility? This will also send
> > the right message to the end users.
> >   
> I would value ABI compatibility much higher than API compatibility. If
> someone is recompiling the application anyway, making a couple of small
> changes (large rework is obviously a different issue) to the code should
> not be a massive issue, I hope. On the other hand, ABI compatibility is
> needed to allow seamless update from one version to another, and it's that
> ABI compatiblity that allows distro's to pick up our latest and greatest
> versions.

Agree with Bruce. What is most important about API is that the function
should change signature if behavior changes. I.e catch API changes at
compile time (not runtime).

Re: [dpdk-dev] ABI and inline functions

[Stephen Hemminger]

On Wed, 17 Apr 2019 09:36:38 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:

> > 
> > Coming to 1), I think DPDK cannot provide ABI compatibility unless all the inline functions are converted to normal functions and symbol versioning is done for those (not bothering about performance).
> >   
> I disagree. I think even in the case of #1, we should be able to manage
> some changes without breaking ABI.
> 
> > In this context, does it make sense to say that we will maintain API
> > compatibility rather than saying ABI compatibility? This will also send
> > the right message to the end users.
> >   
> I would value ABI compatibility much higher than API compatibility. If
> someone is recompiling the application anyway, making a couple of small
> changes (large rework is obviously a different issue) to the code should
> not be a massive issue, I hope. On the other hand, ABI compatibility is
> needed to allow seamless update from one version to another, and it's that
> ABI compatiblity that allows distro's to pick up our latest and greatest
> versions.

Agree with Bruce. What is most important about API is that the function
should change signature if behavior changes. I.e catch API changes at
compile time (not runtime).

Re: [dpdk-dev] ABI and inline functions

[Jerin Jacob Kollanukkaran]

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Bruce Richardson
> Sent: Wednesday, April 17, 2019 2:07 PM
> To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Cc: dev@dpdk.org; Stephen Hemminger <stephen@networkplumber.org>;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>; thomas@monjalon.net;
> Ray Kinsella <mdr@ashroe.eu>; nd <nd@arm.com>
> Subject: Re: [dpdk-dev] ABI and inline functions
> 
> > 2) Every function that is not in the direct datapath (fastpath?)
> > should not be inline. Exceptions or things like rx/tx burst, ring
> > enqueue/dequeue, and packet alloc/free - Stephen
> 
> Agree with this. Anything not data path should not be inline. The next question
> then is for data path items how to determine whether they need to be inline or
> not. In general my rule-of-thumb:
> * anything dealing with bursts of packets should not be inline

# I think, we need consider the how fat is the function too,
If it is light weight, probably we need to make it inline
# Except the forward case, In real world use case, it is not trivial make large 
burst of packet to process it even though function prototype burst.
We may need to consider that
# For the given function if some argument is used with inline other function,
 probably no point in making that function alone  inline as the structure 
is exposed in some other function.

> * anything what works with single packet at a time should be inline
> 
> > In this context, does it make sense to say that we will maintain API
> > compatibility rather than saying ABI compatibility? This will also
> > send the right message to the end users.
> >
> I would value ABI compatibility much higher than API compatibility. If someone
> is recompiling the application anyway, making a couple of small changes (large
> rework is obviously a different issue) to the code should not be a massive issue, I
> hope. On the other hand, ABI compatibility is needed to allow seamless update
> from one version to another, and it's that ABI compatiblity that allows distro's to
> pick up our latest and greatest versions.

IMO, We have two primary use case for DPDK

1) NFV kind of use case 	where the application needs to run on multiple platform
without recompiling it.
2) Fixed appliance use case where embed SoC like Intel Denverton or ARM64 integrated
Controller used. For fixed appliance use case, end user care more of performance
than ABI compatibility as it easy to recompile the end user application vs the cost of
hitting performance impact.

IMO, DPDK needs to cater to both category.

My 2c.

> 
> Personally, I'd be happy enough to allow API changes at any point without
> deprecation notice, so long as function versioning is used to ensure ABI
> compatibility is kept.
> 
> My 2c.
> /Bruce

Re: [dpdk-dev] ABI and inline functions

[Jerin Jacob Kollanukkaran]

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Bruce Richardson
> Sent: Wednesday, April 17, 2019 2:07 PM
> To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Cc: dev@dpdk.org; Stephen Hemminger <stephen@networkplumber.org>;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>; thomas@monjalon.net;
> Ray Kinsella <mdr@ashroe.eu>; nd <nd@arm.com>
> Subject: Re: [dpdk-dev] ABI and inline functions
> 
> > 2) Every function that is not in the direct datapath (fastpath?)
> > should not be inline. Exceptions or things like rx/tx burst, ring
> > enqueue/dequeue, and packet alloc/free - Stephen
> 
> Agree with this. Anything not data path should not be inline. The next question
> then is for data path items how to determine whether they need to be inline or
> not. In general my rule-of-thumb:
> * anything dealing with bursts of packets should not be inline

# I think, we need consider the how fat is the function too,
If it is light weight, probably we need to make it inline
# Except the forward case, In real world use case, it is not trivial make large 
burst of packet to process it even though function prototype burst.
We may need to consider that
# For the given function if some argument is used with inline other function,
 probably no point in making that function alone  inline as the structure 
is exposed in some other function.

> * anything what works with single packet at a time should be inline
> 
> > In this context, does it make sense to say that we will maintain API
> > compatibility rather than saying ABI compatibility? This will also
> > send the right message to the end users.
> >
> I would value ABI compatibility much higher than API compatibility. If someone
> is recompiling the application anyway, making a couple of small changes (large
> rework is obviously a different issue) to the code should not be a massive issue, I
> hope. On the other hand, ABI compatibility is needed to allow seamless update
> from one version to another, and it's that ABI compatiblity that allows distro's to
> pick up our latest and greatest versions.

IMO, We have two primary use case for DPDK

1) NFV kind of use case 	where the application needs to run on multiple platform
without recompiling it.
2) Fixed appliance use case where embed SoC like Intel Denverton or ARM64 integrated
Controller used. For fixed appliance use case, end user care more of performance
than ABI compatibility as it easy to recompile the end user application vs the cost of
hitting performance impact.

IMO, DPDK needs to cater to both category.

My 2c.

> 
> Personally, I'd be happy enough to allow API changes at any point without
> deprecation notice, so long as function versioning is used to ensure ABI
> compatibility is kept.
> 
> My 2c.
> /Bruce

Re: [dpdk-dev] ABI and inline functions

[Stephen Hemminger]

On Wed, 17 Apr 2019 17:46:34 +0000
Jerin Jacob Kollanukkaran <jerinj@marvell.com> wrote:

> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Bruce Richardson
> > Sent: Wednesday, April 17, 2019 2:07 PM
> > To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> > Cc: dev@dpdk.org; Stephen Hemminger <stephen@networkplumber.org>;
> > Ananyev, Konstantin <konstantin.ananyev@intel.com>; thomas@monjalon.net;
> > Ray Kinsella <mdr@ashroe.eu>; nd <nd@arm.com>
> > Subject: Re: [dpdk-dev] ABI and inline functions
> >   
> > > 2) Every function that is not in the direct datapath (fastpath?)
> > > should not be inline. Exceptions or things like rx/tx burst, ring
> > > enqueue/dequeue, and packet alloc/free - Stephen  
> > 
> > Agree with this. Anything not data path should not be inline. The next question
> > then is for data path items how to determine whether they need to be inline or
> > not. In general my rule-of-thumb:
> > * anything dealing with bursts of packets should not be inline  
> 
> # I think, we need consider the how fat is the function too,
> If it is light weight, probably we need to make it inline
> # Except the forward case, In real world use case, it is not trivial make large 
> burst of packet to process it even though function prototype burst.
> We may need to consider that
> # For the given function if some argument is used with inline other function,
>  probably no point in making that function alone  inline as the structure 
> is exposed in some other function.
> 
> > * anything what works with single packet at a time should be inline
> >   
> > > In this context, does it make sense to say that we will maintain API
> > > compatibility rather than saying ABI compatibility? This will also
> > > send the right message to the end users.
> > >  
> > I would value ABI compatibility much higher than API compatibility. If someone
> > is recompiling the application anyway, making a couple of small changes (large
> > rework is obviously a different issue) to the code should not be a massive issue, I
> > hope. On the other hand, ABI compatibility is needed to allow seamless update
> > from one version to another, and it's that ABI compatiblity that allows distro's to
> > pick up our latest and greatest versions.  
> 
> IMO, We have two primary use case for DPDK
> 
> 1) NFV kind of use case 	where the application needs to run on multiple platform
> without recompiling it.
> 2) Fixed appliance use case where embed SoC like Intel Denverton or ARM64 integrated
> Controller used. For fixed appliance use case, end user care more of performance
> than ABI compatibility as it easy to recompile the end user application vs the cost of
> hitting performance impact.

Nobody cares about compatiablity until they have to the first security update.
 

Re: [dpdk-dev] ABI and inline functions

[Stephen Hemminger]

On Wed, 17 Apr 2019 17:46:34 +0000
Jerin Jacob Kollanukkaran <jerinj@marvell.com> wrote:

> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Bruce Richardson
> > Sent: Wednesday, April 17, 2019 2:07 PM
> > To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> > Cc: dev@dpdk.org; Stephen Hemminger <stephen@networkplumber.org>;
> > Ananyev, Konstantin <konstantin.ananyev@intel.com>; thomas@monjalon.net;
> > Ray Kinsella <mdr@ashroe.eu>; nd <nd@arm.com>
> > Subject: Re: [dpdk-dev] ABI and inline functions
> >   
> > > 2) Every function that is not in the direct datapath (fastpath?)
> > > should not be inline. Exceptions or things like rx/tx burst, ring
> > > enqueue/dequeue, and packet alloc/free - Stephen  
> > 
> > Agree with this. Anything not data path should not be inline. The next question
> > then is for data path items how to determine whether they need to be inline or
> > not. In general my rule-of-thumb:
> > * anything dealing with bursts of packets should not be inline  
> 
> # I think, we need consider the how fat is the function too,
> If it is light weight, probably we need to make it inline
> # Except the forward case, In real world use case, it is not trivial make large 
> burst of packet to process it even though function prototype burst.
> We may need to consider that
> # For the given function if some argument is used with inline other function,
>  probably no point in making that function alone  inline as the structure 
> is exposed in some other function.
> 
> > * anything what works with single packet at a time should be inline
> >   
> > > In this context, does it make sense to say that we will maintain API
> > > compatibility rather than saying ABI compatibility? This will also
> > > send the right message to the end users.
> > >  
> > I would value ABI compatibility much higher than API compatibility. If someone
> > is recompiling the application anyway, making a couple of small changes (large
> > rework is obviously a different issue) to the code should not be a massive issue, I
> > hope. On the other hand, ABI compatibility is needed to allow seamless update
> > from one version to another, and it's that ABI compatiblity that allows distro's to
> > pick up our latest and greatest versions.  
> 
> IMO, We have two primary use case for DPDK
> 
> 1) NFV kind of use case 	where the application needs to run on multiple platform
> without recompiling it.
> 2) Fixed appliance use case where embed SoC like Intel Denverton or ARM64 integrated
> Controller used. For fixed appliance use case, end user care more of performance
> than ABI compatibility as it easy to recompile the end user application vs the cost of
> hitting performance impact.

Nobody cares about compatiablity until they have to the first security update.
 

Re: [dpdk-dev] ABI and inline functions

[Honnappa Nagarahalli]

> 
> On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli wrote:
> > Hello,
> > 	There was a conversation [1] in the context of RCU library. I thought
> > it needs participation from broader audience. Summary for the context
> > (please look at [1] for full details)
> >
> 
> Thanks for kicking off this discussion
> 
> > 1) How do we provide ABI compatibility when the code base contains
> inline functions? Unless we freeze development in these inline functions and
> corresponding structures, it might not be possible to claim ABI compatibility.
> Application has to be recompiled.
> 
> I agree that in some cases the application "might" need to be recompiled,
> but on the other hand I also think that there are many cases where ABI
> function versioning can still be used to mitigate things. For example, if we
> think of a couple of various scenarios:
> 
> 1. If everything is inline and variables are allocated by app, e.g.
> spinlock on stack, then there is no issue as everything is application
> contained.
If there is a bug fix which requires the structure to change, the application would need to recompile. I guess you are talking about a scenario when nothing changed in the inline function/variables.

> 
> 2. If the situation is as in #1, but the structures in question are passed to
> non-inline DPDK functions. In this case, any changes to the structures require
> those functions taking the structures to be versioned for old and new
> structures
I think this can get complicated on the inline function side. The application and the DPDK library will end up having different inline functions. The changed inline function needs to be aware of 2 structure formats or the inline function needs to be duplicated (one for each version of the structure). I guess these are the workarounds we have to do.

> 
> 3. If instead we have the case, like in rte_ring, where the data structures are
> allocated using functions, but they are used via inlines in the app. In this
> case the creation functions (and any other function using the
> structures) need to be versioned so that the application gets the expected
> structure back from the creation.
The new structure members have to be added such that the previous layout is not affected. Either add at the end or use the gaps that are left because of cache line alignment.

> 
> It might be useful to think of what other scenarios we have and what ones
> are likely to be problematic, especially those that can't be solved by having
> multiple function versions.
> 
> > 2) Every function that is not in the direct datapath (fastpath?)
> > should not be inline. Exceptions or things like rx/tx burst, ring
> > enqueue/dequeue, and packet alloc/free - Stephen
> 
> Agree with this. Anything not data path should not be inline. The next
Yes, very clear on this.

> question then is for data path items how to determine whether they need to
> be inline or not. In general my rule-of-thumb:
> * anything dealing with bursts of packets should not be inline
> * anything what works with single packet at a time should be inline
> 
> The one exception to this rule is cases where we have to consider "empty
> polling" as a likely use-case. For example, rte_ring_dequeue_burst works
> with bursts of packets, but there is a valid application use case where a
> thread could be polling a set of rings where only a small number are
> actually busy. Right now, polling an empty ring only costs a few cycles,
> meaning that it's neglible to have a few polls of empty rings before you get
> to a busy one. Having that function not-inline will cause that cost to jump
> significantly, so could cause problems. (This leads to the interesting scenario
> where ring enqueue is safe to un-inline, while dequeue is not).
A good way to think about it would be - ratio of amount of work done in the function to cost of function call.

> 
> > 3) Plus synchronization routines: spin/rwlock/barrier, etc. I think
> > rcu should be one of such exceptions - it is just another
> > synchronization mechanism after all (just a bit more sophisticated). -
> > Konstantin
> >
> In general I believe the synchronisation primitives should be inline.
> However, it does come down to cost too - if a function takes 300 cycles, do
> we really care if it takes 305 or 310 instead to make it not inline?
> Hopefully most synchronisation primitives are faster than this so this
> situation should not occur.
> 
> > 2) and 3) can be good guidelines to what functions/APIs can be inline. But,
> I believe this guideline exists today too. Are there any thoughts to change
> this?
> >
> > Coming to 1), I think DPDK cannot provide ABI compatibility unless all the
> inline functions are converted to normal functions and symbol versioning is
> done for those (not bothering about performance).
> >
> I disagree. I think even in the case of #1, we should be able to manage some
> changes without breaking ABI.
I completely agree with you on trying to keep the ABI break surface small and doing due diligence when one is required. However, I think claiming 100% ABI compatibility all the time (or as frequently as other projects claim) might be tough. IMO, we need to look beyond this to solve the end user problem. May be packaging multiple LTS with distros when DPDK could not avoid breaking ABI compatibility?

> 
> > In this context, does it make sense to say that we will maintain API
> > compatibility rather than saying ABI compatibility? This will also
> > send the right message to the end users.
> >
> I would value ABI compatibility much higher than API compatibility. If
> someone is recompiling the application anyway, making a couple of small
> changes (large rework is obviously a different issue) to the code should not
> be a massive issue, I hope. On the other hand, ABI compatibility is needed to
> allow seamless update from one version to another, and it's that ABI
> compatiblity that allows distro's to pick up our latest and greatest versions.
> 
I think it is also important to setup the right expectations to DPDK users. i.e. DPDK will police itself better to provide ABI compatibility but occasionally it might not be possible.

> Personally, I'd be happy enough to allow API changes at any point without
> deprecation notice, so long as function versioning is used to ensure ABI
> compatibility is kept.
> 
> My 2c.
> /Bruce

Re: [dpdk-dev] ABI and inline functions

[Honnappa Nagarahalli]

> 
> On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli wrote:
> > Hello,
> > 	There was a conversation [1] in the context of RCU library. I thought
> > it needs participation from broader audience. Summary for the context
> > (please look at [1] for full details)
> >
> 
> Thanks for kicking off this discussion
> 
> > 1) How do we provide ABI compatibility when the code base contains
> inline functions? Unless we freeze development in these inline functions and
> corresponding structures, it might not be possible to claim ABI compatibility.
> Application has to be recompiled.
> 
> I agree that in some cases the application "might" need to be recompiled,
> but on the other hand I also think that there are many cases where ABI
> function versioning can still be used to mitigate things. For example, if we
> think of a couple of various scenarios:
> 
> 1. If everything is inline and variables are allocated by app, e.g.
> spinlock on stack, then there is no issue as everything is application
> contained.
If there is a bug fix which requires the structure to change, the application would need to recompile. I guess you are talking about a scenario when nothing changed in the inline function/variables.

> 
> 2. If the situation is as in #1, but the structures in question are passed to
> non-inline DPDK functions. In this case, any changes to the structures require
> those functions taking the structures to be versioned for old and new
> structures
I think this can get complicated on the inline function side. The application and the DPDK library will end up having different inline functions. The changed inline function needs to be aware of 2 structure formats or the inline function needs to be duplicated (one for each version of the structure). I guess these are the workarounds we have to do.

> 
> 3. If instead we have the case, like in rte_ring, where the data structures are
> allocated using functions, but they are used via inlines in the app. In this
> case the creation functions (and any other function using the
> structures) need to be versioned so that the application gets the expected
> structure back from the creation.
The new structure members have to be added such that the previous layout is not affected. Either add at the end or use the gaps that are left because of cache line alignment.

> 
> It might be useful to think of what other scenarios we have and what ones
> are likely to be problematic, especially those that can't be solved by having
> multiple function versions.
> 
> > 2) Every function that is not in the direct datapath (fastpath?)
> > should not be inline. Exceptions or things like rx/tx burst, ring
> > enqueue/dequeue, and packet alloc/free - Stephen
> 
> Agree with this. Anything not data path should not be inline. The next
Yes, very clear on this.

> question then is for data path items how to determine whether they need to
> be inline or not. In general my rule-of-thumb:
> * anything dealing with bursts of packets should not be inline
> * anything what works with single packet at a time should be inline
> 
> The one exception to this rule is cases where we have to consider "empty
> polling" as a likely use-case. For example, rte_ring_dequeue_burst works
> with bursts of packets, but there is a valid application use case where a
> thread could be polling a set of rings where only a small number are
> actually busy. Right now, polling an empty ring only costs a few cycles,
> meaning that it's neglible to have a few polls of empty rings before you get
> to a busy one. Having that function not-inline will cause that cost to jump
> significantly, so could cause problems. (This leads to the interesting scenario
> where ring enqueue is safe to un-inline, while dequeue is not).
A good way to think about it would be - ratio of amount of work done in the function to cost of function call.

> 
> > 3) Plus synchronization routines: spin/rwlock/barrier, etc. I think
> > rcu should be one of such exceptions - it is just another
> > synchronization mechanism after all (just a bit more sophisticated). -
> > Konstantin
> >
> In general I believe the synchronisation primitives should be inline.
> However, it does come down to cost too - if a function takes 300 cycles, do
> we really care if it takes 305 or 310 instead to make it not inline?
> Hopefully most synchronisation primitives are faster than this so this
> situation should not occur.
> 
> > 2) and 3) can be good guidelines to what functions/APIs can be inline. But,
> I believe this guideline exists today too. Are there any thoughts to change
> this?
> >
> > Coming to 1), I think DPDK cannot provide ABI compatibility unless all the
> inline functions are converted to normal functions and symbol versioning is
> done for those (not bothering about performance).
> >
> I disagree. I think even in the case of #1, we should be able to manage some
> changes without breaking ABI.
I completely agree with you on trying to keep the ABI break surface small and doing due diligence when one is required. However, I think claiming 100% ABI compatibility all the time (or as frequently as other projects claim) might be tough. IMO, we need to look beyond this to solve the end user problem. May be packaging multiple LTS with distros when DPDK could not avoid breaking ABI compatibility?

> 
> > In this context, does it make sense to say that we will maintain API
> > compatibility rather than saying ABI compatibility? This will also
> > send the right message to the end users.
> >
> I would value ABI compatibility much higher than API compatibility. If
> someone is recompiling the application anyway, making a couple of small
> changes (large rework is obviously a different issue) to the code should not
> be a massive issue, I hope. On the other hand, ABI compatibility is needed to
> allow seamless update from one version to another, and it's that ABI
> compatiblity that allows distro's to pick up our latest and greatest versions.
> 
I think it is also important to setup the right expectations to DPDK users. i.e. DPDK will police itself better to provide ABI compatibility but occasionally it might not be possible.

> Personally, I'd be happy enough to allow API changes at any point without
> deprecation notice, so long as function versioning is used to ensure ABI
> compatibility is kept.
> 
> My 2c.
> /Bruce

Re: [dpdk-dev] [EXT] Re: ABI and inline functions

[Jerin Jacob Kollanukkaran]

> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Thursday, April 18, 2019 12:21 AM
> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> Cc: Bruce Richardson <bruce.richardson@intel.com>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; dev@dpdk.org; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; thomas@monjalon.net; Ray Kinsella
> <mdr@ashroe.eu>; nd <nd@arm.com>
> Subject: [EXT] Re: [dpdk-dev] ABI and inline functions
> > I would value ABI compatibility much higher than API compatibility.
> > > If someone is recompiling the application anyway, making a couple of
> > > small changes (large rework is obviously a different issue) to the
> > > code should not be a massive issue, I hope. On the other hand, ABI
> > > compatibility is needed to allow seamless update from one version to
> > > another, and it's that ABI compatiblity that allows distro's to pick up our
> latest and greatest versions.
> >
> > IMO, We have two primary use case for DPDK
> >
> > 1) NFV kind of use case 	where the application needs to run on multiple
> platform
> > without recompiling it.
> > 2) Fixed appliance use case where embed SoC like Intel Denverton or
> > ARM64 integrated Controller used. For fixed appliance use case, end
> > user care more of performance than ABI compatibility as it easy to
> > recompile the end user application vs the cost of hitting performance impact.
> 
> Nobody cares about compatiablity until they have to the first security update.

For fixed appliance case, The update(FW update) will be  a single blob which
Include all the components. So they can back port the security fix and recompile
the sw as needed.

The very similar category  is DPDK running in smart NICs(Runs as FW in PCIe EP device).



> 

Re: [dpdk-dev] [EXT] Re: ABI and inline functions

[Jerin Jacob Kollanukkaran]

> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Thursday, April 18, 2019 12:21 AM
> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> Cc: Bruce Richardson <bruce.richardson@intel.com>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; dev@dpdk.org; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; thomas@monjalon.net; Ray Kinsella
> <mdr@ashroe.eu>; nd <nd@arm.com>
> Subject: [EXT] Re: [dpdk-dev] ABI and inline functions
> > I would value ABI compatibility much higher than API compatibility.
> > > If someone is recompiling the application anyway, making a couple of
> > > small changes (large rework is obviously a different issue) to the
> > > code should not be a massive issue, I hope. On the other hand, ABI
> > > compatibility is needed to allow seamless update from one version to
> > > another, and it's that ABI compatiblity that allows distro's to pick up our
> latest and greatest versions.
> >
> > IMO, We have two primary use case for DPDK
> >
> > 1) NFV kind of use case 	where the application needs to run on multiple
> platform
> > without recompiling it.
> > 2) Fixed appliance use case where embed SoC like Intel Denverton or
> > ARM64 integrated Controller used. For fixed appliance use case, end
> > user care more of performance than ABI compatibility as it easy to
> > recompile the end user application vs the cost of hitting performance impact.
> 
> Nobody cares about compatiablity until they have to the first security update.

For fixed appliance case, The update(FW update) will be  a single blob which
Include all the components. So they can back port the security fix and recompile
the sw as needed.

The very similar category  is DPDK running in smart NICs(Runs as FW in PCIe EP device).



> 

Re: [dpdk-dev] ABI and inline functions

[Bruce Richardson]

On Thu, Apr 18, 2019 at 04:34:53AM +0000, Honnappa Nagarahalli wrote:
> > 
> > On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli wrote:
> > > Hello,
> > > 	There was a conversation [1] in the context of RCU library. I thought
> > > it needs participation from broader audience. Summary for the context
> > > (please look at [1] for full details)
> > >
> > 
> > Thanks for kicking off this discussion
> > 
> > > 1) How do we provide ABI compatibility when the code base contains
> > inline functions? Unless we freeze development in these inline functions and
> > corresponding structures, it might not be possible to claim ABI compatibility.
> > Application has to be recompiled.
> > 
> > I agree that in some cases the application "might" need to be recompiled,
> > but on the other hand I also think that there are many cases where ABI
> > function versioning can still be used to mitigate things. For example, if we
> > think of a couple of various scenarios:
> > 
> > 1. If everything is inline and variables are allocated by app, e.g.
> > spinlock on stack, then there is no issue as everything is application
> > contained.
> If there is a bug fix which requires the structure to change, the application would need to recompile. I guess you are talking about a scenario when nothing changed in the inline function/variables.
> 

If the application wants the bugfix, then yes. However, if the app is
unaffected by the bug, then it should have the option of updating DPDK
without a recompile.

> > 
> > 2. If the situation is as in #1, but the structures in question are passed to
> > non-inline DPDK functions. In this case, any changes to the structures require
> > those functions taking the structures to be versioned for old and new
> > structures
> I think this can get complicated on the inline function side. The application and the DPDK library will end up having different inline functions. The changed inline function needs to be aware of 2 structure formats or the inline function needs to be duplicated (one for each version of the structure). I guess these are the workarounds we have to do.
> 
No, there is never any need for two versions of the inline functions, only
the newest version is needed. This is because in the case of a newly compiled
application only the newest version of the non-inline functions is ever
used. The other older versions are only used at runtime for compatilibity
with pre-compiled apps with the older inlines.

> > 
> > 3. If instead we have the case, like in rte_ring, where the data
> > structures are allocated using functions, but they are used via inlines
> > in the app. In this case the creation functions (and any other function
> > using the structures) need to be versioned so that the application gets
> > the expected structure back from the creation.
> The new structure members have to be added such that the previous layout
> is not affected. Either add at the end or use the gaps that are left
> because of cache line alignment.
> 
Not necessarily. There is nothing that requires the older and newer
versions of a function to use the same structure. We can rename the
original structure when versioning the old function, and then create a new
structure with the original name for later recompiled code using the newest
ABIs.

> > 
> > It might be useful to think of what other scenarios we have and what ones
> > are likely to be problematic, especially those that can't be solved by having
> > multiple function versions.
> > 
> > > 2) Every function that is not in the direct datapath (fastpath?)
> > > should not be inline. Exceptions or things like rx/tx burst, ring
> > > enqueue/dequeue, and packet alloc/free - Stephen
> > 
> > Agree with this. Anything not data path should not be inline. The next
> Yes, very clear on this.
> 
> > question then is for data path items how to determine whether they need to
> > be inline or not. In general my rule-of-thumb:
> > * anything dealing with bursts of packets should not be inline
> > * anything what works with single packet at a time should be inline
> > 
> > The one exception to this rule is cases where we have to consider "empty
> > polling" as a likely use-case. For example, rte_ring_dequeue_burst works
> > with bursts of packets, but there is a valid application use case where a
> > thread could be polling a set of rings where only a small number are
> > actually busy. Right now, polling an empty ring only costs a few cycles,
> > meaning that it's neglible to have a few polls of empty rings before you get
> > to a busy one. Having that function not-inline will cause that cost to jump
> > significantly, so could cause problems. (This leads to the interesting scenario
> > where ring enqueue is safe to un-inline, while dequeue is not).
> A good way to think about it would be - ratio of amount of work done in the function to cost of function call.
> 

Yes, I would tend to agree in general.  The other thing is the frequency of
calls, and - as already stated - whether it takes a burst of not.  Because
even if it's a trivial function that takes only 10 cycles and we want to
uninline it, the cost may double; but if it takes a burst of packets and is
only used once/twice per burst the cost per packet should still only be a
fraction of a cycle.

> > 
> > > 3) Plus synchronization routines: spin/rwlock/barrier, etc. I think
> > > rcu should be one of such exceptions - it is just another
> > > synchronization mechanism after all (just a bit more sophisticated). -
> > > Konstantin
> > >
> > In general I believe the synchronisation primitives should be inline.
> > However, it does come down to cost too - if a function takes 300 cycles, do
> > we really care if it takes 305 or 310 instead to make it not inline?
> > Hopefully most synchronisation primitives are faster than this so this
> > situation should not occur.
> > 
> > > 2) and 3) can be good guidelines to what functions/APIs can be inline. But,
> > I believe this guideline exists today too. Are there any thoughts to change
> > this?
> > >
> > > Coming to 1), I think DPDK cannot provide ABI compatibility unless all the
> > inline functions are converted to normal functions and symbol versioning is
> > done for those (not bothering about performance).
> > >
> > I disagree. I think even in the case of #1, we should be able to manage some
> > changes without breaking ABI.
> I completely agree with you on trying to keep the ABI break surface small and doing due diligence when one is required. However, I think claiming 100% ABI compatibility all the time (or as frequently as other projects claim) might be tough. IMO, we need to look beyond this to solve the end user problem. May be packaging multiple LTS with distros when DPDK could not avoid breaking ABI compatibility?
> 
Having multiple LTS's per distro would be nice, but it's putting a lot more
work on the distro folks, I think.

> > 
> > > In this context, does it make sense to say that we will maintain API
> > > compatibility rather than saying ABI compatibility? This will also
> > > send the right message to the end users.
> > >
> > I would value ABI compatibility much higher than API compatibility. If
> > someone is recompiling the application anyway, making a couple of small
> > changes (large rework is obviously a different issue) to the code should not
> > be a massive issue, I hope. On the other hand, ABI compatibility is needed to
> > allow seamless update from one version to another, and it's that ABI
> > compatiblity that allows distro's to pick up our latest and greatest versions.
> > 
> I think it is also important to setup the right expectations to DPDK users. i.e. DPDK will police itself better to provide ABI compatibility but occasionally it might not be possible.
> 
The trouble here is that a DPDK release, as a product is either backward
compatible or not. Either a user can take it as a drop-in replacement for
the previous version or not. Users do not want to have to go through a
checklist for each app to see if the app uses only "compatible" APIs or
not. Same for the distro folks, they are not going to take some libs from a
release but not others because the ABI is broken for some but not others.

Regards,
/Bruce

Re: [dpdk-dev] ABI and inline functions

[Bruce Richardson]

On Thu, Apr 18, 2019 at 04:34:53AM +0000, Honnappa Nagarahalli wrote:
> > 
> > On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli wrote:
> > > Hello,
> > > 	There was a conversation [1] in the context of RCU library. I thought
> > > it needs participation from broader audience. Summary for the context
> > > (please look at [1] for full details)
> > >
> > 
> > Thanks for kicking off this discussion
> > 
> > > 1) How do we provide ABI compatibility when the code base contains
> > inline functions? Unless we freeze development in these inline functions and
> > corresponding structures, it might not be possible to claim ABI compatibility.
> > Application has to be recompiled.
> > 
> > I agree that in some cases the application "might" need to be recompiled,
> > but on the other hand I also think that there are many cases where ABI
> > function versioning can still be used to mitigate things. For example, if we
> > think of a couple of various scenarios:
> > 
> > 1. If everything is inline and variables are allocated by app, e.g.
> > spinlock on stack, then there is no issue as everything is application
> > contained.
> If there is a bug fix which requires the structure to change, the application would need to recompile. I guess you are talking about a scenario when nothing changed in the inline function/variables.
> 

If the application wants the bugfix, then yes. However, if the app is
unaffected by the bug, then it should have the option of updating DPDK
without a recompile.

> > 
> > 2. If the situation is as in #1, but the structures in question are passed to
> > non-inline DPDK functions. In this case, any changes to the structures require
> > those functions taking the structures to be versioned for old and new
> > structures
> I think this can get complicated on the inline function side. The application and the DPDK library will end up having different inline functions. The changed inline function needs to be aware of 2 structure formats or the inline function needs to be duplicated (one for each version of the structure). I guess these are the workarounds we have to do.
> 
No, there is never any need for two versions of the inline functions, only
the newest version is needed. This is because in the case of a newly compiled
application only the newest version of the non-inline functions is ever
used. The other older versions are only used at runtime for compatilibity
with pre-compiled apps with the older inlines.

> > 
> > 3. If instead we have the case, like in rte_ring, where the data
> > structures are allocated using functions, but they are used via inlines
> > in the app. In this case the creation functions (and any other function
> > using the structures) need to be versioned so that the application gets
> > the expected structure back from the creation.
> The new structure members have to be added such that the previous layout
> is not affected. Either add at the end or use the gaps that are left
> because of cache line alignment.
> 
Not necessarily. There is nothing that requires the older and newer
versions of a function to use the same structure. We can rename the
original structure when versioning the old function, and then create a new
structure with the original name for later recompiled code using the newest
ABIs.

> > 
> > It might be useful to think of what other scenarios we have and what ones
> > are likely to be problematic, especially those that can't be solved by having
> > multiple function versions.
> > 
> > > 2) Every function that is not in the direct datapath (fastpath?)
> > > should not be inline. Exceptions or things like rx/tx burst, ring
> > > enqueue/dequeue, and packet alloc/free - Stephen
> > 
> > Agree with this. Anything not data path should not be inline. The next
> Yes, very clear on this.
> 
> > question then is for data path items how to determine whether they need to
> > be inline or not. In general my rule-of-thumb:
> > * anything dealing with bursts of packets should not be inline
> > * anything what works with single packet at a time should be inline
> > 
> > The one exception to this rule is cases where we have to consider "empty
> > polling" as a likely use-case. For example, rte_ring_dequeue_burst works
> > with bursts of packets, but there is a valid application use case where a
> > thread could be polling a set of rings where only a small number are
> > actually busy. Right now, polling an empty ring only costs a few cycles,
> > meaning that it's neglible to have a few polls of empty rings before you get
> > to a busy one. Having that function not-inline will cause that cost to jump
> > significantly, so could cause problems. (This leads to the interesting scenario
> > where ring enqueue is safe to un-inline, while dequeue is not).
> A good way to think about it would be - ratio of amount of work done in the function to cost of function call.
> 

Yes, I would tend to agree in general.  The other thing is the frequency of
calls, and - as already stated - whether it takes a burst of not.  Because
even if it's a trivial function that takes only 10 cycles and we want to
uninline it, the cost may double; but if it takes a burst of packets and is
only used once/twice per burst the cost per packet should still only be a
fraction of a cycle.

> > 
> > > 3) Plus synchronization routines: spin/rwlock/barrier, etc. I think
> > > rcu should be one of such exceptions - it is just another
> > > synchronization mechanism after all (just a bit more sophisticated). -
> > > Konstantin
> > >
> > In general I believe the synchronisation primitives should be inline.
> > However, it does come down to cost too - if a function takes 300 cycles, do
> > we really care if it takes 305 or 310 instead to make it not inline?
> > Hopefully most synchronisation primitives are faster than this so this
> > situation should not occur.
> > 
> > > 2) and 3) can be good guidelines to what functions/APIs can be inline. But,
> > I believe this guideline exists today too. Are there any thoughts to change
> > this?
> > >
> > > Coming to 1), I think DPDK cannot provide ABI compatibility unless all the
> > inline functions are converted to normal functions and symbol versioning is
> > done for those (not bothering about performance).
> > >
> > I disagree. I think even in the case of #1, we should be able to manage some
> > changes without breaking ABI.
> I completely agree with you on trying to keep the ABI break surface small and doing due diligence when one is required. However, I think claiming 100% ABI compatibility all the time (or as frequently as other projects claim) might be tough. IMO, we need to look beyond this to solve the end user problem. May be packaging multiple LTS with distros when DPDK could not avoid breaking ABI compatibility?
> 
Having multiple LTS's per distro would be nice, but it's putting a lot more
work on the distro folks, I think.

> > 
> > > In this context, does it make sense to say that we will maintain API
> > > compatibility rather than saying ABI compatibility? This will also
> > > send the right message to the end users.
> > >
> > I would value ABI compatibility much higher than API compatibility. If
> > someone is recompiling the application anyway, making a couple of small
> > changes (large rework is obviously a different issue) to the code should not
> > be a massive issue, I hope. On the other hand, ABI compatibility is needed to
> > allow seamless update from one version to another, and it's that ABI
> > compatiblity that allows distro's to pick up our latest and greatest versions.
> > 
> I think it is also important to setup the right expectations to DPDK users. i.e. DPDK will police itself better to provide ABI compatibility but occasionally it might not be possible.
> 
The trouble here is that a DPDK release, as a product is either backward
compatible or not. Either a user can take it as a drop-in replacement for
the previous version or not. Users do not want to have to go through a
checklist for each app to see if the app uses only "compatible" APIs or
not. Same for the distro folks, they are not going to take some libs from a
release but not others because the ABI is broken for some but not others.

Regards,
/Bruce

Re: [dpdk-dev] ABI and inline functions

[Ray Kinsella]

On 18/04/2019 11:28, Bruce Richardson wrote:
> On Thu, Apr 18, 2019 at 04:34:53AM +0000, Honnappa Nagarahalli wrote:
>>>
>>> On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli wrote:
>>>> Hello,
>>>> 	There was a conversation [1] in the context of RCU library. I thought
>>>> it needs participation from broader audience. Summary for the context
>>>> (please look at [1] for full details)
>>>>
>>>
>>> Thanks for kicking off this discussion
>>>
>>>> 1) How do we provide ABI compatibility when the code base contains
>>> inline functions? Unless we freeze development in these inline functions and
>>> corresponding structures, it might not be possible to claim ABI compatibility.
>>> Application has to be recompiled.
>>>
>>> I agree that in some cases the application "might" need to be recompiled,
>>> but on the other hand I also think that there are many cases where ABI
>>> function versioning can still be used to mitigate things. For example, if we
>>> think of a couple of various scenarios:
>>>
>>> 1. If everything is inline and variables are allocated by app, e.g.
>>> spinlock on stack, then there is no issue as everything is application
>>> contained.
>> If there is a bug fix which requires the structure to change, the application would need to recompile. I guess you are talking about a scenario when nothing changed in the inline function/variables.
>>
> 
> If the application wants the bugfix, then yes. However, if the app is
> unaffected by the bug, then it should have the option of updating DPDK
> without a recompile.

I would also imagine that should be an extremely rare case, that a
bugfix would require a structure change ... perhaps for an alignment issues?

> 
>>>
>>> 2. If the situation is as in #1, but the structures in question are passed to
>>> non-inline DPDK functions. In this case, any changes to the structures require
>>> those functions taking the structures to be versioned for old and new
>>> structures
>> I think this can get complicated on the inline function side. The application and the DPDK library will end up having different inline functions. The changed inline function needs to be aware of 2 structure formats or the inline function needs to be duplicated (one for each version of the structure). I guess these are the workarounds we have to do.
>>
> No, there is never any need for two versions of the inline functions, only
> the newest version is needed. This is because in the case of a newly compiled
> application only the newest version of the non-inline functions is ever
> used. The other older versions are only used at runtime for compatilibity
> with pre-compiled apps with the older inlines.
> 
>>>
>>> 3. If instead we have the case, like in rte_ring, where the data
>>> structures are allocated using functions, but they are used via inlines
>>> in the app. In this case the creation functions (and any other function
>>> using the structures) need to be versioned so that the application gets
>>> the expected structure back from the creation.
>> The new structure members have to be added such that the previous layout
>> is not affected. Either add at the end or use the gaps that are left
>> because of cache line alignment.
>>
> Not necessarily. There is nothing that requires the older and newer
> versions of a function to use the same structure. We can rename the
> original structure when versioning the old function, and then create a new
> structure with the original name for later recompiled code using the newest
> ABIs.
> 
>>>
>>> It might be useful to think of what other scenarios we have and what ones
>>> are likely to be problematic, especially those that can't be solved by having
>>> multiple function versions.
>>>
>>>> 2) Every function that is not in the direct datapath (fastpath?)
>>>> should not be inline. Exceptions or things like rx/tx burst, ring
>>>> enqueue/dequeue, and packet alloc/free - Stephen
>>>
>>> Agree with this. Anything not data path should not be inline. The next
>> Yes, very clear on this.
>>
>>> question then is for data path items how to determine whether they need to
>>> be inline or not. In general my rule-of-thumb:
>>> * anything dealing with bursts of packets should not be inline
>>> * anything what works with single packet at a time should be inline
>>>
>>> The one exception to this rule is cases where we have to consider "empty
>>> polling" as a likely use-case. For example, rte_ring_dequeue_burst works
>>> with bursts of packets, but there is a valid application use case where a
>>> thread could be polling a set of rings where only a small number are
>>> actually busy. Right now, polling an empty ring only costs a few cycles,
>>> meaning that it's neglible to have a few polls of empty rings before you get
>>> to a busy one. Having that function not-inline will cause that cost to jump
>>> significantly, so could cause problems. (This leads to the interesting scenario
>>> where ring enqueue is safe to un-inline, while dequeue is not).
>> A good way to think about it would be - ratio of amount of work done in the function to cost of function call.
>>
> 
> Yes, I would tend to agree in general.  The other thing is the frequency of
> calls, and - as already stated - whether it takes a burst of not.  Because
> even if it's a trivial function that takes only 10 cycles and we want to
> uninline it, the cost may double; but if it takes a burst of packets and is
> only used once/twice per burst the cost per packet should still only be a
> fraction of a cycle.
> 
>>>
>>>> 3) Plus synchronization routines: spin/rwlock/barrier, etc. I think
>>>> rcu should be one of such exceptions - it is just another
>>>> synchronization mechanism after all (just a bit more sophisticated). -
>>>> Konstantin
>>>>
>>> In general I believe the synchronisation primitives should be inline.
>>> However, it does come down to cost too - if a function takes 300 cycles, do
>>> we really care if it takes 305 or 310 instead to make it not inline?
>>> Hopefully most synchronisation primitives are faster than this so this
>>> situation should not occur.
>>>
>>>> 2) and 3) can be good guidelines to what functions/APIs can be inline. But,
>>> I believe this guideline exists today too. Are there any thoughts to change
>>> this?
>>>>
>>>> Coming to 1), I think DPDK cannot provide ABI compatibility unless all the
>>> inline functions are converted to normal functions and symbol versioning is
>>> done for those (not bothering about performance).
>>>>
>>> I disagree. I think even in the case of #1, we should be able to manage some
>>> changes without breaking ABI.
>> I completely agree with you on trying to keep the ABI break surface small and doing due diligence when one is required. However, I think claiming 100% ABI compatibility all the time (or as frequently as other projects claim) might be tough. IMO, we need to look beyond this to solve the end user problem. May be packaging multiple LTS with distros when DPDK could not avoid breaking ABI compatibility?
>>
> Having multiple LTS's per distro would be nice, but it's putting a lot more
> work on the distro folks, I think.

I completely disagree with this approach.
Anytime I have seen this done, most frequently with interpreted
languages, think multiple versions of Java and Python concurrently on a
system, it has always been a usability nightmare.


> 
>>>
>>>> In this context, does it make sense to say that we will maintain API
>>>> compatibility rather than saying ABI compatibility? This will also
>>>> send the right message to the end users.
>>>>
>>> I would value ABI compatibility much higher than API compatibility. If
>>> someone is recompiling the application anyway, making a couple of small
>>> changes (large rework is obviously a different issue) to the code should not
>>> be a massive issue, I hope. On the other hand, ABI compatibility is needed to
>>> allow seamless update from one version to another, and it's that ABI
>>> compatiblity that allows distro's to pick up our latest and greatest versions.
>>>
>> I think it is also important to setup the right expectations to DPDK users. i.e. DPDK will police itself better to provide ABI compatibility but occasionally it might not be possible.
>>
> The trouble here is that a DPDK release, as a product is either backward
> compatible or not. Either a user can take it as a drop-in replacement for
> the previous version or not. Users do not want to have to go through a
> checklist for each app to see if the app uses only "compatible" APIs or
> not. Same for the distro folks, they are not going to take some libs from a
> release but not others because the ABI is broken for some but not others.

Agreed, a DPDK release is either backwards compatible or it isn't, and
to Bruce's point if it isn't, it maters less if this is for a big API
change or a small API change - the failure of the API stability
guarantee is the significant piece.

The reality is that most other system libraries provide strong
guarantees ... to date we have provided very little.

If we start saying "Yes, but except when", the exception case very
quickly becomes the general case and then we are back to the beginning
again.

> 
> Regards,
> /Bruce
> 

Re: [dpdk-dev] ABI and inline functions

[Ray Kinsella]

On 18/04/2019 11:28, Bruce Richardson wrote:
> On Thu, Apr 18, 2019 at 04:34:53AM +0000, Honnappa Nagarahalli wrote:
>>>
>>> On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli wrote:
>>>> Hello,
>>>> 	There was a conversation [1] in the context of RCU library. I thought
>>>> it needs participation from broader audience. Summary for the context
>>>> (please look at [1] for full details)
>>>>
>>>
>>> Thanks for kicking off this discussion
>>>
>>>> 1) How do we provide ABI compatibility when the code base contains
>>> inline functions? Unless we freeze development in these inline functions and
>>> corresponding structures, it might not be possible to claim ABI compatibility.
>>> Application has to be recompiled.
>>>
>>> I agree that in some cases the application "might" need to be recompiled,
>>> but on the other hand I also think that there are many cases where ABI
>>> function versioning can still be used to mitigate things. For example, if we
>>> think of a couple of various scenarios:
>>>
>>> 1. If everything is inline and variables are allocated by app, e.g.
>>> spinlock on stack, then there is no issue as everything is application
>>> contained.
>> If there is a bug fix which requires the structure to change, the application would need to recompile. I guess you are talking about a scenario when nothing changed in the inline function/variables.
>>
> 
> If the application wants the bugfix, then yes. However, if the app is
> unaffected by the bug, then it should have the option of updating DPDK
> without a recompile.

I would also imagine that should be an extremely rare case, that a
bugfix would require a structure change ... perhaps for an alignment issues?

> 
>>>
>>> 2. If the situation is as in #1, but the structures in question are passed to
>>> non-inline DPDK functions. In this case, any changes to the structures require
>>> those functions taking the structures to be versioned for old and new
>>> structures
>> I think this can get complicated on the inline function side. The application and the DPDK library will end up having different inline functions. The changed inline function needs to be aware of 2 structure formats or the inline function needs to be duplicated (one for each version of the structure). I guess these are the workarounds we have to do.
>>
> No, there is never any need for two versions of the inline functions, only
> the newest version is needed. This is because in the case of a newly compiled
> application only the newest version of the non-inline functions is ever
> used. The other older versions are only used at runtime for compatilibity
> with pre-compiled apps with the older inlines.
> 
>>>
>>> 3. If instead we have the case, like in rte_ring, where the data
>>> structures are allocated using functions, but they are used via inlines
>>> in the app. In this case the creation functions (and any other function
>>> using the structures) need to be versioned so that the application gets
>>> the expected structure back from the creation.
>> The new structure members have to be added such that the previous layout
>> is not affected. Either add at the end or use the gaps that are left
>> because of cache line alignment.
>>
> Not necessarily. There is nothing that requires the older and newer
> versions of a function to use the same structure. We can rename the
> original structure when versioning the old function, and then create a new
> structure with the original name for later recompiled code using the newest
> ABIs.
> 
>>>
>>> It might be useful to think of what other scenarios we have and what ones
>>> are likely to be problematic, especially those that can't be solved by having
>>> multiple function versions.
>>>
>>>> 2) Every function that is not in the direct datapath (fastpath?)
>>>> should not be inline. Exceptions or things like rx/tx burst, ring
>>>> enqueue/dequeue, and packet alloc/free - Stephen
>>>
>>> Agree with this. Anything not data path should not be inline. The next
>> Yes, very clear on this.
>>
>>> question then is for data path items how to determine whether they need to
>>> be inline or not. In general my rule-of-thumb:
>>> * anything dealing with bursts of packets should not be inline
>>> * anything what works with single packet at a time should be inline
>>>
>>> The one exception to this rule is cases where we have to consider "empty
>>> polling" as a likely use-case. For example, rte_ring_dequeue_burst works
>>> with bursts of packets, but there is a valid application use case where a
>>> thread could be polling a set of rings where only a small number are
>>> actually busy. Right now, polling an empty ring only costs a few cycles,
>>> meaning that it's neglible to have a few polls of empty rings before you get
>>> to a busy one. Having that function not-inline will cause that cost to jump
>>> significantly, so could cause problems. (This leads to the interesting scenario
>>> where ring enqueue is safe to un-inline, while dequeue is not).
>> A good way to think about it would be - ratio of amount of work done in the function to cost of function call.
>>
> 
> Yes, I would tend to agree in general.  The other thing is the frequency of
> calls, and - as already stated - whether it takes a burst of not.  Because
> even if it's a trivial function that takes only 10 cycles and we want to
> uninline it, the cost may double; but if it takes a burst of packets and is
> only used once/twice per burst the cost per packet should still only be a
> fraction of a cycle.
> 
>>>
>>>> 3) Plus synchronization routines: spin/rwlock/barrier, etc. I think
>>>> rcu should be one of such exceptions - it is just another
>>>> synchronization mechanism after all (just a bit more sophisticated). -
>>>> Konstantin
>>>>
>>> In general I believe the synchronisation primitives should be inline.
>>> However, it does come down to cost too - if a function takes 300 cycles, do
>>> we really care if it takes 305 or 310 instead to make it not inline?
>>> Hopefully most synchronisation primitives are faster than this so this
>>> situation should not occur.
>>>
>>>> 2) and 3) can be good guidelines to what functions/APIs can be inline. But,
>>> I believe this guideline exists today too. Are there any thoughts to change
>>> this?
>>>>
>>>> Coming to 1), I think DPDK cannot provide ABI compatibility unless all the
>>> inline functions are converted to normal functions and symbol versioning is
>>> done for those (not bothering about performance).
>>>>
>>> I disagree. I think even in the case of #1, we should be able to manage some
>>> changes without breaking ABI.
>> I completely agree with you on trying to keep the ABI break surface small and doing due diligence when one is required. However, I think claiming 100% ABI compatibility all the time (or as frequently as other projects claim) might be tough. IMO, we need to look beyond this to solve the end user problem. May be packaging multiple LTS with distros when DPDK could not avoid breaking ABI compatibility?
>>
> Having multiple LTS's per distro would be nice, but it's putting a lot more
> work on the distro folks, I think.

I completely disagree with this approach.
Anytime I have seen this done, most frequently with interpreted
languages, think multiple versions of Java and Python concurrently on a
system, it has always been a usability nightmare.


> 
>>>
>>>> In this context, does it make sense to say that we will maintain API
>>>> compatibility rather than saying ABI compatibility? This will also
>>>> send the right message to the end users.
>>>>
>>> I would value ABI compatibility much higher than API compatibility. If
>>> someone is recompiling the application anyway, making a couple of small
>>> changes (large rework is obviously a different issue) to the code should not
>>> be a massive issue, I hope. On the other hand, ABI compatibility is needed to
>>> allow seamless update from one version to another, and it's that ABI
>>> compatiblity that allows distro's to pick up our latest and greatest versions.
>>>
>> I think it is also important to setup the right expectations to DPDK users. i.e. DPDK will police itself better to provide ABI compatibility but occasionally it might not be possible.
>>
> The trouble here is that a DPDK release, as a product is either backward
> compatible or not. Either a user can take it as a drop-in replacement for
> the previous version or not. Users do not want to have to go through a
> checklist for each app to see if the app uses only "compatible" APIs or
> not. Same for the distro folks, they are not going to take some libs from a
> release but not others because the ABI is broken for some but not others.

Agreed, a DPDK release is either backwards compatible or it isn't, and
to Bruce's point if it isn't, it maters less if this is for a big API
change or a small API change - the failure of the API stability
guarantee is the significant piece.

The reality is that most other system libraries provide strong
guarantees ... to date we have provided very little.

If we start saying "Yes, but except when", the exception case very
quickly becomes the general case and then we are back to the beginning
again.

> 
> Regards,
> /Bruce
> 

Re: [dpdk-dev] ABI and inline functions

[Ray Kinsella]

On 17/04/2019 19:51, Stephen Hemminger wrote:
> On Wed, 17 Apr 2019 17:46:34 +0000
> Jerin Jacob Kollanukkaran <jerinj@marvell.com> wrote:
> 
>>> -----Original Message-----
>>> From: dev <dev-bounces@dpdk.org> On Behalf Of Bruce Richardson
>>> Sent: Wednesday, April 17, 2019 2:07 PM
>>> To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
>>> Cc: dev@dpdk.org; Stephen Hemminger <stephen@networkplumber.org>;
>>> Ananyev, Konstantin <konstantin.ananyev@intel.com>; thomas@monjalon.net;
>>> Ray Kinsella <mdr@ashroe.eu>; nd <nd@arm.com>
>>> Subject: Re: [dpdk-dev] ABI and inline functions
>>>   
>>>> 2) Every function that is not in the direct datapath (fastpath?)
>>>> should not be inline. Exceptions or things like rx/tx burst, ring
>>>> enqueue/dequeue, and packet alloc/free - Stephen  
>>>
>>> Agree with this. Anything not data path should not be inline. The next question
>>> then is for data path items how to determine whether they need to be inline or
>>> not. In general my rule-of-thumb:
>>> * anything dealing with bursts of packets should not be inline  
>>
>> # I think, we need consider the how fat is the function too,
>> If it is light weight, probably we need to make it inline

Well if it light weight however it is not dealing directly with packets,
probably does not matter if is not-inline?

>> # Except the forward case, In real world use case, it is not trivial make large 
>> burst of packet to process it even though function prototype burst.
>> We may need to consider that

>> # For the given function if some argument is used with inline other function,
>>  probably no point in making that function alone  inline as the structure 
>> is exposed in some other function.
>>
>>> * anything what works with single packet at a time should be inline
>>>   
>>>> In this context, does it make sense to say that we will maintain API
>>>> compatibility rather than saying ABI compatibility? This will also
>>>> send the right message to the end users.
>>>>  
>>> I would value ABI compatibility much higher than API compatibility. If someone
>>> is recompiling the application anyway, making a couple of small changes (large
>>> rework is obviously a different issue) to the code should not be a massive issue, I
>>> hope. On the other hand, ABI compatibility is needed to allow seamless update
>>> from one version to another, and it's that ABI compatiblity that allows distro's to
>>> pick up our latest and greatest versions.  
>>
>> IMO, We have two primary use case for DPDK
>>
>> 1) NFV kind of use case 	where the application needs to run on multiple platform
>> without recompiling it.
>> 2) Fixed appliance use case where embed SoC like Intel Denverton or ARM64 integrated
>> Controller used. For fixed appliance use case, end user care more of performance
>> than ABI compatibility as it easy to recompile the end user application vs the cost of
>> hitting performance impact.
> 
> Nobody cares about compatiablity until they have to the first security update.
>  
+1

Re: [dpdk-dev] ABI and inline functions

[Ray Kinsella]

On 17/04/2019 19:51, Stephen Hemminger wrote:
> On Wed, 17 Apr 2019 17:46:34 +0000
> Jerin Jacob Kollanukkaran <jerinj@marvell.com> wrote:
> 
>>> -----Original Message-----
>>> From: dev <dev-bounces@dpdk.org> On Behalf Of Bruce Richardson
>>> Sent: Wednesday, April 17, 2019 2:07 PM
>>> To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
>>> Cc: dev@dpdk.org; Stephen Hemminger <stephen@networkplumber.org>;
>>> Ananyev, Konstantin <konstantin.ananyev@intel.com>; thomas@monjalon.net;
>>> Ray Kinsella <mdr@ashroe.eu>; nd <nd@arm.com>
>>> Subject: Re: [dpdk-dev] ABI and inline functions
>>>   
>>>> 2) Every function that is not in the direct datapath (fastpath?)
>>>> should not be inline. Exceptions or things like rx/tx burst, ring
>>>> enqueue/dequeue, and packet alloc/free - Stephen  
>>>
>>> Agree with this. Anything not data path should not be inline. The next question
>>> then is for data path items how to determine whether they need to be inline or
>>> not. In general my rule-of-thumb:
>>> * anything dealing with bursts of packets should not be inline  
>>
>> # I think, we need consider the how fat is the function too,
>> If it is light weight, probably we need to make it inline

Well if it light weight however it is not dealing directly with packets,
probably does not matter if is not-inline?

>> # Except the forward case, In real world use case, it is not trivial make large 
>> burst of packet to process it even though function prototype burst.
>> We may need to consider that

>> # For the given function if some argument is used with inline other function,
>>  probably no point in making that function alone  inline as the structure 
>> is exposed in some other function.
>>
>>> * anything what works with single packet at a time should be inline
>>>   
>>>> In this context, does it make sense to say that we will maintain API
>>>> compatibility rather than saying ABI compatibility? This will also
>>>> send the right message to the end users.
>>>>  
>>> I would value ABI compatibility much higher than API compatibility. If someone
>>> is recompiling the application anyway, making a couple of small changes (large
>>> rework is obviously a different issue) to the code should not be a massive issue, I
>>> hope. On the other hand, ABI compatibility is needed to allow seamless update
>>> from one version to another, and it's that ABI compatiblity that allows distro's to
>>> pick up our latest and greatest versions.  
>>
>> IMO, We have two primary use case for DPDK
>>
>> 1) NFV kind of use case 	where the application needs to run on multiple platform
>> without recompiling it.
>> 2) Fixed appliance use case where embed SoC like Intel Denverton or ARM64 integrated
>> Controller used. For fixed appliance use case, end user care more of performance
>> than ABI compatibility as it easy to recompile the end user application vs the cost of
>> hitting performance impact.
> 
> Nobody cares about compatiablity until they have to the first security update.
>  
+1

Re: [dpdk-dev] [EXT] Re: ABI and inline functions

[Ray Kinsella]

On 18/04/2019 06:56, Jerin Jacob Kollanukkaran wrote:
> 
>> -----Original Message-----
>> From: Stephen Hemminger <stephen@networkplumber.org>
>> Sent: Thursday, April 18, 2019 12:21 AM
>> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
>> Cc: Bruce Richardson <bruce.richardson@intel.com>; Honnappa Nagarahalli
>> <Honnappa.Nagarahalli@arm.com>; dev@dpdk.org; Ananyev, Konstantin
>> <konstantin.ananyev@intel.com>; thomas@monjalon.net; Ray Kinsella
>> <mdr@ashroe.eu>; nd <nd@arm.com>
>> Subject: [EXT] Re: [dpdk-dev] ABI and inline functions
>>> I would value ABI compatibility much higher than API compatibility.
>>>> If someone is recompiling the application anyway, making a couple of
>>>> small changes (large rework is obviously a different issue) to the
>>>> code should not be a massive issue, I hope. On the other hand, ABI
>>>> compatibility is needed to allow seamless update from one version to
>>>> another, and it's that ABI compatiblity that allows distro's to pick up our
>> latest and greatest versions.
>>>
>>> IMO, We have two primary use case for DPDK
>>>
>>> 1) NFV kind of use case 	where the application needs to run on multiple
>> platform
>>> without recompiling it.
>>> 2) Fixed appliance use case where embed SoC like Intel Denverton or
>>> ARM64 integrated Controller used. For fixed appliance use case, end
>>> user care more of performance than ABI compatibility as it easy to
>>> recompile the end user application vs the cost of hitting performance impact.
>>
>> Nobody cares about compatiablity until they have to the first security update.
> 
> For fixed appliance case, The update(FW update) will be  a single blob which
> Include all the components. So they can back port the security fix and recompile
> the sw as needed.
> 
> The very similar category  is DPDK running in smart NICs(Runs as FW in PCIe EP device).

So is there a real versus a perceived compromise happen here - that we
are compromising optimal performance in order to make API stability
happen? Do we have specific an examples that this is actually the case?

Thanks,

Ray K

Re: [dpdk-dev] [EXT] Re: ABI and inline functions

[Ray Kinsella]

On 18/04/2019 06:56, Jerin Jacob Kollanukkaran wrote:
> 
>> -----Original Message-----
>> From: Stephen Hemminger <stephen@networkplumber.org>
>> Sent: Thursday, April 18, 2019 12:21 AM
>> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
>> Cc: Bruce Richardson <bruce.richardson@intel.com>; Honnappa Nagarahalli
>> <Honnappa.Nagarahalli@arm.com>; dev@dpdk.org; Ananyev, Konstantin
>> <konstantin.ananyev@intel.com>; thomas@monjalon.net; Ray Kinsella
>> <mdr@ashroe.eu>; nd <nd@arm.com>
>> Subject: [EXT] Re: [dpdk-dev] ABI and inline functions
>>> I would value ABI compatibility much higher than API compatibility.
>>>> If someone is recompiling the application anyway, making a couple of
>>>> small changes (large rework is obviously a different issue) to the
>>>> code should not be a massive issue, I hope. On the other hand, ABI
>>>> compatibility is needed to allow seamless update from one version to
>>>> another, and it's that ABI compatiblity that allows distro's to pick up our
>> latest and greatest versions.
>>>
>>> IMO, We have two primary use case for DPDK
>>>
>>> 1) NFV kind of use case 	where the application needs to run on multiple
>> platform
>>> without recompiling it.
>>> 2) Fixed appliance use case where embed SoC like Intel Denverton or
>>> ARM64 integrated Controller used. For fixed appliance use case, end
>>> user care more of performance than ABI compatibility as it easy to
>>> recompile the end user application vs the cost of hitting performance impact.
>>
>> Nobody cares about compatiablity until they have to the first security update.
> 
> For fixed appliance case, The update(FW update) will be  a single blob which
> Include all the components. So they can back port the security fix and recompile
> the sw as needed.
> 
> The very similar category  is DPDK running in smart NICs(Runs as FW in PCIe EP device).

So is there a real versus a perceived compromise happen here - that we
are compromising optimal performance in order to make API stability
happen? Do we have specific an examples that this is actually the case?

Thanks,

Ray K

Re: [dpdk-dev] ABI and inline functions

[Honnappa Nagarahalli]

> > > On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli
> wrote:
> > > > Hello,
> > > > 	There was a conversation [1] in the context of RCU library. I
> > > > thought it needs participation from broader audience. Summary for
> > > > the context (please look at [1] for full details)
> > > >
> > >
> > > Thanks for kicking off this discussion
> > >
> > > > 1) How do we provide ABI compatibility when the code base contains
> > > inline functions? Unless we freeze development in these inline
> > > functions and corresponding structures, it might not be possible to claim
> ABI compatibility.
> > > Application has to be recompiled.
> > >
> > > I agree that in some cases the application "might" need to be
> > > recompiled, but on the other hand I also think that there are many
> > > cases where ABI function versioning can still be used to mitigate
> > > things. For example, if we think of a couple of various scenarios:
> > >
> > > 1. If everything is inline and variables are allocated by app, e.g.
> > > spinlock on stack, then there is no issue as everything is
> > > application contained.
> > If there is a bug fix which requires the structure to change, the application
> would need to recompile. I guess you are talking about a scenario when
> nothing changed in the inline function/variables.
> >
> 
> If the application wants the bugfix, then yes. However, if the app is
> unaffected by the bug, then it should have the option of updating DPDK
> without a recompile.
>
Agree on the requirement.

> > >
> > > 2. If the situation is as in #1, but the structures in question are
> > > passed to non-inline DPDK functions. In this case, any changes to
> > > the structures require those functions taking the structures to be
> > > versioned for old and new structures
> > I think this can get complicated on the inline function side. The application
> and the DPDK library will end up having different inline functions. The
> changed inline function needs to be aware of 2 structure formats or the inline
> function needs to be duplicated (one for each version of the structure). I
> guess these are the workarounds we have to do.
> >
> No, there is never any need for two versions of the inline functions, only the
> newest version is needed. This is because in the case of a newly compiled
> application only the newest version of the non-inline functions is ever used.
> The other older versions are only used at runtime for compatilibity with pre-
> compiled apps with the older inlines.
> 
Since the inline function is used in the application and the DPDK (say a library), we have 2 copies of the same inline function in the final binary (1 with the application and 2nd one behind DPDK non-inline functions). When the DPDK non-inline functions are versioned, the older version of the functions have to support the old structures for the old application to work with the new DPDK. i.e. both copies of the inline function have to be the same.

> > >
> > > 3. If instead we have the case, like in rte_ring, where the data
> > > structures are allocated using functions, but they are used via
> > > inlines in the app. In this case the creation functions (and any
> > > other function using the structures) need to be versioned so that
> > > the application gets the expected structure back from the creation.
> > The new structure members have to be added such that the previous
> > layout is not affected. Either add at the end or use the gaps that are
> > left because of cache line alignment.
> >
> Not necessarily. There is nothing that requires the older and newer versions
> of a function to use the same structure. We can rename the original structure
> when versioning the old function, and then create a new structure with the
> original name for later recompiled code using the newest ABIs.
>
Agree, assuming only the app is using the inline functions.

> > >
> > > It might be useful to think of what other scenarios we have and what
> > > ones are likely to be problematic, especially those that can't be
> > > solved by having multiple function versions.
> > >
> > > > 2) Every function that is not in the direct datapath (fastpath?)
> > > > should not be inline. Exceptions or things like rx/tx burst, ring
> > > > enqueue/dequeue, and packet alloc/free - Stephen
> > >
> > > Agree with this. Anything not data path should not be inline. The
> > > next
> > Yes, very clear on this.
> >
> > > question then is for data path items how to determine whether they
> > > need to be inline or not. In general my rule-of-thumb:
> > > * anything dealing with bursts of packets should not be inline
> > > * anything what works with single packet at a time should be inline
> > >
> > > The one exception to this rule is cases where we have to consider
> > > "empty polling" as a likely use-case. For example,
> > > rte_ring_dequeue_burst works with bursts of packets, but there is a
> > > valid application use case where a thread could be polling a set of
> > > rings where only a small number are actually busy. Right now,
> > > polling an empty ring only costs a few cycles, meaning that it's
> > > neglible to have a few polls of empty rings before you get to a busy
> > > one. Having that function not-inline will cause that cost to jump
> > > significantly, so could cause problems. (This leads to the interesting
> scenario where ring enqueue is safe to un-inline, while dequeue is not).
> > A good way to think about it would be - ratio of amount of work done in
> the function to cost of function call.
> >
> 
> Yes, I would tend to agree in general.  The other thing is the frequency of
> calls, and - as already stated - whether it takes a burst of not.  Because even
> if it's a trivial function that takes only 10 cycles and we want to uninline it,
> the cost may double; but if it takes a burst of packets and is only used
> once/twice per burst the cost per packet should still only be a fraction of a
> cycle.
> 
> > >
> > > > 3) Plus synchronization routines: spin/rwlock/barrier, etc. I
> > > > think rcu should be one of such exceptions - it is just another
> > > > synchronization mechanism after all (just a bit more
> > > > sophisticated). - Konstantin
> > > >
> > > In general I believe the synchronisation primitives should be inline.
> > > However, it does come down to cost too - if a function takes 300
> > > cycles, do we really care if it takes 305 or 310 instead to make it not
> inline?
> > > Hopefully most synchronisation primitives are faster than this so
> > > this situation should not occur.
> > >
> > > > 2) and 3) can be good guidelines to what functions/APIs can be
> > > > inline. But,
> > > I believe this guideline exists today too. Are there any thoughts to
> > > change this?
> > > >
> > > > Coming to 1), I think DPDK cannot provide ABI compatibility unless
> > > > all the
> > > inline functions are converted to normal functions and symbol
> > > versioning is done for those (not bothering about performance).
> > > >
> > > I disagree. I think even in the case of #1, we should be able to
> > > manage some changes without breaking ABI.
> > I completely agree with you on trying to keep the ABI break surface small
> and doing due diligence when one is required. However, I think claiming 100%
> ABI compatibility all the time (or as frequently as other projects claim) might
> be tough. IMO, we need to look beyond this to solve the end user problem.
> May be packaging multiple LTS with distros when DPDK could not avoid
> breaking ABI compatibility?
> >
> Having multiple LTS's per distro would be nice, but it's putting a lot more
> work on the distro folks, I think.
> 
> > >
> > > > In this context, does it make sense to say that we will maintain
> > > > API compatibility rather than saying ABI compatibility? This will
> > > > also send the right message to the end users.
> > > >
> > > I would value ABI compatibility much higher than API compatibility.
> > > If someone is recompiling the application anyway, making a couple of
> > > small changes (large rework is obviously a different issue) to the
> > > code should not be a massive issue, I hope. On the other hand, ABI
> > > compatibility is needed to allow seamless update from one version to
> > > another, and it's that ABI compatiblity that allows distro's to pick up our
> latest and greatest versions.
> > >
> > I think it is also important to setup the right expectations to DPDK users. i.e.
> DPDK will police itself better to provide ABI compatibility but occasionally it
> might not be possible.
> >
> The trouble here is that a DPDK release, as a product is either backward
> compatible or not. Either a user can take it as a drop-in replacement for the
> previous version or not. Users do not want to have to go through a checklist
> for each app to see if the app uses only "compatible" APIs or not. Same for
For the releases where the ABI is broken, would not such a list help the users?

> the distro folks, they are not going to take some libs from a release but not
> others because the ABI is broken for some but not others.
Yes, this is not an option, it becomes a DPDK release of its own 😊.

> 
> Regards,
> /Bruce

Re: [dpdk-dev] ABI and inline functions

[Honnappa Nagarahalli]

> > > On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli
> wrote:
> > > > Hello,
> > > > 	There was a conversation [1] in the context of RCU library. I
> > > > thought it needs participation from broader audience. Summary for
> > > > the context (please look at [1] for full details)
> > > >
> > >
> > > Thanks for kicking off this discussion
> > >
> > > > 1) How do we provide ABI compatibility when the code base contains
> > > inline functions? Unless we freeze development in these inline
> > > functions and corresponding structures, it might not be possible to claim
> ABI compatibility.
> > > Application has to be recompiled.
> > >
> > > I agree that in some cases the application "might" need to be
> > > recompiled, but on the other hand I also think that there are many
> > > cases where ABI function versioning can still be used to mitigate
> > > things. For example, if we think of a couple of various scenarios:
> > >
> > > 1. If everything is inline and variables are allocated by app, e.g.
> > > spinlock on stack, then there is no issue as everything is
> > > application contained.
> > If there is a bug fix which requires the structure to change, the application
> would need to recompile. I guess you are talking about a scenario when
> nothing changed in the inline function/variables.
> >
> 
> If the application wants the bugfix, then yes. However, if the app is
> unaffected by the bug, then it should have the option of updating DPDK
> without a recompile.
>
Agree on the requirement.

> > >
> > > 2. If the situation is as in #1, but the structures in question are
> > > passed to non-inline DPDK functions. In this case, any changes to
> > > the structures require those functions taking the structures to be
> > > versioned for old and new structures
> > I think this can get complicated on the inline function side. The application
> and the DPDK library will end up having different inline functions. The
> changed inline function needs to be aware of 2 structure formats or the inline
> function needs to be duplicated (one for each version of the structure). I
> guess these are the workarounds we have to do.
> >
> No, there is never any need for two versions of the inline functions, only the
> newest version is needed. This is because in the case of a newly compiled
> application only the newest version of the non-inline functions is ever used.
> The other older versions are only used at runtime for compatilibity with pre-
> compiled apps with the older inlines.
> 
Since the inline function is used in the application and the DPDK (say a library), we have 2 copies of the same inline function in the final binary (1 with the application and 2nd one behind DPDK non-inline functions). When the DPDK non-inline functions are versioned, the older version of the functions have to support the old structures for the old application to work with the new DPDK. i.e. both copies of the inline function have to be the same.

> > >
> > > 3. If instead we have the case, like in rte_ring, where the data
> > > structures are allocated using functions, but they are used via
> > > inlines in the app. In this case the creation functions (and any
> > > other function using the structures) need to be versioned so that
> > > the application gets the expected structure back from the creation.
> > The new structure members have to be added such that the previous
> > layout is not affected. Either add at the end or use the gaps that are
> > left because of cache line alignment.
> >
> Not necessarily. There is nothing that requires the older and newer versions
> of a function to use the same structure. We can rename the original structure
> when versioning the old function, and then create a new structure with the
> original name for later recompiled code using the newest ABIs.
>
Agree, assuming only the app is using the inline functions.

> > >
> > > It might be useful to think of what other scenarios we have and what
> > > ones are likely to be problematic, especially those that can't be
> > > solved by having multiple function versions.
> > >
> > > > 2) Every function that is not in the direct datapath (fastpath?)
> > > > should not be inline. Exceptions or things like rx/tx burst, ring
> > > > enqueue/dequeue, and packet alloc/free - Stephen
> > >
> > > Agree with this. Anything not data path should not be inline. The
> > > next
> > Yes, very clear on this.
> >
> > > question then is for data path items how to determine whether they
> > > need to be inline or not. In general my rule-of-thumb:
> > > * anything dealing with bursts of packets should not be inline
> > > * anything what works with single packet at a time should be inline
> > >
> > > The one exception to this rule is cases where we have to consider
> > > "empty polling" as a likely use-case. For example,
> > > rte_ring_dequeue_burst works with bursts of packets, but there is a
> > > valid application use case where a thread could be polling a set of
> > > rings where only a small number are actually busy. Right now,
> > > polling an empty ring only costs a few cycles, meaning that it's
> > > neglible to have a few polls of empty rings before you get to a busy
> > > one. Having that function not-inline will cause that cost to jump
> > > significantly, so could cause problems. (This leads to the interesting
> scenario where ring enqueue is safe to un-inline, while dequeue is not).
> > A good way to think about it would be - ratio of amount of work done in
> the function to cost of function call.
> >
> 
> Yes, I would tend to agree in general.  The other thing is the frequency of
> calls, and - as already stated - whether it takes a burst of not.  Because even
> if it's a trivial function that takes only 10 cycles and we want to uninline it,
> the cost may double; but if it takes a burst of packets and is only used
> once/twice per burst the cost per packet should still only be a fraction of a
> cycle.
> 
> > >
> > > > 3) Plus synchronization routines: spin/rwlock/barrier, etc. I
> > > > think rcu should be one of such exceptions - it is just another
> > > > synchronization mechanism after all (just a bit more
> > > > sophisticated). - Konstantin
> > > >
> > > In general I believe the synchronisation primitives should be inline.
> > > However, it does come down to cost too - if a function takes 300
> > > cycles, do we really care if it takes 305 or 310 instead to make it not
> inline?
> > > Hopefully most synchronisation primitives are faster than this so
> > > this situation should not occur.
> > >
> > > > 2) and 3) can be good guidelines to what functions/APIs can be
> > > > inline. But,
> > > I believe this guideline exists today too. Are there any thoughts to
> > > change this?
> > > >
> > > > Coming to 1), I think DPDK cannot provide ABI compatibility unless
> > > > all the
> > > inline functions are converted to normal functions and symbol
> > > versioning is done for those (not bothering about performance).
> > > >
> > > I disagree. I think even in the case of #1, we should be able to
> > > manage some changes without breaking ABI.
> > I completely agree with you on trying to keep the ABI break surface small
> and doing due diligence when one is required. However, I think claiming 100%
> ABI compatibility all the time (or as frequently as other projects claim) might
> be tough. IMO, we need to look beyond this to solve the end user problem.
> May be packaging multiple LTS with distros when DPDK could not avoid
> breaking ABI compatibility?
> >
> Having multiple LTS's per distro would be nice, but it's putting a lot more
> work on the distro folks, I think.
> 
> > >
> > > > In this context, does it make sense to say that we will maintain
> > > > API compatibility rather than saying ABI compatibility? This will
> > > > also send the right message to the end users.
> > > >
> > > I would value ABI compatibility much higher than API compatibility.
> > > If someone is recompiling the application anyway, making a couple of
> > > small changes (large rework is obviously a different issue) to the
> > > code should not be a massive issue, I hope. On the other hand, ABI
> > > compatibility is needed to allow seamless update from one version to
> > > another, and it's that ABI compatiblity that allows distro's to pick up our
> latest and greatest versions.
> > >
> > I think it is also important to setup the right expectations to DPDK users. i.e.
> DPDK will police itself better to provide ABI compatibility but occasionally it
> might not be possible.
> >
> The trouble here is that a DPDK release, as a product is either backward
> compatible or not. Either a user can take it as a drop-in replacement for the
> previous version or not. Users do not want to have to go through a checklist
> for each app to see if the app uses only "compatible" APIs or not. Same for
For the releases where the ABI is broken, would not such a list help the users?

> the distro folks, they are not going to take some libs from a release but not
> others because the ABI is broken for some but not others.
Yes, this is not an option, it becomes a DPDK release of its own 😊.

> 
> Regards,
> /Bruce

Re: [dpdk-dev] ABI and inline functions

[Honnappa Nagarahalli]

> On 18/04/2019 11:28, Bruce Richardson wrote:
> > On Thu, Apr 18, 2019 at 04:34:53AM +0000, Honnappa Nagarahalli wrote:
> >>>
> >>> On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli
> wrote:
> >>>> Hello,
> >>>> 	There was a conversation [1] in the context of RCU library. I
> >>>> thought it needs participation from broader audience. Summary for
> >>>> the context (please look at [1] for full details)
> >>>>
> >>>
> >>> Thanks for kicking off this discussion
> >>>
> >>>> 1) How do we provide ABI compatibility when the code base contains
> >>> inline functions? Unless we freeze development in these inline
> >>> functions and corresponding structures, it might not be possible to claim
> ABI compatibility.
> >>> Application has to be recompiled.
> >>>
> >>> I agree that in some cases the application "might" need to be
> >>> recompiled, but on the other hand I also think that there are many
> >>> cases where ABI function versioning can still be used to mitigate
> >>> things. For example, if we think of a couple of various scenarios:
> >>>
> >>> 1. If everything is inline and variables are allocated by app, e.g.
> >>> spinlock on stack, then there is no issue as everything is
> >>> application contained.
> >> If there is a bug fix which requires the structure to change, the application
> would need to recompile. I guess you are talking about a scenario when
> nothing changed in the inline function/variables.
> >>
> >
> > If the application wants the bugfix, then yes. However, if the app is
> > unaffected by the bug, then it should have the option of updating DPDK
> > without a recompile.
> 
> I would also imagine that should be an extremely rare case, that a bugfix
> would require a structure change ... perhaps for an alignment issues?
> 
> >
> >>>
> >>> 2. If the situation is as in #1, but the structures in question are
> >>> passed to non-inline DPDK functions. In this case, any changes to
> >>> the structures require those functions taking the structures to be
> >>> versioned for old and new structures
> >> I think this can get complicated on the inline function side. The application
> and the DPDK library will end up having different inline functions. The
> changed inline function needs to be aware of 2 structure formats or the inline
> function needs to be duplicated (one for each version of the structure). I
> guess these are the workarounds we have to do.
> >>
> > No, there is never any need for two versions of the inline functions,
> > only the newest version is needed. This is because in the case of a
> > newly compiled application only the newest version of the non-inline
> > functions is ever used. The other older versions are only used at
> > runtime for compatilibity with pre-compiled apps with the older inlines.
> >
> >>>
> >>> 3. If instead we have the case, like in rte_ring, where the data
> >>> structures are allocated using functions, but they are used via
> >>> inlines in the app. In this case the creation functions (and any
> >>> other function using the structures) need to be versioned so that
> >>> the application gets the expected structure back from the creation.
> >> The new structure members have to be added such that the previous
> >> layout is not affected. Either add at the end or use the gaps that
> >> are left because of cache line alignment.
> >>
> > Not necessarily. There is nothing that requires the older and newer
> > versions of a function to use the same structure. We can rename the
> > original structure when versioning the old function, and then create a
> > new structure with the original name for later recompiled code using
> > the newest ABIs.
> >
> >>>
> >>> It might be useful to think of what other scenarios we have and what
> >>> ones are likely to be problematic, especially those that can't be
> >>> solved by having multiple function versions.
> >>>
> >>>> 2) Every function that is not in the direct datapath (fastpath?)
> >>>> should not be inline. Exceptions or things like rx/tx burst, ring
> >>>> enqueue/dequeue, and packet alloc/free - Stephen
> >>>
> >>> Agree with this. Anything not data path should not be inline. The
> >>> next
> >> Yes, very clear on this.
> >>
> >>> question then is for data path items how to determine whether they
> >>> need to be inline or not. In general my rule-of-thumb:
> >>> * anything dealing with bursts of packets should not be inline
> >>> * anything what works with single packet at a time should be inline
> >>>
> >>> The one exception to this rule is cases where we have to consider
> >>> "empty polling" as a likely use-case. For example,
> >>> rte_ring_dequeue_burst works with bursts of packets, but there is a
> >>> valid application use case where a thread could be polling a set of
> >>> rings where only a small number are actually busy. Right now,
> >>> polling an empty ring only costs a few cycles, meaning that it's
> >>> neglible to have a few polls of empty rings before you get to a busy
> >>> one. Having that function not-inline will cause that cost to jump
> >>> significantly, so could cause problems. (This leads to the interesting
> scenario where ring enqueue is safe to un-inline, while dequeue is not).
> >> A good way to think about it would be - ratio of amount of work done in
> the function to cost of function call.
> >>
> >
> > Yes, I would tend to agree in general.  The other thing is the
> > frequency of calls, and - as already stated - whether it takes a burst
> > of not.  Because even if it's a trivial function that takes only 10
> > cycles and we want to uninline it, the cost may double; but if it
> > takes a burst of packets and is only used once/twice per burst the
> > cost per packet should still only be a fraction of a cycle.
> >
> >>>
> >>>> 3) Plus synchronization routines: spin/rwlock/barrier, etc. I think
> >>>> rcu should be one of such exceptions - it is just another
> >>>> synchronization mechanism after all (just a bit more
> >>>> sophisticated). - Konstantin
> >>>>
> >>> In general I believe the synchronisation primitives should be inline.
> >>> However, it does come down to cost too - if a function takes 300
> >>> cycles, do we really care if it takes 305 or 310 instead to make it not
> inline?
> >>> Hopefully most synchronisation primitives are faster than this so
> >>> this situation should not occur.
> >>>
> >>>> 2) and 3) can be good guidelines to what functions/APIs can be
> >>>> inline. But,
> >>> I believe this guideline exists today too. Are there any thoughts to
> >>> change this?
> >>>>
> >>>> Coming to 1), I think DPDK cannot provide ABI compatibility unless
> >>>> all the
> >>> inline functions are converted to normal functions and symbol
> >>> versioning is done for those (not bothering about performance).
> >>>>
> >>> I disagree. I think even in the case of #1, we should be able to
> >>> manage some changes without breaking ABI.
> >> I completely agree with you on trying to keep the ABI break surface small
> and doing due diligence when one is required. However, I think claiming 100%
> ABI compatibility all the time (or as frequently as other projects claim) might
> be tough. IMO, we need to look beyond this to solve the end user problem.
> May be packaging multiple LTS with distros when DPDK could not avoid
> breaking ABI compatibility?
> >>
> > Having multiple LTS's per distro would be nice, but it's putting a lot
> > more work on the distro folks, I think.
> 
> I completely disagree with this approach.
> Anytime I have seen this done, most frequently with interpreted languages,
> think multiple versions of Java and Python concurrently on a system, it has
> always been a usability nightmare.
> 
> 
> >
> >>>
> >>>> In this context, does it make sense to say that we will maintain
> >>>> API compatibility rather than saying ABI compatibility? This will
> >>>> also send the right message to the end users.
> >>>>
> >>> I would value ABI compatibility much higher than API compatibility.
> >>> If someone is recompiling the application anyway, making a couple of
> >>> small changes (large rework is obviously a different issue) to the
> >>> code should not be a massive issue, I hope. On the other hand, ABI
> >>> compatibility is needed to allow seamless update from one version to
> >>> another, and it's that ABI compatiblity that allows distro's to pick up our
> latest and greatest versions.
> >>>
> >> I think it is also important to setup the right expectations to DPDK users.
> i.e. DPDK will police itself better to provide ABI compatibility but occasionally
> it might not be possible.
> >>
> > The trouble here is that a DPDK release, as a product is either
> > backward compatible or not. Either a user can take it as a drop-in
> > replacement for the previous version or not. Users do not want to have
> > to go through a checklist for each app to see if the app uses only
> > "compatible" APIs or not. Same for the distro folks, they are not
> > going to take some libs from a release but not others because the ABI is
> broken for some but not others.
> 
> Agreed, a DPDK release is either backwards compatible or it isn't, and to
> Bruce's point if it isn't, it maters less if this is for a big API change or a small
> API change - the failure of the API stability guarantee is the significant piece.
> 
> The reality is that most other system libraries provide strong guarantees ... to
> date we have provided very little.
It would be good to look libraries that provide inline functions and ABI compatibility to better understand the methods used.

> 
> If we start saying "Yes, but except when", the exception case very quickly
> becomes the general case and then we are back to the beginning again.
> 
> >
> > Regards,
> > /Bruce
> >

Re: [dpdk-dev] ABI and inline functions

[Honnappa Nagarahalli]

> On 18/04/2019 11:28, Bruce Richardson wrote:
> > On Thu, Apr 18, 2019 at 04:34:53AM +0000, Honnappa Nagarahalli wrote:
> >>>
> >>> On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli
> wrote:
> >>>> Hello,
> >>>> 	There was a conversation [1] in the context of RCU library. I
> >>>> thought it needs participation from broader audience. Summary for
> >>>> the context (please look at [1] for full details)
> >>>>
> >>>
> >>> Thanks for kicking off this discussion
> >>>
> >>>> 1) How do we provide ABI compatibility when the code base contains
> >>> inline functions? Unless we freeze development in these inline
> >>> functions and corresponding structures, it might not be possible to claim
> ABI compatibility.
> >>> Application has to be recompiled.
> >>>
> >>> I agree that in some cases the application "might" need to be
> >>> recompiled, but on the other hand I also think that there are many
> >>> cases where ABI function versioning can still be used to mitigate
> >>> things. For example, if we think of a couple of various scenarios:
> >>>
> >>> 1. If everything is inline and variables are allocated by app, e.g.
> >>> spinlock on stack, then there is no issue as everything is
> >>> application contained.
> >> If there is a bug fix which requires the structure to change, the application
> would need to recompile. I guess you are talking about a scenario when
> nothing changed in the inline function/variables.
> >>
> >
> > If the application wants the bugfix, then yes. However, if the app is
> > unaffected by the bug, then it should have the option of updating DPDK
> > without a recompile.
> 
> I would also imagine that should be an extremely rare case, that a bugfix
> would require a structure change ... perhaps for an alignment issues?
> 
> >
> >>>
> >>> 2. If the situation is as in #1, but the structures in question are
> >>> passed to non-inline DPDK functions. In this case, any changes to
> >>> the structures require those functions taking the structures to be
> >>> versioned for old and new structures
> >> I think this can get complicated on the inline function side. The application
> and the DPDK library will end up having different inline functions. The
> changed inline function needs to be aware of 2 structure formats or the inline
> function needs to be duplicated (one for each version of the structure). I
> guess these are the workarounds we have to do.
> >>
> > No, there is never any need for two versions of the inline functions,
> > only the newest version is needed. This is because in the case of a
> > newly compiled application only the newest version of the non-inline
> > functions is ever used. The other older versions are only used at
> > runtime for compatilibity with pre-compiled apps with the older inlines.
> >
> >>>
> >>> 3. If instead we have the case, like in rte_ring, where the data
> >>> structures are allocated using functions, but they are used via
> >>> inlines in the app. In this case the creation functions (and any
> >>> other function using the structures) need to be versioned so that
> >>> the application gets the expected structure back from the creation.
> >> The new structure members have to be added such that the previous
> >> layout is not affected. Either add at the end or use the gaps that
> >> are left because of cache line alignment.
> >>
> > Not necessarily. There is nothing that requires the older and newer
> > versions of a function to use the same structure. We can rename the
> > original structure when versioning the old function, and then create a
> > new structure with the original name for later recompiled code using
> > the newest ABIs.
> >
> >>>
> >>> It might be useful to think of what other scenarios we have and what
> >>> ones are likely to be problematic, especially those that can't be
> >>> solved by having multiple function versions.
> >>>
> >>>> 2) Every function that is not in the direct datapath (fastpath?)
> >>>> should not be inline. Exceptions or things like rx/tx burst, ring
> >>>> enqueue/dequeue, and packet alloc/free - Stephen
> >>>
> >>> Agree with this. Anything not data path should not be inline. The
> >>> next
> >> Yes, very clear on this.
> >>
> >>> question then is for data path items how to determine whether they
> >>> need to be inline or not. In general my rule-of-thumb:
> >>> * anything dealing with bursts of packets should not be inline
> >>> * anything what works with single packet at a time should be inline
> >>>
> >>> The one exception to this rule is cases where we have to consider
> >>> "empty polling" as a likely use-case. For example,
> >>> rte_ring_dequeue_burst works with bursts of packets, but there is a
> >>> valid application use case where a thread could be polling a set of
> >>> rings where only a small number are actually busy. Right now,
> >>> polling an empty ring only costs a few cycles, meaning that it's
> >>> neglible to have a few polls of empty rings before you get to a busy
> >>> one. Having that function not-inline will cause that cost to jump
> >>> significantly, so could cause problems. (This leads to the interesting
> scenario where ring enqueue is safe to un-inline, while dequeue is not).
> >> A good way to think about it would be - ratio of amount of work done in
> the function to cost of function call.
> >>
> >
> > Yes, I would tend to agree in general.  The other thing is the
> > frequency of calls, and - as already stated - whether it takes a burst
> > of not.  Because even if it's a trivial function that takes only 10
> > cycles and we want to uninline it, the cost may double; but if it
> > takes a burst of packets and is only used once/twice per burst the
> > cost per packet should still only be a fraction of a cycle.
> >
> >>>
> >>>> 3) Plus synchronization routines: spin/rwlock/barrier, etc. I think
> >>>> rcu should be one of such exceptions - it is just another
> >>>> synchronization mechanism after all (just a bit more
> >>>> sophisticated). - Konstantin
> >>>>
> >>> In general I believe the synchronisation primitives should be inline.
> >>> However, it does come down to cost too - if a function takes 300
> >>> cycles, do we really care if it takes 305 or 310 instead to make it not
> inline?
> >>> Hopefully most synchronisation primitives are faster than this so
> >>> this situation should not occur.
> >>>
> >>>> 2) and 3) can be good guidelines to what functions/APIs can be
> >>>> inline. But,
> >>> I believe this guideline exists today too. Are there any thoughts to
> >>> change this?
> >>>>
> >>>> Coming to 1), I think DPDK cannot provide ABI compatibility unless
> >>>> all the
> >>> inline functions are converted to normal functions and symbol
> >>> versioning is done for those (not bothering about performance).
> >>>>
> >>> I disagree. I think even in the case of #1, we should be able to
> >>> manage some changes without breaking ABI.
> >> I completely agree with you on trying to keep the ABI break surface small
> and doing due diligence when one is required. However, I think claiming 100%
> ABI compatibility all the time (or as frequently as other projects claim) might
> be tough. IMO, we need to look beyond this to solve the end user problem.
> May be packaging multiple LTS with distros when DPDK could not avoid
> breaking ABI compatibility?
> >>
> > Having multiple LTS's per distro would be nice, but it's putting a lot
> > more work on the distro folks, I think.
> 
> I completely disagree with this approach.
> Anytime I have seen this done, most frequently with interpreted languages,
> think multiple versions of Java and Python concurrently on a system, it has
> always been a usability nightmare.
> 
> 
> >
> >>>
> >>>> In this context, does it make sense to say that we will maintain
> >>>> API compatibility rather than saying ABI compatibility? This will
> >>>> also send the right message to the end users.
> >>>>
> >>> I would value ABI compatibility much higher than API compatibility.
> >>> If someone is recompiling the application anyway, making a couple of
> >>> small changes (large rework is obviously a different issue) to the
> >>> code should not be a massive issue, I hope. On the other hand, ABI
> >>> compatibility is needed to allow seamless update from one version to
> >>> another, and it's that ABI compatiblity that allows distro's to pick up our
> latest and greatest versions.
> >>>
> >> I think it is also important to setup the right expectations to DPDK users.
> i.e. DPDK will police itself better to provide ABI compatibility but occasionally
> it might not be possible.
> >>
> > The trouble here is that a DPDK release, as a product is either
> > backward compatible or not. Either a user can take it as a drop-in
> > replacement for the previous version or not. Users do not want to have
> > to go through a checklist for each app to see if the app uses only
> > "compatible" APIs or not. Same for the distro folks, they are not
> > going to take some libs from a release but not others because the ABI is
> broken for some but not others.
> 
> Agreed, a DPDK release is either backwards compatible or it isn't, and to
> Bruce's point if it isn't, it maters less if this is for a big API change or a small
> API change - the failure of the API stability guarantee is the significant piece.
> 
> The reality is that most other system libraries provide strong guarantees ... to
> date we have provided very little.
It would be good to look libraries that provide inline functions and ABI compatibility to better understand the methods used.

> 
> If we start saying "Yes, but except when", the exception case very quickly
> becomes the general case and then we are back to the beginning again.
> 
> >
> > Regards,
> > /Bruce
> >

Re: [dpdk-dev] ABI and inline functions

[Bruce Richardson]

On Wed, Apr 24, 2019 at 05:08:46AM +0000, Honnappa Nagarahalli wrote:
> > > > On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli
> > wrote:
> > > > > Hello,
<snip> 
> > > >
> > > > 2. If the situation is as in #1, but the structures in question are
> > > > passed to non-inline DPDK functions. In this case, any changes to
> > > > the structures require those functions taking the structures to be
> > > > versioned for old and new structures
> > > I think this can get complicated on the inline function side. The application
> > and the DPDK library will end up having different inline functions. The
> > changed inline function needs to be aware of 2 structure formats or the inline
> > function needs to be duplicated (one for each version of the structure). I
> > guess these are the workarounds we have to do.
> > >
> > No, there is never any need for two versions of the inline functions, only the
> > newest version is needed. This is because in the case of a newly compiled
> > application only the newest version of the non-inline functions is ever used.
> > The other older versions are only used at runtime for compatilibity with pre-
> > compiled apps with the older inlines.
> > 
> Since the inline function is used in the application and the DPDK (say a library), we have 2 copies of the same inline function in the final binary (1 with the application and 2nd one behind DPDK non-inline functions). When the DPDK non-inline functions are versioned, the older version of the functions have to support the old structures for the old application to work with the new DPDK. i.e. both copies of the inline function have to be the same.
> 

Ok, if the inline function is used both in DPDK libs and apps, then indeed
we have a slightly more complex problem. I was mostly assuming the inlines
were for app use only, but you are right that we have internally used ones
also. However, I expect this only makes the problem a little harder rather
than impossible - we would need two versions of the inline function,
however, only one needs to be exposed publically.

Really, the main upshot is that we need to reduce use of inline functions
except where absolutely necessary, and then hide structures as much as
possible. I don't see anyone disagreeing with that goal. :-)

/Bruce

Re: [dpdk-dev] ABI and inline functions

[Bruce Richardson]

On Wed, Apr 24, 2019 at 05:08:46AM +0000, Honnappa Nagarahalli wrote:
> > > > On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli
> > wrote:
> > > > > Hello,
<snip> 
> > > >
> > > > 2. If the situation is as in #1, but the structures in question are
> > > > passed to non-inline DPDK functions. In this case, any changes to
> > > > the structures require those functions taking the structures to be
> > > > versioned for old and new structures
> > > I think this can get complicated on the inline function side. The application
> > and the DPDK library will end up having different inline functions. The
> > changed inline function needs to be aware of 2 structure formats or the inline
> > function needs to be duplicated (one for each version of the structure). I
> > guess these are the workarounds we have to do.
> > >
> > No, there is never any need for two versions of the inline functions, only the
> > newest version is needed. This is because in the case of a newly compiled
> > application only the newest version of the non-inline functions is ever used.
> > The other older versions are only used at runtime for compatilibity with pre-
> > compiled apps with the older inlines.
> > 
> Since the inline function is used in the application and the DPDK (say a library), we have 2 copies of the same inline function in the final binary (1 with the application and 2nd one behind DPDK non-inline functions). When the DPDK non-inline functions are versioned, the older version of the functions have to support the old structures for the old application to work with the new DPDK. i.e. both copies of the inline function have to be the same.
> 

Ok, if the inline function is used both in DPDK libs and apps, then indeed
we have a slightly more complex problem. I was mostly assuming the inlines
were for app use only, but you are right that we have internally used ones
also. However, I expect this only makes the problem a little harder rather
than impossible - we would need two versions of the inline function,
however, only one needs to be exposed publically.

Really, the main upshot is that we need to reduce use of inline functions
except where absolutely necessary, and then hide structures as much as
possible. I don't see anyone disagreeing with that goal. :-)

/Bruce

Re: [dpdk-dev] ABI and inline functions

[Burakov, Anatoly]

On 23-Apr-19 3:12 PM, Ray Kinsella wrote:
> 
> 
> On 18/04/2019 11:28, Bruce Richardson wrote:
>> On Thu, Apr 18, 2019 at 04:34:53AM +0000, Honnappa Nagarahalli wrote:
>>>>
>>>> On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli wrote:
>>>>> Hello,
>>>>> 	There was a conversation [1] in the context of RCU library. I thought
>>>>> it needs participation from broader audience. Summary for the context
>>>>> (please look at [1] for full details)
>>>>>
>>>>
>>>> Thanks for kicking off this discussion
>>>>
>>>>> 1) How do we provide ABI compatibility when the code base contains
>>>> inline functions? Unless we freeze development in these inline functions and
>>>> corresponding structures, it might not be possible to claim ABI compatibility.
>>>> Application has to be recompiled.
>>>>
>>>> I agree that in some cases the application "might" need to be recompiled,
>>>> but on the other hand I also think that there are many cases where ABI
>>>> function versioning can still be used to mitigate things. For example, if we
>>>> think of a couple of various scenarios:
>>>>
>>>> 1. If everything is inline and variables are allocated by app, e.g.
>>>> spinlock on stack, then there is no issue as everything is application
>>>> contained.
>>> If there is a bug fix which requires the structure to change, the application would need to recompile. I guess you are talking about a scenario when nothing changed in the inline function/variables.
>>>
>>
>> If the application wants the bugfix, then yes. However, if the app is
>> unaffected by the bug, then it should have the option of updating DPDK
>> without a recompile.
> 
> I would also imagine that should be an extremely rare case, that a
> bugfix would require a structure change ... perhaps for an alignment issues?

Multiprocess threading issues is one case i've had to do that more than 
once.

<snip>


> 
> The reality is that most other system libraries provide strong
> guarantees ... to date we have provided very little.
> 

To our credit, the libraries you're likely referring to aren't trying to 
reimplement the Linux kernel :) I don't think we do these API/ABI breaks 
just because we like doing them - DPDK is complex, and getting 
everything right the first time *and* allowing for future evolution is 
not a trivial undertaking.

To me, part of the problem is that DPDK is an "everything and the 
kitchen sink" kind of library where there is a bunch of drivers, a whole 
quasi-OS layer of dealing with hardware in a cross-platform manner, a 
separate memory management system, a bunch of libraries such as hash/lpm 
tables, plus there's QOS, IP Pipeline, flow stuff, etc. - normally, "a 
library" would concentrate on doing one thing well. DPDK, on the other 
hand, tries to do *everything* well. The sheer breadth of DPDK's scope 
is, i think, contributing to the breakages. If you keep 99% of your 
libraries stable between version, but there's a small ABI tweak in an 
LPM library, the entire DPDK stability gets invalidated.

Perhaps limiting DPDK's scope would help with this as well.

> 
>>
>> Regards,
>> /Bruce
>>
> 


-- 
Thanks,
Anatoly

Re: [dpdk-dev] ABI and inline functions

[Burakov, Anatoly]

On 23-Apr-19 3:12 PM, Ray Kinsella wrote:
> 
> 
> On 18/04/2019 11:28, Bruce Richardson wrote:
>> On Thu, Apr 18, 2019 at 04:34:53AM +0000, Honnappa Nagarahalli wrote:
>>>>
>>>> On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli wrote:
>>>>> Hello,
>>>>> 	There was a conversation [1] in the context of RCU library. I thought
>>>>> it needs participation from broader audience. Summary for the context
>>>>> (please look at [1] for full details)
>>>>>
>>>>
>>>> Thanks for kicking off this discussion
>>>>
>>>>> 1) How do we provide ABI compatibility when the code base contains
>>>> inline functions? Unless we freeze development in these inline functions and
>>>> corresponding structures, it might not be possible to claim ABI compatibility.
>>>> Application has to be recompiled.
>>>>
>>>> I agree that in some cases the application "might" need to be recompiled,
>>>> but on the other hand I also think that there are many cases where ABI
>>>> function versioning can still be used to mitigate things. For example, if we
>>>> think of a couple of various scenarios:
>>>>
>>>> 1. If everything is inline and variables are allocated by app, e.g.
>>>> spinlock on stack, then there is no issue as everything is application
>>>> contained.
>>> If there is a bug fix which requires the structure to change, the application would need to recompile. I guess you are talking about a scenario when nothing changed in the inline function/variables.
>>>
>>
>> If the application wants the bugfix, then yes. However, if the app is
>> unaffected by the bug, then it should have the option of updating DPDK
>> without a recompile.
> 
> I would also imagine that should be an extremely rare case, that a
> bugfix would require a structure change ... perhaps for an alignment issues?

Multiprocess threading issues is one case i've had to do that more than 
once.

<snip>


> 
> The reality is that most other system libraries provide strong
> guarantees ... to date we have provided very little.
> 

To our credit, the libraries you're likely referring to aren't trying to 
reimplement the Linux kernel :) I don't think we do these API/ABI breaks 
just because we like doing them - DPDK is complex, and getting 
everything right the first time *and* allowing for future evolution is 
not a trivial undertaking.

To me, part of the problem is that DPDK is an "everything and the 
kitchen sink" kind of library where there is a bunch of drivers, a whole 
quasi-OS layer of dealing with hardware in a cross-platform manner, a 
separate memory management system, a bunch of libraries such as hash/lpm 
tables, plus there's QOS, IP Pipeline, flow stuff, etc. - normally, "a 
library" would concentrate on doing one thing well. DPDK, on the other 
hand, tries to do *everything* well. The sheer breadth of DPDK's scope 
is, i think, contributing to the breakages. If you keep 99% of your 
libraries stable between version, but there's a small ABI tweak in an 
LPM library, the entire DPDK stability gets invalidated.

Perhaps limiting DPDK's scope would help with this as well.

> 
>>
>> Regards,
>> /Bruce
>>
> 


-- 
Thanks,
Anatoly

Re: [dpdk-dev] ABI and inline functions

[Ray Kinsella]

On 24/04/2019 12:08, Burakov, Anatoly wrote:
> On 23-Apr-19 3:12 PM, Ray Kinsella wrote:
>>
>>
>> On 18/04/2019 11:28, Bruce Richardson wrote:
>>> On Thu, Apr 18, 2019 at 04:34:53AM +0000, Honnappa Nagarahalli wrote:
>>>>>
>>>>> On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli wrote:
>>>>>> Hello,
>>>>>>     There was a conversation [1] in the context of RCU library. I
>>>>>> thought
>>>>>> it needs participation from broader audience. Summary for the context
>>>>>> (please look at [1] for full details)
>>>>>>
>>>>>
>>>>> Thanks for kicking off this discussion
>>>>>
>>>>>> 1) How do we provide ABI compatibility when the code base contains
>>>>> inline functions? Unless we freeze development in these inline
>>>>> functions and
>>>>> corresponding structures, it might not be possible to claim ABI
>>>>> compatibility.
>>>>> Application has to be recompiled.
>>>>>
>>>>> I agree that in some cases the application "might" need to be
>>>>> recompiled,
>>>>> but on the other hand I also think that there are many cases where ABI
>>>>> function versioning can still be used to mitigate things. For
>>>>> example, if we
>>>>> think of a couple of various scenarios:
>>>>>
>>>>> 1. If everything is inline and variables are allocated by app, e.g.
>>>>> spinlock on stack, then there is no issue as everything is application
>>>>> contained.
>>>> If there is a bug fix which requires the structure to change, the
>>>> application would need to recompile. I guess you are talking about a
>>>> scenario when nothing changed in the inline function/variables.
>>>>
>>>
>>> If the application wants the bugfix, then yes. However, if the app is
>>> unaffected by the bug, then it should have the option of updating DPDK
>>> without a recompile.
>>
>> I would also imagine that should be an extremely rare case, that a
>> bugfix would require a structure change ... perhaps for an alignment
>> issues?
> 
> Multiprocess threading issues is one case i've had to do that more than
> once.

Another reason to dislike multi process I guess.

> 
> <snip>
> 
> 
>>
>> The reality is that most other system libraries provide strong
>> guarantees ... to date we have provided very little.
>>
> 
> To our credit, the libraries you're likely referring to aren't trying to
> reimplement the Linux kernel :) I don't think we do these API/ABI breaks
> just because we like doing them - DPDK is complex, and getting
> everything right the first time *and* allowing for future evolution is
> not a trivial undertaking.
>
> 
> To me, part of the problem is that DPDK is an "everything and the
> kitchen sink" kind of library where there is a bunch of drivers, a whole
> quasi-OS layer of dealing with hardware in a cross-platform manner, a
> separate memory management system, a bunch of libraries such as hash/lpm
> tables, plus there's QOS, IP Pipeline, flow stuff, etc. - normally, "a
> library" would concentrate on doing one thing well. DPDK, on the other
> hand, tries to do *everything* well. The sheer breadth of DPDK's scope
> is, i think, contributing to the breakages. If you keep 99% of your
> libraries stable between version, but there's a small ABI tweak in an
> LPM library, the entire DPDK stability gets invalidated.

Well I guess DPDK is no more complex than Java or .NET Framework in that
respect, as these also feature OS-layers, memory management systems,
application frameworks, primitives etc but do manage to give their
consumers strong guarantee's on API stability. Clearly ABI stability has
a no meaning when you always being JIT compiled.

I understand that doing everything right the first time, allowing for
future evolution, while keep ABI/APIs relatively stable is a tough ask.

I would propose that API's marked as "experimental" be given greater
freedom to change to give time of API's to bake, so they don't need to
be always "right first time", that there is wriggle room for these.

I would also propose more use of ABI Versioning to enable evolution of
DPDK while preserving backward compatibility.

> 
> Perhaps limiting DPDK's scope would help with this as well.

I agree.

> 
>>
>>>
>>> Regards,
>>> /Bruce
>>>
>>
> 
> 

Re: [dpdk-dev] ABI and inline functions

[Ray Kinsella]

On 24/04/2019 12:08, Burakov, Anatoly wrote:
> On 23-Apr-19 3:12 PM, Ray Kinsella wrote:
>>
>>
>> On 18/04/2019 11:28, Bruce Richardson wrote:
>>> On Thu, Apr 18, 2019 at 04:34:53AM +0000, Honnappa Nagarahalli wrote:
>>>>>
>>>>> On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli wrote:
>>>>>> Hello,
>>>>>>     There was a conversation [1] in the context of RCU library. I
>>>>>> thought
>>>>>> it needs participation from broader audience. Summary for the context
>>>>>> (please look at [1] for full details)
>>>>>>
>>>>>
>>>>> Thanks for kicking off this discussion
>>>>>
>>>>>> 1) How do we provide ABI compatibility when the code base contains
>>>>> inline functions? Unless we freeze development in these inline
>>>>> functions and
>>>>> corresponding structures, it might not be possible to claim ABI
>>>>> compatibility.
>>>>> Application has to be recompiled.
>>>>>
>>>>> I agree that in some cases the application "might" need to be
>>>>> recompiled,
>>>>> but on the other hand I also think that there are many cases where ABI
>>>>> function versioning can still be used to mitigate things. For
>>>>> example, if we
>>>>> think of a couple of various scenarios:
>>>>>
>>>>> 1. If everything is inline and variables are allocated by app, e.g.
>>>>> spinlock on stack, then there is no issue as everything is application
>>>>> contained.
>>>> If there is a bug fix which requires the structure to change, the
>>>> application would need to recompile. I guess you are talking about a
>>>> scenario when nothing changed in the inline function/variables.
>>>>
>>>
>>> If the application wants the bugfix, then yes. However, if the app is
>>> unaffected by the bug, then it should have the option of updating DPDK
>>> without a recompile.
>>
>> I would also imagine that should be an extremely rare case, that a
>> bugfix would require a structure change ... perhaps for an alignment
>> issues?
> 
> Multiprocess threading issues is one case i've had to do that more than
> once.

Another reason to dislike multi process I guess.

> 
> <snip>
> 
> 
>>
>> The reality is that most other system libraries provide strong
>> guarantees ... to date we have provided very little.
>>
> 
> To our credit, the libraries you're likely referring to aren't trying to
> reimplement the Linux kernel :) I don't think we do these API/ABI breaks
> just because we like doing them - DPDK is complex, and getting
> everything right the first time *and* allowing for future evolution is
> not a trivial undertaking.
>
> 
> To me, part of the problem is that DPDK is an "everything and the
> kitchen sink" kind of library where there is a bunch of drivers, a whole
> quasi-OS layer of dealing with hardware in a cross-platform manner, a
> separate memory management system, a bunch of libraries such as hash/lpm
> tables, plus there's QOS, IP Pipeline, flow stuff, etc. - normally, "a
> library" would concentrate on doing one thing well. DPDK, on the other
> hand, tries to do *everything* well. The sheer breadth of DPDK's scope
> is, i think, contributing to the breakages. If you keep 99% of your
> libraries stable between version, but there's a small ABI tweak in an
> LPM library, the entire DPDK stability gets invalidated.

Well I guess DPDK is no more complex than Java or .NET Framework in that
respect, as these also feature OS-layers, memory management systems,
application frameworks, primitives etc but do manage to give their
consumers strong guarantee's on API stability. Clearly ABI stability has
a no meaning when you always being JIT compiled.

I understand that doing everything right the first time, allowing for
future evolution, while keep ABI/APIs relatively stable is a tough ask.

I would propose that API's marked as "experimental" be given greater
freedom to change to give time of API's to bake, so they don't need to
be always "right first time", that there is wriggle room for these.

I would also propose more use of ABI Versioning to enable evolution of
DPDK while preserving backward compatibility.

> 
> Perhaps limiting DPDK's scope would help with this as well.

I agree.

> 
>>
>>>
>>> Regards,
>>> /Bruce
>>>
>>
> 
> 

Re: [dpdk-dev] ABI and inline functions

[Burakov, Anatoly]

On 24-Apr-19 1:22 PM, Ray Kinsella wrote:
> 
> On 24/04/2019 12:08, Burakov, Anatoly wrote:
>> On 23-Apr-19 3:12 PM, Ray Kinsella wrote:
>>>
>>>
>>> On 18/04/2019 11:28, Bruce Richardson wrote:
>>>> On Thu, Apr 18, 2019 at 04:34:53AM +0000, Honnappa Nagarahalli wrote:
>>>>>>
>>>>>> On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli wrote:
>>>>>>> Hello,
>>>>>>>      There was a conversation [1] in the context of RCU library. I
>>>>>>> thought
>>>>>>> it needs participation from broader audience. Summary for the context
>>>>>>> (please look at [1] for full details)
>>>>>>>
>>>>>>
>>>>>> Thanks for kicking off this discussion
>>>>>>
>>>>>>> 1) How do we provide ABI compatibility when the code base contains
>>>>>> inline functions? Unless we freeze development in these inline
>>>>>> functions and
>>>>>> corresponding structures, it might not be possible to claim ABI
>>>>>> compatibility.
>>>>>> Application has to be recompiled.
>>>>>>
>>>>>> I agree that in some cases the application "might" need to be
>>>>>> recompiled,
>>>>>> but on the other hand I also think that there are many cases where ABI
>>>>>> function versioning can still be used to mitigate things. For
>>>>>> example, if we
>>>>>> think of a couple of various scenarios:
>>>>>>
>>>>>> 1. If everything is inline and variables are allocated by app, e.g.
>>>>>> spinlock on stack, then there is no issue as everything is application
>>>>>> contained.
>>>>> If there is a bug fix which requires the structure to change, the
>>>>> application would need to recompile. I guess you are talking about a
>>>>> scenario when nothing changed in the inline function/variables.
>>>>>
>>>>
>>>> If the application wants the bugfix, then yes. However, if the app is
>>>> unaffected by the bug, then it should have the option of updating DPDK
>>>> without a recompile.
>>>
>>> I would also imagine that should be an extremely rare case, that a
>>> bugfix would require a structure change ... perhaps for an alignment
>>> issues?
>>
>> Multiprocess threading issues is one case i've had to do that more than
>> once.
> 
> Another reason to dislike multi process I guess.
> 
>>
>> <snip>
>>
>>
>>>
>>> The reality is that most other system libraries provide strong
>>> guarantees ... to date we have provided very little.
>>>
>>
>> To our credit, the libraries you're likely referring to aren't trying to
>> reimplement the Linux kernel :) I don't think we do these API/ABI breaks
>> just because we like doing them - DPDK is complex, and getting
>> everything right the first time *and* allowing for future evolution is
>> not a trivial undertaking.
>>
>>
>> To me, part of the problem is that DPDK is an "everything and the
>> kitchen sink" kind of library where there is a bunch of drivers, a whole
>> quasi-OS layer of dealing with hardware in a cross-platform manner, a
>> separate memory management system, a bunch of libraries such as hash/lpm
>> tables, plus there's QOS, IP Pipeline, flow stuff, etc. - normally, "a
>> library" would concentrate on doing one thing well. DPDK, on the other
>> hand, tries to do *everything* well. The sheer breadth of DPDK's scope
>> is, i think, contributing to the breakages. If you keep 99% of your
>> libraries stable between version, but there's a small ABI tweak in an
>> LPM library, the entire DPDK stability gets invalidated.
> 
> Well I guess DPDK is no more complex than Java or .NET Framework in that
> respect, as these also feature OS-layers, memory management systems,
> application frameworks, primitives etc but do manage to give their
> consumers strong guarantee's on API stability. Clearly ABI stability has
> a no meaning when you always being JIT compiled.

I was basing my response on your earlier comparisons of DPDK to 
GStreamer :) Comparing it to .NET Framework is perhaps a better fit, but 
then these frameworks generally go through much more rigorous 
development/design cycle than DPDK does - DPDK's API design process is 
pretty ad-hoc, while both Java and .NET have various kinds of procedures 
by which things get into the standard library. If we're prepared to do 
that - i'm all for it. What we can't have is stability *and* keep the 
same approach to design/development that we have now.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] ABI and inline functions

[Burakov, Anatoly]

On 24-Apr-19 1:22 PM, Ray Kinsella wrote:
> 
> On 24/04/2019 12:08, Burakov, Anatoly wrote:
>> On 23-Apr-19 3:12 PM, Ray Kinsella wrote:
>>>
>>>
>>> On 18/04/2019 11:28, Bruce Richardson wrote:
>>>> On Thu, Apr 18, 2019 at 04:34:53AM +0000, Honnappa Nagarahalli wrote:
>>>>>>
>>>>>> On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli wrote:
>>>>>>> Hello,
>>>>>>>      There was a conversation [1] in the context of RCU library. I
>>>>>>> thought
>>>>>>> it needs participation from broader audience. Summary for the context
>>>>>>> (please look at [1] for full details)
>>>>>>>
>>>>>>
>>>>>> Thanks for kicking off this discussion
>>>>>>
>>>>>>> 1) How do we provide ABI compatibility when the code base contains
>>>>>> inline functions? Unless we freeze development in these inline
>>>>>> functions and
>>>>>> corresponding structures, it might not be possible to claim ABI
>>>>>> compatibility.
>>>>>> Application has to be recompiled.
>>>>>>
>>>>>> I agree that in some cases the application "might" need to be
>>>>>> recompiled,
>>>>>> but on the other hand I also think that there are many cases where ABI
>>>>>> function versioning can still be used to mitigate things. For
>>>>>> example, if we
>>>>>> think of a couple of various scenarios:
>>>>>>
>>>>>> 1. If everything is inline and variables are allocated by app, e.g.
>>>>>> spinlock on stack, then there is no issue as everything is application
>>>>>> contained.
>>>>> If there is a bug fix which requires the structure to change, the
>>>>> application would need to recompile. I guess you are talking about a
>>>>> scenario when nothing changed in the inline function/variables.
>>>>>
>>>>
>>>> If the application wants the bugfix, then yes. However, if the app is
>>>> unaffected by the bug, then it should have the option of updating DPDK
>>>> without a recompile.
>>>
>>> I would also imagine that should be an extremely rare case, that a
>>> bugfix would require a structure change ... perhaps for an alignment
>>> issues?
>>
>> Multiprocess threading issues is one case i've had to do that more than
>> once.
> 
> Another reason to dislike multi process I guess.
> 
>>
>> <snip>
>>
>>
>>>
>>> The reality is that most other system libraries provide strong
>>> guarantees ... to date we have provided very little.
>>>
>>
>> To our credit, the libraries you're likely referring to aren't trying to
>> reimplement the Linux kernel :) I don't think we do these API/ABI breaks
>> just because we like doing them - DPDK is complex, and getting
>> everything right the first time *and* allowing for future evolution is
>> not a trivial undertaking.
>>
>>
>> To me, part of the problem is that DPDK is an "everything and the
>> kitchen sink" kind of library where there is a bunch of drivers, a whole
>> quasi-OS layer of dealing with hardware in a cross-platform manner, a
>> separate memory management system, a bunch of libraries such as hash/lpm
>> tables, plus there's QOS, IP Pipeline, flow stuff, etc. - normally, "a
>> library" would concentrate on doing one thing well. DPDK, on the other
>> hand, tries to do *everything* well. The sheer breadth of DPDK's scope
>> is, i think, contributing to the breakages. If you keep 99% of your
>> libraries stable between version, but there's a small ABI tweak in an
>> LPM library, the entire DPDK stability gets invalidated.
> 
> Well I guess DPDK is no more complex than Java or .NET Framework in that
> respect, as these also feature OS-layers, memory management systems,
> application frameworks, primitives etc but do manage to give their
> consumers strong guarantee's on API stability. Clearly ABI stability has
> a no meaning when you always being JIT compiled.

I was basing my response on your earlier comparisons of DPDK to 
GStreamer :) Comparing it to .NET Framework is perhaps a better fit, but 
then these frameworks generally go through much more rigorous 
development/design cycle than DPDK does - DPDK's API design process is 
pretty ad-hoc, while both Java and .NET have various kinds of procedures 
by which things get into the standard library. If we're prepared to do 
that - i'm all for it. What we can't have is stability *and* keep the 
same approach to design/development that we have now.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] ABI and inline functions

[Stephen Hemminger]

On Wed, 24 Apr 2019 13:54:51 +0100
"Burakov, Anatoly" <anatoly.burakov@intel.com> wrote:

> On 24-Apr-19 1:22 PM, Ray Kinsella wrote:
> > 
> > On 24/04/2019 12:08, Burakov, Anatoly wrote:  
> >> On 23-Apr-19 3:12 PM, Ray Kinsella wrote:  
> >>>
> >>>
> >>> On 18/04/2019 11:28, Bruce Richardson wrote:  
> >>>> On Thu, Apr 18, 2019 at 04:34:53AM +0000, Honnappa Nagarahalli wrote:  
> >>>>>>
> >>>>>> On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli wrote:  
> >>>>>>> Hello,
> >>>>>>>      There was a conversation [1] in the context of RCU library. I
> >>>>>>> thought
> >>>>>>> it needs participation from broader audience. Summary for the context
> >>>>>>> (please look at [1] for full details)
> >>>>>>>  
> >>>>>>
> >>>>>> Thanks for kicking off this discussion
> >>>>>>  
> >>>>>>> 1) How do we provide ABI compatibility when the code base contains  
> >>>>>> inline functions? Unless we freeze development in these inline
> >>>>>> functions and
> >>>>>> corresponding structures, it might not be possible to claim ABI
> >>>>>> compatibility.
> >>>>>> Application has to be recompiled.
> >>>>>>
> >>>>>> I agree that in some cases the application "might" need to be
> >>>>>> recompiled,
> >>>>>> but on the other hand I also think that there are many cases where ABI
> >>>>>> function versioning can still be used to mitigate things. For
> >>>>>> example, if we
> >>>>>> think of a couple of various scenarios:
> >>>>>>
> >>>>>> 1. If everything is inline and variables are allocated by app, e.g.
> >>>>>> spinlock on stack, then there is no issue as everything is application
> >>>>>> contained.  
> >>>>> If there is a bug fix which requires the structure to change, the
> >>>>> application would need to recompile. I guess you are talking about a
> >>>>> scenario when nothing changed in the inline function/variables.
> >>>>>  
> >>>>
> >>>> If the application wants the bugfix, then yes. However, if the app is
> >>>> unaffected by the bug, then it should have the option of updating DPDK
> >>>> without a recompile.  
> >>>
> >>> I would also imagine that should be an extremely rare case, that a
> >>> bugfix would require a structure change ... perhaps for an alignment
> >>> issues?  
> >>
> >> Multiprocess threading issues is one case i've had to do that more than
> >> once.  
> > 
> > Another reason to dislike multi process I guess.
> >   
> >>
> >> <snip>
> >>
> >>  
> >>>
> >>> The reality is that most other system libraries provide strong
> >>> guarantees ... to date we have provided very little.
> >>>  
> >>
> >> To our credit, the libraries you're likely referring to aren't trying to
> >> reimplement the Linux kernel :) I don't think we do these API/ABI breaks
> >> just because we like doing them - DPDK is complex, and getting
> >> everything right the first time *and* allowing for future evolution is
> >> not a trivial undertaking.
> >>
> >>
> >> To me, part of the problem is that DPDK is an "everything and the
> >> kitchen sink" kind of library where there is a bunch of drivers, a whole
> >> quasi-OS layer of dealing with hardware in a cross-platform manner, a
> >> separate memory management system, a bunch of libraries such as hash/lpm
> >> tables, plus there's QOS, IP Pipeline, flow stuff, etc. - normally, "a
> >> library" would concentrate on doing one thing well. DPDK, on the other
> >> hand, tries to do *everything* well. The sheer breadth of DPDK's scope
> >> is, i think, contributing to the breakages. If you keep 99% of your
> >> libraries stable between version, but there's a small ABI tweak in an
> >> LPM library, the entire DPDK stability gets invalidated.  
> > 
> > Well I guess DPDK is no more complex than Java or .NET Framework in that
> > respect, as these also feature OS-layers, memory management systems,
> > application frameworks, primitives etc but do manage to give their
> > consumers strong guarantee's on API stability. Clearly ABI stability has
> > a no meaning when you always being JIT compiled.  
> 
> I was basing my response on your earlier comparisons of DPDK to 
> GStreamer :) Comparing it to .NET Framework is perhaps a better fit, but 
> then these frameworks generally go through much more rigorous 
> development/design cycle than DPDK does - DPDK's API design process is 
> pretty ad-hoc, while both Java and .NET have various kinds of procedures 
> by which things get into the standard library. If we're prepared to do 
> that - i'm all for it. What we can't have is stability *and* keep the 
> same approach to design/development that we have now.
> 

Lets step back a bit. I think some of the basic business rules which
go back to the "Innovators dilemma" are that it is NOT a good idea
to pay too much attention to existing customers. Instead, it is important
to pay attention to the need of the potential customers that could
use your product.

In this case, my argument is that it is important for the DPDK project
to pay more attention to the reasons that new users
do not want to use the project; and existing users do not want to
upgrade.  This problem has been repeatedly expressed as:
"DPDK is not stable". The project needs to address that concern
even if it means a 1% drop in performance. Surely that 1% can be
made up by improving other places.  Even using link-time-optimization
can give 2 to 5 % back.

Re: [dpdk-dev] ABI and inline functions

[Stephen Hemminger]

On Wed, 24 Apr 2019 13:54:51 +0100
"Burakov, Anatoly" <anatoly.burakov@intel.com> wrote:

> On 24-Apr-19 1:22 PM, Ray Kinsella wrote:
> > 
> > On 24/04/2019 12:08, Burakov, Anatoly wrote:  
> >> On 23-Apr-19 3:12 PM, Ray Kinsella wrote:  
> >>>
> >>>
> >>> On 18/04/2019 11:28, Bruce Richardson wrote:  
> >>>> On Thu, Apr 18, 2019 at 04:34:53AM +0000, Honnappa Nagarahalli wrote:  
> >>>>>>
> >>>>>> On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli wrote:  
> >>>>>>> Hello,
> >>>>>>>      There was a conversation [1] in the context of RCU library. I
> >>>>>>> thought
> >>>>>>> it needs participation from broader audience. Summary for the context
> >>>>>>> (please look at [1] for full details)
> >>>>>>>  
> >>>>>>
> >>>>>> Thanks for kicking off this discussion
> >>>>>>  
> >>>>>>> 1) How do we provide ABI compatibility when the code base contains  
> >>>>>> inline functions? Unless we freeze development in these inline
> >>>>>> functions and
> >>>>>> corresponding structures, it might not be possible to claim ABI
> >>>>>> compatibility.
> >>>>>> Application has to be recompiled.
> >>>>>>
> >>>>>> I agree that in some cases the application "might" need to be
> >>>>>> recompiled,
> >>>>>> but on the other hand I also think that there are many cases where ABI
> >>>>>> function versioning can still be used to mitigate things. For
> >>>>>> example, if we
> >>>>>> think of a couple of various scenarios:
> >>>>>>
> >>>>>> 1. If everything is inline and variables are allocated by app, e.g.
> >>>>>> spinlock on stack, then there is no issue as everything is application
> >>>>>> contained.  
> >>>>> If there is a bug fix which requires the structure to change, the
> >>>>> application would need to recompile. I guess you are talking about a
> >>>>> scenario when nothing changed in the inline function/variables.
> >>>>>  
> >>>>
> >>>> If the application wants the bugfix, then yes. However, if the app is
> >>>> unaffected by the bug, then it should have the option of updating DPDK
> >>>> without a recompile.  
> >>>
> >>> I would also imagine that should be an extremely rare case, that a
> >>> bugfix would require a structure change ... perhaps for an alignment
> >>> issues?  
> >>
> >> Multiprocess threading issues is one case i've had to do that more than
> >> once.  
> > 
> > Another reason to dislike multi process I guess.
> >   
> >>
> >> <snip>
> >>
> >>  
> >>>
> >>> The reality is that most other system libraries provide strong
> >>> guarantees ... to date we have provided very little.
> >>>  
> >>
> >> To our credit, the libraries you're likely referring to aren't trying to
> >> reimplement the Linux kernel :) I don't think we do these API/ABI breaks
> >> just because we like doing them - DPDK is complex, and getting
> >> everything right the first time *and* allowing for future evolution is
> >> not a trivial undertaking.
> >>
> >>
> >> To me, part of the problem is that DPDK is an "everything and the
> >> kitchen sink" kind of library where there is a bunch of drivers, a whole
> >> quasi-OS layer of dealing with hardware in a cross-platform manner, a
> >> separate memory management system, a bunch of libraries such as hash/lpm
> >> tables, plus there's QOS, IP Pipeline, flow stuff, etc. - normally, "a
> >> library" would concentrate on doing one thing well. DPDK, on the other
> >> hand, tries to do *everything* well. The sheer breadth of DPDK's scope
> >> is, i think, contributing to the breakages. If you keep 99% of your
> >> libraries stable between version, but there's a small ABI tweak in an
> >> LPM library, the entire DPDK stability gets invalidated.  
> > 
> > Well I guess DPDK is no more complex than Java or .NET Framework in that
> > respect, as these also feature OS-layers, memory management systems,
> > application frameworks, primitives etc but do manage to give their
> > consumers strong guarantee's on API stability. Clearly ABI stability has
> > a no meaning when you always being JIT compiled.  
> 
> I was basing my response on your earlier comparisons of DPDK to 
> GStreamer :) Comparing it to .NET Framework is perhaps a better fit, but 
> then these frameworks generally go through much more rigorous 
> development/design cycle than DPDK does - DPDK's API design process is 
> pretty ad-hoc, while both Java and .NET have various kinds of procedures 
> by which things get into the standard library. If we're prepared to do 
> that - i'm all for it. What we can't have is stability *and* keep the 
> same approach to design/development that we have now.
> 

Lets step back a bit. I think some of the basic business rules which
go back to the "Innovators dilemma" are that it is NOT a good idea
to pay too much attention to existing customers. Instead, it is important
to pay attention to the need of the potential customers that could
use your product.

In this case, my argument is that it is important for the DPDK project
to pay more attention to the reasons that new users
do not want to use the project; and existing users do not want to
upgrade.  This problem has been repeatedly expressed as:
"DPDK is not stable". The project needs to address that concern
even if it means a 1% drop in performance. Surely that 1% can be
made up by improving other places.  Even using link-time-optimization
can give 2 to 5 % back.

Re: [dpdk-dev] [EXT] Re: ABI and inline functions

[Jerin Jacob Kollanukkaran]

> -----Original Message-----
> From: Ray Kinsella <mdr@ashroe.eu>
> Sent: Tuesday, April 23, 2019 7:54 PM
> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Stephen Hemminger
> <stephen@networkplumber.org>
> Cc: Bruce Richardson <bruce.richardson@intel.com>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; dev@dpdk.org; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; thomas@monjalon.net; nd <nd@arm.com>
> Subject: Re: [EXT] Re: [dpdk-dev] ABI and inline functions
> 
> 
> 
> On 18/04/2019 06:56, Jerin Jacob Kollanukkaran wrote:
> >
> >> -----Original Message-----
> >> From: Stephen Hemminger <stephen@networkplumber.org>
> >> Sent: Thursday, April 18, 2019 12:21 AM
> >> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> >> Cc: Bruce Richardson <bruce.richardson@intel.com>; Honnappa
> >> Nagarahalli <Honnappa.Nagarahalli@arm.com>; dev@dpdk.org; Ananyev,
> >> Konstantin <konstantin.ananyev@intel.com>; thomas@monjalon.net; Ray
> >> Kinsella <mdr@ashroe.eu>; nd <nd@arm.com>
> >> Subject: [EXT] Re: [dpdk-dev] ABI and inline functions
> >>> I would value ABI compatibility much higher than API compatibility.
> >>>> If someone is recompiling the application anyway, making a couple
> >>>> of small changes (large rework is obviously a different issue) to
> >>>> the code should not be a massive issue, I hope. On the other hand,
> >>>> ABI compatibility is needed to allow seamless update from one
> >>>> version to another, and it's that ABI compatiblity that allows
> >>>> distro's to pick up our
> >> latest and greatest versions.
> >>>
> >>> IMO, We have two primary use case for DPDK
> >>>
> >>> 1) NFV kind of use case 	where the application needs to run on multiple
> >> platform
> >>> without recompiling it.
> >>> 2) Fixed appliance use case where embed SoC like Intel Denverton or
> >>> ARM64 integrated Controller used. For fixed appliance use case, end
> >>> user care more of performance than ABI compatibility as it easy to
> >>> recompile the end user application vs the cost of hitting performance
> impact.
> >>
> >> Nobody cares about compatiablity until they have to the first security update.
> >
> > For fixed appliance case, The update(FW update) will be  a single blob
> > which Include all the components. So they can back port the security
> > fix and recompile the sw as needed.
> >
> > The very similar category  is DPDK running in smart NICs(Runs as FW in PCIe EP
> device).
> 
> So is there a real versus a perceived compromise happen here - that we are
> compromising optimal performance in order to make API stability happen? Do
> we have specific an examples that this is actually the case?

The arm64 integrated SoC would functions as PCIE EP mode as well.
That would means that you will have x86/arm64 server machine and 
Integrated NIC controller SoC connected to server over PCIE. The usecase for
this to have packet processing offload from server to EP device to save server cycles.
Where EP device will be treated as FW and internally it runs DPDK as HW
Abstraction library for FW.


> 
> Thanks,
> 
> Ray K
> 

Re: [dpdk-dev] [EXT] Re: ABI and inline functions

[Jerin Jacob Kollanukkaran]

> -----Original Message-----
> From: Ray Kinsella <mdr@ashroe.eu>
> Sent: Tuesday, April 23, 2019 7:54 PM
> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Stephen Hemminger
> <stephen@networkplumber.org>
> Cc: Bruce Richardson <bruce.richardson@intel.com>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; dev@dpdk.org; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; thomas@monjalon.net; nd <nd@arm.com>
> Subject: Re: [EXT] Re: [dpdk-dev] ABI and inline functions
> 
> 
> 
> On 18/04/2019 06:56, Jerin Jacob Kollanukkaran wrote:
> >
> >> -----Original Message-----
> >> From: Stephen Hemminger <stephen@networkplumber.org>
> >> Sent: Thursday, April 18, 2019 12:21 AM
> >> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> >> Cc: Bruce Richardson <bruce.richardson@intel.com>; Honnappa
> >> Nagarahalli <Honnappa.Nagarahalli@arm.com>; dev@dpdk.org; Ananyev,
> >> Konstantin <konstantin.ananyev@intel.com>; thomas@monjalon.net; Ray
> >> Kinsella <mdr@ashroe.eu>; nd <nd@arm.com>
> >> Subject: [EXT] Re: [dpdk-dev] ABI and inline functions
> >>> I would value ABI compatibility much higher than API compatibility.
> >>>> If someone is recompiling the application anyway, making a couple
> >>>> of small changes (large rework is obviously a different issue) to
> >>>> the code should not be a massive issue, I hope. On the other hand,
> >>>> ABI compatibility is needed to allow seamless update from one
> >>>> version to another, and it's that ABI compatiblity that allows
> >>>> distro's to pick up our
> >> latest and greatest versions.
> >>>
> >>> IMO, We have two primary use case for DPDK
> >>>
> >>> 1) NFV kind of use case 	where the application needs to run on multiple
> >> platform
> >>> without recompiling it.
> >>> 2) Fixed appliance use case where embed SoC like Intel Denverton or
> >>> ARM64 integrated Controller used. For fixed appliance use case, end
> >>> user care more of performance than ABI compatibility as it easy to
> >>> recompile the end user application vs the cost of hitting performance
> impact.
> >>
> >> Nobody cares about compatiablity until they have to the first security update.
> >
> > For fixed appliance case, The update(FW update) will be  a single blob
> > which Include all the components. So they can back port the security
> > fix and recompile the sw as needed.
> >
> > The very similar category  is DPDK running in smart NICs(Runs as FW in PCIe EP
> device).
> 
> So is there a real versus a perceived compromise happen here - that we are
> compromising optimal performance in order to make API stability happen? Do
> we have specific an examples that this is actually the case?

The arm64 integrated SoC would functions as PCIE EP mode as well.
That would means that you will have x86/arm64 server machine and 
Integrated NIC controller SoC connected to server over PCIE. The usecase for
this to have packet processing offload from server to EP device to save server cycles.
Where EP device will be treated as FW and internally it runs DPDK as HW
Abstraction library for FW.


> 
> Thanks,
> 
> Ray K
> 

Re: [dpdk-dev] ABI and inline functions

[Thomas Monjalon]

24/04/2019 14:22, Ray Kinsella:
> On 24/04/2019 12:08, Burakov, Anatoly wrote:
> > To me, part of the problem is that DPDK is an "everything and the
> > kitchen sink" kind of library where there is a bunch of drivers, a whole
> > quasi-OS layer of dealing with hardware in a cross-platform manner, a
> > separate memory management system, a bunch of libraries such as hash/lpm
> > tables, plus there's QOS, IP Pipeline, flow stuff, etc. - normally, "a
> > library" would concentrate on doing one thing well. DPDK, on the other
> > hand, tries to do *everything* well. The sheer breadth of DPDK's scope
> > is, i think, contributing to the breakages. If you keep 99% of your
> > libraries stable between version, but there's a small ABI tweak in an
> > LPM library, the entire DPDK stability gets invalidated.

Yes, that's why we keep the split in libraries.
So we can update LPM library version while keeping the same ethdev
version to allow applications to load the shared library of the same
name without any re-compilation. In this case, the need for
re-compilation is only for applications using LPM.
But this model is not convenient for distributions because they
manage DPDK as one package. It means they have to re-compile all
applications if one LPM ABI is broken, even if it is experimental.
Should the libraries be split in packages?

> Well I guess DPDK is no more complex than Java or .NET Framework in that
> respect, as these also feature OS-layers, memory management systems,
> application frameworks, primitives etc but do manage to give their
> consumers strong guarantee's on API stability. Clearly ABI stability has
> a no meaning when you always being JIT compiled.
> 
> I understand that doing everything right the first time, allowing for
> future evolution, while keep ABI/APIs relatively stable is a tough ask.
> 
> I would propose that API's marked as "experimental" be given greater
> freedom to change to give time of API's to bake, so they don't need to
> be always "right first time", that there is wriggle room for these.

As said above, experimental level does not solve ABI stability.
If we break an experimental API, the ABI is broken, requiring
to re-compile the application.

> I would also propose more use of ABI Versioning to enable evolution of
> DPDK while preserving backward compatibility.

Yes, if all other obvious versioning issues are solved,
we can walk the extra mile of better using ABI versioning.

> > Perhaps limiting DPDK's scope would help with this as well.
> 
> I agree.

Yes, either split the project in more limited scope areas,
or split the packages.

Re: [dpdk-dev] ABI and inline functions

[Thomas Monjalon]

24/04/2019 14:22, Ray Kinsella:
> On 24/04/2019 12:08, Burakov, Anatoly wrote:
> > To me, part of the problem is that DPDK is an "everything and the
> > kitchen sink" kind of library where there is a bunch of drivers, a whole
> > quasi-OS layer of dealing with hardware in a cross-platform manner, a
> > separate memory management system, a bunch of libraries such as hash/lpm
> > tables, plus there's QOS, IP Pipeline, flow stuff, etc. - normally, "a
> > library" would concentrate on doing one thing well. DPDK, on the other
> > hand, tries to do *everything* well. The sheer breadth of DPDK's scope
> > is, i think, contributing to the breakages. If you keep 99% of your
> > libraries stable between version, but there's a small ABI tweak in an
> > LPM library, the entire DPDK stability gets invalidated.

Yes, that's why we keep the split in libraries.
So we can update LPM library version while keeping the same ethdev
version to allow applications to load the shared library of the same
name without any re-compilation. In this case, the need for
re-compilation is only for applications using LPM.
But this model is not convenient for distributions because they
manage DPDK as one package. It means they have to re-compile all
applications if one LPM ABI is broken, even if it is experimental.
Should the libraries be split in packages?

> Well I guess DPDK is no more complex than Java or .NET Framework in that
> respect, as these also feature OS-layers, memory management systems,
> application frameworks, primitives etc but do manage to give their
> consumers strong guarantee's on API stability. Clearly ABI stability has
> a no meaning when you always being JIT compiled.
> 
> I understand that doing everything right the first time, allowing for
> future evolution, while keep ABI/APIs relatively stable is a tough ask.
> 
> I would propose that API's marked as "experimental" be given greater
> freedom to change to give time of API's to bake, so they don't need to
> be always "right first time", that there is wriggle room for these.

As said above, experimental level does not solve ABI stability.
If we break an experimental API, the ABI is broken, requiring
to re-compile the application.

> I would also propose more use of ABI Versioning to enable evolution of
> DPDK while preserving backward compatibility.

Yes, if all other obvious versioning issues are solved,
we can walk the extra mile of better using ABI versioning.

> > Perhaps limiting DPDK's scope would help with this as well.
> 
> I agree.

Yes, either split the project in more limited scope areas,
or split the packages.