From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id F15231B5B8 for ; Wed, 17 Apr 2019 10:36:42 +0200 (CEST) X-Amp-Result: UNKNOWN X-Amp-Original-Verdict: FILE UNKNOWN X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Apr 2019 01:36:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,361,1549958400"; d="scan'208";a="338371893" Received: from bricha3-mobl.ger.corp.intel.com ([10.237.220.103]) by fmsmga005.fm.intel.com with SMTP; 17 Apr 2019 01:36:39 -0700 Received: by (sSMTP sendmail emulation); Wed, 17 Apr 2019 09:36:38 +0100 Date: Wed, 17 Apr 2019 09:36:38 +0100 From: Bruce Richardson To: Honnappa Nagarahalli Cc: "dev@dpdk.org" , Stephen Hemminger , "Ananyev, Konstantin" , "thomas@monjalon.net" , Ray Kinsella , nd Message-ID: <20190417083637.GB1890@bricha3-MOBL.ger.corp.intel.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.11.4 (2019-03-13) Subject: Re: [dpdk-dev] ABI and inline functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2019 08:36:43 -0000 On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli wrote: > Hello, > There was a conversation [1] in the context of RCU library. I thought it needs participation from broader audience. Summary for the context (please look at [1] for full details) > Thanks for kicking off this discussion > 1) How do we provide ABI compatibility when the code base contains inline functions? Unless we freeze development in these inline functions and corresponding structures, it might not be possible to claim ABI compatibility. Application has to be recompiled. I agree that in some cases the application "might" need to be recompiled, but on the other hand I also think that there are many cases where ABI function versioning can still be used to mitigate things. For example, if we think of a couple of various scenarios: 1. If everything is inline and variables are allocated by app, e.g. spinlock on stack, then there is no issue as everything is application contained. 2. If the situation is as in #1, but the structures in question are passed to non-inline DPDK functions. In this case, any changes to the structures require those functions taking the structures to be versioned for old and new structures 3. If instead we have the case, like in rte_ring, where the data structures are allocated using functions, but they are used via inlines in the app. In this case the creation functions (and any other function using the structures) need to be versioned so that the application gets the expected structure back from the creation. It might be useful to think of what other scenarios we have and what ones are likely to be problematic, especially those that can't be solved by having multiple function versions. > 2) Every function that is not in the direct datapath (fastpath?) should not be inline. Exceptions or things like rx/tx burst, ring enqueue/dequeue, and packet alloc/free - Stephen Agree with this. Anything not data path should not be inline. The next question then is for data path items how to determine whether they need to be inline or not. In general my rule-of-thumb: * anything dealing with bursts of packets should not be inline * anything what works with single packet at a time should be inline The one exception to this rule is cases where we have to consider "empty polling" as a likely use-case. For example, rte_ring_dequeue_burst works with bursts of packets, but there is a valid application use case where a thread could be polling a set of rings where only a small number are actually busy. Right now, polling an empty ring only costs a few cycles, meaning that it's neglible to have a few polls of empty rings before you get to a busy one. Having that function not-inline will cause that cost to jump significantly, so could cause problems. (This leads to the interesting scenario where ring enqueue is safe to un-inline, while dequeue is not). > 3) Plus synchronization routines: spin/rwlock/barrier, etc. I think rcu should be one of such exceptions - it is just another synchronization mechanism after all (just a bit more sophisticated). - Konstantin > In general I believe the synchronisation primitives should be inline. However, it does come down to cost too - if a function takes 300 cycles, do we really care if it takes 305 or 310 instead to make it not inline? Hopefully most synchronisation primitives are faster than this so this situation should not occur. > 2) and 3) can be good guidelines to what functions/APIs can be inline. But, I believe this guideline exists today too. Are there any thoughts to change this? > > Coming to 1), I think DPDK cannot provide ABI compatibility unless all the inline functions are converted to normal functions and symbol versioning is done for those (not bothering about performance). > I disagree. I think even in the case of #1, we should be able to manage some changes without breaking ABI. > In this context, does it make sense to say that we will maintain API > compatibility rather than saying ABI compatibility? This will also send > the right message to the end users. > I would value ABI compatibility much higher than API compatibility. If someone is recompiling the application anyway, making a couple of small changes (large rework is obviously a different issue) to the code should not be a massive issue, I hope. On the other hand, ABI compatibility is needed to allow seamless update from one version to another, and it's that ABI compatiblity that allows distro's to pick up our latest and greatest versions. Personally, I'd be happy enough to allow API changes at any point without deprecation notice, so long as function versioning is used to ensure ABI compatibility is kept. My 2c. /Bruce From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id 49325A00E6 for ; Wed, 17 Apr 2019 10:36:44 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0FBF31B5BB; Wed, 17 Apr 2019 10:36:44 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id F15231B5B8 for ; Wed, 17 Apr 2019 10:36:42 +0200 (CEST) X-Amp-Result: UNKNOWN X-Amp-Original-Verdict: FILE UNKNOWN X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Apr 2019 01:36:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,361,1549958400"; d="scan'208";a="338371893" Received: from bricha3-mobl.ger.corp.intel.com ([10.237.220.103]) by fmsmga005.fm.intel.com with SMTP; 17 Apr 2019 01:36:39 -0700 Received: by (sSMTP sendmail emulation); Wed, 17 Apr 2019 09:36:38 +0100 Date: Wed, 17 Apr 2019 09:36:38 +0100 From: Bruce Richardson To: Honnappa Nagarahalli Cc: "dev@dpdk.org" , Stephen Hemminger , "Ananyev, Konstantin" , "thomas@monjalon.net" , Ray Kinsella , nd Message-ID: <20190417083637.GB1890@bricha3-MOBL.ger.corp.intel.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.11.4 (2019-03-13) Subject: Re: [dpdk-dev] ABI and inline functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190417083638.2wA5f0KHnsxMOZ4EOByqca9c8XGI19dKhI4uzHmiVaw@z> On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli wrote: > Hello, > There was a conversation [1] in the context of RCU library. I thought it needs participation from broader audience. Summary for the context (please look at [1] for full details) > Thanks for kicking off this discussion > 1) How do we provide ABI compatibility when the code base contains inline functions? Unless we freeze development in these inline functions and corresponding structures, it might not be possible to claim ABI compatibility. Application has to be recompiled. I agree that in some cases the application "might" need to be recompiled, but on the other hand I also think that there are many cases where ABI function versioning can still be used to mitigate things. For example, if we think of a couple of various scenarios: 1. If everything is inline and variables are allocated by app, e.g. spinlock on stack, then there is no issue as everything is application contained. 2. If the situation is as in #1, but the structures in question are passed to non-inline DPDK functions. In this case, any changes to the structures require those functions taking the structures to be versioned for old and new structures 3. If instead we have the case, like in rte_ring, where the data structures are allocated using functions, but they are used via inlines in the app. In this case the creation functions (and any other function using the structures) need to be versioned so that the application gets the expected structure back from the creation. It might be useful to think of what other scenarios we have and what ones are likely to be problematic, especially those that can't be solved by having multiple function versions. > 2) Every function that is not in the direct datapath (fastpath?) should not be inline. Exceptions or things like rx/tx burst, ring enqueue/dequeue, and packet alloc/free - Stephen Agree with this. Anything not data path should not be inline. The next question then is for data path items how to determine whether they need to be inline or not. In general my rule-of-thumb: * anything dealing with bursts of packets should not be inline * anything what works with single packet at a time should be inline The one exception to this rule is cases where we have to consider "empty polling" as a likely use-case. For example, rte_ring_dequeue_burst works with bursts of packets, but there is a valid application use case where a thread could be polling a set of rings where only a small number are actually busy. Right now, polling an empty ring only costs a few cycles, meaning that it's neglible to have a few polls of empty rings before you get to a busy one. Having that function not-inline will cause that cost to jump significantly, so could cause problems. (This leads to the interesting scenario where ring enqueue is safe to un-inline, while dequeue is not). > 3) Plus synchronization routines: spin/rwlock/barrier, etc. I think rcu should be one of such exceptions - it is just another synchronization mechanism after all (just a bit more sophisticated). - Konstantin > In general I believe the synchronisation primitives should be inline. However, it does come down to cost too - if a function takes 300 cycles, do we really care if it takes 305 or 310 instead to make it not inline? Hopefully most synchronisation primitives are faster than this so this situation should not occur. > 2) and 3) can be good guidelines to what functions/APIs can be inline. But, I believe this guideline exists today too. Are there any thoughts to change this? > > Coming to 1), I think DPDK cannot provide ABI compatibility unless all the inline functions are converted to normal functions and symbol versioning is done for those (not bothering about performance). > I disagree. I think even in the case of #1, we should be able to manage some changes without breaking ABI. > In this context, does it make sense to say that we will maintain API > compatibility rather than saying ABI compatibility? This will also send > the right message to the end users. > I would value ABI compatibility much higher than API compatibility. If someone is recompiling the application anyway, making a couple of small changes (large rework is obviously a different issue) to the code should not be a massive issue, I hope. On the other hand, ABI compatibility is needed to allow seamless update from one version to another, and it's that ABI compatiblity that allows distro's to pick up our latest and greatest versions. Personally, I'd be happy enough to allow API changes at any point without deprecation notice, so long as function versioning is used to ensure ABI compatibility is kept. My 2c. /Bruce