From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id D0D1CA00E6 for ; Thu, 18 Apr 2019 06:34:58 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 968351B91C; Thu, 18 Apr 2019 06:34:57 +0200 (CEST) Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-eopbgr70070.outbound.protection.outlook.com [40.107.7.70]) by dpdk.org (Postfix) with ESMTP id 0AAA41B90B for ; Thu, 18 Apr 2019 06:34:56 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector1-arm-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XIsa29oJuSi01kUWUFNTuWj60xZNNcr+VBtkvljClOA=; b=UVzxCaXkfhgJqSWWcz5WWPJlnsr+5qE/HJyg0n0TWzDvVGlQ8fwz2fznsmiUhg/QTvLWTOs1nIYyopdoqT3h2YAZnZs00RHnMH+Sz/4P/N6fieOTFzbO6b0xHadXcwiOHIwozj/ycguROtgKOPnUH/kgYMiTij2OKL7WO8Ble/E= Received: from VE1PR08MB5149.eurprd08.prod.outlook.com (20.179.30.152) by VE1PR08MB4701.eurprd08.prod.outlook.com (10.255.114.143) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1792.19; Thu, 18 Apr 2019 04:34:53 +0000 Received: from VE1PR08MB5149.eurprd08.prod.outlook.com ([fe80::e0ae:ecad:ec5:8177]) by VE1PR08MB5149.eurprd08.prod.outlook.com ([fe80::e0ae:ecad:ec5:8177%2]) with mapi id 15.20.1792.018; Thu, 18 Apr 2019 04:34:53 +0000 From: Honnappa Nagarahalli To: Bruce Richardson CC: "dev@dpdk.org" , Stephen Hemminger , "Ananyev, Konstantin" , "thomas@monjalon.net" , Ray Kinsella , Honnappa Nagarahalli , nd , nd Thread-Topic: [dpdk-dev] ABI and inline functions Thread-Index: AdT03C2s3ZB+VIuxS0mLCJ+6mgHcvgAHH6YAABZxZtA= Date: Thu, 18 Apr 2019 04:34:53 +0000 Message-ID: References: <20190417083637.GB1890@bricha3-MOBL.ger.corp.intel.com> In-Reply-To: <20190417083637.GB1890@bricha3-MOBL.ger.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Honnappa.Nagarahalli@arm.com; x-originating-ip: [217.140.111.135] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 3bf59689-e2c0-4cb7-46bb-08d6c3b733f1 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(5600141)(711020)(4605104)(4618075)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(2017052603328)(7193020); SRVR:VE1PR08MB4701; x-ms-traffictypediagnostic: VE1PR08MB4701: x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr nodisclaimer: True x-microsoft-antispam-prvs: x-forefront-prvs: 0011612A55 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(396003)(346002)(39860400002)(376002)(136003)(366004)(189003)(52314003)(199004)(6916009)(3846002)(6436002)(6116002)(2906002)(72206003)(97736004)(229853002)(478600001)(4326008)(76176011)(8936002)(8676002)(316002)(71200400001)(26005)(6506007)(81156014)(81166006)(54906003)(7696005)(25786009)(68736007)(52536014)(486006)(476003)(53936002)(99286004)(102836004)(71190400001)(33656002)(86362001)(74316002)(11346002)(6246003)(66066001)(7736002)(256004)(186003)(55016002)(305945005)(5660300002)(14444005)(14454004)(9686003)(446003); DIR:OUT; SFP:1101; SCL:1; SRVR:VE1PR08MB4701; H:VE1PR08MB5149.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: /FUdZ8x9i92eQCf5h+AMV7arWOzAUmd91L06wm3BpeBtmEqwBo3jGnfOrrRhY85QZoxTEYOQh8BoUQXj3AFfGP4eEG+f3CpH8YT0HnE6pWcomKLsK7w6a/YReTDYgdZt8h6e6IenkZ4DIw62neQJPkKflEvao/mnSGlO0pN86whCw+rTuBQRbTV+c4Z4UnNAtqWOiSqqaZXRR73neWk38CqmkhbZoHhrJywfRtS8TNT1iSJQSvIGhzxpsN1+4PSBFIVGKfBRqZEXgb3UdS8JFKebUhbsBiqZaKHMkLIIRRPWBAkxIfA2iL8rbCKwJk0pAHbO680CGkSCC6oa5wWQNWuZpdWmFOO1a6zVnmZSBr288dgR0KIZ1S6Wr25n0AeotqJTVpOLBLSQgY/RYDDepUT1gVQ2R2e2vVWw2JMaSTc= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3bf59689-e2c0-4cb7-46bb-08d6c3b733f1 X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Apr 2019 04:34:53.5734 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: VE1PR08MB4701 Subject: Re: [dpdk-dev] ABI and inline functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190418043453.k9ukp6Nf43NCETFt3CxvBNgqg1yu5ZFzKBdKyqbfMF8@z> >=20 > On Wed, Apr 17, 2019 at 05:12:43AM +0000, Honnappa Nagarahalli wrote: > > Hello, > > There was a conversation [1] in the context of RCU library. I thought > > it needs participation from broader audience. Summary for the context > > (please look at [1] for full details) > > >=20 > Thanks for kicking off this discussion >=20 > > 1) How do we provide ABI compatibility when the code base contains > inline functions? Unless we freeze development in these inline functions = and > corresponding structures, it might not be possible to claim ABI compatibi= lity. > Application has to be recompiled. >=20 > I agree that in some cases the application "might" need to be recompiled, > but on the other hand I also think that there are many cases where ABI > function versioning can still be used to mitigate things. For example, if= we > think of a couple of various scenarios: >=20 > 1. If everything is inline and variables are allocated by app, e.g. > spinlock on stack, then there is no issue as everything is application > contained. If there is a bug fix which requires the structure to change, the applicati= on would need to recompile. I guess you are talking about a scenario when n= othing changed in the inline function/variables. >=20 > 2. If the situation is as in #1, but the structures in question are passe= d to > non-inline DPDK functions. In this case, any changes to the structures re= quire > those functions taking the structures to be versioned for old and new > structures I think this can get complicated on the inline function side. The applicati= on and the DPDK library will end up having different inline functions. The = changed inline function needs to be aware of 2 structure formats or the inl= ine function needs to be duplicated (one for each version of the structure)= . I guess these are the workarounds we have to do. >=20 > 3. If instead we have the case, like in rte_ring, where the data structur= es are > allocated using functions, but they are used via inlines in the app. In t= his > case the creation functions (and any other function using the > structures) need to be versioned so that the application gets the expecte= d > structure back from the creation. The new structure members have to be added such that the previous layout is= not affected. Either add at the end or use the gaps that are left because = of cache line alignment. >=20 > It might be useful to think of what other scenarios we have and what ones > are likely to be problematic, especially those that can't be solved by ha= ving > multiple function versions. >=20 > > 2) Every function that is not in the direct datapath (fastpath?) > > should not be inline. Exceptions or things like rx/tx burst, ring > > enqueue/dequeue, and packet alloc/free - Stephen >=20 > Agree with this. Anything not data path should not be inline. The next Yes, very clear on this. > question then is for data path items how to determine whether they need t= o > be inline or not. In general my rule-of-thumb: > * anything dealing with bursts of packets should not be inline > * anything what works with single packet at a time should be inline >=20 > The one exception to this rule is cases where we have to consider "empty > polling" as a likely use-case. For example, rte_ring_dequeue_burst works > with bursts of packets, but there is a valid application use case where a > thread could be polling a set of rings where only a small number are > actually busy. Right now, polling an empty ring only costs a few cycles, > meaning that it's neglible to have a few polls of empty rings before you = get > to a busy one. Having that function not-inline will cause that cost to ju= mp > significantly, so could cause problems. (This leads to the interesting sc= enario > where ring enqueue is safe to un-inline, while dequeue is not). A good way to think about it would be - ratio of amount of work done in the= function to cost of function call. >=20 > > 3) Plus synchronization routines: spin/rwlock/barrier, etc. I think > > rcu should be one of such exceptions - it is just another > > synchronization mechanism after all (just a bit more sophisticated). - > > Konstantin > > > In general I believe the synchronisation primitives should be inline. > However, it does come down to cost too - if a function takes 300 cycles, = do > we really care if it takes 305 or 310 instead to make it not inline? > Hopefully most synchronisation primitives are faster than this so this > situation should not occur. >=20 > > 2) and 3) can be good guidelines to what functions/APIs can be inline. = But, > I believe this guideline exists today too. Are there any thoughts to chan= ge > this? > > > > Coming to 1), I think DPDK cannot provide ABI compatibility unless all = the > inline functions are converted to normal functions and symbol versioning = is > done for those (not bothering about performance). > > > I disagree. I think even in the case of #1, we should be able to manage s= ome > changes without breaking ABI. I completely agree with you on trying to keep the ABI break surface small a= nd doing due diligence when one is required. However, I think claiming 100%= ABI compatibility all the time (or as frequently as other projects claim) = might be tough. IMO, we need to look beyond this to solve the end user prob= lem. May be packaging multiple LTS with distros when DPDK could not avoid b= reaking ABI compatibility? >=20 > > In this context, does it make sense to say that we will maintain API > > compatibility rather than saying ABI compatibility? This will also > > send the right message to the end users. > > > I would value ABI compatibility much higher than API compatibility. If > someone is recompiling the application anyway, making a couple of small > changes (large rework is obviously a different issue) to the code should = not > be a massive issue, I hope. On the other hand, ABI compatibility is neede= d to > allow seamless update from one version to another, and it's that ABI > compatiblity that allows distro's to pick up our latest and greatest vers= ions. >=20 I think it is also important to setup the right expectations to DPDK users.= i.e. DPDK will police itself better to provide ABI compatibility but occas= ionally it might not be possible. > Personally, I'd be happy enough to allow API changes at any point without > deprecation notice, so long as function versioning is used to ensure ABI > compatibility is kept. >=20 > My 2c. > /Bruce