From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id DBD77A0679 for ; Thu, 4 Apr 2019 14:52:47 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7F5741B43D; Thu, 4 Apr 2019 14:52:47 +0200 (CEST) Received: from relay0145.mxlogin.com (relay0145.mxlogin.com [199.181.239.145]) by dpdk.org (Postfix) with ESMTP id 696671B43B; Thu, 4 Apr 2019 14:52:46 +0200 (CEST) Received: from filter001.mxrelay.co (unknown [116.203.155.46]) by relay0145.mxlogin.com (Postfix) with ESMTP id 74388CC3032C; Thu, 4 Apr 2019 07:52:45 -0500 (CDT) Received: from localhost (unknown [23.92.70.113]) by filter001.mxrelay.co (Postfix) with ESMTPS id 2583D100AC8; Thu, 4 Apr 2019 12:52:44 +0000 (UTC) Received: from [134.134.139.83] by localhost with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.89) (envelope-from ) id 1hC1rz-0002wJ-Mh; Thu, 04 Apr 2019 08:53:28 -0400 To: Bruce Richardson , "Burakov, Anatoly" Cc: dev@dpdk.org, Kevin Traynor , "techboard@dpdk.org" References: <94df3cc4-de54-72d6-84c6-81bebd209a81@intel.com> <20190404105447.GA1351@bricha3-MOBL.ger.corp.intel.com> From: Ray Kinsella Openpgp: preference=signencrypt Autocrypt: addr=mdr@ashroe.eu; keydata= mQINBFv8B3wBEAC+5ImcgbIvadt3axrTnt7Sxch3FsmWTTomXfB8YiuHT8KL8L/bFRQSL1f6 ASCHu3M89EjYazlY+vJUWLr0BhK5t/YI7bQzrOuYrl9K94vlLwzD19s/zB/g5YGGR5plJr0s JtJsFGEvF9LL3e+FKMRXveQxBB8A51nAHfwG0WSyx53d61DYz7lp4/Y4RagxaJoHp9lakn8j HV2N6rrnF+qt5ukj5SbbKWSzGg5HQF2t0QQ5tzWhCAKTfcPlnP0GymTBfNMGOReWivi3Qqzr S51Xo7hoGujUgNAM41sxpxmhx8xSwcQ5WzmxgAhJ/StNV9cb3HWIoE5StCwQ4uXOLplZNGnS uxNdegvKB95NHZjRVRChg/uMTGpg9PqYbTIFoPXjuk27sxZLRJRrueg4tLbb3HM39CJwSB++ YICcqf2N+GVD48STfcIlpp12/HI+EcDSThzfWFhaHDC0hyirHxJyHXjnZ8bUexI/5zATn/ux TpMbc/vicJxeN+qfaVqPkCbkS71cHKuPluM3jE8aNCIBNQY1/j87k5ELzg3qaesLo2n1krBH bKvFfAmQuUuJT84/IqfdVtrSCTabvDuNBDpYBV0dGbTwaRfE7i+LiJJclUr8lOvHUpJ4Y6a5 0cxEPxm498G12Z3NoY/mP5soItPIPtLR0rA0fage44zSPwp6cQARAQABtBxSYXkgS2luc2Vs bGEgPG1kckBhc2hyb2UuZXU+iQJUBBMBCAA+FiEEcDUDlKDJaDuJlfZfdJdaH/sCCpsFAlv8 B3wCGyMFCQlmAYAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQdJdaH/sCCptdtRAAl0oE msa+djBVYLIsax+0f8acidtWg2l9f7kc2hEjp9h9aZCpPchQvhhemtew/nKavik3RSnLTAyn B3C/0GNlmvI1l5PFROOgPZwz4xhJKGN7jOsRrbkJa23a8ly5UXwF3Vqnlny7D3z+7cu1qq/f VRK8qFyWkAb+xgqeZ/hTcbJUWtW+l5Zb+68WGEp8hB7TuJLEWb4+VKgHTpQ4vElYj8H3Z94a 04s2PJMbLIZSgmKDASnyrKY0CzTpPXx5rSJ1q+B1FCsfepHLqt3vKSALa3ld6bJ8fSJtDUJ7 JLiU8dFZrywgDIVme01jPbjJtUScW6jONLvhI8Z2sheR71UoKqGomMHNQpZ03ViVWBEALzEt TcjWgJFn8yAmxqM4nBnZ+hE3LbMo34KCHJD4eg18ojDt3s9VrDLa+V9fNxUHPSib9FD9UX/1 +nGfU/ZABmiTuUDM7WZdXri7HaMpzDRJUKI6b+/uunF8xH/h/MHW16VuMzgI5dkOKKv1LejD dT5mA4R+2zBS+GsM0oa2hUeX9E5WwjaDzXtVDg6kYq8YvEd+m0z3M4e6diFeLS77/sAOgaYL 92UcoKD+Beym/fVuC6/55a0e12ksTmgk5/ZoEdoNQLlVgd2INtvnO+0k5BJcn66ZjKn3GbEC VqFbrnv1GnA58nEInRCTzR1k26h9nmS5Ag0EW/wHfAEQAMth1vHr3fOZkVOPfod3M6DkQir5 xJvUW5EHgYUjYCPIa2qzgIVVuLDqZgSCCinyooG5dUJONVHj3nCbITCpJp4eB3PI84RPfDcC hf/V34N/Gx5mTeoymSZDBmXT8YtvV/uJvn+LvHLO4ZJdvq5ZxmDyxfXFmkm3/lLw0+rrNdK5 pt6OnVlCqEU9tcDBezjUwDtOahyV20XqxtUttN4kQWbDRkhT+HrA9WN9l2HX91yEYC+zmF1S OhBqRoTPLrR6g4sCWgFywqztpvZWhyIicJipnjac7qL/wRS+wrWfsYy6qWLIV80beN7yoa6v ccnuy4pu2uiuhk9/edtlmFE4dNdoRf7843CV9k1yRASTlmPkU59n0TJbw+okTa9fbbQgbIb1 pWsAuicRHyLUIUz4f6kPgdgty2FgTKuPuIzJd1s8s6p2aC1qo+Obm2gnBTduB+/n1Jw+vKpt 07d+CKEKu4CWwvZZ8ktJJLeofi4hMupTYiq+oMzqH+V1k6QgNm0Da489gXllU+3EFC6W1qKj tkvQzg2rYoWeYD1Qn8iXcO4Fpk6wzylclvatBMddVlQ6qrYeTmSbCsk+m2KVrz5vIyja0o5Y yfeN29s9emXnikmNfv/dA5fpi8XCANNnz3zOfA93DOB9DBf0TQ2/OrSPGjB3op7RCfoPBZ7u AjJ9dM7VABEBAAGJAjwEGAEIACYWIQRwNQOUoMloO4mV9l90l1of+wIKmwUCW/wHfAIbDAUJ CWYBgAAKCRB0l1of+wIKm3KlD/9w/LOG5rtgtCUWPl4B3pZvGpNym6XdK8cop9saOnE85zWf u+sKWCrxNgYkYP7aZrYMPwqDvilxhbTsIJl5HhPgpTO1b0i+c0n1Tij3EElj5UCg3q8mEc17 c+5jRrY3oz77g7E3oPftAjaq1ybbXjY4K32o3JHFR6I8wX3m9wJZJe1+Y+UVrrjY65gZFxcA thNVnWKErarVQGjeNgHV4N1uF3pIx3kT1N4GSnxhoz4Bki91kvkbBhUgYfNflGURfZT3wIKK +d50jd7kqRouXUCzTdzmDh7jnYrcEFM4nvyaYu0JjSS5R672d9SK5LVIfWmoUGzqD4AVmUW8 pcv461+PXchuS8+zpltR9zajl72Q3ymlT4BTAQOlCWkD0snBoKNUB5d2EXPNV13nA0qlm4U2 GpROfJMQXjV6fyYRvttKYfM5xYKgRgtP0z5lTAbsjg9WFKq0Fndh7kUlmHjuAIwKIV4Tzo75 QO2zC0/NTaTjmrtiXhP+vkC4pcrOGNsbHuaqvsc/ZZ0siXyYsqbctj/sCd8ka2r94u+c7o4l BGaAm+FtwAfEAkXHu4y5Phuv2IRR+x1wTey1U1RaEPgN8xq0LQ1OitX4t2mQwjdPihZQBCnZ wzOrkbzlJMNrMKJpEgulmxAHmYJKgvZHXZXtLJSejFjR0GdHJcL5rwVOMWB8cg== Message-ID: Date: Thu, 4 Apr 2019 13:52:34 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <20190404105447.GA1351@bricha3-MOBL.ger.corp.intel.com> Content-Type: text/plain; charset="UTF-8" Content-Language: en-US Content-Transfer-Encoding: 8bit X-AuthUser: mdr@ashroe.eu Subject: Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190404125234.5P5t-UDarqdBJ-sqBCr4JGBgsWTHsYuzuCOnoCRJrA0@z> On 04/04/2019 11:54, Bruce Richardson wrote: > On Thu, Apr 04, 2019 at 10:29:19AM +0100, Burakov, Anatoly wrote: >> On 03-Apr-19 4:42 PM, Ray Kinsella wrote: >>> Hi folks, >>> [SNIP] >> >> Hi Ray, >> >> My somewhat rambly 2 cents :) >> >> While i think some solution has to be found for the situation, we also have >> to balance this against speed of development and new features rollout. >> >> For example, let's consider what i am intimately familiar with - the memory >> rework. I have made enormous efforts to ensure that pre-18.05 and post-18.05 >> remain as ABI/API compatible as possible, but there were a couple of API >> calls that were removed, and there couldn't have been any replacements >> (these API's were exposing internal structures that shouldn't have been >> exposed in the first place), and 18.05 also broke the ABI compatibility, >> because there was no way to do it without it (shared internal structures >> needed to change in part to support multiprocess). >> >> So, if i understand your proposal correctly, assuming a 2-year waiting >> period for the deprecation of core API's, you would essentially still be >> waiting for the memory rework to land for a year more. Moreover, even >> *after* it has landed, there was a continuous stream of improvements and >> bugfixes, some of which has broke ABI compatibility as well. Some of them >> were my fault (as in, i could've foreseen the need for those changes, but >> didn't), but others came as a result of people using these new features in >> the wild and reporting issues/problems/suggestions - i am but one man, after >> all. Plus, you know, there's only 24 hours in a day, and some stuff takes >> time to implement :) >> >> Since this rework goes right at the heart of DPDK (arguably there isn't a >> more "core" API than memory!), there is no (sane) way in the universe to 1) >> keep backwards compatibility for this, or 2) keep two parallel versions of >> it. We also need to test all that, and, to be honest, one validation cycle >> for a release wouldn't be enough to figure out all of the kinks and >> implications of such a case. It was really great that memory rework has >> landed in 18.05 and we had time to improve and prepare it for an 18.11 LTS - >> i think everyone can say that it's in much better shape in 18.11 than it was >> in 18.05, but if we couldn't do an ABI break here or there, this rate of >> improvements would have slowed down significantly. >> >> Now, i understand that this is probably a highly exceptional case, but i'm >> sure that maintainers of other parts of DPDK will have their own examples of >> similar things happening. >> >> I have no idea what a proper solution would look like. Any "splitting" of >> the trees into "experimental" vs. "stable" will end up causing the same >> issue - people choose to use stable over experimental because, well, it's >> more stable, and new/experimental features don't get tested as much because >> no one runs the thing in the first place. >> >> TL;DR we have to be careful not to constrain the pace of >> development/bugfixing just for the sake of having a stable API/ABI :) >> > > Actually, I think we *do* need to constrain the pace of development for the > sake of ABI stability. At this stage DPDK has been around for quite a > number of years and so should be considered a fairly mature project - it > should just start acting like it. I 100% agree. If you break your users stuff regularly enough, they will eventually start looking around for an alternative that doesn't break their stuff quiet so regularly. We often use the pace of innovation in DPDK as justification for ABI/API breakages, but that approach is a real rarity among the Open Source community. I can't think of any mature project off-hand that share's it. I would ask is Linux any less innovative because they offer a stable API and have an absolute commitment to never breaking userspace? Would Linux have ever been as popular as it is today it they broke userspace every quarter? They reality is that they (Linux) find workarounds and compromise because there is an uber-maintainer Linus who had a strong ethos from the start not to break their users stuff - we need the same ethos in DPDK. > > Now, in terms of features like the memory rework, that is indeed a case > that there was no alternative other than a massive ABI break. However, for > that rework there was a strong need for improvement in that area that we > can make the case for an ABI break to support it - and it is of a scale > that nothing other than an ABI change would do. For other areas and > examples, I doubt there are many in the last couple of years that are of > that scale. I would also be inclined to agree with Bruce's points on memory rework was somewhat of an outlier, we don't see many like it. > My thoughts on the matter are: > 1. I think we really need to do work to start hiding more of our data > structures - like what Stephen's latest RFC does. This hiding should reduce > the scope for ABI breaks. > 2. Once done, I think we should commit to having an ABI break only in the > rarest of circumstances, and only with very large justification. I want us > to get to the point where DPDK releases can immediately be picked up by all > linux distros and rolled out because they are ABI compatible. The work that Anatoly describes removing APIs that exposed internal structures and Stephen H's RFC similarly are good examples of the kind of work required to prepare for this change. We need to take a good look at the API and reduce the number of unnecessary internal structures exposed. I never expected it going to to be a big bang - but is a definite direction we need to move towards over the next few release. > I'm not sure I like the idea of planned ABI break releases - that strikes > me as a plan where we end up with the same number of ABI breaks as before, > just balled into one release. > I agree, with this approach you just delaying the evil day and when that day comes it will 10-times more painful. This is no substitute for a don't break our users stuff ethos. > Question for Kevin, Luca and others who look at distro-packaging: is it the > case that each distro will only ship one version of DPDK, or is it possible > that if we have ABI breaks, a distro will provide two copies of DPDK > simultaneously, e.g. a 19.11 ABI version and a 20.11 ABI version? > > > So, in short, I'm generally in favour of a zero-tolerance approach for DPDK > ABI breaks, and having ABI breaks as a major event reserved only for > massive rework changes, such as major mbuf changes, or new memory layout or > similar. > > Regards, > /Bruce >