Re: [dpdk-dev] How to manage new APIs added after major ABI release?

[Luca Boccassi <bluca@debian.org>]

On Tue, 2019-12-10 at 18:04 +0100, Thomas Monjalon wrote:
> 10/12/2019 17:32, Bruce Richardson:
> > On Tue, Dec 10, 2019 at 04:20:41PM +0000, Luca Boccassi wrote:
> > > On Tue, 2019-12-10 at 15:46 +0000, Bruce Richardson wrote:
> > > > On Tue, Dec 10, 2019 at 03:03:51PM +0000, Luca Boccassi wrote:
> > > > > On Tue, 2019-12-10 at 14:36 +0000, Bruce Richardson wrote:
> > > > > > On Tue, Dec 10, 2019 at 12:40:53PM +0000, Ferruh Yigit
> > > > > > wrote:
> > > > > > > On 12/10/2019 12:04 PM, Bruce Richardson wrote:
> > > > > > > > On Tue, Dec 10, 2019 at 11:56:28AM +0000, Ferruh Yigit
> > > > > > > > wrote:
> > > > > > > > > Hi,
> > > > > > > > > 
> > > > > > > > > With new process, the major ABI releases will be
> > > > > > > > > compatible
> > > > > > > > > until it is
> > > > > > > > > deprecated (until next LTS for now),
> > > > > > > > > like current ABI version is 20 in DPDK_19.11 and DPDK
> > > > > > > > > versions
> > > > > > > > > until DPDK_20.11
> > > > > > > > > will be ABI compatible with this version.
> > > > > > > > > 
> > > > > > > > > But if we introduce a new API after major ABI, say in
> > > > > > > > > 20.02
> > > > > > > > > release, are we
> > > > > > > > > allowed to break the ABI for that API before
> > > > > > > > > DPDK_20.11?
> > > > > > > > > 
> > > > > > > > > If we allow it break, following problem will be
> > > > > > > > > observed:
> > > > > > > > > Assume an application using .so.20.1 library, and
> > > > > > > > > using the
> > > > > > > > > new
> > > > > > > > > API introduced
> > > > > > > > > in 20.02, lets say foo(),
> > > > > > > > > but when application switches to .so.20.2 (released
> > > > > > > > > via
> > > > > > > > > DPDK_20.05), application
> > > > > > > > > will fail because of ABI breakage in foo().
> > > > > > > > > 
> > > > > > > > > I think it is fair that application expects forward
> > > > > > > > > compatibility in minor
> > > > > > > > > versions of a shared library.
> > > > > > > > > Like if application linked against .so.20.2, fair to
> > > > > > > > > expect
> > > > > > > > > .so.20.3, .so.20.4
> > > > > > > > > etc will work fine. I think currently only .so.20.0
> > > > > > > > > is
> > > > > > > > > fully
> > > > > > > > > forward compatible.
> > > > > > > > > 
> > > > > > > > > If we all agree on this, we may need to tweak the
> > > > > > > > > process a
> > > > > > > > > little, but before
> > > > > > > > > diving into implementation details, I would like to
> > > > > > > > > be sure
> > > > > > > > > we
> > > > > > > > > are in same page.
> > > > > > > > > 
> > > > > > > > 
> > > > > > > > Well, any new API's generally come in as experimental,
> > > > > > > > in
> > > > > > > > which
> > > > > > > > case
> > > > > > > > changes are allowed, and breakage can be expected. If
> > > > > > > > they
> > > > > > > > are
> > > > > > > > not
> > > > > > > > experiemental, then the ABI policy applies to them in
> > > > > > > > that
> > > > > > > > they
> > > > > > > > cannot
> > > > > > > > change since they are part of the .21 ABI, even if that
> > > > > > > > ABI
> > > > > > > > is
> > > > > > > > not fully
> > > > > > > > complete yet. For any application only using stable,
> > > > > > > > non-
> > > > > > > > experimental
> > > > > > > > functions, forward compatibility must be maintained
> > > > > > > > IMHO.
> > > > > > > > 
> > > > > > > 
> > > > > > > Talking about not experimental APIs, experimental ones
> > > > > > > free
> > > > > > > from
> > > > > > > the process.
> > > > > > > 
> > > > > > > And when and API added in 20.02 (ABI_20.1) it is kind of
> > > > > > > still
> > > > > > > ABI_20, because
> > > > > > > it should be supported for following ABI_20.x, instead of
> > > > > > > calling
> > > > > > > it ABI_21, and
> > > > > > > this minor tweak (and mind shift) in .map files can be
> > > > > > > our
> > > > > > > solution.
> > > > > > 
> > > > > > Related at what to do with adding versions between major
> > > > > > ABI
> > > > > > versions, when
> > > > > > investigating with Kevin the ABI checking we have made an
> > > > > > unpleasant
> > > > > > discovery:
> > > > > > 
> > > > > > This minor version bumping from 20.0 to 20.1 has apparently
> > > > > > already
> > > > > > broken
> > > > > > our ABI according to libabigail.
> > > > > > 
> > > > > > The Gory Details [skip to the end for suggestions to fix]
> > > > > > ---------------------------------------------------------
> > > > > > ---
> > > > > > 
> > > > > > The reason for this is that the soversion encoded in each
> > > > > > library
> > > > > > -
> > > > > > whether
> > > > > > built with meson or make - is the full 20.0 version, not
> > > > > > just the
> > > > > > major ABI
> > > > > > .20 part. Then when apps link against DPDK, they actually
> > > > > > encode
> > > > > > the
> > > > > > 20.0.
> > > > > > 
> > > > > > So what this means is that currently - using a make build
> > > > > > as an
> > > > > > example
> > > > > > here - ldd on the latest head build gives:
> > > > > > 
> > > > > >  LD_LIBRARY_PATH=$(pwd)/x86_64-native-linux-gcc/lib ldd
> > > > > > x86_64-
> > > > > > native-linux-gcc/app/testpmd | head
> > > > > >         linux-vdso.so.1 (0x00007fff6813d000)
> > > > > >         librte_pmd_bond.so.20.1 =>
> > > > > > /home/bruce/dpdk.org/x86_64-
> > > > > > native-linux-gcc/lib/librte_pmd_bond.so.20.1
> > > > > > (0x00007f36d723c000)
> > > > > >         librte_pmd_dpaa.so.20.1 =>
> > > > > > /home/bruce/dpdk.org/x86_64-
> > > > > > native-linux-gcc/lib/librte_pmd_dpaa.so.20.1
> > > > > > (0x00007f36d7229000)
> > > > > >         librte_mempool_dpaa.so.20.1 =>
> > > > > > /home/bruce/dpdk.org/x86_64-
> > > > > > native-linux-gcc/lib/librte_mempool_dpaa.so.20.1
> > > > > > (0x00007f36d7224000)
> > > > > >         librte_pmd_ixgbe.so.20.1 =>
> > > > > > /home/bruce/dpdk.org/x86_64-
> > > > > > native-linux-gcc/lib/librte_pmd_ixgbe.so.20.1
> > > > > > (0x00007f36d71ba000)
> > > > > >         librte_pmd_i40e.so.20.1 =>
> > > > > > /home/bruce/dpdk.org/x86_64-
> > > > > > native-linux-gcc/lib/librte_pmd_i40e.so.20.1
> > > > > > (0x00007f36d7126000)
> > > > > >         librte_pmd_bnxt.so.20.1 =>
> > > > > > /home/bruce/dpdk.org/x86_64-
> > > > > > native-linux-gcc/lib/librte_pmd_bnxt.so.20.1
> > > > > > (0x00007f36d70e5000)
> > > > > >         librte_pmd_softnic.so.20.1 =>
> > > > > > /home/bruce/dpdk.org/x86_64-
> > > > > > native-linux-gcc/lib/librte_pmd_softnic.so.20.1
> > > > > > (0x00007f36d70b7000)
> > > > > >         librte_flow_classify.so.0.201 =>
> > > > > > /home/bruce/dpdk.org/x86_64-
> > > > > > native-linux-gcc/lib/librte_flow_classify.so.0.201
> > > > > > (0x00007f36d70b1000)
> > > > > >         librte_pipeline.so.20.1 =>
> > > > > > /home/bruce/dpdk.org/x86_64-
> > > > > > native-linux-gcc/lib/librte_pipeline.so.20.1
> > > > > > (0x00007f36d7088000)
> > > > > > ...
> > > > > > 
> > > > > > Similarly ldd on a 19.11 checkout gives:
> > > > > > 
> > > > > > LD_LIBRARY_PATH=$(pwd)/x86_64-native-linux-gcc_v19.11/lib
> > > > > > ldd
> > > > > > x86_64-
> > > > > > native-linux-gcc_v19.11/app/testpmd | head
> > > > > >         linux-vdso.so.1 (0x00007ffc2a964000)
> > > > > >         librte_pmd_bond.so.20.0 =>
> > > > > > /home/bruce/dpdk.org/x86_64-
> > > > > > native-linux-gcc_v19.11/lib/librte_pmd_bond.so.20.0
> > > > > > (0x00007fd4dc6b6000)
> > > > > >         librte_pmd_dpaa.so.20.0 =>
> > > > > > /home/bruce/dpdk.org/x86_64-
> > > > > > native-linux-gcc_v19.11/lib/librte_pmd_dpaa.so.20.0
> > > > > > (0x00007fd4dc6a3000)
> > > > > >         librte_mempool_dpaa.so.20.0 =>
> > > > > > /home/bruce/dpdk.org/x86_64-
> > > > > > native-linux-gcc_v19.11/lib/librte_mempool_dpaa.so.20.0
> > > > > > (0x00007fd4dc69e000)
> > > > > >         librte_pmd_ixgbe.so.20.0 =>
> > > > > > /home/bruce/dpdk.org/x86_64-
> > > > > > native-linux-gcc_v19.11/lib/librte_pmd_ixgbe.so.20.0
> > > > > > (0x00007fd4dc634000)
> > > > > >         librte_pmd_i40e.so.20.0 =>
> > > > > > /home/bruce/dpdk.org/x86_64-
> > > > > > native-linux-gcc_v19.11/lib/librte_pmd_i40e.so.20.0
> > > > > > (0x00007fd4dc5a0000)
> > > > > >         librte_pmd_bnxt.so.20.0 =>
> > > > > > /home/bruce/dpdk.org/x86_64-
> > > > > > native-linux-gcc_v19.11/lib/librte_pmd_bnxt.so.20.0
> > > > > > (0x00007fd4dc55d000)
> > > > > >         librte_pmd_softnic.so.20.0 =>
> > > > > > /home/bruce/dpdk.org/x86_64-
> > > > > > native-linux-gcc_v19.11/lib/librte_pmd_softnic.so.20.0
> > > > > > (0x00007fd4dc531000)
> > > > > >         librte_flow_classify.so.0.200 =>
> > > > > > /home/bruce/dpdk.org/x86_64-
> > > > > > native-linux-gcc_v19.11/lib/librte_flow_classify.so.0.200
> > > > > > (0x00007fd4dc52b000)
> > > > > >         librte_pipeline.so.20.0 =>
> > > > > > /home/bruce/dpdk.org/x86_64-
> > > > > > native-linux-gcc_v19.11/lib/librte_pipeline.so.20.0
> > > > > > (0x00007fd4dc502000)
> > > > > > 
> > > > > > The final check - using the 19.11 compiled testpmd with the
> > > > > > library
> > > > > > path
> > > > > > set to 20.02 versionned libs:
> > > > > > 
> > > > > > LD_LIBRARY_PATH=$(pwd)/x86_64-native-linux-gcc/lib ldd
> > > > > > x86_64-
> > > > > > native-
> > > > > > linux-gcc_v19.11/app/testpmd | head
> > > > > >         linux-vdso.so.1 (0x00007ffc711fc000)
> > > > > >         librte_pmd_bond.so.20.0 => not found
> > > > > >         librte_pmd_dpaa.so.20.0 => not found
> > > > > >         librte_mempool_dpaa.so.20.0 => not found
> > > > > >         librte_pmd_ixgbe.so.20.0 => not found
> > > > > >         librte_pmd_i40e.so.20.0 => not found
> > > > > >         librte_pmd_bnxt.so.20.0 => not found
> > > > > >         librte_pmd_softnic.so.20.0 => not found
> > > > > >         librte_flow_classify.so.0.200 => not found
> > > > > >         librte_pipeline.so.20.0 => not found
> > > > > > 
> > > > > > Fixing This
> > > > > > -----------
> > > > > > 
> > > > > > To fix this, we need to ensure that the SONAME remains
> > > > > > constant
> > > > > > across the
> > > > > > releases. Therefore, I currently see two options:
> > > > > > 
> > > > > > 1. keep 20.0 as the version and soname across all releases
> > > > > > in
> > > > > > 2020,
> > > > > > i.e.
> > > > > >   just revert the ABIVERSION change patch. Trouble there is
> > > > > > how
> > > > > > to
> > > > > > track
> > > > > >   20.02 vs 20.05 etc. etc.
> > > > > > 
> > > > > > 2. remove the .0, .1 from the SONAMES stored in the
> > > > > > libraries.
> > > > > > This
> > > > > > has the
> > > > > >   advantage of keeping the existing planned schemes, but
> > > > > > has the
> > > > > > really big
> > > > > >   downside of breaking ABI compatibility with anyone who
> > > > > > has
> > > > > > already
> > > > > >   compiled with 19.11.
> > > > > > 
> > > > > > Personally, of the two options - unless someone can come up
> > > > > > with
> > > > > > a
> > > > > > third
> > > > > > option - I'd tend towards the second, fixing the builds to
> > > > > > remove
> > > > > > the
> > > > > > .0 in
> > > > > > the soname, and releasing that ASAP as 19.11.1 before 19.11
> > > > > > gets
> > > > > > widespread
> > > > > > adoption. Since this ABI stability is new, teething
> > > > > > problems may
> > > > > > be
> > > > > > expected.
> > > > > > 
> > > > > > Thoughts and comments?
> > > > > > /Bruce
> > > > > > 
> > > > > > BTW: For meson, the patch for option 2 is just to remove
> > > > > > the
> > > > > > so_version
> > > > > > variable and all references to it from lib/meson.build and
> > > > > > drivers/meson.build. Haven't looked into a "make" fix yet.
> > > > > 
> > > > > Hi,
> > > > > 
> > > > > With libtool and its (arguably arcane) format, only the first
> > > > > digit
> > > > > is
> > > > > the ABI current version and gets encoded in the elf header.
> > > > > The
> > > > > other
> > > > > digits can be used to track compatible minor increments, and
> > > > > are
> > > > > mostly
> > > > > ignored. On the system a symlink libfoo.so.major ->
> > > > > libfoo.so.major.minor is added.
> > > > > 
> > > > > Eg:
> > > > > 
> > > > > $ readelf -d /usr/lib/x86_64-linux-gnu/libzmq.so.5.2.3 | grep
> > > > > SONAME
> > > > >  0x000000000000000e (SONAME)             Library soname:
> > > > > [libzmq.so.5]
> > > > > $ ls -l /usr/lib/x86_64-linux-gnu/libzmq.so.5
> > > > > lrwxrwxrwx 1 root root 15 Dec 31  2014 /usr/lib/x86_64-linux-
> > > > > gnu/libzmq.so.5 -> libzmq.so.5.2.3
> > > > > 
> > > > > Can we do the same? Not sure what the right incantation is
> > > > > for
> > > > > Meson,
> > > > > but it should be possibly.
> > > > > 
> > > > 
> > > > That's essentially option 2, and it's still an ABI break
> > > > because
> > > > existing
> > > > builds of 19.11 have the soname will the full version number in
> > > > it.
> > > > The
> > > > default behaviour for meson is exactly how you described it,
> > > > except
> > > > that
> > > > previously we needed more exact control over the version info
> > > > (for
> > > > your
> > > > dpdk-specific versions in the sonames) and so overrode the
> > > > soversion
> > > > explicitly. The fix for meson is to remove this overriding i.e.
> > > > remove
> > > > "soversion:" parameter for each shared_library() call.
> > > > 
> > > >  
> > > > > Also, we should leave the current at 20.0 - let's not break
> > > > > compatibility already, please :-)
> > > > > 
> > > > 
> > > > If we do this, maybe we can use 20.0.1 and 20.0.2 version
> > > > numbers?
> > > 
> > > Yes, that's what I meant - IMHO we should just take the hit and
> > > use the
> > > slightly weird 20.0 format until next year, and add a third digit
> > > for
> > > compatible updates. Then for v21 we can drop it.
> > > 
> > 
> > My concern with that is us forgetting, because we'll put in place
> > hacks to
> > have the soversion be the first two numbers of the version. Then we
> > need to
> > remember to remove those hacks before 20.11 goes out or we'll end
> > up with
> > 21.0 being the soversion again.
> > 
> > For that reason, I'd rather see us fix it now before 19.11 gets
> > widely
> > adopted.
> 
> Me too I vote for fixing soname as 20 in a small 19.11.1
> and release it quickly.
> 
> We are supposed to experience tooling and scheme adoption during this
> first year of new ABI policy. I think we should stick to the planned
> versionning, in order to avoid any surprise for the next major ABI.

If the concern is forgetting, I'll happily set a reminder in my
calendar and then nag everybody :-)

In my opinion declaring stability and compatibility, with press
releases and articles, and then immediately breaking it in the first
point release does not send the right message. Not because there was a
bug, which happens all the time, but because mantaining backward
compatibility sometimes means having to carry less-than-ideal or
downright ugly hacks for a while.

If we are not ready to maintain compatibility by carrying an ugly
workaround when the implications are minor or non-existant (I'm pretty
sure few would really cares what format the soname has, it's all
handled by tools), what happens next time when the implications are
that a broken feature stays broken, or that lower performance stays
lower, and so on?

-- 
Kind regards,
Luca Boccassi