Re: [dpdk-dev] [EXT] Re: [PATCH v2 4/4] add ABI checks

[Anoob Joseph <anoobj@marvell.com>]

Hi Akhil, Neil, Fiona

Sorry for the late response. I want to propose a new change in line with what you folks had proposed.

May be we can treat the new features EXPERIMENTAL until a new stable release.

enum rte_crypto_aead_algorithm {
         RTE_CRYPTO_AEAD_AES_CCM = 1,
         /**< AES algorithm in CCM mode. */
         RTE_CRYPTO_AEAD_AES_GCM,
         /**< AES algorithm in GCM mode. */
         RTE_CRYPTO_AEAD_LIST_END,
         /**< List end for stable */
         /** EXPERIMENTAL */
         RTE_CRYPTO_AEAD_CHACHA20_POLY1305,
         /**< Chacha20 cipher with poly1305 authenticator */
         RTE_CRYPTO_AEAD_LIST_END_EXPERIMENTAL
         /**< List end */
 };

And then introduce an experimental API,

const struct rte_cryptodev_capabilities *
rte_cryptodev_get_exp_capabilites(uint8_t dev_id);

The PMD owner is expected to add new capabilities (only new ones) to this one until the new feature is deemed stable (ie, in one of the next stable releases). We don't expect users to change their API/ABI. For applications using EXPERIMENTAL is allowed to use the above capabilities to get the EXPERIMENTAL features.

This does involve moving around code in PMD when one feature is added, but that's the risk PMD owner is taking by upstreaming as EXPERIMENTAL and not in stable release.

Thanks,
Anoob

> -----Original Message-----
> From: Neil Horman <nhorman@tuxdriver.com>
> Sent: Wednesday, February 5, 2020 3:40 AM
> To: Akhil Goyal <akhil.goyal@nxp.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>; Ferruh Yigit
> <ferruh.yigit@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> dev@dpdk.org; David Marchand <david.marchand@redhat.com>; Anoob Joseph
> <anoobj@marvell.com>; Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>;
> Richardson, Bruce <bruce.richardson@intel.com>; Mcnamara, John
> <john.mcnamara@intel.com>; dodji@seketeli.net; Andrew Rybchenko
> <arybchenko@solarflare.com>; aconole@redhat.com; bluca@debian.org;
> ktraynor@redhat.com
> Subject: [EXT] Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
> 
> External Email
> 
> ----------------------------------------------------------------------
> On Tue, Feb 04, 2020 at 10:32:01AM +0000, Akhil Goyal wrote:
> >
> > >
> > > 04/02/2020 11:16, Akhil Goyal:
> > > > Hi,
> > > > > On 2/3/2020 5:09 PM, Thomas Monjalon wrote:
> > > > > > 03/02/2020 10:30, Ferruh Yigit:
> > > > > >> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
> > > > > >>> 02/02/2020 14:05, Thomas Monjalon:
> > > > > >>>> 31/01/2020 15:16, Trahe, Fiona:
> > > > > >>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> > > > > >>>>>> 30/01/2020 17:09, Ferruh Yigit:
> > > > > >>>>>>> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>> I believe these enums will be used only in case of ASYM
> > > > > >>>>>>>> case which
> > > is
> > > > > experimental.
> > > > > >>>>>>>
> > > > > >>>>>>> Independent from being experiment and not, this
> > > > > >>>>>>> shouldn't be a
> > > > > problem, I think
> > > > > >>>>>>> this is a false positive.
> > > > > >>>>>>>
> > > > > >>>>>>> The ABI break can happen when a struct has been shared
> > > > > >>>>>>> between
> > > the
> > > > > application
> > > > > >>>>>>> and the library (DPDK) and the layout of that memory
> > > > > >>>>>>> know
> > > differently
> > > > > by
> > > > > >>>>>>> application and the library.
> > > > > >>>>>>>
> > > > > >>>>>>> Here in all cases, there is no layout/size change.
> > > > > >>>>>>>
> > > > > >>>>>>> As to the value changes of the enums, since application
> > > > > >>>>>>> compiled
> > > with
> > > > > old DPDK,
> > > > > >>>>>>> it will know only up to '6', 7 and more means invalid to
> > > > > >>>>>>> the
> > > application.
> > > > > So it
> > > > > >>>>>>> won't send these values also it should ignore these
> > > > > >>>>>>> values from
> > > library.
> > > > > Only
> > > > > >>>>>>> consequence is old application won't able to use new
> > > > > >>>>>>> features
> > > those
> > > > > new enums
> > > > > >>>>>>> provide but that is expected/normal.
> > > > > >>>>>>
> > > > > >>>>>> If library give higher value than expected by the
> > > > > >>>>>> application, if the application uses this value as array
> > > > > >>>>>> index, there can be an access out of bounds.
> > > > > >>>>>
> > > > > >>>>> [Fiona] All asymmetric APIs are experimental so above
> > > > > >>>>> shouldn't be a
> > > > > problem.
> > > > > >>>>> But for the same issue with sym crypto below, I believe
> > > > > >>>>> Ferruh's
> > > > > explanation makes
> > > > > >>>>> sense and I don't see how there can be an API breakage.
> > > > > >>>>> So if an application hasn't compiled against the new lib
> > > > > >>>>> it will be still
> > > using
> > > > > the old value
> > > > > >>>>> which will be within bounds. If it's picking up the higher
> > > > > >>>>> new value
> > > from
> > > > > the lib it must
> > > > > >>>>> have been compiled against the lib so shouldn't have problems.
> > > > > >>>>
> > > > > >>>> You say there is no ABI issue because the application will
> > > > > >>>> be re-
> > > compiled
> > > > > >>>> for the updated library. Indeed, compilation fixes compatibility
> issues.
> > > > > >>>> But this is not relevant for ABI compatibility.
> > > > > >>>> ABI compatibility means we can upgrade the library without
> > > recompiling
> > > > > >>>> the application and it must work.
> > > > > >>>> You think it is a false positive because you assume the
> > > > > >>>> application "picks" the new value. I think you miss the
> > > > > >>>> case where the new value is returned by a function in the upgraded
> library.
> > > > > >>>>
> > > > > >>>>> There are also no structs on the API which contain arrays
> > > > > >>>>> using this for sizing, so I don't see an opportunity for
> > > > > >>>>> an appl to have a mismatch in memory addresses.
> > > > > >>>>
> > > > > >>>> Let me demonstrate where the API may "use" the new value
> > > > > >>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the
> > > > > application.
> > > > > >>>>
> > > > > >>>> Once upon a time a DPDK application counting the number of
> > > > > >>>> devices supporting each AEAD algo (in order to find the best
> supported algo).
> > > > > >>>> It is done in an array indexed by algo id:
> > > > > >>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
> > > > > >>>> The application is compiled with DPDK 19.11, where
> > > > > >>>> RTE_CRYPTO_AEAD_LIST_END = 3.
> > > > > >>>> So the size of the application array aead_dev_count is 3.
> > > > > >>>> This binary is run with DPDK 20.02, where
> > > > > >>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
> > > > > >>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
> > > > > >>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
> > > > > >>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
> > > > > >>>> The application uses this value:
> > > > > >>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
> > > > > >>>> The application is crashing because of out of bound access.
> > > > > >>>
> > > > > >>> I'd say this is an example of bad written app.
> > > > > >>> It probably should check that returned by library value
> > > > > >>> doesn't exceed its internal array size.
> > > > > >>
> > > > > >> +1
> > > > > >>
> > > > > >> Application should ignore values >= MAX.
> > > > > >
> > > > > > Of course, blaming the API user is a lot easier than looking at the API.
> > > > > > Here the API has RTE_CRYPTO_AEAD_LIST_END which can be
> > > > > > understood as the max value for the application.
> > > > > > Value ranges are part of the ABI compatibility contract.
> > > > > > It seems you expect the application developer to be aware that
> > > > > > DPDK could return a higher value, so the application should
> > > > > > check every enum values after calling an API. CRAZY.
> > > > > >
> > > > > > When we decide to announce an ABI compatibility and do some
> > > > > > marketing, everyone is OK. But when we need to really make our
> > > > > > ABI compatible, I see little or no effort. DISAPPOINTING.
> > > > >
> > > > > This is not to blame the user or to do less work, this is more
> > > > > sane approach that library provides the _END/_MAX value and
> > > > > application uses it as valid
> > > range
> > > > > check.
> > > > >
> > > > > >
> > > > > >> Do you suggest we don't extend any enum or define between ABI
> > > breakage
> > > > > releases
> > > > > >> to be sure bad written applications not affected?
> > > > > >
> > > > > > I suggest we must consider not breaking any assumption made on the
> API.
> > > > > > Here we are breaking the enum range because nothing mentions
> > > _LIST_END
> > > > > > is not really the absolute end of the enum.
> > > > > > The solution is to make the change below in 20.02 + backport in
> 19.11.1:
> > > > > >
> > > > > > - _LIST_END
> > > > > > + _LIST_END, /* an ABI-compatible version may increase this
> > > > > > + value */ _LIST_MAX = _LIST_END + 42 /* room for
> > > > > > + ABI-compatible additions */
> > > > > > };
> > > > > >
> > > > >
> > > > > What is the point of "_LIST_MAX" here?
> > > > >
> > > > > Application should know the "_LIST_END" of when it has been
> > > > > compiled for
> > > the
> > > > > valid range check. Next time it is compiled "_LIST_END" may be
> > > > > different
> > > value
> > > > > but same logic applies.
> > > > >
> > > > > When "_LIST_END" is missing, application can't protect itself,
> > > > > in that case library should send only the values application
> > > > > knows when it is compiled,
> > > this
> > > > > means either we can't extend our enum/defines until next ABI
> > > > > breakage, or
> > > we
> > > > > need to do ABI versioning to the functions that returns an enum
> > > > > each time
> > > enum
> > > > > value extended.
> > > > >
> > > > > I believe it is saner to provide _END/_MAX values to the application to
> use.
> > > And
> > > > > if required comment them to clarify the expected usage.
> > > > >
> > > > > But in above suggestion application can't use or rely on
> > > > > "_LIST_MAX", it
> > > doesn't
> > > > > mean anything to application.
> > > > >
> > > >
> > > > Can we have something like
> > > > enum rte_crypto_aead_algorithm {
> > > >         RTE_CRYPTO_AEAD_AES_CCM = 1,
> > > >         /**< AES algorithm in CCM mode. */
> > > >         RTE_CRYPTO_AEAD_AES_GCM,
> > > >         /**< AES algorithm in GCM mode. */
> > > >         RTE_CRYPTO_AEAD_LIST_END,
> > > >         /**< List end for 19.11 ABI compatibility */
> > > >         RTE_CRYPTO_AEAD_CHACHA20_POLY1305,
> > > >         /**< Chacha20 cipher with poly1305 authenticator */
> > > >         RTE_CRYPTO_AEAD_LIST_END_2011
> > > >         /**< List end for 20.11 ABI compatibility */ };
> > > >
> > > > And in 20.11 release we alter the RTE_CRYPTO_AEAD_LIST_END to the
> > > > end
> > > and remove RTE_CRYPTO_AEAD_LIST_END_2011
> > > >
> > > > I believe it will be ok for any application which need to use the
> > > > chacha poly
> > > assume that this algo is
> > > > Experimental and will move to formal list in 20.11. This can be
> > > > documented in
> > > the documentation.
> > > > I believe there is no way to add a new enum as experimental so
> > > > far. This way
> > > we can formalize this
> > > > requirement as well.
> > > >
> > > > I believe this way effect of ABI breakage will be nullified.
> > >
> > > This is a possibility in the (a) proposal.
> > > But it breaks API (and ABI) because a high value is returned while
> > > not expected by the application.
> > >
> > > I guess ABI and release maintainers will vote no to such breakage.
> > > Note: I vote no.
> > >
> >
> > If that is the case, I would say we should go with b).
> >
> > Versioned APIs does not look good and adds more confusion.
> >
> What makes you say that?
> 
> Versioned APIs are the way you maintain backward compatibility.
> 
> If a library doesn't use versioned API's, then they either:
> 
> 1) break frequently, causing application headaches
> 2) have APIS that are so mature, strictly defined, and small, they never change
> anyway
> 3) go to the trouble of creating compat libs for as far back as they need to
> support
> 
> DPDK doesn't yet have a mature, stable API, so we have to do (1) or (2).  (1) has
> already been declared a bad idea, because application developers and distros
> have declared a desire for backwards compatibility.  We could go with (3)
> instead of ABI versioning, but between compat libs and versioning, the latter is
> the much less difficult way to handle that.
> 
> Neil