* Re: [dpdk-dev] [PATCH] devtools: skip experimental libraries in ABI check
2020-02-21 16:10 17% [dpdk-dev] [PATCH] devtools: skip experimental libraries in ABI check David Marchand
@ 2020-02-21 16:13 4% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-21 16:13 UTC (permalink / raw)
To: David Marchand, dev; +Cc: thomas, dodji
On 2/21/2020 4:10 PM, David Marchand wrote:
> We don't provide ABI compatibility for experimental libraries.
> Skip those libraries by catching a soname containing a version starting
> with '0.'.
I am not familiar with the script and how libabigail works, but +1 to the idea.
>
> Align the special case for the glue libraries by using the soname too.
> Once libabigail has support for it, we will have a single type of rule.
>
> Fixes: 777014e56d07 ("devtools: add ABI checks")
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> devtools/check-abi.sh | 9 +++++++--
> 1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/devtools/check-abi.sh b/devtools/check-abi.sh
> index 0b4d1a37e..dd9120e69 100755
> --- a/devtools/check-abi.sh
> +++ b/devtools/check-abi.sh
> @@ -40,8 +40,13 @@ for dump in $(find $refdir -name "*.dump"); do
> # skip glue drivers, example librte_pmd_mlx5_glue.dump
> # We can't rely on a suppression rule for now:
> # https://sourceware.org/bugzilla/show_bug.cgi?id=25480
> - if [ "$name" != "${name%%_glue.dump}" ]; then
> - echo "Skipping ${dump}..."
> + if grep -qE "\<soname='[^']*_glue\.so\.[^']*'" $dump; then
> + echo "Skipped glue library $name."
> + continue
> + fi
> + # skip experimental libraries, with a sover starting with 0.
> + if grep -qE "\<soname='[^']*\.so\.0\.[^']*'" $dump; then
> + echo "Skipped experimental library $name."
> continue
> fi
> dump2=$(find $newdir -name $name)
>
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH] devtools: skip experimental libraries in ABI check
@ 2020-02-21 16:10 17% David Marchand
2020-02-21 16:13 4% ` Ferruh Yigit
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-21 16:10 UTC (permalink / raw)
To: dev; +Cc: thomas, ferruh.yigit, dodji
We don't provide ABI compatibility for experimental libraries.
Skip those libraries by catching a soname containing a version starting
with '0.'.
Align the special case for the glue libraries by using the soname too.
Once libabigail has support for it, we will have a single type of rule.
Fixes: 777014e56d07 ("devtools: add ABI checks")
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
devtools/check-abi.sh | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/devtools/check-abi.sh b/devtools/check-abi.sh
index 0b4d1a37e..dd9120e69 100755
--- a/devtools/check-abi.sh
+++ b/devtools/check-abi.sh
@@ -40,8 +40,13 @@ for dump in $(find $refdir -name "*.dump"); do
# skip glue drivers, example librte_pmd_mlx5_glue.dump
# We can't rely on a suppression rule for now:
# https://sourceware.org/bugzilla/show_bug.cgi?id=25480
- if [ "$name" != "${name%%_glue.dump}" ]; then
- echo "Skipping ${dump}..."
+ if grep -qE "\<soname='[^']*_glue\.so\.[^']*'" $dump; then
+ echo "Skipped glue library $name."
+ continue
+ fi
+ # skip experimental libraries, with a sover starting with 0.
+ if grep -qE "\<soname='[^']*\.so\.0\.[^']*'" $dump; then
+ echo "Skipped experimental library $name."
continue
fi
dump2=$(find $newdir -name $name)
--
2.23.0
^ permalink raw reply [relevance 17%]
* Re: [dpdk-dev] [PATCH] build: fix experimental library versioning
2020-02-21 15:24 5% ` David Marchand
@ 2020-02-21 15:34 3% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-21 15:34 UTC (permalink / raw)
To: David Marchand
Cc: Thomas Monjalon, Neil Horman, Bruce Richardson, dev, Kinsella,
Ray, Luca Boccassi, Kevin Traynor
On 2/21/2020 3:24 PM, David Marchand wrote:
> On Thu, Feb 20, 2020 at 8:55 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>>
>> The problem occurred when workaround that makes soname two digits
>> applied. With this change for the ABI version "20.0.1" the experimental
>> library version become ".so.2001".
>> After workaround removed in ABI version 21.0, the experimental library
>> version will become ".so.210".
>> "2001" is bigger value than "201" although it is a previous version of
>> the library version, this can break the version comparisons.
>>
>> To fix this, introducing a temporary sub level versioning for the
>> experimental libraries, so that minor version comparison will continue
>> to work.
>>
>> After change experimental libraries will follow below versioning:
>>
>> DPDK version ABI version soname library name
>> ------------ ----------- ------ ------------
>> DPDK 19.11 20.0 .so.0.200 .so.0.200
>> DPDK 20.02 20.0.1 .so.0.200.1 .so.0.200.1
>> DPDK 20.05 20.0.2 .so.0.200.2 .so.0.200.2
>> DPDK 20.11 21.0 .so.0.210 .so.0.210
>> DPDK 21.02 21.1 .so.0.211 .so.0.211
>>
>> Note: After workaround removed in DPDK 20.11 and soname switch back to
>> single digit this patch won't work and needs to be updated.
>>
>> Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>
> The soname change breaks existing applications like testpmd (linked to
> various experimental libraries).
> I have the full details if needed in a pastebin (v19.11, master, patch_applied).
>
> $ LD_LIBRARY_PATH=$HOME/builds/x86_64-native-linux-gcc+shared+kmods/install/lib
> $HOME/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/bin/testpmd
> /home/dmarchan/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/bin/testpmd:
> error while loading shared libraries: librte_flow_classify.so.0.200:
> cannot open shared object file: No such file or directory
>
>
> libabigail reports this change as an error, so if it is intended, we
> need to bypass the check for those libraries.
soname change was intended, we don't provide any ABI compatibility in the
experimental libraries, keeping same soname may mislead as if we do.
The impact of it is, any application linked against experimental library will
need to recompile in new release.
Can we add exception to the the libabigail for experimental libraries?
>
> Functions changes summary: 0 Removed, 0 Changed (2 filtered out), 0
> Added functions
> Variables changes summary: 0 Removed, 0 Changed, 0 Added variable
> ELF SONAME changed
> Functions changes summary: 0 Removed, 0 Changed (7 filtered out), 0
> Added functions
> Variables changes summary: 0 Removed, 0 Changed, 0 Added variable
> SONAME changed from 'librte_flow_classify.so.0.200' to
> 'librte_flow_classify.so.0.200.1'
> Error: ABI issue reported for 'abidiff --suppr
> /home/dmarchan/dpdk/devtools/libabigail.abignore --no-added-syms
> --headers-dir1 /home/dmarchan/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/include
> --headers-dir2 /home/dmarchan/builds/x86_64-native-linux-gcc+shared+kmods/install/include
> /home/dmarchan/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/dump/librte_flow_classify.dump
> /home/dmarchan/builds/x86_64-native-linux-gcc+shared+kmods/install/dump/librte_flow_classify.dump'
>
>
>
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] build: fix experimental library versioning
2020-02-20 19:54 10% ` [dpdk-dev] [PATCH] build: fix experimental library versioning Ferruh Yigit
2020-02-20 22:14 0% ` Luca Boccassi
2020-02-21 12:36 0% ` Ray Kinsella
@ 2020-02-21 15:24 5% ` David Marchand
2020-02-21 15:34 3% ` Ferruh Yigit
2 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-21 15:24 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Thomas Monjalon, Neil Horman, Bruce Richardson, dev, Kinsella,
Ray, Luca Boccassi, Kevin Traynor
On Thu, Feb 20, 2020 at 8:55 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> The problem occurred when workaround that makes soname two digits
> applied. With this change for the ABI version "20.0.1" the experimental
> library version become ".so.2001".
> After workaround removed in ABI version 21.0, the experimental library
> version will become ".so.210".
> "2001" is bigger value than "201" although it is a previous version of
> the library version, this can break the version comparisons.
>
> To fix this, introducing a temporary sub level versioning for the
> experimental libraries, so that minor version comparison will continue
> to work.
>
> After change experimental libraries will follow below versioning:
>
> DPDK version ABI version soname library name
> ------------ ----------- ------ ------------
> DPDK 19.11 20.0 .so.0.200 .so.0.200
> DPDK 20.02 20.0.1 .so.0.200.1 .so.0.200.1
> DPDK 20.05 20.0.2 .so.0.200.2 .so.0.200.2
> DPDK 20.11 21.0 .so.0.210 .so.0.210
> DPDK 21.02 21.1 .so.0.211 .so.0.211
>
> Note: After workaround removed in DPDK 20.11 and soname switch back to
> single digit this patch won't work and needs to be updated.
>
> Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
The soname change breaks existing applications like testpmd (linked to
various experimental libraries).
I have the full details if needed in a pastebin (v19.11, master, patch_applied).
$ LD_LIBRARY_PATH=$HOME/builds/x86_64-native-linux-gcc+shared+kmods/install/lib
$HOME/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/bin/testpmd
/home/dmarchan/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/bin/testpmd:
error while loading shared libraries: librte_flow_classify.so.0.200:
cannot open shared object file: No such file or directory
libabigail reports this change as an error, so if it is intended, we
need to bypass the check for those libraries.
Functions changes summary: 0 Removed, 0 Changed (2 filtered out), 0
Added functions
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable
ELF SONAME changed
Functions changes summary: 0 Removed, 0 Changed (7 filtered out), 0
Added functions
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable
SONAME changed from 'librte_flow_classify.so.0.200' to
'librte_flow_classify.so.0.200.1'
Error: ABI issue reported for 'abidiff --suppr
/home/dmarchan/dpdk/devtools/libabigail.abignore --no-added-syms
--headers-dir1 /home/dmarchan/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/include
--headers-dir2 /home/dmarchan/builds/x86_64-native-linux-gcc+shared+kmods/install/include
/home/dmarchan/abi/v19.11/x86_64-native-linux-gcc+shared+kmods/dump/librte_flow_classify.dump
/home/dmarchan/builds/x86_64-native-linux-gcc+shared+kmods/install/dump/librte_flow_classify.dump'
--
David Marchand
^ permalink raw reply [relevance 5%]
* Re: [dpdk-dev] [PATCH] build: fix experimental library versioning
2020-02-20 19:54 10% ` [dpdk-dev] [PATCH] build: fix experimental library versioning Ferruh Yigit
2020-02-20 22:14 0% ` Luca Boccassi
@ 2020-02-21 12:36 0% ` Ray Kinsella
2020-02-21 15:24 5% ` David Marchand
2 siblings, 0 replies; 200+ results
From: Ray Kinsella @ 2020-02-21 12:36 UTC (permalink / raw)
To: Ferruh Yigit, Thomas Monjalon, Neil Horman, Bruce Richardson
Cc: dev, ray.kinsella, bluca, david.marchand, ktraynor
On 20/02/2020 19:54, Ferruh Yigit wrote:
> The problem occurred when workaround that makes soname two digits
> applied. With this change for the ABI version "20.0.1" the experimental
> library version become ".so.2001".
> After workaround removed in ABI version 21.0, the experimental library
> version will become ".so.210".
> "2001" is bigger value than "201" although it is a previous version of
> the library version, this can break the version comparisons.
>
> To fix this, introducing a temporary sub level versioning for the
> experimental libraries, so that minor version comparison will continue
> to work.
>
> After change experimental libraries will follow below versioning:
>
> DPDK version ABI version soname library name
> ------------ ----------- ------ ------------
> DPDK 19.11 20.0 .so.0.200 .so.0.200
> DPDK 20.02 20.0.1 .so.0.200.1 .so.0.200.1
> DPDK 20.05 20.0.2 .so.0.200.2 .so.0.200.2
> DPDK 20.11 21.0 .so.0.210 .so.0.210
> DPDK 21.02 21.1 .so.0.211 .so.0.211
>
> Note: After workaround removed in DPDK 20.11 and soname switch back to
> single digit this patch won't work and needs to be updated.
>
> Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
>
Acked-by: Ray Kinsella <mdr@ashroe.eu>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> Cc: Thomas Monjalon <thomas@monjalon.net>
> Cc: ray.kinsella@intel.com
> Cc: nhorman@tuxdriver.com
> Cc: bruce.richardson@intel.com
> Cc: bluca@debian.org
> Cc: david.marchand@redhat.com
> Cc: ktraynor@redhat.com
> ---
> config/meson.build | 4 ++--
> mk/rte.lib.mk | 8 +++++---
> 2 files changed, 7 insertions(+), 5 deletions(-)
>
> diff --git a/config/meson.build b/config/meson.build
> index 6c46767e3..d3d2370ce 100644
> --- a/config/meson.build
> +++ b/config/meson.build
> @@ -31,8 +31,8 @@ abi_version = run_command(find_program('cat', 'more'),
> # sonames => librte_stable.so.20.0, librte_experimental.so.0.200
> abi_va = abi_version.split('.')
> stable_so_version = abi_va.length() == 2 ? abi_va[0] : abi_va[0] + '.' + abi_va[1]
> -experimental_abi_version = '0.' + ''.join(abi_va)
> -experimental_so_version = '0.' + ''.join(stable_so_version.split('.'))
> +experimental_abi_version = '0.' + abi_va[0] + abi_va[1] + '.' + abi_va[2]
> +experimental_so_version = experimental_abi_version
>
> # extract all version information into the build configuration
> dpdk_conf.set('RTE_VER_YEAR', pver.get(0).to_int())
> diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
> index b1a8372cc..682b590db 100644
> --- a/mk/rte.lib.mk
> +++ b/mk/rte.lib.mk
> @@ -15,8 +15,8 @@ LIBABIVER ?= $(shell cat $(RTE_SRCDIR)/ABI_VERSION)
> SOVER := $(basename $(LIBABIVER))
> ifeq ($(shell grep -s "^DPDK_" $(SRCDIR)/$(EXPORT_MAP)),)
> # EXPERIMENTAL ABI is versioned as 0.major+minor, e.g. 0.201 for 20.1 ABI
> -LIBABIVER := 0.$(shell echo $(LIBABIVER) | tr -d '.')
> -SOVER := 0.$(shell echo $(SOVER) | tr -d '.')
> +LIBABIVER := 0.$(shell echo $(LIBABIVER) | awk 'BEGIN { FS="." }; { print $$1$$2"."$$3 }')
> +SOVER := $(LIBABIVER)
> endif
>
> ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
> @@ -135,7 +135,9 @@ $(RTE_OUTPUT)/lib/$(LIB): $(LIB)
> $(Q)cp -f $(LIB) $(RTE_OUTPUT)/lib
> ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
> $(Q)ln -s -f $< $(shell echo $@ | sed 's/\.so.*/.so/')
> - $(Q)ln -s -f $< $(shell echo $@ | sed 's/\.so.*/.so.$(SOVER)/')
> + $(Q)if [ $(SOVER) != $(LIBABIVER) ]; then \
> + ln -s -f $< $(shell echo $@ | sed 's/\.so.*/.so.$(SOVER)/') ; \
> + fi
> endif
>
> #
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
2020-02-18 6:01 0% ` Stephen Hemminger
@ 2020-02-21 10:40 0% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-21 10:40 UTC (permalink / raw)
To: Stephen Hemminger
Cc: Neil Horman, John McNamara, Marko Kovacevic, dev, David Marchand,
Thomas Monjalon, Andrew Rybchenko
On 2/18/2020 6:01 AM, Stephen Hemminger wrote:
> On Mon, 17 Feb 2020 15:38:05 +0000
> Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
>> For the ABI compatibility it is better to hide internal data structures
>> from the application as much as possible. But because of some inline
>> functions 'struct eth_dev_ops' can't be hidden completely.
>>
>> Plan is to split the 'struct eth_dev_ops' into two as ones used by
>> inline functions and ones not used, and hide the second part that not
>> used by inline functions completely to the application.
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> ---
>> Cc: David Marchand <david.marchand@redhat.com>
>> Cc: Thomas Monjalon <thomas@monjalon.net>
>> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
>> ---
>> doc/guides/rel_notes/deprecation.rst | 6 ++++++
>> 1 file changed, 6 insertions(+)
>>
>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
>> index dfcca87ab..2aa431028 100644
>> --- a/doc/guides/rel_notes/deprecation.rst
>> +++ b/doc/guides/rel_notes/deprecation.rst
>> @@ -72,6 +72,12 @@ Deprecation Notices
>> In 19.11 PMDs will still update the field even when the offload is not
>> enabled.
>>
>> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
>> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
>> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
>> + access the struct directly. The struct will be separate in two, the ops used
>> + by inline functions still will be accessible to user but rest will be hidden.
>> +
>> * cryptodev: support for using IV with all sizes is added, J0 still can
>> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
>> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
>
> Good luck, truely hiding internals is hard. The mbuf structure is already split but not really
> hidden at all (just look at dwarf output). It doesn't make sense to do it unless
> you can really hide it.
I believe this can be done, only following [1] dev_ops are used by inline
functions, rest can be moved into separate struct and moved into ethdev driver
looking header.
[1]
rx_queue_count
rx_descriptor_done
rx_descriptor_status
tx_descriptor_status
>
> I would attack the rte_device stuff first. Make rte_device opaque to the application
> that would help for future versions. Then work backwards to rte_tehtdev.
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: deprecate using MAX values as array size
@ 2020-02-21 10:25 0% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-21 10:25 UTC (permalink / raw)
To: Neil Horman, John McNamara, Marko Kovacevic
Cc: dev, Thomas Monjalon, Bruce Richardson, Akhil Goyal,
Konstantin Ananyev, David Marchand, Jerin Jacob,
Andrew Rybchenko, Ori Kam
On 1/30/2020 2:20 PM, Ferruh Yigit wrote:
> Adding the deprecation notice as reminder for next ABI breakage release
> (20.11).
> This one time breakage is required to be able to extend enum/define
> without breaking ABI.
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> doc/guides/rel_notes/deprecation.rst | 14 ++++++++++++++
> 1 file changed, 14 insertions(+)
>
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index dfcca87ab..99d81564a 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -38,6 +38,20 @@ Deprecation Notices
> remove it from the externally visible ABI and allow it to be updated in the
> future.
>
> +* lib: will fix extending some enum/define breaking the ABI. There are multiple
> + samples in DPDK that enum/define terminated with a ``.*MAX.*`` value which is
> + used by iterators, and arrays holding these values are sized with this
> + ``.*MAX.*`` value. So extending this enum/define increases the ``.*MAX.*``
> + value which increases the size of the array and depending on how/where the
> + array is used this may break the ABI.
> + ``RTE_ETH_FLOW_MAX`` is one sample of the mentioned case, adding a new flow
> + type will break the ABI because of ``flex_mask[RTE_ETH_FLOW_MAX]`` array
> + usage in following public struct hierarchy:
> + ``rte_eth_fdir_flex_conf -> rte_fdir_conf -> rte_eth_conf (in the middle)``.
> + Need to identify this kind of usages and fix in 20.11, otherwise this blocks
> + us extending existing enum/define.
> + One solution can be using a fixed size array instead of ``.*MAX.*`` value.
> +
> * dpaa2: removal of ``rte_dpaa2_memsegs`` structure which has been replaced
> by a pa-va search library. This structure was earlier being used for holding
> memory segments used by dpaa2 driver for faster pa->va translation. This
>
Reminder of this deprecation notice.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] build: fix experimental library versioning
2020-02-20 19:54 10% ` [dpdk-dev] [PATCH] build: fix experimental library versioning Ferruh Yigit
@ 2020-02-20 22:14 0% ` Luca Boccassi
2020-02-21 12:36 0% ` Ray Kinsella
2020-02-21 15:24 5% ` David Marchand
2 siblings, 0 replies; 200+ results
From: Luca Boccassi @ 2020-02-20 22:14 UTC (permalink / raw)
To: Ferruh Yigit, Thomas Monjalon, Neil Horman, Bruce Richardson
Cc: dev, ray.kinsella, david.marchand, ktraynor
On Thu, 2020-02-20 at 19:54 +0000, Ferruh Yigit wrote:
> The problem occurred when workaround that makes soname two digits
> applied. With this change for the ABI version "20.0.1" the
> experimental
> library version become ".so.2001".
> After workaround removed in ABI version 21.0, the experimental
> library
> version will become ".so.210".
> "2001" is bigger value than "201" although it is a previous version
> of
> the library version, this can break the version comparisons.
>
> To fix this, introducing a temporary sub level versioning for the
> experimental libraries, so that minor version comparison will
> continue
> to work.
>
> After change experimental libraries will follow below versioning:
>
> DPDK version ABI version soname library name
> ------------ ----------- ------ ------------
> DPDK 19.11 20.0 .so.0.200 .so.0.200
> DPDK 20.02 20.0.1 .so.0.200.1 .so.0.200.1
> DPDK 20.05 20.0.2 .so.0.200.2 .so.0.200.2
> DPDK 20.11 21.0 .so.0.210 .so.0.210
> DPDK 21.02 21.1 .so.0.211 .so.0.211
>
> Note: After workaround removed in DPDK 20.11 and soname switch back
> to
> single digit this patch won't work and needs to be updated.
>
> Fixes: f26c2b39b271 ("build: fix soname info for 19.11
> compatibility")
>
> Signed-off-by: Ferruh Yigit <
> ferruh.yigit@intel.com
Acked-by: Luca Boccassi <bluca@debian.org>
This solution looks like a good compromise to me. Thanks Ferruh.
--
Kind regards,
Luca Boccassi
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] build: fix experimental library versioning
2020-02-20 19:50 8% ` Ferruh Yigit
@ 2020-02-20 19:54 10% ` Ferruh Yigit
2020-02-20 22:14 0% ` Luca Boccassi
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Ferruh Yigit @ 2020-02-20 19:54 UTC (permalink / raw)
To: Thomas Monjalon, Neil Horman, Bruce Richardson
Cc: dev, Ferruh Yigit, ray.kinsella, bluca, david.marchand, ktraynor
The problem occurred when workaround that makes soname two digits
applied. With this change for the ABI version "20.0.1" the experimental
library version become ".so.2001".
After workaround removed in ABI version 21.0, the experimental library
version will become ".so.210".
"2001" is bigger value than "201" although it is a previous version of
the library version, this can break the version comparisons.
To fix this, introducing a temporary sub level versioning for the
experimental libraries, so that minor version comparison will continue
to work.
After change experimental libraries will follow below versioning:
DPDK version ABI version soname library name
------------ ----------- ------ ------------
DPDK 19.11 20.0 .so.0.200 .so.0.200
DPDK 20.02 20.0.1 .so.0.200.1 .so.0.200.1
DPDK 20.05 20.0.2 .so.0.200.2 .so.0.200.2
DPDK 20.11 21.0 .so.0.210 .so.0.210
DPDK 21.02 21.1 .so.0.211 .so.0.211
Note: After workaround removed in DPDK 20.11 and soname switch back to
single digit this patch won't work and needs to be updated.
Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
Cc: Thomas Monjalon <thomas@monjalon.net>
Cc: ray.kinsella@intel.com
Cc: nhorman@tuxdriver.com
Cc: bruce.richardson@intel.com
Cc: bluca@debian.org
Cc: david.marchand@redhat.com
Cc: ktraynor@redhat.com
---
config/meson.build | 4 ++--
mk/rte.lib.mk | 8 +++++---
2 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/config/meson.build b/config/meson.build
index 6c46767e3..d3d2370ce 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -31,8 +31,8 @@ abi_version = run_command(find_program('cat', 'more'),
# sonames => librte_stable.so.20.0, librte_experimental.so.0.200
abi_va = abi_version.split('.')
stable_so_version = abi_va.length() == 2 ? abi_va[0] : abi_va[0] + '.' + abi_va[1]
-experimental_abi_version = '0.' + ''.join(abi_va)
-experimental_so_version = '0.' + ''.join(stable_so_version.split('.'))
+experimental_abi_version = '0.' + abi_va[0] + abi_va[1] + '.' + abi_va[2]
+experimental_so_version = experimental_abi_version
# extract all version information into the build configuration
dpdk_conf.set('RTE_VER_YEAR', pver.get(0).to_int())
diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
index b1a8372cc..682b590db 100644
--- a/mk/rte.lib.mk
+++ b/mk/rte.lib.mk
@@ -15,8 +15,8 @@ LIBABIVER ?= $(shell cat $(RTE_SRCDIR)/ABI_VERSION)
SOVER := $(basename $(LIBABIVER))
ifeq ($(shell grep -s "^DPDK_" $(SRCDIR)/$(EXPORT_MAP)),)
# EXPERIMENTAL ABI is versioned as 0.major+minor, e.g. 0.201 for 20.1 ABI
-LIBABIVER := 0.$(shell echo $(LIBABIVER) | tr -d '.')
-SOVER := 0.$(shell echo $(SOVER) | tr -d '.')
+LIBABIVER := 0.$(shell echo $(LIBABIVER) | awk 'BEGIN { FS="." }; { print $$1$$2"."$$3 }')
+SOVER := $(LIBABIVER)
endif
ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
@@ -135,7 +135,9 @@ $(RTE_OUTPUT)/lib/$(LIB): $(LIB)
$(Q)cp -f $(LIB) $(RTE_OUTPUT)/lib
ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
$(Q)ln -s -f $< $(shell echo $@ | sed 's/\.so.*/.so/')
- $(Q)ln -s -f $< $(shell echo $@ | sed 's/\.so.*/.so.$(SOVER)/')
+ $(Q)if [ $(SOVER) != $(LIBABIVER) ]; then \
+ ln -s -f $< $(shell echo $@ | sed 's/\.so.*/.so.$(SOVER)/') ; \
+ fi
endif
#
--
2.24.1
^ permalink raw reply [relevance 10%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-18 10:36 4% ` Kinsella, Ray
@ 2020-02-20 19:50 8% ` Ferruh Yigit
2020-02-20 19:54 10% ` [dpdk-dev] [PATCH] build: fix experimental library versioning Ferruh Yigit
0 siblings, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-02-20 19:50 UTC (permalink / raw)
To: Kinsella, Ray, Thomas Monjalon, Richardson, Bruce
Cc: nhorman, bluca, david.marchand, ktraynor, dev
On 2/18/2020 10:36 AM, Kinsella, Ray wrote:
>
>
>> -----Original Message-----
>> From: Thomas Monjalon <thomas@monjalon.net>
>> Sent: Tuesday 18 February 2020 09:50
>> To: Richardson, Bruce <bruce.richardson@intel.com>
>> Cc: Kinsella, Ray <ray.kinsella@intel.com>; nhorman@tuxdriver.com;
>> bluca@debian.org; david.marchand@redhat.com; ktraynor@redhat.com;
>> dev@dpdk.org
>> Subject: Re: ABI version of experimental libraries
>>
>> 18/02/2020 10:42, Bruce Richardson:
>>> On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
>>>> Hi,
>>>>
>>>> I would like to remind everybody our mistake when defining ABI
>> versions.
>>>> It has been "fixed" in this commit:
>>>> http://git.dpdk.org/dpdk/commit/?id=f26c2b39
>>>>
>>>> Please let's think about the consequence for the experimental
>> libraries.
>>>>
>>>> In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In
>> DPDK
>>>> 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
>>>> increasing, that's fine. When we'll switch to the new major ABI
>> and
>>>> use a normal numbering: In DPDK 20.11, we will use the ABI version
>>>> 0.210 with soname 0.21 Numbers are dropping.
>>>>
>>>> In short, for experimental libs, ABI 20.1 > ABI 21.0
>>>>
>>>> Are we OK with this or do we prefer reverting to normal numbering
>>>> for experimental libraries in DPDK 20.02?
>>>>
>>> Personally, I would not be too concerned about the verions of
>>> experimental libs, so long as they don't conflict across versions and
>>> have some similarity to the major ABI version for the release.
>>
>> You think sorting of the version numbers is not important?
>> If we don't care comparing experimental version numbers, then OK, let's
>> drop this patch. But please we need a small vote.
>>
>> Note: there would be no problem if we did not vote for having a special
>> numbering for pure experimental libraries (I am still against).
>>
>
> So while experimental library version numbers are not "important".
> I do agree with Thomas they should be sane, increase and should have a consistent format.
>
> Should we always pad them to 4 places?
> i.e.
>
> DPDK 19.11 ... 0.20 (needs to remain 0.20).
> DPDK 20.02 ... 0.2001
> DPDK 20.11 ... 0.2100
> DPDK 21.02 ... 0.2101
>
> Make sense?
>
What about following:
DPDK version ABI version soname library name
------------ ----------- ------ ------------
DPDK 19.11 20.0 .so.0.200 .so.0.200
DPDK 20.02 20.0.1 .so.0.200.1 .so.0.200.1
DPDK 20.05 20.0.2 .so.0.200.2 .so.0.200.2
DPDK 20.11 21.0 .so.0.210 .so.0.210
DPDK 21.02 21.1 .so.0.211 .so.0.211
This let us turn back to sane versions when soname fixed in 20.11.
Also I think for experimental libraries soname should NOT be same in minor
versions, otherwise it can create false impression that it is ABI compatible
between two minor versions.
I will reply the patch to this email.
^ permalink raw reply [relevance 8%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-20 14:35 0% ` Aaron Conole
@ 2020-02-20 16:01 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-20 16:01 UTC (permalink / raw)
To: ci; +Cc: Thomas Monjalon, dev, Michael Santana, Aaron Conole, Bruce Richardson
On Thu, Feb 20, 2020 at 3:35 PM Aaron Conole <aconole@redhat.com> wrote:
>
> David Marchand <david.marchand@redhat.com> writes:
>
> > On Thu, Feb 20, 2020 at 11:42 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> >>
> >> 19/02/2020 22:39, Aaron Conole:
> >> > David Marchand <david.marchand@redhat.com> writes:
> >> >
> >> > > Let's prune the jobs list to limit the amount of time spent by the robot
> >> > > in Travis.
> >> > >
> >> > > Since meson enables automatically the relevant components, there is not
> >> > > much gain in testing with extra_packages vs required_packages only.
> >> > >
> >> > > For a given arch/compiler/env combination, compilation is first tested
> >> > > in all jobs that run tests or build the docs or run the ABI checks.
> >> > > In the same context, for jobs that accumulates running tests, building
> >> > > the docs etc..., those steps are independent and can be split to save
> >> > > some cpu on Travis.
> >> > >
> >> > > With this, we go down from 21 to 15 jobs.
> >> > >
> >> > > Note: this patch requires a flush of the existing caches in Travis.
> >> > >
> >> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> >> > > ---
> >> >
> >> > In general, I think the idea with required vs. extra was to have a build
> >> > that did the minimum required, and one that did all the packages (to
> >> > allow a minimum vs. full DPDK).
> >> >
> >> > At least, that's from
> >> > http://mails.dpdk.org/archives/dev/2019-January/124007.html
> >>
> >> I think the benefit of a minimum build is to have a quick report,
> >> and easy to setup.
> >
> > Yes, Travis serves as a first gate when submitting patches.
> > But since Travis is best effort/free, we can't have a full coverage.
> >
> >
> >> > Not sure if that's still something anyone cares about.
> >>
> >> Given that Travis knows how to satisfy the dependencies,
> >> and that we must wait for all jobs to finish,
> >> I don't see any benefit of a minimal setup.
> >
> > This minimal setup also tests that dpdk dependencies are correct.
> > If a change makes something rely on libX and libX is in the packages
> > always installed in Travis, the missing dependency would not get
> > caught.
> >
> > But here, this adds too many jobs.
> >
> > UNH, Intel and other CIs should step in and fill this kind of gap.
>
> Okay, makes sense to me. Are one of these CI providers offering to
> cover this?
Maybe it is already covered, the best is to ask, so sending to ci@dpdk.org.
For the CI guys, which packages are installed on the systems/vms that
do compilation tests?
Is it possible to have a summary of the different setups?
Thanks.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-20 12:22 0% ` David Marchand
@ 2020-02-20 14:35 0% ` Aaron Conole
2020-02-20 16:01 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Aaron Conole @ 2020-02-20 14:35 UTC (permalink / raw)
To: David Marchand; +Cc: Thomas Monjalon, dev, Michael Santana, Bruce Richardson
David Marchand <david.marchand@redhat.com> writes:
> On Thu, Feb 20, 2020 at 11:42 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>>
>> 19/02/2020 22:39, Aaron Conole:
>> > David Marchand <david.marchand@redhat.com> writes:
>> >
>> > > Let's prune the jobs list to limit the amount of time spent by the robot
>> > > in Travis.
>> > >
>> > > Since meson enables automatically the relevant components, there is not
>> > > much gain in testing with extra_packages vs required_packages only.
>> > >
>> > > For a given arch/compiler/env combination, compilation is first tested
>> > > in all jobs that run tests or build the docs or run the ABI checks.
>> > > In the same context, for jobs that accumulates running tests, building
>> > > the docs etc..., those steps are independent and can be split to save
>> > > some cpu on Travis.
>> > >
>> > > With this, we go down from 21 to 15 jobs.
>> > >
>> > > Note: this patch requires a flush of the existing caches in Travis.
>> > >
>> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
>> > > ---
>> >
>> > In general, I think the idea with required vs. extra was to have a build
>> > that did the minimum required, and one that did all the packages (to
>> > allow a minimum vs. full DPDK).
>> >
>> > At least, that's from
>> > http://mails.dpdk.org/archives/dev/2019-January/124007.html
>>
>> I think the benefit of a minimum build is to have a quick report,
>> and easy to setup.
>
> Yes, Travis serves as a first gate when submitting patches.
> But since Travis is best effort/free, we can't have a full coverage.
>
>
>> > Not sure if that's still something anyone cares about.
>>
>> Given that Travis knows how to satisfy the dependencies,
>> and that we must wait for all jobs to finish,
>> I don't see any benefit of a minimal setup.
>
> This minimal setup also tests that dpdk dependencies are correct.
> If a change makes something rely on libX and libX is in the packages
> always installed in Travis, the missing dependency would not get
> caught.
>
> But here, this adds too many jobs.
>
> UNH, Intel and other CIs should step in and fill this kind of gap.
Okay, makes sense to me. Are one of these CI providers offering to
cover this?
Also,
Acked-by: Aaron Conole <aconole@redhat.com>
>
> --
> David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-20 10:42 0% ` Thomas Monjalon
@ 2020-02-20 12:22 0% ` David Marchand
2020-02-20 14:35 0% ` Aaron Conole
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-20 12:22 UTC (permalink / raw)
To: Thomas Monjalon, Aaron Conole; +Cc: dev, Michael Santana, Bruce Richardson
On Thu, Feb 20, 2020 at 11:42 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 19/02/2020 22:39, Aaron Conole:
> > David Marchand <david.marchand@redhat.com> writes:
> >
> > > Let's prune the jobs list to limit the amount of time spent by the robot
> > > in Travis.
> > >
> > > Since meson enables automatically the relevant components, there is not
> > > much gain in testing with extra_packages vs required_packages only.
> > >
> > > For a given arch/compiler/env combination, compilation is first tested
> > > in all jobs that run tests or build the docs or run the ABI checks.
> > > In the same context, for jobs that accumulates running tests, building
> > > the docs etc..., those steps are independent and can be split to save
> > > some cpu on Travis.
> > >
> > > With this, we go down from 21 to 15 jobs.
> > >
> > > Note: this patch requires a flush of the existing caches in Travis.
> > >
> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > > ---
> >
> > In general, I think the idea with required vs. extra was to have a build
> > that did the minimum required, and one that did all the packages (to
> > allow a minimum vs. full DPDK).
> >
> > At least, that's from
> > http://mails.dpdk.org/archives/dev/2019-January/124007.html
>
> I think the benefit of a minimum build is to have a quick report,
> and easy to setup.
Yes, Travis serves as a first gate when submitting patches.
But since Travis is best effort/free, we can't have a full coverage.
> > Not sure if that's still something anyone cares about.
>
> Given that Travis knows how to satisfy the dependencies,
> and that we must wait for all jobs to finish,
> I don't see any benefit of a minimal setup.
This minimal setup also tests that dpdk dependencies are correct.
If a change makes something rely on libX and libX is in the packages
always installed in Travis, the missing dependency would not get
caught.
But here, this adds too many jobs.
UNH, Intel and other CIs should step in and fill this kind of gap.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-19 19:41 3% ` [dpdk-dev] [PATCH 4/4] ci: reorganise " David Marchand
2020-02-19 21:39 0% ` Aaron Conole
2020-02-20 0:18 0% ` dwilder
@ 2020-02-20 11:07 0% ` Thomas Monjalon
2 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-20 11:07 UTC (permalink / raw)
To: David Marchand; +Cc: aconole, dev, Michael Santana
19/02/2020 20:41, David Marchand:
> Let's prune the jobs list to limit the amount of time spent by the robot
> in Travis.
>
> Since meson enables automatically the relevant components, there is not
> much gain in testing with extra_packages vs required_packages only.
>
> For a given arch/compiler/env combination, compilation is first tested
> in all jobs that run tests or build the docs or run the ABI checks.
> In the same context, for jobs that accumulates running tests, building
> the docs etc..., those steps are independent and can be split to save
> some cpu on Travis.
>
> With this, we go down from 21 to 15 jobs.
>
> Note: this patch requires a flush of the existing caches in Travis.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
A small step for DPDK but a big save for the earth ;)
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-19 21:39 0% ` Aaron Conole
@ 2020-02-20 10:42 0% ` Thomas Monjalon
2020-02-20 12:22 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-20 10:42 UTC (permalink / raw)
To: David Marchand, Aaron Conole; +Cc: dev, Michael Santana, Bruce Richardson
19/02/2020 22:39, Aaron Conole:
> David Marchand <david.marchand@redhat.com> writes:
>
> > Let's prune the jobs list to limit the amount of time spent by the robot
> > in Travis.
> >
> > Since meson enables automatically the relevant components, there is not
> > much gain in testing with extra_packages vs required_packages only.
> >
> > For a given arch/compiler/env combination, compilation is first tested
> > in all jobs that run tests or build the docs or run the ABI checks.
> > In the same context, for jobs that accumulates running tests, building
> > the docs etc..., those steps are independent and can be split to save
> > some cpu on Travis.
> >
> > With this, we go down from 21 to 15 jobs.
> >
> > Note: this patch requires a flush of the existing caches in Travis.
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
>
> In general, I think the idea with required vs. extra was to have a build
> that did the minimum required, and one that did all the packages (to
> allow a minimum vs. full DPDK).
>
> At least, that's from
> http://mails.dpdk.org/archives/dev/2019-January/124007.html
I think the benefit of a minimum build is to have a quick report,
and easy to setup.
> Not sure if that's still something anyone cares about.
Given that Travis knows how to satisfy the dependencies,
and that we must wait for all jobs to finish,
I don't see any benefit of a minimal setup.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [dpdk-announce] release candidate 20.02-rc3
@ 2020-02-20 9:41 4% ` Peng, Yuan
0 siblings, 0 replies; 200+ results
From: Peng, Yuan @ 2020-02-20 9:41 UTC (permalink / raw)
To: Thomas Monjalon, dev
DPDK 20.02 RC3 Test Report from intel:
Summary
• Totally 9898 test cases for 20.02 RC3, 9629 cases are executed, 9428 test cases passed, 178 test cases failed, 23 test case blocked, Execution rate is 97.28%, pass rate is 97.91%.
• Totally create ~278 new tests for DPDK20.02 new features.
• New to create 9 issues from STV in RC3 till now, not include some issues have found and reported before RC3.
• Met 1 build issue on Centos 7.4 with DPDK 20.02 RC3, now it has been fixed, and patch applied.
Details:
• Intel NIC PMD PF & VF regression: 99% executed, 0 critical issue and 1 high issue is tracked(verified).
Function cases include rx/tx/forward packets, packet processing, jumboframe, dual vlan, filter cases and so on: 5 bugs found, no critical and high level.
Performance cases such as l2fwd, l3fwd, single core and so on: no new bug is found.
• Crypto cases include ipsec_gw cryptodev, fips cryptodev, l2fwd crypto, performance cases and so on: no critical and no high bugs found.
• Vhost/virtio: PVP/loopback, qemu/virtio-user, multi-queue, dequeue zero copy cases and so on: 1 new medium bug is found.
• Power relate cases, no new bug is found.
• ABI test includes unit test, vhost and flow related tests. no new bug is found.
• 20.02 new features include ABI test, L2tpv3, crypto_dev related new features and so no. no new bug is found.
• Scecurity tests include Nic malformed to PF/VF, virtio, Compreqssion/cryptodev, power and invalid command line, all passed.
Thanks.
Yuan.
-----Original Message-----
From: dev <dev-bounces@dpdk.org> On Behalf Of Thomas Monjalon
Sent: Monday, February 17, 2020 5:59 AM
To: announce@dpdk.org
Subject: [dpdk-dev] [dpdk-announce] release candidate 20.02-rc3
A new DPDK release candidate is ready for testing:
https://git.dpdk.org/dpdk/tag/?id=v20.02-rc3
121 patches were integrated.
The release notes so far:
http://doc.dpdk.org/guides/rel_notes/release_20_02.html
Some deprecation notices might be added if reviewed on time.
The -rc4 will include only bug fixes, doc and tooling.
This next milestone is expected during next week.
Please hurry up to do the last checks and bug fixes.
You may share some release validation results by replying to this message (at dev@dpdk.org).
If you are preparing the next release cycle, please send your v1 patches before the 20.05 proposal deadline, which will happen on March 18th.
It is also time to build an estimated roadmap for the next cycles.
Thank you everyone
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-19 19:41 3% ` [dpdk-dev] [PATCH 4/4] ci: reorganise " David Marchand
2020-02-19 21:39 0% ` Aaron Conole
@ 2020-02-20 0:18 0% ` dwilder
2020-02-20 11:07 0% ` Thomas Monjalon
2 siblings, 0 replies; 200+ results
From: dwilder @ 2020-02-20 0:18 UTC (permalink / raw)
To: David Marchand; +Cc: thomas, aconole, dev, Michael Santana
On 2020-02-19 11:41, David Marchand wrote:
> Let's prune the jobs list to limit the amount of time spent by the
> robot
> in Travis.
>
> Since meson enables automatically the relevant components, there is not
> much gain in testing with extra_packages vs required_packages only.
>
> For a given arch/compiler/env combination, compilation is first tested
> in all jobs that run tests or build the docs or run the ABI checks.
> In the same context, for jobs that accumulates running tests, building
> the docs etc..., those steps are independent and can be split to save
> some cpu on Travis.
>
> With this, we go down from 21 to 15 jobs.
>
> Note: this patch requires a flush of the existing caches in Travis.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> .travis.yml | 51 +++++++++------------------------------------------
> 1 file changed, 9 insertions(+), 42 deletions(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index c174d5f30..b64a81bd0 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -15,15 +15,12 @@ addons:
> update: true
> packages: &required_packages
> - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip,
> ninja-build]
> + - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev,
> libfdt-dev, libjansson4]
>
> _aarch64_packages: &aarch64_packages
> - *required_packages
> - [gcc-aarch64-linux-gnu, libc6-dev-arm64-cross,
> pkg-config-aarch64-linux-gnu]
>
> -_extra_packages: &extra_packages
> - - *required_packages
> - - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev,
> libfdt-dev, libjansson4]
> -
> _libabigail_build_packages: &libabigail_build_packages
> - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
>
> @@ -43,66 +40,39 @@ jobs:
> - env: DEF_LIB="static"
> arch: amd64
> compiler: gcc
> - - env: DEF_LIB="shared"
> - arch: amd64
> - compiler: gcc
> - env: DEF_LIB="shared" RUN_TESTS=1
> arch: amd64
> compiler: gcc
> - - env: DEF_LIB="static" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: gcc
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
> + - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: amd64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *doc_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> + - env: DEF_LIB="shared" ABI_CHECKS=1
> arch: amd64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *libabigail_build_packages
> # x86_64 clang jobs
> - env: DEF_LIB="static"
> arch: amd64
> compiler: clang
> - - env: DEF_LIB="shared"
> - arch: amd64
> - compiler: clang
> - env: DEF_LIB="shared" RUN_TESTS=1
> arch: amd64
> compiler: clang
> - - env: DEF_LIB="static" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: clang
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: clang
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
> + - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: amd64
> compiler: clang
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *doc_packages
> # x86_64 cross-compiling 32-bits jobs
> - env: DEF_LIB="static" BUILD_32BIT=1
> @@ -131,9 +101,6 @@ jobs:
> - env: DEF_LIB="static"
> arch: arm64
> compiler: gcc
> - - env: DEF_LIB="shared"
> - arch: arm64
> - compiler: gcc
> - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: arm64
> compiler: gcc
> @@ -142,13 +109,13 @@ jobs:
> packages:
> - *required_packages
> - *doc_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> + - env: DEF_LIB="shared" ABI_CHECKS=1
> arch: arm64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *libabigail_build_packages
> # aarch64 clang jobs
> - env: DEF_LIB="static"
Much easier to follow, I had no issues adding ppc64le jobs.
All my travis jobs ran fine.
LGTM (the series)
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
2020-02-19 19:41 3% ` [dpdk-dev] [PATCH 4/4] ci: reorganise " David Marchand
@ 2020-02-19 21:39 0% ` Aaron Conole
2020-02-20 10:42 0% ` Thomas Monjalon
2020-02-20 0:18 0% ` dwilder
2020-02-20 11:07 0% ` Thomas Monjalon
2 siblings, 1 reply; 200+ results
From: Aaron Conole @ 2020-02-19 21:39 UTC (permalink / raw)
To: David Marchand; +Cc: thomas, dev, Michael Santana, Bruce Richardson
David Marchand <david.marchand@redhat.com> writes:
> Let's prune the jobs list to limit the amount of time spent by the robot
> in Travis.
>
> Since meson enables automatically the relevant components, there is not
> much gain in testing with extra_packages vs required_packages only.
>
> For a given arch/compiler/env combination, compilation is first tested
> in all jobs that run tests or build the docs or run the ABI checks.
> In the same context, for jobs that accumulates running tests, building
> the docs etc..., those steps are independent and can be split to save
> some cpu on Travis.
>
> With this, we go down from 21 to 15 jobs.
>
> Note: this patch requires a flush of the existing caches in Travis.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
In general, I think the idea with required vs. extra was to have a build
that did the minimum required, and one that did all the packages (to
allow a minimum vs. full DPDK).
At least, that's from
http://mails.dpdk.org/archives/dev/2019-January/124007.html
Not sure if that's still something anyone cares about.
> .travis.yml | 51 +++++++++------------------------------------------
> 1 file changed, 9 insertions(+), 42 deletions(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index c174d5f30..b64a81bd0 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -15,15 +15,12 @@ addons:
> update: true
> packages: &required_packages
> - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
> + - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4]
>
> _aarch64_packages: &aarch64_packages
> - *required_packages
> - [gcc-aarch64-linux-gnu, libc6-dev-arm64-cross, pkg-config-aarch64-linux-gnu]
>
> -_extra_packages: &extra_packages
> - - *required_packages
> - - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4]
> -
> _libabigail_build_packages: &libabigail_build_packages
> - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
>
> @@ -43,66 +40,39 @@ jobs:
> - env: DEF_LIB="static"
> arch: amd64
> compiler: gcc
> - - env: DEF_LIB="shared"
> - arch: amd64
> - compiler: gcc
> - env: DEF_LIB="shared" RUN_TESTS=1
> arch: amd64
> compiler: gcc
> - - env: DEF_LIB="static" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: gcc
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
> + - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: amd64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *doc_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> + - env: DEF_LIB="shared" ABI_CHECKS=1
> arch: amd64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *libabigail_build_packages
> # x86_64 clang jobs
> - env: DEF_LIB="static"
> arch: amd64
> compiler: clang
> - - env: DEF_LIB="shared"
> - arch: amd64
> - compiler: clang
> - env: DEF_LIB="shared" RUN_TESTS=1
> arch: amd64
> compiler: clang
> - - env: DEF_LIB="static" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: clang
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: clang
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
> + - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: amd64
> compiler: clang
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *doc_packages
> # x86_64 cross-compiling 32-bits jobs
> - env: DEF_LIB="static" BUILD_32BIT=1
> @@ -131,9 +101,6 @@ jobs:
> - env: DEF_LIB="static"
> arch: arm64
> compiler: gcc
> - - env: DEF_LIB="shared"
> - arch: arm64
> - compiler: gcc
> - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: arm64
> compiler: gcc
> @@ -142,13 +109,13 @@ jobs:
> packages:
> - *required_packages
> - *doc_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> + - env: DEF_LIB="shared" ABI_CHECKS=1
> arch: arm64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *libabigail_build_packages
> # aarch64 clang jobs
> - env: DEF_LIB="static"
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-18 9:50 4% ` Thomas Monjalon
@ 2020-02-18 10:36 4% ` Kinsella, Ray
2020-02-20 19:50 8% ` Ferruh Yigit
2020-02-19 11:43 7% ` [dpdk-dev] ABI version of experimental libraries Neil Horman
1 sibling, 1 reply; 200+ results
From: Kinsella, Ray @ 2020-02-18 10:36 UTC (permalink / raw)
To: Thomas Monjalon, Richardson, Bruce
Cc: nhorman, bluca, david.marchand, ktraynor, dev
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Tuesday 18 February 2020 09:50
> To: Richardson, Bruce <bruce.richardson@intel.com>
> Cc: Kinsella, Ray <ray.kinsella@intel.com>; nhorman@tuxdriver.com;
> bluca@debian.org; david.marchand@redhat.com; ktraynor@redhat.com;
> dev@dpdk.org
> Subject: Re: ABI version of experimental libraries
>
> 18/02/2020 10:42, Bruce Richardson:
> > On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
> > > Hi,
> > >
> > > I would like to remind everybody our mistake when defining ABI
> versions.
> > > It has been "fixed" in this commit:
> > > http://git.dpdk.org/dpdk/commit/?id=f26c2b39
> > >
> > > Please let's think about the consequence for the experimental
> libraries.
> > >
> > > In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In
> DPDK
> > > 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
> > > increasing, that's fine. When we'll switch to the new major ABI
> and
> > > use a normal numbering: In DPDK 20.11, we will use the ABI version
> > > 0.210 with soname 0.21 Numbers are dropping.
> > >
> > > In short, for experimental libs, ABI 20.1 > ABI 21.0
> > >
> > > Are we OK with this or do we prefer reverting to normal numbering
> > > for experimental libraries in DPDK 20.02?
> > >
> > Personally, I would not be too concerned about the verions of
> > experimental libs, so long as they don't conflict across versions and
> > have some similarity to the major ABI version for the release.
>
> You think sorting of the version numbers is not important?
> If we don't care comparing experimental version numbers, then OK, let's
> drop this patch. But please we need a small vote.
>
> Note: there would be no problem if we did not vote for having a special
> numbering for pure experimental libraries (I am still against).
>
So while experimental library version numbers are not "important".
I do agree with Thomas they should be sane, increase and should have a consistent format.
Should we always pad them to 4 places?
i.e.
DPDK 19.11 ... 0.20 (needs to remain 0.20).
DPDK 20.02 ... 0.2001
DPDK 20.11 ... 0.2100
DPDK 21.02 ... 0.2101
Make sense?
Ray K
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-19 12:43 4% ` Thomas Monjalon
2020-02-19 13:50 4% ` Ray Kinsella
@ 2020-02-19 21:17 4% ` Neil Horman
1 sibling, 0 replies; 200+ results
From: Neil Horman @ 2020-02-19 21:17 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Bruce Richardson, ray.kinsella, bluca, david.marchand, ktraynor, dev
On Wed, Feb 19, 2020 at 01:43:01PM +0100, Thomas Monjalon wrote:
> 19/02/2020 12:43, Neil Horman:
> > On Tue, Feb 18, 2020 at 10:50:09AM +0100, Thomas Monjalon wrote:
> > > 18/02/2020 10:42, Bruce Richardson:
> > > > On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
> > > > > Hi,
> > > > >
> > > > > I would like to remind everybody our mistake when defining ABI versions.
> > > > > It has been "fixed" in this commit:
> > > > > http://git.dpdk.org/dpdk/commit/?id=f26c2b39
> > > > >
> > > > > Please let's think about the consequence for the experimental libraries.
> > > > >
> > > > > In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
> > > > > 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
> > > > > increasing, that's fine. When we'll switch to the new major ABI and use
> > > > > a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
> > > > > soname 0.21 Numbers are dropping.
> > > > >
> > > > > In short, for experimental libs, ABI 20.1 > ABI 21.0
> > > > >
> > > > > Are we OK with this or do we prefer reverting to normal numbering for
> > > > > experimental libraries in DPDK 20.02?
> > > > >
> > > > Personally, I would not be too concerned about the verions of experimental
> > > > libs, so long as they don't conflict across versions and have some
> > > > similarity to the major ABI version for the release.
> > >
> > > You think sorting of the version numbers is not important?
> > > If we don't care comparing experimental version numbers,
> > > then OK, let's drop this patch. But please we need a small vote.
> > >
> > > Note: there would be no problem if we did not vote for having
> > > a special numbering for pure experimental libraries (I am still against).
> > >
> > I don't understand. Why would we change the ABI_VERSION at all in an LTS release at
> > all? This operation is meant to take an an experimental API and mark it as
> > stable by promoting its version number to the next major releases number. As
> > such, in the LTS release, we should keep the soname the same, as there should be
> > no other ABI changes in the promoted API.
>
> The library version number is updated because we add new symbols.
But thats a matter of policy (and possibly build mechanics). Not saying agree
with this approach necessecarily (as I've not fully thought it through), but we
could make room in the policy for this eventuality.
Neil
>
>
>
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs
@ 2020-02-19 19:41 3% ` David Marchand
2020-02-19 21:39 0% ` Aaron Conole
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: David Marchand @ 2020-02-19 19:41 UTC (permalink / raw)
To: thomas, aconole; +Cc: dev, Michael Santana
Let's prune the jobs list to limit the amount of time spent by the robot
in Travis.
Since meson enables automatically the relevant components, there is not
much gain in testing with extra_packages vs required_packages only.
For a given arch/compiler/env combination, compilation is first tested
in all jobs that run tests or build the docs or run the ABI checks.
In the same context, for jobs that accumulates running tests, building
the docs etc..., those steps are independent and can be split to save
some cpu on Travis.
With this, we go down from 21 to 15 jobs.
Note: this patch requires a flush of the existing caches in Travis.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
.travis.yml | 51 +++++++++------------------------------------------
1 file changed, 9 insertions(+), 42 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index c174d5f30..b64a81bd0 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -15,15 +15,12 @@ addons:
update: true
packages: &required_packages
- [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+ - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4]
_aarch64_packages: &aarch64_packages
- *required_packages
- [gcc-aarch64-linux-gnu, libc6-dev-arm64-cross, pkg-config-aarch64-linux-gnu]
-_extra_packages: &extra_packages
- - *required_packages
- - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4]
-
_libabigail_build_packages: &libabigail_build_packages
- [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
@@ -43,66 +40,39 @@ jobs:
- env: DEF_LIB="static"
arch: amd64
compiler: gcc
- - env: DEF_LIB="shared"
- arch: amd64
- compiler: gcc
- env: DEF_LIB="shared" RUN_TESTS=1
arch: amd64
compiler: gcc
- - env: DEF_LIB="static" EXTRA_PACKAGES=1
- arch: amd64
- compiler: gcc
- addons:
- apt:
- packages:
- - *extra_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
+ - env: DEF_LIB="shared" BUILD_DOCS=1
arch: amd64
compiler: gcc
addons:
apt:
packages:
- - *extra_packages
+ - *required_packages
- *doc_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
+ - env: DEF_LIB="shared" ABI_CHECKS=1
arch: amd64
compiler: gcc
addons:
apt:
packages:
- - *extra_packages
+ - *required_packages
- *libabigail_build_packages
# x86_64 clang jobs
- env: DEF_LIB="static"
arch: amd64
compiler: clang
- - env: DEF_LIB="shared"
- arch: amd64
- compiler: clang
- env: DEF_LIB="shared" RUN_TESTS=1
arch: amd64
compiler: clang
- - env: DEF_LIB="static" EXTRA_PACKAGES=1
- arch: amd64
- compiler: clang
- addons:
- apt:
- packages:
- - *extra_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1
- arch: amd64
- compiler: clang
- addons:
- apt:
- packages:
- - *extra_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
+ - env: DEF_LIB="shared" BUILD_DOCS=1
arch: amd64
compiler: clang
addons:
apt:
packages:
- - *extra_packages
+ - *required_packages
- *doc_packages
# x86_64 cross-compiling 32-bits jobs
- env: DEF_LIB="static" BUILD_32BIT=1
@@ -131,9 +101,6 @@ jobs:
- env: DEF_LIB="static"
arch: arm64
compiler: gcc
- - env: DEF_LIB="shared"
- arch: arm64
- compiler: gcc
- env: DEF_LIB="shared" BUILD_DOCS=1
arch: arm64
compiler: gcc
@@ -142,13 +109,13 @@ jobs:
packages:
- *required_packages
- *doc_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
+ - env: DEF_LIB="shared" ABI_CHECKS=1
arch: arm64
compiler: gcc
addons:
apt:
packages:
- - *extra_packages
+ - *required_packages
- *libabigail_build_packages
# aarch64 clang jobs
- env: DEF_LIB="static"
--
2.23.0
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v2 0/7] vfio/pci: SR-IOV support
@ 2020-02-19 18:53 2% Alex Williamson
0 siblings, 0 replies; 200+ results
From: Alex Williamson @ 2020-02-19 18:53 UTC (permalink / raw)
To: kvm
Cc: linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, bruce.richardson, cohuck
Changes since v1 are primarily to patch 3/7 where the commit log is
rewritten, along with option parsing and failure logging based on
upstream discussions. The primary user visible difference is that
option parsing is now much more strict. If a vf_token option is
provided that cannot be used, we generate an error. As a result of
this, opening a PF with a vf_token option will serve as a mechanism of
setting the vf_token. This seems like a more user friendly API than
the alternative of sometimes requiring the option (VFs in use) and
sometimes rejecting it, and upholds our desire that the option is
always either used or rejected.
This also means that the VFIO_DEVICE_FEATURE ioctl is not the only
means of setting the VF token, which might call into question whether
we absolutely need this new ioctl. Currently I'm keeping it because I
can imagine use cases, for example if a hypervisor were to support
SR-IOV, the PF device might be opened without consideration for a VF
token and we'd require the hypservisor to close and re-open the PF in
order to set a known VF token, which is impractical.
Series overview (same as provided with v1):
The synopsis of this series is that we have an ongoing desire to drive
PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
for this with DPDK drivers and potentially interesting future use
cases in virtualization. We've been reluctant to add this support
previously due to the dependency and trust relationship between the
VF device and PF driver. Minimally the PF driver can induce a denial
of service to the VF, but depending on the specific implementation,
the PF driver might also be responsible for moving data between VFs
or have direct access to the state of the VF, including data or state
otherwise private to the VF or VF driver.
To help resolve these concerns, we introduce a VF token into the VFIO
PCI ABI, which acts as a shared secret key between drivers. The
userspace PF driver is required to set the VF token to a known value
and userspace VF drivers are required to provide the token to access
the VF device. If a PF driver is restarted with VF drivers in use, it
must also provide the current token in order to prevent a rogue
untrusted PF driver from replacing a known driver. The degree to
which this new token is considered secret is left to the userspace
drivers, the kernel intentionally provides no means to retrieve the
current token.
Note that the above token is only required for this new model where
both the PF and VF devices are usable through vfio-pci. Existing
models of VFIO drivers where the PF is used without SR-IOV enabled
or the VF is bound to a userspace driver with an in-kernel, host PF
driver are unaffected.
The latter configuration above also highlights a new inverted scenario
that is now possible, a userspace PF driver with in-kernel VF drivers.
I believe this is a scenario that should be allowed, but should not be
enabled by default. This series includes code to set a default
driver_override for VFs sourced from a vfio-pci user owned PF, such
that the VFs are also bound to vfio-pci. This model is compatible
with tools like driverctl and allows the system administrator to
decide if other bindings should be enabled. The VF token interface
above exists only between vfio-pci PF and VF drivers, once a VF is
bound to another driver, the administrator has effectively pronounced
the device as trusted. The vfio-pci driver will note alternate
binding in dmesg for logging and debugging purposes.
Please review, comment, and test. The example QEMU implementation
provided with the RFC is still current for this version. Thanks,
Alex
RFC: https://lore.kernel.org/lkml/158085337582.9445.17682266437583505502.stgit@gimli.home/
v1: https://lore.kernel.org/lkml/158145472604.16827.15751375540102298130.stgit@gimli.home/
---
Alex Williamson (7):
vfio: Include optional device match in vfio_device_ops callbacks
vfio/pci: Implement match ops
vfio/pci: Introduce VF token
vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
vfio/pci: Add sriov_configure support
vfio/pci: Remove dev_fmt definition
vfio/pci: Cleanup .probe() exit paths
drivers/vfio/pci/vfio_pci.c | 383 +++++++++++++++++++++++++++++++++--
drivers/vfio/pci/vfio_pci_private.h | 10 +
drivers/vfio/vfio.c | 20 +-
include/linux/vfio.h | 4
include/uapi/linux/vfio.h | 37 +++
5 files changed, 426 insertions(+), 28 deletions(-)
^ permalink raw reply [relevance 2%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-19 12:43 4% ` Thomas Monjalon
@ 2020-02-19 13:50 4% ` Ray Kinsella
2020-02-19 21:17 4% ` Neil Horman
1 sibling, 0 replies; 200+ results
From: Ray Kinsella @ 2020-02-19 13:50 UTC (permalink / raw)
To: dev
On 19/02/2020 12:43, Thomas Monjalon wrote:
> 19/02/2020 12:43, Neil Horman:
>> On Tue, Feb 18, 2020 at 10:50:09AM +0100, Thomas Monjalon wrote:
>>> 18/02/2020 10:42, Bruce Richardson:
>>>> On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
>>>>> Hi,
>>>>>
>>>>> I would like to remind everybody our mistake when defining ABI versions.
>>>>> It has been "fixed" in this commit:
>>>>> http://git.dpdk.org/dpdk/commit/?id=f26c2b39
>>>>>
>>>>> Please let's think about the consequence for the experimental libraries.
>>>>>
>>>>> In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
>>>>> 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
>>>>> increasing, that's fine. When we'll switch to the new major ABI and use
>>>>> a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
>>>>> soname 0.21 Numbers are dropping.
>>>>>
>>>>> In short, for experimental libs, ABI 20.1 > ABI 21.0
>>>>>
>>>>> Are we OK with this or do we prefer reverting to normal numbering for
>>>>> experimental libraries in DPDK 20.02?
>>>>>
>>>> Personally, I would not be too concerned about the verions of experimental
>>>> libs, so long as they don't conflict across versions and have some
>>>> similarity to the major ABI version for the release.
>>>
>>> You think sorting of the version numbers is not important?
>>> If we don't care comparing experimental version numbers,
>>> then OK, let's drop this patch. But please we need a small vote.
>>>
>>> Note: there would be no problem if we did not vote for having
>>> a special numbering for pure experimental libraries (I am still against).
>>>
>> I don't understand. Why would we change the ABI_VERSION at all in an LTS release at
>> all? This operation is meant to take an an experimental API and mark it as
>> stable by promoting its version number to the next major releases number. As
>> such, in the LTS release, we should keep the soname the same, as there should be
>> no other ABI changes in the promoted API.
>
> The library version number is updated because we add new symbols.
>
>
So while experimental library version numbers are not "important".
I do agree with Thomas they should be sane, increase and should have a consistent format.
Should we always just pad them to 4 places as the simple solution?
i.e.
DPDK 19.11 ... 0.20 (needs to remain 0.20).
DPDK 20.02 ... 0.2001
DPDK 20.11 ... 0.2100
DPDK 21.02 ... 0.2101
Make sense?
Ray K
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-19 11:43 7% ` [dpdk-dev] ABI version of experimental libraries Neil Horman
@ 2020-02-19 12:43 4% ` Thomas Monjalon
2020-02-19 13:50 4% ` Ray Kinsella
2020-02-19 21:17 4% ` Neil Horman
0 siblings, 2 replies; 200+ results
From: Thomas Monjalon @ 2020-02-19 12:43 UTC (permalink / raw)
To: Neil Horman
Cc: Bruce Richardson, ray.kinsella, bluca, david.marchand, ktraynor, dev
19/02/2020 12:43, Neil Horman:
> On Tue, Feb 18, 2020 at 10:50:09AM +0100, Thomas Monjalon wrote:
> > 18/02/2020 10:42, Bruce Richardson:
> > > On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
> > > > Hi,
> > > >
> > > > I would like to remind everybody our mistake when defining ABI versions.
> > > > It has been "fixed" in this commit:
> > > > http://git.dpdk.org/dpdk/commit/?id=f26c2b39
> > > >
> > > > Please let's think about the consequence for the experimental libraries.
> > > >
> > > > In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
> > > > 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
> > > > increasing, that's fine. When we'll switch to the new major ABI and use
> > > > a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
> > > > soname 0.21 Numbers are dropping.
> > > >
> > > > In short, for experimental libs, ABI 20.1 > ABI 21.0
> > > >
> > > > Are we OK with this or do we prefer reverting to normal numbering for
> > > > experimental libraries in DPDK 20.02?
> > > >
> > > Personally, I would not be too concerned about the verions of experimental
> > > libs, so long as they don't conflict across versions and have some
> > > similarity to the major ABI version for the release.
> >
> > You think sorting of the version numbers is not important?
> > If we don't care comparing experimental version numbers,
> > then OK, let's drop this patch. But please we need a small vote.
> >
> > Note: there would be no problem if we did not vote for having
> > a special numbering for pure experimental libraries (I am still against).
> >
> I don't understand. Why would we change the ABI_VERSION at all in an LTS release at
> all? This operation is meant to take an an experimental API and mark it as
> stable by promoting its version number to the next major releases number. As
> such, in the LTS release, we should keep the soname the same, as there should be
> no other ABI changes in the promoted API.
The library version number is updated because we add new symbols.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] net/ionic: fix ABI version
2020-02-17 16:13 7% [dpdk-dev] [PATCH] net/ionic: fix ABI version Ferruh Yigit
@ 2020-02-19 12:37 4% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-19 12:37 UTC (permalink / raw)
To: Alfredo Cardigliano, Shannon Nelson; +Cc: dev, David Marchand
On 2/17/2020 4:13 PM, Ferruh Yigit wrote:
> Changed the ABI version to 20.0.1.
>
> Fixes: a72cf98cc93e ("net/ionic: add skeleton")
>
> Reported-by: David Marchand <david.marchand@redhat.com>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
Applied to dpdk-next-net/master, thanks.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-18 9:50 4% ` Thomas Monjalon
2020-02-18 10:36 4% ` Kinsella, Ray
@ 2020-02-19 11:43 7% ` Neil Horman
2020-02-19 12:43 4% ` Thomas Monjalon
1 sibling, 1 reply; 200+ results
From: Neil Horman @ 2020-02-19 11:43 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Bruce Richardson, ray.kinsella, bluca, david.marchand, ktraynor, dev
On Tue, Feb 18, 2020 at 10:50:09AM +0100, Thomas Monjalon wrote:
> 18/02/2020 10:42, Bruce Richardson:
> > On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
> > > Hi,
> > >
> > > I would like to remind everybody our mistake when defining ABI versions.
> > > It has been "fixed" in this commit:
> > > http://git.dpdk.org/dpdk/commit/?id=f26c2b39
> > >
> > > Please let's think about the consequence for the experimental libraries.
> > >
> > > In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
> > > 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
> > > increasing, that's fine. When we'll switch to the new major ABI and use
> > > a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
> > > soname 0.21 Numbers are dropping.
> > >
> > > In short, for experimental libs, ABI 20.1 > ABI 21.0
> > >
> > > Are we OK with this or do we prefer reverting to normal numbering for
> > > experimental libraries in DPDK 20.02?
> > >
> > Personally, I would not be too concerned about the verions of experimental
> > libs, so long as they don't conflict across versions and have some
> > similarity to the major ABI version for the release.
>
> You think sorting of the version numbers is not important?
> If we don't care comparing experimental version numbers,
> then OK, let's drop this patch. But please we need a small vote.
>
> Note: there would be no problem if we did not vote for having
> a special numbering for pure experimental libraries (I am still against).
>
I don't understand. Why would we change the ABI_VERSION at all in an LTS release at
all? This operation is meant to take an an experimental API and mark it as
stable by promoting its version number to the next major releases number. As
such, in the LTS release, we should keep the soname the same, as there should be
no other ABI changes in the promoted API.
Neil
>
>
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH v2] ci: build and use libabigail 1.6
2020-02-18 16:38 0% ` David Marchand
@ 2020-02-19 8:13 0% ` Dodji Seketeli
0 siblings, 0 replies; 200+ results
From: Dodji Seketeli @ 2020-02-19 8:13 UTC (permalink / raw)
To: David Marchand
Cc: Thomas Monjalon, dev, Aaron Conole, Christian Ehrhardt,
Kevin Laatz, Michael Santana
Hello,
David Marchand <david.marchand@redhat.com> writes:
>> > + LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
>> > + LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
>> > +
>> > + if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
>> > + rm -rf libabigail
>> > + # if we change libabigail, invalidate existing abi cache
>> > + rm -rf reference
>> > + fi
>> > +
>> > + if [ ! -d libabigail ]; then
>> > + git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
>>
>> Why not using the tarball?
>> http://mirrors.kernel.org/sourceware/libabigail/libabigail-1.6.tar.gz
>
> No good reason "now".
>
> I was first bitten by a reference to redhat-hardened-ld in some
> libtool script in the tarball (/me looks in Dodji direction).
> I then considered switching to different versions of libabigail by
> just setting the LIBABIGAIL_VERSION env variable from .travis.yml.
> I ended up with the current latest version which is also what is in
> Ubuntu latest releases.
I think either way (distro package, tarball and git) has some benefits
and drawbacks.
I think one benefit of /being able/ to use the code from git is for
cases where you guys need some new features (and we tend to continuously
add fixes/functionalities to the git repository) that is available in
git only for now. Hopefully, with time, you'll only need to use the
distro package.
FWIW, I like the fact that your setup lets you choose between the distro
package, the tarball or the git code. That's powerful.
Cheers,
--
Dodji
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v3 0/2] improve Travis coverage
@ 2020-02-18 22:36 3% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-18 22:36 UTC (permalink / raw)
To: Thomas Monjalon; +Cc: dev, Aaron Conole, Bruce Richardson
On Tue, Feb 18, 2020 at 10:20 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> Remove some useless jobs and add more packages
> to build more PMDs in Travis.
>
> v3: reword commit logs
> v2: remove libbpf because not available
>
> Thomas Monjalon (2):
> ci: remove redundant configs disabling kmods
> ci: enable more drivers in Travis builds
>
> .travis.yml | 30 +++++-------------------------
> 1 file changed, 5 insertions(+), 25 deletions(-)
About enabling kmod compilation again, this can be done in a followup patch.
For the series,
Acked-by: David Marchand <david.marchand@redhat.com>
Rebased on master, cache for abi checks jobs deleted.
Series applied.
--
David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v3] ci: build and use libabigail 1.6
2020-02-18 20:31 0% ` Aaron Conole
@ 2020-02-18 20:59 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-18 20:59 UTC (permalink / raw)
To: David Marchand
Cc: Thomas Monjalon, dev, Christian Ehrhardt, Dodji Seketeli,
Kevin Laatz, Michael Santana, Aaron Conole
On Tue, Feb 18, 2020 at 9:31 PM Aaron Conole <aconole@redhat.com> wrote:
> David Marchand <david.marchand@redhat.com> writes:
>
> > libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> > breakage [1].
> > This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> > in launchpad [3].
> >
> > But for now, build and use the last version 1.6 so that the ABI checks
> > can be kept.
> >
> > 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> > 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b
> > 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > Acked-by: Thomas Monjalon <thomas@monjalon.net>
> > ---
>
> Acked-by: Aaron Conole <aconole@redhat.com>
Applied.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v3] ci: build and use libabigail 1.6
2020-02-18 20:13 9% ` [dpdk-dev] [PATCH v3] " David Marchand
@ 2020-02-18 20:31 0% ` Aaron Conole
2020-02-18 20:59 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Aaron Conole @ 2020-02-18 20:31 UTC (permalink / raw)
To: David Marchand
Cc: thomas, dev, christian.ehrhardt, dodji, kevin.laatz, Michael Santana
David Marchand <david.marchand@redhat.com> writes:
> libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> breakage [1].
> This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> in launchpad [3].
>
> But for now, build and use the last version 1.6 so that the ABI checks
> can be kept.
>
> 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b
> 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Thomas Monjalon <thomas@monjalon.net>
> ---
Acked-by: Aaron Conole <aconole@redhat.com>
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH v3] ci: build and use libabigail 1.6
2020-02-17 13:59 9% [dpdk-dev] [PATCH] ci: build and use libabigail 1.6 David Marchand
` (2 preceding siblings ...)
2020-02-18 14:29 9% ` [dpdk-dev] [PATCH v2] " David Marchand
@ 2020-02-18 20:13 9% ` David Marchand
2020-02-18 20:31 0% ` Aaron Conole
3 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-18 20:13 UTC (permalink / raw)
To: thomas
Cc: dev, aconole, christian.ehrhardt, dodji, kevin.laatz, Michael Santana
libabigail 1.2 (at least) reports changes in 'const' property as an ABI
breakage [1].
This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
in launchpad [3].
But for now, build and use the last version 1.6 so that the ABI checks
can be kept.
1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b
3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
---
Changelog since v2:
- used tarball instead of git,
- moved libabigail compilation/install in a function,
Changelog since v1:
- keep extra_packages in ABI checks jobs,
- fix configure step,
---
.ci/linux-build.sh | 29 +++++++++++++++++++++++++++++
.travis.yml | 8 +++++++-
2 files changed, 36 insertions(+), 1 deletion(-)
diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index c7c3840fc..d500c4c00 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -14,6 +14,20 @@ on_error() {
}
trap on_error EXIT
+install_libabigail() {
+ version=$1
+ instdir=$2
+
+ wget -q "http://mirrors.kernel.org/sourceware/libabigail/${version}.tar.gz"
+ tar -xf ${version}.tar.gz
+ cd $version && autoreconf -vfi && cd -
+ mkdir $version/build
+ cd $version/build && ../configure --prefix=$instdir && cd -
+ make -C $version/build all install
+ rm -rf $version
+ rm ${version}.tar.gz
+}
+
if [ "$AARCH64" = "1" ]; then
# convert the arch specifier
OPTS="$OPTS --cross-file config/arm/arm64_armv8_linux_gcc"
@@ -38,6 +52,21 @@ if [ "$AARCH64" != "1" ]; then
fi
if [ "$ABI_CHECKS" = "1" ]; then
+ LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
+
+ if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
+ rm -rf libabigail
+ # if we change libabigail, invalidate existing abi cache
+ rm -rf reference
+ fi
+
+ if [ ! -d libabigail ]; then
+ install_libabigail $LIBABIGAIL_VERSION $(pwd)/libabigail
+ echo $LIBABIGAIL_VERSION > libabigail/VERSION
+ fi
+
+ export PATH=$(pwd)/libabigail/bin:$PATH
+
REF_GIT_REPO=${REF_GIT_REPO:-https://dpdk.org/git/dpdk}
REF_GIT_TAG=${REF_GIT_TAG:-v19.11}
diff --git a/.travis.yml b/.travis.yml
index 22539d823..030064c94 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -2,6 +2,7 @@ language: c
cache:
ccache: true
directories:
+ - libabigail
- reference
compiler:
- gcc
@@ -24,7 +25,10 @@ aarch64_packages: &aarch64_packages
extra_packages: &extra_packages
- *required_packages
- - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4, abigail-tools]
+ - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
+
+libabigail_build_packages: &libabigail_build_packages
+ - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
build_32b_packages: &build_32b_packages
- *required_packages
@@ -160,6 +164,7 @@ matrix:
apt:
packages:
- *extra_packages
+ - *libabigail_build_packages
- env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
arch: arm64
compiler: gcc
@@ -167,5 +172,6 @@ matrix:
apt:
packages:
- *extra_packages
+ - *libabigail_build_packages
script: ./.ci/${TRAVIS_OS_NAME}-build.sh
--
2.23.0
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH v2] ethdev: add comment to warn of ABI breakage
2020-02-18 13:39 13% ` [dpdk-dev] [PATCH v2] " Ciara Power
@ 2020-02-18 18:40 4% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-18 18:40 UTC (permalink / raw)
To: Ciara Power, thomas, arybchenko; +Cc: dev
On 2/18/2020 1:39 PM, Ciara Power wrote:
> If a function is added to the eth_dev_ops struct before
> tx_descriptor_status function, this will cause ABI breakage. This is due
> to static inline functions using this function, and some other functions
> above it in the struct, so they cannot change position. A comment is
> added to inform developers of this possible breakage.
>
> Signed-off-by: Ciara Power <ciara.power@intel.com>
> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
> Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
Applied to dpdk-next-net/master, thanks.
(uppercase ABOVE/BELOW as suggested applied while merging)
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2] ci: build and use libabigail 1.6
2020-02-18 15:46 0% ` Thomas Monjalon
@ 2020-02-18 16:38 0% ` David Marchand
2020-02-19 8:13 0% ` Dodji Seketeli
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-18 16:38 UTC (permalink / raw)
To: Thomas Monjalon
Cc: dev, Aaron Conole, Christian Ehrhardt, Dodji Seketeli,
Kevin Laatz, Michael Santana
On Tue, Feb 18, 2020 at 4:46 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 18/02/2020 15:29, David Marchand:
> > libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> > breakage [1].
> > This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> > in launchpad [3].
> >
> > But for now, build and use the last version 1.6 so that the ABI checks
> > can be kept.
> >
> > 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> > 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b
> > 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
>
> Acked-by: Thomas Monjalon <thomas@monjalon.net>
>
> I suggest few improvements below:
>
> > --- a/.ci/linux-build.sh
> > +++ b/.ci/linux-build.sh
> > if [ "$ABI_CHECKS" = "1" ]; then
>
> What do you think about moving the libabigail install in a function?
No strong opinion, we had everything inline so far.
>
> We could justify this with a comment about installing the latest version.
>
> > + LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
> > + LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
> > +
> > + if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
> > + rm -rf libabigail
> > + # if we change libabigail, invalidate existing abi cache
> > + rm -rf reference
> > + fi
> > +
> > + if [ ! -d libabigail ]; then
> > + git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
>
> Why not using the tarball?
> http://mirrors.kernel.org/sourceware/libabigail/libabigail-1.6.tar.gz
No good reason "now".
I was first bitten by a reference to redhat-hardened-ld in some
libtool script in the tarball (/me looks in Dodji direction).
I then considered switching to different versions of libabigail by
just setting the LIBABIGAIL_VERSION env variable from .travis.yml.
I ended up with the current latest version which is also what is in
Ubuntu latest releases.
The tarball is smaller than a git clone, so best to use it.
v3 incoming.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2] ci: build and use libabigail 1.6
2020-02-18 14:29 9% ` [dpdk-dev] [PATCH v2] " David Marchand
@ 2020-02-18 15:46 0% ` Thomas Monjalon
2020-02-18 16:38 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-18 15:46 UTC (permalink / raw)
To: David Marchand
Cc: dev, aconole, christian.ehrhardt, dodji, kevin.laatz, Michael Santana
18/02/2020 15:29, David Marchand:
> libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> breakage [1].
> This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> in launchpad [3].
>
> But for now, build and use the last version 1.6 so that the ABI checks
> can be kept.
>
> 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b
> 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
I suggest few improvements below:
> --- a/.ci/linux-build.sh
> +++ b/.ci/linux-build.sh
> if [ "$ABI_CHECKS" = "1" ]; then
What do you think about moving the libabigail install in a function?
We could justify this with a comment about installing the latest version.
> + LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
> + LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
> +
> + if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
> + rm -rf libabigail
> + # if we change libabigail, invalidate existing abi cache
> + rm -rf reference
> + fi
> +
> + if [ ! -d libabigail ]; then
> + git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
Why not using the tarball?
http://mirrors.kernel.org/sourceware/libabigail/libabigail-1.6.tar.gz
> + cd libabigail/src && autoreconf -i && cd -
> + instdir=$(pwd)/libabigail
> + mkdir libabigail/src/build
> + cd libabigail/src/build && ../configure --prefix=$instdir && cd -
> + make -C libabigail/src/build all install
> +
> + rm -rf libabigail/src
> + echo $LIBABIGAIL_VERSION > libabigail/VERSION
> + fi
> +
> + export PATH=$(pwd)/libabigail/bin:$PATH
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
2020-02-18 14:55 0% ` Aaron Conole
@ 2020-02-18 15:07 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-18 15:07 UTC (permalink / raw)
To: Aaron Conole
Cc: Thomas Monjalon, dev, Christian Ehrhardt, Dodji Seketeli,
Michael Santana
On Tue, Feb 18, 2020 at 3:55 PM Aaron Conole <aconole@redhat.com> wrote:
>
> David Marchand <david.marchand@redhat.com> writes:
>
> > On Tue, Feb 18, 2020 at 10:40 AM David Marchand
> > <david.marchand@redhat.com> wrote:
> >>
> >> On Mon, Feb 17, 2020 at 7:48 PM Aaron Conole <aconole@redhat.com> wrote:
> >> >
> >> > David Marchand <david.marchand@redhat.com> writes:
> >> >
> >> > > libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> >> > > breakage [1].
> >> > > This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> >> > > in launchpad [3].
> >> > >
> >> > > But for now, build and use the last version 1.6 so that the ABI checks
> >> > > can be kept.
> >> > >
> >> > > 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> >> > > 2:
> >> > > https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
> >> > > 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
> >> > >
> >> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> >> > > ---
> >> >
> >> > Does it make sense to base libabigail required ontop of extra packages?
> >> > Otherwise some libraries won't get built / checked, no?
> >>
> >> The only change I see is the pcap driver being enabled.
> >> On the principle, I agree that trying to build all possible
> >> libraries/drivers is better when checking the ABI.
> >> So I'll keep extra_packages yes.
> >>
> >> I am currently testing that touching extra_packages (well, testing
> >> Thomas patches) results in Travis treating the job as a new one (i.e.
> >> with no cache).
> >
> > Travis bases each job cache on the job description:
> > https://docs.travis-ci.com/user/caching/
> >
> > I tested Thomas change on extra_packages content, and the job used the
> > old cache.
> > My idea was to try to put *extra_packages in an env variable, but it
> > does not work (my yaml-fu is lacking).
> >
> > If there is no easy way, I will invalidate the cache manually.
>
> We don't actually use the EXTRA_PACKAGES variable for anything, so I
> guess it's probably okay to change the value and that should invalidate
> the cache. Most of the variables, in fact, could be checked for
> non-zero value rather than a specific positive value, and then it's easy
> to invalidate the cache by just bumping them. It's a thought (and
> kindof a hack). Or we can just use the travis CLI tool and delete the
> caches (we'll have to do that for the ovsrobot as well, I think).
>
What I had in mind was to convert the extra_packages yaml thing into a
string to pass into EXTRA_PACKAGES.
But I did not manage.
About bumping the value, users are likely to be unaware of this step
if they submit a patch touching .travis.yml.
Deleting the caches from ovsrobot if .travis.yml has been touched seems simpler.
On master, I will stick to manual cache invalidation.
--
David Marchand
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
2020-02-18 11:18 0% ` David Marchand
@ 2020-02-18 14:55 0% ` Aaron Conole
2020-02-18 15:07 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Aaron Conole @ 2020-02-18 14:55 UTC (permalink / raw)
To: David Marchand
Cc: Thomas Monjalon, dev, Christian Ehrhardt, Dodji Seketeli,
Michael Santana
David Marchand <david.marchand@redhat.com> writes:
> On Tue, Feb 18, 2020 at 10:40 AM David Marchand
> <david.marchand@redhat.com> wrote:
>>
>> On Mon, Feb 17, 2020 at 7:48 PM Aaron Conole <aconole@redhat.com> wrote:
>> >
>> > David Marchand <david.marchand@redhat.com> writes:
>> >
>> > > libabigail 1.2 (at least) reports changes in 'const' property as an ABI
>> > > breakage [1].
>> > > This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
>> > > in launchpad [3].
>> > >
>> > > But for now, build and use the last version 1.6 so that the ABI checks
>> > > can be kept.
>> > >
>> > > 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
>> > > 2:
>> > > https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
>> > > 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
>> > >
>> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
>> > > ---
>> >
>> > Does it make sense to base libabigail required ontop of extra packages?
>> > Otherwise some libraries won't get built / checked, no?
>>
>> The only change I see is the pcap driver being enabled.
>> On the principle, I agree that trying to build all possible
>> libraries/drivers is better when checking the ABI.
>> So I'll keep extra_packages yes.
>>
>> I am currently testing that touching extra_packages (well, testing
>> Thomas patches) results in Travis treating the job as a new one (i.e.
>> with no cache).
>
> Travis bases each job cache on the job description:
> https://docs.travis-ci.com/user/caching/
>
> I tested Thomas change on extra_packages content, and the job used the
> old cache.
> My idea was to try to put *extra_packages in an env variable, but it
> does not work (my yaml-fu is lacking).
>
> If there is no easy way, I will invalidate the cache manually.
We don't actually use the EXTRA_PACKAGES variable for anything, so I
guess it's probably okay to change the value and that should invalidate
the cache. Most of the variables, in fact, could be checked for
non-zero value rather than a specific positive value, and then it's easy
to invalidate the cache by just bumping them. It's a thought (and
kindof a hack). Or we can just use the travis CLI tool and delete the
caches (we'll have to do that for the ovsrobot as well, I think).
>
> --
> David Marchand
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH v2] ci: build and use libabigail 1.6
2020-02-17 13:59 9% [dpdk-dev] [PATCH] ci: build and use libabigail 1.6 David Marchand
2020-02-17 15:15 0% ` Thomas Monjalon
2020-02-17 18:47 0% ` Aaron Conole
@ 2020-02-18 14:29 9% ` David Marchand
2020-02-18 15:46 0% ` Thomas Monjalon
2020-02-18 20:13 9% ` [dpdk-dev] [PATCH v3] " David Marchand
3 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-18 14:29 UTC (permalink / raw)
To: thomas
Cc: dev, aconole, christian.ehrhardt, dodji, kevin.laatz, Michael Santana
libabigail 1.2 (at least) reports changes in 'const' property as an ABI
breakage [1].
This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
in launchpad [3].
But for now, build and use the last version 1.6 so that the ABI checks
can be kept.
1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b
3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
Changelog since v1:
- keep extra_packages in ABI checks jobs,
- fix configure step,
---
.ci/linux-build.sh | 23 +++++++++++++++++++++++
.travis.yml | 8 +++++++-
2 files changed, 30 insertions(+), 1 deletion(-)
diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index c7c3840fc..cb9627bd5 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -38,6 +38,29 @@ if [ "$AARCH64" != "1" ]; then
fi
if [ "$ABI_CHECKS" = "1" ]; then
+ LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
+ LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
+
+ if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
+ rm -rf libabigail
+ # if we change libabigail, invalidate existing abi cache
+ rm -rf reference
+ fi
+
+ if [ ! -d libabigail ]; then
+ git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
+ cd libabigail/src && autoreconf -i && cd -
+ instdir=$(pwd)/libabigail
+ mkdir libabigail/src/build
+ cd libabigail/src/build && ../configure --prefix=$instdir && cd -
+ make -C libabigail/src/build all install
+
+ rm -rf libabigail/src
+ echo $LIBABIGAIL_VERSION > libabigail/VERSION
+ fi
+
+ export PATH=$(pwd)/libabigail/bin:$PATH
+
REF_GIT_REPO=${REF_GIT_REPO:-https://dpdk.org/git/dpdk}
REF_GIT_TAG=${REF_GIT_TAG:-v19.11}
diff --git a/.travis.yml b/.travis.yml
index 22539d823..030064c94 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -2,6 +2,7 @@ language: c
cache:
ccache: true
directories:
+ - libabigail
- reference
compiler:
- gcc
@@ -24,7 +25,10 @@ aarch64_packages: &aarch64_packages
extra_packages: &extra_packages
- *required_packages
- - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4, abigail-tools]
+ - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
+
+libabigail_build_packages: &libabigail_build_packages
+ - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
build_32b_packages: &build_32b_packages
- *required_packages
@@ -160,6 +164,7 @@ matrix:
apt:
packages:
- *extra_packages
+ - *libabigail_build_packages
- env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
arch: arm64
compiler: gcc
@@ -167,5 +172,6 @@ matrix:
apt:
packages:
- *extra_packages
+ - *libabigail_build_packages
script: ./.ci/${TRAVIS_OS_NAME}-build.sh
--
2.23.0
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage
2020-02-18 13:13 4% ` Andrew Rybchenko
@ 2020-02-18 14:20 4% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-18 14:20 UTC (permalink / raw)
To: Ciara Power; +Cc: Ferruh Yigit, Andrew Rybchenko, dev
18/02/2020 14:13, Andrew Rybchenko:
> On 2/18/20 2:59 PM, Ferruh Yigit wrote:
> > On 2/18/2020 10:46 AM, Ciara Power wrote:
> >> If a function is added to the eth_dev_ops struct before
> >> tx_descriptor_status function, this will cause ABI breakage. This is due
> >> to static inline functions using this function, and some other functions
> >> above it in the struct, so they cannot change position. A comment is
> >> added to inform developers of this possible breakage.
> >>
> >> Signed-off-by: Ciara Power <ciara.power@intel.com>
> >> ---
> >> lib/librte_ethdev/rte_ethdev_core.h | 3 +++
> >> 1 file changed, 3 insertions(+)
> >>
> >> diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
> >> index 7bf97e24e..1ad04a129 100644
> >> --- a/lib/librte_ethdev/rte_ethdev_core.h
> >> +++ b/lib/librte_ethdev/rte_ethdev_core.h
> >> @@ -667,6 +667,9 @@ struct eth_dev_ops {
> >> /**< Check the status of a Rx descriptor. */
> >> eth_tx_descriptor_status_t tx_descriptor_status;
> >> /**< Check the status of a Tx descriptor. */
> >> + /* Static inline functions use functions above this comment.
> >> + * New dev_ops functions should be added below to avoid breaking ABI.
> >> + */
> >> eth_rx_enable_intr_t rx_queue_intr_enable; /**< Enable Rx queue interrupt. */
> >> eth_rx_disable_intr_t rx_queue_intr_disable; /**< Disable Rx queue interrupt. */
> >> eth_tx_queue_setup_t tx_queue_setup;/**< Set up device TX queue. */
> >>
> >
> > Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
>
> LGTM, but may I suggest to make the first line in the comment
> empty to make it easier to notice. I.e.
> /*
> * Static inline functions use functions above this comment.
> * ...
> */
>
> Or even add empty lines before and after the comment.
>
> Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
In the same idea as Andrew, I would suggest some uppercases,
maybe "ABOVE" and "BELOW".
Acked-by: Thomas Monjalon <thomas@monjalon.net>
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH v2] ethdev: add comment to warn of ABI breakage
2020-02-18 10:46 13% [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage Ciara Power
2020-02-18 11:59 4% ` Ferruh Yigit
@ 2020-02-18 13:39 13% ` Ciara Power
2020-02-18 18:40 4% ` Ferruh Yigit
1 sibling, 1 reply; 200+ results
From: Ciara Power @ 2020-02-18 13:39 UTC (permalink / raw)
To: thomas, ferruh.yigit, arybchenko; +Cc: dev, Ciara Power
If a function is added to the eth_dev_ops struct before
tx_descriptor_status function, this will cause ABI breakage. This is due
to static inline functions using this function, and some other functions
above it in the struct, so they cannot change position. A comment is
added to inform developers of this possible breakage.
Signed-off-by: Ciara Power <ciara.power@intel.com>
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
---
V2: Added empty first line in comment
---
lib/librte_ethdev/rte_ethdev_core.h | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
index 7bf97e24e..9933ffdc1 100644
--- a/lib/librte_ethdev/rte_ethdev_core.h
+++ b/lib/librte_ethdev/rte_ethdev_core.h
@@ -667,6 +667,10 @@ struct eth_dev_ops {
/**< Check the status of a Rx descriptor. */
eth_tx_descriptor_status_t tx_descriptor_status;
/**< Check the status of a Tx descriptor. */
+ /*
+ * Static inline functions use functions above this comment.
+ * New dev_ops functions should be added below to avoid breaking ABI.
+ */
eth_rx_enable_intr_t rx_queue_intr_enable; /**< Enable Rx queue interrupt. */
eth_rx_disable_intr_t rx_queue_intr_disable; /**< Disable Rx queue interrupt. */
eth_tx_queue_setup_t tx_queue_setup;/**< Set up device TX queue. */
--
2.17.1
^ permalink raw reply [relevance 13%]
* Re: [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage
2020-02-18 11:59 4% ` Ferruh Yigit
@ 2020-02-18 13:13 4% ` Andrew Rybchenko
2020-02-18 14:20 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Andrew Rybchenko @ 2020-02-18 13:13 UTC (permalink / raw)
To: Ferruh Yigit, Ciara Power, thomas, arybchenko; +Cc: dev
On 2/18/20 2:59 PM, Ferruh Yigit wrote:
> On 2/18/2020 10:46 AM, Ciara Power wrote:
>> If a function is added to the eth_dev_ops struct before
>> tx_descriptor_status function, this will cause ABI breakage. This is due
>> to static inline functions using this function, and some other functions
>> above it in the struct, so they cannot change position. A comment is
>> added to inform developers of this possible breakage.
>>
>> Signed-off-by: Ciara Power <ciara.power@intel.com>
>> ---
>> lib/librte_ethdev/rte_ethdev_core.h | 3 +++
>> 1 file changed, 3 insertions(+)
>>
>> diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
>> index 7bf97e24e..1ad04a129 100644
>> --- a/lib/librte_ethdev/rte_ethdev_core.h
>> +++ b/lib/librte_ethdev/rte_ethdev_core.h
>> @@ -667,6 +667,9 @@ struct eth_dev_ops {
>> /**< Check the status of a Rx descriptor. */
>> eth_tx_descriptor_status_t tx_descriptor_status;
>> /**< Check the status of a Tx descriptor. */
>> + /* Static inline functions use functions above this comment.
>> + * New dev_ops functions should be added below to avoid breaking ABI.
>> + */
>> eth_rx_enable_intr_t rx_queue_intr_enable; /**< Enable Rx queue interrupt. */
>> eth_rx_disable_intr_t rx_queue_intr_disable; /**< Disable Rx queue interrupt. */
>> eth_tx_queue_setup_t tx_queue_setup;/**< Set up device TX queue. */
>>
>
> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
LGTM, but may I suggest to make the first line in the comment
empty to make it easier to notice. I.e.
/*
* Static inline functions use functions above this comment.
* ...
*/
Or even add empty lines before and after the comment.
Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage
2020-02-18 10:46 13% [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage Ciara Power
@ 2020-02-18 11:59 4% ` Ferruh Yigit
2020-02-18 13:13 4% ` Andrew Rybchenko
2020-02-18 13:39 13% ` [dpdk-dev] [PATCH v2] " Ciara Power
1 sibling, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-02-18 11:59 UTC (permalink / raw)
To: Ciara Power, thomas, arybchenko; +Cc: dev
On 2/18/2020 10:46 AM, Ciara Power wrote:
> If a function is added to the eth_dev_ops struct before
> tx_descriptor_status function, this will cause ABI breakage. This is due
> to static inline functions using this function, and some other functions
> above it in the struct, so they cannot change position. A comment is
> added to inform developers of this possible breakage.
>
> Signed-off-by: Ciara Power <ciara.power@intel.com>
> ---
> lib/librte_ethdev/rte_ethdev_core.h | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
> index 7bf97e24e..1ad04a129 100644
> --- a/lib/librte_ethdev/rte_ethdev_core.h
> +++ b/lib/librte_ethdev/rte_ethdev_core.h
> @@ -667,6 +667,9 @@ struct eth_dev_ops {
> /**< Check the status of a Rx descriptor. */
> eth_tx_descriptor_status_t tx_descriptor_status;
> /**< Check the status of a Tx descriptor. */
> + /* Static inline functions use functions above this comment.
> + * New dev_ops functions should be added below to avoid breaking ABI.
> + */
> eth_rx_enable_intr_t rx_queue_intr_enable; /**< Enable Rx queue interrupt. */
> eth_rx_disable_intr_t rx_queue_intr_disable; /**< Disable Rx queue interrupt. */
> eth_tx_queue_setup_t tx_queue_setup;/**< Set up device TX queue. */
>
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
2020-02-18 9:40 3% ` David Marchand
@ 2020-02-18 11:18 0% ` David Marchand
2020-02-18 14:55 0% ` Aaron Conole
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-18 11:18 UTC (permalink / raw)
To: Aaron Conole
Cc: Thomas Monjalon, dev, Christian Ehrhardt, Dodji Seketeli,
Michael Santana
On Tue, Feb 18, 2020 at 10:40 AM David Marchand
<david.marchand@redhat.com> wrote:
>
> On Mon, Feb 17, 2020 at 7:48 PM Aaron Conole <aconole@redhat.com> wrote:
> >
> > David Marchand <david.marchand@redhat.com> writes:
> >
> > > libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> > > breakage [1].
> > > This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> > > in launchpad [3].
> > >
> > > But for now, build and use the last version 1.6 so that the ABI checks
> > > can be kept.
> > >
> > > 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> > > 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
> > > 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
> > >
> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > > ---
> >
> > Does it make sense to base libabigail required ontop of extra packages?
> > Otherwise some libraries won't get built / checked, no?
>
> The only change I see is the pcap driver being enabled.
> On the principle, I agree that trying to build all possible
> libraries/drivers is better when checking the ABI.
> So I'll keep extra_packages yes.
>
> I am currently testing that touching extra_packages (well, testing
> Thomas patches) results in Travis treating the job as a new one (i.e.
> with no cache).
Travis bases each job cache on the job description:
https://docs.travis-ci.com/user/caching/
I tested Thomas change on extra_packages content, and the job used the
old cache.
My idea was to try to put *extra_packages in an env variable, but it
does not work (my yaml-fu is lacking).
If there is no easy way, I will invalidate the cache manually.
--
David Marchand
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage
@ 2020-02-18 10:46 13% Ciara Power
2020-02-18 11:59 4% ` Ferruh Yigit
2020-02-18 13:39 13% ` [dpdk-dev] [PATCH v2] " Ciara Power
0 siblings, 2 replies; 200+ results
From: Ciara Power @ 2020-02-18 10:46 UTC (permalink / raw)
To: thomas, ferruh.yigit, arybchenko; +Cc: dev, Ciara Power
If a function is added to the eth_dev_ops struct before
tx_descriptor_status function, this will cause ABI breakage. This is due
to static inline functions using this function, and some other functions
above it in the struct, so they cannot change position. A comment is
added to inform developers of this possible breakage.
Signed-off-by: Ciara Power <ciara.power@intel.com>
---
lib/librte_ethdev/rte_ethdev_core.h | 3 +++
1 file changed, 3 insertions(+)
diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
index 7bf97e24e..1ad04a129 100644
--- a/lib/librte_ethdev/rte_ethdev_core.h
+++ b/lib/librte_ethdev/rte_ethdev_core.h
@@ -667,6 +667,9 @@ struct eth_dev_ops {
/**< Check the status of a Rx descriptor. */
eth_tx_descriptor_status_t tx_descriptor_status;
/**< Check the status of a Tx descriptor. */
+ /* Static inline functions use functions above this comment.
+ * New dev_ops functions should be added below to avoid breaking ABI.
+ */
eth_rx_enable_intr_t rx_queue_intr_enable; /**< Enable Rx queue interrupt. */
eth_rx_disable_intr_t rx_queue_intr_disable; /**< Disable Rx queue interrupt. */
eth_tx_queue_setup_t tx_queue_setup;/**< Set up device TX queue. */
--
2.17.1
^ permalink raw reply [relevance 13%]
* [dpdk-dev] [PATCH v4] doc: alias to experimental tag for stable apis
2020-02-05 15:17 17% [dpdk-dev] [PATCH] doc: alias to experimental tag for stable apis Ray Kinsella
2020-02-05 15:43 17% ` [dpdk-dev] [PATCH v3] " Ray Kinsella
@ 2020-02-18 10:46 17% ` Ray Kinsella
1 sibling, 0 replies; 200+ results
From: Ray Kinsella @ 2020-02-18 10:46 UTC (permalink / raw)
To: dev
Cc: john.mcnamara, nhorman, ferruh.yigit, marko.kovacevic,
bruce.richardson, Ray Kinsella
When a maintainer is promoting an API to become part of the next major ABI
version by removing the experimental tag, possibly a few releases in
advance of the declaration of the next ABI version. The maintainer may
choose to offer an alias to the experimental tag, as removing the tag
before the declaration of the next major ABI version, would cause an ABI
breakage for applications using the API.
Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
---
doc/guides/contributing/abi_policy.rst | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/doc/guides/contributing/abi_policy.rst b/doc/guides/contributing/abi_policy.rst
index 05ca959..7ae7de7 100644
--- a/doc/guides/contributing/abi_policy.rst
+++ b/doc/guides/contributing/abi_policy.rst
@@ -159,6 +159,11 @@ The requirements for changing the ABI are:
``experimental``, as described in the section on :ref:`Experimental APIs
and Libraries <experimental_apis>`.
+ - In situations in which an ``experimental`` API has been stable for some
+ time. When promoting the API to become part of the next ABI version, the
+ maintainer may choose to provide an alias to the ``experimental`` tag, so
+ as not to break consuming applications.
+
#. If a newly proposed API functionally replaces an existing one, when the new
API becomes non-experimental, then the old one is marked with
``__rte_deprecated``.
@@ -317,6 +322,11 @@ not required. Though, an API should remain in experimental state for at least
one release. Thereafter, the normal process of posting patch for review to
mailing list can be followed.
+After the experimental tag has been formally removed, a tree/sub-tree maintainer
+may choose to offer an alias to the experimental tag so as not to break
+applications using the API. The alias is then dropped at the declaration of next
+major ABI version.
+
Libraries
~~~~~~~~~
--
2.7.4
^ permalink raw reply [relevance 17%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-18 9:42 7% ` [dpdk-dev] ABI version of " Bruce Richardson
@ 2020-02-18 9:50 4% ` Thomas Monjalon
2020-02-18 10:36 4% ` Kinsella, Ray
2020-02-19 11:43 7% ` [dpdk-dev] ABI version of experimental libraries Neil Horman
0 siblings, 2 replies; 200+ results
From: Thomas Monjalon @ 2020-02-18 9:50 UTC (permalink / raw)
To: Bruce Richardson
Cc: ray.kinsella, nhorman, bluca, david.marchand, ktraynor, dev
18/02/2020 10:42, Bruce Richardson:
> On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
> > Hi,
> >
> > I would like to remind everybody our mistake when defining ABI versions.
> > It has been "fixed" in this commit:
> > http://git.dpdk.org/dpdk/commit/?id=f26c2b39
> >
> > Please let's think about the consequence for the experimental libraries.
> >
> > In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
> > 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
> > increasing, that's fine. When we'll switch to the new major ABI and use
> > a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
> > soname 0.21 Numbers are dropping.
> >
> > In short, for experimental libs, ABI 20.1 > ABI 21.0
> >
> > Are we OK with this or do we prefer reverting to normal numbering for
> > experimental libraries in DPDK 20.02?
> >
> Personally, I would not be too concerned about the verions of experimental
> libs, so long as they don't conflict across versions and have some
> similarity to the major ABI version for the release.
You think sorting of the version numbers is not important?
If we don't care comparing experimental version numbers,
then OK, let's drop this patch. But please we need a small vote.
Note: there would be no problem if we did not vote for having
a special numbering for pure experimental libraries (I am still against).
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] build: fix soname for experimental libraries
2020-02-18 9:40 0% ` Bruce Richardson
@ 2020-02-18 9:47 0% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-18 9:47 UTC (permalink / raw)
To: Bruce Richardson
Cc: dev, ray.kinsella, nhorman, bluca, david.marchand, ktraynor
18/02/2020 10:40, Bruce Richardson:
> On Tue, Feb 18, 2020 at 12:44:02AM +0100, Thomas Monjalon wrote:
> > Because of an original mistake in ABI numbering,
> > and a temporary workaround for ABI 20,
> > for experimental libs, numbering would lead to consider
> > ABI 20.1 > ABI 21.0
> >
> > Before this patch:
> >
> > DPDK 19.11: ABI version 0.200 and soname 0.20
> > DPDK 20.02: ABI version 0.2001 and soname 0.201
> > Numbers are increasing, that's fine.
> > For the next major ABI, back to normal numbering:
> > DPDK 20.11: ABI version 0.210 and soname 0.21
> > Numbers are decreasing!
> >
> > After this patch:
> >
> > DPDK 19.11: ABI version 0.200 and soname 0.20
> > DPDK 20.02: ABI version 0.201 and soname 0.20
> > DPDK 20.11: ABI version 0.210 and soname 0.21
> >
> > Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
> >
> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > ---
> > config/meson.build | 8 ++++----
> > mk/rte.lib.mk | 4 ++--
> > 2 files changed, 6 insertions(+), 6 deletions(-)
> >
> > diff --git a/config/meson.build b/config/meson.build
> > index 6c46767e3e..e7cd74e2c2 100644
> > --- a/config/meson.build
> > +++ b/config/meson.build
> > @@ -27,12 +27,12 @@ abi_version = run_command(find_program('cat', 'more'),
> > # and the filename suffix as 0.majorminor versions,
> > # e.g. v20.1 => librte_stable.so.20.1, librte_experimental.so.0.201
> > # sonames => librte_stable.so.20, librte_experimental.so.0.20
> > -# e.g. v20.0.1 => librte_stable.so.20.0.1, librte_experimental.so.0.2001
> > -# sonames => librte_stable.so.20.0, librte_experimental.so.0.200
> > +# e.g. v20.0.1 => librte_stable.so.20.0.1, librte_experimental.so.0.201
> > +# sonames => librte_stable.so.20.0, librte_experimental.so.0.20
> > abi_va = abi_version.split('.')
> > stable_so_version = abi_va.length() == 2 ? abi_va[0] : abi_va[0] + '.' + abi_va[1]
> > -experimental_abi_version = '0.' + ''.join(abi_va)
> > -experimental_so_version = '0.' + ''.join(stable_so_version.split('.'))
> > +experimental_abi_version = '0.' + ''.join([abi_va[0], abi_va[2]])
> > +experimental_so_version = '0.' + ''.join([abi_va[0]])
> >
>
> My concern about this is that it will break, or rather need to be changed
> again for the 20.11 release. While I see the numbering as not-ideal in
> terms of version numbers, the existing scheme was originally designed to
> work with either 3-digit or 2-digit version numbers.
It could be improved to work with 2-digit too.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] ABI version of experimental libraries
2020-02-17 23:15 9% [dpdk-dev] ABI version of experimental libraries Thomas Monjalon
2020-02-17 23:44 11% ` [dpdk-dev] [PATCH] build: fix soname for " Thomas Monjalon
@ 2020-02-18 9:42 7% ` Bruce Richardson
2020-02-18 9:50 4% ` Thomas Monjalon
1 sibling, 1 reply; 200+ results
From: Bruce Richardson @ 2020-02-18 9:42 UTC (permalink / raw)
To: Thomas Monjalon
Cc: ray.kinsella, nhorman, bluca, david.marchand, ktraynor, dev
On Tue, Feb 18, 2020 at 12:15:56AM +0100, Thomas Monjalon wrote:
> Hi,
>
> I would like to remind everybody our mistake when defining ABI versions.
> It has been "fixed" in this commit:
> http://git.dpdk.org/dpdk/commit/?id=f26c2b39
>
> Please let's think about the consequence for the experimental libraries.
>
> In DPDK 19.11, we use the ABI version 0.200 with soname 0.20 In DPDK
> 20.02, we use the ABI version 0.2001 with soname 0.201 Numbers are
> increasing, that's fine. When we'll switch to the new major ABI and use
> a normal numbering: In DPDK 20.11, we will use the ABI version 0.210 with
> soname 0.21 Numbers are dropping.
>
> In short, for experimental libs, ABI 20.1 > ABI 21.0
>
> Are we OK with this or do we prefer reverting to normal numbering for
> experimental libraries in DPDK 20.02?
>
Personally, I would not be too concerned about the verions of experimental
libs, so long as they don't conflict across versions and have some
similarity to the major ABI version for the release.
/Bruce
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
2020-02-17 18:47 0% ` Aaron Conole
@ 2020-02-18 9:40 3% ` David Marchand
2020-02-18 11:18 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-18 9:40 UTC (permalink / raw)
To: Aaron Conole
Cc: Thomas Monjalon, dev, Christian Ehrhardt, Dodji Seketeli,
Michael Santana
On Mon, Feb 17, 2020 at 7:48 PM Aaron Conole <aconole@redhat.com> wrote:
>
> David Marchand <david.marchand@redhat.com> writes:
>
> > libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> > breakage [1].
> > This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> > in launchpad [3].
> >
> > But for now, build and use the last version 1.6 so that the ABI checks
> > can be kept.
> >
> > 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> > 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
> > 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
>
> Does it make sense to base libabigail required ontop of extra packages?
> Otherwise some libraries won't get built / checked, no?
The only change I see is the pcap driver being enabled.
On the principle, I agree that trying to build all possible
libraries/drivers is better when checking the ABI.
So I'll keep extra_packages yes.
I am currently testing that touching extra_packages (well, testing
Thomas patches) results in Travis treating the job as a new one (i.e.
with no cache).
>
> > .ci/linux-build.sh | 22 ++++++++++++++++++++++
> > .travis.yml | 15 ++++++++++-----
> > 2 files changed, 32 insertions(+), 5 deletions(-)
> >
> > diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
> > index c7c3840fc..0d4bc9a62 100755
> > --- a/.ci/linux-build.sh
> > +++ b/.ci/linux-build.sh
> > @@ -38,6 +38,28 @@ if [ "$AARCH64" != "1" ]; then
> > fi
> >
> > if [ "$ABI_CHECKS" = "1" ]; then
> > + LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
> > + LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
> > +
> > + if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
> > + rm -rf libabigail
> > + # if we change libabigail, invalidate existing abi cache
> > + rm -rf reference
> > + fi
> > +
> > + if [ ! -d libabigail ]; then
> > + git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
> > + cd libabigail/src && autoconf -vfi && cd -
And I managed to send a "oh yeah, seems better this way"/untested version...
Fixed.
--
David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] build: fix soname for experimental libraries
2020-02-17 23:44 11% ` [dpdk-dev] [PATCH] build: fix soname for " Thomas Monjalon
@ 2020-02-18 9:40 0% ` Bruce Richardson
2020-02-18 9:47 0% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Bruce Richardson @ 2020-02-18 9:40 UTC (permalink / raw)
To: Thomas Monjalon
Cc: dev, ray.kinsella, nhorman, bluca, david.marchand, ktraynor
On Tue, Feb 18, 2020 at 12:44:02AM +0100, Thomas Monjalon wrote:
> Because of an original mistake in ABI numbering,
> and a temporary workaround for ABI 20,
> for experimental libs, numbering would lead to consider
> ABI 20.1 > ABI 21.0
>
> Before this patch:
>
> DPDK 19.11: ABI version 0.200 and soname 0.20
> DPDK 20.02: ABI version 0.2001 and soname 0.201
> Numbers are increasing, that's fine.
> For the next major ABI, back to normal numbering:
> DPDK 20.11: ABI version 0.210 and soname 0.21
> Numbers are decreasing!
>
> After this patch:
>
> DPDK 19.11: ABI version 0.200 and soname 0.20
> DPDK 20.02: ABI version 0.201 and soname 0.20
> DPDK 20.11: ABI version 0.210 and soname 0.21
>
> Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
>
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---
> config/meson.build | 8 ++++----
> mk/rte.lib.mk | 4 ++--
> 2 files changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/config/meson.build b/config/meson.build
> index 6c46767e3e..e7cd74e2c2 100644
> --- a/config/meson.build
> +++ b/config/meson.build
> @@ -27,12 +27,12 @@ abi_version = run_command(find_program('cat', 'more'),
> # and the filename suffix as 0.majorminor versions,
> # e.g. v20.1 => librte_stable.so.20.1, librte_experimental.so.0.201
> # sonames => librte_stable.so.20, librte_experimental.so.0.20
> -# e.g. v20.0.1 => librte_stable.so.20.0.1, librte_experimental.so.0.2001
> -# sonames => librte_stable.so.20.0, librte_experimental.so.0.200
> +# e.g. v20.0.1 => librte_stable.so.20.0.1, librte_experimental.so.0.201
> +# sonames => librte_stable.so.20.0, librte_experimental.so.0.20
> abi_va = abi_version.split('.')
> stable_so_version = abi_va.length() == 2 ? abi_va[0] : abi_va[0] + '.' + abi_va[1]
> -experimental_abi_version = '0.' + ''.join(abi_va)
> -experimental_so_version = '0.' + ''.join(stable_so_version.split('.'))
> +experimental_abi_version = '0.' + ''.join([abi_va[0], abi_va[2]])
> +experimental_so_version = '0.' + ''.join([abi_va[0]])
>
My concern about this is that it will break, or rather need to be changed
again for the 20.11 release. While I see the numbering as not-ideal in
terms of version numbers, the existing scheme was originally designed to
work with either 3-digit or 2-digit version numbers.
/Bruce
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
2020-02-17 15:38 3% [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct Ferruh Yigit
2020-02-18 5:07 0% ` Jerin Jacob
@ 2020-02-18 6:01 0% ` Stephen Hemminger
2020-02-21 10:40 0% ` Ferruh Yigit
1 sibling, 1 reply; 200+ results
From: Stephen Hemminger @ 2020-02-18 6:01 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Neil Horman, John McNamara, Marko Kovacevic, dev, David Marchand,
Thomas Monjalon, Andrew Rybchenko
On Mon, 17 Feb 2020 15:38:05 +0000
Ferruh Yigit <ferruh.yigit@intel.com> wrote:
> For the ABI compatibility it is better to hide internal data structures
> from the application as much as possible. But because of some inline
> functions 'struct eth_dev_ops' can't be hidden completely.
>
> Plan is to split the 'struct eth_dev_ops' into two as ones used by
> inline functions and ones not used, and hide the second part that not
> used by inline functions completely to the application.
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> Cc: David Marchand <david.marchand@redhat.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>
> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
> ---
> doc/guides/rel_notes/deprecation.rst | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index dfcca87ab..2aa431028 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -72,6 +72,12 @@ Deprecation Notices
> In 19.11 PMDs will still update the field even when the offload is not
> enabled.
>
> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
> + access the struct directly. The struct will be separate in two, the ops used
> + by inline functions still will be accessible to user but rest will be hidden.
> +
> * cryptodev: support for using IV with all sizes is added, J0 still can
> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
Good luck, truely hiding internals is hard. The mbuf structure is already split but not really
hidden at all (just look at dwarf output). It doesn't make sense to do it unless
you can really hide it.
I would attack the rte_device stuff first. Make rte_device opaque to the application
that would help for future versions. Then work backwards to rte_tehtdev.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
2020-02-17 15:38 3% [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct Ferruh Yigit
@ 2020-02-18 5:07 0% ` Jerin Jacob
2020-02-18 6:01 0% ` Stephen Hemminger
1 sibling, 0 replies; 200+ results
From: Jerin Jacob @ 2020-02-18 5:07 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Neil Horman, John McNamara, Marko Kovacevic, dpdk-dev,
David Marchand, Thomas Monjalon, Andrew Rybchenko
On Mon, Feb 17, 2020 at 9:08 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> For the ABI compatibility it is better to hide internal data structures
> from the application as much as possible. But because of some inline
> functions 'struct eth_dev_ops' can't be hidden completely.
>
> Plan is to split the 'struct eth_dev_ops' into two as ones used by
> inline functions and ones not used, and hide the second part that not
> used by inline functions completely to the application.
It is a good improvement. IMO, If anything used in fast-path it
should be in ``struct rte_eth_dev``
and rest can completely be moved to internal. In this case, if
`rte_eth_tx_descriptor_status`
not used on fastpath, Maybe we don't need to maintain the inline
status and move completely
to .c file.
Those may be specifics of the work. In general, this change looks good to me.
Acked-by: Jerin Jacob <jerinj@marvell.com>
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> Cc: David Marchand <david.marchand@redhat.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>
> Cc: Andrew Rybchenko <arybchenko@solarflare.com>
> ---
> doc/guides/rel_notes/deprecation.rst | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index dfcca87ab..2aa431028 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -72,6 +72,12 @@ Deprecation Notices
> In 19.11 PMDs will still update the field even when the offload is not
> enabled.
>
> +* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
> + Currently the ``struct eth_dev_ops`` struct is accessible by the application
> + because some inline functions, like ``rte_eth_tx_descriptor_status()``,
> + access the struct directly. The struct will be separate in two, the ops used
> + by inline functions still will be accessible to user but rest will be hidden.
> +
> * cryptodev: support for using IV with all sizes is added, J0 still can
> be used but only when IV length in following structs ``rte_crypto_auth_xform``,
> ``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
> --
> 2.24.1
>
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] build: fix soname for experimental libraries
2020-02-17 23:15 9% [dpdk-dev] ABI version of experimental libraries Thomas Monjalon
@ 2020-02-17 23:44 11% ` Thomas Monjalon
2020-02-18 9:40 0% ` Bruce Richardson
2020-02-18 9:42 7% ` [dpdk-dev] ABI version of " Bruce Richardson
1 sibling, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-17 23:44 UTC (permalink / raw)
To: dev
Cc: ray.kinsella, nhorman, bluca, david.marchand, ktraynor, Bruce Richardson
Because of an original mistake in ABI numbering,
and a temporary workaround for ABI 20,
for experimental libs, numbering would lead to consider
ABI 20.1 > ABI 21.0
Before this patch:
DPDK 19.11: ABI version 0.200 and soname 0.20
DPDK 20.02: ABI version 0.2001 and soname 0.201
Numbers are increasing, that's fine.
For the next major ABI, back to normal numbering:
DPDK 20.11: ABI version 0.210 and soname 0.21
Numbers are decreasing!
After this patch:
DPDK 19.11: ABI version 0.200 and soname 0.20
DPDK 20.02: ABI version 0.201 and soname 0.20
DPDK 20.11: ABI version 0.210 and soname 0.21
Fixes: f26c2b39b271 ("build: fix soname info for 19.11 compatibility")
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
config/meson.build | 8 ++++----
mk/rte.lib.mk | 4 ++--
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/config/meson.build b/config/meson.build
index 6c46767e3e..e7cd74e2c2 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -27,12 +27,12 @@ abi_version = run_command(find_program('cat', 'more'),
# and the filename suffix as 0.majorminor versions,
# e.g. v20.1 => librte_stable.so.20.1, librte_experimental.so.0.201
# sonames => librte_stable.so.20, librte_experimental.so.0.20
-# e.g. v20.0.1 => librte_stable.so.20.0.1, librte_experimental.so.0.2001
-# sonames => librte_stable.so.20.0, librte_experimental.so.0.200
+# e.g. v20.0.1 => librte_stable.so.20.0.1, librte_experimental.so.0.201
+# sonames => librte_stable.so.20.0, librte_experimental.so.0.20
abi_va = abi_version.split('.')
stable_so_version = abi_va.length() == 2 ? abi_va[0] : abi_va[0] + '.' + abi_va[1]
-experimental_abi_version = '0.' + ''.join(abi_va)
-experimental_so_version = '0.' + ''.join(stable_so_version.split('.'))
+experimental_abi_version = '0.' + ''.join([abi_va[0], abi_va[2]])
+experimental_so_version = '0.' + ''.join([abi_va[0]])
# extract all version information into the build configuration
dpdk_conf.set('RTE_VER_YEAR', pver.get(0).to_int())
diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
index b1a8372cc2..8730c084ce 100644
--- a/mk/rte.lib.mk
+++ b/mk/rte.lib.mk
@@ -15,8 +15,8 @@ LIBABIVER ?= $(shell cat $(RTE_SRCDIR)/ABI_VERSION)
SOVER := $(basename $(LIBABIVER))
ifeq ($(shell grep -s "^DPDK_" $(SRCDIR)/$(EXPORT_MAP)),)
# EXPERIMENTAL ABI is versioned as 0.major+minor, e.g. 0.201 for 20.1 ABI
-LIBABIVER := 0.$(shell echo $(LIBABIVER) | tr -d '.')
-SOVER := 0.$(shell echo $(SOVER) | tr -d '.')
+LIBABIVER := 0.$(shell echo $(LIBABIVER) | cut -d '.' -f1,3 | tr -d '.')
+SOVER := 0.$(shell echo $(SOVER) | cut -d '.' -f1)
endif
ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
--
2.25.0
^ permalink raw reply [relevance 11%]
* [dpdk-dev] ABI version of experimental libraries
@ 2020-02-17 23:15 9% Thomas Monjalon
2020-02-17 23:44 11% ` [dpdk-dev] [PATCH] build: fix soname for " Thomas Monjalon
2020-02-18 9:42 7% ` [dpdk-dev] ABI version of " Bruce Richardson
0 siblings, 2 replies; 200+ results
From: Thomas Monjalon @ 2020-02-17 23:15 UTC (permalink / raw)
To: ray.kinsella, nhorman
Cc: bruce.richardson, bluca, david.marchand, ktraynor, dev
Hi,
I would like to remind everybody our mistake when defining ABI versions.
It has been "fixed" in this commit:
http://git.dpdk.org/dpdk/commit/?id=f26c2b39
Please let's think about the consequence for the experimental libraries.
In DPDK 19.11, we use the ABI version 0.200 with soname 0.20
In DPDK 20.02, we use the ABI version 0.2001 with soname 0.201
Numbers are increasing, that's fine.
When we'll switch to the new major ABI and use a normal numbering:
In DPDK 20.11, we will use the ABI version 0.210 with soname 0.21
Numbers are dropping.
In short, for experimental libs, ABI 20.1 > ABI 21.0
Are we OK with this or do we prefer reverting to normal numbering
for experimental libraries in DPDK 20.02?
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
2020-02-17 13:59 9% [dpdk-dev] [PATCH] ci: build and use libabigail 1.6 David Marchand
2020-02-17 15:15 0% ` Thomas Monjalon
@ 2020-02-17 18:47 0% ` Aaron Conole
2020-02-18 9:40 3% ` David Marchand
2020-02-18 14:29 9% ` [dpdk-dev] [PATCH v2] " David Marchand
2020-02-18 20:13 9% ` [dpdk-dev] [PATCH v3] " David Marchand
3 siblings, 1 reply; 200+ results
From: Aaron Conole @ 2020-02-17 18:47 UTC (permalink / raw)
To: David Marchand; +Cc: thomas, dev, christian.ehrhardt, dodji, Michael Santana
David Marchand <david.marchand@redhat.com> writes:
> libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> breakage [1].
> This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> in launchpad [3].
>
> But for now, build and use the last version 1.6 so that the ABI checks
> can be kept.
>
> 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
> 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
Does it make sense to base libabigail required ontop of extra packages?
Otherwise some libraries won't get built / checked, no?
> .ci/linux-build.sh | 22 ++++++++++++++++++++++
> .travis.yml | 15 ++++++++++-----
> 2 files changed, 32 insertions(+), 5 deletions(-)
>
> diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
> index c7c3840fc..0d4bc9a62 100755
> --- a/.ci/linux-build.sh
> +++ b/.ci/linux-build.sh
> @@ -38,6 +38,28 @@ if [ "$AARCH64" != "1" ]; then
> fi
>
> if [ "$ABI_CHECKS" = "1" ]; then
> + LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
> + LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
> +
> + if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
> + rm -rf libabigail
> + # if we change libabigail, invalidate existing abi cache
> + rm -rf reference
> + fi
> +
> + if [ ! -d libabigail ]; then
> + git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
> + cd libabigail/src && autoconf -vfi && cd -
> + mkdir libabigail/src/build
> + cd libabigail/src/build && ../configure --prefix=$(pwd)/libabigail && cd -
> + make -C libabigail/src/build all install
> +
> + rm -rf libabigail/src
> + echo $LIBABIGAIL_VERSION > libabigail/VERSION
> + fi
> +
> + export PATH=$(pwd)/libabigail/bin:$PATH
> +
> REF_GIT_REPO=${REF_GIT_REPO:-https://dpdk.org/git/dpdk}
> REF_GIT_TAG=${REF_GIT_TAG:-v19.11}
>
> diff --git a/.travis.yml b/.travis.yml
> index 22539d823..d8253fdd4 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -2,6 +2,7 @@ language: c
> cache:
> ccache: true
> directories:
> + - libabigail
> - reference
> compiler:
> - gcc
> @@ -24,7 +25,11 @@ aarch64_packages: &aarch64_packages
>
> extra_packages: &extra_packages
> - *required_packages
> - - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4, abigail-tools]
> + - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +
> +libabigail_build_packages: &libabigail_build_packages
> + - *required_packages
> + - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
>
> build_32b_packages: &build_32b_packages
> - *required_packages
> @@ -154,18 +159,18 @@ matrix:
> packages:
> - *required_packages
> - *doc_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> + - env: DEF_LIB="shared" ABI_CHECKS=1
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> + - *libabigail_build_packages
> + - env: DEF_LIB="shared" ABI_CHECKS=1
> arch: arm64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *libabigail_build_packages
>
> script: ./.ci/${TRAVIS_OS_NAME}-build.sh
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] net/ionic: fix ABI version
@ 2020-02-17 16:13 7% Ferruh Yigit
2020-02-19 12:37 4% ` Ferruh Yigit
0 siblings, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-02-17 16:13 UTC (permalink / raw)
To: Alfredo Cardigliano, Shannon Nelson; +Cc: dev, Ferruh Yigit, David Marchand
Changed the ABI version to 20.0.1.
Fixes: a72cf98cc93e ("net/ionic: add skeleton")
Reported-by: David Marchand <david.marchand@redhat.com>
Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
Cc: Alfredo Cardigliano <cardigliano@ntop.org>
---
drivers/net/ionic/rte_pmd_ionic_version.map | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/drivers/net/ionic/rte_pmd_ionic_version.map b/drivers/net/ionic/rte_pmd_ionic_version.map
index 3fb4ff7eb..bc8fd6d4d 100644
--- a/drivers/net/ionic/rte_pmd_ionic_version.map
+++ b/drivers/net/ionic/rte_pmd_ionic_version.map
@@ -1,5 +1,4 @@
-DPDK_20.0 {
+DPDK_20.0.1 {
local: *;
};
-
--
2.24.1
^ permalink raw reply [relevance 7%]
* [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct
@ 2020-02-17 15:38 3% Ferruh Yigit
2020-02-18 5:07 0% ` Jerin Jacob
2020-02-18 6:01 0% ` Stephen Hemminger
0 siblings, 2 replies; 200+ results
From: Ferruh Yigit @ 2020-02-17 15:38 UTC (permalink / raw)
To: Neil Horman, John McNamara, Marko Kovacevic
Cc: dev, Ferruh Yigit, David Marchand, Thomas Monjalon, Andrew Rybchenko
For the ABI compatibility it is better to hide internal data structures
from the application as much as possible. But because of some inline
functions 'struct eth_dev_ops' can't be hidden completely.
Plan is to split the 'struct eth_dev_ops' into two as ones used by
inline functions and ones not used, and hide the second part that not
used by inline functions completely to the application.
Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
Cc: David Marchand <david.marchand@redhat.com>
Cc: Thomas Monjalon <thomas@monjalon.net>
Cc: Andrew Rybchenko <arybchenko@solarflare.com>
---
doc/guides/rel_notes/deprecation.rst | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index dfcca87ab..2aa431028 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -72,6 +72,12 @@ Deprecation Notices
In 19.11 PMDs will still update the field even when the offload is not
enabled.
+* ethdev: Split the ``struct eth_dev_ops`` struct to hide it as much as possible.
+ Currently the ``struct eth_dev_ops`` struct is accessible by the application
+ because some inline functions, like ``rte_eth_tx_descriptor_status()``,
+ access the struct directly. The struct will be separate in two, the ops used
+ by inline functions still will be accessible to user but rest will be hidden.
+
* cryptodev: support for using IV with all sizes is added, J0 still can
be used but only when IV length in following structs ``rte_crypto_auth_xform``,
``rte_crypto_aead_xform`` is set to zero. When IV length is greater or equal
--
2.24.1
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-17 14:23 8% ` Ray Kinsella
@ 2020-02-17 15:37 4% ` Neil Horman
0 siblings, 0 replies; 200+ results
From: Neil Horman @ 2020-02-17 15:37 UTC (permalink / raw)
To: Ray Kinsella
Cc: Bruce Richardson, Luca Boccassi, Ferruh Yigit,
Cristian Dumitrescu, Eelco Chaudron, dev, Thomas Monjalon,
David Marchand, Ian Stokes
On Mon, Feb 17, 2020 at 02:23:15PM +0000, Ray Kinsella wrote:
>
>
> On 14/02/2020 20:48, Neil Horman wrote:
> > On Fri, Feb 14, 2020 at 11:36:34AM +0000, Bruce Richardson wrote:
> >> On Thu, Feb 13, 2020 at 09:40:40PM -0500, Neil Horman wrote:
> >>> On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
> >>>>
> >>>>
> >>>> On 05/02/2020 11:32, Neil Horman wrote:
> >>>>> On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
> >>>>>> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
> >>>>>>>> But if we can do the versioning in the master, LTS can backport it
> >>>>>>>> and can have mature version of that API in LTS without breaking
> >>>>>>>> the existing users.
> >>>>>>>>
> >>>>>>>
> >>>>>>> But why bother? The only thing you've changed is the version
> >>>>>>> tagging. Its ok to leave that alone in LTS, you just cant change
> >>>>>>> it.
> >>>>>>>
> >>>>>>> Thats part of the burden of an LTS release, it will have some drift
> >>>>>>> from the upstream head, because you have to keep things stable. So
> >>>>>>> you stabilize the upstream ABI version for this API and just never
> >>>>>>> backport it to the current LTS release.
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> A customer (OVS) explicitly and specifically requested backporting
> >>>>>> the symbol change to 19.11, as they don't want to enable
> >>>>>> experimental APIs in their releases. I replied that it would only be
> >>>>>> acceptable with aliasing to keep compatibility, and Ferruh very
> >>>>>> kindly did the work to implement that.
> >>>>>>
> >>>>> but, thats part of the agreement, no? You can't always have new
> >>>>> features and stability at the same time.
> >>>>>
> >>>>> I get that this is an odd corner case, because strictly speaking you
> >>>>> could waive the ABI change that libabigail is reporting, and most
> >>>>> application users (like OVS) could get away with it, because their
> >>>>> application does all the right things to make it ok, but I don't
> >>>>> think you can make a decsion like this for all users based on the
> >>>>> request of a single user.
> >>>>>
> >>>>> It seems like the right thing is for OVS to augment their build time
> >>>>> configuration to allow developers to select the ability to use
> >>>>> experimental apis at compile time, and then the decision is placed in
> >>>>> the hands of end users.
> >>>>
> >>>> So this is not isolated case right ... it is common from API's to
> >>>> graduate from experimental to mature, we _want_ that to happen, we
> >>>> _want_ APIs to mature.
> >>>>
> >>> Sure, I can absolutely agree with that, though I would suggest that the
> >>> maturity of the ABI is orthogonal to its labeling as such (more on that
> >>> below)
> >>>
> >>>> I am worried what you are proposing will encourage a behavior whereby
> >>>> maintainers to wait until the declaration of the next major ABI version
> >>>> to make these symbol changes, causing these kinds of changes to queue
> >>>> up for that release.
> >>>>
> >>> I think you're probably right about that, and would make the agrument
> >>> that thats perfectly fine (again I'll clarify below)
> >>>
> >>>> And you would have to ask why? In the case of a reasonably mature API,
> >>>> there will be no functional or signature change - its mature! So what
> >>>> is the harm of providing an alias back to Experimental until the next
> >>>> ABI version is declared?
> >>>>
> >>> From a philosophical standpoint, there is absoluely no harm, and I don't
> >>> think anyone would disagree, the harm comes from the details of the
> >>> implementation, as you've noted.
> >>>
> >>>> So while yes, you are 100% right - experimental mean no guarantees.
> >>>> But if the API is baked, it is not going to change, so I don't see the
> >>>> harm.
> >>>>
> >>>> We don't want the unnecessary triumph of policy over pragmatism.
> >>>>
> >>> I would make the converse argument here. While I agree that when an API
> >>> is mature, theres no point in calling it experimental anymore, I would
> >>> also suggest that, if an API is mature, and not expected to change,
> >>> theres no harm in leaving its version tag as experimental for an LTS
> >>> release. This is what I was referring to above. Once an application
> >>> developer has done the work to integrate an API from DPDK into its
> >>> application, that work is done. If the API remains stable, then I
> >>> honestly don't know that they care that the version label is EXPERIMENTAL
> >>> versus 20.11 or 20.05 or whatever. They care about the API being stable
> >>> more so than its version label. As such, it seems....reasonable to me to
> >>> have developers queue their migration of experimental APIs to official
> >>> versioned APIs at the next major release deliniation point.
> >>>
> >>> I'd welcome counter arguments, but it seems pretty natural to me to make
> >>> these sorts of changes at the major release mark. People using
> >>> experimantal apis have already implicity agreed to manage changes to
> >>> them, so if we just hold it stable in an LTS release (and don't update
> >>> the version label), its just gravy for them.
> >>>
> >> The counter argument that I see is that while the experimental tag remains
> >> in place the user has no way to know that an API is stable or not, and so
> >> in many projects cannot use the API. If for example an API that is
> >> experimental in 19.11 is unchanged through 20.05 at which point we decide
> >> to promote it to stable. Once the change to the exp tag it is made, any
> >> users of 19.11 can then see that it is an unchanged and now-stable ABI and
> >> can start using it in their software, if they wish, without having to wait
> >> for the 20.11 release. Changing the tag early allows ABIs to be potentially
> >> used immediately.
> >>
> >
> > I would agree with this, however, when using an LTS release, in my mind at
> > least, part of the agreement there is you get stability in the fuctions that
> > were declared stable at the time of release. I'm not sure there should be an
> > expectation of additional stabilization within the lifetime of the release
> > (thats really what the 12 month LTS release cycle is for, no)? You never have
> > to wait more than a year for a new set of stable features. If you want faster
> > feature integration than that, you can choose to enable the experimental API's
> > and accept the benefits and drawbacks thereof.
> >
> > That said, if (as I understand it) the goal is to provide a mechanism to allow
> > experimental features to be promoted to stable status, perhaps we can find a
> > happy middle ground. What if we were to create a construct such as this:
> >
> > #pragma push_macro("ALLOW_EXPERIMENTAL_APIS")
> > #undef ALLOW_EXPERIMENTAL_APIS
> > void __rte_experimental func(...);
> > #pragma pop_macro("ALLOW_EXPERIMENTAL_APIS")
> >
> > Such a consruct would allow the maintainer of an API to pseudo-promote an API
> > from a experimental to stable status, at least so far as compilation would be
> > concerned. Its messy and clunky, and it wouldn't change the function version at
> > all, but the end result would be that:
> > a) such a wraped experimental function would no longer issue a warning when used
> > during compilation/linking
> > and
> > b) provide a semi-easy grepable pattern for application writers to look for when
> > considering the use of an API that was previously experimental
> >
> > such a construct would have to be used very judiciously, in that once its
> > implemented, the API has to be treated as stable, even though its 'excused' from
> > the normal checking, but it could provide something of the more rapid promotion
> > path being sought.
> >
> > Thoughts?
> > Neil
> >> Regards,
> >> /Bruce
> >>
>
>
> Then method above seems clunky, and it is unclear to me if it solves the original problem.
> It seems simpler to me to promote a symbol the next ABI version, and then provide an alias.
>
> At the declaration of the next major ABI version (v21), we would then only need to check
> for those symbols that are both EXPERIMENTAL and v21, to know which aliases need to be removed.
>
> Thanks,
>
> Ray K
>
i don't disagree, just looking for other potential options.
Neil
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
2020-02-17 13:59 9% [dpdk-dev] [PATCH] ci: build and use libabigail 1.6 David Marchand
@ 2020-02-17 15:15 0% ` Thomas Monjalon
2020-02-17 18:47 0% ` Aaron Conole
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-17 15:15 UTC (permalink / raw)
To: David Marchand, christian.ehrhardt
Cc: dev, dodji, Aaron Conole, Michael Santana, kevin.laatz
17/02/2020 14:59, David Marchand:
> libabigail 1.2 (at least) reports changes in 'const' property as an ABI
> breakage [1].
> This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
> in launchpad [3].
>
> But for now, build and use the last version 1.6 so that the ABI checks
> can be kept.
>
> 1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
> 2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
> 3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> + LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
> + LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
> +
> + if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
> + rm -rf libabigail
> + # if we change libabigail, invalidate existing abi cache
> + rm -rf reference
> + fi
> +
> + if [ ! -d libabigail ]; then
> + git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
> + cd libabigail/src && autoconf -vfi && cd -
> + mkdir libabigail/src/build
> + cd libabigail/src/build && ../configure --prefix=$(pwd)/libabigail && cd -
> + make -C libabigail/src/build all install
> +
> + rm -rf libabigail/src
> + echo $LIBABIGAIL_VERSION > libabigail/VERSION
> + fi
Can we avoid compiling libabigail ourself?
Is there an up-to-date Ubuntu package somewhere?
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-14 20:48 4% ` Neil Horman
2020-02-14 21:52 4% ` Bruce Richardson
@ 2020-02-17 14:23 8% ` Ray Kinsella
2020-02-17 15:37 4% ` Neil Horman
1 sibling, 1 reply; 200+ results
From: Ray Kinsella @ 2020-02-17 14:23 UTC (permalink / raw)
To: Neil Horman, Bruce Richardson
Cc: Luca Boccassi, Ferruh Yigit, Cristian Dumitrescu, Eelco Chaudron,
dev, Thomas Monjalon, David Marchand, Ian Stokes
On 14/02/2020 20:48, Neil Horman wrote:
> On Fri, Feb 14, 2020 at 11:36:34AM +0000, Bruce Richardson wrote:
>> On Thu, Feb 13, 2020 at 09:40:40PM -0500, Neil Horman wrote:
>>> On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
>>>>
>>>>
>>>> On 05/02/2020 11:32, Neil Horman wrote:
>>>>> On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
>>>>>> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
>>>>>>>> But if we can do the versioning in the master, LTS can backport it
>>>>>>>> and can have mature version of that API in LTS without breaking
>>>>>>>> the existing users.
>>>>>>>>
>>>>>>>
>>>>>>> But why bother? The only thing you've changed is the version
>>>>>>> tagging. Its ok to leave that alone in LTS, you just cant change
>>>>>>> it.
>>>>>>>
>>>>>>> Thats part of the burden of an LTS release, it will have some drift
>>>>>>> from the upstream head, because you have to keep things stable. So
>>>>>>> you stabilize the upstream ABI version for this API and just never
>>>>>>> backport it to the current LTS release.
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> A customer (OVS) explicitly and specifically requested backporting
>>>>>> the symbol change to 19.11, as they don't want to enable
>>>>>> experimental APIs in their releases. I replied that it would only be
>>>>>> acceptable with aliasing to keep compatibility, and Ferruh very
>>>>>> kindly did the work to implement that.
>>>>>>
>>>>> but, thats part of the agreement, no? You can't always have new
>>>>> features and stability at the same time.
>>>>>
>>>>> I get that this is an odd corner case, because strictly speaking you
>>>>> could waive the ABI change that libabigail is reporting, and most
>>>>> application users (like OVS) could get away with it, because their
>>>>> application does all the right things to make it ok, but I don't
>>>>> think you can make a decsion like this for all users based on the
>>>>> request of a single user.
>>>>>
>>>>> It seems like the right thing is for OVS to augment their build time
>>>>> configuration to allow developers to select the ability to use
>>>>> experimental apis at compile time, and then the decision is placed in
>>>>> the hands of end users.
>>>>
>>>> So this is not isolated case right ... it is common from API's to
>>>> graduate from experimental to mature, we _want_ that to happen, we
>>>> _want_ APIs to mature.
>>>>
>>> Sure, I can absolutely agree with that, though I would suggest that the
>>> maturity of the ABI is orthogonal to its labeling as such (more on that
>>> below)
>>>
>>>> I am worried what you are proposing will encourage a behavior whereby
>>>> maintainers to wait until the declaration of the next major ABI version
>>>> to make these symbol changes, causing these kinds of changes to queue
>>>> up for that release.
>>>>
>>> I think you're probably right about that, and would make the agrument
>>> that thats perfectly fine (again I'll clarify below)
>>>
>>>> And you would have to ask why? In the case of a reasonably mature API,
>>>> there will be no functional or signature change - its mature! So what
>>>> is the harm of providing an alias back to Experimental until the next
>>>> ABI version is declared?
>>>>
>>> From a philosophical standpoint, there is absoluely no harm, and I don't
>>> think anyone would disagree, the harm comes from the details of the
>>> implementation, as you've noted.
>>>
>>>> So while yes, you are 100% right - experimental mean no guarantees.
>>>> But if the API is baked, it is not going to change, so I don't see the
>>>> harm.
>>>>
>>>> We don't want the unnecessary triumph of policy over pragmatism.
>>>>
>>> I would make the converse argument here. While I agree that when an API
>>> is mature, theres no point in calling it experimental anymore, I would
>>> also suggest that, if an API is mature, and not expected to change,
>>> theres no harm in leaving its version tag as experimental for an LTS
>>> release. This is what I was referring to above. Once an application
>>> developer has done the work to integrate an API from DPDK into its
>>> application, that work is done. If the API remains stable, then I
>>> honestly don't know that they care that the version label is EXPERIMENTAL
>>> versus 20.11 or 20.05 or whatever. They care about the API being stable
>>> more so than its version label. As such, it seems....reasonable to me to
>>> have developers queue their migration of experimental APIs to official
>>> versioned APIs at the next major release deliniation point.
>>>
>>> I'd welcome counter arguments, but it seems pretty natural to me to make
>>> these sorts of changes at the major release mark. People using
>>> experimantal apis have already implicity agreed to manage changes to
>>> them, so if we just hold it stable in an LTS release (and don't update
>>> the version label), its just gravy for them.
>>>
>> The counter argument that I see is that while the experimental tag remains
>> in place the user has no way to know that an API is stable or not, and so
>> in many projects cannot use the API. If for example an API that is
>> experimental in 19.11 is unchanged through 20.05 at which point we decide
>> to promote it to stable. Once the change to the exp tag it is made, any
>> users of 19.11 can then see that it is an unchanged and now-stable ABI and
>> can start using it in their software, if they wish, without having to wait
>> for the 20.11 release. Changing the tag early allows ABIs to be potentially
>> used immediately.
>>
>
> I would agree with this, however, when using an LTS release, in my mind at
> least, part of the agreement there is you get stability in the fuctions that
> were declared stable at the time of release. I'm not sure there should be an
> expectation of additional stabilization within the lifetime of the release
> (thats really what the 12 month LTS release cycle is for, no)? You never have
> to wait more than a year for a new set of stable features. If you want faster
> feature integration than that, you can choose to enable the experimental API's
> and accept the benefits and drawbacks thereof.
>
> That said, if (as I understand it) the goal is to provide a mechanism to allow
> experimental features to be promoted to stable status, perhaps we can find a
> happy middle ground. What if we were to create a construct such as this:
>
> #pragma push_macro("ALLOW_EXPERIMENTAL_APIS")
> #undef ALLOW_EXPERIMENTAL_APIS
> void __rte_experimental func(...);
> #pragma pop_macro("ALLOW_EXPERIMENTAL_APIS")
>
> Such a consruct would allow the maintainer of an API to pseudo-promote an API
> from a experimental to stable status, at least so far as compilation would be
> concerned. Its messy and clunky, and it wouldn't change the function version at
> all, but the end result would be that:
> a) such a wraped experimental function would no longer issue a warning when used
> during compilation/linking
> and
> b) provide a semi-easy grepable pattern for application writers to look for when
> considering the use of an API that was previously experimental
>
> such a construct would have to be used very judiciously, in that once its
> implemented, the API has to be treated as stable, even though its 'excused' from
> the normal checking, but it could provide something of the more rapid promotion
> path being sought.
>
> Thoughts?
> Neil
>> Regards,
>> /Bruce
>>
Then method above seems clunky, and it is unclear to me if it solves the original problem.
It seems simpler to me to promote a symbol the next ABI version, and then provide an alias.
At the declaration of the next major ABI version (v21), we would then only need to check
for those symbols that are both EXPERIMENTAL and v21, to know which aliases need to be removed.
Thanks,
Ray K
^ permalink raw reply [relevance 8%]
* [dpdk-dev] [PATCH] ci: build and use libabigail 1.6
@ 2020-02-17 13:59 9% David Marchand
2020-02-17 15:15 0% ` Thomas Monjalon
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: David Marchand @ 2020-02-17 13:59 UTC (permalink / raw)
To: thomas; +Cc: dev, christian.ehrhardt, dodji, Aaron Conole, Michael Santana
libabigail 1.2 (at least) reports changes in 'const' property as an ABI
breakage [1].
This was fixed upstream in libabigail 1.4 [2], and a bug has been opened
in launchpad [3].
But for now, build and use the last version 1.6 so that the ABI checks
can be kept.
1: https://travis-ci.com/DPDK/dpdk/jobs/287872118#L2242
2: https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commitdiff;h=215b7eb4fe8b986fe1cc87d9d8e7412998038392
3: https://bugs.launchpad.net/ubuntu/+source/libabigail/+bug/1863607
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
.ci/linux-build.sh | 22 ++++++++++++++++++++++
.travis.yml | 15 ++++++++++-----
2 files changed, 32 insertions(+), 5 deletions(-)
diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index c7c3840fc..0d4bc9a62 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -38,6 +38,28 @@ if [ "$AARCH64" != "1" ]; then
fi
if [ "$ABI_CHECKS" = "1" ]; then
+ LIBABIGAIL_REPO=${LIBABIGAIL_REPO:-https://sourceware.org/git/libabigail.git}
+ LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
+
+ if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
+ rm -rf libabigail
+ # if we change libabigail, invalidate existing abi cache
+ rm -rf reference
+ fi
+
+ if [ ! -d libabigail ]; then
+ git clone --single-branch -b $LIBABIGAIL_VERSION $LIBABIGAIL_REPO libabigail/src
+ cd libabigail/src && autoconf -vfi && cd -
+ mkdir libabigail/src/build
+ cd libabigail/src/build && ../configure --prefix=$(pwd)/libabigail && cd -
+ make -C libabigail/src/build all install
+
+ rm -rf libabigail/src
+ echo $LIBABIGAIL_VERSION > libabigail/VERSION
+ fi
+
+ export PATH=$(pwd)/libabigail/bin:$PATH
+
REF_GIT_REPO=${REF_GIT_REPO:-https://dpdk.org/git/dpdk}
REF_GIT_TAG=${REF_GIT_TAG:-v19.11}
diff --git a/.travis.yml b/.travis.yml
index 22539d823..d8253fdd4 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -2,6 +2,7 @@ language: c
cache:
ccache: true
directories:
+ - libabigail
- reference
compiler:
- gcc
@@ -24,7 +25,11 @@ aarch64_packages: &aarch64_packages
extra_packages: &extra_packages
- *required_packages
- - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4, abigail-tools]
+ - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
+
+libabigail_build_packages: &libabigail_build_packages
+ - *required_packages
+ - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
build_32b_packages: &build_32b_packages
- *required_packages
@@ -154,18 +159,18 @@ matrix:
packages:
- *required_packages
- *doc_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
+ - env: DEF_LIB="shared" ABI_CHECKS=1
compiler: gcc
addons:
apt:
packages:
- - *extra_packages
- - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
+ - *libabigail_build_packages
+ - env: DEF_LIB="shared" ABI_CHECKS=1
arch: arm64
compiler: gcc
addons:
apt:
packages:
- - *extra_packages
+ - *libabigail_build_packages
script: ./.ci/${TRAVIS_OS_NAME}-build.sh
--
2.23.0
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH v8 1/1] fbarray: fix duplicated fbarray file in secondary
2020-02-14 15:29 3% ` Thomas Monjalon
@ 2020-02-17 12:54 0% ` Yasufumi Ogawa
0 siblings, 0 replies; 200+ results
From: Yasufumi Ogawa @ 2020-02-17 12:54 UTC (permalink / raw)
To: Thomas Monjalon, David Marchand
Cc: Burakov, Anatoly, Ananyev, Konstantin, dev
> 14/02/2020 16:08, David Marchand:
>> On Fri, Feb 14, 2020 at 8:46 AM Yasufumi Ogawa <yasufum.o@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> Could I confirm that this patch is going to be merged in 20.02?
>>
>> Sorry, but I can't take this patch in 20.02.
>> It breaks compilation on FreeBSD.
>> http://mails.dpdk.org/archives/test-report/2019-November/109435.html
Sorry. I didn't find it. I'd see it.
>>
>>
>> I am still unconvinced on the need to change the size to something so
>> huge to accommodate with this new use case (secondary processes in
>> containers).
It is not so common actually, but serious issue for some NFV usecases. I
remember, in a talk in the last DPDK summit, ZTE was also suffered from
the same problem.
>> Why can't we truncate the container hostname so that it fits in 64 bytes?
It is just a possible maximum length of format of
"fbarray_memseg-1048576k-0-0_PID_HOSTNAME", so I think it can be
truncated if dropping some information.
>>
>>
>> Thomas, opinion?
>
> If the use case is justified enough, I would prefer merging such change in
> 20.11 avoiding an ABI breakage in a core library, even if it is experimental.I understand, thanks.
Yasufumi
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-14 21:52 4% ` Bruce Richardson
@ 2020-02-15 13:43 4% ` Neil Horman
0 siblings, 0 replies; 200+ results
From: Neil Horman @ 2020-02-15 13:43 UTC (permalink / raw)
To: Bruce Richardson
Cc: Ray Kinsella, Luca Boccassi, Ferruh Yigit, Cristian Dumitrescu,
Eelco Chaudron, dev, Thomas Monjalon, David Marchand, Ian Stokes
On Fri, Feb 14, 2020 at 09:52:53PM +0000, Bruce Richardson wrote:
> On Fri, Feb 14, 2020 at 03:48:48PM -0500, Neil Horman wrote:
> > On Fri, Feb 14, 2020 at 11:36:34AM +0000, Bruce Richardson wrote:
> > > On Thu, Feb 13, 2020 at 09:40:40PM -0500, Neil Horman wrote:
> > > > On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
> > > > >
> > > > >
> > > > > On 05/02/2020 11:32, Neil Horman wrote:
> > > > > > On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
> > > > > >> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
> > > > > >>>> But if we can do the versioning in the master, LTS can backport it
> > > > > >>>> and can have mature version of that API in LTS without breaking
> > > > > >>>> the existing users.
> > > > > >>>>
> > > > > >>>
> > > > > >>> But why bother? The only thing you've changed is the version
> > > > > >>> tagging. Its ok to leave that alone in LTS, you just cant change
> > > > > >>> it.
> > > > > >>>
> > > > > >>> Thats part of the burden of an LTS release, it will have some drift
> > > > > >>> from the upstream head, because you have to keep things stable. So
> > > > > >>> you stabilize the upstream ABI version for this API and just never
> > > > > >>> backport it to the current LTS release.
> > > > > >>
> > > > > >> Hi,
> > > > > >>
> > > > > >> A customer (OVS) explicitly and specifically requested backporting
> > > > > >> the symbol change to 19.11, as they don't want to enable
> > > > > >> experimental APIs in their releases. I replied that it would only be
> > > > > >> acceptable with aliasing to keep compatibility, and Ferruh very
> > > > > >> kindly did the work to implement that.
> > > > > >>
> > > > > > but, thats part of the agreement, no? You can't always have new
> > > > > > features and stability at the same time.
> > > > > >
> > > > > > I get that this is an odd corner case, because strictly speaking you
> > > > > > could waive the ABI change that libabigail is reporting, and most
> > > > > > application users (like OVS) could get away with it, because their
> > > > > > application does all the right things to make it ok, but I don't
> > > > > > think you can make a decsion like this for all users based on the
> > > > > > request of a single user.
> > > > > >
> > > > > > It seems like the right thing is for OVS to augment their build time
> > > > > > configuration to allow developers to select the ability to use
> > > > > > experimental apis at compile time, and then the decision is placed in
> > > > > > the hands of end users.
> > > > >
> > > > > So this is not isolated case right ... it is common from API's to
> > > > > graduate from experimental to mature, we _want_ that to happen, we
> > > > > _want_ APIs to mature.
> > > > >
> > > > Sure, I can absolutely agree with that, though I would suggest that the
> > > > maturity of the ABI is orthogonal to its labeling as such (more on that
> > > > below)
> > > >
> > > > > I am worried what you are proposing will encourage a behavior whereby
> > > > > maintainers to wait until the declaration of the next major ABI version
> > > > > to make these symbol changes, causing these kinds of changes to queue
> > > > > up for that release.
> > > > >
> > > > I think you're probably right about that, and would make the agrument
> > > > that thats perfectly fine (again I'll clarify below)
> > > >
> > > > > And you would have to ask why? In the case of a reasonably mature API,
> > > > > there will be no functional or signature change - its mature! So what
> > > > > is the harm of providing an alias back to Experimental until the next
> > > > > ABI version is declared?
> > > > >
> > > > From a philosophical standpoint, there is absoluely no harm, and I don't
> > > > think anyone would disagree, the harm comes from the details of the
> > > > implementation, as you've noted.
> > > >
> > > > > So while yes, you are 100% right - experimental mean no guarantees.
> > > > > But if the API is baked, it is not going to change, so I don't see the
> > > > > harm.
> > > > >
> > > > > We don't want the unnecessary triumph of policy over pragmatism.
> > > > >
> > > > I would make the converse argument here. While I agree that when an API
> > > > is mature, theres no point in calling it experimental anymore, I would
> > > > also suggest that, if an API is mature, and not expected to change,
> > > > theres no harm in leaving its version tag as experimental for an LTS
> > > > release. This is what I was referring to above. Once an application
> > > > developer has done the work to integrate an API from DPDK into its
> > > > application, that work is done. If the API remains stable, then I
> > > > honestly don't know that they care that the version label is EXPERIMENTAL
> > > > versus 20.11 or 20.05 or whatever. They care about the API being stable
> > > > more so than its version label. As such, it seems....reasonable to me to
> > > > have developers queue their migration of experimental APIs to official
> > > > versioned APIs at the next major release deliniation point.
> > > >
> > > > I'd welcome counter arguments, but it seems pretty natural to me to make
> > > > these sorts of changes at the major release mark. People using
> > > > experimantal apis have already implicity agreed to manage changes to
> > > > them, so if we just hold it stable in an LTS release (and don't update
> > > > the version label), its just gravy for them.
> > > >
> > > The counter argument that I see is that while the experimental tag remains
> > > in place the user has no way to know that an API is stable or not, and so
> > > in many projects cannot use the API. If for example an API that is
> > > experimental in 19.11 is unchanged through 20.05 at which point we decide
> > > to promote it to stable. Once the change to the exp tag it is made, any
> > > users of 19.11 can then see that it is an unchanged and now-stable ABI and
> > > can start using it in their software, if they wish, without having to wait
> > > for the 20.11 release. Changing the tag early allows ABIs to be potentially
> > > used immediately.
> > >
> >
> > I would agree with this, however, when using an LTS release, in my mind at
> > least, part of the agreement there is you get stability in the fuctions that
> > were declared stable at the time of release. I'm not sure there should be an
> > expectation of additional stabilization within the lifetime of the release
> > (thats really what the 12 month LTS release cycle is for, no)? You never have
> > to wait more than a year for a new set of stable features. If you want faster
> > feature integration than that, you can choose to enable the experimental API's
> > and accept the benefits and drawbacks thereof.
> >
> > That said, if (as I understand it) the goal is to provide a mechanism to allow
> > experimental features to be promoted to stable status, perhaps we can find a
> > happy middle ground. What if we were to create a construct such as this:
> >
> > #pragma push_macro("ALLOW_EXPERIMENTAL_APIS")
> > #undef ALLOW_EXPERIMENTAL_APIS
> > void __rte_experimental func(...);
> > #pragma pop_macro("ALLOW_EXPERIMENTAL_APIS")
> >
> > Such a consruct would allow the maintainer of an API to pseudo-promote an API
> > from a experimental to stable status, at least so far as compilation would be
> > concerned. Its messy and clunky, and it wouldn't change the function version at
> > all, but the end result would be that:
> > a) such a wraped experimental function would no longer issue a warning when used
> > during compilation/linking
> > and
> > b) provide a semi-easy grepable pattern for application writers to look for when
> > considering the use of an API that was previously experimental
> >
> > such a construct would have to be used very judiciously, in that once its
> > implemented, the API has to be treated as stable, even though its 'excused' from
> > the normal checking, but it could provide something of the more rapid promotion
> > path being sought.
> >
> Sure it could work and does meet the end goal. I'm not really sure it's a
> better solution than just promoting the function and putting in an alias
> for it back to experimental, though.
>
It probably isn't, I was just trying to think of alternative
implementations, that dont make life harder in keeping up with LTS
maintenence.
Neil
> /Bruce
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-14 20:48 4% ` Neil Horman
@ 2020-02-14 21:52 4% ` Bruce Richardson
2020-02-15 13:43 4% ` Neil Horman
2020-02-17 14:23 8% ` Ray Kinsella
1 sibling, 1 reply; 200+ results
From: Bruce Richardson @ 2020-02-14 21:52 UTC (permalink / raw)
To: Neil Horman
Cc: Ray Kinsella, Luca Boccassi, Ferruh Yigit, Cristian Dumitrescu,
Eelco Chaudron, dev, Thomas Monjalon, David Marchand, Ian Stokes
On Fri, Feb 14, 2020 at 03:48:48PM -0500, Neil Horman wrote:
> On Fri, Feb 14, 2020 at 11:36:34AM +0000, Bruce Richardson wrote:
> > On Thu, Feb 13, 2020 at 09:40:40PM -0500, Neil Horman wrote:
> > > On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
> > > >
> > > >
> > > > On 05/02/2020 11:32, Neil Horman wrote:
> > > > > On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
> > > > >> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
> > > > >>>> But if we can do the versioning in the master, LTS can backport it
> > > > >>>> and can have mature version of that API in LTS without breaking
> > > > >>>> the existing users.
> > > > >>>>
> > > > >>>
> > > > >>> But why bother? The only thing you've changed is the version
> > > > >>> tagging. Its ok to leave that alone in LTS, you just cant change
> > > > >>> it.
> > > > >>>
> > > > >>> Thats part of the burden of an LTS release, it will have some drift
> > > > >>> from the upstream head, because you have to keep things stable. So
> > > > >>> you stabilize the upstream ABI version for this API and just never
> > > > >>> backport it to the current LTS release.
> > > > >>
> > > > >> Hi,
> > > > >>
> > > > >> A customer (OVS) explicitly and specifically requested backporting
> > > > >> the symbol change to 19.11, as they don't want to enable
> > > > >> experimental APIs in their releases. I replied that it would only be
> > > > >> acceptable with aliasing to keep compatibility, and Ferruh very
> > > > >> kindly did the work to implement that.
> > > > >>
> > > > > but, thats part of the agreement, no? You can't always have new
> > > > > features and stability at the same time.
> > > > >
> > > > > I get that this is an odd corner case, because strictly speaking you
> > > > > could waive the ABI change that libabigail is reporting, and most
> > > > > application users (like OVS) could get away with it, because their
> > > > > application does all the right things to make it ok, but I don't
> > > > > think you can make a decsion like this for all users based on the
> > > > > request of a single user.
> > > > >
> > > > > It seems like the right thing is for OVS to augment their build time
> > > > > configuration to allow developers to select the ability to use
> > > > > experimental apis at compile time, and then the decision is placed in
> > > > > the hands of end users.
> > > >
> > > > So this is not isolated case right ... it is common from API's to
> > > > graduate from experimental to mature, we _want_ that to happen, we
> > > > _want_ APIs to mature.
> > > >
> > > Sure, I can absolutely agree with that, though I would suggest that the
> > > maturity of the ABI is orthogonal to its labeling as such (more on that
> > > below)
> > >
> > > > I am worried what you are proposing will encourage a behavior whereby
> > > > maintainers to wait until the declaration of the next major ABI version
> > > > to make these symbol changes, causing these kinds of changes to queue
> > > > up for that release.
> > > >
> > > I think you're probably right about that, and would make the agrument
> > > that thats perfectly fine (again I'll clarify below)
> > >
> > > > And you would have to ask why? In the case of a reasonably mature API,
> > > > there will be no functional or signature change - its mature! So what
> > > > is the harm of providing an alias back to Experimental until the next
> > > > ABI version is declared?
> > > >
> > > From a philosophical standpoint, there is absoluely no harm, and I don't
> > > think anyone would disagree, the harm comes from the details of the
> > > implementation, as you've noted.
> > >
> > > > So while yes, you are 100% right - experimental mean no guarantees.
> > > > But if the API is baked, it is not going to change, so I don't see the
> > > > harm.
> > > >
> > > > We don't want the unnecessary triumph of policy over pragmatism.
> > > >
> > > I would make the converse argument here. While I agree that when an API
> > > is mature, theres no point in calling it experimental anymore, I would
> > > also suggest that, if an API is mature, and not expected to change,
> > > theres no harm in leaving its version tag as experimental for an LTS
> > > release. This is what I was referring to above. Once an application
> > > developer has done the work to integrate an API from DPDK into its
> > > application, that work is done. If the API remains stable, then I
> > > honestly don't know that they care that the version label is EXPERIMENTAL
> > > versus 20.11 or 20.05 or whatever. They care about the API being stable
> > > more so than its version label. As such, it seems....reasonable to me to
> > > have developers queue their migration of experimental APIs to official
> > > versioned APIs at the next major release deliniation point.
> > >
> > > I'd welcome counter arguments, but it seems pretty natural to me to make
> > > these sorts of changes at the major release mark. People using
> > > experimantal apis have already implicity agreed to manage changes to
> > > them, so if we just hold it stable in an LTS release (and don't update
> > > the version label), its just gravy for them.
> > >
> > The counter argument that I see is that while the experimental tag remains
> > in place the user has no way to know that an API is stable or not, and so
> > in many projects cannot use the API. If for example an API that is
> > experimental in 19.11 is unchanged through 20.05 at which point we decide
> > to promote it to stable. Once the change to the exp tag it is made, any
> > users of 19.11 can then see that it is an unchanged and now-stable ABI and
> > can start using it in their software, if they wish, without having to wait
> > for the 20.11 release. Changing the tag early allows ABIs to be potentially
> > used immediately.
> >
>
> I would agree with this, however, when using an LTS release, in my mind at
> least, part of the agreement there is you get stability in the fuctions that
> were declared stable at the time of release. I'm not sure there should be an
> expectation of additional stabilization within the lifetime of the release
> (thats really what the 12 month LTS release cycle is for, no)? You never have
> to wait more than a year for a new set of stable features. If you want faster
> feature integration than that, you can choose to enable the experimental API's
> and accept the benefits and drawbacks thereof.
>
> That said, if (as I understand it) the goal is to provide a mechanism to allow
> experimental features to be promoted to stable status, perhaps we can find a
> happy middle ground. What if we were to create a construct such as this:
>
> #pragma push_macro("ALLOW_EXPERIMENTAL_APIS")
> #undef ALLOW_EXPERIMENTAL_APIS
> void __rte_experimental func(...);
> #pragma pop_macro("ALLOW_EXPERIMENTAL_APIS")
>
> Such a consruct would allow the maintainer of an API to pseudo-promote an API
> from a experimental to stable status, at least so far as compilation would be
> concerned. Its messy and clunky, and it wouldn't change the function version at
> all, but the end result would be that:
> a) such a wraped experimental function would no longer issue a warning when used
> during compilation/linking
> and
> b) provide a semi-easy grepable pattern for application writers to look for when
> considering the use of an API that was previously experimental
>
> such a construct would have to be used very judiciously, in that once its
> implemented, the API has to be treated as stable, even though its 'excused' from
> the normal checking, but it could provide something of the more rapid promotion
> path being sought.
>
Sure it could work and does meet the end goal. I'm not really sure it's a
better solution than just promoting the function and putting in an alias
for it back to experimental, though.
/Bruce
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-14 11:36 7% ` Bruce Richardson
@ 2020-02-14 20:48 4% ` Neil Horman
2020-02-14 21:52 4% ` Bruce Richardson
2020-02-17 14:23 8% ` Ray Kinsella
0 siblings, 2 replies; 200+ results
From: Neil Horman @ 2020-02-14 20:48 UTC (permalink / raw)
To: Bruce Richardson
Cc: Ray Kinsella, Luca Boccassi, Ferruh Yigit, Cristian Dumitrescu,
Eelco Chaudron, dev, Thomas Monjalon, David Marchand, Ian Stokes
On Fri, Feb 14, 2020 at 11:36:34AM +0000, Bruce Richardson wrote:
> On Thu, Feb 13, 2020 at 09:40:40PM -0500, Neil Horman wrote:
> > On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
> > >
> > >
> > > On 05/02/2020 11:32, Neil Horman wrote:
> > > > On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
> > > >> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
> > > >>>> But if we can do the versioning in the master, LTS can backport it
> > > >>>> and can have mature version of that API in LTS without breaking
> > > >>>> the existing users.
> > > >>>>
> > > >>>
> > > >>> But why bother? The only thing you've changed is the version
> > > >>> tagging. Its ok to leave that alone in LTS, you just cant change
> > > >>> it.
> > > >>>
> > > >>> Thats part of the burden of an LTS release, it will have some drift
> > > >>> from the upstream head, because you have to keep things stable. So
> > > >>> you stabilize the upstream ABI version for this API and just never
> > > >>> backport it to the current LTS release.
> > > >>
> > > >> Hi,
> > > >>
> > > >> A customer (OVS) explicitly and specifically requested backporting
> > > >> the symbol change to 19.11, as they don't want to enable
> > > >> experimental APIs in their releases. I replied that it would only be
> > > >> acceptable with aliasing to keep compatibility, and Ferruh very
> > > >> kindly did the work to implement that.
> > > >>
> > > > but, thats part of the agreement, no? You can't always have new
> > > > features and stability at the same time.
> > > >
> > > > I get that this is an odd corner case, because strictly speaking you
> > > > could waive the ABI change that libabigail is reporting, and most
> > > > application users (like OVS) could get away with it, because their
> > > > application does all the right things to make it ok, but I don't
> > > > think you can make a decsion like this for all users based on the
> > > > request of a single user.
> > > >
> > > > It seems like the right thing is for OVS to augment their build time
> > > > configuration to allow developers to select the ability to use
> > > > experimental apis at compile time, and then the decision is placed in
> > > > the hands of end users.
> > >
> > > So this is not isolated case right ... it is common from API's to
> > > graduate from experimental to mature, we _want_ that to happen, we
> > > _want_ APIs to mature.
> > >
> > Sure, I can absolutely agree with that, though I would suggest that the
> > maturity of the ABI is orthogonal to its labeling as such (more on that
> > below)
> >
> > > I am worried what you are proposing will encourage a behavior whereby
> > > maintainers to wait until the declaration of the next major ABI version
> > > to make these symbol changes, causing these kinds of changes to queue
> > > up for that release.
> > >
> > I think you're probably right about that, and would make the agrument
> > that thats perfectly fine (again I'll clarify below)
> >
> > > And you would have to ask why? In the case of a reasonably mature API,
> > > there will be no functional or signature change - its mature! So what
> > > is the harm of providing an alias back to Experimental until the next
> > > ABI version is declared?
> > >
> > From a philosophical standpoint, there is absoluely no harm, and I don't
> > think anyone would disagree, the harm comes from the details of the
> > implementation, as you've noted.
> >
> > > So while yes, you are 100% right - experimental mean no guarantees.
> > > But if the API is baked, it is not going to change, so I don't see the
> > > harm.
> > >
> > > We don't want the unnecessary triumph of policy over pragmatism.
> > >
> > I would make the converse argument here. While I agree that when an API
> > is mature, theres no point in calling it experimental anymore, I would
> > also suggest that, if an API is mature, and not expected to change,
> > theres no harm in leaving its version tag as experimental for an LTS
> > release. This is what I was referring to above. Once an application
> > developer has done the work to integrate an API from DPDK into its
> > application, that work is done. If the API remains stable, then I
> > honestly don't know that they care that the version label is EXPERIMENTAL
> > versus 20.11 or 20.05 or whatever. They care about the API being stable
> > more so than its version label. As such, it seems....reasonable to me to
> > have developers queue their migration of experimental APIs to official
> > versioned APIs at the next major release deliniation point.
> >
> > I'd welcome counter arguments, but it seems pretty natural to me to make
> > these sorts of changes at the major release mark. People using
> > experimantal apis have already implicity agreed to manage changes to
> > them, so if we just hold it stable in an LTS release (and don't update
> > the version label), its just gravy for them.
> >
> The counter argument that I see is that while the experimental tag remains
> in place the user has no way to know that an API is stable or not, and so
> in many projects cannot use the API. If for example an API that is
> experimental in 19.11 is unchanged through 20.05 at which point we decide
> to promote it to stable. Once the change to the exp tag it is made, any
> users of 19.11 can then see that it is an unchanged and now-stable ABI and
> can start using it in their software, if they wish, without having to wait
> for the 20.11 release. Changing the tag early allows ABIs to be potentially
> used immediately.
>
I would agree with this, however, when using an LTS release, in my mind at
least, part of the agreement there is you get stability in the fuctions that
were declared stable at the time of release. I'm not sure there should be an
expectation of additional stabilization within the lifetime of the release
(thats really what the 12 month LTS release cycle is for, no)? You never have
to wait more than a year for a new set of stable features. If you want faster
feature integration than that, you can choose to enable the experimental API's
and accept the benefits and drawbacks thereof.
That said, if (as I understand it) the goal is to provide a mechanism to allow
experimental features to be promoted to stable status, perhaps we can find a
happy middle ground. What if we were to create a construct such as this:
#pragma push_macro("ALLOW_EXPERIMENTAL_APIS")
#undef ALLOW_EXPERIMENTAL_APIS
void __rte_experimental func(...);
#pragma pop_macro("ALLOW_EXPERIMENTAL_APIS")
Such a consruct would allow the maintainer of an API to pseudo-promote an API
from a experimental to stable status, at least so far as compilation would be
concerned. Its messy and clunky, and it wouldn't change the function version at
all, but the end result would be that:
a) such a wraped experimental function would no longer issue a warning when used
during compilation/linking
and
b) provide a semi-easy grepable pattern for application writers to look for when
considering the use of an API that was previously experimental
such a construct would have to be used very judiciously, in that once its
implemented, the API has to be treated as stable, even though its 'excused' from
the normal checking, but it could provide something of the more rapid promotion
path being sought.
Thoughts?
Neil
> Regards,
> /Bruce
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2] service: don't walk out of bounds when checking services
2020-02-07 14:27 3% ` Aaron Conole
@ 2020-02-14 16:38 0% ` Kevin Traynor
0 siblings, 0 replies; 200+ results
From: Kevin Traynor @ 2020-02-14 16:38 UTC (permalink / raw)
To: Aaron Conole
Cc: Harry van Haaren, nikhil.rao, David Marchand, dev,
Bruce Richardson, Pavan Nikhilesh, Gage Eads, Thomas Monjalon,
dpdk stable
On 07/02/2020 14:27, Aaron Conole wrote:
> Kevin Traynor <ktraynor@redhat.com> writes:
>
>> On 20/12/2019 14:43, David Marchand wrote:
>>> On Wed, Dec 4, 2019 at 9:34 AM David Marchand <david.marchand@redhat.com> wrote:
>>>>
>>>> On Wed, Dec 4, 2019 at 9:33 AM David Marchand <david.marchand@redhat.com> wrote:
>>>>>
>>>>> On Tue, Dec 3, 2019 at 10:15 PM Aaron Conole <aconole@redhat.com> wrote:
>>>>>>
>>>>>> The service_valid call is used without properly bounds checking the
>>>>>> input parameter. Almost all instances of the service_valid call are
>>>>>> inside a for() loop that prevents excessive walks, but some of the
>>>>>> public APIs don't bounds check and will pass invalid arguments.
>>>>>>
>>>>>> Prevent this by using SERVICE_GET_OR_ERR_RET where it makes sense,
>>>>>> and adding a bounds check to one service_valid() use.
>>>>>>
>>>>>> Fixes: 8d39d3e237c2 ("service: fix race in service on app lcore function")
>>>>>> Fixes: e9139a32f6e8 ("service: add function to run on app lcore")
>>>>>> Fixes: e30dd31847d2 ("service: add mechanism for quiescing")
>>>> Cc: stable@dpdk.org
>>>>
>>
>> With the commit below, this patch will apply cleanly on 18.11.
>>
>> Seems ok to me to add below commit, wdyt?
>
> If I'm reading it correctly, the move is for an internal data structure
> in librte_eal, so I think it shouldn't be an ABI breakage.
>
> Looks safe to me as well.
>
Thanks, both patches applied.
>> commit e484ccddbe1b41886fef1e445ef2fdfa55086198
>> Author: Nikhil Rao <nikhil.rao@intel.com>
>> Date: Mon Sep 16 15:31:02 2019 +0530
>>
>> service: avoid false sharing on core state
>>
>>
>>>>>> Signed-off-by: Aaron Conole <aconole@redhat.com>
>>>>>
>>>>> Reviewed-by: David Marchand <david.marchand@redhat.com>
>>>
>>> Applied, thanks.
>>>
>>>
>>> --
>>> David Marchand
>>>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v8 1/1] fbarray: fix duplicated fbarray file in secondary
@ 2020-02-14 15:29 3% ` Thomas Monjalon
2020-02-17 12:54 0% ` Yasufumi Ogawa
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-14 15:29 UTC (permalink / raw)
To: Yasufumi Ogawa, David Marchand; +Cc: Burakov, Anatoly, Ananyev, Konstantin, dev
14/02/2020 16:08, David Marchand:
> On Fri, Feb 14, 2020 at 8:46 AM Yasufumi Ogawa <yasufum.o@gmail.com> wrote:
> >
> > Hi,
> >
> > Could I confirm that this patch is going to be merged in 20.02?
>
> Sorry, but I can't take this patch in 20.02.
> It breaks compilation on FreeBSD.
> http://mails.dpdk.org/archives/test-report/2019-November/109435.html
>
>
> I am still unconvinced on the need to change the size to something so
> huge to accommodate with this new use case (secondary processes in
> containers).
> Why can't we truncate the container hostname so that it fits in 64 bytes?
>
>
> Thomas, opinion?
If the use case is justified enough, I would prefer merging such change in
20.11 avoiding an ABI breakage in a core library, even if it is experimental.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH 0/7] vfio/pci: SR-IOV support
2020-02-14 4:57 0% ` Alexey Kardashevskiy
@ 2020-02-14 15:27 0% ` Alex Williamson
0 siblings, 0 replies; 200+ results
From: Alex Williamson @ 2020-02-14 15:27 UTC (permalink / raw)
To: Alexey Kardashevskiy
Cc: kvm, linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, bruce.richardson, cohuck
On Fri, 14 Feb 2020 15:57:04 +1100
Alexey Kardashevskiy <aik@ozlabs.ru> wrote:
> On 12/02/2020 10:05, Alex Williamson wrote:
> > Given the mostly positive feedback from the RFC[1], here's a new
> > non-RFC revision. Changes since RFC:
> >
> > - vfio_device_ops.match semantics refined
> > - Use helpers for struct pci_dev.physfn to avoid breakage without
> > CONFIG_PCI_IOV
> > - Relax to allow SR-IOV configuration changes while PF is opened.
> > There are potentially interesting use cases here, including
> > perhaps QEMU emulating an SR-IOV capability and calling out
> > to a privileged entity to manipulate sriov_numvfs and corral
> > the resulting devices.
> > - Retest vfio_device_feature.argsz to include uuid length.
> > - Add Connie's R-b on 6/7
> >
> > I still wish we had a solution to make it less opaque to the user
> > why a VFIO_GROUP_GET_DEVICE_FD() has failed if a VF token is
> > required, but this is still the best I've been able to come up with.
> > If there are objections or better ideas, please raise them now.
> >
> > The synopsis of this series is that we have an ongoing desire to drive
> > PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
> > for this with DPDK drivers and potentially interesting future use
> > cases in virtualization. We've been reluctant to add this support
> > previously due to the dependency and trust relationship between the
> > VF device and PF driver. Minimally the PF driver can induce a denial
> > of service to the VF, but depending on the specific implementation,
> > the PF driver might also be responsible for moving data between VFs
> > or have direct access to the state of the VF, including data or state
> > otherwise private to the VF or VF driver.
> >
> > To help resolve these concerns, we introduce a VF token into the VFIO
> > PCI ABI, which acts as a shared secret key between drivers. The
> > userspace PF driver is required to set the VF token to a known value
> > and userspace VF drivers are required to provide the token to access
> > the VF device. If a PF driver is restarted with VF drivers in use, it
> > must also provide the current token in order to prevent a rogue
> > untrusted PF driver from replacing a known driver. The degree to
> > which this new token is considered secret is left to the userspace
> > drivers, the kernel intentionally provides no means to retrieve the
> > current token.
> >
> > Note that the above token is only required for this new model where
> > both the PF and VF devices are usable through vfio-pci. Existing
> > models of VFIO drivers where the PF is used without SR-IOV enabled
> > or the VF is bound to a userspace driver with an in-kernel, host PF
> > driver are unaffected.
> >
> > The latter configuration above also highlights a new inverted scenario
> > that is now possible, a userspace PF driver with in-kernel VF drivers.
> > I believe this is a scenario that should be allowed, but should not be
> > enabled by default. This series includes code to set a default
> > driver_override for VFs sourced from a vfio-pci user owned PF, such
> > that the VFs are also bound to vfio-pci. This model is compatible
> > with tools like driverctl and allows the system administrator to
> > decide if other bindings should be enabled. The VF token interface
> > above exists only between vfio-pci PF and VF drivers, once a VF is
> > bound to another driver, the administrator has effectively pronounced
> > the device as trusted. The vfio-pci driver will note alternate
> > binding in dmesg for logging and debugging purposes.
> >
> > Please review, comment, and test. The example QEMU implementation
> > provided with the RFC[2] is still current for this version. Thanks,
>
>
> It is a cool feature. One question - what device have you tested it with?
>
> Does not a PF want to control/manage VFs on a PF driver side? I am
> thinking of Mellanox CX5 or similar NIC and it acts as an managed
> ethernet switch which might want to do something to VFs and VFs may not
> work as expected without PF's native driver doing things to it, or this
> is not a concern, is it? Thanks,
TBH, I'm starting with the premise that a userspace PF driver already
works. The DPDK folks have produced some "interesting" code that
allows SR-IOV to be enabled on a PF underneath vfio-pci. There's also
a non-upstream igb-uio driver associated with DPDK that seems to be
recommended for SR-IOV PF driver use cases, particularly for an FPGA
device. The testing I've done, and what's provided by the QEMU patch I
reference, is really only unit testing the vf_token support and
DEVICE_FEATURE ioctl provided here. I used this with an Intel 82576
(igb) where the PF driver doesn't particularly like being assigned to a
VM with SR-IOV enabled. Likewise, I can prove that the interfaces here
provide the correct restrictions for the VF, but the VF doesn't work in
a VM due to the state of the PF. I'm hoping we'll have some
confirmation from the DPDK folks that this provides what they need to
abandon the non-upstream drivers and more nefarious hacks. There's a
lot more virtualization work to be done in QEMU before I'd propose
patch I reference above upstream.
To your specific question regarding CX5, I think there are very few
SR-IOV devices where the PF doesn't act as some kind of packet router
or ring management engine. The Amazon device listed in the pci-pf-stub
driver seems to be one of the few SR-IOV devices which claim the PF has
no special interfaces other than exposing the SR-IOV capability itself.
So I think we generally expect a device specific SR-IOV aware driver
running on the PF via this interface. That's certainly the case for
the DPDK code for the FPGA device above. Thanks,
Alex
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-14 2:40 7% ` Neil Horman
@ 2020-02-14 11:36 7% ` Bruce Richardson
2020-02-14 20:48 4% ` Neil Horman
0 siblings, 1 reply; 200+ results
From: Bruce Richardson @ 2020-02-14 11:36 UTC (permalink / raw)
To: Neil Horman
Cc: Ray Kinsella, Luca Boccassi, Ferruh Yigit, Cristian Dumitrescu,
Eelco Chaudron, dev, Thomas Monjalon, David Marchand, Ian Stokes
On Thu, Feb 13, 2020 at 09:40:40PM -0500, Neil Horman wrote:
> On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
> >
> >
> > On 05/02/2020 11:32, Neil Horman wrote:
> > > On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
> > >> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
> > >>>> But if we can do the versioning in the master, LTS can backport it
> > >>>> and can have mature version of that API in LTS without breaking
> > >>>> the existing users.
> > >>>>
> > >>>
> > >>> But why bother? The only thing you've changed is the version
> > >>> tagging. Its ok to leave that alone in LTS, you just cant change
> > >>> it.
> > >>>
> > >>> Thats part of the burden of an LTS release, it will have some drift
> > >>> from the upstream head, because you have to keep things stable. So
> > >>> you stabilize the upstream ABI version for this API and just never
> > >>> backport it to the current LTS release.
> > >>
> > >> Hi,
> > >>
> > >> A customer (OVS) explicitly and specifically requested backporting
> > >> the symbol change to 19.11, as they don't want to enable
> > >> experimental APIs in their releases. I replied that it would only be
> > >> acceptable with aliasing to keep compatibility, and Ferruh very
> > >> kindly did the work to implement that.
> > >>
> > > but, thats part of the agreement, no? You can't always have new
> > > features and stability at the same time.
> > >
> > > I get that this is an odd corner case, because strictly speaking you
> > > could waive the ABI change that libabigail is reporting, and most
> > > application users (like OVS) could get away with it, because their
> > > application does all the right things to make it ok, but I don't
> > > think you can make a decsion like this for all users based on the
> > > request of a single user.
> > >
> > > It seems like the right thing is for OVS to augment their build time
> > > configuration to allow developers to select the ability to use
> > > experimental apis at compile time, and then the decision is placed in
> > > the hands of end users.
> >
> > So this is not isolated case right ... it is common from API's to
> > graduate from experimental to mature, we _want_ that to happen, we
> > _want_ APIs to mature.
> >
> Sure, I can absolutely agree with that, though I would suggest that the
> maturity of the ABI is orthogonal to its labeling as such (more on that
> below)
>
> > I am worried what you are proposing will encourage a behavior whereby
> > maintainers to wait until the declaration of the next major ABI version
> > to make these symbol changes, causing these kinds of changes to queue
> > up for that release.
> >
> I think you're probably right about that, and would make the agrument
> that thats perfectly fine (again I'll clarify below)
>
> > And you would have to ask why? In the case of a reasonably mature API,
> > there will be no functional or signature change - its mature! So what
> > is the harm of providing an alias back to Experimental until the next
> > ABI version is declared?
> >
> From a philosophical standpoint, there is absoluely no harm, and I don't
> think anyone would disagree, the harm comes from the details of the
> implementation, as you've noted.
>
> > So while yes, you are 100% right - experimental mean no guarantees.
> > But if the API is baked, it is not going to change, so I don't see the
> > harm.
> >
> > We don't want the unnecessary triumph of policy over pragmatism.
> >
> I would make the converse argument here. While I agree that when an API
> is mature, theres no point in calling it experimental anymore, I would
> also suggest that, if an API is mature, and not expected to change,
> theres no harm in leaving its version tag as experimental for an LTS
> release. This is what I was referring to above. Once an application
> developer has done the work to integrate an API from DPDK into its
> application, that work is done. If the API remains stable, then I
> honestly don't know that they care that the version label is EXPERIMENTAL
> versus 20.11 or 20.05 or whatever. They care about the API being stable
> more so than its version label. As such, it seems....reasonable to me to
> have developers queue their migration of experimental APIs to official
> versioned APIs at the next major release deliniation point.
>
> I'd welcome counter arguments, but it seems pretty natural to me to make
> these sorts of changes at the major release mark. People using
> experimantal apis have already implicity agreed to manage changes to
> them, so if we just hold it stable in an LTS release (and don't update
> the version label), its just gravy for them.
>
The counter argument that I see is that while the experimental tag remains
in place the user has no way to know that an API is stable or not, and so
in many projects cannot use the API. If for example an API that is
experimental in 19.11 is unchanged through 20.05 at which point we decide
to promote it to stable. Once the change to the exp tag it is made, any
users of 19.11 can then see that it is an unchanged and now-stable ABI and
can start using it in their software, if they wish, without having to wait
for the 20.11 release. Changing the tag early allows ABIs to be potentially
used immediately.
Regards,
/Bruce
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH v8 1/1] fbarray: fix duplicated fbarray file in secondary
@ 2020-02-14 7:46 0% ` Yasufumi Ogawa
0 siblings, 1 reply; 200+ results
From: Yasufumi Ogawa @ 2020-02-14 7:46 UTC (permalink / raw)
To: Burakov, Anatoly, konstantin.ananyev, david.marchand, dev
Cc: yasufumi.ogawa.gy, Yasufumi Ogawa
Hi,
Could I confirm that this patch is going to be merged in 20.02?
Regards,
Yasufumi
On 2019/12/06 19:44, Burakov, Anatoly wrote:
> On 27-Nov-19 8:48 AM, Yasufumi Ogawa wrote:
>> From: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
>>
>> In secondary_msl_create_walk(), it creates a file for fbarrays with its
>> PID for reserving unique name among secondary processes. However, it
>> does not work if several secondaries run as app containers because each
>> of containerized secondary has PID 1, and failed to reserve unique name
>> other than first one. To reserve unique name in each of containers, use
>> hostname in addition to PID.
>>
>> Signed-off-by: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
>> ---
>> lib/librte_eal/common/include/rte_fbarray.h | 7 ++++++-
>> lib/librte_eal/linux/eal/eal_memalloc.c | 11 ++++++++---
>> 2 files changed, 14 insertions(+), 4 deletions(-)
>>
>> diff --git a/lib/librte_eal/common/include/rte_fbarray.h
>> b/lib/librte_eal/common/include/rte_fbarray.h
>> index 6dccdbec9..df003b8dc 100644
>> --- a/lib/librte_eal/common/include/rte_fbarray.h
>> +++ b/lib/librte_eal/common/include/rte_fbarray.h
>> @@ -39,7 +39,12 @@ extern "C" {
>> #include <rte_compat.h>
>> #include <rte_rwlock.h>
>> -#define RTE_FBARRAY_NAME_LEN 64
>> +/* Filename of fbarray is defined as a combination of several params
>> + * such as "fbarray_memseg-1048576k-0-0_PID_HOSTNAME".
>> + * The length of string before PID can be 32bytes, and the length of
>> + * PID can be 7bytes maximamly. Final 1 byte is for null terminator.
>> + */
>> +#define RTE_FBARRAY_NAME_LEN (32 + 7 + 1 + HOST_NAME_MAX + 1)
>
> This breaks ABI, but i believe this doesn't break *stable* ABI, so it is
> OK.
>
> Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 0/7] vfio/pci: SR-IOV support
2020-02-11 23:05 2% [dpdk-dev] [PATCH 0/7] vfio/pci: SR-IOV support Alex Williamson
@ 2020-02-14 4:57 0% ` Alexey Kardashevskiy
2020-02-14 15:27 0% ` Alex Williamson
0 siblings, 1 reply; 200+ results
From: Alexey Kardashevskiy @ 2020-02-14 4:57 UTC (permalink / raw)
To: Alex Williamson, kvm
Cc: linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, bruce.richardson, cohuck
On 12/02/2020 10:05, Alex Williamson wrote:
> Given the mostly positive feedback from the RFC[1], here's a new
> non-RFC revision. Changes since RFC:
>
> - vfio_device_ops.match semantics refined
> - Use helpers for struct pci_dev.physfn to avoid breakage without
> CONFIG_PCI_IOV
> - Relax to allow SR-IOV configuration changes while PF is opened.
> There are potentially interesting use cases here, including
> perhaps QEMU emulating an SR-IOV capability and calling out
> to a privileged entity to manipulate sriov_numvfs and corral
> the resulting devices.
> - Retest vfio_device_feature.argsz to include uuid length.
> - Add Connie's R-b on 6/7
>
> I still wish we had a solution to make it less opaque to the user
> why a VFIO_GROUP_GET_DEVICE_FD() has failed if a VF token is
> required, but this is still the best I've been able to come up with.
> If there are objections or better ideas, please raise them now.
>
> The synopsis of this series is that we have an ongoing desire to drive
> PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
> for this with DPDK drivers and potentially interesting future use
> cases in virtualization. We've been reluctant to add this support
> previously due to the dependency and trust relationship between the
> VF device and PF driver. Minimally the PF driver can induce a denial
> of service to the VF, but depending on the specific implementation,
> the PF driver might also be responsible for moving data between VFs
> or have direct access to the state of the VF, including data or state
> otherwise private to the VF or VF driver.
>
> To help resolve these concerns, we introduce a VF token into the VFIO
> PCI ABI, which acts as a shared secret key between drivers. The
> userspace PF driver is required to set the VF token to a known value
> and userspace VF drivers are required to provide the token to access
> the VF device. If a PF driver is restarted with VF drivers in use, it
> must also provide the current token in order to prevent a rogue
> untrusted PF driver from replacing a known driver. The degree to
> which this new token is considered secret is left to the userspace
> drivers, the kernel intentionally provides no means to retrieve the
> current token.
>
> Note that the above token is only required for this new model where
> both the PF and VF devices are usable through vfio-pci. Existing
> models of VFIO drivers where the PF is used without SR-IOV enabled
> or the VF is bound to a userspace driver with an in-kernel, host PF
> driver are unaffected.
>
> The latter configuration above also highlights a new inverted scenario
> that is now possible, a userspace PF driver with in-kernel VF drivers.
> I believe this is a scenario that should be allowed, but should not be
> enabled by default. This series includes code to set a default
> driver_override for VFs sourced from a vfio-pci user owned PF, such
> that the VFs are also bound to vfio-pci. This model is compatible
> with tools like driverctl and allows the system administrator to
> decide if other bindings should be enabled. The VF token interface
> above exists only between vfio-pci PF and VF drivers, once a VF is
> bound to another driver, the administrator has effectively pronounced
> the device as trusted. The vfio-pci driver will note alternate
> binding in dmesg for logging and debugging purposes.
>
> Please review, comment, and test. The example QEMU implementation
> provided with the RFC[2] is still current for this version. Thanks,
It is a cool feature. One question - what device have you tested it with?
Does not a PF want to control/manage VFs on a PF driver side? I am
thinking of Mellanox CX5 or similar NIC and it acts as an managed
ethernet switch which might want to do something to VFs and VFs may not
work as expected without PF's native driver doing things to it, or this
is not a concern, is it? Thanks,
>
> Alex
>
> [1] https://lore.kernel.org/lkml/158085337582.9445.17682266437583505502.stgit@gimli.home/
> [2] https://lore.kernel.org/lkml/20200204161737.34696b91@w520.home/
> ---
>
> Alex Williamson (7):
> vfio: Include optional device match in vfio_device_ops callbacks
> vfio/pci: Implement match ops
> vfio/pci: Introduce VF token
> vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
> vfio/pci: Add sriov_configure support
> vfio/pci: Remove dev_fmt definition
> vfio/pci: Cleanup .probe() exit paths
>
>
> drivers/vfio/pci/vfio_pci.c | 312 ++++++++++++++++++++++++++++++++---
> drivers/vfio/pci/vfio_pci_private.h | 10 +
> drivers/vfio/vfio.c | 20 ++
> include/linux/vfio.h | 4
> include/uapi/linux/vfio.h | 37 ++++
> 5 files changed, 355 insertions(+), 28 deletions(-)
>
--
Alexey
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-13 17:40 8% ` Ray Kinsella
@ 2020-02-14 2:40 7% ` Neil Horman
2020-02-14 11:36 7% ` Bruce Richardson
0 siblings, 1 reply; 200+ results
From: Neil Horman @ 2020-02-14 2:40 UTC (permalink / raw)
To: Ray Kinsella
Cc: Luca Boccassi, Ferruh Yigit, Cristian Dumitrescu, Eelco Chaudron,
dev, Thomas Monjalon, David Marchand, Bruce Richardson,
Ian Stokes
On Thu, Feb 13, 2020 at 05:40:43PM +0000, Ray Kinsella wrote:
>
>
> On 05/02/2020 11:32, Neil Horman wrote:
> > On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
> >> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
> >>>> But if we can do the versioning in the master, LTS can backport it
> >>>> and can have
> >>>> mature version of that API in LTS without breaking the existing
> >>>> users.
> >>>>
> >>>
> >>> But why bother? The only thing you've changed is the version
> >>> tagging. Its ok
> >>> to leave that alone in LTS, you just cant change it.
> >>>
> >>> Thats part of the burden of an LTS release, it will have some drift
> >>> from the
> >>> upstream head, because you have to keep things stable. So you
> >>> stabilize the
> >>> upstream ABI version for this API and just never backport it to the
> >>> current LTS
> >>> release.
> >>
> >> Hi,
> >>
> >> A customer (OVS) explicitly and specifically requested backporting the
> >> symbol change to 19.11, as they don't want to enable experimental APIs
> >> in their releases. I replied that it would only be acceptable with
> >> aliasing to keep compatibility, and Ferruh very kindly did the work to
> >> implement that.
> >>
> > but, thats part of the agreement, no? You can't always have new features and
> > stability at the same time.
> >
> > I get that this is an odd corner case, because strictly speaking you could waive
> > the ABI change that libabigail is reporting, and most application users (like
> > OVS) could get away with it, because their application does all the right things
> > to make it ok, but I don't think you can make a decsion like this for all users
> > based on the request of a single user.
> >
> > It seems like the right thing is for OVS to augment their build time
> > configuration to allow developers to select the ability to use experimental apis
> > at compile time, and then the decision is placed in the hands of end users.
>
> So this is not isolated case right ... it is common from API's to graduate
> from experimental to mature, we _want_ that to happen, we _want_ APIs to mature.
>
Sure, I can absolutely agree with that, though I would suggest that
the maturity of the ABI is orthogonal to its labeling as such (more on
that below)
> I am worried what you are proposing will encourage a behavior whereby maintainers
> to wait until the declaration of the next major ABI version to make these symbol changes,
> causing these kinds of changes to queue up for that release.
>
I think you're probably right about that, and would make the agrument
that thats perfectly fine (again I'll clarify below)
> And you would have to ask why?
> In the case of a reasonably mature API, there will be no functional or signature change - its mature!
> So what is the harm of providing an alias back to Experimental until the next ABI version is declared?
>
From a philosophical standpoint, there is absoluely no harm, and I don't
think anyone would disagree, the harm comes from the details of the
implementation, as you've noted.
> So while yes, you are 100% right - experimental mean no guarantees.
> But if the API is baked, it is not going to change, so I don't see the harm.
>
> We don't want the unnecessary triumph of policy over pragmatism.
>
I would make the converse argument here. While I agree that when an API
is mature, theres no point in calling it experimental anymore, I would
also suggest that, if an API is mature, and not expected to change,
theres no harm in leaving its version tag as experimental for an LTS
release. This is what I was referring to above. Once an application
developer has done the work to integrate an API from DPDK into its
application, that work is done. If the API remains stable, then I
honestly don't know that they care that the version label is
EXPERIMENTAL versus 20.11 or 20.05 or whatever. They care about the API
being stable more so than its version label. As such, it
seems....reasonable to me to have developers queue their migration of
experimental APIs to official versioned APIs at the next major release
deliniation point.
I'd welcome counter arguments, but it seems pretty natural to me to make
these sorts of changes at the major release mark. People using
experimantal apis have already implicity agreed to manage changes to
them, so if we just hold it stable in an LTS release (and don't update
the version label), its just gravy for them.
Regards
Neil
>
>
>
>
>
>
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH] doc: remove major in designation of normal releases
@ 2020-02-13 20:51 0% ` Kevin Traynor
0 siblings, 0 replies; 200+ results
From: Kevin Traynor @ 2020-02-13 20:51 UTC (permalink / raw)
To: Thomas Monjalon, John McNamara, Marko Kovacevic
Cc: dev, orika, David Marchand
On 05/08/2019 13:30, Thomas Monjalon wrote:
> The word "major" was used to differentiate with release candidates
> or stable maintenance releases.
> However the word "major" can be understood as "LTS",
> so it is less confusing to avoid this word.
>
> Reported-by: Ori Kam <orika@mellanox.com>
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---
> doc/guides/contributing/documentation.rst | 2 +-
> doc/guides/contributing/stable.rst | 2 +-
> 2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/doc/guides/contributing/documentation.rst b/doc/guides/contributing/documentation.rst
> index 27e4b13be..550d8dec2 100644
> --- a/doc/guides/contributing/documentation.rst
> +++ b/doc/guides/contributing/documentation.rst
> @@ -62,7 +62,7 @@ added to by the developer.
>
> The Release Notes document which features have been added in the current and previous releases of DPDK and highlight
> any known issues.
> - The Releases Notes also contain notifications of features that will change ABI compatibility in the next major release.
> + The Releases Notes also contain notifications of features that will change ABI compatibility in the next release.
>
> Developers should include updates to the Release Notes with patch sets that relate to any of the following sections:
>
> diff --git a/doc/guides/contributing/stable.rst b/doc/guides/contributing/stable.rst
> index 6a5eee9bd..24b9e8b7d 100644
> --- a/doc/guides/contributing/stable.rst
> +++ b/doc/guides/contributing/stable.rst
> @@ -25,7 +25,7 @@ Release to indicate longer term support.
> Stable Releases
> ---------------
>
> -Any major release of DPDK can be designated as a Stable Release if a
> +Any release of DPDK can be designated as a Stable Release if a
> maintainer volunteers to maintain it and there is a commitment from major
> contributors to validate it before releases. If a release is to be designated
> as a Stable Release, it should be done by 1 month after the master release.
>
Acked-by: Kevin Traynor <ktraynor@redhat.com>
(Prompted-by: David Marchand <david.marchand@redhat.com> ;-))
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-05 11:32 7% ` Neil Horman
@ 2020-02-13 17:40 8% ` Ray Kinsella
2020-02-14 2:40 7% ` Neil Horman
0 siblings, 1 reply; 200+ results
From: Ray Kinsella @ 2020-02-13 17:40 UTC (permalink / raw)
To: Neil Horman, Luca Boccassi
Cc: Ferruh Yigit, Cristian Dumitrescu, Eelco Chaudron, dev,
Thomas Monjalon, David Marchand, Bruce Richardson, Ian Stokes
On 05/02/2020 11:32, Neil Horman wrote:
> On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
>> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
>>>> But if we can do the versioning in the master, LTS can backport it
>>>> and can have
>>>> mature version of that API in LTS without breaking the existing
>>>> users.
>>>>
>>>
>>> But why bother? The only thing you've changed is the version
>>> tagging. Its ok
>>> to leave that alone in LTS, you just cant change it.
>>>
>>> Thats part of the burden of an LTS release, it will have some drift
>>> from the
>>> upstream head, because you have to keep things stable. So you
>>> stabilize the
>>> upstream ABI version for this API and just never backport it to the
>>> current LTS
>>> release.
>>
>> Hi,
>>
>> A customer (OVS) explicitly and specifically requested backporting the
>> symbol change to 19.11, as they don't want to enable experimental APIs
>> in their releases. I replied that it would only be acceptable with
>> aliasing to keep compatibility, and Ferruh very kindly did the work to
>> implement that.
>>
> but, thats part of the agreement, no? You can't always have new features and
> stability at the same time.
>
> I get that this is an odd corner case, because strictly speaking you could waive
> the ABI change that libabigail is reporting, and most application users (like
> OVS) could get away with it, because their application does all the right things
> to make it ok, but I don't think you can make a decsion like this for all users
> based on the request of a single user.
>
> It seems like the right thing is for OVS to augment their build time
> configuration to allow developers to select the ability to use experimental apis
> at compile time, and then the decision is placed in the hands of end users.
So this is not isolated case right ... it is common from API's to graduate
from experimental to mature, we _want_ that to happen, we _want_ APIs to mature.
I am worried what you are proposing will encourage a behavior whereby maintainers
to wait until the declaration of the next major ABI version to make these symbol changes,
causing these kinds of changes to queue up for that release.
And you would have to ask why?
In the case of a reasonably mature API, there will be no functional or signature change - its mature!
So what is the harm of providing an alias back to Experimental until the next ABI version is declared?
So while yes, you are 100% right - experimental mean no guarantees.
But if the API is baked, it is not going to change, so I don't see the harm.
We don't want the unnecessary triumph of policy over pragmatism.
^ permalink raw reply [relevance 8%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-04 17:46 4% ` Trahe, Fiona
@ 2020-02-13 14:51 4% ` Kusztal, ArkadiuszX
0 siblings, 0 replies; 200+ results
From: Kusztal, ArkadiuszX @ 2020-02-13 14:51 UTC (permalink / raw)
To: Trahe, Fiona, Thomas Monjalon
Cc: David Marchand, nhorman, bluca, ktraynor, Ray Kinsella, dev,
Akhil Goyal, Yigit, Ferruh, Ananyev, Konstantin, dev,
Anoob Joseph, Richardson, Bruce, Mcnamara, John, dodji,
Andrew Rybchenko, aconole
Hi,
Two comments from me,
> > > The patch we're working on will provide two versions of
> > > rte_cryptodev_info_get(), both call the same PMD function from the
> dev_ops info_get fn ptr.
> > > The default version operates s as normal, the 19.11 version searches
> > > through the list returned by the PMD, looking for sym.aead.algo =
> > > ChaChaPoly, it needs to strip it from
> > the list.
> > > As PMDs just pass a ptr to their capabilities list ( it isn't a
> > > linked list, but an array with an end marker =
> > > RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST) if the API layer detects
> > > Chacha it must allocate some space and store a local copy of the trimmed
> list. This must be stored only once per device.
[Arek] The problem with this solution is that we need to allocate memory.
So the question is how to handle unlikely case of malloc error when we operate inside void function rte_cryptodev_info_get?
And even if we would pass somehow error condition to the caller then what to do is another question.
> >
> > I don't understand what you have to store.
> > Can't you just set the algo to 0 if it is ChaCha?
> [Fiona] it returns a pointer to data in the PMD domain, which the API couldn't
> and shouldn't overwrite, e.g.
> static const struct rte_cryptodev_capabilities qat_gen3_sym_capabilities[]
Should we print user some information
>
> >
> > > This versioning will apply to any PMD which wants to take advantage
> > > of the new API between now and
> > 20.11.
> > >
> > > Note, I expect the ABI checker tools will still complain of ABI
> > > breakage as the LIST_END value will still
> > change.
> >
> > Right, you need to update the ignore list for the tool.
> >
> > > We are also reviewing all other cryptodev APIs in case there is any other
> API which needs versioning.
> > >
> > > Anyone see any problem with this approach?
> >
> > The other issue is with all other functions accepting this enum as input.
> > We should continue returning an error if getting Chacha as input with
> > 19.11 version of these functions.
> > But I would tend to consider this small ABI breakage can be ignored as
> > it is in the error path.
> [Fiona] The QAT PMD tests for and handles this error. I expect other PMDs
> do too.
[Arek] - Yes, it is error path but on the other hand we explicitly specify what value we will return when calling
rte_cryptodev_sym_session_init so caller may expect EINVAL when wrong algorithm value selected (usually it probably will be ENOTSUP).
In this case when setting 3 (LIST_END) on 19.11 app and linking against 20.02 (assuming with Chacha) shared build, caller would get success on return and fully set chacha session,
which will probably result in undefined behavior.
So shouldn't this function be versioned as well then?
Regards,
Arek
^ permalink raw reply [relevance 4%]
* [dpdk-dev] DPDK Release Status Meeting 13/02/2020
@ 2020-02-13 11:20 3% Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-13 11:20 UTC (permalink / raw)
To: dpdk-dev; +Cc: Thomas Monjalon
Minutes 13 February 2020
------------------------
Agenda:
* Release Dates
* -rc2 status
* Subtrees
* OvS
Participants:
* Debian/Microsoft
* Intel
* Mellanox
* NXP
* Red Hat
Release Dates
-------------
* v20.02 dates:
* -rc2 is released on 6 February
* https://mails.dpdk.org/archives/dev/2020-February/157366.html
* -rc3 Friday 14 February 2020
* Release Friday 21 February 2020
* Release pushed because of extended PRC holidays
* v20.05 *updated* proposal, please comment:
* Proposal/V1: Wednesday 18 March 2020
* Integration/Merge/RC1: Friday 17 April 2020
* Release: Wednesday 20 May 2020
* The update request is based on delays in 20.02
* Release date postponed more taking into PRC holiday on 1-5 May into account
-rc2 Status
-----------
* -rc2 has been released last week, but it has not been fully tested by Intel
because of extended PRC holiday, expecting to have a test report tomorrow.
No issue reported from Mellanox testing.
* !! Can all vendors/contributors that tested the -rc1/-rc2 please share their
test results, either via email to mail list or in the release status meeting,
to increase our confidence to the release.
* Decided go on with the existing -rc3 plan, but if we find late critical issues
because of delayed testing, we may need to have interim -rc4/-rc5 releases.
* There is still some risk around release date, based on full validation output.
Subtrees
--------
* main
* Sub-trees are not ready yet, will pull them tomorrow for -rc3
* ABI fixes merged
* ACL patches under review
* Windows patches has been merged
* Lots of doc updates for -rc3 & -rc4
* Reminder of deprecation notices, some discussed but not sent yet
* Should be ready for -rc3 tomorrow
* next-net
* Some fixes merged, pulled from sub-trees, will ready for -rc3 end of today
* Will pull from mlx and virtio trees when they are ready today
* af_xdp fix series reviewed by @Xiaolong, will merge today
* Pulling from -brcm & -mrvl trees as well regularly
* next-net-crypto
* Only a couple of patches left for -rc3
* Doc patch for inline protocol
* One test crypto patch waiting reply from @Fiona
* next-net-eventdev
* No update
* next-net-virtio
* There will be a few more fixes for -rc3, waiting @Ivan
* next-net-intel
* Some fixes merged for -rc3
* next-net-mlx
* Some small fixes merged for -rc3, there will be a few more
* LTS
* 19.11.x
* Started working on 19.11.1, going smooth for now
* Will wait 20.02 release for having a -rc1 date
* Proposed to create 19.11 branch in stable tree
* 17.11.10-rc1 released, please test and report results
* https://mails.dpdk.org/archives/dev/2020-January/154915.html
* Got test results from Intel, Red Hat & Mellanox
* QAT regression reported by Intel testing, need help from developers
* mlx reported regressions issue resolved
* Waiting ETA for QAT issue until end of today
* If no update received, will proceed with the release
* 18.11.x
* Started to work on next LTS version 18.11.7
OvS
---
* 2.13 release should be around this week
* The experimental tag dropped in master may remain as experimental in LTS
This means the OvS may have to keep consuming API as experimental, DPDK
discussion thread on the issue:
* https://mails.dpdk.org/archives/dev/2020-February/157149.html
DPDK Release Status Meetings
============================
The DPDK Release Status Meeting is intended for DPDK Committers to discuss
the status of the master tree and sub-trees, and for project managers to
track progress or milestone dates.
The meeting occurs on Thursdays at 8:30 UTC. If you wish to attend just
send an email to "John McNamara <john.mcnamara@intel.com>" for the invite.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [RFC 1/4] ring: future proof flag settings
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 1/4] ring: future proof flag settings Stephen Hemminger
@ 2020-02-13 11:12 0% ` Ananyev, Konstantin
0 siblings, 0 replies; 200+ results
From: Ananyev, Konstantin @ 2020-02-13 11:12 UTC (permalink / raw)
To: Stephen Hemminger, dev
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Stephen Hemminger
> Sent: Wednesday, February 12, 2020 11:08 PM
> To: dev@dpdk.org
> Cc: Stephen Hemminger <stephen@networkplumber.org>
> Subject: [dpdk-dev] [RFC 1/4] ring: future proof flag settings
>
> All API's should check that they support the flag values
> passed. If an application passes an invalid flag it could
> cause problems in later ABI.
>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
> lib/librte_ring/rte_ring.c | 7 +++++++
> 1 file changed, 7 insertions(+)
>
> diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c
> index 77e5de099b81..6cd8831649ea 100644
> --- a/lib/librte_ring/rte_ring.c
> +++ b/lib/librte_ring/rte_ring.c
> @@ -100,6 +100,13 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
> RTE_BUILD_BUG_ON((offsetof(struct rte_ring, prod) &
> RTE_CACHE_LINE_MASK) != 0);
>
> + /* future proof flags, only allow supported values */
> + if (flags & ~(RING_F_SP_ENQ | RING_F_SC_DEQ)) {
> + RTE_LOG(ERR, RING,
> + "Unsupported flags requested %d\n", flags);
> + return -EINVAL;
> + }
> +
Good idea, I think, but seems you forgot RING_F_EXACT_SZ.
> /* init the ring structure */
> memset(r, 0, sizeof(*r));
> ret = strlcpy(r->name, name, sizeof(r->name));
> --
> 2.20.1
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [RFC 4/4] cfgfile: check flags value
2020-02-12 23:08 3% [dpdk-dev] [RFC 0/4] Enforce flag checking in API's Stephen Hemminger
` (2 preceding siblings ...)
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 3/4] stack: " Stephen Hemminger
@ 2020-02-12 23:08 3% ` Stephen Hemminger
3 siblings, 0 replies; 200+ results
From: Stephen Hemminger @ 2020-02-12 23:08 UTC (permalink / raw)
To: dev; +Cc: Stephen Hemminger
All API's should check that they support the flag values
passed. If an application passes an invalid flag it could
cause problems in later ABI.
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
lib/librte_cfgfile/rte_cfgfile.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/lib/librte_cfgfile/rte_cfgfile.c b/lib/librte_cfgfile/rte_cfgfile.c
index 9049fd9c2319..cec82c82ee20 100644
--- a/lib/librte_cfgfile/rte_cfgfile.c
+++ b/lib/librte_cfgfile/rte_cfgfile.c
@@ -272,6 +272,10 @@ rte_cfgfile_create(int flags)
int i;
struct rte_cfgfile *cfg;
+ /* future proof flags usage */
+ if (flags & ~(CFG_FLAG_GLOBAL_SECTION | CFG_FLAG_EMPTY_VALUES))
+ return NULL;
+
cfg = malloc(sizeof(*cfg));
if (cfg == NULL)
--
2.20.1
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [RFC 3/4] stack: check flags on creation
2020-02-12 23:08 3% [dpdk-dev] [RFC 0/4] Enforce flag checking in API's Stephen Hemminger
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 1/4] ring: future proof flag settings Stephen Hemminger
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 2/4] hash: check flags on creation Stephen Hemminger
@ 2020-02-12 23:08 3% ` Stephen Hemminger
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 4/4] cfgfile: check flags value Stephen Hemminger
3 siblings, 0 replies; 200+ results
From: Stephen Hemminger @ 2020-02-12 23:08 UTC (permalink / raw)
To: dev; +Cc: Stephen Hemminger
All API's should check that they support the flag values
passed. If an application passes an invalid flag it could
cause problems in later ABI.
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
lib/librte_stack/rte_stack.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/lib/librte_stack/rte_stack.c b/lib/librte_stack/rte_stack.c
index d19824f00439..4b09ff30d6bd 100644
--- a/lib/librte_stack/rte_stack.c
+++ b/lib/librte_stack/rte_stack.c
@@ -59,6 +59,11 @@ rte_stack_create(const char *name, unsigned int count, int socket_id,
unsigned int sz;
int ret;
+ if (flags & ~(RTE_STACK_F_LF)) {
+ STACK_LOG_ERR("Unsupported stack flags %#x\n", flags);
+ return NULL;
+ }
+
#ifdef RTE_ARCH_64
RTE_BUILD_BUG_ON(sizeof(struct rte_stack_lf_head) != 16);
#else
--
2.20.1
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [RFC 2/4] hash: check flags on creation
2020-02-12 23:08 3% [dpdk-dev] [RFC 0/4] Enforce flag checking in API's Stephen Hemminger
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 1/4] ring: future proof flag settings Stephen Hemminger
@ 2020-02-12 23:08 3% ` Stephen Hemminger
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 3/4] stack: " Stephen Hemminger
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 4/4] cfgfile: check flags value Stephen Hemminger
3 siblings, 0 replies; 200+ results
From: Stephen Hemminger @ 2020-02-12 23:08 UTC (permalink / raw)
To: dev; +Cc: Stephen Hemminger
All API's should check that they support the flag values
passed. If an application passes an invalid flag it could
cause problems in later ABI.
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
lib/librte_hash/rte_cuckoo_hash.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/lib/librte_hash/rte_cuckoo_hash.c b/lib/librte_hash/rte_cuckoo_hash.c
index 6af8ca42e94f..ab9cd91b4955 100644
--- a/lib/librte_hash/rte_cuckoo_hash.c
+++ b/lib/librte_hash/rte_cuckoo_hash.c
@@ -32,6 +32,9 @@
#include "rte_hash.h"
#include "rte_cuckoo_hash.h"
+/* Mask of all flags supported by this version */
+#define RTE_HASH_EXTRA_FLAGS_MASK 0x2f
+
#define FOR_EACH_BUCKET(CURRENT_BKT, START_BUCKET) \
for (CURRENT_BKT = START_BUCKET; \
CURRENT_BKT != NULL; \
@@ -164,6 +167,12 @@ rte_hash_create(const struct rte_hash_parameters *params)
return NULL;
}
+ if (params->extra_flag & ~RTE_HASH_EXTRA_FLAGS_MASK) {
+ rte_errno = EINVAL;
+ RTE_LOG(ERR, HASH, "rte_hash_create: unsupported extra flags\n");
+ return NULL;
+ }
+
/* Validate correct usage of extra options */
if ((params->extra_flag & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY) &&
(params->extra_flag & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF)) {
--
2.20.1
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [RFC 1/4] ring: future proof flag settings
2020-02-12 23:08 3% [dpdk-dev] [RFC 0/4] Enforce flag checking in API's Stephen Hemminger
@ 2020-02-12 23:08 3% ` Stephen Hemminger
2020-02-13 11:12 0% ` Ananyev, Konstantin
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 2/4] hash: check flags on creation Stephen Hemminger
` (2 subsequent siblings)
3 siblings, 1 reply; 200+ results
From: Stephen Hemminger @ 2020-02-12 23:08 UTC (permalink / raw)
To: dev; +Cc: Stephen Hemminger
All API's should check that they support the flag values
passed. If an application passes an invalid flag it could
cause problems in later ABI.
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
lib/librte_ring/rte_ring.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c
index 77e5de099b81..6cd8831649ea 100644
--- a/lib/librte_ring/rte_ring.c
+++ b/lib/librte_ring/rte_ring.c
@@ -100,6 +100,13 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
RTE_BUILD_BUG_ON((offsetof(struct rte_ring, prod) &
RTE_CACHE_LINE_MASK) != 0);
+ /* future proof flags, only allow supported values */
+ if (flags & ~(RING_F_SP_ENQ | RING_F_SC_DEQ)) {
+ RTE_LOG(ERR, RING,
+ "Unsupported flags requested %d\n", flags);
+ return -EINVAL;
+ }
+
/* init the ring structure */
memset(r, 0, sizeof(*r));
ret = strlcpy(r->name, name, sizeof(r->name));
--
2.20.1
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [RFC 0/4] Enforce flag checking in API's
@ 2020-02-12 23:08 3% Stephen Hemminger
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 1/4] ring: future proof flag settings Stephen Hemminger
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: Stephen Hemminger @ 2020-02-12 23:08 UTC (permalink / raw)
To: dev; +Cc: Stephen Hemminger
Since the DPDK api's are lazy about checking for flag values
it means the flags can never be extended in future releases
without causing ABI breakage. This means we end up doing unnecessary
symbol versioning just to work around applications that might
pass in naughty bits.
This is the DPDK analog of the Linux kernel openat() problem.
Openat api was added but since kernel did not check flags it
ended up that another syscall openat2() was necessary before
the flags could be used.
Stephen Hemminger (4):
ring: future proof flag settings
hash: check flags on creation
stack: check flags on creation
cfgfile: check flags value
lib/librte_cfgfile/rte_cfgfile.c | 4 ++++
lib/librte_hash/rte_cuckoo_hash.c | 9 +++++++++
lib/librte_ring/rte_ring.c | 7 +++++++
lib/librte_stack/rte_stack.c | 5 +++++
4 files changed, 25 insertions(+)
--
2.20.1
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH 1/2] common/mlx5: fix ABI version
2020-02-11 19:48 7% [dpdk-dev] [PATCH 1/2] common/mlx5: fix " Matan Azrad
2020-02-11 19:48 7% ` [dpdk-dev] [PATCH 2/2] vdpa/mlx5: " Matan Azrad
2020-02-11 20:31 4% ` [dpdk-dev] [PATCH 1/2] common/mlx5: " Slava Ovsiienko
@ 2020-02-12 8:53 4% ` David Marchand
2 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-12 8:53 UTC (permalink / raw)
To: Matan Azrad; +Cc: dev, Viacheslav Ovsiienko, Thomas Monjalon
On Tue, Feb 11, 2020 at 8:48 PM Matan Azrad <matan@mellanox.com> wrote:
>
> Changed the ABI version to 20.0.1.
>
> Fixes: 7b4f1e6bd367 ("common/mlx5: introduce common library")
>
> Signed-off-by: Matan Azrad <matan@mellanox.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
Series applied, thanks.
--
David Marchand
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH 0/7] vfio/pci: SR-IOV support
@ 2020-02-11 23:05 2% Alex Williamson
2020-02-14 4:57 0% ` Alexey Kardashevskiy
0 siblings, 1 reply; 200+ results
From: Alex Williamson @ 2020-02-11 23:05 UTC (permalink / raw)
To: kvm
Cc: linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, bruce.richardson, cohuck
Given the mostly positive feedback from the RFC[1], here's a new
non-RFC revision. Changes since RFC:
- vfio_device_ops.match semantics refined
- Use helpers for struct pci_dev.physfn to avoid breakage without
CONFIG_PCI_IOV
- Relax to allow SR-IOV configuration changes while PF is opened.
There are potentially interesting use cases here, including
perhaps QEMU emulating an SR-IOV capability and calling out
to a privileged entity to manipulate sriov_numvfs and corral
the resulting devices.
- Retest vfio_device_feature.argsz to include uuid length.
- Add Connie's R-b on 6/7
I still wish we had a solution to make it less opaque to the user
why a VFIO_GROUP_GET_DEVICE_FD() has failed if a VF token is
required, but this is still the best I've been able to come up with.
If there are objections or better ideas, please raise them now.
The synopsis of this series is that we have an ongoing desire to drive
PCIe SR-IOV PFs from userspace with VFIO. There's an immediate need
for this with DPDK drivers and potentially interesting future use
cases in virtualization. We've been reluctant to add this support
previously due to the dependency and trust relationship between the
VF device and PF driver. Minimally the PF driver can induce a denial
of service to the VF, but depending on the specific implementation,
the PF driver might also be responsible for moving data between VFs
or have direct access to the state of the VF, including data or state
otherwise private to the VF or VF driver.
To help resolve these concerns, we introduce a VF token into the VFIO
PCI ABI, which acts as a shared secret key between drivers. The
userspace PF driver is required to set the VF token to a known value
and userspace VF drivers are required to provide the token to access
the VF device. If a PF driver is restarted with VF drivers in use, it
must also provide the current token in order to prevent a rogue
untrusted PF driver from replacing a known driver. The degree to
which this new token is considered secret is left to the userspace
drivers, the kernel intentionally provides no means to retrieve the
current token.
Note that the above token is only required for this new model where
both the PF and VF devices are usable through vfio-pci. Existing
models of VFIO drivers where the PF is used without SR-IOV enabled
or the VF is bound to a userspace driver with an in-kernel, host PF
driver are unaffected.
The latter configuration above also highlights a new inverted scenario
that is now possible, a userspace PF driver with in-kernel VF drivers.
I believe this is a scenario that should be allowed, but should not be
enabled by default. This series includes code to set a default
driver_override for VFs sourced from a vfio-pci user owned PF, such
that the VFs are also bound to vfio-pci. This model is compatible
with tools like driverctl and allows the system administrator to
decide if other bindings should be enabled. The VF token interface
above exists only between vfio-pci PF and VF drivers, once a VF is
bound to another driver, the administrator has effectively pronounced
the device as trusted. The vfio-pci driver will note alternate
binding in dmesg for logging and debugging purposes.
Please review, comment, and test. The example QEMU implementation
provided with the RFC[2] is still current for this version. Thanks,
Alex
[1] https://lore.kernel.org/lkml/158085337582.9445.17682266437583505502.stgit@gimli.home/
[2] https://lore.kernel.org/lkml/20200204161737.34696b91@w520.home/
---
Alex Williamson (7):
vfio: Include optional device match in vfio_device_ops callbacks
vfio/pci: Implement match ops
vfio/pci: Introduce VF token
vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
vfio/pci: Add sriov_configure support
vfio/pci: Remove dev_fmt definition
vfio/pci: Cleanup .probe() exit paths
drivers/vfio/pci/vfio_pci.c | 312 ++++++++++++++++++++++++++++++++---
drivers/vfio/pci/vfio_pci_private.h | 10 +
drivers/vfio/vfio.c | 20 ++
include/linux/vfio.h | 4
include/uapi/linux/vfio.h | 37 ++++
5 files changed, 355 insertions(+), 28 deletions(-)
^ permalink raw reply [relevance 2%]
* Re: [dpdk-dev] [PATCH 2/2] vdpa/mlx5: fix ABI version
2020-02-11 19:48 7% ` [dpdk-dev] [PATCH 2/2] vdpa/mlx5: " Matan Azrad
@ 2020-02-11 20:31 4% ` Slava Ovsiienko
0 siblings, 0 replies; 200+ results
From: Slava Ovsiienko @ 2020-02-11 20:31 UTC (permalink / raw)
To: Matan Azrad, dev; +Cc: Thomas Monjalon, David Marchand
> -----Original Message-----
> From: Matan Azrad <matan@mellanox.com>
> Sent: Tuesday, February 11, 2020 21:49
> To: dev@dpdk.org
> Cc: Slava Ovsiienko <viacheslavo@mellanox.com>; Thomas Monjalon
> <thomas@monjalon.net>; David Marchand <david.marchand@redhat.com>
> Subject: [PATCH 2/2] vdpa/mlx5: fix ABI version
>
> Changed the ABI version to 20.0.1.
>
> Fixes: 95276abaaf0a ("vdpa/mlx5: introduce Mellanox vDPA driver")
>
> Signed-off-by: Matan Azrad <matan@mellanox.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> ---
> drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map
> b/drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map
> index 143836e..179f7f1 100644
> --- a/drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map
> +++ b/drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map
> @@ -1,3 +1,3 @@
> -DPDK_20.02 {
> +DPDK_20.0.1 {
> local: *;
> };
> --
> 1.8.3.1
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH 1/2] common/mlx5: fix ABI version
2020-02-11 19:48 7% [dpdk-dev] [PATCH 1/2] common/mlx5: fix " Matan Azrad
2020-02-11 19:48 7% ` [dpdk-dev] [PATCH 2/2] vdpa/mlx5: " Matan Azrad
@ 2020-02-11 20:31 4% ` Slava Ovsiienko
2020-02-12 8:53 4% ` David Marchand
2 siblings, 0 replies; 200+ results
From: Slava Ovsiienko @ 2020-02-11 20:31 UTC (permalink / raw)
To: Matan Azrad, dev; +Cc: Thomas Monjalon, David Marchand
> -----Original Message-----
> From: Matan Azrad <matan@mellanox.com>
> Sent: Tuesday, February 11, 2020 21:49
> To: dev@dpdk.org
> Cc: Slava Ovsiienko <viacheslavo@mellanox.com>; Thomas Monjalon
> <thomas@monjalon.net>; David Marchand <david.marchand@redhat.com>
> Subject: [PATCH 1/2] common/mlx5: fix ABI version
>
> Changed the ABI version to 20.0.1.
>
> Fixes: 7b4f1e6bd367 ("common/mlx5: introduce common library")
>
> Signed-off-by: Matan Azrad <matan@mellanox.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> ---
> drivers/common/mlx5/rte_common_mlx5_version.map | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map
> b/drivers/common/mlx5/rte_common_mlx5_version.map
> index b1a2014..aede2a0 100644
> --- a/drivers/common/mlx5/rte_common_mlx5_version.map
> +++ b/drivers/common/mlx5/rte_common_mlx5_version.map
> @@ -1,4 +1,4 @@
> -DPDK_20.02 {
> +DPDK_20.0.1 {
> global:
>
> mlx5_class_get;
> --
> 1.8.3.1
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH 1/2] common/mlx5: fix ABI version
@ 2020-02-11 19:48 7% Matan Azrad
2020-02-11 19:48 7% ` [dpdk-dev] [PATCH 2/2] vdpa/mlx5: " Matan Azrad
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Matan Azrad @ 2020-02-11 19:48 UTC (permalink / raw)
To: dev; +Cc: Viacheslav Ovsiienko, Thomas Monjalon, David Marchand
Changed the ABI version to 20.0.1.
Fixes: 7b4f1e6bd367 ("common/mlx5: introduce common library")
Signed-off-by: Matan Azrad <matan@mellanox.com>
---
drivers/common/mlx5/rte_common_mlx5_version.map | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index b1a2014..aede2a0 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -1,4 +1,4 @@
-DPDK_20.02 {
+DPDK_20.0.1 {
global:
mlx5_class_get;
--
1.8.3.1
^ permalink raw reply [relevance 7%]
* [dpdk-dev] [PATCH 2/2] vdpa/mlx5: fix ABI version
2020-02-11 19:48 7% [dpdk-dev] [PATCH 1/2] common/mlx5: fix " Matan Azrad
@ 2020-02-11 19:48 7% ` Matan Azrad
2020-02-11 20:31 4% ` Slava Ovsiienko
2020-02-11 20:31 4% ` [dpdk-dev] [PATCH 1/2] common/mlx5: " Slava Ovsiienko
2020-02-12 8:53 4% ` David Marchand
2 siblings, 1 reply; 200+ results
From: Matan Azrad @ 2020-02-11 19:48 UTC (permalink / raw)
To: dev; +Cc: Viacheslav Ovsiienko, Thomas Monjalon, David Marchand
Changed the ABI version to 20.0.1.
Fixes: 95276abaaf0a ("vdpa/mlx5: introduce Mellanox vDPA driver")
Signed-off-by: Matan Azrad <matan@mellanox.com>
---
drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map b/drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map
index 143836e..179f7f1 100644
--- a/drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map
+++ b/drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map
@@ -1,3 +1,3 @@
-DPDK_20.02 {
+DPDK_20.0.1 {
local: *;
};
--
1.8.3.1
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH 1/1] raw/octeontx2_ep: update the ABI version
2020-02-11 11:27 7% [dpdk-dev] [PATCH 1/1] raw/octeontx2_ep: update the ABI version Mahipal Challa
@ 2020-02-11 14:05 4% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-11 14:05 UTC (permalink / raw)
To: Mahipal Challa
Cc: Thomas Monjalon, dev, Jerin Jacob Kollanukkaran, pathreya,
snilla, venkatn
On Tue, Feb 11, 2020 at 12:29 PM Mahipal Challa <mchalla@marvell.com> wrote:
>
> Changed the ABI version to 20.0.1.
>
> Fixes: 56d46d13f736 ("raw/octeontx2_ep: add build infra and device probe")
>
> Signed-off-by: Mahipal Challa <mchalla@marvell.com>
> ---
> drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map b/drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map
> index ff357af..bc8fd6d 100644
> --- a/drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map
> +++ b/drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map
> @@ -1,4 +1,4 @@
> -DPDK_20.02 {
> +DPDK_20.0.1 {
>
> local: *;
> };
> --
> 1.8.3.1
>
Acked-by: David Marchand <david.marchand@redhat.com>
Applied, thanks.
--
David Marchand
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] common/octeontx2: update version map
2020-02-10 10:27 3% [dpdk-dev] [PATCH] common/octeontx2: update version map Anoob Joseph
@ 2020-02-11 14:04 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-11 14:04 UTC (permalink / raw)
To: Anoob Joseph
Cc: Akhil Goyal, Jerin Jacob, Thomas Monjalon, Narayana Prasad, dev
On Mon, Feb 10, 2020 at 11:28 AM Anoob Joseph <anoobj@marvell.com> wrote:
>
> Move symbols added in 20.02 release to DPDK_20.0.1 ABI.
>
> Fixes: d06551535a09 ("common/octeontx2: add security capability routine")
> Fixes: 3fe4d07d1678 ("crypto/octeontx2: enable CPT to share QP with ethdev")
> Fixes: f44e71637755 ("net/octeontx2: add security session operations")
>
> Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Small fix below.
Acked-by: David Marchand <david.marchand@redhat.com>
> ---
> .../common/octeontx2/rte_common_octeontx2_version.map | 17 +++++++++++------
> 1 file changed, 11 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/common/octeontx2/rte_common_octeontx2_version.map b/drivers/common/octeontx2/rte_common_octeontx2_version.map
> index 19a7b19..a25fc69 100644
> --- a/drivers/common/octeontx2/rte_common_octeontx2_version.map
> +++ b/drivers/common/octeontx2/rte_common_octeontx2_version.map
> @@ -5,7 +5,6 @@ DPDK_20.0 {
> otx2_dev_fini;
> otx2_dev_priv_init;
> otx2_disable_irqs;
> - otx2_eth_dev_is_sec_capable;
> otx2_intra_dev_get_cfg;
> otx2_logtype_base;
> otx2_logtype_dpi;
> @@ -28,11 +27,6 @@ DPDK_20.0 {
> otx2_npa_pf_func_get;
> otx2_npa_set_defaults;
> otx2_register_irq;
> - otx2_sec_idev_cfg_init;
> - otx2_sec_idev_tx_cpt_qp_add;
> - otx2_sec_idev_tx_cpt_qp_remove;
> - otx2_sec_idev_tx_cpt_qp_get;
> - otx2_sec_idev_tx_cpt_qp_put;
> otx2_sso_pf_func_get;
> otx2_sso_pf_func_set;
> otx2_unregister_irq;
> @@ -40,6 +34,17 @@ DPDK_20.0 {
> local: *;
> };
>
> +DPDK_20.0.1 {
> + global:
> +
> + otx2_eth_dev_is_sec_capable;
> + otx2_sec_idev_cfg_init;
> + otx2_sec_idev_tx_cpt_qp_add;
> + otx2_sec_idev_tx_cpt_qp_remove;
> + otx2_sec_idev_tx_cpt_qp_get;
> + otx2_sec_idev_tx_cpt_qp_put;
> +};
> +
DPDK_20.0.1 must inherit from DPDK_20.0 node.
I fixed it.
Applied, thanks.
--
David Marchand
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH 1/1] raw/octeontx2_ep: update the ABI version
@ 2020-02-11 11:27 7% Mahipal Challa
2020-02-11 14:05 4% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Mahipal Challa @ 2020-02-11 11:27 UTC (permalink / raw)
To: thomas, dev; +Cc: jerinj, pathreya, snilla, venkatn
Changed the ABI version to 20.0.1.
Fixes: 56d46d13f736 ("raw/octeontx2_ep: add build infra and device probe")
Signed-off-by: Mahipal Challa <mchalla@marvell.com>
---
drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map b/drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map
index ff357af..bc8fd6d 100644
--- a/drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map
+++ b/drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map
@@ -1,4 +1,4 @@
-DPDK_20.02 {
+DPDK_20.0.1 {
local: *;
};
--
1.8.3.1
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [RFC PATCH 0/7] vfio/pci: SR-IOV support
2020-02-04 23:05 3% [dpdk-dev] [RFC PATCH 0/7] vfio/pci: SR-IOV support Alex Williamson
@ 2020-02-11 11:18 0% ` Jerin Jacob
0 siblings, 0 replies; 200+ results
From: Jerin Jacob @ 2020-02-11 11:18 UTC (permalink / raw)
To: Alex Williamson
Cc: kvm, linux-pci, linux-kernel, dpdk-dev, mtosatti,
Thomas Monjalon, Luca Boccassi, Richardson, Bruce, cohuck,
Vamsi Attunuru
On Wed, Feb 5, 2020 at 4:35 AM Alex Williamson
<alex.williamson@redhat.com> wrote:
>
> There seems to be an ongoing desire to use userspace, vfio-based
> drivers for both SR-IOV PF and VF devices. The fundamental issue
> with this concept is that the VF is not fully independent of the PF
> driver. Minimally the PF driver might be able to deny service to the
> VF, VF data paths might be dependent on the state of the PF device,
> or the PF my have some degree of ability to inspect or manipulate the
> VF data. It therefore would seem irresponsible to unleash VFs onto
> the system, managed by a user owned PF.
>
> We address this in a few ways in this series. First, we can use a bus
> notifier and the driver_override facility to make sure VFs are bound
> to the vfio-pci driver by default. This should eliminate the chance
> that a VF is accidentally bound and used by host drivers. We don't
> however remove the ability for a host admin to change this override.
>
> The next issue we need to address is how we let userspace drivers
> opt-in to this participation with the PF driver. We do not want an
> admin to be able to unwittingly assign one of these VFs to a tenant
> that isn't working in collaboration with the PF driver. We could use
> IOMMU grouping, but this seems to push too far towards tightly coupled
> PF and VF drivers. This series introduces a "VF token", implemented
> as a UUID, as a shared secret between PF and VF drivers. The token
> needs to be set by the PF driver and used as part of the device
> matching by the VF driver. Provisions in the code also account for
> restarting the PF driver with active VF drivers, requiring the PF to
> use the current token to re-gain access to the PF.
Thanks Alex for the series. DPDK realizes this use-case through, an out of
tree igb_uio module, for non VFIO devices. Supporting this use case, with
VFIO, will be a great enhancement for DPDK as we are planning to
get rid of out of tree modules any focus only on userspace aspects.
From the DPDK perspective, we have following use-cases
1) VF representer or OVS/vSwitch use cases where
DPDK PF acts as an HW switch to steer traffic to VF
using the rte_flow library backed by HW CAMs.
2) Unlike, other PCI class of devices, Network class of PCIe devices
would have additional
capability on the PF devices such as promiscuous mode support etc
leverage that in DPDK
PF and VF use cases.
That would boil down to the use of the following topology.
a) PF bound to DPDK/VFIO and VF bound to Linux
b) PF bound to DPDK/VFIO and VF bound to DPDK/VFIO
Tested the use case (a) and it works this patch. Tested use case(b), it
works with patch provided both PF and VF under the same application.
Regarding the use case where PF bound to DPDK/VFIO and
VF bound to DPDK/VFIO are _two different_ processes then sharing the UUID
will be a little tricky thing in terms of usage. But if that is the
purpose of bringing
UUID to the equation then it fine.
Overall this series looks good to me. We can test the next non-RFC
series and give
Tested-by by after testing with DPDK.
>
> The above solutions introduce a bit of a modification to the VFIO ABI
> and an additional ABI extension. The modification is that the
> VFIO_GROUP_GET_DEVICE_FD ioctl is specified to require a char string
> from the user providing the device name. For this solution, we extend
> the syntax to allow the device name followed by key/value pairs. In
> this case we add "vf_token=3e7e882e-1daf-417f-ad8d-882eea5ee337", for
> example. These options are expected to be space separated. Matching
> these key/value pairs is entirely left to the vfio bus driver (ex.
> vfio-pci) and the internal ops structure is extended to allow this
> optional support. This extension should be fully backwards compatible
> to existing userspace, such code will simply fail to open these newly
> exposed devices, as intended.
>
> I've been debating whether instead of the above we should allow the
> user to get the device fd as normal, but restrict the interfaces until
> the user authenticates, but I'm afraid this would be a less backwards
> compatible solution. It would be just as unclear to the user why a
> device read/write/mmap/ioctl failed as it might be to why getting the
> device fd could fail. However in the latter case, I believe we do a
> better job of restricting how far userspace code might go before they
> ultimately fail. I'd welcome discussion in the space, and or course
> the extension of the GET_DEVICE_FD string.
>
> Finally, the user needs to be able to set a VF token. I add a
> VFIO_DEVICE_FEATURE ioctl for this that's meant to be reusable for
> getting, setting, and probing arbitrary features of a device.
>
> I'll reply to this cover letter with a very basic example of a QEMU
> update to support this interface, though I haven't found a device yet
> that behaves well with the PF running in one VM with the VF in
> another, or really even just a PF running in a VM with SR-IOV enabled.
> I know these devices exist though, and I suspect QEMU will not be the
> primary user of this support for now, but this behavior reaffirms my
> concerns to prevent mis-use.
>
> Please comment. In particular, does this approach meet the DPDK needs
> for userspace PF and VF drivers, with the hopefully minor hurdle of
> sharing a token between drivers. The token is of course left to
> userspace how to manage, and might be static (and not very secret) for
> a given set of drivers. Thanks,
>
> Alex
>
> ---
>
> Alex Williamson (7):
> vfio: Include optional device match in vfio_device_ops callbacks
> vfio/pci: Implement match ops
> vfio/pci: Introduce VF token
> vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
> vfio/pci: Add sriov_configure support
> vfio/pci: Remove dev_fmt definition
> vfio/pci: Cleanup .probe() exit paths
>
>
> drivers/vfio/pci/vfio_pci.c | 315 ++++++++++++++++++++++++++++++++---
> drivers/vfio/pci/vfio_pci_private.h | 10 +
> drivers/vfio/vfio.c | 19 ++
> include/linux/vfio.h | 3
> include/uapi/linux/vfio.h | 37 ++++
> 5 files changed, 356 insertions(+), 28 deletions(-)
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v1] common/iavf: update the ABI version
2020-02-06 12:04 7% [dpdk-dev] [PATCH v1] common/iavf: update the ABI version Haiyue Wang
@ 2020-02-11 2:08 4% ` Ye Xiaolong
0 siblings, 0 replies; 200+ results
From: Ye Xiaolong @ 2020-02-11 2:08 UTC (permalink / raw)
To: Haiyue Wang; +Cc: dev, thomas
On 02/06, Haiyue Wang wrote:
>The new symbols should be in ABI version 20.0.1.
>
>Fixes: 89214fe915b8 ("net/iavf/base: move to drivers common directory")
>
>Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
>---
> drivers/common/iavf/rte_common_iavf_version.map | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
>diff --git a/drivers/common/iavf/rte_common_iavf_version.map b/drivers/common/iavf/rte_common_iavf_version.map
>index 1a0839262..2f11d67c0 100644
>--- a/drivers/common/iavf/rte_common_iavf_version.map
>+++ b/drivers/common/iavf/rte_common_iavf_version.map
>@@ -1,4 +1,4 @@
>-DPDK_20.0 {
>+DPDK_20.0.1 {
> global:
>
> iavf_init_adminq;
>--
>2.17.1
>
Acked-by: Xiaolong Ye <xiaolong.ye@intel.com>
Applied to dpdk-next-net-intel, Thanks.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v7] eal: add manual probing option
2020-02-10 15:27 0% ` Thomas Monjalon
@ 2020-02-10 16:33 0% ` Jerin Jacob
0 siblings, 0 replies; 200+ results
From: Jerin Jacob @ 2020-02-10 16:33 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Jerin Jacob Kollanukkaran, Gaetan Rivet,
Pavan Nikhilesh Bhagavatula, Vamsi Krishna Attunuru,
David Marchand, dpdk-dev, Richardson, Bruce
On Mon, Feb 10, 2020 at 8:57 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 10/02/2020 15:51, Jerin Jacob:
> > On Tue, Feb 4, 2020 at 9:32 PM Gaetan Rivet <grive@u256.net> wrote:
> > >
> > > On 04/02/2020 16:06, Thomas Monjalon wrote:
> > > > 04/02/2020 13:43, Gaetan Rivet:
> > > >> On 04/02/2020 12:07, Thomas Monjalon wrote:
> > > >>> 04/02/2020 11:03, Gaetan Rivet:
> > > >>>> On 03/02/2020 23:21, Thomas Monjalon wrote:
> > > >>>>> 03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
> > > >>>>>> @David Marchand @thomas@monjalon.net
> > > >>>>>>
> > > >>>>>> Ping?
> > > >>>>>>
> > > >>>>>> Are there any more changes required for this patch? It's been in queue since last October.
> > > >>>>>
> > > >>>>> Sorry we have not decided whether it is a good idea or not.
> > > >>>>>
> > > >>>>> All changes related to probing are very sensitive,
> > > >>>>> and we know a big refactoring would be better than stacking
> > > >>>>> more and more options and corner cases.
> > > >>>>>
> > > >>>>> As we are busy with ABI stability stuff, we did not allocate
> > > >>>>> enough time to properly think about this feature.
> > > >>>>> Please accept our apologies, and let's consider it as
> > > >>>>> a high priority for 20.05 cycle.
> > > >>>>>
> > > >>>>
> > > >>>> Hello Thomas,
> > > >>>>
> > > >>>> This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.
> > > >>>>
> > > >>>> The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.
> > > >>>
> > > >>>
> > > >>> Yes, life is full of bad decisions and consequences.
> > > >>
> > > >>
> > > >> Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.
> > > >>
> > > >>>
> > > >>> I still think there is a risk in adding new user expectations,
> > > >>> and maintaining some code to workaround unknown issues.
> > > >>>
> > > >>> The real question here is to know why this patch?
> > > >>> Is it to workaround a broken driver?
> > > >>> Or to workaround a broken design in EAL and bus drivers?
> > > >>
> > > >> Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).
> > > >>
> > > >> I'm not sure having a dependent-probe by default is good, and that would be a big change.
> > > >>
> > > >> If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.
> > > >>
> > > >> [1]: First proposal:
> > > >> http://mails.dpdk.org/archives/dev/2019-September/144166.html
> > > >> My arguments:
> > > >> http://mails.dpdk.org/archives/dev/2019-September/144564.html
> > > >
> > > >
> > > > OK so there are two needs:
> > > >
> > > > 1/ Better control whitelist/blacklist mode.
> > > > We already know that a rework is needed here.
> > > > Unfortunately neither you or me had time to work on it,
> > > > and others who were interested disappeared.
> > > >
> > > > 2/ Associate ports with equivalent properties in applications.
> > > > This must be done in applications.
> > > > Tweaking the probe order is a hack.
> > > >
> > > >
> > >
> > > An application that want to tightly control the port init order, currently (by doing exactly like me here, hotpluging one by one the ports), would still need the bigger hack that consist in inserting a whitelist PCI devargs with a dummy address, depending on a undocumented PCI bus feature consisting in ignoring matching errors but keeping probing policy from failed devargs processing.
> > >
> > > Instead, with this patch this app can do
> > >
> > > rte_eal_manual_probe_set(1);
> > > rte_eal_init();
> > >
> > > to have the same behavior and be able to hotplug ports as it sees fit.
> > >
> > > You are worried about creating user expectations about this behavior (being forced to replicate in some way the functionality during the rewrite, as I understand it?), but then you are currently forcing users to expect this workaround to exist in the PCI bus, blocking devs from touching it as it will thus break current app configurations. I've seen systemd unit file using this -w dummy flag, as well as the programmatic equivalent. Which is better, to have to rework it cutting short these configs, or to propose beforehand a deprecation path?.
> > >
> > > This rework won't happen in 20.05, nor in the medium term unless you decide to drive this change. This workaround serves three needs (PCI normalization, port congruence and port dependency) in a low-risk implementation.
> >
> > Thomas,
> >
> > What would be the resolution of this? What is your recommendation to
> > fix the issue as you have the concern of this patch?
> >
> > Issue:
> > 1) When l2fwd does the forwarding for simplicity and performance
> > reason it just xor the port to find the destination port to forward.
> > 2) If the adjacent ports are not symmetrical(example: one is 40G and
> > other 10G) then forwarding will drop the packets.
> >
> > So, either
> > a) We need to control the probing order
> >
> > b) Or Application need
> > 1) To track the symmetrical ports and maintain the forwarding table OR
> > 2) Have the command-line option to specify destination port like l3fwd.
> >
> > We can fix it in the application, but do we need to complicate l2fwd?
> > I am fine with that, if that is consensus.
>
> You are describing an application issue,
> that's why I believe it should be fixed in the application.
Thanks for the quick reply and I agree.
>
> Should we have a command line option to configure the forwarding rules
> in the application (2)? I think yes.
> Should we implement an application logic to automatically create
> the best forwarding rules (1)? It would be nice, but anyway,
> I think we need manual config (2) as a fallback.
>
>
> > Thoughts? If you think, there is a rework needed in eal then could you
> > enumerate the items for the rework.
>
> Sorry I don't have time to describe dive into EAL probing and
> enumerate the items to rework.
> The most important issues I remind are:
> - white/blacklist policy is a mess and should be done in a higher layer
> - devargs syntax should allow generic matching (thanks to class awareness and generic syntax)
>
> Starting from these 2 items, we could imagine a generic path to
> disable automatic probing, but I think the l2fwd logic should not
> rely on probing order anyway.
+ Bruce as l2fwd maintainer.
Since in any case, l2fwd needs to be updated, We will focus on l2fwd
change for v20.05 and leaving the fate of this patch to EAL
maintainers.
Let us know, Are we are OK with below change in l2fwd as
- Introduce an array-based port lookup table instead of hardcoding to
xor based lookup.
- if no argument specified fill dest port as xor of source
- If argument is specified override the lookup table with a
user-specified destination port.
>
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v7] eal: add manual probing option
2020-02-10 14:51 0% ` Jerin Jacob
@ 2020-02-10 15:27 0% ` Thomas Monjalon
2020-02-10 16:33 0% ` Jerin Jacob
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-10 15:27 UTC (permalink / raw)
To: Jerin Jacob, Jerin Jacob Kollanukkaran
Cc: Gaetan Rivet, Pavan Nikhilesh Bhagavatula,
Vamsi Krishna Attunuru, David Marchand, dpdk-dev, Richardson,
Bruce
10/02/2020 15:51, Jerin Jacob:
> On Tue, Feb 4, 2020 at 9:32 PM Gaetan Rivet <grive@u256.net> wrote:
> >
> > On 04/02/2020 16:06, Thomas Monjalon wrote:
> > > 04/02/2020 13:43, Gaetan Rivet:
> > >> On 04/02/2020 12:07, Thomas Monjalon wrote:
> > >>> 04/02/2020 11:03, Gaetan Rivet:
> > >>>> On 03/02/2020 23:21, Thomas Monjalon wrote:
> > >>>>> 03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
> > >>>>>> @David Marchand @thomas@monjalon.net
> > >>>>>>
> > >>>>>> Ping?
> > >>>>>>
> > >>>>>> Are there any more changes required for this patch? It's been in queue since last October.
> > >>>>>
> > >>>>> Sorry we have not decided whether it is a good idea or not.
> > >>>>>
> > >>>>> All changes related to probing are very sensitive,
> > >>>>> and we know a big refactoring would be better than stacking
> > >>>>> more and more options and corner cases.
> > >>>>>
> > >>>>> As we are busy with ABI stability stuff, we did not allocate
> > >>>>> enough time to properly think about this feature.
> > >>>>> Please accept our apologies, and let's consider it as
> > >>>>> a high priority for 20.05 cycle.
> > >>>>>
> > >>>>
> > >>>> Hello Thomas,
> > >>>>
> > >>>> This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.
> > >>>>
> > >>>> The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.
> > >>>
> > >>>
> > >>> Yes, life is full of bad decisions and consequences.
> > >>
> > >>
> > >> Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.
> > >>
> > >>>
> > >>> I still think there is a risk in adding new user expectations,
> > >>> and maintaining some code to workaround unknown issues.
> > >>>
> > >>> The real question here is to know why this patch?
> > >>> Is it to workaround a broken driver?
> > >>> Or to workaround a broken design in EAL and bus drivers?
> > >>
> > >> Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).
> > >>
> > >> I'm not sure having a dependent-probe by default is good, and that would be a big change.
> > >>
> > >> If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.
> > >>
> > >> [1]: First proposal:
> > >> http://mails.dpdk.org/archives/dev/2019-September/144166.html
> > >> My arguments:
> > >> http://mails.dpdk.org/archives/dev/2019-September/144564.html
> > >
> > >
> > > OK so there are two needs:
> > >
> > > 1/ Better control whitelist/blacklist mode.
> > > We already know that a rework is needed here.
> > > Unfortunately neither you or me had time to work on it,
> > > and others who were interested disappeared.
> > >
> > > 2/ Associate ports with equivalent properties in applications.
> > > This must be done in applications.
> > > Tweaking the probe order is a hack.
> > >
> > >
> >
> > An application that want to tightly control the port init order, currently (by doing exactly like me here, hotpluging one by one the ports), would still need the bigger hack that consist in inserting a whitelist PCI devargs with a dummy address, depending on a undocumented PCI bus feature consisting in ignoring matching errors but keeping probing policy from failed devargs processing.
> >
> > Instead, with this patch this app can do
> >
> > rte_eal_manual_probe_set(1);
> > rte_eal_init();
> >
> > to have the same behavior and be able to hotplug ports as it sees fit.
> >
> > You are worried about creating user expectations about this behavior (being forced to replicate in some way the functionality during the rewrite, as I understand it?), but then you are currently forcing users to expect this workaround to exist in the PCI bus, blocking devs from touching it as it will thus break current app configurations. I've seen systemd unit file using this -w dummy flag, as well as the programmatic equivalent. Which is better, to have to rework it cutting short these configs, or to propose beforehand a deprecation path?.
> >
> > This rework won't happen in 20.05, nor in the medium term unless you decide to drive this change. This workaround serves three needs (PCI normalization, port congruence and port dependency) in a low-risk implementation.
>
> Thomas,
>
> What would be the resolution of this? What is your recommendation to
> fix the issue as you have the concern of this patch?
>
> Issue:
> 1) When l2fwd does the forwarding for simplicity and performance
> reason it just xor the port to find the destination port to forward.
> 2) If the adjacent ports are not symmetrical(example: one is 40G and
> other 10G) then forwarding will drop the packets.
>
> So, either
> a) We need to control the probing order
>
> b) Or Application need
> 1) To track the symmetrical ports and maintain the forwarding table OR
> 2) Have the command-line option to specify destination port like l3fwd.
>
> We can fix it in the application, but do we need to complicate l2fwd?
> I am fine with that, if that is consensus.
You are describing an application issue,
that's why I believe it should be fixed in the application.
Should we have a command line option to configure the forwarding rules
in the application (2)? I think yes.
Should we implement an application logic to automatically create
the best forwarding rules (1)? It would be nice, but anyway,
I think we need manual config (2) as a fallback.
> Thoughts? If you think, there is a rework needed in eal then could you
> enumerate the items for the rework.
Sorry I don't have time to describe dive into EAL probing and
enumerate the items to rework.
The most important issues I remind are:
- white/blacklist policy is a mess and should be done in a higher layer
- devargs syntax should allow generic matching (thanks to class awareness and generic syntax)
Starting from these 2 items, we could imagine a generic path to
disable automatic probing, but I think the l2fwd logic should not
rely on probing order anyway.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v7] eal: add manual probing option
2020-02-04 16:02 0% ` Gaetan Rivet
@ 2020-02-10 14:51 0% ` Jerin Jacob
2020-02-10 15:27 0% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Jerin Jacob @ 2020-02-10 14:51 UTC (permalink / raw)
To: Gaetan Rivet
Cc: Thomas Monjalon, Pavan Nikhilesh Bhagavatula,
Vamsi Krishna Attunuru, David Marchand, dpdk-dev,
Jerin Jacob Kollanukkaran, Richardson, Bruce
On Tue, Feb 4, 2020 at 9:32 PM Gaetan Rivet <grive@u256.net> wrote:
>
> On 04/02/2020 16:06, Thomas Monjalon wrote:
> > 04/02/2020 13:43, Gaetan Rivet:
> >> On 04/02/2020 12:07, Thomas Monjalon wrote:
> >>> 04/02/2020 11:03, Gaetan Rivet:
> >>>> On 03/02/2020 23:21, Thomas Monjalon wrote:
> >>>>> 03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
> >>>>>> @David Marchand @thomas@monjalon.net
> >>>>>>
> >>>>>> Ping?
> >>>>>>
> >>>>>> Are there any more changes required for this patch? It's been in queue since last October.
> >>>>>
> >>>>> Sorry we have not decided whether it is a good idea or not.
> >>>>>
> >>>>> All changes related to probing are very sensitive,
> >>>>> and we know a big refactoring would be better than stacking
> >>>>> more and more options and corner cases.
> >>>>>
> >>>>> As we are busy with ABI stability stuff, we did not allocate
> >>>>> enough time to properly think about this feature.
> >>>>> Please accept our apologies, and let's consider it as
> >>>>> a high priority for 20.05 cycle.
> >>>>>
> >>>>
> >>>> Hello Thomas,
> >>>>
> >>>> This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.
> >>>>
> >>>> The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.
> >>>
> >>>
> >>> Yes, life is full of bad decisions and consequences.
> >>
> >>
> >> Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.
> >>
> >>>
> >>> I still think there is a risk in adding new user expectations,
> >>> and maintaining some code to workaround unknown issues.
> >>>
> >>> The real question here is to know why this patch?
> >>> Is it to workaround a broken driver?
> >>> Or to workaround a broken design in EAL and bus drivers?
> >>
> >> Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).
> >>
> >> I'm not sure having a dependent-probe by default is good, and that would be a big change.
> >>
> >> If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.
> >>
> >> [1]: First proposal:
> >> http://mails.dpdk.org/archives/dev/2019-September/144166.html
> >> My arguments:
> >> http://mails.dpdk.org/archives/dev/2019-September/144564.html
> >
> >
> > OK so there are two needs:
> >
> > 1/ Better control whitelist/blacklist mode.
> > We already know that a rework is needed here.
> > Unfortunately neither you or me had time to work on it,
> > and others who were interested disappeared.
> >
> > 2/ Associate ports with equivalent properties in applications.
> > This must be done in applications.
> > Tweaking the probe order is a hack.
> >
> >
>
> An application that want to tightly control the port init order, currently (by doing exactly like me here, hotpluging one by one the ports), would still need the bigger hack that consist in inserting a whitelist PCI devargs with a dummy address, depending on a undocumented PCI bus feature consisting in ignoring matching errors but keeping probing policy from failed devargs processing.
>
> Instead, with this patch this app can do
>
> rte_eal_manual_probe_set(1);
> rte_eal_init();
>
> to have the same behavior and be able to hotplug ports as it sees fit.
>
> You are worried about creating user expectations about this behavior (being forced to replicate in some way the functionality during the rewrite, as I understand it?), but then you are currently forcing users to expect this workaround to exist in the PCI bus, blocking devs from touching it as it will thus break current app configurations. I've seen systemd unit file using this -w dummy flag, as well as the programmatic equivalent. Which is better, to have to rework it cutting short these configs, or to propose beforehand a deprecation path?.
>
> This rework won't happen in 20.05, nor in the medium term unless you decide to drive this change. This workaround serves three needs (PCI normalization, port congruence and port dependency) in a low-risk implementation.
Thomas,
What would be the resolution of this? What is your recommendation to
fix the issue as you have the concern of this patch?
Issue:
1) When l2fwd does the forwarding for simplicity and performance
reason it just xor the port to find the destination port to forward.
2) If the adjacent ports are not symmetrical(example: one is 40G and
other 10G) then forwarding will drop the packets.
So, either
a) We need to control the probing order
b) Or Application need
1) To track the symmetrical ports and maintain the forwarding table OR
2) Have the command-line option to specify destination port like l3fwd.
We can fix it in the application, but do we need to complicate l2fwd?
I am fine with that, if that is consensus.
Thoughts? If you think, there is a rework needed in eal then could you
enumerate the items for the rework.
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] common/octeontx2: update version map
@ 2020-02-10 10:27 3% Anoob Joseph
2020-02-11 14:04 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Anoob Joseph @ 2020-02-10 10:27 UTC (permalink / raw)
To: Akhil Goyal, Jerin Jacob, Thomas Monjalon
Cc: Anoob Joseph, Narayana Prasad, dev
Move symbols added in 20.02 release to DPDK_20.0.1 ABI.
Fixes: d06551535a09 ("common/octeontx2: add security capability routine")
Fixes: 3fe4d07d1678 ("crypto/octeontx2: enable CPT to share QP with ethdev")
Fixes: f44e71637755 ("net/octeontx2: add security session operations")
Signed-off-by: Anoob Joseph <anoobj@marvell.com>
---
.../common/octeontx2/rte_common_octeontx2_version.map | 17 +++++++++++------
1 file changed, 11 insertions(+), 6 deletions(-)
diff --git a/drivers/common/octeontx2/rte_common_octeontx2_version.map b/drivers/common/octeontx2/rte_common_octeontx2_version.map
index 19a7b19..a25fc69 100644
--- a/drivers/common/octeontx2/rte_common_octeontx2_version.map
+++ b/drivers/common/octeontx2/rte_common_octeontx2_version.map
@@ -5,7 +5,6 @@ DPDK_20.0 {
otx2_dev_fini;
otx2_dev_priv_init;
otx2_disable_irqs;
- otx2_eth_dev_is_sec_capable;
otx2_intra_dev_get_cfg;
otx2_logtype_base;
otx2_logtype_dpi;
@@ -28,11 +27,6 @@ DPDK_20.0 {
otx2_npa_pf_func_get;
otx2_npa_set_defaults;
otx2_register_irq;
- otx2_sec_idev_cfg_init;
- otx2_sec_idev_tx_cpt_qp_add;
- otx2_sec_idev_tx_cpt_qp_remove;
- otx2_sec_idev_tx_cpt_qp_get;
- otx2_sec_idev_tx_cpt_qp_put;
otx2_sso_pf_func_get;
otx2_sso_pf_func_set;
otx2_unregister_irq;
@@ -40,6 +34,17 @@ DPDK_20.0 {
local: *;
};
+DPDK_20.0.1 {
+ global:
+
+ otx2_eth_dev_is_sec_capable;
+ otx2_sec_idev_cfg_init;
+ otx2_sec_idev_tx_cpt_qp_add;
+ otx2_sec_idev_tx_cpt_qp_remove;
+ otx2_sec_idev_tx_cpt_qp_get;
+ otx2_sec_idev_tx_cpt_qp_put;
+};
+
EXPERIMENTAL {
global:
--
2.7.4
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2] service: don't walk out of bounds when checking services
@ 2020-02-07 14:27 3% ` Aaron Conole
2020-02-14 16:38 0% ` Kevin Traynor
0 siblings, 1 reply; 200+ results
From: Aaron Conole @ 2020-02-07 14:27 UTC (permalink / raw)
To: Kevin Traynor
Cc: Harry van Haaren, nikhil.rao, David Marchand, dev,
Bruce Richardson, Pavan Nikhilesh, Gage Eads, Thomas Monjalon,
dpdk stable
Kevin Traynor <ktraynor@redhat.com> writes:
> On 20/12/2019 14:43, David Marchand wrote:
>> On Wed, Dec 4, 2019 at 9:34 AM David Marchand <david.marchand@redhat.com> wrote:
>>>
>>> On Wed, Dec 4, 2019 at 9:33 AM David Marchand <david.marchand@redhat.com> wrote:
>>>>
>>>> On Tue, Dec 3, 2019 at 10:15 PM Aaron Conole <aconole@redhat.com> wrote:
>>>>>
>>>>> The service_valid call is used without properly bounds checking the
>>>>> input parameter. Almost all instances of the service_valid call are
>>>>> inside a for() loop that prevents excessive walks, but some of the
>>>>> public APIs don't bounds check and will pass invalid arguments.
>>>>>
>>>>> Prevent this by using SERVICE_GET_OR_ERR_RET where it makes sense,
>>>>> and adding a bounds check to one service_valid() use.
>>>>>
>>>>> Fixes: 8d39d3e237c2 ("service: fix race in service on app lcore function")
>>>>> Fixes: e9139a32f6e8 ("service: add function to run on app lcore")
>>>>> Fixes: e30dd31847d2 ("service: add mechanism for quiescing")
>>> Cc: stable@dpdk.org
>>>
>
> With the commit below, this patch will apply cleanly on 18.11.
>
> Seems ok to me to add below commit, wdyt?
If I'm reading it correctly, the move is for an internal data structure
in librte_eal, so I think it shouldn't be an ABI breakage.
Looks safe to me as well.
> commit e484ccddbe1b41886fef1e445ef2fdfa55086198
> Author: Nikhil Rao <nikhil.rao@intel.com>
> Date: Mon Sep 16 15:31:02 2019 +0530
>
> service: avoid false sharing on core state
>
>
>>>>> Signed-off-by: Aaron Conole <aconole@redhat.com>
>>>>
>>>> Reviewed-by: David Marchand <david.marchand@redhat.com>
>>
>> Applied, thanks.
>>
>>
>> --
>> David Marchand
>>
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [dpdk-announce] release candidate 20.02-rc2
@ 2020-02-06 22:33 4% Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-06 22:33 UTC (permalink / raw)
To: announce
A new DPDK release candidate is ready for testing:
https://git.dpdk.org/dpdk/tag/?id=v20.02-rc2
226 patches were integrated.
The release notes so far:
http://doc.dpdk.org/guides/rel_notes/release_20_02.html
Highlights of 20.02-rc2:
- ABI check tooling
- Mellanox vDPA driver
- Marvell OCTEON TX2 inline IPsec
- synchronous CPU crypto API
- CPU crypto based on new libraries (intel-ipsec-mb and AArch64cryptolib)
- reverted Chacha20-Poly1305 algorithm (because of ABI issue)
- event mode in l3fwd example
The -rc3 should include only some bug fixes, simple cleanups, doc
and tooling. We have one week to complete this milestone.
Then one more week (allowing -rc4) should be needed before the release.
The schedule is shifting a bit to let more time to Chinese contributors.
Thank you everyone
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v7 1/2] eal: add API to check if its interrupt context
@ 2020-02-06 15:35 3% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-06 15:35 UTC (permalink / raw)
To: Sunil Kumar Kori; +Cc: Bruce Richardson, dev, Harman Kalra
On Tue, Jan 14, 2020 at 10:05 AM Sunil Kumar Kori <skori@marvell.com> wrote:
>
> From: Harman Kalra <hkalra@marvell.com>
>
> Added an API to check if current execution is in interrupt
> context. This will be helpful to handle nested interrupt cases.
>
> Signed-off-by: Harman Kalra <hkalra@marvell.com>
> Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
> Reviewed-by: Jerin Jacob <jerinj@marvell.com>
> ---
> v7:
> - No changes.
> v6:
> - No changes.
> v5:
> - Fix shared library compilation error
> v4:
> - No changes.
> v3:
> - API Comment is updated as per man page.
> - Scope updated within the library/driver only.
> - Remove experimental tag
> v2:
> - Rebased patch on 19.11-rc4
>
> lib/librte_eal/common/include/rte_eal_interrupts.h | 11 +++++++++++
> lib/librte_eal/freebsd/eal/eal_interrupts.c | 5 +++++
> lib/librte_eal/linux/eal/eal_interrupts.c | 5 +++++
> lib/librte_eal/rte_eal_version.map | 1 +
> 4 files changed, 22 insertions(+)
>
> diff --git a/lib/librte_eal/common/include/rte_eal_interrupts.h b/lib/librte_eal/common/include/rte_eal_interrupts.h
> index b370c0d26..19d1d45ab 100644
> --- a/lib/librte_eal/common/include/rte_eal_interrupts.h
> +++ b/lib/librte_eal/common/include/rte_eal_interrupts.h
> @@ -220,4 +220,15 @@ rte_intr_allow_others(struct rte_intr_handle *intr_handle);
> int
> rte_intr_cap_multiple(struct rte_intr_handle *intr_handle);
>
> +/**
> + * @internal
> + * Check if currently executing in interrupt context
> + *
> + * @return
> + * - non zero in case of interrupt context
> + * - zero in case of process context
> + */
> +int
> +rte_thread_is_intr(void);
> +
> #endif /* _RTE_EAL_INTERRUPTS_H_ */
> diff --git a/lib/librte_eal/freebsd/eal/eal_interrupts.c b/lib/librte_eal/freebsd/eal/eal_interrupts.c
> index f6831b790..ce2a27b4a 100644
> --- a/lib/librte_eal/freebsd/eal/eal_interrupts.c
> +++ b/lib/librte_eal/freebsd/eal/eal_interrupts.c
> @@ -671,3 +671,8 @@ rte_intr_free_epoll_fd(struct rte_intr_handle *intr_handle)
> {
> RTE_SET_USED(intr_handle);
> }
> +
> +int rte_thread_is_intr(void)
> +{
> + return pthread_equal(intr_thread, pthread_self());
> +}
> diff --git a/lib/librte_eal/linux/eal/eal_interrupts.c b/lib/librte_eal/linux/eal/eal_interrupts.c
> index 14ebb108c..cb8e10709 100644
> --- a/lib/librte_eal/linux/eal/eal_interrupts.c
> +++ b/lib/librte_eal/linux/eal/eal_interrupts.c
> @@ -1488,3 +1488,8 @@ rte_intr_cap_multiple(struct rte_intr_handle *intr_handle)
>
> return 0;
> }
> +
> +int rte_thread_is_intr(void)
> +{
> + return pthread_equal(intr_thread, pthread_self());
> +}
> diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
> index e38d02530..06ed2e9dc 100644
> --- a/lib/librte_eal/rte_eal_version.map
> +++ b/lib/librte_eal/rte_eal_version.map
> @@ -91,6 +91,7 @@ DPDK_20.0 {
> rte_intr_free_epoll_fd;
> rte_intr_rx_ctl;
> rte_intr_tls_epfd;
> + rte_thread_is_intr;
> rte_keepalive_create;
> rte_keepalive_dispatch_pings;
> rte_keepalive_mark_alive;
> --
> 2.17.1
We can't put this symbol in the stable ABI and just flag it as
internal in the function description.
Users won't notice this warning.
I moved this as an experimental API and enabled the use of
experimental api in octeontx2 common driver.
We can revisit once we have the rte_internal tag Neil had proposed.
Series applied, thanks.
--
David Marchand
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v1] common/iavf: update the ABI version
@ 2020-02-06 12:04 7% Haiyue Wang
2020-02-11 2:08 4% ` Ye Xiaolong
0 siblings, 1 reply; 200+ results
From: Haiyue Wang @ 2020-02-06 12:04 UTC (permalink / raw)
To: dev, xiaolong.ye; +Cc: thomas, Haiyue Wang
The new symbols should be in ABI version 20.0.1.
Fixes: 89214fe915b8 ("net/iavf/base: move to drivers common directory")
Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
---
drivers/common/iavf/rte_common_iavf_version.map | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/common/iavf/rte_common_iavf_version.map b/drivers/common/iavf/rte_common_iavf_version.map
index 1a0839262..2f11d67c0 100644
--- a/drivers/common/iavf/rte_common_iavf_version.map
+++ b/drivers/common/iavf/rte_common_iavf_version.map
@@ -1,4 +1,4 @@
-DPDK_20.0 {
+DPDK_20.0.1 {
global:
iavf_init_adminq;
--
2.17.1
^ permalink raw reply [relevance 7%]
* [dpdk-dev] DPDK Release Status Meeting 6/02/2020
@ 2020-02-06 10:32 4% Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-06 10:32 UTC (permalink / raw)
To: dpdk-dev; +Cc: Thomas Monjalon
Minutes 6 February 2020
-----------------------
Agenda:
* Release Dates
* Subtrees
* OvS
Participants:
* Intel
* Marvell
* Mellanox
* NXP
* Red Hat
Release Dates
-------------
* v20.02 dates:
* -rc2 pushed to *Thursday 6 February 2020*
* -rc3 pushed to *Friday 14 February 2020*
* Release pushed to *Friday 21 February 2020*
* Release pushed because of extended PRC holidays
* v20.05 dates:
* Proposal/V1: Friday 6 March 2020
* Integration/Merge/RC1: Friday 10 April 2020
* Release: Friday 1 May 2020
Subtrees
--------
* main
* Pulled next-net & next-net-crypto
* Dropped two patches from crypto because of ABI breakages
* ccp sha3 & chacha20-poly
* ABI check patches merged
* In travis checks enabled by default
* next-net
* pulled, nothing in backlog for -rc3, there can be fixes
* next-net-crypto
* ipsecgw eventmode can be postponed, Akhil will check
* CPU crypto patches merged
* next-net-eventdev
* There can be a few fixes for -rc3
* next-net-virtio
* Three small fixes for -rc3
* next-net-mlx
* pulled
* LTS
* 17.11.10-rc1 released, please test and report results
* https://mails.dpdk.org/archives/dev/2020-January/154915.html
* Got test results from Intel, Red Hat & Mellanox
* QAT regression reported by Intel testing, need help from developers
* mlx reported regressions, needs more details
* Will wait two more week for above issues clarified
* Rough release date is mid February
* 18.11.6 released
* https://mails.dpdk.org/archives/dev/2020-January/156760.html
OvS
---
* Release on track
* Backporting removing experimental tag under discussion in DPDK
* https://mails.dpdk.org/archives/dev/2020-January/155419.html
DPDK Release Status Meetings
============================
The DPDK Release Status Meeting is intended for DPDK Committers to discuss
the status of the master tree and sub-trees, and for project managers to
track progress or milestone dates.
The meeting occurs on Thursdays at 8:30 UTC. If you wish to attend just
send an email to "John McNamara <john.mcnamara@intel.com>" for the invite.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 1/2] crypto/ccp: sha3 support enabling in ccp
2020-02-05 22:22 3% ` Thomas Monjalon
@ 2020-02-06 9:34 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-06 9:34 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Kumar, Ravi1, Sardar, Shamsher singh, dev, Akhil Goyal, dodji
On Wed, Feb 5, 2020 at 11:22 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> 05/02/2020 13:24, Akhil Goyal:
> > > For series,
> > > Acked-by: Ravi Kumar <ravi1.kumar@amd.com>
> > >
> > > >
> > > >From: Sardar Shamsher Singh <Shamshersingh.Sardar@amd.com>
> > > >
> > > >sha3 support enabled in AMD-CCP crypto controller
> > > >
> > > >Signed-off-by: Sardar Shamsher Singh <Shamshersingh.Sardar@amd.com>
> > > >---
> > Change patch title and description as below
> > cryptodev: fix missing SHA3 algo strings
> >
> > SHA3 support was added earlier but algo strings were
> > missing. This patch add the missing strings.
> >
> > Fixes: 1df800f89518 ("crypto/ccp: support SHA3 family")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Sardar Shamsher Singh <Shamshersingh.Sardar@amd.com>
> > Acked-by: Ravi Kumar <ravi1.kumar@amd.com>
> > Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
> >
> > Applied to dpdk-next-crypto
>
> Sorry I must drop this patch because it triggers an ABI warning:
> [C]'const char* rte_crypto_auth_algorithm_strings[]' was changed at rte_crypto_sym.h:320:1:
> size of symbol changed from 168 to 232
This is still not clear to me, but here is how I understand the issue.
An exposed array (and its size) ends up in both the shared library and
the final binary data section.
[dmarchan@wsfd-netdev66 dpdk]$ readelf -sW
~/builds/build-gcc-shared/app/dpdk-test-crypto-perf |grep
rte_crypto_auth_algorithm_strings
86: 00000000004141a0 168 OBJECT GLOBAL DEFAULT 24
rte_crypto_auth_algorithm_strings@DPDK_20.0 (4)
308: 00000000004141a0 168 OBJECT GLOBAL DEFAULT 24
rte_crypto_auth_algorithm_strings@@DPDK_20.0
[dmarchan@wsfd-netdev66 dpdk]$ readelf -sW
~/builds/build-gcc-shared/lib/librte_cryptodev.so |grep
rte_crypto_auth_algorithm_strings
57: 000000000000b220 168 OBJECT GLOBAL DEFAULT 23
rte_crypto_auth_algorithm_strings@@DPDK_20.0
158: 000000000000b220 168 OBJECT GLOBAL DEFAULT 23
rte_crypto_auth_algorithm_strings
At runtime, the linker chooses to rewire all access to the final
binary data section, not the shared library local representation.
Now, if we update the array size, the shared library code is built
with the assumption of the increased size.
But at runtime with an "old" binary, the shared library code runs with
a shorter array, with potential out of bound access.
Interesting article:
https://developers.redhat.com/blog/2019/05/06/how-c-array-sizes-become-part-of-the-binary-interface-of-a-library/
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2 1/2] crypto/ccp: sha3 support enabling in ccp
@ 2020-02-05 22:22 3% ` Thomas Monjalon
2020-02-06 9:34 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-05 22:22 UTC (permalink / raw)
To: Kumar, Ravi1, Sardar, Shamsher singh; +Cc: dev, Akhil Goyal, david.marchand
05/02/2020 13:24, Akhil Goyal:
> > For series,
> > Acked-by: Ravi Kumar <ravi1.kumar@amd.com>
> >
> > >
> > >From: Sardar Shamsher Singh <Shamshersingh.Sardar@amd.com>
> > >
> > >sha3 support enabled in AMD-CCP crypto controller
> > >
> > >Signed-off-by: Sardar Shamsher Singh <Shamshersingh.Sardar@amd.com>
> > >---
> Change patch title and description as below
> cryptodev: fix missing SHA3 algo strings
>
> SHA3 support was added earlier but algo strings were
> missing. This patch add the missing strings.
>
> Fixes: 1df800f89518 ("crypto/ccp: support SHA3 family")
> Cc: stable@dpdk.org
>
> Signed-off-by: Sardar Shamsher Singh <Shamshersingh.Sardar@amd.com>
> Acked-by: Ravi Kumar <ravi1.kumar@amd.com>
> Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
>
> Applied to dpdk-next-crypto
Sorry I must drop this patch because it triggers an ABI warning:
[C]'const char* rte_crypto_auth_algorithm_strings[]' was changed at rte_crypto_sym.h:320:1:
size of symbol changed from 168 to 232
I don't understand how the size of this array can impact the application,
but I prefer to stay on the safe side for now.
Please let's discuss this patch in 20.02-rc3 timeframe.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2] build: remove unused function versioning
2020-01-30 16:11 3% ` David Marchand
@ 2020-02-05 20:44 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-05 20:44 UTC (permalink / raw)
To: Andrzej Ostruszka
Cc: dev, David Hunt, Bruce Richardson, Vladimir Medvedkin,
Robert Sanford, Erik Gabriel Carrillo, Marcin Baran, dpdk stable
On Thu, Jan 30, 2020 at 5:11 PM David Marchand
<david.marchand@redhat.com> wrote:
>
> On Thu, Jan 30, 2020 at 10:25 AM Andrzej Ostruszka
> <aostruszka@marvell.com> wrote:
> >
> > Timer, LPM and Distributor libraries no longer use function versioning
> > and therefore do not need separate build for static and shared version
> > of libraries.
> >
> > This patch removes use_function_versioning from their meson build files
> > and corresponding include from the sources.
> >
> > Signed-off-by: Andrzej Ostruszka <aostruszka@marvell.com>
> > Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> >
> > Fixes: f2fb215843a9 ("timer: remove deprecated code")
> > Fixes: 6e5b51676176 ("distributor: remove deprecated code")
> > Fixes: c381a8d554b7 ("lpm: remove deprecated code")
> > Cc: marcinx.baran@intel.com
> >
> > ---
> > V2:
> > - removed also includes (updating only meson files causes build
> > failures), and updated commit msg accordingly
>
> Thanks, I had the exact same patch in store (caught when working with
> Olivier on some ABI stuff for mempool).
>
> Cc: stable@dpdk.org
>
> Acked-by: David Marchand <david.marchand@redhat.com>
Applied, thanks.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] buildtools: allow pedantic empty pmdinfo
@ 2020-02-05 17:29 3% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-05 17:29 UTC (permalink / raw)
To: Bruce Richardson
Cc: dev, ferruh.yigit, matan, maxime.coquelin, david.marchand
05/02/2020 18:14, Bruce Richardson:
> On Wed, Feb 05, 2020 at 06:07:23PM +0100, Thomas Monjalon wrote:
> > If a driver has no info generated by pmdinfogen,
> > and if this driver is compiled as pedantic,
> > then an failure would occur:
> > drivers/rte_common_mlx5.pmd.c:1: error:
> > ISO C forbids an empty translation unit [-Werror=pedantic]
> > Such error is triggered with the new mlx5 common directory.
> >
> > In order to allow an "empty driver info" compiled in pedantic mode,
> > the script generating .pmd.c file is modified to add a static string
> > unconditionnaly.
> > The minimal generated code is:
> > static __attribute__((unused)) const char *generator =
> > "/path/to/dpdk/buildtools/gen-pmdinfo-cfile.sh";
> >
> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
>
> LGTM
>
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Applied, thanks, it will allow pulling next-net
without breaking debug compilation for the new ABI tooling.
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v3] doc: alias to experimental tag for stable apis
2020-02-05 15:17 17% [dpdk-dev] [PATCH] doc: alias to experimental tag for stable apis Ray Kinsella
@ 2020-02-05 15:43 17% ` Ray Kinsella
2020-02-18 10:46 17% ` [dpdk-dev] [PATCH v4] " Ray Kinsella
1 sibling, 0 replies; 200+ results
From: Ray Kinsella @ 2020-02-05 15:43 UTC (permalink / raw)
To: dev; +Cc: john.mcnamara, nhorman, ferruh.yigit, marko.kovacevic, Ray Kinsella
When a maintainer is promoting an API to become part of the next major ABI
version by removing the experimental tag, possibly a few releases in
advance of the declaration of the next ABI version. The maintainer may
choose to offer an alias to the experimental tag, as removing the tag
before the declaration of the next major ABI version, would cause an ABI
breakage for applications using the API.
Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
---
doc/guides/contributing/abi_policy.rst | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/doc/guides/contributing/abi_policy.rst b/doc/guides/contributing/abi_policy.rst
index 05ca959..9a4a102 100644
--- a/doc/guides/contributing/abi_policy.rst
+++ b/doc/guides/contributing/abi_policy.rst
@@ -159,6 +159,11 @@ The requirements for changing the ABI are:
``experimental``, as described in the section on :ref:`Experimental APIs
and Libraries <experimental_apis>`.
+ - In situations where an ``experimental`` API has been stable for some time.
+ When promoting the API to become part of the next ABI version, the
+ maintainer may choose to provide an alias to the ``experimental`` tag, so
+ as not to break consuming applications.
+
#. If a newly proposed API functionally replaces an existing one, when the new
API becomes non-experimental, then the old one is marked with
``__rte_deprecated``.
@@ -317,6 +322,11 @@ not required. Though, an API should remain in experimental state for at least
one release. Thereafter, the normal process of posting patch for review to
mailing list can be followed.
+After the experimental tag has been formally removed, a tree/sub-tree maintainer
+may choose to offer an alias to the experimental tag so as not to break
+applications using the API. This alias can then be dropped at the declaration of
+next major ABI version.
+
Libraries
~~~~~~~~~
--
2.7.4
^ permalink raw reply [relevance 17%]
* [dpdk-dev] [PATCH v2] doc: alias to experimental tag for stable apis
@ 2020-02-05 15:28 17% Ray Kinsella
0 siblings, 0 replies; 200+ results
From: Ray Kinsella @ 2020-02-05 15:28 UTC (permalink / raw)
To: dev; +Cc: john.mcnamara, nhorman, ferruh.yigit, marko.kovacevic, Ray Kinsella
When a maintainer is promoting an API to become part of the next major ABI
version by removing the experimental tag, possibly a few releases in
advance of the declaration of the next ABI version. The maintainer may
choose to offer an alias to the experimental tag, as removing the tag
before the declaration of the next major ABI version, would cause an ABI
breakage for applications using the API.
Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
---
doc/guides/contributing/abi_policy.rst | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/doc/guides/contributing/abi_policy.rst b/doc/guides/contributing/abi_policy.rst
index 05ca959..9a4a102 100644
--- a/doc/guides/contributing/abi_policy.rst
+++ b/doc/guides/contributing/abi_policy.rst
@@ -159,6 +159,11 @@ The requirements for changing the ABI are:
``experimental``, as described in the section on :ref:`Experimental APIs
and Libraries <experimental_apis>`.
+ - In situations where an ``experimental`` API has been stable for some time.
+ When promoting the API to become part of the next ABI version, the
+ maintainer may choose to provide an alias to the ``experimental`` tag, so
+ as not to break consuming applications.
+
#. If a newly proposed API functionally replaces an existing one, when the new
API becomes non-experimental, then the old one is marked with
``__rte_deprecated``.
@@ -317,6 +322,11 @@ not required. Though, an API should remain in experimental state for at least
one release. Thereafter, the normal process of posting patch for review to
mailing list can be followed.
+After the experimental tag has been formally removed, a tree/sub-tree maintainer
+may choose to offer an alias to the experimental tag so as not to break
+applications using the API. This alias can then be dropped at the declaration of
+next major ABI version.
+
Libraries
~~~~~~~~~
--
2.7.4
^ permalink raw reply [relevance 17%]
* Re: [dpdk-dev] [PATCH v3] cryptodev: add chacha20-poly1305 aead algorithm
2020-02-05 14:41 0% ` Trahe, Fiona
@ 2020-02-05 15:21 0% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-05 15:21 UTC (permalink / raw)
To: Trahe, Fiona
Cc: Anoob Joseph, Kusztal, ArkadiuszX, Akhil Goyal, dev,
Narayana Prasad Raju Athreya, Tejasree Kondoj,
Jerin Jacob Kollanukkaran, bluca, ktraynor, david.marchand,
nhorman, Kinsella, Ray, Yigit, Ferruh, Richardson, Bruce
05/02/2020 15:41, Trahe, Fiona:
> From: Thomas Monjalon <thomas@monjalon.net>
> > 15/01/2020 16:53, Akhil Goyal:
> > > > > This patch adds Chacha20-Poly1305 AEAD algorithm to Cryptodev.
> > > > >
> > > > > Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
> > > > > Acked-by: Fiona Trahe <fiona.trahe@intel.com>
> > > >
> > > > Acked-by: Anoob Joseph <anoobj@marvell.com>
> > > >
> > > Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
> > >
> > > Applied to dpdk-next-crypto
> >
> > As discussed in other email threads,
> > this patch was causing an ABI breakage and had to be reverted:
> > http://git.dpdk.org/dpdk/commit/?id=f5862ae99
> > Sorry, this feature will miss 20.02 release I'm afraid.
> >
> > 2 actions should be taken:
> > - try to re-intoduce the feature with function versioning
> > - improve ABI stability in DPDK 20.11 for future features
>
> [Fiona] Agreed, it's underway.
Any estimation of time?
I expect a lot of discussions on the solution,
especially because it is a primer.
> The QAT ChaChaPoly patchset including the unit tests will also need to be reverted.
> https://patches.dpdk.org/patch/64729/
> https://patches.dpdk.org/patch/64730/
Yes, this is what I did in one commit.
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] doc: alias to experimental tag for stable apis
@ 2020-02-05 15:17 17% Ray Kinsella
2020-02-05 15:43 17% ` [dpdk-dev] [PATCH v3] " Ray Kinsella
2020-02-18 10:46 17% ` [dpdk-dev] [PATCH v4] " Ray Kinsella
0 siblings, 2 replies; 200+ results
From: Ray Kinsella @ 2020-02-05 15:17 UTC (permalink / raw)
To: dev; +Cc: john.mcnamara, nhorman, ferruh.yigit, marko.kovacevic, Ray Kinsella
When a maintainer is promoting an API to become part of the next major ABI
version by removing the experimental tag, possibly a few releases in advance of
the declaration of the next ABI version. The maintainer may choose to offer an
alias to the experimental tag, as removing the tag before the declaration of the
next major ABI version, would cause an ABI breakage for applications using the
API.
Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
---
doc/guides/contributing/abi_policy.rst | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/doc/guides/contributing/abi_policy.rst b/doc/guides/contributing/abi_policy.rst
index 05ca959..9a4a102 100644
--- a/doc/guides/contributing/abi_policy.rst
+++ b/doc/guides/contributing/abi_policy.rst
@@ -159,6 +159,11 @@ The requirements for changing the ABI are:
``experimental``, as described in the section on :ref:`Experimental APIs
and Libraries <experimental_apis>`.
+ - In situations where an ``experimental`` API has been stable for some time.
+ When promoting the API to become part of the next ABI version, the
+ maintainer may choose to provide an alias to the ``experimental`` tag, so
+ as not to break consuming applications.
+
#. If a newly proposed API functionally replaces an existing one, when the new
API becomes non-experimental, then the old one is marked with
``__rte_deprecated``.
@@ -317,6 +322,11 @@ not required. Though, an API should remain in experimental state for at least
one release. Thereafter, the normal process of posting patch for review to
mailing list can be followed.
+After the experimental tag has been formally removed, a tree/sub-tree maintainer
+may choose to offer an alias to the experimental tag so as not to break
+applications using the API. This alias can then be dropped at the declaration of
+next major ABI version.
+
Libraries
~~~~~~~~~
--
2.7.4
^ permalink raw reply [relevance 17%]
* Re: [dpdk-dev] [PATCH v3] cryptodev: add chacha20-poly1305 aead algorithm
2020-02-05 14:28 5% ` Thomas Monjalon
@ 2020-02-05 14:41 0% ` Trahe, Fiona
2020-02-05 15:21 0% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Trahe, Fiona @ 2020-02-05 14:41 UTC (permalink / raw)
To: Thomas Monjalon, Anoob Joseph, Kusztal, ArkadiuszX, Akhil Goyal
Cc: dev, Narayana Prasad Raju Athreya, Tejasree Kondoj,
Jerin Jacob Kollanukkaran, bluca, ktraynor, david.marchand,
nhorman, Kinsella, Ray, Yigit, Ferruh, Richardson, Bruce, Trahe,
Fiona
Hi Thomas,
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, February 5, 2020 2:29 PM
> To: Anoob Joseph <anoobj@marvell.com>; Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>; Trahe,
> Fiona <fiona.trahe@intel.com>; Akhil Goyal <akhil.goyal@nxp.com>
> Cc: dev@dpdk.org; Narayana Prasad Raju Athreya <pathreya@marvell.com>; Tejasree Kondoj
> <ktejasree@marvell.com>; Jerin Jacob Kollanukkaran <jerinj@marvell.com>; bluca@debian.org;
> ktraynor@redhat.com; david.marchand@redhat.com; nhorman@tuxdriver.com; Kinsella, Ray
> <ray.kinsella@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v3] cryptodev: add chacha20-poly1305 aead algorithm
>
> 15/01/2020 16:53, Akhil Goyal:
> > > > This patch adds Chacha20-Poly1305 AEAD algorithm to Cryptodev.
> > > >
> > > > Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
> > > > Acked-by: Fiona Trahe <fiona.trahe@intel.com>
> > >
> > > Acked-by: Anoob Joseph <anoobj@marvell.com>
> > >
> > Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
> >
> > Applied to dpdk-next-crypto
>
> As discussed in other email threads,
> this patch was causing an ABI breakage and had to be reverted:
> http://git.dpdk.org/dpdk/commit/?id=f5862ae99
> Sorry, this feature will miss 20.02 release I'm afraid.
>
> 2 actions should be taken:
> - try to re-intoduce the feature with function versioning
> - improve ABI stability in DPDK 20.11 for future features
[Fiona] Agreed, it's underway.
The QAT ChaChaPoly patchset including the unit tests will also need to be reverted.
https://patches.dpdk.org/patch/64729/
https://patches.dpdk.org/patch/64730/
^ permalink raw reply [relevance 0%]
* [dpdk-dev] Travis warning about job configs
@ 2020-02-05 14:34 3% David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-05 14:34 UTC (permalink / raw)
To: Aaron Conole, Michael Santana; +Cc: dev
While monitoring the newly pushed ABI check in Travis, I noticed that
Travis has now a feature to validate the configuration we pass.
You can notice a new "Build config validation" box in
https://travis-ci.com/DPDK/dpdk/builds/147554312/config
Not sure those warnings are a real issue, but it could become one in the future.
--
David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [EXT] Re: [PATCH v2 4/4] add ABI checks
2020-02-05 6:16 7% ` [dpdk-dev] [EXT] " Anoob Joseph
@ 2020-02-05 14:33 4% ` Trahe, Fiona
0 siblings, 0 replies; 200+ results
From: Trahe, Fiona @ 2020-02-05 14:33 UTC (permalink / raw)
To: Anoob Joseph, Neil Horman, Akhil Goyal
Cc: Thomas Monjalon, Yigit, Ferruh, Ananyev, Konstantin, dev,
David Marchand, Kusztal, ArkadiuszX, Richardson, Bruce, Mcnamara,
John, dodji, Andrew Rybchenko, aconole, bluca, ktraynor, Trahe,
Fiona
Hi Anoob,
> -----Original Message-----
> From: Anoob Joseph <anoobj@marvell.com>
> Sent: Wednesday, February 5, 2020 6:16 AM
> To: Neil Horman <nhorman@tuxdriver.com>; Akhil Goyal <akhil.goyal@nxp.com>; Trahe, Fiona
> <fiona.trahe@intel.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>; Yigit, Ferruh <ferruh.yigit@intel.com>; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>; dev@dpdk.org;
> David Marchand <david.marchand@redhat.com>; Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>;
> Richardson, Bruce <bruce.richardson@intel.com>; Mcnamara, John <john.mcnamara@intel.com>;
> dodji@seketeli.net; Andrew Rybchenko <arybchenko@solarflare.com>; aconole@redhat.com;
> bluca@debian.org; ktraynor@redhat.com
> Subject: RE: [EXT] Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
>
> Hi Akhil, Neil, Fiona
>
> Sorry for the late response. I want to propose a new change in line with what you folks had proposed.
>
> May be we can treat the new features EXPERIMENTAL until a new stable release.
>
> enum rte_crypto_aead_algorithm {
> RTE_CRYPTO_AEAD_AES_CCM = 1,
> /**< AES algorithm in CCM mode. */
> RTE_CRYPTO_AEAD_AES_GCM,
> /**< AES algorithm in GCM mode. */
> RTE_CRYPTO_AEAD_LIST_END,
> /**< List end for stable */
> /** EXPERIMENTAL */
> RTE_CRYPTO_AEAD_CHACHA20_POLY1305,
> /**< Chacha20 cipher with poly1305 authenticator */
> RTE_CRYPTO_AEAD_LIST_END_EXPERIMENTAL
> /**< List end */
> };
>
> And then introduce an experimental API,
>
> const struct rte_cryptodev_capabilities *
> rte_cryptodev_get_exp_capabilites(uint8_t dev_id);
>
> The PMD owner is expected to add new capabilities (only new ones) to this one until the new feature is
> deemed stable (ie, in one of the next stable releases). We don't expect users to change their API/ABI.
> For applications using EXPERIMENTAL is allowed to use the above capabilities to get the EXPERIMENTAL
> features.
>
> This does involve moving around code in PMD when one feature is added, but that's the risk PMD
> owner is taking by upstreaming as EXPERIMENTAL and not in stable release.
>
> Thanks,
> Anoob
[Fiona] Thanks for the suggestion Anoob.
I like the enum part of the idea - but not the new temporary API as the applications need to be aware of it and would have to change again when it's removed.
I explored an alternative way of using the current experimental infrastructure, i.e.:
enum rte_crypto_aead_algorithm {
RTE_CRYPTO_AEAD_AES_CCM = 1,
/**< AES algorithm in CCM mode. */
RTE_CRYPTO_AEAD_AES_GCM,
/**< AES algorithm in GCM mode. */
#ifdef ALLOW_EXPERIMENTAL_API
RTE_CRYPTO_AEAD_CHACHA20_POLY1305,
/**< Chacha20 cipher with poly1305 authenticator */
#endif
RTE_CRYPTO_AEAD_LIST_END,
/**< List end */
};
No new rte_cryptodev_get_exp_capabilities() needed.
Any PMD that implements the experimental API must do the same:
#ifdef ALLOW_EXPERIMENTAL_API
<PMD code processing new enum>
#endif
Same with test code.
Any 19.11 production code that wants to run against shared objects from
20.02 can be expected to build DPDK with ALLOW_EXPERIMENTAL_API disabled,
so will not pick up the new feature.
However, it appears the flag is not globally consistent, i.e. most PMDs have it set, even if the application doesn't set it. So this probably wouldn't work.
We're testing the approach outlined yesterday and believe it satisfactorily resolves the issue, so will stick with that.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v3] cryptodev: add chacha20-poly1305 aead algorithm
@ 2020-02-05 14:28 5% ` Thomas Monjalon
2020-02-05 14:41 0% ` Trahe, Fiona
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-05 14:28 UTC (permalink / raw)
To: Anoob Joseph, Arek Kusztal, fiona.trahe, Akhil Goyal
Cc: dev, Narayana Prasad Raju Athreya, Tejasree Kondoj,
Jerin Jacob Kollanukkaran, bluca, ktraynor, david.marchand,
nhorman, ray.kinsella, ferruh.yigit, bruce.richardson
15/01/2020 16:53, Akhil Goyal:
> > > This patch adds Chacha20-Poly1305 AEAD algorithm to Cryptodev.
> > >
> > > Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
> > > Acked-by: Fiona Trahe <fiona.trahe@intel.com>
> >
> > Acked-by: Anoob Joseph <anoobj@marvell.com>
> >
> Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
>
> Applied to dpdk-next-crypto
As discussed in other email threads,
this patch was causing an ABI breakage and had to be reverted:
http://git.dpdk.org/dpdk/commit/?id=f5862ae99
Sorry, this feature will miss 20.02 release I'm afraid.
2 actions should be taken:
- try to re-intoduce the feature with function versioning
- improve ABI stability in DPDK 20.11 for future features
We are all (hardly) learning how to maintain ABI compatibility,
as this is our first real issue since the new ABI policy.
Thanks for understanding.
^ permalink raw reply [relevance 5%]
* Re: [dpdk-dev] [PATCH v4 0/3] add ABI checks
2020-02-02 21:08 9% ` [dpdk-dev] [PATCH v4 0/3] " David Marchand
2020-02-02 21:08 28% ` [dpdk-dev] [PATCH v4 3/3] " David Marchand
@ 2020-02-05 14:13 4% ` Thomas Monjalon
1 sibling, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-05 14:13 UTC (permalink / raw)
To: David Marchand
Cc: dev, bruce.richardson, kevin.laatz, aconole, nhorman,
akhil.goyal, anoobj, bluca, fiona.trahe, ferruh.yigit,
Arek Kusztal
02/02/2020 22:08, David Marchand:
> Here is the current state of the ABI checks.
>
> libabigail has some issues when mixing dump and so files compiled with
> clang [1], so for now, all checks are done on dumps only.
> libabigail 1.0-rc3 in Xenial reported issues that disappear with the
> version 1.2 in Bionic.
>
> To avoid getting warnings on internal types like [2], the checks now make
> use of the public headers part of a dpdk installation (patch 2 and 3 to
> prepare for this).
>
> Some internal rte_hash headers were installed by meson, so patch 1 fixes
> this.
>
> The most important point, abidiff complains on the rc1 cryptodev changes:
> - Chacha20-Poly1305 AEAD support,
> - ECPM and ECDSA support
>
> A suppression rule has been put on the internal type rte_cryptodev_ops.
> But other changes are an ABI breakage afaiu. I put suppression rules on
> them so that we can run the checks, but some action must be taken for
> 20.02 if my analysis is confirmed.
The suppression rules for Chacha-Poly are removed,
and Chacha-Poly commits are reverted while merging this series.
> Special thanks to Dodji the libabigail maintainer for helping on this
> topic.
>
> 1: https://sourceware.org/bugzilla/show_bug.cgi?id=25409
> 2: http://inbox.dpdk.org/dev/CAJFAV8yFKoDZROX9Mkyp7pDMvXw3e7mHwxjfrcjD5ZoFB2tZ8w@mail.gmail.com/
Applied, thanks
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH 00/14] cleanup resources on shutdown
2020-02-05 12:07 0% ` Stephen Hemminger
@ 2020-02-05 12:32 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-05 12:32 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: dev, Luca Boccassi, Bruce Richardson
On Wed, Feb 5, 2020 at 1:07 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Wed, 5 Feb 2020 10:32:49 +0100
> David Marchand <david.marchand@redhat.com> wrote:
>
> > Hello Stephen,
> >
> > On Sat, Jan 4, 2020 at 2:34 AM Stephen Hemminger
> > <stephen@networkplumber.org> wrote:
> > >
> > > Recently started using valgrind with DPDK, and the results
> > > are not clean.
> > >
> > > The DPDK has a function that applications can use to tell it
> > > to cleanup resources on shutdown (rte_eal_cleanup). But the
> > > current coverage of that API is spotty. Many internal parts of
> > > DPDK leave files and allocated memory behind.
> > >
> > > This patch set is a start at getting the sub-parts of
> > > DPDK to cleanup after themselves. These are the easier ones,
> > > the harder and more critical ones are in the drivers
> > > and the memory subsystem.
> > >
> > > There are no visible API or ABI changes here.
> >
> > Could you share what you did to run a dpdk application with valgrind?
> >
> > I tried with testpmd and a 3.15 valgrind (fc30), but I get an init
> > failure on the cpu flags.
> >
> > $ LD_LIBRARY_PATH=/home/dmarchan/builds/build-gcc-shared/install/usr/local/lib64
> > valgrind /home/dmarchan/builds/build-gcc-shared/install/usr/local/bin/dpdk-testpmd
> > -c 3 --no-huge -m 20 -d librte_mempool_ring.so -d librte_pmd_null.so
> > -w 0:0.0 --vdev net_null1 --vdev net_null2 -- --no-mlockall
> > --total-num-mbufs=2048 -ia
> > ==10258== Memcheck, a memory error detector
> > ==10258== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
> > ==10258== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
> > ==10258== Command:
> > /home/dmarchan/builds/build-gcc-shared/install/usr/local/bin/dpdk-testpmd
> > -c 3 --no-huge -m 20 -d librte_mempool_ring.so -d librte_pmd_null.so
> > -w 0:0.0 --vdev net_null1 --vdev net_null2 -- --no-mlockall
> > --total-num-mbufs=2048 -ia
> > ==10258==
> > ERROR: This system does not support "RDSEED".
> > Please check that RTE_MACHINE is set correctly.
> > EAL: FATAL: unsupported cpu type.
> > EAL: unsupported cpu type.
> > EAL: Error - exiting with code: 1
> > Cause: Cannot init EAL: Operation not supported
> > ==10258==
> > ==10258== HEAP SUMMARY:
> > ==10258== in use at exit: 1,388 bytes in 49 blocks
> > ==10258== total heap usage: 97 allocs, 48 frees, 89,426 bytes allocated
> > ==10258==
> > ==10258== LEAK SUMMARY:
> > ==10258== definitely lost: 0 bytes in 0 blocks
> > ==10258== indirectly lost: 0 bytes in 0 blocks
> > ==10258== possibly lost: 0 bytes in 0 blocks
> > ==10258== still reachable: 1,388 bytes in 49 blocks
> > ==10258== suppressed: 0 bytes in 0 blocks
> > ==10258== Rerun with --leak-check=full to see details of leaked memory
> > ==10258==
> > ==10258== For lists of detected and suppressed errors, rerun with: -s
> > ==10258== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
> >
>
> I am testing with valgrind on ARM.
> It should be possible on x86 but you need to dial down the RTE_MACHINE
> choice to something valgrind understands.
>
Ok, so no black magic in valgrind :-)
Yeah I managed to run with the x86-default target we have in
test-meson-builds.sh.
Thanks.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 00/14] cleanup resources on shutdown
2020-02-05 9:32 0% ` David Marchand
@ 2020-02-05 12:07 0% ` Stephen Hemminger
2020-02-05 12:32 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Stephen Hemminger @ 2020-02-05 12:07 UTC (permalink / raw)
To: David Marchand; +Cc: dev, Luca Boccassi, Bruce Richardson
On Wed, 5 Feb 2020 10:32:49 +0100
David Marchand <david.marchand@redhat.com> wrote:
> Hello Stephen,
>
> On Sat, Jan 4, 2020 at 2:34 AM Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> >
> > Recently started using valgrind with DPDK, and the results
> > are not clean.
> >
> > The DPDK has a function that applications can use to tell it
> > to cleanup resources on shutdown (rte_eal_cleanup). But the
> > current coverage of that API is spotty. Many internal parts of
> > DPDK leave files and allocated memory behind.
> >
> > This patch set is a start at getting the sub-parts of
> > DPDK to cleanup after themselves. These are the easier ones,
> > the harder and more critical ones are in the drivers
> > and the memory subsystem.
> >
> > There are no visible API or ABI changes here.
>
> Could you share what you did to run a dpdk application with valgrind?
>
> I tried with testpmd and a 3.15 valgrind (fc30), but I get an init
> failure on the cpu flags.
>
> $ LD_LIBRARY_PATH=/home/dmarchan/builds/build-gcc-shared/install/usr/local/lib64
> valgrind /home/dmarchan/builds/build-gcc-shared/install/usr/local/bin/dpdk-testpmd
> -c 3 --no-huge -m 20 -d librte_mempool_ring.so -d librte_pmd_null.so
> -w 0:0.0 --vdev net_null1 --vdev net_null2 -- --no-mlockall
> --total-num-mbufs=2048 -ia
> ==10258== Memcheck, a memory error detector
> ==10258== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
> ==10258== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
> ==10258== Command:
> /home/dmarchan/builds/build-gcc-shared/install/usr/local/bin/dpdk-testpmd
> -c 3 --no-huge -m 20 -d librte_mempool_ring.so -d librte_pmd_null.so
> -w 0:0.0 --vdev net_null1 --vdev net_null2 -- --no-mlockall
> --total-num-mbufs=2048 -ia
> ==10258==
> ERROR: This system does not support "RDSEED".
> Please check that RTE_MACHINE is set correctly.
> EAL: FATAL: unsupported cpu type.
> EAL: unsupported cpu type.
> EAL: Error - exiting with code: 1
> Cause: Cannot init EAL: Operation not supported
> ==10258==
> ==10258== HEAP SUMMARY:
> ==10258== in use at exit: 1,388 bytes in 49 blocks
> ==10258== total heap usage: 97 allocs, 48 frees, 89,426 bytes allocated
> ==10258==
> ==10258== LEAK SUMMARY:
> ==10258== definitely lost: 0 bytes in 0 blocks
> ==10258== indirectly lost: 0 bytes in 0 blocks
> ==10258== possibly lost: 0 bytes in 0 blocks
> ==10258== still reachable: 1,388 bytes in 49 blocks
> ==10258== suppressed: 0 bytes in 0 blocks
> ==10258== Rerun with --leak-check=full to see details of leaked memory
> ==10258==
> ==10258== For lists of detected and suppressed errors, rerun with: -s
> ==10258== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
>
I am testing with valgrind on ARM.
It should be possible on x86 but you need to dial down the RTE_MACHINE
choice to something valgrind understands.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-05 10:04 4% ` Luca Boccassi
@ 2020-02-05 11:32 7% ` Neil Horman
2020-02-13 17:40 8% ` Ray Kinsella
0 siblings, 1 reply; 200+ results
From: Neil Horman @ 2020-02-05 11:32 UTC (permalink / raw)
To: Luca Boccassi
Cc: Ferruh Yigit, Cristian Dumitrescu, Eelco Chaudron, dev,
Thomas Monjalon, David Marchand, Bruce Richardson, Ian Stokes
On Wed, Feb 05, 2020 at 11:04:29AM +0100, Luca Boccassi wrote:
> On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
> > > But if we can do the versioning in the master, LTS can backport it
> > > and can have
> > > mature version of that API in LTS without breaking the existing
> > > users.
> > >
> >
> > But why bother? The only thing you've changed is the version
> > tagging. Its ok
> > to leave that alone in LTS, you just cant change it.
> >
> > Thats part of the burden of an LTS release, it will have some drift
> > from the
> > upstream head, because you have to keep things stable. So you
> > stabilize the
> > upstream ABI version for this API and just never backport it to the
> > current LTS
> > release.
>
> Hi,
>
> A customer (OVS) explicitly and specifically requested backporting the
> symbol change to 19.11, as they don't want to enable experimental APIs
> in their releases. I replied that it would only be acceptable with
> aliasing to keep compatibility, and Ferruh very kindly did the work to
> implement that.
>
but, thats part of the agreement, no? You can't always have new features and
stability at the same time.
I get that this is an odd corner case, because strictly speaking you could waive
the ABI change that libabigail is reporting, and most application users (like
OVS) could get away with it, because their application does all the right things
to make it ok, but I don't think you can make a decsion like this for all users
based on the request of a single user.
It seems like the right thing is for OVS to augment their build time
configuration to allow developers to select the ability to use experimental apis
at compile time, and then the decision is placed in the hands of end users.
Neil
> --
> Kind regards,
> Luca Boccassi
>
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-04 9:51 4% ` David Marchand
2020-02-04 10:10 4% ` Trahe, Fiona
@ 2020-02-05 11:10 4% ` Ray Kinsella
1 sibling, 0 replies; 200+ results
From: Ray Kinsella @ 2020-02-05 11:10 UTC (permalink / raw)
To: David Marchand
Cc: Thomas Monjalon, Neil Horman, Luca Boccassi, Kevin Traynor,
Ananyev, Konstantin, Akhil Goyal, Trahe, Fiona, Ferruh Yigit,
dev, Anoob Joseph, Kusztal, ArkadiuszX, Richardson, Bruce,
Mcnamara, John, dodji, Andrew Rybchenko, Aaron Conole
On 04/02/2020 09:51, David Marchand wrote:
> On Mon, Feb 3, 2020 at 7:56 PM Ray Kinsella <mdr@ashroe.eu> wrote:
>> On 03/02/2020 17:34, Thomas Monjalon wrote:
>>> 03/02/2020 18:09, Thomas Monjalon:
>>>> 03/02/2020 10:30, Ferruh Yigit:
>>>>> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
>>>>>> 02/02/2020 14:05, Thomas Monjalon:
>>>>>>> 31/01/2020 15:16, Trahe, Fiona:
>>>>>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
>>>>>>>>> If library give higher value than expected by the application,
>>>>>>>>> if the application uses this value as array index,
>>>>>>>>> there can be an access out of bounds.
>>>>>>>>
>>>>>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
>>>>>>>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
>>>>>>>> sense and I don't see how there can be an API breakage.
>>>>>>>> So if an application hasn't compiled against the new lib it will be still using the old value
>>>>>>>> which will be within bounds. If it's picking up the higher new value from the lib it must
>>>>>>>> have been compiled against the lib so shouldn't have problems.
>>>>>>>
>>>>>>> You say there is no ABI issue because the application will be re-compiled
>>>>>>> for the updated library. Indeed, compilation fixes compatibility issues.
>>>>>>> But this is not relevant for ABI compatibility.
>>>>>>> ABI compatibility means we can upgrade the library without recompiling
>>>>>>> the application and it must work.
>>>>>>> You think it is a false positive because you assume the application
>>>>>>> "picks" the new value. I think you miss the case where the new value
>>>>>>> is returned by a function in the upgraded library.
>>>>>>>
>>>>>>>> There are also no structs on the API which contain arrays using this
>>>>>>>> for sizing, so I don't see an opportunity for an appl to have a
>>>>>>>> mismatch in memory addresses.
>>>>>>>
>>>>>>> Let me demonstrate where the API may "use" the new value
>>>>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
>>>>>>>
>>>>>>> Once upon a time a DPDK application counting the number of devices
>>>>>>> supporting each AEAD algo (in order to find the best supported algo).
>>>>>>> It is done in an array indexed by algo id:
>>>>>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
>>>>>>> The application is compiled with DPDK 19.11,
>>>>>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
>>>>>>> So the size of the application array aead_dev_count is 3.
>>>>>>> This binary is run with DPDK 20.02,
>>>>>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
>>>>>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
>>>>>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
>>>>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
>>>>>>> The application uses this value:
>>>>>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
>>>>>>> The application is crashing because of out of bound access.
>>>>>>
>>>>>> I'd say this is an example of bad written app.
>>>>>> It probably should check that returned by library value doesn't
>>>>>> exceed its internal array size.
>>>>>
>>>>> +1
>>>>>
>>>>> Application should ignore values >= MAX.
>>>>
>>>> Of course, blaming the API user is a lot easier than looking at the API.
>>>> Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
>>>> as the max value for the application.
>>>> Value ranges are part of the ABI compatibility contract.
>>>> It seems you expect the application developer to be aware that
>>>> DPDK could return a higher value, so the application should
>>>> check every enum values after calling an API. CRAZY.
>>>>
>>>> When we decide to announce an ABI compatibility and do some marketing,
>>>> everyone is OK. But when we need to really make our ABI compatible,
>>>> I see little or no effort. DISAPPOINTING.
>>>>
>>>>> Do you suggest we don't extend any enum or define between ABI breakage releases
>>>>> to be sure bad written applications not affected?
>>>>
>>>> I suggest we must consider not breaking any assumption made on the API.
>>>> Here we are breaking the enum range because nothing mentions _LIST_END
>>>> is not really the absolute end of the enum.
>>>> The solution is to make the change below in 20.02 + backport in 19.11.1:
>>>
>>> Thinking twice, merging such change before 20.11 is breaking the
>>> ABI assumption based on the API 19.11.0.
>>> I ask the release maintainers (Luca, Kevin, David and me) and
>>> the ABI maintainers (Neil and Ray) to vote for a or b solution:
>>> a) add comment and LIST_MAX as below in 20.02 + 19.11.1
>>
>> That would still be an ABI breakage though right.
>
> Yes.
>
>
>>
>>> b) wait 20.11 and revert Chacha-Poly from 20.02
>>
>> Thanks for analysis above Fiona, Ferruh and all.
>>
>> That is a nasty one alright - there is no "good" answer here.
>> I agree with Ferruh's sentiments overall, we should rethink this API for 20.11.
>> Could do without an enumeration?
>>
>> There a c) though right.
>> We could work around the issue by api versioning rte_cryptodev_info_get() and friends.
>
> It has a lot of friends, but it sounds like the right approach.
+1
> Is someone looking into this?
Looks to be in hand now.
>
>
>> So they only support/acknowledge the existence of Chacha-Poly for applications build against > 20.02.
>>
>> It would be painful I know.
>> It would also mean that Chacha-Poly would only be available to those building against >= 20.02.
>
> Yes.
>
>
> --
> David Marchand
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-04 12:02 9% ` Neil Horman
@ 2020-02-05 10:04 4% ` Luca Boccassi
2020-02-05 11:32 7% ` Neil Horman
0 siblings, 1 reply; 200+ results
From: Luca Boccassi @ 2020-02-05 10:04 UTC (permalink / raw)
To: Neil Horman, Ferruh Yigit
Cc: Cristian Dumitrescu, Eelco Chaudron, dev, Thomas Monjalon,
David Marchand, Bruce Richardson, Ian Stokes
On Tue, 2020-02-04 at 07:02 -0500, Neil Horman wrote:
> > But if we can do the versioning in the master, LTS can backport it
> > and can have
> > mature version of that API in LTS without breaking the existing
> > users.
> >
>
> But why bother? The only thing you've changed is the version
> tagging. Its ok
> to leave that alone in LTS, you just cant change it.
>
> Thats part of the burden of an LTS release, it will have some drift
> from the
> upstream head, because you have to keep things stable. So you
> stabilize the
> upstream ABI version for this API and just never backport it to the
> current LTS
> release.
Hi,
A customer (OVS) explicitly and specifically requested backporting the
symbol change to 19.11, as they don't want to enable experimental APIs
in their releases. I replied that it would only be acceptable with
aliasing to keep compatibility, and Ferruh very kindly did the work to
implement that.
--
Kind regards,
Luca Boccassi
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH 00/14] cleanup resources on shutdown
@ 2020-02-05 9:32 0% ` David Marchand
2020-02-05 12:07 0% ` Stephen Hemminger
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-02-05 9:32 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: dev, Luca Boccassi, Bruce Richardson
Hello Stephen,
On Sat, Jan 4, 2020 at 2:34 AM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> Recently started using valgrind with DPDK, and the results
> are not clean.
>
> The DPDK has a function that applications can use to tell it
> to cleanup resources on shutdown (rte_eal_cleanup). But the
> current coverage of that API is spotty. Many internal parts of
> DPDK leave files and allocated memory behind.
>
> This patch set is a start at getting the sub-parts of
> DPDK to cleanup after themselves. These are the easier ones,
> the harder and more critical ones are in the drivers
> and the memory subsystem.
>
> There are no visible API or ABI changes here.
Could you share what you did to run a dpdk application with valgrind?
I tried with testpmd and a 3.15 valgrind (fc30), but I get an init
failure on the cpu flags.
$ LD_LIBRARY_PATH=/home/dmarchan/builds/build-gcc-shared/install/usr/local/lib64
valgrind /home/dmarchan/builds/build-gcc-shared/install/usr/local/bin/dpdk-testpmd
-c 3 --no-huge -m 20 -d librte_mempool_ring.so -d librte_pmd_null.so
-w 0:0.0 --vdev net_null1 --vdev net_null2 -- --no-mlockall
--total-num-mbufs=2048 -ia
==10258== Memcheck, a memory error detector
==10258== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==10258== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==10258== Command:
/home/dmarchan/builds/build-gcc-shared/install/usr/local/bin/dpdk-testpmd
-c 3 --no-huge -m 20 -d librte_mempool_ring.so -d librte_pmd_null.so
-w 0:0.0 --vdev net_null1 --vdev net_null2 -- --no-mlockall
--total-num-mbufs=2048 -ia
==10258==
ERROR: This system does not support "RDSEED".
Please check that RTE_MACHINE is set correctly.
EAL: FATAL: unsupported cpu type.
EAL: unsupported cpu type.
EAL: Error - exiting with code: 1
Cause: Cannot init EAL: Operation not supported
==10258==
==10258== HEAP SUMMARY:
==10258== in use at exit: 1,388 bytes in 49 blocks
==10258== total heap usage: 97 allocs, 48 frees, 89,426 bytes allocated
==10258==
==10258== LEAK SUMMARY:
==10258== definitely lost: 0 bytes in 0 blocks
==10258== indirectly lost: 0 bytes in 0 blocks
==10258== possibly lost: 0 bytes in 0 blocks
==10258== still reachable: 1,388 bytes in 49 blocks
==10258== suppressed: 0 bytes in 0 blocks
==10258== Rerun with --leak-check=full to see details of leaked memory
==10258==
==10258== For lists of detected and suppressed errors, rerun with: -s
==10258== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
Thanks.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [EXT] Re: [PATCH v2 4/4] add ABI checks
2020-02-04 22:10 7% ` Neil Horman
@ 2020-02-05 6:16 7% ` Anoob Joseph
2020-02-05 14:33 4% ` Trahe, Fiona
0 siblings, 1 reply; 200+ results
From: Anoob Joseph @ 2020-02-05 6:16 UTC (permalink / raw)
To: Neil Horman, Akhil Goyal, Trahe, Fiona
Cc: Thomas Monjalon, Ferruh Yigit, Ananyev, Konstantin, Trahe, Fiona,
dev, David Marchand, Kusztal, ArkadiuszX, Richardson, Bruce,
Mcnamara, John, dodji, Andrew Rybchenko, aconole, bluca,
ktraynor
Hi Akhil, Neil, Fiona
Sorry for the late response. I want to propose a new change in line with what you folks had proposed.
May be we can treat the new features EXPERIMENTAL until a new stable release.
enum rte_crypto_aead_algorithm {
RTE_CRYPTO_AEAD_AES_CCM = 1,
/**< AES algorithm in CCM mode. */
RTE_CRYPTO_AEAD_AES_GCM,
/**< AES algorithm in GCM mode. */
RTE_CRYPTO_AEAD_LIST_END,
/**< List end for stable */
/** EXPERIMENTAL */
RTE_CRYPTO_AEAD_CHACHA20_POLY1305,
/**< Chacha20 cipher with poly1305 authenticator */
RTE_CRYPTO_AEAD_LIST_END_EXPERIMENTAL
/**< List end */
};
And then introduce an experimental API,
const struct rte_cryptodev_capabilities *
rte_cryptodev_get_exp_capabilites(uint8_t dev_id);
The PMD owner is expected to add new capabilities (only new ones) to this one until the new feature is deemed stable (ie, in one of the next stable releases). We don't expect users to change their API/ABI. For applications using EXPERIMENTAL is allowed to use the above capabilities to get the EXPERIMENTAL features.
This does involve moving around code in PMD when one feature is added, but that's the risk PMD owner is taking by upstreaming as EXPERIMENTAL and not in stable release.
Thanks,
Anoob
> -----Original Message-----
> From: Neil Horman <nhorman@tuxdriver.com>
> Sent: Wednesday, February 5, 2020 3:40 AM
> To: Akhil Goyal <akhil.goyal@nxp.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>; Ferruh Yigit
> <ferruh.yigit@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> dev@dpdk.org; David Marchand <david.marchand@redhat.com>; Anoob Joseph
> <anoobj@marvell.com>; Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>;
> Richardson, Bruce <bruce.richardson@intel.com>; Mcnamara, John
> <john.mcnamara@intel.com>; dodji@seketeli.net; Andrew Rybchenko
> <arybchenko@solarflare.com>; aconole@redhat.com; bluca@debian.org;
> ktraynor@redhat.com
> Subject: [EXT] Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
>
> External Email
>
> ----------------------------------------------------------------------
> On Tue, Feb 04, 2020 at 10:32:01AM +0000, Akhil Goyal wrote:
> >
> > >
> > > 04/02/2020 11:16, Akhil Goyal:
> > > > Hi,
> > > > > On 2/3/2020 5:09 PM, Thomas Monjalon wrote:
> > > > > > 03/02/2020 10:30, Ferruh Yigit:
> > > > > >> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
> > > > > >>> 02/02/2020 14:05, Thomas Monjalon:
> > > > > >>>> 31/01/2020 15:16, Trahe, Fiona:
> > > > > >>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> > > > > >>>>>> 30/01/2020 17:09, Ferruh Yigit:
> > > > > >>>>>>> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>> I believe these enums will be used only in case of ASYM
> > > > > >>>>>>>> case which
> > > is
> > > > > experimental.
> > > > > >>>>>>>
> > > > > >>>>>>> Independent from being experiment and not, this
> > > > > >>>>>>> shouldn't be a
> > > > > problem, I think
> > > > > >>>>>>> this is a false positive.
> > > > > >>>>>>>
> > > > > >>>>>>> The ABI break can happen when a struct has been shared
> > > > > >>>>>>> between
> > > the
> > > > > application
> > > > > >>>>>>> and the library (DPDK) and the layout of that memory
> > > > > >>>>>>> know
> > > differently
> > > > > by
> > > > > >>>>>>> application and the library.
> > > > > >>>>>>>
> > > > > >>>>>>> Here in all cases, there is no layout/size change.
> > > > > >>>>>>>
> > > > > >>>>>>> As to the value changes of the enums, since application
> > > > > >>>>>>> compiled
> > > with
> > > > > old DPDK,
> > > > > >>>>>>> it will know only up to '6', 7 and more means invalid to
> > > > > >>>>>>> the
> > > application.
> > > > > So it
> > > > > >>>>>>> won't send these values also it should ignore these
> > > > > >>>>>>> values from
> > > library.
> > > > > Only
> > > > > >>>>>>> consequence is old application won't able to use new
> > > > > >>>>>>> features
> > > those
> > > > > new enums
> > > > > >>>>>>> provide but that is expected/normal.
> > > > > >>>>>>
> > > > > >>>>>> If library give higher value than expected by the
> > > > > >>>>>> application, if the application uses this value as array
> > > > > >>>>>> index, there can be an access out of bounds.
> > > > > >>>>>
> > > > > >>>>> [Fiona] All asymmetric APIs are experimental so above
> > > > > >>>>> shouldn't be a
> > > > > problem.
> > > > > >>>>> But for the same issue with sym crypto below, I believe
> > > > > >>>>> Ferruh's
> > > > > explanation makes
> > > > > >>>>> sense and I don't see how there can be an API breakage.
> > > > > >>>>> So if an application hasn't compiled against the new lib
> > > > > >>>>> it will be still
> > > using
> > > > > the old value
> > > > > >>>>> which will be within bounds. If it's picking up the higher
> > > > > >>>>> new value
> > > from
> > > > > the lib it must
> > > > > >>>>> have been compiled against the lib so shouldn't have problems.
> > > > > >>>>
> > > > > >>>> You say there is no ABI issue because the application will
> > > > > >>>> be re-
> > > compiled
> > > > > >>>> for the updated library. Indeed, compilation fixes compatibility
> issues.
> > > > > >>>> But this is not relevant for ABI compatibility.
> > > > > >>>> ABI compatibility means we can upgrade the library without
> > > recompiling
> > > > > >>>> the application and it must work.
> > > > > >>>> You think it is a false positive because you assume the
> > > > > >>>> application "picks" the new value. I think you miss the
> > > > > >>>> case where the new value is returned by a function in the upgraded
> library.
> > > > > >>>>
> > > > > >>>>> There are also no structs on the API which contain arrays
> > > > > >>>>> using this for sizing, so I don't see an opportunity for
> > > > > >>>>> an appl to have a mismatch in memory addresses.
> > > > > >>>>
> > > > > >>>> Let me demonstrate where the API may "use" the new value
> > > > > >>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the
> > > > > application.
> > > > > >>>>
> > > > > >>>> Once upon a time a DPDK application counting the number of
> > > > > >>>> devices supporting each AEAD algo (in order to find the best
> supported algo).
> > > > > >>>> It is done in an array indexed by algo id:
> > > > > >>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
> > > > > >>>> The application is compiled with DPDK 19.11, where
> > > > > >>>> RTE_CRYPTO_AEAD_LIST_END = 3.
> > > > > >>>> So the size of the application array aead_dev_count is 3.
> > > > > >>>> This binary is run with DPDK 20.02, where
> > > > > >>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
> > > > > >>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
> > > > > >>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
> > > > > >>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
> > > > > >>>> The application uses this value:
> > > > > >>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
> > > > > >>>> The application is crashing because of out of bound access.
> > > > > >>>
> > > > > >>> I'd say this is an example of bad written app.
> > > > > >>> It probably should check that returned by library value
> > > > > >>> doesn't exceed its internal array size.
> > > > > >>
> > > > > >> +1
> > > > > >>
> > > > > >> Application should ignore values >= MAX.
> > > > > >
> > > > > > Of course, blaming the API user is a lot easier than looking at the API.
> > > > > > Here the API has RTE_CRYPTO_AEAD_LIST_END which can be
> > > > > > understood as the max value for the application.
> > > > > > Value ranges are part of the ABI compatibility contract.
> > > > > > It seems you expect the application developer to be aware that
> > > > > > DPDK could return a higher value, so the application should
> > > > > > check every enum values after calling an API. CRAZY.
> > > > > >
> > > > > > When we decide to announce an ABI compatibility and do some
> > > > > > marketing, everyone is OK. But when we need to really make our
> > > > > > ABI compatible, I see little or no effort. DISAPPOINTING.
> > > > >
> > > > > This is not to blame the user or to do less work, this is more
> > > > > sane approach that library provides the _END/_MAX value and
> > > > > application uses it as valid
> > > range
> > > > > check.
> > > > >
> > > > > >
> > > > > >> Do you suggest we don't extend any enum or define between ABI
> > > breakage
> > > > > releases
> > > > > >> to be sure bad written applications not affected?
> > > > > >
> > > > > > I suggest we must consider not breaking any assumption made on the
> API.
> > > > > > Here we are breaking the enum range because nothing mentions
> > > _LIST_END
> > > > > > is not really the absolute end of the enum.
> > > > > > The solution is to make the change below in 20.02 + backport in
> 19.11.1:
> > > > > >
> > > > > > - _LIST_END
> > > > > > + _LIST_END, /* an ABI-compatible version may increase this
> > > > > > + value */ _LIST_MAX = _LIST_END + 42 /* room for
> > > > > > + ABI-compatible additions */
> > > > > > };
> > > > > >
> > > > >
> > > > > What is the point of "_LIST_MAX" here?
> > > > >
> > > > > Application should know the "_LIST_END" of when it has been
> > > > > compiled for
> > > the
> > > > > valid range check. Next time it is compiled "_LIST_END" may be
> > > > > different
> > > value
> > > > > but same logic applies.
> > > > >
> > > > > When "_LIST_END" is missing, application can't protect itself,
> > > > > in that case library should send only the values application
> > > > > knows when it is compiled,
> > > this
> > > > > means either we can't extend our enum/defines until next ABI
> > > > > breakage, or
> > > we
> > > > > need to do ABI versioning to the functions that returns an enum
> > > > > each time
> > > enum
> > > > > value extended.
> > > > >
> > > > > I believe it is saner to provide _END/_MAX values to the application to
> use.
> > > And
> > > > > if required comment them to clarify the expected usage.
> > > > >
> > > > > But in above suggestion application can't use or rely on
> > > > > "_LIST_MAX", it
> > > doesn't
> > > > > mean anything to application.
> > > > >
> > > >
> > > > Can we have something like
> > > > enum rte_crypto_aead_algorithm {
> > > > RTE_CRYPTO_AEAD_AES_CCM = 1,
> > > > /**< AES algorithm in CCM mode. */
> > > > RTE_CRYPTO_AEAD_AES_GCM,
> > > > /**< AES algorithm in GCM mode. */
> > > > RTE_CRYPTO_AEAD_LIST_END,
> > > > /**< List end for 19.11 ABI compatibility */
> > > > RTE_CRYPTO_AEAD_CHACHA20_POLY1305,
> > > > /**< Chacha20 cipher with poly1305 authenticator */
> > > > RTE_CRYPTO_AEAD_LIST_END_2011
> > > > /**< List end for 20.11 ABI compatibility */ };
> > > >
> > > > And in 20.11 release we alter the RTE_CRYPTO_AEAD_LIST_END to the
> > > > end
> > > and remove RTE_CRYPTO_AEAD_LIST_END_2011
> > > >
> > > > I believe it will be ok for any application which need to use the
> > > > chacha poly
> > > assume that this algo is
> > > > Experimental and will move to formal list in 20.11. This can be
> > > > documented in
> > > the documentation.
> > > > I believe there is no way to add a new enum as experimental so
> > > > far. This way
> > > we can formalize this
> > > > requirement as well.
> > > >
> > > > I believe this way effect of ABI breakage will be nullified.
> > >
> > > This is a possibility in the (a) proposal.
> > > But it breaks API (and ABI) because a high value is returned while
> > > not expected by the application.
> > >
> > > I guess ABI and release maintainers will vote no to such breakage.
> > > Note: I vote no.
> > >
> >
> > If that is the case, I would say we should go with b).
> >
> > Versioned APIs does not look good and adds more confusion.
> >
> What makes you say that?
>
> Versioned APIs are the way you maintain backward compatibility.
>
> If a library doesn't use versioned API's, then they either:
>
> 1) break frequently, causing application headaches
> 2) have APIS that are so mature, strictly defined, and small, they never change
> anyway
> 3) go to the trouble of creating compat libs for as far back as they need to
> support
>
> DPDK doesn't yet have a mature, stable API, so we have to do (1) or (2). (1) has
> already been declared a bad idea, because application developers and distros
> have declared a desire for backwards compatibility. We could go with (3)
> instead of ABI versioning, but between compat libs and versioning, the latter is
> the much less difficult way to handle that.
>
> Neil
^ permalink raw reply [relevance 7%]
* [dpdk-dev] [RFC PATCH 0/7] vfio/pci: SR-IOV support
@ 2020-02-04 23:05 3% Alex Williamson
2020-02-11 11:18 0% ` Jerin Jacob
0 siblings, 1 reply; 200+ results
From: Alex Williamson @ 2020-02-04 23:05 UTC (permalink / raw)
To: kvm
Cc: linux-pci, linux-kernel, dev, mtosatti, thomas, bluca,
jerinjacobk, bruce.richardson, cohuck
There seems to be an ongoing desire to use userspace, vfio-based
drivers for both SR-IOV PF and VF devices. The fundamental issue
with this concept is that the VF is not fully independent of the PF
driver. Minimally the PF driver might be able to deny service to the
VF, VF data paths might be dependent on the state of the PF device,
or the PF my have some degree of ability to inspect or manipulate the
VF data. It therefore would seem irresponsible to unleash VFs onto
the system, managed by a user owned PF.
We address this in a few ways in this series. First, we can use a bus
notifier and the driver_override facility to make sure VFs are bound
to the vfio-pci driver by default. This should eliminate the chance
that a VF is accidentally bound and used by host drivers. We don't
however remove the ability for a host admin to change this override.
The next issue we need to address is how we let userspace drivers
opt-in to this participation with the PF driver. We do not want an
admin to be able to unwittingly assign one of these VFs to a tenant
that isn't working in collaboration with the PF driver. We could use
IOMMU grouping, but this seems to push too far towards tightly coupled
PF and VF drivers. This series introduces a "VF token", implemented
as a UUID, as a shared secret between PF and VF drivers. The token
needs to be set by the PF driver and used as part of the device
matching by the VF driver. Provisions in the code also account for
restarting the PF driver with active VF drivers, requiring the PF to
use the current token to re-gain access to the PF.
The above solutions introduce a bit of a modification to the VFIO ABI
and an additional ABI extension. The modification is that the
VFIO_GROUP_GET_DEVICE_FD ioctl is specified to require a char string
from the user providing the device name. For this solution, we extend
the syntax to allow the device name followed by key/value pairs. In
this case we add "vf_token=3e7e882e-1daf-417f-ad8d-882eea5ee337", for
example. These options are expected to be space separated. Matching
these key/value pairs is entirely left to the vfio bus driver (ex.
vfio-pci) and the internal ops structure is extended to allow this
optional support. This extension should be fully backwards compatible
to existing userspace, such code will simply fail to open these newly
exposed devices, as intended.
I've been debating whether instead of the above we should allow the
user to get the device fd as normal, but restrict the interfaces until
the user authenticates, but I'm afraid this would be a less backwards
compatible solution. It would be just as unclear to the user why a
device read/write/mmap/ioctl failed as it might be to why getting the
device fd could fail. However in the latter case, I believe we do a
better job of restricting how far userspace code might go before they
ultimately fail. I'd welcome discussion in the space, and or course
the extension of the GET_DEVICE_FD string.
Finally, the user needs to be able to set a VF token. I add a
VFIO_DEVICE_FEATURE ioctl for this that's meant to be reusable for
getting, setting, and probing arbitrary features of a device.
I'll reply to this cover letter with a very basic example of a QEMU
update to support this interface, though I haven't found a device yet
that behaves well with the PF running in one VM with the VF in
another, or really even just a PF running in a VM with SR-IOV enabled.
I know these devices exist though, and I suspect QEMU will not be the
primary user of this support for now, but this behavior reaffirms my
concerns to prevent mis-use.
Please comment. In particular, does this approach meet the DPDK needs
for userspace PF and VF drivers, with the hopefully minor hurdle of
sharing a token between drivers. The token is of course left to
userspace how to manage, and might be static (and not very secret) for
a given set of drivers. Thanks,
Alex
---
Alex Williamson (7):
vfio: Include optional device match in vfio_device_ops callbacks
vfio/pci: Implement match ops
vfio/pci: Introduce VF token
vfio: Introduce VFIO_DEVICE_FEATURE ioctl and first user
vfio/pci: Add sriov_configure support
vfio/pci: Remove dev_fmt definition
vfio/pci: Cleanup .probe() exit paths
drivers/vfio/pci/vfio_pci.c | 315 ++++++++++++++++++++++++++++++++---
drivers/vfio/pci/vfio_pci_private.h | 10 +
drivers/vfio/vfio.c | 19 ++
include/linux/vfio.h | 3
include/uapi/linux/vfio.h | 37 ++++
5 files changed, 356 insertions(+), 28 deletions(-)
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-04 10:32 4% ` Akhil Goyal
2020-02-04 11:35 4% ` Bruce Richardson
@ 2020-02-04 22:10 7% ` Neil Horman
2020-02-05 6:16 7% ` [dpdk-dev] [EXT] " Anoob Joseph
1 sibling, 1 reply; 200+ results
From: Neil Horman @ 2020-02-04 22:10 UTC (permalink / raw)
To: Akhil Goyal
Cc: Thomas Monjalon, Ferruh Yigit, Ananyev, Konstantin, Trahe, Fiona,
dev, David Marchand, Anoob Joseph, Kusztal, ArkadiuszX,
Richardson, Bruce, Mcnamara, John, dodji, Andrew Rybchenko,
aconole, bluca, ktraynor
On Tue, Feb 04, 2020 at 10:32:01AM +0000, Akhil Goyal wrote:
>
> >
> > 04/02/2020 11:16, Akhil Goyal:
> > > Hi,
> > > > On 2/3/2020 5:09 PM, Thomas Monjalon wrote:
> > > > > 03/02/2020 10:30, Ferruh Yigit:
> > > > >> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
> > > > >>> 02/02/2020 14:05, Thomas Monjalon:
> > > > >>>> 31/01/2020 15:16, Trahe, Fiona:
> > > > >>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> > > > >>>>>> 30/01/2020 17:09, Ferruh Yigit:
> > > > >>>>>>> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
> > > > >>>>>>>>
> > > > >>>>>>>> I believe these enums will be used only in case of ASYM case which
> > is
> > > > experimental.
> > > > >>>>>>>
> > > > >>>>>>> Independent from being experiment and not, this shouldn't be a
> > > > problem, I think
> > > > >>>>>>> this is a false positive.
> > > > >>>>>>>
> > > > >>>>>>> The ABI break can happen when a struct has been shared between
> > the
> > > > application
> > > > >>>>>>> and the library (DPDK) and the layout of that memory know
> > differently
> > > > by
> > > > >>>>>>> application and the library.
> > > > >>>>>>>
> > > > >>>>>>> Here in all cases, there is no layout/size change.
> > > > >>>>>>>
> > > > >>>>>>> As to the value changes of the enums, since application compiled
> > with
> > > > old DPDK,
> > > > >>>>>>> it will know only up to '6', 7 and more means invalid to the
> > application.
> > > > So it
> > > > >>>>>>> won't send these values also it should ignore these values from
> > library.
> > > > Only
> > > > >>>>>>> consequence is old application won't able to use new features
> > those
> > > > new enums
> > > > >>>>>>> provide but that is expected/normal.
> > > > >>>>>>
> > > > >>>>>> If library give higher value than expected by the application,
> > > > >>>>>> if the application uses this value as array index,
> > > > >>>>>> there can be an access out of bounds.
> > > > >>>>>
> > > > >>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a
> > > > problem.
> > > > >>>>> But for the same issue with sym crypto below, I believe Ferruh's
> > > > explanation makes
> > > > >>>>> sense and I don't see how there can be an API breakage.
> > > > >>>>> So if an application hasn't compiled against the new lib it will be still
> > using
> > > > the old value
> > > > >>>>> which will be within bounds. If it's picking up the higher new value
> > from
> > > > the lib it must
> > > > >>>>> have been compiled against the lib so shouldn't have problems.
> > > > >>>>
> > > > >>>> You say there is no ABI issue because the application will be re-
> > compiled
> > > > >>>> for the updated library. Indeed, compilation fixes compatibility issues.
> > > > >>>> But this is not relevant for ABI compatibility.
> > > > >>>> ABI compatibility means we can upgrade the library without
> > recompiling
> > > > >>>> the application and it must work.
> > > > >>>> You think it is a false positive because you assume the application
> > > > >>>> "picks" the new value. I think you miss the case where the new value
> > > > >>>> is returned by a function in the upgraded library.
> > > > >>>>
> > > > >>>>> There are also no structs on the API which contain arrays using this
> > > > >>>>> for sizing, so I don't see an opportunity for an appl to have a
> > > > >>>>> mismatch in memory addresses.
> > > > >>>>
> > > > >>>> Let me demonstrate where the API may "use" the new value
> > > > >>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the
> > > > application.
> > > > >>>>
> > > > >>>> Once upon a time a DPDK application counting the number of devices
> > > > >>>> supporting each AEAD algo (in order to find the best supported algo).
> > > > >>>> It is done in an array indexed by algo id:
> > > > >>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
> > > > >>>> The application is compiled with DPDK 19.11,
> > > > >>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
> > > > >>>> So the size of the application array aead_dev_count is 3.
> > > > >>>> This binary is run with DPDK 20.02,
> > > > >>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
> > > > >>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
> > > > >>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
> > > > >>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
> > > > >>>> The application uses this value:
> > > > >>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
> > > > >>>> The application is crashing because of out of bound access.
> > > > >>>
> > > > >>> I'd say this is an example of bad written app.
> > > > >>> It probably should check that returned by library value doesn't
> > > > >>> exceed its internal array size.
> > > > >>
> > > > >> +1
> > > > >>
> > > > >> Application should ignore values >= MAX.
> > > > >
> > > > > Of course, blaming the API user is a lot easier than looking at the API.
> > > > > Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
> > > > > as the max value for the application.
> > > > > Value ranges are part of the ABI compatibility contract.
> > > > > It seems you expect the application developer to be aware that
> > > > > DPDK could return a higher value, so the application should
> > > > > check every enum values after calling an API. CRAZY.
> > > > >
> > > > > When we decide to announce an ABI compatibility and do some marketing,
> > > > > everyone is OK. But when we need to really make our ABI compatible,
> > > > > I see little or no effort. DISAPPOINTING.
> > > >
> > > > This is not to blame the user or to do less work, this is more sane approach
> > > > that library provides the _END/_MAX value and application uses it as valid
> > range
> > > > check.
> > > >
> > > > >
> > > > >> Do you suggest we don't extend any enum or define between ABI
> > breakage
> > > > releases
> > > > >> to be sure bad written applications not affected?
> > > > >
> > > > > I suggest we must consider not breaking any assumption made on the API.
> > > > > Here we are breaking the enum range because nothing mentions
> > _LIST_END
> > > > > is not really the absolute end of the enum.
> > > > > The solution is to make the change below in 20.02 + backport in 19.11.1:
> > > > >
> > > > > - _LIST_END
> > > > > + _LIST_END, /* an ABI-compatible version may increase this value */
> > > > > + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
> > > > > };
> > > > >
> > > >
> > > > What is the point of "_LIST_MAX" here?
> > > >
> > > > Application should know the "_LIST_END" of when it has been compiled for
> > the
> > > > valid range check. Next time it is compiled "_LIST_END" may be different
> > value
> > > > but same logic applies.
> > > >
> > > > When "_LIST_END" is missing, application can't protect itself, in that case
> > > > library should send only the values application knows when it is compiled,
> > this
> > > > means either we can't extend our enum/defines until next ABI breakage, or
> > we
> > > > need to do ABI versioning to the functions that returns an enum each time
> > enum
> > > > value extended.
> > > >
> > > > I believe it is saner to provide _END/_MAX values to the application to use.
> > And
> > > > if required comment them to clarify the expected usage.
> > > >
> > > > But in above suggestion application can't use or rely on "_LIST_MAX", it
> > doesn't
> > > > mean anything to application.
> > > >
> > >
> > > Can we have something like
> > > enum rte_crypto_aead_algorithm {
> > > RTE_CRYPTO_AEAD_AES_CCM = 1,
> > > /**< AES algorithm in CCM mode. */
> > > RTE_CRYPTO_AEAD_AES_GCM,
> > > /**< AES algorithm in GCM mode. */
> > > RTE_CRYPTO_AEAD_LIST_END,
> > > /**< List end for 19.11 ABI compatibility */
> > > RTE_CRYPTO_AEAD_CHACHA20_POLY1305,
> > > /**< Chacha20 cipher with poly1305 authenticator */
> > > RTE_CRYPTO_AEAD_LIST_END_2011
> > > /**< List end for 20.11 ABI compatibility */
> > > };
> > >
> > > And in 20.11 release we alter the RTE_CRYPTO_AEAD_LIST_END to the end
> > and remove RTE_CRYPTO_AEAD_LIST_END_2011
> > >
> > > I believe it will be ok for any application which need to use the chacha poly
> > assume that this algo is
> > > Experimental and will move to formal list in 20.11. This can be documented in
> > the documentation.
> > > I believe there is no way to add a new enum as experimental so far. This way
> > we can formalize this
> > > requirement as well.
> > >
> > > I believe this way effect of ABI breakage will be nullified.
> >
> > This is a possibility in the (a) proposal.
> > But it breaks API (and ABI) because a high value is returned
> > while not expected by the application.
> >
> > I guess ABI and release maintainers will vote no to such breakage.
> > Note: I vote no.
> >
>
> If that is the case, I would say we should go with b).
>
> Versioned APIs does not look good and adds more confusion.
>
What makes you say that?
Versioned APIs are the way you maintain backward compatibility.
If a library doesn't use versioned API's, then they either:
1) break frequently, causing application headaches
2) have APIS that are so mature, strictly defined, and small, they never change anyway
3) go to the trouble of creating compat libs for as far back as they need to
support
DPDK doesn't yet have a mature, stable API, so we have to do (1) or (2). (1)
has already been declared a bad idea, because application developers and distros
have declared a desire for backwards compatibility. We could go with (3)
instead of ABI versioning, but between compat libs and versioning, the latter is
the much less difficult way to handle that.
Neil
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-04 10:16 9% ` Akhil Goyal
2020-02-04 10:28 8% ` Thomas Monjalon
@ 2020-02-04 21:59 8% ` Neil Horman
1 sibling, 0 replies; 200+ results
From: Neil Horman @ 2020-02-04 21:59 UTC (permalink / raw)
To: Akhil Goyal
Cc: Ferruh Yigit, Thomas Monjalon, Ananyev, Konstantin, Trahe, Fiona,
David Marchand, Anoob Joseph, Kusztal, ArkadiuszX, dev,
Richardson, Bruce, Mcnamara, John, dodji, Andrew Rybchenko,
aconole, bluca, ktraynor
On Tue, Feb 04, 2020 at 10:16:56AM +0000, Akhil Goyal wrote:
> Hi,
> > On 2/3/2020 5:09 PM, Thomas Monjalon wrote:
> > > 03/02/2020 10:30, Ferruh Yigit:
> > >> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
> > >>> 02/02/2020 14:05, Thomas Monjalon:
> > >>>> 31/01/2020 15:16, Trahe, Fiona:
> > >>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> > >>>>>> 30/01/2020 17:09, Ferruh Yigit:
> > >>>>>>> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
> > >>>>>>>>
> > >>>>>>>> I believe these enums will be used only in case of ASYM case which is
> > experimental.
> > >>>>>>>
> > >>>>>>> Independent from being experiment and not, this shouldn't be a
> > problem, I think
> > >>>>>>> this is a false positive.
> > >>>>>>>
> > >>>>>>> The ABI break can happen when a struct has been shared between the
> > application
> > >>>>>>> and the library (DPDK) and the layout of that memory know differently
> > by
> > >>>>>>> application and the library.
> > >>>>>>>
> > >>>>>>> Here in all cases, there is no layout/size change.
> > >>>>>>>
> > >>>>>>> As to the value changes of the enums, since application compiled with
> > old DPDK,
> > >>>>>>> it will know only up to '6', 7 and more means invalid to the application.
> > So it
> > >>>>>>> won't send these values also it should ignore these values from library.
> > Only
> > >>>>>>> consequence is old application won't able to use new features those
> > new enums
> > >>>>>>> provide but that is expected/normal.
> > >>>>>>
> > >>>>>> If library give higher value than expected by the application,
> > >>>>>> if the application uses this value as array index,
> > >>>>>> there can be an access out of bounds.
> > >>>>>
> > >>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a
> > problem.
> > >>>>> But for the same issue with sym crypto below, I believe Ferruh's
> > explanation makes
> > >>>>> sense and I don't see how there can be an API breakage.
> > >>>>> So if an application hasn't compiled against the new lib it will be still using
> > the old value
> > >>>>> which will be within bounds. If it's picking up the higher new value from
> > the lib it must
> > >>>>> have been compiled against the lib so shouldn't have problems.
> > >>>>
> > >>>> You say there is no ABI issue because the application will be re-compiled
> > >>>> for the updated library. Indeed, compilation fixes compatibility issues.
> > >>>> But this is not relevant for ABI compatibility.
> > >>>> ABI compatibility means we can upgrade the library without recompiling
> > >>>> the application and it must work.
> > >>>> You think it is a false positive because you assume the application
> > >>>> "picks" the new value. I think you miss the case where the new value
> > >>>> is returned by a function in the upgraded library.
> > >>>>
> > >>>>> There are also no structs on the API which contain arrays using this
> > >>>>> for sizing, so I don't see an opportunity for an appl to have a
> > >>>>> mismatch in memory addresses.
> > >>>>
> > >>>> Let me demonstrate where the API may "use" the new value
> > >>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the
> > application.
> > >>>>
> > >>>> Once upon a time a DPDK application counting the number of devices
> > >>>> supporting each AEAD algo (in order to find the best supported algo).
> > >>>> It is done in an array indexed by algo id:
> > >>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
> > >>>> The application is compiled with DPDK 19.11,
> > >>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
> > >>>> So the size of the application array aead_dev_count is 3.
> > >>>> This binary is run with DPDK 20.02,
> > >>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
> > >>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
> > >>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
> > >>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
> > >>>> The application uses this value:
> > >>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
> > >>>> The application is crashing because of out of bound access.
> > >>>
> > >>> I'd say this is an example of bad written app.
> > >>> It probably should check that returned by library value doesn't
> > >>> exceed its internal array size.
> > >>
> > >> +1
> > >>
> > >> Application should ignore values >= MAX.
> > >
> > > Of course, blaming the API user is a lot easier than looking at the API.
> > > Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
> > > as the max value for the application.
> > > Value ranges are part of the ABI compatibility contract.
> > > It seems you expect the application developer to be aware that
> > > DPDK could return a higher value, so the application should
> > > check every enum values after calling an API. CRAZY.
> > >
> > > When we decide to announce an ABI compatibility and do some marketing,
> > > everyone is OK. But when we need to really make our ABI compatible,
> > > I see little or no effort. DISAPPOINTING.
> >
> > This is not to blame the user or to do less work, this is more sane approach
> > that library provides the _END/_MAX value and application uses it as valid range
> > check.
> >
> > >
> > >> Do you suggest we don't extend any enum or define between ABI breakage
> > releases
> > >> to be sure bad written applications not affected?
> > >
> > > I suggest we must consider not breaking any assumption made on the API.
> > > Here we are breaking the enum range because nothing mentions _LIST_END
> > > is not really the absolute end of the enum.
> > > The solution is to make the change below in 20.02 + backport in 19.11.1:
> > >
> > > - _LIST_END
> > > + _LIST_END, /* an ABI-compatible version may increase this value */
> > > + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
> > > };
> > >
> >
> > What is the point of "_LIST_MAX" here?
> >
> > Application should know the "_LIST_END" of when it has been compiled for the
> > valid range check. Next time it is compiled "_LIST_END" may be different value
> > but same logic applies.
> >
> > When "_LIST_END" is missing, application can't protect itself, in that case
> > library should send only the values application knows when it is compiled, this
> > means either we can't extend our enum/defines until next ABI breakage, or we
> > need to do ABI versioning to the functions that returns an enum each time enum
> > value extended.
> >
> > I believe it is saner to provide _END/_MAX values to the application to use. And
> > if required comment them to clarify the expected usage.
> >
> > But in above suggestion application can't use or rely on "_LIST_MAX", it doesn't
> > mean anything to application.
> >
>
> Can we have something like
> enum rte_crypto_aead_algorithm {
> RTE_CRYPTO_AEAD_AES_CCM = 1,
> /**< AES algorithm in CCM mode. */
> RTE_CRYPTO_AEAD_AES_GCM,
> /**< AES algorithm in GCM mode. */
> RTE_CRYPTO_AEAD_LIST_END,
> /**< List end for 19.11 ABI compatibility */
> RTE_CRYPTO_AEAD_CHACHA20_POLY1305,
> /**< Chacha20 cipher with poly1305 authenticator */
> RTE_CRYPTO_AEAD_LIST_END_2011
> /**< List end for 20.11 ABI compatibility */
> };
>
> And in 20.11 release we alter the RTE_CRYPTO_AEAD_LIST_END to the end and remove RTE_CRYPTO_AEAD_LIST_END_2011
>
> I believe it will be ok for any application which need to use the chacha poly assume that this algo is
> Experimental and will move to formal list in 20.11. This can be documented in the documentation.
> I believe there is no way to add a new enum as experimental so far. This way we can formalize this
> requirement as well.
>
> I believe this way effect of ABI breakage will be nullified.
>
Thats not really helpful though, in that libabigail will then complain that
you've aliased an old ennumeration name in the ABI to a new name.
A better solution would be do one of the following:
a) add an API call - something like rte_crypto_get_max_alg(), which returns at
run time the maximum number of algorithms available, so that the application is
forced to learn that number at run time, rather than at compile time
b) Modify the API such that you pass in an algorithm name rather than an index
value defined by an ennumeration. You may also add an API call that dumps all
the available strings back to the user.
a) would be nice, but it would still require an ABI change, which is less than
optimal
b) is nice because it gives you flexibility in how you search for algs - if you
want to add a new alg, you just update your internal tables with a new name
string, and applications get get its information by querying based on that
string, rather than an index.
b is also nice because it can just superscede the existing implementation (i.e.
the current implementation remains unchanged, but only supports the current,
existing algs), if you want to get the new algs, you use the new API calls
(which can also find the already existing algs), and at some point in the future
we can just deprecate the old API.
Neil
>
> -Akhil
>
> > > Then *_LIST_END values could be ignored by libabigail with such a change.
> > >
> > > If such a patch is not done by tomorrow, I will have to revert
> > > Chacha-Poly commits before 20.02-rc2, because
> > >
> > > 1/ LIST_END, without any comment, means "size of range"
> > > 2/ we do not blame users for undocumented ABI changes
> > > 3/ we respect the ABI compatibility contract
> > >
> > >
>
^ permalink raw reply [relevance 8%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-04 14:44 4% ` Aaron Conole
@ 2020-02-04 19:49 7% ` Neil Horman
0 siblings, 0 replies; 200+ results
From: Neil Horman @ 2020-02-04 19:49 UTC (permalink / raw)
To: Aaron Conole
Cc: Thomas Monjalon, David Marchand, bluca, ktraynor, Ray Kinsella,
dev, Akhil Goyal, Trahe, Fiona, Ferruh Yigit, Ananyev,
Konstantin, Anoob Joseph, Kusztal, ArkadiuszX, Richardson, Bruce,
Mcnamara, John, dodji, Andrew Rybchenko
On Tue, Feb 04, 2020 at 09:44:53AM -0500, Aaron Conole wrote:
> Thomas Monjalon <thomas@monjalon.net> writes:
>
> > RED FLAG
> >
> > I don't see a lot of reactions, so I summarize the issue.
> > We need action TODAY!
> >
> > API makes think that rte_cryptodev_info_get() cannot return
> > a value >= 3 (RTE_CRYPTO_AEAD_LIST_END in 19.11).
> > Current 20.02 returns 3 (RTE_CRYPTO_AEAD_CHACHA20_POLY1305).
> > The ABI compatibility contract is broken currently.
> >
> > There are 3 possible outcomes:
> >
> > a) Change the API comments and backport to 19.11.1
> > The details are discussed between Ferruh and me.
> > Either put responsibility on API user (with explicit comment),
> > or expose ABI extension allowance with a new API max value.
> > In both cases, this is breaking the implicit contract of 19.11.0.
> > This option can be chosen only if release and ABI maintainers
> > vote for it.
> >
> > b) Revert Chacha-Poly from 20.02-rc2.
> >
> > c) Add versioned function rte_cryptodev_info_get_v1911()
> > which calls rte_cryptodev_info_get() and filters out
> > RTE_CRYPTO_AEAD_CHACHA20_POLY1305 capability.
> > So Chacha-Poly capability would be seen and usable only
> > if compiling with DPDK 20.02.
> >
> > I hope it is clear what are the actions for everybody:
> > - ABI and release maintainers must say yes or no to the proposal (a)
> > - In the meantime, crypto team must send a patch for the proposal (c)
> > - If (a) and (c) are not possible at the end of today, I will take (b)
> >
> > Note: do not say it is too short for (c), as it was possible to work
> > on such solution since the issue was exposed on last Wednesday.
>
> While I'm not a maintainer, if I my opinion counts for anything, I'd
> choose option c or b. Absolutely NACK to a.
>
Agreed, options c and b are reasonable, a isn't. ABI commitments are ours, not
users.
Neil
> >
> > 03/02/2020 22:07, Thomas Monjalon:
> >> 03/02/2020 19:55, Ray Kinsella:
> >> > On 03/02/2020 17:34, Thomas Monjalon wrote:
> >> > > 03/02/2020 18:09, Thomas Monjalon:
> >> > >> 03/02/2020 10:30, Ferruh Yigit:
> >> > >>> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
> >> > >>>> 02/02/2020 14:05, Thomas Monjalon:
> >> > >>>>> 31/01/2020 15:16, Trahe, Fiona:
> >> > >>>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> >> > >>>>>>> If library give higher value than expected by the application,
> >> > >>>>>>> if the application uses this value as array index,
> >> > >>>>>>> there can be an access out of bounds.
> >> > >>>>>>
> >> > >>>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
> >> > >>>>>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
> >> > >>>>>> sense and I don't see how there can be an API breakage.
> >> > >>>>>> So if an application hasn't compiled against the new lib it
> >> > >>>>>> will be still using the old value
> >> > >>>>>> which will be within bounds. If it's picking up the higher
> >> > >>>>>> new value from the lib it must
> >> > >>>>>> have been compiled against the lib so shouldn't have problems.
> >> > >>>>>
> >> > >>>>> You say there is no ABI issue because the application will be re-compiled
> >> > >>>>> for the updated library. Indeed, compilation fixes compatibility issues.
> >> > >>>>> But this is not relevant for ABI compatibility.
> >> > >>>>> ABI compatibility means we can upgrade the library without recompiling
> >> > >>>>> the application and it must work.
> >> > >>>>> You think it is a false positive because you assume the application
> >> > >>>>> "picks" the new value. I think you miss the case where the new value
> >> > >>>>> is returned by a function in the upgraded library.
> >> > >>>>>
> >> > >>>>>> There are also no structs on the API which contain arrays using this
> >> > >>>>>> for sizing, so I don't see an opportunity for an appl to have a
> >> > >>>>>> mismatch in memory addresses.
> >> > >>>>>
> >> > >>>>> Let me demonstrate where the API may "use" the new value
> >> > >>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
> >> > >>>>>
> >> > >>>>> Once upon a time a DPDK application counting the number of devices
> >> > >>>>> supporting each AEAD algo (in order to find the best supported algo).
> >> > >>>>> It is done in an array indexed by algo id:
> >> > >>>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
> >> > >>>>> The application is compiled with DPDK 19.11,
> >> > >>>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
> >> > >>>>> So the size of the application array aead_dev_count is 3.
> >> > >>>>> This binary is run with DPDK 20.02,
> >> > >>>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
> >> > >>>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
> >> > >>>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
> >> > >>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
> >> > >>>>> The application uses this value:
> >> > >>>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
> >> > >>>>> The application is crashing because of out of bound access.
> >> > >>>>
> >> > >>>> I'd say this is an example of bad written app.
> >> > >>>> It probably should check that returned by library value doesn't
> >> > >>>> exceed its internal array size.
> >> > >>>
> >> > >>> +1
> >> > >>>
> >> > >>> Application should ignore values >= MAX.
> >> > >>
> >> > >> Of course, blaming the API user is a lot easier than looking at the API.
> >> > >> Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
> >> > >> as the max value for the application.
> >> > >> Value ranges are part of the ABI compatibility contract.
> >> > >> It seems you expect the application developer to be aware that
> >> > >> DPDK could return a higher value, so the application should
> >> > >> check every enum values after calling an API. CRAZY.
> >> > >>
> >> > >> When we decide to announce an ABI compatibility and do some marketing,
> >> > >> everyone is OK. But when we need to really make our ABI compatible,
> >> > >> I see little or no effort. DISAPPOINTING.
> >> > >>
> >> > >>> Do you suggest we don't extend any enum or define between ABI breakage releases
> >> > >>> to be sure bad written applications not affected?
> >> > >>
> >> > >> I suggest we must consider not breaking any assumption made on the API.
> >> > >> Here we are breaking the enum range because nothing mentions _LIST_END
> >> > >> is not really the absolute end of the enum.
> >> > >> The solution is to make the change below in 20.02 + backport in 19.11.1:
> >> > >
> >> > > Thinking twice, merging such change before 20.11 is breaking the
> >> > > ABI assumption based on the API 19.11.0.
> >> > > I ask the release maintainers (Luca, Kevin, David and me) and
> >> > > the ABI maintainers (Neil and Ray) to vote for a or b solution:
> >> > > a) add comment and LIST_MAX as below in 20.02 + 19.11.1
> >> >
> >> > That would still be an ABI breakage though right.
> >> >
> >> > > b) wait 20.11 and revert Chacha-Poly from 20.02
> >> >
> >> > Thanks for analysis above Fiona, Ferruh and all.
> >> >
> >> > That is a nasty one alright - there is no "good" answer here.
> >> > I agree with Ferruh's sentiments overall, we should rethink this API for 20.11.
> >> > Could do without an enumeration?
> >> >
> >> > There a c) though right.
> >> > We could work around the issue by api versioning rte_cryptodev_info_get() and friends.
> >> > So they only support/acknowledge the existence of Chacha-Poly for
> >> > applications build against > 20.02.
> >>
> >> I agree there is a c) as I proposed in another email:
> >> http://mails.dpdk.org/archives/dev/2020-February/156919.html
> >> "
> >> In this case, the proper solution is to implement
> >> rte_cryptodev_info_get_v1911() so it filters out
> >> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 capability.
> >> With this solution, an application compiled with DPDK 19.11 will keep
> >> seeing the same range as before, while a 20.02 application could
> >> see and use ChachaPoly.
> >> "
> >>
> >> > It would be painful I know.
> >>
> >> Not so painful in my opinion.
> >> Just need to call rte_cryptodev_info_get() from
> >> rte_cryptodev_info_get_v1911() and filter the value
> >> in the 19.11 range: [0..AES_GCM].
> >>
> >> > It would also mean that Chacha-Poly would only be available to
> >> > those building against >= 20.02.
> >>
> >> Yes exactly.
> >>
> >> The addition of comments and LIST_MAX like below are still valid
> >> to avoid versioning after 20.11.
> >>
> >> > >> - _LIST_END
> >> > >> + _LIST_END, /* an ABI-compatible version may increase this value */
> >> > >> + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
> >> > >> };
> >> > >>
> >> > >> Then *_LIST_END values could be ignored by libabigail with such a change.
> >>
> >> In order to avoid ABI check complaining, the best is to completely
> >> remove LIST_END in DPDK 20.11.
> >>
> >>
> >> > >> If such a patch is not done by tomorrow, I will have to revert
> >> > >> Chacha-Poly commits before 20.02-rc2, because
> >> > >>
> >> > >> 1/ LIST_END, without any comment, means "size of range"
> >> > >> 2/ we do not blame users for undocumented ABI changes
> >> > >> 3/ we respect the ABI compatibility contract
>
>
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-04 15:59 7% ` Thomas Monjalon
@ 2020-02-04 17:46 4% ` Trahe, Fiona
2020-02-13 14:51 4% ` Kusztal, ArkadiuszX
0 siblings, 1 reply; 200+ results
From: Trahe, Fiona @ 2020-02-04 17:46 UTC (permalink / raw)
To: Thomas Monjalon
Cc: David Marchand, nhorman, bluca, ktraynor, Ray Kinsella, dev,
Akhil Goyal, Yigit, Ferruh, Ananyev, Konstantin, dev,
Anoob Joseph, Kusztal, ArkadiuszX, Richardson, Bruce, Mcnamara,
John, dodji, Andrew Rybchenko, aconole, Trahe, Fiona
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Tuesday, February 4, 2020 4:00 PM
> To: Trahe, Fiona <fiona.trahe@intel.com>
> Cc: David Marchand <david.marchand@redhat.com>; nhorman@tuxdriver.com; bluca@debian.org;
> ktraynor@redhat.com; Ray Kinsella <mdr@ashroe.eu>; dev@dpdk.org; Akhil Goyal
> <akhil.goyal@nxp.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; dev@dpdk.org; Anoob Joseph <anoobj@marvell.com>; Kusztal,
> ArkadiuszX <arkadiuszx.kusztal@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>;
> Mcnamara, John <john.mcnamara@intel.com>; dodji@seketeli.net; Andrew Rybchenko
> <arybchenko@solarflare.com>; aconole@redhat.com
> Subject: Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
>
> 04/02/2020 16:52, Trahe, Fiona:
> >
> > > We are working on a patch, when it is ready we will send it.
> > > If it's not ready by end of your today, of course, go ahead with (b) and
> > > we will work towards 20.05.
> >
> > We will not be sending a patch today.
> > The patch we're working on will provide two versions of rte_cryptodev_info_get(),
> > both call the same PMD function from the dev_ops info_get fn ptr.
> > The default version operates s as normal, the 19.11 version searches
> > through the list returned by the PMD, looking for sym.aead.algo = ChaChaPoly, it needs to strip it from
> the list.
> > As PMDs just pass a ptr to their capabilities list ( it isn't a linked list, but an array
> > with an end marker = RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST) if the API
> > layer detects Chacha it must allocate some space and store a local copy of the
> > trimmed list. This must be stored only once per device.
>
> I don't understand what you have to store.
> Can't you just set the algo to 0 if it is ChaCha?
[Fiona] it returns a pointer to data in the PMD domain, which the API couldn't and shouldn't overwrite, e.g.
static const struct rte_cryptodev_capabilities qat_gen3_sym_capabilities[]
>
> > This versioning will apply to any PMD which wants to take advantage of the new API between now and
> 20.11.
> >
> > Note, I expect the ABI checker tools will still complain of ABI breakage as the LIST_END value will still
> change.
>
> Right, you need to update the ignore list for the tool.
>
> > We are also reviewing all other cryptodev APIs in case there is any other API which needs versioning.
> >
> > Anyone see any problem with this approach?
>
> The other issue is with all other functions accepting this enum as input.
> We should continue returning an error if getting Chacha as input with
> 19.11 version of these functions.
> But I would tend to consider this small ABI breakage can be ignored
> as it is in the error path.
[Fiona] The QAT PMD tests for and handles this error. I expect other PMDs do too.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v7] eal: add manual probing option
2020-02-04 15:06 0% ` Thomas Monjalon
@ 2020-02-04 16:02 0% ` Gaetan Rivet
2020-02-10 14:51 0% ` Jerin Jacob
0 siblings, 1 reply; 200+ results
From: Gaetan Rivet @ 2020-02-04 16:02 UTC (permalink / raw)
To: Thomas Monjalon, Pavan Nikhilesh Bhagavatula, Vamsi Krishna Attunuru
Cc: David Marchand, dev, Jerin Jacob Kollanukkaran
On 04/02/2020 16:06, Thomas Monjalon wrote:
> 04/02/2020 13:43, Gaetan Rivet:
>> On 04/02/2020 12:07, Thomas Monjalon wrote:
>>> 04/02/2020 11:03, Gaetan Rivet:
>>>> On 03/02/2020 23:21, Thomas Monjalon wrote:
>>>>> 03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
>>>>>> @David Marchand @thomas@monjalon.net
>>>>>>
>>>>>> Ping?
>>>>>>
>>>>>> Are there any more changes required for this patch? It's been in queue since last October.
>>>>>
>>>>> Sorry we have not decided whether it is a good idea or not.
>>>>>
>>>>> All changes related to probing are very sensitive,
>>>>> and we know a big refactoring would be better than stacking
>>>>> more and more options and corner cases.
>>>>>
>>>>> As we are busy with ABI stability stuff, we did not allocate
>>>>> enough time to properly think about this feature.
>>>>> Please accept our apologies, and let's consider it as
>>>>> a high priority for 20.05 cycle.
>>>>>
>>>>
>>>> Hello Thomas,
>>>>
>>>> This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.
>>>>
>>>> The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.
>>>
>>>
>>> Yes, life is full of bad decisions and consequences.
>>
>>
>> Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.
>>
>>>
>>> I still think there is a risk in adding new user expectations,
>>> and maintaining some code to workaround unknown issues.
>>>
>>> The real question here is to know why this patch?
>>> Is it to workaround a broken driver?
>>> Or to workaround a broken design in EAL and bus drivers?
>>
>> Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).
>>
>> I'm not sure having a dependent-probe by default is good, and that would be a big change.
>>
>> If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.
>>
>> [1]: First proposal:
>> http://mails.dpdk.org/archives/dev/2019-September/144166.html
>> My arguments:
>> http://mails.dpdk.org/archives/dev/2019-September/144564.html
>
>
> OK so there are two needs:
>
> 1/ Better control whitelist/blacklist mode.
> We already know that a rework is needed here.
> Unfortunately neither you or me had time to work on it,
> and others who were interested disappeared.
>
> 2/ Associate ports with equivalent properties in applications.
> This must be done in applications.
> Tweaking the probe order is a hack.
>
>
An application that want to tightly control the port init order, currently (by doing exactly like me here, hotpluging one by one the ports), would still need the bigger hack that consist in inserting a whitelist PCI devargs with a dummy address, depending on a undocumented PCI bus feature consisting in ignoring matching errors but keeping probing policy from failed devargs processing.
Instead, with this patch this app can do
rte_eal_manual_probe_set(1);
rte_eal_init();
to have the same behavior and be able to hotplug ports as it sees fit.
You are worried about creating user expectations about this behavior (being forced to replicate in some way the functionality during the rewrite, as I understand it?), but then you are currently forcing users to expect this workaround to exist in the PCI bus, blocking devs from touching it as it will thus break current app configurations. I've seen systemd unit file using this -w dummy flag, as well as the programmatic equivalent. Which is better, to have to rework it cutting short these configs, or to propose beforehand a deprecation path?.
This rework won't happen in 20.05, nor in the medium term unless you decide to drive this change. This workaround serves three needs (PCI normalization, port congruence and port dependency) in a low-risk implementation.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-04 15:52 8% ` Trahe, Fiona
@ 2020-02-04 15:59 7% ` Thomas Monjalon
2020-02-04 17:46 4% ` Trahe, Fiona
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-04 15:59 UTC (permalink / raw)
To: Trahe, Fiona
Cc: David Marchand, nhorman, bluca, ktraynor, Ray Kinsella, dev,
Akhil Goyal, Yigit, Ferruh, Ananyev, Konstantin, dev,
Anoob Joseph, Kusztal, ArkadiuszX, Richardson, Bruce, Mcnamara,
John, dodji, Andrew Rybchenko, aconole
04/02/2020 16:52, Trahe, Fiona:
>
> > We are working on a patch, when it is ready we will send it.
> > If it's not ready by end of your today, of course, go ahead with (b) and
> > we will work towards 20.05.
>
> We will not be sending a patch today.
> The patch we're working on will provide two versions of rte_cryptodev_info_get(),
> both call the same PMD function from the dev_ops info_get fn ptr.
> The default version operates s as normal, the 19.11 version searches
> through the list returned by the PMD, looking for sym.aead.algo = ChaChaPoly, it needs to strip it from the list.
> As PMDs just pass a ptr to their capabilities list ( it isn't a linked list, but an array
> with an end marker = RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST) if the API
> layer detects Chacha it must allocate some space and store a local copy of the
> trimmed list. This must be stored only once per device.
I don't understand what you have to store.
Can't you just set the algo to 0 if it is ChaCha?
> This versioning will apply to any PMD which wants to take advantage of the new API between now and 20.11.
>
> Note, I expect the ABI checker tools will still complain of ABI breakage as the LIST_END value will still change.
Right, you need to update the ignore list for the tool.
> We are also reviewing all other cryptodev APIs in case there is any other API which needs versioning.
>
> Anyone see any problem with this approach?
The other issue is with all other functions accepting this enum as input.
We should continue returning an error if getting Chacha as input with
19.11 version of these functions.
But I would tend to consider this small ABI breakage can be ignored
as it is in the error path.
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-04 12:44 4% ` Trahe, Fiona
@ 2020-02-04 15:52 8% ` Trahe, Fiona
2020-02-04 15:59 7% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Trahe, Fiona @ 2020-02-04 15:52 UTC (permalink / raw)
To: Thomas Monjalon, David Marchand, nhorman, bluca, ktraynor,
Ray Kinsella, dev, Akhil Goyal, Yigit, Ferruh
Cc: Ananyev, Konstantin, dev, Anoob Joseph, Kusztal, ArkadiuszX,
Richardson, Bruce, Mcnamara, John, dodji, Andrew Rybchenko,
aconole, Trahe, Fiona
> We are working on a patch, when it is ready we will send it.
> If it's not ready by end of your today, of course, go ahead with (b) and
> we will work towards 20.05.
We will not be sending a patch today.
The patch we're working on will provide two versions of rte_cryptodev_info_get(),
both call the same PMD function from the dev_ops info_get fn ptr.
The default version operates s as normal, the 19.11 version searches
through the list returned by the PMD, looking for sym.aead.algo = ChaChaPoly, it needs to strip it from the list.
As PMDs just pass a ptr to their capabilities list ( it isn't a linked list, but an array
with an end marker = RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST) if the API
layer detects Chacha it must allocate some space and store a local copy of the
trimmed list. This must be stored only once per device.
This versioning will apply to any PMD which wants to take advantage of the new API between now and 20.11.
Note, I expect the ABI checker tools will still complain of ABI breakage as the LIST_END value will still change.
We are also reviewing all other cryptodev APIs in case there is any other API which needs versioning.
Anyone see any problem with this approach?
^ permalink raw reply [relevance 8%]
* Re: [dpdk-dev] [PATCH v7] eal: add manual probing option
2020-02-04 12:43 0% ` Gaetan Rivet
@ 2020-02-04 15:06 0% ` Thomas Monjalon
2020-02-04 16:02 0% ` Gaetan Rivet
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-04 15:06 UTC (permalink / raw)
To: Pavan Nikhilesh Bhagavatula, Gaetan Rivet, Vamsi Krishna Attunuru
Cc: David Marchand, dev, Jerin Jacob Kollanukkaran
04/02/2020 13:43, Gaetan Rivet:
> On 04/02/2020 12:07, Thomas Monjalon wrote:
> > 04/02/2020 11:03, Gaetan Rivet:
> >> On 03/02/2020 23:21, Thomas Monjalon wrote:
> >>> 03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
> >>>> @David Marchand @thomas@monjalon.net
> >>>>
> >>>> Ping?
> >>>>
> >>>> Are there any more changes required for this patch? It's been in queue since last October.
> >>>
> >>> Sorry we have not decided whether it is a good idea or not.
> >>>
> >>> All changes related to probing are very sensitive,
> >>> and we know a big refactoring would be better than stacking
> >>> more and more options and corner cases.
> >>>
> >>> As we are busy with ABI stability stuff, we did not allocate
> >>> enough time to properly think about this feature.
> >>> Please accept our apologies, and let's consider it as
> >>> a high priority for 20.05 cycle.
> >>>
> >>
> >> Hello Thomas,
> >>
> >> This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.
> >>
> >> The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.
> >
> >
> > Yes, life is full of bad decisions and consequences.
>
>
> Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.
>
> >
> > I still think there is a risk in adding new user expectations,
> > and maintaining some code to workaround unknown issues.
> >
> > The real question here is to know why this patch?
> > Is it to workaround a broken driver?
> > Or to workaround a broken design in EAL and bus drivers?
>
> Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).
>
> I'm not sure having a dependent-probe by default is good, and that would be a big change.
>
> If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.
>
> [1]: First proposal:
> http://mails.dpdk.org/archives/dev/2019-September/144166.html
> My arguments:
> http://mails.dpdk.org/archives/dev/2019-September/144564.html
OK so there are two needs:
1/ Better control whitelist/blacklist mode.
We already know that a rework is needed here.
Unfortunately neither you or me had time to work on it,
and others who were interested disappeared.
2/ Associate ports with equivalent properties in applications.
This must be done in applications.
Tweaking the probe order is a hack.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-04 10:24 9% ` Thomas Monjalon
2020-02-04 12:44 4% ` Trahe, Fiona
2020-02-04 12:57 4% ` Kevin Traynor
@ 2020-02-04 14:44 4% ` Aaron Conole
2020-02-04 19:49 7% ` Neil Horman
2 siblings, 1 reply; 200+ results
From: Aaron Conole @ 2020-02-04 14:44 UTC (permalink / raw)
To: Thomas Monjalon
Cc: David Marchand, nhorman, bluca, ktraynor, Ray Kinsella, dev,
Akhil Goyal, Trahe, Fiona, Ferruh Yigit, Ananyev, Konstantin,
Anoob Joseph, Kusztal, ArkadiuszX, Richardson, Bruce, Mcnamara,
John, dodji, Andrew Rybchenko
Thomas Monjalon <thomas@monjalon.net> writes:
> RED FLAG
>
> I don't see a lot of reactions, so I summarize the issue.
> We need action TODAY!
>
> API makes think that rte_cryptodev_info_get() cannot return
> a value >= 3 (RTE_CRYPTO_AEAD_LIST_END in 19.11).
> Current 20.02 returns 3 (RTE_CRYPTO_AEAD_CHACHA20_POLY1305).
> The ABI compatibility contract is broken currently.
>
> There are 3 possible outcomes:
>
> a) Change the API comments and backport to 19.11.1
> The details are discussed between Ferruh and me.
> Either put responsibility on API user (with explicit comment),
> or expose ABI extension allowance with a new API max value.
> In both cases, this is breaking the implicit contract of 19.11.0.
> This option can be chosen only if release and ABI maintainers
> vote for it.
>
> b) Revert Chacha-Poly from 20.02-rc2.
>
> c) Add versioned function rte_cryptodev_info_get_v1911()
> which calls rte_cryptodev_info_get() and filters out
> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 capability.
> So Chacha-Poly capability would be seen and usable only
> if compiling with DPDK 20.02.
>
> I hope it is clear what are the actions for everybody:
> - ABI and release maintainers must say yes or no to the proposal (a)
> - In the meantime, crypto team must send a patch for the proposal (c)
> - If (a) and (c) are not possible at the end of today, I will take (b)
>
> Note: do not say it is too short for (c), as it was possible to work
> on such solution since the issue was exposed on last Wednesday.
While I'm not a maintainer, if I my opinion counts for anything, I'd
choose option c or b. Absolutely NACK to a.
>
> 03/02/2020 22:07, Thomas Monjalon:
>> 03/02/2020 19:55, Ray Kinsella:
>> > On 03/02/2020 17:34, Thomas Monjalon wrote:
>> > > 03/02/2020 18:09, Thomas Monjalon:
>> > >> 03/02/2020 10:30, Ferruh Yigit:
>> > >>> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
>> > >>>> 02/02/2020 14:05, Thomas Monjalon:
>> > >>>>> 31/01/2020 15:16, Trahe, Fiona:
>> > >>>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
>> > >>>>>>> If library give higher value than expected by the application,
>> > >>>>>>> if the application uses this value as array index,
>> > >>>>>>> there can be an access out of bounds.
>> > >>>>>>
>> > >>>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
>> > >>>>>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
>> > >>>>>> sense and I don't see how there can be an API breakage.
>> > >>>>>> So if an application hasn't compiled against the new lib it
>> > >>>>>> will be still using the old value
>> > >>>>>> which will be within bounds. If it's picking up the higher
>> > >>>>>> new value from the lib it must
>> > >>>>>> have been compiled against the lib so shouldn't have problems.
>> > >>>>>
>> > >>>>> You say there is no ABI issue because the application will be re-compiled
>> > >>>>> for the updated library. Indeed, compilation fixes compatibility issues.
>> > >>>>> But this is not relevant for ABI compatibility.
>> > >>>>> ABI compatibility means we can upgrade the library without recompiling
>> > >>>>> the application and it must work.
>> > >>>>> You think it is a false positive because you assume the application
>> > >>>>> "picks" the new value. I think you miss the case where the new value
>> > >>>>> is returned by a function in the upgraded library.
>> > >>>>>
>> > >>>>>> There are also no structs on the API which contain arrays using this
>> > >>>>>> for sizing, so I don't see an opportunity for an appl to have a
>> > >>>>>> mismatch in memory addresses.
>> > >>>>>
>> > >>>>> Let me demonstrate where the API may "use" the new value
>> > >>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
>> > >>>>>
>> > >>>>> Once upon a time a DPDK application counting the number of devices
>> > >>>>> supporting each AEAD algo (in order to find the best supported algo).
>> > >>>>> It is done in an array indexed by algo id:
>> > >>>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
>> > >>>>> The application is compiled with DPDK 19.11,
>> > >>>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
>> > >>>>> So the size of the application array aead_dev_count is 3.
>> > >>>>> This binary is run with DPDK 20.02,
>> > >>>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
>> > >>>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
>> > >>>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
>> > >>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
>> > >>>>> The application uses this value:
>> > >>>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
>> > >>>>> The application is crashing because of out of bound access.
>> > >>>>
>> > >>>> I'd say this is an example of bad written app.
>> > >>>> It probably should check that returned by library value doesn't
>> > >>>> exceed its internal array size.
>> > >>>
>> > >>> +1
>> > >>>
>> > >>> Application should ignore values >= MAX.
>> > >>
>> > >> Of course, blaming the API user is a lot easier than looking at the API.
>> > >> Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
>> > >> as the max value for the application.
>> > >> Value ranges are part of the ABI compatibility contract.
>> > >> It seems you expect the application developer to be aware that
>> > >> DPDK could return a higher value, so the application should
>> > >> check every enum values after calling an API. CRAZY.
>> > >>
>> > >> When we decide to announce an ABI compatibility and do some marketing,
>> > >> everyone is OK. But when we need to really make our ABI compatible,
>> > >> I see little or no effort. DISAPPOINTING.
>> > >>
>> > >>> Do you suggest we don't extend any enum or define between ABI breakage releases
>> > >>> to be sure bad written applications not affected?
>> > >>
>> > >> I suggest we must consider not breaking any assumption made on the API.
>> > >> Here we are breaking the enum range because nothing mentions _LIST_END
>> > >> is not really the absolute end of the enum.
>> > >> The solution is to make the change below in 20.02 + backport in 19.11.1:
>> > >
>> > > Thinking twice, merging such change before 20.11 is breaking the
>> > > ABI assumption based on the API 19.11.0.
>> > > I ask the release maintainers (Luca, Kevin, David and me) and
>> > > the ABI maintainers (Neil and Ray) to vote for a or b solution:
>> > > a) add comment and LIST_MAX as below in 20.02 + 19.11.1
>> >
>> > That would still be an ABI breakage though right.
>> >
>> > > b) wait 20.11 and revert Chacha-Poly from 20.02
>> >
>> > Thanks for analysis above Fiona, Ferruh and all.
>> >
>> > That is a nasty one alright - there is no "good" answer here.
>> > I agree with Ferruh's sentiments overall, we should rethink this API for 20.11.
>> > Could do without an enumeration?
>> >
>> > There a c) though right.
>> > We could work around the issue by api versioning rte_cryptodev_info_get() and friends.
>> > So they only support/acknowledge the existence of Chacha-Poly for
>> > applications build against > 20.02.
>>
>> I agree there is a c) as I proposed in another email:
>> http://mails.dpdk.org/archives/dev/2020-February/156919.html
>> "
>> In this case, the proper solution is to implement
>> rte_cryptodev_info_get_v1911() so it filters out
>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 capability.
>> With this solution, an application compiled with DPDK 19.11 will keep
>> seeing the same range as before, while a 20.02 application could
>> see and use ChachaPoly.
>> "
>>
>> > It would be painful I know.
>>
>> Not so painful in my opinion.
>> Just need to call rte_cryptodev_info_get() from
>> rte_cryptodev_info_get_v1911() and filter the value
>> in the 19.11 range: [0..AES_GCM].
>>
>> > It would also mean that Chacha-Poly would only be available to
>> > those building against >= 20.02.
>>
>> Yes exactly.
>>
>> The addition of comments and LIST_MAX like below are still valid
>> to avoid versioning after 20.11.
>>
>> > >> - _LIST_END
>> > >> + _LIST_END, /* an ABI-compatible version may increase this value */
>> > >> + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
>> > >> };
>> > >>
>> > >> Then *_LIST_END values could be ignored by libabigail with such a change.
>>
>> In order to avoid ABI check complaining, the best is to completely
>> remove LIST_END in DPDK 20.11.
>>
>>
>> > >> If such a patch is not done by tomorrow, I will have to revert
>> > >> Chacha-Poly commits before 20.02-rc2, because
>> > >>
>> > >> 1/ LIST_END, without any comment, means "size of range"
>> > >> 2/ we do not blame users for undocumented ABI changes
>> > >> 3/ we respect the ABI compatibility contract
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-04 10:24 9% ` Thomas Monjalon
2020-02-04 12:44 4% ` Trahe, Fiona
@ 2020-02-04 12:57 4% ` Kevin Traynor
2020-02-04 14:44 4% ` Aaron Conole
2 siblings, 0 replies; 200+ results
From: Kevin Traynor @ 2020-02-04 12:57 UTC (permalink / raw)
To: Thomas Monjalon, David Marchand, nhorman, bluca, Ray Kinsella,
dev, Akhil Goyal, Trahe, Fiona, Ferruh Yigit
Cc: Ananyev, Konstantin, Anoob Joseph, Kusztal, ArkadiuszX,
Richardson, Bruce, Mcnamara, John, dodji, Andrew Rybchenko,
aconole
On 04/02/2020 10:24, Thomas Monjalon wrote:
> RED FLAG
>
> I don't see a lot of reactions, so I summarize the issue.
> We need action TODAY!
>
> API makes think that rte_cryptodev_info_get() cannot return
> a value >= 3 (RTE_CRYPTO_AEAD_LIST_END in 19.11).
> Current 20.02 returns 3 (RTE_CRYPTO_AEAD_CHACHA20_POLY1305).
> The ABI compatibility contract is broken currently.
>
> There are 3 possible outcomes:
>
> a) Change the API comments and backport to 19.11.1
> The details are discussed between Ferruh and me.
> Either put responsibility on API user (with explicit comment),
> or expose ABI extension allowance with a new API max value.
> In both cases, this is breaking the implicit contract of 19.11.0.
> This option can be chosen only if release and ABI maintainers
> vote for it.
>
> b) Revert Chacha-Poly from 20.02-rc2.
>
> c) Add versioned function rte_cryptodev_info_get_v1911()
> which calls rte_cryptodev_info_get() and filters out
> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 capability.
> So Chacha-Poly capability would be seen and usable only
> if compiling with DPDK 20.02.
>
Maybe a separate version of rte_cryptodev_get_aead_algo_enum() also
needed to handle chacha string differently.
> I hope it is clear what are the actions for everybody:
> - ABI and release maintainers must say yes or no to the proposal (a)
My 2c for a) is No.
> - In the meantime, crypto team must send a patch for the proposal (c)
> - If (a) and (c) are not possible at the end of today, I will take (b)
>
> Note: do not say it is too short for (c), as it was possible to work
> on such solution since the issue was exposed on last Wednesday.
>
Could it be reverted today if necessary and re-added later in the
release cycle? It seems like something modular that should not
invalidate earlier testing.
>
> 03/02/2020 22:07, Thomas Monjalon:
>> 03/02/2020 19:55, Ray Kinsella:
>>> On 03/02/2020 17:34, Thomas Monjalon wrote:
>>>> 03/02/2020 18:09, Thomas Monjalon:
>>>>> 03/02/2020 10:30, Ferruh Yigit:
>>>>>> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
>>>>>>> 02/02/2020 14:05, Thomas Monjalon:
>>>>>>>> 31/01/2020 15:16, Trahe, Fiona:
>>>>>>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
>>>>>>>>>> If library give higher value than expected by the application,
>>>>>>>>>> if the application uses this value as array index,
>>>>>>>>>> there can be an access out of bounds.
>>>>>>>>>
>>>>>>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
>>>>>>>>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
>>>>>>>>> sense and I don't see how there can be an API breakage.
>>>>>>>>> So if an application hasn't compiled against the new lib it will be still using the old value
>>>>>>>>> which will be within bounds. If it's picking up the higher new value from the lib it must
>>>>>>>>> have been compiled against the lib so shouldn't have problems.
>>>>>>>>
>>>>>>>> You say there is no ABI issue because the application will be re-compiled
>>>>>>>> for the updated library. Indeed, compilation fixes compatibility issues.
>>>>>>>> But this is not relevant for ABI compatibility.
>>>>>>>> ABI compatibility means we can upgrade the library without recompiling
>>>>>>>> the application and it must work.
>>>>>>>> You think it is a false positive because you assume the application
>>>>>>>> "picks" the new value. I think you miss the case where the new value
>>>>>>>> is returned by a function in the upgraded library.
>>>>>>>>
>>>>>>>>> There are also no structs on the API which contain arrays using this
>>>>>>>>> for sizing, so I don't see an opportunity for an appl to have a
>>>>>>>>> mismatch in memory addresses.
>>>>>>>>
>>>>>>>> Let me demonstrate where the API may "use" the new value
>>>>>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
>>>>>>>>
>>>>>>>> Once upon a time a DPDK application counting the number of devices
>>>>>>>> supporting each AEAD algo (in order to find the best supported algo).
>>>>>>>> It is done in an array indexed by algo id:
>>>>>>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
>>>>>>>> The application is compiled with DPDK 19.11,
>>>>>>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
>>>>>>>> So the size of the application array aead_dev_count is 3.
>>>>>>>> This binary is run with DPDK 20.02,
>>>>>>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
>>>>>>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
>>>>>>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
>>>>>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
>>>>>>>> The application uses this value:
>>>>>>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
>>>>>>>> The application is crashing because of out of bound access.
>>>>>>>
>>>>>>> I'd say this is an example of bad written app.
>>>>>>> It probably should check that returned by library value doesn't
>>>>>>> exceed its internal array size.
>>>>>>
>>>>>> +1
>>>>>>
>>>>>> Application should ignore values >= MAX.
>>>>>
>>>>> Of course, blaming the API user is a lot easier than looking at the API.
>>>>> Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
>>>>> as the max value for the application.
>>>>> Value ranges are part of the ABI compatibility contract.
>>>>> It seems you expect the application developer to be aware that
>>>>> DPDK could return a higher value, so the application should
>>>>> check every enum values after calling an API. CRAZY.
>>>>>
>>>>> When we decide to announce an ABI compatibility and do some marketing,
>>>>> everyone is OK. But when we need to really make our ABI compatible,
>>>>> I see little or no effort. DISAPPOINTING.
>>>>>
>>>>>> Do you suggest we don't extend any enum or define between ABI breakage releases
>>>>>> to be sure bad written applications not affected?
>>>>>
>>>>> I suggest we must consider not breaking any assumption made on the API.
>>>>> Here we are breaking the enum range because nothing mentions _LIST_END
>>>>> is not really the absolute end of the enum.
>>>>> The solution is to make the change below in 20.02 + backport in 19.11.1:
>>>>
>>>> Thinking twice, merging such change before 20.11 is breaking the
>>>> ABI assumption based on the API 19.11.0.
>>>> I ask the release maintainers (Luca, Kevin, David and me) and
>>>> the ABI maintainers (Neil and Ray) to vote for a or b solution:
>>>> a) add comment and LIST_MAX as below in 20.02 + 19.11.1
>>>
>>> That would still be an ABI breakage though right.
>>>
>>>> b) wait 20.11 and revert Chacha-Poly from 20.02
>>>
>>> Thanks for analysis above Fiona, Ferruh and all.
>>>
>>> That is a nasty one alright - there is no "good" answer here.
>>> I agree with Ferruh's sentiments overall, we should rethink this API for 20.11.
>>> Could do without an enumeration?
>>>
>>> There a c) though right.
>>> We could work around the issue by api versioning rte_cryptodev_info_get() and friends.
>>> So they only support/acknowledge the existence of Chacha-Poly for applications build against > 20.02.
>>
>> I agree there is a c) as I proposed in another email:
>> http://mails.dpdk.org/archives/dev/2020-February/156919.html
>> "
>> In this case, the proper solution is to implement
>> rte_cryptodev_info_get_v1911() so it filters out
>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 capability.
>> With this solution, an application compiled with DPDK 19.11 will keep
>> seeing the same range as before, while a 20.02 application could
>> see and use ChachaPoly.
>> "
>>
>>> It would be painful I know.
>>
>> Not so painful in my opinion.
>> Just need to call rte_cryptodev_info_get() from
>> rte_cryptodev_info_get_v1911() and filter the value
>> in the 19.11 range: [0..AES_GCM].
>>
>>> It would also mean that Chacha-Poly would only be available to
>>> those building against >= 20.02.
>>
>> Yes exactly.
>>
>> The addition of comments and LIST_MAX like below are still valid
>> to avoid versioning after 20.11.
>>
>>>>> - _LIST_END
>>>>> + _LIST_END, /* an ABI-compatible version may increase this value */
>>>>> + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
>>>>> };
>>>>>
>>>>> Then *_LIST_END values could be ignored by libabigail with such a change.
>>
>> In order to avoid ABI check complaining, the best is to completely
>> remove LIST_END in DPDK 20.11.
>>
>>
>>>>> If such a patch is not done by tomorrow, I will have to revert
>>>>> Chacha-Poly commits before 20.02-rc2, because
>>>>>
>>>>> 1/ LIST_END, without any comment, means "size of range"
>>>>> 2/ we do not blame users for undocumented ABI changes
>>>>> 3/ we respect the ABI compatibility contract
>
>
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-04 10:24 9% ` Thomas Monjalon
@ 2020-02-04 12:44 4% ` Trahe, Fiona
2020-02-04 15:52 8% ` Trahe, Fiona
2020-02-04 12:57 4% ` Kevin Traynor
2020-02-04 14:44 4% ` Aaron Conole
2 siblings, 1 reply; 200+ results
From: Trahe, Fiona @ 2020-02-04 12:44 UTC (permalink / raw)
To: Thomas Monjalon, David Marchand, nhorman, bluca, ktraynor,
Ray Kinsella, dev, Akhil Goyal, Yigit, Ferruh
Cc: Ananyev, Konstantin, dev, Anoob Joseph, Kusztal, ArkadiuszX,
Richardson, Bruce, Mcnamara, John, dodji, Andrew Rybchenko,
aconole, Trahe, Fiona
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Tuesday, February 4, 2020 10:24 AM
> To: David Marchand <david.marchand@redhat.com>; nhorman@tuxdriver.com; bluca@debian.org;
> ktraynor@redhat.com; Ray Kinsella <mdr@ashroe.eu>; dev@dpdk.org; Akhil Goyal
> <akhil.goyal@nxp.com>; Trahe, Fiona <fiona.trahe@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org; Anoob Joseph
> <anoobj@marvell.com>; Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Mcnamara, John <john.mcnamara@intel.com>; dodji@seketeli.net;
> Andrew Rybchenko <arybchenko@solarflare.com>; aconole@redhat.com
> Subject: Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
>
> RED FLAG
>
> I don't see a lot of reactions, so I summarize the issue.
> We need action TODAY!
>
> API makes think that rte_cryptodev_info_get() cannot return
> a value >= 3 (RTE_CRYPTO_AEAD_LIST_END in 19.11).
> Current 20.02 returns 3 (RTE_CRYPTO_AEAD_CHACHA20_POLY1305).
> The ABI compatibility contract is broken currently.
>
> There are 3 possible outcomes:
>
> a) Change the API comments and backport to 19.11.1
> The details are discussed between Ferruh and me.
> Either put responsibility on API user (with explicit comment),
> or expose ABI extension allowance with a new API max value.
> In both cases, this is breaking the implicit contract of 19.11.0.
> This option can be chosen only if release and ABI maintainers
> vote for it.
>
> b) Revert Chacha-Poly from 20.02-rc2.
>
> c) Add versioned function rte_cryptodev_info_get_v1911()
> which calls rte_cryptodev_info_get() and filters out
> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 capability.
> So Chacha-Poly capability would be seen and usable only
> if compiling with DPDK 20.02.
>
> I hope it is clear what are the actions for everybody:
> - ABI and release maintainers must say yes or no to the proposal (a)
> - In the meantime, crypto team must send a patch for the proposal (c)
> - If (a) and (c) are not possible at the end of today, I will take (b)
>
> Note: do not say it is too short for (c), as it was possible to work
> on such solution since the issue was exposed on last Wednesday.
>
[Fiona] Thanks for your understanding and help with our scheduling Thomas.
We are working on a patch, when it is ready we will send it.
If it's not ready by end of your today, of course, go ahead with (b) and
we will work towards 20.05.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v7] eal: add manual probing option
2020-02-04 11:07 0% ` Thomas Monjalon
@ 2020-02-04 12:43 0% ` Gaetan Rivet
2020-02-04 15:06 0% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Gaetan Rivet @ 2020-02-04 12:43 UTC (permalink / raw)
To: Thomas Monjalon, Pavan Nikhilesh Bhagavatula, David Marchand
Cc: dev, Vamsi Krishna Attunuru, Jerin Jacob Kollanukkaran
On 04/02/2020 12:07, Thomas Monjalon wrote:
> 04/02/2020 11:03, Gaetan Rivet:
>> On 03/02/2020 23:21, Thomas Monjalon wrote:
>>> 03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
>>>> @David Marchand @thomas@monjalon.net
>>>>
>>>> Ping?
>>>>
>>>> Are there any more changes required for this patch? It's been in queue since last October.
>>>
>>> Sorry we have not decided whether it is a good idea or not.
>>>
>>> All changes related to probing are very sensitive,
>>> and we know a big refactoring would be better than stacking
>>> more and more options and corner cases.
>>>
>>> As we are busy with ABI stability stuff, we did not allocate
>>> enough time to properly think about this feature.
>>> Please accept our apologies, and let's consider it as
>>> a high priority for 20.05 cycle.
>>>
>>
>> Hello Thomas,
>>
>> This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.
>>
>> The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.
>
>
> Yes, life is full of bad decisions and consequences.
Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.
>
> I still think there is a risk in adding new user expectations,
> and maintaining some code to workaround unknown issues.
>
> The real question here is to know why this patch?
> Is it to workaround a broken driver?
> Or to workaround a broken design in EAL and bus drivers?
>
>
Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).
I'm not sure having a dependent-probe by default is good, and that would be a big change.
If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.
[1]: First proposal:
http://mails.dpdk.org/archives/dev/2019-September/144166.html
My arguments:
http://mails.dpdk.org/archives/dev/2019-September/144564.html
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-03 12:53 4% ` Ferruh Yigit
@ 2020-02-04 12:02 9% ` Neil Horman
2020-02-05 10:04 4% ` Luca Boccassi
0 siblings, 1 reply; 200+ results
From: Neil Horman @ 2020-02-04 12:02 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Cristian Dumitrescu, Eelco Chaudron, dev, Thomas Monjalon,
Luca Boccassi, David Marchand, Bruce Richardson, Ian Stokes
On Mon, Feb 03, 2020 at 12:53:28PM +0000, Ferruh Yigit wrote:
> On 2/2/2020 6:53 PM, Neil Horman wrote:
> > On Wed, Jan 29, 2020 at 12:29:53PM +0000, Ferruh Yigit wrote:
> >> Duplicated the existing symbol and versioned one as experimental and
> >> other as stable.
> >>
> >> Created VERSION_SYMBOL_EXPERIMENTAL helper macro.
> >>
> >> Updated the 'check-experimental-syms.sh' buildtool, which was
> >> complaining that the symbol is in EXPERIMENTAL tag in .map file but it
> >> is not in the .experimental section (__rte_experimental tag is missing).
> >> Updated tool in a way it won't complain if the symbol in the
> >> EXPERIMENTAL tag duplicated in some other block in .map file (versioned)
> >>
> >> Updated meson build system to allow the versioning,
> >> 'use_function_versioning = true', not sure why it was disabled by
> >> default.
> >>
> >> Fixes: 30512af820fe ("meter: remove experimental flag from RFC4115 trTCM API")
> >>
> > I'm not sure I understand the purpose here.
> >
> > I think I understand what you are trying to do. I believe that you are
> > trying to move the experimental symbols in librte_meter to be part of
> > the official ABI, while still maintaining backward compatibility with
> > applications built against the experimental symbol versions, correct?
>
> Yes, exactly.
>
Ok, so we're on the same page, good.
> >
> > I can understand the desire, and I'm not exactly opposed to providing a
> > mechanism for that, but it seems somewhat complex, and to be honest,
> > part of the drawback to using experimental symbols as an application
> > developer is understanding that you may need to rebuild when those
> > symbols change (even if the change is the symbol removal and replacement
> > with an identical one that has a versioned tag).
> >
> > I think the right answer for people encountering this condition is to
> > just rebuild. Otherwise, we are creating an environment in which we are
> > implicitly communicating to users that we are maintaining a modicum of
> > stability in experimental symobls.
>
> This will mean, even the the API was perfect it will force its users to
> recompile (and re-package, re-deploy etc..) at least once, this is feeling like
Yes, thats correct.
> we are still breaking user applications easily.
we reserve that right, based on the fact that we marked these symbols as
experimental. Thats the deal we provide. Stable (non experimental) API calls
are guaranteed not to change for the ABI lifetime, and its our responsibility to
maintain that. However, experimental API's are marked as such because we can
change them (even if that change is to remove it and replace it with an
identical stable version), and application developers assume that risk by using
them. I understand the removal of an experimental symbol, replacing it with a
stable identical version is a trivial case of that change, and if we can ease
the burden, thats fine, but I don't think we need to go out of our way, or incur
any additional burden to ease that transition. We created the experimental
symbol marking to ease the burden of this sort of development on us, we should
be willing to use it.
> And this may create a stronger motivation by applications not to use
> experimental APIs, I can't decide if this is a good thing or bad thing.
>
Yeah, for stable, long term support applications, that makes sense, they
shouldn't be using experimental API's when they need the guarantee of support,
or they should be understanding of their responsibility to rebuild, when we
change the API's in both complex and trivial cases like this.
>
> The issue is amplified in the LTS,
> If we remove experimental tag in LTS, will be breaking the apps using that
> experimental API, just to mature the API.
You shouldn't be making this sort of transition in LTS. Just leave the
experimental symbol as is there, and commit the transition for the next LTS
release.
> If we keep it as experimental, the API will be mature in main repo, but the LTS
> has to keep exact same API as experimental up to two years.
Yes, thats the meaning of LTS. Things stay stable for its lifetime. And if the
only thing you are doing is replacing the experimental ABI version with a stable
version (leaving the details of the ABI unchanged), then its a moot point. The
only difference will be that the LTS release will have symbols marked as
experimental.
>
> But if we can do the versioning in the master, LTS can backport it and can have
> mature version of that API in LTS without breaking the existing users.
>
But why bother? The only thing you've changed is the version tagging. Its ok
to leave that alone in LTS, you just cant change it.
Thats part of the burden of an LTS release, it will have some drift from the
upstream head, because you have to keep things stable. So you stabilize the
upstream ABI version for this API and just never backport it to the current LTS
release.
> >
> > A few more nits in line
> >
> >> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> >> ---
> >> Cc: Neil Horman <nhorman@tuxdriver.com>
> >> Cc: Thomas Monjalon <thomas@monjalon.net>
> >> Cc: Luca Boccassi <bluca@debian.org>
> >> Cc: David Marchand <david.marchand@redhat.com>
> >> Cc: Bruce Richardson <bruce.richardson@intel.com>
> >> Cc: Ian Stokes <ian.stokes@intel.com>
> >> Cc: Eelco Chaudron <echaudro@redhat.com>
> >>
> >> I didn't like the idea of duplicating the symbol but not able to find to
> >> alias it without duplicating, if there is a way please share.
> >>
> >> Also not tested with old binaries, only verified the symbols in new
> >> binaries.
> >> ---
> >> buildtools/check-experimental-syms.sh | 3 +-
> >> .../common/include/rte_function_versioning.h | 4 ++
> >> lib/librte_meter/rte_meter.c | 59 ++++++++++++++++++-
> >> lib/librte_meter/rte_meter_version.map | 8 +++
> >> lib/meson.build | 2 +-
> >> 5 files changed, 71 insertions(+), 5 deletions(-)
> >>
> >> diff --git a/buildtools/check-experimental-syms.sh b/buildtools/check-experimental-syms.sh
> >> index f3603e5ba..35c4bf006 100755
> >> --- a/buildtools/check-experimental-syms.sh
> >> +++ b/buildtools/check-experimental-syms.sh
> >> @@ -26,7 +26,8 @@ ret=0
> >> for SYM in `$LIST_SYMBOL -S EXPERIMENTAL $MAPFILE |cut -d ' ' -f 3`
> >> do
> >> if grep -q "\.text.*[[:space:]]$SYM$" $DUMPFILE &&
> >> - ! grep -q "\.text\.experimental.*[[:space:]]$SYM$" $DUMPFILE
> >> + ! grep -q "\.text\.experimental.*[[:space:]]$SYM$" $DUMPFILE &&
> >> + $LIST_SYMBOL -s $SYM $MAPFILE | grep -q EXPERIMENTAL
> >> then
> >> cat >&2 <<- END_OF_MESSAGE
> >> $SYM is not flagged as experimental
> >> diff --git a/lib/librte_eal/common/include/rte_function_versioning.h b/lib/librte_eal/common/include/rte_function_versioning.h
> >> index c924351d5..ab102b06e 100644
> >> --- a/lib/librte_eal/common/include/rte_function_versioning.h
> >> +++ b/lib/librte_eal/common/include/rte_function_versioning.h
> >> @@ -46,6 +46,9 @@
> >> */
> >> #define VERSION_SYMBOL(b, e, n) __asm__(".symver " RTE_STR(b) RTE_STR(e) ", " RTE_STR(b) "@DPDK_" RTE_STR(n))
> >>
> >> +
> >> +#define VERSION_SYMBOL_EXPERIMENTAL(b, e) __asm__(".symver " RTE_STR(b) RTE_STR(e) ", " RTE_STR(b) "@EXPERIMENTAL")
> >> +
> > I don't know that you want to make a explicit clone of this macro.
> > instead what you might want to do is something like:
> >
> > #define __VERSION_SYMBOL(b, e, n, t) __asm__(".symver " RTE_STR(b) RTE_STR(e) ", " RTE_STR(b) "@" RTE_STR(t) "_" RTE_STR(n))
> >
> > #define VERSION_SYMBOL(b, e, n) __VERSION_SYMBOL(b, e, n, DPDK)
> >
> > Thats not exactly right, but you get the idea, then you can emulate
> > VERSION_SYMBOL_EXPERIMENTAL with __VERSION_SYMBOL(b, e, n, EXPERIMENTAL)
>
> +1
>
> >
> >
> >> /*
> >> * BIND_DEFAULT_SYMBOL
> >> * Creates a symbol version entry instructing the linker to bind references to
> >> @@ -79,6 +82,7 @@
> >> * No symbol versioning in use
> >> */
> >> #define VERSION_SYMBOL(b, e, n)
> >> +#define VERSION_SYMBOL_EXPERIMENTAL(b, e)
> >> #define __vsym
> >> #define BIND_DEFAULT_SYMBOL(b, e, n)
> >> #define MAP_STATIC_SYMBOL(f, p) f __attribute__((alias(RTE_STR(p))))
> >> diff --git a/lib/librte_meter/rte_meter.c b/lib/librte_meter/rte_meter.c
> >> index da01429a8..5244537fa 100644
> >> --- a/lib/librte_meter/rte_meter.c
> >> +++ b/lib/librte_meter/rte_meter.c
> >> @@ -9,6 +9,7 @@
> >> #include <rte_common.h>
> >> #include <rte_log.h>
> >> #include <rte_cycles.h>
> >> +#include <rte_function_versioning.h>
> >>
> >> #include "rte_meter.h"
> >>
> >> @@ -119,8 +120,8 @@ rte_meter_trtcm_config(struct rte_meter_trtcm *m,
> >> return 0;
> >> }
> >>
> >> -int
> >> -rte_meter_trtcm_rfc4115_profile_config(
> >> +static int
> >> +rte_meter_trtcm_rfc4115_profile_config_(
> >> struct rte_meter_trtcm_rfc4115_profile *p,
> >> struct rte_meter_trtcm_rfc4115_params *params)
> >> {
> >> @@ -145,7 +146,35 @@ rte_meter_trtcm_rfc4115_profile_config(
> >> }
> >>
> >> int
> >> -rte_meter_trtcm_rfc4115_config(
> >> +rte_meter_trtcm_rfc4115_profile_config_s(
> >> + struct rte_meter_trtcm_rfc4115_profile *p,
> >> + struct rte_meter_trtcm_rfc4115_params *params);
> >> +int
> >> +rte_meter_trtcm_rfc4115_profile_config_s(
> >> + struct rte_meter_trtcm_rfc4115_profile *p,
> >> + struct rte_meter_trtcm_rfc4115_params *params)
> >> +{
> >> + return rte_meter_trtcm_rfc4115_profile_config_(p, params);
> >> +}
> >> +BIND_DEFAULT_SYMBOL(rte_meter_trtcm_rfc4115_profile_config, _s, 20.0.1);
> >> +MAP_STATIC_SYMBOL(int rte_meter_trtcm_rfc4115_profile_config(struct rte_meter_trtcm_rfc4115_profile *p,
> >> + struct rte_meter_trtcm_rfc4115_params *params), rte_meter_trtcm_rfc4115_profile_config_s);
> >> +
> >> +int
> >> +rte_meter_trtcm_rfc4115_profile_config_e(
> >> + struct rte_meter_trtcm_rfc4115_profile *p,
> >> + struct rte_meter_trtcm_rfc4115_params *params);
> >> +int
> >> +rte_meter_trtcm_rfc4115_profile_config_e(
> >> + struct rte_meter_trtcm_rfc4115_profile *p,
> >> + struct rte_meter_trtcm_rfc4115_params *params)
> >> +{
> >> + return rte_meter_trtcm_rfc4115_profile_config_(p, params);
> >> +}
> >> +VERSION_SYMBOL_EXPERIMENTAL(rte_meter_trtcm_rfc4115_profile_config, _e);
> >> +
> >> +static int
> >> +rte_meter_trtcm_rfc4115_config_(
> >> struct rte_meter_trtcm_rfc4115 *m,
> >> struct rte_meter_trtcm_rfc4115_profile *p)
> >> {
> >> @@ -160,3 +189,27 @@ rte_meter_trtcm_rfc4115_config(
> >>
> >> return 0;
> >> }
> >> +
> >> +int
> >> +rte_meter_trtcm_rfc4115_config_s(struct rte_meter_trtcm_rfc4115 *m,
> >> + struct rte_meter_trtcm_rfc4115_profile *p);
> >> +int
> >> +rte_meter_trtcm_rfc4115_config_s(struct rte_meter_trtcm_rfc4115 *m,
> >> + struct rte_meter_trtcm_rfc4115_profile *p)
> >> +{
> >> + return rte_meter_trtcm_rfc4115_config_(m, p);
> >> +}
> >> +BIND_DEFAULT_SYMBOL(rte_meter_trtcm_rfc4115_config, _s, 20.0.1);
> >> +MAP_STATIC_SYMBOL(int rte_meter_trtcm_rfc4115_config(struct rte_meter_trtcm_rfc4115 *m,
> >> + struct rte_meter_trtcm_rfc4115_profile *p), rte_meter_trtcm_rfc4115_config_s);
> >> +
> >> +int
> >> +rte_meter_trtcm_rfc4115_config_e(struct rte_meter_trtcm_rfc4115 *m,
> >> + struct rte_meter_trtcm_rfc4115_profile *p);
> >> +int
> >> +rte_meter_trtcm_rfc4115_config_e(struct rte_meter_trtcm_rfc4115 *m,
> >> + struct rte_meter_trtcm_rfc4115_profile *p)
> >> +{
> >> + return rte_meter_trtcm_rfc4115_config_(m, p);
> >> +}
> >> +VERSION_SYMBOL_EXPERIMENTAL(rte_meter_trtcm_rfc4115_config, _e);
> >> diff --git a/lib/librte_meter/rte_meter_version.map b/lib/librte_meter/rte_meter_version.map
> >> index fc12cc0bf..b4b043174 100644
> >> --- a/lib/librte_meter/rte_meter_version.map
> >> +++ b/lib/librte_meter/rte_meter_version.map
> >> @@ -20,4 +20,12 @@ DPDK_20.0.1 {
> >> rte_meter_trtcm_rfc4115_color_blind_check;
> >> rte_meter_trtcm_rfc4115_config;
> >> rte_meter_trtcm_rfc4115_profile_config;
> >> +
> >> } DPDK_20.0;
> >> +
> >> +EXPERIMENTAL {
> >> + global:
> >> +
> >> + rte_meter_trtcm_rfc4115_config;
> >> + rte_meter_trtcm_rfc4115_profile_config;
> >> +};
> >> diff --git a/lib/meson.build b/lib/meson.build
> >> index 0af3efab2..553964831 100644
> >> --- a/lib/meson.build
> >> +++ b/lib/meson.build
> >> @@ -48,7 +48,7 @@ foreach l:libraries
> >> reason = '<unknown reason>' # set if build == false to explain why
> >> name = l
> >> allow_experimental_apis = false
> >> - use_function_versioning = false
> >> + use_function_versioning = true
> >> sources = []
> >> headers = []
> >> includes = []
> >> --
> >> 2.24.1
> >>
> >>
>
>
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-04 10:32 4% ` Akhil Goyal
@ 2020-02-04 11:35 4% ` Bruce Richardson
2020-02-04 22:10 7% ` Neil Horman
1 sibling, 0 replies; 200+ results
From: Bruce Richardson @ 2020-02-04 11:35 UTC (permalink / raw)
To: Akhil Goyal
Cc: Thomas Monjalon, Ferruh Yigit, Ananyev, Konstantin, Trahe, Fiona,
dev, David Marchand, Anoob Joseph, Kusztal, ArkadiuszX, nhorman,
Mcnamara, John, dodji, Andrew Rybchenko, aconole, bluca,
ktraynor
On Tue, Feb 04, 2020 at 10:32:01AM +0000, Akhil Goyal wrote:
>
> >
> > 04/02/2020 11:16, Akhil Goyal:
> > > Hi,
> > > > On 2/3/2020 5:09 PM, Thomas Monjalon wrote:
> > > > > 03/02/2020 10:30, Ferruh Yigit:
> > > > >> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
> > > > >>> 02/02/2020 14:05, Thomas Monjalon:
> > > > >>>> 31/01/2020 15:16, Trahe, Fiona:
> > > > >>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> > > > >>>>>> 30/01/2020 17:09, Ferruh Yigit:
> > > > >>>>>>> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
> > > > >>>>>>>>
> > > > >>>>>>>> I believe these enums will be used only in case of ASYM case which
> > is
> > > > experimental.
> > > > >>>>>>>
> > > > >>>>>>> Independent from being experiment and not, this shouldn't be a
> > > > problem, I think
> > > > >>>>>>> this is a false positive.
> > > > >>>>>>>
> > > > >>>>>>> The ABI break can happen when a struct has been shared between
> > the
> > > > application
> > > > >>>>>>> and the library (DPDK) and the layout of that memory know
> > differently
> > > > by
> > > > >>>>>>> application and the library.
> > > > >>>>>>>
> > > > >>>>>>> Here in all cases, there is no layout/size change.
> > > > >>>>>>>
> > > > >>>>>>> As to the value changes of the enums, since application compiled
> > with
> > > > old DPDK,
> > > > >>>>>>> it will know only up to '6', 7 and more means invalid to the
> > application.
> > > > So it
> > > > >>>>>>> won't send these values also it should ignore these values from
> > library.
> > > > Only
> > > > >>>>>>> consequence is old application won't able to use new features
> > those
> > > > new enums
> > > > >>>>>>> provide but that is expected/normal.
> > > > >>>>>>
> > > > >>>>>> If library give higher value than expected by the application,
> > > > >>>>>> if the application uses this value as array index,
> > > > >>>>>> there can be an access out of bounds.
> > > > >>>>>
> > > > >>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a
> > > > problem.
> > > > >>>>> But for the same issue with sym crypto below, I believe Ferruh's
> > > > explanation makes
> > > > >>>>> sense and I don't see how there can be an API breakage.
> > > > >>>>> So if an application hasn't compiled against the new lib it will be still
> > using
> > > > the old value
> > > > >>>>> which will be within bounds. If it's picking up the higher new value
> > from
> > > > the lib it must
> > > > >>>>> have been compiled against the lib so shouldn't have problems.
> > > > >>>>
> > > > >>>> You say there is no ABI issue because the application will be re-
> > compiled
> > > > >>>> for the updated library. Indeed, compilation fixes compatibility issues.
> > > > >>>> But this is not relevant for ABI compatibility.
> > > > >>>> ABI compatibility means we can upgrade the library without
> > recompiling
> > > > >>>> the application and it must work.
> > > > >>>> You think it is a false positive because you assume the application
> > > > >>>> "picks" the new value. I think you miss the case where the new value
> > > > >>>> is returned by a function in the upgraded library.
> > > > >>>>
> > > > >>>>> There are also no structs on the API which contain arrays using this
> > > > >>>>> for sizing, so I don't see an opportunity for an appl to have a
> > > > >>>>> mismatch in memory addresses.
> > > > >>>>
> > > > >>>> Let me demonstrate where the API may "use" the new value
> > > > >>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the
> > > > application.
> > > > >>>>
> > > > >>>> Once upon a time a DPDK application counting the number of devices
> > > > >>>> supporting each AEAD algo (in order to find the best supported algo).
> > > > >>>> It is done in an array indexed by algo id:
> > > > >>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
> > > > >>>> The application is compiled with DPDK 19.11,
> > > > >>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
> > > > >>>> So the size of the application array aead_dev_count is 3.
> > > > >>>> This binary is run with DPDK 20.02,
> > > > >>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
> > > > >>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
> > > > >>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
> > > > >>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
> > > > >>>> The application uses this value:
> > > > >>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
> > > > >>>> The application is crashing because of out of bound access.
> > > > >>>
> > > > >>> I'd say this is an example of bad written app.
> > > > >>> It probably should check that returned by library value doesn't
> > > > >>> exceed its internal array size.
> > > > >>
> > > > >> +1
> > > > >>
> > > > >> Application should ignore values >= MAX.
> > > > >
> > > > > Of course, blaming the API user is a lot easier than looking at the API.
> > > > > Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
> > > > > as the max value for the application.
> > > > > Value ranges are part of the ABI compatibility contract.
> > > > > It seems you expect the application developer to be aware that
> > > > > DPDK could return a higher value, so the application should
> > > > > check every enum values after calling an API. CRAZY.
> > > > >
> > > > > When we decide to announce an ABI compatibility and do some marketing,
> > > > > everyone is OK. But when we need to really make our ABI compatible,
> > > > > I see little or no effort. DISAPPOINTING.
> > > >
> > > > This is not to blame the user or to do less work, this is more sane approach
> > > > that library provides the _END/_MAX value and application uses it as valid
> > range
> > > > check.
> > > >
> > > > >
> > > > >> Do you suggest we don't extend any enum or define between ABI
> > breakage
> > > > releases
> > > > >> to be sure bad written applications not affected?
> > > > >
> > > > > I suggest we must consider not breaking any assumption made on the API.
> > > > > Here we are breaking the enum range because nothing mentions
> > _LIST_END
> > > > > is not really the absolute end of the enum.
> > > > > The solution is to make the change below in 20.02 + backport in 19.11.1:
> > > > >
> > > > > - _LIST_END
> > > > > + _LIST_END, /* an ABI-compatible version may increase this value */
> > > > > + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
> > > > > };
> > > > >
> > > >
> > > > What is the point of "_LIST_MAX" here?
> > > >
> > > > Application should know the "_LIST_END" of when it has been compiled for
> > the
> > > > valid range check. Next time it is compiled "_LIST_END" may be different
> > value
> > > > but same logic applies.
> > > >
> > > > When "_LIST_END" is missing, application can't protect itself, in that case
> > > > library should send only the values application knows when it is compiled,
> > this
> > > > means either we can't extend our enum/defines until next ABI breakage, or
> > we
> > > > need to do ABI versioning to the functions that returns an enum each time
> > enum
> > > > value extended.
> > > >
> > > > I believe it is saner to provide _END/_MAX values to the application to use.
> > And
> > > > if required comment them to clarify the expected usage.
> > > >
> > > > But in above suggestion application can't use or rely on "_LIST_MAX", it
> > doesn't
> > > > mean anything to application.
> > > >
> > >
> > > Can we have something like
> > > enum rte_crypto_aead_algorithm {
> > > RTE_CRYPTO_AEAD_AES_CCM = 1,
> > > /**< AES algorithm in CCM mode. */
> > > RTE_CRYPTO_AEAD_AES_GCM,
> > > /**< AES algorithm in GCM mode. */
> > > RTE_CRYPTO_AEAD_LIST_END,
> > > /**< List end for 19.11 ABI compatibility */
> > > RTE_CRYPTO_AEAD_CHACHA20_POLY1305,
> > > /**< Chacha20 cipher with poly1305 authenticator */
> > > RTE_CRYPTO_AEAD_LIST_END_2011
> > > /**< List end for 20.11 ABI compatibility */
> > > };
> > >
> > > And in 20.11 release we alter the RTE_CRYPTO_AEAD_LIST_END to the end
> > and remove RTE_CRYPTO_AEAD_LIST_END_2011
> > >
> > > I believe it will be ok for any application which need to use the chacha poly
> > assume that this algo is
> > > Experimental and will move to formal list in 20.11. This can be documented in
> > the documentation.
> > > I believe there is no way to add a new enum as experimental so far. This way
> > we can formalize this
> > > requirement as well.
> > >
> > > I believe this way effect of ABI breakage will be nullified.
> >
> > This is a possibility in the (a) proposal.
> > But it breaks API (and ABI) because a high value is returned
> > while not expected by the application.
> >
> > I guess ABI and release maintainers will vote no to such breakage.
> > Note: I vote no.
> >
>
> If that is the case, I would say we should go with b).
>
> Versioned APIs does not look good and adds more confusion.
>
How does it add confusion, it's the standard and recommended way to fix
things like this? To maintain stable ABIs in the medium and long term we
need to get used to using versioning and not be afraid of it. Developers
will soon get used to the added bit of complexity it involves.
Regards,
/Bruce
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v7] eal: add manual probing option
2020-02-04 10:03 0% ` Gaetan Rivet
@ 2020-02-04 11:07 0% ` Thomas Monjalon
2020-02-04 12:43 0% ` Gaetan Rivet
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-04 11:07 UTC (permalink / raw)
To: Pavan Nikhilesh Bhagavatula, Gaetan Rivet
Cc: dev, dev, David Marchand, Vamsi Krishna Attunuru,
Jerin Jacob Kollanukkaran
04/02/2020 11:03, Gaetan Rivet:
> On 03/02/2020 23:21, Thomas Monjalon wrote:
> > 03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
> >> @David Marchand @thomas@monjalon.net
> >>
> >> Ping?
> >>
> >> Are there any more changes required for this patch? It's been in queue since last October.
> >
> > Sorry we have not decided whether it is a good idea or not.
> >
> > All changes related to probing are very sensitive,
> > and we know a big refactoring would be better than stacking
> > more and more options and corner cases.
> >
> > As we are busy with ABI stability stuff, we did not allocate
> > enough time to properly think about this feature.
> > Please accept our apologies, and let's consider it as
> > a high priority for 20.05 cycle.
> >
>
> Hello Thomas,
>
> This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.
>
> The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.
Yes, life is full of bad decisions and consequences.
I still think there is a risk in adding new user expectations,
and maintaining some code to workaround unknown issues.
The real question here is to know why this patch?
Is it to workaround a broken driver?
Or to workaround a broken design in EAL and bus drivers?
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-04 10:10 4% ` Trahe, Fiona
@ 2020-02-04 10:38 4% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-02-04 10:38 UTC (permalink / raw)
To: Trahe, Fiona
Cc: David Marchand, Ray Kinsella, dev, Neil Horman, Luca Boccassi,
Kevin Traynor, Ananyev, Konstantin, Akhil Goyal, Yigit, Ferruh,
dev, Anoob Joseph, Kusztal, ArkadiuszX, Richardson, Bruce,
Mcnamara, John, dodji, Andrew Rybchenko, Aaron Conole
04/02/2020 11:10, Trahe, Fiona:
> And not used for sizing > >
> > > There a c) though right.
> > > We could work around the issue by api versioning rte_cryptodev_info_get() and friends.
> >
> > It has a lot of friends, but it sounds like the right approach.
> > Is someone looking into this?
> [Fiona] Yes. But not clear yet if can be done by tomorrow.
Should be done by today now.
> But even if feasible, that only works around the current issue.
> There is a bigger issue to be decided here -
> Should we be removing LIST_END/MAX values from all enums in 20.11?
> Or defining through API comment that they should only be used as a range boundary and
> NOT to size an array. And so having a fixed value is not part of the API contract.
Please let's discuss 20.11 API later. It is not so urgent.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-04 9:45 4% ` Thomas Monjalon
@ 2020-02-04 9:56 4% ` Ferruh Yigit
2020-02-04 10:08 4% ` Bruce Richardson
2020-02-04 10:17 4% ` Kevin Traynor
0 siblings, 2 replies; 200+ results
From: Ferruh Yigit @ 2020-02-04 9:56 UTC (permalink / raw)
To: Thomas Monjalon
Cc: dev, Ananyev, Konstantin, Akhil Goyal, Trahe, Fiona,
David Marchand, Anoob Joseph, Kusztal, ArkadiuszX, Richardson,
Bruce, nhorman, Mcnamara, John, dodji, Andrew Rybchenko, aconole,
bluca, ktraynor
On 2/4/2020 9:45 AM, Thomas Monjalon wrote:
> 04/02/2020 10:19, Ferruh Yigit:
>> On 2/3/2020 6:40 PM, Thomas Monjalon wrote:
>>> 03/02/2020 18:40, Ferruh Yigit:
>>>> On 2/3/2020 5:09 PM, Thomas Monjalon wrote:
>>>>> 03/02/2020 10:30, Ferruh Yigit:
>>>>>> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
>>>>>>> 02/02/2020 14:05, Thomas Monjalon:
>>>>>>>> 31/01/2020 15:16, Trahe, Fiona:
>>>>>>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
>>>>>>>>>> 30/01/2020 17:09, Ferruh Yigit:
>>>>>>>>>>> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I believe these enums will be used only in case of ASYM case which is experimental.
>>>>>>>>>>>
>>>>>>>>>>> Independent from being experiment and not, this shouldn't be a problem, I think
>>>>>>>>>>> this is a false positive.
>>>>>>>>>>>
>>>>>>>>>>> The ABI break can happen when a struct has been shared between the application
>>>>>>>>>>> and the library (DPDK) and the layout of that memory know differently by
>>>>>>>>>>> application and the library.
>>>>>>>>>>>
>>>>>>>>>>> Here in all cases, there is no layout/size change.
>>>>>>>>>>>
>>>>>>>>>>> As to the value changes of the enums, since application compiled with old DPDK,
>>>>>>>>>>> it will know only up to '6', 7 and more means invalid to the application. So it
>>>>>>>>>>> won't send these values also it should ignore these values from library. Only
>>>>>>>>>>> consequence is old application won't able to use new features those new enums
>>>>>>>>>>> provide but that is expected/normal.
>>>>>>>>>>
>>>>>>>>>> If library give higher value than expected by the application,
>>>>>>>>>> if the application uses this value as array index,
>>>>>>>>>> there can be an access out of bounds.
>>>>>>>>>
>>>>>>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
>>>>>>>>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
>>>>>>>>> sense and I don't see how there can be an API breakage.
>>>>>>>>> So if an application hasn't compiled against the new lib it will be still using the old value
>>>>>>>>> which will be within bounds. If it's picking up the higher new value from the lib it must
>>>>>>>>> have been compiled against the lib so shouldn't have problems.
>>>>>>>>
>>>>>>>> You say there is no ABI issue because the application will be re-compiled
>>>>>>>> for the updated library. Indeed, compilation fixes compatibility issues.
>>>>>>>> But this is not relevant for ABI compatibility.
>>>>>>>> ABI compatibility means we can upgrade the library without recompiling
>>>>>>>> the application and it must work.
>>>>>>>> You think it is a false positive because you assume the application
>>>>>>>> "picks" the new value. I think you miss the case where the new value
>>>>>>>> is returned by a function in the upgraded library.
>>>>>>>>
>>>>>>>>> There are also no structs on the API which contain arrays using this
>>>>>>>>> for sizing, so I don't see an opportunity for an appl to have a
>>>>>>>>> mismatch in memory addresses.
>>>>>>>>
>>>>>>>> Let me demonstrate where the API may "use" the new value
>>>>>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
>>>>>>>>
>>>>>>>> Once upon a time a DPDK application counting the number of devices
>>>>>>>> supporting each AEAD algo (in order to find the best supported algo).
>>>>>>>> It is done in an array indexed by algo id:
>>>>>>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
>>>>>>>> The application is compiled with DPDK 19.11,
>>>>>>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
>>>>>>>> So the size of the application array aead_dev_count is 3.
>>>>>>>> This binary is run with DPDK 20.02,
>>>>>>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
>>>>>>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
>>>>>>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
>>>>>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
>>>>>>>> The application uses this value:
>>>>>>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
>>>>>>>> The application is crashing because of out of bound access.
>>>>>>>
>>>>>>> I'd say this is an example of bad written app.
>>>>>>> It probably should check that returned by library value doesn't
>>>>>>> exceed its internal array size.
>>>>>>
>>>>>> +1
>>>>>>
>>>>>> Application should ignore values >= MAX.
>>>>>
>>>>> Of course, blaming the API user is a lot easier than looking at the API.
>>>>> Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
>>>>> as the max value for the application.
>>>>> Value ranges are part of the ABI compatibility contract.
>>>>> It seems you expect the application developer to be aware that
>>>>> DPDK could return a higher value, so the application should
>>>>> check every enum values after calling an API. CRAZY.
>>>>>
>>>>> When we decide to announce an ABI compatibility and do some marketing,
>>>>> everyone is OK. But when we need to really make our ABI compatible,
>>>>> I see little or no effort. DISAPPOINTING.
>>>>
>>>> This is not to blame the user or to do less work, this is more sane approach
>>>> that library provides the _END/_MAX value and application uses it as valid range
>>>> check.
>>>>
>>>>>> Do you suggest we don't extend any enum or define between ABI breakage releases
>>>>>> to be sure bad written applications not affected?
>>>>>
>>>>> I suggest we must consider not breaking any assumption made on the API.
>>>>> Here we are breaking the enum range because nothing mentions _LIST_END
>>>>> is not really the absolute end of the enum.
>>>>> The solution is to make the change below in 20.02 + backport in 19.11.1:
>>>>>
>>>>> - _LIST_END
>>>>> + _LIST_END, /* an ABI-compatible version may increase this value */
>>>>> + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
>>>>> };
>>>>>
>>>>
>>>> What is the point of "_LIST_MAX" here?
>>>
>>> _LIST_MAX is range of value that DPDK can return in the ABI contract.
>>> So the appplication can rely on the range 0.._LIST_MAX.
>>>
>>>> Application should know the "_LIST_END" of when it has been compiled for the
>>>> valid range check. Next time it is compiled "_LIST_END" may be different value
>>>> but same logic applies.
>>>
>>> No, ABI compatibility contract means you can compile your application
>>> with DPDK 19.11.0 and run it with DPDK 20.02.
>>> So _LIST_END comes from 19.11 and does not include ChachaPoly.
>>
>> That is what I mean, let me try to give a sample.
>>
>> DPDK19.11 returns, A=1, B=2, END=3
>>
>> Application compiled with DPDK19.11, it will process A, B and ignore anything ">= 3"
>
> No, the application will not ignore anything ">=3" as I explained above,
> and you blamed the application for it.
> Nothing in the API says the application must filter value higher than 3,
> because as of now, values higher than 3 are PMD bug.
When application compiled, that is the END value, anything bigger than this
value is not valid, if any application use the return value directly I think it
is doing something wrong.
But yes there may be applications relying on library will always send in the
range. We never communicated this. But we can add comments to clarify this.
>
>
>> DPDK20.02 returns A=1, B=2, C=3, D=4, END=5
>>
>> Old application will still only will know/use A, B and can ignore when library
>> sends C=3, D=4 etc...
>>
>>
>> In above, if you add another limit as you suggested, like MAX=10 and ask
>> application to use it,
>>
>> Application compiled with DPDK19.11 will be OK since library only sends A,B and
>> application uses them.
>>
>> But with DPDK20.02 application may have problem, since library will be sending
>> C=3, which is valid according to the check " <= MAX (10)", how application will
>> know to ignore it.
>
> Why application should ignore value C=3 with DPDK 20.02?
This is the application compiled with DPDK19.11, and running with DPDK20.02.
So for the application this is the value >= MAX and something it doesn't know
what to do.
>
>
>> So application should use _END to know the valid ones according it, if so what
>> is the point of having _MAX.
>>
>>
>>>> When "_LIST_END" is missing, application can't protect itself, in that case
>>>> library should send only the values application knows when it is compiled, this
>>>> means either we can't extend our enum/defines until next ABI breakage, or we
>>>> need to do ABI versioning to the functions that returns an enum each time enum
>>>> value extended.
>>>
>>> If we define _LIST_MAX as a bigger value than current _LIST_END,
>>> we have some room to add values in between.
>>>
>>> If (as of now) we don't have _LIST_MAX room, then yes we must version
>>> the functions returning the enum.
>>> In this case, the proper solution is to implement
>>> rte_cryptodev_info_get_v1911() so it filters out
>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 capability.
>>> With this solution, an application compiled with DPDK 19.11 will keep
>>> seeing the same range as before, while a 20.02 application could
>>> see and use ChachaPoly.
>>> This is another proposal that I was expecting from the crypto team,
>>> instead of claiming there is no issue (and wasting precious time).
>>>
>>>
>>>> I believe it is saner to provide _END/_MAX values to the application to use. And
>>>> if required comment them to clarify the expected usage.
>>>>
>>>> But in above suggestion application can't use or rely on "_LIST_MAX", it doesn't
>>>> mean anything to application.
>>>
>>> I don't understand what you mean. I think you misunderstood what is ABI compat.
>>>
>>>
>>>>> Then *_LIST_END values could be ignored by libabigail with such a change.
>>>>>
>>>>> If such a patch is not done by tomorrow, I will have to revert
>>>>> Chacha-Poly commits before 20.02-rc2, because
>>>>>
>>>>> 1/ LIST_END, without any comment, means "size of range"
>>>>> 2/ we do not blame users for undocumented ABI changes
>>>>> 3/ we respect the ABI compatibility contract
>
>
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-04 10:28 8% ` Thomas Monjalon
@ 2020-02-04 10:32 4% ` Akhil Goyal
2020-02-04 11:35 4% ` Bruce Richardson
2020-02-04 22:10 7% ` Neil Horman
0 siblings, 2 replies; 200+ results
From: Akhil Goyal @ 2020-02-04 10:32 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Ferruh Yigit, Ananyev, Konstantin, Trahe, Fiona, dev,
David Marchand, Anoob Joseph, Kusztal, ArkadiuszX, dev,
Richardson, Bruce, nhorman, Mcnamara, John, dodji,
Andrew Rybchenko, aconole, bluca, ktraynor
>
> 04/02/2020 11:16, Akhil Goyal:
> > Hi,
> > > On 2/3/2020 5:09 PM, Thomas Monjalon wrote:
> > > > 03/02/2020 10:30, Ferruh Yigit:
> > > >> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
> > > >>> 02/02/2020 14:05, Thomas Monjalon:
> > > >>>> 31/01/2020 15:16, Trahe, Fiona:
> > > >>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> > > >>>>>> 30/01/2020 17:09, Ferruh Yigit:
> > > >>>>>>> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
> > > >>>>>>>>
> > > >>>>>>>> I believe these enums will be used only in case of ASYM case which
> is
> > > experimental.
> > > >>>>>>>
> > > >>>>>>> Independent from being experiment and not, this shouldn't be a
> > > problem, I think
> > > >>>>>>> this is a false positive.
> > > >>>>>>>
> > > >>>>>>> The ABI break can happen when a struct has been shared between
> the
> > > application
> > > >>>>>>> and the library (DPDK) and the layout of that memory know
> differently
> > > by
> > > >>>>>>> application and the library.
> > > >>>>>>>
> > > >>>>>>> Here in all cases, there is no layout/size change.
> > > >>>>>>>
> > > >>>>>>> As to the value changes of the enums, since application compiled
> with
> > > old DPDK,
> > > >>>>>>> it will know only up to '6', 7 and more means invalid to the
> application.
> > > So it
> > > >>>>>>> won't send these values also it should ignore these values from
> library.
> > > Only
> > > >>>>>>> consequence is old application won't able to use new features
> those
> > > new enums
> > > >>>>>>> provide but that is expected/normal.
> > > >>>>>>
> > > >>>>>> If library give higher value than expected by the application,
> > > >>>>>> if the application uses this value as array index,
> > > >>>>>> there can be an access out of bounds.
> > > >>>>>
> > > >>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a
> > > problem.
> > > >>>>> But for the same issue with sym crypto below, I believe Ferruh's
> > > explanation makes
> > > >>>>> sense and I don't see how there can be an API breakage.
> > > >>>>> So if an application hasn't compiled against the new lib it will be still
> using
> > > the old value
> > > >>>>> which will be within bounds. If it's picking up the higher new value
> from
> > > the lib it must
> > > >>>>> have been compiled against the lib so shouldn't have problems.
> > > >>>>
> > > >>>> You say there is no ABI issue because the application will be re-
> compiled
> > > >>>> for the updated library. Indeed, compilation fixes compatibility issues.
> > > >>>> But this is not relevant for ABI compatibility.
> > > >>>> ABI compatibility means we can upgrade the library without
> recompiling
> > > >>>> the application and it must work.
> > > >>>> You think it is a false positive because you assume the application
> > > >>>> "picks" the new value. I think you miss the case where the new value
> > > >>>> is returned by a function in the upgraded library.
> > > >>>>
> > > >>>>> There are also no structs on the API which contain arrays using this
> > > >>>>> for sizing, so I don't see an opportunity for an appl to have a
> > > >>>>> mismatch in memory addresses.
> > > >>>>
> > > >>>> Let me demonstrate where the API may "use" the new value
> > > >>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the
> > > application.
> > > >>>>
> > > >>>> Once upon a time a DPDK application counting the number of devices
> > > >>>> supporting each AEAD algo (in order to find the best supported algo).
> > > >>>> It is done in an array indexed by algo id:
> > > >>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
> > > >>>> The application is compiled with DPDK 19.11,
> > > >>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
> > > >>>> So the size of the application array aead_dev_count is 3.
> > > >>>> This binary is run with DPDK 20.02,
> > > >>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
> > > >>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
> > > >>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
> > > >>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
> > > >>>> The application uses this value:
> > > >>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
> > > >>>> The application is crashing because of out of bound access.
> > > >>>
> > > >>> I'd say this is an example of bad written app.
> > > >>> It probably should check that returned by library value doesn't
> > > >>> exceed its internal array size.
> > > >>
> > > >> +1
> > > >>
> > > >> Application should ignore values >= MAX.
> > > >
> > > > Of course, blaming the API user is a lot easier than looking at the API.
> > > > Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
> > > > as the max value for the application.
> > > > Value ranges are part of the ABI compatibility contract.
> > > > It seems you expect the application developer to be aware that
> > > > DPDK could return a higher value, so the application should
> > > > check every enum values after calling an API. CRAZY.
> > > >
> > > > When we decide to announce an ABI compatibility and do some marketing,
> > > > everyone is OK. But when we need to really make our ABI compatible,
> > > > I see little or no effort. DISAPPOINTING.
> > >
> > > This is not to blame the user or to do less work, this is more sane approach
> > > that library provides the _END/_MAX value and application uses it as valid
> range
> > > check.
> > >
> > > >
> > > >> Do you suggest we don't extend any enum or define between ABI
> breakage
> > > releases
> > > >> to be sure bad written applications not affected?
> > > >
> > > > I suggest we must consider not breaking any assumption made on the API.
> > > > Here we are breaking the enum range because nothing mentions
> _LIST_END
> > > > is not really the absolute end of the enum.
> > > > The solution is to make the change below in 20.02 + backport in 19.11.1:
> > > >
> > > > - _LIST_END
> > > > + _LIST_END, /* an ABI-compatible version may increase this value */
> > > > + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
> > > > };
> > > >
> > >
> > > What is the point of "_LIST_MAX" here?
> > >
> > > Application should know the "_LIST_END" of when it has been compiled for
> the
> > > valid range check. Next time it is compiled "_LIST_END" may be different
> value
> > > but same logic applies.
> > >
> > > When "_LIST_END" is missing, application can't protect itself, in that case
> > > library should send only the values application knows when it is compiled,
> this
> > > means either we can't extend our enum/defines until next ABI breakage, or
> we
> > > need to do ABI versioning to the functions that returns an enum each time
> enum
> > > value extended.
> > >
> > > I believe it is saner to provide _END/_MAX values to the application to use.
> And
> > > if required comment them to clarify the expected usage.
> > >
> > > But in above suggestion application can't use or rely on "_LIST_MAX", it
> doesn't
> > > mean anything to application.
> > >
> >
> > Can we have something like
> > enum rte_crypto_aead_algorithm {
> > RTE_CRYPTO_AEAD_AES_CCM = 1,
> > /**< AES algorithm in CCM mode. */
> > RTE_CRYPTO_AEAD_AES_GCM,
> > /**< AES algorithm in GCM mode. */
> > RTE_CRYPTO_AEAD_LIST_END,
> > /**< List end for 19.11 ABI compatibility */
> > RTE_CRYPTO_AEAD_CHACHA20_POLY1305,
> > /**< Chacha20 cipher with poly1305 authenticator */
> > RTE_CRYPTO_AEAD_LIST_END_2011
> > /**< List end for 20.11 ABI compatibility */
> > };
> >
> > And in 20.11 release we alter the RTE_CRYPTO_AEAD_LIST_END to the end
> and remove RTE_CRYPTO_AEAD_LIST_END_2011
> >
> > I believe it will be ok for any application which need to use the chacha poly
> assume that this algo is
> > Experimental and will move to formal list in 20.11. This can be documented in
> the documentation.
> > I believe there is no way to add a new enum as experimental so far. This way
> we can formalize this
> > requirement as well.
> >
> > I believe this way effect of ABI breakage will be nullified.
>
> This is a possibility in the (a) proposal.
> But it breaks API (and ABI) because a high value is returned
> while not expected by the application.
>
> I guess ABI and release maintainers will vote no to such breakage.
> Note: I vote no.
>
If that is the case, I would say we should go with b).
Versioned APIs does not look good and adds more confusion.
>
> > > > Then *_LIST_END values could be ignored by libabigail with such a change.
> > > >
> > > > If such a patch is not done by tomorrow, I will have to revert
> > > > Chacha-Poly commits before 20.02-rc2, because
> > > >
> > > > 1/ LIST_END, without any comment, means "size of range"
> > > > 2/ we do not blame users for undocumented ABI changes
> > > > 3/ we respect the ABI compatibility contract
>
>
>
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-04 10:16 9% ` Akhil Goyal
@ 2020-02-04 10:28 8% ` Thomas Monjalon
2020-02-04 10:32 4% ` Akhil Goyal
2020-02-04 21:59 8% ` [dpdk-dev] " Neil Horman
1 sibling, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-04 10:28 UTC (permalink / raw)
To: Akhil Goyal
Cc: Ferruh Yigit, Ananyev, Konstantin, Trahe, Fiona, dev,
David Marchand, Anoob Joseph, Kusztal, ArkadiuszX, dev,
Richardson, Bruce, nhorman, Mcnamara, John, dodji,
Andrew Rybchenko, aconole, bluca, ktraynor
04/02/2020 11:16, Akhil Goyal:
> Hi,
> > On 2/3/2020 5:09 PM, Thomas Monjalon wrote:
> > > 03/02/2020 10:30, Ferruh Yigit:
> > >> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
> > >>> 02/02/2020 14:05, Thomas Monjalon:
> > >>>> 31/01/2020 15:16, Trahe, Fiona:
> > >>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> > >>>>>> 30/01/2020 17:09, Ferruh Yigit:
> > >>>>>>> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
> > >>>>>>>>
> > >>>>>>>> I believe these enums will be used only in case of ASYM case which is
> > experimental.
> > >>>>>>>
> > >>>>>>> Independent from being experiment and not, this shouldn't be a
> > problem, I think
> > >>>>>>> this is a false positive.
> > >>>>>>>
> > >>>>>>> The ABI break can happen when a struct has been shared between the
> > application
> > >>>>>>> and the library (DPDK) and the layout of that memory know differently
> > by
> > >>>>>>> application and the library.
> > >>>>>>>
> > >>>>>>> Here in all cases, there is no layout/size change.
> > >>>>>>>
> > >>>>>>> As to the value changes of the enums, since application compiled with
> > old DPDK,
> > >>>>>>> it will know only up to '6', 7 and more means invalid to the application.
> > So it
> > >>>>>>> won't send these values also it should ignore these values from library.
> > Only
> > >>>>>>> consequence is old application won't able to use new features those
> > new enums
> > >>>>>>> provide but that is expected/normal.
> > >>>>>>
> > >>>>>> If library give higher value than expected by the application,
> > >>>>>> if the application uses this value as array index,
> > >>>>>> there can be an access out of bounds.
> > >>>>>
> > >>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a
> > problem.
> > >>>>> But for the same issue with sym crypto below, I believe Ferruh's
> > explanation makes
> > >>>>> sense and I don't see how there can be an API breakage.
> > >>>>> So if an application hasn't compiled against the new lib it will be still using
> > the old value
> > >>>>> which will be within bounds. If it's picking up the higher new value from
> > the lib it must
> > >>>>> have been compiled against the lib so shouldn't have problems.
> > >>>>
> > >>>> You say there is no ABI issue because the application will be re-compiled
> > >>>> for the updated library. Indeed, compilation fixes compatibility issues.
> > >>>> But this is not relevant for ABI compatibility.
> > >>>> ABI compatibility means we can upgrade the library without recompiling
> > >>>> the application and it must work.
> > >>>> You think it is a false positive because you assume the application
> > >>>> "picks" the new value. I think you miss the case where the new value
> > >>>> is returned by a function in the upgraded library.
> > >>>>
> > >>>>> There are also no structs on the API which contain arrays using this
> > >>>>> for sizing, so I don't see an opportunity for an appl to have a
> > >>>>> mismatch in memory addresses.
> > >>>>
> > >>>> Let me demonstrate where the API may "use" the new value
> > >>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the
> > application.
> > >>>>
> > >>>> Once upon a time a DPDK application counting the number of devices
> > >>>> supporting each AEAD algo (in order to find the best supported algo).
> > >>>> It is done in an array indexed by algo id:
> > >>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
> > >>>> The application is compiled with DPDK 19.11,
> > >>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
> > >>>> So the size of the application array aead_dev_count is 3.
> > >>>> This binary is run with DPDK 20.02,
> > >>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
> > >>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
> > >>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
> > >>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
> > >>>> The application uses this value:
> > >>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
> > >>>> The application is crashing because of out of bound access.
> > >>>
> > >>> I'd say this is an example of bad written app.
> > >>> It probably should check that returned by library value doesn't
> > >>> exceed its internal array size.
> > >>
> > >> +1
> > >>
> > >> Application should ignore values >= MAX.
> > >
> > > Of course, blaming the API user is a lot easier than looking at the API.
> > > Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
> > > as the max value for the application.
> > > Value ranges are part of the ABI compatibility contract.
> > > It seems you expect the application developer to be aware that
> > > DPDK could return a higher value, so the application should
> > > check every enum values after calling an API. CRAZY.
> > >
> > > When we decide to announce an ABI compatibility and do some marketing,
> > > everyone is OK. But when we need to really make our ABI compatible,
> > > I see little or no effort. DISAPPOINTING.
> >
> > This is not to blame the user or to do less work, this is more sane approach
> > that library provides the _END/_MAX value and application uses it as valid range
> > check.
> >
> > >
> > >> Do you suggest we don't extend any enum or define between ABI breakage
> > releases
> > >> to be sure bad written applications not affected?
> > >
> > > I suggest we must consider not breaking any assumption made on the API.
> > > Here we are breaking the enum range because nothing mentions _LIST_END
> > > is not really the absolute end of the enum.
> > > The solution is to make the change below in 20.02 + backport in 19.11.1:
> > >
> > > - _LIST_END
> > > + _LIST_END, /* an ABI-compatible version may increase this value */
> > > + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
> > > };
> > >
> >
> > What is the point of "_LIST_MAX" here?
> >
> > Application should know the "_LIST_END" of when it has been compiled for the
> > valid range check. Next time it is compiled "_LIST_END" may be different value
> > but same logic applies.
> >
> > When "_LIST_END" is missing, application can't protect itself, in that case
> > library should send only the values application knows when it is compiled, this
> > means either we can't extend our enum/defines until next ABI breakage, or we
> > need to do ABI versioning to the functions that returns an enum each time enum
> > value extended.
> >
> > I believe it is saner to provide _END/_MAX values to the application to use. And
> > if required comment them to clarify the expected usage.
> >
> > But in above suggestion application can't use or rely on "_LIST_MAX", it doesn't
> > mean anything to application.
> >
>
> Can we have something like
> enum rte_crypto_aead_algorithm {
> RTE_CRYPTO_AEAD_AES_CCM = 1,
> /**< AES algorithm in CCM mode. */
> RTE_CRYPTO_AEAD_AES_GCM,
> /**< AES algorithm in GCM mode. */
> RTE_CRYPTO_AEAD_LIST_END,
> /**< List end for 19.11 ABI compatibility */
> RTE_CRYPTO_AEAD_CHACHA20_POLY1305,
> /**< Chacha20 cipher with poly1305 authenticator */
> RTE_CRYPTO_AEAD_LIST_END_2011
> /**< List end for 20.11 ABI compatibility */
> };
>
> And in 20.11 release we alter the RTE_CRYPTO_AEAD_LIST_END to the end and remove RTE_CRYPTO_AEAD_LIST_END_2011
>
> I believe it will be ok for any application which need to use the chacha poly assume that this algo is
> Experimental and will move to formal list in 20.11. This can be documented in the documentation.
> I believe there is no way to add a new enum as experimental so far. This way we can formalize this
> requirement as well.
>
> I believe this way effect of ABI breakage will be nullified.
This is a possibility in the (a) proposal.
But it breaks API (and ABI) because a high value is returned
while not expected by the application.
I guess ABI and release maintainers will vote no to such breakage.
Note: I vote no.
> > > Then *_LIST_END values could be ignored by libabigail with such a change.
> > >
> > > If such a patch is not done by tomorrow, I will have to revert
> > > Chacha-Poly commits before 20.02-rc2, because
> > >
> > > 1/ LIST_END, without any comment, means "size of range"
> > > 2/ we do not blame users for undocumented ABI changes
> > > 3/ we respect the ABI compatibility contract
^ permalink raw reply [relevance 8%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-03 21:07 7% ` Thomas Monjalon
@ 2020-02-04 9:46 8% ` Ferruh Yigit
2020-02-04 10:24 9% ` Thomas Monjalon
1 sibling, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-04 9:46 UTC (permalink / raw)
To: Thomas Monjalon, David Marchand, nhorman, bluca, ktraynor, Ray Kinsella
Cc: Ananyev, Konstantin, Akhil Goyal, Trahe, Fiona, dev,
Anoob Joseph, Kusztal, ArkadiuszX, Richardson, Bruce, Mcnamara,
John, dodji, Andrew Rybchenko, aconole
On 2/3/2020 9:07 PM, Thomas Monjalon wrote:
> 03/02/2020 19:55, Ray Kinsella:
>> On 03/02/2020 17:34, Thomas Monjalon wrote:
>>> 03/02/2020 18:09, Thomas Monjalon:
>>>> 03/02/2020 10:30, Ferruh Yigit:
>>>>> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
>>>>>> 02/02/2020 14:05, Thomas Monjalon:
>>>>>>> 31/01/2020 15:16, Trahe, Fiona:
>>>>>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
>>>>>>>>> If library give higher value than expected by the application,
>>>>>>>>> if the application uses this value as array index,
>>>>>>>>> there can be an access out of bounds.
>>>>>>>>
>>>>>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
>>>>>>>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
>>>>>>>> sense and I don't see how there can be an API breakage.
>>>>>>>> So if an application hasn't compiled against the new lib it will be still using the old value
>>>>>>>> which will be within bounds. If it's picking up the higher new value from the lib it must
>>>>>>>> have been compiled against the lib so shouldn't have problems.
>>>>>>>
>>>>>>> You say there is no ABI issue because the application will be re-compiled
>>>>>>> for the updated library. Indeed, compilation fixes compatibility issues.
>>>>>>> But this is not relevant for ABI compatibility.
>>>>>>> ABI compatibility means we can upgrade the library without recompiling
>>>>>>> the application and it must work.
>>>>>>> You think it is a false positive because you assume the application
>>>>>>> "picks" the new value. I think you miss the case where the new value
>>>>>>> is returned by a function in the upgraded library.
>>>>>>>
>>>>>>>> There are also no structs on the API which contain arrays using this
>>>>>>>> for sizing, so I don't see an opportunity for an appl to have a
>>>>>>>> mismatch in memory addresses.
>>>>>>>
>>>>>>> Let me demonstrate where the API may "use" the new value
>>>>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
>>>>>>>
>>>>>>> Once upon a time a DPDK application counting the number of devices
>>>>>>> supporting each AEAD algo (in order to find the best supported algo).
>>>>>>> It is done in an array indexed by algo id:
>>>>>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
>>>>>>> The application is compiled with DPDK 19.11,
>>>>>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
>>>>>>> So the size of the application array aead_dev_count is 3.
>>>>>>> This binary is run with DPDK 20.02,
>>>>>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
>>>>>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
>>>>>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
>>>>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
>>>>>>> The application uses this value:
>>>>>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
>>>>>>> The application is crashing because of out of bound access.
>>>>>>
>>>>>> I'd say this is an example of bad written app.
>>>>>> It probably should check that returned by library value doesn't
>>>>>> exceed its internal array size.
>>>>>
>>>>> +1
>>>>>
>>>>> Application should ignore values >= MAX.
>>>>
>>>> Of course, blaming the API user is a lot easier than looking at the API.
>>>> Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
>>>> as the max value for the application.
>>>> Value ranges are part of the ABI compatibility contract.
>>>> It seems you expect the application developer to be aware that
>>>> DPDK could return a higher value, so the application should
>>>> check every enum values after calling an API. CRAZY.
>>>>
>>>> When we decide to announce an ABI compatibility and do some marketing,
>>>> everyone is OK. But when we need to really make our ABI compatible,
>>>> I see little or no effort. DISAPPOINTING.
>>>>
>>>>> Do you suggest we don't extend any enum or define between ABI breakage releases
>>>>> to be sure bad written applications not affected?
>>>>
>>>> I suggest we must consider not breaking any assumption made on the API.
>>>> Here we are breaking the enum range because nothing mentions _LIST_END
>>>> is not really the absolute end of the enum.
>>>> The solution is to make the change below in 20.02 + backport in 19.11.1:
>>>
>>> Thinking twice, merging such change before 20.11 is breaking the
>>> ABI assumption based on the API 19.11.0.
>>> I ask the release maintainers (Luca, Kevin, David and me) and
>>> the ABI maintainers (Neil and Ray) to vote for a or b solution:
>>> a) add comment and LIST_MAX as below in 20.02 + 19.11.1
>>
>> That would still be an ABI breakage though right.
>>
>>> b) wait 20.11 and revert Chacha-Poly from 20.02
>>
>> Thanks for analysis above Fiona, Ferruh and all.
>>
>> That is a nasty one alright - there is no "good" answer here.
>> I agree with Ferruh's sentiments overall, we should rethink this API for 20.11.
>> Could do without an enumeration?
>>
>> There a c) though right.
>> We could work around the issue by api versioning rte_cryptodev_info_get() and friends.
>> So they only support/acknowledge the existence of Chacha-Poly for applications build against > 20.02.
>
> I agree there is a c) as I proposed in another email:
> http://mails.dpdk.org/archives/dev/2020-February/156919.html
> "
> In this case, the proper solution is to implement
> rte_cryptodev_info_get_v1911() so it filters out
> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 capability.
> With this solution, an application compiled with DPDK 19.11 will keep
> seeing the same range as before, while a 20.02 application could
> see and use ChachaPoly.
> "
>
>> It would be painful I know.
>
> Not so painful in my opinion.
> Just need to call rte_cryptodev_info_get() from
> rte_cryptodev_info_get_v1911() and filter the value
> in the 19.11 range: [0..AES_GCM].
>
>> It would also mean that Chacha-Poly would only be available to
>> those building against >= 20.02.
>
> Yes exactly.
>
> The addition of comments and LIST_MAX like below are still valid
> to avoid versioning after 20.11.
>
>>>> - _LIST_END
>>>> + _LIST_END, /* an ABI-compatible version may increase this value */
>>>> + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
>>>> };
>>>>
>>>> Then *_LIST_END values could be ignored by libabigail with such a change.
>
> In order to avoid ABI check complaining, the best is to completely
> remove LIST_END in DPDK 20.11.
We can remove LIST_END only if we go with option (c).
Two different approach,
- Provide the LIST_END and expect application protect itself against new values
can be coming in newer version of the library
- Do ABI versioning to prevent application receive new values at all, (c).
We can select one, but I believe the selection shouldn't be based on just
silencing the ABI check tool.
>
>
>>>> If such a patch is not done by tomorrow, I will have to revert
>>>> Chacha-Poly commits before 20.02-rc2, because
>>>>
>>>> 1/ LIST_END, without any comment, means "size of range"
>>>> 2/ we do not blame users for undocumented ABI changes
>>>> 3/ we respect the ABI compatibility contract
>
>
>
^ permalink raw reply [relevance 8%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-03 21:07 7% ` Thomas Monjalon
2020-02-04 9:46 8% ` Ferruh Yigit
@ 2020-02-04 10:24 9% ` Thomas Monjalon
2020-02-04 12:44 4% ` Trahe, Fiona
` (2 more replies)
1 sibling, 3 replies; 200+ results
From: Thomas Monjalon @ 2020-02-04 10:24 UTC (permalink / raw)
To: David Marchand, nhorman, bluca, ktraynor, Ray Kinsella, dev,
Akhil Goyal, Trahe, Fiona, Ferruh Yigit
Cc: Ananyev, Konstantin, dev, Anoob Joseph, Kusztal, ArkadiuszX,
Richardson, Bruce, Mcnamara, John, dodji, Andrew Rybchenko,
aconole
RED FLAG
I don't see a lot of reactions, so I summarize the issue.
We need action TODAY!
API makes think that rte_cryptodev_info_get() cannot return
a value >= 3 (RTE_CRYPTO_AEAD_LIST_END in 19.11).
Current 20.02 returns 3 (RTE_CRYPTO_AEAD_CHACHA20_POLY1305).
The ABI compatibility contract is broken currently.
There are 3 possible outcomes:
a) Change the API comments and backport to 19.11.1
The details are discussed between Ferruh and me.
Either put responsibility on API user (with explicit comment),
or expose ABI extension allowance with a new API max value.
In both cases, this is breaking the implicit contract of 19.11.0.
This option can be chosen only if release and ABI maintainers
vote for it.
b) Revert Chacha-Poly from 20.02-rc2.
c) Add versioned function rte_cryptodev_info_get_v1911()
which calls rte_cryptodev_info_get() and filters out
RTE_CRYPTO_AEAD_CHACHA20_POLY1305 capability.
So Chacha-Poly capability would be seen and usable only
if compiling with DPDK 20.02.
I hope it is clear what are the actions for everybody:
- ABI and release maintainers must say yes or no to the proposal (a)
- In the meantime, crypto team must send a patch for the proposal (c)
- If (a) and (c) are not possible at the end of today, I will take (b)
Note: do not say it is too short for (c), as it was possible to work
on such solution since the issue was exposed on last Wednesday.
03/02/2020 22:07, Thomas Monjalon:
> 03/02/2020 19:55, Ray Kinsella:
> > On 03/02/2020 17:34, Thomas Monjalon wrote:
> > > 03/02/2020 18:09, Thomas Monjalon:
> > >> 03/02/2020 10:30, Ferruh Yigit:
> > >>> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
> > >>>> 02/02/2020 14:05, Thomas Monjalon:
> > >>>>> 31/01/2020 15:16, Trahe, Fiona:
> > >>>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> > >>>>>>> If library give higher value than expected by the application,
> > >>>>>>> if the application uses this value as array index,
> > >>>>>>> there can be an access out of bounds.
> > >>>>>>
> > >>>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
> > >>>>>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
> > >>>>>> sense and I don't see how there can be an API breakage.
> > >>>>>> So if an application hasn't compiled against the new lib it will be still using the old value
> > >>>>>> which will be within bounds. If it's picking up the higher new value from the lib it must
> > >>>>>> have been compiled against the lib so shouldn't have problems.
> > >>>>>
> > >>>>> You say there is no ABI issue because the application will be re-compiled
> > >>>>> for the updated library. Indeed, compilation fixes compatibility issues.
> > >>>>> But this is not relevant for ABI compatibility.
> > >>>>> ABI compatibility means we can upgrade the library without recompiling
> > >>>>> the application and it must work.
> > >>>>> You think it is a false positive because you assume the application
> > >>>>> "picks" the new value. I think you miss the case where the new value
> > >>>>> is returned by a function in the upgraded library.
> > >>>>>
> > >>>>>> There are also no structs on the API which contain arrays using this
> > >>>>>> for sizing, so I don't see an opportunity for an appl to have a
> > >>>>>> mismatch in memory addresses.
> > >>>>>
> > >>>>> Let me demonstrate where the API may "use" the new value
> > >>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
> > >>>>>
> > >>>>> Once upon a time a DPDK application counting the number of devices
> > >>>>> supporting each AEAD algo (in order to find the best supported algo).
> > >>>>> It is done in an array indexed by algo id:
> > >>>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
> > >>>>> The application is compiled with DPDK 19.11,
> > >>>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
> > >>>>> So the size of the application array aead_dev_count is 3.
> > >>>>> This binary is run with DPDK 20.02,
> > >>>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
> > >>>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
> > >>>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
> > >>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
> > >>>>> The application uses this value:
> > >>>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
> > >>>>> The application is crashing because of out of bound access.
> > >>>>
> > >>>> I'd say this is an example of bad written app.
> > >>>> It probably should check that returned by library value doesn't
> > >>>> exceed its internal array size.
> > >>>
> > >>> +1
> > >>>
> > >>> Application should ignore values >= MAX.
> > >>
> > >> Of course, blaming the API user is a lot easier than looking at the API.
> > >> Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
> > >> as the max value for the application.
> > >> Value ranges are part of the ABI compatibility contract.
> > >> It seems you expect the application developer to be aware that
> > >> DPDK could return a higher value, so the application should
> > >> check every enum values after calling an API. CRAZY.
> > >>
> > >> When we decide to announce an ABI compatibility and do some marketing,
> > >> everyone is OK. But when we need to really make our ABI compatible,
> > >> I see little or no effort. DISAPPOINTING.
> > >>
> > >>> Do you suggest we don't extend any enum or define between ABI breakage releases
> > >>> to be sure bad written applications not affected?
> > >>
> > >> I suggest we must consider not breaking any assumption made on the API.
> > >> Here we are breaking the enum range because nothing mentions _LIST_END
> > >> is not really the absolute end of the enum.
> > >> The solution is to make the change below in 20.02 + backport in 19.11.1:
> > >
> > > Thinking twice, merging such change before 20.11 is breaking the
> > > ABI assumption based on the API 19.11.0.
> > > I ask the release maintainers (Luca, Kevin, David and me) and
> > > the ABI maintainers (Neil and Ray) to vote for a or b solution:
> > > a) add comment and LIST_MAX as below in 20.02 + 19.11.1
> >
> > That would still be an ABI breakage though right.
> >
> > > b) wait 20.11 and revert Chacha-Poly from 20.02
> >
> > Thanks for analysis above Fiona, Ferruh and all.
> >
> > That is a nasty one alright - there is no "good" answer here.
> > I agree with Ferruh's sentiments overall, we should rethink this API for 20.11.
> > Could do without an enumeration?
> >
> > There a c) though right.
> > We could work around the issue by api versioning rte_cryptodev_info_get() and friends.
> > So they only support/acknowledge the existence of Chacha-Poly for applications build against > 20.02.
>
> I agree there is a c) as I proposed in another email:
> http://mails.dpdk.org/archives/dev/2020-February/156919.html
> "
> In this case, the proper solution is to implement
> rte_cryptodev_info_get_v1911() so it filters out
> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 capability.
> With this solution, an application compiled with DPDK 19.11 will keep
> seeing the same range as before, while a 20.02 application could
> see and use ChachaPoly.
> "
>
> > It would be painful I know.
>
> Not so painful in my opinion.
> Just need to call rte_cryptodev_info_get() from
> rte_cryptodev_info_get_v1911() and filter the value
> in the 19.11 range: [0..AES_GCM].
>
> > It would also mean that Chacha-Poly would only be available to
> > those building against >= 20.02.
>
> Yes exactly.
>
> The addition of comments and LIST_MAX like below are still valid
> to avoid versioning after 20.11.
>
> > >> - _LIST_END
> > >> + _LIST_END, /* an ABI-compatible version may increase this value */
> > >> + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
> > >> };
> > >>
> > >> Then *_LIST_END values could be ignored by libabigail with such a change.
>
> In order to avoid ABI check complaining, the best is to completely
> remove LIST_END in DPDK 20.11.
>
>
> > >> If such a patch is not done by tomorrow, I will have to revert
> > >> Chacha-Poly commits before 20.02-rc2, because
> > >>
> > >> 1/ LIST_END, without any comment, means "size of range"
> > >> 2/ we do not blame users for undocumented ABI changes
> > >> 3/ we respect the ABI compatibility contract
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-04 9:56 4% ` Ferruh Yigit
2020-02-04 10:08 4% ` Bruce Richardson
@ 2020-02-04 10:17 4% ` Kevin Traynor
1 sibling, 0 replies; 200+ results
From: Kevin Traynor @ 2020-02-04 10:17 UTC (permalink / raw)
To: Ferruh Yigit, Thomas Monjalon
Cc: dev, Ananyev, Konstantin, Akhil Goyal, Trahe, Fiona,
David Marchand, Anoob Joseph, Kusztal, ArkadiuszX, Richardson,
Bruce, nhorman, Mcnamara, John, dodji, Andrew Rybchenko, aconole,
bluca
On 04/02/2020 09:56, Ferruh Yigit wrote:
> On 2/4/2020 9:45 AM, Thomas Monjalon wrote:
>> 04/02/2020 10:19, Ferruh Yigit:
>>> On 2/3/2020 6:40 PM, Thomas Monjalon wrote:
>>>> 03/02/2020 18:40, Ferruh Yigit:
>>>>> On 2/3/2020 5:09 PM, Thomas Monjalon wrote:
>>>>>> 03/02/2020 10:30, Ferruh Yigit:
>>>>>>> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
>>>>>>>> 02/02/2020 14:05, Thomas Monjalon:
>>>>>>>>> 31/01/2020 15:16, Trahe, Fiona:
>>>>>>>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
>>>>>>>>>>> 30/01/2020 17:09, Ferruh Yigit:
>>>>>>>>>>>> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> I believe these enums will be used only in case of ASYM case which is experimental.
>>>>>>>>>>>>
>>>>>>>>>>>> Independent from being experiment and not, this shouldn't be a problem, I think
>>>>>>>>>>>> this is a false positive.
>>>>>>>>>>>>
>>>>>>>>>>>> The ABI break can happen when a struct has been shared between the application
>>>>>>>>>>>> and the library (DPDK) and the layout of that memory know differently by
>>>>>>>>>>>> application and the library.
>>>>>>>>>>>>
>>>>>>>>>>>> Here in all cases, there is no layout/size change.
>>>>>>>>>>>>
>>>>>>>>>>>> As to the value changes of the enums, since application compiled with old DPDK,
>>>>>>>>>>>> it will know only up to '6', 7 and more means invalid to the application. So it
>>>>>>>>>>>> won't send these values also it should ignore these values from library. Only
>>>>>>>>>>>> consequence is old application won't able to use new features those new enums
>>>>>>>>>>>> provide but that is expected/normal.
>>>>>>>>>>>
>>>>>>>>>>> If library give higher value than expected by the application,
>>>>>>>>>>> if the application uses this value as array index,
>>>>>>>>>>> there can be an access out of bounds.
>>>>>>>>>>
>>>>>>>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
>>>>>>>>>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
>>>>>>>>>> sense and I don't see how there can be an API breakage.
>>>>>>>>>> So if an application hasn't compiled against the new lib it will be still using the old value
>>>>>>>>>> which will be within bounds. If it's picking up the higher new value from the lib it must
>>>>>>>>>> have been compiled against the lib so shouldn't have problems.
>>>>>>>>>
>>>>>>>>> You say there is no ABI issue because the application will be re-compiled
>>>>>>>>> for the updated library. Indeed, compilation fixes compatibility issues.
>>>>>>>>> But this is not relevant for ABI compatibility.
>>>>>>>>> ABI compatibility means we can upgrade the library without recompiling
>>>>>>>>> the application and it must work.
>>>>>>>>> You think it is a false positive because you assume the application
>>>>>>>>> "picks" the new value. I think you miss the case where the new value
>>>>>>>>> is returned by a function in the upgraded library.
>>>>>>>>>
>>>>>>>>>> There are also no structs on the API which contain arrays using this
>>>>>>>>>> for sizing, so I don't see an opportunity for an appl to have a
>>>>>>>>>> mismatch in memory addresses.
>>>>>>>>>
>>>>>>>>> Let me demonstrate where the API may "use" the new value
>>>>>>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
>>>>>>>>>
>>>>>>>>> Once upon a time a DPDK application counting the number of devices
>>>>>>>>> supporting each AEAD algo (in order to find the best supported algo).
>>>>>>>>> It is done in an array indexed by algo id:
>>>>>>>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
>>>>>>>>> The application is compiled with DPDK 19.11,
>>>>>>>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
>>>>>>>>> So the size of the application array aead_dev_count is 3.
>>>>>>>>> This binary is run with DPDK 20.02,
>>>>>>>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
>>>>>>>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
>>>>>>>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
>>>>>>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
>>>>>>>>> The application uses this value:
>>>>>>>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
>>>>>>>>> The application is crashing because of out of bound access.
>>>>>>>>
>>>>>>>> I'd say this is an example of bad written app.
>>>>>>>> It probably should check that returned by library value doesn't
>>>>>>>> exceed its internal array size.
>>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> Application should ignore values >= MAX.
>>>>>>
>>>>>> Of course, blaming the API user is a lot easier than looking at the API.
>>>>>> Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
>>>>>> as the max value for the application.
>>>>>> Value ranges are part of the ABI compatibility contract.
>>>>>> It seems you expect the application developer to be aware that
>>>>>> DPDK could return a higher value, so the application should
>>>>>> check every enum values after calling an API. CRAZY.
>>>>>>
>>>>>> When we decide to announce an ABI compatibility and do some marketing,
>>>>>> everyone is OK. But when we need to really make our ABI compatible,
>>>>>> I see little or no effort. DISAPPOINTING.
>>>>>
>>>>> This is not to blame the user or to do less work, this is more sane approach
>>>>> that library provides the _END/_MAX value and application uses it as valid range
>>>>> check.
>>>>>
>>>>>>> Do you suggest we don't extend any enum or define between ABI breakage releases
>>>>>>> to be sure bad written applications not affected?
>>>>>>
>>>>>> I suggest we must consider not breaking any assumption made on the API.
>>>>>> Here we are breaking the enum range because nothing mentions _LIST_END
>>>>>> is not really the absolute end of the enum.
>>>>>> The solution is to make the change below in 20.02 + backport in 19.11.1:
>>>>>>
>>>>>> - _LIST_END
>>>>>> + _LIST_END, /* an ABI-compatible version may increase this value */
>>>>>> + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
>>>>>> };
>>>>>>
>>>>>
>>>>> What is the point of "_LIST_MAX" here?
>>>>
>>>> _LIST_MAX is range of value that DPDK can return in the ABI contract.
>>>> So the appplication can rely on the range 0.._LIST_MAX.
>>>>
>>>>> Application should know the "_LIST_END" of when it has been compiled for the
>>>>> valid range check. Next time it is compiled "_LIST_END" may be different value
>>>>> but same logic applies.
>>>>
>>>> No, ABI compatibility contract means you can compile your application
>>>> with DPDK 19.11.0 and run it with DPDK 20.02.
>>>> So _LIST_END comes from 19.11 and does not include ChachaPoly.
>>>
>>> That is what I mean, let me try to give a sample.
>>>
>>> DPDK19.11 returns, A=1, B=2, END=3
>>>
>>> Application compiled with DPDK19.11, it will process A, B and ignore anything ">= 3"
>>
>> No, the application will not ignore anything ">=3" as I explained above,
>> and you blamed the application for it.
>> Nothing in the API says the application must filter value higher than 3,
>> because as of now, values higher than 3 are PMD bug.
>
> When application compiled, that is the END value, anything bigger than this
> value is not valid, if any application use the return value directly I think it
> is doing something wrong.
I don't think we can make an assumption that the application will/should
range check *and* silently ignore what it considers out of bounds values.
An application may not range check, but if it does, it may ignore these
new values (we got lucky), or it could print errors or even decide to
abort because it considers that DPDK is now returning values higher than
the (compiled) max so something must be corrupt.
Versioning sounds the best solution to me too, but I'm not sure how
difficult the mechanics are in this case.
> But yes there may be applications relying on library will always send in the
> range. We never communicated this. But we can add comments to clarify this.
>
>>
>>
>>> DPDK20.02 returns A=1, B=2, C=3, D=4, END=5
>>>
>>> Old application will still only will know/use A, B and can ignore when library
>>> sends C=3, D=4 etc...
>>>
>>>
>>> In above, if you add another limit as you suggested, like MAX=10 and ask
>>> application to use it,
>>>
>>> Application compiled with DPDK19.11 will be OK since library only sends A,B and
>>> application uses them.
>>>
>>> But with DPDK20.02 application may have problem, since library will be sending
>>> C=3, which is valid according to the check " <= MAX (10)", how application will
>>> know to ignore it.
>>
>> Why application should ignore value C=3 with DPDK 20.02?
>
> This is the application compiled with DPDK19.11, and running with DPDK20.02.
>
> So for the application this is the value >= MAX and something it doesn't know
> what to do.
>
>>
>>
>>> So application should use _END to know the valid ones according it, if so what
>>> is the point of having _MAX.
>>>
>>>
>>>>> When "_LIST_END" is missing, application can't protect itself, in that case
>>>>> library should send only the values application knows when it is compiled, this
>>>>> means either we can't extend our enum/defines until next ABI breakage, or we
>>>>> need to do ABI versioning to the functions that returns an enum each time enum
>>>>> value extended.
>>>>
>>>> If we define _LIST_MAX as a bigger value than current _LIST_END,
>>>> we have some room to add values in between.
>>>>
>>>> If (as of now) we don't have _LIST_MAX room, then yes we must version
>>>> the functions returning the enum.
>>>> In this case, the proper solution is to implement
>>>> rte_cryptodev_info_get_v1911() so it filters out
>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 capability.
>>>> With this solution, an application compiled with DPDK 19.11 will keep
>>>> seeing the same range as before, while a 20.02 application could
>>>> see and use ChachaPoly.
>>>> This is another proposal that I was expecting from the crypto team,
>>>> instead of claiming there is no issue (and wasting precious time).
>>>>
>>>>
>>>>> I believe it is saner to provide _END/_MAX values to the application to use. And
>>>>> if required comment them to clarify the expected usage.
>>>>>
>>>>> But in above suggestion application can't use or rely on "_LIST_MAX", it doesn't
>>>>> mean anything to application.
>>>>
>>>> I don't understand what you mean. I think you misunderstood what is ABI compat.
>>>>
>>>>
>>>>>> Then *_LIST_END values could be ignored by libabigail with such a change.
>>>>>>
>>>>>> If such a patch is not done by tomorrow, I will have to revert
>>>>>> Chacha-Poly commits before 20.02-rc2, because
>>>>>>
>>>>>> 1/ LIST_END, without any comment, means "size of range"
>>>>>> 2/ we do not blame users for undocumented ABI changes
>>>>>> 3/ we respect the ABI compatibility contract
>>
>>
>>
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-03 17:40 8% ` Ferruh Yigit
2020-02-03 18:40 9% ` Thomas Monjalon
@ 2020-02-04 10:16 9% ` Akhil Goyal
2020-02-04 10:28 8% ` Thomas Monjalon
2020-02-04 21:59 8% ` [dpdk-dev] " Neil Horman
1 sibling, 2 replies; 200+ results
From: Akhil Goyal @ 2020-02-04 10:16 UTC (permalink / raw)
To: Ferruh Yigit, Thomas Monjalon, Ananyev, Konstantin, Trahe, Fiona
Cc: David Marchand, Anoob Joseph, Kusztal, ArkadiuszX, dev,
Richardson, Bruce, nhorman, Mcnamara, John, dodji,
Andrew Rybchenko, aconole, bluca, ktraynor
Hi,
> On 2/3/2020 5:09 PM, Thomas Monjalon wrote:
> > 03/02/2020 10:30, Ferruh Yigit:
> >> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
> >>> 02/02/2020 14:05, Thomas Monjalon:
> >>>> 31/01/2020 15:16, Trahe, Fiona:
> >>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> >>>>>> 30/01/2020 17:09, Ferruh Yigit:
> >>>>>>> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
> >>>>>>>>
> >>>>>>>> I believe these enums will be used only in case of ASYM case which is
> experimental.
> >>>>>>>
> >>>>>>> Independent from being experiment and not, this shouldn't be a
> problem, I think
> >>>>>>> this is a false positive.
> >>>>>>>
> >>>>>>> The ABI break can happen when a struct has been shared between the
> application
> >>>>>>> and the library (DPDK) and the layout of that memory know differently
> by
> >>>>>>> application and the library.
> >>>>>>>
> >>>>>>> Here in all cases, there is no layout/size change.
> >>>>>>>
> >>>>>>> As to the value changes of the enums, since application compiled with
> old DPDK,
> >>>>>>> it will know only up to '6', 7 and more means invalid to the application.
> So it
> >>>>>>> won't send these values also it should ignore these values from library.
> Only
> >>>>>>> consequence is old application won't able to use new features those
> new enums
> >>>>>>> provide but that is expected/normal.
> >>>>>>
> >>>>>> If library give higher value than expected by the application,
> >>>>>> if the application uses this value as array index,
> >>>>>> there can be an access out of bounds.
> >>>>>
> >>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a
> problem.
> >>>>> But for the same issue with sym crypto below, I believe Ferruh's
> explanation makes
> >>>>> sense and I don't see how there can be an API breakage.
> >>>>> So if an application hasn't compiled against the new lib it will be still using
> the old value
> >>>>> which will be within bounds. If it's picking up the higher new value from
> the lib it must
> >>>>> have been compiled against the lib so shouldn't have problems.
> >>>>
> >>>> You say there is no ABI issue because the application will be re-compiled
> >>>> for the updated library. Indeed, compilation fixes compatibility issues.
> >>>> But this is not relevant for ABI compatibility.
> >>>> ABI compatibility means we can upgrade the library without recompiling
> >>>> the application and it must work.
> >>>> You think it is a false positive because you assume the application
> >>>> "picks" the new value. I think you miss the case where the new value
> >>>> is returned by a function in the upgraded library.
> >>>>
> >>>>> There are also no structs on the API which contain arrays using this
> >>>>> for sizing, so I don't see an opportunity for an appl to have a
> >>>>> mismatch in memory addresses.
> >>>>
> >>>> Let me demonstrate where the API may "use" the new value
> >>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the
> application.
> >>>>
> >>>> Once upon a time a DPDK application counting the number of devices
> >>>> supporting each AEAD algo (in order to find the best supported algo).
> >>>> It is done in an array indexed by algo id:
> >>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
> >>>> The application is compiled with DPDK 19.11,
> >>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
> >>>> So the size of the application array aead_dev_count is 3.
> >>>> This binary is run with DPDK 20.02,
> >>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
> >>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
> >>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
> >>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
> >>>> The application uses this value:
> >>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
> >>>> The application is crashing because of out of bound access.
> >>>
> >>> I'd say this is an example of bad written app.
> >>> It probably should check that returned by library value doesn't
> >>> exceed its internal array size.
> >>
> >> +1
> >>
> >> Application should ignore values >= MAX.
> >
> > Of course, blaming the API user is a lot easier than looking at the API.
> > Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
> > as the max value for the application.
> > Value ranges are part of the ABI compatibility contract.
> > It seems you expect the application developer to be aware that
> > DPDK could return a higher value, so the application should
> > check every enum values after calling an API. CRAZY.
> >
> > When we decide to announce an ABI compatibility and do some marketing,
> > everyone is OK. But when we need to really make our ABI compatible,
> > I see little or no effort. DISAPPOINTING.
>
> This is not to blame the user or to do less work, this is more sane approach
> that library provides the _END/_MAX value and application uses it as valid range
> check.
>
> >
> >> Do you suggest we don't extend any enum or define between ABI breakage
> releases
> >> to be sure bad written applications not affected?
> >
> > I suggest we must consider not breaking any assumption made on the API.
> > Here we are breaking the enum range because nothing mentions _LIST_END
> > is not really the absolute end of the enum.
> > The solution is to make the change below in 20.02 + backport in 19.11.1:
> >
> > - _LIST_END
> > + _LIST_END, /* an ABI-compatible version may increase this value */
> > + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
> > };
> >
>
> What is the point of "_LIST_MAX" here?
>
> Application should know the "_LIST_END" of when it has been compiled for the
> valid range check. Next time it is compiled "_LIST_END" may be different value
> but same logic applies.
>
> When "_LIST_END" is missing, application can't protect itself, in that case
> library should send only the values application knows when it is compiled, this
> means either we can't extend our enum/defines until next ABI breakage, or we
> need to do ABI versioning to the functions that returns an enum each time enum
> value extended.
>
> I believe it is saner to provide _END/_MAX values to the application to use. And
> if required comment them to clarify the expected usage.
>
> But in above suggestion application can't use or rely on "_LIST_MAX", it doesn't
> mean anything to application.
>
Can we have something like
enum rte_crypto_aead_algorithm {
RTE_CRYPTO_AEAD_AES_CCM = 1,
/**< AES algorithm in CCM mode. */
RTE_CRYPTO_AEAD_AES_GCM,
/**< AES algorithm in GCM mode. */
RTE_CRYPTO_AEAD_LIST_END,
/**< List end for 19.11 ABI compatibility */
RTE_CRYPTO_AEAD_CHACHA20_POLY1305,
/**< Chacha20 cipher with poly1305 authenticator */
RTE_CRYPTO_AEAD_LIST_END_2011
/**< List end for 20.11 ABI compatibility */
};
And in 20.11 release we alter the RTE_CRYPTO_AEAD_LIST_END to the end and remove RTE_CRYPTO_AEAD_LIST_END_2011
I believe it will be ok for any application which need to use the chacha poly assume that this algo is
Experimental and will move to formal list in 20.11. This can be documented in the documentation.
I believe there is no way to add a new enum as experimental so far. This way we can formalize this
requirement as well.
I believe this way effect of ABI breakage will be nullified.
-Akhil
> > Then *_LIST_END values could be ignored by libabigail with such a change.
> >
> > If such a patch is not done by tomorrow, I will have to revert
> > Chacha-Poly commits before 20.02-rc2, because
> >
> > 1/ LIST_END, without any comment, means "size of range"
> > 2/ we do not blame users for undocumented ABI changes
> > 3/ we respect the ABI compatibility contract
> >
> >
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-04 9:51 4% ` David Marchand
@ 2020-02-04 10:10 4% ` Trahe, Fiona
2020-02-04 10:38 4% ` Thomas Monjalon
2020-02-05 11:10 4% ` Ray Kinsella
1 sibling, 1 reply; 200+ results
From: Trahe, Fiona @ 2020-02-04 10:10 UTC (permalink / raw)
To: David Marchand, Ray Kinsella
Cc: Thomas Monjalon, Neil Horman, Luca Boccassi, Kevin Traynor,
Ananyev, Konstantin, Akhil Goyal, Yigit, Ferruh, dev,
Anoob Joseph, Kusztal, ArkadiuszX, Richardson, Bruce, Mcnamara,
John, dodji, Andrew Rybchenko, Aaron Conole, Trahe, Fiona
And not used for sizing > >
> > There a c) though right.
> > We could work around the issue by api versioning rte_cryptodev_info_get() and friends.
>
> It has a lot of friends, but it sounds like the right approach.
> Is someone looking into this?
[Fiona] Yes. But not clear yet if can be done by tomorrow.
But even if feasible, that only works around the current issue.
There is a bigger issue to be decided here -
Should we be removing LIST_END/MAX values from all enums in 20.11?
Or defining through API comment that they should only be used as a range boundary and
NOT to size an array. And so having a fixed value is not part of the API contract.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-04 9:56 4% ` Ferruh Yigit
@ 2020-02-04 10:08 4% ` Bruce Richardson
2020-02-04 10:17 4% ` Kevin Traynor
1 sibling, 0 replies; 200+ results
From: Bruce Richardson @ 2020-02-04 10:08 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Thomas Monjalon, dev, Ananyev, Konstantin, Akhil Goyal, Trahe,
Fiona, David Marchand, Anoob Joseph, Kusztal, ArkadiuszX,
nhorman, Mcnamara, John, dodji, Andrew Rybchenko, aconole, bluca,
ktraynor
On Tue, Feb 04, 2020 at 09:56:31AM +0000, Ferruh Yigit wrote:
> On 2/4/2020 9:45 AM, Thomas Monjalon wrote:
> > 04/02/2020 10:19, Ferruh Yigit:
> >> On 2/3/2020 6:40 PM, Thomas Monjalon wrote:
> >>> 03/02/2020 18:40, Ferruh Yigit:
> >>>> On 2/3/2020 5:09 PM, Thomas Monjalon wrote:
> >>>>> 03/02/2020 10:30, Ferruh Yigit:
> >>>>>> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
> >>>>>>> 02/02/2020 14:05, Thomas Monjalon:
> >>>>>>>> 31/01/2020 15:16, Trahe, Fiona:
> >>>>>>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> >>>>>>>>>> 30/01/2020 17:09, Ferruh Yigit:
> >>>>>>>>>>> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> I believe these enums will be used only in case of ASYM case which is experimental.
> >>>>>>>>>>>
> >>>>>>>>>>> Independent from being experiment and not, this shouldn't be a problem, I think
> >>>>>>>>>>> this is a false positive.
> >>>>>>>>>>>
> >>>>>>>>>>> The ABI break can happen when a struct has been shared between the application
> >>>>>>>>>>> and the library (DPDK) and the layout of that memory know differently by
> >>>>>>>>>>> application and the library.
> >>>>>>>>>>>
> >>>>>>>>>>> Here in all cases, there is no layout/size change.
> >>>>>>>>>>>
> >>>>>>>>>>> As to the value changes of the enums, since application compiled with old DPDK,
> >>>>>>>>>>> it will know only up to '6', 7 and more means invalid to the application. So it
> >>>>>>>>>>> won't send these values also it should ignore these values from library. Only
> >>>>>>>>>>> consequence is old application won't able to use new features those new enums
> >>>>>>>>>>> provide but that is expected/normal.
> >>>>>>>>>>
> >>>>>>>>>> If library give higher value than expected by the application,
> >>>>>>>>>> if the application uses this value as array index,
> >>>>>>>>>> there can be an access out of bounds.
> >>>>>>>>>
> >>>>>>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
> >>>>>>>>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
> >>>>>>>>> sense and I don't see how there can be an API breakage.
> >>>>>>>>> So if an application hasn't compiled against the new lib it will be still using the old value
> >>>>>>>>> which will be within bounds. If it's picking up the higher new value from the lib it must
> >>>>>>>>> have been compiled against the lib so shouldn't have problems.
> >>>>>>>>
> >>>>>>>> You say there is no ABI issue because the application will be re-compiled
> >>>>>>>> for the updated library. Indeed, compilation fixes compatibility issues.
> >>>>>>>> But this is not relevant for ABI compatibility.
> >>>>>>>> ABI compatibility means we can upgrade the library without recompiling
> >>>>>>>> the application and it must work.
> >>>>>>>> You think it is a false positive because you assume the application
> >>>>>>>> "picks" the new value. I think you miss the case where the new value
> >>>>>>>> is returned by a function in the upgraded library.
> >>>>>>>>
> >>>>>>>>> There are also no structs on the API which contain arrays using this
> >>>>>>>>> for sizing, so I don't see an opportunity for an appl to have a
> >>>>>>>>> mismatch in memory addresses.
> >>>>>>>>
> >>>>>>>> Let me demonstrate where the API may "use" the new value
> >>>>>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
> >>>>>>>>
> >>>>>>>> Once upon a time a DPDK application counting the number of devices
> >>>>>>>> supporting each AEAD algo (in order to find the best supported algo).
> >>>>>>>> It is done in an array indexed by algo id:
> >>>>>>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
> >>>>>>>> The application is compiled with DPDK 19.11,
> >>>>>>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
> >>>>>>>> So the size of the application array aead_dev_count is 3.
> >>>>>>>> This binary is run with DPDK 20.02,
> >>>>>>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
> >>>>>>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
> >>>>>>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
> >>>>>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
> >>>>>>>> The application uses this value:
> >>>>>>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
> >>>>>>>> The application is crashing because of out of bound access.
> >>>>>>>
> >>>>>>> I'd say this is an example of bad written app.
> >>>>>>> It probably should check that returned by library value doesn't
> >>>>>>> exceed its internal array size.
> >>>>>>
> >>>>>> +1
> >>>>>>
> >>>>>> Application should ignore values >= MAX.
> >>>>>
> >>>>> Of course, blaming the API user is a lot easier than looking at the API.
> >>>>> Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
> >>>>> as the max value for the application.
> >>>>> Value ranges are part of the ABI compatibility contract.
> >>>>> It seems you expect the application developer to be aware that
> >>>>> DPDK could return a higher value, so the application should
> >>>>> check every enum values after calling an API. CRAZY.
> >>>>>
> >>>>> When we decide to announce an ABI compatibility and do some marketing,
> >>>>> everyone is OK. But when we need to really make our ABI compatible,
> >>>>> I see little or no effort. DISAPPOINTING.
> >>>>
> >>>> This is not to blame the user or to do less work, this is more sane approach
> >>>> that library provides the _END/_MAX value and application uses it as valid range
> >>>> check.
> >>>>
> >>>>>> Do you suggest we don't extend any enum or define between ABI breakage releases
> >>>>>> to be sure bad written applications not affected?
> >>>>>
> >>>>> I suggest we must consider not breaking any assumption made on the API.
> >>>>> Here we are breaking the enum range because nothing mentions _LIST_END
> >>>>> is not really the absolute end of the enum.
> >>>>> The solution is to make the change below in 20.02 + backport in 19.11.1:
> >>>>>
> >>>>> - _LIST_END
> >>>>> + _LIST_END, /* an ABI-compatible version may increase this value */
> >>>>> + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
> >>>>> };
> >>>>>
> >>>>
> >>>> What is the point of "_LIST_MAX" here?
> >>>
> >>> _LIST_MAX is range of value that DPDK can return in the ABI contract.
> >>> So the appplication can rely on the range 0.._LIST_MAX.
> >>>
> >>>> Application should know the "_LIST_END" of when it has been compiled for the
> >>>> valid range check. Next time it is compiled "_LIST_END" may be different value
> >>>> but same logic applies.
> >>>
> >>> No, ABI compatibility contract means you can compile your application
> >>> with DPDK 19.11.0 and run it with DPDK 20.02.
> >>> So _LIST_END comes from 19.11 and does not include ChachaPoly.
> >>
> >> That is what I mean, let me try to give a sample.
> >>
> >> DPDK19.11 returns, A=1, B=2, END=3
> >>
> >> Application compiled with DPDK19.11, it will process A, B and ignore anything ">= 3"
> >
> > No, the application will not ignore anything ">=3" as I explained above,
> > and you blamed the application for it.
> > Nothing in the API says the application must filter value higher than 3,
> > because as of now, values higher than 3 are PMD bug.
>
> When application compiled, that is the END value, anything bigger than this
> value is not valid, if any application use the return value directly I think it
> is doing something wrong.
> But yes there may be applications relying on library will always send in the
> range. We never communicated this. But we can add comments to clarify this.
>
I don't think we should do so, as for any function returning an enum by
definition it should never return an out-of-range value. I strongly agree
with the suggestion of versioning the functions so that the ranges seen by
apps are clamped to the expected 19.11 compatible values.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v7] eal: add manual probing option
2020-02-03 22:21 3% ` Thomas Monjalon
@ 2020-02-04 10:03 0% ` Gaetan Rivet
2020-02-04 11:07 0% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Gaetan Rivet @ 2020-02-04 10:03 UTC (permalink / raw)
To: Thomas Monjalon, Pavan Nikhilesh Bhagavatula
Cc: dev, David Marchand, Vamsi Krishna Attunuru, Jerin Jacob Kollanukkaran
On 03/02/2020 23:21, Thomas Monjalon wrote:
> 03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
>> @David Marchand @thomas@monjalon.net
>>
>> Ping?
>>
>> Are there any more changes required for this patch? It's been in queue since last October.
>
> Sorry we have not decided whether it is a good idea or not.
>
> All changes related to probing are very sensitive,
> and we know a big refactoring would be better than stacking
> more and more options and corner cases.
>
> As we are busy with ABI stability stuff, we did not allocate
> enough time to properly think about this feature.
> Please accept our apologies, and let's consider it as
> a high priority for 20.05 cycle.
>
>
>
Hello Thomas,
This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.
The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-03 18:55 7% ` Ray Kinsella
2020-02-03 21:07 7% ` Thomas Monjalon
@ 2020-02-04 9:51 4% ` David Marchand
2020-02-04 10:10 4% ` Trahe, Fiona
2020-02-05 11:10 4% ` Ray Kinsella
1 sibling, 2 replies; 200+ results
From: David Marchand @ 2020-02-04 9:51 UTC (permalink / raw)
To: Ray Kinsella
Cc: Thomas Monjalon, Neil Horman, Luca Boccassi, Kevin Traynor,
Ananyev, Konstantin, Akhil Goyal, Trahe, Fiona, Ferruh Yigit,
dev, Anoob Joseph, Kusztal, ArkadiuszX, Richardson, Bruce,
Mcnamara, John, dodji, Andrew Rybchenko, Aaron Conole
On Mon, Feb 3, 2020 at 7:56 PM Ray Kinsella <mdr@ashroe.eu> wrote:
> On 03/02/2020 17:34, Thomas Monjalon wrote:
> > 03/02/2020 18:09, Thomas Monjalon:
> >> 03/02/2020 10:30, Ferruh Yigit:
> >>> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
> >>>> 02/02/2020 14:05, Thomas Monjalon:
> >>>>> 31/01/2020 15:16, Trahe, Fiona:
> >>>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> >>>>>>> If library give higher value than expected by the application,
> >>>>>>> if the application uses this value as array index,
> >>>>>>> there can be an access out of bounds.
> >>>>>>
> >>>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
> >>>>>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
> >>>>>> sense and I don't see how there can be an API breakage.
> >>>>>> So if an application hasn't compiled against the new lib it will be still using the old value
> >>>>>> which will be within bounds. If it's picking up the higher new value from the lib it must
> >>>>>> have been compiled against the lib so shouldn't have problems.
> >>>>>
> >>>>> You say there is no ABI issue because the application will be re-compiled
> >>>>> for the updated library. Indeed, compilation fixes compatibility issues.
> >>>>> But this is not relevant for ABI compatibility.
> >>>>> ABI compatibility means we can upgrade the library without recompiling
> >>>>> the application and it must work.
> >>>>> You think it is a false positive because you assume the application
> >>>>> "picks" the new value. I think you miss the case where the new value
> >>>>> is returned by a function in the upgraded library.
> >>>>>
> >>>>>> There are also no structs on the API which contain arrays using this
> >>>>>> for sizing, so I don't see an opportunity for an appl to have a
> >>>>>> mismatch in memory addresses.
> >>>>>
> >>>>> Let me demonstrate where the API may "use" the new value
> >>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
> >>>>>
> >>>>> Once upon a time a DPDK application counting the number of devices
> >>>>> supporting each AEAD algo (in order to find the best supported algo).
> >>>>> It is done in an array indexed by algo id:
> >>>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
> >>>>> The application is compiled with DPDK 19.11,
> >>>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
> >>>>> So the size of the application array aead_dev_count is 3.
> >>>>> This binary is run with DPDK 20.02,
> >>>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
> >>>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
> >>>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
> >>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
> >>>>> The application uses this value:
> >>>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
> >>>>> The application is crashing because of out of bound access.
> >>>>
> >>>> I'd say this is an example of bad written app.
> >>>> It probably should check that returned by library value doesn't
> >>>> exceed its internal array size.
> >>>
> >>> +1
> >>>
> >>> Application should ignore values >= MAX.
> >>
> >> Of course, blaming the API user is a lot easier than looking at the API.
> >> Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
> >> as the max value for the application.
> >> Value ranges are part of the ABI compatibility contract.
> >> It seems you expect the application developer to be aware that
> >> DPDK could return a higher value, so the application should
> >> check every enum values after calling an API. CRAZY.
> >>
> >> When we decide to announce an ABI compatibility and do some marketing,
> >> everyone is OK. But when we need to really make our ABI compatible,
> >> I see little or no effort. DISAPPOINTING.
> >>
> >>> Do you suggest we don't extend any enum or define between ABI breakage releases
> >>> to be sure bad written applications not affected?
> >>
> >> I suggest we must consider not breaking any assumption made on the API.
> >> Here we are breaking the enum range because nothing mentions _LIST_END
> >> is not really the absolute end of the enum.
> >> The solution is to make the change below in 20.02 + backport in 19.11.1:
> >
> > Thinking twice, merging such change before 20.11 is breaking the
> > ABI assumption based on the API 19.11.0.
> > I ask the release maintainers (Luca, Kevin, David and me) and
> > the ABI maintainers (Neil and Ray) to vote for a or b solution:
> > a) add comment and LIST_MAX as below in 20.02 + 19.11.1
>
> That would still be an ABI breakage though right.
Yes.
>
> > b) wait 20.11 and revert Chacha-Poly from 20.02
>
> Thanks for analysis above Fiona, Ferruh and all.
>
> That is a nasty one alright - there is no "good" answer here.
> I agree with Ferruh's sentiments overall, we should rethink this API for 20.11.
> Could do without an enumeration?
>
> There a c) though right.
> We could work around the issue by api versioning rte_cryptodev_info_get() and friends.
It has a lot of friends, but it sounds like the right approach.
Is someone looking into this?
> So they only support/acknowledge the existence of Chacha-Poly for applications build against > 20.02.
>
> It would be painful I know.
> It would also mean that Chacha-Poly would only be available to those building against >= 20.02.
Yes.
--
David Marchand
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-04 9:19 4% ` Ferruh Yigit
@ 2020-02-04 9:45 4% ` Thomas Monjalon
2020-02-04 9:56 4% ` Ferruh Yigit
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-04 9:45 UTC (permalink / raw)
To: Ferruh Yigit
Cc: dev, Ananyev, Konstantin, Akhil Goyal, Trahe, Fiona,
David Marchand, Anoob Joseph, Kusztal, ArkadiuszX, dev,
Richardson, Bruce, nhorman, Mcnamara, John, dodji,
Andrew Rybchenko, aconole, bluca, ktraynor
04/02/2020 10:19, Ferruh Yigit:
> On 2/3/2020 6:40 PM, Thomas Monjalon wrote:
> > 03/02/2020 18:40, Ferruh Yigit:
> >> On 2/3/2020 5:09 PM, Thomas Monjalon wrote:
> >>> 03/02/2020 10:30, Ferruh Yigit:
> >>>> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
> >>>>> 02/02/2020 14:05, Thomas Monjalon:
> >>>>>> 31/01/2020 15:16, Trahe, Fiona:
> >>>>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> >>>>>>>> 30/01/2020 17:09, Ferruh Yigit:
> >>>>>>>>> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
> >>>>>>>>>>
> >>>>>>>>>> I believe these enums will be used only in case of ASYM case which is experimental.
> >>>>>>>>>
> >>>>>>>>> Independent from being experiment and not, this shouldn't be a problem, I think
> >>>>>>>>> this is a false positive.
> >>>>>>>>>
> >>>>>>>>> The ABI break can happen when a struct has been shared between the application
> >>>>>>>>> and the library (DPDK) and the layout of that memory know differently by
> >>>>>>>>> application and the library.
> >>>>>>>>>
> >>>>>>>>> Here in all cases, there is no layout/size change.
> >>>>>>>>>
> >>>>>>>>> As to the value changes of the enums, since application compiled with old DPDK,
> >>>>>>>>> it will know only up to '6', 7 and more means invalid to the application. So it
> >>>>>>>>> won't send these values also it should ignore these values from library. Only
> >>>>>>>>> consequence is old application won't able to use new features those new enums
> >>>>>>>>> provide but that is expected/normal.
> >>>>>>>>
> >>>>>>>> If library give higher value than expected by the application,
> >>>>>>>> if the application uses this value as array index,
> >>>>>>>> there can be an access out of bounds.
> >>>>>>>
> >>>>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
> >>>>>>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
> >>>>>>> sense and I don't see how there can be an API breakage.
> >>>>>>> So if an application hasn't compiled against the new lib it will be still using the old value
> >>>>>>> which will be within bounds. If it's picking up the higher new value from the lib it must
> >>>>>>> have been compiled against the lib so shouldn't have problems.
> >>>>>>
> >>>>>> You say there is no ABI issue because the application will be re-compiled
> >>>>>> for the updated library. Indeed, compilation fixes compatibility issues.
> >>>>>> But this is not relevant for ABI compatibility.
> >>>>>> ABI compatibility means we can upgrade the library without recompiling
> >>>>>> the application and it must work.
> >>>>>> You think it is a false positive because you assume the application
> >>>>>> "picks" the new value. I think you miss the case where the new value
> >>>>>> is returned by a function in the upgraded library.
> >>>>>>
> >>>>>>> There are also no structs on the API which contain arrays using this
> >>>>>>> for sizing, so I don't see an opportunity for an appl to have a
> >>>>>>> mismatch in memory addresses.
> >>>>>>
> >>>>>> Let me demonstrate where the API may "use" the new value
> >>>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
> >>>>>>
> >>>>>> Once upon a time a DPDK application counting the number of devices
> >>>>>> supporting each AEAD algo (in order to find the best supported algo).
> >>>>>> It is done in an array indexed by algo id:
> >>>>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
> >>>>>> The application is compiled with DPDK 19.11,
> >>>>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
> >>>>>> So the size of the application array aead_dev_count is 3.
> >>>>>> This binary is run with DPDK 20.02,
> >>>>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
> >>>>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
> >>>>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
> >>>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
> >>>>>> The application uses this value:
> >>>>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
> >>>>>> The application is crashing because of out of bound access.
> >>>>>
> >>>>> I'd say this is an example of bad written app.
> >>>>> It probably should check that returned by library value doesn't
> >>>>> exceed its internal array size.
> >>>>
> >>>> +1
> >>>>
> >>>> Application should ignore values >= MAX.
> >>>
> >>> Of course, blaming the API user is a lot easier than looking at the API.
> >>> Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
> >>> as the max value for the application.
> >>> Value ranges are part of the ABI compatibility contract.
> >>> It seems you expect the application developer to be aware that
> >>> DPDK could return a higher value, so the application should
> >>> check every enum values after calling an API. CRAZY.
> >>>
> >>> When we decide to announce an ABI compatibility and do some marketing,
> >>> everyone is OK. But when we need to really make our ABI compatible,
> >>> I see little or no effort. DISAPPOINTING.
> >>
> >> This is not to blame the user or to do less work, this is more sane approach
> >> that library provides the _END/_MAX value and application uses it as valid range
> >> check.
> >>
> >>>> Do you suggest we don't extend any enum or define between ABI breakage releases
> >>>> to be sure bad written applications not affected?
> >>>
> >>> I suggest we must consider not breaking any assumption made on the API.
> >>> Here we are breaking the enum range because nothing mentions _LIST_END
> >>> is not really the absolute end of the enum.
> >>> The solution is to make the change below in 20.02 + backport in 19.11.1:
> >>>
> >>> - _LIST_END
> >>> + _LIST_END, /* an ABI-compatible version may increase this value */
> >>> + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
> >>> };
> >>>
> >>
> >> What is the point of "_LIST_MAX" here?
> >
> > _LIST_MAX is range of value that DPDK can return in the ABI contract.
> > So the appplication can rely on the range 0.._LIST_MAX.
> >
> >> Application should know the "_LIST_END" of when it has been compiled for the
> >> valid range check. Next time it is compiled "_LIST_END" may be different value
> >> but same logic applies.
> >
> > No, ABI compatibility contract means you can compile your application
> > with DPDK 19.11.0 and run it with DPDK 20.02.
> > So _LIST_END comes from 19.11 and does not include ChachaPoly.
>
> That is what I mean, let me try to give a sample.
>
> DPDK19.11 returns, A=1, B=2, END=3
>
> Application compiled with DPDK19.11, it will process A, B and ignore anything ">= 3"
No, the application will not ignore anything ">=3" as I explained above,
and you blamed the application for it.
Nothing in the API says the application must filter value higher than 3,
because as of now, values higher than 3 are PMD bug.
> DPDK20.02 returns A=1, B=2, C=3, D=4, END=5
>
> Old application will still only will know/use A, B and can ignore when library
> sends C=3, D=4 etc...
>
>
> In above, if you add another limit as you suggested, like MAX=10 and ask
> application to use it,
>
> Application compiled with DPDK19.11 will be OK since library only sends A,B and
> application uses them.
>
> But with DPDK20.02 application may have problem, since library will be sending
> C=3, which is valid according to the check " <= MAX (10)", how application will
> know to ignore it.
Why application should ignore value C=3 with DPDK 20.02?
> So application should use _END to know the valid ones according it, if so what
> is the point of having _MAX.
>
>
> >> When "_LIST_END" is missing, application can't protect itself, in that case
> >> library should send only the values application knows when it is compiled, this
> >> means either we can't extend our enum/defines until next ABI breakage, or we
> >> need to do ABI versioning to the functions that returns an enum each time enum
> >> value extended.
> >
> > If we define _LIST_MAX as a bigger value than current _LIST_END,
> > we have some room to add values in between.
> >
> > If (as of now) we don't have _LIST_MAX room, then yes we must version
> > the functions returning the enum.
> > In this case, the proper solution is to implement
> > rte_cryptodev_info_get_v1911() so it filters out
> > RTE_CRYPTO_AEAD_CHACHA20_POLY1305 capability.
> > With this solution, an application compiled with DPDK 19.11 will keep
> > seeing the same range as before, while a 20.02 application could
> > see and use ChachaPoly.
> > This is another proposal that I was expecting from the crypto team,
> > instead of claiming there is no issue (and wasting precious time).
> >
> >
> >> I believe it is saner to provide _END/_MAX values to the application to use. And
> >> if required comment them to clarify the expected usage.
> >>
> >> But in above suggestion application can't use or rely on "_LIST_MAX", it doesn't
> >> mean anything to application.
> >
> > I don't understand what you mean. I think you misunderstood what is ABI compat.
> >
> >
> >>> Then *_LIST_END values could be ignored by libabigail with such a change.
> >>>
> >>> If such a patch is not done by tomorrow, I will have to revert
> >>> Chacha-Poly commits before 20.02-rc2, because
> >>>
> >>> 1/ LIST_END, without any comment, means "size of range"
> >>> 2/ we do not blame users for undocumented ABI changes
> >>> 3/ we respect the ABI compatibility contract
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-03 18:40 9% ` Thomas Monjalon
@ 2020-02-04 9:19 4% ` Ferruh Yigit
2020-02-04 9:45 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-02-04 9:19 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Ananyev, Konstantin, Akhil Goyal, Trahe, Fiona, David Marchand,
Anoob Joseph, Kusztal, ArkadiuszX, dev, Richardson, Bruce,
nhorman, Mcnamara, John, dodji, Andrew Rybchenko, aconole, bluca,
ktraynor
On 2/3/2020 6:40 PM, Thomas Monjalon wrote:
> 03/02/2020 18:40, Ferruh Yigit:
>> On 2/3/2020 5:09 PM, Thomas Monjalon wrote:
>>> 03/02/2020 10:30, Ferruh Yigit:
>>>> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
>>>>> 02/02/2020 14:05, Thomas Monjalon:
>>>>>> 31/01/2020 15:16, Trahe, Fiona:
>>>>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
>>>>>>>> 30/01/2020 17:09, Ferruh Yigit:
>>>>>>>>> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
>>>>>>>>>>
>>>>>>>>>> I believe these enums will be used only in case of ASYM case which is experimental.
>>>>>>>>>
>>>>>>>>> Independent from being experiment and not, this shouldn't be a problem, I think
>>>>>>>>> this is a false positive.
>>>>>>>>>
>>>>>>>>> The ABI break can happen when a struct has been shared between the application
>>>>>>>>> and the library (DPDK) and the layout of that memory know differently by
>>>>>>>>> application and the library.
>>>>>>>>>
>>>>>>>>> Here in all cases, there is no layout/size change.
>>>>>>>>>
>>>>>>>>> As to the value changes of the enums, since application compiled with old DPDK,
>>>>>>>>> it will know only up to '6', 7 and more means invalid to the application. So it
>>>>>>>>> won't send these values also it should ignore these values from library. Only
>>>>>>>>> consequence is old application won't able to use new features those new enums
>>>>>>>>> provide but that is expected/normal.
>>>>>>>>
>>>>>>>> If library give higher value than expected by the application,
>>>>>>>> if the application uses this value as array index,
>>>>>>>> there can be an access out of bounds.
>>>>>>>
>>>>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
>>>>>>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
>>>>>>> sense and I don't see how there can be an API breakage.
>>>>>>> So if an application hasn't compiled against the new lib it will be still using the old value
>>>>>>> which will be within bounds. If it's picking up the higher new value from the lib it must
>>>>>>> have been compiled against the lib so shouldn't have problems.
>>>>>>
>>>>>> You say there is no ABI issue because the application will be re-compiled
>>>>>> for the updated library. Indeed, compilation fixes compatibility issues.
>>>>>> But this is not relevant for ABI compatibility.
>>>>>> ABI compatibility means we can upgrade the library without recompiling
>>>>>> the application and it must work.
>>>>>> You think it is a false positive because you assume the application
>>>>>> "picks" the new value. I think you miss the case where the new value
>>>>>> is returned by a function in the upgraded library.
>>>>>>
>>>>>>> There are also no structs on the API which contain arrays using this
>>>>>>> for sizing, so I don't see an opportunity for an appl to have a
>>>>>>> mismatch in memory addresses.
>>>>>>
>>>>>> Let me demonstrate where the API may "use" the new value
>>>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
>>>>>>
>>>>>> Once upon a time a DPDK application counting the number of devices
>>>>>> supporting each AEAD algo (in order to find the best supported algo).
>>>>>> It is done in an array indexed by algo id:
>>>>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
>>>>>> The application is compiled with DPDK 19.11,
>>>>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
>>>>>> So the size of the application array aead_dev_count is 3.
>>>>>> This binary is run with DPDK 20.02,
>>>>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
>>>>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
>>>>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
>>>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
>>>>>> The application uses this value:
>>>>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
>>>>>> The application is crashing because of out of bound access.
>>>>>
>>>>> I'd say this is an example of bad written app.
>>>>> It probably should check that returned by library value doesn't
>>>>> exceed its internal array size.
>>>>
>>>> +1
>>>>
>>>> Application should ignore values >= MAX.
>>>
>>> Of course, blaming the API user is a lot easier than looking at the API.
>>> Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
>>> as the max value for the application.
>>> Value ranges are part of the ABI compatibility contract.
>>> It seems you expect the application developer to be aware that
>>> DPDK could return a higher value, so the application should
>>> check every enum values after calling an API. CRAZY.
>>>
>>> When we decide to announce an ABI compatibility and do some marketing,
>>> everyone is OK. But when we need to really make our ABI compatible,
>>> I see little or no effort. DISAPPOINTING.
>>
>> This is not to blame the user or to do less work, this is more sane approach
>> that library provides the _END/_MAX value and application uses it as valid range
>> check.
>>
>>>> Do you suggest we don't extend any enum or define between ABI breakage releases
>>>> to be sure bad written applications not affected?
>>>
>>> I suggest we must consider not breaking any assumption made on the API.
>>> Here we are breaking the enum range because nothing mentions _LIST_END
>>> is not really the absolute end of the enum.
>>> The solution is to make the change below in 20.02 + backport in 19.11.1:
>>>
>>> - _LIST_END
>>> + _LIST_END, /* an ABI-compatible version may increase this value */
>>> + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
>>> };
>>>
>>
>> What is the point of "_LIST_MAX" here?
>
> _LIST_MAX is range of value that DPDK can return in the ABI contract.
> So the appplication can rely on the range 0.._LIST_MAX.
>
>> Application should know the "_LIST_END" of when it has been compiled for the
>> valid range check. Next time it is compiled "_LIST_END" may be different value
>> but same logic applies.
>
> No, ABI compatibility contract means you can compile your application
> with DPDK 19.11.0 and run it with DPDK 20.02.
> So _LIST_END comes from 19.11 and does not include ChachaPoly.
That is what I mean, let me try to give a sample.
DPDK19.11 returns, A=1, B=2, END=3
Application compiled with DPDK19.11, it will process A, B and ignore anything ">= 3"
DPDK20.02 returns A=1, B=2, C=3, D=4, END=5
Old application will still only will know/use A, B and can ignore when library
sends C=3, D=4 etc...
In above, if you add another limit as you suggested, like MAX=10 and ask
application to use it,
Application compiled with DPDK19.11 will be OK since library only sends A,B and
application uses them.
But with DPDK20.02 application may have problem, since library will be sending
C=3, which is valid according to the check " <= MAX (10)", how application will
know to ignore it.
So application should use _END to know the valid ones according it, if so what
is the point of having _MAX.
>
>> When "_LIST_END" is missing, application can't protect itself, in that case
>> library should send only the values application knows when it is compiled, this
>> means either we can't extend our enum/defines until next ABI breakage, or we
>> need to do ABI versioning to the functions that returns an enum each time enum
>> value extended.
>
> If we define _LIST_MAX as a bigger value than current _LIST_END,
> we have some room to add values in between.
>
> If (as of now) we don't have _LIST_MAX room, then yes we must version
> the functions returning the enum.
> In this case, the proper solution is to implement
> rte_cryptodev_info_get_v1911() so it filters out
> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 capability.
> With this solution, an application compiled with DPDK 19.11 will keep
> seeing the same range as before, while a 20.02 application could
> see and use ChachaPoly.
> This is another proposal that I was expecting from the crypto team,
> instead of claiming there is no issue (and wasting precious time).
>
>
>> I believe it is saner to provide _END/_MAX values to the application to use. And
>> if required comment them to clarify the expected usage.
>>
>> But in above suggestion application can't use or rely on "_LIST_MAX", it doesn't
>> mean anything to application.
>
> I don't understand what you mean. I think you misunderstood what is ABI compat.
>
>
>>> Then *_LIST_END values could be ignored by libabigail with such a change.
>>>
>>> If such a patch is not done by tomorrow, I will have to revert
>>> Chacha-Poly commits before 20.02-rc2, because
>>>
>>> 1/ LIST_END, without any comment, means "size of range"
>>> 2/ we do not blame users for undocumented ABI changes
>>> 3/ we respect the ABI compatibility contract
>
>
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v7] eal: add manual probing option
@ 2020-02-03 22:21 3% ` Thomas Monjalon
2020-02-04 10:03 0% ` Gaetan Rivet
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-03 22:21 UTC (permalink / raw)
To: Gaetan Rivet, Pavan Nikhilesh Bhagavatula
Cc: dev, David Marchand, Vamsi Krishna Attunuru, Jerin Jacob Kollanukkaran
03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
> @David Marchand @thomas@monjalon.net
>
> Ping?
>
> Are there any more changes required for this patch? It's been in queue since last October.
Sorry we have not decided whether it is a good idea or not.
All changes related to probing are very sensitive,
and we know a big refactoring would be better than stacking
more and more options and corner cases.
As we are busy with ABI stability stuff, we did not allocate
enough time to properly think about this feature.
Please accept our apologies, and let's consider it as
a high priority for 20.05 cycle.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-03 18:55 7% ` Ray Kinsella
@ 2020-02-03 21:07 7% ` Thomas Monjalon
2020-02-04 9:46 8% ` Ferruh Yigit
2020-02-04 10:24 9% ` Thomas Monjalon
2020-02-04 9:51 4% ` David Marchand
1 sibling, 2 replies; 200+ results
From: Thomas Monjalon @ 2020-02-03 21:07 UTC (permalink / raw)
To: David Marchand, nhorman, bluca, ktraynor, Ray Kinsella
Cc: Ananyev, Konstantin, Akhil Goyal, Trahe, Fiona, Ferruh Yigit,
dev, Anoob Joseph, Kusztal, ArkadiuszX, Richardson, Bruce,
Mcnamara, John, dodji, Andrew Rybchenko, aconole
03/02/2020 19:55, Ray Kinsella:
> On 03/02/2020 17:34, Thomas Monjalon wrote:
> > 03/02/2020 18:09, Thomas Monjalon:
> >> 03/02/2020 10:30, Ferruh Yigit:
> >>> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
> >>>> 02/02/2020 14:05, Thomas Monjalon:
> >>>>> 31/01/2020 15:16, Trahe, Fiona:
> >>>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> >>>>>>> If library give higher value than expected by the application,
> >>>>>>> if the application uses this value as array index,
> >>>>>>> there can be an access out of bounds.
> >>>>>>
> >>>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
> >>>>>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
> >>>>>> sense and I don't see how there can be an API breakage.
> >>>>>> So if an application hasn't compiled against the new lib it will be still using the old value
> >>>>>> which will be within bounds. If it's picking up the higher new value from the lib it must
> >>>>>> have been compiled against the lib so shouldn't have problems.
> >>>>>
> >>>>> You say there is no ABI issue because the application will be re-compiled
> >>>>> for the updated library. Indeed, compilation fixes compatibility issues.
> >>>>> But this is not relevant for ABI compatibility.
> >>>>> ABI compatibility means we can upgrade the library without recompiling
> >>>>> the application and it must work.
> >>>>> You think it is a false positive because you assume the application
> >>>>> "picks" the new value. I think you miss the case where the new value
> >>>>> is returned by a function in the upgraded library.
> >>>>>
> >>>>>> There are also no structs on the API which contain arrays using this
> >>>>>> for sizing, so I don't see an opportunity for an appl to have a
> >>>>>> mismatch in memory addresses.
> >>>>>
> >>>>> Let me demonstrate where the API may "use" the new value
> >>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
> >>>>>
> >>>>> Once upon a time a DPDK application counting the number of devices
> >>>>> supporting each AEAD algo (in order to find the best supported algo).
> >>>>> It is done in an array indexed by algo id:
> >>>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
> >>>>> The application is compiled with DPDK 19.11,
> >>>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
> >>>>> So the size of the application array aead_dev_count is 3.
> >>>>> This binary is run with DPDK 20.02,
> >>>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
> >>>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
> >>>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
> >>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
> >>>>> The application uses this value:
> >>>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
> >>>>> The application is crashing because of out of bound access.
> >>>>
> >>>> I'd say this is an example of bad written app.
> >>>> It probably should check that returned by library value doesn't
> >>>> exceed its internal array size.
> >>>
> >>> +1
> >>>
> >>> Application should ignore values >= MAX.
> >>
> >> Of course, blaming the API user is a lot easier than looking at the API.
> >> Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
> >> as the max value for the application.
> >> Value ranges are part of the ABI compatibility contract.
> >> It seems you expect the application developer to be aware that
> >> DPDK could return a higher value, so the application should
> >> check every enum values after calling an API. CRAZY.
> >>
> >> When we decide to announce an ABI compatibility and do some marketing,
> >> everyone is OK. But when we need to really make our ABI compatible,
> >> I see little or no effort. DISAPPOINTING.
> >>
> >>> Do you suggest we don't extend any enum or define between ABI breakage releases
> >>> to be sure bad written applications not affected?
> >>
> >> I suggest we must consider not breaking any assumption made on the API.
> >> Here we are breaking the enum range because nothing mentions _LIST_END
> >> is not really the absolute end of the enum.
> >> The solution is to make the change below in 20.02 + backport in 19.11.1:
> >
> > Thinking twice, merging such change before 20.11 is breaking the
> > ABI assumption based on the API 19.11.0.
> > I ask the release maintainers (Luca, Kevin, David and me) and
> > the ABI maintainers (Neil and Ray) to vote for a or b solution:
> > a) add comment and LIST_MAX as below in 20.02 + 19.11.1
>
> That would still be an ABI breakage though right.
>
> > b) wait 20.11 and revert Chacha-Poly from 20.02
>
> Thanks for analysis above Fiona, Ferruh and all.
>
> That is a nasty one alright - there is no "good" answer here.
> I agree with Ferruh's sentiments overall, we should rethink this API for 20.11.
> Could do without an enumeration?
>
> There a c) though right.
> We could work around the issue by api versioning rte_cryptodev_info_get() and friends.
> So they only support/acknowledge the existence of Chacha-Poly for applications build against > 20.02.
I agree there is a c) as I proposed in another email:
http://mails.dpdk.org/archives/dev/2020-February/156919.html
"
In this case, the proper solution is to implement
rte_cryptodev_info_get_v1911() so it filters out
RTE_CRYPTO_AEAD_CHACHA20_POLY1305 capability.
With this solution, an application compiled with DPDK 19.11 will keep
seeing the same range as before, while a 20.02 application could
see and use ChachaPoly.
"
> It would be painful I know.
Not so painful in my opinion.
Just need to call rte_cryptodev_info_get() from
rte_cryptodev_info_get_v1911() and filter the value
in the 19.11 range: [0..AES_GCM].
> It would also mean that Chacha-Poly would only be available to
> those building against >= 20.02.
Yes exactly.
The addition of comments and LIST_MAX like below are still valid
to avoid versioning after 20.11.
> >> - _LIST_END
> >> + _LIST_END, /* an ABI-compatible version may increase this value */
> >> + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
> >> };
> >>
> >> Then *_LIST_END values could be ignored by libabigail with such a change.
In order to avoid ABI check complaining, the best is to completely
remove LIST_END in DPDK 20.11.
> >> If such a patch is not done by tomorrow, I will have to revert
> >> Chacha-Poly commits before 20.02-rc2, because
> >>
> >> 1/ LIST_END, without any comment, means "size of range"
> >> 2/ we do not blame users for undocumented ABI changes
> >> 3/ we respect the ABI compatibility contract
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-03 17:34 8% ` Thomas Monjalon
@ 2020-02-03 18:55 7% ` Ray Kinsella
2020-02-03 21:07 7% ` Thomas Monjalon
2020-02-04 9:51 4% ` David Marchand
0 siblings, 2 replies; 200+ results
From: Ray Kinsella @ 2020-02-03 18:55 UTC (permalink / raw)
To: Thomas Monjalon, David Marchand, nhorman, bluca, ktraynor
Cc: Ananyev, Konstantin, Akhil Goyal, Trahe, Fiona, Ferruh Yigit,
dev, Anoob Joseph, Kusztal, ArkadiuszX, Richardson, Bruce,
Mcnamara, John, dodji, Andrew Rybchenko, aconole
On 03/02/2020 17:34, Thomas Monjalon wrote:
> 03/02/2020 18:09, Thomas Monjalon:
>> 03/02/2020 10:30, Ferruh Yigit:
>>> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
>>>> 02/02/2020 14:05, Thomas Monjalon:
>>>>> 31/01/2020 15:16, Trahe, Fiona:
>>>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
>>>>>>> If library give higher value than expected by the application,
>>>>>>> if the application uses this value as array index,
>>>>>>> there can be an access out of bounds.
>>>>>>
>>>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
>>>>>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
>>>>>> sense and I don't see how there can be an API breakage.
>>>>>> So if an application hasn't compiled against the new lib it will be still using the old value
>>>>>> which will be within bounds. If it's picking up the higher new value from the lib it must
>>>>>> have been compiled against the lib so shouldn't have problems.
>>>>>
>>>>> You say there is no ABI issue because the application will be re-compiled
>>>>> for the updated library. Indeed, compilation fixes compatibility issues.
>>>>> But this is not relevant for ABI compatibility.
>>>>> ABI compatibility means we can upgrade the library without recompiling
>>>>> the application and it must work.
>>>>> You think it is a false positive because you assume the application
>>>>> "picks" the new value. I think you miss the case where the new value
>>>>> is returned by a function in the upgraded library.
>>>>>
>>>>>> There are also no structs on the API which contain arrays using this
>>>>>> for sizing, so I don't see an opportunity for an appl to have a
>>>>>> mismatch in memory addresses.
>>>>>
>>>>> Let me demonstrate where the API may "use" the new value
>>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
>>>>>
>>>>> Once upon a time a DPDK application counting the number of devices
>>>>> supporting each AEAD algo (in order to find the best supported algo).
>>>>> It is done in an array indexed by algo id:
>>>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
>>>>> The application is compiled with DPDK 19.11,
>>>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
>>>>> So the size of the application array aead_dev_count is 3.
>>>>> This binary is run with DPDK 20.02,
>>>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
>>>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
>>>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
>>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
>>>>> The application uses this value:
>>>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
>>>>> The application is crashing because of out of bound access.
>>>>
>>>> I'd say this is an example of bad written app.
>>>> It probably should check that returned by library value doesn't
>>>> exceed its internal array size.
>>>
>>> +1
>>>
>>> Application should ignore values >= MAX.
>>
>> Of course, blaming the API user is a lot easier than looking at the API.
>> Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
>> as the max value for the application.
>> Value ranges are part of the ABI compatibility contract.
>> It seems you expect the application developer to be aware that
>> DPDK could return a higher value, so the application should
>> check every enum values after calling an API. CRAZY.
>>
>> When we decide to announce an ABI compatibility and do some marketing,
>> everyone is OK. But when we need to really make our ABI compatible,
>> I see little or no effort. DISAPPOINTING.
>>
>>> Do you suggest we don't extend any enum or define between ABI breakage releases
>>> to be sure bad written applications not affected?
>>
>> I suggest we must consider not breaking any assumption made on the API.
>> Here we are breaking the enum range because nothing mentions _LIST_END
>> is not really the absolute end of the enum.
>> The solution is to make the change below in 20.02 + backport in 19.11.1:
>
> Thinking twice, merging such change before 20.11 is breaking the
> ABI assumption based on the API 19.11.0.
> I ask the release maintainers (Luca, Kevin, David and me) and
> the ABI maintainers (Neil and Ray) to vote for a or b solution:
> a) add comment and LIST_MAX as below in 20.02 + 19.11.1
That would still be an ABI breakage though right.
> b) wait 20.11 and revert Chacha-Poly from 20.02
Thanks for analysis above Fiona, Ferruh and all.
That is a nasty one alright - there is no "good" answer here.
I agree with Ferruh's sentiments overall, we should rethink this API for 20.11.
Could do without an enumeration?
There a c) though right.
We could work around the issue by api versioning rte_cryptodev_info_get() and friends.
So they only support/acknowledge the existence of Chacha-Poly for applications build against > 20.02.
It would be painful I know.
It would also mean that Chacha-Poly would only be available to those building against >= 20.02.
>
>
>> - _LIST_END
>> + _LIST_END, /* an ABI-compatible version may increase this value */
>> + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
>> };
>>
>> Then *_LIST_END values could be ignored by libabigail with such a change.
>>
>> If such a patch is not done by tomorrow, I will have to revert
>> Chacha-Poly commits before 20.02-rc2, because
>>
>> 1/ LIST_END, without any comment, means "size of range"
>> 2/ we do not blame users for undocumented ABI changes
>> 3/ we respect the ABI compatibility contract
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-03 17:40 8% ` Ferruh Yigit
@ 2020-02-03 18:40 9% ` Thomas Monjalon
2020-02-04 9:19 4% ` Ferruh Yigit
2020-02-04 10:16 9% ` Akhil Goyal
1 sibling, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-03 18:40 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Ananyev, Konstantin, Akhil Goyal, Trahe, Fiona, David Marchand,
Anoob Joseph, Kusztal, ArkadiuszX, dev, Richardson, Bruce,
nhorman, Mcnamara, John, dodji, Andrew Rybchenko, aconole, bluca,
ktraynor
03/02/2020 18:40, Ferruh Yigit:
> On 2/3/2020 5:09 PM, Thomas Monjalon wrote:
> > 03/02/2020 10:30, Ferruh Yigit:
> >> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
> >>> 02/02/2020 14:05, Thomas Monjalon:
> >>>> 31/01/2020 15:16, Trahe, Fiona:
> >>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> >>>>>> 30/01/2020 17:09, Ferruh Yigit:
> >>>>>>> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
> >>>>>>>>
> >>>>>>>> I believe these enums will be used only in case of ASYM case which is experimental.
> >>>>>>>
> >>>>>>> Independent from being experiment and not, this shouldn't be a problem, I think
> >>>>>>> this is a false positive.
> >>>>>>>
> >>>>>>> The ABI break can happen when a struct has been shared between the application
> >>>>>>> and the library (DPDK) and the layout of that memory know differently by
> >>>>>>> application and the library.
> >>>>>>>
> >>>>>>> Here in all cases, there is no layout/size change.
> >>>>>>>
> >>>>>>> As to the value changes of the enums, since application compiled with old DPDK,
> >>>>>>> it will know only up to '6', 7 and more means invalid to the application. So it
> >>>>>>> won't send these values also it should ignore these values from library. Only
> >>>>>>> consequence is old application won't able to use new features those new enums
> >>>>>>> provide but that is expected/normal.
> >>>>>>
> >>>>>> If library give higher value than expected by the application,
> >>>>>> if the application uses this value as array index,
> >>>>>> there can be an access out of bounds.
> >>>>>
> >>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
> >>>>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
> >>>>> sense and I don't see how there can be an API breakage.
> >>>>> So if an application hasn't compiled against the new lib it will be still using the old value
> >>>>> which will be within bounds. If it's picking up the higher new value from the lib it must
> >>>>> have been compiled against the lib so shouldn't have problems.
> >>>>
> >>>> You say there is no ABI issue because the application will be re-compiled
> >>>> for the updated library. Indeed, compilation fixes compatibility issues.
> >>>> But this is not relevant for ABI compatibility.
> >>>> ABI compatibility means we can upgrade the library without recompiling
> >>>> the application and it must work.
> >>>> You think it is a false positive because you assume the application
> >>>> "picks" the new value. I think you miss the case where the new value
> >>>> is returned by a function in the upgraded library.
> >>>>
> >>>>> There are also no structs on the API which contain arrays using this
> >>>>> for sizing, so I don't see an opportunity for an appl to have a
> >>>>> mismatch in memory addresses.
> >>>>
> >>>> Let me demonstrate where the API may "use" the new value
> >>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
> >>>>
> >>>> Once upon a time a DPDK application counting the number of devices
> >>>> supporting each AEAD algo (in order to find the best supported algo).
> >>>> It is done in an array indexed by algo id:
> >>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
> >>>> The application is compiled with DPDK 19.11,
> >>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
> >>>> So the size of the application array aead_dev_count is 3.
> >>>> This binary is run with DPDK 20.02,
> >>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
> >>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
> >>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
> >>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
> >>>> The application uses this value:
> >>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
> >>>> The application is crashing because of out of bound access.
> >>>
> >>> I'd say this is an example of bad written app.
> >>> It probably should check that returned by library value doesn't
> >>> exceed its internal array size.
> >>
> >> +1
> >>
> >> Application should ignore values >= MAX.
> >
> > Of course, blaming the API user is a lot easier than looking at the API.
> > Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
> > as the max value for the application.
> > Value ranges are part of the ABI compatibility contract.
> > It seems you expect the application developer to be aware that
> > DPDK could return a higher value, so the application should
> > check every enum values after calling an API. CRAZY.
> >
> > When we decide to announce an ABI compatibility and do some marketing,
> > everyone is OK. But when we need to really make our ABI compatible,
> > I see little or no effort. DISAPPOINTING.
>
> This is not to blame the user or to do less work, this is more sane approach
> that library provides the _END/_MAX value and application uses it as valid range
> check.
>
> >> Do you suggest we don't extend any enum or define between ABI breakage releases
> >> to be sure bad written applications not affected?
> >
> > I suggest we must consider not breaking any assumption made on the API.
> > Here we are breaking the enum range because nothing mentions _LIST_END
> > is not really the absolute end of the enum.
> > The solution is to make the change below in 20.02 + backport in 19.11.1:
> >
> > - _LIST_END
> > + _LIST_END, /* an ABI-compatible version may increase this value */
> > + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
> > };
> >
>
> What is the point of "_LIST_MAX" here?
_LIST_MAX is range of value that DPDK can return in the ABI contract.
So the appplication can rely on the range 0.._LIST_MAX.
> Application should know the "_LIST_END" of when it has been compiled for the
> valid range check. Next time it is compiled "_LIST_END" may be different value
> but same logic applies.
No, ABI compatibility contract means you can compile your application
with DPDK 19.11.0 and run it with DPDK 20.02.
So _LIST_END comes from 19.11 and does not include ChachaPoly.
> When "_LIST_END" is missing, application can't protect itself, in that case
> library should send only the values application knows when it is compiled, this
> means either we can't extend our enum/defines until next ABI breakage, or we
> need to do ABI versioning to the functions that returns an enum each time enum
> value extended.
If we define _LIST_MAX as a bigger value than current _LIST_END,
we have some room to add values in between.
If (as of now) we don't have _LIST_MAX room, then yes we must version
the functions returning the enum.
In this case, the proper solution is to implement
rte_cryptodev_info_get_v1911() so it filters out
RTE_CRYPTO_AEAD_CHACHA20_POLY1305 capability.
With this solution, an application compiled with DPDK 19.11 will keep
seeing the same range as before, while a 20.02 application could
see and use ChachaPoly.
This is another proposal that I was expecting from the crypto team,
instead of claiming there is no issue (and wasting precious time).
> I believe it is saner to provide _END/_MAX values to the application to use. And
> if required comment them to clarify the expected usage.
>
> But in above suggestion application can't use or rely on "_LIST_MAX", it doesn't
> mean anything to application.
I don't understand what you mean. I think you misunderstood what is ABI compat.
> > Then *_LIST_END values could be ignored by libabigail with such a change.
> >
> > If such a patch is not done by tomorrow, I will have to revert
> > Chacha-Poly commits before 20.02-rc2, because
> >
> > 1/ LIST_END, without any comment, means "size of range"
> > 2/ we do not blame users for undocumented ABI changes
> > 3/ we respect the ABI compatibility contract
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-03 17:09 9% ` Thomas Monjalon
2020-02-03 17:34 8% ` Thomas Monjalon
@ 2020-02-03 17:40 8% ` Ferruh Yigit
2020-02-03 18:40 9% ` Thomas Monjalon
2020-02-04 10:16 9% ` Akhil Goyal
1 sibling, 2 replies; 200+ results
From: Ferruh Yigit @ 2020-02-03 17:40 UTC (permalink / raw)
To: Thomas Monjalon, Ananyev, Konstantin, Akhil Goyal, Trahe, Fiona
Cc: David Marchand, Anoob Joseph, Kusztal, ArkadiuszX, dev,
Richardson, Bruce, nhorman, Mcnamara, John, dodji,
Andrew Rybchenko, aconole, bluca, ktraynor
On 2/3/2020 5:09 PM, Thomas Monjalon wrote:
> 03/02/2020 10:30, Ferruh Yigit:
>> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
>>> 02/02/2020 14:05, Thomas Monjalon:
>>>> 31/01/2020 15:16, Trahe, Fiona:
>>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
>>>>>> 30/01/2020 17:09, Ferruh Yigit:
>>>>>>> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
>>>>>>>>
>>>>>>>> I believe these enums will be used only in case of ASYM case which is experimental.
>>>>>>>
>>>>>>> Independent from being experiment and not, this shouldn't be a problem, I think
>>>>>>> this is a false positive.
>>>>>>>
>>>>>>> The ABI break can happen when a struct has been shared between the application
>>>>>>> and the library (DPDK) and the layout of that memory know differently by
>>>>>>> application and the library.
>>>>>>>
>>>>>>> Here in all cases, there is no layout/size change.
>>>>>>>
>>>>>>> As to the value changes of the enums, since application compiled with old DPDK,
>>>>>>> it will know only up to '6', 7 and more means invalid to the application. So it
>>>>>>> won't send these values also it should ignore these values from library. Only
>>>>>>> consequence is old application won't able to use new features those new enums
>>>>>>> provide but that is expected/normal.
>>>>>>
>>>>>> If library give higher value than expected by the application,
>>>>>> if the application uses this value as array index,
>>>>>> there can be an access out of bounds.
>>>>>
>>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
>>>>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
>>>>> sense and I don't see how there can be an API breakage.
>>>>> So if an application hasn't compiled against the new lib it will be still using the old value
>>>>> which will be within bounds. If it's picking up the higher new value from the lib it must
>>>>> have been compiled against the lib so shouldn't have problems.
>>>>
>>>> You say there is no ABI issue because the application will be re-compiled
>>>> for the updated library. Indeed, compilation fixes compatibility issues.
>>>> But this is not relevant for ABI compatibility.
>>>> ABI compatibility means we can upgrade the library without recompiling
>>>> the application and it must work.
>>>> You think it is a false positive because you assume the application
>>>> "picks" the new value. I think you miss the case where the new value
>>>> is returned by a function in the upgraded library.
>>>>
>>>>> There are also no structs on the API which contain arrays using this
>>>>> for sizing, so I don't see an opportunity for an appl to have a
>>>>> mismatch in memory addresses.
>>>>
>>>> Let me demonstrate where the API may "use" the new value
>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
>>>>
>>>> Once upon a time a DPDK application counting the number of devices
>>>> supporting each AEAD algo (in order to find the best supported algo).
>>>> It is done in an array indexed by algo id:
>>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
>>>> The application is compiled with DPDK 19.11,
>>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
>>>> So the size of the application array aead_dev_count is 3.
>>>> This binary is run with DPDK 20.02,
>>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
>>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
>>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
>>>> The application uses this value:
>>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
>>>> The application is crashing because of out of bound access.
>>>
>>> I'd say this is an example of bad written app.
>>> It probably should check that returned by library value doesn't
>>> exceed its internal array size.
>>
>> +1
>>
>> Application should ignore values >= MAX.
>
> Of course, blaming the API user is a lot easier than looking at the API.
> Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
> as the max value for the application.
> Value ranges are part of the ABI compatibility contract.
> It seems you expect the application developer to be aware that
> DPDK could return a higher value, so the application should
> check every enum values after calling an API. CRAZY.
>
> When we decide to announce an ABI compatibility and do some marketing,
> everyone is OK. But when we need to really make our ABI compatible,
> I see little or no effort. DISAPPOINTING.
This is not to blame the user or to do less work, this is more sane approach
that library provides the _END/_MAX value and application uses it as valid range
check.
>
>> Do you suggest we don't extend any enum or define between ABI breakage releases
>> to be sure bad written applications not affected?
>
> I suggest we must consider not breaking any assumption made on the API.
> Here we are breaking the enum range because nothing mentions _LIST_END
> is not really the absolute end of the enum.
> The solution is to make the change below in 20.02 + backport in 19.11.1:
>
> - _LIST_END
> + _LIST_END, /* an ABI-compatible version may increase this value */
> + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
> };
>
What is the point of "_LIST_MAX" here?
Application should know the "_LIST_END" of when it has been compiled for the
valid range check. Next time it is compiled "_LIST_END" may be different value
but same logic applies.
When "_LIST_END" is missing, application can't protect itself, in that case
library should send only the values application knows when it is compiled, this
means either we can't extend our enum/defines until next ABI breakage, or we
need to do ABI versioning to the functions that returns an enum each time enum
value extended.
I believe it is saner to provide _END/_MAX values to the application to use. And
if required comment them to clarify the expected usage.
But in above suggestion application can't use or rely on "_LIST_MAX", it doesn't
mean anything to application.
> Then *_LIST_END values could be ignored by libabigail with such a change.
>
> If such a patch is not done by tomorrow, I will have to revert
> Chacha-Poly commits before 20.02-rc2, because
>
> 1/ LIST_END, without any comment, means "size of range"
> 2/ we do not blame users for undocumented ABI changes
> 3/ we respect the ABI compatibility contract
>
>
^ permalink raw reply [relevance 8%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-03 17:09 9% ` Thomas Monjalon
@ 2020-02-03 17:34 8% ` Thomas Monjalon
2020-02-03 18:55 7% ` Ray Kinsella
2020-02-03 17:40 8% ` Ferruh Yigit
1 sibling, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-03 17:34 UTC (permalink / raw)
To: David Marchand, nhorman, bluca, ktraynor, Ray Kinsella
Cc: Ananyev, Konstantin, Akhil Goyal, Trahe, Fiona, Ferruh Yigit,
dev, Anoob Joseph, Kusztal, ArkadiuszX, dev, Richardson, Bruce,
Mcnamara, John, dodji, Andrew Rybchenko, aconole
03/02/2020 18:09, Thomas Monjalon:
> 03/02/2020 10:30, Ferruh Yigit:
> > On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
> > > 02/02/2020 14:05, Thomas Monjalon:
> > >> 31/01/2020 15:16, Trahe, Fiona:
> > >>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> > >>>> If library give higher value than expected by the application,
> > >>>> if the application uses this value as array index,
> > >>>> there can be an access out of bounds.
> > >>>
> > >>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
> > >>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
> > >>> sense and I don't see how there can be an API breakage.
> > >>> So if an application hasn't compiled against the new lib it will be still using the old value
> > >>> which will be within bounds. If it's picking up the higher new value from the lib it must
> > >>> have been compiled against the lib so shouldn't have problems.
> > >>
> > >> You say there is no ABI issue because the application will be re-compiled
> > >> for the updated library. Indeed, compilation fixes compatibility issues.
> > >> But this is not relevant for ABI compatibility.
> > >> ABI compatibility means we can upgrade the library without recompiling
> > >> the application and it must work.
> > >> You think it is a false positive because you assume the application
> > >> "picks" the new value. I think you miss the case where the new value
> > >> is returned by a function in the upgraded library.
> > >>
> > >>> There are also no structs on the API which contain arrays using this
> > >>> for sizing, so I don't see an opportunity for an appl to have a
> > >>> mismatch in memory addresses.
> > >>
> > >> Let me demonstrate where the API may "use" the new value
> > >> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
> > >>
> > >> Once upon a time a DPDK application counting the number of devices
> > >> supporting each AEAD algo (in order to find the best supported algo).
> > >> It is done in an array indexed by algo id:
> > >> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
> > >> The application is compiled with DPDK 19.11,
> > >> where RTE_CRYPTO_AEAD_LIST_END = 3.
> > >> So the size of the application array aead_dev_count is 3.
> > >> This binary is run with DPDK 20.02,
> > >> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
> > >> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
> > >> rte_cryptodev_info.capabilities.sym.aead.algo is set to
> > >> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
> > >> The application uses this value:
> > >> ++ aead_dev_count[info.capabilities.sym.aead.algo];
> > >> The application is crashing because of out of bound access.
> > >
> > > I'd say this is an example of bad written app.
> > > It probably should check that returned by library value doesn't
> > > exceed its internal array size.
> >
> > +1
> >
> > Application should ignore values >= MAX.
>
> Of course, blaming the API user is a lot easier than looking at the API.
> Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
> as the max value for the application.
> Value ranges are part of the ABI compatibility contract.
> It seems you expect the application developer to be aware that
> DPDK could return a higher value, so the application should
> check every enum values after calling an API. CRAZY.
>
> When we decide to announce an ABI compatibility and do some marketing,
> everyone is OK. But when we need to really make our ABI compatible,
> I see little or no effort. DISAPPOINTING.
>
> > Do you suggest we don't extend any enum or define between ABI breakage releases
> > to be sure bad written applications not affected?
>
> I suggest we must consider not breaking any assumption made on the API.
> Here we are breaking the enum range because nothing mentions _LIST_END
> is not really the absolute end of the enum.
> The solution is to make the change below in 20.02 + backport in 19.11.1:
Thinking twice, merging such change before 20.11 is breaking the
ABI assumption based on the API 19.11.0.
I ask the release maintainers (Luca, Kevin, David and me) and
the ABI maintainers (Neil and Ray) to vote for a or b solution:
a) add comment and LIST_MAX as below in 20.02 + 19.11.1
b) wait 20.11 and revert Chacha-Poly from 20.02
> - _LIST_END
> + _LIST_END, /* an ABI-compatible version may increase this value */
> + _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
> };
>
> Then *_LIST_END values could be ignored by libabigail with such a change.
>
> If such a patch is not done by tomorrow, I will have to revert
> Chacha-Poly commits before 20.02-rc2, because
>
> 1/ LIST_END, without any comment, means "size of range"
> 2/ we do not blame users for undocumented ABI changes
> 3/ we respect the ABI compatibility contract
^ permalink raw reply [relevance 8%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-03 9:30 7% ` Ferruh Yigit
2020-02-03 11:50 9% ` Neil Horman
@ 2020-02-03 17:09 9% ` Thomas Monjalon
2020-02-03 17:34 8% ` Thomas Monjalon
2020-02-03 17:40 8% ` Ferruh Yigit
1 sibling, 2 replies; 200+ results
From: Thomas Monjalon @ 2020-02-03 17:09 UTC (permalink / raw)
To: Ananyev, Konstantin, Akhil Goyal, Trahe, Fiona, Ferruh Yigit
Cc: David Marchand, Anoob Joseph, Kusztal, ArkadiuszX, dev,
Richardson, Bruce, nhorman, Mcnamara, John, dodji,
Andrew Rybchenko, aconole, bluca, ktraynor
03/02/2020 10:30, Ferruh Yigit:
> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
> > 02/02/2020 14:05, Thomas Monjalon:
> >> 31/01/2020 15:16, Trahe, Fiona:
> >>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> >>>> 30/01/2020 17:09, Ferruh Yigit:
> >>>>> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
> >>>>>>
> >>>>>> I believe these enums will be used only in case of ASYM case which is experimental.
> >>>>>
> >>>>> Independent from being experiment and not, this shouldn't be a problem, I think
> >>>>> this is a false positive.
> >>>>>
> >>>>> The ABI break can happen when a struct has been shared between the application
> >>>>> and the library (DPDK) and the layout of that memory know differently by
> >>>>> application and the library.
> >>>>>
> >>>>> Here in all cases, there is no layout/size change.
> >>>>>
> >>>>> As to the value changes of the enums, since application compiled with old DPDK,
> >>>>> it will know only up to '6', 7 and more means invalid to the application. So it
> >>>>> won't send these values also it should ignore these values from library. Only
> >>>>> consequence is old application won't able to use new features those new enums
> >>>>> provide but that is expected/normal.
> >>>>
> >>>> If library give higher value than expected by the application,
> >>>> if the application uses this value as array index,
> >>>> there can be an access out of bounds.
> >>>
> >>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
> >>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
> >>> sense and I don't see how there can be an API breakage.
> >>> So if an application hasn't compiled against the new lib it will be still using the old value
> >>> which will be within bounds. If it's picking up the higher new value from the lib it must
> >>> have been compiled against the lib so shouldn't have problems.
> >>
> >> You say there is no ABI issue because the application will be re-compiled
> >> for the updated library. Indeed, compilation fixes compatibility issues.
> >> But this is not relevant for ABI compatibility.
> >> ABI compatibility means we can upgrade the library without recompiling
> >> the application and it must work.
> >> You think it is a false positive because you assume the application
> >> "picks" the new value. I think you miss the case where the new value
> >> is returned by a function in the upgraded library.
> >>
> >>> There are also no structs on the API which contain arrays using this
> >>> for sizing, so I don't see an opportunity for an appl to have a
> >>> mismatch in memory addresses.
> >>
> >> Let me demonstrate where the API may "use" the new value
> >> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
> >>
> >> Once upon a time a DPDK application counting the number of devices
> >> supporting each AEAD algo (in order to find the best supported algo).
> >> It is done in an array indexed by algo id:
> >> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
> >> The application is compiled with DPDK 19.11,
> >> where RTE_CRYPTO_AEAD_LIST_END = 3.
> >> So the size of the application array aead_dev_count is 3.
> >> This binary is run with DPDK 20.02,
> >> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
> >> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
> >> rte_cryptodev_info.capabilities.sym.aead.algo is set to
> >> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
> >> The application uses this value:
> >> ++ aead_dev_count[info.capabilities.sym.aead.algo];
> >> The application is crashing because of out of bound access.
> >
> > I'd say this is an example of bad written app.
> > It probably should check that returned by library value doesn't
> > exceed its internal array size.
>
> +1
>
> Application should ignore values >= MAX.
Of course, blaming the API user is a lot easier than looking at the API.
Here the API has RTE_CRYPTO_AEAD_LIST_END which can be understood
as the max value for the application.
Value ranges are part of the ABI compatibility contract.
It seems you expect the application developer to be aware that
DPDK could return a higher value, so the application should
check every enum values after calling an API. CRAZY.
When we decide to announce an ABI compatibility and do some marketing,
everyone is OK. But when we need to really make our ABI compatible,
I see little or no effort. DISAPPOINTING.
> Do you suggest we don't extend any enum or define between ABI breakage releases
> to be sure bad written applications not affected?
I suggest we must consider not breaking any assumption made on the API.
Here we are breaking the enum range because nothing mentions _LIST_END
is not really the absolute end of the enum.
The solution is to make the change below in 20.02 + backport in 19.11.1:
- _LIST_END
+ _LIST_END, /* an ABI-compatible version may increase this value */
+ _LIST_MAX = _LIST_END + 42 /* room for ABI-compatible additions */
};
Then *_LIST_END values could be ignored by libabigail with such a change.
If such a patch is not done by tomorrow, I will have to revert
Chacha-Poly commits before 20.02-rc2, because
1/ LIST_END, without any comment, means "size of range"
2/ we do not blame users for undocumented ABI changes
3/ we respect the ABI compatibility contract
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-03 13:09 7% ` Ferruh Yigit
@ 2020-02-03 14:00 7% ` Dodji Seketeli
2020-02-03 14:46 4% ` Ferruh Yigit
2020-02-03 15:08 7% ` Trahe, Fiona
1 sibling, 1 reply; 200+ results
From: Dodji Seketeli @ 2020-02-03 14:00 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Neil Horman, Ananyev, Konstantin, Thomas Monjalon, Akhil Goyal,
Trahe, Fiona, David Marchand, Anoob Joseph, Kusztal, ArkadiuszX,
dev, Richardson, Bruce, Mcnamara, John, Andrew Rybchenko,
aconole
Hello,
Ferruh Yigit <ferruh.yigit@intel.com> a écrit:
[...]
> +1 the change/shuffle of the existing values are problematic, but we don't have
> it in this case.
Right.
[...]
> The concern is when there are cases we can waive, we can't directly rely on the
> tool and automate it. These indicators good for improving the code, but not good
> to use it as build time checker.
Well, it depends. The tooling as it is today have the capability to
automatically "waive" some classes of A{B,P}I change reports that you
guys (the developers) deem harmless, in the context of your project.
For instance, in the precise case of interest here, one could define a
"suppression specification" to teach the ABI verifier that, for the enum
rte_crypto_asym_xform_type, the only enumerator which numerical value is
allowed to change is the one named RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END.
The content of the suppression specification file would look like:
[suppress_type]
# So, in practise, this rule is to allow adding enumerators
# only to the of the the rte_crypto_asym_xform_type enum,
# right before the RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
# enumerator which is meant to always be the last enumerator.
type_kind = enum
name = rte_crypto_asym_xform_type
changed_enumerators = RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
This way, you can hopefully reduce the surface of the changes you want
to see reported, tailored in a way that is specific to your project.
This should hopefully bring the system closer to a state that would
allow you guys to having something that is automated enough to have it
be triggered at build time.
And if there is some sensibly needed tweaking that the libabigail
tooling doesn't allow you guys to do right away, I'd be happy to hear
about it and try to add the functionality to the framework for you guys.
> Is there any way to reduce the failure only to definite ABI breakages?
I hope my comment above somewhat answers this question of yours. If it
does not, please tell me.
Cheers,
--
Dodji
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-03 13:09 7% ` Ferruh Yigit
2020-02-03 14:00 7% ` Dodji Seketeli
@ 2020-02-03 15:08 7% ` Trahe, Fiona
1 sibling, 0 replies; 200+ results
From: Trahe, Fiona @ 2020-02-03 15:08 UTC (permalink / raw)
To: Yigit, Ferruh, Neil Horman
Cc: Ananyev, Konstantin, Thomas Monjalon, Akhil Goyal,
David Marchand, Anoob Joseph, Kusztal, ArkadiuszX, dev,
Richardson, Bruce, Mcnamara, John, dodji, Andrew Rybchenko,
aconole, Trahe, Fiona
> >
> > These sorts of changes are the type that we could collectively waive in terms of
> > ABI checking, as they should be ok, but the errors from libabigail should be
> > taken as an indicator that this API could be rewritten (for example by removing
> > the abi entirely, and adding an API call that returns an array of instance name
> > and ids), so that changes of the above sort arent required.
>
> We can spend more time on it, but I can't see for now how to escape returning
> enumaration as indication of type, and this looks legitimate sage as long as
> other side verifies the received value is valid in the type range.
[Fiona] Regarding re-work to make the original code more robust to ABI breakage
One option would be to remove LIST_END from the enum, but keep the enum and allow appending values to it.
Instead of LIST_END have a static var keeping track of the MAX_NUM_AEAD_ALGOS
and an API call rte_cryptodev_get_max_aead_algos() forcing an application to dynamically size any array to accommodate any new values.
The API is safer I think - but there are other pitfalls with this approach - the MAX can more easily get out-of-sync with the enum.
And the application still needs to safely handle values it doesn't recognise.
Anyone think this is a better way?
I still think the best solution is to suppress changes to the LIST_END element.
^ permalink raw reply [relevance 7%]
* [dpdk-dev] Minutes of Technical Board Meeting, 2020-01-15
@ 2020-02-03 14:53 5% Maxime Coquelin
0 siblings, 0 replies; 200+ results
From: Maxime Coquelin @ 2020-02-03 14:53 UTC (permalink / raw)
To: DPDK Techboard, dev
Minutes of Technical Board Meeting, 2020-01-15
Members Attending
-----------------
- Ferruh
- Kevin
- Bruce
- Konstantin
- Thomas
- Hemant
- Maxime (Chair)
- Olivier
- Stephen
- Jerin
- Honnappa
NOTE: The technical board meetings every second Wednesday on IRC
channel #dpdk-board, at 3pm UTC. Meetings are public and DPDK community
members are welcome to attend.
NOTE: Next meeting will be on Wednesday 2020-01-29 @3pm UTC, and will be
chaired by Olivier.
1) Estimation of the rework needed for the new vDPA devices class
- Matan already posted the rework, including moving IFC driver
- Main driver impacted will be Mellanox
- LTSes maintainers are fine with the rework
- Thomas applied the rework to master branch, so that Matan can post
the Mellanox vDPA series
2) Action request for Thomas to add approved e-mail IDs to the security
pre-release mailing list
- Done by Thomas
3) UNH Lab decisions
- Technical Board proposed to have member companies to assign people to
add more tests in UNH Lab.
- The Governing Board approved the proposal for one quarter.
- Top contributing companies will be asked to assign one contributor
for 20% FTE.
- Thomas volunteered to manage this trial
4) Kubernetes integration status
- Proposal to write a blog post about DPDK status in Kubernetes and
plans
- Maxime volunteered to write this blog post
- Draft review target on 2020-02-12 Techboard meeting
5) ABI check tooling
- Question about where/whether to store ABI dumps
- Techboard agrees it should not be stored in main DPDK git repo
- Techboard voted to have the ABI check being done by the CI with
scripts provided in DPDK git repo, vs. ABI check done at build time
- No decision yet by the Techboard on whether the ABI dumps for the
releases should be stored somewhere, or generated each time
^ permalink raw reply [relevance 5%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-03 14:00 7% ` Dodji Seketeli
@ 2020-02-03 14:46 4% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-02-03 14:46 UTC (permalink / raw)
To: Dodji Seketeli
Cc: Neil Horman, Ananyev, Konstantin, Thomas Monjalon, Akhil Goyal,
Trahe, Fiona, David Marchand, Anoob Joseph, Kusztal, ArkadiuszX,
dev, Richardson, Bruce, Mcnamara, John, Andrew Rybchenko,
aconole
On 2/3/2020 2:00 PM, Dodji Seketeli wrote:
> Hello,
>
> Ferruh Yigit <ferruh.yigit@intel.com> a écrit:
>
> [...]
>
>> +1 the change/shuffle of the existing values are problematic, but we don't have
>> it in this case.
>
> Right.
>
> [...]
>
>> The concern is when there are cases we can waive, we can't directly rely on the
>> tool and automate it. These indicators good for improving the code, but not good
>> to use it as build time checker.
>
> Well, it depends. The tooling as it is today have the capability to
> automatically "waive" some classes of A{B,P}I change reports that you
> guys (the developers) deem harmless, in the context of your project.
>
> For instance, in the precise case of interest here, one could define a
> "suppression specification" to teach the ABI verifier that, for the enum
> rte_crypto_asym_xform_type, the only enumerator which numerical value is
> allowed to change is the one named RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END.
>
> The content of the suppression specification file would look like:
>
> [suppress_type]
> # So, in practise, this rule is to allow adding enumerators
> # only to the of the the rte_crypto_asym_xform_type enum,
> # right before the RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
> # enumerator which is meant to always be the last enumerator.
> type_kind = enum
> name = rte_crypto_asym_xform_type
> changed_enumerators = RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
>
> This way, you can hopefully reduce the surface of the changes you want
> to see reported, tailored in a way that is specific to your project.
> This should hopefully bring the system closer to a state that would
> allow you guys to having something that is automated enough to have it
> be triggered at build time.
Thanks, at least this provides a way to silence the warnings not an issue for us
as we hit them.
Is there a more global, don't warn on new enums kind of option?
Although I assume this is not possible since _END or _MAX enum value will be
changing and tool can't know its usage and will report the change.
>
> And if there is some sensibly needed tweaking that the libabigail
> tooling doesn't allow you guys to do right away, I'd be happy to hear
> about it and try to add the functionality to the framework for you guys.
>
>> Is there any way to reduce the failure only to definite ABI breakages?
>
> I hope my comment above somewhat answers this question of yours. If it
> does not, please tell me.
>
> Cheers,
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-03 11:50 9% ` Neil Horman
@ 2020-02-03 13:09 7% ` Ferruh Yigit
2020-02-03 14:00 7% ` Dodji Seketeli
2020-02-03 15:08 7% ` Trahe, Fiona
0 siblings, 2 replies; 200+ results
From: Ferruh Yigit @ 2020-02-03 13:09 UTC (permalink / raw)
To: Neil Horman
Cc: Ananyev, Konstantin, Thomas Monjalon, Akhil Goyal, Trahe, Fiona,
David Marchand, Anoob Joseph, Kusztal, ArkadiuszX, dev,
Richardson, Bruce, Mcnamara, John, dodji, Andrew Rybchenko,
aconole
On 2/3/2020 11:50 AM, Neil Horman wrote:
> On Mon, Feb 03, 2020 at 09:30:06AM +0000, Ferruh Yigit wrote:
>> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
>>>
>>>> 31/01/2020 15:16, Trahe, Fiona:
>>>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
>>>>>> 30/01/2020 17:09, Ferruh Yigit:
>>>>>>> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
>>>>>>>>
>>>>>>>> I believe these enums will be used only in case of ASYM case which is experimental.
>>>>>>>
>>>>>>> Independent from being experiment and not, this shouldn't be a problem, I think
>>>>>>> this is a false positive.
>>>>>>>
>>>>>>> The ABI break can happen when a struct has been shared between the application
>>>>>>> and the library (DPDK) and the layout of that memory know differently by
>>>>>>> application and the library.
>>>>>>>
>>>>>>> Here in all cases, there is no layout/size change.
>>>>>>>
>>>>>>> As to the value changes of the enums, since application compiled with old DPDK,
>>>>>>> it will know only up to '6', 7 and more means invalid to the application. So it
>>>>>>> won't send these values also it should ignore these values from library. Only
>>>>>>> consequence is old application won't able to use new features those new enums
>>>>>>> provide but that is expected/normal.
>>>>>>
>>>>>> If library give higher value than expected by the application,
>>>>>> if the application uses this value as array index,
>>>>>> there can be an access out of bounds.
>>>>>
>>>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
>>>>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
>>>>> sense and I don't see how there can be an API breakage.
>>>>> So if an application hasn't compiled against the new lib it will be still using the old value
>>>>> which will be within bounds. If it's picking up the higher new value from the lib it must
>>>>> have been compiled against the lib so shouldn't have problems.
>>>>
>>>> You say there is no ABI issue because the application will be re-compiled
>>>> for the updated library. Indeed, compilation fixes compatibility issues.
>>>> But this is not relevant for ABI compatibility.
>>>> ABI compatibility means we can upgrade the library without recompiling
>>>> the application and it must work.
>>>> You think it is a false positive because you assume the application
>>>> "picks" the new value. I think you miss the case where the new value
>>>> is returned by a function in the upgraded library.
>>>>
>>>>> There are also no structs on the API which contain arrays using this
>>>>> for sizing, so I don't see an opportunity for an appl to have a
>>>>> mismatch in memory addresses.
>>>>
>>>> Let me demonstrate where the API may "use" the new value
>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
>>>>
>>>> Once upon a time a DPDK application counting the number of devices
>>>> supporting each AEAD algo (in order to find the best supported algo).
>>>> It is done in an array indexed by algo id:
>>>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
>>>> The application is compiled with DPDK 19.11,
>>>> where RTE_CRYPTO_AEAD_LIST_END = 3.
>>>> So the size of the application array aead_dev_count is 3.
>>>> This binary is run with DPDK 20.02,
>>>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
>>>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
>>>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
>>>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
>>>> The application uses this value:
>>>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
>>>> The application is crashing because of out of bound access.
>>>
>>> I'd say this is an example of bad written app.
>>> It probably should check that returned by library value doesn't
>>> exceed its internal array size.
>>
>> +1
>>
>> Application should ignore values >= MAX.
>>
> The example is still somewhat valid in it general principle though. While
> extending an ennumeration may be flagged by libabigail as an ABI breakage, its
> not necessecarily a false positive. By extending the ennumeration, all the
> previous entries in an array defined by said ennumeration remain constant in
> their offsets, so you can 'get away with such a change' in terms of preserving
> backwards compatibility in the above example, but you cannot, for example,
> shuffle the values in the ennumeration, as doing so would cause a functional
> breakage (i.e. requesting an instance of RTE_CRYPTO_AEAD_CHACHA20_POLY1305 might
> instead give you an instance of RTE_CRYPTO_AEAD_AES_GCM.
+1 the change/shuffle of the existing values are problematic, but we don't have
it in this case.
>
> These sorts of changes are the type that we could collectively waive in terms of
> ABI checking, as they should be ok, but the errors from libabigail should be
> taken as an indicator that this API could be rewritten (for example by removing
> the abi entirely, and adding an API call that returns an array of instance name
> and ids), so that changes of the above sort arent required.
We can spend more time on it, but I can't see for now how to escape returning
enumaration as indication of type, and this looks legitimate sage as long as
other side verifies the received value is valid in the type range.
>
>
>> Do you suggest we don't extend any enum or define between ABI breakage releases
>> to be sure bad written applications not affected?
>>
> As noted above, we could waive such corner cases, and probably be fine, but the
> error from the ABI check still serves a valid purpose in that its an indicator
> that your library API is ABI sensitive to code changes that re-architecture may
> address
>
The concern is when there are cases we can waive, we can't directly rely on the
tool and automate it. These indicators good for improving the code, but not good
to use it as build time checker.
Is there any way to reduce the failure only to definite ABI breakages?
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
2020-02-02 18:53 7% ` [dpdk-dev] [RFC] " Neil Horman
@ 2020-02-03 12:53 4% ` Ferruh Yigit
2020-02-04 12:02 9% ` Neil Horman
0 siblings, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-02-03 12:53 UTC (permalink / raw)
To: Neil Horman
Cc: Cristian Dumitrescu, Eelco Chaudron, dev, Thomas Monjalon,
Luca Boccassi, David Marchand, Bruce Richardson, Ian Stokes
On 2/2/2020 6:53 PM, Neil Horman wrote:
> On Wed, Jan 29, 2020 at 12:29:53PM +0000, Ferruh Yigit wrote:
>> Duplicated the existing symbol and versioned one as experimental and
>> other as stable.
>>
>> Created VERSION_SYMBOL_EXPERIMENTAL helper macro.
>>
>> Updated the 'check-experimental-syms.sh' buildtool, which was
>> complaining that the symbol is in EXPERIMENTAL tag in .map file but it
>> is not in the .experimental section (__rte_experimental tag is missing).
>> Updated tool in a way it won't complain if the symbol in the
>> EXPERIMENTAL tag duplicated in some other block in .map file (versioned)
>>
>> Updated meson build system to allow the versioning,
>> 'use_function_versioning = true', not sure why it was disabled by
>> default.
>>
>> Fixes: 30512af820fe ("meter: remove experimental flag from RFC4115 trTCM API")
>>
> I'm not sure I understand the purpose here.
>
> I think I understand what you are trying to do. I believe that you are
> trying to move the experimental symbols in librte_meter to be part of
> the official ABI, while still maintaining backward compatibility with
> applications built against the experimental symbol versions, correct?
Yes, exactly.
>
> I can understand the desire, and I'm not exactly opposed to providing a
> mechanism for that, but it seems somewhat complex, and to be honest,
> part of the drawback to using experimental symbols as an application
> developer is understanding that you may need to rebuild when those
> symbols change (even if the change is the symbol removal and replacement
> with an identical one that has a versioned tag).
>
> I think the right answer for people encountering this condition is to
> just rebuild. Otherwise, we are creating an environment in which we are
> implicitly communicating to users that we are maintaining a modicum of
> stability in experimental symobls.
This will mean, even the the API was perfect it will force its users to
recompile (and re-package, re-deploy etc..) at least once, this is feeling like
we are still breaking user applications easily.
And this may create a stronger motivation by applications not to use
experimental APIs, I can't decide if this is a good thing or bad thing.
The issue is amplified in the LTS,
If we remove experimental tag in LTS, will be breaking the apps using that
experimental API, just to mature the API.
If we keep it as experimental, the API will be mature in main repo, but the LTS
has to keep exact same API as experimental up to two years.
But if we can do the versioning in the master, LTS can backport it and can have
mature version of that API in LTS without breaking the existing users.
>
> A few more nits in line
>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> ---
>> Cc: Neil Horman <nhorman@tuxdriver.com>
>> Cc: Thomas Monjalon <thomas@monjalon.net>
>> Cc: Luca Boccassi <bluca@debian.org>
>> Cc: David Marchand <david.marchand@redhat.com>
>> Cc: Bruce Richardson <bruce.richardson@intel.com>
>> Cc: Ian Stokes <ian.stokes@intel.com>
>> Cc: Eelco Chaudron <echaudro@redhat.com>
>>
>> I didn't like the idea of duplicating the symbol but not able to find to
>> alias it without duplicating, if there is a way please share.
>>
>> Also not tested with old binaries, only verified the symbols in new
>> binaries.
>> ---
>> buildtools/check-experimental-syms.sh | 3 +-
>> .../common/include/rte_function_versioning.h | 4 ++
>> lib/librte_meter/rte_meter.c | 59 ++++++++++++++++++-
>> lib/librte_meter/rte_meter_version.map | 8 +++
>> lib/meson.build | 2 +-
>> 5 files changed, 71 insertions(+), 5 deletions(-)
>>
>> diff --git a/buildtools/check-experimental-syms.sh b/buildtools/check-experimental-syms.sh
>> index f3603e5ba..35c4bf006 100755
>> --- a/buildtools/check-experimental-syms.sh
>> +++ b/buildtools/check-experimental-syms.sh
>> @@ -26,7 +26,8 @@ ret=0
>> for SYM in `$LIST_SYMBOL -S EXPERIMENTAL $MAPFILE |cut -d ' ' -f 3`
>> do
>> if grep -q "\.text.*[[:space:]]$SYM$" $DUMPFILE &&
>> - ! grep -q "\.text\.experimental.*[[:space:]]$SYM$" $DUMPFILE
>> + ! grep -q "\.text\.experimental.*[[:space:]]$SYM$" $DUMPFILE &&
>> + $LIST_SYMBOL -s $SYM $MAPFILE | grep -q EXPERIMENTAL
>> then
>> cat >&2 <<- END_OF_MESSAGE
>> $SYM is not flagged as experimental
>> diff --git a/lib/librte_eal/common/include/rte_function_versioning.h b/lib/librte_eal/common/include/rte_function_versioning.h
>> index c924351d5..ab102b06e 100644
>> --- a/lib/librte_eal/common/include/rte_function_versioning.h
>> +++ b/lib/librte_eal/common/include/rte_function_versioning.h
>> @@ -46,6 +46,9 @@
>> */
>> #define VERSION_SYMBOL(b, e, n) __asm__(".symver " RTE_STR(b) RTE_STR(e) ", " RTE_STR(b) "@DPDK_" RTE_STR(n))
>>
>> +
>> +#define VERSION_SYMBOL_EXPERIMENTAL(b, e) __asm__(".symver " RTE_STR(b) RTE_STR(e) ", " RTE_STR(b) "@EXPERIMENTAL")
>> +
> I don't know that you want to make a explicit clone of this macro.
> instead what you might want to do is something like:
>
> #define __VERSION_SYMBOL(b, e, n, t) __asm__(".symver " RTE_STR(b) RTE_STR(e) ", " RTE_STR(b) "@" RTE_STR(t) "_" RTE_STR(n))
>
> #define VERSION_SYMBOL(b, e, n) __VERSION_SYMBOL(b, e, n, DPDK)
>
> Thats not exactly right, but you get the idea, then you can emulate
> VERSION_SYMBOL_EXPERIMENTAL with __VERSION_SYMBOL(b, e, n, EXPERIMENTAL)
+1
>
>
>> /*
>> * BIND_DEFAULT_SYMBOL
>> * Creates a symbol version entry instructing the linker to bind references to
>> @@ -79,6 +82,7 @@
>> * No symbol versioning in use
>> */
>> #define VERSION_SYMBOL(b, e, n)
>> +#define VERSION_SYMBOL_EXPERIMENTAL(b, e)
>> #define __vsym
>> #define BIND_DEFAULT_SYMBOL(b, e, n)
>> #define MAP_STATIC_SYMBOL(f, p) f __attribute__((alias(RTE_STR(p))))
>> diff --git a/lib/librte_meter/rte_meter.c b/lib/librte_meter/rte_meter.c
>> index da01429a8..5244537fa 100644
>> --- a/lib/librte_meter/rte_meter.c
>> +++ b/lib/librte_meter/rte_meter.c
>> @@ -9,6 +9,7 @@
>> #include <rte_common.h>
>> #include <rte_log.h>
>> #include <rte_cycles.h>
>> +#include <rte_function_versioning.h>
>>
>> #include "rte_meter.h"
>>
>> @@ -119,8 +120,8 @@ rte_meter_trtcm_config(struct rte_meter_trtcm *m,
>> return 0;
>> }
>>
>> -int
>> -rte_meter_trtcm_rfc4115_profile_config(
>> +static int
>> +rte_meter_trtcm_rfc4115_profile_config_(
>> struct rte_meter_trtcm_rfc4115_profile *p,
>> struct rte_meter_trtcm_rfc4115_params *params)
>> {
>> @@ -145,7 +146,35 @@ rte_meter_trtcm_rfc4115_profile_config(
>> }
>>
>> int
>> -rte_meter_trtcm_rfc4115_config(
>> +rte_meter_trtcm_rfc4115_profile_config_s(
>> + struct rte_meter_trtcm_rfc4115_profile *p,
>> + struct rte_meter_trtcm_rfc4115_params *params);
>> +int
>> +rte_meter_trtcm_rfc4115_profile_config_s(
>> + struct rte_meter_trtcm_rfc4115_profile *p,
>> + struct rte_meter_trtcm_rfc4115_params *params)
>> +{
>> + return rte_meter_trtcm_rfc4115_profile_config_(p, params);
>> +}
>> +BIND_DEFAULT_SYMBOL(rte_meter_trtcm_rfc4115_profile_config, _s, 20.0.1);
>> +MAP_STATIC_SYMBOL(int rte_meter_trtcm_rfc4115_profile_config(struct rte_meter_trtcm_rfc4115_profile *p,
>> + struct rte_meter_trtcm_rfc4115_params *params), rte_meter_trtcm_rfc4115_profile_config_s);
>> +
>> +int
>> +rte_meter_trtcm_rfc4115_profile_config_e(
>> + struct rte_meter_trtcm_rfc4115_profile *p,
>> + struct rte_meter_trtcm_rfc4115_params *params);
>> +int
>> +rte_meter_trtcm_rfc4115_profile_config_e(
>> + struct rte_meter_trtcm_rfc4115_profile *p,
>> + struct rte_meter_trtcm_rfc4115_params *params)
>> +{
>> + return rte_meter_trtcm_rfc4115_profile_config_(p, params);
>> +}
>> +VERSION_SYMBOL_EXPERIMENTAL(rte_meter_trtcm_rfc4115_profile_config, _e);
>> +
>> +static int
>> +rte_meter_trtcm_rfc4115_config_(
>> struct rte_meter_trtcm_rfc4115 *m,
>> struct rte_meter_trtcm_rfc4115_profile *p)
>> {
>> @@ -160,3 +189,27 @@ rte_meter_trtcm_rfc4115_config(
>>
>> return 0;
>> }
>> +
>> +int
>> +rte_meter_trtcm_rfc4115_config_s(struct rte_meter_trtcm_rfc4115 *m,
>> + struct rte_meter_trtcm_rfc4115_profile *p);
>> +int
>> +rte_meter_trtcm_rfc4115_config_s(struct rte_meter_trtcm_rfc4115 *m,
>> + struct rte_meter_trtcm_rfc4115_profile *p)
>> +{
>> + return rte_meter_trtcm_rfc4115_config_(m, p);
>> +}
>> +BIND_DEFAULT_SYMBOL(rte_meter_trtcm_rfc4115_config, _s, 20.0.1);
>> +MAP_STATIC_SYMBOL(int rte_meter_trtcm_rfc4115_config(struct rte_meter_trtcm_rfc4115 *m,
>> + struct rte_meter_trtcm_rfc4115_profile *p), rte_meter_trtcm_rfc4115_config_s);
>> +
>> +int
>> +rte_meter_trtcm_rfc4115_config_e(struct rte_meter_trtcm_rfc4115 *m,
>> + struct rte_meter_trtcm_rfc4115_profile *p);
>> +int
>> +rte_meter_trtcm_rfc4115_config_e(struct rte_meter_trtcm_rfc4115 *m,
>> + struct rte_meter_trtcm_rfc4115_profile *p)
>> +{
>> + return rte_meter_trtcm_rfc4115_config_(m, p);
>> +}
>> +VERSION_SYMBOL_EXPERIMENTAL(rte_meter_trtcm_rfc4115_config, _e);
>> diff --git a/lib/librte_meter/rte_meter_version.map b/lib/librte_meter/rte_meter_version.map
>> index fc12cc0bf..b4b043174 100644
>> --- a/lib/librte_meter/rte_meter_version.map
>> +++ b/lib/librte_meter/rte_meter_version.map
>> @@ -20,4 +20,12 @@ DPDK_20.0.1 {
>> rte_meter_trtcm_rfc4115_color_blind_check;
>> rte_meter_trtcm_rfc4115_config;
>> rte_meter_trtcm_rfc4115_profile_config;
>> +
>> } DPDK_20.0;
>> +
>> +EXPERIMENTAL {
>> + global:
>> +
>> + rte_meter_trtcm_rfc4115_config;
>> + rte_meter_trtcm_rfc4115_profile_config;
>> +};
>> diff --git a/lib/meson.build b/lib/meson.build
>> index 0af3efab2..553964831 100644
>> --- a/lib/meson.build
>> +++ b/lib/meson.build
>> @@ -48,7 +48,7 @@ foreach l:libraries
>> reason = '<unknown reason>' # set if build == false to explain why
>> name = l
>> allow_experimental_apis = false
>> - use_function_versioning = false
>> + use_function_versioning = true
>> sources = []
>> headers = []
>> includes = []
>> --
>> 2.24.1
>>
>>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-03 9:30 7% ` Ferruh Yigit
@ 2020-02-03 11:50 9% ` Neil Horman
2020-02-03 13:09 7% ` Ferruh Yigit
2020-02-03 17:09 9% ` Thomas Monjalon
1 sibling, 1 reply; 200+ results
From: Neil Horman @ 2020-02-03 11:50 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Ananyev, Konstantin, Thomas Monjalon, Akhil Goyal, Trahe, Fiona,
David Marchand, Anoob Joseph, Kusztal, ArkadiuszX, dev,
Richardson, Bruce, Mcnamara, John, dodji, Andrew Rybchenko,
aconole
On Mon, Feb 03, 2020 at 09:30:06AM +0000, Ferruh Yigit wrote:
> On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
> >
> >> 31/01/2020 15:16, Trahe, Fiona:
> >>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> >>>> 30/01/2020 17:09, Ferruh Yigit:
> >>>>> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
> >>>>>>
> >>>>>> I believe these enums will be used only in case of ASYM case which is experimental.
> >>>>>
> >>>>> Independent from being experiment and not, this shouldn't be a problem, I think
> >>>>> this is a false positive.
> >>>>>
> >>>>> The ABI break can happen when a struct has been shared between the application
> >>>>> and the library (DPDK) and the layout of that memory know differently by
> >>>>> application and the library.
> >>>>>
> >>>>> Here in all cases, there is no layout/size change.
> >>>>>
> >>>>> As to the value changes of the enums, since application compiled with old DPDK,
> >>>>> it will know only up to '6', 7 and more means invalid to the application. So it
> >>>>> won't send these values also it should ignore these values from library. Only
> >>>>> consequence is old application won't able to use new features those new enums
> >>>>> provide but that is expected/normal.
> >>>>
> >>>> If library give higher value than expected by the application,
> >>>> if the application uses this value as array index,
> >>>> there can be an access out of bounds.
> >>>
> >>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
> >>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
> >>> sense and I don't see how there can be an API breakage.
> >>> So if an application hasn't compiled against the new lib it will be still using the old value
> >>> which will be within bounds. If it's picking up the higher new value from the lib it must
> >>> have been compiled against the lib so shouldn't have problems.
> >>
> >> You say there is no ABI issue because the application will be re-compiled
> >> for the updated library. Indeed, compilation fixes compatibility issues.
> >> But this is not relevant for ABI compatibility.
> >> ABI compatibility means we can upgrade the library without recompiling
> >> the application and it must work.
> >> You think it is a false positive because you assume the application
> >> "picks" the new value. I think you miss the case where the new value
> >> is returned by a function in the upgraded library.
> >>
> >>> There are also no structs on the API which contain arrays using this
> >>> for sizing, so I don't see an opportunity for an appl to have a
> >>> mismatch in memory addresses.
> >>
> >> Let me demonstrate where the API may "use" the new value
> >> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
> >>
> >> Once upon a time a DPDK application counting the number of devices
> >> supporting each AEAD algo (in order to find the best supported algo).
> >> It is done in an array indexed by algo id:
> >> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
> >> The application is compiled with DPDK 19.11,
> >> where RTE_CRYPTO_AEAD_LIST_END = 3.
> >> So the size of the application array aead_dev_count is 3.
> >> This binary is run with DPDK 20.02,
> >> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
> >> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
> >> rte_cryptodev_info.capabilities.sym.aead.algo is set to
> >> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
> >> The application uses this value:
> >> ++ aead_dev_count[info.capabilities.sym.aead.algo];
> >> The application is crashing because of out of bound access.
> >
> > I'd say this is an example of bad written app.
> > It probably should check that returned by library value doesn't
> > exceed its internal array size.
>
> +1
>
> Application should ignore values >= MAX.
>
The example is still somewhat valid in it general principle though. While
extending an ennumeration may be flagged by libabigail as an ABI breakage, its
not necessecarily a false positive. By extending the ennumeration, all the
previous entries in an array defined by said ennumeration remain constant in
their offsets, so you can 'get away with such a change' in terms of preserving
backwards compatibility in the above example, but you cannot, for example,
shuffle the values in the ennumeration, as doing so would cause a functional
breakage (i.e. requesting an instance of RTE_CRYPTO_AEAD_CHACHA20_POLY1305 might
instead give you an instance of RTE_CRYPTO_AEAD_AES_GCM.
These sorts of changes are the type that we could collectively waive in terms of
ABI checking, as they should be ok, but the errors from libabigail should be
taken as an indicator that this API could be rewritten (for example by removing
the abi entirely, and adding an API call that returns an array of instance name
and ids), so that changes of the above sort arent required.
> Do you suggest we don't extend any enum or define between ABI breakage releases
> to be sure bad written applications not affected?
>
As noted above, we could waive such corner cases, and probably be fine, but the
error from the ABI check still serves a valid purpose in that its an indicator
that your library API is ABI sensitive to code changes that re-architecture may
address
Neil
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [RFC v2] meter: fix ABI break due to experimental tag removal
2020-01-30 20:20 4% ` Thomas Monjalon
@ 2020-02-03 11:10 4% ` Eelco Chaudron
0 siblings, 0 replies; 200+ results
From: Eelco Chaudron @ 2020-02-03 11:10 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Luca Boccassi, Ferruh Yigit, Neil Horman, Cristian Dumitrescu,
dev, David Marchand, Bruce Richardson, Ian Stokes,
Andrzej Ostruszka
On 30 Jan 2020, at 21:20, Thomas Monjalon wrote:
<SNIP>
>>>>
>>>> Yes, sorry, this is what I meant.
>>>> Given 19.11.0 already has the symbol as experimental,
>>>> and that applications like OVS had to accept it as experimental,
>>>> why removing experimental tag in 19.11.1?
>>>
>>> I think it was mentioned that it was preferred not to suppress the
>>> compiler warning to avoid any accidental use in the future, but the
>>> OVS
>>> maintainer(s) should answer as I might remember wrongly.
>>
>> Yes this is the reason, OVS compiles with -Werror so we would like to
>> avoid the warnings. You can not disable them per include, it’s
>> global
>> for all of DPDK.
>
> Yes but anyway OVS must accept the experimental function as the next
> release
> will use it with DPDK 19.11.0.
Yes, we do for now, but we would like to remove it ASAP as it opens up
the code base for the experimental APIs to slip in…
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-02 14:41 4% ` Ananyev, Konstantin
@ 2020-02-03 9:30 7% ` Ferruh Yigit
2020-02-03 11:50 9% ` Neil Horman
2020-02-03 17:09 9% ` Thomas Monjalon
0 siblings, 2 replies; 200+ results
From: Ferruh Yigit @ 2020-02-03 9:30 UTC (permalink / raw)
To: Ananyev, Konstantin, Thomas Monjalon, Akhil Goyal, Trahe, Fiona
Cc: David Marchand, Anoob Joseph, Kusztal, ArkadiuszX, dev,
Richardson, Bruce, nhorman, Mcnamara, John, dodji,
Andrew Rybchenko, aconole
On 2/2/2020 2:41 PM, Ananyev, Konstantin wrote:
>
>> 31/01/2020 15:16, Trahe, Fiona:
>>> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
>>>> 30/01/2020 17:09, Ferruh Yigit:
>>>>> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
>>>>>>
>>>>>> I believe these enums will be used only in case of ASYM case which is experimental.
>>>>>
>>>>> Independent from being experiment and not, this shouldn't be a problem, I think
>>>>> this is a false positive.
>>>>>
>>>>> The ABI break can happen when a struct has been shared between the application
>>>>> and the library (DPDK) and the layout of that memory know differently by
>>>>> application and the library.
>>>>>
>>>>> Here in all cases, there is no layout/size change.
>>>>>
>>>>> As to the value changes of the enums, since application compiled with old DPDK,
>>>>> it will know only up to '6', 7 and more means invalid to the application. So it
>>>>> won't send these values also it should ignore these values from library. Only
>>>>> consequence is old application won't able to use new features those new enums
>>>>> provide but that is expected/normal.
>>>>
>>>> If library give higher value than expected by the application,
>>>> if the application uses this value as array index,
>>>> there can be an access out of bounds.
>>>
>>> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
>>> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
>>> sense and I don't see how there can be an API breakage.
>>> So if an application hasn't compiled against the new lib it will be still using the old value
>>> which will be within bounds. If it's picking up the higher new value from the lib it must
>>> have been compiled against the lib so shouldn't have problems.
>>
>> You say there is no ABI issue because the application will be re-compiled
>> for the updated library. Indeed, compilation fixes compatibility issues.
>> But this is not relevant for ABI compatibility.
>> ABI compatibility means we can upgrade the library without recompiling
>> the application and it must work.
>> You think it is a false positive because you assume the application
>> "picks" the new value. I think you miss the case where the new value
>> is returned by a function in the upgraded library.
>>
>>> There are also no structs on the API which contain arrays using this
>>> for sizing, so I don't see an opportunity for an appl to have a
>>> mismatch in memory addresses.
>>
>> Let me demonstrate where the API may "use" the new value
>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
>>
>> Once upon a time a DPDK application counting the number of devices
>> supporting each AEAD algo (in order to find the best supported algo).
>> It is done in an array indexed by algo id:
>> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
>> The application is compiled with DPDK 19.11,
>> where RTE_CRYPTO_AEAD_LIST_END = 3.
>> So the size of the application array aead_dev_count is 3.
>> This binary is run with DPDK 20.02,
>> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
>> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
>> rte_cryptodev_info.capabilities.sym.aead.algo is set to
>> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
>> The application uses this value:
>> ++ aead_dev_count[info.capabilities.sym.aead.algo];
>> The application is crashing because of out of bound access.
>
> I'd say this is an example of bad written app.
> It probably should check that returned by library value doesn't
> exceed its internal array size.
+1
Application should ignore values >= MAX.
Do you suggest we don't extend any enum or define between ABI breakage releases
to be sure bad written applications not affected?
^ permalink raw reply [relevance 7%]
* [dpdk-dev] [PATCH v4 3/3] add ABI checks
2020-02-02 21:08 9% ` [dpdk-dev] [PATCH v4 0/3] " David Marchand
@ 2020-02-02 21:08 28% ` David Marchand
2020-02-05 14:13 4% ` [dpdk-dev] [PATCH v4 0/3] " Thomas Monjalon
1 sibling, 0 replies; 200+ results
From: David Marchand @ 2020-02-02 21:08 UTC (permalink / raw)
To: dev
Cc: thomas, bruce.richardson, kevin.laatz, aconole, nhorman,
akhil.goyal, anoobj, bluca, fiona.trahe, ferruh.yigit,
Michael Santana, John McNamara, Marko Kovacevic
For normal developers, those checks are disabled.
Enabling them requires a configuration that will trigger the ABI dumps
generation as part of the existing devtools/test-build.sh and
devtools/test-meson-builds.sh scripts.
Those checks are enabled in the CI for the default meson options on x86
and aarch64 so that proposed patches are validated via our CI robot.
A cache of the ABI is stored in travis jobs to avoid rebuilding too
often.
Checks can be informational only, by setting ABI_CHECKS_WARN_ONLY when
breaking the ABI in a future release.
Explicit suppression rules have been added on internal structures
exposed to crypto drivers as the current ABI policy does not apply to
them.
This could be improved in the future by carefully splitting the headers
content with application and driver "users" in mind.
We currently have issues reported for librte_crypto recent changes for
which suppression rules have been added too.
Mellanox glue libraries are explicitly skipped as they are not part of
the application ABI.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Luca Boccassi <bluca@debian.org>
---
Changelog since v3:
- rebased following patch 3 removal,
- added dpdk.abignore to MAINTAINERS,
- avoided meson reconfiguration when unneeded,
- switched to fixed sources location for building ABI references,
Changelog since v2:
- forced -g / buildtype=debugoptimised in the test scripts so that
we can check ABI in existing environments,
- little update on the documentation,
Changelog since v1:
- reworked the scripts so that the build test scripts clone and build the
reference automatically. A developer only needs to set one variable to
enable the checks,
- meson builds are done with debug so that abidiff can inspect the
structures,
- abidiff checks only public types by looking at installed headers,
- abidiff has some issue when comparing a dump with a .so built with clang
so all diff are now done with dump files only,
- suppression rules have been added to waive warnings on exposed internal
types,
- an abi breakage has been reported on changes in cryptodev.
For now, suppression rules have been put in place to let the CI run,
---
.ci/linux-build.sh | 24 ++++++++++++
.travis.yml | 20 +++++++++-
MAINTAINERS | 3 ++
devtools/check-abi.sh | 59 +++++++++++++++++++++++++++++
devtools/dpdk.abignore | 21 ++++++++++
devtools/gen-abi.sh | 26 +++++++++++++
devtools/test-build.sh | 52 ++++++++++++++++++++++---
devtools/test-meson-builds.sh | 50 ++++++++++++++++++++++--
doc/guides/contributing/patches.rst | 15 ++++++++
9 files changed, 260 insertions(+), 10 deletions(-)
create mode 100755 devtools/check-abi.sh
create mode 100644 devtools/dpdk.abignore
create mode 100755 devtools/gen-abi.sh
diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index ccc3a7ccd..c7c3840fc 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -29,6 +29,7 @@ if [ "$BUILD_32BIT" = "1" ]; then
fi
OPTS="$OPTS --default-library=$DEF_LIB"
+OPTS="$OPTS --buildtype=debugoptimized"
meson build --werror -Dexamples=all $OPTS
ninja -C build
@@ -36,6 +37,29 @@ if [ "$AARCH64" != "1" ]; then
devtools/test-null.sh
fi
+if [ "$ABI_CHECKS" = "1" ]; then
+ REF_GIT_REPO=${REF_GIT_REPO:-https://dpdk.org/git/dpdk}
+ REF_GIT_TAG=${REF_GIT_TAG:-v19.11}
+
+ if [ "$(cat reference/VERSION 2>/dev/null)" != "$REF_GIT_TAG" ]; then
+ rm -rf reference
+ fi
+
+ if [ ! -d reference ]; then
+ refsrcdir=$(readlink -f $(pwd)/../dpdk-$REF_GIT_TAG)
+ git clone --single-branch -b $REF_GIT_TAG $REF_GIT_REPO $refsrcdir
+ meson --werror $OPTS $refsrcdir $refsrcdir/build
+ ninja -C $refsrcdir/build
+ DESTDIR=$(pwd)/reference ninja -C $refsrcdir/build install
+ devtools/gen-abi.sh reference
+ echo $REF_GIT_TAG > reference/VERSION
+ fi
+
+ DESTDIR=$(pwd)/install ninja -C build install
+ devtools/gen-abi.sh install
+ devtools/check-abi.sh reference install ${ABI_CHECKS_WARN_ONLY:-}
+fi
+
if [ "$RUN_TESTS" = "1" ]; then
sudo meson test -C build --suite fast-tests -t 3
fi
diff --git a/.travis.yml b/.travis.yml
index 8162f1c05..22539d823 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,5 +1,8 @@
language: c
-cache: ccache
+cache:
+ ccache: true
+ directories:
+ - reference
compiler:
- gcc
- clang
@@ -21,7 +24,7 @@ aarch64_packages: &aarch64_packages
extra_packages: &extra_packages
- *required_packages
- - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
+ - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4, abigail-tools]
build_32b_packages: &build_32b_packages
- *required_packages
@@ -151,5 +154,18 @@ matrix:
packages:
- *required_packages
- *doc_packages
+ - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
+ compiler: gcc
+ addons:
+ apt:
+ packages:
+ - *extra_packages
+ - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
+ arch: arm64
+ compiler: gcc
+ addons:
+ apt:
+ packages:
+ - *extra_packages
script: ./.ci/${TRAVIS_OS_NAME}-build.sh
diff --git a/MAINTAINERS b/MAINTAINERS
index 94bccae6d..db4b8c5b6 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -144,8 +144,11 @@ M: Neil Horman <nhorman@tuxdriver.com>
F: lib/librte_eal/common/include/rte_compat.h
F: lib/librte_eal/common/include/rte_function_versioning.h
F: doc/guides/rel_notes/deprecation.rst
+F: devtools/check-abi.sh
F: devtools/check-abi-version.sh
F: devtools/check-symbol-change.sh
+F: devtools/dpdk.abignore
+F: devtools/gen-abi.sh
F: devtools/update-abi.sh
F: devtools/update_version_map_abi.py
F: devtools/validate-abi.sh
diff --git a/devtools/check-abi.sh b/devtools/check-abi.sh
new file mode 100755
index 000000000..5872499ec
--- /dev/null
+++ b/devtools/check-abi.sh
@@ -0,0 +1,59 @@
+#!/bin/sh -e
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 Red Hat, Inc.
+
+if [ $# != 2 ] && [ $# != 3 ]; then
+ echo "Usage: $0 refdir newdir [warnonly]"
+ exit 1
+fi
+
+refdir=$1
+newdir=$2
+warnonly=${3:-}
+ABIDIFF_OPTIONS="--suppr $(dirname $0)/dpdk.abignore --no-added-syms"
+
+if [ ! -d $refdir ]; then
+ echo "Error: reference directory '$refdir' does not exist."
+ exit 1
+fi
+incdir=$(find $refdir -type d -a -name include)
+if [ -z "$incdir" ] || [ ! -e "$incdir" ]; then
+ echo "WARNING: could not identify a include directory for $refdir, expect false positives..."
+else
+ ABIDIFF_OPTIONS="$ABIDIFF_OPTIONS --headers-dir1 $incdir"
+fi
+
+if [ ! -d $newdir ]; then
+ echo "Error: directory to check '$newdir' does not exist."
+ exit 1
+fi
+incdir2=$(find $newdir -type d -a -name include)
+if [ -z "$incdir2" ] || [ ! -e "$incdir2" ]; then
+ echo "WARNING: could not identify a include directory for $newdir, expect false positives..."
+else
+ ABIDIFF_OPTIONS="$ABIDIFF_OPTIONS --headers-dir2 $incdir2"
+fi
+
+error=
+for dump in $(find $refdir -name "*.dump"); do
+ name=$(basename $dump)
+ # skip glue drivers, example librte_pmd_mlx5_glue.dump
+ # We can't rely on a suppression rule for now:
+ # https://sourceware.org/bugzilla/show_bug.cgi?id=25480
+ if [ "$name" != "${name%%_glue.dump}" ]; then
+ echo "Skipping ${dump}..."
+ continue
+ fi
+ dump2=$(find $newdir -name $name)
+ if [ -z "$dump2" ] || [ ! -e "$dump2" ]; then
+ echo "Error: can't find $name in $newdir"
+ error=1
+ continue
+ fi
+ if ! abidiff $ABIDIFF_OPTIONS $dump $dump2; then
+ echo "Error: ABI issue reported for 'abidiff $ABIDIFF_OPTIONS $dump $dump2'"
+ error=1
+ fi
+done
+
+[ -z "$error" ] || [ -n "$warnonly" ]
diff --git a/devtools/dpdk.abignore b/devtools/dpdk.abignore
new file mode 100644
index 000000000..f2903612c
--- /dev/null
+++ b/devtools/dpdk.abignore
@@ -0,0 +1,21 @@
+[suppress_function]
+ symbol_version = EXPERIMENTAL
+[suppress_variable]
+ symbol_version = EXPERIMENTAL
+
+; Explicit ignore for driver-only ABI
+[suppress_type]
+ name = rte_cryptodev_ops
+; Ignore this enum update as it is part of an experimental API
+[suppress_type]
+ type_kind = enum
+ name = rte_crypto_asym_xform_type
+ changed_enumerators = RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
+
+; FIXME
+[suppress_type]
+ type_kind = enum
+ name = rte_crypto_aead_algorithm
+ changed_enumerators = RTE_CRYPTO_AEAD_LIST_END
+[suppress_variable]
+ name = rte_crypto_aead_algorithm_strings
diff --git a/devtools/gen-abi.sh b/devtools/gen-abi.sh
new file mode 100755
index 000000000..c44b0e228
--- /dev/null
+++ b/devtools/gen-abi.sh
@@ -0,0 +1,26 @@
+#!/bin/sh -e
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 Red Hat, Inc.
+
+if [ $# != 1 ]; then
+ echo "Usage: $0 installdir"
+ exit 1
+fi
+
+installdir=$1
+if [ ! -d $installdir ]; then
+ echo "Error: install directory '$installdir' does not exist."
+ exit 1
+fi
+
+dumpdir=$installdir/dump
+rm -rf $dumpdir
+mkdir -p $dumpdir
+for f in $(find $installdir -name "*.so.*"); do
+ if test -L $f; then
+ continue
+ fi
+
+ libname=$(basename $f)
+ abidw --out-file $dumpdir/${libname%.so*}.dump $f
+done
diff --git a/devtools/test-build.sh b/devtools/test-build.sh
index 52305fbb8..2f55e9147 100755
--- a/devtools/test-build.sh
+++ b/devtools/test-build.sh
@@ -6,6 +6,8 @@ default_path=$PATH
# Load config options:
# - ARMV8_CRYPTO_LIB_PATH
+# - DPDK_ABI_REF_DIR
+# - DPDK_ABI_REF_VERSION
# - DPDK_BUILD_TEST_CONFIGS (defconfig1+option1+option2 defconfig2)
# - DPDK_BUILD_TEST_DIR
# - DPDK_DEP_ARCHIVE
@@ -30,7 +32,8 @@ default_path=$PATH
# - LIBSSO_SNOW3G_PATH
# - LIBSSO_KASUMI_PATH
# - LIBSSO_ZUC_PATH
-. $(dirname $(readlink -f $0))/load-devel-config
+devtools_dir=$(dirname $(readlink -f $0))
+. $devtools_dir/load-devel-config
print_usage () {
echo "usage: $(basename $0) [-h] [-jX] [-s] [config1 [config2] ...]]"
@@ -67,7 +70,8 @@ J=$DPDK_MAKE_JOBS
builds_dir=${DPDK_BUILD_TEST_DIR:-.}
short=false
unset verbose
-maxerr=-Wfatal-errors
+# for ABI checks, we need debuginfo
+test_cflags="-Wfatal-errors -g"
while getopts hj:sv ARG ; do
case $ARG in
j ) J=$OPTARG ;;
@@ -97,7 +101,7 @@ trap "signal=INT ; trap - INT ; kill -INT $$" INT
# notify result on exit
trap on_exit EXIT
-cd $(dirname $(readlink -f $0))/..
+cd $devtools_dir/..
reset_env ()
{
@@ -233,14 +237,14 @@ for conf in $configs ; do
# reload config with DPDK_TARGET set
DPDK_TARGET=$target
reset_env
- . $(dirname $(readlink -f $0))/load-devel-config
+ . $devtools_dir/load-devel-config
options=$(echo $conf | sed 's,[^~+]*,,')
dir=$builds_dir/$conf
config $dir $target $options
echo "================== Build $conf"
- ${MAKE} -j$J EXTRA_CFLAGS="$maxerr $DPDK_DEP_CFLAGS" \
+ ${MAKE} -j$J EXTRA_CFLAGS="$test_cflags $DPDK_DEP_CFLAGS" \
EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS" $verbose O=$dir
! $short || break
export RTE_TARGET=$target
@@ -253,6 +257,44 @@ for conf in $configs ; do
EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS" $verbose \
O=$(readlink -f $dir)/examples
unset RTE_TARGET
+ if [ -n "$DPDK_ABI_REF_VERSION" ]; then
+ abirefdir=${DPDK_ABI_REF_DIR:-reference}/$DPDK_ABI_REF_VERSION
+ if [ ! -d $abirefdir/$conf ]; then
+ # clone current sources
+ if [ ! -d $abirefdir/src ]; then
+ git clone --local --no-hardlinks \
+ --single-branch \
+ -b $DPDK_ABI_REF_VERSION \
+ $(pwd) $abirefdir/src
+ fi
+
+ cd $abirefdir/src
+
+ rm -rf $abirefdir/build
+ config $abirefdir/build $target $options
+
+ echo -n "================== Build $conf "
+ echo "($DPDK_ABI_REF_VERSION)"
+ ${MAKE} -j$J \
+ EXTRA_CFLAGS="$test_cflags $DPDK_DEP_CFLAGS" \
+ EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS" $verbose \
+ O=$abirefdir/build
+ ! $short || break
+ export RTE_TARGET=$target
+ ${MAKE} install O=$abirefdir/build \
+ DESTDIR=$abirefdir/$conf \
+ prefix=
+ unset RTE_TARGET
+ $devtools_dir/gen-abi.sh $abirefdir/$conf
+
+ # back to current workdir
+ cd $devtools_dir/..
+ fi
+
+ echo "================== Check ABI $conf"
+ $devtools_dir/gen-abi.sh $dir/install
+ $devtools_dir/check-abi.sh $abirefdir/$conf $dir/install
+ fi
echo "################## $conf done."
unset dir
done
diff --git a/devtools/test-meson-builds.sh b/devtools/test-meson-builds.sh
index fb6c404e5..c2d33b3b9 100755
--- a/devtools/test-meson-builds.sh
+++ b/devtools/test-meson-builds.sh
@@ -64,9 +64,18 @@ config () # <dir> <builddir> <meson options>
builddir=$1
shift
if [ -f "$builddir/build.ninja" ] ; then
+ # for existing environments, switch to debugoptimized if unset
+ # so that ABI checks can run
+ if ! $MESON configure $builddir |
+ awk '$1=="buildtype" {print $2}' |
+ grep -qw debugoptimized; then
+ $MESON configure --buildtype=debugoptimized $builddir
+ fi
return
fi
- options="--werror -Dexamples=all"
+ options=
+ options="$options --werror -Dexamples=all"
+ options="$options --buildtype=debugoptimized"
for option in $DPDK_MESON_OPTIONS ; do
options="$options -D$option"
done
@@ -92,6 +101,13 @@ compile () # <builddir>
fi
}
+install_target () # <builddir> <installdir>
+{
+ rm -rf $2
+ echo "DESTDIR=$2 $ninja_cmd -C $1 install"
+ DESTDIR=$2 $ninja_cmd -C $1 install
+}
+
build () # <directory> <target compiler> <meson options>
{
targetdir=$1
@@ -103,6 +119,31 @@ build () # <directory> <target compiler> <meson options>
load_env $targetcc || return 0
config $srcdir $builds_dir/$targetdir $*
compile $builds_dir/$targetdir
+ if [ -n "$DPDK_ABI_REF_VERSION" ]; then
+ abirefdir=${DPDK_ABI_REF_DIR:-reference}/$DPDK_ABI_REF_VERSION
+ if [ ! -d $abirefdir/$targetdir ]; then
+ # clone current sources
+ if [ ! -d $abirefdir/src ]; then
+ git clone --local --no-hardlinks \
+ --single-branch \
+ -b $DPDK_ABI_REF_VERSION \
+ $srcdir $abirefdir/src
+ fi
+
+ rm -rf $abirefdir/build
+ config $abirefdir/src $abirefdir/build $*
+ compile $abirefdir/build
+ install_target $abirefdir/build $abirefdir/$targetdir
+ $srcdir/devtools/gen-abi.sh $abirefdir/$targetdir
+ fi
+
+ install_target $builds_dir/$targetdir \
+ $(readlink -f $builds_dir/$targetdir/install)
+ $srcdir/devtools/gen-abi.sh \
+ $(readlink -f $builds_dir/$targetdir/install)
+ $srcdir/devtools/check-abi.sh $abirefdir/$targetdir \
+ $(readlink -f $builds_dir/$targetdir/install)
+ fi
}
if [ "$1" = "-vv" ] ; then
@@ -153,8 +194,11 @@ done
# Test installation of the x86-default target, to be used for checking
# the sample apps build using the pkg-config file for cflags and libs
build_path=$(readlink -f $builds_dir/build-x86-default)
-export DESTDIR=$build_path/install-root
-$ninja_cmd -C $build_path install
+export DESTDIR=$build_path/install
+# No need to reinstall if ABI checks are enabled
+if [ -z "$DPDK_ABI_REF_VERSION" ]; then
+ install_target $build_path $DESTDIR
+fi
load_env cc
pc_file=$(find $DESTDIR -name libdpdk.pc)
diff --git a/doc/guides/contributing/patches.rst b/doc/guides/contributing/patches.rst
index 0686450e4..59442824a 100644
--- a/doc/guides/contributing/patches.rst
+++ b/doc/guides/contributing/patches.rst
@@ -513,6 +513,21 @@ in a single subfolder called "__builds" created in the current directory.
Setting ``DPDK_BUILD_TEST_DIR`` to an absolute directory path e.g. ``/tmp`` is also supported.
+Checking ABI compatibility
+--------------------------
+
+By default, ABI compatibility checks are disabled.
+
+To enable them, a reference version must be selected via the environment
+variable ``DPDK_ABI_REF_VERSION``.
+
+The ``devtools/test-build.sh`` and ``devtools/test-meson-builds.sh`` scripts
+then build this reference version in a temporary directory and store the
+results in a subfolder of the current working directory.
+The environment variable ``DPDK_ABI_REF_DIR`` can be set so that the results go
+to a different location.
+
+
Sending Patches
---------------
--
2.23.0
^ permalink raw reply [relevance 28%]
* [dpdk-dev] [PATCH v4 0/3] add ABI checks
` (2 preceding siblings ...)
2020-01-30 16:00 9% ` [dpdk-dev] [PATCH v3 0/4] " David Marchand
@ 2020-02-02 21:08 9% ` David Marchand
2020-02-02 21:08 28% ` [dpdk-dev] [PATCH v4 3/3] " David Marchand
2020-02-05 14:13 4% ` [dpdk-dev] [PATCH v4 0/3] " Thomas Monjalon
3 siblings, 2 replies; 200+ results
From: David Marchand @ 2020-02-02 21:08 UTC (permalink / raw)
To: dev
Cc: thomas, bruce.richardson, kevin.laatz, aconole, nhorman,
akhil.goyal, anoobj, bluca, fiona.trahe, ferruh.yigit
Here is the current state of the ABI checks.
libabigail has some issues when mixing dump and so files compiled with
clang [1], so for now, all checks are done on dumps only.
libabigail 1.0-rc3 in Xenial reported issues that disappear with the
version 1.2 in Bionic.
To avoid getting warnings on internal types like [2], the checks now make
use of the public headers part of a dpdk installation (patch 2 and 3 to
prepare for this).
Some internal rte_hash headers were installed by meson, so patch 1 fixes
this.
The most important point, abidiff complains on the rc1 cryptodev changes:
- Chacha20-Poly1305 AEAD support,
- ECPM and ECDSA support
A suppression rule has been put on the internal type rte_cryptodev_ops.
But other changes are an ABI breakage afaiu. I put suppression rules on
them so that we can run the checks, but some action must be taken for
20.02 if my analysis is confirmed.
Special thanks to Dodji the libabigail maintainer for helping on this
topic.
1: https://sourceware.org/bugzilla/show_bug.cgi?id=25409
2: http://inbox.dpdk.org/dev/CAJFAV8yFKoDZROX9Mkyp7pDMvXw3e7mHwxjfrcjD5ZoFB2tZ8w@mail.gmail.com/
--
David Marchand
Changelog since v3:
- removed patch 3 based on Thomas feedback,
- switched to fixed sources location for building ABI references,
Changelog since v2:
- incorporated offlist Thomas comments wrt to existing build envs (do
not change --prefix, force builds with debug enabled)
David Marchand (3):
hash: fix meson headers packaging
build: split build helper
add ABI checks
.ci/linux-build.sh | 24 ++++++++
.travis.yml | 20 +++++-
MAINTAINERS | 3 +
devtools/check-abi.sh | 59 ++++++++++++++++++
devtools/dpdk.abignore | 21 +++++++
devtools/gen-abi.sh | 26 ++++++++
devtools/test-build.sh | 52 ++++++++++++++--
devtools/test-meson-builds.sh | 96 ++++++++++++++++++++++++-----
doc/guides/contributing/patches.rst | 15 +++++
lib/librte_hash/meson.build | 5 +-
10 files changed, 293 insertions(+), 28 deletions(-)
create mode 100755 devtools/check-abi.sh
create mode 100644 devtools/dpdk.abignore
create mode 100755 devtools/gen-abi.sh
--
2.23.0
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal
@ 2020-02-02 18:53 7% ` Neil Horman
2020-02-03 12:53 4% ` Ferruh Yigit
2 siblings, 1 reply; 200+ results
From: Neil Horman @ 2020-02-02 18:53 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Cristian Dumitrescu, Eelco Chaudron, dev, Thomas Monjalon,
Luca Boccassi, David Marchand, Bruce Richardson, Ian Stokes
On Wed, Jan 29, 2020 at 12:29:53PM +0000, Ferruh Yigit wrote:
> Duplicated the existing symbol and versioned one as experimental and
> other as stable.
>
> Created VERSION_SYMBOL_EXPERIMENTAL helper macro.
>
> Updated the 'check-experimental-syms.sh' buildtool, which was
> complaining that the symbol is in EXPERIMENTAL tag in .map file but it
> is not in the .experimental section (__rte_experimental tag is missing).
> Updated tool in a way it won't complain if the symbol in the
> EXPERIMENTAL tag duplicated in some other block in .map file (versioned)
>
> Updated meson build system to allow the versioning,
> 'use_function_versioning = true', not sure why it was disabled by
> default.
>
> Fixes: 30512af820fe ("meter: remove experimental flag from RFC4115 trTCM API")
>
I'm not sure I understand the purpose here.
I think I understand what you are trying to do. I believe that you are
trying to move the experimental symbols in librte_meter to be part of
the official ABI, while still maintaining backward compatibility with
applications built against the experimental symbol versions, correct?
I can understand the desire, and I'm not exactly opposed to providing a
mechanism for that, but it seems somewhat complex, and to be honest,
part of the drawback to using experimental symbols as an application
developer is understanding that you may need to rebuild when those
symbols change (even if the change is the symbol removal and replacement
with an identical one that has a versioned tag).
I think the right answer for people encountering this condition is to
just rebuild. Otherwise, we are creating an environment in which we are
implicitly communicating to users that we are maintaining a modicum of
stability in experimental symobls.
A few more nits in line
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> Cc: Neil Horman <nhorman@tuxdriver.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>
> Cc: Luca Boccassi <bluca@debian.org>
> Cc: David Marchand <david.marchand@redhat.com>
> Cc: Bruce Richardson <bruce.richardson@intel.com>
> Cc: Ian Stokes <ian.stokes@intel.com>
> Cc: Eelco Chaudron <echaudro@redhat.com>
>
> I didn't like the idea of duplicating the symbol but not able to find to
> alias it without duplicating, if there is a way please share.
>
> Also not tested with old binaries, only verified the symbols in new
> binaries.
> ---
> buildtools/check-experimental-syms.sh | 3 +-
> .../common/include/rte_function_versioning.h | 4 ++
> lib/librte_meter/rte_meter.c | 59 ++++++++++++++++++-
> lib/librte_meter/rte_meter_version.map | 8 +++
> lib/meson.build | 2 +-
> 5 files changed, 71 insertions(+), 5 deletions(-)
>
> diff --git a/buildtools/check-experimental-syms.sh b/buildtools/check-experimental-syms.sh
> index f3603e5ba..35c4bf006 100755
> --- a/buildtools/check-experimental-syms.sh
> +++ b/buildtools/check-experimental-syms.sh
> @@ -26,7 +26,8 @@ ret=0
> for SYM in `$LIST_SYMBOL -S EXPERIMENTAL $MAPFILE |cut -d ' ' -f 3`
> do
> if grep -q "\.text.*[[:space:]]$SYM$" $DUMPFILE &&
> - ! grep -q "\.text\.experimental.*[[:space:]]$SYM$" $DUMPFILE
> + ! grep -q "\.text\.experimental.*[[:space:]]$SYM$" $DUMPFILE &&
> + $LIST_SYMBOL -s $SYM $MAPFILE | grep -q EXPERIMENTAL
> then
> cat >&2 <<- END_OF_MESSAGE
> $SYM is not flagged as experimental
> diff --git a/lib/librte_eal/common/include/rte_function_versioning.h b/lib/librte_eal/common/include/rte_function_versioning.h
> index c924351d5..ab102b06e 100644
> --- a/lib/librte_eal/common/include/rte_function_versioning.h
> +++ b/lib/librte_eal/common/include/rte_function_versioning.h
> @@ -46,6 +46,9 @@
> */
> #define VERSION_SYMBOL(b, e, n) __asm__(".symver " RTE_STR(b) RTE_STR(e) ", " RTE_STR(b) "@DPDK_" RTE_STR(n))
>
> +
> +#define VERSION_SYMBOL_EXPERIMENTAL(b, e) __asm__(".symver " RTE_STR(b) RTE_STR(e) ", " RTE_STR(b) "@EXPERIMENTAL")
> +
I don't know that you want to make a explicit clone of this macro.
instead what you might want to do is something like:
#define __VERSION_SYMBOL(b, e, n, t) __asm__(".symver " RTE_STR(b) RTE_STR(e) ", " RTE_STR(b) "@" RTE_STR(t) "_" RTE_STR(n))
#define VERSION_SYMBOL(b, e, n) __VERSION_SYMBOL(b, e, n, DPDK)
Thats not exactly right, but you get the idea, then you can emulate
VERSION_SYMBOL_EXPERIMENTAL with __VERSION_SYMBOL(b, e, n, EXPERIMENTAL)
> /*
> * BIND_DEFAULT_SYMBOL
> * Creates a symbol version entry instructing the linker to bind references to
> @@ -79,6 +82,7 @@
> * No symbol versioning in use
> */
> #define VERSION_SYMBOL(b, e, n)
> +#define VERSION_SYMBOL_EXPERIMENTAL(b, e)
> #define __vsym
> #define BIND_DEFAULT_SYMBOL(b, e, n)
> #define MAP_STATIC_SYMBOL(f, p) f __attribute__((alias(RTE_STR(p))))
> diff --git a/lib/librte_meter/rte_meter.c b/lib/librte_meter/rte_meter.c
> index da01429a8..5244537fa 100644
> --- a/lib/librte_meter/rte_meter.c
> +++ b/lib/librte_meter/rte_meter.c
> @@ -9,6 +9,7 @@
> #include <rte_common.h>
> #include <rte_log.h>
> #include <rte_cycles.h>
> +#include <rte_function_versioning.h>
>
> #include "rte_meter.h"
>
> @@ -119,8 +120,8 @@ rte_meter_trtcm_config(struct rte_meter_trtcm *m,
> return 0;
> }
>
> -int
> -rte_meter_trtcm_rfc4115_profile_config(
> +static int
> +rte_meter_trtcm_rfc4115_profile_config_(
> struct rte_meter_trtcm_rfc4115_profile *p,
> struct rte_meter_trtcm_rfc4115_params *params)
> {
> @@ -145,7 +146,35 @@ rte_meter_trtcm_rfc4115_profile_config(
> }
>
> int
> -rte_meter_trtcm_rfc4115_config(
> +rte_meter_trtcm_rfc4115_profile_config_s(
> + struct rte_meter_trtcm_rfc4115_profile *p,
> + struct rte_meter_trtcm_rfc4115_params *params);
> +int
> +rte_meter_trtcm_rfc4115_profile_config_s(
> + struct rte_meter_trtcm_rfc4115_profile *p,
> + struct rte_meter_trtcm_rfc4115_params *params)
> +{
> + return rte_meter_trtcm_rfc4115_profile_config_(p, params);
> +}
> +BIND_DEFAULT_SYMBOL(rte_meter_trtcm_rfc4115_profile_config, _s, 20.0.1);
> +MAP_STATIC_SYMBOL(int rte_meter_trtcm_rfc4115_profile_config(struct rte_meter_trtcm_rfc4115_profile *p,
> + struct rte_meter_trtcm_rfc4115_params *params), rte_meter_trtcm_rfc4115_profile_config_s);
> +
> +int
> +rte_meter_trtcm_rfc4115_profile_config_e(
> + struct rte_meter_trtcm_rfc4115_profile *p,
> + struct rte_meter_trtcm_rfc4115_params *params);
> +int
> +rte_meter_trtcm_rfc4115_profile_config_e(
> + struct rte_meter_trtcm_rfc4115_profile *p,
> + struct rte_meter_trtcm_rfc4115_params *params)
> +{
> + return rte_meter_trtcm_rfc4115_profile_config_(p, params);
> +}
> +VERSION_SYMBOL_EXPERIMENTAL(rte_meter_trtcm_rfc4115_profile_config, _e);
> +
> +static int
> +rte_meter_trtcm_rfc4115_config_(
> struct rte_meter_trtcm_rfc4115 *m,
> struct rte_meter_trtcm_rfc4115_profile *p)
> {
> @@ -160,3 +189,27 @@ rte_meter_trtcm_rfc4115_config(
>
> return 0;
> }
> +
> +int
> +rte_meter_trtcm_rfc4115_config_s(struct rte_meter_trtcm_rfc4115 *m,
> + struct rte_meter_trtcm_rfc4115_profile *p);
> +int
> +rte_meter_trtcm_rfc4115_config_s(struct rte_meter_trtcm_rfc4115 *m,
> + struct rte_meter_trtcm_rfc4115_profile *p)
> +{
> + return rte_meter_trtcm_rfc4115_config_(m, p);
> +}
> +BIND_DEFAULT_SYMBOL(rte_meter_trtcm_rfc4115_config, _s, 20.0.1);
> +MAP_STATIC_SYMBOL(int rte_meter_trtcm_rfc4115_config(struct rte_meter_trtcm_rfc4115 *m,
> + struct rte_meter_trtcm_rfc4115_profile *p), rte_meter_trtcm_rfc4115_config_s);
> +
> +int
> +rte_meter_trtcm_rfc4115_config_e(struct rte_meter_trtcm_rfc4115 *m,
> + struct rte_meter_trtcm_rfc4115_profile *p);
> +int
> +rte_meter_trtcm_rfc4115_config_e(struct rte_meter_trtcm_rfc4115 *m,
> + struct rte_meter_trtcm_rfc4115_profile *p)
> +{
> + return rte_meter_trtcm_rfc4115_config_(m, p);
> +}
> +VERSION_SYMBOL_EXPERIMENTAL(rte_meter_trtcm_rfc4115_config, _e);
> diff --git a/lib/librte_meter/rte_meter_version.map b/lib/librte_meter/rte_meter_version.map
> index fc12cc0bf..b4b043174 100644
> --- a/lib/librte_meter/rte_meter_version.map
> +++ b/lib/librte_meter/rte_meter_version.map
> @@ -20,4 +20,12 @@ DPDK_20.0.1 {
> rte_meter_trtcm_rfc4115_color_blind_check;
> rte_meter_trtcm_rfc4115_config;
> rte_meter_trtcm_rfc4115_profile_config;
> +
> } DPDK_20.0;
> +
> +EXPERIMENTAL {
> + global:
> +
> + rte_meter_trtcm_rfc4115_config;
> + rte_meter_trtcm_rfc4115_profile_config;
> +};
> diff --git a/lib/meson.build b/lib/meson.build
> index 0af3efab2..553964831 100644
> --- a/lib/meson.build
> +++ b/lib/meson.build
> @@ -48,7 +48,7 @@ foreach l:libraries
> reason = '<unknown reason>' # set if build == false to explain why
> name = l
> allow_experimental_apis = false
> - use_function_versioning = false
> + use_function_versioning = true
> sources = []
> headers = []
> includes = []
> --
> 2.24.1
>
>
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-02-02 13:05 9% ` Thomas Monjalon
@ 2020-02-02 14:41 4% ` Ananyev, Konstantin
2020-02-03 9:30 7% ` Ferruh Yigit
0 siblings, 1 reply; 200+ results
From: Ananyev, Konstantin @ 2020-02-02 14:41 UTC (permalink / raw)
To: Thomas Monjalon, Akhil Goyal, Yigit, Ferruh, Trahe, Fiona
Cc: David Marchand, Anoob Joseph, Kusztal, ArkadiuszX, dev,
Richardson, Bruce, nhorman, Mcnamara, John, dodji,
Andrew Rybchenko, Trahe, Fiona, aconole
> 31/01/2020 15:16, Trahe, Fiona:
> > On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> > > 30/01/2020 17:09, Ferruh Yigit:
> > > > On 1/29/2020 8:13 PM, Akhil Goyal wrote:
> > > > >
> > > > > I believe these enums will be used only in case of ASYM case which is experimental.
> > > >
> > > > Independent from being experiment and not, this shouldn't be a problem, I think
> > > > this is a false positive.
> > > >
> > > > The ABI break can happen when a struct has been shared between the application
> > > > and the library (DPDK) and the layout of that memory know differently by
> > > > application and the library.
> > > >
> > > > Here in all cases, there is no layout/size change.
> > > >
> > > > As to the value changes of the enums, since application compiled with old DPDK,
> > > > it will know only up to '6', 7 and more means invalid to the application. So it
> > > > won't send these values also it should ignore these values from library. Only
> > > > consequence is old application won't able to use new features those new enums
> > > > provide but that is expected/normal.
> > >
> > > If library give higher value than expected by the application,
> > > if the application uses this value as array index,
> > > there can be an access out of bounds.
> >
> > [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
> > But for the same issue with sym crypto below, I believe Ferruh's explanation makes
> > sense and I don't see how there can be an API breakage.
> > So if an application hasn't compiled against the new lib it will be still using the old value
> > which will be within bounds. If it's picking up the higher new value from the lib it must
> > have been compiled against the lib so shouldn't have problems.
>
> You say there is no ABI issue because the application will be re-compiled
> for the updated library. Indeed, compilation fixes compatibility issues.
> But this is not relevant for ABI compatibility.
> ABI compatibility means we can upgrade the library without recompiling
> the application and it must work.
> You think it is a false positive because you assume the application
> "picks" the new value. I think you miss the case where the new value
> is returned by a function in the upgraded library.
>
> > There are also no structs on the API which contain arrays using this
> > for sizing, so I don't see an opportunity for an appl to have a
> > mismatch in memory addresses.
>
> Let me demonstrate where the API may "use" the new value
> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
>
> Once upon a time a DPDK application counting the number of devices
> supporting each AEAD algo (in order to find the best supported algo).
> It is done in an array indexed by algo id:
> int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
> The application is compiled with DPDK 19.11,
> where RTE_CRYPTO_AEAD_LIST_END = 3.
> So the size of the application array aead_dev_count is 3.
> This binary is run with DPDK 20.02,
> where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
> When calling rte_cryptodev_info_get() on a device QAT_GEN3,
> rte_cryptodev_info.capabilities.sym.aead.algo is set to
> RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
> The application uses this value:
> ++ aead_dev_count[info.capabilities.sym.aead.algo];
> The application is crashing because of out of bound access.
I'd say this is an example of bad written app.
It probably should check that returned by library value doesn't
exceed its internal array size.
Konstantin
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-01-31 14:16 4% ` Trahe, Fiona
@ 2020-02-02 13:05 9% ` Thomas Monjalon
2020-02-02 14:41 4% ` Ananyev, Konstantin
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-02-02 13:05 UTC (permalink / raw)
To: Akhil Goyal, Yigit, Ferruh, Trahe, Fiona
Cc: David Marchand, Anoob Joseph, Kusztal, ArkadiuszX, dev,
Richardson, Bruce, nhorman, Mcnamara, John, dodji,
Andrew Rybchenko, Trahe, Fiona, aconole
31/01/2020 15:16, Trahe, Fiona:
> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> > 30/01/2020 17:09, Ferruh Yigit:
> > > On 1/29/2020 8:13 PM, Akhil Goyal wrote:
> > > >
> > > > I believe these enums will be used only in case of ASYM case which is experimental.
> > >
> > > Independent from being experiment and not, this shouldn't be a problem, I think
> > > this is a false positive.
> > >
> > > The ABI break can happen when a struct has been shared between the application
> > > and the library (DPDK) and the layout of that memory know differently by
> > > application and the library.
> > >
> > > Here in all cases, there is no layout/size change.
> > >
> > > As to the value changes of the enums, since application compiled with old DPDK,
> > > it will know only up to '6', 7 and more means invalid to the application. So it
> > > won't send these values also it should ignore these values from library. Only
> > > consequence is old application won't able to use new features those new enums
> > > provide but that is expected/normal.
> >
> > If library give higher value than expected by the application,
> > if the application uses this value as array index,
> > there can be an access out of bounds.
>
> [Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
> But for the same issue with sym crypto below, I believe Ferruh's explanation makes
> sense and I don't see how there can be an API breakage.
> So if an application hasn't compiled against the new lib it will be still using the old value
> which will be within bounds. If it's picking up the higher new value from the lib it must
> have been compiled against the lib so shouldn't have problems.
You say there is no ABI issue because the application will be re-compiled
for the updated library. Indeed, compilation fixes compatibility issues.
But this is not relevant for ABI compatibility.
ABI compatibility means we can upgrade the library without recompiling
the application and it must work.
You think it is a false positive because you assume the application
"picks" the new value. I think you miss the case where the new value
is returned by a function in the upgraded library.
> There are also no structs on the API which contain arrays using this
> for sizing, so I don't see an opportunity for an appl to have a
> mismatch in memory addresses.
Let me demonstrate where the API may "use" the new value
RTE_CRYPTO_AEAD_CHACHA20_POLY1305 and how it impacts the application.
Once upon a time a DPDK application counting the number of devices
supporting each AEAD algo (in order to find the best supported algo).
It is done in an array indexed by algo id:
int aead_dev_count[RTE_CRYPTO_AEAD_LIST_END];
The application is compiled with DPDK 19.11,
where RTE_CRYPTO_AEAD_LIST_END = 3.
So the size of the application array aead_dev_count is 3.
This binary is run with DPDK 20.02,
where RTE_CRYPTO_AEAD_CHACHA20_POLY1305 = 3.
When calling rte_cryptodev_info_get() on a device QAT_GEN3,
rte_cryptodev_info.capabilities.sym.aead.algo is set to
RTE_CRYPTO_AEAD_CHACHA20_POLY1305 (= 3).
The application uses this value:
++ aead_dev_count[info.capabilities.sym.aead.algo];
The application is crashing because of out of bound access.
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH v3 4/4] add ABI checks
2020-01-30 22:32 4% ` Thomas Monjalon
@ 2020-02-01 15:29 4% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-02-01 15:29 UTC (permalink / raw)
To: Thomas Monjalon
Cc: dev, Bruce Richardson, Kevin Laatz, Aaron Conole, Neil Horman,
Akhil Goyal, Anoob Joseph, Luca Boccassi, Trahe, Fiona, Yigit,
Ferruh, Michael Santana, John McNamara, Marko Kovacevic
On Thu, Jan 30, 2020 at 11:33 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 30/01/2020 17:00, David Marchand:
> > Enabling them requires a configuration that will trigger the ABI dumps
> > generation as part of the existing devtools/test-build.sh and
> > devtools/test-meson-builds.sh scripts.
> [...]
> > --- a/devtools/test-meson-builds.sh
> > +++ b/devtools/test-meson-builds.sh
> > if [ -f "$builddir/build.ninja" ] ; then
> > + # for existing environments, force debugoptimized so that ABI
> > + # checks can run
> > + $MESON configure --buildtype=debugoptimized $builddir
>
> This is forcing meson to re-run each time, even if the buildtype is
> already "debugoptimized".
> Please query meson configuration to avoid useless re-run:
>
> $MESON configure $builddir | awk '$1=="buildtype" {print $2}'
>
>
Good catch, will fix.
--
David Marchand
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH 2/2] eal: add eal_parse_optionlist to parse user input
2020-01-31 11:25 0% ` Govindharajan, Hariprasad
@ 2020-01-31 14:42 0% ` Govindharajan, Hariprasad
1 sibling, 0 replies; 200+ results
From: Govindharajan, Hariprasad @ 2020-01-31 14:42 UTC (permalink / raw)
To: Stephen Hemminger, Yigit, Ferruh; +Cc: David Marchand, Thomas Monjalon, dev
> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Wednesday, January 29, 2020 7:20 PM
> To: Yigit, Ferruh <ferruh.yigit@intel.com>
> Cc: David Marchand <david.marchand@redhat.com>; Thomas Monjalon
> <thomas@monjalon.net>; Govindharajan, Hariprasad
> <hariprasad.govindharajan@intel.com>; dev <dev@dpdk.org>
> Subject: Re: [dpdk-dev] [PATCH 2/2] eal: add eal_parse_optionlist to parse
> user input
>
> On Wed, 29 Jan 2020 18:07:05 +0000
> Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> > On 1/29/2020 5:44 PM, David Marchand wrote:
> > > On Tue, Jan 28, 2020 at 6:35 PM Ferruh Yigit <ferruh.yigit@intel.com>
> wrote:
> > >> On 1/27/2020 10:30 AM, Hariprasad Govindharajan wrote:
> > >>> In current version, there is a function which parses the corelist
> > >>> based on user value. A new generic function eal_parse_optionlist
> > >>> is added which will parse corelist as well as similar user input
> > >>> so that we can use it as a public API too.
> > >>>
> > >>> Signed-off-by: Hariprasad Govindharajan
> > >>> <hariprasad.govindharajan@intel.com>
> > >>
> > >> Hi David,
> > >>
> > >> Overall this patchset is to add '--portlist' command to testpmd and
> > >> remove existing 64 port limitation.
> > >>
> > >> And in this patch re-uses the exiting parser function in eal and
> > >> converts it to API, question is if eal is good place to have this API, what
> do you think about it?
> > >
> > > Exporting string parsers from the EAL has little value.
> > > Ok we avoid code duplication (and I can see other places in the tree
> > > where it might be used), but in the end we will have to maintain
> > > this API in the ABI when it enters the stable ABI.
> > >
> > > I am for avoiding this.
> > >
> > >
> >
> > The same function can be used in some sample applications too (which
> > are using port mask), so instead of duplicating it multiple times, it
> > would be nice to have this function somewhere that applications can use.
> >
> > Does it makes sense to have a rte_util.c (under in eal or as a
> > separate library) to have this kind of application helper functions?
>
> It makes sense to have a rte library that handles arbitrary size bitvector and
> has string handling functions. Kind of like what kernel has for the cpuset
> parsing. This could be used for cpus in EAL and port-masks or other arrays in
> applications.
>
> But just doing copy/paste of existing code without thinking about how API
> should work is a bad idea.
[Govindharajan, Hariprasad]
Hi,
PLEASE IGNORE MY PREVIOUS EMAIL.
I am planning to move the existing parser function to the testpmd file instead of keeping it in the eal and will revert the eal back.
Also, I am planning to create an util file in eal with this parser and do a RFC.
@Stephen Hemminger, We already investigated the existing function and then decided to re-use it as seen in the patch. For creating an API, is there any other specific requirements that should be addressed? Please clarify us.
Thanks
G Hariprasad
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-01-30 20:18 4% ` Thomas Monjalon
2020-01-31 9:03 7% ` Ferruh Yigit
@ 2020-01-31 14:16 4% ` Trahe, Fiona
2020-02-02 13:05 9% ` Thomas Monjalon
1 sibling, 1 reply; 200+ results
From: Trahe, Fiona @ 2020-01-31 14:16 UTC (permalink / raw)
To: Thomas Monjalon, Akhil Goyal, David Marchand, Anoob Joseph,
Kusztal, ArkadiuszX, Yigit, Ferruh
Cc: dev, Richardson, Bruce, nhorman, Mcnamara, John, dodji,
Andrew Rybchenko, Trahe, Fiona
> > > I believe these enums will be used only in case of ASYM case which is experimental.
> >
> > Independent from being experiment and not, this shouldn't be a problem, I think
> > this is a false positive.
> >
> > The ABI break can happen when a struct has been shared between the application
> > and the library (DPDK) and the layout of that memory know differently by
> > application and the library.
> >
> > Here in all cases, there is no layout/size change.
> >
> > As to the value changes of the enums, since application compiled with old DPDK,
> > it will know only up to '6', 7 and more means invalid to the application. So it
> > won't send these values also it should ignore these values from library. Only
> > consequence is old application won't able to use new features those new enums
> > provide but that is expected/normal.
>
> If library give higher value than expected by the application,
> if the application uses this value as array index,
> there can be an access out of bounds.
[Fiona] All asymmetric APIs are experimental so above shouldn't be a problem.
But for the same issue with sym crypto below, I believe Ferruh's explanation makes
sense and I don't see how there can be an API breakage.
So if an application hasn't compiled against the new lib it will be still using the old value
which will be within bounds. If it's picking up the higher new value from the lib it must
have been compiled against the lib so shouldn't have problems.
There are also no structs on the API which contain arrays using this for sizing, so I don't see an
opportunity for an appl to have a mismatch in memory addresses.
> > >> [C]'function int
> > >> rte_cryptodev_get_aead_algo_enum(rte_crypto_aead_algorithm*, const
> > >> char*)' at rte_cryptodev.c:239:1 has some indirect sub-type changes:
> > >> parameter 1 of type 'rte_crypto_aead_algorithm*' has sub-type changes:
> > >> in pointed to type 'enum rte_crypto_aead_algorithm' at
> > >> rte_crypto_sym.h:346:1:
> > >> type size hasn't changed
> > >> 1 enumerator insertion:
> > >> 'rte_crypto_aead_algorithm::RTE_CRYPTO_AEAD_CHACHA20_POLY1305'
> > >> value '3'
> > >> 1 enumerator change:
> > >> 'rte_crypto_aead_algorithm::RTE_CRYPTO_AEAD_LIST_END' from
> > >> value '3' to '4' at rte_crypto_sym.h:346:1
> >
> > Same as above, no layout change.
> >
> > >>
> > >>
> > >> [C]'const char* rte_crypto_aead_algorithm_strings[1]' was changed at
> > >> rte_crypto_sym.h:358:1:
> > >> size of symbol (in bytes) changed from 24 to 32
> > >>
> >
> > The shared memory size changes, but this is global variable in the library, and
> > the values application can request 'RTE_CRYPTO_AEAD_AES_CCM' &
> > 'RTE_CRYPTO_AEAD_AES_GCM' is already there, so there is no backward
> > compatibility issue here.
>
> For this one, I don't know what is the breakage.
>
>
> > > +Fiona and Arek
> > >
> > > We may need to revert the chacha-poly patches.
> > >
> >
> > I don't see any ABI break in this case, can someone explain if I am missing
> > anything here?
>
>
>
>
^ permalink raw reply [relevance 4%]
* [dpdk-dev] Minutes of Technical Board Meeting, 2020-01-29
@ 2020-01-31 12:36 4% Olivier Matz
0 siblings, 0 replies; 200+ results
From: Olivier Matz @ 2020-01-31 12:36 UTC (permalink / raw)
To: dev
Minutes of Technical Board Meeting, 2020-01-29
Members Attending
-----------------
- Bruce
- Ferruh
- Hemant
- Honnappa
- Jerin
- Kevin
- Konstantin
- Olivier (chair)
- Stephen
- Thomas
NOTE: The technical board meetings every second Wednesday on IRC
channel #dpdk-board, at 3pm UTC. Meetings are public and DPDK community
members are welcome to attend.
NOTE: Next meeting will be on Wednesday 2020-02-12 @3pm UTC, and will be
chaired by Stephen.
1/ Integration of SAD capability in ipsec-secgw
With v5, performance drop is less than 2% on x86, so it can be
integrated if the performance is confirmed on other archs.
2/ Guidelines for examples and apps distributed in DPDK repository
- In dpdk/examples directory, the applications should demonstrate
the API usage in priority, and the performance (ideally both).
- If both goals cannot be reconciled in the example application, a
performance application can be created in dpdk/app (for main
features).
3/ ICC support in DPDK
- It was already agreed that compilers supported in DPDK are GCC and
CLANG: http://mails.dpdk.org/archives/dev/2019-June/135847.html
- However patches for ICC support are accepted.
- An ICC compilation warning should not be fatal.
(note: this is already the default for meson builds).
- Some ICC compilation warnings will be disabled at a global level
4/ Name of repository for kernel modules
On 2019-11-06, we decided to move igb_uio in a separate repository
before 20.11:
http://mails.dpdk.org/archives/dev/2019-November/151763.html
A repository will also be needed to host netuio for Windows.
The name of this repository will be *dpdk-kmods*.
AR for Jerin: write the deprecation notice for removal of igb_uio.
already done: http://inbox.dpdk.org/dev/20200130165853.693270-1-jerinj@marvell.com
5/ Reorganize EAL to ease windows port integration
Thomas will propose a patch to simplify the layout of
dpdk/lib/librte_eal.
6/ ABI topics
In 20.02-rc1, some possible ABI breaks were found in meter, crypto
and security libraries: to be checked and fixed before the release.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH 2/2] eal: add eal_parse_optionlist to parse user input
@ 2020-01-31 11:25 0% ` Govindharajan, Hariprasad
2020-01-31 14:42 0% ` Govindharajan, Hariprasad
1 sibling, 0 replies; 200+ results
From: Govindharajan, Hariprasad @ 2020-01-31 11:25 UTC (permalink / raw)
To: Stephen Hemminger, Yigit, Ferruh; +Cc: David Marchand, Thomas Monjalon, dev
Hi,
I am planning to move the existing parser function to the testpmd file instead of keeping it in the eal and will revert the eal back.
Also, I am planning to create an util file in eal with this parser and do a RFC.
@Stephen Hemminger, We already investigated the existing function and then decided to re-use it as seen in the patch. For creating an API, is there any other specific requirements
that should be addressed? Please clarify us.
Thanks
G Hariprasad
-----Original Message-----
From: Stephen Hemminger <stephen@networkplumber.org>
Sent: Wednesday, January 29, 2020 7:20 PM
To: Yigit, Ferruh <ferruh.yigit@intel.com>
Cc: David Marchand <david.marchand@redhat.com>; Thomas Monjalon <thomas@monjalon.net>; Govindharajan, Hariprasad <hariprasad.govindharajan@intel.com>; dev <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH 2/2] eal: add eal_parse_optionlist to parse user input
On Wed, 29 Jan 2020 18:07:05 +0000
Ferruh Yigit <ferruh.yigit@intel.com> wrote:
> On 1/29/2020 5:44 PM, David Marchand wrote:
> > On Tue, Jan 28, 2020 at 6:35 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
> >> On 1/27/2020 10:30 AM, Hariprasad Govindharajan wrote:
> >>> In current version, there is a function which parses the corelist
> >>> based on user value. A new generic function eal_parse_optionlist
> >>> is added which will parse corelist as well as similar user input
> >>> so that we can use it as a public API too.
> >>>
> >>> Signed-off-by: Hariprasad Govindharajan
> >>> <hariprasad.govindharajan@intel.com>
> >>
> >> Hi David,
> >>
> >> Overall this patchset is to add '--portlist' command to testpmd and
> >> remove existing 64 port limitation.
> >>
> >> And in this patch re-uses the exiting parser function in eal and
> >> converts it to API, question is if eal is good place to have this API, what do you think about it?
> >
> > Exporting string parsers from the EAL has little value.
> > Ok we avoid code duplication (and I can see other places in the tree
> > where it might be used), but in the end we will have to maintain
> > this API in the ABI when it enters the stable ABI.
> >
> > I am for avoiding this.
> >
> >
>
> The same function can be used in some sample applications too (which
> are using port mask), so instead of duplicating it multiple times, it
> would be nice to have this function somewhere that applications can use.
>
> Does it makes sense to have a rte_util.c (under in eal or as a
> separate library) to have this kind of application helper functions?
It makes sense to have a rte library that handles arbitrary size bitvector and has string handling functions. Kind of like what kernel has for the cpuset parsing. This could be used for cpus in EAL and port-masks or other arrays in applications.
But just doing copy/paste of existing code without thinking about how API should work is a bad idea.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-01-31 9:03 7% ` Ferruh Yigit
@ 2020-01-31 10:26 7% ` Ananyev, Konstantin
0 siblings, 0 replies; 200+ results
From: Ananyev, Konstantin @ 2020-01-31 10:26 UTC (permalink / raw)
To: Yigit, Ferruh, Thomas Monjalon, Akhil Goyal, David Marchand,
Anoob Joseph, Trahe, Fiona, Kusztal, ArkadiuszX
Cc: dev, Richardson, Bruce, nhorman, Mcnamara, John, dodji, Andrew Rybchenko
>
> On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> > 30/01/2020 17:09, Ferruh Yigit:
> >> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
> >>>
> >>>
> >>>>
> >>>> On Wed, Jan 29, 2020 at 7:10 PM Anoob Joseph <anoobj@marvell.com> wrote:
> >>>>> The asymmetric crypto library is experimental. Changes to experimental code
> >>>> paths is allowed, right?
> >>>>
> >>>> The asymmetric crypto enum is referenced by a function part of the stable ABI.
> >>>> It is possible to waive this enum, if we are sure no use out of the
> >>>> experimental asym crypto APIs is possible.
> >>>>
> >>>> The rest of the changes touch stable symbols.
> >>>>
> >>>> Adding the abidiff report:
> >>>>
> >>>> [C]'function void rte_cryptodev_info_get(uint8_t,
> >>>> rte_cryptodev_info*)' at rte_cryptodev.c:1115:1 has some indirect
> >>>> sub-type changes:
> >>>> parameter 2 of type 'rte_cryptodev_info*' has sub-type changes:
> >>>> in pointed to type 'struct rte_cryptodev_info' at rte_cryptodev.h:468:1:
> >>>> type size hasn't changed
> >>>> 1 data member change:
> >>>> type of 'const rte_cryptodev_capabilities*
> >>>> rte_cryptodev_info::capabilities' changed:
> >>>> in pointed to type 'const rte_cryptodev_capabilities':
> >>>> in unqualified underlying type 'struct
> >>>> rte_cryptodev_capabilities' at rte_cryptodev.h:176:1:
> >>>> type size hasn't changed
> >>>> 1 data member change:
> >>>> type of '__anonymous_union__ ' changed:
> >>>> type size hasn't changed
> >>>> 1 data member change:
> >>>> type of 'rte_cryptodev_asymmetric_capability
> >>>> __anonymous_union__::asym' changed:
> >>>> type size hasn't changed
> >>>> 1 data member change:
> >>>> type of
> >>>> 'rte_cryptodev_asymmetric_xform_capability
> >>>> rte_cryptodev_asymmetric_capability::xform_capa' changed:
> >>>> type size hasn't changed
> >>>> 1 data member change:
> >>>> type of 'rte_crypto_asym_xform_type
> >>>> rte_cryptodev_asymmetric_xform_capability::xform_type' changed:
> >>>> type size hasn't changed
> >>>> 2 enumerator insertions:
> >>>>
> >>>> 'rte_crypto_asym_xform_type::RTE_CRYPTO_ASYM_XFORM_ECDSA' value '7'
> >>>>
> >>>> 'rte_crypto_asym_xform_type::RTE_CRYPTO_ASYM_XFORM_ECPM' value '8'
> >>>> 1 enumerator change:
> >>>>
> >>>> 'rte_crypto_asym_xform_type::RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END'
> >>>> from
> >>>> value '7' to '9' at rte_crypto_asym.h:60:1
> >>>>
> >>>
> >>> I believe these enums will be used only in case of ASYM case which is experimental.
> >>
> >> Independent from being experiment and not, this shouldn't be a problem, I think
> >> this is a false positive.
> >>
> >> The ABI break can happen when a struct has been shared between the application
> >> and the library (DPDK) and the layout of that memory know differently by
> >> application and the library.
> >>
> >> Here in all cases, there is no layout/size change.
> >>
> >> As to the value changes of the enums, since application compiled with old DPDK,
> >> it will know only up to '6', 7 and more means invalid to the application. So it
> >> won't send these values also it should ignore these values from library. Only
> >> consequence is old application won't able to use new features those new enums
> >> provide but that is expected/normal.
> >
> > If library give higher value than expected by the application,
> > if the application uses this value as array index,
> > there can be an access out of bounds.
>
> First this concern is not an ABI break concern, but application should ignore
> any value bigger than the MAX value it knows.
> Otherwise this would mean we can't add any new enum or define to the project,
> which is wrong I believe.
>
> >
> >
> >>>> [C]'function int
> >>>> rte_cryptodev_get_aead_algo_enum(rte_crypto_aead_algorithm*, const
> >>>> char*)' at rte_cryptodev.c:239:1 has some indirect sub-type changes:
> >>>> parameter 1 of type 'rte_crypto_aead_algorithm*' has sub-type changes:
> >>>> in pointed to type 'enum rte_crypto_aead_algorithm' at
> >>>> rte_crypto_sym.h:346:1:
> >>>> type size hasn't changed
> >>>> 1 enumerator insertion:
> >>>> 'rte_crypto_aead_algorithm::RTE_CRYPTO_AEAD_CHACHA20_POLY1305'
> >>>> value '3'
> >>>> 1 enumerator change:
> >>>> 'rte_crypto_aead_algorithm::RTE_CRYPTO_AEAD_LIST_END' from
> >>>> value '3' to '4' at rte_crypto_sym.h:346:1
> >>
> >> Same as above, no layout change.
> >>
> >>>>
> >>>>
> >>>> [C]'const char* rte_crypto_aead_algorithm_strings[1]' was changed at
> >>>> rte_crypto_sym.h:358:1:
> >>>> size of symbol (in bytes) changed from 24 to 32
> >>>>
> >>
> >> The shared memory size changes, but this is global variable in the library, and
> >> the values application can request 'RTE_CRYPTO_AEAD_AES_CCM' &
> >> 'RTE_CRYPTO_AEAD_AES_GCM' is already there, so there is no backward
> >> compatibility issue here.
> >
> > For this one, I don't know what is the breakage.
Reading through this report, I am also don't see why it is considered as ABI breakage.
Yes, size of rte_crypto_aead_algorithm_strings[] has changed, but this array is not public one.
Also I don't see any place where we use RTE_CRYPTO_AEAD_LIST_END to define array size
in our public API.
At first glance it looks like false positive to me.
Do I miss something obvious here?
Konstantin
> >
> >
> >>> +Fiona and Arek
> >>>
> >>> We may need to revert the chacha-poly patches.
> >>>
> >>
> >> I don't see any ABI break in this case, can someone explain if I am missing
> >> anything here?
> >
> >
> >
> >
> >
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-01-31 9:07 4% ` Ferruh Yigit
@ 2020-01-31 9:37 4% ` Ananyev, Konstantin
0 siblings, 0 replies; 200+ results
From: Ananyev, Konstantin @ 2020-01-31 9:37 UTC (permalink / raw)
To: Yigit, Ferruh, Thomas Monjalon, Anoob Joseph, akhil.goyal, Trahe, Fiona
Cc: dev, David Marchand, Richardson, Bruce, nhorman, Mcnamara, John,
Kusztal, ArkadiuszX
> -----Original Message-----
> From: Yigit, Ferruh <ferruh.yigit@intel.com>
> Sent: Friday, January 31, 2020 9:07 AM
> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; Thomas Monjalon <thomas@monjalon.net>; Anoob Joseph
> <anoobj@marvell.com>; akhil.goyal@nxp.com; Trahe, Fiona <fiona.trahe@intel.com>
> Cc: dev@dpdk.org; David Marchand <david.marchand@redhat.com>; Richardson, Bruce <bruce.richardson@intel.com>;
> nhorman@tuxdriver.com; Mcnamara, John <john.mcnamara@intel.com>; Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
>
> On 1/30/2020 11:49 PM, Ananyev, Konstantin wrote:
> >
> >
> >> -----Original Message-----
> >> From: dev <dev-bounces@dpdk.org> On Behalf Of Thomas Monjalon
> >> Sent: Thursday, January 30, 2020 4:00 PM
> >> To: Anoob Joseph <anoobj@marvell.com>; akhil.goyal@nxp.com; Trahe, Fiona <fiona.trahe@intel.com>
> >> Cc: dev@dpdk.org; David Marchand <david.marchand@redhat.com>; Richardson, Bruce <bruce.richardson@intel.com>;
> >> nhorman@tuxdriver.com; Mcnamara, John <john.mcnamara@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>; Kusztal, ArkadiuszX
> >> <arkadiuszx.kusztal@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>
> >> Subject: Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
> >>
> >> 30/01/2020 14:06, Trahe, Fiona:
> >>> We were unaware the LIST_END change could constitute an ABI breakage, but can see how it affects the array size when picked up.
> >>> We're exploring options.
> >>>
> >>> I agree with Anoob's point that if we don't allow the LIST_END to be modified, then it means no feature can be implemented without
> ABI
> >> breakage.
> >>> Anyone object to removing those LIST_END elements - or have a better suggestion? Would have to be in 20.11 I suppose.
> >>
> >> Yes, having max value right after the last value is ridiculous,
> >> it prevents adding any value.
> >> In 20.11, we should remove all these *_END and *_MAX from API enums
> >> and replace them with a separate #define with reasonnable maximums.
> >>
> >
> > I think we'd better avoid public structs that have array of _MAX elems in them.
> >
>
> That should fix, but we need to wait for 20.11 for the change, and what should
> be the new array size?
Make it dynamic whenever possible?
Make Input/output args to provide both pointer and size,
or use some predefined value for terminating element (NULL, -1, etc.)?
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [RFC v2] meter: fix ABI break due to experimental tag removal
@ 2020-01-31 9:25 4% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-01-31 9:25 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Neil Horman, Cristian Dumitrescu, Eelco Chaudron, dev,
Luca Boccassi, David Marchand, Bruce Richardson, Ian Stokes,
Andrzej Ostruszka
On 1/30/2020 12:33 PM, Thomas Monjalon wrote:
> Hi,
>
> I disagree with the need of this patch.
> The symbol was experimental, meaning we can change it.
> Removing experimental tag is not an ABI break.
In theory what you said is correct, this is experimental API and anything can
happen to it, this is the contract.
And when we need to change an experimental API, the users will be affected,
there is no escape from it.
But for this case limitation is more mechanical I believe,
because API is not changing at all, and it is becoming mature.
So we are breaking the user application because the experimental API they are
using becoming mature without any change.
This looks annoying from the user perspective, and if we can prevent this break,
I am for preventing it.
>
>
> 29/01/2020 17:43, Ferruh Yigit:
>> Duplicated the existing symbol and versioned one as experimental and
>> other as stable.
> [..]
>> --- a/lib/librte_meter/rte_meter_version.map
>> +++ b/lib/librte_meter/rte_meter_version.map
>> @@ -20,4 +20,12 @@ DPDK_20.0.1 {
>> rte_meter_trtcm_rfc4115_color_blind_check;
>> rte_meter_trtcm_rfc4115_config;
>> rte_meter_trtcm_rfc4115_profile_config;
>> +
>> } DPDK_20.0;
>> +
>> +EXPERIMENTAL {
>> + global:
>> +
>> + rte_meter_trtcm_rfc4115_config;
>> + rte_meter_trtcm_rfc4115_profile_config;
>> +};
>
>
>
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-01-30 23:49 4% ` Ananyev, Konstantin
@ 2020-01-31 9:07 4% ` Ferruh Yigit
2020-01-31 9:37 4% ` Ananyev, Konstantin
0 siblings, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-01-31 9:07 UTC (permalink / raw)
To: Ananyev, Konstantin, Thomas Monjalon, Anoob Joseph, akhil.goyal,
Trahe, Fiona
Cc: dev, David Marchand, Richardson, Bruce, nhorman, Mcnamara, John,
Kusztal, ArkadiuszX
On 1/30/2020 11:49 PM, Ananyev, Konstantin wrote:
>
>
>> -----Original Message-----
>> From: dev <dev-bounces@dpdk.org> On Behalf Of Thomas Monjalon
>> Sent: Thursday, January 30, 2020 4:00 PM
>> To: Anoob Joseph <anoobj@marvell.com>; akhil.goyal@nxp.com; Trahe, Fiona <fiona.trahe@intel.com>
>> Cc: dev@dpdk.org; David Marchand <david.marchand@redhat.com>; Richardson, Bruce <bruce.richardson@intel.com>;
>> nhorman@tuxdriver.com; Mcnamara, John <john.mcnamara@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>; Kusztal, ArkadiuszX
>> <arkadiuszx.kusztal@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>
>> Subject: Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
>>
>> 30/01/2020 14:06, Trahe, Fiona:
>>> We were unaware the LIST_END change could constitute an ABI breakage, but can see how it affects the array size when picked up.
>>> We're exploring options.
>>>
>>> I agree with Anoob's point that if we don't allow the LIST_END to be modified, then it means no feature can be implemented without ABI
>> breakage.
>>> Anyone object to removing those LIST_END elements - or have a better suggestion? Would have to be in 20.11 I suppose.
>>
>> Yes, having max value right after the last value is ridiculous,
>> it prevents adding any value.
>> In 20.11, we should remove all these *_END and *_MAX from API enums
>> and replace them with a separate #define with reasonnable maximums.
>>
>
> I think we'd better avoid public structs that have array of _MAX elems in them.
>
That should fix, but we need to wait for 20.11 for the change, and what should
be the new array size?
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-01-30 20:18 4% ` Thomas Monjalon
@ 2020-01-31 9:03 7% ` Ferruh Yigit
2020-01-31 10:26 7% ` Ananyev, Konstantin
2020-01-31 14:16 4% ` Trahe, Fiona
1 sibling, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-01-31 9:03 UTC (permalink / raw)
To: Thomas Monjalon, Akhil Goyal, David Marchand, Anoob Joseph,
Trahe, Fiona, Kusztal, ArkadiuszX
Cc: dev, bruce.richardson, nhorman, John McNamara, dodji, Andrew Rybchenko
On 1/30/2020 8:18 PM, Thomas Monjalon wrote:
> 30/01/2020 17:09, Ferruh Yigit:
>> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
>>>
>>>
>>>>
>>>> On Wed, Jan 29, 2020 at 7:10 PM Anoob Joseph <anoobj@marvell.com> wrote:
>>>>> The asymmetric crypto library is experimental. Changes to experimental code
>>>> paths is allowed, right?
>>>>
>>>> The asymmetric crypto enum is referenced by a function part of the stable ABI.
>>>> It is possible to waive this enum, if we are sure no use out of the
>>>> experimental asym crypto APIs is possible.
>>>>
>>>> The rest of the changes touch stable symbols.
>>>>
>>>> Adding the abidiff report:
>>>>
>>>> [C]'function void rte_cryptodev_info_get(uint8_t,
>>>> rte_cryptodev_info*)' at rte_cryptodev.c:1115:1 has some indirect
>>>> sub-type changes:
>>>> parameter 2 of type 'rte_cryptodev_info*' has sub-type changes:
>>>> in pointed to type 'struct rte_cryptodev_info' at rte_cryptodev.h:468:1:
>>>> type size hasn't changed
>>>> 1 data member change:
>>>> type of 'const rte_cryptodev_capabilities*
>>>> rte_cryptodev_info::capabilities' changed:
>>>> in pointed to type 'const rte_cryptodev_capabilities':
>>>> in unqualified underlying type 'struct
>>>> rte_cryptodev_capabilities' at rte_cryptodev.h:176:1:
>>>> type size hasn't changed
>>>> 1 data member change:
>>>> type of '__anonymous_union__ ' changed:
>>>> type size hasn't changed
>>>> 1 data member change:
>>>> type of 'rte_cryptodev_asymmetric_capability
>>>> __anonymous_union__::asym' changed:
>>>> type size hasn't changed
>>>> 1 data member change:
>>>> type of
>>>> 'rte_cryptodev_asymmetric_xform_capability
>>>> rte_cryptodev_asymmetric_capability::xform_capa' changed:
>>>> type size hasn't changed
>>>> 1 data member change:
>>>> type of 'rte_crypto_asym_xform_type
>>>> rte_cryptodev_asymmetric_xform_capability::xform_type' changed:
>>>> type size hasn't changed
>>>> 2 enumerator insertions:
>>>>
>>>> 'rte_crypto_asym_xform_type::RTE_CRYPTO_ASYM_XFORM_ECDSA' value '7'
>>>>
>>>> 'rte_crypto_asym_xform_type::RTE_CRYPTO_ASYM_XFORM_ECPM' value '8'
>>>> 1 enumerator change:
>>>>
>>>> 'rte_crypto_asym_xform_type::RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END'
>>>> from
>>>> value '7' to '9' at rte_crypto_asym.h:60:1
>>>>
>>>
>>> I believe these enums will be used only in case of ASYM case which is experimental.
>>
>> Independent from being experiment and not, this shouldn't be a problem, I think
>> this is a false positive.
>>
>> The ABI break can happen when a struct has been shared between the application
>> and the library (DPDK) and the layout of that memory know differently by
>> application and the library.
>>
>> Here in all cases, there is no layout/size change.
>>
>> As to the value changes of the enums, since application compiled with old DPDK,
>> it will know only up to '6', 7 and more means invalid to the application. So it
>> won't send these values also it should ignore these values from library. Only
>> consequence is old application won't able to use new features those new enums
>> provide but that is expected/normal.
>
> If library give higher value than expected by the application,
> if the application uses this value as array index,
> there can be an access out of bounds.
First this concern is not an ABI break concern, but application should ignore
any value bigger than the MAX value it knows.
Otherwise this would mean we can't add any new enum or define to the project,
which is wrong I believe.
>
>
>>>> [C]'function int
>>>> rte_cryptodev_get_aead_algo_enum(rte_crypto_aead_algorithm*, const
>>>> char*)' at rte_cryptodev.c:239:1 has some indirect sub-type changes:
>>>> parameter 1 of type 'rte_crypto_aead_algorithm*' has sub-type changes:
>>>> in pointed to type 'enum rte_crypto_aead_algorithm' at
>>>> rte_crypto_sym.h:346:1:
>>>> type size hasn't changed
>>>> 1 enumerator insertion:
>>>> 'rte_crypto_aead_algorithm::RTE_CRYPTO_AEAD_CHACHA20_POLY1305'
>>>> value '3'
>>>> 1 enumerator change:
>>>> 'rte_crypto_aead_algorithm::RTE_CRYPTO_AEAD_LIST_END' from
>>>> value '3' to '4' at rte_crypto_sym.h:346:1
>>
>> Same as above, no layout change.
>>
>>>>
>>>>
>>>> [C]'const char* rte_crypto_aead_algorithm_strings[1]' was changed at
>>>> rte_crypto_sym.h:358:1:
>>>> size of symbol (in bytes) changed from 24 to 32
>>>>
>>
>> The shared memory size changes, but this is global variable in the library, and
>> the values application can request 'RTE_CRYPTO_AEAD_AES_CCM' &
>> 'RTE_CRYPTO_AEAD_AES_GCM' is already there, so there is no backward
>> compatibility issue here.
>
> For this one, I don't know what is the breakage.
>
>
>>> +Fiona and Arek
>>>
>>> We may need to revert the chacha-poly patches.
>>>
>>
>> I don't see any ABI break in this case, can someone explain if I am missing
>> anything here?
>
>
>
>
>
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-01-30 15:59 4% ` Thomas Monjalon
2020-01-30 16:42 7% ` Ferruh Yigit
@ 2020-01-30 23:49 4% ` Ananyev, Konstantin
2020-01-31 9:07 4% ` Ferruh Yigit
1 sibling, 1 reply; 200+ results
From: Ananyev, Konstantin @ 2020-01-30 23:49 UTC (permalink / raw)
To: Thomas Monjalon, Anoob Joseph, akhil.goyal, Trahe, Fiona
Cc: dev, David Marchand, Richardson, Bruce, nhorman, Mcnamara, John,
Trahe, Fiona, Kusztal, ArkadiuszX, Yigit, Ferruh
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Thomas Monjalon
> Sent: Thursday, January 30, 2020 4:00 PM
> To: Anoob Joseph <anoobj@marvell.com>; akhil.goyal@nxp.com; Trahe, Fiona <fiona.trahe@intel.com>
> Cc: dev@dpdk.org; David Marchand <david.marchand@redhat.com>; Richardson, Bruce <bruce.richardson@intel.com>;
> nhorman@tuxdriver.com; Mcnamara, John <john.mcnamara@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>; Kusztal, ArkadiuszX
> <arkadiuszx.kusztal@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
>
> 30/01/2020 14:06, Trahe, Fiona:
> > We were unaware the LIST_END change could constitute an ABI breakage, but can see how it affects the array size when picked up.
> > We're exploring options.
> >
> > I agree with Anoob's point that if we don't allow the LIST_END to be modified, then it means no feature can be implemented without ABI
> breakage.
> > Anyone object to removing those LIST_END elements - or have a better suggestion? Would have to be in 20.11 I suppose.
>
> Yes, having max value right after the last value is ridiculous,
> it prevents adding any value.
> In 20.11, we should remove all these *_END and *_MAX from API enums
> and replace them with a separate #define with reasonnable maximums.
>
I think we'd better avoid public structs that have array of _MAX elems in them.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v3 4/4] add ABI checks
2020-01-30 16:00 28% ` [dpdk-dev] [PATCH v3 4/4] add ABI checks David Marchand
2020-01-30 22:32 4% ` Thomas Monjalon
@ 2020-01-30 22:44 4% ` Thomas Monjalon
1 sibling, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-01-30 22:44 UTC (permalink / raw)
To: David Marchand
Cc: dev, bruce.richardson, kevin.laatz, aconole, nhorman,
akhil.goyal, anoobj, bluca, fiona.trahe, ferruh.yigit,
Michael Santana, John McNamara, Marko Kovacevic
30/01/2020 17:00, David Marchand:
> --- a/devtools/test-meson-builds.sh
> +++ b/devtools/test-meson-builds.sh
> +refsrcdir=$(mktemp -d -t dpdk-${DPDK_ABI_REF_VERSION:-}.XXX)
Instead of a temporary source directory,
could it be inside $DPDK_ABI_REF_DIR/$DPDK_ABI_REF_VERSION ?
I feel it would more "hackable" for debugging of the process.
[..]
> + if [ -n "$DPDK_ABI_REF_VERSION" ]; then
> + DPDK_ABI_REF_DIR=${DPDK_ABI_REF_DIR:-reference}
> + abirefdir=$DPDK_ABI_REF_DIR/$DPDK_ABI_REF_VERSION/$targetdir
> + if [ ! -d $abirefdir ]; then
> + # clone current sources
> + if [ ! -d $refsrcdir/.git ]; then
> + git clone --local --no-hardlinks \
> + --single-branch \
> + -b $DPDK_ABI_REF_VERSION \
> + $srcdir $refsrcdir
> + fi
> +
> + rm -rf $refsrcdir/build
> + config $refsrcdir $refsrcdir/build $*
> + compile $refsrcdir/build $abirefdir
> + $srcdir/devtools/gen-abi.sh $abirefdir
> + fi
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v3 4/4] add ABI checks
2020-01-30 16:00 28% ` [dpdk-dev] [PATCH v3 4/4] add ABI checks David Marchand
@ 2020-01-30 22:32 4% ` Thomas Monjalon
2020-02-01 15:29 4% ` David Marchand
2020-01-30 22:44 4% ` Thomas Monjalon
1 sibling, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-01-30 22:32 UTC (permalink / raw)
To: David Marchand
Cc: dev, bruce.richardson, kevin.laatz, aconole, nhorman,
akhil.goyal, anoobj, bluca, fiona.trahe, ferruh.yigit,
Michael Santana, John McNamara, Marko Kovacevic
30/01/2020 17:00, David Marchand:
> Enabling them requires a configuration that will trigger the ABI dumps
> generation as part of the existing devtools/test-build.sh and
> devtools/test-meson-builds.sh scripts.
[...]
> --- a/devtools/test-meson-builds.sh
> +++ b/devtools/test-meson-builds.sh
> if [ -f "$builddir/build.ninja" ] ; then
> + # for existing environments, force debugoptimized so that ABI
> + # checks can run
> + $MESON configure --buildtype=debugoptimized $builddir
This is forcing meson to re-run each time, even if the buildtype is
already "debugoptimized".
Please query meson configuration to avoid useless re-run:
$MESON configure $builddir | awk '$1=="buildtype" {print $2}'
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v3 3/4] build: test meson installation
2020-01-30 16:00 3% ` [dpdk-dev] [PATCH v3 3/4] build: test meson installation David Marchand
@ 2020-01-30 22:17 3% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-01-30 22:17 UTC (permalink / raw)
To: David Marchand
Cc: dev, bruce.richardson, kevin.laatz, aconole, nhorman,
akhil.goyal, anoobj, bluca, fiona.trahe, ferruh.yigit,
Michael Santana
30/01/2020 17:00, David Marchand:
> Let's test installing with meson as part of Travis and the
> devtools/test-meson-builds.sh script.
For test-meson-builds.sh, I would prefer avoiding install if not needed.
Can we install only if compiling examples or checking ABI?
> -compile () # <builddir>
> +compile () # <builddir> <installdir>
> {
> builddir=$1
> if [ -n "$TEST_MESON_BUILD_VERY_VERBOSE" ] ; then
> @@ -90,6 +90,9 @@ compile () # <builddir>
> echo "$ninja_cmd -C $builddir"
> $ninja_cmd -C $builddir
> fi
> + rm -rf $2
> + echo "DESTDIR=$2 $ninja_cmd -C $builddir install"
> + DESTDIR=$2 $ninja_cmd -C $builddir install
> }
>
> build () # <directory> <target compiler> <meson options>
> @@ -102,7 +105,8 @@ build () # <directory> <target compiler> <meson options>
> command -v ${CC##* } >/dev/null 2>&1 || return 0
> load_env $targetcc || return 0
> config $srcdir $builds_dir/$targetdir $*
> - compile $builds_dir/$targetdir
> + compile $builds_dir/$targetdir \
> + $(readlink -f $builds_dir/$targetdir/install)
> }
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [RFC v2] meter: fix ABI break due to experimental tag removal
2020-01-30 16:15 4% ` Eelco Chaudron
@ 2020-01-30 20:20 4% ` Thomas Monjalon
2020-02-03 11:10 4% ` Eelco Chaudron
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-01-30 20:20 UTC (permalink / raw)
To: Luca Boccassi, Eelco Chaudron
Cc: Ferruh Yigit, Neil Horman, Cristian Dumitrescu, dev,
David Marchand, Bruce Richardson, Ian Stokes, Andrzej Ostruszka
30/01/2020 17:15, Eelco Chaudron:
> On 30 Jan 2020, at 17:04, Luca Boccassi wrote:
> > On Thu, 2020-01-30 at 16:55 +0100, Thomas Monjalon wrote:
> >> 30/01/2020 15:21, Luca Boccassi:
> >>> On Thu, 2020-01-30 at 15:17 +0100, Thomas Monjalon wrote:
> >>>> 30/01/2020 13:57, Luca Boccassi:
> >>>>> On Thu, 2020-01-30 at 13:33 +0100, Thomas Monjalon wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> I disagree with the need of this patch.
> >>>>>> The symbol was experimental, meaning we can change it.
> >>>>>> Removing experimental tag is not an ABI break.
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> This symbol change was requested for backport in 19.11.x, and
> >>>>> experimental or not I'm not too keen on backward incompatible
> >>>>> changes
> >>>>> to the public interface in an _LTS point release_. The
> >>>>> compromise
> >>>>> was
> >>>>> to see if we could support both symbols version, which makes
> >>>>> the
> >>>>> change
> >>>>> backward compatible.
> >>>>>
> >>>>> If you prefer not to have this patch in mainline I'm also fine
> >>>>> in
> >>>>> taking it just for the LTS. I agree with you that it is not
> >>>>> required
> >>>>> for mainline releases (although nicer for me if it's a backport
> >>>>> rather
> >>>>> than a new change).
> >>>>
> >>>> I would like to avoid opening the door for maintaining the
> >>>> experimental ABI
> >>>> in the mainline. Please take it directly in the LTS.
> >>>>
> >>>> The next question is to know whether we really want to have such
> >>>> patch in LTS.
> >>>> Anyway, 19.11.0 has this symbol as experimental.
> >>>> How adding a non-experimental version of the function in 19.11.1
> >>>> will
> >>>> change
> >>>> the ABI status of the whole 19.11 branch?
> >>>
> >>> The problem is not adding the new symbol, but removing the
> >>> experimental
> >>> one. Changing the version of the symbol was requested by OVS for
> >>> inclusion in 19.11.
> >>
> >> Yes, sorry, this is what I meant.
> >> Given 19.11.0 already has the symbol as experimental,
> >> and that applications like OVS had to accept it as experimental,
> >> why removing experimental tag in 19.11.1?
> >
> > I think it was mentioned that it was preferred not to suppress the
> > compiler warning to avoid any accidental use in the future, but the
> > OVS
> > maintainer(s) should answer as I might remember wrongly.
>
> Yes this is the reason, OVS compiles with -Werror so we would like to
> avoid the warnings. You can not disable them per include, it’s global
> for all of DPDK.
Yes but anyway OVS must accept the experimental function as the next release
will use it with DPDK 19.11.0.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-01-30 16:09 8% ` Ferruh Yigit
@ 2020-01-30 20:18 4% ` Thomas Monjalon
2020-01-31 9:03 7% ` Ferruh Yigit
2020-01-31 14:16 4% ` Trahe, Fiona
0 siblings, 2 replies; 200+ results
From: Thomas Monjalon @ 2020-01-30 20:18 UTC (permalink / raw)
To: Akhil Goyal, David Marchand, Anoob Joseph, Trahe, Fiona, Kusztal,
ArkadiuszX, Ferruh Yigit
Cc: dev, bruce.richardson, nhorman, John McNamara, dodji, Andrew Rybchenko
30/01/2020 17:09, Ferruh Yigit:
> On 1/29/2020 8:13 PM, Akhil Goyal wrote:
> >
> >
> >>
> >> On Wed, Jan 29, 2020 at 7:10 PM Anoob Joseph <anoobj@marvell.com> wrote:
> >>> The asymmetric crypto library is experimental. Changes to experimental code
> >> paths is allowed, right?
> >>
> >> The asymmetric crypto enum is referenced by a function part of the stable ABI.
> >> It is possible to waive this enum, if we are sure no use out of the
> >> experimental asym crypto APIs is possible.
> >>
> >> The rest of the changes touch stable symbols.
> >>
> >> Adding the abidiff report:
> >>
> >> [C]'function void rte_cryptodev_info_get(uint8_t,
> >> rte_cryptodev_info*)' at rte_cryptodev.c:1115:1 has some indirect
> >> sub-type changes:
> >> parameter 2 of type 'rte_cryptodev_info*' has sub-type changes:
> >> in pointed to type 'struct rte_cryptodev_info' at rte_cryptodev.h:468:1:
> >> type size hasn't changed
> >> 1 data member change:
> >> type of 'const rte_cryptodev_capabilities*
> >> rte_cryptodev_info::capabilities' changed:
> >> in pointed to type 'const rte_cryptodev_capabilities':
> >> in unqualified underlying type 'struct
> >> rte_cryptodev_capabilities' at rte_cryptodev.h:176:1:
> >> type size hasn't changed
> >> 1 data member change:
> >> type of '__anonymous_union__ ' changed:
> >> type size hasn't changed
> >> 1 data member change:
> >> type of 'rte_cryptodev_asymmetric_capability
> >> __anonymous_union__::asym' changed:
> >> type size hasn't changed
> >> 1 data member change:
> >> type of
> >> 'rte_cryptodev_asymmetric_xform_capability
> >> rte_cryptodev_asymmetric_capability::xform_capa' changed:
> >> type size hasn't changed
> >> 1 data member change:
> >> type of 'rte_crypto_asym_xform_type
> >> rte_cryptodev_asymmetric_xform_capability::xform_type' changed:
> >> type size hasn't changed
> >> 2 enumerator insertions:
> >>
> >> 'rte_crypto_asym_xform_type::RTE_CRYPTO_ASYM_XFORM_ECDSA' value '7'
> >>
> >> 'rte_crypto_asym_xform_type::RTE_CRYPTO_ASYM_XFORM_ECPM' value '8'
> >> 1 enumerator change:
> >>
> >> 'rte_crypto_asym_xform_type::RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END'
> >> from
> >> value '7' to '9' at rte_crypto_asym.h:60:1
> >>
> >
> > I believe these enums will be used only in case of ASYM case which is experimental.
>
> Independent from being experiment and not, this shouldn't be a problem, I think
> this is a false positive.
>
> The ABI break can happen when a struct has been shared between the application
> and the library (DPDK) and the layout of that memory know differently by
> application and the library.
>
> Here in all cases, there is no layout/size change.
>
> As to the value changes of the enums, since application compiled with old DPDK,
> it will know only up to '6', 7 and more means invalid to the application. So it
> won't send these values also it should ignore these values from library. Only
> consequence is old application won't able to use new features those new enums
> provide but that is expected/normal.
If library give higher value than expected by the application,
if the application uses this value as array index,
there can be an access out of bounds.
> >> [C]'function int
> >> rte_cryptodev_get_aead_algo_enum(rte_crypto_aead_algorithm*, const
> >> char*)' at rte_cryptodev.c:239:1 has some indirect sub-type changes:
> >> parameter 1 of type 'rte_crypto_aead_algorithm*' has sub-type changes:
> >> in pointed to type 'enum rte_crypto_aead_algorithm' at
> >> rte_crypto_sym.h:346:1:
> >> type size hasn't changed
> >> 1 enumerator insertion:
> >> 'rte_crypto_aead_algorithm::RTE_CRYPTO_AEAD_CHACHA20_POLY1305'
> >> value '3'
> >> 1 enumerator change:
> >> 'rte_crypto_aead_algorithm::RTE_CRYPTO_AEAD_LIST_END' from
> >> value '3' to '4' at rte_crypto_sym.h:346:1
>
> Same as above, no layout change.
>
> >>
> >>
> >> [C]'const char* rte_crypto_aead_algorithm_strings[1]' was changed at
> >> rte_crypto_sym.h:358:1:
> >> size of symbol (in bytes) changed from 24 to 32
> >>
>
> The shared memory size changes, but this is global variable in the library, and
> the values application can request 'RTE_CRYPTO_AEAD_AES_CCM' &
> 'RTE_CRYPTO_AEAD_AES_GCM' is already there, so there is no backward
> compatibility issue here.
For this one, I don't know what is the breakage.
> > +Fiona and Arek
> >
> > We may need to revert the chacha-poly patches.
> >
>
> I don't see any ABI break in this case, can someone explain if I am missing
> anything here?
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] doc: deprecation notice to move igb_uio to a new repo
2020-01-30 16:58 4% [dpdk-dev] [PATCH] doc: deprecation notice to move igb_uio to a new repo jerinj
@ 2020-01-30 17:30 0% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-01-30 17:30 UTC (permalink / raw)
To: jerinj, dev, Neil Horman, John McNamara, Marko Kovacevic
Cc: thomas, techboard
On 1/30/2020 4:58 PM, jerinj@marvell.com wrote:
> From: Jerin Jacob <jerinj@marvell.com>
>
> Based on the tech board meeting held on 2019-11-06,
> It's been decided to move igb_uio kernel module to a new repository
> hosted by dpdk.org in v20.11 release.
>
> http://mails.dpdk.org/archives/dev/2019-November/151763.html
>
> Adding the deprecation notice for the same in advance.
>
> Signed-off-by: Jerin Jacob <jerinj@marvell.com>
> ---
> doc/guides/rel_notes/deprecation.rst | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index dfcca87ab..2b9d0113f 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -38,6 +38,12 @@ Deprecation Notices
> remove it from the externally visible ABI and allow it to be updated in the
> future.
>
> +* igb_uio: In the view of reducing the kernel dependency from the main tree,
> + as a first step, The technical board has been decided to move ``igb_uio``
> + kernel module to a new repository hosted by dpdk.org in v20.11 release.
> + Minutes of Technical Board Meeting of `2019-11-06
> + <http://mails.dpdk.org/archives/dev/2019-November/151763.html>`_.
> +
> * dpaa2: removal of ``rte_dpaa2_memsegs`` structure which has been replaced
> by a pa-va search library. This structure was earlier being used for holding
> memory segments used by dpaa2 driver for faster pa->va translation. This
>
Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] doc: deprecation notice to move igb_uio to a new repo
@ 2020-01-30 16:58 4% jerinj
2020-01-30 17:30 0% ` Ferruh Yigit
0 siblings, 1 reply; 200+ results
From: jerinj @ 2020-01-30 16:58 UTC (permalink / raw)
To: dev, Neil Horman, John McNamara, Marko Kovacevic
Cc: thomas, techboard, ferruh.yigit, Jerin Jacob
From: Jerin Jacob <jerinj@marvell.com>
Based on the tech board meeting held on 2019-11-06,
It's been decided to move igb_uio kernel module to a new repository
hosted by dpdk.org in v20.11 release.
http://mails.dpdk.org/archives/dev/2019-November/151763.html
Adding the deprecation notice for the same in advance.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
---
doc/guides/rel_notes/deprecation.rst | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index dfcca87ab..2b9d0113f 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -38,6 +38,12 @@ Deprecation Notices
remove it from the externally visible ABI and allow it to be updated in the
future.
+* igb_uio: In the view of reducing the kernel dependency from the main tree,
+ as a first step, The technical board has been decided to move ``igb_uio``
+ kernel module to a new repository hosted by dpdk.org in v20.11 release.
+ Minutes of Technical Board Meeting of `2019-11-06
+ <http://mails.dpdk.org/archives/dev/2019-November/151763.html>`_.
+
* dpaa2: removal of ``rte_dpaa2_memsegs`` structure which has been replaced
by a pa-va search library. This structure was earlier being used for holding
memory segments used by dpaa2 driver for faster pa->va translation. This
--
2.24.1
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
2020-01-30 15:59 4% ` Thomas Monjalon
@ 2020-01-30 16:42 7% ` Ferruh Yigit
2020-01-30 23:49 4% ` Ananyev, Konstantin
1 sibling, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-01-30 16:42 UTC (permalink / raw)
To: Thomas Monjalon, Anoob Joseph, akhil.goyal, Trahe, Fiona
Cc: dev, David Marchand, Richardson, Bruce, nhorman, Mcnamara, John,
Kusztal, ArkadiuszX
On 1/30/2020 3:59 PM, Thomas Monjalon wrote:
> 30/01/2020 14:06, Trahe, Fiona:
>> We were unaware the LIST_END change could constitute an ABI breakage, but can see how it affects the array size when picked up.
>> We're exploring options.
>>
>> I agree with Anoob's point that if we don't allow the LIST_END to be modified, then it means no feature can be implemented without ABI breakage.
>> Anyone object to removing those LIST_END elements - or have a better suggestion? Would have to be in 20.11 I suppose.
>
> Yes, having max value right after the last value is ridiculous,
> it prevents adding any value.
> In 20.11, we should remove all these *_END and *_MAX from API enums
> and replace them with a separate #define with reasonnable maximums.
>
>
I disagree, that kind of usage is common and lets loops iterate on the valid
elements, and it is not a source of ABI break on its own.
Indeed other way around, not having MAX, is problematic, if we don't have the
MAX value to compare and decide if a provided value is valid or not, and when
new version of the library introduces the new values, how old application can
detect unsupported new values?
As far as I can see the problem occurs when that *_END and *_MAX used to define
the size of array in the public struct. This usage prevents adding new values
and I already send a deprecation notice for it:
https://patches.dpdk.org/patch/65359/
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [dpdk-stable] [PATCH] meter: move RFC4115 trTCM APIs as none experimental
@ 2020-01-30 16:33 4% ` Ray Kinsella
0 siblings, 0 replies; 200+ results
From: Ray Kinsella @ 2020-01-30 16:33 UTC (permalink / raw)
To: Ferruh Yigit, Neil Horman, David Marchand
Cc: Thomas Monjalon, Cristian Dumitrescu, dpdk stable, dev,
Eelco Chaudron, Kevin Traynor, Ian Stokes, Ilya Maximets,
Luca Boccassi
On 16/01/2020 12:42, Ferruh Yigit wrote:
> On 1/16/2020 11:54 AM, Neil Horman wrote:
>> On Thu, Jan 16, 2020 at 12:25:06PM +0100, David Marchand wrote:
>>> On Tue, Dec 17, 2019 at 2:08 PM Eelco Chaudron <echaudro@redhat.com> wrote:
>>>>
>>>> Moved RFC4115 APIs to none experimental as they have been there
>>>> since 19.02. Also, these APIs are the same as the none RFC4115 APIs.
>>>>
>>>> Signed-off-by: Eelco Chaudron <echaudro@redhat.com>
>>>
>>> There is a discussion on the OVS ml at the moment to get these symbols
>>> in the stable ABI for 19.11.
>>> I want to understand how this would be done.
>>>
>>> - I take this patch in 20.02, these symbols are added in the 20.0.1 ABI.
>>> On the other hand, the 19.11 release maintains the 20.0 ABI.
>>>
>>> Does it mean the backport adds these symbols with the 20.0 version in
>>> the 19.11 branch?
>>> Or is 20.0.1 version acceptable / a thing we want?
>>>
>>>
>>> - These symbol already existed in the 20.0 ABI, versioned as EXPERIMENTAL.
>>> We can go and remove these entries since we are not bound to preserve
>>> the experimental APIs.
>>> But, on the other hand, nothing should prevent us from keeping some
>>> aliases so that the symbols versioned EXPERIMENTAL are still available
>>> to existing users.
>>>
>> I would say that choice is up to you. If you want to alias them to be nice to
>> prior users, thats fine by me. But experimental means experimental, and so users
>> have to be prepared to rebuild when things change, even if that change is
>> changing the version from experimental to a concrete version.
>>
>
> I would prefer to keep the alias and don't break the existing users, specially
> for the case experimental API is becoming mature without change.
>
Apologies for dragging up an older thread.
I had a thought later that we should add something formal to the ABI Policy.
Such that if we have an API that has been part of DPDK as experimental for a long time,
to the point it has a number of users - out of kindness, the maintainer may ask a
contributor to add an alias until the next major ABI version is declared?
So as not to break consuming projects - make sense?
Thanks,
Ray K
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [RFC v2] meter: fix ABI break due to experimental tag removal
2020-01-30 16:04 4% ` Luca Boccassi
@ 2020-01-30 16:15 4% ` Eelco Chaudron
2020-01-30 20:20 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Eelco Chaudron @ 2020-01-30 16:15 UTC (permalink / raw)
To: Luca Boccassi
Cc: Thomas Monjalon, Ferruh Yigit, Neil Horman, Cristian Dumitrescu,
dev, David Marchand, Bruce Richardson, Ian Stokes,
Andrzej Ostruszka
On 30 Jan 2020, at 17:04, Luca Boccassi wrote:
> On Thu, 2020-01-30 at 16:55 +0100, Thomas Monjalon wrote:
>> 30/01/2020 15:21, Luca Boccassi:
>>> On Thu, 2020-01-30 at 15:17 +0100, Thomas Monjalon wrote:
>>>> 30/01/2020 13:57, Luca Boccassi:
>>>>> On Thu, 2020-01-30 at 13:33 +0100, Thomas Monjalon wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I disagree with the need of this patch.
>>>>>> The symbol was experimental, meaning we can change it.
>>>>>> Removing experimental tag is not an ABI break.
>>>>>
>>>>> Hi,
>>>>>
>>>>> This symbol change was requested for backport in 19.11.x, and
>>>>> experimental or not I'm not too keen on backward incompatible
>>>>> changes
>>>>> to the public interface in an _LTS point release_. The
>>>>> compromise
>>>>> was
>>>>> to see if we could support both symbols version, which makes
>>>>> the
>>>>> change
>>>>> backward compatible.
>>>>>
>>>>> If you prefer not to have this patch in mainline I'm also fine
>>>>> in
>>>>> taking it just for the LTS. I agree with you that it is not
>>>>> required
>>>>> for mainline releases (although nicer for me if it's a backport
>>>>> rather
>>>>> than a new change).
>>>>
>>>> I would like to avoid opening the door for maintaining the
>>>> experimental ABI
>>>> in the mainline. Please take it directly in the LTS.
>>>>
>>>> The next question is to know whether we really want to have such
>>>> patch in LTS.
>>>> Anyway, 19.11.0 has this symbol as experimental.
>>>> How adding a non-experimental version of the function in 19.11.1
>>>> will
>>>> change
>>>> the ABI status of the whole 19.11 branch?
>>>
>>> The problem is not adding the new symbol, but removing the
>>> experimental
>>> one. Changing the version of the symbol was requested by OVS for
>>> inclusion in 19.11.
>>
>> Yes, sorry, this is what I meant.
>> Given 19.11.0 already has the symbol as experimental,
>> and that applications like OVS had to accept it as experimental,
>> why removing experimental tag in 19.11.1?
>
> I think it was mentioned that it was preferred not to suppress the
> compiler warning to avoid any accidental use in the future, but the
> OVS
> maintainer(s) should answer as I might remember wrongly.
Yes this is the reason, OVS compiles with -Werror so we would like to
avoid the warnings. You can not disable them per include, it’s global
for all of DPDK.
//Eelco
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2] build: remove unused function versioning
@ 2020-01-30 16:11 3% ` David Marchand
2020-02-05 20:44 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-01-30 16:11 UTC (permalink / raw)
To: Andrzej Ostruszka
Cc: dev, David Hunt, Bruce Richardson, Vladimir Medvedkin,
Robert Sanford, Erik Gabriel Carrillo, Marcin Baran, dpdk stable
On Thu, Jan 30, 2020 at 10:25 AM Andrzej Ostruszka
<aostruszka@marvell.com> wrote:
>
> Timer, LPM and Distributor libraries no longer use function versioning
> and therefore do not need separate build for static and shared version
> of libraries.
>
> This patch removes use_function_versioning from their meson build files
> and corresponding include from the sources.
>
> Signed-off-by: Andrzej Ostruszka <aostruszka@marvell.com>
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
>
> Fixes: f2fb215843a9 ("timer: remove deprecated code")
> Fixes: 6e5b51676176 ("distributor: remove deprecated code")
> Fixes: c381a8d554b7 ("lpm: remove deprecated code")
> Cc: marcinx.baran@intel.com
>
> ---
> V2:
> - removed also includes (updating only meson files causes build
> failures), and updated commit msg accordingly
Thanks, I had the exact same patch in store (caught when working with
Olivier on some ABI stuff for mempool).
Cc: stable@dpdk.org
Acked-by: David Marchand <david.marchand@redhat.com>
--
David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
@ 2020-01-30 16:09 8% ` Ferruh Yigit
2020-01-30 20:18 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-01-30 16:09 UTC (permalink / raw)
To: Akhil Goyal, David Marchand, Anoob Joseph, Trahe, Fiona, Kusztal,
ArkadiuszX
Cc: Thomas Monjalon, dev, bruce.richardson, nhorman, John McNamara,
dodji, Andrew Rybchenko
On 1/29/2020 8:13 PM, Akhil Goyal wrote:
>
>
>>
>> On Wed, Jan 29, 2020 at 7:10 PM Anoob Joseph <anoobj@marvell.com> wrote:
>>> The asymmetric crypto library is experimental. Changes to experimental code
>> paths is allowed, right?
>>
>> The asymmetric crypto enum is referenced by a function part of the stable ABI.
>> It is possible to waive this enum, if we are sure no use out of the
>> experimental asym crypto APIs is possible.
>>
>> The rest of the changes touch stable symbols.
>>
>> Adding the abidiff report:
>>
>> [C]'function void rte_cryptodev_info_get(uint8_t,
>> rte_cryptodev_info*)' at rte_cryptodev.c:1115:1 has some indirect
>> sub-type changes:
>> parameter 2 of type 'rte_cryptodev_info*' has sub-type changes:
>> in pointed to type 'struct rte_cryptodev_info' at rte_cryptodev.h:468:1:
>> type size hasn't changed
>> 1 data member change:
>> type of 'const rte_cryptodev_capabilities*
>> rte_cryptodev_info::capabilities' changed:
>> in pointed to type 'const rte_cryptodev_capabilities':
>> in unqualified underlying type 'struct
>> rte_cryptodev_capabilities' at rte_cryptodev.h:176:1:
>> type size hasn't changed
>> 1 data member change:
>> type of '__anonymous_union__ ' changed:
>> type size hasn't changed
>> 1 data member change:
>> type of 'rte_cryptodev_asymmetric_capability
>> __anonymous_union__::asym' changed:
>> type size hasn't changed
>> 1 data member change:
>> type of
>> 'rte_cryptodev_asymmetric_xform_capability
>> rte_cryptodev_asymmetric_capability::xform_capa' changed:
>> type size hasn't changed
>> 1 data member change:
>> type of 'rte_crypto_asym_xform_type
>> rte_cryptodev_asymmetric_xform_capability::xform_type' changed:
>> type size hasn't changed
>> 2 enumerator insertions:
>>
>> 'rte_crypto_asym_xform_type::RTE_CRYPTO_ASYM_XFORM_ECDSA' value '7'
>>
>> 'rte_crypto_asym_xform_type::RTE_CRYPTO_ASYM_XFORM_ECPM' value '8'
>> 1 enumerator change:
>>
>> 'rte_crypto_asym_xform_type::RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END'
>> from
>> value '7' to '9' at rte_crypto_asym.h:60:1
>>
>
> I believe these enums will be used only in case of ASYM case which is experimental.
Independent from being experiment and not, this shouldn't be a problem, I think
this is a false positive.
The ABI break can happen when a struct has been shared between the application
and the library (DPDK) and the layout of that memory know differently by
application and the library.
Here in all cases, there is no layout/size change.
As to the value changes of the enums, since application compiled with old DPDK,
it will know only up to '6', 7 and more means invalid to the application. So it
won't send these values also it should ignore these values from library. Only
consequence is old application won't able to use new features those new enums
provide but that is expected/normal.
>
>>
>> [C]'function int
>> rte_cryptodev_get_aead_algo_enum(rte_crypto_aead_algorithm*, const
>> char*)' at rte_cryptodev.c:239:1 has some indirect sub-type changes:
>> parameter 1 of type 'rte_crypto_aead_algorithm*' has sub-type changes:
>> in pointed to type 'enum rte_crypto_aead_algorithm' at
>> rte_crypto_sym.h:346:1:
>> type size hasn't changed
>> 1 enumerator insertion:
>> 'rte_crypto_aead_algorithm::RTE_CRYPTO_AEAD_CHACHA20_POLY1305'
>> value '3'
>> 1 enumerator change:
>> 'rte_crypto_aead_algorithm::RTE_CRYPTO_AEAD_LIST_END' from
>> value '3' to '4' at rte_crypto_sym.h:346:1
Same as above, no layout change.
>>
>>
>> [C]'const char* rte_crypto_aead_algorithm_strings[1]' was changed at
>> rte_crypto_sym.h:358:1:
>> size of symbol (in bytes) changed from 24 to 32
>>
The shared memory size changes, but this is global variable in the library, and
the values application can request 'RTE_CRYPTO_AEAD_AES_CCM' &
'RTE_CRYPTO_AEAD_AES_GCM' is already there, so there is no backward
compatibility issue here.
>>
> +Fiona and Arek
>
> We may need to revert the chacha-poly patches.
>
I don't see any ABI break in this case, can someone explain if I am missing
anything here?
^ permalink raw reply [relevance 8%]
* Re: [dpdk-dev] [RFC v2] meter: fix ABI break due to experimental tag removal
@ 2020-01-30 16:04 4% ` Luca Boccassi
2020-01-30 16:15 4% ` Eelco Chaudron
0 siblings, 1 reply; 200+ results
From: Luca Boccassi @ 2020-01-30 16:04 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Ferruh Yigit, Neil Horman, Cristian Dumitrescu, Eelco Chaudron,
dev, David Marchand, Bruce Richardson, Ian Stokes,
Andrzej Ostruszka
On Thu, 2020-01-30 at 16:55 +0100, Thomas Monjalon wrote:
> 30/01/2020 15:21, Luca Boccassi:
> > On Thu, 2020-01-30 at 15:17 +0100, Thomas Monjalon wrote:
> > > 30/01/2020 13:57, Luca Boccassi:
> > > > On Thu, 2020-01-30 at 13:33 +0100, Thomas Monjalon wrote:
> > > > > Hi,
> > > > >
> > > > > I disagree with the need of this patch.
> > > > > The symbol was experimental, meaning we can change it.
> > > > > Removing experimental tag is not an ABI break.
> > > >
> > > > Hi,
> > > >
> > > > This symbol change was requested for backport in 19.11.x, and
> > > > experimental or not I'm not too keen on backward incompatible
> > > > changes
> > > > to the public interface in an _LTS point release_. The
> > > > compromise
> > > > was
> > > > to see if we could support both symbols version, which makes
> > > > the
> > > > change
> > > > backward compatible.
> > > >
> > > > If you prefer not to have this patch in mainline I'm also fine
> > > > in
> > > > taking it just for the LTS. I agree with you that it is not
> > > > required
> > > > for mainline releases (although nicer for me if it's a backport
> > > > rather
> > > > than a new change).
> > >
> > > I would like to avoid opening the door for maintaining the
> > > experimental ABI
> > > in the mainline. Please take it directly in the LTS.
> > >
> > > The next question is to know whether we really want to have such
> > > patch in LTS.
> > > Anyway, 19.11.0 has this symbol as experimental.
> > > How adding a non-experimental version of the function in 19.11.1
> > > will
> > > change
> > > the ABI status of the whole 19.11 branch?
> >
> > The problem is not adding the new symbol, but removing the
> > experimental
> > one. Changing the version of the symbol was requested by OVS for
> > inclusion in 19.11.
>
> Yes, sorry, this is what I meant.
> Given 19.11.0 already has the symbol as experimental,
> and that applications like OVS had to accept it as experimental,
> why removing experimental tag in 19.11.1?
I think it was mentioned that it was preferred not to suppress the
compiler warning to avoid any accidental use in the future, but the OVS
maintainer(s) should answer as I might remember wrongly.
--
Kind regards,
Luca Boccassi
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH v3 4/4] add ABI checks
2020-01-30 16:00 9% ` [dpdk-dev] [PATCH v3 0/4] " David Marchand
2020-01-30 16:00 3% ` [dpdk-dev] [PATCH v3 3/4] build: test meson installation David Marchand
@ 2020-01-30 16:00 28% ` David Marchand
2020-01-30 22:32 4% ` Thomas Monjalon
2020-01-30 22:44 4% ` Thomas Monjalon
1 sibling, 2 replies; 200+ results
From: David Marchand @ 2020-01-30 16:00 UTC (permalink / raw)
To: dev
Cc: thomas, bruce.richardson, kevin.laatz, aconole, nhorman,
akhil.goyal, anoobj, bluca, fiona.trahe, ferruh.yigit,
Michael Santana, John McNamara, Marko Kovacevic
For normal developers, those checks are disabled.
Enabling them requires a configuration that will trigger the ABI dumps
generation as part of the existing devtools/test-build.sh and
devtools/test-meson-builds.sh scripts.
Those checks are enabled in the CI for the default meson options on x86
and aarch64 so that proposed patches are validated via our CI robot.
A cache of the ABI is stored in travis jobs to avoid rebuilding too
often.
Checks can be only informational by setting ABI_CHECKS_WARN_ONLY when
breaking the ABI in a future release.
Explicit suppression rules have been added on internal structures
exposed to crypto drivers as the current ABI policy does not apply to
them.
This could be improved in the future by carefully splitting the headers
content with application and driver "users" in mind.
We currently have issues reported for librte_crypto recent changes for
which suppression rules have been added too.
Mellanox glue libraries are explicitly skipped as they are not part of
the application ABI.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Luca Boccassi <bluca@debian.org>
---
Changelog since v2:
- forced -g / buildtype=debugoptimised in the test scripts so that
we can check ABI in existing environments,
- little update on the documentation,
Changelog since v1:
- reworked the scripts so that the build test scripts clone and build the
reference automatically. A developer only needs to set one variable to
enable the checks,
- meson builds are done with debug so that abidiff can inspect the
structures,
- abidiff checks only public types by looking at installed headers,
- abidiff has some issue when comparing a dump with a .so built with clang
so all diff are now done with dump files only,
- suppression rules have been added to waive warnings on exposed internal
types,
- an abi breakage has been reported on changes in cryptodev.
For now, suppression rules have been put in place to let the CI run,
---
.ci/linux-build.sh | 23 +++++++++++
.travis.yml | 20 +++++++++-
MAINTAINERS | 2 +
devtools/check-abi.sh | 59 +++++++++++++++++++++++++++++
devtools/dpdk.abignore | 21 ++++++++++
devtools/gen-abi.sh | 26 +++++++++++++
devtools/test-build.sh | 51 ++++++++++++++++++++++---
devtools/test-meson-builds.sh | 37 +++++++++++++++++-
doc/guides/contributing/patches.rst | 15 ++++++++
9 files changed, 246 insertions(+), 8 deletions(-)
create mode 100755 devtools/check-abi.sh
create mode 100644 devtools/dpdk.abignore
create mode 100755 devtools/gen-abi.sh
diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index e61aa2b0a..95bd869c3 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -30,6 +30,7 @@ fi
OPTS="$OPTS --default-library=$DEF_LIB"
OPTS="$OPTS --prefix=/usr -Dlibdir=lib"
+OPTS="$OPTS --buildtype=debugoptimized"
meson build --werror -Dexamples=all $OPTS
ninja -C build
DESTDIR=$(pwd)/install ninja -C build install
@@ -40,6 +41,28 @@ if [ "$AARCH64" != "1" ]; then
unset LD_LIBRARY_PATH
fi
+if [ "$ABI_CHECKS" = "1" ]; then
+ REF_GIT_REPO=${REF_GIT_REPO:-https://dpdk.org/git/dpdk}
+ REF_GIT_TAG=${REF_GIT_TAG:-v19.11}
+
+ if [ "$(cat reference/VERSION 2>/dev/null)" != "$REF_GIT_TAG" ]; then
+ rm -rf reference
+ fi
+
+ if [ ! -d reference ]; then
+ refsrcdir=$(readlink -f $(pwd)/../dpdk-$REF_GIT_TAG)
+ git clone --single-branch -b $REF_GIT_TAG $REF_GIT_REPO $refsrcdir
+ meson --werror $OPTS $refsrcdir $refsrcdir/build
+ ninja -C $refsrcdir/build
+ DESTDIR=$(pwd)/reference ninja -C $refsrcdir/build install
+ devtools/gen-abi.sh reference
+ echo $REF_GIT_TAG > reference/VERSION
+ fi
+
+ devtools/gen-abi.sh install
+ devtools/check-abi.sh reference install ${ABI_CHECKS_WARN_ONLY:-}
+fi
+
if [ "$RUN_TESTS" = "1" ]; then
sudo meson test -C build --suite fast-tests -t 3
fi
diff --git a/.travis.yml b/.travis.yml
index 8162f1c05..22539d823 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,5 +1,8 @@
language: c
-cache: ccache
+cache:
+ ccache: true
+ directories:
+ - reference
compiler:
- gcc
- clang
@@ -21,7 +24,7 @@ aarch64_packages: &aarch64_packages
extra_packages: &extra_packages
- *required_packages
- - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
+ - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4, abigail-tools]
build_32b_packages: &build_32b_packages
- *required_packages
@@ -151,5 +154,18 @@ matrix:
packages:
- *required_packages
- *doc_packages
+ - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
+ compiler: gcc
+ addons:
+ apt:
+ packages:
+ - *extra_packages
+ - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
+ arch: arm64
+ compiler: gcc
+ addons:
+ apt:
+ packages:
+ - *extra_packages
script: ./.ci/${TRAVIS_OS_NAME}-build.sh
diff --git a/MAINTAINERS b/MAINTAINERS
index 94bccae6d..6dae4ee63 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -144,8 +144,10 @@ M: Neil Horman <nhorman@tuxdriver.com>
F: lib/librte_eal/common/include/rte_compat.h
F: lib/librte_eal/common/include/rte_function_versioning.h
F: doc/guides/rel_notes/deprecation.rst
+F: devtools/check-abi.sh
F: devtools/check-abi-version.sh
F: devtools/check-symbol-change.sh
+F: devtools/gen-abi.sh
F: devtools/update-abi.sh
F: devtools/update_version_map_abi.py
F: devtools/validate-abi.sh
diff --git a/devtools/check-abi.sh b/devtools/check-abi.sh
new file mode 100755
index 000000000..5872499ec
--- /dev/null
+++ b/devtools/check-abi.sh
@@ -0,0 +1,59 @@
+#!/bin/sh -e
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 Red Hat, Inc.
+
+if [ $# != 2 ] && [ $# != 3 ]; then
+ echo "Usage: $0 refdir newdir [warnonly]"
+ exit 1
+fi
+
+refdir=$1
+newdir=$2
+warnonly=${3:-}
+ABIDIFF_OPTIONS="--suppr $(dirname $0)/dpdk.abignore --no-added-syms"
+
+if [ ! -d $refdir ]; then
+ echo "Error: reference directory '$refdir' does not exist."
+ exit 1
+fi
+incdir=$(find $refdir -type d -a -name include)
+if [ -z "$incdir" ] || [ ! -e "$incdir" ]; then
+ echo "WARNING: could not identify a include directory for $refdir, expect false positives..."
+else
+ ABIDIFF_OPTIONS="$ABIDIFF_OPTIONS --headers-dir1 $incdir"
+fi
+
+if [ ! -d $newdir ]; then
+ echo "Error: directory to check '$newdir' does not exist."
+ exit 1
+fi
+incdir2=$(find $newdir -type d -a -name include)
+if [ -z "$incdir2" ] || [ ! -e "$incdir2" ]; then
+ echo "WARNING: could not identify a include directory for $newdir, expect false positives..."
+else
+ ABIDIFF_OPTIONS="$ABIDIFF_OPTIONS --headers-dir2 $incdir2"
+fi
+
+error=
+for dump in $(find $refdir -name "*.dump"); do
+ name=$(basename $dump)
+ # skip glue drivers, example librte_pmd_mlx5_glue.dump
+ # We can't rely on a suppression rule for now:
+ # https://sourceware.org/bugzilla/show_bug.cgi?id=25480
+ if [ "$name" != "${name%%_glue.dump}" ]; then
+ echo "Skipping ${dump}..."
+ continue
+ fi
+ dump2=$(find $newdir -name $name)
+ if [ -z "$dump2" ] || [ ! -e "$dump2" ]; then
+ echo "Error: can't find $name in $newdir"
+ error=1
+ continue
+ fi
+ if ! abidiff $ABIDIFF_OPTIONS $dump $dump2; then
+ echo "Error: ABI issue reported for 'abidiff $ABIDIFF_OPTIONS $dump $dump2'"
+ error=1
+ fi
+done
+
+[ -z "$error" ] || [ -n "$warnonly" ]
diff --git a/devtools/dpdk.abignore b/devtools/dpdk.abignore
new file mode 100644
index 000000000..f2903612c
--- /dev/null
+++ b/devtools/dpdk.abignore
@@ -0,0 +1,21 @@
+[suppress_function]
+ symbol_version = EXPERIMENTAL
+[suppress_variable]
+ symbol_version = EXPERIMENTAL
+
+; Explicit ignore for driver-only ABI
+[suppress_type]
+ name = rte_cryptodev_ops
+; Ignore this enum update as it is part of an experimental API
+[suppress_type]
+ type_kind = enum
+ name = rte_crypto_asym_xform_type
+ changed_enumerators = RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
+
+; FIXME
+[suppress_type]
+ type_kind = enum
+ name = rte_crypto_aead_algorithm
+ changed_enumerators = RTE_CRYPTO_AEAD_LIST_END
+[suppress_variable]
+ name = rte_crypto_aead_algorithm_strings
diff --git a/devtools/gen-abi.sh b/devtools/gen-abi.sh
new file mode 100755
index 000000000..c44b0e228
--- /dev/null
+++ b/devtools/gen-abi.sh
@@ -0,0 +1,26 @@
+#!/bin/sh -e
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 Red Hat, Inc.
+
+if [ $# != 1 ]; then
+ echo "Usage: $0 installdir"
+ exit 1
+fi
+
+installdir=$1
+if [ ! -d $installdir ]; then
+ echo "Error: install directory '$installdir' does not exist."
+ exit 1
+fi
+
+dumpdir=$installdir/dump
+rm -rf $dumpdir
+mkdir -p $dumpdir
+for f in $(find $installdir -name "*.so.*"); do
+ if test -L $f; then
+ continue
+ fi
+
+ libname=$(basename $f)
+ abidw --out-file $dumpdir/${libname%.so*}.dump $f
+done
diff --git a/devtools/test-build.sh b/devtools/test-build.sh
index 52305fbb8..a97e1280e 100755
--- a/devtools/test-build.sh
+++ b/devtools/test-build.sh
@@ -30,7 +30,8 @@ default_path=$PATH
# - LIBSSO_SNOW3G_PATH
# - LIBSSO_KASUMI_PATH
# - LIBSSO_ZUC_PATH
-. $(dirname $(readlink -f $0))/load-devel-config
+devtools_dir=$(dirname $(readlink -f $0))
+. $devtools_dir/load-devel-config
print_usage () {
echo "usage: $(basename $0) [-h] [-jX] [-s] [config1 [config2] ...]]"
@@ -64,10 +65,12 @@ print_help () {
[ -z $MAKE ] && echo "Cannot find make or gmake" && exit 1
J=$DPDK_MAKE_JOBS
+refsrcdir=$(mktemp -d -t dpdk-${DPDK_ABI_REF_VERSION:-}.XXX)
builds_dir=${DPDK_BUILD_TEST_DIR:-.}
short=false
unset verbose
-maxerr=-Wfatal-errors
+# for ABI checks, we need debuginfo
+test_cflags="-Wfatal-errors -g"
while getopts hj:sv ARG ; do
case $ARG in
j ) J=$OPTARG ;;
@@ -91,13 +94,14 @@ on_exit ()
[ "$DPDK_NOTIFY" != notify-send ] || \
notify-send -u low --icon=dialog-error 'DPDK build' 'failed'
fi
+ rm -rf $refsrcdir
}
# catch manual interrupt to ignore notification
trap "signal=INT ; trap - INT ; kill -INT $$" INT
# notify result on exit
trap on_exit EXIT
-cd $(dirname $(readlink -f $0))/..
+cd $devtools_dir/..
reset_env ()
{
@@ -233,14 +237,14 @@ for conf in $configs ; do
# reload config with DPDK_TARGET set
DPDK_TARGET=$target
reset_env
- . $(dirname $(readlink -f $0))/load-devel-config
+ . $devtools_dir/load-devel-config
options=$(echo $conf | sed 's,[^~+]*,,')
dir=$builds_dir/$conf
config $dir $target $options
echo "================== Build $conf"
- ${MAKE} -j$J EXTRA_CFLAGS="$maxerr $DPDK_DEP_CFLAGS" \
+ ${MAKE} -j$J EXTRA_CFLAGS="$test_cflags $DPDK_DEP_CFLAGS" \
EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS" $verbose O=$dir
! $short || break
export RTE_TARGET=$target
@@ -253,6 +257,43 @@ for conf in $configs ; do
EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS" $verbose \
O=$(readlink -f $dir)/examples
unset RTE_TARGET
+ if [ -n "$DPDK_ABI_REF_VERSION" ]; then
+ DPDK_ABI_REF_DIR=${DPDK_ABI_REF_DIR:-reference}
+ abirefdir=$DPDK_ABI_REF_DIR/$DPDK_ABI_REF_VERSION/$conf
+ if [ ! -d $abirefdir ]; then
+ # clone current sources
+ if [ ! -d $refsrcdir/.git ]; then
+ git clone --local --no-hardlinks \
+ --single-branch \
+ -b $DPDK_ABI_REF_VERSION \
+ $(pwd) $refsrcdir
+ fi
+
+ cd $refsrcdir
+
+ rm -rf build
+ config build $target $options
+
+ echo -n "================== Build $conf "
+ echo "($DPDK_ABI_REF_VERSION)"
+ ${MAKE} -j$J \
+ EXTRA_CFLAGS="$test_cflags $DPDK_DEP_CFLAGS" \
+ EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS" $verbose \
+ O=build
+ ! $short || break
+ export RTE_TARGET=$target
+ ${MAKE} install O=build DESTDIR=$abirefdir \
+ prefix=
+ $devtools_dir/gen-abi.sh $abirefdir
+
+ # back to current workdir
+ cd $devtools_dir/..
+ fi
+
+ echo "================== Check ABI $conf"
+ $devtools_dir/gen-abi.sh $dir/install
+ $devtools_dir/check-abi.sh $abirefdir $dir/install
+ fi
echo "################## $conf done."
unset dir
done
diff --git a/devtools/test-meson-builds.sh b/devtools/test-meson-builds.sh
index 747fb4a1a..23e93770d 100755
--- a/devtools/test-meson-builds.sh
+++ b/devtools/test-meson-builds.sh
@@ -16,8 +16,15 @@ srcdir=$(dirname $(readlink -f $0))/..
MESON=${MESON:-meson}
use_shared="--default-library=shared"
+refsrcdir=$(mktemp -d -t dpdk-${DPDK_ABI_REF_VERSION:-}.XXX)
builds_dir=${DPDK_BUILD_TEST_DIR:-.}
+on_exit ()
+{
+ rm -rf $refsrcdir
+}
+trap on_exit EXIT
+
if command -v gmake >/dev/null 2>&1 ; then
MAKE=gmake
else
@@ -64,9 +71,14 @@ config () # <dir> <builddir> <meson options>
builddir=$1
shift
if [ -f "$builddir/build.ninja" ] ; then
+ # for existing environments, force debugoptimized so that ABI
+ # checks can run
+ $MESON configure --buildtype=debugoptimized $builddir
return
fi
- options="--werror -Dexamples=all -Dlibdir=lib"
+ options=
+ options="$options --werror -Dexamples=all -Dlibdir=lib"
+ options="$options --buildtype=debugoptimized"
for option in $DPDK_MESON_OPTIONS ; do
options="$options -D$option"
done
@@ -107,6 +119,29 @@ build () # <directory> <target compiler> <meson options>
config $srcdir $builds_dir/$targetdir $*
compile $builds_dir/$targetdir \
$(readlink -f $builds_dir/$targetdir/install)
+ if [ -n "$DPDK_ABI_REF_VERSION" ]; then
+ DPDK_ABI_REF_DIR=${DPDK_ABI_REF_DIR:-reference}
+ abirefdir=$DPDK_ABI_REF_DIR/$DPDK_ABI_REF_VERSION/$targetdir
+ if [ ! -d $abirefdir ]; then
+ # clone current sources
+ if [ ! -d $refsrcdir/.git ]; then
+ git clone --local --no-hardlinks \
+ --single-branch \
+ -b $DPDK_ABI_REF_VERSION \
+ $srcdir $refsrcdir
+ fi
+
+ rm -rf $refsrcdir/build
+ config $refsrcdir $refsrcdir/build $*
+ compile $refsrcdir/build $abirefdir
+ $srcdir/devtools/gen-abi.sh $abirefdir
+ fi
+
+ $srcdir/devtools/gen-abi.sh \
+ $(readlink -f $builds_dir/$targetdir/install)
+ $srcdir/devtools/check-abi.sh $abirefdir \
+ $(readlink -f $builds_dir/$targetdir/install)
+ fi
}
if [ "$1" = "-vv" ] ; then
diff --git a/doc/guides/contributing/patches.rst b/doc/guides/contributing/patches.rst
index 0686450e4..59442824a 100644
--- a/doc/guides/contributing/patches.rst
+++ b/doc/guides/contributing/patches.rst
@@ -513,6 +513,21 @@ in a single subfolder called "__builds" created in the current directory.
Setting ``DPDK_BUILD_TEST_DIR`` to an absolute directory path e.g. ``/tmp`` is also supported.
+Checking ABI compatibility
+--------------------------
+
+By default, ABI compatibility checks are disabled.
+
+To enable them, a reference version must be selected via the environment
+variable ``DPDK_ABI_REF_VERSION``.
+
+The ``devtools/test-build.sh`` and ``devtools/test-meson-builds.sh`` scripts
+then build this reference version in a temporary directory and store the
+results in a subfolder of the current working directory.
+The environment variable ``DPDK_ABI_REF_DIR`` can be set so that the results go
+to a different location.
+
+
Sending Patches
---------------
--
2.23.0
^ permalink raw reply [relevance 28%]
* [dpdk-dev] [PATCH v3 3/4] build: test meson installation
2020-01-30 16:00 9% ` [dpdk-dev] [PATCH v3 0/4] " David Marchand
@ 2020-01-30 16:00 3% ` David Marchand
2020-01-30 22:17 3% ` Thomas Monjalon
2020-01-30 16:00 28% ` [dpdk-dev] [PATCH v3 4/4] add ABI checks David Marchand
1 sibling, 1 reply; 200+ results
From: David Marchand @ 2020-01-30 16:00 UTC (permalink / raw)
To: dev
Cc: thomas, bruce.richardson, kevin.laatz, aconole, nhorman,
akhil.goyal, anoobj, bluca, fiona.trahe, ferruh.yigit,
Michael Santana
Let's test installing with meson as part of Travis and the
devtools/test-meson-builds.sh script.
The resulting headers and binaries make more sense to run checks
against, since they should be the same as what the final users get.
test-null.sh is now called on an installed testpmd.
The (future) ABI compatibility checks will also need to filter its report
against installed public headers.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Luca Boccassi <bluca@debian.org>
---
Changelog since v2:
- removed forced --prefix= parameter in test-meson-builds.sh to avoid
breakage of existing environments,
- simplified install with no special verbose mode,
- removed ldd call in test-null.sh since the main user of this script
finds this change too much for his eyes,
---
.ci/linux-build.sh | 6 +++++-
devtools/test-meson-builds.sh | 21 +++++++++++----------
2 files changed, 16 insertions(+), 11 deletions(-)
diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index ccc3a7ccd..e61aa2b0a 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -29,11 +29,15 @@ if [ "$BUILD_32BIT" = "1" ]; then
fi
OPTS="$OPTS --default-library=$DEF_LIB"
+OPTS="$OPTS --prefix=/usr -Dlibdir=lib"
meson build --werror -Dexamples=all $OPTS
ninja -C build
+DESTDIR=$(pwd)/install ninja -C build install
if [ "$AARCH64" != "1" ]; then
- devtools/test-null.sh
+ export LD_LIBRARY_PATH=$(pwd)/install/usr/lib
+ devtools/test-null.sh install/usr/bin/dpdk-testpmd
+ unset LD_LIBRARY_PATH
fi
if [ "$RUN_TESTS" = "1" ]; then
diff --git a/devtools/test-meson-builds.sh b/devtools/test-meson-builds.sh
index fb6c404e5..747fb4a1a 100755
--- a/devtools/test-meson-builds.sh
+++ b/devtools/test-meson-builds.sh
@@ -66,7 +66,7 @@ config () # <dir> <builddir> <meson options>
if [ -f "$builddir/build.ninja" ] ; then
return
fi
- options="--werror -Dexamples=all"
+ options="--werror -Dexamples=all -Dlibdir=lib"
for option in $DPDK_MESON_OPTIONS ; do
options="$options -D$option"
done
@@ -75,7 +75,7 @@ config () # <dir> <builddir> <meson options>
$MESON $options $dir $builddir
}
-compile () # <builddir>
+compile () # <builddir> <installdir>
{
builddir=$1
if [ -n "$TEST_MESON_BUILD_VERY_VERBOSE" ] ; then
@@ -90,6 +90,9 @@ compile () # <builddir>
echo "$ninja_cmd -C $builddir"
$ninja_cmd -C $builddir
fi
+ rm -rf $2
+ echo "DESTDIR=$2 $ninja_cmd -C $builddir install"
+ DESTDIR=$2 $ninja_cmd -C $builddir install
}
build () # <directory> <target compiler> <meson options>
@@ -102,7 +105,8 @@ build () # <directory> <target compiler> <meson options>
command -v ${CC##* } >/dev/null 2>&1 || return 0
load_env $targetcc || return 0
config $srcdir $builds_dir/$targetdir $*
- compile $builds_dir/$targetdir
+ compile $builds_dir/$targetdir \
+ $(readlink -f $builds_dir/$targetdir/install)
}
if [ "$1" = "-vv" ] ; then
@@ -134,7 +138,7 @@ ok=$(cc -march=$default_machine -E - < /dev/null > /dev/null 2>&1 || echo false)
if [ "$ok" = "false" ] ; then
default_machine='corei7'
fi
-build build-x86-default cc -Dlibdir=lib -Dmachine=$default_machine $use_shared
+build build-x86-default cc -Dmachine=$default_machine $use_shared
c=aarch64-linux-gnu-gcc
# generic armv8a with clang as host compiler
@@ -150,12 +154,9 @@ for f in $srcdir/config/arm/arm64_[bdo]*gcc ; do
unset CC
done
-# Test installation of the x86-default target, to be used for checking
-# the sample apps build using the pkg-config file for cflags and libs
-build_path=$(readlink -f $builds_dir/build-x86-default)
-export DESTDIR=$build_path/install-root
-$ninja_cmd -C $build_path install
-
+# Use the x86-default target, to check the sample apps build using the
+# pkg-config file for cflags and libs
+export DESTDIR=$(readlink -f $builds_dir/build-x86-default/install)
load_env cc
pc_file=$(find $DESTDIR -name libdpdk.pc)
export PKG_CONFIG_PATH=$(dirname $pc_file):$PKG_CONFIG_PATH
--
2.23.0
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v3 0/4] add ABI checks
@ 2020-01-30 16:00 9% ` David Marchand
2020-01-30 16:00 3% ` [dpdk-dev] [PATCH v3 3/4] build: test meson installation David Marchand
2020-01-30 16:00 28% ` [dpdk-dev] [PATCH v3 4/4] add ABI checks David Marchand
2020-02-02 21:08 9% ` [dpdk-dev] [PATCH v4 0/3] " David Marchand
3 siblings, 2 replies; 200+ results
From: David Marchand @ 2020-01-30 16:00 UTC (permalink / raw)
To: dev
Cc: thomas, bruce.richardson, kevin.laatz, aconole, nhorman,
akhil.goyal, anoobj, bluca, fiona.trahe, ferruh.yigit
Here is the current state of the ABI checks.
libabigail has some issues when mixing dump and so files compiled with
clang [1], so for now, all checks are done on dumps only.
libabigail 1.0-rc3 in Xenial reported issues that disappear with the
version 1.2 in Bionic.
To avoid getting warnings on internal types like [2], the checks now make
use of the public headers part of a dpdk installation (patch 2 and 3 to
prepare for this).
Some internal rte_hash headers were installed by meson, so patch 1 fixes
this.
The most important point, abidiff complains on the rc1 cryptodev changes:
- Chacha20-Poly1305 AEAD support,
- ECPM and ECDSA support
A suppression rule has been put on the internal type rte_cryptodev_ops.
But other changes are an ABI breakage afaiu. I put suppression rules on
them so that we can run the checks, but some action must be taken for
20.02 if my analysis is confirmed.
Special thanks to Dodji the libabigail maintainer for helping on this
topic.
1: https://sourceware.org/bugzilla/show_bug.cgi?id=25409
2: http://inbox.dpdk.org/dev/CAJFAV8yFKoDZROX9Mkyp7pDMvXw3e7mHwxjfrcjD5ZoFB2tZ8w@mail.gmail.com/
--
David Marchand
Changelog since v2:
- incorporated offlist Thomas comments wrt to existing build envs (do
not change --prefix, force builds with debug enabled)
David Marchand (4):
hash: fix meson headers packaging
build: split build helper
build: test meson installation
add ABI checks
.ci/linux-build.sh | 29 ++++++++-
.travis.yml | 20 +++++-
MAINTAINERS | 2 +
devtools/check-abi.sh | 59 +++++++++++++++++
devtools/dpdk.abignore | 21 +++++++
devtools/gen-abi.sh | 26 ++++++++
devtools/test-build.sh | 51 +++++++++++++--
devtools/test-meson-builds.sh | 98 ++++++++++++++++++++++-------
doc/guides/contributing/patches.rst | 15 +++++
lib/librte_hash/meson.build | 5 +-
10 files changed, 292 insertions(+), 34 deletions(-)
create mode 100755 devtools/check-abi.sh
create mode 100644 devtools/dpdk.abignore
create mode 100755 devtools/gen-abi.sh
--
2.23.0
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH v2 4/4] add ABI checks
@ 2020-01-30 15:59 4% ` Thomas Monjalon
2020-01-30 16:42 7% ` Ferruh Yigit
2020-01-30 23:49 4% ` Ananyev, Konstantin
0 siblings, 2 replies; 200+ results
From: Thomas Monjalon @ 2020-01-30 15:59 UTC (permalink / raw)
To: Anoob Joseph, akhil.goyal, Trahe, Fiona
Cc: dev, David Marchand, Richardson, Bruce, nhorman, Mcnamara, John,
Trahe, Fiona, Kusztal, ArkadiuszX, Yigit, Ferruh
30/01/2020 14:06, Trahe, Fiona:
> We were unaware the LIST_END change could constitute an ABI breakage, but can see how it affects the array size when picked up.
> We're exploring options.
>
> I agree with Anoob's point that if we don't allow the LIST_END to be modified, then it means no feature can be implemented without ABI breakage.
> Anyone object to removing those LIST_END elements - or have a better suggestion? Would have to be in 20.11 I suppose.
Yes, having max value right after the last value is ridiculous,
it prevents adding any value.
In 20.11, we should remove all these *_END and *_MAX from API enums
and replace them with a separate #define with reasonnable maximums.
^ permalink raw reply [relevance 4%]
Results 6601-6800 of ~18000 next (older) | prev (newer) | reverse | sort options + mbox downloads above
-- links below jump to the message on this page --
2019-08-05 12:30 [dpdk-dev] [PATCH] doc: remove major in designation of normal releases Thomas Monjalon
2020-02-13 20:51 0% ` Kevin Traynor
2019-10-25 15:46 [dpdk-dev] [PATCH v5] eal: add manual probing option Gaetan Rivet
2020-01-23 9:58 ` [dpdk-dev] [PATCH v7] " Gaetan Rivet
2020-02-03 5:16 ` Pavan Nikhilesh Bhagavatula
2020-02-03 22:21 3% ` Thomas Monjalon
2020-02-04 10:03 0% ` Gaetan Rivet
2020-02-04 11:07 0% ` Thomas Monjalon
2020-02-04 12:43 0% ` Gaetan Rivet
2020-02-04 15:06 0% ` Thomas Monjalon
2020-02-04 16:02 0% ` Gaetan Rivet
2020-02-10 14:51 0% ` Jerin Jacob
2020-02-10 15:27 0% ` Thomas Monjalon
2020-02-10 16:33 0% ` Jerin Jacob
2019-11-13 21:43 [dpdk-dev] [PATCH v7 0/1] fbarray: fix duplicated fbarray file in secondary yasufum.o
2019-11-27 8:48 ` [dpdk-dev] [PATCH v8 " Yasufumi Ogawa
2019-11-27 8:48 ` [dpdk-dev] [PATCH v8 1/1] " Yasufumi Ogawa
2019-12-06 10:44 ` Burakov, Anatoly
2020-02-14 7:46 0% ` Yasufumi Ogawa
2020-02-14 15:08 ` David Marchand
2020-02-14 15:29 3% ` Thomas Monjalon
2020-02-17 12:54 0% ` Yasufumi Ogawa
2019-11-26 14:56 [dpdk-dev] [PATCH] service: don't walk out of bounds when checking services Aaron Conole
2019-12-03 21:15 ` [dpdk-dev] [PATCH v2] " Aaron Conole
2019-12-04 8:33 ` David Marchand
2019-12-04 8:34 ` David Marchand
2019-12-20 14:43 ` David Marchand
2020-02-07 12:04 ` Kevin Traynor
2020-02-07 14:27 3% ` Aaron Conole
2020-02-14 16:38 0% ` Kevin Traynor
2019-12-17 13:07 [dpdk-dev] [PATCH] meter: move RFC4115 trTCM APIs as none experimental Eelco Chaudron
2020-01-16 11:25 ` [dpdk-dev] [dpdk-stable] " David Marchand
2020-01-16 11:54 ` Neil Horman
2020-01-16 12:42 ` Ferruh Yigit
2020-01-30 16:33 4% ` Ray Kinsella
2019-12-20 6:56 [dpdk-dev] [PATCH v6 2/2] common/octeontx2: add polling based response mbox message Sunil Kumar Kori
2020-01-14 9:04 ` [dpdk-dev] [PATCH v7 1/2] eal: add API to check if its interrupt context Sunil Kumar Kori
2020-02-06 15:35 3% ` David Marchand
2019-12-20 15:20 [dpdk-dev] [PATCH] add ABI checks David Marchand
2020-01-29 17:26 ` [dpdk-dev] [PATCH v2 0/4] " David Marchand
2020-01-29 17:26 ` [dpdk-dev] [PATCH v2 4/4] " David Marchand
2020-01-29 17:42 ` Thomas Monjalon
2020-01-29 18:10 ` Anoob Joseph
2020-01-29 20:03 ` David Marchand
2020-01-29 20:13 ` Akhil Goyal
2020-01-30 16:09 8% ` Ferruh Yigit
2020-01-30 20:18 4% ` Thomas Monjalon
2020-01-31 9:03 7% ` Ferruh Yigit
2020-01-31 10:26 7% ` Ananyev, Konstantin
2020-01-31 14:16 4% ` Trahe, Fiona
2020-02-02 13:05 9% ` Thomas Monjalon
2020-02-02 14:41 4% ` Ananyev, Konstantin
2020-02-03 9:30 7% ` Ferruh Yigit
2020-02-03 11:50 9% ` Neil Horman
2020-02-03 13:09 7% ` Ferruh Yigit
2020-02-03 14:00 7% ` Dodji Seketeli
2020-02-03 14:46 4% ` Ferruh Yigit
2020-02-03 15:08 7% ` Trahe, Fiona
2020-02-03 17:09 9% ` Thomas Monjalon
2020-02-03 17:34 8% ` Thomas Monjalon
2020-02-03 18:55 7% ` Ray Kinsella
2020-02-03 21:07 7% ` Thomas Monjalon
2020-02-04 9:46 8% ` Ferruh Yigit
2020-02-04 10:24 9% ` Thomas Monjalon
2020-02-04 12:44 4% ` Trahe, Fiona
2020-02-04 15:52 8% ` Trahe, Fiona
2020-02-04 15:59 7% ` Thomas Monjalon
2020-02-04 17:46 4% ` Trahe, Fiona
2020-02-13 14:51 4% ` Kusztal, ArkadiuszX
2020-02-04 12:57 4% ` Kevin Traynor
2020-02-04 14:44 4% ` Aaron Conole
2020-02-04 19:49 7% ` Neil Horman
2020-02-04 9:51 4% ` David Marchand
2020-02-04 10:10 4% ` Trahe, Fiona
2020-02-04 10:38 4% ` Thomas Monjalon
2020-02-05 11:10 4% ` Ray Kinsella
2020-02-03 17:40 8% ` Ferruh Yigit
2020-02-03 18:40 9% ` Thomas Monjalon
2020-02-04 9:19 4% ` Ferruh Yigit
2020-02-04 9:45 4% ` Thomas Monjalon
2020-02-04 9:56 4% ` Ferruh Yigit
2020-02-04 10:08 4% ` Bruce Richardson
2020-02-04 10:17 4% ` Kevin Traynor
2020-02-04 10:16 9% ` Akhil Goyal
2020-02-04 10:28 8% ` Thomas Monjalon
2020-02-04 10:32 4% ` Akhil Goyal
2020-02-04 11:35 4% ` Bruce Richardson
2020-02-04 22:10 7% ` Neil Horman
2020-02-05 6:16 7% ` [dpdk-dev] [EXT] " Anoob Joseph
2020-02-05 14:33 4% ` Trahe, Fiona
2020-02-04 21:59 8% ` [dpdk-dev] " Neil Horman
2020-01-30 13:06 ` Trahe, Fiona
2020-01-30 15:59 4% ` Thomas Monjalon
2020-01-30 16:42 7% ` Ferruh Yigit
2020-01-30 23:49 4% ` Ananyev, Konstantin
2020-01-31 9:07 4% ` Ferruh Yigit
2020-01-31 9:37 4% ` Ananyev, Konstantin
2020-01-30 16:00 9% ` [dpdk-dev] [PATCH v3 0/4] " David Marchand
2020-01-30 16:00 3% ` [dpdk-dev] [PATCH v3 3/4] build: test meson installation David Marchand
2020-01-30 22:17 3% ` Thomas Monjalon
2020-01-30 16:00 28% ` [dpdk-dev] [PATCH v3 4/4] add ABI checks David Marchand
2020-01-30 22:32 4% ` Thomas Monjalon
2020-02-01 15:29 4% ` David Marchand
2020-01-30 22:44 4% ` Thomas Monjalon
2020-02-02 21:08 9% ` [dpdk-dev] [PATCH v4 0/3] " David Marchand
2020-02-02 21:08 28% ` [dpdk-dev] [PATCH v4 3/3] " David Marchand
2020-02-05 14:13 4% ` [dpdk-dev] [PATCH v4 0/3] " Thomas Monjalon
2020-01-04 1:33 [dpdk-dev] [PATCH 00/14] cleanup resources on shutdown Stephen Hemminger
2020-02-05 9:32 0% ` David Marchand
2020-02-05 12:07 0% ` Stephen Hemminger
2020-02-05 12:32 0% ` David Marchand
2020-01-13 10:43 [dpdk-dev] [PATCH v3] cryptodev: add chacha20-poly1305 aead algorithm Arek Kusztal
2020-01-14 13:18 ` Anoob Joseph
2020-01-15 15:53 ` Akhil Goyal
2020-02-05 14:28 5% ` Thomas Monjalon
2020-02-05 14:41 0% ` Trahe, Fiona
2020-02-05 15:21 0% ` Thomas Monjalon
2020-01-22 10:52 [dpdk-dev] [PATCH v2 1/2] crypto/ccp: sha3 support enabling in ccp ssardar
2020-01-22 11:11 ` Kumar, Ravi1
2020-02-05 12:24 ` Akhil Goyal
2020-02-05 22:22 3% ` Thomas Monjalon
2020-02-06 9:34 0% ` David Marchand
2020-01-27 10:30 [dpdk-dev] [PATCH 0/2] app/testpmd: add portlist option to the testpmd Hariprasad Govindharajan
2020-01-27 10:30 ` [dpdk-dev] [PATCH 2/2] eal: add eal_parse_optionlist to parse user input Hariprasad Govindharajan
2020-01-28 17:35 ` Ferruh Yigit
2020-01-29 17:44 ` David Marchand
2020-01-29 18:07 ` Ferruh Yigit
2020-01-29 19:19 ` Stephen Hemminger
2020-01-31 11:25 0% ` Govindharajan, Hariprasad
2020-01-31 14:42 0% ` Govindharajan, Hariprasad
2020-01-29 12:29 [dpdk-dev] [RFC] meter: fix ABI break due to experimental tag removal Ferruh Yigit
2020-01-29 16:43 ` [dpdk-dev] [RFC v2] " Ferruh Yigit
2020-01-30 12:33 ` Thomas Monjalon
2020-01-31 9:25 4% ` Ferruh Yigit
2020-01-30 14:17 ` Thomas Monjalon
2020-01-30 14:21 ` Luca Boccassi
2020-01-30 15:55 ` Thomas Monjalon
2020-01-30 16:04 4% ` Luca Boccassi
2020-01-30 16:15 4% ` Eelco Chaudron
2020-01-30 20:20 4% ` Thomas Monjalon
2020-02-03 11:10 4% ` Eelco Chaudron
2020-02-02 18:53 7% ` [dpdk-dev] [RFC] " Neil Horman
2020-02-03 12:53 4% ` Ferruh Yigit
2020-02-04 12:02 9% ` Neil Horman
2020-02-05 10:04 4% ` Luca Boccassi
2020-02-05 11:32 7% ` Neil Horman
2020-02-13 17:40 8% ` Ray Kinsella
2020-02-14 2:40 7% ` Neil Horman
2020-02-14 11:36 7% ` Bruce Richardson
2020-02-14 20:48 4% ` Neil Horman
2020-02-14 21:52 4% ` Bruce Richardson
2020-02-15 13:43 4% ` Neil Horman
2020-02-17 14:23 8% ` Ray Kinsella
2020-02-17 15:37 4% ` Neil Horman
2020-01-29 16:23 [dpdk-dev] [PATCH] build: remove function versioning from meson files Andrzej Ostruszka
2020-01-30 9:25 ` [dpdk-dev] [PATCH v2] build: remove unused function versioning Andrzej Ostruszka
2020-01-30 16:11 3% ` David Marchand
2020-02-05 20:44 0% ` David Marchand
2020-01-30 14:20 [dpdk-dev] [PATCH] doc: deprecate using MAX values as array size Ferruh Yigit
2020-02-21 10:25 0% ` Ferruh Yigit
2020-01-30 16:58 4% [dpdk-dev] [PATCH] doc: deprecation notice to move igb_uio to a new repo jerinj
2020-01-30 17:30 0% ` Ferruh Yigit
2020-01-31 12:36 4% [dpdk-dev] Minutes of Technical Board Meeting, 2020-01-29 Olivier Matz
2020-02-03 14:53 5% [dpdk-dev] Minutes of Technical Board Meeting, 2020-01-15 Maxime Coquelin
2020-02-04 23:05 3% [dpdk-dev] [RFC PATCH 0/7] vfio/pci: SR-IOV support Alex Williamson
2020-02-11 11:18 0% ` Jerin Jacob
2020-02-05 14:34 3% [dpdk-dev] Travis warning about job configs David Marchand
2020-02-05 15:17 17% [dpdk-dev] [PATCH] doc: alias to experimental tag for stable apis Ray Kinsella
2020-02-05 15:43 17% ` [dpdk-dev] [PATCH v3] " Ray Kinsella
2020-02-18 10:46 17% ` [dpdk-dev] [PATCH v4] " Ray Kinsella
2020-02-05 15:28 17% [dpdk-dev] [PATCH v2] " Ray Kinsella
2020-02-05 17:07 [dpdk-dev] [PATCH] buildtools: allow pedantic empty pmdinfo Thomas Monjalon
2020-02-05 17:14 ` Bruce Richardson
2020-02-05 17:29 3% ` Thomas Monjalon
2020-02-06 10:32 4% [dpdk-dev] DPDK Release Status Meeting 6/02/2020 Ferruh Yigit
2020-02-06 12:04 7% [dpdk-dev] [PATCH v1] common/iavf: update the ABI version Haiyue Wang
2020-02-11 2:08 4% ` Ye Xiaolong
2020-02-06 22:33 4% [dpdk-dev] [dpdk-announce] release candidate 20.02-rc2 Thomas Monjalon
2020-02-10 10:27 3% [dpdk-dev] [PATCH] common/octeontx2: update version map Anoob Joseph
2020-02-11 14:04 0% ` David Marchand
2020-02-11 11:27 7% [dpdk-dev] [PATCH 1/1] raw/octeontx2_ep: update the ABI version Mahipal Challa
2020-02-11 14:05 4% ` David Marchand
2020-02-11 19:48 7% [dpdk-dev] [PATCH 1/2] common/mlx5: fix " Matan Azrad
2020-02-11 19:48 7% ` [dpdk-dev] [PATCH 2/2] vdpa/mlx5: " Matan Azrad
2020-02-11 20:31 4% ` Slava Ovsiienko
2020-02-11 20:31 4% ` [dpdk-dev] [PATCH 1/2] common/mlx5: " Slava Ovsiienko
2020-02-12 8:53 4% ` David Marchand
2020-02-11 23:05 2% [dpdk-dev] [PATCH 0/7] vfio/pci: SR-IOV support Alex Williamson
2020-02-14 4:57 0% ` Alexey Kardashevskiy
2020-02-14 15:27 0% ` Alex Williamson
2020-02-12 23:08 3% [dpdk-dev] [RFC 0/4] Enforce flag checking in API's Stephen Hemminger
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 1/4] ring: future proof flag settings Stephen Hemminger
2020-02-13 11:12 0% ` Ananyev, Konstantin
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 2/4] hash: check flags on creation Stephen Hemminger
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 3/4] stack: " Stephen Hemminger
2020-02-12 23:08 3% ` [dpdk-dev] [RFC 4/4] cfgfile: check flags value Stephen Hemminger
2020-02-13 11:20 3% [dpdk-dev] DPDK Release Status Meeting 13/02/2020 Ferruh Yigit
2020-02-16 21:58 [dpdk-dev] [dpdk-announce] release candidate 20.02-rc3 Thomas Monjalon
2020-02-20 9:41 4% ` Peng, Yuan
2020-02-17 11:35 [dpdk-dev] [PATCH 0/2] improve Travis coverage Thomas Monjalon
2020-02-18 21:20 ` [dpdk-dev] [PATCH v3 " Thomas Monjalon
2020-02-18 22:36 3% ` David Marchand
2020-02-17 13:59 9% [dpdk-dev] [PATCH] ci: build and use libabigail 1.6 David Marchand
2020-02-17 15:15 0% ` Thomas Monjalon
2020-02-17 18:47 0% ` Aaron Conole
2020-02-18 9:40 3% ` David Marchand
2020-02-18 11:18 0% ` David Marchand
2020-02-18 14:55 0% ` Aaron Conole
2020-02-18 15:07 0% ` David Marchand
2020-02-18 14:29 9% ` [dpdk-dev] [PATCH v2] " David Marchand
2020-02-18 15:46 0% ` Thomas Monjalon
2020-02-18 16:38 0% ` David Marchand
2020-02-19 8:13 0% ` Dodji Seketeli
2020-02-18 20:13 9% ` [dpdk-dev] [PATCH v3] " David Marchand
2020-02-18 20:31 0% ` Aaron Conole
2020-02-18 20:59 0% ` David Marchand
2020-02-17 15:38 3% [dpdk-dev] [PATCH] doc: plan splitting the ethdev ops struct Ferruh Yigit
2020-02-18 5:07 0% ` Jerin Jacob
2020-02-18 6:01 0% ` Stephen Hemminger
2020-02-21 10:40 0% ` Ferruh Yigit
2020-02-17 16:13 7% [dpdk-dev] [PATCH] net/ionic: fix ABI version Ferruh Yigit
2020-02-19 12:37 4% ` Ferruh Yigit
2020-02-17 23:15 9% [dpdk-dev] ABI version of experimental libraries Thomas Monjalon
2020-02-17 23:44 11% ` [dpdk-dev] [PATCH] build: fix soname for " Thomas Monjalon
2020-02-18 9:40 0% ` Bruce Richardson
2020-02-18 9:47 0% ` Thomas Monjalon
2020-02-18 9:42 7% ` [dpdk-dev] ABI version of " Bruce Richardson
2020-02-18 9:50 4% ` Thomas Monjalon
2020-02-18 10:36 4% ` Kinsella, Ray
2020-02-20 19:50 8% ` Ferruh Yigit
2020-02-20 19:54 10% ` [dpdk-dev] [PATCH] build: fix experimental library versioning Ferruh Yigit
2020-02-20 22:14 0% ` Luca Boccassi
2020-02-21 12:36 0% ` Ray Kinsella
2020-02-21 15:24 5% ` David Marchand
2020-02-21 15:34 3% ` Ferruh Yigit
2020-02-19 11:43 7% ` [dpdk-dev] ABI version of experimental libraries Neil Horman
2020-02-19 12:43 4% ` Thomas Monjalon
2020-02-19 13:50 4% ` Ray Kinsella
2020-02-19 21:17 4% ` Neil Horman
2020-02-18 10:46 13% [dpdk-dev] [PATCH] ethdev: add comment to warn of ABI breakage Ciara Power
2020-02-18 11:59 4% ` Ferruh Yigit
2020-02-18 13:13 4% ` Andrew Rybchenko
2020-02-18 14:20 4% ` Thomas Monjalon
2020-02-18 13:39 13% ` [dpdk-dev] [PATCH v2] " Ciara Power
2020-02-18 18:40 4% ` Ferruh Yigit
2020-02-19 18:53 2% [dpdk-dev] [PATCH v2 0/7] vfio/pci: SR-IOV support Alex Williamson
2020-02-19 19:41 [dpdk-dev] [PATCH 0/4] Reorganise Travis jobs David Marchand
2020-02-19 19:41 3% ` [dpdk-dev] [PATCH 4/4] ci: reorganise " David Marchand
2020-02-19 21:39 0% ` Aaron Conole
2020-02-20 10:42 0% ` Thomas Monjalon
2020-02-20 12:22 0% ` David Marchand
2020-02-20 14:35 0% ` Aaron Conole
2020-02-20 16:01 0% ` David Marchand
2020-02-20 0:18 0% ` dwilder
2020-02-20 11:07 0% ` Thomas Monjalon
2020-02-21 16:10 17% [dpdk-dev] [PATCH] devtools: skip experimental libraries in ABI check David Marchand
2020-02-21 16:13 4% ` Ferruh Yigit
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).