* Re: [dpdk-dev] [PATCH v2] eal/cpuflags: add x86 based cpu flags
@ 2020-03-27 13:18 5% ` Van Haaren, Harry
2020-03-27 15:04 0% ` David Marchand
2020-03-27 13:44 0% ` Neil Horman
1 sibling, 1 reply; 200+ results
From: Van Haaren, Harry @ 2020-03-27 13:18 UTC (permalink / raw)
To: David Marchand, Laatz, Kevin
Cc: dev, Richardson, Bruce, Neil Horman, Thomas Monjalon,
Honnappa Nagarahalli, Dodji Seketeli
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Friday, March 27, 2020 12:24 PM
> To: Laatz, Kevin <kevin.laatz@intel.com>
> Cc: dev <dev@dpdk.org>; Richardson, Bruce <bruce.richardson@intel.com>; Van
> Haaren, Harry <harry.van.haaren@intel.com>; Neil Horman
> <nhorman@tuxdriver.com>; Thomas Monjalon <thomas@monjalon.net>; Honnappa
> Nagarahalli <Honnappa.Nagarahalli@arm.com>; Dodji Seketeli <dodji@redhat.com>
> Subject: Re: [dpdk-dev] [PATCH v2] eal/cpuflags: add x86 based cpu flags
>
> On Wed, Mar 25, 2020 at 12:11 PM Kevin Laatz <kevin.laatz@intel.com> wrote:
> >
> > This patch adds CPU flags which will enable the detection of ISA
> > features available on more recent x86 based CPUs.
> >
> > The CPUID leaf information can be found in Section 1.7 of this
> > document:
> > https://software.intel.com/sites/default/files/managed/c5/15/architecture-
> instruction-set-extensions-programming-reference.pdf
> >
> > The following CPU flags are added in this patch:
> > - AVX-512 doubleword and quadword instructions.
> > - AVX-512 integer fused multiply-add instructions.
> > - AVX-512 conflict detection instructions.
> > - AVX-512 byte and word instructions.
> > - AVX-512 vector length instructions.
> > - AVX-512 vector bit manipulation instructions.
> > - AVX-512 vector bit manipulation 2 instructions.
> > - Galois field new instructions.
> > - Vector AES instructions.
> > - Vector carry-less multiply instructions.
> > - AVX-512 vector neural network instructions.
> > - AVX-512 for bit algorithm instructions.
> > - AVX-512 vector popcount instructions.
> > - Cache line demote instructions.
> > - Direct store instructions.
> > - Direct store 64B instructions.
> > - AVX-512 two register intersection instructions.
> >
> > Signed-off-by: Kevin Laatz <kevin.laatz@intel.com>
> > ---
> > lib/librte_eal/common/arch/x86/rte_cpuflags.c | 18 ++++++++++++++++++
> > .../common/include/arch/x86/rte_cpuflags.h | 18 ++++++++++++++++++
> > 2 files changed, 36 insertions(+)
> >
> > diff --git a/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> b/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> > index 6492df556..30439e795 100644
> > --- a/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> > +++ b/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> > @@ -120,6 +120,24 @@ const struct feature_entry rte_cpu_feature_table[] = {
> > FEAT_DEF(EM64T, 0x80000001, 0, RTE_REG_EDX, 29)
> >
> > FEAT_DEF(INVTSC, 0x80000007, 0, RTE_REG_EDX, 8)
> > +
> > + FEAT_DEF(AVX512DQ, 0x00000007, 0, RTE_REG_EBX, 17)
> > + FEAT_DEF(AVX512IFMA, 0x00000007, 0, RTE_REG_EBX, 21)
> > + FEAT_DEF(AVX512CD, 0x00000007, 0, RTE_REG_EBX, 28)
> > + FEAT_DEF(AVX512BW, 0x00000007, 0, RTE_REG_EBX, 30)
> > + FEAT_DEF(AVX512VL, 0x00000007, 0, RTE_REG_EBX, 31)
> > + FEAT_DEF(AVX512VBMI, 0x00000007, 0, RTE_REG_ECX, 1)
> > + FEAT_DEF(AVX512VBMI2, 0x00000007, 0, RTE_REG_ECX, 6)
> > + FEAT_DEF(GFNI, 0x00000007, 0, RTE_REG_ECX, 8)
> > + FEAT_DEF(VAES, 0x00000007, 0, RTE_REG_ECX, 9)
> > + FEAT_DEF(VPCLMULQDQ, 0x00000007, 0, RTE_REG_ECX, 10)
> > + FEAT_DEF(AVX512VNNI, 0x00000007, 0, RTE_REG_ECX, 11)
> > + FEAT_DEF(AVX512BITALG, 0x00000007, 0, RTE_REG_ECX, 12)
> > + FEAT_DEF(AVX512VPOPCNTDQ, 0x00000007, 0, RTE_REG_ECX, 14)
> > + FEAT_DEF(CLDEMOTE, 0x00000007, 0, RTE_REG_ECX, 25)
> > + FEAT_DEF(MOVDIRI, 0x00000007, 0, RTE_REG_ECX, 27)
> > + FEAT_DEF(MOVDIR64B, 0x00000007, 0, RTE_REG_ECX, 28)
> > + FEAT_DEF(AVX512VP2INTERSECT, 0x00000007, 0, RTE_REG_EDX, 8)
> > };
> >
> > int
> > diff --git a/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> b/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> > index 25ba47b96..f8f73b19f 100644
> > --- a/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> > +++ b/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> > @@ -113,6 +113,24 @@ enum rte_cpu_flag_t {
> > /* (EAX 80000007h) EDX features */
> > RTE_CPUFLAG_INVTSC, /**< INVTSC */
> >
> > + RTE_CPUFLAG_AVX512DQ, /**< AVX512 Doubleword and
> Quadword */
> > + RTE_CPUFLAG_AVX512IFMA, /**< AVX512 Integer Fused
> Multiply-Add */
> > + RTE_CPUFLAG_AVX512CD, /**< AVX512 Conflict Detection*/
> > + RTE_CPUFLAG_AVX512BW, /**< AVX512 Byte and Word */
> > + RTE_CPUFLAG_AVX512VL, /**< AVX512 Vector Length */
> > + RTE_CPUFLAG_AVX512VBMI, /**< AVX512 Vector Bit
> Manipulation */
> > + RTE_CPUFLAG_AVX512VBMI2, /**< AVX512 Vector Bit
> Manipulation 2 */
> > + RTE_CPUFLAG_GFNI, /**< Galois Field New
> Instructions */
> > + RTE_CPUFLAG_VAES, /**< Vector AES */
> > + RTE_CPUFLAG_VPCLMULQDQ, /**< Vector Carry-less Multiply
> */
> > + RTE_CPUFLAG_AVX512VNNI, /**< AVX512 Vector Neural
> Network Instructions */
> > + RTE_CPUFLAG_AVX512BITALG, /**< AVX512 Bit Algorithms */
> > + RTE_CPUFLAG_AVX512VPOPCNTDQ, /**< AVX512 Vector Popcount */
> > + RTE_CPUFLAG_CLDEMOTE, /**< Cache Line Demote */
> > + RTE_CPUFLAG_MOVDIRI, /**< Direct Store Instructions
> */
> > + RTE_CPUFLAG_MOVDIR64B, /**< Direct Store Instructions
> 64B */
> > + RTE_CPUFLAG_AVX512VP2INTERSECT, /**< AVX512 Two Register
> Intersection */
> > +
> > /* The last item */
> > RTE_CPUFLAG_NUMFLAGS, /**< This should always be the
> last! */
>
> This is seen as an ABI break because of the change on _NUMFLAGS:
> https://travis-ci.com/github/ovsrobot/dpdk/jobs/302524264#L2351
Correct a publicly exposed enum max value has changed - I don't believe this is an ABI break, but a backward compatible ABI change was expected with this patchset.
Code compiled against eg 19.11 or 20.02 is expected to continue operating correctly.
The new flags were only added at the end of the enum, ensuring to not change the meaning of any existing flags which would be compiled-in constants to the application binary.
The actual size of the CPU flags array is a DPDK internal structure (in a .c file), and is hidden from the application, and never allocated by an application - so no possible mismatch in ABI there? Applications compiled against the older ABI will just not know about the newer flags - but suffer no breakage.
@ABI compatibility folks, please review too - but to the best of my understanding this is not an ABI break, but a backwards compatible update of CPU flag lists?
Thanks for flagging the CI results David!
Regards, -Harry
^ permalink raw reply [relevance 5%]
* Re: [dpdk-dev] [PATCH v2] eal/cpuflags: add x86 based cpu flags
2020-03-27 13:18 5% ` Van Haaren, Harry
@ 2020-03-27 13:44 0% ` Neil Horman
2020-03-27 14:15 4% ` Thomas Monjalon
1 sibling, 1 reply; 200+ results
From: Neil Horman @ 2020-03-27 13:44 UTC (permalink / raw)
To: David Marchand
Cc: Kevin Laatz, dev, Bruce Richardson, Van Haaren Harry,
Thomas Monjalon, Honnappa Nagarahalli, Dodji Seketeli
On Fri, Mar 27, 2020 at 01:24:12PM +0100, David Marchand wrote:
> On Wed, Mar 25, 2020 at 12:11 PM Kevin Laatz <kevin.laatz@intel.com> wrote:
> >
> > This patch adds CPU flags which will enable the detection of ISA
> > features available on more recent x86 based CPUs.
> >
> > The CPUID leaf information can be found in Section 1.7 of this
> > document:
> > https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf
> >
> > The following CPU flags are added in this patch:
> > - AVX-512 doubleword and quadword instructions.
> > - AVX-512 integer fused multiply-add instructions.
> > - AVX-512 conflict detection instructions.
> > - AVX-512 byte and word instructions.
> > - AVX-512 vector length instructions.
> > - AVX-512 vector bit manipulation instructions.
> > - AVX-512 vector bit manipulation 2 instructions.
> > - Galois field new instructions.
> > - Vector AES instructions.
> > - Vector carry-less multiply instructions.
> > - AVX-512 vector neural network instructions.
> > - AVX-512 for bit algorithm instructions.
> > - AVX-512 vector popcount instructions.
> > - Cache line demote instructions.
> > - Direct store instructions.
> > - Direct store 64B instructions.
> > - AVX-512 two register intersection instructions.
> >
> > Signed-off-by: Kevin Laatz <kevin.laatz@intel.com>
> > ---
> > lib/librte_eal/common/arch/x86/rte_cpuflags.c | 18 ++++++++++++++++++
> > .../common/include/arch/x86/rte_cpuflags.h | 18 ++++++++++++++++++
> > 2 files changed, 36 insertions(+)
> >
> > diff --git a/lib/librte_eal/common/arch/x86/rte_cpuflags.c b/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> > index 6492df556..30439e795 100644
> > --- a/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> > +++ b/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> > @@ -120,6 +120,24 @@ const struct feature_entry rte_cpu_feature_table[] = {
> > FEAT_DEF(EM64T, 0x80000001, 0, RTE_REG_EDX, 29)
> >
> > FEAT_DEF(INVTSC, 0x80000007, 0, RTE_REG_EDX, 8)
> > +
> > + FEAT_DEF(AVX512DQ, 0x00000007, 0, RTE_REG_EBX, 17)
> > + FEAT_DEF(AVX512IFMA, 0x00000007, 0, RTE_REG_EBX, 21)
> > + FEAT_DEF(AVX512CD, 0x00000007, 0, RTE_REG_EBX, 28)
> > + FEAT_DEF(AVX512BW, 0x00000007, 0, RTE_REG_EBX, 30)
> > + FEAT_DEF(AVX512VL, 0x00000007, 0, RTE_REG_EBX, 31)
> > + FEAT_DEF(AVX512VBMI, 0x00000007, 0, RTE_REG_ECX, 1)
> > + FEAT_DEF(AVX512VBMI2, 0x00000007, 0, RTE_REG_ECX, 6)
> > + FEAT_DEF(GFNI, 0x00000007, 0, RTE_REG_ECX, 8)
> > + FEAT_DEF(VAES, 0x00000007, 0, RTE_REG_ECX, 9)
> > + FEAT_DEF(VPCLMULQDQ, 0x00000007, 0, RTE_REG_ECX, 10)
> > + FEAT_DEF(AVX512VNNI, 0x00000007, 0, RTE_REG_ECX, 11)
> > + FEAT_DEF(AVX512BITALG, 0x00000007, 0, RTE_REG_ECX, 12)
> > + FEAT_DEF(AVX512VPOPCNTDQ, 0x00000007, 0, RTE_REG_ECX, 14)
> > + FEAT_DEF(CLDEMOTE, 0x00000007, 0, RTE_REG_ECX, 25)
> > + FEAT_DEF(MOVDIRI, 0x00000007, 0, RTE_REG_ECX, 27)
> > + FEAT_DEF(MOVDIR64B, 0x00000007, 0, RTE_REG_ECX, 28)
> > + FEAT_DEF(AVX512VP2INTERSECT, 0x00000007, 0, RTE_REG_EDX, 8)
> > };
> >
> > int
> > diff --git a/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h b/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> > index 25ba47b96..f8f73b19f 100644
> > --- a/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> > +++ b/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> > @@ -113,6 +113,24 @@ enum rte_cpu_flag_t {
> > /* (EAX 80000007h) EDX features */
> > RTE_CPUFLAG_INVTSC, /**< INVTSC */
> >
> > + RTE_CPUFLAG_AVX512DQ, /**< AVX512 Doubleword and Quadword */
> > + RTE_CPUFLAG_AVX512IFMA, /**< AVX512 Integer Fused Multiply-Add */
> > + RTE_CPUFLAG_AVX512CD, /**< AVX512 Conflict Detection*/
> > + RTE_CPUFLAG_AVX512BW, /**< AVX512 Byte and Word */
> > + RTE_CPUFLAG_AVX512VL, /**< AVX512 Vector Length */
> > + RTE_CPUFLAG_AVX512VBMI, /**< AVX512 Vector Bit Manipulation */
> > + RTE_CPUFLAG_AVX512VBMI2, /**< AVX512 Vector Bit Manipulation 2 */
> > + RTE_CPUFLAG_GFNI, /**< Galois Field New Instructions */
> > + RTE_CPUFLAG_VAES, /**< Vector AES */
> > + RTE_CPUFLAG_VPCLMULQDQ, /**< Vector Carry-less Multiply */
> > + RTE_CPUFLAG_AVX512VNNI, /**< AVX512 Vector Neural Network Instructions */
> > + RTE_CPUFLAG_AVX512BITALG, /**< AVX512 Bit Algorithms */
> > + RTE_CPUFLAG_AVX512VPOPCNTDQ, /**< AVX512 Vector Popcount */
> > + RTE_CPUFLAG_CLDEMOTE, /**< Cache Line Demote */
> > + RTE_CPUFLAG_MOVDIRI, /**< Direct Store Instructions */
> > + RTE_CPUFLAG_MOVDIR64B, /**< Direct Store Instructions 64B */
> > + RTE_CPUFLAG_AVX512VP2INTERSECT, /**< AVX512 Two Register Intersection */
> > +
> > /* The last item */
> > RTE_CPUFLAG_NUMFLAGS, /**< This should always be the last! */
>
> This is seen as an ABI break because of the change on _NUMFLAGS:
> https://travis-ci.com/github/ovsrobot/dpdk/jobs/302524264#L2351
>
It shouldn't be, as the only API calls we expose that use rte_cpu_flag_t accept
it as an integer parameter to see if the flag is enabled. Theres no use of the
enum in a public array or any struct that is sized based on the number of flags,
so you should be good to go
Neil
>
> --
> David Marchand
>
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2] eal/cpuflags: add x86 based cpu flags
2020-03-27 13:44 0% ` Neil Horman
@ 2020-03-27 14:15 4% ` Thomas Monjalon
2020-03-27 14:32 3% ` Van Haaren, Harry
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-03-27 14:15 UTC (permalink / raw)
To: Neil Horman, Dodji Seketeli, mdr
Cc: David Marchand, Kevin Laatz, dev, Bruce Richardson,
Van Haaren Harry, Honnappa Nagarahalli
27/03/2020 14:44, Neil Horman:
> On Fri, Mar 27, 2020 at 01:24:12PM +0100, David Marchand wrote:
> > On Wed, Mar 25, 2020 at 12:11 PM Kevin Laatz <kevin.laatz@intel.com> wrote:
> > > --- a/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> > > +++ b/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> > > @@ -113,6 +113,24 @@ enum rte_cpu_flag_t {
> > > /* (EAX 80000007h) EDX features */
> > > RTE_CPUFLAG_INVTSC, /**< INVTSC */
> > >
> > > + RTE_CPUFLAG_AVX512DQ, /**< AVX512 Doubleword and Quadword */
> > > + RTE_CPUFLAG_AVX512IFMA, /**< AVX512 Integer Fused Multiply-Add */
> > > + RTE_CPUFLAG_AVX512CD, /**< AVX512 Conflict Detection*/
> > > + RTE_CPUFLAG_AVX512BW, /**< AVX512 Byte and Word */
> > > + RTE_CPUFLAG_AVX512VL, /**< AVX512 Vector Length */
> > > + RTE_CPUFLAG_AVX512VBMI, /**< AVX512 Vector Bit Manipulation */
> > > + RTE_CPUFLAG_AVX512VBMI2, /**< AVX512 Vector Bit Manipulation 2 */
> > > + RTE_CPUFLAG_GFNI, /**< Galois Field New Instructions */
> > > + RTE_CPUFLAG_VAES, /**< Vector AES */
> > > + RTE_CPUFLAG_VPCLMULQDQ, /**< Vector Carry-less Multiply */
> > > + RTE_CPUFLAG_AVX512VNNI, /**< AVX512 Vector Neural Network Instructions */
> > > + RTE_CPUFLAG_AVX512BITALG, /**< AVX512 Bit Algorithms */
> > > + RTE_CPUFLAG_AVX512VPOPCNTDQ, /**< AVX512 Vector Popcount */
> > > + RTE_CPUFLAG_CLDEMOTE, /**< Cache Line Demote */
> > > + RTE_CPUFLAG_MOVDIRI, /**< Direct Store Instructions */
> > > + RTE_CPUFLAG_MOVDIR64B, /**< Direct Store Instructions 64B */
> > > + RTE_CPUFLAG_AVX512VP2INTERSECT, /**< AVX512 Two Register Intersection */
> > > +
> > > /* The last item */
> > > RTE_CPUFLAG_NUMFLAGS, /**< This should always be the last! */
> >
> > This is seen as an ABI break because of the change on _NUMFLAGS:
> > https://travis-ci.com/github/ovsrobot/dpdk/jobs/302524264#L2351
> >
> It shouldn't be, as the only API calls we expose that use rte_cpu_flag_t accept
> it as an integer parameter to see if the flag is enabled. Theres no use of the
> enum in a public array or any struct that is sized based on the number of flags,
> so you should be good to go
Indeed I cannot imagine an ABI incompatibility in this case.
The only behaviour change is to accept new (higher) RTE_CPUFLAG values
in functions rte_cpu_get_flag_enabled() and rte_cpu_get_flag_name().
Is changing the range of valid values an ABI break?
Why is it flagged by libabigail?
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2] eal/cpuflags: add x86 based cpu flags
2020-03-27 14:15 4% ` Thomas Monjalon
@ 2020-03-27 14:32 3% ` Van Haaren, Harry
2020-03-27 14:36 3% ` Ray Kinsella
0 siblings, 1 reply; 200+ results
From: Van Haaren, Harry @ 2020-03-27 14:32 UTC (permalink / raw)
To: Thomas Monjalon, Neil Horman, Dodji Seketeli, mdr
Cc: David Marchand, Laatz, Kevin, dev, Richardson, Bruce,
Honnappa Nagarahalli
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Friday, March 27, 2020 2:16 PM
> To: Neil Horman <nhorman@tuxdriver.com>; Dodji Seketeli <dodji@redhat.com>;
> mdr@ashroe.eu
> Cc: David Marchand <david.marchand@redhat.com>; Laatz, Kevin
> <kevin.laatz@intel.com>; dev <dev@dpdk.org>; Richardson, Bruce
> <bruce.richardson@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Subject: Re: [dpdk-dev] [PATCH v2] eal/cpuflags: add x86 based cpu flags
>
> 27/03/2020 14:44, Neil Horman:
> > On Fri, Mar 27, 2020 at 01:24:12PM +0100, David Marchand wrote:
> > > On Wed, Mar 25, 2020 at 12:11 PM Kevin Laatz <kevin.laatz@intel.com>
> wrote:
> > > > --- a/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> > > > +++ b/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> > > > @@ -113,6 +113,24 @@ enum rte_cpu_flag_t {
> > > > /* (EAX 80000007h) EDX features */
> > > > RTE_CPUFLAG_INVTSC, /**< INVTSC */
> > > >
> > > > + RTE_CPUFLAG_AVX512DQ, /**< AVX512 Doubleword and
> Quadword */
> > > > + RTE_CPUFLAG_AVX512IFMA, /**< AVX512 Integer Fused
> Multiply-Add */
> > > > + RTE_CPUFLAG_AVX512CD, /**< AVX512 Conflict
> Detection*/
> > > > + RTE_CPUFLAG_AVX512BW, /**< AVX512 Byte and Word */
> > > > + RTE_CPUFLAG_AVX512VL, /**< AVX512 Vector Length */
> > > > + RTE_CPUFLAG_AVX512VBMI, /**< AVX512 Vector Bit
> Manipulation */
> > > > + RTE_CPUFLAG_AVX512VBMI2, /**< AVX512 Vector Bit
> Manipulation 2 */
> > > > + RTE_CPUFLAG_GFNI, /**< Galois Field New
> Instructions */
> > > > + RTE_CPUFLAG_VAES, /**< Vector AES */
> > > > + RTE_CPUFLAG_VPCLMULQDQ, /**< Vector Carry-less
> Multiply */
> > > > + RTE_CPUFLAG_AVX512VNNI, /**< AVX512 Vector Neural
> Network Instructions */
> > > > + RTE_CPUFLAG_AVX512BITALG, /**< AVX512 Bit Algorithms
> */
> > > > + RTE_CPUFLAG_AVX512VPOPCNTDQ, /**< AVX512 Vector Popcount
> */
> > > > + RTE_CPUFLAG_CLDEMOTE, /**< Cache Line Demote */
> > > > + RTE_CPUFLAG_MOVDIRI, /**< Direct Store
> Instructions */
> > > > + RTE_CPUFLAG_MOVDIR64B, /**< Direct Store
> Instructions 64B */
> > > > + RTE_CPUFLAG_AVX512VP2INTERSECT, /**< AVX512 Two Register
> Intersection */
> > > > +
> > > > /* The last item */
> > > > RTE_CPUFLAG_NUMFLAGS, /**< This should always be
> the last! */
> > >
> > > This is seen as an ABI break because of the change on _NUMFLAGS:
> > > https://travis-ci.com/github/ovsrobot/dpdk/jobs/302524264#L2351
> > >
> > It shouldn't be, as the only API calls we expose that use rte_cpu_flag_t
> accept
> > it as an integer parameter to see if the flag is enabled. Theres no use of
> the
> > enum in a public array or any struct that is sized based on the number of
> flags,
> > so you should be good to go
>
> Indeed I cannot imagine an ABI incompatibility in this case.
> The only behaviour change is to accept new (higher) RTE_CPUFLAG values
> in functions rte_cpu_get_flag_enabled() and rte_cpu_get_flag_name().
> Is changing the range of valid values an ABI break?
> Why is it flagged by libabigail?
If this enum _MAX value was used by the application to allocate an array, that later our DPDK code would write to it could cause out-of-bounds array accesses of the application supplied array. Abigail doesn't know what applications could use the value for, so it flags it.
IMO Abigail is right to flag it to us - a manual review to understand what that _MAX enum value is used for, and then decide on a case by case basis seems the best way forward to me.
Thanks Neil/Thomas for reviewing, as reply in this thread, I also believe this is not going to break ABI.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2] eal/cpuflags: add x86 based cpu flags
2020-03-27 14:32 3% ` Van Haaren, Harry
@ 2020-03-27 14:36 3% ` Ray Kinsella
2020-03-27 15:19 3% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Ray Kinsella @ 2020-03-27 14:36 UTC (permalink / raw)
To: Van Haaren, Harry, Thomas Monjalon, Neil Horman, Dodji Seketeli
Cc: David Marchand, Laatz, Kevin, dev, Richardson, Bruce,
Honnappa Nagarahalli
On 27/03/2020 14:32, Van Haaren, Harry wrote:
>> -----Original Message-----
>> From: Thomas Monjalon <thomas@monjalon.net>
>> Sent: Friday, March 27, 2020 2:16 PM
>> To: Neil Horman <nhorman@tuxdriver.com>; Dodji Seketeli <dodji@redhat.com>;
>> mdr@ashroe.eu
>> Cc: David Marchand <david.marchand@redhat.com>; Laatz, Kevin
>> <kevin.laatz@intel.com>; dev <dev@dpdk.org>; Richardson, Bruce
>> <bruce.richardson@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
>> Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
>> Subject: Re: [dpdk-dev] [PATCH v2] eal/cpuflags: add x86 based cpu flags
>>
>> 27/03/2020 14:44, Neil Horman:
>>> On Fri, Mar 27, 2020 at 01:24:12PM +0100, David Marchand wrote:
>>>> On Wed, Mar 25, 2020 at 12:11 PM Kevin Laatz <kevin.laatz@intel.com>
>> wrote:
>>>>> --- a/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
>>>>> +++ b/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
>>>>> @@ -113,6 +113,24 @@ enum rte_cpu_flag_t {
>>>>> /* (EAX 80000007h) EDX features */
>>>>> RTE_CPUFLAG_INVTSC, /**< INVTSC */
>>>>>
>>>>> + RTE_CPUFLAG_AVX512DQ, /**< AVX512 Doubleword and
>> Quadword */
>>>>> + RTE_CPUFLAG_AVX512IFMA, /**< AVX512 Integer Fused
>> Multiply-Add */
>>>>> + RTE_CPUFLAG_AVX512CD, /**< AVX512 Conflict
>> Detection*/
>>>>> + RTE_CPUFLAG_AVX512BW, /**< AVX512 Byte and Word */
>>>>> + RTE_CPUFLAG_AVX512VL, /**< AVX512 Vector Length */
>>>>> + RTE_CPUFLAG_AVX512VBMI, /**< AVX512 Vector Bit
>> Manipulation */
>>>>> + RTE_CPUFLAG_AVX512VBMI2, /**< AVX512 Vector Bit
>> Manipulation 2 */
>>>>> + RTE_CPUFLAG_GFNI, /**< Galois Field New
>> Instructions */
>>>>> + RTE_CPUFLAG_VAES, /**< Vector AES */
>>>>> + RTE_CPUFLAG_VPCLMULQDQ, /**< Vector Carry-less
>> Multiply */
>>>>> + RTE_CPUFLAG_AVX512VNNI, /**< AVX512 Vector Neural
>> Network Instructions */
>>>>> + RTE_CPUFLAG_AVX512BITALG, /**< AVX512 Bit Algorithms
>> */
>>>>> + RTE_CPUFLAG_AVX512VPOPCNTDQ, /**< AVX512 Vector Popcount
>> */
>>>>> + RTE_CPUFLAG_CLDEMOTE, /**< Cache Line Demote */
>>>>> + RTE_CPUFLAG_MOVDIRI, /**< Direct Store
>> Instructions */
>>>>> + RTE_CPUFLAG_MOVDIR64B, /**< Direct Store
>> Instructions 64B */
>>>>> + RTE_CPUFLAG_AVX512VP2INTERSECT, /**< AVX512 Two Register
>> Intersection */
>>>>> +
>>>>> /* The last item */
>>>>> RTE_CPUFLAG_NUMFLAGS, /**< This should always be
>> the last! */
>>>>
>>>> This is seen as an ABI break because of the change on _NUMFLAGS:
>>>> https://travis-ci.com/github/ovsrobot/dpdk/jobs/302524264#L2351
>>>>
>>> It shouldn't be, as the only API calls we expose that use rte_cpu_flag_t
>> accept
>>> it as an integer parameter to see if the flag is enabled. Theres no use of
>> the
>>> enum in a public array or any struct that is sized based on the number of
>> flags,
>>> so you should be good to go
>>
>> Indeed I cannot imagine an ABI incompatibility in this case.
>> The only behaviour change is to accept new (higher) RTE_CPUFLAG values
>> in functions rte_cpu_get_flag_enabled() and rte_cpu_get_flag_name().
>> Is changing the range of valid values an ABI break?
>> Why is it flagged by libabigail?
>
> If this enum _MAX value was used by the application to allocate an array, that later our DPDK code would write to it could cause out-of-bounds array accesses of the application supplied array. Abigail doesn't know what applications could use the value for, so it flags it.
>
> IMO Abigail is right to flag it to us - a manual review to understand what that _MAX enum value is used for, and then decide on a case by case basis seems the best way forward to me.
>
> Thanks Neil/Thomas for reviewing, as reply in this thread, I also believe this is not going to break ABI.
>
+1 to Harrry's comments.
I can't immediately see how this might break the ABI either.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2] eal/cpuflags: add x86 based cpu flags
2020-03-27 13:18 5% ` Van Haaren, Harry
@ 2020-03-27 15:04 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-03-27 15:04 UTC (permalink / raw)
To: Van Haaren, Harry, Laatz, Kevin
Cc: dev, Richardson, Bruce, Neil Horman, Thomas Monjalon,
Honnappa Nagarahalli, Dodji Seketeli
On Fri, Mar 27, 2020 at 2:18 PM Van Haaren, Harry
<harry.van.haaren@intel.com> wrote:
>
> > -----Original Message-----
> > From: David Marchand <david.marchand@redhat.com>
> > Sent: Friday, March 27, 2020 12:24 PM
> > To: Laatz, Kevin <kevin.laatz@intel.com>
> > Cc: dev <dev@dpdk.org>; Richardson, Bruce <bruce.richardson@intel.com>; Van
> > Haaren, Harry <harry.van.haaren@intel.com>; Neil Horman
> > <nhorman@tuxdriver.com>; Thomas Monjalon <thomas@monjalon.net>; Honnappa
> > Nagarahalli <Honnappa.Nagarahalli@arm.com>; Dodji Seketeli <dodji@redhat.com>
> > Subject: Re: [dpdk-dev] [PATCH v2] eal/cpuflags: add x86 based cpu flags
> >
> > On Wed, Mar 25, 2020 at 12:11 PM Kevin Laatz <kevin.laatz@intel.com> wrote:
> > >
> > > This patch adds CPU flags which will enable the detection of ISA
> > > features available on more recent x86 based CPUs.
> > >
> > > The CPUID leaf information can be found in Section 1.7 of this
> > > document:
> > > https://software.intel.com/sites/default/files/managed/c5/15/architecture-
> > instruction-set-extensions-programming-reference.pdf
> > >
> > > The following CPU flags are added in this patch:
> > > - AVX-512 doubleword and quadword instructions.
> > > - AVX-512 integer fused multiply-add instructions.
> > > - AVX-512 conflict detection instructions.
> > > - AVX-512 byte and word instructions.
> > > - AVX-512 vector length instructions.
> > > - AVX-512 vector bit manipulation instructions.
> > > - AVX-512 vector bit manipulation 2 instructions.
> > > - Galois field new instructions.
> > > - Vector AES instructions.
> > > - Vector carry-less multiply instructions.
> > > - AVX-512 vector neural network instructions.
> > > - AVX-512 for bit algorithm instructions.
> > > - AVX-512 vector popcount instructions.
> > > - Cache line demote instructions.
> > > - Direct store instructions.
> > > - Direct store 64B instructions.
> > > - AVX-512 two register intersection instructions.
> > >
> > > Signed-off-by: Kevin Laatz <kevin.laatz@intel.com>
> > > ---
> > > lib/librte_eal/common/arch/x86/rte_cpuflags.c | 18 ++++++++++++++++++
> > > .../common/include/arch/x86/rte_cpuflags.h | 18 ++++++++++++++++++
> > > 2 files changed, 36 insertions(+)
> > >
> > > diff --git a/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> > b/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> > > index 6492df556..30439e795 100644
> > > --- a/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> > > +++ b/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> > > @@ -120,6 +120,24 @@ const struct feature_entry rte_cpu_feature_table[] = {
> > > FEAT_DEF(EM64T, 0x80000001, 0, RTE_REG_EDX, 29)
> > >
> > > FEAT_DEF(INVTSC, 0x80000007, 0, RTE_REG_EDX, 8)
> > > +
> > > + FEAT_DEF(AVX512DQ, 0x00000007, 0, RTE_REG_EBX, 17)
> > > + FEAT_DEF(AVX512IFMA, 0x00000007, 0, RTE_REG_EBX, 21)
> > > + FEAT_DEF(AVX512CD, 0x00000007, 0, RTE_REG_EBX, 28)
> > > + FEAT_DEF(AVX512BW, 0x00000007, 0, RTE_REG_EBX, 30)
> > > + FEAT_DEF(AVX512VL, 0x00000007, 0, RTE_REG_EBX, 31)
> > > + FEAT_DEF(AVX512VBMI, 0x00000007, 0, RTE_REG_ECX, 1)
> > > + FEAT_DEF(AVX512VBMI2, 0x00000007, 0, RTE_REG_ECX, 6)
> > > + FEAT_DEF(GFNI, 0x00000007, 0, RTE_REG_ECX, 8)
> > > + FEAT_DEF(VAES, 0x00000007, 0, RTE_REG_ECX, 9)
> > > + FEAT_DEF(VPCLMULQDQ, 0x00000007, 0, RTE_REG_ECX, 10)
> > > + FEAT_DEF(AVX512VNNI, 0x00000007, 0, RTE_REG_ECX, 11)
> > > + FEAT_DEF(AVX512BITALG, 0x00000007, 0, RTE_REG_ECX, 12)
> > > + FEAT_DEF(AVX512VPOPCNTDQ, 0x00000007, 0, RTE_REG_ECX, 14)
> > > + FEAT_DEF(CLDEMOTE, 0x00000007, 0, RTE_REG_ECX, 25)
> > > + FEAT_DEF(MOVDIRI, 0x00000007, 0, RTE_REG_ECX, 27)
> > > + FEAT_DEF(MOVDIR64B, 0x00000007, 0, RTE_REG_ECX, 28)
> > > + FEAT_DEF(AVX512VP2INTERSECT, 0x00000007, 0, RTE_REG_EDX, 8)
> > > };
> > >
> > > int
> > > diff --git a/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> > b/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> > > index 25ba47b96..f8f73b19f 100644
> > > --- a/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> > > +++ b/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> > > @@ -113,6 +113,24 @@ enum rte_cpu_flag_t {
> > > /* (EAX 80000007h) EDX features */
> > > RTE_CPUFLAG_INVTSC, /**< INVTSC */
> > >
> > > + RTE_CPUFLAG_AVX512DQ, /**< AVX512 Doubleword and
> > Quadword */
> > > + RTE_CPUFLAG_AVX512IFMA, /**< AVX512 Integer Fused
> > Multiply-Add */
> > > + RTE_CPUFLAG_AVX512CD, /**< AVX512 Conflict Detection*/
> > > + RTE_CPUFLAG_AVX512BW, /**< AVX512 Byte and Word */
> > > + RTE_CPUFLAG_AVX512VL, /**< AVX512 Vector Length */
> > > + RTE_CPUFLAG_AVX512VBMI, /**< AVX512 Vector Bit
> > Manipulation */
> > > + RTE_CPUFLAG_AVX512VBMI2, /**< AVX512 Vector Bit
> > Manipulation 2 */
> > > + RTE_CPUFLAG_GFNI, /**< Galois Field New
> > Instructions */
> > > + RTE_CPUFLAG_VAES, /**< Vector AES */
> > > + RTE_CPUFLAG_VPCLMULQDQ, /**< Vector Carry-less Multiply
> > */
> > > + RTE_CPUFLAG_AVX512VNNI, /**< AVX512 Vector Neural
> > Network Instructions */
> > > + RTE_CPUFLAG_AVX512BITALG, /**< AVX512 Bit Algorithms */
> > > + RTE_CPUFLAG_AVX512VPOPCNTDQ, /**< AVX512 Vector Popcount */
> > > + RTE_CPUFLAG_CLDEMOTE, /**< Cache Line Demote */
> > > + RTE_CPUFLAG_MOVDIRI, /**< Direct Store Instructions
> > */
> > > + RTE_CPUFLAG_MOVDIR64B, /**< Direct Store Instructions
> > 64B */
> > > + RTE_CPUFLAG_AVX512VP2INTERSECT, /**< AVX512 Two Register
> > Intersection */
> > > +
> > > /* The last item */
> > > RTE_CPUFLAG_NUMFLAGS, /**< This should always be the
> > last! */
> >
> > This is seen as an ABI break because of the change on _NUMFLAGS:
> > https://travis-ci.com/github/ovsrobot/dpdk/jobs/302524264#L2351
>
> Correct a publicly exposed enum max value has changed - I don't believe this is an ABI break, but a backward compatible ABI change was expected with this patchset.
A change is that if an application was passing incorrect values to the
functions taking this enum as input, then now it would succeed.
I don't really see the point in doing this :-).
>
> Code compiled against eg 19.11 or 20.02 is expected to continue operating correctly.
> The new flags were only added at the end of the enum, ensuring to not change the meaning of any existing flags which would be compiled-in constants to the application binary.
>
> The actual size of the CPU flags array is a DPDK internal structure (in a .c file), and is hidden from the application, and never allocated by an application - so no possible mismatch in ABI there? Applications compiled against the older ABI will just not know about the newer flags - but suffer no breakage.
>
> @ABI compatibility folks, please review too - but to the best of my understanding this is not an ABI break, but a backwards compatible update of CPU flag lists?
>
> Thanks for flagging the CI results David!
I'd like people to look at this by themselves, not wait for Thomas,
Aaron or me to check.
When a failure is caught by the robot, a mail is sent to the submitter
afaiu (I suppose with Travis instability wrt ARM jobs, the mail might
not have been sent this time).
It is then the responsibility of the submitter to either discuss the
report on the mailing or/and waive it in devtools/libabigail.abignore.
Thanks.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2] eal/cpuflags: add x86 based cpu flags
2020-03-27 14:36 3% ` Ray Kinsella
@ 2020-03-27 15:19 3% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-03-27 15:19 UTC (permalink / raw)
To: Van Haaren, Harry, Neil Horman, Ray Kinsella, David Marchand,
Laatz, Kevin
Cc: Dodji Seketeli, dev, Richardson, Bruce, Honnappa Nagarahalli
27/03/2020 15:36, Ray Kinsella:
> On 27/03/2020 14:32, Van Haaren, Harry wrote:
> > From: Thomas Monjalon <thomas@monjalon.net>
> >> 27/03/2020 14:44, Neil Horman:
> >>> On Fri, Mar 27, 2020 at 01:24:12PM +0100, David Marchand wrote:
> >>>> On Wed, Mar 25, 2020 at 12:11 PM Kevin Laatz <kevin.laatz@intel.com>
> >> wrote:
> >>>>> --- a/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> >>>>> +++ b/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> >>>>> @@ -113,6 +113,24 @@ enum rte_cpu_flag_t {
> >>>>> /* (EAX 80000007h) EDX features */
> >>>>> RTE_CPUFLAG_INVTSC, /**< INVTSC */
> >>>>>
> >>>>> + RTE_CPUFLAG_AVX512DQ, /**< AVX512 Doubleword and
> >> Quadword */
> >>>>> + RTE_CPUFLAG_AVX512IFMA, /**< AVX512 Integer Fused
> >> Multiply-Add */
> >>>>> + RTE_CPUFLAG_AVX512CD, /**< AVX512 Conflict
> >> Detection*/
> >>>>> + RTE_CPUFLAG_AVX512BW, /**< AVX512 Byte and Word */
> >>>>> + RTE_CPUFLAG_AVX512VL, /**< AVX512 Vector Length */
> >>>>> + RTE_CPUFLAG_AVX512VBMI, /**< AVX512 Vector Bit
> >> Manipulation */
> >>>>> + RTE_CPUFLAG_AVX512VBMI2, /**< AVX512 Vector Bit
> >> Manipulation 2 */
> >>>>> + RTE_CPUFLAG_GFNI, /**< Galois Field New
> >> Instructions */
> >>>>> + RTE_CPUFLAG_VAES, /**< Vector AES */
> >>>>> + RTE_CPUFLAG_VPCLMULQDQ, /**< Vector Carry-less
> >> Multiply */
> >>>>> + RTE_CPUFLAG_AVX512VNNI, /**< AVX512 Vector Neural
> >> Network Instructions */
> >>>>> + RTE_CPUFLAG_AVX512BITALG, /**< AVX512 Bit Algorithms
> >> */
> >>>>> + RTE_CPUFLAG_AVX512VPOPCNTDQ, /**< AVX512 Vector Popcount
> >> */
> >>>>> + RTE_CPUFLAG_CLDEMOTE, /**< Cache Line Demote */
> >>>>> + RTE_CPUFLAG_MOVDIRI, /**< Direct Store
> >> Instructions */
> >>>>> + RTE_CPUFLAG_MOVDIR64B, /**< Direct Store
> >> Instructions 64B */
> >>>>> + RTE_CPUFLAG_AVX512VP2INTERSECT, /**< AVX512 Two Register
> >> Intersection */
> >>>>> +
> >>>>> /* The last item */
> >>>>> RTE_CPUFLAG_NUMFLAGS, /**< This should always be
> >> the last! */
> >>>>
> >>>> This is seen as an ABI break because of the change on _NUMFLAGS:
> >>>> https://travis-ci.com/github/ovsrobot/dpdk/jobs/302524264#L2351
> >>>>
> >>> It shouldn't be, as the only API calls we expose that use rte_cpu_flag_t
> >> accept
> >>> it as an integer parameter to see if the flag is enabled. Theres no use of
> >> the
> >>> enum in a public array or any struct that is sized based on the number of
> >> flags,
> >>> so you should be good to go
> >>
> >> Indeed I cannot imagine an ABI incompatibility in this case.
> >> The only behaviour change is to accept new (higher) RTE_CPUFLAG values
> >> in functions rte_cpu_get_flag_enabled() and rte_cpu_get_flag_name().
> >> Is changing the range of valid values an ABI break?
> >> Why is it flagged by libabigail?
> >
> > If this enum _MAX value was used by the application to allocate an array, that later our DPDK code would write to it could cause out-of-bounds array accesses of the application supplied array. Abigail doesn't know what applications could use the value for, so it flags it.
> >
> > IMO Abigail is right to flag it to us - a manual review to understand what that _MAX enum value is used for, and then decide on a case by case basis seems the best way forward to me.
> >
> > Thanks Neil/Thomas for reviewing, as reply in this thread, I also believe this is not going to break ABI.
> >
>
> +1 to Harrry's comments.
> I can't immediately see how this might break the ABI either.
So we all agree that increasing the range of valid rte_cpu_flag_t values
is OK for ABI compatibility.
This conclusion must be written as a libabigail exception in this patch.
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v3 00/33] DPDK Trace support
@ 2020-03-29 14:43 1% ` jerinj
2020-04-01 8:18 3% ` David Marchand
2020-04-03 15:36 1% ` [dpdk-dev] [PATCH v4 " jerinj
0 siblings, 2 replies; 200+ results
From: jerinj @ 2020-03-29 14:43 UTC (permalink / raw)
Cc: dev, thomas, bruce.richardson, david.marchand, mattias.ronnblom,
skori, Jerin Jacob
From: Jerin Jacob <jerinj@marvell.com>
v3:
~~
1) Fix the following build issues reported by CI
http://mails.dpdk.org/archives/test-report/2020-March/122060.html
a) clang + i686 meson build issue(Fixed in the the patch meson: add libatomic as a global dependency for i686 clang)
b) fixed build issue with FreeBSD with top of tree change.
c) fixed missing experimental API for iavf and ice with meson build in avx512 files.
v2:
~~
Addressed the following review comments from Mattias Rönnblom:
1) Changed
from:
typedef uint64_t* rte_trace_t;
to
typedef uint64_t rte_trace_t;
Initially thought to make the handle as
struct rte_trace {
uint64_t val;
}
but changed to uint64_t for the following reasons
a) It is opaque to the application and it will fix the compile-time type
check as well.
b) The handle has an index that will point to an internal slow-path
structure so no ABI change required in the future.
c) RTE_TRACE_POINT_DEFINE need to expose trace object. So it is better
to keep as uint64_t and avoid one more indirection for no use.
2)
Changed:
from:
enum rte_trace_mode_e {
to:
enum rte_trace_mode {
3) removed [out] "found" param from rte_trace_pattern() and
rte_trace_regexp()
4) Changed rte_trace_from_name to rte_trace_by_name
5) rte_trace_is_dp_enabled() return bool now
6) in __rte_trace_point_register() the argument fn change to register_fn
7) removed !! from rte_trace_is_enabled()
8) Remove uninitialized "rc warning" from rte_trace_pattern() and
rte_trace_regexp()
9) fixup bool return type for trace_entry_compare()
10) fixup calloc casting in trace_mkdir()
11) check fclose() return in trace_meta_save() and trace_mem_save()
12) rte_trace_ctf_* macro cleanup
13) added release notes
14) fix build issues reported by CI
http://mails.dpdk.org/archives/test-report/2020-March/121235.html
This patch set contains
~~~~~~~~~~~~~~~~~~~~~~~~
# The native implementation of common trace format(CTF)[1] based tracer
# Public API to create the trace points.
# Add tracepoints to eal, ethdev, mempool, eventdev and cryptodev
library for tracing support
# A unit test case
# Performance test case to measure the trace overhead. (See eal/trace:
# add trace performance test cases, patch)
# Programmers guide for Trace support(See doc: add trace library guide,
# patch)
# Tested OS:
~~~~~~~~~~~
- Linux
- FreeBSD
# Tested open source CTF trace viewers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Babeltrace
- Tracecompass
# Trace overhead comparison with LTTng
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
trace overhead data on x86:[2]
# 236 cycles with LTTng(>100ns)
# 18 cycles(7ns) with Native DPDK CTF emitter.(See eal/trace: add trace
# performance test cases patch)
trace overhead data on arm64:
# 312 cycles to 1100 cycles with LTTng based on the class of arm64
# CPU.
# 11 cycles to 13 cycles with Native DPDK CTF emitter based on the
class of arm64 CPU.
18 cycles(on x86) vs 11 cycles(on arm64) is due to rdtsc() overhead in
x86. It seems rdtsc takes around 15cycles in x86.
Items that needs to be sort it out
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Makefile and meson.build are updated to allow experimental APIs.
As multiple EXPERIMENTAL symbols, exported by trace library, are
used in various drivers, lib, app and examples. So to fix compilation
warning/error, Makefile and meson.build are updated for all required
components to support EXPERIMENTAL APIs.
It results same code changes at multiple components as well as
increases source code line changes in patchset too.
Suggestions are welcome to resolve this issue with lesser code changes.
More details:
~~~~~~~~~~~~~
# The Native DPDK CTF trace support does not have any dependency on
third-party library.
The generated output file is compatible with LTTng as both are using
CTF trace format.
The performance gain comes from:
1) exploit dpdk worker thread usage model to avoid atomics and use per
core variables
2) use hugepage,
3) avoid a lot function pointers in fast-path etc
4) avoid unaligned store for arm64 etc
Features:
~~~~~~~~~
- APIs and Features are similar to rte_log dynamic framework
API(expect log prints on stdout vs it dumps on trace file)
- No specific limit on the events. A string-based event like rte_log
for pattern matching
- Dynamic enable/disable support.
- Instructmention overhead is ~1 cycle. i.e cost of adding the code
wth out using trace feature.
- Timestamp support for all the events using DPDK rte_rtdsc
- No dependency on another library. Clean room native implementation of
CTF.
Functional test case:
a) echo "trace_autotest" | sudo ./build/app/test/dpdk-test -c 0x3
--trace-level=8
The above command emits the following trace events
<code>
uint8_t i;
rte_trace_lib_eal_generic_void();
rte_trace_lib_eal_generic_u64(0x10000000000000);
rte_trace_lib_eal_generic_u32(0x10000000);
rte_trace_lib_eal_generic_u16(0xffee);
rte_trace_lib_eal_generic_u8(0xc);
rte_trace_lib_eal_generic_i64(-1234);
rte_trace_lib_eal_generic_i32(-1234567);
rte_trace_lib_eal_generic_i16(12);
rte_trace_lib_eal_generic_i8(-3);
rte_trace_lib_eal_generic_string("my string");
rte_trace_lib_eal_generic_function(__func__);
for (i = 0; i < 128; i++)
rte_trace_lib_eal_generic_u8(i);
</code>
Install babeltrace package in Linux and point the generated trace file
to babel trace. By default trace file created under
<user>/dpdk-traces/time_stamp/
example:
# babeltrace /root/dpdk-traces/rte-2020-02-15-PM-02-56-51 | more
[13:27:36.138468807] (+?.?????????) lib.eal.generic.void: { cpu_id =
0, name = "dpdk-test" }, { }
[13:27:36.138468851] (+0.000000044) lib.eal.generic.u64: { cpu_id = 0,
name = "dpdk-test" }, { in = 4503599627370496 }
[13:27:36.138468860] (+0.000000009) lib.eal.generic.u32: { cpu_id = 0,
name = "dpdk-test" }, { in = 268435456 }
[13:27:36.138468934] (+0.000000074) lib.eal.generic.u16: { cpu_id = 0,
name = "dpdk-test" }, { in = 65518 }
[13:27:36.138468949] (+0.000000015) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 12 }
[13:27:36.138468956] (+0.000000007) lib.eal.generic.i64: { cpu_id = 0,
name = "dpdk-test" }, { in = -1234 }
[13:27:36.138468963] (+0.000000007) lib.eal.generic.i32: { cpu_id = 0,
name = "dpdk-test" }, { in = -1234567 }
[13:27:36.138469024] (+0.000000061) lib.eal.generic.i16: { cpu_id = 0,
name = "dpdk-test" }, { in = 12 }
[13:27:36.138469044] (+0.000000020) lib.eal.generic.i8: { cpu_id = 0,
name = "dpdk-test" }, { in = -3 }
[13:27:36.138469051] (+0.000000007) lib.eal.generic.string: { cpu_id =
0, name = "dpdk-test" }, { str = "my string" }
[13:27:36.138469203] (+0.000000152) lib.eal.generic.func: { cpu_id =
0, name = "dpdk-test" }, { func = "test_trace_points" }
[13:27:36.138469239] (+0.000000036) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 0 }
[13:27:36.138469246] (+0.000000007) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 1 }
[13:27:36.138469252] (+0.000000006) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 2 }
[13:27:36.138469262] (+0.000000010) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 3 }
[13:27:36.138469269] (+0.000000007) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 4 }
[13:27:36.138469276] (+0.000000007) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 5 }
# There is a GUI based trace viewer available in Windows, Linux and
# Mac.
It is called as tracecompass.(https://www.eclipse.org/tracecompass/)
The example screenshot and Histogram of above DPDK trace using
Tracecompass.
https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG
File walk through:
~~~~~~~~~~~~~~~~~~
lib/librte_eal/common/include/rte_trace.h - Public API for Trace
provider and Trace control
lib/librte_eal/common/eal_common_trace.c - main trace implementation
lib/librte_eal/common/eal_common_trace_ctf.c - CTF metadata spec
implementation
lib/librte_eal/common/eal_common_trace_utils.c - command line utils
and filesystem operations.
lib/librte_eal/common/eal_common_trace_points.c - trace points for EAL
library
lib/librte_eal/common/include/rte_trace_eal.h - EAL tracepoint public
API.
lib/librte_eal/common/eal_trace.h - Private trace header file.
[1] https://diamon.org/ctf/
[2] The above test is ported to LTTng for finding the LTTng trace
overhead. It available at
https://github.com/jerinjacobk/lttng-overhead
https://github.com/jerinjacobk/lttng-overhead/blob/master/README
Jerin Jacob (21):
eal: introduce API for getting thread name
eal: define the public API for trace support
eal/trace: implement trace register API
eal/trace: implement trace operation APIs
eal/trace: add internal trace init and fini interface
eal/trace: get bootup timestamp for trace
eal/trace: create CTF TDSL metadata in memory
eal/trace: implement trace memory allocation
eal/trace: implement debug dump function
eal/trace: implement trace save
eal/trace: implement registration payload
eal/trace: implement provider payload
eal/trace: hook internal trace APIs to Linux
eal/trace: hook internal trace APIs to FreeBSD
eal/trace: add generic tracepoints
eal/trace: add alarm tracepoints
eal/trace: add memory tracepoints
eal/trace: add memzone tracepoints
eal/trace: add thread tracepoints
eal/trace: add interrupt tracepoints
eal/trace: add trace performance test cases
Pavan Nikhilesh (1):
meson: add libatomic as a global dependency for i686 clang
Sunil Kumar Kori (11):
eal/trace: handle CTF keyword collision
eal/trace: add trace level configuration parameter
eal/trace: add trace dir configuration parameter
eal/trace: add trace bufsize configuration parameter
eal/trace: add trace mode configuration parameter
eal/trace: add unit test cases
ethdev: add tracepoints
eventdev: add tracepoints
cryptodev: add tracepoints
mempool: add tracepoints
doc: add trace library guide
MAINTAINERS | 15 +
app/pdump/Makefile | 1 +
app/pdump/meson.build | 1 +
app/proc-info/Makefile | 1 +
app/proc-info/meson.build | 1 +
app/test-acl/Makefile | 1 +
app/test-acl/meson.build | 1 +
app/test-cmdline/Makefile | 1 +
app/test-cmdline/meson.build | 1 +
app/test-eventdev/Makefile | 1 +
app/test-eventdev/meson.build | 1 +
app/test-pipeline/Makefile | 1 +
app/test-pipeline/meson.build | 1 +
app/test/Makefile | 4 +-
app/test/meson.build | 3 +
app/test/test_trace.c | 618 ++++++++++++++++++
app/test/test_trace.h | 52 ++
app/test/test_trace_perf.c | 179 +++++
app/test/test_trace_register.c | 46 ++
config/common_base | 1 +
config/meson.build | 10 +
config/rte_config.h | 1 +
doc/api/doxy-api-index.md | 3 +-
doc/guides/linux_gsg/eal_args.include.rst | 55 ++
doc/guides/prog_guide/index.rst | 1 +
doc/guides/prog_guide/trace_lib.rst | 265 ++++++++
doc/guides/rel_notes/release_20_05.rst | 9 +
drivers/common/cpt/Makefile | 1 +
drivers/crypto/ccp/Makefile | 1 +
drivers/crypto/ccp/meson.build | 2 +
drivers/crypto/mvsam/Makefile | 1 +
drivers/crypto/mvsam/meson.build | 1 +
drivers/crypto/null/Makefile | 1 +
drivers/crypto/null/meson.build | 1 +
drivers/crypto/scheduler/Makefile | 1 +
drivers/crypto/scheduler/meson.build | 1 +
drivers/crypto/virtio/Makefile | 1 +
drivers/crypto/virtio/meson.build | 1 +
drivers/event/octeontx/Makefile | 1 +
drivers/event/octeontx/meson.build | 6 +-
drivers/event/octeontx2/meson.build | 5 -
drivers/event/opdl/meson.build | 5 -
drivers/event/skeleton/Makefile | 1 +
drivers/event/skeleton/meson.build | 1 +
drivers/event/sw/Makefile | 1 +
drivers/event/sw/meson.build | 1 +
drivers/mempool/ring/Makefile | 1 +
drivers/mempool/ring/meson.build | 1 +
drivers/net/af_packet/Makefile | 1 +
drivers/net/af_packet/meson.build | 1 +
drivers/net/af_xdp/Makefile | 1 +
drivers/net/af_xdp/meson.build | 1 +
drivers/net/ark/Makefile | 1 +
drivers/net/ark/meson.build | 1 +
drivers/net/bnxt/Makefile | 1 +
drivers/net/bnxt/meson.build | 1 +
drivers/net/cxgbe/Makefile | 1 +
drivers/net/cxgbe/meson.build | 1 +
drivers/net/enetc/Makefile | 1 +
drivers/net/enetc/meson.build | 1 +
drivers/net/hinic/Makefile | 1 +
drivers/net/hinic/base/meson.build | 3 +-
drivers/net/hinic/meson.build | 1 +
drivers/net/iavf/meson.build | 3 +
drivers/net/ice/meson.build | 3 +
drivers/net/ionic/ionic_dev.c | 1 +
drivers/net/ionic/ionic_mac_api.c | 1 +
drivers/net/ionic/ionic_main.c | 1 +
drivers/net/kni/Makefile | 1 +
drivers/net/kni/meson.build | 1 +
drivers/net/liquidio/Makefile | 1 +
drivers/net/liquidio/meson.build | 1 +
drivers/net/mvneta/Makefile | 1 +
drivers/net/mvneta/meson.build | 1 +
drivers/net/mvpp2/Makefile | 1 +
drivers/net/mvpp2/meson.build | 1 +
drivers/net/nfb/Makefile | 1 +
drivers/net/nfb/meson.build | 1 +
drivers/net/null/Makefile | 1 +
drivers/net/null/meson.build | 1 +
drivers/net/octeontx/Makefile | 1 +
drivers/net/octeontx2/Makefile | 1 +
drivers/net/octeontx2/meson.build | 1 +
drivers/net/pcap/Makefile | 1 +
drivers/net/pcap/meson.build | 1 +
drivers/net/ring/Makefile | 1 +
drivers/net/ring/meson.build | 1 +
drivers/net/szedata2/Makefile | 1 +
drivers/net/szedata2/meson.build | 1 +
drivers/net/thunderx/base/meson.build | 1 +
drivers/net/vhost/Makefile | 1 +
drivers/net/vhost/meson.build | 1 +
drivers/raw/ioat/Makefile | 1 +
drivers/raw/ioat/meson.build | 1 +
drivers/raw/octeontx2_dma/Makefile | 1 +
drivers/raw/octeontx2_dma/meson.build | 1 +
drivers/raw/octeontx2_ep/Makefile | 1 +
drivers/raw/octeontx2_ep/meson.build | 1 +
drivers/raw/skeleton/Makefile | 1 +
drivers/raw/skeleton/meson.build | 1 +
examples/cmdline/Makefile | 1 +
examples/cmdline/meson.build | 1 +
examples/distributor/Makefile | 1 +
examples/distributor/meson.build | 1 +
examples/ethtool/ethtool-app/Makefile | 1 +
examples/eventdev_pipeline/meson.build | 1 +
examples/flow_filtering/Makefile | 1 +
examples/flow_filtering/meson.build | 1 +
examples/helloworld/Makefile | 1 +
examples/helloworld/meson.build | 1 +
examples/ioat/Makefile | 1 +
examples/ioat/meson.build | 1 +
examples/ip_fragmentation/Makefile | 2 +
examples/ip_fragmentation/meson.build | 1 +
examples/ip_reassembly/Makefile | 1 +
examples/ip_reassembly/meson.build | 1 +
examples/ipv4_multicast/Makefile | 1 +
examples/ipv4_multicast/meson.build | 1 +
examples/l2fwd-cat/Makefile | 1 +
examples/l2fwd-cat/meson.build | 1 +
examples/l2fwd-event/Makefile | 1 +
examples/l2fwd-event/meson.build | 6 +-
examples/l2fwd-jobstats/Makefile | 1 +
examples/l2fwd-jobstats/meson.build | 1 +
examples/l2fwd-keepalive/Makefile | 1 +
examples/l2fwd-keepalive/ka-agent/Makefile | 1 +
examples/l2fwd-keepalive/meson.build | 1 +
examples/l3fwd-acl/Makefile | 1 +
examples/l3fwd-acl/meson.build | 1 +
examples/l3fwd/Makefile | 1 +
examples/l3fwd/meson.build | 1 +
examples/link_status_interrupt/Makefile | 1 +
examples/link_status_interrupt/meson.build | 1 +
.../client_server_mp/mp_client/Makefile | 1 +
.../client_server_mp/mp_client/meson.build | 1 +
.../client_server_mp/mp_server/meson.build | 1 +
examples/multi_process/hotplug_mp/Makefile | 1 +
examples/multi_process/hotplug_mp/meson.build | 1 +
examples/multi_process/simple_mp/Makefile | 1 +
examples/multi_process/simple_mp/meson.build | 1 +
examples/multi_process/symmetric_mp/Makefile | 1 +
.../multi_process/symmetric_mp/meson.build | 1 +
examples/ntb/Makefile | 1 +
examples/ntb/meson.build | 1 +
examples/packet_ordering/Makefile | 1 +
examples/packet_ordering/meson.build | 1 +
.../performance-thread/l3fwd-thread/Makefile | 1 +
.../l3fwd-thread/meson.build | 1 +
.../performance-thread/pthread_shim/Makefile | 1 +
.../pthread_shim/meson.build | 1 +
examples/ptpclient/Makefile | 1 +
examples/ptpclient/meson.build | 1 +
examples/qos_meter/Makefile | 1 +
examples/qos_meter/meson.build | 1 +
examples/qos_sched/Makefile | 1 +
examples/qos_sched/meson.build | 1 +
examples/server_node_efd/node/Makefile | 1 +
examples/server_node_efd/node/meson.build | 1 +
examples/server_node_efd/server/Makefile | 1 +
examples/server_node_efd/server/meson.build | 1 +
examples/service_cores/Makefile | 1 +
examples/service_cores/meson.build | 1 +
examples/skeleton/Makefile | 1 +
examples/skeleton/meson.build | 1 +
examples/timer/Makefile | 1 +
examples/timer/meson.build | 1 +
examples/vm_power_manager/Makefile | 1 +
examples/vm_power_manager/meson.build | 1 +
examples/vmdq/Makefile | 1 +
examples/vmdq/meson.build | 1 +
examples/vmdq_dcb/Makefile | 1 +
examples/vmdq_dcb/meson.build | 1 +
lib/librte_bitratestats/Makefile | 1 +
lib/librte_bitratestats/meson.build | 1 +
lib/librte_cryptodev/Makefile | 4 +-
lib/librte_cryptodev/cryptodev_trace_points.c | 70 ++
lib/librte_cryptodev/meson.build | 7 +-
lib/librte_cryptodev/rte_cryptodev.c | 18 +
lib/librte_cryptodev/rte_cryptodev.h | 6 +
.../rte_cryptodev_version.map | 18 +
lib/librte_cryptodev/rte_trace_cryptodev.h | 133 ++++
lib/librte_cryptodev/rte_trace_cryptodev_fp.h | 34 +
lib/librte_distributor/Makefile | 1 +
lib/librte_distributor/meson.build | 6 +-
lib/librte_eal/common/Makefile | 1 +
lib/librte_eal/common/eal_common_log.c | 9 +-
lib/librte_eal/common/eal_common_memzone.c | 9 +
lib/librte_eal/common/eal_common_options.c | 68 +-
lib/librte_eal/common/eal_common_thread.c | 3 +-
lib/librte_eal/common/eal_common_trace.c | 610 +++++++++++++++++
lib/librte_eal/common/eal_common_trace_ctf.c | 488 ++++++++++++++
.../common/eal_common_trace_points.c | 115 ++++
.../common/eal_common_trace_utils.c | 523 +++++++++++++++
lib/librte_eal/common/eal_options.h | 8 +
lib/librte_eal/common/eal_private.h | 11 +
lib/librte_eal/common/eal_trace.h | 122 ++++
lib/librte_eal/common/include/rte_lcore.h | 17 +
lib/librte_eal/common/include/rte_trace.h | 584 +++++++++++++++++
lib/librte_eal/common/include/rte_trace_eal.h | 247 +++++++
.../common/include/rte_trace_provider.h | 137 ++++
.../common/include/rte_trace_register.h | 53 ++
lib/librte_eal/common/meson.build | 8 +
lib/librte_eal/common/rte_malloc.c | 60 +-
lib/librte_eal/freebsd/eal/Makefile | 4 +
lib/librte_eal/freebsd/eal/eal.c | 10 +
lib/librte_eal/freebsd/eal/eal_alarm.c | 3 +
lib/librte_eal/freebsd/eal/eal_interrupts.c | 54 +-
lib/librte_eal/freebsd/eal/eal_thread.c | 21 +-
lib/librte_eal/linux/eal/Makefile | 4 +
lib/librte_eal/linux/eal/eal.c | 9 +
lib/librte_eal/linux/eal/eal_alarm.c | 4 +
lib/librte_eal/linux/eal/eal_interrupts.c | 84 ++-
lib/librte_eal/linux/eal/eal_thread.c | 27 +-
lib/librte_eal/rte_eal_version.map | 59 ++
lib/librte_ethdev/Makefile | 3 +
lib/librte_ethdev/ethdev_trace_points.c | 43 ++
lib/librte_ethdev/meson.build | 5 +-
lib/librte_ethdev/rte_ethdev.c | 12 +
lib/librte_ethdev/rte_ethdev.h | 5 +
lib/librte_ethdev/rte_ethdev_version.map | 10 +
lib/librte_ethdev/rte_trace_ethdev.h | 90 +++
lib/librte_ethdev/rte_trace_ethdev_fp.h | 40 ++
lib/librte_eventdev/Makefile | 3 +
lib/librte_eventdev/eventdev_trace_points.c | 173 +++++
lib/librte_eventdev/meson.build | 3 +
.../rte_event_crypto_adapter.c | 10 +
.../rte_event_eth_rx_adapter.c | 11 +
.../rte_event_eth_tx_adapter.c | 13 +-
.../rte_event_eth_tx_adapter.h | 2 +
lib/librte_eventdev/rte_event_timer_adapter.c | 8 +-
lib/librte_eventdev/rte_event_timer_adapter.h | 8 +
lib/librte_eventdev/rte_eventdev.c | 9 +
lib/librte_eventdev/rte_eventdev.h | 5 +-
lib/librte_eventdev/rte_eventdev_version.map | 42 ++
lib/librte_eventdev/rte_trace_eventdev.h | 278 ++++++++
lib/librte_eventdev/rte_trace_eventdev_fp.h | 75 +++
lib/librte_gro/Makefile | 1 +
lib/librte_gro/meson.build | 1 +
lib/librte_gso/Makefile | 1 +
lib/librte_gso/meson.build | 1 +
lib/librte_ip_frag/Makefile | 1 +
lib/librte_ip_frag/meson.build | 1 +
lib/librte_kni/Makefile | 1 +
lib/librte_kni/meson.build | 1 +
lib/librte_latencystats/Makefile | 1 +
lib/librte_latencystats/meson.build | 1 +
lib/librte_mempool/Makefile | 3 +
lib/librte_mempool/mempool_trace_points.c | 108 +++
lib/librte_mempool/meson.build | 7 +-
lib/librte_mempool/rte_mempool.c | 16 +
lib/librte_mempool/rte_mempool.h | 13 +
lib/librte_mempool/rte_mempool_ops.c | 7 +
lib/librte_mempool/rte_mempool_version.map | 26 +
lib/librte_mempool/rte_trace_mempool.h | 148 +++++
lib/librte_mempool/rte_trace_mempool_fp.h | 102 +++
lib/librte_port/Makefile | 1 +
lib/librte_port/meson.build | 1 +
lib/librte_rcu/meson.build | 5 -
lib/librte_reorder/Makefile | 1 +
lib/librte_reorder/meson.build | 1 +
lib/librte_sched/Makefile | 1 +
lib/librte_sched/meson.build | 1 +
lib/librte_security/Makefile | 1 +
lib/librte_security/meson.build | 1 +
lib/librte_table/Makefile | 1 +
lib/librte_table/meson.build | 1 +
266 files changed, 6286 insertions(+), 113 deletions(-)
create mode 100644 app/test/test_trace.c
create mode 100644 app/test/test_trace.h
create mode 100644 app/test/test_trace_perf.c
create mode 100644 app/test/test_trace_register.c
create mode 100644 doc/guides/prog_guide/trace_lib.rst
create mode 100644 lib/librte_cryptodev/cryptodev_trace_points.c
create mode 100644 lib/librte_cryptodev/rte_trace_cryptodev.h
create mode 100644 lib/librte_cryptodev/rte_trace_cryptodev_fp.h
create mode 100644 lib/librte_eal/common/eal_common_trace.c
create mode 100644 lib/librte_eal/common/eal_common_trace_ctf.c
create mode 100644 lib/librte_eal/common/eal_common_trace_points.c
create mode 100644 lib/librte_eal/common/eal_common_trace_utils.c
create mode 100644 lib/librte_eal/common/eal_trace.h
create mode 100644 lib/librte_eal/common/include/rte_trace.h
create mode 100644 lib/librte_eal/common/include/rte_trace_eal.h
create mode 100644 lib/librte_eal/common/include/rte_trace_provider.h
create mode 100644 lib/librte_eal/common/include/rte_trace_register.h
create mode 100644 lib/librte_ethdev/ethdev_trace_points.c
create mode 100644 lib/librte_ethdev/rte_trace_ethdev.h
create mode 100644 lib/librte_ethdev/rte_trace_ethdev_fp.h
create mode 100644 lib/librte_eventdev/eventdev_trace_points.c
create mode 100644 lib/librte_eventdev/rte_trace_eventdev.h
create mode 100644 lib/librte_eventdev/rte_trace_eventdev_fp.h
create mode 100644 lib/librte_mempool/mempool_trace_points.c
create mode 100644 lib/librte_mempool/rte_trace_mempool.h
create mode 100644 lib/librte_mempool/rte_trace_mempool_fp.h
--
2.25.1
^ permalink raw reply [relevance 1%]
* [dpdk-dev] [RFC PATCH 7/9] eal/windows: fix rte_page_sizes with Clang on Windows
@ 2020-03-30 4:10 3% ` Dmitry Kozlyuk
1 sibling, 0 replies; 200+ results
From: Dmitry Kozlyuk @ 2020-03-30 4:10 UTC (permalink / raw)
To: dev; +Cc: Dmitry Malloy (MESHCHANINOV), Dmitry Kozlyuk, Anatoly Burakov
Clang on Windows follows MS ABI where enum values are signed 32-bit,
Enum rte_page_size has members valued beyond 2^32. EAL cannot use
-fno-ms-compatibility because its code is OS-dependent. The only option
is to define these values outside enum, but this prohibits using
-fstrict-enums. Another consequence is that enum rte_page_size cannot
be used to hold page size, because on Windows it won't be able to hold
all possible values.
Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
lib/librte_eal/common/include/rte_memory.h | 6 ++++++
lib/librte_eal/meson.build | 1 +
2 files changed, 7 insertions(+)
diff --git a/lib/librte_eal/common/include/rte_memory.h b/lib/librte_eal/common/include/rte_memory.h
index 1742fde9a..5b0e2d8b5 100644
--- a/lib/librte_eal/common/include/rte_memory.h
+++ b/lib/librte_eal/common/include/rte_memory.h
@@ -34,8 +34,14 @@ enum rte_page_sizes {
RTE_PGSIZE_256M = 1ULL << 28,
RTE_PGSIZE_512M = 1ULL << 29,
RTE_PGSIZE_1G = 1ULL << 30,
+/* Work around Clang on Windows being limited to 32-bit underlying type. */
+#if !defined(RTE_TOOLCHAIN_CLANG) || !defined(RTE_EXEC_ENV_WINDOWS)
RTE_PGSIZE_4G = 1ULL << 32,
RTE_PGSIZE_16G = 1ULL << 34,
+#else
+#define RTE_PGSIZE_4G (1ULL << 32)
+#define RTE_PGSIZE_16G (1ULL << 34)
+#endif
};
#define SOCKET_ID_ANY -1 /**< Any NUMA socket. */
diff --git a/lib/librte_eal/meson.build b/lib/librte_eal/meson.build
index 1730d603f..ec80bd6be 100644
--- a/lib/librte_eal/meson.build
+++ b/lib/librte_eal/meson.build
@@ -23,6 +23,7 @@ endif
if cc.has_header('getopt.h')
cflags += ['-DHAVE_GETOPT_H', '-DHAVE_GETOPT', '-DHAVE_GETOPT_LONG']
endif
+cflags += '-fno-strict-enums'
sources = common_sources + env_sources
objs = common_objs + env_objs
headers = common_headers + env_headers
--
2.25.1
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2 0/4] Introduce IF proxy library
@ 2020-03-30 19:23 3% ` Andrzej Ostruszka
0 siblings, 0 replies; 200+ results
From: Andrzej Ostruszka @ 2020-03-30 19:23 UTC (permalink / raw)
To: David Marchand, Andrzej Ostruszka; +Cc: dev
On 3/26/20 1:41 PM, Andrzej Ostruszka wrote:
> Thank you David for taking time to look at this.
>
> On 3/25/20 9:08 AM, David Marchand wrote:
>> Hello Andrzej,
>>
>> On Tue, Mar 10, 2020 at 12:11 PM Andrzej Ostruszka
> [...]
>> I can see we end up exposing structures for registering callbacks.
>
> Right. I was thinking more in terms of user convenience so it seemed
> like a good choice to gather them in one struct and call 'register'
> once. The fact that the same structure is used to keep them is an
> implementation choice and this can be decoupled.
>
>> Did you consider some ways to avoid exposure of those? (thinking of
>> ABI maintenance for when this library will elect to non-experimental).
>
> I will. So far I used the union for the input since I like when things
> are well typed :) and there is no need for casting. However I will
> spend some time on this and will get back to you soon (if you have
> already something in your head please share). Right now I'm thinking
> about taking array of callbacks with each entry being ("event type",
> callback) pair, however need to figure out how to have minimum amount of
> type casting.
David, I thought about this a bit and here is my proposal.
Define "typeful" callback pointer (public):
union rte_ifpx_cb_ptr {
int (*mac_change)(const struct mac_change *ev);
int (*mtu_change)(const struct mtu_change *ev);
...
int (*cfg_done)(void);
};
In implementation make sure its size is as expected:
_Static_assert(sizeof(union rte_ifpx_cb_ptr) == sizeof (int(*)(void*)),
"Size of callback pointer has to be"
"equal to size of function pointer");
Accept as input tagged callbacks (also public type):
struct rte_ifpx_callback {
enum rte_ifpx_event_type type;
union rte_ifpx_cb_ptr callback;
};
The user would be defining array of callbacks:
struct rte_ifpx_callback callbacks[] = {
{RTE_IFPX_MAC_CHANGE, {.mac_change = mac_change}},
{RTE_IFPX_MTU_CHANGE, {.mtu_change = mtu_change}},
...
{RTE_IFPX_CFG_DONE, {.cfg_done = finished}},
};
and passing it to registration together with its length like:
int rte_ifpx_callbacks_register(int len,
const struct rte_ifpx_callback *cbs)
{
for (int i = 0; i < len; ++i) {
switch (cbs[i].type) {
case RTE_IFPX_MAC_CHANGE:
priv_cbs.mac_change = cbs[i].callback.mac_change;
break;
...
}
This way we should be protected from ABI breakage when adding new event
types and how the callbacks are stored would not be visible to the user.
Let me know what do you think about it.
With regards
Andrzej Ostruszka
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
@ 2020-03-30 21:29 0% ` Honnappa Nagarahalli
2020-03-30 23:37 0% ` Honnappa Nagarahalli
0 siblings, 1 reply; 200+ results
From: Honnappa Nagarahalli @ 2020-03-30 21:29 UTC (permalink / raw)
To: Ananyev, Konstantin, dev; +Cc: olivier.matz, nd, Honnappa Nagarahalli, nd
<snip>
> >
> > > Subject: [dpdk-dev] [RFC 0/6] New sync modes for ring
> > >
> > > Upfront note - that RFC is not a complete patch.
> > > It introduces an ABI breakage, plus it doesn't update ring_elem code
> > > properly,
> > As per the current rules, these changes (in the current form) will be
> > accepted only for 20.11 release. How do we address this for immediate
> requirements like RCU defer APIs?
>
> I think I found a way to introduce these new modes without API/ABI breakage.
> Working on v1 right now. Plan to submit it by end of that week/start of next
> one.
ok
>
> > I suggest that we move forward with my RFC (taking into consideration your
> feedback) to make progress on RCU APIs.
> >
> > > etc.
> > > I plan to deal with all these things in later versions.
> > > Right now I seek an initial feedback about proposed ideas.
> > > Would also ask people to repeat performance tests (see below) on
> > > their platforms to confirm the impact.
> > >
> > > More and more customers use(/try to use) DPDK based apps within
> > > overcommitted systems (multiple acttive threads over same pysical cores):
> > > VM, container deployments, etc.
> > > One quite common problem they hit: Lock-Holder-Preemption with
> rte_ring.
> > > LHP is quite a common problem for spin-based sync primitives
> > > (spin-locks, etc.) on overcommitted systems.
> > > The situation gets much worse when some sort of fair-locking
> > > technique is used (ticket-lock, etc.).
> > > As now not only lock-owner but also lock-waiters scheduling order
> > > matters a lot.
> > > This is a well-known problem for kernel within VMs:
> > > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > > The problem with rte_ring is that while head accusion is sort of
> > > un-fair locking, waiting on tail is very similar to ticket lock
> > > schema - tail has to be updated in particular order.
> > > That makes current rte_ring implementation to perform really pure on
> > > some overcommited scenarios.
> > > While it is probably not possible to completely resolve this problem
> > > in userspace only (without some kernel communication/intervention),
> > > removing fairness in tail update can mitigate it significantly.
> > > So this RFC proposes two new optional ring synchronization modes:
> > > 1) Head/Tail Sync (HTS) mode
> > > In that mode enqueue/dequeue operation is fully serialized:
> > > only one thread at a time is allowed to perform given op.
> > > As another enhancement provide ability to split enqueue/dequeue
> > > operation into two phases:
> > > - enqueue/dequeue start
> > > - enqueue/dequeue finish
> > > That allows user to inspect objects in the ring without removing
> > > them from it (aka MT safe peek).
> > IMO, this will not address the problem described above.
>
> It does, please see the results produced by ring_stress_*autotest below.
> Let say for test-case: 8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' it
Had not looked at these tests. Please see the numbers below.
> shows:
> avg number of cycles per object for enqueue /dequeue:
> MP/MC: 280314.32
> HTS: 294.72
> RTS: 318.79
>
> Customer who tried it reported similar level of improvement.
Is this tested with the VM/Container setup described in the slides you referred to?
> Actually if you have time - would be very interesting to see what numbers will
> be on ARM boxes.
> To reproduce, just:
> $cat ring_tests_u4
> ring_stress_autotest
> ring_stress_hts_autotest
> ring_stress_rts_autotest
>
> /app/test/dpdk-test --lcores='6,(10-13)@7,(20-23)@8' -n 4 < ring_tests_u4
> 2>&1 | tee res1
>
> Then look at the ' AGGREGATE' stats.
> Right now it is a bit too verbose, so probably the easiest thing to extract same
> numbers quickly:
> grep 'cycles/obj' res1 | grep 'cycles/obj' | cat -n | awk '{if ($(1)%9==0) print
> $(NF);}'
> 280314.32
> 1057833.55
> 294.72
> 480.10
> 318.79
> 461.52
>
> First 2 numbers will be for MP/MC, next 2 for HTS, last 2 for RTS.
12305.05
12027.09
3.59
7.37
4.41
7.98
>
> > For ex: when a producer updates the head and gets scheduled out, other
> > producers have to spin.
>
> Sure, as I wrote in original cover letter:
> " While it is probably not possible to completely resolve this problem in
> userspace only (without some kernel communication/intervention), removing
> fairness in tail update can mitigate it significantly."
> Results from the ring_stress_*_autotest confirm that.
>
> > The problem is probably worse as with non-HTS case moving of the head
> > and copying of the ring elements can happen in parallel between the
> producers (similarly for consumers).
>
> Yes as we serialize the ring, we remove possibility of simultaneous copy.
> That's why for 'normal' cases (one thread per core) original MP/MC is usually
> faster.
> Though on overcommitted scenarios current MP/MC performance degrades
> dramatically.
> The main problem with current MP/MC implementation is in that tail update
> have to be done in strict order (sort of fair locking scheme).
> Which means that we have much-much worse LHP manifestation, then when
> we use unfair schemes.
> With serialized ring (HTS) we remove that ordering completely (same idea as
> switch from fair to unfair locking for PV spin-locks).
>
> > IMO, HTS should not be a configurable flag.
>
> Why?
>
> > In RCU requirement, a MP enqueue and HTS dequeue are required.
>
> This is supported, user can specify different modes for consumer and
> producer:
> (0 | RING_F_MC_HTS_DEQ).
> Then it is up to the user either to call generic
> rte_ring_enqueue/rte_ring_dequeue,
> or specify mode manually by function name:
> rte_ring_mp_enqueue_bulk/ rte_ring_hts_dequeue_bulk.
Ok, that should be good.
>
> >
> > > 2) Relaxed Tail Sync (RTS)
> > > The main difference from original MP/MC algorithm is that tail value
> > > is increased not by every thread that finished enqueue/dequeue, but
> > > only by the last one.
> > > That allows threads to avoid spinning on ring tail value, leaving
> > > actual tail value change to the last thread in the update queue.
> > This can be a configurable flag on the ring.
> > I am not sure how this solves the problem you have stated above
> > completely. Updating the count from all intermediate threads is still
> > required to update the value of the head. But yes, it reduces the severity of
> the problem by not enforcing the order in which the tail is updated.
>
> As I said above, main source of slowdown here - that we have to update tail in
> particular order.
> So the main objective (same as for HTS) is to remove that ordering.
>
> > I also think it introduces the problem on the other side of the ring
> > because the tail is not updated soon enough (the other side has to wait
> longer for the elements to become available).
>
> Yes, producer/consumer starvation.
> That's why we need max allowed Head-Tail-Distance (htd_max) - to limit how
> far head can go away from tail.
>
> > It also introduces another configuration parameter (HTD_MAX_DEF) which
> > they have to deal with.
>
> If user doesn't provide any value, it will be set by default to ring.capacity / 8.
> From my measurements works quite well.
> Though there possibility for the user to set another value, if needed.
>
> > Users have to still implement the current hypervisor related solutions.
>
> Didn't get what you trying to say with that phrase.
The references you provided talked about resolving LHP by doing co-scheduling of vCPUs (which I think could be applied to DPDK applications). I am saying that we still need such mechanisms along with these solutions.
>
> > IMO, we should run the benchmark for this on an over committed setup to
> understand the benefits.
>
> That's why I created ring_stress_*autotest test-cases and collected numbers
> provided below.
> I suppose they clearly show the problem on overcommitted scenarios, and
> how RTS/HTS improve that situation.
> Would appreciate if you repeat these tests on your machines.
>
> >
> > >
> > > Test results on IA (see below) show significant improvements for
> > > average enqueue/dequeue op times on overcommitted systems.
> > > For 'classic' DPDK deployments (one thread per core) original MP/MC
> > > algorithm still shows best numbers, though for 64-bit target RTS
> > > numbers are not that far away.
> > > Numbers were produced by ring_stress_*autotest (first patch in these
> series).
> > >
> > > X86_64 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
> > > DEQ+ENQ average cycles/obj
> > >
> > > MP/MC HTS RTS
> > > 1thread@1core(--lcores=6-7) 8.00 8.15 8.99
> > > 2thread@2core(--lcores=6-8) 19.14 19.61 20.35
> > > 4thread@4core(--lcores=6-10) 29.43 29.79 31.82
> > > 8thread@8core(--lcores=6-14) 110.59 192.81 119.50
> > > 16thread@16core(--lcores=6-22) 461.03 813.12 495.59
> > > 32thread/@32core(--lcores='6-22,55-70') 982.90 1972.38 1160.51
> > >
> > > 2thread@1core(--lcores='6,(10-11)@7' 20140.50 23.58 25.14
> > > 4thread@2core(--lcores='6,(10-11)@7,(20-21)@8' 153680.60 76.88
> 80.05
> > > 8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 280314.32 294.72
> > > 318.79 16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 643176.59
> > > 1144.02
> > > 1175.14 32thread@2core(--lcores='6,(10-25)@7,(30-45)@8' 4264238.80
> > > 4627.48 4892.68
> > >
> > > 8thread@2core(--lcores='6,(10-17)@(7,8))' 321085.98 298.59 307.47
> > > 16thread@4core(--lcores='6,(20-35)@(7-10))' 1900705.61 575.35
> 678.29
> > > 32thread@4core(--lcores='6,(20-51)@(7-10))' 5510445.85 2164.36
> 2714.12
> > >
> > > i686 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
> > > DEQ+ENQ average cycles/obj
> > >
> > > MP/MC HTS RTS
> > > 1thread@1core(--lcores=6-7) 7.85 12.13 11.31
> > > 2thread@2core(--lcores=6-8) 17.89 24.52 21.86
> > > 8thread@8core(--lcores=6-14) 32.58 354.20 54.58
> > > 32thread/@32core(--lcores='6-22,55-70') 813.77 6072.41 2169.91
> > >
> > > 2thread@1core(--lcores='6,(10-11)@7' 16095.00 36.06 34.74
> > > 8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 1140354.54 346.61
> > > 361.57 16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 1920417.86
> > > 1314.90
> > > 1416.65
> > >
> > > 8thread@2core(--lcores='6,(10-17)@(7,8))' 594358.61 332.70 357.74
> > > 32thread@4core(--lcores='6,(20-51)@(7-10))' 5319896.86 2836.44
> 3028.87
> > >
> > > Konstantin Ananyev (6):
> > > test/ring: add contention stress test
> > > ring: rework ring layout to allow new sync schemes
> > > ring: introduce RTS ring mode
> > > test/ring: add contention stress test for RTS ring
> > > ring: introduce HTS ring mode
> > > test/ring: add contention stress test for HTS ring
> > >
> > > app/test/Makefile | 3 +
> > > app/test/meson.build | 3 +
> > > app/test/test_pdump.c | 6 +-
> > > app/test/test_ring_hts_stress.c | 28 ++
> > > app/test/test_ring_rts_stress.c | 28 ++
> > > app/test/test_ring_stress.c | 27 ++
> > > app/test/test_ring_stress.h | 477 +++++++++++++++++++
> > > lib/librte_pdump/rte_pdump.c | 2 +-
> > > lib/librte_port/rte_port_ring.c | 12 +-
> > > lib/librte_ring/Makefile | 4 +-
> > > lib/librte_ring/meson.build | 4 +-
> > > lib/librte_ring/rte_ring.c | 84 +++-
> > > lib/librte_ring/rte_ring.h | 619 +++++++++++++++++++++++--
> > > lib/librte_ring/rte_ring_elem.h | 8 +-
> > > lib/librte_ring/rte_ring_hts_generic.h | 228 +++++++++
> > > lib/librte_ring/rte_ring_rts_generic.h | 240 ++++++++++
> > > 16 files changed, 1721 insertions(+), 52 deletions(-) create mode
> > > 100644 app/test/test_ring_hts_stress.c create mode 100644
> > > app/test/test_ring_rts_stress.c create mode 100644
> > > app/test/test_ring_stress.c create mode 100644
> > > app/test/test_ring_stress.h create mode 100644
> > > lib/librte_ring/rte_ring_hts_generic.h
> > > create mode 100644 lib/librte_ring/rte_ring_rts_generic.h
> > >
> > > --
> > > 2.17.1
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-03-30 21:29 0% ` Honnappa Nagarahalli
@ 2020-03-30 23:37 0% ` Honnappa Nagarahalli
2020-03-31 17:21 0% ` Ananyev, Konstantin
0 siblings, 1 reply; 200+ results
From: Honnappa Nagarahalli @ 2020-03-30 23:37 UTC (permalink / raw)
To: Ananyev, Konstantin, dev; +Cc: olivier.matz, nd, Honnappa Nagarahalli, nd
<snip>
>
> > >
> > > > Subject: [dpdk-dev] [RFC 0/6] New sync modes for ring
> > > >
> > > > Upfront note - that RFC is not a complete patch.
> > > > It introduces an ABI breakage, plus it doesn't update ring_elem
> > > > code properly,
> > > As per the current rules, these changes (in the current form) will
> > > be accepted only for 20.11 release. How do we address this for
> > > immediate
> > requirements like RCU defer APIs?
> >
> > I think I found a way to introduce these new modes without API/ABI
> breakage.
> > Working on v1 right now. Plan to submit it by end of that week/start
> > of next one.
> ok
RCU defer APIs require the rte_ring_xxx_elem versions. I guess you are adding those as well.
>
<snip>
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH v1] doc: use svg file type for ice nic
@ 2020-03-31 12:01 1% Haiyue Wang
2020-03-31 12:06 1% ` [dpdk-dev] [PATCH v2] doc: use svg file type for ice Haiyue Wang
2020-04-01 5:44 1% ` [dpdk-dev] [PATCH v3] doc: use svg file type for ice PMD Haiyue Wang
0 siblings, 2 replies; 200+ results
From: Haiyue Wang @ 2020-03-31 12:01 UTC (permalink / raw)
To: dev, xiaolong.ye, ferruh.yigit; +Cc: Haiyue Wang
Replace the binary format png with text type svg.
Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
---
doc/guides/nics/img/ice_dcf.png | Bin 39168 -> 0 bytes
doc/guides/nics/img/ice_dcf.svg | 727 ++++++++++++++++++++++++++++++++
2 files changed, 727 insertions(+)
delete mode 100644 doc/guides/nics/img/ice_dcf.png
create mode 100644 doc/guides/nics/img/ice_dcf.svg
diff --git a/doc/guides/nics/img/ice_dcf.png b/doc/guides/nics/img/ice_dcf.png
deleted file mode 100644
index 540823c4a05e9ea9d558d233fea77016d1ae5df3..0000000000000000000000000000000000000000
GIT binary patch
literal 0
HcmV?d00001
literal 39168
zcmZ6ybzD?k*FH`NDBT?jlF}e8B?Hn74BeolfOLbDz);d%Lw70NNOyNPNO%7Z_x(KY
z`+k3a@X<4KW}m&+Uh7)dy4F6yN(xdKsHCWHaBvvXpCpvw;1KwLKlqmjz$g7i%71`=
z;O&*A#NkSZ$ajH@7p7wJVsLO35oq^@NWk^0Z=W>n;oxu`p8w#NOsHJo;1Xh`CBz^u
zdIzZ}I(W`x-6a%La{^jclNOU&`_)rTS4-}zY76d1P4w1CVq)-OmrKC^5#ZY~-}pzu
zy$L?@i@zdO6jl2qp0>O_?^V0~==_TKu;0E)6AT_jK}Ys$GuU9xpl6NmUq4AAAS8rZ
z5<h>~BQL7h+1bg!^v{JkDk^GFZXt;WZK)wU@I4Lh>Nma-8Py=vX58uL5BwsNv0+TC
ztdd|ub|{f>I#qBR_r=Ecc2+(+F$m*+POs4gtgWq$;%5$Zz1W+)IolS4AR!?E!%MzF
zY^hc+(<6BKV$B>q*BptIQ5gaPfd*IJosgp=XT25!8zwm^L5M@!Q9uFabHF7#DiuCK
zt~qeo#^9j@Nyw!ThkWHBd%m>z=MtX)IqO&q%qUT-Q<|7KJNMhNs;a6pC&Kl7XSB)v
z3KtLW;^N}!aM9A*nl<f&LypeAH;UH%@~=<W?^;Mic5d!cle_a$qiaVX7S(%;iC{c>
zPEJmL@k|#dzb#-e#WW$n3TwM=5&|%zK#w5Rj~l++vL_;_xx2ZfV1g5o&-OTVud|dg
zY)p8a+CN(%s#Jb5B@9dFyIC8&!|9!kw<)+dnAUfkUON7RZe*>j5V@t@LFIjKzBWN}
zl6bSB?AXD@u-6mJ^$s$%>{fg;H**<Wy_z$R_0fFyxQl_NC8(r`t&yYbC$Wq!ZhWiR
zaQO>2)_CPI7Ur}ViP+8SHXAZ(qCX-8Pgw>@)`d8#Ub}%C2oieVUWG5$Z;wDPw0oh!
z(MS2V(Mt&eBO3Siyc1oDi4_JnN!?`_!F7~4=LJ<;7i4@c?GuTsp&oXOubJN~j*nRE
zr*(v=l`h&IUu)Y-G0<(qS?nZJwYZ$k+*tNcTf2SvjF$-h6dEDNR%>@OyFp@~wx8sm
zQ@KkHCa?M<DDa*q5JkVi|7}X6ooQlUmnxZR*#{T+GPIBvv|B=xJdNM>M<?%wx7c)l
zdn}q*;Wg|Gzyh^yRZNGE1Xt2+qw~^RE=E{lE|-=pha<-cjHj$U3eSXtS-y9ciA;Qt
zg#X2CIXogbtk+gV#pepYbkK~qZROc+2-nD3e6SS4d#n~;vgmMgi02}r<bB>}Fg(g-
zTDmx2YrO0kRzm_}D5S%J%DnQ0DdV>iV*NTt#-ch%|0xYIm~S%Q_BwvyfdQLqe%iUU
za<4uRAr<XM?#|y&3?~<aXTr-Hgaz#tT;9D;cK$2uv9~aYB8Zi$G+Wuxv-FN9+5PYR
zh4qI=trjcZ;=wL@A=W&9;v>?i^7Yn}O(rs$G1Q(|Z2z1sn<Yns)V;&#c$@TE2Q)Ru
z)6)`Bt!gQksJ*lfL=-F}Q=R&|&-+U>DkcK~7vFVi4u!dmmta<R;msWO$55HcDm$Mq
zYzuC++dJmp5`L=_vy^#UbXhfEraEBcG`ZP)uwPnpl~qe?xmmf+NiO2ax1Om+(c=}a
z!GBS0w-`sWU7FHRP;K|H)xhT98<o%yk#hLBt#8R=)!=YcP~pc5m_`)LD5eB~iYqs_
zj?tLy72jM<E{lrbr1en>*bVr_*W;KI;3E7^sdYFjoC{%b$NVtKVTT^7+2BYNw7pYj
zi4R5^w->W9pqU==f1#Eb;6C!>z+DXeb>TXZwZp;G`tJ%B5BYEu=0)FDN%x4fNM_s}
z&3d^+eysZRo^x@R*na2q(uHz@x%b$Uaf>KVBm(XH446M${l^6#mQ=@EaW_yr=YvCd
z+ccCShFQ?$1}?+o10_*^aev+*-pN^+qy-`EqoDYuWpSd~d>GvmJf>Q%JrCl>)WF6k
zmNOMMMp+hgneL-`A#pMMbJc=vS<;lS;}#5&f&14fUK<#DZv350Jd6LHkN$NrFqGJ8
zlLQ^XPt!kNF7<(R!K>C~vVL6sXh^GeVu@|f9?Xx@{W00`WXRZv(|^df><;rf5PNdx
zBpMO7vZw8Z5tyP#0>Vqx1X*<dv&0Kc;4ZffjQvQ21Vb2S)~HS*e}&q3vP0X5Ck>{p
zL|PgxQ2d3lia{mhxO*i0u!<pNN+=Dr*xkF}rL@3)H_B6SSLCd3U(-4g(^%Ggi&~71
zLC)Opd9psBf9m^tF-R2=i~Y+Wx~{DJ7&!b`+!vr6(LbBVOG$#4e;21&m~FS))*G-=
zPa}wqkl#vdk|f?wwVSv9(#)clt1G+|;%v%L28)#;+6VXk{7iUG+jNKVTH?KH*Mg`P
z4}U@{EuUAnkMX-Hs~^?DoM;li?~cv8zY`mcjZ7JLTz<$aU4RE4@Y}g{&JAw#$bdHS
zIsj~7y#89(M|b^nq}$-hTBu4^z2sUdUo4>v8BBVg%wyU%_MXf@bERkY-D?x<$;O<b
zpvBDRefW<DBArvMWHxmVUG9o}n*WowASDGwz0FLH(Dzku6^(`Vv9TH-Or_3WQek9&
zp{?wvKs0uy5PokudTJp-5ZP7duzj+zYp+MDA<51+GTg}}+XPcyOCY#AF=5lm{bEKD
zuYwz`>Y=}WkR(4(Y2H+%2SHKL8L4sdgQdlyUB8->Fe_=z_$V35@6;>_^Sx+A)!#{y
z%vNutx>hi*td+hI$B#YrfhqfD<l~3DlzhC=#`503zteBJi#)&=g+hzrVxg$DkD{I@
zg^onlA0{~L`RpH-+6gSVF_?>R2n@E#DgbDpEt*<Z@je@xlV4BeZ%k2#RC;@KHPXI=
zTv4h`bGcu3y~M9wU$HGWe-9h0vD0j^8a@f(O+_$Xo~<oWUu<$5exr+)S7gD4{3wVn
zgrav3%g?CuuSSpayqwxe{k^|Fjm)EGSivv!zjH;?(yL3=fBQ=%uWad!A{EMEQA^#0
z1YSLDC4#7>-9zS&zAcI<8lQ|mDf7d6im6Q*Pa;o%x1R@B!{cXusK5DSe-NiT0+q(3
zJAul<M~5YH9~^lh-eTUP{6+YH%@Tq2W2W5mXeE;-FoLq>=#Q(^6LkG!bA|cqcF*pi
zs1U^jfwNgMV;A~2jK#)8*61YsCi9QR)xxWa4oeYf>?DNCwZ`dlvWs4bPp=e=C;^0@
zKx^mvbur_;)>z-nUx7=5Qf6*XICBRQBO-{5^FAFbsff<KpkO!a8>m~+y7p>YE4&LA
zuaOFs0yDxhFwwq*qMEm7{5K*3xDA#58|>mq?5K=|*ax&{U5}%6AKyg@FV2-FFZMsS
zrHF6K$F#kK%8>ZN{<(`uWDrDi_jGHzJr(KJ|1yPYc)l)sxy8H2+vzKLzZc?{M!E7v
zF8j7?Yx*gy+=(y$qRI15nG2g2=Obg4BUlV=;xP}#z2<_JKj<^R-sjvH4CwW2sL>TH
zBFbS!1ZIVePcY_hP7Oo2*3+(G-We}hkFa4=pbOcJV<M%e|C+Y!zUqG{n|J=*>)qtn
zuvYEZ1ox-boipvgxq`t_VP5lxlooZlyR(J3H(<AZF?Y%7yd%9i@7`?6^1pYM<8}vA
z(fcoCgsMf!oK2VWC%GQAgevYXL7o?Eii4GU?~V&J?0;;|X}e4MzW2UeoiA;v_Kja^
zfO*44nT%$s<HJ_iHD0JS+i49q+(`zs=pvVW8i+;xuYtKK!0_&IS4RuYlSNh0LB0N0
z@7M}4b%hGL51O*{(vrSeZ6D8OHCSs;@XmhzeVhf(Slk`m&D&IY@;78t2t{dNPCng^
zl3t`DCq?X(3hcH>ORIJ6{H~eR;2X%NKo&enSZ7_?BtrJ5+S7I!1*N++VP8<HIBoPS
zs4s)hk6=@{5NU`^H+?`wcP3Bej!F{iUVeW-Vmz5k<8mL=To^D!#+Hw%%ierPJ88H_
zPv82kH0g1Ai1s0S9toBo^Lh+BR#Ex!G~`x30&(~+!!jfFHn1X}Pg@a3JmuSLTQ#Qb
zpQ{H2%o6CZ4m_;3gUII{#r0Kl7wZMxrF?pq^i$O~lx>HQg7%WjlKr9^Cy2f+@^BqY
zbx<#R&U+j$A5c+HBI@EZBVkOHH&Z}NE>JzIVq7w@Q)x#`<VnX!98`B2r8m%6Vo+<#
z6|brd_*__wX0{t1GqA?_MGCa@sMxH?;yyPPF1TB!b^Hhn>~1g2KzO8);$a8uAH*&3
zg)D9M>q1*@_xCVo#(yS{1NMP%E6V?!Zdx45c`#Zo`Hos^`dDktCFvy1sVN@{zyDdY
z>><Z7B&4vz?kkf60-^cx)}7#;WUQ;|Ust^A{Cc6ps{`J_Ho33e7$a~vXbL1#t`aT!
z2UJ5oWg&|^mMJOh+0J)(WRFNE*ofb`(PH0t*QCEMv<$k$g`L*rpQiFCT-S{2>uUSm
zw<=5E{Hpn^91&*!WTH|0=HF?*gcfcA(20pesM@XX<znYR-nMzrF*R^)xxTPuJ-mhU
z$llE1dA!{69o)n5LZtfsGN@04B+k+hUE_rgg5;a&za<srs{J-<3Qy6+aIcv(Vh0~)
zkKgAXo_BfQU6|#09=7DXY~^inHrqTNHErF7_5Idg-n+BOIH|92e|zJ_nvaQy)}E~B
z(h@~J+X?{0YlgxkKbSHfz6PX3+q1^PZ8&z^?eZ~;#lzEVFypvJwzmp*NjFhor7Ydu
z;qg_Ml8d`7>U`!J{-r<=sz9Ex<rB%wndw8n3T-!>HqoW|-MF)wvnft=dI;3KrdhMf
zMAysew~LW;pSGKW3p2yd|KC%+rqf%cE><%ubiE&Jx1%0MZ=Vj7xf*$_KBZqWak)pv
z4j1_4XAlThcpZNinfEz(+YMJ<jd{>q^T3Cm4}mOujeA_~ly2A-OCo@BUWkDw$?;>`
zNAbhK1bj9N)%iBl%U+H3#(W(f7`9(~6|64(y#)u!7m=9$ls^wV6HaD*K=}$*?`*(r
z%PAmMv7h_OPY`2waY<I!<1y{9lkvg*S2VsV;C^W>H9yql*ma~+oMt}#AG;;>^x|`8
z#R49-yL6HlpGSwi9Z64GB2a6w($eJZ>gJ#mxlbpK{v}O&29pecI1p8Qp<h1SHvO4N
z_+F)%xO0841x)wd$&8Vw&1HJ3w?MV^H)!Ha$X1O>IO*|zpZSM6>*QdWiv2AWGUOWT
zi-U3Fb<QX6QPsrlDFCm9oEyVYRmw8*Eij=nt7ZU<rce$)Tpe9zp*)({+wiPnEysrt
z%KsJKDNEN+O_ku)jJ2#;k}_W1r057%=`a*slt2m_4Ow4Dxwv;-?t_o4?>G2;iM1t1
zHTq@j=TP}0C)-I{SAz4qYL-K^qweE9?(nlNI~&!^GA8sit}QIH$3qI@ShGe{waqKL
z$gP6l!{HMFCb=eGSRp?HQ$W^k5NfTr!FhVBHe2dKhrT@y7@j_LUMO2X!U<qQcl9^w
zAMYGS1h^`W*Fliy$>29w+j$DuOrED%X?afH7nQAm*uGMru4R&AA=<D2a2dhUa`0r@
zkTJbBK6OSuwLJ5|P9q(YMcV`M$U<Wo(*mXYZ8CH?GyY$Y{x{kJ@CmLnT3u;ei(W?I
zMzAO`+bv3<e%#6GvtLFB?dSqKV^E+x%tEbya&0sX)%=q7bl~KNG#gIFFumkEV<7hU
zq)iJq1xI5@jQ(S?U1mj9Smdun_!|Kx7{jS*z}2gF{sN8ft`X>R+<Q>}*9|j&!o)35
z{$bev7y}Xb>i>Ao|77I<=gZUAg;co|EjBB;+t~N>_f;{P9o0(>CsVsQIY)8(M+{Z0
zX0uI|msFKEtT~ttnsz7Q&HG#`N0?_r9`7GV%pEN$8bm$wuu-KbwUr2qDS>DJ*c!rf
zo^{1mo!8969ZL|;%-@E-i_s)2-OJ(P&$+jIm%hVOr64opW`*}ty|pQ4f6VyiOYbe#
zc=({mBPV?e10QS2MJS$rn8*$>vA(j}V^5F$v@VUb2^BbA=Q!jU&j8~69bkn$Rr&R-
zO%6w?WLzx_VTtLj2CV!KGIodUcx>y$1FafezYvGt?k~pa`aUwZ1UZA|ee$tkNv=Cc
z{JGZu?=XZZBj@Ofc4+btE(a9d-5DgAA*&B(**VS-x*~(_)GHAM5wR(${3Wbl#jKod
z%!O|u`G9Q+|2vP8d@VU)x3xNg57CaY!$@wqG0aZHEbB+A(>?I5v+>m<Uz;jVsLJ*J
zW~+@y6zO<$tl?|lNJMu4hC=%f7vE5WJWMT(IHLy3^({EaKP?YB7ke1l@pqa|SOJ6V
zP+$pl?BYl#$__WuF_p)}{q`1!uG%lf;Sa^0L}IxMpnos@Sn`)SbPo_}$YtIzMIKdC
zPs>vd6ZxC5y|K}HBV94S<<b$`HYa;Z!TZt^q6v86x58P{wkI2UoRXG>hZ~Pm)?ev(
zXVwAnw+Kn2g}4PvdewlLVqV+5V`7&VqCwG_bUp~XJRAD!ZJ?#2yhSQ*$+5}6kL+Ty
z7n`c?qP19mF#uIzF`@8+FFFm+?x5Zv7Kf||U6sF$ie@YBg?|s!O5Yd)@?QvRXPw9W
zzAFH=P#$oWOh*kT(7*Ed?XvkR4WNH}In6(}n`hUC2FjAX9nI&<5ll~Xlj0N(M~gM9
z7W;3xm?1qmFD1O@Yps*F!w=>vJEt4sE)}`;KqRl-pB?W8e==fRQHbNR&yXn9-CPzT
zX|2+4M8JL)^24fW-lP2u?B&<ST&l#FFZ)D-uFKJC<Kab1+ygzELfogXQQ&I#70tnM
zy54n{`>H0Fw|hk<)fQE<3!TG@ud#0no9o!?oNKC!@}4eEg2_~&7N3qT-Qsh9Iv>z{
zm)0<yr)u#7_;6@3bvjJz+ggqdL`H_^?hjYdPzWpB%A2F0&w3<JB_EHSqr-ppKXP7$
zrWt>;AauX_U`)J@pMV`1Q@7$z#}5b8W-NWwOd}B|VY?aWP5mhB2UD^`MzduiO()As
zi*rJP!fC{s1^J?Ot$QRN*3s2*LB2-Q@>P<~K2Ce@@ChWHB0qvX-#@OXT6O<z6!)t8
zuHHuvKO2MQPf~XiwS5gw^y}|f;*G`8qVqEEj|<{jf0-d9z;+?Hsd8<$bJw!TNAoog
z!lOHQ@wE`4mlgY=k-F55@ok?~VYgZ9ii?83DUY{@BQPW0Kt&6>3A?52TkfiF>Svqg
z?k5mH0g<X3N|2TzLCY-?>EQvDCi4-yJ&q_ZOB?^Ly!aEOJF0~0B(D@5kKF=wuhD|l
zU+C#Q8yn1*!P{yZ%n10gV=xIA)5(n8w~GulmX=Xr!2;1WoHvT$;rkC#$)4|!vFT+X
zw1r+D=B(0|$vmpR_ND*2G9vq;Sw=C};CZDWLZX{Kby1`C&o%_Y=2S+BH<R2XLYGSE
z_=^|+ssvIdqk*Vve*U?%e-cz*yJPIOZgw1c$H2N3J9pcuYz={bj%-@o*jDZLI~QwC
zBAzvlhUpVeBJ9G0MuPyI?ZP@V^Kz#H7sx;!p1xk4&&-S`|3LqHbLRJvK%QV=Qkb{S
zvFK<?FJ<lVt5;392rQqy5=su0>1Ji8iLbg#!*Kvdbu_1a%bzZ^ESUW(q;MjVZ0*f;
z!p?7bS9101myQ#T`rwFer2+tPS}~s>qQQ4@qljMnW0?dl0Z$d)m31QSYb~j2ibvAI
z#x<niUH!DZdzYp5SBs_*horD3f0JK!rz1mnfewB4Gb?v?I8aoYjvl{!QI?LHC9;+~
zoO4O`Ss;?Z^Aldl-U@c;ZGbzXt2gmFZFi(IS+89!*Lk<F@bP}#ZyFV(^jK1WN90`~
zt=Tthn9XCnm;+Qjz8G0Z>*7Ne;3SWus}*&Z1bGmhWqu<dAv#}%Gd(=ndllitwoRIo
z=Y;_o^88@Qse8?9I6Pwi)bibdeYC$Yfnv85*zhy4D|RdlSSDYL0)*rCoha8?w<DT&
z1A2C)NtZt0&W>)}@#IQP`74&+aOIqCHv=OPy;3=g?ZXcQU_r3<9`^sxN#N(Sl-g5+
zhm%cbigdiDjZN%d1~Qg0ODQFKeyoJR`kd#k|IXA-l{D|)&PM<Niw)F*#IwjCReL1}
zYT#Hf>PQn`j?{K~c<F;FKn7kjrJhC8ZhL_tqV0FFyv`9W-^|cIJ!3w5CB}a^^%s2A
z!kZ-9CkOI{{doQP-#T1pWlE_zQTCS9g~NcX{+SG4b?_OC`xuy3j+4X;3IK1etFU|_
zwU)#L2MPfH`45d!QU<#gu>th)=$7BR;X|@wqUv5<r@H;<-%;IVy_9xGyk!FH?40`i
zh&xk1X(J4^;^f7B!0d@?mW)-O{}o#QN$MY^K{-1A05(IRQ<dU{w)@%ijcKn;(QdbU
zS>2MQW2XvUa)ii)_u`-rnYHKB?wPOvE8o>{*St4#(@XuD6t}M<Kjr}^p&@SEy-a^s
zicMrbiSmDPIzRp-EIO&}#9Z*`{%WHma(;pwaXmQj(l?ztiZhz)?1y!o?2<o%oYP^B
z6=Bgzb0(P_rjGYJ4x0Ex{g{L_v_D*Sp*kJ(f4vqey-LLa(*Ttf96(dRO5oVI*kvHS
z)Yrni5#mxk52&L1ST~|zsG-JRTfTf6_;CA&S~zwWFHy*R*chhF3+&o6)p-`N@F9?l
zoSB%vtJ+66h^$SOipE>?*&Oc(2)=DpfDruZ?u)O3FB}0j^bIi0vt%bB26EIJ=b_`j
zOA1%H`Vt?X?rOD8woVkiMxan81My~V*uaT99~-NAswewgQT>020z-bjLUXyB$=(5F
z9G~s*y0#nZkD*LO`6f}2YVh8EaNq&r+Ak|aWUE>bp(T*mSAqFG%OgQ+P>a@EZ6~h^
z>o?rr#(V;ElS_BiX$tkX(wtspX6DCW{Fg{%=6j`jPqn|>OY@nhl4Zk*<q)K-4WtE2
zT=Xw+#p}(Eb7^W#JwAPHY`QOYdJj{syXN6!1lUnlx{1dr@tqdHzW4wjf0jn517Qdg
z=!?1^@thZ2vBvf)uiJyS<5q(c)qhhM#%1R}>Uml%abT1~`U|=XaVE#c%g7PeQvGo0
z)-=6kkfAb6jV(L}fX;C^-057|Q>o|d{FD#WUvE&jnVy&A0k!CR%(MCH2e^R#JaS_N
znggoLNCg_gLo2g1SejKFNA;^kwYJ$B9v)jOBm2@Ez1!lKMa;9uhM-!^XPe#-5Da_T
z&m`v{tLN!0^5385dIR!-r_<Mz`Kp^bCju;?82igI@{uap{k-ljoM?_CwNSE!ijVV0
zz3^eE&kIb4-5Snu?08)+a0#C*>;Y-^pa!G+!0pq(t;w3<q&Bj#lW>Mmesfu0*hRc!
z9+d~O{*%Db-?Y1~yh*E<m_qESjvJr`{%e1IS9+h$W$(57K`FKllA9RK+##=SwgngC
zEztt?2qCD<Mnjl+!{OwtG*ENSHW;<0Z9W(_c)JW{tnWY8)wyTyk2BfSs+*cg7X2EQ
z=d9!RQ!e9TV3MPe5(d$xXXIzg%MsRqLI|_E<WHe8nFI9>+r4Eij&q(H_&ItXP8kY0
zhglRO!te5?*<|T1*jqku^}AWsT}=UkmNZ?_-w6}<0=A6MGLhUG6ZHH2?K<P9jEQ}8
zDC&66za{7|^II6Mi%l4p#)<PZPsg3p@u<d2FBtkyiJhz@lgzafkLJ#UM3Wkm?;XTj
zmT!p+XUhU%EmKC<2#;9CaPjnKoFz;fD2UKSp1>Vu{yM{NH3}k_`}6Qyb0P^HP783l
zsV!U7SEgrT(W^G`bA^LWUP5e($^kB;9^Du?Kna&UdyvR;Wg6I|skPDw-YJr4w#ER(
znm0{dZBp;MSFJjxQVq*<YpI?-@nYFvH8Z-zqQ4`s2@8mGHbH9ELNAkL)kb@!ZvOzL
zbaE~m)6AOw^82--vSzn$W^!$Licpx+R^5ALo10FnjySnP4)afW9Ejf9@0+$`DLTa1
zH`e)E4e_21xXsPBr%mY(v^fmgTT&&@_u<6rf17?ToV8FwGlSDB1?s>OEw3~CeL_5B
zXD#8)VY7!#1e-|^Uc=Q>)#3wU7FH2@(uewe0EqooU%ubdPph|-Zb{6lZ$-_+c=Fu$
z6$Wz0#!NBZb&#$q9@yuSlAOHCLTk97!#j{y#Pj;7dm7`?qvAWeS>%51QL`21@+god
zr-pamJ8Ep7Y5(wTr=3<(`G1$X<Jg6Mdr)LDm=W=YXEqk0J%gk~N1(175Fo?^mDiec
zM!sZqrY0JrI6@L<(R4uPysw!?#(=sMP};O-=5t2qzi1Q!!exVOpqdSv9IMm3MDf9y
zuD5@Fv(3pxMvK>!09T}tA7WAoy;OJj-=}>aRYYTSwU-{%bR=a0zf6?!E=9~z(>@6&
zK8C=U>}FrAk$gC=0rbZ&^TqERW<P)XLdr+5L|<EF)G7sjxbCO{#0Q@#47%PgmKZ=3
z{#$yRkDr5LjjD42;*<2=GR{(ean|!S`bG5m{QY(6P(ZIo*^`Hp{Z$X3(?(%!(@IPh
zS(4dY1Z)srZKK^fSh_<tsh*Vy<*<C9z#oj^8y;D|LHPEHTZ7SL?TeV#u?h0|JxM+F
z&v$a-Cm}G-mJe0>S58vXkI_a<dWy~KlD|J@-8tEqdfE`m;leCV4yK19wqxDYYM##e
zpA+IgG$D#dULW#0qAVl*HY$G&UF|!Im(?`qu;GVQWCXSgr075=y$@$)BfPSPQP`6|
zmkxU2MMw5&0g6{+u6G1DibM`6IA9M_94K+Hl3+~%XeZ%rpotU5r(X{R0jrbdfAqT9
zpnHFA+gctLP7gX6g{}tQ-p1!$-ZxzoaYY-zy(yIbD43H}_9niN20+k%wYhBhaUy~a
zogDY>^k#-ahKEh#r<;;>HWpeIB#Hg91m44lI^)OV>+6|oWTMHiMVf~DxxLd%!zj=S
zpkh(X{r2$k&`JB)d4s8sKY94&!OMIc0Wwk9$@`W)7}h?)^1ln8<=Nkg*hiLE-ASh1
z#JxLb_z8neh`oAU$Z!1k%|-|OK|tqCp=r0pn?ez<JMjL%A9Ps)>6LkUH%{PifZBg>
zCU|i5;<{*hMp}gt1~raLsrM+_UiP#R8v9x7jZ=rGn%OV;+t0WesG8>QBy9_Y_t~Zl
zI^~ok8{%#h2=bz$9n<5{@-Uw~H_y|6Le!wL4de)@>mmY=4n`dYz0}{_)>{s=YFwOM
zEl#?tC;@{231|1ZA8T%#4%kgr<NrIAVV3z{mQQR4f;-v`U0kxg^_FJ7eQ^{t-SV8y
zXm3N_iycbP9)oj|ix20U7BkXB+=cW-_0%QaOj|zyF*)}*)4tJ#-W!cbMM=t4zCCQp
zgJtNsV6X8Jtht!{zR@NyDX4FCb%BJYp|$utVtWfdqZeU2EbI1(r&uwtRXrqtYFOzB
zmXBU42b3lAKw;WVR{Az13Sfx8QU76x+49QGBSXAKJ;z{sLe}vrhwve$A28WZ?T0jZ
zl40o%_w8W`%%vOGY}C#$guce=W?{is)$3f64ji=tEC9M5H4%Kq&iJkIu;J2SIqq_p
zET#ZpEg<6wtS&1kL~O^oePwi--?(*H8i>@5m)l!C{Y+^qt%3m}fK@pR4zx&~-%Tn4
z3K&qP%<@>;9FXw4t=Pm@W5{x9VwAO)YuI~E2Q3T!s;_x!IYW#neb^pe)?OB5PfB<}
z=Cd?cb3=Ixf<5(D=sJI!eC=YhU4t3sT0_3%9obKJoO19m`$zf=$Z#dp7+{b|Z2r6y
z59t+iJEcZ|3Uo|#`Ca5`Dm8`zLebx_L_eU{g%($mR%g(AUWX>81#~y-+@2DhAxvM6
zi6W)IZMj0jhJJwfS%`LLop4suSKKl+eB<7+(BjN;fZ|r}%^t`{Md5%XK38MZ+U45B
zJ9wx4WlMja@ZtmP6zl7Lujh~V_yii*S}FWl?=Y<?z$7t%iq8ao)mYO-TKv7)>JHXS
zw*82~hH+$dcE1D^C^9^n@|R9syt#gfU6_0MUUmp_YUkc}l70g#@{)V3^x;iyMfwnm
zYA3a_=MD~U15x7*SMra9r4;bx$!CXWCye}&Pl~tvtI~34_NX0=!BN6ix`(Ce5`EOg
z)b=P~s3cD-LZ6J7-vx!vMCHfi3s-ysvEc5wa^C*XQbliv&wm#^vna_)&kp-03=U|&
z2G75Ybua2_)XsEv-FxuMG7JUa1bo}yJFg74OUMK?s9fz0IW^%ct)`lUc>!S=v%>u+
zoEABdTY;uWk*%G0?uFLRoSTD0pxP2mCI1M=hH42X)y&61S)UJ(Cw}21G$v}NTbx#9
zAWe(#@gYCzR|HL~0gxM}f@k1Sxoc7WnuEFXD+aoYlZQ~Ft472{lV*0DXxFhkiB784
zj&g*o=d0(m1tOX>K3a&hh`bvh_95};V}22-4}OCI(8X7<5f}o0$w=mJ;a#j!Hv$Py
z_7_0g_Fu-G3Az{&ygJ&?9K>{cau|F1&QBwt{)yx}1EAXeTLI{RkxWjvn?h|I+p#2&
z<Xj1j$#p-G`2F7-7Oy@#HQ>u>pqhEp8MN@z@L(k~ZHWAb?ZPr(Py<vG%Rj#bSgrG3
z(usssK#OWmk;PvA?+tE$JN8_JmBy4s^Uiz%na6R@KrwAWV?aUsk75bJS4Gg~egJ*#
zSDa_`l~#l`k$j0c`QPsY@KkZ7vT<6i-TTsE48EmAeu}duN`zwnBLgy|<gd-bfC5FE
z{3M3~nsM_cog)H<z*Sx#&h+0(z$loBLw_SttNw(-vrb`+(~6Y+5`3PnF8*UFBq4_q
z<$DKVi-OB)H;g%mf&Rb7VFr(8X7%CSX2-g6$-JU;*(}_lTe2NK`NW8{pdurrEc>yl
zpAn6^FL**iwePkySaJh__2YenuX%leBxJK4tI^K2n?C0{PWFefIDAg{YtaHx8Cbgk
zdsu>nL=N`q@rz#zQ44E!C)~(q=KmZGlNtq1e?fp_VZgyJumhMN{?$z5>6l%n3#s1?
zcONTLrjo|MCINwc=QwdFg5)TOpogInYC>Nj1AH)$2X+Dz;1hO^?L^wXs&oCS_v$gt
z0G|`eTFC1&5aNX1G9muYlVY?Wo0|w)c7;t1HL+EJf!^Odlu=z|U;lZ)B1Q^<h3o~`
zToQEu8aX%m_jYv%btD5X+cG`h!a`eYYggyE2sbAeeSH5r9ISQ&{5QY@@|HBO7CDXV
zU4XN1TjLC6`7-@|FubU*x=o+JI2r<Vfs&ns_yqAMH?O%)%+Mvnvvy#W!G?&M0bl<8
zK|y}2sJZ5>wU-aRY~+ml^&eVVBHKFg1-zihJBIpv_JqBxY#0+G3kGV$kFDS}dQ@vK
zF-`tD4=;WvHDcFrSO}_D%dtATihK+NQo&u{)PMz&O*cNVRWvYg`Rpa;<06eN`3T#A
z$qc=d-i}#wgd`NgLSaVss6q(h%;bv<T7~#hipwiK2?OLNB<XB=lzx#_5GE#to4{h_
zPJ=w#$|Pq61S!!1nompJ2c=`dGOzI^|Ip3UgNHM(Q}VC>kgMr|IpWoowe<AD>D3H2
zcw(W@!Ka^kFyj0jT0Q%I=pd2O-pi=F6Lhj1JXJTR@DN2&`MHAHKQU39LKv^I;DF6$
znN+CVjZD>G$ezHFAOWeWbzNZvm)l*!VQDtFXz;5$bArquU!@tBiV*&}zkrn1^2H{H
z3$+Y|SPU3KGCPnFv;!=`e`8`IR4#pm9h&mi_VSW#ZS4ab2P%Ldu~y2q1_~rOL9f4a
zLZP}!Jx1I1s3F&zo8p=5Om9XD_qZllQWrFD$lp*VwfA`9>q1^C6UexQ%5e6Y?<%Bu
z;fqL%n`XzKoETaYyGDqIFg1%353Q}kGi56)L8$D`W`xBi4<69M!aMFB4L3wkS(uJO
z$LJsHtP6yIDZ=4TqWo0)1R)10M7dm}0FD|gWz2p`8bBE9>ai;MvEJ)#pt2M;9Oy!f
z0k@NmPFu~hus$|gFuJqrT|T=VWdPyi{?EqLT*h46fEWQ^IEF&#+q^G)E)Zw7oQ+>{
zUi_4~?LCh8S`aor&A<}ksb9}_fx}k?CAj|GQ&h$Z3GYUkG~N&brlmBYB9dz=K$#`Z
z1P@Y;q-JGfrc0xym-g^&pFz6z{~D?zl?j^U{<Iz(#g`)=x*;ySIsX+np-F)AGT>Ej
z0Y1O9(ZS!y&H;jmE`tqD2Bt*3uAUFhSLS#h*P96by>%_y<g==G#9Yj;(9=^z9(dwV
z9Z)7{GZjplq#dVIJTnc>%+G!Sl}XXznHOY;0Uq9Nw1#%VP5{Kl2yD(K@pgoso>~3U
zJ#h~aDkGDT{|Ek6fVz7^A1|K}TRIJ8SX+-iu-1^@>ysXKM*N#G^+rChC|m|6+rIvO
zJQd>SIUXHGN5=lot!Ka2C~rB}(eUpKAq-56J}_LVx41c`n=`d8%J&jRt2=QqQIfN$
z2JEMJz?MM0VAc{|sOlBzC;yCF_!QMYbajrEdo3MxoZxT7swhMX#0bN$e7QAL@dT*n
zcT`cZ^eg;HZ$_pqHShPaHsAO{N<u|QLd7MXZTa6X*W>APbHN*LgQMg1-PEdGxRGg0
z!8%}2MSq8;hg)R3$UG5<mR1S!1bU16g~Ptcm9r@)OB>;XDg!>0sMTPD5&DiPdIllP
zK&^aDWa-~~)8DeCukYcBo<zT0<#;8eSMWTlA~5RWS>NoJq_vp#oCyM;brmBD_Kc!t
z<>jugH&wZL5x}AO^%;!PV70LNos_y^p~=|HW9MwpKkJ}pTtACeZvK|Q^!|g&^IG#5
zO7p|#b9L?eQ3qK6cR<|8Pk#oCwIO%dof@PRRIs4P4&cV6AKNVvTN0a$msX75wSxEV
zn@rUy%@@2fJ=cQ?J>(M|FGl;Jnu@<!7ra^d(e`T6T7G~S99PzrNo&$unjbar=-5Oc
zfsy67s5NO#Kc-`K%H5P8r^E_knSOa}or+QZ-GAawZ`MYMLl-1$L6e{JWAjTzMjcL1
z{aW1cb_#gYo1;>YDDAGW2ww&;?j{a9pwpVx-5}e_%HXMtr8VDVV=lO!%uUi~gbrhc
zg(8`23e=y-$QS}exmcZq#$OM2XR5x+$?veJ?r*NNZz4GWcUq&sj|&MbU_T|oj~y^)
z0PiNvVFwaJp>>q-Z=$L)BKu6X>||wrg)!6af3HuIR$~oO(KtU%Wn^U1h^Q|e%|Y3F
z%Dh}Vvo0omUdgq4_H9vtPUC~h7}uXH4&u6C@uOQbQ^tEQk&H}<FZ+A3@x#4r3@_HA
zqG(OwS2<2-bH0=|`#4WHzTKabBs_NJDL#PB*(*aB%}Ep;(ccD60zE+?Ne2AN8FB#k
zDx5BZ6h7fT$0r>sQ{hjYN1Z`FcQ=ll(je}j{KJC%7gAlEXETwuzuro^QRcM{w8f^N
ziL%8e$ET;op7o8SFbhw45s^};4{W+#t2-Dl6y~N=vrd=J1v31Q=sLdHd^WuYAZTvJ
zbtRt)b<w>7(UvC9@SAQ|wj1MdV+^D;-pls)0WsZM*BUEDYAekM(n<5l`Jnm2y-nGm
ztbx?V02efQuA;0(icKgT8U1!iER)%Thd^=&AXIL6U3vww_VTLNMV>y$II*UpFZ0TK
zX8WH`5gk3b=G$Rwp(zU}TJ;-nt%JOPHcm{+W|nPmdW*-Rt$qYsgZ7IC!)NE$HsT7<
zeBg;xz$T`=nEC@HR{5v|u_N_z<!7k_0zj2KSEi!_LVN@Dh4yimuEcoO>&Lb0OCU|?
zLz0f<qLs(15}0Yo=o28*TnY;P3|prezp0V2g}fdx_xPDF1<?c(JTfiuV}JhnT+vGz
zf=>{VdbCDiV6iO@w@%eGM&C7ojUSr=?8sK-8KYg)pQ*`rOyP9Tz$FDJ$C0j^3cWfY
z@6Xg?51`Gz6mTWVE^_yH2hayc=ZW89TpnJoGH!Gn-`~C#DlAVxL-4yG*5YSBo8jOV
zsH0rCm_A3OqPRvWeS8BQ5mTvtNgVn$D+6XBGob?EO>csR$}+PNeUDps@D&vYye86?
z+ehi;rQ5@jxScJRJCRwY5nlAgpp~wt-zggEqK98k9RpL&%a+CkL57L;$U+9(vQ{Mp
zvB+l}K>;*!*PQr~5dRpMud*Vzsj7U~q_Wb^?qk+EVms`mFOBg|sKX33=+CIA$i|vQ
zQa#5jY1f;Spc#J`Hw!zhAGtn^{6wG}e4ph@7dL2*O6--g-Q{U_GLRyfpbUi@Mx$OK
z69&2MA}f9wX)si15G&J`fZMgMcOeL~;a*91cp^7Do13j=<??Jgs?_cT^SdNeMI$D;
z`x4_7{XQT#Ga{r@Uf0yv6`syQToO=eY`_xWfQ`6|CX%h(>nnNs{D}O+7LVkp$i(RW
zsrSD#Lm}fH$pfF*h+fJaq_7n<z23KxD~=lH3=lU|uucoeBGW`sg7o3;{BD!P7l~{Q
zY-M|^6XcxN*&~my05NaGB77BqmnPwE8lz;w$nYC?a#D&-{y=xEs?yW1qfBIqyTU#^
zFVF2Z(8Eh@>7tvBg<NNxv&s`t;Ew3s4^CvI6h$OWc)o@NolQuKLXHNk08jI4&tlB0
zo(4i#2jy|h_6n<XOJR@FORTdI_Zgw5z<l5wJJ9gGz54E0QY)hHOAA^i5Il%>8Xnw9
zH&&u6#Nb8kQ=J+aN&D<y0?-*Sf1X`B%0|})Zvfc?&<Oz2T_LNcT%6V9m9noU_~hA6
zY~Vk}5P9@}@`7qSr|@!HN8lPgE}9hYQLEC467X0ta4&Gq2uK^MM3J91R;j*fUetM)
z3j5IJP<i_khZbiAb>~lb*qH^$ucqO@0G{$D54LvVa(8(^_QCB}YAQv1e}IgPOkag@
z_YdzR>1aCLdIv^kW;q6YTr@9AdU|@|I+v5R)>cv5-6v@onYk*{;nj{nLyWaC(_op5
z*?KkC5L6Vb<<PIWE@m`zyu4SPAw>OgELGLjgz@Y$RGql0q(>h=Q?_)+u8imie~DZ3
z1t@?WQ{md$8gL@Z^X{O=&CTt2D+8B^sOEET)KIFRU+Z6yuCcMxllAWM3gq;*&*tWg
z70k3he*6HA=~#~E4t9l+1*=8|L!n6=#$CQhXpN1He{zPSB2z!k-d%O|d>ziD;o2T*
zag!=pvnceJI2n<YGcnn!v)^{RIxNY`qG2_R5r)z-ndOm3*EN*}_-E#C0PVO~KnwAH
zxR^fM9uDR>M(Z`I$jI55saT&VP)Qu<6C@Ld3?}m)Ej2YHCMJGb>&r_Dg7&Z1x!lP@
zWmJa0^+Ht_-R@mCCirFPosV9Vkmrhn4TU3&f}whECtjH<x)S0gPD?1<0Ifx3aUT%q
zEgPH7Shlpt{V`%sZgFu`2k3R7T8Y-h{&bo5)8k6j-@jk&L=5F6ko=kuKC;l3=0@QY
z$O*XJp4@nx>~x_*QO{iQWg)$t)cyXjam-!$_2%6d7eujje)F=PCoFeBFrj&)3cB11
zyOnqY4o#FUy3*J2#Cnb7F!*)h!Ec#ZUC3zfWZ!v%0s-EX&6~nlwinHR9~oNMH=zSz
zcV2GxWFMQEA&$dUr$vC>ZpQ1!vFUql4J015JUt#RHV9EhB&ZhT13YegeB97Vy;R2s
z#H!!y@i3)pg=nmzO+PXguwS#qqBWMM-@;=xA5JRhb(8da-uXy&xThXi;cS&@(cWHA
zt0JKqT1RDN<x9=DxVYP!n<kGNT)oZn-SOika&mIfoIftq$jHdNHcQ!~X2Qb4$4u-4
zy}i5kV6nWH%R|2{r_V1gioD4Za!CUOB?&qRU9wM<>!|yfJ98dkP=!wj>TN{4Y0Z@m
zrx{$FjehN|8a4W6+nYqkGBV2v8oFK?3g}+9UD#bj_O2^$6Z|!i0ze9Sk3&n3<@g`o
zf8Yo>8EVITD-~Ks#!$$l@(xVF_YLS5&sAG#o0;td;n3Wk?;`4Be;C<@`KB>bQ?CK8
z0=WKsgY!}5$H0}RGa}}hUsl~J@)X}DXZxp32NI`?oR6lg=FrLbEmp?}$f^sGr6h)v
zMOuLNPk4AZE$y1S>I-D#cJ23ETr|JoACrYW1sW4$V(_sE?lT-69c#P>+Wb(Q0Vi}l
z+hWnK)@kwjz|Wtk@uJdVvhea?zC#3S2ldCna?A46R2((a3=b_WZJS%Y^ARRze{b)b
zxK0LDz-i>8e&9~h{w8uWsjdh<AK9s;r#cFg2m(?V86}?g-HoMDxnH0St6)ocegeL#
zmI;rC?c-9_zT&S?tCU37zxlO)=7bp|5uRPeVtP4-QAMlz>+b3*c=rb1KVKBFTwzDo
zU?Y-7%HShb;k6C~9T`y|jq-z=s9<bzGMW`?NT)AduKv@b;1$)_d!8qc>GYiF<Yb$S
zuKxHE?HVA;f4<m3jkB1qKG^7uPEAeCL`aS}K0XG3VK{|9nlECstIK3(Br}AFt-8AU
zN4g;XM{)6R3PB#1V}ox5b!2!F7wZusLy275K)4$4iC09yeBb*a7>a4!q<wJb<UAhj
zkRc72zu5jcV8w}b{n`v0tSRVjv6uWqqO)scXM-zW6jJAUgH!R!^@f~+M%Jv5vU$u;
zS_$lAO$ma$^;sP)ZuGFzH(CS`H!&#)RSxlpI4MCPRlujHkU!BEn?QqJP>>8bf;raN
z-j1N>OI>(+YNtWwVsyArcdYW7FtD}Uu;a&IV#{%2+|V|x^4quHwkO%6xPnf9YmGXg
zfBrzD;Z~pCsh578EF>sC0QOtSC*Cg&E9UHcSLG*mM7JNjZi5rjIQ9}XE<XN?GECXd
zt}a{xaY%Orm7=^nEfcmfq)e|#Mpm}L^Nz<u1sL}`N6Z1fq$^AV1~rOjbN?99%ci&v
zHf3bRpawRb&&)51gyg7%qE{e@DX_EJ@xtiy7Zi}h^0dqP#sQV0qVy*j!_S{pX_+jE
z6nAPE6m^P$&H<B3Y`)ji!xiw_UV}IKCy=hL=k8d}#~<jn_Vz{Ra;PsEnXXA8FCOG;
ztQR2=Nb3t692{ULY@P#O<mvuwYcM%D{xu>Shxy()%CNo{D^qLVtluCb@p}tMy09k#
zB4S%mZf>qho&v=OH*N=rp!1<WGMYe%Y&@&jI3gxuXZzFL#Xhh3*js)5<p>)Ua`wRj
zmAsYtiK!_%CjWfy_wNn4Bp|Q-^B=F*v98S*8eQ?4#YQMychIs@z^}m^E(F7}_cwB)
zMsJ)T>2R<<HijNHhR~p#58h{wYY*EJsra#BMe33~<2_v*tc<ho*eAGNXOn{4rV293
zUxH{$fI~({&3Bm97l@Bk-#QYxOvS~;3ui@Tq0k<{&E|fYBc>+i<>h7Px6+o61`;7D
z2I;Rn?atR6LEi(XT=L<7EOCvvbAN;Onja4j@A?g;pwn02ElB6Xh1qH=vPOx|pWlW_
zBIfxf23^4k-`yRxsH&>2-cd72XfRI!HqVsu?+O9VD^^xjMGMvZ#8wR<WLe3BU$O$i
z=I-jKx5JotGZY~h^*);@QDjZ*?cr?}9IVp$3aB|g4fkP&gwtTLMZ)UuMN{LV;`Ehb
z;u5p0KEIkzw~*NUdw>OV`(nnb`R_JI>w3Q*002();%M1>rrfZcX~+E&m?LpuzQ*Qb
z^68DNXvYihI&7_59S9*+y8Pza8t~SpI(A+}1Xi&<fQAN2-zGD}{D7BI;XxW)1GH+y
z<m3wgLWYs?cK!K-O+Z~&S9f)^i~<(3k%GJgM@o$b{cLzN7#z{O+?%vKUTIrhT^-?s
zN*Tnkp}sA2314x2ZF!UIzp<8U-$yl~M6aUbevX?G6C1rxM(%Y1?&Nm|+$<N>M3dir
z!KF|&HXkToogRWsggy{#342>g;Qy24KB+=}qF_-v?2dhzt^D$EdqD$2TCR(WQwzkT
z0DzR<RhK<m=k+mGSg{P`dsv96CJEJ4wb>{PNY5XOoY%WBSmV)BM}Rby8Si$nN6<)Z
zYlP(AW`Iu4_YJ@%YinzXfKSTGV?Kx|qAs_$w-PyS;N6{_V;3PZGP0fb^onWdMLCW3
zp{Ei)5=dUBGHk9CNy96d`NBTf1Rl0hy4{1k3@sTWf((D#AGV`lQRG-;RMKET2+?uN
z@Vd!F{$(_JLSR}tUw5<MrgoBYC*TJw<QyX;2<c4XHXplM_Fisu1?o<0AU1GlB<~-t
zmZMBrv0p`m(Wl7+X&dhPlh0~K6$$6(p1-qICB?-Cj4tyHP6vBSO?P{fMVWy#&R|~!
zC%_k*J@4`q)7!C98N?f#n!3X&(9-;-p!ptP2>UCM(*Ah1r{fN+5tvU=5C`#>5WwL+
zt$Ugo#UTLob1Nc?Zh8j0nsE2fV#5!ZFNQ2Xj?e4lUf<IFEiRVc6Gu3Z>X&R5ChQ*-
z6(`@}5dm&T$5`0dDCFpPvFvq+K}thJM0CCV1J=^=1SE!~<fK_|51!b}fP?DAJAwP#
z`GGn#y%sJyQ0&hE7gGbc)9*D+-cMenC2L^S<1itQ>*M{h765d)i1*h+g7C=51i$p*
zn-y|%bBoFzX9ZrOB?-BxAE!y@OiDnGcm31HQuyrw*A1KoGg|$VC2vFaZt({Du{xRR
zzdg+Y6@-njyW?hL*NyuyQ?pCH?c?|f`*oK}NfDKeh^_Pi(sk&It_~dyB|8raAUBYh
zhoB-X)9++|48)9rA$$<6`eJNcWxZH$3wRGXANDW!GIxN4f6u@!YlU_|OfFd?-s`bw
z*KG}_c|R=u-d}ms@e+*mLyC5EcSJ$V^lWeP-5RrmQV_sjXLg1~o;<D&DNdLqu06aU
zkck}G_^sh|5n#>0hJ}-GZLPHVO-)T5Fa7bepkty2u-EhU9G=F)+`O$NrxprMV)&U)
zt#SXZRMR#k*Z%CS?`?<-rWi2k>n|apq37r4pE(o%uu(>IhlGb?zI^%Jikl<%htK=#
z>uc->4BFn_-sa|J1GL^$n+!tMcrq#~tg=lYOb-t!NlBB^()2Vm@bPiv`FeZi#AK(B
z57)P+4~1(V%3*iN<U$^(s38DWiZVqJIkxq6w6_~rjg$t<K4E!Sl8}%9qbw~g9gDLg
zFX4XD61p?aIPoQrY5w?9gB{>9?CeUFuD*|Ke0;=9T#fTxEf23Hx4`&dj#}!#jWrxB
zSkl;FG|<yg7FPh#2EzM*I>(Ds-eIb!LIHhFPZ;G#Fd)y7@<OQ#uMvFIURwHVz0FyA
zAqtKh;m<iEWSe)sci{7rCd}eIJ8T-!s*=7)yDgI7i3i%(_^K>XDRGB?FM4T27~baO
z#-5zKunqkRV}POdZ|XY^>O8RqE88Gs<bN`AT_9x93Z)e3WfKm_|DIF&!9&1gZG*OG
zI*HGfj%aF)|1(3O*4y~hBRqUbKvQXp<j|qia}0@;#|J^cuA5qKTm@EdV%+fHKp^0l
z6mH)6P*uD!y}V|%NmFE&b_p{Qa54}_Pulzwgc(q&Ydsn<bpYwle1QR)9~BaiS$98k
zpX)@2^c_YWiBFKKa@*JAahu}WKUM601Q*AYh5kqlJYNEkuScSoQx_P4qEr*;!wAU*
zL`nlQkSE3`c*V#R(c?m)QUtu4&Z<$ZygXNd83%O1e#-(<LDoOo-dirt>*xA3guy>w
z847wcz^RTLp=?767~m!NVsHFtB9;s|&_vE-F<@-f#AgEkMFRskCcrwve)B@zIf3ad
zQ+3cgFJE@#@dWToFsQsQ9i>uX)>LUv9uau{o5Jb?FG>Lmhi-*bqB1SZ!>wo?rFbVm
zy`5z3)X_6@5PDwu9a05<y3Uz@8@B?!IE2!?F^k8XD7A{=A{QPC^<TKyCMU0Zx=u_3
zUdo<#?)e}T-d588a0-U}4iRMt&CAo^s|N#`^5);j)c$rqNJ3YHcu*f*g_)mNlmc-w
zuoelE@}owNdA<dRx4X8VRt$3e6l;u472T&aIh?qpDvOR>;B9juw?jKK3!X2Z<$=iw
z05YbN^%Bs7h>D79V|)5+?9;|A4onVtG{rT^c0?V#_sQyHm?_LCKF?Dv9`7v(DVh?;
zwu+GTaIEC6bKb&n20jQ@=-BL%z{D+3T7KP<*Kf=oh8KXX`n9<=tj*HR`Nq=2DLGI0
zv_C9%I{hsO((3L$ZL5-taqGOJlIpxsP6`~kaTFCdoTSV>MMXEPI6(<A{|jtVP@a2_
z6;%d9LMH)c9Tmr;G`B;3jHX@tHcW<*$@^Z$`M4YhFK?LL?i;<-)C+vo5=1ftElxi`
z^9TB4oRv=vj4a6Ge8674xuDM!Eq~M<8_s)}PeJ9|Phd=sSMX+KdQBbXx;8dx?R}Y2
zXAo&D%BT4Rcv}5i7OwfJ$>@JY3pa~;(brxX_`pz+-r(8s5Sv<4cka=UeA2MI$2xC)
ztcTm5if12%&wj0yn1N#8PvQLD{W`(4!-~j}C6Dwq(y*xhD4L+{@)e;d;h(6~1r-11
zoJL^C@d^Yu<ajMG?yi4@`rg>U8XBWx_SOh0()gsYHszJ2(+Js)i|}!hw(d}t3bPW#
z@zo~=q1^fo$dXTHo$K|ExLlUxBj{p4#JO3`L)3{eCwwAB%BwDw?S-dfe!Ft(bfBkN
zQ;us6?+G0NNHrc1SfvZh|LI|@M#m+x0ICkR%!Ur+1j$#Ma+b+Wx>x?tH8|`eY1vxW
zS#(;bhxbH$Iey#n^#8}yR|ZrSHSJ1=bfZY;p^@%7bf<JTlF}t04bmmuB_g17NH<6c
zNOwth-^KgA-@W%YoW1v|nOV=w^Nhry_ofOW4?FmR2pQ_ZVH4eEjT<Z|z;tvu;4sxq
zUcGf;f#~M_26(516DjzLIDo$)lnlF<JhUf#$NRljI`*oyJ2#VmDNBX|>^gREQSoRe
zkIbP)iB+WETB|K%iy^#OzK~`m=v~LxaH9-ibWzxP^kUxF?-EW(p(OI<xrdTR)ySyB
zLN)vFoEuH1$%;`?&1#O>@QY6<pYp}`6ztnpKGGC;S){=J%{*PUl~iW>I`sY=4;D}J
zje74kT~V?6b9$u2(bhZV8GwA~Z!GoYkUQAzDf9eTepz@T{Aufy(2nlBk@$AzC)Uc0
zYalt@waD>V`p3GG`;zwXKr&W~4qCX#^3T|jl5>q-DN344S$`%5gjg65Kz_pJ2Vn0c
zL_rC&-txSn=2~sPEkX)rj2vFy|5Obiv!yGOr8CQK<MNruoZgc@x1_aGBS*XY%QWwK
zl>G7kaynkT%q`s`br;J^OeOnoCL$>%5~T9fJ_|-c$eYhU8QijGaLT<Yd(8^_F6=9j
zyR2ee2q-o$_h|t3<v*5n#RQ^YCY24fNPOoz%ze<tlc*-W*V{}#`u0h`UBRM}5*?~}
zm6Uu^(f(y{ZeHZS85kD`zNYY#gjB@8>S)%MwFDcip1djSLfY?OwLgQ^ws55fI5QY1
zKZWdb#s>U<vWzfRxNy=-D6B~jyB>TQ94K{s1L^?nj|>+j@BoB+eed#&Fn%f1&`6_A
z4<9z=*52i2qA%wU)mFm+j`lyuyvbL=dlS=FR4$C=6D}qUOGQOuwGVy?7MvM+#rh(e
z|6RCZ4Ue?Jo_P@(W{KywTLNNHV03<}KVSAa><MS_Qu;aXgdm49IkkHPgFN`;Xcc;M
zO_0&+?Rg6kR{h_fI0mH={v-BhfUemnJb?QHw=z|4NH4ZBqayeTW@=xp+gbi~M0e*n
zGd*Qf3klfMg`|!6(@>vYoq3JfmdFzJ3we9)qc|@Y8SiROT^nGtd<F-v+bCAasU!1c
zT7tyqNEUVj<GnZ3H7sCP@xy-58S2n%dZv||B=j*~$upa-4&Z_6wUQS7xVTK+n9|JF
z2lZoHNc1^ee$$8Dgp7=gX2G6;o`d4g)9P?~ph7@>hL@KY><OYgm87B3Dk+V+Z-((F
zoX^v0yL#fOMMTpt@xI`ym|zWM^JQc?3%1|2imsOB!J6y|E(-6k2&XuxJRl`1lIOc8
z*48)Fk0g6Ud}&19v=%2droNOBo(~=!D7r$7r8bf^CG-&dGOFP@`!exKr<(`aBBW?N
zti2;)l;KSg0d~L2k~+QWf2KBCS+35-y#!3vzz}ARVu)%dd?A6uTLsRTVO(Xiy*DM-
z_cyF&R2<1O4`8*WsBtzow&pZiug0G96yMg^@S{InRz#1nvJ$Qol(Q)cABnq+&8^{U
zpc5;${|mMpkY!r92+J?``z7eExqLp>IX)u5mNd3ac;Xp7{I8!^S5AsxN3I1pVM*6H
z_JcfW%NiUq6`Nz;Ue~V$bN1P%hF>D`%e6s<f~i9IYHV<>t7ksv`fo>9{r9w}H~7T^
zebkwoP*!@T<Yv1#_DF!Cw&F<0*Em<Y9Fd7;c49m!D=4m~u2-=X@0KW#L!Pd!G0}?i
z{<GK!_^A(n0d~wv18dzV2qz9oF^;tfgFB`muzCF=KYw#lDh~)uDN82n+{2WSB9;iU
zWW9+`IwX8`u=U#dXjOg0qMgjS*7*XZV7rholD<MbuDGw6t?qS}{5CSM`VNJ7;VgWX
z*)@u>mMQ9%Twd=4tBc5U!Xb?-aB4!bH=R4@u(5GO@^rCdnHnf5qomn+&JkXlNf+cY
zlT@KJ`la|)q;Z5JIBJ#@Zd5GFyxSJ{ZZnz4#UD8;1`=tU;S7O;lr>M4E^@s$RkPP`
zt#B+c{ub<E{6PxF0v&mR+F4;k$rtC0HYqWUHGf!q??u)b1#jZ=gG;!Lf-0=u?m}$7
z?F*cD7J29PS1UyF+YsgA@9r0u5pwz4>qe>JYR8=-ju-Qi8J|p_O_XdVW+1Ue26}=-
z?ci?C$otdW(pS-G!{Se~8dmLW6=NSPb~jen$Vd}4C3WD%zCaws&!z>_{dX;r_|<qs
zm8zNYUS^eKzJWxRQ^*byN^zX}QsN#6!BJnq|Mj{?qBB=Pq>{l!e7r}J@7Y{sfiHv+
zRpk==A?WX9TC8VW7?XkdZt!o<<XcXbY#lk`P;z87+?`BJ)kfI$QgNDvU1^YkC8*Zv
z@gE=fEZp%8W)-`PX;2^Drqi=H#hYC&`8cgTDSQl9x0|5SH!-bWb>c0sZEBygS)=7x
zk(CD&J6-9qEY=znZ<*`={bmvd{w^l&>**Or`u;^i(j}f89)1uxI?DaDh`P{WyvH{j
z!$T_c9RxD{otE^y>amA+vLt)@G%(N*a+zSUZC~X64Uc9#dlG}BU!Wx8)eGIWex_pU
z*;bk5a}Gf%1RHBr=T8hI!a?Mb3I`5_ur3qG#sHh&kEJW-ApsL*hLftM0V$*}Z^5&Q
ze{z5q!%Lm`UXirC&zjJUt<yu;XCg42D_S8}Mvp&%r&+U()czhhh@QnFkC35#hAS|I
zRqbz<MmVS>pg>lm`H6#nuBO`P2@93L8NjnfsMcIAQP~?rj6@ZSgOnwgQl<s%f8!%0
zsaZP-#Y6w3=Q4nj5xvg6o77Yjm@kf1C<ih~?dLVbtfpk%+iN%e=snF&qn+9(juL$T
z?_XHTJ?r{;o19h-jz30o6tymtm_!WWwxRXwRiqN-pQkDnlw!e`C47m3Brtd$=G5!c
z#RETv##!tzKqI9ZlGKC)4rHcUyO3pvCGxMUdBvMmdVVPB{5So%#dA;3lGUdoTbA-!
z!Mryyc0>_+nuV+2o~RV~wE8oi3@dl`>BVMP!HzkfgCqB$@npe%OXyQ-`7}GE5GG;}
zAKv^pPq}C+85d7;6Wk1drcn?Y6gor*-9R?g@vJclHX+u&|IIkV5FuU0NJn2jD5e?@
z*dGYQk*JTCGias)bjd1vGK*mM7`lxbmPmWZXzU5lfq#J@L7~qAQsy)e{_S8Jh4F+L
zPC?OWm4=cedmcHMW9$L=|MxVV2uKQz24;W8r6A%hWRR)Qfk+aMq)u)I*;suKP;ARx
zR5YmRAy6@`{8M?x=Dmd0dC57)v&oO}mPvEt4gR}DNll;*u$%<kOd{n}<C04|T?S(c
zUM`0`*Go;P{C}V084U{fB-Q_v6jH^!`2Qa){t62egf14BVYZ|tH6;MJCKHrw3num}
z6_mubn~9diwtc<1>-IAH)>1sR320UM<hZI{#VKv*LmrB~Lw>J9@3I^9&=QyF9}KJ+
zz#B_Cpj`Wvj#2J!LUVj2%P^NbHp4y}JPA2vsa4_I1<Cg<bMJ6CFR9Hi4Su}^q6&EC
zlo0f%mdc|<gB|*EL~<78hMqY8C5pH7xMMl@AmHDmPbt+xI(-y!THhyFX00hux11&e
z-|_o8Ke8WN-(~o(xoB+fHVrp?n~g=^*Gac7URe^sl`9l5(ecTpe?BEK|4<_G)2TSR
zm1xyI2)!NVS*i$@3vZOqoa|j@C;EP|vCQa!zTj?||791a<cGx}Uu(*HI}z(CZKHri
zvnAa$u7jd8$P=Y$JosKpzT57#hzP;~IEkyFsy7;ruh8avQ#lfHpPDjWSM3@6uF-z`
zA?)P%sh0J<cJ_>oz&;+k@dJLV528cDK6OL+0q=G=e`EHmiR=6t#TC*YG6Ie{vSn0m
zR6z_*3L6NF4-!JcNKj2a4(#bK-%t6Df^NSqKFz32!_1GOE%`J)ZLP0lqj)S=9S<0-
zj*xnvV<=o21UMZpF+Tlk6uh4*Xp}NxY&k_d5L~tyj^0}Ko%HB3N#l|q3veUXKGg)}
zP?6m{kfJ!f(_Z17A5G{^dSgA^LY;2~WoIZn2X?iSIwmm$)$5NhKIa0e_cD)D@QJn?
ziHVL-UtEL_DXtmdiA8x9EybS;+<xokN4YFZk6ykA*Vjb1@l^#?o<>O98B*)$3|PC*
z+tLPzo}`GlXEk~5#5X5=AQic5&iRXzue+9naeLGF#Acl#Di2xhL?EaR7=HK1X6lEI
zxBuBlK|l^${hdC394=ZY%$WjJD!<jQnTGu;ttn{@;>Jt_IzybqH(Oa*UORk0BqMg2
zV%ctaQGVXnp47$LdCC#!&2t;C%8raijY?FHd{s}p6khV4H2$>bXG%l?`Tg!{Gc`ia
zFK&Rnyg`uS@X8J=S19Cjq?#lEXOgMVRs9|ouMUu6_8P4dsYR|{pDGS%eJU-sY5G9Q
zQ*oC$T|s@D=Ul)1^WSf;4FtDi^%)y^<3qIfW128IoC*NF7+1-se9^byv)#+U64(~W
zv^3#1YKl+P9FnM1o<vIyDHJ<aWSw^$b?S9hD+e$mAIIDU&t<X0KB?cWBR_=CI8d`Q
z*1#KM)*EQK-N#XRG>sX&weD+l=D8?tcQLZhsi956Gy{M~z6lfoZP}ZZr=6bfZPi)+
zOQ1V&-Zvo+Q;kO3Y(3J$v#cy6Ma&c-9yLhybBY0Q{4|1{<{0d{1e)FoPZgj(j}})o
zrY#@ANL}*hMAdh<t0libhe#-u$D=vLtGuEY=BW{9{K{!(V~0^^iUr1Kk>y&P4W)<<
z4fxq<D2mC0MLf$2f-T!^&a@RTAh=hNu~5yBwMdAtvgaW8dM@;f@~frTZ*A}vrzOHs
z)5!f%;``b#!0?z;mPcSJ%wv*_OIuXQHGCvtGZ;DuQwsP60I4~1=SC#gjXA3~h_Nt%
z9n|5r{dx2B<*z1I^7jf{7Wu-p`k5D9$F7@YrAdjrzQ1f-!X7Yl7tD<=6b%PFWvAY}
zZMZ+n)Ym>2KLJNEFZwG=3<HFa-!aPMl%|9U1T4%TVEGbge{FKhrgEeCyhYQ)ZNI6$
z8-u<Uh{>*-jnBi;S}mXMDEbZgxi|HGO`;aR4IIgr+DI;%ogonH{Kg{*pJAh`{og3~
zT|shEMick9cPVEh`6evYTcNN}sG5XoE8EooE2ONk^dsP~zk3VQ61m3=H`xE>cUr#p
zVUEWv#~U)>53WnjZ0uXy%jI5aYHTj)c^&G>3Dt%Bij={@l4+KvdO1)qijz$P84f_l
z$${%KX8DSac-&f-5G4c43R4>nL;0?~-ZnzuYx`+O4p_**l-^}->%{_PvgC)CUvac<
z2o=4}Pt@z%Z%hB_{Y&GYC$~O+v;#KW&^)JRnAvKh(W)JdNZ4uH>zxYCR}$)({E@Mz
ze{Btg;x3zZRFWI=Q@D8evRwHtC`$0E2W@&3<!8F?jW<2QTm;FKhquiKqb>U#X#MmB
zQ}McDRXe4>m5qu|BsDjhcDD}Wfkci#%TrZEZw0j#Ox=}L?9S7EW_18o;^tEM(yNyN
zca0W91k1joK`4uhIb-bs#ve+I{CR6_kQ;GiXEFAb-{05oS5=PMlXe19|LsDdd!^pj
z%ttz+&m1*sz*!g<8Fgr&_Z_;<_&%LgE&s5eHu+tOZI#1%Wgh6MbxRN}c-1nwvn%pp
z=X_E5z5k(zegEn_qw$|U1kFwawwp<^0KYSd3KKdqO(x0{$OQ~owdB&rMPE9;#2y9~
z9p}dj9$ZXT!4nQXMpg))&uJT7Qam1>wnoGa<%_yn5V^XFs&=H`rv}=$oZJ)IXs8-2
zvfs};eu10ui0QA^q2rs`JNz2p{lM|Q`>2~beDJI%h+D*;r{<|@rSO@Z1Z1b=h3Y*8
zbC#Q?Jr)HXy4@F!y`r{?+MIw!81Cc2#<{0q+oaUUByNXO6rEe@|F|%>5nX&0g|e!1
zR{gD%hvgV*N=Z9$Ic?9{)FyKlx6yZ79s2nFnm4ng9Zx1Fo7s)>25-S%m|U$o)gahU
z2RnM3Mraj5MUXx8dN?dF<g(JX9G4GE&M}>93Gj%un|-eLxog<j-3)8*n~IlhZ=DA^
zo(C=SfHK4)Vwu=>?IzA7?D#0(FZ!&|w?E$ab11Pf!RMeHXZpJ1jTyRDTo-j<ojp*)
zIPkYM@pDZ1mC>Yp)RD(-!2d({?K*4mqIX~{q@PJQz^nY`)!_GXYbR9(7J5j69F)uH
zfmuDazXkR%hDXW#r2DPBlv10A`RUZdSpV8KeSWFwyUwflkR{KJc7N*4sDCyKCRbG1
zceUROZYepirkiwH0VB=@Q1wxcv=dzS$;*AEopXM8`5&ppK4Cb(@YU_RGat-UNjLXN
z_XZVFTTKHsH535EQkH*{XAvJW-y%+r*E2pOMo8S~S9&?tUW4+n$qxDbkzZoFmSKo#
ztVL?9AbWt#Zu8BpfN2_K<esASqtZHF0li>Mz9GEjsq(zV=%4T?<CJyMDOO52Rhz8r
z22ayiAQ{&tC`7SrL&aW?0f?>@p=~;U&#wLe*(oTG4l@HaV6Ya)>~L5OlP&GNG_IL_
zU%Ot=2zmQd(;y_j!I)tX5cx7P5?)eMt@kQQiTx}b-rm<turq-KcJ139@8jK^pUquW
zvzKmHF9W?WIqft%8)@(SZBJ7OZ5vQ+X+EQ1m%@g+{mO*lrClp3yPy=5K8Da6(ZS~k
zXi}MzLlR$4Dmn}kO*B8WI4}Bg65xQDgOJZ#E?)Ndwcp$_c2zvHw+3zj>+JT*+Qaui
z1jv1{$&wV<Ua7zQ31{{RxVxZy%5)zEI9>V#z*Gi=bRmo7#B$C_i-8_lpPa9}C9SKG
zq|rtmIv(S)BPM!B4{VCQE4KHC(sr!vZ}PmK`gZC1i}n<#fO=PfeQ(t4{2?DUpQnv&
ztIRNAh()UO_S$&aD~i{u7IR+(ey%P9?F@q@VWuuy=HW^?jrOO%w(BiRVSmsl$N5Ae
zY?`x)5M6>0GI$*8KDc0}v^Jz!=G8d3SD64~)XJ6)3O%1|!Bwr{RYNRObE=2FN3trd
z-D^wQLYd@=5)70WD5t4jtKv7m#|UV)2Xa~Kh;E#1>{&-Ye?nB<9uE56$lqoyowE)!
z52?EzbSuH`voZZS-(R{+<z5C1VpMrCY%MPDUsZNi7m>dWq|hW(pEU^bfAa#Vwz5bQ
z69G*R66=Kw2R@7lec*`@(<4wJ%CRx8XV10|qV9T$zkQwmCh|hd2XG0O{U+~wB2F@6
zrCV6afi7U>!N?+2gf@(d1~67d^qikd<8HAA@JhYdwObW`gfoJBe7RUkzFfV2^Ly3h
zJU@OxevZq<<T5v0{AC}zUlmXw62%jsUyoPI#dLQ8sM<tV%5-dGd{VUeYAB3GdtZd~
zM2siQqMYl`=@$G%3?{wuG+`}f<=TV1id7SHbi2|{JkVC6_>LsBJZl-ckS%v_M?DX!
zhl3e(^l|n1``FCy4?HUFqf&ipXhq82;fE0a%}$7#LVdm~yjM-{#q%mm>R!+C-;=|U
zHpek&=;OSIbmg`nP!p=b!#DPJ?VS~S)4^WC{iw&I3bRtK`VjE`&g^CE_Dcmv&CFtF
z>x*A_lhYd?dkK@}(u~`5&ci2_eVCV*-2Sd9nOWWo`zytIK_0kC(~I`CPpofZXO6#}
zvVO_6oUmF9T&5d6;5BvGcCKw6G2{}x;Or0e^hXL2A^iH}x8@t^meOeP#OG)*WF)}q
zESFQXz1-w;^he11iQAN;g?=D+k&uS44FCBCR-T`~8MW0C?X!{f`g4iuR5E<sP9xbq
zh;yvuP@advL4ZH++;zh&ap&g9(=fx@+EiDT0Yrn!F)?Cn?$o0g1wyMb7GM4;aCQmW
zNY;#DE$_J;c;F`4<F(yuNBV|qFeUdSZe7VAn>F|CN@S&+i^4)BGw)#B|2R#gm?izT
z)1k-_29I!VJg@4N`~9O_ySU4H>YOM!8hzenrma4ffJ@y*S50!q%Vtrm>EC8osXiMC
zW_$m+7G{?MR6<Z)K6Ejb^D|4d)p~@jTOUvBFbWyJrjI!j<k!(=eo>;2@8Uh6bAOpu
z)Sz5`?;Oh5<aX-}1X}`Z&K`iLzUL&A$FMe66%*jvI6la^!ys;Xy4@2c)iJ`=5+0+@
zpFUb_9?@-_;==t>F6P7Xc&wKizik+2R=(}3T8T#g#p&pwUZgJjy5CpPnlG}P_E|-Q
zqP*L8oGbNTvZHmIjVIw>e(`M$Q3gsb$;*6LP9?>ii9F3LDD=Ch+7-|sv3Z=<{+(Z>
zS1vPZYp%-dCU?YPVX|FaTWr^;W?6|!Puj3<e|Ds+F!MnNSD(XpzAWVpKc>jVC(qJ8
z!W(@J`NbOJkmaWZlSXaQG)$-_?jA$ABi)-Dn_iT)I)8X0WnQ%8KW~+A5|7<7bm-dC
zJ<~{D`10IX12wR4y(g|5r=9J&5?SM8O!e3fIqJR{nB53SHuK-}&yQ2F_&wbZ{%$Y|
zWC@z2@xfL_GuVw-_+8?gK!DOaa_=C70m%DJH$Gsx+q%-%zS7Z9&z}21##ysAM}kq7
zq+=(3GilW5H(4`%oQMe=df%e|%;k}o&{p}dADJAn4nt0qZMMw@>qD5^E3TL{N!963
z-jTLjdE1A15!+k`{c>c<`*r@+ZK?y^g?iSe;;&q%+__v{e1EKQpsf(?W5H@WtUFJ*
z_Bb@N-fgVE&^M;pSZSg-NMxW)+q_nYY%;;t`?2BH$#<s!>=H!`-LnQ4&K0wL7`la_
z9s`d6;ccnQBKJ0*J^hC{NrIu)`kFV45Uv&-qx5YpES%#T-13!8IidOu0SEe<^OTR%
zDrBmP4UYG&%M{a5wjcM(EF9+S#0Ps9{~YnUKxQ{%%c4Txw2R1#e4i*#hsRT`#8^}G
zw*FJ&*QZK26@f}o?%4LU3hs&iGh0?cOLmvQFCj)oiYrR?Fm5jUZQpL6GD=#kW%se=
z#_64~8KuDkt#!G8tN@I^Wk9P*kp0Be9N0Nuq;KuApga<ko-8GNbLiec{j)>+o=d?!
zZ=Da%t?)#$=-rgO##J;RYd`8KeJstbf#iBdxb{Z#7<MTOA!3U8Ot}rN5mznHq*CUE
zqY{w(IMc5fT_kwd{an!oSC^GtN!IY2Hr_bh2ZOyH=#9}&NouS{5Q{lSGu-9iB){=k
zDly-rv8a(z4gArT^#Q)^(Q<9|(sitb?dl{(PRD<Xi@nW3MpnbwKF@&&$)}9}>=&cQ
zy1?!Jif@a&ch)nM<tNE?6QOw*NL5Za{hP`|+jH|l8<WA{a6s2Qk@qBUfNV0=jfBsl
zO^1^&6$CS0wDF6URsRGX2%Iq}J~ltvic^K}b@*JZT>r3h%=G@&0DF)GguC8}H)+E^
zxT<2?H(L-%4bA$ErG1@(u?dGPKiv<BUMgJh`l@(mRexr!3ui{{nWvqG0}u?jdcUk(
z3m_9I*-wJjF5RkFR$^?R^SNvTL?(Dx3(1+mHL!?<7+?_hU5*C5OP=!S2r`j!G)W>i
z5f=L*cNh!)DI_3?g%U$vE?;%P#q?nRV7f)3DRTXLxRX(?t8<1XO)I%HC37MO!~O|K
z<*OkTQj~ufdpjSk#{HVsj7P=FRHqZ%*y{>`57gwlz@FaAO$Zbits~=gxh|8g2nI}g
z;2rWbJ6YjEJxI9T#~UQbHDI+A;b<vgyhv%>MC0?l12|OQvNxR$T^!ytNBzG5%q8o^
zha;OWqZ@$frf5x~syO#_-fEZDXjLd-yufU*?Uw3RfjvFw`Zo_{_mmbAcy#F$_I8~V
z+8#y!tNHgI7y#Y2#7hT*fo0x?qo$c*O}UO2ixe}5^jb%#ztYe`IJIm$dGwk727HRL
z%3GZ`8qR*xrDuCxpDVys#dEuBLvFc~Vay&_&Xf7?5k?CaJTvrP`vj9eQ<caqIimcJ
znF3Mj9zT3L+Z9Dl(yYe;`nK^asD?p`zF>g^?wGNzx3uE?(}&)U?dkBnZdXR2jsAKK
z3Gycfkx(;z>@#_oydK&Pz<Ra3E1i`Y<}4D{lDfML%O9Viy6k*iTs3ZkNsC{9)LXUw
z?bP4we0*PmWG^&lb<ifVUG>^e#}2q1=a)qw0x(f%SL{P9%?@IJ;30eR=@~T~s*i0g
zd-*<n0~G}n49M*tAH930<pSVETEHav)uVm+je-;C*&J+V+3?)qeUOirP1n`(@77n-
z2<BOub02cL#J;GjRnnv>&#}x*bJbWJ75URG<U5hw*ZpcwDmbcq!NaF%%wqrAbg+tB
zP3Sxb)7h&NZYx^T(dKgmIb`oeA&@zG9tG?#&6i)-#Gt`f1_#ZnDo>SkQkh<IyrJ;+
z@G;|QcKpHa1|QjTE3iv)hJE)X&h>KM9;PgnMtzC0-_31Wr6c>fe|PlBM=Aj>g_i^1
zjAplXSDG+A2^Y5?^KUcL!?O1ULf+j;8cjRH+oXF*@+0;t&s}{=10|Itm@1N*U`*#_
zJ$cewgg^f=wdCjS>jpsUyIN{N3`Af<Qjj1h1nlY$-|>Cp%Y{^c3|iYuy$1Q}O+Xb=
zv&goYGLqK}crEOe#}&g-(IKYlrv#L*tVsQx*DGs7L#jfHSbeMSzjE_kUB4abDf#ht
zgilfH<Kw;qsgG@$_vDMVM~hKnX1f5KcfL#PF~Gvee6E5+XM_~r35)Ej4}aWGTDIGW
zqTYx)^kzG1cuRFGMT!m&fEZ8^keKt%jfA~uCNe9j31kKsP|<;=PtF_nV#0bs@2nNS
z)Ki}rUu^FVB<sQ#{G?g7<j<(lQNFkRxa-Ll%9SF$n+;Gw!viIC#NUt(cS|cZCkHT_
z<ZfD}$|<s$Pfe3(*UIkEf%4e%%VM5m5o=M^8@(L<f|Gx(<C=m}ny%LN@dmISsm_ok
z-#RdMO$>13d5&>&OYKKkNCQp;`~1iv%R_(jk?ALTH7lrTx5!4D=a;@B`oF-vcvrT9
z7&w6xCUct}3$C<+pWPMKKzi)r8AYXs;D9I;<bfy%s-g2S>Pw%t)+e!xo`I<Uf!6cH
zB$M}i`}_73^4CpFaiE$KIJ5c4#V>LuSpknS8Q{G@3k?;W5@J@sNN6N{@>5t*A-^>4
zFHUNgkJ;f=!0#11%qg>4kvZ(W*BNa`*Edi`jU}CCxj*edDCK|O@tAMgbR<x9a~|DW
zOLFBWi2&6UqR_ebAsx<em~-};m5+!RfcRhj*_d~17jn6@{a_it!4&IsJ9wc7x(qy=
zW`&qP00Ghe%`NP)LhL5x!?NvmUK<huS#bz`xox;rltc5jy{~ites`~33CL)~ApOr&
zf?Wo~U1C4XLL)#%$w*&b9&YvWXVC2HrEyk1i&Y718ZlKaQVV|{`=I3^g8kg9Ug6t6
zVxdXiR9#o|(C3s(59$2j(!!$kLPIrSE?!Y!ldj)W5o)VDq2~e1Q-A=d`YI1fVVywk
z5B@wDGFmXK28KO|uq>~6cHXT~%LPE6B%Sd5xP!Xo=~MlNMQ1sKCHk{Dilk#)l*sZ|
z_iqPbIj*n&nLSI<iUYOT?<%{?1&fW$60Qd`>RXhkFv}ic)#KC2S8hjk70I4Qg~Q2=
zod>aeX$Hm(B!yF|8acSx>KKuowRGh`-B>co)1JLuXg$O6%hSY`50G4bnaQJuNPddB
zcoVl*FjE=Z$lP2(H>RlBc=Y*o71^QB<;2)6o&`Z<B<#7evhTaG`*$!vKtTr?CQ>e9
zh4;N+KF7}fmJyfTpMT5k^UWd|MthAR9EkB&@NM%y&mGZQjHmzQdW^kwx8=9pDEn|K
z)wCbzhWWv8XVIkbsAyPRRTegdBbGtGG}&p9i;gW(IXw!i+tQT_y%N+{0G+YZpBd88
zS;xIoVl|IY!0in0_jujqJF?OqVDqRMQc8ZOg%-}EPjTK6tceQOwfU?3&M|;@4I}^d
z@cZ^H62Fea9fR2hympi;-INjhUzK@=7f>2mQSZ4yo*!^qHcQcJgG`@3^{(kPAw5hr
zjcgepQBA(2w$ky2A2X<N`4d#61jC+i!3wBP(sB#+yux4P=_C6wncKaboX|+Ws^Ja#
z)r<K1RL$t<feN8G`IIBHD%H>d6#qXZZ-!aTwLy&|K1$YEhhVXFJ8>6RdYHDNrZTdS
z$J!4agQg27wCD({lnK84LSK^)S|=lYUkZAam)x$+hf6%ZUQ?>!;WPFq5_w{k!RcVk
zD?5@x+^3!<tEBbLe9o@@@92ZE`92-Gm=3zFJ7WH)c3N&T&>?-<e^7?6g!-Of^GcUb
zwo`B952AHB(?am$&dCf%atzShc|1u6?J_XAE_xNxYMgpr*;TO(0=*ijDur9`TMjSK
z(8g3u)_gj&660|kC~1(m400Dt-jGkI!NWO**&(evrmV&PBDi5Yo{IPj!<<y9d>!eg
z$N#u=vG3RxI5GqJ7bqFD9pI2C6YJmtPAC#{)xzxfU4~;j%?5QkAkk#Ed|;%Ek<fvU
zXS|;n3ax<p`*ep<mTUDrq}WW-2U{Swt^_N|JW?I$w^WOBfzZ?tX*<>%i@1X+EQiQ~
zMGZZ|UQ9vDnU_k0DfvxG0%5u#R;qeJ5^FA61qwZ+5$^8>934ye^tsKmVv$JGlh=AY
zq6N0(w|6C}Q<q6k+Bv$Y*L!hSALmVI$3v1%YJx^@zN;{<rjs!y=%l_6<G@O0{Egh7
zam}ea!pIX5+*f1{gisEh(8AL>?6Th}<`hGtjslMUz8@Q(6dp$#R0|h;k?{xsxBz_&
z2mkQDO4#=<w8ml1s1^+AqC<qcA%qIPIO}rTtF#S}m8!_OSI((fwUoa%IS`;UM!!!7
zFDmr~+#dlks7m1eZ?*E17-*dFcvjV3e<<`}^-q_y?rjsnC`-o)yUulkofo`h5#i{Z
zsf<y%qTc-%?kkJ?9BqjvBMee+R@2{E$89{Fw~J^QR~*8;@|Vqj0Wx?X#W15^lV_7V
zEG~0DIb;~NfwW4aUV{i;PW}BFZ0~J{(nbH<y`n$_YuWiJhu`g8ABIW9)lW@p8)a`i
z*9jVFd83ZL4A&}e{GDvzKIs{9v$nfP0BZuGyx-{SE$Ew~LkJ1EOw-q*(iX1_@ZwZM
z52m)Mze8X6#;37H_UO?J0FLAhtRVcN+B=9qAdT=4Jw?6!{RXi|j-5n2Nhm$4)xq0u
zyjEeWbEuJU^n+|QqEzIp<Z!PUya1l_f<(~Irt7lx(qA=+pYmyz8gG8Q!Nm=3@>(!Y
zS}sBxvZR-XgEK2`&3Tp1KFVi)yS{k8fryt3YrIP;`2I3T`>4`}=^huejP*JF`F>(q
z+TX?T^`~Eyb=Ik}5MvSp<ca)Ys5wHU98CBSgHD_KWQbnu{rX#mS}ZTbS7pZ}f%<PY
z!c9<qN}O8RbT_W8KSCtl#`#S_40xgwKk~>X(1GlD$4j9}MS;uq`YRx>;ngeB`cG4(
zjal)t()b6ZwF5ReKFchnGAekn^!pViVP{$dN)eDP51zGO0&QFzgvL8@C-IR9y$7V=
zZr{X`$Qgvb>&35}ly_0+R34{;w><WnJk|+61=5oy4ZcUahXqy&nZAj$o~u!&#w3~z
zy9TFbWCTSl62^KYnGq%A{^Hi|H^g_*L!&yvHlCNCGS!Moi9*P*8Y!7*JT7jT!;}n=
zl?IE4#WxaXI{tMLpml}A${Em7YYW<nNitAYGj-@RNAHfAElImr+f!JuP||HU^`r#(
zQbL^3sg%bjCICq$F+M(Ch8ic8&4r}C8@7OrfVo~$kh%F+A5GxEnIc2?Sd6*cPza_F
zN1sUKryMk2=5MAO0`w3sWfPNCz>LBq;f>4AW??be;1!RTdGRz5NWv@F0T(;`CRCO$
z#Mc?+Y!&W)AHD>?XEP}&rJw_!7W@{|{%m1kK~#g)umvlXE$vk|H_OvUof-!$+!h8F
z6X8(S>+8Nxo~Q*cKu99$DM<Rxc6&4{J3D&~N6JuQh}nc-pe=ZSVa>?hn50%=fQ1P+
zHA%qcrYKhMZO)x}R0%sOK3=jZ;DRhPx!HlfXR4~I;Fph}ZcW4@(H6IhKzWFzXp?ez
zLp-(+_8$rP@l2PFUN54Mt^&8RekZ*K*ap~nDnz8D{(Dnpes`A^=H}rQ9CFUG>B<_X
z7d)e&8=1}rHCK>(m53dVUg%7}!P!oPSeVv&&7<KeQk`{Aa-i2MywitZDbqXuQqz7j
z$-Pe(R5uh-`aT2=EWMhV+S=+WOn#HgW@H0()Un(ROUDm<E2E=UN0)<`#Ny;9Ke+m7
z!Vk1Ze)$fE86zw>vFd}o-J3^Tc>=wz-IFykHjQjK+N101H}-4h_$eLjQR=W^tfKM^
zpdVahQfH_5%a>8Wk6T+?|7y={_}jp+nJPVA%gN;1N|l&?13j)V8fJ7wmnHdrxOHt1
zt1p4uv!29p>3lPjIRQsh^W&$H?W%nHzqk(;M-D^ql=%TT$WU^}A4%d!&>~|Kj&y#0
zegHnV`QEs=yVu?8>;L>WT=9J2cSFWeX0%Y7Eix{%pUc89)PS@YkMTOB<75_3RWe%-
zye<31%`L**t^p<)gieu!*GO;K*}>}Yx$S9qdh+J5-nkAp6y0xH8P%t|o&J3PvEfrZ
z^xxWPeM<rHr!4}8lW_*6h+iCE{JJSO1Enn~<uA1j&GtZvggD^uGLe%T|M{weghBHC
z#}B|z#YQ%7G&zlxCH}FhmD)q7xIooSvnk)GlBBAlrB6E)Ea74Bmej{`BH3sEA68EL
zgWnp$-H%eHh>T-nJP=Mp(bE{f$7yQ1Z4M@bje2}}IXXNH9~fc9V>*$Z*?JWdpy%Xp
zfshpkzuD?1#b5HT$jJSxg7U{pWoFaw^2zrLb=oj|w-k-XV-(rF^!`8*mkcZm;9Pw(
zY}Hj%9GRZxqNSzf;^JatjOsD%Xt!~$#=|U+lFXm7zvNS#UnVr#X+JD(w4$z3@H+Zq
zMy6LiKN4j~pY5=4d@XMA2iJH1>zYg3i>g<4UqE0RKRY+~9&p3@<0xFWhWP*u1-xl$
zdKyqxD=RA#6B2&u9gi)1v@-nlW$=1amiX(1_Jx1CtM|XrFoGr35d9)yr^?+_zur*n
zbeiDLsJir4ufj&uC$v!8>@2aLs;vm9+BI?)7R}X1=<<UCOeOPVb7^$n=?eK>-k+z}
z>eav3tG1oqU{DX(pR(C3*Nn{(1QZp9b?y6cS)FnWC;1yg?15HIhkLe*>lWNEfzHu5
z{ykc1>3sI_rwcp+S?W?{@~;DrB@_r8Kj}8ODfp}E^n~h`t-OS$hgi^6s)5T6s?vH2
z@~f5gn0~l~0B2vy@*fG9pj40|*5@q3`zkvihDhKsCgi<fMH)wL9W{PR2K>0sx5mej
zZYm_h+1?s?2Dqfeur>uoT~+bGIJH#n2xw>H7w0=B;#DSL3#~*tP@=80IjRzyb<D}n
zS<jR#{pKS*i$X+jG^kl@-GG}o2Rc7HnG<RDqxe7jsCuI_TAixPYNcJoh%MfYILC}-
zd#-q8=#(h!H*^OVc}~F$qO`tC8pQ8lB`RXk#NiDZVo85mK=Cj;UYTZaO%5z=ap7{b
zR8eAp6A4p{(#K17!Rcc{R#0g6y;}WLHd!U&OOTu{O{{|3SsTf+(^Q}C=RFa)UHuhj
zP`@x<cGLfO3^E-%9M%@eQDKv)#*~kby7CfH8PACP<CFX-^vesNR6$EAq&?Jcx|Fcl
z$7+aDY^Ad0kA3wiGdcAlz`*J7*2i&>$zscZx{o1Uo-rwqIX6CN=KY3}Qt0of5X+Mg
zq)H-f{aX6kxMH9`G#4?(>}}>yQDZURwr;i1X-e-}(ML&bu~Ub{TugFM|9IIAH^#8z
z-J{yeh~hJ>mW{X}wUw;CHvv*+?O)aMD^y6=cwsB!T7KTjHjk)hsWl`uJ{P}-_~vdp
zD;l)8lkquU|AQW-Q4cQIB9F0n?_=L}a~G=lCg>!c48E8(J*~BF4ZoM#FU^f|QD9lJ
z%ijA^uK(IjUU!_YFM8i$ZnR8)Q;@bqFt;=gRV`{L=FOt!6{(>6s;wk{A_rj^1rVgD
zi+}<_3V{-**9V^dqV99h?O|rAfxJfF!LP+J0a|4SYR*ndi^>S%cnEm-l(d1GUvHM|
zAL_E|b8J;CDY&RBH-fn;cYa8C7w}PKQ~gFZt(+(#?!i;B<dWTiiU-qwW(-cHkCONp
z{uR_7K4GWFdni9lCul-$xj!fA$SA&0h1L%FamelT#QY$RSx5drHJh6G*5sv9`t*5t
zKV$oLR_!$XjR78An`k3OKLap98@Rsti5NMp4If6Da{uPQhxfzl=CLWDxZlzczN~7p
zWcGa%(8D#vQ;UZw#{iNypCZ*&Hgu=o5FVBSmdgd3XTGszx`&jkT!FoW4MHa}fk<`%
zUJMk7L5K&6LPbt2oDB7aFV*CbB2n|iht}J;t4X-M?a7&AI9`^87K*P5Id19nMf2_s
z!RcF*YF`VI&t!EJ440#Nn|R4c1)3=oLuDfM#4#+s`6gVRkdkeT;l7f#FH8aj6>2E{
zCkz?x|K{ATJzXfW)il*tSRXW-BDS;Mz82FECN63oQFG5IzP3zJv-DFE_t$aa@8e|y
z5Y-}<=1<VUf<>@Gf|x4oV`Aq!->6UG>qHn91o`%0ZRKQ`33!p9kJHUn5~4&vAt}li
z<Nko`%g~WC*Q`*b<lle`2?4hr8rKT!R$8lDpn7IdbqzMWPi5n{_f{D|-|n4nQEmt|
zU8?)vwr+ftxcx+@3IszKT(K?SpdlJc8U%O8=`Y`IGl!2c7Y>UH)0N3sztUzq_lK_(
z-LqP5EOzn8C|S5N`LmJjKdx|?P4}m(PDI~h{ec;GS-mO?K<3=C(25J3l8Qc}K1h$v
zmCObB>_HQ}m%gTnk2~X&e7qb*r6-4og$>lM&}<u;7^f#s&xoX0537<XMg-l~BvY8y
z?}Cjoak@-X&ar^)6H<Hq4b6eGndB(=GdR_Sa%-+pf65MLeeN|hXfwc4;oXQ#J@ry)
zIY6O|cr8)G9&#+9Rm3f^W3poM6l7wm6q>M#$<mVgIgfA;xN0C8mL<|x4LZNNhPi5W
z{XlhEFze>%w5g@xP}VYXlt^ILXBSre`oP6TQWfmu>3Fq%GC|)BiAo-JKF=-9<Zc)E
zzj^;~>vog8yYg{+vrCAP^2^5-g8@%~wi<zPdBQ!ts3mGma%=Ta_!AJxxPb-6$Ka=b
ztuj-VnzW4XTlxA7>gCg<KRvQGG-UT2@`TnsS+2v#{pB<3cAM&8bI_0*r)snWlc_x)
zftL;N(lHsFJyShVEybvRnOfQ+KXu^hjH8ychMy=@?f$I)i6Ci_uJU0jbhUS-2ybSe
zEiE$~5<?ehnc9wF(MO0p;fYUIKu7qpI$j*$CLz9ovm~QabFQ%l4X64u$JtGPdIT^r
zavF~%rA7E(j(sovs6{^&J8X24LOV^xD8`4+W0g-hePxag8s?bB|7JmH$*5bTrU~ll
zdrKP!-~{nf>hF<nU^_X{6&;0CMSLDc0~4+8(1@lyR&pt*qyQ1Ge)7oRqmZ<6ulagd
zVNfAup7bkaRyBx74Bj_-w!5LL>{31sLVV9f!8YRzY`<pN8cP|{12btTaWFZaiUxA>
zto`TEz=0hUTyVL3xJzMfe{On`vvt(=0NNJjGlg*S=8A@yAA0pgBmSE<XL%-8I@@<R
z&vT|~*c>oGzmNjUAZJ_si-y{J>psMk)h~tEbvvWkTRlZhDOUv9>ZHiI`NO*s#^B~Y
zCbukqoJy6r%ejwD6u7EI{{O&rNG`mYoWqk+&<6H7LtKah#O`{PNVA$h;8(~}GPgbX
z-#toDoGT0heM?u=t3XWkSFypUCa0{8<}AM>YxD6{EvLhJ2{pw7f3Z|w=ezd0)!Vnf
zS`a4`1LdFK#3{HxR=?ENeLRM<m@xu=T<G^yF`Aar4z#n+e;A=^eUVrn)X%ObbuQX`
zF(q33m0FdKKo`4o;km4CCublhdUh`SrB*k!?jBuHAUdlMK9)kN5GE3}Ea7^glc9FY
z7s%U~O8X6O5Z5jVG^j~I=p4|NR5N~bajSXRYD2Uo88xQv{siZQv##jPI>vkdj$HjL
zT^Q&6F<-o#I)*N&t=w#Tx!rat^ScI6Oz<ET#E~&m0Z(!4Jga|AC($QbHiLAmZkiWo
z*dA46F5i<L$ybEmH+}p;xG)f$A0yHEsq!wzHz=xZyMG=Uzfo9xoLI|JH6=AlQ~4d|
z7Yh|%#kT8}FrqE`M*p$s*FwNDGT%`ecKpn7*df*?{PTX@5PvL>$<V8}fy@8G>sMTz
zhy<R|xe{UAyr8nZ4pZ!mV0V!ys=<B}d8LnU4Z5BIvK^gBtSY8y&-bC~0`XU<c=)Fl
zzUw>dw2C(4Yj*}=u{V7sDq9c=S!gF>J7$Onzhj2;DMq5XHli{%&&i7Y@^HewgN_55
zp?Dw`a|kCnIEO=QkqA}M!{96)Ye<v50`vl!?k`#&=4jM;@+kro^XE20RY@3iO={Li
za$)u6$D@9B%>wbR-#VQFVXPZZdHUg?^Hc)6)L7?qoj|{cw$^KAGT)6^N_05`;fV0W
z!?Pj4Lp<P?uU7o(*=;1zu@ZrU3Loi0W3Lw9&t0|?Epe;%<}Y@}4;2P|*-&z7dI<5^
zzqt9pjw<}1EV!J*2kBt7BxTrO7Y31S#BRC*^Lm{<i`1{zu~#~!bYgrKz|t85>_2lo
z-O%MPg;}iEo_auv1P38&oOmJVRnQB_*m%fgk$Zk<+h5H>RTbbk-**hxJP?%mjHn4t
z;7Mrs>l8_j5st29^y2@e#pf9<h{2{og3^Er)d|t^{L@i-XN4`Pt?B*l7S3$?Pde_u
zBdeuiFS#=n7vPm00zLLqbzJ7D?!vjc_(Cc~rN`p{K~}C8bRG!QfKHRMdE#&Pzvq;h
zi0W64bCeHEQCLU){Ej_kpE}d6-${8ruQW~yz&!tXnOr`q#c5uaZQLQ<!Ftxrn}n#%
zvmFdiyd|RQRLsXGCUTD|uX6VuI4@^uGD;wI!~uSYUPxWu=ilhj;^MpS{PI}i8$D;L
zW7cdOkEG)}QB}Zru9JmUNz84Xgs8}_Yc9aN`6XX~rSrUA^bi|zbk#uPKRYx_m%}J#
z(<d7LHcYWO&1V*Cx!X(BEe9|kb&6tr(~mK<+lm%1hl_`ySm_`Q@!ww>xDV(*hFYd-
z^x2iDHBgRmE-~Dl&#{_~tU>tkZ(y3}KU$|{Bh{p|7iD=2A`f<0NmHD{f|=>p*C?Ug
z%gJ%L=G!hN&P+c&?B;C^eo`akW7kUMxOV;g*;hiRoE^wKYwb;B-)&3P?t9jdfzQ%^
zyKU2(;kS@OOZPhp>P!;oXeq0rMC|+{+K{?CDj&-W4$LDkk}^D@*ULtflt5lHEk|+N
zlejl4GdI8~yl#UM<JIb#v*z|D4B-Iu6*0q`f^s_co6V)u4<@W1R71xzTer%Bk3w^R
z&l+VYKoiB{0}5ki!2L-|QpT&0n-&o+8d>>yHefV@#E6z?u=pkYsjINKTy;NvlkwlE
z+4VXbaI#F}zr#4f#gd}~EU33XqSQqVHQMnyUf}Q>3+K4db~08PLhUDURbBg~i=N^Q
z%1@=deEM{i$S&BZik35EtumgoUslt#VD;Pwzp^I0qxRex0{r(~z$xXBr~Qo@qV*%c
z&wq^!!`BWR!V?mhB`Jb<SFjnRlq8dr)1J1HkaPrjCGW1^8N{h4`F#1VPft(Sxv;+e
zA)@h9>8wtVR&3A!bR#kp2j6}!2jP3MU~EMH`b-@$YCmMrvEyA?#&v%zG3~xj9pZM9
z(D{JQw?sDSw}-_(+)G!;s({$bAteljI$5WU7BIUP=?kMBb4Rudj-{0{MwBKrQug5;
zUel>wwYcs%`3G7$En(Y*x)TXR8jQwWkcu=H=7UC?GYd6LIH(e77PI0MkpR-`1#DD|
z`@7qn()yOp$T68r$KH|6ElC4eN&G}s&#j#1j*l1*DsL2U;|%tam)Pp7OPPO&2MV=b
zr@epGkFo@(kw1sO?VOvBg#f@>vke^$q^D_NA=w-(%|e0oUbVOR->CER2~ie_1?6>B
z(Z@J|h(<Uo)`HW}=@=wA8Iwpb-shELtQYFFTNSk28|!5bUI-_sQ{w>aR5$bP*NO8m
z_X#dU##Pw<{>2ok+Sz6rx%9vBi{3E+eXwo3LztndgWiGc-IgRNJp7g)Mp_JB%>}>y
zO~|vPWBR$~l|>GuJ{U^Fmx%zwMwOG#FQ~u02aCH{n?3j8vkE<AE2(YvKqk9iIy_Qa
zgJ3(u<bn57&<T3G>tv5yBK`%oCPaB7zIPJUWahpxJQtM*(Xq;FkY&<*JuQeAV<(SY
z5<gA-l0V6BfX}sxZ#_YtKqq#j5BDaH_CGTlQHq)-7~uN%&aMr<*aNm@sn3FDv+FB|
zVPmQZ$($0*r_yp%f!dwyNPNC`w4gxZ@pnA0)gh;B#7NbyBK-~&|6oeC`TW<v8cqgA
zND7q<Jr)k-4<#mk+pM~2darf_3DyKEJ~Xi}Kz5dxiY`B2>Re2Pg^2-TR!neBM5k|H
z?P9$Xz*9mlsi~!w9sdiXx<LJSDz@?T=?u;SngzxGGSt4GT)QjupvarS5d~XH`AG-S
zA9bX%ArJ04LW{oVjJ8w9R~vDRJkqs~v5)Cn9IrGLdGi!^HS!YT(mArt-`rC3=*;l)
zB)#nPU!Ttg$ipw>Vx43!*gppZj2!hN?-eyGUK1tD+IKkAPR$$6P21ms{!mP9_?0^;
z(w91YXm(#TvghJ^D`pF1T4Dg~0E|H5C2_?E+|*s_7UcfQ?XwVU$!by60U|Hq^n3_V
zL@v`j$~u!{4J0kPMMU5)DS>H}|AEU*M7=BR5?=C>8?Q`X703zj)J_E&+LIcs#0`f~
z$_!E09cS}Qa+jrr&g@YRg8oU*4nj;bUv4$BQ|F$X_5z0B`$g;~^3L=UHOfB(W(;1c
zY8j^?s_jr|cZsYU^ok;U)CgRJ)%PtZW6Q>(WLMO-w^3<!4%d?OASxJmg_q0;nxvv-
z_lI=M-)H;J@lmPfTfr)qB5f@>jkzdQS{n%?;o<aQx5TdTmN<3LCpBKc{6Vy<Q-n;Q
z4V(>?H*gSsu*MpJ!S*grrN--DF*SjYm-w%T0XaTt-2upA_tEFuM_0Q0wD<hi5vA?O
z<|8Vxtvi`IyKioQDqMl}DDuz#$NEl|W^=B87q!MC6jzk|ndl@p<fnd|*>z#$AQAl^
z-*MbnnyKzpaSmYuS;s5D`}Y^zwSqFjYA>EyD*0hoY>h;lYr4$Ob&KaoIV~k^vrE69
z&{u2x0o$={W(#{R+m>DgWITLE)Om<$dtAFq#0Aey#Gmkzy29GZ*{iUnuzFzs$Dw|t
zg!u3idY5U@fgZi^E9C{;XupGP>;vDcss8BpGfRvL`OL78+TnG~WzEngij}F32xm)!
zqnr0jpa5Q%^WAHOp!_YH9@g=^H`SIq(5=1)BaQDaPVW{s!pH|f8Sa04d|_#3LL=bD
z|K!_L)<w1U@3-n|rv9xg9(aw1LS&=NDr`HFR)No>{Pr+N9kDU=OV>)oq0d_&VrgmJ
zJvtTcw=#lerTGsMJ<RA~b=)!`ACzu^f&jbbVgN5@3mSAkd&_!l9l`nCW8>hFMjON=
z9ZhG8Koq7^>)M0OMkydN0m}f{N%^x!S10nC+=gMLLSiW~8+vDzd~1=Mu$Z7JzN&cj
z6@X&}M4ugLFNQ@?^BOH2yUxaV&F$H8yX1GIweAN$9vg;i2%V1jy&2^G%>40U=2$Bb
zIY{GqjNtu>IN};o><B|k)V>Z3K^RsbCEHsyE<c50*5})zr>0@(W$bPvNh%U)fMJBI
ztzL90`6mKOcQARCR`hLJxoJNcH%Z-@5fwz9AnKpI+y&56_Vs2bH6Dz0p0V>nUKlz0
z^GKf~rh21M+<>hQ7isP#B6I6>vOuh&i%b%13fF0xM5ox-!5$;bJ{78`mJ+wv`<%56
zu=GKt3Z^eH-f{er1&GUXpHnVeus4bHN|xc;=t{`u>Fdi(H+V66U?l)L|Ek5qcz7!O
zz1P<@l6G5_ab<WoA<TXnCH|`sCclkTp9z+S7WL3Cpegm{x&eip@W)87w&b<3VX380
zLC8Teukn(j#Z+bCaQl7$<1YUJ$J+q&=CBmAN10kcF%dnV7hG^RF4`INRpowOm55>$
zo`9oC1db;8XH~JV50g|r)$aEq9$g|LYSZ^|R|q9&rXr+4<fq`%{xkZG-CY<yi!T&j
z^NI;b%XgC6Mr4;>uSZi_QgOh4fzp%z?l@$Cx2?~@{+<Te(e});-<r*nj}B-KTpmC?
zT~g`UR)Kc{Z9w>Q(>zyHk1$!e<&3-`Hyqd+xs$(o@bLlvJ=o<q_$a|wVyp=g`V1lD
z;hFz?K0%ThIbZa@bCz3*RS2|G|8sJ|#pyqgVJ~NK>F5~SIs0ej*pW%0b+->2uWKAU
zd%_6P@J&<ieM{gJglDZ2WrvheNB#Y#l1U_7oKxc0hmeU@a!HjmMiqHJPK!fvbVZ9Z
z9EKl${@Dlug*B-E<3%RZ@|!sO=tp@Zei43IL=oh^WmfeKnC1}@j>R7F21b2&!$ZZZ
zZ3tT!{a#Hh>2)v>RYZqHPNq2q+A&W7c5+nha+hrgY+kAuiL<s_>zN_xZS?^5aCcm$
z2ce%*^fV2jBMG>APXJBV0iroj*-DHlrQu+!aR^+bRY@e-nOL|SH4c8YJQ~Xl3F(Q^
zk(*VrMT5GnS2`A=iX;s^;g@Cns>^QP#fp~-qBiGjmbQ2mTgpwg^#Avv#F27-Rjz7I
z?sD+B-V6g_s)41SHpRbd$YPU`xI5R4cToK8W*S23;QOG2-TdQ6no&$CvXZj@_B|w_
zYvc9{_0+wIHahJ8&Ic8vYAgR5!FFPNhL*KEPn;<>mZn!ad15byhumr^DiaxE4p9lx
z|0(Ru<DuNU04^pm*2t2C7|cwCCX^-al)*%pY}chy=vJ0)mh9UIV^4_4nq@4>MY=+a
z?b?!Ly{<J%5tFi1gZB)#`);4l`_5m#&*xd@{FZZ`^PK1WzGr#YKBuX5T*67WE5TM-
zzbZx|d+I0B6S|1V_F#|G19?2(Z}(~~M*yjF<ovtON^(!K_NYss&xY;J$q9d>IX=l^
zhZFqtNbv}pW%XQZ84meDrSY6c&NVbP=V~Vr;}!VavRuLWdpA=cm0cgr%IVNt;T`JG
zb2vTCMSrg@1(_;e78^(M!^H75jb(f^jB=n&#Z?WZj&V}6sS+;bF)R9=ooHOWB-$!j
z6^%t)1+cw+bji*2cj>F$`nDClT<r<#`ue2(;zj<>zg|^%5R_z*B9WY&Y{ea7Y}8X+
zP0d(zH{@vRBDTiDM(?N0HH*`{$Lg2Ri8hRS9^&$DdM)yS2_lzDA?iKkaKYa*K_fkh
zbrl_~SQU|^1*=CT&3;!V&d<~A%s9w>yFsPnv#D8ApRwgZ?FJlOKHd%YY7r^YMHQyp
zjp7LjC(E;qq6@RB6OL)}2fhrq1Y1OiXln?_nSUP#<{J8(>ltuaxy*=i;+%7x$_+7f
zf?4J#0%`x=Qa4v}#@F-mKN5M<?La?7hp+stPvt^FSyf}$4w)EhllS@iPTl{J^fPH#
zlk<f3D|#PdJv`0XDKk8;rF;n&`A{-E<9?R0!U#Y%{?bMWE79L1>iO85CmtEc%dpDL
z$zeZ+&u(IDU7ONPGpT<h7^uMa*zNH(w2OQ{!T;g4oxAaw9u3^o^WebEVo8wh(*1>Y
zPM_uj)TSnPTkdGoE|Z`Z<JdtN^VQA%_XezNv2os9#nQ>w48Qj_rFBoz(%)tPOg4mL
z@b+AFS%w~qw2KdZUu#Cix8(YbRXkyL6Fxlbzvjh%BAv+++P0iP8)&<WGJ01z{OOtF
z%SKsLBo+1MlSEyB#?7fjD=y)8R>Sh;-qRDzTps3YCj1ls@KQBZndiCOi<A?dAIp!u
zsj95K<jMp>ZJ9ZU!%&>EDIG)YcC_*&F3XYuC~$4t-+dOa*i3V|C0|r7)$m-c@I#vu
zb_-+?pT*Z!gv59i=DQud<3l}Asq)Z0{oSYZ@FccfeARx7q~kK-xRNmD%IqKcExDlX
zc9<;oSB1dG2Vg_`?VQxdwl60xHigHEubX^2<HWmtE<XZ)n_bs+F!OHqa2e^r&t2A8
zj#v3SVAmwG7YMwVD3L^)&eP!Y*%<Y0fB$%3EEp5&*Upq9REmSFth9$DJ1w-R=Cia~
z8dx%1*)JoVK@&R73q&z20?(ft3@O|k`9GJKj=r3UbE%0H=QDUXY6W4tt%I;TAn4Q&
zLybbkzKs4k#zWu27S^p~EDo}iKAUMFRn%tE(XMm1UF-6G`U|!*`~P_`p;eG@xs3d7
zFNQF4XBlNg)bVi<l?lK5ds2xdA51SlBHj_^pDabO3EAq&v>1~b!67{es}cUZntiJS
zZ7N%$=#)ZO8cud)nw<Z>G!;uoviiCMj^OQii?O5#M^SF}5Lz3sZJ3StIjG_V4Vgi3
z)1jeT8p5oW1GO_Cph$y8HE$#`1RAdUp+XKcrq$8dts8j!zZ<+2Bqb%@m1g#%3o1Lb
zq~=gM{-g<QNfK*mOZm}sTYez*{SW#7U)M33tpmJq=WET_xeqy^$2G`O9k3E7y&dhV
zep1WHexUbd%%NN3=$y!-v9_ahUd&?kj+ZK`1-18UKaTarE@kB13a;jv$(!NvH$B+l
zYXRA+NGLNKrs<Wh5a6~X_P9V14VP5k{8qZ=Fs%}^a?iV@6ca%7eqN_%j9IjTxFkjs
z-o?l{0`<|g>;wYLO&g?-a^p!67j(}<Dkw~$J-8CExSJJa9sv>px?7GQ$bz>&l^hRK
z+6J>~1^RaYjs4i!1~YGOVn#Ux;h2^A;nx!rw?AK%mX^Msp8g_TLtdiNXBp7uHa0fR
z71}X1HQxZ$+WcA!4LltLf#Ch(EO_EzDUor?IvxNUZEMrl%3PeC^_m-aI45amXP0pQ
zKKD~H^2j`ZGH=|tL5~QwY8J&Q8JL?Vi4K?m(hh{^@VyIfmxTbN@XD1h>g_%2?EC>h
z-U)<R>eABE{QUgN!YCJfoS|POSzeF+@EEEWHAe{2*(xh5OUx$eulolF!*y4Hh^xdI
z;$OtU7>XcV6HwdT51eyJO)nPz72s^P<QcoIE}ZlA^*wc}>49FH#PMH#(9_dnj0T7;
zRaiE)6;OeJ%$|v`wjMdH%?SwDz;7#u88KdOyZ5KulBBGqZ5|R)u#pljBr?b9sxk?m
zq2-^Nnu>IN3YcI>w!J1>1W4tR2FU!tmHB(Wp?0qg1JfTRCs$NhNJGt@mg3JXC~))_
z9*Px=>O;9o7kSJL)CH`IRVf0q5vZvE3ZI&uMz~!Z;Ta!=L1r)*zuCCx>r>WzHv%&>
zz0VdKozo)htUvCqp58QB>-F)Z&7C$N*<MsYJPT?E-f@HcfrAGP7&5o@^R(NG;M;~h
z5grOK2DbUp(a~>TWHzB7q?Pl>lk1t))oKj0_4V}@*@%WiL#}u|f;+bqRx6V)&Et|w
zx{P+s25>faiiPo%=F54cJ#!o^ot?d6a(qZE5gz|ldb+y0Q@cH4{35SGeh#ETI5;@i
zUb;b)Zf$L)YXHEORZ<l8yU^qh$jODJSifbiw%(K+vIUEJ(a_M3fL&tBK|Z52yDNa!
z@R=?sY^nXGEnWtX#{<iDTDSIH1X8ue!os4ZrA0zQVybtOEpS!oUKf>$zI4GNL8Koj
zqj>Gm&`>oswH~2|Ru#IMcFXT!D4JHgFa7H7?oNR(S4=I5k;4w9Jguo|dUX5o<7L1K
zXPpG*{LItpYFbqox0C&o>*Gd;`h%b>6Fyc_n>RH)GX=Pu({DHfx_hqa=;+wg`6N-5
zPmu6^eSNkgCm+0=s6Z+Bb2~w~X~1mv{6w=)5D`M$LM&h~m?KI`LG`xH{r$z0ib-8v
zhO>%~F4P-AHWJe|#Ai?^2Af)1cB(3PYPXJ!WzcLJBsb`3r?_Qwb~Zsq5v%Ma7nk$J
zb#GM+1^N47CH_F7i?BF(0{NLD&A?2|&vVzi!|>)#q~%g4*aXGIko!`B`Wt0XYzaHs
zj;X2b_2F7_%j0Ik)klCV+}Y9b!^pm4E~rR4v8<w^c!-gjmXL|qBLtc7pkz>EEG0di
z2TPV?5fB&WE+sq1c=0cMEv}<Vfog=7?g_X8sW?(L9`i#-hPd+z3Iq#C{5wcV$;t9^
za_1S!)=yl7;;2^Br_JbZF4PC`@W}ezZBu3K;u+a5DJi+u&JWrO6Vy`}RsSI__~~@)
zY^sh6VP`NOi=TjP2-o=WFT|iybs3r0P@|w(l!JoQ;-o*xS}UM$_N^@O!3TT;u}`8x
zvS4H=NHLDt(-&W{1!`RXqmmN~?Do#*&qL1~b)rNZ;pb1%M`1CTU<C`@oj^3Gl`NLS
z>E=B{ImQlm#@dY>*0jqKajR1;Q2={q%ETJXQl&R_{hM85V<W{sL?uo$3i&KKD2Qdv
zjelc9W@DqLzdy8FAo5Off0Hpc2-S(+z%aCv70wAjcbS?G?|Zk^Dc4@N#sF&iz=rM^
z(597zfw?Nq8-85mo?*hD%e#UXXq}y0RXA9}N}R5&*J+nm0ma_TLcz^{4QaSO0LeWK
zztf3Bb-u^^1pXWyoqQ7Ey3D%p6$1WSkAgnD8ULE8`oV)-0W-Eyev^%%VCg=8c3+xM
zi_gav@$6Ft0CZgEjGVlDQ~vP4fUwqO&=as(B5@{SB)plq`B5A$>moUh_T|&t$)8yh
znmwL@iD#}vrua`Iu7Pz{P*5oIh8(b)9v-3N>Q@v|Px@}>W3e2{#yux#p8j8!X25z$
z#CgInw!29s>6xXeI=i~Ml2^pE*!kDY0h}E&uJv&~4o`x#(op!Ca_YTY5fl_?tP)l2
z0nHp99`^S3270-)F)OO~7U8k)stdrI*A^y@vX5L5i_wj*Tw>0<ywQCr1*)_bXOGKS
zEl3=OlI3*$Q)VC`EaP#&;6rz0H~aGJo5Fa?A|N;F7iVKav2|)RRm!@`Df`vCvK;~(
zh%z2^kN1gQt(&iI5q4U?g&GuY)qj?k<AxSy=jKd}Y)vId_=#k+mX<`o9y*Z{6O*JF
zL+02v6TSRNe)*UP<i?(!moGt;gD+?vKK$^7v5HG$4tajGM&u7nyguSN!rN7KZ=Qe<
z<G|Y2<+-V;OKYeGy{S(JXGnjKta=nu)fhO?tougmd_5uUG4Zg9JwTtKBO8oGsA&WS
z+!--63oQP@M;A8z#&OECfQJsV>M4mb@Q7gm@i_AqXl;Z1MX;=yfdZh+?3uTI8b#Oj
zet%|r2%A43Xc79aKz4zO(O*jvs_-zK1vD0i#_HS!uNJ~B4BQg9GJBcI1knI^{s-*!
z=824i%GU9+|LH&6@w{xYD^A>;y^<#yGhWm6uf7DoECk*Ez80UDD6e>-&}Bs7h33ni
zoZ7eS)7>GzCnhHxt90i<FIt_}9PG`mt*rex`^8|}yV-IpWiZ$t!EJX@@!vho)DA++
z<S*#gOg~dc?k|5cOu7g(6vlYl4OP^SM1DpI3n^P9-hkV2NJB#V_uEdf2yNe2MoUB}
txzm|FM#w=N&LOuUh@{=$)_=2E*kqAV>Wv<iEYD+sACX|BQ+m>t{13q)j6eVY
diff --git a/doc/guides/nics/img/ice_dcf.svg b/doc/guides/nics/img/ice_dcf.svg
new file mode 100644
index 000000000..58851ca5d
--- /dev/null
+++ b/doc/guides/nics/img/ice_dcf.svg
@@ -0,0 +1,727 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px" width="568px" height="513px" viewBox="0 0 568 513" enable-background="new 0 0 568 513" xml:space="preserve"> <image id="image0" width="568" height="513" x="0" y="0"
+ href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAjgAAAIBCAIAAADLYlLEAAAABGdBTUEAALGPC/xhBQAAACBjSFJN
+AAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAABmJLR0QA/wD/AP+gvaeTAAAA
+CXBIWXMAABJ0AAASdAHeZh94AACAAElEQVR42uzdZ3wbx5kw8NmO3gGCvVeJpHrvkm3JlmvcHTtx
+Sb1znOSScy538d29l7tcitN9SWynOO7dlquK1XulRIm9N5DoHdj+fqBMURQpUWIBSTz/nz9Yi8Vi
+dgnsszPzzAwWCoU0Gg0CAAAAppJwOIzjOIZhuCzLiS4MAAAAMJQsy/0RCk90SQAAAIBhyJ+DGhUA
+AICp6EKNCgIVAACAKUuWZTLRZQAAXDWe5+vr6xNdCgAmVkZGRv//QKACYPqJRqPvvPNOTk4OhmGJ
+LgsA408Uxba2tm9/+9v9bX4QqACYlkiSfOCBBwiCSHRBABh/oVDo6aef7v9/SKYAAAAw1UF6OgAA
+gCkNAhUAAIApDQIVAACAqWigZwoCFQAAgCkNsv4AAGBCRKPR7du3v/XWW7/4xS9SUlIGtv/+97/n
+ef6WW245duzYwYMHN2zYcMsttwy86nK5vv71r6elpf3P//yPVqsdYxk6OjreeuutXbt29fX1EQSx
+Zs2axx9/3G634/j411J+8pOf6PX6m2++OTMzc3yPPE6ByncK9e0d99MG10KTjzI2J7oQIGHCPIrw
+iS4EQAghJIpMDClaO7q37jl0w+bb+jdGI+E33npnw423RAlNU2ffjs92xkWUM2tBSmoaQigSDu/e
+vnvbtu3zFy91hMXo2EYfNNaee/+t1zo6O1ZtvN2WmhYJhzzOXmdURhH5asc1nDp6uKe7s7hsdkFx
+6Uj7LLnuVoqiBKW5LzqmYpMYoodsGdPxBrAeFKgZn0OBMcLpsR8DTF+ChGJCogsBziNSMnNLK+Z+
+9NFHqzbe1r+p6sxZXpTzimYptSZeQhZ7ep/Ls2//gZtuvwsh1N3neuWlv2fl5ssIiwuIHvSnDPi9
+J44cOrR3l9vVZ09LX7Fmw/I1G3iO3bNja1NDnSyJrc2NN95214KlKzQaLUIoHAru2bvvxMmT93zp
+sSUr1uj0Bo5jPS6nQmdiJbytqXH7x1saa8+SFFUxd8GKtddlZud6XM6jB/Y4+/ooijy8f49Cpdp8
++90Lliz/7NMP33/zlZ7ODrPVtn7TzWuv2+Ry9u3Z/kl7a7NCqVh7/U03bL4dJ4gjhw6qNRpGrXXV
+NdSerYrH4m5XX1tLU1Zu/qZb7iiZVSHLciwa2frBe4f27ZQRqpy7YN3GzWkZWR++80Z9TbXBaOrq
+aKucv+juex8YcheDPioAAJgo1hR7+bwFDbXnnH2O/i17P9taWl6Znp1DUhRCqKy80mA0nTl1zOXs
+C4dDdWdPh0PBBUtXXDrnyMHdOz/79EOGYSrmLcQw7JMtb584fEAUxca6mtf+9lyvo7t0dmVKWjpF
+Uv37u519XR1tFqtt2ar1eoMRwzCGUaRlZCkUymAg8NLz/9fV3lpYMisnr/DcmVOfvP+W29kXjYRP
+HTvy9it/i8ViC5et1BtML/zxd85eR0ZWTm5eYWZObln5nIKiEpKieZ7T6g3zFy9LSU1/+9W/19Wc
+FQThzMljdWfPRMJhZ2/P9o+2HN6/W28wFs8qdzp6Xvvbcwghnuc+ef+tI/t3Z+fll5VXnj194vD+
+PR63q+7cmQ/ffq2lsS6vqCQrO/fSywh9VAAAMFFUanVeQZHZbDm4+7Nb734gFPDv27ntoa/8g8Vq
+7d/BaLZk6QwtTfXHD+0vLC3bu3P76g2bZFlub2kafByvx33s8H6SJO784pfTM7PrzlW//coLb7/6
+91mVczEMkyRx3Q2bFy9fpVCqBt4SDgY5jrWnZWh1uiGlqjlzqq256bZ7Hlh7/U2SJG5589WzZ06e
+OXW8sLhUkiWcINZef2NeYXFnW+s9N65uaaxfvmaDx+W02FIWLV81d+ESjuMYBZObX2RLsXe0t/7T
+17905uSxvILCwR/B83xOfsGNt9+lVms+fu+tv/7xN72ObrVa88Kffn/Xg4/cdPtdKrXmhT/9rqWh
+rrh0NkJIbzAuWr7quhtvU6nVl15GCFQAADBRcJyw2dPmLlzy6Qfv3nr3A3U11bIs5xeXqDUXsiTK
+yit5nt3+8fvdne1uV9+Dj31zx8dbhhzH7/OKophfVJqTV4gQysjMySss7q+mIIQysnLKyucMjlIX
+DDcZZGdbS35RSXFpuUarRQgVlc5qrK9pbWooLC5lGEV2bn5eYTFCKDMnlyRJl7OPZeOD304QBMey
+VceP9HR1RiORcCDgdbskSRq8j9Fkys0vsqWkIoRSUtN4jnP2OrJy8s6ePrloxeq3Xv4bjhMNNWdF
+QQwG/AihtKzssoq5w0YpBIEKAAAmlF5vmLNg0btvvNTR1rJ728dLV6y12lJx/EIyg8WWUlBctnPr
+xzu3frT5jrvTMoZPmaMoiiQ/v2NjqL8i1f8vhVJ1aVOhWqulabrP0RMOhTSXZA/SNIMT57t+MAzD
+MNQfaXAcVyiUlz8jZ2/Prq0fHT24j2EULBuPRMLDlJamSXqY/vL+yqLf68ExnCDIvMJis9WGECII
+kvy80fJSEKgAAGACMQplbkFRemb226+8sGvbJ//yXz/X6fVD9ikqmXXTbXe2NjVsuvXOYQ+iVmtY
+Nt7e2ux29hrNFo/b1dvTnV9UepnPtdhS0jKy6s+dPXJgz6KlKzU6Hc9xLmevxZZislh37/i0s601
+MztXksTurg6e41LTMi5/IvFYjGVZhFBbS9PRg/uKSmd//TtPtrc2f/8bD49yEn8cxzOzc+964OGl
+q9dqNLpIOEQQJKNQXPGNEKgAAGBiabW6tdff+Mdf/1Sr1ZfMKr+0jc6aYr/7wUf7/z8cCl56hJTU
+tFnlc48e2vfRu2/mFZU01J7t7mz7wn0PXe5DdfpFy1f1dHV+/O6bwYA/LT0zFAo0N9RtvuPuinkL
+3n/z5UP7dgoCL4rCiSMHdAbjvMVLRWHEhFGNVuv1uquOH7HaUnAcNxhNfp/nyP7dbS3NPM91dbSJ
+onjF68AwzAOPfP3A7h2SLBqNZrfLmVdQnFtYdMU3QqACAICJpVRrlq5a+/oLz3/h/odUGs3AdqPJ
+zPMcRV3URIbjuMlizcjKwS8e63TdTbeSNP3plnfefPmvmTl5N99x97qNm1k2bjJb0zOzCXKYm3lx
+WfkXH/3GB2+/9tYrL7idvQRBLF+9nmW5lNT0b373h6+/+Of/e/onFE0vXb3u5tvvTsvI6u3psthS
+Bo8Fzi0o0huMOEEUz644dfzo9o/f93k9t9/zwOKVa156/v+OHzowZ+HiR//hO8/+5mc8x6Wkpun0
+Bpqm1RptSmq6Tm/oP4hao8nJL1QolDSjuP+Rr7761+deev6PXrczNT3zgUe+np2Xb7GlcBxH0yMO
+rcF8Pp/BYBjr36F3B2p/MzFfATCEcQ4q+kaiCwEmViAQeOaZZ5588slL16Pys8jPJrp8AIwBjSOt
+HHr66aefeOIJgiDwiZhFAwAAABhHEKgAAABMadOgj+pUvfupZ4+fbPAMbNGpqIVl1i9uLFw6O0Wr
+psZwbAAAAFPdNAhUnCC5/HEFRbz/i+v7tzi98bd2tvzgmaNfva3krvX5Zj2T6DKiD/e3P/d+XVmu
+8SffXJTosgAAwIwyDQJVP4YmZueZ+v+fzRDNBuZojfPt3a2z800rKu2JLh3yh7nGzoBODRPCAgDA
+OJs2gWowhibKcoxziiw7jna194YGAlW3K/LJwc5DZ/s8AZYksQ0L0m9clpVuVRMEhhD69WvV/jC3
+dHZKnzf62fGeOCeUZBvuXp9fkKH78EDHruM9Dk/UoKU3Lc1cMy/NYrgwBi3OiU1dga2Hu07UueOs
+YDMq18xPWzMv1WpUEjjW44q+vLXx9R3NXc5IOMrf9s/bsuzqJ+4uz8/QIYTCUe69ve37q3pd/jhF
+4ovKrBsWZRRn6Rl6bNP3AwBA0piWgQohRJG4kiF4QeaF82sVN3YGXt3WVNXoXVaeYtTSTl9sy772
+Uw2ef3loTpZdg+PYiTr3gdO9Na2+2XnG5RUpgTB3tMb11LPHVs9L23+6d9Wc1HkllqpGz58/qBdE
+edPSTL2GRgiFY/z+qt5XtzcrGWLJLJtGRTm9sS372k7UuR67pSQ/U69RkQtLrY2dgR53NCNFc+uq
+bKOWMWhpWZbDUf6Xr1WfrPMsKLHML7X6Q+yxWndTV/D+GwqmQi0QgMEcbv+vXvo00aUYf+k24xP3
+35DoUoAxma6BKhDm2nvDZr3CpGMQQn3e2EcHOnafdFy3KOOhGwstBoUnEKdJ4pevVl+3MN1iUGhU
+FEKo0xlZUGZdvzB9bpElEhcoEn/ymSMtPeFHby6697p8k56pbfV/+b92f3igoyTHMKfQjBBq7Ay8
+ubOlozf85IOVyyvsGhXlcEde2Yb/+YO6zBTNvTqmv4LV5YrsP92bn657eHNxfwnjnPje3ra/flj/
+6M0lD9xQkGXXBCO8YmvTO7tad53omVNk1ighDQRMIYFw7M3tRxNdivE3uyADAtV0Ny3T093++Ns7
+W5s7gyvmpBRl6hFCTV2Bncd7KBK/57p8m1GJY5hZr3hgYwHLCTVt/mj8/LwgBg29eJZtYZlVpSBN
+Oqay0CzLiCSwBzYWmvUMjmGz8ow6NdXYEXD6YgihSFw4UeuuavAsKU9ZMz9Nq6YwDKVZ1fdel6+i
+yU8OdnY7IyMVMs6Jz2+pp0j8jjW5WXYNSeAmHbOiMsWkY1q6g74QjMkEAIBRmTY1Kl+I/b+3zy8i
+7A3Ga1v9i2bZ7tmQl5OmRQj1eWOdzjCOYZ8e6hz8LpaXQlFeEM83D1qNihSTkiYJhBCBYwxN0BSe
+n6E1ai/KG+QEURRlhFAowrU5QnFOLM02KAb1KmXY1Dlp2iPnnO5AfKQCC6J0otaVadd8eKBjz6nz
+lSeXP97ljBRk6OLslefFAgAAgKZRoIrGhX1VDllGVQ0ehyf62K0lX7qxsDjL0J+VwAuSIMmRKLev
+yjH4XZtXZM0pMiuZa8xcEESZFyQCx2hqaNXTpGcicZ7jpZHeK8soEhfCUf7I2b7BqRN56dql5Sla
+FbT7AQDAqEybQJVuVb/6X+tFSf6/t2p+/tLpWFwkCbw/nQ8hhOMYTeKV5fa/PbWaJMatPRPHMBzH
+ZFkWJXnIS3FWpEicwC83uz1F4uX5pj/+YKXNeIX1XQAAAIxkmvVRETj2pZsKNy3L/PBA+3t72hzu
+qCzLCCGditIoKXcw5nBHx/HjlArCpGUQQk5fbHCsisT4jt5wpk0zMC8GjmH4xUuyEDiWlaJpc4QC
+IU66JM4BAAAYpWkWqBBCOjX9/S9WlOYYnnuvbtuRrmCERwgVZOjnF1uc3tiH+zvCMb4/MHC86Auy
+vHDtUcKoZeaXWdNt6r0nHR29YUGUEEJxTtxzqrfLGd6wKCPDdn7Gfo2S0qopXpAGEjcYirhnQ163
+K7LjeHff53GO46VQhI9BBxUAAIza9AtUCKGCDP0/P1hp1DF//qD+4Jk+QZTyM3R3b8jPtmv/+G7N
+Wztb23tDbn/8RJ37mXdqOvvCoihd82fNKTTfsTq3pTv469er69r8bn987ynHr1+rzrJrb1uVnWo5
+vwBaYZa+otDc0Rf+6EBHIMIJoqxUEN+5r2JFZepPXqh6bXtTmyPkDbKn6t1vftZyrNaV6EsIAADT
+xrTpoxpi/YL0h24s/N0b5/78QZ1ZzyyaZVsy2/b0t5a88FHDz186/fB/7SZwbE6R5R/uLLMalWPp
+tTLpmAc2FhRl6//0bu2Kr24JRfmcVO1DNxZ+cVNhtl0zcOTSHMOXbyz65avVj/x4T3aq5oWn1s4v
+sVgMihf/Y81z79e/8mnzv/7heIwVFpRav7ixMCtFnejrBwAA08Y0WDhREKVoXJBl1D9VxIA4J7Kc
+SBCYgib6A4YkySwvcrwkSjKGEEFgDE3QJNHfeRSJCYIoKRmCpoiBI4eiPEXig8fehqK8LMtKhqTI
+80FIlpEoSXFO5AUJyQjHMZrCGYrAL86kEMTz++A4plZQJIEhhGRZZnmJ40VRkvvHbNEUQZM4ftks
+jDGBhROTwEQsnFjX5tj0Dz9P9JmNv9kFGR/85ruJLgW4CpcunDgNalQkgQ872auCJhQXz5iH45iS
+IZUjzKWuVpKXHnnICCqE0KWJ4xiGSALXKK9QLRt2HwzDLi0nAACA0ZuWfVQAAACSBwQqAAAAU9o0
+aPoDAExBKgX931/ZlGLUfud37/X5woNfWlCSeeeaCotefehs+58/OjL4pf945IbsFOPv3t5/sqFr
+yAEzbIZv37VKraCf+vMn7kD0hw+uL86yUeTQZvMTdV2/eG1Xos8eTCoIVACAa0GTxE3LyvLTLD96
+/pMhgWpWrv2L1y+wGNR2s+6tPacD4QtTYubYTTcvn+XyR3yhaKvDO7BdpaDmFqY/uHFBV5+f5QSE
+5DVzC1ZU5D3/weHa9r7BBx/8LpAkIFABAMZTZoqxMj+NpgiXP1yUYV0/v+idPWcGXn1375kls3PW
+zivYdqx+cMixGjTr5hXQJPHJkdpg9ELa4ocHa7YerUv0OYEEgz4qAMB4mluYvrA0s6at7+3dZwxa
+5W0rZw9+ddep5sZOZ7pVX5Bh0QzK0E0xalfPyXcHIq/vrEr0GYApBwIVAGDcMBRZlpOSbtUfre34
+8GBNNM5X5KcVZVgHdghG4kdrO0VJWlyWlZNq6t+oVtKFGZasFGN9h7O62XGNnw1mLmj6AwCMm/x0
+c2VBWiTG1bb1eYKRs629i8uy7l4358d/3z6wz47jDdcvLJ5fnFmcZTvb4kAIZacY1y8oinPCx4dq
+hxxw6ewcBXPhNtXR5z91SRYGmPFmYKCSZRSJ86fqPRolWV5gunT+pEhc6HZGwjE+L02rU9O+MFfT
+4su2a9Kt6v51Q2pafe5APNOmyUxRDzv9Um2b3xuMp5hUGVa14loXuwJg5lkzt2BeUfrJhu7DNe0u
+f3j7sfo1c/PXLyj8+Wu7WO78fM2Hzradbem5fVVFSZZVp1YEI/GsFOOKitz2Pu8bu6qGHPALq8vX
+zisY+OenR+ogUCWhGRioJFlqd4Ru/f7WVItqx+9vspuU2MULcNS2+n7x8pk2R+jpJ5bOKzYfPNN7
++z9v+/fH5j9xz+z+KTCeevb427ta/+HOWU8+NCfTdtG8fDJCsbjw/d8d2nm85+HNJd+9vzw/XZfo
+MwZgStAomVk5doNGVdvWV9vWhxA6fK49FInPyrWX59lPNnQPrHdzpLZzRUX+ioq8g2fbjtZ0ZNqM
+agX9aX2X2x8ZcszvPfMBJFOAGdhHReB4ill147Ks2jbf6UYPy120poYky3Xt/qpGT16ariLfdJnj
+HKru21/lGFjGvp8oynurHAdO98FSHQAMsW5+wbzijBjHc7yYZtGnWfQqBdXS49WomG/fvYbEL9xt
+Pjhwrrqlpyw7pSzHvqgs6551czzB6I7jjYk+AzBFzcAaFUJIrSDvWJvz2vamjw90zi+xDm7j9gXZ
+ujY/QvKqealaNRVjhWGPgGHoVIP7g/3t80qsxVn6ge0sJ/zm9WqWFydsTlkApiUMQ4tLs3NTTSad
+6qmHr3/q4es/346RBL52bh5NkZxw/vHOE4g0dLpWVeYVZloQhrLshlON3Vv2n030SYApamYGKiVD
+rl+QbtAyr25veuKe2VaDYuClM03e43XubLt2RUXKZY6QmaLFMay6ybf9SOdAoGI58dBZ587jPfds
+yN+yvyPRZwnAFGLVa2bl2aMs/6Nfvv3SthMD23NTTS//6ItlufZHNy9+/sPDkRjXv726ydHlCq6d
+U7CoJIsiyVMN3Yk+AzB1zcCmv34MTTx0Y1GMFQ6c7g2GuYHttW3+tp5gSbahJMdwmbdrlOS6Balp
+VtWeU71NncH+jb4Q+4PfH0mzqG9dla1TzcwYD8C1uXv9nLIc++mm7rMtjhjLD/zn8ARf3H6CJPHb
+V5Yr6QtLE3x0uHbPqaYMm74iP7Wuve/1z04l+gzA1DVjAxVF4A/ckE+T+L4zvf7PA1WXM3y22Wsz
+KdcvTL/8aooYhhaUWhaW2erb/R8f6kAIBSPctiPdjV3BH399odWgxDFo/APggkWlWWoFteN4Q1VT
+z+DtwUj8g/3nupz+8jx7uk0/8LuLc3xLj9cbjHGiWN/h7HT6E30GYOqasdUCgsBm5ZpKso2fHux8
+cGNhqlVFEXh1s6+m1ZdhVVcUmK54BKNWUZxlPNvs/fRQ58pKu1ZN/e2j+rnF5k1LMxs6Aok+PwAS
+LBxjH/jPlxQM1e0OIoR+8tJn//fOgVaHN8byg3cTJbm913vnj/5Gk2RLt0eULmQnvbe/+kRDJ0US
+vZ6gIEpDjv/d37+vVyvOtfaNoixghpuxgQohpGCIO9fn/eSFU2eavGV5RotecbrBE4kL80usqRbV
+Fd+O49j8Esva+Wl/erfm2ffqKotMjZ2B//76QoOGgdoUAIIoHavrHPhnfz76sDhBPFbbeen2Pm+o
+zxsa6V0wRQUYMGOb/vptWpJpNSi2HenqcUWbu4PVzV6LQTGnyMxQoxqlq9fQS2bbynKN7+9t+9tH
+DUVZ+o1LM/EZfs0AAGBqmeE33dx07YJSa3WLt7k7eLja2dYbKs0xlo+i3W9AcbZhw8L0SFzocUUf
+u6XEZlQm+pwAACC5zOSmP4SQgiY2Lsncf7r3UHVfZ18YITS3yHxVwcagodcvTP/XL89RMOSmZVmJ
+PiEAAEg6MzxQIYRWzLGnW9Xbj3T1uKMbFqWX519FdapfYab+nx+ck+jzAACAJDXDm/4QQlkpmsWz
+bD3uaCTOV+SbctO0iS4RAACAqzDza1QIoTvW5IaifJwTl5Sn6DX04JdwHLMaldcvzshL1w2M8Kgs
+NJv0jN00YguhXkOvmptammNQMUlxAQEAIIGS4j67rCJl2QgTJjEUsWSW7eNfbRq88UePzLv8AUuy
+DX//97WJPi0AphCjVplq1rsDYacv3L9FraQzrAZJlhs7XTiGleak4DjW2OmKcxcm2LSbtCkmrS8U
+7ejzKxmqMMM6kJHLi2Iwwva4g3GOv4bygJkkKQIVAGCirZ6T/6MvX//8B0f+8N6B/i2zc1N//JVN
+LC9s/ufnaYp49T8e1CqZO3/0t9NNPf3De9VK+qu3LP3Gbcte23nqO799vyDd8s7/fFmnUgQjcYSQ
+KEnVzb2/fWvfsdqOGMSq5Dbz+6gAAFNEqll3/4Z5qabzS7htWlJ607Iyhh60uEEo9qPnPim4938K
+7v2f5z84Up6X+r9fv2lxGWTbJjsIVACASXK6uefGpaV28/mEpoq81DSLzh2IDLvzz1/d9dwHhy16
+tdWoSXTBQYJBoAIATJKPD9XIMrp9VUWO3XTXmsoFJZnd7oDTGx5p/zd2V/V6QzcsKp6Vm3I1nwNm
+GghUAIBJwgninqrm5RU52XbjwrKscJQ9Wd8lyfJI+3e7AjGOp0iCgInLkhv8+QEAk+eZdw/E4vx/
+PnLD2nkFB8+1DTtZ7QCDRkGRo5qWE8xsEKgAAOPg4Nm2blegMNOSlWLs30LgGE0NzSt2eIJHazuy
+7MbqZseuk03hGHeZY25cVGrVq4/UdLQ6vIk+P5BIkJ4OABgHTl+4pccztyjjxqWl7+w5Q5PEqjl5
+Bo1y/+mWIXu+tO3E0dqOTqe/odNVlGkb6YBfWF3x5U0Lnb7wudbeUJRN9PmBRIJANQ7e2tlS3exd
+vzB91ZxUhFCvJ/bKtsY+T+yn/7g40UUDYPJsOXDOrFd/YXXFklnZBIalWvSnm3peu2SN+YZOV0On
+a9gjaJT0fdfNqyhIQwgVZ1pZXnjts6qaNlg7MdlNlUAVY4WGjkB9h7+pKxiNCxSJpxiVpbnGWblG
+i0GR6NJdwf7TvR8e6LCbVf2Byhdi393TVt/mh0AFksrhc22iKJXnpWpUDJLR/urWc6291S0OhJAg
+Sr97e9+51t5o/KK2vrMtjuc/OOzwBhFCfd7QM+8c0KvPT13W2Omq63QeqenwBaOJPjNwOTzPP/OL
+/05Lz9x46xd0esPgl6qOHzm8b/fsOfNWrL1uLB+R+EAly8jtj310sPPAmV4lQ8qyTOAYSeCxuBCM
+8LKM1sxLHd9PbOoKHjzTq1XTt6/OSfTZAzBzROP8nqrmPVXNl74kiNJzWw5fur22va+2/XyFyekP
+//7t/Yk+CXDVBJ7/46/+d+7CpcvXbhgSqKpPnfjrH35z14MPT/tAxQviWztb//RebX667ks3ZS8q
+s1oMClGUu92Rxo4ARY7/qu9nW7y/eOVMVooGAhUAAEx9CQ5Uoii19oR+/Vq1XkN/74GKecUWhiYQ
+QiSB8tJ0eWm6wTtLktzjjrb2BP0hTpJli0GRlaKxGZX9b0EIeQLxunY/RRL56drm7pDDHUEI6dR0
+SbbBalSQBB6JCw0d/qPnnMEI1+eNvb+3TcmQC8usBg1zrMYZ58SiLL3TF+txR2mSKM7Wp1vVkiQH
+I1ynM+JwR2OcSJN4mkWVm6bVqCgcu7og6vBE2x1hb5DlBVGrorLt2nSbWkETCCFBlD452JmTpjXr
+mTZH2Btg7WZlQYbOoGUS+wcCAICES3CgisSFd/e0NXYFf/2dpbNyjQMh51IcLzV0Bt7Z3Xq60eMN
+xDleyrJrZuebrluYPjvfpFKQCKGaVt+P/nQcw7Ev31h0st5d1eiJs4Iko7vX592zIS8jRRMIs1sP
+d72/t93tj4ci/C9fqU4xKTNsap2K/snfqxo7A4/dUtLlitS0+KxG5ZdvKko1q3rc0f2ne/edcjT3
+BGOsSJN4Tqp23YK09QvSzQYFgY82VnW7Im/vbD1U3ecKxGNxQa0g55VYN6/IWjLbRhJ4nBPv/OH2
+O9bkLiyznm70dDsjKypT77s+HwIVAAAkOlDFhA/2tyNZXjMvVaUcsTCSJHf0hf7jueP17f4ff33h
+DYszeVHcfdLx07+frm31f+vuWQvKrP31m0hcqGvz61TU//vagop8s8Md+dGzx3/7+tk0i+q2NTlp
+FvUPHppTkmN46tnjWSmaD5/e2H98UZQRQs3dwb9+WP/te8t/8OCc/gwOX4jdsq/t7x83Li1P+dV3
+lhVl6tod4Wfervmn3x7+768tvH1truHi1a1GEo7yz71Xt+1o151rcx/YWKhSkJ8c7Py/t8/Vtvn+
+8OSKNIu6f7cPD3R4g+w/3V+xrCJFo6QS+6cBAIApIsEDfjlBOtvskxFKs6oH1i28VCDC7TrRs+NY
+9z8/NOfGZVkKhtCq6JtXZN+yMvtkvWvrka5w9MIqAKkW1ZMPzqksMGMYSrOqv3RTsUZFnW70Or3x
+K5bny5uLN6/IHsgzPNvs23q4y6Rjvn3v7NJsA4HjuWm6f3t4rlmveH5LXUdvaOTJXy6y80TPm5+1
+rKi0f2FtbopJqVVR1y/OuH1tTmNnsKrBM7Abz0tPPjhn9bxUiFIAADAgwYFKkuRQlLvibk5v7N3d
+bQxF3LQsc3A8W16ZYjeratv9LT2hgY1aFTV4mUS1giQIzB2Ix1jhih+0dl6adVA2/Ml69+lGb3mB
+Kdt+fr5nDENGLXPTssyaNn99e2A0x0QIHaru8wTjlYVmm0kZY4UYKzA0nmJUcbw4OHwurbAVZOoG
+Fo4DAACAEt70RxJ4doq2rTd0+d0ESQ5GORxHRq1icAaDUatQK8lgiAuFrxztrkGcE0VJ0qqG1m/S
+rRqKxL3BeJwT+7vHLi8U5QRReuTHu7/yPxeirChJmTZYvwAAAK4gwYGKpvDyAlN7X6jLGTaoaZK8
+Qg1vSJ5d/z9lhEbXAjduzn/u1XyqWkH9/PElNyzOGLyRIPBLoyAAAEwjGI6lZWYRl0wfzLLxeCyi
+1Rsys3PH+BEJbvpTKYj1C9MRwnYc64nER2xGI3BMo6AkCfnD7ODw4A+zMVbUqinNxNzuGQrHcfzS
+gvV6ooIoGbUMQ43qAqoVFCdIBI6ZDco0q3rgvxSTcjQVMgAAmLJwnFhz/Y11Z890trdy7IVZGR3d
+ne1tLRZbSmn5nLF+RGLPUKWgblmZXZJtePGjhjNN3jgnDrubRa+4fkkGx0u7jjtESRrYfrzG5fTG
+CjP12faraEOjSFwxun6gWXmmkmx9XZu/231+EVJZRsEI99mxnvw0XX6GTsmMKszML7Hq1fTRGle7
+IzSa/QEAYLogSfKeBx/Nzs1/8dlnThw5EI1GEEKtzY1vvfy35oa6NRs2FhaXjfUjEnuGBI6l29RP
+Plj5kxeqfvyXk4/eWrK8IsVmUsbiQn17oKrRbdAym5dnGbT0dYsyth3p/vXr1ToNtaLCLkjSsRrX
+Rwc6CzP16+anXdV4ozSLujBL39QZPHLOuXiW7TJ7VhQYNyxMf3dP21+21D+wsTDbrulxR17+tKnT
+Gf6n+yuyU7U4jiGEaIqQZTQQZQkcUzGkLKNQlNMoaQxDa+an3rgsa99pxyvbmu7ZkJeXpuMFqa7d
+7/bHV861qxXQ+gcSL8NmfPZHj0zax4VZkcAx5ejaJMZC+/nkgWCC4DieV1j8nX/9f59ueee1F55/
+4blnJFGkSEqj0998xz1rrrtRqVKN8SMS3+5EEfjNK7MpEt990vHxgY4t+9oIHMcwpFaQdrMq264l
+CZwk8IIM3ZMPVr63p+2lT5te/rSR4yW1kiovMK1fkDav2DL6gbcIofx07a0rc174qOH//flkWa7h
+G3eUDST1DWHRK25ema2giXMtvv/68wmEYZIkkwT2nXvLb1+dY9SeH0S1ZLbtdKPn7Z2tGiX1lVtL
+TDrmhsWZ51p83/jp/h88NLckW28zKh+7tdigpRs7A//1l1MkgSkZ0mZUzi+xUAQsCQamBI1Kcd2S
+2ZPzWb1BbleDL8+iXJyjG/vRQMJRFL14+WqT2dLT3RUK+AVRUKnU9rSM7Lx8izVl7MdPfKDCMGTS
+MXeszZ2VZ2rqCjg8UY4TSRJPMSkLM/W5aVqKxBFCSoZcOSfVblZVNXr6vFFJlDNTNGV5xtxU7UA3
+T16a7tv3lssXJzlk2NTf/2Kl3aRKMZ1/sNKp6TXzUzUqqqU7qKAJpYLEcezhzcVr5qWlWS+K/ASB
+F2TojVqmptXX2BmIxAQFQxRm6ucUmk06Bv88Oq6otCsZsrEjkGFTI4QMGvr2NTk6DRWK8BoViWEY
+Qqgs1/ilm6iaVl97bzga45UKMtuumZ1voikCIcRQxE//cXGaRWWEqShAEjjREXy/2rU4R59vUVo0
+0KIwE1A0XVYxt6xi7kQcHPP5fAaDYayH6d2B2t+c9CsDhmOcg4q+kehCgIkVCASeeeaZJ598kiCG
+9rb6WeSf2qsMusP8zz9r/7TWW5qi+vqKjDWFhkSXCEwtNI60cujpp59+4oknCILAcRzanQAAk+pQ
+W6CqOxyMCzV90dPdoSgnjf2YYGaDQAUAmDzuCL+t1tsdYBFCobhwpidc54wkulBgqoNABQCYPPub
+/dU94Qh7PkX2THd4b5M/LkClClwOBCoAwCQJseK2Oq87cmEKaXeEP9MdbnbFEl00MKVBoAIATJK9
+Tf5TXaHIxeP66/oiOxt9ojTJ86CB6QQCFQBgMvCi/H61KxAfOvuMI8id7Aj2hfhrOipIChCoAAAT
+TpLlPU3+Y+3B2HDTpDW74x+dcye6jGDqgkAFAJhwgoReP9kXGyETvTvA7m7yuSdmsR4wAyR+ZgoA
+wMwmSrIzxGUYmFsqrP0TxziCnCPAGlWkVUMjhARJ1iqIk52h60vNiS4smIogUAEAJhaOYzYN9dXl
+6QPTmx3vCB1tD8zN0C7K1iOEZCRjGKZlYG1rMLxxClTGuUiZluhzAQghhCjt2I8Bpi8NhabkDR/P
+1NED/8AkgeW4ylTFgnSY3BIMhWOIi160ZZwCFWNGDNTZAUg8EkdXWig78cpsCquaMChJJbTpgOEM
+6a6ErwkAYLLplYReORXrfWBqgkAFAJhsEU7kRFlF4czUr/2BKSDpviWSJEWisXAkOvZDAQCuTW1f
+7GBrqDsAg3zBqCRdjSocifb0uQRRsFstapVKqYC+XAAmW11frMUTt2jIPDP8AMGVJVegEgTR6fF2
+9Dh4QfQFQikWs91qVqtVBKzLBcAkigoSQWAMAb87MCpJFKhkWfYHQy6vP85yCCFfIBiORP3BUGaa
+Xa/VqJSKRBcQgGQhSrKawhUUBCowKkkUqDied3q8gWBoYAsvCL0ut9PjzbDb8rIzFQxNkSSGYYku
+KQAzXI6RQQgZIPEPjE6yBCpZll1en8vr5fih/beSJHX09Dq9vuz01HS7TaVQ4NASCMBE2lRqTHQR
+wHSSLHdkluP6nJ5QeMRkv3icrW9uO1Fd63R7eV6QYXEcACaMLCP4iYHRS5ZA1dbZ4/EHJOkKK14H
+g6Hj1TWna+sCoWCiiwzAjNXui/cEWRZWoAejkxRNf75A0OXzs9yVFxGQEVIyNI4TsTirUvI0RSW6
+7ACMVkBAMXHsh5kMP9vdW2xX3TjLrIUcJnAxGkemS+67Mz9QiZLU5eiLRKPyldoa8rMzrWajSqEg
+CIIiCYKAnl4wnUgyEqdJe1qQFaO8JE6fAoNJIw33lZj5garX5Xb7/DwvXHFPkiB1Wo2CpkdxVADA
+tYvzEoFjFAEZtmBUZngfFcfx3T190VhclmWSJIjLDjCMxWPQwwvAJDCpKZOKokkIVGBUZniNyu3z
+EySRarNoNWqGprp7nS6Pb6SdvYGgIE6TNn4AprM75lotGgqHMYtgdGZ4oFIqFTkZaSRJqpQKAsdD
+kajXFxBHyP2LxuIsx6uUMvx+AJhQ5WnqRBcBTCczPFAZdRctd6tWKhQME4nFht1ZkqRgKKxVq2lq
+hl8WABIowklxXlTTBEyhBEYpub4oGpXq8nP6BUJhQbhy2gUA4Jq5w/z+5kCXj010QcC0kVyBSqVU
+KhSXC1ShMAQqACaWO8wdaw85glce1whAv+QKVAoFo1Qwl+mCCoYiLMdfccQVAOCa+SK8EqZOB1cj
+ub4rOIYpGJphRhwpJUpSJBaH3D8AJo4gyyoKp2EQFRi15ApUCCG1SqlRqy6zQyAYYllolABgoqTq
+mHyLQgdrfIBRS7r0NqVCoVIqERpxNFU0FuOhRgXAhKnM0FRmaBJdirGSZZkbxfSh0wuO49SUnOA0
+CQMVc/nEP18gyLIsQtpRHxIAkHQ8bs87b76Z6FKMs8ysrE2bb0p0KYaRdE1/BI4zNE3TIz41CKLI
+spwgQKUKgAnR7mVdYY4XYY0PMFpJF6gQQiolo9NcbmC8PxSOsTDIA4AJ8cbJvn2N/sB0WZIETAHJ
+GKgYmtGoLpdPEYvF+UtWrAcAjIsOb1xCiKEg6w+MVjIGKiVDq1XKy+wQDEficahRATAhBFHGMQxm
+1ASjl4yBiiAIBcNQ5IiJJCzHxVlOhNw/ACaATkma1RRDJuPNB1ybZPyuYBjGMLR25NFUsiyHo7E4
+jKYCYAJ8Ya611K4mcahRgdFKxkCFEFLQtFZ7uZEc0VgsDvkUAEyARdk6m3YqDtYBU1bSjaPqR9P0
+5eeniECgAmAC+GOCKMlahoTlfcHoJWmNiiIJtVJxmZXpY7F4NMZKEgz1AGA8NfRFj7YHfVHIqgVX
+IUkDFYZhNEWplSPm/kmyHIuzLAs/JwDGU5snfqoj7IvBYjrgKiRpoEII0TRl0Osus0MkGg1Ho4ku
+JgAzSogVSRwjIJMCXI0kDlQUpVNfbn6KaDwei8cTXUwAZpQoJ9r1tIZO3jsPuAZJmkyBEKJIUqtV
+Yxg20jKJ8TgbjcVlWcZgZCIA46Q8TcOQmJJKrjU+MAyjKJKhaVEQYyw7+J7T3w1B05QoinGWG9wv
+ztA0RZGSKMU5bkh/OYZhNEXSFCUIYoxl+w9+6Z1KlmRO4Dlu2ndhJG+gwjCMJkmFgonFhq82iZLE
+8TwnCMyUnPcegOloTZEh0UVIAAVDL6yctW7Zoqa2znc/3RmJxQZeUjLM0gWVyxfM6XL0fbJzv8Pl
+Hnhp+YI5C+fMdvS5tu071Ot0Dz6gVq2cX162bMGcmvrm97bvnlNatH7FEgVDi5KEBkXBQCh8orr2
+wPGqRF+AsUreQIUQoijKYtB3xkZs3wtHY6FgmDEbE11SAMDMlJlmz0lPJXDcbNDPKikYHKh2Hz5e
+kp+dmmK5NO1Lp9WVFObF4uyJc3UDGzt7erfuOdjr9l7YT5alEVqMppekbikmSUKnu9y6UyzLRuKx
+UR8PAHAFuxv8ngik/F1gMug0apU/GBJE0XbxM7EkSX1uL47hWel23cUTFDA0qVOrnW5vd2/fwEZZ
+lgVRFAYTxZkxxia5AxVBGLSXC1SxOBuJQj4FAOPDGeZfPd7XG4Sh9OcZtBqbxSxKUpejzxsIWs2m
+eeWlg3eoqmkIR6KFOVlmg35go81iqigtkmW51+URk2NZr6QOVDiOMwzFXGERRZbn4QEQgHEQjAsx
+XhJnQlvU+MjJSk+3Wx1O99HT51o7utVKRU5G2uAdOh29Hn9Ar9MOXpdcq1an2iz+YOjYmXOJPoNJ
+ktR9VAghkiBMBr3j4o7KwWIsG4xEBj/OAACuTacnbtVSSpg86XNWk1HBMG6Pr6fXqWIYjuOtZpPZ
+qPf4Av07cBzf63Sn2iwFOVkur8/p9ioVCpvZpFYqm9o7A8HQRUczmzauWc5y52fTDkei9c1tDa0d
+iT7LcZDUNSqEEEEQRv3lghDLcuFwJNHFBGAm0CiIfItSSSdXbvpIstNT01JssTjrC4Z4QQhFox5/
+QK9RzyoqGLzb2fomrz+QkZpiMRoQQiaDLjPdHouzDS3tQw4Yi8fbOnvqmtr6/2tu7/IFQqMuzpSW
+7DUqAscN+svmU3B8OALzUwAwDgqsKi1DGlXJftvpl5OZbjEZ2jq7+1wehFAwHG7t6E5PsRXkZO49
+cmJgN6fH6/L6SvINFpNRqWCMep3NbHR7/W2dPUMOGI5E65pbe12eRJ/Z+Ev2bwyGYWqlkiJJXhi+
+I4oXhBjLiqJIEPAYCMCYGFUQpc5Tq5Q2s0mlUOh12tnF+cX5OQSBW4wGnMCNel1eVkZLR9fAzj29
+zpz0tHS7zeVJs5pNFEk63d7Bg7FmvGRv+sMwjCQJvU4z0uwTsiyzHB+OJNF3AoAJ0uaJc0JSZKld
+UXZ6mtVkcHt9vkBIqVDoNGq1UsnxfEe3Q6lg5pWXDp5moq3L4fb6bGbjrKL8tBSrPxhq7uhM9BlM
+Kni6QTiGWUxGrz8w0lxKLMsFQiG9TnOVBwYAXMCL8psnnQ8tsado6USXJfEKcjLUKuXhk9XHz9YM
+TI5DUWR+ZsbtG9elp9g0KmU4Guu/KXn9gV6XJzXFmpWeyvN8U3tne5cj0WcwqZK9RoUQwnHcpNdh
+aMRMJI7nQ5BPAcDY+GL8tlpfKC4muiCJZ9TrLGZjNMY6Pd7BU7jxvOD0entdbp1WXZCTSeAX7s/d
+fU5/MKxVqyRJdnl8iT6DyQY1KoRhmF6nJQhCHGEItyAI4WhUkmUcZqcF4JrIMurysVYNRRHJ+COS
+JDkUifb0OT0+vyRJ2RmpLMt3dvV6/YEhe8Zi8eq6RpIkUyyWc0QLEs/H9c6e3qa2DoLAe3r7Onp6
+h7wrEo05nG6fPzBSX/t0B4EKIYRIglCplHxQGLb1T5JljuNjcVY9aMwdAGD0JFnu9rFpBppOykDF
+ctyps3Wnzp6fl6/qXH3Vufph94yx3PHq2uPVtUO2x1lu39GT+46evPQtsizXNLXWNLUm+iwnEASq
+82xmYzgSEYTh2yU4QfD6/GqlPdHFBGBawjGsyK7CMKRmIHsWXDUIVAghhGGYQafDcRyh4QMVLwjB
+cDjRxQRgusIwVGxTFdtUiS4ImJYgmeI8k0E3uOtyCJ4XAkEIVAAAkAAQqM6jSFLBMPgIsUqWZY4X
+4iyX6GICMC2JkvzWKWeMh5Q/cC0SH6hisdgTTzyxbNmyrVu3RqNRhFB7e/uTTz656GJr1679r//6
+r9OnT19meZXPPvts0aJFd9xxx/Hjx4e81NLS8vjjj3/ve987ffr0sO/FMMxk1JHEyJUqQbg0RWcS
+uN3uZ5999q677vr4448n/9MBGDtZRsG48JudXWEWRvuCa5H4PipJkhoaGo4fP+71ekVRRAjF4/Gm
+pqbu7u4777zzpptuQghxHNfe3r579+5PPvnkP/7jP1avXs0wzKWH8nq9x44da2ho0Ov1v/3tb7WD
+1pqKxWINDQ0KhSI8cleTUafr6XWhQYt6BIOBIwf3N9bXr1q7ft78Bb5gMC3FOsnXh+f57u7u6upq
+j2cGTuEFkoEky71BnoThHeBaJT5QjUSlUpWVlV1//fUIIVEU/X5/bm7uo48++rOf/Wzu3LlW64gB
+IxQK7d69e+fOnbfccgt2NT8Ng05LXjyhH8dxPV1ddTXnyivnCqI4ZFJ9AMBoiJLsCLKZRgWBQ6gC
+12LqBqrBCIIwm81LliwpKSk5dOhQf52JpoeZiEWtVmdnZ2MY9rvf/W7JkiVWqxXHR9u8qWAYmqbx
+aEwadjSVJEVjLM/zFEWN8oAAAIQQhmFGJbU0X0fDSlSDzC8vc3l8HT0OhJBRryvIyWQYur65LRSO
+LJlXiRA6eLyK4/mB/YvzczJSUzq6HI1tHTazadGc2QMvhcKR7j5nR08vx/FXX5BpYHoEqn4URVVU
+VBw4cKCvr4/n+WEDlUKhmDVrVnFx8S9+8YuXX375G9/4hlKpHOmAgiD09fUdOXKktrY2Ho8zDEMy
+qrTsHJ1OjxPEqRPHjh853NRQ5/d59+/Z1d7WYjKaSvOy33zjNa/X+/jjj1+mVhcOh7dt21ZVVfXN
+b37TZrPhOM6ybFVV1dmzZx0ORzweN5lM5eXlFRUVKSkpA+9644032tra7rjjDpfLdeTIEbfbvXbt
+2rKysiEHl2W5pqZmy5Ytubm569evv0wxAJgKSBzLtyj1SkJBJr5TfOooyMlECHX0OFRKZUFO1qyi
+fIfTJQgiRVGVpUVKBdPd29fW2dM/2YTdZplTVpydnirwQmNbh16nKc7LPnamBsOQgmFyM9OzM1Jp
+imrp6JqROV/TKVAhhEZTPTIajV/4whe2bdv23HPPrVu3btasWcPWgTiOq6+vf+211xoaGlJTUymK
+CgQCbe3tlqbGhYuXpaaliYLIcZwoirIsCwLPsVycjfV5vJ/t3Ll927ZVq1atWLFCoVAghI4fP37i
+xAm73b5s2bL+sBEIBP74xz+ePHny0UcfRQiFw+EtW7bs2rULIaTVajmOczgcNTU1J06cuPfee3Ny
+cvqL9O677+7YsSMajapUqt7eXkmSeH7o85Eoiu3t7b/+9a937NjxxBNPjDSRLgBTB4YhnZLQjfzI
+mMxIgshKsxfkZARCoapzdb5AUKtRI4QEUawoKex1uvsDVUF2ps1iGpxKFgxH9h45gWGYgmGKcrNX
+Lp47u6jA5fFBoEownufPnDlDkqTVaiXJEUtOUVRhYeFXv/rVxx9//OWXX/7ud787uNYyoK+v7803
+33z11VcfffTRr33ta2az2ev1/upXv3r3vfcUSpXeoF+weElOfv5H773DcUfXrL9+5Zq1OI6FItFb
+b7117549x44dq6ioUCgU8Xj8vffee+aZZ+bOnavVatetWyeKosfjOXfu3IIFC/rbHvft2/fLX/4y
+KyvrscceW7FiBUmS/THy9ddfRwh997vfHagdxuPxY8eO3Xfffffcc09aWhpFUQ7HhWmSRVHs7e39
+05/+9PHHH99xxx133323zWZL9J8FgCvgRbnRFSuwKGioUV2MJIj0FFtJQY4so9M1DX1ub/92SZLa
+OrszUlNSLOY4y+l1GrvVLMtyNM4OOYIsy7F4/HRtfVa6PTcjTamYmdO8TY/vjSRJPp/v5MmTtbW1
+S5cuLSkpGTbrb4BKpdq8efPSpUtfe+2148ePxy5ZYUwQhPr6+g8++KCsrOyJJ56wWCwYhpnN5vvu
+u0+lVLa3NAcDw2Siy5IcjcXXrF6jUqmqqqpCoRBCqLu7u6mpied5n8939uxZ9HnWYjAYvPXWW0mS
+jMViL7zwgtPpvP3229esWaPT6VQq1dy5c+++++709PTXX3+9paVl8KeUl5evWrUqOzt7SEVQFEWH
+w/Hqq6++8sort9xyy/e///1hAzAAU02EE5/d3xMXJKj/D5Geal08t9yo051raB68UqIky929zkg8
+XlaUp1Epy4ryGYb2+4McN2Jtyenx0jRFkjNzhqqpW6OKxWKNjY179+5FCHEc19raunXr1tTU1Kee
+ekqv11/+vf1R51//9V8feeSR119/vbCwcMjoq2g02tTU1NHRMW/evJMnL5rnUZKkaDQ67BdCRkiS
+ZbPVmpOTc+bMGZ/PJ8tydXW12+3ur13V1dXF4/FIJHLu3DmdTnf99deTJNnZ2VldXZ2fn19YWKhS
+XZhCxmq1VlZWVlVVHT16tKSkZGB7UVHRpScYi8XOnj3rdDp/9rOfbdiw4d///d/7u74S/VcC4Aok
+Gfmjws563//emgf56UNkp6cihHX29Hm8fgLHhyzgcLauqbK0KDPdnmG3uTw+WZaz01NHOpQgCDO4
+I2DqBiqXy/Xqq69u3bq1/58ajWbDhg3/+q//WllZOZobNEmS69atu/nmm1988cWVK1fOmjVr8Kvx
+eDwYDIZCoa1btx49enTwS6IoFpSkDJupgRCSZdnp9t5ww8bf/e63nZ2dRUVFhw4dUiqVmzZtamho
+aG5ubmpqUigUp0+fLioqys/PRwj5fD5BEAoLC41G4+BDqdVqu90uSVIkctFiV+np6YPjWb++vr5X
+X32VoihRFO+8806DwQBRCkwLnCB1+OIFViWkpl/qaNU5QRAqy4qWzCvfd/SUy3vRQlMnztQU5mav
+X74kHIkeaT9r0GkuE6g0ahVCaKbGqqkbqLKysr73ve997WtfG8tB/vEf/3HXrl3PP//8Y489JooX
+pm/BMAzH8dTU1B/+8IdDPsIXCJ5raB5pEgpZlv3B4OrVq//ylz+fPHnSbrfX1dWlpKSsW7cOIXT4
+8OGDBw+Wl5cfO3bsoYce6n8LjuMYhvE8P6RWJ8tyf5GGDPbSaDSXZn/k5OR897vfzcvL++Y3v/nf
+//3fRUVFZWVll+moA2CKECU5GBezTcxVDWpMEnGWO1ffhOP4nLLiBRVlOw8eYy9uy2lsbdcoFadr
+6x1Ol2GERcZxHGNoOiPVHgiF2Rmanj7Dn8qLi4u/+tWv9vb27tixY3BWglKpNJvNsVispqZmyDOI
+VqOmRx4pJcuy1x9cvny5Vqs9e/bsnj17nE5nYWFhcXHx7NmzU1JStm3bVl9fL0nS2rVr+99itVpp
+mq6urnY6nYMPFQwG29vbSZI0GAyjORelUrl8+fLf/OY3NTU13/72tx0Ox0x9egIzCU3is1PVN822
+EjP8ZnON4hx3rrGlvqW9pCB3zdIFQ1pKDp8884eX3jxadTY6aCHgfhiGkSRJUZTZYLhlw5r0FGtN
+Y2swNDMnJZj5350HHnigoKDgww8/rKurG9io0WhKS0tzc3M//fTTzs7OwfuTBKFQ0Jfpk+QFIRqP
+b9iwoa2t7c033zQajbNnz1apVDk5OYWFhSdOnNi2bZvdbl+zZk3//hkZGStWrGhvb6+pqQkGgwPH
+6e3tPXbsWHp6+oYNG0Z5LkqlctWqVT/60Y+OHDny1FNPdXV1jfKNACQKRWDZJsW6Yj3MoDSSYCh8
+ura+trGlOD9nybzyUb4rLcX6L998+Aff+PLXH7zLbNTvPnz8dG1dOBob5dunl5nfdmQ0Gh9//HG3
+292fkjegqKjoscce+9GPfvTEE09861vfWrlyJUmSR48e3bNnT25hkc5oEZCo1WgtVls0Eqk+XTW7
+otJoMvW/1+X1L1u+fOvWrS0tLStXriwuLkYIZWRklJSU/OlPf+I47rbbbht4MsIw7Fvf+lZ9ff2f
+//xniqJuu+02mqYPHTr0/PPPBwKBp556yvT5YUdDpVI98cQThw4dev/99ysrK++//37IUAdTHIYh
+DEGUGmrL9t3C5/0RvkBw16Hj+46d4gWB54W/vfk+zwvCxevKn6lrrG1q7e8vaO3ofvq5FwdekiSJ
+4/mR1n2dAWZ+oMJx/Prrr9+7d29fX9/g7QaD4Y477sjIyHj11VefeuqpQCAgy3JRUdHSpUuzM7M4
+GcW5AM0wFXPmet3uk8eP/uv3v221pvzn//4cIRQIBhcuXKhWqymKmjNnTlZWFkJIpVKVlpbOmTPH
+5/OtWrVq8Gfl5+f/9re/feedd7Zs2fLss8+Gw2Gj0bh48eKHH3541apVV5UWgWGYVqv93//937vv
+vvvnP/+5Xq+/7bbbhqRpADB1eCP8lmrPnXOtGljb92KxQYOiJElmOW5gqG5kuIoRzwv851NmC6IY
+Ckeu/BkzBebz+UbZRzJBRFGsra31+/3FxcUmk4kgiFgs1traGolEMjIyUlNTR38ol8vV1NRktVoL
+CgqGvNTW1tbb26vVajMzM3U6Xf9GWZZjsVhXV1cgEIjH47Is63Q6q9Wq0+kb2jq6e50IIZ7ng36/
+z+dlWZaiqKKSUoSQUsEsnlPe3NQYjUaLiooGJjEKBoOtra0cx+Xn5w+pJ4mi2NfX53a7w+GwIAhK
+pdJqtVqtVrVaPbBPXV2dx+MpLy8fKCFCqH8aC5fLlZWVNVB5EgThzJkzkUgkLy/PZrPB9IPJJhAI
+PPPMM08++SQxaBplH4+iU+yRWpLkOmf037a0vflY2Qybkdbtcr/z5puJLsU4y8zK2rT5psSWgcGR
+hUahUOjpp59+4oknCILAcTzxNSqCIGbPnj14i1KpvHSCu9Hov/UP+1JOTs7ATEUDMAxTqVRFRUVD
+tsuyrGBogiBEUaQoymy1mi8+LMtygijOmTNnSCKTTqerrKwc6TTT0tLS0tIuU/7Bo6kG0DSdnZ2d
+nZ09eCNJkvPmzRvjlQdgosUEqcsXR7I8w6IUGIxj2W89/MW8ouJH/+GJgVul3+fd9sH7xw8fWrB0
+2S133q1QqsbyETM/meIaYBimVimVI09+IcmyPxjiZ26LMADjIsZJjgBXYh/TTQpMcaIobv/4g2MH
+D8Ri0f4tfp93y1uvv/XKi/F4LCevgCDH2uQDgWp4GrVaqbjcLE3+YGhIVycAYAgSx1L1zIbSq0gX
+AtMdy7Kfvv/uR++8lZaeef8jj81dtHjsfROJb/qbmtRKpeKygSoAgQqAK1Ez+NxMLazukTw4lt27
+Y+vWD95LTUu/58uPzF+8dFx60CFQDY9haAVD4zg+ZDqJAZFYnON5WZZhvD0AI6EI3KqBKJUsOI7d
+v2vHS88/q9ZovvDAQwuWLKUoeuyHRRCoRoJjmFKhUDD0pQPC+0mSFI7E9FotRcE1BGAYgih7o3wg
+LhZaYSWqmc/tcn76/rs7P/1EEPjb73tg7sJF4xWlEPRRXYZGpVSrLvcD8wWDLD8zZ9YCYOyCcfFo
+e2h7rW/shwJTn6uvd+uH77c1N2bm5GVm5zDMeK6MBYFqREqFQqm4XKAKhSM8BCoARuCP8c2uGAMd
+VMmhdHbFz5559p6HHjlz4thH77zV0dY6Ur/JNYBmqxEpFIxKyWAYNtLcr6FIlON5GcHkMAAMI8yK
+zhBXnqYe+6GmIIIgdHrd2I8zGqKEcAxNQm/4pQsMXRW1RvvwNx/v6+3e+sF7eoPx/ke/YrWNz+Ku
+EKhGROA4Q9M0RbEjrKopimI8zgqCSM3QVTUBGIsQK3rCfI55Zi6ObjQZ733ggcn5rG4/q2YIvZKc
++s/EFpvtm//0pN/vf/WFP2u02oe++g2KHoeeKqiVX45aqdBpL/c86AuG4iw76uMBkEQq0jTfuy7L
+ph23HvWk9fejfbsa/GF2eswwkFdY/Og3v5WTX/D2Ky9+8v6743JMCFSXwzCM6rLdVJERFq0HAGgY
+Is+iUMNctGNzvD10qCXwxgnnme5wossyWguXrbj3wYdxgnj9xb8c3rd37AeEQHU5KgVz+cS/UDg6
+U5fUBGCMMAzhMMxwbKKc+E6Vq80dO9MdOdsTDcWnR6WKJMlVG67beMutbc3Nz/3+V7XVZ8Z6wESf
+0ZRGEISCoWmK5PjhJ6HgBYFlOVGUCFi+FIBBzjkipzrDy3J1eTCIagx21PkOtwYDMVFG6FRnaHGO
+dk6GZuyHHV+MQvHuzgNqjdpmv7DYhcFkvvdLj67beBPNMGnpmWP8CAhUV6BQMFqN2uMLDPuqLMvB
+SDQWj2vUMO0mABf0BblTXaESuyov0SWZvmK8+O5pd6efFWUZIXS8PXgmTz8FAxWO4xXz5l+60WJL
+sYxT1h/UA65AQdNa9eXyKSKRCORTADBEhBWjrJSmh0yKa/dpje9cTyTOn2/uc4X5M12hNs/MXGz+
+8iBQXQFD05evLYVjsTgL+RQAXBDhxBAr6hSkFVL+roksI39UePuUyxvhB4ZxipJ8sjN0pC2U6NIl
+AASqKyBJQqlgyJFHSrEsF2c5URy3MdgATHc4hhXalGuLDRSsl3hNBEl+65TrdFeIFS66sbR74yc6
+Qr3BpHsyhkB1BRiGMTStGXnAtizL0XgcKlUADFBSeGWGZmW+PtEFmZYESW5yxf5+tDcYF4dMisMK
+co0jcqIj6SpVEKiujKYpvU57mR3CkWg0lowNxwCMhCZwGEF1bWKc9NFZT5snLkrDTN7W6okfaw/G
++eRqwoGsvyujKery+RSxWCwWh3wKAM7r8LIhVihJURHQ9HeVZBnFBandGy+yKQVRRggF4yIrSP1z
++wqSLElyuyfe7I7NSp2ZkygOCwLVlZEkqVYpcRyTpOFnp42xXIxlYRFFABBCvChvr/O2umP/tikH
+AtXVkmWZxNHtlZYbSo0xQZZldLw92OVn861KNUVEeVEQZSWF9wQ4CFTgIjiG0TSlVCgi0eHb9yRJ
+YlmW5TgFw1zlsQGYaVxhLsZLNi2toKBn4arhOGZUUeuKjQNb1BRe74xeX2qaPUPnoR/VZUl0AaYH
+hiTNhsv1DIejsVA4muhiApB47rAgyXKqHh7axodeRRZalQZlUlcqkvrkR48kSZ32cgPC4ywbi8dH
+fTwAxh82NZZGE0VJxxBpenoqFGYGSLbkyWG/NhCoRoWiSJ32col/0Vg8HIXEP5BIegrpqUQXAqFN
+hdqNBRoMIRzaa8A4gUA1KhiG0RSpYOiRxktJksTxPM8LFAWXFCTGFKnBEBiajMVok0a1I5KhZ4yq
+pL6xwDPPaFEkYTYaLrNDNBYPhqbNgjEATARWkFiYpWVcPXOg70BbKMol9VWFQDVaBEEYdLrL7MCy
+bDgK+RQgqVU7oi8dd+9vDSa6IDNHfV9UkmRmGixDP4EgUI0WSRCGy85PEWc56KYCSa7OGesKsBR0
+T42TLj+nYQitgkjyEWnwfRotHMeVCpqmR+ytFkQxzrKCIFzNUQGYUVo8cVGS02F1j3ESZsUiW7Ln
+piMIVFeFJEiDTjvS9BOyLMc5LhSB1j+QpDhRIjDMrqUtmmS/sY4Xm5a6e44ly5jsg9Lg+3QVcBw3
+Gw0uj2+kHViWC4YjRr3uao4KwLUQRXH//v1TatYuXpSp3jDGEEcPwoLX4+lcW6JLMOnYi1ejhUB1
+FXAcv3wQ4jg+FIkkuphg5qNpet68eeHwlMsynWVCCImhUNKtQzFBWjzxVB2tTMrJqBYuXDjw/xCo
+rgKGY1q1iiCIkTqieEGIRmOiJBHQmQwmklKpvOmmmxJdiosIkuyPiUoSVzPw5R8fvCj/bGf35vnW
+XFOSNv35fOebr+ArdRUwhEiC0GnUOD5iNxXH8dEozKUEko4vKuxsDDS54cs/bpxh7sManyfCJ7og
+iQeB6upgGGYzGy/TMcDyvC8Ag0hA0mnzxj9r9Lf6IFCNm7q+mFVD9a9EleSg6e/qYBhm0OlwDBfR
+8APFeV4IhcMIIVmWJUkSJQnHcILAp1SnNwDjrsXD2rV0qhYS08dNu48tsCjUNCyUDIHqKmEYZjTo
+cAJHIwyX4gUhEArzPM8Kgs8fDARDWq0m1WqmqSkwXSgAE6bRE7eoyRQtfM/HTZZRYdfSRhUEKghU
+V48iSSXD8BwvycMs+CvLsi8Y2n/sVP+AKqVCkZ2RRpFwncEMN8umtGkpsxq+6uNmQ1FyLfBxGfCt
+uhZWszESi0n88LUqURT751LCMCwvO0PBMNDuB2a8m2aZcAwl+Uw/YIJAN91VE0SRJEnssosqyLKM
+Y5hRr8uw20gSau5g5qMJjMThiWx8iJJ8tCP0XrU3xIqJLsuUQCKEHDBE9UoEQYiGQxwbjQYD8XiU
+5Xn+CnP6YQRJay3pbpbCIbkUzHR7m3x2HZNlUiggRW08cCLa2hCOC1KRXWdMslk+aByZlUM3kggh
+iNlXJMm4iAh/IBjweiTpytcLJ3CtwaTRm3gZQ3B5wYwW46VXT7pumm2x6hRQpRoXrCAfbg+tzDeI
+CEu2+/Ow3yB4/BkVHMcZlcpkS7OmZVEUffk+JwzDGIXSZEslIIcCJIFuf9wfE5UUwRAQpsaBLKMI
+K9Y4IiUpKkVyL0M1AO6ko0UQpFqrpxVKRqEI+rxBv0cSh2/9I0hKozcpVZpEFxmAydDiiafpabOa
+hEyKcSEjJMnyuiJjjlFBElCXQAgC1dWiKNpgTmEUKoVKHQ74YpGQeHG4wnFCqdIYzDYMpvsDyUFF
+44uzdSYVjKAaHxiGNAz5xYUpWgVUUc+DQHUtlGoto1Sr1JpQwBfye9l4VP58TBVFM3qjBapTIHmU
+2FSiFcHifuMFQ0hB4bmXZhQkMfhuXSMcxzV6k0KtZZSqkM8TjYR4jiVJUq3VawzmRJcOgMlj0cC0
+SeNGllGUF+v7onMztVCdGgDNU2NCkpTJmpqSkWuypqo0OpVGpzdZaCZJ5+QHSajLz3YHWF6Uxn4o
+gBDiRanFHXvjlHPYiW+SFtSoxoFCpWYUSo3eJAqcWmdMdHEAmDwfnnObVeR1xSaDCp56x0GEk050
+hhgCQzJCUKX6HHy3xgeG42qtTme04JBDAZKGKMtVXWElRRCQnDYeZIT8Mf5wW2BZnh7mXRsMvl4A
+gGshy8gbFep6I2kGmJBifIiSHIyLnrBQmaGFODXYODX9yRJMwADANIfJ2FXcEARJPt0VzreqMvQ0
+BXnU44HEsWKb6ie35tlhWa+LjU+g0vl3G/veSfS5AACuHcekOXJ/OPr9KQJbV2RI1dN6GEE1fhQU
+nm9Ostn9RmH8alQyzL0KwDSGycLVvgXHsGKbCiakGF/Q6HcpaFkGAFw1SZYjnIgQgqU9xkuYFXc1
++v5nW3uiCzIVQXo6AOCqRTjpo3Melpfum2+jIZNiPHgifIMzZoIlkocD3zAAwFWLcuKuBq9JTUIW
+9XjpDbK1feHyVJh9bRgQqAAAV4cT5e4A6whyS3J00EE1LlhBCrOiliHK7JBJMQyoZgIArg4nSK4w
+X5KitsIsf+MEx7ACq8qgooyQQjkcCFSIRfSn1IZEl2JKuJHfTqEZlb0Z5tBv9s7AX75OgR5fkbC/
+FElgOSbFXXNtib4MMwdFYNkmRbZJkeiCTFEQqBCLMW/Rtya6FFPCBmE3NbOGGYRZ7N8+nYFP/ZkG
+OYGBSkHixTZooRo3EU6McpKSwjUMkeiyTFHQRwUAuAqCJIfiYpyH6dLHTasnfrg10BfiEl2QqQsC
+FQDgKviiwrHOYE8A7qrjgxOkM93hYx0hWNfjMiBQAQBGS5ZRd4D96KzHE51RTcQJ1BviOnxxFY1n
+GaGDakTQRwUAGC1WkFxhLhjnc6Dbf5z0BjmNgiyxqRgYNz0yCFQAgNEKxARXmC+2qayaGZhLmRB5
+ZoVeSRqUkEZxORCoAACjpaDxkhRVvlmZ6ILMHBYNbYHhaFcCgQoAMFo6hqxM00CnP5hk0CoKABgt
+DEMEjpEwbdJ46A1yv9jZsaXaleiCTAMQqMafv/HsZ1+9vn3rG0O2N7/3l13/eHPL+y8kuoAAXIsu
+P3ug1d/mjSW6IDPE4bZgnJdMamj3uzIIVONPiIa9dVVxj3PI9pjL4as/HXM7xnJwiWPbP31j9z/e
+HHP3JvpEQXKp643sbvTDaJ9xwQrSWUcow8CUpsAcH1cGgWqakSUp3N3qOLxDZOHBFkyeUFx0BDlR
+klO0UAMYB76oYFbTRTaVUQWJAlcG1yhhJJ6LONqdJ/f7G89KHKuyZ9jmrzKVziMYBUIo6uxynTro
+q6viQgFGbzKVzTfPXoBT9Onf/cjXcEYS+NO/f4pS67Ku+4KlfBGpgjVsxtn371ubk2oa+KcgiMEo
+29HnO1HfdbKh69L9Z+XaF5Zk5qSaDRoFRRK8IHqC0YZO19Yjdf5wDCH00MaFcwrTGGroL84fih2o
+bv34cG2iz/gKugOsLyZkGRQqGhKpx4GaIa4rNpnUsELyqECgSgxJFPwN1S0fvijEogpzCqM3ckF/
+w+t/yFx7q33pBiTL7Z++7q2tUpisKluaLMtRZ5fGn6u2ZyqtqeGeNgzDFOYURmckVRoMh2rx+Ltj
+dcWs3JT395+ra3cihAwaxaxc+5q5+WvmFry958w7e84M3nnDgqK7182ZlZPiC8caO1097qBKQRVl
+WkuzU3o9wT1VzQih9fML71hdfqa559DZdl/oQm04FI2HomyiT/fKdAqiMl2jV0CUGh9ahtBaIct/
+tCBQTQgxHus9ulPiL5oPzXnqgBAJ9f9/3N3Xvf8T56kDsx7+furS6yiNPtTeUPX7HzW++SdVaial
+1vYe2UVpDXm3fMlYXMkFfVFnN6Mz0XrTrEeerHv5t84Te4vv+wdNem6iT3QmC0XZN3dVbdl/DiFk
+NWjmFaXfsLjkluWzM21Gpy+8/0xL/26VBWlfv3XZgpLMPVVN7+ytPlbb0eMO6jXKuYVpBRnWGHvR
+VENHazp+/87+Noc30Sd31dL0jFlNiRL0UI2Dur5ouoHRwlzpowaBakLIohBzOfxNZwdvjHv6JOH8
+bSvY3uCqOqDLLkxfvZlUqBBCutySrOvuPPGz7/oazlhmL8JpBskSHwkJsSitM9I6Y6LPKam5/OGt
+R+vrO11IRo9sXvyN25YNBKoHrpu3vDzn4Nm2Z945cLS2o39jIBzbfap596nmRBd8PMEcP+OiN8R9
+Wuu5pdyiZaBGNVoQqCYEqdbm3vRA8f3/OHjj2ef+u+GNP/X/f8ztCHe3ZRaW90epfqbSeQSjjHS1
+psxfbZ2zzFt7svfQ9mhvp9KWrknLVtrScBLmrUmkNof31R0nH7xh/rLZOWkWvcMTzLQZ1swrJHD8
+7T1nzrSMKZ9zKuv0s5wgpepo6KAaux11Xn9MQFA1vRrwiJQYEs8KsciQjUqLHSdJIR7FKarg9kcL
+bntYEvm2j18599eftmx5IdTWkOhSAxTnhR5PyKhTrZ2XT+DYLStmZVj1Dm+o2x2IszNzQnFOkD6t
+8Xx0zhOIC4kuy7TnifC7G/0r8vQpOkievApQo0oMWmtUWVNFLi5LIoaff0r11pwU2JjSYicYFa0z
+2JdssC/ZwPpcHdvfbvngRTbgXfDkrxFCA/uDyccLotsfyU01GTUqHMMKM60KmgxHWZ4XR/N2s15d
+lGnVKM/fpDhe9ASjnkBkNO9NlO4A2xfisk2KVB2T6LJMb5KMTneHc8yKWXa1GuqmVwMCVWJo0nP1
++WWBltqos0dlTcMIQuRY95nDGMJ0eaWEQslHQzhO4LSCNlgsFUucVQe9tacQQgjDGIMJYZgYj8mS
+BCl/iaWgSAy7ij/B7avKr19YLErnl8ft6PP/5eMjz205nOjzuJwGV8ygpEpS1IkuyLSHY2hdkXF2
+qsYEY6eu0kRdL0GUeEHGMURTxJBxAqIkC4KEYYimzj9TSJLMCZI83JB3AsdIApdkWRBlAseo4bpz
+ZRkJoiRKMoFjJIlLoswLEo5jNDV1b+K63JK05ZtO/9+/1/7t6aL7/kFlTe09uqvp3ecz191qKp0b
+6mh0HNymsqbZFq4hGaW7+nCwtS5t5Y0IIZwkTWXzSUbZsf2t/NseVlpTMZxAMBRjEmEYhhDq/7b2
+eIKCII7+6j//weHfvLVvemX9heJimp7OMUF1akxkhGRZxjHMpoWe5qs2IYEqHOP/762a//7byeIc
+46v/b11+um7wq9uPdP378ycsBsVHT29ECAmifLTGeceT24KRYZr4r1uc8U/3l5+sc//sxdOP3lLy
+X19bcOk+/lD8F69Uv/hxw5c2F3/v/or39rZ959eHrl+c/sp/rsen6uyZBM2krdxIG0zN7/1120Mr
+hFjEWFhe+tD3sq+/k9GbaI3Be+5460cvV/3mX4R4VJNVmLPxnoLbHkYIYQRpKJw967F/aXzzT+f+
++rM5T/xv7o33MQZzok8oWSgZKjfVIMmyKxiSZLmqoSscZ3VaJUPP2Gfk2yssCKEp+1OaFiRZ9keF
+U93h1QUGmNL3GkxgjSrGilX17jc/a3lkc7HNdCERU5RklhO5QW36kiTHObGiwPSzxxcXXBzVGJrQ
+qCgMoUhceOnTxu/eX27QMEPqDzVtgbPN3tx03Zp5aTJCoijHWZHjpURdU1PZvM3vnKHU2iHbix/4
+VsEdj5HK800olEqbMn+luWy+yMaRLGMURSrVBKNEGEaqtbmbv5h13RckgUeyjBEUqVASyvP5gRhO
+FN39jbybH5RFkdLoCAXMFTZJFDSZbtWbdepeb+iD/TWCKG09Wt/p9FfkpebYTWolHYlxY/+UqYaA
+G+uYheLi61XORmd0eZ4eAtU1mNjHQF6Qnn2vbvEsq1GXSl1pEAZN4TajMs06TFN4mlW9aWnmtiNd
+h6r7blicQRAXHepEnetknXvV3NSKApNWRd2+JmfJbJtGRSXqGRCnaKXFful2SqWlVIOiF4bhFENT
+w7SoYBhGKtUDIe1SpEpNqqDPYLIVZlgfvWkJL0rv7DnTP5I3zgkfH6zJsZseu3lJR5//sxMzKjMz
+xkvnHJE0PZ2mh3a/aydJcl+I21nvfWxZGkSpazOxvTgPbiyMxPg3drS2OUJjOY5Zp7h+cUacE9/e
+1SqIF3VlOX2x+na/TkMvq0gx6RgCx4xapizXmJUC09+BcZNpMzx846L/+sqmRaVZ+8+0PvPugYGX
+Xvj0+GcnGnPspn/70obHv7CyIN2CEMq1m75335rnn7z7kZsWJ7rs1+5EZ3Bng9cRnIHVxMnkjQl7
+mwNWDb0sVw/V02szsTWqtfPTwjFh98meBWUWq1FpuNYVl7Uqal6JJc2i2nq4q8cdzbJrBv7e51p8
+DZ3BnFTt3CIzgWORuHCouu9vH9bfs6Fg84pMDLIMwLUy6VQ/fHDDIzcuRgjRFKFRMnGO/9snR9/a
+fabT6R/YzeEJ/uLVXU3d7rVz8x+8Yf4tK2ZFYhxDk0qGanV46juc11yAhDvREWJIXK+Ysd1vk4Mm
+8AKLstim0jJwJa/RxF44g5Z+cFPhj/968r097WU5xqXlKdd2HILAUs2qDYvS/7yl7nity25WKj//
+k59qcDtckc3Ls0pzjQghXpA6esMfH+xcWGqTEYIwBa7Nj/++3Wa4UCkXRSnG8p5gpMXhvTRn71xr
+byASP3Cm1WbUKGkKxzFBlELReJcrUNPW17/P8x8e3n68vqHD5fZP6VFTA5xhrqY3cl2JCdb1GCM1
+jc/N0FDE1E1CnvomPMIvrUjZuCTzvb1tnx3vzk7VpllG7Plv6Qn99MXTpkGDCucUmjYuybIaFQgh
+nZq6eUX2c+/Xvr6jed2CtP5A1eWMVDV4lAqyvMBk1EIzOhg3Hx2suar9u5z+rkHVrEvtO92y73Si
+z+pq0AS+KFtXalerYe7UaxXnpZggGZWkXgl1qTGZ8CCvVVH3XJefbddsPdR1+GwfL4yYjIdjmIIm
+lMyF/2jywhgshiYqCk0lWYbdJx1dzkh/T9XpRk9TZ6Ak21BRCPnZAIwng5K8udySZ1ZAr8o16/TF
+t9d5G13RRBdk2puMOF9RYLp5ZfYf3q7ZdqRrVp5ppN1yUjVP3DO7JNsw7Ks4hhm1zO1rcn/+0ukj
+55w5qVq9hj5R5/KFuPJ8U97FSe0AgLGzXWunMkAIhVmxpi9yqitUaocBJGM1SRXSW1bmVDV4jte6
+tx/tshoU13YQhiZuXZ39+zfPfbi/44YlmYIonW70WgzKWXlGdUL7e2VRiPs80d6OmKdXiEZwgmQM
+JlVKpja7cPBufCQYcXRGHO18OIjhOKnSKK2pmrScYZfwcJ85LPG8Lq9EYbQObJQ41t9cE2pvSF+1
+WeTi7jNH+EhwyBtVKRm2eSsSeDXADFDVHTaryBQdQxNQn7oWkiw3uqJNrtjsVE15KmQgj9Uk3d/T
+rapNSzPPtfi2He4qLzBd20EoAp9TZCnNMeyr6u3oDXuC8TZHaM281Nl5iVyrSRbFYFtD37E9vobT
+cZ+zP1DReqPSkpp13Z3G4kqCZhCGCdGw4/BnzhN7Qx2NYiyK0wpKq9dm5FvnLEtddt2l63c0vP4H
+NuCb9cg/Dw5UXMhf9/dfOY5+Zp27IursPvHz7yIMKa0XLf9hnbscAhUYC0GSnzvYvanUvEpJ0QR0
+UF0LUULOEEcR+Op8faLLMhNMXkVk9bzUqgbPm5+1dPaFo9e6XgCOYffdUPDvz53YW+Wob/eHonxp
+jtFuTmTNOtrX2fzun731p1OXbCj70j9pswv5aKjv2O6GV59xHNq++Kk/mUrn4hTtrTtV87dfqMz2
+kvsft1QsJRhF1NntOXss1NFgm79yNAtNyZLE+r3OqgMZa2+l9aaY24EQKrzr63mbv8gYLQm8AmAm
+kRHqDbLOEJ9mYBQUVKeuEUVglRnaAqsqw3iNDUhgsMnLmDRqmU1LM0tzDdXN3sbOwDUWF8NuWJKh
+11Dbj3R/fLCzNNdQXmBmEjdhviyK7dve6j262754Q8kDT/S39VEqbdqKTfP+6edcOFD74q/YgAfJ
+cu+xXVFHR+ryG1KX3UBpdDhFa9Jzs2+4u+SL3750sqVhCWw00HKO9bvTVmyEFRTBBJEkeV9zIN+i
+SNMxNGRUXxNZRrKM7Fo63wJr+I6PSf0iLp5t27g0M3UMFSAMQ8VZhnnFluN1Tl+QnVtoybAlsjoV
+d/c6T+1XmGzWyiWE4sKXEidITUZe3s0Pdu58L9hSJ/GsxLGyJEmCIIujWrjoUkI07Gs4Q9CMbe4K
+HBpkJgVFEloVo2QuPBYQOK5VMTrV+cdkrYrRaxTkxTd0miINGqVaSSOEcBwzaVUD/+k1ChVDEVN4
+cRZRRo3O6OpCoxay0q+JKMn+mBBmYZHJ8TTZP5gblmTee33+GA/yhTV5Ri1jNyuXlqekWxM55Z2v
+8Uzc3avLKdJk5A15Cado++L1OEH6m84K8ZihsIIxWtynD/obq6/ts1ifu2ffJ8aiSkZvgnU9Jse6
+eQUf/vSxf75/3cCWwkzLG//vS1t/+dX+f37w08eOPfedBSWZg99177o5NS89+Yd/uhMhlGk1eD7+
+8cB/x5//7n8+urEwc+o21dIE9tSm3BV5eiU9daPpVFbTG/nx1ta3qlyJLsiMMiF9VGoF9d37Kr5x
+R5laRQ0Zj51uUf3nVxb8y5fmDsxRSxLYktm21nfvIwlcNbrkvbvW5920IgshdGmyn15NP7Cx4I41
+OQqGxCf+bi7EYxJ/hQXIY+5eieey1t+OkNz01vP7vn+vIb/UtmC1Zc4KfW7xKNv9RDYW7myO9nUV
+3vXVwdtP//Zfq//wn+jzJ/Tcmx5Y+C+/neizBoNlWA13ra10esMtDg9CaPWc/Puum6dRXhh+Ho6x
++ff8jyCIdrP2H+5Ycc+6OQaN8rdv7a1u6U102UekYUh4FLoGMU587aTTpKLWFCYyw2vmmZBAhWFI
+wRCK4ZoOcBxTKcghAYkk8KuaV4KmcJoafoQHhiGGIhhqslothlvscegukoQQwik6Y/XNtnmrwj2t
+vtoqz9mjrR++rErJLH3w27YFq654EDbg89aexGnGNm/l4O0lD30nc+1ttP78r2KUYQ+Mo8Yu1+o5
++R8dqu0PVKU5KaXZNlcgPLCDLCNfKMoLYjAa//c/f0oSeEV+WklWyhQMVKwonegILcnRTcJD3oy0
+vd5HE9iKfEOmESbKGU8wsceYEIwCJ69wDRVGa3/uQ//KHQqjWZ9Tkr5io6/xTOuHr5z5438u/Jff
+6fPLLn8QLuANtNXr80qHtDEyBosmPQey/hJo96mmFRV5GxYUtTu8GTbDqso8fygWjg8z47ggSu5A
+5ER91+KybJtpyj1SRDlpf3PgzVPORVk6HPqnrkmBVZlrVmQaFbCcx/iCZugx0WUXMgZLxNERc3YP
+eUkWBM+547Ik6nKLCcWFFFWcYhiDWZOZbylfbCyuDLY1eOtOXf5TZIGPONp89WfMsxbiFEwWMLUE
+IvGTDd1zC9OKs23zijI0SuZYXefgdUGHcPvDgiiSUy+hLsKJR9uD6QYG5nK+ZnlmZZFNpVNAnB9n
+U+7XMr2o7FmmWfNjLoen5oTIsQPbZVGM9nW1f/q6de4KfX4pTinEeEwWL0oEwnCSYBQIw3D8CnUy
+NuD1N51DkpiyaE2izxgM49UdJ8Mx7mu3LN20tKSuvW/v6ZbL7EyRxBTM+hNlzBNDNY7I9SUmaPe7
+Br0hjhNlBYUzJA4XcNxNuR/M9EIwisy1t+rzy/qO7u7Y/la0rwshJMRj7uojtS/9mo8Ei+79ptKS
+iuF464cv1b/6jPPkfi7oRwjx4ZC39qTn3DFtRp6hqPzynxJz9/obq1UpmaaSuYk+4+RS09bX5Qqk
+WXSZNkP/FgzDaGrog0VDp+t0Y09JdgrLCduO1vd5L7dMaG6aGSEUjrFoKhFljEd0sU05O1UNrVZX
+q8MXf/Goo9MXl0bRaQ2uAfRRjZWhsDzv5gcdh3Y4T+73NZzGSRpJkhCPCJHQrEeeTF28nqAVCCFK
+awh3t3Xt/qBn/ycIw2RB4MNBWmvMWHvrpantQ8RcjnB3m3XO8mFnBQQTp9Ppb+p2r6rM27ik5N29
+1aIoLSjOtBnUx2o7huz54cEapz/c6fQfr+9aVJo17NG0KmZlZd6aufmtDk9zlzvRJ3cRHJPtGuyO
+bJs6ccPnpyMZIU6Q3j3t8kZ5WUawBN4EgUA1VjhFW+cuV6ak++pOhzoa2YAHp2hdTomhsHzwnHsZ
+a27W5RT5Gs6Eu1r4SJCgFYbC2cbSuaaSOQQzzCQraSs2CrGYypbOh4OhjiYhHrWULxq8g9Kckn/b
+w6ayeYMHGoNxd7C6tSjTetuK2Tl2kyBKlflpvd7QS9tODNmtqqm7qql72CPQJPHtu1eJoqxXM3OL
+MnhBfG/f2XOfL6g4RZCYbFVIZfZEjkqcjkRJPtwWaHTF7pprTdPTUBmdIBCoxgFO0brsIl120WX2
+IRiFsbjSWFw5ymPmbLqv/3/8Ted8DWdojWHIe1X2zFmPPpnoU5/59le3IgxbO7fAZtAgDNV1OE/U
+d+082dT/6ieHaxs6XZGLc/y63YFPj9T5QlGEUDjGvbLjZGn2+bWta9r69p9pOXyu3R2YQuv8ulhF
+S1iTZ4U8nasmI+SJCGsLjfMydSqojE4YCFRTXaSnLdzZrMst0WYVJLosySgS47Yeqdt6pG7YV3/6
+8s5LN55r7T3Xen6MlCcYeeQnryX6JC6Hk/CTXvObXdlPGjwwg+TVInFsY6kJIaSkoL9/AsHFnfJk
+SZ2ek7r0ukSXA8xMblZRE9KzIm5l4okuy3TCiXJ3gJUkWUUTKprAINNvIkGNaqpLX31z+uqbE10K
+MGN1RVVelrk1o1NFCP5EF2a6EES51R3bUu3++oo0bUJXbU0SUKMCIKl5eYaXsFXWqZXcMZXJMuoK
+xLecdbujPNSjJgc8CwCQ1DbauzfauxFC3JgPlSSCceFER+hsT/hHm3I0DNxCJwPUqAAA4Cp0+OKN
+7thXlqflJ3Rt8aQCjwMAJKmOiKY7prIxsXxtaOxHSx4lKeocs5IhIX9i8kCNCoAkddxn2t6b6uIU
+Yz9UUqEITMsQ9NSbVnjysfHY/t2fbV4599W/PRePRQe/1NvT/av/+Y/vfOWLR/bvHvsHQY0KgGTk
+YhVnAiZRxvI0UJ0aFV+U393o5yXp1nIrQ0KUQgghSZLCoWBD7Tmv2yVJ0uCXeI7rc3R3tLWEQ+Pw
+BYPLDUAyOua1hHhqlt5vpqfW9LhTEyfKnzX4zvSEbRoa1pqafFCjAiAZWZn4BrujVOsnMJjw+wo4
+UTrQEjjriORblHMytAQEqkkHgQopZPZ+7q3ElkGWUZwTFIke307LMy1FWcvIT988eSfFyzhCiMKk
+MR/pyuc1xiOU6AIyCqgIYYzHSQY4homSPDtVPT9Ta1DCPTMB4KIjGnGb+O2JLQMvyC9+0rBhYXqa
+VU0S8Lw2btQ0+u5qfnI+qy+urA3pTRRbYfAl+rwvh5NwEpO15CRdlhmAxLGKdA2GMKMKbpiJAX1U
+iSdJcq83+stXzvz5g3p/GDoMpiVexg+4bX9sKt7ZlxoXp/Qs2nUh3Wd9qZ1RWNHjyoJxwRnmWEGy
+aWirhoLeqUSBQJV4nCCdqnfVdwaeeevcjqPdwchMa39LBl1R1aeO9E8c6QfctoaQLtHFGZEg4584
+Mj50ZPTEYBmzKxAkeX9z4GRnKBiHBtIEg0CVeCwn7qvqlSXZE4j/7MXTVQ0elhcTXShwFUQZ2+20
+H/ZYgjx1ymfa5UwV5Sn66N0XV5wLGGbp/QWQlX5ZgiTXOMLb6rz+mAgje0eCYRhND7+MmSRJkiSR
+FMUw4zBQDwJVgskyCka4Hce6ZRkhhE41uN/e1druCEkS5GJNGz0x5bbetOawFiHUGVOf9Jlc7BQd
+RdsY0qUrY/MMXjMDjcyX44vwr5zoyzUrV+XrLWpYqGt4JEnZUuwKpcrrcQ8ZRxUKBaLRiMlsTU3P
+GPsHQaBKMJYT6zsCVQ0eST4fmZ7/oO7TQ13eENxHpgdBxrb0ZJ3wmqMiiRDiJPxswLjHmZLocg1P
+SYjrUhyZqim0vvAUJMvIEeTsWubWCnOKFhY+HhFJUemZ2fMWLd320Xu9jm6OZWVZliQpGo3UnKny
+uJx5hUVpWdnj8EHjUlwZoyQC+mavhZ+N7j7VO3hLNCb86b2anEzTpmV5BDGlu+UBQqgzxLzTmd0W
+1QxsORswfODIvj2rj5x6s+wstPXPc4NLaOgPVsKh1wohhCQZ4RiqSNdUpGvGfrQZT6vT/9OPfvyP
+X773T7/+2R33PZRfWCII/InDB95/4+WU1NSbbr9HqRyHqXvHJ1CFjKtDxtUJvmDTU1dH2772rw/Z
+WNPqf/W0Xbn8a4VFZYkuILiC5w/21MYdvHShBsxJ+Blu9mvUjasKDIkuHbg6vCi7w3yqHmpRo0VS
+1KyKuX96+e2//uG33/3al1x9DlEQ5sxfdONtd2248ZbcgqLx+ZREn2ZSEwTB2dtzcO/OS1/6+L23
+cvOL7GnpWp0+0cUEI+oLcdvrve5LEjVb3LEPzrqnTqCSEdrf7A/EhBX5BhiyOpIYJx7tCP1xf9ez
+95VoGBIyKEYJx/HCkln/8bPf8DwvyzKSZYIkKYqm6HHr25tyTRNJxet27ftsu8APM/QyFAy8/Jc/
+fvzem4kuI7icZ/Z2NTqjgjg08yXKi3XOyKmuKZNZJ6O3q1wkgcFIoJFEOPFAa+CvR3runW9X0zDq
+/uoQBKFUqXV6g95g1BtNGq2OUShwfNx6LiBQJVIw4D9TdXzYl2iaEUWxq6PN0dWZ6GKCYcgyOt4R
+2tPsD8ZFebhXHQFuW5030cVECCFBkg+3B/0xocSmUtHwkx9egzP6zmnXhmLTuiIjDvnoUww0AiSM
+IAhuZ1/NmVM4TpgtVrPVplKr/V7Puk2bMzJzzBaryWJLTc8wmMyJLikYHi9J1xUb/TFRkhEnSLV9
+EUmSzWqKl2RekDEMdflYb1QwJXreHV6U9zT6big1WbU03IJHkq5n7pmfMtuu1jKQwTTlQKBKGFmW
+rbaUr37r+1qd3mg2K5VqQeSf/n//tmr9xvI58xUKBc0ocByef6coDEPFNlWqjuZEGclIlOS/HnEw
+JL6h2KSgcFGSZRkxJK6kEv8XpAhsY6k53UBPhcJMNbwoI4QoAjOrqcXZOgUsNDUlQaBKGIIg0jKz
+77jvIZVaTVE0QojnOL3B5Op1YBimGI+cTjChTCrKpLrQXVySomJIYk66Rj/FshVIHJuTAZnWw+BF
+uao7FIgJlelaq4aC9TumLHh8SBgcx5Uqld5g7I9SCCGKphcsW3Hm1PE+R0+iSweu2tJc/YIsrWIq
+1Vo4Qer0xbv9MHh8GLwonXNE9jcHOv2sjGAimCltCv2oAEJo3fU3dXe093S2D5sKCKay0hR1sU01
+pRYp90aFj2o8jiAEqmHU9UW313sFSa5M1xinWCUYDDGFflQAIVRYUmY0m9tbW7wed6LLAq4CK0jB
+uBgXJnzJxNGL8VKDK3q6OwxT1Q3LFebVNLE8T1+epqGm3hwiYDD480wtjEKx4cZb2luaOtqaE10W
+cBUCMeFkZ6jNG090QS5wBNhDLYGKNE2OGeZGGkZluube+baFWToKuqamPAhUU86qdTcEg/7WpsZY
+NJrosoDRinDSnmbfOUc40QW5IMqLEU68rsSY6IJMLe4w3xvkWEEyqymTiqJgaO90AIFqylEolXMW
+LG5va6k7d0YSYWGq6cEX5TEZU5BTaAhOoVX17TVZBRZIH73AHxPeP+va0eD1RmEtxOmERAjBI8VU
+c/Mdd/30qR/s37k1OzfXap2iC0aAwUKsYFIRNs2UeEAXJVmSkYrCVVMpBTHhIpz4TpXzSFvgq8vT
+rGpyKvylwKWGbYglEUKZ2kQXDVwsU2u5bs3yffv2tZ7aP/eOO2CB0akvR09oCFVlKp3wX5OMUJMr
+7okKi7O18L0ZwArSs1XO3Y3en92cPTddPaWSM8EVQVLmFHXnnXdWV1fv379/yZIl6enpiS4OuIKV
+eVpZRlNhgqJgTPykzl/bG8szKWxayPc7j8Sx5bna64uNOSYGJkafduCxYorSaDS33norQuijjz5K
+dFnAleEYRuBToup7qCPU6mVX5mktGngMvYDAsfJUdYGFoSFKTUMQqKYoDMMWLVpUWlpaVVW1f//+
+RBcHXEGTO+4KJ36MtjPMH+8I0QS2vkg/Fap3CdflZ3+/3/GnQ30IIYrACLgo0xMEqqlLqVRu2rQp
+Nzf3tddeO3fuXKKLA0bkjQp/PeY83pn43HS9gtxQaNhcZrRqoNEPNbnjzx52Hu+MrMrTJbosYEyg
+cWBKS09Pv+GGG15//fVnn332ySefTEtLS2BhInHuTHPPnlMtZ5p7YiwnI5SfZl5WnruqMt9qUCf6
+UiWSM8S7wzwrJn6+OIbEKtLUsixDzaHJHX+jyt3t5x5dbCu0KhJdHDAmxA9+8AOFAv6KUxSGYXq9
+XqVS1dTUNDY2rlq1KlElibH8kZqO37+9T5ZRfoZlXlFmqllb3+HcdaKJIogMm0GjpBN9tRLmVHek
+2cOWp6rzzAn7KbGCFGZFGSEVjdOQ0oYQJ0okjlWkq5fn6qbEoAFw9eLxOI7jGIZBoJrqSJI0mUw4
+ju/atcvn8+Xm5iqVCZgRR5Jllhdoilw7v/C6BUXLK3Ir8tJSzfq3dp9u6HSV56fm2E2JvlQJ0+yJ
+h1ipzK5M1ycsWp/oCu9pDlIEnqpL3icGhBAnyjJCOIYpKTzTyOSYFBClpq+BQAVNf9OAVqtdsmSJ
+3+/fvXt3b2/vV77yFZvNNskZZgxFzsq1z8q1D2xRK+k1c/NLsm27TzV3u/yJvkiJVGxTGlVkloFJ
+VAHCrPRpnT/GSXPTk7oNlhflo+2hMCfNtiszDAysLzVjQBPB9GC1Wm+99dbNmzfX1dX99re/ra6u
+5qfAOiBRlscwjBdFQZxCs4ZPvmwjszBTk5K4QUsH2oKdPnZehrrImrzzzwbi4tZ6//vnfA2u2FTo
+LwTjCGpU04ZOp7vpppsyMjKeeuqpaDS6fv36RYsWmc1mgkjY/HJnW3rqO5xapUKnSt7WY0GSnWFe
+ryDVdMIe+845onPT1QszNRpmCk02OMmOdoQ+rPGm6pj1hfr8xHUWgokAfVTTCUEQdrt92bJlu3bt
+OnbsGMuyPM+TJMkwzOSHq25X4A/vHjha27lmbv5Ny8pSzUmaAewI8p/W+fQK0pK4jHACxxZkaNIN
+TDIn++1rCRZYlLeXm/ItcEObIQb6qDCfz2cwGBJdHnDVPvjgg3feeYdl2cWLF1dWVqakpKjVapqm
+KYoiCIKiKLV6ororZFnudPr//NGRP394xKBR/s9Xb7xuUbGSTtKBO9vq/G+cdj+8KGV5bgKm+Yvx
+EkPiSdsXIyPkjQg6BUESSRyiZy6fz0cQBI7jEKimt61bt7766qvV1dV2u720tNRqtZpMJrVabTAY
+SkpKVCqVSqXSarXjlXkhy4gXhC5X4D//tu2jA+dKslOe/OK6NXMKtKqE5REk3F+O9LV6uXvmWmbb
+J7t/SJDkvS3BuelqvYJMwlglycgT4d+o8twy25imoyF1YuYZCFQkQkiCfsdp67rrb7ju+huam5vr
+6uq6urq6u7ubWlqCwaDL6aqprVmzevXKlSvvuvNOg9FIkCSGxvpLjnP8qYbuR3/6emevb93Con//
+8nWz81Jpkkzmr1CNM15oVRiVxCRfBFlGjc74v3zU/u/XZ63K16qo5OqdEmXZHeF/t9dxoC28NEdn
+1SIsib+EMwk23H0K8/l8YYUh0WUD4y8Wjb772st/+OXPbPbUx5/84aJlK1VjawzkeOF0Q+f3f/l6
+e7fnH+9f/7W71mqUyVuRGvBulSvPoiyyqZSTm0zBCtITbzTm21SPLrWb1MnV7ipKcrM79tOtHXFB
++s1dBSY1DbWpGYPBkfXzoYAXNf1FIFDNRLIs8xwXj8ffee2lLW+8ZrZav/qt78xZsJiir3FAaGNH
+33/84b2quo5/vH/DQ5uXqRQMdAsghDhRJjCET24aQ5QVXz/pfOuk6zd3F+aaFUnY6uWPCjvqfCsK
+dDYtk3xnP5MxOLJcEqiIH/zgBzwJSTIzEIZhBEkyCkVBccmGG29m4/GX//IcG49n5+Uprn5uC48/
+/OmBs+/tOjW3JOuHj27WqBQQpfoRODb563vgOJ6io2elqUtT1TSRRKMhBVFmRYkicJrEciwKg5JM
+5kTHGYnEkOrzZmyYmSKJaLQ6jVZ36z33pWfnvPPKi36f5+4HH05Jvbr5bf2haG1LD0MRNywv16rh
+yea8/U0Bi4bKNiuUk7voO4Eju442qankWamWE6Uz3ZE9jX6zmvryEjuOYWo6ubrlkhkEqmRhMluW
+r15LUdSnW9594U9/uOO++wuKS0f/9kA41tTpDIRjL35w8NMD1UNe/eqda1bPL070KSbA/mZ/WZo6
+RUdPWqCKC5I7zJvVpJIilEnT5hVmxYMtgZ31fr2KXJKbpCP2khkEqiSi1miWrFil0Wp3fPzhB2+/
+ecd9X8zOzRvle7PTzN/54vWBSGzYV3PTLYk+uQSI8eI5R2RZnl4xWVFKlORWd2zLGc8dcyyFNlWi
+L8DkaXTF2jzxWanqRbnaAkvyThOVtCBQJRelSjVv0WJJFD/d8t77b7z2xce+YjJbR/NGs16zZmFJ
+oos/hcgyanLGRAml6OhJa3/rC/EHW4JRXjKokivNz6ahluXprRoqJbnnhk9aydLADQYQBFk+d/6S
+Vat7e7q2bnk/0cWZtjAkyWhlod6kpianBS7Ciqe7QrW9kU1lpmRYwJcTpAPNgUZXLM5L6QZmdpoa
+olTSghpVMlKp1YuWrQz4fPt37sjIyVm5dkOiSzQt5ZgVRhWpV05Sl37/oohldvW8TE2iT33CeaPC
+jjpvjSO6ocSYlrhVvsAUATWqJGU0mZYsX5VbWPTWS3/vdfQkujjTD4aQXklmmRSTlh2uYYhFOfpb
+KiwzfgFfXpQ+OOPe0+jPMSuyTQrFTD9fcEXwDUheWXl5m269Q6lWf/jOm4kuyzQjyXJfiGv3xoVJ
+mTcpxkneCB9hRSWFJ3CO9skhy4gX5ZreyM3lljvnWDONsP4hgECV3LJz8zZsuunDt990dHUluizT
+CS/IexoDn9Z44/yErxgpSHKNI3KwJdAb5Eb5Fo4XnN5gbavjZG17U6czwRdrdGSEOFEOsyKGIRVN
+fH9D1nUlRo0CRkoBhCBQJTmlSlUyq7y8cu5f//h7nh/tfRDwony0LTg5q2v0+Lltdd7q7shoWvyi
+ca7PEzhV3/HiR4ee/NUbjzz151+9uDXRV+vKREn2RPizPeFzPeH+LRYNBRUpMACSKZJdanrGFx/7
+2tceuPu2e+4rLpudwPWCpwsZoZggHWkLfGNV+kSP8xUkeWe9lxWkjWXmTOMVpgCWZPnYubYtu06e
+rG3v8wRlWZblaTCjOC/KXX52W41nT6P/vgUpiS4OmIqgRpXsCIJISU+/9a57nv5/T8Vj0UQXZxrg
+BKnNEzOpqElYA6nVHXOEuFlpmjmjyPQLhWOHTjc1d7mWVOT/29dufvz+6ZHMWdsX+b+93Yfbgk+s
+y7y5IhlHjoMrgtnTAZJEsauz/d4br//rW+/lF5WQJNSzr4CX5C4vm2Oe8Jl5ZRmJkoxhaDQRUUZI
+FEVJkjEMwzFsz4n6b/747+uXlD3zwwcTfcEuxxPhvVHepqG1Sbn8Ixhi2NnToUYFEE4QZott9Ybr
+3nvj1XAolOjiTAMUjmWZJmOVEwxDJIGNst6GIUQSBE2RFEkQxORP6X4VYry4v8lf1xtBCBlVZJ5Z
+qYMoBUYGgQoghJBSqfzK49/e/uEHkTAEqsuRJDnKSQiNqoozFm+edO5p9PtjQqLPePw1OWM/2dr+
++klnd4DjRAnHsKkdVUHiQaACCCGEE0RWTp5CpTqyf28wGEh0caauCC8dagn8fnf3hH7KznrfZ3U+
+hNDMG+t6tC34h33dCoq4Z37KvEwtCY06YBTgWwLOIynqhs23HNy9y+/1JrosU5c/KhxsDZgmbNSt
+jFCXj33rpHNelrbQqpx5k1DYdfQtFZY759oWZGmNKmjuA6My034GYCxu2HxrV0dbn6NnYEyVwPMw
+vmqAJMueCNfsii3N1U7cp7jCfLFdta7YaNPSM+A+LstyICbsqPNFOFGWUaqeWZanK7AqJ21tFDAD
+QH4XuCA1I4NRKg/t3d3n6IlGwqFgKOj35RUVr7nuBr3BmOjSJR4ryKG4aFJT6QZm7EcbFoZQqp6+
+rdKaqmNIYtqHqbggNTmjn9X7g3FxXqZGRREUgSE07c8LTDIIVEktGom0tTR1trV6XK5QMBgKBro7
+O3u6OpVKVSgY6M8A/ML9D669fmOiSzol4BhKNzC3VVomYiJaQZJdIS5Vz9hnymIWMV6qcYQ/qvZg
+OLYoRzdpiyCDmQcCVVITBaG9teUvz/ymtanJ7/Oy8fiQuQyKSsvyC4t0ekOiSzolMCSeZ1HmTcAK
+s6wgNbvjB5r8X1pqn7Tp2CeaJMuChIxq6oYyc5ENluUF1w4CVVJTaTT5hcU6vaG3Z/g0tpy8/LzC
+okQXc0qI8VKUE/VKctwTAARJ7vDFt5x2B2KCKMkyMY2bxkRJdoa4OC9btZSGIRZmayvTNdAdBcYI
+vkBJjSCI7Lz8rz7xT2kZmZe+yigUuQWFmdm5iS7mlNDhje9u8Puj4z+wyRsR9jUFOryxexbYlNQ0
+7piKcmKtI/rxOc/+5kD/hcIxDKIUGDv4DiU7hmFKZ5d/8bGvKVWqIaMuU9MzyioqbampiS5j4nGC
+VNsb+azBF4jx435wb5R3BLgvLrbPyRjPpXu1akV5YUZ2qnlyLpEky6e7wi8ccZztiaYZaKt2hq+b
+BSYTNP0BpNMbHv2Hbx0/fHDfZ9t5/sKNODe/IC0zK9GlmxI6fGy7ly20qfKtqnE/eKFV+d31GUpq
+nOetX1CW8+bT/zDRV0aSZRzDEM4s83YAAB/YSURBVEKsIO1vDtj1zG3llnzokQLjCgIVQAghlVrz
++7+9vG7e7L5ehyxJCCEcx0tmzc7JzU900aaETh8bjAkbSsYzR1+UZF6UcRyjCUyJT8vVVQRJdoU5
+g5JUkISSIr5/HTzWgAkBTX8AIYQwDNPpDT995k8Wq61/S1pmVuWCRanD9V0loTyL4sbZ5spxbZo7
+3R3+477uD6vdiT65ayRI8rG24AN/rXnpSJ9vAlpEARgANSpwwar116/fdOPH774dDATyi4qtKfZE
+l2iqyDQy6QZmHCeiPdYefPuU266jxreWNjnivHSqM/TsAYcgyd/bkLUiT69VwJ0ETCD4eoELKIp6
+4gf/1tXWduzQwVkVlRnQQYUQK0gYhmhiPNedb/fEPjnn1TD4TeUWLTP9foMkgWUYmXVFhqV5+jQ9
+o6RwmPscTKjp9yMBEyo9I+vRf3xCEITZlXNtqWmJLk7i1fVGT3aG8izK1YWG8TqmQUVtKDaqGDzH
+NOFLL44XTpQanLG+IDcvU2tQkXYds7ncolMQE73cCQAIAtVkOvy3M11VfYkuxZWFg3xmeFHbO+H3
+D+xOdFmubN13FpuydRN3/OqecG+QW5A1nh+hVRBzMjU4hk2LQVOsIDc4IweaA7W9sVyLoiJdgyFE
+EZhRBXcPMEngqzZ5XI2+tiM9iS7FqFjEvFgramubBqVlwxM4uXuHN97p4+w6Js+iGPvReoPcvib/
+4hxdlkmhoqdNml9PgN3fHAzExPXFxgKbUqeYNiUHMwYEKjAMhhj/0ULTUXeANanJUrtKzYz17twX
+4l480itJ8vxxrZxNnC4/a1FTCgrXKYgFWVoFhedblCoa8oRBAkCgAmBEaXrGoKTGOJ25LMuuMP/S
+0T5XmLtxltk2tadskGW528/uawr0hbhNs8x5FqVZTZnVU7rMYMaDQAXAiLJN49DiJyPECpIvwm+e
+bVmQpVWNuXI2cXhRruoKHWgKtHvjJXb1DFgQC8wMEKgAGIYrxEd50aymNGOOKxiGWTT0/YtS8i1K
+ZuotLS/JMi/KsowUFC5Kcm+QY0V5c7llSZ5eDQ19YGqAQAXAMA60BOKCtDxPf82BSpRkb1ToCbCV
+6RolhZfZ1Yk+p6EkWQ7GxE5/vC/IG1Xk/CwtQ+Lri43ri41KipguefMgGUCgAmAoV5g/2BKYm6lV
+XmuVghflngC7t8nf6IxVpo/nxEvjyBHgjrYFT3WFI6y4IFs7P0uLYWgapSOC5AGBCoChPqx286Jc
+nqa2XFMSgSDJXb74JzXeU52hr61IT/TZXESSZYQQhjAMQ6e6wnubA2V21c3lZruOSXTRABgRBKok
+otIz1HDPy7IsS5IscBIb4S5eiR4p1BSjusLNOhpkeVZM9MmNp7OOyPoiQ6bxGu/dvqiwr8l/pivy
+tZXpC7K0iT6b8zhR5gTJF+UpAjeqSIbEry8xXldinILdZgAMAYEqidz3n8tnrx46G7osyWxMCDij
+Tcd7P/r9qXiYk6ULwWrdl2df91jF5Q/75+/sOru7I9EnN56evqNgLG+3aqjb5lhXFxnHJWlwXEiy
+fLw9uK8psK/Jf0uF5dYKS4qOpiFEgWkCAlWyw3BMoaaYHJ01S1eyLP3Ff9nbVecReSnR5ZqWWEFC
+CDEkrlOQU2e2WUGUj7eHnt3vmJ+l+fVdhRlGBmpRYHqZKr8lMGmc7cH9r9We+ex8HYhWkTmVtoU3
+5xcusBtT1Xf+cMlLP9zb1xoY/JaAM3r8o+Z9r9YNe8BokE30OY0PTpCe2dt9W4U108hc7RCiuCB+
+fNZ7vD20PF9/02wzQiixWXMxXqrvi7Z54rdVWkgCm5el+dWdBTSBKSgcppEF0w4EqqQjCVI0yAVc
+0f5/YjgWdMc6ql03f3tB6bK01AJDRqnZ74qy4QtL4UmSHI/wA2+ZkWK8tKPOd6oz/KUl9qu9lff4
+2ZePOXtD3Ip83eKcBM+Q1BfkDjT79zUFIry0usAQ5SQVjdMkDg19YPqCQJXsZElmI7yrM7Tr7+cK
+FqQwKipvXkrbGdfgQDXj8ZLc5o2/cqz39jlWNX3VQ4iqeyIUgW6cZZqfpdUleglBV5jzRoW5WdrZ
+aeoMA8OQUH8C0x4EKoAQQpIodZxzS6KMEDKna2hlcn0xYpzU1BfNs6jWFRkp4qprHuXp6uIUlUUz
+DtNYXHXJebHFHf+s3qei8A0lphyzItOo0CpIJYWb1RS08oGZIbnuR2AkGMLoz5dv4DlxcOJfMlCQ
+WFmqOsOkMGuo0dzaZRlFefHdU661xUa7jk7TJ2AQkiTLrZ74myedobikUxIFVmX/FO96JalPsucM
+MOPBFxoghBBB4fNvyicpHCHU0+CLTeQiT1MQTeJ5ViUvSKOJUrwot7hjH1S7Y5y0XJzUkM6Lcl+I
+6wty87O0GMJUFG7T0kU2qjhFmWlkNFMmzxCA8QXf7GRHULgpVZNTaV32hSKCJiIBtq3KGfVflMin
+0FAFC+zYJV038TB3+rOO4LRNshBlud0Tb3LFl+fp1AwxmnQDXpTbvfH3TrtESb6+1GTVUJPTuuYI
+cud6Ih3euCssWLVU/3RHqXrmrrk2FQ2JfGCGg0CVdFQ6pmhxqkJzfr4Jkias2br8eSkpuXouJlRt
+a+tp8g2ZaUKpoYuXpBUvSRtyKJ8j3F7tnr6BqjfI7W7wOwLc0tzRzh8hy0iUZA1D3jHHatdR2EQm
+oYdYkcAwBYXhGOYMced6Ir6YoKaJAqtyYB8trLcLkgAEqqSjsyoX3VKw6JbPJ1+QkShKfFxwdQSb
+jvftebkm5I4NeQsXF9ydob4W/5DtEV88Gpiug6hcYX5/U6DVG19XZNBeNlVPkmV3mPdFBZOatGro
+Ipsyy8QoqYmKEJ4I740KziDX6WcLLMoSu0rDEFYNta7YkKKj9UoShuuCZAOBKumwUcHbHQp+Ho1k
+SeY5MeyLd9f7jrzXKHDDzNoX8bOntrbu+HN1oss+nlrdsRZ3vCxVtSxPP9I+kix7o0KXj61xREKs
+uCRHZ9XQGIaNb5SSZSRIsiTLDInLCNX3RU91RVrdMUGUtDSeb1UihNL0TEJSNgCYCiBQJR2fI/zZ
+386e+Lgl0QWZDKIsjzQqyqalN882FaWoLlNBYQXpYEtgR61PqyA2lpmLU1TjWTZJjnKSPyaE4kKM
+F5U0WWZXIRmROG7VUKsK9AUW5TWvMwLATAKBCsxMgiQHYgIvSXbtRRWRGCcJssyQeI55xBljWUHC
+MYzEMQxD4bi0KFd3Q6nJqrmWJT+G4EVZkGSKwEgci/LS8Y7QjjpfszuqJPG1xaYyuwrD0KIc7aKc
+qTLnOgBTAQQqMANJstziib90pDfXxNy3yK74vM7Ei/KuBq83KizM0RXbhqkeCZIc56XqrrBJQ2WZ
+FEqKuH+hbVyKFOMlUZJbPbGeADcrVZVhUNAERpPYrFT1Y8tTU7QUrFgIwEggUIEZqM3DPruv550q
+V2mKKsTJ31iZRhEYQmhbrfftU+7l+XrzcItsiZJ8vC34s+2dEU68d0GKSU0pqXFrefvJ1vaqznCn
+j82zKL63ITPDoGBIfHmefnleoi8WAFMeBCow09Q7Y88d6HmnysUJ0llHpNkTW1Wgn5Wqfve0+6Oz
+nptnm9cXGwyXBKo4L+1q9L96rG91kWHTLFOmgVFcU8YEK0jOEF/bGznVGQ6zwg+uz1IzJEJoTrrm
+hjJTvkWlYXDF+MU/AJIBBCowo5xzRP96yPFBtZsTJISQKMkxTtrT6CuwKnNNzPc2ZGQZFVoF2Z9g
+wYtSkzve7olvLDMxJL6qQD8vQ6OgcDVDkKMeQhvjRWeIV9OEXklQBL6n0f9OlZsm8TwzszTPNDBv
+xQ2zTCSO0QSe2OU/AJiOIFCBmaOuN/ry0d5PznkigwYsyzJyBnlRkmelafqzGDAMnXNEdtR6a/qi
+mIwWZGsRQhiG1DShHnVHkSTLb59yH2kLeKMChtCNZeY1xQazGl+Yrcu3KAkcU9K4liEGKk+jPzIA
+YAgIVEnk0z9W7X+9josKnu7QKN9y9IPm5pN9Aiv6+yKJLv4VVPeEXz7a92mN9/+3d2fBbdz3HcB3
+sbtY3AdBgjgI3oQoUhIpiqJEKbJsN4qkqJEtR2lztVNPpmmaaR+amWQ6TR7acWeavmTGD42nVh+c
+xnKiOJZjubZsyrplS6IOUpREihd4gScuAsS9i0UfENMMSVOUCXKX4PfzJGGXf/z+mhW+/P/x3/0H
+Y/zc19NEeigQDydSaSKlZSk1S9EkyfFCvoY5mJdn08vthsdsGO8JJx+ORR9NRkcCiZQgHNli2ldp
+IAnSoKT2lhu0CkqroIoMrJalCIIwqmijCv+tALIJ/6M2kNFH/if9Ee9wyDscErvwx7sxEPrtnclL
+PdPeBdtopdPEw7HIS+8P6ZTUoZq8xhKtTkGXmpSFOjnLyGQkSaSJBC+wtCyVTj8Yi1zrD87EUtMx
+frtDs7/KYNHJ45wwHePS6XSVWWlU0pl17SRJNBRrKRmpZGSY0ANYVQgqWN+SvPBOh+cP97y3h2YC
+UX7Rc3wRfjKcjPHUiD9eY1XrFIQ3wn3iCrm8MUFI56mZL1XoGkt0BEEkOCGWFPRKyqxlivPYzKo/
+o4rZXqTdakvrlbSapWaXAprUWbizCgAeC0EF69tYKHFmKHKtLxjjhM87R0innWZlgUZu0vxxL0GW
+lmkVVIGGkdOkUcVknvUnI8nyfKVOQRtUtE5BKT7dd1DDUmu/IyIAzEJQwfpGkeTeCr2ape6Phof8
+8SS/+P5QOgXVXKZzGNnM48YdRtaqlxNpgqY+m7MjCSJfw+Rn4wkUAJBFCCpY3xxGRVNt/kggcaVv
++sZA0OWNj04n5n1TJSOJQq18i009d+vb5S9ABwBxIaggFziM7Hd2Fj6zyXh/NNw6GLo5GHIHPosr
+kiQrC5QsbrMFWJ8QVJA7bDq5TZe3t1z/cDzS0uX/uD84Hkz6IpxOQRUZWYbCEApgXUJQQa7RsNSu
+Ut12h2bAG3/zrudyb2CLVa1CTAGsWwgqyE1ySrapUPWzwyX/cqjkWn/QpGFwrxPAOoWgghwnI4mn
+KvUrbwcAxIKvlwEAQNIQVAAAIGmY+stNerPq4N/VFdfkXz7Z1X5ugPv0aeJ7jju3Hyyb8cVb3+l9
+dH2srN78jZ82x0KJq7/taj83NK8Ro0Wz57izdr/jwcXh93/ZRhCErkB56Af1pdv+ZNPbSCA22jvd
+dnZg6IFH7H4DQA5CUOUmipbl2TSWCoPawJKf3tm69xubmo87BT7dfm7I/chPEASrYqyVBi6RajxS
+sTCoqvfatv1ZsblUP/LQO7fZwjLd9bd6718a5hMpk0NbVG0q2ZpftNl0+93+9paBeIR7kkoBAB4D
+U38bReOfl+96rjLFCTff6e04PxQOxGcP0XJZybaC5hec1Jw9k4qqTdXN9jybdmFT6TThc88MtE/1
+353sOD909Tddre/0yWRk89edtfsdYncUAHINgmpDqNnnaD7mFIR065n++xeGZ3yx2UMCL0SmEzRD
+7X6hSmdSzg6/NjXbLOX6eIRLLDlCioc5z3Co48LwQNuk3qysbLTo8pVidxcAcgqCKvdt3lv01Leq
+dWZV+7mh+xeGQt7Y3KOCkJ7xxXpujlkrjQ2Hy1gVQxBEWZ3ZucsaDiYm+gNcMvXYt4gE4lNDoUSE
+M5fqbc48sXsMADkFQZXLSBnp3G3d/90aR62p46OhjvPzUyojHuau/76HS/A7vlquN6sUGnn9V0pN
+Dk3f7Ql3l2+Z7zXji4f9cU2ewlSkEbvfAJBTEFS5jCQJhZphlTRJktGZJP85OzZxiZTr3tS9c0N5
+NvWWpx1bnnaUN5i9wzM9N8amp6LLfK94OBmPcDQjkyuxQgcAsglBlcuEVLrj/PDlN7oCE5HmF6q2
+PlOsMSoWPZNLpi6d7JzxxpqPVT393RqVXnH/4shYb2D57yVX0IyC4pNCMsYv/6cAAB4LQZX72lsG
+Ws/0pQVi97Gq6r12pVa+8Jx0Kj01EGz/aEipY61VRvcj31DHVCyUXP67aPOV2jxlZDruHwuL3WMA
+yCmYpdkQLr/eqTUpdx6paHquMh5Odl8fm70FeK4rr3cVOLRak/L673smB4LLb1+upI0WlULDuB/5
+JlxP8IMAAI+FoNooLv7qocaoqH2qqOFwWdgfH3rgTQvzd22f8cde+/HlJ2qWZmSMki6rM1fssCSj
+/ED7VAAjKgDIKgTVRhGZjt843a3JU2zabQt54/7xSMiz3IUSCzEKWqmT0wxlqdRXNdm27C9S6RQ3
+3u5taxn6wm0CACwKQbWBDHZ42z4YMJhVu56rTEb59//r7hdrh6Jlh39Yf/iH9Zm/hjzRwfveM7+4
+03trXOwuAkAOIgOBQERhELuMDeHdn17uvjC4Nu9Fyki5gqZoMhlP8VyK+HSSj2ZkNEuRMpJPpLhE
+iqJlrIpO8elEdPHHT9ByimGpFP/HtXyfNvsna3DS6bSQEvikkOIFYs1953+OWGvz1/59AWA1sDIi
+/9P1XoFAgKIomUyGEVVuSguLZw/PCXPvpkrxQnTJpX18MsXPeTLF5zULALB6sDwdAAC+uGQy+W8/
++dHzz+4dHnDNvpiIx95963ff/9bX/+Nn/zwxPrrCt0BQAQDAF5dOpwf6ejvu3onH//iEtng89v47
+p18/8d+CIOx95lmDYaXP/8TUHwAAZNPlcy2n3zhpyMv75t98b2fzXoVypTsqIKgAACBrbn589Z3f
+/UZnMLzw7e/u2rtPqVKvvE1M/QEAQHbcuv7xr0+8Eo/Fvvr8C7u/9JRKnYWUIjCiAgCArHjz9V/1
+d3ePj7r/4q9f3P2lp9TqrO34g6BaO44GC4MtMLJNaVCsvBEAWLmBvt6pyXE5y+r1BjnLZrFlfG6u
+nfqvbyKITWJXAQCwKn7yr/8+2Nf3y1/85/+9/abJbN69bz+bpbjCd1QAAJAdX/nac0ePf7O/p/ut
+N3796OF9nsvO8wEwogIAgKz55ovfm5qcePu3JxUKxQ/+6celFZUy2UpHRAgqAADIGpVK/eLf/0Nw
+OnDxw7MGk+n7//ijfLN5hW1i6g8AALKp0Gr7q7/9wY7de87+4fTbp06mUqkVNoigAgCALKuu3Xrs
+L79dYC783f/+6g+n3lhha9jmAwAAvrh0Oj0d8CcTCVOBmaY/+zopkYhHwmFBENRqjVKlWmZr2OYD
+AACyjCRJY55p4essq2DZ7NzmiKk/AACQNAQVAABIGoIKAAAkDUEFAACShqACAABJQ1ABAICkIagA
+AEDSEFQAACBpCCoAAJA0BBUAAEgaTRCEjBS7CgAAAIJYNI5ogiCs2dzbHgAAIJsw9QcAAJKGoAIA
+AElDUAEAgKQhqAAAQNIQVAAAIGkIKgAAkDQEFQAASBqCCgAAJA1BBQAAkoagAgAASUNQAQCApCGo
+AABA0miCIOKJhNhlAAAAEDKZTM4w816kCYJouXJd7NoAAAAIk1G/t3H7vBcx9QcAAJKGoAIAAElD
+UAEAgKQhqAAAQNIQVAAAIGkIKgAAkDQEFQAASBqCCgAAJA1BBQAAkkaLXQA8AZ7jxtzDYlexFsxW
+m0KhFLsKAJAEBNV6EotFP7l0Xuwq1sKzh7+msEgrqGLRaDDgF7uKpeiNeUqVapUaD8+EwqGQ2F1c
+Sl5+gZxl1+CN4rHYtN8ndneXojMYVWq12FVkE4IKYFkmx0dvXLkodhVL2bXv6bJK5yo1PuTqu3/3
+tthdXMqzh79mtljX4I08k+MfX/xI7O4uZeeefRWbNotdRTbhOyoAAJA0BBUAAEgaggoAACQNQQUA
+AJKGoAIAAEnDqr+NzmQyHT16tKKiYu6LkUjE4/Fcvny5t7c384rD4Th06FBJScnCFtxu97lz5/r7
++8XuCgDkJgTVRscwjNVqdTgcly9fbmtrS6VSBEHU1dVt3ry5oKDggw8+aGtrIwiCZVmbzWY2m69e
+vXrv3r25LUSj0cnJSbH7AQA5C0EFBEEQgiCMjY09ePCA4ziCIKampliWra+v37Zt28jIiNfrzZzG
+87zb7e7o6BC7XqkrKSk5cOBANBo9c+ZMOByed9RisTQ1NZlMpra2tvb29pKSkqNHjy7aztmzZ/v6
++sTuzZNhWXb79u319fW9vb3nzy9yf3pjY+OWLVuCweCtW7c4jmtoaKisrFx4WigUeu+992avvXXK
+4XAcPHgwmUyePn164ZVgNpt37txZWFh47969O3fuFBUVHTt2bNF2Wlpauru7xe6NaBBUsIiJiYnu
+7u7KysrS0tLi4uL1/mGx9pRKZXl5uU6nc7vdV65cmXe0qqqqubmZ5/nOzk6CIIxGY1NT08TEREtL
+y7wzZ2ZmxO7KE0ulUnq9vr6+3mAwdHZ2jo+Pzzuhrq5u586dn3zyCcdxGo3G6XRu3bq1u7u7vb19
+7mmxWCwej4vdm5ViWbasrCwvL294ePjSpUvzjlZUVOzZs4cgiK6uLoIg9Hp9U1OTx+M5e/bsvDND
+0n4syGpDUMHiQqFQLBbTarVarVbsWtYfn8/X0dFx8ODBhoaGhUFlt9sNBkNra+vsF3uCIPh8vnPn
+zoldeBbwPD81NeXxeMxmc21t7bygMhqNVquV5/mBgQGfz6fRaAiCSCQSLpcrN7o/j9/vb29vP3Lk
+yM6dOxcGld1uNxqNbW1ts+NmQRACgUBO/lOsBFb9weL0er1SqQyFQhv8V7kvJhAIPHr0iOO4oqIi
+i8Uy95Ddbrfb7RzHzZ1TzTFut/vRo0darbaqqmreoW3bthUUFIyPj09NTfE8L3alqy4YDHZ2diYS
+CbvdbrVaSZKcPWSz2ex2eyqVcrvdHo9H7EolDUEFi7Db7TU1NWq12uVyDQ9/9rx2uVzudDr3z1Ff
+X6/T6cSuV3J4nvf7/W6322AwNDU1zf142rp1q91uHx8fn5iYELvM1eLxePr6+jiOs1qtdrt97qH6
++nqVStXZ2Tk1NSV2mWshlUoFAgG3263T6Xbt2kVR1Oyhmpqa4uLiycnJ0dFRscuUOgQVEARByGQy
+h8PR0NDQ2NjY2Nh44MCBsrKywcHBtrY2n++zB0UzDFNaWrpjjurqanVuPac5W4LBYHt7O0VRdXV1
+s0HFMExRUZFCoejt7Z37G0Du8fl8g4ODJpNp165dsy/m5+cXFhZGIpGenp5AICB2jWtkZmbm7t27
+MpmsoaFhNqgYhrHb7Uqlsq+vb3BwUOwapQ7fUQFBEARFUTU1NTabLZ1OEwQRjUa7urquXbs2b8lZ
+JBJpaWm5eFHSDxGXiGAwePfu3QMHDlit1oKCAo/HIwiCxWKxWq2RSGRkZGTunCpJkmq1urq6em4L
+IyMjkUhE7H58QePj4/fu3XM6neXl5TRNZ2b59uzZk5eX9/Dhw2AwOPdkmqbz8/Pndp/juOHh4cwa
+1PVuZmbm9u3bhw4dslgsBQUFY2NjmSvBZrNFo9GFV4JKpZp3Jbjd7oUrBjcUBBUQBEFwHPfhhx+e
+P38+Nz4aJCIejw8ODtbV1R06dOjUqVPxeLyhoaGgoKC/v3/et1MURdnt9hdffHHuiydOnFh3a9Nn
+RSIRt9sdiUTsdrvNZssMH2traxmGuX//vt//Jzt7qVSq7du3l5eXz77i9/tfffXVnBl1JZNJl8u1
+Y8eOL3/5y2+++WYkEqmrqyssLBwaGpo3BUpRlMVimXclvPbaa5llgRsWggpgtczMzFy6dKmurq66
+upphGJ7nHQ4HwzBz1/tl8Dzf09Pz85//XOySsykYDPb09NTV1e3bt+/kyZMqlcpisXi93sHBwWg0
+OvfMUCh04cKFt956S+ySV0s4HL5w4UJDQ8OmTZtYlk0kEkVFRXK5/M6dO7MPf8nged7lcr300kti
+lywt+I4KYLVkVl17vd6SkhKz2VxRUVFcXBwOh9fvhN4TmZqaam1tlclktbW1NE0fP35cp9MtTKmN
+gOO4/v7+ycnJ4uLigoKC8vLy4uLiaDS6Qa6ElcOICmAVJZPJrq6uwsLCZ555RiaT6XS6q1ev5vB6
+v3l993g8Pp8vLy/P6XQ6nU5BEN57772NuRSb5/muri6z2bx//36CIAwGw82bN7Heb5kwogJYRfF4
+/MKFC/F4fMuWLZs3b848jSJXb59aKBQKPXjwQKlUPv/888XFxaOjo/F4PLNgZ6PhOO6jjz6KxWI1
+NTWbN28WBGHjrNFfOYyoNjqv13vixAm5XB4MBpe4AXN4ePiVV16haXreei1YWiqVytwyVVRUJJPJ
+enp6pqenBUEQu641EgwGb968uX//fqfTSdP07du3N+C8X0bmShgfHy8tLaUoqre3d0NdCSuEEdVG
+x/O8x+MZHR0Nh8NL/KqbTCanpqbGxsYwq/6kksnktWvXUqkURVE3btzYUBNfqVTK7/e7XC6GYTK3
+E23YoCIIguO469evcxxHUdStW7ew58DyYUQFsLpSqdSNGzfGx8dpmu7r61v4ST0wMPDyyy+vx+fP
+LkcoFDp16pTBYOA4bnJyct4YYmJi4t1339VoNBthOjSdTre2tk5MTDAM43K5Ft4aNTw8/PLLL+N3
+wYUQVACrK51OBwKBJW4JytwaLHaZqyWZTC6xqWYkEnG5XGLXuHaWvhIyg06xa5QiTP0BAICkIagA
+AEDSEFQAACBpCCoAAJA0LKZYT2iattiLxK5iLchZVuwSAEAqEFTriVKlfvorXxW7CgCANYWgAlgW
+mmE0WknvZcwwzOo1LpezEu/+3M1zVxVNS/9KkItdQpaRgUDgyq12scsAAAAgTEb93sbtmT9PT0/L
+MsSuCgAAYCkIKgAAkDQEFQAASJqMJEmxawAAAPhcGFEBAIBEkSRJkiSCCgAApGh2wg9TfwAAIGkY
+UQEAgBRl5v0IBBUAAEgZSZKY+gMAACmaHVH9P2WCwF9EWBpvAAAAJXRFWHRkYXRlOmNyZWF0ZQAy
+MDIwLTAzLTMxVDExOjUwOjQ5KzAwOjAwciT11gAAACV0RVh0ZGF0ZTptb2RpZnkAMjAyMC0wMy0z
+MVQxMTo1MDo0OSswMDowMAN5TWoAAAAASUVORK5CYII=" />
+</svg>
--
2.26.0
^ permalink raw reply [relevance 1%]
* [dpdk-dev] [PATCH v2] doc: use svg file type for ice
2020-03-31 12:01 1% [dpdk-dev] [PATCH v1] doc: use svg file type for ice nic Haiyue Wang
@ 2020-03-31 12:06 1% ` Haiyue Wang
2020-04-01 5:44 1% ` [dpdk-dev] [PATCH v3] doc: use svg file type for ice PMD Haiyue Wang
1 sibling, 0 replies; 200+ results
From: Haiyue Wang @ 2020-03-31 12:06 UTC (permalink / raw)
To: dev, xiaolong.ye, ferruh.yigit; +Cc: Haiyue Wang
Replace the binary format png with text type svg.
Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
---
v2: Fix the commit tile log format error
---
doc/guides/nics/img/ice_dcf.png | Bin 39168 -> 0 bytes
doc/guides/nics/img/ice_dcf.svg | 727 ++++++++++++++++++++++++++++++++
2 files changed, 727 insertions(+)
delete mode 100644 doc/guides/nics/img/ice_dcf.png
create mode 100644 doc/guides/nics/img/ice_dcf.svg
diff --git a/doc/guides/nics/img/ice_dcf.png b/doc/guides/nics/img/ice_dcf.png
deleted file mode 100644
index 540823c4a05e9ea9d558d233fea77016d1ae5df3..0000000000000000000000000000000000000000
GIT binary patch
literal 0
HcmV?d00001
literal 39168
zcmZ6ybzD?k*FH`NDBT?jlF}e8B?Hn74BeolfOLbDz);d%Lw70NNOyNPNO%7Z_x(KY
z`+k3a@X<4KW}m&+Uh7)dy4F6yN(xdKsHCWHaBvvXpCpvw;1KwLKlqmjz$g7i%71`=
z;O&*A#NkSZ$ajH@7p7wJVsLO35oq^@NWk^0Z=W>n;oxu`p8w#NOsHJo;1Xh`CBz^u
zdIzZ}I(W`x-6a%La{^jclNOU&`_)rTS4-}zY76d1P4w1CVq)-OmrKC^5#ZY~-}pzu
zy$L?@i@zdO6jl2qp0>O_?^V0~==_TKu;0E)6AT_jK}Ys$GuU9xpl6NmUq4AAAS8rZ
z5<h>~BQL7h+1bg!^v{JkDk^GFZXt;WZK)wU@I4Lh>Nma-8Py=vX58uL5BwsNv0+TC
ztdd|ub|{f>I#qBR_r=Ecc2+(+F$m*+POs4gtgWq$;%5$Zz1W+)IolS4AR!?E!%MzF
zY^hc+(<6BKV$B>q*BptIQ5gaPfd*IJosgp=XT25!8zwm^L5M@!Q9uFabHF7#DiuCK
zt~qeo#^9j@Nyw!ThkWHBd%m>z=MtX)IqO&q%qUT-Q<|7KJNMhNs;a6pC&Kl7XSB)v
z3KtLW;^N}!aM9A*nl<f&LypeAH;UH%@~=<W?^;Mic5d!cle_a$qiaVX7S(%;iC{c>
zPEJmL@k|#dzb#-e#WW$n3TwM=5&|%zK#w5Rj~l++vL_;_xx2ZfV1g5o&-OTVud|dg
zY)p8a+CN(%s#Jb5B@9dFyIC8&!|9!kw<)+dnAUfkUON7RZe*>j5V@t@LFIjKzBWN}
zl6bSB?AXD@u-6mJ^$s$%>{fg;H**<Wy_z$R_0fFyxQl_NC8(r`t&yYbC$Wq!ZhWiR
zaQO>2)_CPI7Ur}ViP+8SHXAZ(qCX-8Pgw>@)`d8#Ub}%C2oieVUWG5$Z;wDPw0oh!
z(MS2V(Mt&eBO3Siyc1oDi4_JnN!?`_!F7~4=LJ<;7i4@c?GuTsp&oXOubJN~j*nRE
zr*(v=l`h&IUu)Y-G0<(qS?nZJwYZ$k+*tNcTf2SvjF$-h6dEDNR%>@OyFp@~wx8sm
zQ@KkHCa?M<DDa*q5JkVi|7}X6ooQlUmnxZR*#{T+GPIBvv|B=xJdNM>M<?%wx7c)l
zdn}q*;Wg|Gzyh^yRZNGE1Xt2+qw~^RE=E{lE|-=pha<-cjHj$U3eSXtS-y9ciA;Qt
zg#X2CIXogbtk+gV#pepYbkK~qZROc+2-nD3e6SS4d#n~;vgmMgi02}r<bB>}Fg(g-
zTDmx2YrO0kRzm_}D5S%J%DnQ0DdV>iV*NTt#-ch%|0xYIm~S%Q_BwvyfdQLqe%iUU
za<4uRAr<XM?#|y&3?~<aXTr-Hgaz#tT;9D;cK$2uv9~aYB8Zi$G+Wuxv-FN9+5PYR
zh4qI=trjcZ;=wL@A=W&9;v>?i^7Yn}O(rs$G1Q(|Z2z1sn<Yns)V;&#c$@TE2Q)Ru
z)6)`Bt!gQksJ*lfL=-F}Q=R&|&-+U>DkcK~7vFVi4u!dmmta<R;msWO$55HcDm$Mq
zYzuC++dJmp5`L=_vy^#UbXhfEraEBcG`ZP)uwPnpl~qe?xmmf+NiO2ax1Om+(c=}a
z!GBS0w-`sWU7FHRP;K|H)xhT98<o%yk#hLBt#8R=)!=YcP~pc5m_`)LD5eB~iYqs_
zj?tLy72jM<E{lrbr1en>*bVr_*W;KI;3E7^sdYFjoC{%b$NVtKVTT^7+2BYNw7pYj
zi4R5^w->W9pqU==f1#Eb;6C!>z+DXeb>TXZwZp;G`tJ%B5BYEu=0)FDN%x4fNM_s}
z&3d^+eysZRo^x@R*na2q(uHz@x%b$Uaf>KVBm(XH446M${l^6#mQ=@EaW_yr=YvCd
z+ccCShFQ?$1}?+o10_*^aev+*-pN^+qy-`EqoDYuWpSd~d>GvmJf>Q%JrCl>)WF6k
zmNOMMMp+hgneL-`A#pMMbJc=vS<;lS;}#5&f&14fUK<#DZv350Jd6LHkN$NrFqGJ8
zlLQ^XPt!kNF7<(R!K>C~vVL6sXh^GeVu@|f9?Xx@{W00`WXRZv(|^df><;rf5PNdx
zBpMO7vZw8Z5tyP#0>Vqx1X*<dv&0Kc;4ZffjQvQ21Vb2S)~HS*e}&q3vP0X5Ck>{p
zL|PgxQ2d3lia{mhxO*i0u!<pNN+=Dr*xkF}rL@3)H_B6SSLCd3U(-4g(^%Ggi&~71
zLC)Opd9psBf9m^tF-R2=i~Y+Wx~{DJ7&!b`+!vr6(LbBVOG$#4e;21&m~FS))*G-=
zPa}wqkl#vdk|f?wwVSv9(#)clt1G+|;%v%L28)#;+6VXk{7iUG+jNKVTH?KH*Mg`P
z4}U@{EuUAnkMX-Hs~^?DoM;li?~cv8zY`mcjZ7JLTz<$aU4RE4@Y}g{&JAw#$bdHS
zIsj~7y#89(M|b^nq}$-hTBu4^z2sUdUo4>v8BBVg%wyU%_MXf@bERkY-D?x<$;O<b
zpvBDRefW<DBArvMWHxmVUG9o}n*WowASDGwz0FLH(Dzku6^(`Vv9TH-Or_3WQek9&
zp{?wvKs0uy5PokudTJp-5ZP7duzj+zYp+MDA<51+GTg}}+XPcyOCY#AF=5lm{bEKD
zuYwz`>Y=}WkR(4(Y2H+%2SHKL8L4sdgQdlyUB8->Fe_=z_$V35@6;>_^Sx+A)!#{y
z%vNutx>hi*td+hI$B#YrfhqfD<l~3DlzhC=#`503zteBJi#)&=g+hzrVxg$DkD{I@
zg^onlA0{~L`RpH-+6gSVF_?>R2n@E#DgbDpEt*<Z@je@xlV4BeZ%k2#RC;@KHPXI=
zTv4h`bGcu3y~M9wU$HGWe-9h0vD0j^8a@f(O+_$Xo~<oWUu<$5exr+)S7gD4{3wVn
zgrav3%g?CuuSSpayqwxe{k^|Fjm)EGSivv!zjH;?(yL3=fBQ=%uWad!A{EMEQA^#0
z1YSLDC4#7>-9zS&zAcI<8lQ|mDf7d6im6Q*Pa;o%x1R@B!{cXusK5DSe-NiT0+q(3
zJAul<M~5YH9~^lh-eTUP{6+YH%@Tq2W2W5mXeE;-FoLq>=#Q(^6LkG!bA|cqcF*pi
zs1U^jfwNgMV;A~2jK#)8*61YsCi9QR)xxWa4oeYf>?DNCwZ`dlvWs4bPp=e=C;^0@
zKx^mvbur_;)>z-nUx7=5Qf6*XICBRQBO-{5^FAFbsff<KpkO!a8>m~+y7p>YE4&LA
zuaOFs0yDxhFwwq*qMEm7{5K*3xDA#58|>mq?5K=|*ax&{U5}%6AKyg@FV2-FFZMsS
zrHF6K$F#kK%8>ZN{<(`uWDrDi_jGHzJr(KJ|1yPYc)l)sxy8H2+vzKLzZc?{M!E7v
zF8j7?Yx*gy+=(y$qRI15nG2g2=Obg4BUlV=;xP}#z2<_JKj<^R-sjvH4CwW2sL>TH
zBFbS!1ZIVePcY_hP7Oo2*3+(G-We}hkFa4=pbOcJV<M%e|C+Y!zUqG{n|J=*>)qtn
zuvYEZ1ox-boipvgxq`t_VP5lxlooZlyR(J3H(<AZF?Y%7yd%9i@7`?6^1pYM<8}vA
z(fcoCgsMf!oK2VWC%GQAgevYXL7o?Eii4GU?~V&J?0;;|X}e4MzW2UeoiA;v_Kja^
zfO*44nT%$s<HJ_iHD0JS+i49q+(`zs=pvVW8i+;xuYtKK!0_&IS4RuYlSNh0LB0N0
z@7M}4b%hGL51O*{(vrSeZ6D8OHCSs;@XmhzeVhf(Slk`m&D&IY@;78t2t{dNPCng^
zl3t`DCq?X(3hcH>ORIJ6{H~eR;2X%NKo&enSZ7_?BtrJ5+S7I!1*N++VP8<HIBoPS
zs4s)hk6=@{5NU`^H+?`wcP3Bej!F{iUVeW-Vmz5k<8mL=To^D!#+Hw%%ierPJ88H_
zPv82kH0g1Ai1s0S9toBo^Lh+BR#Ex!G~`x30&(~+!!jfFHn1X}Pg@a3JmuSLTQ#Qb
zpQ{H2%o6CZ4m_;3gUII{#r0Kl7wZMxrF?pq^i$O~lx>HQg7%WjlKr9^Cy2f+@^BqY
zbx<#R&U+j$A5c+HBI@EZBVkOHH&Z}NE>JzIVq7w@Q)x#`<VnX!98`B2r8m%6Vo+<#
z6|brd_*__wX0{t1GqA?_MGCa@sMxH?;yyPPF1TB!b^Hhn>~1g2KzO8);$a8uAH*&3
zg)D9M>q1*@_xCVo#(yS{1NMP%E6V?!Zdx45c`#Zo`Hos^`dDktCFvy1sVN@{zyDdY
z>><Z7B&4vz?kkf60-^cx)}7#;WUQ;|Ust^A{Cc6ps{`J_Ho33e7$a~vXbL1#t`aT!
z2UJ5oWg&|^mMJOh+0J)(WRFNE*ofb`(PH0t*QCEMv<$k$g`L*rpQiFCT-S{2>uUSm
zw<=5E{Hpn^91&*!WTH|0=HF?*gcfcA(20pesM@XX<znYR-nMzrF*R^)xxTPuJ-mhU
z$llE1dA!{69o)n5LZtfsGN@04B+k+hUE_rgg5;a&za<srs{J-<3Qy6+aIcv(Vh0~)
zkKgAXo_BfQU6|#09=7DXY~^inHrqTNHErF7_5Idg-n+BOIH|92e|zJ_nvaQy)}E~B
z(h@~J+X?{0YlgxkKbSHfz6PX3+q1^PZ8&z^?eZ~;#lzEVFypvJwzmp*NjFhor7Ydu
z;qg_Ml8d`7>U`!J{-r<=sz9Ex<rB%wndw8n3T-!>HqoW|-MF)wvnft=dI;3KrdhMf
zMAysew~LW;pSGKW3p2yd|KC%+rqf%cE><%ubiE&Jx1%0MZ=Vj7xf*$_KBZqWak)pv
z4j1_4XAlThcpZNinfEz(+YMJ<jd{>q^T3Cm4}mOujeA_~ly2A-OCo@BUWkDw$?;>`
zNAbhK1bj9N)%iBl%U+H3#(W(f7`9(~6|64(y#)u!7m=9$ls^wV6HaD*K=}$*?`*(r
z%PAmMv7h_OPY`2waY<I!<1y{9lkvg*S2VsV;C^W>H9yql*ma~+oMt}#AG;;>^x|`8
z#R49-yL6HlpGSwi9Z64GB2a6w($eJZ>gJ#mxlbpK{v}O&29pecI1p8Qp<h1SHvO4N
z_+F)%xO0841x)wd$&8Vw&1HJ3w?MV^H)!Ha$X1O>IO*|zpZSM6>*QdWiv2AWGUOWT
zi-U3Fb<QX6QPsrlDFCm9oEyVYRmw8*Eij=nt7ZU<rce$)Tpe9zp*)({+wiPnEysrt
z%KsJKDNEN+O_ku)jJ2#;k}_W1r057%=`a*slt2m_4Ow4Dxwv;-?t_o4?>G2;iM1t1
zHTq@j=TP}0C)-I{SAz4qYL-K^qweE9?(nlNI~&!^GA8sit}QIH$3qI@ShGe{waqKL
z$gP6l!{HMFCb=eGSRp?HQ$W^k5NfTr!FhVBHe2dKhrT@y7@j_LUMO2X!U<qQcl9^w
zAMYGS1h^`W*Fliy$>29w+j$DuOrED%X?afH7nQAm*uGMru4R&AA=<D2a2dhUa`0r@
zkTJbBK6OSuwLJ5|P9q(YMcV`M$U<Wo(*mXYZ8CH?GyY$Y{x{kJ@CmLnT3u;ei(W?I
zMzAO`+bv3<e%#6GvtLFB?dSqKV^E+x%tEbya&0sX)%=q7bl~KNG#gIFFumkEV<7hU
zq)iJq1xI5@jQ(S?U1mj9Smdun_!|Kx7{jS*z}2gF{sN8ft`X>R+<Q>}*9|j&!o)35
z{$bev7y}Xb>i>Ao|77I<=gZUAg;co|EjBB;+t~N>_f;{P9o0(>CsVsQIY)8(M+{Z0
zX0uI|msFKEtT~ttnsz7Q&HG#`N0?_r9`7GV%pEN$8bm$wuu-KbwUr2qDS>DJ*c!rf
zo^{1mo!8969ZL|;%-@E-i_s)2-OJ(P&$+jIm%hVOr64opW`*}ty|pQ4f6VyiOYbe#
zc=({mBPV?e10QS2MJS$rn8*$>vA(j}V^5F$v@VUb2^BbA=Q!jU&j8~69bkn$Rr&R-
zO%6w?WLzx_VTtLj2CV!KGIodUcx>y$1FafezYvGt?k~pa`aUwZ1UZA|ee$tkNv=Cc
z{JGZu?=XZZBj@Ofc4+btE(a9d-5DgAA*&B(**VS-x*~(_)GHAM5wR(${3Wbl#jKod
z%!O|u`G9Q+|2vP8d@VU)x3xNg57CaY!$@wqG0aZHEbB+A(>?I5v+>m<Uz;jVsLJ*J
zW~+@y6zO<$tl?|lNJMu4hC=%f7vE5WJWMT(IHLy3^({EaKP?YB7ke1l@pqa|SOJ6V
zP+$pl?BYl#$__WuF_p)}{q`1!uG%lf;Sa^0L}IxMpnos@Sn`)SbPo_}$YtIzMIKdC
zPs>vd6ZxC5y|K}HBV94S<<b$`HYa;Z!TZt^q6v86x58P{wkI2UoRXG>hZ~Pm)?ev(
zXVwAnw+Kn2g}4PvdewlLVqV+5V`7&VqCwG_bUp~XJRAD!ZJ?#2yhSQ*$+5}6kL+Ty
z7n`c?qP19mF#uIzF`@8+FFFm+?x5Zv7Kf||U6sF$ie@YBg?|s!O5Yd)@?QvRXPw9W
zzAFH=P#$oWOh*kT(7*Ed?XvkR4WNH}In6(}n`hUC2FjAX9nI&<5ll~Xlj0N(M~gM9
z7W;3xm?1qmFD1O@Yps*F!w=>vJEt4sE)}`;KqRl-pB?W8e==fRQHbNR&yXn9-CPzT
zX|2+4M8JL)^24fW-lP2u?B&<ST&l#FFZ)D-uFKJC<Kab1+ygzELfogXQQ&I#70tnM
zy54n{`>H0Fw|hk<)fQE<3!TG@ud#0no9o!?oNKC!@}4eEg2_~&7N3qT-Qsh9Iv>z{
zm)0<yr)u#7_;6@3bvjJz+ggqdL`H_^?hjYdPzWpB%A2F0&w3<JB_EHSqr-ppKXP7$
zrWt>;AauX_U`)J@pMV`1Q@7$z#}5b8W-NWwOd}B|VY?aWP5mhB2UD^`MzduiO()As
zi*rJP!fC{s1^J?Ot$QRN*3s2*LB2-Q@>P<~K2Ce@@ChWHB0qvX-#@OXT6O<z6!)t8
zuHHuvKO2MQPf~XiwS5gw^y}|f;*G`8qVqEEj|<{jf0-d9z;+?Hsd8<$bJw!TNAoog
z!lOHQ@wE`4mlgY=k-F55@ok?~VYgZ9ii?83DUY{@BQPW0Kt&6>3A?52TkfiF>Svqg
z?k5mH0g<X3N|2TzLCY-?>EQvDCi4-yJ&q_ZOB?^Ly!aEOJF0~0B(D@5kKF=wuhD|l
zU+C#Q8yn1*!P{yZ%n10gV=xIA)5(n8w~GulmX=Xr!2;1WoHvT$;rkC#$)4|!vFT+X
zw1r+D=B(0|$vmpR_ND*2G9vq;Sw=C};CZDWLZX{Kby1`C&o%_Y=2S+BH<R2XLYGSE
z_=^|+ssvIdqk*Vve*U?%e-cz*yJPIOZgw1c$H2N3J9pcuYz={bj%-@o*jDZLI~QwC
zBAzvlhUpVeBJ9G0MuPyI?ZP@V^Kz#H7sx;!p1xk4&&-S`|3LqHbLRJvK%QV=Qkb{S
zvFK<?FJ<lVt5;392rQqy5=su0>1Ji8iLbg#!*Kvdbu_1a%bzZ^ESUW(q;MjVZ0*f;
z!p?7bS9101myQ#T`rwFer2+tPS}~s>qQQ4@qljMnW0?dl0Z$d)m31QSYb~j2ibvAI
z#x<niUH!DZdzYp5SBs_*horD3f0JK!rz1mnfewB4Gb?v?I8aoYjvl{!QI?LHC9;+~
zoO4O`Ss;?Z^Aldl-U@c;ZGbzXt2gmFZFi(IS+89!*Lk<F@bP}#ZyFV(^jK1WN90`~
zt=Tthn9XCnm;+Qjz8G0Z>*7Ne;3SWus}*&Z1bGmhWqu<dAv#}%Gd(=ndllitwoRIo
z=Y;_o^88@Qse8?9I6Pwi)bibdeYC$Yfnv85*zhy4D|RdlSSDYL0)*rCoha8?w<DT&
z1A2C)NtZt0&W>)}@#IQP`74&+aOIqCHv=OPy;3=g?ZXcQU_r3<9`^sxN#N(Sl-g5+
zhm%cbigdiDjZN%d1~Qg0ODQFKeyoJR`kd#k|IXA-l{D|)&PM<Niw)F*#IwjCReL1}
zYT#Hf>PQn`j?{K~c<F;FKn7kjrJhC8ZhL_tqV0FFyv`9W-^|cIJ!3w5CB}a^^%s2A
z!kZ-9CkOI{{doQP-#T1pWlE_zQTCS9g~NcX{+SG4b?_OC`xuy3j+4X;3IK1etFU|_
zwU)#L2MPfH`45d!QU<#gu>th)=$7BR;X|@wqUv5<r@H;<-%;IVy_9xGyk!FH?40`i
zh&xk1X(J4^;^f7B!0d@?mW)-O{}o#QN$MY^K{-1A05(IRQ<dU{w)@%ijcKn;(QdbU
zS>2MQW2XvUa)ii)_u`-rnYHKB?wPOvE8o>{*St4#(@XuD6t}M<Kjr}^p&@SEy-a^s
zicMrbiSmDPIzRp-EIO&}#9Z*`{%WHma(;pwaXmQj(l?ztiZhz)?1y!o?2<o%oYP^B
z6=Bgzb0(P_rjGYJ4x0Ex{g{L_v_D*Sp*kJ(f4vqey-LLa(*Ttf96(dRO5oVI*kvHS
z)Yrni5#mxk52&L1ST~|zsG-JRTfTf6_;CA&S~zwWFHy*R*chhF3+&o6)p-`N@F9?l
zoSB%vtJ+66h^$SOipE>?*&Oc(2)=DpfDruZ?u)O3FB}0j^bIi0vt%bB26EIJ=b_`j
zOA1%H`Vt?X?rOD8woVkiMxan81My~V*uaT99~-NAswewgQT>020z-bjLUXyB$=(5F
z9G~s*y0#nZkD*LO`6f}2YVh8EaNq&r+Ak|aWUE>bp(T*mSAqFG%OgQ+P>a@EZ6~h^
z>o?rr#(V;ElS_BiX$tkX(wtspX6DCW{Fg{%=6j`jPqn|>OY@nhl4Zk*<q)K-4WtE2
zT=Xw+#p}(Eb7^W#JwAPHY`QOYdJj{syXN6!1lUnlx{1dr@tqdHzW4wjf0jn517Qdg
z=!?1^@thZ2vBvf)uiJyS<5q(c)qhhM#%1R}>Uml%abT1~`U|=XaVE#c%g7PeQvGo0
z)-=6kkfAb6jV(L}fX;C^-057|Q>o|d{FD#WUvE&jnVy&A0k!CR%(MCH2e^R#JaS_N
znggoLNCg_gLo2g1SejKFNA;^kwYJ$B9v)jOBm2@Ez1!lKMa;9uhM-!^XPe#-5Da_T
z&m`v{tLN!0^5385dIR!-r_<Mz`Kp^bCju;?82igI@{uap{k-ljoM?_CwNSE!ijVV0
zz3^eE&kIb4-5Snu?08)+a0#C*>;Y-^pa!G+!0pq(t;w3<q&Bj#lW>Mmesfu0*hRc!
z9+d~O{*%Db-?Y1~yh*E<m_qESjvJr`{%e1IS9+h$W$(57K`FKllA9RK+##=SwgngC
zEztt?2qCD<Mnjl+!{OwtG*ENSHW;<0Z9W(_c)JW{tnWY8)wyTyk2BfSs+*cg7X2EQ
z=d9!RQ!e9TV3MPe5(d$xXXIzg%MsRqLI|_E<WHe8nFI9>+r4Eij&q(H_&ItXP8kY0
zhglRO!te5?*<|T1*jqku^}AWsT}=UkmNZ?_-w6}<0=A6MGLhUG6ZHH2?K<P9jEQ}8
zDC&66za{7|^II6Mi%l4p#)<PZPsg3p@u<d2FBtkyiJhz@lgzafkLJ#UM3Wkm?;XTj
zmT!p+XUhU%EmKC<2#;9CaPjnKoFz;fD2UKSp1>Vu{yM{NH3}k_`}6Qyb0P^HP783l
zsV!U7SEgrT(W^G`bA^LWUP5e($^kB;9^Du?Kna&UdyvR;Wg6I|skPDw-YJr4w#ER(
znm0{dZBp;MSFJjxQVq*<YpI?-@nYFvH8Z-zqQ4`s2@8mGHbH9ELNAkL)kb@!ZvOzL
zbaE~m)6AOw^82--vSzn$W^!$Licpx+R^5ALo10FnjySnP4)afW9Ejf9@0+$`DLTa1
zH`e)E4e_21xXsPBr%mY(v^fmgTT&&@_u<6rf17?ToV8FwGlSDB1?s>OEw3~CeL_5B
zXD#8)VY7!#1e-|^Uc=Q>)#3wU7FH2@(uewe0EqooU%ubdPph|-Zb{6lZ$-_+c=Fu$
z6$Wz0#!NBZb&#$q9@yuSlAOHCLTk97!#j{y#Pj;7dm7`?qvAWeS>%51QL`21@+god
zr-pamJ8Ep7Y5(wTr=3<(`G1$X<Jg6Mdr)LDm=W=YXEqk0J%gk~N1(175Fo?^mDiec
zM!sZqrY0JrI6@L<(R4uPysw!?#(=sMP};O-=5t2qzi1Q!!exVOpqdSv9IMm3MDf9y
zuD5@Fv(3pxMvK>!09T}tA7WAoy;OJj-=}>aRYYTSwU-{%bR=a0zf6?!E=9~z(>@6&
zK8C=U>}FrAk$gC=0rbZ&^TqERW<P)XLdr+5L|<EF)G7sjxbCO{#0Q@#47%PgmKZ=3
z{#$yRkDr5LjjD42;*<2=GR{(ean|!S`bG5m{QY(6P(ZIo*^`Hp{Z$X3(?(%!(@IPh
zS(4dY1Z)srZKK^fSh_<tsh*Vy<*<C9z#oj^8y;D|LHPEHTZ7SL?TeV#u?h0|JxM+F
z&v$a-Cm}G-mJe0>S58vXkI_a<dWy~KlD|J@-8tEqdfE`m;leCV4yK19wqxDYYM##e
zpA+IgG$D#dULW#0qAVl*HY$G&UF|!Im(?`qu;GVQWCXSgr075=y$@$)BfPSPQP`6|
zmkxU2MMw5&0g6{+u6G1DibM`6IA9M_94K+Hl3+~%XeZ%rpotU5r(X{R0jrbdfAqT9
zpnHFA+gctLP7gX6g{}tQ-p1!$-ZxzoaYY-zy(yIbD43H}_9niN20+k%wYhBhaUy~a
zogDY>^k#-ahKEh#r<;;>HWpeIB#Hg91m44lI^)OV>+6|oWTMHiMVf~DxxLd%!zj=S
zpkh(X{r2$k&`JB)d4s8sKY94&!OMIc0Wwk9$@`W)7}h?)^1ln8<=Nkg*hiLE-ASh1
z#JxLb_z8neh`oAU$Z!1k%|-|OK|tqCp=r0pn?ez<JMjL%A9Ps)>6LkUH%{PifZBg>
zCU|i5;<{*hMp}gt1~raLsrM+_UiP#R8v9x7jZ=rGn%OV;+t0WesG8>QBy9_Y_t~Zl
zI^~ok8{%#h2=bz$9n<5{@-Uw~H_y|6Le!wL4de)@>mmY=4n`dYz0}{_)>{s=YFwOM
zEl#?tC;@{231|1ZA8T%#4%kgr<NrIAVV3z{mQQR4f;-v`U0kxg^_FJ7eQ^{t-SV8y
zXm3N_iycbP9)oj|ix20U7BkXB+=cW-_0%QaOj|zyF*)}*)4tJ#-W!cbMM=t4zCCQp
zgJtNsV6X8Jtht!{zR@NyDX4FCb%BJYp|$utVtWfdqZeU2EbI1(r&uwtRXrqtYFOzB
zmXBU42b3lAKw;WVR{Az13Sfx8QU76x+49QGBSXAKJ;z{sLe}vrhwve$A28WZ?T0jZ
zl40o%_w8W`%%vOGY}C#$guce=W?{is)$3f64ji=tEC9M5H4%Kq&iJkIu;J2SIqq_p
zET#ZpEg<6wtS&1kL~O^oePwi--?(*H8i>@5m)l!C{Y+^qt%3m}fK@pR4zx&~-%Tn4
z3K&qP%<@>;9FXw4t=Pm@W5{x9VwAO)YuI~E2Q3T!s;_x!IYW#neb^pe)?OB5PfB<}
z=Cd?cb3=Ixf<5(D=sJI!eC=YhU4t3sT0_3%9obKJoO19m`$zf=$Z#dp7+{b|Z2r6y
z59t+iJEcZ|3Uo|#`Ca5`Dm8`zLebx_L_eU{g%($mR%g(AUWX>81#~y-+@2DhAxvM6
zi6W)IZMj0jhJJwfS%`LLop4suSKKl+eB<7+(BjN;fZ|r}%^t`{Md5%XK38MZ+U45B
zJ9wx4WlMja@ZtmP6zl7Lujh~V_yii*S}FWl?=Y<?z$7t%iq8ao)mYO-TKv7)>JHXS
zw*82~hH+$dcE1D^C^9^n@|R9syt#gfU6_0MUUmp_YUkc}l70g#@{)V3^x;iyMfwnm
zYA3a_=MD~U15x7*SMra9r4;bx$!CXWCye}&Pl~tvtI~34_NX0=!BN6ix`(Ce5`EOg
z)b=P~s3cD-LZ6J7-vx!vMCHfi3s-ysvEc5wa^C*XQbliv&wm#^vna_)&kp-03=U|&
z2G75Ybua2_)XsEv-FxuMG7JUa1bo}yJFg74OUMK?s9fz0IW^%ct)`lUc>!S=v%>u+
zoEABdTY;uWk*%G0?uFLRoSTD0pxP2mCI1M=hH42X)y&61S)UJ(Cw}21G$v}NTbx#9
zAWe(#@gYCzR|HL~0gxM}f@k1Sxoc7WnuEFXD+aoYlZQ~Ft472{lV*0DXxFhkiB784
zj&g*o=d0(m1tOX>K3a&hh`bvh_95};V}22-4}OCI(8X7<5f}o0$w=mJ;a#j!Hv$Py
z_7_0g_Fu-G3Az{&ygJ&?9K>{cau|F1&QBwt{)yx}1EAXeTLI{RkxWjvn?h|I+p#2&
z<Xj1j$#p-G`2F7-7Oy@#HQ>u>pqhEp8MN@z@L(k~ZHWAb?ZPr(Py<vG%Rj#bSgrG3
z(usssK#OWmk;PvA?+tE$JN8_JmBy4s^Uiz%na6R@KrwAWV?aUsk75bJS4Gg~egJ*#
zSDa_`l~#l`k$j0c`QPsY@KkZ7vT<6i-TTsE48EmAeu}duN`zwnBLgy|<gd-bfC5FE
z{3M3~nsM_cog)H<z*Sx#&h+0(z$loBLw_SttNw(-vrb`+(~6Y+5`3PnF8*UFBq4_q
z<$DKVi-OB)H;g%mf&Rb7VFr(8X7%CSX2-g6$-JU;*(}_lTe2NK`NW8{pdurrEc>yl
zpAn6^FL**iwePkySaJh__2YenuX%leBxJK4tI^K2n?C0{PWFefIDAg{YtaHx8Cbgk
zdsu>nL=N`q@rz#zQ44E!C)~(q=KmZGlNtq1e?fp_VZgyJumhMN{?$z5>6l%n3#s1?
zcONTLrjo|MCINwc=QwdFg5)TOpogInYC>Nj1AH)$2X+Dz;1hO^?L^wXs&oCS_v$gt
z0G|`eTFC1&5aNX1G9muYlVY?Wo0|w)c7;t1HL+EJf!^Odlu=z|U;lZ)B1Q^<h3o~`
zToQEu8aX%m_jYv%btD5X+cG`h!a`eYYggyE2sbAeeSH5r9ISQ&{5QY@@|HBO7CDXV
zU4XN1TjLC6`7-@|FubU*x=o+JI2r<Vfs&ns_yqAMH?O%)%+Mvnvvy#W!G?&M0bl<8
zK|y}2sJZ5>wU-aRY~+ml^&eVVBHKFg1-zihJBIpv_JqBxY#0+G3kGV$kFDS}dQ@vK
zF-`tD4=;WvHDcFrSO}_D%dtATihK+NQo&u{)PMz&O*cNVRWvYg`Rpa;<06eN`3T#A
z$qc=d-i}#wgd`NgLSaVss6q(h%;bv<T7~#hipwiK2?OLNB<XB=lzx#_5GE#to4{h_
zPJ=w#$|Pq61S!!1nompJ2c=`dGOzI^|Ip3UgNHM(Q}VC>kgMr|IpWoowe<AD>D3H2
zcw(W@!Ka^kFyj0jT0Q%I=pd2O-pi=F6Lhj1JXJTR@DN2&`MHAHKQU39LKv^I;DF6$
znN+CVjZD>G$ezHFAOWeWbzNZvm)l*!VQDtFXz;5$bArquU!@tBiV*&}zkrn1^2H{H
z3$+Y|SPU3KGCPnFv;!=`e`8`IR4#pm9h&mi_VSW#ZS4ab2P%Ldu~y2q1_~rOL9f4a
zLZP}!Jx1I1s3F&zo8p=5Om9XD_qZllQWrFD$lp*VwfA`9>q1^C6UexQ%5e6Y?<%Bu
z;fqL%n`XzKoETaYyGDqIFg1%353Q}kGi56)L8$D`W`xBi4<69M!aMFB4L3wkS(uJO
z$LJsHtP6yIDZ=4TqWo0)1R)10M7dm}0FD|gWz2p`8bBE9>ai;MvEJ)#pt2M;9Oy!f
z0k@NmPFu~hus$|gFuJqrT|T=VWdPyi{?EqLT*h46fEWQ^IEF&#+q^G)E)Zw7oQ+>{
zUi_4~?LCh8S`aor&A<}ksb9}_fx}k?CAj|GQ&h$Z3GYUkG~N&brlmBYB9dz=K$#`Z
z1P@Y;q-JGfrc0xym-g^&pFz6z{~D?zl?j^U{<Iz(#g`)=x*;ySIsX+np-F)AGT>Ej
z0Y1O9(ZS!y&H;jmE`tqD2Bt*3uAUFhSLS#h*P96by>%_y<g==G#9Yj;(9=^z9(dwV
z9Z)7{GZjplq#dVIJTnc>%+G!Sl}XXznHOY;0Uq9Nw1#%VP5{Kl2yD(K@pgoso>~3U
zJ#h~aDkGDT{|Ek6fVz7^A1|K}TRIJ8SX+-iu-1^@>ysXKM*N#G^+rChC|m|6+rIvO
zJQd>SIUXHGN5=lot!Ka2C~rB}(eUpKAq-56J}_LVx41c`n=`d8%J&jRt2=QqQIfN$
z2JEMJz?MM0VAc{|sOlBzC;yCF_!QMYbajrEdo3MxoZxT7swhMX#0bN$e7QAL@dT*n
zcT`cZ^eg;HZ$_pqHShPaHsAO{N<u|QLd7MXZTa6X*W>APbHN*LgQMg1-PEdGxRGg0
z!8%}2MSq8;hg)R3$UG5<mR1S!1bU16g~Ptcm9r@)OB>;XDg!>0sMTPD5&DiPdIllP
zK&^aDWa-~~)8DeCukYcBo<zT0<#;8eSMWTlA~5RWS>NoJq_vp#oCyM;brmBD_Kc!t
z<>jugH&wZL5x}AO^%;!PV70LNos_y^p~=|HW9MwpKkJ}pTtACeZvK|Q^!|g&^IG#5
zO7p|#b9L?eQ3qK6cR<|8Pk#oCwIO%dof@PRRIs4P4&cV6AKNVvTN0a$msX75wSxEV
zn@rUy%@@2fJ=cQ?J>(M|FGl;Jnu@<!7ra^d(e`T6T7G~S99PzrNo&$unjbar=-5Oc
zfsy67s5NO#Kc-`K%H5P8r^E_knSOa}or+QZ-GAawZ`MYMLl-1$L6e{JWAjTzMjcL1
z{aW1cb_#gYo1;>YDDAGW2ww&;?j{a9pwpVx-5}e_%HXMtr8VDVV=lO!%uUi~gbrhc
zg(8`23e=y-$QS}exmcZq#$OM2XR5x+$?veJ?r*NNZz4GWcUq&sj|&MbU_T|oj~y^)
z0PiNvVFwaJp>>q-Z=$L)BKu6X>||wrg)!6af3HuIR$~oO(KtU%Wn^U1h^Q|e%|Y3F
z%Dh}Vvo0omUdgq4_H9vtPUC~h7}uXH4&u6C@uOQbQ^tEQk&H}<FZ+A3@x#4r3@_HA
zqG(OwS2<2-bH0=|`#4WHzTKabBs_NJDL#PB*(*aB%}Ep;(ccD60zE+?Ne2AN8FB#k
zDx5BZ6h7fT$0r>sQ{hjYN1Z`FcQ=ll(je}j{KJC%7gAlEXETwuzuro^QRcM{w8f^N
ziL%8e$ET;op7o8SFbhw45s^};4{W+#t2-Dl6y~N=vrd=J1v31Q=sLdHd^WuYAZTvJ
zbtRt)b<w>7(UvC9@SAQ|wj1MdV+^D;-pls)0WsZM*BUEDYAekM(n<5l`Jnm2y-nGm
ztbx?V02efQuA;0(icKgT8U1!iER)%Thd^=&AXIL6U3vww_VTLNMV>y$II*UpFZ0TK
zX8WH`5gk3b=G$Rwp(zU}TJ;-nt%JOPHcm{+W|nPmdW*-Rt$qYsgZ7IC!)NE$HsT7<
zeBg;xz$T`=nEC@HR{5v|u_N_z<!7k_0zj2KSEi!_LVN@Dh4yimuEcoO>&Lb0OCU|?
zLz0f<qLs(15}0Yo=o28*TnY;P3|prezp0V2g}fdx_xPDF1<?c(JTfiuV}JhnT+vGz
zf=>{VdbCDiV6iO@w@%eGM&C7ojUSr=?8sK-8KYg)pQ*`rOyP9Tz$FDJ$C0j^3cWfY
z@6Xg?51`Gz6mTWVE^_yH2hayc=ZW89TpnJoGH!Gn-`~C#DlAVxL-4yG*5YSBo8jOV
zsH0rCm_A3OqPRvWeS8BQ5mTvtNgVn$D+6XBGob?EO>csR$}+PNeUDps@D&vYye86?
z+ehi;rQ5@jxScJRJCRwY5nlAgpp~wt-zggEqK98k9RpL&%a+CkL57L;$U+9(vQ{Mp
zvB+l}K>;*!*PQr~5dRpMud*Vzsj7U~q_Wb^?qk+EVms`mFOBg|sKX33=+CIA$i|vQ
zQa#5jY1f;Spc#J`Hw!zhAGtn^{6wG}e4ph@7dL2*O6--g-Q{U_GLRyfpbUi@Mx$OK
z69&2MA}f9wX)si15G&J`fZMgMcOeL~;a*91cp^7Do13j=<??Jgs?_cT^SdNeMI$D;
z`x4_7{XQT#Ga{r@Uf0yv6`syQToO=eY`_xWfQ`6|CX%h(>nnNs{D}O+7LVkp$i(RW
zsrSD#Lm}fH$pfF*h+fJaq_7n<z23KxD~=lH3=lU|uucoeBGW`sg7o3;{BD!P7l~{Q
zY-M|^6XcxN*&~my05NaGB77BqmnPwE8lz;w$nYC?a#D&-{y=xEs?yW1qfBIqyTU#^
zFVF2Z(8Eh@>7tvBg<NNxv&s`t;Ew3s4^CvI6h$OWc)o@NolQuKLXHNk08jI4&tlB0
zo(4i#2jy|h_6n<XOJR@FORTdI_Zgw5z<l5wJJ9gGz54E0QY)hHOAA^i5Il%>8Xnw9
zH&&u6#Nb8kQ=J+aN&D<y0?-*Sf1X`B%0|})Zvfc?&<Oz2T_LNcT%6V9m9noU_~hA6
zY~Vk}5P9@}@`7qSr|@!HN8lPgE}9hYQLEC467X0ta4&Gq2uK^MM3J91R;j*fUetM)
z3j5IJP<i_khZbiAb>~lb*qH^$ucqO@0G{$D54LvVa(8(^_QCB}YAQv1e}IgPOkag@
z_YdzR>1aCLdIv^kW;q6YTr@9AdU|@|I+v5R)>cv5-6v@onYk*{;nj{nLyWaC(_op5
z*?KkC5L6Vb<<PIWE@m`zyu4SPAw>OgELGLjgz@Y$RGql0q(>h=Q?_)+u8imie~DZ3
z1t@?WQ{md$8gL@Z^X{O=&CTt2D+8B^sOEET)KIFRU+Z6yuCcMxllAWM3gq;*&*tWg
z70k3he*6HA=~#~E4t9l+1*=8|L!n6=#$CQhXpN1He{zPSB2z!k-d%O|d>ziD;o2T*
zag!=pvnceJI2n<YGcnn!v)^{RIxNY`qG2_R5r)z-ndOm3*EN*}_-E#C0PVO~KnwAH
zxR^fM9uDR>M(Z`I$jI55saT&VP)Qu<6C@Ld3?}m)Ej2YHCMJGb>&r_Dg7&Z1x!lP@
zWmJa0^+Ht_-R@mCCirFPosV9Vkmrhn4TU3&f}whECtjH<x)S0gPD?1<0Ifx3aUT%q
zEgPH7Shlpt{V`%sZgFu`2k3R7T8Y-h{&bo5)8k6j-@jk&L=5F6ko=kuKC;l3=0@QY
z$O*XJp4@nx>~x_*QO{iQWg)$t)cyXjam-!$_2%6d7eujje)F=PCoFeBFrj&)3cB11
zyOnqY4o#FUy3*J2#Cnb7F!*)h!Ec#ZUC3zfWZ!v%0s-EX&6~nlwinHR9~oNMH=zSz
zcV2GxWFMQEA&$dUr$vC>ZpQ1!vFUql4J015JUt#RHV9EhB&ZhT13YegeB97Vy;R2s
z#H!!y@i3)pg=nmzO+PXguwS#qqBWMM-@;=xA5JRhb(8da-uXy&xThXi;cS&@(cWHA
zt0JKqT1RDN<x9=DxVYP!n<kGNT)oZn-SOika&mIfoIftq$jHdNHcQ!~X2Qb4$4u-4
zy}i5kV6nWH%R|2{r_V1gioD4Za!CUOB?&qRU9wM<>!|yfJ98dkP=!wj>TN{4Y0Z@m
zrx{$FjehN|8a4W6+nYqkGBV2v8oFK?3g}+9UD#bj_O2^$6Z|!i0ze9Sk3&n3<@g`o
zf8Yo>8EVITD-~Ks#!$$l@(xVF_YLS5&sAG#o0;td;n3Wk?;`4Be;C<@`KB>bQ?CK8
z0=WKsgY!}5$H0}RGa}}hUsl~J@)X}DXZxp32NI`?oR6lg=FrLbEmp?}$f^sGr6h)v
zMOuLNPk4AZE$y1S>I-D#cJ23ETr|JoACrYW1sW4$V(_sE?lT-69c#P>+Wb(Q0Vi}l
z+hWnK)@kwjz|Wtk@uJdVvhea?zC#3S2ldCna?A46R2((a3=b_WZJS%Y^ARRze{b)b
zxK0LDz-i>8e&9~h{w8uWsjdh<AK9s;r#cFg2m(?V86}?g-HoMDxnH0St6)ocegeL#
zmI;rC?c-9_zT&S?tCU37zxlO)=7bp|5uRPeVtP4-QAMlz>+b3*c=rb1KVKBFTwzDo
zU?Y-7%HShb;k6C~9T`y|jq-z=s9<bzGMW`?NT)AduKv@b;1$)_d!8qc>GYiF<Yb$S
zuKxHE?HVA;f4<m3jkB1qKG^7uPEAeCL`aS}K0XG3VK{|9nlECstIK3(Br}AFt-8AU
zN4g;XM{)6R3PB#1V}ox5b!2!F7wZusLy275K)4$4iC09yeBb*a7>a4!q<wJb<UAhj
zkRc72zu5jcV8w}b{n`v0tSRVjv6uWqqO)scXM-zW6jJAUgH!R!^@f~+M%Jv5vU$u;
zS_$lAO$ma$^;sP)ZuGFzH(CS`H!&#)RSxlpI4MCPRlujHkU!BEn?QqJP>>8bf;raN
z-j1N>OI>(+YNtWwVsyArcdYW7FtD}Uu;a&IV#{%2+|V|x^4quHwkO%6xPnf9YmGXg
zfBrzD;Z~pCsh578EF>sC0QOtSC*Cg&E9UHcSLG*mM7JNjZi5rjIQ9}XE<XN?GECXd
zt}a{xaY%Orm7=^nEfcmfq)e|#Mpm}L^Nz<u1sL}`N6Z1fq$^AV1~rOjbN?99%ci&v
zHf3bRpawRb&&)51gyg7%qE{e@DX_EJ@xtiy7Zi}h^0dqP#sQV0qVy*j!_S{pX_+jE
z6nAPE6m^P$&H<B3Y`)ji!xiw_UV}IKCy=hL=k8d}#~<jn_Vz{Ra;PsEnXXA8FCOG;
ztQR2=Nb3t692{ULY@P#O<mvuwYcM%D{xu>Shxy()%CNo{D^qLVtluCb@p}tMy09k#
zB4S%mZf>qho&v=OH*N=rp!1<WGMYe%Y&@&jI3gxuXZzFL#Xhh3*js)5<p>)Ua`wRj
zmAsYtiK!_%CjWfy_wNn4Bp|Q-^B=F*v98S*8eQ?4#YQMychIs@z^}m^E(F7}_cwB)
zMsJ)T>2R<<HijNHhR~p#58h{wYY*EJsra#BMe33~<2_v*tc<ho*eAGNXOn{4rV293
zUxH{$fI~({&3Bm97l@Bk-#QYxOvS~;3ui@Tq0k<{&E|fYBc>+i<>h7Px6+o61`;7D
z2I;Rn?atR6LEi(XT=L<7EOCvvbAN;Onja4j@A?g;pwn02ElB6Xh1qH=vPOx|pWlW_
zBIfxf23^4k-`yRxsH&>2-cd72XfRI!HqVsu?+O9VD^^xjMGMvZ#8wR<WLe3BU$O$i
z=I-jKx5JotGZY~h^*);@QDjZ*?cr?}9IVp$3aB|g4fkP&gwtTLMZ)UuMN{LV;`Ehb
z;u5p0KEIkzw~*NUdw>OV`(nnb`R_JI>w3Q*002();%M1>rrfZcX~+E&m?LpuzQ*Qb
z^68DNXvYihI&7_59S9*+y8Pza8t~SpI(A+}1Xi&<fQAN2-zGD}{D7BI;XxW)1GH+y
z<m3wgLWYs?cK!K-O+Z~&S9f)^i~<(3k%GJgM@o$b{cLzN7#z{O+?%vKUTIrhT^-?s
zN*Tnkp}sA2314x2ZF!UIzp<8U-$yl~M6aUbevX?G6C1rxM(%Y1?&Nm|+$<N>M3dir
z!KF|&HXkToogRWsggy{#342>g;Qy24KB+=}qF_-v?2dhzt^D$EdqD$2TCR(WQwzkT
z0DzR<RhK<m=k+mGSg{P`dsv96CJEJ4wb>{PNY5XOoY%WBSmV)BM}Rby8Si$nN6<)Z
zYlP(AW`Iu4_YJ@%YinzXfKSTGV?Kx|qAs_$w-PyS;N6{_V;3PZGP0fb^onWdMLCW3
zp{Ei)5=dUBGHk9CNy96d`NBTf1Rl0hy4{1k3@sTWf((D#AGV`lQRG-;RMKET2+?uN
z@Vd!F{$(_JLSR}tUw5<MrgoBYC*TJw<QyX;2<c4XHXplM_Fisu1?o<0AU1GlB<~-t
zmZMBrv0p`m(Wl7+X&dhPlh0~K6$$6(p1-qICB?-Cj4tyHP6vBSO?P{fMVWy#&R|~!
zC%_k*J@4`q)7!C98N?f#n!3X&(9-;-p!ptP2>UCM(*Ah1r{fN+5tvU=5C`#>5WwL+
zt$Ugo#UTLob1Nc?Zh8j0nsE2fV#5!ZFNQ2Xj?e4lUf<IFEiRVc6Gu3Z>X&R5ChQ*-
z6(`@}5dm&T$5`0dDCFpPvFvq+K}thJM0CCV1J=^=1SE!~<fK_|51!b}fP?DAJAwP#
z`GGn#y%sJyQ0&hE7gGbc)9*D+-cMenC2L^S<1itQ>*M{h765d)i1*h+g7C=51i$p*
zn-y|%bBoFzX9ZrOB?-BxAE!y@OiDnGcm31HQuyrw*A1KoGg|$VC2vFaZt({Du{xRR
zzdg+Y6@-njyW?hL*NyuyQ?pCH?c?|f`*oK}NfDKeh^_Pi(sk&It_~dyB|8raAUBYh
zhoB-X)9++|48)9rA$$<6`eJNcWxZH$3wRGXANDW!GIxN4f6u@!YlU_|OfFd?-s`bw
z*KG}_c|R=u-d}ms@e+*mLyC5EcSJ$V^lWeP-5RrmQV_sjXLg1~o;<D&DNdLqu06aU
zkck}G_^sh|5n#>0hJ}-GZLPHVO-)T5Fa7bepkty2u-EhU9G=F)+`O$NrxprMV)&U)
zt#SXZRMR#k*Z%CS?`?<-rWi2k>n|apq37r4pE(o%uu(>IhlGb?zI^%Jikl<%htK=#
z>uc->4BFn_-sa|J1GL^$n+!tMcrq#~tg=lYOb-t!NlBB^()2Vm@bPiv`FeZi#AK(B
z57)P+4~1(V%3*iN<U$^(s38DWiZVqJIkxq6w6_~rjg$t<K4E!Sl8}%9qbw~g9gDLg
zFX4XD61p?aIPoQrY5w?9gB{>9?CeUFuD*|Ke0;=9T#fTxEf23Hx4`&dj#}!#jWrxB
zSkl;FG|<yg7FPh#2EzM*I>(Ds-eIb!LIHhFPZ;G#Fd)y7@<OQ#uMvFIURwHVz0FyA
zAqtKh;m<iEWSe)sci{7rCd}eIJ8T-!s*=7)yDgI7i3i%(_^K>XDRGB?FM4T27~baO
z#-5zKunqkRV}POdZ|XY^>O8RqE88Gs<bN`AT_9x93Z)e3WfKm_|DIF&!9&1gZG*OG
zI*HGfj%aF)|1(3O*4y~hBRqUbKvQXp<j|qia}0@;#|J^cuA5qKTm@EdV%+fHKp^0l
z6mH)6P*uD!y}V|%NmFE&b_p{Qa54}_Pulzwgc(q&Ydsn<bpYwle1QR)9~BaiS$98k
zpX)@2^c_YWiBFKKa@*JAahu}WKUM601Q*AYh5kqlJYNEkuScSoQx_P4qEr*;!wAU*
zL`nlQkSE3`c*V#R(c?m)QUtu4&Z<$ZygXNd83%O1e#-(<LDoOo-dirt>*xA3guy>w
z847wcz^RTLp=?767~m!NVsHFtB9;s|&_vE-F<@-f#AgEkMFRskCcrwve)B@zIf3ad
zQ+3cgFJE@#@dWToFsQsQ9i>uX)>LUv9uau{o5Jb?FG>Lmhi-*bqB1SZ!>wo?rFbVm
zy`5z3)X_6@5PDwu9a05<y3Uz@8@B?!IE2!?F^k8XD7A{=A{QPC^<TKyCMU0Zx=u_3
zUdo<#?)e}T-d588a0-U}4iRMt&CAo^s|N#`^5);j)c$rqNJ3YHcu*f*g_)mNlmc-w
zuoelE@}owNdA<dRx4X8VRt$3e6l;u472T&aIh?qpDvOR>;B9juw?jKK3!X2Z<$=iw
z05YbN^%Bs7h>D79V|)5+?9;|A4onVtG{rT^c0?V#_sQyHm?_LCKF?Dv9`7v(DVh?;
zwu+GTaIEC6bKb&n20jQ@=-BL%z{D+3T7KP<*Kf=oh8KXX`n9<=tj*HR`Nq=2DLGI0
zv_C9%I{hsO((3L$ZL5-taqGOJlIpxsP6`~kaTFCdoTSV>MMXEPI6(<A{|jtVP@a2_
z6;%d9LMH)c9Tmr;G`B;3jHX@tHcW<*$@^Z$`M4YhFK?LL?i;<-)C+vo5=1ftElxi`
z^9TB4oRv=vj4a6Ge8674xuDM!Eq~M<8_s)}PeJ9|Phd=sSMX+KdQBbXx;8dx?R}Y2
zXAo&D%BT4Rcv}5i7OwfJ$>@JY3pa~;(brxX_`pz+-r(8s5Sv<4cka=UeA2MI$2xC)
ztcTm5if12%&wj0yn1N#8PvQLD{W`(4!-~j}C6Dwq(y*xhD4L+{@)e;d;h(6~1r-11
zoJL^C@d^Yu<ajMG?yi4@`rg>U8XBWx_SOh0()gsYHszJ2(+Js)i|}!hw(d}t3bPW#
z@zo~=q1^fo$dXTHo$K|ExLlUxBj{p4#JO3`L)3{eCwwAB%BwDw?S-dfe!Ft(bfBkN
zQ;us6?+G0NNHrc1SfvZh|LI|@M#m+x0ICkR%!Ur+1j$#Ma+b+Wx>x?tH8|`eY1vxW
zS#(;bhxbH$Iey#n^#8}yR|ZrSHSJ1=bfZY;p^@%7bf<JTlF}t04bmmuB_g17NH<6c
zNOwth-^KgA-@W%YoW1v|nOV=w^Nhry_ofOW4?FmR2pQ_ZVH4eEjT<Z|z;tvu;4sxq
zUcGf;f#~M_26(516DjzLIDo$)lnlF<JhUf#$NRljI`*oyJ2#VmDNBX|>^gREQSoRe
zkIbP)iB+WETB|K%iy^#OzK~`m=v~LxaH9-ibWzxP^kUxF?-EW(p(OI<xrdTR)ySyB
zLN)vFoEuH1$%;`?&1#O>@QY6<pYp}`6ztnpKGGC;S){=J%{*PUl~iW>I`sY=4;D}J
zje74kT~V?6b9$u2(bhZV8GwA~Z!GoYkUQAzDf9eTepz@T{Aufy(2nlBk@$AzC)Uc0
zYalt@waD>V`p3GG`;zwXKr&W~4qCX#^3T|jl5>q-DN344S$`%5gjg65Kz_pJ2Vn0c
zL_rC&-txSn=2~sPEkX)rj2vFy|5Obiv!yGOr8CQK<MNruoZgc@x1_aGBS*XY%QWwK
zl>G7kaynkT%q`s`br;J^OeOnoCL$>%5~T9fJ_|-c$eYhU8QijGaLT<Yd(8^_F6=9j
zyR2ee2q-o$_h|t3<v*5n#RQ^YCY24fNPOoz%ze<tlc*-W*V{}#`u0h`UBRM}5*?~}
zm6Uu^(f(y{ZeHZS85kD`zNYY#gjB@8>S)%MwFDcip1djSLfY?OwLgQ^ws55fI5QY1
zKZWdb#s>U<vWzfRxNy=-D6B~jyB>TQ94K{s1L^?nj|>+j@BoB+eed#&Fn%f1&`6_A
z4<9z=*52i2qA%wU)mFm+j`lyuyvbL=dlS=FR4$C=6D}qUOGQOuwGVy?7MvM+#rh(e
z|6RCZ4Ue?Jo_P@(W{KywTLNNHV03<}KVSAa><MS_Qu;aXgdm49IkkHPgFN`;Xcc;M
zO_0&+?Rg6kR{h_fI0mH={v-BhfUemnJb?QHw=z|4NH4ZBqayeTW@=xp+gbi~M0e*n
zGd*Qf3klfMg`|!6(@>vYoq3JfmdFzJ3we9)qc|@Y8SiROT^nGtd<F-v+bCAasU!1c
zT7tyqNEUVj<GnZ3H7sCP@xy-58S2n%dZv||B=j*~$upa-4&Z_6wUQS7xVTK+n9|JF
z2lZoHNc1^ee$$8Dgp7=gX2G6;o`d4g)9P?~ph7@>hL@KY><OYgm87B3Dk+V+Z-((F
zoX^v0yL#fOMMTpt@xI`ym|zWM^JQc?3%1|2imsOB!J6y|E(-6k2&XuxJRl`1lIOc8
z*48)Fk0g6Ud}&19v=%2droNOBo(~=!D7r$7r8bf^CG-&dGOFP@`!exKr<(`aBBW?N
zti2;)l;KSg0d~L2k~+QWf2KBCS+35-y#!3vzz}ARVu)%dd?A6uTLsRTVO(Xiy*DM-
z_cyF&R2<1O4`8*WsBtzow&pZiug0G96yMg^@S{InRz#1nvJ$Qol(Q)cABnq+&8^{U
zpc5;${|mMpkY!r92+J?``z7eExqLp>IX)u5mNd3ac;Xp7{I8!^S5AsxN3I1pVM*6H
z_JcfW%NiUq6`Nz;Ue~V$bN1P%hF>D`%e6s<f~i9IYHV<>t7ksv`fo>9{r9w}H~7T^
zebkwoP*!@T<Yv1#_DF!Cw&F<0*Em<Y9Fd7;c49m!D=4m~u2-=X@0KW#L!Pd!G0}?i
z{<GK!_^A(n0d~wv18dzV2qz9oF^;tfgFB`muzCF=KYw#lDh~)uDN82n+{2WSB9;iU
zWW9+`IwX8`u=U#dXjOg0qMgjS*7*XZV7rholD<MbuDGw6t?qS}{5CSM`VNJ7;VgWX
z*)@u>mMQ9%Twd=4tBc5U!Xb?-aB4!bH=R4@u(5GO@^rCdnHnf5qomn+&JkXlNf+cY
zlT@KJ`la|)q;Z5JIBJ#@Zd5GFyxSJ{ZZnz4#UD8;1`=tU;S7O;lr>M4E^@s$RkPP`
zt#B+c{ub<E{6PxF0v&mR+F4;k$rtC0HYqWUHGf!q??u)b1#jZ=gG;!Lf-0=u?m}$7
z?F*cD7J29PS1UyF+YsgA@9r0u5pwz4>qe>JYR8=-ju-Qi8J|p_O_XdVW+1Ue26}=-
z?ci?C$otdW(pS-G!{Se~8dmLW6=NSPb~jen$Vd}4C3WD%zCaws&!z>_{dX;r_|<qs
zm8zNYUS^eKzJWxRQ^*byN^zX}QsN#6!BJnq|Mj{?qBB=Pq>{l!e7r}J@7Y{sfiHv+
zRpk==A?WX9TC8VW7?XkdZt!o<<XcXbY#lk`P;z87+?`BJ)kfI$QgNDvU1^YkC8*Zv
z@gE=fEZp%8W)-`PX;2^Drqi=H#hYC&`8cgTDSQl9x0|5SH!-bWb>c0sZEBygS)=7x
zk(CD&J6-9qEY=znZ<*`={bmvd{w^l&>**Or`u;^i(j}f89)1uxI?DaDh`P{WyvH{j
z!$T_c9RxD{otE^y>amA+vLt)@G%(N*a+zSUZC~X64Uc9#dlG}BU!Wx8)eGIWex_pU
z*;bk5a}Gf%1RHBr=T8hI!a?Mb3I`5_ur3qG#sHh&kEJW-ApsL*hLftM0V$*}Z^5&Q
ze{z5q!%Lm`UXirC&zjJUt<yu;XCg42D_S8}Mvp&%r&+U()czhhh@QnFkC35#hAS|I
zRqbz<MmVS>pg>lm`H6#nuBO`P2@93L8NjnfsMcIAQP~?rj6@ZSgOnwgQl<s%f8!%0
zsaZP-#Y6w3=Q4nj5xvg6o77Yjm@kf1C<ih~?dLVbtfpk%+iN%e=snF&qn+9(juL$T
z?_XHTJ?r{;o19h-jz30o6tymtm_!WWwxRXwRiqN-pQkDnlw!e`C47m3Brtd$=G5!c
z#RETv##!tzKqI9ZlGKC)4rHcUyO3pvCGxMUdBvMmdVVPB{5So%#dA;3lGUdoTbA-!
z!Mryyc0>_+nuV+2o~RV~wE8oi3@dl`>BVMP!HzkfgCqB$@npe%OXyQ-`7}GE5GG;}
zAKv^pPq}C+85d7;6Wk1drcn?Y6gor*-9R?g@vJclHX+u&|IIkV5FuU0NJn2jD5e?@
z*dGYQk*JTCGias)bjd1vGK*mM7`lxbmPmWZXzU5lfq#J@L7~qAQsy)e{_S8Jh4F+L
zPC?OWm4=cedmcHMW9$L=|MxVV2uKQz24;W8r6A%hWRR)Qfk+aMq)u)I*;suKP;ARx
zR5YmRAy6@`{8M?x=Dmd0dC57)v&oO}mPvEt4gR}DNll;*u$%<kOd{n}<C04|T?S(c
zUM`0`*Go;P{C}V084U{fB-Q_v6jH^!`2Qa){t62egf14BVYZ|tH6;MJCKHrw3num}
z6_mubn~9diwtc<1>-IAH)>1sR320UM<hZI{#VKv*LmrB~Lw>J9@3I^9&=QyF9}KJ+
zz#B_Cpj`Wvj#2J!LUVj2%P^NbHp4y}JPA2vsa4_I1<Cg<bMJ6CFR9Hi4Su}^q6&EC
zlo0f%mdc|<gB|*EL~<78hMqY8C5pH7xMMl@AmHDmPbt+xI(-y!THhyFX00hux11&e
z-|_o8Ke8WN-(~o(xoB+fHVrp?n~g=^*Gac7URe^sl`9l5(ecTpe?BEK|4<_G)2TSR
zm1xyI2)!NVS*i$@3vZOqoa|j@C;EP|vCQa!zTj?||791a<cGx}Uu(*HI}z(CZKHri
zvnAa$u7jd8$P=Y$JosKpzT57#hzP;~IEkyFsy7;ruh8avQ#lfHpPDjWSM3@6uF-z`
zA?)P%sh0J<cJ_>oz&;+k@dJLV528cDK6OL+0q=G=e`EHmiR=6t#TC*YG6Ie{vSn0m
zR6z_*3L6NF4-!JcNKj2a4(#bK-%t6Df^NSqKFz32!_1GOE%`J)ZLP0lqj)S=9S<0-
zj*xnvV<=o21UMZpF+Tlk6uh4*Xp}NxY&k_d5L~tyj^0}Ko%HB3N#l|q3veUXKGg)}
zP?6m{kfJ!f(_Z17A5G{^dSgA^LY;2~WoIZn2X?iSIwmm$)$5NhKIa0e_cD)D@QJn?
ziHVL-UtEL_DXtmdiA8x9EybS;+<xokN4YFZk6ykA*Vjb1@l^#?o<>O98B*)$3|PC*
z+tLPzo}`GlXEk~5#5X5=AQic5&iRXzue+9naeLGF#Acl#Di2xhL?EaR7=HK1X6lEI
zxBuBlK|l^${hdC394=ZY%$WjJD!<jQnTGu;ttn{@;>Jt_IzybqH(Oa*UORk0BqMg2
zV%ctaQGVXnp47$LdCC#!&2t;C%8raijY?FHd{s}p6khV4H2$>bXG%l?`Tg!{Gc`ia
zFK&Rnyg`uS@X8J=S19Cjq?#lEXOgMVRs9|ouMUu6_8P4dsYR|{pDGS%eJU-sY5G9Q
zQ*oC$T|s@D=Ul)1^WSf;4FtDi^%)y^<3qIfW128IoC*NF7+1-se9^byv)#+U64(~W
zv^3#1YKl+P9FnM1o<vIyDHJ<aWSw^$b?S9hD+e$mAIIDU&t<X0KB?cWBR_=CI8d`Q
z*1#KM)*EQK-N#XRG>sX&weD+l=D8?tcQLZhsi956Gy{M~z6lfoZP}ZZr=6bfZPi)+
zOQ1V&-Zvo+Q;kO3Y(3J$v#cy6Ma&c-9yLhybBY0Q{4|1{<{0d{1e)FoPZgj(j}})o
zrY#@ANL}*hMAdh<t0libhe#-u$D=vLtGuEY=BW{9{K{!(V~0^^iUr1Kk>y&P4W)<<
z4fxq<D2mC0MLf$2f-T!^&a@RTAh=hNu~5yBwMdAtvgaW8dM@;f@~frTZ*A}vrzOHs
z)5!f%;``b#!0?z;mPcSJ%wv*_OIuXQHGCvtGZ;DuQwsP60I4~1=SC#gjXA3~h_Nt%
z9n|5r{dx2B<*z1I^7jf{7Wu-p`k5D9$F7@YrAdjrzQ1f-!X7Yl7tD<=6b%PFWvAY}
zZMZ+n)Ym>2KLJNEFZwG=3<HFa-!aPMl%|9U1T4%TVEGbge{FKhrgEeCyhYQ)ZNI6$
z8-u<Uh{>*-jnBi;S}mXMDEbZgxi|HGO`;aR4IIgr+DI;%ogonH{Kg{*pJAh`{og3~
zT|shEMick9cPVEh`6evYTcNN}sG5XoE8EooE2ONk^dsP~zk3VQ61m3=H`xE>cUr#p
zVUEWv#~U)>53WnjZ0uXy%jI5aYHTj)c^&G>3Dt%Bij={@l4+KvdO1)qijz$P84f_l
z$${%KX8DSac-&f-5G4c43R4>nL;0?~-ZnzuYx`+O4p_**l-^}->%{_PvgC)CUvac<
z2o=4}Pt@z%Z%hB_{Y&GYC$~O+v;#KW&^)JRnAvKh(W)JdNZ4uH>zxYCR}$)({E@Mz
ze{Btg;x3zZRFWI=Q@D8evRwHtC`$0E2W@&3<!8F?jW<2QTm;FKhquiKqb>U#X#MmB
zQ}McDRXe4>m5qu|BsDjhcDD}Wfkci#%TrZEZw0j#Ox=}L?9S7EW_18o;^tEM(yNyN
zca0W91k1joK`4uhIb-bs#ve+I{CR6_kQ;GiXEFAb-{05oS5=PMlXe19|LsDdd!^pj
z%ttz+&m1*sz*!g<8Fgr&_Z_;<_&%LgE&s5eHu+tOZI#1%Wgh6MbxRN}c-1nwvn%pp
z=X_E5z5k(zegEn_qw$|U1kFwawwp<^0KYSd3KKdqO(x0{$OQ~owdB&rMPE9;#2y9~
z9p}dj9$ZXT!4nQXMpg))&uJT7Qam1>wnoGa<%_yn5V^XFs&=H`rv}=$oZJ)IXs8-2
zvfs};eu10ui0QA^q2rs`JNz2p{lM|Q`>2~beDJI%h+D*;r{<|@rSO@Z1Z1b=h3Y*8
zbC#Q?Jr)HXy4@F!y`r{?+MIw!81Cc2#<{0q+oaUUByNXO6rEe@|F|%>5nX&0g|e!1
zR{gD%hvgV*N=Z9$Ic?9{)FyKlx6yZ79s2nFnm4ng9Zx1Fo7s)>25-S%m|U$o)gahU
z2RnM3Mraj5MUXx8dN?dF<g(JX9G4GE&M}>93Gj%un|-eLxog<j-3)8*n~IlhZ=DA^
zo(C=SfHK4)Vwu=>?IzA7?D#0(FZ!&|w?E$ab11Pf!RMeHXZpJ1jTyRDTo-j<ojp*)
zIPkYM@pDZ1mC>Yp)RD(-!2d({?K*4mqIX~{q@PJQz^nY`)!_GXYbR9(7J5j69F)uH
zfmuDazXkR%hDXW#r2DPBlv10A`RUZdSpV8KeSWFwyUwflkR{KJc7N*4sDCyKCRbG1
zceUROZYepirkiwH0VB=@Q1wxcv=dzS$;*AEopXM8`5&ppK4Cb(@YU_RGat-UNjLXN
z_XZVFTTKHsH535EQkH*{XAvJW-y%+r*E2pOMo8S~S9&?tUW4+n$qxDbkzZoFmSKo#
ztVL?9AbWt#Zu8BpfN2_K<esASqtZHF0li>Mz9GEjsq(zV=%4T?<CJyMDOO52Rhz8r
z22ayiAQ{&tC`7SrL&aW?0f?>@p=~;U&#wLe*(oTG4l@HaV6Ya)>~L5OlP&GNG_IL_
zU%Ot=2zmQd(;y_j!I)tX5cx7P5?)eMt@kQQiTx}b-rm<turq-KcJ139@8jK^pUquW
zvzKmHF9W?WIqft%8)@(SZBJ7OZ5vQ+X+EQ1m%@g+{mO*lrClp3yPy=5K8Da6(ZS~k
zXi}MzLlR$4Dmn}kO*B8WI4}Bg65xQDgOJZ#E?)Ndwcp$_c2zvHw+3zj>+JT*+Qaui
z1jv1{$&wV<Ua7zQ31{{RxVxZy%5)zEI9>V#z*Gi=bRmo7#B$C_i-8_lpPa9}C9SKG
zq|rtmIv(S)BPM!B4{VCQE4KHC(sr!vZ}PmK`gZC1i}n<#fO=PfeQ(t4{2?DUpQnv&
ztIRNAh()UO_S$&aD~i{u7IR+(ey%P9?F@q@VWuuy=HW^?jrOO%w(BiRVSmsl$N5Ae
zY?`x)5M6>0GI$*8KDc0}v^Jz!=G8d3SD64~)XJ6)3O%1|!Bwr{RYNRObE=2FN3trd
z-D^wQLYd@=5)70WD5t4jtKv7m#|UV)2Xa~Kh;E#1>{&-Ye?nB<9uE56$lqoyowE)!
z52?EzbSuH`voZZS-(R{+<z5C1VpMrCY%MPDUsZNi7m>dWq|hW(pEU^bfAa#Vwz5bQ
z69G*R66=Kw2R@7lec*`@(<4wJ%CRx8XV10|qV9T$zkQwmCh|hd2XG0O{U+~wB2F@6
zrCV6afi7U>!N?+2gf@(d1~67d^qikd<8HAA@JhYdwObW`gfoJBe7RUkzFfV2^Ly3h
zJU@OxevZq<<T5v0{AC}zUlmXw62%jsUyoPI#dLQ8sM<tV%5-dGd{VUeYAB3GdtZd~
zM2siQqMYl`=@$G%3?{wuG+`}f<=TV1id7SHbi2|{JkVC6_>LsBJZl-ckS%v_M?DX!
zhl3e(^l|n1``FCy4?HUFqf&ipXhq82;fE0a%}$7#LVdm~yjM-{#q%mm>R!+C-;=|U
zHpek&=;OSIbmg`nP!p=b!#DPJ?VS~S)4^WC{iw&I3bRtK`VjE`&g^CE_Dcmv&CFtF
z>x*A_lhYd?dkK@}(u~`5&ci2_eVCV*-2Sd9nOWWo`zytIK_0kC(~I`CPpofZXO6#}
zvVO_6oUmF9T&5d6;5BvGcCKw6G2{}x;Or0e^hXL2A^iH}x8@t^meOeP#OG)*WF)}q
zESFQXz1-w;^he11iQAN;g?=D+k&uS44FCBCR-T`~8MW0C?X!{f`g4iuR5E<sP9xbq
zh;yvuP@advL4ZH++;zh&ap&g9(=fx@+EiDT0Yrn!F)?Cn?$o0g1wyMb7GM4;aCQmW
zNY;#DE$_J;c;F`4<F(yuNBV|qFeUdSZe7VAn>F|CN@S&+i^4)BGw)#B|2R#gm?izT
z)1k-_29I!VJg@4N`~9O_ySU4H>YOM!8hzenrma4ffJ@y*S50!q%Vtrm>EC8osXiMC
zW_$m+7G{?MR6<Z)K6Ejb^D|4d)p~@jTOUvBFbWyJrjI!j<k!(=eo>;2@8Uh6bAOpu
z)Sz5`?;Oh5<aX-}1X}`Z&K`iLzUL&A$FMe66%*jvI6la^!ys;Xy4@2c)iJ`=5+0+@
zpFUb_9?@-_;==t>F6P7Xc&wKizik+2R=(}3T8T#g#p&pwUZgJjy5CpPnlG}P_E|-Q
zqP*L8oGbNTvZHmIjVIw>e(`M$Q3gsb$;*6LP9?>ii9F3LDD=Ch+7-|sv3Z=<{+(Z>
zS1vPZYp%-dCU?YPVX|FaTWr^;W?6|!Puj3<e|Ds+F!MnNSD(XpzAWVpKc>jVC(qJ8
z!W(@J`NbOJkmaWZlSXaQG)$-_?jA$ABi)-Dn_iT)I)8X0WnQ%8KW~+A5|7<7bm-dC
zJ<~{D`10IX12wR4y(g|5r=9J&5?SM8O!e3fIqJR{nB53SHuK-}&yQ2F_&wbZ{%$Y|
zWC@z2@xfL_GuVw-_+8?gK!DOaa_=C70m%DJH$Gsx+q%-%zS7Z9&z}21##ysAM}kq7
zq+=(3GilW5H(4`%oQMe=df%e|%;k}o&{p}dADJAn4nt0qZMMw@>qD5^E3TL{N!963
z-jTLjdE1A15!+k`{c>c<`*r@+ZK?y^g?iSe;;&q%+__v{e1EKQpsf(?W5H@WtUFJ*
z_Bb@N-fgVE&^M;pSZSg-NMxW)+q_nYY%;;t`?2BH$#<s!>=H!`-LnQ4&K0wL7`la_
z9s`d6;ccnQBKJ0*J^hC{NrIu)`kFV45Uv&-qx5YpES%#T-13!8IidOu0SEe<^OTR%
zDrBmP4UYG&%M{a5wjcM(EF9+S#0Ps9{~YnUKxQ{%%c4Txw2R1#e4i*#hsRT`#8^}G
zw*FJ&*QZK26@f}o?%4LU3hs&iGh0?cOLmvQFCj)oiYrR?Fm5jUZQpL6GD=#kW%se=
z#_64~8KuDkt#!G8tN@I^Wk9P*kp0Be9N0Nuq;KuApga<ko-8GNbLiec{j)>+o=d?!
zZ=Da%t?)#$=-rgO##J;RYd`8KeJstbf#iBdxb{Z#7<MTOA!3U8Ot}rN5mznHq*CUE
zqY{w(IMc5fT_kwd{an!oSC^GtN!IY2Hr_bh2ZOyH=#9}&NouS{5Q{lSGu-9iB){=k
zDly-rv8a(z4gArT^#Q)^(Q<9|(sitb?dl{(PRD<Xi@nW3MpnbwKF@&&$)}9}>=&cQ
zy1?!Jif@a&ch)nM<tNE?6QOw*NL5Za{hP`|+jH|l8<WA{a6s2Qk@qBUfNV0=jfBsl
zO^1^&6$CS0wDF6URsRGX2%Iq}J~ltvic^K}b@*JZT>r3h%=G@&0DF)GguC8}H)+E^
zxT<2?H(L-%4bA$ErG1@(u?dGPKiv<BUMgJh`l@(mRexr!3ui{{nWvqG0}u?jdcUk(
z3m_9I*-wJjF5RkFR$^?R^SNvTL?(Dx3(1+mHL!?<7+?_hU5*C5OP=!S2r`j!G)W>i
z5f=L*cNh!)DI_3?g%U$vE?;%P#q?nRV7f)3DRTXLxRX(?t8<1XO)I%HC37MO!~O|K
z<*OkTQj~ufdpjSk#{HVsj7P=FRHqZ%*y{>`57gwlz@FaAO$Zbits~=gxh|8g2nI}g
z;2rWbJ6YjEJxI9T#~UQbHDI+A;b<vgyhv%>MC0?l12|OQvNxR$T^!ytNBzG5%q8o^
zha;OWqZ@$frf5x~syO#_-fEZDXjLd-yufU*?Uw3RfjvFw`Zo_{_mmbAcy#F$_I8~V
z+8#y!tNHgI7y#Y2#7hT*fo0x?qo$c*O}UO2ixe}5^jb%#ztYe`IJIm$dGwk727HRL
z%3GZ`8qR*xrDuCxpDVys#dEuBLvFc~Vay&_&Xf7?5k?CaJTvrP`vj9eQ<caqIimcJ
znF3Mj9zT3L+Z9Dl(yYe;`nK^asD?p`zF>g^?wGNzx3uE?(}&)U?dkBnZdXR2jsAKK
z3Gycfkx(;z>@#_oydK&Pz<Ra3E1i`Y<}4D{lDfML%O9Viy6k*iTs3ZkNsC{9)LXUw
z?bP4we0*PmWG^&lb<ifVUG>^e#}2q1=a)qw0x(f%SL{P9%?@IJ;30eR=@~T~s*i0g
zd-*<n0~G}n49M*tAH930<pSVETEHav)uVm+je-;C*&J+V+3?)qeUOirP1n`(@77n-
z2<BOub02cL#J;GjRnnv>&#}x*bJbWJ75URG<U5hw*ZpcwDmbcq!NaF%%wqrAbg+tB
zP3Sxb)7h&NZYx^T(dKgmIb`oeA&@zG9tG?#&6i)-#Gt`f1_#ZnDo>SkQkh<IyrJ;+
z@G;|QcKpHa1|QjTE3iv)hJE)X&h>KM9;PgnMtzC0-_31Wr6c>fe|PlBM=Aj>g_i^1
zjAplXSDG+A2^Y5?^KUcL!?O1ULf+j;8cjRH+oXF*@+0;t&s}{=10|Itm@1N*U`*#_
zJ$cewgg^f=wdCjS>jpsUyIN{N3`Af<Qjj1h1nlY$-|>Cp%Y{^c3|iYuy$1Q}O+Xb=
zv&goYGLqK}crEOe#}&g-(IKYlrv#L*tVsQx*DGs7L#jfHSbeMSzjE_kUB4abDf#ht
zgilfH<Kw;qsgG@$_vDMVM~hKnX1f5KcfL#PF~Gvee6E5+XM_~r35)Ej4}aWGTDIGW
zqTYx)^kzG1cuRFGMT!m&fEZ8^keKt%jfA~uCNe9j31kKsP|<;=PtF_nV#0bs@2nNS
z)Ki}rUu^FVB<sQ#{G?g7<j<(lQNFkRxa-Ll%9SF$n+;Gw!viIC#NUt(cS|cZCkHT_
z<ZfD}$|<s$Pfe3(*UIkEf%4e%%VM5m5o=M^8@(L<f|Gx(<C=m}ny%LN@dmISsm_ok
z-#RdMO$>13d5&>&OYKKkNCQp;`~1iv%R_(jk?ALTH7lrTx5!4D=a;@B`oF-vcvrT9
z7&w6xCUct}3$C<+pWPMKKzi)r8AYXs;D9I;<bfy%s-g2S>Pw%t)+e!xo`I<Uf!6cH
zB$M}i`}_73^4CpFaiE$KIJ5c4#V>LuSpknS8Q{G@3k?;W5@J@sNN6N{@>5t*A-^>4
zFHUNgkJ;f=!0#11%qg>4kvZ(W*BNa`*Edi`jU}CCxj*edDCK|O@tAMgbR<x9a~|DW
zOLFBWi2&6UqR_ebAsx<em~-};m5+!RfcRhj*_d~17jn6@{a_it!4&IsJ9wc7x(qy=
zW`&qP00Ghe%`NP)LhL5x!?NvmUK<huS#bz`xox;rltc5jy{~ites`~33CL)~ApOr&
zf?Wo~U1C4XLL)#%$w*&b9&YvWXVC2HrEyk1i&Y718ZlKaQVV|{`=I3^g8kg9Ug6t6
zVxdXiR9#o|(C3s(59$2j(!!$kLPIrSE?!Y!ldj)W5o)VDq2~e1Q-A=d`YI1fVVywk
z5B@wDGFmXK28KO|uq>~6cHXT~%LPE6B%Sd5xP!Xo=~MlNMQ1sKCHk{Dilk#)l*sZ|
z_iqPbIj*n&nLSI<iUYOT?<%{?1&fW$60Qd`>RXhkFv}ic)#KC2S8hjk70I4Qg~Q2=
zod>aeX$Hm(B!yF|8acSx>KKuowRGh`-B>co)1JLuXg$O6%hSY`50G4bnaQJuNPddB
zcoVl*FjE=Z$lP2(H>RlBc=Y*o71^QB<;2)6o&`Z<B<#7evhTaG`*$!vKtTr?CQ>e9
zh4;N+KF7}fmJyfTpMT5k^UWd|MthAR9EkB&@NM%y&mGZQjHmzQdW^kwx8=9pDEn|K
z)wCbzhWWv8XVIkbsAyPRRTegdBbGtGG}&p9i;gW(IXw!i+tQT_y%N+{0G+YZpBd88
zS;xIoVl|IY!0in0_jujqJF?OqVDqRMQc8ZOg%-}EPjTK6tceQOwfU?3&M|;@4I}^d
z@cZ^H62Fea9fR2hympi;-INjhUzK@=7f>2mQSZ4yo*!^qHcQcJgG`@3^{(kPAw5hr
zjcgepQBA(2w$ky2A2X<N`4d#61jC+i!3wBP(sB#+yux4P=_C6wncKaboX|+Ws^Ja#
z)r<K1RL$t<feN8G`IIBHD%H>d6#qXZZ-!aTwLy&|K1$YEhhVXFJ8>6RdYHDNrZTdS
z$J!4agQg27wCD({lnK84LSK^)S|=lYUkZAam)x$+hf6%ZUQ?>!;WPFq5_w{k!RcVk
zD?5@x+^3!<tEBbLe9o@@@92ZE`92-Gm=3zFJ7WH)c3N&T&>?-<e^7?6g!-Of^GcUb
zwo`B952AHB(?am$&dCf%atzShc|1u6?J_XAE_xNxYMgpr*;TO(0=*ijDur9`TMjSK
z(8g3u)_gj&660|kC~1(m400Dt-jGkI!NWO**&(evrmV&PBDi5Yo{IPj!<<y9d>!eg
z$N#u=vG3RxI5GqJ7bqFD9pI2C6YJmtPAC#{)xzxfU4~;j%?5QkAkk#Ed|;%Ek<fvU
zXS|;n3ax<p`*ep<mTUDrq}WW-2U{Swt^_N|JW?I$w^WOBfzZ?tX*<>%i@1X+EQiQ~
zMGZZ|UQ9vDnU_k0DfvxG0%5u#R;qeJ5^FA61qwZ+5$^8>934ye^tsKmVv$JGlh=AY
zq6N0(w|6C}Q<q6k+Bv$Y*L!hSALmVI$3v1%YJx^@zN;{<rjs!y=%l_6<G@O0{Egh7
zam}ea!pIX5+*f1{gisEh(8AL>?6Th}<`hGtjslMUz8@Q(6dp$#R0|h;k?{xsxBz_&
z2mkQDO4#=<w8ml1s1^+AqC<qcA%qIPIO}rTtF#S}m8!_OSI((fwUoa%IS`;UM!!!7
zFDmr~+#dlks7m1eZ?*E17-*dFcvjV3e<<`}^-q_y?rjsnC`-o)yUulkofo`h5#i{Z
zsf<y%qTc-%?kkJ?9BqjvBMee+R@2{E$89{Fw~J^QR~*8;@|Vqj0Wx?X#W15^lV_7V
zEG~0DIb;~NfwW4aUV{i;PW}BFZ0~J{(nbH<y`n$_YuWiJhu`g8ABIW9)lW@p8)a`i
z*9jVFd83ZL4A&}e{GDvzKIs{9v$nfP0BZuGyx-{SE$Ew~LkJ1EOw-q*(iX1_@ZwZM
z52m)Mze8X6#;37H_UO?J0FLAhtRVcN+B=9qAdT=4Jw?6!{RXi|j-5n2Nhm$4)xq0u
zyjEeWbEuJU^n+|QqEzIp<Z!PUya1l_f<(~Irt7lx(qA=+pYmyz8gG8Q!Nm=3@>(!Y
zS}sBxvZR-XgEK2`&3Tp1KFVi)yS{k8fryt3YrIP;`2I3T`>4`}=^huejP*JF`F>(q
z+TX?T^`~Eyb=Ik}5MvSp<ca)Ys5wHU98CBSgHD_KWQbnu{rX#mS}ZTbS7pZ}f%<PY
z!c9<qN}O8RbT_W8KSCtl#`#S_40xgwKk~>X(1GlD$4j9}MS;uq`YRx>;ngeB`cG4(
zjal)t()b6ZwF5ReKFchnGAekn^!pViVP{$dN)eDP51zGO0&QFzgvL8@C-IR9y$7V=
zZr{X`$Qgvb>&35}ly_0+R34{;w><WnJk|+61=5oy4ZcUahXqy&nZAj$o~u!&#w3~z
zy9TFbWCTSl62^KYnGq%A{^Hi|H^g_*L!&yvHlCNCGS!Moi9*P*8Y!7*JT7jT!;}n=
zl?IE4#WxaXI{tMLpml}A${Em7YYW<nNitAYGj-@RNAHfAElImr+f!JuP||HU^`r#(
zQbL^3sg%bjCICq$F+M(Ch8ic8&4r}C8@7OrfVo~$kh%F+A5GxEnIc2?Sd6*cPza_F
zN1sUKryMk2=5MAO0`w3sWfPNCz>LBq;f>4AW??be;1!RTdGRz5NWv@F0T(;`CRCO$
z#Mc?+Y!&W)AHD>?XEP}&rJw_!7W@{|{%m1kK~#g)umvlXE$vk|H_OvUof-!$+!h8F
z6X8(S>+8Nxo~Q*cKu99$DM<Rxc6&4{J3D&~N6JuQh}nc-pe=ZSVa>?hn50%=fQ1P+
zHA%qcrYKhMZO)x}R0%sOK3=jZ;DRhPx!HlfXR4~I;Fph}ZcW4@(H6IhKzWFzXp?ez
zLp-(+_8$rP@l2PFUN54Mt^&8RekZ*K*ap~nDnz8D{(Dnpes`A^=H}rQ9CFUG>B<_X
z7d)e&8=1}rHCK>(m53dVUg%7}!P!oPSeVv&&7<KeQk`{Aa-i2MywitZDbqXuQqz7j
z$-Pe(R5uh-`aT2=EWMhV+S=+WOn#HgW@H0()Un(ROUDm<E2E=UN0)<`#Ny;9Ke+m7
z!Vk1Ze)$fE86zw>vFd}o-J3^Tc>=wz-IFykHjQjK+N101H}-4h_$eLjQR=W^tfKM^
zpdVahQfH_5%a>8Wk6T+?|7y={_}jp+nJPVA%gN;1N|l&?13j)V8fJ7wmnHdrxOHt1
zt1p4uv!29p>3lPjIRQsh^W&$H?W%nHzqk(;M-D^ql=%TT$WU^}A4%d!&>~|Kj&y#0
zegHnV`QEs=yVu?8>;L>WT=9J2cSFWeX0%Y7Eix{%pUc89)PS@YkMTOB<75_3RWe%-
zye<31%`L**t^p<)gieu!*GO;K*}>}Yx$S9qdh+J5-nkAp6y0xH8P%t|o&J3PvEfrZ
z^xxWPeM<rHr!4}8lW_*6h+iCE{JJSO1Enn~<uA1j&GtZvggD^uGLe%T|M{weghBHC
z#}B|z#YQ%7G&zlxCH}FhmD)q7xIooSvnk)GlBBAlrB6E)Ea74Bmej{`BH3sEA68EL
zgWnp$-H%eHh>T-nJP=Mp(bE{f$7yQ1Z4M@bje2}}IXXNH9~fc9V>*$Z*?JWdpy%Xp
zfshpkzuD?1#b5HT$jJSxg7U{pWoFaw^2zrLb=oj|w-k-XV-(rF^!`8*mkcZm;9Pw(
zY}Hj%9GRZxqNSzf;^JatjOsD%Xt!~$#=|U+lFXm7zvNS#UnVr#X+JD(w4$z3@H+Zq
zMy6LiKN4j~pY5=4d@XMA2iJH1>zYg3i>g<4UqE0RKRY+~9&p3@<0xFWhWP*u1-xl$
zdKyqxD=RA#6B2&u9gi)1v@-nlW$=1amiX(1_Jx1CtM|XrFoGr35d9)yr^?+_zur*n
zbeiDLsJir4ufj&uC$v!8>@2aLs;vm9+BI?)7R}X1=<<UCOeOPVb7^$n=?eK>-k+z}
z>eav3tG1oqU{DX(pR(C3*Nn{(1QZp9b?y6cS)FnWC;1yg?15HIhkLe*>lWNEfzHu5
z{ykc1>3sI_rwcp+S?W?{@~;DrB@_r8Kj}8ODfp}E^n~h`t-OS$hgi^6s)5T6s?vH2
z@~f5gn0~l~0B2vy@*fG9pj40|*5@q3`zkvihDhKsCgi<fMH)wL9W{PR2K>0sx5mej
zZYm_h+1?s?2Dqfeur>uoT~+bGIJH#n2xw>H7w0=B;#DSL3#~*tP@=80IjRzyb<D}n
zS<jR#{pKS*i$X+jG^kl@-GG}o2Rc7HnG<RDqxe7jsCuI_TAixPYNcJoh%MfYILC}-
zd#-q8=#(h!H*^OVc}~F$qO`tC8pQ8lB`RXk#NiDZVo85mK=Cj;UYTZaO%5z=ap7{b
zR8eAp6A4p{(#K17!Rcc{R#0g6y;}WLHd!U&OOTu{O{{|3SsTf+(^Q}C=RFa)UHuhj
zP`@x<cGLfO3^E-%9M%@eQDKv)#*~kby7CfH8PACP<CFX-^vesNR6$EAq&?Jcx|Fcl
z$7+aDY^Ad0kA3wiGdcAlz`*J7*2i&>$zscZx{o1Uo-rwqIX6CN=KY3}Qt0of5X+Mg
zq)H-f{aX6kxMH9`G#4?(>}}>yQDZURwr;i1X-e-}(ML&bu~Ub{TugFM|9IIAH^#8z
z-J{yeh~hJ>mW{X}wUw;CHvv*+?O)aMD^y6=cwsB!T7KTjHjk)hsWl`uJ{P}-_~vdp
zD;l)8lkquU|AQW-Q4cQIB9F0n?_=L}a~G=lCg>!c48E8(J*~BF4ZoM#FU^f|QD9lJ
z%ijA^uK(IjUU!_YFM8i$ZnR8)Q;@bqFt;=gRV`{L=FOt!6{(>6s;wk{A_rj^1rVgD
zi+}<_3V{-**9V^dqV99h?O|rAfxJfF!LP+J0a|4SYR*ndi^>S%cnEm-l(d1GUvHM|
zAL_E|b8J;CDY&RBH-fn;cYa8C7w}PKQ~gFZt(+(#?!i;B<dWTiiU-qwW(-cHkCONp
z{uR_7K4GWFdni9lCul-$xj!fA$SA&0h1L%FamelT#QY$RSx5drHJh6G*5sv9`t*5t
zKV$oLR_!$XjR78An`k3OKLap98@Rsti5NMp4If6Da{uPQhxfzl=CLWDxZlzczN~7p
zWcGa%(8D#vQ;UZw#{iNypCZ*&Hgu=o5FVBSmdgd3XTGszx`&jkT!FoW4MHa}fk<`%
zUJMk7L5K&6LPbt2oDB7aFV*CbB2n|iht}J;t4X-M?a7&AI9`^87K*P5Id19nMf2_s
z!RcF*YF`VI&t!EJ440#Nn|R4c1)3=oLuDfM#4#+s`6gVRkdkeT;l7f#FH8aj6>2E{
zCkz?x|K{ATJzXfW)il*tSRXW-BDS;Mz82FECN63oQFG5IzP3zJv-DFE_t$aa@8e|y
z5Y-}<=1<VUf<>@Gf|x4oV`Aq!->6UG>qHn91o`%0ZRKQ`33!p9kJHUn5~4&vAt}li
z<Nko`%g~WC*Q`*b<lle`2?4hr8rKT!R$8lDpn7IdbqzMWPi5n{_f{D|-|n4nQEmt|
zU8?)vwr+ftxcx+@3IszKT(K?SpdlJc8U%O8=`Y`IGl!2c7Y>UH)0N3sztUzq_lK_(
z-LqP5EOzn8C|S5N`LmJjKdx|?P4}m(PDI~h{ec;GS-mO?K<3=C(25J3l8Qc}K1h$v
zmCObB>_HQ}m%gTnk2~X&e7qb*r6-4og$>lM&}<u;7^f#s&xoX0537<XMg-l~BvY8y
z?}Cjoak@-X&ar^)6H<Hq4b6eGndB(=GdR_Sa%-+pf65MLeeN|hXfwc4;oXQ#J@ry)
zIY6O|cr8)G9&#+9Rm3f^W3poM6l7wm6q>M#$<mVgIgfA;xN0C8mL<|x4LZNNhPi5W
z{XlhEFze>%w5g@xP}VYXlt^ILXBSre`oP6TQWfmu>3Fq%GC|)BiAo-JKF=-9<Zc)E
zzj^;~>vog8yYg{+vrCAP^2^5-g8@%~wi<zPdBQ!ts3mGma%=Ta_!AJxxPb-6$Ka=b
ztuj-VnzW4XTlxA7>gCg<KRvQGG-UT2@`TnsS+2v#{pB<3cAM&8bI_0*r)snWlc_x)
zftL;N(lHsFJyShVEybvRnOfQ+KXu^hjH8ychMy=@?f$I)i6Ci_uJU0jbhUS-2ybSe
zEiE$~5<?ehnc9wF(MO0p;fYUIKu7qpI$j*$CLz9ovm~QabFQ%l4X64u$JtGPdIT^r
zavF~%rA7E(j(sovs6{^&J8X24LOV^xD8`4+W0g-hePxag8s?bB|7JmH$*5bTrU~ll
zdrKP!-~{nf>hF<nU^_X{6&;0CMSLDc0~4+8(1@lyR&pt*qyQ1Ge)7oRqmZ<6ulagd
zVNfAup7bkaRyBx74Bj_-w!5LL>{31sLVV9f!8YRzY`<pN8cP|{12btTaWFZaiUxA>
zto`TEz=0hUTyVL3xJzMfe{On`vvt(=0NNJjGlg*S=8A@yAA0pgBmSE<XL%-8I@@<R
z&vT|~*c>oGzmNjUAZJ_si-y{J>psMk)h~tEbvvWkTRlZhDOUv9>ZHiI`NO*s#^B~Y
zCbukqoJy6r%ejwD6u7EI{{O&rNG`mYoWqk+&<6H7LtKah#O`{PNVA$h;8(~}GPgbX
z-#toDoGT0heM?u=t3XWkSFypUCa0{8<}AM>YxD6{EvLhJ2{pw7f3Z|w=ezd0)!Vnf
zS`a4`1LdFK#3{HxR=?ENeLRM<m@xu=T<G^yF`Aar4z#n+e;A=^eUVrn)X%ObbuQX`
zF(q33m0FdKKo`4o;km4CCublhdUh`SrB*k!?jBuHAUdlMK9)kN5GE3}Ea7^glc9FY
z7s%U~O8X6O5Z5jVG^j~I=p4|NR5N~bajSXRYD2Uo88xQv{siZQv##jPI>vkdj$HjL
zT^Q&6F<-o#I)*N&t=w#Tx!rat^ScI6Oz<ET#E~&m0Z(!4Jga|AC($QbHiLAmZkiWo
z*dA46F5i<L$ybEmH+}p;xG)f$A0yHEsq!wzHz=xZyMG=Uzfo9xoLI|JH6=AlQ~4d|
z7Yh|%#kT8}FrqE`M*p$s*FwNDGT%`ecKpn7*df*?{PTX@5PvL>$<V8}fy@8G>sMTz
zhy<R|xe{UAyr8nZ4pZ!mV0V!ys=<B}d8LnU4Z5BIvK^gBtSY8y&-bC~0`XU<c=)Fl
zzUw>dw2C(4Yj*}=u{V7sDq9c=S!gF>J7$Onzhj2;DMq5XHli{%&&i7Y@^HewgN_55
zp?Dw`a|kCnIEO=QkqA}M!{96)Ye<v50`vl!?k`#&=4jM;@+kro^XE20RY@3iO={Li
za$)u6$D@9B%>wbR-#VQFVXPZZdHUg?^Hc)6)L7?qoj|{cw$^KAGT)6^N_05`;fV0W
z!?Pj4Lp<P?uU7o(*=;1zu@ZrU3Loi0W3Lw9&t0|?Epe;%<}Y@}4;2P|*-&z7dI<5^
zzqt9pjw<}1EV!J*2kBt7BxTrO7Y31S#BRC*^Lm{<i`1{zu~#~!bYgrKz|t85>_2lo
z-O%MPg;}iEo_auv1P38&oOmJVRnQB_*m%fgk$Zk<+h5H>RTbbk-**hxJP?%mjHn4t
z;7Mrs>l8_j5st29^y2@e#pf9<h{2{og3^Er)d|t^{L@i-XN4`Pt?B*l7S3$?Pde_u
zBdeuiFS#=n7vPm00zLLqbzJ7D?!vjc_(Cc~rN`p{K~}C8bRG!QfKHRMdE#&Pzvq;h
zi0W64bCeHEQCLU){Ej_kpE}d6-${8ruQW~yz&!tXnOr`q#c5uaZQLQ<!Ftxrn}n#%
zvmFdiyd|RQRLsXGCUTD|uX6VuI4@^uGD;wI!~uSYUPxWu=ilhj;^MpS{PI}i8$D;L
zW7cdOkEG)}QB}Zru9JmUNz84Xgs8}_Yc9aN`6XX~rSrUA^bi|zbk#uPKRYx_m%}J#
z(<d7LHcYWO&1V*Cx!X(BEe9|kb&6tr(~mK<+lm%1hl_`ySm_`Q@!ww>xDV(*hFYd-
z^x2iDHBgRmE-~Dl&#{_~tU>tkZ(y3}KU$|{Bh{p|7iD=2A`f<0NmHD{f|=>p*C?Ug
z%gJ%L=G!hN&P+c&?B;C^eo`akW7kUMxOV;g*;hiRoE^wKYwb;B-)&3P?t9jdfzQ%^
zyKU2(;kS@OOZPhp>P!;oXeq0rMC|+{+K{?CDj&-W4$LDkk}^D@*ULtflt5lHEk|+N
zlejl4GdI8~yl#UM<JIb#v*z|D4B-Iu6*0q`f^s_co6V)u4<@W1R71xzTer%Bk3w^R
z&l+VYKoiB{0}5ki!2L-|QpT&0n-&o+8d>>yHefV@#E6z?u=pkYsjINKTy;NvlkwlE
z+4VXbaI#F}zr#4f#gd}~EU33XqSQqVHQMnyUf}Q>3+K4db~08PLhUDURbBg~i=N^Q
z%1@=deEM{i$S&BZik35EtumgoUslt#VD;Pwzp^I0qxRex0{r(~z$xXBr~Qo@qV*%c
z&wq^!!`BWR!V?mhB`Jb<SFjnRlq8dr)1J1HkaPrjCGW1^8N{h4`F#1VPft(Sxv;+e
zA)@h9>8wtVR&3A!bR#kp2j6}!2jP3MU~EMH`b-@$YCmMrvEyA?#&v%zG3~xj9pZM9
z(D{JQw?sDSw}-_(+)G!;s({$bAteljI$5WU7BIUP=?kMBb4Rudj-{0{MwBKrQug5;
zUel>wwYcs%`3G7$En(Y*x)TXR8jQwWkcu=H=7UC?GYd6LIH(e77PI0MkpR-`1#DD|
z`@7qn()yOp$T68r$KH|6ElC4eN&G}s&#j#1j*l1*DsL2U;|%tam)Pp7OPPO&2MV=b
zr@epGkFo@(kw1sO?VOvBg#f@>vke^$q^D_NA=w-(%|e0oUbVOR->CER2~ie_1?6>B
z(Z@J|h(<Uo)`HW}=@=wA8Iwpb-shELtQYFFTNSk28|!5bUI-_sQ{w>aR5$bP*NO8m
z_X#dU##Pw<{>2ok+Sz6rx%9vBi{3E+eXwo3LztndgWiGc-IgRNJp7g)Mp_JB%>}>y
zO~|vPWBR$~l|>GuJ{U^Fmx%zwMwOG#FQ~u02aCH{n?3j8vkE<AE2(YvKqk9iIy_Qa
zgJ3(u<bn57&<T3G>tv5yBK`%oCPaB7zIPJUWahpxJQtM*(Xq;FkY&<*JuQeAV<(SY
z5<gA-l0V6BfX}sxZ#_YtKqq#j5BDaH_CGTlQHq)-7~uN%&aMr<*aNm@sn3FDv+FB|
zVPmQZ$($0*r_yp%f!dwyNPNC`w4gxZ@pnA0)gh;B#7NbyBK-~&|6oeC`TW<v8cqgA
zND7q<Jr)k-4<#mk+pM~2darf_3DyKEJ~Xi}Kz5dxiY`B2>Re2Pg^2-TR!neBM5k|H
z?P9$Xz*9mlsi~!w9sdiXx<LJSDz@?T=?u;SngzxGGSt4GT)QjupvarS5d~XH`AG-S
zA9bX%ArJ04LW{oVjJ8w9R~vDRJkqs~v5)Cn9IrGLdGi!^HS!YT(mArt-`rC3=*;l)
zB)#nPU!Ttg$ipw>Vx43!*gppZj2!hN?-eyGUK1tD+IKkAPR$$6P21ms{!mP9_?0^;
z(w91YXm(#TvghJ^D`pF1T4Dg~0E|H5C2_?E+|*s_7UcfQ?XwVU$!by60U|Hq^n3_V
zL@v`j$~u!{4J0kPMMU5)DS>H}|AEU*M7=BR5?=C>8?Q`X703zj)J_E&+LIcs#0`f~
z$_!E09cS}Qa+jrr&g@YRg8oU*4nj;bUv4$BQ|F$X_5z0B`$g;~^3L=UHOfB(W(;1c
zY8j^?s_jr|cZsYU^ok;U)CgRJ)%PtZW6Q>(WLMO-w^3<!4%d?OASxJmg_q0;nxvv-
z_lI=M-)H;J@lmPfTfr)qB5f@>jkzdQS{n%?;o<aQx5TdTmN<3LCpBKc{6Vy<Q-n;Q
z4V(>?H*gSsu*MpJ!S*grrN--DF*SjYm-w%T0XaTt-2upA_tEFuM_0Q0wD<hi5vA?O
z<|8Vxtvi`IyKioQDqMl}DDuz#$NEl|W^=B87q!MC6jzk|ndl@p<fnd|*>z#$AQAl^
z-*MbnnyKzpaSmYuS;s5D`}Y^zwSqFjYA>EyD*0hoY>h;lYr4$Ob&KaoIV~k^vrE69
z&{u2x0o$={W(#{R+m>DgWITLE)Om<$dtAFq#0Aey#Gmkzy29GZ*{iUnuzFzs$Dw|t
zg!u3idY5U@fgZi^E9C{;XupGP>;vDcss8BpGfRvL`OL78+TnG~WzEngij}F32xm)!
zqnr0jpa5Q%^WAHOp!_YH9@g=^H`SIq(5=1)BaQDaPVW{s!pH|f8Sa04d|_#3LL=bD
z|K!_L)<w1U@3-n|rv9xg9(aw1LS&=NDr`HFR)No>{Pr+N9kDU=OV>)oq0d_&VrgmJ
zJvtTcw=#lerTGsMJ<RA~b=)!`ACzu^f&jbbVgN5@3mSAkd&_!l9l`nCW8>hFMjON=
z9ZhG8Koq7^>)M0OMkydN0m}f{N%^x!S10nC+=gMLLSiW~8+vDzd~1=Mu$Z7JzN&cj
z6@X&}M4ugLFNQ@?^BOH2yUxaV&F$H8yX1GIweAN$9vg;i2%V1jy&2^G%>40U=2$Bb
zIY{GqjNtu>IN};o><B|k)V>Z3K^RsbCEHsyE<c50*5})zr>0@(W$bPvNh%U)fMJBI
ztzL90`6mKOcQARCR`hLJxoJNcH%Z-@5fwz9AnKpI+y&56_Vs2bH6Dz0p0V>nUKlz0
z^GKf~rh21M+<>hQ7isP#B6I6>vOuh&i%b%13fF0xM5ox-!5$;bJ{78`mJ+wv`<%56
zu=GKt3Z^eH-f{er1&GUXpHnVeus4bHN|xc;=t{`u>Fdi(H+V66U?l)L|Ek5qcz7!O
zz1P<@l6G5_ab<WoA<TXnCH|`sCclkTp9z+S7WL3Cpegm{x&eip@W)87w&b<3VX380
zLC8Teukn(j#Z+bCaQl7$<1YUJ$J+q&=CBmAN10kcF%dnV7hG^RF4`INRpowOm55>$
zo`9oC1db;8XH~JV50g|r)$aEq9$g|LYSZ^|R|q9&rXr+4<fq`%{xkZG-CY<yi!T&j
z^NI;b%XgC6Mr4;>uSZi_QgOh4fzp%z?l@$Cx2?~@{+<Te(e});-<r*nj}B-KTpmC?
zT~g`UR)Kc{Z9w>Q(>zyHk1$!e<&3-`Hyqd+xs$(o@bLlvJ=o<q_$a|wVyp=g`V1lD
z;hFz?K0%ThIbZa@bCz3*RS2|G|8sJ|#pyqgVJ~NK>F5~SIs0ej*pW%0b+->2uWKAU
zd%_6P@J&<ieM{gJglDZ2WrvheNB#Y#l1U_7oKxc0hmeU@a!HjmMiqHJPK!fvbVZ9Z
z9EKl${@Dlug*B-E<3%RZ@|!sO=tp@Zei43IL=oh^WmfeKnC1}@j>R7F21b2&!$ZZZ
zZ3tT!{a#Hh>2)v>RYZqHPNq2q+A&W7c5+nha+hrgY+kAuiL<s_>zN_xZS?^5aCcm$
z2ce%*^fV2jBMG>APXJBV0iroj*-DHlrQu+!aR^+bRY@e-nOL|SH4c8YJQ~Xl3F(Q^
zk(*VrMT5GnS2`A=iX;s^;g@Cns>^QP#fp~-qBiGjmbQ2mTgpwg^#Avv#F27-Rjz7I
z?sD+B-V6g_s)41SHpRbd$YPU`xI5R4cToK8W*S23;QOG2-TdQ6no&$CvXZj@_B|w_
zYvc9{_0+wIHahJ8&Ic8vYAgR5!FFPNhL*KEPn;<>mZn!ad15byhumr^DiaxE4p9lx
z|0(Ru<DuNU04^pm*2t2C7|cwCCX^-al)*%pY}chy=vJ0)mh9UIV^4_4nq@4>MY=+a
z?b?!Ly{<J%5tFi1gZB)#`);4l`_5m#&*xd@{FZZ`^PK1WzGr#YKBuX5T*67WE5TM-
zzbZx|d+I0B6S|1V_F#|G19?2(Z}(~~M*yjF<ovtON^(!K_NYss&xY;J$q9d>IX=l^
zhZFqtNbv}pW%XQZ84meDrSY6c&NVbP=V~Vr;}!VavRuLWdpA=cm0cgr%IVNt;T`JG
zb2vTCMSrg@1(_;e78^(M!^H75jb(f^jB=n&#Z?WZj&V}6sS+;bF)R9=ooHOWB-$!j
z6^%t)1+cw+bji*2cj>F$`nDClT<r<#`ue2(;zj<>zg|^%5R_z*B9WY&Y{ea7Y}8X+
zP0d(zH{@vRBDTiDM(?N0HH*`{$Lg2Ri8hRS9^&$DdM)yS2_lzDA?iKkaKYa*K_fkh
zbrl_~SQU|^1*=CT&3;!V&d<~A%s9w>yFsPnv#D8ApRwgZ?FJlOKHd%YY7r^YMHQyp
zjp7LjC(E;qq6@RB6OL)}2fhrq1Y1OiXln?_nSUP#<{J8(>ltuaxy*=i;+%7x$_+7f
zf?4J#0%`x=Qa4v}#@F-mKN5M<?La?7hp+stPvt^FSyf}$4w)EhllS@iPTl{J^fPH#
zlk<f3D|#PdJv`0XDKk8;rF;n&`A{-E<9?R0!U#Y%{?bMWE79L1>iO85CmtEc%dpDL
z$zeZ+&u(IDU7ONPGpT<h7^uMa*zNH(w2OQ{!T;g4oxAaw9u3^o^WebEVo8wh(*1>Y
zPM_uj)TSnPTkdGoE|Z`Z<JdtN^VQA%_XezNv2os9#nQ>w48Qj_rFBoz(%)tPOg4mL
z@b+AFS%w~qw2KdZUu#Cix8(YbRXkyL6Fxlbzvjh%BAv+++P0iP8)&<WGJ01z{OOtF
z%SKsLBo+1MlSEyB#?7fjD=y)8R>Sh;-qRDzTps3YCj1ls@KQBZndiCOi<A?dAIp!u
zsj95K<jMp>ZJ9ZU!%&>EDIG)YcC_*&F3XYuC~$4t-+dOa*i3V|C0|r7)$m-c@I#vu
zb_-+?pT*Z!gv59i=DQud<3l}Asq)Z0{oSYZ@FccfeARx7q~kK-xRNmD%IqKcExDlX
zc9<;oSB1dG2Vg_`?VQxdwl60xHigHEubX^2<HWmtE<XZ)n_bs+F!OHqa2e^r&t2A8
zj#v3SVAmwG7YMwVD3L^)&eP!Y*%<Y0fB$%3EEp5&*Upq9REmSFth9$DJ1w-R=Cia~
z8dx%1*)JoVK@&R73q&z20?(ft3@O|k`9GJKj=r3UbE%0H=QDUXY6W4tt%I;TAn4Q&
zLybbkzKs4k#zWu27S^p~EDo}iKAUMFRn%tE(XMm1UF-6G`U|!*`~P_`p;eG@xs3d7
zFNQF4XBlNg)bVi<l?lK5ds2xdA51SlBHj_^pDabO3EAq&v>1~b!67{es}cUZntiJS
zZ7N%$=#)ZO8cud)nw<Z>G!;uoviiCMj^OQii?O5#M^SF}5Lz3sZJ3StIjG_V4Vgi3
z)1jeT8p5oW1GO_Cph$y8HE$#`1RAdUp+XKcrq$8dts8j!zZ<+2Bqb%@m1g#%3o1Lb
zq~=gM{-g<QNfK*mOZm}sTYez*{SW#7U)M33tpmJq=WET_xeqy^$2G`O9k3E7y&dhV
zep1WHexUbd%%NN3=$y!-v9_ahUd&?kj+ZK`1-18UKaTarE@kB13a;jv$(!NvH$B+l
zYXRA+NGLNKrs<Wh5a6~X_P9V14VP5k{8qZ=Fs%}^a?iV@6ca%7eqN_%j9IjTxFkjs
z-o?l{0`<|g>;wYLO&g?-a^p!67j(}<Dkw~$J-8CExSJJa9sv>px?7GQ$bz>&l^hRK
z+6J>~1^RaYjs4i!1~YGOVn#Ux;h2^A;nx!rw?AK%mX^Msp8g_TLtdiNXBp7uHa0fR
z71}X1HQxZ$+WcA!4LltLf#Ch(EO_EzDUor?IvxNUZEMrl%3PeC^_m-aI45amXP0pQ
zKKD~H^2j`ZGH=|tL5~QwY8J&Q8JL?Vi4K?m(hh{^@VyIfmxTbN@XD1h>g_%2?EC>h
z-U)<R>eABE{QUgN!YCJfoS|POSzeF+@EEEWHAe{2*(xh5OUx$eulolF!*y4Hh^xdI
z;$OtU7>XcV6HwdT51eyJO)nPz72s^P<QcoIE}ZlA^*wc}>49FH#PMH#(9_dnj0T7;
zRaiE)6;OeJ%$|v`wjMdH%?SwDz;7#u88KdOyZ5KulBBGqZ5|R)u#pljBr?b9sxk?m
zq2-^Nnu>IN3YcI>w!J1>1W4tR2FU!tmHB(Wp?0qg1JfTRCs$NhNJGt@mg3JXC~))_
z9*Px=>O;9o7kSJL)CH`IRVf0q5vZvE3ZI&uMz~!Z;Ta!=L1r)*zuCCx>r>WzHv%&>
zz0VdKozo)htUvCqp58QB>-F)Z&7C$N*<MsYJPT?E-f@HcfrAGP7&5o@^R(NG;M;~h
z5grOK2DbUp(a~>TWHzB7q?Pl>lk1t))oKj0_4V}@*@%WiL#}u|f;+bqRx6V)&Et|w
zx{P+s25>faiiPo%=F54cJ#!o^ot?d6a(qZE5gz|ldb+y0Q@cH4{35SGeh#ETI5;@i
zUb;b)Zf$L)YXHEORZ<l8yU^qh$jODJSifbiw%(K+vIUEJ(a_M3fL&tBK|Z52yDNa!
z@R=?sY^nXGEnWtX#{<iDTDSIH1X8ue!os4ZrA0zQVybtOEpS!oUKf>$zI4GNL8Koj
zqj>Gm&`>oswH~2|Ru#IMcFXT!D4JHgFa7H7?oNR(S4=I5k;4w9Jguo|dUX5o<7L1K
zXPpG*{LItpYFbqox0C&o>*Gd;`h%b>6Fyc_n>RH)GX=Pu({DHfx_hqa=;+wg`6N-5
zPmu6^eSNkgCm+0=s6Z+Bb2~w~X~1mv{6w=)5D`M$LM&h~m?KI`LG`xH{r$z0ib-8v
zhO>%~F4P-AHWJe|#Ai?^2Af)1cB(3PYPXJ!WzcLJBsb`3r?_Qwb~Zsq5v%Ma7nk$J
zb#GM+1^N47CH_F7i?BF(0{NLD&A?2|&vVzi!|>)#q~%g4*aXGIko!`B`Wt0XYzaHs
zj;X2b_2F7_%j0Ik)klCV+}Y9b!^pm4E~rR4v8<w^c!-gjmXL|qBLtc7pkz>EEG0di
z2TPV?5fB&WE+sq1c=0cMEv}<Vfog=7?g_X8sW?(L9`i#-hPd+z3Iq#C{5wcV$;t9^
za_1S!)=yl7;;2^Br_JbZF4PC`@W}ezZBu3K;u+a5DJi+u&JWrO6Vy`}RsSI__~~@)
zY^sh6VP`NOi=TjP2-o=WFT|iybs3r0P@|w(l!JoQ;-o*xS}UM$_N^@O!3TT;u}`8x
zvS4H=NHLDt(-&W{1!`RXqmmN~?Do#*&qL1~b)rNZ;pb1%M`1CTU<C`@oj^3Gl`NLS
z>E=B{ImQlm#@dY>*0jqKajR1;Q2={q%ETJXQl&R_{hM85V<W{sL?uo$3i&KKD2Qdv
zjelc9W@DqLzdy8FAo5Off0Hpc2-S(+z%aCv70wAjcbS?G?|Zk^Dc4@N#sF&iz=rM^
z(597zfw?Nq8-85mo?*hD%e#UXXq}y0RXA9}N}R5&*J+nm0ma_TLcz^{4QaSO0LeWK
zztf3Bb-u^^1pXWyoqQ7Ey3D%p6$1WSkAgnD8ULE8`oV)-0W-Eyev^%%VCg=8c3+xM
zi_gav@$6Ft0CZgEjGVlDQ~vP4fUwqO&=as(B5@{SB)plq`B5A$>moUh_T|&t$)8yh
znmwL@iD#}vrua`Iu7Pz{P*5oIh8(b)9v-3N>Q@v|Px@}>W3e2{#yux#p8j8!X25z$
z#CgInw!29s>6xXeI=i~Ml2^pE*!kDY0h}E&uJv&~4o`x#(op!Ca_YTY5fl_?tP)l2
z0nHp99`^S3270-)F)OO~7U8k)stdrI*A^y@vX5L5i_wj*Tw>0<ywQCr1*)_bXOGKS
zEl3=OlI3*$Q)VC`EaP#&;6rz0H~aGJo5Fa?A|N;F7iVKav2|)RRm!@`Df`vCvK;~(
zh%z2^kN1gQt(&iI5q4U?g&GuY)qj?k<AxSy=jKd}Y)vId_=#k+mX<`o9y*Z{6O*JF
zL+02v6TSRNe)*UP<i?(!moGt;gD+?vKK$^7v5HG$4tajGM&u7nyguSN!rN7KZ=Qe<
z<G|Y2<+-V;OKYeGy{S(JXGnjKta=nu)fhO?tougmd_5uUG4Zg9JwTtKBO8oGsA&WS
z+!--63oQP@M;A8z#&OECfQJsV>M4mb@Q7gm@i_AqXl;Z1MX;=yfdZh+?3uTI8b#Oj
zet%|r2%A43Xc79aKz4zO(O*jvs_-zK1vD0i#_HS!uNJ~B4BQg9GJBcI1knI^{s-*!
z=824i%GU9+|LH&6@w{xYD^A>;y^<#yGhWm6uf7DoECk*Ez80UDD6e>-&}Bs7h33ni
zoZ7eS)7>GzCnhHxt90i<FIt_}9PG`mt*rex`^8|}yV-IpWiZ$t!EJX@@!vho)DA++
z<S*#gOg~dc?k|5cOu7g(6vlYl4OP^SM1DpI3n^P9-hkV2NJB#V_uEdf2yNe2MoUB}
txzm|FM#w=N&LOuUh@{=$)_=2E*kqAV>Wv<iEYD+sACX|BQ+m>t{13q)j6eVY
diff --git a/doc/guides/nics/img/ice_dcf.svg b/doc/guides/nics/img/ice_dcf.svg
new file mode 100644
index 000000000..58851ca5d
--- /dev/null
+++ b/doc/guides/nics/img/ice_dcf.svg
@@ -0,0 +1,727 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px" width="568px" height="513px" viewBox="0 0 568 513" enable-background="new 0 0 568 513" xml:space="preserve"> <image id="image0" width="568" height="513" x="0" y="0"
+ href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAjgAAAIBCAIAAADLYlLEAAAABGdBTUEAALGPC/xhBQAAACBjSFJN
+AAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAABmJLR0QA/wD/AP+gvaeTAAAA
+CXBIWXMAABJ0AAASdAHeZh94AACAAElEQVR42uzdZ3wbx5kw8NmO3gGCvVeJpHrvkm3JlmvcHTtx
+Sb1znOSScy538d29l7tcitN9SWynOO7dlquK1XulRIm9N5DoHdj+fqBMURQpUWIBSTz/nz9Yi8Vi
+dgnsszPzzAwWCoU0Gg0CAAAAppJwOIzjOIZhuCzLiS4MAAAAMJQsy/0RCk90SQAAAIBhyJ+DGhUA
+AICp6EKNCgIVAACAKUuWZTLRZQAAXDWe5+vr6xNdCgAmVkZGRv//QKACYPqJRqPvvPNOTk4OhmGJ
+LgsA408Uxba2tm9/+9v9bX4QqACYlkiSfOCBBwiCSHRBABh/oVDo6aef7v9/SKYAAAAw1UF6OgAA
+gCkNAhUAAIApDQIVAACAqWigZwoCFQAAgCkNsv4AAGBCRKPR7du3v/XWW7/4xS9SUlIGtv/+97/n
+ef6WW245duzYwYMHN2zYcMsttwy86nK5vv71r6elpf3P//yPVqsdYxk6OjreeuutXbt29fX1EQSx
+Zs2axx9/3G634/j411J+8pOf6PX6m2++OTMzc3yPPE6ByncK9e0d99MG10KTjzI2J7oQIGHCPIrw
+iS4EQAghJIpMDClaO7q37jl0w+bb+jdGI+E33npnw423RAlNU2ffjs92xkWUM2tBSmoaQigSDu/e
+vnvbtu3zFy91hMXo2EYfNNaee/+t1zo6O1ZtvN2WmhYJhzzOXmdURhH5asc1nDp6uKe7s7hsdkFx
+6Uj7LLnuVoqiBKW5LzqmYpMYoodsGdPxBrAeFKgZn0OBMcLpsR8DTF+ChGJCogsBziNSMnNLK+Z+
+9NFHqzbe1r+p6sxZXpTzimYptSZeQhZ7ep/Ls2//gZtuvwsh1N3neuWlv2fl5ssIiwuIHvSnDPi9
+J44cOrR3l9vVZ09LX7Fmw/I1G3iO3bNja1NDnSyJrc2NN95214KlKzQaLUIoHAru2bvvxMmT93zp
+sSUr1uj0Bo5jPS6nQmdiJbytqXH7x1saa8+SFFUxd8GKtddlZud6XM6jB/Y4+/ooijy8f49Cpdp8
++90Lliz/7NMP33/zlZ7ODrPVtn7TzWuv2+Ry9u3Z/kl7a7NCqVh7/U03bL4dJ4gjhw6qNRpGrXXV
+NdSerYrH4m5XX1tLU1Zu/qZb7iiZVSHLciwa2frBe4f27ZQRqpy7YN3GzWkZWR++80Z9TbXBaOrq
+aKucv+juex8YcheDPioAAJgo1hR7+bwFDbXnnH2O/i17P9taWl6Znp1DUhRCqKy80mA0nTl1zOXs
+C4dDdWdPh0PBBUtXXDrnyMHdOz/79EOGYSrmLcQw7JMtb584fEAUxca6mtf+9lyvo7t0dmVKWjpF
+Uv37u519XR1tFqtt2ar1eoMRwzCGUaRlZCkUymAg8NLz/9fV3lpYMisnr/DcmVOfvP+W29kXjYRP
+HTvy9it/i8ViC5et1BtML/zxd85eR0ZWTm5eYWZObln5nIKiEpKieZ7T6g3zFy9LSU1/+9W/19Wc
+FQThzMljdWfPRMJhZ2/P9o+2HN6/W28wFs8qdzp6Xvvbcwghnuc+ef+tI/t3Z+fll5VXnj194vD+
+PR63q+7cmQ/ffq2lsS6vqCQrO/fSywh9VAAAMFFUanVeQZHZbDm4+7Nb734gFPDv27ntoa/8g8Vq
+7d/BaLZk6QwtTfXHD+0vLC3bu3P76g2bZFlub2kafByvx33s8H6SJO784pfTM7PrzlW//coLb7/6
+91mVczEMkyRx3Q2bFy9fpVCqBt4SDgY5jrWnZWh1uiGlqjlzqq256bZ7Hlh7/U2SJG5589WzZ06e
+OXW8sLhUkiWcINZef2NeYXFnW+s9N65uaaxfvmaDx+W02FIWLV81d+ESjuMYBZObX2RLsXe0t/7T
+17905uSxvILCwR/B83xOfsGNt9+lVms+fu+tv/7xN72ObrVa88Kffn/Xg4/cdPtdKrXmhT/9rqWh
+rrh0NkJIbzAuWr7quhtvU6nVl15GCFQAADBRcJyw2dPmLlzy6Qfv3nr3A3U11bIs5xeXqDUXsiTK
+yit5nt3+8fvdne1uV9+Dj31zx8dbhhzH7/OKophfVJqTV4gQysjMySss7q+mIIQysnLKyucMjlIX
+DDcZZGdbS35RSXFpuUarRQgVlc5qrK9pbWooLC5lGEV2bn5eYTFCKDMnlyRJl7OPZeOD304QBMey
+VceP9HR1RiORcCDgdbskSRq8j9Fkys0vsqWkIoRSUtN4jnP2OrJy8s6ePrloxeq3Xv4bjhMNNWdF
+QQwG/AihtKzssoq5w0YpBIEKAAAmlF5vmLNg0btvvNTR1rJ728dLV6y12lJx/EIyg8WWUlBctnPr
+xzu3frT5jrvTMoZPmaMoiiQ/v2NjqL8i1f8vhVJ1aVOhWqulabrP0RMOhTSXZA/SNIMT57t+MAzD
+MNQfaXAcVyiUlz8jZ2/Prq0fHT24j2EULBuPRMLDlJamSXqY/vL+yqLf68ExnCDIvMJis9WGECII
+kvy80fJSEKgAAGACMQplbkFRemb226+8sGvbJ//yXz/X6fVD9ikqmXXTbXe2NjVsuvXOYQ+iVmtY
+Nt7e2ux29hrNFo/b1dvTnV9UepnPtdhS0jKy6s+dPXJgz6KlKzU6Hc9xLmevxZZislh37/i0s601
+MztXksTurg6e41LTMi5/IvFYjGVZhFBbS9PRg/uKSmd//TtPtrc2f/8bD49yEn8cxzOzc+964OGl
+q9dqNLpIOEQQJKNQXPGNEKgAAGBiabW6tdff+Mdf/1Sr1ZfMKr+0jc6aYr/7wUf7/z8cCl56hJTU
+tFnlc48e2vfRu2/mFZU01J7t7mz7wn0PXe5DdfpFy1f1dHV+/O6bwYA/LT0zFAo0N9RtvuPuinkL
+3n/z5UP7dgoCL4rCiSMHdAbjvMVLRWHEhFGNVuv1uquOH7HaUnAcNxhNfp/nyP7dbS3NPM91dbSJ
+onjF68AwzAOPfP3A7h2SLBqNZrfLmVdQnFtYdMU3QqACAICJpVRrlq5a+/oLz3/h/odUGs3AdqPJ
+zPMcRV3URIbjuMlizcjKwS8e63TdTbeSNP3plnfefPmvmTl5N99x97qNm1k2bjJb0zOzCXKYm3lx
+WfkXH/3GB2+/9tYrL7idvQRBLF+9nmW5lNT0b373h6+/+Of/e/onFE0vXb3u5tvvTsvI6u3psthS
+Bo8Fzi0o0huMOEEUz644dfzo9o/f93k9t9/zwOKVa156/v+OHzowZ+HiR//hO8/+5mc8x6Wkpun0
+Bpqm1RptSmq6Tm/oP4hao8nJL1QolDSjuP+Rr7761+deev6PXrczNT3zgUe+np2Xb7GlcBxH0yMO
+rcF8Pp/BYBjr36F3B2p/MzFfATCEcQ4q+kaiCwEmViAQeOaZZ5588slL16Pys8jPJrp8AIwBjSOt
+HHr66aefeOIJgiDwiZhFAwAAABhHEKgAAABMadOgj+pUvfupZ4+fbPAMbNGpqIVl1i9uLFw6O0Wr
+psZwbAAAAFPdNAhUnCC5/HEFRbz/i+v7tzi98bd2tvzgmaNfva3krvX5Zj2T6DKiD/e3P/d+XVmu
+8SffXJTosgAAwIwyDQJVP4YmZueZ+v+fzRDNBuZojfPt3a2z800rKu2JLh3yh7nGzoBODRPCAgDA
+OJs2gWowhibKcoxziiw7jna194YGAlW3K/LJwc5DZ/s8AZYksQ0L0m9clpVuVRMEhhD69WvV/jC3
+dHZKnzf62fGeOCeUZBvuXp9fkKH78EDHruM9Dk/UoKU3Lc1cMy/NYrgwBi3OiU1dga2Hu07UueOs
+YDMq18xPWzMv1WpUEjjW44q+vLXx9R3NXc5IOMrf9s/bsuzqJ+4uz8/QIYTCUe69ve37q3pd/jhF
+4ovKrBsWZRRn6Rl6bNP3AwBA0piWgQohRJG4kiF4QeaF82sVN3YGXt3WVNXoXVaeYtTSTl9sy772
+Uw2ef3loTpZdg+PYiTr3gdO9Na2+2XnG5RUpgTB3tMb11LPHVs9L23+6d9Wc1HkllqpGz58/qBdE
+edPSTL2GRgiFY/z+qt5XtzcrGWLJLJtGRTm9sS372k7UuR67pSQ/U69RkQtLrY2dgR53NCNFc+uq
+bKOWMWhpWZbDUf6Xr1WfrPMsKLHML7X6Q+yxWndTV/D+GwqmQi0QgMEcbv+vXvo00aUYf+k24xP3
+35DoUoAxma6BKhDm2nvDZr3CpGMQQn3e2EcHOnafdFy3KOOhGwstBoUnEKdJ4pevVl+3MN1iUGhU
+FEKo0xlZUGZdvzB9bpElEhcoEn/ymSMtPeFHby6697p8k56pbfV/+b92f3igoyTHMKfQjBBq7Ay8
+ubOlozf85IOVyyvsGhXlcEde2Yb/+YO6zBTNvTqmv4LV5YrsP92bn657eHNxfwnjnPje3ra/flj/
+6M0lD9xQkGXXBCO8YmvTO7tad53omVNk1ighDQRMIYFw7M3tRxNdivE3uyADAtV0Ny3T093++Ns7
+W5s7gyvmpBRl6hFCTV2Bncd7KBK/57p8m1GJY5hZr3hgYwHLCTVt/mj8/LwgBg29eJZtYZlVpSBN
+Oqay0CzLiCSwBzYWmvUMjmGz8ow6NdXYEXD6YgihSFw4UeuuavAsKU9ZMz9Nq6YwDKVZ1fdel6+i
+yU8OdnY7IyMVMs6Jz2+pp0j8jjW5WXYNSeAmHbOiMsWkY1q6g74QjMkEAIBRmTY1Kl+I/b+3zy8i
+7A3Ga1v9i2bZ7tmQl5OmRQj1eWOdzjCOYZ8e6hz8LpaXQlFeEM83D1qNihSTkiYJhBCBYwxN0BSe
+n6E1ai/KG+QEURRlhFAowrU5QnFOLM02KAb1KmXY1Dlp2iPnnO5AfKQCC6J0otaVadd8eKBjz6nz
+lSeXP97ljBRk6OLslefFAgAAgKZRoIrGhX1VDllGVQ0ehyf62K0lX7qxsDjL0J+VwAuSIMmRKLev
+yjH4XZtXZM0pMiuZa8xcEESZFyQCx2hqaNXTpGcicZ7jpZHeK8soEhfCUf7I2b7BqRN56dql5Sla
+FbT7AQDAqEybQJVuVb/6X+tFSf6/t2p+/tLpWFwkCbw/nQ8hhOMYTeKV5fa/PbWaJMatPRPHMBzH
+ZFkWJXnIS3FWpEicwC83uz1F4uX5pj/+YKXNeIX1XQAAAIxkmvVRETj2pZsKNy3L/PBA+3t72hzu
+qCzLCCGditIoKXcw5nBHx/HjlArCpGUQQk5fbHCsisT4jt5wpk0zMC8GjmH4xUuyEDiWlaJpc4QC
+IU66JM4BAAAYpWkWqBBCOjX9/S9WlOYYnnuvbtuRrmCERwgVZOjnF1uc3tiH+zvCMb4/MHC86Auy
+vHDtUcKoZeaXWdNt6r0nHR29YUGUEEJxTtxzqrfLGd6wKCPDdn7Gfo2S0qopXpAGEjcYirhnQ163
+K7LjeHff53GO46VQhI9BBxUAAIza9AtUCKGCDP0/P1hp1DF//qD+4Jk+QZTyM3R3b8jPtmv/+G7N
+Wztb23tDbn/8RJ37mXdqOvvCoihd82fNKTTfsTq3pTv469er69r8bn987ynHr1+rzrJrb1uVnWo5
+vwBaYZa+otDc0Rf+6EBHIMIJoqxUEN+5r2JFZepPXqh6bXtTmyPkDbKn6t1vftZyrNaV6EsIAADT
+xrTpoxpi/YL0h24s/N0b5/78QZ1ZzyyaZVsy2/b0t5a88FHDz186/fB/7SZwbE6R5R/uLLMalWPp
+tTLpmAc2FhRl6//0bu2Kr24JRfmcVO1DNxZ+cVNhtl0zcOTSHMOXbyz65avVj/x4T3aq5oWn1s4v
+sVgMihf/Y81z79e/8mnzv/7heIwVFpRav7ixMCtFnejrBwAA08Y0WDhREKVoXJBl1D9VxIA4J7Kc
+SBCYgib6A4YkySwvcrwkSjKGEEFgDE3QJNHfeRSJCYIoKRmCpoiBI4eiPEXig8fehqK8LMtKhqTI
+80FIlpEoSXFO5AUJyQjHMZrCGYrAL86kEMTz++A4plZQJIEhhGRZZnmJ40VRkvvHbNEUQZM4ftks
+jDGBhROTwEQsnFjX5tj0Dz9P9JmNv9kFGR/85ruJLgW4CpcunDgNalQkgQ872auCJhQXz5iH45iS
+IZUjzKWuVpKXHnnICCqE0KWJ4xiGSALXKK9QLRt2HwzDLi0nAACA0ZuWfVQAAACSBwQqAAAAU9o0
+aPoDAExBKgX931/ZlGLUfud37/X5woNfWlCSeeeaCotefehs+58/OjL4pf945IbsFOPv3t5/sqFr
+yAEzbIZv37VKraCf+vMn7kD0hw+uL86yUeTQZvMTdV2/eG1Xos8eTCoIVACAa0GTxE3LyvLTLD96
+/pMhgWpWrv2L1y+wGNR2s+6tPacD4QtTYubYTTcvn+XyR3yhaKvDO7BdpaDmFqY/uHFBV5+f5QSE
+5DVzC1ZU5D3/weHa9r7BBx/8LpAkIFABAMZTZoqxMj+NpgiXP1yUYV0/v+idPWcGXn1375kls3PW
+zivYdqx+cMixGjTr5hXQJPHJkdpg9ELa4ocHa7YerUv0OYEEgz4qAMB4mluYvrA0s6at7+3dZwxa
+5W0rZw9+ddep5sZOZ7pVX5Bh0QzK0E0xalfPyXcHIq/vrEr0GYApBwIVAGDcMBRZlpOSbtUfre34
+8GBNNM5X5KcVZVgHdghG4kdrO0VJWlyWlZNq6t+oVtKFGZasFGN9h7O62XGNnw1mLmj6AwCMm/x0
+c2VBWiTG1bb1eYKRs629i8uy7l4358d/3z6wz47jDdcvLJ5fnFmcZTvb4kAIZacY1y8oinPCx4dq
+hxxw6ewcBXPhNtXR5z91SRYGmPFmYKCSZRSJ86fqPRolWV5gunT+pEhc6HZGwjE+L02rU9O+MFfT
+4su2a9Kt6v51Q2pafe5APNOmyUxRDzv9Um2b3xuMp5hUGVa14loXuwJg5lkzt2BeUfrJhu7DNe0u
+f3j7sfo1c/PXLyj8+Wu7WO78fM2Hzradbem5fVVFSZZVp1YEI/GsFOOKitz2Pu8bu6qGHPALq8vX
+zisY+OenR+ogUCWhGRioJFlqd4Ru/f7WVItqx+9vspuU2MULcNS2+n7x8pk2R+jpJ5bOKzYfPNN7
++z9v+/fH5j9xz+z+KTCeevb427ta/+HOWU8+NCfTdtG8fDJCsbjw/d8d2nm85+HNJd+9vzw/XZfo
+MwZgStAomVk5doNGVdvWV9vWhxA6fK49FInPyrWX59lPNnQPrHdzpLZzRUX+ioq8g2fbjtZ0ZNqM
+agX9aX2X2x8ZcszvPfMBJFOAGdhHReB4ill147Ks2jbf6UYPy120poYky3Xt/qpGT16ariLfdJnj
+HKru21/lGFjGvp8oynurHAdO98FSHQAMsW5+wbzijBjHc7yYZtGnWfQqBdXS49WomG/fvYbEL9xt
+Pjhwrrqlpyw7pSzHvqgs6551czzB6I7jjYk+AzBFzcAaFUJIrSDvWJvz2vamjw90zi+xDm7j9gXZ
+ujY/QvKqealaNRVjhWGPgGHoVIP7g/3t80qsxVn6ge0sJ/zm9WqWFydsTlkApiUMQ4tLs3NTTSad
+6qmHr3/q4es/346RBL52bh5NkZxw/vHOE4g0dLpWVeYVZloQhrLshlON3Vv2n030SYApamYGKiVD
+rl+QbtAyr25veuKe2VaDYuClM03e43XubLt2RUXKZY6QmaLFMay6ybf9SOdAoGI58dBZ587jPfds
+yN+yvyPRZwnAFGLVa2bl2aMs/6Nfvv3SthMD23NTTS//6ItlufZHNy9+/sPDkRjXv726ydHlCq6d
+U7CoJIsiyVMN3Yk+AzB1zcCmv34MTTx0Y1GMFQ6c7g2GuYHttW3+tp5gSbahJMdwmbdrlOS6Balp
+VtWeU71NncH+jb4Q+4PfH0mzqG9dla1TzcwYD8C1uXv9nLIc++mm7rMtjhjLD/zn8ARf3H6CJPHb
+V5Yr6QtLE3x0uHbPqaYMm74iP7Wuve/1z04l+gzA1DVjAxVF4A/ckE+T+L4zvf7PA1WXM3y22Wsz
+KdcvTL/8aooYhhaUWhaW2erb/R8f6kAIBSPctiPdjV3BH399odWgxDFo/APggkWlWWoFteN4Q1VT
+z+DtwUj8g/3nupz+8jx7uk0/8LuLc3xLj9cbjHGiWN/h7HT6E30GYOqasdUCgsBm5ZpKso2fHux8
+cGNhqlVFEXh1s6+m1ZdhVVcUmK54BKNWUZxlPNvs/fRQ58pKu1ZN/e2j+rnF5k1LMxs6Aok+PwAS
+LBxjH/jPlxQM1e0OIoR+8tJn//fOgVaHN8byg3cTJbm913vnj/5Gk2RLt0eULmQnvbe/+kRDJ0US
+vZ6gIEpDjv/d37+vVyvOtfaNoixghpuxgQohpGCIO9fn/eSFU2eavGV5RotecbrBE4kL80usqRbV
+Fd+O49j8Esva+Wl/erfm2ffqKotMjZ2B//76QoOGgdoUAIIoHavrHPhnfz76sDhBPFbbeen2Pm+o
+zxsa6V0wRQUYMGOb/vptWpJpNSi2HenqcUWbu4PVzV6LQTGnyMxQoxqlq9fQS2bbynKN7+9t+9tH
+DUVZ+o1LM/EZfs0AAGBqmeE33dx07YJSa3WLt7k7eLja2dYbKs0xlo+i3W9AcbZhw8L0SFzocUUf
+u6XEZlQm+pwAACC5zOSmP4SQgiY2Lsncf7r3UHVfZ18YITS3yHxVwcagodcvTP/XL89RMOSmZVmJ
+PiEAAEg6MzxQIYRWzLGnW9Xbj3T1uKMbFqWX519FdapfYab+nx+ck+jzAACAJDXDm/4QQlkpmsWz
+bD3uaCTOV+SbctO0iS4RAACAqzDza1QIoTvW5IaifJwTl5Sn6DX04JdwHLMaldcvzshL1w2M8Kgs
+NJv0jN00YguhXkOvmptammNQMUlxAQEAIIGS4j67rCJl2QgTJjEUsWSW7eNfbRq88UePzLv8AUuy
+DX//97WJPi0AphCjVplq1rsDYacv3L9FraQzrAZJlhs7XTiGleak4DjW2OmKcxcm2LSbtCkmrS8U
+7ejzKxmqMMM6kJHLi2Iwwva4g3GOv4bygJkkKQIVAGCirZ6T/6MvX//8B0f+8N6B/i2zc1N//JVN
+LC9s/ufnaYp49T8e1CqZO3/0t9NNPf3De9VK+qu3LP3Gbcte23nqO799vyDd8s7/fFmnUgQjcYSQ
+KEnVzb2/fWvfsdqOGMSq5Dbz+6gAAFNEqll3/4Z5qabzS7htWlJ607Iyhh60uEEo9qPnPim4938K
+7v2f5z84Up6X+r9fv2lxGWTbJjsIVACASXK6uefGpaV28/mEpoq81DSLzh2IDLvzz1/d9dwHhy16
+tdWoSXTBQYJBoAIATJKPD9XIMrp9VUWO3XTXmsoFJZnd7oDTGx5p/zd2V/V6QzcsKp6Vm3I1nwNm
+GghUAIBJwgninqrm5RU52XbjwrKscJQ9Wd8lyfJI+3e7AjGOp0iCgInLkhv8+QEAk+eZdw/E4vx/
+PnLD2nkFB8+1DTtZ7QCDRkGRo5qWE8xsEKgAAOPg4Nm2blegMNOSlWLs30LgGE0NzSt2eIJHazuy
+7MbqZseuk03hGHeZY25cVGrVq4/UdLQ6vIk+P5BIkJ4OABgHTl+4pccztyjjxqWl7+w5Q5PEqjl5
+Bo1y/+mWIXu+tO3E0dqOTqe/odNVlGkb6YBfWF3x5U0Lnb7wudbeUJRN9PmBRIJANQ7e2tlS3exd
+vzB91ZxUhFCvJ/bKtsY+T+yn/7g40UUDYPJsOXDOrFd/YXXFklnZBIalWvSnm3peu2SN+YZOV0On
+a9gjaJT0fdfNqyhIQwgVZ1pZXnjts6qaNlg7MdlNlUAVY4WGjkB9h7+pKxiNCxSJpxiVpbnGWblG
+i0GR6NJdwf7TvR8e6LCbVf2Byhdi393TVt/mh0AFksrhc22iKJXnpWpUDJLR/urWc6291S0OhJAg
+Sr97e9+51t5o/KK2vrMtjuc/OOzwBhFCfd7QM+8c0KvPT13W2Omq63QeqenwBaOJPjNwOTzPP/OL
+/05Lz9x46xd0esPgl6qOHzm8b/fsOfNWrL1uLB+R+EAly8jtj310sPPAmV4lQ8qyTOAYSeCxuBCM
+8LKM1sxLHd9PbOoKHjzTq1XTt6/OSfTZAzBzROP8nqrmPVXNl74kiNJzWw5fur22va+2/XyFyekP
+//7t/Yk+CXDVBJ7/46/+d+7CpcvXbhgSqKpPnfjrH35z14MPT/tAxQviWztb//RebX667ks3ZS8q
+s1oMClGUu92Rxo4ARY7/qu9nW7y/eOVMVooGAhUAAEx9CQ5Uoii19oR+/Vq1XkN/74GKecUWhiYQ
+QiSB8tJ0eWm6wTtLktzjjrb2BP0hTpJli0GRlaKxGZX9b0EIeQLxunY/RRL56drm7pDDHUEI6dR0
+SbbBalSQBB6JCw0d/qPnnMEI1+eNvb+3TcmQC8usBg1zrMYZ58SiLL3TF+txR2mSKM7Wp1vVkiQH
+I1ynM+JwR2OcSJN4mkWVm6bVqCgcu7og6vBE2x1hb5DlBVGrorLt2nSbWkETCCFBlD452JmTpjXr
+mTZH2Btg7WZlQYbOoGUS+wcCAICES3CgisSFd/e0NXYFf/2dpbNyjQMh51IcLzV0Bt7Z3Xq60eMN
+xDleyrJrZuebrluYPjvfpFKQCKGaVt+P/nQcw7Ev31h0st5d1eiJs4Iko7vX592zIS8jRRMIs1sP
+d72/t93tj4ci/C9fqU4xKTNsap2K/snfqxo7A4/dUtLlitS0+KxG5ZdvKko1q3rc0f2ne/edcjT3
+BGOsSJN4Tqp23YK09QvSzQYFgY82VnW7Im/vbD1U3ecKxGNxQa0g55VYN6/IWjLbRhJ4nBPv/OH2
+O9bkLiyznm70dDsjKypT77s+HwIVAAAkOlDFhA/2tyNZXjMvVaUcsTCSJHf0hf7jueP17f4ff33h
+DYszeVHcfdLx07+frm31f+vuWQvKrP31m0hcqGvz61TU//vagop8s8Md+dGzx3/7+tk0i+q2NTlp
+FvUPHppTkmN46tnjWSmaD5/e2H98UZQRQs3dwb9+WP/te8t/8OCc/gwOX4jdsq/t7x83Li1P+dV3
+lhVl6tod4Wfervmn3x7+768tvH1truHi1a1GEo7yz71Xt+1o151rcx/YWKhSkJ8c7Py/t8/Vtvn+
+8OSKNIu6f7cPD3R4g+w/3V+xrCJFo6QS+6cBAIApIsEDfjlBOtvskxFKs6oH1i28VCDC7TrRs+NY
+9z8/NOfGZVkKhtCq6JtXZN+yMvtkvWvrka5w9MIqAKkW1ZMPzqksMGMYSrOqv3RTsUZFnW70Or3x
+K5bny5uLN6/IHsgzPNvs23q4y6Rjvn3v7NJsA4HjuWm6f3t4rlmveH5LXUdvaOTJXy6y80TPm5+1
+rKi0f2FtbopJqVVR1y/OuH1tTmNnsKrBM7Abz0tPPjhn9bxUiFIAADAgwYFKkuRQlLvibk5v7N3d
+bQxF3LQsc3A8W16ZYjeratv9LT2hgY1aFTV4mUS1giQIzB2Ix1jhih+0dl6adVA2/Ml69+lGb3mB
+Kdt+fr5nDENGLXPTssyaNn99e2A0x0QIHaru8wTjlYVmm0kZY4UYKzA0nmJUcbw4OHwurbAVZOoG
+Fo4DAACAEt70RxJ4doq2rTd0+d0ESQ5GORxHRq1icAaDUatQK8lgiAuFrxztrkGcE0VJ0qqG1m/S
+rRqKxL3BeJwT+7vHLi8U5QRReuTHu7/yPxeirChJmTZYvwAAAK4gwYGKpvDyAlN7X6jLGTaoaZK8
+Qg1vSJ5d/z9lhEbXAjduzn/u1XyqWkH9/PElNyzOGLyRIPBLoyAAAEwjGI6lZWYRl0wfzLLxeCyi
+1Rsys3PH+BEJbvpTKYj1C9MRwnYc64nER2xGI3BMo6AkCfnD7ODw4A+zMVbUqinNxNzuGQrHcfzS
+gvV6ooIoGbUMQ43qAqoVFCdIBI6ZDco0q3rgvxSTcjQVMgAAmLJwnFhz/Y11Z890trdy7IVZGR3d
+ne1tLRZbSmn5nLF+RGLPUKWgblmZXZJtePGjhjNN3jgnDrubRa+4fkkGx0u7jjtESRrYfrzG5fTG
+CjP12faraEOjSFwxun6gWXmmkmx9XZu/231+EVJZRsEI99mxnvw0XX6GTsmMKszML7Hq1fTRGle7
+IzSa/QEAYLogSfKeBx/Nzs1/8dlnThw5EI1GEEKtzY1vvfy35oa6NRs2FhaXjfUjEnuGBI6l29RP
+Plj5kxeqfvyXk4/eWrK8IsVmUsbiQn17oKrRbdAym5dnGbT0dYsyth3p/vXr1ToNtaLCLkjSsRrX
+Rwc6CzP16+anXdV4ozSLujBL39QZPHLOuXiW7TJ7VhQYNyxMf3dP21+21D+wsTDbrulxR17+tKnT
+Gf6n+yuyU7U4jiGEaIqQZTQQZQkcUzGkLKNQlNMoaQxDa+an3rgsa99pxyvbmu7ZkJeXpuMFqa7d
+7/bHV861qxXQ+gcSL8NmfPZHj0zax4VZkcAx5ejaJMZC+/nkgWCC4DieV1j8nX/9f59ueee1F55/
+4blnJFGkSEqj0998xz1rrrtRqVKN8SMS3+5EEfjNK7MpEt990vHxgY4t+9oIHMcwpFaQdrMq264l
+CZwk8IIM3ZMPVr63p+2lT5te/rSR4yW1kiovMK1fkDav2DL6gbcIofx07a0rc174qOH//flkWa7h
+G3eUDST1DWHRK25ema2giXMtvv/68wmEYZIkkwT2nXvLb1+dY9SeH0S1ZLbtdKPn7Z2tGiX1lVtL
+TDrmhsWZ51p83/jp/h88NLckW28zKh+7tdigpRs7A//1l1MkgSkZ0mZUzi+xUAQsCQamBI1Kcd2S
+2ZPzWb1BbleDL8+iXJyjG/vRQMJRFL14+WqT2dLT3RUK+AVRUKnU9rSM7Lx8izVl7MdPfKDCMGTS
+MXeszZ2VZ2rqCjg8UY4TSRJPMSkLM/W5aVqKxBFCSoZcOSfVblZVNXr6vFFJlDNTNGV5xtxU7UA3
+T16a7tv3lssXJzlk2NTf/2Kl3aRKMZ1/sNKp6TXzUzUqqqU7qKAJpYLEcezhzcVr5qWlWS+K/ASB
+F2TojVqmptXX2BmIxAQFQxRm6ucUmk06Bv88Oq6otCsZsrEjkGFTI4QMGvr2NTk6DRWK8BoViWEY
+Qqgs1/ilm6iaVl97bzga45UKMtuumZ1voikCIcRQxE//cXGaRWWEqShAEjjREXy/2rU4R59vUVo0
+0KIwE1A0XVYxt6xi7kQcHPP5fAaDYayH6d2B2t+c9CsDhmOcg4q+kehCgIkVCASeeeaZJ598kiCG
+9rb6WeSf2qsMusP8zz9r/7TWW5qi+vqKjDWFhkSXCEwtNI60cujpp59+4oknCILAcRzanQAAk+pQ
+W6CqOxyMCzV90dPdoSgnjf2YYGaDQAUAmDzuCL+t1tsdYBFCobhwpidc54wkulBgqoNABQCYPPub
+/dU94Qh7PkX2THd4b5M/LkClClwOBCoAwCQJseK2Oq87cmEKaXeEP9MdbnbFEl00MKVBoAIATJK9
+Tf5TXaHIxeP66/oiOxt9ojTJ86CB6QQCFQBgMvCi/H61KxAfOvuMI8id7Aj2hfhrOipIChCoAAAT
+TpLlPU3+Y+3B2HDTpDW74x+dcye6jGDqgkAFAJhwgoReP9kXGyETvTvA7m7yuSdmsR4wAyR+ZgoA
+wMwmSrIzxGUYmFsqrP0TxziCnCPAGlWkVUMjhARJ1iqIk52h60vNiS4smIogUAEAJhaOYzYN9dXl
+6QPTmx3vCB1tD8zN0C7K1iOEZCRjGKZlYG1rMLxxClTGuUiZluhzAQghhCjt2I8Bpi8NhabkDR/P
+1NED/8AkgeW4ylTFgnSY3BIMhWOIi160ZZwCFWNGDNTZAUg8EkdXWig78cpsCquaMChJJbTpgOEM
+6a6ErwkAYLLplYReORXrfWBqgkAFAJhsEU7kRFlF4czUr/2BKSDpviWSJEWisXAkOvZDAQCuTW1f
+7GBrqDsAg3zBqCRdjSocifb0uQRRsFstapVKqYC+XAAmW11frMUTt2jIPDP8AMGVJVegEgTR6fF2
+9Dh4QfQFQikWs91qVqtVBKzLBcAkigoSQWAMAb87MCpJFKhkWfYHQy6vP85yCCFfIBiORP3BUGaa
+Xa/VqJSKRBcQgGQhSrKawhUUBCowKkkUqDied3q8gWBoYAsvCL0ut9PjzbDb8rIzFQxNkSSGYYku
+KQAzXI6RQQgZIPEPjE6yBCpZll1en8vr5fih/beSJHX09Dq9vuz01HS7TaVQ4NASCMBE2lRqTHQR
+wHSSLHdkluP6nJ5QeMRkv3icrW9uO1Fd63R7eV6QYXEcACaMLCP4iYHRS5ZA1dbZ4/EHJOkKK14H
+g6Hj1TWna+sCoWCiiwzAjNXui/cEWRZWoAejkxRNf75A0OXzs9yVFxGQEVIyNI4TsTirUvI0RSW6
+7ACMVkBAMXHsh5kMP9vdW2xX3TjLrIUcJnAxGkemS+67Mz9QiZLU5eiLRKPyldoa8rMzrWajSqEg
+CIIiCYKAnl4wnUgyEqdJe1qQFaO8JE6fAoNJIw33lZj5garX5Xb7/DwvXHFPkiB1Wo2CpkdxVADA
+tYvzEoFjFAEZtmBUZngfFcfx3T190VhclmWSJIjLDjCMxWPQwwvAJDCpKZOKokkIVGBUZniNyu3z
+EySRarNoNWqGprp7nS6Pb6SdvYGgIE6TNn4AprM75lotGgqHMYtgdGZ4oFIqFTkZaSRJqpQKAsdD
+kajXFxBHyP2LxuIsx6uUMvx+AJhQ5WnqRBcBTCczPFAZdRctd6tWKhQME4nFht1ZkqRgKKxVq2lq
+hl8WABIowklxXlTTBEyhBEYpub4oGpXq8nP6BUJhQbhy2gUA4Jq5w/z+5kCXj010QcC0kVyBSqVU
+KhSXC1ShMAQqACaWO8wdaw85glce1whAv+QKVAoFo1Qwl+mCCoYiLMdfccQVAOCa+SK8EqZOB1cj
+ub4rOIYpGJphRhwpJUpSJBaH3D8AJo4gyyoKp2EQFRi15ApUCCG1SqlRqy6zQyAYYllolABgoqTq
+mHyLQgdrfIBRS7r0NqVCoVIqERpxNFU0FuOhRgXAhKnM0FRmaBJdirGSZZkbxfSh0wuO49SUnOA0
+CQMVc/nEP18gyLIsQtpRHxIAkHQ8bs87b76Z6FKMs8ysrE2bb0p0KYaRdE1/BI4zNE3TIz41CKLI
+spwgQKUKgAnR7mVdYY4XYY0PMFpJF6gQQiolo9NcbmC8PxSOsTDIA4AJ8cbJvn2N/sB0WZIETAHJ
+GKgYmtGoLpdPEYvF+UtWrAcAjIsOb1xCiKEg6w+MVjIGKiVDq1XKy+wQDEficahRATAhBFHGMQxm
+1ASjl4yBiiAIBcNQ5IiJJCzHxVlOhNw/ACaATkma1RRDJuPNB1ybZPyuYBjGMLR25NFUsiyHo7E4
+jKYCYAJ8Ya611K4mcahRgdFKxkCFEFLQtFZ7uZEc0VgsDvkUAEyARdk6m3YqDtYBU1bSjaPqR9P0
+5eeniECgAmAC+GOCKMlahoTlfcHoJWmNiiIJtVJxmZXpY7F4NMZKEgz1AGA8NfRFj7YHfVHIqgVX
+IUkDFYZhNEWplSPm/kmyHIuzLAs/JwDGU5snfqoj7IvBYjrgKiRpoEII0TRl0Osus0MkGg1Ho4ku
+JgAzSogVSRwjIJMCXI0kDlQUpVNfbn6KaDwei8cTXUwAZpQoJ9r1tIZO3jsPuAZJmkyBEKJIUqtV
+Yxg20jKJ8TgbjcVlWcZgZCIA46Q8TcOQmJJKrjU+MAyjKJKhaVEQYyw7+J7T3w1B05QoinGWG9wv
+ztA0RZGSKMU5bkh/OYZhNEXSFCUIYoxl+w9+6Z1KlmRO4Dlu2ndhJG+gwjCMJkmFgonFhq82iZLE
+8TwnCMyUnPcegOloTZEh0UVIAAVDL6yctW7Zoqa2znc/3RmJxQZeUjLM0gWVyxfM6XL0fbJzv8Pl
+Hnhp+YI5C+fMdvS5tu071Ot0Dz6gVq2cX162bMGcmvrm97bvnlNatH7FEgVDi5KEBkXBQCh8orr2
+wPGqRF+AsUreQIUQoijKYtB3xkZs3wtHY6FgmDEbE11SAMDMlJlmz0lPJXDcbNDPKikYHKh2Hz5e
+kp+dmmK5NO1Lp9WVFObF4uyJc3UDGzt7erfuOdjr9l7YT5alEVqMppekbikmSUKnu9y6UyzLRuKx
+UR8PAHAFuxv8ngik/F1gMug0apU/GBJE0XbxM7EkSX1uL47hWel23cUTFDA0qVOrnW5vd2/fwEZZ
+lgVRFAYTxZkxxia5AxVBGLSXC1SxOBuJQj4FAOPDGeZfPd7XG4Sh9OcZtBqbxSxKUpejzxsIWs2m
+eeWlg3eoqmkIR6KFOVlmg35go81iqigtkmW51+URk2NZr6QOVDiOMwzFXGERRZbn4QEQgHEQjAsx
+XhJnQlvU+MjJSk+3Wx1O99HT51o7utVKRU5G2uAdOh29Hn9Ar9MOXpdcq1an2iz+YOjYmXOJPoNJ
+ktR9VAghkiBMBr3j4o7KwWIsG4xEBj/OAACuTacnbtVSSpg86XNWk1HBMG6Pr6fXqWIYjuOtZpPZ
+qPf4Av07cBzf63Sn2iwFOVkur8/p9ioVCpvZpFYqm9o7A8HQRUczmzauWc5y52fTDkei9c1tDa0d
+iT7LcZDUNSqEEEEQRv3lghDLcuFwJNHFBGAm0CiIfItSSSdXbvpIstNT01JssTjrC4Z4QQhFox5/
+QK9RzyoqGLzb2fomrz+QkZpiMRoQQiaDLjPdHouzDS3tQw4Yi8fbOnvqmtr6/2tu7/IFQqMuzpSW
+7DUqAscN+svmU3B8OALzUwAwDgqsKi1DGlXJftvpl5OZbjEZ2jq7+1wehFAwHG7t6E5PsRXkZO49
+cmJgN6fH6/L6SvINFpNRqWCMep3NbHR7/W2dPUMOGI5E65pbe12eRJ/Z+Ev2bwyGYWqlkiJJXhi+
+I4oXhBjLiqJIEPAYCMCYGFUQpc5Tq5Q2s0mlUOh12tnF+cX5OQSBW4wGnMCNel1eVkZLR9fAzj29
+zpz0tHS7zeVJs5pNFEk63d7Bg7FmvGRv+sMwjCQJvU4z0uwTsiyzHB+OJNF3AoAJ0uaJc0JSZKld
+UXZ6mtVkcHt9vkBIqVDoNGq1UsnxfEe3Q6lg5pWXDp5moq3L4fb6bGbjrKL8tBSrPxhq7uhM9BlM
+Kni6QTiGWUxGrz8w0lxKLMsFQiG9TnOVBwYAXMCL8psnnQ8tsado6USXJfEKcjLUKuXhk9XHz9YM
+TI5DUWR+ZsbtG9elp9g0KmU4Guu/KXn9gV6XJzXFmpWeyvN8U3tne5cj0WcwqZK9RoUQwnHcpNdh
+aMRMJI7nQ5BPAcDY+GL8tlpfKC4muiCJZ9TrLGZjNMY6Pd7BU7jxvOD0entdbp1WXZCTSeAX7s/d
+fU5/MKxVqyRJdnl8iT6DyQY1KoRhmF6nJQhCHGEItyAI4WhUkmUcZqcF4JrIMurysVYNRRHJ+COS
+JDkUifb0OT0+vyRJ2RmpLMt3dvV6/YEhe8Zi8eq6RpIkUyyWc0QLEs/H9c6e3qa2DoLAe3r7Onp6
+h7wrEo05nG6fPzBSX/t0B4EKIYRIglCplHxQGLb1T5JljuNjcVY9aMwdAGD0JFnu9rFpBppOykDF
+ctyps3Wnzp6fl6/qXH3Vufph94yx3PHq2uPVtUO2x1lu39GT+46evPQtsizXNLXWNLUm+iwnEASq
+82xmYzgSEYTh2yU4QfD6/GqlPdHFBGBawjGsyK7CMKRmIHsWXDUIVAghhGGYQafDcRyh4QMVLwjB
+cDjRxQRgusIwVGxTFdtUiS4ImJYgmeI8k0E3uOtyCJ4XAkEIVAAAkAAQqM6jSFLBMPgIsUqWZY4X
+4iyX6GICMC2JkvzWKWeMh5Q/cC0SH6hisdgTTzyxbNmyrVu3RqNRhFB7e/uTTz656GJr1679r//6
+r9OnT19meZXPPvts0aJFd9xxx/Hjx4e81NLS8vjjj3/ve987ffr0sO/FMMxk1JHEyJUqQbg0RWcS
+uN3uZ5999q677vr4448n/9MBGDtZRsG48JudXWEWRvuCa5H4PipJkhoaGo4fP+71ekVRRAjF4/Gm
+pqbu7u4777zzpptuQghxHNfe3r579+5PPvnkP/7jP1avXs0wzKWH8nq9x44da2ho0Ov1v/3tb7WD
+1pqKxWINDQ0KhSI8cleTUafr6XWhQYt6BIOBIwf3N9bXr1q7ft78Bb5gMC3FOsnXh+f57u7u6upq
+j2cGTuEFkoEky71BnoThHeBaJT5QjUSlUpWVlV1//fUIIVEU/X5/bm7uo48++rOf/Wzu3LlW64gB
+IxQK7d69e+fOnbfccgt2NT8Ng05LXjyhH8dxPV1ddTXnyivnCqI4ZFJ9AMBoiJLsCLKZRgWBQ6gC
+12LqBqrBCIIwm81LliwpKSk5dOhQf52JpoeZiEWtVmdnZ2MY9rvf/W7JkiVWqxXHR9u8qWAYmqbx
+aEwadjSVJEVjLM/zFEWN8oAAAIQQhmFGJbU0X0fDSlSDzC8vc3l8HT0OhJBRryvIyWQYur65LRSO
+LJlXiRA6eLyK4/mB/YvzczJSUzq6HI1tHTazadGc2QMvhcKR7j5nR08vx/FXX5BpYHoEqn4URVVU
+VBw4cKCvr4/n+WEDlUKhmDVrVnFx8S9+8YuXX375G9/4hlKpHOmAgiD09fUdOXKktrY2Ho8zDEMy
+qrTsHJ1OjxPEqRPHjh853NRQ5/d59+/Z1d7WYjKaSvOy33zjNa/X+/jjj1+mVhcOh7dt21ZVVfXN
+b37TZrPhOM6ybFVV1dmzZx0ORzweN5lM5eXlFRUVKSkpA+9644032tra7rjjDpfLdeTIEbfbvXbt
+2rKysiEHl2W5pqZmy5Ytubm569evv0wxAJgKSBzLtyj1SkJBJr5TfOooyMlECHX0OFRKZUFO1qyi
+fIfTJQgiRVGVpUVKBdPd29fW2dM/2YTdZplTVpydnirwQmNbh16nKc7LPnamBsOQgmFyM9OzM1Jp
+imrp6JqROV/TKVAhhEZTPTIajV/4whe2bdv23HPPrVu3btasWcPWgTiOq6+vf+211xoaGlJTUymK
+CgQCbe3tlqbGhYuXpaaliYLIcZwoirIsCwLPsVycjfV5vJ/t3Ll927ZVq1atWLFCoVAghI4fP37i
+xAm73b5s2bL+sBEIBP74xz+ePHny0UcfRQiFw+EtW7bs2rULIaTVajmOczgcNTU1J06cuPfee3Ny
+cvqL9O677+7YsSMajapUqt7eXkmSeH7o85Eoiu3t7b/+9a937NjxxBNPjDSRLgBTB4YhnZLQjfzI
+mMxIgshKsxfkZARCoapzdb5AUKtRI4QEUawoKex1uvsDVUF2ps1iGpxKFgxH9h45gWGYgmGKcrNX
+Lp47u6jA5fFBoEownufPnDlDkqTVaiXJEUtOUVRhYeFXv/rVxx9//OWXX/7ud787uNYyoK+v7803
+33z11VcfffTRr33ta2az2ev1/upXv3r3vfcUSpXeoF+weElOfv5H773DcUfXrL9+5Zq1OI6FItFb
+b7117549x44dq6ioUCgU8Xj8vffee+aZZ+bOnavVatetWyeKosfjOXfu3IIFC/rbHvft2/fLX/4y
+KyvrscceW7FiBUmS/THy9ddfRwh997vfHagdxuPxY8eO3Xfffffcc09aWhpFUQ7HhWmSRVHs7e39
+05/+9PHHH99xxx133323zWZL9J8FgCvgRbnRFSuwKGioUV2MJIj0FFtJQY4so9M1DX1ub/92SZLa
+OrszUlNSLOY4y+l1GrvVLMtyNM4OOYIsy7F4/HRtfVa6PTcjTamYmdO8TY/vjSRJPp/v5MmTtbW1
+S5cuLSkpGTbrb4BKpdq8efPSpUtfe+2148ePxy5ZYUwQhPr6+g8++KCsrOyJJ56wWCwYhpnN5vvu
+u0+lVLa3NAcDw2Siy5IcjcXXrF6jUqmqqqpCoRBCqLu7u6mpied5n8939uxZ9HnWYjAYvPXWW0mS
+jMViL7zwgtPpvP3229esWaPT6VQq1dy5c+++++709PTXX3+9paVl8KeUl5evWrUqOzt7SEVQFEWH
+w/Hqq6++8sort9xyy/e///1hAzAAU02EE5/d3xMXJKj/D5Geal08t9yo051raB68UqIky929zkg8
+XlaUp1Epy4ryGYb2+4McN2Jtyenx0jRFkjNzhqqpW6OKxWKNjY179+5FCHEc19raunXr1tTU1Kee
+ekqv11/+vf1R51//9V8feeSR119/vbCwcMjoq2g02tTU1NHRMW/evJMnL5rnUZKkaDQ67BdCRkiS
+ZbPVmpOTc+bMGZ/PJ8tydXW12+3ur13V1dXF4/FIJHLu3DmdTnf99deTJNnZ2VldXZ2fn19YWKhS
+XZhCxmq1VlZWVlVVHT16tKSkZGB7UVHRpScYi8XOnj3rdDp/9rOfbdiw4d///d/7u74S/VcC4Aok
+Gfmjws563//emgf56UNkp6cihHX29Hm8fgLHhyzgcLauqbK0KDPdnmG3uTw+WZaz01NHOpQgCDO4
+I2DqBiqXy/Xqq69u3bq1/58ajWbDhg3/+q//WllZOZobNEmS69atu/nmm1988cWVK1fOmjVr8Kvx
+eDwYDIZCoa1btx49enTwS6IoFpSkDJupgRCSZdnp9t5ww8bf/e63nZ2dRUVFhw4dUiqVmzZtamho
+aG5ubmpqUigUp0+fLioqys/PRwj5fD5BEAoLC41G4+BDqdVqu90uSVIkctFiV+np6YPjWb++vr5X
+X32VoihRFO+8806DwQBRCkwLnCB1+OIFViWkpl/qaNU5QRAqy4qWzCvfd/SUy3vRQlMnztQU5mav
+X74kHIkeaT9r0GkuE6g0ahVCaKbGqqkbqLKysr73ve997WtfG8tB/vEf/3HXrl3PP//8Y489JooX
+pm/BMAzH8dTU1B/+8IdDPsIXCJ5raB5pEgpZlv3B4OrVq//ylz+fPHnSbrfX1dWlpKSsW7cOIXT4
+8OGDBw+Wl5cfO3bsoYce6n8LjuMYhvE8P6RWJ8tyf5GGDPbSaDSXZn/k5OR897vfzcvL++Y3v/nf
+//3fRUVFZWVll+moA2CKECU5GBezTcxVDWpMEnGWO1ffhOP4nLLiBRVlOw8eYy9uy2lsbdcoFadr
+6x1Ol2GERcZxHGNoOiPVHgiF2Rmanj7Dn8qLi4u/+tWv9vb27tixY3BWglKpNJvNsVispqZmyDOI
+VqOmRx4pJcuy1x9cvny5Vqs9e/bsnj17nE5nYWFhcXHx7NmzU1JStm3bVl9fL0nS2rVr+99itVpp
+mq6urnY6nYMPFQwG29vbSZI0GAyjORelUrl8+fLf/OY3NTU13/72tx0Ox0x9egIzCU3is1PVN822
+EjP8ZnON4hx3rrGlvqW9pCB3zdIFQ1pKDp8884eX3jxadTY6aCHgfhiGkSRJUZTZYLhlw5r0FGtN
+Y2swNDMnJZj5350HHnigoKDgww8/rKurG9io0WhKS0tzc3M//fTTzs7OwfuTBKFQ0Jfpk+QFIRqP
+b9iwoa2t7c033zQajbNnz1apVDk5OYWFhSdOnNi2bZvdbl+zZk3//hkZGStWrGhvb6+pqQkGgwPH
+6e3tPXbsWHp6+oYNG0Z5LkqlctWqVT/60Y+OHDny1FNPdXV1jfKNACQKRWDZJsW6Yj3MoDSSYCh8
+ura+trGlOD9nybzyUb4rLcX6L998+Aff+PLXH7zLbNTvPnz8dG1dOBob5dunl5nfdmQ0Gh9//HG3
+292fkjegqKjoscce+9GPfvTEE09861vfWrlyJUmSR48e3bNnT25hkc5oEZCo1WgtVls0Eqk+XTW7
+otJoMvW/1+X1L1u+fOvWrS0tLStXriwuLkYIZWRklJSU/OlPf+I47rbbbht4MsIw7Fvf+lZ9ff2f
+//xniqJuu+02mqYPHTr0/PPPBwKBp556yvT5YUdDpVI98cQThw4dev/99ysrK++//37IUAdTHIYh
+DEGUGmrL9t3C5/0RvkBw16Hj+46d4gWB54W/vfk+zwvCxevKn6lrrG1q7e8vaO3ofvq5FwdekiSJ
+4/mR1n2dAWZ+oMJx/Prrr9+7d29fX9/g7QaD4Y477sjIyHj11VefeuqpQCAgy3JRUdHSpUuzM7M4
+GcW5AM0wFXPmet3uk8eP/uv3v221pvzn//4cIRQIBhcuXKhWqymKmjNnTlZWFkJIpVKVlpbOmTPH
+5/OtWrVq8Gfl5+f/9re/feedd7Zs2fLss8+Gw2Gj0bh48eKHH3541apVV5UWgWGYVqv93//937vv
+vvvnP/+5Xq+/7bbbhqRpADB1eCP8lmrPnXOtGljb92KxQYOiJElmOW5gqG5kuIoRzwv851NmC6IY
+Ckeu/BkzBebz+UbZRzJBRFGsra31+/3FxcUmk4kgiFgs1traGolEMjIyUlNTR38ol8vV1NRktVoL
+CgqGvNTW1tbb26vVajMzM3U6Xf9GWZZjsVhXV1cgEIjH47Is63Q6q9Wq0+kb2jq6e50IIZ7ng36/
+z+dlWZaiqKKSUoSQUsEsnlPe3NQYjUaLiooGJjEKBoOtra0cx+Xn5w+pJ4mi2NfX53a7w+GwIAhK
+pdJqtVqtVrVaPbBPXV2dx+MpLy8fKCFCqH8aC5fLlZWVNVB5EgThzJkzkUgkLy/PZrPB9IPJJhAI
+PPPMM08++SQxaBplH4+iU+yRWpLkOmf037a0vflY2Qybkdbtcr/z5puJLsU4y8zK2rT5psSWgcGR
+hUahUOjpp59+4oknCILAcTzxNSqCIGbPnj14i1KpvHSCu9Hov/UP+1JOTs7ATEUDMAxTqVRFRUVD
+tsuyrGBogiBEUaQoymy1mi8+LMtygijOmTNnSCKTTqerrKwc6TTT0tLS0tIuU/7Bo6kG0DSdnZ2d
+nZ09eCNJkvPmzRvjlQdgosUEqcsXR7I8w6IUGIxj2W89/MW8ouJH/+GJgVul3+fd9sH7xw8fWrB0
+2S133q1QqsbyETM/meIaYBimVimVI09+IcmyPxjiZ26LMADjIsZJjgBXYh/TTQpMcaIobv/4g2MH
+D8Ri0f4tfp93y1uvv/XKi/F4LCevgCDH2uQDgWp4GrVaqbjcLE3+YGhIVycAYAgSx1L1zIbSq0gX
+AtMdy7Kfvv/uR++8lZaeef8jj81dtHjsfROJb/qbmtRKpeKygSoAgQqAK1Ez+NxMLazukTw4lt27
+Y+vWD95LTUu/58uPzF+8dFx60CFQDY9haAVD4zg+ZDqJAZFYnON5WZZhvD0AI6EI3KqBKJUsOI7d
+v2vHS88/q9ZovvDAQwuWLKUoeuyHRRCoRoJjmFKhUDD0pQPC+0mSFI7E9FotRcE1BGAYgih7o3wg
+LhZaYSWqmc/tcn76/rs7P/1EEPjb73tg7sJF4xWlEPRRXYZGpVSrLvcD8wWDLD8zZ9YCYOyCcfFo
+e2h7rW/shwJTn6uvd+uH77c1N2bm5GVm5zDMeK6MBYFqREqFQqm4XKAKhSM8BCoARuCP8c2uGAMd
+VMmhdHbFz5559p6HHjlz4thH77zV0dY6Ur/JNYBmqxEpFIxKyWAYNtLcr6FIlON5GcHkMAAMI8yK
+zhBXnqYe+6GmIIIgdHrd2I8zGqKEcAxNQm/4pQsMXRW1RvvwNx/v6+3e+sF7eoPx/ke/YrWNz+Ku
+EKhGROA4Q9M0RbEjrKopimI8zgqCSM3QVTUBGIsQK3rCfI55Zi6ObjQZ733ggcn5rG4/q2YIvZKc
++s/EFpvtm//0pN/vf/WFP2u02oe++g2KHoeeKqiVX45aqdBpL/c86AuG4iw76uMBkEQq0jTfuy7L
+ph23HvWk9fejfbsa/GF2eswwkFdY/Og3v5WTX/D2Ky9+8v6743JMCFSXwzCM6rLdVJERFq0HAGgY
+Is+iUMNctGNzvD10qCXwxgnnme5wossyWguXrbj3wYdxgnj9xb8c3rd37AeEQHU5KgVz+cS/UDg6
+U5fUBGCMMAzhMMxwbKKc+E6Vq80dO9MdOdsTDcWnR6WKJMlVG67beMutbc3Nz/3+V7XVZ8Z6wESf
+0ZRGEISCoWmK5PjhJ6HgBYFlOVGUCFi+FIBBzjkipzrDy3J1eTCIagx21PkOtwYDMVFG6FRnaHGO
+dk6GZuyHHV+MQvHuzgNqjdpmv7DYhcFkvvdLj67beBPNMGnpmWP8CAhUV6BQMFqN2uMLDPuqLMvB
+SDQWj2vUMO0mABf0BblTXaESuyov0SWZvmK8+O5pd6efFWUZIXS8PXgmTz8FAxWO4xXz5l+60WJL
+sYxT1h/UA65AQdNa9eXyKSKRCORTADBEhBWjrJSmh0yKa/dpje9cTyTOn2/uc4X5M12hNs/MXGz+
+8iBQXQFD05evLYVjsTgL+RQAXBDhxBAr6hSkFVL+roksI39UePuUyxvhB4ZxipJ8sjN0pC2U6NIl
+AASqKyBJQqlgyJFHSrEsF2c5URy3MdgATHc4hhXalGuLDRSsl3hNBEl+65TrdFeIFS66sbR74yc6
+Qr3BpHsyhkB1BRiGMTStGXnAtizL0XgcKlUADFBSeGWGZmW+PtEFmZYESW5yxf5+tDcYF4dMisMK
+co0jcqIj6SpVEKiujKYpvU57mR3CkWg0lowNxwCMhCZwGEF1bWKc9NFZT5snLkrDTN7W6okfaw/G
++eRqwoGsvyujKery+RSxWCwWh3wKAM7r8LIhVihJURHQ9HeVZBnFBandGy+yKQVRRggF4yIrSP1z
++wqSLElyuyfe7I7NSp2ZkygOCwLVlZEkqVYpcRyTpOFnp42xXIxlYRFFABBCvChvr/O2umP/tikH
+AtXVkmWZxNHtlZYbSo0xQZZldLw92OVn861KNUVEeVEQZSWF9wQ4CFTgIjiG0TSlVCgi0eHb9yRJ
+YlmW5TgFw1zlsQGYaVxhLsZLNi2toKBn4arhOGZUUeuKjQNb1BRe74xeX2qaPUPnoR/VZUl0AaYH
+hiTNhsv1DIejsVA4muhiApB47rAgyXKqHh7axodeRRZalQZlUlcqkvrkR48kSZ32cgPC4ywbi8dH
+fTwAxh82NZZGE0VJxxBpenoqFGYGSLbkyWG/NhCoRoWiSJ32col/0Vg8HIXEP5BIegrpqUQXAqFN
+hdqNBRoMIRzaa8A4gUA1KhiG0RSpYOiRxktJksTxPM8LFAWXFCTGFKnBEBiajMVok0a1I5KhZ4yq
+pL6xwDPPaFEkYTYaLrNDNBYPhqbNgjEATARWkFiYpWVcPXOg70BbKMol9VWFQDVaBEEYdLrL7MCy
+bDgK+RQgqVU7oi8dd+9vDSa6IDNHfV9UkmRmGixDP4EgUI0WSRCGy85PEWc56KYCSa7OGesKsBR0
+T42TLj+nYQitgkjyEWnwfRotHMeVCpqmR+ytFkQxzrKCIFzNUQGYUVo8cVGS02F1j3ESZsUiW7Ln
+piMIVFeFJEiDTjvS9BOyLMc5LhSB1j+QpDhRIjDMrqUtmmS/sY4Xm5a6e44ly5jsg9Lg+3QVcBw3
+Gw0uj2+kHViWC4YjRr3uao4KwLUQRXH//v1TatYuXpSp3jDGEEcPwoLX4+lcW6JLMOnYi1ejhUB1
+FXAcv3wQ4jg+FIkkuphg5qNpet68eeHwlMsynWVCCImhUNKtQzFBWjzxVB2tTMrJqBYuXDjw/xCo
+rgKGY1q1iiCIkTqieEGIRmOiJBHQmQwmklKpvOmmmxJdiosIkuyPiUoSVzPw5R8fvCj/bGf35vnW
+XFOSNv35fOebr+ArdRUwhEiC0GnUOD5iNxXH8dEozKUEko4vKuxsDDS54cs/bpxh7sManyfCJ7og
+iQeB6upgGGYzGy/TMcDyvC8Ag0hA0mnzxj9r9Lf6IFCNm7q+mFVD9a9EleSg6e/qYBhm0OlwDBfR
+8APFeV4IhcMIIVmWJUkSJQnHcILAp1SnNwDjrsXD2rV0qhYS08dNu48tsCjUNCyUDIHqKmEYZjTo
+cAJHIwyX4gUhEArzPM8Kgs8fDARDWq0m1WqmqSkwXSgAE6bRE7eoyRQtfM/HTZZRYdfSRhUEKghU
+V48iSSXD8BwvycMs+CvLsi8Y2n/sVP+AKqVCkZ2RRpFwncEMN8umtGkpsxq+6uNmQ1FyLfBxGfCt
+uhZWszESi0n88LUqURT751LCMCwvO0PBMNDuB2a8m2aZcAwl+Uw/YIJAN91VE0SRJEnssosqyLKM
+Y5hRr8uw20gSau5g5qMJjMThiWx8iJJ8tCP0XrU3xIqJLsuUQCKEHDBE9UoEQYiGQxwbjQYD8XiU
+5Xn+CnP6YQRJay3pbpbCIbkUzHR7m3x2HZNlUiggRW08cCLa2hCOC1KRXWdMslk+aByZlUM3kggh
+iNlXJMm4iAh/IBjweiTpytcLJ3CtwaTRm3gZQ3B5wYwW46VXT7pumm2x6hRQpRoXrCAfbg+tzDeI
+CEu2+/Ow3yB4/BkVHMcZlcpkS7OmZVEUffk+JwzDGIXSZEslIIcCJIFuf9wfE5UUwRAQpsaBLKMI
+K9Y4IiUpKkVyL0M1AO6ko0UQpFqrpxVKRqEI+rxBv0cSh2/9I0hKozcpVZpEFxmAydDiiafpabOa
+hEyKcSEjJMnyuiJjjlFBElCXQAgC1dWiKNpgTmEUKoVKHQ74YpGQeHG4wnFCqdIYzDYMpvsDyUFF
+44uzdSYVjKAaHxiGNAz5xYUpWgVUUc+DQHUtlGoto1Sr1JpQwBfye9l4VP58TBVFM3qjBapTIHmU
+2FSiFcHifuMFQ0hB4bmXZhQkMfhuXSMcxzV6k0KtZZSqkM8TjYR4jiVJUq3VawzmRJcOgMlj0cC0
+SeNGllGUF+v7onMztVCdGgDNU2NCkpTJmpqSkWuypqo0OpVGpzdZaCZJ5+QHSajLz3YHWF6Uxn4o
+gBDiRanFHXvjlHPYiW+SFtSoxoFCpWYUSo3eJAqcWmdMdHEAmDwfnnObVeR1xSaDCp56x0GEk050
+hhgCQzJCUKX6HHy3xgeG42qtTme04JBDAZKGKMtVXWElRRCQnDYeZIT8Mf5wW2BZnh7mXRsMvl4A
+gGshy8gbFep6I2kGmJBifIiSHIyLnrBQmaGFODXYODX9yRJMwADANIfJ2FXcEARJPt0VzreqMvQ0
+BXnU44HEsWKb6ie35tlhWa+LjU+g0vl3G/veSfS5AACuHcekOXJ/OPr9KQJbV2RI1dN6GEE1fhQU
+nm9Ostn9RmH8alQyzL0KwDSGycLVvgXHsGKbCiakGF/Q6HcpaFkGAFw1SZYjnIgQgqU9xkuYFXc1
++v5nW3uiCzIVQXo6AOCqRTjpo3Melpfum2+jIZNiPHgifIMzZoIlkocD3zAAwFWLcuKuBq9JTUIW
+9XjpDbK1feHyVJh9bRgQqAAAV4cT5e4A6whyS3J00EE1LlhBCrOiliHK7JBJMQyoZgIArg4nSK4w
+X5KitsIsf+MEx7ACq8qgooyQQjkcCFSIRfSn1IZEl2JKuJHfTqEZlb0Z5tBv9s7AX75OgR5fkbC/
+FElgOSbFXXNtib4MMwdFYNkmRbZJkeiCTFEQqBCLMW/Rtya6FFPCBmE3NbOGGYRZ7N8+nYFP/ZkG
+OYGBSkHixTZooRo3EU6McpKSwjUMkeiyTFHQRwUAuAqCJIfiYpyH6dLHTasnfrg10BfiEl2QqQsC
+FQDgKviiwrHOYE8A7qrjgxOkM93hYx0hWNfjMiBQAQBGS5ZRd4D96KzHE51RTcQJ1BviOnxxFY1n
+GaGDakTQRwUAGC1WkFxhLhjnc6Dbf5z0BjmNgiyxqRgYNz0yCFQAgNEKxARXmC+2qayaGZhLmRB5
+ZoVeSRqUkEZxORCoAACjpaDxkhRVvlmZ6ILMHBYNbYHhaFcCgQoAMFo6hqxM00CnP5hk0CoKABgt
+DEMEjpEwbdJ46A1yv9jZsaXaleiCTAMQqMafv/HsZ1+9vn3rG0O2N7/3l13/eHPL+y8kuoAAXIsu
+P3ug1d/mjSW6IDPE4bZgnJdMamj3uzIIVONPiIa9dVVxj3PI9pjL4as/HXM7xnJwiWPbP31j9z/e
+HHP3JvpEQXKp643sbvTDaJ9xwQrSWUcow8CUpsAcH1cGgWqakSUp3N3qOLxDZOHBFkyeUFx0BDlR
+klO0UAMYB76oYFbTRTaVUQWJAlcG1yhhJJ6LONqdJ/f7G89KHKuyZ9jmrzKVziMYBUIo6uxynTro
+q6viQgFGbzKVzTfPXoBT9Onf/cjXcEYS+NO/f4pS67Ku+4KlfBGpgjVsxtn371ubk2oa+KcgiMEo
+29HnO1HfdbKh69L9Z+XaF5Zk5qSaDRoFRRK8IHqC0YZO19Yjdf5wDCH00MaFcwrTGGroL84fih2o
+bv34cG2iz/gKugOsLyZkGRQqGhKpx4GaIa4rNpnUsELyqECgSgxJFPwN1S0fvijEogpzCqM3ckF/
+w+t/yFx7q33pBiTL7Z++7q2tUpisKluaLMtRZ5fGn6u2ZyqtqeGeNgzDFOYURmckVRoMh2rx+Ltj
+dcWs3JT395+ra3cihAwaxaxc+5q5+WvmFry958w7e84M3nnDgqK7182ZlZPiC8caO1097qBKQRVl
+WkuzU3o9wT1VzQih9fML71hdfqa559DZdl/oQm04FI2HomyiT/fKdAqiMl2jV0CUGh9ahtBaIct/
+tCBQTQgxHus9ulPiL5oPzXnqgBAJ9f9/3N3Xvf8T56kDsx7+furS6yiNPtTeUPX7HzW++SdVaial
+1vYe2UVpDXm3fMlYXMkFfVFnN6Mz0XrTrEeerHv5t84Te4vv+wdNem6iT3QmC0XZN3dVbdl/DiFk
+NWjmFaXfsLjkluWzM21Gpy+8/0xL/26VBWlfv3XZgpLMPVVN7+ytPlbb0eMO6jXKuYVpBRnWGHvR
+VENHazp+/87+Noc30Sd31dL0jFlNiRL0UI2Dur5ouoHRwlzpowaBakLIohBzOfxNZwdvjHv6JOH8
+bSvY3uCqOqDLLkxfvZlUqBBCutySrOvuPPGz7/oazlhmL8JpBskSHwkJsSitM9I6Y6LPKam5/OGt
+R+vrO11IRo9sXvyN25YNBKoHrpu3vDzn4Nm2Z945cLS2o39jIBzbfap596nmRBd8PMEcP+OiN8R9
+Wuu5pdyiZaBGNVoQqCYEqdbm3vRA8f3/OHjj2ef+u+GNP/X/f8ztCHe3ZRaW90epfqbSeQSjjHS1
+psxfbZ2zzFt7svfQ9mhvp9KWrknLVtrScBLmrUmkNof31R0nH7xh/rLZOWkWvcMTzLQZ1swrJHD8
+7T1nzrSMKZ9zKuv0s5wgpepo6KAaux11Xn9MQFA1vRrwiJQYEs8KsciQjUqLHSdJIR7FKarg9kcL
+bntYEvm2j18599eftmx5IdTWkOhSAxTnhR5PyKhTrZ2XT+DYLStmZVj1Dm+o2x2IszNzQnFOkD6t
+8Xx0zhOIC4kuy7TnifC7G/0r8vQpOkievApQo0oMWmtUWVNFLi5LIoaff0r11pwU2JjSYicYFa0z
+2JdssC/ZwPpcHdvfbvngRTbgXfDkrxFCA/uDyccLotsfyU01GTUqHMMKM60KmgxHWZ4XR/N2s15d
+lGnVKM/fpDhe9ASjnkBkNO9NlO4A2xfisk2KVB2T6LJMb5KMTneHc8yKWXa1GuqmVwMCVWJo0nP1
++WWBltqos0dlTcMIQuRY95nDGMJ0eaWEQslHQzhO4LSCNlgsFUucVQe9tacQQgjDGIMJYZgYj8mS
+BCl/iaWgSAy7ij/B7avKr19YLErnl8ft6PP/5eMjz205nOjzuJwGV8ygpEpS1IkuyLSHY2hdkXF2
+qsYEY6eu0kRdL0GUeEHGMURTxJBxAqIkC4KEYYimzj9TSJLMCZI83JB3AsdIApdkWRBlAseo4bpz
+ZRkJoiRKMoFjJIlLoswLEo5jNDV1b+K63JK05ZtO/9+/1/7t6aL7/kFlTe09uqvp3ecz191qKp0b
+6mh0HNymsqbZFq4hGaW7+nCwtS5t5Y0IIZwkTWXzSUbZsf2t/NseVlpTMZxAMBRjEmEYhhDq/7b2
+eIKCII7+6j//weHfvLVvemX9heJimp7OMUF1akxkhGRZxjHMpoWe5qs2IYEqHOP/762a//7byeIc
+46v/b11+um7wq9uPdP378ycsBsVHT29ECAmifLTGeceT24KRYZr4r1uc8U/3l5+sc//sxdOP3lLy
+X19bcOk+/lD8F69Uv/hxw5c2F3/v/or39rZ959eHrl+c/sp/rsen6uyZBM2krdxIG0zN7/1120Mr
+hFjEWFhe+tD3sq+/k9GbaI3Be+5460cvV/3mX4R4VJNVmLPxnoLbHkYIYQRpKJw967F/aXzzT+f+
++rM5T/xv7o33MQZzok8oWSgZKjfVIMmyKxiSZLmqoSscZ3VaJUPP2Gfk2yssCKEp+1OaFiRZ9keF
+U93h1QUGmNL3GkxgjSrGilX17jc/a3lkc7HNdCERU5RklhO5QW36kiTHObGiwPSzxxcXXBzVGJrQ
+qCgMoUhceOnTxu/eX27QMEPqDzVtgbPN3tx03Zp5aTJCoijHWZHjpURdU1PZvM3vnKHU2iHbix/4
+VsEdj5HK800olEqbMn+luWy+yMaRLGMURSrVBKNEGEaqtbmbv5h13RckgUeyjBEUqVASyvP5gRhO
+FN39jbybH5RFkdLoCAXMFTZJFDSZbtWbdepeb+iD/TWCKG09Wt/p9FfkpebYTWolHYlxY/+UqYaA
+G+uYheLi61XORmd0eZ4eAtU1mNjHQF6Qnn2vbvEsq1GXSl1pEAZN4TajMs06TFN4mlW9aWnmtiNd
+h6r7blicQRAXHepEnetknXvV3NSKApNWRd2+JmfJbJtGRSXqGRCnaKXFful2SqWlVIOiF4bhFENT
+w7SoYBhGKtUDIe1SpEpNqqDPYLIVZlgfvWkJL0rv7DnTP5I3zgkfH6zJsZseu3lJR5//sxMzKjMz
+xkvnHJE0PZ2mh3a/aydJcl+I21nvfWxZGkSpazOxvTgPbiyMxPg3drS2OUJjOY5Zp7h+cUacE9/e
+1SqIF3VlOX2x+na/TkMvq0gx6RgCx4xapizXmJUC09+BcZNpMzx846L/+sqmRaVZ+8+0PvPugYGX
+Xvj0+GcnGnPspn/70obHv7CyIN2CEMq1m75335rnn7z7kZsWJ7rs1+5EZ3Bng9cRnIHVxMnkjQl7
+mwNWDb0sVw/V02szsTWqtfPTwjFh98meBWUWq1FpuNYVl7Uqal6JJc2i2nq4q8cdzbJrBv7e51p8
+DZ3BnFTt3CIzgWORuHCouu9vH9bfs6Fg84pMDLIMwLUy6VQ/fHDDIzcuRgjRFKFRMnGO/9snR9/a
+fabT6R/YzeEJ/uLVXU3d7rVz8x+8Yf4tK2ZFYhxDk0qGanV46juc11yAhDvREWJIXK+Ysd1vk4Mm
+8AKLstim0jJwJa/RxF44g5Z+cFPhj/968r097WU5xqXlKdd2HILAUs2qDYvS/7yl7nity25WKj//
+k59qcDtckc3Ls0pzjQghXpA6esMfH+xcWGqTEYIwBa7Nj/++3Wa4UCkXRSnG8p5gpMXhvTRn71xr
+byASP3Cm1WbUKGkKxzFBlELReJcrUNPW17/P8x8e3n68vqHD5fZP6VFTA5xhrqY3cl2JCdb1GCM1
+jc/N0FDE1E1CnvomPMIvrUjZuCTzvb1tnx3vzk7VpllG7Plv6Qn99MXTpkGDCucUmjYuybIaFQgh
+nZq6eUX2c+/Xvr6jed2CtP5A1eWMVDV4lAqyvMBk1EIzOhg3Hx2suar9u5z+rkHVrEvtO92y73Si
+z+pq0AS+KFtXalerYe7UaxXnpZggGZWkXgl1qTGZ8CCvVVH3XJefbddsPdR1+GwfL4yYjIdjmIIm
+lMyF/2jywhgshiYqCk0lWYbdJx1dzkh/T9XpRk9TZ6Ak21BRCPnZAIwng5K8udySZ1ZAr8o16/TF
+t9d5G13RRBdk2puMOF9RYLp5ZfYf3q7ZdqRrVp5ppN1yUjVP3DO7JNsw7Ks4hhm1zO1rcn/+0ukj
+55w5qVq9hj5R5/KFuPJ8U97FSe0AgLGzXWunMkAIhVmxpi9yqitUaocBJGM1SRXSW1bmVDV4jte6
+tx/tshoU13YQhiZuXZ39+zfPfbi/44YlmYIonW70WgzKWXlGdUL7e2VRiPs80d6OmKdXiEZwgmQM
+JlVKpja7cPBufCQYcXRGHO18OIjhOKnSKK2pmrScYZfwcJ85LPG8Lq9EYbQObJQ41t9cE2pvSF+1
+WeTi7jNH+EhwyBtVKRm2eSsSeDXADFDVHTaryBQdQxNQn7oWkiw3uqJNrtjsVE15KmQgj9Uk3d/T
+rapNSzPPtfi2He4qLzBd20EoAp9TZCnNMeyr6u3oDXuC8TZHaM281Nl5iVyrSRbFYFtD37E9vobT
+cZ+zP1DReqPSkpp13Z3G4kqCZhCGCdGw4/BnzhN7Qx2NYiyK0wpKq9dm5FvnLEtddt2l63c0vP4H
+NuCb9cg/Dw5UXMhf9/dfOY5+Zp27IursPvHz7yIMKa0XLf9hnbscAhUYC0GSnzvYvanUvEpJ0QR0
+UF0LUULOEEcR+Op8faLLMhNMXkVk9bzUqgbPm5+1dPaFo9e6XgCOYffdUPDvz53YW+Wob/eHonxp
+jtFuTmTNOtrX2fzun731p1OXbCj70j9pswv5aKjv2O6GV59xHNq++Kk/mUrn4hTtrTtV87dfqMz2
+kvsft1QsJRhF1NntOXss1NFgm79yNAtNyZLE+r3OqgMZa2+l9aaY24EQKrzr63mbv8gYLQm8AmAm
+kRHqDbLOEJ9mYBQUVKeuEUVglRnaAqsqw3iNDUhgsMnLmDRqmU1LM0tzDdXN3sbOwDUWF8NuWJKh
+11Dbj3R/fLCzNNdQXmBmEjdhviyK7dve6j262754Q8kDT/S39VEqbdqKTfP+6edcOFD74q/YgAfJ
+cu+xXVFHR+ryG1KX3UBpdDhFa9Jzs2+4u+SL3750sqVhCWw00HKO9bvTVmyEFRTBBJEkeV9zIN+i
+SNMxNGRUXxNZRrKM7Fo63wJr+I6PSf0iLp5t27g0M3UMFSAMQ8VZhnnFluN1Tl+QnVtoybAlsjoV
+d/c6T+1XmGzWyiWE4sKXEidITUZe3s0Pdu58L9hSJ/GsxLGyJEmCIIujWrjoUkI07Gs4Q9CMbe4K
+HBpkJgVFEloVo2QuPBYQOK5VMTrV+cdkrYrRaxTkxTd0miINGqVaSSOEcBwzaVUD/+k1ChVDEVN4
+cRZRRo3O6OpCoxay0q+JKMn+mBBmYZHJ8TTZP5gblmTee33+GA/yhTV5Ri1jNyuXlqekWxM55Z2v
+8Uzc3avLKdJk5A15Cado++L1OEH6m84K8ZihsIIxWtynD/obq6/ts1ifu2ffJ8aiSkZvgnU9Jse6
+eQUf/vSxf75/3cCWwkzLG//vS1t/+dX+f37w08eOPfedBSWZg99177o5NS89+Yd/uhMhlGk1eD7+
+8cB/x5//7n8+urEwc+o21dIE9tSm3BV5eiU9daPpVFbTG/nx1ta3qlyJLsiMMiF9VGoF9d37Kr5x
+R5laRQ0Zj51uUf3nVxb8y5fmDsxRSxLYktm21nfvIwlcNbrkvbvW5920IgshdGmyn15NP7Cx4I41
+OQqGxCf+bi7EYxJ/hQXIY+5eieey1t+OkNz01vP7vn+vIb/UtmC1Zc4KfW7xKNv9RDYW7myO9nUV
+3vXVwdtP//Zfq//wn+jzJ/Tcmx5Y+C+/neizBoNlWA13ra10esMtDg9CaPWc/Puum6dRXhh+Ho6x
++ff8jyCIdrP2H+5Ycc+6OQaN8rdv7a1u6U102UekYUh4FLoGMU587aTTpKLWFCYyw2vmmZBAhWFI
+wRCK4ZoOcBxTKcghAYkk8KuaV4KmcJoafoQHhiGGIhhqslothlvscegukoQQwik6Y/XNtnmrwj2t
+vtoqz9mjrR++rErJLH3w27YFq654EDbg89aexGnGNm/l4O0lD30nc+1ttP78r2KUYQ+Mo8Yu1+o5
++R8dqu0PVKU5KaXZNlcgPLCDLCNfKMoLYjAa//c/f0oSeEV+WklWyhQMVKwonegILcnRTcJD3oy0
+vd5HE9iKfEOmESbKGU8wsceYEIwCJ69wDRVGa3/uQ//KHQqjWZ9Tkr5io6/xTOuHr5z5438u/Jff
+6fPLLn8QLuANtNXr80qHtDEyBosmPQey/hJo96mmFRV5GxYUtTu8GTbDqso8fygWjg8z47ggSu5A
+5ER91+KybJtpyj1SRDlpf3PgzVPORVk6HPqnrkmBVZlrVmQaFbCcx/iCZugx0WUXMgZLxNERc3YP
+eUkWBM+547Ik6nKLCcWFFFWcYhiDWZOZbylfbCyuDLY1eOtOXf5TZIGPONp89WfMsxbiFEwWMLUE
+IvGTDd1zC9OKs23zijI0SuZYXefgdUGHcPvDgiiSUy+hLsKJR9uD6QYG5nK+ZnlmZZFNpVNAnB9n
+U+7XMr2o7FmmWfNjLoen5oTIsQPbZVGM9nW1f/q6de4KfX4pTinEeEwWL0oEwnCSYBQIw3D8CnUy
+NuD1N51DkpiyaE2izxgM49UdJ8Mx7mu3LN20tKSuvW/v6ZbL7EyRxBTM+hNlzBNDNY7I9SUmaPe7
+Br0hjhNlBYUzJA4XcNxNuR/M9EIwisy1t+rzy/qO7u7Y/la0rwshJMRj7uojtS/9mo8Ei+79ptKS
+iuF464cv1b/6jPPkfi7oRwjx4ZC39qTn3DFtRp6hqPzynxJz9/obq1UpmaaSuYk+4+RS09bX5Qqk
+WXSZNkP/FgzDaGrog0VDp+t0Y09JdgrLCduO1vd5L7dMaG6aGSEUjrFoKhFljEd0sU05O1UNrVZX
+q8MXf/Goo9MXl0bRaQ2uAfRRjZWhsDzv5gcdh3Y4T+73NZzGSRpJkhCPCJHQrEeeTF28nqAVCCFK
+awh3t3Xt/qBn/ycIw2RB4MNBWmvMWHvrpantQ8RcjnB3m3XO8mFnBQQTp9Ppb+p2r6rM27ik5N29
+1aIoLSjOtBnUx2o7huz54cEapz/c6fQfr+9aVJo17NG0KmZlZd6aufmtDk9zlzvRJ3cRHJPtGuyO
+bJs6ccPnpyMZIU6Q3j3t8kZ5WUawBN4EgUA1VjhFW+cuV6ak++pOhzoa2YAHp2hdTomhsHzwnHsZ
+a27W5RT5Gs6Eu1r4SJCgFYbC2cbSuaaSOQQzzCQraSs2CrGYypbOh4OhjiYhHrWULxq8g9Kckn/b
+w6ayeYMHGoNxd7C6tSjTetuK2Tl2kyBKlflpvd7QS9tODNmtqqm7qql72CPQJPHtu1eJoqxXM3OL
+MnhBfG/f2XOfL6g4RZCYbFVIZfZEjkqcjkRJPtwWaHTF7pprTdPTUBmdIBCoxgFO0brsIl120WX2
+IRiFsbjSWFw5ymPmbLqv/3/8Ted8DWdojWHIe1X2zFmPPpnoU5/59le3IgxbO7fAZtAgDNV1OE/U
+d+082dT/6ieHaxs6XZGLc/y63YFPj9T5QlGEUDjGvbLjZGn2+bWta9r69p9pOXyu3R2YQuv8ulhF
+S1iTZ4U8nasmI+SJCGsLjfMydSqojE4YCFRTXaSnLdzZrMst0WYVJLosySgS47Yeqdt6pG7YV3/6
+8s5LN55r7T3Xen6MlCcYeeQnryX6JC6Hk/CTXvObXdlPGjwwg+TVInFsY6kJIaSkoL9/AsHFnfJk
+SZ2ek7r0ukSXA8xMblZRE9KzIm5l4okuy3TCiXJ3gJUkWUUTKprAINNvIkGNaqpLX31z+uqbE10K
+MGN1RVVelrk1o1NFCP5EF2a6EES51R3bUu3++oo0bUJXbU0SUKMCIKl5eYaXsFXWqZXcMZXJMuoK
+xLecdbujPNSjJgc8CwCQ1DbauzfauxFC3JgPlSSCceFER+hsT/hHm3I0DNxCJwPUqAAA4Cp0+OKN
+7thXlqflJ3Rt8aQCjwMAJKmOiKY7prIxsXxtaOxHSx4lKeocs5IhIX9i8kCNCoAkddxn2t6b6uIU
+Yz9UUqEITMsQ9NSbVnjysfHY/t2fbV4599W/PRePRQe/1NvT/av/+Y/vfOWLR/bvHvsHQY0KgGTk
+YhVnAiZRxvI0UJ0aFV+U393o5yXp1nIrQ0KUQgghSZLCoWBD7Tmv2yVJ0uCXeI7rc3R3tLWEQ+Pw
+BYPLDUAyOua1hHhqlt5vpqfW9LhTEyfKnzX4zvSEbRoa1pqafFCjAiAZWZn4BrujVOsnMJjw+wo4
+UTrQEjjriORblHMytAQEqkkHgQopZPZ+7q3ElkGWUZwTFIke307LMy1FWcvIT988eSfFyzhCiMKk
+MR/pyuc1xiOU6AIyCqgIYYzHSQY4homSPDtVPT9Ta1DCPTMB4KIjGnGb+O2JLQMvyC9+0rBhYXqa
+VU0S8Lw2btQ0+u5qfnI+qy+urA3pTRRbYfAl+rwvh5NwEpO15CRdlhmAxLGKdA2GMKMKbpiJAX1U
+iSdJcq83+stXzvz5g3p/GDoMpiVexg+4bX9sKt7ZlxoXp/Qs2nUh3Wd9qZ1RWNHjyoJxwRnmWEGy
+aWirhoLeqUSBQJV4nCCdqnfVdwaeeevcjqPdwchMa39LBl1R1aeO9E8c6QfctoaQLtHFGZEg4584
+Mj50ZPTEYBmzKxAkeX9z4GRnKBiHBtIEg0CVeCwn7qvqlSXZE4j/7MXTVQ0elhcTXShwFUQZ2+20
+H/ZYgjx1ymfa5UwV5Sn66N0XV5wLGGbp/QWQlX5ZgiTXOMLb6rz+mAgje0eCYRhND7+MmSRJkiSR
+FMUw4zBQDwJVgskyCka4Hce6ZRkhhE41uN/e1druCEkS5GJNGz0x5bbetOawFiHUGVOf9Jlc7BQd
+RdsY0qUrY/MMXjMDjcyX44vwr5zoyzUrV+XrLWpYqGt4JEnZUuwKpcrrcQ8ZRxUKBaLRiMlsTU3P
+GPsHQaBKMJYT6zsCVQ0eST4fmZ7/oO7TQ13eENxHpgdBxrb0ZJ3wmqMiiRDiJPxswLjHmZLocg1P
+SYjrUhyZqim0vvAUJMvIEeTsWubWCnOKFhY+HhFJUemZ2fMWLd320Xu9jm6OZWVZliQpGo3UnKny
+uJx5hUVpWdnj8EHjUlwZoyQC+mavhZ+N7j7VO3hLNCb86b2anEzTpmV5BDGlu+UBQqgzxLzTmd0W
+1QxsORswfODIvj2rj5x6s+wstPXPc4NLaOgPVsKh1wohhCQZ4RiqSNdUpGvGfrQZT6vT/9OPfvyP
+X773T7/+2R33PZRfWCII/InDB95/4+WU1NSbbr9HqRyHqXvHJ1CFjKtDxtUJvmDTU1dH2772rw/Z
+WNPqf/W0Xbn8a4VFZYkuILiC5w/21MYdvHShBsxJ+Blu9mvUjasKDIkuHbg6vCi7w3yqHmpRo0VS
+1KyKuX96+e2//uG33/3al1x9DlEQ5sxfdONtd2248ZbcgqLx+ZREn2ZSEwTB2dtzcO/OS1/6+L23
+cvOL7GnpWp0+0cUEI+oLcdvrve5LEjVb3LEPzrqnTqCSEdrf7A/EhBX5BhiyOpIYJx7tCP1xf9ez
+95VoGBIyKEYJx/HCkln/8bPf8DwvyzKSZYIkKYqm6HHr25tyTRNJxet27ftsu8APM/QyFAy8/Jc/
+fvzem4kuI7icZ/Z2NTqjgjg08yXKi3XOyKmuKZNZJ6O3q1wkgcFIoJFEOPFAa+CvR3runW9X0zDq
+/uoQBKFUqXV6g95g1BtNGq2OUShwfNx6LiBQJVIw4D9TdXzYl2iaEUWxq6PN0dWZ6GKCYcgyOt4R
+2tPsD8ZFebhXHQFuW5030cVECCFBkg+3B/0xocSmUtHwkx9egzP6zmnXhmLTuiIjDvnoUww0AiSM
+IAhuZ1/NmVM4TpgtVrPVplKr/V7Puk2bMzJzzBaryWJLTc8wmMyJLikYHi9J1xUb/TFRkhEnSLV9
+EUmSzWqKl2RekDEMdflYb1QwJXreHV6U9zT6big1WbU03IJHkq5n7pmfMtuu1jKQwTTlQKBKGFmW
+rbaUr37r+1qd3mg2K5VqQeSf/n//tmr9xvI58xUKBc0ocByef6coDEPFNlWqjuZEGclIlOS/HnEw
+JL6h2KSgcFGSZRkxJK6kEv8XpAhsY6k53UBPhcJMNbwoI4QoAjOrqcXZOgUsNDUlQaBKGIIg0jKz
+77jvIZVaTVE0QojnOL3B5Op1YBimGI+cTjChTCrKpLrQXVySomJIYk66Rj/FshVIHJuTAZnWw+BF
+uao7FIgJlelaq4aC9TumLHh8SBgcx5Uqld5g7I9SCCGKphcsW3Hm1PE+R0+iSweu2tJc/YIsrWIq
+1Vo4Qer0xbv9MHh8GLwonXNE9jcHOv2sjGAimCltCv2oAEJo3fU3dXe093S2D5sKCKay0hR1sU01
+pRYp90aFj2o8jiAEqmHU9UW313sFSa5M1xinWCUYDDGFflQAIVRYUmY0m9tbW7wed6LLAq4CK0jB
+uBgXJnzJxNGL8VKDK3q6OwxT1Q3LFebVNLE8T1+epqGm3hwiYDD480wtjEKx4cZb2luaOtqaE10W
+cBUCMeFkZ6jNG090QS5wBNhDLYGKNE2OGeZGGkZluube+baFWToKuqamPAhUU86qdTcEg/7WpsZY
+NJrosoDRinDSnmbfOUc40QW5IMqLEU68rsSY6IJMLe4w3xvkWEEyqymTiqJgaO90AIFqylEolXMW
+LG5va6k7d0YSYWGq6cEX5TEZU5BTaAhOoVX17TVZBRZIH73AHxPeP+va0eD1RmEtxOmERAjBI8VU
+c/Mdd/30qR/s37k1OzfXap2iC0aAwUKsYFIRNs2UeEAXJVmSkYrCVVMpBTHhIpz4TpXzSFvgq8vT
+rGpyKvylwKWGbYglEUKZ2kQXDVwsU2u5bs3yffv2tZ7aP/eOO2CB0akvR09oCFVlKp3wX5OMUJMr
+7okKi7O18L0ZwArSs1XO3Y3en92cPTddPaWSM8EVQVLmFHXnnXdWV1fv379/yZIl6enpiS4OuIKV
+eVpZRlNhgqJgTPykzl/bG8szKWxayPc7j8Sx5bna64uNOSYGJkafduCxYorSaDS33norQuijjz5K
+dFnAleEYRuBToup7qCPU6mVX5mktGngMvYDAsfJUdYGFoSFKTUMQqKYoDMMWLVpUWlpaVVW1f//+
+RBcHXEGTO+4KJ36MtjPMH+8I0QS2vkg/Fap3CdflZ3+/3/GnQ30IIYrACLgo0xMEqqlLqVRu2rQp
+Nzf3tddeO3fuXKKLA0bkjQp/PeY83pn43HS9gtxQaNhcZrRqoNEPNbnjzx52Hu+MrMrTJbosYEyg
+cWBKS09Pv+GGG15//fVnn332ySefTEtLS2BhInHuTHPPnlMtZ5p7YiwnI5SfZl5WnruqMt9qUCf6
+UiWSM8S7wzwrJn6+OIbEKtLUsixDzaHJHX+jyt3t5x5dbCu0KhJdHDAmxA9+8AOFAv6KUxSGYXq9
+XqVS1dTUNDY2rlq1KlElibH8kZqO37+9T5ZRfoZlXlFmqllb3+HcdaKJIogMm0GjpBN9tRLmVHek
+2cOWp6rzzAn7KbGCFGZFGSEVjdOQ0oYQJ0okjlWkq5fn6qbEoAFw9eLxOI7jGIZBoJrqSJI0mUw4
+ju/atcvn8+Xm5iqVCZgRR5Jllhdoilw7v/C6BUXLK3Ir8tJSzfq3dp9u6HSV56fm2E2JvlQJ0+yJ
+h1ipzK5M1ycsWp/oCu9pDlIEnqpL3icGhBAnyjJCOIYpKTzTyOSYFBClpq+BQAVNf9OAVqtdsmSJ
+3+/fvXt3b2/vV77yFZvNNskZZgxFzsq1z8q1D2xRK+k1c/NLsm27TzV3u/yJvkiJVGxTGlVkloFJ
+VAHCrPRpnT/GSXPTk7oNlhflo+2hMCfNtiszDAysLzVjQBPB9GC1Wm+99dbNmzfX1dX99re/ra6u
+5qfAOiBRlscwjBdFQZxCs4ZPvmwjszBTk5K4QUsH2oKdPnZehrrImrzzzwbi4tZ6//vnfA2u2FTo
+LwTjCGpU04ZOp7vpppsyMjKeeuqpaDS6fv36RYsWmc1mgkjY/HJnW3rqO5xapUKnSt7WY0GSnWFe
+ryDVdMIe+845onPT1QszNRpmCk02OMmOdoQ+rPGm6pj1hfr8xHUWgokAfVTTCUEQdrt92bJlu3bt
+OnbsGMuyPM+TJMkwzOSHq25X4A/vHjha27lmbv5Ny8pSzUmaAewI8p/W+fQK0pK4jHACxxZkaNIN
+TDIn++1rCRZYlLeXm/ItcEObIQb6qDCfz2cwGBJdHnDVPvjgg3feeYdl2cWLF1dWVqakpKjVapqm
+KYoiCIKiKLV6ororZFnudPr//NGRP394xKBR/s9Xb7xuUbGSTtKBO9vq/G+cdj+8KGV5bgKm+Yvx
+EkPiSdsXIyPkjQg6BUESSRyiZy6fz0cQBI7jEKimt61bt7766qvV1dV2u720tNRqtZpMJrVabTAY
+SkpKVCqVSqXSarXjlXkhy4gXhC5X4D//tu2jA+dKslOe/OK6NXMKtKqE5REk3F+O9LV6uXvmWmbb
+J7t/SJDkvS3BuelqvYJMwlglycgT4d+o8twy25imoyF1YuYZCFQkQkiCfsdp67rrb7ju+huam5vr
+6uq6urq6u7ubWlqCwaDL6aqprVmzevXKlSvvuvNOg9FIkCSGxvpLjnP8qYbuR3/6emevb93Con//
+8nWz81Jpkkzmr1CNM15oVRiVxCRfBFlGjc74v3zU/u/XZ63K16qo5OqdEmXZHeF/t9dxoC28NEdn
+1SIsib+EMwk23H0K8/l8YYUh0WUD4y8Wjb772st/+OXPbPbUx5/84aJlK1VjawzkeOF0Q+f3f/l6
+e7fnH+9f/7W71mqUyVuRGvBulSvPoiyyqZSTm0zBCtITbzTm21SPLrWb1MnV7ipKcrM79tOtHXFB
++s1dBSY1DbWpGYPBkfXzoYAXNf1FIFDNRLIs8xwXj8ffee2lLW+8ZrZav/qt78xZsJiir3FAaGNH
+33/84b2quo5/vH/DQ5uXqRQMdAsghDhRJjCET24aQ5QVXz/pfOuk6zd3F+aaFUnY6uWPCjvqfCsK
+dDYtk3xnP5MxOLJcEqiIH/zgBzwJSTIzEIZhBEkyCkVBccmGG29m4/GX//IcG49n5+Uprn5uC48/
+/OmBs+/tOjW3JOuHj27WqBQQpfoRODb563vgOJ6io2elqUtT1TSRRKMhBVFmRYkicJrEciwKg5JM
+5kTHGYnEkOrzZmyYmSKJaLQ6jVZ36z33pWfnvPPKi36f5+4HH05Jvbr5bf2haG1LD0MRNywv16rh
+yea8/U0Bi4bKNiuUk7voO4Eju442qankWamWE6Uz3ZE9jX6zmvryEjuOYWo6ubrlkhkEqmRhMluW
+r15LUdSnW9594U9/uOO++wuKS0f/9kA41tTpDIRjL35w8NMD1UNe/eqda1bPL070KSbA/mZ/WZo6
+RUdPWqCKC5I7zJvVpJIilEnT5hVmxYMtgZ31fr2KXJKbpCP2khkEqiSi1miWrFil0Wp3fPzhB2+/
+ecd9X8zOzRvle7PTzN/54vWBSGzYV3PTLYk+uQSI8eI5R2RZnl4xWVFKlORWd2zLGc8dcyyFNlWi
+L8DkaXTF2jzxWanqRbnaAkvyThOVtCBQJRelSjVv0WJJFD/d8t77b7z2xce+YjJbR/NGs16zZmFJ
+oos/hcgyanLGRAml6OhJa3/rC/EHW4JRXjKokivNz6ahluXprRoqJbnnhk9aydLADQYQBFk+d/6S
+Vat7e7q2bnk/0cWZtjAkyWhlod6kpianBS7Ciqe7QrW9kU1lpmRYwJcTpAPNgUZXLM5L6QZmdpoa
+olTSghpVMlKp1YuWrQz4fPt37sjIyVm5dkOiSzQt5ZgVRhWpV05Sl37/oohldvW8TE2iT33CeaPC
+jjpvjSO6ocSYlrhVvsAUATWqJGU0mZYsX5VbWPTWS3/vdfQkujjTD4aQXklmmRSTlh2uYYhFOfpb
+KiwzfgFfXpQ+OOPe0+jPMSuyTQrFTD9fcEXwDUheWXl5m269Q6lWf/jOm4kuyzQjyXJfiGv3xoVJ
+mTcpxkneCB9hRSWFJ3CO9skhy4gX5ZreyM3lljvnWDONsP4hgECV3LJz8zZsuunDt990dHUluizT
+CS/IexoDn9Z44/yErxgpSHKNI3KwJdAb5Eb5Fo4XnN5gbavjZG17U6czwRdrdGSEOFEOsyKGIRVN
+fH9D1nUlRo0CRkoBhCBQJTmlSlUyq7y8cu5f//h7nh/tfRDwony0LTg5q2v0+Lltdd7q7shoWvyi
+ca7PEzhV3/HiR4ee/NUbjzz151+9uDXRV+vKREn2RPizPeFzPeH+LRYNBRUpMACSKZJdanrGFx/7
+2tceuPu2e+4rLpudwPWCpwsZoZggHWkLfGNV+kSP8xUkeWe9lxWkjWXmTOMVpgCWZPnYubYtu06e
+rG3v8wRlWZblaTCjOC/KXX52W41nT6P/vgUpiS4OmIqgRpXsCIJISU+/9a57nv5/T8Vj0UQXZxrg
+BKnNEzOpqElYA6nVHXOEuFlpmjmjyPQLhWOHTjc1d7mWVOT/29dufvz+6ZHMWdsX+b+93Yfbgk+s
+y7y5IhlHjoMrgtnTAZJEsauz/d4br//rW+/lF5WQJNSzr4CX5C4vm2Oe8Jl5ZRmJkoxhaDQRUUZI
+FEVJkjEMwzFsz4n6b/747+uXlD3zwwcTfcEuxxPhvVHepqG1Sbn8Ixhi2NnToUYFEE4QZott9Ybr
+3nvj1XAolOjiTAMUjmWZJmOVEwxDJIGNst6GIUQSBE2RFEkQxORP6X4VYry4v8lf1xtBCBlVZJ5Z
+qYMoBUYGgQoghJBSqfzK49/e/uEHkTAEqsuRJDnKSQiNqoozFm+edO5p9PtjQqLPePw1OWM/2dr+
++klnd4DjRAnHsKkdVUHiQaACCCGEE0RWTp5CpTqyf28wGEh0caauCC8dagn8fnf3hH7KznrfZ3U+
+hNDMG+t6tC34h33dCoq4Z37KvEwtCY06YBTgWwLOIynqhs23HNy9y+/1JrosU5c/KhxsDZgmbNSt
+jFCXj33rpHNelrbQqpx5k1DYdfQtFZY759oWZGmNKmjuA6My034GYCxu2HxrV0dbn6NnYEyVwPMw
+vmqAJMueCNfsii3N1U7cp7jCfLFdta7YaNPSM+A+LstyICbsqPNFOFGWUaqeWZanK7AqJ21tFDAD
+QH4XuCA1I4NRKg/t3d3n6IlGwqFgKOj35RUVr7nuBr3BmOjSJR4ryKG4aFJT6QZm7EcbFoZQqp6+
+rdKaqmNIYtqHqbggNTmjn9X7g3FxXqZGRREUgSE07c8LTDIIVEktGom0tTR1trV6XK5QMBgKBro7
+O3u6OpVKVSgY6M8A/ML9D669fmOiSzol4BhKNzC3VVomYiJaQZJdIS5Vz9hnymIWMV6qcYQ/qvZg
+OLYoRzdpiyCDmQcCVVITBaG9teUvz/ymtanJ7/Oy8fiQuQyKSsvyC4t0ekOiSzolMCSeZ1HmTcAK
+s6wgNbvjB5r8X1pqn7Tp2CeaJMuChIxq6oYyc5ENluUF1w4CVVJTaTT5hcU6vaG3Z/g0tpy8/LzC
+okQXc0qI8VKUE/VKctwTAARJ7vDFt5x2B2KCKMkyMY2bxkRJdoa4OC9btZSGIRZmayvTNdAdBcYI
+vkBJjSCI7Lz8rz7xT2kZmZe+yigUuQWFmdm5iS7mlNDhje9u8Puj4z+wyRsR9jUFOryxexbYlNQ0
+7piKcmKtI/rxOc/+5kD/hcIxDKIUGDv4DiU7hmFKZ5d/8bGvKVWqIaMuU9MzyioqbampiS5j4nGC
+VNsb+azBF4jx435wb5R3BLgvLrbPyRjPpXu1akV5YUZ2qnlyLpEky6e7wi8ccZztiaYZaKt2hq+b
+BSYTNP0BpNMbHv2Hbx0/fHDfZ9t5/sKNODe/IC0zK9GlmxI6fGy7ly20qfKtqnE/eKFV+d31GUpq
+nOetX1CW8+bT/zDRV0aSZRzDEM4s83YAAB/YSURBVEKsIO1vDtj1zG3llnzokQLjCgIVQAghlVrz
++7+9vG7e7L5ehyxJCCEcx0tmzc7JzU900aaETh8bjAkbSsYzR1+UZF6UcRyjCUyJT8vVVQRJdoU5
+g5JUkISSIr5/HTzWgAkBTX8AIYQwDNPpDT995k8Wq61/S1pmVuWCRanD9V0loTyL4sbZ5spxbZo7
+3R3+477uD6vdiT65ayRI8rG24AN/rXnpSJ9vAlpEARgANSpwwar116/fdOPH774dDATyi4qtKfZE
+l2iqyDQy6QZmHCeiPdYefPuU266jxreWNjnivHSqM/TsAYcgyd/bkLUiT69VwJ0ETCD4eoELKIp6
+4gf/1tXWduzQwVkVlRnQQYUQK0gYhmhiPNedb/fEPjnn1TD4TeUWLTP9foMkgWUYmXVFhqV5+jQ9
+o6RwmPscTKjp9yMBEyo9I+vRf3xCEITZlXNtqWmJLk7i1fVGT3aG8izK1YWG8TqmQUVtKDaqGDzH
+NOFLL44XTpQanLG+IDcvU2tQkXYds7ncolMQE73cCQAIAtVkOvy3M11VfYkuxZWFg3xmeFHbO+H3
+D+xOdFmubN13FpuydRN3/OqecG+QW5A1nh+hVRBzMjU4hk2LQVOsIDc4IweaA7W9sVyLoiJdgyFE
+EZhRBXcPMEngqzZ5XI2+tiM9iS7FqFjEvFgramubBqVlwxM4uXuHN97p4+w6Js+iGPvReoPcvib/
+4hxdlkmhoqdNml9PgN3fHAzExPXFxgKbUqeYNiUHMwYEKjAMhhj/0ULTUXeANanJUrtKzYz17twX
+4l480itJ8vxxrZxNnC4/a1FTCgrXKYgFWVoFhedblCoa8oRBAkCgAmBEaXrGoKTGOJ25LMuuMP/S
+0T5XmLtxltk2tadskGW528/uawr0hbhNs8x5FqVZTZnVU7rMYMaDQAXAiLJN49DiJyPECpIvwm+e
+bVmQpVWNuXI2cXhRruoKHWgKtHvjJXb1DFgQC8wMEKgAGIYrxEd50aymNGOOKxiGWTT0/YtS8i1K
+ZuotLS/JMi/KsowUFC5Kcm+QY0V5c7llSZ5eDQ19YGqAQAXAMA60BOKCtDxPf82BSpRkb1ToCbCV
+6RolhZfZ1Yk+p6EkWQ7GxE5/vC/IG1Xk/CwtQ+Lri43ri41KipguefMgGUCgAmAoV5g/2BKYm6lV
+XmuVghflngC7t8nf6IxVpo/nxEvjyBHgjrYFT3WFI6y4IFs7P0uLYWgapSOC5AGBCoChPqx286Jc
+nqa2XFMSgSDJXb74JzXeU52hr61IT/TZXESSZYQQhjAMQ6e6wnubA2V21c3lZruOSXTRABgRBKok
+otIz1HDPy7IsS5IscBIb4S5eiR4p1BSjusLNOhpkeVZM9MmNp7OOyPoiQ6bxGu/dvqiwr8l/pivy
+tZXpC7K0iT6b8zhR5gTJF+UpAjeqSIbEry8xXldinILdZgAMAYEqidz3n8tnrx46G7osyWxMCDij
+Tcd7P/r9qXiYk6ULwWrdl2df91jF5Q/75+/sOru7I9EnN56evqNgLG+3aqjb5lhXFxnHJWlwXEiy
+fLw9uK8psK/Jf0uF5dYKS4qOpiFEgWkCAlWyw3BMoaaYHJ01S1eyLP3Ff9nbVecReSnR5ZqWWEFC
+CDEkrlOQU2e2WUGUj7eHnt3vmJ+l+fVdhRlGBmpRYHqZKr8lMGmc7cH9r9We+ex8HYhWkTmVtoU3
+5xcusBtT1Xf+cMlLP9zb1xoY/JaAM3r8o+Z9r9YNe8BokE30OY0PTpCe2dt9W4U108hc7RCiuCB+
+fNZ7vD20PF9/02wzQiixWXMxXqrvi7Z54rdVWkgCm5el+dWdBTSBKSgcppEF0w4EqqQjCVI0yAVc
+0f5/YjgWdMc6ql03f3tB6bK01AJDRqnZ74qy4QtL4UmSHI/wA2+ZkWK8tKPOd6oz/KUl9qu9lff4
+2ZePOXtD3Ip83eKcBM+Q1BfkDjT79zUFIry0usAQ5SQVjdMkDg19YPqCQJXsZElmI7yrM7Tr7+cK
+FqQwKipvXkrbGdfgQDXj8ZLc5o2/cqz39jlWNX3VQ4iqeyIUgW6cZZqfpdUleglBV5jzRoW5WdrZ
+aeoMA8OQUH8C0x4EKoAQQpIodZxzS6KMEDKna2hlcn0xYpzU1BfNs6jWFRkp4qprHuXp6uIUlUUz
+DtNYXHXJebHFHf+s3qei8A0lphyzItOo0CpIJYWb1RS08oGZIbnuR2AkGMLoz5dv4DlxcOJfMlCQ
+WFmqOsOkMGuo0dzaZRlFefHdU661xUa7jk7TJ2AQkiTLrZ74myedobikUxIFVmX/FO96JalPsucM
+MOPBFxoghBBB4fNvyicpHCHU0+CLTeQiT1MQTeJ5ViUvSKOJUrwot7hjH1S7Y5y0XJzUkM6Lcl+I
+6wty87O0GMJUFG7T0kU2qjhFmWlkNFMmzxCA8QXf7GRHULgpVZNTaV32hSKCJiIBtq3KGfVflMin
+0FAFC+zYJV038TB3+rOO4LRNshBlud0Tb3LFl+fp1AwxmnQDXpTbvfH3TrtESb6+1GTVUJPTuuYI
+cud6Ih3euCssWLVU/3RHqXrmrrk2FQ2JfGCGg0CVdFQ6pmhxqkJzfr4Jkias2br8eSkpuXouJlRt
+a+tp8g2ZaUKpoYuXpBUvSRtyKJ8j3F7tnr6BqjfI7W7wOwLc0tzRzh8hy0iUZA1D3jHHatdR2EQm
+oYdYkcAwBYXhGOYMced6Ir6YoKaJAqtyYB8trLcLkgAEqqSjsyoX3VKw6JbPJ1+QkShKfFxwdQSb
+jvftebkm5I4NeQsXF9ydob4W/5DtEV88Gpiug6hcYX5/U6DVG19XZNBeNlVPkmV3mPdFBZOatGro
+Ipsyy8QoqYmKEJ4I740KziDX6WcLLMoSu0rDEFYNta7YkKKj9UoShuuCZAOBKumwUcHbHQp+Ho1k
+SeY5MeyLd9f7jrzXKHDDzNoX8bOntrbu+HN1oss+nlrdsRZ3vCxVtSxPP9I+kix7o0KXj61xREKs
+uCRHZ9XQGIaNb5SSZSRIsiTLDInLCNX3RU91RVrdMUGUtDSeb1UihNL0TEJSNgCYCiBQJR2fI/zZ
+386e+Lgl0QWZDKIsjzQqyqalN882FaWoLlNBYQXpYEtgR61PqyA2lpmLU1TjWTZJjnKSPyaE4kKM
+F5U0WWZXIRmROG7VUKsK9AUW5TWvMwLATAKBCsxMgiQHYgIvSXbtRRWRGCcJssyQeI55xBljWUHC
+MYzEMQxD4bi0KFd3Q6nJqrmWJT+G4EVZkGSKwEgci/LS8Y7QjjpfszuqJPG1xaYyuwrD0KIc7aKc
+qTLnOgBTAQQqMANJstziib90pDfXxNy3yK74vM7Ei/KuBq83KizM0RXbhqkeCZIc56XqrrBJQ2WZ
+FEqKuH+hbVyKFOMlUZJbPbGeADcrVZVhUNAERpPYrFT1Y8tTU7QUrFgIwEggUIEZqM3DPruv550q
+V2mKKsTJ31iZRhEYQmhbrfftU+7l+XrzcItsiZJ8vC34s+2dEU68d0GKSU0pqXFrefvJ1vaqznCn
+j82zKL63ITPDoGBIfHmefnleoi8WAFMeBCow09Q7Y88d6HmnysUJ0llHpNkTW1Wgn5Wqfve0+6Oz
+nptnm9cXGwyXBKo4L+1q9L96rG91kWHTLFOmgVFcU8YEK0jOEF/bGznVGQ6zwg+uz1IzJEJoTrrm
+hjJTvkWlYXDF+MU/AJIBBCowo5xzRP96yPFBtZsTJISQKMkxTtrT6CuwKnNNzPc2ZGQZFVoF2Z9g
+wYtSkzve7olvLDMxJL6qQD8vQ6OgcDVDkKMeQhvjRWeIV9OEXklQBL6n0f9OlZsm8TwzszTPNDBv
+xQ2zTCSO0QSe2OU/AJiOIFCBmaOuN/ry0d5PznkigwYsyzJyBnlRkmelafqzGDAMnXNEdtR6a/qi
+mIwWZGsRQhiG1DShHnVHkSTLb59yH2kLeKMChtCNZeY1xQazGl+Yrcu3KAkcU9K4liEGKk+jPzIA
+YAgIVEnk0z9W7X+9josKnu7QKN9y9IPm5pN9Aiv6+yKJLv4VVPeEXz7a92mN9/+3d2fBbdz3HcB3
+sbtY3AdBgjgI3oQoUhIpiqJEKbJsN4qkqJEtR2lztVNPpmmaaR+amWQ6TR7acWeavmTGD42nVh+c
+xnKiOJZjubZsyrplS6IOUpREihd4gScuAsS9i0UfENMMSVOUCXKX4PfzJGGXf/z+mhW+/P/x3/0H
+Y/zc19NEeigQDydSaSKlZSk1S9EkyfFCvoY5mJdn08vthsdsGO8JJx+ORR9NRkcCiZQgHNli2ldp
+IAnSoKT2lhu0CkqroIoMrJalCIIwqmijCv+tALIJ/6M2kNFH/if9Ee9wyDscErvwx7sxEPrtnclL
+PdPeBdtopdPEw7HIS+8P6ZTUoZq8xhKtTkGXmpSFOjnLyGQkSaSJBC+wtCyVTj8Yi1zrD87EUtMx
+frtDs7/KYNHJ45wwHePS6XSVWWlU0pl17SRJNBRrKRmpZGSY0ANYVQgqWN+SvPBOh+cP97y3h2YC
+UX7Rc3wRfjKcjPHUiD9eY1XrFIQ3wn3iCrm8MUFI56mZL1XoGkt0BEEkOCGWFPRKyqxlivPYzKo/
+o4rZXqTdakvrlbSapWaXAprUWbizCgAeC0EF69tYKHFmKHKtLxjjhM87R0innWZlgUZu0vxxL0GW
+lmkVVIGGkdOkUcVknvUnI8nyfKVOQRtUtE5BKT7dd1DDUmu/IyIAzEJQwfpGkeTeCr2ape6Phof8
+8SS/+P5QOgXVXKZzGNnM48YdRtaqlxNpgqY+m7MjCSJfw+Rn4wkUAJBFCCpY3xxGRVNt/kggcaVv
++sZA0OWNj04n5n1TJSOJQq18i009d+vb5S9ABwBxIaggFziM7Hd2Fj6zyXh/NNw6GLo5GHIHPosr
+kiQrC5QsbrMFWJ8QVJA7bDq5TZe3t1z/cDzS0uX/uD84Hkz6IpxOQRUZWYbCEApgXUJQQa7RsNSu
+Ut12h2bAG3/zrudyb2CLVa1CTAGsWwgqyE1ySrapUPWzwyX/cqjkWn/QpGFwrxPAOoWgghwnI4mn
+KvUrbwcAxIKvlwEAQNIQVAAAIGmY+stNerPq4N/VFdfkXz7Z1X5ugPv0aeJ7jju3Hyyb8cVb3+l9
+dH2srN78jZ82x0KJq7/taj83NK8Ro0Wz57izdr/jwcXh93/ZRhCErkB56Af1pdv+ZNPbSCA22jvd
+dnZg6IFH7H4DQA5CUOUmipbl2TSWCoPawJKf3tm69xubmo87BT7dfm7I/chPEASrYqyVBi6RajxS
+sTCoqvfatv1ZsblUP/LQO7fZwjLd9bd6718a5hMpk0NbVG0q2ZpftNl0+93+9paBeIR7kkoBAB4D
+U38bReOfl+96rjLFCTff6e04PxQOxGcP0XJZybaC5hec1Jw9k4qqTdXN9jybdmFT6TThc88MtE/1
+353sOD909Tddre/0yWRk89edtfsdYncUAHINgmpDqNnnaD7mFIR065n++xeGZ3yx2UMCL0SmEzRD
+7X6hSmdSzg6/NjXbLOX6eIRLLDlCioc5z3Co48LwQNuk3qysbLTo8pVidxcAcgqCKvdt3lv01Leq
+dWZV+7mh+xeGQt7Y3KOCkJ7xxXpujlkrjQ2Hy1gVQxBEWZ3ZucsaDiYm+gNcMvXYt4gE4lNDoUSE
+M5fqbc48sXsMADkFQZXLSBnp3G3d/90aR62p46OhjvPzUyojHuau/76HS/A7vlquN6sUGnn9V0pN
+Dk3f7Ql3l2+Z7zXji4f9cU2ewlSkEbvfAJBTEFS5jCQJhZphlTRJktGZJP85OzZxiZTr3tS9c0N5
+NvWWpx1bnnaUN5i9wzM9N8amp6LLfK94OBmPcDQjkyuxQgcAsglBlcuEVLrj/PDlN7oCE5HmF6q2
+PlOsMSoWPZNLpi6d7JzxxpqPVT393RqVXnH/4shYb2D57yVX0IyC4pNCMsYv/6cAAB4LQZX72lsG
+Ws/0pQVi97Gq6r12pVa+8Jx0Kj01EGz/aEipY61VRvcj31DHVCyUXP67aPOV2jxlZDruHwuL3WMA
+yCmYpdkQLr/eqTUpdx6paHquMh5Odl8fm70FeK4rr3cVOLRak/L673smB4LLb1+upI0WlULDuB/5
+JlxP8IMAAI+FoNooLv7qocaoqH2qqOFwWdgfH3rgTQvzd22f8cde+/HlJ2qWZmSMki6rM1fssCSj
+/ED7VAAjKgDIKgTVRhGZjt843a3JU2zabQt54/7xSMiz3IUSCzEKWqmT0wxlqdRXNdm27C9S6RQ3
+3u5taxn6wm0CACwKQbWBDHZ42z4YMJhVu56rTEb59//r7hdrh6Jlh39Yf/iH9Zm/hjzRwfveM7+4
+03trXOwuAkAOIgOBQERhELuMDeHdn17uvjC4Nu9Fyki5gqZoMhlP8VyK+HSSj2ZkNEuRMpJPpLhE
+iqJlrIpO8elEdPHHT9ByimGpFP/HtXyfNvsna3DS6bSQEvikkOIFYs1953+OWGvz1/59AWA1sDIi
+/9P1XoFAgKIomUyGEVVuSguLZw/PCXPvpkrxQnTJpX18MsXPeTLF5zULALB6sDwdAAC+uGQy+W8/
++dHzz+4dHnDNvpiIx95963ff/9bX/+Nn/zwxPrrCt0BQAQDAF5dOpwf6ejvu3onH//iEtng89v47
+p18/8d+CIOx95lmDYaXP/8TUHwAAZNPlcy2n3zhpyMv75t98b2fzXoVypTsqIKgAACBrbn589Z3f
+/UZnMLzw7e/u2rtPqVKvvE1M/QEAQHbcuv7xr0+8Eo/Fvvr8C7u/9JRKnYWUIjCiAgCArHjz9V/1
+d3ePj7r/4q9f3P2lp9TqrO34g6BaO44GC4MtMLJNaVCsvBEAWLmBvt6pyXE5y+r1BjnLZrFlfG6u
+nfqvbyKITWJXAQCwKn7yr/8+2Nf3y1/85/+9/abJbN69bz+bpbjCd1QAAJAdX/nac0ePf7O/p/ut
+N3796OF9nsvO8wEwogIAgKz55ovfm5qcePu3JxUKxQ/+6celFZUy2UpHRAgqAADIGpVK/eLf/0Nw
+OnDxw7MGk+n7//ijfLN5hW1i6g8AALKp0Gr7q7/9wY7de87+4fTbp06mUqkVNoigAgCALKuu3Xrs
+L79dYC783f/+6g+n3lhha9jmAwAAvrh0Oj0d8CcTCVOBmaY/+zopkYhHwmFBENRqjVKlWmZr2OYD
+AACyjCRJY55p4essq2DZ7NzmiKk/AACQNAQVAABIGoIKAAAkDUEFAACShqACAABJQ1ABAICkIagA
+AEDSEFQAACBpCCoAAJA0BBUAAEgaTRCEjBS7CgAAAIJYNI5ogiCs2dzbHgAAIJsw9QcAAJKGoAIA
+AElDUAEAgKQhqAAAQNIQVAAAIGkIKgAAkDQEFQAASBqCCgAAJA1BBQAAkoagAgAASUNQAQCApCGo
+AABA0miCIOKJhNhlAAAAEDKZTM4w816kCYJouXJd7NoAAAAIk1G/t3H7vBcx9QcAAJKGoAIAAElD
+UAEAgKQhqAAAQNIQVAAAIGkIKgAAkDQEFQAASBqCCgAAJA1BBQAAkkaLXQA8AZ7jxtzDYlexFsxW
+m0KhFLsKAJAEBNV6EotFP7l0Xuwq1sKzh7+msEgrqGLRaDDgF7uKpeiNeUqVapUaD8+EwqGQ2F1c
+Sl5+gZxl1+CN4rHYtN8ndneXojMYVWq12FVkE4IKYFkmx0dvXLkodhVL2bXv6bJK5yo1PuTqu3/3
+tthdXMqzh79mtljX4I08k+MfX/xI7O4uZeeefRWbNotdRTbhOyoAAJA0BBUAAEgaggoAACQNQQUA
+AJKGoAIAAEnDqr+NzmQyHT16tKKiYu6LkUjE4/Fcvny5t7c384rD4Th06FBJScnCFtxu97lz5/r7
++8XuCgDkJgTVRscwjNVqdTgcly9fbmtrS6VSBEHU1dVt3ry5oKDggw8+aGtrIwiCZVmbzWY2m69e
+vXrv3r25LUSj0cnJSbH7AQA5C0EFBEEQgiCMjY09ePCA4ziCIKampliWra+v37Zt28jIiNfrzZzG
+87zb7e7o6BC7XqkrKSk5cOBANBo9c+ZMOByed9RisTQ1NZlMpra2tvb29pKSkqNHjy7aztmzZ/v6
++sTuzZNhWXb79u319fW9vb3nzy9yf3pjY+OWLVuCweCtW7c4jmtoaKisrFx4WigUeu+992avvXXK
+4XAcPHgwmUyePn164ZVgNpt37txZWFh47969O3fuFBUVHTt2bNF2Wlpauru7xe6NaBBUsIiJiYnu
+7u7KysrS0tLi4uL1/mGx9pRKZXl5uU6nc7vdV65cmXe0qqqqubmZ5/nOzk6CIIxGY1NT08TEREtL
+y7wzZ2ZmxO7KE0ulUnq9vr6+3mAwdHZ2jo+Pzzuhrq5u586dn3zyCcdxGo3G6XRu3bq1u7u7vb19
+7mmxWCwej4vdm5ViWbasrCwvL294ePjSpUvzjlZUVOzZs4cgiK6uLoIg9Hp9U1OTx+M5e/bsvDND
+0n4syGpDUMHiQqFQLBbTarVarVbsWtYfn8/X0dFx8ODBhoaGhUFlt9sNBkNra+vsF3uCIPh8vnPn
+zoldeBbwPD81NeXxeMxmc21t7bygMhqNVquV5/mBgQGfz6fRaAiCSCQSLpcrN7o/j9/vb29vP3Lk
+yM6dOxcGld1uNxqNbW1ts+NmQRACgUBO/lOsBFb9weL0er1SqQyFQhv8V7kvJhAIPHr0iOO4oqIi
+i8Uy95Ddbrfb7RzHzZ1TzTFut/vRo0darbaqqmreoW3bthUUFIyPj09NTfE8L3alqy4YDHZ2diYS
+CbvdbrVaSZKcPWSz2ex2eyqVcrvdHo9H7EolDUEFi7Db7TU1NWq12uVyDQ9/9rx2uVzudDr3z1Ff
+X6/T6cSuV3J4nvf7/W6322AwNDU1zf142rp1q91uHx8fn5iYELvM1eLxePr6+jiOs1qtdrt97qH6
++nqVStXZ2Tk1NSV2mWshlUoFAgG3263T6Xbt2kVR1Oyhmpqa4uLiycnJ0dFRscuUOgQVEARByGQy
+h8PR0NDQ2NjY2Nh44MCBsrKywcHBtrY2n++zB0UzDFNaWrpjjurqanVuPac5W4LBYHt7O0VRdXV1
+s0HFMExRUZFCoejt7Z37G0Du8fl8g4ODJpNp165dsy/m5+cXFhZGIpGenp5AICB2jWtkZmbm7t27
+MpmsoaFhNqgYhrHb7Uqlsq+vb3BwUOwapQ7fUQFBEARFUTU1NTabLZ1OEwQRjUa7urquXbs2b8lZ
+JBJpaWm5eFHSDxGXiGAwePfu3QMHDlit1oKCAo/HIwiCxWKxWq2RSGRkZGTunCpJkmq1urq6em4L
+IyMjkUhE7H58QePj4/fu3XM6neXl5TRNZ2b59uzZk5eX9/Dhw2AwOPdkmqbz8/Pndp/juOHh4cwa
+1PVuZmbm9u3bhw4dslgsBQUFY2NjmSvBZrNFo9GFV4JKpZp3Jbjd7oUrBjcUBBUQBEFwHPfhhx+e
+P38+Nz4aJCIejw8ODtbV1R06dOjUqVPxeLyhoaGgoKC/v3/et1MURdnt9hdffHHuiydOnFh3a9Nn
+RSIRt9sdiUTsdrvNZssMH2traxmGuX//vt//Jzt7qVSq7du3l5eXz77i9/tfffXVnBl1JZNJl8u1
+Y8eOL3/5y2+++WYkEqmrqyssLBwaGpo3BUpRlMVimXclvPbaa5llgRsWggpgtczMzFy6dKmurq66
+upphGJ7nHQ4HwzBz1/tl8Dzf09Pz85//XOySsykYDPb09NTV1e3bt+/kyZMqlcpisXi93sHBwWg0
+OvfMUCh04cKFt956S+ySV0s4HL5w4UJDQ8OmTZtYlk0kEkVFRXK5/M6dO7MPf8nged7lcr300kti
+lywt+I4KYLVkVl17vd6SkhKz2VxRUVFcXBwOh9fvhN4TmZqaam1tlclktbW1NE0fP35cp9MtTKmN
+gOO4/v7+ycnJ4uLigoKC8vLy4uLiaDS6Qa6ElcOICmAVJZPJrq6uwsLCZ555RiaT6XS6q1ev5vB6
+v3l993g8Pp8vLy/P6XQ6nU5BEN57772NuRSb5/muri6z2bx//36CIAwGw82bN7Heb5kwogJYRfF4
+/MKFC/F4fMuWLZs3b848jSJXb59aKBQKPXjwQKlUPv/888XFxaOjo/F4PLNgZ6PhOO6jjz6KxWI1
+NTWbN28WBGHjrNFfOYyoNjqv13vixAm5XB4MBpe4AXN4ePiVV16haXreei1YWiqVytwyVVRUJJPJ
+enp6pqenBUEQu641EgwGb968uX//fqfTSdP07du3N+C8X0bmShgfHy8tLaUoqre3d0NdCSuEEdVG
+x/O8x+MZHR0Nh8NL/KqbTCanpqbGxsYwq/6kksnktWvXUqkURVE3btzYUBNfqVTK7/e7XC6GYTK3
+E23YoCIIguO469evcxxHUdStW7ew58DyYUQFsLpSqdSNGzfGx8dpmu7r61v4ST0wMPDyyy+vx+fP
+LkcoFDp16pTBYOA4bnJyct4YYmJi4t1339VoNBthOjSdTre2tk5MTDAM43K5Ft4aNTw8/PLLL+N3
+wYUQVACrK51OBwKBJW4JytwaLHaZqyWZTC6xqWYkEnG5XGLXuHaWvhIyg06xa5QiTP0BAICkIagA
+AEDSEFQAACBpCCoAAJA0LKZYT2iattiLxK5iLchZVuwSAEAqEFTriVKlfvorXxW7CgCANYWgAlgW
+mmE0WknvZcwwzOo1LpezEu/+3M1zVxVNS/9KkItdQpaRgUDgyq12scsAAAAgTEb93sbtmT9PT0/L
+MsSuCgAAYCkIKgAAkDQEFQAASJqMJEmxawAAAPhcGFEBAIBEkSRJkiSCCgAApGh2wg9TfwAAIGkY
+UQEAgBRl5v0IBBUAAEgZSZKY+gMAACmaHVH9P2WCwF9EWBpvAAAAJXRFWHRkYXRlOmNyZWF0ZQAy
+MDIwLTAzLTMxVDExOjUwOjQ5KzAwOjAwciT11gAAACV0RVh0ZGF0ZTptb2RpZnkAMjAyMC0wMy0z
+MVQxMTo1MDo0OSswMDowMAN5TWoAAAAASUVORK5CYII=" />
+</svg>
--
2.26.0
^ permalink raw reply [relevance 1%]
* [dpdk-dev] [PATCH v1 0/8] New sync modes for ring
@ 2020-03-31 16:43 3% ` Konstantin Ananyev
2020-04-02 22:09 3% ` [dpdk-dev] [PATCH v2 0/9] " Konstantin Ananyev
1 sibling, 1 reply; 200+ results
From: Konstantin Ananyev @ 2020-03-31 16:43 UTC (permalink / raw)
To: dev; +Cc: honnappa.nagarahalli, Konstantin Ananyev
RFC - V1 changes:
1. remove ABI brekage (at least I hope I did)
2. Add support for ring_elem
3. Rework peek related API a bit
4. Rework test to make it less verbose and unite all test-cases
in one command
5. Add new test-case for MT peek API
TODO list:
1. Add C11 atomics support
2. Update docs
These days many customers use(/try to use) DPDK based apps within
overcommitted systems (multiple acttive threads over same pysical cores):
VM, container deployments, etc.
One quite common problem they hit:
Lock-Holder-Preemption/Lock-Waiter-Preemption with rte_ring.
LHP/LWP are quite a common problem for spin-based sync primitives
(spin-locks, etc.) on overcommitted systems.
The situation gets much worse when some sort of
fair-locking technique is used (ticket-lock, etc.).
As now not only lock-owner but also lock-waiters scheduling
order matters a lot.
This is a well-known problem for kernel within VMs:
http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
The problem with rte_ring is that while head accusion is sort of
un-fair locking, waiting on tail is very similar to ticket lock schema -
tail has to be updated in particular order.
That makes current rte_ring implementation to perform
really pure on some overcommited scenarios.
While it is probably not possible to completely resolve LHP problem in
userspace only (without some kernel communication/intervention),
removing fairness in tail update can mitigate it significantly.
So this RFC proposes two new optional ring synchronization modes:
1) Head/Tail Sync (HTS) mode
In that mode enqueue/dequeue operation is fully serialized:
only one thread at a time is allowed to perform given op.
As another enhancement provide ability to split enqueue/dequeue
operation into two phases:
- enqueue/dequeue start
- enqueue/dequeue finish
That allows user to inspect objects in the ring without removing
them from it (aka MT safe peek).
2) Relaxed Tail Sync (RTS)
The main difference from original MP/MC algorithm is that
tail value is increased not by every thread that finished enqueue/dequeue,
but only by the last one.
That allows threads to avoid spinning on ring tail value,
leaving actual tail value change to the last thread in the update queue.
Note that these new sync modes are optional.
For current rte_ring users nothing should change
(both in terms of API/ABI and performance).
Existing sync modes MP/MC,SP/SC kept untouched, set up in the same
way (via flags and _init_), and MP/MC remains as default one.
The only thing that changed:
Format of prod/cons now could differ depending on mode selected at _init_.
So user has to stick with one sync model through whole ring lifetime.
In other words, user can't create a ring for let say SP mode and then
in the middle of data-path change his mind and start using MP_RTS mode.
For existing modes (SP/MP, SC/MC) format remains the same and
user can still use them interchangeably, though of course it is an
error prone practice.
Test results on IA (see below) show significant improvements
for average enqueue/dequeue op times on overcommitted systems.
For 'classic' DPDK deployments (one thread per core) original MP/MC
algorithm still shows best numbers, though for 64-bit target
RTS numbers are not that far away.
Numbers were produced by new UT test-case: ring_stress_autotest, i.e.:
echo ring_stress_autotest | ./dpdk-test -n 4 --lcores='...'
X86_64 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
DEQ+ENQ average cycles/obj
MP/MC HTS RTS
1thread@1core(--lcores=6-7) 8.00 8.15 8.99
2thread@2core(--lcores=6-8) 19.14 19.61 20.35
4thread@4core(--lcores=6-10) 29.43 29.79 31.82
8thread@8core(--lcores=6-14) 110.59 192.81 119.50
16thread@16core(--lcores=6-22) 461.03 813.12 495.59
32thread/@32core(--lcores='6-22,55-70') 982.90 1972.38 1160.51
2thread@1core(--lcores='6,(10-11)@7' 20140.50 23.58 25.14
4thread@2core(--lcores='6,(10-11)@7,(20-21)@8' 153680.60 76.88 80.05
8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 280314.32 294.72 318.79
16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 643176.59 1144.02 1175.14
32thread@2core(--lcores='6,(10-25)@7,(30-45)@8' 4264238.80 4627.48 4892.68
8thread@2core(--lcores='6,(10-17)@(7,8))' 321085.98 298.59 307.47
16thread@4core(--lcores='6,(20-35)@(7-10))' 1900705.61 575.35 678.29
32thread@4core(--lcores='6,(20-51)@(7-10))' 5510445.85 2164.36 2714.12
i686 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
DEQ+ENQ average cycles/obj
MP/MC HTS RTS
1thread@1core(--lcores=6-7) 7.85 12.13 11.31
2thread@2core(--lcores=6-8) 17.89 24.52 21.86
8thread@8core(--lcores=6-14) 32.58 354.20 54.58
32thread/@32core(--lcores='6-22,55-70') 813.77 6072.41 2169.91
2thread@1core(--lcores='6,(10-11)@7' 16095.00 36.06 34.74
8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 1140354.54 346.61 361.57
16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 1920417.86 1314.90 1416.65
8thread@2core(--lcores='6,(10-17)@(7,8))' 594358.61 332.70 357.74
32thread@4core(--lcores='6,(20-51)@(7-10))' 5319896.86 2836.44 3028.87
Konstantin Ananyev (8):
test/ring: add contention stress test
ring: prepare ring to allow new sync schemes
ring: introduce RTS ring mode
test/ring: add contention stress test for RTS ring
ring: introduce HTS ring mode
test/ring: add contention stress test for HTS ring
ring: introduce peek style API
test/ring: add stress test for MT peek API
app/test/Makefile | 5 +
app/test/meson.build | 5 +
app/test/test_pdump.c | 6 +-
app/test/test_ring_hts_stress.c | 32 ++
app/test/test_ring_mpmc_stress.c | 31 ++
app/test/test_ring_peek_stress.c | 43 +++
app/test/test_ring_rts_stress.c | 32 ++
app/test/test_ring_stress.c | 57 ++++
app/test/test_ring_stress.h | 37 +++
app/test/test_ring_stress_impl.h | 436 +++++++++++++++++++++++++
lib/librte_pdump/rte_pdump.c | 2 +-
lib/librte_port/rte_port_ring.c | 12 +-
lib/librte_ring/Makefile | 9 +-
lib/librte_ring/meson.build | 9 +-
lib/librte_ring/rte_ring.c | 114 ++++++-
lib/librte_ring/rte_ring.h | 243 ++++++++++++--
lib/librte_ring/rte_ring_c11_mem.h | 44 +++
lib/librte_ring/rte_ring_elem.h | 105 +++++-
lib/librte_ring/rte_ring_generic.h | 48 +++
lib/librte_ring/rte_ring_hts.h | 210 ++++++++++++
lib/librte_ring/rte_ring_hts_elem.h | 205 ++++++++++++
lib/librte_ring/rte_ring_hts_generic.h | 235 +++++++++++++
lib/librte_ring/rte_ring_peek.h | 379 +++++++++++++++++++++
lib/librte_ring/rte_ring_rts.h | 316 ++++++++++++++++++
lib/librte_ring/rte_ring_rts_elem.h | 205 ++++++++++++
lib/librte_ring/rte_ring_rts_generic.h | 210 ++++++++++++
26 files changed, 2974 insertions(+), 56 deletions(-)
create mode 100644 app/test/test_ring_hts_stress.c
create mode 100644 app/test/test_ring_mpmc_stress.c
create mode 100644 app/test/test_ring_peek_stress.c
create mode 100644 app/test/test_ring_rts_stress.c
create mode 100644 app/test/test_ring_stress.c
create mode 100644 app/test/test_ring_stress.h
create mode 100644 app/test/test_ring_stress_impl.h
create mode 100644 lib/librte_ring/rte_ring_hts.h
create mode 100644 lib/librte_ring/rte_ring_hts_elem.h
create mode 100644 lib/librte_ring/rte_ring_hts_generic.h
create mode 100644 lib/librte_ring/rte_ring_peek.h
create mode 100644 lib/librte_ring/rte_ring_rts.h
create mode 100644 lib/librte_ring/rte_ring_rts_elem.h
create mode 100644 lib/librte_ring/rte_ring_rts_generic.h
--
2.17.1
^ permalink raw reply [relevance 3%]
* [dpdk-dev] /validate-abi.sh complains [PATCH v1 3/8] ring: introduce RTS ring mode
@ 2020-03-31 17:05 7% Ananyev, Konstantin
2020-03-31 19:22 8% ` Thomas Monjalon
2020-04-01 6:38 9% ` David Marchand
0 siblings, 2 replies; 200+ results
From: Ananyev, Konstantin @ 2020-03-31 17:05 UTC (permalink / raw)
To: david.marchand, thomas, nhorman
Cc: honnappa.nagarahalli, dev, Richardson, Bruce, Yigit, Ferruh
Hi everyone,
Have a question regarding validate-abi.sh.
It complains on the following changes with that patch:
@@ -111,11 +129,21 @@ struct rte_ring {
char pad0 __rte_cache_aligned; /**< empty cache line */
/** Ring producer status. */
- struct rte_ring_headtail prod __rte_cache_aligned;
+ RTE_STD_C11
+ union {
+ struct rte_ring_headtail prod;
+ struct rte_ring_rts_headtail rts_prod;
+ } __rte_cache_aligned;
+
char pad1 __rte_cache_aligned; /**< empty cache line */
/** Ring consumer status. */
- struct rte_ring_headtail cons __rte_cache_aligned;
+ RTE_STD_C11
+ union {
+ struct rte_ring_headtail cons;
+ struct rte_ring_rts_headtail rts_cons;
+ } __rte_cache_aligned;
+
Complaints:
rte_ring.h
[−] struct rte_ring 2
1
Change: Field cons has been removed from this type.
Effect: Applications will access incorrect memory when attempting to access this field.
2
Change: Field prod has been removed from this type.
Effect: Applications will access incorrect memory when attempting to access this field.
From my perspective it looks false-positive:
*prod* and *cons* fields are still there,
their format, size and offset within rte_ring remain the same.
Is that some limitation with the tool, or am I missing something here?
Thanks
Konstantin
> -----Original Message-----
> From: Ananyev, Konstantin <konstantin.ananyev@intel.com>
> Sent: Tuesday, March 31, 2020 5:43 PM
> To: dev@dpdk.org
> Cc: honnappa.nagarahalli@arm.com; Ananyev, Konstantin <konstantin.ananyev@intel.com>
> Subject: [PATCH v1 3/8] ring: introduce RTS ring mode
>
> Introduce relaxed tail sync (RTS) mode for MT ring synchronization.
> Aim to reduce stall times in case when ring is used on
> overcommited cpus (multiple active threads on the same cpu).
> The main difference from original MP/MC algorithm is that
> tail value is increased not by every thread that finished enqueue/dequeue,
> but only by the last one.
> That allows threads to avoid spinning on ring tail value,
> leaving actual tail value change to the last thread in the update queue.
>
> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> ---
> lib/librte_ring/Makefile | 5 +-
> lib/librte_ring/meson.build | 5 +-
> lib/librte_ring/rte_ring.c | 100 +++++++-
> lib/librte_ring/rte_ring.h | 109 ++++++++-
> lib/librte_ring/rte_ring_elem.h | 86 ++++++-
> lib/librte_ring/rte_ring_rts.h | 316 +++++++++++++++++++++++++
> lib/librte_ring/rte_ring_rts_elem.h | 205 ++++++++++++++++
> lib/librte_ring/rte_ring_rts_generic.h | 210 ++++++++++++++++
> 8 files changed, 1007 insertions(+), 29 deletions(-)
> create mode 100644 lib/librte_ring/rte_ring_rts.h
> create mode 100644 lib/librte_ring/rte_ring_rts_elem.h
> create mode 100644 lib/librte_ring/rte_ring_rts_generic.h
>
> diff --git a/lib/librte_ring/Makefile b/lib/librte_ring/Makefile
> index 917c560ad..8f5c284cc 100644
> --- a/lib/librte_ring/Makefile
> +++ b/lib/librte_ring/Makefile
> @@ -18,6 +18,9 @@ SRCS-$(CONFIG_RTE_LIBRTE_RING) := rte_ring.c
> SYMLINK-$(CONFIG_RTE_LIBRTE_RING)-include := rte_ring.h \
> rte_ring_elem.h \
> rte_ring_generic.h \
> - rte_ring_c11_mem.h
> + rte_ring_c11_mem.h \
> + rte_ring_rts.h \
> + rte_ring_rts_elem.h \
> + rte_ring_rts_generic.h
>
> include $(RTE_SDK)/mk/rte.lib.mk
> diff --git a/lib/librte_ring/meson.build b/lib/librte_ring/meson.build
> index f2f3ccc88..612936afb 100644
> --- a/lib/librte_ring/meson.build
> +++ b/lib/librte_ring/meson.build
> @@ -5,7 +5,10 @@ sources = files('rte_ring.c')
> headers = files('rte_ring.h',
> 'rte_ring_elem.h',
> 'rte_ring_c11_mem.h',
> - 'rte_ring_generic.h')
> + 'rte_ring_generic.h',
> + 'rte_ring_rts.h',
> + 'rte_ring_rts_elem.h',
> + 'rte_ring_rts_generic.h')
>
> # rte_ring_create_elem and rte_ring_get_memsize_elem are experimental
> allow_experimental_apis = true
> diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c
> index fa5733907..222eec0fb 100644
> --- a/lib/librte_ring/rte_ring.c
> +++ b/lib/librte_ring/rte_ring.c
> @@ -45,6 +45,9 @@ EAL_REGISTER_TAILQ(rte_ring_tailq)
> /* true if x is a power of 2 */
> #define POWEROF2(x) ((((x)-1) & (x)) == 0)
>
> +/* by default set head/tail distance as 1/8 of ring capacity */
> +#define HTD_MAX_DEF 8
> +
> /* return the size of memory occupied by a ring */
> ssize_t
> rte_ring_get_memsize_elem(unsigned int esize, unsigned int count)
> @@ -79,11 +82,84 @@ rte_ring_get_memsize(unsigned int count)
> return rte_ring_get_memsize_elem(sizeof(void *), count);
> }
>
> +/*
> + * internal helper function to reset prod/cons head-tail values.
> + */
> +static void
> +reset_headtail(void *p)
> +{
> + struct rte_ring_headtail *ht;
> + struct rte_ring_rts_headtail *ht_rts;
> +
> + ht = p;
> + ht_rts = p;
> +
> + switch (ht->sync_type) {
> + case RTE_RING_SYNC_MT:
> + case RTE_RING_SYNC_ST:
> + ht->head = 0;
> + ht->tail = 0;
> + break;
> + case RTE_RING_SYNC_MT_RTS:
> + ht_rts->head.raw = 0;
> + ht_rts->tail.raw = 0;
> + break;
> + default:
> + /* unknown sync mode */
> + RTE_ASSERT(0);
> + }
> +}
> +
> void
> rte_ring_reset(struct rte_ring *r)
> {
> - r->prod.head = r->cons.head = 0;
> - r->prod.tail = r->cons.tail = 0;
> + reset_headtail(&r->prod);
> + reset_headtail(&r->cons);
> +}
> +
> +/*
> + * helper function, calculates sync_type values for prod and cons
> + * based on input flags. Returns zero at success or negative
> + * errno value otherwise.
> + */
> +static int
> +get_sync_type(uint32_t flags, enum rte_ring_sync_type *prod_st,
> + enum rte_ring_sync_type *cons_st)
> +{
> + static const uint32_t prod_st_flags =
> + (RING_F_SP_ENQ | RING_F_MP_RTS_ENQ);
> + static const uint32_t cons_st_flags =
> + (RING_F_SC_DEQ | RING_F_MC_RTS_DEQ);
> +
> + switch (flags & prod_st_flags) {
> + case 0:
> + *prod_st = RTE_RING_SYNC_MT;
> + break;
> + case RING_F_SP_ENQ:
> + *prod_st = RTE_RING_SYNC_ST;
> + break;
> + case RING_F_MP_RTS_ENQ:
> + *prod_st = RTE_RING_SYNC_MT_RTS;
> + break;
> + default:
> + return -EINVAL;
> + }
> +
> + switch (flags & cons_st_flags) {
> + case 0:
> + *cons_st = RTE_RING_SYNC_MT;
> + break;
> + case RING_F_SC_DEQ:
> + *cons_st = RTE_RING_SYNC_ST;
> + break;
> + case RING_F_MC_RTS_DEQ:
> + *cons_st = RTE_RING_SYNC_MT_RTS;
> + break;
> + default:
> + return -EINVAL;
> + }
> +
> + return 0;
> }
>
> int
> @@ -100,16 +176,20 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
> RTE_BUILD_BUG_ON((offsetof(struct rte_ring, prod) &
> RTE_CACHE_LINE_MASK) != 0);
>
> + RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, sync_type) !=
> + offsetof(struct rte_ring_rts_headtail, sync_type));
> + RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, tail) !=
> + offsetof(struct rte_ring_rts_headtail, tail.val.pos));
> +
> /* init the ring structure */
> memset(r, 0, sizeof(*r));
> ret = strlcpy(r->name, name, sizeof(r->name));
> if (ret < 0 || ret >= (int)sizeof(r->name))
> return -ENAMETOOLONG;
> r->flags = flags;
> - r->prod.sync_type = (flags & RING_F_SP_ENQ) ?
> - RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
> - r->cons.sync_type = (flags & RING_F_SC_DEQ) ?
> - RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
> + ret = get_sync_type(flags, &r->prod.sync_type, &r->cons.sync_type);
> + if (ret != 0)
> + return ret;
>
> if (flags & RING_F_EXACT_SZ) {
> r->size = rte_align32pow2(count + 1);
> @@ -126,8 +206,12 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
> r->mask = count - 1;
> r->capacity = r->mask;
> }
> - r->prod.head = r->cons.head = 0;
> - r->prod.tail = r->cons.tail = 0;
> +
> + /* set default values for head-tail distance */
> + if (flags & RING_F_MP_RTS_ENQ)
> + rte_ring_set_prod_htd_max(r, r->capacity / HTD_MAX_DEF);
> + if (flags & RING_F_MC_RTS_DEQ)
> + rte_ring_set_cons_htd_max(r, r->capacity / HTD_MAX_DEF);
>
> return 0;
> }
> diff --git a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h
> index d4775a063..2b42a0211 100644
> --- a/lib/librte_ring/rte_ring.h
> +++ b/lib/librte_ring/rte_ring.h
> @@ -65,10 +65,13 @@ enum rte_ring_queue_behavior {
> enum rte_ring_sync_type {
> RTE_RING_SYNC_MT, /**< multi-thread safe (default mode) */
> RTE_RING_SYNC_ST, /**< single thread only */
> +#ifdef ALLOW_EXPERIMENTAL_API
> + RTE_RING_SYNC_MT_RTS, /**< multi-thread relaxed tail sync */
> +#endif
> };
>
> /**
> - * structure to hold a pair of head/tail values and other metadata.
> + * structures to hold a pair of head/tail values and other metadata.
> * Depending on sync_type format of that structure might be different,
> * but offset for *sync_type* and *tail* values should remain the same.
> */
> @@ -84,6 +87,21 @@ struct rte_ring_headtail {
> };
> };
>
> +union rte_ring_ht_poscnt {
> + uint64_t raw;
> + struct {
> + uint32_t cnt; /**< head/tail reference counter */
> + uint32_t pos; /**< head/tail position */
> + } val;
> +};
> +
> +struct rte_ring_rts_headtail {
> + volatile union rte_ring_ht_poscnt tail;
> + enum rte_ring_sync_type sync_type; /**< sync type of prod/cons */
> + uint32_t htd_max; /**< max allowed distance between head/tail */
> + volatile union rte_ring_ht_poscnt head;
> +};
> +
> /**
> * An RTE ring structure.
> *
> @@ -111,11 +129,21 @@ struct rte_ring {
> char pad0 __rte_cache_aligned; /**< empty cache line */
>
> /** Ring producer status. */
> - struct rte_ring_headtail prod __rte_cache_aligned;
> + RTE_STD_C11
> + union {
> + struct rte_ring_headtail prod;
> + struct rte_ring_rts_headtail rts_prod;
> + } __rte_cache_aligned;
> +
> char pad1 __rte_cache_aligned; /**< empty cache line */
>
> /** Ring consumer status. */
> - struct rte_ring_headtail cons __rte_cache_aligned;
> + RTE_STD_C11
> + union {
> + struct rte_ring_headtail cons;
> + struct rte_ring_rts_headtail rts_cons;
> + } __rte_cache_aligned;
> +
> char pad2 __rte_cache_aligned; /**< empty cache line */
> };
>
> @@ -132,6 +160,9 @@ struct rte_ring {
> #define RING_F_EXACT_SZ 0x0004
> #define RTE_RING_SZ_MASK (0x7fffffffU) /**< Ring size mask */
>
> +#define RING_F_MP_RTS_ENQ 0x0008 /**< The default enqueue is "MP RTS". */
> +#define RING_F_MC_RTS_DEQ 0x0010 /**< The default dequeue is "MC RTS". */
> +
> #define __IS_SP RTE_RING_SYNC_ST
> #define __IS_MP RTE_RING_SYNC_MT
> #define __IS_SC RTE_RING_SYNC_ST
> @@ -461,6 +492,10 @@ rte_ring_sp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
> RTE_RING_SYNC_ST, free_space);
> }
>
> +#ifdef ALLOW_EXPERIMENTAL_API
> +#include <rte_ring_rts.h>
> +#endif
> +
> /**
> * Enqueue several objects on a ring.
> *
> @@ -484,8 +519,21 @@ static __rte_always_inline unsigned int
> rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
> unsigned int n, unsigned int *free_space)
> {
> - return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
> - r->prod.sync_type, free_space);
> + switch (r->prod.sync_type) {
> + case RTE_RING_SYNC_MT:
> + return rte_ring_mp_enqueue_bulk(r, obj_table, n, free_space);
> + case RTE_RING_SYNC_ST:
> + return rte_ring_sp_enqueue_bulk(r, obj_table, n, free_space);
> +#ifdef ALLOW_EXPERIMENTAL_API
> + case RTE_RING_SYNC_MT_RTS:
> + return rte_ring_mp_rts_enqueue_bulk(r, obj_table, n,
> + free_space);
> +#endif
> + }
> +
> + /* valid ring should never reach this point */
> + RTE_ASSERT(0);
> + return 0;
> }
>
> /**
> @@ -619,8 +667,20 @@ static __rte_always_inline unsigned int
> rte_ring_dequeue_bulk(struct rte_ring *r, void **obj_table, unsigned int n,
> unsigned int *available)
> {
> - return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
> - r->cons.sync_type, available);
> + switch (r->cons.sync_type) {
> + case RTE_RING_SYNC_MT:
> + return rte_ring_mc_dequeue_bulk(r, obj_table, n, available);
> + case RTE_RING_SYNC_ST:
> + return rte_ring_sc_dequeue_bulk(r, obj_table, n, available);
> +#ifdef ALLOW_EXPERIMENTAL_API
> + case RTE_RING_SYNC_MT_RTS:
> + return rte_ring_mc_rts_dequeue_bulk(r, obj_table, n, available);
> +#endif
> + }
> +
> + /* valid ring should never reach this point */
> + RTE_ASSERT(0);
> + return 0;
> }
>
> /**
> @@ -940,8 +1000,21 @@ static __rte_always_inline unsigned
> rte_ring_enqueue_burst(struct rte_ring *r, void * const *obj_table,
> unsigned int n, unsigned int *free_space)
> {
> - return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_VARIABLE,
> - r->prod.sync_type, free_space);
> + switch (r->prod.sync_type) {
> + case RTE_RING_SYNC_MT:
> + return rte_ring_mp_enqueue_burst(r, obj_table, n, free_space);
> + case RTE_RING_SYNC_ST:
> + return rte_ring_sp_enqueue_burst(r, obj_table, n, free_space);
> +#ifdef ALLOW_EXPERIMENTAL_API
> + case RTE_RING_SYNC_MT_RTS:
> + return rte_ring_mp_rts_enqueue_burst(r, obj_table, n,
> + free_space);
> +#endif
> + }
> +
> + /* valid ring should never reach this point */
> + RTE_ASSERT(0);
> + return 0;
> }
>
> /**
> @@ -1020,9 +1093,21 @@ static __rte_always_inline unsigned
> rte_ring_dequeue_burst(struct rte_ring *r, void **obj_table,
> unsigned int n, unsigned int *available)
> {
> - return __rte_ring_do_dequeue(r, obj_table, n,
> - RTE_RING_QUEUE_VARIABLE,
> - r->cons.sync_type, available);
> + switch (r->cons.sync_type) {
> + case RTE_RING_SYNC_MT:
> + return rte_ring_mc_dequeue_burst(r, obj_table, n, available);
> + case RTE_RING_SYNC_ST:
> + return rte_ring_sc_dequeue_burst(r, obj_table, n, available);
> +#ifdef ALLOW_EXPERIMENTAL_API
> + case RTE_RING_SYNC_MT_RTS:
> + return rte_ring_mc_rts_dequeue_burst(r, obj_table, n,
> + available);
> +#endif
> + }
> +
> + /* valid ring should never reach this point */
> + RTE_ASSERT(0);
> + return 0;
> }
>
> #ifdef __cplusplus
> diff --git a/lib/librte_ring/rte_ring_elem.h b/lib/librte_ring/rte_ring_elem.h
> index 28f9836e6..5de0850dc 100644
> --- a/lib/librte_ring/rte_ring_elem.h
> +++ b/lib/librte_ring/rte_ring_elem.h
> @@ -542,6 +542,8 @@ rte_ring_sp_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
> RTE_RING_QUEUE_FIXED, __IS_SP, free_space);
> }
>
> +#include <rte_ring_rts_elem.h>
> +
> /**
> * Enqueue several objects on a ring.
> *
> @@ -571,6 +573,26 @@ rte_ring_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
> {
> return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
> RTE_RING_QUEUE_FIXED, r->prod.sync_type, free_space);
> +
> + switch (r->prod.sync_type) {
> + case RTE_RING_SYNC_MT:
> + return rte_ring_mp_enqueue_bulk_elem(r, obj_table, esize, n,
> + free_space);
> + case RTE_RING_SYNC_ST:
> + return rte_ring_sp_enqueue_bulk_elem(r, obj_table, esize, n,
> + free_space);
> +#ifdef ALLOW_EXPERIMENTAL_API
> + case RTE_RING_SYNC_MT_RTS:
> + return rte_ring_mp_rts_enqueue_bulk_elem(r, obj_table, esize, n,
> + free_space);
> +#endif
> + }
> +
> + /* valid ring should never reach this point */
> + RTE_ASSERT(0);
> + if (free_space != NULL)
> + *free_space = 0;
> + return 0;
> }
>
> /**
> @@ -733,8 +755,25 @@ static __rte_always_inline unsigned int
> rte_ring_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
> unsigned int esize, unsigned int n, unsigned int *available)
> {
> - return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
> - RTE_RING_QUEUE_FIXED, r->cons.sync_type, available);
> + switch (r->cons.sync_type) {
> + case RTE_RING_SYNC_MT:
> + return rte_ring_mc_dequeue_bulk_elem(r, obj_table, esize, n,
> + available);
> + case RTE_RING_SYNC_ST:
> + return rte_ring_sc_dequeue_bulk_elem(r, obj_table, esize, n,
> + available);
> +#ifdef ALLOW_EXPERIMENTAL_API
> + case RTE_RING_SYNC_MT_RTS:
> + return rte_ring_mc_rts_dequeue_bulk_elem(r, obj_table, esize,
> + n, available);
> +#endif
> + }
> +
> + /* valid ring should never reach this point */
> + RTE_ASSERT(0);
> + if (available != NULL)
> + *available = 0;
> + return 0;
> }
>
> /**
> @@ -901,8 +940,25 @@ static __rte_always_inline unsigned
> rte_ring_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
> unsigned int esize, unsigned int n, unsigned int *free_space)
> {
> - return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
> - RTE_RING_QUEUE_VARIABLE, r->prod.sync_type, free_space);
> + switch (r->prod.sync_type) {
> + case RTE_RING_SYNC_MT:
> + return rte_ring_mp_enqueue_burst_elem(r, obj_table, esize, n,
> + free_space);
> + case RTE_RING_SYNC_ST:
> + return rte_ring_sp_enqueue_burst_elem(r, obj_table, esize, n,
> + free_space);
> +#ifdef ALLOW_EXPERIMENTAL_API
> + case RTE_RING_SYNC_MT_RTS:
> + return rte_ring_mp_rts_enqueue_burst_elem(r, obj_table, esize,
> + n, free_space);
> +#endif
> + }
> +
> + /* valid ring should never reach this point */
> + RTE_ASSERT(0);
> + if (free_space != NULL)
> + *free_space = 0;
> + return 0;
> }
>
> /**
> @@ -993,9 +1049,25 @@ static __rte_always_inline unsigned int
> rte_ring_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
> unsigned int esize, unsigned int n, unsigned int *available)
> {
> - return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
> - RTE_RING_QUEUE_VARIABLE,
> - r->cons.sync_type, available);
> + switch (r->cons.sync_type) {
> + case RTE_RING_SYNC_MT:
> + return rte_ring_mc_dequeue_burst_elem(r, obj_table, esize, n,
> + available);
> + case RTE_RING_SYNC_ST:
> + return rte_ring_sc_dequeue_burst_elem(r, obj_table, esize, n,
> + available);
> +#ifdef ALLOW_EXPERIMENTAL_API
> + case RTE_RING_SYNC_MT_RTS:
> + return rte_ring_mc_rts_dequeue_burst_elem(r, obj_table, esize,
> + n, available);
> +#endif
> + }
> +
> + /* valid ring should never reach this point */
> + RTE_ASSERT(0);
> + if (available != NULL)
> + *available = 0;
> + return 0;
> }
>
> #ifdef __cplusplus
> diff --git a/lib/librte_ring/rte_ring_rts.h b/lib/librte_ring/rte_ring_rts.h
> new file mode 100644
> index 000000000..18404fe48
> --- /dev/null
> +++ b/lib/librte_ring/rte_ring_rts.h
> @@ -0,0 +1,316 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + *
> + * Copyright (c) 2010-2017 Intel Corporation
> + * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
> + * All rights reserved.
> + * Derived from FreeBSD's bufring.h
> + * Used as BSD-3 Licensed with permission from Kip Macy.
> + */
> +
> +#ifndef _RTE_RING_RTS_H_
> +#define _RTE_RING_RTS_H_
> +
> +/**
> + * @file rte_ring_rts.h
> + * @b EXPERIMENTAL: this API may change without prior notice
> + * It is not recommended to include this file directly.
> + * Please include <rte_ring.h> instead.
> + *
> + * Contains functions for Relaxed Tail Sync (RTS) ring mode.
> + * The main idea remains the same as for our original MP/MC synchronization
> + * mechanism.
> + * The main difference is that tail value is increased not
> + * by every thread that finished enqueue/dequeue,
> + * but only by the last one doing enqueue/dequeue.
> + * That allows threads to skip spinning on tail value,
> + * leaving actual tail value change to last thread in the update queue.
> + * RTS requires 2 64-bit CAS for each enqueue(/dequeue) operation:
> + * one for head update, second for tail update.
> + * As a gain it allows thread to avoid spinning/waiting on tail value.
> + * In comparision original MP/MC algorithm requires one 32-bit CAS
> + * for head update and waiting/spinning on tail value.
> + *
> + * Brief outline:
> + * - introduce refcnt for both head and tail.
> + * - increment head.refcnt for each head.value update
> + * - write head:value and head:refcnt atomically (64-bit CAS)
> + * - move tail.value ahead only when tail.refcnt + 1 == head.refcnt
> + * - increment tail.refcnt when each enqueue/dequeue op finishes
> + * (no matter is tail:value going to change or not)
> + * - write tail.value and tail.recnt atomically (64-bit CAS)
> + *
> + * To avoid producer/consumer starvation:
> + * - limit max allowed distance between head and tail value (HTD_MAX).
> + * I.E. thread is allowed to proceed with changing head.value,
> + * only when: head.value - tail.value <= HTD_MAX
> + * HTD_MAX is an optional parameter.
> + * With HTD_MAX == 0 we'll have fully serialized ring -
> + * i.e. only one thread at a time will be able to enqueue/dequeue
> + * to/from the ring.
> + * With HTD_MAX >= ring.capacity - no limitation.
> + * By default HTD_MAX == ring.capacity / 8.
> + */
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +#include <rte_ring_rts_generic.h>
> +
> +/**
> + * @internal Enqueue several objects on the RTS ring.
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects).
> + * @param n
> + * The number of objects to add in the ring from the obj_table.
> + * @param behavior
> + * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
> + * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
> + * @param free_space
> + * returns the amount of space after the enqueue operation has finished
> + * @return
> + * Actual number of objects enqueued.
> + * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> + */
> +static __rte_always_inline unsigned int
> +__rte_ring_do_rts_enqueue(struct rte_ring *r, void * const *obj_table,
> + uint32_t n, enum rte_ring_queue_behavior behavior,
> + uint32_t *free_space)
> +{
> + uint32_t free, head;
> +
> + n = __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
> +
> + if (n != 0) {
> + ENQUEUE_PTRS(r, &r[1], head, obj_table, n, void *);
> + __rte_ring_rts_update_tail(&r->rts_prod);
> + }
> +
> + if (free_space != NULL)
> + *free_space = free - n;
> + return n;
> +}
> +
> +/**
> + * @internal Dequeue several objects from the RTS ring.
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects).
> + * @param n
> + * The number of objects to pull from the ring.
> + * @param behavior
> + * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
> + * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
> + * @param available
> + * returns the number of remaining ring entries after the dequeue has finished
> + * @return
> + * - Actual number of objects dequeued.
> + * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> + */
> +static __rte_always_inline unsigned int
> +__rte_ring_do_rts_dequeue(struct rte_ring *r, void **obj_table,
> + uint32_t n, enum rte_ring_queue_behavior behavior,
> + uint32_t *available)
> +{
> + uint32_t entries, head;
> +
> + n = __rte_ring_rts_move_cons_head(r, n, behavior, &head, &entries);
> +
> + if (n != 0) {
> + DEQUEUE_PTRS(r, &r[1], head, obj_table, n, void *);
> + __rte_ring_rts_update_tail(&r->rts_cons);
> + }
> +
> + if (available != NULL)
> + *available = entries - n;
> + return n;
> +}
> +
> +/**
> + * Enqueue several objects on the RTS ring (multi-producers safe).
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects).
> + * @param n
> + * The number of objects to add in the ring from the obj_table.
> + * @param free_space
> + * if non-NULL, returns the amount of space in the ring after the
> + * enqueue operation has finished.
> + * @return
> + * The number of objects enqueued, either 0 or n
> + */
> +__rte_experimental
> +static __rte_always_inline unsigned int
> +rte_ring_mp_rts_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
> + unsigned int n, unsigned int *free_space)
> +{
> + return __rte_ring_do_rts_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
> + free_space);
> +}
> +
> +/**
> + * Dequeue several objects from an RTS ring (multi-consumers safe).
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects) that will be filled.
> + * @param n
> + * The number of objects to dequeue from the ring to the obj_table.
> + * @param available
> + * If non-NULL, returns the number of remaining ring entries after the
> + * dequeue has finished.
> + * @return
> + * The number of objects dequeued, either 0 or n
> + */
> +__rte_experimental
> +static __rte_always_inline unsigned int
> +rte_ring_mc_rts_dequeue_bulk(struct rte_ring *r, void **obj_table,
> + unsigned int n, unsigned int *available)
> +{
> + return __rte_ring_do_rts_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
> + available);
> +}
> +
> +/**
> + * Return producer max Head-Tail-Distance (HTD).
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @return
> + * Producer HTD value, if producer is set in appropriate sync mode,
> + * or UINT32_MAX otherwise.
> + */
> +__rte_experimental
> +static inline uint32_t
> +rte_ring_get_prod_htd_max(const struct rte_ring *r)
> +{
> + if (r->prod.sync_type == RTE_RING_SYNC_MT_RTS)
> + return r->rts_prod.htd_max;
> + return UINT32_MAX;
> +}
> +
> +/**
> + * Set producer max Head-Tail-Distance (HTD).
> + * Note that producer has to use appropriate sync mode (RTS).
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param v
> + * new HTD value to setup.
> + * @return
> + * Zero on success, or negative error code otherwise.
> + */
> +__rte_experimental
> +static inline int
> +rte_ring_set_prod_htd_max(struct rte_ring *r, uint32_t v)
> +{
> + if (r->prod.sync_type != RTE_RING_SYNC_MT_RTS)
> + return -ENOTSUP;
> +
> + r->rts_prod.htd_max = v;
> + return 0;
> +}
> +
> +/**
> + * Return consumer max Head-Tail-Distance (HTD).
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @return
> + * Consumer HTD value, if consumer is set in appropriate sync mode,
> + * or UINT32_MAX otherwise.
> + */
> +__rte_experimental
> +static inline uint32_t
> +rte_ring_get_cons_htd_max(const struct rte_ring *r)
> +{
> + if (r->cons.sync_type == RTE_RING_SYNC_MT_RTS)
> + return r->rts_cons.htd_max;
> + return UINT32_MAX;
> +}
> +
> +/**
> + * Set consumer max Head-Tail-Distance (HTD).
> + * Note that consumer has to use appropriate sync mode (RTS).
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param v
> + * new HTD value to setup.
> + * @return
> + * Zero on success, or negative error code otherwise.
> + */
> +__rte_experimental
> +static inline int
> +rte_ring_set_cons_htd_max(struct rte_ring *r, uint32_t v)
> +{
> + if (r->cons.sync_type != RTE_RING_SYNC_MT_RTS)
> + return -ENOTSUP;
> +
> + r->rts_cons.htd_max = v;
> + return 0;
> +}
> +
> +/**
> + * Enqueue several objects on the RTS ring (multi-producers safe).
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects).
> + * @param n
> + * The number of objects to add in the ring from the obj_table.
> + * @param free_space
> + * if non-NULL, returns the amount of space in the ring after the
> + * enqueue operation has finished.
> + * @return
> + * - n: Actual number of objects enqueued.
> + */
> +__rte_experimental
> +static __rte_always_inline unsigned
> +rte_ring_mp_rts_enqueue_burst(struct rte_ring *r, void * const *obj_table,
> + unsigned int n, unsigned int *free_space)
> +{
> + return __rte_ring_do_rts_enqueue(r, obj_table, n,
> + RTE_RING_QUEUE_VARIABLE, free_space);
> +}
> +
> +/**
> + * Dequeue several objects from an RTS ring (multi-consumers safe).
> + * When the requested objects are more than the available objects,
> + * only dequeue the actual number of objects.
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects) that will be filled.
> + * @param n
> + * The number of objects to dequeue from the ring to the obj_table.
> + * @param available
> + * If non-NULL, returns the number of remaining ring entries after the
> + * dequeue has finished.
> + * @return
> + * - n: Actual number of objects dequeued, 0 if ring is empty
> + */
> +__rte_experimental
> +static __rte_always_inline unsigned
> +rte_ring_mc_rts_dequeue_burst(struct rte_ring *r, void **obj_table,
> + unsigned int n, unsigned int *available)
> +{
> + return __rte_ring_do_rts_dequeue(r, obj_table, n,
> + RTE_RING_QUEUE_VARIABLE, available);
> +}
> +
> +#ifdef __cplusplus
> +}
> +#endif
> +
> +#endif /* _RTE_RING_RTS_H_ */
> diff --git a/lib/librte_ring/rte_ring_rts_elem.h b/lib/librte_ring/rte_ring_rts_elem.h
> new file mode 100644
> index 000000000..71a331b23
> --- /dev/null
> +++ b/lib/librte_ring/rte_ring_rts_elem.h
> @@ -0,0 +1,205 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + *
> + * Copyright (c) 2010-2017 Intel Corporation
> + * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
> + * All rights reserved.
> + * Derived from FreeBSD's bufring.h
> + * Used as BSD-3 Licensed with permission from Kip Macy.
> + */
> +
> +#ifndef _RTE_RING_RTS_ELEM_H_
> +#define _RTE_RING_RTS_ELEM_H_
> +
> +/**
> + * @file rte_ring_rts_elem.h
> + * @b EXPERIMENTAL: this API may change without prior notice
> + *
> + * It is not recommended to include this file directly.
> + * Please include <rte_ring_elem.h> instead.
> + * Contains *ring_elem* functions for Relaxed Tail Sync (RTS) ring mode.
> + * for more details please refer to <rte_ring_rts.h>.
> + */
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +#include <rte_ring_rts_generic.h>
> +
> +/**
> + * @internal Enqueue several objects on the RTS ring.
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects).
> + * @param n
> + * The number of objects to add in the ring from the obj_table.
> + * @param behavior
> + * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
> + * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
> + * @param free_space
> + * returns the amount of space after the enqueue operation has finished
> + * @return
> + * Actual number of objects enqueued.
> + * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> + */
> +static __rte_always_inline unsigned int
> +__rte_ring_do_rts_enqueue_elem(struct rte_ring *r, void * const *obj_table,
> + uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
> + uint32_t *free_space)
> +{
> + uint32_t free, head;
> +
> + n = __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
> +
> + if (n != 0) {
> + __rte_ring_enqueue_elems(r, head, obj_table, esize, n);
> + __rte_ring_rts_update_tail(&r->rts_prod);
> + }
> +
> + if (free_space != NULL)
> + *free_space = free - n;
> + return n;
> +}
> +
> +/**
> + * @internal Dequeue several objects from the RTS ring.
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects).
> + * @param n
> + * The number of objects to pull from the ring.
> + * @param behavior
> + * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
> + * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
> + * @param available
> + * returns the number of remaining ring entries after the dequeue has finished
> + * @return
> + * - Actual number of objects dequeued.
> + * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> + */
> +static __rte_always_inline unsigned int
> +__rte_ring_do_rts_dequeue_elem(struct rte_ring *r, void **obj_table,
> + uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
> + uint32_t *available)
> +{
> + uint32_t entries, head;
> +
> + n = __rte_ring_rts_move_cons_head(r, n, behavior, &head, &entries);
> +
> + if (n != 0) {
> + __rte_ring_dequeue_elems(r, head, obj_table, esize, n);
> + __rte_ring_rts_update_tail(&r->rts_cons);
> + }
> +
> + if (available != NULL)
> + *available = entries - n;
> + return n;
> +}
> +
> +/**
> + * Enqueue several objects on the RTS ring (multi-producers safe).
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects).
> + * @param n
> + * The number of objects to add in the ring from the obj_table.
> + * @param free_space
> + * if non-NULL, returns the amount of space in the ring after the
> + * enqueue operation has finished.
> + * @return
> + * The number of objects enqueued, either 0 or n
> + */
> +__rte_experimental
> +static __rte_always_inline unsigned int
> +rte_ring_mp_rts_enqueue_bulk_elem(struct rte_ring *r, void * const *obj_table,
> + unsigned int esize, unsigned int n, unsigned int *free_space)
> +{
> + return __rte_ring_do_rts_enqueue_elem(r, obj_table, esize, n,
> + RTE_RING_QUEUE_FIXED, free_space);
> +}
> +
> +/**
> + * Dequeue several objects from an RTS ring (multi-consumers safe).
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects) that will be filled.
> + * @param n
> + * The number of objects to dequeue from the ring to the obj_table.
> + * @param available
> + * If non-NULL, returns the number of remaining ring entries after the
> + * dequeue has finished.
> + * @return
> + * The number of objects dequeued, either 0 or n
> + */
> +__rte_experimental
> +static __rte_always_inline unsigned int
> +rte_ring_mc_rts_dequeue_bulk_elem(struct rte_ring *r, void **obj_table,
> + unsigned int esize, unsigned int n, unsigned int *available)
> +{
> + return __rte_ring_do_rts_dequeue_elem(r, obj_table, esize, n,
> + RTE_RING_QUEUE_FIXED, available);
> +}
> +
> +/**
> + * Enqueue several objects on the RTS ring (multi-producers safe).
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects).
> + * @param n
> + * The number of objects to add in the ring from the obj_table.
> + * @param free_space
> + * if non-NULL, returns the amount of space in the ring after the
> + * enqueue operation has finished.
> + * @return
> + * - n: Actual number of objects enqueued.
> + */
> +__rte_experimental
> +static __rte_always_inline unsigned
> +rte_ring_mp_rts_enqueue_burst_elem(struct rte_ring *r, void * const *obj_table,
> + unsigned int esize, unsigned int n, unsigned int *free_space)
> +{
> + return __rte_ring_do_rts_enqueue_elem(r, obj_table, esize, n,
> + RTE_RING_QUEUE_VARIABLE, free_space);
> +}
> +
> +/**
> + * Dequeue several objects from an RTS ring (multi-consumers safe).
> + * When the requested objects are more than the available objects,
> + * only dequeue the actual number of objects.
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects) that will be filled.
> + * @param n
> + * The number of objects to dequeue from the ring to the obj_table.
> + * @param available
> + * If non-NULL, returns the number of remaining ring entries after the
> + * dequeue has finished.
> + * @return
> + * - n: Actual number of objects dequeued, 0 if ring is empty
> + */
> +__rte_experimental
> +static __rte_always_inline unsigned
> +rte_ring_mc_rts_dequeue_burst_elem(struct rte_ring *r, void **obj_table,
> + unsigned int esize, unsigned int n, unsigned int *available)
> +{
> + return __rte_ring_do_rts_dequeue_elem(r, obj_table, esize, n,
> + RTE_RING_QUEUE_VARIABLE, available);
> +}
> +
> +#ifdef __cplusplus
> +}
> +#endif
> +
> +#endif /* _RTE_RING_RTS_ELEM_H_ */
> diff --git a/lib/librte_ring/rte_ring_rts_generic.h b/lib/librte_ring/rte_ring_rts_generic.h
> new file mode 100644
> index 000000000..31a37924c
> --- /dev/null
> +++ b/lib/librte_ring/rte_ring_rts_generic.h
> @@ -0,0 +1,210 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + *
> + * Copyright (c) 2010-2017 Intel Corporation
> + * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
> + * All rights reserved.
> + * Derived from FreeBSD's bufring.h
> + * Used as BSD-3 Licensed with permission from Kip Macy.
> + */
> +
> +#ifndef _RTE_RING_RTS_GENERIC_H_
> +#define _RTE_RING_RTS_GENERIC_H_
> +
> +/**
> + * @file rte_ring_rts_generic.h
> + * It is not recommended to include this file directly,
> + * include <rte_ring.h> instead.
> + * Contains internal helper functions for Relaxed Tail Sync (RTS) ring mode.
> + * For more information please refer to <rte_ring_rts.h>.
> + */
> +
> +/**
> + * @internal This function updates tail values.
> + */
> +static __rte_always_inline void
> +__rte_ring_rts_update_tail(struct rte_ring_rts_headtail *ht)
> +{
> + union rte_ring_ht_poscnt h, ot, nt;
> +
> + /*
> + * If there are other enqueues/dequeues in progress that
> + * might preceded us, then don't update tail with new value.
> + */
> +
> + do {
> + ot.raw = ht->tail.raw;
> + rte_smp_rmb();
> +
> + /* on 32-bit systems we have to do atomic read here */
> + h.raw = rte_atomic64_read((rte_atomic64_t *)
> + (uintptr_t)&ht->head.raw);
> +
> + nt.raw = ot.raw;
> + if (++nt.val.cnt == h.val.cnt)
> + nt.val.pos = h.val.pos;
> +
> + } while (rte_atomic64_cmpset(&ht->tail.raw, ot.raw, nt.raw) == 0);
> +}
> +
> +/**
> + * @internal This function waits till head/tail distance wouldn't
> + * exceed pre-defined max value.
> + */
> +static __rte_always_inline void
> +__rte_ring_rts_head_wait(const struct rte_ring_rts_headtail *ht,
> + union rte_ring_ht_poscnt *h)
> +{
> + uint32_t max;
> +
> + max = ht->htd_max;
> + h->raw = ht->head.raw;
> + rte_smp_rmb();
> +
> + while (h->val.pos - ht->tail.val.pos > max) {
> + rte_pause();
> + h->raw = ht->head.raw;
> + rte_smp_rmb();
> + }
> +}
> +
> +/**
> + * @internal This function updates the producer head for enqueue.
> + *
> + * @param r
> + * A pointer to the ring structure
> + * @param is_sp
> + * Indicates whether multi-producer path is needed or not
> + * @param n
> + * The number of elements we will want to enqueue, i.e. how far should the
> + * head be moved
> + * @param behavior
> + * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
> + * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
> + * @param old_head
> + * Returns head value as it was before the move, i.e. where enqueue starts
> + * @param new_head
> + * Returns the current/new head value i.e. where enqueue finishes
> + * @param free_entries
> + * Returns the amount of free space in the ring BEFORE head was moved
> + * @return
> + * Actual number of objects enqueued.
> + * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> + */
> +static __rte_always_inline uint32_t
> +__rte_ring_rts_move_prod_head(struct rte_ring *r, uint32_t num,
> + enum rte_ring_queue_behavior behavior, uint32_t *old_head,
> + uint32_t *free_entries)
> +{
> + uint32_t n;
> + union rte_ring_ht_poscnt nh, oh;
> +
> + const uint32_t capacity = r->capacity;
> +
> + do {
> + /* Reset n to the initial burst count */
> + n = num;
> +
> + /* read prod head (may spin on prod tail) */
> + __rte_ring_rts_head_wait(&r->rts_prod, &oh);
> +
> + /* add rmb barrier to avoid load/load reorder in weak
> + * memory model. It is noop on x86
> + */
> + rte_smp_rmb();
> +
> + /*
> + * The subtraction is done between two unsigned 32bits value
> + * (the result is always modulo 32 bits even if we have
> + * *old_head > cons_tail). So 'free_entries' is always between 0
> + * and capacity (which is < size).
> + */
> + *free_entries = capacity + r->cons.tail - oh.val.pos;
> +
> + /* check that we have enough room in ring */
> + if (unlikely(n > *free_entries))
> + n = (behavior == RTE_RING_QUEUE_FIXED) ?
> + 0 : *free_entries;
> +
> + if (n == 0)
> + return 0;
> +
> + nh.val.pos = oh.val.pos + n;
> + nh.val.cnt = oh.val.cnt + 1;
> +
> + } while (rte_atomic64_cmpset(&r->rts_prod.head.raw,
> + oh.raw, nh.raw) == 0);
> +
> + *old_head = oh.val.pos;
> + return n;
> +}
> +
> +/**
> + * @internal This function updates the consumer head for dequeue
> + *
> + * @param r
> + * A pointer to the ring structure
> + * @param is_sc
> + * Indicates whether multi-consumer path is needed or not
> + * @param n
> + * The number of elements we will want to enqueue, i.e. how far should the
> + * head be moved
> + * @param behavior
> + * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
> + * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
> + * @param old_head
> + * Returns head value as it was before the move, i.e. where dequeue starts
> + * @param new_head
> + * Returns the current/new head value i.e. where dequeue finishes
> + * @param entries
> + * Returns the number of entries in the ring BEFORE head was moved
> + * @return
> + * - Actual number of objects dequeued.
> + * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> + */
> +static __rte_always_inline unsigned int
> +__rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num,
> + enum rte_ring_queue_behavior behavior, uint32_t *old_head,
> + uint32_t *entries)
> +{
> + uint32_t n;
> + union rte_ring_ht_poscnt nh, oh;
> +
> + /* move cons.head atomically */
> + do {
> + /* Restore n as it may change every loop */
> + n = num;
> +
> + /* read cons head (may spin on cons tail) */
> + __rte_ring_rts_head_wait(&r->rts_cons, &oh);
> +
> +
> + /* add rmb barrier to avoid load/load reorder in weak
> + * memory model. It is noop on x86
> + */
> + rte_smp_rmb();
> +
> + /* The subtraction is done between two unsigned 32bits value
> + * (the result is always modulo 32 bits even if we have
> + * cons_head > prod_tail). So 'entries' is always between 0
> + * and size(ring)-1.
> + */
> + *entries = r->prod.tail - oh.val.pos;
> +
> + /* Set the actual entries for dequeue */
> + if (n > *entries)
> + n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 : *entries;
> +
> + if (unlikely(n == 0))
> + return 0;
> +
> + nh.val.pos = oh.val.pos + n;
> + nh.val.cnt = oh.val.cnt + 1;
> +
> + } while (rte_atomic64_cmpset(&r->rts_cons.head.raw,
> + oh.raw, nh.raw) == 0);
> +
> + *old_head = oh.val.pos;
> + return n;
> +}
> +
> +#endif /* _RTE_RING_RTS_GENERIC_H_ */
> --
> 2.17.1
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
2020-03-30 23:37 0% ` Honnappa Nagarahalli
@ 2020-03-31 17:21 0% ` Ananyev, Konstantin
0 siblings, 0 replies; 200+ results
From: Ananyev, Konstantin @ 2020-03-31 17:21 UTC (permalink / raw)
To: Honnappa Nagarahalli, dev; +Cc: olivier.matz, nd, nd
> <snip>
> >
> > > >
> > > > > Subject: [dpdk-dev] [RFC 0/6] New sync modes for ring
> > > > >
> > > > > Upfront note - that RFC is not a complete patch.
> > > > > It introduces an ABI breakage, plus it doesn't update ring_elem
> > > > > code properly,
> > > > As per the current rules, these changes (in the current form) will
> > > > be accepted only for 20.11 release. How do we address this for
> > > > immediate
> > > requirements like RCU defer APIs?
> > >
> > > I think I found a way to introduce these new modes without API/ABI
> > breakage.
> > > Working on v1 right now. Plan to submit it by end of that week/start
> > > of next one.
> > ok
> RCU defer APIs require the rte_ring_xxx_elem versions. I guess you are adding those as well.
Yes, I added it into V1, please have a look.
Also I made 'legacy' peek API (enqueue/dequeue_start/finish) to call
'elem' peek API (enqueue/dequeue_elem_start/finish).
About naming: thought about changing start/finish to reserve/commit,
but decided to left as it is for now - in case you would like to go ahead
with SG API and use reserve/commit there.
Konstantin
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] /validate-abi.sh complains [PATCH v1 3/8] ring: introduce RTS ring mode
2020-03-31 17:05 7% [dpdk-dev] /validate-abi.sh complains [PATCH v1 3/8] ring: introduce RTS ring mode Ananyev, Konstantin
@ 2020-03-31 19:22 8% ` Thomas Monjalon
2020-03-31 19:24 4% ` Thomas Monjalon
2020-04-01 6:38 9% ` David Marchand
1 sibling, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-03-31 19:22 UTC (permalink / raw)
To: Ananyev, Konstantin
Cc: david.marchand, nhorman, honnappa.nagarahalli, dev, Richardson,
Bruce, Yigit, Ferruh, kevin.laatz
31/03/2020 19:05, Ananyev, Konstantin:
> Hi everyone,
>
> Have a question regarding validate-abi.sh.
devtools/validate-abi.sh should be removed.
Please use the new devtools/check-abi.sh
^ permalink raw reply [relevance 8%]
* Re: [dpdk-dev] /validate-abi.sh complains [PATCH v1 3/8] ring: introduce RTS ring mode
2020-03-31 19:22 8% ` Thomas Monjalon
@ 2020-03-31 19:24 4% ` Thomas Monjalon
2020-04-01 12:52 4% ` Neil Horman
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-03-31 19:24 UTC (permalink / raw)
To: nhorman, mdr
Cc: Ananyev, Konstantin, david.marchand, honnappa.nagarahalli, dev,
Richardson, Bruce, Yigit, Ferruh, kevin.laatz
31/03/2020 21:22, Thomas Monjalon:
> 31/03/2020 19:05, Ananyev, Konstantin:
> > Hi everyone,
> >
> > Have a question regarding validate-abi.sh.
>
> devtools/validate-abi.sh should be removed.
> Please use the new devtools/check-abi.sh
The file doc/guides/contributing/abi_versioning.rst
should be updated as well.
Neil? Ray?
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [RFC] ring: make ring implementation non-inlined
@ 2020-03-31 23:25 5% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-03-31 23:25 UTC (permalink / raw)
To: Jerin Jacob, Konstantin Ananyev, Jerin Jacob, Morten Brørup
Cc: dev, Olivier Matz, Honnappa Nagarahalli, David Christensen,
Stephen Hemminger
26/03/2020 09:04, Morten Brørup:
> From: Jerin Jacob
> > On Fri, Mar 20, 2020 Konstantin Ananyev wrote:
> > >
> > > As was discussed here:
> > > http://mails.dpdk.org/archives/dev/2020-February/158586.html
> > > this RFC aimed to hide ring internals into .c and make all
> > > ring functions non-inlined. In theory that might help to
> > > maintain ABI stability in future.
> > > This is just a POC to measure the impact of proposed idea,
> > > proper implementation would definetly need some extra effort.
> > > On IA box (SKX) ring_perf_autotest shows ~20-30 cycles extra for
> > > enqueue+dequeue pair. On some more realistic code, I suspect
> > > the impact it might be a bit higher.
> > > For MP/MC bulk transfers degradation seems quite small,
> > > though for SP/SC and/or small transfers it is more then noticable
> > > (see exact numbers below).
> > > From my perspective we'd probably keep it inlined for now
> > > to avoid any non-anticipated perfomance degradations.
> > > Though intersted to see perf results and opinions from
> > > other interested parties.
> >
> > +1
Konstantin, thank you for doing some measures
> > My reasoning is a bit different, DPDK is using in embedded boxes too
> > where performance has more weight than ABI stuff.
>
> As a network appliance vendor I can confirm that we certainly care
> more about performance than ABI stability.
> ABI stability is irrelevant for us;
> and API instability is a non-recurring engineering cost each time
> we choose to switch to a new DPDK version, which we only do if we
> cannot avoid it, e.g. due to new drivers, security fixes or
> new features that we want to use.
>
> For us, the trend pointed in the wrong direction when DPDK switched
> the preference towards runtime configurability and deprecated compile
> time configurability. I do understand the reasoning behind it,
> and the impact is minimal, so we accept it.
The code can be optimized by removing some instructions with #ifdef.
But the complexity of managing #ifdef enabling/disabling,
depending on the platform and the use case, would be huge.
We try to have a reasonable code "always enabled" which performs well
in all cases. This is a design choice which makes DPDK a library,
not a pool of code to cherry-pick.
> However, if DPDK starts sacrificing performance of the core
> libraries for the benefits of the GNU/Linux distributors,
> network appliance vendors may put more effort into sticking
> with old DPDK versions instead of updating.
The initial choice regarding ABI compatibility was "do not care".
Recently, the decision was done to care about ABI compatibility
as priority number 2. The priority number 1 remains the performance.
That's a reason for allowing some ABI breakages in some specific
releases announced in advance.
> > I think we need to focus first on slow
> > path APIs ABI stuff.
Yes we should not degrade fast path performance for the sake
of avoiding uncertain future ABI issues.
Morten, Jerin, thank you for the feedback.
^ permalink raw reply [relevance 5%]
* [dpdk-dev] [PATCH v3] doc: use svg file type for ice PMD
2020-03-31 12:01 1% [dpdk-dev] [PATCH v1] doc: use svg file type for ice nic Haiyue Wang
2020-03-31 12:06 1% ` [dpdk-dev] [PATCH v2] doc: use svg file type for ice Haiyue Wang
@ 2020-04-01 5:44 1% ` Haiyue Wang
1 sibling, 0 replies; 200+ results
From: Haiyue Wang @ 2020-04-01 5:44 UTC (permalink / raw)
To: dev, xiaolong.ye, ferruh.yigit; +Cc: Haiyue Wang
Replace the PNG binary format with SVG text format.
Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
---
v3: Recreate it as .svg from scratch
v2: Fix the commit tile log format error
---
doc/guides/nics/img/ice_dcf.png | Bin 39168 -> 0 bytes
doc/guides/nics/img/ice_dcf.svg | 516 ++++++++++++++++++++++++++++++++
2 files changed, 516 insertions(+)
delete mode 100644 doc/guides/nics/img/ice_dcf.png
create mode 100644 doc/guides/nics/img/ice_dcf.svg
diff --git a/doc/guides/nics/img/ice_dcf.png b/doc/guides/nics/img/ice_dcf.png
deleted file mode 100644
index 540823c4a05e9ea9d558d233fea77016d1ae5df3..0000000000000000000000000000000000000000
GIT binary patch
literal 0
HcmV?d00001
literal 39168
zcmZ6ybzD?k*FH`NDBT?jlF}e8B?Hn74BeolfOLbDz);d%Lw70NNOyNPNO%7Z_x(KY
z`+k3a@X<4KW}m&+Uh7)dy4F6yN(xdKsHCWHaBvvXpCpvw;1KwLKlqmjz$g7i%71`=
z;O&*A#NkSZ$ajH@7p7wJVsLO35oq^@NWk^0Z=W>n;oxu`p8w#NOsHJo;1Xh`CBz^u
zdIzZ}I(W`x-6a%La{^jclNOU&`_)rTS4-}zY76d1P4w1CVq)-OmrKC^5#ZY~-}pzu
zy$L?@i@zdO6jl2qp0>O_?^V0~==_TKu;0E)6AT_jK}Ys$GuU9xpl6NmUq4AAAS8rZ
z5<h>~BQL7h+1bg!^v{JkDk^GFZXt;WZK)wU@I4Lh>Nma-8Py=vX58uL5BwsNv0+TC
ztdd|ub|{f>I#qBR_r=Ecc2+(+F$m*+POs4gtgWq$;%5$Zz1W+)IolS4AR!?E!%MzF
zY^hc+(<6BKV$B>q*BptIQ5gaPfd*IJosgp=XT25!8zwm^L5M@!Q9uFabHF7#DiuCK
zt~qeo#^9j@Nyw!ThkWHBd%m>z=MtX)IqO&q%qUT-Q<|7KJNMhNs;a6pC&Kl7XSB)v
z3KtLW;^N}!aM9A*nl<f&LypeAH;UH%@~=<W?^;Mic5d!cle_a$qiaVX7S(%;iC{c>
zPEJmL@k|#dzb#-e#WW$n3TwM=5&|%zK#w5Rj~l++vL_;_xx2ZfV1g5o&-OTVud|dg
zY)p8a+CN(%s#Jb5B@9dFyIC8&!|9!kw<)+dnAUfkUON7RZe*>j5V@t@LFIjKzBWN}
zl6bSB?AXD@u-6mJ^$s$%>{fg;H**<Wy_z$R_0fFyxQl_NC8(r`t&yYbC$Wq!ZhWiR
zaQO>2)_CPI7Ur}ViP+8SHXAZ(qCX-8Pgw>@)`d8#Ub}%C2oieVUWG5$Z;wDPw0oh!
z(MS2V(Mt&eBO3Siyc1oDi4_JnN!?`_!F7~4=LJ<;7i4@c?GuTsp&oXOubJN~j*nRE
zr*(v=l`h&IUu)Y-G0<(qS?nZJwYZ$k+*tNcTf2SvjF$-h6dEDNR%>@OyFp@~wx8sm
zQ@KkHCa?M<DDa*q5JkVi|7}X6ooQlUmnxZR*#{T+GPIBvv|B=xJdNM>M<?%wx7c)l
zdn}q*;Wg|Gzyh^yRZNGE1Xt2+qw~^RE=E{lE|-=pha<-cjHj$U3eSXtS-y9ciA;Qt
zg#X2CIXogbtk+gV#pepYbkK~qZROc+2-nD3e6SS4d#n~;vgmMgi02}r<bB>}Fg(g-
zTDmx2YrO0kRzm_}D5S%J%DnQ0DdV>iV*NTt#-ch%|0xYIm~S%Q_BwvyfdQLqe%iUU
za<4uRAr<XM?#|y&3?~<aXTr-Hgaz#tT;9D;cK$2uv9~aYB8Zi$G+Wuxv-FN9+5PYR
zh4qI=trjcZ;=wL@A=W&9;v>?i^7Yn}O(rs$G1Q(|Z2z1sn<Yns)V;&#c$@TE2Q)Ru
z)6)`Bt!gQksJ*lfL=-F}Q=R&|&-+U>DkcK~7vFVi4u!dmmta<R;msWO$55HcDm$Mq
zYzuC++dJmp5`L=_vy^#UbXhfEraEBcG`ZP)uwPnpl~qe?xmmf+NiO2ax1Om+(c=}a
z!GBS0w-`sWU7FHRP;K|H)xhT98<o%yk#hLBt#8R=)!=YcP~pc5m_`)LD5eB~iYqs_
zj?tLy72jM<E{lrbr1en>*bVr_*W;KI;3E7^sdYFjoC{%b$NVtKVTT^7+2BYNw7pYj
zi4R5^w->W9pqU==f1#Eb;6C!>z+DXeb>TXZwZp;G`tJ%B5BYEu=0)FDN%x4fNM_s}
z&3d^+eysZRo^x@R*na2q(uHz@x%b$Uaf>KVBm(XH446M${l^6#mQ=@EaW_yr=YvCd
z+ccCShFQ?$1}?+o10_*^aev+*-pN^+qy-`EqoDYuWpSd~d>GvmJf>Q%JrCl>)WF6k
zmNOMMMp+hgneL-`A#pMMbJc=vS<;lS;}#5&f&14fUK<#DZv350Jd6LHkN$NrFqGJ8
zlLQ^XPt!kNF7<(R!K>C~vVL6sXh^GeVu@|f9?Xx@{W00`WXRZv(|^df><;rf5PNdx
zBpMO7vZw8Z5tyP#0>Vqx1X*<dv&0Kc;4ZffjQvQ21Vb2S)~HS*e}&q3vP0X5Ck>{p
zL|PgxQ2d3lia{mhxO*i0u!<pNN+=Dr*xkF}rL@3)H_B6SSLCd3U(-4g(^%Ggi&~71
zLC)Opd9psBf9m^tF-R2=i~Y+Wx~{DJ7&!b`+!vr6(LbBVOG$#4e;21&m~FS))*G-=
zPa}wqkl#vdk|f?wwVSv9(#)clt1G+|;%v%L28)#;+6VXk{7iUG+jNKVTH?KH*Mg`P
z4}U@{EuUAnkMX-Hs~^?DoM;li?~cv8zY`mcjZ7JLTz<$aU4RE4@Y}g{&JAw#$bdHS
zIsj~7y#89(M|b^nq}$-hTBu4^z2sUdUo4>v8BBVg%wyU%_MXf@bERkY-D?x<$;O<b
zpvBDRefW<DBArvMWHxmVUG9o}n*WowASDGwz0FLH(Dzku6^(`Vv9TH-Or_3WQek9&
zp{?wvKs0uy5PokudTJp-5ZP7duzj+zYp+MDA<51+GTg}}+XPcyOCY#AF=5lm{bEKD
zuYwz`>Y=}WkR(4(Y2H+%2SHKL8L4sdgQdlyUB8->Fe_=z_$V35@6;>_^Sx+A)!#{y
z%vNutx>hi*td+hI$B#YrfhqfD<l~3DlzhC=#`503zteBJi#)&=g+hzrVxg$DkD{I@
zg^onlA0{~L`RpH-+6gSVF_?>R2n@E#DgbDpEt*<Z@je@xlV4BeZ%k2#RC;@KHPXI=
zTv4h`bGcu3y~M9wU$HGWe-9h0vD0j^8a@f(O+_$Xo~<oWUu<$5exr+)S7gD4{3wVn
zgrav3%g?CuuSSpayqwxe{k^|Fjm)EGSivv!zjH;?(yL3=fBQ=%uWad!A{EMEQA^#0
z1YSLDC4#7>-9zS&zAcI<8lQ|mDf7d6im6Q*Pa;o%x1R@B!{cXusK5DSe-NiT0+q(3
zJAul<M~5YH9~^lh-eTUP{6+YH%@Tq2W2W5mXeE;-FoLq>=#Q(^6LkG!bA|cqcF*pi
zs1U^jfwNgMV;A~2jK#)8*61YsCi9QR)xxWa4oeYf>?DNCwZ`dlvWs4bPp=e=C;^0@
zKx^mvbur_;)>z-nUx7=5Qf6*XICBRQBO-{5^FAFbsff<KpkO!a8>m~+y7p>YE4&LA
zuaOFs0yDxhFwwq*qMEm7{5K*3xDA#58|>mq?5K=|*ax&{U5}%6AKyg@FV2-FFZMsS
zrHF6K$F#kK%8>ZN{<(`uWDrDi_jGHzJr(KJ|1yPYc)l)sxy8H2+vzKLzZc?{M!E7v
zF8j7?Yx*gy+=(y$qRI15nG2g2=Obg4BUlV=;xP}#z2<_JKj<^R-sjvH4CwW2sL>TH
zBFbS!1ZIVePcY_hP7Oo2*3+(G-We}hkFa4=pbOcJV<M%e|C+Y!zUqG{n|J=*>)qtn
zuvYEZ1ox-boipvgxq`t_VP5lxlooZlyR(J3H(<AZF?Y%7yd%9i@7`?6^1pYM<8}vA
z(fcoCgsMf!oK2VWC%GQAgevYXL7o?Eii4GU?~V&J?0;;|X}e4MzW2UeoiA;v_Kja^
zfO*44nT%$s<HJ_iHD0JS+i49q+(`zs=pvVW8i+;xuYtKK!0_&IS4RuYlSNh0LB0N0
z@7M}4b%hGL51O*{(vrSeZ6D8OHCSs;@XmhzeVhf(Slk`m&D&IY@;78t2t{dNPCng^
zl3t`DCq?X(3hcH>ORIJ6{H~eR;2X%NKo&enSZ7_?BtrJ5+S7I!1*N++VP8<HIBoPS
zs4s)hk6=@{5NU`^H+?`wcP3Bej!F{iUVeW-Vmz5k<8mL=To^D!#+Hw%%ierPJ88H_
zPv82kH0g1Ai1s0S9toBo^Lh+BR#Ex!G~`x30&(~+!!jfFHn1X}Pg@a3JmuSLTQ#Qb
zpQ{H2%o6CZ4m_;3gUII{#r0Kl7wZMxrF?pq^i$O~lx>HQg7%WjlKr9^Cy2f+@^BqY
zbx<#R&U+j$A5c+HBI@EZBVkOHH&Z}NE>JzIVq7w@Q)x#`<VnX!98`B2r8m%6Vo+<#
z6|brd_*__wX0{t1GqA?_MGCa@sMxH?;yyPPF1TB!b^Hhn>~1g2KzO8);$a8uAH*&3
zg)D9M>q1*@_xCVo#(yS{1NMP%E6V?!Zdx45c`#Zo`Hos^`dDktCFvy1sVN@{zyDdY
z>><Z7B&4vz?kkf60-^cx)}7#;WUQ;|Ust^A{Cc6ps{`J_Ho33e7$a~vXbL1#t`aT!
z2UJ5oWg&|^mMJOh+0J)(WRFNE*ofb`(PH0t*QCEMv<$k$g`L*rpQiFCT-S{2>uUSm
zw<=5E{Hpn^91&*!WTH|0=HF?*gcfcA(20pesM@XX<znYR-nMzrF*R^)xxTPuJ-mhU
z$llE1dA!{69o)n5LZtfsGN@04B+k+hUE_rgg5;a&za<srs{J-<3Qy6+aIcv(Vh0~)
zkKgAXo_BfQU6|#09=7DXY~^inHrqTNHErF7_5Idg-n+BOIH|92e|zJ_nvaQy)}E~B
z(h@~J+X?{0YlgxkKbSHfz6PX3+q1^PZ8&z^?eZ~;#lzEVFypvJwzmp*NjFhor7Ydu
z;qg_Ml8d`7>U`!J{-r<=sz9Ex<rB%wndw8n3T-!>HqoW|-MF)wvnft=dI;3KrdhMf
zMAysew~LW;pSGKW3p2yd|KC%+rqf%cE><%ubiE&Jx1%0MZ=Vj7xf*$_KBZqWak)pv
z4j1_4XAlThcpZNinfEz(+YMJ<jd{>q^T3Cm4}mOujeA_~ly2A-OCo@BUWkDw$?;>`
zNAbhK1bj9N)%iBl%U+H3#(W(f7`9(~6|64(y#)u!7m=9$ls^wV6HaD*K=}$*?`*(r
z%PAmMv7h_OPY`2waY<I!<1y{9lkvg*S2VsV;C^W>H9yql*ma~+oMt}#AG;;>^x|`8
z#R49-yL6HlpGSwi9Z64GB2a6w($eJZ>gJ#mxlbpK{v}O&29pecI1p8Qp<h1SHvO4N
z_+F)%xO0841x)wd$&8Vw&1HJ3w?MV^H)!Ha$X1O>IO*|zpZSM6>*QdWiv2AWGUOWT
zi-U3Fb<QX6QPsrlDFCm9oEyVYRmw8*Eij=nt7ZU<rce$)Tpe9zp*)({+wiPnEysrt
z%KsJKDNEN+O_ku)jJ2#;k}_W1r057%=`a*slt2m_4Ow4Dxwv;-?t_o4?>G2;iM1t1
zHTq@j=TP}0C)-I{SAz4qYL-K^qweE9?(nlNI~&!^GA8sit}QIH$3qI@ShGe{waqKL
z$gP6l!{HMFCb=eGSRp?HQ$W^k5NfTr!FhVBHe2dKhrT@y7@j_LUMO2X!U<qQcl9^w
zAMYGS1h^`W*Fliy$>29w+j$DuOrED%X?afH7nQAm*uGMru4R&AA=<D2a2dhUa`0r@
zkTJbBK6OSuwLJ5|P9q(YMcV`M$U<Wo(*mXYZ8CH?GyY$Y{x{kJ@CmLnT3u;ei(W?I
zMzAO`+bv3<e%#6GvtLFB?dSqKV^E+x%tEbya&0sX)%=q7bl~KNG#gIFFumkEV<7hU
zq)iJq1xI5@jQ(S?U1mj9Smdun_!|Kx7{jS*z}2gF{sN8ft`X>R+<Q>}*9|j&!o)35
z{$bev7y}Xb>i>Ao|77I<=gZUAg;co|EjBB;+t~N>_f;{P9o0(>CsVsQIY)8(M+{Z0
zX0uI|msFKEtT~ttnsz7Q&HG#`N0?_r9`7GV%pEN$8bm$wuu-KbwUr2qDS>DJ*c!rf
zo^{1mo!8969ZL|;%-@E-i_s)2-OJ(P&$+jIm%hVOr64opW`*}ty|pQ4f6VyiOYbe#
zc=({mBPV?e10QS2MJS$rn8*$>vA(j}V^5F$v@VUb2^BbA=Q!jU&j8~69bkn$Rr&R-
zO%6w?WLzx_VTtLj2CV!KGIodUcx>y$1FafezYvGt?k~pa`aUwZ1UZA|ee$tkNv=Cc
z{JGZu?=XZZBj@Ofc4+btE(a9d-5DgAA*&B(**VS-x*~(_)GHAM5wR(${3Wbl#jKod
z%!O|u`G9Q+|2vP8d@VU)x3xNg57CaY!$@wqG0aZHEbB+A(>?I5v+>m<Uz;jVsLJ*J
zW~+@y6zO<$tl?|lNJMu4hC=%f7vE5WJWMT(IHLy3^({EaKP?YB7ke1l@pqa|SOJ6V
zP+$pl?BYl#$__WuF_p)}{q`1!uG%lf;Sa^0L}IxMpnos@Sn`)SbPo_}$YtIzMIKdC
zPs>vd6ZxC5y|K}HBV94S<<b$`HYa;Z!TZt^q6v86x58P{wkI2UoRXG>hZ~Pm)?ev(
zXVwAnw+Kn2g}4PvdewlLVqV+5V`7&VqCwG_bUp~XJRAD!ZJ?#2yhSQ*$+5}6kL+Ty
z7n`c?qP19mF#uIzF`@8+FFFm+?x5Zv7Kf||U6sF$ie@YBg?|s!O5Yd)@?QvRXPw9W
zzAFH=P#$oWOh*kT(7*Ed?XvkR4WNH}In6(}n`hUC2FjAX9nI&<5ll~Xlj0N(M~gM9
z7W;3xm?1qmFD1O@Yps*F!w=>vJEt4sE)}`;KqRl-pB?W8e==fRQHbNR&yXn9-CPzT
zX|2+4M8JL)^24fW-lP2u?B&<ST&l#FFZ)D-uFKJC<Kab1+ygzELfogXQQ&I#70tnM
zy54n{`>H0Fw|hk<)fQE<3!TG@ud#0no9o!?oNKC!@}4eEg2_~&7N3qT-Qsh9Iv>z{
zm)0<yr)u#7_;6@3bvjJz+ggqdL`H_^?hjYdPzWpB%A2F0&w3<JB_EHSqr-ppKXP7$
zrWt>;AauX_U`)J@pMV`1Q@7$z#}5b8W-NWwOd}B|VY?aWP5mhB2UD^`MzduiO()As
zi*rJP!fC{s1^J?Ot$QRN*3s2*LB2-Q@>P<~K2Ce@@ChWHB0qvX-#@OXT6O<z6!)t8
zuHHuvKO2MQPf~XiwS5gw^y}|f;*G`8qVqEEj|<{jf0-d9z;+?Hsd8<$bJw!TNAoog
z!lOHQ@wE`4mlgY=k-F55@ok?~VYgZ9ii?83DUY{@BQPW0Kt&6>3A?52TkfiF>Svqg
z?k5mH0g<X3N|2TzLCY-?>EQvDCi4-yJ&q_ZOB?^Ly!aEOJF0~0B(D@5kKF=wuhD|l
zU+C#Q8yn1*!P{yZ%n10gV=xIA)5(n8w~GulmX=Xr!2;1WoHvT$;rkC#$)4|!vFT+X
zw1r+D=B(0|$vmpR_ND*2G9vq;Sw=C};CZDWLZX{Kby1`C&o%_Y=2S+BH<R2XLYGSE
z_=^|+ssvIdqk*Vve*U?%e-cz*yJPIOZgw1c$H2N3J9pcuYz={bj%-@o*jDZLI~QwC
zBAzvlhUpVeBJ9G0MuPyI?ZP@V^Kz#H7sx;!p1xk4&&-S`|3LqHbLRJvK%QV=Qkb{S
zvFK<?FJ<lVt5;392rQqy5=su0>1Ji8iLbg#!*Kvdbu_1a%bzZ^ESUW(q;MjVZ0*f;
z!p?7bS9101myQ#T`rwFer2+tPS}~s>qQQ4@qljMnW0?dl0Z$d)m31QSYb~j2ibvAI
z#x<niUH!DZdzYp5SBs_*horD3f0JK!rz1mnfewB4Gb?v?I8aoYjvl{!QI?LHC9;+~
zoO4O`Ss;?Z^Aldl-U@c;ZGbzXt2gmFZFi(IS+89!*Lk<F@bP}#ZyFV(^jK1WN90`~
zt=Tthn9XCnm;+Qjz8G0Z>*7Ne;3SWus}*&Z1bGmhWqu<dAv#}%Gd(=ndllitwoRIo
z=Y;_o^88@Qse8?9I6Pwi)bibdeYC$Yfnv85*zhy4D|RdlSSDYL0)*rCoha8?w<DT&
z1A2C)NtZt0&W>)}@#IQP`74&+aOIqCHv=OPy;3=g?ZXcQU_r3<9`^sxN#N(Sl-g5+
zhm%cbigdiDjZN%d1~Qg0ODQFKeyoJR`kd#k|IXA-l{D|)&PM<Niw)F*#IwjCReL1}
zYT#Hf>PQn`j?{K~c<F;FKn7kjrJhC8ZhL_tqV0FFyv`9W-^|cIJ!3w5CB}a^^%s2A
z!kZ-9CkOI{{doQP-#T1pWlE_zQTCS9g~NcX{+SG4b?_OC`xuy3j+4X;3IK1etFU|_
zwU)#L2MPfH`45d!QU<#gu>th)=$7BR;X|@wqUv5<r@H;<-%;IVy_9xGyk!FH?40`i
zh&xk1X(J4^;^f7B!0d@?mW)-O{}o#QN$MY^K{-1A05(IRQ<dU{w)@%ijcKn;(QdbU
zS>2MQW2XvUa)ii)_u`-rnYHKB?wPOvE8o>{*St4#(@XuD6t}M<Kjr}^p&@SEy-a^s
zicMrbiSmDPIzRp-EIO&}#9Z*`{%WHma(;pwaXmQj(l?ztiZhz)?1y!o?2<o%oYP^B
z6=Bgzb0(P_rjGYJ4x0Ex{g{L_v_D*Sp*kJ(f4vqey-LLa(*Ttf96(dRO5oVI*kvHS
z)Yrni5#mxk52&L1ST~|zsG-JRTfTf6_;CA&S~zwWFHy*R*chhF3+&o6)p-`N@F9?l
zoSB%vtJ+66h^$SOipE>?*&Oc(2)=DpfDruZ?u)O3FB}0j^bIi0vt%bB26EIJ=b_`j
zOA1%H`Vt?X?rOD8woVkiMxan81My~V*uaT99~-NAswewgQT>020z-bjLUXyB$=(5F
z9G~s*y0#nZkD*LO`6f}2YVh8EaNq&r+Ak|aWUE>bp(T*mSAqFG%OgQ+P>a@EZ6~h^
z>o?rr#(V;ElS_BiX$tkX(wtspX6DCW{Fg{%=6j`jPqn|>OY@nhl4Zk*<q)K-4WtE2
zT=Xw+#p}(Eb7^W#JwAPHY`QOYdJj{syXN6!1lUnlx{1dr@tqdHzW4wjf0jn517Qdg
z=!?1^@thZ2vBvf)uiJyS<5q(c)qhhM#%1R}>Uml%abT1~`U|=XaVE#c%g7PeQvGo0
z)-=6kkfAb6jV(L}fX;C^-057|Q>o|d{FD#WUvE&jnVy&A0k!CR%(MCH2e^R#JaS_N
znggoLNCg_gLo2g1SejKFNA;^kwYJ$B9v)jOBm2@Ez1!lKMa;9uhM-!^XPe#-5Da_T
z&m`v{tLN!0^5385dIR!-r_<Mz`Kp^bCju;?82igI@{uap{k-ljoM?_CwNSE!ijVV0
zz3^eE&kIb4-5Snu?08)+a0#C*>;Y-^pa!G+!0pq(t;w3<q&Bj#lW>Mmesfu0*hRc!
z9+d~O{*%Db-?Y1~yh*E<m_qESjvJr`{%e1IS9+h$W$(57K`FKllA9RK+##=SwgngC
zEztt?2qCD<Mnjl+!{OwtG*ENSHW;<0Z9W(_c)JW{tnWY8)wyTyk2BfSs+*cg7X2EQ
z=d9!RQ!e9TV3MPe5(d$xXXIzg%MsRqLI|_E<WHe8nFI9>+r4Eij&q(H_&ItXP8kY0
zhglRO!te5?*<|T1*jqku^}AWsT}=UkmNZ?_-w6}<0=A6MGLhUG6ZHH2?K<P9jEQ}8
zDC&66za{7|^II6Mi%l4p#)<PZPsg3p@u<d2FBtkyiJhz@lgzafkLJ#UM3Wkm?;XTj
zmT!p+XUhU%EmKC<2#;9CaPjnKoFz;fD2UKSp1>Vu{yM{NH3}k_`}6Qyb0P^HP783l
zsV!U7SEgrT(W^G`bA^LWUP5e($^kB;9^Du?Kna&UdyvR;Wg6I|skPDw-YJr4w#ER(
znm0{dZBp;MSFJjxQVq*<YpI?-@nYFvH8Z-zqQ4`s2@8mGHbH9ELNAkL)kb@!ZvOzL
zbaE~m)6AOw^82--vSzn$W^!$Licpx+R^5ALo10FnjySnP4)afW9Ejf9@0+$`DLTa1
zH`e)E4e_21xXsPBr%mY(v^fmgTT&&@_u<6rf17?ToV8FwGlSDB1?s>OEw3~CeL_5B
zXD#8)VY7!#1e-|^Uc=Q>)#3wU7FH2@(uewe0EqooU%ubdPph|-Zb{6lZ$-_+c=Fu$
z6$Wz0#!NBZb&#$q9@yuSlAOHCLTk97!#j{y#Pj;7dm7`?qvAWeS>%51QL`21@+god
zr-pamJ8Ep7Y5(wTr=3<(`G1$X<Jg6Mdr)LDm=W=YXEqk0J%gk~N1(175Fo?^mDiec
zM!sZqrY0JrI6@L<(R4uPysw!?#(=sMP};O-=5t2qzi1Q!!exVOpqdSv9IMm3MDf9y
zuD5@Fv(3pxMvK>!09T}tA7WAoy;OJj-=}>aRYYTSwU-{%bR=a0zf6?!E=9~z(>@6&
zK8C=U>}FrAk$gC=0rbZ&^TqERW<P)XLdr+5L|<EF)G7sjxbCO{#0Q@#47%PgmKZ=3
z{#$yRkDr5LjjD42;*<2=GR{(ean|!S`bG5m{QY(6P(ZIo*^`Hp{Z$X3(?(%!(@IPh
zS(4dY1Z)srZKK^fSh_<tsh*Vy<*<C9z#oj^8y;D|LHPEHTZ7SL?TeV#u?h0|JxM+F
z&v$a-Cm}G-mJe0>S58vXkI_a<dWy~KlD|J@-8tEqdfE`m;leCV4yK19wqxDYYM##e
zpA+IgG$D#dULW#0qAVl*HY$G&UF|!Im(?`qu;GVQWCXSgr075=y$@$)BfPSPQP`6|
zmkxU2MMw5&0g6{+u6G1DibM`6IA9M_94K+Hl3+~%XeZ%rpotU5r(X{R0jrbdfAqT9
zpnHFA+gctLP7gX6g{}tQ-p1!$-ZxzoaYY-zy(yIbD43H}_9niN20+k%wYhBhaUy~a
zogDY>^k#-ahKEh#r<;;>HWpeIB#Hg91m44lI^)OV>+6|oWTMHiMVf~DxxLd%!zj=S
zpkh(X{r2$k&`JB)d4s8sKY94&!OMIc0Wwk9$@`W)7}h?)^1ln8<=Nkg*hiLE-ASh1
z#JxLb_z8neh`oAU$Z!1k%|-|OK|tqCp=r0pn?ez<JMjL%A9Ps)>6LkUH%{PifZBg>
zCU|i5;<{*hMp}gt1~raLsrM+_UiP#R8v9x7jZ=rGn%OV;+t0WesG8>QBy9_Y_t~Zl
zI^~ok8{%#h2=bz$9n<5{@-Uw~H_y|6Le!wL4de)@>mmY=4n`dYz0}{_)>{s=YFwOM
zEl#?tC;@{231|1ZA8T%#4%kgr<NrIAVV3z{mQQR4f;-v`U0kxg^_FJ7eQ^{t-SV8y
zXm3N_iycbP9)oj|ix20U7BkXB+=cW-_0%QaOj|zyF*)}*)4tJ#-W!cbMM=t4zCCQp
zgJtNsV6X8Jtht!{zR@NyDX4FCb%BJYp|$utVtWfdqZeU2EbI1(r&uwtRXrqtYFOzB
zmXBU42b3lAKw;WVR{Az13Sfx8QU76x+49QGBSXAKJ;z{sLe}vrhwve$A28WZ?T0jZ
zl40o%_w8W`%%vOGY}C#$guce=W?{is)$3f64ji=tEC9M5H4%Kq&iJkIu;J2SIqq_p
zET#ZpEg<6wtS&1kL~O^oePwi--?(*H8i>@5m)l!C{Y+^qt%3m}fK@pR4zx&~-%Tn4
z3K&qP%<@>;9FXw4t=Pm@W5{x9VwAO)YuI~E2Q3T!s;_x!IYW#neb^pe)?OB5PfB<}
z=Cd?cb3=Ixf<5(D=sJI!eC=YhU4t3sT0_3%9obKJoO19m`$zf=$Z#dp7+{b|Z2r6y
z59t+iJEcZ|3Uo|#`Ca5`Dm8`zLebx_L_eU{g%($mR%g(AUWX>81#~y-+@2DhAxvM6
zi6W)IZMj0jhJJwfS%`LLop4suSKKl+eB<7+(BjN;fZ|r}%^t`{Md5%XK38MZ+U45B
zJ9wx4WlMja@ZtmP6zl7Lujh~V_yii*S}FWl?=Y<?z$7t%iq8ao)mYO-TKv7)>JHXS
zw*82~hH+$dcE1D^C^9^n@|R9syt#gfU6_0MUUmp_YUkc}l70g#@{)V3^x;iyMfwnm
zYA3a_=MD~U15x7*SMra9r4;bx$!CXWCye}&Pl~tvtI~34_NX0=!BN6ix`(Ce5`EOg
z)b=P~s3cD-LZ6J7-vx!vMCHfi3s-ysvEc5wa^C*XQbliv&wm#^vna_)&kp-03=U|&
z2G75Ybua2_)XsEv-FxuMG7JUa1bo}yJFg74OUMK?s9fz0IW^%ct)`lUc>!S=v%>u+
zoEABdTY;uWk*%G0?uFLRoSTD0pxP2mCI1M=hH42X)y&61S)UJ(Cw}21G$v}NTbx#9
zAWe(#@gYCzR|HL~0gxM}f@k1Sxoc7WnuEFXD+aoYlZQ~Ft472{lV*0DXxFhkiB784
zj&g*o=d0(m1tOX>K3a&hh`bvh_95};V}22-4}OCI(8X7<5f}o0$w=mJ;a#j!Hv$Py
z_7_0g_Fu-G3Az{&ygJ&?9K>{cau|F1&QBwt{)yx}1EAXeTLI{RkxWjvn?h|I+p#2&
z<Xj1j$#p-G`2F7-7Oy@#HQ>u>pqhEp8MN@z@L(k~ZHWAb?ZPr(Py<vG%Rj#bSgrG3
z(usssK#OWmk;PvA?+tE$JN8_JmBy4s^Uiz%na6R@KrwAWV?aUsk75bJS4Gg~egJ*#
zSDa_`l~#l`k$j0c`QPsY@KkZ7vT<6i-TTsE48EmAeu}duN`zwnBLgy|<gd-bfC5FE
z{3M3~nsM_cog)H<z*Sx#&h+0(z$loBLw_SttNw(-vrb`+(~6Y+5`3PnF8*UFBq4_q
z<$DKVi-OB)H;g%mf&Rb7VFr(8X7%CSX2-g6$-JU;*(}_lTe2NK`NW8{pdurrEc>yl
zpAn6^FL**iwePkySaJh__2YenuX%leBxJK4tI^K2n?C0{PWFefIDAg{YtaHx8Cbgk
zdsu>nL=N`q@rz#zQ44E!C)~(q=KmZGlNtq1e?fp_VZgyJumhMN{?$z5>6l%n3#s1?
zcONTLrjo|MCINwc=QwdFg5)TOpogInYC>Nj1AH)$2X+Dz;1hO^?L^wXs&oCS_v$gt
z0G|`eTFC1&5aNX1G9muYlVY?Wo0|w)c7;t1HL+EJf!^Odlu=z|U;lZ)B1Q^<h3o~`
zToQEu8aX%m_jYv%btD5X+cG`h!a`eYYggyE2sbAeeSH5r9ISQ&{5QY@@|HBO7CDXV
zU4XN1TjLC6`7-@|FubU*x=o+JI2r<Vfs&ns_yqAMH?O%)%+Mvnvvy#W!G?&M0bl<8
zK|y}2sJZ5>wU-aRY~+ml^&eVVBHKFg1-zihJBIpv_JqBxY#0+G3kGV$kFDS}dQ@vK
zF-`tD4=;WvHDcFrSO}_D%dtATihK+NQo&u{)PMz&O*cNVRWvYg`Rpa;<06eN`3T#A
z$qc=d-i}#wgd`NgLSaVss6q(h%;bv<T7~#hipwiK2?OLNB<XB=lzx#_5GE#to4{h_
zPJ=w#$|Pq61S!!1nompJ2c=`dGOzI^|Ip3UgNHM(Q}VC>kgMr|IpWoowe<AD>D3H2
zcw(W@!Ka^kFyj0jT0Q%I=pd2O-pi=F6Lhj1JXJTR@DN2&`MHAHKQU39LKv^I;DF6$
znN+CVjZD>G$ezHFAOWeWbzNZvm)l*!VQDtFXz;5$bArquU!@tBiV*&}zkrn1^2H{H
z3$+Y|SPU3KGCPnFv;!=`e`8`IR4#pm9h&mi_VSW#ZS4ab2P%Ldu~y2q1_~rOL9f4a
zLZP}!Jx1I1s3F&zo8p=5Om9XD_qZllQWrFD$lp*VwfA`9>q1^C6UexQ%5e6Y?<%Bu
z;fqL%n`XzKoETaYyGDqIFg1%353Q}kGi56)L8$D`W`xBi4<69M!aMFB4L3wkS(uJO
z$LJsHtP6yIDZ=4TqWo0)1R)10M7dm}0FD|gWz2p`8bBE9>ai;MvEJ)#pt2M;9Oy!f
z0k@NmPFu~hus$|gFuJqrT|T=VWdPyi{?EqLT*h46fEWQ^IEF&#+q^G)E)Zw7oQ+>{
zUi_4~?LCh8S`aor&A<}ksb9}_fx}k?CAj|GQ&h$Z3GYUkG~N&brlmBYB9dz=K$#`Z
z1P@Y;q-JGfrc0xym-g^&pFz6z{~D?zl?j^U{<Iz(#g`)=x*;ySIsX+np-F)AGT>Ej
z0Y1O9(ZS!y&H;jmE`tqD2Bt*3uAUFhSLS#h*P96by>%_y<g==G#9Yj;(9=^z9(dwV
z9Z)7{GZjplq#dVIJTnc>%+G!Sl}XXznHOY;0Uq9Nw1#%VP5{Kl2yD(K@pgoso>~3U
zJ#h~aDkGDT{|Ek6fVz7^A1|K}TRIJ8SX+-iu-1^@>ysXKM*N#G^+rChC|m|6+rIvO
zJQd>SIUXHGN5=lot!Ka2C~rB}(eUpKAq-56J}_LVx41c`n=`d8%J&jRt2=QqQIfN$
z2JEMJz?MM0VAc{|sOlBzC;yCF_!QMYbajrEdo3MxoZxT7swhMX#0bN$e7QAL@dT*n
zcT`cZ^eg;HZ$_pqHShPaHsAO{N<u|QLd7MXZTa6X*W>APbHN*LgQMg1-PEdGxRGg0
z!8%}2MSq8;hg)R3$UG5<mR1S!1bU16g~Ptcm9r@)OB>;XDg!>0sMTPD5&DiPdIllP
zK&^aDWa-~~)8DeCukYcBo<zT0<#;8eSMWTlA~5RWS>NoJq_vp#oCyM;brmBD_Kc!t
z<>jugH&wZL5x}AO^%;!PV70LNos_y^p~=|HW9MwpKkJ}pTtACeZvK|Q^!|g&^IG#5
zO7p|#b9L?eQ3qK6cR<|8Pk#oCwIO%dof@PRRIs4P4&cV6AKNVvTN0a$msX75wSxEV
zn@rUy%@@2fJ=cQ?J>(M|FGl;Jnu@<!7ra^d(e`T6T7G~S99PzrNo&$unjbar=-5Oc
zfsy67s5NO#Kc-`K%H5P8r^E_knSOa}or+QZ-GAawZ`MYMLl-1$L6e{JWAjTzMjcL1
z{aW1cb_#gYo1;>YDDAGW2ww&;?j{a9pwpVx-5}e_%HXMtr8VDVV=lO!%uUi~gbrhc
zg(8`23e=y-$QS}exmcZq#$OM2XR5x+$?veJ?r*NNZz4GWcUq&sj|&MbU_T|oj~y^)
z0PiNvVFwaJp>>q-Z=$L)BKu6X>||wrg)!6af3HuIR$~oO(KtU%Wn^U1h^Q|e%|Y3F
z%Dh}Vvo0omUdgq4_H9vtPUC~h7}uXH4&u6C@uOQbQ^tEQk&H}<FZ+A3@x#4r3@_HA
zqG(OwS2<2-bH0=|`#4WHzTKabBs_NJDL#PB*(*aB%}Ep;(ccD60zE+?Ne2AN8FB#k
zDx5BZ6h7fT$0r>sQ{hjYN1Z`FcQ=ll(je}j{KJC%7gAlEXETwuzuro^QRcM{w8f^N
ziL%8e$ET;op7o8SFbhw45s^};4{W+#t2-Dl6y~N=vrd=J1v31Q=sLdHd^WuYAZTvJ
zbtRt)b<w>7(UvC9@SAQ|wj1MdV+^D;-pls)0WsZM*BUEDYAekM(n<5l`Jnm2y-nGm
ztbx?V02efQuA;0(icKgT8U1!iER)%Thd^=&AXIL6U3vww_VTLNMV>y$II*UpFZ0TK
zX8WH`5gk3b=G$Rwp(zU}TJ;-nt%JOPHcm{+W|nPmdW*-Rt$qYsgZ7IC!)NE$HsT7<
zeBg;xz$T`=nEC@HR{5v|u_N_z<!7k_0zj2KSEi!_LVN@Dh4yimuEcoO>&Lb0OCU|?
zLz0f<qLs(15}0Yo=o28*TnY;P3|prezp0V2g}fdx_xPDF1<?c(JTfiuV}JhnT+vGz
zf=>{VdbCDiV6iO@w@%eGM&C7ojUSr=?8sK-8KYg)pQ*`rOyP9Tz$FDJ$C0j^3cWfY
z@6Xg?51`Gz6mTWVE^_yH2hayc=ZW89TpnJoGH!Gn-`~C#DlAVxL-4yG*5YSBo8jOV
zsH0rCm_A3OqPRvWeS8BQ5mTvtNgVn$D+6XBGob?EO>csR$}+PNeUDps@D&vYye86?
z+ehi;rQ5@jxScJRJCRwY5nlAgpp~wt-zggEqK98k9RpL&%a+CkL57L;$U+9(vQ{Mp
zvB+l}K>;*!*PQr~5dRpMud*Vzsj7U~q_Wb^?qk+EVms`mFOBg|sKX33=+CIA$i|vQ
zQa#5jY1f;Spc#J`Hw!zhAGtn^{6wG}e4ph@7dL2*O6--g-Q{U_GLRyfpbUi@Mx$OK
z69&2MA}f9wX)si15G&J`fZMgMcOeL~;a*91cp^7Do13j=<??Jgs?_cT^SdNeMI$D;
z`x4_7{XQT#Ga{r@Uf0yv6`syQToO=eY`_xWfQ`6|CX%h(>nnNs{D}O+7LVkp$i(RW
zsrSD#Lm}fH$pfF*h+fJaq_7n<z23KxD~=lH3=lU|uucoeBGW`sg7o3;{BD!P7l~{Q
zY-M|^6XcxN*&~my05NaGB77BqmnPwE8lz;w$nYC?a#D&-{y=xEs?yW1qfBIqyTU#^
zFVF2Z(8Eh@>7tvBg<NNxv&s`t;Ew3s4^CvI6h$OWc)o@NolQuKLXHNk08jI4&tlB0
zo(4i#2jy|h_6n<XOJR@FORTdI_Zgw5z<l5wJJ9gGz54E0QY)hHOAA^i5Il%>8Xnw9
zH&&u6#Nb8kQ=J+aN&D<y0?-*Sf1X`B%0|})Zvfc?&<Oz2T_LNcT%6V9m9noU_~hA6
zY~Vk}5P9@}@`7qSr|@!HN8lPgE}9hYQLEC467X0ta4&Gq2uK^MM3J91R;j*fUetM)
z3j5IJP<i_khZbiAb>~lb*qH^$ucqO@0G{$D54LvVa(8(^_QCB}YAQv1e}IgPOkag@
z_YdzR>1aCLdIv^kW;q6YTr@9AdU|@|I+v5R)>cv5-6v@onYk*{;nj{nLyWaC(_op5
z*?KkC5L6Vb<<PIWE@m`zyu4SPAw>OgELGLjgz@Y$RGql0q(>h=Q?_)+u8imie~DZ3
z1t@?WQ{md$8gL@Z^X{O=&CTt2D+8B^sOEET)KIFRU+Z6yuCcMxllAWM3gq;*&*tWg
z70k3he*6HA=~#~E4t9l+1*=8|L!n6=#$CQhXpN1He{zPSB2z!k-d%O|d>ziD;o2T*
zag!=pvnceJI2n<YGcnn!v)^{RIxNY`qG2_R5r)z-ndOm3*EN*}_-E#C0PVO~KnwAH
zxR^fM9uDR>M(Z`I$jI55saT&VP)Qu<6C@Ld3?}m)Ej2YHCMJGb>&r_Dg7&Z1x!lP@
zWmJa0^+Ht_-R@mCCirFPosV9Vkmrhn4TU3&f}whECtjH<x)S0gPD?1<0Ifx3aUT%q
zEgPH7Shlpt{V`%sZgFu`2k3R7T8Y-h{&bo5)8k6j-@jk&L=5F6ko=kuKC;l3=0@QY
z$O*XJp4@nx>~x_*QO{iQWg)$t)cyXjam-!$_2%6d7eujje)F=PCoFeBFrj&)3cB11
zyOnqY4o#FUy3*J2#Cnb7F!*)h!Ec#ZUC3zfWZ!v%0s-EX&6~nlwinHR9~oNMH=zSz
zcV2GxWFMQEA&$dUr$vC>ZpQ1!vFUql4J015JUt#RHV9EhB&ZhT13YegeB97Vy;R2s
z#H!!y@i3)pg=nmzO+PXguwS#qqBWMM-@;=xA5JRhb(8da-uXy&xThXi;cS&@(cWHA
zt0JKqT1RDN<x9=DxVYP!n<kGNT)oZn-SOika&mIfoIftq$jHdNHcQ!~X2Qb4$4u-4
zy}i5kV6nWH%R|2{r_V1gioD4Za!CUOB?&qRU9wM<>!|yfJ98dkP=!wj>TN{4Y0Z@m
zrx{$FjehN|8a4W6+nYqkGBV2v8oFK?3g}+9UD#bj_O2^$6Z|!i0ze9Sk3&n3<@g`o
zf8Yo>8EVITD-~Ks#!$$l@(xVF_YLS5&sAG#o0;td;n3Wk?;`4Be;C<@`KB>bQ?CK8
z0=WKsgY!}5$H0}RGa}}hUsl~J@)X}DXZxp32NI`?oR6lg=FrLbEmp?}$f^sGr6h)v
zMOuLNPk4AZE$y1S>I-D#cJ23ETr|JoACrYW1sW4$V(_sE?lT-69c#P>+Wb(Q0Vi}l
z+hWnK)@kwjz|Wtk@uJdVvhea?zC#3S2ldCna?A46R2((a3=b_WZJS%Y^ARRze{b)b
zxK0LDz-i>8e&9~h{w8uWsjdh<AK9s;r#cFg2m(?V86}?g-HoMDxnH0St6)ocegeL#
zmI;rC?c-9_zT&S?tCU37zxlO)=7bp|5uRPeVtP4-QAMlz>+b3*c=rb1KVKBFTwzDo
zU?Y-7%HShb;k6C~9T`y|jq-z=s9<bzGMW`?NT)AduKv@b;1$)_d!8qc>GYiF<Yb$S
zuKxHE?HVA;f4<m3jkB1qKG^7uPEAeCL`aS}K0XG3VK{|9nlECstIK3(Br}AFt-8AU
zN4g;XM{)6R3PB#1V}ox5b!2!F7wZusLy275K)4$4iC09yeBb*a7>a4!q<wJb<UAhj
zkRc72zu5jcV8w}b{n`v0tSRVjv6uWqqO)scXM-zW6jJAUgH!R!^@f~+M%Jv5vU$u;
zS_$lAO$ma$^;sP)ZuGFzH(CS`H!&#)RSxlpI4MCPRlujHkU!BEn?QqJP>>8bf;raN
z-j1N>OI>(+YNtWwVsyArcdYW7FtD}Uu;a&IV#{%2+|V|x^4quHwkO%6xPnf9YmGXg
zfBrzD;Z~pCsh578EF>sC0QOtSC*Cg&E9UHcSLG*mM7JNjZi5rjIQ9}XE<XN?GECXd
zt}a{xaY%Orm7=^nEfcmfq)e|#Mpm}L^Nz<u1sL}`N6Z1fq$^AV1~rOjbN?99%ci&v
zHf3bRpawRb&&)51gyg7%qE{e@DX_EJ@xtiy7Zi}h^0dqP#sQV0qVy*j!_S{pX_+jE
z6nAPE6m^P$&H<B3Y`)ji!xiw_UV}IKCy=hL=k8d}#~<jn_Vz{Ra;PsEnXXA8FCOG;
ztQR2=Nb3t692{ULY@P#O<mvuwYcM%D{xu>Shxy()%CNo{D^qLVtluCb@p}tMy09k#
zB4S%mZf>qho&v=OH*N=rp!1<WGMYe%Y&@&jI3gxuXZzFL#Xhh3*js)5<p>)Ua`wRj
zmAsYtiK!_%CjWfy_wNn4Bp|Q-^B=F*v98S*8eQ?4#YQMychIs@z^}m^E(F7}_cwB)
zMsJ)T>2R<<HijNHhR~p#58h{wYY*EJsra#BMe33~<2_v*tc<ho*eAGNXOn{4rV293
zUxH{$fI~({&3Bm97l@Bk-#QYxOvS~;3ui@Tq0k<{&E|fYBc>+i<>h7Px6+o61`;7D
z2I;Rn?atR6LEi(XT=L<7EOCvvbAN;Onja4j@A?g;pwn02ElB6Xh1qH=vPOx|pWlW_
zBIfxf23^4k-`yRxsH&>2-cd72XfRI!HqVsu?+O9VD^^xjMGMvZ#8wR<WLe3BU$O$i
z=I-jKx5JotGZY~h^*);@QDjZ*?cr?}9IVp$3aB|g4fkP&gwtTLMZ)UuMN{LV;`Ehb
z;u5p0KEIkzw~*NUdw>OV`(nnb`R_JI>w3Q*002();%M1>rrfZcX~+E&m?LpuzQ*Qb
z^68DNXvYihI&7_59S9*+y8Pza8t~SpI(A+}1Xi&<fQAN2-zGD}{D7BI;XxW)1GH+y
z<m3wgLWYs?cK!K-O+Z~&S9f)^i~<(3k%GJgM@o$b{cLzN7#z{O+?%vKUTIrhT^-?s
zN*Tnkp}sA2314x2ZF!UIzp<8U-$yl~M6aUbevX?G6C1rxM(%Y1?&Nm|+$<N>M3dir
z!KF|&HXkToogRWsggy{#342>g;Qy24KB+=}qF_-v?2dhzt^D$EdqD$2TCR(WQwzkT
z0DzR<RhK<m=k+mGSg{P`dsv96CJEJ4wb>{PNY5XOoY%WBSmV)BM}Rby8Si$nN6<)Z
zYlP(AW`Iu4_YJ@%YinzXfKSTGV?Kx|qAs_$w-PyS;N6{_V;3PZGP0fb^onWdMLCW3
zp{Ei)5=dUBGHk9CNy96d`NBTf1Rl0hy4{1k3@sTWf((D#AGV`lQRG-;RMKET2+?uN
z@Vd!F{$(_JLSR}tUw5<MrgoBYC*TJw<QyX;2<c4XHXplM_Fisu1?o<0AU1GlB<~-t
zmZMBrv0p`m(Wl7+X&dhPlh0~K6$$6(p1-qICB?-Cj4tyHP6vBSO?P{fMVWy#&R|~!
zC%_k*J@4`q)7!C98N?f#n!3X&(9-;-p!ptP2>UCM(*Ah1r{fN+5tvU=5C`#>5WwL+
zt$Ugo#UTLob1Nc?Zh8j0nsE2fV#5!ZFNQ2Xj?e4lUf<IFEiRVc6Gu3Z>X&R5ChQ*-
z6(`@}5dm&T$5`0dDCFpPvFvq+K}thJM0CCV1J=^=1SE!~<fK_|51!b}fP?DAJAwP#
z`GGn#y%sJyQ0&hE7gGbc)9*D+-cMenC2L^S<1itQ>*M{h765d)i1*h+g7C=51i$p*
zn-y|%bBoFzX9ZrOB?-BxAE!y@OiDnGcm31HQuyrw*A1KoGg|$VC2vFaZt({Du{xRR
zzdg+Y6@-njyW?hL*NyuyQ?pCH?c?|f`*oK}NfDKeh^_Pi(sk&It_~dyB|8raAUBYh
zhoB-X)9++|48)9rA$$<6`eJNcWxZH$3wRGXANDW!GIxN4f6u@!YlU_|OfFd?-s`bw
z*KG}_c|R=u-d}ms@e+*mLyC5EcSJ$V^lWeP-5RrmQV_sjXLg1~o;<D&DNdLqu06aU
zkck}G_^sh|5n#>0hJ}-GZLPHVO-)T5Fa7bepkty2u-EhU9G=F)+`O$NrxprMV)&U)
zt#SXZRMR#k*Z%CS?`?<-rWi2k>n|apq37r4pE(o%uu(>IhlGb?zI^%Jikl<%htK=#
z>uc->4BFn_-sa|J1GL^$n+!tMcrq#~tg=lYOb-t!NlBB^()2Vm@bPiv`FeZi#AK(B
z57)P+4~1(V%3*iN<U$^(s38DWiZVqJIkxq6w6_~rjg$t<K4E!Sl8}%9qbw~g9gDLg
zFX4XD61p?aIPoQrY5w?9gB{>9?CeUFuD*|Ke0;=9T#fTxEf23Hx4`&dj#}!#jWrxB
zSkl;FG|<yg7FPh#2EzM*I>(Ds-eIb!LIHhFPZ;G#Fd)y7@<OQ#uMvFIURwHVz0FyA
zAqtKh;m<iEWSe)sci{7rCd}eIJ8T-!s*=7)yDgI7i3i%(_^K>XDRGB?FM4T27~baO
z#-5zKunqkRV}POdZ|XY^>O8RqE88Gs<bN`AT_9x93Z)e3WfKm_|DIF&!9&1gZG*OG
zI*HGfj%aF)|1(3O*4y~hBRqUbKvQXp<j|qia}0@;#|J^cuA5qKTm@EdV%+fHKp^0l
z6mH)6P*uD!y}V|%NmFE&b_p{Qa54}_Pulzwgc(q&Ydsn<bpYwle1QR)9~BaiS$98k
zpX)@2^c_YWiBFKKa@*JAahu}WKUM601Q*AYh5kqlJYNEkuScSoQx_P4qEr*;!wAU*
zL`nlQkSE3`c*V#R(c?m)QUtu4&Z<$ZygXNd83%O1e#-(<LDoOo-dirt>*xA3guy>w
z847wcz^RTLp=?767~m!NVsHFtB9;s|&_vE-F<@-f#AgEkMFRskCcrwve)B@zIf3ad
zQ+3cgFJE@#@dWToFsQsQ9i>uX)>LUv9uau{o5Jb?FG>Lmhi-*bqB1SZ!>wo?rFbVm
zy`5z3)X_6@5PDwu9a05<y3Uz@8@B?!IE2!?F^k8XD7A{=A{QPC^<TKyCMU0Zx=u_3
zUdo<#?)e}T-d588a0-U}4iRMt&CAo^s|N#`^5);j)c$rqNJ3YHcu*f*g_)mNlmc-w
zuoelE@}owNdA<dRx4X8VRt$3e6l;u472T&aIh?qpDvOR>;B9juw?jKK3!X2Z<$=iw
z05YbN^%Bs7h>D79V|)5+?9;|A4onVtG{rT^c0?V#_sQyHm?_LCKF?Dv9`7v(DVh?;
zwu+GTaIEC6bKb&n20jQ@=-BL%z{D+3T7KP<*Kf=oh8KXX`n9<=tj*HR`Nq=2DLGI0
zv_C9%I{hsO((3L$ZL5-taqGOJlIpxsP6`~kaTFCdoTSV>MMXEPI6(<A{|jtVP@a2_
z6;%d9LMH)c9Tmr;G`B;3jHX@tHcW<*$@^Z$`M4YhFK?LL?i;<-)C+vo5=1ftElxi`
z^9TB4oRv=vj4a6Ge8674xuDM!Eq~M<8_s)}PeJ9|Phd=sSMX+KdQBbXx;8dx?R}Y2
zXAo&D%BT4Rcv}5i7OwfJ$>@JY3pa~;(brxX_`pz+-r(8s5Sv<4cka=UeA2MI$2xC)
ztcTm5if12%&wj0yn1N#8PvQLD{W`(4!-~j}C6Dwq(y*xhD4L+{@)e;d;h(6~1r-11
zoJL^C@d^Yu<ajMG?yi4@`rg>U8XBWx_SOh0()gsYHszJ2(+Js)i|}!hw(d}t3bPW#
z@zo~=q1^fo$dXTHo$K|ExLlUxBj{p4#JO3`L)3{eCwwAB%BwDw?S-dfe!Ft(bfBkN
zQ;us6?+G0NNHrc1SfvZh|LI|@M#m+x0ICkR%!Ur+1j$#Ma+b+Wx>x?tH8|`eY1vxW
zS#(;bhxbH$Iey#n^#8}yR|ZrSHSJ1=bfZY;p^@%7bf<JTlF}t04bmmuB_g17NH<6c
zNOwth-^KgA-@W%YoW1v|nOV=w^Nhry_ofOW4?FmR2pQ_ZVH4eEjT<Z|z;tvu;4sxq
zUcGf;f#~M_26(516DjzLIDo$)lnlF<JhUf#$NRljI`*oyJ2#VmDNBX|>^gREQSoRe
zkIbP)iB+WETB|K%iy^#OzK~`m=v~LxaH9-ibWzxP^kUxF?-EW(p(OI<xrdTR)ySyB
zLN)vFoEuH1$%;`?&1#O>@QY6<pYp}`6ztnpKGGC;S){=J%{*PUl~iW>I`sY=4;D}J
zje74kT~V?6b9$u2(bhZV8GwA~Z!GoYkUQAzDf9eTepz@T{Aufy(2nlBk@$AzC)Uc0
zYalt@waD>V`p3GG`;zwXKr&W~4qCX#^3T|jl5>q-DN344S$`%5gjg65Kz_pJ2Vn0c
zL_rC&-txSn=2~sPEkX)rj2vFy|5Obiv!yGOr8CQK<MNruoZgc@x1_aGBS*XY%QWwK
zl>G7kaynkT%q`s`br;J^OeOnoCL$>%5~T9fJ_|-c$eYhU8QijGaLT<Yd(8^_F6=9j
zyR2ee2q-o$_h|t3<v*5n#RQ^YCY24fNPOoz%ze<tlc*-W*V{}#`u0h`UBRM}5*?~}
zm6Uu^(f(y{ZeHZS85kD`zNYY#gjB@8>S)%MwFDcip1djSLfY?OwLgQ^ws55fI5QY1
zKZWdb#s>U<vWzfRxNy=-D6B~jyB>TQ94K{s1L^?nj|>+j@BoB+eed#&Fn%f1&`6_A
z4<9z=*52i2qA%wU)mFm+j`lyuyvbL=dlS=FR4$C=6D}qUOGQOuwGVy?7MvM+#rh(e
z|6RCZ4Ue?Jo_P@(W{KywTLNNHV03<}KVSAa><MS_Qu;aXgdm49IkkHPgFN`;Xcc;M
zO_0&+?Rg6kR{h_fI0mH={v-BhfUemnJb?QHw=z|4NH4ZBqayeTW@=xp+gbi~M0e*n
zGd*Qf3klfMg`|!6(@>vYoq3JfmdFzJ3we9)qc|@Y8SiROT^nGtd<F-v+bCAasU!1c
zT7tyqNEUVj<GnZ3H7sCP@xy-58S2n%dZv||B=j*~$upa-4&Z_6wUQS7xVTK+n9|JF
z2lZoHNc1^ee$$8Dgp7=gX2G6;o`d4g)9P?~ph7@>hL@KY><OYgm87B3Dk+V+Z-((F
zoX^v0yL#fOMMTpt@xI`ym|zWM^JQc?3%1|2imsOB!J6y|E(-6k2&XuxJRl`1lIOc8
z*48)Fk0g6Ud}&19v=%2droNOBo(~=!D7r$7r8bf^CG-&dGOFP@`!exKr<(`aBBW?N
zti2;)l;KSg0d~L2k~+QWf2KBCS+35-y#!3vzz}ARVu)%dd?A6uTLsRTVO(Xiy*DM-
z_cyF&R2<1O4`8*WsBtzow&pZiug0G96yMg^@S{InRz#1nvJ$Qol(Q)cABnq+&8^{U
zpc5;${|mMpkY!r92+J?``z7eExqLp>IX)u5mNd3ac;Xp7{I8!^S5AsxN3I1pVM*6H
z_JcfW%NiUq6`Nz;Ue~V$bN1P%hF>D`%e6s<f~i9IYHV<>t7ksv`fo>9{r9w}H~7T^
zebkwoP*!@T<Yv1#_DF!Cw&F<0*Em<Y9Fd7;c49m!D=4m~u2-=X@0KW#L!Pd!G0}?i
z{<GK!_^A(n0d~wv18dzV2qz9oF^;tfgFB`muzCF=KYw#lDh~)uDN82n+{2WSB9;iU
zWW9+`IwX8`u=U#dXjOg0qMgjS*7*XZV7rholD<MbuDGw6t?qS}{5CSM`VNJ7;VgWX
z*)@u>mMQ9%Twd=4tBc5U!Xb?-aB4!bH=R4@u(5GO@^rCdnHnf5qomn+&JkXlNf+cY
zlT@KJ`la|)q;Z5JIBJ#@Zd5GFyxSJ{ZZnz4#UD8;1`=tU;S7O;lr>M4E^@s$RkPP`
zt#B+c{ub<E{6PxF0v&mR+F4;k$rtC0HYqWUHGf!q??u)b1#jZ=gG;!Lf-0=u?m}$7
z?F*cD7J29PS1UyF+YsgA@9r0u5pwz4>qe>JYR8=-ju-Qi8J|p_O_XdVW+1Ue26}=-
z?ci?C$otdW(pS-G!{Se~8dmLW6=NSPb~jen$Vd}4C3WD%zCaws&!z>_{dX;r_|<qs
zm8zNYUS^eKzJWxRQ^*byN^zX}QsN#6!BJnq|Mj{?qBB=Pq>{l!e7r}J@7Y{sfiHv+
zRpk==A?WX9TC8VW7?XkdZt!o<<XcXbY#lk`P;z87+?`BJ)kfI$QgNDvU1^YkC8*Zv
z@gE=fEZp%8W)-`PX;2^Drqi=H#hYC&`8cgTDSQl9x0|5SH!-bWb>c0sZEBygS)=7x
zk(CD&J6-9qEY=znZ<*`={bmvd{w^l&>**Or`u;^i(j}f89)1uxI?DaDh`P{WyvH{j
z!$T_c9RxD{otE^y>amA+vLt)@G%(N*a+zSUZC~X64Uc9#dlG}BU!Wx8)eGIWex_pU
z*;bk5a}Gf%1RHBr=T8hI!a?Mb3I`5_ur3qG#sHh&kEJW-ApsL*hLftM0V$*}Z^5&Q
ze{z5q!%Lm`UXirC&zjJUt<yu;XCg42D_S8}Mvp&%r&+U()czhhh@QnFkC35#hAS|I
zRqbz<MmVS>pg>lm`H6#nuBO`P2@93L8NjnfsMcIAQP~?rj6@ZSgOnwgQl<s%f8!%0
zsaZP-#Y6w3=Q4nj5xvg6o77Yjm@kf1C<ih~?dLVbtfpk%+iN%e=snF&qn+9(juL$T
z?_XHTJ?r{;o19h-jz30o6tymtm_!WWwxRXwRiqN-pQkDnlw!e`C47m3Brtd$=G5!c
z#RETv##!tzKqI9ZlGKC)4rHcUyO3pvCGxMUdBvMmdVVPB{5So%#dA;3lGUdoTbA-!
z!Mryyc0>_+nuV+2o~RV~wE8oi3@dl`>BVMP!HzkfgCqB$@npe%OXyQ-`7}GE5GG;}
zAKv^pPq}C+85d7;6Wk1drcn?Y6gor*-9R?g@vJclHX+u&|IIkV5FuU0NJn2jD5e?@
z*dGYQk*JTCGias)bjd1vGK*mM7`lxbmPmWZXzU5lfq#J@L7~qAQsy)e{_S8Jh4F+L
zPC?OWm4=cedmcHMW9$L=|MxVV2uKQz24;W8r6A%hWRR)Qfk+aMq)u)I*;suKP;ARx
zR5YmRAy6@`{8M?x=Dmd0dC57)v&oO}mPvEt4gR}DNll;*u$%<kOd{n}<C04|T?S(c
zUM`0`*Go;P{C}V084U{fB-Q_v6jH^!`2Qa){t62egf14BVYZ|tH6;MJCKHrw3num}
z6_mubn~9diwtc<1>-IAH)>1sR320UM<hZI{#VKv*LmrB~Lw>J9@3I^9&=QyF9}KJ+
zz#B_Cpj`Wvj#2J!LUVj2%P^NbHp4y}JPA2vsa4_I1<Cg<bMJ6CFR9Hi4Su}^q6&EC
zlo0f%mdc|<gB|*EL~<78hMqY8C5pH7xMMl@AmHDmPbt+xI(-y!THhyFX00hux11&e
z-|_o8Ke8WN-(~o(xoB+fHVrp?n~g=^*Gac7URe^sl`9l5(ecTpe?BEK|4<_G)2TSR
zm1xyI2)!NVS*i$@3vZOqoa|j@C;EP|vCQa!zTj?||791a<cGx}Uu(*HI}z(CZKHri
zvnAa$u7jd8$P=Y$JosKpzT57#hzP;~IEkyFsy7;ruh8avQ#lfHpPDjWSM3@6uF-z`
zA?)P%sh0J<cJ_>oz&;+k@dJLV528cDK6OL+0q=G=e`EHmiR=6t#TC*YG6Ie{vSn0m
zR6z_*3L6NF4-!JcNKj2a4(#bK-%t6Df^NSqKFz32!_1GOE%`J)ZLP0lqj)S=9S<0-
zj*xnvV<=o21UMZpF+Tlk6uh4*Xp}NxY&k_d5L~tyj^0}Ko%HB3N#l|q3veUXKGg)}
zP?6m{kfJ!f(_Z17A5G{^dSgA^LY;2~WoIZn2X?iSIwmm$)$5NhKIa0e_cD)D@QJn?
ziHVL-UtEL_DXtmdiA8x9EybS;+<xokN4YFZk6ykA*Vjb1@l^#?o<>O98B*)$3|PC*
z+tLPzo}`GlXEk~5#5X5=AQic5&iRXzue+9naeLGF#Acl#Di2xhL?EaR7=HK1X6lEI
zxBuBlK|l^${hdC394=ZY%$WjJD!<jQnTGu;ttn{@;>Jt_IzybqH(Oa*UORk0BqMg2
zV%ctaQGVXnp47$LdCC#!&2t;C%8raijY?FHd{s}p6khV4H2$>bXG%l?`Tg!{Gc`ia
zFK&Rnyg`uS@X8J=S19Cjq?#lEXOgMVRs9|ouMUu6_8P4dsYR|{pDGS%eJU-sY5G9Q
zQ*oC$T|s@D=Ul)1^WSf;4FtDi^%)y^<3qIfW128IoC*NF7+1-se9^byv)#+U64(~W
zv^3#1YKl+P9FnM1o<vIyDHJ<aWSw^$b?S9hD+e$mAIIDU&t<X0KB?cWBR_=CI8d`Q
z*1#KM)*EQK-N#XRG>sX&weD+l=D8?tcQLZhsi956Gy{M~z6lfoZP}ZZr=6bfZPi)+
zOQ1V&-Zvo+Q;kO3Y(3J$v#cy6Ma&c-9yLhybBY0Q{4|1{<{0d{1e)FoPZgj(j}})o
zrY#@ANL}*hMAdh<t0libhe#-u$D=vLtGuEY=BW{9{K{!(V~0^^iUr1Kk>y&P4W)<<
z4fxq<D2mC0MLf$2f-T!^&a@RTAh=hNu~5yBwMdAtvgaW8dM@;f@~frTZ*A}vrzOHs
z)5!f%;``b#!0?z;mPcSJ%wv*_OIuXQHGCvtGZ;DuQwsP60I4~1=SC#gjXA3~h_Nt%
z9n|5r{dx2B<*z1I^7jf{7Wu-p`k5D9$F7@YrAdjrzQ1f-!X7Yl7tD<=6b%PFWvAY}
zZMZ+n)Ym>2KLJNEFZwG=3<HFa-!aPMl%|9U1T4%TVEGbge{FKhrgEeCyhYQ)ZNI6$
z8-u<Uh{>*-jnBi;S}mXMDEbZgxi|HGO`;aR4IIgr+DI;%ogonH{Kg{*pJAh`{og3~
zT|shEMick9cPVEh`6evYTcNN}sG5XoE8EooE2ONk^dsP~zk3VQ61m3=H`xE>cUr#p
zVUEWv#~U)>53WnjZ0uXy%jI5aYHTj)c^&G>3Dt%Bij={@l4+KvdO1)qijz$P84f_l
z$${%KX8DSac-&f-5G4c43R4>nL;0?~-ZnzuYx`+O4p_**l-^}->%{_PvgC)CUvac<
z2o=4}Pt@z%Z%hB_{Y&GYC$~O+v;#KW&^)JRnAvKh(W)JdNZ4uH>zxYCR}$)({E@Mz
ze{Btg;x3zZRFWI=Q@D8evRwHtC`$0E2W@&3<!8F?jW<2QTm;FKhquiKqb>U#X#MmB
zQ}McDRXe4>m5qu|BsDjhcDD}Wfkci#%TrZEZw0j#Ox=}L?9S7EW_18o;^tEM(yNyN
zca0W91k1joK`4uhIb-bs#ve+I{CR6_kQ;GiXEFAb-{05oS5=PMlXe19|LsDdd!^pj
z%ttz+&m1*sz*!g<8Fgr&_Z_;<_&%LgE&s5eHu+tOZI#1%Wgh6MbxRN}c-1nwvn%pp
z=X_E5z5k(zegEn_qw$|U1kFwawwp<^0KYSd3KKdqO(x0{$OQ~owdB&rMPE9;#2y9~
z9p}dj9$ZXT!4nQXMpg))&uJT7Qam1>wnoGa<%_yn5V^XFs&=H`rv}=$oZJ)IXs8-2
zvfs};eu10ui0QA^q2rs`JNz2p{lM|Q`>2~beDJI%h+D*;r{<|@rSO@Z1Z1b=h3Y*8
zbC#Q?Jr)HXy4@F!y`r{?+MIw!81Cc2#<{0q+oaUUByNXO6rEe@|F|%>5nX&0g|e!1
zR{gD%hvgV*N=Z9$Ic?9{)FyKlx6yZ79s2nFnm4ng9Zx1Fo7s)>25-S%m|U$o)gahU
z2RnM3Mraj5MUXx8dN?dF<g(JX9G4GE&M}>93Gj%un|-eLxog<j-3)8*n~IlhZ=DA^
zo(C=SfHK4)Vwu=>?IzA7?D#0(FZ!&|w?E$ab11Pf!RMeHXZpJ1jTyRDTo-j<ojp*)
zIPkYM@pDZ1mC>Yp)RD(-!2d({?K*4mqIX~{q@PJQz^nY`)!_GXYbR9(7J5j69F)uH
zfmuDazXkR%hDXW#r2DPBlv10A`RUZdSpV8KeSWFwyUwflkR{KJc7N*4sDCyKCRbG1
zceUROZYepirkiwH0VB=@Q1wxcv=dzS$;*AEopXM8`5&ppK4Cb(@YU_RGat-UNjLXN
z_XZVFTTKHsH535EQkH*{XAvJW-y%+r*E2pOMo8S~S9&?tUW4+n$qxDbkzZoFmSKo#
ztVL?9AbWt#Zu8BpfN2_K<esASqtZHF0li>Mz9GEjsq(zV=%4T?<CJyMDOO52Rhz8r
z22ayiAQ{&tC`7SrL&aW?0f?>@p=~;U&#wLe*(oTG4l@HaV6Ya)>~L5OlP&GNG_IL_
zU%Ot=2zmQd(;y_j!I)tX5cx7P5?)eMt@kQQiTx}b-rm<turq-KcJ139@8jK^pUquW
zvzKmHF9W?WIqft%8)@(SZBJ7OZ5vQ+X+EQ1m%@g+{mO*lrClp3yPy=5K8Da6(ZS~k
zXi}MzLlR$4Dmn}kO*B8WI4}Bg65xQDgOJZ#E?)Ndwcp$_c2zvHw+3zj>+JT*+Qaui
z1jv1{$&wV<Ua7zQ31{{RxVxZy%5)zEI9>V#z*Gi=bRmo7#B$C_i-8_lpPa9}C9SKG
zq|rtmIv(S)BPM!B4{VCQE4KHC(sr!vZ}PmK`gZC1i}n<#fO=PfeQ(t4{2?DUpQnv&
ztIRNAh()UO_S$&aD~i{u7IR+(ey%P9?F@q@VWuuy=HW^?jrOO%w(BiRVSmsl$N5Ae
zY?`x)5M6>0GI$*8KDc0}v^Jz!=G8d3SD64~)XJ6)3O%1|!Bwr{RYNRObE=2FN3trd
z-D^wQLYd@=5)70WD5t4jtKv7m#|UV)2Xa~Kh;E#1>{&-Ye?nB<9uE56$lqoyowE)!
z52?EzbSuH`voZZS-(R{+<z5C1VpMrCY%MPDUsZNi7m>dWq|hW(pEU^bfAa#Vwz5bQ
z69G*R66=Kw2R@7lec*`@(<4wJ%CRx8XV10|qV9T$zkQwmCh|hd2XG0O{U+~wB2F@6
zrCV6afi7U>!N?+2gf@(d1~67d^qikd<8HAA@JhYdwObW`gfoJBe7RUkzFfV2^Ly3h
zJU@OxevZq<<T5v0{AC}zUlmXw62%jsUyoPI#dLQ8sM<tV%5-dGd{VUeYAB3GdtZd~
zM2siQqMYl`=@$G%3?{wuG+`}f<=TV1id7SHbi2|{JkVC6_>LsBJZl-ckS%v_M?DX!
zhl3e(^l|n1``FCy4?HUFqf&ipXhq82;fE0a%}$7#LVdm~yjM-{#q%mm>R!+C-;=|U
zHpek&=;OSIbmg`nP!p=b!#DPJ?VS~S)4^WC{iw&I3bRtK`VjE`&g^CE_Dcmv&CFtF
z>x*A_lhYd?dkK@}(u~`5&ci2_eVCV*-2Sd9nOWWo`zytIK_0kC(~I`CPpofZXO6#}
zvVO_6oUmF9T&5d6;5BvGcCKw6G2{}x;Or0e^hXL2A^iH}x8@t^meOeP#OG)*WF)}q
zESFQXz1-w;^he11iQAN;g?=D+k&uS44FCBCR-T`~8MW0C?X!{f`g4iuR5E<sP9xbq
zh;yvuP@advL4ZH++;zh&ap&g9(=fx@+EiDT0Yrn!F)?Cn?$o0g1wyMb7GM4;aCQmW
zNY;#DE$_J;c;F`4<F(yuNBV|qFeUdSZe7VAn>F|CN@S&+i^4)BGw)#B|2R#gm?izT
z)1k-_29I!VJg@4N`~9O_ySU4H>YOM!8hzenrma4ffJ@y*S50!q%Vtrm>EC8osXiMC
zW_$m+7G{?MR6<Z)K6Ejb^D|4d)p~@jTOUvBFbWyJrjI!j<k!(=eo>;2@8Uh6bAOpu
z)Sz5`?;Oh5<aX-}1X}`Z&K`iLzUL&A$FMe66%*jvI6la^!ys;Xy4@2c)iJ`=5+0+@
zpFUb_9?@-_;==t>F6P7Xc&wKizik+2R=(}3T8T#g#p&pwUZgJjy5CpPnlG}P_E|-Q
zqP*L8oGbNTvZHmIjVIw>e(`M$Q3gsb$;*6LP9?>ii9F3LDD=Ch+7-|sv3Z=<{+(Z>
zS1vPZYp%-dCU?YPVX|FaTWr^;W?6|!Puj3<e|Ds+F!MnNSD(XpzAWVpKc>jVC(qJ8
z!W(@J`NbOJkmaWZlSXaQG)$-_?jA$ABi)-Dn_iT)I)8X0WnQ%8KW~+A5|7<7bm-dC
zJ<~{D`10IX12wR4y(g|5r=9J&5?SM8O!e3fIqJR{nB53SHuK-}&yQ2F_&wbZ{%$Y|
zWC@z2@xfL_GuVw-_+8?gK!DOaa_=C70m%DJH$Gsx+q%-%zS7Z9&z}21##ysAM}kq7
zq+=(3GilW5H(4`%oQMe=df%e|%;k}o&{p}dADJAn4nt0qZMMw@>qD5^E3TL{N!963
z-jTLjdE1A15!+k`{c>c<`*r@+ZK?y^g?iSe;;&q%+__v{e1EKQpsf(?W5H@WtUFJ*
z_Bb@N-fgVE&^M;pSZSg-NMxW)+q_nYY%;;t`?2BH$#<s!>=H!`-LnQ4&K0wL7`la_
z9s`d6;ccnQBKJ0*J^hC{NrIu)`kFV45Uv&-qx5YpES%#T-13!8IidOu0SEe<^OTR%
zDrBmP4UYG&%M{a5wjcM(EF9+S#0Ps9{~YnUKxQ{%%c4Txw2R1#e4i*#hsRT`#8^}G
zw*FJ&*QZK26@f}o?%4LU3hs&iGh0?cOLmvQFCj)oiYrR?Fm5jUZQpL6GD=#kW%se=
z#_64~8KuDkt#!G8tN@I^Wk9P*kp0Be9N0Nuq;KuApga<ko-8GNbLiec{j)>+o=d?!
zZ=Da%t?)#$=-rgO##J;RYd`8KeJstbf#iBdxb{Z#7<MTOA!3U8Ot}rN5mznHq*CUE
zqY{w(IMc5fT_kwd{an!oSC^GtN!IY2Hr_bh2ZOyH=#9}&NouS{5Q{lSGu-9iB){=k
zDly-rv8a(z4gArT^#Q)^(Q<9|(sitb?dl{(PRD<Xi@nW3MpnbwKF@&&$)}9}>=&cQ
zy1?!Jif@a&ch)nM<tNE?6QOw*NL5Za{hP`|+jH|l8<WA{a6s2Qk@qBUfNV0=jfBsl
zO^1^&6$CS0wDF6URsRGX2%Iq}J~ltvic^K}b@*JZT>r3h%=G@&0DF)GguC8}H)+E^
zxT<2?H(L-%4bA$ErG1@(u?dGPKiv<BUMgJh`l@(mRexr!3ui{{nWvqG0}u?jdcUk(
z3m_9I*-wJjF5RkFR$^?R^SNvTL?(Dx3(1+mHL!?<7+?_hU5*C5OP=!S2r`j!G)W>i
z5f=L*cNh!)DI_3?g%U$vE?;%P#q?nRV7f)3DRTXLxRX(?t8<1XO)I%HC37MO!~O|K
z<*OkTQj~ufdpjSk#{HVsj7P=FRHqZ%*y{>`57gwlz@FaAO$Zbits~=gxh|8g2nI}g
z;2rWbJ6YjEJxI9T#~UQbHDI+A;b<vgyhv%>MC0?l12|OQvNxR$T^!ytNBzG5%q8o^
zha;OWqZ@$frf5x~syO#_-fEZDXjLd-yufU*?Uw3RfjvFw`Zo_{_mmbAcy#F$_I8~V
z+8#y!tNHgI7y#Y2#7hT*fo0x?qo$c*O}UO2ixe}5^jb%#ztYe`IJIm$dGwk727HRL
z%3GZ`8qR*xrDuCxpDVys#dEuBLvFc~Vay&_&Xf7?5k?CaJTvrP`vj9eQ<caqIimcJ
znF3Mj9zT3L+Z9Dl(yYe;`nK^asD?p`zF>g^?wGNzx3uE?(}&)U?dkBnZdXR2jsAKK
z3Gycfkx(;z>@#_oydK&Pz<Ra3E1i`Y<}4D{lDfML%O9Viy6k*iTs3ZkNsC{9)LXUw
z?bP4we0*PmWG^&lb<ifVUG>^e#}2q1=a)qw0x(f%SL{P9%?@IJ;30eR=@~T~s*i0g
zd-*<n0~G}n49M*tAH930<pSVETEHav)uVm+je-;C*&J+V+3?)qeUOirP1n`(@77n-
z2<BOub02cL#J;GjRnnv>&#}x*bJbWJ75URG<U5hw*ZpcwDmbcq!NaF%%wqrAbg+tB
zP3Sxb)7h&NZYx^T(dKgmIb`oeA&@zG9tG?#&6i)-#Gt`f1_#ZnDo>SkQkh<IyrJ;+
z@G;|QcKpHa1|QjTE3iv)hJE)X&h>KM9;PgnMtzC0-_31Wr6c>fe|PlBM=Aj>g_i^1
zjAplXSDG+A2^Y5?^KUcL!?O1ULf+j;8cjRH+oXF*@+0;t&s}{=10|Itm@1N*U`*#_
zJ$cewgg^f=wdCjS>jpsUyIN{N3`Af<Qjj1h1nlY$-|>Cp%Y{^c3|iYuy$1Q}O+Xb=
zv&goYGLqK}crEOe#}&g-(IKYlrv#L*tVsQx*DGs7L#jfHSbeMSzjE_kUB4abDf#ht
zgilfH<Kw;qsgG@$_vDMVM~hKnX1f5KcfL#PF~Gvee6E5+XM_~r35)Ej4}aWGTDIGW
zqTYx)^kzG1cuRFGMT!m&fEZ8^keKt%jfA~uCNe9j31kKsP|<;=PtF_nV#0bs@2nNS
z)Ki}rUu^FVB<sQ#{G?g7<j<(lQNFkRxa-Ll%9SF$n+;Gw!viIC#NUt(cS|cZCkHT_
z<ZfD}$|<s$Pfe3(*UIkEf%4e%%VM5m5o=M^8@(L<f|Gx(<C=m}ny%LN@dmISsm_ok
z-#RdMO$>13d5&>&OYKKkNCQp;`~1iv%R_(jk?ALTH7lrTx5!4D=a;@B`oF-vcvrT9
z7&w6xCUct}3$C<+pWPMKKzi)r8AYXs;D9I;<bfy%s-g2S>Pw%t)+e!xo`I<Uf!6cH
zB$M}i`}_73^4CpFaiE$KIJ5c4#V>LuSpknS8Q{G@3k?;W5@J@sNN6N{@>5t*A-^>4
zFHUNgkJ;f=!0#11%qg>4kvZ(W*BNa`*Edi`jU}CCxj*edDCK|O@tAMgbR<x9a~|DW
zOLFBWi2&6UqR_ebAsx<em~-};m5+!RfcRhj*_d~17jn6@{a_it!4&IsJ9wc7x(qy=
zW`&qP00Ghe%`NP)LhL5x!?NvmUK<huS#bz`xox;rltc5jy{~ites`~33CL)~ApOr&
zf?Wo~U1C4XLL)#%$w*&b9&YvWXVC2HrEyk1i&Y718ZlKaQVV|{`=I3^g8kg9Ug6t6
zVxdXiR9#o|(C3s(59$2j(!!$kLPIrSE?!Y!ldj)W5o)VDq2~e1Q-A=d`YI1fVVywk
z5B@wDGFmXK28KO|uq>~6cHXT~%LPE6B%Sd5xP!Xo=~MlNMQ1sKCHk{Dilk#)l*sZ|
z_iqPbIj*n&nLSI<iUYOT?<%{?1&fW$60Qd`>RXhkFv}ic)#KC2S8hjk70I4Qg~Q2=
zod>aeX$Hm(B!yF|8acSx>KKuowRGh`-B>co)1JLuXg$O6%hSY`50G4bnaQJuNPddB
zcoVl*FjE=Z$lP2(H>RlBc=Y*o71^QB<;2)6o&`Z<B<#7evhTaG`*$!vKtTr?CQ>e9
zh4;N+KF7}fmJyfTpMT5k^UWd|MthAR9EkB&@NM%y&mGZQjHmzQdW^kwx8=9pDEn|K
z)wCbzhWWv8XVIkbsAyPRRTegdBbGtGG}&p9i;gW(IXw!i+tQT_y%N+{0G+YZpBd88
zS;xIoVl|IY!0in0_jujqJF?OqVDqRMQc8ZOg%-}EPjTK6tceQOwfU?3&M|;@4I}^d
z@cZ^H62Fea9fR2hympi;-INjhUzK@=7f>2mQSZ4yo*!^qHcQcJgG`@3^{(kPAw5hr
zjcgepQBA(2w$ky2A2X<N`4d#61jC+i!3wBP(sB#+yux4P=_C6wncKaboX|+Ws^Ja#
z)r<K1RL$t<feN8G`IIBHD%H>d6#qXZZ-!aTwLy&|K1$YEhhVXFJ8>6RdYHDNrZTdS
z$J!4agQg27wCD({lnK84LSK^)S|=lYUkZAam)x$+hf6%ZUQ?>!;WPFq5_w{k!RcVk
zD?5@x+^3!<tEBbLe9o@@@92ZE`92-Gm=3zFJ7WH)c3N&T&>?-<e^7?6g!-Of^GcUb
zwo`B952AHB(?am$&dCf%atzShc|1u6?J_XAE_xNxYMgpr*;TO(0=*ijDur9`TMjSK
z(8g3u)_gj&660|kC~1(m400Dt-jGkI!NWO**&(evrmV&PBDi5Yo{IPj!<<y9d>!eg
z$N#u=vG3RxI5GqJ7bqFD9pI2C6YJmtPAC#{)xzxfU4~;j%?5QkAkk#Ed|;%Ek<fvU
zXS|;n3ax<p`*ep<mTUDrq}WW-2U{Swt^_N|JW?I$w^WOBfzZ?tX*<>%i@1X+EQiQ~
zMGZZ|UQ9vDnU_k0DfvxG0%5u#R;qeJ5^FA61qwZ+5$^8>934ye^tsKmVv$JGlh=AY
zq6N0(w|6C}Q<q6k+Bv$Y*L!hSALmVI$3v1%YJx^@zN;{<rjs!y=%l_6<G@O0{Egh7
zam}ea!pIX5+*f1{gisEh(8AL>?6Th}<`hGtjslMUz8@Q(6dp$#R0|h;k?{xsxBz_&
z2mkQDO4#=<w8ml1s1^+AqC<qcA%qIPIO}rTtF#S}m8!_OSI((fwUoa%IS`;UM!!!7
zFDmr~+#dlks7m1eZ?*E17-*dFcvjV3e<<`}^-q_y?rjsnC`-o)yUulkofo`h5#i{Z
zsf<y%qTc-%?kkJ?9BqjvBMee+R@2{E$89{Fw~J^QR~*8;@|Vqj0Wx?X#W15^lV_7V
zEG~0DIb;~NfwW4aUV{i;PW}BFZ0~J{(nbH<y`n$_YuWiJhu`g8ABIW9)lW@p8)a`i
z*9jVFd83ZL4A&}e{GDvzKIs{9v$nfP0BZuGyx-{SE$Ew~LkJ1EOw-q*(iX1_@ZwZM
z52m)Mze8X6#;37H_UO?J0FLAhtRVcN+B=9qAdT=4Jw?6!{RXi|j-5n2Nhm$4)xq0u
zyjEeWbEuJU^n+|QqEzIp<Z!PUya1l_f<(~Irt7lx(qA=+pYmyz8gG8Q!Nm=3@>(!Y
zS}sBxvZR-XgEK2`&3Tp1KFVi)yS{k8fryt3YrIP;`2I3T`>4`}=^huejP*JF`F>(q
z+TX?T^`~Eyb=Ik}5MvSp<ca)Ys5wHU98CBSgHD_KWQbnu{rX#mS}ZTbS7pZ}f%<PY
z!c9<qN}O8RbT_W8KSCtl#`#S_40xgwKk~>X(1GlD$4j9}MS;uq`YRx>;ngeB`cG4(
zjal)t()b6ZwF5ReKFchnGAekn^!pViVP{$dN)eDP51zGO0&QFzgvL8@C-IR9y$7V=
zZr{X`$Qgvb>&35}ly_0+R34{;w><WnJk|+61=5oy4ZcUahXqy&nZAj$o~u!&#w3~z
zy9TFbWCTSl62^KYnGq%A{^Hi|H^g_*L!&yvHlCNCGS!Moi9*P*8Y!7*JT7jT!;}n=
zl?IE4#WxaXI{tMLpml}A${Em7YYW<nNitAYGj-@RNAHfAElImr+f!JuP||HU^`r#(
zQbL^3sg%bjCICq$F+M(Ch8ic8&4r}C8@7OrfVo~$kh%F+A5GxEnIc2?Sd6*cPza_F
zN1sUKryMk2=5MAO0`w3sWfPNCz>LBq;f>4AW??be;1!RTdGRz5NWv@F0T(;`CRCO$
z#Mc?+Y!&W)AHD>?XEP}&rJw_!7W@{|{%m1kK~#g)umvlXE$vk|H_OvUof-!$+!h8F
z6X8(S>+8Nxo~Q*cKu99$DM<Rxc6&4{J3D&~N6JuQh}nc-pe=ZSVa>?hn50%=fQ1P+
zHA%qcrYKhMZO)x}R0%sOK3=jZ;DRhPx!HlfXR4~I;Fph}ZcW4@(H6IhKzWFzXp?ez
zLp-(+_8$rP@l2PFUN54Mt^&8RekZ*K*ap~nDnz8D{(Dnpes`A^=H}rQ9CFUG>B<_X
z7d)e&8=1}rHCK>(m53dVUg%7}!P!oPSeVv&&7<KeQk`{Aa-i2MywitZDbqXuQqz7j
z$-Pe(R5uh-`aT2=EWMhV+S=+WOn#HgW@H0()Un(ROUDm<E2E=UN0)<`#Ny;9Ke+m7
z!Vk1Ze)$fE86zw>vFd}o-J3^Tc>=wz-IFykHjQjK+N101H}-4h_$eLjQR=W^tfKM^
zpdVahQfH_5%a>8Wk6T+?|7y={_}jp+nJPVA%gN;1N|l&?13j)V8fJ7wmnHdrxOHt1
zt1p4uv!29p>3lPjIRQsh^W&$H?W%nHzqk(;M-D^ql=%TT$WU^}A4%d!&>~|Kj&y#0
zegHnV`QEs=yVu?8>;L>WT=9J2cSFWeX0%Y7Eix{%pUc89)PS@YkMTOB<75_3RWe%-
zye<31%`L**t^p<)gieu!*GO;K*}>}Yx$S9qdh+J5-nkAp6y0xH8P%t|o&J3PvEfrZ
z^xxWPeM<rHr!4}8lW_*6h+iCE{JJSO1Enn~<uA1j&GtZvggD^uGLe%T|M{weghBHC
z#}B|z#YQ%7G&zlxCH}FhmD)q7xIooSvnk)GlBBAlrB6E)Ea74Bmej{`BH3sEA68EL
zgWnp$-H%eHh>T-nJP=Mp(bE{f$7yQ1Z4M@bje2}}IXXNH9~fc9V>*$Z*?JWdpy%Xp
zfshpkzuD?1#b5HT$jJSxg7U{pWoFaw^2zrLb=oj|w-k-XV-(rF^!`8*mkcZm;9Pw(
zY}Hj%9GRZxqNSzf;^JatjOsD%Xt!~$#=|U+lFXm7zvNS#UnVr#X+JD(w4$z3@H+Zq
zMy6LiKN4j~pY5=4d@XMA2iJH1>zYg3i>g<4UqE0RKRY+~9&p3@<0xFWhWP*u1-xl$
zdKyqxD=RA#6B2&u9gi)1v@-nlW$=1amiX(1_Jx1CtM|XrFoGr35d9)yr^?+_zur*n
zbeiDLsJir4ufj&uC$v!8>@2aLs;vm9+BI?)7R}X1=<<UCOeOPVb7^$n=?eK>-k+z}
z>eav3tG1oqU{DX(pR(C3*Nn{(1QZp9b?y6cS)FnWC;1yg?15HIhkLe*>lWNEfzHu5
z{ykc1>3sI_rwcp+S?W?{@~;DrB@_r8Kj}8ODfp}E^n~h`t-OS$hgi^6s)5T6s?vH2
z@~f5gn0~l~0B2vy@*fG9pj40|*5@q3`zkvihDhKsCgi<fMH)wL9W{PR2K>0sx5mej
zZYm_h+1?s?2Dqfeur>uoT~+bGIJH#n2xw>H7w0=B;#DSL3#~*tP@=80IjRzyb<D}n
zS<jR#{pKS*i$X+jG^kl@-GG}o2Rc7HnG<RDqxe7jsCuI_TAixPYNcJoh%MfYILC}-
zd#-q8=#(h!H*^OVc}~F$qO`tC8pQ8lB`RXk#NiDZVo85mK=Cj;UYTZaO%5z=ap7{b
zR8eAp6A4p{(#K17!Rcc{R#0g6y;}WLHd!U&OOTu{O{{|3SsTf+(^Q}C=RFa)UHuhj
zP`@x<cGLfO3^E-%9M%@eQDKv)#*~kby7CfH8PACP<CFX-^vesNR6$EAq&?Jcx|Fcl
z$7+aDY^Ad0kA3wiGdcAlz`*J7*2i&>$zscZx{o1Uo-rwqIX6CN=KY3}Qt0of5X+Mg
zq)H-f{aX6kxMH9`G#4?(>}}>yQDZURwr;i1X-e-}(ML&bu~Ub{TugFM|9IIAH^#8z
z-J{yeh~hJ>mW{X}wUw;CHvv*+?O)aMD^y6=cwsB!T7KTjHjk)hsWl`uJ{P}-_~vdp
zD;l)8lkquU|AQW-Q4cQIB9F0n?_=L}a~G=lCg>!c48E8(J*~BF4ZoM#FU^f|QD9lJ
z%ijA^uK(IjUU!_YFM8i$ZnR8)Q;@bqFt;=gRV`{L=FOt!6{(>6s;wk{A_rj^1rVgD
zi+}<_3V{-**9V^dqV99h?O|rAfxJfF!LP+J0a|4SYR*ndi^>S%cnEm-l(d1GUvHM|
zAL_E|b8J;CDY&RBH-fn;cYa8C7w}PKQ~gFZt(+(#?!i;B<dWTiiU-qwW(-cHkCONp
z{uR_7K4GWFdni9lCul-$xj!fA$SA&0h1L%FamelT#QY$RSx5drHJh6G*5sv9`t*5t
zKV$oLR_!$XjR78An`k3OKLap98@Rsti5NMp4If6Da{uPQhxfzl=CLWDxZlzczN~7p
zWcGa%(8D#vQ;UZw#{iNypCZ*&Hgu=o5FVBSmdgd3XTGszx`&jkT!FoW4MHa}fk<`%
zUJMk7L5K&6LPbt2oDB7aFV*CbB2n|iht}J;t4X-M?a7&AI9`^87K*P5Id19nMf2_s
z!RcF*YF`VI&t!EJ440#Nn|R4c1)3=oLuDfM#4#+s`6gVRkdkeT;l7f#FH8aj6>2E{
zCkz?x|K{ATJzXfW)il*tSRXW-BDS;Mz82FECN63oQFG5IzP3zJv-DFE_t$aa@8e|y
z5Y-}<=1<VUf<>@Gf|x4oV`Aq!->6UG>qHn91o`%0ZRKQ`33!p9kJHUn5~4&vAt}li
z<Nko`%g~WC*Q`*b<lle`2?4hr8rKT!R$8lDpn7IdbqzMWPi5n{_f{D|-|n4nQEmt|
zU8?)vwr+ftxcx+@3IszKT(K?SpdlJc8U%O8=`Y`IGl!2c7Y>UH)0N3sztUzq_lK_(
z-LqP5EOzn8C|S5N`LmJjKdx|?P4}m(PDI~h{ec;GS-mO?K<3=C(25J3l8Qc}K1h$v
zmCObB>_HQ}m%gTnk2~X&e7qb*r6-4og$>lM&}<u;7^f#s&xoX0537<XMg-l~BvY8y
z?}Cjoak@-X&ar^)6H<Hq4b6eGndB(=GdR_Sa%-+pf65MLeeN|hXfwc4;oXQ#J@ry)
zIY6O|cr8)G9&#+9Rm3f^W3poM6l7wm6q>M#$<mVgIgfA;xN0C8mL<|x4LZNNhPi5W
z{XlhEFze>%w5g@xP}VYXlt^ILXBSre`oP6TQWfmu>3Fq%GC|)BiAo-JKF=-9<Zc)E
zzj^;~>vog8yYg{+vrCAP^2^5-g8@%~wi<zPdBQ!ts3mGma%=Ta_!AJxxPb-6$Ka=b
ztuj-VnzW4XTlxA7>gCg<KRvQGG-UT2@`TnsS+2v#{pB<3cAM&8bI_0*r)snWlc_x)
zftL;N(lHsFJyShVEybvRnOfQ+KXu^hjH8ychMy=@?f$I)i6Ci_uJU0jbhUS-2ybSe
zEiE$~5<?ehnc9wF(MO0p;fYUIKu7qpI$j*$CLz9ovm~QabFQ%l4X64u$JtGPdIT^r
zavF~%rA7E(j(sovs6{^&J8X24LOV^xD8`4+W0g-hePxag8s?bB|7JmH$*5bTrU~ll
zdrKP!-~{nf>hF<nU^_X{6&;0CMSLDc0~4+8(1@lyR&pt*qyQ1Ge)7oRqmZ<6ulagd
zVNfAup7bkaRyBx74Bj_-w!5LL>{31sLVV9f!8YRzY`<pN8cP|{12btTaWFZaiUxA>
zto`TEz=0hUTyVL3xJzMfe{On`vvt(=0NNJjGlg*S=8A@yAA0pgBmSE<XL%-8I@@<R
z&vT|~*c>oGzmNjUAZJ_si-y{J>psMk)h~tEbvvWkTRlZhDOUv9>ZHiI`NO*s#^B~Y
zCbukqoJy6r%ejwD6u7EI{{O&rNG`mYoWqk+&<6H7LtKah#O`{PNVA$h;8(~}GPgbX
z-#toDoGT0heM?u=t3XWkSFypUCa0{8<}AM>YxD6{EvLhJ2{pw7f3Z|w=ezd0)!Vnf
zS`a4`1LdFK#3{HxR=?ENeLRM<m@xu=T<G^yF`Aar4z#n+e;A=^eUVrn)X%ObbuQX`
zF(q33m0FdKKo`4o;km4CCublhdUh`SrB*k!?jBuHAUdlMK9)kN5GE3}Ea7^glc9FY
z7s%U~O8X6O5Z5jVG^j~I=p4|NR5N~bajSXRYD2Uo88xQv{siZQv##jPI>vkdj$HjL
zT^Q&6F<-o#I)*N&t=w#Tx!rat^ScI6Oz<ET#E~&m0Z(!4Jga|AC($QbHiLAmZkiWo
z*dA46F5i<L$ybEmH+}p;xG)f$A0yHEsq!wzHz=xZyMG=Uzfo9xoLI|JH6=AlQ~4d|
z7Yh|%#kT8}FrqE`M*p$s*FwNDGT%`ecKpn7*df*?{PTX@5PvL>$<V8}fy@8G>sMTz
zhy<R|xe{UAyr8nZ4pZ!mV0V!ys=<B}d8LnU4Z5BIvK^gBtSY8y&-bC~0`XU<c=)Fl
zzUw>dw2C(4Yj*}=u{V7sDq9c=S!gF>J7$Onzhj2;DMq5XHli{%&&i7Y@^HewgN_55
zp?Dw`a|kCnIEO=QkqA}M!{96)Ye<v50`vl!?k`#&=4jM;@+kro^XE20RY@3iO={Li
za$)u6$D@9B%>wbR-#VQFVXPZZdHUg?^Hc)6)L7?qoj|{cw$^KAGT)6^N_05`;fV0W
z!?Pj4Lp<P?uU7o(*=;1zu@ZrU3Loi0W3Lw9&t0|?Epe;%<}Y@}4;2P|*-&z7dI<5^
zzqt9pjw<}1EV!J*2kBt7BxTrO7Y31S#BRC*^Lm{<i`1{zu~#~!bYgrKz|t85>_2lo
z-O%MPg;}iEo_auv1P38&oOmJVRnQB_*m%fgk$Zk<+h5H>RTbbk-**hxJP?%mjHn4t
z;7Mrs>l8_j5st29^y2@e#pf9<h{2{og3^Er)d|t^{L@i-XN4`Pt?B*l7S3$?Pde_u
zBdeuiFS#=n7vPm00zLLqbzJ7D?!vjc_(Cc~rN`p{K~}C8bRG!QfKHRMdE#&Pzvq;h
zi0W64bCeHEQCLU){Ej_kpE}d6-${8ruQW~yz&!tXnOr`q#c5uaZQLQ<!Ftxrn}n#%
zvmFdiyd|RQRLsXGCUTD|uX6VuI4@^uGD;wI!~uSYUPxWu=ilhj;^MpS{PI}i8$D;L
zW7cdOkEG)}QB}Zru9JmUNz84Xgs8}_Yc9aN`6XX~rSrUA^bi|zbk#uPKRYx_m%}J#
z(<d7LHcYWO&1V*Cx!X(BEe9|kb&6tr(~mK<+lm%1hl_`ySm_`Q@!ww>xDV(*hFYd-
z^x2iDHBgRmE-~Dl&#{_~tU>tkZ(y3}KU$|{Bh{p|7iD=2A`f<0NmHD{f|=>p*C?Ug
z%gJ%L=G!hN&P+c&?B;C^eo`akW7kUMxOV;g*;hiRoE^wKYwb;B-)&3P?t9jdfzQ%^
zyKU2(;kS@OOZPhp>P!;oXeq0rMC|+{+K{?CDj&-W4$LDkk}^D@*ULtflt5lHEk|+N
zlejl4GdI8~yl#UM<JIb#v*z|D4B-Iu6*0q`f^s_co6V)u4<@W1R71xzTer%Bk3w^R
z&l+VYKoiB{0}5ki!2L-|QpT&0n-&o+8d>>yHefV@#E6z?u=pkYsjINKTy;NvlkwlE
z+4VXbaI#F}zr#4f#gd}~EU33XqSQqVHQMnyUf}Q>3+K4db~08PLhUDURbBg~i=N^Q
z%1@=deEM{i$S&BZik35EtumgoUslt#VD;Pwzp^I0qxRex0{r(~z$xXBr~Qo@qV*%c
z&wq^!!`BWR!V?mhB`Jb<SFjnRlq8dr)1J1HkaPrjCGW1^8N{h4`F#1VPft(Sxv;+e
zA)@h9>8wtVR&3A!bR#kp2j6}!2jP3MU~EMH`b-@$YCmMrvEyA?#&v%zG3~xj9pZM9
z(D{JQw?sDSw}-_(+)G!;s({$bAteljI$5WU7BIUP=?kMBb4Rudj-{0{MwBKrQug5;
zUel>wwYcs%`3G7$En(Y*x)TXR8jQwWkcu=H=7UC?GYd6LIH(e77PI0MkpR-`1#DD|
z`@7qn()yOp$T68r$KH|6ElC4eN&G}s&#j#1j*l1*DsL2U;|%tam)Pp7OPPO&2MV=b
zr@epGkFo@(kw1sO?VOvBg#f@>vke^$q^D_NA=w-(%|e0oUbVOR->CER2~ie_1?6>B
z(Z@J|h(<Uo)`HW}=@=wA8Iwpb-shELtQYFFTNSk28|!5bUI-_sQ{w>aR5$bP*NO8m
z_X#dU##Pw<{>2ok+Sz6rx%9vBi{3E+eXwo3LztndgWiGc-IgRNJp7g)Mp_JB%>}>y
zO~|vPWBR$~l|>GuJ{U^Fmx%zwMwOG#FQ~u02aCH{n?3j8vkE<AE2(YvKqk9iIy_Qa
zgJ3(u<bn57&<T3G>tv5yBK`%oCPaB7zIPJUWahpxJQtM*(Xq;FkY&<*JuQeAV<(SY
z5<gA-l0V6BfX}sxZ#_YtKqq#j5BDaH_CGTlQHq)-7~uN%&aMr<*aNm@sn3FDv+FB|
zVPmQZ$($0*r_yp%f!dwyNPNC`w4gxZ@pnA0)gh;B#7NbyBK-~&|6oeC`TW<v8cqgA
zND7q<Jr)k-4<#mk+pM~2darf_3DyKEJ~Xi}Kz5dxiY`B2>Re2Pg^2-TR!neBM5k|H
z?P9$Xz*9mlsi~!w9sdiXx<LJSDz@?T=?u;SngzxGGSt4GT)QjupvarS5d~XH`AG-S
zA9bX%ArJ04LW{oVjJ8w9R~vDRJkqs~v5)Cn9IrGLdGi!^HS!YT(mArt-`rC3=*;l)
zB)#nPU!Ttg$ipw>Vx43!*gppZj2!hN?-eyGUK1tD+IKkAPR$$6P21ms{!mP9_?0^;
z(w91YXm(#TvghJ^D`pF1T4Dg~0E|H5C2_?E+|*s_7UcfQ?XwVU$!by60U|Hq^n3_V
zL@v`j$~u!{4J0kPMMU5)DS>H}|AEU*M7=BR5?=C>8?Q`X703zj)J_E&+LIcs#0`f~
z$_!E09cS}Qa+jrr&g@YRg8oU*4nj;bUv4$BQ|F$X_5z0B`$g;~^3L=UHOfB(W(;1c
zY8j^?s_jr|cZsYU^ok;U)CgRJ)%PtZW6Q>(WLMO-w^3<!4%d?OASxJmg_q0;nxvv-
z_lI=M-)H;J@lmPfTfr)qB5f@>jkzdQS{n%?;o<aQx5TdTmN<3LCpBKc{6Vy<Q-n;Q
z4V(>?H*gSsu*MpJ!S*grrN--DF*SjYm-w%T0XaTt-2upA_tEFuM_0Q0wD<hi5vA?O
z<|8Vxtvi`IyKioQDqMl}DDuz#$NEl|W^=B87q!MC6jzk|ndl@p<fnd|*>z#$AQAl^
z-*MbnnyKzpaSmYuS;s5D`}Y^zwSqFjYA>EyD*0hoY>h;lYr4$Ob&KaoIV~k^vrE69
z&{u2x0o$={W(#{R+m>DgWITLE)Om<$dtAFq#0Aey#Gmkzy29GZ*{iUnuzFzs$Dw|t
zg!u3idY5U@fgZi^E9C{;XupGP>;vDcss8BpGfRvL`OL78+TnG~WzEngij}F32xm)!
zqnr0jpa5Q%^WAHOp!_YH9@g=^H`SIq(5=1)BaQDaPVW{s!pH|f8Sa04d|_#3LL=bD
z|K!_L)<w1U@3-n|rv9xg9(aw1LS&=NDr`HFR)No>{Pr+N9kDU=OV>)oq0d_&VrgmJ
zJvtTcw=#lerTGsMJ<RA~b=)!`ACzu^f&jbbVgN5@3mSAkd&_!l9l`nCW8>hFMjON=
z9ZhG8Koq7^>)M0OMkydN0m}f{N%^x!S10nC+=gMLLSiW~8+vDzd~1=Mu$Z7JzN&cj
z6@X&}M4ugLFNQ@?^BOH2yUxaV&F$H8yX1GIweAN$9vg;i2%V1jy&2^G%>40U=2$Bb
zIY{GqjNtu>IN};o><B|k)V>Z3K^RsbCEHsyE<c50*5})zr>0@(W$bPvNh%U)fMJBI
ztzL90`6mKOcQARCR`hLJxoJNcH%Z-@5fwz9AnKpI+y&56_Vs2bH6Dz0p0V>nUKlz0
z^GKf~rh21M+<>hQ7isP#B6I6>vOuh&i%b%13fF0xM5ox-!5$;bJ{78`mJ+wv`<%56
zu=GKt3Z^eH-f{er1&GUXpHnVeus4bHN|xc;=t{`u>Fdi(H+V66U?l)L|Ek5qcz7!O
zz1P<@l6G5_ab<WoA<TXnCH|`sCclkTp9z+S7WL3Cpegm{x&eip@W)87w&b<3VX380
zLC8Teukn(j#Z+bCaQl7$<1YUJ$J+q&=CBmAN10kcF%dnV7hG^RF4`INRpowOm55>$
zo`9oC1db;8XH~JV50g|r)$aEq9$g|LYSZ^|R|q9&rXr+4<fq`%{xkZG-CY<yi!T&j
z^NI;b%XgC6Mr4;>uSZi_QgOh4fzp%z?l@$Cx2?~@{+<Te(e});-<r*nj}B-KTpmC?
zT~g`UR)Kc{Z9w>Q(>zyHk1$!e<&3-`Hyqd+xs$(o@bLlvJ=o<q_$a|wVyp=g`V1lD
z;hFz?K0%ThIbZa@bCz3*RS2|G|8sJ|#pyqgVJ~NK>F5~SIs0ej*pW%0b+->2uWKAU
zd%_6P@J&<ieM{gJglDZ2WrvheNB#Y#l1U_7oKxc0hmeU@a!HjmMiqHJPK!fvbVZ9Z
z9EKl${@Dlug*B-E<3%RZ@|!sO=tp@Zei43IL=oh^WmfeKnC1}@j>R7F21b2&!$ZZZ
zZ3tT!{a#Hh>2)v>RYZqHPNq2q+A&W7c5+nha+hrgY+kAuiL<s_>zN_xZS?^5aCcm$
z2ce%*^fV2jBMG>APXJBV0iroj*-DHlrQu+!aR^+bRY@e-nOL|SH4c8YJQ~Xl3F(Q^
zk(*VrMT5GnS2`A=iX;s^;g@Cns>^QP#fp~-qBiGjmbQ2mTgpwg^#Avv#F27-Rjz7I
z?sD+B-V6g_s)41SHpRbd$YPU`xI5R4cToK8W*S23;QOG2-TdQ6no&$CvXZj@_B|w_
zYvc9{_0+wIHahJ8&Ic8vYAgR5!FFPNhL*KEPn;<>mZn!ad15byhumr^DiaxE4p9lx
z|0(Ru<DuNU04^pm*2t2C7|cwCCX^-al)*%pY}chy=vJ0)mh9UIV^4_4nq@4>MY=+a
z?b?!Ly{<J%5tFi1gZB)#`);4l`_5m#&*xd@{FZZ`^PK1WzGr#YKBuX5T*67WE5TM-
zzbZx|d+I0B6S|1V_F#|G19?2(Z}(~~M*yjF<ovtON^(!K_NYss&xY;J$q9d>IX=l^
zhZFqtNbv}pW%XQZ84meDrSY6c&NVbP=V~Vr;}!VavRuLWdpA=cm0cgr%IVNt;T`JG
zb2vTCMSrg@1(_;e78^(M!^H75jb(f^jB=n&#Z?WZj&V}6sS+;bF)R9=ooHOWB-$!j
z6^%t)1+cw+bji*2cj>F$`nDClT<r<#`ue2(;zj<>zg|^%5R_z*B9WY&Y{ea7Y}8X+
zP0d(zH{@vRBDTiDM(?N0HH*`{$Lg2Ri8hRS9^&$DdM)yS2_lzDA?iKkaKYa*K_fkh
zbrl_~SQU|^1*=CT&3;!V&d<~A%s9w>yFsPnv#D8ApRwgZ?FJlOKHd%YY7r^YMHQyp
zjp7LjC(E;qq6@RB6OL)}2fhrq1Y1OiXln?_nSUP#<{J8(>ltuaxy*=i;+%7x$_+7f
zf?4J#0%`x=Qa4v}#@F-mKN5M<?La?7hp+stPvt^FSyf}$4w)EhllS@iPTl{J^fPH#
zlk<f3D|#PdJv`0XDKk8;rF;n&`A{-E<9?R0!U#Y%{?bMWE79L1>iO85CmtEc%dpDL
z$zeZ+&u(IDU7ONPGpT<h7^uMa*zNH(w2OQ{!T;g4oxAaw9u3^o^WebEVo8wh(*1>Y
zPM_uj)TSnPTkdGoE|Z`Z<JdtN^VQA%_XezNv2os9#nQ>w48Qj_rFBoz(%)tPOg4mL
z@b+AFS%w~qw2KdZUu#Cix8(YbRXkyL6Fxlbzvjh%BAv+++P0iP8)&<WGJ01z{OOtF
z%SKsLBo+1MlSEyB#?7fjD=y)8R>Sh;-qRDzTps3YCj1ls@KQBZndiCOi<A?dAIp!u
zsj95K<jMp>ZJ9ZU!%&>EDIG)YcC_*&F3XYuC~$4t-+dOa*i3V|C0|r7)$m-c@I#vu
zb_-+?pT*Z!gv59i=DQud<3l}Asq)Z0{oSYZ@FccfeARx7q~kK-xRNmD%IqKcExDlX
zc9<;oSB1dG2Vg_`?VQxdwl60xHigHEubX^2<HWmtE<XZ)n_bs+F!OHqa2e^r&t2A8
zj#v3SVAmwG7YMwVD3L^)&eP!Y*%<Y0fB$%3EEp5&*Upq9REmSFth9$DJ1w-R=Cia~
z8dx%1*)JoVK@&R73q&z20?(ft3@O|k`9GJKj=r3UbE%0H=QDUXY6W4tt%I;TAn4Q&
zLybbkzKs4k#zWu27S^p~EDo}iKAUMFRn%tE(XMm1UF-6G`U|!*`~P_`p;eG@xs3d7
zFNQF4XBlNg)bVi<l?lK5ds2xdA51SlBHj_^pDabO3EAq&v>1~b!67{es}cUZntiJS
zZ7N%$=#)ZO8cud)nw<Z>G!;uoviiCMj^OQii?O5#M^SF}5Lz3sZJ3StIjG_V4Vgi3
z)1jeT8p5oW1GO_Cph$y8HE$#`1RAdUp+XKcrq$8dts8j!zZ<+2Bqb%@m1g#%3o1Lb
zq~=gM{-g<QNfK*mOZm}sTYez*{SW#7U)M33tpmJq=WET_xeqy^$2G`O9k3E7y&dhV
zep1WHexUbd%%NN3=$y!-v9_ahUd&?kj+ZK`1-18UKaTarE@kB13a;jv$(!NvH$B+l
zYXRA+NGLNKrs<Wh5a6~X_P9V14VP5k{8qZ=Fs%}^a?iV@6ca%7eqN_%j9IjTxFkjs
z-o?l{0`<|g>;wYLO&g?-a^p!67j(}<Dkw~$J-8CExSJJa9sv>px?7GQ$bz>&l^hRK
z+6J>~1^RaYjs4i!1~YGOVn#Ux;h2^A;nx!rw?AK%mX^Msp8g_TLtdiNXBp7uHa0fR
z71}X1HQxZ$+WcA!4LltLf#Ch(EO_EzDUor?IvxNUZEMrl%3PeC^_m-aI45amXP0pQ
zKKD~H^2j`ZGH=|tL5~QwY8J&Q8JL?Vi4K?m(hh{^@VyIfmxTbN@XD1h>g_%2?EC>h
z-U)<R>eABE{QUgN!YCJfoS|POSzeF+@EEEWHAe{2*(xh5OUx$eulolF!*y4Hh^xdI
z;$OtU7>XcV6HwdT51eyJO)nPz72s^P<QcoIE}ZlA^*wc}>49FH#PMH#(9_dnj0T7;
zRaiE)6;OeJ%$|v`wjMdH%?SwDz;7#u88KdOyZ5KulBBGqZ5|R)u#pljBr?b9sxk?m
zq2-^Nnu>IN3YcI>w!J1>1W4tR2FU!tmHB(Wp?0qg1JfTRCs$NhNJGt@mg3JXC~))_
z9*Px=>O;9o7kSJL)CH`IRVf0q5vZvE3ZI&uMz~!Z;Ta!=L1r)*zuCCx>r>WzHv%&>
zz0VdKozo)htUvCqp58QB>-F)Z&7C$N*<MsYJPT?E-f@HcfrAGP7&5o@^R(NG;M;~h
z5grOK2DbUp(a~>TWHzB7q?Pl>lk1t))oKj0_4V}@*@%WiL#}u|f;+bqRx6V)&Et|w
zx{P+s25>faiiPo%=F54cJ#!o^ot?d6a(qZE5gz|ldb+y0Q@cH4{35SGeh#ETI5;@i
zUb;b)Zf$L)YXHEORZ<l8yU^qh$jODJSifbiw%(K+vIUEJ(a_M3fL&tBK|Z52yDNa!
z@R=?sY^nXGEnWtX#{<iDTDSIH1X8ue!os4ZrA0zQVybtOEpS!oUKf>$zI4GNL8Koj
zqj>Gm&`>oswH~2|Ru#IMcFXT!D4JHgFa7H7?oNR(S4=I5k;4w9Jguo|dUX5o<7L1K
zXPpG*{LItpYFbqox0C&o>*Gd;`h%b>6Fyc_n>RH)GX=Pu({DHfx_hqa=;+wg`6N-5
zPmu6^eSNkgCm+0=s6Z+Bb2~w~X~1mv{6w=)5D`M$LM&h~m?KI`LG`xH{r$z0ib-8v
zhO>%~F4P-AHWJe|#Ai?^2Af)1cB(3PYPXJ!WzcLJBsb`3r?_Qwb~Zsq5v%Ma7nk$J
zb#GM+1^N47CH_F7i?BF(0{NLD&A?2|&vVzi!|>)#q~%g4*aXGIko!`B`Wt0XYzaHs
zj;X2b_2F7_%j0Ik)klCV+}Y9b!^pm4E~rR4v8<w^c!-gjmXL|qBLtc7pkz>EEG0di
z2TPV?5fB&WE+sq1c=0cMEv}<Vfog=7?g_X8sW?(L9`i#-hPd+z3Iq#C{5wcV$;t9^
za_1S!)=yl7;;2^Br_JbZF4PC`@W}ezZBu3K;u+a5DJi+u&JWrO6Vy`}RsSI__~~@)
zY^sh6VP`NOi=TjP2-o=WFT|iybs3r0P@|w(l!JoQ;-o*xS}UM$_N^@O!3TT;u}`8x
zvS4H=NHLDt(-&W{1!`RXqmmN~?Do#*&qL1~b)rNZ;pb1%M`1CTU<C`@oj^3Gl`NLS
z>E=B{ImQlm#@dY>*0jqKajR1;Q2={q%ETJXQl&R_{hM85V<W{sL?uo$3i&KKD2Qdv
zjelc9W@DqLzdy8FAo5Off0Hpc2-S(+z%aCv70wAjcbS?G?|Zk^Dc4@N#sF&iz=rM^
z(597zfw?Nq8-85mo?*hD%e#UXXq}y0RXA9}N}R5&*J+nm0ma_TLcz^{4QaSO0LeWK
zztf3Bb-u^^1pXWyoqQ7Ey3D%p6$1WSkAgnD8ULE8`oV)-0W-Eyev^%%VCg=8c3+xM
zi_gav@$6Ft0CZgEjGVlDQ~vP4fUwqO&=as(B5@{SB)plq`B5A$>moUh_T|&t$)8yh
znmwL@iD#}vrua`Iu7Pz{P*5oIh8(b)9v-3N>Q@v|Px@}>W3e2{#yux#p8j8!X25z$
z#CgInw!29s>6xXeI=i~Ml2^pE*!kDY0h}E&uJv&~4o`x#(op!Ca_YTY5fl_?tP)l2
z0nHp99`^S3270-)F)OO~7U8k)stdrI*A^y@vX5L5i_wj*Tw>0<ywQCr1*)_bXOGKS
zEl3=OlI3*$Q)VC`EaP#&;6rz0H~aGJo5Fa?A|N;F7iVKav2|)RRm!@`Df`vCvK;~(
zh%z2^kN1gQt(&iI5q4U?g&GuY)qj?k<AxSy=jKd}Y)vId_=#k+mX<`o9y*Z{6O*JF
zL+02v6TSRNe)*UP<i?(!moGt;gD+?vKK$^7v5HG$4tajGM&u7nyguSN!rN7KZ=Qe<
z<G|Y2<+-V;OKYeGy{S(JXGnjKta=nu)fhO?tougmd_5uUG4Zg9JwTtKBO8oGsA&WS
z+!--63oQP@M;A8z#&OECfQJsV>M4mb@Q7gm@i_AqXl;Z1MX;=yfdZh+?3uTI8b#Oj
zet%|r2%A43Xc79aKz4zO(O*jvs_-zK1vD0i#_HS!uNJ~B4BQg9GJBcI1knI^{s-*!
z=824i%GU9+|LH&6@w{xYD^A>;y^<#yGhWm6uf7DoECk*Ez80UDD6e>-&}Bs7h33ni
zoZ7eS)7>GzCnhHxt90i<FIt_}9PG`mt*rex`^8|}yV-IpWiZ$t!EJX@@!vho)DA++
z<S*#gOg~dc?k|5cOu7g(6vlYl4OP^SM1DpI3n^P9-hkV2NJB#V_uEdf2yNe2MoUB}
txzm|FM#w=N&LOuUh@{=$)_=2E*kqAV>Wv<iEYD+sACX|BQ+m>t{13q)j6eVY
diff --git a/doc/guides/nics/img/ice_dcf.svg b/doc/guides/nics/img/ice_dcf.svg
new file mode 100644
index 000000000..c6de820a0
--- /dev/null
+++ b/doc/guides/nics/img/ice_dcf.svg
@@ -0,0 +1,516 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+ xmlns:dc="http://purl.org/dc/elements/1.1/"
+ xmlns:cc="http://creativecommons.org/ns#"
+ xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+ xmlns:svg="http://www.w3.org/2000/svg"
+ xmlns="http://www.w3.org/2000/svg"
+ xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+ xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+ width="226.02977mm"
+ height="174.625mm"
+ viewBox="0 0 226.02977 174.625"
+ version="1.1"
+ id="svg8"
+ inkscape:version="0.92.4 (5da689c313, 2019-01-14)"
+ sodipodi:docname="ice_dcf.svg">
+ <defs
+ id="defs2" />
+ <sodipodi:namedview
+ id="base"
+ pagecolor="#ffffff"
+ bordercolor="#666666"
+ borderopacity="1.0"
+ inkscape:pageopacity="0.0"
+ inkscape:pageshadow="2"
+ inkscape:zoom="0.35"
+ inkscape:cx="415.71428"
+ inkscape:cy="-62.441805"
+ inkscape:document-units="mm"
+ inkscape:current-layer="layer1"
+ showgrid="false"
+ inkscape:window-width="1920"
+ inkscape:window-height="1001"
+ inkscape:window-x="-9"
+ inkscape:window-y="-9"
+ inkscape:window-maximized="1" />
+ <metadata
+ id="metadata5">
+ <rdf:RDF>
+ <cc:Work
+ rdf:about="">
+ <dc:format>image/svg+xml</dc:format>
+ <dc:type
+ rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+ <dc:title></dc:title>
+ </cc:Work>
+ </rdf:RDF>
+ </metadata>
+ <g
+ inkscape:label="Layer 1"
+ inkscape:groupmode="layer"
+ id="layer1"
+ transform="translate(4.1577365,-1.8005958)">
+ <g
+ id="layer1-9"
+ inkscape:label="Layer 1"
+ transform="translate(3.0238119,1.5119048)">
+ <g
+ transform="translate(-2.6458356,-49.514882)"
+ inkscape:label="Layer 1"
+ id="layer1-7">
+ <g
+ style="fill:none;stroke:none;stroke-linecap:square;stroke-miterlimit:10"
+ id="g4657"
+ transform="matrix(0.23544767,0,0,0.24253472,-4.5357128,49.803573)">
+ <g
+ id="g4558"
+ clip-path="url(#p.0)">
+ <path
+ id="path4382"
+ d="M 0,0 H 960 V 720 H 0 Z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
+ <path
+ id="path4384"
+ d="M 72.002625,606.8819 H 876.63256 v 77.03937 H 72.002625 Z"
+ inkscape:connector-curvature="0"
+ style="fill:#b7b7b7;fill-rule:evenodd" />
+ <path
+ id="path4386"
+ d="M 72.002625,606.8819 H 876.63256 v 77.03937 H 72.002625 Z"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#b7b7b7;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
+ <path
+ id="path4388"
+ d="m 164.20998,627.07874 h 104.85039 v 37.88977 H 164.20998 Z"
+ inkscape:connector-curvature="0"
+ style="fill:#434343;fill-rule:evenodd" />
+ <path
+ id="path4390"
+ d="m 164.20998,627.07874 h 104.85039 v 37.88977 H 164.20998 Z"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
+ <path
+ id="path4392"
+ d="m 206.1529,653.9987 v -13.35938 h 5.04687 q 1.32813,0 2.03125,0.125 0.96875,0.17188 1.64063,0.64063 0.67187,0.45312 1.07812,1.28125 0.40625,0.82812 0.40625,1.82812 0,1.70313 -1.09375,2.89063 -1.07812,1.17187 -3.92187,1.17187 h -3.42188 v 5.42188 z m 1.76562,-7 h 3.45313 q 1.71875,0 2.4375,-0.64063 0.71875,-0.64062 0.71875,-1.79687 0,-0.84375 -0.42188,-1.4375 -0.42187,-0.59375 -1.125,-0.78125 -0.4375,-0.125 -1.64062,-0.125 h -3.42188 z m 10.7717,7 v -13.35938 h 9.01563 v 1.57813 h -7.25 v 4.14062 h 6.26562 v 1.57813 h -6.26562 v 6.0625 z"
+ inkscape:connector-curvature="0"
+ style="fill:#ffffff;fill-rule:nonzero" />
+ <path
+ id="path4394"
+ d="m 363.36746,627.07874 h 104.85037 v 37.88977 H 363.36746 Z"
+ inkscape:connector-curvature="0"
+ style="fill:#434343;fill-rule:evenodd" />
+ <path
+ id="path4396"
+ d="m 363.36746,627.07874 h 104.85037 v 37.88977 H 363.36746 Z"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
+ <path
+ id="path4398"
+ d="m 409.12286,653.9987 -5.17188,-13.35938 h 1.92188 l 3.46875,9.70313 q 0.42187,1.17187 0.70312,2.1875 0.3125,-1.09375 0.71875,-2.1875 l 3.60938,-9.70313 h 1.79687 l -5.23437,13.35938 z m 8.72482,0 v -13.35938 h 9.01563 v 1.57813 h -7.25 v 4.14062 h 6.26562 v 1.57813 h -6.26562 v 6.0625 z"
+ inkscape:connector-curvature="0"
+ style="fill:#ffffff;fill-rule:nonzero" />
+ <path
+ id="path4400"
+ d="m 527.15485,627.07874 h 104.8504 v 37.88977 h -104.8504 z"
+ inkscape:connector-curvature="0"
+ style="fill:#434343;fill-rule:evenodd" />
+ <path
+ id="path4402"
+ d="m 527.15485,627.07874 h 104.8504 v 37.88977 h -104.8504 z"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
+ <path
+ id="path4404"
+ d="m 572.9103,653.9987 -5.17188,-13.35938 h 1.92188 l 3.46875,9.70313 q 0.42187,1.17187 0.70312,2.1875 0.3125,-1.09375 0.71875,-2.1875 l 3.60938,-9.70313 h 1.79687 l -5.23437,13.35938 z m 8.72479,0 v -13.35938 h 9.01563 v 1.57813 h -7.25 v 4.14062 h 6.26562 v 1.57813 h -6.26562 v 6.0625 z"
+ inkscape:connector-curvature="0"
+ style="fill:#ffffff;fill-rule:nonzero" />
+ <path
+ id="path4406"
+ d="m 683.36743,627.07874 h 104.8504 v 37.88977 h -104.8504 z"
+ inkscape:connector-curvature="0"
+ style="fill:#434343;fill-rule:evenodd" />
+ <path
+ id="path4408"
+ d="m 683.36743,627.07874 h 104.8504 v 37.88977 h -104.8504 z"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
+ <path
+ id="path4410"
+ d="m 729.12286,653.9987 -5.17188,-13.35938 h 1.92188 l 3.46875,9.70313 q 0.42187,1.17187 0.70312,2.1875 0.3125,-1.09375 0.71875,-2.1875 l 3.60938,-9.70313 h 1.79687 l -5.23437,13.35938 z m 8.72485,0 v -13.35938 h 9.01563 v 1.57813 h -7.25 v 4.14062 h 6.26562 v 1.57813 h -6.26562 v 6.0625 z"
+ inkscape:connector-curvature="0"
+ style="fill:#ffffff;fill-rule:nonzero" />
+ <path
+ id="path4412"
+ d="M 72,440.96588 H 876.62994 V 581.18634 H 72 Z"
+ inkscape:connector-curvature="0"
+ style="fill:#cfe2f3;fill-rule:evenodd" />
+ <path
+ id="path4414"
+ d="M 72,440.96588 H 876.62994 V 581.18634 H 72 Z"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#c9daf8;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
+ <path
+ id="path4416"
+ d="m 840.4199,529.82153 h 36.62994 V 567.7113 H 840.4199 Z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
+ <path
+ id="path4418"
+ d="m 850.81055,556.7415 v -13.35938 h 2.70313 v 5.9375 l 5.4375,-5.9375 h 3.625 l -5.01563,5.20313 5.29688,8.15625 h -3.48438 l -3.67187,-6.26563 -2.1875,2.23438 v 4.03125 z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-rule:nonzero" />
+ <path
+ id="path4420"
+ d="m 242.51706,463.70865 h 168.8504 v 82.11026 h -168.8504 z"
+ inkscape:connector-curvature="0"
+ style="fill:#9900ff;fill-rule:evenodd" />
+ <path
+ id="path4422"
+ d="M 308.82507,503.42865 V 480.5224 h 7.42187 q 4.21875,0 5.5,0.34375 1.96875,0.51562 3.29688,2.25 1.32812,1.71875 1.32812,4.45312 0,2.10938 -0.76562,3.54688 -0.76563,1.4375 -1.95313,2.26562 -1.17187,0.8125 -2.39062,1.07813 -1.65625,0.32812 -4.79688,0.32812 h -3.01562 v 8.64063 z m 4.625,-19.03125 v 6.5 h 2.53125 q 2.73437,0 3.65625,-0.35938 0.92187,-0.35937 1.4375,-1.125 0.53125,-0.76562 0.53125,-1.78125 0,-1.25 -0.73438,-2.0625 -0.73437,-0.8125 -1.85937,-1.01562 -0.82813,-0.15625 -3.32813,-0.15625 z m 16.75,19.03125 V 480.5224 h 15.70312 v 3.875 h -11.07812 v 5.42187 h 9.5625 v 3.875 h -9.5625 v 9.73438 z"
+ inkscape:connector-curvature="0"
+ style="fill:#ffffff;fill-rule:nonzero" />
+ <path
+ id="path4424"
+ d="m 302.04187,533.74866 v -17.1875 h 2.26563 v 8.53125 l 8.53125,-8.53125 h 3.07812 l -7.20312,6.96875 7.53125,10.21875 h -3 l -6.125,-8.70313 -2.8125,2.75 v 5.95313 z m 16.02344,0 v -17.1875 h 3.42187 l 4.0625,12.17187 q 0.5625,1.70313 0.82813,2.54688 0.29687,-0.9375 0.90625,-2.76563 l 4.125,-11.95312 h 3.04687 v 17.1875 h -2.1875 v -14.375 l -4.98437,14.375 h -2.0625 l -4.96875,-14.625 v 14.625 z m 20.07031,0 v -17.1875 h 5.90625 q 2.01563,0 3.0625,0.25 1.48438,0.34375 2.51563,1.23437 1.35937,1.14063 2.03125,2.9375 0.6875,1.78125 0.6875,4.07813 0,1.95312 -0.46875,3.46875 -0.45313,1.51562 -1.17188,2.51562 -0.70312,0.98438 -1.5625,1.54688 -0.84375,0.5625 -2.04687,0.85937 -1.20313,0.29688 -2.76563,0.29688 z m 2.26563,-2.03125 h 3.67187 q 1.70313,0 2.65625,-0.3125 0.96875,-0.3125 1.54688,-0.89063 0.8125,-0.8125 1.26562,-2.17187 0.45313,-1.375 0.45313,-3.3125 0,-2.70313 -0.89063,-4.14063 -0.89062,-1.45312 -2.15625,-1.9375 -0.90625,-0.35937 -2.9375,-0.35937 h -3.60937 z"
+ inkscape:connector-curvature="0"
+ style="fill:#ffffff;fill-rule:nonzero" />
+ <path
+ id="path4426"
+ d="M 72,240.5538 H 876.62994 V 400.96325 H 72 Z"
+ inkscape:connector-curvature="0"
+ style="fill:#c9daf8;fill-rule:evenodd" />
+ <path
+ id="path4428"
+ d="M 72,240.5538 H 876.62994 V 400.96325 H 72 Z"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#c9daf8;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
+ <path
+ id="path4430"
+ d="M 103.58006,269.6063 H 527.17061 V 370.64567 H 103.58006 Z"
+ inkscape:connector-curvature="0"
+ style="fill:#ffd966;fill-rule:evenodd" />
+ <path
+ id="path4432"
+ d="M 103.58006,269.6063 H 527.17061 V 370.64567 H 103.58006 Z"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#ffd966;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
+ <path
+ id="path4434"
+ d="m 109.89764,300.6404 h 68.22047 v 51.77954 h -68.22047 z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
+ <path
+ id="path4436"
+ d="m 120.28826,327.56042 v -13.35937 h 2.625 l 5.45313,8.92187 v -8.92187 h 2.51563 v 13.35937 h -2.70313 l -5.39063,-8.70312 v 8.70312 z m 13.45733,0 v -13.35937 h 9.15625 v 2.26562 h -6.45313 v 3.15625 h 5.5625 v 2.26563 h -5.5625 v 5.67187 z m 14.7866,0 -4.78125,-13.35937 h 2.9375 l 3.375,9.89062 3.26563,-9.89062 h 2.85937 l -4.78125,13.35937 z m 8.68447,-10.98437 v -2.375 h 2.5625 v 2.375 z m 0,10.98437 v -9.67187 h 2.5625 v 9.67187 z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-rule:nonzero" />
+ <path
+ id="path4438"
+ d="m 194.51706,290.53018 h 104.8504 v 68.56692 h -104.8504 z"
+ inkscape:connector-curvature="0"
+ style="fill:#00ffff;fill-rule:evenodd" />
+ <path
+ id="path4440"
+ d="m 227.57297,317.4502 v -13.35938 h 2.70312 v 5.26563 h 5.28125 v -5.26563 h 2.70313 v 13.35938 h -2.70313 v -5.84375 h -5.28125 v 5.84375 z m 12.86357,-4.96875 q 0,-1.28125 0.625,-2.46875 0.625,-1.20313 1.78125,-1.82813 1.15625,-0.625 2.57813,-0.625 2.1875,0 3.59375,1.42188 1.40625,1.42187 1.40625,3.60937 0,2.1875 -1.42188,3.64063 -1.42187,1.4375 -3.5625,1.4375 -1.32812,0 -2.54687,-0.59375 -1.20313,-0.60938 -1.82813,-1.76563 -0.625,-1.17187 -0.625,-2.82812 z m 2.625,0.125 q 0,1.45312 0.67188,2.21875 0.6875,0.75 1.6875,0.75 1,0 1.67187,-0.75 0.6875,-0.76563 0.6875,-2.23438 0,-1.42187 -0.6875,-2.1875 -0.67187,-0.76562 -1.67187,-0.76562 -1,0 -1.6875,0.76562 -0.67188,0.76563 -0.67188,2.20313 z m 8.45848,2.07812 2.5625,-0.39062 q 0.17188,0.75 0.67188,1.14062 0.5,0.39063 1.40625,0.39063 0.98437,0 1.48437,-0.375 0.34375,-0.25 0.34375,-0.67188 0,-0.29687 -0.1875,-0.48437 -0.1875,-0.1875 -0.85937,-0.34375 -3.09375,-0.6875 -3.92188,-1.25 -1.14062,-0.78125 -1.14062,-2.17188 0,-1.26562 0.98437,-2.10937 1,-0.85938 3.07813,-0.85938 1.98437,0 2.95312,0.65625 0.96875,0.64063 1.32813,1.90625 l -2.40625,0.4375 q -0.15625,-0.5625 -0.59375,-0.85937 -0.42188,-0.29688 -1.23438,-0.29688 -1,0 -1.4375,0.28125 -0.29687,0.20313 -0.29687,0.51563 0,0.26562 0.25,0.46875 0.34375,0.25 2.39062,0.71875 2.04688,0.45312 2.85938,1.14062 0.79687,0.67188 0.79687,1.89063 0,1.34375 -1.10937,2.29687 -1.10938,0.95313 -3.28125,0.95313 -1.98438,0 -3.14063,-0.79688 -1.14062,-0.8125 -1.5,-2.1875 z m 15.71948,-6.90625 v 2.03125 h -1.75 v 3.90625 q 0,1.1875 0.0469,1.39063 0.0469,0.1875 0.21875,0.3125 0.1875,0.125 0.4375,0.125 0.35937,0 1.03125,-0.25 l 0.21875,2 q -0.89063,0.375 -2.01563,0.375 -0.70312,0 -1.26562,-0.23438 -0.54688,-0.23437 -0.8125,-0.59375 -0.25,-0.375 -0.34375,-1 -0.0937,-0.45312 -0.0937,-1.8125 v -4.21875 h -1.17188 v -2.03125 h 1.17188 v -1.92187 l 2.57812,-1.5 v 3.42187 z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-rule:nonzero" />
+ <path
+ id="path4442"
+ d="m 220.8043,332.85645 q 0,-2.04688 0.60937,-3.42188 0.45313,-1.03125 1.23438,-1.82812 0.79687,-0.8125 1.73437,-1.20313 1.25,-0.53125 2.875,-0.53125 2.95313,0 4.71875,1.82813 1.78125,1.82812 1.78125,5.09375 0,3.23437 -1.76562,5.0625 -1.75,1.82812 -4.6875,1.82812 -2.98438,0 -4.75,-1.8125 -1.75,-1.82812 -1.75,-5.01562 z m 2.78125,-0.0937 q 0,2.26562 1.04687,3.4375 1.04688,1.17187 2.65625,1.17187 1.60938,0 2.64063,-1.15625 1.04687,-1.17187 1.04687,-3.48437 0,-2.29688 -1.01562,-3.42188 -1,-1.14062 -2.67188,-1.14062 -1.67187,0 -2.6875,1.15625 -1.01562,1.14062 -1.01562,3.4375 z m 11.58955,2.34375 2.625,-0.25 q 0.23438,1.3125 0.95313,1.9375 0.73437,0.60937 1.96875,0.60937 1.29687,0 1.95312,-0.54687 0.67188,-0.54688 0.67188,-1.28125 0,-0.48438 -0.28125,-0.8125 -0.28125,-0.32813 -0.96875,-0.57813 -0.48438,-0.15625 -2.17188,-0.57812 -2.15625,-0.54688 -3.03125,-1.32813 -1.23437,-1.09375 -1.23437,-2.6875 0,-1.01562 0.57812,-1.90625 0.57813,-0.89062 1.65625,-1.34375 1.09375,-0.46875 2.64063,-0.46875 2.51562,0 3.78125,1.10938 1.28125,1.09375 1.34375,2.9375 l -2.70313,0.10937 q -0.17187,-1.03125 -0.75,-1.46875 -0.5625,-0.45312 -1.70312,-0.45312 -1.17188,0 -1.84375,0.46875 -0.42188,0.3125 -0.42188,0.84375 0,0.46875 0.40625,0.79687 0.5,0.4375 2.46875,0.90625 1.96875,0.45313 2.90625,0.95313 0.95313,0.5 1.48438,1.35937 0.53125,0.85938 0.53125,2.125 0,1.15625 -0.64063,2.15625 -0.64062,1 -1.8125,1.48438 -1.15625,0.48437 -2.89062,0.48437 -2.53125,0 -3.89063,-1.17187 -1.35937,-1.17188 -1.625,-3.40625 z m 18.23626,4.34375 v -13.35938 h 2.68751 v 13.35938 z m 3.8708,0.23437 3.3125,-13.8125 h 1.92187 l -3.34375,13.8125 z m 6.58957,-0.23437 v -13.35938 h 9.15625 v 2.26563 h -6.45313 v 3.15625 h 5.5625 v 2.26562 h -5.5625 v 5.67188 z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-rule:nonzero" />
+ <path
+ id="path4444"
+ d="m 389.88715,290.53806 h 104.85037 v 68.56692 H 389.88715 Z"
+ inkscape:connector-curvature="0"
+ style="fill:#1c4587;fill-rule:evenodd" />
+ <path
+ id="path4446"
+ d="M 424.5789,317.45807 V 304.0987 h 4.60937 q 1.54688,0 2.375,0.20312 1.14063,0.25 1.95313,0.95313 1.0625,0.89062 1.57812,2.28125 0.53125,1.39062 0.53125,3.17187 0,1.51563 -0.35937,2.70313 -0.35938,1.17187 -0.92188,1.9375 -0.54687,0.76562 -1.20312,1.21875 -0.65625,0.4375 -1.59375,0.67187 -0.9375,0.21875 -2.14063,0.21875 z m 1.76562,-1.57812 h 2.85938 q 1.3125,0 2.0625,-0.23438 0.75,-0.25 1.20312,-0.70312 0.625,-0.625 0.96875,-1.6875 0.35938,-1.0625 0.35938,-2.57813 0,-2.09375 -0.6875,-3.21875 -0.6875,-1.125 -1.67188,-1.5 -0.70312,-0.28125 -2.28125,-0.28125 h -2.8125 z m 21.23859,-3.10938 1.76563,0.45313 q -0.5625,2.17187 -2,3.32812 -1.4375,1.14063 -3.53125,1.14063 -2.15625,0 -3.51563,-0.875 -1.34375,-0.89063 -2.0625,-2.54688 -0.70312,-1.67187 -0.70312,-3.59375 0,-2.07812 0.79687,-3.625 0.79688,-1.5625 2.26563,-2.35937 1.48437,-0.8125 3.25,-0.8125 2,0 3.35937,1.01562 1.375,1.01563 1.90625,2.875 l -1.73437,0.40625 q -0.46875,-1.45312 -1.35938,-2.10937 -0.875,-0.67188 -2.20312,-0.67188 -1.54688,0 -2.57813,0.73438 -1.03125,0.73437 -1.45312,1.98437 -0.42188,1.23438 -0.42188,2.5625 0,1.70313 0.5,2.96875 0.5,1.26563 1.54688,1.90625 1.04687,0.625 2.26562,0.625 1.48438,0 2.51563,-0.85937 1.03125,-0.85938 1.39062,-2.54688 z m 4.03543,4.6875 V 304.0987 h 9.01563 v 1.57812 h -7.25 v 4.14063 h 6.26562 v 1.57812 h -6.26562 v 6.0625 z"
+ inkscape:connector-curvature="0"
+ style="fill:#ffffff;fill-rule:nonzero" />
+ <path
+ id="path4448"
+ d="m 434.4172,324.79117 h 1.26562 v 5.51563 q 0,1.4375 -0.32812,2.29687 -0.3125,0.84375 -1.17188,1.375 -0.84375,0.51563 -2.21875,0.51563 -1.34375,0 -2.20312,-0.45313 -0.84375,-0.46875 -1.21875,-1.34375 -0.35938,-0.875 -0.35938,-2.39062 v -5.51563 h 1.26563 v 5.51563 q 0,1.23437 0.21875,1.82812 0.23437,0.59375 0.79687,0.92188 0.5625,0.3125 1.39063,0.3125 1.39062,0 1.96875,-0.625 0.59375,-0.64063 0.59375,-2.4375 z m 3.32828,9.54688 v -9.54688 h 1.90625 l 2.25,6.76563 q 0.3125,0.9375 0.46875,1.40625 0.15625,-0.51563 0.5,-1.53125 l 2.28125,-6.64063 h 1.70312 v 9.54688 h -1.21875 v -7.98438 l -2.76562,7.98438 h -1.14063 l -2.76562,-8.125 v 8.125 z m 11.14932,0 v -9.54688 h 3.28125 q 1.10938,0 1.70313,0.14063 0.8125,0.1875 1.39062,0.6875 0.76563,0.64062 1.14063,1.64062 0.375,0.98438 0.375,2.25 0,1.09375 -0.26563,1.9375 -0.25,0.82813 -0.65625,1.39063 -0.39062,0.54687 -0.85937,0.85937 -0.46875,0.3125 -1.14063,0.48438 -0.65625,0.15625 -1.53125,0.15625 z m 1.26563,-1.125 h 2.03125 q 0.9375,0 1.46875,-0.17188 0.54687,-0.1875 0.875,-0.5 0.4375,-0.45312 0.6875,-1.20312 0.25,-0.76563 0.25,-1.84375 0,-1.5 -0.5,-2.29688 -0.48438,-0.8125 -1.1875,-1.07812 -0.5,-0.20313 -1.625,-0.20313 h -2 z"
+ inkscape:connector-curvature="0"
+ style="fill:#ffffff;fill-rule:nonzero" />
+ <path
+ id="path4450"
+ d="m 840.41205,357.18372 h 36.62994 v 37.88977 h -36.62994 z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
+ <path
+ id="path4452"
+ d="m 850.7558,370.74435 h 2.6875 v 7.23437 q 0,1.71875 0.10938,2.23438 0.17187,0.82812 0.8125,1.32812 0.65625,0.48438 1.78125,0.48438 1.15625,0 1.73437,-0.46875 0.59375,-0.46875 0.70313,-1.14063 0.125,-0.6875 0.125,-2.28125 v -7.39062 h 2.6875 v 7.01562 q 0,2.40625 -0.21875,3.40625 -0.21875,0.98438 -0.8125,1.67188 -0.57813,0.6875 -1.5625,1.09375 -0.98438,0.40625 -2.5625,0.40625 -1.92188,0 -2.90625,-0.4375 -0.98438,-0.45313 -1.5625,-1.15625 -0.57813,-0.70313 -0.75,-1.48438 -0.26563,-1.14062 -0.26563,-3.39062 z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-rule:nonzero" />
+ <path
+ id="path4454"
+ d="m 612.63257,1.816273 h 264 V 222.4462 h -264 z"
+ inkscape:connector-curvature="0"
+ style="fill:#ffffff;fill-rule:evenodd" />
+ <path
+ id="path4456"
+ d="m 612.63257,1.816273 h 264 V 222.4462 h -264 z"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
+ <path
+ id="path4458"
+ d="m 622.73755,11.92126 h 240 v 90.96063 h -240 z"
+ inkscape:connector-curvature="0"
+ style="fill:#c9daf8;fill-rule:evenodd" />
+ <path
+ id="path4460"
+ d="m 622.73755,11.92126 h 240 v 90.96063 h -240 z"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#d0e0e3;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
+ <path
+ id="path4462"
+ d="m 627.79004,119.28871 h 234.96063 v 90.96063 H 627.79004 Z"
+ inkscape:connector-curvature="0"
+ style="fill:#cfe2f3;fill-rule:evenodd" />
+ <path
+ id="path4464"
+ d="m 627.79004,119.28871 h 234.96063 v 90.96063 H 627.79004 Z"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#c9daf8;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
+ <path
+ id="path4466"
+ d="m 636.63257,27.078741 h 84.62988 v 60.44094 h -84.62988 z"
+ inkscape:connector-curvature="0"
+ style="fill:#1c4587;fill-rule:evenodd" />
+ <path
+ id="path4468"
+ d="m 660.66064,42.529987 v -1.890625 h 1.64062 v 1.890625 z m 0,11.46875 v -9.671875 h 1.64062 v 9.671875 z m 2.87915,0 5.125,-13.359375 h 1.90625 l 5.46875,13.359375 h -2.01562 l -1.54688,-4.046875 h -5.59375 l -1.46875,4.046875 z m 3.85938,-5.484375 h 4.53125 l -1.40625,-3.703125 q -0.625,-1.6875 -0.9375,-2.765625 -0.26563,1.28125 -0.71875,2.546875 z m 12.48004,5.484375 -5.17188,-13.359375 h 1.92188 l 3.46875,9.703125 q 0.42187,1.171875 0.70312,2.1875 0.3125,-1.09375 0.71875,-2.1875 l 3.60938,-9.703125 h 1.79687 l -5.23437,13.359375 z m 8.72485,0 V 40.639362 h 9.01563 v 1.578125 h -7.25 v 4.140625 h 6.26562 v 1.578125 h -6.26562 v 6.0625 z"
+ inkscape:connector-curvature="0"
+ style="fill:#ffffff;fill-rule:nonzero" />
+ <path
+ id="path4470"
+ d="m 671.05237,61.331863 h 1.26562 v 5.515625 q 0,1.4375 -0.32812,2.296875 -0.3125,0.84375 -1.17188,1.375 -0.84375,0.515625 -2.21875,0.515625 -1.34375,0 -2.20312,-0.453125 -0.84375,-0.46875 -1.21875,-1.34375 -0.35938,-0.875 -0.35938,-2.390625 v -5.515625 h 1.26563 v 5.515625 q 0,1.234375 0.21875,1.828125 0.23437,0.59375 0.79687,0.921875 0.5625,0.3125 1.39063,0.3125 1.39062,0 1.96875,-0.625 0.59375,-0.640625 0.59375,-2.4375 z m 3.32831,9.546875 v -9.546875 h 1.90625 l 2.25,6.765625 q 0.3125,0.9375 0.46875,1.40625 0.15625,-0.515625 0.5,-1.53125 l 2.28125,-6.640625 h 1.70312 v 9.546875 h -1.21875 v -7.984375 l -2.76562,7.984375 h -1.14063 l -2.76562,-8.125 v 8.125 z m 11.14929,0 v -9.546875 h 3.28125 q 1.10937,0 1.70312,0.140625 0.8125,0.1875 1.39063,0.6875 0.76562,0.640625 1.14062,1.640625 0.375,0.984375 0.375,2.25 0,1.09375 -0.26562,1.9375 -0.25,0.828125 -0.65625,1.390625 -0.39063,0.546875 -0.85938,0.859375 -0.46875,0.3125 -1.14062,0.484375 -0.65625,0.15625 -1.53125,0.15625 z m 1.26562,-1.125 h 2.03125 q 0.9375,0 1.46875,-0.171875 0.54688,-0.1875 0.875,-0.5 0.4375,-0.453125 0.6875,-1.203125 0.25,-0.765625 0.25,-1.84375 0,-1.5 -0.5,-2.296875 -0.48437,-0.8125 -1.1875,-1.078125 -0.5,-0.203125 -1.625,-0.203125 h -2 z"
+ inkscape:connector-curvature="0"
+ style="fill:#ffffff;fill-rule:nonzero" />
+ <path
+ id="path4472"
+ d="M 736.853,3.0787401 H 882.11283 V 27.07874 H 736.853 Z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
+ <path
+ id="path4474"
+ d="m 749.603,26.158741 -3.75,-10.484376 h 2.29687 l 2.64063,7.750001 2.57812,-7.750001 h 2.25 L 751.853,26.158741 Z m 7.05682,0 V 15.674365 h 3.17188 l 1.90625,7.156251 1.89062,-7.156251 h 3.17188 v 10.484376 h -1.96875 v -8.265625 l -2.07813,8.265625 h -2.04687 l -2.07813,-8.265625 v 8.265625 z m 15.84327,-3.90625 q 0,-1 0.48437,-1.9375 0.5,-0.9375 1.40625,-1.421875 0.90625,-0.5 2.01563,-0.5 1.71875,0 2.82812,1.125 1.10938,1.109375 1.10938,2.8125 0,1.734375 -1.125,2.875 -1.10938,1.125 -2.79688,1.125 -1.04687,0 -2,-0.46875 -0.9375,-0.46875 -1.4375,-1.375 -0.48437,-0.921875 -0.48437,-2.234375 z m 2.04687,0.109375 q 0,1.125 0.53125,1.734375 0.54688,0.59375 1.34375,0.59375 0.78125,0 1.3125,-0.59375 0.53125,-0.609375 0.53125,-1.75 0,-1.125 -0.53125,-1.71875 -0.53125,-0.609375 -1.3125,-0.609375 -0.79687,0 -1.34375,0.609375 -0.53125,0.59375 -0.53125,1.734375 z m 9.29639,3.796875 h -2.01563 v -7.59375 h 1.85938 v 1.078125 q 0.48437,-0.765625 0.85937,-1 0.39063,-0.25 0.875,-0.25 0.6875,0 1.32813,0.375 l -0.625,1.75 q -0.5,-0.328125 -0.9375,-0.328125 -0.42188,0 -0.71875,0.234375 -0.29688,0.234375 -0.46875,0.84375 -0.15625,0.609375 -0.15625,2.546875 z m 14.57245,-3.859375 2.04687,0.65625 q -0.46875,1.71875 -1.57812,2.546875 -1.09375,0.828125 -2.78125,0.828125 -2.07813,0 -3.4375,-1.421875 -1.34375,-1.421875 -1.34375,-3.90625 0,-2.609375 1.35937,-4.0625 1.35938,-1.453126 3.5625,-1.453126 1.92188,0 3.125,1.140626 0.71875,0.671875 1.0625,1.9375 l -2.09375,0.5 q -0.1875,-0.828125 -0.78125,-1.296875 -0.57812,-0.46875 -1.42187,-0.46875 -1.17188,0 -1.90625,0.84375 -0.71875,0.828125 -0.71875,2.703125 0,1.984375 0.71875,2.84375 0.71875,0.84375 1.85937,0.84375 0.84375,0 1.45313,-0.53125 0.60937,-0.546875 0.875,-1.703125 z m 3.39679,-0.04687 q 0,-1 0.48437,-1.9375 0.5,-0.9375 1.40625,-1.421875 0.90625,-0.5 2.01563,-0.5 1.71875,0 2.82812,1.125 1.10938,1.109375 1.10938,2.8125 0,1.734375 -1.125,2.875 -1.10938,1.125 -2.79688,1.125 -1.04687,0 -2,-0.46875 -0.9375,-0.46875 -1.4375,-1.375 -0.48437,-0.921875 -0.48437,-2.234375 z m 2.04687,0.109375 q 0,1.125 0.53125,1.734375 0.54688,0.59375 1.34375,0.59375 0.78125,0 1.3125,-0.59375 0.53125,-0.609375 0.53125,-1.75 0,-1.125 -0.53125,-1.71875 -0.53125,-0.609375 -1.3125,-0.609375 -0.79687,0 -1.34375,0.609375 -0.53125,0.59375 -0.53125,1.734375 z m 14.28076,3.796875 h -2.01562 v -3.875 q 0,-1.234375 -0.125,-1.59375 -0.125,-0.359375 -0.42188,-0.5625 -0.29687,-0.203125 -0.70312,-0.203125 -0.51563,0 -0.9375,0.296875 -0.40625,0.28125 -0.5625,0.75 -0.15625,0.46875 -0.15625,1.75 v 3.4375 h -2.01563 v -7.59375 h 1.875 v 1.109375 q 1,-1.28125 2.5,-1.28125 0.67188,0 1.21875,0.234375 0.54688,0.234375 0.82813,0.609375 0.29687,0.375 0.40625,0.84375 0.10937,0.46875 0.10937,1.359375 z m 5.51514,-7.59375 v 1.59375 h -1.375 v 3.0625 q 0,0.9375 0.0312,1.09375 0.0469,0.140625 0.1875,0.25 0.14062,0.09375 0.34375,0.09375 0.28125,0 0.8125,-0.1875 l 0.17187,1.5625 q -0.70312,0.296875 -1.59375,0.296875 -0.54687,0 -0.98437,-0.171875 -0.42188,-0.1875 -0.64063,-0.46875 -0.20312,-0.296875 -0.28125,-0.796875 -0.0625,-0.359375 -0.0625,-1.421875 v -3.3125 h -0.92187 v -1.59375 h 0.92187 v -1.515625 l 2.01563,-1.171876 v 2.687501 z m 2.91187,2.3125 -1.82813,-0.328125 q 0.29688,-1.109375 1.04688,-1.625 0.76562,-0.53125 2.23437,-0.53125 1.35938,0 2.01563,0.3125 0.65625,0.3125 0.92187,0.8125 0.26563,0.484375 0.26563,1.796875 l -0.0156,2.34375 q 0,1 0.0937,1.484375 0.0937,0.46875 0.35938,1.015625 h -1.98438 q -0.0781,-0.203125 -0.20312,-0.59375 -0.0469,-0.171875 -0.0625,-0.234375 -0.51563,0.5 -1.10938,0.75 -0.57812,0.25 -1.25,0.25 -1.17187,0 -1.85937,-0.640625 -0.67188,-0.640625 -0.67188,-1.609375 0,-0.640625 0.3125,-1.140625 0.3125,-0.515625 0.85938,-0.78125 0.5625,-0.265625 1.60937,-0.46875 1.40625,-0.265625 1.95313,-0.484375 v -0.203125 q 0,-0.578125 -0.29688,-0.828125 -0.28125,-0.25 -1.07812,-0.25 -0.53125,0 -0.84375,0.21875 -0.29688,0.203125 -0.46875,0.734375 z m 2.6875,1.625 q -0.39063,0.140625 -1.23438,0.328125 -0.82812,0.171875 -1.09375,0.34375 -0.39062,0.265625 -0.39062,0.703125 0,0.421875 0.3125,0.734375 0.3125,0.296875 0.8125,0.296875 0.53125,0 1.03125,-0.359375 0.35937,-0.265625 0.48437,-0.65625 0.0781,-0.265625 0.0781,-0.984375 z m 3.94799,-4.96875 V 15.67437 h 2.01563 v 1.859376 z m 0,8.625 v -7.59375 h 2.01563 v 7.59375 z m 10.99384,0 h -2.01563 v -3.875 q 0,-1.234375 -0.125,-1.59375 -0.125,-0.359375 -0.42187,-0.5625 -0.29688,-0.203125 -0.70313,-0.203125 -0.51562,0 -0.9375,0.296875 -0.40625,0.28125 -0.5625,0.75 -0.15625,0.46875 -0.15625,1.75 v 3.4375 h -2.01562 v -7.59375 h 1.875 v 1.109375 q 1,-1.28125 2.5,-1.28125 0.67187,0 1.21875,0.234375 0.54687,0.234375 0.82812,0.609375 0.29688,0.375 0.40625,0.84375 0.10938,0.46875 0.10938,1.359375 z m 6.43701,-2.421875 2,0.34375 q -0.375,1.09375 -1.21875,1.671875 -0.82813,0.578125 -2.07813,0.578125 -1.98437,0 -2.9375,-1.296875 -0.75,-1.03125 -0.75,-2.625 0,-1.875 0.98438,-2.9375 0.98437,-1.078125 2.5,-1.078125 1.6875,0 2.67187,1.125 0.98438,1.109375 0.9375,3.421875 h -5.03125 q 0.0156,0.890625 0.48438,1.390625 0.46875,0.5 1.15625,0.5 0.46875,0 0.78125,-0.25 0.32812,-0.265625 0.5,-0.84375 z m 0.10937,-2.03125 q -0.0156,-0.875 -0.45312,-1.328125 -0.42188,-0.453125 -1.03125,-0.453125 -0.67188,0 -1.09375,0.484375 -0.42188,0.46875 -0.42188,1.296875 z m 5.573,4.453125 h -2.01562 v -7.59375 h 1.85937 v 1.078125 q 0.48438,-0.765625 0.85938,-1 0.39062,-0.25 0.875,-0.25 0.6875,0 1.32812,0.375 l -0.625,1.75 q -0.5,-0.328125 -0.9375,-0.328125 -0.42187,0 -0.71875,0.234375 -0.29687,0.234375 -0.46875,0.84375 -0.15625,0.609375 -0.15625,2.546875 z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-rule:nonzero" />
+ <path
+ id="path4476"
+ d="m 826.52496,59.92651 h 36.62988 v 37.889767 h -36.62988 z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
+ <path
+ id="path4478"
+ d="m 836.8687,73.48713 h 2.6875 v 7.234375 q 0,1.71875 0.10937,2.234375 0.17188,0.828125 0.8125,1.328125 0.65625,0.484375 1.78125,0.484375 1.15625,0 1.73438,-0.46875 0.59375,-0.46875 0.70312,-1.140625 0.125,-0.6875 0.125,-2.28125 V 73.48713 h 2.6875 v 7.015625 q 0,2.40625 -0.21875,3.40625 -0.21875,0.984375 -0.8125,1.671875 -0.57812,0.6875 -1.5625,1.09375 -0.98437,0.40625 -2.5625,0.40625 -1.92187,0 -2.90625,-0.4375 -0.98437,-0.453125 -1.5625,-1.15625 -0.57812,-0.703125 -0.75,-1.484375 -0.26562,-1.140625 -0.26562,-3.390625 z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-rule:nonzero" />
+ <path
+ id="path4480"
+ d="m 826.5276,169.82152 h 36.62988 v 37.88977 H 826.5276 Z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
+ <path
+ id="path4482"
+ d="m 836.9182,196.74152 v -13.35938 h 2.70312 v 5.9375 l 5.4375,-5.9375 h 3.625 l -5.01562,5.20313 5.29687,8.15625 h -3.48437 l -3.67188,-6.26563 -2.1875,2.23438 v 4.03125 z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-rule:nonzero" />
+ <path
+ id="path4484"
+ d="m 737.26245,130.83989 h 84.62994 v 60.44095 h -84.62994 z"
+ inkscape:connector-curvature="0"
+ style="fill:#999999;fill-rule:evenodd" />
+ <path
+ id="path4486"
+ d="m 761.2905,146.29114 v -1.89062 h 1.64062 v 1.89062 z m 0,11.46875 v -9.67187 h 1.64062 v 9.67187 z m 2.87921,0 5.125,-13.35937 h 1.90625 l 5.46875,13.35937 h -2.01562 l -1.54688,-4.04687 h -5.59375 l -1.46875,4.04687 z m 3.85938,-5.48437 h 4.53125 l -1.40625,-3.70313 q -0.625,-1.6875 -0.9375,-2.76562 -0.26563,1.28125 -0.71875,2.54687 z m 12.48004,5.48437 -5.17188,-13.35937 h 1.92188 l 3.46875,9.70312 q 0.42187,1.17188 0.70312,2.1875 0.3125,-1.09375 0.71875,-2.1875 l 3.60938,-9.70312 h 1.79687 l -5.23437,13.35937 z m 8.72479,0 v -13.35937 h 9.01563 v 1.57812 h -7.25 v 4.14063 h 6.26562 v 1.57812 h -6.26562 v 6.0625 z"
+ inkscape:connector-curvature="0"
+ style="fill:#ffffff;fill-rule:nonzero" />
+ <path
+ id="path4488"
+ d="m 765.7375,174.6399 v -9.54688 h 1.26562 v 4.73438 l 4.73438,-4.73438 h 1.71875 l -4,3.875 4.17187,5.67188 h -1.65625 l -3.40625,-4.82813 -1.5625,1.51563 v 3.3125 z m 8.9054,0 v -9.54688 h 1.90625 l 2.25,6.76563 q 0.3125,0.9375 0.46875,1.40625 0.15625,-0.51563 0.5,-1.53125 l 2.28125,-6.64063 h 1.70312 v 9.54688 h -1.21875 v -7.98438 l -2.76562,7.98438 h -1.14063 l -2.76562,-8.125 v 8.125 z m 11.14929,0 v -9.54688 h 3.28125 q 1.10937,0 1.70312,0.14063 0.8125,0.1875 1.39063,0.6875 0.76562,0.64062 1.14062,1.64062 0.375,0.98438 0.375,2.25 0,1.09375 -0.26562,1.9375 -0.25,0.82813 -0.65625,1.39063 -0.39063,0.54687 -0.85938,0.85937 -0.46875,0.3125 -1.14062,0.48438 -0.65625,0.15625 -1.53125,0.15625 z m 1.26562,-1.125 h 2.03125 q 0.9375,0 1.46875,-0.17188 0.54688,-0.1875 0.875,-0.5 0.4375,-0.45312 0.6875,-1.20312 0.25,-0.76563 0.25,-1.84375 0,-1.5 -0.5,-2.29688 -0.48437,-0.8125 -1.1875,-1.07812 -0.5,-0.20313 -1.625,-0.20313 h -2 z"
+ inkscape:connector-curvature="0"
+ style="fill:#ffffff;fill-rule:nonzero" />
+ <path
+ id="path4490"
+ d="m 104.41995,29.606298 h 195.7795 v 90.960632 h -195.7795 z"
+ inkscape:connector-curvature="0"
+ style="fill:#ffd966;fill-rule:evenodd" />
+ <path
+ id="path4492"
+ d="m 104.41995,29.606298 h 195.7795 v 90.960632 h -195.7795 z"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#ffd966;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
+ <path
+ id="path4494"
+ d="M 134.73753,27.921259 H 264.84777 V 118.88189 H 134.73753 Z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
+ <path
+ id="path4496"
+ d="M 166.94131,54.84126 V 41.481885 h 5.6875 q 2.14062,0 3.10937,0.359375 0.96875,0.359375 1.54688,1.28125 0.57812,0.921875 0.57812,2.109375 0,1.5 -0.89062,2.484375 -0.875,0.96875 -2.625,1.234375 0.875,0.5 1.4375,1.125 0.57812,0.609375 1.53125,2.15625 l 1.64062,2.609375 h -3.23437 l -1.9375,-2.90625 q -1.04688,-1.5625 -1.4375,-1.96875 -0.375,-0.40625 -0.8125,-0.546875 -0.42188,-0.15625 -1.34375,-0.15625 h -0.54688 v 5.578125 z m 2.70312,-7.703125 h 2 q 1.9375,0 2.42188,-0.15625 0.48437,-0.171875 0.75,-0.578125 0.28125,-0.40625 0.28125,-1 0,-0.671875 -0.35938,-1.078125 -0.35937,-0.421875 -1.01562,-0.53125 -0.32813,-0.04687 -1.96875,-0.04687 h -2.10938 z m 16.34795,4.625 2.54688,0.421875 q -0.48438,1.40625 -1.54688,2.140625 -1.0625,0.734375 -2.65625,0.734375 -2.51562,0 -3.73437,-1.65625 -0.95313,-1.3125 -0.95313,-3.328125 0,-2.40625 1.25,-3.765625 1.26563,-1.359375 3.1875,-1.359375 2.15625,0 3.40625,1.421875 1.25,1.421875 1.1875,4.375 h -6.40625 q 0.0312,1.140625 0.60938,1.78125 0.59375,0.625 1.48437,0.625 0.59375,0 1,-0.328125 0.42188,-0.328125 0.625,-1.0625 z m 0.15625,-2.59375 q -0.0312,-1.109375 -0.57812,-1.6875 -0.54688,-0.578125 -1.32813,-0.578125 -0.84375,0 -1.39062,0.609375 -0.54688,0.609375 -0.53125,1.65625 z m 4.42259,-4 h 2.35938 v 1.3125 q 1.26562,-1.53125 3.01562,-1.53125 0.9375,0 1.60938,0.390625 0.6875,0.375 1.125,1.140625 0.64062,-0.765625 1.375,-1.140625 0.75,-0.390625 1.57812,-0.390625 1.0625,0 1.79688,0.4375 0.75,0.421875 1.10937,1.265625 0.26563,0.625 0.26563,2 v 6.1875 h -2.5625 v -5.53125 q 0,-1.4375 -0.26563,-1.859375 -0.34375,-0.546875 -1.09375,-0.546875 -0.53125,0 -1.01562,0.328125 -0.46875,0.328125 -0.67188,0.96875 -0.20312,0.625 -0.20312,2 v 4.640625 h -2.5625 v -5.296875 q 0,-1.421875 -0.14063,-1.828125 -0.14062,-0.40625 -0.42187,-0.609375 -0.28125,-0.203125 -0.78125,-0.203125 -0.59375,0 -1.0625,0.328125 -0.46875,0.3125 -0.6875,0.921875 -0.20313,0.59375 -0.20313,1.984375 v 4.703125 h -2.5625 z m 16.19777,4.703125 q 0,-1.28125 0.625,-2.46875 0.625,-1.203125 1.78125,-1.828125 1.15625,-0.625 2.57813,-0.625 2.1875,0 3.59375,1.421875 1.40625,1.421875 1.40625,3.609375 0,2.1875 -1.42188,3.640625 -1.42187,1.4375 -3.5625,1.4375 -1.32812,0 -2.54687,-0.59375 -1.20313,-0.609375 -1.82813,-1.765625 -0.625,-1.171875 -0.625,-2.828125 z m 2.625,0.125 q 0,1.453125 0.67188,2.21875 0.6875,0.75 1.6875,0.75 1,0 1.67187,-0.75 0.6875,-0.765625 0.6875,-2.234375 0,-1.421875 -0.6875,-2.1875 -0.67187,-0.765625 -1.67187,-0.765625 -1,0 -1.6875,0.765625 -0.67188,0.765625 -0.67188,2.203125 z m 13.80223,-4.828125 v 2.03125 h -1.75 v 3.90625 q 0,1.1875 0.0469,1.390625 0.0469,0.1875 0.21875,0.3125 0.1875,0.125 0.4375,0.125 0.35937,0 1.03125,-0.25 l 0.21875,2 q -0.89063,0.375 -2.01563,0.375 -0.70312,0 -1.26562,-0.234375 -0.54688,-0.234375 -0.8125,-0.59375 -0.25,-0.375 -0.34375,-1 -0.0937,-0.453125 -0.0937,-1.8125 v -4.21875 h -1.17188 v -2.03125 h 1.17188 V 43.24751 l 2.57812,-1.5 v 3.421875 z m 7.36893,6.59375 2.54688,0.421875 q -0.48438,1.40625 -1.54688,2.140625 -1.0625,0.734375 -2.65625,0.734375 -2.51562,0 -3.73437,-1.65625 -0.95313,-1.3125 -0.95313,-3.328125 0,-2.40625 1.25,-3.765625 1.26563,-1.359375 3.1875,-1.359375 2.15625,0 3.40625,1.421875 1.25,1.421875 1.1875,4.375 h -6.40625 q 0.0312,1.140625 0.60938,1.78125 0.59375,0.625 1.48437,0.625 0.59375,0 1,-0.328125 0.42188,-0.328125 0.625,-1.0625 z m 0.15625,-2.59375 q -0.0312,-1.109375 -0.57812,-1.6875 -0.54688,-0.578125 -1.32813,-0.578125 -0.84375,0 -1.39062,0.609375 -0.54688,0.609375 -0.53125,1.65625 z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-rule:nonzero" />
+ <path
+ id="path4498"
+ d="m 187.97444,76.841255 -4.78125,-13.359371 h 2.9375 l 3.375,9.890621 3.26562,-9.890621 h 2.85938 l -4.78125,13.359371 z m 8.9592,0 V 63.481884 h 2.6875 v 13.359371 z m 5.23018,0 V 63.481884 h 4.03125 l 2.42187,9.109371 2.39063,-9.109371 h 4.04687 v 13.359371 h -2.5 V 66.32563 l -2.65625,10.515625 h -2.59375 L 204.66382,66.32563 v 10.515625 z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-rule:nonzero" />
+ <path
+ id="path4500"
+ d="m 165.13069,93.935005 2.60937,0.828125 q -0.59375,2.1875 -2,3.25 -1.39062,1.0625 -3.54687,1.0625 -2.65625,0 -4.375,-1.8125 -1.70313,-1.828125 -1.70313,-4.984375 0,-3.328125 1.71875,-5.171875 1.71875,-1.84375 4.51563,-1.84375 2.45312,0 3.98437,1.4375 0.92188,0.859375 1.375,2.46875 l -2.67187,0.640625 q -0.23438,-1.046875 -0.98438,-1.640625 -0.75,-0.609375 -1.82812,-0.609375 -1.48438,0 -2.42188,1.078125 -0.92187,1.0625 -0.92187,3.4375 0,2.53125 0.90625,3.609375 0.92187,1.078125 2.375,1.078125 1.07812,0 1.84375,-0.671875 0.78125,-0.6875 1.125,-2.15625 z m 4.3167,-0.0625 q 0,-1.28125 0.625,-2.46875 0.625,-1.203125 1.78125,-1.828125 1.15625,-0.625 2.57812,-0.625 2.1875,0 3.59375,1.421875 1.40625,1.421875 1.40625,3.609375 0,2.1875 -1.42187,3.640625 -1.42188,1.4375 -3.5625,1.4375 -1.32813,0 -2.54688,-0.59375 -1.20312,-0.609375 -1.82812,-1.765625 -0.625,-1.171875 -0.625,-2.828125 z m 2.625,0.125 q 0,1.453125 0.67187,2.21875 0.6875,0.75 1.6875,0.75 1,0 1.67188,-0.75 0.6875,-0.765625 0.6875,-2.234375 0,-1.421875 -0.6875,-2.1875 -0.67188,-0.765625 -1.67188,-0.765625 -1,0 -1.6875,0.765625 -0.67187,0.765625 -0.67187,2.203125 z m 18.1616,4.84375 h -2.5625 v -4.9375 q 0,-1.5625 -0.17187,-2.015625 -0.15625,-0.46875 -0.53125,-0.71875 -0.35938,-0.265625 -0.875,-0.265625 -0.67188,0 -1.20313,0.375 -0.53125,0.359375 -0.73437,0.96875 -0.1875,0.59375 -0.1875,2.21875 v 4.375 h -2.54688 V 89.16938 h 2.375 v 1.421875 q 1.26563,-1.640625 3.1875,-1.640625 0.84375,0 1.54688,0.3125 0.70312,0.296875 1.0625,0.78125 0.35937,0.46875 0.5,1.078125 0.14062,0.59375 0.14062,1.703125 z m 7.03661,-9.671875 v 2.03125 h -1.75 v 3.90625 q 0,1.1875 0.0469,1.390625 0.0469,0.1875 0.21875,0.3125 0.1875,0.125 0.4375,0.125 0.35938,0 1.03125,-0.25 l 0.21875,2 q -0.89062,0.375 -2.01562,0.375 -0.70313,0 -1.26563,-0.234375 -0.54687,-0.234375 -0.8125,-0.59375 -0.25,-0.375 -0.34375,-1 -0.0937,-0.453125 -0.0937,-1.8125 v -4.21875 h -1.17187 v -2.03125 h 1.17187 v -1.921875 l 2.57813,-1.5 v 3.421875 z m 4.2283,9.671875 h -2.5625 V 89.16938 h 2.375 v 1.375 q 0.60938,-0.984375 1.09375,-1.28125 0.48438,-0.3125 1.10938,-0.3125 0.875,0 1.6875,0.484375 l -0.79688,2.234375 q -0.64062,-0.421875 -1.20312,-0.421875 -0.53125,0 -0.90625,0.296875 -0.375,0.296875 -0.59375,1.078125 -0.20313,0.765625 -0.20313,3.234375 z m 4.21339,-4.96875 q 0,-1.28125 0.625,-2.46875 0.625,-1.203125 1.78125,-1.828125 1.15625,-0.625 2.57813,-0.625 2.1875,0 3.59375,1.421875 1.40625,1.421875 1.40625,3.609375 0,2.1875 -1.42188,3.640625 -1.42187,1.4375 -3.5625,1.4375 -1.32812,0 -2.54687,-0.59375 -1.20313,-0.609375 -1.82813,-1.765625 -0.625,-1.171875 -0.625,-2.828125 z m 2.625,0.125 q 0,1.453125 0.67188,2.21875 0.6875,0.75 1.6875,0.75 1,0 1.67187,-0.75 0.6875,-0.765625 0.6875,-2.234375 0,-1.421875 -0.6875,-2.1875 -0.67187,-0.765625 -1.67187,-0.765625 -1,0 -1.6875,0.765625 -0.67188,0.765625 -0.67188,2.203125 z m 9.36474,4.84375 V 85.48188 h 2.5625 v 13.359375 z m 5.1833,0 V 85.48188 h 2.5625 v 13.359375 z m 10.77705,-3.078125 2.54688,0.421875 q -0.48438,1.40625 -1.54688,2.140625 -1.0625,0.734375 -2.65625,0.734375 -2.51562,0 -3.73437,-1.65625 -0.95313,-1.3125 -0.95313,-3.328125 0,-2.40625 1.25,-3.765625 1.26563,-1.359375 3.1875,-1.359375 2.15625,0 3.40625,1.421875 1.25,1.421875 1.1875,4.375 h -6.40625 q 0.0312,1.140625 0.60938,1.78125 0.59375,0.625 1.48437,0.625 0.59375,0 1,-0.328125 0.42188,-0.328125 0.625,-1.0625 z m 0.15625,-2.59375 q -0.0312,-1.109375 -0.57812,-1.6875 -0.54688,-0.578125 -1.32813,-0.578125 -0.84375,0 -1.39062,0.609375 -0.54688,0.609375 -0.53125,1.65625 z m 7.07885,5.671875 h -2.5625 V 89.16938 h 2.375 v 1.375 q 0.60937,-0.984375 1.09375,-1.28125 0.48437,-0.3125 1.10937,-0.3125 0.875,0 1.6875,0.484375 l -0.79687,2.234375 q -0.64063,-0.421875 -1.20313,-0.421875 -0.53125,0 -0.90625,0.296875 -0.375,0.296875 -0.59375,1.078125 -0.20312,0.765625 -0.20312,3.234375 z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-rule:nonzero" />
+ <path
+ id="path4502"
+ d="m 189.47244,133.82152 9.48032,-9.48031 9.48031,9.48031 h -4.74016 v 117.25984 h 4.74016 l -9.48031,9.48032 -9.48032,-9.48032 h 4.74016 V 133.82152 Z"
+ inkscape:connector-curvature="0"
+ style="fill:#b7b7b7;fill-rule:evenodd" />
+ <path
+ id="path4504"
+ d="m 189.47244,133.82152 9.48032,-9.48031 9.48031,9.48031 h -4.74016 v 117.25984 h 4.74016 l -9.48031,9.48032 -9.48032,-9.48032 h 4.74016 V 133.82152 Z"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#b7b7b7;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
+ <path
+ id="path4506"
+ d="m 137.68504,150.86876 h 131.37007 v 40.40945 H 137.68504 Z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
+ <path
+ id="path4508"
+ d="m 147.9663,177.78876 v -13.35938 h 2.6875 v 13.35938 z m 5.26143,0 v -13.35938 h 4.32812 q 2.45313,0 3.20313,0.20313 1.14062,0.29687 1.92187,1.3125 0.78125,1 0.78125,2.59375 0,1.23437 -0.45312,2.07812 -0.45313,0.82813 -1.14063,1.3125 -0.6875,0.46875 -1.39062,0.625 -0.96875,0.20313 -2.79688,0.20313 h -1.76562 v 5.03125 z m 2.6875,-11.09375 v 3.78125 h 1.48437 q 1.59375,0 2.125,-0.20313 0.54688,-0.20312 0.84375,-0.65625 0.3125,-0.45312 0.3125,-1.03125 0,-0.73437 -0.4375,-1.20312 -0.42187,-0.48438 -1.07812,-0.59375 -0.48438,-0.0937 -1.9375,-0.0937 z m 14.63339,11.09375 v -13.35938 h 2.625 l 5.45313,8.92188 v -8.92188 h 2.51562 v 13.35938 h -2.70312 l -5.39063,-8.70313 v 8.70313 z m 19.01982,-3.07813 2.54688,0.42188 q -0.48438,1.40625 -1.54688,2.14062 -1.0625,0.73438 -2.65625,0.73438 -2.51562,0 -3.73437,-1.65625 -0.95313,-1.3125 -0.95313,-3.32813 0,-2.40625 1.25,-3.76562 1.26563,-1.35938 3.1875,-1.35938 2.15625,0 3.40625,1.42188 1.25,1.42187 1.1875,4.375 h -6.40625 q 0.0312,1.14062 0.60938,1.78125 0.59375,0.625 1.48437,0.625 0.59375,0 1,-0.32813 0.42188,-0.32812 0.625,-1.0625 z m 0.15625,-2.59375 q -0.0312,-1.10937 -0.57812,-1.6875 -0.54688,-0.57812 -1.32813,-0.57812 -0.84375,0 -1.39062,0.60937 -0.54688,0.60938 -0.53125,1.65625 z m 9.06322,-4 v 2.03125 h -1.75 v 3.90625 q 0,1.1875 0.0469,1.39063 0.0469,0.1875 0.21875,0.3125 0.1875,0.125 0.4375,0.125 0.35938,0 1.03125,-0.25 l 0.21875,2 q -0.89062,0.375 -2.01562,0.375 -0.70313,0 -1.26563,-0.23438 -0.54687,-0.23437 -0.8125,-0.59375 -0.25,-0.375 -0.34375,-1 -0.0937,-0.45312 -0.0937,-1.8125 v -4.21875 h -1.17187 v -2.03125 h 1.17187 v -1.92187 l 2.57813,-1.5 v 3.42187 z m 3.57205,9.67188 -3.0625,-9.67188 h 2.48438 l 1.8125,6.34375 1.67187,-6.34375 h 2.46875 l 1.60938,6.34375 1.85937,-6.34375 h 2.51563 l -3.10938,9.67188 h -2.45312 l -1.67188,-6.21875 -1.64062,6.21875 z m 12.1208,-4.96875 q 0,-1.28125 0.625,-2.46875 0.625,-1.20313 1.78125,-1.82813 1.15625,-0.625 2.57813,-0.625 2.1875,0 3.59375,1.42188 1.40625,1.42187 1.40625,3.60937 0,2.1875 -1.42188,3.64063 -1.42187,1.4375 -3.5625,1.4375 -1.32812,0 -2.54687,-0.59375 -1.20313,-0.60938 -1.82813,-1.76563 -0.625,-1.17187 -0.625,-2.82812 z m 2.625,0.125 q 0,1.45312 0.67188,2.21875 0.6875,0.75 1.6875,0.75 1,0 1.67187,-0.75 0.6875,-0.76563 0.6875,-2.23438 0,-1.42187 -0.6875,-2.1875 -0.67187,-0.76562 -1.67187,-0.76562 -1,0 -1.6875,0.76562 -0.67188,0.76563 -0.67188,2.20313 z m 11.81786,4.84375 h -2.5625 v -9.67188 h 2.375 v 1.375 q 0.60938,-0.98437 1.09375,-1.28125 0.48438,-0.3125 1.10938,-0.3125 0.875,0 1.6875,0.48438 l -0.79688,2.23437 q -0.64062,-0.42187 -1.20312,-0.42187 -0.53125,0 -0.90625,0.29687 -0.375,0.29688 -0.59375,1.07813 -0.20313,0.76562 -0.20313,3.23437 z m 4.71339,0 v -13.35938 h 2.5625 v 7.09375 l 3,-3.40625 h 3.14063 l -3.29688,3.53125 3.53125,6.14063 h -2.75 l -2.4375,-4.34375 -1.1875,1.25 v 3.09375 z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-rule:nonzero" />
+ <path
+ id="path4510"
+ d="m 229.9029,361.8189 c 0,25.47244 24.25984,38.20865 48.51968,50.94488 24.25983,12.73621 48.51969,25.47242 48.51969,50.94486"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
+ <path
+ id="path4512"
+ d="m 229.9029,361.8189 c 0,25.47244 24.25984,38.20865 48.51968,50.94488 24.25983,12.73621 48.51969,25.47242 48.51969,50.94486"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#000000;stroke-width:3;stroke-linecap:butt;stroke-linejoin:round" />
+ <path
+ id="path4514"
+ d="m 313.46194,390.02625 h 42.96063 v 51.77951 h -42.96063 z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
+ <path
+ id="path4516"
+ d="m 338.64944,425.66812 v 4.07812 h -15.39062 q 0.25,-2.3125 1.5,-4.375 1.25,-2.07812 4.9375,-5.5 2.96875,-2.76562 3.64062,-3.75 0.90625,-1.35937 0.90625,-2.6875 0,-1.46875 -0.79687,-2.25 -0.78125,-0.79687 -2.17188,-0.79687 -1.375,0 -2.1875,0.82812 -0.8125,0.82813 -0.9375,2.75 l -4.375,-0.4375 q 0.39063,-3.625 2.45313,-5.20312 2.0625,-1.57813 5.15625,-1.57813 3.39062,0 5.32812,1.82813 1.9375,1.82812 1.9375,4.54687 0,1.54688 -0.5625,2.95313 -0.54687,1.39062 -1.75,2.92187 -0.79687,1.01563 -2.875,2.92188 -2.07812,1.90625 -2.64062,2.53125 -0.54688,0.625 -0.89063,1.21875 z"
+ inkscape:connector-curvature="0"
+ style="fill:#1155cc;fill-rule:nonzero" />
+ <path
+ id="path4518"
+ d="m 537.4619,422.02625 h 42.96063 v 51.77951 H 537.4619 Z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
+ <path
+ id="path4520"
+ d="m 559.05566,461.74625 h -4.39063 v -16.54688 q -2.40625,2.25 -5.67187,3.32813 v -3.98438 q 1.71875,-0.5625 3.73437,-2.125 2.01563,-1.57812 2.76563,-3.67187 h 3.5625 z"
+ inkscape:connector-curvature="0"
+ style="fill:#1155cc;fill-rule:nonzero" />
+ <path
+ id="path4522"
+ d="m 442.3281,359.09448 c 0,72.83466 -15.48032,145.66931 -30.96063,145.66931"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
+ <path
+ id="path4524"
+ d="m 442.3281,359.09448 c 0,72.83466 -15.48032,145.66931 -30.96063,145.66931"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#4a86e8;stroke-width:2;stroke-linecap:butt;stroke-linejoin:round;stroke-dasharray:8, 6" />
+ <path
+ id="path4526"
+ d="m 678.9475,87.519684 c 0,208.629906 -133.79529,417.259856 -267.59055,417.259856"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
+ <path
+ id="path4528"
+ d="m 678.9475,87.519684 c 0,208.629906 -133.79529,417.259856 -267.59055,417.259856"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#4a86e8;stroke-width:2;stroke-linecap:butt;stroke-linejoin:round;stroke-dasharray:8, 6" />
+ <path
+ id="path4530"
+ d="m 779.57745,191.28084 c 0,156.7559 -184.09454,313.51184 -368.189,313.51184"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
+ <path
+ id="path4532"
+ d="m 779.57745,191.28084 c 0,156.7559 -184.09454,313.51184 -368.189,313.51184"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#4a86e8;stroke-width:2;stroke-linecap:butt;stroke-linejoin:round;stroke-dasharray:8, 6" />
+ <path
+ id="path4534"
+ d="m 221.88715,372.1496 9.4803,-16.85037 9.48032,16.85037 z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-rule:evenodd" />
+ <path
+ id="path4536"
+ d="m 221.88715,372.1496 9.4803,-16.85037 9.48032,16.85037 z"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
+ <path
+ id="path4538"
+ d="m 335.87573,450.3371 -7.59055,17.74802 -11.24411,-15.70078 z"
+ inkscape:connector-curvature="0"
+ style="fill:#000000;fill-rule:evenodd" />
+ <path
+ id="path4540"
+ d="m 335.87573,450.3371 -7.59055,17.74802 -11.24411,-15.70078 z"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
+ <path
+ id="path4542"
+ d="m 432.83203,372.1496 9.48032,-16.85037 9.48031,16.85037 z"
+ inkscape:connector-curvature="0"
+ style="fill:#4a86e8;fill-rule:evenodd" />
+ <path
+ id="path4544"
+ d="m 432.83203,372.1496 9.48032,-16.85037 9.48031,16.85037 z"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#4a86e8;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
+ <path
+ id="path4546"
+ d="m 669.46716,97.81628 9.48035,-16.850395 9.48028,16.850395 z"
+ inkscape:connector-curvature="0"
+ style="fill:#4a86e8;fill-rule:evenodd" />
+ <path
+ id="path4548"
+ d="m 669.46716,97.81628 9.48035,-16.850395 9.48028,16.850395 z"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#4a86e8;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
+ <path
+ id="path4550"
+ d="m 768.83203,204.14961 9.48029,-16.8504 9.48034,16.8504 z"
+ inkscape:connector-curvature="0"
+ style="fill:#4a86e8;fill-rule:evenodd" />
+ <path
+ id="path4552"
+ d="m 768.83203,204.14961 9.48029,-16.8504 9.48034,16.8504 z"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#4a86e8;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
+ <path
+ id="path4554"
+ d="m 421.14697,514.3213 -18.88187,-4.11023 13.30707,-14 z"
+ inkscape:connector-curvature="0"
+ style="fill:#4a86e8;fill-rule:evenodd" />
+ <path
+ id="path4556"
+ d="m 421.14697,514.3213 -18.88187,-4.11023 13.30707,-14 z"
+ inkscape:connector-curvature="0"
+ style="fill-rule:evenodd;stroke:#4a86e8;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
+ </g>
+ </g>
+ </g>
+ </g>
+ </g>
+</svg>
--
2.26.0
^ permalink raw reply [relevance 1%]
* Re: [dpdk-dev] /validate-abi.sh complains [PATCH v1 3/8] ring: introduce RTS ring mode
2020-03-31 17:05 7% [dpdk-dev] /validate-abi.sh complains [PATCH v1 3/8] ring: introduce RTS ring mode Ananyev, Konstantin
2020-03-31 19:22 8% ` Thomas Monjalon
@ 2020-04-01 6:38 9% ` David Marchand
2020-04-02 12:36 4% ` Ananyev, Konstantin
1 sibling, 1 reply; 200+ results
From: David Marchand @ 2020-04-01 6:38 UTC (permalink / raw)
To: Ananyev, Konstantin
Cc: thomas, nhorman, honnappa.nagarahalli, dev, Richardson, Bruce,
Yigit, Ferruh, Dodji Seketeli, Kevin Laatz
On Tue, Mar 31, 2020 at 7:05 PM Ananyev, Konstantin
<konstantin.ananyev@intel.com> wrote:
>
> Hi everyone,
>
> Have a question regarding validate-abi.sh.
> It complains on the following changes with that patch:
>
> @@ -111,11 +129,21 @@ struct rte_ring {
> char pad0 __rte_cache_aligned; /**< empty cache line */
>
> /** Ring producer status. */
> - struct rte_ring_headtail prod __rte_cache_aligned;
> + RTE_STD_C11
> + union {
> + struct rte_ring_headtail prod;
> + struct rte_ring_rts_headtail rts_prod;
> + } __rte_cache_aligned;
> +
> char pad1 __rte_cache_aligned; /**< empty cache line */
>
> /** Ring consumer status. */
> - struct rte_ring_headtail cons __rte_cache_aligned;
> + RTE_STD_C11
> + union {
> + struct rte_ring_headtail cons;
> + struct rte_ring_rts_headtail rts_cons;
> + } __rte_cache_aligned;
> +
>
> Complaints:
> rte_ring.h
> [−] struct rte_ring 2
>
> 1
> Change: Field cons has been removed from this type.
> Effect: Applications will access incorrect memory when attempting to access this field.
> 2
> Change: Field prod has been removed from this type.
> Effect: Applications will access incorrect memory when attempting to access this field.
>
> From my perspective it looks false-positive:
> *prod* and *cons* fields are still there,
> their format, size and offset within rte_ring remain the same.
> Is that some limitation with the tool, or am I missing something here?
- Side note, we have build failures with clang and ARM jobs:
https://travis-ci.com/github/ovsrobot/dpdk/builds/157277423
- We switched to libabigail called from devtools/check-abi.sh which
you can run locally
(https://doc.dpdk.org/guides/contributing/patches.html?highlight=abi#checking-abi-compatibility).
Or you can count on Aaron's robot to do this check in Travis.
It reported a warning on those fields:
https://travis-ci.com/github/ovsrobot/dpdk/jobs/310689008#L2380
I understand this as a false positive too.
It seems similar to the bz I opened about fields moved to anonymous
constructs: https://sourceware.org/bugzilla/show_bug.cgi?id=25661
Cc: Dodji.
For the time being, you can waive this by adding a rule in
devtools/libabigail.abignore.
--
David Marchand
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH v3 00/33] DPDK Trace support
2020-03-29 14:43 1% ` [dpdk-dev] [PATCH v3 00/33] " jerinj
@ 2020-04-01 8:18 3% ` David Marchand
2020-04-01 10:04 0% ` Jerin Jacob
2020-04-03 15:36 1% ` [dpdk-dev] [PATCH v4 " jerinj
1 sibling, 1 reply; 200+ results
From: David Marchand @ 2020-04-01 8:18 UTC (permalink / raw)
To: Jerin Jacob Kollanukkaran
Cc: dev, Thomas Monjalon, Bruce Richardson, Mattias Rönnblom,
Sunil Kumar Kori, Yigit, Ferruh, Andrew Rybchenko,
Declan Doherty, Olivier Matz, Neil Horman
Hello Jerin,
On Sun, Mar 29, 2020 at 4:43 PM <jerinj@marvell.com> wrote:
> Items that needs to be sort it out
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> # Makefile and meson.build are updated to allow experimental APIs.
>
> As multiple EXPERIMENTAL symbols, exported by trace library, are
> used in various drivers, lib, app and examples. So to fix compilation
> warning/error, Makefile and meson.build are updated for all required
> components to support EXPERIMENTAL APIs.
> It results same code changes at multiple components as well as
> increases source code line changes in patchset too.
> Suggestions are welcome to resolve this issue with lesser code changes.
- Regardless of the trace framework, the ALLOW_EXPERIMENTAL_API flag
gates access to APIs so that external users are aware of their status.
I have been considering setting this flag unconditionally for internal
users in the top Makefile/meson for app/ lib/ and drivers/.
I could look at this and prepare a patch about this, but this is not
enough here.
With the trace framework, we add experimental symbols in mempool or
ethdev inlines, which then inherit the experimental check.
This would break compilation of external code. Users are then forced
to enable experimental API.
How about:
* we introduce a global config switch that enables/disables the trace
framework (off by default): the people who want to test it and help
stabilize it will have to deal with the experimental flag for now,
* in 20.11, the trace points are put into the stable ABI, and the
option is removed,
- With the patchset rebased on the current master (could be my fault,
so take it with a grain of salt), the ALLOW_EXPERIMENTAL_API flag is
not passed when compiling the l3fwd example against an installed dpdk.
- On the MAINTAINERS update:
Trace
M: Jerin Jacob <jerinj@marvell.com>
M: Sunil Kumar Kori <skori@marvell.com>
F: lib/librte_eal/include/rte_trace*.h
F: lib/librte_eal/common/eal_common_trace*.c
F: lib/librte_eal/common/eal_trace.h
F: lib/librte_ethdev/ethdev_trace_points.c
F: lib/librte_ethdev/rte_trace_ethdev*.h
F: lib/librte_eventdev/eventdev_trace_points.c
F: lib/librte_eventdev/rte_trace_eventdev*.h
F: lib/librte_cryptodev/cryptodev_trace_points.c
F: lib/librte_cryptodev/rte_trace_cryptodev*.h
F: lib/librte_mempool/mempool_trace_points.c
F: lib/librte_mempool/rte_trace_mempool*.h
Once we exclude the trace framework, the trace points themselves
should be the responsibility of the component (ethdev, eventdev...)
maintainers (copied them).
- I had something more dynamic in mind for gathering traces: like
enabling/disabling trace points in a running process and getting the
traces on the fly.
But I suppose the current framework is enough and we can work on this later.
>
> More details:
> ~~~~~~~~~~~~~
>
> # The Native DPDK CTF trace support does not have any dependency on
> third-party library.
> The generated output file is compatible with LTTng as both are using
> CTF trace format.
>
> The performance gain comes from:
> 1) exploit dpdk worker thread usage model to avoid atomics and use per
> core variables
> 2) use hugepage,
> 3) avoid a lot function pointers in fast-path etc
> 4) avoid unaligned store for arm64 etc
>
> Features:
> ~~~~~~~~~
> - APIs and Features are similar to rte_log dynamic framework
> API(expect log prints on stdout vs it dumps on trace file)
I am not sure about the global and per tracepoint levels.
We must enable each tracepoint anyway, the level is just a commodity.
I don't have strong arguments against it, but it feels odd to
copy/paste the rte_log framework.
--
David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v3 00/33] DPDK Trace support
2020-04-01 8:18 3% ` David Marchand
@ 2020-04-01 10:04 0% ` Jerin Jacob
2020-04-01 14:12 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Jerin Jacob @ 2020-04-01 10:04 UTC (permalink / raw)
To: David Marchand
Cc: Jerin Jacob Kollanukkaran, dev, Thomas Monjalon,
Bruce Richardson, Mattias Rönnblom, Sunil Kumar Kori, Yigit,
Ferruh, Andrew Rybchenko, Declan Doherty, Olivier Matz,
Neil Horman, Ray Kinsella
On Wed, Apr 1, 2020 at 1:49 PM David Marchand <david.marchand@redhat.com> wrote:
>
> Hello Jerin,
Hello David,
Thanks for the review.
>
> On Sun, Mar 29, 2020 at 4:43 PM <jerinj@marvell.com> wrote:
> > Items that needs to be sort it out
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > # Makefile and meson.build are updated to allow experimental APIs.
> >
> > As multiple EXPERIMENTAL symbols, exported by trace library, are
> > used in various drivers, lib, app and examples. So to fix compilation
> > warning/error, Makefile and meson.build are updated for all required
> > components to support EXPERIMENTAL APIs.
> > It results same code changes at multiple components as well as
> > increases source code line changes in patchset too.
> > Suggestions are welcome to resolve this issue with lesser code changes.
>
> - Regardless of the trace framework, the ALLOW_EXPERIMENTAL_API flag
> gates access to APIs so that external users are aware of their status.
> I have been considering setting this flag unconditionally for internal
> users in the top Makefile/meson for app/ lib/ and drivers/.
> I could look at this and prepare a patch about this, but this is not
> enough here.
It makes sense to me. Let me know when you are planning to send that patch,
I will rebase this series on top of that.
If you don't have time then I can send the patch too.
I assume the patch content will be:
1) Removing experimental API from app, lib, drivers, examples with
make and meson
2) Have it enabled at the global level.
> With the trace framework, we add experimental symbols in mempool or
> ethdev inlines, which then inherit the experimental check.
> This would break compilation of external code. Users are then forced
> to enable experimental API.
I agree with the analysis.
>
> How about:
> * we introduce a global config switch that enables/disables the trace
> framework (off by default): the people who want to test it and help
> stabilize it will have to deal with the experimental flag for now,
> * in 20.11, the trace points are put into the stable ABI, and the
> option is removed,
IMO, the better alternative would be:
Since the trace changes in the "inline" functions of the specific
library already
disabled under _compile time_ RTE_ENABLE_TRACE_DP flag and
it is using RTE_TRACE_POINT_DP() to define the trace unlike slow path
trace like RTE_TRACE_POINT().
So I can improve RTE_TRACE_POINT_DP() to make absolute NOP if
ALLOW_EXPERIMENTAL_API not defined.
On the upside,
The tracing code will be enabled by default(runtime it is disabled by
default anyway).
If some need to fastpath API tracing then ALLOW_EXPERIMENTAL_API need
to enable.
So this won't break applications.
>
>
> - With the patchset rebased on the current master (could be my fault,
> so take it with a grain of salt), the ALLOW_EXPERIMENTAL_API flag is
> not passed when compiling the l3fwd example against an installed dpdk.
I will check. We have added ALLOW_EXPERIMENTAL_API flag where we got
compilation issues.
>
>
> - On the MAINTAINERS update:
>
> Trace
> M: Jerin Jacob <jerinj@marvell.com>
> M: Sunil Kumar Kori <skori@marvell.com>
> F: lib/librte_eal/include/rte_trace*.h
> F: lib/librte_eal/common/eal_common_trace*.c
> F: lib/librte_eal/common/eal_trace.h
> F: lib/librte_ethdev/ethdev_trace_points.c
> F: lib/librte_ethdev/rte_trace_ethdev*.h
> F: lib/librte_eventdev/eventdev_trace_points.c
> F: lib/librte_eventdev/rte_trace_eventdev*.h
> F: lib/librte_cryptodev/cryptodev_trace_points.c
> F: lib/librte_cryptodev/rte_trace_cryptodev*.h
> F: lib/librte_mempool/mempool_trace_points.c
> F: lib/librte_mempool/rte_trace_mempool*.h
>
> Once we exclude the trace framework, the trace points themselves
> should be the responsibility of the component (ethdev, eventdev...)
> maintainers (copied them).
Yes. I will remove the following from the MAINTAINERS update in the
next version.
F: lib/librte_ethdev/ethdev_trace_points.c
F: lib/librte_ethdev/rte_trace_ethdev*.h
F: lib/librte_eventdev/eventdev_trace_points.c
F: lib/librte_eventdev/rte_trace_eventdev*.h
F: lib/librte_cryptodev/cryptodev_trace_points.c
F: lib/librte_cryptodev/rte_trace_cryptodev*.h
F: lib/librte_mempool/mempool_trace_points.c
F: lib/librte_mempool/rte_trace_mempool*.h
>
>
> - I had something more dynamic in mind for gathering traces: like
> enabling/disabling trace points in a running process and getting the
> traces on the fly.
rte_trace_enable() and rte_trace_disable() already avilable for this.
> But I suppose the current framework is enough and we can work on this later.
Yes. for all new features.
>
>
>
> >
> > More details:
> > ~~~~~~~~~~~~~
> >
> > # The Native DPDK CTF trace support does not have any dependency on
> > third-party library.
> > The generated output file is compatible with LTTng as both are using
> > CTF trace format.
> >
> > The performance gain comes from:
> > 1) exploit dpdk worker thread usage model to avoid atomics and use per
> > core variables
> > 2) use hugepage,
> > 3) avoid a lot function pointers in fast-path etc
> > 4) avoid unaligned store for arm64 etc
> >
> > Features:
> > ~~~~~~~~~
> > - APIs and Features are similar to rte_log dynamic framework
> > API(expect log prints on stdout vs it dumps on trace file)
>
> I am not sure about the global and per tracepoint levels.
> We must enable each tracepoint anyway, the level is just a commodity.
>
> I don't have strong arguments against it, but it feels odd to
> copy/paste the rte_log framework.
I have intentionally kept public API similar to rte_log wherever it is
possible. Reason being,
1) In the future, it is easy to replace rte_log with trace _if needed_.
2) Avoid the new API learning curve.
/Jerin
>
>
> --
> David Marchand
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] /validate-abi.sh complains [PATCH v1 3/8] ring: introduce RTS ring mode
2020-03-31 19:24 4% ` Thomas Monjalon
@ 2020-04-01 12:52 4% ` Neil Horman
2020-04-06 14:02 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Neil Horman @ 2020-04-01 12:52 UTC (permalink / raw)
To: Thomas Monjalon
Cc: mdr, Ananyev, Konstantin, david.marchand, honnappa.nagarahalli,
dev, Richardson, Bruce, Yigit, Ferruh, kevin.laatz
On Tue, Mar 31, 2020 at 09:24:41PM +0200, Thomas Monjalon wrote:
> 31/03/2020 21:22, Thomas Monjalon:
> > 31/03/2020 19:05, Ananyev, Konstantin:
> > > Hi everyone,
> > >
> > > Have a question regarding validate-abi.sh.
> >
> > devtools/validate-abi.sh should be removed.
> > Please use the new devtools/check-abi.sh
>
> The file doc/guides/contributing/abi_versioning.rst
> should be updated as well.
> Neil? Ray?
>
Ack, we should remove validate_abi and update the docs
Neil
>
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v3 00/33] DPDK Trace support
2020-04-01 10:04 0% ` Jerin Jacob
@ 2020-04-01 14:12 0% ` David Marchand
2020-04-01 14:16 0% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-04-01 14:12 UTC (permalink / raw)
To: Jerin Jacob Kollanukkaran, Thomas Monjalon
Cc: dev, Bruce Richardson, Mattias Rönnblom, Sunil Kumar Kori,
Yigit, Ferruh, Andrew Rybchenko, Declan Doherty, Olivier Matz,
Neil Horman, Ray Kinsella
On Wed, Apr 1, 2020 at 12:05 PM Jerin Jacob <jerinjacobk@gmail.com> wrote:
> On Wed, Apr 1, 2020 at 1:49 PM David Marchand <david.marchand@redhat.com> wrote:
> > - Regardless of the trace framework, the ALLOW_EXPERIMENTAL_API flag
> > gates access to APIs so that external users are aware of their status.
> > I have been considering setting this flag unconditionally for internal
> > users in the top Makefile/meson for app/ lib/ and drivers/.
> > I could look at this and prepare a patch about this, but this is not
> > enough here.
>
> It makes sense to me. Let me know when you are planning to send that patch,
> I will rebase this series on top of that.
Feel free to take over, thanks.
>
> If you don't have time then I can send the patch too.
> I assume the patch content will be:
> 1) Removing experimental API from app, lib, drivers, examples with
> make and meson
> 2) Have it enabled at the global level.
examples are a special case as they can be compiled out of the dpdk sources.
This is why I excluded them of the list in my mail before.
> > How about:
> > * we introduce a global config switch that enables/disables the trace
> > framework (off by default): the people who want to test it and help
> > stabilize it will have to deal with the experimental flag for now,
> > * in 20.11, the trace points are put into the stable ABI, and the
> > option is removed,
>
> IMO, the better alternative would be:
>
> Since the trace changes in the "inline" functions of the specific
> library already
> disabled under _compile time_ RTE_ENABLE_TRACE_DP flag and
> it is using RTE_TRACE_POINT_DP() to define the trace unlike slow path
> trace like RTE_TRACE_POINT().
Ah indeed.
Note: RTE_ENABLE_TRACE_DP is not plugged with meson.
> So I can improve RTE_TRACE_POINT_DP() to make absolute NOP if
> ALLOW_EXPERIMENTAL_API not defined.
>
> On the upside,
> The tracing code will be enabled by default(runtime it is disabled by
> default anyway).
> If some need to fastpath API tracing then ALLOW_EXPERIMENTAL_API need
> to enable.
> So this won't break applications.
So either keep the RTE_ENABLE_TRACE_DP flag or use
ALLOW_EXPERIMENTAL_API... no opinion.
Thomas?
> > - With the patchset rebased on the current master (could be my fault,
> > so take it with a grain of salt), the ALLOW_EXPERIMENTAL_API flag is
> > not passed when compiling the l3fwd example against an installed dpdk.
>
> I will check. We have added ALLOW_EXPERIMENTAL_API flag where we got
> compilation issues.
No compilation issue, just big fat warnings reporting that l3fwd did
not enable ALLOW_EXPERIMENTAL_API and it is not built with -Werror.
I caught it with ./devtools/test-*build*.sh scripts.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v3 00/33] DPDK Trace support
2020-04-01 14:12 0% ` David Marchand
@ 2020-04-01 14:16 0% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-04-01 14:16 UTC (permalink / raw)
To: Jerin Jacob Kollanukkaran, David Marchand
Cc: dev, Bruce Richardson, Mattias Rönnblom, Sunil Kumar Kori,
Yigit, Ferruh, Andrew Rybchenko, Declan Doherty, Olivier Matz,
Neil Horman, Ray Kinsella
01/04/2020 16:12, David Marchand:
> On Wed, Apr 1, 2020 at 12:05 PM Jerin Jacob <jerinjacobk@gmail.com> wrote:
> > On Wed, Apr 1, 2020 at 1:49 PM David Marchand <david.marchand@redhat.com> wrote:
> > > - Regardless of the trace framework, the ALLOW_EXPERIMENTAL_API flag
> > > gates access to APIs so that external users are aware of their status.
> > > I have been considering setting this flag unconditionally for internal
> > > users in the top Makefile/meson for app/ lib/ and drivers/.
> > > I could look at this and prepare a patch about this, but this is not
> > > enough here.
> >
> > It makes sense to me. Let me know when you are planning to send that patch,
> > I will rebase this series on top of that.
>
> Feel free to take over, thanks.
>
>
> >
> > If you don't have time then I can send the patch too.
> > I assume the patch content will be:
> > 1) Removing experimental API from app, lib, drivers, examples with
> > make and meson
> > 2) Have it enabled at the global level.
>
> examples are a special case as they can be compiled out of the dpdk sources.
> This is why I excluded them of the list in my mail before.
>
>
> > > How about:
> > > * we introduce a global config switch that enables/disables the trace
> > > framework (off by default): the people who want to test it and help
> > > stabilize it will have to deal with the experimental flag for now,
> > > * in 20.11, the trace points are put into the stable ABI, and the
> > > option is removed,
> >
> > IMO, the better alternative would be:
> >
> > Since the trace changes in the "inline" functions of the specific
> > library already
> > disabled under _compile time_ RTE_ENABLE_TRACE_DP flag and
> > it is using RTE_TRACE_POINT_DP() to define the trace unlike slow path
> > trace like RTE_TRACE_POINT().
>
> Ah indeed.
> Note: RTE_ENABLE_TRACE_DP is not plugged with meson.
>
>
> > So I can improve RTE_TRACE_POINT_DP() to make absolute NOP if
> > ALLOW_EXPERIMENTAL_API not defined.
> >
> > On the upside,
> > The tracing code will be enabled by default(runtime it is disabled by
> > default anyway).
> > If some need to fastpath API tracing then ALLOW_EXPERIMENTAL_API need
> > to enable.
> > So this won't break applications.
>
> So either keep the RTE_ENABLE_TRACE_DP flag or use
> ALLOW_EXPERIMENTAL_API... no opinion.
> Thomas?
Anyway we need a compile-time option?
The option is just for compatibility?
Then ALLOW_EXPERIMENTAL_API looks to be the right option.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2 1/6] rte_ethdev: add function to check if device is owned
@ 2020-04-01 22:24 3% ` Stephen Hemminger
2020-04-02 8:04 3% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Stephen Hemminger @ 2020-04-01 22:24 UTC (permalink / raw)
To: Thomas Monjalon; +Cc: dev, ferruh.yigit, arybchenko
On Wed, 01 Apr 2020 23:42:44 +0200
Thomas Monjalon <thomas@monjalon.net> wrote:
> 16/03/2020 17:09, Stephen Hemminger:
> > This is a simple helper function to check if device is owned
> > (being used as a sub-device).
>
> I would suggest not restricting ownership to sub-device case.
>
> > It is more convienent than having
> > applications call rte_eth_dev_owner_get and check the result.
>
> Yes it is more convenient but I don't like adding such simple wrapper.
>
> I propose to extend rte_eth_dev_owner_get() behaviour:
> if the owner pointer is NULL, the function returns 0 only
> if an owner (not RTE_ETH_DEV_NO_OWNER) is found.
>
> So instead of using your wrapper:
> if (rte_eth_dev_is_owned(port_id))
> you can use:
> if (rte_eth_dev_owner_get(port_id, NULL) == 0)
That is not how rte_eth_dev_owner_get works now.
Passing NULL to it would crash.
And if devices is not owned rte_eth_dev_owner_get() returns 0
and owner is set to a magic value. We could change the ABI for this
since it is marked experimental. But that seems more risky.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2 1/6] rte_ethdev: add function to check if device is owned
2020-04-01 22:24 3% ` Stephen Hemminger
@ 2020-04-02 8:04 3% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-04-02 8:04 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: dev, ferruh.yigit, arybchenko
02/04/2020 00:24, Stephen Hemminger:
> On Wed, 01 Apr 2020 23:42:44 +0200
> Thomas Monjalon <thomas@monjalon.net> wrote:
>
> > 16/03/2020 17:09, Stephen Hemminger:
> > > This is a simple helper function to check if device is owned
> > > (being used as a sub-device).
> >
> > I would suggest not restricting ownership to sub-device case.
> >
> > > It is more convienent than having
> > > applications call rte_eth_dev_owner_get and check the result.
> >
> > Yes it is more convenient but I don't like adding such simple wrapper.
> >
> > I propose to extend rte_eth_dev_owner_get() behaviour:
> > if the owner pointer is NULL, the function returns 0 only
> > if an owner (not RTE_ETH_DEV_NO_OWNER) is found.
> >
> > So instead of using your wrapper:
> > if (rte_eth_dev_is_owned(port_id))
> > you can use:
> > if (rte_eth_dev_owner_get(port_id, NULL) == 0)
>
> That is not how rte_eth_dev_owner_get works now.
> Passing NULL to it would crash.
Indeed this is why I am proposing to extend it
and fill this case.
> And if devices is not owned rte_eth_dev_owner_get() returns 0
> and owner is set to a magic value. We could change the ABI for this
> since it is marked experimental. But that seems more risky.
This is not an ABI change.
Just manage a case which was crashing previously.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] /validate-abi.sh complains [PATCH v1 3/8] ring: introduce RTS ring mode
2020-04-01 6:38 9% ` David Marchand
@ 2020-04-02 12:36 4% ` Ananyev, Konstantin
2020-04-02 13:13 4% ` Dodji Seketeli
2020-04-02 14:27 4% ` David Marchand
0 siblings, 2 replies; 200+ results
From: Ananyev, Konstantin @ 2020-04-02 12:36 UTC (permalink / raw)
To: David Marchand
Cc: thomas, nhorman, honnappa.nagarahalli, dev, Richardson, Bruce,
Yigit, Ferruh, Dodji Seketeli, Laatz, Kevin
Hi David,
> > Have a question regarding validate-abi.sh.
> > It complains on the following changes with that patch:
> >
> > @@ -111,11 +129,21 @@ struct rte_ring {
> > char pad0 __rte_cache_aligned; /**< empty cache line */
> >
> > /** Ring producer status. */
> > - struct rte_ring_headtail prod __rte_cache_aligned;
> > + RTE_STD_C11
> > + union {
> > + struct rte_ring_headtail prod;
> > + struct rte_ring_rts_headtail rts_prod;
> > + } __rte_cache_aligned;
> > +
> > char pad1 __rte_cache_aligned; /**< empty cache line */
> >
> > /** Ring consumer status. */
> > - struct rte_ring_headtail cons __rte_cache_aligned;
> > + RTE_STD_C11
> > + union {
> > + struct rte_ring_headtail cons;
> > + struct rte_ring_rts_headtail rts_cons;
> > + } __rte_cache_aligned;
> > +
> >
> > Complaints:
> > rte_ring.h
> > [−] struct rte_ring 2
> >
> > 1
> > Change: Field cons has been removed from this type.
> > Effect: Applications will access incorrect memory when attempting to access this field.
> > 2
> > Change: Field prod has been removed from this type.
> > Effect: Applications will access incorrect memory when attempting to access this field.
> >
> > From my perspective it looks false-positive:
> > *prod* and *cons* fields are still there,
> > their format, size and offset within rte_ring remain the same.
> > Is that some limitation with the tool, or am I missing something here?
>
> - Side note, we have build failures with clang and ARM jobs:
> https://travis-ci.com/github/ovsrobot/dpdk/builds/157277423
ack, will fix.
>
>
> - We switched to libabigail called from devtools/check-abi.sh which
> you can run locally
> (https://doc.dpdk.org/guides/contributing/patches.html?highlight=abi#checking-abi-compatibility).
> Or you can count on Aaron's robot to do this check in Travis.
>
> It reported a warning on those fields:
> https://travis-ci.com/github/ovsrobot/dpdk/jobs/310689008#L2380
> I understand this as a false positive too.
>
> It seems similar to the bz I opened about fields moved to anonymous
> constructs: https://sourceware.org/bugzilla/show_bug.cgi?id=25661
Yes, looks the same.
> Cc: Dodji.
>
>
> For the time being, you can waive this by adding a rule in
> devtools/libabigail.abignore.
Ok, so what is the procedure here?
Should I submit changes to devtools/libabigail.abignore together with the
patch that introduced the problem?
BTW, I used the following changes in libabigail.abignore to supress errors:
$ git diff devtools/libabigail.abignore
diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore
index a59df8f13..032479b9f 100644
--- a/devtools/libabigail.abignore
+++ b/devtools/libabigail.abignore
@@ -11,3 +11,9 @@
type_kind = enum
name = rte_crypto_asym_xform_type
changed_enumerators = RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
+; Ignore updates of ring prod/cons
+[suppress_type]
+ type_kind = struct
+ name = rte_ring
+ has_data_members_inserted_at = offsetof(prod)
+ has_data_members_inserted_at = offsetof(cons)
Do you know, is there a better (more fine-grained) approach?
Thanks
Konstantin
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] /validate-abi.sh complains [PATCH v1 3/8] ring: introduce RTS ring mode
2020-04-02 12:36 4% ` Ananyev, Konstantin
@ 2020-04-02 13:13 4% ` Dodji Seketeli
2020-04-02 14:27 4% ` David Marchand
1 sibling, 0 replies; 200+ results
From: Dodji Seketeli @ 2020-04-02 13:13 UTC (permalink / raw)
To: Ananyev, Konstantin
Cc: David Marchand, thomas, nhorman, honnappa.nagarahalli, dev,
Richardson, Bruce, Yigit, Ferruh, Dodji Seketeli, Laatz, Kevin
Hello,
"Ananyev, Konstantin" <konstantin.ananyev@intel.com> writes:
> Hi David,
>
>> > Have a question regarding validate-abi.sh.
>> > It complains on the following changes with that patch:
>> >
>> > @@ -111,11 +129,21 @@ struct rte_ring {
>> > char pad0 __rte_cache_aligned; /**< empty cache line */
>> >
>> > /** Ring producer status. */
>> > - struct rte_ring_headtail prod __rte_cache_aligned;
>> > + RTE_STD_C11
>> > + union {
>> > + struct rte_ring_headtail prod;
>> > + struct rte_ring_rts_headtail rts_prod;
>> > + } __rte_cache_aligned;
>> > +
>> > char pad1 __rte_cache_aligned; /**< empty cache line */
>> >
>> > /** Ring consumer status. */
>> > - struct rte_ring_headtail cons __rte_cache_aligned;
>> > + RTE_STD_C11
>> > + union {
>> > + struct rte_ring_headtail cons;
>> > + struct rte_ring_rts_headtail rts_cons;
>> > + } __rte_cache_aligned;
>> > +
[...]
>> It reported a warning on those fields:
>> https://travis-ci.com/github/ovsrobot/dpdk/jobs/310689008#L2380
>> I understand this as a false positive too.
>>
>> It seems similar to the bz I opened about fields moved to anonymous
>> constructs: https://sourceware.org/bugzilla/show_bug.cgi?id=25661
>
> Yes, looks the same.
>
>
>> Cc: Dodji.
>>
>>
>> For the time being, you can waive this by adding a rule in
>> devtools/libabigail.abignore.
>
> Ok, so what is the procedure here?
> Should I submit changes to devtools/libabigail.abignore together with the
> patch that introduced the problem?
> BTW, I used the following changes in libabigail.abignore to supress errors:
> $ git diff devtools/libabigail.abignore
> diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore
> index a59df8f13..032479b9f 100644
> --- a/devtools/libabigail.abignore
> +++ b/devtools/libabigail.abignore
> @@ -11,3 +11,9 @@
> type_kind = enum
> name = rte_crypto_asym_xform_type
> changed_enumerators = RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
> +; Ignore updates of ring prod/cons
> +[suppress_type]
> + type_kind = struct
> + name = rte_ring
> + has_data_members_inserted_at = offsetof(prod)
> + has_data_members_inserted_at = offsetof(cons)
>
> Do you know, is there a better (more fine-grained) approach?
I think what you did is correct.
I am teaching the tool to recognize this kind of change construct and
avoid emitting a false positive. So that you can drop these kind of
suppression specification.
Sorry for the inconvenience.
Cheers,
--
Dodji
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] /validate-abi.sh complains [PATCH v1 3/8] ring: introduce RTS ring mode
2020-04-02 12:36 4% ` Ananyev, Konstantin
2020-04-02 13:13 4% ` Dodji Seketeli
@ 2020-04-02 14:27 4% ` David Marchand
2020-04-02 15:25 4% ` Ananyev, Konstantin
1 sibling, 1 reply; 200+ results
From: David Marchand @ 2020-04-02 14:27 UTC (permalink / raw)
To: Ananyev, Konstantin, Dodji Seketeli
Cc: thomas, nhorman, honnappa.nagarahalli, dev, Richardson, Bruce,
Yigit, Ferruh, Laatz, Kevin
On Thu, Apr 2, 2020 at 2:37 PM Ananyev, Konstantin
<konstantin.ananyev@intel.com> wrote:
> > For the time being, you can waive this by adding a rule in
> > devtools/libabigail.abignore.
>
> Ok, so what is the procedure here?
> Should I submit changes to devtools/libabigail.abignore together with the
> patch that introduced the problem?
> BTW, I used the following changes in libabigail.abignore to supress errors:
> $ git diff devtools/libabigail.abignore
> diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore
> index a59df8f13..032479b9f 100644
> --- a/devtools/libabigail.abignore
> +++ b/devtools/libabigail.abignore
> @@ -11,3 +11,9 @@
> type_kind = enum
> name = rte_crypto_asym_xform_type
> changed_enumerators = RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
> +; Ignore updates of ring prod/cons
> +[suppress_type]
> + type_kind = struct
> + name = rte_ring
> + has_data_members_inserted_at = offsetof(prod)
> + has_data_members_inserted_at = offsetof(cons)
Hum, I could not find offsetof() in the manual but I did find offset_of().
But anyway, I understand this has_data_members_inserted_at is a way to
select structs.
Not a way to filter out changes that touches fields offsets inside the
rte_ring structure.
So you might as well simplify the rule and suppress the whole rte_ring
struct changes.
I suspect this is what is happening here: abidiff did not complain
about the offsetof vs offset_of thing, so the
has_data_members_inserted_at criterias may have been ignored.
--
David Marchand
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] /validate-abi.sh complains [PATCH v1 3/8] ring: introduce RTS ring mode
2020-04-02 14:27 4% ` David Marchand
@ 2020-04-02 15:25 4% ` Ananyev, Konstantin
0 siblings, 0 replies; 200+ results
From: Ananyev, Konstantin @ 2020-04-02 15:25 UTC (permalink / raw)
To: David Marchand, Dodji Seketeli
Cc: thomas, nhorman, honnappa.nagarahalli, dev, Richardson, Bruce,
Yigit, Ferruh, Laatz, Kevin
> >
> > Ok, so what is the procedure here?
> > Should I submit changes to devtools/libabigail.abignore together with the
> > patch that introduced the problem?
> > BTW, I used the following changes in libabigail.abignore to supress errors:
> > $ git diff devtools/libabigail.abignore
> > diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore
> > index a59df8f13..032479b9f 100644
> > --- a/devtools/libabigail.abignore
> > +++ b/devtools/libabigail.abignore
> > @@ -11,3 +11,9 @@
> > type_kind = enum
> > name = rte_crypto_asym_xform_type
> > changed_enumerators = RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
> > +; Ignore updates of ring prod/cons
> > +[suppress_type]
> > + type_kind = struct
> > + name = rte_ring
> > + has_data_members_inserted_at = offsetof(prod)
> > + has_data_members_inserted_at = offsetof(cons)
>
> Hum, I could not find offsetof() in the manual but I did find offset_of().
Yep, typo from me.
>
> But anyway, I understand this has_data_members_inserted_at is a way to
> select structs.
> Not a way to filter out changes that touches fields offsets inside the
> rte_ring structure.
Seems so.
> So you might as well simplify the rule and suppress the whole rte_ring
> struct changes.
Ok.
> I suspect this is what is happening here: abidiff did not complain
> about the offsetof vs offset_of thing, so the
> has_data_members_inserted_at criterias may have been ignored.
>
Konstantin
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH v2 0/9] New sync modes for ring
2020-03-31 16:43 3% ` [dpdk-dev] [PATCH v1 0/8] " Konstantin Ananyev
@ 2020-04-02 22:09 3% ` Konstantin Ananyev
2020-04-02 22:09 1% ` [dpdk-dev] [PATCH v2 3/9] ring: introduce RTS ring mode Konstantin Ananyev
2020-04-03 17:42 3% ` [dpdk-dev] [PATCH v3 0/9] New sync modes for ring Konstantin Ananyev
0 siblings, 2 replies; 200+ results
From: Konstantin Ananyev @ 2020-04-02 22:09 UTC (permalink / raw)
To: dev; +Cc: honnappa.nagarahalli, david.marchand, jielong.zjl, Konstantin Ananyev
V1 - v2 changes:
1. Fix compilation issues
2. Add C11 atomics support
3. Updates devtools/libabigail.abignore (workaround)
RFC - V1 changes:
1. remove ABI brekage (at least I hope I did)
2. Add support for ring_elem
3. Rework peek related API a bit
4. Rework test to make it less verbose and unite all test-cases
in one command
5. Add new test-case for MT peek API
TODO list:
1. Update docs
These days more and more customers use(/try to use) DPDK based apps within
overcommitted systems (multiple acttive threads over same pysical cores):
VM, container deployments, etc.
One quite common problem they hit:
Lock-Holder-Preemption/Lock-Waiter-Preemption with rte_ring.
LHP is quite a common problem for spin-based sync primitives
(spin-locks, etc.) on overcommitted systems.
The situation gets much worse when some sort of
fair-locking technique is used (ticket-lock, etc.).
As now not only lock-owner but also lock-waiters scheduling
order matters a lot (LWP).
These two problems are well-known for kernel within VMs:
http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
The problem with rte_ring is that while head accusion is sort of
un-fair locking, waiting on tail is very similar to ticket lock schema -
tail has to be updated in particular order.
So while head update exibits only LHP scneario,
tail wait and update can cause an LWP.
That makes current rte_ring implementation to perform
really pure on some overcommited scenarios.
While it is probably not possible to completely resolve LHP problem in
userspace only (without some kernel communication/intervention),
removing fairness in tail update can mitigate current LWP significantly.
This RFC proposes two new optional ring synchronization modes:
1) Head/Tail Sync (HTS) mode
In that mode enqueue/dequeue operation is fully serialized:
only one thread at a time is allowed to perform given op.
As another enhancement provide ability to split enqueue/dequeue
operation into two phases:
- enqueue/dequeue start
- enqueue/dequeue finish
That allows user to inspect objects in the ring without removing
them from it (aka MT safe peek).
2) Relaxed Tail Sync (RTS)
The main difference from original MP/MC algorithm is that
tail value is increased not by every thread that finished enqueue/dequeue,
but only by the last one.
That allows threads to avoid spinning on ring tail value,
leaving actual tail value change to the last thread in the update queue.
Note that these new sync modes are optional.
For current rte_ring users nothing should change
(both in terms of API/ABI and performance).
Existing sync modes MP/MC,SP/SC kept untouched, set up in the same
way (via flags and _init_), and MP/MC remains as default one.
The only thing that changed:
Format of prod/cons now could differ depending on mode selected at _init_.
So user has to stick with one sync model through whole ring lifetime.
In other words, user can't create a ring for let say SP mode and then
in the middle of data-path change his mind and start using MP_RTS mode.
For existing modes (SP/MP, SC/MC) format remains the same and
user can still use them interchangeably, though of course it is an
error prone practice.
Test results on IA (see below) show significant improvements
for average enqueue/dequeue op times on overcommitted systems.
For 'classic' DPDK deployments (one thread per core) original MP/MC
algorithm still shows best numbers, though for 64-bit target
RTS numbers are not that far away.
Numbers were produced by new UT test-case: ring_stress_autotest, i.e.:
echo ring_stress_autotest | ./dpdk-test -n 4 --lcores='...'
X86_64 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
DEQ+ENQ average cycles/obj
MP/MC HTS RTS
1thread@1core(--lcores=6-7) 8.00 8.15 8.99
2thread@2core(--lcores=6-8) 19.14 19.61 20.35
4thread@4core(--lcores=6-10) 29.43 29.79 31.82
8thread@8core(--lcores=6-14) 110.59 192.81 119.50
16thread@16core(--lcores=6-22) 461.03 813.12 495.59
32thread/@32core(--lcores='6-22,55-70') 982.90 1972.38 1160.51
2thread@1core(--lcores='6,(10-11)@7' 20140.50 23.58 25.14
4thread@2core(--lcores='6,(10-11)@7,(20-21)@8' 153680.60 76.88 80.05
8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 280314.32 294.72 318.79
16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 643176.59 1144.02 1175.14
32thread@2core(--lcores='6,(10-25)@7,(30-45)@8' 4264238.80 4627.48 4892.68
8thread@2core(--lcores='6,(10-17)@(7,8))' 321085.98 298.59 307.47
16thread@4core(--lcores='6,(20-35)@(7-10))' 1900705.61 575.35 678.29
32thread@4core(--lcores='6,(20-51)@(7-10))' 5510445.85 2164.36 2714.12
i686 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
DEQ+ENQ average cycles/obj
MP/MC HTS RTS
1thread@1core(--lcores=6-7) 7.85 12.13 11.31
2thread@2core(--lcores=6-8) 17.89 24.52 21.86
8thread@8core(--lcores=6-14) 32.58 354.20 54.58
32thread/@32core(--lcores='6-22,55-70') 813.77 6072.41 2169.91
2thread@1core(--lcores='6,(10-11)@7' 16095.00 36.06 34.74
8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 1140354.54 346.61 361.57
16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 1920417.86 1314.90 1416.65
8thread@2core(--lcores='6,(10-17)@(7,8))' 594358.61 332.70 357.74
32thread@4core(--lcores='6,(20-51)@(7-10))' 5319896.86 2836.44 3028.87
Konstantin Ananyev (9):
test/ring: add contention stress test
ring: prepare ring to allow new sync schemes
ring: introduce RTS ring mode
test/ring: add contention stress test for RTS ring
ring: introduce HTS ring mode
test/ring: add contention stress test for HTS ring
ring: introduce peek style API
test/ring: add stress test for MT peek API
ring: add C11 memory model for new sync modes
app/test/Makefile | 5 +
app/test/meson.build | 5 +
app/test/test_pdump.c | 6 +-
app/test/test_ring_hts_stress.c | 32 ++
app/test/test_ring_mpmc_stress.c | 31 ++
app/test/test_ring_peek_stress.c | 43 +++
app/test/test_ring_rts_stress.c | 32 ++
app/test/test_ring_stress.c | 57 ++++
app/test/test_ring_stress.h | 38 +++
app/test/test_ring_stress_impl.h | 444 +++++++++++++++++++++++++
devtools/libabigail.abignore | 4 +
lib/librte_pdump/rte_pdump.c | 2 +-
lib/librte_port/rte_port_ring.c | 12 +-
lib/librte_ring/Makefile | 11 +-
lib/librte_ring/meson.build | 11 +-
lib/librte_ring/rte_ring.c | 114 ++++++-
lib/librte_ring/rte_ring.h | 244 ++++++++++++--
lib/librte_ring/rte_ring_c11_mem.h | 44 +++
lib/librte_ring/rte_ring_elem.h | 105 +++++-
lib/librte_ring/rte_ring_generic.h | 48 +++
lib/librte_ring/rte_ring_hts.h | 214 ++++++++++++
lib/librte_ring/rte_ring_hts_c11_mem.h | 222 +++++++++++++
lib/librte_ring/rte_ring_hts_elem.h | 209 ++++++++++++
lib/librte_ring/rte_ring_hts_generic.h | 235 +++++++++++++
lib/librte_ring/rte_ring_peek.h | 379 +++++++++++++++++++++
lib/librte_ring/rte_ring_rts.h | 320 ++++++++++++++++++
lib/librte_ring/rte_ring_rts_c11_mem.h | 198 +++++++++++
lib/librte_ring/rte_ring_rts_elem.h | 209 ++++++++++++
lib/librte_ring/rte_ring_rts_generic.h | 210 ++++++++++++
29 files changed, 3428 insertions(+), 56 deletions(-)
create mode 100644 app/test/test_ring_hts_stress.c
create mode 100644 app/test/test_ring_mpmc_stress.c
create mode 100644 app/test/test_ring_peek_stress.c
create mode 100644 app/test/test_ring_rts_stress.c
create mode 100644 app/test/test_ring_stress.c
create mode 100644 app/test/test_ring_stress.h
create mode 100644 app/test/test_ring_stress_impl.h
create mode 100644 lib/librte_ring/rte_ring_hts.h
create mode 100644 lib/librte_ring/rte_ring_hts_c11_mem.h
create mode 100644 lib/librte_ring/rte_ring_hts_elem.h
create mode 100644 lib/librte_ring/rte_ring_hts_generic.h
create mode 100644 lib/librte_ring/rte_ring_peek.h
create mode 100644 lib/librte_ring/rte_ring_rts.h
create mode 100644 lib/librte_ring/rte_ring_rts_c11_mem.h
create mode 100644 lib/librte_ring/rte_ring_rts_elem.h
create mode 100644 lib/librte_ring/rte_ring_rts_generic.h
--
2.17.1
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v2 3/9] ring: introduce RTS ring mode
2020-04-02 22:09 3% ` [dpdk-dev] [PATCH v2 0/9] " Konstantin Ananyev
@ 2020-04-02 22:09 1% ` Konstantin Ananyev
2020-04-03 17:42 3% ` [dpdk-dev] [PATCH v3 0/9] New sync modes for ring Konstantin Ananyev
1 sibling, 0 replies; 200+ results
From: Konstantin Ananyev @ 2020-04-02 22:09 UTC (permalink / raw)
To: dev; +Cc: honnappa.nagarahalli, david.marchand, jielong.zjl, Konstantin Ananyev
Introduce relaxed tail sync (RTS) mode for MT ring synchronization.
Aim to reduce stall times in case when ring is used on
overcommited cpus (multiple active threads on the same cpu).
The main difference from original MP/MC algorithm is that
tail value is increased not by every thread that finished enqueue/dequeue,
but only by the last one.
That allows threads to avoid spinning on ring tail value,
leaving actual tail value change to the last thread in the update queue.
check-abi.sh reports what I believe is a false-positive about
ring cons/prod changes. As a workaround, devtools/libabigail.abignore is
updated to suppress *struct ring* related errors.
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
devtools/libabigail.abignore | 4 +
lib/librte_ring/Makefile | 5 +-
lib/librte_ring/meson.build | 5 +-
lib/librte_ring/rte_ring.c | 100 +++++++-
lib/librte_ring/rte_ring.h | 110 ++++++++-
lib/librte_ring/rte_ring_elem.h | 86 ++++++-
lib/librte_ring/rte_ring_rts.h | 316 +++++++++++++++++++++++++
lib/librte_ring/rte_ring_rts_elem.h | 205 ++++++++++++++++
lib/librte_ring/rte_ring_rts_generic.h | 210 ++++++++++++++++
9 files changed, 1012 insertions(+), 29 deletions(-)
create mode 100644 lib/librte_ring/rte_ring_rts.h
create mode 100644 lib/librte_ring/rte_ring_rts_elem.h
create mode 100644 lib/librte_ring/rte_ring_rts_generic.h
diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore
index a59df8f13..ece014111 100644
--- a/devtools/libabigail.abignore
+++ b/devtools/libabigail.abignore
@@ -11,3 +11,7 @@
type_kind = enum
name = rte_crypto_asym_xform_type
changed_enumerators = RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
+; Ignore updates of ring prod/cons
+[suppress_type]
+ type_kind = struct
+ name = rte_ring
diff --git a/lib/librte_ring/Makefile b/lib/librte_ring/Makefile
index 917c560ad..8f5c284cc 100644
--- a/lib/librte_ring/Makefile
+++ b/lib/librte_ring/Makefile
@@ -18,6 +18,9 @@ SRCS-$(CONFIG_RTE_LIBRTE_RING) := rte_ring.c
SYMLINK-$(CONFIG_RTE_LIBRTE_RING)-include := rte_ring.h \
rte_ring_elem.h \
rte_ring_generic.h \
- rte_ring_c11_mem.h
+ rte_ring_c11_mem.h \
+ rte_ring_rts.h \
+ rte_ring_rts_elem.h \
+ rte_ring_rts_generic.h
include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_ring/meson.build b/lib/librte_ring/meson.build
index f2f3ccc88..612936afb 100644
--- a/lib/librte_ring/meson.build
+++ b/lib/librte_ring/meson.build
@@ -5,7 +5,10 @@ sources = files('rte_ring.c')
headers = files('rte_ring.h',
'rte_ring_elem.h',
'rte_ring_c11_mem.h',
- 'rte_ring_generic.h')
+ 'rte_ring_generic.h',
+ 'rte_ring_rts.h',
+ 'rte_ring_rts_elem.h',
+ 'rte_ring_rts_generic.h')
# rte_ring_create_elem and rte_ring_get_memsize_elem are experimental
allow_experimental_apis = true
diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c
index fa5733907..222eec0fb 100644
--- a/lib/librte_ring/rte_ring.c
+++ b/lib/librte_ring/rte_ring.c
@@ -45,6 +45,9 @@ EAL_REGISTER_TAILQ(rte_ring_tailq)
/* true if x is a power of 2 */
#define POWEROF2(x) ((((x)-1) & (x)) == 0)
+/* by default set head/tail distance as 1/8 of ring capacity */
+#define HTD_MAX_DEF 8
+
/* return the size of memory occupied by a ring */
ssize_t
rte_ring_get_memsize_elem(unsigned int esize, unsigned int count)
@@ -79,11 +82,84 @@ rte_ring_get_memsize(unsigned int count)
return rte_ring_get_memsize_elem(sizeof(void *), count);
}
+/*
+ * internal helper function to reset prod/cons head-tail values.
+ */
+static void
+reset_headtail(void *p)
+{
+ struct rte_ring_headtail *ht;
+ struct rte_ring_rts_headtail *ht_rts;
+
+ ht = p;
+ ht_rts = p;
+
+ switch (ht->sync_type) {
+ case RTE_RING_SYNC_MT:
+ case RTE_RING_SYNC_ST:
+ ht->head = 0;
+ ht->tail = 0;
+ break;
+ case RTE_RING_SYNC_MT_RTS:
+ ht_rts->head.raw = 0;
+ ht_rts->tail.raw = 0;
+ break;
+ default:
+ /* unknown sync mode */
+ RTE_ASSERT(0);
+ }
+}
+
void
rte_ring_reset(struct rte_ring *r)
{
- r->prod.head = r->cons.head = 0;
- r->prod.tail = r->cons.tail = 0;
+ reset_headtail(&r->prod);
+ reset_headtail(&r->cons);
+}
+
+/*
+ * helper function, calculates sync_type values for prod and cons
+ * based on input flags. Returns zero at success or negative
+ * errno value otherwise.
+ */
+static int
+get_sync_type(uint32_t flags, enum rte_ring_sync_type *prod_st,
+ enum rte_ring_sync_type *cons_st)
+{
+ static const uint32_t prod_st_flags =
+ (RING_F_SP_ENQ | RING_F_MP_RTS_ENQ);
+ static const uint32_t cons_st_flags =
+ (RING_F_SC_DEQ | RING_F_MC_RTS_DEQ);
+
+ switch (flags & prod_st_flags) {
+ case 0:
+ *prod_st = RTE_RING_SYNC_MT;
+ break;
+ case RING_F_SP_ENQ:
+ *prod_st = RTE_RING_SYNC_ST;
+ break;
+ case RING_F_MP_RTS_ENQ:
+ *prod_st = RTE_RING_SYNC_MT_RTS;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ switch (flags & cons_st_flags) {
+ case 0:
+ *cons_st = RTE_RING_SYNC_MT;
+ break;
+ case RING_F_SC_DEQ:
+ *cons_st = RTE_RING_SYNC_ST;
+ break;
+ case RING_F_MC_RTS_DEQ:
+ *cons_st = RTE_RING_SYNC_MT_RTS;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
}
int
@@ -100,16 +176,20 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
RTE_BUILD_BUG_ON((offsetof(struct rte_ring, prod) &
RTE_CACHE_LINE_MASK) != 0);
+ RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, sync_type) !=
+ offsetof(struct rte_ring_rts_headtail, sync_type));
+ RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, tail) !=
+ offsetof(struct rte_ring_rts_headtail, tail.val.pos));
+
/* init the ring structure */
memset(r, 0, sizeof(*r));
ret = strlcpy(r->name, name, sizeof(r->name));
if (ret < 0 || ret >= (int)sizeof(r->name))
return -ENAMETOOLONG;
r->flags = flags;
- r->prod.sync_type = (flags & RING_F_SP_ENQ) ?
- RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
- r->cons.sync_type = (flags & RING_F_SC_DEQ) ?
- RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
+ ret = get_sync_type(flags, &r->prod.sync_type, &r->cons.sync_type);
+ if (ret != 0)
+ return ret;
if (flags & RING_F_EXACT_SZ) {
r->size = rte_align32pow2(count + 1);
@@ -126,8 +206,12 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
r->mask = count - 1;
r->capacity = r->mask;
}
- r->prod.head = r->cons.head = 0;
- r->prod.tail = r->cons.tail = 0;
+
+ /* set default values for head-tail distance */
+ if (flags & RING_F_MP_RTS_ENQ)
+ rte_ring_set_prod_htd_max(r, r->capacity / HTD_MAX_DEF);
+ if (flags & RING_F_MC_RTS_DEQ)
+ rte_ring_set_cons_htd_max(r, r->capacity / HTD_MAX_DEF);
return 0;
}
diff --git a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h
index d4775a063..f6f084d79 100644
--- a/lib/librte_ring/rte_ring.h
+++ b/lib/librte_ring/rte_ring.h
@@ -48,6 +48,7 @@ extern "C" {
#include <rte_branch_prediction.h>
#include <rte_memzone.h>
#include <rte_pause.h>
+#include <rte_debug.h>
#define RTE_TAILQ_RING_NAME "RTE_RING"
@@ -65,10 +66,13 @@ enum rte_ring_queue_behavior {
enum rte_ring_sync_type {
RTE_RING_SYNC_MT, /**< multi-thread safe (default mode) */
RTE_RING_SYNC_ST, /**< single thread only */
+#ifdef ALLOW_EXPERIMENTAL_API
+ RTE_RING_SYNC_MT_RTS, /**< multi-thread relaxed tail sync */
+#endif
};
/**
- * structure to hold a pair of head/tail values and other metadata.
+ * structures to hold a pair of head/tail values and other metadata.
* Depending on sync_type format of that structure might be different,
* but offset for *sync_type* and *tail* values should remain the same.
*/
@@ -84,6 +88,21 @@ struct rte_ring_headtail {
};
};
+union rte_ring_ht_poscnt {
+ uint64_t raw;
+ struct {
+ uint32_t cnt; /**< head/tail reference counter */
+ uint32_t pos; /**< head/tail position */
+ } val;
+};
+
+struct rte_ring_rts_headtail {
+ volatile union rte_ring_ht_poscnt tail;
+ enum rte_ring_sync_type sync_type; /**< sync type of prod/cons */
+ uint32_t htd_max; /**< max allowed distance between head/tail */
+ volatile union rte_ring_ht_poscnt head;
+};
+
/**
* An RTE ring structure.
*
@@ -111,11 +130,21 @@ struct rte_ring {
char pad0 __rte_cache_aligned; /**< empty cache line */
/** Ring producer status. */
- struct rte_ring_headtail prod __rte_cache_aligned;
+ RTE_STD_C11
+ union {
+ struct rte_ring_headtail prod;
+ struct rte_ring_rts_headtail rts_prod;
+ } __rte_cache_aligned;
+
char pad1 __rte_cache_aligned; /**< empty cache line */
/** Ring consumer status. */
- struct rte_ring_headtail cons __rte_cache_aligned;
+ RTE_STD_C11
+ union {
+ struct rte_ring_headtail cons;
+ struct rte_ring_rts_headtail rts_cons;
+ } __rte_cache_aligned;
+
char pad2 __rte_cache_aligned; /**< empty cache line */
};
@@ -132,6 +161,9 @@ struct rte_ring {
#define RING_F_EXACT_SZ 0x0004
#define RTE_RING_SZ_MASK (0x7fffffffU) /**< Ring size mask */
+#define RING_F_MP_RTS_ENQ 0x0008 /**< The default enqueue is "MP RTS". */
+#define RING_F_MC_RTS_DEQ 0x0010 /**< The default dequeue is "MC RTS". */
+
#define __IS_SP RTE_RING_SYNC_ST
#define __IS_MP RTE_RING_SYNC_MT
#define __IS_SC RTE_RING_SYNC_ST
@@ -461,6 +493,10 @@ rte_ring_sp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
RTE_RING_SYNC_ST, free_space);
}
+#ifdef ALLOW_EXPERIMENTAL_API
+#include <rte_ring_rts.h>
+#endif
+
/**
* Enqueue several objects on a ring.
*
@@ -484,8 +520,21 @@ static __rte_always_inline unsigned int
rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
- return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- r->prod.sync_type, free_space);
+ switch (r->prod.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mp_enqueue_bulk(r, obj_table, n, free_space);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sp_enqueue_bulk(r, obj_table, n, free_space);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mp_rts_enqueue_bulk(r, obj_table, n,
+ free_space);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ return 0;
}
/**
@@ -619,8 +668,20 @@ static __rte_always_inline unsigned int
rte_ring_dequeue_bulk(struct rte_ring *r, void **obj_table, unsigned int n,
unsigned int *available)
{
- return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- r->cons.sync_type, available);
+ switch (r->cons.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mc_dequeue_bulk(r, obj_table, n, available);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sc_dequeue_bulk(r, obj_table, n, available);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mc_rts_dequeue_bulk(r, obj_table, n, available);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ return 0;
}
/**
@@ -940,8 +1001,21 @@ static __rte_always_inline unsigned
rte_ring_enqueue_burst(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
- return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_VARIABLE,
- r->prod.sync_type, free_space);
+ switch (r->prod.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mp_enqueue_burst(r, obj_table, n, free_space);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sp_enqueue_burst(r, obj_table, n, free_space);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mp_rts_enqueue_burst(r, obj_table, n,
+ free_space);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ return 0;
}
/**
@@ -1020,9 +1094,21 @@ static __rte_always_inline unsigned
rte_ring_dequeue_burst(struct rte_ring *r, void **obj_table,
unsigned int n, unsigned int *available)
{
- return __rte_ring_do_dequeue(r, obj_table, n,
- RTE_RING_QUEUE_VARIABLE,
- r->cons.sync_type, available);
+ switch (r->cons.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mc_dequeue_burst(r, obj_table, n, available);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sc_dequeue_burst(r, obj_table, n, available);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mc_rts_dequeue_burst(r, obj_table, n,
+ available);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ return 0;
}
#ifdef __cplusplus
diff --git a/lib/librte_ring/rte_ring_elem.h b/lib/librte_ring/rte_ring_elem.h
index 28f9836e6..5de0850dc 100644
--- a/lib/librte_ring/rte_ring_elem.h
+++ b/lib/librte_ring/rte_ring_elem.h
@@ -542,6 +542,8 @@ rte_ring_sp_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
RTE_RING_QUEUE_FIXED, __IS_SP, free_space);
}
+#include <rte_ring_rts_elem.h>
+
/**
* Enqueue several objects on a ring.
*
@@ -571,6 +573,26 @@ rte_ring_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
{
return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
RTE_RING_QUEUE_FIXED, r->prod.sync_type, free_space);
+
+ switch (r->prod.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mp_enqueue_bulk_elem(r, obj_table, esize, n,
+ free_space);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sp_enqueue_bulk_elem(r, obj_table, esize, n,
+ free_space);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mp_rts_enqueue_bulk_elem(r, obj_table, esize, n,
+ free_space);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ if (free_space != NULL)
+ *free_space = 0;
+ return 0;
}
/**
@@ -733,8 +755,25 @@ static __rte_always_inline unsigned int
rte_ring_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
unsigned int esize, unsigned int n, unsigned int *available)
{
- return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, r->cons.sync_type, available);
+ switch (r->cons.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mc_dequeue_bulk_elem(r, obj_table, esize, n,
+ available);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sc_dequeue_bulk_elem(r, obj_table, esize, n,
+ available);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mc_rts_dequeue_bulk_elem(r, obj_table, esize,
+ n, available);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ if (available != NULL)
+ *available = 0;
+ return 0;
}
/**
@@ -901,8 +940,25 @@ static __rte_always_inline unsigned
rte_ring_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
unsigned int esize, unsigned int n, unsigned int *free_space)
{
- return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE, r->prod.sync_type, free_space);
+ switch (r->prod.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mp_enqueue_burst_elem(r, obj_table, esize, n,
+ free_space);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sp_enqueue_burst_elem(r, obj_table, esize, n,
+ free_space);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mp_rts_enqueue_burst_elem(r, obj_table, esize,
+ n, free_space);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ if (free_space != NULL)
+ *free_space = 0;
+ return 0;
}
/**
@@ -993,9 +1049,25 @@ static __rte_always_inline unsigned int
rte_ring_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
unsigned int esize, unsigned int n, unsigned int *available)
{
- return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE,
- r->cons.sync_type, available);
+ switch (r->cons.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mc_dequeue_burst_elem(r, obj_table, esize, n,
+ available);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sc_dequeue_burst_elem(r, obj_table, esize, n,
+ available);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mc_rts_dequeue_burst_elem(r, obj_table, esize,
+ n, available);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ if (available != NULL)
+ *available = 0;
+ return 0;
}
#ifdef __cplusplus
diff --git a/lib/librte_ring/rte_ring_rts.h b/lib/librte_ring/rte_ring_rts.h
new file mode 100644
index 000000000..18404fe48
--- /dev/null
+++ b/lib/librte_ring/rte_ring_rts.h
@@ -0,0 +1,316 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#ifndef _RTE_RING_RTS_H_
+#define _RTE_RING_RTS_H_
+
+/**
+ * @file rte_ring_rts.h
+ * @b EXPERIMENTAL: this API may change without prior notice
+ * It is not recommended to include this file directly.
+ * Please include <rte_ring.h> instead.
+ *
+ * Contains functions for Relaxed Tail Sync (RTS) ring mode.
+ * The main idea remains the same as for our original MP/MC synchronization
+ * mechanism.
+ * The main difference is that tail value is increased not
+ * by every thread that finished enqueue/dequeue,
+ * but only by the last one doing enqueue/dequeue.
+ * That allows threads to skip spinning on tail value,
+ * leaving actual tail value change to last thread in the update queue.
+ * RTS requires 2 64-bit CAS for each enqueue(/dequeue) operation:
+ * one for head update, second for tail update.
+ * As a gain it allows thread to avoid spinning/waiting on tail value.
+ * In comparision original MP/MC algorithm requires one 32-bit CAS
+ * for head update and waiting/spinning on tail value.
+ *
+ * Brief outline:
+ * - introduce refcnt for both head and tail.
+ * - increment head.refcnt for each head.value update
+ * - write head:value and head:refcnt atomically (64-bit CAS)
+ * - move tail.value ahead only when tail.refcnt + 1 == head.refcnt
+ * - increment tail.refcnt when each enqueue/dequeue op finishes
+ * (no matter is tail:value going to change or not)
+ * - write tail.value and tail.recnt atomically (64-bit CAS)
+ *
+ * To avoid producer/consumer starvation:
+ * - limit max allowed distance between head and tail value (HTD_MAX).
+ * I.E. thread is allowed to proceed with changing head.value,
+ * only when: head.value - tail.value <= HTD_MAX
+ * HTD_MAX is an optional parameter.
+ * With HTD_MAX == 0 we'll have fully serialized ring -
+ * i.e. only one thread at a time will be able to enqueue/dequeue
+ * to/from the ring.
+ * With HTD_MAX >= ring.capacity - no limitation.
+ * By default HTD_MAX == ring.capacity / 8.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_ring_rts_generic.h>
+
+/**
+ * @internal Enqueue several objects on the RTS ring.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
+ * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
+ * @param free_space
+ * returns the amount of space after the enqueue operation has finished
+ * @return
+ * Actual number of objects enqueued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_enqueue(struct rte_ring *r, void * const *obj_table,
+ uint32_t n, enum rte_ring_queue_behavior behavior,
+ uint32_t *free_space)
+{
+ uint32_t free, head;
+
+ n = __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
+
+ if (n != 0) {
+ ENQUEUE_PTRS(r, &r[1], head, obj_table, n, void *);
+ __rte_ring_rts_update_tail(&r->rts_prod);
+ }
+
+ if (free_space != NULL)
+ *free_space = free - n;
+ return n;
+}
+
+/**
+ * @internal Dequeue several objects from the RTS ring.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to pull from the ring.
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
+ * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
+ * @param available
+ * returns the number of remaining ring entries after the dequeue has finished
+ * @return
+ * - Actual number of objects dequeued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_dequeue(struct rte_ring *r, void **obj_table,
+ uint32_t n, enum rte_ring_queue_behavior behavior,
+ uint32_t *available)
+{
+ uint32_t entries, head;
+
+ n = __rte_ring_rts_move_cons_head(r, n, behavior, &head, &entries);
+
+ if (n != 0) {
+ DEQUEUE_PTRS(r, &r[1], head, obj_table, n, void *);
+ __rte_ring_rts_update_tail(&r->rts_cons);
+ }
+
+ if (available != NULL)
+ *available = entries - n;
+ return n;
+}
+
+/**
+ * Enqueue several objects on the RTS ring (multi-producers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * The number of objects enqueued, either 0 or n
+ */
+__rte_experimental
+static __rte_always_inline unsigned int
+rte_ring_mp_rts_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
+ unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_rts_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
+ free_space);
+}
+
+/**
+ * Dequeue several objects from an RTS ring (multi-consumers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * The number of objects dequeued, either 0 or n
+ */
+__rte_experimental
+static __rte_always_inline unsigned int
+rte_ring_mc_rts_dequeue_bulk(struct rte_ring *r, void **obj_table,
+ unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_rts_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
+ available);
+}
+
+/**
+ * Return producer max Head-Tail-Distance (HTD).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * Producer HTD value, if producer is set in appropriate sync mode,
+ * or UINT32_MAX otherwise.
+ */
+__rte_experimental
+static inline uint32_t
+rte_ring_get_prod_htd_max(const struct rte_ring *r)
+{
+ if (r->prod.sync_type == RTE_RING_SYNC_MT_RTS)
+ return r->rts_prod.htd_max;
+ return UINT32_MAX;
+}
+
+/**
+ * Set producer max Head-Tail-Distance (HTD).
+ * Note that producer has to use appropriate sync mode (RTS).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param v
+ * new HTD value to setup.
+ * @return
+ * Zero on success, or negative error code otherwise.
+ */
+__rte_experimental
+static inline int
+rte_ring_set_prod_htd_max(struct rte_ring *r, uint32_t v)
+{
+ if (r->prod.sync_type != RTE_RING_SYNC_MT_RTS)
+ return -ENOTSUP;
+
+ r->rts_prod.htd_max = v;
+ return 0;
+}
+
+/**
+ * Return consumer max Head-Tail-Distance (HTD).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * Consumer HTD value, if consumer is set in appropriate sync mode,
+ * or UINT32_MAX otherwise.
+ */
+__rte_experimental
+static inline uint32_t
+rte_ring_get_cons_htd_max(const struct rte_ring *r)
+{
+ if (r->cons.sync_type == RTE_RING_SYNC_MT_RTS)
+ return r->rts_cons.htd_max;
+ return UINT32_MAX;
+}
+
+/**
+ * Set consumer max Head-Tail-Distance (HTD).
+ * Note that consumer has to use appropriate sync mode (RTS).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param v
+ * new HTD value to setup.
+ * @return
+ * Zero on success, or negative error code otherwise.
+ */
+__rte_experimental
+static inline int
+rte_ring_set_cons_htd_max(struct rte_ring *r, uint32_t v)
+{
+ if (r->cons.sync_type != RTE_RING_SYNC_MT_RTS)
+ return -ENOTSUP;
+
+ r->rts_cons.htd_max = v;
+ return 0;
+}
+
+/**
+ * Enqueue several objects on the RTS ring (multi-producers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * - n: Actual number of objects enqueued.
+ */
+__rte_experimental
+static __rte_always_inline unsigned
+rte_ring_mp_rts_enqueue_burst(struct rte_ring *r, void * const *obj_table,
+ unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_rts_enqueue(r, obj_table, n,
+ RTE_RING_QUEUE_VARIABLE, free_space);
+}
+
+/**
+ * Dequeue several objects from an RTS ring (multi-consumers safe).
+ * When the requested objects are more than the available objects,
+ * only dequeue the actual number of objects.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * - n: Actual number of objects dequeued, 0 if ring is empty
+ */
+__rte_experimental
+static __rte_always_inline unsigned
+rte_ring_mc_rts_dequeue_burst(struct rte_ring *r, void **obj_table,
+ unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_rts_dequeue(r, obj_table, n,
+ RTE_RING_QUEUE_VARIABLE, available);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_RING_RTS_H_ */
diff --git a/lib/librte_ring/rte_ring_rts_elem.h b/lib/librte_ring/rte_ring_rts_elem.h
new file mode 100644
index 000000000..71a331b23
--- /dev/null
+++ b/lib/librte_ring/rte_ring_rts_elem.h
@@ -0,0 +1,205 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#ifndef _RTE_RING_RTS_ELEM_H_
+#define _RTE_RING_RTS_ELEM_H_
+
+/**
+ * @file rte_ring_rts_elem.h
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * It is not recommended to include this file directly.
+ * Please include <rte_ring_elem.h> instead.
+ * Contains *ring_elem* functions for Relaxed Tail Sync (RTS) ring mode.
+ * for more details please refer to <rte_ring_rts.h>.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_ring_rts_generic.h>
+
+/**
+ * @internal Enqueue several objects on the RTS ring.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
+ * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
+ * @param free_space
+ * returns the amount of space after the enqueue operation has finished
+ * @return
+ * Actual number of objects enqueued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_enqueue_elem(struct rte_ring *r, void * const *obj_table,
+ uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+ uint32_t *free_space)
+{
+ uint32_t free, head;
+
+ n = __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
+
+ if (n != 0) {
+ __rte_ring_enqueue_elems(r, head, obj_table, esize, n);
+ __rte_ring_rts_update_tail(&r->rts_prod);
+ }
+
+ if (free_space != NULL)
+ *free_space = free - n;
+ return n;
+}
+
+/**
+ * @internal Dequeue several objects from the RTS ring.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to pull from the ring.
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
+ * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
+ * @param available
+ * returns the number of remaining ring entries after the dequeue has finished
+ * @return
+ * - Actual number of objects dequeued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_dequeue_elem(struct rte_ring *r, void **obj_table,
+ uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+ uint32_t *available)
+{
+ uint32_t entries, head;
+
+ n = __rte_ring_rts_move_cons_head(r, n, behavior, &head, &entries);
+
+ if (n != 0) {
+ __rte_ring_dequeue_elems(r, head, obj_table, esize, n);
+ __rte_ring_rts_update_tail(&r->rts_cons);
+ }
+
+ if (available != NULL)
+ *available = entries - n;
+ return n;
+}
+
+/**
+ * Enqueue several objects on the RTS ring (multi-producers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * The number of objects enqueued, either 0 or n
+ */
+__rte_experimental
+static __rte_always_inline unsigned int
+rte_ring_mp_rts_enqueue_bulk_elem(struct rte_ring *r, void * const *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_rts_enqueue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_FIXED, free_space);
+}
+
+/**
+ * Dequeue several objects from an RTS ring (multi-consumers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * The number of objects dequeued, either 0 or n
+ */
+__rte_experimental
+static __rte_always_inline unsigned int
+rte_ring_mc_rts_dequeue_bulk_elem(struct rte_ring *r, void **obj_table,
+ unsigned int esize, unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_rts_dequeue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_FIXED, available);
+}
+
+/**
+ * Enqueue several objects on the RTS ring (multi-producers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * - n: Actual number of objects enqueued.
+ */
+__rte_experimental
+static __rte_always_inline unsigned
+rte_ring_mp_rts_enqueue_burst_elem(struct rte_ring *r, void * const *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_rts_enqueue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_VARIABLE, free_space);
+}
+
+/**
+ * Dequeue several objects from an RTS ring (multi-consumers safe).
+ * When the requested objects are more than the available objects,
+ * only dequeue the actual number of objects.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * - n: Actual number of objects dequeued, 0 if ring is empty
+ */
+__rte_experimental
+static __rte_always_inline unsigned
+rte_ring_mc_rts_dequeue_burst_elem(struct rte_ring *r, void **obj_table,
+ unsigned int esize, unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_rts_dequeue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_VARIABLE, available);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_RING_RTS_ELEM_H_ */
diff --git a/lib/librte_ring/rte_ring_rts_generic.h b/lib/librte_ring/rte_ring_rts_generic.h
new file mode 100644
index 000000000..31a37924c
--- /dev/null
+++ b/lib/librte_ring/rte_ring_rts_generic.h
@@ -0,0 +1,210 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#ifndef _RTE_RING_RTS_GENERIC_H_
+#define _RTE_RING_RTS_GENERIC_H_
+
+/**
+ * @file rte_ring_rts_generic.h
+ * It is not recommended to include this file directly,
+ * include <rte_ring.h> instead.
+ * Contains internal helper functions for Relaxed Tail Sync (RTS) ring mode.
+ * For more information please refer to <rte_ring_rts.h>.
+ */
+
+/**
+ * @internal This function updates tail values.
+ */
+static __rte_always_inline void
+__rte_ring_rts_update_tail(struct rte_ring_rts_headtail *ht)
+{
+ union rte_ring_ht_poscnt h, ot, nt;
+
+ /*
+ * If there are other enqueues/dequeues in progress that
+ * might preceded us, then don't update tail with new value.
+ */
+
+ do {
+ ot.raw = ht->tail.raw;
+ rte_smp_rmb();
+
+ /* on 32-bit systems we have to do atomic read here */
+ h.raw = rte_atomic64_read((rte_atomic64_t *)
+ (uintptr_t)&ht->head.raw);
+
+ nt.raw = ot.raw;
+ if (++nt.val.cnt == h.val.cnt)
+ nt.val.pos = h.val.pos;
+
+ } while (rte_atomic64_cmpset(&ht->tail.raw, ot.raw, nt.raw) == 0);
+}
+
+/**
+ * @internal This function waits till head/tail distance wouldn't
+ * exceed pre-defined max value.
+ */
+static __rte_always_inline void
+__rte_ring_rts_head_wait(const struct rte_ring_rts_headtail *ht,
+ union rte_ring_ht_poscnt *h)
+{
+ uint32_t max;
+
+ max = ht->htd_max;
+ h->raw = ht->head.raw;
+ rte_smp_rmb();
+
+ while (h->val.pos - ht->tail.val.pos > max) {
+ rte_pause();
+ h->raw = ht->head.raw;
+ rte_smp_rmb();
+ }
+}
+
+/**
+ * @internal This function updates the producer head for enqueue.
+ *
+ * @param r
+ * A pointer to the ring structure
+ * @param is_sp
+ * Indicates whether multi-producer path is needed or not
+ * @param n
+ * The number of elements we will want to enqueue, i.e. how far should the
+ * head be moved
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
+ * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
+ * @param old_head
+ * Returns head value as it was before the move, i.e. where enqueue starts
+ * @param new_head
+ * Returns the current/new head value i.e. where enqueue finishes
+ * @param free_entries
+ * Returns the amount of free space in the ring BEFORE head was moved
+ * @return
+ * Actual number of objects enqueued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline uint32_t
+__rte_ring_rts_move_prod_head(struct rte_ring *r, uint32_t num,
+ enum rte_ring_queue_behavior behavior, uint32_t *old_head,
+ uint32_t *free_entries)
+{
+ uint32_t n;
+ union rte_ring_ht_poscnt nh, oh;
+
+ const uint32_t capacity = r->capacity;
+
+ do {
+ /* Reset n to the initial burst count */
+ n = num;
+
+ /* read prod head (may spin on prod tail) */
+ __rte_ring_rts_head_wait(&r->rts_prod, &oh);
+
+ /* add rmb barrier to avoid load/load reorder in weak
+ * memory model. It is noop on x86
+ */
+ rte_smp_rmb();
+
+ /*
+ * The subtraction is done between two unsigned 32bits value
+ * (the result is always modulo 32 bits even if we have
+ * *old_head > cons_tail). So 'free_entries' is always between 0
+ * and capacity (which is < size).
+ */
+ *free_entries = capacity + r->cons.tail - oh.val.pos;
+
+ /* check that we have enough room in ring */
+ if (unlikely(n > *free_entries))
+ n = (behavior == RTE_RING_QUEUE_FIXED) ?
+ 0 : *free_entries;
+
+ if (n == 0)
+ return 0;
+
+ nh.val.pos = oh.val.pos + n;
+ nh.val.cnt = oh.val.cnt + 1;
+
+ } while (rte_atomic64_cmpset(&r->rts_prod.head.raw,
+ oh.raw, nh.raw) == 0);
+
+ *old_head = oh.val.pos;
+ return n;
+}
+
+/**
+ * @internal This function updates the consumer head for dequeue
+ *
+ * @param r
+ * A pointer to the ring structure
+ * @param is_sc
+ * Indicates whether multi-consumer path is needed or not
+ * @param n
+ * The number of elements we will want to enqueue, i.e. how far should the
+ * head be moved
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
+ * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
+ * @param old_head
+ * Returns head value as it was before the move, i.e. where dequeue starts
+ * @param new_head
+ * Returns the current/new head value i.e. where dequeue finishes
+ * @param entries
+ * Returns the number of entries in the ring BEFORE head was moved
+ * @return
+ * - Actual number of objects dequeued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num,
+ enum rte_ring_queue_behavior behavior, uint32_t *old_head,
+ uint32_t *entries)
+{
+ uint32_t n;
+ union rte_ring_ht_poscnt nh, oh;
+
+ /* move cons.head atomically */
+ do {
+ /* Restore n as it may change every loop */
+ n = num;
+
+ /* read cons head (may spin on cons tail) */
+ __rte_ring_rts_head_wait(&r->rts_cons, &oh);
+
+
+ /* add rmb barrier to avoid load/load reorder in weak
+ * memory model. It is noop on x86
+ */
+ rte_smp_rmb();
+
+ /* The subtraction is done between two unsigned 32bits value
+ * (the result is always modulo 32 bits even if we have
+ * cons_head > prod_tail). So 'entries' is always between 0
+ * and size(ring)-1.
+ */
+ *entries = r->prod.tail - oh.val.pos;
+
+ /* Set the actual entries for dequeue */
+ if (n > *entries)
+ n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 : *entries;
+
+ if (unlikely(n == 0))
+ return 0;
+
+ nh.val.pos = oh.val.pos + n;
+ nh.val.cnt = oh.val.cnt + 1;
+
+ } while (rte_atomic64_cmpset(&r->rts_cons.head.raw,
+ oh.raw, nh.raw) == 0);
+
+ *old_head = oh.val.pos;
+ return n;
+}
+
+#endif /* _RTE_RING_RTS_GENERIC_H_ */
--
2.17.1
^ permalink raw reply [relevance 1%]
* Re: [dpdk-dev] [RFC] ethdev: use special speed for virtual Ethernetdevices
@ 2020-04-03 9:45 3% ` Morten Brørup
2020-04-03 11:01 0% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Morten Brørup @ 2020-04-03 9:45 UTC (permalink / raw)
To: Ivan Dyukov, Thomas Monjalon, Ferruh Yigit, Andrew Rybchenko
Cc: dev, Matan Azrad, Benoit Ganne (bganne),
maxime.coquelin, Vladimir Kuramshin, amorenoz, zhihong.wang,
xiaolong.ye, Stephen Hemminger
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ivan Dyukov
> Sent: Friday, April 3, 2020 10:06 AM
>
> 02.04.2020 23:58, Thomas Monjalon пишет:
> > 02/04/2020 22:41, Ivan Dyukov:
> >> 02.04.2020 16:50, Morten Brørup пишет:
> >>>>>> Yes, if speed is unknown, it should be reported as 0.
> >>> Could the DPDK vNIC PMDs be updated accordingly? At least the
> virtio driver...
> >> Current version of dpdk code on master always returns 10G speed for
> >> virtio device and many application rely on it. e.g. pktgen. If we'll
> >> change it, we break the apps.
> > I am OK with breaking such strange assumption.
> > I can understand the need for specifying the underlying hardware
> speed
> > through virtio driver. But hardcoded 10G... no!
> >
> >
> >
> OK. I'll redefine it to 0xffffffff, like in kernel virtio.
>
Thomas, you were opposed to using 1 as the special value for "unknown", as it is likely interpreted as 1 Mbps instead, and I agree with your reasoning on that.
Since using an extremely large value is as close to infinity we can get, Ivan's suggested value makes sense to me instead:
+#define ETH_SPEED_NUM_UNKNOWN 0xffffffff /**< Unknown */
In theory, it also addresses Stephen's concern about breaking the ABI for existing applications that look at speed. They will get a non-zero speed, which is not a randomly chosen fake 10G speed; so I consider it an improvement.
Although in reality, such applications may still break if they are unable to handle the extreme speed of 4.3 Pbps.
I considered "Unlimited" instead of "Unknown", but Unlimited is not really correct, so I settled with Unknown.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [RFC] ethdev: use special speed for virtual Ethernetdevices
2020-04-03 9:45 3% ` Morten Brørup
@ 2020-04-03 11:01 0% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-04-03 11:01 UTC (permalink / raw)
To: Ivan Dyukov, Morten Brørup
Cc: Ferruh Yigit, Andrew Rybchenko, dev, Matan Azrad,
Benoit Ganne (bganne),
maxime.coquelin, Vladimir Kuramshin, amorenoz, zhihong.wang,
xiaolong.ye, Stephen Hemminger
03/04/2020 11:45, Morten Brørup:
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ivan Dyukov
> > Sent: Friday, April 3, 2020 10:06 AM
> >
> > 02.04.2020 23:58, Thomas Monjalon пишет:
> > > 02/04/2020 22:41, Ivan Dyukov:
> > >> 02.04.2020 16:50, Morten Brørup пишет:
> > >>>>>> Yes, if speed is unknown, it should be reported as 0.
> > >>> Could the DPDK vNIC PMDs be updated accordingly? At least the
> > virtio driver...
> > >> Current version of dpdk code on master always returns 10G speed for
> > >> virtio device and many application rely on it. e.g. pktgen. If we'll
> > >> change it, we break the apps.
> > > I am OK with breaking such strange assumption.
> > > I can understand the need for specifying the underlying hardware
> > speed
> > > through virtio driver. But hardcoded 10G... no!
> > >
> > >
> > >
> > OK. I'll redefine it to 0xffffffff, like in kernel virtio.
> >
>
> Thomas, you were opposed to using 1 as the special value for "unknown", as it is likely interpreted as 1 Mbps instead, and I agree with your reasoning on that.
>
> Since using an extremely large value is as close to infinity we can get, Ivan's suggested value makes sense to me instead:
>
> +#define ETH_SPEED_NUM_UNKNOWN 0xffffffff /**< Unknown */
>
> In theory, it also addresses Stephen's concern about breaking the ABI for existing applications that look at speed. They will get a non-zero speed, which is not a randomly chosen fake 10G speed; so I consider it an improvement.
> Although in reality, such applications may still break if they are unable to handle the extreme speed of 4.3 Pbps.
>
> I considered "Unlimited" instead of "Unknown", but Unlimited is not really correct, so I settled with Unknown.
Yes it makes sense.
The only drawback is that the information "speed is accurate" will
be checked against two constants:
ETH_SPEED_NUM_NONE or ETH_SPEED_NUM_UNKNOWN
It can be mitigated with a function though.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v3 07/12] service: remove rte prefix from static functions
@ 2020-04-03 11:57 3% ` Van Haaren, Harry
2020-04-08 10:14 0% ` Phil Yang
0 siblings, 1 reply; 200+ results
From: Van Haaren, Harry @ 2020-04-03 11:57 UTC (permalink / raw)
To: Phil Yang, thomas, Ananyev, Konstantin, stephen, maxime.coquelin, dev
Cc: david.marchand, jerinj, hemant.agrawal, Honnappa.Nagarahalli,
gavin.hu, ruifeng.wang, joyce.kong, nd, stable
> From: Phil Yang <phil.yang@arm.com>
> Sent: Tuesday, March 17, 2020 1:18 AM
> To: thomas@monjalon.net; Van Haaren, Harry <harry.van.haaren@intel.com>;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> stephen@networkplumber.org; maxime.coquelin@redhat.com; dev@dpdk.org
> Cc: david.marchand@redhat.com; jerinj@marvell.com; hemant.agrawal@nxp.com;
> Honnappa.Nagarahalli@arm.com; gavin.hu@arm.com; ruifeng.wang@arm.com;
> joyce.kong@arm.com; nd@arm.com; stable@dpdk.org
> Subject: [PATCH v3 07/12] service: remove rte prefix from static functions
>
> Fixes: 3cf5eb1546ed ("service: fix and refactor atomic service accesses")
> Fixes: 21698354c832 ("service: introduce service cores concept")
> Cc: stable@dpdk.org
>
> Signed-off-by: Phil Yang <phil.yang@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
This patchset needs a rebase since the EAL file movement got merged,
however I'll review here so we can include some Acks etc and make
progress.
Is this really a "Fix"? The internal function names were not exported
in the .map file, so are not part of public ABI. This is an internal
naming improvement (thanks for doing cleanup), but I don't think the
Fixes: tags make sense?
Also I'm not sure if we want to port this patch back to stable? Changing (internal) function names seems like unnecessary churn, and hence risk to a stable release, without any benefit?
---
<snip patch diff>
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v4 00/33] DPDK Trace support
2020-03-29 14:43 1% ` [dpdk-dev] [PATCH v3 00/33] " jerinj
2020-04-01 8:18 3% ` David Marchand
@ 2020-04-03 15:36 1% ` jerinj
2020-04-13 15:00 1% ` [dpdk-dev] [PATCH v5 " jerinj
1 sibling, 1 reply; 200+ results
From: jerinj @ 2020-04-03 15:36 UTC (permalink / raw)
Cc: dev, thomas, bruce.richardson, david.marchand, mattias.ronnblom,
skori, Jerin Jacob
From: Jerin Jacob <jerinj@marvell.com>
v4:
~~
This patch depends on http://patches.dpdk.org/patch/67758/
Depends-on:series-9191
1) Rebased to master.
2) Adapted to latest EAL directory structure change.
3) Fix possible build issue with out of tree application wherein
it does not define -DALLOW_EXPERIMENTAL_API. Fixed by making
fast path trace functions as NOP as it was getting included
in the inline functions of ethdev,mempool, cryptodev, etc(David)
4) Removed DALLOW_EXPERIMENTAL_API definition from individual driver files.
Now it set as global using http://patches.dpdk.org/patch/67758/ patch (David)
5) Added new meson option(-Denable_trace_dp=true)for enabling the datapath trace point (David, Bruce)
6) Changed the authorship and Rewrote the programmer's guide based on the below feedback(Thomas)
http://patches.dpdk.org/patch/67352/
v3:
~~
1) Fix the following build issues reported by CI
http://mails.dpdk.org/archives/test-report/2020-March/122060.html
a) clang + i686 meson build issue(Fixed in the the patch meson: add libatomic as a global dependency for i686 clang)
b) fixed build issue with FreeBSD with top of tree change.
c) fixed missing experimental API for iavf and ice with meson build in avx512 files.
v2:
~~
Addressed the following review comments from Mattias Rönnblom:
1) Changed
from:
typedef uint64_t* rte_trace_t;
to
typedef uint64_t rte_trace_t;
Initially thought to make the handle as
struct rte_trace {
uint64_t val;
}
but changed to uint64_t for the following reasons
a) It is opaque to the application and it will fix the compile-time type
check as well.
b) The handle has an index that will point to an internal slow-path
structure so no ABI change required in the future.
c) RTE_TRACE_POINT_DEFINE need to expose trace object. So it is better
to keep as uint64_t and avoid one more indirection for no use.
2)
Changed:
from:
enum rte_trace_mode_e {
to:
enum rte_trace_mode {
3) removed [out] "found" param from rte_trace_pattern() and
rte_trace_regexp()
4) Changed rte_trace_from_name to rte_trace_by_name
5) rte_trace_is_dp_enabled() return bool now
6) in __rte_trace_point_register() the argument fn change to register_fn
7) removed !! from rte_trace_is_enabled()
8) Remove uninitialized "rc warning" from rte_trace_pattern() and
rte_trace_regexp()
9) fixup bool return type for trace_entry_compare()
10) fixup calloc casting in trace_mkdir()
11) check fclose() return in trace_meta_save() and trace_mem_save()
12) rte_trace_ctf_* macro cleanup
13) added release notes
14) fix build issues reported by CI
http://mails.dpdk.org/archives/test-report/2020-March/121235.html
This patch set contains
~~~~~~~~~~~~~~~~~~~~~~~~
# The native implementation of common trace format(CTF)[1] based tracer
# Public API to create the trace points.
# Add tracepoints to eal, ethdev, mempool, eventdev and cryptodev
library for tracing support
# A unit test case
# Performance test case to measure the trace overhead. (See eal/trace:
# add trace performance test cases, patch)
# Programmers guide for Trace support(See doc: add trace library guide,
# patch)
# Tested OS:
~~~~~~~~~~~
- Linux
- FreeBSD
# Tested open source CTF trace viewers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Babeltrace
- Tracecompass
# Trace overhead comparison with LTTng
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
trace overhead data on x86:[2]
# 236 cycles with LTTng(>100ns)
# 18 cycles(7ns) with Native DPDK CTF emitter.(See eal/trace: add trace
# performance test cases patch)
trace overhead data on arm64:
# 312 cycles to 1100 cycles with LTTng based on the class of arm64
# CPU.
# 11 cycles to 13 cycles with Native DPDK CTF emitter based on the
class of arm64 CPU.
18 cycles(on x86) vs 11 cycles(on arm64) is due to rdtsc() overhead in
x86. It seems rdtsc takes around 15cycles in x86.
More details:
~~~~~~~~~~~~~
# The Native DPDK CTF trace support does not have any dependency on
third-party library.
The generated output file is compatible with LTTng as both are using
CTF trace format.
The performance gain comes from:
1) exploit dpdk worker thread usage model to avoid atomics and use per
core variables
2) use hugepage,
3) avoid a lot function pointers in fast-path etc
4) avoid unaligned store for arm64 etc
Features:
~~~~~~~~~
- APIs and Features are similar to rte_log dynamic framework
API(expect log prints on stdout vs it dumps on trace file)
- No specific limit on the events. A string-based event like rte_log
for pattern matching
- Dynamic enable/disable support.
- Instructmention overhead is ~1 cycle. i.e cost of adding the code
wth out using trace feature.
- Timestamp support for all the events using DPDK rte_rtdsc
- No dependency on another library. Clean room native implementation of
CTF.
Functional test case:
a) echo "trace_autotest" | sudo ./build/app/test/dpdk-test -c 0x3
--trace-level=8
The above command emits the following trace events
<code>
uint8_t i;
rte_trace_lib_eal_generic_void();
rte_trace_lib_eal_generic_u64(0x10000000000000);
rte_trace_lib_eal_generic_u32(0x10000000);
rte_trace_lib_eal_generic_u16(0xffee);
rte_trace_lib_eal_generic_u8(0xc);
rte_trace_lib_eal_generic_i64(-1234);
rte_trace_lib_eal_generic_i32(-1234567);
rte_trace_lib_eal_generic_i16(12);
rte_trace_lib_eal_generic_i8(-3);
rte_trace_lib_eal_generic_string("my string");
rte_trace_lib_eal_generic_function(__func__);
for (i = 0; i < 128; i++)
rte_trace_lib_eal_generic_u8(i);
</code>
Install babeltrace package in Linux and point the generated trace file
to babel trace. By default trace file created under
<user>/dpdk-traces/time_stamp/
example:
# babeltrace /root/dpdk-traces/rte-2020-02-15-PM-02-56-51 | more
[13:27:36.138468807] (+?.?????????) lib.eal.generic.void: { cpu_id =
0, name = "dpdk-test" }, { }
[13:27:36.138468851] (+0.000000044) lib.eal.generic.u64: { cpu_id = 0,
name = "dpdk-test" }, { in = 4503599627370496 }
[13:27:36.138468860] (+0.000000009) lib.eal.generic.u32: { cpu_id = 0,
name = "dpdk-test" }, { in = 268435456 }
[13:27:36.138468934] (+0.000000074) lib.eal.generic.u16: { cpu_id = 0,
name = "dpdk-test" }, { in = 65518 }
[13:27:36.138468949] (+0.000000015) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 12 }
[13:27:36.138468956] (+0.000000007) lib.eal.generic.i64: { cpu_id = 0,
name = "dpdk-test" }, { in = -1234 }
[13:27:36.138468963] (+0.000000007) lib.eal.generic.i32: { cpu_id = 0,
name = "dpdk-test" }, { in = -1234567 }
[13:27:36.138469024] (+0.000000061) lib.eal.generic.i16: { cpu_id = 0,
name = "dpdk-test" }, { in = 12 }
[13:27:36.138469044] (+0.000000020) lib.eal.generic.i8: { cpu_id = 0,
name = "dpdk-test" }, { in = -3 }
[13:27:36.138469051] (+0.000000007) lib.eal.generic.string: { cpu_id =
0, name = "dpdk-test" }, { str = "my string" }
[13:27:36.138469203] (+0.000000152) lib.eal.generic.func: { cpu_id =
0, name = "dpdk-test" }, { func = "test_trace_points" }
[13:27:36.138469239] (+0.000000036) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 0 }
[13:27:36.138469246] (+0.000000007) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 1 }
[13:27:36.138469252] (+0.000000006) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 2 }
[13:27:36.138469262] (+0.000000010) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 3 }
[13:27:36.138469269] (+0.000000007) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 4 }
[13:27:36.138469276] (+0.000000007) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 5 }
# There is a GUI based trace viewer available in Windows, Linux and
# Mac.
It is called as tracecompass.(https://www.eclipse.org/tracecompass/)
The example screenshot and Histogram of above DPDK trace using
Tracecompass.
https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG
File walk through:
~~~~~~~~~~~~~~~~~~
lib/librte_eal/common/include/rte_trace.h - Public API for Trace
provider and Trace control
lib/librte_eal/common/eal_common_trace.c - main trace implementation
lib/librte_eal/common/eal_common_trace_ctf.c - CTF metadata spec
implementation
lib/librte_eal/common/eal_common_trace_utils.c - command line utils
and filesystem operations.
lib/librte_eal/common/eal_common_trace_points.c - trace points for EAL
library
lib/librte_eal/common/include/rte_trace_eal.h - EAL tracepoint public
API.
lib/librte_eal/common/eal_trace.h - Private trace header file.
[1] https://diamon.org/ctf/
[2] The above test is ported to LTTng for finding the LTTng trace
overhead. It available at
https://github.com/jerinjacobk/lttng-overhead
https://github.com/jerinjacobk/lttng-overhead/blob/master/README
Jerin Jacob (22):
eal: introduce API for getting thread name
eal/trace: define the public API for trace support
eal/trace: implement trace register API
eal/trace: implement trace operation APIs
eal/trace: add internal trace init and fini interface
eal/trace: get bootup timestamp for trace
eal/trace: create CTF TDSL metadata in memory
eal/trace: implement trace memory allocation
eal/trace: implement debug dump function
eal/trace: implement trace save
eal/trace: implement registration payload
eal/trace: implement provider payload
eal/trace: hook internal trace APIs to Linux
eal/trace: hook internal trace APIs to FreeBSD
eal/trace: add generic tracepoints
eal/trace: add alarm tracepoints
eal/trace: add memory tracepoints
eal/trace: add memzone tracepoints
eal/trace: add thread tracepoints
eal/trace: add interrupt tracepoints
eal/trace: add trace performance test cases
doc: add trace library guide
Pavan Nikhilesh (1):
meson: add libatomic as a global dependency for i686 clang
Sunil Kumar Kori (10):
eal/trace: handle CTF keyword collision
eal/trace: add trace level configuration parameter
eal/trace: add trace dir configuration parameter
eal/trace: add trace bufsize configuration parameter
eal/trace: add trace mode configuration parameter
eal/trace: add unit test cases
ethdev: add tracepoints
eventdev: add tracepoints
cryptodev: add tracepoints
mempool: add tracepoints
MAINTAINERS | 8 +
app/test/Makefile | 4 +-
app/test/meson.build | 3 +
app/test/test_trace.c | 618 ++++++++++++++++++
app/test/test_trace.h | 52 ++
app/test/test_trace_perf.c | 179 +++++
app/test/test_trace_register.c | 46 ++
config/common_base | 1 +
config/meson.build | 9 +
doc/api/doxy-api-index.md | 3 +-
doc/guides/linux_gsg/eal_args.include.rst | 55 ++
doc/guides/prog_guide/build-sdk-meson.rst | 5 +
doc/guides/prog_guide/index.rst | 1 +
doc/guides/prog_guide/trace_lib.rst | 339 ++++++++++
doc/guides/rel_notes/release_20_05.rst | 9 +
drivers/event/octeontx/meson.build | 5 -
drivers/event/octeontx2/meson.build | 5 -
drivers/event/opdl/meson.build | 5 -
examples/cmdline/Makefile | 1 +
examples/cmdline/meson.build | 1 +
examples/distributor/Makefile | 1 +
examples/distributor/meson.build | 1 +
examples/ethtool/ethtool-app/Makefile | 1 +
examples/eventdev_pipeline/meson.build | 1 +
examples/flow_filtering/Makefile | 1 +
examples/flow_filtering/meson.build | 1 +
examples/helloworld/Makefile | 1 +
examples/helloworld/meson.build | 1 +
examples/ioat/Makefile | 1 +
examples/ioat/meson.build | 1 +
examples/ip_fragmentation/Makefile | 2 +
examples/ip_fragmentation/meson.build | 1 +
examples/ip_reassembly/Makefile | 1 +
examples/ip_reassembly/meson.build | 1 +
examples/ipv4_multicast/Makefile | 1 +
examples/ipv4_multicast/meson.build | 1 +
examples/l2fwd-cat/Makefile | 1 +
examples/l2fwd-cat/meson.build | 1 +
examples/l2fwd-event/Makefile | 1 +
examples/l2fwd-event/meson.build | 6 +-
examples/l2fwd-jobstats/Makefile | 1 +
examples/l2fwd-jobstats/meson.build | 1 +
examples/l2fwd-keepalive/Makefile | 1 +
examples/l2fwd-keepalive/ka-agent/Makefile | 1 +
examples/l2fwd-keepalive/meson.build | 1 +
examples/l3fwd-acl/Makefile | 1 +
examples/l3fwd-acl/meson.build | 1 +
examples/l3fwd/Makefile | 1 +
examples/l3fwd/meson.build | 1 +
examples/link_status_interrupt/Makefile | 1 +
examples/link_status_interrupt/meson.build | 1 +
.../client_server_mp/mp_client/Makefile | 1 +
.../client_server_mp/mp_client/meson.build | 1 +
.../client_server_mp/mp_server/meson.build | 1 +
examples/multi_process/hotplug_mp/Makefile | 1 +
examples/multi_process/hotplug_mp/meson.build | 1 +
examples/multi_process/simple_mp/Makefile | 1 +
examples/multi_process/simple_mp/meson.build | 1 +
examples/multi_process/symmetric_mp/Makefile | 1 +
.../multi_process/symmetric_mp/meson.build | 1 +
examples/ntb/Makefile | 1 +
examples/ntb/meson.build | 1 +
examples/packet_ordering/Makefile | 1 +
examples/packet_ordering/meson.build | 1 +
.../performance-thread/l3fwd-thread/Makefile | 1 +
.../l3fwd-thread/meson.build | 1 +
.../performance-thread/pthread_shim/Makefile | 1 +
.../pthread_shim/meson.build | 1 +
examples/ptpclient/Makefile | 1 +
examples/ptpclient/meson.build | 1 +
examples/qos_meter/Makefile | 1 +
examples/qos_meter/meson.build | 1 +
examples/qos_sched/Makefile | 1 +
examples/qos_sched/meson.build | 1 +
examples/server_node_efd/node/Makefile | 1 +
examples/server_node_efd/node/meson.build | 1 +
examples/server_node_efd/server/Makefile | 1 +
examples/server_node_efd/server/meson.build | 1 +
examples/service_cores/Makefile | 1 +
examples/service_cores/meson.build | 1 +
examples/skeleton/Makefile | 1 +
examples/skeleton/meson.build | 1 +
examples/timer/Makefile | 1 +
examples/timer/meson.build | 1 +
examples/vm_power_manager/Makefile | 1 +
examples/vm_power_manager/meson.build | 1 +
examples/vmdq/Makefile | 1 +
examples/vmdq/meson.build | 1 +
examples/vmdq_dcb/Makefile | 1 +
examples/vmdq_dcb/meson.build | 1 +
lib/librte_cryptodev/Makefile | 4 +-
lib/librte_cryptodev/cryptodev_trace_points.c | 70 ++
lib/librte_cryptodev/meson.build | 6 +-
lib/librte_cryptodev/rte_cryptodev.c | 18 +
lib/librte_cryptodev/rte_cryptodev.h | 6 +
.../rte_cryptodev_version.map | 18 +
lib/librte_cryptodev/rte_trace_cryptodev.h | 133 ++++
lib/librte_cryptodev/rte_trace_cryptodev_fp.h | 34 +
lib/librte_distributor/meson.build | 5 -
lib/librte_eal/common/eal_common_log.c | 9 +-
lib/librte_eal/common/eal_common_memzone.c | 9 +
lib/librte_eal/common/eal_common_options.c | 68 +-
lib/librte_eal/common/eal_common_thread.c | 3 +-
lib/librte_eal/common/eal_common_trace.c | 610 +++++++++++++++++
lib/librte_eal/common/eal_common_trace_ctf.c | 488 ++++++++++++++
.../common/eal_common_trace_points.c | 115 ++++
.../common/eal_common_trace_utils.c | 523 +++++++++++++++
lib/librte_eal/common/eal_options.h | 8 +
lib/librte_eal/common/eal_private.h | 11 +
lib/librte_eal/common/eal_trace.h | 122 ++++
lib/librte_eal/common/meson.build | 4 +
lib/librte_eal/common/rte_malloc.c | 60 +-
lib/librte_eal/freebsd/Makefile | 4 +
lib/librte_eal/freebsd/eal.c | 10 +
lib/librte_eal/freebsd/eal_alarm.c | 3 +
lib/librte_eal/freebsd/eal_interrupts.c | 54 +-
lib/librte_eal/freebsd/eal_thread.c | 21 +-
lib/librte_eal/include/meson.build | 4 +
lib/librte_eal/include/rte_lcore.h | 17 +
lib/librte_eal/include/rte_trace.h | 584 +++++++++++++++++
lib/librte_eal/include/rte_trace_eal.h | 247 +++++++
lib/librte_eal/include/rte_trace_provider.h | 160 +++++
lib/librte_eal/include/rte_trace_register.h | 53 ++
lib/librte_eal/linux/Makefile | 4 +
lib/librte_eal/linux/eal.c | 9 +
lib/librte_eal/linux/eal_alarm.c | 4 +
lib/librte_eal/linux/eal_interrupts.c | 84 ++-
lib/librte_eal/linux/eal_thread.c | 27 +-
lib/librte_eal/rte_eal_version.map | 59 ++
lib/librte_ethdev/Makefile | 3 +
lib/librte_ethdev/ethdev_trace_points.c | 43 ++
lib/librte_ethdev/meson.build | 5 +-
lib/librte_ethdev/rte_ethdev.c | 12 +
lib/librte_ethdev/rte_ethdev.h | 5 +
lib/librte_ethdev/rte_ethdev_version.map | 10 +
lib/librte_ethdev/rte_trace_ethdev.h | 90 +++
lib/librte_ethdev/rte_trace_ethdev_fp.h | 40 ++
lib/librte_eventdev/Makefile | 3 +
lib/librte_eventdev/eventdev_trace_points.c | 173 +++++
lib/librte_eventdev/meson.build | 3 +
.../rte_event_crypto_adapter.c | 10 +
.../rte_event_eth_rx_adapter.c | 11 +
.../rte_event_eth_tx_adapter.c | 13 +-
.../rte_event_eth_tx_adapter.h | 2 +
lib/librte_eventdev/rte_event_timer_adapter.c | 8 +-
lib/librte_eventdev/rte_event_timer_adapter.h | 8 +
lib/librte_eventdev/rte_eventdev.c | 9 +
lib/librte_eventdev/rte_eventdev.h | 5 +-
lib/librte_eventdev/rte_eventdev_version.map | 42 ++
lib/librte_eventdev/rte_trace_eventdev.h | 278 ++++++++
lib/librte_eventdev/rte_trace_eventdev_fp.h | 75 +++
lib/librte_mempool/Makefile | 3 +
lib/librte_mempool/mempool_trace_points.c | 108 +++
lib/librte_mempool/meson.build | 5 +-
lib/librte_mempool/rte_mempool.c | 16 +
lib/librte_mempool/rte_mempool.h | 13 +
lib/librte_mempool/rte_mempool_ops.c | 7 +
lib/librte_mempool/rte_mempool_version.map | 26 +
lib/librte_mempool/rte_trace_mempool.h | 148 +++++
lib/librte_mempool/rte_trace_mempool_fp.h | 102 +++
lib/librte_rcu/meson.build | 5 -
meson_options.txt | 2 +
162 files changed, 6266 insertions(+), 110 deletions(-)
create mode 100644 app/test/test_trace.c
create mode 100644 app/test/test_trace.h
create mode 100644 app/test/test_trace_perf.c
create mode 100644 app/test/test_trace_register.c
create mode 100644 doc/guides/prog_guide/trace_lib.rst
create mode 100644 lib/librte_cryptodev/cryptodev_trace_points.c
create mode 100644 lib/librte_cryptodev/rte_trace_cryptodev.h
create mode 100644 lib/librte_cryptodev/rte_trace_cryptodev_fp.h
create mode 100644 lib/librte_eal/common/eal_common_trace.c
create mode 100644 lib/librte_eal/common/eal_common_trace_ctf.c
create mode 100644 lib/librte_eal/common/eal_common_trace_points.c
create mode 100644 lib/librte_eal/common/eal_common_trace_utils.c
create mode 100644 lib/librte_eal/common/eal_trace.h
create mode 100644 lib/librte_eal/include/rte_trace.h
create mode 100644 lib/librte_eal/include/rte_trace_eal.h
create mode 100644 lib/librte_eal/include/rte_trace_provider.h
create mode 100644 lib/librte_eal/include/rte_trace_register.h
create mode 100644 lib/librte_ethdev/ethdev_trace_points.c
create mode 100644 lib/librte_ethdev/rte_trace_ethdev.h
create mode 100644 lib/librte_ethdev/rte_trace_ethdev_fp.h
create mode 100644 lib/librte_eventdev/eventdev_trace_points.c
create mode 100644 lib/librte_eventdev/rte_trace_eventdev.h
create mode 100644 lib/librte_eventdev/rte_trace_eventdev_fp.h
create mode 100644 lib/librte_mempool/mempool_trace_points.c
create mode 100644 lib/librte_mempool/rte_trace_mempool.h
create mode 100644 lib/librte_mempool/rte_trace_mempool_fp.h
--
2.25.1
^ permalink raw reply [relevance 1%]
* [dpdk-dev] [PATCH v3 0/9] New sync modes for ring
2020-04-02 22:09 3% ` [dpdk-dev] [PATCH v2 0/9] " Konstantin Ananyev
2020-04-02 22:09 1% ` [dpdk-dev] [PATCH v2 3/9] ring: introduce RTS ring mode Konstantin Ananyev
@ 2020-04-03 17:42 3% ` Konstantin Ananyev
` (2 more replies)
1 sibling, 3 replies; 200+ results
From: Konstantin Ananyev @ 2020-04-03 17:42 UTC (permalink / raw)
To: dev; +Cc: honnappa.nagarahalli, david.marchand, jielong.zjl, Konstantin Ananyev
V2 - V3
1. Few more compilation fixes (for gcc 4.8.X)
2. Extra update devtools/libabigail.abignore (workaround)
V1 - v2 changes:
1. Fix compilation issues
2. Add C11 atomics support
3. Updates devtools/libabigail.abignore (workaround)
RFC - V1 changes:
1. remove ABI brekage (at least I hope I did)
2. Add support for ring_elem
3. Rework peek related API a bit
4. Rework test to make it less verbose and unite all test-cases
in one command
5. Add new test-case for MT peek API
TODO list:
1. Update docs
These days more and more customers use(/try to use) DPDK based apps within
overcommitted systems (multiple acttive threads over same pysical cores):
VM, container deployments, etc.
One quite common problem they hit:
Lock-Holder-Preemption/Lock-Waiter-Preemption with rte_ring.
LHP is quite a common problem for spin-based sync primitives
(spin-locks, etc.) on overcommitted systems.
The situation gets much worse when some sort of
fair-locking technique is used (ticket-lock, etc.).
As now not only lock-owner but also lock-waiters scheduling
order matters a lot (LWP).
These two problems are well-known for kernel within VMs:
http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
The problem with rte_ring is that while head accusion is sort of
un-fair locking, waiting on tail is very similar to ticket lock schema -
tail has to be updated in particular order.
That makes current rte_ring implementation to perform
really pure on some overcommited scenarios.
It is probably not possible to completely resolve LHP problem in
userspace only (without some kernel communication/intervention).
But removing fairness at tail update helps to avoid LWP and
can mitigate the situation significantly.
This patch proposes two new optional ring synchronization modes:
1) Head/Tail Sync (HTS) mode
In that mode enqueue/dequeue operation is fully serialized:
only one thread at a time is allowed to perform given op.
As another enhancement provide ability to split enqueue/dequeue
operation into two phases:
- enqueue/dequeue start
- enqueue/dequeue finish
That allows user to inspect objects in the ring without removing
them from it (aka MT safe peek).
2) Relaxed Tail Sync (RTS)
The main difference from original MP/MC algorithm is that
tail value is increased not by every thread that finished enqueue/dequeue,
but only by the last one.
That allows threads to avoid spinning on ring tail value,
leaving actual tail value change to the last thread in the update queue.
Note that these new sync modes are optional.
For current rte_ring users nothing should change
(both in terms of API/ABI and performance).
Existing sync modes MP/MC,SP/SC kept untouched, set up in the same
way (via flags and _init_), and MP/MC remains as default one.
The only thing that changed:
Format of prod/cons now could differ depending on mode selected at _init_.
So user has to stick with one sync model through whole ring lifetime.
In other words, user can't create a ring for let say SP mode and then
in the middle of data-path change his mind and start using MP_RTS mode.
For existing modes (SP/MP, SC/MC) format remains the same and
user can still use them interchangeably, though of course it is an
error prone practice.
Test results on IA (see below) show significant improvements
for average enqueue/dequeue op times on overcommitted systems.
For 'classic' DPDK deployments (one thread per core) original MP/MC
algorithm still shows best numbers, though for 64-bit target
RTS numbers are not that far away.
Numbers were produced by new UT test-case: ring_stress_autotest, i.e.:
echo ring_stress_autotest | ./dpdk-test -n 4 --lcores='...'
X86_64 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
DEQ+ENQ average cycles/obj
MP/MC HTS RTS
1thread@1core(--lcores=6-7) 8.00 8.15 8.99
2thread@2core(--lcores=6-8) 19.14 19.61 20.35
4thread@4core(--lcores=6-10) 29.43 29.79 31.82
8thread@8core(--lcores=6-14) 110.59 192.81 119.50
16thread@16core(--lcores=6-22) 461.03 813.12 495.59
32thread/@32core(--lcores='6-22,55-70') 982.90 1972.38 1160.51
2thread@1core(--lcores='6,(10-11)@7' 20140.50 23.58 25.14
4thread@2core(--lcores='6,(10-11)@7,(20-21)@8' 153680.60 76.88 80.05
8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 280314.32 294.72 318.79
16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 643176.59 1144.02 1175.14
32thread@2core(--lcores='6,(10-25)@7,(30-45)@8' 4264238.80 4627.48 4892.68
8thread@2core(--lcores='6,(10-17)@(7,8))' 321085.98 298.59 307.47
16thread@4core(--lcores='6,(20-35)@(7-10))' 1900705.61 575.35 678.29
32thread@4core(--lcores='6,(20-51)@(7-10))' 5510445.85 2164.36 2714.12
i686 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
DEQ+ENQ average cycles/obj
MP/MC HTS RTS
1thread@1core(--lcores=6-7) 7.85 12.13 11.31
2thread@2core(--lcores=6-8) 17.89 24.52 21.86
8thread@8core(--lcores=6-14) 32.58 354.20 54.58
32thread/@32core(--lcores='6-22,55-70') 813.77 6072.41 2169.91
2thread@1core(--lcores='6,(10-11)@7' 16095.00 36.06 34.74
8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 1140354.54 346.61 361.57
16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 1920417.86 1314.90 1416.65
8thread@2core(--lcores='6,(10-17)@(7,8))' 594358.61 332.70 357.74
32thread@4core(--lcores='6,(20-51)@(7-10))' 5319896.86 2836.44 3028.87
Konstantin Ananyev (9):
test/ring: add contention stress test
ring: prepare ring to allow new sync schemes
ring: introduce RTS ring mode
test/ring: add contention stress test for RTS ring
ring: introduce HTS ring mode
test/ring: add contention stress test for HTS ring
ring: introduce peek style API
test/ring: add stress test for MT peek API
ring: add C11 memory model for new sync modes
app/test/Makefile | 5 +
app/test/meson.build | 5 +
app/test/test_pdump.c | 6 +-
app/test/test_ring_hts_stress.c | 32 ++
app/test/test_ring_mpmc_stress.c | 31 ++
app/test/test_ring_peek_stress.c | 43 +++
app/test/test_ring_rts_stress.c | 32 ++
app/test/test_ring_stress.c | 57 ++++
app/test/test_ring_stress.h | 38 +++
app/test/test_ring_stress_impl.h | 444 +++++++++++++++++++++++++
devtools/libabigail.abignore | 7 +
lib/librte_pdump/rte_pdump.c | 2 +-
lib/librte_port/rte_port_ring.c | 12 +-
lib/librte_ring/Makefile | 11 +-
lib/librte_ring/meson.build | 11 +-
lib/librte_ring/rte_ring.c | 114 ++++++-
lib/librte_ring/rte_ring.h | 244 ++++++++++++--
lib/librte_ring/rte_ring_c11_mem.h | 44 +++
lib/librte_ring/rte_ring_elem.h | 105 +++++-
lib/librte_ring/rte_ring_generic.h | 48 +++
lib/librte_ring/rte_ring_hts.h | 214 ++++++++++++
lib/librte_ring/rte_ring_hts_c11_mem.h | 222 +++++++++++++
lib/librte_ring/rte_ring_hts_elem.h | 209 ++++++++++++
lib/librte_ring/rte_ring_hts_generic.h | 235 +++++++++++++
lib/librte_ring/rte_ring_peek.h | 379 +++++++++++++++++++++
lib/librte_ring/rte_ring_rts.h | 320 ++++++++++++++++++
lib/librte_ring/rte_ring_rts_c11_mem.h | 198 +++++++++++
lib/librte_ring/rte_ring_rts_elem.h | 209 ++++++++++++
lib/librte_ring/rte_ring_rts_generic.h | 210 ++++++++++++
29 files changed, 3431 insertions(+), 56 deletions(-)
create mode 100644 app/test/test_ring_hts_stress.c
create mode 100644 app/test/test_ring_mpmc_stress.c
create mode 100644 app/test/test_ring_peek_stress.c
create mode 100644 app/test/test_ring_rts_stress.c
create mode 100644 app/test/test_ring_stress.c
create mode 100644 app/test/test_ring_stress.h
create mode 100644 app/test/test_ring_stress_impl.h
create mode 100644 lib/librte_ring/rte_ring_hts.h
create mode 100644 lib/librte_ring/rte_ring_hts_c11_mem.h
create mode 100644 lib/librte_ring/rte_ring_hts_elem.h
create mode 100644 lib/librte_ring/rte_ring_hts_generic.h
create mode 100644 lib/librte_ring/rte_ring_peek.h
create mode 100644 lib/librte_ring/rte_ring_rts.h
create mode 100644 lib/librte_ring/rte_ring_rts_c11_mem.h
create mode 100644 lib/librte_ring/rte_ring_rts_elem.h
create mode 100644 lib/librte_ring/rte_ring_rts_generic.h
--
2.17.1
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v3 3/9] ring: introduce RTS ring mode
2020-04-03 17:42 3% ` [dpdk-dev] [PATCH v3 0/9] New sync modes for ring Konstantin Ananyev
@ 2020-04-03 17:42 1% ` Konstantin Ananyev
2020-04-04 17:27 0% ` Wang, Haiyue
2020-04-08 5:00 0% ` Honnappa Nagarahalli
2020-04-17 13:36 3% ` [dpdk-dev] [PATCH v4 0/9] New sync modes for ring Konstantin Ananyev
2 siblings, 2 replies; 200+ results
From: Konstantin Ananyev @ 2020-04-03 17:42 UTC (permalink / raw)
To: dev; +Cc: honnappa.nagarahalli, david.marchand, jielong.zjl, Konstantin Ananyev
Introduce relaxed tail sync (RTS) mode for MT ring synchronization.
Aim to reduce stall times in case when ring is used on
overcommited cpus (multiple active threads on the same cpu).
The main difference from original MP/MC algorithm is that
tail value is increased not by every thread that finished enqueue/dequeue,
but only by the last one.
That allows threads to avoid spinning on ring tail value,
leaving actual tail value change to the last thread in the update queue.
check-abi.sh reports what I believe is a false-positive about
ring cons/prod changes. As a workaround, devtools/libabigail.abignore is
updated to suppress *struct ring* related errors.
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
devtools/libabigail.abignore | 7 +
lib/librte_ring/Makefile | 5 +-
lib/librte_ring/meson.build | 5 +-
lib/librte_ring/rte_ring.c | 100 +++++++-
lib/librte_ring/rte_ring.h | 110 ++++++++-
lib/librte_ring/rte_ring_elem.h | 86 ++++++-
lib/librte_ring/rte_ring_rts.h | 316 +++++++++++++++++++++++++
lib/librte_ring/rte_ring_rts_elem.h | 205 ++++++++++++++++
lib/librte_ring/rte_ring_rts_generic.h | 210 ++++++++++++++++
9 files changed, 1015 insertions(+), 29 deletions(-)
create mode 100644 lib/librte_ring/rte_ring_rts.h
create mode 100644 lib/librte_ring/rte_ring_rts_elem.h
create mode 100644 lib/librte_ring/rte_ring_rts_generic.h
diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore
index a59df8f13..cd86d89ca 100644
--- a/devtools/libabigail.abignore
+++ b/devtools/libabigail.abignore
@@ -11,3 +11,10 @@
type_kind = enum
name = rte_crypto_asym_xform_type
changed_enumerators = RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
+; Ignore updates of ring prod/cons
+[suppress_type]
+ type_kind = struct
+ name = rte_ring
+[suppress_type]
+ type_kind = struct
+ name = rte_event_ring
diff --git a/lib/librte_ring/Makefile b/lib/librte_ring/Makefile
index 917c560ad..8f5c284cc 100644
--- a/lib/librte_ring/Makefile
+++ b/lib/librte_ring/Makefile
@@ -18,6 +18,9 @@ SRCS-$(CONFIG_RTE_LIBRTE_RING) := rte_ring.c
SYMLINK-$(CONFIG_RTE_LIBRTE_RING)-include := rte_ring.h \
rte_ring_elem.h \
rte_ring_generic.h \
- rte_ring_c11_mem.h
+ rte_ring_c11_mem.h \
+ rte_ring_rts.h \
+ rte_ring_rts_elem.h \
+ rte_ring_rts_generic.h
include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_ring/meson.build b/lib/librte_ring/meson.build
index f2f3ccc88..612936afb 100644
--- a/lib/librte_ring/meson.build
+++ b/lib/librte_ring/meson.build
@@ -5,7 +5,10 @@ sources = files('rte_ring.c')
headers = files('rte_ring.h',
'rte_ring_elem.h',
'rte_ring_c11_mem.h',
- 'rte_ring_generic.h')
+ 'rte_ring_generic.h',
+ 'rte_ring_rts.h',
+ 'rte_ring_rts_elem.h',
+ 'rte_ring_rts_generic.h')
# rte_ring_create_elem and rte_ring_get_memsize_elem are experimental
allow_experimental_apis = true
diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c
index fa5733907..222eec0fb 100644
--- a/lib/librte_ring/rte_ring.c
+++ b/lib/librte_ring/rte_ring.c
@@ -45,6 +45,9 @@ EAL_REGISTER_TAILQ(rte_ring_tailq)
/* true if x is a power of 2 */
#define POWEROF2(x) ((((x)-1) & (x)) == 0)
+/* by default set head/tail distance as 1/8 of ring capacity */
+#define HTD_MAX_DEF 8
+
/* return the size of memory occupied by a ring */
ssize_t
rte_ring_get_memsize_elem(unsigned int esize, unsigned int count)
@@ -79,11 +82,84 @@ rte_ring_get_memsize(unsigned int count)
return rte_ring_get_memsize_elem(sizeof(void *), count);
}
+/*
+ * internal helper function to reset prod/cons head-tail values.
+ */
+static void
+reset_headtail(void *p)
+{
+ struct rte_ring_headtail *ht;
+ struct rte_ring_rts_headtail *ht_rts;
+
+ ht = p;
+ ht_rts = p;
+
+ switch (ht->sync_type) {
+ case RTE_RING_SYNC_MT:
+ case RTE_RING_SYNC_ST:
+ ht->head = 0;
+ ht->tail = 0;
+ break;
+ case RTE_RING_SYNC_MT_RTS:
+ ht_rts->head.raw = 0;
+ ht_rts->tail.raw = 0;
+ break;
+ default:
+ /* unknown sync mode */
+ RTE_ASSERT(0);
+ }
+}
+
void
rte_ring_reset(struct rte_ring *r)
{
- r->prod.head = r->cons.head = 0;
- r->prod.tail = r->cons.tail = 0;
+ reset_headtail(&r->prod);
+ reset_headtail(&r->cons);
+}
+
+/*
+ * helper function, calculates sync_type values for prod and cons
+ * based on input flags. Returns zero at success or negative
+ * errno value otherwise.
+ */
+static int
+get_sync_type(uint32_t flags, enum rte_ring_sync_type *prod_st,
+ enum rte_ring_sync_type *cons_st)
+{
+ static const uint32_t prod_st_flags =
+ (RING_F_SP_ENQ | RING_F_MP_RTS_ENQ);
+ static const uint32_t cons_st_flags =
+ (RING_F_SC_DEQ | RING_F_MC_RTS_DEQ);
+
+ switch (flags & prod_st_flags) {
+ case 0:
+ *prod_st = RTE_RING_SYNC_MT;
+ break;
+ case RING_F_SP_ENQ:
+ *prod_st = RTE_RING_SYNC_ST;
+ break;
+ case RING_F_MP_RTS_ENQ:
+ *prod_st = RTE_RING_SYNC_MT_RTS;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ switch (flags & cons_st_flags) {
+ case 0:
+ *cons_st = RTE_RING_SYNC_MT;
+ break;
+ case RING_F_SC_DEQ:
+ *cons_st = RTE_RING_SYNC_ST;
+ break;
+ case RING_F_MC_RTS_DEQ:
+ *cons_st = RTE_RING_SYNC_MT_RTS;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
}
int
@@ -100,16 +176,20 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
RTE_BUILD_BUG_ON((offsetof(struct rte_ring, prod) &
RTE_CACHE_LINE_MASK) != 0);
+ RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, sync_type) !=
+ offsetof(struct rte_ring_rts_headtail, sync_type));
+ RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, tail) !=
+ offsetof(struct rte_ring_rts_headtail, tail.val.pos));
+
/* init the ring structure */
memset(r, 0, sizeof(*r));
ret = strlcpy(r->name, name, sizeof(r->name));
if (ret < 0 || ret >= (int)sizeof(r->name))
return -ENAMETOOLONG;
r->flags = flags;
- r->prod.sync_type = (flags & RING_F_SP_ENQ) ?
- RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
- r->cons.sync_type = (flags & RING_F_SC_DEQ) ?
- RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
+ ret = get_sync_type(flags, &r->prod.sync_type, &r->cons.sync_type);
+ if (ret != 0)
+ return ret;
if (flags & RING_F_EXACT_SZ) {
r->size = rte_align32pow2(count + 1);
@@ -126,8 +206,12 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
r->mask = count - 1;
r->capacity = r->mask;
}
- r->prod.head = r->cons.head = 0;
- r->prod.tail = r->cons.tail = 0;
+
+ /* set default values for head-tail distance */
+ if (flags & RING_F_MP_RTS_ENQ)
+ rte_ring_set_prod_htd_max(r, r->capacity / HTD_MAX_DEF);
+ if (flags & RING_F_MC_RTS_DEQ)
+ rte_ring_set_cons_htd_max(r, r->capacity / HTD_MAX_DEF);
return 0;
}
diff --git a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h
index d4775a063..f6f084d79 100644
--- a/lib/librte_ring/rte_ring.h
+++ b/lib/librte_ring/rte_ring.h
@@ -48,6 +48,7 @@ extern "C" {
#include <rte_branch_prediction.h>
#include <rte_memzone.h>
#include <rte_pause.h>
+#include <rte_debug.h>
#define RTE_TAILQ_RING_NAME "RTE_RING"
@@ -65,10 +66,13 @@ enum rte_ring_queue_behavior {
enum rte_ring_sync_type {
RTE_RING_SYNC_MT, /**< multi-thread safe (default mode) */
RTE_RING_SYNC_ST, /**< single thread only */
+#ifdef ALLOW_EXPERIMENTAL_API
+ RTE_RING_SYNC_MT_RTS, /**< multi-thread relaxed tail sync */
+#endif
};
/**
- * structure to hold a pair of head/tail values and other metadata.
+ * structures to hold a pair of head/tail values and other metadata.
* Depending on sync_type format of that structure might be different,
* but offset for *sync_type* and *tail* values should remain the same.
*/
@@ -84,6 +88,21 @@ struct rte_ring_headtail {
};
};
+union rte_ring_ht_poscnt {
+ uint64_t raw;
+ struct {
+ uint32_t cnt; /**< head/tail reference counter */
+ uint32_t pos; /**< head/tail position */
+ } val;
+};
+
+struct rte_ring_rts_headtail {
+ volatile union rte_ring_ht_poscnt tail;
+ enum rte_ring_sync_type sync_type; /**< sync type of prod/cons */
+ uint32_t htd_max; /**< max allowed distance between head/tail */
+ volatile union rte_ring_ht_poscnt head;
+};
+
/**
* An RTE ring structure.
*
@@ -111,11 +130,21 @@ struct rte_ring {
char pad0 __rte_cache_aligned; /**< empty cache line */
/** Ring producer status. */
- struct rte_ring_headtail prod __rte_cache_aligned;
+ RTE_STD_C11
+ union {
+ struct rte_ring_headtail prod;
+ struct rte_ring_rts_headtail rts_prod;
+ } __rte_cache_aligned;
+
char pad1 __rte_cache_aligned; /**< empty cache line */
/** Ring consumer status. */
- struct rte_ring_headtail cons __rte_cache_aligned;
+ RTE_STD_C11
+ union {
+ struct rte_ring_headtail cons;
+ struct rte_ring_rts_headtail rts_cons;
+ } __rte_cache_aligned;
+
char pad2 __rte_cache_aligned; /**< empty cache line */
};
@@ -132,6 +161,9 @@ struct rte_ring {
#define RING_F_EXACT_SZ 0x0004
#define RTE_RING_SZ_MASK (0x7fffffffU) /**< Ring size mask */
+#define RING_F_MP_RTS_ENQ 0x0008 /**< The default enqueue is "MP RTS". */
+#define RING_F_MC_RTS_DEQ 0x0010 /**< The default dequeue is "MC RTS". */
+
#define __IS_SP RTE_RING_SYNC_ST
#define __IS_MP RTE_RING_SYNC_MT
#define __IS_SC RTE_RING_SYNC_ST
@@ -461,6 +493,10 @@ rte_ring_sp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
RTE_RING_SYNC_ST, free_space);
}
+#ifdef ALLOW_EXPERIMENTAL_API
+#include <rte_ring_rts.h>
+#endif
+
/**
* Enqueue several objects on a ring.
*
@@ -484,8 +520,21 @@ static __rte_always_inline unsigned int
rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
- return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- r->prod.sync_type, free_space);
+ switch (r->prod.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mp_enqueue_bulk(r, obj_table, n, free_space);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sp_enqueue_bulk(r, obj_table, n, free_space);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mp_rts_enqueue_bulk(r, obj_table, n,
+ free_space);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ return 0;
}
/**
@@ -619,8 +668,20 @@ static __rte_always_inline unsigned int
rte_ring_dequeue_bulk(struct rte_ring *r, void **obj_table, unsigned int n,
unsigned int *available)
{
- return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- r->cons.sync_type, available);
+ switch (r->cons.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mc_dequeue_bulk(r, obj_table, n, available);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sc_dequeue_bulk(r, obj_table, n, available);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mc_rts_dequeue_bulk(r, obj_table, n, available);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ return 0;
}
/**
@@ -940,8 +1001,21 @@ static __rte_always_inline unsigned
rte_ring_enqueue_burst(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
- return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_VARIABLE,
- r->prod.sync_type, free_space);
+ switch (r->prod.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mp_enqueue_burst(r, obj_table, n, free_space);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sp_enqueue_burst(r, obj_table, n, free_space);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mp_rts_enqueue_burst(r, obj_table, n,
+ free_space);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ return 0;
}
/**
@@ -1020,9 +1094,21 @@ static __rte_always_inline unsigned
rte_ring_dequeue_burst(struct rte_ring *r, void **obj_table,
unsigned int n, unsigned int *available)
{
- return __rte_ring_do_dequeue(r, obj_table, n,
- RTE_RING_QUEUE_VARIABLE,
- r->cons.sync_type, available);
+ switch (r->cons.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mc_dequeue_burst(r, obj_table, n, available);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sc_dequeue_burst(r, obj_table, n, available);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mc_rts_dequeue_burst(r, obj_table, n,
+ available);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ return 0;
}
#ifdef __cplusplus
diff --git a/lib/librte_ring/rte_ring_elem.h b/lib/librte_ring/rte_ring_elem.h
index 28f9836e6..5de0850dc 100644
--- a/lib/librte_ring/rte_ring_elem.h
+++ b/lib/librte_ring/rte_ring_elem.h
@@ -542,6 +542,8 @@ rte_ring_sp_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
RTE_RING_QUEUE_FIXED, __IS_SP, free_space);
}
+#include <rte_ring_rts_elem.h>
+
/**
* Enqueue several objects on a ring.
*
@@ -571,6 +573,26 @@ rte_ring_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
{
return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
RTE_RING_QUEUE_FIXED, r->prod.sync_type, free_space);
+
+ switch (r->prod.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mp_enqueue_bulk_elem(r, obj_table, esize, n,
+ free_space);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sp_enqueue_bulk_elem(r, obj_table, esize, n,
+ free_space);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mp_rts_enqueue_bulk_elem(r, obj_table, esize, n,
+ free_space);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ if (free_space != NULL)
+ *free_space = 0;
+ return 0;
}
/**
@@ -733,8 +755,25 @@ static __rte_always_inline unsigned int
rte_ring_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
unsigned int esize, unsigned int n, unsigned int *available)
{
- return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, r->cons.sync_type, available);
+ switch (r->cons.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mc_dequeue_bulk_elem(r, obj_table, esize, n,
+ available);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sc_dequeue_bulk_elem(r, obj_table, esize, n,
+ available);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mc_rts_dequeue_bulk_elem(r, obj_table, esize,
+ n, available);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ if (available != NULL)
+ *available = 0;
+ return 0;
}
/**
@@ -901,8 +940,25 @@ static __rte_always_inline unsigned
rte_ring_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
unsigned int esize, unsigned int n, unsigned int *free_space)
{
- return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE, r->prod.sync_type, free_space);
+ switch (r->prod.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mp_enqueue_burst_elem(r, obj_table, esize, n,
+ free_space);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sp_enqueue_burst_elem(r, obj_table, esize, n,
+ free_space);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mp_rts_enqueue_burst_elem(r, obj_table, esize,
+ n, free_space);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ if (free_space != NULL)
+ *free_space = 0;
+ return 0;
}
/**
@@ -993,9 +1049,25 @@ static __rte_always_inline unsigned int
rte_ring_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
unsigned int esize, unsigned int n, unsigned int *available)
{
- return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE,
- r->cons.sync_type, available);
+ switch (r->cons.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mc_dequeue_burst_elem(r, obj_table, esize, n,
+ available);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sc_dequeue_burst_elem(r, obj_table, esize, n,
+ available);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mc_rts_dequeue_burst_elem(r, obj_table, esize,
+ n, available);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ if (available != NULL)
+ *available = 0;
+ return 0;
}
#ifdef __cplusplus
diff --git a/lib/librte_ring/rte_ring_rts.h b/lib/librte_ring/rte_ring_rts.h
new file mode 100644
index 000000000..18404fe48
--- /dev/null
+++ b/lib/librte_ring/rte_ring_rts.h
@@ -0,0 +1,316 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#ifndef _RTE_RING_RTS_H_
+#define _RTE_RING_RTS_H_
+
+/**
+ * @file rte_ring_rts.h
+ * @b EXPERIMENTAL: this API may change without prior notice
+ * It is not recommended to include this file directly.
+ * Please include <rte_ring.h> instead.
+ *
+ * Contains functions for Relaxed Tail Sync (RTS) ring mode.
+ * The main idea remains the same as for our original MP/MC synchronization
+ * mechanism.
+ * The main difference is that tail value is increased not
+ * by every thread that finished enqueue/dequeue,
+ * but only by the last one doing enqueue/dequeue.
+ * That allows threads to skip spinning on tail value,
+ * leaving actual tail value change to last thread in the update queue.
+ * RTS requires 2 64-bit CAS for each enqueue(/dequeue) operation:
+ * one for head update, second for tail update.
+ * As a gain it allows thread to avoid spinning/waiting on tail value.
+ * In comparision original MP/MC algorithm requires one 32-bit CAS
+ * for head update and waiting/spinning on tail value.
+ *
+ * Brief outline:
+ * - introduce refcnt for both head and tail.
+ * - increment head.refcnt for each head.value update
+ * - write head:value and head:refcnt atomically (64-bit CAS)
+ * - move tail.value ahead only when tail.refcnt + 1 == head.refcnt
+ * - increment tail.refcnt when each enqueue/dequeue op finishes
+ * (no matter is tail:value going to change or not)
+ * - write tail.value and tail.recnt atomically (64-bit CAS)
+ *
+ * To avoid producer/consumer starvation:
+ * - limit max allowed distance between head and tail value (HTD_MAX).
+ * I.E. thread is allowed to proceed with changing head.value,
+ * only when: head.value - tail.value <= HTD_MAX
+ * HTD_MAX is an optional parameter.
+ * With HTD_MAX == 0 we'll have fully serialized ring -
+ * i.e. only one thread at a time will be able to enqueue/dequeue
+ * to/from the ring.
+ * With HTD_MAX >= ring.capacity - no limitation.
+ * By default HTD_MAX == ring.capacity / 8.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_ring_rts_generic.h>
+
+/**
+ * @internal Enqueue several objects on the RTS ring.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
+ * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
+ * @param free_space
+ * returns the amount of space after the enqueue operation has finished
+ * @return
+ * Actual number of objects enqueued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_enqueue(struct rte_ring *r, void * const *obj_table,
+ uint32_t n, enum rte_ring_queue_behavior behavior,
+ uint32_t *free_space)
+{
+ uint32_t free, head;
+
+ n = __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
+
+ if (n != 0) {
+ ENQUEUE_PTRS(r, &r[1], head, obj_table, n, void *);
+ __rte_ring_rts_update_tail(&r->rts_prod);
+ }
+
+ if (free_space != NULL)
+ *free_space = free - n;
+ return n;
+}
+
+/**
+ * @internal Dequeue several objects from the RTS ring.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to pull from the ring.
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
+ * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
+ * @param available
+ * returns the number of remaining ring entries after the dequeue has finished
+ * @return
+ * - Actual number of objects dequeued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_dequeue(struct rte_ring *r, void **obj_table,
+ uint32_t n, enum rte_ring_queue_behavior behavior,
+ uint32_t *available)
+{
+ uint32_t entries, head;
+
+ n = __rte_ring_rts_move_cons_head(r, n, behavior, &head, &entries);
+
+ if (n != 0) {
+ DEQUEUE_PTRS(r, &r[1], head, obj_table, n, void *);
+ __rte_ring_rts_update_tail(&r->rts_cons);
+ }
+
+ if (available != NULL)
+ *available = entries - n;
+ return n;
+}
+
+/**
+ * Enqueue several objects on the RTS ring (multi-producers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * The number of objects enqueued, either 0 or n
+ */
+__rte_experimental
+static __rte_always_inline unsigned int
+rte_ring_mp_rts_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
+ unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_rts_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
+ free_space);
+}
+
+/**
+ * Dequeue several objects from an RTS ring (multi-consumers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * The number of objects dequeued, either 0 or n
+ */
+__rte_experimental
+static __rte_always_inline unsigned int
+rte_ring_mc_rts_dequeue_bulk(struct rte_ring *r, void **obj_table,
+ unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_rts_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
+ available);
+}
+
+/**
+ * Return producer max Head-Tail-Distance (HTD).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * Producer HTD value, if producer is set in appropriate sync mode,
+ * or UINT32_MAX otherwise.
+ */
+__rte_experimental
+static inline uint32_t
+rte_ring_get_prod_htd_max(const struct rte_ring *r)
+{
+ if (r->prod.sync_type == RTE_RING_SYNC_MT_RTS)
+ return r->rts_prod.htd_max;
+ return UINT32_MAX;
+}
+
+/**
+ * Set producer max Head-Tail-Distance (HTD).
+ * Note that producer has to use appropriate sync mode (RTS).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param v
+ * new HTD value to setup.
+ * @return
+ * Zero on success, or negative error code otherwise.
+ */
+__rte_experimental
+static inline int
+rte_ring_set_prod_htd_max(struct rte_ring *r, uint32_t v)
+{
+ if (r->prod.sync_type != RTE_RING_SYNC_MT_RTS)
+ return -ENOTSUP;
+
+ r->rts_prod.htd_max = v;
+ return 0;
+}
+
+/**
+ * Return consumer max Head-Tail-Distance (HTD).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * Consumer HTD value, if consumer is set in appropriate sync mode,
+ * or UINT32_MAX otherwise.
+ */
+__rte_experimental
+static inline uint32_t
+rte_ring_get_cons_htd_max(const struct rte_ring *r)
+{
+ if (r->cons.sync_type == RTE_RING_SYNC_MT_RTS)
+ return r->rts_cons.htd_max;
+ return UINT32_MAX;
+}
+
+/**
+ * Set consumer max Head-Tail-Distance (HTD).
+ * Note that consumer has to use appropriate sync mode (RTS).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param v
+ * new HTD value to setup.
+ * @return
+ * Zero on success, or negative error code otherwise.
+ */
+__rte_experimental
+static inline int
+rte_ring_set_cons_htd_max(struct rte_ring *r, uint32_t v)
+{
+ if (r->cons.sync_type != RTE_RING_SYNC_MT_RTS)
+ return -ENOTSUP;
+
+ r->rts_cons.htd_max = v;
+ return 0;
+}
+
+/**
+ * Enqueue several objects on the RTS ring (multi-producers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * - n: Actual number of objects enqueued.
+ */
+__rte_experimental
+static __rte_always_inline unsigned
+rte_ring_mp_rts_enqueue_burst(struct rte_ring *r, void * const *obj_table,
+ unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_rts_enqueue(r, obj_table, n,
+ RTE_RING_QUEUE_VARIABLE, free_space);
+}
+
+/**
+ * Dequeue several objects from an RTS ring (multi-consumers safe).
+ * When the requested objects are more than the available objects,
+ * only dequeue the actual number of objects.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * - n: Actual number of objects dequeued, 0 if ring is empty
+ */
+__rte_experimental
+static __rte_always_inline unsigned
+rte_ring_mc_rts_dequeue_burst(struct rte_ring *r, void **obj_table,
+ unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_rts_dequeue(r, obj_table, n,
+ RTE_RING_QUEUE_VARIABLE, available);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_RING_RTS_H_ */
diff --git a/lib/librte_ring/rte_ring_rts_elem.h b/lib/librte_ring/rte_ring_rts_elem.h
new file mode 100644
index 000000000..71a331b23
--- /dev/null
+++ b/lib/librte_ring/rte_ring_rts_elem.h
@@ -0,0 +1,205 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#ifndef _RTE_RING_RTS_ELEM_H_
+#define _RTE_RING_RTS_ELEM_H_
+
+/**
+ * @file rte_ring_rts_elem.h
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * It is not recommended to include this file directly.
+ * Please include <rte_ring_elem.h> instead.
+ * Contains *ring_elem* functions for Relaxed Tail Sync (RTS) ring mode.
+ * for more details please refer to <rte_ring_rts.h>.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_ring_rts_generic.h>
+
+/**
+ * @internal Enqueue several objects on the RTS ring.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
+ * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
+ * @param free_space
+ * returns the amount of space after the enqueue operation has finished
+ * @return
+ * Actual number of objects enqueued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_enqueue_elem(struct rte_ring *r, void * const *obj_table,
+ uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+ uint32_t *free_space)
+{
+ uint32_t free, head;
+
+ n = __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
+
+ if (n != 0) {
+ __rte_ring_enqueue_elems(r, head, obj_table, esize, n);
+ __rte_ring_rts_update_tail(&r->rts_prod);
+ }
+
+ if (free_space != NULL)
+ *free_space = free - n;
+ return n;
+}
+
+/**
+ * @internal Dequeue several objects from the RTS ring.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to pull from the ring.
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
+ * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
+ * @param available
+ * returns the number of remaining ring entries after the dequeue has finished
+ * @return
+ * - Actual number of objects dequeued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_dequeue_elem(struct rte_ring *r, void **obj_table,
+ uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+ uint32_t *available)
+{
+ uint32_t entries, head;
+
+ n = __rte_ring_rts_move_cons_head(r, n, behavior, &head, &entries);
+
+ if (n != 0) {
+ __rte_ring_dequeue_elems(r, head, obj_table, esize, n);
+ __rte_ring_rts_update_tail(&r->rts_cons);
+ }
+
+ if (available != NULL)
+ *available = entries - n;
+ return n;
+}
+
+/**
+ * Enqueue several objects on the RTS ring (multi-producers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * The number of objects enqueued, either 0 or n
+ */
+__rte_experimental
+static __rte_always_inline unsigned int
+rte_ring_mp_rts_enqueue_bulk_elem(struct rte_ring *r, void * const *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_rts_enqueue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_FIXED, free_space);
+}
+
+/**
+ * Dequeue several objects from an RTS ring (multi-consumers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * The number of objects dequeued, either 0 or n
+ */
+__rte_experimental
+static __rte_always_inline unsigned int
+rte_ring_mc_rts_dequeue_bulk_elem(struct rte_ring *r, void **obj_table,
+ unsigned int esize, unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_rts_dequeue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_FIXED, available);
+}
+
+/**
+ * Enqueue several objects on the RTS ring (multi-producers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * - n: Actual number of objects enqueued.
+ */
+__rte_experimental
+static __rte_always_inline unsigned
+rte_ring_mp_rts_enqueue_burst_elem(struct rte_ring *r, void * const *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_rts_enqueue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_VARIABLE, free_space);
+}
+
+/**
+ * Dequeue several objects from an RTS ring (multi-consumers safe).
+ * When the requested objects are more than the available objects,
+ * only dequeue the actual number of objects.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * - n: Actual number of objects dequeued, 0 if ring is empty
+ */
+__rte_experimental
+static __rte_always_inline unsigned
+rte_ring_mc_rts_dequeue_burst_elem(struct rte_ring *r, void **obj_table,
+ unsigned int esize, unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_rts_dequeue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_VARIABLE, available);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_RING_RTS_ELEM_H_ */
diff --git a/lib/librte_ring/rte_ring_rts_generic.h b/lib/librte_ring/rte_ring_rts_generic.h
new file mode 100644
index 000000000..f88460d47
--- /dev/null
+++ b/lib/librte_ring/rte_ring_rts_generic.h
@@ -0,0 +1,210 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#ifndef _RTE_RING_RTS_GENERIC_H_
+#define _RTE_RING_RTS_GENERIC_H_
+
+/**
+ * @file rte_ring_rts_generic.h
+ * It is not recommended to include this file directly,
+ * include <rte_ring.h> instead.
+ * Contains internal helper functions for Relaxed Tail Sync (RTS) ring mode.
+ * For more information please refer to <rte_ring_rts.h>.
+ */
+
+/**
+ * @internal This function updates tail values.
+ */
+static __rte_always_inline void
+__rte_ring_rts_update_tail(struct rte_ring_rts_headtail *ht)
+{
+ union rte_ring_ht_poscnt h, ot, nt;
+
+ /*
+ * If there are other enqueues/dequeues in progress that
+ * might preceded us, then don't update tail with new value.
+ */
+
+ do {
+ ot.raw = ht->tail.raw;
+ rte_smp_rmb();
+
+ /* on 32-bit systems we have to do atomic read here */
+ h.raw = rte_atomic64_read((rte_atomic64_t *)
+ (uintptr_t)&ht->head.raw);
+
+ nt.raw = ot.raw;
+ if (++nt.val.cnt == h.val.cnt)
+ nt.val.pos = h.val.pos;
+
+ } while (rte_atomic64_cmpset(&ht->tail.raw, ot.raw, nt.raw) == 0);
+}
+
+/**
+ * @internal This function waits till head/tail distance wouldn't
+ * exceed pre-defined max value.
+ */
+static __rte_always_inline void
+__rte_ring_rts_head_wait(const struct rte_ring_rts_headtail *ht,
+ union rte_ring_ht_poscnt *h)
+{
+ uint32_t max;
+
+ max = ht->htd_max;
+ h->raw = ht->head.raw;
+ rte_smp_rmb();
+
+ while (h->val.pos - ht->tail.val.pos > max) {
+ rte_pause();
+ h->raw = ht->head.raw;
+ rte_smp_rmb();
+ }
+}
+
+/**
+ * @internal This function updates the producer head for enqueue.
+ *
+ * @param r
+ * A pointer to the ring structure
+ * @param is_sp
+ * Indicates whether multi-producer path is needed or not
+ * @param n
+ * The number of elements we will want to enqueue, i.e. how far should the
+ * head be moved
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
+ * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
+ * @param old_head
+ * Returns head value as it was before the move, i.e. where enqueue starts
+ * @param new_head
+ * Returns the current/new head value i.e. where enqueue finishes
+ * @param free_entries
+ * Returns the amount of free space in the ring BEFORE head was moved
+ * @return
+ * Actual number of objects enqueued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline uint32_t
+__rte_ring_rts_move_prod_head(struct rte_ring *r, uint32_t num,
+ enum rte_ring_queue_behavior behavior, uint32_t *old_head,
+ uint32_t *free_entries)
+{
+ uint32_t n;
+ union rte_ring_ht_poscnt nh, oh;
+
+ const uint32_t capacity = r->capacity;
+
+ do {
+ /* Reset n to the initial burst count */
+ n = num;
+
+ /* read prod head (may spin on prod tail) */
+ __rte_ring_rts_head_wait(&r->rts_prod, &oh);
+
+ /* add rmb barrier to avoid load/load reorder in weak
+ * memory model. It is noop on x86
+ */
+ rte_smp_rmb();
+
+ /*
+ * The subtraction is done between two unsigned 32bits value
+ * (the result is always modulo 32 bits even if we have
+ * *old_head > cons_tail). So 'free_entries' is always between 0
+ * and capacity (which is < size).
+ */
+ *free_entries = capacity + r->cons.tail - oh.val.pos;
+
+ /* check that we have enough room in ring */
+ if (unlikely(n > *free_entries))
+ n = (behavior == RTE_RING_QUEUE_FIXED) ?
+ 0 : *free_entries;
+
+ if (n == 0)
+ break;
+
+ nh.val.pos = oh.val.pos + n;
+ nh.val.cnt = oh.val.cnt + 1;
+
+ } while (rte_atomic64_cmpset(&r->rts_prod.head.raw,
+ oh.raw, nh.raw) == 0);
+
+ *old_head = oh.val.pos;
+ return n;
+}
+
+/**
+ * @internal This function updates the consumer head for dequeue
+ *
+ * @param r
+ * A pointer to the ring structure
+ * @param is_sc
+ * Indicates whether multi-consumer path is needed or not
+ * @param n
+ * The number of elements we will want to enqueue, i.e. how far should the
+ * head be moved
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
+ * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
+ * @param old_head
+ * Returns head value as it was before the move, i.e. where dequeue starts
+ * @param new_head
+ * Returns the current/new head value i.e. where dequeue finishes
+ * @param entries
+ * Returns the number of entries in the ring BEFORE head was moved
+ * @return
+ * - Actual number of objects dequeued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num,
+ enum rte_ring_queue_behavior behavior, uint32_t *old_head,
+ uint32_t *entries)
+{
+ uint32_t n;
+ union rte_ring_ht_poscnt nh, oh;
+
+ /* move cons.head atomically */
+ do {
+ /* Restore n as it may change every loop */
+ n = num;
+
+ /* read cons head (may spin on cons tail) */
+ __rte_ring_rts_head_wait(&r->rts_cons, &oh);
+
+
+ /* add rmb barrier to avoid load/load reorder in weak
+ * memory model. It is noop on x86
+ */
+ rte_smp_rmb();
+
+ /* The subtraction is done between two unsigned 32bits value
+ * (the result is always modulo 32 bits even if we have
+ * cons_head > prod_tail). So 'entries' is always between 0
+ * and size(ring)-1.
+ */
+ *entries = r->prod.tail - oh.val.pos;
+
+ /* Set the actual entries for dequeue */
+ if (n > *entries)
+ n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 : *entries;
+
+ if (unlikely(n == 0))
+ break;
+
+ nh.val.pos = oh.val.pos + n;
+ nh.val.cnt = oh.val.cnt + 1;
+
+ } while (rte_atomic64_cmpset(&r->rts_cons.head.raw,
+ oh.raw, nh.raw) == 0);
+
+ *old_head = oh.val.pos;
+ return n;
+}
+
+#endif /* _RTE_RING_RTS_GENERIC_H_ */
--
2.17.1
^ permalink raw reply [relevance 1%]
* Re: [dpdk-dev] [PATCH v3 3/9] ring: introduce RTS ring mode
2020-04-03 17:42 1% ` [dpdk-dev] [PATCH v3 3/9] ring: introduce RTS ring mode Konstantin Ananyev
@ 2020-04-04 17:27 0% ` Wang, Haiyue
2020-04-08 5:00 0% ` Honnappa Nagarahalli
1 sibling, 0 replies; 200+ results
From: Wang, Haiyue @ 2020-04-04 17:27 UTC (permalink / raw)
To: Ananyev, Konstantin, dev
Cc: honnappa.nagarahalli, david.marchand, jielong.zjl, Ananyev, Konstantin
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Konstantin Ananyev
> Sent: Saturday, April 4, 2020 01:42
> To: dev@dpdk.org
> Cc: honnappa.nagarahalli@arm.com; david.marchand@redhat.com; jielong.zjl@antfin.com; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>
> Subject: [dpdk-dev] [PATCH v3 3/9] ring: introduce RTS ring mode
>
> Introduce relaxed tail sync (RTS) mode for MT ring synchronization.
> Aim to reduce stall times in case when ring is used on
> overcommited cpus (multiple active threads on the same cpu).
> The main difference from original MP/MC algorithm is that
> tail value is increased not by every thread that finished enqueue/dequeue,
> but only by the last one.
> That allows threads to avoid spinning on ring tail value,
> leaving actual tail value change to the last thread in the update queue.
>
> check-abi.sh reports what I believe is a false-positive about
> ring cons/prod changes. As a workaround, devtools/libabigail.abignore is
> updated to suppress *struct ring* related errors.
>
> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> ---
> devtools/libabigail.abignore | 7 +
> lib/librte_ring/Makefile | 5 +-
> lib/librte_ring/meson.build | 5 +-
> lib/librte_ring/rte_ring.c | 100 +++++++-
> lib/librte_ring/rte_ring.h | 110 ++++++++-
> lib/librte_ring/rte_ring_elem.h | 86 ++++++-
> lib/librte_ring/rte_ring_rts.h | 316 +++++++++++++++++++++++++
> lib/librte_ring/rte_ring_rts_elem.h | 205 ++++++++++++++++
> lib/librte_ring/rte_ring_rts_generic.h | 210 ++++++++++++++++
> 9 files changed, 1015 insertions(+), 29 deletions(-)
> create mode 100644 lib/librte_ring/rte_ring_rts.h
> create mode 100644 lib/librte_ring/rte_ring_rts_elem.h
> create mode 100644 lib/librte_ring/rte_ring_rts_generic.h
>
> #ifdef __cplusplus
> diff --git a/lib/librte_ring/rte_ring_rts.h b/lib/librte_ring/rte_ring_rts.h
> new file mode 100644
> index 000000000..18404fe48
> --- /dev/null
> +++ b/lib/librte_ring/rte_ring_rts.h
> @@ -0,0 +1,316 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + *
> + * Copyright (c) 2010-2017 Intel Corporation
> + * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
> + * All rights reserved.
> + * Derived from FreeBSD's bufring.h
Find that it is buf_ring.h in real ;-)
> + * Used as BSD-3 Licensed with permission from Kip Macy.
> + */
> --
> 2.17.1
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [EXT] [PATCH v1 4/4] regexdev: implement regex rte level functions
@ 2020-04-06 11:16 3% ` Thomas Monjalon
2020-04-06 12:33 3% ` Pavan Nikhilesh Bhagavatula
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-04-06 11:16 UTC (permalink / raw)
To: Pavan Nikhilesh Bhagavatula, Ori Kam
Cc: Jerin Jacob Kollanukkaran, xiang.w.wang, dev, Shahaf Shuler,
hemant.agrawal, Opher Reviv, Alex Rosenbaum, Dovrat Zifroni,
Prasun Kapoor, nipun.gupta, bruce.richardson, yang.a.hong,
harry.chang, gu.jian1, shanjiangh, zhangy.yun, lixingfu, wushuai,
yuyingxia, fanchenggang, davidfgao, liuzhong1, zhaoyong11, oc,
jim, hongjun.ni, j.bromhead, deri, fc, arthur.su, david.marchand
05/04/2020 17:04, Ori Kam:
> From: Pavan Nikhilesh Bhagavatula
> > >+uint16_t
> > >+rte_regexdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
> > >+ struct rte_regex_ops **ops, uint16_t nb_ops)
> > >+{
> > >+ return regex_devices[dev_id]-
> > >>enqueue(regex_devices[dev_id], qp_id,
> > >+ ops, nb_ops);
> > >+}
> >
> > Move these functions to .h in-lining them.
> > Also, please add debug checks @see rte_eth_rx_burst/rte_eth_tx_burst.
>
> O.K will update.
In general, inlining is a pain for ABI compatibility.
Please inline only if the gain is very significant.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [EXT] [PATCH v1 4/4] regexdev: implement regex rte level functions
2020-04-06 11:16 3% ` Thomas Monjalon
@ 2020-04-06 12:33 3% ` Pavan Nikhilesh Bhagavatula
2020-04-06 13:14 3% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Pavan Nikhilesh Bhagavatula @ 2020-04-06 12:33 UTC (permalink / raw)
To: Thomas Monjalon, Ori Kam
Cc: Jerin Jacob Kollanukkaran, xiang.w.wang, dev, Shahaf Shuler,
hemant.agrawal, Opher Reviv, Alex Rosenbaum, Dovrat Zifroni,
Prasun Kapoor, nipun.gupta, bruce.richardson, yang.a.hong,
harry.chang, gu.jian1, shanjiangh, zhangy.yun, lixingfu, wushuai,
yuyingxia, fanchenggang, davidfgao, liuzhong1, zhaoyong11, oc,
jim, hongjun.ni, j.bromhead, deri, fc, arthur.su, david.marchand
>> From: Pavan Nikhilesh Bhagavatula
>> > >+uint16_t
>> > >+rte_regexdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
>> > >+ struct rte_regex_ops **ops, uint16_t nb_ops)
>> > >+{
>> > >+ return regex_devices[dev_id]-
>> > >>enqueue(regex_devices[dev_id], qp_id,
>> > >+ ops, nb_ops);
>> > >+}
>> >
>> > Move these functions to .h in-lining them.
>> > Also, please add debug checks @see
>rte_eth_rx_burst/rte_eth_tx_burst.
>>
>> O.K will update.
>
>In general, inlining is a pain for ABI compatibility.
>Please inline only if the gain is very significant.
>
The performance gain mostly comes from hoisting `regex_devices[dev_id]` load above the
poll loop.
Since, the performance measurement application is still in pipeline and regexdev would be
experimental for next couple of releases I suggest inlining it now and worrying about ABI when
experimental tag needs to be removed.
We can follow the same path as done by ethdev [https://www.mail-archive.com/dev@dpdk.org/msg142392.html]
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [EXT] [PATCH v1 4/4] regexdev: implement regex rte level functions
2020-04-06 12:33 3% ` Pavan Nikhilesh Bhagavatula
@ 2020-04-06 13:14 3% ` Thomas Monjalon
2020-04-06 13:20 0% ` Jerin Jacob
2020-04-06 13:22 0% ` Pavan Nikhilesh Bhagavatula
0 siblings, 2 replies; 200+ results
From: Thomas Monjalon @ 2020-04-06 13:14 UTC (permalink / raw)
To: Pavan Nikhilesh Bhagavatula
Cc: Ori Kam, Jerin Jacob Kollanukkaran, xiang.w.wang, dev,
Shahaf Shuler, hemant.agrawal, Opher Reviv, Alex Rosenbaum,
Dovrat Zifroni, Prasun Kapoor, nipun.gupta, bruce.richardson,
yang.a.hong, harry.chang, gu.jian1, shanjiangh, zhangy.yun,
lixingfu, wushuai, yuyingxia, fanchenggang, davidfgao, liuzhong1,
zhaoyong11, oc, jim, hongjun.ni, j.bromhead, deri, fc, arthur.su,
david.marchand
06/04/2020 14:33, Pavan Nikhilesh Bhagavatula:
> >> From: Pavan Nikhilesh Bhagavatula
> >> > >+uint16_t
> >> > >+rte_regexdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
> >> > >+ struct rte_regex_ops **ops, uint16_t nb_ops)
> >> > >+{
> >> > >+ return regex_devices[dev_id]-
> >> > >>enqueue(regex_devices[dev_id], qp_id,
> >> > >+ ops, nb_ops);
> >> > >+}
> >> >
> >> > Move these functions to .h in-lining them.
> >> > Also, please add debug checks @see
> >rte_eth_rx_burst/rte_eth_tx_burst.
> >>
> >> O.K will update.
> >
> >In general, inlining is a pain for ABI compatibility.
> >Please inline only if the gain is very significant.
> >
>
> The performance gain mostly comes from hoisting `regex_devices[dev_id]` load above the
> poll loop.
> Since, the performance measurement application is still in pipeline and regexdev would be
> experimental for next couple of releases I suggest inlining it now and worrying about ABI when
> experimental tag needs to be removed.
No, we must worry about ABI from the beginning.
> We can follow the same path as done by ethdev [https://www.mail-archive.com/dev@dpdk.org/msg142392.html]
ethdev is not an argument.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [EXT] [PATCH v1 4/4] regexdev: implement regex rte level functions
2020-04-06 13:14 3% ` Thomas Monjalon
@ 2020-04-06 13:20 0% ` Jerin Jacob
2020-04-06 13:22 0% ` Pavan Nikhilesh Bhagavatula
1 sibling, 0 replies; 200+ results
From: Jerin Jacob @ 2020-04-06 13:20 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Pavan Nikhilesh Bhagavatula, Ori Kam, Jerin Jacob Kollanukkaran,
xiang.w.wang, dev, Shahaf Shuler, hemant.agrawal, Opher Reviv,
Alex Rosenbaum, Dovrat Zifroni, Prasun Kapoor, nipun.gupta,
bruce.richardson, yang.a.hong, harry.chang, gu.jian1, shanjiangh,
zhangy.yun, lixingfu, wushuai, yuyingxia, fanchenggang,
davidfgao, liuzhong1, zhaoyong11, oc, jim, hongjun.ni,
j.bromhead, deri, fc, arthur.su, david.marchand
On Mon, Apr 6, 2020 at 6:44 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 06/04/2020 14:33, Pavan Nikhilesh Bhagavatula:
> > >> From: Pavan Nikhilesh Bhagavatula
> > >> > >+uint16_t
> > >> > >+rte_regexdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
> > >> > >+ struct rte_regex_ops **ops, uint16_t nb_ops)
> > >> > >+{
> > >> > >+ return regex_devices[dev_id]-
> > >> > >>enqueue(regex_devices[dev_id], qp_id,
> > >> > >+ ops, nb_ops);
> > >> > >+}
> > >> >
> > >> > Move these functions to .h in-lining them.
> > >> > Also, please add debug checks @see
> > >rte_eth_rx_burst/rte_eth_tx_burst.
> > >>
> > >> O.K will update.
> > >
> > >In general, inlining is a pain for ABI compatibility.
> > >Please inline only if the gain is very significant.
> > >
> >
> > The performance gain mostly comes from hoisting `regex_devices[dev_id]` load above the
> > poll loop.
> > Since, the performance measurement application is still in pipeline and regexdev would be
> > experimental for next couple of releases I suggest inlining it now and worrying about ABI when
> > experimental tag needs to be removed.
>
> No, we must worry about ABI from the beginning.
I think, we need to have the performance number first before we decide
one or another.
>
> > We can follow the same path as done by ethdev [https://www.mail-archive.com/dev@dpdk.org/msg142392.html]
>
> ethdev is not an argument.
Actually this thread explains how to make it inline without exposing
the internal structure unlike existing ethdev. The effort
is stalled due to PMD changes required. In this case, regexdev is new
so it is the correct time to add such code.
>
>
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [EXT] [PATCH v1 4/4] regexdev: implement regex rte level functions
2020-04-06 13:14 3% ` Thomas Monjalon
2020-04-06 13:20 0% ` Jerin Jacob
@ 2020-04-06 13:22 0% ` Pavan Nikhilesh Bhagavatula
2020-04-06 13:36 0% ` Thomas Monjalon
1 sibling, 1 reply; 200+ results
From: Pavan Nikhilesh Bhagavatula @ 2020-04-06 13:22 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Ori Kam, Jerin Jacob Kollanukkaran, xiang.w.wang, dev,
Shahaf Shuler, hemant.agrawal, Opher Reviv, Alex Rosenbaum,
Dovrat Zifroni, Prasun Kapoor, nipun.gupta, bruce.richardson,
yang.a.hong, harry.chang, gu.jian1, shanjiangh, zhangy.yun,
lixingfu, wushuai, yuyingxia, fanchenggang, davidfgao, liuzhong1,
zhaoyong11, oc, jim, hongjun.ni, j.bromhead, deri, fc, arthur.su,
david.marchand
>06/04/2020 14:33, Pavan Nikhilesh Bhagavatula:
>> >> From: Pavan Nikhilesh Bhagavatula
>> >> > >+uint16_t
>> >> > >+rte_regexdev_enqueue_burst(uint8_t dev_id, uint16_t
>qp_id,
>> >> > >+ struct rte_regex_ops **ops, uint16_t
>nb_ops)
>> >> > >+{
>> >> > >+ return regex_devices[dev_id]-
>> >> > >>enqueue(regex_devices[dev_id], qp_id,
>> >> > >+ ops, nb_ops);
>> >> > >+}
>> >> >
>> >> > Move these functions to .h in-lining them.
>> >> > Also, please add debug checks @see
>> >rte_eth_rx_burst/rte_eth_tx_burst.
>> >>
>> >> O.K will update.
>> >
>> >In general, inlining is a pain for ABI compatibility.
>> >Please inline only if the gain is very significant.
>> >
>>
>> The performance gain mostly comes from hoisting
>`regex_devices[dev_id]` load above the
>> poll loop.
>> Since, the performance measurement application is still in pipeline
>and regexdev would be
>> experimental for next couple of releases I suggest inlining it now and
>worrying about ABI when
>> experimental tag needs to be removed.
>
>No, we must worry about ABI from the beginning.
I though performance was the primary objective :-).
>
>> We can follow the same path as done by ethdev
>[https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mail-
>2Darchive.com_dev-
>40dpdk.org_msg142392.html&d=DwICAg&c=nKjWec2b6R0mOyPaz7xtf
>Q&r=E3SgYMjtKCMVsB-fmvgGV3o-
>g_fjLhk5Pupi9ijohpc&m=7Gqb6WKmZV5uY3xa7FRVrRVDz8Usrsd-
>rDjIKr6CUQQ&s=sQo2Kx9fzTNXwiQ2Fzki3s5GSuiiAEzz2VtN68_KKXo&e=
>]
>
>ethdev is not an argument.
What about ring? [http://mails.dpdk.org/archives/dev/2020-April/161506.html]
Why do we need to prove the same performance advantage using in-lining for datapath
critical functions again and again?
>
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [EXT] [PATCH v1 4/4] regexdev: implement regex rte level functions
2020-04-06 13:22 0% ` Pavan Nikhilesh Bhagavatula
@ 2020-04-06 13:36 0% ` Thomas Monjalon
2020-04-06 13:50 0% ` Pavan Nikhilesh Bhagavatula
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-04-06 13:36 UTC (permalink / raw)
To: Pavan Nikhilesh Bhagavatula
Cc: Ori Kam, Jerin Jacob Kollanukkaran, xiang.w.wang, dev,
Shahaf Shuler, hemant.agrawal, Opher Reviv, Alex Rosenbaum,
Dovrat Zifroni, Prasun Kapoor, nipun.gupta, bruce.richardson,
yang.a.hong, harry.chang, gu.jian1, shanjiangh, zhangy.yun,
lixingfu, wushuai, yuyingxia, fanchenggang, davidfgao, liuzhong1,
zhaoyong11, oc, jim, hongjun.ni, j.bromhead, deri, fc, arthur.su,
david.marchand
06/04/2020 15:22, Pavan Nikhilesh Bhagavatula:
> >06/04/2020 14:33, Pavan Nikhilesh Bhagavatula:
> >> >> From: Pavan Nikhilesh Bhagavatula
> >> >> > >+uint16_t
> >> >> > >+rte_regexdev_enqueue_burst(uint8_t dev_id, uint16_t
> >qp_id,
> >> >> > >+ struct rte_regex_ops **ops, uint16_t
> >nb_ops)
> >> >> > >+{
> >> >> > >+ return regex_devices[dev_id]-
> >> >> > >>enqueue(regex_devices[dev_id], qp_id,
> >> >> > >+ ops, nb_ops);
> >> >> > >+}
> >> >> >
> >> >> > Move these functions to .h in-lining them.
> >> >> > Also, please add debug checks @see
> >> >rte_eth_rx_burst/rte_eth_tx_burst.
> >> >>
> >> >> O.K will update.
> >> >
> >> >In general, inlining is a pain for ABI compatibility.
> >> >Please inline only if the gain is very significant.
> >> >
> >>
> >> The performance gain mostly comes from hoisting
> >`regex_devices[dev_id]` load above the
> >> poll loop.
> >> Since, the performance measurement application is still in pipeline
> >and regexdev would be
> >> experimental for next couple of releases I suggest inlining it now and
> >worrying about ABI when
> >> experimental tag needs to be removed.
> >
> >No, we must worry about ABI from the beginning.
>
> I though performance was the primary objective :-).
It is.
> >> We can follow the same path as done by ethdev
> >[https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mail-
> >2Darchive.com_dev-
> >40dpdk.org_msg142392.html&d=DwICAg&c=nKjWec2b6R0mOyPaz7xtf
> >Q&r=E3SgYMjtKCMVsB-fmvgGV3o-
> >g_fjLhk5Pupi9ijohpc&m=7Gqb6WKmZV5uY3xa7FRVrRVDz8Usrsd-
> >rDjIKr6CUQQ&s=sQo2Kx9fzTNXwiQ2Fzki3s5GSuiiAEzz2VtN68_KKXo&e=
> >]
> >
> >ethdev is not an argument.
>
> What about ring? [http://mails.dpdk.org/archives/dev/2020-April/161506.html]
>
> Why do we need to prove the same performance advantage using in-lining for datapath
> critical functions again and again?
Because every libraries have not the same usage and load.
We should compare how much cycle is saved with inline vs
how much cycles is, "in average", a regex burst?
If you tell me regex processing is fast, OK, let's inline.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [EXT] [PATCH v1 4/4] regexdev: implement regex rte level functions
2020-04-06 13:36 0% ` Thomas Monjalon
@ 2020-04-06 13:50 0% ` Pavan Nikhilesh Bhagavatula
2020-04-06 14:00 0% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Pavan Nikhilesh Bhagavatula @ 2020-04-06 13:50 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Ori Kam, Jerin Jacob Kollanukkaran, xiang.w.wang, dev,
Shahaf Shuler, hemant.agrawal, Opher Reviv, Alex Rosenbaum,
Dovrat Zifroni, Prasun Kapoor, nipun.gupta, bruce.richardson,
yang.a.hong, harry.chang, gu.jian1, shanjiangh, zhangy.yun,
lixingfu, wushuai, yuyingxia, fanchenggang, davidfgao, liuzhong1,
zhaoyong11, oc, jim, hongjun.ni, j.bromhead, deri, fc, arthur.su,
david.marchand
>-----Original Message-----
>From: Thomas Monjalon <thomas@monjalon.net>
>Sent: Monday, April 6, 2020 7:07 PM
>To: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>
>Cc: Ori Kam <orika@mellanox.com>; Jerin Jacob Kollanukkaran
><jerinj@marvell.com>; xiang.w.wang@intel.com; dev@dpdk.org;
>Shahaf Shuler <shahafs@mellanox.com>; hemant.agrawal@nxp.com;
>Opher Reviv <opher@mellanox.com>; Alex Rosenbaum
><alexr@mellanox.com>; Dovrat Zifroni <dovrat@marvell.com>; Prasun
>Kapoor <pkapoor@marvell.com>; nipun.gupta@nxp.com;
>bruce.richardson@intel.com; yang.a.hong@intel.com;
>harry.chang@intel.com; gu.jian1@zte.com.cn;
>shanjiangh@chinatelecom.cn; zhangy.yun@chinatelecom.cn;
>lixingfu@huachentel.com; wushuai@inspur.com;
>yuyingxia@yxlink.com; fanchenggang@sunyainfo.com;
>davidfgao@tencent.com; liuzhong1@chinaunicom.cn;
>zhaoyong11@huawei.com; oc@yunify.com; jim@netgate.com;
>hongjun.ni@intel.com; j.bromhead@titan-ic.com; deri@ntop.org;
>fc@napatech.com; arthur.su@lionic.com; david.marchand@redhat.com
>Subject: Re: [EXT] [PATCH v1 4/4] regexdev: implement regex rte level
>functions
>
>06/04/2020 15:22, Pavan Nikhilesh Bhagavatula:
>> >06/04/2020 14:33, Pavan Nikhilesh Bhagavatula:
>> >> >> From: Pavan Nikhilesh Bhagavatula
>> >> >> > >+uint16_t
>> >> >> > >+rte_regexdev_enqueue_burst(uint8_t dev_id, uint16_t
>> >qp_id,
>> >> >> > >+ struct rte_regex_ops **ops, uint16_t
>> >nb_ops)
>> >> >> > >+{
>> >> >> > >+ return regex_devices[dev_id]-
>> >> >> > >>enqueue(regex_devices[dev_id], qp_id,
>> >> >> > >+ ops, nb_ops);
>> >> >> > >+}
>> >> >> >
>> >> >> > Move these functions to .h in-lining them.
>> >> >> > Also, please add debug checks @see
>> >> >rte_eth_rx_burst/rte_eth_tx_burst.
>> >> >>
>> >> >> O.K will update.
>> >> >
>> >> >In general, inlining is a pain for ABI compatibility.
>> >> >Please inline only if the gain is very significant.
>> >> >
>> >>
>> >> The performance gain mostly comes from hoisting
>> >`regex_devices[dev_id]` load above the
>> >> poll loop.
>> >> Since, the performance measurement application is still in pipeline
>> >and regexdev would be
>> >> experimental for next couple of releases I suggest inlining it now
>and
>> >worrying about ABI when
>> >> experimental tag needs to be removed.
>> >
>> >No, we must worry about ABI from the beginning.
>>
>> I though performance was the primary objective :-).
>
>It is.
>
>> >> We can follow the same path as done by ethdev
>> >[https://urldefense.proofpoint.com/v2/url?u=https-
>3A__www.mail-
>> >2Darchive.com_dev-
>>
>>40dpdk.org_msg142392.html&d=DwICAg&c=nKjWec2b6R0mOyPaz7xt
>f
>> >Q&r=E3SgYMjtKCMVsB-fmvgGV3o-
>> >g_fjLhk5Pupi9ijohpc&m=7Gqb6WKmZV5uY3xa7FRVrRVDz8Usrsd-
>>
>>rDjIKr6CUQQ&s=sQo2Kx9fzTNXwiQ2Fzki3s5GSuiiAEzz2VtN68_KKXo&e
>=
>> >]
>> >
>> >ethdev is not an argument.
>>
>> What about ring? [https://urldefense.proofpoint.com/v2/url?u=http-
>3A__mails.dpdk.org_archives_dev_2020-
>2DApril_161506.html&d=DwICAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=E3S
>gYMjtKCMVsB-fmvgGV3o-
>g_fjLhk5Pupi9ijohpc&m=u9gnM_YrOJDusN4yR8YUcUuUkri4tOjnHrQ0A
>Qd5zTw&s=uv6AQA-
>Zu7o6ugyhpGHLxFOk4SfEdkHfFGDmhzANRME&e= ]
>>
>> Why do we need to prove the same performance advantage using in-
>lining for datapath
>> critical functions again and again?
>
>Because every libraries have not the same usage and load.
>We should compare how much cycle is saved with inline vs
>how much cycles is, "in average", a regex burst?
>
>If you tell me regex processing is fast, OK, let's inline.
>
Regex processing speed would still be dependent on underlying device capabilities.
All we are trying to do is reduce the enqueue/dequeue completion time which would
bring down the overall latency.
Thanks,
Pavan.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [EXT] [PATCH v1 4/4] regexdev: implement regex rte level functions
2020-04-06 13:50 0% ` Pavan Nikhilesh Bhagavatula
@ 2020-04-06 14:00 0% ` Thomas Monjalon
2020-04-06 18:53 0% ` Ori Kam
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-04-06 14:00 UTC (permalink / raw)
To: Pavan Nikhilesh Bhagavatula, Ori Kam
Cc: Jerin Jacob Kollanukkaran, xiang.w.wang, dev, Shahaf Shuler,
hemant.agrawal, Opher Reviv, Alex Rosenbaum, Dovrat Zifroni,
Prasun Kapoor, nipun.gupta, bruce.richardson, yang.a.hong,
harry.chang, gu.jian1, shanjiangh, zhangy.yun, lixingfu, wushuai,
yuyingxia, fanchenggang, davidfgao, liuzhong1, zhaoyong11, oc,
jim, hongjun.ni, j.bromhead, deri, fc, arthur.su, david.marchand
06/04/2020 15:50, Pavan Nikhilesh Bhagavatula:
> From: Thomas Monjalon <thomas@monjalon.net>
> >> >06/04/2020 14:33, Pavan Nikhilesh Bhagavatula:
> >> >> >> From: Pavan Nikhilesh Bhagavatula
> >> >> >> > >+uint16_t
> >> >> >> > >+rte_regexdev_enqueue_burst(uint8_t dev_id, uint16_t
> >> >qp_id,
> >> >> >> > >+ struct rte_regex_ops **ops, uint16_t
> >> >nb_ops)
> >> >> >> > >+{
> >> >> >> > >+ return regex_devices[dev_id]-
> >> >> >> > >>enqueue(regex_devices[dev_id], qp_id,
> >> >> >> > >+ ops, nb_ops);
> >> >> >> > >+}
> >> >> >> >
> >> >> >> > Move these functions to .h in-lining them.
> >> >> >> > Also, please add debug checks @see
> >> >> >rte_eth_rx_burst/rte_eth_tx_burst.
> >> >> >>
> >> >> >> O.K will update.
> >> >> >
> >> >> >In general, inlining is a pain for ABI compatibility.
> >> >> >Please inline only if the gain is very significant.
> >> >> >
> >> >>
> >> >> The performance gain mostly comes from hoisting
> >> >`regex_devices[dev_id]` load above the
> >> >> poll loop.
> >> >> Since, the performance measurement application is still in pipeline
> >> >and regexdev would be
> >> >> experimental for next couple of releases I suggest inlining it now
> >and
> >> >worrying about ABI when
> >> >> experimental tag needs to be removed.
> >> >
> >> >No, we must worry about ABI from the beginning.
> >>
> >> I though performance was the primary objective :-).
> >
> >It is.
> >
> >> >> We can follow the same path as done by ethdev
> >> >[https://urldefense.proofpoint.com/v2/url?u=https-
> >3A__www.mail-
> >> >2Darchive.com_dev-
> >>
> >>40dpdk.org_msg142392.html&d=DwICAg&c=nKjWec2b6R0mOyPaz7xt
> >f
> >> >Q&r=E3SgYMjtKCMVsB-fmvgGV3o-
> >> >g_fjLhk5Pupi9ijohpc&m=7Gqb6WKmZV5uY3xa7FRVrRVDz8Usrsd-
> >>
> >>rDjIKr6CUQQ&s=sQo2Kx9fzTNXwiQ2Fzki3s5GSuiiAEzz2VtN68_KKXo&e
> >=
> >> >]
> >> >
> >> >ethdev is not an argument.
> >>
> >> What about ring? [https://urldefense.proofpoint.com/v2/url?u=http-
> >3A__mails.dpdk.org_archives_dev_2020-
> >2DApril_161506.html&d=DwICAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=E3S
> >gYMjtKCMVsB-fmvgGV3o-
> >g_fjLhk5Pupi9ijohpc&m=u9gnM_YrOJDusN4yR8YUcUuUkri4tOjnHrQ0A
> >Qd5zTw&s=uv6AQA-
> >Zu7o6ugyhpGHLxFOk4SfEdkHfFGDmhzANRME&e= ]
> >>
> >> Why do we need to prove the same performance advantage using in-
> >lining for datapath
> >> critical functions again and again?
> >
> >Because every libraries have not the same usage and load.
> >We should compare how much cycle is saved with inline vs
> >how much cycles is, "in average", a regex burst?
> >
> >If you tell me regex processing is fast, OK, let's inline.
> >
>
> Regex processing speed would still be dependent on underlying device capabilities.
>
> All we are trying to do is reduce the enqueue/dequeue completion time which would
> bring down the overall latency.
Take your regex HW and do a simple regex processing burst.
How many cycles it takes to complete?
How many cycles you lose if not inline?
If the ratio is lower than 1/200, I think inline is not a must.
Ori, please consider the same measure for your HW.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] /validate-abi.sh complains [PATCH v1 3/8] ring: introduce RTS ring mode
2020-04-01 12:52 4% ` Neil Horman
@ 2020-04-06 14:02 4% ` Thomas Monjalon
2020-04-06 14:36 4% ` Neil Horman
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-04-06 14:02 UTC (permalink / raw)
To: Neil Horman
Cc: dev, mdr, Ananyev, Konstantin, david.marchand,
honnappa.nagarahalli, dev, Richardson, Bruce, Yigit, Ferruh,
kevin.laatz
01/04/2020 14:52, Neil Horman:
> On Tue, Mar 31, 2020 at 09:24:41PM +0200, Thomas Monjalon wrote:
> > 31/03/2020 21:22, Thomas Monjalon:
> > > 31/03/2020 19:05, Ananyev, Konstantin:
> > > > Hi everyone,
> > > >
> > > > Have a question regarding validate-abi.sh.
> > >
> > > devtools/validate-abi.sh should be removed.
> > > Please use the new devtools/check-abi.sh
> >
> > The file doc/guides/contributing/abi_versioning.rst
> > should be updated as well.
> > Neil? Ray?
> >
> Ack, we should remove validate_abi and update the docs
Neil, are you going to do this update?
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] /validate-abi.sh complains [PATCH v1 3/8] ring: introduce RTS ring mode
2020-04-06 14:02 4% ` Thomas Monjalon
@ 2020-04-06 14:36 4% ` Neil Horman
2020-04-07 7:32 4% ` Ray Kinsella
0 siblings, 1 reply; 200+ results
From: Neil Horman @ 2020-04-06 14:36 UTC (permalink / raw)
To: Thomas Monjalon
Cc: dev, mdr, Ananyev, Konstantin, david.marchand,
honnappa.nagarahalli, Richardson, Bruce, Yigit, Ferruh,
kevin.laatz
On Mon, Apr 06, 2020 at 04:02:22PM +0200, Thomas Monjalon wrote:
> 01/04/2020 14:52, Neil Horman:
> > On Tue, Mar 31, 2020 at 09:24:41PM +0200, Thomas Monjalon wrote:
> > > 31/03/2020 21:22, Thomas Monjalon:
> > > > 31/03/2020 19:05, Ananyev, Konstantin:
> > > > > Hi everyone,
> > > > >
> > > > > Have a question regarding validate-abi.sh.
> > > >
> > > > devtools/validate-abi.sh should be removed.
> > > > Please use the new devtools/check-abi.sh
> > >
> > > The file doc/guides/contributing/abi_versioning.rst
> > > should be updated as well.
> > > Neil? Ray?
> > >
> > Ack, we should remove validate_abi and update the docs
>
> Neil, are you going to do this update?
I can, I had though it was already in progress though. I'll take care of it
this week.
Best
Neil
>
>
>
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [EXT] [PATCH v1 4/4] regexdev: implement regex rte level functions
2020-04-06 14:00 0% ` Thomas Monjalon
@ 2020-04-06 18:53 0% ` Ori Kam
0 siblings, 0 replies; 200+ results
From: Ori Kam @ 2020-04-06 18:53 UTC (permalink / raw)
To: Thomas Monjalon, Pavan Nikhilesh Bhagavatula
Cc: Jerin Jacob Kollanukkaran, xiang.w.wang, dev, Shahaf Shuler,
hemant.agrawal, Opher Reviv, Alex Rosenbaum, Dovrat Zifroni,
Prasun Kapoor, nipun.gupta, bruce.richardson, yang.a.hong,
harry.chang, gu.jian1, shanjiangh, zhangy.yun, lixingfu, wushuai,
yuyingxia, fanchenggang, davidfgao, liuzhong1, zhaoyong11, oc,
jim, hongjun.ni, j.bromhead, deri, fc, arthur.su, david.marchand
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Thomas Monjalon
> Sent: Monday, April 6, 2020 5:00 PM
> To: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>; Ori Kam
> <orika@mellanox.com>
> Cc: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; xiang.w.wang@intel.com;
> dev@dpdk.org; Shahaf Shuler <shahafs@mellanox.com>;
> hemant.agrawal@nxp.com; Opher Reviv <opher@mellanox.com>; Alex
> Rosenbaum <alexr@mellanox.com>; Dovrat Zifroni <dovrat@marvell.com>;
> Prasun Kapoor <pkapoor@marvell.com>; nipun.gupta@nxp.com;
> bruce.richardson@intel.com; yang.a.hong@intel.com; harry.chang@intel.com;
> gu.jian1@zte.com.cn; shanjiangh@chinatelecom.cn;
> zhangy.yun@chinatelecom.cn; lixingfu@huachentel.com; wushuai@inspur.com;
> yuyingxia@yxlink.com; fanchenggang@sunyainfo.com;
> davidfgao@tencent.com; liuzhong1@chinaunicom.cn;
> zhaoyong11@huawei.com; oc@yunify.com; jim@netgate.com;
> hongjun.ni@intel.com; j.bromhead@titan-ic.com; deri@ntop.org;
> fc@napatech.com; arthur.su@lionic.com; david.marchand@redhat.com
> Subject: Re: [dpdk-dev] [EXT] [PATCH v1 4/4] regexdev: implement regex rte
> level functions
>
> 06/04/2020 15:50, Pavan Nikhilesh Bhagavatula:
> > From: Thomas Monjalon <thomas@monjalon.net>
> > >> >06/04/2020 14:33, Pavan Nikhilesh Bhagavatula:
> > >> >> >> From: Pavan Nikhilesh Bhagavatula
> > >> >> >> > >+uint16_t
> > >> >> >> > >+rte_regexdev_enqueue_burst(uint8_t dev_id, uint16_t
> > >> >qp_id,
> > >> >> >> > >+ struct rte_regex_ops **ops, uint16_t
> > >> >nb_ops)
> > >> >> >> > >+{
> > >> >> >> > >+ return regex_devices[dev_id]-
> > >> >> >> > >>enqueue(regex_devices[dev_id], qp_id,
> > >> >> >> > >+ ops, nb_ops);
> > >> >> >> > >+}
> > >> >> >> >
> > >> >> >> > Move these functions to .h in-lining them.
> > >> >> >> > Also, please add debug checks @see
> > >> >> >rte_eth_rx_burst/rte_eth_tx_burst.
> > >> >> >>
> > >> >> >> O.K will update.
> > >> >> >
> > >> >> >In general, inlining is a pain for ABI compatibility.
> > >> >> >Please inline only if the gain is very significant.
> > >> >> >
> > >> >>
> > >> >> The performance gain mostly comes from hoisting
> > >> >`regex_devices[dev_id]` load above the
> > >> >> poll loop.
> > >> >> Since, the performance measurement application is still in pipeline
> > >> >and regexdev would be
> > >> >> experimental for next couple of releases I suggest inlining it now
> > >and
> > >> >worrying about ABI when
> > >> >> experimental tag needs to be removed.
> > >> >
> > >> >No, we must worry about ABI from the beginning.
> > >>
> > >> I though performance was the primary objective :-).
> > >
> > >It is.
> > >
> > >> >> We can follow the same path as done by ethdev
> > >>
> >[https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefe
> nse.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-
> &data=02%7C01%7Corika%40mellanox.com%7Cc3e07910f52846f64b5008
> d7da32d82e%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C6372177
> 84252928490&sdata=8Pxq8ciUVW7bMiB1XmQBNm%2Fsmy8m1Wa7yKXK
> JRL4d%2B4%3D&reserved=0
> >
> >3A__https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fww
> w.mail-
> %2F&data=02%7C01%7Corika%40mellanox.com%7Cc3e07910f52846f64b
> 5008d7da32d82e%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C637
> 217784252928490&sdata=sTDsN85xShTU7WOLQQ2iF8eh%2FyoVHYwDq%
> 2FTdgdHdlbM%3D&reserved=0
> > >> >2Darchive.com_dev-
> > >>
> > >>40dpdk.org_msg142392.html&d=DwICAg&c=nKjWec2b6R0mOyPaz7xt
> > >f
> > >> >Q&r=E3SgYMjtKCMVsB-fmvgGV3o-
> > >> >g_fjLhk5Pupi9ijohpc&m=7Gqb6WKmZV5uY3xa7FRVrRVDz8Usrsd-
> > >>
> > >>rDjIKr6CUQQ&s=sQo2Kx9fzTNXwiQ2Fzki3s5GSuiiAEzz2VtN68_KKXo&e
> > >=
> > >> >]
> > >> >
> > >> >ethdev is not an argument.
> > >>
> > >> What about ring?
> [https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefens
> e.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-
> &data=02%7C01%7Corika%40mellanox.com%7Cc3e07910f52846f64b5008
> d7da32d82e%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C6372177
> 84252928490&sdata=t2qUIj51WPzqQhJngheYh1s7DWzOxd%2FcAyAQK39C
> HgQ%3D&reserved=0
> > >3A__mails.dpdk.org_archives_dev_2020-
> > >2DApril_161506.html&d=DwICAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=E3S
> > >gYMjtKCMVsB-fmvgGV3o-
> > >g_fjLhk5Pupi9ijohpc&m=u9gnM_YrOJDusN4yR8YUcUuUkri4tOjnHrQ0A
> > >Qd5zTw&s=uv6AQA-
> > >Zu7o6ugyhpGHLxFOk4SfEdkHfFGDmhzANRME&e= ]
> > >>
> > >> Why do we need to prove the same performance advantage using in-
> > >lining for datapath
> > >> critical functions again and again?
> > >
> > >Because every libraries have not the same usage and load.
> > >We should compare how much cycle is saved with inline vs
> > >how much cycles is, "in average", a regex burst?
> > >
> > >If you tell me regex processing is fast, OK, let's inline.
> > >
> >
> > Regex processing speed would still be dependent on underlying device
> capabilities.
> >
> > All we are trying to do is reduce the enqueue/dequeue completion time
> which would
> > bring down the overall latency.
>
> Take your regex HW and do a simple regex processing burst.
> How many cycles it takes to complete?
> How many cycles you lose if not inline?
> If the ratio is lower than 1/200, I think inline is not a must.
>
> Ori, please consider the same measure for your HW.
>
I think that the default for data path should be inline.
In this specific case maybe the benefit is small.
I guess that by definition the RegEx is slower than the net device.
It is much harder to check the performance of RegEx device due to the
fact that it depends on the data itself and on the rules.
I don't have HW to check the numbers right now. Even if I have them
this is for Mellanox implementation other PMD may have different values.
I suggest that we change to inline, and after getting the HW we will run some tests
and based on them we can decide if the API should be changed.
Please note that the API change will be before the first integration of this patch
so it is not real API change.
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] Remove abi_versioning.sh from tree
@ 2020-04-06 19:34 38% Neil Horman
2020-04-07 7:36 4% ` David Marchand
2020-04-08 14:49 0% ` Ray Kinsella
0 siblings, 2 replies; 200+ results
From: Neil Horman @ 2020-04-06 19:34 UTC (permalink / raw)
To: dev
Since we've moved away from our initial abi_versioning.sh script, in
favor of check_abi.sh, which uses libabigail, remove the old script from
the tree, and update the docs accordingly
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
CC: thomas@monjalon.net
---
MAINTAINERS | 1 -
devtools/validate-abi.sh | 251 ---------------------
doc/guides/contributing/abi_versioning.rst | 18 +-
3 files changed, 9 insertions(+), 261 deletions(-)
delete mode 100755 devtools/validate-abi.sh
diff --git a/MAINTAINERS b/MAINTAINERS
index 4800f6884..84b633431 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -152,7 +152,6 @@ F: devtools/gen-abi.sh
F: devtools/libabigail.abignore
F: devtools/update-abi.sh
F: devtools/update_version_map_abi.py
-F: devtools/validate-abi.sh
F: buildtools/check-experimental-syms.sh
F: buildtools/map-list-symbol.sh
diff --git a/devtools/validate-abi.sh b/devtools/validate-abi.sh
deleted file mode 100755
index f64e19d38..000000000
--- a/devtools/validate-abi.sh
+++ /dev/null
@@ -1,251 +0,0 @@
-#!/usr/bin/env bash
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2015 Neil Horman. All rights reserved.
-# Copyright(c) 2017 6WIND S.A.
-# All rights reserved
-
-set -e
-
-abicheck=abi-compliance-checker
-abidump=abi-dumper
-default_dst=abi-check
-default_target=x86_64-native-linuxapp-gcc
-
-# trap on error
-err_report() {
- echo "$0: error at line $1"
-}
-trap 'err_report $LINENO' ERR
-
-print_usage () {
- cat <<- END_OF_HELP
- $(basename $0) [options] <rev1> <rev2>
-
- This script compares the ABI of 2 git revisions of the current
- workspace. The output is a html report and a compilation log.
-
- The objective is to make sure that applications built against
- DSOs from the first revision can still run when executed using
- the DSOs built from the second revision.
-
- <rev1> and <rev2> are git commit id or tags.
-
- Options:
- -h show this help
- -j <num> enable parallel compilation with <num> threads
- -v show compilation logs on the console
- -d <dir> change working directory (default is ${default_dst})
- -t <target> the dpdk target to use (default is ${default_target})
- -f overwrite existing files in destination directory
-
- The script returns 0 on success, or the value of last failing
- call of ${abicheck} (incompatible abi or the tool has run with errors).
- The errors returned by ${abidump} are ignored.
-
- END_OF_HELP
-}
-
-# log in the file, and on stdout if verbose
-# $1: level string
-# $2: string to be logged
-log() {
- echo "$1: $2"
- if [ "${verbose}" != "true" ]; then
- echo "$1: $2" >&3
- fi
-}
-
-# launch a command and log it, taking care of surrounding spaces with quotes
-cmd() {
- local i s whitespace ret
- s=""
- whitespace="[[:space:]]"
- for i in "$@"; do
- if [[ $i =~ $whitespace ]]; then
- i=\"$i\"
- fi
- if [ -z "$s" ]; then
- s="$i"
- else
- s="$s $i"
- fi
- done
-
- ret=0
- log "CMD" "$s"
- "$@" || ret=$?
- if [ "$ret" != "0" ]; then
- log "CMD" "previous command returned $ret"
- fi
-
- return $ret
-}
-
-# redirect or copy stderr/stdout to a file
-# the syntax is unfamiliar, but it makes the rest of the
-# code easier to read, avoiding the use of pipes
-set_log_file() {
- # save original stdout and stderr in fd 3 and 4
- exec 3>&1
- exec 4>&2
- # create a new fd 5 that send to a file
- exec 5> >(cat > $1)
- # send stdout and stderr to fd 5
- if [ "${verbose}" = "true" ]; then
- exec 1> >(tee /dev/fd/5 >&3)
- exec 2> >(tee /dev/fd/5 >&4)
- else
- exec 1>&5
- exec 2>&5
- fi
-}
-
-# Make sure we configure SHARED libraries
-# Also turn off IGB and KNI as those require kernel headers to build
-fixup_config() {
- local conf=config/defconfig_$target
- cmd sed -i -e"$ a\CONFIG_RTE_BUILD_SHARED_LIB=y" $conf
- cmd sed -i -e"$ a\CONFIG_RTE_NEXT_ABI=n" $conf
- cmd sed -i -e"$ a\CONFIG_RTE_EAL_IGB_UIO=n" $conf
- cmd sed -i -e"$ a\CONFIG_RTE_LIBRTE_KNI=n" $conf
- cmd sed -i -e"$ a\CONFIG_RTE_KNI_KMOD=n" $conf
-}
-
-# build dpdk for the given tag and dump abi
-# $1: hash of the revision
-gen_abi() {
- local i
-
- cmd git clone ${dpdkroot} ${dst}/${1}
- cmd cd ${dst}/${1}
-
- log "INFO" "Checking out version ${1} of the dpdk"
- # Move to the old version of the tree
- cmd git checkout ${1}
-
- fixup_config
-
- # Now configure the build
- log "INFO" "Configuring DPDK ${1}"
- cmd make config T=$target O=$target
-
- # Checking abi compliance relies on using the dwarf information in
- # the shared objects. Build with -g to include them.
- log "INFO" "Building DPDK ${1}. This might take a moment"
- cmd make -j$parallel O=$target V=1 EXTRA_CFLAGS="-g -Og -Wno-error" \
- EXTRA_LDFLAGS="-g" || log "INFO" "The build failed"
-
- # Move to the lib directory
- cmd cd ${PWD}/$target/lib
- log "INFO" "Collecting ABI information for ${1}"
- for i in *.so; do
- [ -e "$i" ] || break
- cmd $abidump ${i} -o $dst/${1}/${i}.dump -lver ${1} || true
- # hack to ignore empty SymbolsInfo section (no public ABI)
- if grep -q "'SymbolInfo' => {}," $dst/${1}/${i}.dump \
- 2> /dev/null; then
- log "INFO" "${i} has no public ABI, remove dump file"
- cmd rm -f $dst/${1}/${i}.dump
- fi
- done
-}
-
-verbose=false
-parallel=1
-dst=${default_dst}
-target=${default_target}
-force=0
-while getopts j:vd:t:fh ARG ; do
- case $ARG in
- j ) parallel=$OPTARG ;;
- v ) verbose=true ;;
- d ) dst=$OPTARG ;;
- t ) target=$OPTARG ;;
- f ) force=1 ;;
- h ) print_usage ; exit 0 ;;
- ? ) print_usage ; exit 1 ;;
- esac
-done
-shift $(($OPTIND - 1))
-
-if [ $# != 2 ]; then
- print_usage
- exit 1
-fi
-
-tag1=$1
-tag2=$2
-
-# convert path to absolute
-case "${dst}" in
- /*) ;;
- *) dst=${PWD}/${dst} ;;
-esac
-dpdkroot=$(readlink -f $(dirname $0)/..)
-
-if [ -e "${dst}" -a "$force" = 0 ]; then
- echo "The ${dst} directory is not empty. Remove it, use another"
- echo "one (-d <dir>), or force overriding (-f)"
- exit 1
-fi
-
-rm -rf ${dst}
-mkdir -p ${dst}
-set_log_file ${dst}/abi-check.log
-log "INFO" "Logs available in ${dst}/abi-check.log"
-
-command -v ${abicheck} || log "INFO" "Can't find ${abicheck} utility"
-command -v ${abidump} || log "INFO" "Can't find ${abidump} utility"
-
-hash1=$(git show -s --format=%h "$tag1" -- 2> /dev/null | tail -1)
-hash2=$(git show -s --format=%h "$tag2" -- 2> /dev/null | tail -1)
-
-# Make hashes available in output for non-local reference
-tag1="$tag1 ($hash1)"
-tag2="$tag2 ($hash2)"
-
-if [ "$hash1" = "$hash2" ]; then
- log "ERROR" "$tag1 and $tag2 are the same revisions"
- exit 1
-fi
-
-cmd mkdir -p ${dst}
-
-# dump abi for each revision
-gen_abi ${hash1}
-gen_abi ${hash2}
-
-# compare the abi dumps
-cmd cd ${dst}
-ret=0
-list=""
-for i in ${hash2}/*.dump; do
- name=`basename $i`
- libname=${name%.dump}
-
- if [ ! -f ${hash1}/$name ]; then
- log "INFO" "$NAME does not exist in $tag1. skipping..."
- continue
- fi
-
- local_ret=0
- cmd $abicheck -l $libname \
- -old ${hash1}/$name -new ${hash2}/$name || local_ret=$?
- if [ $local_ret != 0 ]; then
- log "NOTICE" "$abicheck returned $local_ret"
- ret=$local_ret
- list="$list $libname"
- fi
-done
-
-if [ $ret != 0 ]; then
- log "NOTICE" "ABI may be incompatible, check reports/logs for details."
- log "NOTICE" "Incompatible list: $list"
-else
- log "NOTICE" "No error detected, ABI is compatible."
-fi
-
-log "INFO" "Logs are in ${dst}/abi-check.log"
-log "INFO" "HTML reports are in ${dst}/compat_reports directory"
-
-exit $ret
diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst
index a21f4e7a4..1c4a3f927 100644
--- a/doc/guides/contributing/abi_versioning.rst
+++ b/doc/guides/contributing/abi_versioning.rst
@@ -482,9 +482,9 @@ Running the ABI Validator
-------------------------
The ``devtools`` directory in the DPDK source tree contains a utility program,
-``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI
-Compliance Checker
-<http://ispras.linuxbase.org/index.php/ABI_compliance_checker>`_.
+``check-abi.sh``, for validating the DPDK ABI based on the libabigail abidiff
+utility:
+https://sourceware.org/libabigail/manual/abidiff.html
This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper``
utilities which can be installed via a package manager. For example::
@@ -492,9 +492,9 @@ utilities which can be installed via a package manager. For example::
sudo yum install abi-compliance-checker
sudo yum install abi-dumper
-The syntax of the ``validate-abi.sh`` utility is::
+The syntax of the ``check-abi.sh`` utility is::
- ./devtools/validate-abi.sh <REV1> <REV2>
+ ./devtools/check-abi.sh <REV1> <REV2>
Where ``REV1`` and ``REV2`` are valid gitrevisions(7)
https://www.kernel.org/pub/software/scm/git/docs/gitrevisions.html
@@ -503,16 +503,16 @@ on the local repo.
For example::
# Check between the previous and latest commit:
- ./devtools/validate-abi.sh HEAD~1 HEAD
+ ./devtools/check-abi.sh HEAD~1 HEAD
# Check on a specific compilation target:
- ./devtools/validate-abi.sh -t x86_64-native-linux-gcc HEAD~1 HEAD
+ ./devtools/check-abi.sh -t x86_64-native-linux-gcc HEAD~1 HEAD
# Check between two tags:
- ./devtools/validate-abi.sh v2.0.0 v2.1.0
+ ./devtools/check-abi.sh v2.0.0 v2.1.0
# Check between git master and local topic-branch "vhost-hacking":
- ./devtools/validate-abi.sh master vhost-hacking
+ ./devtools/check-abi.sh master vhost-hacking
After the validation script completes (it can take a while since it need to
compile both tags) it will create compatibility reports in the
--
2.25.1
^ permalink raw reply [relevance 38%]
* Re: [dpdk-dev] /validate-abi.sh complains [PATCH v1 3/8] ring: introduce RTS ring mode
2020-04-06 14:36 4% ` Neil Horman
@ 2020-04-07 7:32 4% ` Ray Kinsella
2020-04-07 11:31 4% ` Neil Horman
0 siblings, 1 reply; 200+ results
From: Ray Kinsella @ 2020-04-07 7:32 UTC (permalink / raw)
To: Neil Horman, Thomas Monjalon
Cc: dev, Ananyev, Konstantin, david.marchand, honnappa.nagarahalli,
Richardson, Bruce, Yigit, Ferruh, kevin.laatz
On 06/04/2020 15:36, Neil Horman wrote:
> On Mon, Apr 06, 2020 at 04:02:22PM +0200, Thomas Monjalon wrote:
>> 01/04/2020 14:52, Neil Horman:
>>> On Tue, Mar 31, 2020 at 09:24:41PM +0200, Thomas Monjalon wrote:
>>>> 31/03/2020 21:22, Thomas Monjalon:
>>>>> 31/03/2020 19:05, Ananyev, Konstantin:
>>>>>> Hi everyone,
>>>>>>
>>>>>> Have a question regarding validate-abi.sh.
>>>>>
>>>>> devtools/validate-abi.sh should be removed.
>>>>> Please use the new devtools/check-abi.sh
>>>>
>>>> The file doc/guides/contributing/abi_versioning.rst
>>>> should be updated as well.
>>>> Neil? Ray?
>>>>
>>> Ack, we should remove validate_abi and update the docs
>>
>> Neil, are you going to do this update?
> I can, I had though it was already in progress though. I'll take care of it
> this week.
Thanks Neil - don't know how I missed this thread.
>
> Best
> Neil
>
>>
>>
>>
>>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] Remove abi_versioning.sh from tree
2020-04-06 19:34 38% [dpdk-dev] [PATCH] Remove abi_versioning.sh from tree Neil Horman
@ 2020-04-07 7:36 4% ` David Marchand
2020-04-07 11:33 0% ` Neil Horman
2020-04-08 14:50 0% ` Ray Kinsella
2020-04-08 14:49 0% ` Ray Kinsella
1 sibling, 2 replies; 200+ results
From: David Marchand @ 2020-04-07 7:36 UTC (permalink / raw)
To: Neil Horman; +Cc: dev, Thomas Monjalon, Kevin Laatz
On Mon, Apr 6, 2020 at 9:34 PM Neil Horman <nhorman@tuxdriver.com> wrote:
>
> Since we've moved away from our initial abi_versioning.sh script, in
abi_versioning.sh does not exist (idem with the patch title).
I suppose you meant validate-abi.sh.
> favor of check_abi.sh, which uses libabigail, remove the old script from
check-abi.sh
> the tree, and update the docs accordingly
>
[snip]
> diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst
> index a21f4e7a4..1c4a3f927 100644
> --- a/doc/guides/contributing/abi_versioning.rst
> +++ b/doc/guides/contributing/abi_versioning.rst
> @@ -482,9 +482,9 @@ Running the ABI Validator
> -------------------------
>
> The ``devtools`` directory in the DPDK source tree contains a utility program,
> -``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI
> -Compliance Checker
> -<http://ispras.linuxbase.org/index.php/ABI_compliance_checker>`_.
> +``check-abi.sh``, for validating the DPDK ABI based on the libabigail abidiff
> +utility:
> +https://sourceware.org/libabigail/manual/abidiff.html
>
> This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper``
> utilities which can be installed via a package manager. For example::
> @@ -492,9 +492,9 @@ utilities which can be installed via a package manager. For example::
> sudo yum install abi-compliance-checker
> sudo yum install abi-dumper
>
> -The syntax of the ``validate-abi.sh`` utility is::
> +The syntax of the ``check-abi.sh`` utility is::
>
> - ./devtools/validate-abi.sh <REV1> <REV2>
> + ./devtools/check-abi.sh <REV1> <REV2>
The new script is not a direct replacement.
It won't take git revisions, but build directories where versions of
dpdk have been compiled.
devtools/test-build.sh and devtools/test-meson-builds.sh illustrate its use.
https://git.dpdk.org/dpdk/tree/devtools/test-meson-builds.sh#n127
--
David Marchand
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH 00/16] NXP DPAAx fixes and enhancements
@ 2020-04-07 10:25 3% ` Hemant Agrawal
2020-04-07 12:20 0% ` Thomas Monjalon
2020-04-08 7:20 4% ` Dodji Seketeli
0 siblings, 2 replies; 200+ results
From: Hemant Agrawal @ 2020-04-07 10:25 UTC (permalink / raw)
To: Dodji Seketeli, David Marchand
Cc: Hemant Agrawal (OSS), Yigit, Ferruh, dev, Neil Horman, Thomas Monjalon
HI Dodji,
>
> David Marchand <david.marchand@redhat.com> writes:
>
> > On Thu, Mar 5, 2020 at 10:19 AM Hemant Agrawal (OSS)
> > <hemant.agrawal@oss.nxp.com> wrote:
> >> > On Thu, Mar 5, 2020 at 10:06 AM Hemant Agrawal (OSS)
> >> > <hemant.agrawal@oss.nxp.com> wrote:
> >> > >
> >> > > Hi David,
> >> > > > On Mon, Mar 2, 2020 at 10:26 AM Hemant Agrawal
> >> > > > <hemant.agrawal@nxp.com> wrote:
> >> > > > >
> >> > > > > This patch series add various patches for enhancing and
> >> > > > > fixing NXP fslmc bus, dpaa bus, and dpaax.
> >> > > > >
> >> > > > > - the main change is support to allow thread migration across
> >> > > > > lcores
> >> > > > > - improving the multi-process support
> >> > > >
> >> > > > This series triggers an ABI warning that must be investigated.
> >> > > >https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%
> >> > > >2Ftravis-
> ci.com%2Fovsrobot%2Fdpdk%2Fjobs%2F292904119%23L2233&
> >> > >
> >;data=02%7C01%7Chemant.agrawal%40nxp.com%7C91a3230cfd334c28bbd
> b0
> >> > >
> >8d7c4dee590%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6371
> 943
> >> > >
> >33920176015&sdata=%2BViKwS2sNucwLFD9VtvwxOK1huq0g%2B6TfT6
> Fqp
> >> > > >Nyz5w%3D&reserved=0
> >> > >
> >> > > [Hemant]
> >> > > As per the logs:
> >> > >
> >> > > Variables changes summary: 1 Removed, 2 Changed, 0 Added
> >> > > variables
> >> > > 1 Removed variable:
> >> > > 'dpaa2_portal_dqrr per_lcore_dpaa2_held_bufs'
> >> > {per_lcore_dpaa2_held_bufs@@DPDK_20.0}
> >> > > 2 Changed variables:
> >> > > [C]'dpaa2_io_portal_t dpaa2_io_portal[128]' was changed at
> >> > dpaa2_hw_dpio.h:40:1: size of symbol changed from 5120 to 2048
> >> > > [C]'dpaa2_io_portal_t per_lcore__dpaa2_io' was changed at
> >> > > dpaa2_hw_dpio.h:20:1: size of symbol changed from 40 to 16
> >> > >
> >> > > Error: ABI issue reported for 'abidiff --suppr
> >> > > devtools/libabigail.abignore --
> >> > no-added-syms --headers-dir1 reference/usr/local/include
> >> > --headers-dir2 install/usr/local/include
> >> > reference/dump/librte_bus_fslmc.dump
> >> > install/dump/librte_bus_fslmc.dump'
> >> > >
> >> > > ---------------
> >> > >
> >> > > These changes are w.r.t modifications in internal structures and
> variables.
> >> > They may be ignored.
> >> >
> >> > The ABI check considers symbol exposed in headers available to final
> users.
> >> > If those are internal, why are the headers public?
> >> >
> >>
> >> [Hemant] These symbols are not part of any public header files, but
> >> they are part of *.map files to share them between different driver
> >> libs i.e bus_fslmc and net_dpaa2
> >
> > I would expect libabigail to skip those symbols, so there is something
> > I have missed in how --headers-dirX work.
>
> In libabigail speak, we make a difference between *ELF symbols* and types.
>
> --header-dirX is about telling the tool what the public *types* are. As you
> rightfully implied, types that are defined in files that are not found in the
> directories specified by --header-dirX are considered to be private types and
> are thus not shown in the ABI change report.
>
> ELF symbols however are a different matter. Header files don't usually define
> ELF symbols, be they variable of function symbols. Header files can at most
> declare variables or functions that would be actually defined elsewhere in
> source code, leading to the definition of ELF variable or function symbols in the
> final binary. At this point, we aren't talking about types anymore, as the ELF
> format doesn't know what types (in C or any other language) are. So --header-
> dirX don't deal with ELF symbols.
>
> And from what I understand from the message quoted above, the changes we
> are talking about have to do with EFL variable symbols which size have
> changed. So in practise, these are global arrays (exposed by at the binary level
> as an ELF variable symbol of a given size) with public visibility which size have
> changed.
>
> So my guess would be that if you guys don't want these arrays to be part the
> binary interface of this library, they should probably be declared static at the C
> level and accessed through some accessor function or something like that. At
> least, that's my humble uninformed opinion.
[Hemant] Actually some of these are in datapath, there is a performance impact of accessing them via function calls.
>
> In the mean time, the tooling can be tought to ignore changes to these ELF
> symbols, as you you guys all know already.
>
[Hemant] will you please help me about adding entry to libagigail.abignore
I tried doing following, but it is not helping
--- a/devtools/libabigail.abignore
+++ b/devtools/libabigail.abignore
@@ -2,10 +2,15 @@
symbol_version = EXPERIMENTAL
[suppress_variable]
symbol_version = EXPERIMENTAL
+ name = per_lcore__dpaa2_io
+ name = dpaa2_io_portal
; Explicit ignore for driver-only ABI
[suppress_type]
name = rte_cryptodev_ops
+ name = dpaa2_io_portal_t
>
> > Anyway, all of those symbols in dpaa are part of the driver ABI.
> > We are still missing a way to mark internal symbols.
> > Neil had posted a framework for this
> >
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpatchwo
> rk.dpdk.org%2Fproject%2Fdpdk%2Flist%2F%3Fseries%3D5004&data=02
> %7C01%7Chemant.agrawal%40nxp.com%7C91a3230cfd334c28bbdb08d7c4d
> ee590%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6371943339
> 20186005&sdata=1Is%2BqQwP%2Bn0QVJ2HYK2%2Bx7TJooEvry1sNUUN
> fWMygkM%3D&reserved=0.
> >
> > In order to get this series passing the checks, I recommend NXP
> > rebasing Neil scripts (I will help reviewing this part), then mark all
> > those symbols as internal in its drivers.
> > Other vendor will convert their drivers later, as there is no need at
> > the moment.
> >
[Hemant] I have commented on Neil's series. It needs more changes in existing code. An approach like __rte_experimental will work better.
> > Thanks.
>
Regards,
Hemant
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] /validate-abi.sh complains [PATCH v1 3/8] ring: introduce RTS ring mode
2020-04-07 7:32 4% ` Ray Kinsella
@ 2020-04-07 11:31 4% ` Neil Horman
0 siblings, 0 replies; 200+ results
From: Neil Horman @ 2020-04-07 11:31 UTC (permalink / raw)
To: Ray Kinsella
Cc: Thomas Monjalon, dev, Ananyev, Konstantin, david.marchand,
honnappa.nagarahalli, Richardson, Bruce, Yigit, Ferruh,
kevin.laatz
On Tue, Apr 07, 2020 at 08:32:15AM +0100, Ray Kinsella wrote:
>
>
> On 06/04/2020 15:36, Neil Horman wrote:
> > On Mon, Apr 06, 2020 at 04:02:22PM +0200, Thomas Monjalon wrote:
> >> 01/04/2020 14:52, Neil Horman:
> >>> On Tue, Mar 31, 2020 at 09:24:41PM +0200, Thomas Monjalon wrote:
> >>>> 31/03/2020 21:22, Thomas Monjalon:
> >>>>> 31/03/2020 19:05, Ananyev, Konstantin:
> >>>>>> Hi everyone,
> >>>>>>
> >>>>>> Have a question regarding validate-abi.sh.
> >>>>>
> >>>>> devtools/validate-abi.sh should be removed.
> >>>>> Please use the new devtools/check-abi.sh
> >>>>
> >>>> The file doc/guides/contributing/abi_versioning.rst
> >>>> should be updated as well.
> >>>> Neil? Ray?
> >>>>
> >>> Ack, we should remove validate_abi and update the docs
> >>
> >> Neil, are you going to do this update?
> > I can, I had though it was already in progress though. I'll take care of it
> > this week.
>
> Thanks Neil - don't know how I missed this thread.
>
No sweat, apologies, I just realized I didn't copy you, but its on the list now
Neil
> >
> > Best
> > Neil
> >
> >>
> >>
> >>
> >>
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] Remove abi_versioning.sh from tree
2020-04-07 7:36 4% ` David Marchand
@ 2020-04-07 11:33 0% ` Neil Horman
2020-04-07 11:40 0% ` David Marchand
2020-04-07 11:58 5% ` Thomas Monjalon
2020-04-08 14:50 0% ` Ray Kinsella
1 sibling, 2 replies; 200+ results
From: Neil Horman @ 2020-04-07 11:33 UTC (permalink / raw)
To: David Marchand; +Cc: dev, Thomas Monjalon, Kevin Laatz
On Tue, Apr 07, 2020 at 09:36:17AM +0200, David Marchand wrote:
> On Mon, Apr 6, 2020 at 9:34 PM Neil Horman <nhorman@tuxdriver.com> wrote:
> >
> > Since we've moved away from our initial abi_versioning.sh script, in
>
> abi_versioning.sh does not exist (idem with the patch title).
> I suppose you meant validate-abi.sh.
>
Crud, you're right, I was convoluting terms, sorry. Shall I repost with a
corrected changelog?
Neil
> > favor of check_abi.sh, which uses libabigail, remove the old script from
>
> check-abi.sh
>
> > the tree, and update the docs accordingly
> >
>
> [snip]
>
> > diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst
> > index a21f4e7a4..1c4a3f927 100644
> > --- a/doc/guides/contributing/abi_versioning.rst
> > +++ b/doc/guides/contributing/abi_versioning.rst
> > @@ -482,9 +482,9 @@ Running the ABI Validator
> > -------------------------
> >
> > The ``devtools`` directory in the DPDK source tree contains a utility program,
> > -``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI
> > -Compliance Checker
> > -<http://ispras.linuxbase.org/index.php/ABI_compliance_checker>`_.
> > +``check-abi.sh``, for validating the DPDK ABI based on the libabigail abidiff
> > +utility:
> > +https://sourceware.org/libabigail/manual/abidiff.html
> >
> > This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper``
> > utilities which can be installed via a package manager. For example::
> > @@ -492,9 +492,9 @@ utilities which can be installed via a package manager. For example::
> > sudo yum install abi-compliance-checker
> > sudo yum install abi-dumper
> >
> > -The syntax of the ``validate-abi.sh`` utility is::
> > +The syntax of the ``check-abi.sh`` utility is::
> >
> > - ./devtools/validate-abi.sh <REV1> <REV2>
> > + ./devtools/check-abi.sh <REV1> <REV2>
>
> The new script is not a direct replacement.
> It won't take git revisions, but build directories where versions of
> dpdk have been compiled.
>
> devtools/test-build.sh and devtools/test-meson-builds.sh illustrate its use.
> https://git.dpdk.org/dpdk/tree/devtools/test-meson-builds.sh#n127
>
>
>
> --
> David Marchand
>
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] Remove abi_versioning.sh from tree
2020-04-07 11:33 0% ` Neil Horman
@ 2020-04-07 11:40 0% ` David Marchand
2020-04-07 11:58 5% ` Thomas Monjalon
1 sibling, 0 replies; 200+ results
From: David Marchand @ 2020-04-07 11:40 UTC (permalink / raw)
To: Neil Horman; +Cc: dev, Thomas Monjalon, Kevin Laatz, Kinsella, Ray
On Tue, Apr 7, 2020 at 1:33 PM Neil Horman <nhorman@tuxdriver.com> wrote:
>
> On Tue, Apr 07, 2020 at 09:36:17AM +0200, David Marchand wrote:
> > On Mon, Apr 6, 2020 at 9:34 PM Neil Horman <nhorman@tuxdriver.com> wrote:
> > >
> > > Since we've moved away from our initial abi_versioning.sh script, in
> >
> > abi_versioning.sh does not exist (idem with the patch title).
> > I suppose you meant validate-abi.sh.
> >
> Crud, you're right, I was convoluting terms, sorry. Shall I repost with a
> corrected changelog?
Yes, and with the other comment I did earlier, addressed too.
> > > favor of check_abi.sh, which uses libabigail, remove the old script from
> >
> > check-abi.sh
> >
> > > the tree, and update the docs accordingly
> > >
> >
> > [snip]
> >
> > > diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst
> > > index a21f4e7a4..1c4a3f927 100644
> > > --- a/doc/guides/contributing/abi_versioning.rst
> > > +++ b/doc/guides/contributing/abi_versioning.rst
> > > @@ -482,9 +482,9 @@ Running the ABI Validator
> > > -------------------------
> > >
> > > The ``devtools`` directory in the DPDK source tree contains a utility program,
> > > -``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI
> > > -Compliance Checker
> > > -<http://ispras.linuxbase.org/index.php/ABI_compliance_checker>`_.
> > > +``check-abi.sh``, for validating the DPDK ABI based on the libabigail abidiff
> > > +utility:
> > > +https://sourceware.org/libabigail/manual/abidiff.html
> > >
> > > This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper``
> > > utilities which can be installed via a package manager. For example::
> > > @@ -492,9 +492,9 @@ utilities which can be installed via a package manager. For example::
> > > sudo yum install abi-compliance-checker
> > > sudo yum install abi-dumper
> > >
> > > -The syntax of the ``validate-abi.sh`` utility is::
> > > +The syntax of the ``check-abi.sh`` utility is::
> > >
> > > - ./devtools/validate-abi.sh <REV1> <REV2>
> > > + ./devtools/check-abi.sh <REV1> <REV2>
> >
> > The new script is not a direct replacement.
> > It won't take git revisions, but build directories where versions of
> > dpdk have been compiled.
> >
> > devtools/test-build.sh and devtools/test-meson-builds.sh illustrate its use.
> > https://git.dpdk.org/dpdk/tree/devtools/test-meson-builds.sh#n127
> >
Thanks.
--
David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] Remove abi_versioning.sh from tree
2020-04-07 11:33 0% ` Neil Horman
2020-04-07 11:40 0% ` David Marchand
@ 2020-04-07 11:58 5% ` Thomas Monjalon
2020-04-07 19:52 3% ` Neil Horman
2020-04-08 14:34 0% ` Ray Kinsella
1 sibling, 2 replies; 200+ results
From: Thomas Monjalon @ 2020-04-07 11:58 UTC (permalink / raw)
To: David Marchand, Neil Horman, Kevin Laatz, mdr; +Cc: dev
07/04/2020 13:33, Neil Horman:
> On Tue, Apr 07, 2020 at 09:36:17AM +0200, David Marchand wrote:
> > On Mon, Apr 6, 2020 at 9:34 PM Neil Horman <nhorman@tuxdriver.com> wrote:
> > >
> > > Since we've moved away from our initial abi_versioning.sh script, in
> >
> > abi_versioning.sh does not exist (idem with the patch title).
> > I suppose you meant validate-abi.sh.
> >
> Crud, you're right, I was convoluting terms, sorry. Shall I repost with a
> corrected changelog?
Not only the commit log, look below how you did not care about the basic
usage of the new tool.
> > > favor of check_abi.sh, which uses libabigail, remove the old script from
> >
> > check-abi.sh
> >
> > > the tree, and update the docs accordingly
[...]
> > > --- a/doc/guides/contributing/abi_versioning.rst
> > > +++ b/doc/guides/contributing/abi_versioning.rst
The maintainer of doc/guides/contributing/abi_*.rst
is Ray Kinsella so I add him as Cc.
[...]
> > > -The syntax of the ``validate-abi.sh`` utility is::
> > > +The syntax of the ``check-abi.sh`` utility is::
> > >
> > > - ./devtools/validate-abi.sh <REV1> <REV2>
> > > + ./devtools/check-abi.sh <REV1> <REV2>
> >
> > The new script is not a direct replacement.
> > It won't take git revisions, but build directories where versions of
> > dpdk have been compiled.
> >
> > devtools/test-build.sh and devtools/test-meson-builds.sh illustrate its use.
> > https://git.dpdk.org/dpdk/tree/devtools/test-meson-builds.sh#n127
David, I think Neil did not take time to understand what changed in
ABI tooling.
I really wonder who is the real maintainer of ABI tooling and policy.
Neil, Ray, I was expecting a better involvement in this major
policy enforcement.
This is where we are:
- Neil asked first for ABI compatibility
- Neil created validate-abi.sh
- Ray asked for a strict policy
- Kevin worked on a new tooling
- David completed the tooling work
- David integrated ABI checks in Travis
There are many people partly involved.
I think we need one person truly involved in ABI questions,
someone who feels responsible and will take care of details
like the documentation update requested above.
Please don't rely on David and myself, we are already very busy
with making sure every patches are properly reviewed.
We need good help on the ABI topic in general.
^ permalink raw reply [relevance 5%]
* Re: [dpdk-dev] [PATCH 00/16] NXP DPAAx fixes and enhancements
2020-04-07 10:25 3% ` Hemant Agrawal
@ 2020-04-07 12:20 0% ` Thomas Monjalon
2020-04-08 7:20 4% ` Dodji Seketeli
1 sibling, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-04-07 12:20 UTC (permalink / raw)
To: Hemant Agrawal (OSS), Hemant Agrawal
Cc: Dodji Seketeli, David Marchand, dev, Yigit, Ferruh, dev, Neil Horman
07/04/2020 12:25, Hemant Agrawal:
> > In the mean time, the tooling can be tought to ignore changes to these ELF
> > symbols, as you you guys all know already.
> >
> [Hemant] will you please help me about adding entry to libagigail.abignore
> I tried doing following, but it is not helping
>
> --- a/devtools/libabigail.abignore
> +++ b/devtools/libabigail.abignore
> @@ -2,10 +2,15 @@
> symbol_version = EXPERIMENTAL
> [suppress_variable]
> symbol_version = EXPERIMENTAL
> + name = per_lcore__dpaa2_io
> + name = dpaa2_io_portal
>
> ; Explicit ignore for driver-only ABI
> [suppress_type]
> name = rte_cryptodev_ops
> + name = dpaa2_io_portal_t
> >
> > > Anyway, all of those symbols in dpaa are part of the driver ABI.
> > > We are still missing a way to mark internal symbols.
> > > Neil had posted a framework for this
> > >
> > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpatchwo
> > rk.dpdk.org%2Fproject%2Fdpdk%2Flist%2F%3Fseries%3D5004&data=02
> > %7C01%7Chemant.agrawal%40nxp.com%7C91a3230cfd334c28bbdb08d7c4d
> > ee590%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6371943339
> > 20186005&sdata=1Is%2BqQwP%2Bn0QVJ2HYK2%2Bx7TJooEvry1sNUUN
> > fWMygkM%3D&reserved=0.
> > >
> > > In order to get this series passing the checks, I recommend NXP
> > > rebasing Neil scripts (I will help reviewing this part), then mark all
> > > those symbols as internal in its drivers.
> > > Other vendor will convert their drivers later, as there is no need at
> > > the moment.
> > >
> [Hemant] I have commented on Neil's series.
> It needs more changes in existing code.
> An approach like __rte_experimental will work better.
I guess you mean __rte_internal?
Please Hemant don't wait for someone else filling the gap.
If __rte_internal is the right approach, please complete and use it.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2 0/4] introduce multi-function processing support
@ 2020-04-07 18:05 2% ` Trahe, Fiona
2020-04-09 9:25 0% ` Coyle, David
0 siblings, 1 reply; 200+ results
From: Trahe, Fiona @ 2020-04-07 18:05 UTC (permalink / raw)
To: Coyle, David, Yigit, Ferruh, dev
Cc: Doherty, Declan, De Lara Guarch, Pablo, Ryan, Brendan,
shreyansh.jain, hemant.agrawal, Akhil Goyal, Trahe, Fiona
Hi David, Ferruh,
> -----Original Message-----
> From: Coyle, David <david.coyle@intel.com>
> Sent: Tuesday, April 7, 2020 12:28 PM
> To: Yigit, Ferruh <ferruh.yigit@intel.com>; dev@dpdk.org
> Cc: Doherty, Declan <declan.doherty@intel.com>; Trahe, Fiona <fiona.trahe@intel.com>; De Lara
> Guarch, Pablo <pablo.de.lara.guarch@intel.com>; Ryan, Brendan <brendan.ryan@intel.com>;
> shreyansh.jain@nxp.com; hemant.agrawal@nxp.com; Akhil Goyal <akhil.goyal@nxp.com>
> Subject: RE: [dpdk-dev] [PATCH v2 0/4] introduce multi-function processing support
>
> Hi Ferruh, see below
>
> > >
> > > While DPDK's rte_cryptodev and rte_compressdev allow many
> > > cryptographic and compression algorithms to be chained together in one
> > > operation, there is no way to chain these with any error detection or
> > > checksum algorithms. And there is no way to chain crypto and
> > > compression algorithms together. The multi-function interface will
> > > allow these chains to be created, and also allow any future type of
> > operation to be easily added.
> >
> > I was thinking if the cryptodev can be used instead but this paragraph already
> > seems explained it. But again can you please elaborate why rawdev is used?
>
> [DC] There are a number of reasons the rawdev approach was ultimately chosen:
>
> 1) As the paragraph above explains, our primary use-case was to chain a crypto operation with error
> detection algorithms such as CRC or BIP as this could leverage optimized multi-function
> implementations such as in the IPSec Multi-Buffer library and have a significant impact on
> performance of network access dataplane processing such as for vCMTS (DOCSIS MAC).
> However such error detection algorithms are not Crypto functions so some early advice we took was
> that it would not be suitable to add these to cryptodev.
> Also, with a view to the future, the multi-function rawdev approach allows crypto operations to be
> chained with compression operations.
> Again, neither cryptodev or compressdev allows this type chaining.
>
> 2) An earlier version of multi-function suggested adding a new library called rte_accelerator, as
> described here http://mails.dpdk.org/archives/dev/2020-February/157045.html
> We received some comments on the dev mailing list that we should not add yet another acceleration
> library to DPDK.
> And we also subsequently felt that the rawdev approach is better - that rationale is described below.
>
> rte_accelerator was also built on top of crypto and compress devices which already existed e.g.
> drivers/crypto/aesni_mb, drivers/crypto/qat and drivers/compress/qat .
> We subsequently realized that this was somewhat confusing when performing multi-function type
> operations. For example, for combined Crypto-Compression operations in the future, it would use
> either an existing crypto or compress device, but neither really made sense when the operations are
> combined.
> What was needed was a raw device which allowed an application to configure any type of device and
> it's queue pairs and send any type of operation to that device.
>
> For both of these reasons, we decided to go down the rawdev route, with a multi-function interface
> which can be used by several raw device drivers.
>
> 3) rawdev is the ideal place to try out a new approach like this to accessing devices.
> Adding it here allows potential consumers of this such as VNF solution providers to study and try out
> this approach, and take advantage of the multi-function operations already supported in the IPSec
> Multi-Buffer library such as Crypto-CRC and Crypto-CRC-BIP, all without DPDK committing to a new
> library upfront.
> We would hope that the multi-function rawdev approach will mature over time (through feedback
> from customers, new use-cases arising etc.), at which point it could be potentially be moved into the
> main DPDK library set.
>
[Fiona] agree with above, in particular item (2). Just to expand a bit more on this: To do a crypto+compression op
one would only send one op to one device. That device could have been either a crypto device which also
implemented multi-fn, by adding compression, crc, etc to its capabilities OR a compression device which added crypto, crc, bip capabilities.
Both were confusing and both raised questions about whether one could still do "normal" ops on the device, e.g. whether
a normal crypto op could be interleaved on same qp as a multi-fn op. And how the capabilities would reflect what the device could do.
It seems better to me to have a multifn device, which does explicitly just multifn ops.
Building this on top of rawdev is a good fit in my opinion, for the following reasons:
* avoids duplication of device APIs, as rawdev configure, start, stop, qp_setup, etc are all there already, also a nice set of stats APIs
* no impact or dependency added to rawdev lib
* avoids breakages on cryptodev or compressdev APIs
* avoids code duplication where functionality is already in a lib, e.g. re-uses cryptodev and compressdev headers. This adds a dependency, but I think that's ok as multi-function inherently depends on these functions.
* allows easy extension to add new functionality not currently available in any lib (CRC and BIP)
* allows evolution - range of useful chains which may emerge is not yet clear.
I do have some concerns, but these are resolvable in my opinion.
(A) as there's no rawdev capability APIs and capabilities are essentially opaque to the rawdev API, the application uses explicit device naming to create or find a device that it knows will fulfil the multifunction APIs. I can see how this works for rawdevs which expect to have only one PMD that will fulfil the service, however I'd expect multi-fn to have at least 2 driver types, probably more eventually. To be extensible I'd suggest a naming convention for a class of devices. E.g. all devices and drivers that implement multi-fn should create a rawdev named mfn_xxx, e.g. mfn_aesni_mb, mfn_qat. The "mfn_" string should be defined in the mfn hdr. This would allow creation of apis like rte_multi_fn_count() which could find rawdevs which implement mfn_ without hardcoding specific driver names.
(B) version control of the multi-function APIs. Putting the multifn API into the drivers/raw/common directory gives a lot of freedom while it's experimental. But can it benefit from API/ABI breakage infrastructure once the experimental tag is removed? Is there any reason not to move the common files to a lib/librte_multi_fn API?
(C) xstat name strings should be moved from aesni_mb PMD to common and maybe use same naming convention, so appl can query same stats from any device, e.g. "mfn_successful_enqueues" could be implemented by all PMDs. If PMDs want to add driver-specific stats they can add their own without the mfn_, instead create their own unique stat name.
(D) The unit test code is not extensible - again probably as based on previous rawdevs where there's only 1 implementation. For mfn I'd suggest replacing test_rawdev_selftest_aesni_mb() with a test_rawdev_selftest_multi_function(), which finds and/or creates all the raw PMDs implementing the mfn API and runs a test on each. And move the test files from the drivers/raw/aesni_mb dir to app/test and make generic so can run against any device named mfn_xxx
(E) the main reason to piggyback onto crypto_perf_tool is to get the benefit of parsing and of all the crypto setup. However this code has been inflated a lot, in part due to name diffs like rte_cryptodev_enqueue_burst() vs rte_multi_fn_enqueue_burst(). Maybe could be a lot slimmer with macros like ENQUEUE_BURST(dev, qp, void *op, burst_size) ? would mean a compile time decision to do either multifn OR cryptodev API calls, but I think that may work and simplify it.
(F) ok, this is a bit pedantic, (sorry David!) but should the aesni_mb rawdev be renamed aesni_mb_mfn throughout (files, fns, dev and driver name). I mean it's implementing the mfn type of rawdev. I'm thinking ahead to QAT - it can implement a sym device, an asym device, a compression device and in future a multi-fn device. I'd propose to name it qat_multifn in case there'll be some other kind of rawdev device it could also implement in future. So the name qat_raw wouldn't be so helpful. (we made that mistake with qat_crypto, which should probably have been qat_sym_crypto - in my opinion more specific names are better)
I have a few minor comment- I'll reply on specific patches.
^ permalink raw reply [relevance 2%]
* Re: [dpdk-dev] [PATCH] Remove abi_versioning.sh from tree
2020-04-07 11:58 5% ` Thomas Monjalon
@ 2020-04-07 19:52 3% ` Neil Horman
2020-04-08 14:34 0% ` Ray Kinsella
1 sibling, 0 replies; 200+ results
From: Neil Horman @ 2020-04-07 19:52 UTC (permalink / raw)
To: Thomas Monjalon; +Cc: David Marchand, Kevin Laatz, mdr, dev
On Tue, Apr 07, 2020 at 01:58:57PM +0200, Thomas Monjalon wrote:
> 07/04/2020 13:33, Neil Horman:
> > On Tue, Apr 07, 2020 at 09:36:17AM +0200, David Marchand wrote:
> > > On Mon, Apr 6, 2020 at 9:34 PM Neil Horman <nhorman@tuxdriver.com> wrote:
> > > >
> > > > Since we've moved away from our initial abi_versioning.sh script, in
> > >
> > > abi_versioning.sh does not exist (idem with the patch title).
> > > I suppose you meant validate-abi.sh.
> > >
> > Crud, you're right, I was convoluting terms, sorry. Shall I repost with a
> > corrected changelog?
>
> Not only the commit log, look below how you did not care about the basic
> usage of the new tool.
>
>
> > > > favor of check_abi.sh, which uses libabigail, remove the old script from
> > >
> > > check-abi.sh
> > >
> > > > the tree, and update the docs accordingly
> [...]
> > > > --- a/doc/guides/contributing/abi_versioning.rst
> > > > +++ b/doc/guides/contributing/abi_versioning.rst
>
> The maintainer of doc/guides/contributing/abi_*.rst
> is Ray Kinsella so I add him as Cc.
>
> [...]
> > > > -The syntax of the ``validate-abi.sh`` utility is::
> > > > +The syntax of the ``check-abi.sh`` utility is::
> > > >
> > > > - ./devtools/validate-abi.sh <REV1> <REV2>
> > > > + ./devtools/check-abi.sh <REV1> <REV2>
> > >
> > > The new script is not a direct replacement.
> > > It won't take git revisions, but build directories where versions of
> > > dpdk have been compiled.
> > >
> > > devtools/test-build.sh and devtools/test-meson-builds.sh illustrate its use.
> > > https://git.dpdk.org/dpdk/tree/devtools/test-meson-builds.sh#n127
>
> David, I think Neil did not take time to understand what changed in
> ABI tooling.
> I really wonder who is the real maintainer of ABI tooling and policy.
> Neil, Ray, I was expecting a better involvement in this major
> policy enforcement.
>
> This is where we are:
> - Neil asked first for ABI compatibility
> - Neil created validate-abi.sh
> - Ray asked for a strict policy
> - Kevin worked on a new tooling
> - David completed the tooling work
> - David integrated ABI checks in Travis
>
> There are many people partly involved.
> I think we need one person truly involved in ABI questions,
> someone who feels responsible and will take care of details
> like the documentation update requested above.
>
> Please don't rely on David and myself, we are already very busy
> with making sure every patches are properly reviewed.
> We need good help on the ABI topic in general.
>
We're all very busy Thomas, theres no need to get bent out of shape. You're
right, I thought this was easier than it was, thinking that check-abi.sh was
interface compatible, apologies. I'm going through this more carefully, and will
repost soon.
FWIW, I don't think I'll be able to fill this role full time. I'm not sure if
anyone else is either, i'm afraid.
Neil
>
>
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v3 2/9] ring: prepare ring to allow new sync schemes
@ 2020-04-08 4:59 3% ` Honnappa Nagarahalli
2020-04-09 13:39 3% ` Ananyev, Konstantin
0 siblings, 1 reply; 200+ results
From: Honnappa Nagarahalli @ 2020-04-08 4:59 UTC (permalink / raw)
To: Konstantin Ananyev, dev
Cc: david.marchand, jielong.zjl, nd, Honnappa Nagarahalli, nd
<snip>
> Subject: [PATCH v3 2/9] ring: prepare ring to allow new sync schemes
>
> Change from *single* to *sync_type* to allow different synchronisation
> schemes to be applied.
> Mark *single* as deprecated in comments.
> Add new functions to allow user to query ring sync types.
> Replace direct access to *single* with appopriate function call.
>
> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> ---
> app/test/test_pdump.c | 6 +-
> lib/librte_pdump/rte_pdump.c | 2 +-
> lib/librte_port/rte_port_ring.c | 12 ++--
> lib/librte_ring/rte_ring.c | 6 +-
> lib/librte_ring/rte_ring.h | 113 ++++++++++++++++++++++++++------
> lib/librte_ring/rte_ring_elem.h | 8 +--
> 6 files changed, 108 insertions(+), 39 deletions(-)
>
> diff --git a/app/test/test_pdump.c b/app/test/test_pdump.c index
> ad183184c..6a1180bcb 100644
> --- a/app/test/test_pdump.c
> +++ b/app/test/test_pdump.c
> @@ -57,8 +57,7 @@ run_pdump_client_tests(void)
> if (ret < 0)
> return -1;
> mp->flags = 0x0000;
> - ring_client = rte_ring_create("SR0", RING_SIZE, rte_socket_id(),
> - RING_F_SP_ENQ | RING_F_SC_DEQ);
> + ring_client = rte_ring_create("SR0", RING_SIZE, rte_socket_id(), 0);
Are you saying to get SP and SC behavior we now have to set the flags to 0? Isn't that a ABI break?
> if (ring_client == NULL) {
> printf("rte_ring_create SR0 failed");
> return -1;
> @@ -71,9 +70,6 @@ run_pdump_client_tests(void)
> }
> rte_eth_dev_probing_finish(eth_dev);
>
> - ring_client->prod.single = 0;
> - ring_client->cons.single = 0;
Just wondering if users outside of DPDK have done the same. I hope not, otherwise, we have an API break?
> -
> printf("\n***** flags = RTE_PDUMP_FLAG_TX *****\n");
>
> for (itr = 0; itr < NUM_ITR; itr++) {
> diff --git a/lib/librte_pdump/rte_pdump.c b/lib/librte_pdump/rte_pdump.c
> index 8a01ac510..65364f2c5 100644
> --- a/lib/librte_pdump/rte_pdump.c
> +++ b/lib/librte_pdump/rte_pdump.c
> @@ -380,7 +380,7 @@ pdump_validate_ring_mp(struct rte_ring *ring, struct
> rte_mempool *mp)
> rte_errno = EINVAL;
> return -1;
> }
> - if (ring->prod.single || ring->cons.single) {
> + if (rte_ring_prod_single(ring) || rte_ring_cons_single(ring)) {
> PDUMP_LOG(ERR, "ring with either SP or SC settings"
> " is not valid for pdump, should have MP and MC settings\n");
> rte_errno = EINVAL;
> diff --git a/lib/librte_port/rte_port_ring.c b/lib/librte_port/rte_port_ring.c
> index 47fcdd06a..2f6c050fa 100644
> --- a/lib/librte_port/rte_port_ring.c
> +++ b/lib/librte_port/rte_port_ring.c
> @@ -44,8 +44,8 @@ rte_port_ring_reader_create_internal(void *params, int
> socket_id,
> /* Check input parameters */
> if ((conf == NULL) ||
> (conf->ring == NULL) ||
> - (conf->ring->cons.single && is_multi) ||
> - (!(conf->ring->cons.single) && !is_multi)) {
> + (rte_ring_cons_single(conf->ring) && is_multi) ||
> + (!rte_ring_cons_single(conf->ring) && !is_multi)) {
> RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
> return NULL;
> }
> @@ -171,8 +171,8 @@ rte_port_ring_writer_create_internal(void *params,
> int socket_id,
> /* Check input parameters */
> if ((conf == NULL) ||
> (conf->ring == NULL) ||
> - (conf->ring->prod.single && is_multi) ||
> - (!(conf->ring->prod.single) && !is_multi) ||
> + (rte_ring_prod_single(conf->ring) && is_multi) ||
> + (!rte_ring_prod_single(conf->ring) && !is_multi) ||
> (conf->tx_burst_sz > RTE_PORT_IN_BURST_SIZE_MAX)) {
> RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
> return NULL;
> @@ -440,8 +440,8 @@ rte_port_ring_writer_nodrop_create_internal(void
> *params, int socket_id,
> /* Check input parameters */
> if ((conf == NULL) ||
> (conf->ring == NULL) ||
> - (conf->ring->prod.single && is_multi) ||
> - (!(conf->ring->prod.single) && !is_multi) ||
> + (rte_ring_prod_single(conf->ring) && is_multi) ||
> + (!rte_ring_prod_single(conf->ring) && !is_multi) ||
> (conf->tx_burst_sz > RTE_PORT_IN_BURST_SIZE_MAX)) {
> RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
> return NULL;
> diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c index
> 77e5de099..fa5733907 100644
> --- a/lib/librte_ring/rte_ring.c
> +++ b/lib/librte_ring/rte_ring.c
> @@ -106,8 +106,10 @@ rte_ring_init(struct rte_ring *r, const char *name,
> unsigned count,
> if (ret < 0 || ret >= (int)sizeof(r->name))
> return -ENAMETOOLONG;
> r->flags = flags;
> - r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
> - r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
> + r->prod.sync_type = (flags & RING_F_SP_ENQ) ?
> + RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
> + r->cons.sync_type = (flags & RING_F_SC_DEQ) ?
> + RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
>
> if (flags & RING_F_EXACT_SZ) {
> r->size = rte_align32pow2(count + 1); diff --git
> a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h index
> 18fc5d845..d4775a063 100644
> --- a/lib/librte_ring/rte_ring.h
> +++ b/lib/librte_ring/rte_ring.h
> @@ -61,11 +61,27 @@ enum rte_ring_queue_behavior { #define
> RTE_RING_NAMESIZE (RTE_MEMZONE_NAMESIZE - \
> sizeof(RTE_RING_MZ_PREFIX) + 1)
>
> -/* structure to hold a pair of head/tail values and other metadata */
> +/** prod/cons sync types */
> +enum rte_ring_sync_type {
> + RTE_RING_SYNC_MT, /**< multi-thread safe (default mode) */
> + RTE_RING_SYNC_ST, /**< single thread only */
> +};
> +
> +/**
> + * structure to hold a pair of head/tail values and other metadata.
> + * Depending on sync_type format of that structure might be different,
> + * but offset for *sync_type* and *tail* values should remain the same.
> + */
> struct rte_ring_headtail {
> - volatile uint32_t head; /**< Prod/consumer head. */
> - volatile uint32_t tail; /**< Prod/consumer tail. */
> - uint32_t single; /**< True if single prod/cons */
> + volatile uint32_t head; /**< prod/consumer head. */
> + volatile uint32_t tail; /**< prod/consumer tail. */
> + RTE_STD_C11
> + union {
> + /** sync type of prod/cons */
> + enum rte_ring_sync_type sync_type;
> + /** deprecated - True if single prod/cons */
> + uint32_t single;
> + };
> };
>
> /**
> @@ -116,11 +132,10 @@ struct rte_ring {
> #define RING_F_EXACT_SZ 0x0004
> #define RTE_RING_SZ_MASK (0x7fffffffU) /**< Ring size mask */
>
> -/* @internal defines for passing to the enqueue dequeue worker functions
> */ -#define __IS_SP 1 -#define __IS_MP 0 -#define __IS_SC 1 -#define
> __IS_MC 0
> +#define __IS_SP RTE_RING_SYNC_ST
> +#define __IS_MP RTE_RING_SYNC_MT
> +#define __IS_SC RTE_RING_SYNC_ST
> +#define __IS_MC RTE_RING_SYNC_MT
I think we can remove these #defines and use the new SYNC types
>
> /**
> * Calculate the memory size needed for a ring @@ -420,7 +435,7 @@
> rte_ring_mp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
> unsigned int n, unsigned int *free_space) {
> return __rte_ring_do_enqueue(r, obj_table, n,
> RTE_RING_QUEUE_FIXED,
> - __IS_MP, free_space);
> + RTE_RING_SYNC_MT, free_space);
> }
>
> /**
> @@ -443,7 +458,7 @@ rte_ring_sp_enqueue_bulk(struct rte_ring *r, void *
> const *obj_table,
> unsigned int n, unsigned int *free_space) {
> return __rte_ring_do_enqueue(r, obj_table, n,
> RTE_RING_QUEUE_FIXED,
> - __IS_SP, free_space);
> + RTE_RING_SYNC_ST, free_space);
> }
>
> /**
> @@ -470,7 +485,7 @@ rte_ring_enqueue_bulk(struct rte_ring *r, void *
> const *obj_table,
> unsigned int n, unsigned int *free_space) {
> return __rte_ring_do_enqueue(r, obj_table, n,
> RTE_RING_QUEUE_FIXED,
> - r->prod.single, free_space);
> + r->prod.sync_type, free_space);
> }
>
> /**
> @@ -554,7 +569,7 @@ rte_ring_mc_dequeue_bulk(struct rte_ring *r, void
> **obj_table,
> unsigned int n, unsigned int *available) {
> return __rte_ring_do_dequeue(r, obj_table, n,
> RTE_RING_QUEUE_FIXED,
> - __IS_MC, available);
> + RTE_RING_SYNC_MT, available);
> }
>
> /**
> @@ -578,7 +593,7 @@ rte_ring_sc_dequeue_bulk(struct rte_ring *r, void
> **obj_table,
> unsigned int n, unsigned int *available) {
> return __rte_ring_do_dequeue(r, obj_table, n,
> RTE_RING_QUEUE_FIXED,
> - __IS_SC, available);
> + RTE_RING_SYNC_ST, available);
> }
>
> /**
> @@ -605,7 +620,7 @@ rte_ring_dequeue_bulk(struct rte_ring *r, void
> **obj_table, unsigned int n,
> unsigned int *available)
> {
> return __rte_ring_do_dequeue(r, obj_table, n,
> RTE_RING_QUEUE_FIXED,
> - r->cons.single, available);
> + r->cons.sync_type, available);
> }
>
> /**
> @@ -777,6 +792,62 @@ rte_ring_get_capacity(const struct rte_ring *r)
> return r->capacity;
> }
>
> +/**
> + * Return sync type used by producer in the ring.
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @return
> + * Producer sync type value.
> + */
> +static inline enum rte_ring_sync_type
> +rte_ring_get_prod_sync_type(const struct rte_ring *r) {
> + return r->prod.sync_type;
> +}
> +
> +/**
> + * Check is the ring for single producer.
^^ if
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @return
> + * true if ring is SP, zero otherwise.
> + */
> +static inline int
> +rte_ring_prod_single(const struct rte_ring *r) {
would rte_ring_is_prod_single better?
> + return (rte_ring_get_prod_sync_type(r) == RTE_RING_SYNC_ST); }
> +
> +/**
> + * Return sync type used by consumer in the ring.
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @return
> + * Consumer sync type value.
> + */
> +static inline enum rte_ring_sync_type
> +rte_ring_get_cons_sync_type(const struct rte_ring *r) {
> + return r->cons.sync_type;
> +}
> +
> +/**
> + * Check is the ring for single consumer.
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @return
> + * true if ring is SC, zero otherwise.
> + */
> +static inline int
> +rte_ring_cons_single(const struct rte_ring *r) {
> + return (rte_ring_get_cons_sync_type(r) == RTE_RING_SYNC_ST); }
> +
All these new functions are not required to be called in the data path. They can be made non-inline.
> /**
> * Dump the status of all rings on the console
> *
> @@ -820,7 +891,7 @@ rte_ring_mp_enqueue_burst(struct rte_ring *r, void *
> const *obj_table,
> unsigned int n, unsigned int *free_space) {
> return __rte_ring_do_enqueue(r, obj_table, n,
> - RTE_RING_QUEUE_VARIABLE, __IS_MP, free_space);
> + RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_MT,
> free_space);
> }
>
> /**
> @@ -843,7 +914,7 @@ rte_ring_sp_enqueue_burst(struct rte_ring *r, void *
> const *obj_table,
> unsigned int n, unsigned int *free_space) {
> return __rte_ring_do_enqueue(r, obj_table, n,
> - RTE_RING_QUEUE_VARIABLE, __IS_SP, free_space);
> + RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_ST,
> free_space);
> }
>
> /**
> @@ -870,7 +941,7 @@ rte_ring_enqueue_burst(struct rte_ring *r, void *
> const *obj_table,
> unsigned int n, unsigned int *free_space) {
> return __rte_ring_do_enqueue(r, obj_table, n,
> RTE_RING_QUEUE_VARIABLE,
> - r->prod.single, free_space);
> + r->prod.sync_type, free_space);
> }
>
> /**
> @@ -898,7 +969,7 @@ rte_ring_mc_dequeue_burst(struct rte_ring *r, void
> **obj_table,
> unsigned int n, unsigned int *available) {
> return __rte_ring_do_dequeue(r, obj_table, n,
> - RTE_RING_QUEUE_VARIABLE, __IS_MC, available);
> + RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_MT,
> available);
> }
>
> /**
> @@ -923,7 +994,7 @@ rte_ring_sc_dequeue_burst(struct rte_ring *r, void
> **obj_table,
> unsigned int n, unsigned int *available) {
> return __rte_ring_do_dequeue(r, obj_table, n,
> - RTE_RING_QUEUE_VARIABLE, __IS_SC, available);
> + RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_ST,
> available);
> }
>
> /**
> @@ -951,7 +1022,7 @@ rte_ring_dequeue_burst(struct rte_ring *r, void
> **obj_table, {
> return __rte_ring_do_dequeue(r, obj_table, n,
> RTE_RING_QUEUE_VARIABLE,
> - r->cons.single, available);
> + r->cons.sync_type, available);
> }
>
> #ifdef __cplusplus
> diff --git a/lib/librte_ring/rte_ring_elem.h b/lib/librte_ring/rte_ring_elem.h
> index 663addc73..28f9836e6 100644
> --- a/lib/librte_ring/rte_ring_elem.h
> +++ b/lib/librte_ring/rte_ring_elem.h
> @@ -570,7 +570,7 @@ rte_ring_enqueue_bulk_elem(struct rte_ring *r, const
> void *obj_table,
> unsigned int esize, unsigned int n, unsigned int *free_space) {
> return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
> - RTE_RING_QUEUE_FIXED, r->prod.single, free_space);
> + RTE_RING_QUEUE_FIXED, r->prod.sync_type,
> free_space);
> }
>
> /**
> @@ -734,7 +734,7 @@ rte_ring_dequeue_bulk_elem(struct rte_ring *r, void
> *obj_table,
> unsigned int esize, unsigned int n, unsigned int *available) {
> return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
> - RTE_RING_QUEUE_FIXED, r->cons.single, available);
> + RTE_RING_QUEUE_FIXED, r->cons.sync_type,
> available);
> }
>
> /**
> @@ -902,7 +902,7 @@ rte_ring_enqueue_burst_elem(struct rte_ring *r,
> const void *obj_table,
> unsigned int esize, unsigned int n, unsigned int *free_space) {
> return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
> - RTE_RING_QUEUE_VARIABLE, r->prod.single,
> free_space);
> + RTE_RING_QUEUE_VARIABLE, r->prod.sync_type,
> free_space);
> }
>
> /**
> @@ -995,7 +995,7 @@ rte_ring_dequeue_burst_elem(struct rte_ring *r, void
> *obj_table, {
> return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
> RTE_RING_QUEUE_VARIABLE,
> - r->cons.single, available);
> + r->cons.sync_type, available);
> }
>
> #ifdef __cplusplus
> --
> 2.17.1
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v3 3/9] ring: introduce RTS ring mode
2020-04-03 17:42 1% ` [dpdk-dev] [PATCH v3 3/9] ring: introduce RTS ring mode Konstantin Ananyev
2020-04-04 17:27 0% ` Wang, Haiyue
@ 2020-04-08 5:00 0% ` Honnappa Nagarahalli
2020-04-09 14:52 0% ` Ananyev, Konstantin
1 sibling, 1 reply; 200+ results
From: Honnappa Nagarahalli @ 2020-04-08 5:00 UTC (permalink / raw)
To: Konstantin Ananyev, dev
Cc: david.marchand, jielong.zjl, Honnappa Nagarahalli, nd, nd
<snip>
>
> Introduce relaxed tail sync (RTS) mode for MT ring synchronization.
> Aim to reduce stall times in case when ring is used on overcommited cpus
> (multiple active threads on the same cpu).
> The main difference from original MP/MC algorithm is that tail value is
> increased not by every thread that finished enqueue/dequeue, but only by the
> last one.
> That allows threads to avoid spinning on ring tail value, leaving actual tail
> value change to the last thread in the update queue.
>
> check-abi.sh reports what I believe is a false-positive about ring cons/prod
> changes. As a workaround, devtools/libabigail.abignore is updated to suppress
> *struct ring* related errors.
This can be removed from the commit message.
>
> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> ---
> devtools/libabigail.abignore | 7 +
> lib/librte_ring/Makefile | 5 +-
> lib/librte_ring/meson.build | 5 +-
> lib/librte_ring/rte_ring.c | 100 +++++++-
> lib/librte_ring/rte_ring.h | 110 ++++++++-
> lib/librte_ring/rte_ring_elem.h | 86 ++++++-
> lib/librte_ring/rte_ring_rts.h | 316 +++++++++++++++++++++++++
> lib/librte_ring/rte_ring_rts_elem.h | 205 ++++++++++++++++
> lib/librte_ring/rte_ring_rts_generic.h | 210 ++++++++++++++++
> 9 files changed, 1015 insertions(+), 29 deletions(-) create mode 100644
> lib/librte_ring/rte_ring_rts.h create mode 100644
> lib/librte_ring/rte_ring_rts_elem.h
> create mode 100644 lib/librte_ring/rte_ring_rts_generic.h
>
> diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore index
> a59df8f13..cd86d89ca 100644
> --- a/devtools/libabigail.abignore
> +++ b/devtools/libabigail.abignore
> @@ -11,3 +11,10 @@
> type_kind = enum
> name = rte_crypto_asym_xform_type
> changed_enumerators = RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
> +; Ignore updates of ring prod/cons
> +[suppress_type]
> + type_kind = struct
> + name = rte_ring
> +[suppress_type]
> + type_kind = struct
> + name = rte_event_ring
Does this block the reporting of these structures forever?
> diff --git a/lib/librte_ring/Makefile b/lib/librte_ring/Makefile index
> 917c560ad..8f5c284cc 100644
> --- a/lib/librte_ring/Makefile
> +++ b/lib/librte_ring/Makefile
> @@ -18,6 +18,9 @@ SRCS-$(CONFIG_RTE_LIBRTE_RING) := rte_ring.c
> SYMLINK-$(CONFIG_RTE_LIBRTE_RING)-include := rte_ring.h \
> rte_ring_elem.h \
> rte_ring_generic.h \
> - rte_ring_c11_mem.h
> + rte_ring_c11_mem.h \
> + rte_ring_rts.h \
> + rte_ring_rts_elem.h \
> + rte_ring_rts_generic.h
>
> include $(RTE_SDK)/mk/rte.lib.mk
> diff --git a/lib/librte_ring/meson.build b/lib/librte_ring/meson.build index
> f2f3ccc88..612936afb 100644
> --- a/lib/librte_ring/meson.build
> +++ b/lib/librte_ring/meson.build
> @@ -5,7 +5,10 @@ sources = files('rte_ring.c') headers = files('rte_ring.h',
> 'rte_ring_elem.h',
> 'rte_ring_c11_mem.h',
> - 'rte_ring_generic.h')
> + 'rte_ring_generic.h',
> + 'rte_ring_rts.h',
> + 'rte_ring_rts_elem.h',
> + 'rte_ring_rts_generic.h')
>
> # rte_ring_create_elem and rte_ring_get_memsize_elem are experimental
> allow_experimental_apis = true diff --git a/lib/librte_ring/rte_ring.c
> b/lib/librte_ring/rte_ring.c index fa5733907..222eec0fb 100644
> --- a/lib/librte_ring/rte_ring.c
> +++ b/lib/librte_ring/rte_ring.c
> @@ -45,6 +45,9 @@ EAL_REGISTER_TAILQ(rte_ring_tailq)
> /* true if x is a power of 2 */
> #define POWEROF2(x) ((((x)-1) & (x)) == 0)
>
> +/* by default set head/tail distance as 1/8 of ring capacity */
> +#define HTD_MAX_DEF 8
> +
> /* return the size of memory occupied by a ring */ ssize_t
> rte_ring_get_memsize_elem(unsigned int esize, unsigned int count) @@ -
> 79,11 +82,84 @@ rte_ring_get_memsize(unsigned int count)
> return rte_ring_get_memsize_elem(sizeof(void *), count); }
>
> +/*
> + * internal helper function to reset prod/cons head-tail values.
> + */
> +static void
> +reset_headtail(void *p)
> +{
> + struct rte_ring_headtail *ht;
> + struct rte_ring_rts_headtail *ht_rts;
> +
> + ht = p;
> + ht_rts = p;
> +
> + switch (ht->sync_type) {
> + case RTE_RING_SYNC_MT:
> + case RTE_RING_SYNC_ST:
> + ht->head = 0;
> + ht->tail = 0;
> + break;
> + case RTE_RING_SYNC_MT_RTS:
> + ht_rts->head.raw = 0;
> + ht_rts->tail.raw = 0;
> + break;
> + default:
> + /* unknown sync mode */
> + RTE_ASSERT(0);
> + }
> +}
> +
> void
> rte_ring_reset(struct rte_ring *r)
> {
> - r->prod.head = r->cons.head = 0;
> - r->prod.tail = r->cons.tail = 0;
> + reset_headtail(&r->prod);
> + reset_headtail(&r->cons);
> +}
> +
> +/*
> + * helper function, calculates sync_type values for prod and cons
> + * based on input flags. Returns zero at success or negative
> + * errno value otherwise.
> + */
> +static int
> +get_sync_type(uint32_t flags, enum rte_ring_sync_type *prod_st,
> + enum rte_ring_sync_type *cons_st)
> +{
> + static const uint32_t prod_st_flags =
> + (RING_F_SP_ENQ | RING_F_MP_RTS_ENQ);
> + static const uint32_t cons_st_flags =
> + (RING_F_SC_DEQ | RING_F_MC_RTS_DEQ);
> +
> + switch (flags & prod_st_flags) {
> + case 0:
> + *prod_st = RTE_RING_SYNC_MT;
> + break;
> + case RING_F_SP_ENQ:
> + *prod_st = RTE_RING_SYNC_ST;
> + break;
> + case RING_F_MP_RTS_ENQ:
> + *prod_st = RTE_RING_SYNC_MT_RTS;
> + break;
> + default:
> + return -EINVAL;
> + }
> +
> + switch (flags & cons_st_flags) {
> + case 0:
> + *cons_st = RTE_RING_SYNC_MT;
> + break;
> + case RING_F_SC_DEQ:
> + *cons_st = RTE_RING_SYNC_ST;
> + break;
> + case RING_F_MC_RTS_DEQ:
> + *cons_st = RTE_RING_SYNC_MT_RTS;
> + break;
> + default:
> + return -EINVAL;
> + }
> +
> + return 0;
> }
>
> int
> @@ -100,16 +176,20 @@ rte_ring_init(struct rte_ring *r, const char *name,
> unsigned count,
> RTE_BUILD_BUG_ON((offsetof(struct rte_ring, prod) &
> RTE_CACHE_LINE_MASK) != 0);
>
> + RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, sync_type) !=
> + offsetof(struct rte_ring_rts_headtail, sync_type));
> + RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, tail) !=
> + offsetof(struct rte_ring_rts_headtail, tail.val.pos));
> +
> /* init the ring structure */
> memset(r, 0, sizeof(*r));
> ret = strlcpy(r->name, name, sizeof(r->name));
> if (ret < 0 || ret >= (int)sizeof(r->name))
> return -ENAMETOOLONG;
> r->flags = flags;
> - r->prod.sync_type = (flags & RING_F_SP_ENQ) ?
> - RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
> - r->cons.sync_type = (flags & RING_F_SC_DEQ) ?
> - RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
> + ret = get_sync_type(flags, &r->prod.sync_type, &r->cons.sync_type);
> + if (ret != 0)
> + return ret;
>
> if (flags & RING_F_EXACT_SZ) {
> r->size = rte_align32pow2(count + 1); @@ -126,8 +206,12
> @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
> r->mask = count - 1;
> r->capacity = r->mask;
> }
> - r->prod.head = r->cons.head = 0;
> - r->prod.tail = r->cons.tail = 0;
> +
> + /* set default values for head-tail distance */
> + if (flags & RING_F_MP_RTS_ENQ)
> + rte_ring_set_prod_htd_max(r, r->capacity / HTD_MAX_DEF);
> + if (flags & RING_F_MC_RTS_DEQ)
> + rte_ring_set_cons_htd_max(r, r->capacity / HTD_MAX_DEF);
>
> return 0;
> }
> diff --git a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h index
> d4775a063..f6f084d79 100644
> --- a/lib/librte_ring/rte_ring.h
> +++ b/lib/librte_ring/rte_ring.h
> @@ -48,6 +48,7 @@ extern "C" {
> #include <rte_branch_prediction.h>
> #include <rte_memzone.h>
> #include <rte_pause.h>
> +#include <rte_debug.h>
>
> #define RTE_TAILQ_RING_NAME "RTE_RING"
>
> @@ -65,10 +66,13 @@ enum rte_ring_queue_behavior { enum
> rte_ring_sync_type {
> RTE_RING_SYNC_MT, /**< multi-thread safe (default mode) */
> RTE_RING_SYNC_ST, /**< single thread only */
> +#ifdef ALLOW_EXPERIMENTAL_API
> + RTE_RING_SYNC_MT_RTS, /**< multi-thread relaxed tail sync */
> #endif
> };
>
> /**
> - * structure to hold a pair of head/tail values and other metadata.
> + * structures to hold a pair of head/tail values and other metadata.
> * Depending on sync_type format of that structure might be different,
> * but offset for *sync_type* and *tail* values should remain the same.
> */
> @@ -84,6 +88,21 @@ struct rte_ring_headtail {
> };
> };
>
> +union rte_ring_ht_poscnt {
nit, this is specific to RTS, may be change this to rte_ring_rts_ht_poscnt?
> + uint64_t raw;
> + struct {
> + uint32_t cnt; /**< head/tail reference counter */
> + uint32_t pos; /**< head/tail position */
> + } val;
> +};
> +
> +struct rte_ring_rts_headtail {
> + volatile union rte_ring_ht_poscnt tail;
> + enum rte_ring_sync_type sync_type; /**< sync type of prod/cons */
> + uint32_t htd_max; /**< max allowed distance between head/tail */
> + volatile union rte_ring_ht_poscnt head; };
> +
> /**
> * An RTE ring structure.
> *
> @@ -111,11 +130,21 @@ struct rte_ring {
> char pad0 __rte_cache_aligned; /**< empty cache line */
>
> /** Ring producer status. */
> - struct rte_ring_headtail prod __rte_cache_aligned;
> + RTE_STD_C11
> + union {
> + struct rte_ring_headtail prod;
> + struct rte_ring_rts_headtail rts_prod;
> + } __rte_cache_aligned;
> +
> char pad1 __rte_cache_aligned; /**< empty cache line */
>
> /** Ring consumer status. */
> - struct rte_ring_headtail cons __rte_cache_aligned;
> + RTE_STD_C11
> + union {
> + struct rte_ring_headtail cons;
> + struct rte_ring_rts_headtail rts_cons;
> + } __rte_cache_aligned;
> +
> char pad2 __rte_cache_aligned; /**< empty cache line */ };
>
> @@ -132,6 +161,9 @@ struct rte_ring {
> #define RING_F_EXACT_SZ 0x0004
> #define RTE_RING_SZ_MASK (0x7fffffffU) /**< Ring size mask */
>
> +#define RING_F_MP_RTS_ENQ 0x0008 /**< The default enqueue is "MP RTS".
> +*/ #define RING_F_MC_RTS_DEQ 0x0010 /**< The default dequeue is "MC
> +RTS". */
> +
> #define __IS_SP RTE_RING_SYNC_ST
> #define __IS_MP RTE_RING_SYNC_MT
> #define __IS_SC RTE_RING_SYNC_ST
> @@ -461,6 +493,10 @@ rte_ring_sp_enqueue_bulk(struct rte_ring *r, void *
> const *obj_table,
> RTE_RING_SYNC_ST, free_space);
> }
>
> +#ifdef ALLOW_EXPERIMENTAL_API
> +#include <rte_ring_rts.h>
> +#endif
> +
> /**
> * Enqueue several objects on a ring.
> *
> @@ -484,8 +520,21 @@ static __rte_always_inline unsigned int
> rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
> unsigned int n, unsigned int *free_space) {
> - return __rte_ring_do_enqueue(r, obj_table, n,
> RTE_RING_QUEUE_FIXED,
> - r->prod.sync_type, free_space);
> + switch (r->prod.sync_type) {
> + case RTE_RING_SYNC_MT:
> + return rte_ring_mp_enqueue_bulk(r, obj_table, n, free_space);
> + case RTE_RING_SYNC_ST:
> + return rte_ring_sp_enqueue_bulk(r, obj_table, n, free_space);
Have you validated if these affect the performance for the existing APIs?
I am also wondering why should we support these new modes in the legacy APIs?
I think users should move to use rte_ring_xxx_elem APIs. If users want to use RTS/HTS it will be a good time for them to move to new APIs. They anyway have to test their code for RTS/HTS, might as well make the change to new APIs and test both.
It will be less code to maintain for the community as well.
> #ifdef
> +ALLOW_EXPERIMENTAL_API
> + case RTE_RING_SYNC_MT_RTS:
> + return rte_ring_mp_rts_enqueue_bulk(r, obj_table, n,
> + free_space);
> +#endif
> + }
> +
> + /* valid ring should never reach this point */
> + RTE_ASSERT(0);
> + return 0;
> }
>
> /**
> @@ -619,8 +668,20 @@ static __rte_always_inline unsigned int
> rte_ring_dequeue_bulk(struct rte_ring *r, void **obj_table, unsigned int n,
> unsigned int *available)
> {
> - return __rte_ring_do_dequeue(r, obj_table, n,
> RTE_RING_QUEUE_FIXED,
> - r->cons.sync_type, available);
> + switch (r->cons.sync_type) {
> + case RTE_RING_SYNC_MT:
> + return rte_ring_mc_dequeue_bulk(r, obj_table, n, available);
> + case RTE_RING_SYNC_ST:
> + return rte_ring_sc_dequeue_bulk(r, obj_table, n, available);
> #ifdef
> +ALLOW_EXPERIMENTAL_API
> + case RTE_RING_SYNC_MT_RTS:
> + return rte_ring_mc_rts_dequeue_bulk(r, obj_table, n,
> available);
> +#endif
> + }
> +
> + /* valid ring should never reach this point */
> + RTE_ASSERT(0);
> + return 0;
> }
>
> /**
> @@ -940,8 +1001,21 @@ static __rte_always_inline unsigned
> rte_ring_enqueue_burst(struct rte_ring *r, void * const *obj_table,
> unsigned int n, unsigned int *free_space) {
> - return __rte_ring_do_enqueue(r, obj_table, n,
> RTE_RING_QUEUE_VARIABLE,
> - r->prod.sync_type, free_space);
> + switch (r->prod.sync_type) {
> + case RTE_RING_SYNC_MT:
> + return rte_ring_mp_enqueue_burst(r, obj_table, n,
> free_space);
> + case RTE_RING_SYNC_ST:
> + return rte_ring_sp_enqueue_burst(r, obj_table, n, free_space);
> #ifdef
> +ALLOW_EXPERIMENTAL_API
> + case RTE_RING_SYNC_MT_RTS:
> + return rte_ring_mp_rts_enqueue_burst(r, obj_table, n,
> + free_space);
> +#endif
> + }
> +
> + /* valid ring should never reach this point */
> + RTE_ASSERT(0);
> + return 0;
> }
>
> /**
> @@ -1020,9 +1094,21 @@ static __rte_always_inline unsigned
> rte_ring_dequeue_burst(struct rte_ring *r, void **obj_table,
> unsigned int n, unsigned int *available) {
> - return __rte_ring_do_dequeue(r, obj_table, n,
> - RTE_RING_QUEUE_VARIABLE,
> - r->cons.sync_type, available);
> + switch (r->cons.sync_type) {
> + case RTE_RING_SYNC_MT:
> + return rte_ring_mc_dequeue_burst(r, obj_table, n, available);
> + case RTE_RING_SYNC_ST:
> + return rte_ring_sc_dequeue_burst(r, obj_table, n, available);
> #ifdef
> +ALLOW_EXPERIMENTAL_API
> + case RTE_RING_SYNC_MT_RTS:
> + return rte_ring_mc_rts_dequeue_burst(r, obj_table, n,
> + available);
> +#endif
> + }
> +
> + /* valid ring should never reach this point */
> + RTE_ASSERT(0);
> + return 0;
> }
>
> #ifdef __cplusplus
> diff --git a/lib/librte_ring/rte_ring_elem.h b/lib/librte_ring/rte_ring_elem.h
> index 28f9836e6..5de0850dc 100644
> --- a/lib/librte_ring/rte_ring_elem.h
> +++ b/lib/librte_ring/rte_ring_elem.h
> @@ -542,6 +542,8 @@ rte_ring_sp_enqueue_bulk_elem(struct rte_ring *r,
> const void *obj_table,
> RTE_RING_QUEUE_FIXED, __IS_SP, free_space); }
>
> +#include <rte_ring_rts_elem.h>
> +
> /**
> * Enqueue several objects on a ring.
> *
> @@ -571,6 +573,26 @@ rte_ring_enqueue_bulk_elem(struct rte_ring *r,
> const void *obj_table, {
> return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
> RTE_RING_QUEUE_FIXED, r->prod.sync_type,
> free_space);
> +
> + switch (r->prod.sync_type) {
> + case RTE_RING_SYNC_MT:
> + return rte_ring_mp_enqueue_bulk_elem(r, obj_table, esize, n,
> + free_space);
> + case RTE_RING_SYNC_ST:
> + return rte_ring_sp_enqueue_bulk_elem(r, obj_table, esize, n,
> + free_space);
> +#ifdef ALLOW_EXPERIMENTAL_API
> + case RTE_RING_SYNC_MT_RTS:
> + return rte_ring_mp_rts_enqueue_bulk_elem(r, obj_table,
> esize, n,
> + free_space);
> +#endif
> + }
> +
> + /* valid ring should never reach this point */
> + RTE_ASSERT(0);
> + if (free_space != NULL)
> + *free_space = 0;
> + return 0;
> }
>
> /**
> @@ -733,8 +755,25 @@ static __rte_always_inline unsigned int
> rte_ring_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
> unsigned int esize, unsigned int n, unsigned int *available) {
> - return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
> - RTE_RING_QUEUE_FIXED, r->cons.sync_type,
> available);
> + switch (r->cons.sync_type) {
> + case RTE_RING_SYNC_MT:
> + return rte_ring_mc_dequeue_bulk_elem(r, obj_table, esize, n,
> + available);
> + case RTE_RING_SYNC_ST:
> + return rte_ring_sc_dequeue_bulk_elem(r, obj_table, esize, n,
> + available);
> +#ifdef ALLOW_EXPERIMENTAL_API
> + case RTE_RING_SYNC_MT_RTS:
> + return rte_ring_mc_rts_dequeue_bulk_elem(r, obj_table,
> esize,
> + n, available);
> +#endif
> + }
> +
> + /* valid ring should never reach this point */
> + RTE_ASSERT(0);
> + if (available != NULL)
> + *available = 0;
> + return 0;
> }
>
> /**
> @@ -901,8 +940,25 @@ static __rte_always_inline unsigned
> rte_ring_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
> unsigned int esize, unsigned int n, unsigned int *free_space) {
> - return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
> - RTE_RING_QUEUE_VARIABLE, r->prod.sync_type,
> free_space);
> + switch (r->prod.sync_type) {
> + case RTE_RING_SYNC_MT:
> + return rte_ring_mp_enqueue_burst_elem(r, obj_table, esize, n,
> + free_space);
> + case RTE_RING_SYNC_ST:
> + return rte_ring_sp_enqueue_burst_elem(r, obj_table, esize, n,
> + free_space);
> +#ifdef ALLOW_EXPERIMENTAL_API
> + case RTE_RING_SYNC_MT_RTS:
> + return rte_ring_mp_rts_enqueue_burst_elem(r, obj_table,
> esize,
> + n, free_space);
> +#endif
> + }
> +
> + /* valid ring should never reach this point */
> + RTE_ASSERT(0);
> + if (free_space != NULL)
> + *free_space = 0;
> + return 0;
> }
>
> /**
> @@ -993,9 +1049,25 @@ static __rte_always_inline unsigned int
> rte_ring_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
> unsigned int esize, unsigned int n, unsigned int *available) {
> - return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
> - RTE_RING_QUEUE_VARIABLE,
> - r->cons.sync_type, available);
> + switch (r->cons.sync_type) {
> + case RTE_RING_SYNC_MT:
> + return rte_ring_mc_dequeue_burst_elem(r, obj_table, esize, n,
> + available);
> + case RTE_RING_SYNC_ST:
> + return rte_ring_sc_dequeue_burst_elem(r, obj_table, esize, n,
> + available);
> +#ifdef ALLOW_EXPERIMENTAL_API
> + case RTE_RING_SYNC_MT_RTS:
> + return rte_ring_mc_rts_dequeue_burst_elem(r, obj_table,
> esize,
> + n, available);
> +#endif
> + }
> +
> + /* valid ring should never reach this point */
> + RTE_ASSERT(0);
> + if (available != NULL)
> + *available = 0;
> + return 0;
> }
>
> #ifdef __cplusplus
> diff --git a/lib/librte_ring/rte_ring_rts.h b/lib/librte_ring/rte_ring_rts.h new
> file mode 100644 index 000000000..18404fe48
> --- /dev/null
> +++ b/lib/librte_ring/rte_ring_rts.h
IMO, we should not provide these APIs.
> @@ -0,0 +1,316 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + *
> + * Copyright (c) 2010-2017 Intel Corporation
nit, the year should change to 2020? Look at others too.
> + * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
> + * All rights reserved.
> + * Derived from FreeBSD's bufring.h
> + * Used as BSD-3 Licensed with permission from Kip Macy.
> + */
> +
> +#ifndef _RTE_RING_RTS_H_
> +#define _RTE_RING_RTS_H_
> +
> +/**
> + * @file rte_ring_rts.h
> + * @b EXPERIMENTAL: this API may change without prior notice
> + * It is not recommended to include this file directly.
> + * Please include <rte_ring.h> instead.
> + *
> + * Contains functions for Relaxed Tail Sync (RTS) ring mode.
> + * The main idea remains the same as for our original MP/MC
^^^ the
> +synchronization
> + * mechanism.
> + * The main difference is that tail value is increased not
> + * by every thread that finished enqueue/dequeue,
> + * but only by the last one doing enqueue/dequeue.
should we say 'current last' or 'last thread at a given instance'?
> + * That allows threads to skip spinning on tail value,
> + * leaving actual tail value change to last thread in the update queue.
nit, I understand what you mean by 'update queue' here. IMO, we should remove it as it might confuse some.
> + * RTS requires 2 64-bit CAS for each enqueue(/dequeue) operation:
> + * one for head update, second for tail update.
> + * As a gain it allows thread to avoid spinning/waiting on tail value.
> + * In comparision original MP/MC algorithm requires one 32-bit CAS
> + * for head update and waiting/spinning on tail value.
> + *
> + * Brief outline:
> + * - introduce refcnt for both head and tail.
Suggesting using the same names as used in the structures.
> + * - increment head.refcnt for each head.value update
> + * - write head:value and head:refcnt atomically (64-bit CAS)
> + * - move tail.value ahead only when tail.refcnt + 1 == head.refcnt
May be add '(indicating that this is the last thread updating the tail)'
> + * - increment tail.refcnt when each enqueue/dequeue op finishes
May be add 'otherwise' at the beginning.
> + * (no matter is tail:value going to change or not)
nit ^^ if
> + * - write tail.value and tail.recnt atomically (64-bit CAS)
> + *
> + * To avoid producer/consumer starvation:
> + * - limit max allowed distance between head and tail value (HTD_MAX).
> + * I.E. thread is allowed to proceed with changing head.value,
> + * only when: head.value - tail.value <= HTD_MAX
> + * HTD_MAX is an optional parameter.
> + * With HTD_MAX == 0 we'll have fully serialized ring -
> + * i.e. only one thread at a time will be able to enqueue/dequeue
> + * to/from the ring.
> + * With HTD_MAX >= ring.capacity - no limitation.
> + * By default HTD_MAX == ring.capacity / 8.
> + */
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +#include <rte_ring_rts_generic.h>
> +
> +/**
> + * @internal Enqueue several objects on the RTS ring.
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects).
> + * @param n
> + * The number of objects to add in the ring from the obj_table.
> + * @param behavior
> + * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
> + * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from
> ring
> + * @param free_space
> + * returns the amount of space after the enqueue operation has finished
> + * @return
> + * Actual number of objects enqueued.
> + * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> + */
> +static __rte_always_inline unsigned int
> +__rte_ring_do_rts_enqueue(struct rte_ring *r, void * const *obj_table,
> + uint32_t n, enum rte_ring_queue_behavior behavior,
> + uint32_t *free_space)
> +{
> + uint32_t free, head;
> +
> + n = __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
> +
> + if (n != 0) {
> + ENQUEUE_PTRS(r, &r[1], head, obj_table, n, void *);
> + __rte_ring_rts_update_tail(&r->rts_prod);
> + }
> +
> + if (free_space != NULL)
> + *free_space = free - n;
> + return n;
> +}
> +
> +/**
> + * @internal Dequeue several objects from the RTS ring.
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects).
> + * @param n
> + * The number of objects to pull from the ring.
> + * @param behavior
> + * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
> + * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from
> ring
> + * @param available
> + * returns the number of remaining ring entries after the dequeue has
> finished
> + * @return
> + * - Actual number of objects dequeued.
> + * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> + */
> +static __rte_always_inline unsigned int
> +__rte_ring_do_rts_dequeue(struct rte_ring *r, void **obj_table,
> + uint32_t n, enum rte_ring_queue_behavior behavior,
> + uint32_t *available)
> +{
> + uint32_t entries, head;
> +
> + n = __rte_ring_rts_move_cons_head(r, n, behavior, &head, &entries);
> +
> + if (n != 0) {
> + DEQUEUE_PTRS(r, &r[1], head, obj_table, n, void *);
> + __rte_ring_rts_update_tail(&r->rts_cons);
> + }
> +
> + if (available != NULL)
> + *available = entries - n;
> + return n;
> +}
> +
> +/**
> + * Enqueue several objects on the RTS ring (multi-producers safe).
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects).
> + * @param n
> + * The number of objects to add in the ring from the obj_table.
> + * @param free_space
> + * if non-NULL, returns the amount of space in the ring after the
> + * enqueue operation has finished.
> + * @return
> + * The number of objects enqueued, either 0 or n
> + */
> +__rte_experimental
> +static __rte_always_inline unsigned int
> +rte_ring_mp_rts_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
> + unsigned int n, unsigned int *free_space) {
> + return __rte_ring_do_rts_enqueue(r, obj_table, n,
> RTE_RING_QUEUE_FIXED,
> + free_space);
> +}
> +
> +/**
> + * Dequeue several objects from an RTS ring (multi-consumers safe).
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects) that will be filled.
> + * @param n
> + * The number of objects to dequeue from the ring to the obj_table.
> + * @param available
> + * If non-NULL, returns the number of remaining ring entries after the
> + * dequeue has finished.
> + * @return
> + * The number of objects dequeued, either 0 or n
> + */
> +__rte_experimental
> +static __rte_always_inline unsigned int
> +rte_ring_mc_rts_dequeue_bulk(struct rte_ring *r, void **obj_table,
> + unsigned int n, unsigned int *available) {
> + return __rte_ring_do_rts_dequeue(r, obj_table, n,
> RTE_RING_QUEUE_FIXED,
> + available);
> +}
> +
> +/**
> + * Return producer max Head-Tail-Distance (HTD).
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @return
> + * Producer HTD value, if producer is set in appropriate sync mode,
> + * or UINT32_MAX otherwise.
> + */
> +__rte_experimental
> +static inline uint32_t
> +rte_ring_get_prod_htd_max(const struct rte_ring *r) {
> + if (r->prod.sync_type == RTE_RING_SYNC_MT_RTS)
> + return r->rts_prod.htd_max;
> + return UINT32_MAX;
> +}
> +
> +/**
> + * Set producer max Head-Tail-Distance (HTD).
> + * Note that producer has to use appropriate sync mode (RTS).
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param v
> + * new HTD value to setup.
> + * @return
> + * Zero on success, or negative error code otherwise.
> + */
> +__rte_experimental
> +static inline int
> +rte_ring_set_prod_htd_max(struct rte_ring *r, uint32_t v) {
> + if (r->prod.sync_type != RTE_RING_SYNC_MT_RTS)
> + return -ENOTSUP;
> +
> + r->rts_prod.htd_max = v;
> + return 0;
> +}
> +
> +/**
> + * Return consumer max Head-Tail-Distance (HTD).
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @return
> + * Consumer HTD value, if consumer is set in appropriate sync mode,
> + * or UINT32_MAX otherwise.
> + */
> +__rte_experimental
> +static inline uint32_t
> +rte_ring_get_cons_htd_max(const struct rte_ring *r) {
> + if (r->cons.sync_type == RTE_RING_SYNC_MT_RTS)
> + return r->rts_cons.htd_max;
> + return UINT32_MAX;
> +}
> +
> +/**
> + * Set consumer max Head-Tail-Distance (HTD).
> + * Note that consumer has to use appropriate sync mode (RTS).
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param v
> + * new HTD value to setup.
> + * @return
> + * Zero on success, or negative error code otherwise.
> + */
> +__rte_experimental
> +static inline int
> +rte_ring_set_cons_htd_max(struct rte_ring *r, uint32_t v) {
> + if (r->cons.sync_type != RTE_RING_SYNC_MT_RTS)
> + return -ENOTSUP;
> +
> + r->rts_cons.htd_max = v;
> + return 0;
> +}
> +
> +/**
> + * Enqueue several objects on the RTS ring (multi-producers safe).
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects).
> + * @param n
> + * The number of objects to add in the ring from the obj_table.
> + * @param free_space
> + * if non-NULL, returns the amount of space in the ring after the
> + * enqueue operation has finished.
> + * @return
> + * - n: Actual number of objects enqueued.
> + */
> +__rte_experimental
> +static __rte_always_inline unsigned
> +rte_ring_mp_rts_enqueue_burst(struct rte_ring *r, void * const *obj_table,
> + unsigned int n, unsigned int *free_space) {
> + return __rte_ring_do_rts_enqueue(r, obj_table, n,
> + RTE_RING_QUEUE_VARIABLE, free_space); }
> +
> +/**
> + * Dequeue several objects from an RTS ring (multi-consumers safe).
> + * When the requested objects are more than the available objects,
> + * only dequeue the actual number of objects.
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects) that will be filled.
> + * @param n
> + * The number of objects to dequeue from the ring to the obj_table.
> + * @param available
> + * If non-NULL, returns the number of remaining ring entries after the
> + * dequeue has finished.
> + * @return
> + * - n: Actual number of objects dequeued, 0 if ring is empty
> + */
> +__rte_experimental
> +static __rte_always_inline unsigned
> +rte_ring_mc_rts_dequeue_burst(struct rte_ring *r, void **obj_table,
> + unsigned int n, unsigned int *available) {
> + return __rte_ring_do_rts_dequeue(r, obj_table, n,
> + RTE_RING_QUEUE_VARIABLE, available); }
> +
> +#ifdef __cplusplus
> +}
> +#endif
> +
> +#endif /* _RTE_RING_RTS_H_ */
> diff --git a/lib/librte_ring/rte_ring_rts_elem.h
> b/lib/librte_ring/rte_ring_rts_elem.h
> new file mode 100644
> index 000000000..71a331b23
> --- /dev/null
> +++ b/lib/librte_ring/rte_ring_rts_elem.h
> @@ -0,0 +1,205 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + *
> + * Copyright (c) 2010-2017 Intel Corporation
> + * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
> + * All rights reserved.
> + * Derived from FreeBSD's bufring.h
> + * Used as BSD-3 Licensed with permission from Kip Macy.
> + */
> +
> +#ifndef _RTE_RING_RTS_ELEM_H_
> +#define _RTE_RING_RTS_ELEM_H_
> +
> +/**
> + * @file rte_ring_rts_elem.h
> + * @b EXPERIMENTAL: this API may change without prior notice
> + *
> + * It is not recommended to include this file directly.
> + * Please include <rte_ring_elem.h> instead.
> + * Contains *ring_elem* functions for Relaxed Tail Sync (RTS) ring mode.
> + * for more details please refer to <rte_ring_rts.h>.
> + */
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +#include <rte_ring_rts_generic.h>
> +
> +/**
> + * @internal Enqueue several objects on the RTS ring.
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects).
> + * @param n
> + * The number of objects to add in the ring from the obj_table.
> + * @param behavior
> + * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
> + * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from
> ring
> + * @param free_space
> + * returns the amount of space after the enqueue operation has finished
> + * @return
> + * Actual number of objects enqueued.
> + * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> + */
> +static __rte_always_inline unsigned int
> +__rte_ring_do_rts_enqueue_elem(struct rte_ring *r, void * const *obj_table,
obj_table should be of type 'const void * obj_table' (looks like copy paste error). Please check the other APIs below too.
> + uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
'esize' is not documented in the comments above the function. You can copy the header from rte_ring_elem.h file. Please check other APIs as well.
> + uint32_t *free_space)
> +{
> + uint32_t free, head;
> +
> + n = __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
> +
> + if (n != 0) {
> + __rte_ring_enqueue_elems(r, head, obj_table, esize, n);
> + __rte_ring_rts_update_tail(&r->rts_prod);
> + }
> +
> + if (free_space != NULL)
> + *free_space = free - n;
> + return n;
> +}
> +
> +/**
> + * @internal Dequeue several objects from the RTS ring.
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects).
> + * @param n
> + * The number of objects to pull from the ring.
> + * @param behavior
> + * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
> + * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from
> ring
> + * @param available
> + * returns the number of remaining ring entries after the dequeue has
> finished
> + * @return
> + * - Actual number of objects dequeued.
> + * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> + */
> +static __rte_always_inline unsigned int
> +__rte_ring_do_rts_dequeue_elem(struct rte_ring *r, void **obj_table,
> + uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
> + uint32_t *available)
> +{
> + uint32_t entries, head;
> +
> + n = __rte_ring_rts_move_cons_head(r, n, behavior, &head, &entries);
> +
> + if (n != 0) {
> + __rte_ring_dequeue_elems(r, head, obj_table, esize, n);
> + __rte_ring_rts_update_tail(&r->rts_cons);
> + }
> +
> + if (available != NULL)
> + *available = entries - n;
> + return n;
> +}
> +
> +/**
> + * Enqueue several objects on the RTS ring (multi-producers safe).
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects).
> + * @param n
> + * The number of objects to add in the ring from the obj_table.
> + * @param free_space
> + * if non-NULL, returns the amount of space in the ring after the
> + * enqueue operation has finished.
> + * @return
> + * The number of objects enqueued, either 0 or n
> + */
> +__rte_experimental
> +static __rte_always_inline unsigned int
> +rte_ring_mp_rts_enqueue_bulk_elem(struct rte_ring *r, void * const
> *obj_table,
> + unsigned int esize, unsigned int n, unsigned int *free_space) {
> + return __rte_ring_do_rts_enqueue_elem(r, obj_table, esize, n,
> + RTE_RING_QUEUE_FIXED, free_space);
> +}
> +
> +/**
> + * Dequeue several objects from an RTS ring (multi-consumers safe).
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects) that will be filled.
> + * @param n
> + * The number of objects to dequeue from the ring to the obj_table.
> + * @param available
> + * If non-NULL, returns the number of remaining ring entries after the
> + * dequeue has finished.
> + * @return
> + * The number of objects dequeued, either 0 or n
> + */
> +__rte_experimental
> +static __rte_always_inline unsigned int
> +rte_ring_mc_rts_dequeue_bulk_elem(struct rte_ring *r, void **obj_table,
> + unsigned int esize, unsigned int n, unsigned int *available) {
> + return __rte_ring_do_rts_dequeue_elem(r, obj_table, esize, n,
> + RTE_RING_QUEUE_FIXED, available);
> +}
> +
> +/**
> + * Enqueue several objects on the RTS ring (multi-producers safe).
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects).
> + * @param n
> + * The number of objects to add in the ring from the obj_table.
> + * @param free_space
> + * if non-NULL, returns the amount of space in the ring after the
> + * enqueue operation has finished.
> + * @return
> + * - n: Actual number of objects enqueued.
> + */
> +__rte_experimental
> +static __rte_always_inline unsigned
> +rte_ring_mp_rts_enqueue_burst_elem(struct rte_ring *r, void * const
> *obj_table,
> + unsigned int esize, unsigned int n, unsigned int *free_space) {
> + return __rte_ring_do_rts_enqueue_elem(r, obj_table, esize, n,
> + RTE_RING_QUEUE_VARIABLE, free_space); }
> +
> +/**
> + * Dequeue several objects from an RTS ring (multi-consumers safe).
> + * When the requested objects are more than the available objects,
> + * only dequeue the actual number of objects.
> + *
> + * @param r
> + * A pointer to the ring structure.
> + * @param obj_table
> + * A pointer to a table of void * pointers (objects) that will be filled.
> + * @param n
> + * The number of objects to dequeue from the ring to the obj_table.
> + * @param available
> + * If non-NULL, returns the number of remaining ring entries after the
> + * dequeue has finished.
> + * @return
> + * - n: Actual number of objects dequeued, 0 if ring is empty
> + */
> +__rte_experimental
> +static __rte_always_inline unsigned
> +rte_ring_mc_rts_dequeue_burst_elem(struct rte_ring *r, void **obj_table,
> + unsigned int esize, unsigned int n, unsigned int *available) {
> + return __rte_ring_do_rts_dequeue_elem(r, obj_table, esize, n,
> + RTE_RING_QUEUE_VARIABLE, available); }
> +
> +#ifdef __cplusplus
> +}
> +#endif
> +
> +#endif /* _RTE_RING_RTS_ELEM_H_ */
> diff --git a/lib/librte_ring/rte_ring_rts_generic.h
> b/lib/librte_ring/rte_ring_rts_generic.h
> new file mode 100644
> index 000000000..f88460d47
> --- /dev/null
> +++ b/lib/librte_ring/rte_ring_rts_generic.h
I do not know the benefit to providing the generic version. Do you know why this was done in the legacy APIs?
If there is no performance difference between generic and C11 versions, should we just skip the generic version?
The oldest compiler in CI are GCC 4.8.5 and Clang 3.4.2 and C11 built-ins are supported earlier than these compiler versions.
I feel the code is growing exponentially in rte_ring library and we should try to cut non-value-ass code/APIs aggressively.
> @@ -0,0 +1,210 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + *
> + * Copyright (c) 2010-2017 Intel Corporation
> + * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
> + * All rights reserved.
> + * Derived from FreeBSD's bufring.h
> + * Used as BSD-3 Licensed with permission from Kip Macy.
> + */
> +
> +#ifndef _RTE_RING_RTS_GENERIC_H_
> +#define _RTE_RING_RTS_GENERIC_H_
> +
> +/**
> + * @file rte_ring_rts_generic.h
> + * It is not recommended to include this file directly,
> + * include <rte_ring.h> instead.
> + * Contains internal helper functions for Relaxed Tail Sync (RTS) ring mode.
> + * For more information please refer to <rte_ring_rts.h>.
> + */
> +
> +/**
> + * @internal This function updates tail values.
> + */
> +static __rte_always_inline void
> +__rte_ring_rts_update_tail(struct rte_ring_rts_headtail *ht) {
> + union rte_ring_ht_poscnt h, ot, nt;
> +
> + /*
> + * If there are other enqueues/dequeues in progress that
> + * might preceded us, then don't update tail with new value.
> + */
> +
> + do {
> + ot.raw = ht->tail.raw;
> + rte_smp_rmb();
> +
> + /* on 32-bit systems we have to do atomic read here */
> + h.raw = rte_atomic64_read((rte_atomic64_t *)
> + (uintptr_t)&ht->head.raw);
> +
> + nt.raw = ot.raw;
> + if (++nt.val.cnt == h.val.cnt)
> + nt.val.pos = h.val.pos;
> +
> + } while (rte_atomic64_cmpset(&ht->tail.raw, ot.raw, nt.raw) == 0); }
> +
> +/**
> + * @internal This function waits till head/tail distance wouldn't
> + * exceed pre-defined max value.
> + */
> +static __rte_always_inline void
> +__rte_ring_rts_head_wait(const struct rte_ring_rts_headtail *ht,
> + union rte_ring_ht_poscnt *h)
> +{
> + uint32_t max;
> +
> + max = ht->htd_max;
> + h->raw = ht->head.raw;
> + rte_smp_rmb();
> +
> + while (h->val.pos - ht->tail.val.pos > max) {
> + rte_pause();
> + h->raw = ht->head.raw;
> + rte_smp_rmb();
> + }
> +}
> +
> +/**
> + * @internal This function updates the producer head for enqueue.
> + *
> + * @param r
> + * A pointer to the ring structure
> + * @param is_sp
> + * Indicates whether multi-producer path is needed or not
> + * @param n
> + * The number of elements we will want to enqueue, i.e. how far should the
> + * head be moved
> + * @param behavior
> + * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
> + * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from
> ring
> + * @param old_head
> + * Returns head value as it was before the move, i.e. where enqueue starts
> + * @param new_head
> + * Returns the current/new head value i.e. where enqueue finishes
> + * @param free_entries
> + * Returns the amount of free space in the ring BEFORE head was moved
> + * @return
> + * Actual number of objects enqueued.
> + * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> + */
> +static __rte_always_inline uint32_t
> +__rte_ring_rts_move_prod_head(struct rte_ring *r, uint32_t num,
> + enum rte_ring_queue_behavior behavior, uint32_t *old_head,
> + uint32_t *free_entries)
> +{
> + uint32_t n;
> + union rte_ring_ht_poscnt nh, oh;
> +
> + const uint32_t capacity = r->capacity;
> +
> + do {
> + /* Reset n to the initial burst count */
> + n = num;
> +
> + /* read prod head (may spin on prod tail) */
> + __rte_ring_rts_head_wait(&r->rts_prod, &oh);
> +
> + /* add rmb barrier to avoid load/load reorder in weak
> + * memory model. It is noop on x86
> + */
> + rte_smp_rmb();
> +
> + /*
> + * The subtraction is done between two unsigned 32bits value
> + * (the result is always modulo 32 bits even if we have
> + * *old_head > cons_tail). So 'free_entries' is always between
> 0
> + * and capacity (which is < size).
> + */
> + *free_entries = capacity + r->cons.tail - oh.val.pos;
> +
> + /* check that we have enough room in ring */
> + if (unlikely(n > *free_entries))
> + n = (behavior == RTE_RING_QUEUE_FIXED) ?
> + 0 : *free_entries;
> +
> + if (n == 0)
> + break;
> +
> + nh.val.pos = oh.val.pos + n;
> + nh.val.cnt = oh.val.cnt + 1;
> +
> + } while (rte_atomic64_cmpset(&r->rts_prod.head.raw,
> + oh.raw, nh.raw) == 0);
> +
> + *old_head = oh.val.pos;
> + return n;
> +}
> +
> +/**
> + * @internal This function updates the consumer head for dequeue
> + *
> + * @param r
> + * A pointer to the ring structure
> + * @param is_sc
> + * Indicates whether multi-consumer path is needed or not
> + * @param n
> + * The number of elements we will want to enqueue, i.e. how far should the
> + * head be moved
> + * @param behavior
> + * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
> + * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from
> ring
> + * @param old_head
> + * Returns head value as it was before the move, i.e. where dequeue starts
> + * @param new_head
> + * Returns the current/new head value i.e. where dequeue finishes
> + * @param entries
> + * Returns the number of entries in the ring BEFORE head was moved
> + * @return
> + * - Actual number of objects dequeued.
> + * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> + */
> +static __rte_always_inline unsigned int
> +__rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num,
> + enum rte_ring_queue_behavior behavior, uint32_t *old_head,
> + uint32_t *entries)
> +{
> + uint32_t n;
> + union rte_ring_ht_poscnt nh, oh;
> +
> + /* move cons.head atomically */
> + do {
> + /* Restore n as it may change every loop */
> + n = num;
> +
> + /* read cons head (may spin on cons tail) */
> + __rte_ring_rts_head_wait(&r->rts_cons, &oh);
> +
> +
> + /* add rmb barrier to avoid load/load reorder in weak
> + * memory model. It is noop on x86
> + */
> + rte_smp_rmb();
> +
> + /* The subtraction is done between two unsigned 32bits value
> + * (the result is always modulo 32 bits even if we have
> + * cons_head > prod_tail). So 'entries' is always between 0
> + * and size(ring)-1.
> + */
> + *entries = r->prod.tail - oh.val.pos;
> +
> + /* Set the actual entries for dequeue */
> + if (n > *entries)
> + n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 :
> *entries;
> +
> + if (unlikely(n == 0))
> + break;
> +
> + nh.val.pos = oh.val.pos + n;
> + nh.val.cnt = oh.val.cnt + 1;
> +
> + } while (rte_atomic64_cmpset(&r->rts_cons.head.raw,
> + oh.raw, nh.raw) == 0);
> +
> + *old_head = oh.val.pos;
> + return n;
> +}
> +
> +#endif /* _RTE_RING_RTS_GENERIC_H_ */
> --
> 2.17.1
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 00/16] NXP DPAAx fixes and enhancements
2020-04-07 10:25 3% ` Hemant Agrawal
2020-04-07 12:20 0% ` Thomas Monjalon
@ 2020-04-08 7:20 4% ` Dodji Seketeli
2020-04-08 7:52 0% ` Dodji Seketeli
1 sibling, 1 reply; 200+ results
From: Dodji Seketeli @ 2020-04-08 7:20 UTC (permalink / raw)
To: Hemant Agrawal
Cc: Dodji Seketeli, David Marchand, Hemant Agrawal (OSS),
Yigit, Ferruh, dev, Neil Horman, Thomas Monjalon
Hello Hemant,
Hemant Agrawal <hemant.agrawal@nxp.com> writes:
[...]
>> >> > > [Hemant]
>> >> > > As per the logs:
>> >> > >
>> >> > > Variables changes summary: 1 Removed, 2 Changed, 0 Added
>> >> > > variables
>> >> > > 1 Removed variable:
>> >> > > 'dpaa2_portal_dqrr per_lcore_dpaa2_held_bufs'
>> >> > {per_lcore_dpaa2_held_bufs@@DPDK_20.0}
>> >> > > 2 Changed variables:
>> >> > > [C]'dpaa2_io_portal_t dpaa2_io_portal[128]' was changed at
>> >> > dpaa2_hw_dpio.h:40:1: size of symbol changed from 5120 to 2048
>> >> > > [C]'dpaa2_io_portal_t per_lcore__dpaa2_io' was changed at
>> >> > > dpaa2_hw_dpio.h:20:1: size of symbol changed from 40 to 16
>> >> > >
>> >> > > Error: ABI issue reported for 'abidiff --suppr
>> >> > > devtools/libabigail.abignore --
>> >> > no-added-syms --headers-dir1 reference/usr/local/include
>> >> > --headers-dir2 install/usr/local/include
>> >> > reference/dump/librte_bus_fslmc.dump
>> >> > install/dump/librte_bus_fslmc.dump'
[...]
>> In the mean time, the tooling can be tought to ignore changes to these ELF
>> symbols, as you you guys all know already.
>>
> [Hemant] will you please help me about adding entry to libagigail.abignore
> I tried doing following, but it is not helping
> --- a/devtools/libabigail.abignore
> +++ b/devtools/libabigail.abignore
> @@ -2,10 +2,15 @@
> symbol_version = EXPERIMENTAL
> [suppress_variable]
> symbol_version = EXPERIMENTAL
> + name = per_lcore__dpaa2_io
> + name = dpaa2_io_portal
>
> ; Explicit ignore for driver-only ABI
> [suppress_type]
> name = rte_cryptodev_ops
> + name = dpaa2_io_portal_t
So, I understand you want the tooling to ignore changes to the global
arrays dpaa2_io_portal and per_lcore__dpaa2_io, right?
If that is correct, then here are the entries you should add to the
libabigail.abignore file (please make sure the comments I have added
there is accurate):
[suppress_variable]
# This global variable is exported by the binary but is not part of
# the logical ABI. In a perfect world, that variable should not be
# global, and we should access it via an accessor function. We do
# that right now because of performance concerns.
name = dpaa2_io_portal
[suppress_variable]
# This global variable is exported by the binary but is not part of
# the logical ABI. In a perfect world, that variable should not be
# global, and we should access it via an accessor function. We do
# that right now because of performance concerns.
name = per_lcore__dpaa2_io
I hope this helps.
Cheers,
--
Dodji
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH 00/16] NXP DPAAx fixes and enhancements
2020-04-08 7:20 4% ` Dodji Seketeli
@ 2020-04-08 7:52 0% ` Dodji Seketeli
0 siblings, 0 replies; 200+ results
From: Dodji Seketeli @ 2020-04-08 7:52 UTC (permalink / raw)
To: Dodji Seketeli
Cc: Hemant Agrawal, David Marchand, Hemant Agrawal (OSS),
Yigit, Ferruh, dev, Neil Horman, Thomas Monjalon
Hello Thomas, Hemant,
Thomas Monjalon <thomas@monjalon.net> writes:
> 07/04/2020 12:25, Hemant Agrawal:
[...]
>> [Hemant] I have commented on Neil's series.
>> It needs more changes in existing code.
>> An approach like __rte_experimental will work better.
>
> I guess you mean __rte_internal?
>
> Please Hemant don't wait for someone else filling the gap.
> If __rte_internal is the right approach, please complete and use it.
Just so that I understand, is __rte_internal an ELF version that the
symbols per_lcore_dpaa2_held_bufs, dpaa2_io_portal and
per_lcore__dpaa2_io should have in the binary?
If that is the case, then it seems to me that the __rte_internal
approach that you are suggesting would be a much better approach that
the one I replied to Hemant about below.
I didn't mean to tell Hemant what approach he should take :-) I was just
trying to help him get the syntax of a libabigail suppression
specification right.
Sorry for the confusion I might have induced.
Dodji Seketeli <dseketel@redhat.com> writes:
> Hello Hemant,
>
> Hemant Agrawal <hemant.agrawal@nxp.com> writes:
>
> [...]
>
>>> >> > > [Hemant]
>>> >> > > As per the logs:
>>> >> > >
>>> >> > > Variables changes summary: 1 Removed, 2 Changed, 0 Added
>>> >> > > variables
>>> >> > > 1 Removed variable:
>>> >> > > 'dpaa2_portal_dqrr per_lcore_dpaa2_held_bufs'
>>> >> > {per_lcore_dpaa2_held_bufs@@DPDK_20.0}
>>> >> > > 2 Changed variables:
>>> >> > > [C]'dpaa2_io_portal_t dpaa2_io_portal[128]' was changed at
>>> >> > dpaa2_hw_dpio.h:40:1: size of symbol changed from 5120 to 2048
>>> >> > > [C]'dpaa2_io_portal_t per_lcore__dpaa2_io' was changed at
>>> >> > > dpaa2_hw_dpio.h:20:1: size of symbol changed from 40 to 16
>>> >> > >
>>> >> > > Error: ABI issue reported for 'abidiff --suppr
>>> >> > > devtools/libabigail.abignore --
>>> >> > no-added-syms --headers-dir1 reference/usr/local/include
>>> >> > --headers-dir2 install/usr/local/include
>>> >> > reference/dump/librte_bus_fslmc.dump
>>> >> > install/dump/librte_bus_fslmc.dump'
>
> [...]
>
>>> In the mean time, the tooling can be tought to ignore changes to these ELF
>>> symbols, as you you guys all know already.
>>>
>> [Hemant] will you please help me about adding entry to libagigail.abignore
>> I tried doing following, but it is not helping
>> --- a/devtools/libabigail.abignore
>> +++ b/devtools/libabigail.abignore
>> @@ -2,10 +2,15 @@
>> symbol_version = EXPERIMENTAL
>> [suppress_variable]
>> symbol_version = EXPERIMENTAL
>> + name = per_lcore__dpaa2_io
>> + name = dpaa2_io_portal
>>
>> ; Explicit ignore for driver-only ABI
>> [suppress_type]
>> name = rte_cryptodev_ops
>> + name = dpaa2_io_portal_t
>
> So, I understand you want the tooling to ignore changes to the global
> arrays dpaa2_io_portal and per_lcore__dpaa2_io, right?
>
> If that is correct, then here are the entries you should add to the
> libabigail.abignore file (please make sure the comments I have added
> there is accurate):
>
> [suppress_variable]
> # This global variable is exported by the binary but is not part of
> # the logical ABI. In a perfect world, that variable should not be
> # global, and we should access it via an accessor function. We do
> # that right now because of performance concerns.
> name = dpaa2_io_portal
>
> [suppress_variable]
> # This global variable is exported by the binary but is not part of
> # the logical ABI. In a perfect world, that variable should not be
> # global, and we should access it via an accessor function. We do
> # that right now because of performance concerns.
> name = per_lcore__dpaa2_io
>
> I hope this helps.
>
> Cheers,
--
Dodji
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v3 07/12] service: remove rte prefix from static functions
2020-04-03 11:57 3% ` Van Haaren, Harry
@ 2020-04-08 10:14 0% ` Phil Yang
2020-04-08 10:36 0% ` Van Haaren, Harry
0 siblings, 1 reply; 200+ results
From: Phil Yang @ 2020-04-08 10:14 UTC (permalink / raw)
To: Van Haaren, Harry, thomas, Ananyev, Konstantin, stephen,
maxime.coquelin, dev
Cc: david.marchand, jerinj, hemant.agrawal, Honnappa Nagarahalli,
Gavin Hu, Ruifeng Wang, Joyce Kong, nd, stable, nd
> -----Original Message-----
> From: Van Haaren, Harry <harry.van.haaren@intel.com>
> Sent: Friday, April 3, 2020 7:58 PM
> To: Phil Yang <Phil.Yang@arm.com>; thomas@monjalon.net; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>;
> stephen@networkplumber.org; maxime.coquelin@redhat.com;
> dev@dpdk.org
> Cc: david.marchand@redhat.com; jerinj@marvell.com;
> hemant.agrawal@nxp.com; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; Gavin Hu <Gavin.Hu@arm.com>;
> Ruifeng Wang <Ruifeng.Wang@arm.com>; Joyce Kong
> <Joyce.Kong@arm.com>; nd <nd@arm.com>; stable@dpdk.org
> Subject: RE: [PATCH v3 07/12] service: remove rte prefix from static functions
>
> > From: Phil Yang <phil.yang@arm.com>
> > Sent: Tuesday, March 17, 2020 1:18 AM
> > To: thomas@monjalon.net; Van Haaren, Harry
> <harry.van.haaren@intel.com>;
> > Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> > stephen@networkplumber.org; maxime.coquelin@redhat.com;
> dev@dpdk.org
> > Cc: david.marchand@redhat.com; jerinj@marvell.com;
> hemant.agrawal@nxp.com;
> > Honnappa.Nagarahalli@arm.com; gavin.hu@arm.com;
> ruifeng.wang@arm.com;
> > joyce.kong@arm.com; nd@arm.com; stable@dpdk.org
> > Subject: [PATCH v3 07/12] service: remove rte prefix from static functions
> >
> > Fixes: 3cf5eb1546ed ("service: fix and refactor atomic service accesses")
> > Fixes: 21698354c832 ("service: introduce service cores concept")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Phil Yang <phil.yang@arm.com>
> > Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
>
>
> This patchset needs a rebase since the EAL file movement got merged,
> however I'll review here so we can include some Acks etc and make
> progress.
>
> Is this really a "Fix"? The internal function names were not exported
> in the .map file, so are not part of public ABI. This is an internal
> naming improvement (thanks for doing cleanup), but I don't think the
> Fixes: tags make sense?
>
> Also I'm not sure if we want to port this patch back to stable? Changing
> (internal) function names seems like unnecessary churn, and hence risk to a
> stable release, without any benefit?
OK.
I will remove these tags in the next version and split the service core patches from the original series into a series by itself.
Thanks,
Phil
>
> ---
>
> <snip patch diff>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v3 07/12] service: remove rte prefix from static functions
2020-04-08 10:14 0% ` Phil Yang
@ 2020-04-08 10:36 0% ` Van Haaren, Harry
2020-04-08 10:49 0% ` Phil Yang
0 siblings, 1 reply; 200+ results
From: Van Haaren, Harry @ 2020-04-08 10:36 UTC (permalink / raw)
To: Phil Yang, thomas, Ananyev, Konstantin, stephen, maxime.coquelin, dev
Cc: david.marchand, jerinj, hemant.agrawal, Honnappa Nagarahalli,
Gavin Hu, Ruifeng Wang, Joyce Kong, nd, stable, nd
> -----Original Message-----
> From: Phil Yang <Phil.Yang@arm.com>
> Sent: Wednesday, April 8, 2020 11:15 AM
> To: Van Haaren, Harry <harry.van.haaren@intel.com>; thomas@monjalon.net;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> stephen@networkplumber.org; maxime.coquelin@redhat.com; dev@dpdk.org
> Cc: david.marchand@redhat.com; jerinj@marvell.com; hemant.agrawal@nxp.com;
> Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Gavin Hu
> <Gavin.Hu@arm.com>; Ruifeng Wang <Ruifeng.Wang@arm.com>; Joyce Kong
> <Joyce.Kong@arm.com>; nd <nd@arm.com>; stable@dpdk.org; nd <nd@arm.com>
> Subject: RE: [PATCH v3 07/12] service: remove rte prefix from static functions
<snip>
> > Is this really a "Fix"? The internal function names were not exported
> > in the .map file, so are not part of public ABI. This is an internal
> > naming improvement (thanks for doing cleanup), but I don't think the
> > Fixes: tags make sense?
> >
> > Also I'm not sure if we want to port this patch back to stable? Changing
> > (internal) function names seems like unnecessary churn, and hence risk to a
> > stable release, without any benefit?
> OK.
> I will remove these tags in the next version and split the service core
> patches from the original series into a series by itself.
Cool - good idea to split.
Perhaps we should focus on getting bugfixes in for the existing code, before doing cleanup? It would make backports easier if churn is minimal.
Suggesting patches order (first to last)
1. bugfixes/things to backport
2. cleanups
3. C11 atomic optimizations
> Thanks,
> Phil
Thanks, and I'll get to reading/reviewing your and Honnappa's feedback later today.
-H
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v3 07/12] service: remove rte prefix from static functions
2020-04-08 10:36 0% ` Van Haaren, Harry
@ 2020-04-08 10:49 0% ` Phil Yang
0 siblings, 0 replies; 200+ results
From: Phil Yang @ 2020-04-08 10:49 UTC (permalink / raw)
To: Van Haaren, Harry, thomas, Ananyev, Konstantin, stephen,
maxime.coquelin, dev
Cc: david.marchand, jerinj, hemant.agrawal, Honnappa Nagarahalli,
Gavin Hu, Ruifeng Wang, Joyce Kong, nd, stable, nd, nd
> -----Original Message-----
> From: Van Haaren, Harry <harry.van.haaren@intel.com>
> Sent: Wednesday, April 8, 2020 6:37 PM
> To: Phil Yang <Phil.Yang@arm.com>; thomas@monjalon.net; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>;
> stephen@networkplumber.org; maxime.coquelin@redhat.com;
> dev@dpdk.org
> Cc: david.marchand@redhat.com; jerinj@marvell.com;
> hemant.agrawal@nxp.com; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; Gavin Hu <Gavin.Hu@arm.com>;
> Ruifeng Wang <Ruifeng.Wang@arm.com>; Joyce Kong
> <Joyce.Kong@arm.com>; nd <nd@arm.com>; stable@dpdk.org; nd
> <nd@arm.com>
> Subject: RE: [PATCH v3 07/12] service: remove rte prefix from static functions
>
> > -----Original Message-----
> > From: Phil Yang <Phil.Yang@arm.com>
> > Sent: Wednesday, April 8, 2020 11:15 AM
> > To: Van Haaren, Harry <harry.van.haaren@intel.com>;
> thomas@monjalon.net;
> > Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> > stephen@networkplumber.org; maxime.coquelin@redhat.com;
> dev@dpdk.org
> > Cc: david.marchand@redhat.com; jerinj@marvell.com;
> hemant.agrawal@nxp.com;
> > Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Gavin Hu
> > <Gavin.Hu@arm.com>; Ruifeng Wang <Ruifeng.Wang@arm.com>; Joyce
> Kong
> > <Joyce.Kong@arm.com>; nd <nd@arm.com>; stable@dpdk.org; nd
> <nd@arm.com>
> > Subject: RE: [PATCH v3 07/12] service: remove rte prefix from static
> functions
> <snip>
> > > Is this really a "Fix"? The internal function names were not exported
> > > in the .map file, so are not part of public ABI. This is an internal
> > > naming improvement (thanks for doing cleanup), but I don't think the
> > > Fixes: tags make sense?
> > >
> > > Also I'm not sure if we want to port this patch back to stable? Changing
> > > (internal) function names seems like unnecessary churn, and hence risk
> to a
> > > stable release, without any benefit?
> > OK.
> > I will remove these tags in the next version and split the service core
> > patches from the original series into a series by itself.
>
> Cool - good idea to split.
>
> Perhaps we should focus on getting bugfixes in for the existing code, before
> doing cleanup? It would make backports easier if churn is minimal.
>
> Suggesting patches order (first to last)
> 1. bugfixes/things to backport
> 2. cleanups
> 3. C11 atomic optimizations
That is a good idea. I will follow this order.
>
>
> > Thanks,
> > Phil
>
> Thanks, and I'll get to reading/reviewing your and Honnappa's feedback later
> today.
>
> -H
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] Remove abi_versioning.sh from tree
2020-04-07 11:58 5% ` Thomas Monjalon
2020-04-07 19:52 3% ` Neil Horman
@ 2020-04-08 14:34 0% ` Ray Kinsella
2020-04-08 17:41 5% ` Thomas Monjalon
1 sibling, 1 reply; 200+ results
From: Ray Kinsella @ 2020-04-08 14:34 UTC (permalink / raw)
To: Thomas Monjalon, David Marchand, Neil Horman, Kevin Laatz; +Cc: dev
On 07/04/2020 12:58, Thomas Monjalon wrote:
> 07/04/2020 13:33, Neil Horman:
>> On Tue, Apr 07, 2020 at 09:36:17AM +0200, David Marchand wrote:
>>> On Mon, Apr 6, 2020 at 9:34 PM Neil Horman <nhorman@tuxdriver.com> wrote:
>>>>
>>>> Since we've moved away from our initial abi_versioning.sh script, in
>>>
>>> abi_versioning.sh does not exist (idem with the patch title).
>>> I suppose you meant validate-abi.sh.
>>>
>> Crud, you're right, I was convoluting terms, sorry. Shall I repost with a
>> corrected changelog?
>
> Not only the commit log, look below how you did not care about the basic
> usage of the new tool.
>
>
>>>> favor of check_abi.sh, which uses libabigail, remove the old script from
>>>
>>> check-abi.sh
>>>
>>>> the tree, and update the docs accordingly
> [...]
>>>> --- a/doc/guides/contributing/abi_versioning.rst
>>>> +++ b/doc/guides/contributing/abi_versioning.rst
>
> The maintainer of doc/guides/contributing/abi_*.rst
> is Ray Kinsella so I add him as Cc.
Thanks for that ...
>
> [...]
>>>> -The syntax of the ``validate-abi.sh`` utility is::
>>>> +The syntax of the ``check-abi.sh`` utility is::
>>>>
>>>> - ./devtools/validate-abi.sh <REV1> <REV2>
>>>> + ./devtools/check-abi.sh <REV1> <REV2>
>>>
>>> The new script is not a direct replacement.
>>> It won't take git revisions, but build directories where versions of
>>> dpdk have been compiled.
>>>
>>> devtools/test-build.sh and devtools/test-meson-builds.sh illustrate its use.
>>> https://git.dpdk.org/dpdk/tree/devtools/test-meson-builds.sh#n127
>
> David, I think Neil did not take time to understand what changed in
> ABI tooling.
> I really wonder who is the real maintainer of ABI tooling and policy.
> Neil, Ray, I was expecting a better involvement in this major
> policy enforcement.
I missed it on the ML, was check-maintainers.sh run?
I am just getting FYI'ed now, right?
>
> This is where we are:
> - Neil asked first for ABI compatibility
> - Neil created validate-abi.sh
> - Ray asked for a strict policy
Feedback on that separately.
> - Kevin worked on a new tooling
> - David completed the tooling work
> - David integrated ABI checks in Travis
>
> There are many people partly involved.
> I think we need one person truly involved in ABI questions,
> someone who feels responsible and will take care of details
> like the documentation update requested above.
>
> Please don't rely on David and myself, we are already very busy
> with making sure every patches are properly reviewed.
> We need good help on the ABI topic in general.
Always happy to help.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] Remove abi_versioning.sh from tree
2020-04-06 19:34 38% [dpdk-dev] [PATCH] Remove abi_versioning.sh from tree Neil Horman
2020-04-07 7:36 4% ` David Marchand
@ 2020-04-08 14:49 0% ` Ray Kinsella
2020-04-09 9:26 3% ` Bruce Richardson
1 sibling, 1 reply; 200+ results
From: Ray Kinsella @ 2020-04-08 14:49 UTC (permalink / raw)
To: Neil Horman, dev, Richardson, Bruce
On 06/04/2020 20:34, Neil Horman wrote:
> Since we've moved away from our initial abi_versioning.sh script, in
> favor of check_abi.sh, which uses libabigail, remove the old script from
> the tree, and update the docs accordingly
>
> Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
> CC: thomas@monjalon.net
> ---
> MAINTAINERS | 1 -
> devtools/validate-abi.sh | 251 ---------------------
> doc/guides/contributing/abi_versioning.rst | 18 +-
> 3 files changed, 9 insertions(+), 261 deletions(-)
> delete mode 100755 devtools/validate-abi.sh
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 4800f6884..84b633431 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -152,7 +152,6 @@ F: devtools/gen-abi.sh
> F: devtools/libabigail.abignore
> F: devtools/update-abi.sh
> F: devtools/update_version_map_abi.py
> -F: devtools/validate-abi.sh
> F: buildtools/check-experimental-syms.sh
> F: buildtools/map-list-symbol.sh
>
> diff --git a/devtools/validate-abi.sh b/devtools/validate-abi.sh
> deleted file mode 100755
> index f64e19d38..000000000
> --- a/devtools/validate-abi.sh
> +++ /dev/null
> @@ -1,251 +0,0 @@
> -#!/usr/bin/env bash
> -# SPDX-License-Identifier: BSD-3-Clause
> -# Copyright(c) 2015 Neil Horman. All rights reserved.
> -# Copyright(c) 2017 6WIND S.A.
> -# All rights reserved
> -
> -set -e
> -
> -abicheck=abi-compliance-checker
> -abidump=abi-dumper
> -default_dst=abi-check
> -default_target=x86_64-native-linuxapp-gcc
> -
> -# trap on error
> -err_report() {
> - echo "$0: error at line $1"
> -}
> -trap 'err_report $LINENO' ERR
> -
> -print_usage () {
> - cat <<- END_OF_HELP
> - $(basename $0) [options] <rev1> <rev2>
> -
> - This script compares the ABI of 2 git revisions of the current
> - workspace. The output is a html report and a compilation log.
> -
> - The objective is to make sure that applications built against
> - DSOs from the first revision can still run when executed using
> - the DSOs built from the second revision.
> -
> - <rev1> and <rev2> are git commit id or tags.
> -
> - Options:
> - -h show this help
> - -j <num> enable parallel compilation with <num> threads
> - -v show compilation logs on the console
> - -d <dir> change working directory (default is ${default_dst})
> - -t <target> the dpdk target to use (default is ${default_target})
> - -f overwrite existing files in destination directory
> -
> - The script returns 0 on success, or the value of last failing
> - call of ${abicheck} (incompatible abi or the tool has run with errors).
> - The errors returned by ${abidump} are ignored.
> -
> - END_OF_HELP
> -}
> -
> -# log in the file, and on stdout if verbose
> -# $1: level string
> -# $2: string to be logged
> -log() {
> - echo "$1: $2"
> - if [ "${verbose}" != "true" ]; then
> - echo "$1: $2" >&3
> - fi
> -}
> -
> -# launch a command and log it, taking care of surrounding spaces with quotes
> -cmd() {
> - local i s whitespace ret
> - s=""
> - whitespace="[[:space:]]"
> - for i in "$@"; do
> - if [[ $i =~ $whitespace ]]; then
> - i=\"$i\"
> - fi
> - if [ -z "$s" ]; then
> - s="$i"
> - else
> - s="$s $i"
> - fi
> - done
> -
> - ret=0
> - log "CMD" "$s"
> - "$@" || ret=$?
> - if [ "$ret" != "0" ]; then
> - log "CMD" "previous command returned $ret"
> - fi
> -
> - return $ret
> -}
> -
> -# redirect or copy stderr/stdout to a file
> -# the syntax is unfamiliar, but it makes the rest of the
> -# code easier to read, avoiding the use of pipes
> -set_log_file() {
> - # save original stdout and stderr in fd 3 and 4
> - exec 3>&1
> - exec 4>&2
> - # create a new fd 5 that send to a file
> - exec 5> >(cat > $1)
> - # send stdout and stderr to fd 5
> - if [ "${verbose}" = "true" ]; then
> - exec 1> >(tee /dev/fd/5 >&3)
> - exec 2> >(tee /dev/fd/5 >&4)
> - else
> - exec 1>&5
> - exec 2>&5
> - fi
> -}
> -
> -# Make sure we configure SHARED libraries
> -# Also turn off IGB and KNI as those require kernel headers to build
> -fixup_config() {
> - local conf=config/defconfig_$target
> - cmd sed -i -e"$ a\CONFIG_RTE_BUILD_SHARED_LIB=y" $conf
> - cmd sed -i -e"$ a\CONFIG_RTE_NEXT_ABI=n" $conf
> - cmd sed -i -e"$ a\CONFIG_RTE_EAL_IGB_UIO=n" $conf
> - cmd sed -i -e"$ a\CONFIG_RTE_LIBRTE_KNI=n" $conf
> - cmd sed -i -e"$ a\CONFIG_RTE_KNI_KMOD=n" $conf
> -}
> -
> -# build dpdk for the given tag and dump abi
> -# $1: hash of the revision
> -gen_abi() {
> - local i
> -
> - cmd git clone ${dpdkroot} ${dst}/${1}
> - cmd cd ${dst}/${1}
> -
> - log "INFO" "Checking out version ${1} of the dpdk"
> - # Move to the old version of the tree
> - cmd git checkout ${1}
> -
> - fixup_config
> -
> - # Now configure the build
> - log "INFO" "Configuring DPDK ${1}"
> - cmd make config T=$target O=$target
> -
> - # Checking abi compliance relies on using the dwarf information in
> - # the shared objects. Build with -g to include them.
> - log "INFO" "Building DPDK ${1}. This might take a moment"
> - cmd make -j$parallel O=$target V=1 EXTRA_CFLAGS="-g -Og -Wno-error" \
> - EXTRA_LDFLAGS="-g" || log "INFO" "The build failed"
> -
> - # Move to the lib directory
> - cmd cd ${PWD}/$target/lib
> - log "INFO" "Collecting ABI information for ${1}"
> - for i in *.so; do
> - [ -e "$i" ] || break
> - cmd $abidump ${i} -o $dst/${1}/${i}.dump -lver ${1} || true
> - # hack to ignore empty SymbolsInfo section (no public ABI)
> - if grep -q "'SymbolInfo' => {}," $dst/${1}/${i}.dump \
> - 2> /dev/null; then
> - log "INFO" "${i} has no public ABI, remove dump file"
> - cmd rm -f $dst/${1}/${i}.dump
> - fi
> - done
> -}
> -
> -verbose=false
> -parallel=1
> -dst=${default_dst}
> -target=${default_target}
> -force=0
> -while getopts j:vd:t:fh ARG ; do
> - case $ARG in
> - j ) parallel=$OPTARG ;;
> - v ) verbose=true ;;
> - d ) dst=$OPTARG ;;
> - t ) target=$OPTARG ;;
> - f ) force=1 ;;
> - h ) print_usage ; exit 0 ;;
> - ? ) print_usage ; exit 1 ;;
> - esac
> -done
> -shift $(($OPTIND - 1))
> -
> -if [ $# != 2 ]; then
> - print_usage
> - exit 1
> -fi
> -
> -tag1=$1
> -tag2=$2
> -
> -# convert path to absolute
> -case "${dst}" in
> - /*) ;;
> - *) dst=${PWD}/${dst} ;;
> -esac
> -dpdkroot=$(readlink -f $(dirname $0)/..)
> -
> -if [ -e "${dst}" -a "$force" = 0 ]; then
> - echo "The ${dst} directory is not empty. Remove it, use another"
> - echo "one (-d <dir>), or force overriding (-f)"
> - exit 1
> -fi
> -
> -rm -rf ${dst}
> -mkdir -p ${dst}
> -set_log_file ${dst}/abi-check.log
> -log "INFO" "Logs available in ${dst}/abi-check.log"
> -
> -command -v ${abicheck} || log "INFO" "Can't find ${abicheck} utility"
> -command -v ${abidump} || log "INFO" "Can't find ${abidump} utility"
> -
> -hash1=$(git show -s --format=%h "$tag1" -- 2> /dev/null | tail -1)
> -hash2=$(git show -s --format=%h "$tag2" -- 2> /dev/null | tail -1)
> -
> -# Make hashes available in output for non-local reference
> -tag1="$tag1 ($hash1)"
> -tag2="$tag2 ($hash2)"
> -
> -if [ "$hash1" = "$hash2" ]; then
> - log "ERROR" "$tag1 and $tag2 are the same revisions"
> - exit 1
> -fi
> -
> -cmd mkdir -p ${dst}
> -
> -# dump abi for each revision
> -gen_abi ${hash1}
> -gen_abi ${hash2}
> -
> -# compare the abi dumps
> -cmd cd ${dst}
> -ret=0
> -list=""
> -for i in ${hash2}/*.dump; do
> - name=`basename $i`
> - libname=${name%.dump}
> -
> - if [ ! -f ${hash1}/$name ]; then
> - log "INFO" "$NAME does not exist in $tag1. skipping..."
> - continue
> - fi
> -
> - local_ret=0
> - cmd $abicheck -l $libname \
> - -old ${hash1}/$name -new ${hash2}/$name || local_ret=$?
> - if [ $local_ret != 0 ]; then
> - log "NOTICE" "$abicheck returned $local_ret"
> - ret=$local_ret
> - list="$list $libname"
> - fi
> -done
> -
> -if [ $ret != 0 ]; then
> - log "NOTICE" "ABI may be incompatible, check reports/logs for details."
> - log "NOTICE" "Incompatible list: $list"
> -else
> - log "NOTICE" "No error detected, ABI is compatible."
> -fi
> -
> -log "INFO" "Logs are in ${dst}/abi-check.log"
> -log "INFO" "HTML reports are in ${dst}/compat_reports directory"
> -
> -exit $ret
> diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst
> index a21f4e7a4..1c4a3f927 100644
> --- a/doc/guides/contributing/abi_versioning.rst
> +++ b/doc/guides/contributing/abi_versioning.rst
> @@ -482,9 +482,9 @@ Running the ABI Validator
> -------------------------
>
Could we simplify this all greatly, by telling people to use the meson/ninja build,
so they get this checking out of the box, without all the headache below?
> The ``devtools`` directory in the DPDK source tree contains a utility program,
> -``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI
> -Compliance Checker
> -<http://ispras.linuxbase.org/index.php/ABI_compliance_checker>`_.
> +``check-abi.sh``, for validating the DPDK ABI based on the libabigail abidiff
> +utility:
> +https://sourceware.org/libabigail/manual/abidiff.html
Links should be in the format.
`PEP8 (Style Guide for Python Code) <https://www.python.org/dev/peps/pep-0008/>`_.
>
> This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper``
> utilities which can be installed via a package manager. For example::
> @@ -492,9 +492,9 @@ utilities which can be installed via a package manager. For example::
> sudo yum install abi-compliance-checker
> sudo yum install abi-dumper
>
> -The syntax of the ``validate-abi.sh`` utility is::
> +The syntax of the ``check-abi.sh`` utility is::
>
> - ./devtools/validate-abi.sh <REV1> <REV2>
> + ./devtools/check-abi.sh <REV1> <REV2>
>
> Where ``REV1`` and ``REV2`` are valid gitrevisions(7)
> https://www.kernel.org/pub/software/scm/git/docs/gitrevisions.html
> @@ -503,16 +503,16 @@ on the local repo.
> For example::
>
> # Check between the previous and latest commit:
> - ./devtools/validate-abi.sh HEAD~1 HEAD
> + ./devtools/check-abi.sh HEAD~1 HEAD
>
> # Check on a specific compilation target:
> - ./devtools/validate-abi.sh -t x86_64-native-linux-gcc HEAD~1 HEAD
> + ./devtools/check-abi.sh -t x86_64-native-linux-gcc HEAD~1 HEAD
>
> # Check between two tags:
> - ./devtools/validate-abi.sh v2.0.0 v2.1.0
> + ./devtools/check-abi.sh v2.0.0 v2.1.0
>
> # Check between git master and local topic-branch "vhost-hacking":
> - ./devtools/validate-abi.sh master vhost-hacking
> + ./devtools/check-abi.sh master vhost-hacking
>
> After the validation script completes (it can take a while since it need to
> compile both tags) it will create compatibility reports in the
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] Remove abi_versioning.sh from tree
2020-04-07 7:36 4% ` David Marchand
2020-04-07 11:33 0% ` Neil Horman
@ 2020-04-08 14:50 0% ` Ray Kinsella
1 sibling, 0 replies; 200+ results
From: Ray Kinsella @ 2020-04-08 14:50 UTC (permalink / raw)
To: dev
On 07/04/2020 08:36, David Marchand wrote:
> On Mon, Apr 6, 2020 at 9:34 PM Neil Horman <nhorman@tuxdriver.com> wrote:
>>
>> Since we've moved away from our initial abi_versioning.sh script, in
>
> abi_versioning.sh does not exist (idem with the patch title).
> I suppose you meant validate-abi.sh.
>
>> favor of check_abi.sh, which uses libabigail, remove the old script from
>
> check-abi.sh
>
>> the tree, and update the docs accordingly
>>
>
> [snip]
>
>> diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst
>> index a21f4e7a4..1c4a3f927 100644
>> --- a/doc/guides/contributing/abi_versioning.rst
>> +++ b/doc/guides/contributing/abi_versioning.rst
>> @@ -482,9 +482,9 @@ Running the ABI Validator
>> -------------------------
>>
>> The ``devtools`` directory in the DPDK source tree contains a utility program,
>> -``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI
>> -Compliance Checker
>> -<http://ispras.linuxbase.org/index.php/ABI_compliance_checker>`_.
>> +``check-abi.sh``, for validating the DPDK ABI based on the libabigail abidiff
>> +utility:
>> +https://sourceware.org/libabigail/manual/abidiff.html
>>
>> This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper``
>> utilities which can be installed via a package manager. For example::
>> @@ -492,9 +492,9 @@ utilities which can be installed via a package manager. For example::
>> sudo yum install abi-compliance-checker
>> sudo yum install abi-dumper
>>
>> -The syntax of the ``validate-abi.sh`` utility is::
>> +The syntax of the ``check-abi.sh`` utility is::
>>
>> - ./devtools/validate-abi.sh <REV1> <REV2>
>> + ./devtools/check-abi.sh <REV1> <REV2>
>
> The new script is not a direct replacement.
> It won't take git revisions, but build directories where versions of
> dpdk have been compiled.
>
> devtools/test-build.sh and devtools/test-meson-builds.sh illustrate its use.
> https://git.dpdk.org/dpdk/tree/devtools/test-meson-builds.sh#n127
>
>
As described in my other - should we just direct contributors to use the meson/ninja build?
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [dpdk-stable] [PATCH v2] mbuf: replace zero-length marker with unnamed union
@ 2020-04-08 15:22 3% ` David Marchand
2020-04-09 9:48 3% ` Gavin Hu
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-04-08 15:22 UTC (permalink / raw)
To: Gavin Hu
Cc: Kevin Traynor, Bruce Richardson, Morten Brørup,
Ferruh Yigit, dev, nd, thomas, jerinj, Honnappa Nagarahalli,
Ruifeng Wang, Phil Yang, Joyce Kong, stable, Olivier MATZ,
Konstantin Ananyev, Andrew Rybchenko
On Wed, Apr 8, 2020 at 5:05 PM Gavin Hu <Gavin.Hu@arm.com> wrote:
> > -----Original Message-----
> > From: Kevin Traynor <ktraynor@redhat.com>
> > Hi Gavin, I lost track if v2 is still a candidate for merge. fwiw, it
> > compiles without giving the zero-length-bounds warning on my system.
> >
> > Kevin.
>
> Yes, this path alone is a candidate for merge.
This patch is not mergeable, it would trigger failures in the ABI checks.
You can see in patchwork that the robot reported a warning in Travis.
http://mails.dpdk.org/archives/test-report/2020-March/119919.html
https://travis-ci.com/github/ovsrobot/dpdk/jobs/295652710#L4476
I opened a bz to libabigail.
https://sourceware.org/bugzilla/show_bug.cgi?id=25661
Either a different solution is found, or your patch will have to deal
with this issue (libabigail fix won't be ready soon afaik) and waive
this.
--
David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v1] common/mlx5: remove devx depndency on ibv and dv
@ 2020-04-08 17:10 4% ` Ferruh Yigit
2020-04-09 6:21 0% ` Ray Kinsella
2020-04-09 7:24 4% ` David Marchand
0 siblings, 2 replies; 200+ results
From: Ferruh Yigit @ 2020-04-08 17:10 UTC (permalink / raw)
To: David Marchand
Cc: Raslan Darawsheh, Ophir Munk, dev, Matan Azrad, Thomas Monjalon,
Olga Shern, Asaf Penso, Kinsella, Ray, Neil Horman
On 4/1/2020 10:59 AM, Raslan Darawsheh wrote:
> Hi,
>
>> -----Original Message-----
>> From: Ophir Munk <ophirmu@mellanox.com>
>> Sent: Monday, March 30, 2020 1:32 AM
>> To: dev@dpdk.org; Matan Azrad <matan@mellanox.com>
>> Cc: Thomas Monjalon <thomas@monjalon.net>; Olga Shern
>> <olgas@mellanox.com>; Raslan Darawsheh <rasland@mellanox.com>; Ophir
>> Munk <ophirmu@mellanox.com>; Asaf Penso <asafp@mellanox.com>
>> Subject: [PATCH v1] common/mlx5: remove devx depndency on ibv and dv
>>
>> File mlx5_devx_cmds.c should contain pure DevX calls. It must be OS
>> agnostic and not include any references to ibv or dv structs (defined in
>> ibverbs and rdma-core linux libraries). This commit replaces all ibv and
>> dv references with 'void *'. Specifically, the following struct were
>> replaced:
>> 1. struct ibv_context *
>> 2. struct ibv_qp *
>> 3. struct mlx5dv_devx_cmd_comp *
>>
>> Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
>
> Patch applied to next-net-mlx,
>
Hi David,
This patch is failing in the travis for ABI checks [1], since mlx has APIs now
[2], are they public APIs or internal ones, and are they part of the ABI policy,
can you please check this?
@Thomas, please don't pull from next-net until this resolved, since this patch
is merged into next-net.
Thanks,
ferruh
[1]
[C]'function mlx5_devx_obj* mlx5_devx_cmd_create_cq(ibv_context*,
mlx5_devx_cq_attr*)' at mlx5_devx_cmds.c:1153:1 has some indirect sub-type changes:
parameter 1 of type 'ibv_context*' changed:
in pointed to type 'struct ibv_context':
entity changed from 'struct ibv_context' to 'void'
type size changed from 2624 to 0 (in bits)
<multiple similar warnings>
[2]
https://git.dpdk.org/dpdk/tree/drivers/common/mlx5/rte_common_mlx5_version.map?h=v20.02#n6
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] Remove abi_versioning.sh from tree
2020-04-08 14:34 0% ` Ray Kinsella
@ 2020-04-08 17:41 5% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-04-08 17:41 UTC (permalink / raw)
To: Neil Horman, Kevin Laatz, Ray Kinsella; +Cc: David Marchand, dev
08/04/2020 16:34, Ray Kinsella:
> On 07/04/2020 12:58, Thomas Monjalon wrote:
> > I really wonder who is the real maintainer of ABI tooling and policy.
> > Neil, Ray, I was expecting a better involvement in this major
> > policy enforcement.
>
> I missed it on the ML, was check-maintainers.sh run?
> I am just getting FYI'ed now, right?
Sorry I was probably not clear.
I am talking about all ABI discussions we have in general
on the mailing list.
Sometimes we need help with the tools, sometimes with the doc,
more often we need helping those having ABI issues or questions.
If we don't timely reply to all concerns, ABI enforcement fails.
And let's be even clearer:
I feel too many ABI concerns are managed by David and myself.
> > This is where we are:
> > - Neil asked first for ABI compatibility
> > - Neil created validate-abi.sh
> > - Ray asked for a strict policy
>
> Feedback on that separately.
>
> > - Kevin worked on a new tooling
> > - David completed the tooling work
> > - David integrated ABI checks in Travis
> >
> > There are many people partly involved.
> > I think we need one person truly involved in ABI questions,
> > someone who feels responsible and will take care of details
> > like the documentation update requested above.
> >
> > Please don't rely on David and myself, we are already very busy
> > with making sure every patches are properly reviewed.
> > We need good help on the ABI topic in general.
>
> Always happy to help.
Please help in emails about ABI issues, including rte_internal marker:
http://inbox.dpdk.org/dev/?q=ABI
Thanks
^ permalink raw reply [relevance 5%]
* [dpdk-dev] [PATCHv2] Remove validate-abi.sh from tree
@ 2020-04-08 19:56 38% Neil Horman
2020-04-09 7:57 4% ` Ray Kinsella
0 siblings, 1 reply; 200+ results
From: Neil Horman @ 2020-04-08 19:56 UTC (permalink / raw)
To: dev; +Cc: Neil Horman, thomas, david.marchand, mdr
Since we've moved away from our initial validate-abi.sh script, in
favor of check-abi.sh, which uses libabigail, remove the old script from
the tree, and update the docs accordingly
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
CC: thomas@monjalon.net
CC: david.marchand@redhat.com
CC: mdr@ashroe.eu
---
MAINTAINERS | 1 -
devtools/validate-abi.sh | 251 ---------------------
doc/guides/contributing/abi_versioning.rst | 61 ++---
3 files changed, 23 insertions(+), 290 deletions(-)
delete mode 100755 devtools/validate-abi.sh
diff --git a/MAINTAINERS b/MAINTAINERS
index 4800f6884..84b633431 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -152,7 +152,6 @@ F: devtools/gen-abi.sh
F: devtools/libabigail.abignore
F: devtools/update-abi.sh
F: devtools/update_version_map_abi.py
-F: devtools/validate-abi.sh
F: buildtools/check-experimental-syms.sh
F: buildtools/map-list-symbol.sh
diff --git a/devtools/validate-abi.sh b/devtools/validate-abi.sh
deleted file mode 100755
index f64e19d38..000000000
--- a/devtools/validate-abi.sh
+++ /dev/null
@@ -1,251 +0,0 @@
-#!/usr/bin/env bash
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2015 Neil Horman. All rights reserved.
-# Copyright(c) 2017 6WIND S.A.
-# All rights reserved
-
-set -e
-
-abicheck=abi-compliance-checker
-abidump=abi-dumper
-default_dst=abi-check
-default_target=x86_64-native-linuxapp-gcc
-
-# trap on error
-err_report() {
- echo "$0: error at line $1"
-}
-trap 'err_report $LINENO' ERR
-
-print_usage () {
- cat <<- END_OF_HELP
- $(basename $0) [options] <rev1> <rev2>
-
- This script compares the ABI of 2 git revisions of the current
- workspace. The output is a html report and a compilation log.
-
- The objective is to make sure that applications built against
- DSOs from the first revision can still run when executed using
- the DSOs built from the second revision.
-
- <rev1> and <rev2> are git commit id or tags.
-
- Options:
- -h show this help
- -j <num> enable parallel compilation with <num> threads
- -v show compilation logs on the console
- -d <dir> change working directory (default is ${default_dst})
- -t <target> the dpdk target to use (default is ${default_target})
- -f overwrite existing files in destination directory
-
- The script returns 0 on success, or the value of last failing
- call of ${abicheck} (incompatible abi or the tool has run with errors).
- The errors returned by ${abidump} are ignored.
-
- END_OF_HELP
-}
-
-# log in the file, and on stdout if verbose
-# $1: level string
-# $2: string to be logged
-log() {
- echo "$1: $2"
- if [ "${verbose}" != "true" ]; then
- echo "$1: $2" >&3
- fi
-}
-
-# launch a command and log it, taking care of surrounding spaces with quotes
-cmd() {
- local i s whitespace ret
- s=""
- whitespace="[[:space:]]"
- for i in "$@"; do
- if [[ $i =~ $whitespace ]]; then
- i=\"$i\"
- fi
- if [ -z "$s" ]; then
- s="$i"
- else
- s="$s $i"
- fi
- done
-
- ret=0
- log "CMD" "$s"
- "$@" || ret=$?
- if [ "$ret" != "0" ]; then
- log "CMD" "previous command returned $ret"
- fi
-
- return $ret
-}
-
-# redirect or copy stderr/stdout to a file
-# the syntax is unfamiliar, but it makes the rest of the
-# code easier to read, avoiding the use of pipes
-set_log_file() {
- # save original stdout and stderr in fd 3 and 4
- exec 3>&1
- exec 4>&2
- # create a new fd 5 that send to a file
- exec 5> >(cat > $1)
- # send stdout and stderr to fd 5
- if [ "${verbose}" = "true" ]; then
- exec 1> >(tee /dev/fd/5 >&3)
- exec 2> >(tee /dev/fd/5 >&4)
- else
- exec 1>&5
- exec 2>&5
- fi
-}
-
-# Make sure we configure SHARED libraries
-# Also turn off IGB and KNI as those require kernel headers to build
-fixup_config() {
- local conf=config/defconfig_$target
- cmd sed -i -e"$ a\CONFIG_RTE_BUILD_SHARED_LIB=y" $conf
- cmd sed -i -e"$ a\CONFIG_RTE_NEXT_ABI=n" $conf
- cmd sed -i -e"$ a\CONFIG_RTE_EAL_IGB_UIO=n" $conf
- cmd sed -i -e"$ a\CONFIG_RTE_LIBRTE_KNI=n" $conf
- cmd sed -i -e"$ a\CONFIG_RTE_KNI_KMOD=n" $conf
-}
-
-# build dpdk for the given tag and dump abi
-# $1: hash of the revision
-gen_abi() {
- local i
-
- cmd git clone ${dpdkroot} ${dst}/${1}
- cmd cd ${dst}/${1}
-
- log "INFO" "Checking out version ${1} of the dpdk"
- # Move to the old version of the tree
- cmd git checkout ${1}
-
- fixup_config
-
- # Now configure the build
- log "INFO" "Configuring DPDK ${1}"
- cmd make config T=$target O=$target
-
- # Checking abi compliance relies on using the dwarf information in
- # the shared objects. Build with -g to include them.
- log "INFO" "Building DPDK ${1}. This might take a moment"
- cmd make -j$parallel O=$target V=1 EXTRA_CFLAGS="-g -Og -Wno-error" \
- EXTRA_LDFLAGS="-g" || log "INFO" "The build failed"
-
- # Move to the lib directory
- cmd cd ${PWD}/$target/lib
- log "INFO" "Collecting ABI information for ${1}"
- for i in *.so; do
- [ -e "$i" ] || break
- cmd $abidump ${i} -o $dst/${1}/${i}.dump -lver ${1} || true
- # hack to ignore empty SymbolsInfo section (no public ABI)
- if grep -q "'SymbolInfo' => {}," $dst/${1}/${i}.dump \
- 2> /dev/null; then
- log "INFO" "${i} has no public ABI, remove dump file"
- cmd rm -f $dst/${1}/${i}.dump
- fi
- done
-}
-
-verbose=false
-parallel=1
-dst=${default_dst}
-target=${default_target}
-force=0
-while getopts j:vd:t:fh ARG ; do
- case $ARG in
- j ) parallel=$OPTARG ;;
- v ) verbose=true ;;
- d ) dst=$OPTARG ;;
- t ) target=$OPTARG ;;
- f ) force=1 ;;
- h ) print_usage ; exit 0 ;;
- ? ) print_usage ; exit 1 ;;
- esac
-done
-shift $(($OPTIND - 1))
-
-if [ $# != 2 ]; then
- print_usage
- exit 1
-fi
-
-tag1=$1
-tag2=$2
-
-# convert path to absolute
-case "${dst}" in
- /*) ;;
- *) dst=${PWD}/${dst} ;;
-esac
-dpdkroot=$(readlink -f $(dirname $0)/..)
-
-if [ -e "${dst}" -a "$force" = 0 ]; then
- echo "The ${dst} directory is not empty. Remove it, use another"
- echo "one (-d <dir>), or force overriding (-f)"
- exit 1
-fi
-
-rm -rf ${dst}
-mkdir -p ${dst}
-set_log_file ${dst}/abi-check.log
-log "INFO" "Logs available in ${dst}/abi-check.log"
-
-command -v ${abicheck} || log "INFO" "Can't find ${abicheck} utility"
-command -v ${abidump} || log "INFO" "Can't find ${abidump} utility"
-
-hash1=$(git show -s --format=%h "$tag1" -- 2> /dev/null | tail -1)
-hash2=$(git show -s --format=%h "$tag2" -- 2> /dev/null | tail -1)
-
-# Make hashes available in output for non-local reference
-tag1="$tag1 ($hash1)"
-tag2="$tag2 ($hash2)"
-
-if [ "$hash1" = "$hash2" ]; then
- log "ERROR" "$tag1 and $tag2 are the same revisions"
- exit 1
-fi
-
-cmd mkdir -p ${dst}
-
-# dump abi for each revision
-gen_abi ${hash1}
-gen_abi ${hash2}
-
-# compare the abi dumps
-cmd cd ${dst}
-ret=0
-list=""
-for i in ${hash2}/*.dump; do
- name=`basename $i`
- libname=${name%.dump}
-
- if [ ! -f ${hash1}/$name ]; then
- log "INFO" "$NAME does not exist in $tag1. skipping..."
- continue
- fi
-
- local_ret=0
- cmd $abicheck -l $libname \
- -old ${hash1}/$name -new ${hash2}/$name || local_ret=$?
- if [ $local_ret != 0 ]; then
- log "NOTICE" "$abicheck returned $local_ret"
- ret=$local_ret
- list="$list $libname"
- fi
-done
-
-if [ $ret != 0 ]; then
- log "NOTICE" "ABI may be incompatible, check reports/logs for details."
- log "NOTICE" "Incompatible list: $list"
-else
- log "NOTICE" "No error detected, ABI is compatible."
-fi
-
-log "INFO" "Logs are in ${dst}/abi-check.log"
-log "INFO" "HTML reports are in ${dst}/compat_reports directory"
-
-exit $ret
diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst
index a21f4e7a4..d32cf440a 100644
--- a/doc/guides/contributing/abi_versioning.rst
+++ b/doc/guides/contributing/abi_versioning.rst
@@ -482,41 +482,26 @@ Running the ABI Validator
-------------------------
The ``devtools`` directory in the DPDK source tree contains a utility program,
-``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI
-Compliance Checker
-<http://ispras.linuxbase.org/index.php/ABI_compliance_checker>`_.
-
-This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper``
-utilities which can be installed via a package manager. For example::
-
- sudo yum install abi-compliance-checker
- sudo yum install abi-dumper
-
-The syntax of the ``validate-abi.sh`` utility is::
-
- ./devtools/validate-abi.sh <REV1> <REV2>
-
-Where ``REV1`` and ``REV2`` are valid gitrevisions(7)
-https://www.kernel.org/pub/software/scm/git/docs/gitrevisions.html
-on the local repo.
-
-For example::
-
- # Check between the previous and latest commit:
- ./devtools/validate-abi.sh HEAD~1 HEAD
-
- # Check on a specific compilation target:
- ./devtools/validate-abi.sh -t x86_64-native-linux-gcc HEAD~1 HEAD
-
- # Check between two tags:
- ./devtools/validate-abi.sh v2.0.0 v2.1.0
-
- # Check between git master and local topic-branch "vhost-hacking":
- ./devtools/validate-abi.sh master vhost-hacking
-
-After the validation script completes (it can take a while since it need to
-compile both tags) it will create compatibility reports in the
-``./abi-check/compat_report`` directory. Listed incompatibilities can be found
-as follows::
-
- grep -lr Incompatible abi-check/compat_reports/
+``check-abi.sh``, for validating the DPDK ABI based on the libabigail abidiff
+utility:
+https://sourceware.org/libabigail/manual/abidiff.html
+
+The syntax of the ``check-abi.sh`` utility is::
+
+ ./devtools/check-abi.sh <refdir> <newdir>
+
+Where <refdir> specifies the directory housing the reference build of dpdk, and
+<newdir> specifies the dpdk build directory to check the abi of
+
+Example:
+ # To compare your build branch to the ABI of the master branch
+ # after you have built your branch
+ $ cd <path to dpdk src tree>
+ $ mkdir ~/ref
+ $ git clone --local --no-hardlinks --single-branch -b master . ~/ref
+ $ cd ~/ref
+ $ cp <path to dpdk src tree from above>/.config ./.config
+ $ make defconfig
+ $ make
+ $ <path to dpdk src tree>/devtools/check-abi.sh ~/ref <path to dpdk src tree>
+
--
2.25.2
^ permalink raw reply [relevance 38%]
* Re: [dpdk-dev] [PATCH v1] common/mlx5: remove devx depndency on ibv and dv
2020-04-08 17:10 4% ` Ferruh Yigit
@ 2020-04-09 6:21 0% ` Ray Kinsella
2020-04-09 7:24 4% ` David Marchand
1 sibling, 0 replies; 200+ results
From: Ray Kinsella @ 2020-04-09 6:21 UTC (permalink / raw)
To: Ferruh Yigit, David Marchand
Cc: Raslan Darawsheh, Ophir Munk, dev, Matan Azrad, Thomas Monjalon,
Olga Shern, Asaf Penso, Kinsella, Ray, Neil Horman
On 08/04/2020 18:10, Ferruh Yigit wrote:
> On 4/1/2020 10:59 AM, Raslan Darawsheh wrote:
>> Hi,
>>
>>> -----Original Message-----
>>> From: Ophir Munk <ophirmu@mellanox.com>
>>> Sent: Monday, March 30, 2020 1:32 AM
>>> To: dev@dpdk.org; Matan Azrad <matan@mellanox.com>
>>> Cc: Thomas Monjalon <thomas@monjalon.net>; Olga Shern
>>> <olgas@mellanox.com>; Raslan Darawsheh <rasland@mellanox.com>; Ophir
>>> Munk <ophirmu@mellanox.com>; Asaf Penso <asafp@mellanox.com>
>>> Subject: [PATCH v1] common/mlx5: remove devx depndency on ibv and dv
>>>
>>> File mlx5_devx_cmds.c should contain pure DevX calls. It must be OS
>>> agnostic and not include any references to ibv or dv structs (defined in
>>> ibverbs and rdma-core linux libraries). This commit replaces all ibv and
>>> dv references with 'void *'. Specifically, the following struct were
>>> replaced:
>>> 1. struct ibv_context *
>>> 2. struct ibv_qp *
>>> 3. struct mlx5dv_devx_cmd_comp *
>>>
>>> Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
>>
>> Patch applied to next-net-mlx,
>>
>
> Hi David,
>
> This patch is failing in the travis for ABI checks [1], since mlx has APIs now
> [2], are they public APIs or internal ones, and are they part of the ABI policy,
> can you please check this?
>
> @Thomas, please don't pull from next-net until this resolved, since this patch
> is merged into next-net.
>
> Thanks,
> ferruh
>
So these were added in the 20.02 release.
http://inbox.dpdk.org/dev/1579539790-3882-16-git-send-email-matan@mellanox.com/
Assuming that they are public?
They have the appearance of APIs that perhaps should have been added as experimental in 20.02?
>
> [1]
> [C]'function mlx5_devx_obj* mlx5_devx_cmd_create_cq(ibv_context*,
> mlx5_devx_cq_attr*)' at mlx5_devx_cmds.c:1153:1 has some indirect sub-type changes:
> parameter 1 of type 'ibv_context*' changed:
> in pointed to type 'struct ibv_context':
> entity changed from 'struct ibv_context' to 'void'
> type size changed from 2624 to 0 (in bits)
>
> <multiple similar warnings>
>
> [2]
> https://git.dpdk.org/dpdk/tree/drivers/common/mlx5/rte_common_mlx5_version.map?h=v20.02#n6
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v1] common/mlx5: remove devx depndency on ibv and dv
2020-04-08 17:10 4% ` Ferruh Yigit
2020-04-09 6:21 0% ` Ray Kinsella
@ 2020-04-09 7:24 4% ` David Marchand
2020-04-16 17:35 3% ` Ferruh Yigit
1 sibling, 1 reply; 200+ results
From: David Marchand @ 2020-04-09 7:24 UTC (permalink / raw)
To: Ferruh Yigit, Matan Azrad, Raslan Darawsheh, Thomas Monjalon
Cc: Ophir Munk, dev, Olga Shern, Asaf Penso, Kinsella, Ray,
Neil Horman, Kevin Laatz
Hello,
On Wed, Apr 8, 2020 at 7:12 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> On 4/1/2020 10:59 AM, Raslan Darawsheh wrote:
> > Hi,
> >
> >> -----Original Message-----
> >> From: Ophir Munk <ophirmu@mellanox.com>
> >> Sent: Monday, March 30, 2020 1:32 AM
> >> To: dev@dpdk.org; Matan Azrad <matan@mellanox.com>
> >> Cc: Thomas Monjalon <thomas@monjalon.net>; Olga Shern
> >> <olgas@mellanox.com>; Raslan Darawsheh <rasland@mellanox.com>; Ophir
> >> Munk <ophirmu@mellanox.com>; Asaf Penso <asafp@mellanox.com>
> >> Subject: [PATCH v1] common/mlx5: remove devx depndency on ibv and dv
> >>
> >> File mlx5_devx_cmds.c should contain pure DevX calls. It must be OS
> >> agnostic and not include any references to ibv or dv structs (defined in
> >> ibverbs and rdma-core linux libraries). This commit replaces all ibv and
> >> dv references with 'void *'. Specifically, the following struct were
> >> replaced:
> >> 1. struct ibv_context *
> >> 2. struct ibv_qp *
> >> 3. struct mlx5dv_devx_cmd_comp *
> >>
> >> Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
> >
> > Patch applied to next-net-mlx,
> >
>
> Hi David,
>
> This patch is failing in the travis for ABI checks [1], since mlx has APIs now
> [2], are they public APIs or internal ones, and are they part of the ABI policy,
> can you please check this?
- What I see on patchwork and test-report ml for this patch:
http://patchwork.dpdk.org/patch/67367/
Ophir proposed a patch on 03/30.
The robot reported an issue on 03/30, and I suppose Ophir got a report.
https://mails.dpdk.org/archives/test-report/2020-March/122623.html
https://travis-ci.com/github/ovsrobot/dpdk/jobs/308057800#L2337
Matan acked the patch on 03/31.
Rasland merged the patch on 04/01.
I understand that the abi checks are not perfect, and people need help
with the new abi checks.
Prove me wrong, but here, I get the feeling that it was just ignored
by 3 people in a row.
- On the question if these should be public API or internal, that is
not for me to reply/investigate.
This is a question for Mellanox.
--
David Marchand
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCHv2] Remove validate-abi.sh from tree
2020-04-08 19:56 38% [dpdk-dev] [PATCHv2] Remove validate-abi.sh from tree Neil Horman
@ 2020-04-09 7:57 4% ` Ray Kinsella
2020-04-09 10:39 4% ` Neil Horman
0 siblings, 1 reply; 200+ results
From: Ray Kinsella @ 2020-04-09 7:57 UTC (permalink / raw)
To: Neil Horman, dev; +Cc: thomas, david.marchand
On 08/04/2020 20:56, Neil Horman wrote:
> Since we've moved away from our initial validate-abi.sh script, in
> favor of check-abi.sh, which uses libabigail, remove the old script from
> the tree, and update the docs accordingly
>
> Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
> CC: thomas@monjalon.net
> CC: david.marchand@redhat.com
> CC: mdr@ashroe.eu
> ---
> MAINTAINERS | 1 -
> devtools/validate-abi.sh | 251 ---------------------
> doc/guides/contributing/abi_versioning.rst | 61 ++---
> 3 files changed, 23 insertions(+), 290 deletions(-)
> delete mode 100755 devtools/validate-abi.sh
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 4800f6884..84b633431 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -152,7 +152,6 @@ F: devtools/gen-abi.sh
> F: devtools/libabigail.abignore
> F: devtools/update-abi.sh
> F: devtools/update_version_map_abi.py
> -F: devtools/validate-abi.sh
> F: buildtools/check-experimental-syms.sh
> F: buildtools/map-list-symbol.sh
>
> diff --git a/devtools/validate-abi.sh b/devtools/validate-abi.sh
> deleted file mode 100755
> index f64e19d38..000000000
> --- a/devtools/validate-abi.sh
> +++ /dev/null
> @@ -1,251 +0,0 @@
> -#!/usr/bin/env bash
> -# SPDX-License-Identifier: BSD-3-Clause
> -# Copyright(c) 2015 Neil Horman. All rights reserved.
> -# Copyright(c) 2017 6WIND S.A.
> -# All rights reserved
> -
> -set -e
> -
> -abicheck=abi-compliance-checker
> -abidump=abi-dumper
> -default_dst=abi-check
> -default_target=x86_64-native-linuxapp-gcc
> -
> -# trap on error
> -err_report() {
> - echo "$0: error at line $1"
> -}
> -trap 'err_report $LINENO' ERR
> -
> -print_usage () {
> - cat <<- END_OF_HELP
> - $(basename $0) [options] <rev1> <rev2>
> -
> - This script compares the ABI of 2 git revisions of the current
> - workspace. The output is a html report and a compilation log.
> -
> - The objective is to make sure that applications built against
> - DSOs from the first revision can still run when executed using
> - the DSOs built from the second revision.
> -
> - <rev1> and <rev2> are git commit id or tags.
> -
> - Options:
> - -h show this help
> - -j <num> enable parallel compilation with <num> threads
> - -v show compilation logs on the console
> - -d <dir> change working directory (default is ${default_dst})
> - -t <target> the dpdk target to use (default is ${default_target})
> - -f overwrite existing files in destination directory
> -
> - The script returns 0 on success, or the value of last failing
> - call of ${abicheck} (incompatible abi or the tool has run with errors).
> - The errors returned by ${abidump} are ignored.
> -
> - END_OF_HELP
> -}
> -
> -# log in the file, and on stdout if verbose
> -# $1: level string
> -# $2: string to be logged
> -log() {
> - echo "$1: $2"
> - if [ "${verbose}" != "true" ]; then
> - echo "$1: $2" >&3
> - fi
> -}
> -
> -# launch a command and log it, taking care of surrounding spaces with quotes
> -cmd() {
> - local i s whitespace ret
> - s=""
> - whitespace="[[:space:]]"
> - for i in "$@"; do
> - if [[ $i =~ $whitespace ]]; then
> - i=\"$i\"
> - fi
> - if [ -z "$s" ]; then
> - s="$i"
> - else
> - s="$s $i"
> - fi
> - done
> -
> - ret=0
> - log "CMD" "$s"
> - "$@" || ret=$?
> - if [ "$ret" != "0" ]; then
> - log "CMD" "previous command returned $ret"
> - fi
> -
> - return $ret
> -}
> -
> -# redirect or copy stderr/stdout to a file
> -# the syntax is unfamiliar, but it makes the rest of the
> -# code easier to read, avoiding the use of pipes
> -set_log_file() {
> - # save original stdout and stderr in fd 3 and 4
> - exec 3>&1
> - exec 4>&2
> - # create a new fd 5 that send to a file
> - exec 5> >(cat > $1)
> - # send stdout and stderr to fd 5
> - if [ "${verbose}" = "true" ]; then
> - exec 1> >(tee /dev/fd/5 >&3)
> - exec 2> >(tee /dev/fd/5 >&4)
> - else
> - exec 1>&5
> - exec 2>&5
> - fi
> -}
> -
> -# Make sure we configure SHARED libraries
> -# Also turn off IGB and KNI as those require kernel headers to build
> -fixup_config() {
> - local conf=config/defconfig_$target
> - cmd sed -i -e"$ a\CONFIG_RTE_BUILD_SHARED_LIB=y" $conf
> - cmd sed -i -e"$ a\CONFIG_RTE_NEXT_ABI=n" $conf
> - cmd sed -i -e"$ a\CONFIG_RTE_EAL_IGB_UIO=n" $conf
> - cmd sed -i -e"$ a\CONFIG_RTE_LIBRTE_KNI=n" $conf
> - cmd sed -i -e"$ a\CONFIG_RTE_KNI_KMOD=n" $conf
> -}
> -
> -# build dpdk for the given tag and dump abi
> -# $1: hash of the revision
> -gen_abi() {
> - local i
> -
> - cmd git clone ${dpdkroot} ${dst}/${1}
> - cmd cd ${dst}/${1}
> -
> - log "INFO" "Checking out version ${1} of the dpdk"
> - # Move to the old version of the tree
> - cmd git checkout ${1}
> -
> - fixup_config
> -
> - # Now configure the build
> - log "INFO" "Configuring DPDK ${1}"
> - cmd make config T=$target O=$target
> -
> - # Checking abi compliance relies on using the dwarf information in
> - # the shared objects. Build with -g to include them.
> - log "INFO" "Building DPDK ${1}. This might take a moment"
> - cmd make -j$parallel O=$target V=1 EXTRA_CFLAGS="-g -Og -Wno-error" \
> - EXTRA_LDFLAGS="-g" || log "INFO" "The build failed"
> -
> - # Move to the lib directory
> - cmd cd ${PWD}/$target/lib
> - log "INFO" "Collecting ABI information for ${1}"
> - for i in *.so; do
> - [ -e "$i" ] || break
> - cmd $abidump ${i} -o $dst/${1}/${i}.dump -lver ${1} || true
> - # hack to ignore empty SymbolsInfo section (no public ABI)
> - if grep -q "'SymbolInfo' => {}," $dst/${1}/${i}.dump \
> - 2> /dev/null; then
> - log "INFO" "${i} has no public ABI, remove dump file"
> - cmd rm -f $dst/${1}/${i}.dump
> - fi
> - done
> -}
> -
> -verbose=false
> -parallel=1
> -dst=${default_dst}
> -target=${default_target}
> -force=0
> -while getopts j:vd:t:fh ARG ; do
> - case $ARG in
> - j ) parallel=$OPTARG ;;
> - v ) verbose=true ;;
> - d ) dst=$OPTARG ;;
> - t ) target=$OPTARG ;;
> - f ) force=1 ;;
> - h ) print_usage ; exit 0 ;;
> - ? ) print_usage ; exit 1 ;;
> - esac
> -done
> -shift $(($OPTIND - 1))
> -
> -if [ $# != 2 ]; then
> - print_usage
> - exit 1
> -fi
> -
> -tag1=$1
> -tag2=$2
> -
> -# convert path to absolute
> -case "${dst}" in
> - /*) ;;
> - *) dst=${PWD}/${dst} ;;
> -esac
> -dpdkroot=$(readlink -f $(dirname $0)/..)
> -
> -if [ -e "${dst}" -a "$force" = 0 ]; then
> - echo "The ${dst} directory is not empty. Remove it, use another"
> - echo "one (-d <dir>), or force overriding (-f)"
> - exit 1
> -fi
> -
> -rm -rf ${dst}
> -mkdir -p ${dst}
> -set_log_file ${dst}/abi-check.log
> -log "INFO" "Logs available in ${dst}/abi-check.log"
> -
> -command -v ${abicheck} || log "INFO" "Can't find ${abicheck} utility"
> -command -v ${abidump} || log "INFO" "Can't find ${abidump} utility"
> -
> -hash1=$(git show -s --format=%h "$tag1" -- 2> /dev/null | tail -1)
> -hash2=$(git show -s --format=%h "$tag2" -- 2> /dev/null | tail -1)
> -
> -# Make hashes available in output for non-local reference
> -tag1="$tag1 ($hash1)"
> -tag2="$tag2 ($hash2)"
> -
> -if [ "$hash1" = "$hash2" ]; then
> - log "ERROR" "$tag1 and $tag2 are the same revisions"
> - exit 1
> -fi
> -
> -cmd mkdir -p ${dst}
> -
> -# dump abi for each revision
> -gen_abi ${hash1}
> -gen_abi ${hash2}
> -
> -# compare the abi dumps
> -cmd cd ${dst}
> -ret=0
> -list=""
> -for i in ${hash2}/*.dump; do
> - name=`basename $i`
> - libname=${name%.dump}
> -
> - if [ ! -f ${hash1}/$name ]; then
> - log "INFO" "$NAME does not exist in $tag1. skipping..."
> - continue
> - fi
> -
> - local_ret=0
> - cmd $abicheck -l $libname \
> - -old ${hash1}/$name -new ${hash2}/$name || local_ret=$?
> - if [ $local_ret != 0 ]; then
> - log "NOTICE" "$abicheck returned $local_ret"
> - ret=$local_ret
> - list="$list $libname"
> - fi
> -done
> -
> -if [ $ret != 0 ]; then
> - log "NOTICE" "ABI may be incompatible, check reports/logs for details."
> - log "NOTICE" "Incompatible list: $list"
> -else
> - log "NOTICE" "No error detected, ABI is compatible."
> -fi
> -
> -log "INFO" "Logs are in ${dst}/abi-check.log"
> -log "INFO" "HTML reports are in ${dst}/compat_reports directory"
> -
> -exit $ret
> diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst
> index a21f4e7a4..d32cf440a 100644
> --- a/doc/guides/contributing/abi_versioning.rst
> +++ b/doc/guides/contributing/abi_versioning.rst
> @@ -482,41 +482,26 @@ Running the ABI Validator
> -------------------------
>
> The ``devtools`` directory in the DPDK source tree contains a utility program,
> -``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI
> -Compliance Checker
> -<http://ispras.linuxbase.org/index.php/ABI_compliance_checker>`_.
> -
> -This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper``
> -utilities which can be installed via a package manager. For example::
> -
> - sudo yum install abi-compliance-checker
> - sudo yum install abi-dumper
> -
> -The syntax of the ``validate-abi.sh`` utility is::
> -
> - ./devtools/validate-abi.sh <REV1> <REV2>
> -
> -Where ``REV1`` and ``REV2`` are valid gitrevisions(7)
> -https://www.kernel.org/pub/software/scm/git/docs/gitrevisions.html
> -on the local repo.
> -
> -For example::
> -
> - # Check between the previous and latest commit:
> - ./devtools/validate-abi.sh HEAD~1 HEAD
> -
> - # Check on a specific compilation target:
> - ./devtools/validate-abi.sh -t x86_64-native-linux-gcc HEAD~1 HEAD
> -
> - # Check between two tags:
> - ./devtools/validate-abi.sh v2.0.0 v2.1.0
> -
> - # Check between git master and local topic-branch "vhost-hacking":
> - ./devtools/validate-abi.sh master vhost-hacking
> -
> -After the validation script completes (it can take a while since it need to
> -compile both tags) it will create compatibility reports in the
> -``./abi-check/compat_report`` directory. Listed incompatibilities can be found
> -as follows::
> -
> - grep -lr Incompatible abi-check/compat_reports/
> +``check-abi.sh``, for validating the DPDK ABI based on the libabigail abidiff
> +utility:
> +https://sourceware.org/libabigail/manual/abidiff.html
(from v1 feedback)
Links should be in the format.
`PEP8 (Style Guide for Python Code) <https://www.python.org/dev/peps/pep-0008/>`_.
> +
> +The syntax of the ``check-abi.sh`` utility is::
> +
> + ./devtools/check-abi.sh <refdir> <newdir>
(from v1 feedback)
Could we simplify this all greatly, by telling people to use the meson/ninja build,
so they get this checking out of the box, without all the headache below?
> +
> +Where <refdir> specifies the directory housing the reference build of dpdk, and
> +<newdir> specifies the dpdk build directory to check the abi of
> +
> +Example:
> + # To compare your build branch to the ABI of the master branch
> + # after you have built your branch
> + $ cd <path to dpdk src tree>
> + $ mkdir ~/ref
> + $ git clone --local --no-hardlinks --single-branch -b master . ~/ref
> + $ cd ~/ref
> + $ cp <path to dpdk src tree from above>/.config ./.config
> + $ make defconfig
> + $ make
> + $ <path to dpdk src tree>/devtools/check-abi.sh ~/ref <path to dpdk src tree>
> +
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 0/4] introduce multi-function processing support
2020-04-07 18:05 2% ` Trahe, Fiona
@ 2020-04-09 9:25 0% ` Coyle, David
2020-04-09 9:37 0% ` Trahe, Fiona
0 siblings, 1 reply; 200+ results
From: Coyle, David @ 2020-04-09 9:25 UTC (permalink / raw)
To: Trahe, Fiona, Yigit, Ferruh, dev
Cc: Doherty, Declan, De Lara Guarch, Pablo, Ryan, Brendan,
shreyansh.jain, hemant.agrawal, Akhil Goyal, O'loingsigh,
Mairtin
Thanks for the detailed review Fiona.
Based on your feedback, we will reduce the scope of our plans for multi-function processing support in DPDK.
We will focus on implementing a rawdev-based AESNI-MB PMD for Crypto-CRC and Crypto-CRC-BIP processing and we will add QAT Crypto-CRC support in a later release.
This functionality is specific to accelerated dataplane processing for DOCSIS and PON MAC workloads.
We also note that there hasn't been much community engagement in the broader scope, so these simpler rawdev PMDs should be sufficient.
If the DPDK community is interested in expanding this concept later, then this can be explored, but it would not seem necessary for now.
We will also remove crypto-perf-tester updates to test rawdev multi-function processing as this would seem like too much code churn on that test tool.
> -----Original Message-----
> From: Trahe, Fiona <fiona.trahe@intel.com>
> Sent: Tuesday, April 7, 2020 7:06 PM
>
> Hi David, Ferruh,
>
> > -----Original Message-----
> > From: Coyle, David <david.coyle@intel.com>
> > Sent: Tuesday, April 7, 2020 12:28 PM
> > To: Yigit, Ferruh <ferruh.yigit@intel.com>; dev@dpdk.org
> >
> > Hi Ferruh, see below
> >
> > > >
> > > > While DPDK's rte_cryptodev and rte_compressdev allow many
> > > > cryptographic and compression algorithms to be chained together in
> > > > one operation, there is no way to chain these with any error
> > > > detection or checksum algorithms. And there is no way to chain
> > > > crypto and compression algorithms together. The multi-function
> > > > interface will allow these chains to be created, and also allow
> > > > any future type of
> > > operation to be easily added.
> > >
> > > I was thinking if the cryptodev can be used instead but this
> > > paragraph already seems explained it. But again can you please elaborate
> why rawdev is used?
> >
> > [DC] There are a number of reasons the rawdev approach was ultimately
> chosen:
> >
> > 1) As the paragraph above explains, our primary use-case was to chain
> > a crypto operation with error detection algorithms such as CRC or BIP
> > as this could leverage optimized multi-function implementations such
> > as in the IPSec Multi-Buffer library and have a significant impact on
> performance of network access dataplane processing such as for vCMTS
> (DOCSIS MAC).
> > However such error detection algorithms are not Crypto functions so
> > some early advice we took was that it would not be suitable to add these to
> cryptodev.
> > Also, with a view to the future, the multi-function rawdev approach
> > allows crypto operations to be chained with compression operations.
> > Again, neither cryptodev or compressdev allows this type chaining.
> >
> > 2) An earlier version of multi-function suggested adding a new library
> > called rte_accelerator, as described here
> > http://mails.dpdk.org/archives/dev/2020-February/157045.html
> > We received some comments on the dev mailing list that we should not
> > add yet another acceleration library to DPDK.
> > And we also subsequently felt that the rawdev approach is better - that
> rationale is described below.
> >
> > rte_accelerator was also built on top of crypto and compress devices which
> already existed e.g.
> > drivers/crypto/aesni_mb, drivers/crypto/qat and drivers/compress/qat .
> > We subsequently realized that this was somewhat confusing when
> > performing multi-function type operations. For example, for combined
> > Crypto-Compression operations in the future, it would use either an
> > existing crypto or compress device, but neither really made sense when
> the operations are combined.
> > What was needed was a raw device which allowed an application to
> > configure any type of device and it's queue pairs and send any type of
> operation to that device.
> >
> > For both of these reasons, we decided to go down the rawdev route,
> > with a multi-function interface which can be used by several raw device
> drivers.
> >
> > 3) rawdev is the ideal place to try out a new approach like this to accessing
> devices.
> > Adding it here allows potential consumers of this such as VNF solution
> > providers to study and try out this approach, and take advantage of
> > the multi-function operations already supported in the IPSec
> > Multi-Buffer library such as Crypto-CRC and Crypto-CRC-BIP, all without
> DPDK committing to a new library upfront.
> > We would hope that the multi-function rawdev approach will mature over
> > time (through feedback from customers, new use-cases arising etc.), at
> > which point it could be potentially be moved into the main DPDK library set.
> >
> [Fiona] agree with above, in particular item (2). Just to expand a bit more on
> this: To do a crypto+compression op one would only send one op to one
> device. That device could have been either a crypto device which also
> implemented multi-fn, by adding compression, crc, etc to its capabilities OR a
> compression device which added crypto, crc, bip capabilities.
> Both were confusing and both raised questions about whether one could still
> do "normal" ops on the device, e.g. whether a normal crypto op could be
> interleaved on same qp as a multi-fn op. And how the capabilities would
> reflect what the device could do.
> It seems better to me to have a multifn device, which does explicitly just
> multifn ops.
>
> Building this on top of rawdev is a good fit in my opinion, for the following
> reasons:
> * avoids duplication of device APIs, as rawdev configure, start, stop,
> qp_setup, etc are all there already, also a nice set of stats APIs
> * no impact or dependency added to rawdev lib
> * avoids breakages on cryptodev or compressdev APIs
> * avoids code duplication where functionality is already in a lib, e.g. re-uses
> cryptodev and compressdev headers. This adds a dependency, but I think
> that's ok as multi-function inherently depends on these functions.
> * allows easy extension to add new functionality not currently available in
> any lib (CRC and BIP)
> * allows evolution - range of useful chains which may emerge is not yet
> clear.
>
> I do have some concerns, but these are resolvable in my opinion.
> (A) as there's no rawdev capability APIs and capabilities are essentially
> opaque to the rawdev API, the application uses explicit device naming to
> create or find a device that it knows will fulfil the multifunction APIs. I can see
> how this works for rawdevs which expect to have only one PMD that will
> fulfil the service, however I'd expect multi-fn to have at least 2 driver types,
> probably more eventually. To be extensible I'd suggest a naming convention
> for a class of devices. E.g. all devices and drivers that implement multi-fn
> should create a rawdev named mfn_xxx, e.g. mfn_aesni_mb, mfn_qat. The
> "mfn_" string should be defined in the mfn hdr. This would allow creation of
> apis like rte_multi_fn_count() which could find rawdevs which implement
> mfn_ without hardcoding specific driver names.
> (B) version control of the multi-function APIs. Putting the multifn API into
> the drivers/raw/common directory gives a lot of freedom while it's
> experimental. But can it benefit from API/ABI breakage infrastructure once
> the experimental tag is removed? Is there any reason not to move the
> common files to a lib/librte_multi_fn API?
> (C) xstat name strings should be moved from aesni_mb PMD to common
> and maybe use same naming convention, so appl can query same stats from
> any device, e.g. "mfn_successful_enqueues" could be implemented by all
> PMDs. If PMDs want to add driver-specific stats they can add their own
> without the mfn_, instead create their own unique stat name.
> (D) The unit test code is not extensible - again probably as based on
> previous rawdevs where there's only 1 implementation. For mfn I'd suggest
> replacing test_rawdev_selftest_aesni_mb() with a
> test_rawdev_selftest_multi_function(), which finds and/or creates all the
> raw PMDs implementing the mfn API and runs a test on each. And move the
> test files from the drivers/raw/aesni_mb dir to app/test and make generic so
> can run against any device named mfn_xxx
> (E) the main reason to piggyback onto crypto_perf_tool is to get the
> benefit of parsing and of all the crypto setup. However this code has been
> inflated a lot, in part due to name diffs like rte_cryptodev_enqueue_burst()
> vs rte_multi_fn_enqueue_burst(). Maybe could be a lot slimmer with
> macros like ENQUEUE_BURST(dev, qp, void *op, burst_size) ? would mean a
> compile time decision to do either multifn OR cryptodev API calls, but I think
> that may work and simplify it.
> (F) ok, this is a bit pedantic, (sorry David!) but should the aesni_mb
> rawdev be renamed aesni_mb_mfn throughout (files, fns, dev and driver
> name). I mean it's implementing the mfn type of rawdev. I'm thinking ahead
> to QAT - it can implement a sym device, an asym device, a compression
> device and in future a multi-fn device. I'd propose to name it qat_multifn in
> case there'll be some other kind of rawdev device it could also implement in
> future. So the name qat_raw wouldn't be so helpful. (we made that mistake
> with qat_crypto, which should probably have been qat_sym_crypto - in my
> opinion more specific names are better)
>
> I have a few minor comment- I'll reply on specific patches.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] Remove abi_versioning.sh from tree
2020-04-08 14:49 0% ` Ray Kinsella
@ 2020-04-09 9:26 3% ` Bruce Richardson
2020-04-09 10:04 0% ` Ray Kinsella
0 siblings, 1 reply; 200+ results
From: Bruce Richardson @ 2020-04-09 9:26 UTC (permalink / raw)
To: Ray Kinsella; +Cc: Neil Horman, dev
On Wed, Apr 08, 2020 at 03:49:49PM +0100, Ray Kinsella wrote:
>
>
> On 06/04/2020 20:34, Neil Horman wrote:
> > Since we've moved away from our initial abi_versioning.sh script, in
> > favor of check_abi.sh, which uses libabigail, remove the old script from
> > the tree, and update the docs accordingly
> >
> > Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
> > CC: thomas@monjalon.net
> > ---
> > MAINTAINERS | 1 -
> > devtools/validate-abi.sh | 251 ---------------------
> > doc/guides/contributing/abi_versioning.rst | 18 +-
> > 3 files changed, 9 insertions(+), 261 deletions(-)
> > delete mode 100755 devtools/validate-abi.sh
> >
<snip>
> diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst
> > index a21f4e7a4..1c4a3f927 100644
> > --- a/doc/guides/contributing/abi_versioning.rst
> > +++ b/doc/guides/contributing/abi_versioning.rst
> > @@ -482,9 +482,9 @@ Running the ABI Validator
> > -------------------------
> >
>
> Could we simplify this all greatly, by telling people to use the meson/ninja build,
> so they get this checking out of the box, without all the headache below?
>
The abi checks are not merged into the meson build - the idea was proposed
and prototyped but never merged.
Regards,
/Bruce
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2 0/4] introduce multi-function processing support
2020-04-09 9:25 0% ` Coyle, David
@ 2020-04-09 9:37 0% ` Trahe, Fiona
2020-04-09 11:55 0% ` Coyle, David
0 siblings, 1 reply; 200+ results
From: Trahe, Fiona @ 2020-04-09 9:37 UTC (permalink / raw)
To: Coyle, David, Yigit, Ferruh, dev
Cc: Doherty, Declan, De Lara Guarch, Pablo, Ryan, Brendan,
shreyansh.jain, hemant.agrawal, Akhil Goyal, O'loingsigh,
Mairtin, Trahe, Fiona
Hi David,
Answer inline below
> -----Original Message-----
> From: Coyle, David <david.coyle@intel.com>
> Sent: Thursday, April 9, 2020 10:26 AM
> To: Trahe, Fiona <fiona.trahe@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; dev@dpdk.org
> Cc: Doherty, Declan <declan.doherty@intel.com>; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Ryan, Brendan <brendan.ryan@intel.com>;
> shreyansh.jain@nxp.com; hemant.agrawal@nxp.com; Akhil Goyal <akhil.goyal@nxp.com>;
> O'loingsigh, Mairtin <mairtin.oloingsigh@intel.com>
> Subject: RE: [dpdk-dev] [PATCH v2 0/4] introduce multi-function processing support
>
> Thanks for the detailed review Fiona.
>
> Based on your feedback, we will reduce the scope of our plans for multi-function processing support
> in DPDK.
>
> We will focus on implementing a rawdev-based AESNI-MB PMD for Crypto-CRC and Crypto-CRC-BIP
> processing and we will add QAT Crypto-CRC support in a later release.
> This functionality is specific to accelerated dataplane processing for DOCSIS and PON MAC workloads.
>
> We also note that there hasn't been much community engagement in the broader scope, so these
> simpler rawdev PMDs should be sufficient.
> If the DPDK community is interested in expanding this concept later, then this can be explored, but it
> would not seem necessary for now.
>
> We will also remove crypto-perf-tester updates to test rawdev multi-function processing as this would
> seem like too much code churn on that test tool.
[Fiona] That sounds like a good idea. In that case my comments B, D and E are not relevant as assuming a broader scope.
Comments A, C and F can still be considered, but are just suggestions, not blockers to this being
applied in 20.05, they could easily be done in a later release.
///snip///
> > I do have some concerns, but these are resolvable in my opinion.
> > (A) as there's no rawdev capability APIs and capabilities are essentially
> > opaque to the rawdev API, the application uses explicit device naming to
> > create or find a device that it knows will fulfil the multifunction APIs. I can see
> > how this works for rawdevs which expect to have only one PMD that will
> > fulfil the service, however I'd expect multi-fn to have at least 2 driver types,
> > probably more eventually. To be extensible I'd suggest a naming convention
> > for a class of devices. E.g. all devices and drivers that implement multi-fn
> > should create a rawdev named mfn_xxx, e.g. mfn_aesni_mb, mfn_qat. The
> > "mfn_" string should be defined in the mfn hdr. This would allow creation of
> > apis like rte_multi_fn_count() which could find rawdevs which implement
> > mfn_ without hardcoding specific driver names.
> > (B) version control of the multi-function APIs. Putting the multifn API into
> > the drivers/raw/common directory gives a lot of freedom while it's
> > experimental. But can it benefit from API/ABI breakage infrastructure once
> > the experimental tag is removed? Is there any reason not to move the
> > common files to a lib/librte_multi_fn API?
> > (C) xstat name strings should be moved from aesni_mb PMD to common
> > and maybe use same naming convention, so appl can query same stats from
> > any device, e.g. "mfn_successful_enqueues" could be implemented by all
> > PMDs. If PMDs want to add driver-specific stats they can add their own
> > without the mfn_, instead create their own unique stat name.
> > (D) The unit test code is not extensible - again probably as based on
> > previous rawdevs where there's only 1 implementation. For mfn I'd suggest
> > replacing test_rawdev_selftest_aesni_mb() with a
> > test_rawdev_selftest_multi_function(), which finds and/or creates all the
> > raw PMDs implementing the mfn API and runs a test on each. And move the
> > test files from the drivers/raw/aesni_mb dir to app/test and make generic so
> > can run against any device named mfn_xxx
> > (E) the main reason to piggyback onto crypto_perf_tool is to get the
> > benefit of parsing and of all the crypto setup. However this code has been
> > inflated a lot, in part due to name diffs like rte_cryptodev_enqueue_burst()
> > vs rte_multi_fn_enqueue_burst(). Maybe could be a lot slimmer with
> > macros like ENQUEUE_BURST(dev, qp, void *op, burst_size) ? would mean a
> > compile time decision to do either multifn OR cryptodev API calls, but I think
> > that may work and simplify it.
> > (F) ok, this is a bit pedantic, (sorry David!) but should the aesni_mb
> > rawdev be renamed aesni_mb_mfn throughout (files, fns, dev and driver
> > name). I mean it's implementing the mfn type of rawdev. I'm thinking ahead
> > to QAT - it can implement a sym device, an asym device, a compression
> > device and in future a multi-fn device. I'd propose to name it qat_multifn in
> > case there'll be some other kind of rawdev device it could also implement in
> > future. So the name qat_raw wouldn't be so helpful. (we made that mistake
> > with qat_crypto, which should probably have been qat_sym_crypto - in my
> > opinion more specific names are better)
> >
> > I have a few minor comment- I'll reply on specific patches.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [dpdk-stable] [PATCH v2] mbuf: replace zero-length marker with unnamed union
2020-04-08 15:22 3% ` [dpdk-dev] [dpdk-stable] " David Marchand
@ 2020-04-09 9:48 3% ` Gavin Hu
2020-04-09 10:49 3% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Gavin Hu @ 2020-04-09 9:48 UTC (permalink / raw)
To: David Marchand
Cc: Kevin Traynor, Bruce Richardson, Morten Brørup,
Ferruh Yigit, dev, nd, thomas, jerinj, Honnappa Nagarahalli,
Ruifeng Wang, Phil Yang, Joyce Kong, stable, Olivier MATZ,
Konstantin Ananyev, Andrew Rybchenko, nd
Hi David,
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Wednesday, April 8, 2020 11:22 PM
> To: Gavin Hu <Gavin.Hu@arm.com>
> Cc: Kevin Traynor <ktraynor@redhat.com>; Bruce Richardson
> <bruce.richardson@intel.com>; Morten Brørup
> <mb@smartsharesystems.com>; Ferruh Yigit <ferruh.yigit@intel.com>;
> dev@dpdk.org; nd <nd@arm.com>; thomas@monjalon.net;
> jerinj@marvell.com; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; Ruifeng Wang
> <Ruifeng.Wang@arm.com>; Phil Yang <Phil.Yang@arm.com>; Joyce Kong
> <Joyce.Kong@arm.com>; stable@dpdk.org; Olivier MATZ
> <olivier.matz@6wind.com>; Konstantin Ananyev
> <konstantin.ananyev@intel.com>; Andrew Rybchenko
> <arybchenko@solarflare.com>
> Subject: Re: [dpdk-stable] [dpdk-dev] [PATCH v2] mbuf: replace zero-length
> marker with unnamed union
>
> On Wed, Apr 8, 2020 at 5:05 PM Gavin Hu <Gavin.Hu@arm.com> wrote:
> > > -----Original Message-----
> > > From: Kevin Traynor <ktraynor@redhat.com>
> > > Hi Gavin, I lost track if v2 is still a candidate for merge. fwiw, it
> > > compiles without giving the zero-length-bounds warning on my system.
> > >
> > > Kevin.
> >
> > Yes, this path alone is a candidate for merge.
>
> This patch is not mergeable, it would trigger failures in the ABI checks.
Isn't it a false failure? If yes, is it ignorable?
>
> You can see in patchwork that the robot reported a warning in Travis.
> http://mails.dpdk.org/archives/test-report/2020-March/119919.html
> https://travis-ci.com/github/ovsrobot/dpdk/jobs/295652710#L4476
>
>
> I opened a bz to libabigail.
> https://sourceware.org/bugzilla/show_bug.cgi?id=25661
>
>
> Either a different solution is found, or your patch will have to deal
> with this issue (libabigail fix won't be ready soon afaik) and waive
> this.
Maybe we come back to 'disable the warning', before the libabigail fix ready? or alternatively ignore this ABI false failure, if it is.
I do not have ideas of what otherwise the options are.
>
> --
> David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] Remove abi_versioning.sh from tree
2020-04-09 9:26 3% ` Bruce Richardson
@ 2020-04-09 10:04 0% ` Ray Kinsella
0 siblings, 0 replies; 200+ results
From: Ray Kinsella @ 2020-04-09 10:04 UTC (permalink / raw)
To: Bruce Richardson; +Cc: Neil Horman, dev
On 09/04/2020 10:26, Bruce Richardson wrote:
> On Wed, Apr 08, 2020 at 03:49:49PM +0100, Ray Kinsella wrote:
>>
>>
>> On 06/04/2020 20:34, Neil Horman wrote:
>>> Since we've moved away from our initial abi_versioning.sh script, in
>>> favor of check_abi.sh, which uses libabigail, remove the old script from
>>> the tree, and update the docs accordingly
>>>
>>> Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
>>> CC: thomas@monjalon.net
>>> ---
>>> MAINTAINERS | 1 -
>>> devtools/validate-abi.sh | 251 ---------------------
>>> doc/guides/contributing/abi_versioning.rst | 18 +-
>>> 3 files changed, 9 insertions(+), 261 deletions(-)
>>> delete mode 100755 devtools/validate-abi.sh
>>>
> <snip>
>> diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst
>>> index a21f4e7a4..1c4a3f927 100644
>>> --- a/doc/guides/contributing/abi_versioning.rst
>>> +++ b/doc/guides/contributing/abi_versioning.rst
>>> @@ -482,9 +482,9 @@ Running the ABI Validator
>>> -------------------------
>>>
>>
>> Could we simplify this all greatly, by telling people to use the meson/ninja build,
>> so they get this checking out of the box, without all the headache below?
>>
> The abi checks are not merged into the meson build - the idea was proposed
> and prototyped but never merged.
ah my mistake ... that is a shame
we need these kind of self check tooling.
>
> Regards,
> /Bruce
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCHv2] Remove validate-abi.sh from tree
2020-04-09 7:57 4% ` Ray Kinsella
@ 2020-04-09 10:39 4% ` Neil Horman
2020-04-09 10:43 4% ` Bruce Richardson
0 siblings, 1 reply; 200+ results
From: Neil Horman @ 2020-04-09 10:39 UTC (permalink / raw)
To: Ray Kinsella; +Cc: dev, thomas, david.marchand
On Thu, Apr 09, 2020 at 08:57:34AM +0100, Ray Kinsella wrote:
>
>
> On 08/04/2020 20:56, Neil Horman wrote:
> > Since we've moved away from our initial validate-abi.sh script, in
> > favor of check-abi.sh, which uses libabigail, remove the old script from
> > the tree, and update the docs accordingly
> >
> > Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
> > CC: thomas@monjalon.net
> > CC: david.marchand@redhat.com
> > CC: mdr@ashroe.eu
> > ---
> > MAINTAINERS | 1 -
> > devtools/validate-abi.sh | 251 ---------------------
> > doc/guides/contributing/abi_versioning.rst | 61 ++---
> > 3 files changed, 23 insertions(+), 290 deletions(-)
> > delete mode 100755 devtools/validate-abi.sh
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index 4800f6884..84b633431 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -152,7 +152,6 @@ F: devtools/gen-abi.sh
> > F: devtools/libabigail.abignore
> > F: devtools/update-abi.sh
> > F: devtools/update_version_map_abi.py
> > -F: devtools/validate-abi.sh
> > F: buildtools/check-experimental-syms.sh
> > F: buildtools/map-list-symbol.sh
> >
> > diff --git a/devtools/validate-abi.sh b/devtools/validate-abi.sh
> > deleted file mode 100755
> > index f64e19d38..000000000
> > --- a/devtools/validate-abi.sh
> > +++ /dev/null
> > @@ -1,251 +0,0 @@
> > -#!/usr/bin/env bash
> > -# SPDX-License-Identifier: BSD-3-Clause
> > -# Copyright(c) 2015 Neil Horman. All rights reserved.
> > -# Copyright(c) 2017 6WIND S.A.
> > -# All rights reserved
> > -
> > -set -e
> > -
> > -abicheck=abi-compliance-checker
> > -abidump=abi-dumper
> > -default_dst=abi-check
> > -default_target=x86_64-native-linuxapp-gcc
> > -
> > -# trap on error
> > -err_report() {
> > - echo "$0: error at line $1"
> > -}
> > -trap 'err_report $LINENO' ERR
> > -
> > -print_usage () {
> > - cat <<- END_OF_HELP
> > - $(basename $0) [options] <rev1> <rev2>
> > -
> > - This script compares the ABI of 2 git revisions of the current
> > - workspace. The output is a html report and a compilation log.
> > -
> > - The objective is to make sure that applications built against
> > - DSOs from the first revision can still run when executed using
> > - the DSOs built from the second revision.
> > -
> > - <rev1> and <rev2> are git commit id or tags.
> > -
> > - Options:
> > - -h show this help
> > - -j <num> enable parallel compilation with <num> threads
> > - -v show compilation logs on the console
> > - -d <dir> change working directory (default is ${default_dst})
> > - -t <target> the dpdk target to use (default is ${default_target})
> > - -f overwrite existing files in destination directory
> > -
> > - The script returns 0 on success, or the value of last failing
> > - call of ${abicheck} (incompatible abi or the tool has run with errors).
> > - The errors returned by ${abidump} are ignored.
> > -
> > - END_OF_HELP
> > -}
> > -
> > -# log in the file, and on stdout if verbose
> > -# $1: level string
> > -# $2: string to be logged
> > -log() {
> > - echo "$1: $2"
> > - if [ "${verbose}" != "true" ]; then
> > - echo "$1: $2" >&3
> > - fi
> > -}
> > -
> > -# launch a command and log it, taking care of surrounding spaces with quotes
> > -cmd() {
> > - local i s whitespace ret
> > - s=""
> > - whitespace="[[:space:]]"
> > - for i in "$@"; do
> > - if [[ $i =~ $whitespace ]]; then
> > - i=\"$i\"
> > - fi
> > - if [ -z "$s" ]; then
> > - s="$i"
> > - else
> > - s="$s $i"
> > - fi
> > - done
> > -
> > - ret=0
> > - log "CMD" "$s"
> > - "$@" || ret=$?
> > - if [ "$ret" != "0" ]; then
> > - log "CMD" "previous command returned $ret"
> > - fi
> > -
> > - return $ret
> > -}
> > -
> > -# redirect or copy stderr/stdout to a file
> > -# the syntax is unfamiliar, but it makes the rest of the
> > -# code easier to read, avoiding the use of pipes
> > -set_log_file() {
> > - # save original stdout and stderr in fd 3 and 4
> > - exec 3>&1
> > - exec 4>&2
> > - # create a new fd 5 that send to a file
> > - exec 5> >(cat > $1)
> > - # send stdout and stderr to fd 5
> > - if [ "${verbose}" = "true" ]; then
> > - exec 1> >(tee /dev/fd/5 >&3)
> > - exec 2> >(tee /dev/fd/5 >&4)
> > - else
> > - exec 1>&5
> > - exec 2>&5
> > - fi
> > -}
> > -
> > -# Make sure we configure SHARED libraries
> > -# Also turn off IGB and KNI as those require kernel headers to build
> > -fixup_config() {
> > - local conf=config/defconfig_$target
> > - cmd sed -i -e"$ a\CONFIG_RTE_BUILD_SHARED_LIB=y" $conf
> > - cmd sed -i -e"$ a\CONFIG_RTE_NEXT_ABI=n" $conf
> > - cmd sed -i -e"$ a\CONFIG_RTE_EAL_IGB_UIO=n" $conf
> > - cmd sed -i -e"$ a\CONFIG_RTE_LIBRTE_KNI=n" $conf
> > - cmd sed -i -e"$ a\CONFIG_RTE_KNI_KMOD=n" $conf
> > -}
> > -
> > -# build dpdk for the given tag and dump abi
> > -# $1: hash of the revision
> > -gen_abi() {
> > - local i
> > -
> > - cmd git clone ${dpdkroot} ${dst}/${1}
> > - cmd cd ${dst}/${1}
> > -
> > - log "INFO" "Checking out version ${1} of the dpdk"
> > - # Move to the old version of the tree
> > - cmd git checkout ${1}
> > -
> > - fixup_config
> > -
> > - # Now configure the build
> > - log "INFO" "Configuring DPDK ${1}"
> > - cmd make config T=$target O=$target
> > -
> > - # Checking abi compliance relies on using the dwarf information in
> > - # the shared objects. Build with -g to include them.
> > - log "INFO" "Building DPDK ${1}. This might take a moment"
> > - cmd make -j$parallel O=$target V=1 EXTRA_CFLAGS="-g -Og -Wno-error" \
> > - EXTRA_LDFLAGS="-g" || log "INFO" "The build failed"
> > -
> > - # Move to the lib directory
> > - cmd cd ${PWD}/$target/lib
> > - log "INFO" "Collecting ABI information for ${1}"
> > - for i in *.so; do
> > - [ -e "$i" ] || break
> > - cmd $abidump ${i} -o $dst/${1}/${i}.dump -lver ${1} || true
> > - # hack to ignore empty SymbolsInfo section (no public ABI)
> > - if grep -q "'SymbolInfo' => {}," $dst/${1}/${i}.dump \
> > - 2> /dev/null; then
> > - log "INFO" "${i} has no public ABI, remove dump file"
> > - cmd rm -f $dst/${1}/${i}.dump
> > - fi
> > - done
> > -}
> > -
> > -verbose=false
> > -parallel=1
> > -dst=${default_dst}
> > -target=${default_target}
> > -force=0
> > -while getopts j:vd:t:fh ARG ; do
> > - case $ARG in
> > - j ) parallel=$OPTARG ;;
> > - v ) verbose=true ;;
> > - d ) dst=$OPTARG ;;
> > - t ) target=$OPTARG ;;
> > - f ) force=1 ;;
> > - h ) print_usage ; exit 0 ;;
> > - ? ) print_usage ; exit 1 ;;
> > - esac
> > -done
> > -shift $(($OPTIND - 1))
> > -
> > -if [ $# != 2 ]; then
> > - print_usage
> > - exit 1
> > -fi
> > -
> > -tag1=$1
> > -tag2=$2
> > -
> > -# convert path to absolute
> > -case "${dst}" in
> > - /*) ;;
> > - *) dst=${PWD}/${dst} ;;
> > -esac
> > -dpdkroot=$(readlink -f $(dirname $0)/..)
> > -
> > -if [ -e "${dst}" -a "$force" = 0 ]; then
> > - echo "The ${dst} directory is not empty. Remove it, use another"
> > - echo "one (-d <dir>), or force overriding (-f)"
> > - exit 1
> > -fi
> > -
> > -rm -rf ${dst}
> > -mkdir -p ${dst}
> > -set_log_file ${dst}/abi-check.log
> > -log "INFO" "Logs available in ${dst}/abi-check.log"
> > -
> > -command -v ${abicheck} || log "INFO" "Can't find ${abicheck} utility"
> > -command -v ${abidump} || log "INFO" "Can't find ${abidump} utility"
> > -
> > -hash1=$(git show -s --format=%h "$tag1" -- 2> /dev/null | tail -1)
> > -hash2=$(git show -s --format=%h "$tag2" -- 2> /dev/null | tail -1)
> > -
> > -# Make hashes available in output for non-local reference
> > -tag1="$tag1 ($hash1)"
> > -tag2="$tag2 ($hash2)"
> > -
> > -if [ "$hash1" = "$hash2" ]; then
> > - log "ERROR" "$tag1 and $tag2 are the same revisions"
> > - exit 1
> > -fi
> > -
> > -cmd mkdir -p ${dst}
> > -
> > -# dump abi for each revision
> > -gen_abi ${hash1}
> > -gen_abi ${hash2}
> > -
> > -# compare the abi dumps
> > -cmd cd ${dst}
> > -ret=0
> > -list=""
> > -for i in ${hash2}/*.dump; do
> > - name=`basename $i`
> > - libname=${name%.dump}
> > -
> > - if [ ! -f ${hash1}/$name ]; then
> > - log "INFO" "$NAME does not exist in $tag1. skipping..."
> > - continue
> > - fi
> > -
> > - local_ret=0
> > - cmd $abicheck -l $libname \
> > - -old ${hash1}/$name -new ${hash2}/$name || local_ret=$?
> > - if [ $local_ret != 0 ]; then
> > - log "NOTICE" "$abicheck returned $local_ret"
> > - ret=$local_ret
> > - list="$list $libname"
> > - fi
> > -done
> > -
> > -if [ $ret != 0 ]; then
> > - log "NOTICE" "ABI may be incompatible, check reports/logs for details."
> > - log "NOTICE" "Incompatible list: $list"
> > -else
> > - log "NOTICE" "No error detected, ABI is compatible."
> > -fi
> > -
> > -log "INFO" "Logs are in ${dst}/abi-check.log"
> > -log "INFO" "HTML reports are in ${dst}/compat_reports directory"
> > -
> > -exit $ret
> > diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst
> > index a21f4e7a4..d32cf440a 100644
> > --- a/doc/guides/contributing/abi_versioning.rst
> > +++ b/doc/guides/contributing/abi_versioning.rst
> > @@ -482,41 +482,26 @@ Running the ABI Validator
> > -------------------------
> >
> > The ``devtools`` directory in the DPDK source tree contains a utility program,
> > -``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI
> > -Compliance Checker
> > -<http://ispras.linuxbase.org/index.php/ABI_compliance_checker>`_.
> > -
> > -This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper``
> > -utilities which can be installed via a package manager. For example::
> > -
> > - sudo yum install abi-compliance-checker
> > - sudo yum install abi-dumper
> > -
> > -The syntax of the ``validate-abi.sh`` utility is::
> > -
> > - ./devtools/validate-abi.sh <REV1> <REV2>
> > -
> > -Where ``REV1`` and ``REV2`` are valid gitrevisions(7)
> > -https://www.kernel.org/pub/software/scm/git/docs/gitrevisions.html
> > -on the local repo.
> > -
> > -For example::
> > -
> > - # Check between the previous and latest commit:
> > - ./devtools/validate-abi.sh HEAD~1 HEAD
> > -
> > - # Check on a specific compilation target:
> > - ./devtools/validate-abi.sh -t x86_64-native-linux-gcc HEAD~1 HEAD
> > -
> > - # Check between two tags:
> > - ./devtools/validate-abi.sh v2.0.0 v2.1.0
> > -
> > - # Check between git master and local topic-branch "vhost-hacking":
> > - ./devtools/validate-abi.sh master vhost-hacking
> > -
> > -After the validation script completes (it can take a while since it need to
> > -compile both tags) it will create compatibility reports in the
> > -``./abi-check/compat_report`` directory. Listed incompatibilities can be found
> > -as follows::
> > -
> > - grep -lr Incompatible abi-check/compat_reports/
> > +``check-abi.sh``, for validating the DPDK ABI based on the libabigail abidiff
> > +utility:
> > +https://sourceware.org/libabigail/manual/abidiff.html
>
> (from v1 feedback)
> Links should be in the format.
> `PEP8 (Style Guide for Python Code) <https://www.python.org/dev/peps/pep-0008/>`_.
>
copy that, I'll fix it up
> > +
> > +The syntax of the ``check-abi.sh`` utility is::
> > +
> > + ./devtools/check-abi.sh <refdir> <newdir>
>
> (from v1 feedback)
> Could we simplify this all greatly, by telling people to use the meson/ninja build,
> so they get this checking out of the box, without all the headache below?
>
I think bruce noted that was never merged, correct?
> > +
> > +Where <refdir> specifies the directory housing the reference build of dpdk, and
> > +<newdir> specifies the dpdk build directory to check the abi of
> > +
> > +Example:
> > + # To compare your build branch to the ABI of the master branch
> > + # after you have built your branch
> > + $ cd <path to dpdk src tree>
> > + $ mkdir ~/ref
> > + $ git clone --local --no-hardlinks --single-branch -b master . ~/ref
> > + $ cd ~/ref
> > + $ cp <path to dpdk src tree from above>/.config ./.config
> > + $ make defconfig
> > + $ make
> > + $ <path to dpdk src tree>/devtools/check-abi.sh ~/ref <path to dpdk src tree>
> > +
> >
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCHv2] Remove validate-abi.sh from tree
2020-04-09 10:39 4% ` Neil Horman
@ 2020-04-09 10:43 4% ` Bruce Richardson
2020-04-09 10:45 4% ` Ray Kinsella
0 siblings, 1 reply; 200+ results
From: Bruce Richardson @ 2020-04-09 10:43 UTC (permalink / raw)
To: Neil Horman; +Cc: Ray Kinsella, dev, thomas, david.marchand
On Thu, Apr 09, 2020 at 06:39:54AM -0400, Neil Horman wrote:
> On Thu, Apr 09, 2020 at 08:57:34AM +0100, Ray Kinsella wrote:
> >
> >
> > On 08/04/2020 20:56, Neil Horman wrote:
> > > Since we've moved away from our initial validate-abi.sh script, in
> > > favor of check-abi.sh, which uses libabigail, remove the old script from
> > > the tree, and update the docs accordingly
> > >
> > > Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
> > > CC: thomas@monjalon.net
> > > CC: david.marchand@redhat.com
> > > CC: mdr@ashroe.eu
> > > ---
> > > MAINTAINERS | 1 -
> > > devtools/validate-abi.sh | 251 ---------------------
> > > doc/guides/contributing/abi_versioning.rst | 61 ++---
> > > 3 files changed, 23 insertions(+), 290 deletions(-)
> > > delete mode 100755 devtools/validate-abi.sh
> > >
> > > diff --git a/MAINTAINERS b/MAINTAINERS
> > > index 4800f6884..84b633431 100644
> > > --- a/MAINTAINERS
> > > +++ b/MAINTAINERS
> > > @@ -152,7 +152,6 @@ F: devtools/gen-abi.sh
> > > F: devtools/libabigail.abignore
> > > F: devtools/update-abi.sh
> > > F: devtools/update_version_map_abi.py
> > > -F: devtools/validate-abi.sh
> > > F: buildtools/check-experimental-syms.sh
> > > F: buildtools/map-list-symbol.sh
> > >
> > > diff --git a/devtools/validate-abi.sh b/devtools/validate-abi.sh
> > > deleted file mode 100755
> > > index f64e19d38..000000000
> > > --- a/devtools/validate-abi.sh
> > > +++ /dev/null
> > > @@ -1,251 +0,0 @@
> > > -#!/usr/bin/env bash
> > > -# SPDX-License-Identifier: BSD-3-Clause
> > > -# Copyright(c) 2015 Neil Horman. All rights reserved.
> > > -# Copyright(c) 2017 6WIND S.A.
> > > -# All rights reserved
> > > -
> > > -set -e
> > > -
> > > -abicheck=abi-compliance-checker
> > > -abidump=abi-dumper
> > > -default_dst=abi-check
> > > -default_target=x86_64-native-linuxapp-gcc
> > > -
> > > -# trap on error
> > > -err_report() {
> > > - echo "$0: error at line $1"
> > > -}
> > > -trap 'err_report $LINENO' ERR
> > > -
> > > -print_usage () {
> > > - cat <<- END_OF_HELP
> > > - $(basename $0) [options] <rev1> <rev2>
> > > -
> > > - This script compares the ABI of 2 git revisions of the current
> > > - workspace. The output is a html report and a compilation log.
> > > -
> > > - The objective is to make sure that applications built against
> > > - DSOs from the first revision can still run when executed using
> > > - the DSOs built from the second revision.
> > > -
> > > - <rev1> and <rev2> are git commit id or tags.
> > > -
> > > - Options:
> > > - -h show this help
> > > - -j <num> enable parallel compilation with <num> threads
> > > - -v show compilation logs on the console
> > > - -d <dir> change working directory (default is ${default_dst})
> > > - -t <target> the dpdk target to use (default is ${default_target})
> > > - -f overwrite existing files in destination directory
> > > -
> > > - The script returns 0 on success, or the value of last failing
> > > - call of ${abicheck} (incompatible abi or the tool has run with errors).
> > > - The errors returned by ${abidump} are ignored.
> > > -
> > > - END_OF_HELP
> > > -}
> > > -
> > > -# log in the file, and on stdout if verbose
> > > -# $1: level string
> > > -# $2: string to be logged
> > > -log() {
> > > - echo "$1: $2"
> > > - if [ "${verbose}" != "true" ]; then
> > > - echo "$1: $2" >&3
> > > - fi
> > > -}
> > > -
> > > -# launch a command and log it, taking care of surrounding spaces with quotes
> > > -cmd() {
> > > - local i s whitespace ret
> > > - s=""
> > > - whitespace="[[:space:]]"
> > > - for i in "$@"; do
> > > - if [[ $i =~ $whitespace ]]; then
> > > - i=\"$i\"
> > > - fi
> > > - if [ -z "$s" ]; then
> > > - s="$i"
> > > - else
> > > - s="$s $i"
> > > - fi
> > > - done
> > > -
> > > - ret=0
> > > - log "CMD" "$s"
> > > - "$@" || ret=$?
> > > - if [ "$ret" != "0" ]; then
> > > - log "CMD" "previous command returned $ret"
> > > - fi
> > > -
> > > - return $ret
> > > -}
> > > -
> > > -# redirect or copy stderr/stdout to a file
> > > -# the syntax is unfamiliar, but it makes the rest of the
> > > -# code easier to read, avoiding the use of pipes
> > > -set_log_file() {
> > > - # save original stdout and stderr in fd 3 and 4
> > > - exec 3>&1
> > > - exec 4>&2
> > > - # create a new fd 5 that send to a file
> > > - exec 5> >(cat > $1)
> > > - # send stdout and stderr to fd 5
> > > - if [ "${verbose}" = "true" ]; then
> > > - exec 1> >(tee /dev/fd/5 >&3)
> > > - exec 2> >(tee /dev/fd/5 >&4)
> > > - else
> > > - exec 1>&5
> > > - exec 2>&5
> > > - fi
> > > -}
> > > -
> > > -# Make sure we configure SHARED libraries
> > > -# Also turn off IGB and KNI as those require kernel headers to build
> > > -fixup_config() {
> > > - local conf=config/defconfig_$target
> > > - cmd sed -i -e"$ a\CONFIG_RTE_BUILD_SHARED_LIB=y" $conf
> > > - cmd sed -i -e"$ a\CONFIG_RTE_NEXT_ABI=n" $conf
> > > - cmd sed -i -e"$ a\CONFIG_RTE_EAL_IGB_UIO=n" $conf
> > > - cmd sed -i -e"$ a\CONFIG_RTE_LIBRTE_KNI=n" $conf
> > > - cmd sed -i -e"$ a\CONFIG_RTE_KNI_KMOD=n" $conf
> > > -}
> > > -
> > > -# build dpdk for the given tag and dump abi
> > > -# $1: hash of the revision
> > > -gen_abi() {
> > > - local i
> > > -
> > > - cmd git clone ${dpdkroot} ${dst}/${1}
> > > - cmd cd ${dst}/${1}
> > > -
> > > - log "INFO" "Checking out version ${1} of the dpdk"
> > > - # Move to the old version of the tree
> > > - cmd git checkout ${1}
> > > -
> > > - fixup_config
> > > -
> > > - # Now configure the build
> > > - log "INFO" "Configuring DPDK ${1}"
> > > - cmd make config T=$target O=$target
> > > -
> > > - # Checking abi compliance relies on using the dwarf information in
> > > - # the shared objects. Build with -g to include them.
> > > - log "INFO" "Building DPDK ${1}. This might take a moment"
> > > - cmd make -j$parallel O=$target V=1 EXTRA_CFLAGS="-g -Og -Wno-error" \
> > > - EXTRA_LDFLAGS="-g" || log "INFO" "The build failed"
> > > -
> > > - # Move to the lib directory
> > > - cmd cd ${PWD}/$target/lib
> > > - log "INFO" "Collecting ABI information for ${1}"
> > > - for i in *.so; do
> > > - [ -e "$i" ] || break
> > > - cmd $abidump ${i} -o $dst/${1}/${i}.dump -lver ${1} || true
> > > - # hack to ignore empty SymbolsInfo section (no public ABI)
> > > - if grep -q "'SymbolInfo' => {}," $dst/${1}/${i}.dump \
> > > - 2> /dev/null; then
> > > - log "INFO" "${i} has no public ABI, remove dump file"
> > > - cmd rm -f $dst/${1}/${i}.dump
> > > - fi
> > > - done
> > > -}
> > > -
> > > -verbose=false
> > > -parallel=1
> > > -dst=${default_dst}
> > > -target=${default_target}
> > > -force=0
> > > -while getopts j:vd:t:fh ARG ; do
> > > - case $ARG in
> > > - j ) parallel=$OPTARG ;;
> > > - v ) verbose=true ;;
> > > - d ) dst=$OPTARG ;;
> > > - t ) target=$OPTARG ;;
> > > - f ) force=1 ;;
> > > - h ) print_usage ; exit 0 ;;
> > > - ? ) print_usage ; exit 1 ;;
> > > - esac
> > > -done
> > > -shift $(($OPTIND - 1))
> > > -
> > > -if [ $# != 2 ]; then
> > > - print_usage
> > > - exit 1
> > > -fi
> > > -
> > > -tag1=$1
> > > -tag2=$2
> > > -
> > > -# convert path to absolute
> > > -case "${dst}" in
> > > - /*) ;;
> > > - *) dst=${PWD}/${dst} ;;
> > > -esac
> > > -dpdkroot=$(readlink -f $(dirname $0)/..)
> > > -
> > > -if [ -e "${dst}" -a "$force" = 0 ]; then
> > > - echo "The ${dst} directory is not empty. Remove it, use another"
> > > - echo "one (-d <dir>), or force overriding (-f)"
> > > - exit 1
> > > -fi
> > > -
> > > -rm -rf ${dst}
> > > -mkdir -p ${dst}
> > > -set_log_file ${dst}/abi-check.log
> > > -log "INFO" "Logs available in ${dst}/abi-check.log"
> > > -
> > > -command -v ${abicheck} || log "INFO" "Can't find ${abicheck} utility"
> > > -command -v ${abidump} || log "INFO" "Can't find ${abidump} utility"
> > > -
> > > -hash1=$(git show -s --format=%h "$tag1" -- 2> /dev/null | tail -1)
> > > -hash2=$(git show -s --format=%h "$tag2" -- 2> /dev/null | tail -1)
> > > -
> > > -# Make hashes available in output for non-local reference
> > > -tag1="$tag1 ($hash1)"
> > > -tag2="$tag2 ($hash2)"
> > > -
> > > -if [ "$hash1" = "$hash2" ]; then
> > > - log "ERROR" "$tag1 and $tag2 are the same revisions"
> > > - exit 1
> > > -fi
> > > -
> > > -cmd mkdir -p ${dst}
> > > -
> > > -# dump abi for each revision
> > > -gen_abi ${hash1}
> > > -gen_abi ${hash2}
> > > -
> > > -# compare the abi dumps
> > > -cmd cd ${dst}
> > > -ret=0
> > > -list=""
> > > -for i in ${hash2}/*.dump; do
> > > - name=`basename $i`
> > > - libname=${name%.dump}
> > > -
> > > - if [ ! -f ${hash1}/$name ]; then
> > > - log "INFO" "$NAME does not exist in $tag1. skipping..."
> > > - continue
> > > - fi
> > > -
> > > - local_ret=0
> > > - cmd $abicheck -l $libname \
> > > - -old ${hash1}/$name -new ${hash2}/$name || local_ret=$?
> > > - if [ $local_ret != 0 ]; then
> > > - log "NOTICE" "$abicheck returned $local_ret"
> > > - ret=$local_ret
> > > - list="$list $libname"
> > > - fi
> > > -done
> > > -
> > > -if [ $ret != 0 ]; then
> > > - log "NOTICE" "ABI may be incompatible, check reports/logs for details."
> > > - log "NOTICE" "Incompatible list: $list"
> > > -else
> > > - log "NOTICE" "No error detected, ABI is compatible."
> > > -fi
> > > -
> > > -log "INFO" "Logs are in ${dst}/abi-check.log"
> > > -log "INFO" "HTML reports are in ${dst}/compat_reports directory"
> > > -
> > > -exit $ret
> > > diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst
> > > index a21f4e7a4..d32cf440a 100644
> > > --- a/doc/guides/contributing/abi_versioning.rst
> > > +++ b/doc/guides/contributing/abi_versioning.rst
> > > @@ -482,41 +482,26 @@ Running the ABI Validator
> > > -------------------------
> > >
> > > The ``devtools`` directory in the DPDK source tree contains a utility program,
> > > -``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI
> > > -Compliance Checker
> > > -<http://ispras.linuxbase.org/index.php/ABI_compliance_checker>`_.
> > > -
> > > -This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper``
> > > -utilities which can be installed via a package manager. For example::
> > > -
> > > - sudo yum install abi-compliance-checker
> > > - sudo yum install abi-dumper
> > > -
> > > -The syntax of the ``validate-abi.sh`` utility is::
> > > -
> > > - ./devtools/validate-abi.sh <REV1> <REV2>
> > > -
> > > -Where ``REV1`` and ``REV2`` are valid gitrevisions(7)
> > > -https://www.kernel.org/pub/software/scm/git/docs/gitrevisions.html
> > > -on the local repo.
> > > -
> > > -For example::
> > > -
> > > - # Check between the previous and latest commit:
> > > - ./devtools/validate-abi.sh HEAD~1 HEAD
> > > -
> > > - # Check on a specific compilation target:
> > > - ./devtools/validate-abi.sh -t x86_64-native-linux-gcc HEAD~1 HEAD
> > > -
> > > - # Check between two tags:
> > > - ./devtools/validate-abi.sh v2.0.0 v2.1.0
> > > -
> > > - # Check between git master and local topic-branch "vhost-hacking":
> > > - ./devtools/validate-abi.sh master vhost-hacking
> > > -
> > > -After the validation script completes (it can take a while since it need to
> > > -compile both tags) it will create compatibility reports in the
> > > -``./abi-check/compat_report`` directory. Listed incompatibilities can be found
> > > -as follows::
> > > -
> > > - grep -lr Incompatible abi-check/compat_reports/
> > > +``check-abi.sh``, for validating the DPDK ABI based on the libabigail abidiff
> > > +utility:
> > > +https://sourceware.org/libabigail/manual/abidiff.html
> >
> > (from v1 feedback)
> > Links should be in the format.
> > `PEP8 (Style Guide for Python Code) <https://www.python.org/dev/peps/pep-0008/>`_.
> >
> copy that, I'll fix it up
>
> > > +
> > > +The syntax of the ``check-abi.sh`` utility is::
> > > +
> > > + ./devtools/check-abi.sh <refdir> <newdir>
> >
> > (from v1 feedback)
> > Could we simplify this all greatly, by telling people to use the meson/ninja build,
> > so they get this checking out of the box, without all the headache below?
> >
> I think bruce noted that was never merged, correct?
>
Yep, correct. :-(
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCHv2] Remove validate-abi.sh from tree
2020-04-09 10:43 4% ` Bruce Richardson
@ 2020-04-09 10:45 4% ` Ray Kinsella
2020-04-09 10:59 7% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Ray Kinsella @ 2020-04-09 10:45 UTC (permalink / raw)
To: Bruce Richardson, Neil Horman; +Cc: dev, thomas, david.marchand
On 09/04/2020 11:43, Bruce Richardson wrote:
> On Thu, Apr 09, 2020 at 06:39:54AM -0400, Neil Horman wrote:
>> On Thu, Apr 09, 2020 at 08:57:34AM +0100, Ray Kinsella wrote:
>>>
>>>
>>> On 08/04/2020 20:56, Neil Horman wrote:
>>>> Since we've moved away from our initial validate-abi.sh script, in
>>>> favor of check-abi.sh, which uses libabigail, remove the old script from
>>>> the tree, and update the docs accordingly
>>>>
>>>> Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
>>>> CC: thomas@monjalon.net
>>>> CC: david.marchand@redhat.com
>>>> CC: mdr@ashroe.eu
>>>> ---
>>>> MAINTAINERS | 1 -
>>>> devtools/validate-abi.sh | 251 ---------------------
>>>> doc/guides/contributing/abi_versioning.rst | 61 ++---
>>>> 3 files changed, 23 insertions(+), 290 deletions(-)
>>>> delete mode 100755 devtools/validate-abi.sh
>>>>
>>>> diff --git a/MAINTAINERS b/MAINTAINERS
>>>> index 4800f6884..84b633431 100644
>>>> --- a/MAINTAINERS
>>>> +++ b/MAINTAINERS
>>>> @@ -152,7 +152,6 @@ F: devtools/gen-abi.sh
>>>> F: devtools/libabigail.abignore
>>>> F: devtools/update-abi.sh
>>>> F: devtools/update_version_map_abi.py
>>>> -F: devtools/validate-abi.sh
>>>> F: buildtools/check-experimental-syms.sh
>>>> F: buildtools/map-list-symbol.sh
>>>>
>>>> diff --git a/devtools/validate-abi.sh b/devtools/validate-abi.sh
>>>> deleted file mode 100755
>>>> index f64e19d38..000000000
>>>> --- a/devtools/validate-abi.sh
>>>> +++ /dev/null
>>>> @@ -1,251 +0,0 @@
>>>> -#!/usr/bin/env bash
>>>> -# SPDX-License-Identifier: BSD-3-Clause
>>>> -# Copyright(c) 2015 Neil Horman. All rights reserved.
>>>> -# Copyright(c) 2017 6WIND S.A.
>>>> -# All rights reserved
>>>> -
>>>> -set -e
>>>> -
>>>> -abicheck=abi-compliance-checker
>>>> -abidump=abi-dumper
>>>> -default_dst=abi-check
>>>> -default_target=x86_64-native-linuxapp-gcc
>>>> -
>>>> -# trap on error
>>>> -err_report() {
>>>> - echo "$0: error at line $1"
>>>> -}
>>>> -trap 'err_report $LINENO' ERR
>>>> -
>>>> -print_usage () {
>>>> - cat <<- END_OF_HELP
>>>> - $(basename $0) [options] <rev1> <rev2>
>>>> -
>>>> - This script compares the ABI of 2 git revisions of the current
>>>> - workspace. The output is a html report and a compilation log.
>>>> -
>>>> - The objective is to make sure that applications built against
>>>> - DSOs from the first revision can still run when executed using
>>>> - the DSOs built from the second revision.
>>>> -
>>>> - <rev1> and <rev2> are git commit id or tags.
>>>> -
>>>> - Options:
>>>> - -h show this help
>>>> - -j <num> enable parallel compilation with <num> threads
>>>> - -v show compilation logs on the console
>>>> - -d <dir> change working directory (default is ${default_dst})
>>>> - -t <target> the dpdk target to use (default is ${default_target})
>>>> - -f overwrite existing files in destination directory
>>>> -
>>>> - The script returns 0 on success, or the value of last failing
>>>> - call of ${abicheck} (incompatible abi or the tool has run with errors).
>>>> - The errors returned by ${abidump} are ignored.
>>>> -
>>>> - END_OF_HELP
>>>> -}
>>>> -
>>>> -# log in the file, and on stdout if verbose
>>>> -# $1: level string
>>>> -# $2: string to be logged
>>>> -log() {
>>>> - echo "$1: $2"
>>>> - if [ "${verbose}" != "true" ]; then
>>>> - echo "$1: $2" >&3
>>>> - fi
>>>> -}
>>>> -
>>>> -# launch a command and log it, taking care of surrounding spaces with quotes
>>>> -cmd() {
>>>> - local i s whitespace ret
>>>> - s=""
>>>> - whitespace="[[:space:]]"
>>>> - for i in "$@"; do
>>>> - if [[ $i =~ $whitespace ]]; then
>>>> - i=\"$i\"
>>>> - fi
>>>> - if [ -z "$s" ]; then
>>>> - s="$i"
>>>> - else
>>>> - s="$s $i"
>>>> - fi
>>>> - done
>>>> -
>>>> - ret=0
>>>> - log "CMD" "$s"
>>>> - "$@" || ret=$?
>>>> - if [ "$ret" != "0" ]; then
>>>> - log "CMD" "previous command returned $ret"
>>>> - fi
>>>> -
>>>> - return $ret
>>>> -}
>>>> -
>>>> -# redirect or copy stderr/stdout to a file
>>>> -# the syntax is unfamiliar, but it makes the rest of the
>>>> -# code easier to read, avoiding the use of pipes
>>>> -set_log_file() {
>>>> - # save original stdout and stderr in fd 3 and 4
>>>> - exec 3>&1
>>>> - exec 4>&2
>>>> - # create a new fd 5 that send to a file
>>>> - exec 5> >(cat > $1)
>>>> - # send stdout and stderr to fd 5
>>>> - if [ "${verbose}" = "true" ]; then
>>>> - exec 1> >(tee /dev/fd/5 >&3)
>>>> - exec 2> >(tee /dev/fd/5 >&4)
>>>> - else
>>>> - exec 1>&5
>>>> - exec 2>&5
>>>> - fi
>>>> -}
>>>> -
>>>> -# Make sure we configure SHARED libraries
>>>> -# Also turn off IGB and KNI as those require kernel headers to build
>>>> -fixup_config() {
>>>> - local conf=config/defconfig_$target
>>>> - cmd sed -i -e"$ a\CONFIG_RTE_BUILD_SHARED_LIB=y" $conf
>>>> - cmd sed -i -e"$ a\CONFIG_RTE_NEXT_ABI=n" $conf
>>>> - cmd sed -i -e"$ a\CONFIG_RTE_EAL_IGB_UIO=n" $conf
>>>> - cmd sed -i -e"$ a\CONFIG_RTE_LIBRTE_KNI=n" $conf
>>>> - cmd sed -i -e"$ a\CONFIG_RTE_KNI_KMOD=n" $conf
>>>> -}
>>>> -
>>>> -# build dpdk for the given tag and dump abi
>>>> -# $1: hash of the revision
>>>> -gen_abi() {
>>>> - local i
>>>> -
>>>> - cmd git clone ${dpdkroot} ${dst}/${1}
>>>> - cmd cd ${dst}/${1}
>>>> -
>>>> - log "INFO" "Checking out version ${1} of the dpdk"
>>>> - # Move to the old version of the tree
>>>> - cmd git checkout ${1}
>>>> -
>>>> - fixup_config
>>>> -
>>>> - # Now configure the build
>>>> - log "INFO" "Configuring DPDK ${1}"
>>>> - cmd make config T=$target O=$target
>>>> -
>>>> - # Checking abi compliance relies on using the dwarf information in
>>>> - # the shared objects. Build with -g to include them.
>>>> - log "INFO" "Building DPDK ${1}. This might take a moment"
>>>> - cmd make -j$parallel O=$target V=1 EXTRA_CFLAGS="-g -Og -Wno-error" \
>>>> - EXTRA_LDFLAGS="-g" || log "INFO" "The build failed"
>>>> -
>>>> - # Move to the lib directory
>>>> - cmd cd ${PWD}/$target/lib
>>>> - log "INFO" "Collecting ABI information for ${1}"
>>>> - for i in *.so; do
>>>> - [ -e "$i" ] || break
>>>> - cmd $abidump ${i} -o $dst/${1}/${i}.dump -lver ${1} || true
>>>> - # hack to ignore empty SymbolsInfo section (no public ABI)
>>>> - if grep -q "'SymbolInfo' => {}," $dst/${1}/${i}.dump \
>>>> - 2> /dev/null; then
>>>> - log "INFO" "${i} has no public ABI, remove dump file"
>>>> - cmd rm -f $dst/${1}/${i}.dump
>>>> - fi
>>>> - done
>>>> -}
>>>> -
>>>> -verbose=false
>>>> -parallel=1
>>>> -dst=${default_dst}
>>>> -target=${default_target}
>>>> -force=0
>>>> -while getopts j:vd:t:fh ARG ; do
>>>> - case $ARG in
>>>> - j ) parallel=$OPTARG ;;
>>>> - v ) verbose=true ;;
>>>> - d ) dst=$OPTARG ;;
>>>> - t ) target=$OPTARG ;;
>>>> - f ) force=1 ;;
>>>> - h ) print_usage ; exit 0 ;;
>>>> - ? ) print_usage ; exit 1 ;;
>>>> - esac
>>>> -done
>>>> -shift $(($OPTIND - 1))
>>>> -
>>>> -if [ $# != 2 ]; then
>>>> - print_usage
>>>> - exit 1
>>>> -fi
>>>> -
>>>> -tag1=$1
>>>> -tag2=$2
>>>> -
>>>> -# convert path to absolute
>>>> -case "${dst}" in
>>>> - /*) ;;
>>>> - *) dst=${PWD}/${dst} ;;
>>>> -esac
>>>> -dpdkroot=$(readlink -f $(dirname $0)/..)
>>>> -
>>>> -if [ -e "${dst}" -a "$force" = 0 ]; then
>>>> - echo "The ${dst} directory is not empty. Remove it, use another"
>>>> - echo "one (-d <dir>), or force overriding (-f)"
>>>> - exit 1
>>>> -fi
>>>> -
>>>> -rm -rf ${dst}
>>>> -mkdir -p ${dst}
>>>> -set_log_file ${dst}/abi-check.log
>>>> -log "INFO" "Logs available in ${dst}/abi-check.log"
>>>> -
>>>> -command -v ${abicheck} || log "INFO" "Can't find ${abicheck} utility"
>>>> -command -v ${abidump} || log "INFO" "Can't find ${abidump} utility"
>>>> -
>>>> -hash1=$(git show -s --format=%h "$tag1" -- 2> /dev/null | tail -1)
>>>> -hash2=$(git show -s --format=%h "$tag2" -- 2> /dev/null | tail -1)
>>>> -
>>>> -# Make hashes available in output for non-local reference
>>>> -tag1="$tag1 ($hash1)"
>>>> -tag2="$tag2 ($hash2)"
>>>> -
>>>> -if [ "$hash1" = "$hash2" ]; then
>>>> - log "ERROR" "$tag1 and $tag2 are the same revisions"
>>>> - exit 1
>>>> -fi
>>>> -
>>>> -cmd mkdir -p ${dst}
>>>> -
>>>> -# dump abi for each revision
>>>> -gen_abi ${hash1}
>>>> -gen_abi ${hash2}
>>>> -
>>>> -# compare the abi dumps
>>>> -cmd cd ${dst}
>>>> -ret=0
>>>> -list=""
>>>> -for i in ${hash2}/*.dump; do
>>>> - name=`basename $i`
>>>> - libname=${name%.dump}
>>>> -
>>>> - if [ ! -f ${hash1}/$name ]; then
>>>> - log "INFO" "$NAME does not exist in $tag1. skipping..."
>>>> - continue
>>>> - fi
>>>> -
>>>> - local_ret=0
>>>> - cmd $abicheck -l $libname \
>>>> - -old ${hash1}/$name -new ${hash2}/$name || local_ret=$?
>>>> - if [ $local_ret != 0 ]; then
>>>> - log "NOTICE" "$abicheck returned $local_ret"
>>>> - ret=$local_ret
>>>> - list="$list $libname"
>>>> - fi
>>>> -done
>>>> -
>>>> -if [ $ret != 0 ]; then
>>>> - log "NOTICE" "ABI may be incompatible, check reports/logs for details."
>>>> - log "NOTICE" "Incompatible list: $list"
>>>> -else
>>>> - log "NOTICE" "No error detected, ABI is compatible."
>>>> -fi
>>>> -
>>>> -log "INFO" "Logs are in ${dst}/abi-check.log"
>>>> -log "INFO" "HTML reports are in ${dst}/compat_reports directory"
>>>> -
>>>> -exit $ret
>>>> diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst
>>>> index a21f4e7a4..d32cf440a 100644
>>>> --- a/doc/guides/contributing/abi_versioning.rst
>>>> +++ b/doc/guides/contributing/abi_versioning.rst
>>>> @@ -482,41 +482,26 @@ Running the ABI Validator
>>>> -------------------------
>>>>
>>>> The ``devtools`` directory in the DPDK source tree contains a utility program,
>>>> -``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI
>>>> -Compliance Checker
>>>> -<http://ispras.linuxbase.org/index.php/ABI_compliance_checker>`_.
>>>> -
>>>> -This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper``
>>>> -utilities which can be installed via a package manager. For example::
>>>> -
>>>> - sudo yum install abi-compliance-checker
>>>> - sudo yum install abi-dumper
>>>> -
>>>> -The syntax of the ``validate-abi.sh`` utility is::
>>>> -
>>>> - ./devtools/validate-abi.sh <REV1> <REV2>
>>>> -
>>>> -Where ``REV1`` and ``REV2`` are valid gitrevisions(7)
>>>> -https://www.kernel.org/pub/software/scm/git/docs/gitrevisions.html
>>>> -on the local repo.
>>>> -
>>>> -For example::
>>>> -
>>>> - # Check between the previous and latest commit:
>>>> - ./devtools/validate-abi.sh HEAD~1 HEAD
>>>> -
>>>> - # Check on a specific compilation target:
>>>> - ./devtools/validate-abi.sh -t x86_64-native-linux-gcc HEAD~1 HEAD
>>>> -
>>>> - # Check between two tags:
>>>> - ./devtools/validate-abi.sh v2.0.0 v2.1.0
>>>> -
>>>> - # Check between git master and local topic-branch "vhost-hacking":
>>>> - ./devtools/validate-abi.sh master vhost-hacking
>>>> -
>>>> -After the validation script completes (it can take a while since it need to
>>>> -compile both tags) it will create compatibility reports in the
>>>> -``./abi-check/compat_report`` directory. Listed incompatibilities can be found
>>>> -as follows::
>>>> -
>>>> - grep -lr Incompatible abi-check/compat_reports/
>>>> +``check-abi.sh``, for validating the DPDK ABI based on the libabigail abidiff
>>>> +utility:
>>>> +https://sourceware.org/libabigail/manual/abidiff.html
>>>
>>> (from v1 feedback)
>>> Links should be in the format.
>>> `PEP8 (Style Guide for Python Code) <https://www.python.org/dev/peps/pep-0008/>`_.
>>>
>> copy that, I'll fix it up
>>
>>>> +
>>>> +The syntax of the ``check-abi.sh`` utility is::
>>>> +
>>>> + ./devtools/check-abi.sh <refdir> <newdir>
>>>
>>> (from v1 feedback)
>>> Could we simplify this all greatly, by telling people to use the meson/ninja build,
>>> so they get this checking out of the box, without all the headache below?
>>>
>> I think bruce noted that was never merged, correct?
>>
> Yep, correct. :-(
>
apologies, was there a reason?
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [dpdk-stable] [PATCH v2] mbuf: replace zero-length marker with unnamed union
2020-04-09 9:48 3% ` Gavin Hu
@ 2020-04-09 10:49 3% ` Thomas Monjalon
2020-04-09 16:09 0% ` Ray Kinsella
2020-04-11 2:50 0% ` Gavin Hu
0 siblings, 2 replies; 200+ results
From: Thomas Monjalon @ 2020-04-09 10:49 UTC (permalink / raw)
To: Gavin Hu
Cc: David Marchand, Kevin Traynor, Bruce Richardson,
Morten Brørup, Ferruh Yigit, dev, nd, jerinj,
Honnappa Nagarahalli, Ruifeng Wang, Phil Yang, Joyce Kong,
stable, Olivier MATZ, Konstantin Ananyev, Andrew Rybchenko, nd,
mdr
09/04/2020 11:48, Gavin Hu:
> From: David Marchand <david.marchand@redhat.com>
> > On Wed, Apr 8, 2020 at 5:05 PM Gavin Hu <Gavin.Hu@arm.com> wrote:
> > > From: Kevin Traynor <ktraynor@redhat.com>
> > > > Hi Gavin, I lost track if v2 is still a candidate for merge. fwiw, it
> > > > compiles without giving the zero-length-bounds warning on my system.
> > > >
> > > > Kevin.
> > >
> > > Yes, this path alone is a candidate for merge.
> >
> > This patch is not mergeable, it would trigger failures in the ABI checks.
>
> Isn't it a false failure? If yes, is it ignorable?
>
> > You can see in patchwork that the robot reported a warning in Travis.
> > http://mails.dpdk.org/archives/test-report/2020-March/119919.html
> > https://travis-ci.com/github/ovsrobot/dpdk/jobs/295652710#L4476
> >
> >
> > I opened a bz to libabigail.
> > https://sourceware.org/bugzilla/show_bug.cgi?id=25661
> >
> >
> > Either a different solution is found, or your patch will have to deal
> > with this issue (libabigail fix won't be ready soon afaik) and waive
> > this.
>
> Maybe we come back to 'disable the warning', before the libabigail fix ready? or alternatively ignore this ABI false failure, if it is.
> I do not have ideas of what otherwise the options are.
Gavin,
I did not check this case.
But in general, we do not skip checks, except some checkpatch ones.
The policy with ABI checks is "NEVER SKIP".
We prefer postponing patches, waiting for someone to fix tooling.
There is a lack of motivation currently for general concerns.
We need to avoid being "write-only" contributors.
So two things need to be done:
1/ improve tooling where it needs
2/ review patches from others
I published a review list recently:
http://mails.dpdk.org/archives/announce/2020-April/000315.html
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCHv2] Remove validate-abi.sh from tree
2020-04-09 10:45 4% ` Ray Kinsella
@ 2020-04-09 10:59 7% ` Thomas Monjalon
2020-04-09 13:02 4% ` Neil Horman
2020-04-09 14:52 9% ` Ray Kinsella
0 siblings, 2 replies; 200+ results
From: Thomas Monjalon @ 2020-04-09 10:59 UTC (permalink / raw)
To: Bruce Richardson, Neil Horman, Ray Kinsella; +Cc: dev, david.marchand
09/04/2020 12:45, Ray Kinsella:
> On 09/04/2020 11:43, Bruce Richardson wrote:
> > On Thu, Apr 09, 2020 at 06:39:54AM -0400, Neil Horman wrote:
> >> On Thu, Apr 09, 2020 at 08:57:34AM +0100, Ray Kinsella wrote:
> >>> On 08/04/2020 20:56, Neil Horman wrote:
> >>>> +The syntax of the ``check-abi.sh`` utility is::
> >>>> +
> >>>> + ./devtools/check-abi.sh <refdir> <newdir>
> >>>
> >>> (from v1 feedback)
> >>> Could we simplify this all greatly, by telling people to use the meson/ninja build,
> >>> so they get this checking out of the box, without all the headache below?
> >>>
> >> I think bruce noted that was never merged, correct?
> >>
> > Yep, correct. :-(
>
> apologies, was there a reason?
Because build tool job is building, not checking.
It would be wrong to make (slow) checks mandatory in all builds.
The need is to enforce checking ABI.
The result is already published by Travis in patchwork and in an
email to the author I believe.
Not checking email and patchwork is not a good excuse.
Patchwork must be a mandatory read for everybody for all checks
in general. Let's not give up on general CI workflow.
^ permalink raw reply [relevance 7%]
* Re: [dpdk-dev] [PATCH v2 0/4] introduce multi-function processing support
2020-04-09 9:37 0% ` Trahe, Fiona
@ 2020-04-09 11:55 0% ` Coyle, David
2020-04-09 13:05 0% ` Trahe, Fiona
0 siblings, 1 reply; 200+ results
From: Coyle, David @ 2020-04-09 11:55 UTC (permalink / raw)
To: Trahe, Fiona, Yigit, Ferruh, dev
Cc: Doherty, Declan, De Lara Guarch, Pablo, Ryan, Brendan,
shreyansh.jain, hemant.agrawal, Akhil Goyal, O'loingsigh,
Mairtin
Hi Fiona, see below
> -----Original Message-----
> From: Trahe, Fiona <fiona.trahe@intel.com>
> Sent: Thursday, April 9, 2020 10:37 AM
>
> Hi David,
>
> Answer inline below
>
> > -----Original Message-----
> > From: Coyle, David <david.coyle@intel.com>
> > Sent: Thursday, April 9, 2020 10:26 AM
> >
> > Thanks for the detailed review Fiona.
> >
> > Based on your feedback, we will reduce the scope of our plans for
> > multi-function processing support in DPDK.
> >
> > We will focus on implementing a rawdev-based AESNI-MB PMD for
> > Crypto-CRC and Crypto-CRC-BIP processing and we will add QAT Crypto-
> CRC support in a later release.
> > This functionality is specific to accelerated dataplane processing for DOCSIS
> and PON MAC workloads.
> >
> > We also note that there hasn't been much community engagement in the
> > broader scope, so these simpler rawdev PMDs should be sufficient.
> > If the DPDK community is interested in expanding this concept later,
> > then this can be explored, but it would not seem necessary for now.
> >
> > We will also remove crypto-perf-tester updates to test rawdev
> > multi-function processing as this would seem like too much code churn on
> that test tool.
>
> [Fiona] That sounds like a good idea. In that case my comments B, D and E are
> not relevant as assuming a broader scope.
> Comments A, C and F can still be considered, but are just suggestions, not
> blockers to this being applied in 20.05, they could easily be done in a later
> release.
[DC] For 20.05, I plan to address A, C and F from below.
We will look to address D in a later release when we add QAT multi-function PMD to see if unit test extensibility can be improved.
And B and E are now no longer applicable due to reduced scope.
>
> ///snip///
>
> > > I do have some concerns, but these are resolvable in my opinion.
> > > (A) as there's no rawdev capability APIs and capabilities are essentially
> > > opaque to the rawdev API, the application uses explicit device
> > > naming to create or find a device that it knows will fulfil the
> > > multifunction APIs. I can see how this works for rawdevs which
> > > expect to have only one PMD that will fulfil the service, however
> > > I'd expect multi-fn to have at least 2 driver types, probably more
> > > eventually. To be extensible I'd suggest a naming convention for a
> > > class of devices. E.g. all devices and drivers that implement
> > > multi-fn should create a rawdev named mfn_xxx, e.g. mfn_aesni_mb,
> > > mfn_qat. The "mfn_" string should be defined in the mfn hdr. This
> > > would allow creation of apis like rte_multi_fn_count() which could find
> rawdevs which implement mfn_ without hardcoding specific driver names.
[DC] The AESNI-MB rawdev will be renamed to rawdev_mfn_aesni_mb.
Keeping "rawdev_" as first prefix keeps this consistent with other rawdevs
Adding "mfn_" allows rawdevs implementing multi-function be found as you suggested
> > > (B) version control of the multi-function APIs. Putting the multifn API
> into
> > > the drivers/raw/common directory gives a lot of freedom while it's
> > > experimental. But can it benefit from API/ABI breakage
> > > infrastructure once the experimental tag is removed? Is there any
> > > reason not to move the common files to a lib/librte_multi_fn API?
[DC] As stated above, this is no longer applicable due to reduced scope
> > > (C) xstat name strings should be moved from aesni_mb PMD to
> common
> > > and maybe use same naming convention, so appl can query same stats
> > > from any device, e.g. "mfn_successful_enqueues" could be
> implemented
> > > by all PMDs. If PMDs want to add driver-specific stats they can add
> > > their own without the mfn_, instead create their own unique stat name.
[DC] This is a good suggestion as these same stats will also be needed by the QAT PMD.
I will make this change.
> > > (D) The unit test code is not extensible - again probably as based on
> > > previous rawdevs where there's only 1 implementation. For mfn I'd
> > > suggest replacing test_rawdev_selftest_aesni_mb() with a
> > > test_rawdev_selftest_multi_function(), which finds and/or creates
> > > all the raw PMDs implementing the mfn API and runs a test on each.
> > > And move the test files from the drivers/raw/aesni_mb dir to
> > > app/test and make generic so can run against any device named mfn_xxx
[DC] As stated above we will look at making the unit tests more extensible when we add the QAT PMD
For now, keeping the tests in the drivers/raw/aesni_mb_mfn directory is consistent with existing rawdevs
> > > (E) the main reason to piggyback onto crypto_perf_tool is to get the
> > > benefit of parsing and of all the crypto setup. However this code
> > > has been inflated a lot, in part due to name diffs like
> > > rte_cryptodev_enqueue_burst() vs rte_multi_fn_enqueue_burst().
> Maybe
> > > could be a lot slimmer with macros like ENQUEUE_BURST(dev, qp, void
> > > *op, burst_size) ? would mean a compile time decision to do either
> > > multifn OR cryptodev API calls, but I think that may work and simplify it.
[DC] We will remove support for multi-function from the crypto-perf too due to amount of code churn it required
> > > (F) ok, this is a bit pedantic, (sorry David!) but should the aesni_mb
> > > rawdev be renamed aesni_mb_mfn throughout (files, fns, dev and
> > > driver name). I mean it's implementing the mfn type of rawdev. I'm
> > > thinking ahead to QAT - it can implement a sym device, an asym
> > > device, a compression device and in future a multi-fn device. I'd
> > > propose to name it qat_multifn in case there'll be some other kind
> > > of rawdev device it could also implement in future. So the name
> > > qat_raw wouldn't be so helpful. (we made that mistake with
> > > qat_crypto, which should probably have been qat_sym_crypto - in my
> > > opinion more specific names are better)
[DC] To keep naming of files, directories, functions etc. consistent with the driver name which now includes the mfn tag,
the files, directories, functions etc. will be renamed to also include 'mfn' e.g. aesni_mb_mfn_rawdev.c/h, aesni_mb_mfn_probe()
A similar naming convention will be used for the QAT driver when we add that.
> > >
> > > I have a few minor comment- I'll reply on specific patches.
^ permalink raw reply [relevance 0%]
* [dpdk-dev] DPDK Release Status Meeting 9/04/2020
@ 2020-04-09 12:42 4% Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-04-09 12:42 UTC (permalink / raw)
To: dpdk-dev; +Cc: Thomas Monjalon
Minutes 9 April 2020
--------------------
Agenda:
* Release Dates
* Highlights
* Subtrees
Participants:
* Arm
* Intel
* Marvell
* Mellanox
* NXP
* Red Hat
Release Dates
-------------
* v20.05 dates:
* Integration/Merge/RC1: Friday 17 April 2020
* Release: Wednesday 20 May 2020
* Marvell, Arm, Mellanox & Broadcom already sent roadmap for 20.05, all
vendors please share the roadmap for the release.
Highlights
----------
* CI failures (ones reported in patchwork)
**We will be more strict and failing patches will be blocked**
Subtrees
--------
* main
* crypto tree will be pulled today
* There is lack of review, some patches stalled
* gcc10 patches
* bit operations (may be postponed to next release)
* cleanup resources on shutdown
* windows memory management
* __rte_internal for ABI checks
* high priority and not very active now, looking for volunteer
* CI failures are not taken into account
* Failures are ignored until maintainer points, this is not scalable
* **We will be more strict and failing patches will be blocked**
* If there is a false positive author should send note about it
* In long term false positives will be fixed
* graph library
* No major blocking issue, trending to be merged in this release
* There will be a new version
* Thomas will review it
* tracing library
* Some concerns on the API, it is copy/paste of log API
* Lacking more reviews
* ifproxy library
* Will be postponed to next release
* telemetry and rawdev patches are in the backlog
* next-net
* Pulled from sub-trees and some patches merged
* It is progressing, there are ~60 patches in backlog, nothing big
* New igc PMD reviewed, waiting for changes
* ABI issues with mlx PMD should be resolved before next pull
* next-crypto
* Test for security API and bbdev patchset are under review
* next-eventdev
* pull request sent, no more patches expected for -rc1
* next-virtio
* There will be reviews this week
* Tree was not active for a while but may prepare a pull request based
on merged patches
* next-net-intel
* Two series merged, reviews are going on
* next-net-mlx
* Some series merged and pulled to next-net
* next-net-mrvl
* All patches merged for -rc1
* LTS
* 18.11.7-rc1 is out, please test
* http://inbox.dpdk.org/dev/20200320193403.18959-1-ktraynor@redhat.com/
* RedHat, Intel & Mellanox sent test reports
* Release planned for next week
DPDK Release Status Meetings
============================
The DPDK Release Status Meeting is intended for DPDK Committers to discuss
the status of the master tree and sub-trees, and for project managers to
track progress or milestone dates.
The meeting occurs on Thursdays at 8:30 UTC. If you wish to attend just
send an email to "John McNamara <john.mcnamara@intel.com>" for the invite.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCHv2] Remove validate-abi.sh from tree
2020-04-09 10:59 7% ` Thomas Monjalon
@ 2020-04-09 13:02 4% ` Neil Horman
2020-04-09 13:37 4% ` Thomas Monjalon
2020-04-09 14:52 9% ` Ray Kinsella
1 sibling, 1 reply; 200+ results
From: Neil Horman @ 2020-04-09 13:02 UTC (permalink / raw)
To: Thomas Monjalon; +Cc: Bruce Richardson, Ray Kinsella, dev, david.marchand
On Thu, Apr 09, 2020 at 12:59:50PM +0200, Thomas Monjalon wrote:
> 09/04/2020 12:45, Ray Kinsella:
> > On 09/04/2020 11:43, Bruce Richardson wrote:
> > > On Thu, Apr 09, 2020 at 06:39:54AM -0400, Neil Horman wrote:
> > >> On Thu, Apr 09, 2020 at 08:57:34AM +0100, Ray Kinsella wrote:
> > >>> On 08/04/2020 20:56, Neil Horman wrote:
> > >>>> +The syntax of the ``check-abi.sh`` utility is::
> > >>>> +
> > >>>> + ./devtools/check-abi.sh <refdir> <newdir>
> > >>>
> > >>> (from v1 feedback)
> > >>> Could we simplify this all greatly, by telling people to use the meson/ninja build,
> > >>> so they get this checking out of the box, without all the headache below?
> > >>>
> > >> I think bruce noted that was never merged, correct?
> > >>
> > > Yep, correct. :-(
> >
> > apologies, was there a reason?
>
> Because build tool job is building, not checking.
> It would be wrong to make (slow) checks mandatory in all builds.
>
> The need is to enforce checking ABI.
> The result is already published by Travis in patchwork and in an
> email to the author I believe.
> Not checking email and patchwork is not a good excuse.
>
> Patchwork must be a mandatory read for everybody for all checks
> in general. Let's not give up on general CI workflow.
>
But it would be helpful to at least codify the commands in the example patch
into a build target, for the convienience of local checking prior to CI
submission, no?
Neil
>
>
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 0/4] introduce multi-function processing support
2020-04-09 11:55 0% ` Coyle, David
@ 2020-04-09 13:05 0% ` Trahe, Fiona
0 siblings, 0 replies; 200+ results
From: Trahe, Fiona @ 2020-04-09 13:05 UTC (permalink / raw)
To: Coyle, David, Yigit, Ferruh, dev
Cc: Doherty, Declan, De Lara Guarch, Pablo, Ryan, Brendan,
shreyansh.jain, hemant.agrawal, Akhil Goyal, O'loingsigh,
Mairtin
Thanks David,
Sounds good.
I look forward to the v3.
Fiona
> -----Original Message-----
> From: Coyle, David <david.coyle@intel.com>
> Sent: Thursday, April 9, 2020 12:56 PM
> To: Trahe, Fiona <fiona.trahe@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; dev@dpdk.org
> Cc: Doherty, Declan <declan.doherty@intel.com>; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Ryan, Brendan <brendan.ryan@intel.com>;
> shreyansh.jain@nxp.com; hemant.agrawal@nxp.com; Akhil Goyal <akhil.goyal@nxp.com>;
> O'loingsigh, Mairtin <mairtin.oloingsigh@intel.com>
> Subject: RE: [dpdk-dev] [PATCH v2 0/4] introduce multi-function processing support
>
> Hi Fiona, see below
>
> > -----Original Message-----
> > From: Trahe, Fiona <fiona.trahe@intel.com>
> > Sent: Thursday, April 9, 2020 10:37 AM
> >
> > Hi David,
> >
> > Answer inline below
> >
> > > -----Original Message-----
> > > From: Coyle, David <david.coyle@intel.com>
> > > Sent: Thursday, April 9, 2020 10:26 AM
> > >
> > > Thanks for the detailed review Fiona.
> > >
> > > Based on your feedback, we will reduce the scope of our plans for
> > > multi-function processing support in DPDK.
> > >
> > > We will focus on implementing a rawdev-based AESNI-MB PMD for
> > > Crypto-CRC and Crypto-CRC-BIP processing and we will add QAT Crypto-
> > CRC support in a later release.
> > > This functionality is specific to accelerated dataplane processing for DOCSIS
> > and PON MAC workloads.
> > >
> > > We also note that there hasn't been much community engagement in the
> > > broader scope, so these simpler rawdev PMDs should be sufficient.
> > > If the DPDK community is interested in expanding this concept later,
> > > then this can be explored, but it would not seem necessary for now.
> > >
> > > We will also remove crypto-perf-tester updates to test rawdev
> > > multi-function processing as this would seem like too much code churn on
> > that test tool.
> >
> > [Fiona] That sounds like a good idea. In that case my comments B, D and E are
> > not relevant as assuming a broader scope.
> > Comments A, C and F can still be considered, but are just suggestions, not
> > blockers to this being applied in 20.05, they could easily be done in a later
> > release.
>
> [DC] For 20.05, I plan to address A, C and F from below.
> We will look to address D in a later release when we add QAT multi-function PMD to see if unit test
> extensibility can be improved.
> And B and E are now no longer applicable due to reduced scope.
>
> >
> > ///snip///
> >
> > > > I do have some concerns, but these are resolvable in my opinion.
> > > > (A) as there's no rawdev capability APIs and capabilities are essentially
> > > > opaque to the rawdev API, the application uses explicit device
> > > > naming to create or find a device that it knows will fulfil the
> > > > multifunction APIs. I can see how this works for rawdevs which
> > > > expect to have only one PMD that will fulfil the service, however
> > > > I'd expect multi-fn to have at least 2 driver types, probably more
> > > > eventually. To be extensible I'd suggest a naming convention for a
> > > > class of devices. E.g. all devices and drivers that implement
> > > > multi-fn should create a rawdev named mfn_xxx, e.g. mfn_aesni_mb,
> > > > mfn_qat. The "mfn_" string should be defined in the mfn hdr. This
> > > > would allow creation of apis like rte_multi_fn_count() which could find
> > rawdevs which implement mfn_ without hardcoding specific driver names.
>
> [DC] The AESNI-MB rawdev will be renamed to rawdev_mfn_aesni_mb.
> Keeping "rawdev_" as first prefix keeps this consistent with other rawdevs
> Adding "mfn_" allows rawdevs implementing multi-function be found as you suggested
>
> > > > (B) version control of the multi-function APIs. Putting the multifn API
> > into
> > > > the drivers/raw/common directory gives a lot of freedom while it's
> > > > experimental. But can it benefit from API/ABI breakage
> > > > infrastructure once the experimental tag is removed? Is there any
> > > > reason not to move the common files to a lib/librte_multi_fn API?
>
> [DC] As stated above, this is no longer applicable due to reduced scope
>
> > > > (C) xstat name strings should be moved from aesni_mb PMD to
> > common
> > > > and maybe use same naming convention, so appl can query same stats
> > > > from any device, e.g. "mfn_successful_enqueues" could be
> > implemented
> > > > by all PMDs. If PMDs want to add driver-specific stats they can add
> > > > their own without the mfn_, instead create their own unique stat name.
>
> [DC] This is a good suggestion as these same stats will also be needed by the QAT PMD.
> I will make this change.
>
> > > > (D) The unit test code is not extensible - again probably as based on
> > > > previous rawdevs where there's only 1 implementation. For mfn I'd
> > > > suggest replacing test_rawdev_selftest_aesni_mb() with a
> > > > test_rawdev_selftest_multi_function(), which finds and/or creates
> > > > all the raw PMDs implementing the mfn API and runs a test on each.
> > > > And move the test files from the drivers/raw/aesni_mb dir to
> > > > app/test and make generic so can run against any device named mfn_xxx
>
> [DC] As stated above we will look at making the unit tests more extensible when we add the QAT PMD
> For now, keeping the tests in the drivers/raw/aesni_mb_mfn directory is consistent with existing
> rawdevs
>
> > > > (E) the main reason to piggyback onto crypto_perf_tool is to get the
> > > > benefit of parsing and of all the crypto setup. However this code
> > > > has been inflated a lot, in part due to name diffs like
> > > > rte_cryptodev_enqueue_burst() vs rte_multi_fn_enqueue_burst().
> > Maybe
> > > > could be a lot slimmer with macros like ENQUEUE_BURST(dev, qp, void
> > > > *op, burst_size) ? would mean a compile time decision to do either
> > > > multifn OR cryptodev API calls, but I think that may work and simplify it.
>
> [DC] We will remove support for multi-function from the crypto-perf too due to amount of code
> churn it required
>
> > > > (F) ok, this is a bit pedantic, (sorry David!) but should the aesni_mb
> > > > rawdev be renamed aesni_mb_mfn throughout (files, fns, dev and
> > > > driver name). I mean it's implementing the mfn type of rawdev. I'm
> > > > thinking ahead to QAT - it can implement a sym device, an asym
> > > > device, a compression device and in future a multi-fn device. I'd
> > > > propose to name it qat_multifn in case there'll be some other kind
> > > > of rawdev device it could also implement in future. So the name
> > > > qat_raw wouldn't be so helpful. (we made that mistake with
> > > > qat_crypto, which should probably have been qat_sym_crypto - in my
> > > > opinion more specific names are better)
>
> [DC] To keep naming of files, directories, functions etc. consistent with the driver name which now
> includes the mfn tag,
> the files, directories, functions etc. will be renamed to also include 'mfn' e.g.
> aesni_mb_mfn_rawdev.c/h, aesni_mb_mfn_probe()
> A similar naming convention will be used for the QAT driver when we add that.
>
> > > >
> > > > I have a few minor comment- I'll reply on specific patches.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [RFC 1/3] eventdev: allow for event devices requiring maintenance
@ 2020-04-09 13:33 3% ` Eads, Gage
2020-04-09 14:14 0% ` Mattias Rönnblom
0 siblings, 1 reply; 200+ results
From: Eads, Gage @ 2020-04-09 13:33 UTC (permalink / raw)
To: Mattias Rönnblom, Jerin Jacob
Cc: dpdk-dev, Jerin Jacob, Richardson, Bruce
> >> diff --git a/lib/librte_eventdev/rte_eventdev.h
> >> b/lib/librte_eventdev/rte_eventdev.h
> >> index 226f352ad..d69150792 100644
> >> --- a/lib/librte_eventdev/rte_eventdev.h
> >> +++ b/lib/librte_eventdev/rte_eventdev.h
> >> @@ -289,6 +289,15 @@ struct rte_event;
> >> * single queue to each port or map a single queue to many port.
> >> */
> >>
> >> +#define RTE_EVENT_DEV_CAP_REQUIRES_MAINT (1ULL << 9) /**<
> Event
> >> +device requires calls to rte_event_maintain() during
> > This scheme would call for DSW specific API handling in fastpath.
>
>
> Initially this would be so, but buffering events might yield performance
> benefits for more event devices than DSW.
>
I agree. For applications that process and enqueue one event at a time, buffering in the PMD could give a performance boost with minimal code changes (assuming the application can tolerate higher packet latency caused by buffering).
>
> In an application, it's often convenient, but sub-optimal from a
> performance point of view, to do single-event enqueue operations. The
> alternative is to use an application-level buffer, and the flush this
> buffer with rte_event_enqueue_burst(). If you allow the event device to
> buffer, you get the simplicity of single-event enqueue operations, but
> without taking any noticeable performance hit.
>
>
> >> + * periods when neither rte_event_dequeue_burst() nor
> > The typical worker thread will be
> > while (1) {
> > rte_event_dequeue_burst();
> > ..proess..
> > rte_event_enqueue_burst();
> > }
> > If so, Why DSW driver can't do the maintenance in driver context in
> > dequeue() call.
> >
>
> DSW already does maintenance on dequeue, and works well in the above
> scenario. The typical worker does not need to care about the
> rte_event_maintain() functions, since it dequeues events on a regular basis.
>
>
> What this RFC addresses is the more atypical (but still fairly common)
> case of a port being neither dequeued to or enqueued from on a regular
> basis. The timer and ethernet rx adapters are examples of such.
>
Those two adapters have application-level buffering already, so adding PMD-level buffering feels unnecessary. Could DSW support this behavior on a port-by-port basis?
If so, I'm picturing something like:
- Add a "PMD buffering" eventdev capability
- If an eventdev has that capability, its ports can be configured for PMD-level buffering (default: no buffering)
-- Convert " uint8_t disable_implicit_release" to a flags bitmap (e.g. "uint8_t event_port_cfg"), with one flag for implicit release disable and another for PMD-level buffering
-- I suspect we can maintain ABI compatibility with function versioning on rte_event_port_setup() and rte_event_port_default_conf_get(), and this flags bitmap could be extended out to 32 bits in 20.11.
- Add "flush" semantics either to a new interface or extend an existing one. I'm partial to a new interface, to avoid an additional check in e.g. the dequeue code. And putting the flush in dequeue doesn't allow an app to batch across multiple iterations of the dequeue-process-enqueue loop.
- Extend rte_event_port_attr_get() to allow users to query this new setting. Adapters that don't call the flush function could error out if the adapter's port is configured for PMD-level buffering.
(eventdev should also forbid "PMD-level buffering" and "implicit release" used together...it's easy to imagine double-release errors occurring otherwise.)
I think this accomplishes Mattias' objective, and there's no effect on existing apps or adapters unless they choose to enable this behavior.
Granted, existing apps would likely see performance loss with dsw until they enable this config option. But perhaps it's worth it to get this behavior properly supported in the interface.
Thanks,
Gage
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCHv2] Remove validate-abi.sh from tree
2020-04-09 13:02 4% ` Neil Horman
@ 2020-04-09 13:37 4% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-04-09 13:37 UTC (permalink / raw)
To: Neil Horman; +Cc: Bruce Richardson, Ray Kinsella, dev, david.marchand
09/04/2020 15:02, Neil Horman:
> On Thu, Apr 09, 2020 at 12:59:50PM +0200, Thomas Monjalon wrote:
> > 09/04/2020 12:45, Ray Kinsella:
> > > On 09/04/2020 11:43, Bruce Richardson wrote:
> > > > On Thu, Apr 09, 2020 at 06:39:54AM -0400, Neil Horman wrote:
> > > >> On Thu, Apr 09, 2020 at 08:57:34AM +0100, Ray Kinsella wrote:
> > > >>> On 08/04/2020 20:56, Neil Horman wrote:
> > > >>>> +The syntax of the ``check-abi.sh`` utility is::
> > > >>>> +
> > > >>>> + ./devtools/check-abi.sh <refdir> <newdir>
> > > >>>
> > > >>> (from v1 feedback)
> > > >>> Could we simplify this all greatly, by telling people to use the meson/ninja build,
> > > >>> so they get this checking out of the box, without all the headache below?
> > > >>>
> > > >> I think bruce noted that was never merged, correct?
> > > >>
> > > > Yep, correct. :-(
> > >
> > > apologies, was there a reason?
> >
> > Because build tool job is building, not checking.
> > It would be wrong to make (slow) checks mandatory in all builds.
> >
> > The need is to enforce checking ABI.
> > The result is already published by Travis in patchwork and in an
> > email to the author I believe.
> > Not checking email and patchwork is not a good excuse.
> >
> > Patchwork must be a mandatory read for everybody for all checks
> > in general. Let's not give up on general CI workflow.
> >
> But it would be helpful to at least codify the commands in the example patch
> into a build target, for the convienience of local checking prior to CI
> submission, no?
I don't understand what you mean.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v3 2/9] ring: prepare ring to allow new sync schemes
2020-04-08 4:59 3% ` Honnappa Nagarahalli
@ 2020-04-09 13:39 3% ` Ananyev, Konstantin
2020-04-10 20:15 0% ` Honnappa Nagarahalli
0 siblings, 1 reply; 200+ results
From: Ananyev, Konstantin @ 2020-04-09 13:39 UTC (permalink / raw)
To: Honnappa Nagarahalli, dev; +Cc: david.marchand, jielong.zjl, nd, nd
> > Subject: [PATCH v3 2/9] ring: prepare ring to allow new sync schemes
> >
> > Change from *single* to *sync_type* to allow different synchronisation
> > schemes to be applied.
> > Mark *single* as deprecated in comments.
> > Add new functions to allow user to query ring sync types.
> > Replace direct access to *single* with appopriate function call.
> >
> > Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> > ---
> > app/test/test_pdump.c | 6 +-
> > lib/librte_pdump/rte_pdump.c | 2 +-
> > lib/librte_port/rte_port_ring.c | 12 ++--
> > lib/librte_ring/rte_ring.c | 6 +-
> > lib/librte_ring/rte_ring.h | 113 ++++++++++++++++++++++++++------
> > lib/librte_ring/rte_ring_elem.h | 8 +--
> > 6 files changed, 108 insertions(+), 39 deletions(-)
> >
> > diff --git a/app/test/test_pdump.c b/app/test/test_pdump.c index
> > ad183184c..6a1180bcb 100644
> > --- a/app/test/test_pdump.c
> > +++ b/app/test/test_pdump.c
> > @@ -57,8 +57,7 @@ run_pdump_client_tests(void)
> > if (ret < 0)
> > return -1;
> > mp->flags = 0x0000;
> > - ring_client = rte_ring_create("SR0", RING_SIZE, rte_socket_id(),
> > - RING_F_SP_ENQ | RING_F_SC_DEQ);
> > + ring_client = rte_ring_create("SR0", RING_SIZE, rte_socket_id(), 0);
> Are you saying to get SP and SC behavior we now have to set the flags to 0?
No.
What the original cause does:
creates SP/SC ring:
ring_client = rte_ring_create("SR0", RING_SIZE, rte_socket_id(),
RING_F_SP_ENQ | RING_F_SC_DEQ);
Then manually makes it MP/MC by:
ring_client->prod.single = 0;
ring_client->cons.single = 0;
Instead it should just create MP/MC ring straightway, as the patch does:
ring_client = rte_ring_create("SR0", RING_SIZE, rte_socket_id(), 0);
>Isn't that a ABI break?
I don't see any.
>
> > if (ring_client == NULL) {
> > printf("rte_ring_create SR0 failed");
> > return -1;
> > @@ -71,9 +70,6 @@ run_pdump_client_tests(void)
> > }
> > rte_eth_dev_probing_finish(eth_dev);
> >
> > - ring_client->prod.single = 0;
> > - ring_client->cons.single = 0;
> Just wondering if users outside of DPDK have done the same. I hope not, otherwise, we have an API break?
I think no. While it is completely wrong practise, it would keep working
even with these changes.
>
> > -
> > printf("\n***** flags = RTE_PDUMP_FLAG_TX *****\n");
> >
> > for (itr = 0; itr < NUM_ITR; itr++) {
> > diff --git a/lib/librte_pdump/rte_pdump.c b/lib/librte_pdump/rte_pdump.c
> > index 8a01ac510..65364f2c5 100644
> > --- a/lib/librte_pdump/rte_pdump.c
> > +++ b/lib/librte_pdump/rte_pdump.c
> > @@ -380,7 +380,7 @@ pdump_validate_ring_mp(struct rte_ring *ring, struct
> > rte_mempool *mp)
> > rte_errno = EINVAL;
> > return -1;
> > }
> > - if (ring->prod.single || ring->cons.single) {
> > + if (rte_ring_prod_single(ring) || rte_ring_cons_single(ring)) {
> > PDUMP_LOG(ERR, "ring with either SP or SC settings"
> > " is not valid for pdump, should have MP and MC settings\n");
> > rte_errno = EINVAL;
> > diff --git a/lib/librte_port/rte_port_ring.c b/lib/librte_port/rte_port_ring.c
> > index 47fcdd06a..2f6c050fa 100644
> > --- a/lib/librte_port/rte_port_ring.c
> > +++ b/lib/librte_port/rte_port_ring.c
> > @@ -44,8 +44,8 @@ rte_port_ring_reader_create_internal(void *params, int
> > socket_id,
> > /* Check input parameters */
> > if ((conf == NULL) ||
> > (conf->ring == NULL) ||
> > - (conf->ring->cons.single && is_multi) ||
> > - (!(conf->ring->cons.single) && !is_multi)) {
> > + (rte_ring_cons_single(conf->ring) && is_multi) ||
> > + (!rte_ring_cons_single(conf->ring) && !is_multi)) {
> > RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
> > return NULL;
> > }
> > @@ -171,8 +171,8 @@ rte_port_ring_writer_create_internal(void *params,
> > int socket_id,
> > /* Check input parameters */
> > if ((conf == NULL) ||
> > (conf->ring == NULL) ||
> > - (conf->ring->prod.single && is_multi) ||
> > - (!(conf->ring->prod.single) && !is_multi) ||
> > + (rte_ring_prod_single(conf->ring) && is_multi) ||
> > + (!rte_ring_prod_single(conf->ring) && !is_multi) ||
> > (conf->tx_burst_sz > RTE_PORT_IN_BURST_SIZE_MAX)) {
> > RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
> > return NULL;
> > @@ -440,8 +440,8 @@ rte_port_ring_writer_nodrop_create_internal(void
> > *params, int socket_id,
> > /* Check input parameters */
> > if ((conf == NULL) ||
> > (conf->ring == NULL) ||
> > - (conf->ring->prod.single && is_multi) ||
> > - (!(conf->ring->prod.single) && !is_multi) ||
> > + (rte_ring_prod_single(conf->ring) && is_multi) ||
> > + (!rte_ring_prod_single(conf->ring) && !is_multi) ||
> > (conf->tx_burst_sz > RTE_PORT_IN_BURST_SIZE_MAX)) {
> > RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
> > return NULL;
> > diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c index
> > 77e5de099..fa5733907 100644
> > --- a/lib/librte_ring/rte_ring.c
> > +++ b/lib/librte_ring/rte_ring.c
> > @@ -106,8 +106,10 @@ rte_ring_init(struct rte_ring *r, const char *name,
> > unsigned count,
> > if (ret < 0 || ret >= (int)sizeof(r->name))
> > return -ENAMETOOLONG;
> > r->flags = flags;
> > - r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
> > - r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
> > + r->prod.sync_type = (flags & RING_F_SP_ENQ) ?
> > + RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
> > + r->cons.sync_type = (flags & RING_F_SC_DEQ) ?
> > + RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
> >
> > if (flags & RING_F_EXACT_SZ) {
> > r->size = rte_align32pow2(count + 1); diff --git
> > a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h index
> > 18fc5d845..d4775a063 100644
> > --- a/lib/librte_ring/rte_ring.h
> > +++ b/lib/librte_ring/rte_ring.h
> > @@ -61,11 +61,27 @@ enum rte_ring_queue_behavior { #define
> > RTE_RING_NAMESIZE (RTE_MEMZONE_NAMESIZE - \
> > sizeof(RTE_RING_MZ_PREFIX) + 1)
> >
> > -/* structure to hold a pair of head/tail values and other metadata */
> > +/** prod/cons sync types */
> > +enum rte_ring_sync_type {
> > + RTE_RING_SYNC_MT, /**< multi-thread safe (default mode) */
> > + RTE_RING_SYNC_ST, /**< single thread only */
> > +};
> > +
> > +/**
> > + * structure to hold a pair of head/tail values and other metadata.
> > + * Depending on sync_type format of that structure might be different,
> > + * but offset for *sync_type* and *tail* values should remain the same.
> > + */
> > struct rte_ring_headtail {
> > - volatile uint32_t head; /**< Prod/consumer head. */
> > - volatile uint32_t tail; /**< Prod/consumer tail. */
> > - uint32_t single; /**< True if single prod/cons */
> > + volatile uint32_t head; /**< prod/consumer head. */
> > + volatile uint32_t tail; /**< prod/consumer tail. */
> > + RTE_STD_C11
> > + union {
> > + /** sync type of prod/cons */
> > + enum rte_ring_sync_type sync_type;
> > + /** deprecated - True if single prod/cons */
> > + uint32_t single;
> > + };
> > };
> >
> > /**
> > @@ -116,11 +132,10 @@ struct rte_ring {
> > #define RING_F_EXACT_SZ 0x0004
> > #define RTE_RING_SZ_MASK (0x7fffffffU) /**< Ring size mask */
> >
> > -/* @internal defines for passing to the enqueue dequeue worker functions
> > */ -#define __IS_SP 1 -#define __IS_MP 0 -#define __IS_SC 1 -#define
> > __IS_MC 0
> > +#define __IS_SP RTE_RING_SYNC_ST
> > +#define __IS_MP RTE_RING_SYNC_MT
> > +#define __IS_SC RTE_RING_SYNC_ST
> > +#define __IS_MC RTE_RING_SYNC_MT
> I think we can remove these #defines and use the new SYNC types
Wouldn't that introduce an API breakage?
Or we are ok here, as they are marked as internal?
I think I can for sure mark them as deprecated.
> >
> > /**
> > * Calculate the memory size needed for a ring @@ -420,7 +435,7 @@
> > rte_ring_mp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
> > unsigned int n, unsigned int *free_space) {
> > return __rte_ring_do_enqueue(r, obj_table, n,
> > RTE_RING_QUEUE_FIXED,
> > - __IS_MP, free_space);
> > + RTE_RING_SYNC_MT, free_space);
> > }
> >
> > /**
> > @@ -443,7 +458,7 @@ rte_ring_sp_enqueue_bulk(struct rte_ring *r, void *
> > const *obj_table,
> > unsigned int n, unsigned int *free_space) {
> > return __rte_ring_do_enqueue(r, obj_table, n,
> > RTE_RING_QUEUE_FIXED,
> > - __IS_SP, free_space);
> > + RTE_RING_SYNC_ST, free_space);
> > }
> >
> > /**
> > @@ -470,7 +485,7 @@ rte_ring_enqueue_bulk(struct rte_ring *r, void *
> > const *obj_table,
> > unsigned int n, unsigned int *free_space) {
> > return __rte_ring_do_enqueue(r, obj_table, n,
> > RTE_RING_QUEUE_FIXED,
> > - r->prod.single, free_space);
> > + r->prod.sync_type, free_space);
> > }
> >
> > /**
> > @@ -554,7 +569,7 @@ rte_ring_mc_dequeue_bulk(struct rte_ring *r, void
> > **obj_table,
> > unsigned int n, unsigned int *available) {
> > return __rte_ring_do_dequeue(r, obj_table, n,
> > RTE_RING_QUEUE_FIXED,
> > - __IS_MC, available);
> > + RTE_RING_SYNC_MT, available);
> > }
> >
> > /**
> > @@ -578,7 +593,7 @@ rte_ring_sc_dequeue_bulk(struct rte_ring *r, void
> > **obj_table,
> > unsigned int n, unsigned int *available) {
> > return __rte_ring_do_dequeue(r, obj_table, n,
> > RTE_RING_QUEUE_FIXED,
> > - __IS_SC, available);
> > + RTE_RING_SYNC_ST, available);
> > }
> >
> > /**
> > @@ -605,7 +620,7 @@ rte_ring_dequeue_bulk(struct rte_ring *r, void
> > **obj_table, unsigned int n,
> > unsigned int *available)
> > {
> > return __rte_ring_do_dequeue(r, obj_table, n,
> > RTE_RING_QUEUE_FIXED,
> > - r->cons.single, available);
> > + r->cons.sync_type, available);
> > }
> >
> > /**
> > @@ -777,6 +792,62 @@ rte_ring_get_capacity(const struct rte_ring *r)
> > return r->capacity;
> > }
> >
> > +/**
> > + * Return sync type used by producer in the ring.
> > + *
> > + * @param r
> > + * A pointer to the ring structure.
> > + * @return
> > + * Producer sync type value.
> > + */
> > +static inline enum rte_ring_sync_type
> > +rte_ring_get_prod_sync_type(const struct rte_ring *r) {
> > + return r->prod.sync_type;
> > +}
> > +
> > +/**
> > + * Check is the ring for single producer.
> ^^ if
> > + *
> > + * @param r
> > + * A pointer to the ring structure.
> > + * @return
> > + * true if ring is SP, zero otherwise.
> > + */
> > +static inline int
> > +rte_ring_prod_single(const struct rte_ring *r) {
> would rte_ring_is_prod_single better?
Ok, can rename.
>
> > + return (rte_ring_get_prod_sync_type(r) == RTE_RING_SYNC_ST); }
> > +
> > +/**
> > + * Return sync type used by consumer in the ring.
> > + *
> > + * @param r
> > + * A pointer to the ring structure.
> > + * @return
> > + * Consumer sync type value.
> > + */
> > +static inline enum rte_ring_sync_type
> > +rte_ring_get_cons_sync_type(const struct rte_ring *r) {
> > + return r->cons.sync_type;
> > +}
> > +
> > +/**
> > + * Check is the ring for single consumer.
> > + *
> > + * @param r
> > + * A pointer to the ring structure.
> > + * @return
> > + * true if ring is SC, zero otherwise.
> > + */
> > +static inline int
> > +rte_ring_cons_single(const struct rte_ring *r) {
> > + return (rte_ring_get_cons_sync_type(r) == RTE_RING_SYNC_ST); }
> > +
> All these new functions are not required to be called in the data path. They can be made non-inline.
Well, all these functions are introduced to encourage people not to
access ring fields sync_type/single directly but use functions instead.
I don't know do people access ring.single directly at data-path or not,
but assuming that they do - making these functions not-inline would
force them to ignore these functions and keep accessing it directly.
That was my thoughts besides making them inline.
I think we have the same for get_size/get_capacity().
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [RFC 1/3] eventdev: allow for event devices requiring maintenance
2020-04-09 13:33 3% ` Eads, Gage
@ 2020-04-09 14:14 0% ` Mattias Rönnblom
0 siblings, 0 replies; 200+ results
From: Mattias Rönnblom @ 2020-04-09 14:14 UTC (permalink / raw)
To: Eads, Gage, Jerin Jacob; +Cc: dpdk-dev, Jerin Jacob, Richardson, Bruce
On 2020-04-09 15:33, Eads, Gage wrote:
>>>> diff --git a/lib/librte_eventdev/rte_eventdev.h
>>>> b/lib/librte_eventdev/rte_eventdev.h
>>>> index 226f352ad..d69150792 100644
>>>> --- a/lib/librte_eventdev/rte_eventdev.h
>>>> +++ b/lib/librte_eventdev/rte_eventdev.h
>>>> @@ -289,6 +289,15 @@ struct rte_event;
>>>> * single queue to each port or map a single queue to many port.
>>>> */
>>>>
>>>> +#define RTE_EVENT_DEV_CAP_REQUIRES_MAINT (1ULL << 9) /**<
>> Event
>>>> +device requires calls to rte_event_maintain() during
>>> This scheme would call for DSW specific API handling in fastpath.
>>
>> Initially this would be so, but buffering events might yield performance
>> benefits for more event devices than DSW.
>>
> I agree. For applications that process and enqueue one event at a time, buffering in the PMD could give a performance boost with minimal code changes (assuming the application can tolerate higher packet latency caused by buffering).
>
>> In an application, it's often convenient, but sub-optimal from a
>> performance point of view, to do single-event enqueue operations. The
>> alternative is to use an application-level buffer, and the flush this
>> buffer with rte_event_enqueue_burst(). If you allow the event device to
>> buffer, you get the simplicity of single-event enqueue operations, but
>> without taking any noticeable performance hit.
>>
>>
>>>> + * periods when neither rte_event_dequeue_burst() nor
>>> The typical worker thread will be
>>> while (1) {
>>> rte_event_dequeue_burst();
>>> ..proess..
>>> rte_event_enqueue_burst();
>>> }
>>> If so, Why DSW driver can't do the maintenance in driver context in
>>> dequeue() call.
>>>
>> DSW already does maintenance on dequeue, and works well in the above
>> scenario. The typical worker does not need to care about the
>> rte_event_maintain() functions, since it dequeues events on a regular basis.
>>
>>
>> What this RFC addresses is the more atypical (but still fairly common)
>> case of a port being neither dequeued to or enqueued from on a regular
>> basis. The timer and ethernet rx adapters are examples of such.
>>
> Those two adapters have application-level buffering already, so adding PMD-level buffering feels unnecessary. Could DSW support this behavior on a port-by-port basis?
Flushing event buffers is just one of DSW's "background tasks". It also
updates the load estimate, so that other ports, considering migrating
flows to this port, has a recent-enough data to work with. In addition,
DSW periodically considered flow migration (i.e. load balancing), which
includes signaling between the ports. Even idle ports needs to respond
to these signals, thus they need to be "maintained". While buffering can
be made optional, the rest of the above can't.
DPDK eventdev seems to aspire to allow distributed scheduler
implementation, considering it has the
RTE_EVENT_DEV_CAP_DISTRIBUTED_SCHED flag since long. I've put some
thought into this, and I have yet to find a solution which can avoid
this kind "background tasks" for an efficient atomic scheduler with
dynamic load balancing.
> If so, I'm picturing something like:
> - Add a "PMD buffering" eventdev capability
> - If an eventdev has that capability, its ports can be configured for PMD-level buffering (default: no buffering)
> -- Convert " uint8_t disable_implicit_release" to a flags bitmap (e.g. "uint8_t event_port_cfg"), with one flag for implicit release disable and another for PMD-level buffering
> -- I suspect we can maintain ABI compatibility with function versioning on rte_event_port_setup() and rte_event_port_default_conf_get(), and this flags bitmap could be extended out to 32 bits in 20.11.
> - Add "flush" semantics either to a new interface or extend an existing one. I'm partial to a new interface, to avoid an additional check in e.g. the dequeue code. And putting the flush in dequeue doesn't allow an app to batch across multiple iterations of the dequeue-process-enqueue loop.
> - Extend rte_event_port_attr_get() to allow users to query this new setting. Adapters that don't call the flush function could error out if the adapter's port is configured for PMD-level buffering.
>
> (eventdev should also forbid "PMD-level buffering" and "implicit release" used together...it's easy to imagine double-release errors occurring otherwise.)
> I think this accomplishes Mattias' objective, and there's no effect on existing apps or adapters unless they choose to enable this behavior.
>
> Granted, existing apps would likely see performance loss with dsw until they enable this config option. But perhaps it's worth it to get this behavior properly supported in the interface.
>
> Thanks,
> Gage
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH] build: add arm 32bit cross compilation to Meson
@ 2020-04-09 14:14 3% Juraj Linkeš
2020-04-13 6:23 0% ` Ruifeng Wang
0 siblings, 1 reply; 200+ results
From: Juraj Linkeš @ 2020-04-09 14:14 UTC (permalink / raw)
To: bruce.richardson; +Cc: dev, Juraj Linkeš
* Add arm-linux-gnueabihf cross-file
* Add generic and default arm 32 bit flags to arm meson.build
* Add support for disabling drivers using flags defined in Meson
* Change checks from dpdk_conf.has() to dpdk.conf.get()
* When processing which drivers to build, check whether the
appropriate RTE flag isn't set to false
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
app/test-pmd/meson.build | 4 +-
app/test/meson.build | 2 +-
config/arm/arm_armv7a_linux_gcc | 17 ++++
config/arm/meson.build | 157 +++++++++++++++++++++-------------
drivers/meson.build | 13 ++-
drivers/net/kni/meson.build | 2 +-
examples/ethtool/meson.build | 2 +-
examples/ioat/meson.build | 2 +-
examples/kni/meson.build | 2 +-
examples/vm_power_manager/meson.build | 4 +-
lib/librte_port/meson.build | 2 +-
lib/meson.build | 2 +-
12 files changed, 133 insertions(+), 76 deletions(-)
create mode 100644 config/arm/arm_armv7a_linux_gcc
diff --git a/app/test-pmd/meson.build b/app/test-pmd/meson.build
index 6006c60f9..eac092d94 100644
--- a/app/test-pmd/meson.build
+++ b/app/test-pmd/meson.build
@@ -31,10 +31,10 @@ endif
if dpdk_conf.has('RTE_LIBRTE_BNXT_PMD')
deps += 'pmd_bnxt'
endif
-if dpdk_conf.has('RTE_LIBRTE_I40E_PMD')
+if dpdk_conf.has('RTE_LIBRTE_I40E_PMD') and dpdk_conf.get('RTE_LIBRTE_I40E_PMD')
deps += 'pmd_i40e'
endif
-if dpdk_conf.has('RTE_LIBRTE_IXGBE_PMD')
+if dpdk_conf.has('RTE_LIBRTE_IXGBE_PMD') and dpdk_conf.get('RTE_LIBRTE_IXGBE_PMD')
deps += 'pmd_ixgbe'
endif
if dpdk_conf.has('RTE_LIBRTE_SOFTNIC_PMD')
diff --git a/app/test/meson.build b/app/test/meson.build
index 351d29cb6..64d5d7bdb 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -348,7 +348,7 @@ endif
if dpdk_conf.has('RTE_LIBRTE_POWER')
test_deps += 'power'
endif
-if dpdk_conf.has('RTE_LIBRTE_KNI')
+if dpdk_conf.has('RTE_LIBRTE_KNI') and dpdk_conf.get('RTE_LIBRTE_KNI')
test_deps += 'kni'
endif
if dpdk_conf.has('RTE_LIBRTE_PDUMP')
diff --git a/config/arm/arm_armv7a_linux_gcc b/config/arm/arm_armv7a_linux_gcc
new file mode 100644
index 000000000..39566d2ea
--- /dev/null
+++ b/config/arm/arm_armv7a_linux_gcc
@@ -0,0 +1,17 @@
+[binaries]
+c = 'arm-linux-gnueabihf-gcc'
+cpp = 'arm-linux-gnueabihf-cpp'
+ar = 'arm-linux-gnueabihf-gcc-ar'
+strip = 'arm-linux-gnueabihf-strip'
+pkgconfig = 'arm-linux-gnueabihf-pkg-config'
+pcap-config = ''
+
+[host_machine]
+system = 'linux'
+cpu_family = 'arm'
+cpu = 'armv7-a'
+endian = 'little'
+
+[properties]
+implementor_id = 'generic_arm32'
+implementor_pn = 'default_arm32'
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 6e75e6d97..d359f2e14 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -34,6 +34,11 @@ flags_generic = [
['RTE_MAX_LCORE', 256],
['RTE_USE_C11_MEM_MODEL', true],
['RTE_CACHE_LINE_SIZE', 128]]
+flags_generic_arm32 = [
+ ['RTE_MACHINE', '"armv7a"'],
+ ['RTE_MAX_LCORE', 128],
+ ['RTE_USE_C11_MEM_MODEL', false],
+ ['RTE_CACHE_LINE_SIZE', 64]]
flags_arm = [
['RTE_MACHINE', '"armv8a"'],
['RTE_MAX_LCORE', 16],
@@ -63,6 +68,32 @@ flags_armada = [
['RTE_MAX_LCORE', 16]]
flags_default_extra = []
+flags_default_arm32_extra = [
+ ['RTE_ARCH_ARM_NEON_MEMCPY', false],
+ ['RTE_FORCE_INTRINSICS', true],
+ ['RTE_ARCH_STRICT_ALIGN', true],
+ ['RTE_EAL_NUMA_AWARE_HUGEPAGES', false],
+ ['RTE_LIBRTE_VHOST_NUMA', false],
+ ['RTE_LIBRTE_EAL_VMWARE_TSC_MAP_SUPPORT', false],
+ ['RTE_LIBRTE_KNI', false],
+ ['RTE_LIBRTE_ARK_PMD', false],
+ ['RTE_LIBRTE_EM_PMD', false],
+ ['RTE_LIBRTE_IGB_PMD', false],
+ ['RTE_LIBRTE_CXGBE_PMD', false],
+ ['RTE_LIBRTE_E1000_PMD', false],
+ ['RTE_LIBRTE_ENIC_PMD', false],
+ ['RTE_LIBRTE_FM10K_PMD', false],
+ ['RTE_LIBRTE_I40E_PMD', false],
+ ['RTE_LIBRTE_IXGBE_PMD', false],
+ ['RTE_LIBRTE_VMXNET3_PMD', false],
+ ['RTE_LIBRTE_QEDE_PMD', false],
+ ['RTE_LIBRTE_SFC_EFX_PMD', false],
+ ['RTE_LIBRTE_AVP_PMD', false],
+ ['RTE_LIBRTE_NFP_PMD', false],
+ ['RTE_LIBRTE_HINIC_PMD', false],
+ ['RTE_LIBRTE_HNS3_PMD', false],
+ ['RTE_LIBRTE_PMD_IOAT_RAWDEV', false],
+ ['RTE_LIBRTE_IONIC_PMD', false]]
flags_n1sdp_extra = [
['RTE_MACHINE', '"n1sdp"'],
['RTE_MAX_NUMA_NODES', 1],
@@ -99,6 +130,9 @@ machine_args_generic = [
['0xd0b', ['-mcpu=cortex-a76']],
['0xd0c', ['-march=armv8.2-a+crc+crypto', '-mcpu=neoverse-n1'], flags_n1sdp_extra]]
+machine_args_generic_arm32 = [
+ ['default_arm32', ['-march=armv7-a', '-mtune=cortex-a9', '-mfpu=neon'], flags_default_arm32_extra]]
+
machine_args_cavium = [
['default', ['-march=armv8-a+crc+crypto','-mcpu=thunderx']],
['native', ['-march=native']],
@@ -114,6 +148,7 @@ machine_args_emag = [
## Arm implementer ID (ARM DDI 0487C.a, Section G7.2.106, Page G7-5321)
impl_generic = ['Generic armv8', flags_generic, machine_args_generic]
+impl_generic_arm32 = ['Generic armv7', flags_generic_arm32, machine_args_generic_arm32]
impl_0x41 = ['Arm', flags_arm, machine_args_generic]
impl_0x42 = ['Broadcom', flags_generic, machine_args_generic]
impl_0x43 = ['Cavium', flags_cavium, machine_args_cavium]
@@ -136,74 +171,74 @@ if not dpdk_conf.get('RTE_ARCH_64')
dpdk_conf.set('RTE_ARCH_ARMv7', 1)
# the minimum architecture supported, armv7-a, needs the following,
# mk/machine/armv7a/rte.vars.mk sets it too
- machine_args += '-mfpu=neon'
else
dpdk_conf.set('RTE_CACHE_LINE_SIZE', 128)
dpdk_conf.set('RTE_ARCH_ARM64', 1)
+endif
- machine = []
- cmd_generic = ['generic', '', '', 'default', '']
- cmd_output = cmd_generic # Set generic by default
- machine_args = [] # Clear previous machine args
- if arm_force_default_march and not meson.is_cross_build()
+machine = []
+machine_args = [] # Clear previous machine args
+cmd_generic = ['generic', '', '', 'default', '']
+cmd_output = cmd_generic # Set generic by default
+if arm_force_default_march and not meson.is_cross_build()
+ machine = impl_generic
+ impl_pn = 'default'
+elif not meson.is_cross_build()
+ # The script returns ['Implementer', 'Variant', 'Architecture',
+ # 'Primary Part number', 'Revision']
+ detect_vendor = find_program(join_paths(
+ meson.current_source_dir(), 'armv8_machine.py'))
+ cmd = run_command(detect_vendor.path())
+ if cmd.returncode() == 0
+ cmd_output = cmd.stdout().to_lower().strip().split(' ')
+ endif
+ # Set to generic if variable is not found
+ machine = get_variable('impl_' + cmd_output[0], ['generic'])
+ if machine[0] == 'generic'
machine = impl_generic
- impl_pn = 'default'
- elif not meson.is_cross_build()
- # The script returns ['Implementer', 'Variant', 'Architecture',
- # 'Primary Part number', 'Revision']
- detect_vendor = find_program(join_paths(
- meson.current_source_dir(), 'armv8_machine.py'))
- cmd = run_command(detect_vendor.path())
- if cmd.returncode() == 0
- cmd_output = cmd.stdout().to_lower().strip().split(' ')
- endif
- # Set to generic if variable is not found
- machine = get_variable('impl_' + cmd_output[0], ['generic'])
- if machine[0] == 'generic'
- machine = impl_generic
- cmd_output = cmd_generic
- endif
- impl_pn = cmd_output[3]
- if arm_force_native_march == true
- impl_pn = 'native'
- endif
- else
- impl_id = meson.get_cross_property('implementor_id', 'generic')
- impl_pn = meson.get_cross_property('implementor_pn', 'default')
- machine = get_variable('impl_' + impl_id)
+ cmd_output = cmd_generic
endif
-
- # Apply Common Defaults. These settings may be overwritten by machine
- # settings later.
- foreach flag: flags_common_default
- if flag.length() > 0
- dpdk_conf.set(flag[0], flag[1])
- endif
- endforeach
-
- message('Implementer : ' + machine[0])
- foreach flag: machine[1]
- if flag.length() > 0
- dpdk_conf.set(flag[0], flag[1])
- endif
- endforeach
-
- foreach marg: machine[2]
- if marg[0] == impl_pn
- foreach flag: marg[1]
- if cc.has_argument(flag)
- machine_args += flag
- endif
- endforeach
- # Apply any extra machine specific flags.
- foreach flag: marg.get(2, flags_default_extra)
- if flag.length() > 0
- dpdk_conf.set(flag[0], flag[1])
- endif
- endforeach
- endif
- endforeach
+ impl_pn = cmd_output[3]
+ if arm_force_native_march == true
+ impl_pn = 'native'
+ endif
+else
+ impl_id = meson.get_cross_property('implementor_id', 'generic')
+ impl_pn = meson.get_cross_property('implementor_pn', 'default')
+ machine = get_variable('impl_' + impl_id)
endif
+
+# Apply Common Defaults. These settings may be overwritten by machine
+# settings later.
+foreach flag: flags_common_default
+ if flag.length() > 0
+ dpdk_conf.set(flag[0], flag[1])
+ endif
+endforeach
+
+message('Implementer : ' + machine[0])
+foreach flag: machine[1]
+ if flag.length() > 0
+ dpdk_conf.set(flag[0], flag[1])
+ endif
+endforeach
+
+foreach marg: machine[2]
+ if marg[0] == impl_pn
+ foreach flag: marg[1]
+ if cc.has_argument(flag)
+ machine_args += flag
+ endif
+ endforeach
+ # Apply any extra machine specific flags.
+ foreach flag: marg.get(2, flags_default_extra)
+ if flag.length() > 0
+ dpdk_conf.set(flag[0], flag[1])
+ message('Setting flag: @0@: @1@'.format(flag[0], flag[1]))
+ endif
+ endforeach
+ endif
+endforeach
message(machine_args)
if (cc.get_define('__ARM_NEON', args: machine_args) != '' or
diff --git a/drivers/meson.build b/drivers/meson.build
index 5502bf992..5763752fc 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -71,6 +71,14 @@ foreach class:dpdk_driver_classes
reason = 'Explicitly disabled via build config'
endif
endforeach
+ if fmt_name == ''
+ fmt_name = name
+ endif
+ conf_flag = config_flag_fmt.format(fmt_name.to_upper())
+ if dpdk_conf.has(conf_flag) and not dpdk_conf.get(conf_flag)
+ build = false
+ reason = 'Explicitly disabled via build config'
+ endif
if build
# get dependency objs from strings
shared_deps = ext_deps
@@ -100,10 +108,7 @@ foreach class:dpdk_driver_classes
else
class_drivers += name
- if fmt_name == ''
- fmt_name = name
- endif
- dpdk_conf.set(config_flag_fmt.format(fmt_name.to_upper()),1)
+ dpdk_conf.set(conf_flag, true)
lib_name = driver_name_fmt.format(fmt_name)
if allow_experimental_apis
diff --git a/drivers/net/kni/meson.build b/drivers/net/kni/meson.build
index 0539b4768..79a16ae5b 100644
--- a/drivers/net/kni/meson.build
+++ b/drivers/net/kni/meson.build
@@ -2,7 +2,7 @@
# Copyright(c) 2018 Intel Corporation
# this driver can be built if-and-only-if KNI library is buildable
-build = dpdk_conf.has('RTE_LIBRTE_KNI')
+build = dpdk_conf.has('RTE_LIBRTE_KNI') and dpdk_conf.get('RTE_LIBRTE_KNI')
reason = 'missing dependency, DPDK KNI library'
sources = files('rte_eth_kni.c')
deps += 'kni'
diff --git a/examples/ethtool/meson.build b/examples/ethtool/meson.build
index bc7a35514..814b8ada1 100644
--- a/examples/ethtool/meson.build
+++ b/examples/ethtool/meson.build
@@ -13,7 +13,7 @@ sources = files('lib/rte_ethtool.c',
includes = include_directories('lib', 'ethtool-app')
deps += 'bus_pci'
-if dpdk_conf.has('RTE_LIBRTE_IXGBE_PMD')
+if dpdk_conf.has('RTE_LIBRTE_IXGBE_PMD') and dpdk_conf.get('RTE_LIBRTE_IXGBE_PMD')
deps += 'pmd_ixgbe'
endif
diff --git a/examples/ioat/meson.build b/examples/ioat/meson.build
index ed8328963..029126353 100644
--- a/examples/ioat/meson.build
+++ b/examples/ioat/meson.build
@@ -6,7 +6,7 @@
# To build this example as a standalone application with an already-installed
# DPDK instance, use 'make'
-build = dpdk_conf.has('RTE_LIBRTE_PMD_IOAT_RAWDEV')
+build = dpdk_conf.has('RTE_LIBRTE_PMD_IOAT_RAWDEV') and dpdk_conf.get('RTE_LIBRTE_PMD_IOAT_RAWDEV')
deps += ['rawdev_ioat']
diff --git a/examples/kni/meson.build b/examples/kni/meson.build
index fd6ae4442..d9514b155 100644
--- a/examples/kni/meson.build
+++ b/examples/kni/meson.build
@@ -7,7 +7,7 @@
# DPDK instance, use 'make'
# this app can be built if-and-only-if KNI library is buildable
-build = dpdk_conf.has('RTE_LIBRTE_KNI')
+build = dpdk_conf.has('RTE_LIBRTE_KNI') and dpdk_conf.get('RTE_LIBRTE_KNI')
deps += ['kni', 'bus_pci']
sources = files(
'main.c'
diff --git a/examples/vm_power_manager/meson.build b/examples/vm_power_manager/meson.build
index 20a4a05b3..9d591813c 100644
--- a/examples/vm_power_manager/meson.build
+++ b/examples/vm_power_manager/meson.build
@@ -17,11 +17,11 @@ if dpdk_conf.has('RTE_LIBRTE_BNXT_PMD')
deps += ['pmd_bnxt']
endif
-if dpdk_conf.has('RTE_LIBRTE_I40E_PMD')
+if dpdk_conf.has('RTE_LIBRTE_I40E_PMD') and dpdk_conf.get('RTE_LIBRTE_I40E_PMD')
deps += ['pmd_i40e']
endif
-if dpdk_conf.has('RTE_LIBRTE_IXGBE_PMD')
+if dpdk_conf.has('RTE_LIBRTE_IXGBE_PMD') and dpdk_conf.get('RTE_LIBRTE_IXGBE_PMD')
deps += ['pmd_ixgbe']
endif
diff --git a/lib/librte_port/meson.build b/lib/librte_port/meson.build
index 0d5ede44a..5f3d190fa 100644
--- a/lib/librte_port/meson.build
+++ b/lib/librte_port/meson.build
@@ -28,7 +28,7 @@ if dpdk_conf.has('RTE_PORT_PCAP')
ext_deps += pcap_dep # dependency provided in config/meson.build
endif
-if dpdk_conf.has('RTE_LIBRTE_KNI')
+if dpdk_conf.has('RTE_LIBRTE_KNI') and dpdk_conf.get('RTE_LIBRTE_KNI')
sources += files('rte_port_kni.c')
headers += files('rte_port_kni.h')
deps += 'kni'
diff --git a/lib/meson.build b/lib/meson.build
index 9c3cc55d5..59ccfed9d 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -86,7 +86,7 @@ foreach l:libraries
set_variable(name.underscorify() + '_disable_reason', reason)
else
enabled_libs += name
- dpdk_conf.set('RTE_LIBRTE_' + name.to_upper(), 1)
+ dpdk_conf.set('RTE_LIBRTE_' + name.to_upper(), true)
install_headers(headers)
libname = 'rte_' + name
--
2.11.0
NOTES: tested here: https://travis-ci.com/github/jlinkes/dpdk/builds/159597484
There are two issues I would like to get feedback for:
1. the aarch64 -> arm cross compilation fails when compiling l3fwd example [0].
I think this failure needs to be fixed by arm devs, but I would like to have
this confirmed.
2. the patch breaks ABI checks on aarch64 [1]. The reason is that the
RTE_LIBRTE_FM10K_PMD and RTE_LIBRTE_AVP_PMD flags in config/arm/meson.build
get properly applied, the libs don't get built and then check the fails when
it doesn't find them. I don't know whether the application of these flags is
desirable (and we would need to fix the ABI check) or whether we should
remove the flags.
[0] https://travis-ci.com/github/jlinkes/dpdk/jobs/317622662#L2672
[1] https://travis-ci.com/github/jlinkes/dpdk/jobs/317622661#L4488
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v3 3/9] ring: introduce RTS ring mode
2020-04-08 5:00 0% ` Honnappa Nagarahalli
@ 2020-04-09 14:52 0% ` Ananyev, Konstantin
2020-04-10 23:10 0% ` Honnappa Nagarahalli
0 siblings, 1 reply; 200+ results
From: Ananyev, Konstantin @ 2020-04-09 14:52 UTC (permalink / raw)
To: Honnappa Nagarahalli, dev; +Cc: david.marchand, jielong.zjl, nd, nd
> > Introduce relaxed tail sync (RTS) mode for MT ring synchronization.
> > Aim to reduce stall times in case when ring is used on overcommited cpus
> > (multiple active threads on the same cpu).
> > The main difference from original MP/MC algorithm is that tail value is
> > increased not by every thread that finished enqueue/dequeue, but only by the
> > last one.
> > That allows threads to avoid spinning on ring tail value, leaving actual tail
> > value change to the last thread in the update queue.
> >
> > check-abi.sh reports what I believe is a false-positive about ring cons/prod
> > changes. As a workaround, devtools/libabigail.abignore is updated to suppress
> > *struct ring* related errors.
> This can be removed from the commit message.
>
> >
> > Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> > ---
> > devtools/libabigail.abignore | 7 +
> > lib/librte_ring/Makefile | 5 +-
> > lib/librte_ring/meson.build | 5 +-
> > lib/librte_ring/rte_ring.c | 100 +++++++-
> > lib/librte_ring/rte_ring.h | 110 ++++++++-
> > lib/librte_ring/rte_ring_elem.h | 86 ++++++-
> > lib/librte_ring/rte_ring_rts.h | 316 +++++++++++++++++++++++++
> > lib/librte_ring/rte_ring_rts_elem.h | 205 ++++++++++++++++
> > lib/librte_ring/rte_ring_rts_generic.h | 210 ++++++++++++++++
> > 9 files changed, 1015 insertions(+), 29 deletions(-) create mode 100644
> > lib/librte_ring/rte_ring_rts.h create mode 100644
> > lib/librte_ring/rte_ring_rts_elem.h
> > create mode 100644 lib/librte_ring/rte_ring_rts_generic.h
> >
> > diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore index
> > a59df8f13..cd86d89ca 100644
> > --- a/devtools/libabigail.abignore
> > +++ b/devtools/libabigail.abignore
> > @@ -11,3 +11,10 @@
> > type_kind = enum
> > name = rte_crypto_asym_xform_type
> > changed_enumerators = RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
> > +; Ignore updates of ring prod/cons
> > +[suppress_type]
> > + type_kind = struct
> > + name = rte_ring
> > +[suppress_type]
> > + type_kind = struct
> > + name = rte_event_ring
> Does this block the reporting of these structures forever?
Till we'll have a fix in libabigail, then we can remove these lines.
I don't know any better alternative.
>
> > diff --git a/lib/librte_ring/Makefile b/lib/librte_ring/Makefile index
> > 917c560ad..8f5c284cc 100644
> > --- a/lib/librte_ring/Makefile
> > +++ b/lib/librte_ring/Makefile
> > @@ -18,6 +18,9 @@ SRCS-$(CONFIG_RTE_LIBRTE_RING) := rte_ring.c
> > SYMLINK-$(CONFIG_RTE_LIBRTE_RING)-include := rte_ring.h \
> > rte_ring_elem.h \
> > rte_ring_generic.h \
> > - rte_ring_c11_mem.h
> > + rte_ring_c11_mem.h \
> > + rte_ring_rts.h \
> > + rte_ring_rts_elem.h \
> > + rte_ring_rts_generic.h
> >
> > include $(RTE_SDK)/mk/rte.lib.mk
> > diff --git a/lib/librte_ring/meson.build b/lib/librte_ring/meson.build index
> > f2f3ccc88..612936afb 100644
> > --- a/lib/librte_ring/meson.build
> > +++ b/lib/librte_ring/meson.build
> > @@ -5,7 +5,10 @@ sources = files('rte_ring.c') headers = files('rte_ring.h',
> > 'rte_ring_elem.h',
> > 'rte_ring_c11_mem.h',
> > - 'rte_ring_generic.h')
> > + 'rte_ring_generic.h',
> > + 'rte_ring_rts.h',
> > + 'rte_ring_rts_elem.h',
> > + 'rte_ring_rts_generic.h')
> >
> > # rte_ring_create_elem and rte_ring_get_memsize_elem are experimental
> > allow_experimental_apis = true diff --git a/lib/librte_ring/rte_ring.c
> > b/lib/librte_ring/rte_ring.c index fa5733907..222eec0fb 100644
> > --- a/lib/librte_ring/rte_ring.c
> > +++ b/lib/librte_ring/rte_ring.c
> > @@ -45,6 +45,9 @@ EAL_REGISTER_TAILQ(rte_ring_tailq)
> > /* true if x is a power of 2 */
> > #define POWEROF2(x) ((((x)-1) & (x)) == 0)
> >
> > +/* by default set head/tail distance as 1/8 of ring capacity */
> > +#define HTD_MAX_DEF 8
> > +
> > /* return the size of memory occupied by a ring */ ssize_t
> > rte_ring_get_memsize_elem(unsigned int esize, unsigned int count) @@ -
> > 79,11 +82,84 @@ rte_ring_get_memsize(unsigned int count)
> > return rte_ring_get_memsize_elem(sizeof(void *), count); }
> >
> > +/*
> > + * internal helper function to reset prod/cons head-tail values.
> > + */
> > +static void
> > +reset_headtail(void *p)
> > +{
> > + struct rte_ring_headtail *ht;
> > + struct rte_ring_rts_headtail *ht_rts;
> > +
> > + ht = p;
> > + ht_rts = p;
> > +
> > + switch (ht->sync_type) {
> > + case RTE_RING_SYNC_MT:
> > + case RTE_RING_SYNC_ST:
> > + ht->head = 0;
> > + ht->tail = 0;
> > + break;
> > + case RTE_RING_SYNC_MT_RTS:
> > + ht_rts->head.raw = 0;
> > + ht_rts->tail.raw = 0;
> > + break;
> > + default:
> > + /* unknown sync mode */
> > + RTE_ASSERT(0);
> > + }
> > +}
> > +
> > void
> > rte_ring_reset(struct rte_ring *r)
> > {
> > - r->prod.head = r->cons.head = 0;
> > - r->prod.tail = r->cons.tail = 0;
> > + reset_headtail(&r->prod);
> > + reset_headtail(&r->cons);
> > +}
> > +
> > +/*
> > + * helper function, calculates sync_type values for prod and cons
> > + * based on input flags. Returns zero at success or negative
> > + * errno value otherwise.
> > + */
> > +static int
> > +get_sync_type(uint32_t flags, enum rte_ring_sync_type *prod_st,
> > + enum rte_ring_sync_type *cons_st)
> > +{
> > + static const uint32_t prod_st_flags =
> > + (RING_F_SP_ENQ | RING_F_MP_RTS_ENQ);
> > + static const uint32_t cons_st_flags =
> > + (RING_F_SC_DEQ | RING_F_MC_RTS_DEQ);
> > +
> > + switch (flags & prod_st_flags) {
> > + case 0:
> > + *prod_st = RTE_RING_SYNC_MT;
> > + break;
> > + case RING_F_SP_ENQ:
> > + *prod_st = RTE_RING_SYNC_ST;
> > + break;
> > + case RING_F_MP_RTS_ENQ:
> > + *prod_st = RTE_RING_SYNC_MT_RTS;
> > + break;
> > + default:
> > + return -EINVAL;
> > + }
> > +
> > + switch (flags & cons_st_flags) {
> > + case 0:
> > + *cons_st = RTE_RING_SYNC_MT;
> > + break;
> > + case RING_F_SC_DEQ:
> > + *cons_st = RTE_RING_SYNC_ST;
> > + break;
> > + case RING_F_MC_RTS_DEQ:
> > + *cons_st = RTE_RING_SYNC_MT_RTS;
> > + break;
> > + default:
> > + return -EINVAL;
> > + }
> > +
> > + return 0;
> > }
> >
> > int
> > @@ -100,16 +176,20 @@ rte_ring_init(struct rte_ring *r, const char *name,
> > unsigned count,
> > RTE_BUILD_BUG_ON((offsetof(struct rte_ring, prod) &
> > RTE_CACHE_LINE_MASK) != 0);
> >
> > + RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, sync_type) !=
> > + offsetof(struct rte_ring_rts_headtail, sync_type));
> > + RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, tail) !=
> > + offsetof(struct rte_ring_rts_headtail, tail.val.pos));
> > +
> > /* init the ring structure */
> > memset(r, 0, sizeof(*r));
> > ret = strlcpy(r->name, name, sizeof(r->name));
> > if (ret < 0 || ret >= (int)sizeof(r->name))
> > return -ENAMETOOLONG;
> > r->flags = flags;
> > - r->prod.sync_type = (flags & RING_F_SP_ENQ) ?
> > - RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
> > - r->cons.sync_type = (flags & RING_F_SC_DEQ) ?
> > - RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
> > + ret = get_sync_type(flags, &r->prod.sync_type, &r->cons.sync_type);
> > + if (ret != 0)
> > + return ret;
> >
> > if (flags & RING_F_EXACT_SZ) {
> > r->size = rte_align32pow2(count + 1); @@ -126,8 +206,12
> > @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
> > r->mask = count - 1;
> > r->capacity = r->mask;
> > }
> > - r->prod.head = r->cons.head = 0;
> > - r->prod.tail = r->cons.tail = 0;
> > +
> > + /* set default values for head-tail distance */
> > + if (flags & RING_F_MP_RTS_ENQ)
> > + rte_ring_set_prod_htd_max(r, r->capacity / HTD_MAX_DEF);
> > + if (flags & RING_F_MC_RTS_DEQ)
> > + rte_ring_set_cons_htd_max(r, r->capacity / HTD_MAX_DEF);
> >
> > return 0;
> > }
> > diff --git a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h index
> > d4775a063..f6f084d79 100644
> > --- a/lib/librte_ring/rte_ring.h
> > +++ b/lib/librte_ring/rte_ring.h
> > @@ -48,6 +48,7 @@ extern "C" {
> > #include <rte_branch_prediction.h>
> > #include <rte_memzone.h>
> > #include <rte_pause.h>
> > +#include <rte_debug.h>
> >
> > #define RTE_TAILQ_RING_NAME "RTE_RING"
> >
> > @@ -65,10 +66,13 @@ enum rte_ring_queue_behavior { enum
> > rte_ring_sync_type {
> > RTE_RING_SYNC_MT, /**< multi-thread safe (default mode) */
> > RTE_RING_SYNC_ST, /**< single thread only */
> > +#ifdef ALLOW_EXPERIMENTAL_API
> > + RTE_RING_SYNC_MT_RTS, /**< multi-thread relaxed tail sync */
> > #endif
> > };
> >
> > /**
> > - * structure to hold a pair of head/tail values and other metadata.
> > + * structures to hold a pair of head/tail values and other metadata.
> > * Depending on sync_type format of that structure might be different,
> > * but offset for *sync_type* and *tail* values should remain the same.
> > */
> > @@ -84,6 +88,21 @@ struct rte_ring_headtail {
> > };
> > };
> >
> > +union rte_ring_ht_poscnt {
> nit, this is specific to RTS, may be change this to rte_ring_rts_ht_poscnt?
Ok.
>
> > + uint64_t raw;
> > + struct {
> > + uint32_t cnt; /**< head/tail reference counter */
> > + uint32_t pos; /**< head/tail position */
> > + } val;
> > +};
> > +
> > +struct rte_ring_rts_headtail {
> > + volatile union rte_ring_ht_poscnt tail;
> > + enum rte_ring_sync_type sync_type; /**< sync type of prod/cons */
> > + uint32_t htd_max; /**< max allowed distance between head/tail */
> > + volatile union rte_ring_ht_poscnt head; };
> > +
> > /**
> > * An RTE ring structure.
> > *
> > @@ -111,11 +130,21 @@ struct rte_ring {
> > char pad0 __rte_cache_aligned; /**< empty cache line */
> >
> > /** Ring producer status. */
> > - struct rte_ring_headtail prod __rte_cache_aligned;
> > + RTE_STD_C11
> > + union {
> > + struct rte_ring_headtail prod;
> > + struct rte_ring_rts_headtail rts_prod;
> > + } __rte_cache_aligned;
> > +
> > char pad1 __rte_cache_aligned; /**< empty cache line */
> >
> > /** Ring consumer status. */
> > - struct rte_ring_headtail cons __rte_cache_aligned;
> > + RTE_STD_C11
> > + union {
> > + struct rte_ring_headtail cons;
> > + struct rte_ring_rts_headtail rts_cons;
> > + } __rte_cache_aligned;
> > +
> > char pad2 __rte_cache_aligned; /**< empty cache line */ };
> >
> > @@ -132,6 +161,9 @@ struct rte_ring {
> > #define RING_F_EXACT_SZ 0x0004
> > #define RTE_RING_SZ_MASK (0x7fffffffU) /**< Ring size mask */
> >
> > +#define RING_F_MP_RTS_ENQ 0x0008 /**< The default enqueue is "MP RTS".
> > +*/ #define RING_F_MC_RTS_DEQ 0x0010 /**< The default dequeue is "MC
> > +RTS". */
> > +
> > #define __IS_SP RTE_RING_SYNC_ST
> > #define __IS_MP RTE_RING_SYNC_MT
> > #define __IS_SC RTE_RING_SYNC_ST
> > @@ -461,6 +493,10 @@ rte_ring_sp_enqueue_bulk(struct rte_ring *r, void *
> > const *obj_table,
> > RTE_RING_SYNC_ST, free_space);
> > }
> >
> > +#ifdef ALLOW_EXPERIMENTAL_API
> > +#include <rte_ring_rts.h>
> > +#endif
> > +
> > /**
> > * Enqueue several objects on a ring.
> > *
> > @@ -484,8 +520,21 @@ static __rte_always_inline unsigned int
> > rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
> > unsigned int n, unsigned int *free_space) {
> > - return __rte_ring_do_enqueue(r, obj_table, n,
> > RTE_RING_QUEUE_FIXED,
> > - r->prod.sync_type, free_space);
> > + switch (r->prod.sync_type) {
> > + case RTE_RING_SYNC_MT:
> > + return rte_ring_mp_enqueue_bulk(r, obj_table, n, free_space);
> > + case RTE_RING_SYNC_ST:
> > + return rte_ring_sp_enqueue_bulk(r, obj_table, n, free_space);
> Have you validated if these affect the performance for the existing APIs?
I run ring_pmd_perf_autotest
(AFAIK, that's the only one of our perf tests that calls rte_ring_enqueue/dequeue),
and didn't see any real difference in perf numbers.
> I am also wondering why should we support these new modes in the legacy APIs?
Majority of DPDK users still do use legacy API,
and I am not sure all of them will be happy to switch to _elem_ one manually.
Plus I can't see how we can justify that after let say:
rte_ring_init(ring, ..., RING_F_MP_HTS_ENQ | RING_F_MC_HTS_DEQ);
returns with success
valid call to rte_ring_enqueue(ring,...) should fail.
> I think users should move to use rte_ring_xxx_elem APIs. If users want to use RTS/HTS it will be a good time for them to move to new APIs.
If they use rte_ring_enqueue/dequeue all they have to do - just change flags in ring_create/ring_init call.
With what you suggest - they have to change every rte_ring_enqueue/dequeue
to rte_ring_elem_enqueue/dequeue.
That's much bigger code churn.
> They anyway have to test their code for RTS/HTS, might as well make the change to new APIs and test both.
> It will be less code to maintain for the community as well.
That's true, right now there is a lot of duplication between
_elem_ and legacy code.
Actually the only real diff between them - actual copying of the objects.
But I thought we are going to deal with that, just by
changing one day all legacy API to wrappers around _elem_ calls,
i.e something like:
static __rte_always_inline unsigned int
rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
return rte_ring_enqueue_elem_bulk(r, obj_table, sizeof(uintptr_t), n, free_space);
}
That way users will switch to new API automatically,
without any extra effort for them, and we will be able to remove legacy code.
Do you have some other thoughts here how to deal with this legacy/elem conversion?
>
> > #ifdef
> > +ALLOW_EXPERIMENTAL_API
> > + case RTE_RING_SYNC_MT_RTS:
> > + return rte_ring_mp_rts_enqueue_bulk(r, obj_table, n,
> > + free_space);
> > +#endif
> > + }
> > +
> > + /* valid ring should never reach this point */
> > + RTE_ASSERT(0);
> > + return 0;
> > }
> >
> > /**
> > @@ -619,8 +668,20 @@ static __rte_always_inline unsigned int
> > rte_ring_dequeue_bulk(struct rte_ring *r, void **obj_table, unsigned int n,
> > unsigned int *available)
> > {
> > - return __rte_ring_do_dequeue(r, obj_table, n,
> > RTE_RING_QUEUE_FIXED,
> > - r->cons.sync_type, available);
> > + switch (r->cons.sync_type) {
> > + case RTE_RING_SYNC_MT:
> > + return rte_ring_mc_dequeue_bulk(r, obj_table, n, available);
> > + case RTE_RING_SYNC_ST:
> > + return rte_ring_sc_dequeue_bulk(r, obj_table, n, available);
> > #ifdef
> > +ALLOW_EXPERIMENTAL_API
> > + case RTE_RING_SYNC_MT_RTS:
> > + return rte_ring_mc_rts_dequeue_bulk(r, obj_table, n,
> > available);
> > +#endif
> > + }
> > +
> > + /* valid ring should never reach this point */
> > + RTE_ASSERT(0);
> > + return 0;
> > }
> >
> > /**
> > @@ -940,8 +1001,21 @@ static __rte_always_inline unsigned
> > rte_ring_enqueue_burst(struct rte_ring *r, void * const *obj_table,
> > unsigned int n, unsigned int *free_space) {
> > - return __rte_ring_do_enqueue(r, obj_table, n,
> > RTE_RING_QUEUE_VARIABLE,
> > - r->prod.sync_type, free_space);
> > + switch (r->prod.sync_type) {
> > + case RTE_RING_SYNC_MT:
> > + return rte_ring_mp_enqueue_burst(r, obj_table, n,
> > free_space);
> > + case RTE_RING_SYNC_ST:
> > + return rte_ring_sp_enqueue_burst(r, obj_table, n, free_space);
> > #ifdef
> > +ALLOW_EXPERIMENTAL_API
> > + case RTE_RING_SYNC_MT_RTS:
> > + return rte_ring_mp_rts_enqueue_burst(r, obj_table, n,
> > + free_space);
> > +#endif
> > + }
> > +
> > + /* valid ring should never reach this point */
> > + RTE_ASSERT(0);
> > + return 0;
> > }
> >
> > /**
> > @@ -1020,9 +1094,21 @@ static __rte_always_inline unsigned
> > rte_ring_dequeue_burst(struct rte_ring *r, void **obj_table,
> > unsigned int n, unsigned int *available) {
> > - return __rte_ring_do_dequeue(r, obj_table, n,
> > - RTE_RING_QUEUE_VARIABLE,
> > - r->cons.sync_type, available);
> > + switch (r->cons.sync_type) {
> > + case RTE_RING_SYNC_MT:
> > + return rte_ring_mc_dequeue_burst(r, obj_table, n, available);
> > + case RTE_RING_SYNC_ST:
> > + return rte_ring_sc_dequeue_burst(r, obj_table, n, available);
> > #ifdef
> > +ALLOW_EXPERIMENTAL_API
> > + case RTE_RING_SYNC_MT_RTS:
> > + return rte_ring_mc_rts_dequeue_burst(r, obj_table, n,
> > + available);
> > +#endif
> > + }
> > +
> > + /* valid ring should never reach this point */
> > + RTE_ASSERT(0);
> > + return 0;
> > }
> >
> > #ifdef __cplusplus
> > diff --git a/lib/librte_ring/rte_ring_elem.h b/lib/librte_ring/rte_ring_elem.h
> > index 28f9836e6..5de0850dc 100644
> > --- a/lib/librte_ring/rte_ring_elem.h
> > +++ b/lib/librte_ring/rte_ring_elem.h
> > @@ -542,6 +542,8 @@ rte_ring_sp_enqueue_bulk_elem(struct rte_ring *r,
> > const void *obj_table,
> > RTE_RING_QUEUE_FIXED, __IS_SP, free_space); }
> >
> > +#include <rte_ring_rts_elem.h>
> > +
> > /**
> > * Enqueue several objects on a ring.
> > *
> > @@ -571,6 +573,26 @@ rte_ring_enqueue_bulk_elem(struct rte_ring *r,
> > const void *obj_table, {
> > return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
> > RTE_RING_QUEUE_FIXED, r->prod.sync_type,
> > free_space);
> > +
> > + switch (r->prod.sync_type) {
> > + case RTE_RING_SYNC_MT:
> > + return rte_ring_mp_enqueue_bulk_elem(r, obj_table, esize, n,
> > + free_space);
> > + case RTE_RING_SYNC_ST:
> > + return rte_ring_sp_enqueue_bulk_elem(r, obj_table, esize, n,
> > + free_space);
> > +#ifdef ALLOW_EXPERIMENTAL_API
> > + case RTE_RING_SYNC_MT_RTS:
> > + return rte_ring_mp_rts_enqueue_bulk_elem(r, obj_table,
> > esize, n,
> > + free_space);
> > +#endif
> > + }
> > +
> > + /* valid ring should never reach this point */
> > + RTE_ASSERT(0);
> > + if (free_space != NULL)
> > + *free_space = 0;
> > + return 0;
> > }
> >
> > /**
> > @@ -733,8 +755,25 @@ static __rte_always_inline unsigned int
> > rte_ring_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
> > unsigned int esize, unsigned int n, unsigned int *available) {
> > - return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
> > - RTE_RING_QUEUE_FIXED, r->cons.sync_type,
> > available);
> > + switch (r->cons.sync_type) {
> > + case RTE_RING_SYNC_MT:
> > + return rte_ring_mc_dequeue_bulk_elem(r, obj_table, esize, n,
> > + available);
> > + case RTE_RING_SYNC_ST:
> > + return rte_ring_sc_dequeue_bulk_elem(r, obj_table, esize, n,
> > + available);
> > +#ifdef ALLOW_EXPERIMENTAL_API
> > + case RTE_RING_SYNC_MT_RTS:
> > + return rte_ring_mc_rts_dequeue_bulk_elem(r, obj_table,
> > esize,
> > + n, available);
> > +#endif
> > + }
> > +
> > + /* valid ring should never reach this point */
> > + RTE_ASSERT(0);
> > + if (available != NULL)
> > + *available = 0;
> > + return 0;
> > }
> >
> > /**
> > @@ -901,8 +940,25 @@ static __rte_always_inline unsigned
> > rte_ring_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
> > unsigned int esize, unsigned int n, unsigned int *free_space) {
> > - return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
> > - RTE_RING_QUEUE_VARIABLE, r->prod.sync_type,
> > free_space);
> > + switch (r->prod.sync_type) {
> > + case RTE_RING_SYNC_MT:
> > + return rte_ring_mp_enqueue_burst_elem(r, obj_table, esize, n,
> > + free_space);
> > + case RTE_RING_SYNC_ST:
> > + return rte_ring_sp_enqueue_burst_elem(r, obj_table, esize, n,
> > + free_space);
> > +#ifdef ALLOW_EXPERIMENTAL_API
> > + case RTE_RING_SYNC_MT_RTS:
> > + return rte_ring_mp_rts_enqueue_burst_elem(r, obj_table,
> > esize,
> > + n, free_space);
> > +#endif
> > + }
> > +
> > + /* valid ring should never reach this point */
> > + RTE_ASSERT(0);
> > + if (free_space != NULL)
> > + *free_space = 0;
> > + return 0;
> > }
> >
> > /**
> > @@ -993,9 +1049,25 @@ static __rte_always_inline unsigned int
> > rte_ring_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
> > unsigned int esize, unsigned int n, unsigned int *available) {
> > - return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
> > - RTE_RING_QUEUE_VARIABLE,
> > - r->cons.sync_type, available);
> > + switch (r->cons.sync_type) {
> > + case RTE_RING_SYNC_MT:
> > + return rte_ring_mc_dequeue_burst_elem(r, obj_table, esize, n,
> > + available);
> > + case RTE_RING_SYNC_ST:
> > + return rte_ring_sc_dequeue_burst_elem(r, obj_table, esize, n,
> > + available);
> > +#ifdef ALLOW_EXPERIMENTAL_API
> > + case RTE_RING_SYNC_MT_RTS:
> > + return rte_ring_mc_rts_dequeue_burst_elem(r, obj_table,
> > esize,
> > + n, available);
> > +#endif
> > + }
> > +
> > + /* valid ring should never reach this point */
> > + RTE_ASSERT(0);
> > + if (available != NULL)
> > + *available = 0;
> > + return 0;
> > }
> >
> > #ifdef __cplusplus
> > diff --git a/lib/librte_ring/rte_ring_rts.h b/lib/librte_ring/rte_ring_rts.h new
> > file mode 100644 index 000000000..18404fe48
> > --- /dev/null
> > +++ b/lib/librte_ring/rte_ring_rts.h
> IMO, we should not provide these APIs.
You mean only _elem_ ones, as discussed above?
>
> > @@ -0,0 +1,316 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + *
> > + * Copyright (c) 2010-2017 Intel Corporation
> nit, the year should change to 2020? Look at others too.
ack, will do.
>
> > + * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
> > + * All rights reserved.
> > + * Derived from FreeBSD's bufring.h
> > + * Used as BSD-3 Licensed with permission from Kip Macy.
> > + */
> > +
> > +#ifndef _RTE_RING_RTS_H_
> > +#define _RTE_RING_RTS_H_
> > +
> > +/**
> > + * @file rte_ring_rts.h
> > + * @b EXPERIMENTAL: this API may change without prior notice
> > + * It is not recommended to include this file directly.
> > + * Please include <rte_ring.h> instead.
> > + *
> > + * Contains functions for Relaxed Tail Sync (RTS) ring mode.
> > + * The main idea remains the same as for our original MP/MC
> ^^^ the
> > +synchronization
> > + * mechanism.
> > + * The main difference is that tail value is increased not
> > + * by every thread that finished enqueue/dequeue,
> > + * but only by the last one doing enqueue/dequeue.
> should we say 'current last' or 'last thread at a given instance'?
>
> > + * That allows threads to skip spinning on tail value,
> > + * leaving actual tail value change to last thread in the update queue.
> nit, I understand what you mean by 'update queue' here. IMO, we should remove it as it might confuse some.
>
> > + * RTS requires 2 64-bit CAS for each enqueue(/dequeue) operation:
> > + * one for head update, second for tail update.
> > + * As a gain it allows thread to avoid spinning/waiting on tail value.
> > + * In comparision original MP/MC algorithm requires one 32-bit CAS
> > + * for head update and waiting/spinning on tail value.
> > + *
> > + * Brief outline:
> > + * - introduce refcnt for both head and tail.
> Suggesting using the same names as used in the structures.
>
> > + * - increment head.refcnt for each head.value update
> > + * - write head:value and head:refcnt atomically (64-bit CAS)
> > + * - move tail.value ahead only when tail.refcnt + 1 == head.refcnt
> May be add '(indicating that this is the last thread updating the tail)'
>
> > + * - increment tail.refcnt when each enqueue/dequeue op finishes
> May be add 'otherwise' at the beginning.
>
> > + * (no matter is tail:value going to change or not)
> nit ^^ if
> > + * - write tail.value and tail.recnt atomically (64-bit CAS)
> > + *
> > + * To avoid producer/consumer starvation:
> > + * - limit max allowed distance between head and tail value (HTD_MAX).
> > + * I.E. thread is allowed to proceed with changing head.value,
> > + * only when: head.value - tail.value <= HTD_MAX
> > + * HTD_MAX is an optional parameter.
> > + * With HTD_MAX == 0 we'll have fully serialized ring -
> > + * i.e. only one thread at a time will be able to enqueue/dequeue
> > + * to/from the ring.
> > + * With HTD_MAX >= ring.capacity - no limitation.
> > + * By default HTD_MAX == ring.capacity / 8.
> > + */
> > +
> > +#ifdef __cplusplus
> > +extern "C" {
> > +#endif
> > +
> > +#include <rte_ring_rts_generic.h>
> > +
> > +/**
> > + * @internal Enqueue several objects on the RTS ring.
> > + *
> > + * @param r
> > + * A pointer to the ring structure.
> > + * @param obj_table
> > + * A pointer to a table of void * pointers (objects).
> > + * @param n
> > + * The number of objects to add in the ring from the obj_table.
> > + * @param behavior
> > + * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
> > + * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from
> > ring
> > + * @param free_space
> > + * returns the amount of space after the enqueue operation has finished
> > + * @return
> > + * Actual number of objects enqueued.
> > + * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> > + */
> > +static __rte_always_inline unsigned int
> > +__rte_ring_do_rts_enqueue(struct rte_ring *r, void * const *obj_table,
> > + uint32_t n, enum rte_ring_queue_behavior behavior,
> > + uint32_t *free_space)
> > +{
> > + uint32_t free, head;
> > +
> > + n = __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
> > +
> > + if (n != 0) {
> > + ENQUEUE_PTRS(r, &r[1], head, obj_table, n, void *);
> > + __rte_ring_rts_update_tail(&r->rts_prod);
> > + }
> > +
> > + if (free_space != NULL)
> > + *free_space = free - n;
> > + return n;
> > +}
> > +
> > +/**
> > + * @internal Dequeue several objects from the RTS ring.
> > + *
> > + * @param r
> > + * A pointer to the ring structure.
> > + * @param obj_table
> > + * A pointer to a table of void * pointers (objects).
> > + * @param n
> > + * The number of objects to pull from the ring.
> > + * @param behavior
> > + * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
> > + * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from
> > ring
> > + * @param available
> > + * returns the number of remaining ring entries after the dequeue has
> > finished
> > + * @return
> > + * - Actual number of objects dequeued.
> > + * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> > + */
> > +static __rte_always_inline unsigned int
> > +__rte_ring_do_rts_dequeue(struct rte_ring *r, void **obj_table,
> > + uint32_t n, enum rte_ring_queue_behavior behavior,
> > + uint32_t *available)
> > +{
> > + uint32_t entries, head;
> > +
> > + n = __rte_ring_rts_move_cons_head(r, n, behavior, &head, &entries);
> > +
> > + if (n != 0) {
> > + DEQUEUE_PTRS(r, &r[1], head, obj_table, n, void *);
> > + __rte_ring_rts_update_tail(&r->rts_cons);
> > + }
> > +
> > + if (available != NULL)
> > + *available = entries - n;
> > + return n;
> > +}
> > +
> > +/**
> > + * Enqueue several objects on the RTS ring (multi-producers safe).
> > + *
> > + * @param r
> > + * A pointer to the ring structure.
> > + * @param obj_table
> > + * A pointer to a table of void * pointers (objects).
> > + * @param n
> > + * The number of objects to add in the ring from the obj_table.
> > + * @param free_space
> > + * if non-NULL, returns the amount of space in the ring after the
> > + * enqueue operation has finished.
> > + * @return
> > + * The number of objects enqueued, either 0 or n
> > + */
> > +__rte_experimental
> > +static __rte_always_inline unsigned int
> > +rte_ring_mp_rts_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
> > + unsigned int n, unsigned int *free_space) {
> > + return __rte_ring_do_rts_enqueue(r, obj_table, n,
> > RTE_RING_QUEUE_FIXED,
> > + free_space);
> > +}
> > +
> > +/**
> > + * Dequeue several objects from an RTS ring (multi-consumers safe).
> > + *
> > + * @param r
> > + * A pointer to the ring structure.
> > + * @param obj_table
> > + * A pointer to a table of void * pointers (objects) that will be filled.
> > + * @param n
> > + * The number of objects to dequeue from the ring to the obj_table.
> > + * @param available
> > + * If non-NULL, returns the number of remaining ring entries after the
> > + * dequeue has finished.
> > + * @return
> > + * The number of objects dequeued, either 0 or n
> > + */
> > +__rte_experimental
> > +static __rte_always_inline unsigned int
> > +rte_ring_mc_rts_dequeue_bulk(struct rte_ring *r, void **obj_table,
> > + unsigned int n, unsigned int *available) {
> > + return __rte_ring_do_rts_dequeue(r, obj_table, n,
> > RTE_RING_QUEUE_FIXED,
> > + available);
> > +}
> > +
> > +/**
> > + * Return producer max Head-Tail-Distance (HTD).
> > + *
> > + * @param r
> > + * A pointer to the ring structure.
> > + * @return
> > + * Producer HTD value, if producer is set in appropriate sync mode,
> > + * or UINT32_MAX otherwise.
> > + */
> > +__rte_experimental
> > +static inline uint32_t
> > +rte_ring_get_prod_htd_max(const struct rte_ring *r) {
> > + if (r->prod.sync_type == RTE_RING_SYNC_MT_RTS)
> > + return r->rts_prod.htd_max;
> > + return UINT32_MAX;
> > +}
> > +
> > +/**
> > + * Set producer max Head-Tail-Distance (HTD).
> > + * Note that producer has to use appropriate sync mode (RTS).
> > + *
> > + * @param r
> > + * A pointer to the ring structure.
> > + * @param v
> > + * new HTD value to setup.
> > + * @return
> > + * Zero on success, or negative error code otherwise.
> > + */
> > +__rte_experimental
> > +static inline int
> > +rte_ring_set_prod_htd_max(struct rte_ring *r, uint32_t v) {
> > + if (r->prod.sync_type != RTE_RING_SYNC_MT_RTS)
> > + return -ENOTSUP;
> > +
> > + r->rts_prod.htd_max = v;
> > + return 0;
> > +}
> > +
> > +/**
> > + * Return consumer max Head-Tail-Distance (HTD).
> > + *
> > + * @param r
> > + * A pointer to the ring structure.
> > + * @return
> > + * Consumer HTD value, if consumer is set in appropriate sync mode,
> > + * or UINT32_MAX otherwise.
> > + */
> > +__rte_experimental
> > +static inline uint32_t
> > +rte_ring_get_cons_htd_max(const struct rte_ring *r) {
> > + if (r->cons.sync_type == RTE_RING_SYNC_MT_RTS)
> > + return r->rts_cons.htd_max;
> > + return UINT32_MAX;
> > +}
> > +
> > +/**
> > + * Set consumer max Head-Tail-Distance (HTD).
> > + * Note that consumer has to use appropriate sync mode (RTS).
> > + *
> > + * @param r
> > + * A pointer to the ring structure.
> > + * @param v
> > + * new HTD value to setup.
> > + * @return
> > + * Zero on success, or negative error code otherwise.
> > + */
> > +__rte_experimental
> > +static inline int
> > +rte_ring_set_cons_htd_max(struct rte_ring *r, uint32_t v) {
> > + if (r->cons.sync_type != RTE_RING_SYNC_MT_RTS)
> > + return -ENOTSUP;
> > +
> > + r->rts_cons.htd_max = v;
> > + return 0;
> > +}
> > +
> > +/**
> > + * Enqueue several objects on the RTS ring (multi-producers safe).
> > + *
> > + * @param r
> > + * A pointer to the ring structure.
> > + * @param obj_table
> > + * A pointer to a table of void * pointers (objects).
> > + * @param n
> > + * The number of objects to add in the ring from the obj_table.
> > + * @param free_space
> > + * if non-NULL, returns the amount of space in the ring after the
> > + * enqueue operation has finished.
> > + * @return
> > + * - n: Actual number of objects enqueued.
> > + */
> > +__rte_experimental
> > +static __rte_always_inline unsigned
> > +rte_ring_mp_rts_enqueue_burst(struct rte_ring *r, void * const *obj_table,
> > + unsigned int n, unsigned int *free_space) {
> > + return __rte_ring_do_rts_enqueue(r, obj_table, n,
> > + RTE_RING_QUEUE_VARIABLE, free_space); }
> > +
> > +/**
> > + * Dequeue several objects from an RTS ring (multi-consumers safe).
> > + * When the requested objects are more than the available objects,
> > + * only dequeue the actual number of objects.
> > + *
> > + * @param r
> > + * A pointer to the ring structure.
> > + * @param obj_table
> > + * A pointer to a table of void * pointers (objects) that will be filled.
> > + * @param n
> > + * The number of objects to dequeue from the ring to the obj_table.
> > + * @param available
> > + * If non-NULL, returns the number of remaining ring entries after the
> > + * dequeue has finished.
> > + * @return
> > + * - n: Actual number of objects dequeued, 0 if ring is empty
> > + */
> > +__rte_experimental
> > +static __rte_always_inline unsigned
> > +rte_ring_mc_rts_dequeue_burst(struct rte_ring *r, void **obj_table,
> > + unsigned int n, unsigned int *available) {
> > + return __rte_ring_do_rts_dequeue(r, obj_table, n,
> > + RTE_RING_QUEUE_VARIABLE, available); }
> > +
> > +#ifdef __cplusplus
> > +}
> > +#endif
> > +
> > +#endif /* _RTE_RING_RTS_H_ */
> > diff --git a/lib/librte_ring/rte_ring_rts_elem.h
> > b/lib/librte_ring/rte_ring_rts_elem.h
> > new file mode 100644
> > index 000000000..71a331b23
> > --- /dev/null
> > +++ b/lib/librte_ring/rte_ring_rts_elem.h
> > @@ -0,0 +1,205 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + *
> > + * Copyright (c) 2010-2017 Intel Corporation
> > + * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
> > + * All rights reserved.
> > + * Derived from FreeBSD's bufring.h
> > + * Used as BSD-3 Licensed with permission from Kip Macy.
> > + */
> > +
> > +#ifndef _RTE_RING_RTS_ELEM_H_
> > +#define _RTE_RING_RTS_ELEM_H_
> > +
> > +/**
> > + * @file rte_ring_rts_elem.h
> > + * @b EXPERIMENTAL: this API may change without prior notice
> > + *
> > + * It is not recommended to include this file directly.
> > + * Please include <rte_ring_elem.h> instead.
> > + * Contains *ring_elem* functions for Relaxed Tail Sync (RTS) ring mode.
> > + * for more details please refer to <rte_ring_rts.h>.
> > + */
> > +
> > +#ifdef __cplusplus
> > +extern "C" {
> > +#endif
> > +
> > +#include <rte_ring_rts_generic.h>
> > +
> > +/**
> > + * @internal Enqueue several objects on the RTS ring.
> > + *
> > + * @param r
> > + * A pointer to the ring structure.
> > + * @param obj_table
> > + * A pointer to a table of void * pointers (objects).
> > + * @param n
> > + * The number of objects to add in the ring from the obj_table.
> > + * @param behavior
> > + * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
> > + * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from
> > ring
> > + * @param free_space
> > + * returns the amount of space after the enqueue operation has finished
> > + * @return
> > + * Actual number of objects enqueued.
> > + * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> > + */
> > +static __rte_always_inline unsigned int
> > +__rte_ring_do_rts_enqueue_elem(struct rte_ring *r, void * const *obj_table,
> obj_table should be of type 'const void * obj_table' (looks like copy paste error). Please check the other APIs below too.
>
> > + uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
> 'esize' is not documented in the comments above the function. You can copy the header from rte_ring_elem.h file. Please check other APIs
> as well.
Ack to both, will fix.
>
> > + uint32_t *free_space)
> > +{
> > + uint32_t free, head;
> > +
> > + n = __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
> > +
> > + if (n != 0) {
> > + __rte_ring_enqueue_elems(r, head, obj_table, esize, n);
> > + __rte_ring_rts_update_tail(&r->rts_prod);
> > + }
> > +
> > + if (free_space != NULL)
> > + *free_space = free - n;
> > + return n;
> > +}
> > +
> > +/**
> > + * @internal Dequeue several objects from the RTS ring.
> > + *
> > + * @param r
> > + * A pointer to the ring structure.
> > + * @param obj_table
> > + * A pointer to a table of void * pointers (objects).
> > + * @param n
> > + * The number of objects to pull from the ring.
> > + * @param behavior
> > + * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
> > + * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from
> > ring
> > + * @param available
> > + * returns the number of remaining ring entries after the dequeue has
> > finished
> > + * @return
> > + * - Actual number of objects dequeued.
> > + * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> > + */
> > +static __rte_always_inline unsigned int
> > +__rte_ring_do_rts_dequeue_elem(struct rte_ring *r, void **obj_table,
> > + uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
> > + uint32_t *available)
> > +{
> > + uint32_t entries, head;
> > +
> > + n = __rte_ring_rts_move_cons_head(r, n, behavior, &head, &entries);
> > +
> > + if (n != 0) {
> > + __rte_ring_dequeue_elems(r, head, obj_table, esize, n);
> > + __rte_ring_rts_update_tail(&r->rts_cons);
> > + }
> > +
> > + if (available != NULL)
> > + *available = entries - n;
> > + return n;
> > +}
> > +
> > +/**
> > + * Enqueue several objects on the RTS ring (multi-producers safe).
> > + *
> > + * @param r
> > + * A pointer to the ring structure.
> > + * @param obj_table
> > + * A pointer to a table of void * pointers (objects).
> > + * @param n
> > + * The number of objects to add in the ring from the obj_table.
> > + * @param free_space
> > + * if non-NULL, returns the amount of space in the ring after the
> > + * enqueue operation has finished.
> > + * @return
> > + * The number of objects enqueued, either 0 or n
> > + */
> > +__rte_experimental
> > +static __rte_always_inline unsigned int
> > +rte_ring_mp_rts_enqueue_bulk_elem(struct rte_ring *r, void * const
> > *obj_table,
> > + unsigned int esize, unsigned int n, unsigned int *free_space) {
> > + return __rte_ring_do_rts_enqueue_elem(r, obj_table, esize, n,
> > + RTE_RING_QUEUE_FIXED, free_space);
> > +}
> > +
> > +/**
> > + * Dequeue several objects from an RTS ring (multi-consumers safe).
> > + *
> > + * @param r
> > + * A pointer to the ring structure.
> > + * @param obj_table
> > + * A pointer to a table of void * pointers (objects) that will be filled.
> > + * @param n
> > + * The number of objects to dequeue from the ring to the obj_table.
> > + * @param available
> > + * If non-NULL, returns the number of remaining ring entries after the
> > + * dequeue has finished.
> > + * @return
> > + * The number of objects dequeued, either 0 or n
> > + */
> > +__rte_experimental
> > +static __rte_always_inline unsigned int
> > +rte_ring_mc_rts_dequeue_bulk_elem(struct rte_ring *r, void **obj_table,
> > + unsigned int esize, unsigned int n, unsigned int *available) {
> > + return __rte_ring_do_rts_dequeue_elem(r, obj_table, esize, n,
> > + RTE_RING_QUEUE_FIXED, available);
> > +}
> > +
> > +/**
> > + * Enqueue several objects on the RTS ring (multi-producers safe).
> > + *
> > + * @param r
> > + * A pointer to the ring structure.
> > + * @param obj_table
> > + * A pointer to a table of void * pointers (objects).
> > + * @param n
> > + * The number of objects to add in the ring from the obj_table.
> > + * @param free_space
> > + * if non-NULL, returns the amount of space in the ring after the
> > + * enqueue operation has finished.
> > + * @return
> > + * - n: Actual number of objects enqueued.
> > + */
> > +__rte_experimental
> > +static __rte_always_inline unsigned
> > +rte_ring_mp_rts_enqueue_burst_elem(struct rte_ring *r, void * const
> > *obj_table,
> > + unsigned int esize, unsigned int n, unsigned int *free_space) {
> > + return __rte_ring_do_rts_enqueue_elem(r, obj_table, esize, n,
> > + RTE_RING_QUEUE_VARIABLE, free_space); }
> > +
> > +/**
> > + * Dequeue several objects from an RTS ring (multi-consumers safe).
> > + * When the requested objects are more than the available objects,
> > + * only dequeue the actual number of objects.
> > + *
> > + * @param r
> > + * A pointer to the ring structure.
> > + * @param obj_table
> > + * A pointer to a table of void * pointers (objects) that will be filled.
> > + * @param n
> > + * The number of objects to dequeue from the ring to the obj_table.
> > + * @param available
> > + * If non-NULL, returns the number of remaining ring entries after the
> > + * dequeue has finished.
> > + * @return
> > + * - n: Actual number of objects dequeued, 0 if ring is empty
> > + */
> > +__rte_experimental
> > +static __rte_always_inline unsigned
> > +rte_ring_mc_rts_dequeue_burst_elem(struct rte_ring *r, void **obj_table,
> > + unsigned int esize, unsigned int n, unsigned int *available) {
> > + return __rte_ring_do_rts_dequeue_elem(r, obj_table, esize, n,
> > + RTE_RING_QUEUE_VARIABLE, available); }
> > +
> > +#ifdef __cplusplus
> > +}
> > +#endif
> > +
> > +#endif /* _RTE_RING_RTS_ELEM_H_ */
> > diff --git a/lib/librte_ring/rte_ring_rts_generic.h
> > b/lib/librte_ring/rte_ring_rts_generic.h
> > new file mode 100644
> > index 000000000..f88460d47
> > --- /dev/null
> > +++ b/lib/librte_ring/rte_ring_rts_generic.h
> I do not know the benefit to providing the generic version. Do you know why this was done in the legacy APIs?
I think at first we had generic API only, then later C11 was added.
As I remember, C11 one on IA was measured as a bit slower then generic,
so it was decided to keep both.
> If there is no performance difference between generic and C11 versions, should we just skip the generic version?
> The oldest compiler in CI are GCC 4.8.5 and Clang 3.4.2 and C11 built-ins are supported earlier than these compiler versions.
> I feel the code is growing exponentially in rte_ring library and we should try to cut non-value-ass code/APIs aggressively.
I'll check is there perf difference for RTS and HTS between generic and C11 versions on IA.
Meanwhile please have a proper look at C11 implementation, I am not that familiar with C11 atomics yet.
If there would be no problems with it and no noticeable diff in performance -
I am ok to have for RTS/HTS modes C11 version only.
>
> > @@ -0,0 +1,210 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + *
> > + * Copyright (c) 2010-2017 Intel Corporation
> > + * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
> > + * All rights reserved.
> > + * Derived from FreeBSD's bufring.h
> > + * Used as BSD-3 Licensed with permission from Kip Macy.
> > + */
> > +
> > +#ifndef _RTE_RING_RTS_GENERIC_H_
> > +#define _RTE_RING_RTS_GENERIC_H_
> > +
> > +/**
> > + * @file rte_ring_rts_generic.h
> > + * It is not recommended to include this file directly,
> > + * include <rte_ring.h> instead.
> > + * Contains internal helper functions for Relaxed Tail Sync (RTS) ring mode.
> > + * For more information please refer to <rte_ring_rts.h>.
> > + */
> > +
> > +/**
> > + * @internal This function updates tail values.
> > + */
> > +static __rte_always_inline void
> > +__rte_ring_rts_update_tail(struct rte_ring_rts_headtail *ht) {
> > + union rte_ring_ht_poscnt h, ot, nt;
> > +
> > + /*
> > + * If there are other enqueues/dequeues in progress that
> > + * might preceded us, then don't update tail with new value.
> > + */
> > +
> > + do {
> > + ot.raw = ht->tail.raw;
> > + rte_smp_rmb();
> > +
> > + /* on 32-bit systems we have to do atomic read here */
> > + h.raw = rte_atomic64_read((rte_atomic64_t *)
> > + (uintptr_t)&ht->head.raw);
> > +
> > + nt.raw = ot.raw;
> > + if (++nt.val.cnt == h.val.cnt)
> > + nt.val.pos = h.val.pos;
> > +
> > + } while (rte_atomic64_cmpset(&ht->tail.raw, ot.raw, nt.raw) == 0); }
> > +
> > +/**
> > + * @internal This function waits till head/tail distance wouldn't
> > + * exceed pre-defined max value.
> > + */
> > +static __rte_always_inline void
> > +__rte_ring_rts_head_wait(const struct rte_ring_rts_headtail *ht,
> > + union rte_ring_ht_poscnt *h)
> > +{
> > + uint32_t max;
> > +
> > + max = ht->htd_max;
> > + h->raw = ht->head.raw;
> > + rte_smp_rmb();
> > +
> > + while (h->val.pos - ht->tail.val.pos > max) {
> > + rte_pause();
> > + h->raw = ht->head.raw;
> > + rte_smp_rmb();
> > + }
> > +}
> > +
> > +/**
> > + * @internal This function updates the producer head for enqueue.
> > + *
> > + * @param r
> > + * A pointer to the ring structure
> > + * @param is_sp
> > + * Indicates whether multi-producer path is needed or not
> > + * @param n
> > + * The number of elements we will want to enqueue, i.e. how far should the
> > + * head be moved
> > + * @param behavior
> > + * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
> > + * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from
> > ring
> > + * @param old_head
> > + * Returns head value as it was before the move, i.e. where enqueue starts
> > + * @param new_head
> > + * Returns the current/new head value i.e. where enqueue finishes
> > + * @param free_entries
> > + * Returns the amount of free space in the ring BEFORE head was moved
> > + * @return
> > + * Actual number of objects enqueued.
> > + * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> > + */
> > +static __rte_always_inline uint32_t
> > +__rte_ring_rts_move_prod_head(struct rte_ring *r, uint32_t num,
> > + enum rte_ring_queue_behavior behavior, uint32_t *old_head,
> > + uint32_t *free_entries)
> > +{
> > + uint32_t n;
> > + union rte_ring_ht_poscnt nh, oh;
> > +
> > + const uint32_t capacity = r->capacity;
> > +
> > + do {
> > + /* Reset n to the initial burst count */
> > + n = num;
> > +
> > + /* read prod head (may spin on prod tail) */
> > + __rte_ring_rts_head_wait(&r->rts_prod, &oh);
> > +
> > + /* add rmb barrier to avoid load/load reorder in weak
> > + * memory model. It is noop on x86
> > + */
> > + rte_smp_rmb();
> > +
> > + /*
> > + * The subtraction is done between two unsigned 32bits value
> > + * (the result is always modulo 32 bits even if we have
> > + * *old_head > cons_tail). So 'free_entries' is always between
> > 0
> > + * and capacity (which is < size).
> > + */
> > + *free_entries = capacity + r->cons.tail - oh.val.pos;
> > +
> > + /* check that we have enough room in ring */
> > + if (unlikely(n > *free_entries))
> > + n = (behavior == RTE_RING_QUEUE_FIXED) ?
> > + 0 : *free_entries;
> > +
> > + if (n == 0)
> > + break;
> > +
> > + nh.val.pos = oh.val.pos + n;
> > + nh.val.cnt = oh.val.cnt + 1;
> > +
> > + } while (rte_atomic64_cmpset(&r->rts_prod.head.raw,
> > + oh.raw, nh.raw) == 0);
> > +
> > + *old_head = oh.val.pos;
> > + return n;
> > +}
> > +
> > +/**
> > + * @internal This function updates the consumer head for dequeue
> > + *
> > + * @param r
> > + * A pointer to the ring structure
> > + * @param is_sc
> > + * Indicates whether multi-consumer path is needed or not
> > + * @param n
> > + * The number of elements we will want to enqueue, i.e. how far should the
> > + * head be moved
> > + * @param behavior
> > + * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
> > + * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from
> > ring
> > + * @param old_head
> > + * Returns head value as it was before the move, i.e. where dequeue starts
> > + * @param new_head
> > + * Returns the current/new head value i.e. where dequeue finishes
> > + * @param entries
> > + * Returns the number of entries in the ring BEFORE head was moved
> > + * @return
> > + * - Actual number of objects dequeued.
> > + * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> > + */
> > +static __rte_always_inline unsigned int
> > +__rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num,
> > + enum rte_ring_queue_behavior behavior, uint32_t *old_head,
> > + uint32_t *entries)
> > +{
> > + uint32_t n;
> > + union rte_ring_ht_poscnt nh, oh;
> > +
> > + /* move cons.head atomically */
> > + do {
> > + /* Restore n as it may change every loop */
> > + n = num;
> > +
> > + /* read cons head (may spin on cons tail) */
> > + __rte_ring_rts_head_wait(&r->rts_cons, &oh);
> > +
> > +
> > + /* add rmb barrier to avoid load/load reorder in weak
> > + * memory model. It is noop on x86
> > + */
> > + rte_smp_rmb();
> > +
> > + /* The subtraction is done between two unsigned 32bits value
> > + * (the result is always modulo 32 bits even if we have
> > + * cons_head > prod_tail). So 'entries' is always between 0
> > + * and size(ring)-1.
> > + */
> > + *entries = r->prod.tail - oh.val.pos;
> > +
> > + /* Set the actual entries for dequeue */
> > + if (n > *entries)
> > + n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 :
> > *entries;
> > +
> > + if (unlikely(n == 0))
> > + break;
> > +
> > + nh.val.pos = oh.val.pos + n;
> > + nh.val.cnt = oh.val.cnt + 1;
> > +
> > + } while (rte_atomic64_cmpset(&r->rts_cons.head.raw,
> > + oh.raw, nh.raw) == 0);
> > +
> > + *old_head = oh.val.pos;
> > + return n;
> > +}
> > +
> > +#endif /* _RTE_RING_RTS_GENERIC_H_ */
> > --
> > 2.17.1
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCHv2] Remove validate-abi.sh from tree
2020-04-09 10:59 7% ` Thomas Monjalon
2020-04-09 13:02 4% ` Neil Horman
@ 2020-04-09 14:52 9% ` Ray Kinsella
2020-04-09 15:18 8% ` Thomas Monjalon
1 sibling, 1 reply; 200+ results
From: Ray Kinsella @ 2020-04-09 14:52 UTC (permalink / raw)
To: Thomas Monjalon, Bruce Richardson, Neil Horman; +Cc: dev, david.marchand
On 09/04/2020 11:59, Thomas Monjalon wrote:
> 09/04/2020 12:45, Ray Kinsella:
>> On 09/04/2020 11:43, Bruce Richardson wrote:
>>> On Thu, Apr 09, 2020 at 06:39:54AM -0400, Neil Horman wrote:
>>>> On Thu, Apr 09, 2020 at 08:57:34AM +0100, Ray Kinsella wrote:
>>>>> On 08/04/2020 20:56, Neil Horman wrote:
>>>>>> +The syntax of the ``check-abi.sh`` utility is::
>>>>>> +
>>>>>> + ./devtools/check-abi.sh <refdir> <newdir>
>>>>>
>>>>> (from v1 feedback)
>>>>> Could we simplify this all greatly, by telling people to use the meson/ninja build,
>>>>> so they get this checking out of the box, without all the headache below?
>>>>>
>>>> I think bruce noted that was never merged, correct?
>>>>
>>> Yep, correct. :-(
>>
>> apologies, was there a reason?
>
> Because build tool job is building, not checking.
> It would be wrong to make (slow) checks mandatory in all builds.
>
> The need is to enforce checking ABI.
> The result is already published by Travis in patchwork and in an
> email to the author I believe.
> Not checking email and patchwork is not a good excuse.
>
> Patchwork must be a mandatory read for everybody for all checks
> in general. Let's not give up on general CI workflow.
>
Thomas
You are trying to solve two problems at once; CI tooling and ABI.
Let's try to solve one at a time.
1. The ABI check, will make the build _marginally_ slower.
You _should_ only need to rebuild the changes between A and B.
2. The meson/ninja are an order of magnitude faster than GNU Make.
We can afford this check.
3. If we want to lessen the ABI burden and send the correct message.
It should be a build blocker, contributors need to hear the message loud and clear.
Most important people _consuming_ DPDK will never see this message.
Only people _changing_ the ABI will see it - the people we want to hear the message loud and clear.
Ray K
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCHv2] Remove validate-abi.sh from tree
2020-04-09 14:52 9% ` Ray Kinsella
@ 2020-04-09 15:18 8% ` Thomas Monjalon
2020-04-09 16:29 4% ` Ray Kinsella
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-04-09 15:18 UTC (permalink / raw)
To: Bruce Richardson, Neil Horman, Ray Kinsella; +Cc: dev, david.marchand
09/04/2020 16:52, Ray Kinsella:
> On 09/04/2020 11:59, Thomas Monjalon wrote:
> > 09/04/2020 12:45, Ray Kinsella:
> >> On 09/04/2020 11:43, Bruce Richardson wrote:
> >>> On Thu, Apr 09, 2020 at 06:39:54AM -0400, Neil Horman wrote:
> >>>> On Thu, Apr 09, 2020 at 08:57:34AM +0100, Ray Kinsella wrote:
> >>>>> On 08/04/2020 20:56, Neil Horman wrote:
> >>>>>> +The syntax of the ``check-abi.sh`` utility is::
> >>>>>> +
> >>>>>> + ./devtools/check-abi.sh <refdir> <newdir>
> >>>>>
> >>>>> (from v1 feedback)
> >>>>> Could we simplify this all greatly, by telling people to use the meson/ninja build,
> >>>>> so they get this checking out of the box, without all the headache below?
> >>>>>
> >>>> I think bruce noted that was never merged, correct?
> >>>>
> >>> Yep, correct. :-(
> >>
> >> apologies, was there a reason?
> >
> > Because build tool job is building, not checking.
> > It would be wrong to make (slow) checks mandatory in all builds.
> >
> > The need is to enforce checking ABI.
> > The result is already published by Travis in patchwork and in an
> > email to the author I believe.
> > Not checking email and patchwork is not a good excuse.
> >
> > Patchwork must be a mandatory read for everybody for all checks
> > in general. Let's not give up on general CI workflow.
> >
>
> Thomas
>
> You are trying to solve two problems at once; CI tooling and ABI.
> Let's try to solve one at a time.
No, you want to mix two problems in a single tool :-)
> 1. The ABI check, will make the build _marginally_ slower.
> You _should_ only need to rebuild the changes between A and B.
Not so marginal.
A re-build takes less than a second. A mandatory check takes 10 secs
on my machine.
> 2. The meson/ninja are an order of magnitude faster than GNU Make.
> We can afford this check.
I am doing such build 10 times (each target) per patch.
But that's not the main issue.
> 3. If we want to lessen the ABI burden and send the correct message.
> It should be a build blocker, contributors need to hear the message loud and clear.
The developer needs to get or build/save the ABI reference.
Making such ABI reference for each target is not so obvious:
- all symbols must be enabled (dependencies)
- some fixes may be needed for some compilers
> Most important people _consuming_ DPDK will never see this message.
> Only people _changing_ the ABI will see it - the people we want to hear the message loud and clear.
No, they will have issue in DPDK compilation if something in the check
goes wrong. We should not bother end users with internal checks.
The message is
a) run the check by
1) setting DPDK_ABI_REF_VERSION on command line or in devel.config file
2) running devtools/test-build.sh or devtools/test-meson-builds.sh
b) check what Travis is reporting in
- email to you
- patchwork reports
I think Travis report is convenient to use.
The local check is integrated in build scripts
but cannot be run by default because of the reasons above.
^ permalink raw reply [relevance 8%]
* Re: [dpdk-dev] [dpdk-stable] [PATCH v2] mbuf: replace zero-length marker with unnamed union
2020-04-09 10:49 3% ` Thomas Monjalon
@ 2020-04-09 16:09 0% ` Ray Kinsella
2020-04-11 2:50 0% ` Gavin Hu
1 sibling, 0 replies; 200+ results
From: Ray Kinsella @ 2020-04-09 16:09 UTC (permalink / raw)
To: Thomas Monjalon, Gavin Hu
Cc: David Marchand, Kevin Traynor, Bruce Richardson,
Morten Brørup, Ferruh Yigit, dev, nd, jerinj,
Honnappa Nagarahalli, Ruifeng Wang, Phil Yang, Joyce Kong,
stable, Olivier MATZ, Konstantin Ananyev, Andrew Rybchenko
On 09/04/2020 11:49, Thomas Monjalon wrote:
> 09/04/2020 11:48, Gavin Hu:
>> From: David Marchand <david.marchand@redhat.com>
>>> On Wed, Apr 8, 2020 at 5:05 PM Gavin Hu <Gavin.Hu@arm.com> wrote:
>>>> From: Kevin Traynor <ktraynor@redhat.com>
>>>>> Hi Gavin, I lost track if v2 is still a candidate for merge. fwiw, it
>>>>> compiles without giving the zero-length-bounds warning on my system.
>>>>>
>>>>> Kevin.
>>>>
>>>> Yes, this path alone is a candidate for merge.
>>>
>>> This patch is not mergeable, it would trigger failures in the ABI checks.
>>
>> Isn't it a false failure? If yes, is it ignorable?
>>
>>> You can see in patchwork that the robot reported a warning in Travis.
>>> http://mails.dpdk.org/archives/test-report/2020-March/119919.html
>>> https://travis-ci.com/github/ovsrobot/dpdk/jobs/295652710#L4476
>>>
>>>
>>> I opened a bz to libabigail.
>>> https://sourceware.org/bugzilla/show_bug.cgi?id=25661
>>>
>>>
>>> Either a different solution is found, or your patch will have to deal
>>> with this issue (libabigail fix won't be ready soon afaik) and waive
>>> this.
>>
>> Maybe we come back to 'disable the warning', before the libabigail fix ready? or alternatively ignore this ABI false failure, if it is.
>> I do not have ideas of what otherwise the options are.
>
> Gavin,
> I did not check this case.
> But in general, we do not skip checks, except some checkpatch ones.
> The policy with ABI checks is "NEVER SKIP".
> We prefer postponing patches, waiting for someone to fix tooling.
In this case Dave Marchand has more than adequately shown the issue is because of a libabigail failure.
and has raised an appropriate BZ in the right place, much kudos.
My read of this, is that libabigail fix, will be complicated.
> There is a lack of motivation currently for general concerns.
> We need to avoid being "write-only" contributors.
> So two things need to be done:
> 1/ improve tooling where it needs
> 2/ review patches from others
> I published a review list recently:
> http://mails.dpdk.org/archives/announce/2020-April/000315.html
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCHv2] Remove validate-abi.sh from tree
2020-04-09 15:18 8% ` Thomas Monjalon
@ 2020-04-09 16:29 4% ` Ray Kinsella
2020-04-09 16:51 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Ray Kinsella @ 2020-04-09 16:29 UTC (permalink / raw)
To: Thomas Monjalon, Bruce Richardson, Neil Horman; +Cc: dev, david.marchand
On 09/04/2020 16:18, Thomas Monjalon wrote:
> 09/04/2020 16:52, Ray Kinsella:
>> On 09/04/2020 11:59, Thomas Monjalon wrote:
>>> 09/04/2020 12:45, Ray Kinsella:
>>>> On 09/04/2020 11:43, Bruce Richardson wrote:
>>>>> On Thu, Apr 09, 2020 at 06:39:54AM -0400, Neil Horman wrote:
>>>>>> On Thu, Apr 09, 2020 at 08:57:34AM +0100, Ray Kinsella wrote:
>>>>>>> On 08/04/2020 20:56, Neil Horman wrote:
>>>>>>>> +The syntax of the ``check-abi.sh`` utility is::
>>>>>>>> +
>>>>>>>> + ./devtools/check-abi.sh <refdir> <newdir>
>>>>>>>
>>>>>>> (from v1 feedback)
>>>>>>> Could we simplify this all greatly, by telling people to use the meson/ninja build,
>>>>>>> so they get this checking out of the box, without all the headache below?
>>>>>>>
>>>>>> I think bruce noted that was never merged, correct?
>>>>>>
>>>>> Yep, correct. :-(
>>>>
>>>> apologies, was there a reason?
>>>
>>> Because build tool job is building, not checking.
>>> It would be wrong to make (slow) checks mandatory in all builds.
>>>
>>> The need is to enforce checking ABI.
>>> The result is already published by Travis in patchwork and in an
>>> email to the author I believe.
>>> Not checking email and patchwork is not a good excuse.
>>>
>>> Patchwork must be a mandatory read for everybody for all checks
>>> in general. Let's not give up on general CI workflow.
>>>
>>
>> Thomas
>>
>> You are trying to solve two problems at once; CI tooling and ABI.
>> Let's try to solve one at a time.
>
> No, you want to mix two problems in a single tool :-)
>
>
>> 1. The ABI check, will make the build _marginally_ slower.
>> You _should_ only need to rebuild the changes between A and B.
>
> Not so marginal.
> A re-build takes less than a second. A mandatory check takes 10 secs
> on my machine.
>
>
>> 2. The meson/ninja are an order of magnitude faster than GNU Make.
>> We can afford this check.
>
> I am doing such build 10 times (each target) per patch.
> But that's not the main issue.
>
>
>> 3. If we want to lessen the ABI burden and send the correct message.
>> It should be a build blocker, contributors need to hear the message loud and clear.
>
> The developer needs to get or build/save the ABI reference.
> Making such ABI reference for each target is not so obvious:
> - all symbols must be enabled (dependencies)
> - some fixes may be needed for some compilers
>
>
>> Most important people _consuming_ DPDK will never see this message.
>> Only people _changing_ the ABI will see it - the people we want to hear the message loud and clear.
>
> No, they will have issue in DPDK compilation if something in the check
> goes wrong. We should not bother end users with internal checks.
>
>
> The message is
> a) run the check by
> 1) setting DPDK_ABI_REF_VERSION on command line or in devel.config file
> 2) running devtools/test-build.sh or devtools/test-meson-builds.sh
> b) check what Travis is reporting in
> - email to you
> - patchwork reports
>
> I think Travis report is convenient to use.
> The local check is integrated in build scripts
> but cannot be run by default because of the reasons above.
>
>
Thomas the reality on this is that people have a tendency to filter this messages into an email folder
and don't always see them.
My 2c is that this will always be a struggle unless we find a way to make it un-ignore-able.
Hence my build-wiring suggestion.
Ray K
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCHv2] Remove validate-abi.sh from tree
2020-04-09 16:29 4% ` Ray Kinsella
@ 2020-04-09 16:51 4% ` Thomas Monjalon
2020-04-10 6:26 4% ` Ray Kinsella
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-04-09 16:51 UTC (permalink / raw)
To: Bruce Richardson, Neil Horman, Ray Kinsella; +Cc: dev, david.marchand
09/04/2020 18:29, Ray Kinsella:
> On 09/04/2020 16:18, Thomas Monjalon wrote:
> > 09/04/2020 16:52, Ray Kinsella:
> >> On 09/04/2020 11:59, Thomas Monjalon wrote:
> >>> 09/04/2020 12:45, Ray Kinsella:
> >>>> On 09/04/2020 11:43, Bruce Richardson wrote:
> >>>>> On Thu, Apr 09, 2020 at 06:39:54AM -0400, Neil Horman wrote:
> >>>>>> On Thu, Apr 09, 2020 at 08:57:34AM +0100, Ray Kinsella wrote:
> >>>>>>> On 08/04/2020 20:56, Neil Horman wrote:
> >>>>>>>> +The syntax of the ``check-abi.sh`` utility is::
> >>>>>>>> +
> >>>>>>>> + ./devtools/check-abi.sh <refdir> <newdir>
> >>>>>>>
> >>>>>>> (from v1 feedback)
> >>>>>>> Could we simplify this all greatly, by telling people to use the meson/ninja build,
> >>>>>>> so they get this checking out of the box, without all the headache below?
> >>>>>>>
> >>>>>> I think bruce noted that was never merged, correct?
> >>>>>>
> >>>>> Yep, correct. :-(
> >>>>
> >>>> apologies, was there a reason?
> >>>
> >>> Because build tool job is building, not checking.
> >>> It would be wrong to make (slow) checks mandatory in all builds.
> >>>
> >>> The need is to enforce checking ABI.
> >>> The result is already published by Travis in patchwork and in an
> >>> email to the author I believe.
> >>> Not checking email and patchwork is not a good excuse.
> >>>
> >>> Patchwork must be a mandatory read for everybody for all checks
> >>> in general. Let's not give up on general CI workflow.
> >>>
> >>
> >> Thomas
> >>
> >> You are trying to solve two problems at once; CI tooling and ABI.
> >> Let's try to solve one at a time.
> >
> > No, you want to mix two problems in a single tool :-)
> >
> >
> >> 1. The ABI check, will make the build _marginally_ slower.
> >> You _should_ only need to rebuild the changes between A and B.
> >
> > Not so marginal.
> > A re-build takes less than a second. A mandatory check takes 10 secs
> > on my machine.
> >
> >
> >> 2. The meson/ninja are an order of magnitude faster than GNU Make.
> >> We can afford this check.
> >
> > I am doing such build 10 times (each target) per patch.
> > But that's not the main issue.
> >
> >
> >> 3. If we want to lessen the ABI burden and send the correct message.
> >> It should be a build blocker, contributors need to hear the message loud and clear.
> >
> > The developer needs to get or build/save the ABI reference.
> > Making such ABI reference for each target is not so obvious:
> > - all symbols must be enabled (dependencies)
> > - some fixes may be needed for some compilers
> >
> >
> >> Most important people _consuming_ DPDK will never see this message.
> >> Only people _changing_ the ABI will see it - the people we want to hear the message loud and clear.
> >
> > No, they will have issue in DPDK compilation if something in the check
> > goes wrong. We should not bother end users with internal checks.
> >
> >
> > The message is
> > a) run the check by
> > 1) setting DPDK_ABI_REF_VERSION on command line or in devel.config file
> > 2) running devtools/test-build.sh or devtools/test-meson-builds.sh
> > b) check what Travis is reporting in
> > - email to you
> > - patchwork reports
> >
> > I think Travis report is convenient to use.
> > The local check is integrated in build scripts
> > but cannot be run by default because of the reasons above.
>
> Thomas the reality on this is that people have a tendency to filter
> this messages into an email folder and don't always see them.
>
> My 2c is that this will always be a struggle unless we find a way
> to make it un-ignore-able.
> Hence my build-wiring suggestion.
My other concern is that we will have the same issue with all checks
done in a CI.
I think the right approach is to enforce people checking CI results.
They will be used to check CI in patchwork because the patches will
be blocked.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCHv2] Remove validate-abi.sh from tree
2020-04-09 16:51 4% ` Thomas Monjalon
@ 2020-04-10 6:26 4% ` Ray Kinsella
2020-04-10 7:57 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Ray Kinsella @ 2020-04-10 6:26 UTC (permalink / raw)
To: Thomas Monjalon, Bruce Richardson, Neil Horman; +Cc: dev, david.marchand
On 09/04/2020 17:51, Thomas Monjalon wrote:
> 09/04/2020 18:29, Ray Kinsella:
>> On 09/04/2020 16:18, Thomas Monjalon wrote:
>>> 09/04/2020 16:52, Ray Kinsella:
>>>> On 09/04/2020 11:59, Thomas Monjalon wrote:
>>>>> 09/04/2020 12:45, Ray Kinsella:
>>>>>> On 09/04/2020 11:43, Bruce Richardson wrote:
>>>>>>> On Thu, Apr 09, 2020 at 06:39:54AM -0400, Neil Horman wrote:
>>>>>>>> On Thu, Apr 09, 2020 at 08:57:34AM +0100, Ray Kinsella wrote:
>>>>>>>>> On 08/04/2020 20:56, Neil Horman wrote:
>>>>>>>>>> +The syntax of the ``check-abi.sh`` utility is::
>>>>>>>>>> +
>>>>>>>>>> + ./devtools/check-abi.sh <refdir> <newdir>
>>>>>>>>>
>>>>>>>>> (from v1 feedback)
>>>>>>>>> Could we simplify this all greatly, by telling people to use the meson/ninja build,
>>>>>>>>> so they get this checking out of the box, without all the headache below?
>>>>>>>>>
>>>>>>>> I think bruce noted that was never merged, correct?
>>>>>>>>
>>>>>>> Yep, correct. :-(
>>>>>>
>>>>>> apologies, was there a reason?
>>>>>
>>>>> Because build tool job is building, not checking.
>>>>> It would be wrong to make (slow) checks mandatory in all builds.
>>>>>
>>>>> The need is to enforce checking ABI.
>>>>> The result is already published by Travis in patchwork and in an
>>>>> email to the author I believe.
>>>>> Not checking email and patchwork is not a good excuse.
>>>>>
>>>>> Patchwork must be a mandatory read for everybody for all checks
>>>>> in general. Let's not give up on general CI workflow.
>>>>>
>>>>
>>>> Thomas
>>>>
>>>> You are trying to solve two problems at once; CI tooling and ABI.
>>>> Let's try to solve one at a time.
>>>
>>> No, you want to mix two problems in a single tool :-)
>>>
>>>
>>>> 1. The ABI check, will make the build _marginally_ slower.
>>>> You _should_ only need to rebuild the changes between A and B.
>>>
>>> Not so marginal.
>>> A re-build takes less than a second. A mandatory check takes 10 secs
>>> on my machine.
>>>
>>>
>>>> 2. The meson/ninja are an order of magnitude faster than GNU Make.
>>>> We can afford this check.
>>>
>>> I am doing such build 10 times (each target) per patch.
>>> But that's not the main issue.
>>>
>>>
>>>> 3. If we want to lessen the ABI burden and send the correct message.
>>>> It should be a build blocker, contributors need to hear the message loud and clear.
>>>
>>> The developer needs to get or build/save the ABI reference.
>>> Making such ABI reference for each target is not so obvious:
>>> - all symbols must be enabled (dependencies)
>>> - some fixes may be needed for some compilers
>>>
>>>
>>>> Most important people _consuming_ DPDK will never see this message.
>>>> Only people _changing_ the ABI will see it - the people we want to hear the message loud and clear.
>>>
>>> No, they will have issue in DPDK compilation if something in the check
>>> goes wrong. We should not bother end users with internal checks.
>>>
>>>
>>> The message is
>>> a) run the check by
>>> 1) setting DPDK_ABI_REF_VERSION on command line or in devel.config file
>>> 2) running devtools/test-build.sh or devtools/test-meson-builds.sh
>>> b) check what Travis is reporting in
>>> - email to you
>>> - patchwork reports
>>>
>>> I think Travis report is convenient to use.
>>> The local check is integrated in build scripts
>>> but cannot be run by default because of the reasons above.
>>
>> Thomas the reality on this is that people have a tendency to filter
>> this messages into an email folder and don't always see them.
>>
>> My 2c is that this will always be a struggle unless we find a way
>> to make it un-ignore-able.
>> Hence my build-wiring suggestion.
>
> My other concern is that we will have the same issue with all checks
> done in a CI.
> I think the right approach is to enforce people checking CI results.
Beyond asking maintainers to check, how would we enforce?
> They will be used to check CI in patchwork because the patches will
> be blocked.
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCHv2] Remove validate-abi.sh from tree
2020-04-10 6:26 4% ` Ray Kinsella
@ 2020-04-10 7:57 4% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-04-10 7:57 UTC (permalink / raw)
To: Bruce Richardson, Neil Horman, Ray Kinsella; +Cc: dev, david.marchand
10/04/2020 08:26, Ray Kinsella:
> On 09/04/2020 17:51, Thomas Monjalon wrote:
> > 09/04/2020 18:29, Ray Kinsella:
> >> On 09/04/2020 16:18, Thomas Monjalon wrote:
> >>> 09/04/2020 16:52, Ray Kinsella:
> >>>> On 09/04/2020 11:59, Thomas Monjalon wrote:
> >>>>> 09/04/2020 12:45, Ray Kinsella:
> >>>>>> On 09/04/2020 11:43, Bruce Richardson wrote:
> >>>>>>> On Thu, Apr 09, 2020 at 06:39:54AM -0400, Neil Horman wrote:
> >>>>>>>> On Thu, Apr 09, 2020 at 08:57:34AM +0100, Ray Kinsella wrote:
> >>>>>>>>> On 08/04/2020 20:56, Neil Horman wrote:
> >>>>>>>>>> +The syntax of the ``check-abi.sh`` utility is::
> >>>>>>>>>> +
> >>>>>>>>>> + ./devtools/check-abi.sh <refdir> <newdir>
> >>>>>>>>>
> >>>>>>>>> (from v1 feedback)
> >>>>>>>>> Could we simplify this all greatly, by telling people to use the meson/ninja build,
> >>>>>>>>> so they get this checking out of the box, without all the headache below?
> >>>>>>>>>
> >>>>>>>> I think bruce noted that was never merged, correct?
> >>>>>>>>
> >>>>>>> Yep, correct. :-(
> >>>>>>
> >>>>>> apologies, was there a reason?
> >>>>>
> >>>>> Because build tool job is building, not checking.
> >>>>> It would be wrong to make (slow) checks mandatory in all builds.
> >>>>>
> >>>>> The need is to enforce checking ABI.
> >>>>> The result is already published by Travis in patchwork and in an
> >>>>> email to the author I believe.
> >>>>> Not checking email and patchwork is not a good excuse.
> >>>>>
> >>>>> Patchwork must be a mandatory read for everybody for all checks
> >>>>> in general. Let's not give up on general CI workflow.
> >>>>>
> >>>>
> >>>> Thomas
> >>>>
> >>>> You are trying to solve two problems at once; CI tooling and ABI.
> >>>> Let's try to solve one at a time.
> >>>
> >>> No, you want to mix two problems in a single tool :-)
> >>>
> >>>
> >>>> 1. The ABI check, will make the build _marginally_ slower.
> >>>> You _should_ only need to rebuild the changes between A and B.
> >>>
> >>> Not so marginal.
> >>> A re-build takes less than a second. A mandatory check takes 10 secs
> >>> on my machine.
> >>>
> >>>
> >>>> 2. The meson/ninja are an order of magnitude faster than GNU Make.
> >>>> We can afford this check.
> >>>
> >>> I am doing such build 10 times (each target) per patch.
> >>> But that's not the main issue.
> >>>
> >>>
> >>>> 3. If we want to lessen the ABI burden and send the correct message.
> >>>> It should be a build blocker, contributors need to hear the message loud and clear.
> >>>
> >>> The developer needs to get or build/save the ABI reference.
> >>> Making such ABI reference for each target is not so obvious:
> >>> - all symbols must be enabled (dependencies)
> >>> - some fixes may be needed for some compilers
> >>>
> >>>
> >>>> Most important people _consuming_ DPDK will never see this message.
> >>>> Only people _changing_ the ABI will see it - the people we want to hear the message loud and clear.
> >>>
> >>> No, they will have issue in DPDK compilation if something in the check
> >>> goes wrong. We should not bother end users with internal checks.
> >>>
> >>>
> >>> The message is
> >>> a) run the check by
> >>> 1) setting DPDK_ABI_REF_VERSION on command line or in devel.config file
> >>> 2) running devtools/test-build.sh or devtools/test-meson-builds.sh
> >>> b) check what Travis is reporting in
> >>> - email to you
> >>> - patchwork reports
> >>>
> >>> I think Travis report is convenient to use.
> >>> The local check is integrated in build scripts
> >>> but cannot be run by default because of the reasons above.
> >>
> >> Thomas the reality on this is that people have a tendency to filter
> >> this messages into an email folder and don't always see them.
> >>
> >> My 2c is that this will always be a struggle unless we find a way
> >> to make it un-ignore-able.
> >> Hence my build-wiring suggestion.
> >
> > My other concern is that we will have the same issue with all checks
> > done in a CI.
> > I think the right approach is to enforce people checking CI results.
>
> Beyond asking maintainers to check, how would we enforce?
Because of the reason below...
> > They will be used to check CI in patchwork because the patches will
> > be blocked.
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH v2 08/10] eal/windows: fix rte_page_sizes with Clang on Windows
@ 2020-04-10 16:43 4% ` Dmitry Kozlyuk
1 sibling, 0 replies; 200+ results
From: Dmitry Kozlyuk @ 2020-04-10 16:43 UTC (permalink / raw)
To: dev
Cc: Dmitry Malloy (MESHCHANINOV),
Narcisa Ana Maria Vasile, Fady Bader, Tal Shnaiderman,
Dmitry Kozlyuk, Anatoly Burakov
Clang on Windows follows MS ABI where enum values are limited to 2^31-1.
Enum rte_page_size has members valued above this limit, which get
wrapped to zero, resulting in compilation error (duplicate values in
enum). Using MS ABI is mandatory for Windows EAL to call Win32 APIs.
Define these values outside of the enum for Clang on Windows only.
This does not affect runtime, because Windows doesn't run on machines
with 4GiB and 16GiB hugepages.
Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
lib/librte_eal/include/rte_memory.h | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/lib/librte_eal/include/rte_memory.h b/lib/librte_eal/include/rte_memory.h
index 1b7c3e5df..3ec673f51 100644
--- a/lib/librte_eal/include/rte_memory.h
+++ b/lib/librte_eal/include/rte_memory.h
@@ -34,8 +34,14 @@ enum rte_page_sizes {
RTE_PGSIZE_256M = 1ULL << 28,
RTE_PGSIZE_512M = 1ULL << 29,
RTE_PGSIZE_1G = 1ULL << 30,
+/* Work around Clang on Windows being limited to 32-bit underlying type. */
+#if !defined(RTE_CC_CLANG) || !defined(RTE_EXEC_ENV_WINDOWS)
RTE_PGSIZE_4G = 1ULL << 32,
RTE_PGSIZE_16G = 1ULL << 34,
+#else
+#define RTE_PGSIZE_4G (1ULL << 32)
+#define RTE_PGSIZE_16G (1ULL << 34)
+#endif
};
#define SOCKET_ID_ANY -1 /**< Any NUMA socket. */
--
2.25.1
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 00/16] update and simplify telemetrylibrary.
@ 2020-04-10 18:06 4% ` Morten Brørup
0 siblings, 0 replies; 200+ results
From: Morten Brørup @ 2020-04-10 18:06 UTC (permalink / raw)
To: Wiles, Keith
Cc: Power, Ciara, dev, Laatz, Kevin, Pattan, Reshma, jerinjacobk,
david.marchand, thomas
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Wiles, Keith
> Sent: Friday, April 10, 2020 4:22 PM
>
> > On Apr 10, 2020, at 5:49 AM, Morten Brørup <mb@smartsharesystems.com>
> wrote:
> >
> >> From: Ciara Power [mailto:ciara.power@intel.com]
> >> Sent: Wednesday, April 8, 2020 6:50 PM
> >>
> >> This patchset extensively reworks the telemetry library adding new
> >> functionality and simplifying much of the existing code, while
> >> maintaining backward compatibility.
> >>
> >> This work is based on the previously sent RFC for a "process info"
> >> library: https://patchwork.dpdk.org/project/dpdk/list/?series=7741
> >> However, rather than creating a new library, this patchset takes
> >> that work and merges it into the existing telemetry library, as
> >> mentioned above.
> >>
> >> The telemetry library as shipped in 19.11 is based upon the metrics
> >> library and outputs all statistics based on that as a source. However,
> >> this limits the telemetry output to only port-level statistics
> >> information, rather than allowing it to be used as a general scheme for
> >> telemetry information across all DPDK libraries.
> >>
> >> With this patchset applied, rather than the telemetry library being
> >> responsible for pulling ethdev stats and pushing them into the metrics
> >> library for retrieval later, each library e.g. ethdev, rawdev, and even
> >> the metrics library itself (for backwards compatiblity) now handle
> >> their
> >> own stats. Any library or app can register a callback function with
> >> telemetry, which will be called if requested by the client connected
> >> via
> >> the telemetry socket.
> >
> > Great. Standardization across libraries is a good improvement.
> >
> >> The callback function in the library/app then
> >> formats its stats, or other data, into a JSON string, and returns it to
> >> telemetry to be sent to the client.
> >
> > I am strongly opposed to using JSON as the standard format in DPDK, and
> instead prefer a binary format with zero (or minimal) type conversion.
> >
> > Here is one reason why I dislike JSON for this: A part of our application
> samples 100k+ counters every second to be able to provide drill-down
> statistics through the GUI. Converting these counters from uint64_t to JSON
> and back to uint64_t for data processing is not going to fly. And I assume
> that we are not the only company developing equipment with drill-down
> statistics.
> >
> > I am aware that there is a difference between statistics for *drill-down
> and data processing* purposes and statistics for *telemetry eyeball viewing
> only* purposes, but the line is blurry, and I see a big risk of setting a
> path that leads to JSON being used in places where it shouldn't.
> >
> > Here is another reason why I dislike JSON for this: JSON is fine for the
> LAMP stack with REST protocols. But other systems use other protocols with
> other formats, e.g. the TICK stack uses an even simpler text based format.
> So DPDK based systems supporting the TICK stack will need to convert to
> first JSON format (in the DPDK libraries), and then from JSON format to
> InfluxDB format (in the DPDK application).
> >
> > I think that type conversion does not belong inside deep inside the DPDK
> libraries, but is a job for the DPDK application. However, DPDK could
> provide libraries for efficient bulk conversion to popular formats like
> JSON. And other formats, if they are relevant, e.g. ASN.1 used by old
> school SNMP.
>
> I believe JSON has it place in this library and in DPDK as it is a good
> conversion tool and easy to utilize with a huge number of tools/languages.
JSON is extremely heavy compared to a raw binary format.
It makes sense for low volume, hierarchical structured data, but not for large tables or arrays of counters.
> Binary output gets into endianness issues and a number of other problems,
> so I would not want all of the data exported from DPDK to be in binary
> format.
Endianness considerations are only relevant for data exchanged across the network; not data exchanged across processes inside the same machine.
And if you are exchanging data across the network, you would usually implement one or more well known protocols for that, e.g. JSON over HTTPS, or ASN.1 over SNMP, or InfluxDB over UDP. This means that the application needs to implement a protocol handler, which - in my opinion - should handle the relevant data type conversions from the raw format provided by DPDK.
I think it would be silly for DPDK core libraries to provide counters in JSON format, so an SNMP Agent would need to convert them from JSON back to binary and then to ASN.1.
> If the layout of the structure changes then the code would need to
> know that on both side to be able to convert the data into the correct
> values.
I may be exaggerating here, but trying to prove a point: This is what we have ABI stability for. Structures should be designed cleverly and future proof, e.g. like the ethdev xstats. Using text based APIs is a circumvention of ABI stability.
>
> With that stated, the new telemetry code allows the application to add new
> commands and with that you can create a binary set of commands along side
> the JSON or any other output format. With the new register command we can
> create say a ‘/ethdevraw/stats,X’ set of commands that can emit binary
> format.
That would be silly. The protocol handler should make the protocol specific conversion, not the driver! Again, going to the extreme to prove a point: If I understand you correctly, this would mean that PMDs would have provide counters in ASN.1 format for SNMP.
Our application provides a HTTPS/REST based communication interface for multiple purposes, e.g. getting tables of data. And if you want to get a table of some data via this interface, you can specify the output format in the request, so you can get it in e.g. TSV format (tabulator separated with a headline) for scripts, HTML format for human eyeballs. This data conversion happens at a common location, so we can easily add other output formats. You don't want to push this all the way down to the originator of the data.
>
> Using this method we get the best of both worlds and when using languages
> like Go or Python to collect these stats we have a standard format for
> conversion. In Go it is pretty hard to do binary conversion and JSON
> conversion is just a few lines.
I don't think DPDK should provide preferential treatment to Go or Python. DPDK is based on C, and should mainly cater for C.
> JSON may not be the fastest, but if you are
> requesting stats faster than a second then use the raw commands to get the
> data, which anyone can add to its application or we can add them to DPDK as
> a standard command set.
APIs in the libraries are currently available to get data in raw format. My main concern is that libraries in the future will not provide functions to get raw data, like they do now, but only JSON formatted data for the telemetry library. This is what I want to avoid.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v3 2/9] ring: prepare ring to allow new sync schemes
2020-04-09 13:39 3% ` Ananyev, Konstantin
@ 2020-04-10 20:15 0% ` Honnappa Nagarahalli
0 siblings, 0 replies; 200+ results
From: Honnappa Nagarahalli @ 2020-04-10 20:15 UTC (permalink / raw)
To: Ananyev, Konstantin, dev
Cc: david.marchand, jielong.zjl, nd, Honnappa Nagarahalli, nd
<snip>
>
> > > Subject: [PATCH v3 2/9] ring: prepare ring to allow new sync schemes
> > >
> > > Change from *single* to *sync_type* to allow different
> > > synchronisation schemes to be applied.
> > > Mark *single* as deprecated in comments.
> > > Add new functions to allow user to query ring sync types.
> > > Replace direct access to *single* with appopriate function call.
> > >
> > > Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> > > ---
> > > app/test/test_pdump.c | 6 +-
> > > lib/librte_pdump/rte_pdump.c | 2 +-
> > > lib/librte_port/rte_port_ring.c | 12 ++--
> > > lib/librte_ring/rte_ring.c | 6 +-
> > > lib/librte_ring/rte_ring.h | 113 ++++++++++++++++++++++++++------
> > > lib/librte_ring/rte_ring_elem.h | 8 +--
> > > 6 files changed, 108 insertions(+), 39 deletions(-)
> > >
> > > diff --git a/app/test/test_pdump.c b/app/test/test_pdump.c index
> > > ad183184c..6a1180bcb 100644
> > > --- a/app/test/test_pdump.c
> > > +++ b/app/test/test_pdump.c
> > > @@ -57,8 +57,7 @@ run_pdump_client_tests(void)
> > > if (ret < 0)
> > > return -1;
> > > mp->flags = 0x0000;
> > > - ring_client = rte_ring_create("SR0", RING_SIZE, rte_socket_id(),
> > > - RING_F_SP_ENQ | RING_F_SC_DEQ);
> > > + ring_client = rte_ring_create("SR0", RING_SIZE, rte_socket_id(),
> > > +0);
> > Are you saying to get SP and SC behavior we now have to set the flags to 0?
>
> No.
> What the original cause does:
> creates SP/SC ring:
> ring_client = rte_ring_create("SR0", RING_SIZE, rte_socket_id(),
> RING_F_SP_ENQ | RING_F_SC_DEQ); Then
> manually makes it MP/MC by:
> ring_client->prod.single = 0;
> ring_client->cons.single = 0;
>
> Instead it should just create MP/MC ring straightway, as the patch does:
> ring_client = rte_ring_create("SR0", RING_SIZE, rte_socket_id(), 0);
>
> >Isn't that a ABI break?
> I don't see any.
Ack
>
> >
> > > if (ring_client == NULL) {
> > > printf("rte_ring_create SR0 failed");
> > > return -1;
> > > @@ -71,9 +70,6 @@ run_pdump_client_tests(void)
> > > }
> > > rte_eth_dev_probing_finish(eth_dev);
> > >
> > > - ring_client->prod.single = 0;
> > > - ring_client->cons.single = 0;
> > Just wondering if users outside of DPDK have done the same. I hope not,
> otherwise, we have an API break?
>
> I think no. While it is completely wrong practise, it would keep working even
> with these changes.
Ack
>
> >
> > > -
> > > printf("\n***** flags = RTE_PDUMP_FLAG_TX *****\n");
> > >
> > > for (itr = 0; itr < NUM_ITR; itr++) { diff --git
> > > a/lib/librte_pdump/rte_pdump.c b/lib/librte_pdump/rte_pdump.c index
> > > 8a01ac510..65364f2c5 100644
> > > --- a/lib/librte_pdump/rte_pdump.c
> > > +++ b/lib/librte_pdump/rte_pdump.c
> > > @@ -380,7 +380,7 @@ pdump_validate_ring_mp(struct rte_ring *ring,
> > > struct rte_mempool *mp)
> > > rte_errno = EINVAL;
> > > return -1;
> > > }
> > > - if (ring->prod.single || ring->cons.single) {
> > > + if (rte_ring_prod_single(ring) || rte_ring_cons_single(ring)) {
> > > PDUMP_LOG(ERR, "ring with either SP or SC settings"
> > > " is not valid for pdump, should have MP and MC settings\n");
> > > rte_errno = EINVAL;
> > > diff --git a/lib/librte_port/rte_port_ring.c
> > > b/lib/librte_port/rte_port_ring.c index 47fcdd06a..2f6c050fa 100644
> > > --- a/lib/librte_port/rte_port_ring.c
> > > +++ b/lib/librte_port/rte_port_ring.c
> > > @@ -44,8 +44,8 @@ rte_port_ring_reader_create_internal(void *params,
> > > int socket_id,
> > > /* Check input parameters */
> > > if ((conf == NULL) ||
> > > (conf->ring == NULL) ||
> > > - (conf->ring->cons.single && is_multi) ||
> > > - (!(conf->ring->cons.single) && !is_multi)) {
> > > + (rte_ring_cons_single(conf->ring) && is_multi) ||
> > > + (!rte_ring_cons_single(conf->ring) && !is_multi)) {
> > > RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
> > > return NULL;
> > > }
> > > @@ -171,8 +171,8 @@ rte_port_ring_writer_create_internal(void
> > > *params, int socket_id,
> > > /* Check input parameters */
> > > if ((conf == NULL) ||
> > > (conf->ring == NULL) ||
> > > - (conf->ring->prod.single && is_multi) ||
> > > - (!(conf->ring->prod.single) && !is_multi) ||
> > > + (rte_ring_prod_single(conf->ring) && is_multi) ||
> > > + (!rte_ring_prod_single(conf->ring) && !is_multi) ||
> > > (conf->tx_burst_sz > RTE_PORT_IN_BURST_SIZE_MAX)) {
> > > RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
> > > return NULL;
> > > @@ -440,8 +440,8 @@
> rte_port_ring_writer_nodrop_create_internal(void
> > > *params, int socket_id,
> > > /* Check input parameters */
> > > if ((conf == NULL) ||
> > > (conf->ring == NULL) ||
> > > - (conf->ring->prod.single && is_multi) ||
> > > - (!(conf->ring->prod.single) && !is_multi) ||
> > > + (rte_ring_prod_single(conf->ring) && is_multi) ||
> > > + (!rte_ring_prod_single(conf->ring) && !is_multi) ||
> > > (conf->tx_burst_sz > RTE_PORT_IN_BURST_SIZE_MAX)) {
> > > RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
> > > return NULL;
> > > diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c
> > > index
> > > 77e5de099..fa5733907 100644
> > > --- a/lib/librte_ring/rte_ring.c
> > > +++ b/lib/librte_ring/rte_ring.c
> > > @@ -106,8 +106,10 @@ rte_ring_init(struct rte_ring *r, const char
> > > *name, unsigned count,
> > > if (ret < 0 || ret >= (int)sizeof(r->name))
> > > return -ENAMETOOLONG;
> > > r->flags = flags;
> > > - r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
> > > - r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
> > > + r->prod.sync_type = (flags & RING_F_SP_ENQ) ?
> > > + RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
> > > + r->cons.sync_type = (flags & RING_F_SC_DEQ) ?
> > > + RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
> > >
> > > if (flags & RING_F_EXACT_SZ) {
> > > r->size = rte_align32pow2(count + 1); diff --git
> > > a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h index
> > > 18fc5d845..d4775a063 100644
> > > --- a/lib/librte_ring/rte_ring.h
> > > +++ b/lib/librte_ring/rte_ring.h
> > > @@ -61,11 +61,27 @@ enum rte_ring_queue_behavior { #define
> > > RTE_RING_NAMESIZE (RTE_MEMZONE_NAMESIZE - \
> > > sizeof(RTE_RING_MZ_PREFIX) + 1)
> > >
> > > -/* structure to hold a pair of head/tail values and other metadata
> > > */
> > > +/** prod/cons sync types */
> > > +enum rte_ring_sync_type {
> > > + RTE_RING_SYNC_MT, /**< multi-thread safe (default mode) */
> > > + RTE_RING_SYNC_ST, /**< single thread only */
> > > +};
> > > +
> > > +/**
> > > + * structure to hold a pair of head/tail values and other metadata.
> > > + * Depending on sync_type format of that structure might be
> > > +different,
> > > + * but offset for *sync_type* and *tail* values should remain the same.
> > > + */
> > > struct rte_ring_headtail {
> > > - volatile uint32_t head; /**< Prod/consumer head. */
> > > - volatile uint32_t tail; /**< Prod/consumer tail. */
> > > - uint32_t single; /**< True if single prod/cons */
> > > + volatile uint32_t head; /**< prod/consumer head. */
> > > + volatile uint32_t tail; /**< prod/consumer tail. */
> > > + RTE_STD_C11
> > > + union {
> > > + /** sync type of prod/cons */
> > > + enum rte_ring_sync_type sync_type;
> > > + /** deprecated - True if single prod/cons */
> > > + uint32_t single;
> > > + };
> > > };
> > >
> > > /**
> > > @@ -116,11 +132,10 @@ struct rte_ring { #define RING_F_EXACT_SZ
> > > 0x0004 #define RTE_RING_SZ_MASK (0x7fffffffU) /**< Ring size mask
> > > */
> > >
> > > -/* @internal defines for passing to the enqueue dequeue worker
> > > functions */ -#define __IS_SP 1 -#define __IS_MP 0 -#define __IS_SC
> > > 1 -#define __IS_MC 0
> > > +#define __IS_SP RTE_RING_SYNC_ST
> > > +#define __IS_MP RTE_RING_SYNC_MT
> > > +#define __IS_SC RTE_RING_SYNC_ST
> > > +#define __IS_MC RTE_RING_SYNC_MT
> > I think we can remove these #defines and use the new SYNC types
>
> Wouldn't that introduce an API breakage?
> Or we are ok here, as they are marked as internal?
> I think I can for sure mark them as deprecated.
I think they are internal.
Although it does not apply to your patch, rte_ring_queue_behavior also should be internal.
>
> > >
> > > /**
> > > * Calculate the memory size needed for a ring @@ -420,7 +435,7 @@
> > > rte_ring_mp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
> > > unsigned int n, unsigned int *free_space) {
> > > return __rte_ring_do_enqueue(r, obj_table, n,
> > > RTE_RING_QUEUE_FIXED,
> > > - __IS_MP, free_space);
> > > + RTE_RING_SYNC_MT, free_space);
> > > }
> > >
> > > /**
> > > @@ -443,7 +458,7 @@ rte_ring_sp_enqueue_bulk(struct rte_ring *r,
> > > void * const *obj_table,
> > > unsigned int n, unsigned int *free_space) {
> > > return __rte_ring_do_enqueue(r, obj_table, n,
> > > RTE_RING_QUEUE_FIXED,
> > > - __IS_SP, free_space);
> > > + RTE_RING_SYNC_ST, free_space);
> > > }
> > >
> > > /**
> > > @@ -470,7 +485,7 @@ rte_ring_enqueue_bulk(struct rte_ring *r, void *
> > > const *obj_table,
> > > unsigned int n, unsigned int *free_space) {
> > > return __rte_ring_do_enqueue(r, obj_table, n,
> > > RTE_RING_QUEUE_FIXED,
> > > - r->prod.single, free_space);
> > > + r->prod.sync_type, free_space);
> > > }
> > >
> > > /**
> > > @@ -554,7 +569,7 @@ rte_ring_mc_dequeue_bulk(struct rte_ring *r,
> > > void **obj_table,
> > > unsigned int n, unsigned int *available) {
> > > return __rte_ring_do_dequeue(r, obj_table, n,
> > > RTE_RING_QUEUE_FIXED,
> > > - __IS_MC, available);
> > > + RTE_RING_SYNC_MT, available);
> > > }
> > >
> > > /**
> > > @@ -578,7 +593,7 @@ rte_ring_sc_dequeue_bulk(struct rte_ring *r,
> > > void **obj_table,
> > > unsigned int n, unsigned int *available) {
> > > return __rte_ring_do_dequeue(r, obj_table, n,
> > > RTE_RING_QUEUE_FIXED,
> > > - __IS_SC, available);
> > > + RTE_RING_SYNC_ST, available);
> > > }
> > >
> > > /**
> > > @@ -605,7 +620,7 @@ rte_ring_dequeue_bulk(struct rte_ring *r, void
> > > **obj_table, unsigned int n,
> > > unsigned int *available)
> > > {
> > > return __rte_ring_do_dequeue(r, obj_table, n,
> > > RTE_RING_QUEUE_FIXED,
> > > - r->cons.single, available);
> > > + r->cons.sync_type, available);
> > > }
> > >
> > > /**
> > > @@ -777,6 +792,62 @@ rte_ring_get_capacity(const struct rte_ring *r)
> > > return r->capacity;
> > > }
> > >
> > > +/**
> > > + * Return sync type used by producer in the ring.
> > > + *
> > > + * @param r
> > > + * A pointer to the ring structure.
> > > + * @return
> > > + * Producer sync type value.
> > > + */
> > > +static inline enum rte_ring_sync_type
> > > +rte_ring_get_prod_sync_type(const struct rte_ring *r) {
> > > + return r->prod.sync_type;
> > > +}
> > > +
> > > +/**
> > > + * Check is the ring for single producer.
> > ^^ if
> > > + *
> > > + * @param r
> > > + * A pointer to the ring structure.
> > > + * @return
> > > + * true if ring is SP, zero otherwise.
> > > + */
> > > +static inline int
> > > +rte_ring_prod_single(const struct rte_ring *r) {
> > would rte_ring_is_prod_single better?
>
> Ok, can rename.
>
> >
> > > + return (rte_ring_get_prod_sync_type(r) == RTE_RING_SYNC_ST); }
> > > +
> > > +/**
> > > + * Return sync type used by consumer in the ring.
> > > + *
> > > + * @param r
> > > + * A pointer to the ring structure.
> > > + * @return
> > > + * Consumer sync type value.
> > > + */
> > > +static inline enum rte_ring_sync_type
> > > +rte_ring_get_cons_sync_type(const struct rte_ring *r) {
> > > + return r->cons.sync_type;
> > > +}
> > > +
> > > +/**
> > > + * Check is the ring for single consumer.
> > > + *
> > > + * @param r
> > > + * A pointer to the ring structure.
> > > + * @return
> > > + * true if ring is SC, zero otherwise.
> > > + */
> > > +static inline int
> > > +rte_ring_cons_single(const struct rte_ring *r) {
> > > + return (rte_ring_get_cons_sync_type(r) == RTE_RING_SYNC_ST); }
> > > +
> > All these new functions are not required to be called in the data path. They
> can be made non-inline.
>
> Well, all these functions are introduced to encourage people not to access
> ring fields sync_type/single directly but use functions instead.
> I don't know do people access ring.single directly at data-path or not, but
> assuming that they do - making these functions not-inline would force them
> to ignore these functions and keep accessing it directly.
> That was my thoughts besides making them inline.
> I think we have the same for get_size/get_capacity().
Ack
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v3 3/9] ring: introduce RTS ring mode
2020-04-09 14:52 0% ` Ananyev, Konstantin
@ 2020-04-10 23:10 0% ` Honnappa Nagarahalli
2020-04-13 14:29 0% ` David Marchand
0 siblings, 1 reply; 200+ results
From: Honnappa Nagarahalli @ 2020-04-10 23:10 UTC (permalink / raw)
To: Ananyev, Konstantin, dev
Cc: david.marchand, jielong.zjl, nd, Honnappa Nagarahalli, nd
<snip>
> Subject: RE: [PATCH v3 3/9] ring: introduce RTS ring mode
>
> > > Introduce relaxed tail sync (RTS) mode for MT ring synchronization.
> > > Aim to reduce stall times in case when ring is used on overcommited
> > > cpus (multiple active threads on the same cpu).
> > > The main difference from original MP/MC algorithm is that tail value
> > > is increased not by every thread that finished enqueue/dequeue, but
> > > only by the last one.
> > > That allows threads to avoid spinning on ring tail value, leaving
> > > actual tail value change to the last thread in the update queue.
> > >
> > > check-abi.sh reports what I believe is a false-positive about ring
> > > cons/prod changes. As a workaround, devtools/libabigail.abignore is
> > > updated to suppress *struct ring* related errors.
> > This can be removed from the commit message.
> >
> > >
> > > Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> > > ---
> > > devtools/libabigail.abignore | 7 +
> > > lib/librte_ring/Makefile | 5 +-
> > > lib/librte_ring/meson.build | 5 +-
> > > lib/librte_ring/rte_ring.c | 100 +++++++-
> > > lib/librte_ring/rte_ring.h | 110 ++++++++-
> > > lib/librte_ring/rte_ring_elem.h | 86 ++++++-
> > > lib/librte_ring/rte_ring_rts.h | 316 +++++++++++++++++++++++++
> > > lib/librte_ring/rte_ring_rts_elem.h | 205 ++++++++++++++++
> > > lib/librte_ring/rte_ring_rts_generic.h | 210 ++++++++++++++++
> > > 9 files changed, 1015 insertions(+), 29 deletions(-) create mode
> > > 100644 lib/librte_ring/rte_ring_rts.h create mode 100644
> > > lib/librte_ring/rte_ring_rts_elem.h
> > > create mode 100644 lib/librte_ring/rte_ring_rts_generic.h
> > >
> > > diff --git a/devtools/libabigail.abignore
> > > b/devtools/libabigail.abignore index a59df8f13..cd86d89ca 100644
> > > --- a/devtools/libabigail.abignore
> > > +++ b/devtools/libabigail.abignore
> > > @@ -11,3 +11,10 @@
> > > type_kind = enum
> > > name = rte_crypto_asym_xform_type
> > > changed_enumerators =
> RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
> > > +; Ignore updates of ring prod/cons
> > > +[suppress_type]
> > > + type_kind = struct
> > > + name = rte_ring
> > > +[suppress_type]
> > > + type_kind = struct
> > > + name = rte_event_ring
> > Does this block the reporting of these structures forever?
>
> Till we'll have a fix in libabigail, then we can remove these lines.
> I don't know any better alternative.
David, does this block all issues in the future for rte_ring library?
>
> >
> > > diff --git a/lib/librte_ring/Makefile b/lib/librte_ring/Makefile
> > > index 917c560ad..8f5c284cc 100644
> > > --- a/lib/librte_ring/Makefile
> > > +++ b/lib/librte_ring/Makefile
> > > @@ -18,6 +18,9 @@ SRCS-$(CONFIG_RTE_LIBRTE_RING) := rte_ring.c
> > > SYMLINK-$(CONFIG_RTE_LIBRTE_RING)-include := rte_ring.h \
> > > rte_ring_elem.h \
> > > rte_ring_generic.h \
> > > - rte_ring_c11_mem.h
> > > + rte_ring_c11_mem.h \
> > > + rte_ring_rts.h \
> > > + rte_ring_rts_elem.h \
> > > + rte_ring_rts_generic.h
> > >
> > > include $(RTE_SDK)/mk/rte.lib.mk
> > > diff --git a/lib/librte_ring/meson.build
> > > b/lib/librte_ring/meson.build index f2f3ccc88..612936afb 100644
> > > --- a/lib/librte_ring/meson.build
> > > +++ b/lib/librte_ring/meson.build
> > > @@ -5,7 +5,10 @@ sources = files('rte_ring.c') headers = files('rte_ring.h',
> > > 'rte_ring_elem.h',
> > > 'rte_ring_c11_mem.h',
> > > - 'rte_ring_generic.h')
> > > + 'rte_ring_generic.h',
> > > + 'rte_ring_rts.h',
> > > + 'rte_ring_rts_elem.h',
> > > + 'rte_ring_rts_generic.h')
> > >
> > > # rte_ring_create_elem and rte_ring_get_memsize_elem are
> > > experimental allow_experimental_apis = true diff --git
> > > a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c index
> > > fa5733907..222eec0fb 100644
> > > --- a/lib/librte_ring/rte_ring.c
> > > +++ b/lib/librte_ring/rte_ring.c
> > > @@ -45,6 +45,9 @@ EAL_REGISTER_TAILQ(rte_ring_tailq)
> > > /* true if x is a power of 2 */
> > > #define POWEROF2(x) ((((x)-1) & (x)) == 0)
> > >
> > > +/* by default set head/tail distance as 1/8 of ring capacity */
> > > +#define HTD_MAX_DEF 8
> > > +
> > > /* return the size of memory occupied by a ring */ ssize_t
> > > rte_ring_get_memsize_elem(unsigned int esize, unsigned int count) @@
> > > -
> > > 79,11 +82,84 @@ rte_ring_get_memsize(unsigned int count)
> > > return rte_ring_get_memsize_elem(sizeof(void *), count); }
> > >
> > > +/*
> > > + * internal helper function to reset prod/cons head-tail values.
> > > + */
> > > +static void
> > > +reset_headtail(void *p)
> > > +{
> > > + struct rte_ring_headtail *ht;
> > > + struct rte_ring_rts_headtail *ht_rts;
> > > +
> > > + ht = p;
> > > + ht_rts = p;
> > > +
> > > + switch (ht->sync_type) {
> > > + case RTE_RING_SYNC_MT:
> > > + case RTE_RING_SYNC_ST:
> > > + ht->head = 0;
> > > + ht->tail = 0;
> > > + break;
> > > + case RTE_RING_SYNC_MT_RTS:
> > > + ht_rts->head.raw = 0;
> > > + ht_rts->tail.raw = 0;
> > > + break;
> > > + default:
> > > + /* unknown sync mode */
> > > + RTE_ASSERT(0);
> > > + }
> > > +}
> > > +
> > > void
> > > rte_ring_reset(struct rte_ring *r)
> > > {
> > > - r->prod.head = r->cons.head = 0;
> > > - r->prod.tail = r->cons.tail = 0;
> > > + reset_headtail(&r->prod);
> > > + reset_headtail(&r->cons);
> > > +}
> > > +
> > > +/*
> > > + * helper function, calculates sync_type values for prod and cons
> > > + * based on input flags. Returns zero at success or negative
> > > + * errno value otherwise.
> > > + */
> > > +static int
> > > +get_sync_type(uint32_t flags, enum rte_ring_sync_type *prod_st,
> > > + enum rte_ring_sync_type *cons_st)
> > > +{
> > > + static const uint32_t prod_st_flags =
> > > + (RING_F_SP_ENQ | RING_F_MP_RTS_ENQ);
> > > + static const uint32_t cons_st_flags =
> > > + (RING_F_SC_DEQ | RING_F_MC_RTS_DEQ);
> > > +
> > > + switch (flags & prod_st_flags) {
> > > + case 0:
> > > + *prod_st = RTE_RING_SYNC_MT;
> > > + break;
> > > + case RING_F_SP_ENQ:
> > > + *prod_st = RTE_RING_SYNC_ST;
> > > + break;
> > > + case RING_F_MP_RTS_ENQ:
> > > + *prod_st = RTE_RING_SYNC_MT_RTS;
> > > + break;
> > > + default:
> > > + return -EINVAL;
> > > + }
> > > +
> > > + switch (flags & cons_st_flags) {
> > > + case 0:
> > > + *cons_st = RTE_RING_SYNC_MT;
> > > + break;
> > > + case RING_F_SC_DEQ:
> > > + *cons_st = RTE_RING_SYNC_ST;
> > > + break;
> > > + case RING_F_MC_RTS_DEQ:
> > > + *cons_st = RTE_RING_SYNC_MT_RTS;
> > > + break;
> > > + default:
> > > + return -EINVAL;
> > > + }
> > > +
> > > + return 0;
> > > }
> > >
> > > int
> > > @@ -100,16 +176,20 @@ rte_ring_init(struct rte_ring *r, const char
> > > *name, unsigned count,
> > > RTE_BUILD_BUG_ON((offsetof(struct rte_ring, prod) &
> > > RTE_CACHE_LINE_MASK) != 0);
> > >
> > > + RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, sync_type) !=
> > > + offsetof(struct rte_ring_rts_headtail, sync_type));
> > > + RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, tail) !=
> > > + offsetof(struct rte_ring_rts_headtail, tail.val.pos));
> > > +
> > > /* init the ring structure */
> > > memset(r, 0, sizeof(*r));
> > > ret = strlcpy(r->name, name, sizeof(r->name));
> > > if (ret < 0 || ret >= (int)sizeof(r->name))
> > > return -ENAMETOOLONG;
> > > r->flags = flags;
> > > - r->prod.sync_type = (flags & RING_F_SP_ENQ) ?
> > > - RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
> > > - r->cons.sync_type = (flags & RING_F_SC_DEQ) ?
> > > - RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
> > > + ret = get_sync_type(flags, &r->prod.sync_type, &r->cons.sync_type);
> > > + if (ret != 0)
> > > + return ret;
> > >
> > > if (flags & RING_F_EXACT_SZ) {
> > > r->size = rte_align32pow2(count + 1); @@ -126,8 +206,12
> @@
> > > rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
> > > r->mask = count - 1;
> > > r->capacity = r->mask;
> > > }
> > > - r->prod.head = r->cons.head = 0;
> > > - r->prod.tail = r->cons.tail = 0;
> > > +
> > > + /* set default values for head-tail distance */
> > > + if (flags & RING_F_MP_RTS_ENQ)
> > > + rte_ring_set_prod_htd_max(r, r->capacity / HTD_MAX_DEF);
> > > + if (flags & RING_F_MC_RTS_DEQ)
> > > + rte_ring_set_cons_htd_max(r, r->capacity / HTD_MAX_DEF);
> > >
> > > return 0;
> > > }
> > > diff --git a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h
> > > index
> > > d4775a063..f6f084d79 100644
> > > --- a/lib/librte_ring/rte_ring.h
> > > +++ b/lib/librte_ring/rte_ring.h
> > > @@ -48,6 +48,7 @@ extern "C" {
> > > #include <rte_branch_prediction.h>
> > > #include <rte_memzone.h>
> > > #include <rte_pause.h>
> > > +#include <rte_debug.h>
> > >
> > > #define RTE_TAILQ_RING_NAME "RTE_RING"
> > >
> > > @@ -65,10 +66,13 @@ enum rte_ring_queue_behavior { enum
> > > rte_ring_sync_type {
> > > RTE_RING_SYNC_MT, /**< multi-thread safe (default mode) */
> > > RTE_RING_SYNC_ST, /**< single thread only */
> > > +#ifdef ALLOW_EXPERIMENTAL_API
> > > + RTE_RING_SYNC_MT_RTS, /**< multi-thread relaxed tail sync */
> > > #endif
> > > };
> > >
> > > /**
> > > - * structure to hold a pair of head/tail values and other metadata.
> > > + * structures to hold a pair of head/tail values and other metadata.
> > > * Depending on sync_type format of that structure might be different,
> > > * but offset for *sync_type* and *tail* values should remain the same.
> > > */
> > > @@ -84,6 +88,21 @@ struct rte_ring_headtail {
> > > };
> > > };
> > >
> > > +union rte_ring_ht_poscnt {
> > nit, this is specific to RTS, may be change this to rte_ring_rts_ht_poscnt?
>
> Ok.
>
> >
> > > + uint64_t raw;
> > > + struct {
> > > + uint32_t cnt; /**< head/tail reference counter */
> > > + uint32_t pos; /**< head/tail position */
> > > + } val;
> > > +};
> > > +
> > > +struct rte_ring_rts_headtail {
> > > + volatile union rte_ring_ht_poscnt tail;
> > > + enum rte_ring_sync_type sync_type; /**< sync type of prod/cons */
> > > + uint32_t htd_max; /**< max allowed distance between head/tail */
> > > + volatile union rte_ring_ht_poscnt head; };
> > > +
> > > /**
> > > * An RTE ring structure.
> > > *
> > > @@ -111,11 +130,21 @@ struct rte_ring {
> > > char pad0 __rte_cache_aligned; /**< empty cache line */
> > >
> > > /** Ring producer status. */
> > > - struct rte_ring_headtail prod __rte_cache_aligned;
> > > + RTE_STD_C11
> > > + union {
> > > + struct rte_ring_headtail prod;
> > > + struct rte_ring_rts_headtail rts_prod;
> > > + } __rte_cache_aligned;
> > > +
> > > char pad1 __rte_cache_aligned; /**< empty cache line */
> > >
> > > /** Ring consumer status. */
> > > - struct rte_ring_headtail cons __rte_cache_aligned;
> > > + RTE_STD_C11
> > > + union {
> > > + struct rte_ring_headtail cons;
> > > + struct rte_ring_rts_headtail rts_cons;
> > > + } __rte_cache_aligned;
> > > +
> > > char pad2 __rte_cache_aligned; /**< empty cache line */ };
> > >
> > > @@ -132,6 +161,9 @@ struct rte_ring { #define RING_F_EXACT_SZ
> > > 0x0004 #define RTE_RING_SZ_MASK (0x7fffffffU) /**< Ring size mask
> > > */
> > >
> > > +#define RING_F_MP_RTS_ENQ 0x0008 /**< The default enqueue is "MP
> RTS".
> > > +*/ #define RING_F_MC_RTS_DEQ 0x0010 /**< The default dequeue is
> "MC
> > > +RTS". */
> > > +
> > > #define __IS_SP RTE_RING_SYNC_ST
> > > #define __IS_MP RTE_RING_SYNC_MT
> > > #define __IS_SC RTE_RING_SYNC_ST
> > > @@ -461,6 +493,10 @@ rte_ring_sp_enqueue_bulk(struct rte_ring *r,
> > > void * const *obj_table,
> > > RTE_RING_SYNC_ST, free_space);
> > > }
> > >
> > > +#ifdef ALLOW_EXPERIMENTAL_API
> > > +#include <rte_ring_rts.h>
> > > +#endif
> > > +
> > > /**
> > > * Enqueue several objects on a ring.
> > > *
> > > @@ -484,8 +520,21 @@ static __rte_always_inline unsigned int
> > > rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
> > > unsigned int n, unsigned int *free_space) {
> > > - return __rte_ring_do_enqueue(r, obj_table, n,
> > > RTE_RING_QUEUE_FIXED,
> > > - r->prod.sync_type, free_space);
> > > + switch (r->prod.sync_type) {
> > > + case RTE_RING_SYNC_MT:
> > > + return rte_ring_mp_enqueue_bulk(r, obj_table, n, free_space);
> > > + case RTE_RING_SYNC_ST:
> > > + return rte_ring_sp_enqueue_bulk(r, obj_table, n, free_space);
> > Have you validated if these affect the performance for the existing APIs?
>
> I run ring_pmd_perf_autotest
> (AFAIK, that's the only one of our perf tests that calls
> rte_ring_enqueue/dequeue), and didn't see any real difference in perf
> numbers.
>
> > I am also wondering why should we support these new modes in the legacy
> APIs?
>
> Majority of DPDK users still do use legacy API, and I am not sure all of them
> will be happy to switch to _elem_ one manually.
> Plus I can't see how we can justify that after let say:
> rte_ring_init(ring, ..., RING_F_MP_HTS_ENQ | RING_F_MC_HTS_DEQ); returns
> with success valid call to rte_ring_enqueue(ring,...) should fail.
Agree, I think the only way right now is through documentation.
>
> > I think users should move to use rte_ring_xxx_elem APIs. If users want to
> use RTS/HTS it will be a good time for them to move to new APIs.
>
> If they use rte_ring_enqueue/dequeue all they have to do - just change flags
> in ring_create/ring_init call.
> With what you suggest - they have to change every
> rte_ring_enqueue/dequeue to rte_ring_elem_enqueue/dequeue.
> That's much bigger code churn.
But these are just 1 to 1 mapped. I would think, there are not a whole lot of them in the application code, may be ~10 lines?
I think the bigger factor for the user here is the algorithm changes in rte_ring library. Bigger effort for the users would be testing rather than code changes in the applications.
>
> > They anyway have to test their code for RTS/HTS, might as well make the
> change to new APIs and test both.
> > It will be less code to maintain for the community as well.
>
> That's true, right now there is a lot of duplication between _elem_ and legacy
> code.
> Actually the only real diff between them - actual copying of the objects.
> But I thought we are going to deal with that, just by changing one day all
> legacy API to wrappers around _elem_ calls, i.e something like:
>
> static __rte_always_inline unsigned int
> rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
> unsigned int n, unsigned int *free_space) {
> return rte_ring_enqueue_elem_bulk(r, obj_table, sizeof(uintptr_t), n,
> free_space); }
>
> That way users will switch to new API automatically, without any extra effort
> for them, and we will be able to remove legacy code.
> Do you have some other thoughts here how to deal with this legacy/elem
> conversion?
Yes, that is what I was thinking, but had not considered any addition of new APIs.
But, I am wondering if we should look at deprecation? If we decide to deprecate, it would be good to avoid making the users of RTS/HTS do the work twice (once to make the switch to RTS/HTS and then another to _elem_ APIs).
One thing we can do is to implement the wrappers you mentioned above for RTS/HTS now. I also it is worth considering to switch to these wrappers 20.05 so that come 20.11, we have a code base that has gone through couple of releases' testing.
>
> >
> > > #ifdef
> > > +ALLOW_EXPERIMENTAL_API
> > > + case RTE_RING_SYNC_MT_RTS:
> > > + return rte_ring_mp_rts_enqueue_bulk(r, obj_table, n,
> > > + free_space);
> > > +#endif
> > > + }
> > > +
> > > + /* valid ring should never reach this point */
> > > + RTE_ASSERT(0);
> > > + return 0;
> > > }
> > >
<snip>
> > >
> > > #ifdef __cplusplus
> > > diff --git a/lib/librte_ring/rte_ring_elem.h
> > > b/lib/librte_ring/rte_ring_elem.h index 28f9836e6..5de0850dc 100644
> > > --- a/lib/librte_ring/rte_ring_elem.h
> > > +++ b/lib/librte_ring/rte_ring_elem.h
> > > @@ -542,6 +542,8 @@ rte_ring_sp_enqueue_bulk_elem(struct rte_ring
> > > *r, const void *obj_table,
> > > RTE_RING_QUEUE_FIXED, __IS_SP, free_space); }
> > >
> > > +#include <rte_ring_rts_elem.h>
<snip>
> > >
> > > #ifdef __cplusplus
> > > diff --git a/lib/librte_ring/rte_ring_rts.h
> > > b/lib/librte_ring/rte_ring_rts.h new file mode 100644 index
> > > 000000000..18404fe48
> > > --- /dev/null
> > > +++ b/lib/librte_ring/rte_ring_rts.h
> > IMO, we should not provide these APIs.
>
> You mean only _elem_ ones, as discussed above?
Yes
>
> >
> > > @@ -0,0 +1,316 @@
> > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > + *
> > > + * Copyright (c) 2010-2017 Intel Corporation
> > nit, the year should change to 2020? Look at others too.
>
> ack, will do.
>
> >
> > > + * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
> > > + * All rights reserved.
> > > + * Derived from FreeBSD's bufring.h
> > > + * Used as BSD-3 Licensed with permission from Kip Macy.
> > > + */
> > > +
> > > +#ifndef _RTE_RING_RTS_H_
> > > +#define _RTE_RING_RTS_H_
> > > +
> > > +/**
> > > + * @file rte_ring_rts.h
> > > + * @b EXPERIMENTAL: this API may change without prior notice
> > > + * It is not recommended to include this file directly.
> > > + * Please include <rte_ring.h> instead.
> > > + *
> > > + * Contains functions for Relaxed Tail Sync (RTS) ring mode.
> > > + * The main idea remains the same as for our original MP/MC
> >
> > ^^^ the
> > > +synchronization
> > > + * mechanism.
> > > + * The main difference is that tail value is increased not
> > > + * by every thread that finished enqueue/dequeue,
> > > + * but only by the last one doing enqueue/dequeue.
> > should we say 'current last' or 'last thread at a given instance'?
> >
> > > + * That allows threads to skip spinning on tail value,
> > > + * leaving actual tail value change to last thread in the update queue.
> > nit, I understand what you mean by 'update queue' here. IMO, we should
> remove it as it might confuse some.
> >
> > > + * RTS requires 2 64-bit CAS for each enqueue(/dequeue) operation:
> > > + * one for head update, second for tail update.
> > > + * As a gain it allows thread to avoid spinning/waiting on tail value.
> > > + * In comparision original MP/MC algorithm requires one 32-bit CAS
> > > + * for head update and waiting/spinning on tail value.
> > > + *
> > > + * Brief outline:
> > > + * - introduce refcnt for both head and tail.
> > Suggesting using the same names as used in the structures.
> >
> > > + * - increment head.refcnt for each head.value update
> > > + * - write head:value and head:refcnt atomically (64-bit CAS)
> > > + * - move tail.value ahead only when tail.refcnt + 1 ==
> > > + head.refcnt
> > May be add '(indicating that this is the last thread updating the tail)'
> >
> > > + * - increment tail.refcnt when each enqueue/dequeue op finishes
> > May be add 'otherwise' at the beginning.
> >
> > > + * (no matter is tail:value going to change or not)
> > nit ^^ if
> > > + * - write tail.value and tail.recnt atomically (64-bit CAS)
> > > + *
> > > + * To avoid producer/consumer starvation:
> > > + * - limit max allowed distance between head and tail value (HTD_MAX).
> > > + * I.E. thread is allowed to proceed with changing head.value,
> > > + * only when: head.value - tail.value <= HTD_MAX
> > > + * HTD_MAX is an optional parameter.
> > > + * With HTD_MAX == 0 we'll have fully serialized ring -
> > > + * i.e. only one thread at a time will be able to enqueue/dequeue
> > > + * to/from the ring.
> > > + * With HTD_MAX >= ring.capacity - no limitation.
> > > + * By default HTD_MAX == ring.capacity / 8.
> > > + */
> > > +
> > > +#ifdef __cplusplus
> > > +extern "C" {
> > > +#endif
> > > +
> > > +#include <rte_ring_rts_generic.h>
> > > +
<snip>
> > > +
> > > +#endif /* _RTE_RING_RTS_H_ */
> > > diff --git a/lib/librte_ring/rte_ring_rts_elem.h
> > > b/lib/librte_ring/rte_ring_rts_elem.h
> > > new file mode 100644
> > > index 000000000..71a331b23
> > > --- /dev/null
> > > +++ b/lib/librte_ring/rte_ring_rts_elem.h
> > > @@ -0,0 +1,205 @@
> > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > + *
> > > + * Copyright (c) 2010-2017 Intel Corporation
> > > + * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
> > > + * All rights reserved.
> > > + * Derived from FreeBSD's bufring.h
> > > + * Used as BSD-3 Licensed with permission from Kip Macy.
> > > + */
> > > +
> > > +#ifndef _RTE_RING_RTS_ELEM_H_
> > > +#define _RTE_RING_RTS_ELEM_H_
> > > +
> > > +/**
> > > + * @file rte_ring_rts_elem.h
> > > + * @b EXPERIMENTAL: this API may change without prior notice
> > > + *
> > > + * It is not recommended to include this file directly.
> > > + * Please include <rte_ring_elem.h> instead.
> > > + * Contains *ring_elem* functions for Relaxed Tail Sync (RTS) ring mode.
> > > + * for more details please refer to <rte_ring_rts.h>.
> > > + */
> > > +
> > > +#ifdef __cplusplus
> > > +extern "C" {
> > > +#endif
> > > +
> > > +#include <rte_ring_rts_generic.h>
> > > +
> > > +/**
> > > + * @internal Enqueue several objects on the RTS ring.
> > > + *
> > > + * @param r
> > > + * A pointer to the ring structure.
> > > + * @param obj_table
> > > + * A pointer to a table of void * pointers (objects).
> > > + * @param n
> > > + * The number of objects to add in the ring from the obj_table.
> > > + * @param behavior
> > > + * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a
> ring
> > > + * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible
> from
> > > ring
> > > + * @param free_space
> > > + * returns the amount of space after the enqueue operation has finished
> > > + * @return
> > > + * Actual number of objects enqueued.
> > > + * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> > > + */
> > > +static __rte_always_inline unsigned int
> > > +__rte_ring_do_rts_enqueue_elem(struct rte_ring *r, void * const
> > > +*obj_table,
> > obj_table should be of type 'const void * obj_table' (looks like copy paste
> error). Please check the other APIs below too.
> >
> > > + uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
> > 'esize' is not documented in the comments above the function. You can
> > copy the header from rte_ring_elem.h file. Please check other APIs as well.
>
> Ack to both, will fix.
>
> >
> > > + uint32_t *free_space)
> > > +{
> > > + uint32_t free, head;
> > > +
> > > + n = __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
> > > +
> > > + if (n != 0) {
> > > + __rte_ring_enqueue_elems(r, head, obj_table, esize, n);
> > > + __rte_ring_rts_update_tail(&r->rts_prod);
> > > + }
> > > +
> > > + if (free_space != NULL)
> > > + *free_space = free - n;
> > > + return n;
> > > +}
> > > +
<snip>
> > > +
> > > +#endif /* _RTE_RING_RTS_ELEM_H_ */
> > > diff --git a/lib/librte_ring/rte_ring_rts_generic.h
> > > b/lib/librte_ring/rte_ring_rts_generic.h
> > > new file mode 100644
> > > index 000000000..f88460d47
> > > --- /dev/null
> > > +++ b/lib/librte_ring/rte_ring_rts_generic.h
> > I do not know the benefit to providing the generic version. Do you know
> why this was done in the legacy APIs?
>
> I think at first we had generic API only, then later C11 was added.
> As I remember, C11 one on IA was measured as a bit slower then generic,
> so it was decided to keep both.
>
> > If there is no performance difference between generic and C11 versions,
> should we just skip the generic version?
> > The oldest compiler in CI are GCC 4.8.5 and Clang 3.4.2 and C11 built-ins
> are supported earlier than these compiler versions.
> > I feel the code is growing exponentially in rte_ring library and we should try
> to cut non-value-ass code/APIs aggressively.
>
> I'll check is there perf difference for RTS and HTS between generic and C11
> versions on IA.
> Meanwhile please have a proper look at C11 implementation, I am not that
> familiar with C11 atomics yet.
ok
> If there would be no problems with it and no noticeable diff in performance -
> I am ok to have for RTS/HTS modes C11 version only.
>
> >
> > > @@ -0,0 +1,210 @@
> > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > + *
> > > + * Copyright (c) 2010-2017 Intel Corporation
> > > + * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
> > > + * All rights reserved.
> > > + * Derived from FreeBSD's bufring.h
> > > + * Used as BSD-3 Licensed with permission from Kip Macy.
> > > + */
> > > +
> > > +#ifndef _RTE_RING_RTS_GENERIC_H_
> > > +#define _RTE_RING_RTS_GENERIC_H_
> > > +
> > > +/**
> > > + * @file rte_ring_rts_generic.h
> > > + * It is not recommended to include this file directly,
> > > + * include <rte_ring.h> instead.
> > > + * Contains internal helper functions for Relaxed Tail Sync (RTS) ring
> mode.
> > > + * For more information please refer to <rte_ring_rts.h>.
> > > + */
<snip>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [dpdk-stable] [PATCH v2] mbuf: replace zero-length marker with unnamed union
2020-04-09 10:49 3% ` Thomas Monjalon
2020-04-09 16:09 0% ` Ray Kinsella
@ 2020-04-11 2:50 0% ` Gavin Hu
1 sibling, 0 replies; 200+ results
From: Gavin Hu @ 2020-04-11 2:50 UTC (permalink / raw)
To: thomas
Cc: David Marchand, Kevin Traynor, Bruce Richardson,
Morten Brørup, Ferruh Yigit, dev, nd, jerinj,
Honnappa Nagarahalli, Ruifeng Wang, Phil Yang, Joyce Kong,
stable, Olivier MATZ, Konstantin Ananyev, Andrew Rybchenko, nd,
mdr, nd
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Thursday, April 9, 2020 6:50 PM
> To: Gavin Hu <Gavin.Hu@arm.com>
> Cc: David Marchand <david.marchand@redhat.com>; Kevin Traynor
> <ktraynor@redhat.com>; Bruce Richardson <bruce.richardson@intel.com>;
> Morten Brørup <mb@smartsharesystems.com>; Ferruh Yigit
> <ferruh.yigit@intel.com>; dev@dpdk.org; nd <nd@arm.com>;
> jerinj@marvell.com; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; Ruifeng Wang
> <Ruifeng.Wang@arm.com>; Phil Yang <Phil.Yang@arm.com>; Joyce Kong
> <Joyce.Kong@arm.com>; stable@dpdk.org; Olivier MATZ
> <olivier.matz@6wind.com>; Konstantin Ananyev
> <konstantin.ananyev@intel.com>; Andrew Rybchenko
> <arybchenko@solarflare.com>; nd <nd@arm.com>; mdr@ashroe.eu
> Subject: Re: [dpdk-stable] [dpdk-dev] [PATCH v2] mbuf: replace zero-length
> marker with unnamed union
>
> 09/04/2020 11:48, Gavin Hu:
> > From: David Marchand <david.marchand@redhat.com>
> > > On Wed, Apr 8, 2020 at 5:05 PM Gavin Hu <Gavin.Hu@arm.com> wrote:
> > > > From: Kevin Traynor <ktraynor@redhat.com>
> > > > > Hi Gavin, I lost track if v2 is still a candidate for merge. fwiw, it
> > > > > compiles without giving the zero-length-bounds warning on my
> system.
> > > > >
> > > > > Kevin.
> > > >
> > > > Yes, this path alone is a candidate for merge.
> > >
> > > This patch is not mergeable, it would trigger failures in the ABI checks.
> >
> > Isn't it a false failure? If yes, is it ignorable?
> >
> > > You can see in patchwork that the robot reported a warning in Travis.
> > > http://mails.dpdk.org/archives/test-report/2020-March/119919.html
> > > https://travis-ci.com/github/ovsrobot/dpdk/jobs/295652710#L4476
> > >
> > >
> > > I opened a bz to libabigail.
> > > https://sourceware.org/bugzilla/show_bug.cgi?id=25661
> > >
> > >
> > > Either a different solution is found, or your patch will have to deal
> > > with this issue (libabigail fix won't be ready soon afaik) and waive
> > > this.
> >
> > Maybe we come back to 'disable the warning', before the libabigail fix
> ready? or alternatively ignore this ABI false failure, if it is.
> > I do not have ideas of what otherwise the options are.
>
> Gavin,
> I did not check this case.
> But in general, we do not skip checks, except some checkpatch ones.
> The policy with ABI checks is "NEVER SKIP".
> We prefer postponing patches, waiting for someone to fix tooling.
Ok, I am fine with this.
> There is a lack of motivation currently for general concerns.
> We need to avoid being "write-only" contributors.
> So two things need to be done:
> 1/ improve tooling where it needs
> 2/ review patches from others
> I published a review list recently:
> http://mails.dpdk.org/archives/announce/2020-April/000315.html
Thanks!
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH dpdk-dev v2 1/2] eal: introduce rte-init queue for libraries initialization
@ 2020-04-12 3:20 3% ` Tonghao Zhang
2020-04-12 3:32 0% ` Tonghao Zhang
0 siblings, 1 reply; 200+ results
From: Tonghao Zhang @ 2020-04-12 3:20 UTC (permalink / raw)
To: Jerin Jacob
Cc: Olivier Matz, Andrew Rybchenko, Gage Eads, Artem V. Andreev,
Jerin Jacob, Nithin Dabilpuram, Vamsi Attunuru, Hemant Agrawal,
David Marchand, Anatoly Burakov, Richardson, Bruce, dpdk-dev
On Fri, Apr 10, 2020 at 9:11 PM Jerin Jacob <jerinjacobk@gmail.com> wrote:
>
> On Fri, Apr 10, 2020 at 11:48 AM Jerin Jacob <jerinjacobk@gmail.com> wrote:
> >
>
> Three more items are missing in this patch
>
> 1) Unit test case for new API
> 2) Make the new API __rte_experimal
Hi Jerin
This API will be invoked in mempool, if use that prefix, there is a
compile warning:
include/rte_mempool.h:933:2: error: ‘rte_init_register’ is deprecated
(declared at x86_64-native-linuxapp-gcc/include/rte_init.h:61): Symbol
is not yet part of stable ABI [-Werror=deprecated-declarations]
remove the __rte_experimal in the patch 2?
> 3) Update the .map file
>
>
> > On Thu, Apr 9, 2020 at 8:33 PM <xiangxia.m.yue@gmail.com> wrote:
> > >
> > > From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > >
> > > This patch introduces last-init queue, user can register a
> > > callback for theirs initialization. Running rte_last_init_run(),
> >
> > The above section needs to be rewritten wrt v2 changes.
> >
> > > the almost resource of DPDK are available, such as memzone, ring.
> > > With this way, user don't introduce additional codes in eal layer.
> > >
> > > [This patch will be used for next patch.]
> >
> > See below
> >
> >
> > >
> > > Signed-off-by: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > ---
> > See above
> >
> > Move [This patch will be used for next patch.] here to avoid
> > unnecessary information in the git commit.
> >
> > > v2:
> > > * rename rte_last_init_register ->rte_init_register
> > > * rename rte_last_init struct ->rte_init
> > > * rename rte_init_cb ->rte_init_cb_t
> > > * free the rte_init node when not used.
> > > * remove rte_init and others to eal_private.h
> > > * add comments
> > > * fix checkpatch warning
> > > ---
> > > diff --git a/lib/librte_eal/include/rte_init.h b/lib/librte_eal/include/rte_init.h
> > > new file mode 100644
> > > index 0000000..636efff
> > > --- /dev/null
> > > +++ b/lib/librte_eal/include/rte_init.h
> > > @@ -0,0 +1,59 @@
> > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > + * Copyright 2020 DPDK Community
> > > + */
> > > +
> > > +#ifndef _RTE_INIT_H_
> > > +#define _RTE_INIT_H_
> >
> > @file section is missing. See
> > lib/librte_eal/common/include/rte_errno.h as example.
> >
> >
> > > +#ifdef __cplusplus
> > > +extern "C" {
> > > +#endif
> > > +
> > > +#include <stdio.h>
> > > +#include <sys/queue.h>
> >
> > <sys/queue.h> is not required in public API header file.
> >
> > > +
> > > +/**
> > > + * Implementation specific callback function which is
> > > + * responsible for specificed initialization.
> > > + *
> > > + * This is called when almost resources are available.
> > > + *
> > > + * @return
> > > + * 0 for successful callback
> > > + * Negative for unsuccessful callback with error value
> > > + */
> > > +typedef int (*rte_init_cb_t)(const void *arg);
> > > +
> > > +/**
> > > + * rte_init type.
> > > + *
> > > + * The rte_init of RTE_INIT_PRE are called firstly,
> > > + * and then RTE_INIT_POST.
> > > + */
> > > +enum rte_init_type {
> > > + RTE_INIT_PRE,
> >
> > Type specific comment is missing.
> >
> > Example as reference for formatting.
> >
> > /**
> > * Enumerate trace mode operation.
> > */
> > enum rte_trace_mode_e {
> > /**
> > * In this mode, When no space left in trace buffer, the subsequent
> > * events overwrite the old events in the trace buffer.
> > */
> > RTE_TRACE_MODE_OVERWRITE,
> > /**
> > * In this mode, When no space left on trace buffer, the subsequent
> > * events shall not be recorded in the trace buffer.
> > */
> > RTE_TRACE_MODE_DISCARD,
> > };
> >
> > > + RTE_INIT_POST
> > > +};
> >
> >
> > > +
> > > +/**
> > > + * Register a rte_init callback.
> > > + *
> > > + * @param cb
> > > + * A pointer to a rte_init_cb structure, which will be used
> >
> > s/used/invoked?
> >
> > > + * in rte_eal_init().
> > > + *
> > > + * @param arg
> > > + * The cb will use that as param.
> > > + *
> > > + * @param type
> > > + * The type of rte_init registered.
> > > + */
> > > +
> > > +void rte_init_register(rte_init_cb_t cb, const void *arg,
> > > + enum rte_init_type type);
> > > +
> > > +#ifdef __cplusplus
> > > +}
> > > +#endif
> > > +
> > > +#endif /* _RTE_INIT_H_ */
--
Best regards, Tonghao
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH dpdk-dev v2 1/2] eal: introduce rte-init queue for libraries initialization
2020-04-12 3:20 3% ` Tonghao Zhang
@ 2020-04-12 3:32 0% ` Tonghao Zhang
2020-04-13 11:32 0% ` Jerin Jacob
0 siblings, 1 reply; 200+ results
From: Tonghao Zhang @ 2020-04-12 3:32 UTC (permalink / raw)
To: Jerin Jacob
Cc: Olivier Matz, Andrew Rybchenko, Gage Eads, Artem V. Andreev,
Jerin Jacob, Nithin Dabilpuram, Vamsi Attunuru, Hemant Agrawal,
David Marchand, Anatoly Burakov, Richardson, Bruce, dpdk-dev
On Sun, Apr 12, 2020 at 11:20 AM Tonghao Zhang <xiangxia.m.yue@gmail.com> wrote:
>
> On Fri, Apr 10, 2020 at 9:11 PM Jerin Jacob <jerinjacobk@gmail.com> wrote:
> >
> > On Fri, Apr 10, 2020 at 11:48 AM Jerin Jacob <jerinjacobk@gmail.com> wrote:
> > >
> >
> > Three more items are missing in this patch
Can I add "Co-authored-by" for you ?
> > 1) Unit test case for new API
> > 2) Make the new API __rte_experimal
> Hi Jerin
> This API will be invoked in mempool, if use that prefix, there is a
> compile warning:
> include/rte_mempool.h:933:2: error: ‘rte_init_register’ is deprecated
> (declared at x86_64-native-linuxapp-gcc/include/rte_init.h:61): Symbol
> is not yet part of stable ABI [-Werror=deprecated-declarations]
>
> remove the __rte_experimal in the patch 2?
>
> > 3) Update the .map file
> >
> >
> > > On Thu, Apr 9, 2020 at 8:33 PM <xiangxia.m.yue@gmail.com> wrote:
> > > >
> > > > From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > >
> > > > This patch introduces last-init queue, user can register a
> > > > callback for theirs initialization. Running rte_last_init_run(),
> > >
> > > The above section needs to be rewritten wrt v2 changes.
> > >
> > > > the almost resource of DPDK are available, such as memzone, ring.
> > > > With this way, user don't introduce additional codes in eal layer.
> > > >
> > > > [This patch will be used for next patch.]
> > >
> > > See below
> > >
> > >
> > > >
> > > > Signed-off-by: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > > ---
> > > See above
> > >
> > > Move [This patch will be used for next patch.] here to avoid
> > > unnecessary information in the git commit.
> > >
> > > > v2:
> > > > * rename rte_last_init_register ->rte_init_register
> > > > * rename rte_last_init struct ->rte_init
> > > > * rename rte_init_cb ->rte_init_cb_t
> > > > * free the rte_init node when not used.
> > > > * remove rte_init and others to eal_private.h
> > > > * add comments
> > > > * fix checkpatch warning
> > > > ---
> > > > diff --git a/lib/librte_eal/include/rte_init.h b/lib/librte_eal/include/rte_init.h
> > > > new file mode 100644
> > > > index 0000000..636efff
> > > > --- /dev/null
> > > > +++ b/lib/librte_eal/include/rte_init.h
> > > > @@ -0,0 +1,59 @@
> > > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > > + * Copyright 2020 DPDK Community
> > > > + */
> > > > +
> > > > +#ifndef _RTE_INIT_H_
> > > > +#define _RTE_INIT_H_
> > >
> > > @file section is missing. See
> > > lib/librte_eal/common/include/rte_errno.h as example.
> > >
> > >
> > > > +#ifdef __cplusplus
> > > > +extern "C" {
> > > > +#endif
> > > > +
> > > > +#include <stdio.h>
> > > > +#include <sys/queue.h>
> > >
> > > <sys/queue.h> is not required in public API header file.
> > >
> > > > +
> > > > +/**
> > > > + * Implementation specific callback function which is
> > > > + * responsible for specificed initialization.
> > > > + *
> > > > + * This is called when almost resources are available.
> > > > + *
> > > > + * @return
> > > > + * 0 for successful callback
> > > > + * Negative for unsuccessful callback with error value
> > > > + */
> > > > +typedef int (*rte_init_cb_t)(const void *arg);
> > > > +
> > > > +/**
> > > > + * rte_init type.
> > > > + *
> > > > + * The rte_init of RTE_INIT_PRE are called firstly,
> > > > + * and then RTE_INIT_POST.
> > > > + */
> > > > +enum rte_init_type {
> > > > + RTE_INIT_PRE,
> > >
> > > Type specific comment is missing.
> > >
> > > Example as reference for formatting.
> > >
> > > /**
> > > * Enumerate trace mode operation.
> > > */
> > > enum rte_trace_mode_e {
> > > /**
> > > * In this mode, When no space left in trace buffer, the subsequent
> > > * events overwrite the old events in the trace buffer.
> > > */
> > > RTE_TRACE_MODE_OVERWRITE,
> > > /**
> > > * In this mode, When no space left on trace buffer, the subsequent
> > > * events shall not be recorded in the trace buffer.
> > > */
> > > RTE_TRACE_MODE_DISCARD,
> > > };
> > >
> > > > + RTE_INIT_POST
> > > > +};
> > >
> > >
> > > > +
> > > > +/**
> > > > + * Register a rte_init callback.
> > > > + *
> > > > + * @param cb
> > > > + * A pointer to a rte_init_cb structure, which will be used
> > >
> > > s/used/invoked?
> > >
> > > > + * in rte_eal_init().
> > > > + *
> > > > + * @param arg
> > > > + * The cb will use that as param.
> > > > + *
> > > > + * @param type
> > > > + * The type of rte_init registered.
> > > > + */
> > > > +
> > > > +void rte_init_register(rte_init_cb_t cb, const void *arg,
> > > > + enum rte_init_type type);
> > > > +
> > > > +#ifdef __cplusplus
> > > > +}
> > > > +#endif
> > > > +
> > > > +#endif /* _RTE_INIT_H_ */
>
>
>
> --
> Best regards, Tonghao
--
Best regards, Tonghao
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] build: add arm 32bit cross compilation to Meson
2020-04-09 14:14 3% [dpdk-dev] [PATCH] build: add arm 32bit cross compilation to Meson Juraj Linkeš
@ 2020-04-13 6:23 0% ` Ruifeng Wang
2020-04-14 7:05 0% ` Juraj Linkeš
0 siblings, 1 reply; 200+ results
From: Ruifeng Wang @ 2020-04-13 6:23 UTC (permalink / raw)
To: Juraj Linkeš, bruce.richardson
Cc: dev, Honnappa Nagarahalli, Gavin Hu, nd, nd
Hi Juraj,
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Juraj Linke?
> Sent: Thursday, April 9, 2020 10:15 PM
> To: bruce.richardson@intel.com
> Cc: dev@dpdk.org; Juraj Linkeš <juraj.linkes@pantheon.tech>
> Subject: [dpdk-dev] [PATCH] build: add arm 32bit cross compilation to Meson
>
> * Add arm-linux-gnueabihf cross-file
> * Add generic and default arm 32 bit flags to arm meson.build
> * Add support for disabling drivers using flags defined in Meson
> * Change checks from dpdk_conf.has() to dpdk.conf.get()
> * When processing which drivers to build, check whether the
> appropriate RTE flag isn't set to false
>
The changes are not small as a single patch.
Could you split it into series of smaller patches?
I think each bullet in commit message can be a separate patch.
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
> app/test-pmd/meson.build | 4 +-
> app/test/meson.build | 2 +-
> config/arm/arm_armv7a_linux_gcc | 17 ++++
> config/arm/meson.build | 157 +++++++++++++++++++++-------------
> drivers/meson.build | 13 ++-
> drivers/net/kni/meson.build | 2 +-
> examples/ethtool/meson.build | 2 +-
> examples/ioat/meson.build | 2 +-
> examples/kni/meson.build | 2 +-
> examples/vm_power_manager/meson.build | 4 +-
> lib/librte_port/meson.build | 2 +-
> lib/meson.build | 2 +-
> 12 files changed, 133 insertions(+), 76 deletions(-) create mode 100644
> config/arm/arm_armv7a_linux_gcc
>
<snip>
> --
> 2.11.0
> NOTES: tested here: https://travis-
> ci.com/github/jlinkes/dpdk/builds/159597484
> There are two issues I would like to get feedback for:
> 1. the aarch64 -> arm cross compilation fails when compiling l3fwd example
> [0].
> I think this failure needs to be fixed by arm devs, but I would like to have
> this confirmed.
Yes, this should be fixed in source code.
> 2. the patch breaks ABI checks on aarch64 [1]. The reason is that the
> RTE_LIBRTE_FM10K_PMD and RTE_LIBRTE_AVP_PMD flags in
> config/arm/meson.build
> get properly applied, the libs don't get built and then check the fails when
> it doesn't find them. I don't know whether the application of these flags is
> desirable (and we would need to fix the ABI check) or whether we should
> remove the flags.
Does the changes impact not only aarch32 but also aarch64?
>
> [0] https://travis-ci.com/github/jlinkes/dpdk/jobs/317622662#L2672
> [1] https://travis-ci.com/github/jlinkes/dpdk/jobs/317622661#L4488
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH v4] eal: add VFIO-PCI SR-IOV support
@ 2020-04-13 8:29 2% ` Haiyue Wang
2020-04-13 12:18 3% ` Thomas Monjalon
2020-04-13 15:37 0% ` Andrew Rybchenko
2020-04-14 3:06 4% ` [dpdk-dev] [PATCH v5 0/2] support for VFIO-PCI VF token interface Haiyue Wang
` (3 subsequent siblings)
4 siblings, 2 replies; 200+ results
From: Haiyue Wang @ 2020-04-13 8:29 UTC (permalink / raw)
To: dev, thomas, vattunuru, jerinj, alex.williamson, david.marchand
Cc: Haiyue Wang
The kernel module vfio-pci introduces the VF token to enable SR-IOV
support since 5.7.
The VF token can be set by a vfio-pci based PF driver and must be known
by the vfio-pci based VF driver in order to gain access to the device.
An example VF token option would take this form:
1. Install vfio-pci with option 'enable_sriov=1'
2. ./usertools/dpdk-devbind.py -b vfio-pci 0000:87:00.0
3. echo 2 > /sys/bus/pci/devices/0000:87:00.0/sriov_numvfs
4. Start the PF:
./x86_64-native-linux-gcc/app/testpmd -l 22-25 -n 4 \
-w 87:00.0,vf_token=2ab74924-c335-45f4-9b16-8569e5b08258 \
--file-prefix=pf -- -i
5. Start the VF:
./x86_64-native-linux-gcc/app/testpmd -l 26-29 -n 4 \
-w 87:02.0,vf_token=2ab74924-c335-45f4-9b16-8569e5b08258 \
--file-prefix=vf1 -- -i
Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
Acked-by: Vamsi Attunuru <vattunuru@marvell.com>
---
v4: 1. Ignore rte_vfio_setup_device ABI check since it is
for Linux driver use.
v3: https://patchwork.dpdk.org/patch/68254/
Fix the Travis build failed:
(1). rte_uuid.h:97:55: error: unknown type name ‘size_t’
(2). rte_uuid.h:58:2: error: implicit declaration of function ‘memcpy’
v2: https://patchwork.dpdk.org/patch/68240/
Fix the FreeBSD build error.
v1: https://patchwork.dpdk.org/patch/68237/
Update the commit message.
RFC v2: https://patchwork.dpdk.org/patch/68114/
Based on Vamsi's RFC v1, and Alex's patch for Qemu
[https://lore.kernel.org/lkml/20200204161737.34696b91@w520.home/]:
Use the devarg to pass-down the VF token.
RFC v1: https://patchwork.dpdk.org/patch/66281/ by Vamsi.
---
devtools/libabigail.abignore | 3 ++
drivers/bus/pci/linux/pci_vfio.c | 56 +++++++++++++++++++++++++++++--
lib/librte_eal/freebsd/eal.c | 3 +-
lib/librte_eal/include/rte_uuid.h | 2 ++
lib/librte_eal/include/rte_vfio.h | 8 ++++-
lib/librte_eal/linux/eal_vfio.c | 20 +++++++++--
6 files changed, 85 insertions(+), 7 deletions(-)
diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore
index a59df8f13..d918746b4 100644
--- a/devtools/libabigail.abignore
+++ b/devtools/libabigail.abignore
@@ -11,3 +11,6 @@
type_kind = enum
name = rte_crypto_asym_xform_type
changed_enumerators = RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
+; Ignore this function which is only relevant to linux for driver
+[suppress_type]
+ name = rte_vfio_setup_device
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index 64cd84a68..7f99337c7 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -11,6 +11,7 @@
#include <sys/mman.h>
#include <stdbool.h>
+#include <rte_devargs.h>
#include <rte_log.h>
#include <rte_pci.h>
#include <rte_bus_pci.h>
@@ -644,11 +645,59 @@ pci_vfio_msix_is_mappable(int vfio_dev_fd, int msix_region)
return ret;
}
+static void
+vfio_pci_vf_token_arg(struct rte_devargs *devargs, rte_uuid_t uu)
+{
+#define VF_TOKEN_ARG "vf_token="
+ char c, *p, *vf_token;
+
+ if (devargs == NULL)
+ return;
+
+ p = strstr(devargs->args, VF_TOKEN_ARG);
+ if (!p)
+ return;
+
+ vf_token = p + strlen(VF_TOKEN_ARG);
+ if (strlen(vf_token) < (RTE_UUID_STRLEN - 1))
+ return;
+
+ c = vf_token[RTE_UUID_STRLEN - 1];
+ if (c != '\0' && c != ',')
+ return;
+
+ vf_token[RTE_UUID_STRLEN - 1] = '\0';
+ if (rte_uuid_parse(vf_token, uu)) {
+ RTE_LOG(ERR, EAL,
+ "The VF token is not a valid uuid : %s\n", vf_token);
+ vf_token[RTE_UUID_STRLEN - 1] = c;
+ return;
+ }
+
+ RTE_LOG(DEBUG, EAL,
+ "The VF token is found : %s\n", vf_token);
+
+ vf_token[RTE_UUID_STRLEN - 1] = c;
+
+ /* Purge this vfio-pci specific token from the device arguments */
+ if (c != '\0') {
+ /* 1. Handle the case : 'vf_token=uuid,arg1=val1' */
+ memmove(p, vf_token + RTE_UUID_STRLEN,
+ strlen(vf_token + RTE_UUID_STRLEN) + 1);
+ } else {
+ /* 2. Handle the case : 'arg1=val1,vf_token=uuid' */
+ if (p != devargs->args)
+ p--;
+
+ *p = '\0';
+ }
+}
static int
pci_vfio_map_resource_primary(struct rte_pci_device *dev)
{
struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ rte_uuid_t vf_token = RTE_UUID_INIT(0, 0, 0, 0, 0ULL);
char pci_addr[PATH_MAX] = {0};
int vfio_dev_fd;
struct rte_pci_addr *loc = &dev->addr;
@@ -668,8 +717,9 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
loc->domain, loc->bus, loc->devid, loc->function);
+ vfio_pci_vf_token_arg(dev->device.devargs, vf_token);
ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd, &device_info, vf_token);
if (ret)
return ret;
@@ -797,6 +847,7 @@ static int
pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
{
struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ rte_uuid_t vf_token = RTE_UUID_INIT(0, 0, 0, 0, 0ULL);
char pci_addr[PATH_MAX] = {0};
int vfio_dev_fd;
struct rte_pci_addr *loc = &dev->addr;
@@ -830,8 +881,9 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
return -1;
}
+ vfio_pci_vf_token_arg(dev->device.devargs, vf_token);
ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd, &device_info, vf_token);
if (ret)
return ret;
diff --git a/lib/librte_eal/freebsd/eal.c b/lib/librte_eal/freebsd/eal.c
index 6ae37e7e6..a92584795 100644
--- a/lib/librte_eal/freebsd/eal.c
+++ b/lib/librte_eal/freebsd/eal.c
@@ -995,7 +995,8 @@ rte_eal_vfio_intr_mode(void)
int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int *vfio_dev_fd,
- __rte_unused struct vfio_device_info *device_info)
+ __rte_unused struct vfio_device_info *device_info,
+ __rte_unused rte_uuid_t vf_token)
{
return -1;
}
diff --git a/lib/librte_eal/include/rte_uuid.h b/lib/librte_eal/include/rte_uuid.h
index 044afbdfa..8b42e070a 100644
--- a/lib/librte_eal/include/rte_uuid.h
+++ b/lib/librte_eal/include/rte_uuid.h
@@ -15,6 +15,8 @@ extern "C" {
#endif
#include <stdbool.h>
+#include <stddef.h>
+#include <string.h>
/**
* Struct describing a Universal Unique Identifier
diff --git a/lib/librte_eal/include/rte_vfio.h b/lib/librte_eal/include/rte_vfio.h
index 20ed8c45a..1f9e22d82 100644
--- a/lib/librte_eal/include/rte_vfio.h
+++ b/lib/librte_eal/include/rte_vfio.h
@@ -16,6 +16,8 @@ extern "C" {
#include <stdint.h>
+#include <rte_uuid.h>
+
/*
* determine if VFIO is present on the system
*/
@@ -102,13 +104,17 @@ struct vfio_device_info;
* @param device_info
* Device information.
*
+ * @param vf_token
+ * VF token.
+ *
* @return
* 0 on success.
* <0 on failure.
* >1 if the device cannot be managed this way.
*/
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info);
+ int *vfio_dev_fd, struct vfio_device_info *device_info,
+ rte_uuid_t vf_token);
/**
* Release a device mapped to a VFIO-managed I/O MMU group.
diff --git a/lib/librte_eal/linux/eal_vfio.c b/lib/librte_eal/linux/eal_vfio.c
index 4502aefed..916082b5d 100644
--- a/lib/librte_eal/linux/eal_vfio.c
+++ b/lib/librte_eal/linux/eal_vfio.c
@@ -702,7 +702,8 @@ rte_vfio_clear_group(int vfio_group_fd)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info)
+ int *vfio_dev_fd, struct vfio_device_info *device_info,
+ rte_uuid_t vf_token)
{
struct vfio_group_status group_status = {
.argsz = sizeof(group_status)
@@ -712,6 +713,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int vfio_container_fd;
int vfio_group_fd;
int iommu_group_num;
+ char dev[PATH_MAX];
int i, ret;
/* get group number */
@@ -895,8 +897,19 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
t->type_id, t->name);
}
+ if (!rte_uuid_is_null(vf_token)) {
+ char vf_token_str[RTE_UUID_STRLEN];
+
+ rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
+ snprintf(dev, sizeof(dev),
+ "%s vf_token=%s", dev_addr, vf_token_str);
+ } else {
+ snprintf(dev, sizeof(dev),
+ "%s", dev_addr);
+ }
+
/* get a file descriptor for the device */
- *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
+ *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, dev);
if (*vfio_dev_fd < 0) {
/* if we cannot get a device fd, this implies a problem with
* the VFIO group or the container not having IOMMU configured.
@@ -2081,7 +2094,8 @@ int
rte_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int *vfio_dev_fd,
- __rte_unused struct vfio_device_info *device_info)
+ __rte_unused struct vfio_device_info *device_info,
+ __rte_unused rte_uuid_t vf_token)
{
return -1;
}
--
2.26.0
^ permalink raw reply [relevance 2%]
* Re: [dpdk-dev] [PATCH dpdk-dev v2 1/2] eal: introduce rte-init queue for libraries initialization
2020-04-12 3:32 0% ` Tonghao Zhang
@ 2020-04-13 11:32 0% ` Jerin Jacob
0 siblings, 0 replies; 200+ results
From: Jerin Jacob @ 2020-04-13 11:32 UTC (permalink / raw)
To: Tonghao Zhang
Cc: Olivier Matz, Andrew Rybchenko, Gage Eads, Artem V. Andreev,
Jerin Jacob, Nithin Dabilpuram, Vamsi Attunuru, Hemant Agrawal,
David Marchand, Anatoly Burakov, Richardson, Bruce, dpdk-dev
On Sun, Apr 12, 2020 at 9:03 AM Tonghao Zhang <xiangxia.m.yue@gmail.com> wrote:
>
> On Sun, Apr 12, 2020 at 11:20 AM Tonghao Zhang <xiangxia.m.yue@gmail.com> wrote:
> >
> > On Fri, Apr 10, 2020 at 9:11 PM Jerin Jacob <jerinjacobk@gmail.com> wrote:
> > >
> > > On Fri, Apr 10, 2020 at 11:48 AM Jerin Jacob <jerinjacobk@gmail.com> wrote:
> > > >
> > >
> > > Three more items are missing in this patch
> Can I add "Co-authored-by" for you ?
We are not using that tag in DPDK. Feel free to add, Suggested-by:
Jerin Jacob <jerinj@marvell.com>
>
> > > 1) Unit test case for new API
> > > 2) Make the new API __rte_experimal
> > Hi Jerin
> > This API will be invoked in mempool, if use that prefix, there is a
> > compile warning:
> > include/rte_mempool.h:933:2: error: ‘rte_init_register’ is deprecated
> > (declared at x86_64-native-linuxapp-gcc/include/rte_init.h:61): Symbol
> > is not yet part of stable ABI [-Werror=deprecated-declarations]
http://patches.dpdk.org/patch/68119/ will be applied soon. So it will
not a problem,
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v4] eal: add VFIO-PCI SR-IOV support
2020-04-13 8:29 2% ` [dpdk-dev] [PATCH v4] eal: add VFIO-PCI SR-IOV support Haiyue Wang
@ 2020-04-13 12:18 3% ` Thomas Monjalon
2020-04-13 17:01 3% ` Wang, Haiyue
2020-04-13 15:37 0% ` Andrew Rybchenko
1 sibling, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-04-13 12:18 UTC (permalink / raw)
To: Haiyue Wang; +Cc: dev, vattunuru, jerinj, alex.williamson, david.marchand
Hi,
About the title, I think it does not convey what is new here.
VFIO is not new, SR-IOV is already supported.
The title should mention the new VFIO feature in few simple words.
Is it only about using VFIO for PF?
13/04/2020 10:29, Haiyue Wang:
> v4: 1. Ignore rte_vfio_setup_device ABI check since it is
> for Linux driver use.
[...]
> +; Ignore this function which is only relevant to linux for driver
> +[suppress_type]
> + name = rte_vfio_setup_device
Adding such exception for all internal "driver interface" functions
is not scaling. Please use __rte_internal.
I am waiting for the patchset about rte_internal to be reviewed or completed.
As it is not progressing, the decision is to block any patch having
ABI issue because of internal false positive.
Please help, thanks.
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v3 3/9] ring: introduce RTS ring mode
2020-04-10 23:10 0% ` Honnappa Nagarahalli
@ 2020-04-13 14:29 0% ` David Marchand
0 siblings, 0 replies; 200+ results
From: David Marchand @ 2020-04-13 14:29 UTC (permalink / raw)
To: Honnappa Nagarahalli
Cc: Ananyev, Konstantin, dev, jielong.zjl, nd, Kinsella, Ray,
Thomas Monjalon, Jerin Jacob Kollanukkaran
On Sat, Apr 11, 2020 at 1:10 AM Honnappa Nagarahalli
<Honnappa.Nagarahalli@arm.com> wrote:
>
> <snip>
>
> > Subject: RE: [PATCH v3 3/9] ring: introduce RTS ring mode
> >
> > > > Introduce relaxed tail sync (RTS) mode for MT ring synchronization.
> > > > Aim to reduce stall times in case when ring is used on overcommited
> > > > cpus (multiple active threads on the same cpu).
> > > > The main difference from original MP/MC algorithm is that tail value
> > > > is increased not by every thread that finished enqueue/dequeue, but
> > > > only by the last one.
> > > > That allows threads to avoid spinning on ring tail value, leaving
> > > > actual tail value change to the last thread in the update queue.
> > > >
> > > > check-abi.sh reports what I believe is a false-positive about ring
> > > > cons/prod changes. As a workaround, devtools/libabigail.abignore is
> > > > updated to suppress *struct ring* related errors.
> > > This can be removed from the commit message.
> > >
> > > >
> > > > Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> > > > ---
> > > > devtools/libabigail.abignore | 7 +
> > > > lib/librte_ring/Makefile | 5 +-
> > > > lib/librte_ring/meson.build | 5 +-
> > > > lib/librte_ring/rte_ring.c | 100 +++++++-
> > > > lib/librte_ring/rte_ring.h | 110 ++++++++-
> > > > lib/librte_ring/rte_ring_elem.h | 86 ++++++-
> > > > lib/librte_ring/rte_ring_rts.h | 316 +++++++++++++++++++++++++
> > > > lib/librte_ring/rte_ring_rts_elem.h | 205 ++++++++++++++++
> > > > lib/librte_ring/rte_ring_rts_generic.h | 210 ++++++++++++++++
> > > > 9 files changed, 1015 insertions(+), 29 deletions(-) create mode
> > > > 100644 lib/librte_ring/rte_ring_rts.h create mode 100644
> > > > lib/librte_ring/rte_ring_rts_elem.h
> > > > create mode 100644 lib/librte_ring/rte_ring_rts_generic.h
> > > >
> > > > diff --git a/devtools/libabigail.abignore
> > > > b/devtools/libabigail.abignore index a59df8f13..cd86d89ca 100644
> > > > --- a/devtools/libabigail.abignore
> > > > +++ b/devtools/libabigail.abignore
> > > > @@ -11,3 +11,10 @@
> > > > type_kind = enum
> > > > name = rte_crypto_asym_xform_type
> > > > changed_enumerators =
> > RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
> > > > +; Ignore updates of ring prod/cons
> > > > +[suppress_type]
> > > > + type_kind = struct
> > > > + name = rte_ring
> > > > +[suppress_type]
> > > > + type_kind = struct
> > > > + name = rte_event_ring
> > > Does this block the reporting of these structures forever?
> >
> > Till we'll have a fix in libabigail, then we can remove these lines.
> > I don't know any better alternative.
> David, does this block all issues in the future for rte_ring library?
These two "suppression rules" make libabigail consider as harmless any
change on the structures rte_ring and rte_event_ring.
With those suppression rules, you won't get any complaint from
libabigail (if this is what you call issues :-)).
Reviews on those structures must be extra careful, as we are blind
with those rules in place.
--
David Marchand
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH v5 00/33] DPDK Trace support
2020-04-03 15:36 1% ` [dpdk-dev] [PATCH v4 " jerinj
@ 2020-04-13 15:00 1% ` jerinj
0 siblings, 0 replies; 200+ results
From: jerinj @ 2020-04-13 15:00 UTC (permalink / raw)
Cc: dev, thomas, bruce.richardson, david.marchand, mattias.ronnblom,
skori, Jerin Jacob
From: Jerin Jacob <jerinj@marvell.com>
This patch depends on http://patches.dpdk.org/patch/68119/
Depends-on:series-9295
v5
~~
Following API rework based David and Thomas feedback.
1) Rename - "Shell pattern" to "Globbing pattern"
2) Remove the log "level" notion from trace library.
3) Remove rte_trace_global_[level/mode]* API from trace library.
4) EAL command line arg to --trace=regex/globing from --trace-level=regex/globing:level
5) Remove "@b EXPERIMENTAL: this API may change without prior notice" from each functions.
6) Use FP instead of DP for fastpath reference.
7) Updated documentation to reflect the above rework.
8) Updated UT to to reflect the above rework.
8) Updated UT for a FP trace point emit.
9) Updated performance test case for a FP trace point emission.
10) Updated git commit comments to reflect the above rework.
v4:
~~
1) Rebased to master.
2) Adapted to latest EAL directory structure change.
3) Fix possible build issue with out of tree application wherein
it does not define -DALLOW_EXPERIMENTAL_API. Fixed by making
fast path trace functions as NOP as it was getting included
in the inline functions of ethdev,mempool, cryptodev, etc(David)
4) Removed DALLOW_EXPERIMENTAL_API definition from individual driver files.
Now it set as global using http://patches.dpdk.org/patch/67758/ patch (David)
5) Added new meson option(-Denable_trace_dp=true)for enabling the datapath trace point (David, Bruce)
6) Changed the authorship and Rewrote the programmer's guide based on the below feedback(Thomas)
http://patches.dpdk.org/patch/67352/
v3:
~~
1) Fix the following build issues reported by CI
http://mails.dpdk.org/archives/test-report/2020-March/122060.html
a) clang + i686 meson build issue(Fixed in the the patch meson: add libatomic as a global dependency for i686 clang)
b) fixed build issue with FreeBSD with top of tree change.
c) fixed missing experimental API for iavf and ice with meson build in avx512 files.
v2:
~~
Addressed the following review comments from Mattias Rönnblom:
1) Changed
from:
typedef uint64_t* rte_trace_t;
to
typedef uint64_t rte_trace_t;
Initially thought to make the handle as
struct rte_trace {
uint64_t val;
}
but changed to uint64_t for the following reasons
a) It is opaque to the application and it will fix the compile-time type
check as well.
b) The handle has an index that will point to an internal slow-path
structure so no ABI change required in the future.
c) RTE_TRACE_POINT_DEFINE need to expose trace object. So it is better
to keep as uint64_t and avoid one more indirection for no use.
2)
Changed:
from:
enum rte_trace_mode_e {
to:
enum rte_trace_mode {
3) removed [out] "found" param from rte_trace_pattern() and
rte_trace_regexp()
4) Changed rte_trace_from_name to rte_trace_by_name
5) rte_trace_is_dp_enabled() return bool now
6) in __rte_trace_point_register() the argument fn change to register_fn
7) removed !! from rte_trace_is_enabled()
8) Remove uninitialized "rc warning" from rte_trace_pattern() and
rte_trace_regexp()
9) fixup bool return type for trace_entry_compare()
10) fixup calloc casting in trace_mkdir()
11) check fclose() return in trace_meta_save() and trace_mem_save()
12) rte_trace_ctf_* macro cleanup
13) added release notes
14) fix build issues reported by CI
http://mails.dpdk.org/archives/test-report/2020-March/121235.html
This patch set contains
~~~~~~~~~~~~~~~~~~~~~~~~
# The native implementation of common trace format(CTF)[1] based tracer
# Public API to create the trace points.
# Add tracepoints to eal, ethdev, mempool, eventdev and cryptodev
library for tracing support
# A unit test case
# Performance test case to measure the trace overhead. (See eal/trace:
# add trace performance test cases, patch)
# Programmers guide for Trace support(See doc: add trace library guide,
# patch)
# Tested OS:
~~~~~~~~~~~
- Linux
- FreeBSD
# Tested open source CTF trace viewers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Babeltrace
- Tracecompass
# Trace overhead comparison with LTTng
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
trace overhead data on x86:[2]
# 236 cycles with LTTng(>100ns)
# 18 cycles(7ns) with Native DPDK CTF emitter.(See eal/trace: add trace
# performance test cases patch)
trace overhead data on arm64:
# 312 cycles to 1100 cycles with LTTng based on the class of arm64
# CPU.
# 11 cycles to 13 cycles with Native DPDK CTF emitter based on the
class of arm64 CPU.
18 cycles(on x86) vs 11 cycles(on arm64) is due to rdtsc() overhead in
x86. It seems rdtsc takes around 15cycles in x86.
More details:
~~~~~~~~~~~~~
# The Native DPDK CTF trace support does not have any dependency on
third-party library.
The generated output file is compatible with LTTng as both are using
CTF trace format.
The performance gain comes from:
1) exploit dpdk worker thread usage model to avoid atomics and use per
core variables
2) use hugepage,
3) avoid a lot function pointers in fast-path etc
4) avoid unaligned store for arm64 etc
Features:
~~~~~~~~~
- No specific limit on the events. A string-based event like rte_log
for pattern matching
- Dynamic enable/disable support.
- Instructmention overhead is ~1 cycle. i.e cost of adding the code
wth out using trace feature.
- Timestamp support for all the events using DPDK rte_rtdsc
- No dependency on another library. Clean room native implementation of
CTF.
Functional test case:
a) echo "trace_autotest" | sudo ./build/app/test/dpdk-test -c 0x3 --trace=.*
The above command emits the following trace events
<code>
uint8_t i;
rte_trace_lib_eal_generic_void();
rte_trace_lib_eal_generic_u64(0x10000000000000);
rte_trace_lib_eal_generic_u32(0x10000000);
rte_trace_lib_eal_generic_u16(0xffee);
rte_trace_lib_eal_generic_u8(0xc);
rte_trace_lib_eal_generic_i64(-1234);
rte_trace_lib_eal_generic_i32(-1234567);
rte_trace_lib_eal_generic_i16(12);
rte_trace_lib_eal_generic_i8(-3);
rte_trace_lib_eal_generic_string("my string");
rte_trace_lib_eal_generic_function(__func__);
</code>
Install babeltrace package in Linux and point the generated trace file
to babel trace. By default trace file created under
<user>/dpdk-traces/time_stamp/
example:
# babeltrace /root/dpdk-traces/rte-2020-02-15-PM-02-56-51 | more
[13:27:36.138468807] (+?.?????????) lib.eal.generic.void: { cpu_id =0, name = "dpdk-test" }, { }
[13:27:36.138468851] (+0.000000044) lib.eal.generic.u64: { cpu_id = 0, name = "dpdk-test" }, { in = 4503599627370496 }
[13:27:36.138468860] (+0.000000009) lib.eal.generic.u32: { cpu_id = 0, name = "dpdk-test" }, { in = 268435456 }
[13:27:36.138468934] (+0.000000074) lib.eal.generic.u16: { cpu_id = 0, name = "dpdk-test" }, { in = 65518 }
[13:27:36.138468949] (+0.000000015) lib.eal.generic.u8: { cpu_id = 0, name = "dpdk-test" }, { in = 12 }
[13:27:36.138468956] (+0.000000007) lib.eal.generic.i64: { cpu_id = 0, name = "dpdk-test" }, { in = -1234 }
[13:27:36.138468963] (+0.000000007) lib.eal.generic.i32: { cpu_id = 0, name = "dpdk-test" }, { in = -1234567 }
[13:27:36.138469024] (+0.000000061) lib.eal.generic.i16: { cpu_id = 0, name = "dpdk-test" }, { in = 12 }
[13:27:36.138469044] (+0.000000020) lib.eal.generic.i8: { cpu_id = 0, name = "dpdk-test" }, { in = -3 }
[13:27:36.138469051] (+0.000000007) lib.eal.generic.string: { cpu_id = 0, name = "dpdk-test" }, { str = "my string" }
[13:27:36.138469203] (+0.000000152) lib.eal.generic.func: { cpu_id = 0, name = "dpdk-test" }, { func = "test_trace_points" }
# There is a GUI based trace viewer available in Windows, Linux and Mac.
It is called as tracecompass.(https://www.eclipse.org/tracecompass/)
The example screenshot and Histogram of above DPDK trace using
Tracecompass.
https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG
File walk through:
~~~~~~~~~~~~~~~~~~
lib/librte_eal/common/include/rte_trace.h - Public API for Trace
provider and Trace control
lib/librte_eal/common/eal_common_trace.c - main trace implementation
lib/librte_eal/common/eal_common_trace_ctf.c - CTF metadata spec
implementation
lib/librte_eal/common/eal_common_trace_utils.c - command line utils
and filesystem operations.
lib/librte_eal/common/eal_common_trace_points.c - trace points for EAL
library
lib/librte_eal/common/include/rte_trace_eal.h - EAL tracepoint public
API.
lib/librte_eal/common/eal_trace.h - Private trace header file.
[1] https://diamon.org/ctf/
[2] The above test is ported to LTTng for finding the LTTng trace
overhead. It available at
https://github.com/jerinjacobk/lttng-overhead
https://github.com/jerinjacobk/lttng-overhead/blob/master/README
Jerin Jacob (22):
eal: introduce API for getting thread name
eal/trace: define the public API for trace support
eal/trace: implement trace register API
eal/trace: implement trace operation APIs
eal/trace: add internal trace init and fini interface
eal/trace: get bootup timestamp for trace
eal/trace: create CTF TDSL metadata in memory
eal/trace: implement trace memory allocation
eal/trace: implement debug dump function
eal/trace: implement trace save
eal/trace: implement registration payload
eal/trace: implement provider payload
eal/trace: hook internal trace APIs to Linux
eal/trace: hook internal trace APIs to FreeBSD
eal/trace: add generic tracepoints
eal/trace: add alarm tracepoints
eal/trace: add memory tracepoints
eal/trace: add memzone tracepoints
eal/trace: add thread tracepoints
eal/trace: add interrupt tracepoints
eal/trace: add trace performance test cases
doc: add trace library guide
Pavan Nikhilesh (1):
meson: add libatomic as a global dependency for i686 clang
Sunil Kumar Kori (10):
eal/trace: handle CTF keyword collision
eal/trace: add trace configuration parameter
eal/trace: add trace dir configuration parameter
eal/trace: add trace bufsize configuration parameter
eal/trace: add trace mode configuration parameter
eal/trace: add unit test cases
ethdev: add tracepoints
eventdev: add tracepoints
cryptodev: add tracepoints
mempool: add tracepoints
MAINTAINERS | 8 +
app/test/Makefile | 4 +-
app/test/meson.build | 3 +
app/test/test_trace.c | 258 +++++++++
app/test/test_trace.h | 15 +
app/test/test_trace_perf.c | 183 +++++++
app/test/test_trace_register.c | 19 +
config/common_base | 1 +
config/meson.build | 9 +
doc/api/doxy-api-index.md | 3 +-
doc/guides/linux_gsg/eal_args.include.rst | 52 ++
doc/guides/prog_guide/build-sdk-meson.rst | 5 +
doc/guides/prog_guide/index.rst | 1 +
doc/guides/prog_guide/trace_lib.rst | 334 +++++++++++
doc/guides/rel_notes/release_20_05.rst | 9 +
drivers/event/octeontx/meson.build | 5 -
drivers/event/octeontx2/meson.build | 5 -
drivers/event/opdl/meson.build | 5 -
examples/cmdline/Makefile | 1 +
examples/cmdline/meson.build | 1 +
examples/distributor/Makefile | 1 +
examples/distributor/meson.build | 1 +
examples/ethtool/ethtool-app/Makefile | 1 +
examples/eventdev_pipeline/meson.build | 1 +
examples/flow_filtering/Makefile | 1 +
examples/flow_filtering/meson.build | 1 +
examples/helloworld/Makefile | 1 +
examples/helloworld/meson.build | 1 +
examples/ioat/Makefile | 1 +
examples/ioat/meson.build | 1 +
examples/ip_fragmentation/Makefile | 2 +
examples/ip_fragmentation/meson.build | 1 +
examples/ip_reassembly/Makefile | 1 +
examples/ip_reassembly/meson.build | 1 +
examples/ipv4_multicast/Makefile | 1 +
examples/ipv4_multicast/meson.build | 1 +
examples/l2fwd-cat/Makefile | 1 +
examples/l2fwd-cat/meson.build | 1 +
examples/l2fwd-event/Makefile | 1 +
examples/l2fwd-event/meson.build | 6 +-
examples/l2fwd-jobstats/Makefile | 1 +
examples/l2fwd-jobstats/meson.build | 1 +
examples/l2fwd-keepalive/Makefile | 1 +
examples/l2fwd-keepalive/ka-agent/Makefile | 1 +
examples/l2fwd-keepalive/meson.build | 1 +
examples/l3fwd-acl/Makefile | 1 +
examples/l3fwd-acl/meson.build | 1 +
examples/l3fwd/Makefile | 1 +
examples/l3fwd/meson.build | 1 +
examples/link_status_interrupt/Makefile | 1 +
examples/link_status_interrupt/meson.build | 1 +
.../client_server_mp/mp_client/Makefile | 1 +
.../client_server_mp/mp_client/meson.build | 1 +
.../client_server_mp/mp_server/meson.build | 1 +
examples/multi_process/hotplug_mp/Makefile | 1 +
examples/multi_process/hotplug_mp/meson.build | 1 +
examples/multi_process/simple_mp/Makefile | 1 +
examples/multi_process/simple_mp/meson.build | 1 +
examples/multi_process/symmetric_mp/Makefile | 1 +
.../multi_process/symmetric_mp/meson.build | 1 +
examples/ntb/Makefile | 1 +
examples/ntb/meson.build | 1 +
examples/packet_ordering/Makefile | 1 +
examples/packet_ordering/meson.build | 1 +
.../performance-thread/l3fwd-thread/Makefile | 1 +
.../l3fwd-thread/meson.build | 1 +
.../performance-thread/pthread_shim/Makefile | 1 +
.../pthread_shim/meson.build | 1 +
examples/ptpclient/Makefile | 1 +
examples/ptpclient/meson.build | 1 +
examples/qos_meter/Makefile | 1 +
examples/qos_meter/meson.build | 1 +
examples/qos_sched/Makefile | 1 +
examples/qos_sched/meson.build | 1 +
examples/server_node_efd/node/Makefile | 1 +
examples/server_node_efd/node/meson.build | 1 +
examples/server_node_efd/server/Makefile | 1 +
examples/server_node_efd/server/meson.build | 1 +
examples/service_cores/Makefile | 1 +
examples/service_cores/meson.build | 1 +
examples/skeleton/Makefile | 1 +
examples/skeleton/meson.build | 1 +
examples/timer/Makefile | 1 +
examples/timer/meson.build | 1 +
examples/vm_power_manager/Makefile | 1 +
examples/vm_power_manager/meson.build | 1 +
examples/vmdq/Makefile | 1 +
examples/vmdq/meson.build | 1 +
examples/vmdq_dcb/Makefile | 1 +
examples/vmdq_dcb/meson.build | 1 +
lib/librte_cryptodev/Makefile | 4 +-
lib/librte_cryptodev/cryptodev_trace_points.c | 70 +++
lib/librte_cryptodev/meson.build | 6 +-
lib/librte_cryptodev/rte_cryptodev.c | 18 +
lib/librte_cryptodev/rte_cryptodev.h | 6 +
.../rte_cryptodev_version.map | 18 +
lib/librte_cryptodev/rte_trace_cryptodev.h | 133 +++++
lib/librte_cryptodev/rte_trace_cryptodev_fp.h | 34 ++
lib/librte_distributor/meson.build | 5 -
lib/librte_eal/common/eal_common_memzone.c | 9 +
lib/librte_eal/common/eal_common_options.c | 65 +++
lib/librte_eal/common/eal_common_thread.c | 3 +-
lib/librte_eal/common/eal_common_trace.c | 518 ++++++++++++++++++
lib/librte_eal/common/eal_common_trace_ctf.c | 488 +++++++++++++++++
.../common/eal_common_trace_points.c | 115 ++++
.../common/eal_common_trace_utils.c | 478 ++++++++++++++++
lib/librte_eal/common/eal_options.h | 8 +
lib/librte_eal/common/eal_private.h | 5 +
lib/librte_eal/common/eal_trace.h | 121 ++++
lib/librte_eal/common/meson.build | 4 +
lib/librte_eal/common/rte_malloc.c | 60 +-
lib/librte_eal/freebsd/Makefile | 4 +
lib/librte_eal/freebsd/eal.c | 10 +
lib/librte_eal/freebsd/eal_alarm.c | 3 +
lib/librte_eal/freebsd/eal_interrupts.c | 54 +-
lib/librte_eal/freebsd/eal_thread.c | 21 +-
lib/librte_eal/include/meson.build | 4 +
lib/librte_eal/include/rte_lcore.h | 17 +
lib/librte_eal/include/rte_trace.h | 424 ++++++++++++++
lib/librte_eal/include/rte_trace_eal.h | 247 +++++++++
lib/librte_eal/include/rte_trace_provider.h | 157 ++++++
lib/librte_eal/include/rte_trace_register.h | 52 ++
lib/librte_eal/linux/Makefile | 4 +
lib/librte_eal/linux/eal.c | 9 +
lib/librte_eal/linux/eal_alarm.c | 4 +
lib/librte_eal/linux/eal_interrupts.c | 84 +--
lib/librte_eal/linux/eal_thread.c | 27 +-
lib/librte_eal/rte_eal_version.map | 53 ++
lib/librte_ethdev/Makefile | 3 +
lib/librte_ethdev/ethdev_trace_points.c | 43 ++
lib/librte_ethdev/meson.build | 5 +-
lib/librte_ethdev/rte_ethdev.c | 12 +
lib/librte_ethdev/rte_ethdev.h | 5 +
lib/librte_ethdev/rte_ethdev_version.map | 10 +
lib/librte_ethdev/rte_trace_ethdev.h | 90 +++
lib/librte_ethdev/rte_trace_ethdev_fp.h | 40 ++
lib/librte_eventdev/Makefile | 3 +
lib/librte_eventdev/eventdev_trace_points.c | 173 ++++++
lib/librte_eventdev/meson.build | 3 +
.../rte_event_crypto_adapter.c | 10 +
.../rte_event_eth_rx_adapter.c | 11 +
.../rte_event_eth_tx_adapter.c | 13 +-
.../rte_event_eth_tx_adapter.h | 2 +
lib/librte_eventdev/rte_event_timer_adapter.c | 8 +-
lib/librte_eventdev/rte_event_timer_adapter.h | 8 +
lib/librte_eventdev/rte_eventdev.c | 9 +
lib/librte_eventdev/rte_eventdev.h | 5 +-
lib/librte_eventdev/rte_eventdev_version.map | 42 ++
lib/librte_eventdev/rte_trace_eventdev.h | 278 ++++++++++
lib/librte_eventdev/rte_trace_eventdev_fp.h | 75 +++
lib/librte_mempool/Makefile | 3 +
lib/librte_mempool/mempool_trace_points.c | 108 ++++
lib/librte_mempool/meson.build | 5 +-
lib/librte_mempool/rte_mempool.c | 16 +
lib/librte_mempool/rte_mempool.h | 13 +
lib/librte_mempool/rte_mempool_ops.c | 7 +
lib/librte_mempool/rte_mempool_version.map | 26 +
lib/librte_mempool/rte_trace_mempool.h | 148 +++++
lib/librte_mempool/rte_trace_mempool_fp.h | 102 ++++
lib/librte_rcu/meson.build | 5 -
meson_options.txt | 2 +
161 files changed, 5517 insertions(+), 105 deletions(-)
create mode 100644 app/test/test_trace.c
create mode 100644 app/test/test_trace.h
create mode 100644 app/test/test_trace_perf.c
create mode 100644 app/test/test_trace_register.c
create mode 100644 doc/guides/prog_guide/trace_lib.rst
create mode 100644 lib/librte_cryptodev/cryptodev_trace_points.c
create mode 100644 lib/librte_cryptodev/rte_trace_cryptodev.h
create mode 100644 lib/librte_cryptodev/rte_trace_cryptodev_fp.h
create mode 100644 lib/librte_eal/common/eal_common_trace.c
create mode 100644 lib/librte_eal/common/eal_common_trace_ctf.c
create mode 100644 lib/librte_eal/common/eal_common_trace_points.c
create mode 100644 lib/librte_eal/common/eal_common_trace_utils.c
create mode 100644 lib/librte_eal/common/eal_trace.h
create mode 100644 lib/librte_eal/include/rte_trace.h
create mode 100644 lib/librte_eal/include/rte_trace_eal.h
create mode 100644 lib/librte_eal/include/rte_trace_provider.h
create mode 100644 lib/librte_eal/include/rte_trace_register.h
create mode 100644 lib/librte_ethdev/ethdev_trace_points.c
create mode 100644 lib/librte_ethdev/rte_trace_ethdev.h
create mode 100644 lib/librte_ethdev/rte_trace_ethdev_fp.h
create mode 100644 lib/librte_eventdev/eventdev_trace_points.c
create mode 100644 lib/librte_eventdev/rte_trace_eventdev.h
create mode 100644 lib/librte_eventdev/rte_trace_eventdev_fp.h
create mode 100644 lib/librte_mempool/mempool_trace_points.c
create mode 100644 lib/librte_mempool/rte_trace_mempool.h
create mode 100644 lib/librte_mempool/rte_trace_mempool_fp.h
--
2.25.1
^ permalink raw reply [relevance 1%]
* [dpdk-dev] [PATCH dpdk-dev v3 1/2] eal: introduce rte-init queue for libraries initialization
@ 2020-04-13 14:21 3% ` xiangxia.m.yue
1 sibling, 0 replies; 200+ results
From: xiangxia.m.yue @ 2020-04-13 14:21 UTC (permalink / raw)
To: olivier.matz, arybchenko, gage.eads, artem.andreev, jerinj,
ndabilpuram, vattunuru, hemant.agrawal, david.marchand,
anatoly.burakov, bruce.richardson
Cc: dev, Tonghao Zhang
From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
This patch introduces rte-init queue, users can register a callback
for theirs initialization. After the almost resource of DPDK are
available (e.g. memzone, ring), invoke eal_rte_init_run(). With this
way, users don't introduce additional codes in the eal layer.
Suggested-by: Olivier Matz <olivier.matz@6wind.com>
Suggested-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Tonghao Zhang <xiangxia.m.yue@gmail.com>
---
This patch will be used for next patch.
and should be applied after http://patches.dpdk.org/patch/68119/
because of __rte_experimal
v3:
* add rte-init autotest
* add __rte_experimal prefix
* change return type of rte_init_register to int
* update doc/api/doxy-api-index.md
* update rte_eal_version.map
* update comments
* remove unused *.h in rte_init.h
v2:
* rename rte_last_init_register ->rte_init_register
* rename rte_last_init struct ->rte_init
* rename rte_init_cb ->rte_init_cb_t
* free the rte_init node when not used
* remove rte_init and others to eal_private.h
* add comments
* fix checkpatch warning
---
app/test/Makefile | 1 +
app/test/autotest_data.py | 6 +++
app/test/meson.build | 1 +
app/test/test_rte_init.c | 35 +++++++++++++
doc/api/doxy-api-index.md | 1 +
lib/librte_eal/common/eal_common_init.c | 87 +++++++++++++++++++++++++++++++++
lib/librte_eal/common/eal_private.h | 23 +++++++++
lib/librte_eal/common/meson.build | 1 +
lib/librte_eal/freebsd/Makefile | 1 +
lib/librte_eal/freebsd/eal.c | 6 +++
lib/librte_eal/include/rte_init.h | 68 ++++++++++++++++++++++++++
lib/librte_eal/linux/Makefile | 1 +
lib/librte_eal/linux/eal.c | 6 +++
lib/librte_eal/rte_eal_version.map | 1 +
14 files changed, 238 insertions(+)
create mode 100644 app/test/test_rte_init.c
create mode 100644 lib/librte_eal/common/eal_common_init.c
create mode 100644 lib/librte_eal/include/rte_init.h
diff --git a/app/test/Makefile b/app/test/Makefile
index 1f080d162659..494606ad4226 100644
--- a/app/test/Makefile
+++ b/app/test/Makefile
@@ -148,6 +148,7 @@ SRCS-y += test_alarm.c
SRCS-y += test_interrupts.c
SRCS-y += test_version.c
SRCS-y += test_func_reentrancy.c
+SRCS-y += test_rte_init.c
SRCS-y += test_service_cores.c
diff --git a/app/test/autotest_data.py b/app/test/autotest_data.py
index 7b1d01389be4..d06c27d68b4a 100644
--- a/app/test/autotest_data.py
+++ b/app/test/autotest_data.py
@@ -267,6 +267,12 @@
"Report": None,
},
{
+ "Name": "RTE INIT autotest",
+ "Command": "rte_init_autotest",
+ "Func": default_autotest,
+ "Report": None,
+ },
+ {
"Name": "Atomics autotest",
"Command": "atomic_autotest",
"Func": default_autotest,
diff --git a/app/test/meson.build b/app/test/meson.build
index 351d29cb657d..98a952a9285d 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -79,6 +79,7 @@ test_sources = files('commands.c',
'test_mempool.c',
'test_mempool_perf.c',
'test_memzone.c',
+ 'test_rte_init.c',
'test_meter.c',
'test_metrics.c',
'test_mcslock.c',
diff --git a/app/test/test_rte_init.c b/app/test/test_rte_init.c
new file mode 100644
index 000000000000..f6143bc76310
--- /dev/null
+++ b/app/test/test_rte_init.c
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 DPDK Community
+ */
+
+#include <stdio.h>
+
+#include <rte_init.h>
+
+#include "test.h"
+
+static int
+rte_init_cb(__rte_unused const void *arg)
+{
+ return 0;
+}
+
+static int
+test_rte_init(void)
+{
+ printf("test rte-init register API\n");
+ if (rte_init_register(rte_init_cb, NULL, RTE_INIT_PRE) != 0)
+ return -1;
+
+ printf("test rte-init cb\n");
+ if (rte_init_register(NULL, NULL, RTE_INIT_PRE) != -EINVAL)
+ return -1;
+
+ printf("test rte-init type\n");
+ if (rte_init_register(NULL, NULL, 10) != -EINVAL)
+ return -1;
+
+ return 0;
+}
+
+REGISTER_TEST_COMMAND(rte_init_autotest, test_rte_init);
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index dff496be0980..fa02c5f9f287 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -178,6 +178,7 @@ The public API headers are grouped by topics:
- **misc**:
[EAL config] (@ref rte_eal.h),
+ [RTE INIT] (@ref rte_init.h),
[common] (@ref rte_common.h),
[experimental APIs] (@ref rte_compat.h),
[ABI versioning] (@ref rte_function_versioning.h),
diff --git a/lib/librte_eal/common/eal_common_init.c b/lib/librte_eal/common/eal_common_init.c
new file mode 100644
index 000000000000..f5ae5c51b667
--- /dev/null
+++ b/lib/librte_eal/common/eal_common_init.c
@@ -0,0 +1,87 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 DPDK Community
+ */
+
+#include <sys/queue.h>
+
+#include <rte_init.h>
+#include <rte_tailq.h>
+#include <rte_log.h>
+
+#include "eal_private.h"
+
+static struct rte_init_list rte_init_list =
+ TAILQ_HEAD_INITIALIZER(rte_init_list);
+
+int
+rte_init_register(rte_init_cb_t cb, const void *arg, enum rte_init_type type)
+{
+ struct rte_init *last;
+
+ if (cb == NULL) {
+ RTE_LOG(ERR, EAL, "RTE INIT cb is NULL.\n");
+ return -EINVAL;
+ }
+
+ if (type != RTE_INIT_PRE && type != RTE_INIT_POST) {
+ RTE_LOG(ERR, EAL, "RTE INIT type is invalid.\n");
+ return -EINVAL;
+ }
+
+ last = malloc(sizeof(*last));
+ if (last == NULL) {
+ RTE_LOG(ERR, EAL,
+ "Allocate memory for rte_init node failed.\n");
+ return -ENOMEM;
+ }
+
+ last->type = type;
+ last->arg = arg;
+ last->cb = cb;
+
+ TAILQ_INSERT_TAIL(&rte_init_list, last, next);
+
+ return 0;
+}
+
+static int
+eal_rte_init_run_type(enum rte_init_type type)
+{
+ struct rte_init *last;
+ int ret;
+
+ TAILQ_FOREACH(last, &rte_init_list, next) {
+ if (last->type != type)
+ continue;
+
+ ret = last->cb(last->arg);
+ if (ret < 0)
+ return ret;
+ }
+
+ return 0;
+}
+
+int
+eal_rte_init_run(void)
+{
+ struct rte_init *last;
+ struct rte_init *tmp;
+ int ret;
+
+ ret = eal_rte_init_run_type(RTE_INIT_PRE);
+ if (ret < 0)
+ return ret;
+
+ ret = eal_rte_init_run_type(RTE_INIT_POST);
+ if (ret < 0)
+ return ret;
+
+ /* Free rte_init node, not used anymore. */
+ TAILQ_FOREACH_SAFE(last, &rte_init_list, next, tmp) {
+ TAILQ_REMOVE(&rte_init_list, last, next);
+ free(last);
+ }
+
+ return 0;
+}
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index ddcfbe2e4442..f0bcc977eb9d 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -11,6 +11,7 @@
#include <rte_dev.h>
#include <rte_lcore.h>
+#include <rte_init.h>
/**
* Structure storing internal configuration (per-lcore)
@@ -60,6 +61,28 @@ struct rte_config {
} __attribute__((__packed__));
/**
+ * A structure describing a generic initialization.
+ */
+struct rte_init {
+ TAILQ_ENTRY(rte_init) next;
+ enum rte_init_type type;
+ rte_init_cb_t cb;
+ const void *arg;
+};
+
+/** Double linked list of rte_init. */
+TAILQ_HEAD(rte_init_list, rte_init);
+
+/**
+ * Run the callback registered in the global double linked list.
+ *
+ * @return
+ * - 0 on success
+ * - Negative on error
+ */
+int eal_rte_init_run(void);
+
+/**
* Get the global configuration structure.
*
* @return
diff --git a/lib/librte_eal/common/meson.build b/lib/librte_eal/common/meson.build
index 02d9280cc351..ad0ce6a19015 100644
--- a/lib/librte_eal/common/meson.build
+++ b/lib/librte_eal/common/meson.build
@@ -43,6 +43,7 @@ sources += files(
'eal_common_thread.c',
'eal_common_timer.c',
'eal_common_uuid.c',
+ 'eal_common_init.c',
'hotplug_mp.c',
'malloc_elem.c',
'malloc_heap.c',
diff --git a/lib/librte_eal/freebsd/Makefile b/lib/librte_eal/freebsd/Makefile
index e5d4d8ff260b..89c5649f16e6 100644
--- a/lib/librte_eal/freebsd/Makefile
+++ b/lib/librte_eal/freebsd/Makefile
@@ -60,6 +60,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_thread.c
SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_proc.c
SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_fbarray.c
SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_uuid.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_common_init.c
SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_malloc.c
SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += hotplug_mp.c
SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += malloc_elem.c
diff --git a/lib/librte_eal/freebsd/eal.c b/lib/librte_eal/freebsd/eal.c
index 6ae37e7e69be..63af98cb6ca7 100644
--- a/lib/librte_eal/freebsd/eal.c
+++ b/lib/librte_eal/freebsd/eal.c
@@ -874,6 +874,12 @@ static void rte_eal_init_alert(const char *msg)
eal_check_mem_on_local_socket();
+ if (eal_rte_init_run() < 0) {
+ rte_eal_init_alert("Cannot init objects in rte-init queue");
+ rte_errno = EFAULT;
+ return -1;
+ }
+
eal_thread_init_master(rte_config.master_lcore);
ret = eal_thread_dump_affinity(cpuset, sizeof(cpuset));
diff --git a/lib/librte_eal/include/rte_init.h b/lib/librte_eal/include/rte_init.h
new file mode 100644
index 000000000000..83c7fd47daec
--- /dev/null
+++ b/lib/librte_eal/include/rte_init.h
@@ -0,0 +1,68 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 DPDK Community
+ */
+
+#ifndef _RTE_INIT_H_
+#define _RTE_INIT_H_
+
+#include <rte_compat.h>
+
+/**
+ * @file
+ *
+ * RTE INIT Registration Interface
+ *
+ * This file exposes API for other libraries initialization callback.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Implementation specific callback function which is
+ * responsible for specificed initialization.
+ *
+ * This is invoked when almost resources are available.
+ *
+ * @return
+ * 0 for successfully invoked
+ * Negative for unsuccessfully invoked with error value
+ */
+typedef int (*rte_init_cb_t)(const void *arg);
+
+/**
+ * The RTE INIT type of callback function registered.
+ */
+enum rte_init_type {
+ RTE_INIT_PRE, /**< RTE INITs are invoked with high priority. */
+ RTE_INIT_POST /**< Last RTE INITs invoked. */
+};
+
+/**
+ * Register a rte_init callback.
+ *
+ * @param cb
+ * A pointer to a rte_init_cb structure, which will be invoked
+ * in rte_eal_init().
+ *
+ * @param arg
+ * The cb will use that as param.
+ *
+ * @param type
+ * The type of rte_init registered.
+ *
+ * @return
+ * 0 for success register callback.
+ * -EINVAL one of the parameters was invalid.
+ * -ENOMEM no appropriate memory allocated.
+ */
+__rte_experimental
+int rte_init_register(rte_init_cb_t cb, const void *arg,
+ enum rte_init_type type);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_INIT_H_ */
diff --git a/lib/librte_eal/linux/Makefile b/lib/librte_eal/linux/Makefile
index e5f44959c66c..918d94bec91e 100644
--- a/lib/librte_eal/linux/Makefile
+++ b/lib/librte_eal/linux/Makefile
@@ -67,6 +67,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_thread.c
SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_proc.c
SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_fbarray.c
SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_uuid.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_init.c
SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_malloc.c
SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += hotplug_mp.c
SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += malloc_elem.c
diff --git a/lib/librte_eal/linux/eal.c b/lib/librte_eal/linux/eal.c
index 9530ee55f8e3..dd0c2589f6c4 100644
--- a/lib/librte_eal/linux/eal.c
+++ b/lib/librte_eal/linux/eal.c
@@ -1203,6 +1203,12 @@ static void rte_eal_init_alert(const char *msg)
eal_check_mem_on_local_socket();
+ if (eal_rte_init_run() < 0) {
+ rte_eal_init_alert("Cannot init objects in rte-init queue");
+ rte_errno = EFAULT;
+ return -1;
+ }
+
eal_thread_init_master(rte_config.master_lcore);
ret = eal_thread_dump_affinity(cpuset, sizeof(cpuset));
diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
index f9ede5b41c69..23aa30f542db 100644
--- a/lib/librte_eal/rte_eal_version.map
+++ b/lib/librte_eal/rte_eal_version.map
@@ -338,4 +338,5 @@ EXPERIMENTAL {
# added in 20.05
rte_log_can_log;
+ rte_init_register;
};
--
1.8.3.1
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v4] eal: add VFIO-PCI SR-IOV support
2020-04-13 8:29 2% ` [dpdk-dev] [PATCH v4] eal: add VFIO-PCI SR-IOV support Haiyue Wang
2020-04-13 12:18 3% ` Thomas Monjalon
@ 2020-04-13 15:37 0% ` Andrew Rybchenko
2020-04-13 16:45 0% ` Wang, Haiyue
1 sibling, 1 reply; 200+ results
From: Andrew Rybchenko @ 2020-04-13 15:37 UTC (permalink / raw)
To: Haiyue Wang, dev, thomas, vattunuru, jerinj, alex.williamson,
david.marchand
On 4/13/20 11:29 AM, Haiyue Wang wrote:
> The kernel module vfio-pci introduces the VF token to enable SR-IOV
> support since 5.7.
>
> The VF token can be set by a vfio-pci based PF driver and must be known
> by the vfio-pci based VF driver in order to gain access to the device.
>
> An example VF token option would take this form:
>
> 1. Install vfio-pci with option 'enable_sriov=1'
>
> 2. ./usertools/dpdk-devbind.py -b vfio-pci 0000:87:00.0
>
> 3. echo 2 > /sys/bus/pci/devices/0000:87:00.0/sriov_numvfs
>
> 4. Start the PF:
> ./x86_64-native-linux-gcc/app/testpmd -l 22-25 -n 4 \
> -w 87:00.0,vf_token=2ab74924-c335-45f4-9b16-8569e5b08258 \
> --file-prefix=pf -- -i
Should I get a token from my head? Any?
> 5. Start the VF:
> ./x86_64-native-linux-gcc/app/testpmd -l 26-29 -n 4 \
> -w 87:02.0,vf_token=2ab74924-c335-45f4-9b16-8569e5b08258 \
> --file-prefix=vf1 -- -i
>
> Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
> Acked-by: Vamsi Attunuru <vattunuru@marvell.com>
> ---
> v4: 1. Ignore rte_vfio_setup_device ABI check since it is
> for Linux driver use.
>
> v3: https://patchwork.dpdk.org/patch/68254/
> Fix the Travis build failed:
> (1). rte_uuid.h:97:55: error: unknown type name ‘size_t’
> (2). rte_uuid.h:58:2: error: implicit declaration of function ‘memcpy’
>
> v2: https://patchwork.dpdk.org/patch/68240/
> Fix the FreeBSD build error.
>
> v1: https://patchwork.dpdk.org/patch/68237/
> Update the commit message.
>
> RFC v2: https://patchwork.dpdk.org/patch/68114/
> Based on Vamsi's RFC v1, and Alex's patch for Qemu
> [https://lore.kernel.org/lkml/20200204161737.34696b91@w520.home/]:
> Use the devarg to pass-down the VF token.
>
> RFC v1: https://patchwork.dpdk.org/patch/66281/ by Vamsi.
> ---
> devtools/libabigail.abignore | 3 ++
> drivers/bus/pci/linux/pci_vfio.c | 56 +++++++++++++++++++++++++++++--
> lib/librte_eal/freebsd/eal.c | 3 +-
> lib/librte_eal/include/rte_uuid.h | 2 ++
> lib/librte_eal/include/rte_vfio.h | 8 ++++-
> lib/librte_eal/linux/eal_vfio.c | 20 +++++++++--
> 6 files changed, 85 insertions(+), 7 deletions(-)
>
> diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore
> index a59df8f13..d918746b4 100644
> --- a/devtools/libabigail.abignore
> +++ b/devtools/libabigail.abignore
> @@ -11,3 +11,6 @@
> type_kind = enum
> name = rte_crypto_asym_xform_type
> changed_enumerators = RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
> +; Ignore this function which is only relevant to linux for driver
> +[suppress_type]
> + name = rte_vfio_setup_device
> diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
> index 64cd84a68..7f99337c7 100644
> --- a/drivers/bus/pci/linux/pci_vfio.c
> +++ b/drivers/bus/pci/linux/pci_vfio.c
> @@ -11,6 +11,7 @@
> #include <sys/mman.h>
> #include <stdbool.h>
>
> +#include <rte_devargs.h>
> #include <rte_log.h>
> #include <rte_pci.h>
> #include <rte_bus_pci.h>
> @@ -644,11 +645,59 @@ pci_vfio_msix_is_mappable(int vfio_dev_fd, int msix_region)
> return ret;
> }
>
> +static void
> +vfio_pci_vf_token_arg(struct rte_devargs *devargs, rte_uuid_t uu)
> +{
> +#define VF_TOKEN_ARG "vf_token="
> + char c, *p, *vf_token;
> +
> + if (devargs == NULL)
> + return;
> +
> + p = strstr(devargs->args, VF_TOKEN_ARG);
> + if (!p)
> + return;
> +
> + vf_token = p + strlen(VF_TOKEN_ARG);
> + if (strlen(vf_token) < (RTE_UUID_STRLEN - 1))
> + return;
> +
> + c = vf_token[RTE_UUID_STRLEN - 1];
> + if (c != '\0' && c != ',')
> + return;
> +
> + vf_token[RTE_UUID_STRLEN - 1] = '\0';
Is it possible to parse and handle devargs using rte_kvargs.h?
> + if (rte_uuid_parse(vf_token, uu)) {
> + RTE_LOG(ERR, EAL,
> + "The VF token is not a valid uuid : %s\n", vf_token);
> + vf_token[RTE_UUID_STRLEN - 1] = c;
> + return;
I think that the function must return error which is handled
by the caller when something bad happens (e.g. invalid
UUID).
> + }
> +
> + RTE_LOG(DEBUG, EAL,
> + "The VF token is found : %s\n", vf_token);
> +
> + vf_token[RTE_UUID_STRLEN - 1] = c;
> +
> + /* Purge this vfio-pci specific token from the device arguments */
> + if (c != '\0') {
> + /* 1. Handle the case : 'vf_token=uuid,arg1=val1' */
> + memmove(p, vf_token + RTE_UUID_STRLEN,
> + strlen(vf_token + RTE_UUID_STRLEN) + 1);
> + } else {
> + /* 2. Handle the case : 'arg1=val1,vf_token=uuid' */
> + if (p != devargs->args)
> + p--;
> +
> + *p = '\0';
> + }
Is it really required to purge? Why? If yes, it should be explained in
the comment above.
> +}
>
> static int
> pci_vfio_map_resource_primary(struct rte_pci_device *dev)
> {
> struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
> + rte_uuid_t vf_token = RTE_UUID_INIT(0, 0, 0, 0, 0ULL);
May be it would be better if vfio_pci_vf_token_arg()
initializes it anyway instead of duplication init
in two places?
> char pci_addr[PATH_MAX] = {0};
> int vfio_dev_fd;
> struct rte_pci_addr *loc = &dev->addr;
> @@ -668,8 +717,9 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
> snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
> loc->domain, loc->bus, loc->devid, loc->function);
>
> + vfio_pci_vf_token_arg(dev->device.devargs, vf_token);
> ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
> - &vfio_dev_fd, &device_info);
> + &vfio_dev_fd, &device_info, vf_token);
> if (ret)
> return ret;
>
> @@ -797,6 +847,7 @@ static int
> pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
> {
> struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
> + rte_uuid_t vf_token = RTE_UUID_INIT(0, 0, 0, 0, 0ULL);
> char pci_addr[PATH_MAX] = {0};
> int vfio_dev_fd;
> struct rte_pci_addr *loc = &dev->addr;
> @@ -830,8 +881,9 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
> return -1;
> }
>
> + vfio_pci_vf_token_arg(dev->device.devargs, vf_token);
> ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
> - &vfio_dev_fd, &device_info);
> + &vfio_dev_fd, &device_info, vf_token);
> if (ret)
> return ret;
>
> diff --git a/lib/librte_eal/freebsd/eal.c b/lib/librte_eal/freebsd/eal.c
> index 6ae37e7e6..a92584795 100644
> --- a/lib/librte_eal/freebsd/eal.c
> +++ b/lib/librte_eal/freebsd/eal.c
> @@ -995,7 +995,8 @@ rte_eal_vfio_intr_mode(void)
> int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
> __rte_unused const char *dev_addr,
> __rte_unused int *vfio_dev_fd,
> - __rte_unused struct vfio_device_info *device_info)
> + __rte_unused struct vfio_device_info *device_info,
> + __rte_unused rte_uuid_t vf_token)
> {
> return -1;
> }
> diff --git a/lib/librte_eal/include/rte_uuid.h b/lib/librte_eal/include/rte_uuid.h
> index 044afbdfa..8b42e070a 100644
> --- a/lib/librte_eal/include/rte_uuid.h
> +++ b/lib/librte_eal/include/rte_uuid.h
> @@ -15,6 +15,8 @@ extern "C" {
> #endif
>
> #include <stdbool.h>
> +#include <stddef.h>
> +#include <string.h>
>
> /**
> * Struct describing a Universal Unique Identifier
> diff --git a/lib/librte_eal/include/rte_vfio.h b/lib/librte_eal/include/rte_vfio.h
> index 20ed8c45a..1f9e22d82 100644
> --- a/lib/librte_eal/include/rte_vfio.h
> +++ b/lib/librte_eal/include/rte_vfio.h
> @@ -16,6 +16,8 @@ extern "C" {
>
> #include <stdint.h>
>
> +#include <rte_uuid.h>
> +
> /*
> * determine if VFIO is present on the system
> */
> @@ -102,13 +104,17 @@ struct vfio_device_info;
> * @param device_info
> * Device information.
> *
> + * @param vf_token
> + * VF token.
Such comments are useles and just eat space adding nothing
useful. Please, make it useful and explain what is behind the
parameter, when it is necessary, why? Should it be specified
for PF case, VF case, both?
> + *
> * @return
> * 0 on success.
> * <0 on failure.
> * >1 if the device cannot be managed this way.
> */
> int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
> - int *vfio_dev_fd, struct vfio_device_info *device_info);
> + int *vfio_dev_fd, struct vfio_device_info *device_info,
> + rte_uuid_t vf_token);
"rte_uuid_t vf_token" looks confusing. Shouldn't it be
"rte_uuid_t *vf_token"?
>
> /**
> * Release a device mapped to a VFIO-managed I/O MMU group.
> diff --git a/lib/librte_eal/linux/eal_vfio.c b/lib/librte_eal/linux/eal_vfio.c
> index 4502aefed..916082b5d 100644
> --- a/lib/librte_eal/linux/eal_vfio.c
> +++ b/lib/librte_eal/linux/eal_vfio.c
> @@ -702,7 +702,8 @@ rte_vfio_clear_group(int vfio_group_fd)
>
> int
> rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
> - int *vfio_dev_fd, struct vfio_device_info *device_info)
> + int *vfio_dev_fd, struct vfio_device_info *device_info,
> + rte_uuid_t vf_token)
> {
> struct vfio_group_status group_status = {
> .argsz = sizeof(group_status)
> @@ -712,6 +713,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
> int vfio_container_fd;
> int vfio_group_fd;
> int iommu_group_num;
> + char dev[PATH_MAX];
Why PATH_MAX?
> int i, ret;
>
> /* get group number */
> @@ -895,8 +897,19 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
> t->type_id, t->name);
> }
>
> + if (!rte_uuid_is_null(vf_token)) {
> + char vf_token_str[RTE_UUID_STRLEN];
> +
> + rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
> + snprintf(dev, sizeof(dev),
> + "%s vf_token=%s", dev_addr, vf_token_str);
> + } else {
> + snprintf(dev, sizeof(dev),
> + "%s", dev_addr);
> + }
> +
> /* get a file descriptor for the device */
> - *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
> + *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, dev);
> if (*vfio_dev_fd < 0) {
> /* if we cannot get a device fd, this implies a problem with
> * the VFIO group or the container not having IOMMU configured.
> @@ -2081,7 +2094,8 @@ int
> rte_vfio_setup_device(__rte_unused const char *sysfs_base,
> __rte_unused const char *dev_addr,
> __rte_unused int *vfio_dev_fd,
> - __rte_unused struct vfio_device_info *device_info)
> + __rte_unused struct vfio_device_info *device_info,
> + __rte_unused rte_uuid_t vf_token)
> {
> return -1;
> }
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v4] eal: add VFIO-PCI SR-IOV support
2020-04-13 15:37 0% ` Andrew Rybchenko
@ 2020-04-13 16:45 0% ` Wang, Haiyue
0 siblings, 0 replies; 200+ results
From: Wang, Haiyue @ 2020-04-13 16:45 UTC (permalink / raw)
To: Andrew Rybchenko, dev, thomas, vattunuru, jerinj,
alex.williamson, david.marchand
Hi Andrew,
> -----Original Message-----
> From: Andrew Rybchenko <arybchenko@solarflare.com>
> Sent: Monday, April 13, 2020 23:38
> To: Wang, Haiyue <haiyue.wang@intel.com>; dev@dpdk.org; thomas@monjalon.net; vattunuru@marvell.com;
> jerinj@marvell.com; alex.williamson@redhat.com; david.marchand@redhat.com
> Subject: Re: [dpdk-dev] [PATCH v4] eal: add VFIO-PCI SR-IOV support
>
> On 4/13/20 11:29 AM, Haiyue Wang wrote:
> > The kernel module vfio-pci introduces the VF token to enable SR-IOV
> > support since 5.7.
> >
> > The VF token can be set by a vfio-pci based PF driver and must be known
> > by the vfio-pci based VF driver in order to gain access to the device.
> >
> > An example VF token option would take this form:
> >
> > 1. Install vfio-pci with option 'enable_sriov=1'
> >
> > 2. ./usertools/dpdk-devbind.py -b vfio-pci 0000:87:00.0
> >
> > 3. echo 2 > /sys/bus/pci/devices/0000:87:00.0/sriov_numvfs
> >
> > 4. Start the PF:
> > ./x86_64-native-linux-gcc/app/testpmd -l 22-25 -n 4 \
> > -w 87:00.0,vf_token=2ab74924-c335-45f4-9b16-8569e5b08258 \
> > --file-prefix=pf -- -i
>
> Should I get a token from my head? Any?
>
> > 5. Start the VF:
> > ./x86_64-native-linux-gcc/app/testpmd -l 26-29 -n 4 \
> > -w 87:02.0,vf_token=2ab74924-c335-45f4-9b16-8569e5b08258 \
> > --file-prefix=vf1 -- -i
> >
> > Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
> > Acked-by: Vamsi Attunuru <vattunuru@marvell.com>
> > ---
> > v4: 1. Ignore rte_vfio_setup_device ABI check since it is
> > for Linux driver use.
> >
> > v3: https://patchwork.dpdk.org/patch/68254/
> > Fix the Travis build failed:
> > (1). rte_uuid.h:97:55: error: unknown type name ‘size_t’
> > (2). rte_uuid.h:58:2: error: implicit declaration of function ‘memcpy’
> >
> > v2: https://patchwork.dpdk.org/patch/68240/
> > Fix the FreeBSD build error.
> >
> > v1: https://patchwork.dpdk.org/patch/68237/
> > Update the commit message.
> >
> > RFC v2: https://patchwork.dpdk.org/patch/68114/
> > Based on Vamsi's RFC v1, and Alex's patch for Qemu
> > [https://lore.kernel.org/lkml/20200204161737.34696b91@w520.home/]:
> > Use the devarg to pass-down the VF token.
> >
> > RFC v1: https://patchwork.dpdk.org/patch/66281/ by Vamsi.
> > ---
> > devtools/libabigail.abignore | 3 ++
> > drivers/bus/pci/linux/pci_vfio.c | 56 +++++++++++++++++++++++++++++--
> > lib/librte_eal/freebsd/eal.c | 3 +-
> > lib/librte_eal/include/rte_uuid.h | 2 ++
> > lib/librte_eal/include/rte_vfio.h | 8 ++++-
> > lib/librte_eal/linux/eal_vfio.c | 20 +++++++++--
> > 6 files changed, 85 insertions(+), 7 deletions(-)
> >
> > diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore
> > index a59df8f13..d918746b4 100644
> > --- a/devtools/libabigail.abignore
> > +++ b/devtools/libabigail.abignore
> > @@ -11,3 +11,6 @@
> > type_kind = enum
> > name = rte_crypto_asym_xform_type
> > changed_enumerators = RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
> > +; Ignore this function which is only relevant to linux for driver
> > +[suppress_type]
> > + name = rte_vfio_setup_device
> > diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
> > index 64cd84a68..7f99337c7 100644
> > --- a/drivers/bus/pci/linux/pci_vfio.c
> > +++ b/drivers/bus/pci/linux/pci_vfio.c
> > @@ -11,6 +11,7 @@
> > #include <sys/mman.h>
> > #include <stdbool.h>
> >
> > +#include <rte_devargs.h>
> > #include <rte_log.h>
> > #include <rte_pci.h>
> > #include <rte_bus_pci.h>
> > @@ -644,11 +645,59 @@ pci_vfio_msix_is_mappable(int vfio_dev_fd, int msix_region)
> > return ret;
> > }
> >
> > +static void
> > +vfio_pci_vf_token_arg(struct rte_devargs *devargs, rte_uuid_t uu)
> > +{
> > +#define VF_TOKEN_ARG "vf_token="
> > + char c, *p, *vf_token;
> > +
> > + if (devargs == NULL)
> > + return;
> > +
> > + p = strstr(devargs->args, VF_TOKEN_ARG);
> > + if (!p)
> > + return;
> > +
> > + vf_token = p + strlen(VF_TOKEN_ARG);
> > + if (strlen(vf_token) < (RTE_UUID_STRLEN - 1))
> > + return;
> > +
> > + c = vf_token[RTE_UUID_STRLEN - 1];
> > + if (c != '\0' && c != ',')
> > + return;
> > +
> > + vf_token[RTE_UUID_STRLEN - 1] = '\0';
>
> Is it possible to parse and handle devargs using rte_kvargs.h?
>
Since it needs to remove the 'vf_token', as 'vf_token' is not a
valid PMD related args, so need to parse and delete it.
rte_kvargs_parse(const char *args, const char * const valid_keys[])
> > + if (rte_uuid_parse(vf_token, uu)) {
> > + RTE_LOG(ERR, EAL,
> > + "The VF token is not a valid uuid : %s\n", vf_token);
> > + vf_token[RTE_UUID_STRLEN - 1] = c;
> > + return;
>
> I think that the function must return error which is handled
> by the caller when something bad happens (e.g. invalid
> UUID).
>
Yes, make sense, will add the error handling.
> > + }
> > +
> > + RTE_LOG(DEBUG, EAL,
> > + "The VF token is found : %s\n", vf_token);
> > +
> > + vf_token[RTE_UUID_STRLEN - 1] = c;
> > +
> > + /* Purge this vfio-pci specific token from the device arguments */
> > + if (c != '\0') {
> > + /* 1. Handle the case : 'vf_token=uuid,arg1=val1' */
> > + memmove(p, vf_token + RTE_UUID_STRLEN,
> > + strlen(vf_token + RTE_UUID_STRLEN) + 1);
> > + } else {
> > + /* 2. Handle the case : 'arg1=val1,vf_token=uuid' */
> > + if (p != devargs->args)
> > + p--;
> > +
> > + *p = '\0';
> > + }
>
> Is it really required to purge? Why? If yes, it should be explained in
> the comment above.
Please see above reply.
>
> > +}
> >
> > static int
> > pci_vfio_map_resource_primary(struct rte_pci_device *dev)
> > {
> > struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
> > + rte_uuid_t vf_token = RTE_UUID_INIT(0, 0, 0, 0, 0ULL);
>
> May be it would be better if vfio_pci_vf_token_arg()
> initializes it anyway instead of duplication init
> in two places?
+1, will update it.
>
> > char pci_addr[PATH_MAX] = {0};
> > int vfio_dev_fd;
> > struct rte_pci_addr *loc = &dev->addr;
> > @@ -668,8 +717,9 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
> > snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
> > loc->domain, loc->bus, loc->devid, loc->function);
> >
> > + vfio_pci_vf_token_arg(dev->device.devargs, vf_token);
> > ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
> > - &vfio_dev_fd, &device_info);
> > + &vfio_dev_fd, &device_info, vf_token);
> > if (ret)
> > return ret;
> >
> > @@ -797,6 +847,7 @@ static int
> > pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
> > {
> > struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
> > + rte_uuid_t vf_token = RTE_UUID_INIT(0, 0, 0, 0, 0ULL);
> > char pci_addr[PATH_MAX] = {0};
> > int vfio_dev_fd;
> > struct rte_pci_addr *loc = &dev->addr;
> > @@ -830,8 +881,9 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
> > return -1;
> > }
> >
> > + vfio_pci_vf_token_arg(dev->device.devargs, vf_token);
> > ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
> > - &vfio_dev_fd, &device_info);
> > + &vfio_dev_fd, &device_info, vf_token);
> > if (ret)
> > return ret;
> >
> > diff --git a/lib/librte_eal/freebsd/eal.c b/lib/librte_eal/freebsd/eal.c
> > index 6ae37e7e6..a92584795 100644
> > --- a/lib/librte_eal/freebsd/eal.c
> > +++ b/lib/librte_eal/freebsd/eal.c
> > @@ -995,7 +995,8 @@ rte_eal_vfio_intr_mode(void)
> > int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
> > __rte_unused const char *dev_addr,
> > __rte_unused int *vfio_dev_fd,
> > - __rte_unused struct vfio_device_info *device_info)
> > + __rte_unused struct vfio_device_info *device_info,
> > + __rte_unused rte_uuid_t vf_token)
> > {
> > return -1;
> > }
> > diff --git a/lib/librte_eal/include/rte_uuid.h b/lib/librte_eal/include/rte_uuid.h
> > index 044afbdfa..8b42e070a 100644
> > --- a/lib/librte_eal/include/rte_uuid.h
> > +++ b/lib/librte_eal/include/rte_uuid.h
> > @@ -15,6 +15,8 @@ extern "C" {
> > #endif
> >
> > #include <stdbool.h>
> > +#include <stddef.h>
> > +#include <string.h>
> >
> > /**
> > * Struct describing a Universal Unique Identifier
> > diff --git a/lib/librte_eal/include/rte_vfio.h b/lib/librte_eal/include/rte_vfio.h
> > index 20ed8c45a..1f9e22d82 100644
> > --- a/lib/librte_eal/include/rte_vfio.h
> > +++ b/lib/librte_eal/include/rte_vfio.h
> > @@ -16,6 +16,8 @@ extern "C" {
> >
> > #include <stdint.h>
> >
> > +#include <rte_uuid.h>
> > +
> > /*
> > * determine if VFIO is present on the system
> > */
> > @@ -102,13 +104,17 @@ struct vfio_device_info;
> > * @param device_info
> > * Device information.
> > *
> > + * @param vf_token
> > + * VF token.
>
> Such comments are useles and just eat space adding nothing
> useful. Please, make it useful and explain what is behind the
> parameter, when it is necessary, why? Should it be specified
> for PF case, VF case, both?
>
Will add more comments, yes for both PF and VF, as Alex's linux patch
explained.
> > + *
> > * @return
> > * 0 on success.
> > * <0 on failure.
> > * >1 if the device cannot be managed this way.
> > */
> > int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
> > - int *vfio_dev_fd, struct vfio_device_info *device_info);
> > + int *vfio_dev_fd, struct vfio_device_info *device_info,
> > + rte_uuid_t vf_token);
>
> "rte_uuid_t vf_token" looks confusing. Shouldn't it be
> "rte_uuid_t *vf_token"?
This is UUID API design and type definition:
bool rte_uuid_is_null(const rte_uuid_t uu);
DPDK: typedef unsigned char rte_uuid_t[16];
vs
Linux: typedef struct {
__u8 b[UUID_SIZE];
} uuid_t;
>
> >
> > /**
> > * Release a device mapped to a VFIO-managed I/O MMU group.
> > diff --git a/lib/librte_eal/linux/eal_vfio.c b/lib/librte_eal/linux/eal_vfio.c
> > index 4502aefed..916082b5d 100644
> > --- a/lib/librte_eal/linux/eal_vfio.c
> > +++ b/lib/librte_eal/linux/eal_vfio.c
> > @@ -702,7 +702,8 @@ rte_vfio_clear_group(int vfio_group_fd)
> >
> > int
> > rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
> > - int *vfio_dev_fd, struct vfio_device_info *device_info)
> > + int *vfio_dev_fd, struct vfio_device_info *device_info,
> > + rte_uuid_t vf_token)
> > {
> > struct vfio_group_status group_status = {
> > .argsz = sizeof(group_status)
> > @@ -712,6 +713,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
> > int vfio_container_fd;
> > int vfio_group_fd;
> > int iommu_group_num;
> > + char dev[PATH_MAX];
>
> Why PATH_MAX?
Based on Vamsi's RFC v1, and found that it looked a little reasonable, ' char pci_addr[PATH_MAX] = {0}; '
static int
pci_vfio_map_resource_primary(struct rte_pci_device *dev)
{
struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
char pci_addr[PATH_MAX] = {0}; <----
>
> > int i, ret;
> >
> > /* get group number */
> > @@ -895,8 +897,19 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
> > t->type_id, t->name);
> > }
> >
> > + if (!rte_uuid_is_null(vf_token)) {
> > + char vf_token_str[RTE_UUID_STRLEN];
> > +
> > + rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
> > + snprintf(dev, sizeof(dev),
> > + "%s vf_token=%s", dev_addr, vf_token_str);
> > + } else {
> > + snprintf(dev, sizeof(dev),
> > + "%s", dev_addr);
> > + }
> > +
> > /* get a file descriptor for the device */
> > - *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
> > + *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, dev);
> > if (*vfio_dev_fd < 0) {
> > /* if we cannot get a device fd, this implies a problem with
> > * the VFIO group or the container not having IOMMU configured.
> > @@ -2081,7 +2094,8 @@ int
> > rte_vfio_setup_device(__rte_unused const char *sysfs_base,
> > __rte_unused const char *dev_addr,
> > __rte_unused int *vfio_dev_fd,
> > - __rte_unused struct vfio_device_info *device_info)
> > + __rte_unused struct vfio_device_info *device_info,
> > + __rte_unused rte_uuid_t vf_token)
> > {
> > return -1;
> > }
> >
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v4] eal: add VFIO-PCI SR-IOV support
2020-04-13 12:18 3% ` Thomas Monjalon
@ 2020-04-13 17:01 3% ` Wang, Haiyue
0 siblings, 0 replies; 200+ results
From: Wang, Haiyue @ 2020-04-13 17:01 UTC (permalink / raw)
To: Thomas Monjalon; +Cc: dev, vattunuru, jerinj, alex.williamson, david.marchand
Hi Thomas,
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Monday, April 13, 2020 20:18
> To: Wang, Haiyue <haiyue.wang@intel.com>
> Cc: dev@dpdk.org; vattunuru@marvell.com; jerinj@marvell.com; alex.williamson@redhat.com;
> david.marchand@redhat.com
> Subject: Re: [PATCH v4] eal: add VFIO-PCI SR-IOV support
>
> Hi,
>
> About the title, I think it does not convey what is new here.
> VFIO is not new, SR-IOV is already supported.
> The title should mention the new VFIO feature in few simple words.
> Is it only about using VFIO for PF?
>
For Both. Align with Alex's : vfio-pci: QEMU support for vfio-pci VF tokens
How about: 'eal: support for VFIO-PCI VF tokens' ?
>
> 13/04/2020 10:29, Haiyue Wang:
> > v4: 1. Ignore rte_vfio_setup_device ABI check since it is
> > for Linux driver use.
> [...]
> > +; Ignore this function which is only relevant to linux for driver
> > +[suppress_type]
> > + name = rte_vfio_setup_device
>
> Adding such exception for all internal "driver interface" functions
> is not scaling. Please use __rte_internal.
> I am waiting for the patchset about rte_internal to be reviewed or completed.
> As it is not progressing, the decision is to block any patch having
> ABI issue because of internal false positive.
> Please help, thanks.
>
I will drop this patch ABI workaround, and try to catch the '__rte_internal ' design.
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v5 0/2] support for VFIO-PCI VF token interface
2020-04-13 8:29 2% ` [dpdk-dev] [PATCH v4] eal: add VFIO-PCI SR-IOV support Haiyue Wang
@ 2020-04-14 3:06 4% ` Haiyue Wang
2020-04-14 3:21 4% ` [dpdk-dev] [PATCH v6 " Haiyue Wang
` (2 subsequent siblings)
4 siblings, 0 replies; 200+ results
From: Haiyue Wang @ 2020-04-14 3:06 UTC (permalink / raw)
To: dev, thomas, vattunuru, jerinj, alex.williamson, david.marchand
Cc: Haiyue Wang
v5: 1. Add the VF token parse error handling.
2. Split into two patches for different logic module.
3. Add more comments into the code for explaining the design.
4. Drop the ABI change workaround, this patch set focuses on code review.
v4: 1. Ignore rte_vfio_setup_device ABI check since it is
for Linux driver use.
https://patchwork.dpdk.org/patch/68255/
v3: Fix the Travis build failed:
(1). rte_uuid.h:97:55: error: unknown type name ‘size_t’
(2). rte_uuid.h:58:2: error: implicit declaration of function ‘memcpy’
https://patchwork.dpdk.org/patch/68254/
v2: Fix the FreeBSD build error.
https://patchwork.dpdk.org/patch/68240/
v1: Update the commit message.
https://patchwork.dpdk.org/patch/68237/
RFC v2: https://patchwork.dpdk.org/patch/68114/
Based on Vamsi's RFC v1, and Alex's patch for Qemu
[https://lore.kernel.org/lkml/20200204161737.34696b91@w520.home/]:
Use the devarg to pass-down the VF token.
RFC v1: https://patchwork.dpdk.org/patch/66281/ by Vamsi.
Haiyue Wang (2):
eal: add uuid dependent header files explicitly
eal: support for VFIO-PCI VF token
drivers/bus/pci/linux/pci_vfio.c | 74 ++++++++++++++++++++++++++++++-
lib/librte_eal/freebsd/eal.c | 3 +-
lib/librte_eal/include/rte_uuid.h | 2 +
lib/librte_eal/include/rte_vfio.h | 21 ++++++++-
lib/librte_eal/linux/eal_vfio.c | 20 +++++++--
5 files changed, 113 insertions(+), 7 deletions(-)
--
2.26.0
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH v6 0/2] support for VFIO-PCI VF token interface
2020-04-13 8:29 2% ` [dpdk-dev] [PATCH v4] eal: add VFIO-PCI SR-IOV support Haiyue Wang
2020-04-14 3:06 4% ` [dpdk-dev] [PATCH v5 0/2] support for VFIO-PCI VF token interface Haiyue Wang
@ 2020-04-14 3:21 4% ` Haiyue Wang
2020-04-14 13:18 0% ` [dpdk-dev] [EXT] " Vamsi Krishna Attunuru
2020-04-18 11:16 4% ` [dpdk-dev] [PATCH v7 " Haiyue Wang
2020-04-18 17:30 4% ` [dpdk-dev] [PATCH v8 " Haiyue Wang
4 siblings, 1 reply; 200+ results
From: Haiyue Wang @ 2020-04-14 3:21 UTC (permalink / raw)
To: dev, thomas, vattunuru, jerinj, alex.williamson, david.marchand
Cc: Haiyue Wang
v6: Drop the Fixes tag in uuid, since the file has been
moved to another place, not suitable to apply on stable.
And this is not a bug, just some kind of enhancement.
v5: 1. Add the VF token parse error handling.
2. Split into two patches for different logic module.
3. Add more comments into the code for explaining the design.
4. Drop the ABI change workaround, this patch set focuses on code review.
https://patchwork.dpdk.org/cover/68364/
v4: 1. Ignore rte_vfio_setup_device ABI check since it is
for Linux driver use.
https://patchwork.dpdk.org/patch/68255/
v3: Fix the Travis build failed:
(1). rte_uuid.h:97:55: error: unknown type name ‘size_t’
(2). rte_uuid.h:58:2: error: implicit declaration of function ‘memcpy’
https://patchwork.dpdk.org/patch/68254/
v2: Fix the FreeBSD build error.
https://patchwork.dpdk.org/patch/68240/
v1: Update the commit message.
https://patchwork.dpdk.org/patch/68237/
RFC v2: https://patchwork.dpdk.org/patch/68114/
Based on Vamsi's RFC v1, and Alex's patch for Qemu
[https://lore.kernel.org/lkml/20200204161737.34696b91@w520.home/]:
Use the devarg to pass-down the VF token.
RFC v1: https://patchwork.dpdk.org/patch/66281/ by Vamsi.
Haiyue Wang (2):
eal: add uuid dependent header files explicitly
eal: support for VFIO-PCI VF token
drivers/bus/pci/linux/pci_vfio.c | 74 ++++++++++++++++++++++++++++++-
lib/librte_eal/freebsd/eal.c | 3 +-
lib/librte_eal/include/rte_uuid.h | 2 +
lib/librte_eal/include/rte_vfio.h | 21 ++++++++-
lib/librte_eal/linux/eal_vfio.c | 20 +++++++--
5 files changed, 113 insertions(+), 7 deletions(-)
--
2.26.0
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] build: add arm 32bit cross compilation to Meson
2020-04-13 6:23 0% ` Ruifeng Wang
@ 2020-04-14 7:05 0% ` Juraj Linkeš
0 siblings, 0 replies; 200+ results
From: Juraj Linkeš @ 2020-04-14 7:05 UTC (permalink / raw)
To: Ruifeng Wang, bruce.richardson
Cc: dev, Honnappa Nagarahalli, Gavin Hu, nd, nd
Hi Ruifeng,
> -----Original Message-----
> From: Ruifeng Wang <Ruifeng.Wang@arm.com>
> Sent: Monday, April 13, 2020 8:24 AM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>; bruce.richardson@intel.com
> Cc: dev@dpdk.org; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>;
> Gavin Hu <Gavin.Hu@arm.com>; nd <nd@arm.com>; nd <nd@arm.com>
> Subject: RE: [dpdk-dev] [PATCH] build: add arm 32bit cross compilation to
> Meson
>
> Hi Juraj,
>
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Juraj Linke?
> > Sent: Thursday, April 9, 2020 10:15 PM
> > To: bruce.richardson@intel.com
> > Cc: dev@dpdk.org; Juraj Linkeš <juraj.linkes@pantheon.tech>
> > Subject: [dpdk-dev] [PATCH] build: add arm 32bit cross compilation to
> > Meson
> >
> > * Add arm-linux-gnueabihf cross-file
> > * Add generic and default arm 32 bit flags to arm meson.build
> > * Add support for disabling drivers using flags defined in Meson
> > * Change checks from dpdk_conf.has() to dpdk.conf.get()
> > * When processing which drivers to build, check whether the
> > appropriate RTE flag isn't set to false
> >
> The changes are not small as a single patch.
> Could you split it into series of smaller patches?
> I think each bullet in commit message can be a separate patch.
Sure, I'll split the patch.
> > --
> > 2.11.0
> > NOTES: tested here: https://travis-
> > ci.com/github/jlinkes/dpdk/builds/159597484
> > There are two issues I would like to get feedback for:
> > 1. the aarch64 -> arm cross compilation fails when compiling l3fwd
> > example [0].
> > I think this failure needs to be fixed by arm devs, but I would like to have
> > this confirmed.
> Yes, this should be fixed in source code.
Ok, thanks.
> > 2. the patch breaks ABI checks on aarch64 [1]. The reason is that the
> > RTE_LIBRTE_FM10K_PMD and RTE_LIBRTE_AVP_PMD flags in
> > config/arm/meson.build
> > get properly applied, the libs don't get built and then check the fails when
> > it doesn't find them. I don't know whether the application of these flags is
> > desirable (and we would need to fix the ABI check) or whether we should
> > remove the flags.
> Does the changes impact not only aarch32 but also aarch64?
Yes, it does, but only aarch64 cross build (when --cross-file is used). Bruce mentioned that they're moving away from statically defining the driver flags, so the mechanism will change, probably in a way that won't affect aarch64 cross build.
Juraj
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [dpdk-dev v4 1/3] ethdev: add new RSS offload types
@ 2020-04-14 9:42 3% ` Ori Kam
2020-04-15 2:31 0% ` Zhang, Qi Z
0 siblings, 1 reply; 200+ results
From: Ori Kam @ 2020-04-14 9:42 UTC (permalink / raw)
To: Jeff Guo, bernard.iremonger, xiaolong.ye, qi.z.zhang
Cc: dev, jingjing.wu, yahui.cao, simei.su
Hi Jeff,
PSB
Best,
Ori
> -----Original Message-----
> From: Jeff Guo <jia.guo@intel.com>
> Sent: Tuesday, April 14, 2020 8:42 PM
> To: Ori Kam <orika@mellanox.com>; bernard.iremonger@intel.com;
> xiaolong.ye@intel.com; qi.z.zhang@intel.com
> Cc: dev@dpdk.org; jingjing.wu@intel.com; yahui.cao@intel.com;
> simei.su@intel.com; jia.guo@intel.com
> Subject: [dpdk-dev v4 1/3] ethdev: add new RSS offload types
>
> Defines some new RSS offload types for ETH/SVLAN/CVLAN/GTPU/L2TPV3/
> ESP/AH/PFCP.
>
> Signed-off-by: Jeff Guo <jia.guo@intel.com>
> Reviewed-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
> v4->v3:
> no change
> ---
> lib/librte_ethdev/rte_ethdev.h | 11 ++++++++++-
> 1 file changed, 10 insertions(+), 1 deletion(-)
>
> diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
> index d1a593ad1..efe705ff0 100644
> --- a/lib/librte_ethdev/rte_ethdev.h
> +++ b/lib/librte_ethdev/rte_ethdev.h
> @@ -511,6 +511,13 @@ struct rte_eth_rss_conf {
> #define ETH_RSS_GENEVE (1ULL << 20)
> #define ETH_RSS_NVGRE (1ULL << 21)
> #define ETH_RSS_GTPU (1ULL << 23)
> +#define ETH_RSS_ETH (1ULL << 24)
> +#define ETH_RSS_S_VLAN (1ULL << 25)
> +#define ETH_RSS_C_VLAN (1ULL << 26)
> +#define ETH_RSS_ESP (1ULL << 27)
> +#define ETH_RSS_AH (1ULL << 28)
> +#define ETH_RSS_L2TPV3 (1ULL << 29)
> +#define ETH_RSS_PFCP (1ULL << 30)
>
> /*
> * We use the following macros to combine with above ETH_RSS_* for
> @@ -524,7 +531,9 @@ struct rte_eth_rss_conf {
> #define ETH_RSS_L3_SRC_ONLY (1ULL << 63)
> #define ETH_RSS_L3_DST_ONLY (1ULL << 62)
> #define ETH_RSS_L4_SRC_ONLY (1ULL << 61)
> -#define ETH_RSS_L4_DST_ONLY (1ULL << 60)
> +#define ETH_RSS_L4_DST_ONLY (1ULL << 60)
> +#define ETH_RSS_ETH_SRC_ONLY (1ULL << 59)
> +#define ETH_RSS_ETH_DST_ONLY (1ULL << 58)
>
Why not change to L2_RSS_SRC_ONLY? Same for DST?
Also if it is not ABI/API change I would suggest to move the
ETH defines to be 63.
> /**
> * For input set change of hash filter, if SRC_ONLY and DST_ONLY of
> --
> 2.20.1
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
@ 2020-04-14 12:13 3% ` Akhil Goyal
2020-04-14 13:03 0% ` Thomas Monjalon
2020-04-14 13:54 0% ` Ray Kinsella
1 sibling, 1 reply; 200+ results
From: Akhil Goyal @ 2020-04-14 12:13 UTC (permalink / raw)
To: Ray Kinsella, thomas, bruce.richardson; +Cc: fiona.trahe, Arek Kusztal, dev
Hi Ray/Thomas/Bruce,
Could you please help review this patch wrt ABI policy?
>
> This patch adds versioned function rte_cryptodev_info_get.
> Node 20.05 function works the same way it was working before.
> Node 20.0 function strips capability added in 20.05 release
> to prevent some issues with ABI policy. To do that new capability
> array is allocated per device and returned to user instead of the
> original array passed by PMD.
>
> Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
> ---
Regards,
Akhil
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
2020-04-14 12:13 3% ` Akhil Goyal
@ 2020-04-14 13:03 0% ` Thomas Monjalon
2020-04-14 13:52 0% ` Trahe, Fiona
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-04-14 13:03 UTC (permalink / raw)
To: Arek Kusztal
Cc: Ray Kinsella, bruce.richardson, dev, fiona.trahe, dev, Akhil Goyal
14/04/2020 14:13, Akhil Goyal:
> Hi Ray/Thomas/Bruce,
>
> Could you please help review this patch wrt ABI policy?
> >
> > This patch adds versioned function rte_cryptodev_info_get.
> > Node 20.05 function works the same way it was working before.
> > Node 20.0 function strips capability added in 20.05 release
What are Nodes 20.05 and 20.0?
> > to prevent some issues with ABI policy. To do that new capability
> > array is allocated per device and returned to user instead of the
> > original array passed by PMD.
rte_cryptodev_info is provided by the caller of the function.
I don't understand what you explained above, and the code is missing
some comments explaining the reason of the logic.
I still don't know whether this patch or not because it is missing
some clear explanations.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [EXT] [PATCH v6 0/2] support for VFIO-PCI VF token interface
2020-04-14 3:21 4% ` [dpdk-dev] [PATCH v6 " Haiyue Wang
@ 2020-04-14 13:18 0% ` Vamsi Krishna Attunuru
0 siblings, 0 replies; 200+ results
From: Vamsi Krishna Attunuru @ 2020-04-14 13:18 UTC (permalink / raw)
To: Haiyue Wang, dev, thomas, Jerin Jacob Kollanukkaran,
alex.williamson, david.marchand
> -----Original Message-----
> From: Haiyue Wang <haiyue.wang@intel.com>
> Sent: Tuesday, April 14, 2020 8:51 AM
> To: dev@dpdk.org; thomas@monjalon.net; Vamsi Krishna Attunuru
> <vattunuru@marvell.com>; Jerin Jacob Kollanukkaran <jerinj@marvell.com>;
> alex.williamson@redhat.com; david.marchand@redhat.com
> Cc: Haiyue Wang <haiyue.wang@intel.com>
> Subject: [EXT] [PATCH v6 0/2] support for VFIO-PCI VF token interface
>
> External Email
>
> ----------------------------------------------------------------------
> v6: Drop the Fixes tag in uuid, since the file has been
> moved to another place, not suitable to apply on stable.
> And this is not a bug, just some kind of enhancement.
>
> v5: 1. Add the VF token parse error handling.
> 2. Split into two patches for different logic module.
> 3. Add more comments into the code for explaining the design.
> 4. Drop the ABI change workaround, this patch set focuses on code review.
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__patchwork.dpdk.org_cover_68364_&d=DwIDaQ&c=nKjWec2b6R0mOyP
> az7xtfQ&r=WllrYaumVkxaWjgKto6E_rtDQshhIhik2jkvzFyRhW8&m=SxW07R9
> V0iyd_fAAZm4bU3jimkCD0khdQeabCt9pESg&s=mAkB5KtaFDGFdmqfNZV4dy
> w1A8Xy61tm22TNM8kGSrQ&e=
>
> v4: 1. Ignore rte_vfio_setup_device ABI check since it is
> for Linux driver use.
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__patchwork.dpdk.org_patch_68255_&d=DwIDaQ&c=nKjWec2b6R0mOyP
> az7xtfQ&r=WllrYaumVkxaWjgKto6E_rtDQshhIhik2jkvzFyRhW8&m=SxW07R9
> V0iyd_fAAZm4bU3jimkCD0khdQeabCt9pESg&s=Qv5ArXmk_WeQn1bTnM6JS
> XpNp3NEhuenHP1VsjUPzGE&e=
>
> v3: Fix the Travis build failed:
> (1). rte_uuid.h:97:55: error: unknown type name ‘size_t’
> (2). rte_uuid.h:58:2: error: implicit declaration of function ‘memcpy’
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__patchwork.dpdk.org_patch_68254_&d=DwIDaQ&c=nKjWec2b6R0mOyP
> az7xtfQ&r=WllrYaumVkxaWjgKto6E_rtDQshhIhik2jkvzFyRhW8&m=SxW07R9
> V0iyd_fAAZm4bU3jimkCD0khdQeabCt9pESg&s=8U5HwQ-j-
> dkzQPk3E99wJauItTQLH0f48qay37pnTZ8&e=
>
> v2: Fix the FreeBSD build error.
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__patchwork.dpdk.org_patch_68240_&d=DwIDaQ&c=nKjWec2b6R0mOyP
> az7xtfQ&r=WllrYaumVkxaWjgKto6E_rtDQshhIhik2jkvzFyRhW8&m=SxW07R9
> V0iyd_fAAZm4bU3jimkCD0khdQeabCt9pESg&s=7gq0iGeXay4YQ4ydHRdCZ9_
> oWDRQFtGOxGu3chq0z_s&e=
>
> v1: Update the commit message.
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__patchwork.dpdk.org_patch_68237_&d=DwIDaQ&c=nKjWec2b6R0mOyP
> az7xtfQ&r=WllrYaumVkxaWjgKto6E_rtDQshhIhik2jkvzFyRhW8&m=SxW07R9
> V0iyd_fAAZm4bU3jimkCD0khdQeabCt9pESg&s=ibSeCUMMleRvQRH9WvJDYV
> TfBAt-Fi4lgfdTBOZ5xwc&e=
>
> RFC v2: https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__patchwork.dpdk.org_patch_68114_&d=DwIDaQ&c=nKjWec2b6R0mOyP
> az7xtfQ&r=WllrYaumVkxaWjgKto6E_rtDQshhIhik2jkvzFyRhW8&m=SxW07R9
> V0iyd_fAAZm4bU3jimkCD0khdQeabCt9pESg&s=JmSNjqCLE3Xh0_VymptqrViX
> VlRgpFN_PZMvM6C2X2s&e=
> Based on Vamsi's RFC v1, and Alex's patch for Qemu
> [https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__lore.kernel.org_lkml_20200204161737.34696b91-
> 40w520.home_&d=DwIDaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=WllrYaumVkx
> aWjgKto6E_rtDQshhIhik2jkvzFyRhW8&m=SxW07R9V0iyd_fAAZm4bU3jimkCD
> 0khdQeabCt9pESg&s=7tRxVtn0y5xpxqC38hgAB7Bjz7kHR8mbD-
> pIFgP1E7o&e= ]:
> Use the devarg to pass-down the VF token.
>
> RFC v1: https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__patchwork.dpdk.org_patch_66281_&d=DwIDaQ&c=nKjWec2b6R0mOyP
> az7xtfQ&r=WllrYaumVkxaWjgKto6E_rtDQshhIhik2jkvzFyRhW8&m=SxW07R9
> V0iyd_fAAZm4bU3jimkCD0khdQeabCt9pESg&s=riYbXycUMPQxA30y75nRzs08
> f2wM-uAGcQEnK9ENR7w&e= by Vamsi.
>
> Haiyue Wang (2):
> eal: add uuid dependent header files explicitly
> eal: support for VFIO-PCI VF token
>
> drivers/bus/pci/linux/pci_vfio.c | 74 ++++++++++++++++++++++++++++++-
> lib/librte_eal/freebsd/eal.c | 3 +-
> lib/librte_eal/include/rte_uuid.h | 2 + lib/librte_eal/include/rte_vfio.h | 21
> ++++++++-
> lib/librte_eal/linux/eal_vfio.c | 20 +++++++--
> 5 files changed, 113 insertions(+), 7 deletions(-)
>
> --
> 2.26.0
Tested following combinations:
* Both PF & VFs are with DPDK (Single process & multi process)
* PF is with DPDK & VFs are with Kernel driver
* PF is with kernel driver & VFs are with DPDK
Tested-by: Vamsi Attunuru <vattunuru@marvell.com>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
2020-04-14 13:03 0% ` Thomas Monjalon
@ 2020-04-14 13:52 0% ` Trahe, Fiona
0 siblings, 0 replies; 200+ results
From: Trahe, Fiona @ 2020-04-14 13:52 UTC (permalink / raw)
To: Thomas Monjalon, Kusztal, ArkadiuszX
Cc: Ray Kinsella, Richardson, Bruce, dev, dev, Akhil Goyal, Trahe, Fiona
Hi Akhil, Thomas,
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Tuesday, April 14, 2020 2:04 PM
> To: Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>
> Cc: Ray Kinsella <mdr@ashroe.eu>; Richardson, Bruce <bruce.richardson@intel.com>; dev@dpdk.org;
> Trahe, Fiona <fiona.trahe@intel.com>; dev@dpdk.org; Akhil Goyal <akhil.goyal@nxp.com>
> Subject: Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
>
> 14/04/2020 14:13, Akhil Goyal:
> > Hi Ray/Thomas/Bruce,
> >
> > Could you please help review this patch wrt ABI policy?
> > >
> > > This patch adds versioned function rte_cryptodev_info_get.
> > > Node 20.05 function works the same way it was working before.
> > > Node 20.0 function strips capability added in 20.05 release
>
> What are Nodes 20.05 and 20.0?
[Fiona] 20.0 is the designation for the API before 20.05 release.
>
> > > to prevent some issues with ABI policy. To do that new capability
> > > array is allocated per device and returned to user instead of the
> > > original array passed by PMD.
>
> rte_cryptodev_info is provided by the caller of the function.
> I don't understand what you explained above, and the code is missing
> some comments explaining the reason of the logic.
>
> I still don't know whether this patch or not because it is missing
> some clear explanations.
[Fiona] We've identified that we also need to version the rte_cryptodev_capability_get() API
as it calls rte_cryptodev_info_get(). And the version handling oly works
on direct API calls, but not on calls within the API.
We will send a v2 to handle this.
And will add more explanation in the comments.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
2020-04-14 12:13 3% ` Akhil Goyal
@ 2020-04-14 13:54 0% ` Ray Kinsella
2020-04-14 18:27 5% ` Trahe, Fiona
1 sibling, 1 reply; 200+ results
From: Ray Kinsella @ 2020-04-14 13:54 UTC (permalink / raw)
To: dev
On 18/03/2020 20:41, Arek Kusztal wrote:
> This patch adds versioned function rte_cryptodev_info_get.
> Node 20.05 function works the same way it was working before.
> Node 20.0 function strips capability added in 20.05 release
> to prevent some issues with ABI policy. To do that new capability
> array is allocated per device and returned to user instead of the
> original array passed by PMD.
>
> Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
> ---
> lib/librte_cryptodev/rte_cryptodev.c | 97 +++++++++++++++++++++++++-
> lib/librte_cryptodev/rte_cryptodev.h | 12 +++-
> lib/librte_cryptodev/rte_cryptodev_version.map | 7 ++
> 3 files changed, 113 insertions(+), 3 deletions(-)
>
> diff --git a/lib/librte_cryptodev/rte_cryptodev.c b/lib/librte_cryptodev/rte_cryptodev.c
> index 6d1d0e9..2d030ba 100644
> --- a/lib/librte_cryptodev/rte_cryptodev.c
> +++ b/lib/librte_cryptodev/rte_cryptodev.c
> @@ -41,6 +41,9 @@
> #include "rte_cryptodev.h"
> #include "rte_cryptodev_pmd.h"
>
> +#include <rte_compat.h>
> +#include <rte_function_versioning.h>
> +
> static uint8_t nb_drivers;
>
> static struct rte_cryptodev rte_crypto_devices[RTE_CRYPTO_MAX_DEVS];
> @@ -56,6 +59,13 @@ static struct rte_cryptodev_global cryptodev_globals = {
> /* spinlock for crypto device callbacks */
> static rte_spinlock_t rte_cryptodev_cb_lock = RTE_SPINLOCK_INITIALIZER;
>
> +static const struct rte_cryptodev_capabilities
> + cryptodev_undefined_capabilities[] = {
> + RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST()
> +};
> +
> +struct rte_cryptodev_capabilities *capability_copies[RTE_CRYPTO_MAX_DEVS];
> +uint8_t is_capability_checked[RTE_CRYPTO_MAX_DEVS];
>
> /**
> * The user application callback description.
> @@ -999,6 +1009,13 @@ rte_cryptodev_close(uint8_t dev_id)
> RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_close, -ENOTSUP);
> retval = (*dev->dev_ops->dev_close)(dev);
>
> +
> + if (capability_copies[dev_id]) {
> + free(capability_copies[dev_id]);
> + capability_copies[dev_id] = NULL;
> + }
> + is_capability_checked[dev_id] = 0;
> +
> if (retval < 0)
> return retval;
>
> @@ -1111,11 +1128,12 @@ rte_cryptodev_stats_reset(uint8_t dev_id)
> (*dev->dev_ops->stats_reset)(dev);
> }
>
> -
> void
> -rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
> +rte_cryptodev_info_get_v20(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
> {
> struct rte_cryptodev *dev;
> + const struct rte_cryptodev_capabilities *capability;
> + uint8_t counter = 0;
>
> if (!rte_cryptodev_pmd_is_valid_dev(dev_id)) {
> CDEV_LOG_ERR("Invalid dev_id=%d", dev_id);
> @@ -1129,10 +1147,85 @@ rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
> RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
> (*dev->dev_ops->dev_infos_get)(dev, dev_info);
>
> + if (capability_copies[dev_id] == NULL) {
> + if (!is_capability_checked[dev_id]) {
> + uint8_t found_invalid_capa = 0;
> +
> + for (capability = dev_info->capabilities;
> + capability->op != RTE_CRYPTO_OP_TYPE_UNDEFINED;
> + ++capability, ++counter) {
> + if (capability->op == RTE_CRYPTO_OP_TYPE_SYMMETRIC &&
> + capability->sym.xform_type ==
> + RTE_CRYPTO_SYM_XFORM_AEAD
> + && capability->sym.aead.algo >=
> + RTE_CRYPTO_AEAD_CHACHA20_POLY1305) {
> + found_invalid_capa = 1;
> + counter--;
> + }
> + }
> + is_capability_checked[dev_id] = 1;
> + if (found_invalid_capa) {
> + capability_copies[dev_id] = malloc(counter *
> + sizeof(struct rte_cryptodev_capabilities));
> + if (capability_copies[dev_id] == NULL) {
> + /*
> + * error case - no memory to store the trimmed list,
> + * so have to return an empty list
> + */
> + dev_info->capabilities =
> + cryptodev_undefined_capabilities;
> + is_capability_checked[dev_id] = 0;
> + } else {
> + counter = 0;
> + for (capability = dev_info->capabilities;
> + capability->op !=
> + RTE_CRYPTO_OP_TYPE_UNDEFINED;
> + capability++) {
> + if (!(capability->op ==
> + RTE_CRYPTO_OP_TYPE_SYMMETRIC
> + && capability->sym.xform_type ==
> + RTE_CRYPTO_SYM_XFORM_AEAD
> + && capability->sym.aead.algo >=
> + RTE_CRYPTO_AEAD_CHACHA20_POLY1305)) {
> + capability_copies[dev_id][counter++] =
> + *capability;
> + }
> + }
> + dev_info->capabilities =
> + capability_copies[dev_id];
> + }
> + }
> + }
> + } else
> + dev_info->capabilities = capability_copies[dev_id];
> +
> dev_info->driver_name = dev->device->driver->name;
> dev_info->device = dev->device;
> }
> +VERSION_SYMBOL(rte_cryptodev_info_get, _v20, 20.0);
> +
> +void
> +rte_cryptodev_info_get_v2005(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
should be rte_cryptodev_info_get_v2002
> +{
> + struct rte_cryptodev *dev;
>
> + if (!rte_cryptodev_pmd_is_valid_dev(dev_id)) {
> + CDEV_LOG_ERR("Invalid dev_id=%d", dev_id);
> + return;
> + }
> +
> + dev = &rte_crypto_devices[dev_id];
> +
> + memset(dev_info, 0, sizeof(struct rte_cryptodev_info));
> +
> + RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
> + (*dev->dev_ops->dev_infos_get)(dev, dev_info);
> +
> + dev_info->driver_name = dev->device->driver->name;
> + dev_info->device = dev->device;
> +}
> +MAP_STATIC_SYMBOL(void rte_cryptodev_info_get(uint8_t dev_id,
> + struct rte_cryptodev_info *dev_info), rte_cryptodev_info_get_v2005);
should be rte_cryptodev_info_get_v2002
>
> int
> rte_cryptodev_callback_register(uint8_t dev_id,
> diff --git a/lib/librte_cryptodev/rte_cryptodev.h b/lib/librte_cryptodev/rte_cryptodev.h
> index 437b8a9..06ce2f2 100644
> --- a/lib/librte_cryptodev/rte_cryptodev.h
> +++ b/lib/librte_cryptodev/rte_cryptodev.h
> @@ -24,6 +24,9 @@ extern "C" {
> #include <rte_common.h>
> #include <rte_config.h>
>
> +#include <rte_compat.h>
> +#include <rte_function_versioning.h>
> +
> extern const char **rte_cyptodev_names;
>
> /* Logging Macros */
> @@ -758,7 +761,14 @@ rte_cryptodev_stats_reset(uint8_t dev_id);
> * the last valid element has it's op field set to
> * RTE_CRYPTO_OP_TYPE_UNDEFINED.
> */
> -extern void
> +void
> +rte_cryptodev_info_get_v20(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
> +
> +void
> +rte_cryptodev_info_get_v2005(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
> +BIND_DEFAULT_SYMBOL(rte_cryptodev_info_get, _v2005, 20.05);
> +
> +void
> rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
do we still need this forward declaration?
>
>
> diff --git a/lib/librte_cryptodev/rte_cryptodev_version.map b/lib/librte_cryptodev/rte_cryptodev_version.map
> index 6e41b4b..d6c945a 100644
> --- a/lib/librte_cryptodev/rte_cryptodev_version.map
> +++ b/lib/librte_cryptodev/rte_cryptodev_version.map
> @@ -58,6 +58,13 @@ DPDK_20.0 {
> local: *;
> };
>
> +
> +DPDK_20.05 {
should be DPDK_20.0.2
> + global:
> + rte_cryptodev_info_get;
> +} DPDK_20.0;
> +
> +
> EXPERIMENTAL {
> global:
>
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
2020-04-14 13:54 0% ` Ray Kinsella
@ 2020-04-14 18:27 5% ` Trahe, Fiona
2020-04-15 17:24 5% ` Trahe, Fiona
0 siblings, 1 reply; 200+ results
From: Trahe, Fiona @ 2020-04-14 18:27 UTC (permalink / raw)
To: Ray Kinsella, dev; +Cc: Trahe, Fiona, Kusztal, ArkadiuszX
Hi Ray,
We're going to need hep to understand the numbering.
The examples here http://doc.dpdk.org/guides/contributing/abi_versioning.html#major-abi-versions
show only major numbers in the .map file.
Whereas the map files all have major.minor.
The example shows that to add a new version of an existing fn one should use the next major number,
"When an ABI change is made between major ABI versions to a given library, a new section is added to
that library’s version map describing the impending new ABI version"
so
- Old fn becomes fn_v20()
- VERSION_SYMBOL(fn, _v20, 20); - maps node 20 to fn_v20() for dynamic linking
- New fn becomes fn_v21()
- BIND_DEFAULT_SYMBOL(fn, _v21, 21); - maps new builds to fn_v21 for dynamic linking and makes fn_v21 default
- MAP_STATIC_SYMBOL(fn_proto, fn_v21); - for static linking
And the map file should have:
DPDK_21 {
global:
rte_cryptodev_info_get;
} DPDK_20;
When the ABI version moves to node 21 in DPDK20.11, the _v20 symbols can be removed.
You suggest using DPDK_20.0.2 and _v2002
Can you explain why?
In 20.0.2 - which number is minor?
And why 3 numbers?
Regards,
Fiona
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Ray Kinsella
> Sent: Tuesday, April 14, 2020 2:54 PM
> To: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
>
>
>
> On 18/03/2020 20:41, Arek Kusztal wrote:
> > This patch adds versioned function rte_cryptodev_info_get.
> > Node 20.05 function works the same way it was working before.
> > Node 20.0 function strips capability added in 20.05 release
> > to prevent some issues with ABI policy. To do that new capability
> > array is allocated per device and returned to user instead of the
> > original array passed by PMD.
> >
> > Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
> > ---
> > lib/librte_cryptodev/rte_cryptodev.c | 97 +++++++++++++++++++++++++-
> > lib/librte_cryptodev/rte_cryptodev.h | 12 +++-
> > lib/librte_cryptodev/rte_cryptodev_version.map | 7 ++
> > 3 files changed, 113 insertions(+), 3 deletions(-)
> >
> > diff --git a/lib/librte_cryptodev/rte_cryptodev.c b/lib/librte_cryptodev/rte_cryptodev.c
> > index 6d1d0e9..2d030ba 100644
> > --- a/lib/librte_cryptodev/rte_cryptodev.c
> > +++ b/lib/librte_cryptodev/rte_cryptodev.c
> > @@ -41,6 +41,9 @@
> > #include "rte_cryptodev.h"
> > #include "rte_cryptodev_pmd.h"
> >
> > +#include <rte_compat.h>
> > +#include <rte_function_versioning.h>
> > +
> > static uint8_t nb_drivers;
> >
> > static struct rte_cryptodev rte_crypto_devices[RTE_CRYPTO_MAX_DEVS];
> > @@ -56,6 +59,13 @@ static struct rte_cryptodev_global cryptodev_globals = {
> > /* spinlock for crypto device callbacks */
> > static rte_spinlock_t rte_cryptodev_cb_lock = RTE_SPINLOCK_INITIALIZER;
> >
> > +static const struct rte_cryptodev_capabilities
> > + cryptodev_undefined_capabilities[] = {
> > + RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST()
> > +};
> > +
> > +struct rte_cryptodev_capabilities *capability_copies[RTE_CRYPTO_MAX_DEVS];
> > +uint8_t is_capability_checked[RTE_CRYPTO_MAX_DEVS];
> >
> > /**
> > * The user application callback description.
> > @@ -999,6 +1009,13 @@ rte_cryptodev_close(uint8_t dev_id)
> > RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_close, -ENOTSUP);
> > retval = (*dev->dev_ops->dev_close)(dev);
> >
> > +
> > + if (capability_copies[dev_id]) {
> > + free(capability_copies[dev_id]);
> > + capability_copies[dev_id] = NULL;
> > + }
> > + is_capability_checked[dev_id] = 0;
> > +
> > if (retval < 0)
> > return retval;
> >
> > @@ -1111,11 +1128,12 @@ rte_cryptodev_stats_reset(uint8_t dev_id)
> > (*dev->dev_ops->stats_reset)(dev);
> > }
> >
> > -
> > void
> > -rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
> > +rte_cryptodev_info_get_v20(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
> > {
> > struct rte_cryptodev *dev;
> > + const struct rte_cryptodev_capabilities *capability;
> > + uint8_t counter = 0;
> >
> > if (!rte_cryptodev_pmd_is_valid_dev(dev_id)) {
> > CDEV_LOG_ERR("Invalid dev_id=%d", dev_id);
> > @@ -1129,10 +1147,85 @@ rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info
> *dev_info)
> > RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
> > (*dev->dev_ops->dev_infos_get)(dev, dev_info);
> >
> > + if (capability_copies[dev_id] == NULL) {
> > + if (!is_capability_checked[dev_id]) {
> > + uint8_t found_invalid_capa = 0;
> > +
> > + for (capability = dev_info->capabilities;
> > + capability->op != RTE_CRYPTO_OP_TYPE_UNDEFINED;
> > + ++capability, ++counter) {
> > + if (capability->op == RTE_CRYPTO_OP_TYPE_SYMMETRIC &&
> > + capability->sym.xform_type ==
> > + RTE_CRYPTO_SYM_XFORM_AEAD
> > + && capability->sym.aead.algo >=
> > + RTE_CRYPTO_AEAD_CHACHA20_POLY1305) {
> > + found_invalid_capa = 1;
> > + counter--;
> > + }
> > + }
> > + is_capability_checked[dev_id] = 1;
> > + if (found_invalid_capa) {
> > + capability_copies[dev_id] = malloc(counter *
> > + sizeof(struct rte_cryptodev_capabilities));
> > + if (capability_copies[dev_id] == NULL) {
> > + /*
> > + * error case - no memory to store the trimmed list,
> > + * so have to return an empty list
> > + */
> > + dev_info->capabilities =
> > + cryptodev_undefined_capabilities;
> > + is_capability_checked[dev_id] = 0;
> > + } else {
> > + counter = 0;
> > + for (capability = dev_info->capabilities;
> > + capability->op !=
> > + RTE_CRYPTO_OP_TYPE_UNDEFINED;
> > + capability++) {
> > + if (!(capability->op ==
> > + RTE_CRYPTO_OP_TYPE_SYMMETRIC
> > + && capability->sym.xform_type ==
> > + RTE_CRYPTO_SYM_XFORM_AEAD
> > + && capability->sym.aead.algo >=
> > +
> RTE_CRYPTO_AEAD_CHACHA20_POLY1305)) {
> > + capability_copies[dev_id][counter++] =
> > + *capability;
> > + }
> > + }
> > + dev_info->capabilities =
> > + capability_copies[dev_id];
> > + }
> > + }
> > + }
> > + } else
> > + dev_info->capabilities = capability_copies[dev_id];
> > +
> > dev_info->driver_name = dev->device->driver->name;
> > dev_info->device = dev->device;
> > }
> > +VERSION_SYMBOL(rte_cryptodev_info_get, _v20, 20.0);
> > +
> > +void
> > +rte_cryptodev_info_get_v2005(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
>
> should be rte_cryptodev_info_get_v2002
>
> > +{
> > + struct rte_cryptodev *dev;
> >
> > + if (!rte_cryptodev_pmd_is_valid_dev(dev_id)) {
> > + CDEV_LOG_ERR("Invalid dev_id=%d", dev_id);
> > + return;
> > + }
> > +
> > + dev = &rte_crypto_devices[dev_id];
> > +
> > + memset(dev_info, 0, sizeof(struct rte_cryptodev_info));
> > +
> > + RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
> > + (*dev->dev_ops->dev_infos_get)(dev, dev_info);
> > +
> > + dev_info->driver_name = dev->device->driver->name;
> > + dev_info->device = dev->device;
> > +}
> > +MAP_STATIC_SYMBOL(void rte_cryptodev_info_get(uint8_t dev_id,
> > + struct rte_cryptodev_info *dev_info), rte_cryptodev_info_get_v2005);
>
> should be rte_cryptodev_info_get_v2002
>
> >
> > int
> > rte_cryptodev_callback_register(uint8_t dev_id,
> > diff --git a/lib/librte_cryptodev/rte_cryptodev.h b/lib/librte_cryptodev/rte_cryptodev.h
> > index 437b8a9..06ce2f2 100644
> > --- a/lib/librte_cryptodev/rte_cryptodev.h
> > +++ b/lib/librte_cryptodev/rte_cryptodev.h
> > @@ -24,6 +24,9 @@ extern "C" {
> > #include <rte_common.h>
> > #include <rte_config.h>
> >
> > +#include <rte_compat.h>
> > +#include <rte_function_versioning.h>
> > +
> > extern const char **rte_cyptodev_names;
> >
> > /* Logging Macros */
> > @@ -758,7 +761,14 @@ rte_cryptodev_stats_reset(uint8_t dev_id);
> > * the last valid element has it's op field set to
> > * RTE_CRYPTO_OP_TYPE_UNDEFINED.
> > */
> > -extern void
> > +void
> > +rte_cryptodev_info_get_v20(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
> > +
> > +void
> > +rte_cryptodev_info_get_v2005(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
> > +BIND_DEFAULT_SYMBOL(rte_cryptodev_info_get, _v2005, 20.05);
> > +
> > +void
> > rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
> do we still need this forward declaration?
>
> >
> >
> > diff --git a/lib/librte_cryptodev/rte_cryptodev_version.map
> b/lib/librte_cryptodev/rte_cryptodev_version.map
> > index 6e41b4b..d6c945a 100644
> > --- a/lib/librte_cryptodev/rte_cryptodev_version.map
> > +++ b/lib/librte_cryptodev/rte_cryptodev_version.map
> > @@ -58,6 +58,13 @@ DPDK_20.0 {
> > local: *;
> > };
> >
> > +
> > +DPDK_20.05 {
>
> should be DPDK_20.0.2
>
> > + global:
> > + rte_cryptodev_info_get;
> > +} DPDK_20.0;
> > +
> > +
> > EXPERIMENTAL {
> > global:
> >
> >
^ permalink raw reply [relevance 5%]
* [dpdk-dev] [PATCH v3 08/10] eal/windows: fix rte_page_sizes with Clang on Windows
@ 2020-04-14 19:44 4% ` Dmitry Kozlyuk
2020-04-15 9:34 0% ` Jerin Jacob
0 siblings, 1 reply; 200+ results
From: Dmitry Kozlyuk @ 2020-04-14 19:44 UTC (permalink / raw)
To: dev
Cc: Dmitry Malloy (MESHCHANINOV),
Narcisa Ana Maria Vasile, Fady Bader, Tal Shnaiderman,
Dmitry Kozlyuk, Anatoly Burakov
Clang on Windows follows MS ABI where enum values are limited to 2^31-1.
Enum rte_page_size has members valued above this limit, which get
wrapped to zero, resulting in compilation error (duplicate values in
enum). Using MS ABI is mandatory for Windows EAL to call Win32 APIs.
Define these values outside of the enum for Clang on Windows only.
This does not affect runtime, because Windows doesn't run on machines
with 4GiB and 16GiB hugepages.
Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
lib/librte_eal/include/rte_memory.h | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/lib/librte_eal/include/rte_memory.h b/lib/librte_eal/include/rte_memory.h
index 1b7c3e5df..3ec673f51 100644
--- a/lib/librte_eal/include/rte_memory.h
+++ b/lib/librte_eal/include/rte_memory.h
@@ -34,8 +34,14 @@ enum rte_page_sizes {
RTE_PGSIZE_256M = 1ULL << 28,
RTE_PGSIZE_512M = 1ULL << 29,
RTE_PGSIZE_1G = 1ULL << 30,
+/* Work around Clang on Windows being limited to 32-bit underlying type. */
+#if !defined(RTE_CC_CLANG) || !defined(RTE_EXEC_ENV_WINDOWS)
RTE_PGSIZE_4G = 1ULL << 32,
RTE_PGSIZE_16G = 1ULL << 34,
+#else
+#define RTE_PGSIZE_4G (1ULL << 32)
+#define RTE_PGSIZE_16G (1ULL << 34)
+#endif
};
#define SOCKET_ID_ANY -1 /**< Any NUMA socket. */
--
2.25.1
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [dpdk-dev v4 1/3] ethdev: add new RSS offload types
2020-04-14 9:42 3% ` Ori Kam
@ 2020-04-15 2:31 0% ` Zhang, Qi Z
2020-04-15 3:11 3% ` Jeff Guo
0 siblings, 1 reply; 200+ results
From: Zhang, Qi Z @ 2020-04-15 2:31 UTC (permalink / raw)
To: Ori Kam, Guo, Jia, Iremonger, Bernard, Ye, Xiaolong
Cc: dev, Wu, Jingjing, Cao, Yahui, Su, Simei
> -----Original Message-----
> From: Ori Kam <orika@mellanox.com>
> Sent: Tuesday, April 14, 2020 5:43 PM
> To: Guo, Jia <jia.guo@intel.com>; Iremonger, Bernard
> <bernard.iremonger@intel.com>; Ye, Xiaolong <xiaolong.ye@intel.com>;
> Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Wu, Jingjing <jingjing.wu@intel.com>; Cao, Yahui
> <yahui.cao@intel.com>; Su, Simei <simei.su@intel.com>
> Subject: RE: [dpdk-dev v4 1/3] ethdev: add new RSS offload types
>
> Hi Jeff,
> PSB
>
> Best,
> Ori
> > -----Original Message-----
> > From: Jeff Guo <jia.guo@intel.com>
> > Sent: Tuesday, April 14, 2020 8:42 PM
> > To: Ori Kam <orika@mellanox.com>; bernard.iremonger@intel.com;
> > xiaolong.ye@intel.com; qi.z.zhang@intel.com
> > Cc: dev@dpdk.org; jingjing.wu@intel.com; yahui.cao@intel.com;
> > simei.su@intel.com; jia.guo@intel.com
> > Subject: [dpdk-dev v4 1/3] ethdev: add new RSS offload types
> >
> > Defines some new RSS offload types for ETH/SVLAN/CVLAN/GTPU/L2TPV3/
> > ESP/AH/PFCP.
> >
> > Signed-off-by: Jeff Guo <jia.guo@intel.com>
> > Reviewed-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> > v4->v3:
> > no change
> > ---
> > lib/librte_ethdev/rte_ethdev.h | 11 ++++++++++-
> > 1 file changed, 10 insertions(+), 1 deletion(-)
> >
> > diff --git a/lib/librte_ethdev/rte_ethdev.h
> > b/lib/librte_ethdev/rte_ethdev.h index d1a593ad1..efe705ff0 100644
> > --- a/lib/librte_ethdev/rte_ethdev.h
> > +++ b/lib/librte_ethdev/rte_ethdev.h
> > @@ -511,6 +511,13 @@ struct rte_eth_rss_conf {
> > #define ETH_RSS_GENEVE (1ULL << 20)
> > #define ETH_RSS_NVGRE (1ULL << 21)
> > #define ETH_RSS_GTPU (1ULL << 23)
> > +#define ETH_RSS_ETH (1ULL << 24)
> > +#define ETH_RSS_S_VLAN (1ULL << 25)
> > +#define ETH_RSS_C_VLAN (1ULL << 26)
> > +#define ETH_RSS_ESP (1ULL << 27)
> > +#define ETH_RSS_AH (1ULL << 28)
> > +#define ETH_RSS_L2TPV3 (1ULL << 29)
> > +#define ETH_RSS_PFCP (1ULL << 30)
> >
> > /*
> > * We use the following macros to combine with above ETH_RSS_* for
> @@
> > -524,7 +531,9 @@ struct rte_eth_rss_conf {
> > #define ETH_RSS_L3_SRC_ONLY (1ULL << 63)
> > #define ETH_RSS_L3_DST_ONLY (1ULL << 62)
> > #define ETH_RSS_L4_SRC_ONLY (1ULL << 61)
> > -#define ETH_RSS_L4_DST_ONLY (1ULL << 60)
> > +#define ETH_RSS_L4_DST_ONLY (1ULL << 60)
> > +#define ETH_RSS_ETH_SRC_ONLY (1ULL << 59)
> > +#define ETH_RSS_ETH_DST_ONLY (1ULL << 58)
> >
> Why not change to L2_RSS_SRC_ONLY? Same for DST?
> Also if it is not ABI/API change I would suggest to move the ETH defines to be
> 63.
I think reorder will break Application which already use 19.11.
>
> > /**
> > * For input set change of hash filter, if SRC_ONLY and DST_ONLY of
> > --
> > 2.20.1
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [dpdk-dev v4 1/3] ethdev: add new RSS offload types
2020-04-15 2:31 0% ` Zhang, Qi Z
@ 2020-04-15 3:11 3% ` Jeff Guo
0 siblings, 0 replies; 200+ results
From: Jeff Guo @ 2020-04-15 3:11 UTC (permalink / raw)
To: Zhang, Qi Z, Ori Kam, Iremonger, Bernard, Ye, Xiaolong
Cc: dev, Wu, Jingjing, Cao, Yahui, Su, Simei
Ori and qi
On 4/15/2020 10:31 AM, Zhang, Qi Z wrote:
>
>> -----Original Message-----
>> From: Ori Kam <orika@mellanox.com>
>> Sent: Tuesday, April 14, 2020 5:43 PM
>> To: Guo, Jia <jia.guo@intel.com>; Iremonger, Bernard
>> <bernard.iremonger@intel.com>; Ye, Xiaolong <xiaolong.ye@intel.com>;
>> Zhang, Qi Z <qi.z.zhang@intel.com>
>> Cc: dev@dpdk.org; Wu, Jingjing <jingjing.wu@intel.com>; Cao, Yahui
>> <yahui.cao@intel.com>; Su, Simei <simei.su@intel.com>
>> Subject: RE: [dpdk-dev v4 1/3] ethdev: add new RSS offload types
>>
>> Hi Jeff,
>> PSB
>>
>> Best,
>> Ori
>>> -----Original Message-----
>>> From: Jeff Guo <jia.guo@intel.com>
>>> Sent: Tuesday, April 14, 2020 8:42 PM
>>> To: Ori Kam <orika@mellanox.com>; bernard.iremonger@intel.com;
>>> xiaolong.ye@intel.com; qi.z.zhang@intel.com
>>> Cc: dev@dpdk.org; jingjing.wu@intel.com; yahui.cao@intel.com;
>>> simei.su@intel.com; jia.guo@intel.com
>>> Subject: [dpdk-dev v4 1/3] ethdev: add new RSS offload types
>>>
>>> Defines some new RSS offload types for ETH/SVLAN/CVLAN/GTPU/L2TPV3/
>>> ESP/AH/PFCP.
>>>
>>> Signed-off-by: Jeff Guo <jia.guo@intel.com>
>>> Reviewed-by: Qi Zhang <qi.z.zhang@intel.com>
>>> ---
>>> v4->v3:
>>> no change
>>> ---
>>> lib/librte_ethdev/rte_ethdev.h | 11 ++++++++++-
>>> 1 file changed, 10 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/lib/librte_ethdev/rte_ethdev.h
>>> b/lib/librte_ethdev/rte_ethdev.h index d1a593ad1..efe705ff0 100644
>>> --- a/lib/librte_ethdev/rte_ethdev.h
>>> +++ b/lib/librte_ethdev/rte_ethdev.h
>>> @@ -511,6 +511,13 @@ struct rte_eth_rss_conf {
>>> #define ETH_RSS_GENEVE (1ULL << 20)
>>> #define ETH_RSS_NVGRE (1ULL << 21)
>>> #define ETH_RSS_GTPU (1ULL << 23)
>>> +#define ETH_RSS_ETH (1ULL << 24)
>>> +#define ETH_RSS_S_VLAN (1ULL << 25)
>>> +#define ETH_RSS_C_VLAN (1ULL << 26)
>>> +#define ETH_RSS_ESP (1ULL << 27)
>>> +#define ETH_RSS_AH (1ULL << 28)
>>> +#define ETH_RSS_L2TPV3 (1ULL << 29)
>>> +#define ETH_RSS_PFCP (1ULL << 30)
>>>
>>> /*
>>> * We use the following macros to combine with above ETH_RSS_* for
>> @@
>>> -524,7 +531,9 @@ struct rte_eth_rss_conf {
>>> #define ETH_RSS_L3_SRC_ONLY (1ULL << 63)
>>> #define ETH_RSS_L3_DST_ONLY (1ULL << 62)
>>> #define ETH_RSS_L4_SRC_ONLY (1ULL << 61)
>>> -#define ETH_RSS_L4_DST_ONLY (1ULL << 60)
>>> +#define ETH_RSS_L4_DST_ONLY (1ULL << 60)
>>> +#define ETH_RSS_ETH_SRC_ONLY (1ULL << 59)
>>> +#define ETH_RSS_ETH_DST_ONLY (1ULL << 58)
>>>
>> Why not change to L2_RSS_SRC_ONLY? Same for DST?
>> Also if it is not ABI/API change I would suggest to move the ETH defines to be
>> 63.
> I think reorder will break Application which already use 19.11.
Will rename it to ETH_RSS_L2_XXX and keep the order which will not break
the ABI/API, if no objection. Thanks.
>>> /**
>>> * For input set change of hash filter, if SRC_ONLY and DST_ONLY of
>>> --
>>> 2.20.1
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v3 08/10] eal/windows: fix rte_page_sizes with Clang on Windows
2020-04-14 19:44 4% ` [dpdk-dev] [PATCH v3 08/10] eal/windows: fix rte_page_sizes with Clang on Windows Dmitry Kozlyuk
@ 2020-04-15 9:34 0% ` Jerin Jacob
2020-04-15 10:32 0% ` Dmitry Kozlyuk
0 siblings, 1 reply; 200+ results
From: Jerin Jacob @ 2020-04-15 9:34 UTC (permalink / raw)
To: Dmitry Kozlyuk
Cc: dpdk-dev, Dmitry Malloy (MESHCHANINOV),
Narcisa Ana Maria Vasile, Fady Bader, Tal Shnaiderman,
Anatoly Burakov
On Wed, Apr 15, 2020 at 1:16 AM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
>
> Clang on Windows follows MS ABI where enum values are limited to 2^31-1.
> Enum rte_page_size has members valued above this limit, which get
> wrapped to zero, resulting in compilation error (duplicate values in
> enum). Using MS ABI is mandatory for Windows EAL to call Win32 APIs.
>
> Define these values outside of the enum for Clang on Windows only.
> This does not affect runtime, because Windows doesn't run on machines
> with 4GiB and 16GiB hugepages.
>
> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> ---
> lib/librte_eal/include/rte_memory.h | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/lib/librte_eal/include/rte_memory.h b/lib/librte_eal/include/rte_memory.h
> index 1b7c3e5df..3ec673f51 100644
> --- a/lib/librte_eal/include/rte_memory.h
> +++ b/lib/librte_eal/include/rte_memory.h
> @@ -34,8 +34,14 @@ enum rte_page_sizes {
> RTE_PGSIZE_256M = 1ULL << 28,
> RTE_PGSIZE_512M = 1ULL << 29,
> RTE_PGSIZE_1G = 1ULL << 30,
> +/* Work around Clang on Windows being limited to 32-bit underlying type. */
It does look like "enum rte_page_sizes" NOT used as enum anywhere.
[master][dpdk.org] $ grep -ri "enum rte_page_sizes" lib/
lib/librte_eal/include/rte_memory.h:enum rte_page_sizes {
Why not remove this workaround and define all items as #define to
avoid below ifdef clutter.
> +#if !defined(RTE_CC_CLANG) || !defined(RTE_EXEC_ENV_WINDOWS)
See above.
> RTE_PGSIZE_4G = 1ULL << 32,
> RTE_PGSIZE_16G = 1ULL << 34,
> +#else
> +#define RTE_PGSIZE_4G (1ULL << 32)
> +#define RTE_PGSIZE_16G (1ULL << 34)
> +#endif
> };
>
> #define SOCKET_ID_ANY -1 /**< Any NUMA socket. */
> --
> 2.25.1
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v3 08/10] eal/windows: fix rte_page_sizes with Clang on Windows
2020-04-15 9:34 0% ` Jerin Jacob
@ 2020-04-15 10:32 0% ` Dmitry Kozlyuk
2020-04-15 10:57 3% ` Jerin Jacob
0 siblings, 1 reply; 200+ results
From: Dmitry Kozlyuk @ 2020-04-15 10:32 UTC (permalink / raw)
To: Jerin Jacob
Cc: dpdk-dev, Dmitry Malloy (MESHCHANINOV),
Narcisa Ana Maria Vasile, Fady Bader, Tal Shnaiderman,
Anatoly Burakov
> On Wed, Apr 15, 2020 at 1:16 AM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
> >
> > Clang on Windows follows MS ABI where enum values are limited to 2^31-1.
> > Enum rte_page_size has members valued above this limit, which get
> > wrapped to zero, resulting in compilation error (duplicate values in
> > enum). Using MS ABI is mandatory for Windows EAL to call Win32 APIs.
> >
> > Define these values outside of the enum for Clang on Windows only.
> > This does not affect runtime, because Windows doesn't run on machines
> > with 4GiB and 16GiB hugepages.
> >
> > Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> > ---
> > lib/librte_eal/include/rte_memory.h | 6 ++++++
> > 1 file changed, 6 insertions(+)
> >
> > diff --git a/lib/librte_eal/include/rte_memory.h b/lib/librte_eal/include/rte_memory.h
> > index 1b7c3e5df..3ec673f51 100644
> > --- a/lib/librte_eal/include/rte_memory.h
> > +++ b/lib/librte_eal/include/rte_memory.h
> > @@ -34,8 +34,14 @@ enum rte_page_sizes {
> > RTE_PGSIZE_256M = 1ULL << 28,
> > RTE_PGSIZE_512M = 1ULL << 29,
> > RTE_PGSIZE_1G = 1ULL << 30,
> > +/* Work around Clang on Windows being limited to 32-bit underlying type. */
>
> It does look like "enum rte_page_sizes" NOT used as enum anywhere.
>
> [master][dpdk.org] $ grep -ri "enum rte_page_sizes" lib/
> lib/librte_eal/include/rte_memory.h:enum rte_page_sizes {
>
> Why not remove this workaround and define all items as #define to
> avoid below ifdef clutter.
>
> > +#if !defined(RTE_CC_CLANG) || !defined(RTE_EXEC_ENV_WINDOWS)
>
> See above.
>
> > RTE_PGSIZE_4G = 1ULL << 32,
> > RTE_PGSIZE_16G = 1ULL << 34,
> > +#else
> > +#define RTE_PGSIZE_4G (1ULL << 32)
> > +#define RTE_PGSIZE_16G (1ULL << 34)
> > +#endif
> > };
> >
> > #define SOCKET_ID_ANY -1 /**< Any NUMA socket. */
> > --
> > 2.25.1
> >
This is a public header and removing enum rte_page_sizes will break API.
Moving members out of enum while keeping enum itself might break compilation
because of integer constants being converted to enum (with -Werror).
--
Dmitry Kozlyuk
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v3 08/10] eal/windows: fix rte_page_sizes with Clang on Windows
2020-04-15 10:32 0% ` Dmitry Kozlyuk
@ 2020-04-15 10:57 3% ` Jerin Jacob
2020-04-15 11:09 0% ` Dmitry Kozlyuk
0 siblings, 1 reply; 200+ results
From: Jerin Jacob @ 2020-04-15 10:57 UTC (permalink / raw)
To: Dmitry Kozlyuk
Cc: dpdk-dev, Dmitry Malloy (MESHCHANINOV),
Narcisa Ana Maria Vasile, Fady Bader, Tal Shnaiderman,
Anatoly Burakov, Ray Kinsella
On Wed, Apr 15, 2020 at 4:02 PM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
>
> > On Wed, Apr 15, 2020 at 1:16 AM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
> > >
> > > Clang on Windows follows MS ABI where enum values are limited to 2^31-1.
> > > Enum rte_page_size has members valued above this limit, which get
> > > wrapped to zero, resulting in compilation error (duplicate values in
> > > enum). Using MS ABI is mandatory for Windows EAL to call Win32 APIs.
> > >
> > > Define these values outside of the enum for Clang on Windows only.
> > > This does not affect runtime, because Windows doesn't run on machines
> > > with 4GiB and 16GiB hugepages.
> > >
> > > Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> > > ---
> > > lib/librte_eal/include/rte_memory.h | 6 ++++++
> > > 1 file changed, 6 insertions(+)
> > >
> > > diff --git a/lib/librte_eal/include/rte_memory.h b/lib/librte_eal/include/rte_memory.h
> > > index 1b7c3e5df..3ec673f51 100644
> > > --- a/lib/librte_eal/include/rte_memory.h
> > > +++ b/lib/librte_eal/include/rte_memory.h
> > > @@ -34,8 +34,14 @@ enum rte_page_sizes {
> > > RTE_PGSIZE_256M = 1ULL << 28,
> > > RTE_PGSIZE_512M = 1ULL << 29,
> > > RTE_PGSIZE_1G = 1ULL << 30,
> > > +/* Work around Clang on Windows being limited to 32-bit underlying type. */
> >
> > It does look like "enum rte_page_sizes" NOT used as enum anywhere.
> >
> > [master][dpdk.org] $ grep -ri "enum rte_page_sizes" lib/
> > lib/librte_eal/include/rte_memory.h:enum rte_page_sizes {
> >
> > Why not remove this workaround and define all items as #define to
> > avoid below ifdef clutter.
> >
> > > +#if !defined(RTE_CC_CLANG) || !defined(RTE_EXEC_ENV_WINDOWS)
> >
> > See above.
> >
> > > RTE_PGSIZE_4G = 1ULL << 32,
> > > RTE_PGSIZE_16G = 1ULL << 34,
> > > +#else
> > > +#define RTE_PGSIZE_4G (1ULL << 32)
> > > +#define RTE_PGSIZE_16G (1ULL << 34)
> > > +#endif
> > > };
> > >
> > > #define SOCKET_ID_ANY -1 /**< Any NUMA socket. */
> > > --
> > > 2.25.1
> > >
>
> This is a public header and removing enum rte_page_sizes will break API.
> Moving members out of enum while keeping enum itself might break compilation
> because of integer constants being converted to enum (with -Werror).
If none of the public API is using this enum then I think, we may not
need to make this enum as public.
Since it has ULL, I believe both cases(enum or define), it will be
treated as unsigned long long. ie. NO ABI breakage.
>
> --
> Dmitry Kozlyuk
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v3 08/10] eal/windows: fix rte_page_sizes with Clang on Windows
2020-04-15 10:57 3% ` Jerin Jacob
@ 2020-04-15 11:09 0% ` Dmitry Kozlyuk
2020-04-15 11:17 4% ` Jerin Jacob
0 siblings, 1 reply; 200+ results
From: Dmitry Kozlyuk @ 2020-04-15 11:09 UTC (permalink / raw)
To: Jerin Jacob
Cc: dpdk-dev, Dmitry Malloy (MESHCHANINOV),
Narcisa Ana Maria Vasile, Fady Bader, Tal Shnaiderman,
Anatoly Burakov, Ray Kinsella
> On Wed, Apr 15, 2020 at 4:02 PM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
> >
> > > On Wed, Apr 15, 2020 at 1:16 AM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
> > > >
> > > > Clang on Windows follows MS ABI where enum values are limited to 2^31-1.
> > > > Enum rte_page_size has members valued above this limit, which get
> > > > wrapped to zero, resulting in compilation error (duplicate values in
> > > > enum). Using MS ABI is mandatory for Windows EAL to call Win32 APIs.
> > > >
> > > > Define these values outside of the enum for Clang on Windows only.
> > > > This does not affect runtime, because Windows doesn't run on machines
> > > > with 4GiB and 16GiB hugepages.
> > > >
> > > > Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> > > > ---
> > > > lib/librte_eal/include/rte_memory.h | 6 ++++++
> > > > 1 file changed, 6 insertions(+)
> > > >
> > > > diff --git a/lib/librte_eal/include/rte_memory.h b/lib/librte_eal/include/rte_memory.h
> > > > index 1b7c3e5df..3ec673f51 100644
> > > > --- a/lib/librte_eal/include/rte_memory.h
> > > > +++ b/lib/librte_eal/include/rte_memory.h
> > > > @@ -34,8 +34,14 @@ enum rte_page_sizes {
> > > > RTE_PGSIZE_256M = 1ULL << 28,
> > > > RTE_PGSIZE_512M = 1ULL << 29,
> > > > RTE_PGSIZE_1G = 1ULL << 30,
> > > > +/* Work around Clang on Windows being limited to 32-bit underlying type. */
> > >
> > > It does look like "enum rte_page_sizes" NOT used as enum anywhere.
> > >
> > > [master][dpdk.org] $ grep -ri "enum rte_page_sizes" lib/
> > > lib/librte_eal/include/rte_memory.h:enum rte_page_sizes {
> > >
> > > Why not remove this workaround and define all items as #define to
> > > avoid below ifdef clutter.
> > >
> > > > +#if !defined(RTE_CC_CLANG) || !defined(RTE_EXEC_ENV_WINDOWS)
> > >
> > > See above.
> > >
> > > > RTE_PGSIZE_4G = 1ULL << 32,
> > > > RTE_PGSIZE_16G = 1ULL << 34,
> > > > +#else
> > > > +#define RTE_PGSIZE_4G (1ULL << 32)
> > > > +#define RTE_PGSIZE_16G (1ULL << 34)
> > > > +#endif
> > > > };
> > > >
> > > > #define SOCKET_ID_ANY -1 /**< Any NUMA socket. */
> > > > --
> > > > 2.25.1
> > > >
> >
> > This is a public header and removing enum rte_page_sizes will break API.
> > Moving members out of enum while keeping enum itself might break compilation
> > because of integer constants being converted to enum (with -Werror).
>
> If none of the public API is using this enum then I think, we may not
> need to make this enum as public.
Agreed.
> Since it has ULL, I believe both cases(enum or define), it will be
> treated as unsigned long long. ie. NO ABI breakage.
I was talking about API only (compile-time compatibility). Getting rid of
#ifdef and workarounds sounds right, we'll just need a notice in release
notes.
--
Dmitry Kozlyuk
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v3 08/10] eal/windows: fix rte_page_sizes with Clang on Windows
2020-04-15 11:09 0% ` Dmitry Kozlyuk
@ 2020-04-15 11:17 4% ` Jerin Jacob
0 siblings, 0 replies; 200+ results
From: Jerin Jacob @ 2020-04-15 11:17 UTC (permalink / raw)
To: Dmitry Kozlyuk
Cc: dpdk-dev, Dmitry Malloy (MESHCHANINOV),
Narcisa Ana Maria Vasile, Fady Bader, Tal Shnaiderman,
Anatoly Burakov, Ray Kinsella
On Wed, Apr 15, 2020 at 4:39 PM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
>
>
>
> > On Wed, Apr 15, 2020 at 4:02 PM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
> > >
> > > > On Wed, Apr 15, 2020 at 1:16 AM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
> > > > >
> > > > > Clang on Windows follows MS ABI where enum values are limited to 2^31-1.
> > > > > Enum rte_page_size has members valued above this limit, which get
> > > > > wrapped to zero, resulting in compilation error (duplicate values in
> > > > > enum). Using MS ABI is mandatory for Windows EAL to call Win32 APIs.
> > > > >
> > > > > Define these values outside of the enum for Clang on Windows only.
> > > > > This does not affect runtime, because Windows doesn't run on machines
> > > > > with 4GiB and 16GiB hugepages.
> > > > >
> > > > > Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> > > > > ---
> > > > > lib/librte_eal/include/rte_memory.h | 6 ++++++
> > > > > 1 file changed, 6 insertions(+)
> > > > >
> > > > > diff --git a/lib/librte_eal/include/rte_memory.h b/lib/librte_eal/include/rte_memory.h
> > > > > index 1b7c3e5df..3ec673f51 100644
> > > > > --- a/lib/librte_eal/include/rte_memory.h
> > > > > +++ b/lib/librte_eal/include/rte_memory.h
> > > > > @@ -34,8 +34,14 @@ enum rte_page_sizes {
> > > > > RTE_PGSIZE_256M = 1ULL << 28,
> > > > > RTE_PGSIZE_512M = 1ULL << 29,
> > > > > RTE_PGSIZE_1G = 1ULL << 30,
> > > > > +/* Work around Clang on Windows being limited to 32-bit underlying type. */
> > > >
> > > > It does look like "enum rte_page_sizes" NOT used as enum anywhere.
> > > >
> > > > [master][dpdk.org] $ grep -ri "enum rte_page_sizes" lib/
> > > > lib/librte_eal/include/rte_memory.h:enum rte_page_sizes {
> > > >
> > > > Why not remove this workaround and define all items as #define to
> > > > avoid below ifdef clutter.
> > > >
> > > > > +#if !defined(RTE_CC_CLANG) || !defined(RTE_EXEC_ENV_WINDOWS)
> > > >
> > > > See above.
> > > >
> > > > > RTE_PGSIZE_4G = 1ULL << 32,
> > > > > RTE_PGSIZE_16G = 1ULL << 34,
> > > > > +#else
> > > > > +#define RTE_PGSIZE_4G (1ULL << 32)
> > > > > +#define RTE_PGSIZE_16G (1ULL << 34)
> > > > > +#endif
> > > > > };
> > > > >
> > > > > #define SOCKET_ID_ANY -1 /**< Any NUMA socket. */
> > > > > --
> > > > > 2.25.1
> > > > >
> > >
> > > This is a public header and removing enum rte_page_sizes will break API.
> > > Moving members out of enum while keeping enum itself might break compilation
> > > because of integer constants being converted to enum (with -Werror).
> >
> > If none of the public API is using this enum then I think, we may not
> > need to make this enum as public.
>
> Agreed.
>
> > Since it has ULL, I believe both cases(enum or define), it will be
> > treated as unsigned long long. ie. NO ABI breakage.
>
> I was talking about API only (compile-time compatibility). Getting rid of
> #ifdef and workarounds sounds right, we'll just need a notice in release
> notes.
Good to check ./devtools/check-abi.sh for any ABI breakage.
>
> --
> Dmitry Kozlyuk
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v4 6/8] eal/x86: identify AVX512 extensions flag
@ 2020-04-15 13:31 3% ` David Marchand
2020-04-15 14:57 0% ` Liu, Yong
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-04-15 13:31 UTC (permalink / raw)
To: Marvin Liu
Cc: Maxime Coquelin, Xiaolong Ye, Zhihong Wang, Van Haaren Harry,
dev, Kevin Laatz, Kinsella, Ray
On Wed, Apr 15, 2020 at 11:14 AM Marvin Liu <yong.liu@intel.com> wrote:
>
> Read CPUID to check if AVX512 extensions are supported.
>
> Signed-off-by: Marvin Liu <yong.liu@intel.com>
>
> diff --git a/lib/librte_eal/common/arch/x86/rte_cpuflags.c b/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> index 6492df556..54e9f6185 100644
> --- a/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> +++ b/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> @@ -109,6 +109,9 @@ const struct feature_entry rte_cpu_feature_table[] = {
> FEAT_DEF(RTM, 0x00000007, 0, RTE_REG_EBX, 11)
> FEAT_DEF(AVX512F, 0x00000007, 0, RTE_REG_EBX, 16)
> FEAT_DEF(RDSEED, 0x00000007, 0, RTE_REG_EBX, 18)
> + FEAT_DEF(AVX512CD, 0x00000007, 0, RTE_REG_EBX, 28)
> + FEAT_DEF(AVX512BW, 0x00000007, 0, RTE_REG_EBX, 30)
> + FEAT_DEF(AVX512VL, 0x00000007, 0, RTE_REG_EBX, 31)
>
> FEAT_DEF(LAHF_SAHF, 0x80000001, 0, RTE_REG_ECX, 0)
> FEAT_DEF(LZCNT, 0x80000001, 0, RTE_REG_ECX, 4)
> diff --git a/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h b/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> index 25ba47b96..5bf99e05f 100644
> --- a/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> +++ b/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> @@ -98,6 +98,9 @@ enum rte_cpu_flag_t {
> RTE_CPUFLAG_RTM, /**< Transactional memory */
> RTE_CPUFLAG_AVX512F, /**< AVX512F */
> RTE_CPUFLAG_RDSEED, /**< RDSEED instruction */
> + RTE_CPUFLAG_AVX512CD, /**< AVX512CD */
> + RTE_CPUFLAG_AVX512BW, /**< AVX512BW */
> + RTE_CPUFLAG_AVX512VL, /**< AVX512VL */
>
> /* (EAX 80000001h) ECX features */
> RTE_CPUFLAG_LAHF_SAHF, /**< LAHF_SAHF */
This patch most likely breaks the ABI (renumbering flags after
RTE_CPUFLAG_LAHF_SAHF).
This change should not go through the virtio tree and is not rebased on master.
A similar patch had been proposed by Kevin:
http://patchwork.dpdk.org/patch/67438/
--
David Marchand
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH 1/4] vhost: inroduce operation to get vDPA queue stats
@ 2020-04-15 14:36 3% ` Maxime Coquelin
2020-04-16 9:06 3% ` Matan Azrad
0 siblings, 1 reply; 200+ results
From: Maxime Coquelin @ 2020-04-15 14:36 UTC (permalink / raw)
To: Matan Azrad, dev; +Cc: Viacheslav Ovsiienko, Shahaf Shuler
Hi Matan,
On 4/2/20 1:26 PM, Matan Azrad wrote:
> The vDPA device offloads all the datapath of the vhost device to the HW
> device.
>
> In order to expose to the user traffic information this patch introduce
> new API to get traffic statistics per virtio queue.
>
> The statistics are taken directly from the vDPA driver managing the HW
> device.
>
> Signed-off-by: Matan Azrad <matan@mellanox.com>
> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> ---
> doc/guides/rel_notes/release_20_05.rst | 4 +++
> doc/guides/vdpadevs/features/default.ini | 1 +
> doc/guides/vdpadevs/features_overview.rst | 3 +++
> lib/librte_vhost/rte_vdpa.h | 45 ++++++++++++++++++++++++++++++-
> lib/librte_vhost/rte_vhost_version.map | 1 +
> lib/librte_vhost/vdpa.c | 14 ++++++++++
> 6 files changed, 67 insertions(+), 1 deletion(-)
...
> diff --git a/lib/librte_vhost/rte_vdpa.h b/lib/librte_vhost/rte_vdpa.h
> index 9a3deb3..d6cbf48 100644
> --- a/lib/librte_vhost/rte_vdpa.h
> +++ b/lib/librte_vhost/rte_vdpa.h
> @@ -37,6 +37,27 @@ struct rte_vdpa_dev_addr {
> };
> };
>
> +struct rte_vdpa_queue_stats {
> + /** Number of descriptors received by device */
> + uint64_t received_desc;
> + /** Number of descriptors completed by the device */
> + uint64_t completed_desc;
> + /** Number of bad descriptors received by device */
> + uint32_t bad_desc;
> + /**
> + * Number of chained descriptors received that exceed the max allowed
> + * chain by device
> + */
> + uint32_t exceed_max_chain;
> + /**
> + * Number of times device tried to read or write buffer that is not
> + * registered to the device
> + */
> + uint32_t invalid_buffer;
> + /** Number of errors detected by the device */
> + uint32_t errors;
> +};
> +
I think doing it like that, we risk to keep the rte_vdpa_get_stats API
always experimental, as every vendor will want to add their own counters
and so break the ABI.
How about implementing something similar to rte_eth_xstat?
As these stats are for debugging purpose, it would give you much more
flexibility in adding new counters as HW or firmwares evolves.
What do you think?
Thanks,
Maxime
> /**
> * vdpa device operations
> */
> @@ -73,8 +94,11 @@ struct rte_vdpa_dev_ops {
> int (*get_notify_area)(int vid, int qid,
> uint64_t *offset, uint64_t *size);
>
> + /** Get statistics of the queue */
> + int (*get_stats)(int did, int qid, struct rte_vdpa_queue_stats *stats);
> +
> /** Reserved for future extension */
> - void *reserved[5];
> + void *reserved[4];
> };
>
> /**
> @@ -200,4 +224,23 @@ struct rte_vdpa_device *
> __rte_experimental
> int
> rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m);
> +
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice
> + *
> + * Get vDPA device queue statistics.
> + *
> + * @param did
> + * device id
> + * @param qid
> + * queue id
> + * @param stats
> + * queue statistics pointer.
> + * @return
> + * 0 on success, non-zero on failure.
> + */
> +__rte_experimental
> +int
> +rte_vdpa_get_stats(int did, uint16_t qid, struct rte_vdpa_queue_stats *stats);
> #endif /* _RTE_VDPA_H_ */
> diff --git a/lib/librte_vhost/rte_vhost_version.map b/lib/librte_vhost/rte_vhost_version.map
> index 051d08c..c9dcff4 100644
> --- a/lib/librte_vhost/rte_vhost_version.map
> +++ b/lib/librte_vhost/rte_vhost_version.map
> @@ -38,6 +38,7 @@ EXPERIMENTAL {
> rte_vdpa_find_device_id;
> rte_vdpa_get_device;
> rte_vdpa_get_device_num;
> + rte_vdpa_get_stats;
> rte_vhost_driver_attach_vdpa_device;
> rte_vhost_driver_detach_vdpa_device;
> rte_vhost_driver_get_vdpa_device_id;
> diff --git a/lib/librte_vhost/vdpa.c b/lib/librte_vhost/vdpa.c
> index 2b86708..57900fc 100644
> --- a/lib/librte_vhost/vdpa.c
> +++ b/lib/librte_vhost/vdpa.c
> @@ -227,3 +227,17 @@ struct rte_vdpa_device *
> free_ind_table(idesc);
> return -1;
> }
> +
> +int
> +rte_vdpa_get_stats(int did, uint16_t qid, struct rte_vdpa_queue_stats *stats)
> +{
> + struct rte_vdpa_device *vdpa_dev;
> +
> + vdpa_dev = rte_vdpa_get_device(did);
> + if (!vdpa_dev)
> + return -ENODEV;
> +
> + RTE_FUNC_PTR_OR_ERR_RET(vdpa_dev->ops->get_stats, -ENOTSUP);
> +
> + return vdpa_dev->ops->get_stats(did, qid, stats);
> +}
>
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v4 6/8] eal/x86: identify AVX512 extensions flag
2020-04-15 13:31 3% ` David Marchand
@ 2020-04-15 14:57 0% ` Liu, Yong
0 siblings, 0 replies; 200+ results
From: Liu, Yong @ 2020-04-15 14:57 UTC (permalink / raw)
To: David Marchand
Cc: Maxime Coquelin, Ye, Xiaolong, Wang, Zhihong, Van Haaren, Harry,
dev, Laatz, Kevin, Kinsella, Ray
Thanks for note, David. Kevin's patch can fully cover this one.
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Wednesday, April 15, 2020 9:32 PM
> To: Liu, Yong <yong.liu@intel.com>
> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>; Ye, Xiaolong
> <xiaolong.ye@intel.com>; Wang, Zhihong <zhihong.wang@intel.com>; Van
> Haaren, Harry <harry.van.haaren@intel.com>; dev <dev@dpdk.org>; Laatz,
> Kevin <kevin.laatz@intel.com>; Kinsella, Ray <ray.kinsella@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v4 6/8] eal/x86: identify AVX512 extensions
> flag
>
> On Wed, Apr 15, 2020 at 11:14 AM Marvin Liu <yong.liu@intel.com> wrote:
> >
> > Read CPUID to check if AVX512 extensions are supported.
> >
> > Signed-off-by: Marvin Liu <yong.liu@intel.com>
> >
> > diff --git a/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> b/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> > index 6492df556..54e9f6185 100644
> > --- a/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> > +++ b/lib/librte_eal/common/arch/x86/rte_cpuflags.c
> > @@ -109,6 +109,9 @@ const struct feature_entry rte_cpu_feature_table[]
> = {
> > FEAT_DEF(RTM, 0x00000007, 0, RTE_REG_EBX, 11)
> > FEAT_DEF(AVX512F, 0x00000007, 0, RTE_REG_EBX, 16)
> > FEAT_DEF(RDSEED, 0x00000007, 0, RTE_REG_EBX, 18)
> > + FEAT_DEF(AVX512CD, 0x00000007, 0, RTE_REG_EBX, 28)
> > + FEAT_DEF(AVX512BW, 0x00000007, 0, RTE_REG_EBX, 30)
> > + FEAT_DEF(AVX512VL, 0x00000007, 0, RTE_REG_EBX, 31)
> >
> > FEAT_DEF(LAHF_SAHF, 0x80000001, 0, RTE_REG_ECX, 0)
> > FEAT_DEF(LZCNT, 0x80000001, 0, RTE_REG_ECX, 4)
> > diff --git a/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> b/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> > index 25ba47b96..5bf99e05f 100644
> > --- a/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> > +++ b/lib/librte_eal/common/include/arch/x86/rte_cpuflags.h
> > @@ -98,6 +98,9 @@ enum rte_cpu_flag_t {
> > RTE_CPUFLAG_RTM, /**< Transactional memory */
> > RTE_CPUFLAG_AVX512F, /**< AVX512F */
> > RTE_CPUFLAG_RDSEED, /**< RDSEED instruction */
> > + RTE_CPUFLAG_AVX512CD, /**< AVX512CD */
> > + RTE_CPUFLAG_AVX512BW, /**< AVX512BW */
> > + RTE_CPUFLAG_AVX512VL, /**< AVX512VL */
> >
> > /* (EAX 80000001h) ECX features */
> > RTE_CPUFLAG_LAHF_SAHF, /**< LAHF_SAHF */
>
> This patch most likely breaks the ABI (renumbering flags after
> RTE_CPUFLAG_LAHF_SAHF).
> This change should not go through the virtio tree and is not rebased on
> master.
> A similar patch had been proposed by Kevin:
> http://patchwork.dpdk.org/patch/67438/
>
>
> --
> David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
2020-04-14 18:27 5% ` Trahe, Fiona
@ 2020-04-15 17:24 5% ` Trahe, Fiona
2020-04-16 9:51 3% ` Bruce Richardson
0 siblings, 1 reply; 200+ results
From: Trahe, Fiona @ 2020-04-15 17:24 UTC (permalink / raw)
To: Ray Kinsella, dev, Richardson, Bruce, Thomas Monjalon
Cc: Kusztal, ArkadiuszX, Trahe, Fiona
Hi Ray/Bruce/Thomas,
We've been trying to make sense of the versioning scheme - but as the docs describe and give examples
of a 2-number scheme and the code has a 3-number scheme it's pretty confusing.
If you can help me fill out the following table this may help:
VERSION | ABI_VERSION | LIBABIVER| stable soname | lib filename | new section with ABI break in map file
19.11.0 | 20.0 | 1 | stable.so.20.0 | stable.so.20.0.0 |
20.02.0-rc0| 20.1 | 1 | n/a
20.02.0 | 20.0.1 | 0.1 | stable.so.20.0 | stable.so.20.0.1 |
20.05.0 | 20.0.2 | 0.1 | stable.so.20.0 | stable.so.20.0.2 | 20.0.2 (or 21? use impending stable ABI_VERSION?)
20.08.0 | 20.0.3 | 0.1 | stable.so.20.0 | stable.so.20.0.3 |
20.11.0 | 21 | 0.1 | stable.so.21 | stable.so.21.0 | (Can we move back to a 2 number ABI_VERSION at this stage? )
21.02.0 | 21.1 | 0.1 | stable.so.21 | stable.so.21.1 |
Note, first 2 ABI_VERSIONS above were mistakes, soname needs to be same as 19.11 to avoid ABI breakage, so it was changed to 3-part number, with first 2 used in soname.
Questions:
1. For this year, is major a 2-part num 20.0 (and there will never be a 20.1,20.2, etc) and minor is the 3rd number?
2. Can we move back to a 2-number scheme in 20.11?
3. What uses lib filename? (This is described in config/meson.build) Guessing stable soname is name on which apps depend, and is a sym link, pointing to lib filename?
4. What does LIBABIVER have to do with this ? it was 1, changed to 0.1 in 20.2
5a. If in 20.05 we add a version of a fn which breaks ABI 20.0, what should the name of the original function be? fn_v20, or fn_v20.0
Or does it even matter, i.e. is the middle param to this macro arbitrary and just needs to match the fn name, e.g. if fn name was fn_vxyz could macro be:
VERSION_SYMBOL(fn, _vxyz, 20.0);
5b. what should the name of the new fn be? fn_v2002 or fn_v21? Does it need to correspond to the new section in .map? If v2002, doesn't that imply it's compatible with 20.0 ABI
So shouldn't the new section be DPDK_21 and the new fn be fn_v21.
6. Assume we go with DPDK_21 and fn_v21. An app built against 20.05 has a dependency on fn_v21 (because of the BIND_DEFAULT) but on otherfn_v20.0 of all other fns?
If fn_v21 is changed again in 20.08, is there any expectation that the app built against 20.05 must work against the 20.08 fn of the same name?
i.e. is compatibility required on fn changes across minor nonstable releases? If not then jumping ahead to v21 seems the simplest option.
And the "final" fn_v21 in 20.11 is the one that "sticks" and belongs to the new ABI which then must remain stable til 21.11
> -----Original Message-----
> From: Trahe, Fiona <fiona.trahe@intel.com>
> Sent: Tuesday, April 14, 2020 7:27 PM
> To: Ray Kinsella <mdr@ashroe.eu>; dev@dpdk.org
> Cc: Trahe, Fiona <fiona.trahe@intel.com>; Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>
> Subject: RE: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
>
> Hi Ray,
>
> We're going to need hep to understand the numbering.
> The examples here http://doc.dpdk.org/guides/contributing/abi_versioning.html#major-abi-versions
> show only major numbers in the .map file.
> Whereas the map files all have major.minor.
>
> The example shows that to add a new version of an existing fn one should use the next major number,
> "When an ABI change is made between major ABI versions to a given library, a new section is added to
> that library’s version map describing the impending new ABI version"
> so
> - Old fn becomes fn_v20()
> - VERSION_SYMBOL(fn, _v20, 20); - maps node 20 to fn_v20() for dynamic linking
> - New fn becomes fn_v21()
> - BIND_DEFAULT_SYMBOL(fn, _v21, 21); - maps new builds to fn_v21 for dynamic linking and makes
> fn_v21 default
> - MAP_STATIC_SYMBOL(fn_proto, fn_v21); - for static linking
> And the map file should have:
>
> DPDK_21 {
> global:
> rte_cryptodev_info_get;
> } DPDK_20;
>
> When the ABI version moves to node 21 in DPDK20.11, the _v20 symbols can be removed.
>
> You suggest using DPDK_20.0.2 and _v2002
> Can you explain why?
> In 20.0.2 - which number is minor?
> And why 3 numbers?
>
> Regards,
> Fiona
>
>
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Ray Kinsella
> > Sent: Tuesday, April 14, 2020 2:54 PM
> > To: dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
> >
> >
> >
> > On 18/03/2020 20:41, Arek Kusztal wrote:
> > > This patch adds versioned function rte_cryptodev_info_get.
> > > Node 20.05 function works the same way it was working before.
> > > Node 20.0 function strips capability added in 20.05 release
> > > to prevent some issues with ABI policy. To do that new capability
> > > array is allocated per device and returned to user instead of the
> > > original array passed by PMD.
> > >
> > > Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
> > > ---
> > > lib/librte_cryptodev/rte_cryptodev.c | 97 +++++++++++++++++++++++++-
> > > lib/librte_cryptodev/rte_cryptodev.h | 12 +++-
> > > lib/librte_cryptodev/rte_cryptodev_version.map | 7 ++
> > > 3 files changed, 113 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/lib/librte_cryptodev/rte_cryptodev.c b/lib/librte_cryptodev/rte_cryptodev.c
> > > index 6d1d0e9..2d030ba 100644
> > > --- a/lib/librte_cryptodev/rte_cryptodev.c
> > > +++ b/lib/librte_cryptodev/rte_cryptodev.c
> > > @@ -41,6 +41,9 @@
> > > #include "rte_cryptodev.h"
> > > #include "rte_cryptodev_pmd.h"
> > >
> > > +#include <rte_compat.h>
> > > +#include <rte_function_versioning.h>
> > > +
> > > static uint8_t nb_drivers;
> > >
> > > static struct rte_cryptodev rte_crypto_devices[RTE_CRYPTO_MAX_DEVS];
> > > @@ -56,6 +59,13 @@ static struct rte_cryptodev_global cryptodev_globals = {
> > > /* spinlock for crypto device callbacks */
> > > static rte_spinlock_t rte_cryptodev_cb_lock = RTE_SPINLOCK_INITIALIZER;
> > >
> > > +static const struct rte_cryptodev_capabilities
> > > + cryptodev_undefined_capabilities[] = {
> > > + RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST()
> > > +};
> > > +
> > > +struct rte_cryptodev_capabilities *capability_copies[RTE_CRYPTO_MAX_DEVS];
> > > +uint8_t is_capability_checked[RTE_CRYPTO_MAX_DEVS];
> > >
> > > /**
> > > * The user application callback description.
> > > @@ -999,6 +1009,13 @@ rte_cryptodev_close(uint8_t dev_id)
> > > RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_close, -ENOTSUP);
> > > retval = (*dev->dev_ops->dev_close)(dev);
> > >
> > > +
> > > + if (capability_copies[dev_id]) {
> > > + free(capability_copies[dev_id]);
> > > + capability_copies[dev_id] = NULL;
> > > + }
> > > + is_capability_checked[dev_id] = 0;
> > > +
> > > if (retval < 0)
> > > return retval;
> > >
> > > @@ -1111,11 +1128,12 @@ rte_cryptodev_stats_reset(uint8_t dev_id)
> > > (*dev->dev_ops->stats_reset)(dev);
> > > }
> > >
> > > -
> > > void
> > > -rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
> > > +rte_cryptodev_info_get_v20(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
> > > {
> > > struct rte_cryptodev *dev;
> > > + const struct rte_cryptodev_capabilities *capability;
> > > + uint8_t counter = 0;
> > >
> > > if (!rte_cryptodev_pmd_is_valid_dev(dev_id)) {
> > > CDEV_LOG_ERR("Invalid dev_id=%d", dev_id);
> > > @@ -1129,10 +1147,85 @@ rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info
> > *dev_info)
> > > RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
> > > (*dev->dev_ops->dev_infos_get)(dev, dev_info);
> > >
> > > + if (capability_copies[dev_id] == NULL) {
> > > + if (!is_capability_checked[dev_id]) {
> > > + uint8_t found_invalid_capa = 0;
> > > +
> > > + for (capability = dev_info->capabilities;
> > > + capability->op != RTE_CRYPTO_OP_TYPE_UNDEFINED;
> > > + ++capability, ++counter) {
> > > + if (capability->op == RTE_CRYPTO_OP_TYPE_SYMMETRIC &&
> > > + capability->sym.xform_type ==
> > > + RTE_CRYPTO_SYM_XFORM_AEAD
> > > + && capability->sym.aead.algo >=
> > > + RTE_CRYPTO_AEAD_CHACHA20_POLY1305) {
> > > + found_invalid_capa = 1;
> > > + counter--;
> > > + }
> > > + }
> > > + is_capability_checked[dev_id] = 1;
> > > + if (found_invalid_capa) {
> > > + capability_copies[dev_id] = malloc(counter *
> > > + sizeof(struct rte_cryptodev_capabilities));
> > > + if (capability_copies[dev_id] == NULL) {
> > > + /*
> > > + * error case - no memory to store the trimmed list,
> > > + * so have to return an empty list
> > > + */
> > > + dev_info->capabilities =
> > > + cryptodev_undefined_capabilities;
> > > + is_capability_checked[dev_id] = 0;
> > > + } else {
> > > + counter = 0;
> > > + for (capability = dev_info->capabilities;
> > > + capability->op !=
> > > + RTE_CRYPTO_OP_TYPE_UNDEFINED;
> > > + capability++) {
> > > + if (!(capability->op ==
> > > + RTE_CRYPTO_OP_TYPE_SYMMETRIC
> > > + && capability->sym.xform_type ==
> > > + RTE_CRYPTO_SYM_XFORM_AEAD
> > > + && capability->sym.aead.algo >=
> > > +
> > RTE_CRYPTO_AEAD_CHACHA20_POLY1305)) {
> > > + capability_copies[dev_id][counter++] =
> > > + *capability;
> > > + }
> > > + }
> > > + dev_info->capabilities =
> > > + capability_copies[dev_id];
> > > + }
> > > + }
> > > + }
> > > + } else
> > > + dev_info->capabilities = capability_copies[dev_id];
> > > +
> > > dev_info->driver_name = dev->device->driver->name;
> > > dev_info->device = dev->device;
> > > }
> > > +VERSION_SYMBOL(rte_cryptodev_info_get, _v20, 20.0);
> > > +
> > > +void
> > > +rte_cryptodev_info_get_v2005(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
> >
> > should be rte_cryptodev_info_get_v2002
> >
> > > +{
> > > + struct rte_cryptodev *dev;
> > >
> > > + if (!rte_cryptodev_pmd_is_valid_dev(dev_id)) {
> > > + CDEV_LOG_ERR("Invalid dev_id=%d", dev_id);
> > > + return;
> > > + }
> > > +
> > > + dev = &rte_crypto_devices[dev_id];
> > > +
> > > + memset(dev_info, 0, sizeof(struct rte_cryptodev_info));
> > > +
> > > + RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
> > > + (*dev->dev_ops->dev_infos_get)(dev, dev_info);
> > > +
> > > + dev_info->driver_name = dev->device->driver->name;
> > > + dev_info->device = dev->device;
> > > +}
> > > +MAP_STATIC_SYMBOL(void rte_cryptodev_info_get(uint8_t dev_id,
> > > + struct rte_cryptodev_info *dev_info), rte_cryptodev_info_get_v2005);
> >
> > should be rte_cryptodev_info_get_v2002
> >
> > >
> > > int
> > > rte_cryptodev_callback_register(uint8_t dev_id,
> > > diff --git a/lib/librte_cryptodev/rte_cryptodev.h b/lib/librte_cryptodev/rte_cryptodev.h
> > > index 437b8a9..06ce2f2 100644
> > > --- a/lib/librte_cryptodev/rte_cryptodev.h
> > > +++ b/lib/librte_cryptodev/rte_cryptodev.h
> > > @@ -24,6 +24,9 @@ extern "C" {
> > > #include <rte_common.h>
> > > #include <rte_config.h>
> > >
> > > +#include <rte_compat.h>
> > > +#include <rte_function_versioning.h>
> > > +
> > > extern const char **rte_cyptodev_names;
> > >
> > > /* Logging Macros */
> > > @@ -758,7 +761,14 @@ rte_cryptodev_stats_reset(uint8_t dev_id);
> > > * the last valid element has it's op field set to
> > > * RTE_CRYPTO_OP_TYPE_UNDEFINED.
> > > */
> > > -extern void
> > > +void
> > > +rte_cryptodev_info_get_v20(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
> > > +
> > > +void
> > > +rte_cryptodev_info_get_v2005(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
> > > +BIND_DEFAULT_SYMBOL(rte_cryptodev_info_get, _v2005, 20.05);
> > > +
> > > +void
> > > rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
> > do we still need this forward declaration?
> >
> > >
> > >
> > > diff --git a/lib/librte_cryptodev/rte_cryptodev_version.map
> > b/lib/librte_cryptodev/rte_cryptodev_version.map
> > > index 6e41b4b..d6c945a 100644
> > > --- a/lib/librte_cryptodev/rte_cryptodev_version.map
> > > +++ b/lib/librte_cryptodev/rte_cryptodev_version.map
> > > @@ -58,6 +58,13 @@ DPDK_20.0 {
> > > local: *;
> > > };
> > >
> > > +
> > > +DPDK_20.05 {
> >
> > should be DPDK_20.0.2
> >
> > > + global:
> > > + rte_cryptodev_info_get;
> > > +} DPDK_20.0;
> > > +
> > > +
> > > EXPERIMENTAL {
> > > global:
> > >
> > >
^ permalink raw reply [relevance 5%]
* Re: [dpdk-dev] [dpdk-dev v5 3/3] app/testpmd: add new types to RSS hash commands
@ 2020-04-16 7:51 3% ` Jeff Guo
2020-04-16 9:14 0% ` Iremonger, Bernard
0 siblings, 1 reply; 200+ results
From: Jeff Guo @ 2020-04-16 7:51 UTC (permalink / raw)
To: Iremonger, Bernard, orika, Ye, Xiaolong, Zhang, Qi Z
Cc: dev, Wu, Jingjing, Cao, Yahui, Su, Simei
hi, bernard
On 4/15/2020 11:01 PM, Iremonger, Bernard wrote:
> Hi Jeff,
>
>> -----Original Message-----
>> From: Guo, Jia <jia.guo@intel.com>
>> Sent: Wednesday, April 15, 2020 6:11 PM
>> To: Iremonger, Bernard <bernard.iremonger@intel.com>;
>> orika@mellanox.com; Ye, Xiaolong <xiaolong.ye@intel.com>; Zhang, Qi Z
>> <qi.z.zhang@intel.com>
>> Cc: dev@dpdk.org; Wu, Jingjing <jingjing.wu@intel.com>; Cao, Yahui
>> <yahui.cao@intel.com>; Su, Simei <simei.su@intel.com>; Guo, Jia
>> <jia.guo@intel.com>
>> Subject: [dpdk-dev v5 3/3] app/testpmd: add new types to RSS hash
>> commands
>>
>> Add some new types, such as eth/l2-src-only/l2-dst-only/svlan/cvlan/
>> l2tpv3/esp/ah/pfcp types into RSS hash commands, it could be used to
>> configure these rss input set by cmdline.
>>
>> Example testpmd commands are:
>> Eth:
>> testpmd>flow create 0 ingress pattern eth / ipv4 / end actions rss \
>> types eth l2-src-only end key_len 0 queues end / end
>>
>> testpmd>flow create 0 ingress pattern eth / ipv4 / end actions rss \
>> types eth l2-dst-only end key_len 0 queues end / end
>>
>> testpmd>flow create 0 ingress pattern eth / ipv4 / end actions rss \
>> types eth end key_len 0 queues end / end
>>
>> s-vlan:
>> testpmd>flow create 0 ingress pattern eth / ipv4 / end actions rss \
>> types c-vlan end key_len 0 queues end / end
>>
>> c-vlan:
>> testpmd>flow create 0 ingress pattern eth / ipv4 / end actions rss \
>> types c-vlan end key_len 0 queues end / end
>>
>> l2tpv3:
>> testpmd>flow create 0 ingress pattern eth / ipv4 / l2tpv3oip / end \
>> actions rss types ipv4 l2tpv3 end key_len 0 queues end / end
>>
>> esp:
>> testpmd>flow create 0 ingress pattern eth / ipv4 / esp / end actions \
>> rss types ipv4 esp end key_len 0 queues end / end
> Should "rss types ipv4 esp" be "rss types esp" ?
yes, it should be.
>> ah:
>> testpmd>flow create 0 ingress pattern eth / ipv4 / ah / end actions \
>> rss types ipv4 ah end key_len 0 queues end / end
>>
> Should "rss types ah" be "rss types ah" ?
yes, the same as above and i think you are right, all action should be
specific, i think the parse logic should be modify to use pattern to
distinguish the ipv4 ah or ipv6 ah but not use rss type.
>> pfcp:
>> testpmd>flow create 0 ingress pattern eth / ipv4 / udp / pfcp / end \
>> actions rss types ipv4-udp pfcp end key_len 0 queues end / end
> The above sample commands might be better in a doc file, rather than in the commit message.
> For example doc/guides/howto/rte_flow.rst
>
> The flow create/validate commands are normally parsed in app/testpmd/cmdline_flow.c
> This patch is updating the following commands:
> "port config all rss all"
> "port config <port_id> rss-hash-key"
Since this is not the command initial and just leverage the command
which is already exist, so i think i would suppose that no need to
modify the doc if no change, only change
the part if the change bring in, and i could briefly introduce a example
only in commit log for simplify.
>> Signed-off-by: Jeff Guo <jia.guo@intel.com>
>> ---
>> v5->v4:
>> rename eth to l2 and refine commit log
>> ---
>> app/test-pmd/cmdline.c | 28 +++++++++++++++++++++++++--- app/test-
>> pmd/config.c | 9 +++++++++
>> 2 files changed, 34 insertions(+), 3 deletions(-)
>
>> diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c index
>> 863b567c1..6b688ab66 100644
>> --- a/app/test-pmd/cmdline.c
>> +++ b/app/test-pmd/cmdline.c
>> @@ -2270,9 +2270,11 @@ cmd_config_rss_parsed(void *parsed_result,
>> int ret;
>>
>> if (!strcmp(res->value, "all"))
>> -rss_conf.rss_hf = ETH_RSS_IP | ETH_RSS_TCP |
>> +rss_conf.rss_hf = ETH_RSS_ETH | ETH_RSS_IP |
>> ETH_RSS_TCP |
>> ETH_RSS_UDP | ETH_RSS_SCTP |
>> ETH_RSS_L2_PAYLOAD;
>> +else if (!strcmp(res->value, "eth"))
>> +rss_conf.rss_hf = ETH_RSS_ETH;
>> else if (!strcmp(res->value, "ip"))
>> rss_conf.rss_hf = ETH_RSS_IP;
>> else if (!strcmp(res->value, "udp"))
>> @@ -2299,6 +2301,22 @@ cmd_config_rss_parsed(void *parsed_result,
>> rss_conf.rss_hf = ETH_RSS_L4_SRC_ONLY;
>> else if (!strcmp(res->value, "l4-dst-only"))
>> rss_conf.rss_hf = ETH_RSS_L4_DST_ONLY;
>> +else if (!strcmp(res->value, "l2-src-only"))
>> +rss_conf.rss_hf = ETH_RSS_L2_SRC_ONLY;
>> +else if (!strcmp(res->value, "l2-dst-only"))
>> +rss_conf.rss_hf = ETH_RSS_L2_DST_ONLY;
>> +else if (!strcmp(res->value, "s-vlan"))
>> +rss_conf.rss_hf = ETH_RSS_S_VLAN;
>> +else if (!strcmp(res->value, "c-vlan"))
>> +rss_conf.rss_hf = ETH_RSS_C_VLAN;
>> +else if (!strcmp(res->value, "l2tpv3"))
>> +rss_conf.rss_hf = ETH_RSS_L2TPV3;
>> +else if (!strcmp(res->value, "esp"))
>> +rss_conf.rss_hf = ETH_RSS_ESP;
>> +else if (!strcmp(res->value, "ah"))
>> +rss_conf.rss_hf = ETH_RSS_AH;
>> +else if (!strcmp(res->value, "pfcp"))
>> +rss_conf.rss_hf = ETH_RSS_PFCP;
>> else if (!strcmp(res->value, "none"))
>> rss_conf.rss_hf = 0;
>> else if (!strcmp(res->value, "default")) @@ -2467,7 +2485,9 @@
>> cmdline_parse_token_string_t cmd_config_rss_hash_key_rss_type =
>> "ipv4-other#ipv6#ipv6-frag#ipv6-tcp#ipv6-
>> udp#"
>> "ipv6-sctp#ipv6-other#l2-payload#ipv6-ex#"
>> "ipv6-tcp-ex#ipv6-udp-ex#"
>> - "l3-src-only#l3-dst-only#l4-src-only#l4-dst-
>> only");
>> + "l3-src-only#l3-dst-only#l4-src-only#l4-dst-
>> only#"
>> + "l2-src-only#l2-dst-only#s-vlan#c-vlan#"
>> + "l2tpv3#esp#ah#pfcp");
>> cmdline_parse_token_string_t cmd_config_rss_hash_key_value =
>> TOKEN_STRING_INITIALIZER(struct cmd_config_rss_hash_key, key,
>> NULL);
>>
>> @@ -2478,7 +2498,9 @@ cmdline_parse_inst_t cmd_config_rss_hash_key =
>> {
>> "ipv4|ipv4-frag|ipv4-tcp|ipv4-udp|ipv4-sctp|ipv4-other|"
>> "ipv6|ipv6-frag|ipv6-tcp|ipv6-udp|ipv6-sctp|ipv6-other|"
>> "l2-payload|ipv6-ex|ipv6-tcp-ex|ipv6-udp-ex|"
>> -"l3-src-only|l3-dst-only|l4-src-only|l4-dst-only "
>> +"l3-src-only|l3-dst-only|l4-src-only|l4-dst-only|"
>> +"l2-src-only|l2-dst-only|s-vlan|c-vlan|"
>> +"l2tpv3|esp|ah|pfcp "
>> "<string of hex digits (variable length, NIC dependent)>",
>> .tokens = {
>> (void *)&cmd_config_rss_hash_key_port, diff --git
>> a/app/test-pmd/config.c b/app/test-pmd/config.c index
>> 71aeb5413..e4a97388b 100644
>> --- a/app/test-pmd/config.c
>> +++ b/app/test-pmd/config.c
>> @@ -79,6 +79,11 @@ const struct rss_type_info rss_type_table[] = {
>> ETH_RSS_UDP | ETH_RSS_SCTP |
>> ETH_RSS_L2_PAYLOAD },
>> { "none", 0 },
>> +{ "eth", ETH_RSS_ETH },
>> +{ "l2-src-only", ETH_RSS_L2_SRC_ONLY },
>> +{ "l2-dst-only", ETH_RSS_L2_DST_ONLY },
>> +{ "s-vlan", ETH_RSS_S_VLAN },
>> +{ "c-vlan", ETH_RSS_C_VLAN },
> Should these additions be at the bottom of the array with the other additions ?
I suppose this is not in a library code or application binary interface,
so it would not break ABI/API, and re-order it as up-down layer must be
better.
>> { "ipv4", ETH_RSS_IPV4 },
>> { "ipv4-frag", ETH_RSS_FRAG_IPV4 },
>> { "ipv4-tcp", ETH_RSS_NONFRAG_IPV4_TCP }, @@ -108,6 +113,10
>> @@ const struct rss_type_info rss_type_table[] = {
>> { "l3-dst-only", ETH_RSS_L3_DST_ONLY },
>> { "l4-src-only", ETH_RSS_L4_SRC_ONLY },
>> { "l4-dst-only", ETH_RSS_L4_DST_ONLY },
>> +{ "l2tpv3", ETH_RSS_L2TPV3 },
>> +{ "esp", ETH_RSS_ESP },
>> +{ "ah", ETH_RSS_AH },
>> +{ "pfcp", ETH_RSS_PFCP },
>> { NULL, 0 },
>> };
>>
>> --
>> 2.20.1
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v2] cryptodev: version rte_cryptodev_info_get function
@ 2020-04-16 8:12 9% Arek Kusztal
2020-04-16 8:18 3% ` Akhil Goyal
0 siblings, 1 reply; 200+ results
From: Arek Kusztal @ 2020-04-16 8:12 UTC (permalink / raw)
To: dev; +Cc: akhil.goyal, fiona.trahe, Arek Kusztal
This patch adds versioned function rte_cryptodev_info_get.
Node 20.0.2 function works the same way it was working before.
Node 20.0 function strips capability added in 20.05 release
to prevent some issues with ABI policy. To do that new capability
array is allocated per device and returned to user instead of the
original array passed by PMD.
Because rte_cryptodev_info_get is called by other API functions,
rte_cryptodev_sym_capability_get function was versioned the same way.
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
---
This patch depends on following patches:
[1] - "[v3] cryptodev: add chacha20-poly1305 aead algorithm"
(http://patchwork.dpdk.org/patch/64549/)
lib/librte_cryptodev/rte_cryptodev.c | 139 ++++++++++++++++++++++++-
lib/librte_cryptodev/rte_cryptodev.h | 38 ++++++-
lib/librte_cryptodev/rte_cryptodev_version.map | 7 ++
3 files changed, 180 insertions(+), 4 deletions(-)
diff --git a/lib/librte_cryptodev/rte_cryptodev.c b/lib/librte_cryptodev/rte_cryptodev.c
index 6d1d0e9..158e7d6 100644
--- a/lib/librte_cryptodev/rte_cryptodev.c
+++ b/lib/librte_cryptodev/rte_cryptodev.c
@@ -41,6 +41,9 @@
#include "rte_cryptodev.h"
#include "rte_cryptodev_pmd.h"
+#include <rte_compat.h>
+#include <rte_function_versioning.h>
+
static uint8_t nb_drivers;
static struct rte_cryptodev rte_crypto_devices[RTE_CRYPTO_MAX_DEVS];
@@ -56,6 +59,13 @@ static struct rte_cryptodev_global cryptodev_globals = {
/* spinlock for crypto device callbacks */
static rte_spinlock_t rte_cryptodev_cb_lock = RTE_SPINLOCK_INITIALIZER;
+static const struct rte_cryptodev_capabilities
+ cryptodev_undefined_capabilities[] = {
+ RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST()
+};
+
+static struct rte_cryptodev_capabilities *capability_copies[RTE_CRYPTO_MAX_DEVS];
+static uint8_t is_capability_checked[RTE_CRYPTO_MAX_DEVS];
/**
* The user application callback description.
@@ -280,7 +290,43 @@ rte_crypto_auth_operation_strings[] = {
};
const struct rte_cryptodev_symmetric_capability *
-rte_cryptodev_sym_capability_get(uint8_t dev_id,
+rte_cryptodev_sym_capability_get_v20(uint8_t dev_id,
+ const struct rte_cryptodev_sym_capability_idx *idx)
+{
+ const struct rte_cryptodev_capabilities *capability;
+ struct rte_cryptodev_info dev_info;
+ int i = 0;
+
+ rte_cryptodev_info_get_v20(dev_id, &dev_info);
+
+ while ((capability = &dev_info.capabilities[i++])->op !=
+ RTE_CRYPTO_OP_TYPE_UNDEFINED) {
+ if (capability->op != RTE_CRYPTO_OP_TYPE_SYMMETRIC)
+ continue;
+
+ if (capability->sym.xform_type != idx->type)
+ continue;
+
+ if (idx->type == RTE_CRYPTO_SYM_XFORM_AUTH &&
+ capability->sym.auth.algo == idx->algo.auth)
+ return &capability->sym;
+
+ if (idx->type == RTE_CRYPTO_SYM_XFORM_CIPHER &&
+ capability->sym.cipher.algo == idx->algo.cipher)
+ return &capability->sym;
+
+ if (idx->type == RTE_CRYPTO_SYM_XFORM_AEAD &&
+ capability->sym.aead.algo == idx->algo.aead)
+ return &capability->sym;
+ }
+
+ return NULL;
+
+}
+VERSION_SYMBOL(rte_cryptodev_sym_capability_get, _v20, 20.0);
+
+const struct rte_cryptodev_symmetric_capability *
+rte_cryptodev_sym_capability_get_v2002(uint8_t dev_id,
const struct rte_cryptodev_sym_capability_idx *idx)
{
const struct rte_cryptodev_capabilities *capability;
@@ -313,6 +359,10 @@ rte_cryptodev_sym_capability_get(uint8_t dev_id,
return NULL;
}
+MAP_STATIC_SYMBOL(const struct rte_cryptodev_symmetric_capability *
+ rte_cryptodev_sym_capability_get(uint8_t dev_id,
+ const struct rte_cryptodev_sym_capability_idx *idx),
+ rte_cryptodev_sym_capability_get_v2002);
static int
param_range_check(uint16_t size, const struct rte_crypto_param_range *range)
@@ -999,6 +1049,13 @@ rte_cryptodev_close(uint8_t dev_id)
RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_close, -ENOTSUP);
retval = (*dev->dev_ops->dev_close)(dev);
+
+ if (capability_copies[dev_id]) {
+ free(capability_copies[dev_id]);
+ capability_copies[dev_id] = NULL;
+ }
+ is_capability_checked[dev_id] = 0;
+
if (retval < 0)
return retval;
@@ -1111,11 +1168,12 @@ rte_cryptodev_stats_reset(uint8_t dev_id)
(*dev->dev_ops->stats_reset)(dev);
}
-
void
-rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
+rte_cryptodev_info_get_v20(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
{
struct rte_cryptodev *dev;
+ const struct rte_cryptodev_capabilities *capability;
+ uint8_t counter = 0;
if (!rte_cryptodev_pmd_is_valid_dev(dev_id)) {
CDEV_LOG_ERR("Invalid dev_id=%d", dev_id);
@@ -1129,10 +1187,85 @@ rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
(*dev->dev_ops->dev_infos_get)(dev, dev_info);
+ if (capability_copies[dev_id] == NULL) {
+ if (!is_capability_checked[dev_id]) {
+ uint8_t found_invalid_capa = 0;
+
+ for (capability = dev_info->capabilities;
+ capability->op != RTE_CRYPTO_OP_TYPE_UNDEFINED;
+ ++capability, ++counter) {
+ if (capability->op == RTE_CRYPTO_OP_TYPE_SYMMETRIC &&
+ capability->sym.xform_type ==
+ RTE_CRYPTO_SYM_XFORM_AEAD
+ && capability->sym.aead.algo >=
+ RTE_CRYPTO_AEAD_CHACHA20_POLY1305) {
+ found_invalid_capa = 1;
+ counter--;
+ }
+ }
+ is_capability_checked[dev_id] = 1;
+ if (found_invalid_capa) {
+ capability_copies[dev_id] = malloc(counter *
+ sizeof(struct rte_cryptodev_capabilities));
+ if (capability_copies[dev_id] == NULL) {
+ /*
+ * error case - no memory to store the trimmed list,
+ * so have to return an empty list
+ */
+ dev_info->capabilities =
+ cryptodev_undefined_capabilities;
+ is_capability_checked[dev_id] = 0;
+ } else {
+ counter = 0;
+ for (capability = dev_info->capabilities;
+ capability->op !=
+ RTE_CRYPTO_OP_TYPE_UNDEFINED;
+ capability++) {
+ if (!(capability->op ==
+ RTE_CRYPTO_OP_TYPE_SYMMETRIC
+ && capability->sym.xform_type ==
+ RTE_CRYPTO_SYM_XFORM_AEAD
+ && capability->sym.aead.algo >=
+ RTE_CRYPTO_AEAD_CHACHA20_POLY1305)) {
+ capability_copies[dev_id][counter++] =
+ *capability;
+ }
+ }
+ dev_info->capabilities =
+ capability_copies[dev_id];
+ }
+ }
+ }
+ } else
+ dev_info->capabilities = capability_copies[dev_id];
+
dev_info->driver_name = dev->device->driver->name;
dev_info->device = dev->device;
}
+VERSION_SYMBOL(rte_cryptodev_info_get, _v20, 20.0);
+
+void
+rte_cryptodev_info_get_v2002(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
+{
+ struct rte_cryptodev *dev;
+ if (!rte_cryptodev_pmd_is_valid_dev(dev_id)) {
+ CDEV_LOG_ERR("Invalid dev_id=%d", dev_id);
+ return;
+ }
+
+ dev = &rte_crypto_devices[dev_id];
+
+ memset(dev_info, 0, sizeof(struct rte_cryptodev_info));
+
+ RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
+ (*dev->dev_ops->dev_infos_get)(dev, dev_info);
+
+ dev_info->driver_name = dev->device->driver->name;
+ dev_info->device = dev->device;
+}
+MAP_STATIC_SYMBOL(void rte_cryptodev_info_get(uint8_t dev_id,
+ struct rte_cryptodev_info *dev_info), rte_cryptodev_info_get_v2002);
int
rte_cryptodev_callback_register(uint8_t dev_id,
diff --git a/lib/librte_cryptodev/rte_cryptodev.h b/lib/librte_cryptodev/rte_cryptodev.h
index 437b8a9..42b7b42 100644
--- a/lib/librte_cryptodev/rte_cryptodev.h
+++ b/lib/librte_cryptodev/rte_cryptodev.h
@@ -24,6 +24,9 @@ extern "C" {
#include <rte_common.h>
#include <rte_config.h>
+#include <rte_compat.h>
+#include <rte_function_versioning.h>
+
extern const char **rte_cyptodev_names;
/* Logging Macros */
@@ -217,6 +220,15 @@ struct rte_cryptodev_asym_capability_idx {
* - Return NULL if the capability not exist.
*/
const struct rte_cryptodev_symmetric_capability *
+rte_cryptodev_sym_capability_get_v20(uint8_t dev_id,
+ const struct rte_cryptodev_sym_capability_idx *idx);
+
+const struct rte_cryptodev_symmetric_capability *
+rte_cryptodev_sym_capability_get_v2002(uint8_t dev_id,
+ const struct rte_cryptodev_sym_capability_idx *idx);
+BIND_DEFAULT_SYMBOL(rte_cryptodev_sym_capability_get, _v2002, 20.0.2);
+
+const struct rte_cryptodev_symmetric_capability *
rte_cryptodev_sym_capability_get(uint8_t dev_id,
const struct rte_cryptodev_sym_capability_idx *idx);
@@ -758,9 +770,33 @@ rte_cryptodev_stats_reset(uint8_t dev_id);
* the last valid element has it's op field set to
* RTE_CRYPTO_OP_TYPE_UNDEFINED.
*/
-extern void
+
+void
rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
+/* An extra element RTE_CRYPTO_AEAD_CHACHA20_POLY1305 is added
+ * to enum rte_crypto_aead_algorithm, also changing the value of
+ * RTE_CRYPTO_AEAD_LIST_END. To maintain ABI compatibility with applications
+ * which linked against earlier versions, preventing them, for example, from
+ * picking up the new value and using it to index into an array sized too small
+ * for it, it is necessary to have two versions of rte_cryptodev_info_get()
+ * The latest version just returns directly the capabilities retrieved from the device.
+ * The compatible version inspects the capabilities retrieved from the device, but only
+ * returns them directly if the new value is not included. If the new value is
+ * included, it allocates space for a copy of the device capabilities,
+ * trims the new value from this and returns this copy. It only needs to do this
+ * once per device. For the corner case of a corner case when the alloc may fail,
+ * an empty capability list is returned, as there is no mechanism to return an error
+ * and adding such a mechanism would itself be an ABI breakage.
+ * The compatible version can be removed after the next major ABI release.
+ */
+
+void
+rte_cryptodev_info_get_v20(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
+
+void
+rte_cryptodev_info_get_v2002(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
+BIND_DEFAULT_SYMBOL(rte_cryptodev_info_get, _v2002, 20.0.2);
/**
* Register a callback function for specific device id.
diff --git a/lib/librte_cryptodev/rte_cryptodev_version.map b/lib/librte_cryptodev/rte_cryptodev_version.map
index 6e41b4b..1f382d1 100644
--- a/lib/librte_cryptodev/rte_cryptodev_version.map
+++ b/lib/librte_cryptodev/rte_cryptodev_version.map
@@ -58,6 +58,13 @@ DPDK_20.0 {
local: *;
};
+DPDK_20.0.2 {
+ global:
+ rte_cryptodev_info_get;
+ rte_cryptodev_sym_capability_get;
+} DPDK_20.0;
+
+
EXPERIMENTAL {
global:
--
2.1.0
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH v2] cryptodev: version rte_cryptodev_info_get function
2020-04-16 8:12 9% [dpdk-dev] [PATCH v2] cryptodev: version rte_cryptodev_info_get function Arek Kusztal
@ 2020-04-16 8:18 3% ` Akhil Goyal
2020-04-16 8:31 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Akhil Goyal @ 2020-04-16 8:18 UTC (permalink / raw)
To: Arek Kusztal, dev, Ray Kinsella, thomas; +Cc: fiona.trahe, bruce.richardson
++Ray and Thomas for review on ABI
> This patch adds versioned function rte_cryptodev_info_get.
> Node 20.0.2 function works the same way it was working before.
> Node 20.0 function strips capability added in 20.05 release
> to prevent some issues with ABI policy. To do that new capability
> array is allocated per device and returned to user instead of the
> original array passed by PMD.
> Because rte_cryptodev_info_get is called by other API functions,
> rte_cryptodev_sym_capability_get function was versioned the same way.
>
> Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
> ---
>
> This patch depends on following patches:
>
> [1] - "[v3] cryptodev: add chacha20-poly1305 aead algorithm"
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2] cryptodev: version rte_cryptodev_info_get function
2020-04-16 8:18 3% ` Akhil Goyal
@ 2020-04-16 8:31 4% ` Thomas Monjalon
2020-04-16 14:03 0% ` Ray Kinsella
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-04-16 8:31 UTC (permalink / raw)
To: Ray Kinsella, Akhil Goyal
Cc: Arek Kusztal, dev, fiona.trahe, bruce.richardson
16/04/2020 10:18, Akhil Goyal:
> ++Ray and Thomas for review on ABI
Akhil, please work with Ray to understand what needs to be checked
in general for ABI. We need you to maintain ABI in crypto area.
First step: run the new ABI checking tool.
> > This patch adds versioned function rte_cryptodev_info_get.
> > Node 20.0.2 function works the same way it was working before.
> > Node 20.0 function strips capability added in 20.05 release
> > to prevent some issues with ABI policy. To do that new capability
> > array is allocated per device and returned to user instead of the
> > original array passed by PMD.
> > Because rte_cryptodev_info_get is called by other API functions,
> > rte_cryptodev_sym_capability_get function was versioned the same way.
> >
> > Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
> > ---
> >
> > This patch depends on following patches:
> >
> > [1] - "[v3] cryptodev: add chacha20-poly1305 aead algorithm"
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH 1/4] vhost: inroduce operation to get vDPA queue stats
2020-04-15 14:36 3% ` Maxime Coquelin
@ 2020-04-16 9:06 3% ` Matan Azrad
2020-04-16 13:19 3% ` Maxime Coquelin
0 siblings, 1 reply; 200+ results
From: Matan Azrad @ 2020-04-16 9:06 UTC (permalink / raw)
To: Maxime Coquelin, dev; +Cc: Slava Ovsiienko, Shahaf Shuler
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="iso-8859-8-i", Size: 5229 bytes --]
Hi Maxime
Can you point on specific vendor specific counter I suggested?
I think all of them come directly from virtio protocols.
äùâ àú Outlook òáåø Android<https://aka.ms/ghei36>
________________________________
From: Maxime Coquelin <maxime.coquelin@redhat.com>
Sent: Wednesday, April 15, 2020 5:36:59 PM
To: Matan Azrad <matan@mellanox.com>; dev@dpdk.org <dev@dpdk.org>
Cc: Slava Ovsiienko <viacheslavo@mellanox.com>; Shahaf Shuler <shahafs@mellanox.com>
Subject: Re: [PATCH 1/4] vhost: inroduce operation to get vDPA queue stats
Hi Matan,
On 4/2/20 1:26 PM, Matan Azrad wrote:
> The vDPA device offloads all the datapath of the vhost device to the HW
> device.
>
> In order to expose to the user traffic information this patch introduce
> new API to get traffic statistics per virtio queue.
>
> The statistics are taken directly from the vDPA driver managing the HW
> device.
>
> Signed-off-by: Matan Azrad <matan@mellanox.com>
> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> ---
> doc/guides/rel_notes/release_20_05.rst | 4 +++
> doc/guides/vdpadevs/features/default.ini | 1 +
> doc/guides/vdpadevs/features_overview.rst | 3 +++
> lib/librte_vhost/rte_vdpa.h | 45 ++++++++++++++++++++++++++++++-
> lib/librte_vhost/rte_vhost_version.map | 1 +
> lib/librte_vhost/vdpa.c | 14 ++++++++++
> 6 files changed, 67 insertions(+), 1 deletion(-)
...
> diff --git a/lib/librte_vhost/rte_vdpa.h b/lib/librte_vhost/rte_vdpa.h
> index 9a3deb3..d6cbf48 100644
> --- a/lib/librte_vhost/rte_vdpa.h
> +++ b/lib/librte_vhost/rte_vdpa.h
> @@ -37,6 +37,27 @@ struct rte_vdpa_dev_addr {
> };
> };
>
> +struct rte_vdpa_queue_stats {
> + /** Number of descriptors received by device */
> + uint64_t received_desc;
> + /** Number of descriptors completed by the device */
> + uint64_t completed_desc;
> + /** Number of bad descriptors received by device */
> + uint32_t bad_desc;
> + /**
> + * Number of chained descriptors received that exceed the max allowed
> + * chain by device
> + */
> + uint32_t exceed_max_chain;
> + /**
> + * Number of times device tried to read or write buffer that is not
> + * registered to the device
> + */
> + uint32_t invalid_buffer;
> + /** Number of errors detected by the device */
> + uint32_t errors;
> +};
> +
I think doing it like that, we risk to keep the rte_vdpa_get_stats API
always experimental, as every vendor will want to add their own counters
and so break the ABI.
How about implementing something similar to rte_eth_xstat?
As these stats are for debugging purpose, it would give you much more
flexibility in adding new counters as HW or firmwares evolves.
What do you think?
Thanks,
Maxime
> /**
> * vdpa device operations
> */
> @@ -73,8 +94,11 @@ struct rte_vdpa_dev_ops {
> int (*get_notify_area)(int vid, int qid,
> uint64_t *offset, uint64_t *size);
>
> + /** Get statistics of the queue */
> + int (*get_stats)(int did, int qid, struct rte_vdpa_queue_stats *stats);
> +
> /** Reserved for future extension */
> - void *reserved[5];
> + void *reserved[4];
> };
>
> /**
> @@ -200,4 +224,23 @@ struct rte_vdpa_device *
> __rte_experimental
> int
> rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m);
> +
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice
> + *
> + * Get vDPA device queue statistics.
> + *
> + * @param did
> + * device id
> + * @param qid
> + * queue id
> + * @param stats
> + * queue statistics pointer.
> + * @return
> + * 0 on success, non-zero on failure.
> + */
> +__rte_experimental
> +int
> +rte_vdpa_get_stats(int did, uint16_t qid, struct rte_vdpa_queue_stats *stats);
> #endif /* _RTE_VDPA_H_ */
> diff --git a/lib/librte_vhost/rte_vhost_version.map b/lib/librte_vhost/rte_vhost_version.map
> index 051d08c..c9dcff4 100644
> --- a/lib/librte_vhost/rte_vhost_version.map
> +++ b/lib/librte_vhost/rte_vhost_version.map
> @@ -38,6 +38,7 @@ EXPERIMENTAL {
> rte_vdpa_find_device_id;
> rte_vdpa_get_device;
> rte_vdpa_get_device_num;
> + rte_vdpa_get_stats;
> rte_vhost_driver_attach_vdpa_device;
> rte_vhost_driver_detach_vdpa_device;
> rte_vhost_driver_get_vdpa_device_id;
> diff --git a/lib/librte_vhost/vdpa.c b/lib/librte_vhost/vdpa.c
> index 2b86708..57900fc 100644
> --- a/lib/librte_vhost/vdpa.c
> +++ b/lib/librte_vhost/vdpa.c
> @@ -227,3 +227,17 @@ struct rte_vdpa_device *
> free_ind_table(idesc);
> return -1;
> }
> +
> +int
> +rte_vdpa_get_stats(int did, uint16_t qid, struct rte_vdpa_queue_stats *stats)
> +{
> + struct rte_vdpa_device *vdpa_dev;
> +
> + vdpa_dev = rte_vdpa_get_device(did);
> + if (!vdpa_dev)
> + return -ENODEV;
> +
> + RTE_FUNC_PTR_OR_ERR_RET(vdpa_dev->ops->get_stats, -ENOTSUP);
> +
> + return vdpa_dev->ops->get_stats(did, qid, stats);
> +}
>
äùâ àú Outlook òáåø Android<https://aka.ms/ghei36>
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [dpdk-dev v5 3/3] app/testpmd: add new types to RSS hash commands
2020-04-16 7:51 3% ` Jeff Guo
@ 2020-04-16 9:14 0% ` Iremonger, Bernard
2020-04-16 10:22 0% ` Jeff Guo
0 siblings, 1 reply; 200+ results
From: Iremonger, Bernard @ 2020-04-16 9:14 UTC (permalink / raw)
To: Guo, Jia, orika, Ye, Xiaolong, Zhang, Qi Z
Cc: dev, Wu, Jingjing, Cao, Yahui, Su, Simei
Hi Jeff,
> -----Original Message-----
> From: Guo, Jia <jia.guo@intel.com>
> Sent: Thursday, April 16, 2020 8:52 AM
> To: Iremonger, Bernard <bernard.iremonger@intel.com>;
> orika@mellanox.com; Ye, Xiaolong <xiaolong.ye@intel.com>; Zhang, Qi Z
> <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Wu, Jingjing <jingjing.wu@intel.com>; Cao, Yahui
> <yahui.cao@intel.com>; Su, Simei <simei.su@intel.com>
> Subject: Re: [dpdk-dev v5 3/3] app/testpmd: add new types to RSS hash
> commands
>
> hi, bernard
>
>
> On 4/15/2020 11:01 PM, Iremonger, Bernard wrote:
> > Hi Jeff,
> >
> >> -----Original Message-----
> >> From: Guo, Jia <jia.guo@intel.com>
> >> Sent: Wednesday, April 15, 2020 6:11 PM
> >> To: Iremonger, Bernard <bernard.iremonger@intel.com>;
> >> orika@mellanox.com; Ye, Xiaolong <xiaolong.ye@intel.com>; Zhang, Qi Z
> >> <qi.z.zhang@intel.com>
> >> Cc: dev@dpdk.org; Wu, Jingjing <jingjing.wu@intel.com>; Cao, Yahui
> >> <yahui.cao@intel.com>; Su, Simei <simei.su@intel.com>; Guo, Jia
> >> <jia.guo@intel.com>
> >> Subject: [dpdk-dev v5 3/3] app/testpmd: add new types to RSS hash
> >> commands
> >>
> >> Add some new types, such as eth/l2-src-only/l2-dst-only/svlan/cvlan/
> >> l2tpv3/esp/ah/pfcp types into RSS hash commands, it could be used to
> >> configure these rss input set by cmdline.
> >>
> >> Example testpmd commands are:
> >> Eth:
> >> testpmd>flow create 0 ingress pattern eth / ipv4 / end actions rss \
> >> types eth l2-src-only end key_len 0 queues end / end
> >>
> >> testpmd>flow create 0 ingress pattern eth / ipv4 / end actions rss \
> >> types eth l2-dst-only end key_len 0 queues end / end
> >>
> >> testpmd>flow create 0 ingress pattern eth / ipv4 / end actions rss \
> >> types eth end key_len 0 queues end / end
> >>
> >> s-vlan:
> >> testpmd>flow create 0 ingress pattern eth / ipv4 / end actions rss \
> >> types c-vlan end key_len 0 queues end / end
> >>
> >> c-vlan:
> >> testpmd>flow create 0 ingress pattern eth / ipv4 / end actions rss \
> >> types c-vlan end key_len 0 queues end / end
> >>
> >> l2tpv3:
> >> testpmd>flow create 0 ingress pattern eth / ipv4 / l2tpv3oip / end \
> >> actions rss types ipv4 l2tpv3 end key_len 0 queues end / end
> >>
> >> esp:
> >> testpmd>flow create 0 ingress pattern eth / ipv4 / esp / end actions
> >> testpmd>\
> >> rss types ipv4 esp end key_len 0 queues end / end
> > Should "rss types ipv4 esp" be "rss types esp" ?
>
>
> yes, it should be.
>
>
> >> ah:
> >> testpmd>flow create 0 ingress pattern eth / ipv4 / ah / end actions \
> >> rss types ipv4 ah end key_len 0 queues end / end
> >>
> > Should "rss types ah" be "rss types ah" ?
>
>
> yes, the same as above and i think you are right, all action should be specific,
> i think the parse logic should be modify to use pattern to distinguish the ipv4
> ah or ipv6 ah but not use rss type.
>
>
> >> pfcp:
> >> testpmd>flow create 0 ingress pattern eth / ipv4 / udp / pfcp / end \
> >> actions rss types ipv4-udp pfcp end key_len 0 queues end / end
> > The above sample commands might be better in a doc file, rather than in
> the commit message.
> > For example doc/guides/howto/rte_flow.rst
> >
> > The flow create/validate commands are normally parsed in
> > app/testpmd/cmdline_flow.c This patch is updating the following
> commands:
> > "port config all rss all"
> > "port config <port_id> rss-hash-key"
>
>
> Since this is not the command initial and just leverage the command which is
> already exist, so i think i would suppose that no need to modify the doc if no
> change, only change
>
> the part if the change bring in, and i could briefly introduce a example only in
> commit log for simplify.
I still think it would be useful to add the examples above to the doc/guides/howto/rte_flow.rst file which contains other rte_flow examples.
>
>
> >> Signed-off-by: Jeff Guo <jia.guo@intel.com>
> >> ---
> >> v5->v4:
> >> rename eth to l2 and refine commit log
> >> ---
> >> app/test-pmd/cmdline.c | 28 +++++++++++++++++++++++++---
> app/test-
> >> pmd/config.c | 9 +++++++++
> >> 2 files changed, 34 insertions(+), 3 deletions(-)
> >
> >> diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c index
> >> 863b567c1..6b688ab66 100644
> >> --- a/app/test-pmd/cmdline.c
> >> +++ b/app/test-pmd/cmdline.c
> >> @@ -2270,9 +2270,11 @@ cmd_config_rss_parsed(void *parsed_result,
> >> int ret;
> >>
> >> if (!strcmp(res->value, "all"))
> >> -rss_conf.rss_hf = ETH_RSS_IP | ETH_RSS_TCP |
> >> +rss_conf.rss_hf = ETH_RSS_ETH | ETH_RSS_IP |
> >> ETH_RSS_TCP |
> >> ETH_RSS_UDP | ETH_RSS_SCTP |
> >> ETH_RSS_L2_PAYLOAD;
> >> +else if (!strcmp(res->value, "eth"))
> >> +rss_conf.rss_hf = ETH_RSS_ETH;
> >> else if (!strcmp(res->value, "ip"))
> >> rss_conf.rss_hf = ETH_RSS_IP;
> >> else if (!strcmp(res->value, "udp"))
> >> @@ -2299,6 +2301,22 @@ cmd_config_rss_parsed(void *parsed_result,
> >> rss_conf.rss_hf = ETH_RSS_L4_SRC_ONLY;
> >> else if (!strcmp(res->value, "l4-dst-only"))
> >> rss_conf.rss_hf = ETH_RSS_L4_DST_ONLY;
> >> +else if (!strcmp(res->value, "l2-src-only"))
> >> +rss_conf.rss_hf = ETH_RSS_L2_SRC_ONLY;
> >> +else if (!strcmp(res->value, "l2-dst-only"))
> >> +rss_conf.rss_hf = ETH_RSS_L2_DST_ONLY;
> >> +else if (!strcmp(res->value, "s-vlan"))
> >> +rss_conf.rss_hf = ETH_RSS_S_VLAN;
> >> +else if (!strcmp(res->value, "c-vlan"))
> >> +rss_conf.rss_hf = ETH_RSS_C_VLAN;
> >> +else if (!strcmp(res->value, "l2tpv3"))
> >> +rss_conf.rss_hf = ETH_RSS_L2TPV3;
> >> +else if (!strcmp(res->value, "esp"))
> >> +rss_conf.rss_hf = ETH_RSS_ESP;
> >> +else if (!strcmp(res->value, "ah"))
> >> +rss_conf.rss_hf = ETH_RSS_AH;
> >> +else if (!strcmp(res->value, "pfcp"))
> >> +rss_conf.rss_hf = ETH_RSS_PFCP;
> >> else if (!strcmp(res->value, "none"))
> >> rss_conf.rss_hf = 0;
> >> else if (!strcmp(res->value, "default")) @@ -2467,7 +2485,9 @@
> >> cmdline_parse_token_string_t cmd_config_rss_hash_key_rss_type =
> >> "ipv4-other#ipv6#ipv6-frag#ipv6-tcp#ipv6-
> >> udp#"
> >> "ipv6-sctp#ipv6-other#l2-payload#ipv6-ex#"
> >> "ipv6-tcp-ex#ipv6-udp-ex#"
> >> - "l3-src-only#l3-dst-only#l4-src-only#l4-dst-
> >> only");
> >> + "l3-src-only#l3-dst-only#l4-src-only#l4-dst-
> >> only#"
> >> + "l2-src-only#l2-dst-only#s-vlan#c-vlan#"
> >> + "l2tpv3#esp#ah#pfcp");
> >> cmdline_parse_token_string_t cmd_config_rss_hash_key_value =
> >> TOKEN_STRING_INITIALIZER(struct cmd_config_rss_hash_key, key,
> >> NULL);
> >>
> >> @@ -2478,7 +2498,9 @@ cmdline_parse_inst_t
> cmd_config_rss_hash_key =
> >> {
> >> "ipv4|ipv4-frag|ipv4-tcp|ipv4-udp|ipv4-sctp|ipv4-other|"
> >> "ipv6|ipv6-frag|ipv6-tcp|ipv6-udp|ipv6-sctp|ipv6-other|"
> >> "l2-payload|ipv6-ex|ipv6-tcp-ex|ipv6-udp-ex|"
> >> -"l3-src-only|l3-dst-only|l4-src-only|l4-dst-only "
> >> +"l3-src-only|l3-dst-only|l4-src-only|l4-dst-only|"
> >> +"l2-src-only|l2-dst-only|s-vlan|c-vlan|"
> >> +"l2tpv3|esp|ah|pfcp "
> >> "<string of hex digits (variable length, NIC dependent)>",
> >> .tokens = {
> >> (void *)&cmd_config_rss_hash_key_port, diff --git
> >> a/app/test-pmd/config.c b/app/test-pmd/config.c index
> >> 71aeb5413..e4a97388b 100644
> >> --- a/app/test-pmd/config.c
> >> +++ b/app/test-pmd/config.c
> >> @@ -79,6 +79,11 @@ const struct rss_type_info rss_type_table[] = {
> >> ETH_RSS_UDP | ETH_RSS_SCTP |
> >> ETH_RSS_L2_PAYLOAD },
> >> { "none", 0 },
> >> +{ "eth", ETH_RSS_ETH },
> >> +{ "l2-src-only", ETH_RSS_L2_SRC_ONLY },
> >> +{ "l2-dst-only", ETH_RSS_L2_DST_ONLY },
> >> +{ "s-vlan", ETH_RSS_S_VLAN },
> >> +{ "c-vlan", ETH_RSS_C_VLAN },
> > Should these additions be at the bottom of the array with the other
> additions ?
>
>
> I suppose this is not in a library code or application binary interface,
> so it would not break ABI/API, and re-order it as up-down layer must be
> better.
Ok.
>
>
> >> { "ipv4", ETH_RSS_IPV4 },
> >> { "ipv4-frag", ETH_RSS_FRAG_IPV4 },
> >> { "ipv4-tcp", ETH_RSS_NONFRAG_IPV4_TCP }, @@ -108,6 +113,10
> >> @@ const struct rss_type_info rss_type_table[] = {
> >> { "l3-dst-only", ETH_RSS_L3_DST_ONLY },
> >> { "l4-src-only", ETH_RSS_L4_SRC_ONLY },
> >> { "l4-dst-only", ETH_RSS_L4_DST_ONLY },
> >> +{ "l2tpv3", ETH_RSS_L2TPV3 },
> >> +{ "esp", ETH_RSS_ESP },
> >> +{ "ah", ETH_RSS_AH },
> >> +{ "pfcp", ETH_RSS_PFCP },
> >> { NULL, 0 },
> >> };
> >>
> >> --
> >> 2.20.1
Regards,
Bernard.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
2020-04-15 17:24 5% ` Trahe, Fiona
@ 2020-04-16 9:51 3% ` Bruce Richardson
2020-04-16 10:01 3% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Bruce Richardson @ 2020-04-16 9:51 UTC (permalink / raw)
To: Trahe, Fiona; +Cc: Ray Kinsella, dev, Thomas Monjalon, Kusztal, ArkadiuszX
On Wed, Apr 15, 2020 at 06:24:19PM +0100, Trahe, Fiona wrote:
> Hi Ray/Bruce/Thomas,
>
Hi Fiona,
answers as best I can give are inline below.
> We've been trying to make sense of the versioning scheme - but as the docs describe and give examples
> of a 2-number scheme and the code has a 3-number scheme it's pretty confusing.
> If you can help me fill out the following table this may help:
>
> VERSION | ABI_VERSION | LIBABIVER| stable soname | lib filename | new section with ABI break in map file
> 19.11.0 | 20.0 | 1 | stable.so.20.0 | stable.so.20.0.0 |
> 20.02.0-rc0| 20.1 | 1 | n/a
> 20.02.0 | 20.0.1 | 0.1 | stable.so.20.0 | stable.so.20.0.1 |
> 20.05.0 | 20.0.2 | 0.1 | stable.so.20.0 | stable.so.20.0.2 | 20.0.2 (or 21? use impending stable ABI_VERSION?)
> 20.08.0 | 20.0.3 | 0.1 | stable.so.20.0 | stable.so.20.0.3 |
> 20.11.0 | 21 | 0.1 | stable.so.21 | stable.so.21.0 | (Can we move back to a 2 number ABI_VERSION at this stage? )
> 21.02.0 | 21.1 | 0.1 | stable.so.21 | stable.so.21.1 |
>
> Note, first 2 ABI_VERSIONS above were mistakes, soname needs to be same as 19.11 to avoid ABI breakage, so it was changed to 3-part number, with first 2 used in soname.
>
> Questions:
> 1. For this year, is major a 2-part num 20.0 (and there will never be a 20.1,20.2, etc) and minor is the 3rd number?
Yes, for this year only.
> 2. Can we move back to a 2-number scheme in 20.11?
That is the plan, yes.
> 3. What uses lib filename? (This is described in config/meson.build) Guessing stable soname is name on which apps depend, and is a sym link, pointing to lib filename?
In general, yes.
> 4. What does LIBABIVER have to do with this ? it was 1, changed to 0.1 in 20.2
Not sure about this one, what it refers to.
> 5a. If in 20.05 we add a version of a fn which breaks ABI 20.0, what should the name of the original function be? fn_v20, or fn_v20.0
In technical terms it really doesn't matter, it's just a name that will be
looked up in a table. I don't think we strictly enforce the naming, so
whatever is clearest is best. I'd suggest the former.
> Or does it even matter, i.e. is the middle param to this macro arbitrary and just needs to match the fn name, e.g. if fn name was fn_vxyz could macro be:
> VERSION_SYMBOL(fn, _vxyz, 20.0);
Yes.
> 5b. what should the name of the new fn be? fn_v2002 or fn_v21? Does it need to correspond to the new section in .map? If v2002, doesn't that imply it's compatible with 20.0 ABI
> So shouldn't the new section be DPDK_21 and the new fn be fn_v21.
> 6. Assume we go with DPDK_21 and fn_v21. An app built against 20.05 has a dependency on fn_v21 (because of the BIND_DEFAULT) but on otherfn_v20.0 of all other fns?
> If fn_v21 is changed again in 20.08, is there any expectation that the app built against 20.05 must work against the 20.08 fn of the same name?
> i.e. is compatibility required on fn changes across minor nonstable releases? If not then jumping ahead to v21 seems the simplest option.
> And the "final" fn_v21 in 20.11 is the one that "sticks" and belongs to the new ABI which then must remain stable til 21.11
For functions that are part of the stable ABI, each change requires a new
version, since there is an expectation that 20.05 builds will also work
with 20.08.
Regards,
/Bruce
>
>
> > -----Original Message-----
> > From: Trahe, Fiona <fiona.trahe@intel.com>
> > Sent: Tuesday, April 14, 2020 7:27 PM
> > To: Ray Kinsella <mdr@ashroe.eu>; dev@dpdk.org
> > Cc: Trahe, Fiona <fiona.trahe@intel.com>; Kusztal, ArkadiuszX <arkadiuszx.kusztal@intel.com>
> > Subject: RE: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
> >
> > Hi Ray,
> >
> > We're going to need hep to understand the numbering.
> > The examples here http://doc.dpdk.org/guides/contributing/abi_versioning.html#major-abi-versions
> > show only major numbers in the .map file.
> > Whereas the map files all have major.minor.
> >
> > The example shows that to add a new version of an existing fn one should use the next major number,
> > "When an ABI change is made between major ABI versions to a given library, a new section is added to
> > that library’s version map describing the impending new ABI version"
> > so
> > - Old fn becomes fn_v20()
> > - VERSION_SYMBOL(fn, _v20, 20); - maps node 20 to fn_v20() for dynamic linking
> > - New fn becomes fn_v21()
> > - BIND_DEFAULT_SYMBOL(fn, _v21, 21); - maps new builds to fn_v21 for dynamic linking and makes
> > fn_v21 default
> > - MAP_STATIC_SYMBOL(fn_proto, fn_v21); - for static linking
> > And the map file should have:
> >
> > DPDK_21 {
> > global:
> > rte_cryptodev_info_get;
> > } DPDK_20;
> >
> > When the ABI version moves to node 21 in DPDK20.11, the _v20 symbols can be removed.
> >
> > You suggest using DPDK_20.0.2 and _v2002
> > Can you explain why?
> > In 20.0.2 - which number is minor?
> > And why 3 numbers?
> >
> > Regards,
> > Fiona
> >
> >
> > > -----Original Message-----
> > > From: dev <dev-bounces@dpdk.org> On Behalf Of Ray Kinsella
> > > Sent: Tuesday, April 14, 2020 2:54 PM
> > > To: dev@dpdk.org
> > > Subject: Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
> > >
> > >
> > >
> > > On 18/03/2020 20:41, Arek Kusztal wrote:
> > > > This patch adds versioned function rte_cryptodev_info_get.
> > > > Node 20.05 function works the same way it was working before.
> > > > Node 20.0 function strips capability added in 20.05 release
> > > > to prevent some issues with ABI policy. To do that new capability
> > > > array is allocated per device and returned to user instead of the
> > > > original array passed by PMD.
> > > >
> > > > Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
> > > > ---
> > > > lib/librte_cryptodev/rte_cryptodev.c | 97 +++++++++++++++++++++++++-
> > > > lib/librte_cryptodev/rte_cryptodev.h | 12 +++-
> > > > lib/librte_cryptodev/rte_cryptodev_version.map | 7 ++
> > > > 3 files changed, 113 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/lib/librte_cryptodev/rte_cryptodev.c b/lib/librte_cryptodev/rte_cryptodev.c
> > > > index 6d1d0e9..2d030ba 100644
> > > > --- a/lib/librte_cryptodev/rte_cryptodev.c
> > > > +++ b/lib/librte_cryptodev/rte_cryptodev.c
> > > > @@ -41,6 +41,9 @@
> > > > #include "rte_cryptodev.h"
> > > > #include "rte_cryptodev_pmd.h"
> > > >
> > > > +#include <rte_compat.h>
> > > > +#include <rte_function_versioning.h>
> > > > +
> > > > static uint8_t nb_drivers;
> > > >
> > > > static struct rte_cryptodev rte_crypto_devices[RTE_CRYPTO_MAX_DEVS];
> > > > @@ -56,6 +59,13 @@ static struct rte_cryptodev_global cryptodev_globals = {
> > > > /* spinlock for crypto device callbacks */
> > > > static rte_spinlock_t rte_cryptodev_cb_lock = RTE_SPINLOCK_INITIALIZER;
> > > >
> > > > +static const struct rte_cryptodev_capabilities
> > > > +cryptodev_undefined_capabilities[] = {
> > > > +RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST()
> > > > +};
> > > > +
> > > > +struct rte_cryptodev_capabilities *capability_copies[RTE_CRYPTO_MAX_DEVS];
> > > > +uint8_t is_capability_checked[RTE_CRYPTO_MAX_DEVS];
> > > >
> > > > /**
> > > > * The user application callback description.
> > > > @@ -999,6 +1009,13 @@ rte_cryptodev_close(uint8_t dev_id)
> > > > RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_close, -ENOTSUP);
> > > > retval = (*dev->dev_ops->dev_close)(dev);
> > > >
> > > > +
> > > > +if (capability_copies[dev_id]) {
> > > > +free(capability_copies[dev_id]);
> > > > +capability_copies[dev_id] = NULL;
> > > > +}
> > > > +is_capability_checked[dev_id] = 0;
> > > > +
> > > > if (retval < 0)
> > > > return retval;
> > > >
> > > > @@ -1111,11 +1128,12 @@ rte_cryptodev_stats_reset(uint8_t dev_id)
> > > > (*dev->dev_ops->stats_reset)(dev);
> > > > }
> > > >
> > > > -
> > > > void
> > > > -rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
> > > > +rte_cryptodev_info_get_v20(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
> > > > {
> > > > struct rte_cryptodev *dev;
> > > > +const struct rte_cryptodev_capabilities *capability;
> > > > +uint8_t counter = 0;
> > > >
> > > > if (!rte_cryptodev_pmd_is_valid_dev(dev_id)) {
> > > > CDEV_LOG_ERR("Invalid dev_id=%d", dev_id);
> > > > @@ -1129,10 +1147,85 @@ rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info
> > > *dev_info)
> > > > RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
> > > > (*dev->dev_ops->dev_infos_get)(dev, dev_info);
> > > >
> > > > +if (capability_copies[dev_id] == NULL) {
> > > > +if (!is_capability_checked[dev_id]) {
> > > > +uint8_t found_invalid_capa = 0;
> > > > +
> > > > +for (capability = dev_info->capabilities;
> > > > +capability->op != RTE_CRYPTO_OP_TYPE_UNDEFINED;
> > > > +++capability, ++counter) {
> > > > +if (capability->op == RTE_CRYPTO_OP_TYPE_SYMMETRIC &&
> > > > +capability->sym.xform_type ==
> > > > +RTE_CRYPTO_SYM_XFORM_AEAD
> > > > +&& capability->sym.aead.algo >=
> > > > +RTE_CRYPTO_AEAD_CHACHA20_POLY1305) {
> > > > +found_invalid_capa = 1;
> > > > +counter--;
> > > > +}
> > > > +}
> > > > +is_capability_checked[dev_id] = 1;
> > > > +if (found_invalid_capa) {
> > > > +capability_copies[dev_id] = malloc(counter *
> > > > +sizeof(struct rte_cryptodev_capabilities));
> > > > +if (capability_copies[dev_id] == NULL) {
> > > > + /*
> > > > + * error case - no memory to store the trimmed list,
> > > > + * so have to return an empty list
> > > > + */
> > > > +dev_info->capabilities =
> > > > +cryptodev_undefined_capabilities;
> > > > +is_capability_checked[dev_id] = 0;
> > > > +} else {
> > > > +counter = 0;
> > > > +for (capability = dev_info->capabilities;
> > > > +capability->op !=
> > > > +RTE_CRYPTO_OP_TYPE_UNDEFINED;
> > > > +capability++) {
> > > > +if (!(capability->op ==
> > > > +RTE_CRYPTO_OP_TYPE_SYMMETRIC
> > > > +&& capability->sym.xform_type ==
> > > > +RTE_CRYPTO_SYM_XFORM_AEAD
> > > > +&& capability->sym.aead.algo >=
> > > > +
> > > RTE_CRYPTO_AEAD_CHACHA20_POLY1305)) {
> > > > +capability_copies[dev_id][counter++] =
> > > > +*capability;
> > > > +}
> > > > +}
> > > > +dev_info->capabilities =
> > > > +capability_copies[dev_id];
> > > > +}
> > > > +}
> > > > +}
> > > > +} else
> > > > +dev_info->capabilities = capability_copies[dev_id];
> > > > +
> > > > dev_info->driver_name = dev->device->driver->name;
> > > > dev_info->device = dev->device;
> > > > }
> > > > +VERSION_SYMBOL(rte_cryptodev_info_get, _v20, 20.0);
> > > > +
> > > > +void
> > > > +rte_cryptodev_info_get_v2005(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
> > >
> > > should be rte_cryptodev_info_get_v2002
> > >
> > > > +{
> > > > +struct rte_cryptodev *dev;
> > > >
> > > > +if (!rte_cryptodev_pmd_is_valid_dev(dev_id)) {
> > > > +CDEV_LOG_ERR("Invalid dev_id=%d", dev_id);
> > > > +return;
> > > > +}
> > > > +
> > > > +dev = &rte_crypto_devices[dev_id];
> > > > +
> > > > +memset(dev_info, 0, sizeof(struct rte_cryptodev_info));
> > > > +
> > > > +RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
> > > > +(*dev->dev_ops->dev_infos_get)(dev, dev_info);
> > > > +
> > > > +dev_info->driver_name = dev->device->driver->name;
> > > > +dev_info->device = dev->device;
> > > > +}
> > > > +MAP_STATIC_SYMBOL(void rte_cryptodev_info_get(uint8_t dev_id,
> > > > +struct rte_cryptodev_info *dev_info), rte_cryptodev_info_get_v2005);
> > >
> > > should be rte_cryptodev_info_get_v2002
> > >
> > > >
> > > > int
> > > > rte_cryptodev_callback_register(uint8_t dev_id,
> > > > diff --git a/lib/librte_cryptodev/rte_cryptodev.h b/lib/librte_cryptodev/rte_cryptodev.h
> > > > index 437b8a9..06ce2f2 100644
> > > > --- a/lib/librte_cryptodev/rte_cryptodev.h
> > > > +++ b/lib/librte_cryptodev/rte_cryptodev.h
> > > > @@ -24,6 +24,9 @@ extern "C" {
> > > > #include <rte_common.h>
> > > > #include <rte_config.h>
> > > >
> > > > +#include <rte_compat.h>
> > > > +#include <rte_function_versioning.h>
> > > > +
> > > > extern const char **rte_cyptodev_names;
> > > >
> > > > /* Logging Macros */
> > > > @@ -758,7 +761,14 @@ rte_cryptodev_stats_reset(uint8_t dev_id);
> > > > * the last valid element has it's op field set to
> > > > * RTE_CRYPTO_OP_TYPE_UNDEFINED.
> > > > */
> > > > -extern void
> > > > +void
> > > > +rte_cryptodev_info_get_v20(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
> > > > +
> > > > +void
> > > > +rte_cryptodev_info_get_v2005(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
> > > > +BIND_DEFAULT_SYMBOL(rte_cryptodev_info_get, _v2005, 20.05);
> > > > +
> > > > +void
> > > > rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
> > > do we still need this forward declaration?
> > >
> > > >
> > > >
> > > > diff --git a/lib/librte_cryptodev/rte_cryptodev_version.map
> > > b/lib/librte_cryptodev/rte_cryptodev_version.map
> > > > index 6e41b4b..d6c945a 100644
> > > > --- a/lib/librte_cryptodev/rte_cryptodev_version.map
> > > > +++ b/lib/librte_cryptodev/rte_cryptodev_version.map
> > > > @@ -58,6 +58,13 @@ DPDK_20.0 {
> > > > local: *;
> > > > };
> > > >
> > > > +
> > > > +DPDK_20.05 {
> > >
> > > should be DPDK_20.0.2
> > >
> > > > +global:
> > > > +rte_cryptodev_info_get;
> > > > +} DPDK_20.0;
> > > > +
> > > > +
> > > > EXPERIMENTAL {
> > > > global:
> > > >
> > > >
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
2020-04-16 9:51 3% ` Bruce Richardson
@ 2020-04-16 10:01 3% ` Thomas Monjalon
2020-04-17 7:24 5% ` Ray Kinsella
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-04-16 10:01 UTC (permalink / raw)
To: Trahe, Fiona, Bruce Richardson; +Cc: Ray Kinsella, dev, Kusztal, ArkadiuszX
16/04/2020 11:51, Bruce Richardson:
> On Wed, Apr 15, 2020 at 06:24:19PM +0100, Trahe, Fiona wrote:
> > 5a. If in 20.05 we add a version of a fn which breaks ABI 20.0, what should the name of the original function be? fn_v20, or fn_v20.0
>
> In technical terms it really doesn't matter, it's just a name that will be
> looked up in a table. I don't think we strictly enforce the naming, so
> whatever is clearest is best. I'd suggest the former.
Each release can have a new ABI.
The same function can have a different version in 20.02, 20.05 and 20.08.
If you name it fn_v20 in 20.05, what will be the name for the new version
in 20.08? I suggest using the release number when versioning a function.
^ permalink raw reply [relevance 3%]
* [dpdk-dev] DPDK Release Status Meeting 16/04/2020
@ 2020-04-16 10:06 5% Ferruh Yigit
2020-04-16 10:49 0% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-04-16 10:06 UTC (permalink / raw)
To: dpdk-dev; +Cc: Thomas Monjalon
Minutes 16 April 2020
---------------------
Agenda:
* Release Dates
* Subtrees
* OvS
Participants:
* Arm
* Debian/Microsoft
* Intel
* Marvell
* Mellanox
* NXP
* Red Hat
Release Dates
-------------
* v20.05 dates:
* Integration/Merge/RC1 pushed to *Thursday 23 April 2020*
* Release: Wednesday 20 May 2020
* Marvell, Arm, Mellanox & Broadcom already sent roadmap for 20.05, all
vendors please share the roadmap for the release.
Subtrees
--------
* main
* There are still lots of patches, a lot of reviews are missing
* Sub-trees will be pulled next week Monday/Tuesday
* Sub-trees should be ready by EOD Friday or Saturday
* Thomas did review on graph library, planning to get it as it is
* Trace library will be merged for -rc1, the review was missing but at least
feedback requested
* Some library features may go in for -rc2 due to big backlog
* If the work is not finished, planning to block related patches
* Example: ABI breaks for false positive, don't get patches that
breaks ABI before fixing the ABI check tools.
* next-net
* Less than 50 patches in the backlog
* Some ethdev patches are not merged yet, close but not ready for -rc1
* mlx set that break the ABI not clarified yet, it may be dropped from tree
* next-crypto
* ~50 patches in backlog, reviewed most of them, will apply today
* ipsec patch depends on hash change which is for main repo
* Thomas will look to it with priority
* ABI related crypto patch will be reviewed today
* There is a reported build error from Raslan
* next-eventdev
* No more patches for -rc1, already pulled to main
* next-virtio
* Lots of reviews done last and this week, still have bunch of patches
* Some reviewed already and can be merged quickly
* Ones waiting for new version can go into next -rc
* There is a performance optimization for Arm patch waiting for review
* one way barrier for split vring idx
* next-net-intel
* Waiting for some major patchsets related to iavf
* They may not be ready for tomorrow
* next-net-mlx
* There are some big patchsets waiting, they are not blocking the -rc1
* flow aging patches waiting to be merged in next-net
* next-net-mrvl
* All patches merged for -rc1
* next-net-brcm
* There is a big patch merged recently, will pull it to next-net today
* LTS
* 18.11.7 released
* http://inbox.dpdk.org/dev/20200415173701.19706-1-ktraynor@redhat.com/
OvS
---
* There was another meeting last week, ping Kevin for details.
DPDK Release Status Meetings
============================
The DPDK Release Status Meeting is intended for DPDK Committers to discuss
the status of the master tree and sub-trees, and for project managers to
track progress or milestone dates.
The meeting occurs on Thursdays at 8:30 UTC. If you wish to attend just
send an email to "John McNamara <john.mcnamara@intel.com>" for the invite.
^ permalink raw reply [relevance 5%]
* Re: [dpdk-dev] [dpdk-dev v5 3/3] app/testpmd: add new types to RSS hash commands
2020-04-16 9:14 0% ` Iremonger, Bernard
@ 2020-04-16 10:22 0% ` Jeff Guo
0 siblings, 0 replies; 200+ results
From: Jeff Guo @ 2020-04-16 10:22 UTC (permalink / raw)
To: Iremonger, Bernard, orika, Ye, Xiaolong, Zhang, Qi Z
Cc: dev, Wu, Jingjing, Cao, Yahui, Su, Simei
hi, bernard
On 4/16/2020 5:14 PM, Iremonger, Bernard wrote:
> Hi Jeff,
>
>> -----Original Message-----
>> From: Guo, Jia <jia.guo@intel.com>
>> Sent: Thursday, April 16, 2020 8:52 AM
>> To: Iremonger, Bernard <bernard.iremonger@intel.com>;
>> orika@mellanox.com; Ye, Xiaolong <xiaolong.ye@intel.com>; Zhang, Qi Z
>> <qi.z.zhang@intel.com>
>> Cc: dev@dpdk.org; Wu, Jingjing <jingjing.wu@intel.com>; Cao, Yahui
>> <yahui.cao@intel.com>; Su, Simei <simei.su@intel.com>
>> Subject: Re: [dpdk-dev v5 3/3] app/testpmd: add new types to RSS hash
>> commands
>>
>> hi, bernard
>>
>>
>> On 4/15/2020 11:01 PM, Iremonger, Bernard wrote:
>>> Hi Jeff,
>>>
>>>> -----Original Message-----
>>>> From: Guo, Jia <jia.guo@intel.com>
>>>> Sent: Wednesday, April 15, 2020 6:11 PM
>>>> To: Iremonger, Bernard <bernard.iremonger@intel.com>;
>>>> orika@mellanox.com; Ye, Xiaolong <xiaolong.ye@intel.com>; Zhang, Qi Z
>>>> <qi.z.zhang@intel.com>
>>>> Cc: dev@dpdk.org; Wu, Jingjing <jingjing.wu@intel.com>; Cao, Yahui
>>>> <yahui.cao@intel.com>; Su, Simei <simei.su@intel.com>; Guo, Jia
>>>> <jia.guo@intel.com>
>>>> Subject: [dpdk-dev v5 3/3] app/testpmd: add new types to RSS hash
>>>> commands
>>>>
>>>> Add some new types, such as eth/l2-src-only/l2-dst-only/svlan/cvlan/
>>>> l2tpv3/esp/ah/pfcp types into RSS hash commands, it could be used to
>>>> configure these rss input set by cmdline.
>>>>
>>>> Example testpmd commands are:
>>>> Eth:
>>>> testpmd>flow create 0 ingress pattern eth / ipv4 / end actions rss \
>>>> types eth l2-src-only end key_len 0 queues end / end
>>>>
>>>> testpmd>flow create 0 ingress pattern eth / ipv4 / end actions rss \
>>>> types eth l2-dst-only end key_len 0 queues end / end
>>>>
>>>> testpmd>flow create 0 ingress pattern eth / ipv4 / end actions rss \
>>>> types eth end key_len 0 queues end / end
>>>>
>>>> s-vlan:
>>>> testpmd>flow create 0 ingress pattern eth / ipv4 / end actions rss \
>>>> types c-vlan end key_len 0 queues end / end
>>>>
>>>> c-vlan:
>>>> testpmd>flow create 0 ingress pattern eth / ipv4 / end actions rss \
>>>> types c-vlan end key_len 0 queues end / end
>>>>
>>>> l2tpv3:
>>>> testpmd>flow create 0 ingress pattern eth / ipv4 / l2tpv3oip / end \
>>>> actions rss types ipv4 l2tpv3 end key_len 0 queues end / end
>>>>
>>>> esp:
>>>> testpmd>flow create 0 ingress pattern eth / ipv4 / esp / end actions
>>>> testpmd>\
>>>> rss types ipv4 esp end key_len 0 queues end / end
>>> Should "rss types ipv4 esp" be "rss types esp" ?
>>
>> yes, it should be.
>>
>>
>>>> ah:
>>>> testpmd>flow create 0 ingress pattern eth / ipv4 / ah / end actions \
>>>> rss types ipv4 ah end key_len 0 queues end / end
>>>>
>>> Should "rss types ah" be "rss types ah" ?
>>
>> yes, the same as above and i think you are right, all action should be specific,
>> i think the parse logic should be modify to use pattern to distinguish the ipv4
>> ah or ipv6 ah but not use rss type.
>>
>>
>>>> pfcp:
>>>> testpmd>flow create 0 ingress pattern eth / ipv4 / udp / pfcp / end \
>>>> actions rss types ipv4-udp pfcp end key_len 0 queues end / end
>>> The above sample commands might be better in a doc file, rather than in
>> the commit message.
>>> For example doc/guides/howto/rte_flow.rst
>>>
>>> The flow create/validate commands are normally parsed in
>>> app/testpmd/cmdline_flow.c This patch is updating the following
>> commands:
>>> "port config all rss all"
>>> "port config <port_id> rss-hash-key"
>>
>> Since this is not the command initial and just leverage the command which is
>> already exist, so i think i would suppose that no need to modify the doc if no
>> change, only change
>>
>> the part if the change bring in, and i could briefly introduce a example only in
>> commit log for simplify.
>
> I still think it would be useful to add the examples above to the doc/guides/howto/rte_flow.rst file which contains other rte_flow examples.
I think what your suppose is that there is lack of an doc to describe
this rss rule feature what ever pf and vf in the history, so i suppose
to add it in a separate patch next
after leverage all pf and vf about that, what is your or other guys
opinion, i think i am both ok for that.
>>
>>>> Signed-off-by: Jeff Guo <jia.guo@intel.com>
>>>> ---
>>>> v5->v4:
>>>> rename eth to l2 and refine commit log
>>>> ---
>>>> app/test-pmd/cmdline.c | 28 +++++++++++++++++++++++++---
>> app/test-
>>>> pmd/config.c | 9 +++++++++
>>>> 2 files changed, 34 insertions(+), 3 deletions(-)
>>>> diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c index
>>>> 863b567c1..6b688ab66 100644
>>>> --- a/app/test-pmd/cmdline.c
>>>> +++ b/app/test-pmd/cmdline.c
>>>> @@ -2270,9 +2270,11 @@ cmd_config_rss_parsed(void *parsed_result,
>>>> int ret;
>>>>
>>>> if (!strcmp(res->value, "all"))
>>>> -rss_conf.rss_hf = ETH_RSS_IP | ETH_RSS_TCP |
>>>> +rss_conf.rss_hf = ETH_RSS_ETH | ETH_RSS_IP |
>>>> ETH_RSS_TCP |
>>>> ETH_RSS_UDP | ETH_RSS_SCTP |
>>>> ETH_RSS_L2_PAYLOAD;
>>>> +else if (!strcmp(res->value, "eth"))
>>>> +rss_conf.rss_hf = ETH_RSS_ETH;
>>>> else if (!strcmp(res->value, "ip"))
>>>> rss_conf.rss_hf = ETH_RSS_IP;
>>>> else if (!strcmp(res->value, "udp"))
>>>> @@ -2299,6 +2301,22 @@ cmd_config_rss_parsed(void *parsed_result,
>>>> rss_conf.rss_hf = ETH_RSS_L4_SRC_ONLY;
>>>> else if (!strcmp(res->value, "l4-dst-only"))
>>>> rss_conf.rss_hf = ETH_RSS_L4_DST_ONLY;
>>>> +else if (!strcmp(res->value, "l2-src-only"))
>>>> +rss_conf.rss_hf = ETH_RSS_L2_SRC_ONLY;
>>>> +else if (!strcmp(res->value, "l2-dst-only"))
>>>> +rss_conf.rss_hf = ETH_RSS_L2_DST_ONLY;
>>>> +else if (!strcmp(res->value, "s-vlan"))
>>>> +rss_conf.rss_hf = ETH_RSS_S_VLAN;
>>>> +else if (!strcmp(res->value, "c-vlan"))
>>>> +rss_conf.rss_hf = ETH_RSS_C_VLAN;
>>>> +else if (!strcmp(res->value, "l2tpv3"))
>>>> +rss_conf.rss_hf = ETH_RSS_L2TPV3;
>>>> +else if (!strcmp(res->value, "esp"))
>>>> +rss_conf.rss_hf = ETH_RSS_ESP;
>>>> +else if (!strcmp(res->value, "ah"))
>>>> +rss_conf.rss_hf = ETH_RSS_AH;
>>>> +else if (!strcmp(res->value, "pfcp"))
>>>> +rss_conf.rss_hf = ETH_RSS_PFCP;
>>>> else if (!strcmp(res->value, "none"))
>>>> rss_conf.rss_hf = 0;
>>>> else if (!strcmp(res->value, "default")) @@ -2467,7 +2485,9 @@
>>>> cmdline_parse_token_string_t cmd_config_rss_hash_key_rss_type =
>>>> "ipv4-other#ipv6#ipv6-frag#ipv6-tcp#ipv6-
>>>> udp#"
>>>> "ipv6-sctp#ipv6-other#l2-payload#ipv6-ex#"
>>>> "ipv6-tcp-ex#ipv6-udp-ex#"
>>>> - "l3-src-only#l3-dst-only#l4-src-only#l4-dst-
>>>> only");
>>>> + "l3-src-only#l3-dst-only#l4-src-only#l4-dst-
>>>> only#"
>>>> + "l2-src-only#l2-dst-only#s-vlan#c-vlan#"
>>>> + "l2tpv3#esp#ah#pfcp");
>>>> cmdline_parse_token_string_t cmd_config_rss_hash_key_value =
>>>> TOKEN_STRING_INITIALIZER(struct cmd_config_rss_hash_key, key,
>>>> NULL);
>>>>
>>>> @@ -2478,7 +2498,9 @@ cmdline_parse_inst_t
>> cmd_config_rss_hash_key =
>>>> {
>>>> "ipv4|ipv4-frag|ipv4-tcp|ipv4-udp|ipv4-sctp|ipv4-other|"
>>>> "ipv6|ipv6-frag|ipv6-tcp|ipv6-udp|ipv6-sctp|ipv6-other|"
>>>> "l2-payload|ipv6-ex|ipv6-tcp-ex|ipv6-udp-ex|"
>>>> -"l3-src-only|l3-dst-only|l4-src-only|l4-dst-only "
>>>> +"l3-src-only|l3-dst-only|l4-src-only|l4-dst-only|"
>>>> +"l2-src-only|l2-dst-only|s-vlan|c-vlan|"
>>>> +"l2tpv3|esp|ah|pfcp "
>>>> "<string of hex digits (variable length, NIC dependent)>",
>>>> .tokens = {
>>>> (void *)&cmd_config_rss_hash_key_port, diff --git
>>>> a/app/test-pmd/config.c b/app/test-pmd/config.c index
>>>> 71aeb5413..e4a97388b 100644
>>>> --- a/app/test-pmd/config.c
>>>> +++ b/app/test-pmd/config.c
>>>> @@ -79,6 +79,11 @@ const struct rss_type_info rss_type_table[] = {
>>>> ETH_RSS_UDP | ETH_RSS_SCTP |
>>>> ETH_RSS_L2_PAYLOAD },
>>>> { "none", 0 },
>>>> +{ "eth", ETH_RSS_ETH },
>>>> +{ "l2-src-only", ETH_RSS_L2_SRC_ONLY },
>>>> +{ "l2-dst-only", ETH_RSS_L2_DST_ONLY },
>>>> +{ "s-vlan", ETH_RSS_S_VLAN },
>>>> +{ "c-vlan", ETH_RSS_C_VLAN },
>>> Should these additions be at the bottom of the array with the other
>> additions ?
>>
>>
>> I suppose this is not in a library code or application binary interface,
>> so it would not break ABI/API, and re-order it as up-down layer must be
>> better.
> Ok.
>
>>
>>>> { "ipv4", ETH_RSS_IPV4 },
>>>> { "ipv4-frag", ETH_RSS_FRAG_IPV4 },
>>>> { "ipv4-tcp", ETH_RSS_NONFRAG_IPV4_TCP }, @@ -108,6 +113,10
>>>> @@ const struct rss_type_info rss_type_table[] = {
>>>> { "l3-dst-only", ETH_RSS_L3_DST_ONLY },
>>>> { "l4-src-only", ETH_RSS_L4_SRC_ONLY },
>>>> { "l4-dst-only", ETH_RSS_L4_DST_ONLY },
>>>> +{ "l2tpv3", ETH_RSS_L2TPV3 },
>>>> +{ "esp", ETH_RSS_ESP },
>>>> +{ "ah", ETH_RSS_AH },
>>>> +{ "pfcp", ETH_RSS_PFCP },
>>>> { NULL, 0 },
>>>> };
>>>>
>>>> --
>>>> 2.20.1
> Regards,
>
> Bernard.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] DPDK Release Status Meeting 16/04/2020
2020-04-16 10:06 5% [dpdk-dev] DPDK Release Status Meeting 16/04/2020 Ferruh Yigit
@ 2020-04-16 10:49 0% ` Thomas Monjalon
2020-04-16 11:21 0% ` Ferruh Yigit
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-04-16 10:49 UTC (permalink / raw)
To: Ferruh Yigit; +Cc: dpdk-dev
16/04/2020 12:06, Ferruh Yigit:
> * Marvell, Arm, Mellanox & Broadcom already sent roadmap for 20.05, all
> vendors please share the roadmap for the release.
I think it is too late now for 20.05 roadmap :-)
> Subtrees
> --------
>
> * main
> * There are still lots of patches, a lot of reviews are missing
> * Sub-trees will be pulled next week Monday/Tuesday
> * Sub-trees should be ready by EOD Friday or Saturday
> * Thomas did review on graph library, planning to get it as it is
No I did not review rte_graph.
But as it is here for long, it should be merged.
> * Trace library will be merged for -rc1, the review was missing but at least
> feedback requested
David worked on rte_trace reviews.
More reviews were expected.
We hope there will be more feedback after the merge as experimental.
Please let's have open feedback, don't be shy asking for rework
if you have any enhancement idea.
> * Some library features may go in for -rc2 due to big backlog
We will try to minimize feature additions in -rc2 if any.
> * If the work is not finished, planning to block related patches
> * Example: ABI breaks for false positive, don't get patches that
> breaks ABI before fixing the ABI check tools.
Yes we should block patches more often to get work done.
We can find other examples in crypto or ethdev.
We need to define clear deadlines.
Example: any patch in a PMD will be blocked after a deadline
if the PMD does not comply with the requested rework,
as RTE_ETH_DEV_CLOSE_REMOVE behaviour.
> * next-net
> * Less than 50 patches in the backlog
> * Some ethdev patches are not merged yet, close but not ready for -rc1
> * mlx set that break the ABI not clarified yet, it may be dropped from tree
>
> * next-crypto
> * ~50 patches in backlog, reviewed most of them, will apply today
> * ipsec patch depends on hash change which is for main repo
> * Thomas will look to it with priority
It seems hash patches are not ready or not reviewed.
> * ABI related crypto patch will be reviewed today
> * There is a reported build error from Raslan
Looks to be not reproduced.
[...]
> * 18.11.7 released
> * http://inbox.dpdk.org/dev/20200415173701.19706-1-ktraynor@redhat.com/
Congratulations Kevin!
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] DPDK Release Status Meeting 16/04/2020
2020-04-16 10:49 0% ` Thomas Monjalon
@ 2020-04-16 11:21 0% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-04-16 11:21 UTC (permalink / raw)
To: Thomas Monjalon; +Cc: dpdk-dev
On 4/16/2020 11:49 AM, Thomas Monjalon wrote:
> 16/04/2020 12:06, Ferruh Yigit:
>> * Marvell, Arm, Mellanox & Broadcom already sent roadmap for 20.05, all
>> vendors please share the roadmap for the release.
>
> I think it is too late now for 20.05 roadmap :-)
>
>> Subtrees
>> --------
>>
>> * main
>> * There are still lots of patches, a lot of reviews are missing
>> * Sub-trees will be pulled next week Monday/Tuesday
>> * Sub-trees should be ready by EOD Friday or Saturday
>> * Thomas did review on graph library, planning to get it as it is
>
> No I did not review rte_graph.
> But as it is here for long, it should be merged.
>
>> * Trace library will be merged for -rc1, the review was missing but at least
>> feedback requested
>
> David worked on rte_trace reviews.
> More reviews were expected.
> We hope there will be more feedback after the merge as experimental.
> Please let's have open feedback, don't be shy asking for rework
> if you have any enhancement idea.
>
>> * Some library features may go in for -rc2 due to big backlog
>
> We will try to minimize feature additions in -rc2 if any.
>
>> * If the work is not finished, planning to block related patches
>> * Example: ABI breaks for false positive, don't get patches that
>> breaks ABI before fixing the ABI check tools.
>
> Yes we should block patches more often to get work done.
> We can find other examples in crypto or ethdev.
> We need to define clear deadlines.
> Example: any patch in a PMD will be blocked after a deadline
> if the PMD does not comply with the requested rework,
> as RTE_ETH_DEV_CLOSE_REMOVE behaviour.
>
>> * next-net
>> * Less than 50 patches in the backlog
>> * Some ethdev patches are not merged yet, close but not ready for -rc1
>> * mlx set that break the ABI not clarified yet, it may be dropped from tree
>>
>> * next-crypto
>> * ~50 patches in backlog, reviewed most of them, will apply today
>> * ipsec patch depends on hash change which is for main repo
>> * Thomas will look to it with priority
>
> It seems hash patches are not ready or not reviewed.
>
>> * ABI related crypto patch will be reviewed today
>> * There is a reported build error from Raslan
>
> Looks to be not reproduced.
I think this may be valid issue, but reproduced randomly, because in make build,
the ipsec dependency to hash library looks missing.
>
> [...]
>> * 18.11.7 released
>> * http://inbox.dpdk.org/dev/20200415173701.19706-1-ktraynor@redhat.com/
>
> Congratulations Kevin!
>
>
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH 1/4] vhost: inroduce operation to get vDPA queue stats
2020-04-16 9:06 3% ` Matan Azrad
@ 2020-04-16 13:19 3% ` Maxime Coquelin
0 siblings, 0 replies; 200+ results
From: Maxime Coquelin @ 2020-04-16 13:19 UTC (permalink / raw)
To: Matan Azrad, dev, Xiao Wang; +Cc: Slava Ovsiienko, Shahaf Shuler
Hi Matan,
On 4/16/20 11:06 AM, Matan Azrad wrote:
> Hi Maxime
>
> Can you point on specific vendor specific counter I suggested?
No, I can't, but I think we can expect that other vendors may have other
counters they would be interested to dump.
Maybe Intel has some counters in the IFC that they could dump.
Xiao, any thoughts?
> I think all of them come directly from virtio protocols.
exceed_max_chain, for example. Doesn't the spec specify that a
descriptors chain can be as long as the size of the virtqueue?
Here it seems to indicate the device could support less.
Also, as the spec evolves, we may have new counters that comes up,
so better to have something flexible from the start IMHO to avoid ABI
breakages.
Maybe we can have some common xstats names for the Virtio related
counters define in vdpa lib, and then the vendors can specify more
vendor-specific counters if they wish?
Thanks,
Maxime
>
> השג את Outlook עבור Android <https://aka.ms/ghei36>
> ------------------------------------------------------------------------
> *From:* Maxime Coquelin <maxime.coquelin@redhat.com>
> *Sent:* Wednesday, April 15, 2020 5:36:59 PM
> *To:* Matan Azrad <matan@mellanox.com>; dev@dpdk.org <dev@dpdk.org>
> *Cc:* Slava Ovsiienko <viacheslavo@mellanox.com>; Shahaf Shuler
> <shahafs@mellanox.com>
> *Subject:* Re: [PATCH 1/4] vhost: inroduce operation to get vDPA queue
> stats
>
> Hi Matan,
>
> On 4/2/20 1:26 PM, Matan Azrad wrote:
>> The vDPA device offloads all the datapath of the vhost device to the HW
>> device.
>>
>> In order to expose to the user traffic information this patch introduce
>> new API to get traffic statistics per virtio queue.
>>
>> The statistics are taken directly from the vDPA driver managing the HW
>> device.
>>
>> Signed-off-by: Matan Azrad <matan@mellanox.com>
>> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
>> ---
>> doc/guides/rel_notes/release_20_05.rst | 4 +++
>> doc/guides/vdpadevs/features/default.ini | 1 +
>> doc/guides/vdpadevs/features_overview.rst | 3 +++
>> lib/librte_vhost/rte_vdpa.h | 45 ++++++++++++++++++++++++++++++-
>> lib/librte_vhost/rte_vhost_version.map | 1 +
>> lib/librte_vhost/vdpa.c | 14 ++++++++++
>> 6 files changed, 67 insertions(+), 1 deletion(-)
>
> ...
>
>> diff --git a/lib/librte_vhost/rte_vdpa.h b/lib/librte_vhost/rte_vdpa.h
>> index 9a3deb3..d6cbf48 100644
>> --- a/lib/librte_vhost/rte_vdpa.h
>> +++ b/lib/librte_vhost/rte_vdpa.h
>> @@ -37,6 +37,27 @@ struct rte_vdpa_dev_addr {
>> };
>> };
>>
>> +struct rte_vdpa_queue_stats {
>> + /** Number of descriptors received by device */
>> + uint64_t received_desc;
>> + /** Number of descriptors completed by the device */
>> + uint64_t completed_desc;
>> + /** Number of bad descriptors received by device */
>> + uint32_t bad_desc;
>> + /**
>> + * Number of chained descriptors received that exceed the max allowed
>> + * chain by device
>> + */
>> + uint32_t exceed_max_chain;
>> + /**
>> + * Number of times device tried to read or write buffer that is not
>> + * registered to the device
>> + */
>> + uint32_t invalid_buffer;
>> + /** Number of errors detected by the device */
>> + uint32_t errors;
>> +};
>> +
>
> I think doing it like that, we risk to keep the rte_vdpa_get_stats API
> always experimental, as every vendor will want to add their own counters
> and so break the ABI.
>
> How about implementing something similar to rte_eth_xstat?
> As these stats are for debugging purpose, it would give you much more
> flexibility in adding new counters as HW or firmwares evolves.
>
> What do you think?
>
> Thanks,
> Maxime
>
>> /**
>> * vdpa device operations
>> */
>> @@ -73,8 +94,11 @@ struct rte_vdpa_dev_ops {
>> int (*get_notify_area)(int vid, int qid,
>> uint64_t *offset, uint64_t *size);
>>
>> + /** Get statistics of the queue */
>> + int (*get_stats)(int did, int qid, struct rte_vdpa_queue_stats *stats);
>> +
>> /** Reserved for future extension */
>> - void *reserved[5];
>> + void *reserved[4];
>> };
>>
>> /**
>> @@ -200,4 +224,23 @@ struct rte_vdpa_device *
>> __rte_experimental
>> int
>> rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m);
>> +
>> +/**
>> + * @warning
>> + * @b EXPERIMENTAL: this API may change without prior notice
>> + *
>> + * Get vDPA device queue statistics.
>> + *
>> + * @param did
>> + * device id
>> + * @param qid
>> + * queue id
>> + * @param stats
>> + * queue statistics pointer.
>> + * @return
>> + * 0 on success, non-zero on failure.
>> + */
>> +__rte_experimental
>> +int
>> +rte_vdpa_get_stats(int did, uint16_t qid, struct rte_vdpa_queue_stats *stats);
>> #endif /* _RTE_VDPA_H_ */
>> diff --git a/lib/librte_vhost/rte_vhost_version.map b/lib/librte_vhost/rte_vhost_version.map
>> index 051d08c..c9dcff4 100644
>> --- a/lib/librte_vhost/rte_vhost_version.map
>> +++ b/lib/librte_vhost/rte_vhost_version.map
>> @@ -38,6 +38,7 @@ EXPERIMENTAL {
>> rte_vdpa_find_device_id;
>> rte_vdpa_get_device;
>> rte_vdpa_get_device_num;
>> + rte_vdpa_get_stats;
>> rte_vhost_driver_attach_vdpa_device;
>> rte_vhost_driver_detach_vdpa_device;
>> rte_vhost_driver_get_vdpa_device_id;
>> diff --git a/lib/librte_vhost/vdpa.c b/lib/librte_vhost/vdpa.c
>> index 2b86708..57900fc 100644
>> --- a/lib/librte_vhost/vdpa.c
>> +++ b/lib/librte_vhost/vdpa.c
>> @@ -227,3 +227,17 @@ struct rte_vdpa_device *
>> free_ind_table(idesc);
>> return -1;
>> }
>> +
>> +int
>> +rte_vdpa_get_stats(int did, uint16_t qid, struct rte_vdpa_queue_stats *stats)
>> +{
>> + struct rte_vdpa_device *vdpa_dev;
>> +
>> + vdpa_dev = rte_vdpa_get_device(did);
>> + if (!vdpa_dev)
>> + return -ENODEV;
>> +
>> + RTE_FUNC_PTR_OR_ERR_RET(vdpa_dev->ops->get_stats, -ENOTSUP);
>> +
>> + return vdpa_dev->ops->get_stats(did, qid, stats);
>> +}
>>
>
>
> השג את Outlook עבור Android <https://aka.ms/ghei36>
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2] cryptodev: version rte_cryptodev_info_get function
2020-04-16 8:31 4% ` Thomas Monjalon
@ 2020-04-16 14:03 0% ` Ray Kinsella
0 siblings, 0 replies; 200+ results
From: Ray Kinsella @ 2020-04-16 14:03 UTC (permalink / raw)
To: Thomas Monjalon, Akhil Goyal
Cc: Arek Kusztal, dev, fiona.trahe, bruce.richardson
FYI ... not ignoring this, by any means.
Need to check a few points before I respond.
Ray K
On 16/04/2020 09:31, Thomas Monjalon wrote:
> 16/04/2020 10:18, Akhil Goyal:
>> ++Ray and Thomas for review on ABI
>
> Akhil, please work with Ray to understand what needs to be checked
> in general for ABI. We need you to maintain ABI in crypto area.
> First step: run the new ABI checking tool.
>
>
>>> This patch adds versioned function rte_cryptodev_info_get.
>>> Node 20.0.2 function works the same way it was working before.
>>> Node 20.0 function strips capability added in 20.05 release
>>> to prevent some issues with ABI policy. To do that new capability
>>> array is allocated per device and returned to user instead of the
>>> original array passed by PMD.
>>> Because rte_cryptodev_info_get is called by other API functions,
>>> rte_cryptodev_sym_capability_get function was versioned the same way.
>>>
>>> Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
>>> ---
>>>
>>> This patch depends on following patches:
>>>
>>> [1] - "[v3] cryptodev: add chacha20-poly1305 aead algorithm"
>
>
>
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCHv3] Remove validate-abi.sh from tree
@ 2020-04-16 14:54 38% Neil Horman
2020-04-17 10:11 9% ` Ray Kinsella
0 siblings, 1 reply; 200+ results
From: Neil Horman @ 2020-04-16 14:54 UTC (permalink / raw)
To: dev; +Cc: Neil Horman, thomas, david.marchand, mdr
Since we've moved away from our initial validate-abi.sh script, in
favor of check-abi.sh, which uses libabigail, remove the old script from
the tree, and update the docs accordingly
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
CC: thomas@monjalon.net
CC: david.marchand@redhat.com
CC: mdr@ashroe.eu
---
MAINTAINERS | 1 -
devtools/validate-abi.sh | 251 ---------------------
doc/guides/contributing/abi_versioning.rst | 50 ++--
3 files changed, 18 insertions(+), 284 deletions(-)
delete mode 100755 devtools/validate-abi.sh
diff --git a/MAINTAINERS b/MAINTAINERS
index 4800f6884..84b633431 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -152,7 +152,6 @@ F: devtools/gen-abi.sh
F: devtools/libabigail.abignore
F: devtools/update-abi.sh
F: devtools/update_version_map_abi.py
-F: devtools/validate-abi.sh
F: buildtools/check-experimental-syms.sh
F: buildtools/map-list-symbol.sh
diff --git a/devtools/validate-abi.sh b/devtools/validate-abi.sh
deleted file mode 100755
index f64e19d38..000000000
--- a/devtools/validate-abi.sh
+++ /dev/null
@@ -1,251 +0,0 @@
-#!/usr/bin/env bash
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2015 Neil Horman. All rights reserved.
-# Copyright(c) 2017 6WIND S.A.
-# All rights reserved
-
-set -e
-
-abicheck=abi-compliance-checker
-abidump=abi-dumper
-default_dst=abi-check
-default_target=x86_64-native-linuxapp-gcc
-
-# trap on error
-err_report() {
- echo "$0: error at line $1"
-}
-trap 'err_report $LINENO' ERR
-
-print_usage () {
- cat <<- END_OF_HELP
- $(basename $0) [options] <rev1> <rev2>
-
- This script compares the ABI of 2 git revisions of the current
- workspace. The output is a html report and a compilation log.
-
- The objective is to make sure that applications built against
- DSOs from the first revision can still run when executed using
- the DSOs built from the second revision.
-
- <rev1> and <rev2> are git commit id or tags.
-
- Options:
- -h show this help
- -j <num> enable parallel compilation with <num> threads
- -v show compilation logs on the console
- -d <dir> change working directory (default is ${default_dst})
- -t <target> the dpdk target to use (default is ${default_target})
- -f overwrite existing files in destination directory
-
- The script returns 0 on success, or the value of last failing
- call of ${abicheck} (incompatible abi or the tool has run with errors).
- The errors returned by ${abidump} are ignored.
-
- END_OF_HELP
-}
-
-# log in the file, and on stdout if verbose
-# $1: level string
-# $2: string to be logged
-log() {
- echo "$1: $2"
- if [ "${verbose}" != "true" ]; then
- echo "$1: $2" >&3
- fi
-}
-
-# launch a command and log it, taking care of surrounding spaces with quotes
-cmd() {
- local i s whitespace ret
- s=""
- whitespace="[[:space:]]"
- for i in "$@"; do
- if [[ $i =~ $whitespace ]]; then
- i=\"$i\"
- fi
- if [ -z "$s" ]; then
- s="$i"
- else
- s="$s $i"
- fi
- done
-
- ret=0
- log "CMD" "$s"
- "$@" || ret=$?
- if [ "$ret" != "0" ]; then
- log "CMD" "previous command returned $ret"
- fi
-
- return $ret
-}
-
-# redirect or copy stderr/stdout to a file
-# the syntax is unfamiliar, but it makes the rest of the
-# code easier to read, avoiding the use of pipes
-set_log_file() {
- # save original stdout and stderr in fd 3 and 4
- exec 3>&1
- exec 4>&2
- # create a new fd 5 that send to a file
- exec 5> >(cat > $1)
- # send stdout and stderr to fd 5
- if [ "${verbose}" = "true" ]; then
- exec 1> >(tee /dev/fd/5 >&3)
- exec 2> >(tee /dev/fd/5 >&4)
- else
- exec 1>&5
- exec 2>&5
- fi
-}
-
-# Make sure we configure SHARED libraries
-# Also turn off IGB and KNI as those require kernel headers to build
-fixup_config() {
- local conf=config/defconfig_$target
- cmd sed -i -e"$ a\CONFIG_RTE_BUILD_SHARED_LIB=y" $conf
- cmd sed -i -e"$ a\CONFIG_RTE_NEXT_ABI=n" $conf
- cmd sed -i -e"$ a\CONFIG_RTE_EAL_IGB_UIO=n" $conf
- cmd sed -i -e"$ a\CONFIG_RTE_LIBRTE_KNI=n" $conf
- cmd sed -i -e"$ a\CONFIG_RTE_KNI_KMOD=n" $conf
-}
-
-# build dpdk for the given tag and dump abi
-# $1: hash of the revision
-gen_abi() {
- local i
-
- cmd git clone ${dpdkroot} ${dst}/${1}
- cmd cd ${dst}/${1}
-
- log "INFO" "Checking out version ${1} of the dpdk"
- # Move to the old version of the tree
- cmd git checkout ${1}
-
- fixup_config
-
- # Now configure the build
- log "INFO" "Configuring DPDK ${1}"
- cmd make config T=$target O=$target
-
- # Checking abi compliance relies on using the dwarf information in
- # the shared objects. Build with -g to include them.
- log "INFO" "Building DPDK ${1}. This might take a moment"
- cmd make -j$parallel O=$target V=1 EXTRA_CFLAGS="-g -Og -Wno-error" \
- EXTRA_LDFLAGS="-g" || log "INFO" "The build failed"
-
- # Move to the lib directory
- cmd cd ${PWD}/$target/lib
- log "INFO" "Collecting ABI information for ${1}"
- for i in *.so; do
- [ -e "$i" ] || break
- cmd $abidump ${i} -o $dst/${1}/${i}.dump -lver ${1} || true
- # hack to ignore empty SymbolsInfo section (no public ABI)
- if grep -q "'SymbolInfo' => {}," $dst/${1}/${i}.dump \
- 2> /dev/null; then
- log "INFO" "${i} has no public ABI, remove dump file"
- cmd rm -f $dst/${1}/${i}.dump
- fi
- done
-}
-
-verbose=false
-parallel=1
-dst=${default_dst}
-target=${default_target}
-force=0
-while getopts j:vd:t:fh ARG ; do
- case $ARG in
- j ) parallel=$OPTARG ;;
- v ) verbose=true ;;
- d ) dst=$OPTARG ;;
- t ) target=$OPTARG ;;
- f ) force=1 ;;
- h ) print_usage ; exit 0 ;;
- ? ) print_usage ; exit 1 ;;
- esac
-done
-shift $(($OPTIND - 1))
-
-if [ $# != 2 ]; then
- print_usage
- exit 1
-fi
-
-tag1=$1
-tag2=$2
-
-# convert path to absolute
-case "${dst}" in
- /*) ;;
- *) dst=${PWD}/${dst} ;;
-esac
-dpdkroot=$(readlink -f $(dirname $0)/..)
-
-if [ -e "${dst}" -a "$force" = 0 ]; then
- echo "The ${dst} directory is not empty. Remove it, use another"
- echo "one (-d <dir>), or force overriding (-f)"
- exit 1
-fi
-
-rm -rf ${dst}
-mkdir -p ${dst}
-set_log_file ${dst}/abi-check.log
-log "INFO" "Logs available in ${dst}/abi-check.log"
-
-command -v ${abicheck} || log "INFO" "Can't find ${abicheck} utility"
-command -v ${abidump} || log "INFO" "Can't find ${abidump} utility"
-
-hash1=$(git show -s --format=%h "$tag1" -- 2> /dev/null | tail -1)
-hash2=$(git show -s --format=%h "$tag2" -- 2> /dev/null | tail -1)
-
-# Make hashes available in output for non-local reference
-tag1="$tag1 ($hash1)"
-tag2="$tag2 ($hash2)"
-
-if [ "$hash1" = "$hash2" ]; then
- log "ERROR" "$tag1 and $tag2 are the same revisions"
- exit 1
-fi
-
-cmd mkdir -p ${dst}
-
-# dump abi for each revision
-gen_abi ${hash1}
-gen_abi ${hash2}
-
-# compare the abi dumps
-cmd cd ${dst}
-ret=0
-list=""
-for i in ${hash2}/*.dump; do
- name=`basename $i`
- libname=${name%.dump}
-
- if [ ! -f ${hash1}/$name ]; then
- log "INFO" "$NAME does not exist in $tag1. skipping..."
- continue
- fi
-
- local_ret=0
- cmd $abicheck -l $libname \
- -old ${hash1}/$name -new ${hash2}/$name || local_ret=$?
- if [ $local_ret != 0 ]; then
- log "NOTICE" "$abicheck returned $local_ret"
- ret=$local_ret
- list="$list $libname"
- fi
-done
-
-if [ $ret != 0 ]; then
- log "NOTICE" "ABI may be incompatible, check reports/logs for details."
- log "NOTICE" "Incompatible list: $list"
-else
- log "NOTICE" "No error detected, ABI is compatible."
-fi
-
-log "INFO" "Logs are in ${dst}/abi-check.log"
-log "INFO" "HTML reports are in ${dst}/compat_reports directory"
-
-exit $ret
diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst
index a21f4e7a4..219823923 100644
--- a/doc/guides/contributing/abi_versioning.rst
+++ b/doc/guides/contributing/abi_versioning.rst
@@ -482,41 +482,27 @@ Running the ABI Validator
-------------------------
The ``devtools`` directory in the DPDK source tree contains a utility program,
-``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI
-Compliance Checker
-<http://ispras.linuxbase.org/index.php/ABI_compliance_checker>`_.
+``check-abi.sh``, for validating the DPDK ABI based on the libabigail `abidiff
+utility: <https://sourceware.org/libabigail/manual/abidiff.html>`_
-This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper``
-utilities which can be installed via a package manager. For example::
+The syntax of the ``check-abi.sh`` utility is::
- sudo yum install abi-compliance-checker
- sudo yum install abi-dumper
+ ./devtools/check-abi.sh <refdir> <newdir>
-The syntax of the ``validate-abi.sh`` utility is::
+Where <refdir> specifies the directory housing the reference build of dpdk, and
+<newdir> specifies the dpdk build directory to check the abi of
- ./devtools/validate-abi.sh <REV1> <REV2>
+Example:
+To compare your build branch to the ABI of the master branch
+after you have built your branch
-Where ``REV1`` and ``REV2`` are valid gitrevisions(7)
-https://www.kernel.org/pub/software/scm/git/docs/gitrevisions.html
-on the local repo.
+#. $ cd <path to dpdk src tree>
+#. $ mkdir ~/ref
+#. $ git clone --local --no-hardlinks --single-branch -b master . ~/ref
+#. $ cd ~/ref
+#. $ cp <path to dpdk src tree from above>/.config ./.config
+#. $ make defconfig
+#. $ make
+#. $ <path to dpdk src tree>/devtools/check-abi.sh ~/ref <path to dpdk src tree>
-For example::
-
- # Check between the previous and latest commit:
- ./devtools/validate-abi.sh HEAD~1 HEAD
-
- # Check on a specific compilation target:
- ./devtools/validate-abi.sh -t x86_64-native-linux-gcc HEAD~1 HEAD
-
- # Check between two tags:
- ./devtools/validate-abi.sh v2.0.0 v2.1.0
-
- # Check between git master and local topic-branch "vhost-hacking":
- ./devtools/validate-abi.sh master vhost-hacking
-
-After the validation script completes (it can take a while since it need to
-compile both tags) it will create compatibility reports in the
-``./abi-check/compat_report`` directory. Listed incompatibilities can be found
-as follows::
-
- grep -lr Incompatible abi-check/compat_reports/
+
--
2.25.2
^ permalink raw reply [relevance 38%]
* Re: [dpdk-dev] [PATCH v1] common/mlx5: remove devx depndency on ibv and dv
2020-04-09 7:24 4% ` David Marchand
@ 2020-04-16 17:35 3% ` Ferruh Yigit
2020-04-16 20:00 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Ferruh Yigit @ 2020-04-16 17:35 UTC (permalink / raw)
To: Matan Azrad, Raslan Darawsheh, Ophir Munk
Cc: David Marchand, Thomas Monjalon, dev, Olga Shern, Asaf Penso,
Kinsella, Ray, Neil Horman, Kevin Laatz
On 4/9/2020 8:24 AM, David Marchand wrote:
> Hello,
>
> On Wed, Apr 8, 2020 at 7:12 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>>
>> On 4/1/2020 10:59 AM, Raslan Darawsheh wrote:
>>> Hi,
>>>
>>>> -----Original Message-----
>>>> From: Ophir Munk <ophirmu@mellanox.com>
>>>> Sent: Monday, March 30, 2020 1:32 AM
>>>> To: dev@dpdk.org; Matan Azrad <matan@mellanox.com>
>>>> Cc: Thomas Monjalon <thomas@monjalon.net>; Olga Shern
>>>> <olgas@mellanox.com>; Raslan Darawsheh <rasland@mellanox.com>; Ophir
>>>> Munk <ophirmu@mellanox.com>; Asaf Penso <asafp@mellanox.com>
>>>> Subject: [PATCH v1] common/mlx5: remove devx depndency on ibv and dv
>>>>
>>>> File mlx5_devx_cmds.c should contain pure DevX calls. It must be OS
>>>> agnostic and not include any references to ibv or dv structs (defined in
>>>> ibverbs and rdma-core linux libraries). This commit replaces all ibv and
>>>> dv references with 'void *'. Specifically, the following struct were
>>>> replaced:
>>>> 1. struct ibv_context *
>>>> 2. struct ibv_qp *
>>>> 3. struct mlx5dv_devx_cmd_comp *
>>>>
>>>> Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
>>>
>>> Patch applied to next-net-mlx,
>>>
>>
>> Hi David,
>>
>> This patch is failing in the travis for ABI checks [1], since mlx has APIs now
>> [2], are they public APIs or internal ones, and are they part of the ABI policy,
>> can you please check this?
>
> - What I see on patchwork and test-report ml for this patch:
> http://patchwork.dpdk.org/patch/67367/
>
> Ophir proposed a patch on 03/30.
>
> The robot reported an issue on 03/30, and I suppose Ophir got a report.
> https://mails.dpdk.org/archives/test-report/2020-March/122623.html
> https://travis-ci.com/github/ovsrobot/dpdk/jobs/308057800#L2337
>
> Matan acked the patch on 03/31.
>
> Rasland merged the patch on 04/01.
>
> I understand that the abi checks are not perfect, and people need help
> with the new abi checks.
> Prove me wrong, but here, I get the feeling that it was just ignored
> by 3 people in a row.
>
>
> - On the question if these should be public API or internal, that is
> not for me to reply/investigate.
> This is a question for Mellanox.
>
>
Hi Matan, Raslan, Ophir,
First can you please clarify if these APIs are internal or public?
And later if the ABI break issue is not clarified I may need to drop these
patches. Right now they fail in travis!
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v1] common/mlx5: remove devx depndency on ibv and dv
2020-04-16 17:35 3% ` Ferruh Yigit
@ 2020-04-16 20:00 4% ` Thomas Monjalon
2020-04-17 16:19 0% ` Ferruh Yigit
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-04-16 20:00 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Matan Azrad, Raslan Darawsheh, Ophir Munk, David Marchand, dev,
Olga Shern, Asaf Penso, Kinsella, Ray, Neil Horman, Kevin Laatz,
hemant.agrawal, Haiyue Wang, Sunil Kumar Kori
16/04/2020 19:35, Ferruh Yigit:
> On 4/9/2020 8:24 AM, David Marchand wrote:
> > On Wed, Apr 8, 2020 at 7:12 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
> >> On 4/1/2020 10:59 AM, Raslan Darawsheh wrote:
> >>> From: Ophir Munk <ophirmu@mellanox.com>
> >>>>
> >>>> File mlx5_devx_cmds.c should contain pure DevX calls. It must be OS
> >>>> agnostic and not include any references to ibv or dv structs (defined in
> >>>> ibverbs and rdma-core linux libraries). This commit replaces all ibv and
> >>>> dv references with 'void *'. Specifically, the following struct were
> >>>> replaced:
> >>>> 1. struct ibv_context *
> >>>> 2. struct ibv_qp *
> >>>> 3. struct mlx5dv_devx_cmd_comp *
> >>>>
> >>>> Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
> >>>
> >>> Patch applied to next-net-mlx,
> >>>
> >>
> >> Hi David,
> >>
> >> This patch is failing in the travis for ABI checks [1], since mlx has APIs now
> >> [2], are they public APIs or internal ones, and are they part of the ABI policy,
> >> can you please check this?
> >
> > - What I see on patchwork and test-report ml for this patch:
> > http://patchwork.dpdk.org/patch/67367/
> >
> > Ophir proposed a patch on 03/30.
> >
> > The robot reported an issue on 03/30, and I suppose Ophir got a report.
> > https://mails.dpdk.org/archives/test-report/2020-March/122623.html
> > https://travis-ci.com/github/ovsrobot/dpdk/jobs/308057800#L2337
> >
> > Matan acked the patch on 03/31.
> >
> > Rasland merged the patch on 04/01.
> >
> > I understand that the abi checks are not perfect, and people need help
> > with the new abi checks.
> > Prove me wrong, but here, I get the feeling that it was just ignored
> > by 3 people in a row.
> >
> > - On the question if these should be public API or internal, that is
> > not for me to reply/investigate.
> > This is a question for Mellanox.
> >
>
> Hi Matan, Raslan, Ophir,
>
> First can you please clarify if these APIs are internal or public?
As most of common drivers, some functions are exported to be
used by some PMDs. So they are not part of the API/ABI and should be skipped
by ABI checks.
> And later if the ABI break issue is not clarified I may need to drop these
> patches. Right now they fail in travis!
Yes, it fails and could it be avoided with some libabigail config.
But the real solution is to mark internal symbols, and we are waiting
for rte_internal patchset to be completed and merged.
Ferruh, please let's not bloat libabigail config,
and reject any patch failing ABI checks.
As a consequence, this patch must be dropped until it uses rte_internal.
Thanks
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 01/10] Add __rte_internal tag for functions and version target
@ 2020-04-17 2:04 3% ` Wang, Haiyue
2020-04-17 2:38 4% ` Neil Horman
0 siblings, 1 reply; 200+ results
From: Wang, Haiyue @ 2020-04-17 2:04 UTC (permalink / raw)
To: Neil Horman, dev
Cc: Jerin Jacob Kollanukkaran, Richardson, Bruce, Thomas Monjalon,
David Marchand, Yigit, Ferruh
Hi Neil,
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Neil Horman
> Sent: Thursday, June 13, 2019 22:24
> To: dev@dpdk.org
> Cc: Neil Horman <nhorman@tuxdriver.com>; Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Richardson,
> Bruce <bruce.richardson@intel.com>; Thomas Monjalon <thomas@monjalon.net>
> Subject: [dpdk-dev] [PATCH v2 01/10] Add __rte_internal tag for functions and version target
>
> This tag is meant to be used on function prototypes to identify
> functions that are only meant to be used by internal DPDK libraries
> (i.e. libraries that are built while building the SDK itself, as
> identified by the defining of the BUILDING_RTE_SDK macro). When that
> flag is not set, it will resolve to an error function attribute, causing
> build breakage for any compilation unit attempting to build it
>
> Validate the use of this tag in much the same way we validate
> __rte_experimental. By adding an INTERNAL version to library map files,
> we can exempt internal-only functions from ABI checking, and handle them
> to ensure that symbols we wish to only be for internal use between dpdk
> libraries are properly tagged with __rte_experimental
>
> Note this patch updates the check-experimental-syms.sh script, which
> normally only check the EXPERIMENTAL section to also check the INTERNAL
> section now. As such its been renamed to the now more appropriate
> check-special-syms.sh
>
> Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
> CC: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> CC: Bruce Richardson <bruce.richardson@intel.com>
> CC: Thomas Monjalon <thomas@monjalon.net>
> ---
> ...rimental-syms.sh => check-special-syms.sh} | 24 ++++++++++++++++++-
> lib/librte_eal/common/include/rte_compat.h | 12 ++++++++++
> mk/internal/rte.compile-pre.mk | 6 ++---
> mk/target/generic/rte.vars.mk | 2 +-
> 4 files changed, 39 insertions(+), 5 deletions(-)
> rename buildtools/{check-experimental-syms.sh => check-special-syms.sh} (53%)
>
....
> diff --git a/lib/librte_eal/common/include/rte_compat.h b/lib/librte_eal/common/include/rte_compat.h
> index 92ff28faf..739e8485c 100644
> --- a/lib/librte_eal/common/include/rte_compat.h
> +++ b/lib/librte_eal/common/include/rte_compat.h
> @@ -89,4 +89,16 @@ __attribute__((section(".text.experimental")))
>
> #endif
>
> +/*
> + * __rte_internal tags mark functions as internal only, If specified in public
> + * header files, this tag will resolve to an error directive, preventing
> + * external applications from attempting to make calls to functions not meant
> + * for consumption outside the dpdk library
> + */
> +#ifdef BUILDING_RTE_SDK
> +#define __rte_internal __attribute__((section(".text.internal")))
> +#else
> +#define __rte_internal __attribute__((error("This function cannot be used outside of the core DPDK
> library"), \
> + section(".text.internal")))
> +#endif
> #endif /* _RTE_COMPAT_H_ */
Since struct definition is also a kind of ABI (am I right ? ;-) ), like:
drivers/bus/pci/rte_bus_pci.h
struct rte_pci_device {
...
struct rte_intr_handle vfio_req_intr_handle;
/**< Handler of VFIO request interrupt */
} __rte_internal;
Then will capture the errors anyway by using one of __rte_internal definition.
error: 'section' attribute does not apply to types [-Werror=attributes]
error: 'error' attribute does not apply to types
> 2.20.1
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2 01/10] Add __rte_internal tag for functions and version target
2020-04-17 2:04 3% ` Wang, Haiyue
@ 2020-04-17 2:38 4% ` Neil Horman
2020-04-17 4:40 4% ` Wang, Haiyue
0 siblings, 1 reply; 200+ results
From: Neil Horman @ 2020-04-17 2:38 UTC (permalink / raw)
To: Wang, Haiyue
Cc: dev, Jerin Jacob Kollanukkaran, Richardson, Bruce,
Thomas Monjalon, David Marchand, Yigit, Ferruh
On Fri, Apr 17, 2020 at 02:04:30AM +0000, Wang, Haiyue wrote:
> Hi Neil,
>
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Neil Horman
> > Sent: Thursday, June 13, 2019 22:24
> > To: dev@dpdk.org
> > Cc: Neil Horman <nhorman@tuxdriver.com>; Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Richardson,
> > Bruce <bruce.richardson@intel.com>; Thomas Monjalon <thomas@monjalon.net>
> > Subject: [dpdk-dev] [PATCH v2 01/10] Add __rte_internal tag for functions and version target
> >
> > This tag is meant to be used on function prototypes to identify
> > functions that are only meant to be used by internal DPDK libraries
> > (i.e. libraries that are built while building the SDK itself, as
> > identified by the defining of the BUILDING_RTE_SDK macro). When that
> > flag is not set, it will resolve to an error function attribute, causing
> > build breakage for any compilation unit attempting to build it
> >
> > Validate the use of this tag in much the same way we validate
> > __rte_experimental. By adding an INTERNAL version to library map files,
> > we can exempt internal-only functions from ABI checking, and handle them
> > to ensure that symbols we wish to only be for internal use between dpdk
> > libraries are properly tagged with __rte_experimental
> >
> > Note this patch updates the check-experimental-syms.sh script, which
> > normally only check the EXPERIMENTAL section to also check the INTERNAL
> > section now. As such its been renamed to the now more appropriate
> > check-special-syms.sh
> >
> > Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
> > CC: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> > CC: Bruce Richardson <bruce.richardson@intel.com>
> > CC: Thomas Monjalon <thomas@monjalon.net>
> > ---
> > ...rimental-syms.sh => check-special-syms.sh} | 24 ++++++++++++++++++-
> > lib/librte_eal/common/include/rte_compat.h | 12 ++++++++++
> > mk/internal/rte.compile-pre.mk | 6 ++---
> > mk/target/generic/rte.vars.mk | 2 +-
> > 4 files changed, 39 insertions(+), 5 deletions(-)
> > rename buildtools/{check-experimental-syms.sh => check-special-syms.sh} (53%)
> >
> ....
>
> > diff --git a/lib/librte_eal/common/include/rte_compat.h b/lib/librte_eal/common/include/rte_compat.h
> > index 92ff28faf..739e8485c 100644
> > --- a/lib/librte_eal/common/include/rte_compat.h
> > +++ b/lib/librte_eal/common/include/rte_compat.h
> > @@ -89,4 +89,16 @@ __attribute__((section(".text.experimental")))
> >
> > #endif
> >
> > +/*
> > + * __rte_internal tags mark functions as internal only, If specified in public
> > + * header files, this tag will resolve to an error directive, preventing
> > + * external applications from attempting to make calls to functions not meant
> > + * for consumption outside the dpdk library
> > + */
> > +#ifdef BUILDING_RTE_SDK
> > +#define __rte_internal __attribute__((section(".text.internal")))
> > +#else
> > +#define __rte_internal __attribute__((error("This function cannot be used outside of the core DPDK
> > library"), \
> > + section(".text.internal")))
> > +#endif
> > #endif /* _RTE_COMPAT_H_ */
>
> Since struct definition is also a kind of ABI (am I right ? ;-) ), like:
>
Yes, thats correct, which is why I've advocated for making structs
opaque as part of the abi, but I suppose thats not where we are. :)
> drivers/bus/pci/rte_bus_pci.h
> struct rte_pci_device {
> ...
> struct rte_intr_handle vfio_req_intr_handle;
> /**< Handler of VFIO request interrupt */
> } __rte_internal;
>
> Then will capture the errors anyway by using one of __rte_internal definition.
> error: 'section' attribute does not apply to types [-Werror=attributes]
> error: 'error' attribute does not apply to types
>
As it is currently written, the __rte_internal macro is only written to
work on functions. If you don't want a struct to be part of the ABI, we
would need to either:
a) make a simmilar macro (say __rte_internal_data) which uses a simmilar
gcc attibute to catch external usage.
or
b) just move the strucute definition to a location that isn't exposed as
part of the external ABI
Neil
> > 2.20.1
>
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v2 01/10] Add __rte_internal tag for functions and version target
2020-04-17 2:38 4% ` Neil Horman
@ 2020-04-17 4:40 4% ` Wang, Haiyue
0 siblings, 0 replies; 200+ results
From: Wang, Haiyue @ 2020-04-17 4:40 UTC (permalink / raw)
To: Neil Horman
Cc: dev, Jerin Jacob Kollanukkaran, Richardson, Bruce,
Thomas Monjalon, David Marchand, Yigit, Ferruh
> -----Original Message-----
> From: Neil Horman <nhorman@tuxdriver.com>
> Sent: Friday, April 17, 2020 10:38
> To: Wang, Haiyue <haiyue.wang@intel.com>
> Cc: dev@dpdk.org; Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Thomas Monjalon <thomas@monjalon.net>; David Marchand
> <david.marchand@redhat.com>; Yigit, Ferruh <ferruh.yigit@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v2 01/10] Add __rte_internal tag for functions and version target
>
> On Fri, Apr 17, 2020 at 02:04:30AM +0000, Wang, Haiyue wrote:
> > Hi Neil,
> >
> > > -----Original Message-----
> > > From: dev <dev-bounces@dpdk.org> On Behalf Of Neil Horman
> > > Sent: Thursday, June 13, 2019 22:24
> > > To: dev@dpdk.org
> > > Cc: Neil Horman <nhorman@tuxdriver.com>; Jerin Jacob Kollanukkaran <jerinj@marvell.com>;
> Richardson,
> > > Bruce <bruce.richardson@intel.com>; Thomas Monjalon <thomas@monjalon.net>
> > > Subject: [dpdk-dev] [PATCH v2 01/10] Add __rte_internal tag for functions and version target
> > >
> > > This tag is meant to be used on function prototypes to identify
> > > functions that are only meant to be used by internal DPDK libraries
> > > (i.e. libraries that are built while building the SDK itself, as
> > > identified by the defining of the BUILDING_RTE_SDK macro). When that
> > > flag is not set, it will resolve to an error function attribute, causing
> > > build breakage for any compilation unit attempting to build it
> > >
> > > Validate the use of this tag in much the same way we validate
> > > __rte_experimental. By adding an INTERNAL version to library map files,
> > > we can exempt internal-only functions from ABI checking, and handle them
> > > to ensure that symbols we wish to only be for internal use between dpdk
> > > libraries are properly tagged with __rte_experimental
> > >
> > > Note this patch updates the check-experimental-syms.sh script, which
> > > normally only check the EXPERIMENTAL section to also check the INTERNAL
> > > section now. As such its been renamed to the now more appropriate
> > > check-special-syms.sh
> > >
> > > Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
> > > CC: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> > > CC: Bruce Richardson <bruce.richardson@intel.com>
> > > CC: Thomas Monjalon <thomas@monjalon.net>
> > > ---
> > > ...rimental-syms.sh => check-special-syms.sh} | 24 ++++++++++++++++++-
> > > lib/librte_eal/common/include/rte_compat.h | 12 ++++++++++
> > > mk/internal/rte.compile-pre.mk | 6 ++---
> > > mk/target/generic/rte.vars.mk | 2 +-
> > > 4 files changed, 39 insertions(+), 5 deletions(-)
> > > rename buildtools/{check-experimental-syms.sh => check-special-syms.sh} (53%)
> > >
> > ....
> >
> > > diff --git a/lib/librte_eal/common/include/rte_compat.h
> b/lib/librte_eal/common/include/rte_compat.h
> > > index 92ff28faf..739e8485c 100644
> > > --- a/lib/librte_eal/common/include/rte_compat.h
> > > +++ b/lib/librte_eal/common/include/rte_compat.h
> > > @@ -89,4 +89,16 @@ __attribute__((section(".text.experimental")))
> > >
> > > #endif
> > >
> > > +/*
> > > + * __rte_internal tags mark functions as internal only, If specified in public
> > > + * header files, this tag will resolve to an error directive, preventing
> > > + * external applications from attempting to make calls to functions not meant
> > > + * for consumption outside the dpdk library
> > > + */
> > > +#ifdef BUILDING_RTE_SDK
> > > +#define __rte_internal __attribute__((section(".text.internal")))
> > > +#else
> > > +#define __rte_internal __attribute__((error("This function cannot be used outside of the core
> DPDK
> > > library"), \
> > > + section(".text.internal")))
> > > +#endif
> > > #endif /* _RTE_COMPAT_H_ */
> >
> > Since struct definition is also a kind of ABI (am I right ? ;-) ), like:
> >
> Yes, thats correct, which is why I've advocated for making structs
> opaque as part of the abi, but I suppose thats not where we are. :)
>
Make sense, normally structs can't live alone without function API. A little
paranoid for type only ABI checking. ;-) And this definition is good for ABI
checking as we did it for EXPERIMENTAL.
Thanks!
Haiyue
> > drivers/bus/pci/rte_bus_pci.h
> > struct rte_pci_device {
> > ...
> > struct rte_intr_handle vfio_req_intr_handle;
> > /**< Handler of VFIO request interrupt */
> > } __rte_internal;
> >
> > Then will capture the errors anyway by using one of __rte_internal definition.
> > error: 'section' attribute does not apply to types [-Werror=attributes]
> > error: 'error' attribute does not apply to types
> >
> As it is currently written, the __rte_internal macro is only written to
> work on functions. If you don't want a struct to be part of the ABI, we
> would need to either:
>
> a) make a simmilar macro (say __rte_internal_data) which uses a simmilar
> gcc attibute to catch external usage.
>
> or
>
> b) just move the strucute definition to a location that isn't exposed as
> part of the external ABI
>
> Neil
>
> > > 2.20.1
> >
> >
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
2020-04-16 10:01 3% ` Thomas Monjalon
@ 2020-04-17 7:24 5% ` Ray Kinsella
2020-04-17 9:31 4% ` Bruce Richardson
0 siblings, 1 reply; 200+ results
From: Ray Kinsella @ 2020-04-17 7:24 UTC (permalink / raw)
To: Thomas Monjalon, Trahe, Fiona, Bruce Richardson
Cc: dev, Kusztal, ArkadiuszX, Neil Horman, Luca Boccassi,
Kevin Traynor, Yigit, Ferruh
On 16/04/2020 11:01, Thomas Monjalon wrote:
> 16/04/2020 11:51, Bruce Richardson:
>> On Wed, Apr 15, 2020 at 06:24:19PM +0100, Trahe, Fiona wrote:
>>> 5a. If in 20.05 we add a version of a fn which breaks ABI 20.0, what should the name of the original function be? fn_v20, or fn_v20.0
>>
>> In technical terms it really doesn't matter, it's just a name that will be
>> looked up in a table. I don't think we strictly enforce the naming, so
>> whatever is clearest is best. I'd suggest the former.
>
> Each release can have a new ABI.
How many ABI's do we want to support?
> The same function can have a different version in 20.02, 20.05 and 20.08.
> If you name it fn_v20 in 20.05, what will be the name for the new version
> in 20.08? I suggest using the release number when versioning a function.
>
ok - so this is exactly _not_ what we wanted at the outset.
We committed to support a single ABI, that is the 19.11 (v20) ABI until the 20.11 (v21) ABI.
We do _not_ support ABI stability in quarterly releases, as the support overhead would balloon overtime.
Really why would we do this - who would benefit?
Do we envisage a situation that someone who built against the say 20.02 shared libraries.
would expect their binaries to port to 20.05 without a rebuild?
It would also defeat the purpose of EXPERIMENTAL, if the community where willing to support any permutation of an API.
Why would ever bother to use experimental?
I have a bit more checking to do, but IMHO the following we should fix the following commits such that APIs are either EXPERIMENTAL or staged for v21.
root@ashroe.eu:/build/dpdk# find . -name *.map | xargs grep 20.0.1
./lib/librte_meter/rte_meter_version.map:DPDK_20.0.1 {
./drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map:DPDK_20.0.1 {
./drivers/net/ionic/rte_pmd_ionic_version.map:DPDK_20.0.1 {
./drivers/common/octeontx2/rte_common_octeontx2_version.map:DPDK_20.0.1 {
./drivers/common/mlx5/rte_common_mlx5_version.map:DPDK_20.0.1 {
./drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map:DPDK_20.0.1 {
Thanks,
Ray K
^ permalink raw reply [relevance 5%]
* Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
2020-04-17 7:24 5% ` Ray Kinsella
@ 2020-04-17 9:31 4% ` Bruce Richardson
2020-04-17 9:42 3% ` Ray Kinsella
0 siblings, 1 reply; 200+ results
From: Bruce Richardson @ 2020-04-17 9:31 UTC (permalink / raw)
To: Ray Kinsella
Cc: Thomas Monjalon, Trahe, Fiona, dev, Kusztal, ArkadiuszX,
Neil Horman, Luca Boccassi, Kevin Traynor, Yigit, Ferruh
On Fri, Apr 17, 2020 at 08:24:30AM +0100, Ray Kinsella wrote:
>
>
> On 16/04/2020 11:01, Thomas Monjalon wrote:
> > 16/04/2020 11:51, Bruce Richardson:
> >> On Wed, Apr 15, 2020 at 06:24:19PM +0100, Trahe, Fiona wrote:
> >>> 5a. If in 20.05 we add a version of a fn which breaks ABI 20.0, what should the name of the original function be? fn_v20, or fn_v20.0
> >>
> >> In technical terms it really doesn't matter, it's just a name that will be
> >> looked up in a table. I don't think we strictly enforce the naming, so
> >> whatever is clearest is best. I'd suggest the former.
> >
> > Each release can have a new ABI.
>
> How many ABI's do we want to support?
>
It's not how many we want to support, but for me it's a matter of how many
do we need to support. If an API is part of the stable set, it can't just
drop to being experimental for one or two releases - it's always stable
until deprecated. We also shouldn't have a situation where release 20.08 is
ABI compatible with 19.11 but not 20.02 and 20.05.
> > The same function can have a different version in 20.02, 20.05 and 20.08.
> > If you name it fn_v20 in 20.05, what will be the name for the new version
> > in 20.08? I suggest using the release number when versioning a function.
> >
>
> ok - so this is exactly _not_ what we wanted at the outset.
>
> We committed to support a single ABI, that is the 19.11 (v20) ABI until the 20.11 (v21) ABI.
> We do _not_ support ABI stability in quarterly releases, as the support overhead would balloon overtime.
>
> Really why would we do this - who would benefit?
>
> Do we envisage a situation that someone who built against the say 20.02 shared libraries.
> would expect their binaries to port to 20.05 without a rebuild?
>
I would have expected that, yes, as all have the v20 ABI.
Maybe I need to change my expectations, though.
/Bruce
> It would also defeat the purpose of EXPERIMENTAL, if the community where willing to support any permutation of an API.
> Why would ever bother to use experimental?
>
> I have a bit more checking to do, but IMHO the following we should fix the following commits such that APIs are either EXPERIMENTAL or staged for v21.
>
> root@ashroe.eu:/build/dpdk# find . -name *.map | xargs grep 20.0.1
> ./lib/librte_meter/rte_meter_version.map:DPDK_20.0.1 {
> ./drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map:DPDK_20.0.1 {
> ./drivers/net/ionic/rte_pmd_ionic_version.map:DPDK_20.0.1 {
> ./drivers/common/octeontx2/rte_common_octeontx2_version.map:DPDK_20.0.1 {
> ./drivers/common/mlx5/rte_common_mlx5_version.map:DPDK_20.0.1 {
> ./drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map:DPDK_20.0.1 {
>
> Thanks,
>
> Ray K
>
>
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
2020-04-17 9:31 4% ` Bruce Richardson
@ 2020-04-17 9:42 3% ` Ray Kinsella
2020-04-17 10:17 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Ray Kinsella @ 2020-04-17 9:42 UTC (permalink / raw)
To: Bruce Richardson
Cc: Thomas Monjalon, Trahe, Fiona, dev, Kusztal, ArkadiuszX,
Neil Horman, Luca Boccassi, Kevin Traynor, Yigit, Ferruh
On 17/04/2020 10:31, Bruce Richardson wrote:
> On Fri, Apr 17, 2020 at 08:24:30AM +0100, Ray Kinsella wrote:
>>
>>
>> On 16/04/2020 11:01, Thomas Monjalon wrote:
>>> 16/04/2020 11:51, Bruce Richardson:
>>>> On Wed, Apr 15, 2020 at 06:24:19PM +0100, Trahe, Fiona wrote:
>>>>> 5a. If in 20.05 we add a version of a fn which breaks ABI 20.0, what should the name of the original function be? fn_v20, or fn_v20.0
>>>>
>>>> In technical terms it really doesn't matter, it's just a name that will be
>>>> looked up in a table. I don't think we strictly enforce the naming, so
>>>> whatever is clearest is best. I'd suggest the former.
>>>
>>> Each release can have a new ABI.
>>
>> How many ABI's do we want to support?
>>
> It's not how many we want to support, but for me it's a matter of how many
> do we need to support. If an API is part of the stable set, it can't just
> drop to being experimental for one or two releases - it's always stable
> until deprecated. We also shouldn't have a situation where release 20.08 is
> ABI compatible with 19.11 but not 20.02 and 20.05.
True. Let me say it differently.
Our only commitment is to support v20 - 19.11
However you are correct, if something gets committed as v21 in 20.02, in practise should also be there in 20.05+ also.
Because if it is committed as v21 and not as experimental, it should not be changing once committed.
In answering Thomas,
I was more commenting on the proliferation of ABI numbers & symbols we need to track in the build.
With v20, v21 & Experimental we need to keep track of 3.
If we start allowing quarterly builds to have managed ABI's, it will get confusing.
>
>>> The same function can have a different version in 20.02, 20.05 and 20.08.
>>> If you name it fn_v20 in 20.05, what will be the name for the new version
>>> in 20.08? I suggest using the release number when versioning a function.
>>>
>>
>> ok - so this is exactly _not_ what we wanted at the outset.
>>
>> We committed to support a single ABI, that is the 19.11 (v20) ABI until the 20.11 (v21) ABI.
>> We do _not_ support ABI stability in quarterly releases, as the support overhead would balloon overtime.
>>
>> Really why would we do this - who would benefit?
>>
>> Do we envisage a situation that someone who built against the say 20.02 shared libraries.
>> would expect their binaries to port to 20.05 without a rebuild?
>>
> I would have expected that, yes, as all have the v20 ABI.
> Maybe I need to change my expectations, though.
You are correct.
However I guess, I would still see them as slightly different levels of commitment.
>
> /Bruce
>
>> It would also defeat the purpose of EXPERIMENTAL, if the community where willing to support any permutation of an API.
>> Why would ever bother to use experimental?
>>
>> I have a bit more checking to do, but IMHO the following we should fix the following commits such that APIs are either EXPERIMENTAL or staged for v21.
>>
>> root@ashroe.eu:/build/dpdk# find . -name *.map | xargs grep 20.0.1
>> ./lib/librte_meter/rte_meter_version.map:DPDK_20.0.1 {
>> ./drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map:DPDK_20.0.1 {
>> ./drivers/net/ionic/rte_pmd_ionic_version.map:DPDK_20.0.1 {
>> ./drivers/common/octeontx2/rte_common_octeontx2_version.map:DPDK_20.0.1 {
>> ./drivers/common/mlx5/rte_common_mlx5_version.map:DPDK_20.0.1 {
>> ./drivers/raw/octeontx2_ep/rte_rawdev_octeontx2_ep_version.map:DPDK_20.0.1 {
>>
>> Thanks,
>>
>> Ray K
>>
>>
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCHv3] Remove validate-abi.sh from tree
2020-04-16 14:54 38% [dpdk-dev] [PATCHv3] Remove validate-abi.sh from tree Neil Horman
@ 2020-04-17 10:11 9% ` Ray Kinsella
2020-04-17 10:20 8% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Ray Kinsella @ 2020-04-17 10:11 UTC (permalink / raw)
To: Neil Horman, dev; +Cc: thomas, david.marchand
On 16/04/2020 15:54, Neil Horman wrote:
> Since we've moved away from our initial validate-abi.sh script, in
> favor of check-abi.sh, which uses libabigail, remove the old script from
> the tree, and update the docs accordingly
>
> Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
> CC: thomas@monjalon.net
> CC: david.marchand@redhat.com
> CC: mdr@ashroe.eu
> ---
> MAINTAINERS | 1 -
> devtools/validate-abi.sh | 251 ---------------------
> doc/guides/contributing/abi_versioning.rst | 50 ++--
> 3 files changed, 18 insertions(+), 284 deletions(-)
> delete mode 100755 devtools/validate-abi.sh
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 4800f6884..84b633431 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -152,7 +152,6 @@ F: devtools/gen-abi.sh
> F: devtools/libabigail.abignore
> F: devtools/update-abi.sh
> F: devtools/update_version_map_abi.py
> -F: devtools/validate-abi.sh
> F: buildtools/check-experimental-syms.sh
> F: buildtools/map-list-symbol.sh
>
> diff --git a/devtools/validate-abi.sh b/devtools/validate-abi.sh
> deleted file mode 100755
> index f64e19d38..000000000
> --- a/devtools/validate-abi.sh
> +++ /dev/null
> @@ -1,251 +0,0 @@
> -#!/usr/bin/env bash
> -# SPDX-License-Identifier: BSD-3-Clause
> -# Copyright(c) 2015 Neil Horman. All rights reserved.
> -# Copyright(c) 2017 6WIND S.A.
> -# All rights reserved
> -
> -set -e
> -
> -abicheck=abi-compliance-checker
> -abidump=abi-dumper
> -default_dst=abi-check
> -default_target=x86_64-native-linuxapp-gcc
> -
> -# trap on error
> -err_report() {
> - echo "$0: error at line $1"
> -}
> -trap 'err_report $LINENO' ERR
> -
> -print_usage () {
> - cat <<- END_OF_HELP
> - $(basename $0) [options] <rev1> <rev2>
> -
> - This script compares the ABI of 2 git revisions of the current
> - workspace. The output is a html report and a compilation log.
> -
> - The objective is to make sure that applications built against
> - DSOs from the first revision can still run when executed using
> - the DSOs built from the second revision.
> -
> - <rev1> and <rev2> are git commit id or tags.
> -
> - Options:
> - -h show this help
> - -j <num> enable parallel compilation with <num> threads
> - -v show compilation logs on the console
> - -d <dir> change working directory (default is ${default_dst})
> - -t <target> the dpdk target to use (default is ${default_target})
> - -f overwrite existing files in destination directory
> -
> - The script returns 0 on success, or the value of last failing
> - call of ${abicheck} (incompatible abi or the tool has run with errors).
> - The errors returned by ${abidump} are ignored.
> -
> - END_OF_HELP
> -}
> -
> -# log in the file, and on stdout if verbose
> -# $1: level string
> -# $2: string to be logged
> -log() {
> - echo "$1: $2"
> - if [ "${verbose}" != "true" ]; then
> - echo "$1: $2" >&3
> - fi
> -}
> -
> -# launch a command and log it, taking care of surrounding spaces with quotes
> -cmd() {
> - local i s whitespace ret
> - s=""
> - whitespace="[[:space:]]"
> - for i in "$@"; do
> - if [[ $i =~ $whitespace ]]; then
> - i=\"$i\"
> - fi
> - if [ -z "$s" ]; then
> - s="$i"
> - else
> - s="$s $i"
> - fi
> - done
> -
> - ret=0
> - log "CMD" "$s"
> - "$@" || ret=$?
> - if [ "$ret" != "0" ]; then
> - log "CMD" "previous command returned $ret"
> - fi
> -
> - return $ret
> -}
> -
> -# redirect or copy stderr/stdout to a file
> -# the syntax is unfamiliar, but it makes the rest of the
> -# code easier to read, avoiding the use of pipes
> -set_log_file() {
> - # save original stdout and stderr in fd 3 and 4
> - exec 3>&1
> - exec 4>&2
> - # create a new fd 5 that send to a file
> - exec 5> >(cat > $1)
> - # send stdout and stderr to fd 5
> - if [ "${verbose}" = "true" ]; then
> - exec 1> >(tee /dev/fd/5 >&3)
> - exec 2> >(tee /dev/fd/5 >&4)
> - else
> - exec 1>&5
> - exec 2>&5
> - fi
> -}
> -
> -# Make sure we configure SHARED libraries
> -# Also turn off IGB and KNI as those require kernel headers to build
> -fixup_config() {
> - local conf=config/defconfig_$target
> - cmd sed -i -e"$ a\CONFIG_RTE_BUILD_SHARED_LIB=y" $conf
> - cmd sed -i -e"$ a\CONFIG_RTE_NEXT_ABI=n" $conf
> - cmd sed -i -e"$ a\CONFIG_RTE_EAL_IGB_UIO=n" $conf
> - cmd sed -i -e"$ a\CONFIG_RTE_LIBRTE_KNI=n" $conf
> - cmd sed -i -e"$ a\CONFIG_RTE_KNI_KMOD=n" $conf
> -}
> -
> -# build dpdk for the given tag and dump abi
> -# $1: hash of the revision
> -gen_abi() {
> - local i
> -
> - cmd git clone ${dpdkroot} ${dst}/${1}
> - cmd cd ${dst}/${1}
> -
> - log "INFO" "Checking out version ${1} of the dpdk"
> - # Move to the old version of the tree
> - cmd git checkout ${1}
> -
> - fixup_config
> -
> - # Now configure the build
> - log "INFO" "Configuring DPDK ${1}"
> - cmd make config T=$target O=$target
> -
> - # Checking abi compliance relies on using the dwarf information in
> - # the shared objects. Build with -g to include them.
> - log "INFO" "Building DPDK ${1}. This might take a moment"
> - cmd make -j$parallel O=$target V=1 EXTRA_CFLAGS="-g -Og -Wno-error" \
> - EXTRA_LDFLAGS="-g" || log "INFO" "The build failed"
> -
> - # Move to the lib directory
> - cmd cd ${PWD}/$target/lib
> - log "INFO" "Collecting ABI information for ${1}"
> - for i in *.so; do
> - [ -e "$i" ] || break
> - cmd $abidump ${i} -o $dst/${1}/${i}.dump -lver ${1} || true
> - # hack to ignore empty SymbolsInfo section (no public ABI)
> - if grep -q "'SymbolInfo' => {}," $dst/${1}/${i}.dump \
> - 2> /dev/null; then
> - log "INFO" "${i} has no public ABI, remove dump file"
> - cmd rm -f $dst/${1}/${i}.dump
> - fi
> - done
> -}
> -
> -verbose=false
> -parallel=1
> -dst=${default_dst}
> -target=${default_target}
> -force=0
> -while getopts j:vd:t:fh ARG ; do
> - case $ARG in
> - j ) parallel=$OPTARG ;;
> - v ) verbose=true ;;
> - d ) dst=$OPTARG ;;
> - t ) target=$OPTARG ;;
> - f ) force=1 ;;
> - h ) print_usage ; exit 0 ;;
> - ? ) print_usage ; exit 1 ;;
> - esac
> -done
> -shift $(($OPTIND - 1))
> -
> -if [ $# != 2 ]; then
> - print_usage
> - exit 1
> -fi
> -
> -tag1=$1
> -tag2=$2
> -
> -# convert path to absolute
> -case "${dst}" in
> - /*) ;;
> - *) dst=${PWD}/${dst} ;;
> -esac
> -dpdkroot=$(readlink -f $(dirname $0)/..)
> -
> -if [ -e "${dst}" -a "$force" = 0 ]; then
> - echo "The ${dst} directory is not empty. Remove it, use another"
> - echo "one (-d <dir>), or force overriding (-f)"
> - exit 1
> -fi
> -
> -rm -rf ${dst}
> -mkdir -p ${dst}
> -set_log_file ${dst}/abi-check.log
> -log "INFO" "Logs available in ${dst}/abi-check.log"
> -
> -command -v ${abicheck} || log "INFO" "Can't find ${abicheck} utility"
> -command -v ${abidump} || log "INFO" "Can't find ${abidump} utility"
> -
> -hash1=$(git show -s --format=%h "$tag1" -- 2> /dev/null | tail -1)
> -hash2=$(git show -s --format=%h "$tag2" -- 2> /dev/null | tail -1)
> -
> -# Make hashes available in output for non-local reference
> -tag1="$tag1 ($hash1)"
> -tag2="$tag2 ($hash2)"
> -
> -if [ "$hash1" = "$hash2" ]; then
> - log "ERROR" "$tag1 and $tag2 are the same revisions"
> - exit 1
> -fi
> -
> -cmd mkdir -p ${dst}
> -
> -# dump abi for each revision
> -gen_abi ${hash1}
> -gen_abi ${hash2}
> -
> -# compare the abi dumps
> -cmd cd ${dst}
> -ret=0
> -list=""
> -for i in ${hash2}/*.dump; do
> - name=`basename $i`
> - libname=${name%.dump}
> -
> - if [ ! -f ${hash1}/$name ]; then
> - log "INFO" "$NAME does not exist in $tag1. skipping..."
> - continue
> - fi
> -
> - local_ret=0
> - cmd $abicheck -l $libname \
> - -old ${hash1}/$name -new ${hash2}/$name || local_ret=$?
> - if [ $local_ret != 0 ]; then
> - log "NOTICE" "$abicheck returned $local_ret"
> - ret=$local_ret
> - list="$list $libname"
> - fi
> -done
> -
> -if [ $ret != 0 ]; then
> - log "NOTICE" "ABI may be incompatible, check reports/logs for details."
> - log "NOTICE" "Incompatible list: $list"
> -else
> - log "NOTICE" "No error detected, ABI is compatible."
> -fi
> -
> -log "INFO" "Logs are in ${dst}/abi-check.log"
> -log "INFO" "HTML reports are in ${dst}/compat_reports directory"
> -
> -exit $ret
> diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst
> index a21f4e7a4..219823923 100644
> --- a/doc/guides/contributing/abi_versioning.rst
> +++ b/doc/guides/contributing/abi_versioning.rst
> @@ -482,41 +482,27 @@ Running the ABI Validator
> -------------------------
>
> The ``devtools`` directory in the DPDK source tree contains a utility program,
> -``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI
> -Compliance Checker
> -<http://ispras.linuxbase.org/index.php/ABI_compliance_checker>`_.
> +``check-abi.sh``, for validating the DPDK ABI based on the libabigail `abidiff
> +utility: <https://sourceware.org/libabigail/manual/abidiff.html>`_
>
> -This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper``
> -utilities which can be installed via a package manager. For example::
> +The syntax of the ``check-abi.sh`` utility is::
>
> - sudo yum install abi-compliance-checker
> - sudo yum install abi-dumper
> + ./devtools/check-abi.sh <refdir> <newdir>
>
> -The syntax of the ``validate-abi.sh`` utility is::
> +Where <refdir> specifies the directory housing the reference build of dpdk, and
> +<newdir> specifies the dpdk build directory to check the abi of
>
> - ./devtools/validate-abi.sh <REV1> <REV2>
> +Example:
> +To compare your build branch to the ABI of the master branch
> +after you have built your branch
>
> -Where ``REV1`` and ``REV2`` are valid gitrevisions(7)
> -https://www.kernel.org/pub/software/scm/git/docs/gitrevisions.html
> -on the local repo.
> +#. $ cd <path to dpdk src tree>
> +#. $ mkdir ~/ref
> +#. $ git clone --local --no-hardlinks --single-branch -b master . ~/ref
> +#. $ cd ~/ref
> +#. $ cp <path to dpdk src tree from above>/.config ./.config
> +#. $ make defconfig
> +#. $ make
> +#. $ <path to dpdk src tree>/devtools/check-abi.sh ~/ref <path to dpdk src tree>
>
> -For example::
> -
> - # Check between the previous and latest commit:
> - ./devtools/validate-abi.sh HEAD~1 HEAD
> -
> - # Check on a specific compilation target:
> - ./devtools/validate-abi.sh -t x86_64-native-linux-gcc HEAD~1 HEAD
> -
> - # Check between two tags:
> - ./devtools/validate-abi.sh v2.0.0 v2.1.0
> -
> - # Check between git master and local topic-branch "vhost-hacking":
> - ./devtools/validate-abi.sh master vhost-hacking
> -
> -After the validation script completes (it can take a while since it need to
> -compile both tags) it will create compatibility reports in the
> -``./abi-check/compat_report`` directory. Listed incompatibilities can be found
> -as follows::
> -
> - grep -lr Incompatible abi-check/compat_reports/
> +
>
check-abi.sh appears to be backward step in terms of usability.
With validate-abi.sh I do can do a "validate-abi.sh HEAD~1 HEAD".
And it will do the build, install, dump and comparison for me.
And it picked up my 20.0.2 - > 21.0 changes no problem.
With check-abi on the other hand, I need to the build and install myself.
check-abi requires dump files, but I see no reference in the documentation to how these are created.
It silently fails when it doesn't find any ...
Do I run abi-dumper on the so's myself, or how does it work?
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
2020-04-17 9:42 3% ` Ray Kinsella
@ 2020-04-17 10:17 4% ` Thomas Monjalon
2020-04-17 10:33 3% ` Ray Kinsella
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-04-17 10:17 UTC (permalink / raw)
To: Bruce Richardson, Ray Kinsella
Cc: Trahe, Fiona, dev, Kusztal, ArkadiuszX, Neil Horman,
Luca Boccassi, Kevin Traynor, Yigit, Ferruh
17/04/2020 11:42, Ray Kinsella:
> On 17/04/2020 10:31, Bruce Richardson wrote:
> > On Fri, Apr 17, 2020 at 08:24:30AM +0100, Ray Kinsella wrote:
> >> On 16/04/2020 11:01, Thomas Monjalon wrote:
> >>> 16/04/2020 11:51, Bruce Richardson:
> >>>> On Wed, Apr 15, 2020 at 06:24:19PM +0100, Trahe, Fiona wrote:
> >>>>> 5a. If in 20.05 we add a version of a fn which breaks ABI 20.0, what should the name of the original function be? fn_v20, or fn_v20.0
> >>>>
> >>>> In technical terms it really doesn't matter, it's just a name that will be
> >>>> looked up in a table. I don't think we strictly enforce the naming, so
> >>>> whatever is clearest is best. I'd suggest the former.
> >>>
> >>> Each release can have a new ABI.
> >>
> >> How many ABI's do we want to support?
> >>
> > It's not how many we want to support, but for me it's a matter of how many
> > do we need to support. If an API is part of the stable set, it can't just
> > drop to being experimental for one or two releases - it's always stable
> > until deprecated. We also shouldn't have a situation where release 20.08 is
> > ABI compatible with 19.11 but not 20.02 and 20.05.
>
> True. Let me say it differently.
>
> Our only commitment is to support v20 - 19.11
> However you are correct, if something gets committed as v21 in 20.02, in practise should also be there in 20.05+ also.
> Because if it is committed as v21 and not as experimental, it should not be changing once committed.
>
> In answering Thomas,
> I was more commenting on the proliferation of ABI numbers & symbols we need to track in the build.
> With v20, v21 & Experimental we need to keep track of 3.
> If we start allowing quarterly builds to have managed ABI's, it will get confusing.
I don't remember why we are using intermediate ABI versions
between v20 and v21.
If we can use v21 for new ABI and make sure compatibility is maintained
between all versions from 19.11 to 20.08, I'm fine.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCHv3] Remove validate-abi.sh from tree
2020-04-17 10:11 9% ` Ray Kinsella
@ 2020-04-17 10:20 8% ` Thomas Monjalon
2020-04-17 10:35 4% ` Ray Kinsella
2020-04-17 11:47 4% ` Ray Kinsella
0 siblings, 2 replies; 200+ results
From: Thomas Monjalon @ 2020-04-17 10:20 UTC (permalink / raw)
To: Neil Horman, Ray Kinsella; +Cc: dev, david.marchand
17/04/2020 12:11, Ray Kinsella:
>
> On 16/04/2020 15:54, Neil Horman wrote:
> > Since we've moved away from our initial validate-abi.sh script, in
> > favor of check-abi.sh, which uses libabigail, remove the old script from
> > the tree, and update the docs accordingly
> >
> > Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
> > CC: thomas@monjalon.net
> > CC: david.marchand@redhat.com
> > CC: mdr@ashroe.eu
> > ---
> > MAINTAINERS | 1 -
> > devtools/validate-abi.sh | 251 ---------------------
> > doc/guides/contributing/abi_versioning.rst | 50 ++--
> > 3 files changed, 18 insertions(+), 284 deletions(-)
> > delete mode 100755 devtools/validate-abi.sh
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index 4800f6884..84b633431 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -152,7 +152,6 @@ F: devtools/gen-abi.sh
> > F: devtools/libabigail.abignore
> > F: devtools/update-abi.sh
> > F: devtools/update_version_map_abi.py
> > -F: devtools/validate-abi.sh
> > F: buildtools/check-experimental-syms.sh
> > F: buildtools/map-list-symbol.sh
> >
> > diff --git a/devtools/validate-abi.sh b/devtools/validate-abi.sh
> > deleted file mode 100755
> > index f64e19d38..000000000
> > --- a/devtools/validate-abi.sh
> > +++ /dev/null
> > @@ -1,251 +0,0 @@
> > -#!/usr/bin/env bash
> > -# SPDX-License-Identifier: BSD-3-Clause
> > -# Copyright(c) 2015 Neil Horman. All rights reserved.
> > -# Copyright(c) 2017 6WIND S.A.
> > -# All rights reserved
> > -
> > -set -e
> > -
> > -abicheck=abi-compliance-checker
> > -abidump=abi-dumper
> > -default_dst=abi-check
> > -default_target=x86_64-native-linuxapp-gcc
> > -
> > -# trap on error
> > -err_report() {
> > - echo "$0: error at line $1"
> > -}
> > -trap 'err_report $LINENO' ERR
> > -
> > -print_usage () {
> > - cat <<- END_OF_HELP
> > - $(basename $0) [options] <rev1> <rev2>
> > -
> > - This script compares the ABI of 2 git revisions of the current
> > - workspace. The output is a html report and a compilation log.
> > -
> > - The objective is to make sure that applications built against
> > - DSOs from the first revision can still run when executed using
> > - the DSOs built from the second revision.
> > -
> > - <rev1> and <rev2> are git commit id or tags.
> > -
> > - Options:
> > - -h show this help
> > - -j <num> enable parallel compilation with <num> threads
> > - -v show compilation logs on the console
> > - -d <dir> change working directory (default is ${default_dst})
> > - -t <target> the dpdk target to use (default is ${default_target})
> > - -f overwrite existing files in destination directory
> > -
> > - The script returns 0 on success, or the value of last failing
> > - call of ${abicheck} (incompatible abi or the tool has run with errors).
> > - The errors returned by ${abidump} are ignored.
> > -
> > - END_OF_HELP
> > -}
> > -
> > -# log in the file, and on stdout if verbose
> > -# $1: level string
> > -# $2: string to be logged
> > -log() {
> > - echo "$1: $2"
> > - if [ "${verbose}" != "true" ]; then
> > - echo "$1: $2" >&3
> > - fi
> > -}
> > -
> > -# launch a command and log it, taking care of surrounding spaces with quotes
> > -cmd() {
> > - local i s whitespace ret
> > - s=""
> > - whitespace="[[:space:]]"
> > - for i in "$@"; do
> > - if [[ $i =~ $whitespace ]]; then
> > - i=\"$i\"
> > - fi
> > - if [ -z "$s" ]; then
> > - s="$i"
> > - else
> > - s="$s $i"
> > - fi
> > - done
> > -
> > - ret=0
> > - log "CMD" "$s"
> > - "$@" || ret=$?
> > - if [ "$ret" != "0" ]; then
> > - log "CMD" "previous command returned $ret"
> > - fi
> > -
> > - return $ret
> > -}
> > -
> > -# redirect or copy stderr/stdout to a file
> > -# the syntax is unfamiliar, but it makes the rest of the
> > -# code easier to read, avoiding the use of pipes
> > -set_log_file() {
> > - # save original stdout and stderr in fd 3 and 4
> > - exec 3>&1
> > - exec 4>&2
> > - # create a new fd 5 that send to a file
> > - exec 5> >(cat > $1)
> > - # send stdout and stderr to fd 5
> > - if [ "${verbose}" = "true" ]; then
> > - exec 1> >(tee /dev/fd/5 >&3)
> > - exec 2> >(tee /dev/fd/5 >&4)
> > - else
> > - exec 1>&5
> > - exec 2>&5
> > - fi
> > -}
> > -
> > -# Make sure we configure SHARED libraries
> > -# Also turn off IGB and KNI as those require kernel headers to build
> > -fixup_config() {
> > - local conf=config/defconfig_$target
> > - cmd sed -i -e"$ a\CONFIG_RTE_BUILD_SHARED_LIB=y" $conf
> > - cmd sed -i -e"$ a\CONFIG_RTE_NEXT_ABI=n" $conf
> > - cmd sed -i -e"$ a\CONFIG_RTE_EAL_IGB_UIO=n" $conf
> > - cmd sed -i -e"$ a\CONFIG_RTE_LIBRTE_KNI=n" $conf
> > - cmd sed -i -e"$ a\CONFIG_RTE_KNI_KMOD=n" $conf
> > -}
> > -
> > -# build dpdk for the given tag and dump abi
> > -# $1: hash of the revision
> > -gen_abi() {
> > - local i
> > -
> > - cmd git clone ${dpdkroot} ${dst}/${1}
> > - cmd cd ${dst}/${1}
> > -
> > - log "INFO" "Checking out version ${1} of the dpdk"
> > - # Move to the old version of the tree
> > - cmd git checkout ${1}
> > -
> > - fixup_config
> > -
> > - # Now configure the build
> > - log "INFO" "Configuring DPDK ${1}"
> > - cmd make config T=$target O=$target
> > -
> > - # Checking abi compliance relies on using the dwarf information in
> > - # the shared objects. Build with -g to include them.
> > - log "INFO" "Building DPDK ${1}. This might take a moment"
> > - cmd make -j$parallel O=$target V=1 EXTRA_CFLAGS="-g -Og -Wno-error" \
> > - EXTRA_LDFLAGS="-g" || log "INFO" "The build failed"
> > -
> > - # Move to the lib directory
> > - cmd cd ${PWD}/$target/lib
> > - log "INFO" "Collecting ABI information for ${1}"
> > - for i in *.so; do
> > - [ -e "$i" ] || break
> > - cmd $abidump ${i} -o $dst/${1}/${i}.dump -lver ${1} || true
> > - # hack to ignore empty SymbolsInfo section (no public ABI)
> > - if grep -q "'SymbolInfo' => {}," $dst/${1}/${i}.dump \
> > - 2> /dev/null; then
> > - log "INFO" "${i} has no public ABI, remove dump file"
> > - cmd rm -f $dst/${1}/${i}.dump
> > - fi
> > - done
> > -}
> > -
> > -verbose=false
> > -parallel=1
> > -dst=${default_dst}
> > -target=${default_target}
> > -force=0
> > -while getopts j:vd:t:fh ARG ; do
> > - case $ARG in
> > - j ) parallel=$OPTARG ;;
> > - v ) verbose=true ;;
> > - d ) dst=$OPTARG ;;
> > - t ) target=$OPTARG ;;
> > - f ) force=1 ;;
> > - h ) print_usage ; exit 0 ;;
> > - ? ) print_usage ; exit 1 ;;
> > - esac
> > -done
> > -shift $(($OPTIND - 1))
> > -
> > -if [ $# != 2 ]; then
> > - print_usage
> > - exit 1
> > -fi
> > -
> > -tag1=$1
> > -tag2=$2
> > -
> > -# convert path to absolute
> > -case "${dst}" in
> > - /*) ;;
> > - *) dst=${PWD}/${dst} ;;
> > -esac
> > -dpdkroot=$(readlink -f $(dirname $0)/..)
> > -
> > -if [ -e "${dst}" -a "$force" = 0 ]; then
> > - echo "The ${dst} directory is not empty. Remove it, use another"
> > - echo "one (-d <dir>), or force overriding (-f)"
> > - exit 1
> > -fi
> > -
> > -rm -rf ${dst}
> > -mkdir -p ${dst}
> > -set_log_file ${dst}/abi-check.log
> > -log "INFO" "Logs available in ${dst}/abi-check.log"
> > -
> > -command -v ${abicheck} || log "INFO" "Can't find ${abicheck} utility"
> > -command -v ${abidump} || log "INFO" "Can't find ${abidump} utility"
> > -
> > -hash1=$(git show -s --format=%h "$tag1" -- 2> /dev/null | tail -1)
> > -hash2=$(git show -s --format=%h "$tag2" -- 2> /dev/null | tail -1)
> > -
> > -# Make hashes available in output for non-local reference
> > -tag1="$tag1 ($hash1)"
> > -tag2="$tag2 ($hash2)"
> > -
> > -if [ "$hash1" = "$hash2" ]; then
> > - log "ERROR" "$tag1 and $tag2 are the same revisions"
> > - exit 1
> > -fi
> > -
> > -cmd mkdir -p ${dst}
> > -
> > -# dump abi for each revision
> > -gen_abi ${hash1}
> > -gen_abi ${hash2}
> > -
> > -# compare the abi dumps
> > -cmd cd ${dst}
> > -ret=0
> > -list=""
> > -for i in ${hash2}/*.dump; do
> > - name=`basename $i`
> > - libname=${name%.dump}
> > -
> > - if [ ! -f ${hash1}/$name ]; then
> > - log "INFO" "$NAME does not exist in $tag1. skipping..."
> > - continue
> > - fi
> > -
> > - local_ret=0
> > - cmd $abicheck -l $libname \
> > - -old ${hash1}/$name -new ${hash2}/$name || local_ret=$?
> > - if [ $local_ret != 0 ]; then
> > - log "NOTICE" "$abicheck returned $local_ret"
> > - ret=$local_ret
> > - list="$list $libname"
> > - fi
> > -done
> > -
> > -if [ $ret != 0 ]; then
> > - log "NOTICE" "ABI may be incompatible, check reports/logs for details."
> > - log "NOTICE" "Incompatible list: $list"
> > -else
> > - log "NOTICE" "No error detected, ABI is compatible."
> > -fi
> > -
> > -log "INFO" "Logs are in ${dst}/abi-check.log"
> > -log "INFO" "HTML reports are in ${dst}/compat_reports directory"
> > -
> > -exit $ret
> > diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst
> > index a21f4e7a4..219823923 100644
> > --- a/doc/guides/contributing/abi_versioning.rst
> > +++ b/doc/guides/contributing/abi_versioning.rst
> > @@ -482,41 +482,27 @@ Running the ABI Validator
> > -------------------------
> >
> > The ``devtools`` directory in the DPDK source tree contains a utility program,
> > -``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI
> > -Compliance Checker
> > -<http://ispras.linuxbase.org/index.php/ABI_compliance_checker>`_.
> > +``check-abi.sh``, for validating the DPDK ABI based on the libabigail `abidiff
> > +utility: <https://sourceware.org/libabigail/manual/abidiff.html>`_
> >
> > -This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper``
> > -utilities which can be installed via a package manager. For example::
> > +The syntax of the ``check-abi.sh`` utility is::
> >
> > - sudo yum install abi-compliance-checker
> > - sudo yum install abi-dumper
> > + ./devtools/check-abi.sh <refdir> <newdir>
> >
> > -The syntax of the ``validate-abi.sh`` utility is::
> > +Where <refdir> specifies the directory housing the reference build of dpdk, and
> > +<newdir> specifies the dpdk build directory to check the abi of
> >
> > - ./devtools/validate-abi.sh <REV1> <REV2>
> > +Example:
> > +To compare your build branch to the ABI of the master branch
> > +after you have built your branch
> >
> > -Where ``REV1`` and ``REV2`` are valid gitrevisions(7)
> > -https://www.kernel.org/pub/software/scm/git/docs/gitrevisions.html
> > -on the local repo.
> > +#. $ cd <path to dpdk src tree>
> > +#. $ mkdir ~/ref
> > +#. $ git clone --local --no-hardlinks --single-branch -b master . ~/ref
> > +#. $ cd ~/ref
> > +#. $ cp <path to dpdk src tree from above>/.config ./.config
> > +#. $ make defconfig
> > +#. $ make
> > +#. $ <path to dpdk src tree>/devtools/check-abi.sh ~/ref <path to dpdk src tree>
> >
> > -For example::
> > -
> > - # Check between the previous and latest commit:
> > - ./devtools/validate-abi.sh HEAD~1 HEAD
> > -
> > - # Check on a specific compilation target:
> > - ./devtools/validate-abi.sh -t x86_64-native-linux-gcc HEAD~1 HEAD
> > -
> > - # Check between two tags:
> > - ./devtools/validate-abi.sh v2.0.0 v2.1.0
> > -
> > - # Check between git master and local topic-branch "vhost-hacking":
> > - ./devtools/validate-abi.sh master vhost-hacking
> > -
> > -After the validation script completes (it can take a while since it need to
> > -compile both tags) it will create compatibility reports in the
> > -``./abi-check/compat_report`` directory. Listed incompatibilities can be found
> > -as follows::
> > -
> > - grep -lr Incompatible abi-check/compat_reports/
> > +
> >
>
> check-abi.sh appears to be backward step in terms of usability.
No, check-abi.sh benefits from a nice integration in build scripts.
See below.
> With validate-abi.sh I do can do a "validate-abi.sh HEAD~1 HEAD".
> And it will do the build, install, dump and comparison for me.
> And it picked up my 20.0.2 - > 21.0 changes no problem.
>
> With check-abi on the other hand, I need to the build and install myself.
> check-abi requires dump files, but I see no reference in the documentation to how these are created.
> It silently fails when it doesn't find any ...
>
> Do I run abi-dumper on the so's myself, or how does it work?
check-abi.sh is integrated in test-build.sh and test-meson-builds.sh.
Probably we should document usage in these scripts.
^ permalink raw reply [relevance 8%]
* Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
2020-04-17 10:17 4% ` Thomas Monjalon
@ 2020-04-17 10:33 3% ` Ray Kinsella
2020-04-17 11:46 5% ` Trahe, Fiona
0 siblings, 1 reply; 200+ results
From: Ray Kinsella @ 2020-04-17 10:33 UTC (permalink / raw)
To: Thomas Monjalon, Bruce Richardson
Cc: Trahe, Fiona, dev, Kusztal, ArkadiuszX, Neil Horman,
Luca Boccassi, Kevin Traynor, Yigit, Ferruh
On 17/04/2020 11:17, Thomas Monjalon wrote:
> 17/04/2020 11:42, Ray Kinsella:
>> On 17/04/2020 10:31, Bruce Richardson wrote:
>>> On Fri, Apr 17, 2020 at 08:24:30AM +0100, Ray Kinsella wrote:
>>>> On 16/04/2020 11:01, Thomas Monjalon wrote:
>>>>> 16/04/2020 11:51, Bruce Richardson:
>>>>>> On Wed, Apr 15, 2020 at 06:24:19PM +0100, Trahe, Fiona wrote:
>>>>>>> 5a. If in 20.05 we add a version of a fn which breaks ABI 20.0, what should the name of the original function be? fn_v20, or fn_v20.0
>>>>>>
>>>>>> In technical terms it really doesn't matter, it's just a name that will be
>>>>>> looked up in a table. I don't think we strictly enforce the naming, so
>>>>>> whatever is clearest is best. I'd suggest the former.
>>>>>
>>>>> Each release can have a new ABI.
>>>>
>>>> How many ABI's do we want to support?
>>>>
>>> It's not how many we want to support, but for me it's a matter of how many
>>> do we need to support. If an API is part of the stable set, it can't just
>>> drop to being experimental for one or two releases - it's always stable
>>> until deprecated. We also shouldn't have a situation where release 20.08 is
>>> ABI compatible with 19.11 but not 20.02 and 20.05.
>>
>> True. Let me say it differently.
>>
>> Our only commitment is to support v20 - 19.11
>> However you are correct, if something gets committed as v21 in 20.02, in practise should also be there in 20.05+ also.
>> Because if it is committed as v21 and not as experimental, it should not be changing once committed.
>>
>> In answering Thomas,
>> I was more commenting on the proliferation of ABI numbers & symbols we need to track in the build.
>> With v20, v21 & Experimental we need to keep track of 3.
>> If we start allowing quarterly builds to have managed ABI's, it will get confusing.
>
> I don't remember why we are using intermediate ABI versions
> between v20 and v21.
> If we can use v21 for new ABI and make sure compatibility is maintained
> between all versions from 19.11 to 20.08, I'm fine.
>
Well I guess we missed this in 20.02, so I recommend that we fix in 20.05.
It will mean that a couple of symbols versioned 20.0.2 in DPDK 20.02,
Will becomes 21.0 in DPDK 20.05 ... ABI checker will complain.
Apart from that I doubt anyone will notice?
Ray K
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCHv3] Remove validate-abi.sh from tree
2020-04-17 10:20 8% ` Thomas Monjalon
@ 2020-04-17 10:35 4% ` Ray Kinsella
2020-04-17 11:46 4% ` Thomas Monjalon
2020-04-17 11:47 4% ` Ray Kinsella
1 sibling, 1 reply; 200+ results
From: Ray Kinsella @ 2020-04-17 10:35 UTC (permalink / raw)
To: Thomas Monjalon, Neil Horman; +Cc: dev, david.marchand
On 17/04/2020 11:20, Thomas Monjalon wrote:
> 17/04/2020 12:11, Ray Kinsella:
>>
>> On 16/04/2020 15:54, Neil Horman wrote:
>>> Since we've moved away from our initial validate-abi.sh script, in
>>> favor of check-abi.sh, which uses libabigail, remove the old script from
>>> the tree, and update the docs accordingly
>>>
>>> Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
>>> CC: thomas@monjalon.net
>>> CC: david.marchand@redhat.com
>>> CC: mdr@ashroe.eu
>>> ---
>>> MAINTAINERS | 1 -
>>> devtools/validate-abi.sh | 251 ---------------------
>>> doc/guides/contributing/abi_versioning.rst | 50 ++--
>>> 3 files changed, 18 insertions(+), 284 deletions(-)
>>> delete mode 100755 devtools/validate-abi.sh
>>>
>>> diff --git a/MAINTAINERS b/MAINTAINERS
>>> index 4800f6884..84b633431 100644
>>> --- a/MAINTAINERS
>>> +++ b/MAINTAINERS
>>> @@ -152,7 +152,6 @@ F: devtools/gen-abi.sh
>>> F: devtools/libabigail.abignore
>>> F: devtools/update-abi.sh
>>> F: devtools/update_version_map_abi.py
>>> -F: devtools/validate-abi.sh
>>> F: buildtools/check-experimental-syms.sh
>>> F: buildtools/map-list-symbol.sh
>>>
>>> diff --git a/devtools/validate-abi.sh b/devtools/validate-abi.sh
>>> deleted file mode 100755
>>> index f64e19d38..000000000
>>> --- a/devtools/validate-abi.sh
>>> +++ /dev/null
>>> @@ -1,251 +0,0 @@
>>> -#!/usr/bin/env bash
>>> -# SPDX-License-Identifier: BSD-3-Clause
>>> -# Copyright(c) 2015 Neil Horman. All rights reserved.
>>> -# Copyright(c) 2017 6WIND S.A.
>>> -# All rights reserved
>>> -
>>> -set -e
>>> -
>>> -abicheck=abi-compliance-checker
>>> -abidump=abi-dumper
>>> -default_dst=abi-check
>>> -default_target=x86_64-native-linuxapp-gcc
>>> -
>>> -# trap on error
>>> -err_report() {
>>> - echo "$0: error at line $1"
>>> -}
>>> -trap 'err_report $LINENO' ERR
>>> -
>>> -print_usage () {
>>> - cat <<- END_OF_HELP
>>> - $(basename $0) [options] <rev1> <rev2>
>>> -
>>> - This script compares the ABI of 2 git revisions of the current
>>> - workspace. The output is a html report and a compilation log.
>>> -
>>> - The objective is to make sure that applications built against
>>> - DSOs from the first revision can still run when executed using
>>> - the DSOs built from the second revision.
>>> -
>>> - <rev1> and <rev2> are git commit id or tags.
>>> -
>>> - Options:
>>> - -h show this help
>>> - -j <num> enable parallel compilation with <num> threads
>>> - -v show compilation logs on the console
>>> - -d <dir> change working directory (default is ${default_dst})
>>> - -t <target> the dpdk target to use (default is ${default_target})
>>> - -f overwrite existing files in destination directory
>>> -
>>> - The script returns 0 on success, or the value of last failing
>>> - call of ${abicheck} (incompatible abi or the tool has run with errors).
>>> - The errors returned by ${abidump} are ignored.
>>> -
>>> - END_OF_HELP
>>> -}
>>> -
>>> -# log in the file, and on stdout if verbose
>>> -# $1: level string
>>> -# $2: string to be logged
>>> -log() {
>>> - echo "$1: $2"
>>> - if [ "${verbose}" != "true" ]; then
>>> - echo "$1: $2" >&3
>>> - fi
>>> -}
>>> -
>>> -# launch a command and log it, taking care of surrounding spaces with quotes
>>> -cmd() {
>>> - local i s whitespace ret
>>> - s=""
>>> - whitespace="[[:space:]]"
>>> - for i in "$@"; do
>>> - if [[ $i =~ $whitespace ]]; then
>>> - i=\"$i\"
>>> - fi
>>> - if [ -z "$s" ]; then
>>> - s="$i"
>>> - else
>>> - s="$s $i"
>>> - fi
>>> - done
>>> -
>>> - ret=0
>>> - log "CMD" "$s"
>>> - "$@" || ret=$?
>>> - if [ "$ret" != "0" ]; then
>>> - log "CMD" "previous command returned $ret"
>>> - fi
>>> -
>>> - return $ret
>>> -}
>>> -
>>> -# redirect or copy stderr/stdout to a file
>>> -# the syntax is unfamiliar, but it makes the rest of the
>>> -# code easier to read, avoiding the use of pipes
>>> -set_log_file() {
>>> - # save original stdout and stderr in fd 3 and 4
>>> - exec 3>&1
>>> - exec 4>&2
>>> - # create a new fd 5 that send to a file
>>> - exec 5> >(cat > $1)
>>> - # send stdout and stderr to fd 5
>>> - if [ "${verbose}" = "true" ]; then
>>> - exec 1> >(tee /dev/fd/5 >&3)
>>> - exec 2> >(tee /dev/fd/5 >&4)
>>> - else
>>> - exec 1>&5
>>> - exec 2>&5
>>> - fi
>>> -}
>>> -
>>> -# Make sure we configure SHARED libraries
>>> -# Also turn off IGB and KNI as those require kernel headers to build
>>> -fixup_config() {
>>> - local conf=config/defconfig_$target
>>> - cmd sed -i -e"$ a\CONFIG_RTE_BUILD_SHARED_LIB=y" $conf
>>> - cmd sed -i -e"$ a\CONFIG_RTE_NEXT_ABI=n" $conf
>>> - cmd sed -i -e"$ a\CONFIG_RTE_EAL_IGB_UIO=n" $conf
>>> - cmd sed -i -e"$ a\CONFIG_RTE_LIBRTE_KNI=n" $conf
>>> - cmd sed -i -e"$ a\CONFIG_RTE_KNI_KMOD=n" $conf
>>> -}
>>> -
>>> -# build dpdk for the given tag and dump abi
>>> -# $1: hash of the revision
>>> -gen_abi() {
>>> - local i
>>> -
>>> - cmd git clone ${dpdkroot} ${dst}/${1}
>>> - cmd cd ${dst}/${1}
>>> -
>>> - log "INFO" "Checking out version ${1} of the dpdk"
>>> - # Move to the old version of the tree
>>> - cmd git checkout ${1}
>>> -
>>> - fixup_config
>>> -
>>> - # Now configure the build
>>> - log "INFO" "Configuring DPDK ${1}"
>>> - cmd make config T=$target O=$target
>>> -
>>> - # Checking abi compliance relies on using the dwarf information in
>>> - # the shared objects. Build with -g to include them.
>>> - log "INFO" "Building DPDK ${1}. This might take a moment"
>>> - cmd make -j$parallel O=$target V=1 EXTRA_CFLAGS="-g -Og -Wno-error" \
>>> - EXTRA_LDFLAGS="-g" || log "INFO" "The build failed"
>>> -
>>> - # Move to the lib directory
>>> - cmd cd ${PWD}/$target/lib
>>> - log "INFO" "Collecting ABI information for ${1}"
>>> - for i in *.so; do
>>> - [ -e "$i" ] || break
>>> - cmd $abidump ${i} -o $dst/${1}/${i}.dump -lver ${1} || true
>>> - # hack to ignore empty SymbolsInfo section (no public ABI)
>>> - if grep -q "'SymbolInfo' => {}," $dst/${1}/${i}.dump \
>>> - 2> /dev/null; then
>>> - log "INFO" "${i} has no public ABI, remove dump file"
>>> - cmd rm -f $dst/${1}/${i}.dump
>>> - fi
>>> - done
>>> -}
>>> -
>>> -verbose=false
>>> -parallel=1
>>> -dst=${default_dst}
>>> -target=${default_target}
>>> -force=0
>>> -while getopts j:vd:t:fh ARG ; do
>>> - case $ARG in
>>> - j ) parallel=$OPTARG ;;
>>> - v ) verbose=true ;;
>>> - d ) dst=$OPTARG ;;
>>> - t ) target=$OPTARG ;;
>>> - f ) force=1 ;;
>>> - h ) print_usage ; exit 0 ;;
>>> - ? ) print_usage ; exit 1 ;;
>>> - esac
>>> -done
>>> -shift $(($OPTIND - 1))
>>> -
>>> -if [ $# != 2 ]; then
>>> - print_usage
>>> - exit 1
>>> -fi
>>> -
>>> -tag1=$1
>>> -tag2=$2
>>> -
>>> -# convert path to absolute
>>> -case "${dst}" in
>>> - /*) ;;
>>> - *) dst=${PWD}/${dst} ;;
>>> -esac
>>> -dpdkroot=$(readlink -f $(dirname $0)/..)
>>> -
>>> -if [ -e "${dst}" -a "$force" = 0 ]; then
>>> - echo "The ${dst} directory is not empty. Remove it, use another"
>>> - echo "one (-d <dir>), or force overriding (-f)"
>>> - exit 1
>>> -fi
>>> -
>>> -rm -rf ${dst}
>>> -mkdir -p ${dst}
>>> -set_log_file ${dst}/abi-check.log
>>> -log "INFO" "Logs available in ${dst}/abi-check.log"
>>> -
>>> -command -v ${abicheck} || log "INFO" "Can't find ${abicheck} utility"
>>> -command -v ${abidump} || log "INFO" "Can't find ${abidump} utility"
>>> -
>>> -hash1=$(git show -s --format=%h "$tag1" -- 2> /dev/null | tail -1)
>>> -hash2=$(git show -s --format=%h "$tag2" -- 2> /dev/null | tail -1)
>>> -
>>> -# Make hashes available in output for non-local reference
>>> -tag1="$tag1 ($hash1)"
>>> -tag2="$tag2 ($hash2)"
>>> -
>>> -if [ "$hash1" = "$hash2" ]; then
>>> - log "ERROR" "$tag1 and $tag2 are the same revisions"
>>> - exit 1
>>> -fi
>>> -
>>> -cmd mkdir -p ${dst}
>>> -
>>> -# dump abi for each revision
>>> -gen_abi ${hash1}
>>> -gen_abi ${hash2}
>>> -
>>> -# compare the abi dumps
>>> -cmd cd ${dst}
>>> -ret=0
>>> -list=""
>>> -for i in ${hash2}/*.dump; do
>>> - name=`basename $i`
>>> - libname=${name%.dump}
>>> -
>>> - if [ ! -f ${hash1}/$name ]; then
>>> - log "INFO" "$NAME does not exist in $tag1. skipping..."
>>> - continue
>>> - fi
>>> -
>>> - local_ret=0
>>> - cmd $abicheck -l $libname \
>>> - -old ${hash1}/$name -new ${hash2}/$name || local_ret=$?
>>> - if [ $local_ret != 0 ]; then
>>> - log "NOTICE" "$abicheck returned $local_ret"
>>> - ret=$local_ret
>>> - list="$list $libname"
>>> - fi
>>> -done
>>> -
>>> -if [ $ret != 0 ]; then
>>> - log "NOTICE" "ABI may be incompatible, check reports/logs for details."
>>> - log "NOTICE" "Incompatible list: $list"
>>> -else
>>> - log "NOTICE" "No error detected, ABI is compatible."
>>> -fi
>>> -
>>> -log "INFO" "Logs are in ${dst}/abi-check.log"
>>> -log "INFO" "HTML reports are in ${dst}/compat_reports directory"
>>> -
>>> -exit $ret
>>> diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst
>>> index a21f4e7a4..219823923 100644
>>> --- a/doc/guides/contributing/abi_versioning.rst
>>> +++ b/doc/guides/contributing/abi_versioning.rst
>>> @@ -482,41 +482,27 @@ Running the ABI Validator
>>> -------------------------
>>>
>>> The ``devtools`` directory in the DPDK source tree contains a utility program,
>>> -``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI
>>> -Compliance Checker
>>> -<http://ispras.linuxbase.org/index.php/ABI_compliance_checker>`_.
>>> +``check-abi.sh``, for validating the DPDK ABI based on the libabigail `abidiff
>>> +utility: <https://sourceware.org/libabigail/manual/abidiff.html>`_
>>>
>>> -This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper``
>>> -utilities which can be installed via a package manager. For example::
>>> +The syntax of the ``check-abi.sh`` utility is::
>>>
>>> - sudo yum install abi-compliance-checker
>>> - sudo yum install abi-dumper
>>> + ./devtools/check-abi.sh <refdir> <newdir>
>>>
>>> -The syntax of the ``validate-abi.sh`` utility is::
>>> +Where <refdir> specifies the directory housing the reference build of dpdk, and
>>> +<newdir> specifies the dpdk build directory to check the abi of
>>>
>>> - ./devtools/validate-abi.sh <REV1> <REV2>
>>> +Example:
>>> +To compare your build branch to the ABI of the master branch
>>> +after you have built your branch
>>>
>>> -Where ``REV1`` and ``REV2`` are valid gitrevisions(7)
>>> -https://www.kernel.org/pub/software/scm/git/docs/gitrevisions.html
>>> -on the local repo.
>>> +#. $ cd <path to dpdk src tree>
>>> +#. $ mkdir ~/ref
>>> +#. $ git clone --local --no-hardlinks --single-branch -b master . ~/ref
>>> +#. $ cd ~/ref
>>> +#. $ cp <path to dpdk src tree from above>/.config ./.config
>>> +#. $ make defconfig
>>> +#. $ make
>>> +#. $ <path to dpdk src tree>/devtools/check-abi.sh ~/ref <path to dpdk src tree>
>>>
>>> -For example::
>>> -
>>> - # Check between the previous and latest commit:
>>> - ./devtools/validate-abi.sh HEAD~1 HEAD
>>> -
>>> - # Check on a specific compilation target:
>>> - ./devtools/validate-abi.sh -t x86_64-native-linux-gcc HEAD~1 HEAD
>>> -
>>> - # Check between two tags:
>>> - ./devtools/validate-abi.sh v2.0.0 v2.1.0
>>> -
>>> - # Check between git master and local topic-branch "vhost-hacking":
>>> - ./devtools/validate-abi.sh master vhost-hacking
>>> -
>>> -After the validation script completes (it can take a while since it need to
>>> -compile both tags) it will create compatibility reports in the
>>> -``./abi-check/compat_report`` directory. Listed incompatibilities can be found
>>> -as follows::
>>> -
>>> - grep -lr Incompatible abi-check/compat_reports/
>>> +
>>>
>>
>> check-abi.sh appears to be backward step in terms of usability.
>
> No, check-abi.sh benefits from a nice integration in build scripts.
> See below.
>
>> With validate-abi.sh I do can do a "validate-abi.sh HEAD~1 HEAD".
>> And it will do the build, install, dump and comparison for me.
>> And it picked up my 20.0.2 - > 21.0 changes no problem.
>>
>> With check-abi on the other hand, I need to the build and install myself.
>> check-abi requires dump files, but I see no reference in the documentation to how these are created.
>> It silently fails when it doesn't find any ...
>>
>> Do I run abi-dumper on the so's myself, or how does it work?
>
> check-abi.sh is integrated in test-build.sh and test-meson-builds.sh.
> Probably we should document usage in these scripts.
>
ok in that case, this documentation should reference those scripts instead?
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCHv3] Remove validate-abi.sh from tree
2020-04-17 10:35 4% ` Ray Kinsella
@ 2020-04-17 11:46 4% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-04-17 11:46 UTC (permalink / raw)
To: Neil Horman, Ray Kinsella; +Cc: dev, david.marchand
17/04/2020 12:35, Ray Kinsella:
> On 17/04/2020 11:20, Thomas Monjalon wrote:
> > 17/04/2020 12:11, Ray Kinsella:
> >> On 16/04/2020 15:54, Neil Horman wrote:
> >>> Since we've moved away from our initial validate-abi.sh script, in
> >>> favor of check-abi.sh, which uses libabigail, remove the old script from
> >>> the tree, and update the docs accordingly
> >>>
> >>> Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
> >>> CC: thomas@monjalon.net
> >>> CC: david.marchand@redhat.com
> >>> CC: mdr@ashroe.eu
> >>> ---
> >>> MAINTAINERS | 1 -
> >>> devtools/validate-abi.sh | 251 ---------------------
> >>> doc/guides/contributing/abi_versioning.rst | 50 ++--
> >>> 3 files changed, 18 insertions(+), 284 deletions(-)
> >>> delete mode 100755 devtools/validate-abi.sh
> >>>
> >>> diff --git a/MAINTAINERS b/MAINTAINERS
> >>> index 4800f6884..84b633431 100644
> >>> --- a/MAINTAINERS
> >>> +++ b/MAINTAINERS
> >>> @@ -152,7 +152,6 @@ F: devtools/gen-abi.sh
> >>> F: devtools/libabigail.abignore
> >>> F: devtools/update-abi.sh
> >>> F: devtools/update_version_map_abi.py
> >>> -F: devtools/validate-abi.sh
> >>> F: buildtools/check-experimental-syms.sh
> >>> F: buildtools/map-list-symbol.sh
> >>>
> >>> diff --git a/devtools/validate-abi.sh b/devtools/validate-abi.sh
> >>> deleted file mode 100755
> >>> index f64e19d38..000000000
> >>> --- a/devtools/validate-abi.sh
> >>> +++ /dev/null
> >>> @@ -1,251 +0,0 @@
> >>> -#!/usr/bin/env bash
> >>> -# SPDX-License-Identifier: BSD-3-Clause
> >>> -# Copyright(c) 2015 Neil Horman. All rights reserved.
> >>> -# Copyright(c) 2017 6WIND S.A.
> >>> -# All rights reserved
> >>> -
> >>> -set -e
> >>> -
> >>> -abicheck=abi-compliance-checker
> >>> -abidump=abi-dumper
> >>> -default_dst=abi-check
> >>> -default_target=x86_64-native-linuxapp-gcc
> >>> -
> >>> -# trap on error
> >>> -err_report() {
> >>> - echo "$0: error at line $1"
> >>> -}
> >>> -trap 'err_report $LINENO' ERR
> >>> -
> >>> -print_usage () {
> >>> - cat <<- END_OF_HELP
> >>> - $(basename $0) [options] <rev1> <rev2>
> >>> -
> >>> - This script compares the ABI of 2 git revisions of the current
> >>> - workspace. The output is a html report and a compilation log.
> >>> -
> >>> - The objective is to make sure that applications built against
> >>> - DSOs from the first revision can still run when executed using
> >>> - the DSOs built from the second revision.
> >>> -
> >>> - <rev1> and <rev2> are git commit id or tags.
> >>> -
> >>> - Options:
> >>> - -h show this help
> >>> - -j <num> enable parallel compilation with <num> threads
> >>> - -v show compilation logs on the console
> >>> - -d <dir> change working directory (default is ${default_dst})
> >>> - -t <target> the dpdk target to use (default is ${default_target})
> >>> - -f overwrite existing files in destination directory
> >>> -
> >>> - The script returns 0 on success, or the value of last failing
> >>> - call of ${abicheck} (incompatible abi or the tool has run with errors).
> >>> - The errors returned by ${abidump} are ignored.
> >>> -
> >>> - END_OF_HELP
> >>> -}
> >>> -
> >>> -# log in the file, and on stdout if verbose
> >>> -# $1: level string
> >>> -# $2: string to be logged
> >>> -log() {
> >>> - echo "$1: $2"
> >>> - if [ "${verbose}" != "true" ]; then
> >>> - echo "$1: $2" >&3
> >>> - fi
> >>> -}
> >>> -
> >>> -# launch a command and log it, taking care of surrounding spaces with quotes
> >>> -cmd() {
> >>> - local i s whitespace ret
> >>> - s=""
> >>> - whitespace="[[:space:]]"
> >>> - for i in "$@"; do
> >>> - if [[ $i =~ $whitespace ]]; then
> >>> - i=\"$i\"
> >>> - fi
> >>> - if [ -z "$s" ]; then
> >>> - s="$i"
> >>> - else
> >>> - s="$s $i"
> >>> - fi
> >>> - done
> >>> -
> >>> - ret=0
> >>> - log "CMD" "$s"
> >>> - "$@" || ret=$?
> >>> - if [ "$ret" != "0" ]; then
> >>> - log "CMD" "previous command returned $ret"
> >>> - fi
> >>> -
> >>> - return $ret
> >>> -}
> >>> -
> >>> -# redirect or copy stderr/stdout to a file
> >>> -# the syntax is unfamiliar, but it makes the rest of the
> >>> -# code easier to read, avoiding the use of pipes
> >>> -set_log_file() {
> >>> - # save original stdout and stderr in fd 3 and 4
> >>> - exec 3>&1
> >>> - exec 4>&2
> >>> - # create a new fd 5 that send to a file
> >>> - exec 5> >(cat > $1)
> >>> - # send stdout and stderr to fd 5
> >>> - if [ "${verbose}" = "true" ]; then
> >>> - exec 1> >(tee /dev/fd/5 >&3)
> >>> - exec 2> >(tee /dev/fd/5 >&4)
> >>> - else
> >>> - exec 1>&5
> >>> - exec 2>&5
> >>> - fi
> >>> -}
> >>> -
> >>> -# Make sure we configure SHARED libraries
> >>> -# Also turn off IGB and KNI as those require kernel headers to build
> >>> -fixup_config() {
> >>> - local conf=config/defconfig_$target
> >>> - cmd sed -i -e"$ a\CONFIG_RTE_BUILD_SHARED_LIB=y" $conf
> >>> - cmd sed -i -e"$ a\CONFIG_RTE_NEXT_ABI=n" $conf
> >>> - cmd sed -i -e"$ a\CONFIG_RTE_EAL_IGB_UIO=n" $conf
> >>> - cmd sed -i -e"$ a\CONFIG_RTE_LIBRTE_KNI=n" $conf
> >>> - cmd sed -i -e"$ a\CONFIG_RTE_KNI_KMOD=n" $conf
> >>> -}
> >>> -
> >>> -# build dpdk for the given tag and dump abi
> >>> -# $1: hash of the revision
> >>> -gen_abi() {
> >>> - local i
> >>> -
> >>> - cmd git clone ${dpdkroot} ${dst}/${1}
> >>> - cmd cd ${dst}/${1}
> >>> -
> >>> - log "INFO" "Checking out version ${1} of the dpdk"
> >>> - # Move to the old version of the tree
> >>> - cmd git checkout ${1}
> >>> -
> >>> - fixup_config
> >>> -
> >>> - # Now configure the build
> >>> - log "INFO" "Configuring DPDK ${1}"
> >>> - cmd make config T=$target O=$target
> >>> -
> >>> - # Checking abi compliance relies on using the dwarf information in
> >>> - # the shared objects. Build with -g to include them.
> >>> - log "INFO" "Building DPDK ${1}. This might take a moment"
> >>> - cmd make -j$parallel O=$target V=1 EXTRA_CFLAGS="-g -Og -Wno-error" \
> >>> - EXTRA_LDFLAGS="-g" || log "INFO" "The build failed"
> >>> -
> >>> - # Move to the lib directory
> >>> - cmd cd ${PWD}/$target/lib
> >>> - log "INFO" "Collecting ABI information for ${1}"
> >>> - for i in *.so; do
> >>> - [ -e "$i" ] || break
> >>> - cmd $abidump ${i} -o $dst/${1}/${i}.dump -lver ${1} || true
> >>> - # hack to ignore empty SymbolsInfo section (no public ABI)
> >>> - if grep -q "'SymbolInfo' => {}," $dst/${1}/${i}.dump \
> >>> - 2> /dev/null; then
> >>> - log "INFO" "${i} has no public ABI, remove dump file"
> >>> - cmd rm -f $dst/${1}/${i}.dump
> >>> - fi
> >>> - done
> >>> -}
> >>> -
> >>> -verbose=false
> >>> -parallel=1
> >>> -dst=${default_dst}
> >>> -target=${default_target}
> >>> -force=0
> >>> -while getopts j:vd:t:fh ARG ; do
> >>> - case $ARG in
> >>> - j ) parallel=$OPTARG ;;
> >>> - v ) verbose=true ;;
> >>> - d ) dst=$OPTARG ;;
> >>> - t ) target=$OPTARG ;;
> >>> - f ) force=1 ;;
> >>> - h ) print_usage ; exit 0 ;;
> >>> - ? ) print_usage ; exit 1 ;;
> >>> - esac
> >>> -done
> >>> -shift $(($OPTIND - 1))
> >>> -
> >>> -if [ $# != 2 ]; then
> >>> - print_usage
> >>> - exit 1
> >>> -fi
> >>> -
> >>> -tag1=$1
> >>> -tag2=$2
> >>> -
> >>> -# convert path to absolute
> >>> -case "${dst}" in
> >>> - /*) ;;
> >>> - *) dst=${PWD}/${dst} ;;
> >>> -esac
> >>> -dpdkroot=$(readlink -f $(dirname $0)/..)
> >>> -
> >>> -if [ -e "${dst}" -a "$force" = 0 ]; then
> >>> - echo "The ${dst} directory is not empty. Remove it, use another"
> >>> - echo "one (-d <dir>), or force overriding (-f)"
> >>> - exit 1
> >>> -fi
> >>> -
> >>> -rm -rf ${dst}
> >>> -mkdir -p ${dst}
> >>> -set_log_file ${dst}/abi-check.log
> >>> -log "INFO" "Logs available in ${dst}/abi-check.log"
> >>> -
> >>> -command -v ${abicheck} || log "INFO" "Can't find ${abicheck} utility"
> >>> -command -v ${abidump} || log "INFO" "Can't find ${abidump} utility"
> >>> -
> >>> -hash1=$(git show -s --format=%h "$tag1" -- 2> /dev/null | tail -1)
> >>> -hash2=$(git show -s --format=%h "$tag2" -- 2> /dev/null | tail -1)
> >>> -
> >>> -# Make hashes available in output for non-local reference
> >>> -tag1="$tag1 ($hash1)"
> >>> -tag2="$tag2 ($hash2)"
> >>> -
> >>> -if [ "$hash1" = "$hash2" ]; then
> >>> - log "ERROR" "$tag1 and $tag2 are the same revisions"
> >>> - exit 1
> >>> -fi
> >>> -
> >>> -cmd mkdir -p ${dst}
> >>> -
> >>> -# dump abi for each revision
> >>> -gen_abi ${hash1}
> >>> -gen_abi ${hash2}
> >>> -
> >>> -# compare the abi dumps
> >>> -cmd cd ${dst}
> >>> -ret=0
> >>> -list=""
> >>> -for i in ${hash2}/*.dump; do
> >>> - name=`basename $i`
> >>> - libname=${name%.dump}
> >>> -
> >>> - if [ ! -f ${hash1}/$name ]; then
> >>> - log "INFO" "$NAME does not exist in $tag1. skipping..."
> >>> - continue
> >>> - fi
> >>> -
> >>> - local_ret=0
> >>> - cmd $abicheck -l $libname \
> >>> - -old ${hash1}/$name -new ${hash2}/$name || local_ret=$?
> >>> - if [ $local_ret != 0 ]; then
> >>> - log "NOTICE" "$abicheck returned $local_ret"
> >>> - ret=$local_ret
> >>> - list="$list $libname"
> >>> - fi
> >>> -done
> >>> -
> >>> -if [ $ret != 0 ]; then
> >>> - log "NOTICE" "ABI may be incompatible, check reports/logs for details."
> >>> - log "NOTICE" "Incompatible list: $list"
> >>> -else
> >>> - log "NOTICE" "No error detected, ABI is compatible."
> >>> -fi
> >>> -
> >>> -log "INFO" "Logs are in ${dst}/abi-check.log"
> >>> -log "INFO" "HTML reports are in ${dst}/compat_reports directory"
> >>> -
> >>> -exit $ret
> >>> diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst
> >>> index a21f4e7a4..219823923 100644
> >>> --- a/doc/guides/contributing/abi_versioning.rst
> >>> +++ b/doc/guides/contributing/abi_versioning.rst
> >>> @@ -482,41 +482,27 @@ Running the ABI Validator
> >>> -------------------------
> >>>
> >>> The ``devtools`` directory in the DPDK source tree contains a utility program,
> >>> -``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI
> >>> -Compliance Checker
> >>> -<http://ispras.linuxbase.org/index.php/ABI_compliance_checker>`_.
> >>> +``check-abi.sh``, for validating the DPDK ABI based on the libabigail `abidiff
> >>> +utility: <https://sourceware.org/libabigail/manual/abidiff.html>`_
> >>>
> >>> -This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper``
> >>> -utilities which can be installed via a package manager. For example::
> >>> +The syntax of the ``check-abi.sh`` utility is::
> >>>
> >>> - sudo yum install abi-compliance-checker
> >>> - sudo yum install abi-dumper
> >>> + ./devtools/check-abi.sh <refdir> <newdir>
> >>>
> >>> -The syntax of the ``validate-abi.sh`` utility is::
> >>> +Where <refdir> specifies the directory housing the reference build of dpdk, and
> >>> +<newdir> specifies the dpdk build directory to check the abi of
> >>>
> >>> - ./devtools/validate-abi.sh <REV1> <REV2>
> >>> +Example:
> >>> +To compare your build branch to the ABI of the master branch
> >>> +after you have built your branch
> >>>
> >>> -Where ``REV1`` and ``REV2`` are valid gitrevisions(7)
> >>> -https://www.kernel.org/pub/software/scm/git/docs/gitrevisions.html
> >>> -on the local repo.
> >>> +#. $ cd <path to dpdk src tree>
> >>> +#. $ mkdir ~/ref
> >>> +#. $ git clone --local --no-hardlinks --single-branch -b master . ~/ref
> >>> +#. $ cd ~/ref
> >>> +#. $ cp <path to dpdk src tree from above>/.config ./.config
> >>> +#. $ make defconfig
> >>> +#. $ make
> >>> +#. $ <path to dpdk src tree>/devtools/check-abi.sh ~/ref <path to dpdk src tree>
> >>>
> >>> -For example::
> >>> -
> >>> - # Check between the previous and latest commit:
> >>> - ./devtools/validate-abi.sh HEAD~1 HEAD
> >>> -
> >>> - # Check on a specific compilation target:
> >>> - ./devtools/validate-abi.sh -t x86_64-native-linux-gcc HEAD~1 HEAD
> >>> -
> >>> - # Check between two tags:
> >>> - ./devtools/validate-abi.sh v2.0.0 v2.1.0
> >>> -
> >>> - # Check between git master and local topic-branch "vhost-hacking":
> >>> - ./devtools/validate-abi.sh master vhost-hacking
> >>> -
> >>> -After the validation script completes (it can take a while since it need to
> >>> -compile both tags) it will create compatibility reports in the
> >>> -``./abi-check/compat_report`` directory. Listed incompatibilities can be found
> >>> -as follows::
> >>> -
> >>> - grep -lr Incompatible abi-check/compat_reports/
> >>> +
> >>>
> >>
> >> check-abi.sh appears to be backward step in terms of usability.
> >
> > No, check-abi.sh benefits from a nice integration in build scripts.
> > See below.
> >
> >> With validate-abi.sh I do can do a "validate-abi.sh HEAD~1 HEAD".
> >> And it will do the build, install, dump and comparison for me.
> >> And it picked up my 20.0.2 - > 21.0 changes no problem.
> >>
> >> With check-abi on the other hand, I need to the build and install myself.
> >> check-abi requires dump files, but I see no reference in the documentation to how these are created.
> >> It silently fails when it doesn't find any ...
> >>
> >> Do I run abi-dumper on the so's myself, or how does it work?
> >
> > check-abi.sh is integrated in test-build.sh and test-meson-builds.sh.
> > Probably we should document usage in these scripts.
> >
>
> ok in that case, this documentation should reference those scripts instead?
Maybe both?
No strong opinion.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
2020-04-17 10:33 3% ` Ray Kinsella
@ 2020-04-17 11:46 5% ` Trahe, Fiona
2020-04-17 16:01 0% ` Ray Kinsella
0 siblings, 1 reply; 200+ results
From: Trahe, Fiona @ 2020-04-17 11:46 UTC (permalink / raw)
To: Ray Kinsella, Thomas Monjalon, Richardson, Bruce
Cc: dev, Kusztal, ArkadiuszX, Neil Horman, Luca Boccassi,
Kevin Traynor, Yigit, Ferruh, Trahe, Fiona
Hi all,
> -----Original Message-----
> From: Ray Kinsella <mdr@ashroe.eu>
> Sent: Friday, April 17, 2020 11:34 AM
> To: Thomas Monjalon <thomas@monjalon.net>; Richardson, Bruce <bruce.richardson@intel.com>
> Cc: Trahe, Fiona <fiona.trahe@intel.com>; dev@dpdk.org; Kusztal, ArkadiuszX
> <arkadiuszx.kusztal@intel.com>; Neil Horman <nhorman@tuxdriver.com>; Luca Boccassi
> <bluca@debian.org>; Kevin Traynor <ktraynor@redhat.com>; Yigit, Ferruh <ferruh.yigit@intel.com>
> Subject: Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
>
>
>
> On 17/04/2020 11:17, Thomas Monjalon wrote:
> > 17/04/2020 11:42, Ray Kinsella:
> >> On 17/04/2020 10:31, Bruce Richardson wrote:
> >>> On Fri, Apr 17, 2020 at 08:24:30AM +0100, Ray Kinsella wrote:
> >>>> On 16/04/2020 11:01, Thomas Monjalon wrote:
> >>>>> 16/04/2020 11:51, Bruce Richardson:
> >>>>>> On Wed, Apr 15, 2020 at 06:24:19PM +0100, Trahe, Fiona wrote:
> >>>>>>> 5a. If in 20.05 we add a version of a fn which breaks ABI 20.0, what should the name of the
> original function be? fn_v20, or fn_v20.0
> >>>>>>
> >>>>>> In technical terms it really doesn't matter, it's just a name that will be
> >>>>>> looked up in a table. I don't think we strictly enforce the naming, so
> >>>>>> whatever is clearest is best. I'd suggest the former.
> >>>>>
> >>>>> Each release can have a new ABI.
> >>>>
> >>>> How many ABI's do we want to support?
> >>>>
> >>> It's not how many we want to support, but for me it's a matter of how many
> >>> do we need to support. If an API is part of the stable set, it can't just
> >>> drop to being experimental for one or two releases - it's always stable
> >>> until deprecated. We also shouldn't have a situation where release 20.08 is
> >>> ABI compatible with 19.11 but not 20.02 and 20.05.
> >>
> >> True. Let me say it differently.
> >>
> >> Our only commitment is to support v20 - 19.11
> >> However you are correct, if something gets committed as v21 in 20.02, in practise should also be
> there in 20.05+ also.
> >> Because if it is committed as v21 and not as experimental, it should not be changing once
> committed.
> >>
> >> In answering Thomas,
> >> I was more commenting on the proliferation of ABI numbers & symbols we need to track in the
> build.
> >> With v20, v21 & Experimental we need to keep track of 3.
> >> If we start allowing quarterly builds to have managed ABI's, it will get confusing.
> >
> > I don't remember why we are using intermediate ABI versions
> > between v20 and v21.
> > If we can use v21 for new ABI and make sure compatibility is maintained
> > between all versions from 19.11 to 20.08, I'm fine.
[Fiona] Here's a hypothetical case, but it illustrates why I don't think there
should be an expectation to maintain ABI compatibility here.
Example: in 20.05 add a new info_get_v21() which includes ChaChaPoly.
In 20.08 add another new algorithm. info_get_v21() return now includes this.
info_get_v21() will become stable in 20.11 and compatibility must be maintained from then on.
In the meantime, the fn is not experimental - that wouldn't be appropriate as it was a stable API.
But an app either wants stability and so should build against 19.11, or if prepared to move up to
one non-stable-ABI quarterly release should be willing to rebuild for the next non-stable-ABI quarterly release.
I think it's an unnecessary burden to require ABI compatibility across quarterly releases.
And if required could end up with the version tracking hassle Ray referred to above with fn versions
of 20.0.1, 20.0.2, 20.0.3, v21, and potentially several versions of same fn.
^ permalink raw reply [relevance 5%]
* Re: [dpdk-dev] [PATCHv3] Remove validate-abi.sh from tree
2020-04-17 10:20 8% ` Thomas Monjalon
2020-04-17 10:35 4% ` Ray Kinsella
@ 2020-04-17 11:47 4% ` Ray Kinsella
2020-04-17 12:10 4% ` Thomas Monjalon
1 sibling, 1 reply; 200+ results
From: Ray Kinsella @ 2020-04-17 11:47 UTC (permalink / raw)
To: Thomas Monjalon, Neil Horman; +Cc: dev, david.marchand
On 17/04/2020 11:20, Thomas Monjalon wrote:
> 17/04/2020 12:11, Ray Kinsella:
>>
>> On 16/04/2020 15:54, Neil Horman wrote:
>>> Since we've moved away from our initial validate-abi.sh script, in
>>> favor of check-abi.sh, which uses libabigail, remove the old script from
>>> the tree, and update the docs accordingly
>>>
>>> Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
>>> CC: thomas@monjalon.net
>>> CC: david.marchand@redhat.com
>>> CC: mdr@ashroe.eu
>>> ---
>>> MAINTAINERS | 1 -
>>> devtools/validate-abi.sh | 251 ---------------------
>>> doc/guides/contributing/abi_versioning.rst | 50 ++--
>>> 3 files changed, 18 insertions(+), 284 deletions(-)
>>> delete mode 100755 devtools/validate-abi.sh
>>>
>>> diff --git a/MAINTAINERS b/MAINTAINERS
>>> index 4800f6884..84b633431 100644
>>> --- a/MAINTAINERS
>>> +++ b/MAINTAINERS
>>> @@ -152,7 +152,6 @@ F: devtools/gen-abi.sh
>>> F: devtools/libabigail.abignore
>>> F: devtools/update-abi.sh
>>> F: devtools/update_version_map_abi.py
>>> -F: devtools/validate-abi.sh
>>> F: buildtools/check-experimental-syms.sh
>>> F: buildtools/map-list-symbol.sh
>>>
>>> diff --git a/devtools/validate-abi.sh b/devtools/validate-abi.sh
>>> deleted file mode 100755
>>> index f64e19d38..000000000
>>> --- a/devtools/validate-abi.sh
>>> +++ /dev/null
>>> @@ -1,251 +0,0 @@
>>> -#!/usr/bin/env bash
>>> -# SPDX-License-Identifier: BSD-3-Clause
>>> -# Copyright(c) 2015 Neil Horman. All rights reserved.
>>> -# Copyright(c) 2017 6WIND S.A.
>>> -# All rights reserved
>>> -
>>> -set -e
>>> -
>>> -abicheck=abi-compliance-checker
>>> -abidump=abi-dumper
>>> -default_dst=abi-check
>>> -default_target=x86_64-native-linuxapp-gcc
>>> -
>>> -# trap on error
>>> -err_report() {
>>> - echo "$0: error at line $1"
>>> -}
>>> -trap 'err_report $LINENO' ERR
>>> -
>>> -print_usage () {
>>> - cat <<- END_OF_HELP
>>> - $(basename $0) [options] <rev1> <rev2>
>>> -
>>> - This script compares the ABI of 2 git revisions of the current
>>> - workspace. The output is a html report and a compilation log.
>>> -
>>> - The objective is to make sure that applications built against
>>> - DSOs from the first revision can still run when executed using
>>> - the DSOs built from the second revision.
>>> -
>>> - <rev1> and <rev2> are git commit id or tags.
>>> -
>>> - Options:
>>> - -h show this help
>>> - -j <num> enable parallel compilation with <num> threads
>>> - -v show compilation logs on the console
>>> - -d <dir> change working directory (default is ${default_dst})
>>> - -t <target> the dpdk target to use (default is ${default_target})
>>> - -f overwrite existing files in destination directory
>>> -
>>> - The script returns 0 on success, or the value of last failing
>>> - call of ${abicheck} (incompatible abi or the tool has run with errors).
>>> - The errors returned by ${abidump} are ignored.
>>> -
>>> - END_OF_HELP
>>> -}
>>> -
>>> -# log in the file, and on stdout if verbose
>>> -# $1: level string
>>> -# $2: string to be logged
>>> -log() {
>>> - echo "$1: $2"
>>> - if [ "${verbose}" != "true" ]; then
>>> - echo "$1: $2" >&3
>>> - fi
>>> -}
>>> -
>>> -# launch a command and log it, taking care of surrounding spaces with quotes
>>> -cmd() {
>>> - local i s whitespace ret
>>> - s=""
>>> - whitespace="[[:space:]]"
>>> - for i in "$@"; do
>>> - if [[ $i =~ $whitespace ]]; then
>>> - i=\"$i\"
>>> - fi
>>> - if [ -z "$s" ]; then
>>> - s="$i"
>>> - else
>>> - s="$s $i"
>>> - fi
>>> - done
>>> -
>>> - ret=0
>>> - log "CMD" "$s"
>>> - "$@" || ret=$?
>>> - if [ "$ret" != "0" ]; then
>>> - log "CMD" "previous command returned $ret"
>>> - fi
>>> -
>>> - return $ret
>>> -}
>>> -
>>> -# redirect or copy stderr/stdout to a file
>>> -# the syntax is unfamiliar, but it makes the rest of the
>>> -# code easier to read, avoiding the use of pipes
>>> -set_log_file() {
>>> - # save original stdout and stderr in fd 3 and 4
>>> - exec 3>&1
>>> - exec 4>&2
>>> - # create a new fd 5 that send to a file
>>> - exec 5> >(cat > $1)
>>> - # send stdout and stderr to fd 5
>>> - if [ "${verbose}" = "true" ]; then
>>> - exec 1> >(tee /dev/fd/5 >&3)
>>> - exec 2> >(tee /dev/fd/5 >&4)
>>> - else
>>> - exec 1>&5
>>> - exec 2>&5
>>> - fi
>>> -}
>>> -
>>> -# Make sure we configure SHARED libraries
>>> -# Also turn off IGB and KNI as those require kernel headers to build
>>> -fixup_config() {
>>> - local conf=config/defconfig_$target
>>> - cmd sed -i -e"$ a\CONFIG_RTE_BUILD_SHARED_LIB=y" $conf
>>> - cmd sed -i -e"$ a\CONFIG_RTE_NEXT_ABI=n" $conf
>>> - cmd sed -i -e"$ a\CONFIG_RTE_EAL_IGB_UIO=n" $conf
>>> - cmd sed -i -e"$ a\CONFIG_RTE_LIBRTE_KNI=n" $conf
>>> - cmd sed -i -e"$ a\CONFIG_RTE_KNI_KMOD=n" $conf
>>> -}
>>> -
>>> -# build dpdk for the given tag and dump abi
>>> -# $1: hash of the revision
>>> -gen_abi() {
>>> - local i
>>> -
>>> - cmd git clone ${dpdkroot} ${dst}/${1}
>>> - cmd cd ${dst}/${1}
>>> -
>>> - log "INFO" "Checking out version ${1} of the dpdk"
>>> - # Move to the old version of the tree
>>> - cmd git checkout ${1}
>>> -
>>> - fixup_config
>>> -
>>> - # Now configure the build
>>> - log "INFO" "Configuring DPDK ${1}"
>>> - cmd make config T=$target O=$target
>>> -
>>> - # Checking abi compliance relies on using the dwarf information in
>>> - # the shared objects. Build with -g to include them.
>>> - log "INFO" "Building DPDK ${1}. This might take a moment"
>>> - cmd make -j$parallel O=$target V=1 EXTRA_CFLAGS="-g -Og -Wno-error" \
>>> - EXTRA_LDFLAGS="-g" || log "INFO" "The build failed"
>>> -
>>> - # Move to the lib directory
>>> - cmd cd ${PWD}/$target/lib
>>> - log "INFO" "Collecting ABI information for ${1}"
>>> - for i in *.so; do
>>> - [ -e "$i" ] || break
>>> - cmd $abidump ${i} -o $dst/${1}/${i}.dump -lver ${1} || true
>>> - # hack to ignore empty SymbolsInfo section (no public ABI)
>>> - if grep -q "'SymbolInfo' => {}," $dst/${1}/${i}.dump \
>>> - 2> /dev/null; then
>>> - log "INFO" "${i} has no public ABI, remove dump file"
>>> - cmd rm -f $dst/${1}/${i}.dump
>>> - fi
>>> - done
>>> -}
>>> -
>>> -verbose=false
>>> -parallel=1
>>> -dst=${default_dst}
>>> -target=${default_target}
>>> -force=0
>>> -while getopts j:vd:t:fh ARG ; do
>>> - case $ARG in
>>> - j ) parallel=$OPTARG ;;
>>> - v ) verbose=true ;;
>>> - d ) dst=$OPTARG ;;
>>> - t ) target=$OPTARG ;;
>>> - f ) force=1 ;;
>>> - h ) print_usage ; exit 0 ;;
>>> - ? ) print_usage ; exit 1 ;;
>>> - esac
>>> -done
>>> -shift $(($OPTIND - 1))
>>> -
>>> -if [ $# != 2 ]; then
>>> - print_usage
>>> - exit 1
>>> -fi
>>> -
>>> -tag1=$1
>>> -tag2=$2
>>> -
>>> -# convert path to absolute
>>> -case "${dst}" in
>>> - /*) ;;
>>> - *) dst=${PWD}/${dst} ;;
>>> -esac
>>> -dpdkroot=$(readlink -f $(dirname $0)/..)
>>> -
>>> -if [ -e "${dst}" -a "$force" = 0 ]; then
>>> - echo "The ${dst} directory is not empty. Remove it, use another"
>>> - echo "one (-d <dir>), or force overriding (-f)"
>>> - exit 1
>>> -fi
>>> -
>>> -rm -rf ${dst}
>>> -mkdir -p ${dst}
>>> -set_log_file ${dst}/abi-check.log
>>> -log "INFO" "Logs available in ${dst}/abi-check.log"
>>> -
>>> -command -v ${abicheck} || log "INFO" "Can't find ${abicheck} utility"
>>> -command -v ${abidump} || log "INFO" "Can't find ${abidump} utility"
>>> -
>>> -hash1=$(git show -s --format=%h "$tag1" -- 2> /dev/null | tail -1)
>>> -hash2=$(git show -s --format=%h "$tag2" -- 2> /dev/null | tail -1)
>>> -
>>> -# Make hashes available in output for non-local reference
>>> -tag1="$tag1 ($hash1)"
>>> -tag2="$tag2 ($hash2)"
>>> -
>>> -if [ "$hash1" = "$hash2" ]; then
>>> - log "ERROR" "$tag1 and $tag2 are the same revisions"
>>> - exit 1
>>> -fi
>>> -
>>> -cmd mkdir -p ${dst}
>>> -
>>> -# dump abi for each revision
>>> -gen_abi ${hash1}
>>> -gen_abi ${hash2}
>>> -
>>> -# compare the abi dumps
>>> -cmd cd ${dst}
>>> -ret=0
>>> -list=""
>>> -for i in ${hash2}/*.dump; do
>>> - name=`basename $i`
>>> - libname=${name%.dump}
>>> -
>>> - if [ ! -f ${hash1}/$name ]; then
>>> - log "INFO" "$NAME does not exist in $tag1. skipping..."
>>> - continue
>>> - fi
>>> -
>>> - local_ret=0
>>> - cmd $abicheck -l $libname \
>>> - -old ${hash1}/$name -new ${hash2}/$name || local_ret=$?
>>> - if [ $local_ret != 0 ]; then
>>> - log "NOTICE" "$abicheck returned $local_ret"
>>> - ret=$local_ret
>>> - list="$list $libname"
>>> - fi
>>> -done
>>> -
>>> -if [ $ret != 0 ]; then
>>> - log "NOTICE" "ABI may be incompatible, check reports/logs for details."
>>> - log "NOTICE" "Incompatible list: $list"
>>> -else
>>> - log "NOTICE" "No error detected, ABI is compatible."
>>> -fi
>>> -
>>> -log "INFO" "Logs are in ${dst}/abi-check.log"
>>> -log "INFO" "HTML reports are in ${dst}/compat_reports directory"
>>> -
>>> -exit $ret
>>> diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst
>>> index a21f4e7a4..219823923 100644
>>> --- a/doc/guides/contributing/abi_versioning.rst
>>> +++ b/doc/guides/contributing/abi_versioning.rst
>>> @@ -482,41 +482,27 @@ Running the ABI Validator
>>> -------------------------
>>>
>>> The ``devtools`` directory in the DPDK source tree contains a utility program,
>>> -``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI
>>> -Compliance Checker
>>> -<http://ispras.linuxbase.org/index.php/ABI_compliance_checker>`_.
>>> +``check-abi.sh``, for validating the DPDK ABI based on the libabigail `abidiff
>>> +utility: <https://sourceware.org/libabigail/manual/abidiff.html>`_
>>>
>>> -This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper``
>>> -utilities which can be installed via a package manager. For example::
>>> +The syntax of the ``check-abi.sh`` utility is::
>>>
>>> - sudo yum install abi-compliance-checker
>>> - sudo yum install abi-dumper
>>> + ./devtools/check-abi.sh <refdir> <newdir>
>>>
>>> -The syntax of the ``validate-abi.sh`` utility is::
>>> +Where <refdir> specifies the directory housing the reference build of dpdk, and
>>> +<newdir> specifies the dpdk build directory to check the abi of
>>>
>>> - ./devtools/validate-abi.sh <REV1> <REV2>
>>> +Example:
>>> +To compare your build branch to the ABI of the master branch
>>> +after you have built your branch
>>>
>>> -Where ``REV1`` and ``REV2`` are valid gitrevisions(7)
>>> -https://www.kernel.org/pub/software/scm/git/docs/gitrevisions.html
>>> -on the local repo.
>>> +#. $ cd <path to dpdk src tree>
>>> +#. $ mkdir ~/ref
>>> +#. $ git clone --local --no-hardlinks --single-branch -b master . ~/ref
>>> +#. $ cd ~/ref
>>> +#. $ cp <path to dpdk src tree from above>/.config ./.config
>>> +#. $ make defconfig
>>> +#. $ make
>>> +#. $ <path to dpdk src tree>/devtools/check-abi.sh ~/ref <path to dpdk src tree>
>>>
>>> -For example::
>>> -
>>> - # Check between the previous and latest commit:
>>> - ./devtools/validate-abi.sh HEAD~1 HEAD
>>> -
>>> - # Check on a specific compilation target:
>>> - ./devtools/validate-abi.sh -t x86_64-native-linux-gcc HEAD~1 HEAD
>>> -
>>> - # Check between two tags:
>>> - ./devtools/validate-abi.sh v2.0.0 v2.1.0
>>> -
>>> - # Check between git master and local topic-branch "vhost-hacking":
>>> - ./devtools/validate-abi.sh master vhost-hacking
>>> -
>>> -After the validation script completes (it can take a while since it need to
>>> -compile both tags) it will create compatibility reports in the
>>> -``./abi-check/compat_report`` directory. Listed incompatibilities can be found
>>> -as follows::
>>> -
>>> - grep -lr Incompatible abi-check/compat_reports/
>>> +
>>>
>>
>> check-abi.sh appears to be backward step in terms of usability.
>
> No, check-abi.sh benefits from a nice integration in build scripts.
> See below.
>
>> With validate-abi.sh I do can do a "validate-abi.sh HEAD~1 HEAD".
>> And it will do the build, install, dump and comparison for me.
>> And it picked up my 20.0.2 - > 21.0 changes no problem.
>>
>> With check-abi on the other hand, I need to the build and install myself.
>> check-abi requires dump files, but I see no reference in the documentation to how these are created.
>> It silently fails when it doesn't find any ...
>>
>> Do I run abi-dumper on the so's myself, or how does it work?
>
> check-abi.sh is integrated in test-build.sh and test-meson-builds.sh.
> Probably we should document usage in these scripts.
>
Looks like I need to set DPDK_ABI_REF_VERSION=master, not obvious.
Any tips or tricks would be welcome.
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCHv3] Remove validate-abi.sh from tree
2020-04-17 11:47 4% ` Ray Kinsella
@ 2020-04-17 12:10 4% ` Thomas Monjalon
2020-04-17 15:42 9% ` Ray Kinsella
0 siblings, 1 reply; 200+ results
From: Thomas Monjalon @ 2020-04-17 12:10 UTC (permalink / raw)
To: Neil Horman, Ray Kinsella; +Cc: dev, david.marchand
17/04/2020 13:47, Ray Kinsella:
> On 17/04/2020 11:20, Thomas Monjalon wrote:
> > 17/04/2020 12:11, Ray Kinsella:
> >> check-abi.sh appears to be backward step in terms of usability.
> >
> > No, check-abi.sh benefits from a nice integration in build scripts.
> > See below.
> >
> >> With validate-abi.sh I do can do a "validate-abi.sh HEAD~1 HEAD".
> >> And it will do the build, install, dump and comparison for me.
> >> And it picked up my 20.0.2 - > 21.0 changes no problem.
> >>
> >> With check-abi on the other hand, I need to the build and install myself.
> >> check-abi requires dump files, but I see no reference in the documentation to how these are created.
> >> It silently fails when it doesn't find any ...
> >>
> >> Do I run abi-dumper on the so's myself, or how does it work?
> >
> > check-abi.sh is integrated in test-build.sh and test-meson-builds.sh.
> > Probably we should document usage in these scripts.
>
> Looks like I need to set DPDK_ABI_REF_VERSION=master, not obvious.
> Any tips or tricks would be welcome.
export DPDK_ABI_REF_VERSION=v20.02
or
export DPDK_ABI_REF_VERSION=v19.11
Depends on which compatibility you want to test...
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH v4 0/9] New sync modes for ring
2020-04-03 17:42 3% ` [dpdk-dev] [PATCH v3 0/9] New sync modes for ring Konstantin Ananyev
2020-04-03 17:42 1% ` [dpdk-dev] [PATCH v3 3/9] ring: introduce RTS ring mode Konstantin Ananyev
@ 2020-04-17 13:36 3% ` Konstantin Ananyev
2020-04-17 13:36 9% ` [dpdk-dev] [PATCH v4 2/9] ring: prepare ring to allow new sync schemes Konstantin Ananyev
` (2 more replies)
2 siblings, 3 replies; 200+ results
From: Konstantin Ananyev @ 2020-04-17 13:36 UTC (permalink / raw)
To: dev; +Cc: honnappa.nagarahalli, david.marchand, jielong.zjl, Konstantin Ananyev
V3 - V4 changes:
Address comments from Honnappa:
1. for new sync modes make legacy API wrappers around _elem_ calls
2. remove rte_ring_(hts|rts)_generic.h
3. few changes in C11 version
4. peek API - add missing functions for _elem_
5. remove _IS_SP/_IS_MP, etc. internal macros
6. fix param types (obj_table) for _elem_functions
7. fix formal API comments
8. deduplicate code for test_ring_stress
9. added functional tests for new sync modes
V2 - V3 changes:
1. Few more compilation fixes (for gcc 4.8.X)
2. Extra update devtools/libabigail.abignore (workaround)
V1 - V2 changes:
1. Fix compilation issues
2. Add C11 atomics support
3. Updates devtools/libabigail.abignore (workaround)
RFC - V1 changes:
1. remove ABI brekage (at least I hope I did)
2. Add support for ring_elem
3. Rework peek related API a bit
4. Rework test to make it less verbose and unite all test-cases
in one command
5. Add new test-case for MT peek API
TODO list:
1. Update docs
These days more and more customers use(/try to use) DPDK based apps within
overcommitted systems (multiple acttive threads over same pysical cores):
VM, container deployments, etc.
One quite common problem they hit:
Lock-Holder-Preemption/Lock-Waiter-Preemption with rte_ring.
LHP is quite a common problem for spin-based sync primitives
(spin-locks, etc.) on overcommitted systems.
The situation gets much worse when some sort of
fair-locking technique is used (ticket-lock, etc.).
As now not only lock-owner but also lock-waiters scheduling
order matters a lot (LWP).
These two problems are well-known for kernel within VMs:
http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
The problem with rte_ring is that while head accusion is sort of
un-fair locking, waiting on tail is very similar to ticket lock schema -
tail has to be updated in particular order.
That makes current rte_ring implementation to perform
really pure on some overcommited scenarios.
It is probably not possible to completely resolve LHP problem in
userspace only (without some kernel communication/intervention).
But removing fairness at tail update helps to avoid LWP and
can mitigate the situation significantly.
This patch proposes two new optional ring synchronization modes:
1) Head/Tail Sync (HTS) mode
In that mode enqueue/dequeue operation is fully serialized:
only one thread at a time is allowed to perform given op.
As another enhancement provide ability to split enqueue/dequeue
operation into two phases:
- enqueue/dequeue start
- enqueue/dequeue finish
That allows user to inspect objects in the ring without removing
them from it (aka MT safe peek).
2) Relaxed Tail Sync (RTS)
The main difference from original MP/MC algorithm is that
tail value is increased not by every thread that finished enqueue/dequeue,
but only by the last one.
That allows threads to avoid spinning on ring tail value,
leaving actual tail value change to the last thread in the update queue.
Note that these new sync modes are optional.
For current rte_ring users nothing should change
(both in terms of API/ABI and performance).
Existing sync modes MP/MC,SP/SC kept untouched, set up in the same
way (via flags and _init_), and MP/MC remains as default one.
The only thing that changed:
Format of prod/cons now could differ depending on mode selected at _init_.
So user has to stick with one sync model through whole ring lifetime.
In other words, user can't create a ring for let say SP mode and then
in the middle of data-path change his mind and start using MP_RTS mode.
For existing modes (SP/MP, SC/MC) format remains the same and
user can still use them interchangeably, though of course it is an
error prone practice.
Test results on IA (see below) show significant improvements
for average enqueue/dequeue op times on overcommitted systems.
For 'classic' DPDK deployments (one thread per core) original MP/MC
algorithm still shows best numbers, though for 64-bit target
RTS numbers are not that far away.
Numbers were produced by new UT test-case: ring_stress_autotest, i.e.:
echo ring_stress_autotest | ./dpdk-test -n 4 --lcores='...'
X86_64 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
DEQ+ENQ average cycles/obj
MP/MC HTS RTS
1thread@1core(--lcores=6-7) 8.00 8.15 8.99
2thread@2core(--lcores=6-8) 19.14 19.61 20.35
4thread@4core(--lcores=6-10) 29.43 29.79 31.82
8thread@8core(--lcores=6-14) 110.59 192.81 119.50
16thread@16core(--lcores=6-22) 461.03 813.12 495.59
32thread/@32core(--lcores='6-22,55-70') 982.90 1972.38 1160.51
2thread@1core(--lcores='6,(10-11)@7' 20140.50 23.58 25.14
4thread@2core(--lcores='6,(10-11)@7,(20-21)@8' 153680.60 76.88 80.05
8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 280314.32 294.72 318.79
16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 643176.59 1144.02 1175.14
32thread@2core(--lcores='6,(10-25)@7,(30-45)@8' 4264238.80 4627.48 4892.68
8thread@2core(--lcores='6,(10-17)@(7,8))' 321085.98 298.59 307.47
16thread@4core(--lcores='6,(20-35)@(7-10))' 1900705.61 575.35 678.29
32thread@4core(--lcores='6,(20-51)@(7-10))' 5510445.85 2164.36 2714.12
i686 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
DEQ+ENQ average cycles/obj
MP/MC HTS RTS
1thread@1core(--lcores=6-7) 7.85 12.13 11.31
2thread@2core(--lcores=6-8) 17.89 24.52 21.86
8thread@8core(--lcores=6-14) 32.58 354.20 54.58
32thread/@32core(--lcores='6-22,55-70') 813.77 6072.41 2169.91
2thread@1core(--lcores='6,(10-11)@7' 16095.00 36.06 34.74
8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 1140354.54 346.61 361.57
16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 1920417.86 1314.90 1416.65
8thread@2core(--lcores='6,(10-17)@(7,8))' 594358.61 332.70 357.74
32thread@4core(--lcores='6,(20-51)@(7-10))' 5319896.86 2836.44 3028.87
Konstantin Ananyev (9):
test/ring: add contention stress test
ring: prepare ring to allow new sync schemes
ring: introduce RTS ring mode
test/ring: add contention stress test for RTS ring
ring: introduce HTS ring mode
test/ring: add contention stress test for HTS ring
ring: introduce peek style API
test/ring: add stress test for MT peek API
test/ring: add functional tests for new sync modes
app/test/Makefile | 5 +
app/test/meson.build | 5 +
app/test/test_pdump.c | 6 +-
app/test/test_ring.c | 93 ++++--
app/test/test_ring_hts_stress.c | 32 ++
app/test/test_ring_mpmc_stress.c | 31 ++
app/test/test_ring_peek_stress.c | 43 +++
app/test/test_ring_rts_stress.c | 32 ++
app/test/test_ring_stress.c | 57 ++++
app/test/test_ring_stress.h | 38 +++
app/test/test_ring_stress_impl.h | 396 ++++++++++++++++++++++
devtools/libabigail.abignore | 7 +
lib/librte_pdump/rte_pdump.c | 2 +-
lib/librte_port/rte_port_ring.c | 12 +-
lib/librte_ring/Makefile | 8 +-
lib/librte_ring/meson.build | 11 +-
lib/librte_ring/rte_ring.c | 114 ++++++-
lib/librte_ring/rte_ring.h | 243 ++++++++------
lib/librte_ring/rte_ring_c11_mem.h | 44 +++
lib/librte_ring/rte_ring_core.h | 181 ++++++++++
lib/librte_ring/rte_ring_elem.h | 141 ++++++--
lib/librte_ring/rte_ring_generic.h | 48 +++
lib/librte_ring/rte_ring_hts.h | 332 ++++++++++++++++++
lib/librte_ring/rte_ring_hts_c11_mem.h | 207 ++++++++++++
lib/librte_ring/rte_ring_peek.h | 446 +++++++++++++++++++++++++
lib/librte_ring/rte_ring_rts.h | 439 ++++++++++++++++++++++++
lib/librte_ring/rte_ring_rts_c11_mem.h | 179 ++++++++++
27 files changed, 2978 insertions(+), 174 deletions(-)
create mode 100644 app/test/test_ring_hts_stress.c
create mode 100644 app/test/test_ring_mpmc_stress.c
create mode 100644 app/test/test_ring_peek_stress.c
create mode 100644 app/test/test_ring_rts_stress.c
create mode 100644 app/test/test_ring_stress.c
create mode 100644 app/test/test_ring_stress.h
create mode 100644 app/test/test_ring_stress_impl.h
create mode 100644 lib/librte_ring/rte_ring_core.h
create mode 100644 lib/librte_ring/rte_ring_hts.h
create mode 100644 lib/librte_ring/rte_ring_hts_c11_mem.h
create mode 100644 lib/librte_ring/rte_ring_peek.h
create mode 100644 lib/librte_ring/rte_ring_rts.h
create mode 100644 lib/librte_ring/rte_ring_rts_c11_mem.h
--
2.17.1
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v4 2/9] ring: prepare ring to allow new sync schemes
2020-04-17 13:36 3% ` [dpdk-dev] [PATCH v4 0/9] New sync modes for ring Konstantin Ananyev
@ 2020-04-17 13:36 9% ` Konstantin Ananyev
2020-04-17 13:36 1% ` [dpdk-dev] [PATCH v4 3/9] ring: introduce RTS ring mode Konstantin Ananyev
2020-04-18 16:32 3% ` [dpdk-dev] [PATCH v5 0/9] New sync modes for ring Konstantin Ananyev
2 siblings, 0 replies; 200+ results
From: Konstantin Ananyev @ 2020-04-17 13:36 UTC (permalink / raw)
To: dev; +Cc: honnappa.nagarahalli, david.marchand, jielong.zjl, Konstantin Ananyev
To make these preparations two main things are done:
- Change from *single* to *sync_type* to allow different
synchronisation schemes to be applied.
Mark *single* as deprecated in comments.
Add new functions to allow user to query ring sync types.
Replace direct access to *single* with appropriate function call.
- Move actual rte_ring and related structures definitions into a
separate file: <rte_ring_core.h>. It allows to refer contents
of <rte_ring_elem.h> from <rte_ring.h> without introducing a
circular dependency.
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
app/test/test_pdump.c | 6 +-
lib/librte_pdump/rte_pdump.c | 2 +-
lib/librte_port/rte_port_ring.c | 12 +--
lib/librte_ring/Makefile | 1 +
lib/librte_ring/meson.build | 1 +
lib/librte_ring/rte_ring.c | 6 +-
lib/librte_ring/rte_ring.h | 170 ++++++++++++++------------------
lib/librte_ring/rte_ring_core.h | 131 ++++++++++++++++++++++++
lib/librte_ring/rte_ring_elem.h | 42 +++-----
9 files changed, 233 insertions(+), 138 deletions(-)
create mode 100644 lib/librte_ring/rte_ring_core.h
diff --git a/app/test/test_pdump.c b/app/test/test_pdump.c
index ad183184c..6a1180bcb 100644
--- a/app/test/test_pdump.c
+++ b/app/test/test_pdump.c
@@ -57,8 +57,7 @@ run_pdump_client_tests(void)
if (ret < 0)
return -1;
mp->flags = 0x0000;
- ring_client = rte_ring_create("SR0", RING_SIZE, rte_socket_id(),
- RING_F_SP_ENQ | RING_F_SC_DEQ);
+ ring_client = rte_ring_create("SR0", RING_SIZE, rte_socket_id(), 0);
if (ring_client == NULL) {
printf("rte_ring_create SR0 failed");
return -1;
@@ -71,9 +70,6 @@ run_pdump_client_tests(void)
}
rte_eth_dev_probing_finish(eth_dev);
- ring_client->prod.single = 0;
- ring_client->cons.single = 0;
-
printf("\n***** flags = RTE_PDUMP_FLAG_TX *****\n");
for (itr = 0; itr < NUM_ITR; itr++) {
diff --git a/lib/librte_pdump/rte_pdump.c b/lib/librte_pdump/rte_pdump.c
index 8a01ac510..f96709f95 100644
--- a/lib/librte_pdump/rte_pdump.c
+++ b/lib/librte_pdump/rte_pdump.c
@@ -380,7 +380,7 @@ pdump_validate_ring_mp(struct rte_ring *ring, struct rte_mempool *mp)
rte_errno = EINVAL;
return -1;
}
- if (ring->prod.single || ring->cons.single) {
+ if (rte_ring_is_prod_single(ring) || rte_ring_is_cons_single(ring)) {
PDUMP_LOG(ERR, "ring with either SP or SC settings"
" is not valid for pdump, should have MP and MC settings\n");
rte_errno = EINVAL;
diff --git a/lib/librte_port/rte_port_ring.c b/lib/librte_port/rte_port_ring.c
index 47fcdd06a..52b2d8e55 100644
--- a/lib/librte_port/rte_port_ring.c
+++ b/lib/librte_port/rte_port_ring.c
@@ -44,8 +44,8 @@ rte_port_ring_reader_create_internal(void *params, int socket_id,
/* Check input parameters */
if ((conf == NULL) ||
(conf->ring == NULL) ||
- (conf->ring->cons.single && is_multi) ||
- (!(conf->ring->cons.single) && !is_multi)) {
+ (rte_ring_is_cons_single(conf->ring) && is_multi) ||
+ (!rte_ring_is_cons_single(conf->ring) && !is_multi)) {
RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
return NULL;
}
@@ -171,8 +171,8 @@ rte_port_ring_writer_create_internal(void *params, int socket_id,
/* Check input parameters */
if ((conf == NULL) ||
(conf->ring == NULL) ||
- (conf->ring->prod.single && is_multi) ||
- (!(conf->ring->prod.single) && !is_multi) ||
+ (rte_ring_is_prod_single(conf->ring) && is_multi) ||
+ (!rte_ring_is_prod_single(conf->ring) && !is_multi) ||
(conf->tx_burst_sz > RTE_PORT_IN_BURST_SIZE_MAX)) {
RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
return NULL;
@@ -440,8 +440,8 @@ rte_port_ring_writer_nodrop_create_internal(void *params, int socket_id,
/* Check input parameters */
if ((conf == NULL) ||
(conf->ring == NULL) ||
- (conf->ring->prod.single && is_multi) ||
- (!(conf->ring->prod.single) && !is_multi) ||
+ (rte_ring_is_prod_single(conf->ring) && is_multi) ||
+ (!rte_ring_is_prod_single(conf->ring) && !is_multi) ||
(conf->tx_burst_sz > RTE_PORT_IN_BURST_SIZE_MAX)) {
RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
return NULL;
diff --git a/lib/librte_ring/Makefile b/lib/librte_ring/Makefile
index 28368e6d1..6572768c9 100644
--- a/lib/librte_ring/Makefile
+++ b/lib/librte_ring/Makefile
@@ -16,6 +16,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_RING) := rte_ring.c
# install includes
SYMLINK-$(CONFIG_RTE_LIBRTE_RING)-include := rte_ring.h \
+ rte_ring_core.h \
rte_ring_elem.h \
rte_ring_generic.h \
rte_ring_c11_mem.h
diff --git a/lib/librte_ring/meson.build b/lib/librte_ring/meson.build
index 05402e4f0..c656781da 100644
--- a/lib/librte_ring/meson.build
+++ b/lib/librte_ring/meson.build
@@ -3,6 +3,7 @@
sources = files('rte_ring.c')
headers = files('rte_ring.h',
+ 'rte_ring_core.h',
'rte_ring_elem.h',
'rte_ring_c11_mem.h',
'rte_ring_generic.h')
diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c
index 77e5de099..fa5733907 100644
--- a/lib/librte_ring/rte_ring.c
+++ b/lib/librte_ring/rte_ring.c
@@ -106,8 +106,10 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
if (ret < 0 || ret >= (int)sizeof(r->name))
return -ENAMETOOLONG;
r->flags = flags;
- r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
- r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
+ r->prod.sync_type = (flags & RING_F_SP_ENQ) ?
+ RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
+ r->cons.sync_type = (flags & RING_F_SC_DEQ) ?
+ RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
if (flags & RING_F_EXACT_SZ) {
r->size = rte_align32pow2(count + 1);
diff --git a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h
index 18fc5d845..35ee4491c 100644
--- a/lib/librte_ring/rte_ring.h
+++ b/lib/librte_ring/rte_ring.h
@@ -36,91 +36,7 @@
extern "C" {
#endif
-#include <stdio.h>
-#include <stdint.h>
-#include <sys/queue.h>
-#include <errno.h>
-#include <rte_common.h>
-#include <rte_config.h>
-#include <rte_memory.h>
-#include <rte_lcore.h>
-#include <rte_atomic.h>
-#include <rte_branch_prediction.h>
-#include <rte_memzone.h>
-#include <rte_pause.h>
-
-#define RTE_TAILQ_RING_NAME "RTE_RING"
-
-enum rte_ring_queue_behavior {
- RTE_RING_QUEUE_FIXED = 0, /* Enq/Deq a fixed number of items from a ring */
- RTE_RING_QUEUE_VARIABLE /* Enq/Deq as many items as possible from ring */
-};
-
-#define RTE_RING_MZ_PREFIX "RG_"
-/** The maximum length of a ring name. */
-#define RTE_RING_NAMESIZE (RTE_MEMZONE_NAMESIZE - \
- sizeof(RTE_RING_MZ_PREFIX) + 1)
-
-/* structure to hold a pair of head/tail values and other metadata */
-struct rte_ring_headtail {
- volatile uint32_t head; /**< Prod/consumer head. */
- volatile uint32_t tail; /**< Prod/consumer tail. */
- uint32_t single; /**< True if single prod/cons */
-};
-
-/**
- * An RTE ring structure.
- *
- * The producer and the consumer have a head and a tail index. The particularity
- * of these index is that they are not between 0 and size(ring). These indexes
- * are between 0 and 2^32, and we mask their value when we access the ring[]
- * field. Thanks to this assumption, we can do subtractions between 2 index
- * values in a modulo-32bit base: that's why the overflow of the indexes is not
- * a problem.
- */
-struct rte_ring {
- /*
- * Note: this field kept the RTE_MEMZONE_NAMESIZE size due to ABI
- * compatibility requirements, it could be changed to RTE_RING_NAMESIZE
- * next time the ABI changes
- */
- char name[RTE_MEMZONE_NAMESIZE] __rte_cache_aligned; /**< Name of the ring. */
- int flags; /**< Flags supplied at creation. */
- const struct rte_memzone *memzone;
- /**< Memzone, if any, containing the rte_ring */
- uint32_t size; /**< Size of ring. */
- uint32_t mask; /**< Mask (size-1) of ring. */
- uint32_t capacity; /**< Usable size of ring */
-
- char pad0 __rte_cache_aligned; /**< empty cache line */
-
- /** Ring producer status. */
- struct rte_ring_headtail prod __rte_cache_aligned;
- char pad1 __rte_cache_aligned; /**< empty cache line */
-
- /** Ring consumer status. */
- struct rte_ring_headtail cons __rte_cache_aligned;
- char pad2 __rte_cache_aligned; /**< empty cache line */
-};
-
-#define RING_F_SP_ENQ 0x0001 /**< The default enqueue is "single-producer". */
-#define RING_F_SC_DEQ 0x0002 /**< The default dequeue is "single-consumer". */
-/**
- * Ring is to hold exactly requested number of entries.
- * Without this flag set, the ring size requested must be a power of 2, and the
- * usable space will be that size - 1. With the flag, the requested size will
- * be rounded up to the next power of two, but the usable space will be exactly
- * that requested. Worst case, if a power-of-2 size is requested, half the
- * ring space will be wasted.
- */
-#define RING_F_EXACT_SZ 0x0004
-#define RTE_RING_SZ_MASK (0x7fffffffU) /**< Ring size mask */
-
-/* @internal defines for passing to the enqueue dequeue worker functions */
-#define __IS_SP 1
-#define __IS_MP 0
-#define __IS_SC 1
-#define __IS_MC 0
+#include <rte_ring_core.h>
/**
* Calculate the memory size needed for a ring
@@ -420,7 +336,7 @@ rte_ring_mp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- __IS_MP, free_space);
+ RTE_RING_SYNC_MT, free_space);
}
/**
@@ -443,9 +359,13 @@ rte_ring_sp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- __IS_SP, free_space);
+ RTE_RING_SYNC_ST, free_space);
}
+#ifdef ALLOW_EXPERIMENTAL_API
+#include <rte_ring_elem.h>
+#endif
+
/**
* Enqueue several objects on a ring.
*
@@ -470,7 +390,7 @@ rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- r->prod.single, free_space);
+ r->prod.sync_type, free_space);
}
/**
@@ -554,7 +474,7 @@ rte_ring_mc_dequeue_bulk(struct rte_ring *r, void **obj_table,
unsigned int n, unsigned int *available)
{
return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- __IS_MC, available);
+ RTE_RING_SYNC_MT, available);
}
/**
@@ -578,7 +498,7 @@ rte_ring_sc_dequeue_bulk(struct rte_ring *r, void **obj_table,
unsigned int n, unsigned int *available)
{
return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- __IS_SC, available);
+ RTE_RING_SYNC_ST, available);
}
/**
@@ -605,7 +525,7 @@ rte_ring_dequeue_bulk(struct rte_ring *r, void **obj_table, unsigned int n,
unsigned int *available)
{
return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- r->cons.single, available);
+ r->cons.sync_type, available);
}
/**
@@ -777,6 +697,62 @@ rte_ring_get_capacity(const struct rte_ring *r)
return r->capacity;
}
+/**
+ * Return sync type used by producer in the ring.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * Producer sync type value.
+ */
+static inline enum rte_ring_sync_type
+rte_ring_get_prod_sync_type(const struct rte_ring *r)
+{
+ return r->prod.sync_type;
+}
+
+/**
+ * Check is the ring for single producer.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * true if ring is SP, zero otherwise.
+ */
+static inline int
+rte_ring_is_prod_single(const struct rte_ring *r)
+{
+ return (rte_ring_get_prod_sync_type(r) == RTE_RING_SYNC_ST);
+}
+
+/**
+ * Return sync type used by consumer in the ring.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * Consumer sync type value.
+ */
+static inline enum rte_ring_sync_type
+rte_ring_get_cons_sync_type(const struct rte_ring *r)
+{
+ return r->cons.sync_type;
+}
+
+/**
+ * Check is the ring for single consumer.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * true if ring is SC, zero otherwise.
+ */
+static inline int
+rte_ring_is_cons_single(const struct rte_ring *r)
+{
+ return (rte_ring_get_cons_sync_type(r) == RTE_RING_SYNC_ST);
+}
+
/**
* Dump the status of all rings on the console
*
@@ -820,7 +796,7 @@ rte_ring_mp_enqueue_burst(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue(r, obj_table, n,
- RTE_RING_QUEUE_VARIABLE, __IS_MP, free_space);
+ RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_MT, free_space);
}
/**
@@ -843,7 +819,7 @@ rte_ring_sp_enqueue_burst(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue(r, obj_table, n,
- RTE_RING_QUEUE_VARIABLE, __IS_SP, free_space);
+ RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_ST, free_space);
}
/**
@@ -870,7 +846,7 @@ rte_ring_enqueue_burst(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_VARIABLE,
- r->prod.single, free_space);
+ r->prod.sync_type, free_space);
}
/**
@@ -898,7 +874,7 @@ rte_ring_mc_dequeue_burst(struct rte_ring *r, void **obj_table,
unsigned int n, unsigned int *available)
{
return __rte_ring_do_dequeue(r, obj_table, n,
- RTE_RING_QUEUE_VARIABLE, __IS_MC, available);
+ RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_MT, available);
}
/**
@@ -923,7 +899,7 @@ rte_ring_sc_dequeue_burst(struct rte_ring *r, void **obj_table,
unsigned int n, unsigned int *available)
{
return __rte_ring_do_dequeue(r, obj_table, n,
- RTE_RING_QUEUE_VARIABLE, __IS_SC, available);
+ RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_ST, available);
}
/**
@@ -951,7 +927,7 @@ rte_ring_dequeue_burst(struct rte_ring *r, void **obj_table,
{
return __rte_ring_do_dequeue(r, obj_table, n,
RTE_RING_QUEUE_VARIABLE,
- r->cons.single, available);
+ r->cons.sync_type, available);
}
#ifdef __cplusplus
diff --git a/lib/librte_ring/rte_ring_core.h b/lib/librte_ring/rte_ring_core.h
new file mode 100644
index 000000000..459a0ffa1
--- /dev/null
+++ b/lib/librte_ring/rte_ring_core.h
@@ -0,0 +1,131 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2010-2020 Intel Corporation
+ * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#ifndef _RTE_RING_CORE_H_
+#define _RTE_RING_CORE_H_
+
+/**
+ * @file
+ * This file contains definion of RTE ring structure itself,
+ * init flags and some related macros.
+ * For majority of DPDK entities, it is not recommended to include
+ * this file directly, use include <rte_ring.h> or <rte_ring_elem.h>
+ * instead.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdio.h>
+#include <stdint.h>
+#include <string.h>
+#include <sys/queue.h>
+#include <errno.h>
+#include <rte_common.h>
+#include <rte_config.h>
+#include <rte_memory.h>
+#include <rte_lcore.h>
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_memzone.h>
+#include <rte_pause.h>
+#include <rte_debug.h>
+
+#define RTE_TAILQ_RING_NAME "RTE_RING"
+
+enum rte_ring_queue_behavior {
+ /** Enq/Deq a fixed number of items from a ring */
+ RTE_RING_QUEUE_FIXED = 0,
+ /** Enq/Deq as many items as possible from ring */
+ RTE_RING_QUEUE_VARIABLE
+};
+
+#define RTE_RING_MZ_PREFIX "RG_"
+/** The maximum length of a ring name. */
+#define RTE_RING_NAMESIZE (RTE_MEMZONE_NAMESIZE - \
+ sizeof(RTE_RING_MZ_PREFIX) + 1)
+
+/** prod/cons sync types */
+enum rte_ring_sync_type {
+ RTE_RING_SYNC_MT, /**< multi-thread safe (default mode) */
+ RTE_RING_SYNC_ST, /**< single thread only */
+};
+
+/**
+ * structures to hold a pair of head/tail values and other metadata.
+ * Depending on sync_type format of that structure might be different,
+ * but offset for *sync_type* and *tail* values should remain the same.
+ */
+struct rte_ring_headtail {
+ volatile uint32_t head; /**< prod/consumer head. */
+ volatile uint32_t tail; /**< prod/consumer tail. */
+ RTE_STD_C11
+ union {
+ /** sync type of prod/cons */
+ enum rte_ring_sync_type sync_type;
+ /** deprecated - True if single prod/cons */
+ uint32_t single;
+ };
+};
+
+/**
+ * An RTE ring structure.
+ *
+ * The producer and the consumer have a head and a tail index. The particularity
+ * of these index is that they are not between 0 and size(ring). These indexes
+ * are between 0 and 2^32, and we mask their value when we access the ring[]
+ * field. Thanks to this assumption, we can do subtractions between 2 index
+ * values in a modulo-32bit base: that's why the overflow of the indexes is not
+ * a problem.
+ */
+struct rte_ring {
+ /*
+ * Note: this field kept the RTE_MEMZONE_NAMESIZE size due to ABI
+ * compatibility requirements, it could be changed to RTE_RING_NAMESIZE
+ * next time the ABI changes
+ */
+ char name[RTE_MEMZONE_NAMESIZE] __rte_cache_aligned;
+ /**< Name of the ring. */
+ int flags; /**< Flags supplied at creation. */
+ const struct rte_memzone *memzone;
+ /**< Memzone, if any, containing the rte_ring */
+ uint32_t size; /**< Size of ring. */
+ uint32_t mask; /**< Mask (size-1) of ring. */
+ uint32_t capacity; /**< Usable size of ring */
+
+ char pad0 __rte_cache_aligned; /**< empty cache line */
+
+ /** Ring producer status. */
+ struct rte_ring_headtail prod __rte_cache_aligned;
+ char pad1 __rte_cache_aligned; /**< empty cache line */
+
+ /** Ring consumer status. */
+ struct rte_ring_headtail cons __rte_cache_aligned;
+ char pad2 __rte_cache_aligned; /**< empty cache line */
+};
+
+#define RING_F_SP_ENQ 0x0001 /**< The default enqueue is "single-producer". */
+#define RING_F_SC_DEQ 0x0002 /**< The default dequeue is "single-consumer". */
+/**
+ * Ring is to hold exactly requested number of entries.
+ * Without this flag set, the ring size requested must be a power of 2, and the
+ * usable space will be that size - 1. With the flag, the requested size will
+ * be rounded up to the next power of two, but the usable space will be exactly
+ * that requested. Worst case, if a power-of-2 size is requested, half the
+ * ring space will be wasted.
+ */
+#define RING_F_EXACT_SZ 0x0004
+#define RTE_RING_SZ_MASK (0x7fffffffU) /**< Ring size mask */
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_RING_CORE_H_ */
diff --git a/lib/librte_ring/rte_ring_elem.h b/lib/librte_ring/rte_ring_elem.h
index 663addc73..7406c0b0f 100644
--- a/lib/librte_ring/rte_ring_elem.h
+++ b/lib/librte_ring/rte_ring_elem.h
@@ -20,21 +20,7 @@
extern "C" {
#endif
-#include <stdio.h>
-#include <stdint.h>
-#include <string.h>
-#include <sys/queue.h>
-#include <errno.h>
-#include <rte_common.h>
-#include <rte_config.h>
-#include <rte_memory.h>
-#include <rte_lcore.h>
-#include <rte_atomic.h>
-#include <rte_branch_prediction.h>
-#include <rte_memzone.h>
-#include <rte_pause.h>
-
-#include "rte_ring.h"
+#include <rte_ring_core.h>
/**
* @warning
@@ -510,7 +496,7 @@ rte_ring_mp_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
unsigned int esize, unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, __IS_MP, free_space);
+ RTE_RING_QUEUE_FIXED, RTE_RING_SYNC_MT, free_space);
}
/**
@@ -539,7 +525,7 @@ rte_ring_sp_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
unsigned int esize, unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, __IS_SP, free_space);
+ RTE_RING_QUEUE_FIXED, RTE_RING_SYNC_ST, free_space);
}
/**
@@ -570,7 +556,7 @@ rte_ring_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
unsigned int esize, unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, r->prod.single, free_space);
+ RTE_RING_QUEUE_FIXED, r->prod.sync_type, free_space);
}
/**
@@ -675,7 +661,7 @@ rte_ring_mc_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
unsigned int esize, unsigned int n, unsigned int *available)
{
return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, __IS_MC, available);
+ RTE_RING_QUEUE_FIXED, RTE_RING_SYNC_MT, available);
}
/**
@@ -703,7 +689,7 @@ rte_ring_sc_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
unsigned int esize, unsigned int n, unsigned int *available)
{
return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, __IS_SC, available);
+ RTE_RING_QUEUE_FIXED, RTE_RING_SYNC_ST, available);
}
/**
@@ -734,7 +720,7 @@ rte_ring_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
unsigned int esize, unsigned int n, unsigned int *available)
{
return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, r->cons.single, available);
+ RTE_RING_QUEUE_FIXED, r->cons.sync_type, available);
}
/**
@@ -842,7 +828,7 @@ rte_ring_mp_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
unsigned int esize, unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE, __IS_MP, free_space);
+ RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_MT, free_space);
}
/**
@@ -871,7 +857,7 @@ rte_ring_sp_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
unsigned int esize, unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE, __IS_SP, free_space);
+ RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_ST, free_space);
}
/**
@@ -902,7 +888,7 @@ rte_ring_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
unsigned int esize, unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE, r->prod.single, free_space);
+ RTE_RING_QUEUE_VARIABLE, r->prod.sync_type, free_space);
}
/**
@@ -934,7 +920,7 @@ rte_ring_mc_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
unsigned int esize, unsigned int n, unsigned int *available)
{
return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE, __IS_MC, available);
+ RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_MT, available);
}
/**
@@ -963,7 +949,7 @@ rte_ring_sc_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
unsigned int esize, unsigned int n, unsigned int *available)
{
return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE, __IS_SC, available);
+ RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_ST, available);
}
/**
@@ -995,9 +981,11 @@ rte_ring_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
{
return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
RTE_RING_QUEUE_VARIABLE,
- r->cons.single, available);
+ r->cons.sync_type, available);
}
+#include <rte_ring.h>
+
#ifdef __cplusplus
}
#endif
--
2.17.1
^ permalink raw reply [relevance 9%]
* [dpdk-dev] [PATCH v4 3/9] ring: introduce RTS ring mode
2020-04-17 13:36 3% ` [dpdk-dev] [PATCH v4 0/9] New sync modes for ring Konstantin Ananyev
2020-04-17 13:36 9% ` [dpdk-dev] [PATCH v4 2/9] ring: prepare ring to allow new sync schemes Konstantin Ananyev
@ 2020-04-17 13:36 1% ` Konstantin Ananyev
2020-04-18 16:32 3% ` [dpdk-dev] [PATCH v5 0/9] New sync modes for ring Konstantin Ananyev
2 siblings, 0 replies; 200+ results
From: Konstantin Ananyev @ 2020-04-17 13:36 UTC (permalink / raw)
To: dev; +Cc: honnappa.nagarahalli, david.marchand, jielong.zjl, Konstantin Ananyev
Introduce relaxed tail sync (RTS) mode for MT ring synchronization.
Aim to reduce stall times in case when ring is used on
overcommited cpus (multiple active threads on the same cpu).
The main difference from original MP/MC algorithm is that
tail value is increased not by every thread that finished enqueue/dequeue,
but only by the last one.
That allows threads to avoid spinning on ring tail value,
leaving actual tail value change to the last thread in the update queue.
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
check-abi.sh reports what I believe is a false-positive about
ring cons/prod changes. As a workaround, devtools/libabigail.abignore is
updated to suppress *struct ring* related errors.
devtools/libabigail.abignore | 7 +
lib/librte_ring/Makefile | 4 +-
lib/librte_ring/meson.build | 7 +-
lib/librte_ring/rte_ring.c | 100 +++++-
lib/librte_ring/rte_ring.h | 70 +++-
lib/librte_ring/rte_ring_core.h | 35 +-
lib/librte_ring/rte_ring_elem.h | 90 ++++-
lib/librte_ring/rte_ring_rts.h | 439 +++++++++++++++++++++++++
lib/librte_ring/rte_ring_rts_c11_mem.h | 179 ++++++++++
9 files changed, 901 insertions(+), 30 deletions(-)
create mode 100644 lib/librte_ring/rte_ring_rts.h
create mode 100644 lib/librte_ring/rte_ring_rts_c11_mem.h
diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore
index a59df8f13..cd86d89ca 100644
--- a/devtools/libabigail.abignore
+++ b/devtools/libabigail.abignore
@@ -11,3 +11,10 @@
type_kind = enum
name = rte_crypto_asym_xform_type
changed_enumerators = RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
+; Ignore updates of ring prod/cons
+[suppress_type]
+ type_kind = struct
+ name = rte_ring
+[suppress_type]
+ type_kind = struct
+ name = rte_event_ring
diff --git a/lib/librte_ring/Makefile b/lib/librte_ring/Makefile
index 6572768c9..04e446e37 100644
--- a/lib/librte_ring/Makefile
+++ b/lib/librte_ring/Makefile
@@ -19,6 +19,8 @@ SYMLINK-$(CONFIG_RTE_LIBRTE_RING)-include := rte_ring.h \
rte_ring_core.h \
rte_ring_elem.h \
rte_ring_generic.h \
- rte_ring_c11_mem.h
+ rte_ring_c11_mem.h \
+ rte_ring_rts.h \
+ rte_ring_rts_c11_mem.h
include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_ring/meson.build b/lib/librte_ring/meson.build
index c656781da..a95598032 100644
--- a/lib/librte_ring/meson.build
+++ b/lib/librte_ring/meson.build
@@ -6,4 +6,9 @@ headers = files('rte_ring.h',
'rte_ring_core.h',
'rte_ring_elem.h',
'rte_ring_c11_mem.h',
- 'rte_ring_generic.h')
+ 'rte_ring_generic.h',
+ 'rte_ring_rts.h',
+ 'rte_ring_rts_c11_mem.h')
+
+# rte_ring_create_elem and rte_ring_get_memsize_elem are experimental
+allow_experimental_apis = true
diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c
index fa5733907..222eec0fb 100644
--- a/lib/librte_ring/rte_ring.c
+++ b/lib/librte_ring/rte_ring.c
@@ -45,6 +45,9 @@ EAL_REGISTER_TAILQ(rte_ring_tailq)
/* true if x is a power of 2 */
#define POWEROF2(x) ((((x)-1) & (x)) == 0)
+/* by default set head/tail distance as 1/8 of ring capacity */
+#define HTD_MAX_DEF 8
+
/* return the size of memory occupied by a ring */
ssize_t
rte_ring_get_memsize_elem(unsigned int esize, unsigned int count)
@@ -79,11 +82,84 @@ rte_ring_get_memsize(unsigned int count)
return rte_ring_get_memsize_elem(sizeof(void *), count);
}
+/*
+ * internal helper function to reset prod/cons head-tail values.
+ */
+static void
+reset_headtail(void *p)
+{
+ struct rte_ring_headtail *ht;
+ struct rte_ring_rts_headtail *ht_rts;
+
+ ht = p;
+ ht_rts = p;
+
+ switch (ht->sync_type) {
+ case RTE_RING_SYNC_MT:
+ case RTE_RING_SYNC_ST:
+ ht->head = 0;
+ ht->tail = 0;
+ break;
+ case RTE_RING_SYNC_MT_RTS:
+ ht_rts->head.raw = 0;
+ ht_rts->tail.raw = 0;
+ break;
+ default:
+ /* unknown sync mode */
+ RTE_ASSERT(0);
+ }
+}
+
void
rte_ring_reset(struct rte_ring *r)
{
- r->prod.head = r->cons.head = 0;
- r->prod.tail = r->cons.tail = 0;
+ reset_headtail(&r->prod);
+ reset_headtail(&r->cons);
+}
+
+/*
+ * helper function, calculates sync_type values for prod and cons
+ * based on input flags. Returns zero at success or negative
+ * errno value otherwise.
+ */
+static int
+get_sync_type(uint32_t flags, enum rte_ring_sync_type *prod_st,
+ enum rte_ring_sync_type *cons_st)
+{
+ static const uint32_t prod_st_flags =
+ (RING_F_SP_ENQ | RING_F_MP_RTS_ENQ);
+ static const uint32_t cons_st_flags =
+ (RING_F_SC_DEQ | RING_F_MC_RTS_DEQ);
+
+ switch (flags & prod_st_flags) {
+ case 0:
+ *prod_st = RTE_RING_SYNC_MT;
+ break;
+ case RING_F_SP_ENQ:
+ *prod_st = RTE_RING_SYNC_ST;
+ break;
+ case RING_F_MP_RTS_ENQ:
+ *prod_st = RTE_RING_SYNC_MT_RTS;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ switch (flags & cons_st_flags) {
+ case 0:
+ *cons_st = RTE_RING_SYNC_MT;
+ break;
+ case RING_F_SC_DEQ:
+ *cons_st = RTE_RING_SYNC_ST;
+ break;
+ case RING_F_MC_RTS_DEQ:
+ *cons_st = RTE_RING_SYNC_MT_RTS;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
}
int
@@ -100,16 +176,20 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
RTE_BUILD_BUG_ON((offsetof(struct rte_ring, prod) &
RTE_CACHE_LINE_MASK) != 0);
+ RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, sync_type) !=
+ offsetof(struct rte_ring_rts_headtail, sync_type));
+ RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, tail) !=
+ offsetof(struct rte_ring_rts_headtail, tail.val.pos));
+
/* init the ring structure */
memset(r, 0, sizeof(*r));
ret = strlcpy(r->name, name, sizeof(r->name));
if (ret < 0 || ret >= (int)sizeof(r->name))
return -ENAMETOOLONG;
r->flags = flags;
- r->prod.sync_type = (flags & RING_F_SP_ENQ) ?
- RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
- r->cons.sync_type = (flags & RING_F_SC_DEQ) ?
- RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
+ ret = get_sync_type(flags, &r->prod.sync_type, &r->cons.sync_type);
+ if (ret != 0)
+ return ret;
if (flags & RING_F_EXACT_SZ) {
r->size = rte_align32pow2(count + 1);
@@ -126,8 +206,12 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
r->mask = count - 1;
r->capacity = r->mask;
}
- r->prod.head = r->cons.head = 0;
- r->prod.tail = r->cons.tail = 0;
+
+ /* set default values for head-tail distance */
+ if (flags & RING_F_MP_RTS_ENQ)
+ rte_ring_set_prod_htd_max(r, r->capacity / HTD_MAX_DEF);
+ if (flags & RING_F_MC_RTS_DEQ)
+ rte_ring_set_cons_htd_max(r, r->capacity / HTD_MAX_DEF);
return 0;
}
diff --git a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h
index 35ee4491c..77f206ca7 100644
--- a/lib/librte_ring/rte_ring.h
+++ b/lib/librte_ring/rte_ring.h
@@ -1,6 +1,6 @@
/* SPDX-License-Identifier: BSD-3-Clause
*
- * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2010-2020 Intel Corporation
* Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
* All rights reserved.
* Derived from FreeBSD's bufring.h
@@ -389,8 +389,21 @@ static __rte_always_inline unsigned int
rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
- return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- r->prod.sync_type, free_space);
+ switch (r->prod.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mp_enqueue_bulk(r, obj_table, n, free_space);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sp_enqueue_bulk(r, obj_table, n, free_space);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mp_rts_enqueue_bulk(r, obj_table, n,
+ free_space);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ return 0;
}
/**
@@ -524,8 +537,20 @@ static __rte_always_inline unsigned int
rte_ring_dequeue_bulk(struct rte_ring *r, void **obj_table, unsigned int n,
unsigned int *available)
{
- return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- r->cons.sync_type, available);
+ switch (r->cons.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mc_dequeue_bulk(r, obj_table, n, available);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sc_dequeue_bulk(r, obj_table, n, available);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mc_rts_dequeue_bulk(r, obj_table, n, available);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ return 0;
}
/**
@@ -845,8 +870,21 @@ static __rte_always_inline unsigned
rte_ring_enqueue_burst(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
- return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_VARIABLE,
- r->prod.sync_type, free_space);
+ switch (r->prod.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mp_enqueue_burst(r, obj_table, n, free_space);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sp_enqueue_burst(r, obj_table, n, free_space);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mp_rts_enqueue_burst(r, obj_table, n,
+ free_space);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ return 0;
}
/**
@@ -925,9 +963,21 @@ static __rte_always_inline unsigned
rte_ring_dequeue_burst(struct rte_ring *r, void **obj_table,
unsigned int n, unsigned int *available)
{
- return __rte_ring_do_dequeue(r, obj_table, n,
- RTE_RING_QUEUE_VARIABLE,
- r->cons.sync_type, available);
+ switch (r->cons.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mc_dequeue_burst(r, obj_table, n, available);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sc_dequeue_burst(r, obj_table, n, available);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mc_rts_dequeue_burst(r, obj_table, n,
+ available);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ return 0;
}
#ifdef __cplusplus
diff --git a/lib/librte_ring/rte_ring_core.h b/lib/librte_ring/rte_ring_core.h
index 459a0ffa1..173b5f68d 100644
--- a/lib/librte_ring/rte_ring_core.h
+++ b/lib/librte_ring/rte_ring_core.h
@@ -56,6 +56,9 @@ enum rte_ring_queue_behavior {
enum rte_ring_sync_type {
RTE_RING_SYNC_MT, /**< multi-thread safe (default mode) */
RTE_RING_SYNC_ST, /**< single thread only */
+#ifdef ALLOW_EXPERIMENTAL_API
+ RTE_RING_SYNC_MT_RTS, /**< multi-thread relaxed tail sync */
+#endif
};
/**
@@ -75,6 +78,21 @@ struct rte_ring_headtail {
};
};
+union rte_ring_rts_poscnt {
+ uint64_t raw;
+ struct {
+ uint32_t cnt; /**< head/tail reference counter */
+ uint32_t pos; /**< head/tail position */
+ } val;
+};
+
+struct rte_ring_rts_headtail {
+ volatile union rte_ring_rts_poscnt tail;
+ enum rte_ring_sync_type sync_type; /**< sync type of prod/cons */
+ uint32_t htd_max; /**< max allowed distance between head/tail */
+ volatile union rte_ring_rts_poscnt head;
+};
+
/**
* An RTE ring structure.
*
@@ -103,11 +121,21 @@ struct rte_ring {
char pad0 __rte_cache_aligned; /**< empty cache line */
/** Ring producer status. */
- struct rte_ring_headtail prod __rte_cache_aligned;
+ RTE_STD_C11
+ union {
+ struct rte_ring_headtail prod;
+ struct rte_ring_rts_headtail rts_prod;
+ } __rte_cache_aligned;
+
char pad1 __rte_cache_aligned; /**< empty cache line */
/** Ring consumer status. */
- struct rte_ring_headtail cons __rte_cache_aligned;
+ RTE_STD_C11
+ union {
+ struct rte_ring_headtail cons;
+ struct rte_ring_rts_headtail rts_cons;
+ } __rte_cache_aligned;
+
char pad2 __rte_cache_aligned; /**< empty cache line */
};
@@ -124,6 +152,9 @@ struct rte_ring {
#define RING_F_EXACT_SZ 0x0004
#define RTE_RING_SZ_MASK (0x7fffffffU) /**< Ring size mask */
+#define RING_F_MP_RTS_ENQ 0x0008 /**< The default enqueue is "MP RTS". */
+#define RING_F_MC_RTS_DEQ 0x0010 /**< The default dequeue is "MC RTS". */
+
#ifdef __cplusplus
}
#endif
diff --git a/lib/librte_ring/rte_ring_elem.h b/lib/librte_ring/rte_ring_elem.h
index 7406c0b0f..6da0a917b 100644
--- a/lib/librte_ring/rte_ring_elem.h
+++ b/lib/librte_ring/rte_ring_elem.h
@@ -528,6 +528,10 @@ rte_ring_sp_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
RTE_RING_QUEUE_FIXED, RTE_RING_SYNC_ST, free_space);
}
+#ifdef ALLOW_EXPERIMENTAL_API
+#include <rte_ring_rts.h>
+#endif
+
/**
* Enqueue several objects on a ring.
*
@@ -557,6 +561,26 @@ rte_ring_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
{
return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
RTE_RING_QUEUE_FIXED, r->prod.sync_type, free_space);
+
+ switch (r->prod.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mp_enqueue_bulk_elem(r, obj_table, esize, n,
+ free_space);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sp_enqueue_bulk_elem(r, obj_table, esize, n,
+ free_space);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mp_rts_enqueue_bulk_elem(r, obj_table, esize, n,
+ free_space);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ if (free_space != NULL)
+ *free_space = 0;
+ return 0;
}
/**
@@ -661,7 +685,7 @@ rte_ring_mc_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
unsigned int esize, unsigned int n, unsigned int *available)
{
return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, RTE_RING_SYNC_MT, available);
+ RTE_RING_QUEUE_FIXED, RTE_RING_SYNC_MT, available);
}
/**
@@ -719,8 +743,25 @@ static __rte_always_inline unsigned int
rte_ring_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
unsigned int esize, unsigned int n, unsigned int *available)
{
- return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, r->cons.sync_type, available);
+ switch (r->cons.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mc_dequeue_bulk_elem(r, obj_table, esize, n,
+ available);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sc_dequeue_bulk_elem(r, obj_table, esize, n,
+ available);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mc_rts_dequeue_bulk_elem(r, obj_table, esize,
+ n, available);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ if (available != NULL)
+ *available = 0;
+ return 0;
}
/**
@@ -887,8 +928,25 @@ static __rte_always_inline unsigned
rte_ring_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
unsigned int esize, unsigned int n, unsigned int *free_space)
{
- return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE, r->prod.sync_type, free_space);
+ switch (r->prod.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mp_enqueue_burst_elem(r, obj_table, esize, n,
+ free_space);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sp_enqueue_burst_elem(r, obj_table, esize, n,
+ free_space);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mp_rts_enqueue_burst_elem(r, obj_table, esize,
+ n, free_space);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ if (free_space != NULL)
+ *free_space = 0;
+ return 0;
}
/**
@@ -979,9 +1037,25 @@ static __rte_always_inline unsigned int
rte_ring_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
unsigned int esize, unsigned int n, unsigned int *available)
{
- return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE,
- r->cons.sync_type, available);
+ switch (r->cons.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mc_dequeue_burst_elem(r, obj_table, esize, n,
+ available);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sc_dequeue_burst_elem(r, obj_table, esize, n,
+ available);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mc_rts_dequeue_burst_elem(r, obj_table, esize,
+ n, available);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ if (available != NULL)
+ *available = 0;
+ return 0;
}
#include <rte_ring.h>
diff --git a/lib/librte_ring/rte_ring_rts.h b/lib/librte_ring/rte_ring_rts.h
new file mode 100644
index 000000000..8ced07096
--- /dev/null
+++ b/lib/librte_ring/rte_ring_rts.h
@@ -0,0 +1,439 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2010-2020 Intel Corporation
+ * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#ifndef _RTE_RING_RTS_H_
+#define _RTE_RING_RTS_H_
+
+/**
+ * @file rte_ring_rts.h
+ * @b EXPERIMENTAL: this API may change without prior notice
+ * It is not recommended to include this file directly.
+ * Please include <rte_ring.h> instead.
+ *
+ * Contains functions for Relaxed Tail Sync (RTS) ring mode.
+ * The main idea remains the same as for our original MP/MC synchronization
+ * mechanism.
+ * The main difference is that tail value is increased not
+ * by every thread that finished enqueue/dequeue,
+ * but only by the current last one doing enqueue/dequeue.
+ * That allows threads to skip spinning on tail value,
+ * leaving actual tail value change to last thread at a given instance.
+ * RTS requires 2 64-bit CAS for each enqueue(/dequeue) operation:
+ * one for head update, second for tail update.
+ * As a gain it allows thread to avoid spinning/waiting on tail value.
+ * In comparision original MP/MC algorithm requires one 32-bit CAS
+ * for head update and waiting/spinning on tail value.
+ *
+ * Brief outline:
+ * - introduce update counter (cnt) for both head and tail.
+ * - increment head.cnt for each head.value update
+ * - write head.value and head.cnt atomically (64-bit CAS)
+ * - move tail.value ahead only when tail.cnt + 1 == head.cnt
+ * (indicating that this is the last thread updating the tail)
+ * - increment tail.cnt when each enqueue/dequeue op finishes
+ * (no matter if tail.value going to change or not)
+ * - write tail.value and tail.cnt atomically (64-bit CAS)
+ *
+ * To avoid producer/consumer starvation:
+ * - limit max allowed distance between head and tail value (HTD_MAX).
+ * I.E. thread is allowed to proceed with changing head.value,
+ * only when: head.value - tail.value <= HTD_MAX
+ * HTD_MAX is an optional parameter.
+ * With HTD_MAX == 0 we'll have fully serialized ring -
+ * i.e. only one thread at a time will be able to enqueue/dequeue
+ * to/from the ring.
+ * With HTD_MAX >= ring.capacity - no limitation.
+ * By default HTD_MAX == ring.capacity / 8.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_ring_rts_c11_mem.h>
+
+/**
+ * @internal Enqueue several objects on the RTS ring.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
+ * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
+ * @param free_space
+ * returns the amount of space after the enqueue operation has finished
+ * @return
+ * Actual number of objects enqueued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_enqueue_elem(struct rte_ring *r, const void *obj_table,
+ uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+ uint32_t *free_space)
+{
+ uint32_t free, head;
+
+ n = __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
+
+ if (n != 0) {
+ __rte_ring_enqueue_elems(r, head, obj_table, esize, n);
+ __rte_ring_rts_update_tail(&r->rts_prod);
+ }
+
+ if (free_space != NULL)
+ *free_space = free - n;
+ return n;
+}
+
+/**
+ * @internal Dequeue several objects from the RTS ring.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to pull from the ring.
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
+ * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
+ * @param available
+ * returns the number of remaining ring entries after the dequeue has finished
+ * @return
+ * - Actual number of objects dequeued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_dequeue_elem(struct rte_ring *r, void *obj_table,
+ uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+ uint32_t *available)
+{
+ uint32_t entries, head;
+
+ n = __rte_ring_rts_move_cons_head(r, n, behavior, &head, &entries);
+
+ if (n != 0) {
+ __rte_ring_dequeue_elems(r, head, obj_table, esize, n);
+ __rte_ring_rts_update_tail(&r->rts_cons);
+ }
+
+ if (available != NULL)
+ *available = entries - n;
+ return n;
+}
+
+/**
+ * Enqueue several objects on the RTS ring (multi-producers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * The number of objects enqueued, either 0 or n
+ */
+__rte_experimental
+static __rte_always_inline unsigned int
+rte_ring_mp_rts_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_rts_enqueue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_FIXED, free_space);
+}
+
+/**
+ * Dequeue several objects from an RTS ring (multi-consumers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects that will be filled.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * The number of objects dequeued, either 0 or n
+ */
+__rte_experimental
+static __rte_always_inline unsigned int
+rte_ring_mc_rts_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_rts_dequeue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_FIXED, available);
+}
+
+/**
+ * Enqueue several objects on the RTS ring (multi-producers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * - n: Actual number of objects enqueued.
+ */
+__rte_experimental
+static __rte_always_inline unsigned
+rte_ring_mp_rts_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_rts_enqueue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_VARIABLE, free_space);
+}
+
+/**
+ * Dequeue several objects from an RTS ring (multi-consumers safe).
+ * When the requested objects are more than the available objects,
+ * only dequeue the actual number of objects.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects that will be filled.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * - n: Actual number of objects dequeued, 0 if ring is empty
+ */
+__rte_experimental
+static __rte_always_inline unsigned
+rte_ring_mc_rts_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_rts_dequeue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_VARIABLE, available);
+}
+
+/**
+ * Enqueue several objects on the RTS ring (multi-producers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * The number of objects enqueued, either 0 or n
+ */
+__rte_experimental
+static __rte_always_inline unsigned int
+rte_ring_mp_rts_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
+ unsigned int n, unsigned int *free_space)
+{
+ return rte_ring_mp_rts_enqueue_bulk_elem(r, obj_table,
+ sizeof(uintptr_t), n, free_space);
+}
+
+/**
+ * Dequeue several objects from an RTS ring (multi-consumers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * The number of objects dequeued, either 0 or n
+ */
+__rte_experimental
+static __rte_always_inline unsigned int
+rte_ring_mc_rts_dequeue_bulk(struct rte_ring *r, void **obj_table,
+ unsigned int n, unsigned int *available)
+{
+ return rte_ring_mc_rts_dequeue_bulk_elem(r, obj_table,
+ sizeof(uintptr_t), n, available);
+}
+
+/**
+ * Enqueue several objects on the RTS ring (multi-producers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * - n: Actual number of objects enqueued.
+ */
+__rte_experimental
+static __rte_always_inline unsigned
+rte_ring_mp_rts_enqueue_burst(struct rte_ring *r, void * const *obj_table,
+ unsigned int n, unsigned int *free_space)
+{
+ return rte_ring_mp_rts_enqueue_burst_elem(r, obj_table,
+ sizeof(uintptr_t), n, free_space);
+}
+
+/**
+ * Dequeue several objects from an RTS ring (multi-consumers safe).
+ * When the requested objects are more than the available objects,
+ * only dequeue the actual number of objects.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * - n: Actual number of objects dequeued, 0 if ring is empty
+ */
+__rte_experimental
+static __rte_always_inline unsigned
+rte_ring_mc_rts_dequeue_burst(struct rte_ring *r, void **obj_table,
+ unsigned int n, unsigned int *available)
+{
+ return rte_ring_mc_rts_dequeue_burst_elem(r, obj_table,
+ sizeof(uintptr_t), n, available);
+}
+
+/**
+ * Return producer max Head-Tail-Distance (HTD).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * Producer HTD value, if producer is set in appropriate sync mode,
+ * or UINT32_MAX otherwise.
+ */
+__rte_experimental
+static inline uint32_t
+rte_ring_get_prod_htd_max(const struct rte_ring *r)
+{
+ if (r->prod.sync_type == RTE_RING_SYNC_MT_RTS)
+ return r->rts_prod.htd_max;
+ return UINT32_MAX;
+}
+
+/**
+ * Set producer max Head-Tail-Distance (HTD).
+ * Note that producer has to use appropriate sync mode (RTS).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param v
+ * new HTD value to setup.
+ * @return
+ * Zero on success, or negative error code otherwise.
+ */
+__rte_experimental
+static inline int
+rte_ring_set_prod_htd_max(struct rte_ring *r, uint32_t v)
+{
+ if (r->prod.sync_type != RTE_RING_SYNC_MT_RTS)
+ return -ENOTSUP;
+
+ r->rts_prod.htd_max = v;
+ return 0;
+}
+
+/**
+ * Return consumer max Head-Tail-Distance (HTD).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * Consumer HTD value, if consumer is set in appropriate sync mode,
+ * or UINT32_MAX otherwise.
+ */
+__rte_experimental
+static inline uint32_t
+rte_ring_get_cons_htd_max(const struct rte_ring *r)
+{
+ if (r->cons.sync_type == RTE_RING_SYNC_MT_RTS)
+ return r->rts_cons.htd_max;
+ return UINT32_MAX;
+}
+
+/**
+ * Set consumer max Head-Tail-Distance (HTD).
+ * Note that consumer has to use appropriate sync mode (RTS).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param v
+ * new HTD value to setup.
+ * @return
+ * Zero on success, or negative error code otherwise.
+ */
+__rte_experimental
+static inline int
+rte_ring_set_cons_htd_max(struct rte_ring *r, uint32_t v)
+{
+ if (r->cons.sync_type != RTE_RING_SYNC_MT_RTS)
+ return -ENOTSUP;
+
+ r->rts_cons.htd_max = v;
+ return 0;
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_RING_RTS_H_ */
diff --git a/lib/librte_ring/rte_ring_rts_c11_mem.h b/lib/librte_ring/rte_ring_rts_c11_mem.h
new file mode 100644
index 000000000..9f26817c0
--- /dev/null
+++ b/lib/librte_ring/rte_ring_rts_c11_mem.h
@@ -0,0 +1,179 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2010-2020 Intel Corporation
+ * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#ifndef _RTE_RING_RTS_C11_MEM_H_
+#define _RTE_RING_RTS_C11_MEM_H_
+
+/**
+ * @file rte_ring_rts_c11_mem.h
+ * It is not recommended to include this file directly,
+ * include <rte_ring.h> instead.
+ * Contains internal helper functions for Relaxed Tail Sync (RTS) ring mode.
+ * For more information please refer to <rte_ring_rts.h>.
+ */
+
+/**
+ * @internal This function updates tail values.
+ */
+static __rte_always_inline void
+__rte_ring_rts_update_tail(struct rte_ring_rts_headtail *ht)
+{
+ union rte_ring_rts_poscnt h, ot, nt;
+
+ /*
+ * If there are other enqueues/dequeues in progress that
+ * might preceded us, then don't update tail with new value.
+ */
+
+ ot.raw = __atomic_load_n(&ht->tail.raw, __ATOMIC_ACQUIRE);
+
+ do {
+ /* on 32-bit systems we have to do atomic read here */
+ h.raw = __atomic_load_n(&ht->head.raw, __ATOMIC_RELAXED);
+
+ nt.raw = ot.raw;
+ if (++nt.val.cnt == h.val.cnt)
+ nt.val.pos = h.val.pos;
+
+ } while (__atomic_compare_exchange_n(&ht->tail.raw, &ot.raw, nt.raw,
+ 0, __ATOMIC_RELEASE, __ATOMIC_ACQUIRE) == 0);
+}
+
+/**
+ * @internal This function waits till head/tail distance wouldn't
+ * exceed pre-defined max value.
+ */
+static __rte_always_inline void
+__rte_ring_rts_head_wait(const struct rte_ring_rts_headtail *ht,
+ union rte_ring_rts_poscnt *h)
+{
+ uint32_t max;
+
+ max = ht->htd_max;
+
+ while (h->val.pos - ht->tail.val.pos > max) {
+ rte_pause();
+ h->raw = __atomic_load_n(&ht->head.raw, __ATOMIC_ACQUIRE);
+ }
+}
+
+/**
+ * @internal This function updates the producer head for enqueue.
+ */
+static __rte_always_inline uint32_t
+__rte_ring_rts_move_prod_head(struct rte_ring *r, uint32_t num,
+ enum rte_ring_queue_behavior behavior, uint32_t *old_head,
+ uint32_t *free_entries)
+{
+ uint32_t n;
+ union rte_ring_rts_poscnt nh, oh;
+
+ const uint32_t capacity = r->capacity;
+
+ oh.raw = __atomic_load_n(&r->rts_prod.head.raw, __ATOMIC_ACQUIRE);
+
+ do {
+ /* Reset n to the initial burst count */
+ n = num;
+
+ /*
+ * wait for prod head/tail distance,
+ * make sure that we read prod head *before*
+ * reading cons tail.
+ */
+ __rte_ring_rts_head_wait(&r->rts_prod, &oh);
+
+ /*
+ * The subtraction is done between two unsigned 32bits value
+ * (the result is always modulo 32 bits even if we have
+ * *old_head > cons_tail). So 'free_entries' is always between 0
+ * and capacity (which is < size).
+ */
+ *free_entries = capacity + r->cons.tail - oh.val.pos;
+
+ /* check that we have enough room in ring */
+ if (unlikely(n > *free_entries))
+ n = (behavior == RTE_RING_QUEUE_FIXED) ?
+ 0 : *free_entries;
+
+ if (n == 0)
+ break;
+
+ nh.val.pos = oh.val.pos + n;
+ nh.val.cnt = oh.val.cnt + 1;
+
+ /*
+ * this CAS(ACQUIRE, ACQUIRE) serves as a hoist barrier to prevent:
+ * - OOO reads of cons tail value
+ * - OOO copy of elems to the ring
+ */
+ } while (__atomic_compare_exchange_n(&r->rts_prod.head.raw,
+ &oh.raw, nh.raw,
+ 0, __ATOMIC_ACQUIRE, __ATOMIC_ACQUIRE) == 0);
+
+ *old_head = oh.val.pos;
+ return n;
+}
+
+/**
+ * @internal This function updates the consumer head for dequeue
+ */
+static __rte_always_inline unsigned int
+__rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num,
+ enum rte_ring_queue_behavior behavior, uint32_t *old_head,
+ uint32_t *entries)
+{
+ uint32_t n;
+ union rte_ring_rts_poscnt nh, oh;
+
+ oh.raw = __atomic_load_n(&r->rts_cons.head.raw, __ATOMIC_ACQUIRE);
+
+ /* move cons.head atomically */
+ do {
+ /* Restore n as it may change every loop */
+ n = num;
+
+ /*
+ * wait for cons head/tail distance,
+ * make sure that we read cons head *before*
+ * reading prod tail.
+ */
+ __rte_ring_rts_head_wait(&r->rts_cons, &oh);
+
+ /* The subtraction is done between two unsigned 32bits value
+ * (the result is always modulo 32 bits even if we have
+ * cons_head > prod_tail). So 'entries' is always between 0
+ * and size(ring)-1.
+ */
+ *entries = r->prod.tail - oh.val.pos;
+
+ /* Set the actual entries for dequeue */
+ if (n > *entries)
+ n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 : *entries;
+
+ if (unlikely(n == 0))
+ break;
+
+ nh.val.pos = oh.val.pos + n;
+ nh.val.cnt = oh.val.cnt + 1;
+
+ /*
+ * this CAS(ACQUIRE, ACQUIRE) serves as a hoist barrier to prevent:
+ * - OOO reads of prod tail value
+ * - OOO copy of elems from the ring
+ */
+ } while (__atomic_compare_exchange_n(&r->rts_cons.head.raw,
+ &oh.raw, nh.raw,
+ 0, __ATOMIC_ACQUIRE, __ATOMIC_ACQUIRE) == 0);
+
+ *old_head = oh.val.pos;
+ return n;
+}
+
+#endif /* _RTE_RING_RTS_C11_MEM_H_ */
--
2.17.1
^ permalink raw reply [relevance 1%]
* Re: [dpdk-dev] [PATCH v5] build: disable experimental API check internally
@ 2020-04-17 13:56 4% ` David Marchand
2020-04-17 15:05 0% ` Trahe, Fiona
0 siblings, 1 reply; 200+ results
From: David Marchand @ 2020-04-17 13:56 UTC (permalink / raw)
To: Trahe, Fiona
Cc: dev, Jerin Jacob, Pavan Nikhilesh, Richardson, Bruce,
Thomas Monjalon, Yigit, Ferruh, Hemant Agrawal
On Fri, Apr 17, 2020 at 3:44 PM Trahe, Fiona <fiona.trahe@intel.com> wrote:
>
> Hi David,
>
> > -----Original Message-----
> > From: David Marchand <david.marchand@redhat.com>
> > Sent: Friday, April 17, 2020 2:23 PM
> > To: Trahe, Fiona <fiona.trahe@intel.com>
> > Cc: dev <dev@dpdk.org>; Jerin Jacob <jerinjacobk@gmail.com>; Pavan Nikhilesh
> > <pbhagavatula@marvell.com>; Richardson, Bruce <bruce.richardson@intel.com>; Thomas Monjalon
> > <thomas@monjalon.net>; Yigit, Ferruh <ferruh.yigit@intel.com>; Hemant Agrawal
> > <hemant.agrawal@nxp.com>
> > Subject: Re: [dpdk-dev] [PATCH v5] build: disable experimental API check internally
> >
> > On Fri, Apr 17, 2020 at 12:21 PM Trahe, Fiona <fiona.trahe@intel.com> wrote:
> > > I see this is already applied.
> > >
> > > However,
> > > rte_cryptodev_queue_pair_setup() calls
> > > rte_cryptodev_sym_get_existing_header_session_size()
> > > The former is a stable API, the latter is experimental.
> > > So I expect the build to break when ALLOW_EXPERIMENTAL_API is disabled.
> [Fiona] Thanks for confirming where the flag is.
> But I think you've missed my point.
> What about this problem?
- dpdk-test-crypto-perf is built as part of the dpdk compilation itself.
There is no user to be made aware of its use of experimental API.
Now if you are talking about how the crypto API is bent in that it
exposes a stable ABI with an underlying experimental ABI, this has
nothing to do with the flag change.
- But if you want to check crypto experimental api, then try to
disable the flag in examples making use of them.
--
David Marchand
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH v3] cryptodev: version cryptodev info get function
@ 2020-04-17 14:59 11% Arek Kusztal
0 siblings, 0 replies; 200+ results
From: Arek Kusztal @ 2020-04-17 14:59 UTC (permalink / raw)
To: dev; +Cc: akhil.goyal, fiona.trahe, Arek Kusztal
This patch adds versioned function rte_cryptodev_info_get()
to prevent some issues with ABI policy.
Node v21 works in same way as before, returning driver capabilities
directly to the API caller. These capabilities may include new elements
not part of the v20 ABI.
Node v20 function maintains compatibility with v20 ABI releases
by stripping out elements not supported in v20 ABI. Because
rte_cryptodev_info_get is called by other API functions,
rte_cryptodev_sym_capability_get function is versioned the same way.
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
---
v2:
- changed version numbers of symbols to 20.0.2
v3:
- added v2/v3 informations
- changed version numbers of symbols to 21
- fixed checkpatch issues
This patch depends on following patches:
[1] - "[v3] cryptodev: add chacha20-poly1305 aead algorithm"
(http://patchwork.dpdk.org/patch/64549/)
lib/librte_cryptodev/rte_cryptodev.c | 143 ++++++++++++++++++++++++-
lib/librte_cryptodev/rte_cryptodev.h | 39 ++++++-
lib/librte_cryptodev/rte_cryptodev_version.map | 7 ++
3 files changed, 186 insertions(+), 3 deletions(-)
diff --git a/lib/librte_cryptodev/rte_cryptodev.c b/lib/librte_cryptodev/rte_cryptodev.c
index 6d1d0e9..b061447 100644
--- a/lib/librte_cryptodev/rte_cryptodev.c
+++ b/lib/librte_cryptodev/rte_cryptodev.c
@@ -41,6 +41,9 @@
#include "rte_cryptodev.h"
#include "rte_cryptodev_pmd.h"
+#include <rte_compat.h>
+#include <rte_function_versioning.h>
+
static uint8_t nb_drivers;
static struct rte_cryptodev rte_crypto_devices[RTE_CRYPTO_MAX_DEVS];
@@ -56,6 +59,14 @@ static struct rte_cryptodev_global cryptodev_globals = {
/* spinlock for crypto device callbacks */
static rte_spinlock_t rte_cryptodev_cb_lock = RTE_SPINLOCK_INITIALIZER;
+static const struct rte_cryptodev_capabilities
+ cryptodev_undefined_capabilities[] = {
+ RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST()
+};
+
+static struct rte_cryptodev_capabilities
+ *capability_copies[RTE_CRYPTO_MAX_DEVS];
+static uint8_t is_capability_checked[RTE_CRYPTO_MAX_DEVS];
/**
* The user application callback description.
@@ -280,7 +291,43 @@ rte_crypto_auth_operation_strings[] = {
};
const struct rte_cryptodev_symmetric_capability *
-rte_cryptodev_sym_capability_get(uint8_t dev_id,
+rte_cryptodev_sym_capability_get_v20(uint8_t dev_id,
+ const struct rte_cryptodev_sym_capability_idx *idx)
+{
+ const struct rte_cryptodev_capabilities *capability;
+ struct rte_cryptodev_info dev_info;
+ int i = 0;
+
+ rte_cryptodev_info_get_v20(dev_id, &dev_info);
+
+ while ((capability = &dev_info.capabilities[i++])->op !=
+ RTE_CRYPTO_OP_TYPE_UNDEFINED) {
+ if (capability->op != RTE_CRYPTO_OP_TYPE_SYMMETRIC)
+ continue;
+
+ if (capability->sym.xform_type != idx->type)
+ continue;
+
+ if (idx->type == RTE_CRYPTO_SYM_XFORM_AUTH &&
+ capability->sym.auth.algo == idx->algo.auth)
+ return &capability->sym;
+
+ if (idx->type == RTE_CRYPTO_SYM_XFORM_CIPHER &&
+ capability->sym.cipher.algo == idx->algo.cipher)
+ return &capability->sym;
+
+ if (idx->type == RTE_CRYPTO_SYM_XFORM_AEAD &&
+ capability->sym.aead.algo == idx->algo.aead)
+ return &capability->sym;
+ }
+
+ return NULL;
+
+}
+VERSION_SYMBOL(rte_cryptodev_sym_capability_get, _v20, 20.0);
+
+const struct rte_cryptodev_symmetric_capability *
+rte_cryptodev_sym_capability_get_v21(uint8_t dev_id,
const struct rte_cryptodev_sym_capability_idx *idx)
{
const struct rte_cryptodev_capabilities *capability;
@@ -313,6 +360,10 @@ rte_cryptodev_sym_capability_get(uint8_t dev_id,
return NULL;
}
+MAP_STATIC_SYMBOL(const struct rte_cryptodev_symmetric_capability *
+ rte_cryptodev_sym_capability_get(uint8_t dev_id,
+ const struct rte_cryptodev_sym_capability_idx *idx),
+ rte_cryptodev_sym_capability_get_v21);
static int
param_range_check(uint16_t size, const struct rte_crypto_param_range *range)
@@ -999,6 +1050,13 @@ rte_cryptodev_close(uint8_t dev_id)
RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_close, -ENOTSUP);
retval = (*dev->dev_ops->dev_close)(dev);
+
+ if (capability_copies[dev_id]) {
+ free(capability_copies[dev_id]);
+ capability_copies[dev_id] = NULL;
+ }
+ is_capability_checked[dev_id] = 0;
+
if (retval < 0)
return retval;
@@ -1111,9 +1169,61 @@ rte_cryptodev_stats_reset(uint8_t dev_id)
(*dev->dev_ops->stats_reset)(dev);
}
+static void
+get_v20_capabilities(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
+{
+ const struct rte_cryptodev_capabilities *capability;
+ uint8_t found_invalid_capa = 0;
+ uint8_t counter = 0;
+
+ for (capability = dev_info->capabilities;
+ capability->op != RTE_CRYPTO_OP_TYPE_UNDEFINED;
+ ++capability, ++counter) {
+ if (capability->op == RTE_CRYPTO_OP_TYPE_SYMMETRIC &&
+ capability->sym.xform_type ==
+ RTE_CRYPTO_SYM_XFORM_AEAD
+ && capability->sym.aead.algo >=
+ RTE_CRYPTO_AEAD_CHACHA20_POLY1305) {
+ found_invalid_capa = 1;
+ counter--;
+ }
+ }
+ is_capability_checked[dev_id] = 1;
+ if (found_invalid_capa) {
+ capability_copies[dev_id] = malloc(counter *
+ sizeof(struct rte_cryptodev_capabilities));
+ if (capability_copies[dev_id] == NULL) {
+ /*
+ * error case - no memory to store the trimmed
+ * list, so have to return an empty list
+ */
+ dev_info->capabilities =
+ cryptodev_undefined_capabilities;
+ is_capability_checked[dev_id] = 0;
+ } else {
+ counter = 0;
+ for (capability = dev_info->capabilities;
+ capability->op !=
+ RTE_CRYPTO_OP_TYPE_UNDEFINED;
+ capability++) {
+ if (!(capability->op ==
+ RTE_CRYPTO_OP_TYPE_SYMMETRIC
+ && capability->sym.xform_type ==
+ RTE_CRYPTO_SYM_XFORM_AEAD
+ && capability->sym.aead.algo >=
+ RTE_CRYPTO_AEAD_CHACHA20_POLY1305)) {
+ capability_copies[dev_id][counter++] =
+ *capability;
+ }
+ }
+ dev_info->capabilities =
+ capability_copies[dev_id];
+ }
+ }
+}
void
-rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
+rte_cryptodev_info_get_v20(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
{
struct rte_cryptodev *dev;
@@ -1129,10 +1239,39 @@ rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
(*dev->dev_ops->dev_infos_get)(dev, dev_info);
+ if (capability_copies[dev_id] == NULL) {
+ if (!is_capability_checked[dev_id])
+ get_v20_capabilities(dev_id, dev_info);
+ } else
+ dev_info->capabilities = capability_copies[dev_id];
+
dev_info->driver_name = dev->device->driver->name;
dev_info->device = dev->device;
}
+VERSION_SYMBOL(rte_cryptodev_info_get, _v20, 20.0);
+
+void
+rte_cryptodev_info_get_v21(uint8_t dev_id, struct rte_cryptodev_info *dev_info)
+{
+ struct rte_cryptodev *dev;
+
+ if (!rte_cryptodev_pmd_is_valid_dev(dev_id)) {
+ CDEV_LOG_ERR("Invalid dev_id=%d", dev_id);
+ return;
+ }
+
+ dev = &rte_crypto_devices[dev_id];
+ memset(dev_info, 0, sizeof(struct rte_cryptodev_info));
+
+ RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
+ (*dev->dev_ops->dev_infos_get)(dev, dev_info);
+
+ dev_info->driver_name = dev->device->driver->name;
+ dev_info->device = dev->device;
+}
+MAP_STATIC_SYMBOL(void rte_cryptodev_info_get(uint8_t dev_id,
+ struct rte_cryptodev_info *dev_info), rte_cryptodev_info_get_v21);
int
rte_cryptodev_callback_register(uint8_t dev_id,
diff --git a/lib/librte_cryptodev/rte_cryptodev.h b/lib/librte_cryptodev/rte_cryptodev.h
index 437b8a9..7f1bc90 100644
--- a/lib/librte_cryptodev/rte_cryptodev.h
+++ b/lib/librte_cryptodev/rte_cryptodev.h
@@ -24,6 +24,9 @@ extern "C" {
#include <rte_common.h>
#include <rte_config.h>
+#include <rte_compat.h>
+#include <rte_function_versioning.h>
+
extern const char **rte_cyptodev_names;
/* Logging Macros */
@@ -217,6 +220,15 @@ struct rte_cryptodev_asym_capability_idx {
* - Return NULL if the capability not exist.
*/
const struct rte_cryptodev_symmetric_capability *
+rte_cryptodev_sym_capability_get_v20(uint8_t dev_id,
+ const struct rte_cryptodev_sym_capability_idx *idx);
+
+const struct rte_cryptodev_symmetric_capability *
+rte_cryptodev_sym_capability_get_v21(uint8_t dev_id,
+ const struct rte_cryptodev_sym_capability_idx *idx);
+BIND_DEFAULT_SYMBOL(rte_cryptodev_sym_capability_get, _v21, 21);
+
+const struct rte_cryptodev_symmetric_capability *
rte_cryptodev_sym_capability_get(uint8_t dev_id,
const struct rte_cryptodev_sym_capability_idx *idx);
@@ -758,9 +770,34 @@ rte_cryptodev_stats_reset(uint8_t dev_id);
* the last valid element has it's op field set to
* RTE_CRYPTO_OP_TYPE_UNDEFINED.
*/
-extern void
+
+void
rte_cryptodev_info_get(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
+/* An extra element RTE_CRYPTO_AEAD_CHACHA20_POLY1305 is added
+ * to enum rte_crypto_aead_algorithm, also changing the value of
+ * RTE_CRYPTO_AEAD_LIST_END. To maintain ABI compatibility with applications
+ * which linked against earlier versions, preventing them, for example, from
+ * picking up the new value and using it to index into an array sized too small
+ * for it, it is necessary to have two versions of rte_cryptodev_info_get()
+ * The latest version just returns directly the capabilities retrieved from
+ * the device. The compatible version inspects the capabilities retrieved
+ * from the device, but only returns them directly if the new value
+ * is not included. If the new value is included, it allocates space
+ * for a copy of the device capabilities, trims the new value from this
+ * and returns this copy. It only needs to do this once per device.
+ * For the corner case of a corner case when the alloc may fail,
+ * an empty capability list is returned, as there is no mechanism to return
+ * an error and adding such a mechanism would itself be an ABI breakage.
+ * The compatible version can be removed after the next major ABI release.
+ */
+
+void
+rte_cryptodev_info_get_v20(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
+
+void
+rte_cryptodev_info_get_v21(uint8_t dev_id, struct rte_cryptodev_info *dev_info);
+BIND_DEFAULT_SYMBOL(rte_cryptodev_info_get, _v21, 21);
/**
* Register a callback function for specific device id.
diff --git a/lib/librte_cryptodev/rte_cryptodev_version.map b/lib/librte_cryptodev/rte_cryptodev_version.map
index 6e41b4b..512a4a7 100644
--- a/lib/librte_cryptodev/rte_cryptodev_version.map
+++ b/lib/librte_cryptodev/rte_cryptodev_version.map
@@ -58,6 +58,13 @@ DPDK_20.0 {
local: *;
};
+DPDK_21 {
+ global:
+ rte_cryptodev_info_get;
+ rte_cryptodev_sym_capability_get;
+} DPDK_20.0;
+
+
EXPERIMENTAL {
global:
--
2.1.0
^ permalink raw reply [relevance 11%]
* Re: [dpdk-dev] [PATCH v5] build: disable experimental API check internally
2020-04-17 13:56 4% ` David Marchand
@ 2020-04-17 15:05 0% ` Trahe, Fiona
2020-04-17 15:52 0% ` Trahe, Fiona
0 siblings, 1 reply; 200+ results
From: Trahe, Fiona @ 2020-04-17 15:05 UTC (permalink / raw)
To: David Marchand
Cc: dev, Jerin Jacob, Pavan Nikhilesh, Richardson, Bruce,
Thomas Monjalon, Yigit, Ferruh, Hemant Agrawal, Trahe, Fiona
Hi Davd,
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Friday, April 17, 2020 2:56 PM
> To: Trahe, Fiona <fiona.trahe@intel.com>
> Cc: dev <dev@dpdk.org>; Jerin Jacob <jerinjacobk@gmail.com>; Pavan Nikhilesh
> <pbhagavatula@marvell.com>; Richardson, Bruce <bruce.richardson@intel.com>; Thomas Monjalon
> <thomas@monjalon.net>; Yigit, Ferruh <ferruh.yigit@intel.com>; Hemant Agrawal
> <hemant.agrawal@nxp.com>
> Subject: Re: [dpdk-dev] [PATCH v5] build: disable experimental API check internally
>
> On Fri, Apr 17, 2020 at 3:44 PM Trahe, Fiona <fiona.trahe@intel.com> wrote:
> >
> > Hi David,
> >
> > > -----Original Message-----
> > > From: David Marchand <david.marchand@redhat.com>
> > > Sent: Friday, April 17, 2020 2:23 PM
> > > To: Trahe, Fiona <fiona.trahe@intel.com>
> > > Cc: dev <dev@dpdk.org>; Jerin Jacob <jerinjacobk@gmail.com>; Pavan Nikhilesh
> > > <pbhagavatula@marvell.com>; Richardson, Bruce <bruce.richardson@intel.com>; Thomas
> Monjalon
> > > <thomas@monjalon.net>; Yigit, Ferruh <ferruh.yigit@intel.com>; Hemant Agrawal
> > > <hemant.agrawal@nxp.com>
> > > Subject: Re: [dpdk-dev] [PATCH v5] build: disable experimental API check internally
> > >
> > > On Fri, Apr 17, 2020 at 12:21 PM Trahe, Fiona <fiona.trahe@intel.com> wrote:
> > > > I see this is already applied.
> > > >
> > > > However,
> > > > rte_cryptodev_queue_pair_setup() calls
> > > > rte_cryptodev_sym_get_existing_header_session_size()
> > > > The former is a stable API, the latter is experimental.
> > > > So I expect the build to break when ALLOW_EXPERIMENTAL_API is disabled.
> > [Fiona] Thanks for confirming where the flag is.
> > But I think you've missed my point.
> > What about this problem?
>
> - dpdk-test-crypto-perf is built as part of the dpdk compilation itself.
> There is no user to be made aware of its use of experimental API.
>
> Now if you are talking about how the crypto API is bent in that it
> exposes a stable ABI with an underlying experimental ABI, this has
> nothing to do with the flag change.
[Fiona] Before this if an application didn't set ALLOW_EXPERIMENTAL_API
(which I expect is the default for many apps) then the build worked fine
as long as the app didn't directly call an experimental API. The crypto lib
still built ok as its own Makefile had the flag.
I just tried this with dpdk-test-crypto-perf. I had to remove one direct call,
without that it built ok. When I remove the flag from the crypto lib Makefile, the
build breaks as expected due to the above issue.
So, yes, cryptodev lib should be fixed.
However this patch just applied will potentially break builds for many apps!
It could expose many other issues of internal dependencies on experimental APIs.
>
> - But if you want to check crypto experimental api, then try to
> disable the flag in examples making use of them.
>
>
> --
> David Marchand
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCHv3] Remove validate-abi.sh from tree
2020-04-17 12:10 4% ` Thomas Monjalon
@ 2020-04-17 15:42 9% ` Ray Kinsella
2020-04-17 16:10 4% ` Thomas Monjalon
0 siblings, 1 reply; 200+ results
From: Ray Kinsella @ 2020-04-17 15:42 UTC (permalink / raw)
To: Thomas Monjalon, Neil Horman; +Cc: dev, david.marchand
On 17/04/2020 13:10, Thomas Monjalon wrote:
> 17/04/2020 13:47, Ray Kinsella:
>> On 17/04/2020 11:20, Thomas Monjalon wrote:
>>> 17/04/2020 12:11, Ray Kinsella:
>>>> check-abi.sh appears to be backward step in terms of usability.
>>>
>>> No, check-abi.sh benefits from a nice integration in build scripts.
>>> See below.
>>>
>>>> With validate-abi.sh I do can do a "validate-abi.sh HEAD~1 HEAD".
>>>> And it will do the build, install, dump and comparison for me.
>>>> And it picked up my 20.0.2 - > 21.0 changes no problem.
>>>>
>>>> With check-abi on the other hand, I need to the build and install myself.
>>>> check-abi requires dump files, but I see no reference in the documentation to how these are created.
>>>> It silently fails when it doesn't find any ...
>>>>
>>>> Do I run abi-dumper on the so's myself, or how does it work?
>>>
>>> check-abi.sh is integrated in test-build.sh and test-meson-builds.sh.
>>> Probably we should document usage in these scripts.
>>
>> Looks like I need to set DPDK_ABI_REF_VERSION=master, not obvious.
>> Any tips or tricks would be welcome.
>
> export DPDK_ABI_REF_VERSION=v20.02
> or
> export DPDK_ABI_REF_VERSION=v19.11
>
> Depends on which compatibility you want to test...
>
Few things ...
1. test-meson-build.sh keep barfing complaining about reference paths.
ValueError: dst_dir must be absolute, got reference/v19.11/build-gcc-static/usr/local/share/dpdk/examples/bbdev_app
Under the hood, ninja install is failing complaining that it needs an absolute path.
I fixed this in test_meson_build.sh and will send a patch in a minute.
Though it's strange no-one else has seen it?
2. test-meson-build.sh compares the abi for the static builds, which doesn't make any sense.
3. test-meson-build.sh will only take a branch in DPDK_ABI_REF_VERSION that exists locally.
In order to get it to compare HEAD against HEAD~1, which you would imagine is a pretty common case.
I had a create a branch for HEAD~1, in validate-abi this a pretty simple `validate-abi HEAD~1 HEAD`
Thanks,
Ray K
Ray K
^ permalink raw reply [relevance 9%]
* Re: [dpdk-dev] [PATCH v5] build: disable experimental API check internally
2020-04-17 15:05 0% ` Trahe, Fiona
@ 2020-04-17 15:52 0% ` Trahe, Fiona
2020-04-18 19:43 3% ` Chautru, Nicolas
0 siblings, 1 reply; 200+ results
From: Trahe, Fiona @ 2020-04-17 15:52 UTC (permalink / raw)
To: David Marchand
Cc: dev, Jerin Jacob, Pavan Nikhilesh, Richardson, Bruce,
Thomas Monjalon, Yigit, Ferruh, Hemant Agrawal, Trahe, Fiona
Hi David,
> -----Original Message-----
> From: Trahe, Fiona <fiona.trahe@intel.com>
> Sent: Friday, April 17, 2020 4:05 PM
> To: David Marchand <david.marchand@redhat.com>
> Cc: dev <dev@dpdk.org>; Jerin Jacob <jerinjacobk@gmail.com>; Pavan Nikhilesh
> <pbhagavatula@marvell.com>; Richardson, Bruce <bruce.richardson@intel.com>; Thomas Monjalon
> <thomas@monjalon.net>; Yigit, Ferruh <ferruh.yigit@intel.com>; Hemant Agrawal
> <hemant.agrawal@nxp.com>; Trahe, Fiona <fiona.trahe@intel.com>
> Subject: RE: [dpdk-dev] [PATCH v5] build: disable experimental API check internally
>
> Hi Davd,
>
> > -----Original Message-----
> > From: David Marchand <david.marchand@redhat.com>
> > Sent: Friday, April 17, 2020 2:56 PM
> > To: Trahe, Fiona <fiona.trahe@intel.com>
> > Cc: dev <dev@dpdk.org>; Jerin Jacob <jerinjacobk@gmail.com>; Pavan Nikhilesh
> > <pbhagavatula@marvell.com>; Richardson, Bruce <bruce.richardson@intel.com>; Thomas
> Monjalon
> > <thomas@monjalon.net>; Yigit, Ferruh <ferruh.yigit@intel.com>; Hemant Agrawal
> > <hemant.agrawal@nxp.com>
> > Subject: Re: [dpdk-dev] [PATCH v5] build: disable experimental API check internally
> >
> > On Fri, Apr 17, 2020 at 3:44 PM Trahe, Fiona <fiona.trahe@intel.com> wrote:
> > >
> > > Hi David,
> > >
> > > > -----Original Message-----
> > > > From: David Marchand <david.marchand@redhat.com>
> > > > Sent: Friday, April 17, 2020 2:23 PM
> > > > To: Trahe, Fiona <fiona.trahe@intel.com>
> > > > Cc: dev <dev@dpdk.org>; Jerin Jacob <jerinjacobk@gmail.com>; Pavan Nikhilesh
> > > > <pbhagavatula@marvell.com>; Richardson, Bruce <bruce.richardson@intel.com>; Thomas
> > Monjalon
> > > > <thomas@monjalon.net>; Yigit, Ferruh <ferruh.yigit@intel.com>; Hemant Agrawal
> > > > <hemant.agrawal@nxp.com>
> > > > Subject: Re: [dpdk-dev] [PATCH v5] build: disable experimental API check internally
> > > >
> > > > On Fri, Apr 17, 2020 at 12:21 PM Trahe, Fiona <fiona.trahe@intel.com> wrote:
> > > > > I see this is already applied.
> > > > >
> > > > > However,
> > > > > rte_cryptodev_queue_pair_setup() calls
> > > > > rte_cryptodev_sym_get_existing_header_session_size()
> > > > > The former is a stable API, the latter is experimental.
> > > > > So I expect the build to break when ALLOW_EXPERIMENTAL_API is disabled.
> > > [Fiona] Thanks for confirming where the flag is.
> > > But I think you've missed my point.
> > > What about this problem?
> >
> > - dpdk-test-crypto-perf is built as part of the dpdk compilation itself.
> > There is no user to be made aware of its use of experimental API.
> >
> > Now if you are talking about how the crypto API is bent in that it
> > exposes a stable ABI with an underlying experimental ABI, this has
> > nothing to do with the flag change.
> [Fiona] Before this if an application didn't set ALLOW_EXPERIMENTAL_API
> (which I expect is the default for many apps) then the build worked fine
> as long as the app didn't directly call an experimental API. The crypto lib
> still built ok as its own Makefile had the flag.
> I just tried this with dpdk-test-crypto-perf. I had to remove one direct call,
> without that it built ok. When I remove the flag from the crypto lib Makefile, the
> build breaks as expected due to the above issue.
>
> So, yes, cryptodev lib should be fixed.
> However this patch just applied will potentially break builds for many apps!
> It could expose many other issues of internal dependencies on experimental APIs.
[Fiona] Ok, I get it now, there's no issue.
I see now why the patch name was changed from "global" to "internal"!
I had understood that the flag the application set or didn't set would ripple down
through the whole build. Instead the flag is set (or not set) once by the application
only affecting direct API calls and once for the rest of the build, which includes
apps in the app folder, but not apps in the examples folder.
So allowing experimental APIs internally, while disallowing them externally.
A bit confusing, but ok, doesn't break anything and definitely better than
having flags in every Make/meson file.
Fiona
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
2020-04-17 11:46 5% ` Trahe, Fiona
@ 2020-04-17 16:01 0% ` Ray Kinsella
0 siblings, 0 replies; 200+ results
From: Ray Kinsella @ 2020-04-17 16:01 UTC (permalink / raw)
To: Trahe, Fiona, Thomas Monjalon, Richardson, Bruce
Cc: dev, Kusztal, ArkadiuszX, Neil Horman, Luca Boccassi,
Kevin Traynor, Yigit, Ferruh
On 17/04/2020 12:46, Trahe, Fiona wrote:
> Hi all,
>
>> -----Original Message-----
>> From: Ray Kinsella <mdr@ashroe.eu>
>> Sent: Friday, April 17, 2020 11:34 AM
>> To: Thomas Monjalon <thomas@monjalon.net>; Richardson, Bruce <bruce.richardson@intel.com>
>> Cc: Trahe, Fiona <fiona.trahe@intel.com>; dev@dpdk.org; Kusztal, ArkadiuszX
>> <arkadiuszx.kusztal@intel.com>; Neil Horman <nhorman@tuxdriver.com>; Luca Boccassi
>> <bluca@debian.org>; Kevin Traynor <ktraynor@redhat.com>; Yigit, Ferruh <ferruh.yigit@intel.com>
>> Subject: Re: [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function
>>
>>
>>
>> On 17/04/2020 11:17, Thomas Monjalon wrote:
>>> 17/04/2020 11:42, Ray Kinsella:
>>>> On 17/04/2020 10:31, Bruce Richardson wrote:
>>>>> On Fri, Apr 17, 2020 at 08:24:30AM +0100, Ray Kinsella wrote:
>>>>>> On 16/04/2020 11:01, Thomas Monjalon wrote:
>>>>>>> 16/04/2020 11:51, Bruce Richardson:
>>>>>>>> On Wed, Apr 15, 2020 at 06:24:19PM +0100, Trahe, Fiona wrote:
>>>>>>>>> 5a. If in 20.05 we add a version of a fn which breaks ABI 20.0, what should the name of the
>> original function be? fn_v20, or fn_v20.0
>>>>>>>>
>>>>>>>> In technical terms it really doesn't matter, it's just a name that will be
>>>>>>>> looked up in a table. I don't think we strictly enforce the naming, so
>>>>>>>> whatever is clearest is best. I'd suggest the former.
>>>>>>>
>>>>>>> Each release can have a new ABI.
>>>>>>
>>>>>> How many ABI's do we want to support?
>>>>>>
>>>>> It's not how many we want to support, but for me it's a matter of how many
>>>>> do we need to support. If an API is part of the stable set, it can't just
>>>>> drop to being experimental for one or two releases - it's always stable
>>>>> until deprecated. We also shouldn't have a situation where release 20.08 is
>>>>> ABI compatible with 19.11 but not 20.02 and 20.05.
>>>>
>>>> True. Let me say it differently.
>>>>
>>>> Our only commitment is to support v20 - 19.11
>>>> However you are correct, if something gets committed as v21 in 20.02, in practise should also be
>> there in 20.05+ also.
>>>> Because if it is committed as v21 and not as experimental, it should not be changing once
>> committed.
>>>>
>>>> In answering Thomas,
>>>> I was more commenting on the proliferation of ABI numbers & symbols we need to track in the
>> build.
>>>> With v20, v21 & Experimental we need to keep track of 3.
>>>> If we start allowing quarterly builds to have managed ABI's, it will get confusing.
>>>
>>> I don't remember why we are using intermediate ABI versions
>>> between v20 and v21.
>>> If we can use v21 for new ABI and make sure compatibility is maintained
>>> between all versions from 19.11 to 20.08, I'm fine.
> [Fiona] Here's a hypothetical case, but it illustrates why I don't think there
> should be an expectation to maintain ABI compatibility here.
> Example: in 20.05 add a new info_get_v21() which includes ChaChaPoly.
> In 20.08 add another new algorithm. info_get_v21() return now includes this.
> info_get_v21() will become stable in 20.11 and compatibility must be maintained from then on.
Yes, and the v20 version gets removed in 20.11.
> In the meantime, the fn is not experimental - that wouldn't be appropriate as it was a stable API.
Well it could be ... these are just labels after all.
So it entirely possible and permitted to have a info_get@DPDK_20.0 and info_get@EXPERIMENTAL simultaneously.
In fact, if it is your expectation that new version will change in successively quarterly releases.
That suggests it should be experimental.
There in a gap here, we appear to be missing the required MACROS to support a v20.0 + EXPERIMENTAL simultaneously.
As I remember this api was particularly sensitive to size of an enumeration?
Has that been resolved to make the api less brittle?
> But an app either wants stability and so should build against 19.11, or if prepared to move up to
> one non-stable-ABI quarterly release should be willing to rebuild for the next non-stable-ABI quarterly release.
> I think it's an unnecessary burden to require ABI compatibility across quarterly releases.
I generally agree with this point.
Our only cast iron solid commitment is compatibility with 19.11.
That said, once any API has dropped the EXPERIMENTAL label, it shouldn't be changing.
All things being equal quarterly releases should be compatible with each other, as v21 APIs should not be changing.
> And if required could end up with the version tracking hassle Ray referred to above with fn versions
> of 20.0.1, 20.0.2, 20.0.3, v21, and potentially several versions of same fn.
>
>
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCHv3] Remove validate-abi.sh from tree
2020-04-17 15:42 9% ` Ray Kinsella
@ 2020-04-17 16:10 4% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-04-17 16:10 UTC (permalink / raw)
To: Neil Horman, Ray Kinsella; +Cc: dev, david.marchand
17/04/2020 17:42, Ray Kinsella:
> On 17/04/2020 13:10, Thomas Monjalon wrote:
> > 17/04/2020 13:47, Ray Kinsella:
> >> On 17/04/2020 11:20, Thomas Monjalon wrote:
> >>> 17/04/2020 12:11, Ray Kinsella:
> >>>> check-abi.sh appears to be backward step in terms of usability.
> >>>
> >>> No, check-abi.sh benefits from a nice integration in build scripts.
> >>> See below.
> >>>
> >>>> With validate-abi.sh I do can do a "validate-abi.sh HEAD~1 HEAD".
> >>>> And it will do the build, install, dump and comparison for me.
> >>>> And it picked up my 20.0.2 - > 21.0 changes no problem.
> >>>>
> >>>> With check-abi on the other hand, I need to the build and install myself.
> >>>> check-abi requires dump files, but I see no reference in the documentation to how these are created.
> >>>> It silently fails when it doesn't find any ...
> >>>>
> >>>> Do I run abi-dumper on the so's myself, or how does it work?
> >>>
> >>> check-abi.sh is integrated in test-build.sh and test-meson-builds.sh.
> >>> Probably we should document usage in these scripts.
> >>
> >> Looks like I need to set DPDK_ABI_REF_VERSION=master, not obvious.
> >> Any tips or tricks would be welcome.
> >
> > export DPDK_ABI_REF_VERSION=v20.02
> > or
> > export DPDK_ABI_REF_VERSION=v19.11
> >
> > Depends on which compatibility you want to test...
> >
>
> Few things ...
>
> 1. test-meson-build.sh keep barfing complaining about reference paths.
> ValueError: dst_dir must be absolute, got reference/v19.11/build-gcc-static/usr/local/share/dpdk/examples/bbdev_app
>
> Under the hood, ninja install is failing complaining that it needs an absolute path.
> I fixed this in test_meson_build.sh and will send a patch in a minute.
> Though it's strange no-one else has seen it?
I set an absolute path in DPDK_ABI_REF_DIR.
Not sure you can really fix it. What would be the root dir?
> 2. test-meson-build.sh compares the abi for the static builds, which doesn't make any sense.
Yes
> 3. test-meson-build.sh will only take a branch in DPDK_ABI_REF_VERSION that exists locally.
> In order to get it to compare HEAD against HEAD~1, which you would imagine is a pretty common case.
> I had a create a branch for HEAD~1, in validate-abi this a pretty simple `validate-abi HEAD~1 HEAD`
Why is it a common case? You want to compare with a tag. Why something else?
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v1] common/mlx5: remove devx depndency on ibv and dv
2020-04-16 20:00 4% ` Thomas Monjalon
@ 2020-04-17 16:19 0% ` Ferruh Yigit
0 siblings, 0 replies; 200+ results
From: Ferruh Yigit @ 2020-04-17 16:19 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Matan Azrad, Raslan Darawsheh, Ophir Munk, David Marchand, dev,
Olga Shern, Asaf Penso, Kinsella, Ray, Neil Horman, Kevin Laatz,
hemant.agrawal, Haiyue Wang, Sunil Kumar Kori
On 4/16/2020 9:00 PM, Thomas Monjalon wrote:
> 16/04/2020 19:35, Ferruh Yigit:
>> On 4/9/2020 8:24 AM, David Marchand wrote:
>>> On Wed, Apr 8, 2020 at 7:12 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>>>> On 4/1/2020 10:59 AM, Raslan Darawsheh wrote:
>>>>> From: Ophir Munk <ophirmu@mellanox.com>
>>>>>>
>>>>>> File mlx5_devx_cmds.c should contain pure DevX calls. It must be OS
>>>>>> agnostic and not include any references to ibv or dv structs (defined in
>>>>>> ibverbs and rdma-core linux libraries). This commit replaces all ibv and
>>>>>> dv references with 'void *'. Specifically, the following struct were
>>>>>> replaced:
>>>>>> 1. struct ibv_context *
>>>>>> 2. struct ibv_qp *
>>>>>> 3. struct mlx5dv_devx_cmd_comp *
>>>>>>
>>>>>> Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
>>>>>
>>>>> Patch applied to next-net-mlx,
>>>>>
>>>>
>>>> Hi David,
>>>>
>>>> This patch is failing in the travis for ABI checks [1], since mlx has APIs now
>>>> [2], are they public APIs or internal ones, and are they part of the ABI policy,
>>>> can you please check this?
>>>
>>> - What I see on patchwork and test-report ml for this patch:
>>> http://patchwork.dpdk.org/patch/67367/
>>>
>>> Ophir proposed a patch on 03/30.
>>>
>>> The robot reported an issue on 03/30, and I suppose Ophir got a report.
>>> https://mails.dpdk.org/archives/test-report/2020-March/122623.html
>>> https://travis-ci.com/github/ovsrobot/dpdk/jobs/308057800#L2337
>>>
>>> Matan acked the patch on 03/31.
>>>
>>> Rasland merged the patch on 04/01.
>>>
>>> I understand that the abi checks are not perfect, and people need help
>>> with the new abi checks.
>>> Prove me wrong, but here, I get the feeling that it was just ignored
>>> by 3 people in a row.
>>>
>>> - On the question if these should be public API or internal, that is
>>> not for me to reply/investigate.
>>> This is a question for Mellanox.
>>>
>>
>> Hi Matan, Raslan, Ophir,
>>
>> First can you please clarify if these APIs are internal or public?
>
> As most of common drivers, some functions are exported to be
> used by some PMDs. So they are not part of the API/ABI and should be skipped
> by ABI checks.
>
>> And later if the ABI break issue is not clarified I may need to drop these
>> patches. Right now they fail in travis!
>
> Yes, it fails and could it be avoided with some libabigail config.
> But the real solution is to mark internal symbols, and we are waiting
> for rte_internal patchset to be completed and merged.
>
> Ferruh, please let's not bloat libabigail config,
> and reject any patch failing ABI checks.
>
> As a consequence, this patch must be dropped until it uses rte_internal.
> Thanks
>
>
dropped the patch from next-net and updated its patchwork status.
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2] ethdev: support flow aging
@ 2020-04-17 22:00 3% ` Ferruh Yigit
2020-04-17 22:07 0% ` Stephen Hemminger
2020-04-18 5:04 0% ` Bill Zhou
0 siblings, 2 replies; 200+ results
From: Ferruh Yigit @ 2020-04-17 22:00 UTC (permalink / raw)
To: Dong Zhou, orika, matan, wenzhuo.lu, jingjing.wu,
bernard.iremonger, john.mcnamara, marko.kovacevic, thomas,
arybchenko
Cc: dev
On 4/14/2020 9:32 AM, Dong Zhou wrote:
> One of the reasons to destroy a flow is the fact that no packet matches the
> flow for "timeout" time.
> For example, when TCP\UDP sessions are suddenly closed.
>
> Currently, there is not any DPDK mechanism for flow aging and the
> applications use their own ways to detect and destroy aged-out flows.
>
> The flow aging implementation need include:
> - A new rte_flow action: RTE_FLOW_ACTION_TYPE_AGE to set the timeout and
> the application flow context for each flow.
> - A new ethdev event: RTE_ETH_EVENT_FLOW_AGED for the driver to report
> that there are new aged-out flows.
> - A new rte_flow API: rte_flow_get_aged_flows to get the aged-out flows
> contexts from the port.
> - Support input flow aging command line in Testpmd.
>
> Signed-off-by: Dong Zhou <dongz@mellanox.com>
<...>
> --- a/lib/librte_ethdev/rte_ethdev.h
> +++ b/lib/librte_ethdev/rte_ethdev.h
> @@ -3015,6 +3015,7 @@ enum rte_eth_event_type {
> RTE_ETH_EVENT_NEW, /**< port is probed */
> RTE_ETH_EVENT_DESTROY, /**< port is released */
> RTE_ETH_EVENT_IPSEC, /**< IPsec offload related event */
> + RTE_ETH_EVENT_FLOW_AGED,/**< New aged-out flows is detected */
> RTE_ETH_EVENT_MAX /**< max value of this enum */
> };
Just recognized that this is failing in ABI check [1], as far as last time for a
similar enum warning a QAT patch has been dropped, should this need to wait for
20.11 too?
[1]
[C]'function int _rte_eth_dev_callback_process(rte_eth_dev*,
rte_eth_event_type, void*)' at rte_ethdev.c:4063:1 has some indirect sub-type
changes:
parameter 2 of type 'enum rte_eth_event_type' has sub-type changes:
type size hasn't changed
1 enumerator insertion:
'rte_eth_event_type::RTE_ETH_EVENT_FLOW_AGED' value '10'
1 enumerator change:
'rte_eth_event_type::RTE_ETH_EVENT_MAX' from value '10' to '11' at
rte_ethdev.h:3008:1
^ permalink raw reply [relevance 3%]
* Re: [dpdk-dev] [PATCH v2] ethdev: support flow aging
2020-04-17 22:00 3% ` Ferruh Yigit
@ 2020-04-17 22:07 0% ` Stephen Hemminger
2020-04-18 5:04 0% ` Bill Zhou
1 sibling, 0 replies; 200+ results
From: Stephen Hemminger @ 2020-04-17 22:07 UTC (permalink / raw)
To: Ferruh Yigit
Cc: Dong Zhou, orika, matan, wenzhuo.lu, jingjing.wu,
bernard.iremonger, john.mcnamara, marko.kovacevic, thomas,
arybchenko, dev
On Fri, 17 Apr 2020 23:00:57 +0100
Ferruh Yigit <ferruh.yigit@intel.com> wrote:
> On 4/14/2020 9:32 AM, Dong Zhou wrote:
> > One of the reasons to destroy a flow is the fact that no packet matches the
> > flow for "timeout" time.
> > For example, when TCP\UDP sessions are suddenly closed.
> >
> > Currently, there is not any DPDK mechanism for flow aging and the
> > applications use their own ways to detect and destroy aged-out flows.
> >
> > The flow aging implementation need include:
> > - A new rte_flow action: RTE_FLOW_ACTION_TYPE_AGE to set the timeout and
> > the application flow context for each flow.
> > - A new ethdev event: RTE_ETH_EVENT_FLOW_AGED for the driver to report
> > that there are new aged-out flows.
> > - A new rte_flow API: rte_flow_get_aged_flows to get the aged-out flows
> > contexts from the port.
> > - Support input flow aging command line in Testpmd.
> >
> > Signed-off-by: Dong Zhou <dongz@mellanox.com>
>
> <...>
>
> > --- a/lib/librte_ethdev/rte_ethdev.h
> > +++ b/lib/librte_ethdev/rte_ethdev.h
> > @@ -3015,6 +3015,7 @@ enum rte_eth_event_type {
> > RTE_ETH_EVENT_NEW, /**< port is probed */
> > RTE_ETH_EVENT_DESTROY, /**< port is released */
> > RTE_ETH_EVENT_IPSEC, /**< IPsec offload related event */
> > + RTE_ETH_EVENT_FLOW_AGED,/**< New aged-out flows is detected */
> > RTE_ETH_EVENT_MAX /**< max value of this enum */
> > };
>
>
> Just recognized that this is failing in ABI check [1], as far as last time for a
> similar enum warning a QAT patch has been dropped, should this need to wait for
> 20.11 too?
>
>
> [1]
> [C]'function int _rte_eth_dev_callback_process(rte_eth_dev*,
> rte_eth_event_type, void*)' at rte_ethdev.c:4063:1 has some indirect sub-type
> changes:
> parameter 2 of type 'enum rte_eth_event_type' has sub-type changes:
> type size hasn't changed
> 1 enumerator insertion:
> 'rte_eth_event_type::RTE_ETH_EVENT_FLOW_AGED' value '10'
> 1 enumerator change:
> 'rte_eth_event_type::RTE_ETH_EVENT_MAX' from value '10' to '11' at
> rte_ethdev.h:3008:1
>
For 20.11, those _MAX values need to be removed from enums
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2] ethdev: support flow aging
2020-04-17 22:00 3% ` Ferruh Yigit
2020-04-17 22:07 0% ` Stephen Hemminger
@ 2020-04-18 5:04 0% ` Bill Zhou
2020-04-18 9:44 0% ` Thomas Monjalon
1 sibling, 1 reply; 200+ results
From: Bill Zhou @ 2020-04-18 5:04 UTC (permalink / raw)
To: Ferruh Yigit, Ori Kam, Matan Azrad, wenzhuo.lu, jingjing.wu,
bernard.iremonger, john.mcnamara, marko.kovacevic,
Thomas Monjalon, arybchenko
Cc: dev
> -----Original Message-----
> From: Ferruh Yigit <ferruh.yigit@intel.com>
> Sent: Saturday, April 18, 2020 6:01 AM
> To: Bill Zhou <dongz@mellanox.com>; Ori Kam <orika@mellanox.com>;
> Matan Azrad <matan@mellanox.com>; wenzhuo.lu@intel.com;
> jingjing.wu@intel.com; bernard.iremonger@intel.com;
> john.mcnamara@intel.com; marko.kovacevic@intel.com; Thomas Monjalon
> <thomas@monjalon.net>; arybchenko@solarflare.com
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v2] ethdev: support flow aging
>
> On 4/14/2020 9:32 AM, Dong Zhou wrote:
> > One of the reasons to destroy a flow is the fact that no packet
> > matches the flow for "timeout" time.
> > For example, when TCP\UDP sessions are suddenly closed.
> >
> > Currently, there is not any DPDK mechanism for flow aging and the
> > applications use their own ways to detect and destroy aged-out flows.
> >
> > The flow aging implementation need include:
> > - A new rte_flow action: RTE_FLOW_ACTION_TYPE_AGE to set the
> timeout and
> > the application flow context for each flow.
> > - A new ethdev event: RTE_ETH_EVENT_FLOW_AGED for the driver to
> report
> > that there are new aged-out flows.
> > - A new rte_flow API: rte_flow_get_aged_flows to get the aged-out flows
> > contexts from the port.
> > - Support input flow aging command line in Testpmd.
> >
> > Signed-off-by: Dong Zhou <dongz@mellanox.com>
>
> <...>
>
> > --- a/lib/librte_ethdev/rte_ethdev.h
> > +++ b/lib/librte_ethdev/rte_ethdev.h
> > @@ -3015,6 +3015,7 @@ enum rte_eth_event_type {
> > RTE_ETH_EVENT_NEW, /**< port is probed */
> > RTE_ETH_EVENT_DESTROY, /**< port is released */
> > RTE_ETH_EVENT_IPSEC, /**< IPsec offload related event */
> > + RTE_ETH_EVENT_FLOW_AGED,/**< New aged-out flows is detected
> */
> > RTE_ETH_EVENT_MAX /**< max value of this enum */
> > };
>
>
> Just recognized that this is failing in ABI check [1], as far as last time for a
> similar enum warning a QAT patch has been dropped, should this need to
> wait for
> 20.11 too?
This patch is commonly used for flow aging, there are 2 other patches have
implement flow aging in mlx5 driver reply to this patch.
In our schedule, this feature is merged in 20.05 for some customers. Can it
be fixed?
>
>
> [1]
> [C]'function int _rte_eth_dev_callback_process(rte_eth_dev*,
> rte_eth_event_type, void*)' at rte_ethdev.c:4063:1 has some indirect sub-
> type
> changes:
> parameter 2 of type 'enum rte_eth_event_type' has sub-type changes:
> type size hasn't changed
> 1 enumerator insertion:
> 'rte_eth_event_type::RTE_ETH_EVENT_FLOW_AGED' value '10'
> 1 enumerator change:
> 'rte_eth_event_type::RTE_ETH_EVENT_MAX' from value '10' to '11' at
> rte_ethdev.h:3008:1
^ permalink raw reply [relevance 0%]
* Re: [dpdk-dev] [PATCH v2] ethdev: support flow aging
2020-04-18 5:04 0% ` Bill Zhou
@ 2020-04-18 9:44 0% ` Thomas Monjalon
0 siblings, 0 replies; 200+ results
From: Thomas Monjalon @ 2020-04-18 9:44 UTC (permalink / raw)
To: Ferruh Yigit, Ori Kam, Matan Azrad, wenzhuo.lu, jingjing.wu,
bernard.iremonger, john.mcnamara, marko.kovacevic, arybchenko,
Bill Zhou
Cc: dev
18/04/2020 07:04, Bill Zhou:
> From: Ferruh Yigit <ferruh.yigit@intel.com>
> > On 4/14/2020 9:32 AM, Dong Zhou wrote:
> > > --- a/lib/librte_ethdev/rte_ethdev.h
> > > +++ b/lib/librte_ethdev/rte_ethdev.h
> > > @@ -3015,6 +3015,7 @@ enum rte_eth_event_type {
> > > RTE_ETH_EVENT_NEW, /**< port is probed */
> > > RTE_ETH_EVENT_DESTROY, /**< port is released */
> > > RTE_ETH_EVENT_IPSEC, /**< IPsec offload related event */
> > > + RTE_ETH_EVENT_FLOW_AGED,/**< New aged-out flows is detected
> > */
> > > RTE_ETH_EVENT_MAX /**< max value of this enum */
> > > };
> >
> >
> > Just recognized that this is failing in ABI check [1], as far as last time for a
> > similar enum warning a QAT patch has been dropped, should this need to
> > wait for
> > 20.11 too?
>
> This patch is commonly used for flow aging, there are 2 other patches have
> implement flow aging in mlx5 driver reply to this patch.
> In our schedule, this feature is merged in 20.05 for some customers. Can it
> be fixed?
These MAX values in enums are a pain.
We can try to think what can be done, waiting 20.11.
Not sure there is a solution, except hijacking an existing value
not used in the PMD, waiting the definitive value in 20.11...
^ permalink raw reply [relevance 0%]
* [dpdk-dev] [PATCH v7 0/2] support for VFIO-PCI VF token interface
` (2 preceding siblings ...)
2020-04-14 3:21 4% ` [dpdk-dev] [PATCH v6 " Haiyue Wang
@ 2020-04-18 11:16 4% ` Haiyue Wang
2020-04-18 17:30 4% ` [dpdk-dev] [PATCH v8 " Haiyue Wang
4 siblings, 0 replies; 200+ results
From: Haiyue Wang @ 2020-04-18 11:16 UTC (permalink / raw)
To: dev, anatoly.burakov, thomas, vattunuru, jerinj, alex.williamson,
david.marchand
Cc: Haiyue Wang
v7: Add the Fixes tag in uuid, the release note and help
document.
v6: Drop the Fixes tag in uuid, since the file has been
moved to another place, not suitable to apply on stable.
And this is not a bug, just some kind of enhancement.
https://patchwork.dpdk.org/cover/68367/
v5: 1. Add the VF token parse error handling.
2. Split into two patches for different logic module.
3. Add more comments into the code for explaining the design.
4. Drop the ABI change workaround, this patch set focuses on code review.
https://patchwork.dpdk.org/cover/68364/
v4: 1. Ignore rte_vfio_setup_device ABI check since it is
for Linux driver use.
https://patchwork.dpdk.org/patch/68255/
v3: Fix the Travis build failed:
(1). rte_uuid.h:97:55: error: unknown type name ‘size_t’
(2). rte_uuid.h:58:2: error: implicit declaration of function ‘memcpy’
https://patchwork.dpdk.org/patch/68254/
v2: Fix the FreeBSD build error.
https://patchwork.dpdk.org/patch/68240/
v1: Update the commit message.
https://patchwork.dpdk.org/patch/68237/
RFC v2: https://patchwork.dpdk.org/patch/68114/
Based on Vamsi's RFC v1, and Alex's patch for Qemu
[https://lore.kernel.org/lkml/20200204161737.34696b91@w520.home/]:
Use the devarg to pass-down the VF token.
RFC v1: https://patchwork.dpdk.org/patch/66281/ by Vamsi.
Haiyue Wang (2):
eal: add uuid dependent header files explicitly
eal: support for VFIO-PCI VF token
doc/guides/linux_gsg/linux_drivers.rst | 23 +++++++-
doc/guides/rel_notes/release_20_05.rst | 5 ++
drivers/bus/pci/linux/pci_vfio.c | 74 +++++++++++++++++++++++++-
lib/librte_eal/freebsd/eal.c | 3 +-
lib/librte_eal/include/rte_uuid.h | 2 +
lib/librte_eal/include/rte_vfio.h | 21 +++++++-
lib/librte_eal/linux/eal_vfio.c | 20 +++++--
7 files changed, 140 insertions(+), 8 deletions(-)
--
2.26.1
^ permalink raw reply [relevance 4%]
* [dpdk-dev] [PATCH v5 0/9] New sync modes for ring
2020-04-17 13:36 3% ` [dpdk-dev] [PATCH v4 0/9] New sync modes for ring Konstantin Ananyev
2020-04-17 13:36 9% ` [dpdk-dev] [PATCH v4 2/9] ring: prepare ring to allow new sync schemes Konstantin Ananyev
2020-04-17 13:36 1% ` [dpdk-dev] [PATCH v4 3/9] ring: introduce RTS ring mode Konstantin Ananyev
@ 2020-04-18 16:32 3% ` Konstantin Ananyev
2020-04-18 16:32 9% ` [dpdk-dev] [PATCH v5 2/9] ring: prepare ring to allow new sync schemes Konstantin Ananyev
2020-04-18 16:32 1% ` [dpdk-dev] [PATCH v5 3/9] ring: introduce RTS ring mode Konstantin Ananyev
2 siblings, 2 replies; 200+ results
From: Konstantin Ananyev @ 2020-04-18 16:32 UTC (permalink / raw)
To: dev; +Cc: honnappa.nagarahalli, david.marchand, jielong.zjl, Konstantin Ananyev
V4 - V5:
1. fix i686 clang build problem
2. fix formal API comments
V3 - V4 changes:
Address comments from Honnappa:
1. for new sync modes make legacy API wrappers around _elem_ calls
2. remove rte_ring_(hts|rts)_generic.h
3. few changes in C11 version
4. peek API - add missing functions for _elem_
5. remove _IS_SP/_IS_MP, etc. internal macros
6. fix param types (obj_table) for _elem_functions
7. fix formal API comments
8. deduplicate code for test_ring_stress
9. added functional tests for new sync modes
V2 - V3 changes:
1. few more compilation fixes (for gcc 4.8.X)
2. extra update devtools/libabigail.abignore (workaround)
V1 - V2 changes:
1. fix compilation issues
2. add C11 atomics support
3. updates devtools/libabigail.abignore (workaround)
RFC - V1 changes:
1. remove ABI brekage (at least I hope I did)
2. Add support for ring_elem
3. rework peek related API a bit
4. rework test to make it less verbose and unite all test-cases
in one command
5. add new test-case for MT peek API
TODO list:
1. Update docs
These days more and more customers use(/try to use) DPDK based apps within
overcommitted systems (multiple acttive threads over same pysical cores):
VM, container deployments, etc.
One quite common problem they hit:
Lock-Holder-Preemption/Lock-Waiter-Preemption with rte_ring.
LHP is quite a common problem for spin-based sync primitives
(spin-locks, etc.) on overcommitted systems.
The situation gets much worse when some sort of
fair-locking technique is used (ticket-lock, etc.).
As now not only lock-owner but also lock-waiters scheduling
order matters a lot (LWP).
These two problems are well-known for kernel within VMs:
http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
The problem with rte_ring is that while head accusion is sort of
un-fair locking, waiting on tail is very similar to ticket lock schema -
tail has to be updated in particular order.
That makes current rte_ring implementation to perform
really pure on some overcommited scenarios.
It is probably not possible to completely resolve LHP problem in
userspace only (without some kernel communication/intervention).
But removing fairness at tail update helps to avoid LWP and
can mitigate the situation significantly.
This patch proposes two new optional ring synchronization modes:
1) Head/Tail Sync (HTS) mode
In that mode enqueue/dequeue operation is fully serialized:
only one thread at a time is allowed to perform given op.
As another enhancement provide ability to split enqueue/dequeue
operation into two phases:
- enqueue/dequeue start
- enqueue/dequeue finish
That allows user to inspect objects in the ring without removing
them from it (aka MT safe peek).
2) Relaxed Tail Sync (RTS)
The main difference from original MP/MC algorithm is that
tail value is increased not by every thread that finished enqueue/dequeue,
but only by the last one.
That allows threads to avoid spinning on ring tail value,
leaving actual tail value change to the last thread in the update queue.
Note that these new sync modes are optional.
For current rte_ring users nothing should change
(both in terms of API/ABI and performance).
Existing sync modes MP/MC,SP/SC kept untouched, set up in the same
way (via flags and _init_), and MP/MC remains as default one.
The only thing that changed:
Format of prod/cons now could differ depending on mode selected at _init_.
So user has to stick with one sync model through whole ring lifetime.
In other words, user can't create a ring for let say SP mode and then
in the middle of data-path change his mind and start using MP_RTS mode.
For existing modes (SP/MP, SC/MC) format remains the same and
user can still use them interchangeably, though of course it is an
error prone practice.
Test results on IA (see below) show significant improvements
for average enqueue/dequeue op times on overcommitted systems.
For 'classic' DPDK deployments (one thread per core) original MP/MC
algorithm still shows best numbers, though for 64-bit target
RTS numbers are not that far away.
Numbers were produced by new UT test-case: ring_stress_autotest, i.e.:
echo ring_stress_autotest | ./dpdk-test -n 4 --lcores='...'
X86_64 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
DEQ+ENQ average cycles/obj
MP/MC HTS RTS
1thread@1core(--lcores=6-7) 8.00 8.15 8.99
2thread@2core(--lcores=6-8) 19.14 19.61 20.35
4thread@4core(--lcores=6-10) 29.43 29.79 31.82
8thread@8core(--lcores=6-14) 110.59 192.81 119.50
16thread@16core(--lcores=6-22) 461.03 813.12 495.59
32thread/@32core(--lcores='6-22,55-70') 982.90 1972.38 1160.51
2thread@1core(--lcores='6,(10-11)@7' 20140.50 23.58 25.14
4thread@2core(--lcores='6,(10-11)@7,(20-21)@8' 153680.60 76.88 80.05
8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 280314.32 294.72 318.79
16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 643176.59 1144.02 1175.14
32thread@2core(--lcores='6,(10-25)@7,(30-45)@8' 4264238.80 4627.48 4892.68
8thread@2core(--lcores='6,(10-17)@(7,8))' 321085.98 298.59 307.47
16thread@4core(--lcores='6,(20-35)@(7-10))' 1900705.61 575.35 678.29
32thread@4core(--lcores='6,(20-51)@(7-10))' 5510445.85 2164.36 2714.12
i686 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
DEQ+ENQ average cycles/obj
MP/MC HTS RTS
1thread@1core(--lcores=6-7) 7.85 12.13 11.31
2thread@2core(--lcores=6-8) 17.89 24.52 21.86
8thread@8core(--lcores=6-14) 32.58 354.20 54.58
32thread/@32core(--lcores='6-22,55-70') 813.77 6072.41 2169.91
2thread@1core(--lcores='6,(10-11)@7' 16095.00 36.06 34.74
8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 1140354.54 346.61 361.57
16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 1920417.86 1314.90 1416.65
8thread@2core(--lcores='6,(10-17)@(7,8))' 594358.61 332.70 357.74
32thread@4core(--lcores='6,(20-51)@(7-10))' 5319896.86 2836.44 3028.87
Konstantin Ananyev (9):
test/ring: add contention stress test
ring: prepare ring to allow new sync schemes
ring: introduce RTS ring mode
test/ring: add contention stress test for RTS ring
ring: introduce HTS ring mode
test/ring: add contention stress test for HTS ring
ring: introduce peek style API
test/ring: add stress test for MT peek API
test/ring: add functional tests for new sync modes
app/test/Makefile | 5 +
app/test/meson.build | 5 +
app/test/test_pdump.c | 6 +-
app/test/test_ring.c | 93 ++++--
app/test/test_ring_hts_stress.c | 32 ++
app/test/test_ring_mpmc_stress.c | 31 ++
app/test/test_ring_peek_stress.c | 43 +++
app/test/test_ring_rts_stress.c | 32 ++
app/test/test_ring_stress.c | 57 ++++
app/test/test_ring_stress.h | 38 +++
app/test/test_ring_stress_impl.h | 396 ++++++++++++++++++++++
devtools/libabigail.abignore | 7 +
lib/librte_pdump/rte_pdump.c | 2 +-
lib/librte_port/rte_port_ring.c | 12 +-
lib/librte_ring/Makefile | 8 +-
lib/librte_ring/meson.build | 11 +-
lib/librte_ring/rte_ring.c | 114 ++++++-
lib/librte_ring/rte_ring.h | 243 ++++++++------
lib/librte_ring/rte_ring_c11_mem.h | 44 +++
lib/librte_ring/rte_ring_core.h | 184 ++++++++++
lib/librte_ring/rte_ring_elem.h | 141 ++++++--
lib/librte_ring/rte_ring_generic.h | 48 +++
lib/librte_ring/rte_ring_hts.h | 332 +++++++++++++++++++
lib/librte_ring/rte_ring_hts_c11_mem.h | 207 ++++++++++++
lib/librte_ring/rte_ring_peek.h | 442 +++++++++++++++++++++++++
lib/librte_ring/rte_ring_rts.h | 439 ++++++++++++++++++++++++
lib/librte_ring/rte_ring_rts_c11_mem.h | 179 ++++++++++
27 files changed, 2977 insertions(+), 174 deletions(-)
create mode 100644 app/test/test_ring_hts_stress.c
create mode 100644 app/test/test_ring_mpmc_stress.c
create mode 100644 app/test/test_ring_peek_stress.c
create mode 100644 app/test/test_ring_rts_stress.c
create mode 100644 app/test/test_ring_stress.c
create mode 100644 app/test/test_ring_stress.h
create mode 100644 app/test/test_ring_stress_impl.h
create mode 100644 lib/librte_ring/rte_ring_core.h
create mode 100644 lib/librte_ring/rte_ring_hts.h
create mode 100644 lib/librte_ring/rte_ring_hts_c11_mem.h
create mode 100644 lib/librte_ring/rte_ring_peek.h
create mode 100644 lib/librte_ring/rte_ring_rts.h
create mode 100644 lib/librte_ring/rte_ring_rts_c11_mem.h
--
2.17.1
^ permalink raw reply [relevance 3%]
* [dpdk-dev] [PATCH v5 2/9] ring: prepare ring to allow new sync schemes
2020-04-18 16:32 3% ` [dpdk-dev] [PATCH v5 0/9] New sync modes for ring Konstantin Ananyev
@ 2020-04-18 16:32 9% ` Konstantin Ananyev
2020-04-18 16:32 1% ` [dpdk-dev] [PATCH v5 3/9] ring: introduce RTS ring mode Konstantin Ananyev
1 sibling, 0 replies; 200+ results
From: Konstantin Ananyev @ 2020-04-18 16:32 UTC (permalink / raw)
To: dev; +Cc: honnappa.nagarahalli, david.marchand, jielong.zjl, Konstantin Ananyev
To make these preparations two main things are done:
- Change from *single* to *sync_type* to allow different
synchronisation schemes to be applied.
Mark *single* as deprecated in comments.
Add new functions to allow user to query ring sync types.
Replace direct access to *single* with appropriate function call.
- Move actual rte_ring and related structures definitions into a
separate file: <rte_ring_core.h>. It allows to refer contents
of <rte_ring_elem.h> from <rte_ring.h> without introducing a
circular dependency.
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
app/test/test_pdump.c | 6 +-
lib/librte_pdump/rte_pdump.c | 2 +-
lib/librte_port/rte_port_ring.c | 12 +--
lib/librte_ring/Makefile | 1 +
lib/librte_ring/meson.build | 1 +
lib/librte_ring/rte_ring.c | 6 +-
lib/librte_ring/rte_ring.h | 170 ++++++++++++++------------------
lib/librte_ring/rte_ring_core.h | 132 +++++++++++++++++++++++++
lib/librte_ring/rte_ring_elem.h | 42 +++-----
9 files changed, 234 insertions(+), 138 deletions(-)
create mode 100644 lib/librte_ring/rte_ring_core.h
diff --git a/app/test/test_pdump.c b/app/test/test_pdump.c
index ad183184c..6a1180bcb 100644
--- a/app/test/test_pdump.c
+++ b/app/test/test_pdump.c
@@ -57,8 +57,7 @@ run_pdump_client_tests(void)
if (ret < 0)
return -1;
mp->flags = 0x0000;
- ring_client = rte_ring_create("SR0", RING_SIZE, rte_socket_id(),
- RING_F_SP_ENQ | RING_F_SC_DEQ);
+ ring_client = rte_ring_create("SR0", RING_SIZE, rte_socket_id(), 0);
if (ring_client == NULL) {
printf("rte_ring_create SR0 failed");
return -1;
@@ -71,9 +70,6 @@ run_pdump_client_tests(void)
}
rte_eth_dev_probing_finish(eth_dev);
- ring_client->prod.single = 0;
- ring_client->cons.single = 0;
-
printf("\n***** flags = RTE_PDUMP_FLAG_TX *****\n");
for (itr = 0; itr < NUM_ITR; itr++) {
diff --git a/lib/librte_pdump/rte_pdump.c b/lib/librte_pdump/rte_pdump.c
index 8a01ac510..f96709f95 100644
--- a/lib/librte_pdump/rte_pdump.c
+++ b/lib/librte_pdump/rte_pdump.c
@@ -380,7 +380,7 @@ pdump_validate_ring_mp(struct rte_ring *ring, struct rte_mempool *mp)
rte_errno = EINVAL;
return -1;
}
- if (ring->prod.single || ring->cons.single) {
+ if (rte_ring_is_prod_single(ring) || rte_ring_is_cons_single(ring)) {
PDUMP_LOG(ERR, "ring with either SP or SC settings"
" is not valid for pdump, should have MP and MC settings\n");
rte_errno = EINVAL;
diff --git a/lib/librte_port/rte_port_ring.c b/lib/librte_port/rte_port_ring.c
index 47fcdd06a..52b2d8e55 100644
--- a/lib/librte_port/rte_port_ring.c
+++ b/lib/librte_port/rte_port_ring.c
@@ -44,8 +44,8 @@ rte_port_ring_reader_create_internal(void *params, int socket_id,
/* Check input parameters */
if ((conf == NULL) ||
(conf->ring == NULL) ||
- (conf->ring->cons.single && is_multi) ||
- (!(conf->ring->cons.single) && !is_multi)) {
+ (rte_ring_is_cons_single(conf->ring) && is_multi) ||
+ (!rte_ring_is_cons_single(conf->ring) && !is_multi)) {
RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
return NULL;
}
@@ -171,8 +171,8 @@ rte_port_ring_writer_create_internal(void *params, int socket_id,
/* Check input parameters */
if ((conf == NULL) ||
(conf->ring == NULL) ||
- (conf->ring->prod.single && is_multi) ||
- (!(conf->ring->prod.single) && !is_multi) ||
+ (rte_ring_is_prod_single(conf->ring) && is_multi) ||
+ (!rte_ring_is_prod_single(conf->ring) && !is_multi) ||
(conf->tx_burst_sz > RTE_PORT_IN_BURST_SIZE_MAX)) {
RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
return NULL;
@@ -440,8 +440,8 @@ rte_port_ring_writer_nodrop_create_internal(void *params, int socket_id,
/* Check input parameters */
if ((conf == NULL) ||
(conf->ring == NULL) ||
- (conf->ring->prod.single && is_multi) ||
- (!(conf->ring->prod.single) && !is_multi) ||
+ (rte_ring_is_prod_single(conf->ring) && is_multi) ||
+ (!rte_ring_is_prod_single(conf->ring) && !is_multi) ||
(conf->tx_burst_sz > RTE_PORT_IN_BURST_SIZE_MAX)) {
RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
return NULL;
diff --git a/lib/librte_ring/Makefile b/lib/librte_ring/Makefile
index 28368e6d1..6572768c9 100644
--- a/lib/librte_ring/Makefile
+++ b/lib/librte_ring/Makefile
@@ -16,6 +16,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_RING) := rte_ring.c
# install includes
SYMLINK-$(CONFIG_RTE_LIBRTE_RING)-include := rte_ring.h \
+ rte_ring_core.h \
rte_ring_elem.h \
rte_ring_generic.h \
rte_ring_c11_mem.h
diff --git a/lib/librte_ring/meson.build b/lib/librte_ring/meson.build
index 05402e4f0..c656781da 100644
--- a/lib/librte_ring/meson.build
+++ b/lib/librte_ring/meson.build
@@ -3,6 +3,7 @@
sources = files('rte_ring.c')
headers = files('rte_ring.h',
+ 'rte_ring_core.h',
'rte_ring_elem.h',
'rte_ring_c11_mem.h',
'rte_ring_generic.h')
diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c
index 77e5de099..fa5733907 100644
--- a/lib/librte_ring/rte_ring.c
+++ b/lib/librte_ring/rte_ring.c
@@ -106,8 +106,10 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
if (ret < 0 || ret >= (int)sizeof(r->name))
return -ENAMETOOLONG;
r->flags = flags;
- r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
- r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
+ r->prod.sync_type = (flags & RING_F_SP_ENQ) ?
+ RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
+ r->cons.sync_type = (flags & RING_F_SC_DEQ) ?
+ RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
if (flags & RING_F_EXACT_SZ) {
r->size = rte_align32pow2(count + 1);
diff --git a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h
index 18fc5d845..35ee4491c 100644
--- a/lib/librte_ring/rte_ring.h
+++ b/lib/librte_ring/rte_ring.h
@@ -36,91 +36,7 @@
extern "C" {
#endif
-#include <stdio.h>
-#include <stdint.h>
-#include <sys/queue.h>
-#include <errno.h>
-#include <rte_common.h>
-#include <rte_config.h>
-#include <rte_memory.h>
-#include <rte_lcore.h>
-#include <rte_atomic.h>
-#include <rte_branch_prediction.h>
-#include <rte_memzone.h>
-#include <rte_pause.h>
-
-#define RTE_TAILQ_RING_NAME "RTE_RING"
-
-enum rte_ring_queue_behavior {
- RTE_RING_QUEUE_FIXED = 0, /* Enq/Deq a fixed number of items from a ring */
- RTE_RING_QUEUE_VARIABLE /* Enq/Deq as many items as possible from ring */
-};
-
-#define RTE_RING_MZ_PREFIX "RG_"
-/** The maximum length of a ring name. */
-#define RTE_RING_NAMESIZE (RTE_MEMZONE_NAMESIZE - \
- sizeof(RTE_RING_MZ_PREFIX) + 1)
-
-/* structure to hold a pair of head/tail values and other metadata */
-struct rte_ring_headtail {
- volatile uint32_t head; /**< Prod/consumer head. */
- volatile uint32_t tail; /**< Prod/consumer tail. */
- uint32_t single; /**< True if single prod/cons */
-};
-
-/**
- * An RTE ring structure.
- *
- * The producer and the consumer have a head and a tail index. The particularity
- * of these index is that they are not between 0 and size(ring). These indexes
- * are between 0 and 2^32, and we mask their value when we access the ring[]
- * field. Thanks to this assumption, we can do subtractions between 2 index
- * values in a modulo-32bit base: that's why the overflow of the indexes is not
- * a problem.
- */
-struct rte_ring {
- /*
- * Note: this field kept the RTE_MEMZONE_NAMESIZE size due to ABI
- * compatibility requirements, it could be changed to RTE_RING_NAMESIZE
- * next time the ABI changes
- */
- char name[RTE_MEMZONE_NAMESIZE] __rte_cache_aligned; /**< Name of the ring. */
- int flags; /**< Flags supplied at creation. */
- const struct rte_memzone *memzone;
- /**< Memzone, if any, containing the rte_ring */
- uint32_t size; /**< Size of ring. */
- uint32_t mask; /**< Mask (size-1) of ring. */
- uint32_t capacity; /**< Usable size of ring */
-
- char pad0 __rte_cache_aligned; /**< empty cache line */
-
- /** Ring producer status. */
- struct rte_ring_headtail prod __rte_cache_aligned;
- char pad1 __rte_cache_aligned; /**< empty cache line */
-
- /** Ring consumer status. */
- struct rte_ring_headtail cons __rte_cache_aligned;
- char pad2 __rte_cache_aligned; /**< empty cache line */
-};
-
-#define RING_F_SP_ENQ 0x0001 /**< The default enqueue is "single-producer". */
-#define RING_F_SC_DEQ 0x0002 /**< The default dequeue is "single-consumer". */
-/**
- * Ring is to hold exactly requested number of entries.
- * Without this flag set, the ring size requested must be a power of 2, and the
- * usable space will be that size - 1. With the flag, the requested size will
- * be rounded up to the next power of two, but the usable space will be exactly
- * that requested. Worst case, if a power-of-2 size is requested, half the
- * ring space will be wasted.
- */
-#define RING_F_EXACT_SZ 0x0004
-#define RTE_RING_SZ_MASK (0x7fffffffU) /**< Ring size mask */
-
-/* @internal defines for passing to the enqueue dequeue worker functions */
-#define __IS_SP 1
-#define __IS_MP 0
-#define __IS_SC 1
-#define __IS_MC 0
+#include <rte_ring_core.h>
/**
* Calculate the memory size needed for a ring
@@ -420,7 +336,7 @@ rte_ring_mp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- __IS_MP, free_space);
+ RTE_RING_SYNC_MT, free_space);
}
/**
@@ -443,9 +359,13 @@ rte_ring_sp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- __IS_SP, free_space);
+ RTE_RING_SYNC_ST, free_space);
}
+#ifdef ALLOW_EXPERIMENTAL_API
+#include <rte_ring_elem.h>
+#endif
+
/**
* Enqueue several objects on a ring.
*
@@ -470,7 +390,7 @@ rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- r->prod.single, free_space);
+ r->prod.sync_type, free_space);
}
/**
@@ -554,7 +474,7 @@ rte_ring_mc_dequeue_bulk(struct rte_ring *r, void **obj_table,
unsigned int n, unsigned int *available)
{
return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- __IS_MC, available);
+ RTE_RING_SYNC_MT, available);
}
/**
@@ -578,7 +498,7 @@ rte_ring_sc_dequeue_bulk(struct rte_ring *r, void **obj_table,
unsigned int n, unsigned int *available)
{
return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- __IS_SC, available);
+ RTE_RING_SYNC_ST, available);
}
/**
@@ -605,7 +525,7 @@ rte_ring_dequeue_bulk(struct rte_ring *r, void **obj_table, unsigned int n,
unsigned int *available)
{
return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- r->cons.single, available);
+ r->cons.sync_type, available);
}
/**
@@ -777,6 +697,62 @@ rte_ring_get_capacity(const struct rte_ring *r)
return r->capacity;
}
+/**
+ * Return sync type used by producer in the ring.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * Producer sync type value.
+ */
+static inline enum rte_ring_sync_type
+rte_ring_get_prod_sync_type(const struct rte_ring *r)
+{
+ return r->prod.sync_type;
+}
+
+/**
+ * Check is the ring for single producer.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * true if ring is SP, zero otherwise.
+ */
+static inline int
+rte_ring_is_prod_single(const struct rte_ring *r)
+{
+ return (rte_ring_get_prod_sync_type(r) == RTE_RING_SYNC_ST);
+}
+
+/**
+ * Return sync type used by consumer in the ring.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * Consumer sync type value.
+ */
+static inline enum rte_ring_sync_type
+rte_ring_get_cons_sync_type(const struct rte_ring *r)
+{
+ return r->cons.sync_type;
+}
+
+/**
+ * Check is the ring for single consumer.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * true if ring is SC, zero otherwise.
+ */
+static inline int
+rte_ring_is_cons_single(const struct rte_ring *r)
+{
+ return (rte_ring_get_cons_sync_type(r) == RTE_RING_SYNC_ST);
+}
+
/**
* Dump the status of all rings on the console
*
@@ -820,7 +796,7 @@ rte_ring_mp_enqueue_burst(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue(r, obj_table, n,
- RTE_RING_QUEUE_VARIABLE, __IS_MP, free_space);
+ RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_MT, free_space);
}
/**
@@ -843,7 +819,7 @@ rte_ring_sp_enqueue_burst(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue(r, obj_table, n,
- RTE_RING_QUEUE_VARIABLE, __IS_SP, free_space);
+ RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_ST, free_space);
}
/**
@@ -870,7 +846,7 @@ rte_ring_enqueue_burst(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_VARIABLE,
- r->prod.single, free_space);
+ r->prod.sync_type, free_space);
}
/**
@@ -898,7 +874,7 @@ rte_ring_mc_dequeue_burst(struct rte_ring *r, void **obj_table,
unsigned int n, unsigned int *available)
{
return __rte_ring_do_dequeue(r, obj_table, n,
- RTE_RING_QUEUE_VARIABLE, __IS_MC, available);
+ RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_MT, available);
}
/**
@@ -923,7 +899,7 @@ rte_ring_sc_dequeue_burst(struct rte_ring *r, void **obj_table,
unsigned int n, unsigned int *available)
{
return __rte_ring_do_dequeue(r, obj_table, n,
- RTE_RING_QUEUE_VARIABLE, __IS_SC, available);
+ RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_ST, available);
}
/**
@@ -951,7 +927,7 @@ rte_ring_dequeue_burst(struct rte_ring *r, void **obj_table,
{
return __rte_ring_do_dequeue(r, obj_table, n,
RTE_RING_QUEUE_VARIABLE,
- r->cons.single, available);
+ r->cons.sync_type, available);
}
#ifdef __cplusplus
diff --git a/lib/librte_ring/rte_ring_core.h b/lib/librte_ring/rte_ring_core.h
new file mode 100644
index 000000000..d9cef763f
--- /dev/null
+++ b/lib/librte_ring/rte_ring_core.h
@@ -0,0 +1,132 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2010-2020 Intel Corporation
+ * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#ifndef _RTE_RING_CORE_H_
+#define _RTE_RING_CORE_H_
+
+/**
+ * @file
+ * This file contains definion of RTE ring structure itself,
+ * init flags and some related macros.
+ * For majority of DPDK entities, it is not recommended to include
+ * this file directly, use include <rte_ring.h> or <rte_ring_elem.h>
+ * instead.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdio.h>
+#include <stdint.h>
+#include <string.h>
+#include <sys/queue.h>
+#include <errno.h>
+#include <rte_common.h>
+#include <rte_config.h>
+#include <rte_memory.h>
+#include <rte_lcore.h>
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_memzone.h>
+#include <rte_pause.h>
+#include <rte_debug.h>
+
+#define RTE_TAILQ_RING_NAME "RTE_RING"
+
+/** enqueue/dequeue behavior types */
+enum rte_ring_queue_behavior {
+ /** Enq/Deq a fixed number of items from a ring */
+ RTE_RING_QUEUE_FIXED = 0,
+ /** Enq/Deq as many items as possible from ring */
+ RTE_RING_QUEUE_VARIABLE
+};
+
+#define RTE_RING_MZ_PREFIX "RG_"
+/** The maximum length of a ring name. */
+#define RTE_RING_NAMESIZE (RTE_MEMZONE_NAMESIZE - \
+ sizeof(RTE_RING_MZ_PREFIX) + 1)
+
+/** prod/cons sync types */
+enum rte_ring_sync_type {
+ RTE_RING_SYNC_MT, /**< multi-thread safe (default mode) */
+ RTE_RING_SYNC_ST, /**< single thread only */
+};
+
+/**
+ * structures to hold a pair of head/tail values and other metadata.
+ * Depending on sync_type format of that structure might be different,
+ * but offset for *sync_type* and *tail* values should remain the same.
+ */
+struct rte_ring_headtail {
+ volatile uint32_t head; /**< prod/consumer head. */
+ volatile uint32_t tail; /**< prod/consumer tail. */
+ RTE_STD_C11
+ union {
+ /** sync type of prod/cons */
+ enum rte_ring_sync_type sync_type;
+ /** deprecated - True if single prod/cons */
+ uint32_t single;
+ };
+};
+
+/**
+ * An RTE ring structure.
+ *
+ * The producer and the consumer have a head and a tail index. The particularity
+ * of these index is that they are not between 0 and size(ring). These indexes
+ * are between 0 and 2^32, and we mask their value when we access the ring[]
+ * field. Thanks to this assumption, we can do subtractions between 2 index
+ * values in a modulo-32bit base: that's why the overflow of the indexes is not
+ * a problem.
+ */
+struct rte_ring {
+ /*
+ * Note: this field kept the RTE_MEMZONE_NAMESIZE size due to ABI
+ * compatibility requirements, it could be changed to RTE_RING_NAMESIZE
+ * next time the ABI changes
+ */
+ char name[RTE_MEMZONE_NAMESIZE] __rte_cache_aligned;
+ /**< Name of the ring. */
+ int flags; /**< Flags supplied at creation. */
+ const struct rte_memzone *memzone;
+ /**< Memzone, if any, containing the rte_ring */
+ uint32_t size; /**< Size of ring. */
+ uint32_t mask; /**< Mask (size-1) of ring. */
+ uint32_t capacity; /**< Usable size of ring */
+
+ char pad0 __rte_cache_aligned; /**< empty cache line */
+
+ /** Ring producer status. */
+ struct rte_ring_headtail prod __rte_cache_aligned;
+ char pad1 __rte_cache_aligned; /**< empty cache line */
+
+ /** Ring consumer status. */
+ struct rte_ring_headtail cons __rte_cache_aligned;
+ char pad2 __rte_cache_aligned; /**< empty cache line */
+};
+
+#define RING_F_SP_ENQ 0x0001 /**< The default enqueue is "single-producer". */
+#define RING_F_SC_DEQ 0x0002 /**< The default dequeue is "single-consumer". */
+/**
+ * Ring is to hold exactly requested number of entries.
+ * Without this flag set, the ring size requested must be a power of 2, and the
+ * usable space will be that size - 1. With the flag, the requested size will
+ * be rounded up to the next power of two, but the usable space will be exactly
+ * that requested. Worst case, if a power-of-2 size is requested, half the
+ * ring space will be wasted.
+ */
+#define RING_F_EXACT_SZ 0x0004
+#define RTE_RING_SZ_MASK (0x7fffffffU) /**< Ring size mask */
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_RING_CORE_H_ */
diff --git a/lib/librte_ring/rte_ring_elem.h b/lib/librte_ring/rte_ring_elem.h
index 663addc73..7406c0b0f 100644
--- a/lib/librte_ring/rte_ring_elem.h
+++ b/lib/librte_ring/rte_ring_elem.h
@@ -20,21 +20,7 @@
extern "C" {
#endif
-#include <stdio.h>
-#include <stdint.h>
-#include <string.h>
-#include <sys/queue.h>
-#include <errno.h>
-#include <rte_common.h>
-#include <rte_config.h>
-#include <rte_memory.h>
-#include <rte_lcore.h>
-#include <rte_atomic.h>
-#include <rte_branch_prediction.h>
-#include <rte_memzone.h>
-#include <rte_pause.h>
-
-#include "rte_ring.h"
+#include <rte_ring_core.h>
/**
* @warning
@@ -510,7 +496,7 @@ rte_ring_mp_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
unsigned int esize, unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, __IS_MP, free_space);
+ RTE_RING_QUEUE_FIXED, RTE_RING_SYNC_MT, free_space);
}
/**
@@ -539,7 +525,7 @@ rte_ring_sp_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
unsigned int esize, unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, __IS_SP, free_space);
+ RTE_RING_QUEUE_FIXED, RTE_RING_SYNC_ST, free_space);
}
/**
@@ -570,7 +556,7 @@ rte_ring_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
unsigned int esize, unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, r->prod.single, free_space);
+ RTE_RING_QUEUE_FIXED, r->prod.sync_type, free_space);
}
/**
@@ -675,7 +661,7 @@ rte_ring_mc_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
unsigned int esize, unsigned int n, unsigned int *available)
{
return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, __IS_MC, available);
+ RTE_RING_QUEUE_FIXED, RTE_RING_SYNC_MT, available);
}
/**
@@ -703,7 +689,7 @@ rte_ring_sc_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
unsigned int esize, unsigned int n, unsigned int *available)
{
return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, __IS_SC, available);
+ RTE_RING_QUEUE_FIXED, RTE_RING_SYNC_ST, available);
}
/**
@@ -734,7 +720,7 @@ rte_ring_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
unsigned int esize, unsigned int n, unsigned int *available)
{
return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, r->cons.single, available);
+ RTE_RING_QUEUE_FIXED, r->cons.sync_type, available);
}
/**
@@ -842,7 +828,7 @@ rte_ring_mp_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
unsigned int esize, unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE, __IS_MP, free_space);
+ RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_MT, free_space);
}
/**
@@ -871,7 +857,7 @@ rte_ring_sp_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
unsigned int esize, unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE, __IS_SP, free_space);
+ RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_ST, free_space);
}
/**
@@ -902,7 +888,7 @@ rte_ring_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
unsigned int esize, unsigned int n, unsigned int *free_space)
{
return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE, r->prod.single, free_space);
+ RTE_RING_QUEUE_VARIABLE, r->prod.sync_type, free_space);
}
/**
@@ -934,7 +920,7 @@ rte_ring_mc_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
unsigned int esize, unsigned int n, unsigned int *available)
{
return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE, __IS_MC, available);
+ RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_MT, available);
}
/**
@@ -963,7 +949,7 @@ rte_ring_sc_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
unsigned int esize, unsigned int n, unsigned int *available)
{
return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE, __IS_SC, available);
+ RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_ST, available);
}
/**
@@ -995,9 +981,11 @@ rte_ring_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
{
return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
RTE_RING_QUEUE_VARIABLE,
- r->cons.single, available);
+ r->cons.sync_type, available);
}
+#include <rte_ring.h>
+
#ifdef __cplusplus
}
#endif
--
2.17.1
^ permalink raw reply [relevance 9%]
* [dpdk-dev] [PATCH v5 3/9] ring: introduce RTS ring mode
2020-04-18 16:32 3% ` [dpdk-dev] [PATCH v5 0/9] New sync modes for ring Konstantin Ananyev
2020-04-18 16:32 9% ` [dpdk-dev] [PATCH v5 2/9] ring: prepare ring to allow new sync schemes Konstantin Ananyev
@ 2020-04-18 16:32 1% ` Konstantin Ananyev
1 sibling, 0 replies; 200+ results
From: Konstantin Ananyev @ 2020-04-18 16:32 UTC (permalink / raw)
To: dev; +Cc: honnappa.nagarahalli, david.marchand, jielong.zjl, Konstantin Ananyev
Introduce relaxed tail sync (RTS) mode for MT ring synchronization.
Aim to reduce stall times in case when ring is used on
overcommited cpus (multiple active threads on the same cpu).
The main difference from original MP/MC algorithm is that
tail value is increased not by every thread that finished enqueue/dequeue,
but only by the last one.
That allows threads to avoid spinning on ring tail value,
leaving actual tail value change to the last thread in the update queue.
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
check-abi.sh reports what I believe is a false-positive about
ring cons/prod changes. As a workaround, devtools/libabigail.abignore is
updated to suppress *struct ring* related errors.
devtools/libabigail.abignore | 7 +
lib/librte_ring/Makefile | 4 +-
lib/librte_ring/meson.build | 7 +-
lib/librte_ring/rte_ring.c | 100 +++++-
lib/librte_ring/rte_ring.h | 70 +++-
lib/librte_ring/rte_ring_core.h | 36 +-
lib/librte_ring/rte_ring_elem.h | 90 ++++-
lib/librte_ring/rte_ring_rts.h | 439 +++++++++++++++++++++++++
lib/librte_ring/rte_ring_rts_c11_mem.h | 179 ++++++++++
9 files changed, 902 insertions(+), 30 deletions(-)
create mode 100644 lib/librte_ring/rte_ring_rts.h
create mode 100644 lib/librte_ring/rte_ring_rts_c11_mem.h
diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore
index a59df8f13..cd86d89ca 100644
--- a/devtools/libabigail.abignore
+++ b/devtools/libabigail.abignore
@@ -11,3 +11,10 @@
type_kind = enum
name = rte_crypto_asym_xform_type
changed_enumerators = RTE_CRYPTO_ASYM_XFORM_TYPE_LIST_END
+; Ignore updates of ring prod/cons
+[suppress_type]
+ type_kind = struct
+ name = rte_ring
+[suppress_type]
+ type_kind = struct
+ name = rte_event_ring
diff --git a/lib/librte_ring/Makefile b/lib/librte_ring/Makefile
index 6572768c9..04e446e37 100644
--- a/lib/librte_ring/Makefile
+++ b/lib/librte_ring/Makefile
@@ -19,6 +19,8 @@ SYMLINK-$(CONFIG_RTE_LIBRTE_RING)-include := rte_ring.h \
rte_ring_core.h \
rte_ring_elem.h \
rte_ring_generic.h \
- rte_ring_c11_mem.h
+ rte_ring_c11_mem.h \
+ rte_ring_rts.h \
+ rte_ring_rts_c11_mem.h
include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_ring/meson.build b/lib/librte_ring/meson.build
index c656781da..a95598032 100644
--- a/lib/librte_ring/meson.build
+++ b/lib/librte_ring/meson.build
@@ -6,4 +6,9 @@ headers = files('rte_ring.h',
'rte_ring_core.h',
'rte_ring_elem.h',
'rte_ring_c11_mem.h',
- 'rte_ring_generic.h')
+ 'rte_ring_generic.h',
+ 'rte_ring_rts.h',
+ 'rte_ring_rts_c11_mem.h')
+
+# rte_ring_create_elem and rte_ring_get_memsize_elem are experimental
+allow_experimental_apis = true
diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c
index fa5733907..222eec0fb 100644
--- a/lib/librte_ring/rte_ring.c
+++ b/lib/librte_ring/rte_ring.c
@@ -45,6 +45,9 @@ EAL_REGISTER_TAILQ(rte_ring_tailq)
/* true if x is a power of 2 */
#define POWEROF2(x) ((((x)-1) & (x)) == 0)
+/* by default set head/tail distance as 1/8 of ring capacity */
+#define HTD_MAX_DEF 8
+
/* return the size of memory occupied by a ring */
ssize_t
rte_ring_get_memsize_elem(unsigned int esize, unsigned int count)
@@ -79,11 +82,84 @@ rte_ring_get_memsize(unsigned int count)
return rte_ring_get_memsize_elem(sizeof(void *), count);
}
+/*
+ * internal helper function to reset prod/cons head-tail values.
+ */
+static void
+reset_headtail(void *p)
+{
+ struct rte_ring_headtail *ht;
+ struct rte_ring_rts_headtail *ht_rts;
+
+ ht = p;
+ ht_rts = p;
+
+ switch (ht->sync_type) {
+ case RTE_RING_SYNC_MT:
+ case RTE_RING_SYNC_ST:
+ ht->head = 0;
+ ht->tail = 0;
+ break;
+ case RTE_RING_SYNC_MT_RTS:
+ ht_rts->head.raw = 0;
+ ht_rts->tail.raw = 0;
+ break;
+ default:
+ /* unknown sync mode */
+ RTE_ASSERT(0);
+ }
+}
+
void
rte_ring_reset(struct rte_ring *r)
{
- r->prod.head = r->cons.head = 0;
- r->prod.tail = r->cons.tail = 0;
+ reset_headtail(&r->prod);
+ reset_headtail(&r->cons);
+}
+
+/*
+ * helper function, calculates sync_type values for prod and cons
+ * based on input flags. Returns zero at success or negative
+ * errno value otherwise.
+ */
+static int
+get_sync_type(uint32_t flags, enum rte_ring_sync_type *prod_st,
+ enum rte_ring_sync_type *cons_st)
+{
+ static const uint32_t prod_st_flags =
+ (RING_F_SP_ENQ | RING_F_MP_RTS_ENQ);
+ static const uint32_t cons_st_flags =
+ (RING_F_SC_DEQ | RING_F_MC_RTS_DEQ);
+
+ switch (flags & prod_st_flags) {
+ case 0:
+ *prod_st = RTE_RING_SYNC_MT;
+ break;
+ case RING_F_SP_ENQ:
+ *prod_st = RTE_RING_SYNC_ST;
+ break;
+ case RING_F_MP_RTS_ENQ:
+ *prod_st = RTE_RING_SYNC_MT_RTS;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ switch (flags & cons_st_flags) {
+ case 0:
+ *cons_st = RTE_RING_SYNC_MT;
+ break;
+ case RING_F_SC_DEQ:
+ *cons_st = RTE_RING_SYNC_ST;
+ break;
+ case RING_F_MC_RTS_DEQ:
+ *cons_st = RTE_RING_SYNC_MT_RTS;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
}
int
@@ -100,16 +176,20 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
RTE_BUILD_BUG_ON((offsetof(struct rte_ring, prod) &
RTE_CACHE_LINE_MASK) != 0);
+ RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, sync_type) !=
+ offsetof(struct rte_ring_rts_headtail, sync_type));
+ RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, tail) !=
+ offsetof(struct rte_ring_rts_headtail, tail.val.pos));
+
/* init the ring structure */
memset(r, 0, sizeof(*r));
ret = strlcpy(r->name, name, sizeof(r->name));
if (ret < 0 || ret >= (int)sizeof(r->name))
return -ENAMETOOLONG;
r->flags = flags;
- r->prod.sync_type = (flags & RING_F_SP_ENQ) ?
- RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
- r->cons.sync_type = (flags & RING_F_SC_DEQ) ?
- RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
+ ret = get_sync_type(flags, &r->prod.sync_type, &r->cons.sync_type);
+ if (ret != 0)
+ return ret;
if (flags & RING_F_EXACT_SZ) {
r->size = rte_align32pow2(count + 1);
@@ -126,8 +206,12 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
r->mask = count - 1;
r->capacity = r->mask;
}
- r->prod.head = r->cons.head = 0;
- r->prod.tail = r->cons.tail = 0;
+
+ /* set default values for head-tail distance */
+ if (flags & RING_F_MP_RTS_ENQ)
+ rte_ring_set_prod_htd_max(r, r->capacity / HTD_MAX_DEF);
+ if (flags & RING_F_MC_RTS_DEQ)
+ rte_ring_set_cons_htd_max(r, r->capacity / HTD_MAX_DEF);
return 0;
}
diff --git a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h
index 35ee4491c..77f206ca7 100644
--- a/lib/librte_ring/rte_ring.h
+++ b/lib/librte_ring/rte_ring.h
@@ -1,6 +1,6 @@
/* SPDX-License-Identifier: BSD-3-Clause
*
- * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2010-2020 Intel Corporation
* Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
* All rights reserved.
* Derived from FreeBSD's bufring.h
@@ -389,8 +389,21 @@ static __rte_always_inline unsigned int
rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
- return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- r->prod.sync_type, free_space);
+ switch (r->prod.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mp_enqueue_bulk(r, obj_table, n, free_space);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sp_enqueue_bulk(r, obj_table, n, free_space);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mp_rts_enqueue_bulk(r, obj_table, n,
+ free_space);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ return 0;
}
/**
@@ -524,8 +537,20 @@ static __rte_always_inline unsigned int
rte_ring_dequeue_bulk(struct rte_ring *r, void **obj_table, unsigned int n,
unsigned int *available)
{
- return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
- r->cons.sync_type, available);
+ switch (r->cons.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mc_dequeue_bulk(r, obj_table, n, available);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sc_dequeue_bulk(r, obj_table, n, available);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mc_rts_dequeue_bulk(r, obj_table, n, available);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ return 0;
}
/**
@@ -845,8 +870,21 @@ static __rte_always_inline unsigned
rte_ring_enqueue_burst(struct rte_ring *r, void * const *obj_table,
unsigned int n, unsigned int *free_space)
{
- return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_VARIABLE,
- r->prod.sync_type, free_space);
+ switch (r->prod.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mp_enqueue_burst(r, obj_table, n, free_space);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sp_enqueue_burst(r, obj_table, n, free_space);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mp_rts_enqueue_burst(r, obj_table, n,
+ free_space);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ return 0;
}
/**
@@ -925,9 +963,21 @@ static __rte_always_inline unsigned
rte_ring_dequeue_burst(struct rte_ring *r, void **obj_table,
unsigned int n, unsigned int *available)
{
- return __rte_ring_do_dequeue(r, obj_table, n,
- RTE_RING_QUEUE_VARIABLE,
- r->cons.sync_type, available);
+ switch (r->cons.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mc_dequeue_burst(r, obj_table, n, available);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sc_dequeue_burst(r, obj_table, n, available);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mc_rts_dequeue_burst(r, obj_table, n,
+ available);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ return 0;
}
#ifdef __cplusplus
diff --git a/lib/librte_ring/rte_ring_core.h b/lib/librte_ring/rte_ring_core.h
index d9cef763f..ded0fa0b7 100644
--- a/lib/librte_ring/rte_ring_core.h
+++ b/lib/librte_ring/rte_ring_core.h
@@ -57,6 +57,9 @@ enum rte_ring_queue_behavior {
enum rte_ring_sync_type {
RTE_RING_SYNC_MT, /**< multi-thread safe (default mode) */
RTE_RING_SYNC_ST, /**< single thread only */
+#ifdef ALLOW_EXPERIMENTAL_API
+ RTE_RING_SYNC_MT_RTS, /**< multi-thread relaxed tail sync */
+#endif
};
/**
@@ -76,6 +79,22 @@ struct rte_ring_headtail {
};
};
+union rte_ring_rts_poscnt {
+ /** raw 8B value to read/write *cnt* and *pos* as one atomic op */
+ uint64_t raw __rte_aligned(8);
+ struct {
+ uint32_t cnt; /**< head/tail reference counter */
+ uint32_t pos; /**< head/tail position */
+ } val;
+};
+
+struct rte_ring_rts_headtail {
+ volatile union rte_ring_rts_poscnt tail;
+ enum rte_ring_sync_type sync_type; /**< sync type of prod/cons */
+ uint32_t htd_max; /**< max allowed distance between head/tail */
+ volatile union rte_ring_rts_poscnt head;
+};
+
/**
* An RTE ring structure.
*
@@ -104,11 +123,21 @@ struct rte_ring {
char pad0 __rte_cache_aligned; /**< empty cache line */
/** Ring producer status. */
- struct rte_ring_headtail prod __rte_cache_aligned;
+ RTE_STD_C11
+ union {
+ struct rte_ring_headtail prod;
+ struct rte_ring_rts_headtail rts_prod;
+ } __rte_cache_aligned;
+
char pad1 __rte_cache_aligned; /**< empty cache line */
/** Ring consumer status. */
- struct rte_ring_headtail cons __rte_cache_aligned;
+ RTE_STD_C11
+ union {
+ struct rte_ring_headtail cons;
+ struct rte_ring_rts_headtail rts_cons;
+ } __rte_cache_aligned;
+
char pad2 __rte_cache_aligned; /**< empty cache line */
};
@@ -125,6 +154,9 @@ struct rte_ring {
#define RING_F_EXACT_SZ 0x0004
#define RTE_RING_SZ_MASK (0x7fffffffU) /**< Ring size mask */
+#define RING_F_MP_RTS_ENQ 0x0008 /**< The default enqueue is "MP RTS". */
+#define RING_F_MC_RTS_DEQ 0x0010 /**< The default dequeue is "MC RTS". */
+
#ifdef __cplusplus
}
#endif
diff --git a/lib/librte_ring/rte_ring_elem.h b/lib/librte_ring/rte_ring_elem.h
index 7406c0b0f..6da0a917b 100644
--- a/lib/librte_ring/rte_ring_elem.h
+++ b/lib/librte_ring/rte_ring_elem.h
@@ -528,6 +528,10 @@ rte_ring_sp_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
RTE_RING_QUEUE_FIXED, RTE_RING_SYNC_ST, free_space);
}
+#ifdef ALLOW_EXPERIMENTAL_API
+#include <rte_ring_rts.h>
+#endif
+
/**
* Enqueue several objects on a ring.
*
@@ -557,6 +561,26 @@ rte_ring_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
{
return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
RTE_RING_QUEUE_FIXED, r->prod.sync_type, free_space);
+
+ switch (r->prod.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mp_enqueue_bulk_elem(r, obj_table, esize, n,
+ free_space);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sp_enqueue_bulk_elem(r, obj_table, esize, n,
+ free_space);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mp_rts_enqueue_bulk_elem(r, obj_table, esize, n,
+ free_space);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ if (free_space != NULL)
+ *free_space = 0;
+ return 0;
}
/**
@@ -661,7 +685,7 @@ rte_ring_mc_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
unsigned int esize, unsigned int n, unsigned int *available)
{
return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, RTE_RING_SYNC_MT, available);
+ RTE_RING_QUEUE_FIXED, RTE_RING_SYNC_MT, available);
}
/**
@@ -719,8 +743,25 @@ static __rte_always_inline unsigned int
rte_ring_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
unsigned int esize, unsigned int n, unsigned int *available)
{
- return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_FIXED, r->cons.sync_type, available);
+ switch (r->cons.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mc_dequeue_bulk_elem(r, obj_table, esize, n,
+ available);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sc_dequeue_bulk_elem(r, obj_table, esize, n,
+ available);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mc_rts_dequeue_bulk_elem(r, obj_table, esize,
+ n, available);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ if (available != NULL)
+ *available = 0;
+ return 0;
}
/**
@@ -887,8 +928,25 @@ static __rte_always_inline unsigned
rte_ring_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
unsigned int esize, unsigned int n, unsigned int *free_space)
{
- return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE, r->prod.sync_type, free_space);
+ switch (r->prod.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mp_enqueue_burst_elem(r, obj_table, esize, n,
+ free_space);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sp_enqueue_burst_elem(r, obj_table, esize, n,
+ free_space);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mp_rts_enqueue_burst_elem(r, obj_table, esize,
+ n, free_space);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ if (free_space != NULL)
+ *free_space = 0;
+ return 0;
}
/**
@@ -979,9 +1037,25 @@ static __rte_always_inline unsigned int
rte_ring_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
unsigned int esize, unsigned int n, unsigned int *available)
{
- return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
- RTE_RING_QUEUE_VARIABLE,
- r->cons.sync_type, available);
+ switch (r->cons.sync_type) {
+ case RTE_RING_SYNC_MT:
+ return rte_ring_mc_dequeue_burst_elem(r, obj_table, esize, n,
+ available);
+ case RTE_RING_SYNC_ST:
+ return rte_ring_sc_dequeue_burst_elem(r, obj_table, esize, n,
+ available);
+#ifdef ALLOW_EXPERIMENTAL_API
+ case RTE_RING_SYNC_MT_RTS:
+ return rte_ring_mc_rts_dequeue_burst_elem(r, obj_table, esize,
+ n, available);
+#endif
+ }
+
+ /* valid ring should never reach this point */
+ RTE_ASSERT(0);
+ if (available != NULL)
+ *available = 0;
+ return 0;
}
#include <rte_ring.h>
diff --git a/lib/librte_ring/rte_ring_rts.h b/lib/librte_ring/rte_ring_rts.h
new file mode 100644
index 000000000..8ced07096
--- /dev/null
+++ b/lib/librte_ring/rte_ring_rts.h
@@ -0,0 +1,439 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2010-2020 Intel Corporation
+ * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#ifndef _RTE_RING_RTS_H_
+#define _RTE_RING_RTS_H_
+
+/**
+ * @file rte_ring_rts.h
+ * @b EXPERIMENTAL: this API may change without prior notice
+ * It is not recommended to include this file directly.
+ * Please include <rte_ring.h> instead.
+ *
+ * Contains functions for Relaxed Tail Sync (RTS) ring mode.
+ * The main idea remains the same as for our original MP/MC synchronization
+ * mechanism.
+ * The main difference is that tail value is increased not
+ * by every thread that finished enqueue/dequeue,
+ * but only by the current last one doing enqueue/dequeue.
+ * That allows threads to skip spinning on tail value,
+ * leaving actual tail value change to last thread at a given instance.
+ * RTS requires 2 64-bit CAS for each enqueue(/dequeue) operation:
+ * one for head update, second for tail update.
+ * As a gain it allows thread to avoid spinning/waiting on tail value.
+ * In comparision original MP/MC algorithm requires one 32-bit CAS
+ * for head update and waiting/spinning on tail value.
+ *
+ * Brief outline:
+ * - introduce update counter (cnt) for both head and tail.
+ * - increment head.cnt for each head.value update
+ * - write head.value and head.cnt atomically (64-bit CAS)
+ * - move tail.value ahead only when tail.cnt + 1 == head.cnt
+ * (indicating that this is the last thread updating the tail)
+ * - increment tail.cnt when each enqueue/dequeue op finishes
+ * (no matter if tail.value going to change or not)
+ * - write tail.value and tail.cnt atomically (64-bit CAS)
+ *
+ * To avoid producer/consumer starvation:
+ * - limit max allowed distance between head and tail value (HTD_MAX).
+ * I.E. thread is allowed to proceed with changing head.value,
+ * only when: head.value - tail.value <= HTD_MAX
+ * HTD_MAX is an optional parameter.
+ * With HTD_MAX == 0 we'll have fully serialized ring -
+ * i.e. only one thread at a time will be able to enqueue/dequeue
+ * to/from the ring.
+ * With HTD_MAX >= ring.capacity - no limitation.
+ * By default HTD_MAX == ring.capacity / 8.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_ring_rts_c11_mem.h>
+
+/**
+ * @internal Enqueue several objects on the RTS ring.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a ring
+ * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
+ * @param free_space
+ * returns the amount of space after the enqueue operation has finished
+ * @return
+ * Actual number of objects enqueued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_enqueue_elem(struct rte_ring *r, const void *obj_table,
+ uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+ uint32_t *free_space)
+{
+ uint32_t free, head;
+
+ n = __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
+
+ if (n != 0) {
+ __rte_ring_enqueue_elems(r, head, obj_table, esize, n);
+ __rte_ring_rts_update_tail(&r->rts_prod);
+ }
+
+ if (free_space != NULL)
+ *free_space = free - n;
+ return n;
+}
+
+/**
+ * @internal Dequeue several objects from the RTS ring.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to pull from the ring.
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a ring
+ * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
+ * @param available
+ * returns the number of remaining ring entries after the dequeue has finished
+ * @return
+ * - Actual number of objects dequeued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_dequeue_elem(struct rte_ring *r, void *obj_table,
+ uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+ uint32_t *available)
+{
+ uint32_t entries, head;
+
+ n = __rte_ring_rts_move_cons_head(r, n, behavior, &head, &entries);
+
+ if (n != 0) {
+ __rte_ring_dequeue_elems(r, head, obj_table, esize, n);
+ __rte_ring_rts_update_tail(&r->rts_cons);
+ }
+
+ if (available != NULL)
+ *available = entries - n;
+ return n;
+}
+
+/**
+ * Enqueue several objects on the RTS ring (multi-producers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * The number of objects enqueued, either 0 or n
+ */
+__rte_experimental
+static __rte_always_inline unsigned int
+rte_ring_mp_rts_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_rts_enqueue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_FIXED, free_space);
+}
+
+/**
+ * Dequeue several objects from an RTS ring (multi-consumers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects that will be filled.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * The number of objects dequeued, either 0 or n
+ */
+__rte_experimental
+static __rte_always_inline unsigned int
+rte_ring_mc_rts_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_rts_dequeue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_FIXED, available);
+}
+
+/**
+ * Enqueue several objects on the RTS ring (multi-producers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * - n: Actual number of objects enqueued.
+ */
+__rte_experimental
+static __rte_always_inline unsigned
+rte_ring_mp_rts_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *free_space)
+{
+ return __rte_ring_do_rts_enqueue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_VARIABLE, free_space);
+}
+
+/**
+ * Dequeue several objects from an RTS ring (multi-consumers safe).
+ * When the requested objects are more than the available objects,
+ * only dequeue the actual number of objects.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of objects that will be filled.
+ * @param esize
+ * The size of ring element, in bytes. It must be a multiple of 4.
+ * This must be the same value used while creating the ring. Otherwise
+ * the results are undefined.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * - n: Actual number of objects dequeued, 0 if ring is empty
+ */
+__rte_experimental
+static __rte_always_inline unsigned
+rte_ring_mc_rts_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
+ unsigned int esize, unsigned int n, unsigned int *available)
+{
+ return __rte_ring_do_rts_dequeue_elem(r, obj_table, esize, n,
+ RTE_RING_QUEUE_VARIABLE, available);
+}
+
+/**
+ * Enqueue several objects on the RTS ring (multi-producers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * The number of objects enqueued, either 0 or n
+ */
+__rte_experimental
+static __rte_always_inline unsigned int
+rte_ring_mp_rts_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
+ unsigned int n, unsigned int *free_space)
+{
+ return rte_ring_mp_rts_enqueue_bulk_elem(r, obj_table,
+ sizeof(uintptr_t), n, free_space);
+}
+
+/**
+ * Dequeue several objects from an RTS ring (multi-consumers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * The number of objects dequeued, either 0 or n
+ */
+__rte_experimental
+static __rte_always_inline unsigned int
+rte_ring_mc_rts_dequeue_bulk(struct rte_ring *r, void **obj_table,
+ unsigned int n, unsigned int *available)
+{
+ return rte_ring_mc_rts_dequeue_bulk_elem(r, obj_table,
+ sizeof(uintptr_t), n, available);
+}
+
+/**
+ * Enqueue several objects on the RTS ring (multi-producers safe).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ * if non-NULL, returns the amount of space in the ring after the
+ * enqueue operation has finished.
+ * @return
+ * - n: Actual number of objects enqueued.
+ */
+__rte_experimental
+static __rte_always_inline unsigned
+rte_ring_mp_rts_enqueue_burst(struct rte_ring *r, void * const *obj_table,
+ unsigned int n, unsigned int *free_space)
+{
+ return rte_ring_mp_rts_enqueue_burst_elem(r, obj_table,
+ sizeof(uintptr_t), n, free_space);
+}
+
+/**
+ * Dequeue several objects from an RTS ring (multi-consumers safe).
+ * When the requested objects are more than the available objects,
+ * only dequeue the actual number of objects.
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ * The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ * If non-NULL, returns the number of remaining ring entries after the
+ * dequeue has finished.
+ * @return
+ * - n: Actual number of objects dequeued, 0 if ring is empty
+ */
+__rte_experimental
+static __rte_always_inline unsigned
+rte_ring_mc_rts_dequeue_burst(struct rte_ring *r, void **obj_table,
+ unsigned int n, unsigned int *available)
+{
+ return rte_ring_mc_rts_dequeue_burst_elem(r, obj_table,
+ sizeof(uintptr_t), n, available);
+}
+
+/**
+ * Return producer max Head-Tail-Distance (HTD).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * Producer HTD value, if producer is set in appropriate sync mode,
+ * or UINT32_MAX otherwise.
+ */
+__rte_experimental
+static inline uint32_t
+rte_ring_get_prod_htd_max(const struct rte_ring *r)
+{
+ if (r->prod.sync_type == RTE_RING_SYNC_MT_RTS)
+ return r->rts_prod.htd_max;
+ return UINT32_MAX;
+}
+
+/**
+ * Set producer max Head-Tail-Distance (HTD).
+ * Note that producer has to use appropriate sync mode (RTS).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param v
+ * new HTD value to setup.
+ * @return
+ * Zero on success, or negative error code otherwise.
+ */
+__rte_experimental
+static inline int
+rte_ring_set_prod_htd_max(struct rte_ring *r, uint32_t v)
+{
+ if (r->prod.sync_type != RTE_RING_SYNC_MT_RTS)
+ return -ENOTSUP;
+
+ r->rts_prod.htd_max = v;
+ return 0;
+}
+
+/**
+ * Return consumer max Head-Tail-Distance (HTD).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @return
+ * Consumer HTD value, if consumer is set in appropriate sync mode,
+ * or UINT32_MAX otherwise.
+ */
+__rte_experimental
+static inline uint32_t
+rte_ring_get_cons_htd_max(const struct rte_ring *r)
+{
+ if (r->cons.sync_type == RTE_RING_SYNC_MT_RTS)
+ return r->rts_cons.htd_max;
+ return UINT32_MAX;
+}
+
+/**
+ * Set consumer max Head-Tail-Distance (HTD).
+ * Note that consumer has to use appropriate sync mode (RTS).
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param v
+ * new HTD value to setup.
+ * @return
+ * Zero on success, or negative error code otherwise.
+ */
+__rte_experimental
+static inline int
+rte_ring_set_cons_htd_max(struct rte_ring *r, uint32_t v)
+{
+ if (r->cons.sync_type != RTE_RING_SYNC_MT_RTS)
+ return -ENOTSUP;
+
+ r->rts_cons.htd_max = v;
+ return 0;
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_RING_RTS_H_ */
diff --git a/lib/librte_ring/rte_ring_rts_c11_mem.h b/lib/librte_ring/rte_ring_rts_c11_mem.h
new file mode 100644
index 000000000..9f26817c0
--- /dev/null
+++ b/lib/librte_ring/rte_ring_rts_c11_mem.h
@@ -0,0 +1,179 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2010-2020 Intel Corporation
+ * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#ifndef _RTE_RING_RTS_C11_MEM_H_
+#define _RTE_RING_RTS_C11_MEM_H_
+
+/**
+ * @file rte_ring_rts_c11_mem.h
+ * It is not recommended to include this file directly,
+ * include <rte_ring.h> instead.
+ * Contains internal helper functions for Relaxed Tail Sync (RTS) ring mode.
+ * For more information please refer to <rte_ring_rts.h>.
+ */
+
+/**
+ * @internal This function updates tail values.
+ */
+static __rte_always_inline void
+__rte_ring_rts_update_tail(struct rte_ring_rts_headtail *ht)
+{
+ union rte_ring_rts_poscnt h, ot, nt;
+
+ /*
+ * If there are other enqueues/dequeues in progress that
+ * might preceded us, then don't update tail with new value.
+ */
+
+ ot.raw = __atomic_load_n(&ht->tail.raw, __ATOMIC_ACQUIRE);
+
+ do {
+ /* on 32-bit systems we have to do atomic read here */
+ h.raw = __atomic_load_n(&ht->head.raw, __ATOMIC_RELAXED);
+
+ nt.raw = ot.raw;
+ if (++nt.val.cnt == h.val.cnt)
+ nt.val.pos = h.val.pos;
+
+ } while (__atomic_compare_exchange_n(&ht->tail.raw, &ot.raw, nt.raw,
+ 0, __ATOMIC_RELEASE, __ATOMIC_ACQUIRE) == 0);
+}
+
+/**
+ * @internal This function waits till head/tail distance wouldn't
+ * exceed pre-defined max value.
+ */
+static __rte_always_inline void
+__rte_ring_rts_head_wait(const struct rte_ring_rts_headtail *ht,
+ union rte_ring_rts_poscnt *h)
+{
+ uint32_t max;
+
+ max = ht->htd_max;
+
+ while (h->val.pos - ht->tail.val.pos > max) {
+ rte_pause();
+ h->raw = __atomic_load_n(&ht->head.raw, __ATOMIC_ACQUIRE);
+ }
+}
+
+/**
+ * @internal This function updates the producer head for enqueue.
+ */
+static __rte_always_inline uint32_t
+__rte_ring_rts_move_prod_head(struct rte_ring *r, uint32_t num,
+ enum rte_ring_queue_behavior behavior, uint32_t *old_head,
+ uint32_t *free_entries)
+{
+ uint32_t n;
+ union rte_ring_rts_poscnt nh, oh;
+
+ const uint32_t capacity = r->capacity;
+
+ oh.raw = __atomic_load_n(&r->rts_prod.head.raw, __ATOMIC_ACQUIRE);
+
+ do {
+ /* Reset n to the initial burst count */
+ n = num;
+
+ /*
+ * wait for prod head/tail distance,
+ * make sure that we read prod head *before*
+ * reading cons tail.
+ */
+ __rte_ring_rts_head_wait(&r->rts_prod, &oh);
+
+ /*
+ * The subtraction is done between two unsigned 32bits value
+ * (the result is always modulo 32 bits even if we have
+ * *old_head > cons_tail). So 'free_entries' is always between 0
+ * and capacity (which is < size).
+ */
+ *free_entries = capacity + r->cons.tail - oh.val.pos;
+
+ /* check that we have enough room in ring */
+ if (unlikely(n > *free_entries))
+ n = (behavior == RTE_RING_QUEUE_FIXED) ?
+ 0 : *free_entries;
+
+ if (n == 0)
+ break;
+
+ nh.val.pos = oh.val.pos + n;
+ nh.val.cnt = oh.val.cnt + 1;
+
+ /*
+ * this CAS(ACQUIRE, ACQUIRE) serves as a hoist barrier to prevent:
+ * - OOO reads of cons tail value
+ * - OOO copy of elems to the ring
+ */
+ } while (__atomic_compare_exchange_n(&r->rts_prod.head.raw,
+ &oh.raw, nh.raw,
+ 0, __ATOMIC_ACQUIRE, __ATOMIC_ACQUIRE) == 0);
+
+ *old_head = oh.val.pos;
+ return n;
+}
+
+/**
+ * @internal This function updates the consumer head for dequeue
+ */
+static __rte_always_inline unsigned int
+__rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num,
+ enum rte_ring_queue_behavior behavior, uint32_t *old_head,
+ uint32_t *entries)
+{
+ uint32_t n;
+ union rte_ring_rts_poscnt nh, oh;
+
+ oh.raw = __atomic_load_n(&r->rts_cons.head.raw, __ATOMIC_ACQUIRE);
+
+ /* move cons.head atomically */
+ do {
+ /* Restore n as it may change every loop */
+ n = num;
+
+ /*
+ * wait for cons head/tail distance,
+ * make sure that we read cons head *before*
+ * reading prod tail.
+ */
+ __rte_ring_rts_head_wait(&r->rts_cons, &oh);
+
+ /* The subtraction is done between two unsigned 32bits value
+ * (the result is always modulo 32 bits even if we have
+ * cons_head > prod_tail). So 'entries' is always between 0
+ * and size(ring)-1.
+ */
+ *entries = r->prod.tail - oh.val.pos;
+
+ /* Set the actual entries for dequeue */
+ if (n > *entries)
+ n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 : *entries;
+
+ if (unlikely(n == 0))
+ break;
+
+ nh.val.pos = oh.val.pos + n;
+ nh.val.cnt = oh.val.cnt + 1;
+
+ /*
+ * this CAS(ACQUIRE, ACQUIRE) serves as a hoist barrier to prevent:
+ * - OOO reads of prod tail value
+ * - OOO copy of elems from the ring
+ */
+ } while (__atomic_compare_exchange_n(&r->rts_cons.head.raw,
+ &oh.raw, nh.raw,
+ 0, __ATOMIC_ACQUIRE, __ATOMIC_ACQUIRE) == 0);
+
+ *old_head = oh.val.pos;
+ return n;
+}
+
+#endif /* _RTE_RING_RTS_C11_MEM_H_ */
--
2.17.1
^ permalink raw reply [relevance 1%]
* [dpdk-dev] [PATCH v8 0/2] support for VFIO-PCI VF token interface
` (3 preceding siblings ...)
2020-04-18 11:16 4% ` [dpdk-dev] [PATCH v7 " Haiyue Wang
@ 2020-04-18 17:30 4% ` Haiyue Wang
4 siblings, 0 replies; 200+ results
From: Haiyue Wang @ 2020-04-18 17:30 UTC (permalink / raw)
To: dev, anatoly.burakov, thomas, vattunuru, jerinj, alex.williamson,
david.marchand
Cc: Haiyue Wang
v8: Update the document.
v7: Add the Fixes tag in uuid, the release note and help
document.
https://patchwork.dpdk.org/cover/68845/
v6: Drop the Fixes tag in uuid, since the file has been
moved to another place, not suitable to apply on stable.
And this is not a bug, just some kind of enhancement.
https://patchwork.dpdk.org/cover/68367/
v5: 1. Add the VF token parse error handling.
2. Split into two patches for different logic module.
3. Add more comments into the code for explaining the design.
4. Drop the ABI change workaround, this patch set focuses on code review.
https://patchwork.dpdk.org/cover/68364/
v4: 1. Ignore rte_vfio_setup_device ABI check since it is
for Linux driver use.
https://patchwork.dpdk.org/patch/68255/
v3: Fix the Travis build failed:
(1). rte_uuid.h:97:55: error: unknown type name ‘size_t’
(2). rte_uuid.h:58:2: error: implicit declaration of function ‘memcpy’
https://patchwork.dpdk.org/patch/68254/
v2: Fix the FreeBSD build error.
https://patchwork.dpdk.org/patch/68240/
v1: Update the commit message.
https://patchwork.dpdk.org/patch/68237/
RFC v2: https://patchwork.dpdk.org/patch/68114/
Based on Vamsi's RFC v1, and Alex's patch for Qemu
[https://lore.kernel.org/lkml/20200204161737.34696b91@w520.home/]:
Use the devarg to pass-down the VF token.
RFC v1: https://patchwork.dpdk.org/patch/66281/ by Vamsi.
Haiyue Wang (2):
eal: add uuid dependent header files explicitly
eal: support for VFIO-PCI VF token
doc/guides/linux_gsg/linux_drivers.rst | 38 ++++++++++++-
doc/guides/rel_notes/release_20_05.rst | 5 ++
drivers/bus/pci/linux/pci_vfio.c | 74 +++++++++++++++++++++++++-
lib/librte_eal/freebsd/eal.c | 3 +-
lib/librte_eal/include/rte_uuid.h | 2 +
lib/librte_eal/include/rte_vfio.h | 21 +++++++-
lib/librte_eal/linux/eal_vfio.c | 20 +++++--
7 files changed, 155 insertions(+), 8 deletions(-)
--
2.26.1
^ permalink raw reply [relevance 4%]
* Re: [dpdk-dev] [PATCH v5] build: disable experimental API check internally
2020-04-17 15:52 0% ` Trahe, Fiona
@ 2020-04-18 19:43 3% ` Chautru, Nicolas
0 siblings, 0 replies; 200+ results
From: Chautru, Nicolas @ 2020-04-18 19:43 UTC (permalink / raw)
To: Trahe, Fiona, David Marchand, Akhil Goyal
Cc: dev, Jerin Jacob, Pavan Nikhilesh, Richardson, Bruce,
Thomas Monjalon, Yigit, Ferruh, Hemant Agrawal, Trahe, Fiona
Hi,
It is probably just me but I having issue with this new patch.
I typically rebuild the PMD libraries directly when doing incremental changes from the Makefile in the PMD directory (ie. not rebuilding full DPDK each time, which still work okay obviously).
With this new change it doesn't seem to work any longer with the updated Makefiles without the ALLOW_EXPERIMENTAL_API:
Symbol is not yet part of stable ABI [-Werror=deprecated-declarations]
I would need to have a further look but checking whether it is just me.
Thanks
Nic
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Trahe, Fiona
> Sent: Friday, April 17, 2020 8:52 AM
> To: David Marchand <david.marchand@redhat.com>
> Cc: dev <dev@dpdk.org>; Jerin Jacob <jerinjacobk@gmail.com>; Pavan
> Nikhilesh <pbhagavatula@marvell.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Thomas Monjalon <thomas@monjalon.net>;
> Yigit, Ferruh <ferruh.yigit@intel.com>; Hemant Agrawal
> <hemant.agrawal@nxp.com>; Trahe, Fiona <fiona.trahe@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v5] build: disable experimental API check
> internally
>
> Hi David,
>
> > -----Original Message-----
> > From: Trahe, Fiona <fiona.trahe@intel.com>
> > Sent: Friday, April 17, 2020 4:05 PM
> > To: David Marchand <david.marchand@redhat.com>
> > Cc: dev <dev@dpdk.org>; Jerin Jacob <jerinjacobk@gmail.com>; Pavan
> > Nikhilesh <pbhagavatula@marvell.com>; Richardson, Bruce
> > <bruce.richardson@intel.com>; Thomas Monjalon <thomas@monjalon.net>;
> > Yigit, Ferruh <ferruh.yigit@intel.com>; Hemant Agrawal
> > <hemant.agrawal@nxp.com>; Trahe, Fiona <fiona.trahe@intel.com>
> > Subject: RE: [dpdk-dev] [PATCH v5] build: disable experimental API
> > check internally
> >
> > Hi Davd,
> >
> > > -----Original Message-----
> > > From: David Marchand <david.marchand@redhat.com>
> > > Sent: Friday, April 17, 2020 2:56 PM
> > > To: Trahe, Fiona <fiona.trahe@intel.com>
> > > Cc: dev <dev@dpdk.org>; Jerin Jacob <jerinjacobk@gmail.com>; Pavan
> > > Nikhilesh <pbhagavatula@marvell.com>; Richardson, Bruce
> > > <bruce.richardson@intel.com>; Thomas
> > Monjalon
> > > <thomas@monjalon.net>; Yigit, Ferruh <ferruh.yigit@intel.com>;
> > > Hemant Agrawal <hemant.agrawal@nxp.com>
> > > Subject: Re: [dpdk-dev] [PATCH v5] build: disable experimental API
> > > check internally
> > >
> > > On Fri, Apr 17, 2020 at 3:44 PM Trahe, Fiona <fiona.trahe@intel.com>
> wrote:
> > > >
> > > > Hi David,
> > > >
> > > > > -----Original Message-----
> > > > > From: David Marchand <david.marchand@redhat.com>
> > > > > Sent: Friday, April 17, 2020 2:23 PM
> > > > > To: Trahe, Fiona <fiona.trahe@intel.com>
> > > > > Cc: dev <dev@dpdk.org>; Jerin Jacob <jerinjacobk@gmail.com>;
> > > > > Pavan Nikhilesh <pbhagavatula@marvell.com>; Richardson, Bruce
> > > > > <bruce.richardson@intel.com>; Thomas
> > > Monjalon
> > > > > <thomas@monjalon.net>; Yigit, Ferruh <ferruh.yigit@intel.com>;
> > > > > Hemant Agrawal <hemant.agrawal@nxp.com>
> > > > > Subject: Re: [dpdk-dev] [PATCH v5] build: disable experimental
> > > > > API check internally
> > > > >
> > > > > On Fri, Apr 17, 2020 at 12:21 PM Trahe, Fiona <fiona.trahe@intel.com>
> wrote:
> > > > > > I see this is already applied.
> > > > > >
> > > > > > However,
> > > > > > rte_cryptodev_queue_pair_setup() calls
> > > > > > rte_cryptodev_sym_get_existing_header_session_size()
> > > > > > The former is a stable API, the latter is experimental.
> > > > > > So I expect the build to break when ALLOW_EXPERIMENTAL_API is
> disabled.
> > > > [Fiona] Thanks for confirming where the flag is.
> > > > But I think you've missed my point.
> > > > What about this problem?
> > >
> > > - dpdk-test-crypto-perf is built as part of the dpdk compilation itself.
> > > There is no user to be made aware of its use of experimental API.
> > >
> > > Now if you are talking about how the crypto API is bent in that it
> > > exposes a stable ABI with an underlying experimental ABI, this has
> > > nothing to do with the flag change.
> > [Fiona] Before this if an application didn't set
> > ALLOW_EXPERIMENTAL_API (which I expect is the default for many apps)
> > then the build worked fine as long as the app didn't directly call an
> > experimental API. The crypto lib still built ok as its own Makefile had the flag.
> > I just tried this with dpdk-test-crypto-perf. I had to remove one
> > direct call, without that it built ok. When I remove the flag from the
> > crypto lib Makefile, the build breaks as expected due to the above issue.
> >
> > So, yes, cryptodev lib should be fixed.
> > However this patch just applied will potentially break builds for many apps!
> > It could expose many other issues of internal dependencies on experimental
> APIs.
>
> [Fiona] Ok, I get it now, there's no issue.
> I see now why the patch name was changed from "global" to "internal"!
> I had understood that the flag the application set or didn't set would ripple
> down through the whole build. Instead the flag is set (or not set) once by the
> application only affecting direct API calls and once for the rest of the build,
> which includes apps in the app folder, but not apps in the examples folder.
> So allowing experimental APIs internally, while disallowing them externally.
> A bit confusing, but ok, doesn't break anything and definitely better than having
> flags in every Make/meson file.
>
> Fiona
>
^ permalink raw reply [relevance 3%]
Results 7601-7800 of ~18000 next (newer) | prev (older) | reverse | sort options + mbox downloads above
-- links below jump to the message on this page --
2019-05-25 18:43 [dpdk-dev] [RFC PATCH 0/2] introduce __rte_internal tag Neil Horman
2019-06-13 14:23 ` [dpdk-dev] [PATCH v2 0/10] dpdk: " Neil Horman
2019-06-13 14:23 ` [dpdk-dev] [PATCH v2 01/10] Add __rte_internal tag for functions and version target Neil Horman
2020-04-17 2:04 3% ` Wang, Haiyue
2020-04-17 2:38 4% ` Neil Horman
2020-04-17 4:40 4% ` Wang, Haiyue
2020-02-24 11:35 [dpdk-dev] [RFC 0/6] New sync modes for ring Konstantin Ananyev
2020-03-25 20:43 ` Honnappa Nagarahalli
2020-03-26 1:50 ` Ananyev, Konstantin
2020-03-30 21:29 0% ` Honnappa Nagarahalli
2020-03-30 23:37 0% ` Honnappa Nagarahalli
2020-03-31 17:21 0% ` Ananyev, Konstantin
2020-03-31 16:43 3% ` [dpdk-dev] [PATCH v1 0/8] " Konstantin Ananyev
2020-04-02 22:09 3% ` [dpdk-dev] [PATCH v2 0/9] " Konstantin Ananyev
2020-04-02 22:09 1% ` [dpdk-dev] [PATCH v2 3/9] ring: introduce RTS ring mode Konstantin Ananyev
2020-04-03 17:42 3% ` [dpdk-dev] [PATCH v3 0/9] New sync modes for ring Konstantin Ananyev
2020-04-03 17:42 ` [dpdk-dev] [PATCH v3 2/9] ring: prepare ring to allow new sync schemes Konstantin Ananyev
2020-04-08 4:59 3% ` Honnappa Nagarahalli
2020-04-09 13:39 3% ` Ananyev, Konstantin
2020-04-10 20:15 0% ` Honnappa Nagarahalli
2020-04-03 17:42 1% ` [dpdk-dev] [PATCH v3 3/9] ring: introduce RTS ring mode Konstantin Ananyev
2020-04-04 17:27 0% ` Wang, Haiyue
2020-04-08 5:00 0% ` Honnappa Nagarahalli
2020-04-09 14:52 0% ` Ananyev, Konstantin
2020-04-10 23:10 0% ` Honnappa Nagarahalli
2020-04-13 14:29 0% ` David Marchand
2020-04-17 13:36 3% ` [dpdk-dev] [PATCH v4 0/9] New sync modes for ring Konstantin Ananyev
2020-04-17 13:36 9% ` [dpdk-dev] [PATCH v4 2/9] ring: prepare ring to allow new sync schemes Konstantin Ananyev
2020-04-17 13:36 1% ` [dpdk-dev] [PATCH v4 3/9] ring: introduce RTS ring mode Konstantin Ananyev
2020-04-18 16:32 3% ` [dpdk-dev] [PATCH v5 0/9] New sync modes for ring Konstantin Ananyev
2020-04-18 16:32 9% ` [dpdk-dev] [PATCH v5 2/9] ring: prepare ring to allow new sync schemes Konstantin Ananyev
2020-04-18 16:32 1% ` [dpdk-dev] [PATCH v5 3/9] ring: introduce RTS ring mode Konstantin Ananyev
2020-03-02 1:57 [dpdk-dev] [PATCH] mempool: sort the rte_mempool_ops by name xiangxia.m.yue
2020-04-09 15:02 ` [dpdk-dev] [PATCH dpdk-dev v2 1/2] eal: introduce rte-init queue for libraries initialization xiangxia.m.yue
2020-04-10 6:18 ` Jerin Jacob
2020-04-10 13:11 ` Jerin Jacob
2020-04-12 3:20 3% ` Tonghao Zhang
2020-04-12 3:32 0% ` Tonghao Zhang
2020-04-13 11:32 0% ` Jerin Jacob
2020-04-13 14:21 3% ` [dpdk-dev] [PATCH dpdk-dev v3 " xiangxia.m.yue
2020-03-02 14:58 [dpdk-dev] [PATCH 00/16] NXP DPAAx fixes and enhancements Hemant Agrawal
2020-03-02 13:01 ` David Marchand
2020-03-05 9:06 ` Hemant Agrawal (OSS)
2020-03-05 9:09 ` David Marchand
2020-03-05 9:19 ` Hemant Agrawal (OSS)
2020-03-06 10:12 ` David Marchand
2020-03-10 10:36 ` Dodji Seketeli
2020-04-07 10:25 3% ` Hemant Agrawal
2020-04-07 12:20 0% ` Thomas Monjalon
2020-04-08 7:20 4% ` Dodji Seketeli
2020-04-08 7:52 0% ` Dodji Seketeli
2020-03-03 16:27 [dpdk-dev] [PATCH v1] mbuf: replace zero-length marker with unnamed union Gavin Hu
2020-03-07 15:56 ` [dpdk-dev] [PATCH v2] " Gavin Hu
2020-03-09 8:55 ` Ferruh Yigit
2020-03-09 9:45 ` Gavin Hu
2020-03-09 11:29 ` Ferruh Yigit
2020-03-09 13:30 ` Morten Brørup
2020-03-11 7:50 ` Gavin Hu
2020-03-11 9:04 ` Morten Brørup
2020-03-11 12:07 ` Bruce Richardson
2020-03-13 9:22 ` Gavin Hu
2020-04-07 17:13 ` Kevin Traynor
2020-04-08 15:04 ` Gavin Hu
2020-04-08 15:22 3% ` [dpdk-dev] [dpdk-stable] " David Marchand
2020-04-09 9:48 3% ` Gavin Hu
2020-04-09 10:49 3% ` Thomas Monjalon
2020-04-09 16:09 0% ` Ray Kinsella
2020-04-11 2:50 0% ` Gavin Hu
2020-03-05 4:33 [dpdk-dev] [RFC v1 1/1] vfio: set vf token and gain vf device access vattunuru
2020-04-13 8:29 2% ` [dpdk-dev] [PATCH v4] eal: add VFIO-PCI SR-IOV support Haiyue Wang
2020-04-13 12:18 3% ` Thomas Monjalon
2020-04-13 17:01 3% ` Wang, Haiyue
2020-04-13 15:37 0% ` Andrew Rybchenko
2020-04-13 16:45 0% ` Wang, Haiyue
2020-04-14 3:06 4% ` [dpdk-dev] [PATCH v5 0/2] support for VFIO-PCI VF token interface Haiyue Wang
2020-04-14 3:21 4% ` [dpdk-dev] [PATCH v6 " Haiyue Wang
2020-04-14 13:18 0% ` [dpdk-dev] [EXT] " Vamsi Krishna Attunuru
2020-04-18 11:16 4% ` [dpdk-dev] [PATCH v7 " Haiyue Wang
2020-04-18 17:30 4% ` [dpdk-dev] [PATCH v8 " Haiyue Wang
2020-03-06 16:41 [dpdk-dev] [PATCH 0/4] Introduce IF proxy library Andrzej Ostruszka
2020-03-10 11:10 ` [dpdk-dev] [PATCH v2 " Andrzej Ostruszka
2020-03-25 8:08 ` David Marchand
2020-03-26 12:41 ` Andrzej Ostruszka
2020-03-30 19:23 3% ` Andrzej Ostruszka
2020-03-12 7:44 [dpdk-dev] [PATCH v2 00/10] generic rte atomic APIs deprecate proposal Phil Yang
2020-03-17 1:17 ` [dpdk-dev] [PATCH v3 00/12] " Phil Yang
2020-03-17 1:17 ` [dpdk-dev] [PATCH v3 07/12] service: remove rte prefix from static functions Phil Yang
2020-04-03 11:57 3% ` Van Haaren, Harry
2020-04-08 10:14 0% ` Phil Yang
2020-04-08 10:36 0% ` Van Haaren, Harry
2020-04-08 10:49 0% ` Phil Yang
2020-03-12 17:20 [dpdk-dev] [PATCH 0/7] checking for owned ports in portmask Stephen Hemminger
2020-03-16 16:09 ` [dpdk-dev] [PATCH v2 0/6] check " Stephen Hemminger
2020-03-16 16:09 ` [dpdk-dev] [PATCH v2 1/6] rte_ethdev: add function to check if device is owned Stephen Hemminger
2020-04-01 21:42 ` Thomas Monjalon
2020-04-01 22:24 3% ` Stephen Hemminger
2020-04-02 8:04 3% ` Thomas Monjalon
2020-03-13 17:42 [dpdk-dev] [PATCH v1 0/7] vectorize virtio packed ring datapath Marvin Liu
2020-04-15 16:47 ` [dpdk-dev] [PATCH v4 0/8] add packed ring vectorized datapath Marvin Liu
2020-04-15 16:47 ` [dpdk-dev] [PATCH v4 6/8] eal/x86: identify AVX512 extensions flag Marvin Liu
2020-04-15 13:31 3% ` David Marchand
2020-04-15 14:57 0% ` Liu, Yong
2020-03-18 17:04 [dpdk-dev] [dpdk-dev 4/4] app/testpmd: add new types to RSS hash commands Jeff Guo
2020-04-14 17:42 ` [dpdk-dev] [dpdk-dev v4 0/3] add RSS configuration for iavf Jeff Guo
2020-04-14 17:42 ` [dpdk-dev] [dpdk-dev v4 1/3] ethdev: add new RSS offload types Jeff Guo
2020-04-14 9:42 3% ` Ori Kam
2020-04-15 2:31 0% ` Zhang, Qi Z
2020-04-15 3:11 3% ` Jeff Guo
2020-04-15 17:11 ` [dpdk-dev] [dpdk-dev v5 0/3] add RSS configuration for iavf Jeff Guo
2020-04-15 17:11 ` [dpdk-dev] [dpdk-dev v5 3/3] app/testpmd: add new types to RSS hash commands Jeff Guo
2020-04-15 15:01 ` Iremonger, Bernard
2020-04-16 7:51 3% ` Jeff Guo
2020-04-16 9:14 0% ` Iremonger, Bernard
2020-04-16 10:22 0% ` Jeff Guo
2020-03-18 20:41 [dpdk-dev] [PATCH] cryptodev: version rte_cryptodev_info_get function Arek Kusztal
2020-04-14 12:13 3% ` Akhil Goyal
2020-04-14 13:03 0% ` Thomas Monjalon
2020-04-14 13:52 0% ` Trahe, Fiona
2020-04-14 13:54 0% ` Ray Kinsella
2020-04-14 18:27 5% ` Trahe, Fiona
2020-04-15 17:24 5% ` Trahe, Fiona
2020-04-16 9:51 3% ` Bruce Richardson
2020-04-16 10:01 3% ` Thomas Monjalon
2020-04-17 7:24 5% ` Ray Kinsella
2020-04-17 9:31 4% ` Bruce Richardson
2020-04-17 9:42 3% ` Ray Kinsella
2020-04-17 10:17 4% ` Thomas Monjalon
2020-04-17 10:33 3% ` Ray Kinsella
2020-04-17 11:46 5% ` Trahe, Fiona
2020-04-17 16:01 0% ` Ray Kinsella
2020-03-19 17:18 [dpdk-dev] [PATCH 00/12] update and simplify telemetry library Ciara Power
2020-04-08 16:49 ` [dpdk-dev] [PATCH v2 00/16] " Ciara Power
2020-04-10 10:49 ` Morten Brørup
2020-04-10 14:21 ` Wiles, Keith
2020-04-10 18:06 4% ` [dpdk-dev] [PATCH v2 00/16] update and simplify telemetrylibrary Morten Brørup
2020-03-20 16:41 [dpdk-dev] [RFC] ring: make ring implementation non-inlined Konstantin Ananyev
2020-03-25 21:09 ` Jerin Jacob
2020-03-26 8:04 ` Morten Brørup
2020-03-31 23:25 5% ` Thomas Monjalon
2020-03-24 11:49 [dpdk-dev] [PATCH 00/17] Add CPU flags Kevin Laatz
2020-03-25 11:10 ` [dpdk-dev] [PATCH v2] eal/cpuflags: add x86 based cpu flags Kevin Laatz
2020-03-27 12:24 ` David Marchand
2020-03-27 13:18 5% ` Van Haaren, Harry
2020-03-27 15:04 0% ` David Marchand
2020-03-27 13:44 0% ` Neil Horman
2020-03-27 14:15 4% ` Thomas Monjalon
2020-03-27 14:32 3% ` Van Haaren, Harry
2020-03-27 14:36 3% ` Ray Kinsella
2020-03-27 15:19 3% ` Thomas Monjalon
2020-03-25 21:15 [dpdk-dev] [PATCH v2 00/32] DPDK Trace support jerinj
2020-03-29 14:43 1% ` [dpdk-dev] [PATCH v3 00/33] " jerinj
2020-04-01 8:18 3% ` David Marchand
2020-04-01 10:04 0% ` Jerin Jacob
2020-04-01 14:12 0% ` David Marchand
2020-04-01 14:16 0% ` Thomas Monjalon
2020-04-03 15:36 1% ` [dpdk-dev] [PATCH v4 " jerinj
2020-04-13 15:00 1% ` [dpdk-dev] [PATCH v5 " jerinj
2020-03-29 6:47 [dpdk-dev] [PATCH v1 0/4] add RegEx class Ori Kam
2020-04-04 14:27 ` [dpdk-dev] [EXT] [PATCH v1 4/4] regexdev: implement regex rte level functions Pavan Nikhilesh Bhagavatula
2020-04-05 15:04 ` Ori Kam
2020-04-06 11:16 3% ` Thomas Monjalon
2020-04-06 12:33 3% ` Pavan Nikhilesh Bhagavatula
2020-04-06 13:14 3% ` Thomas Monjalon
2020-04-06 13:20 0% ` Jerin Jacob
2020-04-06 13:22 0% ` Pavan Nikhilesh Bhagavatula
2020-04-06 13:36 0% ` Thomas Monjalon
2020-04-06 13:50 0% ` Pavan Nikhilesh Bhagavatula
2020-04-06 14:00 0% ` Thomas Monjalon
2020-04-06 18:53 0% ` Ori Kam
2020-03-29 22:32 [dpdk-dev] [PATCH v1] common/mlx5: remove devx depndency on ibv and dv Ophir Munk
2020-04-01 9:59 ` Raslan Darawsheh
2020-04-08 17:10 4% ` Ferruh Yigit
2020-04-09 6:21 0% ` Ray Kinsella
2020-04-09 7:24 4% ` David Marchand
2020-04-16 17:35 3% ` Ferruh Yigit
2020-04-16 20:00 4% ` Thomas Monjalon
2020-04-17 16:19 0% ` Ferruh Yigit
2020-03-30 4:10 [dpdk-dev] [RFC PATCH 0/9] Windows basic memory management Dmitry Kozlyuk
2020-03-30 4:10 3% ` [dpdk-dev] [RFC PATCH 7/9] eal/windows: fix rte_page_sizes with Clang on Windows Dmitry Kozlyuk
2020-04-10 16:43 ` [dpdk-dev] [PATCH v2 00/10] eal: Windows basic memory management Dmitry Kozlyuk
2020-04-10 16:43 4% ` [dpdk-dev] [PATCH v2 08/10] eal/windows: fix rte_page_sizes with Clang on Windows Dmitry Kozlyuk
2020-04-14 19:44 ` [dpdk-dev] [PATCH v3 00/10] Windows basic memory management Dmitry Kozlyuk
2020-04-14 19:44 4% ` [dpdk-dev] [PATCH v3 08/10] eal/windows: fix rte_page_sizes with Clang on Windows Dmitry Kozlyuk
2020-04-15 9:34 0% ` Jerin Jacob
2020-04-15 10:32 0% ` Dmitry Kozlyuk
2020-04-15 10:57 3% ` Jerin Jacob
2020-04-15 11:09 0% ` Dmitry Kozlyuk
2020-04-15 11:17 4% ` Jerin Jacob
2020-03-31 12:01 1% [dpdk-dev] [PATCH v1] doc: use svg file type for ice nic Haiyue Wang
2020-03-31 12:06 1% ` [dpdk-dev] [PATCH v2] doc: use svg file type for ice Haiyue Wang
2020-04-01 5:44 1% ` [dpdk-dev] [PATCH v3] doc: use svg file type for ice PMD Haiyue Wang
2020-03-31 17:05 7% [dpdk-dev] /validate-abi.sh complains [PATCH v1 3/8] ring: introduce RTS ring mode Ananyev, Konstantin
2020-03-31 19:22 8% ` Thomas Monjalon
2020-03-31 19:24 4% ` Thomas Monjalon
2020-04-01 12:52 4% ` Neil Horman
2020-04-06 14:02 4% ` Thomas Monjalon
2020-04-06 14:36 4% ` Neil Horman
2020-04-07 7:32 4% ` Ray Kinsella
2020-04-07 11:31 4% ` Neil Horman
2020-04-01 6:38 9% ` David Marchand
2020-04-02 12:36 4% ` Ananyev, Konstantin
2020-04-02 13:13 4% ` Dodji Seketeli
2020-04-02 14:27 4% ` David Marchand
2020-04-02 15:25 4% ` Ananyev, Konstantin
2020-04-01 9:33 [dpdk-dev] [RFC] ethdev: use special speed for virtual Ethernet devices Morten Brørup
2020-04-02 13:50 ` [dpdk-dev] [RFC] ethdev: use special speed for virtual Ethernetdevices Morten Brørup
2020-04-02 20:41 ` Ivan Dyukov
[not found] ` <CGME20200402205851eucas1p15d9b25f7e5238379754bd0959b04adc8@eucas1p1.samsung.com>
2020-04-02 20:58 ` Thomas Monjalon
2020-04-03 8:05 ` Ivan Dyukov
2020-04-03 9:45 3% ` Morten Brørup
2020-04-03 11:01 0% ` Thomas Monjalon
2020-04-02 11:26 [dpdk-dev] [PATCH 0/4] vhost: support vDPA virtio queue statistics Matan Azrad
2020-04-02 11:26 ` [dpdk-dev] [PATCH 1/4] vhost: inroduce operation to get vDPA queue stats Matan Azrad
2020-04-15 14:36 3% ` Maxime Coquelin
2020-04-16 9:06 3% ` Matan Azrad
2020-04-16 13:19 3% ` Maxime Coquelin
2020-04-02 18:32 [dpdk-dev] [PATCH v2 1/2] build: meson make experimental tag as global Jerin Jacob
2020-04-13 14:55 ` [dpdk-dev] [PATCH v5] build: disable experimental API check internally David Marchand
2020-04-14 14:23 ` David Marchand
2020-04-17 10:21 ` Trahe, Fiona
2020-04-17 13:23 ` David Marchand
2020-04-17 13:44 ` Trahe, Fiona
2020-04-17 13:56 4% ` David Marchand
2020-04-17 15:05 0% ` Trahe, Fiona
2020-04-17 15:52 0% ` Trahe, Fiona
2020-04-18 19:43 3% ` Chautru, Nicolas
2020-04-03 16:36 [dpdk-dev] [PATCH v2 0/4] introduce multi-function processing support David Coyle
2020-04-06 14:28 ` Ferruh Yigit
2020-04-07 11:27 ` Coyle, David
2020-04-07 18:05 2% ` Trahe, Fiona
2020-04-09 9:25 0% ` Coyle, David
2020-04-09 9:37 0% ` Trahe, Fiona
2020-04-09 11:55 0% ` Coyle, David
2020-04-09 13:05 0% ` Trahe, Fiona
2020-04-06 19:34 38% [dpdk-dev] [PATCH] Remove abi_versioning.sh from tree Neil Horman
2020-04-07 7:36 4% ` David Marchand
2020-04-07 11:33 0% ` Neil Horman
2020-04-07 11:40 0% ` David Marchand
2020-04-07 11:58 5% ` Thomas Monjalon
2020-04-07 19:52 3% ` Neil Horman
2020-04-08 14:34 0% ` Ray Kinsella
2020-04-08 17:41 5% ` Thomas Monjalon
2020-04-08 14:50 0% ` Ray Kinsella
2020-04-08 14:49 0% ` Ray Kinsella
2020-04-09 9:26 3% ` Bruce Richardson
2020-04-09 10:04 0% ` Ray Kinsella
2020-04-08 17:56 [dpdk-dev] [RFC 1/3] eventdev: allow for event devices requiring maintenance Mattias Rönnblom
2020-04-08 19:36 ` Jerin Jacob
2020-04-09 12:21 ` Mattias Rönnblom
2020-04-09 13:33 3% ` Eads, Gage
2020-04-09 14:14 0% ` Mattias Rönnblom
2020-04-08 19:56 38% [dpdk-dev] [PATCHv2] Remove validate-abi.sh from tree Neil Horman
2020-04-09 7:57 4% ` Ray Kinsella
2020-04-09 10:39 4% ` Neil Horman
2020-04-09 10:43 4% ` Bruce Richardson
2020-04-09 10:45 4% ` Ray Kinsella
2020-04-09 10:59 7% ` Thomas Monjalon
2020-04-09 13:02 4% ` Neil Horman
2020-04-09 13:37 4% ` Thomas Monjalon
2020-04-09 14:52 9% ` Ray Kinsella
2020-04-09 15:18 8% ` Thomas Monjalon
2020-04-09 16:29 4% ` Ray Kinsella
2020-04-09 16:51 4% ` Thomas Monjalon
2020-04-10 6:26 4% ` Ray Kinsella
2020-04-10 7:57 4% ` Thomas Monjalon
2020-04-09 12:42 4% [dpdk-dev] DPDK Release Status Meeting 9/04/2020 Ferruh Yigit
2020-04-09 14:14 3% [dpdk-dev] [PATCH] build: add arm 32bit cross compilation to Meson Juraj Linkeš
2020-04-13 6:23 0% ` Ruifeng Wang
2020-04-14 7:05 0% ` Juraj Linkeš
2020-04-10 9:46 [dpdk-dev] [PATCH] ethdev: support flow aging BillZhou
2020-04-14 8:32 ` [dpdk-dev] [PATCH v2] " Dong Zhou
2020-04-17 22:00 3% ` Ferruh Yigit
2020-04-17 22:07 0% ` Stephen Hemminger
2020-04-18 5:04 0% ` Bill Zhou
2020-04-18 9:44 0% ` Thomas Monjalon
2020-04-16 8:12 9% [dpdk-dev] [PATCH v2] cryptodev: version rte_cryptodev_info_get function Arek Kusztal
2020-04-16 8:18 3% ` Akhil Goyal
2020-04-16 8:31 4% ` Thomas Monjalon
2020-04-16 14:03 0% ` Ray Kinsella
2020-04-16 10:06 5% [dpdk-dev] DPDK Release Status Meeting 16/04/2020 Ferruh Yigit
2020-04-16 10:49 0% ` Thomas Monjalon
2020-04-16 11:21 0% ` Ferruh Yigit
2020-04-16 14:54 38% [dpdk-dev] [PATCHv3] Remove validate-abi.sh from tree Neil Horman
2020-04-17 10:11 9% ` Ray Kinsella
2020-04-17 10:20 8% ` Thomas Monjalon
2020-04-17 10:35 4% ` Ray Kinsella
2020-04-17 11:46 4% ` Thomas Monjalon
2020-04-17 11:47 4% ` Ray Kinsella
2020-04-17 12:10 4% ` Thomas Monjalon
2020-04-17 15:42 9% ` Ray Kinsella
2020-04-17 16:10 4% ` Thomas Monjalon
2020-04-17 14:59 11% [dpdk-dev] [PATCH v3] cryptodev: version cryptodev info get function Arek Kusztal
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).