[RFC 0/2] add pointer compression API

[RFC 0/2] add pointer compression API

[Paul Szczepanek]

This patchset is proposing adding a new EAL header with utility functions
that allow compression of arrays of pointers.

When passing caches full of pointers between threads, memory containing
the pointers is copied multiple times which is especially costly between
cores. A compression method will allow us to shrink the memory size
copied.

The compression takes advantage of the fact that pointers are usually
located in a limited memory region (like a mempool). We can compress them
by converting them to offsets from a base memory address.

Offsets can be stored in fewer bytes (dictated by the memory region size
and alignment of the pointer). For example: an 8 byte aligned pointer
which is part of a 32GB memory pool can be stored in 4 bytes. The API is
very generic and does not assume mempool pointers, any pointer can be
passed in.

Compression is based on few and fast operations and especially with vector
instructions leveraged creates minimal overhead.

The API accepts and returns arrays because the overhead means it only is
worth it when done in bulk.

Test is added that shows potential performance gain from compression. In
this test an array of pointers is passed through a ring between two cores.
It shows the gain which is dependent on the bulk operation size. In this
synthetic test run on ampere altra a substantial (up to 25%) performance
gain is seen if done in bulk size larger than 32. At 32 it breaks even and
lower sizes create a small (less than 5%) slowdown due to overhead.

In a more realistic mock application running the l3 forwarding dpdk
example that works in pipeline mode this translated into a ~5% throughput
increase on an ampere altra.

Paul Szczepanek (2):
  eal: add pointer compression functions
  test: add pointer compress tests to ring perf test

 .mailmap                           |   1 +
 app/test/test_ring.h               |  59 +++++-
 app/test/test_ring_perf.c          | 324 ++++++++++++++++++-----------
 lib/eal/include/meson.build        |   1 +
 lib/eal/include/rte_ptr_compress.h | 158 ++++++++++++++
 5 files changed, 419 insertions(+), 124 deletions(-)
 create mode 100644 lib/eal/include/rte_ptr_compress.h

-- 
2.25.1

[RFC 1/2] eal: add pointer compression functions

[Paul Szczepanek]

Add a new utility header for compressing pointers. Pointers are
compressed by taking advantage of their locality. Instead of
storing the full address only an offset from a known base is stored.

The provided functions can store pointers in 32bit offsets.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .mailmap                           |   1 +
 lib/eal/include/meson.build        |   1 +
 lib/eal/include/rte_ptr_compress.h | 158 +++++++++++++++++++++++++++++
 3 files changed, 160 insertions(+)
 create mode 100644 lib/eal/include/rte_ptr_compress.h

diff --git a/.mailmap b/.mailmap
index 864d33ee46..3f0c9d32f5 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1058,6 +1058,7 @@ Paul Greenwalt <paul.greenwalt@intel.com>
 Paulis Gributs <paulis.gributs@intel.com>
 Paul Luse <paul.e.luse@intel.com>
 Paul M Stillwell Jr <paul.m.stillwell.jr@intel.com>
+Paul Szczepanek <paul.szczepanek@arm.com>
 Pavan Kumar Linga <pavan.kumar.linga@intel.com>
 Pavan Nikhilesh <pbhagavatula@marvell.com> <pbhagavatula@caviumnetworks.com>
 Pavel Belous <pavel.belous@aquantia.com>
diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index a0463efac7..60b056ef96 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -35,6 +35,7 @@ headers += files(
         'rte_pci_dev_feature_defs.h',
         'rte_pci_dev_features.h',
         'rte_per_lcore.h',
+	'rte_ptr_compress.h',
         'rte_pflock.h',
         'rte_random.h',
         'rte_reciprocal.h',
diff --git a/lib/eal/include/rte_ptr_compress.h b/lib/eal/include/rte_ptr_compress.h
new file mode 100644
index 0000000000..6498587c0b
--- /dev/null
+++ b/lib/eal/include/rte_ptr_compress.h
@@ -0,0 +1,158 @@
+/* SPDX-License-Identifier: BSD-shift-Clause
+ * Copyright(c) 2023 Arm Limited
+ */
+
+#ifndef _RTE_PTR_COMPRESS_H_
+#define _RTE_PTR_COMPRESS_H_
+
+/**
+ * @file
+ * RTE pointer compression and decompression.
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_vect.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Compress pointers into 32 bit offsets from base pointer.
+ *
+ * @note Offsets from the base pointer must fit within 32bits. Alignment allows
+ * us to drop bits from the offsets - this means that for pointers aligned by
+ * 8 bytes they must be within 32GB of the base pointer. Unaligned pointers
+ * must be within 4GB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_32(void *ptr_base, void **src_table,
+		uint32_t *dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_src_table;
+	svuint64_t v_dest_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_src_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_dest_table = svsub_x(pg, v_src_table, (uint64_t)ptr_base);
+		v_dest_table = svlsr_x(pg, v_dest_table, bit_shift);
+		svst1w(pg, &dest_table[i], v_dest_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON
+	uint64_t ptr_diff;
+	uint64x2_t v_src_table;
+	uint64x2_t v_dest_table;
+	/* right shift is done by left shifting by negative int */
+	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_src_table = vld1q_u64((const uint64_t *)src_table + i);
+		v_dest_table = vsubq_u64(v_src_table, v_ptr_base);
+		v_dest_table = vshlq_u64(v_dest_table, v_shift);
+		vst1_u32(dest_table + i, vqmovn_u64(v_dest_table));
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
+	}
+#else
+	uint64_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		/* save extra bits that are redundant due to alignment */
+		ptr_diff = ptr_diff >> bit_shift;
+		/* make sure no truncation will happen when casting */
+		RTE_ASSERT(ptr_diff <= UINT32_MAX);
+		dest_table[i] = (uint32_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 32 bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_32(void *ptr_base, uint32_t *src_table,
+		void **dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_src_table;
+	svuint64_t v_dest_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_src_table = svld1uw_u64(pg, &src_table[i]);
+		v_src_table = svlsl_x(pg, v_src_table, bit_shift);
+		v_dest_table = svadd_x(pg, v_src_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_dest_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON
+	uint64_t ptr_diff;
+	uint64x2_t v_src_table;
+	uint64x2_t v_dest_table;
+	int64x2_t v_shift = vdupq_n_s64(bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_src_table = vmovl_u32(vld1_u32(src_table + i));
+		v_src_table = vshlq_u64(v_dest_table, v_shift);
+		v_dest_table = vaddq_u64(v_src_table, v_ptr_base);
+		vst1q_u64((uint64_t *)dest_table + i, v_dest_table);
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#else
+	uint64_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PTR_COMPRESS_H_ */
-- 
2.25.1

Re: [RFC 1/2] eal: add pointer compression functions

[Thomas Monjalon]

27/09/2023 17:08, Paul Szczepanek:
> Add a new utility header for compressing pointers. Pointers are
> compressed by taking advantage of their locality. Instead of
> storing the full address only an offset from a known base is stored.

You probably need to insert some explanations from the cover letter.

> The provided functions can store pointers in 32bit offsets.
> 
> Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
[...]
> --- a/lib/eal/include/meson.build
> +++ b/lib/eal/include/meson.build
> @@ -35,6 +35,7 @@ headers += files(
>          'rte_pci_dev_feature_defs.h',
>          'rte_pci_dev_features.h',
>          'rte_per_lcore.h',
> +	'rte_ptr_compress.h',
>          'rte_pflock.h',
>          'rte_random.h',
>          'rte_reciprocal.h',

Did you try to sort alphabetically? failed :)

> +#ifndef _RTE_PTR_COMPRESS_H_
> +#define _RTE_PTR_COMPRESS_H_

No need extra underscores.

> +
> +/**
> + * @file
> + * RTE pointer compression and decompression.

RTE has no mean here I think.

> + */
> +
> +#include <stdint.h>
> +#include <inttypes.h>
> +
> +#include <rte_branch_prediction.h>
> +#include <rte_common.h>
> +#include <rte_debug.h>
> +#include <rte_vect.h>
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +/**
> + * Compress pointers into 32 bit offsets from base pointer.

I think it should be "32-bit".

> + *
> + * @note Offsets from the base pointer must fit within 32bits. Alignment allows
> + * us to drop bits from the offsets - this means that for pointers aligned by
> + * 8 bytes they must be within 32GB of the base pointer. Unaligned pointers
> + * must be within 4GB.

Not clear what is "alignment".
> + *
> + * @param ptr_base
> + *   A pointer used to calculate offsets of pointers in src_table.
> + * @param src_table
> + *   A pointer to an array of pointers.
> + * @param dest_table
> + *   A pointer to an array of compressed pointers returned by this function.
> + * @param n
> + *   The number of objects to compress, must be strictly positive.
> + * @param bit_shift
> + *   Byte alignment of memory pointed to by the pointers allows for
> + *   bits to be dropped from the offset and hence widen the memory region that
> + *   can be covered. This controls how many bits are right shifted.
> + **/
> +static __rte_always_inline void
> +rte_ptr_compress_32(void *ptr_base, void **src_table,
> +		uint32_t *dest_table, unsigned int n, unsigned int bit_shift)
> +{
> +	unsigned int i = 0;
> +#if defined RTE_HAS_SVE_ACLE
> +	svuint64_t v_src_table;
> +	svuint64_t v_dest_table;
> +	svbool_t pg = svwhilelt_b64(i, n);
> +	do {
> +		v_src_table = svld1_u64(pg, (uint64_t *)src_table + i);
> +		v_dest_table = svsub_x(pg, v_src_table, (uint64_t)ptr_base);
> +		v_dest_table = svlsr_x(pg, v_dest_table, bit_shift);
> +		svst1w(pg, &dest_table[i], v_dest_table);
> +		i += svcntd();
> +		pg = svwhilelt_b64(i, n);
> +	} while (svptest_any(svptrue_b64(), pg));
> +#elif defined __ARM_NEON
> +	uint64_t ptr_diff;
> +	uint64x2_t v_src_table;
> +	uint64x2_t v_dest_table;
> +	/* right shift is done by left shifting by negative int */
> +	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
> +	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
> +	for (; i < (n & ~0x1); i += 2) {
> +		v_src_table = vld1q_u64((const uint64_t *)src_table + i);
> +		v_dest_table = vsubq_u64(v_src_table, v_ptr_base);
> +		v_dest_table = vshlq_u64(v_dest_table, v_shift);
> +		vst1_u32(dest_table + i, vqmovn_u64(v_dest_table));
> +	}
> +	/* process leftover single item in case of odd number of n */
> +	if (unlikely(n & 0x1)) {
> +		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
> +		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
> +	}
> +#else
> +	uint64_t ptr_diff;
> +	for (; i < n; i++) {
> +		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
> +		/* save extra bits that are redundant due to alignment */
> +		ptr_diff = ptr_diff >> bit_shift;
> +		/* make sure no truncation will happen when casting */
> +		RTE_ASSERT(ptr_diff <= UINT32_MAX);
> +		dest_table[i] = (uint32_t) ptr_diff;
> +	}
> +#endif
> +}

I see it is providing some per-CPU optimizations,
so it is in favor of having it in DPDK.
Other than that, it looks very generic, so it is questionable to have in DPDK.

RE: [RFC 1/2] eal: add pointer compression functions

[Honnappa Nagarahalli]

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Monday, October 9, 2023 10:54 AM
> To: Paul Szczepanek <Paul.Szczepanek@arm.com>
> Cc: dev@dpdk.org; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>;
> Kamalakshitha Aligeri <Kamalakshitha.Aligeri@arm.com>
> Subject: Re: [RFC 1/2] eal: add pointer compression functions
> 
> 27/09/2023 17:08, Paul Szczepanek:
> > Add a new utility header for compressing pointers. Pointers are
> > compressed by taking advantage of their locality. Instead of storing
> > the full address only an offset from a known base is stored.
> 
> You probably need to insert some explanations from the cover letter.
> 
> > The provided functions can store pointers in 32bit offsets.
> >
> > Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> > Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
> > Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
> > Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> [...]
> > --- a/lib/eal/include/meson.build
> > +++ b/lib/eal/include/meson.build
> > @@ -35,6 +35,7 @@ headers += files(
> >          'rte_pci_dev_feature_defs.h',
> >          'rte_pci_dev_features.h',
> >          'rte_per_lcore.h',
> > +	'rte_ptr_compress.h',
> >          'rte_pflock.h',
> >          'rte_random.h',
> >          'rte_reciprocal.h',
> 
> Did you try to sort alphabetically? failed :)
> 
> > +#ifndef _RTE_PTR_COMPRESS_H_
> > +#define _RTE_PTR_COMPRESS_H_
> 
> No need extra underscores.
> 
> > +
> > +/**
> > + * @file
> > + * RTE pointer compression and decompression.
> 
> RTE has no mean here I think.
> 
> > + */
> > +
> > +#include <stdint.h>
> > +#include <inttypes.h>
> > +
> > +#include <rte_branch_prediction.h>
> > +#include <rte_common.h>
> > +#include <rte_debug.h>
> > +#include <rte_vect.h>
> > +
> > +#ifdef __cplusplus
> > +extern "C" {
> > +#endif
> > +
> > +/**
> > + * Compress pointers into 32 bit offsets from base pointer.
> 
> I think it should be "32-bit".
> 
> > + *
> > + * @note Offsets from the base pointer must fit within 32bits.
> > + Alignment allows
> > + * us to drop bits from the offsets - this means that for pointers
> > + aligned by
> > + * 8 bytes they must be within 32GB of the base pointer. Unaligned
> > + pointers
> > + * must be within 4GB.
> 
> Not clear what is "alignment".
> > + *
> > + * @param ptr_base
> > + *   A pointer used to calculate offsets of pointers in src_table.
> > + * @param src_table
> > + *   A pointer to an array of pointers.
> > + * @param dest_table
> > + *   A pointer to an array of compressed pointers returned by this function.
> > + * @param n
> > + *   The number of objects to compress, must be strictly positive.
> > + * @param bit_shift
> > + *   Byte alignment of memory pointed to by the pointers allows for
> > + *   bits to be dropped from the offset and hence widen the memory region
> that
> > + *   can be covered. This controls how many bits are right shifted.
> > + **/
> > +static __rte_always_inline void
> > +rte_ptr_compress_32(void *ptr_base, void **src_table,
> > +		uint32_t *dest_table, unsigned int n, unsigned int bit_shift) {
> > +	unsigned int i = 0;
> > +#if defined RTE_HAS_SVE_ACLE
> > +	svuint64_t v_src_table;
> > +	svuint64_t v_dest_table;
> > +	svbool_t pg = svwhilelt_b64(i, n);
> > +	do {
> > +		v_src_table = svld1_u64(pg, (uint64_t *)src_table + i);
> > +		v_dest_table = svsub_x(pg, v_src_table, (uint64_t)ptr_base);
> > +		v_dest_table = svlsr_x(pg, v_dest_table, bit_shift);
> > +		svst1w(pg, &dest_table[i], v_dest_table);
> > +		i += svcntd();
> > +		pg = svwhilelt_b64(i, n);
> > +	} while (svptest_any(svptrue_b64(), pg)); #elif defined __ARM_NEON
> > +	uint64_t ptr_diff;
> > +	uint64x2_t v_src_table;
> > +	uint64x2_t v_dest_table;
> > +	/* right shift is done by left shifting by negative int */
> > +	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
> > +	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
> > +	for (; i < (n & ~0x1); i += 2) {
> > +		v_src_table = vld1q_u64((const uint64_t *)src_table + i);
> > +		v_dest_table = vsubq_u64(v_src_table, v_ptr_base);
> > +		v_dest_table = vshlq_u64(v_dest_table, v_shift);
> > +		vst1_u32(dest_table + i, vqmovn_u64(v_dest_table));
> > +	}
> > +	/* process leftover single item in case of odd number of n */
> > +	if (unlikely(n & 0x1)) {
> > +		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
> > +		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
> > +	}
> > +#else
> > +	uint64_t ptr_diff;
> > +	for (; i < n; i++) {
> > +		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
> > +		/* save extra bits that are redundant due to alignment */
> > +		ptr_diff = ptr_diff >> bit_shift;
> > +		/* make sure no truncation will happen when casting */
> > +		RTE_ASSERT(ptr_diff <= UINT32_MAX);
> > +		dest_table[i] = (uint32_t) ptr_diff;
> > +	}
> > +#endif
> > +}
> 
> I see it is providing some per-CPU optimizations, so it is in favor of having it in
> DPDK.
> Other than that, it looks very generic, so it is questionable to have in DPDK.
We had it done for mbuf pointers. But then, we thought it could be generic.

Right now the API results in 32b indices. We could make it generic further by allowing for 16b indices. 8b indices does not make sense.
> 
> 

Re: [RFC 1/2] eal: add pointer compression functions

[Paul Szczepanek]

On 11/10/2023 14:36, Honnappa Nagarahalli wrote:
>> -----Original Message-----
>> From: Thomas Monjalon <thomas@monjalon.net>
>> Sent: Monday, October 9, 2023 10:54 AM
>> To: Paul Szczepanek <Paul.Szczepanek@arm.com>
>> Cc: dev@dpdk.org; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>;
>> Kamalakshitha Aligeri <Kamalakshitha.Aligeri@arm.com>
>> Subject: Re: [RFC 1/2] eal: add pointer compression functions
[...]
>> I see it is providing some per-CPU optimizations, so it is in favor of having it in
>> DPDK.
>> Other than that, it looks very generic, so it is questionable to have in DPDK.
> We had it done for mbuf pointers. But then, we thought it could be generic.
>
> Right now the API results in 32b indices. We could make it generic further by allowing for 16b indices. 8b indices does not make sense.

To add to this, this being generic is I think a good thing.

I think it belongs in DPDK as it will make it easy for other 
architectures to add their versions and maintain the abstraction.

[RFC v2 0/2] add pointer compression API

[Paul Szczepanek]

This patchset is proposing adding a new EAL header with utility functions
that allow compression of arrays of pointers.

When passing caches full of pointers between threads, memory containing
the pointers is copied multiple times which is especially costly between
cores. A compression method will allow us to shrink the memory size
copied.

The compression takes advantage of the fact that pointers are usually
located in a limited memory region (like a mempool). We can compress them
by converting them to offsets from a base memory address.

Offsets can be stored in fewer bytes (dictated by the memory region size
and alignment of the pointer). For example: an 8 byte aligned pointer
which is part of a 32GB memory pool can be stored in 4 bytes. The API is
very generic and does not assume mempool pointers, any pointer can be
passed in.

Compression is based on few and fast operations and especially with vector
instructions leveraged creates minimal overhead.

The API accepts and returns arrays because the overhead means it only is
worth it when done in bulk.

Test is added that shows potential performance gain from compression. In
this test an array of pointers is passed through a ring between two cores.
It shows the gain which is dependent on the bulk operation size. In this
synthetic test run on ampere altra a substantial (up to 25%) performance
gain is seen if done in bulk size larger than 32. At 32 it breaks even and
lower sizes create a small (less than 5%) slowdown due to overhead.

In a more realistic mock application running the l3 forwarding dpdk
example that works in pipeline mode this translated into a ~5% throughput
increase on an ampere altra.

v2:
* addressed review comments (style, explanations and typos)
* lowered bulk iterations closer to original numbers to keep runtime short
* fixed pointer size warning on 32-bit arch

Paul Szczepanek (2):
  eal: add pointer compression functions
  test: add pointer compress tests to ring perf test

 .mailmap                           |   1 +
 app/test/test_ring.h               |  59 +++++-
 app/test/test_ring_perf.c          | 324 ++++++++++++++++++-----------
 lib/eal/include/meson.build        |   1 +
 lib/eal/include/rte_ptr_compress.h | 160 ++++++++++++++
 5 files changed, 421 insertions(+), 124 deletions(-)
 create mode 100644 lib/eal/include/rte_ptr_compress.h

-- 
2.25.1

[RFC v2 1/2] eal: add pointer compression functions

[Paul Szczepanek]

Add a new utility header for compressing pointers. The provided
functions can store pointers in 32-bit offsets.

The compression takes advantage of the fact that pointers are
usually located in a limited memory region (like a mempool).
We can compress them by converting them to offsets from a base
memory address. Offsets can be stored in fewer bytes (dictated
by the memory region size and alignment of the pointer).
For example: an 8 byte aligned pointer which is part of a 32GB
memory pool can be stored in 4 bytes.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .mailmap                           |   1 +
 lib/eal/include/meson.build        |   1 +
 lib/eal/include/rte_ptr_compress.h | 160 +++++++++++++++++++++++++++++
 3 files changed, 162 insertions(+)
 create mode 100644 lib/eal/include/rte_ptr_compress.h

diff --git a/.mailmap b/.mailmap
index 864d33ee46..3f0c9d32f5 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1058,6 +1058,7 @@ Paul Greenwalt <paul.greenwalt@intel.com>
 Paulis Gributs <paulis.gributs@intel.com>
 Paul Luse <paul.e.luse@intel.com>
 Paul M Stillwell Jr <paul.m.stillwell.jr@intel.com>
+Paul Szczepanek <paul.szczepanek@arm.com>
 Pavan Kumar Linga <pavan.kumar.linga@intel.com>
 Pavan Nikhilesh <pbhagavatula@marvell.com> <pbhagavatula@caviumnetworks.com>
 Pavel Belous <pavel.belous@aquantia.com>
diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index a0463efac7..17d8373648 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -36,6 +36,7 @@ headers += files(
         'rte_pci_dev_features.h',
         'rte_per_lcore.h',
         'rte_pflock.h',
+	'rte_ptr_compress.h',
         'rte_random.h',
         'rte_reciprocal.h',
         'rte_seqcount.h',
diff --git a/lib/eal/include/rte_ptr_compress.h b/lib/eal/include/rte_ptr_compress.h
new file mode 100644
index 0000000000..73bde22973
--- /dev/null
+++ b/lib/eal/include/rte_ptr_compress.h
@@ -0,0 +1,160 @@
+/* SPDX-License-Identifier: BSD-shift-Clause
+ * Copyright(c) 2023 Arm Limited
+ */
+
+#ifndef RTE_PTR_COMPRESS_H
+#define RTE_PTR_COMPRESS_H
+
+/**
+ * @file
+ * Pointer compression and decompression.
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_vect.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Compress pointers into 32-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 32 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 32GB of the base pointer. If there is no such alignment guarantee they
+ * must be within 4GB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_32(void *ptr_base, void **src_table,
+		uint32_t *dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_src_table;
+	svuint64_t v_dest_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_src_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_dest_table = svsub_x(pg, v_src_table, (uint64_t)ptr_base);
+		v_dest_table = svlsr_x(pg, v_dest_table, bit_shift);
+		svst1w(pg, &dest_table[i], v_dest_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON
+	uint64_t ptr_diff;
+	uint64x2_t v_src_table;
+	uint64x2_t v_dest_table;
+	/* right shift is done by left shifting by negative int */
+	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_src_table = vld1q_u64((const uint64_t *)src_table + i);
+		v_dest_table = vsubq_u64(v_src_table, v_ptr_base);
+		v_dest_table = vshlq_u64(v_dest_table, v_shift);
+		vst1_u32(dest_table + i, vqmovn_u64(v_dest_table));
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		/* save extra bits that are redundant due to alignment */
+		ptr_diff = ptr_diff >> bit_shift;
+		/* make sure no truncation will happen when casting */
+		RTE_ASSERT(ptr_diff <= UINT32_MAX);
+		dest_table[i] = (uint32_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 32-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_32(void *ptr_base, uint32_t *src_table,
+		void **dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_src_table;
+	svuint64_t v_dest_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_src_table = svld1uw_u64(pg, &src_table[i]);
+		v_src_table = svlsl_x(pg, v_src_table, bit_shift);
+		v_dest_table = svadd_x(pg, v_src_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_dest_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON
+	uint64_t ptr_diff;
+	uint64x2_t v_src_table;
+	uint64x2_t v_dest_table;
+	int64x2_t v_shift = vdupq_n_s64(bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_src_table = vmovl_u32(vld1_u32(src_table + i));
+		v_src_table = vshlq_u64(v_dest_table, v_shift);
+		v_dest_table = vaddq_u64(v_src_table, v_ptr_base);
+		vst1q_u64((uint64_t *)dest_table + i, v_dest_table);
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_PTR_COMPRESS_H */
-- 
2.25.1

[RFC v2 2/2] test: add pointer compress tests to ring perf test

[Paul Szczepanek]

Add a test that runs a zero copy burst enqueue and dequeue on a ring
of raw pointers and compressed pointers at different burst sizes to
showcase performance benefits of newly added pointer compression APIs.

Refactored threading code to pass more parameters to threads to
reuse existing code. Added more bulk sizes to showcase their effects
on compression. Adjusted loop iteration numbers to take into account
bulk sizes to keep runtime constant (instead of number of operations).

Adjusted old printfs to match new ones which have aligned numbers.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 app/test/test_ring.h      |  59 ++++++-
 app/test/test_ring_perf.c | 324 +++++++++++++++++++++++---------------
 2 files changed, 259 insertions(+), 124 deletions(-)

diff --git a/app/test/test_ring.h b/app/test/test_ring.h
index 45c263f3ff..e8b7525c23 100644
--- a/app/test/test_ring.h
+++ b/app/test/test_ring.h
@@ -1,10 +1,12 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */
 
 #include <rte_malloc.h>
 #include <rte_ring.h>
 #include <rte_ring_elem.h>
+#include <rte_memcpy.h>
+#include <rte_ptr_compress.h>
 
 /* API type to call
  * rte_ring_<sp/mp or sc/mc>_enqueue_<bulk/burst>
@@ -25,6 +27,9 @@
 #define TEST_RING_ELEM_BULK 16
 #define TEST_RING_ELEM_BURST 32
 
+#define TEST_RING_ELEM_BURST_ZC 64
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 128
+
 #define TEST_RING_IGNORE_API_TYPE ~0U
 
 /* This function is placed here as it is required for both
@@ -101,6 +106,9 @@ static inline unsigned int
 test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -152,6 +160,29 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n,
 								NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy(zcd.ptr2,
+						(char *)obj + zcd.n1 * esize,
+						(ret - zcd.n1) * esize);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_32(0, obj, zcd.ptr1, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_32(0, obj + zcd.n1,
+						zcd.ptr2, ret - zcd.n1, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
@@ -162,6 +193,9 @@ static inline unsigned int
 test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -213,6 +247,29 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mc_dequeue_burst_elem(r, obj, esize,
 								n, NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy((char *)obj + zcd.n1 * esize,
+						zcd.ptr2,
+						(ret - zcd.n1) * esize);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_32(0, zcd.ptr1, obj, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_32(0, zcd.ptr2,
+						obj + zcd.n1, ret - zcd.n1, 3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c
index d7c5a4c30b..3c9dc24447 100644
--- a/app/test/test_ring_perf.c
+++ b/app/test/test_ring_perf.c
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2010-2014 Intel Corporation
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */
 
 
@@ -21,14 +21,15 @@
  */
 
 #define RING_NAME "RING_PERF"
+#define RING_NAME_COMPRESS "RING_COMP"
 #define RING_SIZE 4096
-#define MAX_BURST 32
+#define MAX_BURST 128
 
 /*
  * the sizes to enqueue and dequeue in testing
  * (marked volatile so they won't be seen as compile-time constants)
  */
-static const volatile unsigned bulk_sizes[] = { 8, 32 };
+static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128 };
 
 struct lcore_pair {
 	unsigned c1, c2;
@@ -43,26 +44,29 @@ test_ring_print_test_string(unsigned int api_type, int esize,
 	if (esize == -1)
 		printf("legacy APIs");
 	else
-		printf("elem APIs: element size %dB", esize);
+		printf("elem APIs (size:%2dB)", esize);
 
 	if (api_type == TEST_RING_IGNORE_API_TYPE)
 		return;
 
 	if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF)
-		printf(": default enqueue/dequeue: ");
+		printf(" - default enqueue/dequeue");
 	else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC)
-		printf(": SP/SC: ");
+		printf(" - SP/SC");
 	else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC)
-		printf(": MP/MC: ");
+		printf(" - MP/MC");
 
 	if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE)
-		printf("single: ");
+		printf(" - single - ");
 	else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK)
-		printf("bulk (size: %u): ", bsz);
+		printf(" - bulk (n:%-3u) - ", bsz);
 	else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST)
-		printf("burst (size: %u): ", bsz);
+		printf(" - burst (n:%-3u) - ", bsz);
+	else if ((api_type & (TEST_RING_ELEM_BURST_ZC |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0)
+		printf(" - burst zero copy (n:%-3u) - ", bsz);
 
-	printf("%.2F\n", value);
+	printf("cycles per elem: %.3F\n", value);
 }
 
 /**** Functions to analyse our core mask to get cores for different tests ***/
@@ -144,28 +148,33 @@ static void
 test_empty_dequeue(struct rte_ring *r, const int esize,
 			const unsigned int api_type)
 {
-	const unsigned int iter_shift = 26;
+	const unsigned int iter_shift = 30;
 	const unsigned int iterations = 1 << iter_shift;
 	unsigned int i = 0;
 	void *burst[MAX_BURST];
 
+	const unsigned int bulk_iterations = iterations / bulk_sizes[0];
 	const uint64_t start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	for (i = 0; i < bulk_iterations; i++)
 		test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type);
 	const uint64_t end = rte_rdtsc();
 
 	test_ring_print_test_string(api_type, esize, bulk_sizes[0],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 }
 
-/*
- * for the separate enqueue and dequeue threads they take in one param
- * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc
- */
-struct thread_params {
+/* describes the ring used by the enqueue and dequeue thread */
+struct ring_params {
 	struct rte_ring *r;
-	unsigned size;        /* input value, the burst size */
-	double spsc, mpmc;    /* output value, the single or multi timings */
+	unsigned int elem_size;
+	unsigned int bulk_sizes_i; /* index into bulk_size array */
+	unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */
+};
+
+/* Used do specify enqueue and dequeue ring operations and their results */
+struct thread_params {
+	struct ring_params *ring_params;
+	double *results; /* result array size must be equal to bulk_sizes */
 };
 
 /*
@@ -174,16 +183,15 @@ struct thread_params {
  * flag == 1 -> dequeue
  */
 static __rte_always_inline int
-enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
-	struct thread_params *p)
+enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 15;
+	const unsigned int iter_shift = 30;
 	const unsigned int iterations = 1 << iter_shift;
-	struct rte_ring *r = p->r;
-	unsigned int bsize = p->size;
 	unsigned int i;
 	void *burst = NULL;
+	unsigned int n_remaining;
+	const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i];
 
 #ifdef RTE_USE_C11_MEM_MODEL
 	if (__atomic_fetch_add(&lcore_count, 1, __ATOMIC_RELAXED) + 1 != 2)
@@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 		while(lcore_count != 2)
 			rte_pause();
 
-	burst = test_ring_calloc(MAX_BURST, esize);
+	burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size);
 	if (burst == NULL)
 		return -1;
 
 	const uint64_t sp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	const unsigned int bulk_iterations = iterations / bulk_n;
+	for (i = 0; i < bulk_iterations; i++) {
+		n_remaining = bulk_n;
 		do {
 			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_enqueue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_dequeue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			if (ret == 0)
 				rte_pause();
-		} while (!ret);
+			else
+				n_remaining -= ret;
+		} while (n_remaining > 0);
+	}
 	const uint64_t sp_end = rte_rdtsc();
 
-	const uint64_t mp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
-		do {
-			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			if (ret == 0)
-				rte_pause();
-		} while (!ret);
-	const uint64_t mp_end = rte_rdtsc();
+	p->results[p->ring_params->bulk_sizes_i] =
+			((double)sp_end - sp_start) / iterations;
 
-	p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize);
-	p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize);
 	return 0;
 }
 
@@ -243,15 +245,7 @@ enqueue_bulk(void *p)
 {
 	struct thread_params *params = p;
 
-	return enqueue_dequeue_bulk_helper(0, -1, params);
-}
-
-static int
-enqueue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(0, 16, params);
+	return enqueue_dequeue_bulk_helper(0, params);
 }
 
 /*
@@ -263,15 +257,7 @@ dequeue_bulk(void *p)
 {
 	struct thread_params *params = p;
 
-	return enqueue_dequeue_bulk_helper(1, -1, params);
-}
-
-static int
-dequeue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(1, 16, params);
+	return enqueue_dequeue_bulk_helper(1, params);
 }
 
 /*
@@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p)
  * used to measure ring perf between hyperthreads, cores and sockets.
  */
 static int
-run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize)
+run_on_core_pair(struct lcore_pair *cores,
+		struct thread_params *param1, struct thread_params *param2)
 {
-	lcore_function_t *f1, *f2;
-	struct thread_params param1 = {0}, param2 = {0};
 	unsigned i;
-
-	if (esize == -1) {
-		f1 = enqueue_bulk;
-		f2 = dequeue_bulk;
-	} else {
-		f1 = enqueue_bulk_16B;
-		f2 = dequeue_bulk_16B;
-	}
+	struct ring_params *ring_params = param1->ring_params;
 
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		lcore_count = 0;
-		param1.size = param2.size = bulk_sizes[i];
-		param1.r = param2.r = r;
+		ring_params->bulk_sizes_i = i;
 		if (cores->c1 == rte_get_main_lcore()) {
-			rte_eal_remote_launch(f2, &param2, cores->c2);
-			f1(&param1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
+			enqueue_bulk(param1);
 			rte_eal_wait_lcore(cores->c2);
 		} else {
-			rte_eal_remote_launch(f1, &param1, cores->c1);
-			rte_eal_remote_launch(f2, &param2, cores->c2);
+			rte_eal_remote_launch(enqueue_bulk, param1, cores->c1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
 			if (rte_eal_wait_lcore(cores->c1) < 0)
 				return -1;
 			if (rte_eal_wait_lcore(cores->c2) < 0)
 				return -1;
 		}
 		test_ring_print_test_string(
-			TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.spsc + param2.spsc);
-		test_ring_print_test_string(
-			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.mpmc + param2.mpmc);
+				ring_params->ring_flags,
+				ring_params->elem_size,
+				bulk_sizes[i],
+				param1->results[i] + param2->results[i]);
 	}
 
 	return 0;
@@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 	uint64_t hz = rte_get_timer_hz();
 	uint64_t lcount = 0;
 	const unsigned int lcore = rte_lcore_id();
-	struct thread_params *params = p;
+	struct ring_params *ring_params = p->ring_params;
 	void *burst = NULL;
 
 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -346,9 +322,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 
 	begin = rte_get_timer_cycles();
 	while (time_diff < hz * TIME_MS / 1000) {
-		test_ring_enqueue(params->r, burst, esize, params->size,
+		test_ring_enqueue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
-		test_ring_dequeue(params->r, burst, esize, params->size,
+		test_ring_dequeue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 		lcount++;
 		time_diff = rte_get_timer_cycles() - begin;
@@ -380,7 +358,8 @@ static int
 run_on_all_cores(struct rte_ring *r, const int esize)
 {
 	uint64_t total;
-	struct thread_params param;
+	struct ring_params ring_params = {0};
+	struct thread_params params = { .ring_params = &ring_params };
 	lcore_function_t *lcore_f;
 	unsigned int i, c;
 
@@ -389,21 +368,20 @@ run_on_all_cores(struct rte_ring *r, const int esize)
 	else
 		lcore_f = load_loop_fn_16B;
 
-	memset(&param, 0, sizeof(struct thread_params));
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		total = 0;
 		printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]);
-		param.size = bulk_sizes[i];
-		param.r = r;
+		params.ring_params->bulk_sizes_i = i;
+		params.ring_params->r = r;
 
 		/* clear synchro and start workers */
 		__atomic_store_n(&synchro, 0, __ATOMIC_RELAXED);
-		if (rte_eal_mp_remote_launch(lcore_f, &param, SKIP_MAIN) < 0)
+		if (rte_eal_mp_remote_launch(lcore_f, &params, SKIP_MAIN) < 0)
 			return -1;
 
 		/* start synchro and launch test on main */
 		__atomic_store_n(&synchro, 1, __ATOMIC_RELAXED);
-		lcore_f(&param);
+		lcore_f(&params);
 
 		rte_eal_mp_wait_lcore();
 
@@ -462,9 +440,9 @@ static int
 test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	const unsigned int api_type)
 {
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 27;
 	const unsigned int iterations = 1 << iter_shift;
-	unsigned int sz, i = 0;
+	unsigned int sz, i;
 	void **burst = NULL;
 
 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -472,17 +450,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 		return -1;
 
 	for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) {
+		const unsigned int n = iterations / bulk_sizes[sz];
 		const uint64_t start = rte_rdtsc();
-		for (i = 0; i < iterations; i++) {
+		for (i = 0; i < n; i++) {
 			test_ring_enqueue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 			test_ring_dequeue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 		}
 		const uint64_t end = rte_rdtsc();
 
 		test_ring_print_test_string(api_type, esize, bulk_sizes[sz],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 	}
 
 	rte_free(burst);
@@ -490,12 +469,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	return 0;
 }
 
+static __rte_always_inline int
+test_ring_perf_esize_run_on_two_cores(
+		struct thread_params *param1, struct thread_params *param2)
+{
+	struct lcore_pair cores;
+
+	if (get_two_hyperthreads(&cores) == 0) {
+		printf("\n### Testing using two hyperthreads ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing using two physical cores ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_sockets(&cores) == 0) {
+		printf("\n### Testing using two NUMA nodes ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	return 0;
+}
+
 /* Run all tests for a given element size */
 static __rte_always_inline int
 test_ring_perf_esize(const int esize)
 {
-	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
+	double results_enq[RTE_DIM(bulk_sizes)];
+	double results_deq[RTE_DIM(bulk_sizes)];
+	struct ring_params ring_params = {
+			.elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results_enq };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results_deq };
 
 	/*
 	 * Performance test for legacy/_elem APIs
@@ -535,22 +545,13 @@ test_ring_perf_esize(const int esize)
 	test_empty_dequeue(r, esize,
 			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 
-	if (get_two_hyperthreads(&cores) == 0) {
-		printf("\n### Testing using two hyperthreads ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.r = r;
 
-	if (get_two_cores(&cores) == 0) {
-		printf("\n### Testing using two physical cores ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
-	if (get_two_sockets(&cores) == 0) {
-		printf("\n### Testing using two NUMA nodes ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
+
+	ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
 
 	printf("\n### Testing using all worker nodes ###\n");
 	if (run_on_all_cores(r, esize) < 0)
@@ -566,6 +567,79 @@ test_ring_perf_esize(const int esize)
 	return -1;
 }
 
+
+static __rte_always_inline int
+test_ring_perf_compression(void)
+{
+	double results1[RTE_DIM(bulk_sizes)];
+	double results2[RTE_DIM(bulk_sizes)];
+	double results1_comp[RTE_DIM(bulk_sizes)];
+	double results2_comp[RTE_DIM(bulk_sizes)];
+
+	struct lcore_pair cores;
+	int ret = -1;
+	unsigned int i;
+	struct ring_params ring_params = { .elem_size = sizeof(void *) };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results1 };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results2 };
+
+	printf("\n### Testing compression gain ###");
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(void *),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy ###\n");
+		ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME_COMPRESS, sizeof(uint32_t),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	/* swap out the result arrays for the compression ones */
+	param1.results = results1_comp;
+	param2.results = results2_comp;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	printf("\n\n### Potential gain from compression ###\n");
+	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
+		const double result = results1[i] + results2[i];
+		const double result_comp = results1_comp[i] + results1_comp[i];
+		const double gain = 100 - (result_comp / result) * 100;
+
+		printf("Gain of %5.1F%% for burst of %-3u elems\n",
+				gain, bulk_sizes[i]);
+	}
+
+	return ret;
+}
+
 static int
 test_ring_perf(void)
 {
@@ -576,6 +650,10 @@ test_ring_perf(void)
 	if (test_ring_perf_esize(16) == -1)
 		return -1;
 
+	/* Test for performance gain of compression */
+	if (test_ring_perf_compression() == -1)
+		return -1;
+
 	return 0;
 }
 
-- 
2.25.1

[PATCH v3 0/3] add pointer compression API

[Paul Szczepanek]

This patchset is proposing adding a new EAL header with utility functions
that allow compression of arrays of pointers.

When passing caches full of pointers between threads, memory containing
the pointers is copied multiple times which is especially costly between
cores. A compression method will allow us to shrink the memory size
copied.

The compression takes advantage of the fact that pointers are usually
located in a limited memory region (like a mempool). We can compress them
by converting them to offsets from a base memory address.

Offsets can be stored in fewer bytes (dictated by the memory region size
and alignment of the pointer). For example: an 8 byte aligned pointer
which is part of a 32GB memory pool can be stored in 4 bytes. The API is
very generic and does not assume mempool pointers, any pointer can be
passed in.

Compression is based on few and fast operations and especially with vector
instructions leveraged creates minimal overhead.

The API accepts and returns arrays because the overhead means it only is
worth it when done in bulk.

Test is added that shows potential performance gain from compression. In
this test an array of pointers is passed through a ring between two cores.
It shows the gain which is dependent on the bulk operation size. In this
synthetic test run on ampere altra a substantial (up to 25%) performance
gain is seen if done in bulk size larger than 32. At 32 it breaks even and
lower sizes create a small (less than 5%) slowdown due to overhead.

In a more realistic mock application running the l3 forwarding dpdk
example that works in pipeline mode this translated into a ~5% throughput
increase on an ampere altra.

v2:
* addressed review comments (style, explanations and typos)
* lowered bulk iterations closer to original numbers to keep runtime short
* fixed pointer size warning on 32-bit arch
v3:
* added 16 bit versions of compression functions and tests
* added documentation of these new utility functions in the EAL guide

Paul Szczepanek (3):
  eal: add pointer compression functions
  test: add pointer compress tests to ring perf test
  docs: add pointer compression to the EAL guide

 .mailmap                                      |   1 +
 app/test/test_ring.h                          |  94 ++++-
 app/test/test_ring_perf.c                     | 354 ++++++++++++------
 .../prog_guide/env_abstraction_layer.rst      | 142 +++++++
 lib/eal/include/meson.build                   |   1 +
 lib/eal/include/rte_ptr_compress.h            | 272 ++++++++++++++
 6 files changed, 740 insertions(+), 124 deletions(-)
 create mode 100644 lib/eal/include/rte_ptr_compress.h

--
2.25.1

[PATCH v3 1/3] eal: add pointer compression functions

[Paul Szczepanek]

Add a new utility header for compressing pointers. The provided
functions can store pointers in 32-bit offsets.

The compression takes advantage of the fact that pointers are
usually located in a limited memory region (like a mempool).
We can compress them by converting them to offsets from a base
memory address. Offsets can be stored in fewer bytes (dictated
by the memory region size and alignment of the pointer).
For example: an 8 byte aligned pointer which is part of a 32GB
memory pool can be stored in 4 bytes.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .mailmap                           |   1 +
 lib/eal/include/meson.build        |   1 +
 lib/eal/include/rte_ptr_compress.h | 272 +++++++++++++++++++++++++++++
 3 files changed, 274 insertions(+)
 create mode 100644 lib/eal/include/rte_ptr_compress.h

diff --git a/.mailmap b/.mailmap
index 864d33ee46..3f0c9d32f5 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1058,6 +1058,7 @@ Paul Greenwalt <paul.greenwalt@intel.com>
 Paulis Gributs <paulis.gributs@intel.com>
 Paul Luse <paul.e.luse@intel.com>
 Paul M Stillwell Jr <paul.m.stillwell.jr@intel.com>
+Paul Szczepanek <paul.szczepanek@arm.com>
 Pavan Kumar Linga <pavan.kumar.linga@intel.com>
 Pavan Nikhilesh <pbhagavatula@marvell.com> <pbhagavatula@caviumnetworks.com>
 Pavel Belous <pavel.belous@aquantia.com>
diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index a0463efac7..17d8373648 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -36,6 +36,7 @@ headers += files(
         'rte_pci_dev_features.h',
         'rte_per_lcore.h',
         'rte_pflock.h',
+	'rte_ptr_compress.h',
         'rte_random.h',
         'rte_reciprocal.h',
         'rte_seqcount.h',
diff --git a/lib/eal/include/rte_ptr_compress.h b/lib/eal/include/rte_ptr_compress.h
new file mode 100644
index 0000000000..ceb4662c14
--- /dev/null
+++ b/lib/eal/include/rte_ptr_compress.h
@@ -0,0 +1,272 @@
+/* SPDX-License-Identifier: BSD-shift-Clause
+ * Copyright(c) 2023 Arm Limited
+ */
+
+#ifndef RTE_PTR_COMPRESS_H
+#define RTE_PTR_COMPRESS_H
+
+/**
+ * @file
+ * Pointer compression and decompression functions.
+ *
+ * When passing arrays full of pointers between threads, memory containing
+ * the pointers is copied multiple times which is especially costly between
+ * cores. These functions allow us to compress the pointers.
+ *
+ * Compression takes advantage of the fact that pointers are usually located in
+ * a limited memory region (like a mempool). We compress them by converting them
+ * to offsets from a base memory address. Offsets can be stored in fewer bytes.
+ *
+ * The compression functions come in two varieties: 32-bit and 16-bit.
+ *
+ * To determine how many bits are needed to compress the pointer calculate
+ * the biggest offset possible (highest value pointer - base pointer)
+ * and shift the value right according to alignment (shift by exponent of the
+ * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by
+ * 3, etc.). The resulting value must fit in either 32 or 16 bits.
+ *
+ * For usage example and further explanation please see "Pointer Compression" in
+ * doc/guides/prog_guide/env_abstraction_layer.rst
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_vect.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Compress pointers into 32-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 32 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 32GB of the base pointer. If there is no such alignment guarantee they
+ * must be within 4GB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_32(void *ptr_base, void **src_table,
+		uint32_t *dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_src_table;
+	svuint64_t v_dest_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_src_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_dest_table = svsub_x(pg, v_src_table, (uint64_t)ptr_base);
+		v_dest_table = svlsr_x(pg, v_dest_table, bit_shift);
+		svst1w(pg, &dest_table[i], v_dest_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON
+	uint64_t ptr_diff;
+	uint64x2_t v_src_table;
+	uint64x2_t v_dest_table;
+	/* right shift is done by left shifting by negative int */
+	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_src_table = vld1q_u64((const uint64_t *)src_table + i);
+		v_dest_table = vsubq_u64(v_src_table, v_ptr_base);
+		v_dest_table = vshlq_u64(v_dest_table, v_shift);
+		vst1_u32(dest_table + i, vqmovn_u64(v_dest_table));
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT32_MAX);
+		dest_table[i] = (uint32_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 32-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_32(void *ptr_base, uint32_t *src_table,
+		void **dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_src_table;
+	svuint64_t v_dest_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_src_table = svld1uw_u64(pg, &src_table[i]);
+		v_src_table = svlsl_x(pg, v_src_table, bit_shift);
+		v_dest_table = svadd_x(pg, v_src_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_dest_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON
+	uint64_t ptr_diff;
+	uint64x2_t v_src_table;
+	uint64x2_t v_dest_table;
+	int64x2_t v_shift = vdupq_n_s64(bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_src_table = vmovl_u32(vld1_u32(src_table + i));
+		v_src_table = vshlq_u64(v_dest_table, v_shift);
+		v_dest_table = vaddq_u64(v_src_table, v_ptr_base);
+		vst1q_u64((uint64_t *)dest_table + i, v_dest_table);
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+/**
+ * Compress pointers into 16-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 16 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 256KB of the base pointer. If there is no such alignment guarantee
+ * they must be within 64KB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_16(void *ptr_base, void **src_table,
+		uint16_t *dest_table, unsigned int n, unsigned int bit_shift)
+{
+
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_src_table;
+	svuint64_t v_dest_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_src_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_dest_table = svsub_x(pg, v_src_table, (uint64_t)ptr_base);
+		v_dest_table = svlsr_x(pg, v_dest_table, bit_shift);
+		svst1h(pg, &dest_table[i], v_dest_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT16_MAX);
+		dest_table[i] = (uint16_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 16-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_16(void *ptr_base, uint16_t *src_table,
+		void **dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_src_table;
+	svuint64_t v_dest_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_src_table = svld1uh_u64(pg, &src_table[i]);
+		v_src_table = svlsl_x(pg, v_src_table, bit_shift);
+		v_dest_table = svadd_x(pg, v_src_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_dest_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_PTR_COMPRESS_H */
--
2.25.1

[PATCH v3 2/3] test: add pointer compress tests to ring perf test

[Paul Szczepanek]

Add a test that runs a zero copy burst enqueue and dequeue on a ring
of raw pointers and compressed pointers at different burst sizes to
showcase performance benefits of newly added pointer compression APIs.

Refactored threading code to pass more parameters to threads to
reuse existing code. Added more bulk sizes to showcase their effects
on compression. Adjusted loop iteration numbers to take into account
bulk sizes to keep runtime constant (instead of number of operations).

Adjusted old printfs to match new ones which have aligned numbers.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 app/test/test_ring.h      |  94 +++++++++-
 app/test/test_ring_perf.c | 354 +++++++++++++++++++++++++-------------
 2 files changed, 324 insertions(+), 124 deletions(-)

diff --git a/app/test/test_ring.h b/app/test/test_ring.h
index 45c263f3ff..3b00f2465d 100644
--- a/app/test/test_ring.h
+++ b/app/test/test_ring.h
@@ -1,10 +1,12 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */

 #include <rte_malloc.h>
 #include <rte_ring.h>
 #include <rte_ring_elem.h>
+#include <rte_memcpy.h>
+#include <rte_ptr_compress.h>

 /* API type to call
  * rte_ring_<sp/mp or sc/mc>_enqueue_<bulk/burst>
@@ -25,6 +27,10 @@
 #define TEST_RING_ELEM_BULK 16
 #define TEST_RING_ELEM_BURST 32

+#define TEST_RING_ELEM_BURST_ZC 64
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 128
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 256
+
 #define TEST_RING_IGNORE_API_TYPE ~0U

 /* This function is placed here as it is required for both
@@ -101,6 +107,9 @@ static inline unsigned int
 test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -152,6 +161,46 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n,
 								NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy(zcd.ptr2,
+						(char *)obj + zcd.n1 * esize,
+						(ret - zcd.n1) * esize);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_16(0, obj, zcd.ptr1, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_16(0,
+						obj + (zcd.n1 * 2),
+						zcd.ptr2,
+						(ret - zcd.n1) * 2, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_32(0, obj, zcd.ptr1, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_32(0, obj + zcd.n1,
+						zcd.ptr2, ret - zcd.n1, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
@@ -162,6 +211,9 @@ static inline unsigned int
 test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -213,6 +265,46 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mc_dequeue_burst_elem(r, obj, esize,
 								n, NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy((char *)obj + zcd.n1 * esize,
+						zcd.ptr2,
+						(ret - zcd.n1) * esize);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_16(0, zcd.ptr1, obj, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_16(0, zcd.ptr2,
+						obj + zcd.n1,
+						(ret - zcd.n1) * 2,
+						3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_32(0, zcd.ptr1, obj, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_32(0, zcd.ptr2,
+						obj + zcd.n1, ret - zcd.n1, 3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c
index d7c5a4c30b..8392d0c781 100644
--- a/app/test/test_ring_perf.c
+++ b/app/test/test_ring_perf.c
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2010-2014 Intel Corporation
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */


@@ -22,13 +22,13 @@

 #define RING_NAME "RING_PERF"
 #define RING_SIZE 4096
-#define MAX_BURST 32
+#define MAX_BURST 256

 /*
  * the sizes to enqueue and dequeue in testing
  * (marked volatile so they won't be seen as compile-time constants)
  */
-static const volatile unsigned bulk_sizes[] = { 8, 32 };
+static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128, 256 };

 struct lcore_pair {
 	unsigned c1, c2;
@@ -43,26 +43,30 @@ test_ring_print_test_string(unsigned int api_type, int esize,
 	if (esize == -1)
 		printf("legacy APIs");
 	else
-		printf("elem APIs: element size %dB", esize);
+		printf("elem APIs (size:%2dB)", esize);

 	if (api_type == TEST_RING_IGNORE_API_TYPE)
 		return;

 	if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF)
-		printf(": default enqueue/dequeue: ");
+		printf(" - default enqueue/dequeue");
 	else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC)
-		printf(": SP/SC: ");
+		printf(" - SP/SC");
 	else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC)
-		printf(": MP/MC: ");
+		printf(" - MP/MC");

 	if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE)
-		printf("single: ");
+		printf(" - single - ");
 	else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK)
-		printf("bulk (size: %u): ", bsz);
+		printf(" - bulk (n:%-3u) - ", bsz);
 	else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST)
-		printf("burst (size: %u): ", bsz);
+		printf(" - burst (n:%-3u) - ", bsz);
+	else if ((api_type & (TEST_RING_ELEM_BURST_ZC |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0)
+		printf(" - burst zero copy (n:%-3u) - ", bsz);

-	printf("%.2F\n", value);
+	printf("cycles per elem: %.3F\n", value);
 }

 /**** Functions to analyse our core mask to get cores for different tests ***/
@@ -144,28 +148,33 @@ static void
 test_empty_dequeue(struct rte_ring *r, const int esize,
 			const unsigned int api_type)
 {
-	const unsigned int iter_shift = 26;
+	const unsigned int iter_shift = 29;
 	const unsigned int iterations = 1 << iter_shift;
 	unsigned int i = 0;
 	void *burst[MAX_BURST];

+	const unsigned int bulk_iterations = iterations / bulk_sizes[0];
 	const uint64_t start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	for (i = 0; i < bulk_iterations; i++)
 		test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type);
 	const uint64_t end = rte_rdtsc();

 	test_ring_print_test_string(api_type, esize, bulk_sizes[0],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 }

-/*
- * for the separate enqueue and dequeue threads they take in one param
- * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc
- */
-struct thread_params {
+/* describes the ring used by the enqueue and dequeue thread */
+struct ring_params {
 	struct rte_ring *r;
-	unsigned size;        /* input value, the burst size */
-	double spsc, mpmc;    /* output value, the single or multi timings */
+	unsigned int elem_size;
+	unsigned int bulk_sizes_i; /* index into bulk_size array */
+	unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */
+};
+
+/* Used do specify enqueue and dequeue ring operations and their results */
+struct thread_params {
+	struct ring_params *ring_params;
+	double *results; /* result array size must be equal to bulk_sizes */
 };

 /*
@@ -174,16 +183,15 @@ struct thread_params {
  * flag == 1 -> dequeue
  */
 static __rte_always_inline int
-enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
-	struct thread_params *p)
+enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 15;
+	const unsigned int iter_shift = 22;
 	const unsigned int iterations = 1 << iter_shift;
-	struct rte_ring *r = p->r;
-	unsigned int bsize = p->size;
 	unsigned int i;
 	void *burst = NULL;
+	unsigned int n_remaining;
+	const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i];

 #ifdef RTE_USE_C11_MEM_MODEL
 	if (__atomic_fetch_add(&lcore_count, 1, __ATOMIC_RELAXED) + 1 != 2)
@@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 		while(lcore_count != 2)
 			rte_pause();

-	burst = test_ring_calloc(MAX_BURST, esize);
+	burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size);
 	if (burst == NULL)
 		return -1;

 	const uint64_t sp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	const unsigned int bulk_iterations = iterations / bulk_n;
+	for (i = 0; i < bulk_iterations; i++) {
+		n_remaining = bulk_n;
 		do {
 			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_enqueue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_dequeue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			if (ret == 0)
 				rte_pause();
-		} while (!ret);
+			else
+				n_remaining -= ret;
+		} while (n_remaining > 0);
+	}
 	const uint64_t sp_end = rte_rdtsc();

-	const uint64_t mp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
-		do {
-			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			if (ret == 0)
-				rte_pause();
-		} while (!ret);
-	const uint64_t mp_end = rte_rdtsc();
+	p->results[p->ring_params->bulk_sizes_i] =
+			((double)sp_end - sp_start) / iterations;

-	p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize);
-	p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize);
 	return 0;
 }

@@ -243,15 +245,7 @@ enqueue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(0, -1, params);
-}
-
-static int
-enqueue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(0, 16, params);
+	return enqueue_dequeue_bulk_helper(0, params);
 }

 /*
@@ -263,15 +257,7 @@ dequeue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(1, -1, params);
-}
-
-static int
-dequeue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(1, 16, params);
+	return enqueue_dequeue_bulk_helper(1, params);
 }

 /*
@@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p)
  * used to measure ring perf between hyperthreads, cores and sockets.
  */
 static int
-run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize)
+run_on_core_pair(struct lcore_pair *cores,
+		struct thread_params *param1, struct thread_params *param2)
 {
-	lcore_function_t *f1, *f2;
-	struct thread_params param1 = {0}, param2 = {0};
 	unsigned i;
-
-	if (esize == -1) {
-		f1 = enqueue_bulk;
-		f2 = dequeue_bulk;
-	} else {
-		f1 = enqueue_bulk_16B;
-		f2 = dequeue_bulk_16B;
-	}
+	struct ring_params *ring_params = param1->ring_params;

 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		lcore_count = 0;
-		param1.size = param2.size = bulk_sizes[i];
-		param1.r = param2.r = r;
+		ring_params->bulk_sizes_i = i;
 		if (cores->c1 == rte_get_main_lcore()) {
-			rte_eal_remote_launch(f2, &param2, cores->c2);
-			f1(&param1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
+			enqueue_bulk(param1);
 			rte_eal_wait_lcore(cores->c2);
 		} else {
-			rte_eal_remote_launch(f1, &param1, cores->c1);
-			rte_eal_remote_launch(f2, &param2, cores->c2);
+			rte_eal_remote_launch(enqueue_bulk, param1, cores->c1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
 			if (rte_eal_wait_lcore(cores->c1) < 0)
 				return -1;
 			if (rte_eal_wait_lcore(cores->c2) < 0)
 				return -1;
 		}
 		test_ring_print_test_string(
-			TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.spsc + param2.spsc);
-		test_ring_print_test_string(
-			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.mpmc + param2.mpmc);
+				ring_params->ring_flags,
+				ring_params->elem_size,
+				bulk_sizes[i],
+				param1->results[i] + param2->results[i]);
 	}

 	return 0;
@@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 	uint64_t hz = rte_get_timer_hz();
 	uint64_t lcount = 0;
 	const unsigned int lcore = rte_lcore_id();
-	struct thread_params *params = p;
+	struct ring_params *ring_params = p->ring_params;
 	void *burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -346,9 +322,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize)

 	begin = rte_get_timer_cycles();
 	while (time_diff < hz * TIME_MS / 1000) {
-		test_ring_enqueue(params->r, burst, esize, params->size,
+		test_ring_enqueue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
-		test_ring_dequeue(params->r, burst, esize, params->size,
+		test_ring_dequeue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 		lcount++;
 		time_diff = rte_get_timer_cycles() - begin;
@@ -380,7 +358,8 @@ static int
 run_on_all_cores(struct rte_ring *r, const int esize)
 {
 	uint64_t total;
-	struct thread_params param;
+	struct ring_params ring_params = {0};
+	struct thread_params params = { .ring_params = &ring_params };
 	lcore_function_t *lcore_f;
 	unsigned int i, c;

@@ -389,21 +368,20 @@ run_on_all_cores(struct rte_ring *r, const int esize)
 	else
 		lcore_f = load_loop_fn_16B;

-	memset(&param, 0, sizeof(struct thread_params));
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		total = 0;
 		printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]);
-		param.size = bulk_sizes[i];
-		param.r = r;
+		params.ring_params->bulk_sizes_i = i;
+		params.ring_params->r = r;

 		/* clear synchro and start workers */
 		__atomic_store_n(&synchro, 0, __ATOMIC_RELAXED);
-		if (rte_eal_mp_remote_launch(lcore_f, &param, SKIP_MAIN) < 0)
+		if (rte_eal_mp_remote_launch(lcore_f, &params, SKIP_MAIN) < 0)
 			return -1;

 		/* start synchro and launch test on main */
 		__atomic_store_n(&synchro, 1, __ATOMIC_RELAXED);
-		lcore_f(&param);
+		lcore_f(&params);

 		rte_eal_mp_wait_lcore();

@@ -462,9 +440,9 @@ static int
 test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	const unsigned int api_type)
 {
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 26;
 	const unsigned int iterations = 1 << iter_shift;
-	unsigned int sz, i = 0;
+	unsigned int sz, i;
 	void **burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -472,17 +450,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 		return -1;

 	for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) {
+		const unsigned int n = iterations / bulk_sizes[sz];
 		const uint64_t start = rte_rdtsc();
-		for (i = 0; i < iterations; i++) {
+		for (i = 0; i < n; i++) {
 			test_ring_enqueue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 			test_ring_dequeue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 		}
 		const uint64_t end = rte_rdtsc();

 		test_ring_print_test_string(api_type, esize, bulk_sizes[sz],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 	}

 	rte_free(burst);
@@ -490,12 +469,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	return 0;
 }

+static __rte_always_inline int
+test_ring_perf_esize_run_on_two_cores(
+		struct thread_params *param1, struct thread_params *param2)
+{
+	struct lcore_pair cores;
+
+	if (get_two_hyperthreads(&cores) == 0) {
+		printf("\n### Testing using two hyperthreads ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing using two physical cores ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_sockets(&cores) == 0) {
+		printf("\n### Testing using two NUMA nodes ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	return 0;
+}
+
 /* Run all tests for a given element size */
 static __rte_always_inline int
 test_ring_perf_esize(const int esize)
 {
-	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
+	double results_enq[RTE_DIM(bulk_sizes)];
+	double results_deq[RTE_DIM(bulk_sizes)];
+	struct ring_params ring_params = {
+			.elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results_enq };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results_deq };

 	/*
 	 * Performance test for legacy/_elem APIs
@@ -535,22 +545,13 @@ test_ring_perf_esize(const int esize)
 	test_empty_dequeue(r, esize,
 			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);

-	if (get_two_hyperthreads(&cores) == 0) {
-		printf("\n### Testing using two hyperthreads ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.r = r;

-	if (get_two_cores(&cores) == 0) {
-		printf("\n### Testing using two physical cores ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
-	if (get_two_sockets(&cores) == 0) {
-		printf("\n### Testing using two NUMA nodes ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
+
+	ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);

 	printf("\n### Testing using all worker nodes ###\n");
 	if (run_on_all_cores(r, esize) < 0)
@@ -566,6 +567,109 @@ test_ring_perf_esize(const int esize)
 	return -1;
 }

+
+static __rte_always_inline int
+test_ring_perf_compression(void)
+{
+	double results1[RTE_DIM(bulk_sizes)];
+	double results2[RTE_DIM(bulk_sizes)];
+	double results1_comp[2][RTE_DIM(bulk_sizes)];
+	double results2_comp[2][RTE_DIM(bulk_sizes)];
+
+	struct lcore_pair cores;
+	int ret = -1;
+	unsigned int i, j;
+	struct ring_params ring_params = { .elem_size = sizeof(void *) };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results1 };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results2 };
+
+	printf("\n### Testing compression gain ###");
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(void *),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy ###\n");
+		ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	/* rings allow only multiples of 4 as sizes,
+	 * we allocate size 4 despite only using 2 bytes
+	 * and use half of RING_SIZE as the number of elements
+	 */
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE / 2, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[0];
+	param2.results = results2_comp[0];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (16b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[1];
+	param2.results = results2_comp[1];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (32b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	for (j = 0; j < 2; j++) {
+		printf("\n### Potential gain from compression (%d-bit offsets) "
+		"###\n", (j + 1) * 16);
+		for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
+			const double result = results1[i] + results2[i];
+			const double result_comp = results1_comp[j][i] +
+				results2_comp[j][i];
+			const double gain = 100 - (result_comp / result) * 100;
+
+			printf("Gain of %5.1F%% for burst of %-3u elems\n",
+					gain, bulk_sizes[i]);
+		}
+	}
+
+	return ret;
+}
+
 static int
 test_ring_perf(void)
 {
@@ -576,6 +680,10 @@ test_ring_perf(void)
 	if (test_ring_perf_esize(16) == -1)
 		return -1;

+	/* Test for performance gain of compression */
+	if (test_ring_perf_compression() == -1)
+		return -1;
+
 	return 0;
 }

--
2.25.1

[PATCH v3 3/3] docs: add pointer compression to the EAL guide

[Paul Szczepanek]

Documentation added in the EAL guide for the new
utility functions for pointer compression
showing example code and potential usecases

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .../prog_guide/env_abstraction_layer.rst      | 142 ++++++++++++++++++
 1 file changed, 142 insertions(+)

diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 89014789de..88cf1f16e2 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -1192,3 +1192,145 @@ will not be deallocated.

 Any successful deallocation event will trigger a callback, for which user
 applications and other DPDK subsystems can register.
+
+.. _pointer_compression:
+
+Pointer Compression
+-------------------
+
+Use ``rte_ptr_compress_16()`` and ``rte_ptr_decompress_16()`` to compress and
+decompress pointers into 16-bit offsets. Use ``rte_ptr_compress_32()`` and
+``rte_ptr_decompress_32()`` to compress and decompress pointers into 32-bit
+offsets.
+
+Compression takes advantage of the fact that pointers are usually located in a
+limited memory region (like a mempool). By converting them to offsets from a
+base memory address they can be stored in fewer bytes. How many bytes are needed
+to store the offset is dictated by the memory region size and alignment of
+objects the pointers point to.
+
+For example, a pointer which is part of a 4GB memory pool can be stored as 32
+bit offset. If the pointer points to memory that is 8 bytes aligned then 3 bits
+can be dropped from the offset and a 32GB memory pool can now fit in 32 bits.
+
+For performance reasons these requirements are not enforced programmatically.
+The programmer is responsible for ensuring that the combination of distance
+from the base pointer and memory alignment allow for storing of the offset in
+the number of bits indicated by the function name (16 or 32). Start of mempool
+memory would be a good candidate for the base pointer. Otherwise any pointer
+that precedes all pointers, is close enough and has the same alignment as the
+pointers being compressed will work.
+
+.. note::
+
+    Performance gains depend on the batch size of pointers and CPU capabilities
+    such as vector extensions. It's important to measure the performance
+    increase on target hardware. A test called ``ring_perf_autotest`` in
+    ``dpdk-test`` can provide the measurements.
+
+Example usage
+~~~~~~~~~~~~~
+
+In this example we send pointers between two cores through a ring. While this
+is a realistic use case the code is simplified for demonstration purposes and
+does not have error handling.
+
+.. code-block:: c
+
+    #include <rte_launch.h>
+    #include <rte_ring.h>
+    #include <rte_ring_elem.h>
+    #include <rte_ptr_compress.h>
+
+    #define ITEMS_ARRAY_SIZE (1024)
+    #define BATCH_SIZE (128)
+    #define ALIGN_EXPONENT (3)
+    #define ITEM_ALIGN (1<<ALIGN_EXPONENT)
+    #define CORE_SEND (1)
+    #define CORE_RECV (2)
+
+    struct item {
+      int a;
+    } __rte_aligned(ITEM_ALIGN);
+
+    static struct item items[ITEMS_ARRAY_SIZE] = {0};
+    static struct rte_ring *ring = NULL;
+
+    static int
+    send_compressed(void *args)
+    {
+      struct item *ptrs_send[BATCH_SIZE] = {0};
+      unsigned int n_send = 0;
+      struct rte_ring_zc_data zcd = {0};
+
+      /* in this toy example we only fill the ptrs_send once and re-use */
+      for (;n_send < BATCH_SIZE; n_send++)
+        ptrs_send[n_send] = &items[n_send];
+
+      for(;;) {
+        n_send = rte_ring_enqueue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        /* compress ptrs_send into offsets */
+        rte_ptr_compress_32(items, /* base pointer */
+          ptrs_send, /* source array to be compressed */
+          zcd.ptr1, /* destination array to store offsets */
+          zcd.n1, /* how many pointers to compress */
+          ALIGN_EXPONENT /* how many bits can we drop from the offset */);
+
+        if (zcd.ptr2 != NULL)
+          rte_ptr_compress_32(items, ptrs_send + zcd.n1,
+            zcd.ptr2, n_send - zcd.n1, ALIGN_EXPONENT);
+
+        rte_ring_enqueue_zc_finish(ring, n_send);
+      }
+      return 1;
+    }
+
+    static int
+    recv_compressed(void *args)
+    {
+      struct item *ptrs_recv[BATCH_SIZE] = {0};
+      unsigned int n_recv;
+      struct rte_ring_zc_data zcd = {0};
+
+      for(;;) {
+        /* receive compressed pointers from the ring */
+        n_recv = rte_ring_dequeue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        rte_ptr_decompress_32(items, /* base pointer */
+          zcd.ptr1, /* source array to decompress */
+          ptrs_recv, /* destination array to store pointers */
+          zcd.n1, /* how many pointers to decompress */
+          ALIGN_EXPONENT /* how many bits were dropped from the offset */);
+
+        /* handle the potential secondary buffer (caused by ring boundary) */
+        if (zcd.ptr2 != NULL)
+          rte_ptr_decompress_32(items,
+            zcd.ptr2,
+            ptrs_recv + zcd.n1,
+            n_recv - zcd.n1,
+            ALIGN_EXPONENT);
+
+        rte_ring_dequeue_zc_finish(ring, n_recv);
+
+        /* ptrs_recv contains what ptrs_send contained in the other thread */
+        /* (...) */
+      }
+      return 1;
+    }
+
+    void
+    compression_example(void)
+    {
+      ring = rte_ring_create_elem(
+        "COMPR_PTRS", sizeof(uint32_t),
+        1024, rte_socket_id(),
+        RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+      rte_eal_remote_launch(send_compressed, NULL, CORE_SEND);
+      rte_eal_remote_launch(recv_compressed, NULL, CORE_RECV);
+
+      for(;;) {}
+    }
--
2.25.1

RE: [PATCH v3 0/3] add pointer compression API

[Morten Brørup]

> From: Paul Szczepanek [mailto:paul.szczepanek@arm.com]
> Sent: Tuesday, 31 October 2023 19.11

[...]

> Test is added that shows potential performance gain from compression.
> In
> this test an array of pointers is passed through a ring between two
> cores.
> It shows the gain which is dependent on the bulk operation size. In
> this
> synthetic test run on ampere altra a substantial (up to 25%)
> performance
> gain is seen if done in bulk size larger than 32. At 32 it breaks even
> and
> lower sizes create a small (less than 5%) slowdown due to overhead.
> 
> In a more realistic mock application running the l3 forwarding dpdk
> example that works in pipeline mode this translated into a ~5%
> throughput
> increase on an ampere altra.

What was the bulk size in this test?

And were the pipeline stages running on the same lcore or individual lcores per pipeline stage?

Re: [PATCH v3 0/3] add pointer compression API

[Paul Szczepanek]

On 01/11/2023 07:42, Morten Brørup wrote:
>> From: Paul Szczepanek [mailto:paul.szczepanek@arm.com]
>> Sent: Tuesday, 31 October 2023 19.11
> [...]
>
>> In a more realistic mock application running the l3 forwarding dpdk
>> example that works in pipeline mode this translated into a ~5%
>> throughput
>> increase on an ampere altra.
> What was the bulk size in this test?
>
> And were the pipeline stages running on the same lcore or individual lcores per pipeline stage?
>

The pipeline mode was run on separate cores and used 128 as the bulk size.

[PATCH v4 0/4] add pointer compression API

[Paul Szczepanek]

This patchset is proposing adding a new EAL header with utility functions
that allow compression of arrays of pointers.

When passing caches full of pointers between threads, memory containing
the pointers is copied multiple times which is especially costly between
cores. A compression method will allow us to shrink the memory size
copied.

The compression takes advantage of the fact that pointers are usually
located in a limited memory region (like a mempool). We can compress them
by converting them to offsets from a base memory address.

Offsets can be stored in fewer bytes (dictated by the memory region size
and alignment of the pointer). For example: an 8 byte aligned pointer
which is part of a 32GB memory pool can be stored in 4 bytes. The API is
very generic and does not assume mempool pointers, any pointer can be
passed in.

Compression is based on few and fast operations and especially with vector
instructions leveraged creates minimal overhead.

The API accepts and returns arrays because the overhead means it only is
worth it when done in bulk.

Test is added that shows potential performance gain from compression. In
this test an array of pointers is passed through a ring between two cores.
It shows the gain which is dependent on the bulk operation size. In this
synthetic test run on ampere altra a substantial (up to 25%) performance
gain is seen if done in bulk size larger than 32. At 32 it breaks even and
lower sizes create a small (less than 5%) slowdown due to overhead.

In a more realistic mock application running the l3 forwarding dpdk
example that works in pipeline mode on two cores this translated into a
~5% throughput increase on an ampere altra.

v2:
* addressed review comments (style, explanations and typos)
* lowered bulk iterations closer to original numbers to keep runtime short
* fixed pointer size warning on 32-bit arch
v3:
* added 16-bit versions of compression functions and tests
* added documentation of these new utility functions in the EAL guide
v4:
* added unit test
* fix bug in NEON implementation of 32-bit decompress

Paul Szczepanek (4):
  eal: add pointer compression functions
  test: add pointer compress tests to ring perf test
  docs: add pointer compression to the EAL guide
  test: add unit test for ptr compression

 .mailmap                                      |   1 +
 app/test/meson.build                          |   1 +
 app/test/test_eal_ptr_compress.c              | 108 ++++++
 app/test/test_ring.h                          |  94 ++++-
 app/test/test_ring_perf.c                     | 354 ++++++++++++------
 .../prog_guide/env_abstraction_layer.rst      | 142 +++++++
 lib/eal/include/meson.build                   |   1 +
 lib/eal/include/rte_ptr_compress.h            | 266 +++++++++++++
 8 files changed, 843 insertions(+), 124 deletions(-)
 create mode 100644 app/test/test_eal_ptr_compress.c
 create mode 100644 lib/eal/include/rte_ptr_compress.h

--
2.25.1

[PATCH v4 1/4] eal: add pointer compression functions

[Paul Szczepanek]

Add a new utility header for compressing pointers. The provided
functions can store pointers in 32-bit offsets.

The compression takes advantage of the fact that pointers are
usually located in a limited memory region (like a mempool).
We can compress them by converting them to offsets from a base
memory address. Offsets can be stored in fewer bytes (dictated
by the memory region size and alignment of the pointer).
For example: an 8 byte aligned pointer which is part of a 32GB
memory pool can be stored in 4 bytes.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .mailmap                           |   1 +
 lib/eal/include/meson.build        |   1 +
 lib/eal/include/rte_ptr_compress.h | 266 +++++++++++++++++++++++++++++
 3 files changed, 268 insertions(+)
 create mode 100644 lib/eal/include/rte_ptr_compress.h

diff --git a/.mailmap b/.mailmap
index 864d33ee46..3f0c9d32f5 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1058,6 +1058,7 @@ Paul Greenwalt <paul.greenwalt@intel.com>
 Paulis Gributs <paulis.gributs@intel.com>
 Paul Luse <paul.e.luse@intel.com>
 Paul M Stillwell Jr <paul.m.stillwell.jr@intel.com>
+Paul Szczepanek <paul.szczepanek@arm.com>
 Pavan Kumar Linga <pavan.kumar.linga@intel.com>
 Pavan Nikhilesh <pbhagavatula@marvell.com> <pbhagavatula@caviumnetworks.com>
 Pavel Belous <pavel.belous@aquantia.com>
diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index a0463efac7..17d8373648 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -36,6 +36,7 @@ headers += files(
         'rte_pci_dev_features.h',
         'rte_per_lcore.h',
         'rte_pflock.h',
+	'rte_ptr_compress.h',
         'rte_random.h',
         'rte_reciprocal.h',
         'rte_seqcount.h',
diff --git a/lib/eal/include/rte_ptr_compress.h b/lib/eal/include/rte_ptr_compress.h
new file mode 100644
index 0000000000..6697385113
--- /dev/null
+++ b/lib/eal/include/rte_ptr_compress.h
@@ -0,0 +1,266 @@
+/* SPDX-License-Identifier: BSD-shift-Clause
+ * Copyright(c) 2023 Arm Limited
+ */
+
+#ifndef RTE_PTR_COMPRESS_H
+#define RTE_PTR_COMPRESS_H
+
+/**
+ * @file
+ * Pointer compression and decompression functions.
+ *
+ * When passing arrays full of pointers between threads, memory containing
+ * the pointers is copied multiple times which is especially costly between
+ * cores. These functions allow us to compress the pointers.
+ *
+ * Compression takes advantage of the fact that pointers are usually located in
+ * a limited memory region (like a mempool). We compress them by converting them
+ * to offsets from a base memory address. Offsets can be stored in fewer bytes.
+ *
+ * The compression functions come in two varieties: 32-bit and 16-bit.
+ *
+ * To determine how many bits are needed to compress the pointer calculate
+ * the biggest offset possible (highest value pointer - base pointer)
+ * and shift the value right according to alignment (shift by exponent of the
+ * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by
+ * 3, etc.). The resulting value must fit in either 32 or 16 bits.
+ *
+ * For usage example and further explanation please see "Pointer Compression" in
+ * doc/guides/prog_guide/env_abstraction_layer.rst
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_vect.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Compress pointers into 32-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 32 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 32GB of the base pointer. If there is no such alignment guarantee they
+ * must be within 4GB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_32(void *ptr_base, void **src_table,
+		uint32_t *dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1w(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	/* right shift is done by left shifting by negative int */
+	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_ptr_table = vld1q_u64((const uint64_t *)src_table + i);
+		v_ptr_table = vsubq_u64(v_ptr_table, v_ptr_base);
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		vst1_u32(dest_table + i, vqmovn_u64(v_ptr_table));
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT32_MAX);
+		dest_table[i] = (uint32_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 32-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_32(void *ptr_base, uint32_t *src_table,
+		void **dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1uw_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	int64x2_t v_shift = vdupq_n_s64(bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_ptr_table = vmovl_u32(vld1_u32(src_table + i));
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		v_ptr_table = vaddq_u64(v_ptr_table, v_ptr_base);
+		vst1q_u64((uint64_t *)dest_table + i, v_ptr_table);
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+/**
+ * Compress pointers into 16-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 16 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 256KB of the base pointer. If there is no such alignment guarantee
+ * they must be within 64KB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_16(void *ptr_base, void **src_table,
+		uint16_t *dest_table, unsigned int n, unsigned int bit_shift)
+{
+
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1h(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT16_MAX);
+		dest_table[i] = (uint16_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 16-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_16(void *ptr_base, uint16_t *src_table,
+		void **dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1uh_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_PTR_COMPRESS_H */
--
2.25.1

[PATCH v4 2/4] test: add pointer compress tests to ring perf test

[Paul Szczepanek]

Add a test that runs a zero copy burst enqueue and dequeue on a ring
of raw pointers and compressed pointers at different burst sizes to
showcase performance benefits of newly added pointer compression APIs.

Refactored threading code to pass more parameters to threads to
reuse existing code. Added more bulk sizes to showcase their effects
on compression. Adjusted loop iteration numbers to take into account
bulk sizes to keep runtime constant (instead of number of operations).

Adjusted old printfs to match new ones which have aligned numbers.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 app/test/test_ring.h      |  94 +++++++++-
 app/test/test_ring_perf.c | 354 +++++++++++++++++++++++++-------------
 2 files changed, 324 insertions(+), 124 deletions(-)

diff --git a/app/test/test_ring.h b/app/test/test_ring.h
index 45c263f3ff..3b00f2465d 100644
--- a/app/test/test_ring.h
+++ b/app/test/test_ring.h
@@ -1,10 +1,12 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */

 #include <rte_malloc.h>
 #include <rte_ring.h>
 #include <rte_ring_elem.h>
+#include <rte_memcpy.h>
+#include <rte_ptr_compress.h>

 /* API type to call
  * rte_ring_<sp/mp or sc/mc>_enqueue_<bulk/burst>
@@ -25,6 +27,10 @@
 #define TEST_RING_ELEM_BULK 16
 #define TEST_RING_ELEM_BURST 32

+#define TEST_RING_ELEM_BURST_ZC 64
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 128
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 256
+
 #define TEST_RING_IGNORE_API_TYPE ~0U

 /* This function is placed here as it is required for both
@@ -101,6 +107,9 @@ static inline unsigned int
 test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -152,6 +161,46 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n,
 								NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy(zcd.ptr2,
+						(char *)obj + zcd.n1 * esize,
+						(ret - zcd.n1) * esize);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_16(0, obj, zcd.ptr1, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_16(0,
+						obj + (zcd.n1 * 2),
+						zcd.ptr2,
+						(ret - zcd.n1) * 2, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_32(0, obj, zcd.ptr1, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_32(0, obj + zcd.n1,
+						zcd.ptr2, ret - zcd.n1, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
@@ -162,6 +211,9 @@ static inline unsigned int
 test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -213,6 +265,46 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mc_dequeue_burst_elem(r, obj, esize,
 								n, NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy((char *)obj + zcd.n1 * esize,
+						zcd.ptr2,
+						(ret - zcd.n1) * esize);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_16(0, zcd.ptr1, obj, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_16(0, zcd.ptr2,
+						obj + zcd.n1,
+						(ret - zcd.n1) * 2,
+						3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_32(0, zcd.ptr1, obj, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_32(0, zcd.ptr2,
+						obj + zcd.n1, ret - zcd.n1, 3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c
index d7c5a4c30b..8392d0c781 100644
--- a/app/test/test_ring_perf.c
+++ b/app/test/test_ring_perf.c
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2010-2014 Intel Corporation
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */


@@ -22,13 +22,13 @@

 #define RING_NAME "RING_PERF"
 #define RING_SIZE 4096
-#define MAX_BURST 32
+#define MAX_BURST 256

 /*
  * the sizes to enqueue and dequeue in testing
  * (marked volatile so they won't be seen as compile-time constants)
  */
-static const volatile unsigned bulk_sizes[] = { 8, 32 };
+static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128, 256 };

 struct lcore_pair {
 	unsigned c1, c2;
@@ -43,26 +43,30 @@ test_ring_print_test_string(unsigned int api_type, int esize,
 	if (esize == -1)
 		printf("legacy APIs");
 	else
-		printf("elem APIs: element size %dB", esize);
+		printf("elem APIs (size:%2dB)", esize);

 	if (api_type == TEST_RING_IGNORE_API_TYPE)
 		return;

 	if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF)
-		printf(": default enqueue/dequeue: ");
+		printf(" - default enqueue/dequeue");
 	else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC)
-		printf(": SP/SC: ");
+		printf(" - SP/SC");
 	else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC)
-		printf(": MP/MC: ");
+		printf(" - MP/MC");

 	if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE)
-		printf("single: ");
+		printf(" - single - ");
 	else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK)
-		printf("bulk (size: %u): ", bsz);
+		printf(" - bulk (n:%-3u) - ", bsz);
 	else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST)
-		printf("burst (size: %u): ", bsz);
+		printf(" - burst (n:%-3u) - ", bsz);
+	else if ((api_type & (TEST_RING_ELEM_BURST_ZC |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0)
+		printf(" - burst zero copy (n:%-3u) - ", bsz);

-	printf("%.2F\n", value);
+	printf("cycles per elem: %.3F\n", value);
 }

 /**** Functions to analyse our core mask to get cores for different tests ***/
@@ -144,28 +148,33 @@ static void
 test_empty_dequeue(struct rte_ring *r, const int esize,
 			const unsigned int api_type)
 {
-	const unsigned int iter_shift = 26;
+	const unsigned int iter_shift = 29;
 	const unsigned int iterations = 1 << iter_shift;
 	unsigned int i = 0;
 	void *burst[MAX_BURST];

+	const unsigned int bulk_iterations = iterations / bulk_sizes[0];
 	const uint64_t start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	for (i = 0; i < bulk_iterations; i++)
 		test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type);
 	const uint64_t end = rte_rdtsc();

 	test_ring_print_test_string(api_type, esize, bulk_sizes[0],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 }

-/*
- * for the separate enqueue and dequeue threads they take in one param
- * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc
- */
-struct thread_params {
+/* describes the ring used by the enqueue and dequeue thread */
+struct ring_params {
 	struct rte_ring *r;
-	unsigned size;        /* input value, the burst size */
-	double spsc, mpmc;    /* output value, the single or multi timings */
+	unsigned int elem_size;
+	unsigned int bulk_sizes_i; /* index into bulk_size array */
+	unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */
+};
+
+/* Used do specify enqueue and dequeue ring operations and their results */
+struct thread_params {
+	struct ring_params *ring_params;
+	double *results; /* result array size must be equal to bulk_sizes */
 };

 /*
@@ -174,16 +183,15 @@ struct thread_params {
  * flag == 1 -> dequeue
  */
 static __rte_always_inline int
-enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
-	struct thread_params *p)
+enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 15;
+	const unsigned int iter_shift = 22;
 	const unsigned int iterations = 1 << iter_shift;
-	struct rte_ring *r = p->r;
-	unsigned int bsize = p->size;
 	unsigned int i;
 	void *burst = NULL;
+	unsigned int n_remaining;
+	const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i];

 #ifdef RTE_USE_C11_MEM_MODEL
 	if (__atomic_fetch_add(&lcore_count, 1, __ATOMIC_RELAXED) + 1 != 2)
@@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 		while(lcore_count != 2)
 			rte_pause();

-	burst = test_ring_calloc(MAX_BURST, esize);
+	burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size);
 	if (burst == NULL)
 		return -1;

 	const uint64_t sp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	const unsigned int bulk_iterations = iterations / bulk_n;
+	for (i = 0; i < bulk_iterations; i++) {
+		n_remaining = bulk_n;
 		do {
 			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_enqueue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_dequeue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			if (ret == 0)
 				rte_pause();
-		} while (!ret);
+			else
+				n_remaining -= ret;
+		} while (n_remaining > 0);
+	}
 	const uint64_t sp_end = rte_rdtsc();

-	const uint64_t mp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
-		do {
-			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			if (ret == 0)
-				rte_pause();
-		} while (!ret);
-	const uint64_t mp_end = rte_rdtsc();
+	p->results[p->ring_params->bulk_sizes_i] =
+			((double)sp_end - sp_start) / iterations;

-	p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize);
-	p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize);
 	return 0;
 }

@@ -243,15 +245,7 @@ enqueue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(0, -1, params);
-}
-
-static int
-enqueue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(0, 16, params);
+	return enqueue_dequeue_bulk_helper(0, params);
 }

 /*
@@ -263,15 +257,7 @@ dequeue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(1, -1, params);
-}
-
-static int
-dequeue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(1, 16, params);
+	return enqueue_dequeue_bulk_helper(1, params);
 }

 /*
@@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p)
  * used to measure ring perf between hyperthreads, cores and sockets.
  */
 static int
-run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize)
+run_on_core_pair(struct lcore_pair *cores,
+		struct thread_params *param1, struct thread_params *param2)
 {
-	lcore_function_t *f1, *f2;
-	struct thread_params param1 = {0}, param2 = {0};
 	unsigned i;
-
-	if (esize == -1) {
-		f1 = enqueue_bulk;
-		f2 = dequeue_bulk;
-	} else {
-		f1 = enqueue_bulk_16B;
-		f2 = dequeue_bulk_16B;
-	}
+	struct ring_params *ring_params = param1->ring_params;

 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		lcore_count = 0;
-		param1.size = param2.size = bulk_sizes[i];
-		param1.r = param2.r = r;
+		ring_params->bulk_sizes_i = i;
 		if (cores->c1 == rte_get_main_lcore()) {
-			rte_eal_remote_launch(f2, &param2, cores->c2);
-			f1(&param1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
+			enqueue_bulk(param1);
 			rte_eal_wait_lcore(cores->c2);
 		} else {
-			rte_eal_remote_launch(f1, &param1, cores->c1);
-			rte_eal_remote_launch(f2, &param2, cores->c2);
+			rte_eal_remote_launch(enqueue_bulk, param1, cores->c1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
 			if (rte_eal_wait_lcore(cores->c1) < 0)
 				return -1;
 			if (rte_eal_wait_lcore(cores->c2) < 0)
 				return -1;
 		}
 		test_ring_print_test_string(
-			TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.spsc + param2.spsc);
-		test_ring_print_test_string(
-			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.mpmc + param2.mpmc);
+				ring_params->ring_flags,
+				ring_params->elem_size,
+				bulk_sizes[i],
+				param1->results[i] + param2->results[i]);
 	}

 	return 0;
@@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 	uint64_t hz = rte_get_timer_hz();
 	uint64_t lcount = 0;
 	const unsigned int lcore = rte_lcore_id();
-	struct thread_params *params = p;
+	struct ring_params *ring_params = p->ring_params;
 	void *burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -346,9 +322,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize)

 	begin = rte_get_timer_cycles();
 	while (time_diff < hz * TIME_MS / 1000) {
-		test_ring_enqueue(params->r, burst, esize, params->size,
+		test_ring_enqueue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
-		test_ring_dequeue(params->r, burst, esize, params->size,
+		test_ring_dequeue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 		lcount++;
 		time_diff = rte_get_timer_cycles() - begin;
@@ -380,7 +358,8 @@ static int
 run_on_all_cores(struct rte_ring *r, const int esize)
 {
 	uint64_t total;
-	struct thread_params param;
+	struct ring_params ring_params = {0};
+	struct thread_params params = { .ring_params = &ring_params };
 	lcore_function_t *lcore_f;
 	unsigned int i, c;

@@ -389,21 +368,20 @@ run_on_all_cores(struct rte_ring *r, const int esize)
 	else
 		lcore_f = load_loop_fn_16B;

-	memset(&param, 0, sizeof(struct thread_params));
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		total = 0;
 		printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]);
-		param.size = bulk_sizes[i];
-		param.r = r;
+		params.ring_params->bulk_sizes_i = i;
+		params.ring_params->r = r;

 		/* clear synchro and start workers */
 		__atomic_store_n(&synchro, 0, __ATOMIC_RELAXED);
-		if (rte_eal_mp_remote_launch(lcore_f, &param, SKIP_MAIN) < 0)
+		if (rte_eal_mp_remote_launch(lcore_f, &params, SKIP_MAIN) < 0)
 			return -1;

 		/* start synchro and launch test on main */
 		__atomic_store_n(&synchro, 1, __ATOMIC_RELAXED);
-		lcore_f(&param);
+		lcore_f(&params);

 		rte_eal_mp_wait_lcore();

@@ -462,9 +440,9 @@ static int
 test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	const unsigned int api_type)
 {
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 26;
 	const unsigned int iterations = 1 << iter_shift;
-	unsigned int sz, i = 0;
+	unsigned int sz, i;
 	void **burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -472,17 +450,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 		return -1;

 	for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) {
+		const unsigned int n = iterations / bulk_sizes[sz];
 		const uint64_t start = rte_rdtsc();
-		for (i = 0; i < iterations; i++) {
+		for (i = 0; i < n; i++) {
 			test_ring_enqueue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 			test_ring_dequeue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 		}
 		const uint64_t end = rte_rdtsc();

 		test_ring_print_test_string(api_type, esize, bulk_sizes[sz],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 	}

 	rte_free(burst);
@@ -490,12 +469,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	return 0;
 }

+static __rte_always_inline int
+test_ring_perf_esize_run_on_two_cores(
+		struct thread_params *param1, struct thread_params *param2)
+{
+	struct lcore_pair cores;
+
+	if (get_two_hyperthreads(&cores) == 0) {
+		printf("\n### Testing using two hyperthreads ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing using two physical cores ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_sockets(&cores) == 0) {
+		printf("\n### Testing using two NUMA nodes ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	return 0;
+}
+
 /* Run all tests for a given element size */
 static __rte_always_inline int
 test_ring_perf_esize(const int esize)
 {
-	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
+	double results_enq[RTE_DIM(bulk_sizes)];
+	double results_deq[RTE_DIM(bulk_sizes)];
+	struct ring_params ring_params = {
+			.elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results_enq };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results_deq };

 	/*
 	 * Performance test for legacy/_elem APIs
@@ -535,22 +545,13 @@ test_ring_perf_esize(const int esize)
 	test_empty_dequeue(r, esize,
 			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);

-	if (get_two_hyperthreads(&cores) == 0) {
-		printf("\n### Testing using two hyperthreads ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.r = r;

-	if (get_two_cores(&cores) == 0) {
-		printf("\n### Testing using two physical cores ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
-	if (get_two_sockets(&cores) == 0) {
-		printf("\n### Testing using two NUMA nodes ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
+
+	ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);

 	printf("\n### Testing using all worker nodes ###\n");
 	if (run_on_all_cores(r, esize) < 0)
@@ -566,6 +567,109 @@ test_ring_perf_esize(const int esize)
 	return -1;
 }

+
+static __rte_always_inline int
+test_ring_perf_compression(void)
+{
+	double results1[RTE_DIM(bulk_sizes)];
+	double results2[RTE_DIM(bulk_sizes)];
+	double results1_comp[2][RTE_DIM(bulk_sizes)];
+	double results2_comp[2][RTE_DIM(bulk_sizes)];
+
+	struct lcore_pair cores;
+	int ret = -1;
+	unsigned int i, j;
+	struct ring_params ring_params = { .elem_size = sizeof(void *) };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results1 };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results2 };
+
+	printf("\n### Testing compression gain ###");
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(void *),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy ###\n");
+		ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	/* rings allow only multiples of 4 as sizes,
+	 * we allocate size 4 despite only using 2 bytes
+	 * and use half of RING_SIZE as the number of elements
+	 */
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE / 2, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[0];
+	param2.results = results2_comp[0];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (16b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[1];
+	param2.results = results2_comp[1];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (32b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	for (j = 0; j < 2; j++) {
+		printf("\n### Potential gain from compression (%d-bit offsets) "
+		"###\n", (j + 1) * 16);
+		for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
+			const double result = results1[i] + results2[i];
+			const double result_comp = results1_comp[j][i] +
+				results2_comp[j][i];
+			const double gain = 100 - (result_comp / result) * 100;
+
+			printf("Gain of %5.1F%% for burst of %-3u elems\n",
+					gain, bulk_sizes[i]);
+		}
+	}
+
+	return ret;
+}
+
 static int
 test_ring_perf(void)
 {
@@ -576,6 +680,10 @@ test_ring_perf(void)
 	if (test_ring_perf_esize(16) == -1)
 		return -1;

+	/* Test for performance gain of compression */
+	if (test_ring_perf_compression() == -1)
+		return -1;
+
 	return 0;
 }

--
2.25.1

[PATCH v4 3/4] docs: add pointer compression to the EAL guide

[Paul Szczepanek]

Documentation added in the EAL guide for the new
utility functions for pointer compression
showing example code and potential usecases

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .../prog_guide/env_abstraction_layer.rst      | 142 ++++++++++++++++++
 1 file changed, 142 insertions(+)

diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 89014789de..cc56784e3d 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -1192,3 +1192,145 @@ will not be deallocated.

 Any successful deallocation event will trigger a callback, for which user
 applications and other DPDK subsystems can register.
+
+.. _pointer_compression:
+
+Pointer Compression
+-------------------
+
+Use ``rte_ptr_compress_16()`` and ``rte_ptr_decompress_16()`` to compress and
+decompress pointers into 16-bit offsets. Use ``rte_ptr_compress_32()`` and
+``rte_ptr_decompress_32()`` to compress and decompress pointers into 32-bit
+offsets.
+
+Compression takes advantage of the fact that pointers are usually located in a
+limited memory region (like a mempool). By converting them to offsets from a
+base memory address they can be stored in fewer bytes. How many bytes are needed
+to store the offset is dictated by the memory region size and alignment of
+objects the pointers point to.
+
+For example, a pointer which is part of a 4GB memory pool can be stored as 32
+bit offset. If the pointer points to memory that is 8 bytes aligned then 3 bits
+can be dropped from the offset and a 32GB memory pool can now fit in 32 bits.
+
+For performance reasons these requirements are not enforced programmatically.
+The programmer is responsible for ensuring that the combination of distance
+from the base pointer and memory alignment allow for storing of the offset in
+the number of bits indicated by the function name (16 or 32). Start of mempool
+memory would be a good candidate for the base pointer. Otherwise any pointer
+that precedes all pointers, is close enough and has the same alignment as the
+pointers being compressed will work.
+
+.. note::
+
+    Performance gains depend on the batch size of pointers and CPU capabilities
+    such as vector extensions. It's important to measure the performance
+    increase on target hardware. A test called ``ring_perf_autotest`` in
+    ``dpdk-test`` can provide the measurements.
+
+Example usage
+~~~~~~~~~~~~~
+
+In this example we send pointers between two cores through a ring. While this
+is a realistic use case the code is simplified for demonstration purposes and
+does not have error handling.
+
+.. code-block:: c
+
+    #include <rte_launch.h>
+    #include <rte_ring.h>
+    #include <rte_ring_elem.h>
+    #include <rte_ptr_compress.h>
+
+    #define ITEMS_ARRAY_SIZE (1024)
+    #define BATCH_SIZE (128)
+    #define ALIGN_EXPONENT (3)
+    #define ITEM_ALIGN (1<<ALIGN_EXPONENT)
+    #define CORE_SEND (1)
+    #define CORE_RECV (2)
+
+    struct item {
+      int a;
+    } __rte_aligned(ITEM_ALIGN);
+
+    static struct item items[ITEMS_ARRAY_SIZE] = {0};
+    static struct rte_ring *ring = NULL;
+
+    static int
+    send_compressed(void *args)
+    {
+      struct item *ptrs_send[BATCH_SIZE] = {0};
+      unsigned int n_send = 0;
+      struct rte_ring_zc_data zcd = {0};
+
+      /* in this example we only fill the ptrs_send once and reuse */
+      for (;n_send < BATCH_SIZE; n_send++)
+        ptrs_send[n_send] = &items[n_send];
+
+      for(;;) {
+        n_send = rte_ring_enqueue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        /* compress ptrs_send into offsets */
+        rte_ptr_compress_32(items, /* base pointer */
+          ptrs_send, /* source array to be compressed */
+          zcd.ptr1, /* destination array to store offsets */
+          zcd.n1, /* how many pointers to compress */
+          ALIGN_EXPONENT /* how many bits can we drop from the offset */);
+
+        if (zcd.ptr2 != NULL)
+          rte_ptr_compress_32(items, ptrs_send + zcd.n1,
+            zcd.ptr2, n_send - zcd.n1, ALIGN_EXPONENT);
+
+        rte_ring_enqueue_zc_finish(ring, n_send);
+      }
+      return 1;
+    }
+
+    static int
+    recv_compressed(void *args)
+    {
+      struct item *ptrs_recv[BATCH_SIZE] = {0};
+      unsigned int n_recv;
+      struct rte_ring_zc_data zcd = {0};
+
+      for(;;) {
+        /* receive compressed pointers from the ring */
+        n_recv = rte_ring_dequeue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        rte_ptr_decompress_32(items, /* base pointer */
+          zcd.ptr1, /* source array to decompress */
+          ptrs_recv, /* destination array to store pointers */
+          zcd.n1, /* how many pointers to decompress */
+          ALIGN_EXPONENT /* how many bits were dropped from the offset */);
+
+        /* handle the potential secondary buffer (caused by ring boundary) */
+        if (zcd.ptr2 != NULL)
+          rte_ptr_decompress_32(items,
+            zcd.ptr2,
+            ptrs_recv + zcd.n1,
+            n_recv - zcd.n1,
+            ALIGN_EXPONENT);
+
+        rte_ring_dequeue_zc_finish(ring, n_recv);
+
+        /* ptrs_recv contains what ptrs_send contained in the other thread */
+        /* (...) */
+      }
+      return 1;
+    }
+
+    void
+    compression_example(void)
+    {
+      ring = rte_ring_create_elem(
+        "COMPR_PTRS", sizeof(uint32_t),
+        1024, rte_socket_id(),
+        RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+      rte_eal_remote_launch(send_compressed, NULL, CORE_SEND);
+      rte_eal_remote_launch(recv_compressed, NULL, CORE_RECV);
+
+      for(;;) {}
+    }
--
2.25.1

[PATCH v4 4/4] test: add unit test for ptr compression

[Paul Szczepanek]

Test compresses and decompresses pointers with various combinations
of memory regions and alignments and verifies the pointers are
recovered correctly.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 app/test/meson.build             |   1 +
 app/test/test_eal_ptr_compress.c | 108 +++++++++++++++++++++++++++++++
 2 files changed, 109 insertions(+)
 create mode 100644 app/test/test_eal_ptr_compress.c

diff --git a/app/test/meson.build b/app/test/meson.build
index 05bae9216d..753de4bbd3 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -61,6 +61,7 @@ source_file_deps = {
     'test_dmadev_api.c': ['dmadev'],
     'test_eal_flags.c': [],
     'test_eal_fs.c': [],
+    'test_eal_ptr_compress.c': [],
     'test_efd.c': ['efd', 'net'],
     'test_efd_perf.c': ['efd', 'hash'],
     'test_errno.c': [],
diff --git a/app/test/test_eal_ptr_compress.c b/app/test/test_eal_ptr_compress.c
new file mode 100644
index 0000000000..c1c9a98be7
--- /dev/null
+++ b/app/test/test_eal_ptr_compress.c
@@ -0,0 +1,108 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2014 Intel Corporation
+ */
+
+#include "test.h"
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_ptr_compress.h>
+
+#define MAX_ALIGN_EXPONENT 3
+#define PTRS_SIZE 16
+#define NUM_BASES 2
+#define NUM_REGIONS 4
+#define MAX_32BIT_REGION ((uint64_t)UINT32_MAX + 1)
+#define MAX_16BIT_REGION (UINT16_MAX + 1)
+
+static int
+test_eal_ptr_compress_params(
+	void *base,
+	uint64_t mem_sz,
+	unsigned int align_exp,
+	unsigned int num_ptrs,
+	bool use_32_bit)
+{
+	unsigned int i;
+	unsigned int align = 1 << align_exp;
+	void *ptrs[PTRS_SIZE] = {0};
+	void *ptrs_out[PTRS_SIZE] = {0};
+	uint32_t offsets32[PTRS_SIZE] = {0};
+	uint16_t offsets16[PTRS_SIZE] = {0};
+
+	for (i = 0; i < num_ptrs; i++) {
+		/* make pointers point at memory in steps of align */
+		/* alternate steps from the start and end of memory region */
+		if ((i & 1) == 1)
+			ptrs[i] = (char *)base + mem_sz - i * align;
+		else
+			ptrs[i] = (char *)base + i * align;
+	}
+
+	if (use_32_bit) {
+		rte_ptr_compress_32(base, ptrs, offsets32, num_ptrs, align_exp);
+		rte_ptr_decompress_32(base, offsets32, ptrs_out, num_ptrs,
+				align_exp);
+	} else {
+		rte_ptr_compress_16(base, ptrs, offsets16, num_ptrs, align_exp);
+		rte_ptr_decompress_16(base, offsets16, ptrs_out, num_ptrs,
+				align_exp);
+	}
+
+	TEST_ASSERT_BUFFERS_ARE_EQUAL(ptrs, ptrs_out, sizeof(void *) * num_ptrs,
+		"Decompressed pointers corrupted\nbase pointer: %p, "
+		"memory region size: %" PRIu64 ", alignment exponent: %u, "
+		"num of pointers: %u, using %s offsets",
+		base, mem_sz, align_exp, num_ptrs,
+		use_32_bit ? "32-bit" : "16-bit");
+
+	return 0;
+}
+
+static int
+test_eal_ptr_compress(void)
+{
+	unsigned int j, k, n;
+	int ret = 0;
+	void * const bases[NUM_BASES] = { (void *)0, (void *)UINT16_MAX };
+	/* maximum size for pointers aligned by consecutive powers of 2 */
+	const uint64_t region_sizes_16[NUM_REGIONS] = {
+		MAX_16BIT_REGION,
+		MAX_16BIT_REGION * 2,
+		MAX_16BIT_REGION * 4,
+		MAX_16BIT_REGION * 8,
+	};
+	const uint64_t region_sizes_32[NUM_REGIONS] = {
+		MAX_32BIT_REGION,
+		MAX_32BIT_REGION * 2,
+		MAX_32BIT_REGION * 4,
+		MAX_32BIT_REGION * 8,
+	};
+
+	for (j = 0; j < NUM_REGIONS; j++) {
+		for (k = 0; k < NUM_BASES; k++) {
+			for (n = 1; n < PTRS_SIZE; n++) {
+				ret |= test_eal_ptr_compress_params(
+					bases[k],
+					region_sizes_16[j],
+					j /* exponent of alignment */,
+					n,
+					false
+				);
+				ret |= test_eal_ptr_compress_params(
+					bases[k],
+					region_sizes_32[j],
+					j /* exponent of alignment */,
+					n,
+					true
+				);
+				if (ret != 0)
+					return ret;
+			}
+		}
+	}
+
+	return ret;
+}
+
+REGISTER_FAST_TEST(eal_ptr_compress_autotest, true, true, test_eal_ptr_compress);
--
2.25.1

[PATCH v5 0/4] add pointer compression API

[Paul Szczepanek]

This patchset is proposing adding a new EAL header with utility functions
that allow compression of arrays of pointers.

When passing caches full of pointers between threads, memory containing
the pointers is copied multiple times which is especially costly between
cores. A compression method will allow us to shrink the memory size
copied.

The compression takes advantage of the fact that pointers are usually
located in a limited memory region (like a mempool). We can compress them
by converting them to offsets from a base memory address.

Offsets can be stored in fewer bytes (dictated by the memory region size
and alignment of the pointer). For example: an 8 byte aligned pointer
which is part of a 32GB memory pool can be stored in 4 bytes. The API is
very generic and does not assume mempool pointers, any pointer can be
passed in.

Compression is based on few and fast operations and especially with vector
instructions leveraged creates minimal overhead.

The API accepts and returns arrays because the overhead means it only is
worth it when done in bulk.

Test is added that shows potential performance gain from compression. In
this test an array of pointers is passed through a ring between two cores.
It shows the gain which is dependent on the bulk operation size. In this
synthetic test run on ampere altra a substantial (up to 25%) performance
gain is seen if done in bulk size larger than 32. At 32 it breaks even and
lower sizes create a small (less than 5%) slowdown due to overhead.

In a more realistic mock application running the l3 forwarding dpdk
example that works in pipeline mode on two cores this translated into a
~5% throughput increase on an ampere altra.

v2:
* addressed review comments (style, explanations and typos)
* lowered bulk iterations closer to original numbers to keep runtime short
* fixed pointer size warning on 32-bit arch
v3:
* added 16-bit versions of compression functions and tests
* added documentation of these new utility functions in the EAL guide
v4:
* added unit test
* fix bug in NEON implementation of 32-bit decompress
v5:
* disable NEON and SVE implementation on AARCH32 due to wrong pointer size

Paul Szczepanek (4):
  eal: add pointer compression functions
  test: add pointer compress tests to ring perf test
  docs: add pointer compression to the EAL guide
  test: add unit test for ptr compression

 .mailmap                                      |   1 +
 app/test/meson.build                          |   1 +
 app/test/test_eal_ptr_compress.c              | 108 ++++++
 app/test/test_ring.h                          |  94 ++++-
 app/test/test_ring_perf.c                     | 354 ++++++++++++------
 .../prog_guide/env_abstraction_layer.rst      | 142 +++++++
 lib/eal/include/meson.build                   |   1 +
 lib/eal/include/rte_ptr_compress.h            | 266 +++++++++++++
 8 files changed, 843 insertions(+), 124 deletions(-)
 create mode 100644 app/test/test_eal_ptr_compress.c
 create mode 100644 lib/eal/include/rte_ptr_compress.h

--
2.25.1

[PATCH v5 1/4] eal: add pointer compression functions

[Paul Szczepanek]

Add a new utility header for compressing pointers. The provided
functions can store pointers in 32-bit offsets.

The compression takes advantage of the fact that pointers are
usually located in a limited memory region (like a mempool).
We can compress them by converting them to offsets from a base
memory address. Offsets can be stored in fewer bytes (dictated
by the memory region size and alignment of the pointer).
For example: an 8 byte aligned pointer which is part of a 32GB
memory pool can be stored in 4 bytes.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .mailmap                           |   1 +
 lib/eal/include/meson.build        |   1 +
 lib/eal/include/rte_ptr_compress.h | 266 +++++++++++++++++++++++++++++
 3 files changed, 268 insertions(+)
 create mode 100644 lib/eal/include/rte_ptr_compress.h

diff --git a/.mailmap b/.mailmap
index 3f5bab26a8..004751d27a 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1069,6 +1069,7 @@ Paul Greenwalt <paul.greenwalt@intel.com>
 Paulis Gributs <paulis.gributs@intel.com>
 Paul Luse <paul.e.luse@intel.com>
 Paul M Stillwell Jr <paul.m.stillwell.jr@intel.com>
+Paul Szczepanek <paul.szczepanek@arm.com>
 Pavan Kumar Linga <pavan.kumar.linga@intel.com>
 Pavan Nikhilesh <pbhagavatula@marvell.com> <pbhagavatula@caviumnetworks.com>
 Pavel Belous <pavel.belous@aquantia.com>
diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index e94b056d46..ce2c733633 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -36,6 +36,7 @@ headers += files(
         'rte_pci_dev_features.h',
         'rte_per_lcore.h',
         'rte_pflock.h',
+	'rte_ptr_compress.h',
         'rte_random.h',
         'rte_reciprocal.h',
         'rte_seqcount.h',
diff --git a/lib/eal/include/rte_ptr_compress.h b/lib/eal/include/rte_ptr_compress.h
new file mode 100644
index 0000000000..47a72e4213
--- /dev/null
+++ b/lib/eal/include/rte_ptr_compress.h
@@ -0,0 +1,266 @@
+/* SPDX-License-Identifier: BSD-shift-Clause
+ * Copyright(c) 2023 Arm Limited
+ */
+
+#ifndef RTE_PTR_COMPRESS_H
+#define RTE_PTR_COMPRESS_H
+
+/**
+ * @file
+ * Pointer compression and decompression functions.
+ *
+ * When passing arrays full of pointers between threads, memory containing
+ * the pointers is copied multiple times which is especially costly between
+ * cores. These functions allow us to compress the pointers.
+ *
+ * Compression takes advantage of the fact that pointers are usually located in
+ * a limited memory region (like a mempool). We compress them by converting them
+ * to offsets from a base memory address. Offsets can be stored in fewer bytes.
+ *
+ * The compression functions come in two varieties: 32-bit and 16-bit.
+ *
+ * To determine how many bits are needed to compress the pointer calculate
+ * the biggest offset possible (highest value pointer - base pointer)
+ * and shift the value right according to alignment (shift by exponent of the
+ * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by
+ * 3, etc.). The resulting value must fit in either 32 or 16 bits.
+ *
+ * For usage example and further explanation please see "Pointer Compression" in
+ * doc/guides/prog_guide/env_abstraction_layer.rst
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_vect.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Compress pointers into 32-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 32 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 32GB of the base pointer. If there is no such alignment guarantee they
+ * must be within 4GB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_32(void *ptr_base, void **src_table,
+		uint32_t *dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1w(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	/* right shift is done by left shifting by negative int */
+	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_ptr_table = vld1q_u64((const uint64_t *)src_table + i);
+		v_ptr_table = vsubq_u64(v_ptr_table, v_ptr_base);
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		vst1_u32(dest_table + i, vqmovn_u64(v_ptr_table));
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT32_MAX);
+		dest_table[i] = (uint32_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 32-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_32(void *ptr_base, uint32_t *src_table,
+		void **dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1uw_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	int64x2_t v_shift = vdupq_n_s64(bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_ptr_table = vmovl_u32(vld1_u32(src_table + i));
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		v_ptr_table = vaddq_u64(v_ptr_table, v_ptr_base);
+		vst1q_u64((uint64_t *)dest_table + i, v_ptr_table);
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+/**
+ * Compress pointers into 16-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 16 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 256KB of the base pointer. If there is no such alignment guarantee
+ * they must be within 64KB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_16(void *ptr_base, void **src_table,
+		uint16_t *dest_table, unsigned int n, unsigned int bit_shift)
+{
+
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1h(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT16_MAX);
+		dest_table[i] = (uint16_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 16-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_16(void *ptr_base, uint16_t *src_table,
+		void **dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1uh_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_PTR_COMPRESS_H */
--
2.25.1

Re: [PATCH v5 1/4] eal: add pointer compression functions

[Konstantin Ananyev]

> Add a new utility header for compressing pointers. The provided
> functions can store pointers in 32-bit offsets.
> 
> The compression takes advantage of the fact that pointers are
> usually located in a limited memory region (like a mempool).
> We can compress them by converting them to offsets from a base
> memory address. Offsets can be stored in fewer bytes (dictated
> by the memory region size and alignment of the pointer).
> For example: an 8 byte aligned pointer which is part of a 32GB
> memory pool can be stored in 4 bytes.
> 
> Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>


 From one side the code itself is very small and straightforward,
from other side - it is not clear to me what is intended usage for it
within DPDK and it's applianances?
Konstantin


> ---
>   .mailmap                           |   1 +
>   lib/eal/include/meson.build        |   1 +
>   lib/eal/include/rte_ptr_compress.h | 266 +++++++++++++++++++++++++++++
>   3 files changed, 268 insertions(+)
>   create mode 100644 lib/eal/include/rte_ptr_compress.h
> 
> diff --git a/.mailmap b/.mailmap
> index 3f5bab26a8..004751d27a 100644
> --- a/.mailmap
> +++ b/.mailmap
> @@ -1069,6 +1069,7 @@ Paul Greenwalt <paul.greenwalt@intel.com>
>   Paulis Gributs <paulis.gributs@intel.com>
>   Paul Luse <paul.e.luse@intel.com>
>   Paul M Stillwell Jr <paul.m.stillwell.jr@intel.com>
> +Paul Szczepanek <paul.szczepanek@arm.com>
>   Pavan Kumar Linga <pavan.kumar.linga@intel.com>
>   Pavan Nikhilesh <pbhagavatula@marvell.com> <pbhagavatula@caviumnetworks.com>
>   Pavel Belous <pavel.belous@aquantia.com>
> diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
> index e94b056d46..ce2c733633 100644
> --- a/lib/eal/include/meson.build
> +++ b/lib/eal/include/meson.build
> @@ -36,6 +36,7 @@ headers += files(
>           'rte_pci_dev_features.h',
>           'rte_per_lcore.h',
>           'rte_pflock.h',
> +	'rte_ptr_compress.h',
>           'rte_random.h',
>           'rte_reciprocal.h',
>           'rte_seqcount.h',
> diff --git a/lib/eal/include/rte_ptr_compress.h b/lib/eal/include/rte_ptr_compress.h
> new file mode 100644
> index 0000000000..47a72e4213
> --- /dev/null
> +++ b/lib/eal/include/rte_ptr_compress.h
> @@ -0,0 +1,266 @@
> +/* SPDX-License-Identifier: BSD-shift-Clause
> + * Copyright(c) 2023 Arm Limited
> + */
> +
> +#ifndef RTE_PTR_COMPRESS_H
> +#define RTE_PTR_COMPRESS_H
> +
> +/**
> + * @file
> + * Pointer compression and decompression functions.
> + *
> + * When passing arrays full of pointers between threads, memory containing
> + * the pointers is copied multiple times which is especially costly between
> + * cores. These functions allow us to compress the pointers.
> + *
> + * Compression takes advantage of the fact that pointers are usually located in
> + * a limited memory region (like a mempool). We compress them by converting them
> + * to offsets from a base memory address. Offsets can be stored in fewer bytes.
> + *
> + * The compression functions come in two varieties: 32-bit and 16-bit.
> + *
> + * To determine how many bits are needed to compress the pointer calculate
> + * the biggest offset possible (highest value pointer - base pointer)
> + * and shift the value right according to alignment (shift by exponent of the
> + * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by
> + * 3, etc.). The resulting value must fit in either 32 or 16 bits.
> + *
> + * For usage example and further explanation please see "Pointer Compression" in
> + * doc/guides/prog_guide/env_abstraction_layer.rst
> + */
> +
> +#include <stdint.h>
> +#include <inttypes.h>
> +
> +#include <rte_branch_prediction.h>
> +#include <rte_common.h>
> +#include <rte_debug.h>
> +#include <rte_vect.h>
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +/**
> + * Compress pointers into 32-bit offsets from base pointer.
> + *
> + * @note It is programmer's responsibility to ensure the resulting offsets fit
> + * into 32 bits. Alignment of the structures pointed to by the pointers allows
> + * us to drop bits from the offsets. This is controlled by the bit_shift
> + * parameter. This means that if structures are aligned by 8 bytes they must be
> + * within 32GB of the base pointer. If there is no such alignment guarantee they
> + * must be within 4GB.
> + *
> + * @param ptr_base
> + *   A pointer used to calculate offsets of pointers in src_table.
> + * @param src_table
> + *   A pointer to an array of pointers.
> + * @param dest_table
> + *   A pointer to an array of compressed pointers returned by this function.
> + * @param n
> + *   The number of objects to compress, must be strictly positive.
> + * @param bit_shift
> + *   Byte alignment of memory pointed to by the pointers allows for
> + *   bits to be dropped from the offset and hence widen the memory region that
> + *   can be covered. This controls how many bits are right shifted.
> + **/
> +static __rte_always_inline void
> +rte_ptr_compress_32(void *ptr_base, void **src_table,
> +		uint32_t *dest_table, unsigned int n, unsigned int bit_shift)
> +{
> +	unsigned int i = 0;
> +#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
> +	svuint64_t v_ptr_table;
> +	svbool_t pg = svwhilelt_b64(i, n);
> +	do {
> +		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
> +		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
> +		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
> +		svst1w(pg, &dest_table[i], v_ptr_table);
> +		i += svcntd();
> +		pg = svwhilelt_b64(i, n);
> +	} while (svptest_any(svptrue_b64(), pg));
> +#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
> +	uint64_t ptr_diff;
> +	uint64x2_t v_ptr_table;
> +	/* right shift is done by left shifting by negative int */
> +	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
> +	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
> +	for (; i < (n & ~0x1); i += 2) {
> +		v_ptr_table = vld1q_u64((const uint64_t *)src_table + i);
> +		v_ptr_table = vsubq_u64(v_ptr_table, v_ptr_base);
> +		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
> +		vst1_u32(dest_table + i, vqmovn_u64(v_ptr_table));
> +	}
> +	/* process leftover single item in case of odd number of n */
> +	if (unlikely(n & 0x1)) {
> +		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
> +		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
> +	}
> +#else
> +	uintptr_t ptr_diff;
> +	for (; i < n; i++) {
> +		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
> +		ptr_diff = ptr_diff >> bit_shift;
> +		RTE_ASSERT(ptr_diff <= UINT32_MAX);
> +		dest_table[i] = (uint32_t) ptr_diff;
> +	}
> +#endif
> +}
> +
> +/**
> + * Decompress pointers from 32-bit offsets from base pointer.
> + *
> + * @param ptr_base
> + *   A pointer which was used to calculate offsets in src_table.
> + * @param src_table
> + *   A pointer to an array to compressed pointers.
> + * @param dest_table
> + *   A pointer to an array of decompressed pointers returned by this function.
> + * @param n
> + *   The number of objects to decompress, must be strictly positive.
> + * @param bit_shift
> + *   Byte alignment of memory pointed to by the pointers allows for
> + *   bits to be dropped from the offset and hence widen the memory region that
> + *   can be covered. This controls how many bits are left shifted when pointers
> + *   are recovered from the offsets.
> + **/
> +static __rte_always_inline void
> +rte_ptr_decompress_32(void *ptr_base, uint32_t *src_table,
> +		void **dest_table, unsigned int n, unsigned int bit_shift)
> +{
> +	unsigned int i = 0;
> +#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
> +	svuint64_t v_ptr_table;
> +	svbool_t pg = svwhilelt_b64(i, n);
> +	do {
> +		v_ptr_table = svld1uw_u64(pg, &src_table[i]);
> +		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
> +		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
> +		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
> +		i += svcntd();
> +		pg = svwhilelt_b64(i, n);
> +	} while (svptest_any(svptrue_b64(), pg));
> +#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
> +	uint64_t ptr_diff;
> +	uint64x2_t v_ptr_table;
> +	int64x2_t v_shift = vdupq_n_s64(bit_shift);
> +	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
> +	for (; i < (n & ~0x1); i += 2) {
> +		v_ptr_table = vmovl_u32(vld1_u32(src_table + i));
> +		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
> +		v_ptr_table = vaddq_u64(v_ptr_table, v_ptr_base);
> +		vst1q_u64((uint64_t *)dest_table + i, v_ptr_table);
> +	}
> +	/* process leftover single item in case of odd number of n */
> +	if (unlikely(n & 0x1)) {
> +		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
> +		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
> +	}
> +#else
> +	uintptr_t ptr_diff;
> +	for (; i < n; i++) {
> +		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
> +		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
> +	}
> +#endif
> +}
> +
> +/**
> + * Compress pointers into 16-bit offsets from base pointer.
> + *
> + * @note It is programmer's responsibility to ensure the resulting offsets fit
> + * into 16 bits. Alignment of the structures pointed to by the pointers allows
> + * us to drop bits from the offsets. This is controlled by the bit_shift
> + * parameter. This means that if structures are aligned by 8 bytes they must be
> + * within 256KB of the base pointer. If there is no such alignment guarantee
> + * they must be within 64KB.
> + *
> + * @param ptr_base
> + *   A pointer used to calculate offsets of pointers in src_table.
> + * @param src_table
> + *   A pointer to an array of pointers.
> + * @param dest_table
> + *   A pointer to an array of compressed pointers returned by this function.
> + * @param n
> + *   The number of objects to compress, must be strictly positive.
> + * @param bit_shift
> + *   Byte alignment of memory pointed to by the pointers allows for
> + *   bits to be dropped from the offset and hence widen the memory region that
> + *   can be covered. This controls how many bits are right shifted.
> + **/
> +static __rte_always_inline void
> +rte_ptr_compress_16(void *ptr_base, void **src_table,
> +		uint16_t *dest_table, unsigned int n, unsigned int bit_shift)
> +{
> +
> +	unsigned int i = 0;
> +#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
> +	svuint64_t v_ptr_table;
> +	svbool_t pg = svwhilelt_b64(i, n);
> +	do {
> +		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
> +		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
> +		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
> +		svst1h(pg, &dest_table[i], v_ptr_table);
> +		i += svcntd();
> +		pg = svwhilelt_b64(i, n);
> +	} while (svptest_any(svptrue_b64(), pg));
> +#else
> +	uintptr_t ptr_diff;
> +	for (; i < n; i++) {
> +		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
> +		ptr_diff = ptr_diff >> bit_shift;
> +		RTE_ASSERT(ptr_diff <= UINT16_MAX);
> +		dest_table[i] = (uint16_t) ptr_diff;
> +	}
> +#endif
> +}
> +
> +/**
> + * Decompress pointers from 16-bit offsets from base pointer.
> + *
> + * @param ptr_base
> + *   A pointer which was used to calculate offsets in src_table.
> + * @param src_table
> + *   A pointer to an array to compressed pointers.
> + * @param dest_table
> + *   A pointer to an array of decompressed pointers returned by this function.
> + * @param n
> + *   The number of objects to decompress, must be strictly positive.
> + * @param bit_shift
> + *   Byte alignment of memory pointed to by the pointers allows for
> + *   bits to be dropped from the offset and hence widen the memory region that
> + *   can be covered. This controls how many bits are left shifted when pointers
> + *   are recovered from the offsets.
> + **/
> +static __rte_always_inline void
> +rte_ptr_decompress_16(void *ptr_base, uint16_t *src_table,
> +		void **dest_table, unsigned int n, unsigned int bit_shift)
> +{
> +	unsigned int i = 0;
> +#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
> +	svuint64_t v_ptr_table;
> +	svbool_t pg = svwhilelt_b64(i, n);
> +	do {
> +		v_ptr_table = svld1uh_u64(pg, &src_table[i]);
> +		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
> +		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
> +		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
> +		i += svcntd();
> +		pg = svwhilelt_b64(i, n);
> +	} while (svptest_any(svptrue_b64(), pg));
> +#else
> +	uintptr_t ptr_diff;
> +	for (; i < n; i++) {
> +		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
> +		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
> +	}
> +#endif
> +}
> +
> +#ifdef __cplusplus
> +}
> +#endif
> +
> +#endif /* RTE_PTR_COMPRESS_H */
> --
> 2.25.1
> 

[PATCH v5 2/4] test: add pointer compress tests to ring perf test

[Paul Szczepanek]

Add a test that runs a zero copy burst enqueue and dequeue on a ring
of raw pointers and compressed pointers at different burst sizes to
showcase performance benefits of newly added pointer compression APIs.

Refactored threading code to pass more parameters to threads to
reuse existing code. Added more bulk sizes to showcase their effects
on compression. Adjusted loop iteration numbers to take into account
bulk sizes to keep runtime constant (instead of number of operations).

Adjusted old printfs to match new ones which have aligned numbers.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 app/test/test_ring.h      |  94 +++++++++-
 app/test/test_ring_perf.c | 354 +++++++++++++++++++++++++-------------
 2 files changed, 324 insertions(+), 124 deletions(-)

diff --git a/app/test/test_ring.h b/app/test/test_ring.h
index 45c263f3ff..3b00f2465d 100644
--- a/app/test/test_ring.h
+++ b/app/test/test_ring.h
@@ -1,10 +1,12 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */

 #include <rte_malloc.h>
 #include <rte_ring.h>
 #include <rte_ring_elem.h>
+#include <rte_memcpy.h>
+#include <rte_ptr_compress.h>

 /* API type to call
  * rte_ring_<sp/mp or sc/mc>_enqueue_<bulk/burst>
@@ -25,6 +27,10 @@
 #define TEST_RING_ELEM_BULK 16
 #define TEST_RING_ELEM_BURST 32

+#define TEST_RING_ELEM_BURST_ZC 64
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 128
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 256
+
 #define TEST_RING_IGNORE_API_TYPE ~0U

 /* This function is placed here as it is required for both
@@ -101,6 +107,9 @@ static inline unsigned int
 test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -152,6 +161,46 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n,
 								NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy(zcd.ptr2,
+						(char *)obj + zcd.n1 * esize,
+						(ret - zcd.n1) * esize);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_16(0, obj, zcd.ptr1, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_16(0,
+						obj + (zcd.n1 * 2),
+						zcd.ptr2,
+						(ret - zcd.n1) * 2, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_32(0, obj, zcd.ptr1, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_32(0, obj + zcd.n1,
+						zcd.ptr2, ret - zcd.n1, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
@@ -162,6 +211,9 @@ static inline unsigned int
 test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -213,6 +265,46 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mc_dequeue_burst_elem(r, obj, esize,
 								n, NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy((char *)obj + zcd.n1 * esize,
+						zcd.ptr2,
+						(ret - zcd.n1) * esize);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_16(0, zcd.ptr1, obj, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_16(0, zcd.ptr2,
+						obj + zcd.n1,
+						(ret - zcd.n1) * 2,
+						3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_32(0, zcd.ptr1, obj, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_32(0, zcd.ptr2,
+						obj + zcd.n1, ret - zcd.n1, 3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c
index d7c5a4c30b..8392d0c781 100644
--- a/app/test/test_ring_perf.c
+++ b/app/test/test_ring_perf.c
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2010-2014 Intel Corporation
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */


@@ -22,13 +22,13 @@

 #define RING_NAME "RING_PERF"
 #define RING_SIZE 4096
-#define MAX_BURST 32
+#define MAX_BURST 256

 /*
  * the sizes to enqueue and dequeue in testing
  * (marked volatile so they won't be seen as compile-time constants)
  */
-static const volatile unsigned bulk_sizes[] = { 8, 32 };
+static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128, 256 };

 struct lcore_pair {
 	unsigned c1, c2;
@@ -43,26 +43,30 @@ test_ring_print_test_string(unsigned int api_type, int esize,
 	if (esize == -1)
 		printf("legacy APIs");
 	else
-		printf("elem APIs: element size %dB", esize);
+		printf("elem APIs (size:%2dB)", esize);

 	if (api_type == TEST_RING_IGNORE_API_TYPE)
 		return;

 	if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF)
-		printf(": default enqueue/dequeue: ");
+		printf(" - default enqueue/dequeue");
 	else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC)
-		printf(": SP/SC: ");
+		printf(" - SP/SC");
 	else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC)
-		printf(": MP/MC: ");
+		printf(" - MP/MC");

 	if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE)
-		printf("single: ");
+		printf(" - single - ");
 	else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK)
-		printf("bulk (size: %u): ", bsz);
+		printf(" - bulk (n:%-3u) - ", bsz);
 	else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST)
-		printf("burst (size: %u): ", bsz);
+		printf(" - burst (n:%-3u) - ", bsz);
+	else if ((api_type & (TEST_RING_ELEM_BURST_ZC |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0)
+		printf(" - burst zero copy (n:%-3u) - ", bsz);

-	printf("%.2F\n", value);
+	printf("cycles per elem: %.3F\n", value);
 }

 /**** Functions to analyse our core mask to get cores for different tests ***/
@@ -144,28 +148,33 @@ static void
 test_empty_dequeue(struct rte_ring *r, const int esize,
 			const unsigned int api_type)
 {
-	const unsigned int iter_shift = 26;
+	const unsigned int iter_shift = 29;
 	const unsigned int iterations = 1 << iter_shift;
 	unsigned int i = 0;
 	void *burst[MAX_BURST];

+	const unsigned int bulk_iterations = iterations / bulk_sizes[0];
 	const uint64_t start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	for (i = 0; i < bulk_iterations; i++)
 		test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type);
 	const uint64_t end = rte_rdtsc();

 	test_ring_print_test_string(api_type, esize, bulk_sizes[0],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 }

-/*
- * for the separate enqueue and dequeue threads they take in one param
- * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc
- */
-struct thread_params {
+/* describes the ring used by the enqueue and dequeue thread */
+struct ring_params {
 	struct rte_ring *r;
-	unsigned size;        /* input value, the burst size */
-	double spsc, mpmc;    /* output value, the single or multi timings */
+	unsigned int elem_size;
+	unsigned int bulk_sizes_i; /* index into bulk_size array */
+	unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */
+};
+
+/* Used do specify enqueue and dequeue ring operations and their results */
+struct thread_params {
+	struct ring_params *ring_params;
+	double *results; /* result array size must be equal to bulk_sizes */
 };

 /*
@@ -174,16 +183,15 @@ struct thread_params {
  * flag == 1 -> dequeue
  */
 static __rte_always_inline int
-enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
-	struct thread_params *p)
+enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 15;
+	const unsigned int iter_shift = 22;
 	const unsigned int iterations = 1 << iter_shift;
-	struct rte_ring *r = p->r;
-	unsigned int bsize = p->size;
 	unsigned int i;
 	void *burst = NULL;
+	unsigned int n_remaining;
+	const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i];

 #ifdef RTE_USE_C11_MEM_MODEL
 	if (__atomic_fetch_add(&lcore_count, 1, __ATOMIC_RELAXED) + 1 != 2)
@@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 		while(lcore_count != 2)
 			rte_pause();

-	burst = test_ring_calloc(MAX_BURST, esize);
+	burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size);
 	if (burst == NULL)
 		return -1;

 	const uint64_t sp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	const unsigned int bulk_iterations = iterations / bulk_n;
+	for (i = 0; i < bulk_iterations; i++) {
+		n_remaining = bulk_n;
 		do {
 			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_enqueue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_dequeue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			if (ret == 0)
 				rte_pause();
-		} while (!ret);
+			else
+				n_remaining -= ret;
+		} while (n_remaining > 0);
+	}
 	const uint64_t sp_end = rte_rdtsc();

-	const uint64_t mp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
-		do {
-			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			if (ret == 0)
-				rte_pause();
-		} while (!ret);
-	const uint64_t mp_end = rte_rdtsc();
+	p->results[p->ring_params->bulk_sizes_i] =
+			((double)sp_end - sp_start) / iterations;

-	p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize);
-	p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize);
 	return 0;
 }

@@ -243,15 +245,7 @@ enqueue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(0, -1, params);
-}
-
-static int
-enqueue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(0, 16, params);
+	return enqueue_dequeue_bulk_helper(0, params);
 }

 /*
@@ -263,15 +257,7 @@ dequeue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(1, -1, params);
-}
-
-static int
-dequeue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(1, 16, params);
+	return enqueue_dequeue_bulk_helper(1, params);
 }

 /*
@@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p)
  * used to measure ring perf between hyperthreads, cores and sockets.
  */
 static int
-run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize)
+run_on_core_pair(struct lcore_pair *cores,
+		struct thread_params *param1, struct thread_params *param2)
 {
-	lcore_function_t *f1, *f2;
-	struct thread_params param1 = {0}, param2 = {0};
 	unsigned i;
-
-	if (esize == -1) {
-		f1 = enqueue_bulk;
-		f2 = dequeue_bulk;
-	} else {
-		f1 = enqueue_bulk_16B;
-		f2 = dequeue_bulk_16B;
-	}
+	struct ring_params *ring_params = param1->ring_params;

 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		lcore_count = 0;
-		param1.size = param2.size = bulk_sizes[i];
-		param1.r = param2.r = r;
+		ring_params->bulk_sizes_i = i;
 		if (cores->c1 == rte_get_main_lcore()) {
-			rte_eal_remote_launch(f2, &param2, cores->c2);
-			f1(&param1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
+			enqueue_bulk(param1);
 			rte_eal_wait_lcore(cores->c2);
 		} else {
-			rte_eal_remote_launch(f1, &param1, cores->c1);
-			rte_eal_remote_launch(f2, &param2, cores->c2);
+			rte_eal_remote_launch(enqueue_bulk, param1, cores->c1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
 			if (rte_eal_wait_lcore(cores->c1) < 0)
 				return -1;
 			if (rte_eal_wait_lcore(cores->c2) < 0)
 				return -1;
 		}
 		test_ring_print_test_string(
-			TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.spsc + param2.spsc);
-		test_ring_print_test_string(
-			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.mpmc + param2.mpmc);
+				ring_params->ring_flags,
+				ring_params->elem_size,
+				bulk_sizes[i],
+				param1->results[i] + param2->results[i]);
 	}

 	return 0;
@@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 	uint64_t hz = rte_get_timer_hz();
 	uint64_t lcount = 0;
 	const unsigned int lcore = rte_lcore_id();
-	struct thread_params *params = p;
+	struct ring_params *ring_params = p->ring_params;
 	void *burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -346,9 +322,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize)

 	begin = rte_get_timer_cycles();
 	while (time_diff < hz * TIME_MS / 1000) {
-		test_ring_enqueue(params->r, burst, esize, params->size,
+		test_ring_enqueue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
-		test_ring_dequeue(params->r, burst, esize, params->size,
+		test_ring_dequeue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 		lcount++;
 		time_diff = rte_get_timer_cycles() - begin;
@@ -380,7 +358,8 @@ static int
 run_on_all_cores(struct rte_ring *r, const int esize)
 {
 	uint64_t total;
-	struct thread_params param;
+	struct ring_params ring_params = {0};
+	struct thread_params params = { .ring_params = &ring_params };
 	lcore_function_t *lcore_f;
 	unsigned int i, c;

@@ -389,21 +368,20 @@ run_on_all_cores(struct rte_ring *r, const int esize)
 	else
 		lcore_f = load_loop_fn_16B;

-	memset(&param, 0, sizeof(struct thread_params));
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		total = 0;
 		printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]);
-		param.size = bulk_sizes[i];
-		param.r = r;
+		params.ring_params->bulk_sizes_i = i;
+		params.ring_params->r = r;

 		/* clear synchro and start workers */
 		__atomic_store_n(&synchro, 0, __ATOMIC_RELAXED);
-		if (rte_eal_mp_remote_launch(lcore_f, &param, SKIP_MAIN) < 0)
+		if (rte_eal_mp_remote_launch(lcore_f, &params, SKIP_MAIN) < 0)
 			return -1;

 		/* start synchro and launch test on main */
 		__atomic_store_n(&synchro, 1, __ATOMIC_RELAXED);
-		lcore_f(&param);
+		lcore_f(&params);

 		rte_eal_mp_wait_lcore();

@@ -462,9 +440,9 @@ static int
 test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	const unsigned int api_type)
 {
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 26;
 	const unsigned int iterations = 1 << iter_shift;
-	unsigned int sz, i = 0;
+	unsigned int sz, i;
 	void **burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -472,17 +450,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 		return -1;

 	for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) {
+		const unsigned int n = iterations / bulk_sizes[sz];
 		const uint64_t start = rte_rdtsc();
-		for (i = 0; i < iterations; i++) {
+		for (i = 0; i < n; i++) {
 			test_ring_enqueue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 			test_ring_dequeue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 		}
 		const uint64_t end = rte_rdtsc();

 		test_ring_print_test_string(api_type, esize, bulk_sizes[sz],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 	}

 	rte_free(burst);
@@ -490,12 +469,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	return 0;
 }

+static __rte_always_inline int
+test_ring_perf_esize_run_on_two_cores(
+		struct thread_params *param1, struct thread_params *param2)
+{
+	struct lcore_pair cores;
+
+	if (get_two_hyperthreads(&cores) == 0) {
+		printf("\n### Testing using two hyperthreads ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing using two physical cores ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_sockets(&cores) == 0) {
+		printf("\n### Testing using two NUMA nodes ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	return 0;
+}
+
 /* Run all tests for a given element size */
 static __rte_always_inline int
 test_ring_perf_esize(const int esize)
 {
-	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
+	double results_enq[RTE_DIM(bulk_sizes)];
+	double results_deq[RTE_DIM(bulk_sizes)];
+	struct ring_params ring_params = {
+			.elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results_enq };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results_deq };

 	/*
 	 * Performance test for legacy/_elem APIs
@@ -535,22 +545,13 @@ test_ring_perf_esize(const int esize)
 	test_empty_dequeue(r, esize,
 			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);

-	if (get_two_hyperthreads(&cores) == 0) {
-		printf("\n### Testing using two hyperthreads ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.r = r;

-	if (get_two_cores(&cores) == 0) {
-		printf("\n### Testing using two physical cores ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
-	if (get_two_sockets(&cores) == 0) {
-		printf("\n### Testing using two NUMA nodes ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
+
+	ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);

 	printf("\n### Testing using all worker nodes ###\n");
 	if (run_on_all_cores(r, esize) < 0)
@@ -566,6 +567,109 @@ test_ring_perf_esize(const int esize)
 	return -1;
 }

+
+static __rte_always_inline int
+test_ring_perf_compression(void)
+{
+	double results1[RTE_DIM(bulk_sizes)];
+	double results2[RTE_DIM(bulk_sizes)];
+	double results1_comp[2][RTE_DIM(bulk_sizes)];
+	double results2_comp[2][RTE_DIM(bulk_sizes)];
+
+	struct lcore_pair cores;
+	int ret = -1;
+	unsigned int i, j;
+	struct ring_params ring_params = { .elem_size = sizeof(void *) };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results1 };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results2 };
+
+	printf("\n### Testing compression gain ###");
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(void *),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy ###\n");
+		ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	/* rings allow only multiples of 4 as sizes,
+	 * we allocate size 4 despite only using 2 bytes
+	 * and use half of RING_SIZE as the number of elements
+	 */
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE / 2, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[0];
+	param2.results = results2_comp[0];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (16b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[1];
+	param2.results = results2_comp[1];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (32b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	for (j = 0; j < 2; j++) {
+		printf("\n### Potential gain from compression (%d-bit offsets) "
+		"###\n", (j + 1) * 16);
+		for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
+			const double result = results1[i] + results2[i];
+			const double result_comp = results1_comp[j][i] +
+				results2_comp[j][i];
+			const double gain = 100 - (result_comp / result) * 100;
+
+			printf("Gain of %5.1F%% for burst of %-3u elems\n",
+					gain, bulk_sizes[i]);
+		}
+	}
+
+	return ret;
+}
+
 static int
 test_ring_perf(void)
 {
@@ -576,6 +680,10 @@ test_ring_perf(void)
 	if (test_ring_perf_esize(16) == -1)
 		return -1;

+	/* Test for performance gain of compression */
+	if (test_ring_perf_compression() == -1)
+		return -1;
+
 	return 0;
 }

--
2.25.1

[PATCH v5 3/4] docs: add pointer compression to the EAL guide

[Paul Szczepanek]

Documentation added in the EAL guide for the new
utility functions for pointer compression
showing example code and potential usecases.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .../prog_guide/env_abstraction_layer.rst      | 142 ++++++++++++++++++
 1 file changed, 142 insertions(+)

diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 6debf54efb..f04d032442 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -1192,3 +1192,145 @@ will not be deallocated.

 Any successful deallocation event will trigger a callback, for which user
 applications and other DPDK subsystems can register.
+
+.. _pointer_compression:
+
+Pointer Compression
+-------------------
+
+Use ``rte_ptr_compress_16()`` and ``rte_ptr_decompress_16()`` to compress and
+decompress pointers into 16-bit offsets. Use ``rte_ptr_compress_32()`` and
+``rte_ptr_decompress_32()`` to compress and decompress pointers into 32-bit
+offsets.
+
+Compression takes advantage of the fact that pointers are usually located in a
+limited memory region (like a mempool). By converting them to offsets from a
+base memory address they can be stored in fewer bytes. How many bytes are needed
+to store the offset is dictated by the memory region size and alignment of
+objects the pointers point to.
+
+For example, a pointer which is part of a 4GB memory pool can be stored as 32
+bit offset. If the pointer points to memory that is 8 bytes aligned then 3 bits
+can be dropped from the offset and a 32GB memory pool can now fit in 32 bits.
+
+For performance reasons these requirements are not enforced programmatically.
+The programmer is responsible for ensuring that the combination of distance
+from the base pointer and memory alignment allow for storing of the offset in
+the number of bits indicated by the function name (16 or 32). Start of mempool
+memory would be a good candidate for the base pointer. Otherwise any pointer
+that precedes all pointers, is close enough and has the same alignment as the
+pointers being compressed will work.
+
+.. note::
+
+    Performance gains depend on the batch size of pointers and CPU capabilities
+    such as vector extensions. It's important to measure the performance
+    increase on target hardware. A test called ``ring_perf_autotest`` in
+    ``dpdk-test`` can provide the measurements.
+
+Example usage
+~~~~~~~~~~~~~
+
+In this example we send pointers between two cores through a ring. While this
+is a realistic use case the code is simplified for demonstration purposes and
+does not have error handling.
+
+.. code-block:: c
+
+    #include <rte_launch.h>
+    #include <rte_ring.h>
+    #include <rte_ring_elem.h>
+    #include <rte_ptr_compress.h>
+
+    #define ITEMS_ARRAY_SIZE (1024)
+    #define BATCH_SIZE (128)
+    #define ALIGN_EXPONENT (3)
+    #define ITEM_ALIGN (1<<ALIGN_EXPONENT)
+    #define CORE_SEND (1)
+    #define CORE_RECV (2)
+
+    struct item {
+      int a;
+    } __rte_aligned(ITEM_ALIGN);
+
+    static struct item items[ITEMS_ARRAY_SIZE] = {0};
+    static struct rte_ring *ring = NULL;
+
+    static int
+    send_compressed(void *args)
+    {
+      struct item *ptrs_send[BATCH_SIZE] = {0};
+      unsigned int n_send = 0;
+      struct rte_ring_zc_data zcd = {0};
+
+      /* in this example we only fill the ptrs_send once and reuse */
+      for (;n_send < BATCH_SIZE; n_send++)
+        ptrs_send[n_send] = &items[n_send];
+
+      for(;;) {
+        n_send = rte_ring_enqueue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        /* compress ptrs_send into offsets */
+        rte_ptr_compress_32(items, /* base pointer */
+          ptrs_send, /* source array to be compressed */
+          zcd.ptr1, /* destination array to store offsets */
+          zcd.n1, /* how many pointers to compress */
+          ALIGN_EXPONENT /* how many bits can we drop from the offset */);
+
+        if (zcd.ptr2 != NULL)
+          rte_ptr_compress_32(items, ptrs_send + zcd.n1,
+            zcd.ptr2, n_send - zcd.n1, ALIGN_EXPONENT);
+
+        rte_ring_enqueue_zc_finish(ring, n_send);
+      }
+      return 1;
+    }
+
+    static int
+    recv_compressed(void *args)
+    {
+      struct item *ptrs_recv[BATCH_SIZE] = {0};
+      unsigned int n_recv;
+      struct rte_ring_zc_data zcd = {0};
+
+      for(;;) {
+        /* receive compressed pointers from the ring */
+        n_recv = rte_ring_dequeue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        rte_ptr_decompress_32(items, /* base pointer */
+          zcd.ptr1, /* source array to decompress */
+          ptrs_recv, /* destination array to store pointers */
+          zcd.n1, /* how many pointers to decompress */
+          ALIGN_EXPONENT /* how many bits were dropped from the offset */);
+
+        /* handle the potential secondary buffer (caused by ring boundary) */
+        if (zcd.ptr2 != NULL)
+          rte_ptr_decompress_32(items,
+            zcd.ptr2,
+            ptrs_recv + zcd.n1,
+            n_recv - zcd.n1,
+            ALIGN_EXPONENT);
+
+        rte_ring_dequeue_zc_finish(ring, n_recv);
+
+        /* ptrs_recv contains what ptrs_send contained in the other thread */
+        /* (...) */
+      }
+      return 1;
+    }
+
+    void
+    compression_example(void)
+    {
+      ring = rte_ring_create_elem(
+        "COMPR_PTRS", sizeof(uint32_t),
+        1024, rte_socket_id(),
+        RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+      rte_eal_remote_launch(send_compressed, NULL, CORE_SEND);
+      rte_eal_remote_launch(recv_compressed, NULL, CORE_RECV);
+
+      for(;;) {}
+    }
--
2.25.1

[PATCH v5 4/4] test: add unit test for ptr compression

[Paul Szczepanek]

Test compresses and decompresses pointers with various combinations
of memory regions and alignments and verifies the pointers are
recovered correctly.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 app/test/meson.build             |   1 +
 app/test/test_eal_ptr_compress.c | 108 +++++++++++++++++++++++++++++++
 2 files changed, 109 insertions(+)
 create mode 100644 app/test/test_eal_ptr_compress.c

diff --git a/app/test/meson.build b/app/test/meson.build
index 4183d66b0e..3e172b154d 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -66,6 +66,7 @@ source_file_deps = {
     'test_dmadev_api.c': ['dmadev'],
     'test_eal_flags.c': [],
     'test_eal_fs.c': [],
+    'test_eal_ptr_compress.c': [],
     'test_efd.c': ['efd', 'net'],
     'test_efd_perf.c': ['efd', 'hash'],
     'test_errno.c': [],
diff --git a/app/test/test_eal_ptr_compress.c b/app/test/test_eal_ptr_compress.c
new file mode 100644
index 0000000000..c1c9a98be7
--- /dev/null
+++ b/app/test/test_eal_ptr_compress.c
@@ -0,0 +1,108 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2014 Intel Corporation
+ */
+
+#include "test.h"
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_ptr_compress.h>
+
+#define MAX_ALIGN_EXPONENT 3
+#define PTRS_SIZE 16
+#define NUM_BASES 2
+#define NUM_REGIONS 4
+#define MAX_32BIT_REGION ((uint64_t)UINT32_MAX + 1)
+#define MAX_16BIT_REGION (UINT16_MAX + 1)
+
+static int
+test_eal_ptr_compress_params(
+	void *base,
+	uint64_t mem_sz,
+	unsigned int align_exp,
+	unsigned int num_ptrs,
+	bool use_32_bit)
+{
+	unsigned int i;
+	unsigned int align = 1 << align_exp;
+	void *ptrs[PTRS_SIZE] = {0};
+	void *ptrs_out[PTRS_SIZE] = {0};
+	uint32_t offsets32[PTRS_SIZE] = {0};
+	uint16_t offsets16[PTRS_SIZE] = {0};
+
+	for (i = 0; i < num_ptrs; i++) {
+		/* make pointers point at memory in steps of align */
+		/* alternate steps from the start and end of memory region */
+		if ((i & 1) == 1)
+			ptrs[i] = (char *)base + mem_sz - i * align;
+		else
+			ptrs[i] = (char *)base + i * align;
+	}
+
+	if (use_32_bit) {
+		rte_ptr_compress_32(base, ptrs, offsets32, num_ptrs, align_exp);
+		rte_ptr_decompress_32(base, offsets32, ptrs_out, num_ptrs,
+				align_exp);
+	} else {
+		rte_ptr_compress_16(base, ptrs, offsets16, num_ptrs, align_exp);
+		rte_ptr_decompress_16(base, offsets16, ptrs_out, num_ptrs,
+				align_exp);
+	}
+
+	TEST_ASSERT_BUFFERS_ARE_EQUAL(ptrs, ptrs_out, sizeof(void *) * num_ptrs,
+		"Decompressed pointers corrupted\nbase pointer: %p, "
+		"memory region size: %" PRIu64 ", alignment exponent: %u, "
+		"num of pointers: %u, using %s offsets",
+		base, mem_sz, align_exp, num_ptrs,
+		use_32_bit ? "32-bit" : "16-bit");
+
+	return 0;
+}
+
+static int
+test_eal_ptr_compress(void)
+{
+	unsigned int j, k, n;
+	int ret = 0;
+	void * const bases[NUM_BASES] = { (void *)0, (void *)UINT16_MAX };
+	/* maximum size for pointers aligned by consecutive powers of 2 */
+	const uint64_t region_sizes_16[NUM_REGIONS] = {
+		MAX_16BIT_REGION,
+		MAX_16BIT_REGION * 2,
+		MAX_16BIT_REGION * 4,
+		MAX_16BIT_REGION * 8,
+	};
+	const uint64_t region_sizes_32[NUM_REGIONS] = {
+		MAX_32BIT_REGION,
+		MAX_32BIT_REGION * 2,
+		MAX_32BIT_REGION * 4,
+		MAX_32BIT_REGION * 8,
+	};
+
+	for (j = 0; j < NUM_REGIONS; j++) {
+		for (k = 0; k < NUM_BASES; k++) {
+			for (n = 1; n < PTRS_SIZE; n++) {
+				ret |= test_eal_ptr_compress_params(
+					bases[k],
+					region_sizes_16[j],
+					j /* exponent of alignment */,
+					n,
+					false
+				);
+				ret |= test_eal_ptr_compress_params(
+					bases[k],
+					region_sizes_32[j],
+					j /* exponent of alignment */,
+					n,
+					true
+				);
+				if (ret != 0)
+					return ret;
+			}
+		}
+	}
+
+	return ret;
+}
+
+REGISTER_FAST_TEST(eal_ptr_compress_autotest, true, true, test_eal_ptr_compress);
--
2.25.1

Re: [PATCH v5 0/4] add pointer compression API

[Paul Szczepanek]

For some reason your email is not visible to me, even though it's in the 
archive.

On 02/11/202416:32,Konstantin Ananyev konstantin.v.ananyev  wrote:

> From one side the code itself is very small and straightforward, > from other side - it is not clear to me what is intended usage for it
> within DPDK and it's applianances?
> Konstantin

The intended usage is explained in the cover email (see below) and demonstrated
in the test supplied in the following patch - when sending arrays of pointers
between cores as it happens in a forwarding example.

On 01/11/2023 18:12, Paul Szczepanek wrote:

> This patchset is proposing adding a new EAL header with utility functions
> that allow compression of arrays of pointers.
>
> When passing caches full of pointers between threads, memory containing
> the pointers is copied multiple times which is especially costly between
> cores. A compression method will allow us to shrink the memory size
> copied.
>
> The compression takes advantage of the fact that pointers are usually
> located in a limited memory region (like a mempool). We can compress them
> by converting them to offsets from a base memory address.
>
> Offsets can be stored in fewer bytes (dictated by the memory region size
> and alignment of the pointer). For example: an 8 byte aligned pointer
> which is part of a 32GB memory pool can be stored in 4 bytes. The API is
> very generic and does not assume mempool pointers, any pointer can be
> passed in.
>
> Compression is based on few and fast operations and especially with vector
> instructions leveraged creates minimal overhead.
>
> The API accepts and returns arrays because the overhead means it only is
> worth it when done in bulk.
>
> Test is added that shows potential performance gain from compression. In
> this test an array of pointers is passed through a ring between two cores.
> It shows the gain which is dependent on the bulk operation size. In this
> synthetic test run on ampere altra a substantial (up to 25%) performance
> gain is seen if done in bulk size larger than 32. At 32 it breaks even and
> lower sizes create a small (less than 5%) slowdown due to overhead.
>
> In a more realistic mock application running the l3 forwarding dpdk
> example that works in pipeline mode on two cores this translated into a
> ~5% throughput increase on an ampere altra.
>
> v2:
> * addressed review comments (style, explanations and typos)
> * lowered bulk iterations closer to original numbers to keep runtime short
> * fixed pointer size warning on 32-bit arch
> v3:
> * added 16-bit versions of compression functions and tests
> * added documentation of these new utility functions in the EAL guide
> v4:
> * added unit test
> * fix bug in NEON implementation of 32-bit decompress
> v5:
> * disable NEON and SVE implementation on AARCH32 due to wrong pointer size
>
> Paul Szczepanek (4):
>    eal: add pointer compression functions
>    test: add pointer compress tests to ring perf test
>    docs: add pointer compression to the EAL guide
>    test: add unit test for ptr compression
>
>   .mailmap                                      |   1 +
>   app/test/meson.build                          |   1 +
>   app/test/test_eal_ptr_compress.c              | 108 ++++++
>   app/test/test_ring.h                          |  94 ++++-
>   app/test/test_ring_perf.c                     | 354 ++++++++++++------
>   .../prog_guide/env_abstraction_layer.rst      | 142 +++++++
>   lib/eal/include/meson.build                   |   1 +
>   lib/eal/include/rte_ptr_compress.h            | 266 +++++++++++++
>   8 files changed, 843 insertions(+), 124 deletions(-)
>   create mode 100644 app/test/test_eal_ptr_compress.c
>   create mode 100644 lib/eal/include/rte_ptr_compress.h
>
> --
> 2.25.1
>

RE: [PATCH v5 0/4] add pointer compression API

[Konstantin Ananyev]

> For some reason your email is not visible to me, even though it's in the
> archive.

No worries.

> 
> On 02/11/202416:32,Konstantin Ananyev konstantin.v.ananyev  wrote:
> 
> > From one side the code itself is very small and straightforward, > from other side - it is not clear to me what is intended usage for it
> > within DPDK and it's applianances?
> > Konstantin
> 
> The intended usage is explained in the cover email (see below) and demonstrated
> in the test supplied in the following patch - when sending arrays of pointers
> between cores as it happens in a forwarding example.

Yes, I saw that. The thing is that test is a 'synthetic' one.
My question was about how do you expect people to use it in more realistic scenarios?
Let say user has a bunch of mbuf pointers, possibly from different mempools.
How he can use this API: how to deduce the base pointer for all of them and what to
do if it can't be done?
 
> On 01/11/2023 18:12, Paul Szczepanek wrote:
> 
> > This patchset is proposing adding a new EAL header with utility functions
> > that allow compression of arrays of pointers.
> >
> > When passing caches full of pointers between threads, memory containing
> > the pointers is copied multiple times which is especially costly between
> > cores. A compression method will allow us to shrink the memory size
> > copied.
> >
> > The compression takes advantage of the fact that pointers are usually
> > located in a limited memory region (like a mempool). We can compress them
> > by converting them to offsets from a base memory address.
> >
> > Offsets can be stored in fewer bytes (dictated by the memory region size
> > and alignment of the pointer). For example: an 8 byte aligned pointer
> > which is part of a 32GB memory pool can be stored in 4 bytes. The API is
> > very generic and does not assume mempool pointers, any pointer can be
> > passed in.
> >
> > Compression is based on few and fast operations and especially with vector
> > instructions leveraged creates minimal overhead.
> >
> > The API accepts and returns arrays because the overhead means it only is
> > worth it when done in bulk.
> >
> > Test is added that shows potential performance gain from compression. In
> > this test an array of pointers is passed through a ring between two cores.
> > It shows the gain which is dependent on the bulk operation size. In this
> > synthetic test run on ampere altra a substantial (up to 25%) performance
> > gain is seen if done in bulk size larger than 32. At 32 it breaks even and
> > lower sizes create a small (less than 5%) slowdown due to overhead.
> >
> > In a more realistic mock application running the l3 forwarding dpdk
> > example that works in pipeline mode on two cores this translated into a
> > ~5% throughput increase on an ampere altra.
> >
> > v2:
> > * addressed review comments (style, explanations and typos)
> > * lowered bulk iterations closer to original numbers to keep runtime short
> > * fixed pointer size warning on 32-bit arch
> > v3:
> > * added 16-bit versions of compression functions and tests
> > * added documentation of these new utility functions in the EAL guide
> > v4:
> > * added unit test
> > * fix bug in NEON implementation of 32-bit decompress
> > v5:
> > * disable NEON and SVE implementation on AARCH32 due to wrong pointer size
> >
> > Paul Szczepanek (4):
> >    eal: add pointer compression functions
> >    test: add pointer compress tests to ring perf test
> >    docs: add pointer compression to the EAL guide
> >    test: add unit test for ptr compression
> >
> >   .mailmap                                      |   1 +
> >   app/test/meson.build                          |   1 +
> >   app/test/test_eal_ptr_compress.c              | 108 ++++++
> >   app/test/test_ring.h                          |  94 ++++-
> >   app/test/test_ring_perf.c                     | 354 ++++++++++++------
> >   .../prog_guide/env_abstraction_layer.rst      | 142 +++++++
> >   lib/eal/include/meson.build                   |   1 +
> >   lib/eal/include/rte_ptr_compress.h            | 266 +++++++++++++
> >   8 files changed, 843 insertions(+), 124 deletions(-)
> >   create mode 100644 app/test/test_eal_ptr_compress.c
> >   create mode 100644 lib/eal/include/rte_ptr_compress.h
> >
> > --
> > 2.25.1
> >

RE: [PATCH v5 0/4] add pointer compression API

[Morten Brørup]

> From: Konstantin Ananyev [mailto:konstantin.ananyev@huawei.com]
> Sent: Thursday, 22 February 2024 17.16
> 
> > For some reason your email is not visible to me, even though it's in the
> > archive.
> 
> No worries.
> 
> >
> > On 02/11/202416:32,Konstantin Ananyev konstantin.v.ananyev  wrote:
> >
> > > From one side the code itself is very small and straightforward, > from
> other side - it is not clear to me what is intended usage for it
> > > within DPDK and it's applianances?
> > > Konstantin
> >
> > The intended usage is explained in the cover email (see below) and
> demonstrated
> > in the test supplied in the following patch - when sending arrays of
> pointers
> > between cores as it happens in a forwarding example.
> 
> Yes, I saw that. The thing is that test is a 'synthetic' one.
> My question was about how do you expect people to use it in more realistic
> scenarios?
> Let say user has a bunch of mbuf pointers, possibly from different mempools.
> How he can use this API: how to deduce the base pointer for all of them and
> what to
> do if it can't be done?

I share Konstantin's concerns with this feature.

If we want to compress mbuf pointers in applications with a few mbuf pools, e.g. an mbuf pool per CPU socket, the compression algorithm would be different.

I would like to add:
If we want to offer optimizations specifically for applications with a single mbuf pool, I think it should be considered in a system-wide context to determine if performance could be improved in more areas.
E.g. removing the pool field from the rte_mbuf structure might free up space to move hot fields from the second cache line to the first, so the second cache line rarely needs to be touched. (As an alternative to removing the pool field, it could be moved to the second cache line, only to be used if the global "single mbuf pool" is NULL.)

On the other hand, I agree that pointer compression can be useful for some applications, so we should accept it.

However, pointer compression has nothing to do with the underlying hardware or operating system, so it does not belong in the EAL (which is already too bloated). It should be a separate library.

> 
> > On 01/11/2023 18:12, Paul Szczepanek wrote:
> >
> > > This patchset is proposing adding a new EAL header with utility functions
> > > that allow compression of arrays of pointers.
> > >
> > > When passing caches full of pointers between threads, memory containing
> > > the pointers is copied multiple times which is especially costly between
> > > cores. A compression method will allow us to shrink the memory size
> > > copied.
> > >
> > > The compression takes advantage of the fact that pointers are usually
> > > located in a limited memory region (like a mempool). We can compress them
> > > by converting them to offsets from a base memory address.
> > >
> > > Offsets can be stored in fewer bytes (dictated by the memory region size
> > > and alignment of the pointer). For example: an 8 byte aligned pointer
> > > which is part of a 32GB memory pool can be stored in 4 bytes. The API is
> > > very generic and does not assume mempool pointers, any pointer can be
> > > passed in.
> > >
> > > Compression is based on few and fast operations and especially with vector
> > > instructions leveraged creates minimal overhead.
> > >
> > > The API accepts and returns arrays because the overhead means it only is
> > > worth it when done in bulk.
> > >
> > > Test is added that shows potential performance gain from compression. In
> > > this test an array of pointers is passed through a ring between two cores.
> > > It shows the gain which is dependent on the bulk operation size. In this
> > > synthetic test run on ampere altra a substantial (up to 25%) performance
> > > gain is seen if done in bulk size larger than 32. At 32 it breaks even and
> > > lower sizes create a small (less than 5%) slowdown due to overhead.
> > >
> > > In a more realistic mock application running the l3 forwarding dpdk
> > > example that works in pipeline mode on two cores this translated into a
> > > ~5% throughput increase on an ampere altra.

Which burst size was used to achieve this ~5% throughput increase?

> > >
> > > v2:
> > > * addressed review comments (style, explanations and typos)
> > > * lowered bulk iterations closer to original numbers to keep runtime short
> > > * fixed pointer size warning on 32-bit arch
> > > v3:
> > > * added 16-bit versions of compression functions and tests
> > > * added documentation of these new utility functions in the EAL guide
> > > v4:
> > > * added unit test
> > > * fix bug in NEON implementation of 32-bit decompress
> > > v5:
> > > * disable NEON and SVE implementation on AARCH32 due to wrong pointer size
> > >
> > > Paul Szczepanek (4):
> > >    eal: add pointer compression functions
> > >    test: add pointer compress tests to ring perf test
> > >    docs: add pointer compression to the EAL guide
> > >    test: add unit test for ptr compression
> > >
> > >   .mailmap                                      |   1 +
> > >   app/test/meson.build                          |   1 +
> > >   app/test/test_eal_ptr_compress.c              | 108 ++++++
> > >   app/test/test_ring.h                          |  94 ++++-
> > >   app/test/test_ring_perf.c                     | 354 ++++++++++++------
> > >   .../prog_guide/env_abstraction_layer.rst      | 142 +++++++
> > >   lib/eal/include/meson.build                   |   1 +
> > >   lib/eal/include/rte_ptr_compress.h            | 266 +++++++++++++
> > >   8 files changed, 843 insertions(+), 124 deletions(-)
> > >   create mode 100644 app/test/test_eal_ptr_compress.c
> > >   create mode 100644 lib/eal/include/rte_ptr_compress.h
> > >
> > > --
> > > 2.25.1
> > >

Re: [PATCH v5 0/4] add pointer compression API

[Patrick Robb]

[-- Attachment #1: Type: text/plain, Size: 6507 bytes --]

The Community CI Testing Lab had an infra failure this morning and some
patches including yours were affected with false failures. The issue is now
resolved and we are rerunning the tests in question for all patches
submitted today.

On Fri, Mar 1, 2024 at 6:16 AM Morten Brørup <mb@smartsharesystems.com>
wrote:

> > From: Konstantin Ananyev [mailto:konstantin.ananyev@huawei.com]
> > Sent: Thursday, 22 February 2024 17.16
> >
> > > For some reason your email is not visible to me, even though it's in
> the
> > > archive.
> >
> > No worries.
> >
> > >
> > > On 02/11/202416:32,Konstantin Ananyev konstantin.v.ananyev  wrote:
> > >
> > > > From one side the code itself is very small and straightforward, >
> from
> > other side - it is not clear to me what is intended usage for it
> > > > within DPDK and it's applianances?
> > > > Konstantin
> > >
> > > The intended usage is explained in the cover email (see below) and
> > demonstrated
> > > in the test supplied in the following patch - when sending arrays of
> > pointers
> > > between cores as it happens in a forwarding example.
> >
> > Yes, I saw that. The thing is that test is a 'synthetic' one.
> > My question was about how do you expect people to use it in more
> realistic
> > scenarios?
> > Let say user has a bunch of mbuf pointers, possibly from different
> mempools.
> > How he can use this API: how to deduce the base pointer for all of them
> and
> > what to
> > do if it can't be done?
>
> I share Konstantin's concerns with this feature.
>
> If we want to compress mbuf pointers in applications with a few mbuf
> pools, e.g. an mbuf pool per CPU socket, the compression algorithm would be
> different.
>
> I would like to add:
> If we want to offer optimizations specifically for applications with a
> single mbuf pool, I think it should be considered in a system-wide context
> to determine if performance could be improved in more areas.
> E.g. removing the pool field from the rte_mbuf structure might free up
> space to move hot fields from the second cache line to the first, so the
> second cache line rarely needs to be touched. (As an alternative to
> removing the pool field, it could be moved to the second cache line, only
> to be used if the global "single mbuf pool" is NULL.)
>
> On the other hand, I agree that pointer compression can be useful for some
> applications, so we should accept it.
>
> However, pointer compression has nothing to do with the underlying
> hardware or operating system, so it does not belong in the EAL (which is
> already too bloated). It should be a separate library.
>
> >
> > > On 01/11/2023 18:12, Paul Szczepanek wrote:
> > >
> > > > This patchset is proposing adding a new EAL header with utility
> functions
> > > > that allow compression of arrays of pointers.
> > > >
> > > > When passing caches full of pointers between threads, memory
> containing
> > > > the pointers is copied multiple times which is especially costly
> between
> > > > cores. A compression method will allow us to shrink the memory size
> > > > copied.
> > > >
> > > > The compression takes advantage of the fact that pointers are usually
> > > > located in a limited memory region (like a mempool). We can compress
> them
> > > > by converting them to offsets from a base memory address.
> > > >
> > > > Offsets can be stored in fewer bytes (dictated by the memory region
> size
> > > > and alignment of the pointer). For example: an 8 byte aligned pointer
> > > > which is part of a 32GB memory pool can be stored in 4 bytes. The
> API is
> > > > very generic and does not assume mempool pointers, any pointer can be
> > > > passed in.
> > > >
> > > > Compression is based on few and fast operations and especially with
> vector
> > > > instructions leveraged creates minimal overhead.
> > > >
> > > > The API accepts and returns arrays because the overhead means it
> only is
> > > > worth it when done in bulk.
> > > >
> > > > Test is added that shows potential performance gain from
> compression. In
> > > > this test an array of pointers is passed through a ring between two
> cores.
> > > > It shows the gain which is dependent on the bulk operation size. In
> this
> > > > synthetic test run on ampere altra a substantial (up to 25%)
> performance
> > > > gain is seen if done in bulk size larger than 32. At 32 it breaks
> even and
> > > > lower sizes create a small (less than 5%) slowdown due to overhead.
> > > >
> > > > In a more realistic mock application running the l3 forwarding dpdk
> > > > example that works in pipeline mode on two cores this translated
> into a
> > > > ~5% throughput increase on an ampere altra.
>
> Which burst size was used to achieve this ~5% throughput increase?
>
> > > >
> > > > v2:
> > > > * addressed review comments (style, explanations and typos)
> > > > * lowered bulk iterations closer to original numbers to keep runtime
> short
> > > > * fixed pointer size warning on 32-bit arch
> > > > v3:
> > > > * added 16-bit versions of compression functions and tests
> > > > * added documentation of these new utility functions in the EAL guide
> > > > v4:
> > > > * added unit test
> > > > * fix bug in NEON implementation of 32-bit decompress
> > > > v5:
> > > > * disable NEON and SVE implementation on AARCH32 due to wrong
> pointer size
> > > >
> > > > Paul Szczepanek (4):
> > > >    eal: add pointer compression functions
> > > >    test: add pointer compress tests to ring perf test
> > > >    docs: add pointer compression to the EAL guide
> > > >    test: add unit test for ptr compression
> > > >
> > > >   .mailmap                                      |   1 +
> > > >   app/test/meson.build                          |   1 +
> > > >   app/test/test_eal_ptr_compress.c              | 108 ++++++
> > > >   app/test/test_ring.h                          |  94 ++++-
> > > >   app/test/test_ring_perf.c                     | 354
> ++++++++++++------
> > > >   .../prog_guide/env_abstraction_layer.rst      | 142 +++++++
> > > >   lib/eal/include/meson.build                   |   1 +
> > > >   lib/eal/include/rte_ptr_compress.h            | 266 +++++++++++++
> > > >   8 files changed, 843 insertions(+), 124 deletions(-)
> > > >   create mode 100644 app/test/test_eal_ptr_compress.c
> > > >   create mode 100644 lib/eal/include/rte_ptr_compress.h
> > > >
> > > > --
> > > > 2.25.1
> > > >
>

[-- Attachment #2: Type: text/html, Size: 7962 bytes --]

Re: [PATCH v5 0/4] add pointer compression API

[Honnappa Nagarahalli]

> On Mar 1, 2024, at 5:16 AM, Morten Brørup <mb@smartsharesystems.com> wrote:
> 
>> From: Konstantin Ananyev [mailto:konstantin.ananyev@huawei.com]
>> Sent: Thursday, 22 February 2024 17.16
>> 
>>> For some reason your email is not visible to me, even though it's in the
>>> archive.
>> 
>> No worries.
>> 
>>> 
>>> On 02/11/202416:32,Konstantin Ananyev konstantin.v.ananyev  wrote:
>>> 
>>>> From one side the code itself is very small and straightforward, > from
>> other side - it is not clear to me what is intended usage for it
>>>> within DPDK and it's applianances?
>>>> Konstantin
>>> 
>>> The intended usage is explained in the cover email (see below) and
>> demonstrated
>>> in the test supplied in the following patch - when sending arrays of
>> pointers
>>> between cores as it happens in a forwarding example.
>> 
>> Yes, I saw that. The thing is that test is a 'synthetic' one.
>> My question was about how do you expect people to use it in more realistic
>> scenarios?
>> Let say user has a bunch of mbuf pointers, possibly from different mempools.
>> How he can use this API: how to deduce the base pointer for all of them and
>> what to
>> do if it can't be done?
> 
> I share Konstantin's concerns with this feature.
> 
> If we want to compress mbuf pointers in applications with a few mbuf pools, e.g. an mbuf pool per CPU socket, the compression algorithm would be different.
This feature is targeted for pipeline mode of applications. We see many customers using pipeline mode. This feature helps in reducing the cost of transferring the packets between cores by reducing the copies involved.
For an application with multiple pools, it depends on how the applications are using multiple pools. But, if there is a bunch of packets belonging to multiple mempools, compressing those mbufs may not be possible. But if those mbufs are grouped per mempool and are transferred on different queues, then it is possible. Hence the APIs are implemented very generically.

> 
> I would like to add:
> If we want to offer optimizations specifically for applications with a single mbuf pool, I think it should be considered in a system-wide context to determine if performance could be improved in more areas.
> E.g. removing the pool field from the rte_mbuf structure might free up space to move hot fields from the second cache line to the first, so the second cache line rarely needs to be touched. (As an alternative to removing the pool field, it could be moved to the second cache line, only to be used if the global "single mbuf pool" is NULL.)
Agree on this. The feedback I have received is on similar lines, many are using simple features. I also received feedback that 90% of the applications use less than 4GB of memory for mbuf and burst sizes are up to 256.

> 
> On the other hand, I agree that pointer compression can be useful for some applications, so we should accept it.
> 
> However, pointer compression has nothing to do with the underlying hardware or operating system, so it does not belong in the EAL (which is already too bloated). It should be a separate library.
Yes, this is generic (though there is SIMD code). We could move it out of EAL.

> 
>> 
>>> On 01/11/2023 18:12, Paul Szczepanek wrote:
>>> 
>>>> This patchset is proposing adding a new EAL header with utility functions
>>>> that allow compression of arrays of pointers.
>>>> 
>>>> When passing caches full of pointers between threads, memory containing
>>>> the pointers is copied multiple times which is especially costly between
>>>> cores. A compression method will allow us to shrink the memory size
>>>> copied.
>>>> 
>>>> The compression takes advantage of the fact that pointers are usually
>>>> located in a limited memory region (like a mempool). We can compress them
>>>> by converting them to offsets from a base memory address.
>>>> 
>>>> Offsets can be stored in fewer bytes (dictated by the memory region size
>>>> and alignment of the pointer). For example: an 8 byte aligned pointer
>>>> which is part of a 32GB memory pool can be stored in 4 bytes. The API is
>>>> very generic and does not assume mempool pointers, any pointer can be
>>>> passed in.
>>>> 
>>>> Compression is based on few and fast operations and especially with vector
>>>> instructions leveraged creates minimal overhead.
>>>> 
>>>> The API accepts and returns arrays because the overhead means it only is
>>>> worth it when done in bulk.
>>>> 
>>>> Test is added that shows potential performance gain from compression. In
>>>> this test an array of pointers is passed through a ring between two cores.
>>>> It shows the gain which is dependent on the bulk operation size. In this
>>>> synthetic test run on ampere altra a substantial (up to 25%) performance
>>>> gain is seen if done in bulk size larger than 32. At 32 it breaks even and
>>>> lower sizes create a small (less than 5%) slowdown due to overhead.
>>>> 
>>>> In a more realistic mock application running the l3 forwarding dpdk
>>>> example that works in pipeline mode on two cores this translated into a
>>>> ~5% throughput increase on an ampere altra.
> 
> Which burst size was used to achieve this ~5% throughput increase?
This is the stock L3fwd application which is split into 2 stages: RX, L3fwd, TX. The default burst size 32 is used.

> 
>>>> 
>>>> v2:
>>>> * addressed review comments (style, explanations and typos)
>>>> * lowered bulk iterations closer to original numbers to keep runtime short
>>>> * fixed pointer size warning on 32-bit arch
>>>> v3:
>>>> * added 16-bit versions of compression functions and tests
>>>> * added documentation of these new utility functions in the EAL guide
>>>> v4:
>>>> * added unit test
>>>> * fix bug in NEON implementation of 32-bit decompress
>>>> v5:
>>>> * disable NEON and SVE implementation on AARCH32 due to wrong pointer size
>>>> 
>>>> Paul Szczepanek (4):
>>>>   eal: add pointer compression functions
>>>>   test: add pointer compress tests to ring perf test
>>>>   docs: add pointer compression to the EAL guide
>>>>   test: add unit test for ptr compression
>>>> 
>>>>  .mailmap                                      |   1 +
>>>>  app/test/meson.build                          |   1 +
>>>>  app/test/test_eal_ptr_compress.c              | 108 ++++++
>>>>  app/test/test_ring.h                          |  94 ++++-
>>>>  app/test/test_ring_perf.c                     | 354 ++++++++++++------
>>>>  .../prog_guide/env_abstraction_layer.rst      | 142 +++++++
>>>>  lib/eal/include/meson.build                   |   1 +
>>>>  lib/eal/include/rte_ptr_compress.h            | 266 +++++++++++++
>>>>  8 files changed, 843 insertions(+), 124 deletions(-)
>>>>  create mode 100644 app/test/test_eal_ptr_compress.c
>>>>  create mode 100644 lib/eal/include/rte_ptr_compress.h
>>>> 
>>>> --
>>>> 2.25.1
>>>> 

RE: [PATCH v5 0/4] add pointer compression API

[Morten Brørup]

> From: Honnappa Nagarahalli [mailto:Honnappa.Nagarahalli@arm.com]
> Sent: Friday, 1 March 2024 20.57
> 
> > On Mar 1, 2024, at 5:16 AM, Morten Brørup <mb@smartsharesystems.com>
> wrote:
> >
> >> From: Konstantin Ananyev [mailto:konstantin.ananyev@huawei.com]
> >> Sent: Thursday, 22 February 2024 17.16
> >>
> >>> For some reason your email is not visible to me, even though it's in
> the
> >>> archive.
> >>
> >> No worries.
> >>
> >>>
> >>> On 02/11/202416:32,Konstantin Ananyev konstantin.v.ananyev  wrote:
> >>>
> >>>> From one side the code itself is very small and straightforward, >
> from
> >> other side - it is not clear to me what is intended usage for it
> >>>> within DPDK and it's applianances?
> >>>> Konstantin
> >>>
> >>> The intended usage is explained in the cover email (see below) and
> >> demonstrated
> >>> in the test supplied in the following patch - when sending arrays of
> >> pointers
> >>> between cores as it happens in a forwarding example.
> >>
> >> Yes, I saw that. The thing is that test is a 'synthetic' one.
> >> My question was about how do you expect people to use it in more
> realistic
> >> scenarios?
> >> Let say user has a bunch of mbuf pointers, possibly from different
> mempools.
> >> How he can use this API: how to deduce the base pointer for all of
> them and
> >> what to
> >> do if it can't be done?
> >
> > I share Konstantin's concerns with this feature.
> >
> > If we want to compress mbuf pointers in applications with a few mbuf
> pools, e.g. an mbuf pool per CPU socket, the compression algorithm would
> be different.
> This feature is targeted for pipeline mode of applications. We see many
> customers using pipeline mode. This feature helps in reducing the cost
> of transferring the packets between cores by reducing the copies
> involved.

OK. I agree this is a very common use case, worth optimizing for.

> For an application with multiple pools, it depends on how the
> applications are using multiple pools. But, if there is a bunch of
> packets belonging to multiple mempools, compressing those mbufs may not
> be possible. But if those mbufs are grouped per mempool and are
> transferred on different queues, then it is possible. Hence the APIs are
> implemented very generically.

OK.

<feature creep>
And for a possible future extension:
If there are very few mbuf pools, such as 2 or 4, it might be possible to develop similar functions to efficiently compress/decompress pointers in a shared queue. E.g. the highest bits could identify the pool, and the lowest bits could identify the pointer offset (with bit shift) in that pool. Or if the pools are less than 4 GB each, the lowest bits could identify the pool, and be masked away for getting the offset (no bit shift), taking advantage of lowest bits of the pointer address always being zero anyway.
I am mentioning this, so it can be taken into consideration when designing the pointer compression library and its API. I don't expect it to be implemented at this time. Also, it might not lead to any changes of the already proposed pointer compression API - just give it a few thoughts.
</feature creep>

+1 for the simplicity of the functions and the API in this patch.
E.g. the bit_shift is most likely known constant at build time, so inlining allows the compiler to optimize for this. In many use cases, it might be 1, and thus optimized away.

> 
> >
> > I would like to add:
> > If we want to offer optimizations specifically for applications with a
> single mbuf pool, I think it should be considered in a system-wide
> context to determine if performance could be improved in more areas.
> > E.g. removing the pool field from the rte_mbuf structure might free up
> space to move hot fields from the second cache line to the first, so the
> second cache line rarely needs to be touched. (As an alternative to
> removing the pool field, it could be moved to the second cache line,
> only to be used if the global "single mbuf pool" is NULL.)
> Agree on this. The feedback I have received is on similar lines, many
> are using simple features. I also received feedback that 90% of the
> applications use less than 4GB of memory for mbuf and burst sizes are up
> to 256.

Interesting.
Keeping the most common use cases in mind is important for steering DPDK in the right direction as it evolves.

If a very large percentage of use cases use one mbuf pool of less than 4 GB, we should seriously consider the broader opportunity for optimizing by generally referencing mbufs by an uint32_t pointer offset (no bit shifting) instead of by pointers.

> 
> >
> > On the other hand, I agree that pointer compression can be useful for
> some applications, so we should accept it.
> >
> > However, pointer compression has nothing to do with the underlying
> hardware or operating system, so it does not belong in the EAL (which is
> already too bloated). It should be a separate library.
> Yes, this is generic (though there is SIMD code). We could move it out
> of EAL.

Thank you.

I think that a misconception that arch specific optimizations (such as SIMD code) required stuff to go into EAL has been prevailing, and this misconception is a main reason why EAL has become so bloated.
Moving features like pointer compression out of EAL, thereby showing alternative design patterns for code containing arch specific optimizations, will help eliminate that misconception.

> 
> >
> >>
> >>> On 01/11/2023 18:12, Paul Szczepanek wrote:
> >>>
> >>>> This patchset is proposing adding a new EAL header with utility
> functions
> >>>> that allow compression of arrays of pointers.
> >>>>
> >>>> When passing caches full of pointers between threads, memory
> containing
> >>>> the pointers is copied multiple times which is especially costly
> between
> >>>> cores. A compression method will allow us to shrink the memory size
> >>>> copied.
> >>>>
> >>>> The compression takes advantage of the fact that pointers are
> usually
> >>>> located in a limited memory region (like a mempool). We can
> compress them
> >>>> by converting them to offsets from a base memory address.
> >>>>
> >>>> Offsets can be stored in fewer bytes (dictated by the memory region
> size
> >>>> and alignment of the pointer). For example: an 8 byte aligned
> pointer
> >>>> which is part of a 32GB memory pool can be stored in 4 bytes. The
> API is
> >>>> very generic and does not assume mempool pointers, any pointer can
> be
> >>>> passed in.
> >>>>
> >>>> Compression is based on few and fast operations and especially with
> vector
> >>>> instructions leveraged creates minimal overhead.
> >>>>
> >>>> The API accepts and returns arrays because the overhead means it
> only is
> >>>> worth it when done in bulk.
> >>>>
> >>>> Test is added that shows potential performance gain from
> compression. In
> >>>> this test an array of pointers is passed through a ring between two
> cores.
> >>>> It shows the gain which is dependent on the bulk operation size. In
> this
> >>>> synthetic test run on ampere altra a substantial (up to 25%)
> performance
> >>>> gain is seen if done in bulk size larger than 32. At 32 it breaks
> even and
> >>>> lower sizes create a small (less than 5%) slowdown due to overhead.
> >>>>
> >>>> In a more realistic mock application running the l3 forwarding dpdk
> >>>> example that works in pipeline mode on two cores this translated
> into a
> >>>> ~5% throughput increase on an ampere altra.
> >
> > Which burst size was used to achieve this ~5% throughput increase?
> This is the stock L3fwd application which is split into 2 stages: RX,
> L3fwd, TX. The default burst size 32 is used.

Impressive.
It proves the point that synthetic tests often are too simple to show the benefits of optimizations for reducing cache misses.

Re: [PATCH v5 0/4] add pointer compression API

[Paul Szczepanek]

On 02/03/2024 10:33, Morten Brørup wrote:
> I think that a misconception that arch specific optimizations (such as SIMD code) required stuff to go into EAL has been prevailing, and this misconception is a main reason why EAL has become so bloated.
> Moving features like pointer compression out of EAL, thereby showing alternative design patterns for code containing arch specific optimizations, will help eliminate that misconception.

I have a patch ready that moves the ptr compress into its own library 
but I must first land this patch:
https://patches.dpdk.org/project/dpdk/patch/20240306221709.166722-2-paul.szczepanek@arm.com/
which is required to have header only libraries - otherwise errors stop 
the build.

Re: [PATCH v5 0/4] add pointer compression API

[Honnappa Nagarahalli]

> On Mar 6, 2024, at 4:31 PM, Paul Szczepanek <Paul.Szczepanek@arm.com> wrote:
> 
> On 02/03/2024 10:33, Morten Brørup wrote:
>> I think that a misconception that arch specific optimizations (such as SIMD code) required stuff to go into EAL has been prevailing, and this misconception is a main reason why EAL has become so bloated.
>> Moving features like pointer compression out of EAL, thereby showing alternative design patterns for code containing arch specific optimizations, will help eliminate that misconception.
> 
> I have a patch ready that moves the ptr compress into its own library but I must first land this patch:
> https://patches.dpdk.org/project/dpdk/patch/20240306221709.166722-2-paul.szczepanek@arm.com/
> which is required to have header only libraries - otherwise errors stop the build.
You can add dependencies to your patch. You do not have to wait for the patch to be merged. I believe the CI takes care of the dependencies as well. Please check: https://doc.dpdk.org/guides/contributing/patches.html#patch-dependencies

RE: [PATCH v5 0/4] add pointer compression API

[Konstantin Ananyev]

> > On Mar 1, 2024, at 5:16 AM, Morten Brørup <mb@smartsharesystems.com> wrote:
> >
> >> From: Konstantin Ananyev [mailto:konstantin.ananyev@huawei.com]
> >> Sent: Thursday, 22 February 2024 17.16
> >>
> >>> For some reason your email is not visible to me, even though it's in the
> >>> archive.
> >>
> >> No worries.
> >>
> >>>
> >>> On 02/11/202416:32,Konstantin Ananyev konstantin.v.ananyev  wrote:
> >>>
> >>>> From one side the code itself is very small and straightforward, > from
> >> other side - it is not clear to me what is intended usage for it
> >>>> within DPDK and it's applianances?
> >>>> Konstantin
> >>>
> >>> The intended usage is explained in the cover email (see below) and
> >> demonstrated
> >>> in the test supplied in the following patch - when sending arrays of
> >> pointers
> >>> between cores as it happens in a forwarding example.
> >>
> >> Yes, I saw that. The thing is that test is a 'synthetic' one.
> >> My question was about how do you expect people to use it in more realistic
> >> scenarios?
> >> Let say user has a bunch of mbuf pointers, possibly from different mempools.
> >> How he can use this API: how to deduce the base pointer for all of them and
> >> what to
> >> do if it can't be done?
> >
> > I share Konstantin's concerns with this feature.
> >
> > If we want to compress mbuf pointers in applications with a few mbuf pools, e.g. an mbuf pool per CPU socket, the compression
> algorithm would be different.
> This feature is targeted for pipeline mode of applications. We see many customers using pipeline mode. This feature helps in reducing
> the cost of transferring the packets between cores by reducing the copies involved.

I do understand the intention, and I am not arguing about usefulness of the pipeline model. 
My point is you are introducing new API: compress/decompress pointers,
but don't provide (or even describe) any proper way for the developer to use it in a safe and predictable manner.
Which from my perspective make it nearly useless and misleading.

> For an application with multiple pools, it depends on how the applications are using multiple pools. But, if there is a bunch of packets
> belonging to multiple mempools, compressing those mbufs may not be possible. But if those mbufs are grouped per mempool and
> are transferred on different queues, then it is possible. Hence the APIs are implemented very generically.

Ok, let's consider even more simplistic scenario - all pointers belong to one mempool.
AFAIK, even one mempool can contain elements from different memzones,
and these memzones are not guaranteed to have consecutive VAs.
So even one mempool, with total size <=4GB can contain elements with distances between them more than 4GB. 
Now let say at startup user created a mempool, how he can determine programmatically
can he apply your compress API safely on it or not?
I presume that if you are serious about this API usage, then such ability has to be provided.
Something like:

int compress_pointer_deduce_mempool_base(const struct rte_memepool *mp[],
	uint32_t nb_mp, uint32_t compress_size, uintptr_t *base_ptr);

Or probably even more generic one:

struct mem_buf {uintptr_t base, size_t len;}; 
int compress_pointer_deduce_base(const struct mem_buf *mem_buf[],
	uint32_t nb_membuf, uint32_t compress_size, uintptr_t *base_ptr);

Even with these functions in-place, user has to be extra careful:
 - he can't add new memory chunks to these mempools (or he'll need to re-calcualte the new base_ptr)
 - he needs to make sure that pointers from only these mempools will be used by compress/decompress.
But at least it provides some ability to use this feature in real apps.

With such API in place it should be possible to make the auto-test more realistic:
- allocate mempool 
- deduce base_pointer
- then we can have a loop with producer/consumer to mimic realistic workload.
    As an example:
     producer(s):  mempool_alloc(); <fill mbuf with some values>; ring_enqueue();  
     consumer(s): ring_dequeue(); <read_and_check_mbuf_data>; free_mbuf();
- free mempool

Or probably you can go even further: take some existing pipeline sample app and make it use compress/decompress API.
That will provide people with some ability to test it and measure it's perf impact.
Again, it will provide an example of the amount of changes required to enable it.
My speculation here that majority of users will find the effort too big, 
while the gain way too limited and fragile.
But at least, there would be some realistic reference point for it and users can decide themselves is it worth it or not. 

> >
> > I would like to add:
> > If we want to offer optimizations specifically for applications with a single mbuf pool, I think it should be considered in a system-wide
> context to determine if performance could be improved in more areas.
> > E.g. removing the pool field from the rte_mbuf structure might free up space to move hot fields from the second cache line to the
> first, so the second cache line rarely needs to be touched. (As an alternative to removing the pool field, it could be moved to the
> second cache line, only to be used if the global "single mbuf pool" is NULL.)
> Agree on this. The feedback I have received is on similar lines, many are using simple features. I also received feedback that 90% of
> the applications use less than 4GB of memory for mbuf and burst sizes are up to 256.

Well, from my perspective the story is completely different:
Majority of real-world apps I am aware do use multiple mempools,
it is also not uncommon to have a mempools with size bigger then 4GB (8/16).
Again, there are queries to make mempools growable/shrinkable on demand.

> >
> > On the other hand, I agree that pointer compression can be useful for some applications, so we should accept it.
> >
> > However, pointer compression has nothing to do with the underlying hardware or operating system, so it does not belong in the EAL
> (which is already too bloated). It should be a separate library.
> Yes, this is generic (though there is SIMD code). We could move it out of EAL.
> 
> >
> >>
> >>> On 01/11/2023 18:12, Paul Szczepanek wrote:
> >>>
> >>>> This patchset is proposing adding a new EAL header with utility functions
> >>>> that allow compression of arrays of pointers.
> >>>>
> >>>> When passing caches full of pointers between threads, memory containing
> >>>> the pointers is copied multiple times which is especially costly between
> >>>> cores. A compression method will allow us to shrink the memory size
> >>>> copied.
> >>>>
> >>>> The compression takes advantage of the fact that pointers are usually
> >>>> located in a limited memory region (like a mempool). We can compress them
> >>>> by converting them to offsets from a base memory address.
> >>>>
> >>>> Offsets can be stored in fewer bytes (dictated by the memory region size
> >>>> and alignment of the pointer). For example: an 8 byte aligned pointer
> >>>> which is part of a 32GB memory pool can be stored in 4 bytes. The API is
> >>>> very generic and does not assume mempool pointers, any pointer can be
> >>>> passed in.
> >>>>
> >>>> Compression is based on few and fast operations and especially with vector
> >>>> instructions leveraged creates minimal overhead.
> >>>>
> >>>> The API accepts and returns arrays because the overhead means it only is
> >>>> worth it when done in bulk.
> >>>>
> >>>> Test is added that shows potential performance gain from compression. In
> >>>> this test an array of pointers is passed through a ring between two cores.
> >>>> It shows the gain which is dependent on the bulk operation size. In this
> >>>> synthetic test run on ampere altra a substantial (up to 25%) performance
> >>>> gain is seen if done in bulk size larger than 32. At 32 it breaks even and
> >>>> lower sizes create a small (less than 5%) slowdown due to overhead.
> >>>>
> >>>> In a more realistic mock application running the l3 forwarding dpdk
> >>>> example that works in pipeline mode on two cores this translated into a
> >>>> ~5% throughput increase on an ampere altra.
> >
> > Which burst size was used to achieve this ~5% throughput increase?
> This is the stock L3fwd application which is split into 2 stages: RX, L3fwd, TX. The default burst size 32 is used.
> 
> >
> >>>>
> >>>> v2:
> >>>> * addressed review comments (style, explanations and typos)
> >>>> * lowered bulk iterations closer to original numbers to keep runtime short
> >>>> * fixed pointer size warning on 32-bit arch
> >>>> v3:
> >>>> * added 16-bit versions of compression functions and tests
> >>>> * added documentation of these new utility functions in the EAL guide
> >>>> v4:
> >>>> * added unit test
> >>>> * fix bug in NEON implementation of 32-bit decompress
> >>>> v5:
> >>>> * disable NEON and SVE implementation on AARCH32 due to wrong pointer size
> >>>>
> >>>> Paul Szczepanek (4):
> >>>>   eal: add pointer compression functions
> >>>>   test: add pointer compress tests to ring perf test
> >>>>   docs: add pointer compression to the EAL guide
> >>>>   test: add unit test for ptr compression
> >>>>
> >>>>  .mailmap                                      |   1 +
> >>>>  app/test/meson.build                          |   1 +
> >>>>  app/test/test_eal_ptr_compress.c              | 108 ++++++
> >>>>  app/test/test_ring.h                          |  94 ++++-
> >>>>  app/test/test_ring_perf.c                     | 354 ++++++++++++------
> >>>>  .../prog_guide/env_abstraction_layer.rst      | 142 +++++++
> >>>>  lib/eal/include/meson.build                   |   1 +
> >>>>  lib/eal/include/rte_ptr_compress.h            | 266 +++++++++++++
> >>>>  8 files changed, 843 insertions(+), 124 deletions(-)
> >>>>  create mode 100644 app/test/test_eal_ptr_compress.c
> >>>>  create mode 100644 lib/eal/include/rte_ptr_compress.h
> >>>>
> >>>> --
> >>>> 2.25.1
> >>>>

Re: RE: [PATCH v5 0/4] add pointer compression API

[Paul Szczepanek]

On 04/03/2024 14:44, Konstantin Ananyev wrote:
>> This feature is targeted for pipeline mode of applications. We see many customers using pipeline mode. This feature helps in reducing
>> the cost of transferring the packets between cores by reducing the copies involved.
> 
> I do understand the intention, and I am not arguing about usefulness of the pipeline model. 
> My point is you are introducing new API: compress/decompress pointers,
> but don't provide (or even describe) any proper way for the developer to use it in a safe and predictable manner.
> Which from my perspective make it nearly useless and misleading.

In the latest version there is an example in the docs showing how to use
it. There is an integration test that shows how to use it. The comments
in the header also provide detailed guidance.

>> For an application with multiple pools, it depends on how the applications are using multiple pools. But, if there is a bunch of packets
>> belonging to multiple mempools, compressing those mbufs may not be possible. But if those mbufs are grouped per mempool and
>> are transferred on different queues, then it is possible. Hence the APIs are implemented very generically.
> 
> Ok, let's consider even more simplistic scenario - all pointers belong to one mempool.
> AFAIK, even one mempool can contain elements from different memzones,
> and these memzones are not guaranteed to have consecutive VAs.
> So even one mempool, with total size <=4GB can contain elements with distances between them more than 4GB. 
> Now let say at startup user created a mempool, how he can determine programmatically
> can he apply your compress API safely on it or not?
> I presume that if you are serious about this API usage, then such ability has to be provided.
> Something like:
> 
> int compress_pointer_deduce_mempool_base(const struct rte_memepool *mp[],
> 	uint32_t nb_mp, uint32_t compress_size, uintptr_t *base_ptr);
> 
> Or probably even more generic one:
> 
> struct mem_buf {uintptr_t base, size_t len;}; 
> int compress_pointer_deduce_base(const struct mem_buf *mem_buf[],
> 	uint32_t nb_membuf, uint32_t compress_size, uintptr_t *base_ptr);
> 
> Even with these functions in-place, user has to be extra careful:
>  - he can't add new memory chunks to these mempools (or he'll need to re-calcualte the new base_ptr)
>  - he needs to make sure that pointers from only these mempools will be used by compress/decompress.
> But at least it provides some ability to use this feature in real apps.
> 
> With such API in place it should be possible to make the auto-test more realistic:
> - allocate mempool 
> - deduce base_pointer
> - then we can have a loop with producer/consumer to mimic realistic workload.
>     As an example:
>      producer(s):  mempool_alloc(); <fill mbuf with some values>; ring_enqueue();  
>      consumer(s): ring_dequeue(); <read_and_check_mbuf_data>; free_mbuf();
> - free mempool

I understand your objections and agree that the proposed API is limited
in its applicability due to its strict requirements.

AFAIK DPDK rte_mempool does require the addresses to be virtually
contiguous as the memory reservation is done during creation of the
mempool and a single memzone is reserved. However, I do not require
users to use the rte_mempool as the API accepts pointers so other
mempool implementations could indeed allow non-contiguous VAs.

To help decide at compile time if compression is allowed I will add
extra macros to the header

#define BITS_REQUIRED_TO_STORE_VALUE(x) \
	((x) == 0 ? 1 : ((sizeof(x)) - __builtin_clzl(x)))
	
#define BIT_SHIFT_FROM_ALIGNMENT(x) ((x) == 0 ? 0 : __builtin_clzl(x))

#define CAN_USE_RTE_PTR_COMPRESS_16(memory_range, object_alignment) \
	((BITS_REQUIRED_TO_STORE_VALUE(memory_range) - \
	BIT_SHIFT_FROM_ALIGNMENT(object_alignment)) <= 16 ? 1 : 0)

#define CAN_USE_RTE_PTR_COMPRESS_32(memory_range, object_alignment) \
	((BITS_REQUIRED_TO_STORE_VALUE(memory_range) - \
	BIT_SHIFT_FROM_ALIGNMENT(object_alignment)) <= 32 ? 1 : 0)

And explain usage in the docs.

The API is very generic and does not even require you to use a mempool.
There are no runtime checks to verify or calculate if the pointers can
be compressed. This is because doing this at runtime would remove any
gains achieved through compression. The code doing the compression needs
to remain limited in size, branching and execution time to remain fast.

This is IMHO the nature of C applications, they trade off runtime checks
for performance. Program correctness needs to be enforced through other
means, linting, valgrind, tests, peer review, etc. It is up to the
programmer to calculate and decide on the viability of compression as it
cannot be done at compile time automatically. There is no way for me to
programmatically verify the alignment and distance of the pointers being
passed in at compile time as I don't require the user to use any
particular mempool implementation.

These limitations are clearly documented in the API and the guide.

> Or probably you can go even further: take some existing pipeline sample app and make it use compress/decompress API.
> That will provide people with some ability to test it and measure it's perf impact.
> Again, it will provide an example of the amount of changes required to enable it.
> My speculation here that majority of users will find the effort too big, 
> while the gain way too limited and fragile.
> But at least, there would be some realistic reference point for it and users can decide themselves is it worth it or not. 

I have added a performance test that runs the compression and
decompression loop with different burst sizes so that you can easily
test if attempting compression is worth the the effort in your usecase.
This is documented in the guide.

> 
>>>
>>> I would like to add:
>>> If we want to offer optimizations specifically for applications with a single mbuf pool, I think it should be considered in a system-wide
>> context to determine if performance could be improved in more areas.
>>> E.g. removing the pool field from the rte_mbuf structure might free up space to move hot fields from the second cache line to the
>> first, so the second cache line rarely needs to be touched. (As an alternative to removing the pool field, it could be moved to the
>> second cache line, only to be used if the global "single mbuf pool" is NULL.)
>> Agree on this. The feedback I have received is on similar lines, many are using simple features. I also received feedback that 90% of
>> the applications use less than 4GB of memory for mbuf and burst sizes are up to 256.
> 
> Well, from my perspective the story is completely different:
> Majority of real-world apps I am aware do use multiple mempools,
> it is also not uncommon to have a mempools with size bigger then 4GB (8/16).
> Again, there are queries to make mempools growable/shrinkable on demand.

You can use this API with mempools even as big as 32GB as long as your
alignment allows for sufficient shift (as explained in the headers and
docs) and rte_mempool objects will have at least 8 bytes alignment so
can fit a 32GB mempool in. It is true that you cannot use it if you
cannot guarantee the address range at compile time. This utility is not
a golden bullet to use on every pointer.

RE: RE: [PATCH v5 0/4] add pointer compression API

[Morten Brørup]

> From: Paul Szczepanek [mailto:paul.szczepanek@arm.com]
> Sent: Wednesday, 15 May 2024 19.01
> 
> On 04/03/2024 14:44, Konstantin Ananyev wrote:
> >> This feature is targeted for pipeline mode of applications. We see
> many customers using pipeline mode. This feature helps in reducing
> >> the cost of transferring the packets between cores by reducing the
> copies involved.
> >
> > I do understand the intention, and I am not arguing about usefulness
> of the pipeline model.
> > My point is you are introducing new API: compress/decompress pointers,
> > but don't provide (or even describe) any proper way for the developer
> to use it in a safe and predictable manner.
> > Which from my perspective make it nearly useless and misleading.
> 
> In the latest version there is an example in the docs showing how to use
> it. There is an integration test that shows how to use it. The comments
> in the header also provide detailed guidance.
> 
> >> For an application with multiple pools, it depends on how the
> applications are using multiple pools. But, if there is a bunch of
> packets
> >> belonging to multiple mempools, compressing those mbufs may not be
> possible. But if those mbufs are grouped per mempool and
> >> are transferred on different queues, then it is possible. Hence the
> APIs are implemented very generically.
> >
> > Ok, let's consider even more simplistic scenario - all pointers belong
> to one mempool.
> > AFAIK, even one mempool can contain elements from different memzones,
> > and these memzones are not guaranteed to have consecutive VAs.
> > So even one mempool, with total size <=4GB can contain elements with
> distances between them more than 4GB.
> > Now let say at startup user created a mempool, how he can determine
> programmatically
> > can he apply your compress API safely on it or not?
> > I presume that if you are serious about this API usage, then such
> ability has to be provided.
> > Something like:
> >
> > int compress_pointer_deduce_mempool_base(const struct rte_memepool
> *mp[],
> > 	uint32_t nb_mp, uint32_t compress_size, uintptr_t *base_ptr);
> >
> > Or probably even more generic one:
> >
> > struct mem_buf {uintptr_t base, size_t len;};
> > int compress_pointer_deduce_base(const struct mem_buf *mem_buf[],
> > 	uint32_t nb_membuf, uint32_t compress_size, uintptr_t *base_ptr);
> >
> > Even with these functions in-place, user has to be extra careful:
> >  - he can't add new memory chunks to these mempools (or he'll need to
> re-calcualte the new base_ptr)
> >  - he needs to make sure that pointers from only these mempools will
> be used by compress/decompress.
> > But at least it provides some ability to use this feature in real
> apps.
> >
> > With such API in place it should be possible to make the auto-test
> more realistic:
> > - allocate mempool
> > - deduce base_pointer
> > - then we can have a loop with producer/consumer to mimic realistic
> workload.
> >     As an example:
> >      producer(s):  mempool_alloc(); <fill mbuf with some values>;
> ring_enqueue();
> >      consumer(s): ring_dequeue(); <read_and_check_mbuf_data>;
> free_mbuf();
> > - free mempool
> 
> I understand your objections and agree that the proposed API is limited
> in its applicability due to its strict requirements.
> 
> AFAIK DPDK rte_mempool does require the addresses to be virtually
> contiguous as the memory reservation is done during creation of the
> mempool and a single memzone is reserved.

No, it does not.
rte_pktmbuf_pool_create() first creates an empty mempool using rte_mempool_create_empty(), and then populates it using rte_mempool_populate_default(), which may use multiple memzones.

As one possible solution to this, the application can call rte_pktmbuf_pool_create() as usual, and then check that mp->nb_mem_chunks == 1 to confirm that all objects in the created mempool reside in one contiguous chunk of memory and thus compression can be used.

Or even better, add a new mempool flag to the mempool library, e.g. RTE_MEMPOOL_F_COMPRESSIBLE, specifying that the mempool objects must be allocated as one chunk of memory with contiguous addresses.
Unfortunately, rte_pktmbuf_pool_create() is missing the memzone flags parameter, so a new rte_pktmbuf_pool_create() API with the flags parameter added would also need to be added.

> However, I do not require
> users to use the rte_mempool as the API accepts pointers so other
> mempool implementations could indeed allow non-contiguous VAs.
> 
> To help decide at compile time if compression is allowed I will add
> extra macros to the header
> 
> #define BITS_REQUIRED_TO_STORE_VALUE(x) \
> 	((x) == 0 ? 1 : ((sizeof(x)) - __builtin_clzl(x)))
> 
> #define BIT_SHIFT_FROM_ALIGNMENT(x) ((x) == 0 ? 0 : __builtin_clzl(x))
> 
> #define CAN_USE_RTE_PTR_COMPRESS_16(memory_range, object_alignment) \
> 	((BITS_REQUIRED_TO_STORE_VALUE(memory_range) - \
> 	BIT_SHIFT_FROM_ALIGNMENT(object_alignment)) <= 16 ? 1 : 0)
> 
> #define CAN_USE_RTE_PTR_COMPRESS_32(memory_range, object_alignment) \
> 	((BITS_REQUIRED_TO_STORE_VALUE(memory_range) - \
> 	BIT_SHIFT_FROM_ALIGNMENT(object_alignment)) <= 32 ? 1 : 0)
> 
> And explain usage in the docs.
> 
> The API is very generic and does not even require you to use a mempool.
> There are no runtime checks to verify or calculate if the pointers can
> be compressed. This is because doing this at runtime would remove any
> gains achieved through compression. The code doing the compression needs
> to remain limited in size, branching and execution time to remain fast.
> 
> This is IMHO the nature of C applications, they trade off runtime checks
> for performance. Program correctness needs to be enforced through other
> means, linting, valgrind, tests, peer review, etc. It is up to the
> programmer to calculate and decide on the viability of compression as it
> cannot be done at compile time automatically. There is no way for me to
> programmatically verify the alignment and distance of the pointers being
> passed in at compile time as I don't require the user to use any
> particular mempool implementation.
> 
> These limitations are clearly documented in the API and the guide.
> 
> > Or probably you can go even further: take some existing pipeline
> sample app and make it use compress/decompress API.
> > That will provide people with some ability to test it and measure it's
> perf impact.
> > Again, it will provide an example of the amount of changes required to
> enable it.
> > My speculation here that majority of users will find the effort too
> big,
> > while the gain way too limited and fragile.
> > But at least, there would be some realistic reference point for it and
> users can decide themselves is it worth it or not.
> 
> I have added a performance test that runs the compression and
> decompression loop with different burst sizes so that you can easily
> test if attempting compression is worth the the effort in your usecase.
> This is documented in the guide.
> 
> >
> >>>
> >>> I would like to add:
> >>> If we want to offer optimizations specifically for applications with
> a single mbuf pool, I think it should be considered in a system-wide
> >> context to determine if performance could be improved in more areas.
> >>> E.g. removing the pool field from the rte_mbuf structure might free
> up space to move hot fields from the second cache line to the
> >> first, so the second cache line rarely needs to be touched. (As an
> alternative to removing the pool field, it could be moved to the
> >> second cache line, only to be used if the global "single mbuf pool"
> is NULL.)
> >> Agree on this. The feedback I have received is on similar lines, many
> are using simple features. I also received feedback that 90% of
> >> the applications use less than 4GB of memory for mbuf and burst sizes
> are up to 256.
> >
> > Well, from my perspective the story is completely different:
> > Majority of real-world apps I am aware do use multiple mempools,
> > it is also not uncommon to have a mempools with size bigger then 4GB
> (8/16).
> > Again, there are queries to make mempools growable/shrinkable on
> demand.
> 
> You can use this API with mempools even as big as 32GB as long as your
> alignment allows for sufficient shift (as explained in the headers and
> docs) and rte_mempool objects will have at least 8 bytes alignment so
> can fit a 32GB mempool in. It is true that you cannot use it if you
> cannot guarantee the address range at compile time. This utility is not
> a golden bullet to use on every pointer.

For future proofing, please rename the compression functions to include the compression algorithm, i.e. "shift" or similar, in the function names.

Specifically I'm thinking about an alternative "multiply" compression algorithm based on division/multiplication by a constant "multiplier" parameter (instead of the "bit_shift" parameter).
This "multiplier" would typically be the object size of the packet mbuf mempool.
The "multiplier" could be constant at built time, e.g. 2368, or determined at runtime.
I don't know the performance of division/multiplication compared to bit shifting for various CPUs, but it would make compression to 16 bit compressed pointers viable for more applications.

The perf test in this series could be used to determine compression/decompression performance of such an algorithm, and the application developer can determine which algorithm to use; "shift" with 32 bit compressed pointers, or "multiply" with 16 bit compressed pointers.

Re: [PATCH v5 0/4] add pointer compression API

[Paul Szczepanek]

On 15/05/2024 23:34, Morten Brørup wrote:
>> From: Paul Szczepanek [mailto:paul.szczepanek@arm.com]
>>
>> AFAIK DPDK rte_mempool does require the addresses to be virtually
>> contiguous as the memory reservation is done during creation of the
>> mempool and a single memzone is reserved.
> 
> No, it does not.
> rte_pktmbuf_pool_create() first creates an empty mempool using rte_mempool_create_empty(), and then populates it using rte_mempool_populate_default(), which may use multiple memzones.
> 
> As one possible solution to this, the application can call rte_pktmbuf_pool_create() as usual, and then check that mp->nb_mem_chunks == 1 to confirm that all objects in the created mempool reside in one contiguous chunk of memory and thus compression can be used.
> 
> Or even better, add a new mempool flag to the mempool library, e.g. RTE_MEMPOOL_F_COMPRESSIBLE, specifying that the mempool objects must be allocated as one chunk of memory with contiguous addresses.
> Unfortunately, rte_pktmbuf_pool_create() is missing the memzone flags parameter, so a new rte_pktmbuf_pool_create() API with the flags parameter added would also need to be added.
> 

You're right, my misunderstanding stemmed from only one mz being stored
in the rte_mempool struct, but nb_mem_chunks is in fact the variable to
check in mempool to verify contiguous VAs. I'll look into the
possibility of adding a mempool contiguous option to mempool.

> 
> For future proofing, please rename the compression functions to include the compression algorithm, i.e. "shift" or similar, in the function names.
> 
> Specifically I'm thinking about an alternative "multiply" compression algorithm based on division/multiplication by a constant "multiplier" parameter (instead of the "bit_shift" parameter).
> This "multiplier" would typically be the object size of the packet mbuf mempool.
> The "multiplier" could be constant at built time, e.g. 2368, or determined at runtime.
> I don't know the performance of division/multiplication compared to bit shifting for various CPUs, but it would make compression to 16 bit compressed pointers viable for more applications.
> 
> The perf test in this series could be used to determine compression/decompression performance of such an algorithm, and the application developer can determine which algorithm to use; "shift" with 32 bit compressed pointers, or "multiply" with 16 bit compressed pointers.
> 

Will add shift to name.

RE: RE: [PATCH v5 0/4] add pointer compression API

[Konstantin Ananyev]

> > From: Paul Szczepanek [mailto:paul.szczepanek@arm.com]
> > Sent: Wednesday, 15 May 2024 19.01
> >
> > On 04/03/2024 14:44, Konstantin Ananyev wrote:
> > >> This feature is targeted for pipeline mode of applications. We see
> > many customers using pipeline mode. This feature helps in reducing
> > >> the cost of transferring the packets between cores by reducing the
> > copies involved.
> > >
> > > I do understand the intention, and I am not arguing about usefulness
> > of the pipeline model.
> > > My point is you are introducing new API: compress/decompress pointers,
> > > but don't provide (or even describe) any proper way for the developer
> > to use it in a safe and predictable manner.
> > > Which from my perspective make it nearly useless and misleading.
> >
> > In the latest version there is an example in the docs showing how to use
> > it. There is an integration test that shows how to use it. The comments
> > in the header also provide detailed guidance.
> >
> > >> For an application with multiple pools, it depends on how the
> > applications are using multiple pools. But, if there is a bunch of
> > packets
> > >> belonging to multiple mempools, compressing those mbufs may not be
> > possible. But if those mbufs are grouped per mempool and
> > >> are transferred on different queues, then it is possible. Hence the
> > APIs are implemented very generically.
> > >
> > > Ok, let's consider even more simplistic scenario - all pointers belong
> > to one mempool.
> > > AFAIK, even one mempool can contain elements from different memzones,
> > > and these memzones are not guaranteed to have consecutive VAs.
> > > So even one mempool, with total size <=4GB can contain elements with
> > distances between them more than 4GB.
> > > Now let say at startup user created a mempool, how he can determine
> > programmatically
> > > can he apply your compress API safely on it or not?
> > > I presume that if you are serious about this API usage, then such
> > ability has to be provided.
> > > Something like:
> > >
> > > int compress_pointer_deduce_mempool_base(const struct rte_memepool
> > *mp[],
> > > 	uint32_t nb_mp, uint32_t compress_size, uintptr_t *base_ptr);
> > >
> > > Or probably even more generic one:
> > >
> > > struct mem_buf {uintptr_t base, size_t len;};
> > > int compress_pointer_deduce_base(const struct mem_buf *mem_buf[],
> > > 	uint32_t nb_membuf, uint32_t compress_size, uintptr_t *base_ptr);
> > >
> > > Even with these functions in-place, user has to be extra careful:
> > >  - he can't add new memory chunks to these mempools (or he'll need to
> > re-calcualte the new base_ptr)
> > >  - he needs to make sure that pointers from only these mempools will
> > be used by compress/decompress.
> > > But at least it provides some ability to use this feature in real
> > apps.
> > >
> > > With such API in place it should be possible to make the auto-test
> > more realistic:
> > > - allocate mempool
> > > - deduce base_pointer
> > > - then we can have a loop with producer/consumer to mimic realistic
> > workload.
> > >     As an example:
> > >      producer(s):  mempool_alloc(); <fill mbuf with some values>;
> > ring_enqueue();
> > >      consumer(s): ring_dequeue(); <read_and_check_mbuf_data>;
> > free_mbuf();
> > > - free mempool
> >
> > I understand your objections and agree that the proposed API is limited
> > in its applicability due to its strict requirements.
> >
> > AFAIK DPDK rte_mempool does require the addresses to be virtually
> > contiguous as the memory reservation is done during creation of the
> > mempool and a single memzone is reserved.
> 
> No, it does not.
> rte_pktmbuf_pool_create() first creates an empty mempool using rte_mempool_create_empty(), and then populates it using
> rte_mempool_populate_default(), which may use multiple memzones.

Yep. 

> As one possible solution to this, the application can call rte_pktmbuf_pool_create() as usual, and then check that mp-
> >nb_mem_chunks == 1 to confirm that all objects in the created mempool reside in one contiguous chunk of memory and thus
> compression can be used.

That's exactly what I am asking for: a new API call that would take number of bits to compress as a parameter,
check mempool layout (number of chunks, base addr and size of each chunk)and based on that either
return base ptr to use, or an error code.  
 
> Or even better, add a new mempool flag to the mempool library, e.g. RTE_MEMPOOL_F_COMPRESSIBLE, specifying that the
> mempool objects must be allocated as one chunk of memory with contiguous addresses.
> Unfortunately, rte_pktmbuf_pool_create() is missing the memzone flags parameter, so a new rte_pktmbuf_pool_create() API with
> the flags parameter added would also need to be added.

Hmm.., I didn't think about that, but yes, I think such API might be useful:
at mempool_create() define a memory layout constraints that it has to obey.
If we'll make it generic enough, I suppose might be helpful for different use-cases.  

> > However, I do not require
> > users to use the rte_mempool as the API accepts pointers so other
> > mempool implementations could indeed allow non-contiguous VAs.
> >
> > To help decide at compile time if compression is allowed I will add
> > extra macros to the header
> >
> > #define BITS_REQUIRED_TO_STORE_VALUE(x) \
> > 	((x) == 0 ? 1 : ((sizeof(x)) - __builtin_clzl(x)))
> >
> > #define BIT_SHIFT_FROM_ALIGNMENT(x) ((x) == 0 ? 0 : __builtin_clzl(x))
> >
> > #define CAN_USE_RTE_PTR_COMPRESS_16(memory_range, object_alignment) \
> > 	((BITS_REQUIRED_TO_STORE_VALUE(memory_range) - \
> > 	BIT_SHIFT_FROM_ALIGNMENT(object_alignment)) <= 16 ? 1 : 0)
> >
> > #define CAN_USE_RTE_PTR_COMPRESS_32(memory_range, object_alignment) \
> > 	((BITS_REQUIRED_TO_STORE_VALUE(memory_range) - \
> > 	BIT_SHIFT_FROM_ALIGNMENT(object_alignment)) <= 32 ? 1 : 0)
> >
> > And explain usage in the docs.
> >
> > The API is very generic and does not even require you to use a mempool.
> > There are no runtime checks to verify or calculate if the pointers can
> > be compressed. 

Yes, and that is the main problem, I believe.

>This is because doing this at runtime would remove any
> > gains achieved through compression. The code doing the compression needs
> > to remain limited in size, branching and execution time to remain fast.
> > This is IMHO the nature of C applications, they trade off runtime checks
> > for performance.

I think this is not correct and you are mixing few completely different things together.
I believe that well written C code (as any other) should do runtime checks.
Now, in DPDK we often do omit run-time checks for input parameters at
Performance critical path.
But we do encourage people to do as many checks as they can at control-path.
As an example - we don't do much checks at eth_rx_burst() itself,
but we do a lot of checks at setup phase: dev_configure(), rx_queue_setup(), etc.
With ptr compression API, I think it can done in the same way:
You can provide an API that would do check is ptr compression applicable for
his current memory layout or not.
It doesn't have to be fast, as user will call it just once at setup stage - before data-path is started.
See what Morten suggested above.   

>> Program correctness needs to be enforced through other
> > means, linting, valgrind, tests, peer review, etc. It is up to the
> > programmer to calculate and decide on the viability of compression as it
> > cannot be done at compile time automatically. There is no way for me to
> > programmatically verify the alignment and distance of the pointers being
> > passed in at compile time as I don't require the user to use any
> > particular mempool implementation.
> >
> > These limitations are clearly documented in the API and the guide.
> >
> > > Or probably you can go even further: take some existing pipeline
> > sample app and make it use compress/decompress API.
> > > That will provide people with some ability to test it and measure it's
> > perf impact.
> > > Again, it will provide an example of the amount of changes required to
> > enable it.
> > > My speculation here that majority of users will find the effort too
> > big,
> > > while the gain way too limited and fragile.
> > > But at least, there would be some realistic reference point for it and
> > users can decide themselves is it worth it or not.
> >
> > I have added a performance test that runs the compression and
> > decompression loop with different burst sizes so that you can easily
> > test if attempting compression is worth the the effort in your usecase.
> > This is documented in the guide.
> >
> > >
> > >>>
> > >>> I would like to add:
> > >>> If we want to offer optimizations specifically for applications with
> > a single mbuf pool, I think it should be considered in a system-wide
> > >> context to determine if performance could be improved in more areas.
> > >>> E.g. removing the pool field from the rte_mbuf structure might free
> > up space to move hot fields from the second cache line to the
> > >> first, so the second cache line rarely needs to be touched. (As an
> > alternative to removing the pool field, it could be moved to the
> > >> second cache line, only to be used if the global "single mbuf pool"
> > is NULL.)
> > >> Agree on this. The feedback I have received is on similar lines, many
> > are using simple features. I also received feedback that 90% of
> > >> the applications use less than 4GB of memory for mbuf and burst sizes
> > are up to 256.
> > >
> > > Well, from my perspective the story is completely different:
> > > Majority of real-world apps I am aware do use multiple mempools,
> > > it is also not uncommon to have a mempools with size bigger then 4GB
> > (8/16).
> > > Again, there are queries to make mempools growable/shrinkable on
> > demand.
> >
> > You can use this API with mempools even as big as 32GB as long as your
> > alignment allows for sufficient shift (as explained in the headers and
> > docs) and rte_mempool objects will have at least 8 bytes alignment so
> > can fit a 32GB mempool in. It is true that you cannot use it if you
> > cannot guarantee the address range at compile time. This utility is not
> > a golden bullet to use on every pointer.
> 
> For future proofing, please rename the compression functions to include the compression algorithm, i.e. "shift" or similar, in the
> function names.
> 
> Specifically I'm thinking about an alternative "multiply" compression algorithm based on division/multiplication by a constant
> "multiplier" parameter (instead of the "bit_shift" parameter).
> This "multiplier" would typically be the object size of the packet mbuf mempool.
> The "multiplier" could be constant at built time, e.g. 2368, or determined at runtime.
> I don't know the performance of division/multiplication compared to bit shifting for various CPUs, but it would make compression to
> 16 bit compressed pointers viable for more applications.
> 
> The perf test in this series could be used to determine compression/decompression performance of such an algorithm, and the
> application developer can determine which algorithm to use; "shift" with 32 bit compressed pointers, or "multiply" with 16 bit
> compressed pointers.

Re: [PATCH v5 0/4] add pointer compression API

[Paul Szczepanek]

I have added macros to help find the parameters and I have added mempool
functions that allow you to determine if you can use the mempool and
what params it needs. The new mempool functions are mentioned in the
docs for ptr compress.

Please take a look at v11.

I did not add a new make pkt buf API that takes mempool flags as this
would be a much bigger task requiring a lot of new APIs and I think this
is something that is useful but not directly related to this patchset.
It's something I could I develop separately without having to worry
about 24.07 deadlines.

RE: [PATCH v5 0/4] add pointer compression API

[Konstantin Ananyev]

> I have added macros to help find the parameters and I have added mempool
> functions that allow you to determine if you can use the mempool and
> what params it needs. The new mempool functions are mentioned in the
> docs for ptr compress.
> Please take a look at v11.

Great, thanks.
Will try to have a look in next few days. 
With these functions in place, can we produce a unit-test that
will use together these new mempool functions and compress API? 
Something like: 
- allocate mempool 
- deduce base_pointer for it
- main_loop_start:
    producer(s):  mempool_get(); <fill mbuf with some values>; ring_enqueue();  
    consumer(s): ring_dequeue(); <read_and_check_mbuf_data>; mempool_put();
- main_loop_end
- free mempool
 
> I did not add a new make pkt buf API that takes mempool flags as this
> would be a much bigger task requiring a lot of new APIs and I think this
> is something that is useful but not directly related to this patchset.
> It's something I could I develop separately without having to worry
> about 24.07 deadlines.

Yep, agree, it sounds like a subject for a separate patch series.
Konstantin

Re: [PATCH v5 0/4] add pointer compression API

[Paul Szczepanek]

On 24/05/2024 10:09, Konstantin Ananyev wrote:
> 
> 
>> I have added macros to help find the parameters and I have added mempool
>> functions that allow you to determine if you can use the mempool and
>> what params it needs. The new mempool functions are mentioned in the
>> docs for ptr compress.
>> Please take a look at v11.
> 
> Great, thanks.
> Will try to have a look in next few days. 
> With these functions in place, can we produce a unit-test that
> will use together these new mempool functions and compress API? 
> Something like: 
> - allocate mempool 
> - deduce base_pointer for it
> - main_loop_start:
>     producer(s):  mempool_get(); <fill mbuf with some values>; ring_enqueue();  
>     consumer(s): ring_dequeue(); <read_and_check_mbuf_data>; mempool_put();
> - main_loop_end
> - free mempool

The v11 already includes mempool base pointer and range calculation in
the mempool test and the functions are mentioned in the ptr compress lib
docs. The ptr compress test doesn't use a mempool to minimise dependencies.

I have a v12 pending (awaiting internal reviews) that addresses Morten's
comments (adds prefix, adds tests and doxygen for all the macros, uses
rte_bitops) and a fix for the guide which had the wrong letter case for
the MACRO.

Re: [PATCH v5 0/4] add pointer compression API

[Paul Szczepanek]

On 28/05/2024 20:29, Paul Szczepanek wrote:
> 
> 
> On 24/05/2024 10:09, Konstantin Ananyev wrote:
>>
>>
>>> I have added macros to help find the parameters and I have added mempool
>>> functions that allow you to determine if you can use the mempool and
>>> what params it needs. The new mempool functions are mentioned in the
>>> docs for ptr compress.
>>> Please take a look at v11.
>>
>> Great, thanks.
>> Will try to have a look in next few days. 
>> With these functions in place, can we produce a unit-test that
>> will use together these new mempool functions and compress API? 
>> Something like: 
>> - allocate mempool 
>> - deduce base_pointer for it
>> - main_loop_start:
>>     producer(s):  mempool_get(); <fill mbuf with some values>; ring_enqueue();  
>>     consumer(s): ring_dequeue(); <read_and_check_mbuf_data>; mempool_put();
>> - main_loop_end
>> - free mempool
> 
> The v11 already includes mempool base pointer and range calculation in
> the mempool test and the functions are mentioned in the ptr compress lib
> docs. The ptr compress test doesn't use a mempool to minimise dependencies.
> 
> I have a v12 pending (awaiting internal reviews) that addresses Morten's
> comments (adds prefix, adds tests and doxygen for all the macros, uses
> rte_bitops) and a fix for the guide which had the wrong letter case for
> the MACRO.

v12 is now up ready for your review. I hope that the explanation in the
ptr compress guide document is enough to show users how to use mempool
pointers with it. Between the guide and the doxygen it should be clear
what values are needed as parameters of the compression function.

RE: [PATCH v5 0/4] add pointer compression API

[Konstantin Ananyev]

> >> I have added macros to help find the parameters and I have added mempool
> >> functions that allow you to determine if you can use the mempool and
> >> what params it needs. The new mempool functions are mentioned in the
> >> docs for ptr compress.
> >> Please take a look at v11.
> >
> > Great, thanks.
> > Will try to have a look in next few days.
> > With these functions in place, can we produce a unit-test that
> > will use together these new mempool functions and compress API?
> > Something like:
> > - allocate mempool
> > - deduce base_pointer for it
> > - main_loop_start:
> >     producer(s):  mempool_get(); <fill mbuf with some values>; ring_enqueue();
> >     consumer(s): ring_dequeue(); <read_and_check_mbuf_data>; mempool_put();
> > - main_loop_end
> > - free mempool
> 
> The v11 already includes mempool base pointer and range calculation in
> the mempool test and the functions are mentioned in the ptr compress lib
> docs. The ptr compress test doesn't use a mempool to minimise dependencies.

Sorry, but I didn't get it: why adding mempool dependency for the unit-test
suddenly become a problem?
From other side - you are introducing a new library and new API for mempool
to enable this new lib.
From my perspective it is essential to provide a unit-test (or sample app) to:
- do a proper functional (ensure that compress/decompress doesn't corrupt pointers)
  and performance testing.
- demonstrate how this 2 things (new lib and new mempool API) can(/should) be used together.

> 
> I have a v12 pending (awaiting internal reviews) that addresses Morten's
> comments (adds prefix, adds tests and doxygen for all the macros, uses
> rte_bitops) and a fix for the guide which had the wrong letter case for
> the MACRO.

[PATCH v6 0/4] add pointer compression API

[Paul Szczepanek]

This patchset is proposing adding a new EAL header with utility functions
that allow compression of arrays of pointers.

When passing caches full of pointers between threads, memory containing
the pointers is copied multiple times which is especially costly between
cores. A compression method will allow us to shrink the memory size
copied.

The compression takes advantage of the fact that pointers are usually
located in a limited memory region (like a mempool). We can compress them
by converting them to offsets from a base memory address.

Offsets can be stored in fewer bytes (dictated by the memory region size
and alignment of the pointer). For example: an 8 byte aligned pointer
which is part of a 32GB memory pool can be stored in 4 bytes. The API is
very generic and does not assume mempool pointers, any pointer can be
passed in.

Compression is based on few and fast operations and especially with vector
instructions leveraged creates minimal overhead.

The API accepts and returns arrays because the overhead means it only is
worth it when done in bulk.

Test is added that shows potential performance gain from compression. In
this test an array of pointers is passed through a ring between two cores.
It shows the gain which is dependent on the bulk operation size. In this
synthetic test run on ampere altra a substantial (up to 25%) performance
gain is seen if done in bulk size larger than 32. At 32 it breaks even and
lower sizes create a small (less than 5%) slowdown due to overhead.

In a more realistic mock application running the l3 forwarding dpdk
example that works in pipeline mode on two cores this translated into a
~5% throughput increase on an ampere altra.

v2:
* addressed review comments (style, explanations and typos)
* lowered bulk iterations closer to original numbers to keep runtime short
* fixed pointer size warning on 32-bit arch
v3:
* added 16-bit versions of compression functions and tests
* added documentation of these new utility functions in the EAL guide
v4:
* added unit test
* fix bug in NEON implementation of 32-bit decompress
v5:
* disable NEON and SVE implementation on AARCH32 due to wrong pointer size
v6:
* added example usage to commit message of the initial commit

Paul Szczepanek (4):
  eal: add pointer compression functions
  test: add pointer compress tests to ring perf test
  docs: add pointer compression to the EAL guide
  test: add unit test for ptr compression

 .mailmap                                      |   1 +
 app/test/meson.build                          |   1 +
 app/test/test_eal_ptr_compress.c              | 108 ++++++
 app/test/test_ring.h                          |  94 ++++-
 app/test/test_ring_perf.c                     | 354 ++++++++++++------
 .../prog_guide/env_abstraction_layer.rst      | 142 +++++++
 lib/eal/include/meson.build                   |   1 +
 lib/eal/include/rte_ptr_compress.h            | 266 +++++++++++++
 8 files changed, 843 insertions(+), 124 deletions(-)
 create mode 100644 app/test/test_eal_ptr_compress.c
 create mode 100644 lib/eal/include/rte_ptr_compress.h

--
2.25.1

[PATCH v6 1/4] eal: add pointer compression functions

[Paul Szczepanek]

Add a new utility header for compressing pointers. The provided
functions can store pointers in 32-bit offsets.

The compression takes advantage of the fact that pointers are
usually located in a limited memory region (like a mempool).
We can compress them by converting them to offsets from a base
memory address. Offsets can be stored in fewer bytes (dictated
by the memory region size and alignment of the pointer).
For example: an 8 byte aligned pointer which is part of a 32GB
memory pool can be stored in 4 bytes.

This can be used for example when passing caches full of pointers
between threads. Memory containing the pointers is copied multiple
times which is especially costly between cores. This compression
method will allow us to shrink the memory size copied. Further
commits add a test to evaluate the effectiveness of this approach.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .mailmap                           |   1 +
 lib/eal/include/meson.build        |   1 +
 lib/eal/include/rte_ptr_compress.h | 266 +++++++++++++++++++++++++++++
 3 files changed, 268 insertions(+)
 create mode 100644 lib/eal/include/rte_ptr_compress.h

diff --git a/.mailmap b/.mailmap
index 3f5bab26a8..004751d27a 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1069,6 +1069,7 @@ Paul Greenwalt <paul.greenwalt@intel.com>
 Paulis Gributs <paulis.gributs@intel.com>
 Paul Luse <paul.e.luse@intel.com>
 Paul M Stillwell Jr <paul.m.stillwell.jr@intel.com>
+Paul Szczepanek <paul.szczepanek@arm.com>
 Pavan Kumar Linga <pavan.kumar.linga@intel.com>
 Pavan Nikhilesh <pbhagavatula@marvell.com> <pbhagavatula@caviumnetworks.com>
 Pavel Belous <pavel.belous@aquantia.com>
diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index e94b056d46..ce2c733633 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -36,6 +36,7 @@ headers += files(
         'rte_pci_dev_features.h',
         'rte_per_lcore.h',
         'rte_pflock.h',
+	'rte_ptr_compress.h',
         'rte_random.h',
         'rte_reciprocal.h',
         'rte_seqcount.h',
diff --git a/lib/eal/include/rte_ptr_compress.h b/lib/eal/include/rte_ptr_compress.h
new file mode 100644
index 0000000000..47a72e4213
--- /dev/null
+++ b/lib/eal/include/rte_ptr_compress.h
@@ -0,0 +1,266 @@
+/* SPDX-License-Identifier: BSD-shift-Clause
+ * Copyright(c) 2023 Arm Limited
+ */
+
+#ifndef RTE_PTR_COMPRESS_H
+#define RTE_PTR_COMPRESS_H
+
+/**
+ * @file
+ * Pointer compression and decompression functions.
+ *
+ * When passing arrays full of pointers between threads, memory containing
+ * the pointers is copied multiple times which is especially costly between
+ * cores. These functions allow us to compress the pointers.
+ *
+ * Compression takes advantage of the fact that pointers are usually located in
+ * a limited memory region (like a mempool). We compress them by converting them
+ * to offsets from a base memory address. Offsets can be stored in fewer bytes.
+ *
+ * The compression functions come in two varieties: 32-bit and 16-bit.
+ *
+ * To determine how many bits are needed to compress the pointer calculate
+ * the biggest offset possible (highest value pointer - base pointer)
+ * and shift the value right according to alignment (shift by exponent of the
+ * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by
+ * 3, etc.). The resulting value must fit in either 32 or 16 bits.
+ *
+ * For usage example and further explanation please see "Pointer Compression" in
+ * doc/guides/prog_guide/env_abstraction_layer.rst
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_vect.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Compress pointers into 32-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 32 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 32GB of the base pointer. If there is no such alignment guarantee they
+ * must be within 4GB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_32(void *ptr_base, void **src_table,
+		uint32_t *dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1w(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	/* right shift is done by left shifting by negative int */
+	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_ptr_table = vld1q_u64((const uint64_t *)src_table + i);
+		v_ptr_table = vsubq_u64(v_ptr_table, v_ptr_base);
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		vst1_u32(dest_table + i, vqmovn_u64(v_ptr_table));
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT32_MAX);
+		dest_table[i] = (uint32_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 32-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_32(void *ptr_base, uint32_t *src_table,
+		void **dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1uw_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	int64x2_t v_shift = vdupq_n_s64(bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_ptr_table = vmovl_u32(vld1_u32(src_table + i));
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		v_ptr_table = vaddq_u64(v_ptr_table, v_ptr_base);
+		vst1q_u64((uint64_t *)dest_table + i, v_ptr_table);
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+/**
+ * Compress pointers into 16-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 16 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 256KB of the base pointer. If there is no such alignment guarantee
+ * they must be within 64KB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_16(void *ptr_base, void **src_table,
+		uint16_t *dest_table, unsigned int n, unsigned int bit_shift)
+{
+
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1h(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT16_MAX);
+		dest_table[i] = (uint16_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 16-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_16(void *ptr_base, uint16_t *src_table,
+		void **dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1uh_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_PTR_COMPRESS_H */
--
2.25.1

[PATCH v6 2/4] test: add pointer compress tests to ring perf test

[Paul Szczepanek]

Add a test that runs a zero copy burst enqueue and dequeue on a ring
of raw pointers and compressed pointers at different burst sizes to
showcase performance benefits of newly added pointer compression APIs.

Refactored threading code to pass more parameters to threads to
reuse existing code. Added more bulk sizes to showcase their effects
on compression. Adjusted loop iteration numbers to take into account
bulk sizes to keep runtime constant (instead of number of operations).

Adjusted old printfs to match new ones which have aligned numbers.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 app/test/test_ring.h      |  94 +++++++++-
 app/test/test_ring_perf.c | 354 +++++++++++++++++++++++++-------------
 2 files changed, 324 insertions(+), 124 deletions(-)

diff --git a/app/test/test_ring.h b/app/test/test_ring.h
index 45c263f3ff..3b00f2465d 100644
--- a/app/test/test_ring.h
+++ b/app/test/test_ring.h
@@ -1,10 +1,12 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */

 #include <rte_malloc.h>
 #include <rte_ring.h>
 #include <rte_ring_elem.h>
+#include <rte_memcpy.h>
+#include <rte_ptr_compress.h>

 /* API type to call
  * rte_ring_<sp/mp or sc/mc>_enqueue_<bulk/burst>
@@ -25,6 +27,10 @@
 #define TEST_RING_ELEM_BULK 16
 #define TEST_RING_ELEM_BURST 32

+#define TEST_RING_ELEM_BURST_ZC 64
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 128
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 256
+
 #define TEST_RING_IGNORE_API_TYPE ~0U

 /* This function is placed here as it is required for both
@@ -101,6 +107,9 @@ static inline unsigned int
 test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -152,6 +161,46 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n,
 								NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy(zcd.ptr2,
+						(char *)obj + zcd.n1 * esize,
+						(ret - zcd.n1) * esize);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_16(0, obj, zcd.ptr1, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_16(0,
+						obj + (zcd.n1 * 2),
+						zcd.ptr2,
+						(ret - zcd.n1) * 2, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_32(0, obj, zcd.ptr1, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_32(0, obj + zcd.n1,
+						zcd.ptr2, ret - zcd.n1, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
@@ -162,6 +211,9 @@ static inline unsigned int
 test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -213,6 +265,46 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mc_dequeue_burst_elem(r, obj, esize,
 								n, NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy((char *)obj + zcd.n1 * esize,
+						zcd.ptr2,
+						(ret - zcd.n1) * esize);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_16(0, zcd.ptr1, obj, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_16(0, zcd.ptr2,
+						obj + zcd.n1,
+						(ret - zcd.n1) * 2,
+						3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_32(0, zcd.ptr1, obj, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_32(0, zcd.ptr2,
+						obj + zcd.n1, ret - zcd.n1, 3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c
index d7c5a4c30b..8392d0c781 100644
--- a/app/test/test_ring_perf.c
+++ b/app/test/test_ring_perf.c
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2010-2014 Intel Corporation
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */


@@ -22,13 +22,13 @@

 #define RING_NAME "RING_PERF"
 #define RING_SIZE 4096
-#define MAX_BURST 32
+#define MAX_BURST 256

 /*
  * the sizes to enqueue and dequeue in testing
  * (marked volatile so they won't be seen as compile-time constants)
  */
-static const volatile unsigned bulk_sizes[] = { 8, 32 };
+static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128, 256 };

 struct lcore_pair {
 	unsigned c1, c2;
@@ -43,26 +43,30 @@ test_ring_print_test_string(unsigned int api_type, int esize,
 	if (esize == -1)
 		printf("legacy APIs");
 	else
-		printf("elem APIs: element size %dB", esize);
+		printf("elem APIs (size:%2dB)", esize);

 	if (api_type == TEST_RING_IGNORE_API_TYPE)
 		return;

 	if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF)
-		printf(": default enqueue/dequeue: ");
+		printf(" - default enqueue/dequeue");
 	else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC)
-		printf(": SP/SC: ");
+		printf(" - SP/SC");
 	else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC)
-		printf(": MP/MC: ");
+		printf(" - MP/MC");

 	if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE)
-		printf("single: ");
+		printf(" - single - ");
 	else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK)
-		printf("bulk (size: %u): ", bsz);
+		printf(" - bulk (n:%-3u) - ", bsz);
 	else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST)
-		printf("burst (size: %u): ", bsz);
+		printf(" - burst (n:%-3u) - ", bsz);
+	else if ((api_type & (TEST_RING_ELEM_BURST_ZC |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0)
+		printf(" - burst zero copy (n:%-3u) - ", bsz);

-	printf("%.2F\n", value);
+	printf("cycles per elem: %.3F\n", value);
 }

 /**** Functions to analyse our core mask to get cores for different tests ***/
@@ -144,28 +148,33 @@ static void
 test_empty_dequeue(struct rte_ring *r, const int esize,
 			const unsigned int api_type)
 {
-	const unsigned int iter_shift = 26;
+	const unsigned int iter_shift = 29;
 	const unsigned int iterations = 1 << iter_shift;
 	unsigned int i = 0;
 	void *burst[MAX_BURST];

+	const unsigned int bulk_iterations = iterations / bulk_sizes[0];
 	const uint64_t start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	for (i = 0; i < bulk_iterations; i++)
 		test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type);
 	const uint64_t end = rte_rdtsc();

 	test_ring_print_test_string(api_type, esize, bulk_sizes[0],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 }

-/*
- * for the separate enqueue and dequeue threads they take in one param
- * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc
- */
-struct thread_params {
+/* describes the ring used by the enqueue and dequeue thread */
+struct ring_params {
 	struct rte_ring *r;
-	unsigned size;        /* input value, the burst size */
-	double spsc, mpmc;    /* output value, the single or multi timings */
+	unsigned int elem_size;
+	unsigned int bulk_sizes_i; /* index into bulk_size array */
+	unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */
+};
+
+/* Used do specify enqueue and dequeue ring operations and their results */
+struct thread_params {
+	struct ring_params *ring_params;
+	double *results; /* result array size must be equal to bulk_sizes */
 };

 /*
@@ -174,16 +183,15 @@ struct thread_params {
  * flag == 1 -> dequeue
  */
 static __rte_always_inline int
-enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
-	struct thread_params *p)
+enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 15;
+	const unsigned int iter_shift = 22;
 	const unsigned int iterations = 1 << iter_shift;
-	struct rte_ring *r = p->r;
-	unsigned int bsize = p->size;
 	unsigned int i;
 	void *burst = NULL;
+	unsigned int n_remaining;
+	const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i];

 #ifdef RTE_USE_C11_MEM_MODEL
 	if (__atomic_fetch_add(&lcore_count, 1, __ATOMIC_RELAXED) + 1 != 2)
@@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 		while(lcore_count != 2)
 			rte_pause();

-	burst = test_ring_calloc(MAX_BURST, esize);
+	burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size);
 	if (burst == NULL)
 		return -1;

 	const uint64_t sp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	const unsigned int bulk_iterations = iterations / bulk_n;
+	for (i = 0; i < bulk_iterations; i++) {
+		n_remaining = bulk_n;
 		do {
 			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_enqueue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_dequeue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			if (ret == 0)
 				rte_pause();
-		} while (!ret);
+			else
+				n_remaining -= ret;
+		} while (n_remaining > 0);
+	}
 	const uint64_t sp_end = rte_rdtsc();

-	const uint64_t mp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
-		do {
-			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			if (ret == 0)
-				rte_pause();
-		} while (!ret);
-	const uint64_t mp_end = rte_rdtsc();
+	p->results[p->ring_params->bulk_sizes_i] =
+			((double)sp_end - sp_start) / iterations;

-	p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize);
-	p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize);
 	return 0;
 }

@@ -243,15 +245,7 @@ enqueue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(0, -1, params);
-}
-
-static int
-enqueue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(0, 16, params);
+	return enqueue_dequeue_bulk_helper(0, params);
 }

 /*
@@ -263,15 +257,7 @@ dequeue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(1, -1, params);
-}
-
-static int
-dequeue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(1, 16, params);
+	return enqueue_dequeue_bulk_helper(1, params);
 }

 /*
@@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p)
  * used to measure ring perf between hyperthreads, cores and sockets.
  */
 static int
-run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize)
+run_on_core_pair(struct lcore_pair *cores,
+		struct thread_params *param1, struct thread_params *param2)
 {
-	lcore_function_t *f1, *f2;
-	struct thread_params param1 = {0}, param2 = {0};
 	unsigned i;
-
-	if (esize == -1) {
-		f1 = enqueue_bulk;
-		f2 = dequeue_bulk;
-	} else {
-		f1 = enqueue_bulk_16B;
-		f2 = dequeue_bulk_16B;
-	}
+	struct ring_params *ring_params = param1->ring_params;

 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		lcore_count = 0;
-		param1.size = param2.size = bulk_sizes[i];
-		param1.r = param2.r = r;
+		ring_params->bulk_sizes_i = i;
 		if (cores->c1 == rte_get_main_lcore()) {
-			rte_eal_remote_launch(f2, &param2, cores->c2);
-			f1(&param1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
+			enqueue_bulk(param1);
 			rte_eal_wait_lcore(cores->c2);
 		} else {
-			rte_eal_remote_launch(f1, &param1, cores->c1);
-			rte_eal_remote_launch(f2, &param2, cores->c2);
+			rte_eal_remote_launch(enqueue_bulk, param1, cores->c1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
 			if (rte_eal_wait_lcore(cores->c1) < 0)
 				return -1;
 			if (rte_eal_wait_lcore(cores->c2) < 0)
 				return -1;
 		}
 		test_ring_print_test_string(
-			TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.spsc + param2.spsc);
-		test_ring_print_test_string(
-			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.mpmc + param2.mpmc);
+				ring_params->ring_flags,
+				ring_params->elem_size,
+				bulk_sizes[i],
+				param1->results[i] + param2->results[i]);
 	}

 	return 0;
@@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 	uint64_t hz = rte_get_timer_hz();
 	uint64_t lcount = 0;
 	const unsigned int lcore = rte_lcore_id();
-	struct thread_params *params = p;
+	struct ring_params *ring_params = p->ring_params;
 	void *burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -346,9 +322,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize)

 	begin = rte_get_timer_cycles();
 	while (time_diff < hz * TIME_MS / 1000) {
-		test_ring_enqueue(params->r, burst, esize, params->size,
+		test_ring_enqueue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
-		test_ring_dequeue(params->r, burst, esize, params->size,
+		test_ring_dequeue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 		lcount++;
 		time_diff = rte_get_timer_cycles() - begin;
@@ -380,7 +358,8 @@ static int
 run_on_all_cores(struct rte_ring *r, const int esize)
 {
 	uint64_t total;
-	struct thread_params param;
+	struct ring_params ring_params = {0};
+	struct thread_params params = { .ring_params = &ring_params };
 	lcore_function_t *lcore_f;
 	unsigned int i, c;

@@ -389,21 +368,20 @@ run_on_all_cores(struct rte_ring *r, const int esize)
 	else
 		lcore_f = load_loop_fn_16B;

-	memset(&param, 0, sizeof(struct thread_params));
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		total = 0;
 		printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]);
-		param.size = bulk_sizes[i];
-		param.r = r;
+		params.ring_params->bulk_sizes_i = i;
+		params.ring_params->r = r;

 		/* clear synchro and start workers */
 		__atomic_store_n(&synchro, 0, __ATOMIC_RELAXED);
-		if (rte_eal_mp_remote_launch(lcore_f, &param, SKIP_MAIN) < 0)
+		if (rte_eal_mp_remote_launch(lcore_f, &params, SKIP_MAIN) < 0)
 			return -1;

 		/* start synchro and launch test on main */
 		__atomic_store_n(&synchro, 1, __ATOMIC_RELAXED);
-		lcore_f(&param);
+		lcore_f(&params);

 		rte_eal_mp_wait_lcore();

@@ -462,9 +440,9 @@ static int
 test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	const unsigned int api_type)
 {
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 26;
 	const unsigned int iterations = 1 << iter_shift;
-	unsigned int sz, i = 0;
+	unsigned int sz, i;
 	void **burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -472,17 +450,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 		return -1;

 	for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) {
+		const unsigned int n = iterations / bulk_sizes[sz];
 		const uint64_t start = rte_rdtsc();
-		for (i = 0; i < iterations; i++) {
+		for (i = 0; i < n; i++) {
 			test_ring_enqueue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 			test_ring_dequeue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 		}
 		const uint64_t end = rte_rdtsc();

 		test_ring_print_test_string(api_type, esize, bulk_sizes[sz],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 	}

 	rte_free(burst);
@@ -490,12 +469,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	return 0;
 }

+static __rte_always_inline int
+test_ring_perf_esize_run_on_two_cores(
+		struct thread_params *param1, struct thread_params *param2)
+{
+	struct lcore_pair cores;
+
+	if (get_two_hyperthreads(&cores) == 0) {
+		printf("\n### Testing using two hyperthreads ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing using two physical cores ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_sockets(&cores) == 0) {
+		printf("\n### Testing using two NUMA nodes ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	return 0;
+}
+
 /* Run all tests for a given element size */
 static __rte_always_inline int
 test_ring_perf_esize(const int esize)
 {
-	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
+	double results_enq[RTE_DIM(bulk_sizes)];
+	double results_deq[RTE_DIM(bulk_sizes)];
+	struct ring_params ring_params = {
+			.elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results_enq };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results_deq };

 	/*
 	 * Performance test for legacy/_elem APIs
@@ -535,22 +545,13 @@ test_ring_perf_esize(const int esize)
 	test_empty_dequeue(r, esize,
 			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);

-	if (get_two_hyperthreads(&cores) == 0) {
-		printf("\n### Testing using two hyperthreads ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.r = r;

-	if (get_two_cores(&cores) == 0) {
-		printf("\n### Testing using two physical cores ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
-	if (get_two_sockets(&cores) == 0) {
-		printf("\n### Testing using two NUMA nodes ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
+
+	ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);

 	printf("\n### Testing using all worker nodes ###\n");
 	if (run_on_all_cores(r, esize) < 0)
@@ -566,6 +567,109 @@ test_ring_perf_esize(const int esize)
 	return -1;
 }

+
+static __rte_always_inline int
+test_ring_perf_compression(void)
+{
+	double results1[RTE_DIM(bulk_sizes)];
+	double results2[RTE_DIM(bulk_sizes)];
+	double results1_comp[2][RTE_DIM(bulk_sizes)];
+	double results2_comp[2][RTE_DIM(bulk_sizes)];
+
+	struct lcore_pair cores;
+	int ret = -1;
+	unsigned int i, j;
+	struct ring_params ring_params = { .elem_size = sizeof(void *) };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results1 };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results2 };
+
+	printf("\n### Testing compression gain ###");
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(void *),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy ###\n");
+		ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	/* rings allow only multiples of 4 as sizes,
+	 * we allocate size 4 despite only using 2 bytes
+	 * and use half of RING_SIZE as the number of elements
+	 */
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE / 2, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[0];
+	param2.results = results2_comp[0];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (16b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[1];
+	param2.results = results2_comp[1];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (32b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	for (j = 0; j < 2; j++) {
+		printf("\n### Potential gain from compression (%d-bit offsets) "
+		"###\n", (j + 1) * 16);
+		for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
+			const double result = results1[i] + results2[i];
+			const double result_comp = results1_comp[j][i] +
+				results2_comp[j][i];
+			const double gain = 100 - (result_comp / result) * 100;
+
+			printf("Gain of %5.1F%% for burst of %-3u elems\n",
+					gain, bulk_sizes[i]);
+		}
+	}
+
+	return ret;
+}
+
 static int
 test_ring_perf(void)
 {
@@ -576,6 +680,10 @@ test_ring_perf(void)
 	if (test_ring_perf_esize(16) == -1)
 		return -1;

+	/* Test for performance gain of compression */
+	if (test_ring_perf_compression() == -1)
+		return -1;
+
 	return 0;
 }

--
2.25.1

[PATCH v6 3/4] docs: add pointer compression to the EAL guide

[Paul Szczepanek]

Documentation added in the EAL guide for the new
utility functions for pointer compression
showing example code and potential usecases.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .../prog_guide/env_abstraction_layer.rst      | 142 ++++++++++++++++++
 1 file changed, 142 insertions(+)

diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 6debf54efb..f04d032442 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -1192,3 +1192,145 @@ will not be deallocated.

 Any successful deallocation event will trigger a callback, for which user
 applications and other DPDK subsystems can register.
+
+.. _pointer_compression:
+
+Pointer Compression
+-------------------
+
+Use ``rte_ptr_compress_16()`` and ``rte_ptr_decompress_16()`` to compress and
+decompress pointers into 16-bit offsets. Use ``rte_ptr_compress_32()`` and
+``rte_ptr_decompress_32()`` to compress and decompress pointers into 32-bit
+offsets.
+
+Compression takes advantage of the fact that pointers are usually located in a
+limited memory region (like a mempool). By converting them to offsets from a
+base memory address they can be stored in fewer bytes. How many bytes are needed
+to store the offset is dictated by the memory region size and alignment of
+objects the pointers point to.
+
+For example, a pointer which is part of a 4GB memory pool can be stored as 32
+bit offset. If the pointer points to memory that is 8 bytes aligned then 3 bits
+can be dropped from the offset and a 32GB memory pool can now fit in 32 bits.
+
+For performance reasons these requirements are not enforced programmatically.
+The programmer is responsible for ensuring that the combination of distance
+from the base pointer and memory alignment allow for storing of the offset in
+the number of bits indicated by the function name (16 or 32). Start of mempool
+memory would be a good candidate for the base pointer. Otherwise any pointer
+that precedes all pointers, is close enough and has the same alignment as the
+pointers being compressed will work.
+
+.. note::
+
+    Performance gains depend on the batch size of pointers and CPU capabilities
+    such as vector extensions. It's important to measure the performance
+    increase on target hardware. A test called ``ring_perf_autotest`` in
+    ``dpdk-test`` can provide the measurements.
+
+Example usage
+~~~~~~~~~~~~~
+
+In this example we send pointers between two cores through a ring. While this
+is a realistic use case the code is simplified for demonstration purposes and
+does not have error handling.
+
+.. code-block:: c
+
+    #include <rte_launch.h>
+    #include <rte_ring.h>
+    #include <rte_ring_elem.h>
+    #include <rte_ptr_compress.h>
+
+    #define ITEMS_ARRAY_SIZE (1024)
+    #define BATCH_SIZE (128)
+    #define ALIGN_EXPONENT (3)
+    #define ITEM_ALIGN (1<<ALIGN_EXPONENT)
+    #define CORE_SEND (1)
+    #define CORE_RECV (2)
+
+    struct item {
+      int a;
+    } __rte_aligned(ITEM_ALIGN);
+
+    static struct item items[ITEMS_ARRAY_SIZE] = {0};
+    static struct rte_ring *ring = NULL;
+
+    static int
+    send_compressed(void *args)
+    {
+      struct item *ptrs_send[BATCH_SIZE] = {0};
+      unsigned int n_send = 0;
+      struct rte_ring_zc_data zcd = {0};
+
+      /* in this example we only fill the ptrs_send once and reuse */
+      for (;n_send < BATCH_SIZE; n_send++)
+        ptrs_send[n_send] = &items[n_send];
+
+      for(;;) {
+        n_send = rte_ring_enqueue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        /* compress ptrs_send into offsets */
+        rte_ptr_compress_32(items, /* base pointer */
+          ptrs_send, /* source array to be compressed */
+          zcd.ptr1, /* destination array to store offsets */
+          zcd.n1, /* how many pointers to compress */
+          ALIGN_EXPONENT /* how many bits can we drop from the offset */);
+
+        if (zcd.ptr2 != NULL)
+          rte_ptr_compress_32(items, ptrs_send + zcd.n1,
+            zcd.ptr2, n_send - zcd.n1, ALIGN_EXPONENT);
+
+        rte_ring_enqueue_zc_finish(ring, n_send);
+      }
+      return 1;
+    }
+
+    static int
+    recv_compressed(void *args)
+    {
+      struct item *ptrs_recv[BATCH_SIZE] = {0};
+      unsigned int n_recv;
+      struct rte_ring_zc_data zcd = {0};
+
+      for(;;) {
+        /* receive compressed pointers from the ring */
+        n_recv = rte_ring_dequeue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        rte_ptr_decompress_32(items, /* base pointer */
+          zcd.ptr1, /* source array to decompress */
+          ptrs_recv, /* destination array to store pointers */
+          zcd.n1, /* how many pointers to decompress */
+          ALIGN_EXPONENT /* how many bits were dropped from the offset */);
+
+        /* handle the potential secondary buffer (caused by ring boundary) */
+        if (zcd.ptr2 != NULL)
+          rte_ptr_decompress_32(items,
+            zcd.ptr2,
+            ptrs_recv + zcd.n1,
+            n_recv - zcd.n1,
+            ALIGN_EXPONENT);
+
+        rte_ring_dequeue_zc_finish(ring, n_recv);
+
+        /* ptrs_recv contains what ptrs_send contained in the other thread */
+        /* (...) */
+      }
+      return 1;
+    }
+
+    void
+    compression_example(void)
+    {
+      ring = rte_ring_create_elem(
+        "COMPR_PTRS", sizeof(uint32_t),
+        1024, rte_socket_id(),
+        RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+      rte_eal_remote_launch(send_compressed, NULL, CORE_SEND);
+      rte_eal_remote_launch(recv_compressed, NULL, CORE_RECV);
+
+      for(;;) {}
+    }
--
2.25.1

[PATCH v6 4/4] test: add unit test for ptr compression

[Paul Szczepanek]

Test compresses and decompresses pointers with various combinations
of memory regions and alignments and verifies the pointers are
recovered correctly.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 app/test/meson.build             |   1 +
 app/test/test_eal_ptr_compress.c | 108 +++++++++++++++++++++++++++++++
 2 files changed, 109 insertions(+)
 create mode 100644 app/test/test_eal_ptr_compress.c

diff --git a/app/test/meson.build b/app/test/meson.build
index 4183d66b0e..3e172b154d 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -66,6 +66,7 @@ source_file_deps = {
     'test_dmadev_api.c': ['dmadev'],
     'test_eal_flags.c': [],
     'test_eal_fs.c': [],
+    'test_eal_ptr_compress.c': [],
     'test_efd.c': ['efd', 'net'],
     'test_efd_perf.c': ['efd', 'hash'],
     'test_errno.c': [],
diff --git a/app/test/test_eal_ptr_compress.c b/app/test/test_eal_ptr_compress.c
new file mode 100644
index 0000000000..c1c9a98be7
--- /dev/null
+++ b/app/test/test_eal_ptr_compress.c
@@ -0,0 +1,108 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2014 Intel Corporation
+ */
+
+#include "test.h"
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_ptr_compress.h>
+
+#define MAX_ALIGN_EXPONENT 3
+#define PTRS_SIZE 16
+#define NUM_BASES 2
+#define NUM_REGIONS 4
+#define MAX_32BIT_REGION ((uint64_t)UINT32_MAX + 1)
+#define MAX_16BIT_REGION (UINT16_MAX + 1)
+
+static int
+test_eal_ptr_compress_params(
+	void *base,
+	uint64_t mem_sz,
+	unsigned int align_exp,
+	unsigned int num_ptrs,
+	bool use_32_bit)
+{
+	unsigned int i;
+	unsigned int align = 1 << align_exp;
+	void *ptrs[PTRS_SIZE] = {0};
+	void *ptrs_out[PTRS_SIZE] = {0};
+	uint32_t offsets32[PTRS_SIZE] = {0};
+	uint16_t offsets16[PTRS_SIZE] = {0};
+
+	for (i = 0; i < num_ptrs; i++) {
+		/* make pointers point at memory in steps of align */
+		/* alternate steps from the start and end of memory region */
+		if ((i & 1) == 1)
+			ptrs[i] = (char *)base + mem_sz - i * align;
+		else
+			ptrs[i] = (char *)base + i * align;
+	}
+
+	if (use_32_bit) {
+		rte_ptr_compress_32(base, ptrs, offsets32, num_ptrs, align_exp);
+		rte_ptr_decompress_32(base, offsets32, ptrs_out, num_ptrs,
+				align_exp);
+	} else {
+		rte_ptr_compress_16(base, ptrs, offsets16, num_ptrs, align_exp);
+		rte_ptr_decompress_16(base, offsets16, ptrs_out, num_ptrs,
+				align_exp);
+	}
+
+	TEST_ASSERT_BUFFERS_ARE_EQUAL(ptrs, ptrs_out, sizeof(void *) * num_ptrs,
+		"Decompressed pointers corrupted\nbase pointer: %p, "
+		"memory region size: %" PRIu64 ", alignment exponent: %u, "
+		"num of pointers: %u, using %s offsets",
+		base, mem_sz, align_exp, num_ptrs,
+		use_32_bit ? "32-bit" : "16-bit");
+
+	return 0;
+}
+
+static int
+test_eal_ptr_compress(void)
+{
+	unsigned int j, k, n;
+	int ret = 0;
+	void * const bases[NUM_BASES] = { (void *)0, (void *)UINT16_MAX };
+	/* maximum size for pointers aligned by consecutive powers of 2 */
+	const uint64_t region_sizes_16[NUM_REGIONS] = {
+		MAX_16BIT_REGION,
+		MAX_16BIT_REGION * 2,
+		MAX_16BIT_REGION * 4,
+		MAX_16BIT_REGION * 8,
+	};
+	const uint64_t region_sizes_32[NUM_REGIONS] = {
+		MAX_32BIT_REGION,
+		MAX_32BIT_REGION * 2,
+		MAX_32BIT_REGION * 4,
+		MAX_32BIT_REGION * 8,
+	};
+
+	for (j = 0; j < NUM_REGIONS; j++) {
+		for (k = 0; k < NUM_BASES; k++) {
+			for (n = 1; n < PTRS_SIZE; n++) {
+				ret |= test_eal_ptr_compress_params(
+					bases[k],
+					region_sizes_16[j],
+					j /* exponent of alignment */,
+					n,
+					false
+				);
+				ret |= test_eal_ptr_compress_params(
+					bases[k],
+					region_sizes_32[j],
+					j /* exponent of alignment */,
+					n,
+					true
+				);
+				if (ret != 0)
+					return ret;
+			}
+		}
+	}
+
+	return ret;
+}
+
+REGISTER_FAST_TEST(eal_ptr_compress_autotest, true, true, test_eal_ptr_compress);
--
2.25.1

[PATCH v7 0/4] add pointer compression API

[Paul Szczepanek]

This patchset is proposing adding a new EAL header with utility functions
that allow compression of arrays of pointers.

When passing caches full of pointers between threads, memory containing
the pointers is copied multiple times which is especially costly between
cores. A compression method will allow us to shrink the memory size
copied.

The compression takes advantage of the fact that pointers are usually
located in a limited memory region (like a mempool). We can compress them
by converting them to offsets from a base memory address.

Offsets can be stored in fewer bytes (dictated by the memory region size
and alignment of the pointer). For example: an 8 byte aligned pointer
which is part of a 32GB memory pool can be stored in 4 bytes. The API is
very generic and does not assume mempool pointers, any pointer can be
passed in.

Compression is based on few and fast operations and especially with vector
instructions leveraged creates minimal overhead.

The API accepts and returns arrays because the overhead means it only is
worth it when done in bulk.

Test is added that shows potential performance gain from compression. In
this test an array of pointers is passed through a ring between two cores.
It shows the gain which is dependent on the bulk operation size. In this
synthetic test run on ampere altra a substantial (up to 25%) performance
gain is seen if done in bulk size larger than 32. At 32 it breaks even and
lower sizes create a small (less than 5%) slowdown due to overhead.

In a more realistic mock application running the l3 forwarding dpdk
example that works in pipeline mode on two cores this translated into a
~5% throughput increase on an ampere altra.

v2:
* addressed review comments (style, explanations and typos)
* lowered bulk iterations closer to original numbers to keep runtime short
* fixed pointer size warning on 32-bit arch
v3:
* added 16-bit versions of compression functions and tests
* added documentation of these new utility functions in the EAL guide
v4:
* added unit test
* fix bug in NEON implementation of 32-bit decompress
v5:
* disable NEON and SVE implementation on AARCH32 due to wrong pointer size
v6:
* added example usage to commit message of the initial commit
v7:
* rebase to remove clashing mailmap changes

Paul Szczepanek (4):
  eal: add pointer compression functions
  test: add pointer compress tests to ring perf test
  docs: add pointer compression to the EAL guide
  test: add unit test for ptr compression

 app/test/meson.build                          |   1 +
 app/test/test_eal_ptr_compress.c              | 108 ++++++
 app/test/test_ring.h                          |  94 ++++-
 app/test/test_ring_perf.c                     | 354 ++++++++++++------
 .../prog_guide/env_abstraction_layer.rst      | 142 +++++++
 lib/eal/include/meson.build                   |   1 +
 lib/eal/include/rte_ptr_compress.h            | 266 +++++++++++++
 7 files changed, 842 insertions(+), 124 deletions(-)
 create mode 100644 app/test/test_eal_ptr_compress.c
 create mode 100644 lib/eal/include/rte_ptr_compress.h

--
2.25.1

[PATCH v7 1/4] eal: add pointer compression functions

[Paul Szczepanek]

Add a new utility header for compressing pointers. The provided
functions can store pointers in 32-bit offsets.

The compression takes advantage of the fact that pointers are
usually located in a limited memory region (like a mempool).
We can compress them by converting them to offsets from a base
memory address. Offsets can be stored in fewer bytes (dictated
by the memory region size and alignment of the pointer).
For example: an 8 byte aligned pointer which is part of a 32GB
memory pool can be stored in 4 bytes.

This can be used for example when passing caches full of pointers
between threads. Memory containing the pointers is copied multiple
times which is especially costly between cores. This compression
method will allow us to shrink the memory size copied. Further
commits add a test to evaluate the effectiveness of this approach.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 lib/eal/include/meson.build        |   1 +
 lib/eal/include/rte_ptr_compress.h | 266 +++++++++++++++++++++++++++++
 2 files changed, 267 insertions(+)
 create mode 100644 lib/eal/include/rte_ptr_compress.h

diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index e94b056d46..ce2c733633 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -36,6 +36,7 @@ headers += files(
         'rte_pci_dev_features.h',
         'rte_per_lcore.h',
         'rte_pflock.h',
+	'rte_ptr_compress.h',
         'rte_random.h',
         'rte_reciprocal.h',
         'rte_seqcount.h',
diff --git a/lib/eal/include/rte_ptr_compress.h b/lib/eal/include/rte_ptr_compress.h
new file mode 100644
index 0000000000..47a72e4213
--- /dev/null
+++ b/lib/eal/include/rte_ptr_compress.h
@@ -0,0 +1,266 @@
+/* SPDX-License-Identifier: BSD-shift-Clause
+ * Copyright(c) 2023 Arm Limited
+ */
+
+#ifndef RTE_PTR_COMPRESS_H
+#define RTE_PTR_COMPRESS_H
+
+/**
+ * @file
+ * Pointer compression and decompression functions.
+ *
+ * When passing arrays full of pointers between threads, memory containing
+ * the pointers is copied multiple times which is especially costly between
+ * cores. These functions allow us to compress the pointers.
+ *
+ * Compression takes advantage of the fact that pointers are usually located in
+ * a limited memory region (like a mempool). We compress them by converting them
+ * to offsets from a base memory address. Offsets can be stored in fewer bytes.
+ *
+ * The compression functions come in two varieties: 32-bit and 16-bit.
+ *
+ * To determine how many bits are needed to compress the pointer calculate
+ * the biggest offset possible (highest value pointer - base pointer)
+ * and shift the value right according to alignment (shift by exponent of the
+ * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by
+ * 3, etc.). The resulting value must fit in either 32 or 16 bits.
+ *
+ * For usage example and further explanation please see "Pointer Compression" in
+ * doc/guides/prog_guide/env_abstraction_layer.rst
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_vect.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Compress pointers into 32-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 32 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 32GB of the base pointer. If there is no such alignment guarantee they
+ * must be within 4GB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_32(void *ptr_base, void **src_table,
+		uint32_t *dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1w(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	/* right shift is done by left shifting by negative int */
+	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_ptr_table = vld1q_u64((const uint64_t *)src_table + i);
+		v_ptr_table = vsubq_u64(v_ptr_table, v_ptr_base);
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		vst1_u32(dest_table + i, vqmovn_u64(v_ptr_table));
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT32_MAX);
+		dest_table[i] = (uint32_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 32-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_32(void *ptr_base, uint32_t *src_table,
+		void **dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1uw_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	int64x2_t v_shift = vdupq_n_s64(bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_ptr_table = vmovl_u32(vld1_u32(src_table + i));
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		v_ptr_table = vaddq_u64(v_ptr_table, v_ptr_base);
+		vst1q_u64((uint64_t *)dest_table + i, v_ptr_table);
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+/**
+ * Compress pointers into 16-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 16 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 256KB of the base pointer. If there is no such alignment guarantee
+ * they must be within 64KB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_16(void *ptr_base, void **src_table,
+		uint16_t *dest_table, unsigned int n, unsigned int bit_shift)
+{
+
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1h(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT16_MAX);
+		dest_table[i] = (uint16_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 16-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_16(void *ptr_base, uint16_t *src_table,
+		void **dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1uh_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_PTR_COMPRESS_H */
--
2.25.1

Re: [PATCH v7 1/4] eal: add pointer compression functions

[David Marchand]

Hello Paul,

Before sending the v8, please fix small issues I found.


On Fri, Mar 1, 2024 at 11:21 AM Paul Szczepanek <paul.szczepanek@arm.com> wrote:
>
> Add a new utility header for compressing pointers. The provided
> functions can store pointers in 32-bit offsets.
>
> The compression takes advantage of the fact that pointers are
> usually located in a limited memory region (like a mempool).
> We can compress them by converting them to offsets from a base
> memory address. Offsets can be stored in fewer bytes (dictated
> by the memory region size and alignment of the pointer).
> For example: an 8 byte aligned pointer which is part of a 32GB
> memory pool can be stored in 4 bytes.
>
> This can be used for example when passing caches full of pointers
> between threads. Memory containing the pointers is copied multiple
> times which is especially costly between cores. This compression
> method will allow us to shrink the memory size copied. Further
> commits add a test to evaluate the effectiveness of this approach.
>
> Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> ---
>  lib/eal/include/meson.build        |   1 +
>  lib/eal/include/rte_ptr_compress.h | 266 +++++++++++++++++++++++++++++
>  2 files changed, 267 insertions(+)
>  create mode 100644 lib/eal/include/rte_ptr_compress.h
>
> diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
> index e94b056d46..ce2c733633 100644
> --- a/lib/eal/include/meson.build
> +++ b/lib/eal/include/meson.build
> @@ -36,6 +36,7 @@ headers += files(
>          'rte_pci_dev_features.h',
>          'rte_per_lcore.h',
>          'rte_pflock.h',
> +       'rte_ptr_compress.h',

I suppose it will not be applicable in v8 but:

$ ./devtools/check-meson.py
Error parsing lib/eal/include/meson.build:38, got some tabulation
Error: Incorrect indent at lib/eal/include/meson.build:39


>          'rte_random.h',
>          'rte_reciprocal.h',
>          'rte_seqcount.h',
> diff --git a/lib/eal/include/rte_ptr_compress.h b/lib/eal/include/rte_ptr_compress.h
> new file mode 100644
> index 0000000000..47a72e4213
> --- /dev/null
> +++ b/lib/eal/include/rte_ptr_compress.h
> @@ -0,0 +1,266 @@
> +/* SPDX-License-Identifier: BSD-shift-Clause

Not sure what the "shift" license is.
Typo?

DPDK uses BSD-3-Clause license.

> + * Copyright(c) 2023 Arm Limited

Should be 2024 now.

> + */
> +
> +#ifndef RTE_PTR_COMPRESS_H
> +#define RTE_PTR_COMPRESS_H

For the rest, I'll trust reviewers.


-- 
David Marchand

[PATCH v7 2/4] test: add pointer compress tests to ring perf test

[Paul Szczepanek]

Add a test that runs a zero copy burst enqueue and dequeue on a ring
of raw pointers and compressed pointers at different burst sizes to
showcase performance benefits of newly added pointer compression APIs.

Refactored threading code to pass more parameters to threads to
reuse existing code. Added more bulk sizes to showcase their effects
on compression. Adjusted loop iteration numbers to take into account
bulk sizes to keep runtime constant (instead of number of operations).

Adjusted old printfs to match new ones which have aligned numbers.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 app/test/test_ring.h      |  94 +++++++++-
 app/test/test_ring_perf.c | 354 +++++++++++++++++++++++++-------------
 2 files changed, 324 insertions(+), 124 deletions(-)

diff --git a/app/test/test_ring.h b/app/test/test_ring.h
index 45c263f3ff..3b00f2465d 100644
--- a/app/test/test_ring.h
+++ b/app/test/test_ring.h
@@ -1,10 +1,12 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */

 #include <rte_malloc.h>
 #include <rte_ring.h>
 #include <rte_ring_elem.h>
+#include <rte_memcpy.h>
+#include <rte_ptr_compress.h>

 /* API type to call
  * rte_ring_<sp/mp or sc/mc>_enqueue_<bulk/burst>
@@ -25,6 +27,10 @@
 #define TEST_RING_ELEM_BULK 16
 #define TEST_RING_ELEM_BURST 32

+#define TEST_RING_ELEM_BURST_ZC 64
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 128
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 256
+
 #define TEST_RING_IGNORE_API_TYPE ~0U

 /* This function is placed here as it is required for both
@@ -101,6 +107,9 @@ static inline unsigned int
 test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -152,6 +161,46 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n,
 								NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy(zcd.ptr2,
+						(char *)obj + zcd.n1 * esize,
+						(ret - zcd.n1) * esize);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_16(0, obj, zcd.ptr1, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_16(0,
+						obj + (zcd.n1 * 2),
+						zcd.ptr2,
+						(ret - zcd.n1) * 2, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_32(0, obj, zcd.ptr1, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_32(0, obj + zcd.n1,
+						zcd.ptr2, ret - zcd.n1, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
@@ -162,6 +211,9 @@ static inline unsigned int
 test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -213,6 +265,46 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mc_dequeue_burst_elem(r, obj, esize,
 								n, NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy((char *)obj + zcd.n1 * esize,
+						zcd.ptr2,
+						(ret - zcd.n1) * esize);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_16(0, zcd.ptr1, obj, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_16(0, zcd.ptr2,
+						obj + zcd.n1,
+						(ret - zcd.n1) * 2,
+						3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_32(0, zcd.ptr1, obj, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_32(0, zcd.ptr2,
+						obj + zcd.n1, ret - zcd.n1, 3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c
index d7c5a4c30b..8392d0c781 100644
--- a/app/test/test_ring_perf.c
+++ b/app/test/test_ring_perf.c
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2010-2014 Intel Corporation
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */


@@ -22,13 +22,13 @@

 #define RING_NAME "RING_PERF"
 #define RING_SIZE 4096
-#define MAX_BURST 32
+#define MAX_BURST 256

 /*
  * the sizes to enqueue and dequeue in testing
  * (marked volatile so they won't be seen as compile-time constants)
  */
-static const volatile unsigned bulk_sizes[] = { 8, 32 };
+static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128, 256 };

 struct lcore_pair {
 	unsigned c1, c2;
@@ -43,26 +43,30 @@ test_ring_print_test_string(unsigned int api_type, int esize,
 	if (esize == -1)
 		printf("legacy APIs");
 	else
-		printf("elem APIs: element size %dB", esize);
+		printf("elem APIs (size:%2dB)", esize);

 	if (api_type == TEST_RING_IGNORE_API_TYPE)
 		return;

 	if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF)
-		printf(": default enqueue/dequeue: ");
+		printf(" - default enqueue/dequeue");
 	else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC)
-		printf(": SP/SC: ");
+		printf(" - SP/SC");
 	else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC)
-		printf(": MP/MC: ");
+		printf(" - MP/MC");

 	if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE)
-		printf("single: ");
+		printf(" - single - ");
 	else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK)
-		printf("bulk (size: %u): ", bsz);
+		printf(" - bulk (n:%-3u) - ", bsz);
 	else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST)
-		printf("burst (size: %u): ", bsz);
+		printf(" - burst (n:%-3u) - ", bsz);
+	else if ((api_type & (TEST_RING_ELEM_BURST_ZC |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0)
+		printf(" - burst zero copy (n:%-3u) - ", bsz);

-	printf("%.2F\n", value);
+	printf("cycles per elem: %.3F\n", value);
 }

 /**** Functions to analyse our core mask to get cores for different tests ***/
@@ -144,28 +148,33 @@ static void
 test_empty_dequeue(struct rte_ring *r, const int esize,
 			const unsigned int api_type)
 {
-	const unsigned int iter_shift = 26;
+	const unsigned int iter_shift = 29;
 	const unsigned int iterations = 1 << iter_shift;
 	unsigned int i = 0;
 	void *burst[MAX_BURST];

+	const unsigned int bulk_iterations = iterations / bulk_sizes[0];
 	const uint64_t start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	for (i = 0; i < bulk_iterations; i++)
 		test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type);
 	const uint64_t end = rte_rdtsc();

 	test_ring_print_test_string(api_type, esize, bulk_sizes[0],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 }

-/*
- * for the separate enqueue and dequeue threads they take in one param
- * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc
- */
-struct thread_params {
+/* describes the ring used by the enqueue and dequeue thread */
+struct ring_params {
 	struct rte_ring *r;
-	unsigned size;        /* input value, the burst size */
-	double spsc, mpmc;    /* output value, the single or multi timings */
+	unsigned int elem_size;
+	unsigned int bulk_sizes_i; /* index into bulk_size array */
+	unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */
+};
+
+/* Used do specify enqueue and dequeue ring operations and their results */
+struct thread_params {
+	struct ring_params *ring_params;
+	double *results; /* result array size must be equal to bulk_sizes */
 };

 /*
@@ -174,16 +183,15 @@ struct thread_params {
  * flag == 1 -> dequeue
  */
 static __rte_always_inline int
-enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
-	struct thread_params *p)
+enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 15;
+	const unsigned int iter_shift = 22;
 	const unsigned int iterations = 1 << iter_shift;
-	struct rte_ring *r = p->r;
-	unsigned int bsize = p->size;
 	unsigned int i;
 	void *burst = NULL;
+	unsigned int n_remaining;
+	const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i];

 #ifdef RTE_USE_C11_MEM_MODEL
 	if (__atomic_fetch_add(&lcore_count, 1, __ATOMIC_RELAXED) + 1 != 2)
@@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 		while(lcore_count != 2)
 			rte_pause();

-	burst = test_ring_calloc(MAX_BURST, esize);
+	burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size);
 	if (burst == NULL)
 		return -1;

 	const uint64_t sp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	const unsigned int bulk_iterations = iterations / bulk_n;
+	for (i = 0; i < bulk_iterations; i++) {
+		n_remaining = bulk_n;
 		do {
 			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_enqueue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_dequeue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			if (ret == 0)
 				rte_pause();
-		} while (!ret);
+			else
+				n_remaining -= ret;
+		} while (n_remaining > 0);
+	}
 	const uint64_t sp_end = rte_rdtsc();

-	const uint64_t mp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
-		do {
-			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			if (ret == 0)
-				rte_pause();
-		} while (!ret);
-	const uint64_t mp_end = rte_rdtsc();
+	p->results[p->ring_params->bulk_sizes_i] =
+			((double)sp_end - sp_start) / iterations;

-	p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize);
-	p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize);
 	return 0;
 }

@@ -243,15 +245,7 @@ enqueue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(0, -1, params);
-}
-
-static int
-enqueue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(0, 16, params);
+	return enqueue_dequeue_bulk_helper(0, params);
 }

 /*
@@ -263,15 +257,7 @@ dequeue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(1, -1, params);
-}
-
-static int
-dequeue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(1, 16, params);
+	return enqueue_dequeue_bulk_helper(1, params);
 }

 /*
@@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p)
  * used to measure ring perf between hyperthreads, cores and sockets.
  */
 static int
-run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize)
+run_on_core_pair(struct lcore_pair *cores,
+		struct thread_params *param1, struct thread_params *param2)
 {
-	lcore_function_t *f1, *f2;
-	struct thread_params param1 = {0}, param2 = {0};
 	unsigned i;
-
-	if (esize == -1) {
-		f1 = enqueue_bulk;
-		f2 = dequeue_bulk;
-	} else {
-		f1 = enqueue_bulk_16B;
-		f2 = dequeue_bulk_16B;
-	}
+	struct ring_params *ring_params = param1->ring_params;

 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		lcore_count = 0;
-		param1.size = param2.size = bulk_sizes[i];
-		param1.r = param2.r = r;
+		ring_params->bulk_sizes_i = i;
 		if (cores->c1 == rte_get_main_lcore()) {
-			rte_eal_remote_launch(f2, &param2, cores->c2);
-			f1(&param1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
+			enqueue_bulk(param1);
 			rte_eal_wait_lcore(cores->c2);
 		} else {
-			rte_eal_remote_launch(f1, &param1, cores->c1);
-			rte_eal_remote_launch(f2, &param2, cores->c2);
+			rte_eal_remote_launch(enqueue_bulk, param1, cores->c1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
 			if (rte_eal_wait_lcore(cores->c1) < 0)
 				return -1;
 			if (rte_eal_wait_lcore(cores->c2) < 0)
 				return -1;
 		}
 		test_ring_print_test_string(
-			TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.spsc + param2.spsc);
-		test_ring_print_test_string(
-			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.mpmc + param2.mpmc);
+				ring_params->ring_flags,
+				ring_params->elem_size,
+				bulk_sizes[i],
+				param1->results[i] + param2->results[i]);
 	}

 	return 0;
@@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 	uint64_t hz = rte_get_timer_hz();
 	uint64_t lcount = 0;
 	const unsigned int lcore = rte_lcore_id();
-	struct thread_params *params = p;
+	struct ring_params *ring_params = p->ring_params;
 	void *burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -346,9 +322,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize)

 	begin = rte_get_timer_cycles();
 	while (time_diff < hz * TIME_MS / 1000) {
-		test_ring_enqueue(params->r, burst, esize, params->size,
+		test_ring_enqueue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
-		test_ring_dequeue(params->r, burst, esize, params->size,
+		test_ring_dequeue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 		lcount++;
 		time_diff = rte_get_timer_cycles() - begin;
@@ -380,7 +358,8 @@ static int
 run_on_all_cores(struct rte_ring *r, const int esize)
 {
 	uint64_t total;
-	struct thread_params param;
+	struct ring_params ring_params = {0};
+	struct thread_params params = { .ring_params = &ring_params };
 	lcore_function_t *lcore_f;
 	unsigned int i, c;

@@ -389,21 +368,20 @@ run_on_all_cores(struct rte_ring *r, const int esize)
 	else
 		lcore_f = load_loop_fn_16B;

-	memset(&param, 0, sizeof(struct thread_params));
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		total = 0;
 		printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]);
-		param.size = bulk_sizes[i];
-		param.r = r;
+		params.ring_params->bulk_sizes_i = i;
+		params.ring_params->r = r;

 		/* clear synchro and start workers */
 		__atomic_store_n(&synchro, 0, __ATOMIC_RELAXED);
-		if (rte_eal_mp_remote_launch(lcore_f, &param, SKIP_MAIN) < 0)
+		if (rte_eal_mp_remote_launch(lcore_f, &params, SKIP_MAIN) < 0)
 			return -1;

 		/* start synchro and launch test on main */
 		__atomic_store_n(&synchro, 1, __ATOMIC_RELAXED);
-		lcore_f(&param);
+		lcore_f(&params);

 		rte_eal_mp_wait_lcore();

@@ -462,9 +440,9 @@ static int
 test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	const unsigned int api_type)
 {
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 26;
 	const unsigned int iterations = 1 << iter_shift;
-	unsigned int sz, i = 0;
+	unsigned int sz, i;
 	void **burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -472,17 +450,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 		return -1;

 	for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) {
+		const unsigned int n = iterations / bulk_sizes[sz];
 		const uint64_t start = rte_rdtsc();
-		for (i = 0; i < iterations; i++) {
+		for (i = 0; i < n; i++) {
 			test_ring_enqueue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 			test_ring_dequeue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 		}
 		const uint64_t end = rte_rdtsc();

 		test_ring_print_test_string(api_type, esize, bulk_sizes[sz],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 	}

 	rte_free(burst);
@@ -490,12 +469,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	return 0;
 }

+static __rte_always_inline int
+test_ring_perf_esize_run_on_two_cores(
+		struct thread_params *param1, struct thread_params *param2)
+{
+	struct lcore_pair cores;
+
+	if (get_two_hyperthreads(&cores) == 0) {
+		printf("\n### Testing using two hyperthreads ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing using two physical cores ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_sockets(&cores) == 0) {
+		printf("\n### Testing using two NUMA nodes ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	return 0;
+}
+
 /* Run all tests for a given element size */
 static __rte_always_inline int
 test_ring_perf_esize(const int esize)
 {
-	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
+	double results_enq[RTE_DIM(bulk_sizes)];
+	double results_deq[RTE_DIM(bulk_sizes)];
+	struct ring_params ring_params = {
+			.elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results_enq };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results_deq };

 	/*
 	 * Performance test for legacy/_elem APIs
@@ -535,22 +545,13 @@ test_ring_perf_esize(const int esize)
 	test_empty_dequeue(r, esize,
 			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);

-	if (get_two_hyperthreads(&cores) == 0) {
-		printf("\n### Testing using two hyperthreads ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.r = r;

-	if (get_two_cores(&cores) == 0) {
-		printf("\n### Testing using two physical cores ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
-	if (get_two_sockets(&cores) == 0) {
-		printf("\n### Testing using two NUMA nodes ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
+
+	ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);

 	printf("\n### Testing using all worker nodes ###\n");
 	if (run_on_all_cores(r, esize) < 0)
@@ -566,6 +567,109 @@ test_ring_perf_esize(const int esize)
 	return -1;
 }

+
+static __rte_always_inline int
+test_ring_perf_compression(void)
+{
+	double results1[RTE_DIM(bulk_sizes)];
+	double results2[RTE_DIM(bulk_sizes)];
+	double results1_comp[2][RTE_DIM(bulk_sizes)];
+	double results2_comp[2][RTE_DIM(bulk_sizes)];
+
+	struct lcore_pair cores;
+	int ret = -1;
+	unsigned int i, j;
+	struct ring_params ring_params = { .elem_size = sizeof(void *) };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results1 };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results2 };
+
+	printf("\n### Testing compression gain ###");
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(void *),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy ###\n");
+		ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	/* rings allow only multiples of 4 as sizes,
+	 * we allocate size 4 despite only using 2 bytes
+	 * and use half of RING_SIZE as the number of elements
+	 */
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE / 2, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[0];
+	param2.results = results2_comp[0];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (16b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[1];
+	param2.results = results2_comp[1];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (32b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	for (j = 0; j < 2; j++) {
+		printf("\n### Potential gain from compression (%d-bit offsets) "
+		"###\n", (j + 1) * 16);
+		for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
+			const double result = results1[i] + results2[i];
+			const double result_comp = results1_comp[j][i] +
+				results2_comp[j][i];
+			const double gain = 100 - (result_comp / result) * 100;
+
+			printf("Gain of %5.1F%% for burst of %-3u elems\n",
+					gain, bulk_sizes[i]);
+		}
+	}
+
+	return ret;
+}
+
 static int
 test_ring_perf(void)
 {
@@ -576,6 +680,10 @@ test_ring_perf(void)
 	if (test_ring_perf_esize(16) == -1)
 		return -1;

+	/* Test for performance gain of compression */
+	if (test_ring_perf_compression() == -1)
+		return -1;
+
 	return 0;
 }

--
2.25.1

Re: [PATCH v7 2/4] test: add pointer compress tests to ring perf test

[David Marchand]

On Fri, Mar 1, 2024 at 11:21 AM Paul Szczepanek <paul.szczepanek@arm.com> wrote:
>
> Add a test that runs a zero copy burst enqueue and dequeue on a ring
> of raw pointers and compressed pointers at different burst sizes to
> showcase performance benefits of newly added pointer compression APIs.
>
> Refactored threading code to pass more parameters to threads to
> reuse existing code. Added more bulk sizes to showcase their effects
> on compression. Adjusted loop iteration numbers to take into account
> bulk sizes to keep runtime constant (instead of number of operations).
>
> Adjusted old printfs to match new ones which have aligned numbers.
>
> Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> ---
>  app/test/test_ring.h      |  94 +++++++++-
>  app/test/test_ring_perf.c | 354 +++++++++++++++++++++++++-------------
>  2 files changed, 324 insertions(+), 124 deletions(-)
>
> diff --git a/app/test/test_ring.h b/app/test/test_ring.h
> index 45c263f3ff..3b00f2465d 100644
> --- a/app/test/test_ring.h
> +++ b/app/test/test_ring.h
> @@ -1,10 +1,12 @@
>  /* SPDX-License-Identifier: BSD-3-Clause
> - * Copyright(c) 2019 Arm Limited
> + * Copyright(c) 2019-2023 Arm Limited

Thomas insisted recently that updating copyrights in this manner is not needed.
https://matija.suklje.name/how-and-why-to-properly-write-copyright-statements-in-your-code#why-not-bump-the-year-on-change


-- 
David Marchand

[PATCH v7 3/4] docs: add pointer compression to the EAL guide

[Paul Szczepanek]

Documentation added in the EAL guide for the new
utility functions for pointer compression
showing example code and potential usecases.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .../prog_guide/env_abstraction_layer.rst      | 142 ++++++++++++++++++
 1 file changed, 142 insertions(+)

diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 9559c12a98..aaa1665c4b 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -1192,3 +1192,145 @@ will not be deallocated.

 Any successful deallocation event will trigger a callback, for which user
 applications and other DPDK subsystems can register.
+
+.. _pointer_compression:
+
+Pointer Compression
+-------------------
+
+Use ``rte_ptr_compress_16()`` and ``rte_ptr_decompress_16()`` to compress and
+decompress pointers into 16-bit offsets. Use ``rte_ptr_compress_32()`` and
+``rte_ptr_decompress_32()`` to compress and decompress pointers into 32-bit
+offsets.
+
+Compression takes advantage of the fact that pointers are usually located in a
+limited memory region (like a mempool). By converting them to offsets from a
+base memory address they can be stored in fewer bytes. How many bytes are needed
+to store the offset is dictated by the memory region size and alignment of
+objects the pointers point to.
+
+For example, a pointer which is part of a 4GB memory pool can be stored as 32
+bit offset. If the pointer points to memory that is 8 bytes aligned then 3 bits
+can be dropped from the offset and a 32GB memory pool can now fit in 32 bits.
+
+For performance reasons these requirements are not enforced programmatically.
+The programmer is responsible for ensuring that the combination of distance
+from the base pointer and memory alignment allow for storing of the offset in
+the number of bits indicated by the function name (16 or 32). Start of mempool
+memory would be a good candidate for the base pointer. Otherwise any pointer
+that precedes all pointers, is close enough and has the same alignment as the
+pointers being compressed will work.
+
+.. note::
+
+    Performance gains depend on the batch size of pointers and CPU capabilities
+    such as vector extensions. It's important to measure the performance
+    increase on target hardware. A test called ``ring_perf_autotest`` in
+    ``dpdk-test`` can provide the measurements.
+
+Example usage
+~~~~~~~~~~~~~
+
+In this example we send pointers between two cores through a ring. While this
+is a realistic use case the code is simplified for demonstration purposes and
+does not have error handling.
+
+.. code-block:: c
+
+    #include <rte_launch.h>
+    #include <rte_ring.h>
+    #include <rte_ring_elem.h>
+    #include <rte_ptr_compress.h>
+
+    #define ITEMS_ARRAY_SIZE (1024)
+    #define BATCH_SIZE (128)
+    #define ALIGN_EXPONENT (3)
+    #define ITEM_ALIGN (1<<ALIGN_EXPONENT)
+    #define CORE_SEND (1)
+    #define CORE_RECV (2)
+
+    struct item {
+      int a;
+    } __rte_aligned(ITEM_ALIGN);
+
+    static struct item items[ITEMS_ARRAY_SIZE] = {0};
+    static struct rte_ring *ring = NULL;
+
+    static int
+    send_compressed(void *args)
+    {
+      struct item *ptrs_send[BATCH_SIZE] = {0};
+      unsigned int n_send = 0;
+      struct rte_ring_zc_data zcd = {0};
+
+      /* in this example we only fill the ptrs_send once and reuse */
+      for (;n_send < BATCH_SIZE; n_send++)
+        ptrs_send[n_send] = &items[n_send];
+
+      for(;;) {
+        n_send = rte_ring_enqueue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        /* compress ptrs_send into offsets */
+        rte_ptr_compress_32(items, /* base pointer */
+          ptrs_send, /* source array to be compressed */
+          zcd.ptr1, /* destination array to store offsets */
+          zcd.n1, /* how many pointers to compress */
+          ALIGN_EXPONENT /* how many bits can we drop from the offset */);
+
+        if (zcd.ptr2 != NULL)
+          rte_ptr_compress_32(items, ptrs_send + zcd.n1,
+            zcd.ptr2, n_send - zcd.n1, ALIGN_EXPONENT);
+
+        rte_ring_enqueue_zc_finish(ring, n_send);
+      }
+      return 1;
+    }
+
+    static int
+    recv_compressed(void *args)
+    {
+      struct item *ptrs_recv[BATCH_SIZE] = {0};
+      unsigned int n_recv;
+      struct rte_ring_zc_data zcd = {0};
+
+      for(;;) {
+        /* receive compressed pointers from the ring */
+        n_recv = rte_ring_dequeue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        rte_ptr_decompress_32(items, /* base pointer */
+          zcd.ptr1, /* source array to decompress */
+          ptrs_recv, /* destination array to store pointers */
+          zcd.n1, /* how many pointers to decompress */
+          ALIGN_EXPONENT /* how many bits were dropped from the offset */);
+
+        /* handle the potential secondary buffer (caused by ring boundary) */
+        if (zcd.ptr2 != NULL)
+          rte_ptr_decompress_32(items,
+            zcd.ptr2,
+            ptrs_recv + zcd.n1,
+            n_recv - zcd.n1,
+            ALIGN_EXPONENT);
+
+        rte_ring_dequeue_zc_finish(ring, n_recv);
+
+        /* ptrs_recv contains what ptrs_send contained in the other thread */
+        /* (...) */
+      }
+      return 1;
+    }
+
+    void
+    compression_example(void)
+    {
+      ring = rte_ring_create_elem(
+        "COMPR_PTRS", sizeof(uint32_t),
+        1024, rte_socket_id(),
+        RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+      rte_eal_remote_launch(send_compressed, NULL, CORE_SEND);
+      rte_eal_remote_launch(recv_compressed, NULL, CORE_RECV);
+
+      for(;;) {}
+    }
--
2.25.1

[PATCH v7 4/4] test: add unit test for ptr compression

[Paul Szczepanek]

Test compresses and decompresses pointers with various combinations
of memory regions and alignments and verifies the pointers are
recovered correctly.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 app/test/meson.build             |   1 +
 app/test/test_eal_ptr_compress.c | 108 +++++++++++++++++++++++++++++++
 2 files changed, 109 insertions(+)
 create mode 100644 app/test/test_eal_ptr_compress.c

diff --git a/app/test/meson.build b/app/test/meson.build
index 7d909039ae..0d1b777199 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -68,6 +68,7 @@ source_file_deps = {
     'test_dmadev_api.c': ['dmadev'],
     'test_eal_flags.c': [],
     'test_eal_fs.c': [],
+    'test_eal_ptr_compress.c': [],
     'test_efd.c': ['efd', 'net'],
     'test_efd_perf.c': ['efd', 'hash'],
     'test_errno.c': [],
diff --git a/app/test/test_eal_ptr_compress.c b/app/test/test_eal_ptr_compress.c
new file mode 100644
index 0000000000..c1c9a98be7
--- /dev/null
+++ b/app/test/test_eal_ptr_compress.c
@@ -0,0 +1,108 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2014 Intel Corporation
+ */
+
+#include "test.h"
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_ptr_compress.h>
+
+#define MAX_ALIGN_EXPONENT 3
+#define PTRS_SIZE 16
+#define NUM_BASES 2
+#define NUM_REGIONS 4
+#define MAX_32BIT_REGION ((uint64_t)UINT32_MAX + 1)
+#define MAX_16BIT_REGION (UINT16_MAX + 1)
+
+static int
+test_eal_ptr_compress_params(
+	void *base,
+	uint64_t mem_sz,
+	unsigned int align_exp,
+	unsigned int num_ptrs,
+	bool use_32_bit)
+{
+	unsigned int i;
+	unsigned int align = 1 << align_exp;
+	void *ptrs[PTRS_SIZE] = {0};
+	void *ptrs_out[PTRS_SIZE] = {0};
+	uint32_t offsets32[PTRS_SIZE] = {0};
+	uint16_t offsets16[PTRS_SIZE] = {0};
+
+	for (i = 0; i < num_ptrs; i++) {
+		/* make pointers point at memory in steps of align */
+		/* alternate steps from the start and end of memory region */
+		if ((i & 1) == 1)
+			ptrs[i] = (char *)base + mem_sz - i * align;
+		else
+			ptrs[i] = (char *)base + i * align;
+	}
+
+	if (use_32_bit) {
+		rte_ptr_compress_32(base, ptrs, offsets32, num_ptrs, align_exp);
+		rte_ptr_decompress_32(base, offsets32, ptrs_out, num_ptrs,
+				align_exp);
+	} else {
+		rte_ptr_compress_16(base, ptrs, offsets16, num_ptrs, align_exp);
+		rte_ptr_decompress_16(base, offsets16, ptrs_out, num_ptrs,
+				align_exp);
+	}
+
+	TEST_ASSERT_BUFFERS_ARE_EQUAL(ptrs, ptrs_out, sizeof(void *) * num_ptrs,
+		"Decompressed pointers corrupted\nbase pointer: %p, "
+		"memory region size: %" PRIu64 ", alignment exponent: %u, "
+		"num of pointers: %u, using %s offsets",
+		base, mem_sz, align_exp, num_ptrs,
+		use_32_bit ? "32-bit" : "16-bit");
+
+	return 0;
+}
+
+static int
+test_eal_ptr_compress(void)
+{
+	unsigned int j, k, n;
+	int ret = 0;
+	void * const bases[NUM_BASES] = { (void *)0, (void *)UINT16_MAX };
+	/* maximum size for pointers aligned by consecutive powers of 2 */
+	const uint64_t region_sizes_16[NUM_REGIONS] = {
+		MAX_16BIT_REGION,
+		MAX_16BIT_REGION * 2,
+		MAX_16BIT_REGION * 4,
+		MAX_16BIT_REGION * 8,
+	};
+	const uint64_t region_sizes_32[NUM_REGIONS] = {
+		MAX_32BIT_REGION,
+		MAX_32BIT_REGION * 2,
+		MAX_32BIT_REGION * 4,
+		MAX_32BIT_REGION * 8,
+	};
+
+	for (j = 0; j < NUM_REGIONS; j++) {
+		for (k = 0; k < NUM_BASES; k++) {
+			for (n = 1; n < PTRS_SIZE; n++) {
+				ret |= test_eal_ptr_compress_params(
+					bases[k],
+					region_sizes_16[j],
+					j /* exponent of alignment */,
+					n,
+					false
+				);
+				ret |= test_eal_ptr_compress_params(
+					bases[k],
+					region_sizes_32[j],
+					j /* exponent of alignment */,
+					n,
+					true
+				);
+				if (ret != 0)
+					return ret;
+			}
+		}
+	}
+
+	return ret;
+}
+
+REGISTER_FAST_TEST(eal_ptr_compress_autotest, true, true, test_eal_ptr_compress);
--
2.25.1

Re: [PATCH v7 4/4] test: add unit test for ptr compression

[David Marchand]

On Fri, Mar 1, 2024 at 11:21 AM Paul Szczepanek <paul.szczepanek@arm.com> wrote:
>
> Test compresses and decompresses pointers with various combinations
> of memory regions and alignments and verifies the pointers are
> recovered correctly.
>
> Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
> ---
>  app/test/meson.build             |   1 +
>  app/test/test_eal_ptr_compress.c | 108 +++++++++++++++++++++++++++++++
>  2 files changed, 109 insertions(+)
>  create mode 100644 app/test/test_eal_ptr_compress.c
>
> diff --git a/app/test/meson.build b/app/test/meson.build
> index 7d909039ae..0d1b777199 100644
> --- a/app/test/meson.build
> +++ b/app/test/meson.build
> @@ -68,6 +68,7 @@ source_file_deps = {
>      'test_dmadev_api.c': ['dmadev'],
>      'test_eal_flags.c': [],
>      'test_eal_fs.c': [],
> +    'test_eal_ptr_compress.c': [],
>      'test_efd.c': ['efd', 'net'],
>      'test_efd_perf.c': ['efd', 'hash'],
>      'test_errno.c': [],
> diff --git a/app/test/test_eal_ptr_compress.c b/app/test/test_eal_ptr_compress.c
> new file mode 100644
> index 0000000000..c1c9a98be7
> --- /dev/null
> +++ b/app/test/test_eal_ptr_compress.c
> @@ -0,0 +1,108 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2010-2014 Intel Corporation

Copyright ARM 2024.


-- 
David Marchand

[PATCH v7 0/4] add pointer compression API

[Paul Szczepanek]

This patchset is proposing adding a new EAL header with utility functions
that allow compression of arrays of pointers.

When passing caches full of pointers between threads, memory containing
the pointers is copied multiple times which is especially costly between
cores. A compression method will allow us to shrink the memory size
copied.

The compression takes advantage of the fact that pointers are usually
located in a limited memory region (like a mempool). We can compress them
by converting them to offsets from a base memory address.

Offsets can be stored in fewer bytes (dictated by the memory region size
and alignment of the pointer). For example: an 8 byte aligned pointer
which is part of a 32GB memory pool can be stored in 4 bytes. The API is
very generic and does not assume mempool pointers, any pointer can be
passed in.

Compression is based on few and fast operations and especially with vector
instructions leveraged creates minimal overhead.

The API accepts and returns arrays because the overhead means it only is
worth it when done in bulk.

Test is added that shows potential performance gain from compression. In
this test an array of pointers is passed through a ring between two cores.
It shows the gain which is dependent on the bulk operation size. In this
synthetic test run on ampere altra a substantial (up to 25%) performance
gain is seen if done in bulk size larger than 32. At 32 it breaks even and
lower sizes create a small (less than 5%) slowdown due to overhead.

In a more realistic mock application running the l3 forwarding dpdk
example that works in pipeline mode on two cores this translated into a
~5% throughput increase on an ampere altra.

v2:
* addressed review comments (style, explanations and typos)
* lowered bulk iterations closer to original numbers to keep runtime short
* fixed pointer size warning on 32-bit arch
v3:
* added 16-bit versions of compression functions and tests
* added documentation of these new utility functions in the EAL guide
v4:
* added unit test
* fix bug in NEON implementation of 32-bit decompress
v5:
* disable NEON and SVE implementation on AARCH32 due to wrong pointer size
v6:
* added example usage to commit message of the initial commit
v7:
* rebase to remove clashing mailmap changes
v8:
* put ptr compress into its own library
* add depends-on tag
* remove copyright bumps
* typos

Paul Szczepanek (4):
  ptr_compress: add pointer compression library
  test: add pointer compress tests to ring perf test
  docs: add pointer compression guide
  test: add unit test for ptr compression

 app/test/meson.build                       |  21 +-
 app/test/test_ptr_compress.c               | 108 +++++++
 app/test/test_ring.h                       |  92 ++++++
 app/test/test_ring_perf.c                  | 352 ++++++++++++++-------
 doc/guides/prog_guide/ptr_compress_lib.rst | 144 +++++++++
 lib/meson.build                            |   1 +
 lib/ptr_compress/meson.build               |   4 +
 lib/ptr_compress/rte_ptr_compress.h        | 266 ++++++++++++++++
 lib/ptr_compress/version.map               |   3 +
 9 files changed, 859 insertions(+), 132 deletions(-)
 create mode 100644 app/test/test_ptr_compress.c
 create mode 100644 doc/guides/prog_guide/ptr_compress_lib.rst
 create mode 100644 lib/ptr_compress/meson.build
 create mode 100644 lib/ptr_compress/rte_ptr_compress.h
 create mode 100644 lib/ptr_compress/version.map

--
2.25.1

[PATCH v8 1/4] ptr_compress: add pointer compression library

[Paul Szczepanek]

Add a new utility header for compressing pointers. The provided
functions can store pointers in 32-bit or 16-bit offsets.

The compression takes advantage of the fact that pointers are
usually located in a limited memory region (like a mempool).
We can compress them by converting them to offsets from a base
memory address. Offsets can be stored in fewer bytes (dictated
by the memory region size and alignment of the pointer).
For example: an 8 byte aligned pointer which is part of a 32GB
memory pool can be stored in 4 bytes.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <Nathan.Brown@arm.com>
---

Depends-on: patch-138071 ("lib: allow libraries with no sources")

 lib/meson.build                     |   1 +
 lib/ptr_compress/meson.build        |   4 +
 lib/ptr_compress/rte_ptr_compress.h | 266 ++++++++++++++++++++++++++++
 lib/ptr_compress/version.map        |   3 +
 4 files changed, 274 insertions(+)
 create mode 100644 lib/ptr_compress/meson.build
 create mode 100644 lib/ptr_compress/rte_ptr_compress.h
 create mode 100644 lib/ptr_compress/version.map

diff --git a/lib/meson.build b/lib/meson.build
index 0fcf3336d1..b46d3e15c6 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -14,6 +14,7 @@ libraries = [
         'argparse',
         'telemetry', # basic info querying
         'eal', # everything depends on eal
+        'ptr_compress',
         'ring',
         'rcu', # rcu depends on ring
         'mempool',
diff --git a/lib/ptr_compress/meson.build b/lib/ptr_compress/meson.build
new file mode 100644
index 0000000000..e92706a45f
--- /dev/null
+++ b/lib/ptr_compress/meson.build
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+headers = files('rte_ptr_compress.h')
diff --git a/lib/ptr_compress/rte_ptr_compress.h b/lib/ptr_compress/rte_ptr_compress.h
new file mode 100644
index 0000000000..97c084003d
--- /dev/null
+++ b/lib/ptr_compress/rte_ptr_compress.h
@@ -0,0 +1,266 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2024 Arm Limited
+ */
+
+#ifndef RTE_PTR_COMPRESS_H
+#define RTE_PTR_COMPRESS_H
+
+/**
+ * @file
+ * Pointer compression and decompression functions.
+ *
+ * When passing arrays full of pointers between threads, memory containing
+ * the pointers is copied multiple times which is especially costly between
+ * cores. These functions allow us to compress the pointers.
+ *
+ * Compression takes advantage of the fact that pointers are usually located in
+ * a limited memory region (like a mempool). We compress them by converting them
+ * to offsets from a base memory address. Offsets can be stored in fewer bytes.
+ *
+ * The compression functions come in two varieties: 32-bit and 16-bit.
+ *
+ * To determine how many bits are needed to compress the pointer calculate
+ * the biggest offset possible (highest value pointer - base pointer)
+ * and shift the value right according to alignment (shift by exponent of the
+ * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by
+ * 3, etc.). The resulting value must fit in either 32 or 16 bits.
+ *
+ * For usage example and further explanation please see "Pointer Compression" in
+ * doc/guides/prog_guide/ptr_compress_lib.rst
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_vect.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Compress pointers into 32-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 32 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 32GB of the base pointer. If there is no such alignment guarantee they
+ * must be within 4GB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_32(void *ptr_base, void **src_table,
+		uint32_t *dest_table, size_t n, uint8_t bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1w(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (i < n);
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	/* right shift is done by left shifting by negative int */
+	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_ptr_table = vld1q_u64((const uint64_t *)src_table + i);
+		v_ptr_table = vsubq_u64(v_ptr_table, v_ptr_base);
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		vst1_u32(dest_table + i, vqmovn_u64(v_ptr_table));
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT32_MAX);
+		dest_table[i] = (uint32_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 32-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_32(void *ptr_base, uint32_t *src_table,
+		void **dest_table, size_t n, uint8_t bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1uw_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (i < n);
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	int64x2_t v_shift = vdupq_n_s64(bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_ptr_table = vmovl_u32(vld1_u32(src_table + i));
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		v_ptr_table = vaddq_u64(v_ptr_table, v_ptr_base);
+		vst1q_u64((uint64_t *)dest_table + i, v_ptr_table);
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+/**
+ * Compress pointers into 16-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 16 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 256KB of the base pointer. If there is no such alignment guarantee
+ * they must be within 64KB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_16(void *ptr_base, void **src_table,
+		uint16_t *dest_table, size_t n, uint8_t bit_shift)
+{
+
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1h(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (i < n);
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT16_MAX);
+		dest_table[i] = (uint16_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 16-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_16(void *ptr_base, uint16_t *src_table,
+		void **dest_table, size_t n, uint8_t bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1uh_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (i < n);
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_PTR_COMPRESS_H */
diff --git a/lib/ptr_compress/version.map b/lib/ptr_compress/version.map
new file mode 100644
index 0000000000..5535c79061
--- /dev/null
+++ b/lib/ptr_compress/version.map
@@ -0,0 +1,3 @@
+DPDK_24 {
+	local: *;
+};
--
2.25.1

[PATCH v8 2/4] test: add pointer compress tests to ring perf test

[Paul Szczepanek]

Add a test that runs a zero copy burst enqueue and dequeue on a ring
of raw pointers and compressed pointers at different burst sizes to
showcase performance benefits of newly added pointer compression APIs.

Refactored threading code to pass more parameters to threads to
reuse existing code. Added more bulk sizes to showcase their effects
on compression. Adjusted loop iteration numbers to take into account
bulk sizes to keep runtime constant (instead of number of operations).

Adjusted old printfs to match new ones which have aligned numbers.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <Nathan.Brown@arm.com>
---
 app/test/meson.build      |  20 +--
 app/test/test_ring.h      |  92 ++++++++++
 app/test/test_ring_perf.c | 352 +++++++++++++++++++++++++-------------
 3 files changed, 332 insertions(+), 132 deletions(-)

diff --git a/app/test/meson.build b/app/test/meson.build
index 7d909039ae..df8cc00730 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -156,16 +156,16 @@ source_file_deps = {
 #    'test_resource.c': [],
     'test_rib.c': ['net', 'rib'],
     'test_rib6.c': ['net', 'rib'],
-    'test_ring.c': [],
-    'test_ring_hts_stress.c': [],
-    'test_ring_mpmc_stress.c': [],
-    'test_ring_mt_peek_stress.c': [],
-    'test_ring_mt_peek_stress_zc.c': [],
-    'test_ring_perf.c': [],
-    'test_ring_rts_stress.c': [],
-    'test_ring_st_peek_stress.c': [],
-    'test_ring_st_peek_stress_zc.c': [],
-    'test_ring_stress.c': [],
+    'test_ring.c': ['ptr_compress'],
+    'test_ring_hts_stress.c': ['ptr_compress'],
+    'test_ring_mpmc_stress.c': ['ptr_compress'],
+    'test_ring_mt_peek_stress.c': ['ptr_compress'],
+    'test_ring_mt_peek_stress_zc.c': ['ptr_compress'],
+    'test_ring_perf.c': ['ptr_compress'],
+    'test_ring_rts_stress.c': ['ptr_compress'],
+    'test_ring_st_peek_stress.c': ['ptr_compress'],
+    'test_ring_st_peek_stress_zc.c': ['ptr_compress'],
+    'test_ring_stress.c': ['ptr_compress'],
     'test_rwlock.c': [],
     'test_sched.c': ['net', 'sched'],
     'test_security.c': ['net', 'security'],
diff --git a/app/test/test_ring.h b/app/test/test_ring.h
index 45c263f3ff..f90662818c 100644
--- a/app/test/test_ring.h
+++ b/app/test/test_ring.h
@@ -5,6 +5,8 @@
 #include <rte_malloc.h>
 #include <rte_ring.h>
 #include <rte_ring_elem.h>
+#include <rte_memcpy.h>
+#include <rte_ptr_compress.h>

 /* API type to call
  * rte_ring_<sp/mp or sc/mc>_enqueue_<bulk/burst>
@@ -25,6 +27,10 @@
 #define TEST_RING_ELEM_BULK 16
 #define TEST_RING_ELEM_BURST 32

+#define TEST_RING_ELEM_BURST_ZC 64
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 128
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 256
+
 #define TEST_RING_IGNORE_API_TYPE ~0U

 /* This function is placed here as it is required for both
@@ -101,6 +107,9 @@ static inline unsigned int
 test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -152,6 +161,46 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n,
 								NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy(zcd.ptr2,
+						(char *)obj + zcd.n1 * esize,
+						(ret - zcd.n1) * esize);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_16(0, obj, zcd.ptr1, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_16(0,
+						obj + (zcd.n1 * 2),
+						zcd.ptr2,
+						(ret - zcd.n1) * 2, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_32(0, obj, zcd.ptr1, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_32(0, obj + zcd.n1,
+						zcd.ptr2, ret - zcd.n1, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
@@ -162,6 +211,9 @@ static inline unsigned int
 test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -213,6 +265,46 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mc_dequeue_burst_elem(r, obj, esize,
 								n, NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy((char *)obj + zcd.n1 * esize,
+						zcd.ptr2,
+						(ret - zcd.n1) * esize);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_16(0, zcd.ptr1, obj, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_16(0, zcd.ptr2,
+						obj + zcd.n1,
+						(ret - zcd.n1) * 2,
+						3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_32(0, zcd.ptr1, obj, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_32(0, zcd.ptr2,
+						obj + zcd.n1, ret - zcd.n1, 3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c
index d7c5a4c30b..05d86cd404 100644
--- a/app/test/test_ring_perf.c
+++ b/app/test/test_ring_perf.c
@@ -22,13 +22,13 @@

 #define RING_NAME "RING_PERF"
 #define RING_SIZE 4096
-#define MAX_BURST 32
+#define MAX_BURST 256

 /*
  * the sizes to enqueue and dequeue in testing
  * (marked volatile so they won't be seen as compile-time constants)
  */
-static const volatile unsigned bulk_sizes[] = { 8, 32 };
+static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128, 256 };

 struct lcore_pair {
 	unsigned c1, c2;
@@ -43,26 +43,30 @@ test_ring_print_test_string(unsigned int api_type, int esize,
 	if (esize == -1)
 		printf("legacy APIs");
 	else
-		printf("elem APIs: element size %dB", esize);
+		printf("elem APIs (size:%2dB)", esize);

 	if (api_type == TEST_RING_IGNORE_API_TYPE)
 		return;

 	if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF)
-		printf(": default enqueue/dequeue: ");
+		printf(" - default enqueue/dequeue");
 	else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC)
-		printf(": SP/SC: ");
+		printf(" - SP/SC");
 	else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC)
-		printf(": MP/MC: ");
+		printf(" - MP/MC");

 	if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE)
-		printf("single: ");
+		printf(" - single - ");
 	else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK)
-		printf("bulk (size: %u): ", bsz);
+		printf(" - bulk (n:%-3u) - ", bsz);
 	else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST)
-		printf("burst (size: %u): ", bsz);
+		printf(" - burst (n:%-3u) - ", bsz);
+	else if ((api_type & (TEST_RING_ELEM_BURST_ZC |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0)
+		printf(" - burst zero copy (n:%-3u) - ", bsz);

-	printf("%.2F\n", value);
+	printf("cycles per elem: %.3F\n", value);
 }

 /**** Functions to analyse our core mask to get cores for different tests ***/
@@ -144,28 +148,33 @@ static void
 test_empty_dequeue(struct rte_ring *r, const int esize,
 			const unsigned int api_type)
 {
-	const unsigned int iter_shift = 26;
+	const unsigned int iter_shift = 29;
 	const unsigned int iterations = 1 << iter_shift;
 	unsigned int i = 0;
 	void *burst[MAX_BURST];

+	const unsigned int bulk_iterations = iterations / bulk_sizes[0];
 	const uint64_t start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	for (i = 0; i < bulk_iterations; i++)
 		test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type);
 	const uint64_t end = rte_rdtsc();

 	test_ring_print_test_string(api_type, esize, bulk_sizes[0],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 }

-/*
- * for the separate enqueue and dequeue threads they take in one param
- * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc
- */
-struct thread_params {
+/* describes the ring used by the enqueue and dequeue thread */
+struct ring_params {
 	struct rte_ring *r;
-	unsigned size;        /* input value, the burst size */
-	double spsc, mpmc;    /* output value, the single or multi timings */
+	unsigned int elem_size;
+	unsigned int bulk_sizes_i; /* index into bulk_size array */
+	unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */
+};
+
+/* Used to specify enqueue and dequeue ring operations and their results */
+struct thread_params {
+	struct ring_params *ring_params;
+	double *results; /* result array size must be equal to bulk_sizes */
 };

 /*
@@ -174,16 +183,15 @@ struct thread_params {
  * flag == 1 -> dequeue
  */
 static __rte_always_inline int
-enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
-	struct thread_params *p)
+enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 15;
+	const unsigned int iter_shift = 22;
 	const unsigned int iterations = 1 << iter_shift;
-	struct rte_ring *r = p->r;
-	unsigned int bsize = p->size;
 	unsigned int i;
 	void *burst = NULL;
+	unsigned int n_remaining;
+	const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i];

 #ifdef RTE_USE_C11_MEM_MODEL
 	if (__atomic_fetch_add(&lcore_count, 1, __ATOMIC_RELAXED) + 1 != 2)
@@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 		while(lcore_count != 2)
 			rte_pause();

-	burst = test_ring_calloc(MAX_BURST, esize);
+	burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size);
 	if (burst == NULL)
 		return -1;

 	const uint64_t sp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	const unsigned int bulk_iterations = iterations / bulk_n;
+	for (i = 0; i < bulk_iterations; i++) {
+		n_remaining = bulk_n;
 		do {
 			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_enqueue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_dequeue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			if (ret == 0)
 				rte_pause();
-		} while (!ret);
+			else
+				n_remaining -= ret;
+		} while (n_remaining > 0);
+	}
 	const uint64_t sp_end = rte_rdtsc();

-	const uint64_t mp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
-		do {
-			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			if (ret == 0)
-				rte_pause();
-		} while (!ret);
-	const uint64_t mp_end = rte_rdtsc();
+	p->results[p->ring_params->bulk_sizes_i] =
+			((double)sp_end - sp_start) / iterations;

-	p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize);
-	p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize);
 	return 0;
 }

@@ -243,15 +245,7 @@ enqueue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(0, -1, params);
-}
-
-static int
-enqueue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(0, 16, params);
+	return enqueue_dequeue_bulk_helper(0, params);
 }

 /*
@@ -263,15 +257,7 @@ dequeue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(1, -1, params);
-}
-
-static int
-dequeue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(1, 16, params);
+	return enqueue_dequeue_bulk_helper(1, params);
 }

 /*
@@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p)
  * used to measure ring perf between hyperthreads, cores and sockets.
  */
 static int
-run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize)
+run_on_core_pair(struct lcore_pair *cores,
+		struct thread_params *param1, struct thread_params *param2)
 {
-	lcore_function_t *f1, *f2;
-	struct thread_params param1 = {0}, param2 = {0};
 	unsigned i;
-
-	if (esize == -1) {
-		f1 = enqueue_bulk;
-		f2 = dequeue_bulk;
-	} else {
-		f1 = enqueue_bulk_16B;
-		f2 = dequeue_bulk_16B;
-	}
+	struct ring_params *ring_params = param1->ring_params;

 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		lcore_count = 0;
-		param1.size = param2.size = bulk_sizes[i];
-		param1.r = param2.r = r;
+		ring_params->bulk_sizes_i = i;
 		if (cores->c1 == rte_get_main_lcore()) {
-			rte_eal_remote_launch(f2, &param2, cores->c2);
-			f1(&param1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
+			enqueue_bulk(param1);
 			rte_eal_wait_lcore(cores->c2);
 		} else {
-			rte_eal_remote_launch(f1, &param1, cores->c1);
-			rte_eal_remote_launch(f2, &param2, cores->c2);
+			rte_eal_remote_launch(enqueue_bulk, param1, cores->c1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
 			if (rte_eal_wait_lcore(cores->c1) < 0)
 				return -1;
 			if (rte_eal_wait_lcore(cores->c2) < 0)
 				return -1;
 		}
 		test_ring_print_test_string(
-			TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.spsc + param2.spsc);
-		test_ring_print_test_string(
-			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.mpmc + param2.mpmc);
+				ring_params->ring_flags,
+				ring_params->elem_size,
+				bulk_sizes[i],
+				param1->results[i] + param2->results[i]);
 	}

 	return 0;
@@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 	uint64_t hz = rte_get_timer_hz();
 	uint64_t lcount = 0;
 	const unsigned int lcore = rte_lcore_id();
-	struct thread_params *params = p;
+	struct ring_params *ring_params = p->ring_params;
 	void *burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -346,9 +322,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize)

 	begin = rte_get_timer_cycles();
 	while (time_diff < hz * TIME_MS / 1000) {
-		test_ring_enqueue(params->r, burst, esize, params->size,
+		test_ring_enqueue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
-		test_ring_dequeue(params->r, burst, esize, params->size,
+		test_ring_dequeue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 		lcount++;
 		time_diff = rte_get_timer_cycles() - begin;
@@ -380,7 +358,8 @@ static int
 run_on_all_cores(struct rte_ring *r, const int esize)
 {
 	uint64_t total;
-	struct thread_params param;
+	struct ring_params ring_params = {0};
+	struct thread_params params = { .ring_params = &ring_params };
 	lcore_function_t *lcore_f;
 	unsigned int i, c;

@@ -389,21 +368,20 @@ run_on_all_cores(struct rte_ring *r, const int esize)
 	else
 		lcore_f = load_loop_fn_16B;

-	memset(&param, 0, sizeof(struct thread_params));
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		total = 0;
 		printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]);
-		param.size = bulk_sizes[i];
-		param.r = r;
+		params.ring_params->bulk_sizes_i = i;
+		params.ring_params->r = r;

 		/* clear synchro and start workers */
 		__atomic_store_n(&synchro, 0, __ATOMIC_RELAXED);
-		if (rte_eal_mp_remote_launch(lcore_f, &param, SKIP_MAIN) < 0)
+		if (rte_eal_mp_remote_launch(lcore_f, &params, SKIP_MAIN) < 0)
 			return -1;

 		/* start synchro and launch test on main */
 		__atomic_store_n(&synchro, 1, __ATOMIC_RELAXED);
-		lcore_f(&param);
+		lcore_f(&params);

 		rte_eal_mp_wait_lcore();

@@ -462,9 +440,9 @@ static int
 test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	const unsigned int api_type)
 {
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 26;
 	const unsigned int iterations = 1 << iter_shift;
-	unsigned int sz, i = 0;
+	unsigned int sz, i;
 	void **burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -472,17 +450,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 		return -1;

 	for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) {
+		const unsigned int n = iterations / bulk_sizes[sz];
 		const uint64_t start = rte_rdtsc();
-		for (i = 0; i < iterations; i++) {
+		for (i = 0; i < n; i++) {
 			test_ring_enqueue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 			test_ring_dequeue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 		}
 		const uint64_t end = rte_rdtsc();

 		test_ring_print_test_string(api_type, esize, bulk_sizes[sz],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 	}

 	rte_free(burst);
@@ -490,12 +469,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	return 0;
 }

+static __rte_always_inline int
+test_ring_perf_esize_run_on_two_cores(
+		struct thread_params *param1, struct thread_params *param2)
+{
+	struct lcore_pair cores;
+
+	if (get_two_hyperthreads(&cores) == 0) {
+		printf("\n### Testing using two hyperthreads ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing using two physical cores ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_sockets(&cores) == 0) {
+		printf("\n### Testing using two NUMA nodes ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	return 0;
+}
+
 /* Run all tests for a given element size */
 static __rte_always_inline int
 test_ring_perf_esize(const int esize)
 {
-	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
+	double results_enq[RTE_DIM(bulk_sizes)];
+	double results_deq[RTE_DIM(bulk_sizes)];
+	struct ring_params ring_params = {
+			.elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results_enq };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results_deq };

 	/*
 	 * Performance test for legacy/_elem APIs
@@ -535,22 +545,13 @@ test_ring_perf_esize(const int esize)
 	test_empty_dequeue(r, esize,
 			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);

-	if (get_two_hyperthreads(&cores) == 0) {
-		printf("\n### Testing using two hyperthreads ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.r = r;

-	if (get_two_cores(&cores) == 0) {
-		printf("\n### Testing using two physical cores ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
-	if (get_two_sockets(&cores) == 0) {
-		printf("\n### Testing using two NUMA nodes ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
+
+	ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);

 	printf("\n### Testing using all worker nodes ###\n");
 	if (run_on_all_cores(r, esize) < 0)
@@ -566,6 +567,109 @@ test_ring_perf_esize(const int esize)
 	return -1;
 }

+
+static __rte_always_inline int
+test_ring_perf_compression(void)
+{
+	double results1[RTE_DIM(bulk_sizes)];
+	double results2[RTE_DIM(bulk_sizes)];
+	double results1_comp[2][RTE_DIM(bulk_sizes)];
+	double results2_comp[2][RTE_DIM(bulk_sizes)];
+
+	struct lcore_pair cores;
+	int ret = -1;
+	unsigned int i, j;
+	struct ring_params ring_params = { .elem_size = sizeof(void *) };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results1 };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results2 };
+
+	printf("\n### Testing compression gain ###");
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(void *),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy ###\n");
+		ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	/* rings allow only multiples of 4 as sizes,
+	 * we allocate size 4 despite only using 2 bytes
+	 * and use half of RING_SIZE as the number of elements
+	 */
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE / 2, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[0];
+	param2.results = results2_comp[0];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (16b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[1];
+	param2.results = results2_comp[1];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (32b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	for (j = 0; j < 2; j++) {
+		printf("\n### Potential gain from compression (%d-bit offsets) "
+		"###\n", (j + 1) * 16);
+		for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
+			const double result = results1[i] + results2[i];
+			const double result_comp = results1_comp[j][i] +
+				results2_comp[j][i];
+			const double gain = 100 - (result_comp / result) * 100;
+
+			printf("Gain of %5.1F%% for burst of %-3u elems\n",
+					gain, bulk_sizes[i]);
+		}
+	}
+
+	return ret;
+}
+
 static int
 test_ring_perf(void)
 {
@@ -576,6 +680,10 @@ test_ring_perf(void)
 	if (test_ring_perf_esize(16) == -1)
 		return -1;

+	/* Test for performance gain of compression */
+	if (test_ring_perf_compression() == -1)
+		return -1;
+
 	return 0;
 }

--
2.25.1

[PATCH v8 3/4] docs: add pointer compression guide

[Paul Szczepanek]

Documentation added in the prog guide for the new
utility functions for pointer compression
showing example code and potential usecases.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <Nathan.Brown@arm.com>
---
 doc/guides/prog_guide/ptr_compress_lib.rst | 144 +++++++++++++++++++++
 1 file changed, 144 insertions(+)
 create mode 100644 doc/guides/prog_guide/ptr_compress_lib.rst

diff --git a/doc/guides/prog_guide/ptr_compress_lib.rst b/doc/guides/prog_guide/ptr_compress_lib.rst
new file mode 100644
index 0000000000..00bc8a3697
--- /dev/null
+++ b/doc/guides/prog_guide/ptr_compress_lib.rst
@@ -0,0 +1,144 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2024 Arm Limited.
+
+.. _Pointer_Compression_Library:
+
+Pointer Compression Library
+===========================
+
+Use ``rte_ptr_compress_16()`` and ``rte_ptr_decompress_16()`` to compress and
+decompress pointers into 16-bit offsets. Use ``rte_ptr_compress_32()`` and
+``rte_ptr_decompress_32()`` to compress and decompress pointers into 32-bit
+offsets.
+
+Compression takes advantage of the fact that pointers are usually located in a
+limited memory region (like a mempool). By converting them to offsets from a
+base memory address they can be stored in fewer bytes. How many bytes are needed
+to store the offset is dictated by the memory region size and alignment of
+objects the pointers point to.
+
+For example, a pointer which is part of a 4GB memory pool can be stored as 32
+bit offset. If the pointer points to memory that is 8 bytes aligned then 3 bits
+can be dropped from the offset and a 32GB memory pool can now fit in 32 bits.
+
+For performance reasons these requirements are not enforced programmatically.
+The programmer is responsible for ensuring that the combination of distance
+from the base pointer and memory alignment allow for storing of the offset in
+the number of bits indicated by the function name (16 or 32). Start of mempool
+memory would be a good candidate for the base pointer. Otherwise any pointer
+that precedes all pointers, is close enough and has the same alignment as the
+pointers being compressed will work.
+
+.. note::
+
+    Performance gains depend on the batch size of pointers and CPU capabilities
+    such as vector extensions. It's important to measure the performance
+    increase on target hardware. A test called ``ring_perf_autotest`` in
+    ``dpdk-test`` can provide the measurements.
+
+Example usage
+~~~~~~~~~~~~~
+
+In this example we send pointers between two cores through a ring. While this
+is a realistic use case the code is simplified for demonstration purposes and
+does not have error handling.
+
+.. code-block:: c
+
+    #include <rte_launch.h>
+    #include <rte_ring.h>
+    #include <rte_ring_elem.h>
+    #include <rte_ptr_compress.h>
+
+    #define ITEMS_ARRAY_SIZE (1024)
+    #define BATCH_SIZE (128)
+    #define ALIGN_EXPONENT (3)
+    #define ITEM_ALIGN (1<<ALIGN_EXPONENT)
+    #define CORE_SEND (1)
+    #define CORE_RECV (2)
+
+    struct item {
+      int a;
+    } __rte_aligned(ITEM_ALIGN);
+
+    static struct item items[ITEMS_ARRAY_SIZE] = {0};
+    static struct rte_ring *ring = NULL;
+
+    static int
+    send_compressed(void *args)
+    {
+      struct item *ptrs_send[BATCH_SIZE] = {0};
+      unsigned int n_send = 0;
+      struct rte_ring_zc_data zcd = {0};
+
+      /* in this example we only fill the ptrs_send once and reuse */
+      for (;n_send < BATCH_SIZE; n_send++)
+        ptrs_send[n_send] = &items[n_send];
+
+      for(;;) {
+        n_send = rte_ring_enqueue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        /* compress ptrs_send into offsets */
+        rte_ptr_compress_32(items, /* base pointer */
+          ptrs_send, /* source array to be compressed */
+          zcd.ptr1, /* destination array to store offsets */
+          zcd.n1, /* how many pointers to compress */
+          ALIGN_EXPONENT /* how many bits can we drop from the offset */);
+
+        if (zcd.ptr2 != NULL)
+          rte_ptr_compress_32(items, ptrs_send + zcd.n1,
+            zcd.ptr2, n_send - zcd.n1, ALIGN_EXPONENT);
+
+        rte_ring_enqueue_zc_finish(ring, n_send);
+      }
+      return 1;
+    }
+
+    static int
+    recv_compressed(void *args)
+    {
+      struct item *ptrs_recv[BATCH_SIZE] = {0};
+      unsigned int n_recv;
+      struct rte_ring_zc_data zcd = {0};
+
+      for(;;) {
+        /* receive compressed pointers from the ring */
+        n_recv = rte_ring_dequeue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        rte_ptr_decompress_32(items, /* base pointer */
+          zcd.ptr1, /* source array to decompress */
+          ptrs_recv, /* destination array to store pointers */
+          zcd.n1, /* how many pointers to decompress */
+          ALIGN_EXPONENT /* how many bits were dropped from the offset */);
+
+        /* handle the potential secondary buffer (caused by ring boundary) */
+        if (zcd.ptr2 != NULL)
+          rte_ptr_decompress_32(items,
+            zcd.ptr2,
+            ptrs_recv + zcd.n1,
+            n_recv - zcd.n1,
+            ALIGN_EXPONENT);
+
+        rte_ring_dequeue_zc_finish(ring, n_recv);
+
+        /* ptrs_recv contains what ptrs_send contained in the other thread */
+        /* (...) */
+      }
+      return 1;
+    }
+
+    void
+    compression_example(void)
+    {
+      ring = rte_ring_create_elem(
+        "COMPR_PTRS", sizeof(uint32_t),
+        1024, rte_socket_id(),
+        RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+      rte_eal_remote_launch(send_compressed, NULL, CORE_SEND);
+      rte_eal_remote_launch(recv_compressed, NULL, CORE_RECV);
+
+      for(;;) {}
+    }
--
2.25.1

[PATCH v8 4/4] test: add unit test for ptr compression

[Paul Szczepanek]

Test compresses and decompresses pointers with various combinations
of memory regions and alignments and verifies the pointers are
recovered correctly.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <Nathan.Brown@arm.com>
---
 app/test/meson.build         |   1 +
 app/test/test_ptr_compress.c | 108 +++++++++++++++++++++++++++++++++++
 2 files changed, 109 insertions(+)
 create mode 100644 app/test/test_ptr_compress.c

diff --git a/app/test/meson.build b/app/test/meson.build
index df8cc00730..e29258e6ec 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -144,6 +144,7 @@ source_file_deps = {
     'test_power_intel_uncore.c': ['power'],
     'test_power_kvm_vm.c': ['power'],
     'test_prefetch.c': [],
+    'test_ptr_compress.c': ['ptr_compress'],
     'test_rand_perf.c': [],
     'test_rawdev.c': ['rawdev', 'bus_vdev'],
     'test_rcu_qsbr.c': ['rcu', 'hash'],
diff --git a/app/test/test_ptr_compress.c b/app/test/test_ptr_compress.c
new file mode 100644
index 0000000000..972f832776
--- /dev/null
+++ b/app/test/test_ptr_compress.c
@@ -0,0 +1,108 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2024 Arm Limited
+ */
+
+#include "test.h"
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_ptr_compress.h>
+
+#define MAX_ALIGN_EXPONENT 3
+#define MAX_PTRS 16
+#define NUM_BASES 2
+#define NUM_REGIONS 4
+#define MAX_32BIT_REGION ((uint64_t)UINT32_MAX + 1)
+#define MAX_16BIT_REGION (UINT16_MAX + 1)
+
+static int
+test_ptr_compress_params(
+	void *base,
+	uint64_t mem_sz,
+	unsigned int align_exp,
+	unsigned int num_ptrs,
+	bool use_32_bit)
+{
+	unsigned int i;
+	unsigned int align = 1 << align_exp;
+	void *ptrs[MAX_PTRS] = {0};
+	void *ptrs_out[MAX_PTRS] = {0};
+	uint32_t offsets32[MAX_PTRS] = {0};
+	uint16_t offsets16[MAX_PTRS] = {0};
+
+	for (i = 0; i < num_ptrs; i++) {
+		/* make pointers point at memory in steps of align */
+		/* alternate steps from the start and end of memory region */
+		if ((i & 1) == 1)
+			ptrs[i] = (char *)base + mem_sz - i * align;
+		else
+			ptrs[i] = (char *)base + i * align;
+	}
+
+	if (use_32_bit) {
+		rte_ptr_compress_32(base, ptrs, offsets32, num_ptrs, align_exp);
+		rte_ptr_decompress_32(base, offsets32, ptrs_out, num_ptrs,
+				align_exp);
+	} else {
+		rte_ptr_compress_16(base, ptrs, offsets16, num_ptrs, align_exp);
+		rte_ptr_decompress_16(base, offsets16, ptrs_out, num_ptrs,
+				align_exp);
+	}
+
+	TEST_ASSERT_BUFFERS_ARE_EQUAL(ptrs, ptrs_out, sizeof(void *) * num_ptrs,
+		"Decompressed pointers corrupted\nbase pointer: %p, "
+		"memory region size: %" PRIu64 ", alignment exponent: %u, "
+		"num of pointers: %u, using %s offsets",
+		base, mem_sz, align_exp, num_ptrs,
+		use_32_bit ? "32-bit" : "16-bit");
+
+	return 0;
+}
+
+static int
+test_ptr_compress(void)
+{
+	unsigned int j, k, n;
+	int ret = 0;
+	void * const bases[NUM_BASES] = { (void *)0, (void *)UINT16_MAX };
+	/* maximum size for pointers aligned by consecutive powers of 2 */
+	const uint64_t region_sizes_16[NUM_REGIONS] = {
+		MAX_16BIT_REGION,
+		MAX_16BIT_REGION * 2,
+		MAX_16BIT_REGION * 4,
+		MAX_16BIT_REGION * 8,
+	};
+	const uint64_t region_sizes_32[NUM_REGIONS] = {
+		MAX_32BIT_REGION,
+		MAX_32BIT_REGION * 2,
+		MAX_32BIT_REGION * 4,
+		MAX_32BIT_REGION * 8,
+	};
+
+	for (j = 0; j < NUM_REGIONS; j++) {
+		for (k = 0; k < NUM_BASES; k++) {
+			for (n = 1; n < MAX_PTRS; n++) {
+				ret |= test_ptr_compress_params(
+					bases[k],
+					region_sizes_16[j],
+					j /* exponent of alignment */,
+					n,
+					false
+				);
+				ret |= test_ptr_compress_params(
+					bases[k],
+					region_sizes_32[j],
+					j /* exponent of alignment */,
+					n,
+					true
+				);
+				if (ret != 0)
+					return ret;
+			}
+		}
+	}
+
+	return ret;
+}
+
+REGISTER_FAST_TEST(ptr_compress_autotest, true, true, test_ptr_compress);
--
2.25.1

Re: [PATCH v7 0/4] add pointer compression API

[David Marchand]

Hello Paul,

On Thu, Mar 7, 2024 at 9:40 PM Paul Szczepanek <paul.szczepanek@arm.com> wrote:
>
> This patchset is proposing adding a new EAL header with utility functions
> that allow compression of arrays of pointers.
>
> When passing caches full of pointers between threads, memory containing
> the pointers is copied multiple times which is especially costly between
> cores. A compression method will allow us to shrink the memory size
> copied.
>
> The compression takes advantage of the fact that pointers are usually
> located in a limited memory region (like a mempool). We can compress them
> by converting them to offsets from a base memory address.
>
> Offsets can be stored in fewer bytes (dictated by the memory region size
> and alignment of the pointer). For example: an 8 byte aligned pointer
> which is part of a 32GB memory pool can be stored in 4 bytes. The API is
> very generic and does not assume mempool pointers, any pointer can be
> passed in.
>
> Compression is based on few and fast operations and especially with vector
> instructions leveraged creates minimal overhead.
>
> The API accepts and returns arrays because the overhead means it only is
> worth it when done in bulk.
>
> Test is added that shows potential performance gain from compression. In
> this test an array of pointers is passed through a ring between two cores.
> It shows the gain which is dependent on the bulk operation size. In this
> synthetic test run on ampere altra a substantial (up to 25%) performance
> gain is seen if done in bulk size larger than 32. At 32 it breaks even and
> lower sizes create a small (less than 5%) slowdown due to overhead.
>
> In a more realistic mock application running the l3 forwarding dpdk
> example that works in pipeline mode on two cores this translated into a
> ~5% throughput increase on an ampere altra.
>
> v2:
> * addressed review comments (style, explanations and typos)
> * lowered bulk iterations closer to original numbers to keep runtime short
> * fixed pointer size warning on 32-bit arch
> v3:
> * added 16-bit versions of compression functions and tests
> * added documentation of these new utility functions in the EAL guide
> v4:
> * added unit test
> * fix bug in NEON implementation of 32-bit decompress
> v5:
> * disable NEON and SVE implementation on AARCH32 due to wrong pointer size
> v6:
> * added example usage to commit message of the initial commit
> v7:
> * rebase to remove clashing mailmap changes
> v8:
> * put ptr compress into its own library
> * add depends-on tag
> * remove copyright bumps
> * typos
>
> Paul Szczepanek (4):
>   ptr_compress: add pointer compression library
>   test: add pointer compress tests to ring perf test
>   docs: add pointer compression guide
>   test: add unit test for ptr compression
>
>  app/test/meson.build                       |  21 +-
>  app/test/test_ptr_compress.c               | 108 +++++++
>  app/test/test_ring.h                       |  92 ++++++
>  app/test/test_ring_perf.c                  | 352 ++++++++++++++-------
>  doc/guides/prog_guide/ptr_compress_lib.rst | 144 +++++++++
>  lib/meson.build                            |   1 +
>  lib/ptr_compress/meson.build               |   4 +
>  lib/ptr_compress/rte_ptr_compress.h        | 266 ++++++++++++++++
>  lib/ptr_compress/version.map               |   3 +
>  9 files changed, 859 insertions(+), 132 deletions(-)
>  create mode 100644 app/test/test_ptr_compress.c
>  create mode 100644 doc/guides/prog_guide/ptr_compress_lib.rst
>  create mode 100644 lib/ptr_compress/meson.build
>  create mode 100644 lib/ptr_compress/rte_ptr_compress.h
>  create mode 100644 lib/ptr_compress/version.map

We mentionned during the weekly release meeting, it seemed too late
for merging this work in the 24.03 release.

Looking at v8, I have comments on this series:
- rather than put a Depends-on: tag, take the lib: patch as part of
your series, there is no need for this patch without the ptr_compress
lib and it will avoid any CI issue (ovsrobot does not support
Depends-on: patch- for example),
- lib/ptr_compress/version.map is unneeded now,
- lib/ptr_compress/, app/test/test_ptr_compress.c and
doc/guides/prog_guide/ptr_compress_lib.rst need a MAINTAINERS entry,
- prefer lowercase characters for mail addresses in commitlogs,
- the documentation is not referenced in doc/guides/prog_guide/index.rst,
- doxygen does not know of this new library, you must update
doc/api/doxy-api-index.md and doc/api/doxy-api.conf.in,
- a RN entry is missing,

There were also comments on the lib: patch.

At this point, it is better to take our time to finish putting this
work in good form and merge it in 24.07.

Thanks.

-- 
David Marchand

Re: [PATCH v7 0/4] add pointer compression API

[Honnappa Nagarahalli]

+ Wathsala


> On Mar 8, 2024, at 2:27 AM, David Marchand <david.marchand@redhat.com> wrote:
> 
> Hello Paul,
> 
> On Thu, Mar 7, 2024 at 9:40 PM Paul Szczepanek <paul.szczepanek@arm.com> wrote:
>> 
>> This patchset is proposing adding a new EAL header with utility functions
>> that allow compression of arrays of pointers.
>> 
>> When passing caches full of pointers between threads, memory containing
>> the pointers is copied multiple times which is especially costly between
>> cores. A compression method will allow us to shrink the memory size
>> copied.
>> 
>> The compression takes advantage of the fact that pointers are usually
>> located in a limited memory region (like a mempool). We can compress them
>> by converting them to offsets from a base memory address.
>> 
>> Offsets can be stored in fewer bytes (dictated by the memory region size
>> and alignment of the pointer). For example: an 8 byte aligned pointer
>> which is part of a 32GB memory pool can be stored in 4 bytes. The API is
>> very generic and does not assume mempool pointers, any pointer can be
>> passed in.
>> 
>> Compression is based on few and fast operations and especially with vector
>> instructions leveraged creates minimal overhead.
>> 
>> The API accepts and returns arrays because the overhead means it only is
>> worth it when done in bulk.
>> 
>> Test is added that shows potential performance gain from compression. In
>> this test an array of pointers is passed through a ring between two cores.
>> It shows the gain which is dependent on the bulk operation size. In this
>> synthetic test run on ampere altra a substantial (up to 25%) performance
>> gain is seen if done in bulk size larger than 32. At 32 it breaks even and
>> lower sizes create a small (less than 5%) slowdown due to overhead.
>> 
>> In a more realistic mock application running the l3 forwarding dpdk
>> example that works in pipeline mode on two cores this translated into a
>> ~5% throughput increase on an ampere altra.
>> 
>> v2:
>> * addressed review comments (style, explanations and typos)
>> * lowered bulk iterations closer to original numbers to keep runtime short
>> * fixed pointer size warning on 32-bit arch
>> v3:
>> * added 16-bit versions of compression functions and tests
>> * added documentation of these new utility functions in the EAL guide
>> v4:
>> * added unit test
>> * fix bug in NEON implementation of 32-bit decompress
>> v5:
>> * disable NEON and SVE implementation on AARCH32 due to wrong pointer size
>> v6:
>> * added example usage to commit message of the initial commit
>> v7:
>> * rebase to remove clashing mailmap changes
>> v8:
>> * put ptr compress into its own library
>> * add depends-on tag
>> * remove copyright bumps
>> * typos
>> 
>> Paul Szczepanek (4):
>>  ptr_compress: add pointer compression library
>>  test: add pointer compress tests to ring perf test
>>  docs: add pointer compression guide
>>  test: add unit test for ptr compression
>> 
>> app/test/meson.build                       |  21 +-
>> app/test/test_ptr_compress.c               | 108 +++++++
>> app/test/test_ring.h                       |  92 ++++++
>> app/test/test_ring_perf.c                  | 352 ++++++++++++++-------
>> doc/guides/prog_guide/ptr_compress_lib.rst | 144 +++++++++
>> lib/meson.build                            |   1 +
>> lib/ptr_compress/meson.build               |   4 +
>> lib/ptr_compress/rte_ptr_compress.h        | 266 ++++++++++++++++
>> lib/ptr_compress/version.map               |   3 +
>> 9 files changed, 859 insertions(+), 132 deletions(-)
>> create mode 100644 app/test/test_ptr_compress.c
>> create mode 100644 doc/guides/prog_guide/ptr_compress_lib.rst
>> create mode 100644 lib/ptr_compress/meson.build
>> create mode 100644 lib/ptr_compress/rte_ptr_compress.h
>> create mode 100644 lib/ptr_compress/version.map
> 
> We mentionned during the weekly release meeting, it seemed too late
> for merging this work in the 24.03 release.
> 
> Looking at v8, I have comments on this series:
> - rather than put a Depends-on: tag, take the lib: patch as part of
> your series, there is no need for this patch without the ptr_compress
> lib and it will avoid any CI issue (ovsrobot does not support
> Depends-on: patch- for example),
Agree, this is a better solution

> - lib/ptr_compress/version.map is unneeded now,
> - lib/ptr_compress/, app/test/test_ptr_compress.c and
> doc/guides/prog_guide/ptr_compress_lib.rst need a MAINTAINERS entry,
> - prefer lowercase characters for mail addresses in commitlogs,
> - the documentation is not referenced in doc/guides/prog_guide/index.rst,
> - doxygen does not know of this new library, you must update
> doc/api/doxy-api-index.md and doc/api/doxy-api.conf.in,
> - a RN entry is missing,
Apologies for missing these.

> 
> There were also comments on the lib: patch.
Not sure which comments you are talking about. Your comments on V7 were addressed in V8.

> 
> At this point, it is better to take our time to finish putting this
> work in good form and merge it in 24.07.
Given your comments do not affect the code and the changes are pretty straightforward, request you reconsider the decision.
Anyway, we will get these changes pushed to community on Monday.

> 
> Thanks.
> 
> -- 
> David Marchand
> 

Re: [PATCH v7 0/4] add pointer compression API

[David Marchand]

On Sun, Mar 10, 2024 at 8:35 PM Honnappa Nagarahalli
<Honnappa.Nagarahalli@arm.com> wrote:
> > We mentionned during the weekly release meeting, it seemed too late
> > for merging this work in the 24.03 release.
> >
> > Looking at v8, I have comments on this series:
> > - rather than put a Depends-on: tag, take the lib: patch as part of
> > your series, there is no need for this patch without the ptr_compress
> > lib and it will avoid any CI issue (ovsrobot does not support
> > Depends-on: patch- for example),
> Agree, this is a better solution
>
> > - lib/ptr_compress/version.map is unneeded now,
> > - lib/ptr_compress/, app/test/test_ptr_compress.c and
> > doc/guides/prog_guide/ptr_compress_lib.rst need a MAINTAINERS entry,
> > - prefer lowercase characters for mail addresses in commitlogs,
> > - the documentation is not referenced in doc/guides/prog_guide/index.rst,
> > - doxygen does not know of this new library, you must update
> > doc/api/doxy-api-index.md and doc/api/doxy-api.conf.in,
> > - a RN entry is missing,
> Apologies for missing these.
>
> >
> > There were also comments on the lib: patch.
> Not sure which comments you are talking about. Your comments on V7 were addressed in V8.

http://inbox.dpdk.org/dev/ZemoDBHt6fMBUqne@bricha3-mobl1.ger.corp.intel.com/T/#m665026fb0c7ed832d3e80b68b16d7549124f6880


-- 
David Marchand

[PATCH v9 0/5] add pointer compression API

[Paul Szczepanek]

This patchset is proposing adding a new header only library
with utility functions that allow compression of arrays of pointers.

Since this is a header only library a patch needed to be added to amend
the build system to allow adding libraries without source files.

When passing caches full of pointers between threads, memory containing
the pointers is copied multiple times which is especially costly between
cores. A compression method will allow us to shrink the memory size
copied.

The compression takes advantage of the fact that pointers are usually
located in a limited memory region (like a mempool). We can compress them
by converting them to offsets from a base memory address.

Offsets can be stored in fewer bytes (dictated by the memory region size
and alignment of the pointer). For example: an 8 byte aligned pointer
which is part of a 32GB memory pool can be stored in 4 bytes. The API is
very generic and does not assume mempool pointers, any pointer can be
passed in.

Compression is based on few and fast operations and especially with vector
instructions leveraged creates minimal overhead.

The API accepts and returns arrays because the overhead means it only is
worth it when done in bulk.

Test is added that shows potential performance gain from compression. In
this test an array of pointers is passed through a ring between two cores.
It shows the gain which is dependent on the bulk operation size. In this
synthetic test run on ampere altra a substantial (up to 25%) performance
gain is seen if done in bulk size larger than 32. At 32 it breaks even and
lower sizes create a small (less than 5%) slowdown due to overhead.

In a more realistic mock application running the l3 forwarding dpdk
example that works in pipeline mode on two cores this translated into a
~5% throughput increase on an ampere altra.

v2:
* addressed review comments (style, explanations and typos)
* lowered bulk iterations closer to original numbers to keep runtime short
* fixed pointer size warning on 32-bit arch
v3:
* added 16-bit versions of compression functions and tests
* added documentation of these new utility functions in the EAL guide
v4:
* added unit test
* fix bug in NEON implementation of 32-bit decompress
v5:
* disable NEON and SVE implementation on AARCH32 due to wrong pointer size
v6:
* added example usage to commit message of the initial commit
v7:
* rebase to remove clashing mailmap changes
v8:
* put ptr compress into its own library
* add depends-on tag
* remove copyright bumps
* typos
v9
* added MAINTAINERS entries, release notes, doc indexes etc.
* added patch for build system to allow header only library

Paul Szczepanek (5):
  lib: allow libraries with no sources
  ptr_compress: add pointer compression library
  test: add pointer compress tests to ring perf test
  docs: add pointer compression guide
  test: add unit test for ptr compression

 MAINTAINERS                                |   6 +
 app/test/meson.build                       |  21 +-
 app/test/test_ptr_compress.c               | 108 +++++++
 app/test/test_ring.h                       |  92 ++++++
 app/test/test_ring_perf.c                  | 352 ++++++++++++++-------
 doc/api/doxy-api-index.md                  |   1 +
 doc/api/doxy-api.conf.in                   |   1 +
 doc/guides/prog_guide/index.rst            |   1 +
 doc/guides/prog_guide/ptr_compress_lib.rst | 142 +++++++++
 doc/guides/rel_notes/release_24_03.rst     |   6 +
 lib/meson.build                            |  15 +
 lib/ptr_compress/meson.build               |   4 +
 lib/ptr_compress/rte_ptr_compress.h        | 266 ++++++++++++++++
 13 files changed, 883 insertions(+), 132 deletions(-)
 create mode 100644 app/test/test_ptr_compress.c
 create mode 100644 doc/guides/prog_guide/ptr_compress_lib.rst
 create mode 100644 lib/ptr_compress/meson.build
 create mode 100644 lib/ptr_compress/rte_ptr_compress.h

--
2.25.1

[PATCH v9 1/5] lib: allow libraries with no sources

[Paul Szczepanek]

Allow header only libraries.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 lib/meson.build | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/lib/meson.build b/lib/meson.build
index 179a272932..e4e31f7ecf 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -222,6 +222,20 @@ foreach l:libraries
     includes += include_directories(l)
     dpdk_includes += include_directories(l)

+    # special case for header only libraries
+    if sources.length() == 0
+        header_lib_dep = declare_dependency(include_directories: includes,
+                dependencies: shared_deps)
+        set_variable('shared_rte_' + name, header_lib_dep)
+        set_variable('static_rte_' + name, header_lib_dep)
+        dpdk_shared_lib_deps += header_lib_dep
+        dpdk_static_lib_deps += header_lib_dep
+        if developer_mode
+            message('lib/@0@: Defining dependency "@1@"'.format(l, name))
+        endif
+        continue
+    endif
+
     if developer_mode and is_windows and use_function_versioning
         message('@0@: Function versioning is not supported by Windows.'.format(name))
     endif
--
2.25.1

Re: [PATCH v9 1/5] lib: allow libraries with no sources

[Bruce Richardson]

On Mon, Mar 11, 2024 at 02:47:02PM +0000, Paul Szczepanek wrote:
> Allow header only libraries.
> 
> Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> ---
>  lib/meson.build | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
> 
> diff --git a/lib/meson.build b/lib/meson.build
> index 179a272932..e4e31f7ecf 100644
> --- a/lib/meson.build
> +++ b/lib/meson.build
> @@ -222,6 +222,20 @@ foreach l:libraries
>      includes += include_directories(l)
>      dpdk_includes += include_directories(l)
> 
> +    # special case for header only libraries
> +    if sources.length() == 0
> +        header_lib_dep = declare_dependency(include_directories: includes,
> +                dependencies: shared_deps)
> +        set_variable('shared_rte_' + name, header_lib_dep)
> +        set_variable('static_rte_' + name, header_lib_dep)
> +        dpdk_shared_lib_deps += header_lib_dep
> +        dpdk_static_lib_deps += header_lib_dep
> +        if developer_mode
> +            message('lib/@0@: Defining dependency "@1@"'.format(l, name))
> +        endif
> +        continue
> +    endif
> +
>      if developer_mode and is_windows and use_function_versioning
>          message('@0@: Function versioning is not supported by Windows.'.format(name))
>      endif

Diff is much clearer this way, thanks.

Acked-by: Bruce Richardson <bruce.richardson@intel.com>

Re: [PATCH v9 1/5] lib: allow libraries with no sources

[Paul Szczepanek]

On 11/03/2024 15:23, Bruce Richardson wrote:
> On Mon, Mar 11, 2024 at 02:47:02PM +0000, Paul Szczepanek wrote:
>> Allow header only libraries.
>>
>> Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
>> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
>> ---
>>  lib/meson.build | 14 ++++++++++++++
>>  1 file changed, 14 insertions(+)
>>
>> diff --git a/lib/meson.build b/lib/meson.build
>> index 179a272932..e4e31f7ecf 100644
>> --- a/lib/meson.build
>> +++ b/lib/meson.build
>> @@ -222,6 +222,20 @@ foreach l:libraries
>>      includes += include_directories(l)
>>      dpdk_includes += include_directories(l)
>>
>> +    # special case for header only libraries
>> +    if sources.length() == 0
>> +        header_lib_dep = declare_dependency(include_directories: includes,
>> +                dependencies: shared_deps)
>> +        set_variable('shared_rte_' + name, header_lib_dep)
>> +        set_variable('static_rte_' + name, header_lib_dep)
>> +        dpdk_shared_lib_deps += header_lib_dep
>> +        dpdk_static_lib_deps += header_lib_dep
>> +        if developer_mode
>> +            message('lib/@0@: Defining dependency "@1@"'.format(l, name))
>> +        endif
>> +        continue
>> +    endif
>> +
>>      if developer_mode and is_windows and use_function_versioning
>>          message('@0@: Function versioning is not supported by Windows.'.format(name))
>>      endif
> 
> Diff is much clearer this way, thank
> 
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>

I had to fix a problem with the addition of `dependencies: shared_deps`
requiring a separate header_lib_dep for static deps - which required a
V10. Can I please get an Ack for the V10?

[PATCH v9 2/5] ptr_compress: add pointer compression library

[Paul Szczepanek]

Add a new utility header for compressing pointers. The provided
functions can store pointers in 32-bit or 16-bit offsets.

The compression takes advantage of the fact that pointers are
usually located in a limited memory region (like a mempool).
We can compress them by converting them to offsets from a base
memory address. Offsets can be stored in fewer bytes (dictated
by the memory region size and alignment of the pointer).
For example: an 8 byte aligned pointer which is part of a 32GB
memory pool can be stored in 4 bytes.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
---
 MAINTAINERS                            |   4 +
 doc/api/doxy-api-index.md              |   1 +
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/rel_notes/release_24_03.rst |   6 +
 lib/meson.build                        |   1 +
 lib/ptr_compress/meson.build           |   4 +
 lib/ptr_compress/rte_ptr_compress.h    | 266 +++++++++++++++++++++++++
 7 files changed, 283 insertions(+)
 create mode 100644 lib/ptr_compress/meson.build
 create mode 100644 lib/ptr_compress/rte_ptr_compress.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 4755a68274..6f703b1b13 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1685,6 +1685,10 @@ M: Chenbo Xia <chenbox@nvidia.com>
 M: Gaetan Rivet <grive@u256.net>
 F: lib/pci/

+Pointer Compression
+M: Paul Szczepanek <paul.szczepanek@arm.com>
+F: lib/ptr_compress/
+
 Power management
 M: Anatoly Burakov <anatoly.burakov@intel.com>
 M: David Hunt <david.hunt@intel.com>
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 8c1eb8fafa..f9283154f8 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -222,6 +222,7 @@ The public API headers are grouped by topics:
   [config file](@ref rte_cfgfile.h),
   [key/value args](@ref rte_kvargs.h),
   [argument parsing](@ref rte_argparse.h),
+  [ptr_compress](@ref rte_ptr_compress.h),
   [string](@ref rte_string_fns.h),
   [thread](@ref rte_thread.h)

diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index 27afec8b3b..a8823c046f 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -71,6 +71,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/pipeline \
                           @TOPDIR@/lib/port \
                           @TOPDIR@/lib/power \
+                          @TOPDIR@/lib/ptr_compress \
                           @TOPDIR@/lib/rawdev \
                           @TOPDIR@/lib/rcu \
                           @TOPDIR@/lib/regexdev \
diff --git a/doc/guides/rel_notes/release_24_03.rst b/doc/guides/rel_notes/release_24_03.rst
index 932688ca4d..b82b8c5c0b 100644
--- a/doc/guides/rel_notes/release_24_03.rst
+++ b/doc/guides/rel_notes/release_24_03.rst
@@ -176,6 +176,12 @@ New Features
   * Added power-saving during polling within the ``rte_event_dequeue_burst()`` API.
   * Added support for DMA adapter.

+* **Introduced pointer compression library.**
+
+  Library provides functions to compress and decompress arrays of pointers
+  which can improve application performance under certain conditions.
+  Performance test was added to help users evaluate performance on their setup.
+

 Removed Items
 -------------
diff --git a/lib/meson.build b/lib/meson.build
index e4e31f7ecf..fe43d137d7 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -14,6 +14,7 @@ libraries = [
         'argparse',
         'telemetry', # basic info querying
         'eal', # everything depends on eal
+        'ptr_compress',
         'ring',
         'rcu', # rcu depends on ring
         'mempool',
diff --git a/lib/ptr_compress/meson.build b/lib/ptr_compress/meson.build
new file mode 100644
index 0000000000..e92706a45f
--- /dev/null
+++ b/lib/ptr_compress/meson.build
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+headers = files('rte_ptr_compress.h')
diff --git a/lib/ptr_compress/rte_ptr_compress.h b/lib/ptr_compress/rte_ptr_compress.h
new file mode 100644
index 0000000000..97c084003d
--- /dev/null
+++ b/lib/ptr_compress/rte_ptr_compress.h
@@ -0,0 +1,266 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2024 Arm Limited
+ */
+
+#ifndef RTE_PTR_COMPRESS_H
+#define RTE_PTR_COMPRESS_H
+
+/**
+ * @file
+ * Pointer compression and decompression functions.
+ *
+ * When passing arrays full of pointers between threads, memory containing
+ * the pointers is copied multiple times which is especially costly between
+ * cores. These functions allow us to compress the pointers.
+ *
+ * Compression takes advantage of the fact that pointers are usually located in
+ * a limited memory region (like a mempool). We compress them by converting them
+ * to offsets from a base memory address. Offsets can be stored in fewer bytes.
+ *
+ * The compression functions come in two varieties: 32-bit and 16-bit.
+ *
+ * To determine how many bits are needed to compress the pointer calculate
+ * the biggest offset possible (highest value pointer - base pointer)
+ * and shift the value right according to alignment (shift by exponent of the
+ * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by
+ * 3, etc.). The resulting value must fit in either 32 or 16 bits.
+ *
+ * For usage example and further explanation please see "Pointer Compression" in
+ * doc/guides/prog_guide/ptr_compress_lib.rst
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_vect.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Compress pointers into 32-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 32 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 32GB of the base pointer. If there is no such alignment guarantee they
+ * must be within 4GB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_32(void *ptr_base, void **src_table,
+		uint32_t *dest_table, size_t n, uint8_t bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1w(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (i < n);
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	/* right shift is done by left shifting by negative int */
+	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_ptr_table = vld1q_u64((const uint64_t *)src_table + i);
+		v_ptr_table = vsubq_u64(v_ptr_table, v_ptr_base);
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		vst1_u32(dest_table + i, vqmovn_u64(v_ptr_table));
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT32_MAX);
+		dest_table[i] = (uint32_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 32-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_32(void *ptr_base, uint32_t *src_table,
+		void **dest_table, size_t n, uint8_t bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1uw_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (i < n);
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	int64x2_t v_shift = vdupq_n_s64(bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_ptr_table = vmovl_u32(vld1_u32(src_table + i));
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		v_ptr_table = vaddq_u64(v_ptr_table, v_ptr_base);
+		vst1q_u64((uint64_t *)dest_table + i, v_ptr_table);
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+/**
+ * Compress pointers into 16-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 16 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 256KB of the base pointer. If there is no such alignment guarantee
+ * they must be within 64KB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_16(void *ptr_base, void **src_table,
+		uint16_t *dest_table, size_t n, uint8_t bit_shift)
+{
+
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1h(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (i < n);
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT16_MAX);
+		dest_table[i] = (uint16_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 16-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_16(void *ptr_base, uint16_t *src_table,
+		void **dest_table, size_t n, uint8_t bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1uh_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (i < n);
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_PTR_COMPRESS_H */
--
2.25.1

[PATCH v9 3/5] test: add pointer compress tests to ring perf test

[Paul Szczepanek]

Add a test that runs a zero copy burst enqueue and dequeue on a ring
of raw pointers and compressed pointers at different burst sizes to
showcase performance benefits of newly added pointer compression APIs.

Refactored threading code to pass more parameters to threads to
reuse existing code. Added more bulk sizes to showcase their effects
on compression. Adjusted loop iteration numbers to take into account
bulk sizes to keep runtime constant (instead of number of operations).

Adjusted old printfs to match new ones which have aligned numbers.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
---
 app/test/meson.build      |  20 +--
 app/test/test_ring.h      |  92 ++++++++++
 app/test/test_ring_perf.c | 352 +++++++++++++++++++++++++-------------
 3 files changed, 332 insertions(+), 132 deletions(-)

diff --git a/app/test/meson.build b/app/test/meson.build
index 7d909039ae..df8cc00730 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -156,16 +156,16 @@ source_file_deps = {
 #    'test_resource.c': [],
     'test_rib.c': ['net', 'rib'],
     'test_rib6.c': ['net', 'rib'],
-    'test_ring.c': [],
-    'test_ring_hts_stress.c': [],
-    'test_ring_mpmc_stress.c': [],
-    'test_ring_mt_peek_stress.c': [],
-    'test_ring_mt_peek_stress_zc.c': [],
-    'test_ring_perf.c': [],
-    'test_ring_rts_stress.c': [],
-    'test_ring_st_peek_stress.c': [],
-    'test_ring_st_peek_stress_zc.c': [],
-    'test_ring_stress.c': [],
+    'test_ring.c': ['ptr_compress'],
+    'test_ring_hts_stress.c': ['ptr_compress'],
+    'test_ring_mpmc_stress.c': ['ptr_compress'],
+    'test_ring_mt_peek_stress.c': ['ptr_compress'],
+    'test_ring_mt_peek_stress_zc.c': ['ptr_compress'],
+    'test_ring_perf.c': ['ptr_compress'],
+    'test_ring_rts_stress.c': ['ptr_compress'],
+    'test_ring_st_peek_stress.c': ['ptr_compress'],
+    'test_ring_st_peek_stress_zc.c': ['ptr_compress'],
+    'test_ring_stress.c': ['ptr_compress'],
     'test_rwlock.c': [],
     'test_sched.c': ['net', 'sched'],
     'test_security.c': ['net', 'security'],
diff --git a/app/test/test_ring.h b/app/test/test_ring.h
index 45c263f3ff..f90662818c 100644
--- a/app/test/test_ring.h
+++ b/app/test/test_ring.h
@@ -5,6 +5,8 @@
 #include <rte_malloc.h>
 #include <rte_ring.h>
 #include <rte_ring_elem.h>
+#include <rte_memcpy.h>
+#include <rte_ptr_compress.h>

 /* API type to call
  * rte_ring_<sp/mp or sc/mc>_enqueue_<bulk/burst>
@@ -25,6 +27,10 @@
 #define TEST_RING_ELEM_BULK 16
 #define TEST_RING_ELEM_BURST 32

+#define TEST_RING_ELEM_BURST_ZC 64
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 128
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 256
+
 #define TEST_RING_IGNORE_API_TYPE ~0U

 /* This function is placed here as it is required for both
@@ -101,6 +107,9 @@ static inline unsigned int
 test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -152,6 +161,46 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n,
 								NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy(zcd.ptr2,
+						(char *)obj + zcd.n1 * esize,
+						(ret - zcd.n1) * esize);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_16(0, obj, zcd.ptr1, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_16(0,
+						obj + (zcd.n1 * 2),
+						zcd.ptr2,
+						(ret - zcd.n1) * 2, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_32(0, obj, zcd.ptr1, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_32(0, obj + zcd.n1,
+						zcd.ptr2, ret - zcd.n1, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
@@ -162,6 +211,9 @@ static inline unsigned int
 test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -213,6 +265,46 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mc_dequeue_burst_elem(r, obj, esize,
 								n, NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy((char *)obj + zcd.n1 * esize,
+						zcd.ptr2,
+						(ret - zcd.n1) * esize);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_16(0, zcd.ptr1, obj, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_16(0, zcd.ptr2,
+						obj + zcd.n1,
+						(ret - zcd.n1) * 2,
+						3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_32(0, zcd.ptr1, obj, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_32(0, zcd.ptr2,
+						obj + zcd.n1, ret - zcd.n1, 3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c
index d7c5a4c30b..367670fd09 100644
--- a/app/test/test_ring_perf.c
+++ b/app/test/test_ring_perf.c
@@ -22,13 +22,13 @@

 #define RING_NAME "RING_PERF"
 #define RING_SIZE 4096
-#define MAX_BURST 32
+#define MAX_BURST 256

 /*
  * the sizes to enqueue and dequeue in testing
  * (marked volatile so they won't be seen as compile-time constants)
  */
-static const volatile unsigned bulk_sizes[] = { 8, 32 };
+static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128, 256 };

 struct lcore_pair {
 	unsigned c1, c2;
@@ -43,26 +43,30 @@ test_ring_print_test_string(unsigned int api_type, int esize,
 	if (esize == -1)
 		printf("legacy APIs");
 	else
-		printf("elem APIs: element size %dB", esize);
+		printf("elem APIs (size:%2dB)", esize);

 	if (api_type == TEST_RING_IGNORE_API_TYPE)
 		return;

 	if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF)
-		printf(": default enqueue/dequeue: ");
+		printf(" - default enqueue/dequeue");
 	else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC)
-		printf(": SP/SC: ");
+		printf(" - SP/SC");
 	else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC)
-		printf(": MP/MC: ");
+		printf(" - MP/MC");

 	if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE)
-		printf("single: ");
+		printf(" - single - ");
 	else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK)
-		printf("bulk (size: %u): ", bsz);
+		printf(" - bulk (n:%-3u) - ", bsz);
 	else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST)
-		printf("burst (size: %u): ", bsz);
+		printf(" - burst (n:%-3u) - ", bsz);
+	else if ((api_type & (TEST_RING_ELEM_BURST_ZC |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0)
+		printf(" - burst zero copy (n:%-3u) - ", bsz);

-	printf("%.2F\n", value);
+	printf("cycles per elem: %.3F\n", value);
 }

 /**** Functions to analyse our core mask to get cores for different tests ***/
@@ -144,28 +148,33 @@ static void
 test_empty_dequeue(struct rte_ring *r, const int esize,
 			const unsigned int api_type)
 {
-	const unsigned int iter_shift = 26;
+	const unsigned int iter_shift = 29;
 	const unsigned int iterations = 1 << iter_shift;
 	unsigned int i = 0;
 	void *burst[MAX_BURST];

+	const unsigned int bulk_iterations = iterations / bulk_sizes[0];
 	const uint64_t start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	for (i = 0; i < bulk_iterations; i++)
 		test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type);
 	const uint64_t end = rte_rdtsc();

 	test_ring_print_test_string(api_type, esize, bulk_sizes[0],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 }

-/*
- * for the separate enqueue and dequeue threads they take in one param
- * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc
- */
-struct thread_params {
+/* describes the ring used by the enqueue and dequeue thread */
+struct ring_params {
 	struct rte_ring *r;
-	unsigned size;        /* input value, the burst size */
-	double spsc, mpmc;    /* output value, the single or multi timings */
+	unsigned int elem_size;
+	unsigned int bulk_sizes_i; /* index into bulk_size array */
+	unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */
+};
+
+/* Used to specify enqueue and dequeue ring operations and their results */
+struct thread_params {
+	struct ring_params *ring_params;
+	double *results; /* result array size must be equal to bulk_sizes */
 };

 /*
@@ -174,16 +183,15 @@ struct thread_params {
  * flag == 1 -> dequeue
  */
 static __rte_always_inline int
-enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
-	struct thread_params *p)
+enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 15;
+	const unsigned int iter_shift = 22;
 	const unsigned int iterations = 1 << iter_shift;
-	struct rte_ring *r = p->r;
-	unsigned int bsize = p->size;
 	unsigned int i;
 	void *burst = NULL;
+	unsigned int n_remaining;
+	const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i];

 #ifdef RTE_USE_C11_MEM_MODEL
 	if (__atomic_fetch_add(&lcore_count, 1, __ATOMIC_RELAXED) + 1 != 2)
@@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 		while(lcore_count != 2)
 			rte_pause();

-	burst = test_ring_calloc(MAX_BURST, esize);
+	burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size);
 	if (burst == NULL)
 		return -1;

 	const uint64_t sp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	const unsigned int bulk_iterations = iterations / bulk_n;
+	for (i = 0; i < bulk_iterations; i++) {
+		n_remaining = bulk_n;
 		do {
 			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_enqueue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_dequeue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			if (ret == 0)
 				rte_pause();
-		} while (!ret);
+			else
+				n_remaining -= ret;
+		} while (n_remaining > 0);
+	}
 	const uint64_t sp_end = rte_rdtsc();

-	const uint64_t mp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
-		do {
-			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			if (ret == 0)
-				rte_pause();
-		} while (!ret);
-	const uint64_t mp_end = rte_rdtsc();
+	p->results[p->ring_params->bulk_sizes_i] =
+			((double)sp_end - sp_start) / iterations;

-	p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize);
-	p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize);
 	return 0;
 }

@@ -243,15 +245,7 @@ enqueue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(0, -1, params);
-}
-
-static int
-enqueue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(0, 16, params);
+	return enqueue_dequeue_bulk_helper(0, params);
 }

 /*
@@ -263,15 +257,7 @@ dequeue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(1, -1, params);
-}
-
-static int
-dequeue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(1, 16, params);
+	return enqueue_dequeue_bulk_helper(1, params);
 }

 /*
@@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p)
  * used to measure ring perf between hyperthreads, cores and sockets.
  */
 static int
-run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize)
+run_on_core_pair(struct lcore_pair *cores,
+		struct thread_params *param1, struct thread_params *param2)
 {
-	lcore_function_t *f1, *f2;
-	struct thread_params param1 = {0}, param2 = {0};
 	unsigned i;
-
-	if (esize == -1) {
-		f1 = enqueue_bulk;
-		f2 = dequeue_bulk;
-	} else {
-		f1 = enqueue_bulk_16B;
-		f2 = dequeue_bulk_16B;
-	}
+	struct ring_params *ring_params = param1->ring_params;

 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		lcore_count = 0;
-		param1.size = param2.size = bulk_sizes[i];
-		param1.r = param2.r = r;
+		ring_params->bulk_sizes_i = i;
 		if (cores->c1 == rte_get_main_lcore()) {
-			rte_eal_remote_launch(f2, &param2, cores->c2);
-			f1(&param1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
+			enqueue_bulk(param1);
 			rte_eal_wait_lcore(cores->c2);
 		} else {
-			rte_eal_remote_launch(f1, &param1, cores->c1);
-			rte_eal_remote_launch(f2, &param2, cores->c2);
+			rte_eal_remote_launch(enqueue_bulk, param1, cores->c1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
 			if (rte_eal_wait_lcore(cores->c1) < 0)
 				return -1;
 			if (rte_eal_wait_lcore(cores->c2) < 0)
 				return -1;
 		}
 		test_ring_print_test_string(
-			TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.spsc + param2.spsc);
-		test_ring_print_test_string(
-			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.mpmc + param2.mpmc);
+				ring_params->ring_flags,
+				ring_params->elem_size,
+				bulk_sizes[i],
+				param1->results[i] + param2->results[i]);
 	}

 	return 0;
@@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 	uint64_t hz = rte_get_timer_hz();
 	uint64_t lcount = 0;
 	const unsigned int lcore = rte_lcore_id();
-	struct thread_params *params = p;
+	struct ring_params *ring_params = p->ring_params;
 	void *burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -346,9 +322,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize)

 	begin = rte_get_timer_cycles();
 	while (time_diff < hz * TIME_MS / 1000) {
-		test_ring_enqueue(params->r, burst, esize, params->size,
+		test_ring_enqueue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
-		test_ring_dequeue(params->r, burst, esize, params->size,
+		test_ring_dequeue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 		lcount++;
 		time_diff = rte_get_timer_cycles() - begin;
@@ -380,7 +358,8 @@ static int
 run_on_all_cores(struct rte_ring *r, const int esize)
 {
 	uint64_t total;
-	struct thread_params param;
+	struct ring_params ring_params = {0};
+	struct thread_params params = { .ring_params = &ring_params };
 	lcore_function_t *lcore_f;
 	unsigned int i, c;

@@ -389,21 +368,20 @@ run_on_all_cores(struct rte_ring *r, const int esize)
 	else
 		lcore_f = load_loop_fn_16B;

-	memset(&param, 0, sizeof(struct thread_params));
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		total = 0;
 		printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]);
-		param.size = bulk_sizes[i];
-		param.r = r;
+		params.ring_params->bulk_sizes_i = i;
+		params.ring_params->r = r;

 		/* clear synchro and start workers */
 		__atomic_store_n(&synchro, 0, __ATOMIC_RELAXED);
-		if (rte_eal_mp_remote_launch(lcore_f, &param, SKIP_MAIN) < 0)
+		if (rte_eal_mp_remote_launch(lcore_f, &params, SKIP_MAIN) < 0)
 			return -1;

 		/* start synchro and launch test on main */
 		__atomic_store_n(&synchro, 1, __ATOMIC_RELAXED);
-		lcore_f(&param);
+		lcore_f(&params);

 		rte_eal_mp_wait_lcore();

@@ -462,9 +440,9 @@ static int
 test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	const unsigned int api_type)
 {
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 26;
 	const unsigned int iterations = 1 << iter_shift;
-	unsigned int sz, i = 0;
+	unsigned int sz, i;
 	void **burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -472,17 +450,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 		return -1;

 	for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) {
+		const unsigned int n = iterations / bulk_sizes[sz];
 		const uint64_t start = rte_rdtsc();
-		for (i = 0; i < iterations; i++) {
+		for (i = 0; i < n; i++) {
 			test_ring_enqueue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 			test_ring_dequeue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 		}
 		const uint64_t end = rte_rdtsc();

 		test_ring_print_test_string(api_type, esize, bulk_sizes[sz],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 	}

 	rte_free(burst);
@@ -490,12 +469,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	return 0;
 }

+static __rte_always_inline int
+test_ring_perf_esize_run_on_two_cores(
+		struct thread_params *param1, struct thread_params *param2)
+{
+	struct lcore_pair cores;
+
+	if (get_two_hyperthreads(&cores) == 0) {
+		printf("\n### Testing using two hyperthreads ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing using two physical cores ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_sockets(&cores) == 0) {
+		printf("\n### Testing using two NUMA nodes ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	return 0;
+}
+
 /* Run all tests for a given element size */
 static __rte_always_inline int
 test_ring_perf_esize(const int esize)
 {
-	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
+	double results_enq[RTE_DIM(bulk_sizes)];
+	double results_deq[RTE_DIM(bulk_sizes)];
+	struct ring_params ring_params = {
+			.elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results_enq };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results_deq };

 	/*
 	 * Performance test for legacy/_elem APIs
@@ -535,22 +545,13 @@ test_ring_perf_esize(const int esize)
 	test_empty_dequeue(r, esize,
 			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);

-	if (get_two_hyperthreads(&cores) == 0) {
-		printf("\n### Testing using two hyperthreads ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.r = r;

-	if (get_two_cores(&cores) == 0) {
-		printf("\n### Testing using two physical cores ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
-	if (get_two_sockets(&cores) == 0) {
-		printf("\n### Testing using two NUMA nodes ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
+
+	ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);

 	printf("\n### Testing using all worker nodes ###\n");
 	if (run_on_all_cores(r, esize) < 0)
@@ -566,6 +567,109 @@ test_ring_perf_esize(const int esize)
 	return -1;
 }

+
+static __rte_always_inline int
+test_ring_perf_compression(void)
+{
+	double results1[RTE_DIM(bulk_sizes)];
+	double results2[RTE_DIM(bulk_sizes)];
+	double results1_comp[2][RTE_DIM(bulk_sizes)];
+	double results2_comp[2][RTE_DIM(bulk_sizes)];
+
+	struct lcore_pair cores;
+	int ret = -1;
+	unsigned int i, j;
+	struct ring_params ring_params = { .elem_size = sizeof(void *) };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results1 };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results2 };
+
+	printf("\n### Testing compression gain ###");
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(void *),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy ###\n");
+		ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	/* rings allow only multiples of 4 as sizes,
+	 * we allocate size 4 despite only using 2 bytes
+	 * and use half of RING_SIZE as the number of elements
+	 */
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE / 2, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[0];
+	param2.results = results2_comp[0];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (16b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[1];
+	param2.results = results2_comp[1];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (32b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	for (j = 0; j < 2; j++) {
+		printf("\n### Potential gain from compression (%d-bit offsets) "
+		"###\n", (j + 1) * 16);
+		for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
+			const double result = results1[i] + results2[i];
+			const double result_comp = results1_comp[j][i] +
+				results2_comp[j][i];
+			const double gain = 100 - (result_comp / result) * 100;
+
+			printf("Gain of %5.1F%% for burst of %-3u elems\n",
+					gain, bulk_sizes[i]);
+		}
+	}
+
+	return ret;
+}
+
 static int
 test_ring_perf(void)
 {
@@ -576,6 +680,10 @@ test_ring_perf(void)
 	if (test_ring_perf_esize(16) == -1)
 		return -1;

+	/* Test for performance gain of compression */
+	if (test_ring_perf_compression() == -1)
+		return -1;
+
 	return 0;
 }

--
2.25.1

[PATCH v9 4/5] docs: add pointer compression guide

[Paul Szczepanek]

Documentation added in the prog guide for the new
utility functions for pointer compression
showing example code and potential usecases.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
---
 MAINTAINERS                                |   1 +
 doc/guides/prog_guide/index.rst            |   1 +
 doc/guides/prog_guide/ptr_compress_lib.rst | 142 +++++++++++++++++++++
 3 files changed, 144 insertions(+)
 create mode 100644 doc/guides/prog_guide/ptr_compress_lib.rst

diff --git a/MAINTAINERS b/MAINTAINERS
index 6f703b1b13..e70f92cdc4 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1688,6 +1688,7 @@ F: lib/pci/
 Pointer Compression
 M: Paul Szczepanek <paul.szczepanek@arm.com>
 F: lib/ptr_compress/
+F: doc/guides/prog_guide/ptr_compress_lib.rst

 Power management
 M: Anatoly Burakov <anatoly.burakov@intel.com>
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index d09d958e6c..6366849eb0 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -73,6 +73,7 @@ Programmer's Guide
     telemetry_lib
     bpf_lib
     graph_lib
+    ptr_compress_lib
     build-sdk-meson
     meson_ut
     build_app
diff --git a/doc/guides/prog_guide/ptr_compress_lib.rst b/doc/guides/prog_guide/ptr_compress_lib.rst
new file mode 100644
index 0000000000..1f9ef24da7
--- /dev/null
+++ b/doc/guides/prog_guide/ptr_compress_lib.rst
@@ -0,0 +1,142 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2024 Arm Limited.
+
+Pointer Compression Library
+===========================
+
+Use ``rte_ptr_compress_16()`` and ``rte_ptr_decompress_16()`` to compress and
+decompress pointers into 16-bit offsets. Use ``rte_ptr_compress_32()`` and
+``rte_ptr_decompress_32()`` to compress and decompress pointers into 32-bit
+offsets.
+
+Compression takes advantage of the fact that pointers are usually located in a
+limited memory region (like a mempool). By converting them to offsets from a
+base memory address they can be stored in fewer bytes. How many bytes are needed
+to store the offset is dictated by the memory region size and alignment of
+objects the pointers point to.
+
+For example, a pointer which is part of a 4GB memory pool can be stored as 32
+bit offset. If the pointer points to memory that is 8 bytes aligned then 3 bits
+can be dropped from the offset and a 32GB memory pool can now fit in 32 bits.
+
+For performance reasons these requirements are not enforced programmatically.
+The programmer is responsible for ensuring that the combination of distance
+from the base pointer and memory alignment allow for storing of the offset in
+the number of bits indicated by the function name (16 or 32). Start of mempool
+memory would be a good candidate for the base pointer. Otherwise any pointer
+that precedes all pointers, is close enough and has the same alignment as the
+pointers being compressed will work.
+
+.. note::
+
+    Performance gains depend on the batch size of pointers and CPU capabilities
+    such as vector extensions. It's important to measure the performance
+    increase on target hardware. A test called ``ring_perf_autotest`` in
+    ``dpdk-test`` can provide the measurements.
+
+Example usage
+~~~~~~~~~~~~~
+
+In this example we send pointers between two cores through a ring. While this
+is a realistic use case the code is simplified for demonstration purposes and
+does not have error handling.
+
+.. code-block:: c
+
+    #include <rte_launch.h>
+    #include <rte_ring.h>
+    #include <rte_ring_elem.h>
+    #include <rte_ptr_compress.h>
+
+    #define ITEMS_ARRAY_SIZE (1024)
+    #define BATCH_SIZE (128)
+    #define ALIGN_EXPONENT (3)
+    #define ITEM_ALIGN (1<<ALIGN_EXPONENT)
+    #define CORE_SEND (1)
+    #define CORE_RECV (2)
+
+    struct item {
+      alignas(ITEM_ALIGN) int a;
+    };
+
+    static struct item items[ITEMS_ARRAY_SIZE] = {0};
+    static struct rte_ring *ring = NULL;
+
+    static int
+    send_compressed(void *args)
+    {
+      struct item *ptrs_send[BATCH_SIZE] = {0};
+      unsigned int n_send = 0;
+      struct rte_ring_zc_data zcd = {0};
+
+      /* in this example we only fill the ptrs_send once and reuse */
+      for (;n_send < BATCH_SIZE; n_send++)
+        ptrs_send[n_send] = &items[n_send];
+
+      for(;;) {
+        n_send = rte_ring_enqueue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        /* compress ptrs_send into offsets */
+        rte_ptr_compress_32(items, /* base pointer */
+          ptrs_send, /* source array to be compressed */
+          zcd.ptr1, /* destination array to store offsets */
+          zcd.n1, /* how many pointers to compress */
+          ALIGN_EXPONENT /* how many bits can we drop from the offset */);
+
+        if (zcd.ptr2 != NULL)
+          rte_ptr_compress_32(items, ptrs_send + zcd.n1,
+            zcd.ptr2, n_send - zcd.n1, ALIGN_EXPONENT);
+
+        rte_ring_enqueue_zc_finish(ring, n_send);
+      }
+      return 1;
+    }
+
+    static int
+    recv_compressed(void *args)
+    {
+      struct item *ptrs_recv[BATCH_SIZE] = {0};
+      unsigned int n_recv;
+      struct rte_ring_zc_data zcd = {0};
+
+      for(;;) {
+        /* receive compressed pointers from the ring */
+        n_recv = rte_ring_dequeue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        rte_ptr_decompress_32(items, /* base pointer */
+          zcd.ptr1, /* source array to decompress */
+          ptrs_recv, /* destination array to store pointers */
+          zcd.n1, /* how many pointers to decompress */
+          ALIGN_EXPONENT /* how many bits were dropped from the offset */);
+
+        /* handle the potential secondary buffer (caused by ring boundary) */
+        if (zcd.ptr2 != NULL)
+          rte_ptr_decompress_32(items,
+            zcd.ptr2,
+            ptrs_recv + zcd.n1,
+            n_recv - zcd.n1,
+            ALIGN_EXPONENT);
+
+        rte_ring_dequeue_zc_finish(ring, n_recv);
+
+        /* ptrs_recv contains what ptrs_send contained in the other thread */
+        /* (...) */
+      }
+      return 1;
+    }
+
+    void
+    compression_example(void)
+    {
+      ring = rte_ring_create_elem(
+        "COMPR_PTRS", sizeof(uint32_t),
+        1024, rte_socket_id(),
+        RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+      rte_eal_remote_launch(send_compressed, NULL, CORE_SEND);
+      rte_eal_remote_launch(recv_compressed, NULL, CORE_RECV);
+
+      for(;;) {}
+    }
--
2.25.1

[PATCH v9 5/5] test: add unit test for ptr compression

[Paul Szczepanek]

Test compresses and decompresses pointers with various combinations
of memory regions and alignments and verify the pointers are
recovered correctly.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
---
 MAINTAINERS                  |   1 +
 app/test/meson.build         |   1 +
 app/test/test_ptr_compress.c | 108 +++++++++++++++++++++++++++++++++++
 3 files changed, 110 insertions(+)
 create mode 100644 app/test/test_ptr_compress.c

diff --git a/MAINTAINERS b/MAINTAINERS
index e70f92cdc4..77ab8a2959 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1688,6 +1688,7 @@ F: lib/pci/
 Pointer Compression
 M: Paul Szczepanek <paul.szczepanek@arm.com>
 F: lib/ptr_compress/
+F: app/test/test_ptr_compress.c
 F: doc/guides/prog_guide/ptr_compress_lib.rst

 Power management
diff --git a/app/test/meson.build b/app/test/meson.build
index df8cc00730..e29258e6ec 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -144,6 +144,7 @@ source_file_deps = {
     'test_power_intel_uncore.c': ['power'],
     'test_power_kvm_vm.c': ['power'],
     'test_prefetch.c': [],
+    'test_ptr_compress.c': ['ptr_compress'],
     'test_rand_perf.c': [],
     'test_rawdev.c': ['rawdev', 'bus_vdev'],
     'test_rcu_qsbr.c': ['rcu', 'hash'],
diff --git a/app/test/test_ptr_compress.c b/app/test/test_ptr_compress.c
new file mode 100644
index 0000000000..972f832776
--- /dev/null
+++ b/app/test/test_ptr_compress.c
@@ -0,0 +1,108 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2024 Arm Limited
+ */
+
+#include "test.h"
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_ptr_compress.h>
+
+#define MAX_ALIGN_EXPONENT 3
+#define MAX_PTRS 16
+#define NUM_BASES 2
+#define NUM_REGIONS 4
+#define MAX_32BIT_REGION ((uint64_t)UINT32_MAX + 1)
+#define MAX_16BIT_REGION (UINT16_MAX + 1)
+
+static int
+test_ptr_compress_params(
+	void *base,
+	uint64_t mem_sz,
+	unsigned int align_exp,
+	unsigned int num_ptrs,
+	bool use_32_bit)
+{
+	unsigned int i;
+	unsigned int align = 1 << align_exp;
+	void *ptrs[MAX_PTRS] = {0};
+	void *ptrs_out[MAX_PTRS] = {0};
+	uint32_t offsets32[MAX_PTRS] = {0};
+	uint16_t offsets16[MAX_PTRS] = {0};
+
+	for (i = 0; i < num_ptrs; i++) {
+		/* make pointers point at memory in steps of align */
+		/* alternate steps from the start and end of memory region */
+		if ((i & 1) == 1)
+			ptrs[i] = (char *)base + mem_sz - i * align;
+		else
+			ptrs[i] = (char *)base + i * align;
+	}
+
+	if (use_32_bit) {
+		rte_ptr_compress_32(base, ptrs, offsets32, num_ptrs, align_exp);
+		rte_ptr_decompress_32(base, offsets32, ptrs_out, num_ptrs,
+				align_exp);
+	} else {
+		rte_ptr_compress_16(base, ptrs, offsets16, num_ptrs, align_exp);
+		rte_ptr_decompress_16(base, offsets16, ptrs_out, num_ptrs,
+				align_exp);
+	}
+
+	TEST_ASSERT_BUFFERS_ARE_EQUAL(ptrs, ptrs_out, sizeof(void *) * num_ptrs,
+		"Decompressed pointers corrupted\nbase pointer: %p, "
+		"memory region size: %" PRIu64 ", alignment exponent: %u, "
+		"num of pointers: %u, using %s offsets",
+		base, mem_sz, align_exp, num_ptrs,
+		use_32_bit ? "32-bit" : "16-bit");
+
+	return 0;
+}
+
+static int
+test_ptr_compress(void)
+{
+	unsigned int j, k, n;
+	int ret = 0;
+	void * const bases[NUM_BASES] = { (void *)0, (void *)UINT16_MAX };
+	/* maximum size for pointers aligned by consecutive powers of 2 */
+	const uint64_t region_sizes_16[NUM_REGIONS] = {
+		MAX_16BIT_REGION,
+		MAX_16BIT_REGION * 2,
+		MAX_16BIT_REGION * 4,
+		MAX_16BIT_REGION * 8,
+	};
+	const uint64_t region_sizes_32[NUM_REGIONS] = {
+		MAX_32BIT_REGION,
+		MAX_32BIT_REGION * 2,
+		MAX_32BIT_REGION * 4,
+		MAX_32BIT_REGION * 8,
+	};
+
+	for (j = 0; j < NUM_REGIONS; j++) {
+		for (k = 0; k < NUM_BASES; k++) {
+			for (n = 1; n < MAX_PTRS; n++) {
+				ret |= test_ptr_compress_params(
+					bases[k],
+					region_sizes_16[j],
+					j /* exponent of alignment */,
+					n,
+					false
+				);
+				ret |= test_ptr_compress_params(
+					bases[k],
+					region_sizes_32[j],
+					j /* exponent of alignment */,
+					n,
+					true
+				);
+				if (ret != 0)
+					return ret;
+			}
+		}
+	}
+
+	return ret;
+}
+
+REGISTER_FAST_TEST(ptr_compress_autotest, true, true, test_ptr_compress);
--
2.25.1

[PATCH v10 0/5] add pointer compression API

[Paul Szczepanek]

This patchset is proposing adding a new header only library
with utility functions that allow compression of arrays of pointers.

Since this is a header only library a patch needed to be added to amend
the build system to allow adding libraries without source files.

When passing caches full of pointers between threads, memory containing
the pointers is copied multiple times which is especially costly between
cores. A compression method will allow us to shrink the memory size
copied.

The compression takes advantage of the fact that pointers are usually
located in a limited memory region (like a mempool). We can compress them
by converting them to offsets from a base memory address.

Offsets can be stored in fewer bytes (dictated by the memory region size
and alignment of the pointer). For example: an 8 byte aligned pointer
which is part of a 32GB memory pool can be stored in 4 bytes. The API is
very generic and does not assume mempool pointers, any pointer can be
passed in.

Compression is based on few and fast operations and especially with vector
instructions leveraged creates minimal overhead.

The API accepts and returns arrays because the overhead means it only is
worth it when done in bulk.

Test is added that shows potential performance gain from compression. In
this test an array of pointers is passed through a ring between two cores.
It shows the gain which is dependent on the bulk operation size. In this
synthetic test run on ampere altra a substantial (up to 25%) performance
gain is seen if done in bulk size larger than 32. At 32 it breaks even and
lower sizes create a small (less than 5%) slowdown due to overhead.

In a more realistic mock application running the l3 forwarding dpdk
example that works in pipeline mode on two cores this translated into a
~5% throughput increase on an ampere altra.

v2:
* addressed review comments (style, explanations and typos)
* lowered bulk iterations closer to original numbers to keep runtime short
* fixed pointer size warning on 32-bit arch
v3:
* added 16-bit versions of compression functions and tests
* added documentation of these new utility functions in the EAL guide
v4:
* added unit test
* fix bug in NEON implementation of 32-bit decompress
v5:
* disable NEON and SVE implementation on AARCH32 due to wrong pointer size
v6:
* added example usage to commit message of the initial commit
v7:
* rebase to remove clashing mailmap changes
v8:
* put ptr compress into its own library
* add depends-on tag
* remove copyright bumps
* typos
v9
* added MAINTAINERS entries, release notes, doc indexes etc.
* added patch for build system to allow header only library
v10
* fixed problem with meson build adding shared deps to static deps

Paul Szczepanek (5):
  lib: allow libraries with no sources
  ptr_compress: add pointer compression library
  test: add pointer compress tests to ring perf test
  docs: add pointer compression guide
  test: add unit test for ptr compression

 MAINTAINERS                                |   6 +
 app/test/meson.build                       |  21 +-
 app/test/test_ptr_compress.c               | 108 +++++++
 app/test/test_ring.h                       |  92 ++++++
 app/test/test_ring_perf.c                  | 352 ++++++++++++++-------
 doc/api/doxy-api-index.md                  |   1 +
 doc/api/doxy-api.conf.in                   |   1 +
 doc/guides/prog_guide/index.rst            |   1 +
 doc/guides/prog_guide/ptr_compress_lib.rst | 142 +++++++++
 doc/guides/rel_notes/release_24_03.rst     |   6 +
 lib/meson.build                            |  17 +
 lib/ptr_compress/meson.build               |   4 +
 lib/ptr_compress/rte_ptr_compress.h        | 266 ++++++++++++++++
 13 files changed, 885 insertions(+), 132 deletions(-)
 create mode 100644 app/test/test_ptr_compress.c
 create mode 100644 doc/guides/prog_guide/ptr_compress_lib.rst
 create mode 100644 lib/ptr_compress/meson.build
 create mode 100644 lib/ptr_compress/rte_ptr_compress.h

--
2.25.1

[PATCH v10 1/5] lib: allow libraries with no sources

[Paul Szczepanek]

Allow header only libraries.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 lib/meson.build | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/lib/meson.build b/lib/meson.build
index 179a272932..7c90602bf5 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -222,6 +222,22 @@ foreach l:libraries
     includes += include_directories(l)
     dpdk_includes += include_directories(l)

+    # special case for header only libraries
+    if sources.length() == 0
+        shared_dep = declare_dependency(include_directories: includes,
+                dependencies: shared_deps)
+        static_dep = declare_dependency(include_directories: includes,
+                dependencies: static_deps)
+        set_variable('shared_rte_' + name, shared_dep)
+        set_variable('static_rte_' + name, static_dep)
+        dpdk_shared_lib_deps += shared_dep
+        dpdk_static_lib_deps += static_dep
+        if developer_mode
+            message('lib/@0@: Defining dependency "@1@"'.format(l, name))
+        endif
+        continue
+    endif
+
     if developer_mode and is_windows and use_function_versioning
         message('@0@: Function versioning is not supported by Windows.'.format(name))
     endif
--
2.25.1

Re: [PATCH v10 1/5] lib: allow libraries with no sources

[Bruce Richardson]

On Mon, Mar 11, 2024 at 08:31:25PM +0000, Paul Szczepanek wrote:
> Allow header only libraries.
> 
> Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>

Acked-by: Bruce Richardson <bruce.richardson@intel.com>

[PATCH v10 2/5] ptr_compress: add pointer compression library

[Paul Szczepanek]

Add a new utility header for compressing pointers. The provided
functions can store pointers in 32-bit or 16-bit offsets.

The compression takes advantage of the fact that pointers are
usually located in a limited memory region (like a mempool).
We can compress them by converting them to offsets from a base
memory address. Offsets can be stored in fewer bytes (dictated
by the memory region size and alignment of the pointer).
For example: an 8 byte aligned pointer which is part of a 32GB
memory pool can be stored in 4 bytes.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
---
 MAINTAINERS                            |   4 +
 doc/api/doxy-api-index.md              |   1 +
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/rel_notes/release_24_03.rst |   6 +
 lib/meson.build                        |   1 +
 lib/ptr_compress/meson.build           |   4 +
 lib/ptr_compress/rte_ptr_compress.h    | 266 +++++++++++++++++++++++++
 7 files changed, 283 insertions(+)
 create mode 100644 lib/ptr_compress/meson.build
 create mode 100644 lib/ptr_compress/rte_ptr_compress.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 4755a68274..6f703b1b13 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1685,6 +1685,10 @@ M: Chenbo Xia <chenbox@nvidia.com>
 M: Gaetan Rivet <grive@u256.net>
 F: lib/pci/

+Pointer Compression
+M: Paul Szczepanek <paul.szczepanek@arm.com>
+F: lib/ptr_compress/
+
 Power management
 M: Anatoly Burakov <anatoly.burakov@intel.com>
 M: David Hunt <david.hunt@intel.com>
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 8c1eb8fafa..f9283154f8 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -222,6 +222,7 @@ The public API headers are grouped by topics:
   [config file](@ref rte_cfgfile.h),
   [key/value args](@ref rte_kvargs.h),
   [argument parsing](@ref rte_argparse.h),
+  [ptr_compress](@ref rte_ptr_compress.h),
   [string](@ref rte_string_fns.h),
   [thread](@ref rte_thread.h)

diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index 27afec8b3b..a8823c046f 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -71,6 +71,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/pipeline \
                           @TOPDIR@/lib/port \
                           @TOPDIR@/lib/power \
+                          @TOPDIR@/lib/ptr_compress \
                           @TOPDIR@/lib/rawdev \
                           @TOPDIR@/lib/rcu \
                           @TOPDIR@/lib/regexdev \
diff --git a/doc/guides/rel_notes/release_24_03.rst b/doc/guides/rel_notes/release_24_03.rst
index 932688ca4d..b82b8c5c0b 100644
--- a/doc/guides/rel_notes/release_24_03.rst
+++ b/doc/guides/rel_notes/release_24_03.rst
@@ -176,6 +176,12 @@ New Features
   * Added power-saving during polling within the ``rte_event_dequeue_burst()`` API.
   * Added support for DMA adapter.

+* **Introduced pointer compression library.**
+
+  Library provides functions to compress and decompress arrays of pointers
+  which can improve application performance under certain conditions.
+  Performance test was added to help users evaluate performance on their setup.
+

 Removed Items
 -------------
diff --git a/lib/meson.build b/lib/meson.build
index 7c90602bf5..63becee142 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -14,6 +14,7 @@ libraries = [
         'argparse',
         'telemetry', # basic info querying
         'eal', # everything depends on eal
+        'ptr_compress',
         'ring',
         'rcu', # rcu depends on ring
         'mempool',
diff --git a/lib/ptr_compress/meson.build b/lib/ptr_compress/meson.build
new file mode 100644
index 0000000000..e92706a45f
--- /dev/null
+++ b/lib/ptr_compress/meson.build
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+headers = files('rte_ptr_compress.h')
diff --git a/lib/ptr_compress/rte_ptr_compress.h b/lib/ptr_compress/rte_ptr_compress.h
new file mode 100644
index 0000000000..97c084003d
--- /dev/null
+++ b/lib/ptr_compress/rte_ptr_compress.h
@@ -0,0 +1,266 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2024 Arm Limited
+ */
+
+#ifndef RTE_PTR_COMPRESS_H
+#define RTE_PTR_COMPRESS_H
+
+/**
+ * @file
+ * Pointer compression and decompression functions.
+ *
+ * When passing arrays full of pointers between threads, memory containing
+ * the pointers is copied multiple times which is especially costly between
+ * cores. These functions allow us to compress the pointers.
+ *
+ * Compression takes advantage of the fact that pointers are usually located in
+ * a limited memory region (like a mempool). We compress them by converting them
+ * to offsets from a base memory address. Offsets can be stored in fewer bytes.
+ *
+ * The compression functions come in two varieties: 32-bit and 16-bit.
+ *
+ * To determine how many bits are needed to compress the pointer calculate
+ * the biggest offset possible (highest value pointer - base pointer)
+ * and shift the value right according to alignment (shift by exponent of the
+ * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by
+ * 3, etc.). The resulting value must fit in either 32 or 16 bits.
+ *
+ * For usage example and further explanation please see "Pointer Compression" in
+ * doc/guides/prog_guide/ptr_compress_lib.rst
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_vect.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Compress pointers into 32-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 32 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 32GB of the base pointer. If there is no such alignment guarantee they
+ * must be within 4GB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_32(void *ptr_base, void **src_table,
+		uint32_t *dest_table, size_t n, uint8_t bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1w(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (i < n);
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	/* right shift is done by left shifting by negative int */
+	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_ptr_table = vld1q_u64((const uint64_t *)src_table + i);
+		v_ptr_table = vsubq_u64(v_ptr_table, v_ptr_base);
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		vst1_u32(dest_table + i, vqmovn_u64(v_ptr_table));
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT32_MAX);
+		dest_table[i] = (uint32_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 32-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_32(void *ptr_base, uint32_t *src_table,
+		void **dest_table, size_t n, uint8_t bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1uw_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (i < n);
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	int64x2_t v_shift = vdupq_n_s64(bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_ptr_table = vmovl_u32(vld1_u32(src_table + i));
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		v_ptr_table = vaddq_u64(v_ptr_table, v_ptr_base);
+		vst1q_u64((uint64_t *)dest_table + i, v_ptr_table);
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+/**
+ * Compress pointers into 16-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 16 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 256KB of the base pointer. If there is no such alignment guarantee
+ * they must be within 64KB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_16(void *ptr_base, void **src_table,
+		uint16_t *dest_table, size_t n, uint8_t bit_shift)
+{
+
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1h(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (i < n);
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT16_MAX);
+		dest_table[i] = (uint16_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 16-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_16(void *ptr_base, uint16_t *src_table,
+		void **dest_table, size_t n, uint8_t bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1uh_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (i < n);
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_PTR_COMPRESS_H */
--
2.25.1

[PATCH v10 3/5] test: add pointer compress tests to ring perf test

[Paul Szczepanek]

Add a test that runs a zero copy burst enqueue and dequeue on a ring
of raw pointers and compressed pointers at different burst sizes to
showcase performance benefits of newly added pointer compression APIs.

Refactored threading code to pass more parameters to threads to
reuse existing code. Added more bulk sizes to showcase their effects
on compression. Adjusted loop iteration numbers to take into account
bulk sizes to keep runtime constant (instead of number of operations).

Adjusted old printfs to match new ones which have aligned numbers.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
---
 app/test/meson.build      |  20 +--
 app/test/test_ring.h      |  92 ++++++++++
 app/test/test_ring_perf.c | 352 +++++++++++++++++++++++++-------------
 3 files changed, 332 insertions(+), 132 deletions(-)

diff --git a/app/test/meson.build b/app/test/meson.build
index 7d909039ae..df8cc00730 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -156,16 +156,16 @@ source_file_deps = {
 #    'test_resource.c': [],
     'test_rib.c': ['net', 'rib'],
     'test_rib6.c': ['net', 'rib'],
-    'test_ring.c': [],
-    'test_ring_hts_stress.c': [],
-    'test_ring_mpmc_stress.c': [],
-    'test_ring_mt_peek_stress.c': [],
-    'test_ring_mt_peek_stress_zc.c': [],
-    'test_ring_perf.c': [],
-    'test_ring_rts_stress.c': [],
-    'test_ring_st_peek_stress.c': [],
-    'test_ring_st_peek_stress_zc.c': [],
-    'test_ring_stress.c': [],
+    'test_ring.c': ['ptr_compress'],
+    'test_ring_hts_stress.c': ['ptr_compress'],
+    'test_ring_mpmc_stress.c': ['ptr_compress'],
+    'test_ring_mt_peek_stress.c': ['ptr_compress'],
+    'test_ring_mt_peek_stress_zc.c': ['ptr_compress'],
+    'test_ring_perf.c': ['ptr_compress'],
+    'test_ring_rts_stress.c': ['ptr_compress'],
+    'test_ring_st_peek_stress.c': ['ptr_compress'],
+    'test_ring_st_peek_stress_zc.c': ['ptr_compress'],
+    'test_ring_stress.c': ['ptr_compress'],
     'test_rwlock.c': [],
     'test_sched.c': ['net', 'sched'],
     'test_security.c': ['net', 'security'],
diff --git a/app/test/test_ring.h b/app/test/test_ring.h
index 45c263f3ff..f90662818c 100644
--- a/app/test/test_ring.h
+++ b/app/test/test_ring.h
@@ -5,6 +5,8 @@
 #include <rte_malloc.h>
 #include <rte_ring.h>
 #include <rte_ring_elem.h>
+#include <rte_memcpy.h>
+#include <rte_ptr_compress.h>

 /* API type to call
  * rte_ring_<sp/mp or sc/mc>_enqueue_<bulk/burst>
@@ -25,6 +27,10 @@
 #define TEST_RING_ELEM_BULK 16
 #define TEST_RING_ELEM_BURST 32

+#define TEST_RING_ELEM_BURST_ZC 64
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 128
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 256
+
 #define TEST_RING_IGNORE_API_TYPE ~0U

 /* This function is placed here as it is required for both
@@ -101,6 +107,9 @@ static inline unsigned int
 test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -152,6 +161,46 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n,
 								NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy(zcd.ptr2,
+						(char *)obj + zcd.n1 * esize,
+						(ret - zcd.n1) * esize);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_16(0, obj, zcd.ptr1, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_16(0,
+						obj + (zcd.n1 * 2),
+						zcd.ptr2,
+						(ret - zcd.n1) * 2, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_32(0, obj, zcd.ptr1, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_32(0, obj + zcd.n1,
+						zcd.ptr2, ret - zcd.n1, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
@@ -162,6 +211,9 @@ static inline unsigned int
 test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -213,6 +265,46 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mc_dequeue_burst_elem(r, obj, esize,
 								n, NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy((char *)obj + zcd.n1 * esize,
+						zcd.ptr2,
+						(ret - zcd.n1) * esize);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_16(0, zcd.ptr1, obj, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_16(0, zcd.ptr2,
+						obj + zcd.n1,
+						(ret - zcd.n1) * 2,
+						3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_32(0, zcd.ptr1, obj, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_32(0, zcd.ptr2,
+						obj + zcd.n1, ret - zcd.n1, 3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c
index d7c5a4c30b..367670fd09 100644
--- a/app/test/test_ring_perf.c
+++ b/app/test/test_ring_perf.c
@@ -22,13 +22,13 @@

 #define RING_NAME "RING_PERF"
 #define RING_SIZE 4096
-#define MAX_BURST 32
+#define MAX_BURST 256

 /*
  * the sizes to enqueue and dequeue in testing
  * (marked volatile so they won't be seen as compile-time constants)
  */
-static const volatile unsigned bulk_sizes[] = { 8, 32 };
+static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128, 256 };

 struct lcore_pair {
 	unsigned c1, c2;
@@ -43,26 +43,30 @@ test_ring_print_test_string(unsigned int api_type, int esize,
 	if (esize == -1)
 		printf("legacy APIs");
 	else
-		printf("elem APIs: element size %dB", esize);
+		printf("elem APIs (size:%2dB)", esize);

 	if (api_type == TEST_RING_IGNORE_API_TYPE)
 		return;

 	if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF)
-		printf(": default enqueue/dequeue: ");
+		printf(" - default enqueue/dequeue");
 	else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC)
-		printf(": SP/SC: ");
+		printf(" - SP/SC");
 	else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC)
-		printf(": MP/MC: ");
+		printf(" - MP/MC");

 	if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE)
-		printf("single: ");
+		printf(" - single - ");
 	else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK)
-		printf("bulk (size: %u): ", bsz);
+		printf(" - bulk (n:%-3u) - ", bsz);
 	else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST)
-		printf("burst (size: %u): ", bsz);
+		printf(" - burst (n:%-3u) - ", bsz);
+	else if ((api_type & (TEST_RING_ELEM_BURST_ZC |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0)
+		printf(" - burst zero copy (n:%-3u) - ", bsz);

-	printf("%.2F\n", value);
+	printf("cycles per elem: %.3F\n", value);
 }

 /**** Functions to analyse our core mask to get cores for different tests ***/
@@ -144,28 +148,33 @@ static void
 test_empty_dequeue(struct rte_ring *r, const int esize,
 			const unsigned int api_type)
 {
-	const unsigned int iter_shift = 26;
+	const unsigned int iter_shift = 29;
 	const unsigned int iterations = 1 << iter_shift;
 	unsigned int i = 0;
 	void *burst[MAX_BURST];

+	const unsigned int bulk_iterations = iterations / bulk_sizes[0];
 	const uint64_t start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	for (i = 0; i < bulk_iterations; i++)
 		test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type);
 	const uint64_t end = rte_rdtsc();

 	test_ring_print_test_string(api_type, esize, bulk_sizes[0],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 }

-/*
- * for the separate enqueue and dequeue threads they take in one param
- * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc
- */
-struct thread_params {
+/* describes the ring used by the enqueue and dequeue thread */
+struct ring_params {
 	struct rte_ring *r;
-	unsigned size;        /* input value, the burst size */
-	double spsc, mpmc;    /* output value, the single or multi timings */
+	unsigned int elem_size;
+	unsigned int bulk_sizes_i; /* index into bulk_size array */
+	unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */
+};
+
+/* Used to specify enqueue and dequeue ring operations and their results */
+struct thread_params {
+	struct ring_params *ring_params;
+	double *results; /* result array size must be equal to bulk_sizes */
 };

 /*
@@ -174,16 +183,15 @@ struct thread_params {
  * flag == 1 -> dequeue
  */
 static __rte_always_inline int
-enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
-	struct thread_params *p)
+enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 15;
+	const unsigned int iter_shift = 22;
 	const unsigned int iterations = 1 << iter_shift;
-	struct rte_ring *r = p->r;
-	unsigned int bsize = p->size;
 	unsigned int i;
 	void *burst = NULL;
+	unsigned int n_remaining;
+	const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i];

 #ifdef RTE_USE_C11_MEM_MODEL
 	if (__atomic_fetch_add(&lcore_count, 1, __ATOMIC_RELAXED) + 1 != 2)
@@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 		while(lcore_count != 2)
 			rte_pause();

-	burst = test_ring_calloc(MAX_BURST, esize);
+	burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size);
 	if (burst == NULL)
 		return -1;

 	const uint64_t sp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	const unsigned int bulk_iterations = iterations / bulk_n;
+	for (i = 0; i < bulk_iterations; i++) {
+		n_remaining = bulk_n;
 		do {
 			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_enqueue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_dequeue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			if (ret == 0)
 				rte_pause();
-		} while (!ret);
+			else
+				n_remaining -= ret;
+		} while (n_remaining > 0);
+	}
 	const uint64_t sp_end = rte_rdtsc();

-	const uint64_t mp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
-		do {
-			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			if (ret == 0)
-				rte_pause();
-		} while (!ret);
-	const uint64_t mp_end = rte_rdtsc();
+	p->results[p->ring_params->bulk_sizes_i] =
+			((double)sp_end - sp_start) / iterations;

-	p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize);
-	p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize);
 	return 0;
 }

@@ -243,15 +245,7 @@ enqueue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(0, -1, params);
-}
-
-static int
-enqueue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(0, 16, params);
+	return enqueue_dequeue_bulk_helper(0, params);
 }

 /*
@@ -263,15 +257,7 @@ dequeue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(1, -1, params);
-}
-
-static int
-dequeue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(1, 16, params);
+	return enqueue_dequeue_bulk_helper(1, params);
 }

 /*
@@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p)
  * used to measure ring perf between hyperthreads, cores and sockets.
  */
 static int
-run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize)
+run_on_core_pair(struct lcore_pair *cores,
+		struct thread_params *param1, struct thread_params *param2)
 {
-	lcore_function_t *f1, *f2;
-	struct thread_params param1 = {0}, param2 = {0};
 	unsigned i;
-
-	if (esize == -1) {
-		f1 = enqueue_bulk;
-		f2 = dequeue_bulk;
-	} else {
-		f1 = enqueue_bulk_16B;
-		f2 = dequeue_bulk_16B;
-	}
+	struct ring_params *ring_params = param1->ring_params;

 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		lcore_count = 0;
-		param1.size = param2.size = bulk_sizes[i];
-		param1.r = param2.r = r;
+		ring_params->bulk_sizes_i = i;
 		if (cores->c1 == rte_get_main_lcore()) {
-			rte_eal_remote_launch(f2, &param2, cores->c2);
-			f1(&param1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
+			enqueue_bulk(param1);
 			rte_eal_wait_lcore(cores->c2);
 		} else {
-			rte_eal_remote_launch(f1, &param1, cores->c1);
-			rte_eal_remote_launch(f2, &param2, cores->c2);
+			rte_eal_remote_launch(enqueue_bulk, param1, cores->c1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
 			if (rte_eal_wait_lcore(cores->c1) < 0)
 				return -1;
 			if (rte_eal_wait_lcore(cores->c2) < 0)
 				return -1;
 		}
 		test_ring_print_test_string(
-			TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.spsc + param2.spsc);
-		test_ring_print_test_string(
-			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.mpmc + param2.mpmc);
+				ring_params->ring_flags,
+				ring_params->elem_size,
+				bulk_sizes[i],
+				param1->results[i] + param2->results[i]);
 	}

 	return 0;
@@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 	uint64_t hz = rte_get_timer_hz();
 	uint64_t lcount = 0;
 	const unsigned int lcore = rte_lcore_id();
-	struct thread_params *params = p;
+	struct ring_params *ring_params = p->ring_params;
 	void *burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -346,9 +322,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize)

 	begin = rte_get_timer_cycles();
 	while (time_diff < hz * TIME_MS / 1000) {
-		test_ring_enqueue(params->r, burst, esize, params->size,
+		test_ring_enqueue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
-		test_ring_dequeue(params->r, burst, esize, params->size,
+		test_ring_dequeue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 		lcount++;
 		time_diff = rte_get_timer_cycles() - begin;
@@ -380,7 +358,8 @@ static int
 run_on_all_cores(struct rte_ring *r, const int esize)
 {
 	uint64_t total;
-	struct thread_params param;
+	struct ring_params ring_params = {0};
+	struct thread_params params = { .ring_params = &ring_params };
 	lcore_function_t *lcore_f;
 	unsigned int i, c;

@@ -389,21 +368,20 @@ run_on_all_cores(struct rte_ring *r, const int esize)
 	else
 		lcore_f = load_loop_fn_16B;

-	memset(&param, 0, sizeof(struct thread_params));
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		total = 0;
 		printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]);
-		param.size = bulk_sizes[i];
-		param.r = r;
+		params.ring_params->bulk_sizes_i = i;
+		params.ring_params->r = r;

 		/* clear synchro and start workers */
 		__atomic_store_n(&synchro, 0, __ATOMIC_RELAXED);
-		if (rte_eal_mp_remote_launch(lcore_f, &param, SKIP_MAIN) < 0)
+		if (rte_eal_mp_remote_launch(lcore_f, &params, SKIP_MAIN) < 0)
 			return -1;

 		/* start synchro and launch test on main */
 		__atomic_store_n(&synchro, 1, __ATOMIC_RELAXED);
-		lcore_f(&param);
+		lcore_f(&params);

 		rte_eal_mp_wait_lcore();

@@ -462,9 +440,9 @@ static int
 test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	const unsigned int api_type)
 {
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 26;
 	const unsigned int iterations = 1 << iter_shift;
-	unsigned int sz, i = 0;
+	unsigned int sz, i;
 	void **burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -472,17 +450,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 		return -1;

 	for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) {
+		const unsigned int n = iterations / bulk_sizes[sz];
 		const uint64_t start = rte_rdtsc();
-		for (i = 0; i < iterations; i++) {
+		for (i = 0; i < n; i++) {
 			test_ring_enqueue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 			test_ring_dequeue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 		}
 		const uint64_t end = rte_rdtsc();

 		test_ring_print_test_string(api_type, esize, bulk_sizes[sz],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 	}

 	rte_free(burst);
@@ -490,12 +469,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	return 0;
 }

+static __rte_always_inline int
+test_ring_perf_esize_run_on_two_cores(
+		struct thread_params *param1, struct thread_params *param2)
+{
+	struct lcore_pair cores;
+
+	if (get_two_hyperthreads(&cores) == 0) {
+		printf("\n### Testing using two hyperthreads ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing using two physical cores ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_sockets(&cores) == 0) {
+		printf("\n### Testing using two NUMA nodes ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	return 0;
+}
+
 /* Run all tests for a given element size */
 static __rte_always_inline int
 test_ring_perf_esize(const int esize)
 {
-	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
+	double results_enq[RTE_DIM(bulk_sizes)];
+	double results_deq[RTE_DIM(bulk_sizes)];
+	struct ring_params ring_params = {
+			.elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results_enq };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results_deq };

 	/*
 	 * Performance test for legacy/_elem APIs
@@ -535,22 +545,13 @@ test_ring_perf_esize(const int esize)
 	test_empty_dequeue(r, esize,
 			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);

-	if (get_two_hyperthreads(&cores) == 0) {
-		printf("\n### Testing using two hyperthreads ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.r = r;

-	if (get_two_cores(&cores) == 0) {
-		printf("\n### Testing using two physical cores ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
-	if (get_two_sockets(&cores) == 0) {
-		printf("\n### Testing using two NUMA nodes ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
+
+	ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);

 	printf("\n### Testing using all worker nodes ###\n");
 	if (run_on_all_cores(r, esize) < 0)
@@ -566,6 +567,109 @@ test_ring_perf_esize(const int esize)
 	return -1;
 }

+
+static __rte_always_inline int
+test_ring_perf_compression(void)
+{
+	double results1[RTE_DIM(bulk_sizes)];
+	double results2[RTE_DIM(bulk_sizes)];
+	double results1_comp[2][RTE_DIM(bulk_sizes)];
+	double results2_comp[2][RTE_DIM(bulk_sizes)];
+
+	struct lcore_pair cores;
+	int ret = -1;
+	unsigned int i, j;
+	struct ring_params ring_params = { .elem_size = sizeof(void *) };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results1 };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results2 };
+
+	printf("\n### Testing compression gain ###");
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(void *),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy ###\n");
+		ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	/* rings allow only multiples of 4 as sizes,
+	 * we allocate size 4 despite only using 2 bytes
+	 * and use half of RING_SIZE as the number of elements
+	 */
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE / 2, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[0];
+	param2.results = results2_comp[0];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (16b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[1];
+	param2.results = results2_comp[1];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (32b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	for (j = 0; j < 2; j++) {
+		printf("\n### Potential gain from compression (%d-bit offsets) "
+		"###\n", (j + 1) * 16);
+		for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
+			const double result = results1[i] + results2[i];
+			const double result_comp = results1_comp[j][i] +
+				results2_comp[j][i];
+			const double gain = 100 - (result_comp / result) * 100;
+
+			printf("Gain of %5.1F%% for burst of %-3u elems\n",
+					gain, bulk_sizes[i]);
+		}
+	}
+
+	return ret;
+}
+
 static int
 test_ring_perf(void)
 {
@@ -576,6 +680,10 @@ test_ring_perf(void)
 	if (test_ring_perf_esize(16) == -1)
 		return -1;

+	/* Test for performance gain of compression */
+	if (test_ring_perf_compression() == -1)
+		return -1;
+
 	return 0;
 }

--
2.25.1

[PATCH v10 4/5] docs: add pointer compression guide

[Paul Szczepanek]

Documentation added in the prog guide for the new
utility functions for pointer compression
showing example code and potential usecases.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
---
 MAINTAINERS                                |   1 +
 doc/guides/prog_guide/index.rst            |   1 +
 doc/guides/prog_guide/ptr_compress_lib.rst | 142 +++++++++++++++++++++
 3 files changed, 144 insertions(+)
 create mode 100644 doc/guides/prog_guide/ptr_compress_lib.rst

diff --git a/MAINTAINERS b/MAINTAINERS
index 6f703b1b13..e70f92cdc4 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1688,6 +1688,7 @@ F: lib/pci/
 Pointer Compression
 M: Paul Szczepanek <paul.szczepanek@arm.com>
 F: lib/ptr_compress/
+F: doc/guides/prog_guide/ptr_compress_lib.rst

 Power management
 M: Anatoly Burakov <anatoly.burakov@intel.com>
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index d09d958e6c..6366849eb0 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -73,6 +73,7 @@ Programmer's Guide
     telemetry_lib
     bpf_lib
     graph_lib
+    ptr_compress_lib
     build-sdk-meson
     meson_ut
     build_app
diff --git a/doc/guides/prog_guide/ptr_compress_lib.rst b/doc/guides/prog_guide/ptr_compress_lib.rst
new file mode 100644
index 0000000000..1f9ef24da7
--- /dev/null
+++ b/doc/guides/prog_guide/ptr_compress_lib.rst
@@ -0,0 +1,142 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2024 Arm Limited.
+
+Pointer Compression Library
+===========================
+
+Use ``rte_ptr_compress_16()`` and ``rte_ptr_decompress_16()`` to compress and
+decompress pointers into 16-bit offsets. Use ``rte_ptr_compress_32()`` and
+``rte_ptr_decompress_32()`` to compress and decompress pointers into 32-bit
+offsets.
+
+Compression takes advantage of the fact that pointers are usually located in a
+limited memory region (like a mempool). By converting them to offsets from a
+base memory address they can be stored in fewer bytes. How many bytes are needed
+to store the offset is dictated by the memory region size and alignment of
+objects the pointers point to.
+
+For example, a pointer which is part of a 4GB memory pool can be stored as 32
+bit offset. If the pointer points to memory that is 8 bytes aligned then 3 bits
+can be dropped from the offset and a 32GB memory pool can now fit in 32 bits.
+
+For performance reasons these requirements are not enforced programmatically.
+The programmer is responsible for ensuring that the combination of distance
+from the base pointer and memory alignment allow for storing of the offset in
+the number of bits indicated by the function name (16 or 32). Start of mempool
+memory would be a good candidate for the base pointer. Otherwise any pointer
+that precedes all pointers, is close enough and has the same alignment as the
+pointers being compressed will work.
+
+.. note::
+
+    Performance gains depend on the batch size of pointers and CPU capabilities
+    such as vector extensions. It's important to measure the performance
+    increase on target hardware. A test called ``ring_perf_autotest`` in
+    ``dpdk-test`` can provide the measurements.
+
+Example usage
+~~~~~~~~~~~~~
+
+In this example we send pointers between two cores through a ring. While this
+is a realistic use case the code is simplified for demonstration purposes and
+does not have error handling.
+
+.. code-block:: c
+
+    #include <rte_launch.h>
+    #include <rte_ring.h>
+    #include <rte_ring_elem.h>
+    #include <rte_ptr_compress.h>
+
+    #define ITEMS_ARRAY_SIZE (1024)
+    #define BATCH_SIZE (128)
+    #define ALIGN_EXPONENT (3)
+    #define ITEM_ALIGN (1<<ALIGN_EXPONENT)
+    #define CORE_SEND (1)
+    #define CORE_RECV (2)
+
+    struct item {
+      alignas(ITEM_ALIGN) int a;
+    };
+
+    static struct item items[ITEMS_ARRAY_SIZE] = {0};
+    static struct rte_ring *ring = NULL;
+
+    static int
+    send_compressed(void *args)
+    {
+      struct item *ptrs_send[BATCH_SIZE] = {0};
+      unsigned int n_send = 0;
+      struct rte_ring_zc_data zcd = {0};
+
+      /* in this example we only fill the ptrs_send once and reuse */
+      for (;n_send < BATCH_SIZE; n_send++)
+        ptrs_send[n_send] = &items[n_send];
+
+      for(;;) {
+        n_send = rte_ring_enqueue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        /* compress ptrs_send into offsets */
+        rte_ptr_compress_32(items, /* base pointer */
+          ptrs_send, /* source array to be compressed */
+          zcd.ptr1, /* destination array to store offsets */
+          zcd.n1, /* how many pointers to compress */
+          ALIGN_EXPONENT /* how many bits can we drop from the offset */);
+
+        if (zcd.ptr2 != NULL)
+          rte_ptr_compress_32(items, ptrs_send + zcd.n1,
+            zcd.ptr2, n_send - zcd.n1, ALIGN_EXPONENT);
+
+        rte_ring_enqueue_zc_finish(ring, n_send);
+      }
+      return 1;
+    }
+
+    static int
+    recv_compressed(void *args)
+    {
+      struct item *ptrs_recv[BATCH_SIZE] = {0};
+      unsigned int n_recv;
+      struct rte_ring_zc_data zcd = {0};
+
+      for(;;) {
+        /* receive compressed pointers from the ring */
+        n_recv = rte_ring_dequeue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        rte_ptr_decompress_32(items, /* base pointer */
+          zcd.ptr1, /* source array to decompress */
+          ptrs_recv, /* destination array to store pointers */
+          zcd.n1, /* how many pointers to decompress */
+          ALIGN_EXPONENT /* how many bits were dropped from the offset */);
+
+        /* handle the potential secondary buffer (caused by ring boundary) */
+        if (zcd.ptr2 != NULL)
+          rte_ptr_decompress_32(items,
+            zcd.ptr2,
+            ptrs_recv + zcd.n1,
+            n_recv - zcd.n1,
+            ALIGN_EXPONENT);
+
+        rte_ring_dequeue_zc_finish(ring, n_recv);
+
+        /* ptrs_recv contains what ptrs_send contained in the other thread */
+        /* (...) */
+      }
+      return 1;
+    }
+
+    void
+    compression_example(void)
+    {
+      ring = rte_ring_create_elem(
+        "COMPR_PTRS", sizeof(uint32_t),
+        1024, rte_socket_id(),
+        RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+      rte_eal_remote_launch(send_compressed, NULL, CORE_SEND);
+      rte_eal_remote_launch(recv_compressed, NULL, CORE_RECV);
+
+      for(;;) {}
+    }
--
2.25.1

[PATCH v10 5/5] test: add unit test for ptr compression

[Paul Szczepanek]

Test compresses and decompresses pointers with various combinations
of memory regions and alignments and verify the pointers are
recovered correctly.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
---
 MAINTAINERS                  |   1 +
 app/test/meson.build         |   1 +
 app/test/test_ptr_compress.c | 108 +++++++++++++++++++++++++++++++++++
 3 files changed, 110 insertions(+)
 create mode 100644 app/test/test_ptr_compress.c

diff --git a/MAINTAINERS b/MAINTAINERS
index e70f92cdc4..77ab8a2959 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1688,6 +1688,7 @@ F: lib/pci/
 Pointer Compression
 M: Paul Szczepanek <paul.szczepanek@arm.com>
 F: lib/ptr_compress/
+F: app/test/test_ptr_compress.c
 F: doc/guides/prog_guide/ptr_compress_lib.rst

 Power management
diff --git a/app/test/meson.build b/app/test/meson.build
index df8cc00730..e29258e6ec 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -144,6 +144,7 @@ source_file_deps = {
     'test_power_intel_uncore.c': ['power'],
     'test_power_kvm_vm.c': ['power'],
     'test_prefetch.c': [],
+    'test_ptr_compress.c': ['ptr_compress'],
     'test_rand_perf.c': [],
     'test_rawdev.c': ['rawdev', 'bus_vdev'],
     'test_rcu_qsbr.c': ['rcu', 'hash'],
diff --git a/app/test/test_ptr_compress.c b/app/test/test_ptr_compress.c
new file mode 100644
index 0000000000..972f832776
--- /dev/null
+++ b/app/test/test_ptr_compress.c
@@ -0,0 +1,108 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2024 Arm Limited
+ */
+
+#include "test.h"
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_ptr_compress.h>
+
+#define MAX_ALIGN_EXPONENT 3
+#define MAX_PTRS 16
+#define NUM_BASES 2
+#define NUM_REGIONS 4
+#define MAX_32BIT_REGION ((uint64_t)UINT32_MAX + 1)
+#define MAX_16BIT_REGION (UINT16_MAX + 1)
+
+static int
+test_ptr_compress_params(
+	void *base,
+	uint64_t mem_sz,
+	unsigned int align_exp,
+	unsigned int num_ptrs,
+	bool use_32_bit)
+{
+	unsigned int i;
+	unsigned int align = 1 << align_exp;
+	void *ptrs[MAX_PTRS] = {0};
+	void *ptrs_out[MAX_PTRS] = {0};
+	uint32_t offsets32[MAX_PTRS] = {0};
+	uint16_t offsets16[MAX_PTRS] = {0};
+
+	for (i = 0; i < num_ptrs; i++) {
+		/* make pointers point at memory in steps of align */
+		/* alternate steps from the start and end of memory region */
+		if ((i & 1) == 1)
+			ptrs[i] = (char *)base + mem_sz - i * align;
+		else
+			ptrs[i] = (char *)base + i * align;
+	}
+
+	if (use_32_bit) {
+		rte_ptr_compress_32(base, ptrs, offsets32, num_ptrs, align_exp);
+		rte_ptr_decompress_32(base, offsets32, ptrs_out, num_ptrs,
+				align_exp);
+	} else {
+		rte_ptr_compress_16(base, ptrs, offsets16, num_ptrs, align_exp);
+		rte_ptr_decompress_16(base, offsets16, ptrs_out, num_ptrs,
+				align_exp);
+	}
+
+	TEST_ASSERT_BUFFERS_ARE_EQUAL(ptrs, ptrs_out, sizeof(void *) * num_ptrs,
+		"Decompressed pointers corrupted\nbase pointer: %p, "
+		"memory region size: %" PRIu64 ", alignment exponent: %u, "
+		"num of pointers: %u, using %s offsets",
+		base, mem_sz, align_exp, num_ptrs,
+		use_32_bit ? "32-bit" : "16-bit");
+
+	return 0;
+}
+
+static int
+test_ptr_compress(void)
+{
+	unsigned int j, k, n;
+	int ret = 0;
+	void * const bases[NUM_BASES] = { (void *)0, (void *)UINT16_MAX };
+	/* maximum size for pointers aligned by consecutive powers of 2 */
+	const uint64_t region_sizes_16[NUM_REGIONS] = {
+		MAX_16BIT_REGION,
+		MAX_16BIT_REGION * 2,
+		MAX_16BIT_REGION * 4,
+		MAX_16BIT_REGION * 8,
+	};
+	const uint64_t region_sizes_32[NUM_REGIONS] = {
+		MAX_32BIT_REGION,
+		MAX_32BIT_REGION * 2,
+		MAX_32BIT_REGION * 4,
+		MAX_32BIT_REGION * 8,
+	};
+
+	for (j = 0; j < NUM_REGIONS; j++) {
+		for (k = 0; k < NUM_BASES; k++) {
+			for (n = 1; n < MAX_PTRS; n++) {
+				ret |= test_ptr_compress_params(
+					bases[k],
+					region_sizes_16[j],
+					j /* exponent of alignment */,
+					n,
+					false
+				);
+				ret |= test_ptr_compress_params(
+					bases[k],
+					region_sizes_32[j],
+					j /* exponent of alignment */,
+					n,
+					true
+				);
+				if (ret != 0)
+					return ret;
+			}
+		}
+	}
+
+	return ret;
+}
+
+REGISTER_FAST_TEST(ptr_compress_autotest, true, true, test_ptr_compress);
--
2.25.1

[PATCH v11 0/6] add pointer compression API

[Paul Szczepanek]

This patchset is proposing adding a new header only library
with utility functions that allow compression of arrays of pointers.

Since this is a header only library a patch needed to be added to amend
the build system to allow adding libraries without source files.

When passing caches full of pointers between threads, memory containing
the pointers is copied multiple times which is especially costly between
cores. A compression method will allow us to shrink the memory size
copied.

The compression takes advantage of the fact that pointers are usually
located in a limited memory region. We can compress them by converting them
to offsets from a base memory address.

Offsets can be stored in fewer bytes (dictated by the memory region size
and alignment of the pointer). For example: an 8 byte aligned pointer
which is part of a 32GB memory pool can be stored in 4 bytes. The API is
very generic and does not assume mempool pointers, any pointer can be
passed in.

Compression is based on few and fast operations and especially with vector
instructions leveraged creates minimal overhead.

The API accepts and returns arrays because the overhead means it only is
worth it when done in bulk.

Test is added that shows potential performance gain from compression. In
this test an array of pointers is passed through a ring between two cores.
It shows the gain which is dependent on the bulk operation size. In this
synthetic test run on ampere altra a substantial (up to 25%) performance
gain is seen if done in bulk size larger than 32. At 32 it breaks even and
lower sizes create a small (less than 5%) slowdown due to overhead.

In a more realistic mock application running the l3 forwarding dpdk
example that works in pipeline mode on two cores this translated into a
~5% throughput increase on an ampere altra.

v2:
* addressed review comments (style, explanations and typos)
* lowered bulk iterations closer to original numbers to keep runtime short
* fixed pointer size warning on 32-bit arch
v3:
* added 16-bit versions of compression functions and tests
* added documentation of these new utility functions in the EAL guide
v4:
* added unit test
* fix bug in NEON implementation of 32-bit decompress
v5:
* disable NEON and SVE implementation on AARCH32 due to wrong pointer size
v6:
* added example usage to commit message of the initial commit
v7:
* rebase to remove clashing mailmap changes
v8:
* put ptr compress into its own library
* add depends-on tag
* remove copyright bumps
* typos
v9
* added MAINTAINERS entries, release notes, doc indexes etc.
* added patch for build system to allow header only library
v10
* fixed problem with meson build adding shared deps to static deps
v11
* added mempool functions to get information about memory range and
alignment
* added tests for the new mempool functions
* added macros to help find the parameters for compression functions
* minor improvement in the SVE compression code
* amended documentation to reflect these changes

Paul Szczepanek (6):
  lib: allow libraries with no sources
  mempool: add functions to get extra mempool info
  ptr_compress: add pointer compression library
  test: add pointer compress tests to ring perf test
  docs: add pointer compression guide
  test: add unit test for ptr compression

 MAINTAINERS                                |   6 +
 app/test/meson.build                       |  21 +-
 app/test/test_mempool.c                    |  61 ++++
 app/test/test_ptr_compress.c               | 110 +++++++
 app/test/test_ring.h                       |  94 ++++++
 app/test/test_ring_perf.c                  | 352 ++++++++++++++-------
 doc/api/doxy-api-index.md                  |   1 +
 doc/api/doxy-api.conf.in                   |   1 +
 doc/guides/prog_guide/index.rst            |   1 +
 doc/guides/prog_guide/ptr_compress_lib.rst | 160 ++++++++++
 doc/guides/rel_notes/release_24_07.rst     |   5 +
 lib/mempool/rte_mempool.c                  |  39 +++
 lib/mempool/rte_mempool.h                  |  37 +++
 lib/mempool/version.map                    |   3 +
 lib/meson.build                            |  17 +
 lib/ptr_compress/meson.build               |   4 +
 lib/ptr_compress/rte_ptr_compress.h        | 278 ++++++++++++++++
 17 files changed, 1058 insertions(+), 132 deletions(-)
 create mode 100644 app/test/test_ptr_compress.c
 create mode 100644 doc/guides/prog_guide/ptr_compress_lib.rst
 create mode 100644 lib/ptr_compress/meson.build
 create mode 100644 lib/ptr_compress/rte_ptr_compress.h

--
2.25.1

[PATCH v11 1/6] lib: allow libraries with no sources

[Paul Szczepanek]

Allow header only libraries.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@arm.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
 lib/meson.build | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/lib/meson.build b/lib/meson.build
index 179a272932..7c90602bf5 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -222,6 +222,22 @@ foreach l:libraries
     includes += include_directories(l)
     dpdk_includes += include_directories(l)

+    # special case for header only libraries
+    if sources.length() == 0
+        shared_dep = declare_dependency(include_directories: includes,
+                dependencies: shared_deps)
+        static_dep = declare_dependency(include_directories: includes,
+                dependencies: static_deps)
+        set_variable('shared_rte_' + name, shared_dep)
+        set_variable('static_rte_' + name, static_dep)
+        dpdk_shared_lib_deps += shared_dep
+        dpdk_static_lib_deps += static_dep
+        if developer_mode
+            message('lib/@0@: Defining dependency "@1@"'.format(l, name))
+        endif
+        continue
+    endif
+
     if developer_mode and is_windows and use_function_versioning
         message('@0@: Function versioning is not supported by Windows.'.format(name))
     endif
--
2.25.1

[PATCH v11 2/6] mempool: add functions to get extra mempool info

[Paul Szczepanek]

Add two functions:
- rte_mempool_get_mem_range - get virtual memory range
of the objects in the mempool,
- rte_mempool_get_obj_alignment - get alignment of
objects in the mempool.

Add two tests that test these new functions.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@arm.com>
Reviewed-by: Yoan Picchi <yoan.picchi@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
---
 app/test/test_mempool.c   | 61 +++++++++++++++++++++++++++++++++++++++
 lib/mempool/rte_mempool.c | 39 +++++++++++++++++++++++++
 lib/mempool/rte_mempool.h | 37 ++++++++++++++++++++++++
 lib/mempool/version.map   |  3 ++
 4 files changed, 140 insertions(+)

diff --git a/app/test/test_mempool.c b/app/test/test_mempool.c
index ad7ebd6363..16eeeb899c 100644
--- a/app/test/test_mempool.c
+++ b/app/test/test_mempool.c
@@ -843,12 +843,16 @@ test_mempool(void)
 	int ret = -1;
 	uint32_t nb_objs = 0;
 	uint32_t nb_mem_chunks = 0;
+	void *start = NULL;
+	size_t length = 0;
+	size_t alignment = 0;
 	struct rte_mempool *mp_cache = NULL;
 	struct rte_mempool *mp_nocache = NULL;
 	struct rte_mempool *mp_stack_anon = NULL;
 	struct rte_mempool *mp_stack_mempool_iter = NULL;
 	struct rte_mempool *mp_stack = NULL;
 	struct rte_mempool *default_pool = NULL;
+	struct rte_mempool *mp_alignment = NULL;
 	struct mp_data cb_arg = {
 		.ret = -1
 	};
@@ -967,6 +971,62 @@ test_mempool(void)
 	}
 	rte_mempool_obj_iter(default_pool, my_obj_init, NULL);

+	if (rte_mempool_get_mem_range(default_pool, &start, &length)) {
+		printf("cannot get mem range from default mempool\n");
+		GOTO_ERR(ret, err);
+	}
+
+	if (rte_mempool_get_mem_range(NULL, NULL, NULL) != -EINVAL) {
+		printf("rte_mempool_get_mem_range failed to return -EINVAL "
+				"when passed invalid arguments\n");
+		GOTO_ERR(ret, err);
+	}
+
+	if (start == NULL || length < (MEMPOOL_SIZE * MEMPOOL_ELT_SIZE)) {
+		printf("mem range of default mempool is invalid\n");
+		GOTO_ERR(ret, err);
+	}
+
+	/* by default mempool objects are aligned by RTE_MEMPOOL_ALIGN */
+	alignment = rte_mempool_get_obj_alignment(default_pool);
+	if (alignment != RTE_MEMPOOL_ALIGN) {
+		printf("rte_mempool_get_obj_alignment returned wrong value, "
+				"expected %zu, returned %zu\n",
+				(size_t)RTE_MEMPOOL_ALIGN, alignment);
+		GOTO_ERR(ret, err);
+	}
+
+	/* create a mempool with a RTE_MEMPOOL_F_NO_CACHE_ALIGN flag */
+	mp_alignment = rte_mempool_create("test_alignment", MEMPOOL_SIZE,
+		MEMPOOL_ELT_SIZE, 0, 0,
+		NULL, NULL,
+		my_obj_init, NULL,
+		SOCKET_ID_ANY, RTE_MEMPOOL_F_NO_CACHE_ALIGN);
+
+	if (mp_alignment == NULL) {
+		printf("cannot allocate mempool with "
+				"RTE_MEMPOOL_F_NO_CACHE_ALIGN flag\n");
+		GOTO_ERR(ret, err);
+	}
+
+	/* mempool was created with RTE_MEMPOOL_F_NO_CACHE_ALIGN
+	 * and minimum alignment is expected which is sizeof(uint64_t)
+	 */
+	alignment = rte_mempool_get_obj_alignment(mp_alignment);
+	if (alignment != sizeof(uint64_t)) {
+		printf("rte_mempool_get_obj_alignment returned wrong value, "
+				"expected %zu, returned %zu\n",
+				(size_t)sizeof(uint64_t), alignment);
+		GOTO_ERR(ret, err);
+	}
+
+	alignment = rte_mempool_get_obj_alignment(NULL);
+	if (alignment != 0) {
+		printf("rte_mempool_get_obj_alignment failed to return 0 for "
+				" an invalid mempool\n");
+		GOTO_ERR(ret, err);
+	}
+
 	/* retrieve the mempool from its name */
 	if (rte_mempool_lookup("test_nocache") != mp_nocache) {
 		printf("Cannot lookup mempool from its name\n");
@@ -1039,6 +1099,7 @@ test_mempool(void)
 	rte_mempool_free(mp_stack_mempool_iter);
 	rte_mempool_free(mp_stack);
 	rte_mempool_free(default_pool);
+	rte_mempool_free(mp_alignment);

 	return ret;
 }
diff --git a/lib/mempool/rte_mempool.c b/lib/mempool/rte_mempool.c
index 12390a2c81..7a4bafb664 100644
--- a/lib/mempool/rte_mempool.c
+++ b/lib/mempool/rte_mempool.c
@@ -1386,6 +1386,45 @@ void rte_mempool_walk(void (*func)(struct rte_mempool *, void *),
 	rte_mcfg_mempool_read_unlock();
 }

+int rte_mempool_get_mem_range(struct rte_mempool *mp,
+		void **mem_range_start, size_t *mem_range_length)
+{
+	if (mp == NULL || mem_range_start == NULL || mem_range_length == NULL)
+		return -EINVAL;
+
+	void *address_low = (void *)UINTPTR_MAX;
+	void *address_high = 0;
+	struct rte_mempool_memhdr *hdr;
+
+	/* go through memory chunks and find the lowest and highest addresses */
+	STAILQ_FOREACH(hdr, &mp->mem_list, next) {
+		if (address_low > hdr->addr)
+			address_low = hdr->addr;
+		if (address_high < RTE_PTR_ADD(hdr->addr, hdr->len))
+			address_high = RTE_PTR_ADD(hdr->addr, hdr->len);
+	}
+
+	/* check if mempool was not populated yet (no memory chunks) */
+	if (address_low == (void *)UINTPTR_MAX)
+		return -EINVAL;
+
+	*mem_range_start = address_low;
+	*mem_range_length = (size_t)RTE_PTR_DIFF(address_high, address_low);
+
+	return 0;
+}
+
+size_t rte_mempool_get_obj_alignment(struct rte_mempool *mp)
+{
+	if (mp == NULL)
+		return 0;
+
+	if (mp->flags & RTE_MEMPOOL_F_NO_CACHE_ALIGN)
+		return sizeof(uint64_t);
+	else
+		return RTE_MEMPOOL_ALIGN;
+}
+
 struct mempool_callback_data {
 	TAILQ_ENTRY(mempool_callback_data) callbacks;
 	rte_mempool_event_callback *func;
diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h
index 23fd5c8465..8a96f97e72 100644
--- a/lib/mempool/rte_mempool.h
+++ b/lib/mempool/rte_mempool.h
@@ -1917,6 +1917,43 @@ uint32_t rte_mempool_calc_obj_size(uint32_t elt_size, uint32_t flags,
 void rte_mempool_walk(void (*func)(struct rte_mempool *, void *arg),
 		      void *arg);

+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Get information about the memory range used by the mempool.
+ *
+ * @param[in] mp
+ *   Pointer to an initialized mempool.
+ * @param[out] mem_range_start
+ *   Returns lowest address in mempool.
+ * @param[out] mem_range_length
+ *   Returns the length of the memory range containing all the addresses
+ *   in the memory pool.
+ * @return
+ *   0 on success, -EINVAL if arguments are not valid.
+ *
+ **/
+__rte_experimental
+int rte_mempool_get_mem_range(struct rte_mempool *mp,
+		void **mem_range_start, size_t *mem_range_length);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Return object alignment.
+ *
+ * @param[in] mp
+ *   Pointer to a mempool.
+ *
+ * @return
+ *   Object alignment if mp is valid. 0 if mp is NULL.
+ *
+ **/
+__rte_experimental
+size_t rte_mempool_get_obj_alignment(struct rte_mempool *mp);
+
 /**
  * @internal Get page size used for mempool object allocation.
  * This function is internal to mempool library and mempool drivers.
diff --git a/lib/mempool/version.map b/lib/mempool/version.map
index 473277400c..02df634b2a 100644
--- a/lib/mempool/version.map
+++ b/lib/mempool/version.map
@@ -50,6 +50,9 @@ EXPERIMENTAL {
 	__rte_mempool_trace_get_contig_blocks;
 	__rte_mempool_trace_default_cache;
 	__rte_mempool_trace_cache_flush;
+	# added in 24.07
+	rte_mempool_get_mem_range;
+	rte_mempool_get_obj_alignment;
 };

 INTERNAL {
--
2.25.1

RE: [PATCH v11 2/6] mempool: add functions to get extra mempool info

[Morten Brørup]

> From: Paul Szczepanek [mailto:paul.szczepanek@arm.com]
> Sent: Friday, 24 May 2024 10.37
> 
> +size_t rte_mempool_get_obj_alignment(struct rte_mempool *mp)
> +{
> +	if (mp == NULL)
> +		return 0;
> +
> +	if (mp->flags & RTE_MEMPOOL_F_NO_CACHE_ALIGN)
> +		return sizeof(uint64_t);
> +	else
> +		return RTE_MEMPOOL_ALIGN;
> +}

The object alignment depends on the underlying mempool driver. You cannot assume that it is either sizeof(uint64_t) or cache line aligned.

Refer to the calc_mem_size driver operation, which also provides object alignment information:
https://elixir.bootlin.com/dpdk/v24.03/source/lib/mempool/rte_mempool.h#L529

If you need this function, you need to add a new driver operation, and your function above can be the default for this operation, like for the the calc_mem_size driver operation:
https://elixir.bootlin.com/dpdk/v24.03/source/lib/mempool/rte_mempool_ops.c#L120

Re: [PATCH v11 2/6] mempool: add functions to get extra mempool info

[Paul Szczepanek]

On 24/05/2024 13:20, Morten Brørup wrote:
>> From: Paul Szczepanek [mailto:paul.szczepanek@arm.com]
>> Sent: Friday, 24 May 2024 10.37
>>
>> +size_t rte_mempool_get_obj_alignment(struct rte_mempool *mp)
>> +{
>> +	if (mp == NULL)
>> +		return 0;
>> +
>> +	if (mp->flags & RTE_MEMPOOL_F_NO_CACHE_ALIGN)
>> +		return sizeof(uint64_t);
>> +	else
>> +		return RTE_MEMPOOL_ALIGN;
>> +}
> 
> The object alignment depends on the underlying mempool driver. You cannot assume that it is either sizeof(uint64_t) or cache line aligned.
> 
> Refer to the calc_mem_size driver operation, which also provides object alignment information:
> https://elixir.bootlin.com/dpdk/v24.03/source/lib/mempool/rte_mempool.h#L529
> 
> If you need this function, you need to add a new driver operation, and your function above can be the default for this operation, like for the the calc_mem_size driver operation:
> https://elixir.bootlin.com/dpdk/v24.03/source/lib/mempool/rte_mempool_ops.c#L120
> 

As discussed on slack the alignment you mention is the memzone alignment
which is distinct from the object alignment which is enforced by the
mempool according to the RTE_MEMPOOL_F_NO_CACHE_ALIGN flag. Objects may
have higher alignment, the alignment returned by the new function is the
minimum guaranteed one.

I addressed your other comments in v12 (pending internal review).

[PATCH v11 3/6] ptr_compress: add pointer compression library

[Paul Szczepanek]

Add a new utility header for compressing pointers. The provided
functions can store pointers as 32-bit or 16-bit offsets.

The compression takes advantage of the fact that pointers are
usually located in a limited memory region (like a mempool).
We can compress them by converting them to offsets from a base
memory address. Offsets can be stored in fewer bytes (dictated
by the memory region size and alignment of the pointer).
For example: an 8 byte aligned pointer which is part of a 32GB
memory pool can be stored in 4 bytes.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@arm.com>
---
 MAINTAINERS                            |   4 +
 doc/api/doxy-api-index.md              |   1 +
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/rel_notes/release_24_07.rst |   5 +
 lib/meson.build                        |   1 +
 lib/ptr_compress/meson.build           |   4 +
 lib/ptr_compress/rte_ptr_compress.h    | 278 +++++++++++++++++++++++++
 7 files changed, 294 insertions(+)
 create mode 100644 lib/ptr_compress/meson.build
 create mode 100644 lib/ptr_compress/rte_ptr_compress.h

diff --git a/MAINTAINERS b/MAINTAINERS
index c9adff9846..27b2f03e6c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1694,6 +1694,10 @@ M: Chenbo Xia <chenbox@nvidia.com>
 M: Gaetan Rivet <grive@u256.net>
 F: lib/pci/

+Pointer Compression
+M: Paul Szczepanek <paul.szczepanek@arm.com>
+F: lib/ptr_compress/
+
 Power management
 M: Anatoly Burakov <anatoly.burakov@intel.com>
 M: David Hunt <david.hunt@intel.com>
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 8c1eb8fafa..f9283154f8 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -222,6 +222,7 @@ The public API headers are grouped by topics:
   [config file](@ref rte_cfgfile.h),
   [key/value args](@ref rte_kvargs.h),
   [argument parsing](@ref rte_argparse.h),
+  [ptr_compress](@ref rte_ptr_compress.h),
   [string](@ref rte_string_fns.h),
   [thread](@ref rte_thread.h)

diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index 27afec8b3b..a8823c046f 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -71,6 +71,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/pipeline \
                           @TOPDIR@/lib/port \
                           @TOPDIR@/lib/power \
+                          @TOPDIR@/lib/ptr_compress \
                           @TOPDIR@/lib/rawdev \
                           @TOPDIR@/lib/rcu \
                           @TOPDIR@/lib/regexdev \
diff --git a/doc/guides/rel_notes/release_24_07.rst b/doc/guides/rel_notes/release_24_07.rst
index a69f24cf99..4711792e61 100644
--- a/doc/guides/rel_notes/release_24_07.rst
+++ b/doc/guides/rel_notes/release_24_07.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================

+* **Introduced pointer compression library.**
+
+  Library provides functions to compress and decompress arrays of pointers
+  which can improve application performance under certain conditions.
+  Performance test was added to help users evaluate performance on their setup.

 Removed Items
 -------------
diff --git a/lib/meson.build b/lib/meson.build
index 7c90602bf5..63becee142 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -14,6 +14,7 @@ libraries = [
         'argparse',
         'telemetry', # basic info querying
         'eal', # everything depends on eal
+        'ptr_compress',
         'ring',
         'rcu', # rcu depends on ring
         'mempool',
diff --git a/lib/ptr_compress/meson.build b/lib/ptr_compress/meson.build
new file mode 100644
index 0000000000..e92706a45f
--- /dev/null
+++ b/lib/ptr_compress/meson.build
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+headers = files('rte_ptr_compress.h')
diff --git a/lib/ptr_compress/rte_ptr_compress.h b/lib/ptr_compress/rte_ptr_compress.h
new file mode 100644
index 0000000000..f697f66075
--- /dev/null
+++ b/lib/ptr_compress/rte_ptr_compress.h
@@ -0,0 +1,278 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2024 Arm Limited
+ */
+
+#ifndef RTE_PTR_COMPRESS_H
+#define RTE_PTR_COMPRESS_H
+
+/**
+ * @file
+ * Pointer compression and decompression functions.
+ *
+ * When passing arrays full of pointers between threads, memory containing
+ * the pointers is copied multiple times which is especially costly between
+ * cores. These functions allow us to compress the pointers.
+ *
+ * Compression takes advantage of the fact that pointers are usually located in
+ * a limited memory region. We compress them by converting them to offsets from
+ * a base memory address. Offsets can be stored in fewer bytes.
+ *
+ * The compression functions come in two varieties: 32-bit and 16-bit.
+ *
+ * To determine how many bits are needed to compress the pointer calculate
+ * the biggest offset possible (highest value pointer - base pointer)
+ * and shift the value right according to alignment (shift by exponent of the
+ * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by
+ * 3, etc.). The resulting value must fit in either 32 or 16 bits.
+ *
+ * For usage example and further explanation please see "Pointer Compression" in
+ * doc/guides/prog_guide/ptr_compress_lib.rst
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_vect.h>
+#include <rte_mempool.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#define BITS_REQUIRED_TO_STORE_VALUE(x) \
+	((x) == 0 ? 1 : (sizeof(size_t) * CHAR_BIT - __builtin_clzl((size_t)x)))
+
+#define BIT_SHIFT_FROM_ALIGNMENT(x) ((x) == 0 ? 0 : __builtin_ctzl(x))
+
+#define CAN_USE_RTE_PTR_COMPRESS_16_SHIFT(mem_range, obj_alignment) \
+	((BITS_REQUIRED_TO_STORE_VALUE(mem_range) - \
+	BIT_SHIFT_FROM_ALIGNMENT(obj_alignment)) <= 16 ? 1 : 0)
+
+#define CAN_USE_RTE_PTR_COMPRESS_32_SHIFT(mem_range, obj_alignment) \
+	((BITS_REQUIRED_TO_STORE_VALUE(mem_range) - \
+	BIT_SHIFT_FROM_ALIGNMENT(obj_alignment)) <= 32 ? 1 : 0)
+
+/**
+ * Compress pointers into 32-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 32 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 32GB of the base pointer. If there is no such alignment guarantee they
+ * must be within 4GB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_32_shift(void *ptr_base, void **src_table,
+		uint32_t *dest_table, size_t n, uint8_t bit_shift)
+{
+	size_t i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	do {
+		svbool_t pg = svwhilelt_b64(i, n);
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1w(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+	} while (i < n);
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	/* right shift is done by left shifting by negative int */
+	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	const size_t n_even = n & ~0x1;
+	for (; i < n_even; i += 2) {
+		v_ptr_table = vld1q_u64((const uint64_t *)src_table + i);
+		v_ptr_table = vsubq_u64(v_ptr_table, v_ptr_base);
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		vst1_u32(dest_table + i, vqmovn_u64(v_ptr_table));
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT32_MAX);
+		dest_table[i] = (uint32_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 32-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_32_shift(void *ptr_base, uint32_t *src_table,
+		void **dest_table, size_t n, uint8_t bit_shift)
+{
+	size_t i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	do {
+		svbool_t pg = svwhilelt_b64(i, n);
+		v_ptr_table = svld1uw_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+	} while (i < n);
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	int64x2_t v_shift = vdupq_n_s64(bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	const size_t n_even = n & ~0x1;
+	for (; i < n_even; i += 2) {
+		v_ptr_table = vmovl_u32(vld1_u32(src_table + i));
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		v_ptr_table = vaddq_u64(v_ptr_table, v_ptr_base);
+		vst1q_u64((uint64_t *)dest_table + i, v_ptr_table);
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+/**
+ * Compress pointers into 16-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 16 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 256KB of the base pointer. If there is no such alignment guarantee
+ * they must be within 64KB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_16_shift(void *ptr_base, void **src_table,
+		uint16_t *dest_table, size_t n, uint8_t bit_shift)
+{
+
+	size_t i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	do {
+		svbool_t pg = svwhilelt_b64(i, n);
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1h(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+	} while (i < n);
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT16_MAX);
+		dest_table[i] = (uint16_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 16-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_16_shift(void *ptr_base, uint16_t *src_table,
+		void **dest_table, size_t n, uint8_t bit_shift)
+{
+	size_t i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	do {
+		svbool_t pg = svwhilelt_b64(i, n);
+		v_ptr_table = svld1uh_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+	} while (i < n);
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_PTR_COMPRESS_H */
--
2.25.1

RE: [PATCH v11 3/6] ptr_compress: add pointer compression library

[Morten Brørup]

> +#define BITS_REQUIRED_TO_STORE_VALUE(x) \
> +	((x) == 0 ? 1 : (sizeof(size_t) * CHAR_BIT - __builtin_clzl((size_t)x)))
> +
> +#define BIT_SHIFT_FROM_ALIGNMENT(x) ((x) == 0 ? 0 : __builtin_ctzl(x))

The two built-ins used are not available with MSVC.
Please see if you can find similar functions in /lib/eal/include/rte_bitops.h instead of defining these macros.

> +
> +#define CAN_USE_RTE_PTR_COMPRESS_16_SHIFT(mem_range, obj_alignment) \
> +	((BITS_REQUIRED_TO_STORE_VALUE(mem_range) - \
> +	BIT_SHIFT_FROM_ALIGNMENT(obj_alignment)) <= 16 ? 1 : 0)
> +
> +#define CAN_USE_RTE_PTR_COMPRESS_32_SHIFT(mem_range, obj_alignment) \
> +	((BITS_REQUIRED_TO_STORE_VALUE(mem_range) - \
> +	BIT_SHIFT_FROM_ALIGNMENT(obj_alignment)) <= 32 ? 1 : 0)
> +

These macros are in a public header file, and thus public.

Their names should begin with RTE_PTR_COMPRESS.
And they should have Doxygen descriptions.

Please also use them in the tests or some other code, for the CI to verify that MSVC can be build them.

RE: [PATCH v11 3/6] ptr_compress: add pointer compression library

[Konstantin Ananyev]

> +/**
> + * Compress pointers into 32-bit offsets from base pointer.
> + *
> + * @note It is programmer's responsibility to ensure the resulting offsets fit
> + * into 32 bits. Alignment of the structures pointed to by the pointers allows
> + * us to drop bits from the offsets. This is controlled by the bit_shift
> + * parameter. This means that if structures are aligned by 8 bytes they must be
> + * within 32GB of the base pointer. If there is no such alignment guarantee they
> + * must be within 4GB.
> + *
> + * @param ptr_base
> + *   A pointer used to calculate offsets of pointers in src_table.
> + * @param src_table
> + *   A pointer to an array of pointers.
> + * @param dest_table
> + *   A pointer to an array of compressed pointers returned by this function.
> + * @param n
> + *   The number of objects to compress, must be strictly positive.
> + * @param bit_shift
> + *   Byte alignment of memory pointed to by the pointers allows for
> + *   bits to be dropped from the offset and hence widen the memory region that
> + *   can be covered. This controls how many bits are right shifted.
> + **/
> +static __rte_always_inline void
> +rte_ptr_compress_32_shift(void *ptr_base, void **src_table,
> +		uint32_t *dest_table, size_t n, uint8_t bit_shift)

Probably: void * const *src_table
And on decompress: const uint32_t *src_table 

> +{
> +	size_t i = 0;
> +#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
> +	svuint64_t v_ptr_table;
> +	do {
> +		svbool_t pg = svwhilelt_b64(i, n);
> +		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
> +		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
> +		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
> +		svst1w(pg, &dest_table[i], v_ptr_table);
> +		i += svcntd();
> +	} while (i < n);
> +#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
> +	uint64_t ptr_diff;
> +	uint64x2_t v_ptr_table;
> +	/* right shift is done by left shifting by negative int */
> +	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
> +	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
> +	const size_t n_even = n & ~0x1;
> +	for (; i < n_even; i += 2) {
> +		v_ptr_table = vld1q_u64((const uint64_t *)src_table + i);
> +		v_ptr_table = vsubq_u64(v_ptr_table, v_ptr_base);
> +		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
> +		vst1_u32(dest_table + i, vqmovn_u64(v_ptr_table));
> +	}
> +	/* process leftover single item in case of odd number of n */
> +	if (unlikely(n & 0x1)) {
> +		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
> +		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
> +	}
> +#else
> +	uintptr_t ptr_diff;
> +	for (; i < n; i++) {
> +		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
> +		ptr_diff = ptr_diff >> bit_shift;
> +		RTE_ASSERT(ptr_diff <= UINT32_MAX);
> +		dest_table[i] = (uint32_t) ptr_diff;
> +	}
> +#endif
> +}
> +

[PATCH v11 4/6] test: add pointer compress tests to ring perf test

[Paul Szczepanek]

Add a test that runs a zero copy burst enqueue and dequeue on a ring
of raw pointers and compressed pointers at different burst sizes to
showcase performance benefits of newly added pointer compression APIs.

Refactored threading code to pass more parameters to threads to
reuse existing code. Added more bulk sizes to showcase their effects
on compression. Adjusted loop iteration numbers to take into account
bulk sizes to keep runtime constant (instead of number of operations).

Adjusted old printfs to match new ones which have aligned numbers.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@arm.com>
---
 app/test/meson.build      |  20 +--
 app/test/test_ring.h      |  94 ++++++++++
 app/test/test_ring_perf.c | 352 +++++++++++++++++++++++++-------------
 3 files changed, 334 insertions(+), 132 deletions(-)

diff --git a/app/test/meson.build b/app/test/meson.build
index 7d909039ae..df8cc00730 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -156,16 +156,16 @@ source_file_deps = {
 #    'test_resource.c': [],
     'test_rib.c': ['net', 'rib'],
     'test_rib6.c': ['net', 'rib'],
-    'test_ring.c': [],
-    'test_ring_hts_stress.c': [],
-    'test_ring_mpmc_stress.c': [],
-    'test_ring_mt_peek_stress.c': [],
-    'test_ring_mt_peek_stress_zc.c': [],
-    'test_ring_perf.c': [],
-    'test_ring_rts_stress.c': [],
-    'test_ring_st_peek_stress.c': [],
-    'test_ring_st_peek_stress_zc.c': [],
-    'test_ring_stress.c': [],
+    'test_ring.c': ['ptr_compress'],
+    'test_ring_hts_stress.c': ['ptr_compress'],
+    'test_ring_mpmc_stress.c': ['ptr_compress'],
+    'test_ring_mt_peek_stress.c': ['ptr_compress'],
+    'test_ring_mt_peek_stress_zc.c': ['ptr_compress'],
+    'test_ring_perf.c': ['ptr_compress'],
+    'test_ring_rts_stress.c': ['ptr_compress'],
+    'test_ring_st_peek_stress.c': ['ptr_compress'],
+    'test_ring_st_peek_stress_zc.c': ['ptr_compress'],
+    'test_ring_stress.c': ['ptr_compress'],
     'test_rwlock.c': [],
     'test_sched.c': ['net', 'sched'],
     'test_security.c': ['net', 'security'],
diff --git a/app/test/test_ring.h b/app/test/test_ring.h
index 45c263f3ff..9e97c5e3e7 100644
--- a/app/test/test_ring.h
+++ b/app/test/test_ring.h
@@ -5,6 +5,8 @@
 #include <rte_malloc.h>
 #include <rte_ring.h>
 #include <rte_ring_elem.h>
+#include <rte_memcpy.h>
+#include <rte_ptr_compress.h>

 /* API type to call
  * rte_ring_<sp/mp or sc/mc>_enqueue_<bulk/burst>
@@ -25,6 +27,10 @@
 #define TEST_RING_ELEM_BULK 16
 #define TEST_RING_ELEM_BURST 32

+#define TEST_RING_ELEM_BURST_ZC 64
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 128
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 256
+
 #define TEST_RING_IGNORE_API_TYPE ~0U

 /* This function is placed here as it is required for both
@@ -101,6 +107,9 @@ static inline unsigned int
 test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -152,6 +161,47 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n,
 								NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy(zcd.ptr2,
+						(char *)obj + zcd.n1 * esize,
+						(ret - zcd.n1) * esize);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_16_shift(
+					0, obj, zcd.ptr1, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_16_shift(0,
+						obj + (zcd.n1 * 2),
+						zcd.ptr2,
+						(ret - zcd.n1) * 2, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_32_shift(0, obj, zcd.ptr1, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_32_shift(0, obj + zcd.n1,
+						zcd.ptr2, ret - zcd.n1, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
@@ -162,6 +212,9 @@ static inline unsigned int
 test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -213,6 +266,47 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mc_dequeue_burst_elem(r, obj, esize,
 								n, NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy((char *)obj + zcd.n1 * esize,
+						zcd.ptr2,
+						(ret - zcd.n1) * esize);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_16_shift(
+					0, zcd.ptr1, obj, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_16_shift(0, zcd.ptr2,
+						obj + zcd.n1,
+						(ret - zcd.n1) * 2,
+						3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_32_shift(0, zcd.ptr1, obj, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_32_shift(0, zcd.ptr2,
+						obj + zcd.n1, ret - zcd.n1, 3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c
index ee328614c1..57cd04a124 100644
--- a/app/test/test_ring_perf.c
+++ b/app/test/test_ring_perf.c
@@ -22,13 +22,13 @@

 #define RING_NAME "RING_PERF"
 #define RING_SIZE 4096
-#define MAX_BURST 32
+#define MAX_BURST 256

 /*
  * the sizes to enqueue and dequeue in testing
  * (marked volatile so they won't be seen as compile-time constants)
  */
-static const volatile unsigned bulk_sizes[] = { 8, 32 };
+static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128, 256 };

 struct lcore_pair {
 	unsigned c1, c2;
@@ -43,26 +43,30 @@ test_ring_print_test_string(unsigned int api_type, int esize,
 	if (esize == -1)
 		printf("legacy APIs");
 	else
-		printf("elem APIs: element size %dB", esize);
+		printf("elem APIs (size:%2dB)", esize);

 	if (api_type == TEST_RING_IGNORE_API_TYPE)
 		return;

 	if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF)
-		printf(": default enqueue/dequeue: ");
+		printf(" - default enqueue/dequeue");
 	else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC)
-		printf(": SP/SC: ");
+		printf(" - SP/SC");
 	else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC)
-		printf(": MP/MC: ");
+		printf(" - MP/MC");

 	if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE)
-		printf("single: ");
+		printf(" - single - ");
 	else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK)
-		printf("bulk (size: %u): ", bsz);
+		printf(" - bulk (n:%-3u) - ", bsz);
 	else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST)
-		printf("burst (size: %u): ", bsz);
+		printf(" - burst (n:%-3u) - ", bsz);
+	else if ((api_type & (TEST_RING_ELEM_BURST_ZC |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0)
+		printf(" - burst zero copy (n:%-3u) - ", bsz);

-	printf("%.2F\n", value);
+	printf("cycles per elem: %.3F\n", value);
 }

 /**** Functions to analyse our core mask to get cores for different tests ***/
@@ -144,28 +148,33 @@ static void
 test_empty_dequeue(struct rte_ring *r, const int esize,
 			const unsigned int api_type)
 {
-	const unsigned int iter_shift = 26;
+	const unsigned int iter_shift = 29;
 	const unsigned int iterations = 1 << iter_shift;
 	unsigned int i = 0;
 	void *burst[MAX_BURST];

+	const unsigned int bulk_iterations = iterations / bulk_sizes[0];
 	const uint64_t start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	for (i = 0; i < bulk_iterations; i++)
 		test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type);
 	const uint64_t end = rte_rdtsc();

 	test_ring_print_test_string(api_type, esize, bulk_sizes[0],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 }

-/*
- * for the separate enqueue and dequeue threads they take in one param
- * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc
- */
-struct thread_params {
+/* describes the ring used by the enqueue and dequeue thread */
+struct ring_params {
 	struct rte_ring *r;
-	unsigned size;        /* input value, the burst size */
-	double spsc, mpmc;    /* output value, the single or multi timings */
+	unsigned int elem_size;
+	unsigned int bulk_sizes_i; /* index into bulk_size array */
+	unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */
+};
+
+/* Used to specify enqueue and dequeue ring operations and their results */
+struct thread_params {
+	struct ring_params *ring_params;
+	double *results; /* result array size must be equal to bulk_sizes */
 };

 /*
@@ -174,16 +183,15 @@ struct thread_params {
  * flag == 1 -> dequeue
  */
 static __rte_always_inline int
-enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
-	struct thread_params *p)
+enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 15;
+	const unsigned int iter_shift = 22;
 	const unsigned int iterations = 1 << iter_shift;
-	struct rte_ring *r = p->r;
-	unsigned int bsize = p->size;
 	unsigned int i;
 	void *burst = NULL;
+	unsigned int n_remaining;
+	const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i];

 #ifdef RTE_USE_C11_MEM_MODEL
 	if (rte_atomic_fetch_add_explicit(&lcore_count, 1, rte_memory_order_relaxed) + 1 != 2)
@@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 		while(lcore_count != 2)
 			rte_pause();

-	burst = test_ring_calloc(MAX_BURST, esize);
+	burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size);
 	if (burst == NULL)
 		return -1;

 	const uint64_t sp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	const unsigned int bulk_iterations = iterations / bulk_n;
+	for (i = 0; i < bulk_iterations; i++) {
+		n_remaining = bulk_n;
 		do {
 			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_enqueue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_dequeue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			if (ret == 0)
 				rte_pause();
-		} while (!ret);
+			else
+				n_remaining -= ret;
+		} while (n_remaining > 0);
+	}
 	const uint64_t sp_end = rte_rdtsc();

-	const uint64_t mp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
-		do {
-			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			if (ret == 0)
-				rte_pause();
-		} while (!ret);
-	const uint64_t mp_end = rte_rdtsc();
+	p->results[p->ring_params->bulk_sizes_i] =
+			((double)sp_end - sp_start) / iterations;

-	p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize);
-	p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize);
 	return 0;
 }

@@ -243,15 +245,7 @@ enqueue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(0, -1, params);
-}
-
-static int
-enqueue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(0, 16, params);
+	return enqueue_dequeue_bulk_helper(0, params);
 }

 /*
@@ -263,15 +257,7 @@ dequeue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(1, -1, params);
-}
-
-static int
-dequeue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(1, 16, params);
+	return enqueue_dequeue_bulk_helper(1, params);
 }

 /*
@@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p)
  * used to measure ring perf between hyperthreads, cores and sockets.
  */
 static int
-run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize)
+run_on_core_pair(struct lcore_pair *cores,
+		struct thread_params *param1, struct thread_params *param2)
 {
-	lcore_function_t *f1, *f2;
-	struct thread_params param1 = {0}, param2 = {0};
 	unsigned i;
-
-	if (esize == -1) {
-		f1 = enqueue_bulk;
-		f2 = dequeue_bulk;
-	} else {
-		f1 = enqueue_bulk_16B;
-		f2 = dequeue_bulk_16B;
-	}
+	struct ring_params *ring_params = param1->ring_params;

 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		lcore_count = 0;
-		param1.size = param2.size = bulk_sizes[i];
-		param1.r = param2.r = r;
+		ring_params->bulk_sizes_i = i;
 		if (cores->c1 == rte_get_main_lcore()) {
-			rte_eal_remote_launch(f2, &param2, cores->c2);
-			f1(&param1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
+			enqueue_bulk(param1);
 			rte_eal_wait_lcore(cores->c2);
 		} else {
-			rte_eal_remote_launch(f1, &param1, cores->c1);
-			rte_eal_remote_launch(f2, &param2, cores->c2);
+			rte_eal_remote_launch(enqueue_bulk, param1, cores->c1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
 			if (rte_eal_wait_lcore(cores->c1) < 0)
 				return -1;
 			if (rte_eal_wait_lcore(cores->c2) < 0)
 				return -1;
 		}
 		test_ring_print_test_string(
-			TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.spsc + param2.spsc);
-		test_ring_print_test_string(
-			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.mpmc + param2.mpmc);
+				ring_params->ring_flags,
+				ring_params->elem_size,
+				bulk_sizes[i],
+				param1->results[i] + param2->results[i]);
 	}

 	return 0;
@@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 	uint64_t hz = rte_get_timer_hz();
 	uint64_t lcount = 0;
 	const unsigned int lcore = rte_lcore_id();
-	struct thread_params *params = p;
+	struct ring_params *ring_params = p->ring_params;
 	void *burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -347,9 +323,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize)

 	begin = rte_get_timer_cycles();
 	while (time_diff < hz * TIME_MS / 1000) {
-		test_ring_enqueue(params->r, burst, esize, params->size,
+		test_ring_enqueue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
-		test_ring_dequeue(params->r, burst, esize, params->size,
+		test_ring_dequeue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 		lcount++;
 		time_diff = rte_get_timer_cycles() - begin;
@@ -381,7 +359,8 @@ static int
 run_on_all_cores(struct rte_ring *r, const int esize)
 {
 	uint64_t total;
-	struct thread_params param;
+	struct ring_params ring_params = {0};
+	struct thread_params params = { .ring_params = &ring_params };
 	lcore_function_t *lcore_f;
 	unsigned int i, c;

@@ -390,21 +369,20 @@ run_on_all_cores(struct rte_ring *r, const int esize)
 	else
 		lcore_f = load_loop_fn_16B;

-	memset(&param, 0, sizeof(struct thread_params));
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		total = 0;
 		printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]);
-		param.size = bulk_sizes[i];
-		param.r = r;
+		params.ring_params->bulk_sizes_i = i;
+		params.ring_params->r = r;

 		/* clear synchro and start workers */
 		rte_atomic_store_explicit(&synchro, 0, rte_memory_order_relaxed);
-		if (rte_eal_mp_remote_launch(lcore_f, &param, SKIP_MAIN) < 0)
+		if (rte_eal_mp_remote_launch(lcore_f, &params, SKIP_MAIN) < 0)
 			return -1;

 		/* start synchro and launch test on main */
 		rte_atomic_store_explicit(&synchro, 1, rte_memory_order_relaxed);
-		lcore_f(&param);
+		lcore_f(&params);

 		rte_eal_mp_wait_lcore();

@@ -463,9 +441,9 @@ static int
 test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	const unsigned int api_type)
 {
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 26;
 	const unsigned int iterations = 1 << iter_shift;
-	unsigned int sz, i = 0;
+	unsigned int sz, i;
 	void **burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -473,17 +451,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 		return -1;

 	for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) {
+		const unsigned int n = iterations / bulk_sizes[sz];
 		const uint64_t start = rte_rdtsc();
-		for (i = 0; i < iterations; i++) {
+		for (i = 0; i < n; i++) {
 			test_ring_enqueue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 			test_ring_dequeue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 		}
 		const uint64_t end = rte_rdtsc();

 		test_ring_print_test_string(api_type, esize, bulk_sizes[sz],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 	}

 	rte_free(burst);
@@ -491,12 +470,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	return 0;
 }

+static __rte_always_inline int
+test_ring_perf_esize_run_on_two_cores(
+		struct thread_params *param1, struct thread_params *param2)
+{
+	struct lcore_pair cores;
+
+	if (get_two_hyperthreads(&cores) == 0) {
+		printf("\n### Testing using two hyperthreads ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing using two physical cores ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_sockets(&cores) == 0) {
+		printf("\n### Testing using two NUMA nodes ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	return 0;
+}
+
 /* Run all tests for a given element size */
 static __rte_always_inline int
 test_ring_perf_esize(const int esize)
 {
-	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
+	double results_enq[RTE_DIM(bulk_sizes)];
+	double results_deq[RTE_DIM(bulk_sizes)];
+	struct ring_params ring_params = {
+			.elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results_enq };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results_deq };

 	/*
 	 * Performance test for legacy/_elem APIs
@@ -536,22 +546,13 @@ test_ring_perf_esize(const int esize)
 	test_empty_dequeue(r, esize,
 			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);

-	if (get_two_hyperthreads(&cores) == 0) {
-		printf("\n### Testing using two hyperthreads ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.r = r;

-	if (get_two_cores(&cores) == 0) {
-		printf("\n### Testing using two physical cores ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
-	if (get_two_sockets(&cores) == 0) {
-		printf("\n### Testing using two NUMA nodes ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
+
+	ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);

 	printf("\n### Testing using all worker nodes ###\n");
 	if (run_on_all_cores(r, esize) < 0)
@@ -567,6 +568,109 @@ test_ring_perf_esize(const int esize)
 	return -1;
 }

+
+static __rte_always_inline int
+test_ring_perf_compression(void)
+{
+	double results1[RTE_DIM(bulk_sizes)];
+	double results2[RTE_DIM(bulk_sizes)];
+	double results1_comp[2][RTE_DIM(bulk_sizes)];
+	double results2_comp[2][RTE_DIM(bulk_sizes)];
+
+	struct lcore_pair cores;
+	int ret = -1;
+	unsigned int i, j;
+	struct ring_params ring_params = { .elem_size = sizeof(void *) };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results1 };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results2 };
+
+	printf("\n### Testing compression gain ###");
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(void *),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy ###\n");
+		ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	/* rings allow only multiples of 4 as sizes,
+	 * we allocate size 4 despite only using 2 bytes
+	 * and use half of RING_SIZE as the number of elements
+	 */
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE / 2, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[0];
+	param2.results = results2_comp[0];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (16b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[1];
+	param2.results = results2_comp[1];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (32b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	for (j = 0; j < 2; j++) {
+		printf("\n### Potential gain from compression (%d-bit offsets) "
+		"###\n", (j + 1) * 16);
+		for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
+			const double result = results1[i] + results2[i];
+			const double result_comp = results1_comp[j][i] +
+				results2_comp[j][i];
+			const double gain = 100 - (result_comp / result) * 100;
+
+			printf("Gain of %5.1F%% for burst of %-3u elems\n",
+					gain, bulk_sizes[i]);
+		}
+	}
+
+	return ret;
+}
+
 static int
 test_ring_perf(void)
 {
@@ -577,6 +681,10 @@ test_ring_perf(void)
 	if (test_ring_perf_esize(16) == -1)
 		return -1;

+	/* Test for performance gain of compression */
+	if (test_ring_perf_compression() == -1)
+		return -1;
+
 	return 0;
 }

--
2.25.1

[PATCH v11 5/6] docs: add pointer compression guide

[Paul Szczepanek]

Documentation added in the prog guide for the new
utility functions for pointer compression
showing example code and potential usecases.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@arm.com>
---
 MAINTAINERS                                |   1 +
 doc/guides/prog_guide/index.rst            |   1 +
 doc/guides/prog_guide/ptr_compress_lib.rst | 160 +++++++++++++++++++++
 3 files changed, 162 insertions(+)
 create mode 100644 doc/guides/prog_guide/ptr_compress_lib.rst

diff --git a/MAINTAINERS b/MAINTAINERS
index 27b2f03e6c..ed50121bd2 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1697,6 +1697,7 @@ F: lib/pci/
 Pointer Compression
 M: Paul Szczepanek <paul.szczepanek@arm.com>
 F: lib/ptr_compress/
+F: doc/guides/prog_guide/ptr_compress_lib.rst

 Power management
 M: Anatoly Burakov <anatoly.burakov@intel.com>
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index d09d958e6c..6366849eb0 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -73,6 +73,7 @@ Programmer's Guide
     telemetry_lib
     bpf_lib
     graph_lib
+    ptr_compress_lib
     build-sdk-meson
     meson_ut
     build_app
diff --git a/doc/guides/prog_guide/ptr_compress_lib.rst b/doc/guides/prog_guide/ptr_compress_lib.rst
new file mode 100644
index 0000000000..349da2695e
--- /dev/null
+++ b/doc/guides/prog_guide/ptr_compress_lib.rst
@@ -0,0 +1,160 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2024 Arm Limited.
+
+Pointer Compression Library
+===========================
+
+Use ``rte_ptr_compress_16_shift()`` and ``rte_ptr_decompress_16_shift()`` to
+compress and decompress pointers into 16-bit offsets.
+Use ``rte_ptr_compress_32_shift()`` and ``rte_ptr_decompress_32_shift()`` to
+compress and decompress pointers into 32-bit offsets.
+
+Compression takes advantage of the fact that pointers are usually located in a
+limited memory region (like a mempool). By converting them to offsets from a
+base memory address they can be stored in fewer bytes. How many bytes are needed
+to store the offset is dictated by the memory region size and alignment of
+objects the pointers point to.
+
+For example, a pointer which is part of a 4GB memory pool can be stored as 32
+bit offset. If the pointer points to memory that is 8 bytes aligned then 3 bits
+can be dropped from the offset and a 32GB memory pool can now fit in 32 bits.
+
+For performance reasons these requirements are not enforced programmatically.
+The programmer is responsible for ensuring that the combination of distance
+from the base pointer and memory alignment allow for storing of the offset in
+the number of bits indicated by the function name (16 or 32). Start of mempool
+memory would be a good candidate for the base pointer. Otherwise any pointer
+that precedes all pointers, is close enough and has the same alignment as the
+pointers being compressed will work.
+
+Macros present in the rte_ptr_compress.h header may be used to evaluate whether
+compression is possible:
+
+*   BITS_REQUIRED_TO_STORE_VALUE
+
+*   BIT_SHIFT_FROM_ALIGNMENT
+
+*   CAN_USE_RTE_PTR_compress_16_shift
+
+*   CAN_USE_RTE_PTR_compress_32_shift
+
+These will help you calculate compression parameters and whether these are
+legal for particular compression function.
+
+If using an rte_mempool you can get the parameters you need to use in the
+compression macros and functions by using ``rte_mempool_get_mem_range()``
+and ``rte_mempool_get_obj_alignment()``.
+
+.. note::
+
+    Performance gains depend on the batch size of pointers and CPU capabilities
+    such as vector extensions. It's important to measure the performance
+    increase on target hardware. A test called ``ring_perf_autotest`` in
+    ``dpdk-test`` can provide the measurements.
+
+Example usage
+~~~~~~~~~~~~~
+
+In this example we send pointers between two cores through a ring. While this
+is a realistic use case the code is simplified for demonstration purposes and
+does not have error handling.
+
+.. code-block:: c
+
+    #include <rte_launch.h>
+    #include <rte_ring.h>
+    #include <rte_ring_elem.h>
+    #include <rte_ptr_compress.h>
+
+    #define ITEMS_ARRAY_SIZE (1024)
+    #define BATCH_SIZE (128)
+    #define ALIGN_EXPONENT (3)
+    #define ITEM_ALIGN (1<<ALIGN_EXPONENT)
+    #define CORE_SEND (1)
+    #define CORE_RECV (2)
+
+    struct item {
+      alignas(ITEM_ALIGN) int a;
+    };
+
+    static struct item items[ITEMS_ARRAY_SIZE] = {0};
+    static struct rte_ring *ring = NULL;
+
+    static int
+    send_compressed(void *args)
+    {
+      struct item *ptrs_send[BATCH_SIZE] = {0};
+      unsigned int n_send = 0;
+      struct rte_ring_zc_data zcd = {0};
+
+      /* in this example we only fill the ptrs_send once and reuse */
+      for (;n_send < BATCH_SIZE; n_send++)
+        ptrs_send[n_send] = &items[n_send];
+
+      for(;;) {
+        n_send = rte_ring_enqueue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        /* compress ptrs_send into offsets */
+        rte_ptr_compress_32_shift(items, /* base pointer */
+          ptrs_send, /* source array to be compressed */
+          zcd.ptr1, /* destination array to store offsets */
+          zcd.n1, /* how many pointers to compress */
+          ALIGN_EXPONENT /* how many bits can we drop from the offset */);
+
+        if (zcd.ptr2 != NULL)
+          rte_ptr_compress_32_shift(items, ptrs_send + zcd.n1,
+            zcd.ptr2, n_send - zcd.n1, ALIGN_EXPONENT);
+
+        rte_ring_enqueue_zc_finish(ring, n_send);
+      }
+      return 1;
+    }
+
+    static int
+    recv_compressed(void *args)
+    {
+      struct item *ptrs_recv[BATCH_SIZE] = {0};
+      unsigned int n_recv;
+      struct rte_ring_zc_data zcd = {0};
+
+      for(;;) {
+        /* receive compressed pointers from the ring */
+        n_recv = rte_ring_dequeue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        rte_ptr_decompress_32_shift(items, /* base pointer */
+          zcd.ptr1, /* source array to decompress */
+          ptrs_recv, /* destination array to store pointers */
+          zcd.n1, /* how many pointers to decompress */
+          ALIGN_EXPONENT /* how many bits were dropped from the offset */);
+
+        /* handle the potential secondary buffer (caused by ring boundary) */
+        if (zcd.ptr2 != NULL)
+          rte_ptr_decompress_32_shift(items,
+            zcd.ptr2,
+            ptrs_recv + zcd.n1,
+            n_recv - zcd.n1,
+            ALIGN_EXPONENT);
+
+        rte_ring_dequeue_zc_finish(ring, n_recv);
+
+        /* ptrs_recv contains what ptrs_send contained in the other thread */
+        /* (...) */
+      }
+      return 1;
+    }
+
+    void
+    compression_example(void)
+    {
+      ring = rte_ring_create_elem(
+        "COMPR_PTRS", sizeof(uint32_t),
+        1024, rte_socket_id(),
+        RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+      rte_eal_remote_launch(send_compressed, NULL, CORE_SEND);
+      rte_eal_remote_launch(recv_compressed, NULL, CORE_RECV);
+
+      for(;;) {}
+    }
--
2.25.1

[PATCH v11 6/6] test: add unit test for ptr compression

[Paul Szczepanek]

Test compresses and decompresses pointers with various combinations
of memory regions and alignments and verifies the pointers are
recovered correctly.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@arm.com>
---
 MAINTAINERS                  |   1 +
 app/test/meson.build         |   1 +
 app/test/test_ptr_compress.c | 110 +++++++++++++++++++++++++++++++++++
 3 files changed, 112 insertions(+)
 create mode 100644 app/test/test_ptr_compress.c

diff --git a/MAINTAINERS b/MAINTAINERS
index ed50121bd2..2565ef5f4b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1697,6 +1697,7 @@ F: lib/pci/
 Pointer Compression
 M: Paul Szczepanek <paul.szczepanek@arm.com>
 F: lib/ptr_compress/
+F: app/test/test_ptr_compress.c
 F: doc/guides/prog_guide/ptr_compress_lib.rst

 Power management
diff --git a/app/test/meson.build b/app/test/meson.build
index df8cc00730..e29258e6ec 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -144,6 +144,7 @@ source_file_deps = {
     'test_power_intel_uncore.c': ['power'],
     'test_power_kvm_vm.c': ['power'],
     'test_prefetch.c': [],
+    'test_ptr_compress.c': ['ptr_compress'],
     'test_rand_perf.c': [],
     'test_rawdev.c': ['rawdev', 'bus_vdev'],
     'test_rcu_qsbr.c': ['rcu', 'hash'],
diff --git a/app/test/test_ptr_compress.c b/app/test/test_ptr_compress.c
new file mode 100644
index 0000000000..a917331d60
--- /dev/null
+++ b/app/test/test_ptr_compress.c
@@ -0,0 +1,110 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2024 Arm Limited
+ */
+
+#include "test.h"
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_ptr_compress.h>
+
+#define MAX_ALIGN_EXPONENT 3
+#define MAX_PTRS 16
+#define NUM_BASES 2
+#define NUM_REGIONS 4
+#define MAX_32BIT_REGION ((uint64_t)UINT32_MAX + 1)
+#define MAX_16BIT_REGION (UINT16_MAX + 1)
+
+static int
+test_ptr_compress_params(
+	void *base,
+	uint64_t mem_sz,
+	unsigned int align_exp,
+	unsigned int num_ptrs,
+	bool use_32_bit)
+{
+	unsigned int i;
+	unsigned int align = 1 << align_exp;
+	void *ptrs[MAX_PTRS] = {0};
+	void *ptrs_out[MAX_PTRS] = {0};
+	uint32_t offsets32[MAX_PTRS] = {0};
+	uint16_t offsets16[MAX_PTRS] = {0};
+
+	for (i = 0; i < num_ptrs; i++) {
+		/* make pointers point at memory in steps of align */
+		/* alternate steps from the start and end of memory region */
+		if ((i & 1) == 1)
+			ptrs[i] = (char *)base + mem_sz - i * align;
+		else
+			ptrs[i] = (char *)base + i * align;
+	}
+
+	if (use_32_bit) {
+		rte_ptr_compress_32_shift(
+				base, ptrs, offsets32, num_ptrs, align_exp);
+		rte_ptr_decompress_32_shift(base, offsets32, ptrs_out, num_ptrs,
+				align_exp);
+	} else {
+		rte_ptr_compress_16_shift(
+				base, ptrs, offsets16, num_ptrs, align_exp);
+		rte_ptr_decompress_16_shift(base, offsets16, ptrs_out, num_ptrs,
+				align_exp);
+	}
+
+	TEST_ASSERT_BUFFERS_ARE_EQUAL(ptrs, ptrs_out, sizeof(void *) * num_ptrs,
+		"Decompressed pointers corrupted\nbase pointer: %p, "
+		"memory region size: %" PRIu64 ", alignment exponent: %u, "
+		"num of pointers: %u, using %s offsets",
+		base, mem_sz, align_exp, num_ptrs,
+		use_32_bit ? "32-bit" : "16-bit");
+
+	return 0;
+}
+
+static int
+test_ptr_compress(void)
+{
+	unsigned int j, k, n;
+	int ret = 0;
+	void * const bases[NUM_BASES] = { (void *)0, (void *)UINT16_MAX };
+	/* maximum size for pointers aligned by consecutive powers of 2 */
+	const uint64_t region_sizes_16[NUM_REGIONS] = {
+		MAX_16BIT_REGION,
+		MAX_16BIT_REGION * 2,
+		MAX_16BIT_REGION * 4,
+		MAX_16BIT_REGION * 8,
+	};
+	const uint64_t region_sizes_32[NUM_REGIONS] = {
+		MAX_32BIT_REGION,
+		MAX_32BIT_REGION * 2,
+		MAX_32BIT_REGION * 4,
+		MAX_32BIT_REGION * 8,
+	};
+
+	for (j = 0; j < NUM_REGIONS; j++) {
+		for (k = 0; k < NUM_BASES; k++) {
+			for (n = 1; n < MAX_PTRS; n++) {
+				ret |= test_ptr_compress_params(
+					bases[k],
+					region_sizes_16[j],
+					j /* exponent of alignment */,
+					n,
+					false
+				);
+				ret |= test_ptr_compress_params(
+					bases[k],
+					region_sizes_32[j],
+					j /* exponent of alignment */,
+					n,
+					true
+				);
+				if (ret != 0)
+					return ret;
+			}
+		}
+	}
+
+	return ret;
+}
+
+REGISTER_FAST_TEST(ptr_compress_autotest, true, true, test_ptr_compress);
--
2.25.1

[PATCH v12 0/6] add pointer compression API

[Paul Szczepanek]

This patchset is proposing adding a new header only library
with utility functions that allow compression of arrays of pointers.

Since this is a header only library a patch needed to be added to amend
the build system to allow adding libraries without source files.

When passing caches full of pointers between threads, memory containing
the pointers is copied multiple times which is especially costly between
cores. A compression method will allow us to shrink the memory size
copied.

The compression takes advantage of the fact that pointers are usually
located in a limited memory region. We can compress them by converting them
to offsets from a base memory address.

Offsets can be stored in fewer bytes (dictated by the memory region size
and alignment of the pointer). For example: an 8 byte aligned pointer
which is part of a 32GB memory pool can be stored in 4 bytes. The API is
very generic and does not assume mempool pointers, any pointer can be
passed in.

Compression is based on few and fast operations and especially with vector
instructions leveraged creates minimal overhead.

The API accepts and returns arrays because the overhead means it only is
worth it when done in bulk.

Test is added that shows potential performance gain from compression. In
this test an array of pointers is passed through a ring between two cores.
It shows the gain which is dependent on the bulk operation size. In this
synthetic test run on ampere altra a substantial (up to 25%) performance
gain is seen if done in bulk size larger than 32. At 32 it breaks even and
lower sizes create a small (less than 5%) slowdown due to overhead.

In a more realistic mock application running the l3 forwarding dpdk
example that works in pipeline mode on two cores this translated into a
~5% throughput increase on an ampere altra.

v2:
* addressed review comments (style, explanations and typos)
* lowered bulk iterations closer to original numbers to keep runtime short
* fixed pointer size warning on 32-bit arch
v3:
* added 16-bit versions of compression functions and tests
* added documentation of these new utility functions in the EAL guide
v4:
* added unit test
* fix bug in NEON implementation of 32-bit decompress
v5:
* disable NEON and SVE implementation on AARCH32 due to wrong pointer size
v6:
* added example usage to commit message of the initial commit
v7:
* rebase to remove clashing mailmap changes
v8:
* put ptr compress into its own library
* add depends-on tag
* remove copyright bumps
* typos
v9
* added MAINTAINERS entries, release notes, doc indexes etc.
* added patch for build system to allow header only library
v10
* fixed problem with meson build adding shared deps to static deps
v11
* added mempool functions to get information about memory range and
alignment
* added tests for the new mempool functions
* added macros to help find the parameters for compression functions
* minor improvement in the SVE compression code
* amended documentation to reflect these changes
v12
* added doxygen and prefixes to macros
* use rte_bitops for clz and ctz
* added unit tests to verify macros
* fixed incorrect letter case in docs

Paul Szczepanek (6):
  lib: allow libraries with no sources
  mempool: add functions to get extra mempool info
  ptr_compress: add pointer compression library
  test: add pointer compress tests to ring perf test
  docs: add pointer compression guide
  test: add unit test for ptr compression

 MAINTAINERS                                |   6 +
 app/test/meson.build                       |  21 +-
 app/test/test_mempool.c                    |  61 ++++
 app/test/test_ptr_compress.c               | 193 +++++++++++
 app/test/test_ring.h                       |  94 ++++++
 app/test/test_ring_perf.c                  | 352 ++++++++++++++-------
 doc/api/doxy-api-index.md                  |   1 +
 doc/api/doxy-api.conf.in                   |   1 +
 doc/guides/prog_guide/index.rst            |   1 +
 doc/guides/prog_guide/ptr_compress_lib.rst | 160 ++++++++++
 doc/guides/rel_notes/release_24_07.rst     |   5 +
 lib/mempool/rte_mempool.c                  |  39 +++
 lib/mempool/rte_mempool.h                  |  37 +++
 lib/mempool/version.map                    |   3 +
 lib/meson.build                            |  17 +
 lib/ptr_compress/meson.build               |   4 +
 lib/ptr_compress/rte_ptr_compress.h        | 324 +++++++++++++++++++
 17 files changed, 1187 insertions(+), 132 deletions(-)
 create mode 100644 app/test/test_ptr_compress.c
 create mode 100644 doc/guides/prog_guide/ptr_compress_lib.rst
 create mode 100644 lib/ptr_compress/meson.build
 create mode 100644 lib/ptr_compress/rte_ptr_compress.h

--
2.25.1

[PATCH v12 1/6] lib: allow libraries with no sources

[Paul Szczepanek]

Allow header only libraries.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
 lib/meson.build | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/lib/meson.build b/lib/meson.build
index 179a272932..7c90602bf5 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -222,6 +222,22 @@ foreach l:libraries
     includes += include_directories(l)
     dpdk_includes += include_directories(l)

+    # special case for header only libraries
+    if sources.length() == 0
+        shared_dep = declare_dependency(include_directories: includes,
+                dependencies: shared_deps)
+        static_dep = declare_dependency(include_directories: includes,
+                dependencies: static_deps)
+        set_variable('shared_rte_' + name, shared_dep)
+        set_variable('static_rte_' + name, static_dep)
+        dpdk_shared_lib_deps += shared_dep
+        dpdk_static_lib_deps += static_dep
+        if developer_mode
+            message('lib/@0@: Defining dependency "@1@"'.format(l, name))
+        endif
+        continue
+    endif
+
     if developer_mode and is_windows and use_function_versioning
         message('@0@: Function versioning is not supported by Windows.'.format(name))
     endif
--
2.25.1

[PATCH v12 2/6] mempool: add functions to get extra mempool info

[Paul Szczepanek]

Add two functions:
- rte_mempool_get_mem_range - get virtual memory range
of the objects in the mempool,
- rte_mempool_get_obj_alignment - get alignment of
objects in the mempool.

Add two tests that test these new functions.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
---
 app/test/test_mempool.c   | 61 +++++++++++++++++++++++++++++++++++++++
 lib/mempool/rte_mempool.c | 39 +++++++++++++++++++++++++
 lib/mempool/rte_mempool.h | 37 ++++++++++++++++++++++++
 lib/mempool/version.map   |  3 ++
 4 files changed, 140 insertions(+)

diff --git a/app/test/test_mempool.c b/app/test/test_mempool.c
index ad7ebd6363..973f4318a8 100644
--- a/app/test/test_mempool.c
+++ b/app/test/test_mempool.c
@@ -843,12 +843,16 @@ test_mempool(void)
 	int ret = -1;
 	uint32_t nb_objs = 0;
 	uint32_t nb_mem_chunks = 0;
+	void *start = NULL;
+	size_t length = 0;
+	size_t alignment = 0;
 	struct rte_mempool *mp_cache = NULL;
 	struct rte_mempool *mp_nocache = NULL;
 	struct rte_mempool *mp_stack_anon = NULL;
 	struct rte_mempool *mp_stack_mempool_iter = NULL;
 	struct rte_mempool *mp_stack = NULL;
 	struct rte_mempool *default_pool = NULL;
+	struct rte_mempool *mp_alignment = NULL;
 	struct mp_data cb_arg = {
 		.ret = -1
 	};
@@ -967,6 +971,62 @@ test_mempool(void)
 	}
 	rte_mempool_obj_iter(default_pool, my_obj_init, NULL);

+	if (rte_mempool_get_mem_range(default_pool, &start, &length)) {
+		printf("cannot get mem range from default mempool\n");
+		GOTO_ERR(ret, err);
+	}
+
+	if (rte_mempool_get_mem_range(NULL, NULL, NULL) != -EINVAL) {
+		printf("rte_mempool_get_mem_range failed to return -EINVAL "
+				"when passed invalid arguments\n");
+		GOTO_ERR(ret, err);
+	}
+
+	if (start == NULL || length < (MEMPOOL_SIZE * MEMPOOL_ELT_SIZE)) {
+		printf("mem range of default mempool is invalid\n");
+		GOTO_ERR(ret, err);
+	}
+
+	/* by default mempool objects are aligned by RTE_MEMPOOL_ALIGN */
+	alignment = rte_mempool_get_obj_alignment(default_pool);
+	if (alignment != RTE_MEMPOOL_ALIGN) {
+		printf("rte_mempool_get_obj_alignment returned wrong value, "
+				"expected %zu, returned %zu\n",
+				(size_t)RTE_MEMPOOL_ALIGN, alignment);
+		GOTO_ERR(ret, err);
+	}
+
+	/* create a mempool with a RTE_MEMPOOL_F_NO_CACHE_ALIGN flag */
+	mp_alignment = rte_mempool_create("test_alignment", MEMPOOL_SIZE,
+		MEMPOOL_ELT_SIZE, 0, 0,
+		NULL, NULL,
+		my_obj_init, NULL,
+		SOCKET_ID_ANY, RTE_MEMPOOL_F_NO_CACHE_ALIGN);
+
+	if (mp_alignment == NULL) {
+		printf("cannot allocate mempool with "
+				"RTE_MEMPOOL_F_NO_CACHE_ALIGN flag\n");
+		GOTO_ERR(ret, err);
+	}
+
+	/* mempool was created with RTE_MEMPOOL_F_NO_CACHE_ALIGN
+	 * and minimum alignment is expected which is sizeof(uint64_t)
+	 */
+	alignment = rte_mempool_get_obj_alignment(mp_alignment);
+	if (alignment != sizeof(uint64_t)) {
+		printf("rte_mempool_get_obj_alignment returned wrong value, "
+				"expected %zu, returned %zu\n",
+				(size_t)sizeof(uint64_t), alignment);
+		GOTO_ERR(ret, err);
+	}
+
+	alignment = rte_mempool_get_obj_alignment(NULL);
+	if (alignment != 0) {
+		printf("rte_mempool_get_obj_alignment failed to return 0 for "
+				" an invalid mempool\n");
+		GOTO_ERR(ret, err);
+	}
+
 	/* retrieve the mempool from its name */
 	if (rte_mempool_lookup("test_nocache") != mp_nocache) {
 		printf("Cannot lookup mempool from its name\n");
@@ -1039,6 +1099,7 @@ test_mempool(void)
 	rte_mempool_free(mp_stack_mempool_iter);
 	rte_mempool_free(mp_stack);
 	rte_mempool_free(default_pool);
+	rte_mempool_free(mp_alignment);

 	return ret;
 }
diff --git a/lib/mempool/rte_mempool.c b/lib/mempool/rte_mempool.c
index 12390a2c81..7a4bafb664 100644
--- a/lib/mempool/rte_mempool.c
+++ b/lib/mempool/rte_mempool.c
@@ -1386,6 +1386,45 @@ void rte_mempool_walk(void (*func)(struct rte_mempool *, void *),
 	rte_mcfg_mempool_read_unlock();
 }

+int rte_mempool_get_mem_range(struct rte_mempool *mp,
+		void **mem_range_start, size_t *mem_range_length)
+{
+	if (mp == NULL || mem_range_start == NULL || mem_range_length == NULL)
+		return -EINVAL;
+
+	void *address_low = (void *)UINTPTR_MAX;
+	void *address_high = 0;
+	struct rte_mempool_memhdr *hdr;
+
+	/* go through memory chunks and find the lowest and highest addresses */
+	STAILQ_FOREACH(hdr, &mp->mem_list, next) {
+		if (address_low > hdr->addr)
+			address_low = hdr->addr;
+		if (address_high < RTE_PTR_ADD(hdr->addr, hdr->len))
+			address_high = RTE_PTR_ADD(hdr->addr, hdr->len);
+	}
+
+	/* check if mempool was not populated yet (no memory chunks) */
+	if (address_low == (void *)UINTPTR_MAX)
+		return -EINVAL;
+
+	*mem_range_start = address_low;
+	*mem_range_length = (size_t)RTE_PTR_DIFF(address_high, address_low);
+
+	return 0;
+}
+
+size_t rte_mempool_get_obj_alignment(struct rte_mempool *mp)
+{
+	if (mp == NULL)
+		return 0;
+
+	if (mp->flags & RTE_MEMPOOL_F_NO_CACHE_ALIGN)
+		return sizeof(uint64_t);
+	else
+		return RTE_MEMPOOL_ALIGN;
+}
+
 struct mempool_callback_data {
 	TAILQ_ENTRY(mempool_callback_data) callbacks;
 	rte_mempool_event_callback *func;
diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h
index 23fd5c8465..8a96f97e72 100644
--- a/lib/mempool/rte_mempool.h
+++ b/lib/mempool/rte_mempool.h
@@ -1917,6 +1917,43 @@ uint32_t rte_mempool_calc_obj_size(uint32_t elt_size, uint32_t flags,
 void rte_mempool_walk(void (*func)(struct rte_mempool *, void *arg),
 		      void *arg);

+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Get information about the memory range used by the mempool.
+ *
+ * @param[in] mp
+ *   Pointer to an initialized mempool.
+ * @param[out] mem_range_start
+ *   Returns lowest address in mempool.
+ * @param[out] mem_range_length
+ *   Returns the length of the memory range containing all the addresses
+ *   in the memory pool.
+ * @return
+ *   0 on success, -EINVAL if arguments are not valid.
+ *
+ **/
+__rte_experimental
+int rte_mempool_get_mem_range(struct rte_mempool *mp,
+		void **mem_range_start, size_t *mem_range_length);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Return object alignment.
+ *
+ * @param[in] mp
+ *   Pointer to a mempool.
+ *
+ * @return
+ *   Object alignment if mp is valid. 0 if mp is NULL.
+ *
+ **/
+__rte_experimental
+size_t rte_mempool_get_obj_alignment(struct rte_mempool *mp);
+
 /**
  * @internal Get page size used for mempool object allocation.
  * This function is internal to mempool library and mempool drivers.
diff --git a/lib/mempool/version.map b/lib/mempool/version.map
index 473277400c..02df634b2a 100644
--- a/lib/mempool/version.map
+++ b/lib/mempool/version.map
@@ -50,6 +50,9 @@ EXPERIMENTAL {
 	__rte_mempool_trace_get_contig_blocks;
 	__rte_mempool_trace_default_cache;
 	__rte_mempool_trace_cache_flush;
+	# added in 24.07
+	rte_mempool_get_mem_range;
+	rte_mempool_get_obj_alignment;
 };

 INTERNAL {
--
2.25.1

RE: [PATCH v12 2/6] mempool: add functions to get extra mempool info

[Morten Brørup]

> From: Paul Szczepanek [mailto:paul.szczepanek@arm.com]
> Sent: Wednesday, 29 May 2024 12.23
> 
> Add two functions:
> - rte_mempool_get_mem_range - get virtual memory range
> of the objects in the mempool,
> - rte_mempool_get_obj_alignment - get alignment of
> objects in the mempool.
> 
> Add two tests that test these new functions.
> 
> Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
> Reviewed-by: Nathan Brown <nathan.brown@arm.com>
> ---

Acked-by: Morten Brørup <mb@smartsharesystems.com>

RE: [PATCH v12 2/6] mempool: add functions to get extra mempool info

[Morten Brørup]

> From: Paul Szczepanek [mailto:paul.szczepanek@arm.com]
> Sent: Wednesday, 29 May 2024 12.23
> 
> Add two functions:
> - rte_mempool_get_mem_range - get virtual memory range
> of the objects in the mempool,
> - rte_mempool_get_obj_alignment - get alignment of
> objects in the mempool.
> 
> Add two tests that test these new functions.
> 
> Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
> Reviewed-by: Nathan Brown <nathan.brown@arm.com>
> ---
> 
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice.
> + *
> + * Get information about the memory range used by the mempool.
> + *
> + * @param[in] mp
> + *   Pointer to an initialized mempool.
> + * @param[out] mem_range_start
> + *   Returns lowest address in mempool.
> + * @param[out] mem_range_length
> + *   Returns the length of the memory range containing all the addresses
> + *   in the memory pool.
> + * @return
> + *   0 on success, -EINVAL if arguments are not valid.
> + *
> + **/
> +__rte_experimental
> +int rte_mempool_get_mem_range(struct rte_mempool *mp,
> +		void **mem_range_start, size_t *mem_range_length);

Paul,

Could you please add one more output parameter "bool *mem_range_contiguous" to this function, returning true if the memory chunks are contiguous.

It will be useful instead of implementing get_memhdr_info() locally in this other patch series:
https://inbox.dpdk.org/dev/MW4PR11MB58724AC82A34A3EEFEF78E898EF22@MW4PR11MB5872.namprd11.prod.outlook.com/

Please coordinate this change directly with Frank Du <frank.du@intel.com>.

-Morten

Re: [PATCH v12 2/6] mempool: add functions to get extra mempool info

[Paul Szczepanek]

On 29/05/2024 14:56, Morten Brørup wrote:
>> From: Paul Szczepanek [mailto:paul.szczepanek@arm.com]
>> Sent: Wednesday, 29 May 2024 12.23
>>
>> Add two functions:
>> - rte_mempool_get_mem_range - get virtual memory range
>> of the objects in the mempool,
>> - rte_mempool_get_obj_alignment - get alignment of
>> objects in the mempool.
>>
>> Add two tests that test these new functions.
>>
>> Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
>> Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
>> Reviewed-by: Nathan Brown <nathan.brown@arm.com>
>> ---
>>
>> +/**
>> + * @warning
>> + * @b EXPERIMENTAL: this API may change without prior notice.
>> + *
>> + * Get information about the memory range used by the mempool.
>> + *
>> + * @param[in] mp
>> + *   Pointer to an initialized mempool.
>> + * @param[out] mem_range_start
>> + *   Returns lowest address in mempool.
>> + * @param[out] mem_range_length
>> + *   Returns the length of the memory range containing all the addresses
>> + *   in the memory pool.
>> + * @return
>> + *   0 on success, -EINVAL if arguments are not valid.
>> + *
>> + **/
>> +__rte_experimental
>> +int rte_mempool_get_mem_range(struct rte_mempool *mp,
>> +		void **mem_range_start, size_t *mem_range_length);
> 
> Paul,
> 
> Could you please add one more output parameter "bool *mem_range_contiguous" to this function, returning true if the memory chunks are contiguous.
> 
> It will be useful instead of implementing get_memhdr_info() locally in this other patch series:
> https://inbox.dpdk.org/dev/MW4PR11MB58724AC82A34A3EEFEF78E898EF22@MW4PR11MB5872.namprd11.prod.outlook.com/
> 
> Please coordinate this change directly with Frank Du <frank.du@intel.com>.
> 
> -Morten
> 

Does this work for you?

int rte_mempool_get_mem_range(struct rte_mempool *mp,
		void **mem_range_start, size_t *mem_range_length,
		bool *contiguous)
{
	if (mp == NULL)
		return -EINVAL;

	void *address_low = (void *)UINTPTR_MAX;
	void *address_high = 0;
	size_t address_diff = 0;
	size_t mem_total_size = 0;
	struct rte_mempool_memhdr *hdr;

	/* go through memory chunks and find the lowest and highest addresses */
	STAILQ_FOREACH(hdr, &mp->mem_list, next) {
		if (address_low > hdr->addr)
			address_low = hdr->addr;
		if (address_high < RTE_PTR_ADD(hdr->addr, hdr->len))
			address_high = RTE_PTR_ADD(hdr->addr, hdr->len);
		mem_total_size += hdr->len;
	}

	/* check if mempool was not populated yet (no memory chunks) */
	if (address_low == (void *)UINTPTR_MAX)
		return -EINVAL;

	address_diff = (size_t)RTE_PTR_DIFF(address_high, address_low);
	if (mem_range_start != NULL)
		*mem_range_start = address_low;
	if (mem_range_length != NULL)
		*mem_range_length = address_diff;
	if (contiguous != NULL)
		*contiguous = (mem_total_size == address_diff) ? true : false;

	return 0;
}

RE: [PATCH v12 2/6] mempool: add functions to get extra mempool info

[Du, Frank]

> -----Original Message-----
> From: Paul Szczepanek <paul.szczepanek@arm.com>
> Sent: Thursday, May 30, 2024 12:18 AM
> To: Morten Brørup <mb@smartsharesystems.com>; dev@dpdk.org; Du, Frank
> <frank.du@intel.com>
> Cc: nd@arm.com; Ferruh Yigit <ferruh.yigit@amd.com>; Andrew Rybchenko
> <andrew.rybchenko@oktetlabs.ru>; Burakov, Anatoly
> <anatoly.burakov@intel.com>; Loftus, Ciara <ciara.loftus@intel.com>
> Subject: Re: [PATCH v12 2/6] mempool: add functions to get extra mempool info
> 
> 
> 
> On 29/05/2024 14:56, Morten Brørup wrote:
> >> From: Paul Szczepanek [mailto:paul.szczepanek@arm.com]
> >> Sent: Wednesday, 29 May 2024 12.23
> >>
> >> Add two functions:
> >> - rte_mempool_get_mem_range - get virtual memory range of the objects
> >> in the mempool,
> >> - rte_mempool_get_obj_alignment - get alignment of objects in the
> >> mempool.
> >>
> >> Add two tests that test these new functions.
> >>
> >> Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
> >> Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
> >> Reviewed-by: Nathan Brown <nathan.brown@arm.com>
> >> ---
> >>
> >> +/**
> >> + * @warning
> >> + * @b EXPERIMENTAL: this API may change without prior notice.
> >> + *
> >> + * Get information about the memory range used by the mempool.
> >> + *
> >> + * @param[in] mp
> >> + *   Pointer to an initialized mempool.
> >> + * @param[out] mem_range_start
> >> + *   Returns lowest address in mempool.
> >> + * @param[out] mem_range_length
> >> + *   Returns the length of the memory range containing all the addresses
> >> + *   in the memory pool.
> >> + * @return
> >> + *   0 on success, -EINVAL if arguments are not valid.
> >> + *
> >> + **/
> >> +__rte_experimental
> >> +int rte_mempool_get_mem_range(struct rte_mempool *mp,
> >> +		void **mem_range_start, size_t *mem_range_length);
> >
> > Paul,
> >
> > Could you please add one more output parameter "bool
> *mem_range_contiguous" to this function, returning true if the memory chunks
> are contiguous.
> >
> > It will be useful instead of implementing get_memhdr_info() locally in this other
> patch series:
> >
> https://inbox.dpdk.org/dev/MW4PR11MB58724AC82A34A3EEFEF78E898EF22@
> MW4P
> > R11MB5872.namprd11.prod.outlook.com/
> >
> > Please coordinate this change directly with Frank Du <frank.du@intel.com>.
> >
> > -Morten
> >
> 
> Does this work for you?
> 
> int rte_mempool_get_mem_range(struct rte_mempool *mp,
> 		void **mem_range_start, size_t *mem_range_length,
> 		bool *contiguous)
> {
> 	if (mp == NULL)
> 		return -EINVAL;
> 
> 	void *address_low = (void *)UINTPTR_MAX;
> 	void *address_high = 0;
> 	size_t address_diff = 0;
> 	size_t mem_total_size = 0;
> 	struct rte_mempool_memhdr *hdr;
> 
> 	/* go through memory chunks and find the lowest and highest addresses
> */
> 	STAILQ_FOREACH(hdr, &mp->mem_list, next) {
> 		if (address_low > hdr->addr)
> 			address_low = hdr->addr;
> 		if (address_high < RTE_PTR_ADD(hdr->addr, hdr->len))
> 			address_high = RTE_PTR_ADD(hdr->addr, hdr->len);
> 		mem_total_size += hdr->len;
> 	}
> 
> 	/* check if mempool was not populated yet (no memory chunks) */
> 	if (address_low == (void *)UINTPTR_MAX)
> 		return -EINVAL;
> 
> 	address_diff = (size_t)RTE_PTR_DIFF(address_high, address_low);
> 	if (mem_range_start != NULL)
> 		*mem_range_start = address_low;
> 	if (mem_range_length != NULL)
> 		*mem_range_length = address_diff;
> 	if (contiguous != NULL)
> 		*contiguous = (mem_total_size == address_diff) ? true : false;
> 
> 	return 0;
> }

Thanks. It looks great. I will rebase the XDP patch to use the new API once it get merged.

[PATCH v12 3/6] ptr_compress: add pointer compression library

[Paul Szczepanek]

Add a new utility header for compressing pointers. The provided
functions can store pointers as 32-bit or 16-bit offsets.

The compression takes advantage of the fact that pointers are
usually located in a limited memory region (like a mempool).
We can compress them by converting them to offsets from a base
memory address. Offsets can be stored in fewer bytes (dictated
by the memory region size and alignment of the pointer).
For example: an 8 byte aligned pointer which is part of a 32GB
memory pool can be stored in 4 bytes.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
---
 MAINTAINERS                            |   4 +
 doc/api/doxy-api-index.md              |   1 +
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/rel_notes/release_24_07.rst |   5 +
 lib/meson.build                        |   1 +
 lib/ptr_compress/meson.build           |   4 +
 lib/ptr_compress/rte_ptr_compress.h    | 324 +++++++++++++++++++++++++
 7 files changed, 340 insertions(+)
 create mode 100644 lib/ptr_compress/meson.build
 create mode 100644 lib/ptr_compress/rte_ptr_compress.h

diff --git a/MAINTAINERS b/MAINTAINERS
index c9adff9846..27b2f03e6c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1694,6 +1694,10 @@ M: Chenbo Xia <chenbox@nvidia.com>
 M: Gaetan Rivet <grive@u256.net>
 F: lib/pci/

+Pointer Compression
+M: Paul Szczepanek <paul.szczepanek@arm.com>
+F: lib/ptr_compress/
+
 Power management
 M: Anatoly Burakov <anatoly.burakov@intel.com>
 M: David Hunt <david.hunt@intel.com>
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 8c1eb8fafa..f9283154f8 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -222,6 +222,7 @@ The public API headers are grouped by topics:
   [config file](@ref rte_cfgfile.h),
   [key/value args](@ref rte_kvargs.h),
   [argument parsing](@ref rte_argparse.h),
+  [ptr_compress](@ref rte_ptr_compress.h),
   [string](@ref rte_string_fns.h),
   [thread](@ref rte_thread.h)

diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index 27afec8b3b..a8823c046f 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -71,6 +71,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/pipeline \
                           @TOPDIR@/lib/port \
                           @TOPDIR@/lib/power \
+                          @TOPDIR@/lib/ptr_compress \
                           @TOPDIR@/lib/rawdev \
                           @TOPDIR@/lib/rcu \
                           @TOPDIR@/lib/regexdev \
diff --git a/doc/guides/rel_notes/release_24_07.rst b/doc/guides/rel_notes/release_24_07.rst
index a69f24cf99..4711792e61 100644
--- a/doc/guides/rel_notes/release_24_07.rst
+++ b/doc/guides/rel_notes/release_24_07.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================

+* **Introduced pointer compression library.**
+
+  Library provides functions to compress and decompress arrays of pointers
+  which can improve application performance under certain conditions.
+  Performance test was added to help users evaluate performance on their setup.

 Removed Items
 -------------
diff --git a/lib/meson.build b/lib/meson.build
index 7c90602bf5..63becee142 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -14,6 +14,7 @@ libraries = [
         'argparse',
         'telemetry', # basic info querying
         'eal', # everything depends on eal
+        'ptr_compress',
         'ring',
         'rcu', # rcu depends on ring
         'mempool',
diff --git a/lib/ptr_compress/meson.build b/lib/ptr_compress/meson.build
new file mode 100644
index 0000000000..e92706a45f
--- /dev/null
+++ b/lib/ptr_compress/meson.build
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+headers = files('rte_ptr_compress.h')
diff --git a/lib/ptr_compress/rte_ptr_compress.h b/lib/ptr_compress/rte_ptr_compress.h
new file mode 100644
index 0000000000..11586246a0
--- /dev/null
+++ b/lib/ptr_compress/rte_ptr_compress.h
@@ -0,0 +1,324 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2024 Arm Limited
+ */
+
+#ifndef RTE_PTR_COMPRESS_H
+#define RTE_PTR_COMPRESS_H
+
+/**
+ * @file
+ * Pointer compression and decompression functions.
+ *
+ * When passing arrays full of pointers between threads, memory containing
+ * the pointers is copied multiple times which is especially costly between
+ * cores. These functions allow us to compress the pointers.
+ *
+ * Compression takes advantage of the fact that pointers are usually located in
+ * a limited memory region. We compress them by converting them to offsets from
+ * a base memory address. Offsets can be stored in fewer bytes.
+ *
+ * The compression functions come in two varieties: 32-bit and 16-bit.
+ *
+ * To determine how many bits are needed to compress the pointer calculate
+ * the biggest offset possible (highest value pointer - base pointer)
+ * and shift the value right according to alignment (shift by exponent of the
+ * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by
+ * 3, etc.). The resulting value must fit in either 32 or 16 bits.
+ *
+ * For usage example and further explanation please see "Pointer Compression" in
+ * doc/guides/prog_guide/ptr_compress_lib.rst
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_vect.h>
+#include <rte_mempool.h>
+#include <rte_bitops.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Calculate how many bits are required to store a value within a given range.
+ * This is to help decide which pointer compression functions can be used to
+ * store pointers contained within a memory range.
+ *
+ * @param range
+ *   The size of the range the value belongs to.
+ * @return
+ *   Number of bits required to store a value.
+ **/
+#define RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(range) \
+	(((uint64_t)range) < 2 ? 1 : \
+		(sizeof(uint64_t) * CHAR_BIT - rte_clz64((uint64_t)range - 1)))
+
+/**
+ * Calculate how many bits in the address can be dropped without losing any
+ * information thanks to the alignment of the address.
+ *
+ * @param alignment
+ *   Memory alignment.
+ * @return
+ *   Size of shift allowed without dropping any information from the pointer.
+ **/
+#define RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(alignment) \
+	((alignment) == 0 ? 0 : rte_ctz64((uint64_t)alignment))
+
+/**
+ * Determine if rte_ptr_compress_16_shift can be used to compress pointers
+ * that contain addresses of memory objects whose memory is aligned by
+ * a given amount and contained in a given memory range.
+ *
+ * @param mem_range
+ *   The size of the memory region that contains the objects pointed to.
+ * @param obj_alignment
+ *   The alignment of objects pointed to.
+ * @return
+ *   1 if function can be used, 0 otherwise.
+ **/
+#define RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT(mem_range, obj_alignment) \
+	((RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(mem_range) - \
+	RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(obj_alignment)) <= 16 ? 1 : 0)
+
+/**
+ * Determine if rte_ptr_compress_32_shift can be used to compress pointers
+ * that contain addresses of memory objects whose memory is aligned by
+ * a given amount and contained in a given memory range.
+ *
+ * @param mem_range
+ *   The size of the memory region that contains the objects pointed to.
+ * @param obj_alignment
+ *   The alignment of objects pointed to.
+ * @return
+ *   1 if function can be used, 0 otherwise.
+ **/
+#define RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT(mem_range, obj_alignment) \
+	((RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(mem_range) - \
+	RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(obj_alignment)) <= 32 ? 1 : 0)
+
+/**
+ * Compress pointers into 32-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 32 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 32GB of the base pointer. If there is no such alignment guarantee they
+ * must be within 4GB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_32_shift(void *ptr_base, void **src_table,
+		uint32_t *dest_table, size_t n, uint8_t bit_shift)
+{
+	size_t i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	do {
+		svbool_t pg = svwhilelt_b64(i, n);
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1w(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+	} while (i < n);
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	/* right shift is done by left shifting by negative int */
+	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	const size_t n_even = n & ~0x1;
+	for (; i < n_even; i += 2) {
+		v_ptr_table = vld1q_u64((const uint64_t *)src_table + i);
+		v_ptr_table = vsubq_u64(v_ptr_table, v_ptr_base);
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		vst1_u32(dest_table + i, vqmovn_u64(v_ptr_table));
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT32_MAX);
+		dest_table[i] = (uint32_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 32-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_32_shift(void *ptr_base, uint32_t *src_table,
+		void **dest_table, size_t n, uint8_t bit_shift)
+{
+	size_t i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	do {
+		svbool_t pg = svwhilelt_b64(i, n);
+		v_ptr_table = svld1uw_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+	} while (i < n);
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	int64x2_t v_shift = vdupq_n_s64(bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	const size_t n_even = n & ~0x1;
+	for (; i < n_even; i += 2) {
+		v_ptr_table = vmovl_u32(vld1_u32(src_table + i));
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		v_ptr_table = vaddq_u64(v_ptr_table, v_ptr_base);
+		vst1q_u64((uint64_t *)dest_table + i, v_ptr_table);
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+/**
+ * Compress pointers into 16-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 16 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 256KB of the base pointer. If there is no such alignment guarantee
+ * they must be within 64KB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_16_shift(void *ptr_base, void **src_table,
+		uint16_t *dest_table, size_t n, uint8_t bit_shift)
+{
+
+	size_t i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	do {
+		svbool_t pg = svwhilelt_b64(i, n);
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1h(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+	} while (i < n);
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT16_MAX);
+		dest_table[i] = (uint16_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 16-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_16_shift(void *ptr_base, uint16_t *src_table,
+		void **dest_table, size_t n, uint8_t bit_shift)
+{
+	size_t i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	do {
+		svbool_t pg = svwhilelt_b64(i, n);
+		v_ptr_table = svld1uh_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+	} while (i < n);
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_PTR_COMPRESS_H */
--
2.25.1

RE: [PATCH v12 3/6] ptr_compress: add pointer compression library

[Morten Brørup]

> From: Paul Szczepanek [mailto:paul.szczepanek@arm.com]
> Sent: Wednesday, 29 May 2024 12.23
> 
> Add a new utility header for compressing pointers. The provided
> functions can store pointers as 32-bit or 16-bit offsets.
> 
> The compression takes advantage of the fact that pointers are
> usually located in a limited memory region (like a mempool).
> We can compress them by converting them to offsets from a base
> memory address. Offsets can be stored in fewer bytes (dictated
> by the memory region size and alignment of the pointer).
> For example: an 8 byte aligned pointer which is part of a 32GB
> memory pool can be stored in 4 bytes.
> 
> Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> Reviewed-by: Nathan Brown <nathan.brown@arm.com>
> Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
> ---

Acked-by: Morten Brørup <mb@smartsharesystems.com>

[PATCH v12 4/6] test: add pointer compress tests to ring perf test

[Paul Szczepanek]

Add a test that runs a zero copy burst enqueue and dequeue on a ring
of raw pointers and compressed pointers at different burst sizes to
showcase performance benefits of newly added pointer compression APIs.

Refactored threading code to pass more parameters to threads to
reuse existing code. Added more bulk sizes to showcase their effects
on compression. Adjusted loop iteration numbers to take into account
bulk sizes to keep runtime constant (instead of number of operations).

Adjusted old printfs to match new ones which have aligned numbers.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
---
 app/test/meson.build      |  20 +--
 app/test/test_ring.h      |  94 ++++++++++
 app/test/test_ring_perf.c | 352 +++++++++++++++++++++++++-------------
 3 files changed, 334 insertions(+), 132 deletions(-)

diff --git a/app/test/meson.build b/app/test/meson.build
index 7d909039ae..df8cc00730 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -156,16 +156,16 @@ source_file_deps = {
 #    'test_resource.c': [],
     'test_rib.c': ['net', 'rib'],
     'test_rib6.c': ['net', 'rib'],
-    'test_ring.c': [],
-    'test_ring_hts_stress.c': [],
-    'test_ring_mpmc_stress.c': [],
-    'test_ring_mt_peek_stress.c': [],
-    'test_ring_mt_peek_stress_zc.c': [],
-    'test_ring_perf.c': [],
-    'test_ring_rts_stress.c': [],
-    'test_ring_st_peek_stress.c': [],
-    'test_ring_st_peek_stress_zc.c': [],
-    'test_ring_stress.c': [],
+    'test_ring.c': ['ptr_compress'],
+    'test_ring_hts_stress.c': ['ptr_compress'],
+    'test_ring_mpmc_stress.c': ['ptr_compress'],
+    'test_ring_mt_peek_stress.c': ['ptr_compress'],
+    'test_ring_mt_peek_stress_zc.c': ['ptr_compress'],
+    'test_ring_perf.c': ['ptr_compress'],
+    'test_ring_rts_stress.c': ['ptr_compress'],
+    'test_ring_st_peek_stress.c': ['ptr_compress'],
+    'test_ring_st_peek_stress_zc.c': ['ptr_compress'],
+    'test_ring_stress.c': ['ptr_compress'],
     'test_rwlock.c': [],
     'test_sched.c': ['net', 'sched'],
     'test_security.c': ['net', 'security'],
diff --git a/app/test/test_ring.h b/app/test/test_ring.h
index 45c263f3ff..9e97c5e3e7 100644
--- a/app/test/test_ring.h
+++ b/app/test/test_ring.h
@@ -5,6 +5,8 @@
 #include <rte_malloc.h>
 #include <rte_ring.h>
 #include <rte_ring_elem.h>
+#include <rte_memcpy.h>
+#include <rte_ptr_compress.h>

 /* API type to call
  * rte_ring_<sp/mp or sc/mc>_enqueue_<bulk/burst>
@@ -25,6 +27,10 @@
 #define TEST_RING_ELEM_BULK 16
 #define TEST_RING_ELEM_BURST 32

+#define TEST_RING_ELEM_BURST_ZC 64
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 128
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 256
+
 #define TEST_RING_IGNORE_API_TYPE ~0U

 /* This function is placed here as it is required for both
@@ -101,6 +107,9 @@ static inline unsigned int
 test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -152,6 +161,47 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n,
 								NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy(zcd.ptr2,
+						(char *)obj + zcd.n1 * esize,
+						(ret - zcd.n1) * esize);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_16_shift(
+					0, obj, zcd.ptr1, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_16_shift(0,
+						obj + (zcd.n1 * 2),
+						zcd.ptr2,
+						(ret - zcd.n1) * 2, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_32_shift(0, obj, zcd.ptr1, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_32_shift(0, obj + zcd.n1,
+						zcd.ptr2, ret - zcd.n1, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
@@ -162,6 +212,9 @@ static inline unsigned int
 test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -213,6 +266,47 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mc_dequeue_burst_elem(r, obj, esize,
 								n, NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy((char *)obj + zcd.n1 * esize,
+						zcd.ptr2,
+						(ret - zcd.n1) * esize);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_16_shift(
+					0, zcd.ptr1, obj, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_16_shift(0, zcd.ptr2,
+						obj + zcd.n1,
+						(ret - zcd.n1) * 2,
+						3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_32_shift(0, zcd.ptr1, obj, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_32_shift(0, zcd.ptr2,
+						obj + zcd.n1, ret - zcd.n1, 3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c
index ee328614c1..57cd04a124 100644
--- a/app/test/test_ring_perf.c
+++ b/app/test/test_ring_perf.c
@@ -22,13 +22,13 @@

 #define RING_NAME "RING_PERF"
 #define RING_SIZE 4096
-#define MAX_BURST 32
+#define MAX_BURST 256

 /*
  * the sizes to enqueue and dequeue in testing
  * (marked volatile so they won't be seen as compile-time constants)
  */
-static const volatile unsigned bulk_sizes[] = { 8, 32 };
+static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128, 256 };

 struct lcore_pair {
 	unsigned c1, c2;
@@ -43,26 +43,30 @@ test_ring_print_test_string(unsigned int api_type, int esize,
 	if (esize == -1)
 		printf("legacy APIs");
 	else
-		printf("elem APIs: element size %dB", esize);
+		printf("elem APIs (size:%2dB)", esize);

 	if (api_type == TEST_RING_IGNORE_API_TYPE)
 		return;

 	if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF)
-		printf(": default enqueue/dequeue: ");
+		printf(" - default enqueue/dequeue");
 	else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC)
-		printf(": SP/SC: ");
+		printf(" - SP/SC");
 	else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC)
-		printf(": MP/MC: ");
+		printf(" - MP/MC");

 	if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE)
-		printf("single: ");
+		printf(" - single - ");
 	else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK)
-		printf("bulk (size: %u): ", bsz);
+		printf(" - bulk (n:%-3u) - ", bsz);
 	else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST)
-		printf("burst (size: %u): ", bsz);
+		printf(" - burst (n:%-3u) - ", bsz);
+	else if ((api_type & (TEST_RING_ELEM_BURST_ZC |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0)
+		printf(" - burst zero copy (n:%-3u) - ", bsz);

-	printf("%.2F\n", value);
+	printf("cycles per elem: %.3F\n", value);
 }

 /**** Functions to analyse our core mask to get cores for different tests ***/
@@ -144,28 +148,33 @@ static void
 test_empty_dequeue(struct rte_ring *r, const int esize,
 			const unsigned int api_type)
 {
-	const unsigned int iter_shift = 26;
+	const unsigned int iter_shift = 29;
 	const unsigned int iterations = 1 << iter_shift;
 	unsigned int i = 0;
 	void *burst[MAX_BURST];

+	const unsigned int bulk_iterations = iterations / bulk_sizes[0];
 	const uint64_t start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	for (i = 0; i < bulk_iterations; i++)
 		test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type);
 	const uint64_t end = rte_rdtsc();

 	test_ring_print_test_string(api_type, esize, bulk_sizes[0],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 }

-/*
- * for the separate enqueue and dequeue threads they take in one param
- * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc
- */
-struct thread_params {
+/* describes the ring used by the enqueue and dequeue thread */
+struct ring_params {
 	struct rte_ring *r;
-	unsigned size;        /* input value, the burst size */
-	double spsc, mpmc;    /* output value, the single or multi timings */
+	unsigned int elem_size;
+	unsigned int bulk_sizes_i; /* index into bulk_size array */
+	unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */
+};
+
+/* Used to specify enqueue and dequeue ring operations and their results */
+struct thread_params {
+	struct ring_params *ring_params;
+	double *results; /* result array size must be equal to bulk_sizes */
 };

 /*
@@ -174,16 +183,15 @@ struct thread_params {
  * flag == 1 -> dequeue
  */
 static __rte_always_inline int
-enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
-	struct thread_params *p)
+enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 15;
+	const unsigned int iter_shift = 22;
 	const unsigned int iterations = 1 << iter_shift;
-	struct rte_ring *r = p->r;
-	unsigned int bsize = p->size;
 	unsigned int i;
 	void *burst = NULL;
+	unsigned int n_remaining;
+	const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i];

 #ifdef RTE_USE_C11_MEM_MODEL
 	if (rte_atomic_fetch_add_explicit(&lcore_count, 1, rte_memory_order_relaxed) + 1 != 2)
@@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 		while(lcore_count != 2)
 			rte_pause();

-	burst = test_ring_calloc(MAX_BURST, esize);
+	burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size);
 	if (burst == NULL)
 		return -1;

 	const uint64_t sp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	const unsigned int bulk_iterations = iterations / bulk_n;
+	for (i = 0; i < bulk_iterations; i++) {
+		n_remaining = bulk_n;
 		do {
 			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_enqueue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_dequeue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			if (ret == 0)
 				rte_pause();
-		} while (!ret);
+			else
+				n_remaining -= ret;
+		} while (n_remaining > 0);
+	}
 	const uint64_t sp_end = rte_rdtsc();

-	const uint64_t mp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
-		do {
-			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			if (ret == 0)
-				rte_pause();
-		} while (!ret);
-	const uint64_t mp_end = rte_rdtsc();
+	p->results[p->ring_params->bulk_sizes_i] =
+			((double)sp_end - sp_start) / iterations;

-	p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize);
-	p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize);
 	return 0;
 }

@@ -243,15 +245,7 @@ enqueue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(0, -1, params);
-}
-
-static int
-enqueue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(0, 16, params);
+	return enqueue_dequeue_bulk_helper(0, params);
 }

 /*
@@ -263,15 +257,7 @@ dequeue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(1, -1, params);
-}
-
-static int
-dequeue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(1, 16, params);
+	return enqueue_dequeue_bulk_helper(1, params);
 }

 /*
@@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p)
  * used to measure ring perf between hyperthreads, cores and sockets.
  */
 static int
-run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize)
+run_on_core_pair(struct lcore_pair *cores,
+		struct thread_params *param1, struct thread_params *param2)
 {
-	lcore_function_t *f1, *f2;
-	struct thread_params param1 = {0}, param2 = {0};
 	unsigned i;
-
-	if (esize == -1) {
-		f1 = enqueue_bulk;
-		f2 = dequeue_bulk;
-	} else {
-		f1 = enqueue_bulk_16B;
-		f2 = dequeue_bulk_16B;
-	}
+	struct ring_params *ring_params = param1->ring_params;

 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		lcore_count = 0;
-		param1.size = param2.size = bulk_sizes[i];
-		param1.r = param2.r = r;
+		ring_params->bulk_sizes_i = i;
 		if (cores->c1 == rte_get_main_lcore()) {
-			rte_eal_remote_launch(f2, &param2, cores->c2);
-			f1(&param1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
+			enqueue_bulk(param1);
 			rte_eal_wait_lcore(cores->c2);
 		} else {
-			rte_eal_remote_launch(f1, &param1, cores->c1);
-			rte_eal_remote_launch(f2, &param2, cores->c2);
+			rte_eal_remote_launch(enqueue_bulk, param1, cores->c1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
 			if (rte_eal_wait_lcore(cores->c1) < 0)
 				return -1;
 			if (rte_eal_wait_lcore(cores->c2) < 0)
 				return -1;
 		}
 		test_ring_print_test_string(
-			TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.spsc + param2.spsc);
-		test_ring_print_test_string(
-			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.mpmc + param2.mpmc);
+				ring_params->ring_flags,
+				ring_params->elem_size,
+				bulk_sizes[i],
+				param1->results[i] + param2->results[i]);
 	}

 	return 0;
@@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 	uint64_t hz = rte_get_timer_hz();
 	uint64_t lcount = 0;
 	const unsigned int lcore = rte_lcore_id();
-	struct thread_params *params = p;
+	struct ring_params *ring_params = p->ring_params;
 	void *burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -347,9 +323,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize)

 	begin = rte_get_timer_cycles();
 	while (time_diff < hz * TIME_MS / 1000) {
-		test_ring_enqueue(params->r, burst, esize, params->size,
+		test_ring_enqueue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
-		test_ring_dequeue(params->r, burst, esize, params->size,
+		test_ring_dequeue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 		lcount++;
 		time_diff = rte_get_timer_cycles() - begin;
@@ -381,7 +359,8 @@ static int
 run_on_all_cores(struct rte_ring *r, const int esize)
 {
 	uint64_t total;
-	struct thread_params param;
+	struct ring_params ring_params = {0};
+	struct thread_params params = { .ring_params = &ring_params };
 	lcore_function_t *lcore_f;
 	unsigned int i, c;

@@ -390,21 +369,20 @@ run_on_all_cores(struct rte_ring *r, const int esize)
 	else
 		lcore_f = load_loop_fn_16B;

-	memset(&param, 0, sizeof(struct thread_params));
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		total = 0;
 		printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]);
-		param.size = bulk_sizes[i];
-		param.r = r;
+		params.ring_params->bulk_sizes_i = i;
+		params.ring_params->r = r;

 		/* clear synchro and start workers */
 		rte_atomic_store_explicit(&synchro, 0, rte_memory_order_relaxed);
-		if (rte_eal_mp_remote_launch(lcore_f, &param, SKIP_MAIN) < 0)
+		if (rte_eal_mp_remote_launch(lcore_f, &params, SKIP_MAIN) < 0)
 			return -1;

 		/* start synchro and launch test on main */
 		rte_atomic_store_explicit(&synchro, 1, rte_memory_order_relaxed);
-		lcore_f(&param);
+		lcore_f(&params);

 		rte_eal_mp_wait_lcore();

@@ -463,9 +441,9 @@ static int
 test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	const unsigned int api_type)
 {
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 26;
 	const unsigned int iterations = 1 << iter_shift;
-	unsigned int sz, i = 0;
+	unsigned int sz, i;
 	void **burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -473,17 +451,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 		return -1;

 	for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) {
+		const unsigned int n = iterations / bulk_sizes[sz];
 		const uint64_t start = rte_rdtsc();
-		for (i = 0; i < iterations; i++) {
+		for (i = 0; i < n; i++) {
 			test_ring_enqueue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 			test_ring_dequeue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 		}
 		const uint64_t end = rte_rdtsc();

 		test_ring_print_test_string(api_type, esize, bulk_sizes[sz],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 	}

 	rte_free(burst);
@@ -491,12 +470,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	return 0;
 }

+static __rte_always_inline int
+test_ring_perf_esize_run_on_two_cores(
+		struct thread_params *param1, struct thread_params *param2)
+{
+	struct lcore_pair cores;
+
+	if (get_two_hyperthreads(&cores) == 0) {
+		printf("\n### Testing using two hyperthreads ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing using two physical cores ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_sockets(&cores) == 0) {
+		printf("\n### Testing using two NUMA nodes ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	return 0;
+}
+
 /* Run all tests for a given element size */
 static __rte_always_inline int
 test_ring_perf_esize(const int esize)
 {
-	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
+	double results_enq[RTE_DIM(bulk_sizes)];
+	double results_deq[RTE_DIM(bulk_sizes)];
+	struct ring_params ring_params = {
+			.elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results_enq };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results_deq };

 	/*
 	 * Performance test for legacy/_elem APIs
@@ -536,22 +546,13 @@ test_ring_perf_esize(const int esize)
 	test_empty_dequeue(r, esize,
 			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);

-	if (get_two_hyperthreads(&cores) == 0) {
-		printf("\n### Testing using two hyperthreads ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.r = r;

-	if (get_two_cores(&cores) == 0) {
-		printf("\n### Testing using two physical cores ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
-	if (get_two_sockets(&cores) == 0) {
-		printf("\n### Testing using two NUMA nodes ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
+
+	ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);

 	printf("\n### Testing using all worker nodes ###\n");
 	if (run_on_all_cores(r, esize) < 0)
@@ -567,6 +568,109 @@ test_ring_perf_esize(const int esize)
 	return -1;
 }

+
+static __rte_always_inline int
+test_ring_perf_compression(void)
+{
+	double results1[RTE_DIM(bulk_sizes)];
+	double results2[RTE_DIM(bulk_sizes)];
+	double results1_comp[2][RTE_DIM(bulk_sizes)];
+	double results2_comp[2][RTE_DIM(bulk_sizes)];
+
+	struct lcore_pair cores;
+	int ret = -1;
+	unsigned int i, j;
+	struct ring_params ring_params = { .elem_size = sizeof(void *) };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results1 };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results2 };
+
+	printf("\n### Testing compression gain ###");
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(void *),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy ###\n");
+		ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	/* rings allow only multiples of 4 as sizes,
+	 * we allocate size 4 despite only using 2 bytes
+	 * and use half of RING_SIZE as the number of elements
+	 */
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE / 2, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[0];
+	param2.results = results2_comp[0];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (16b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[1];
+	param2.results = results2_comp[1];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (32b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	for (j = 0; j < 2; j++) {
+		printf("\n### Potential gain from compression (%d-bit offsets) "
+		"###\n", (j + 1) * 16);
+		for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
+			const double result = results1[i] + results2[i];
+			const double result_comp = results1_comp[j][i] +
+				results2_comp[j][i];
+			const double gain = 100 - (result_comp / result) * 100;
+
+			printf("Gain of %5.1F%% for burst of %-3u elems\n",
+					gain, bulk_sizes[i]);
+		}
+	}
+
+	return ret;
+}
+
 static int
 test_ring_perf(void)
 {
@@ -577,6 +681,10 @@ test_ring_perf(void)
 	if (test_ring_perf_esize(16) == -1)
 		return -1;

+	/* Test for performance gain of compression */
+	if (test_ring_perf_compression() == -1)
+		return -1;
+
 	return 0;
 }

--
2.25.1

[PATCH v12 5/6] docs: add pointer compression guide

[Paul Szczepanek]

Documentation added in the prog guide for the new
utility functions for pointer compression
showing example code and potential usecases.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
---
 MAINTAINERS                                |   1 +
 doc/guides/prog_guide/index.rst            |   1 +
 doc/guides/prog_guide/ptr_compress_lib.rst | 160 +++++++++++++++++++++
 3 files changed, 162 insertions(+)
 create mode 100644 doc/guides/prog_guide/ptr_compress_lib.rst

diff --git a/MAINTAINERS b/MAINTAINERS
index 27b2f03e6c..ed50121bd2 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1697,6 +1697,7 @@ F: lib/pci/
 Pointer Compression
 M: Paul Szczepanek <paul.szczepanek@arm.com>
 F: lib/ptr_compress/
+F: doc/guides/prog_guide/ptr_compress_lib.rst

 Power management
 M: Anatoly Burakov <anatoly.burakov@intel.com>
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index d09d958e6c..6366849eb0 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -73,6 +73,7 @@ Programmer's Guide
     telemetry_lib
     bpf_lib
     graph_lib
+    ptr_compress_lib
     build-sdk-meson
     meson_ut
     build_app
diff --git a/doc/guides/prog_guide/ptr_compress_lib.rst b/doc/guides/prog_guide/ptr_compress_lib.rst
new file mode 100644
index 0000000000..49e94e6c4e
--- /dev/null
+++ b/doc/guides/prog_guide/ptr_compress_lib.rst
@@ -0,0 +1,160 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2024 Arm Limited.
+
+Pointer Compression Library
+===========================
+
+Use ``rte_ptr_compress_16_shift()`` and ``rte_ptr_decompress_16_shift()`` to
+compress and decompress pointers into 16-bit offsets.
+Use ``rte_ptr_compress_32_shift()`` and ``rte_ptr_decompress_32_shift()`` to
+compress and decompress pointers into 32-bit offsets.
+
+Compression takes advantage of the fact that pointers are usually located in a
+limited memory region (like a mempool). By converting them to offsets from a
+base memory address they can be stored in fewer bytes. How many bytes are needed
+to store the offset is dictated by the memory region size and alignment of
+objects the pointers point to.
+
+For example, a pointer which is part of a 4GB memory pool can be stored as 32
+bit offset. If the pointer points to memory that is 8 bytes aligned then 3 bits
+can be dropped from the offset and a 32GB memory pool can now fit in 32 bits.
+
+For performance reasons these requirements are not enforced programmatically.
+The programmer is responsible for ensuring that the combination of distance
+from the base pointer and memory alignment allow for storing of the offset in
+the number of bits indicated by the function name (16 or 32). Start of mempool
+memory would be a good candidate for the base pointer. Otherwise any pointer
+that precedes all pointers, is close enough and has the same alignment as the
+pointers being compressed will work.
+
+Macros present in the rte_ptr_compress.h header may be used to evaluate whether
+compression is possible:
+
+*   RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE
+
+*   RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT
+
+*   RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT
+
+*   RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT
+
+These will help you calculate compression parameters and whether these are
+legal for particular compression function.
+
+If using an rte_mempool you can get the parameters you need to use in the
+compression macros and functions by using ``rte_mempool_get_mem_range()``
+and ``rte_mempool_get_obj_alignment()``.
+
+.. note::
+
+    Performance gains depend on the batch size of pointers and CPU capabilities
+    such as vector extensions. It's important to measure the performance
+    increase on target hardware. A test called ``ring_perf_autotest`` in
+    ``dpdk-test`` can provide the measurements.
+
+Example usage
+~~~~~~~~~~~~~
+
+In this example we send pointers between two cores through a ring. While this
+is a realistic use case the code is simplified for demonstration purposes and
+does not have error handling.
+
+.. code-block:: c
+
+    #include <rte_launch.h>
+    #include <rte_ring.h>
+    #include <rte_ring_elem.h>
+    #include <rte_ptr_compress.h>
+
+    #define ITEMS_ARRAY_SIZE (1024)
+    #define BATCH_SIZE (128)
+    #define ALIGN_EXPONENT (3)
+    #define ITEM_ALIGN (1<<ALIGN_EXPONENT)
+    #define CORE_SEND (1)
+    #define CORE_RECV (2)
+
+    struct item {
+      alignas(ITEM_ALIGN) int a;
+    };
+
+    static struct item items[ITEMS_ARRAY_SIZE] = {0};
+    static struct rte_ring *ring = NULL;
+
+    static int
+    send_compressed(void *args)
+    {
+      struct item *ptrs_send[BATCH_SIZE] = {0};
+      unsigned int n_send = 0;
+      struct rte_ring_zc_data zcd = {0};
+
+      /* in this example we only fill the ptrs_send once and reuse */
+      for (;n_send < BATCH_SIZE; n_send++)
+        ptrs_send[n_send] = &items[n_send];
+
+      for(;;) {
+        n_send = rte_ring_enqueue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        /* compress ptrs_send into offsets */
+        rte_ptr_compress_32_shift(items, /* base pointer */
+          ptrs_send, /* source array to be compressed */
+          zcd.ptr1, /* destination array to store offsets */
+          zcd.n1, /* how many pointers to compress */
+          ALIGN_EXPONENT /* how many bits can we drop from the offset */);
+
+        if (zcd.ptr2 != NULL)
+          rte_ptr_compress_32_shift(items, ptrs_send + zcd.n1,
+            zcd.ptr2, n_send - zcd.n1, ALIGN_EXPONENT);
+
+        rte_ring_enqueue_zc_finish(ring, n_send);
+      }
+      return 1;
+    }
+
+    static int
+    recv_compressed(void *args)
+    {
+      struct item *ptrs_recv[BATCH_SIZE] = {0};
+      unsigned int n_recv;
+      struct rte_ring_zc_data zcd = {0};
+
+      for(;;) {
+        /* receive compressed pointers from the ring */
+        n_recv = rte_ring_dequeue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        rte_ptr_decompress_32_shift(items, /* base pointer */
+          zcd.ptr1, /* source array to decompress */
+          ptrs_recv, /* destination array to store pointers */
+          zcd.n1, /* how many pointers to decompress */
+          ALIGN_EXPONENT /* how many bits were dropped from the offset */);
+
+        /* handle the potential secondary buffer (caused by ring boundary) */
+        if (zcd.ptr2 != NULL)
+          rte_ptr_decompress_32_shift(items,
+            zcd.ptr2,
+            ptrs_recv + zcd.n1,
+            n_recv - zcd.n1,
+            ALIGN_EXPONENT);
+
+        rte_ring_dequeue_zc_finish(ring, n_recv);
+
+        /* ptrs_recv contains what ptrs_send contained in the other thread */
+        /* (...) */
+      }
+      return 1;
+    }
+
+    void
+    compression_example(void)
+    {
+      ring = rte_ring_create_elem(
+        "COMPR_PTRS", sizeof(uint32_t),
+        1024, rte_socket_id(),
+        RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+      rte_eal_remote_launch(send_compressed, NULL, CORE_SEND);
+      rte_eal_remote_launch(recv_compressed, NULL, CORE_RECV);
+
+      for(;;) {}
+    }
--
2.25.1

[PATCH v12 6/6] test: add unit test for ptr compression

[Paul Szczepanek]

Test compresses and decompresses pointers with various combinations
of memory regions and alignments and verifies the pointers are
recovered correctly. Additionally tests helper macros perform
calculations correctly.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
---
 MAINTAINERS                  |   1 +
 app/test/meson.build         |   1 +
 app/test/test_ptr_compress.c | 193 +++++++++++++++++++++++++++++++++++
 3 files changed, 195 insertions(+)
 create mode 100644 app/test/test_ptr_compress.c

diff --git a/MAINTAINERS b/MAINTAINERS
index ed50121bd2..2565ef5f4b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1697,6 +1697,7 @@ F: lib/pci/
 Pointer Compression
 M: Paul Szczepanek <paul.szczepanek@arm.com>
 F: lib/ptr_compress/
+F: app/test/test_ptr_compress.c
 F: doc/guides/prog_guide/ptr_compress_lib.rst

 Power management
diff --git a/app/test/meson.build b/app/test/meson.build
index df8cc00730..e29258e6ec 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -144,6 +144,7 @@ source_file_deps = {
     'test_power_intel_uncore.c': ['power'],
     'test_power_kvm_vm.c': ['power'],
     'test_prefetch.c': [],
+    'test_ptr_compress.c': ['ptr_compress'],
     'test_rand_perf.c': [],
     'test_rawdev.c': ['rawdev', 'bus_vdev'],
     'test_rcu_qsbr.c': ['rcu', 'hash'],
diff --git a/app/test/test_ptr_compress.c b/app/test/test_ptr_compress.c
new file mode 100644
index 0000000000..ab33815974
--- /dev/null
+++ b/app/test/test_ptr_compress.c
@@ -0,0 +1,193 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2024 Arm Limited
+ */
+
+#include "test.h"
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_ptr_compress.h>
+
+#define MAX_ALIGN_EXPONENT 3
+#define MAX_PTRS 16
+#define NUM_BASES 2
+#define NUM_REGIONS 4
+#define MAX_32BIT_REGION ((uint64_t)UINT32_MAX + 1)
+#define MAX_16BIT_REGION (UINT16_MAX + 1)
+
+static int
+test_ptr_compress_params(
+	void *base,
+	uint64_t mem_sz,
+	unsigned int align_exp,
+	unsigned int num_ptrs,
+	bool use_32_bit)
+{
+	unsigned int i;
+	unsigned int align = 1 << align_exp;
+	void *ptrs[MAX_PTRS] = {0};
+	void *ptrs_out[MAX_PTRS] = {0};
+	uint32_t offsets32[MAX_PTRS] = {0};
+	uint16_t offsets16[MAX_PTRS] = {0};
+
+	for (i = 0; i < num_ptrs; i++) {
+		/* make pointers point at memory in steps of align */
+		/* alternate steps from the start and end of memory region */
+		if ((i & 1) == 1)
+			ptrs[i] = (char *)base + mem_sz - i * align;
+		else
+			ptrs[i] = (char *)base + i * align;
+	}
+
+	if (use_32_bit) {
+		rte_ptr_compress_32_shift(
+				base, ptrs, offsets32, num_ptrs, align_exp);
+		rte_ptr_decompress_32_shift(base, offsets32, ptrs_out, num_ptrs,
+				align_exp);
+	} else {
+		rte_ptr_compress_16_shift(
+				base, ptrs, offsets16, num_ptrs, align_exp);
+		rte_ptr_decompress_16_shift(base, offsets16, ptrs_out, num_ptrs,
+				align_exp);
+	}
+
+	TEST_ASSERT_BUFFERS_ARE_EQUAL(ptrs, ptrs_out, sizeof(void *) * num_ptrs,
+		"Decompressed pointers corrupted\nbase pointer: %p, "
+		"memory region size: %" PRIu64 ", alignment exponent: %u, "
+		"num of pointers: %u, using %s offsets",
+		base, mem_sz, align_exp, num_ptrs,
+		use_32_bit ? "32-bit" : "16-bit");
+
+	return 0;
+}
+
+static int
+test_ptr_compress(void)
+{
+	unsigned int j, k, n;
+	int ret = 0;
+	/* the test is run with multiple memory regions and base addresses */
+	void * const bases[NUM_BASES] = { (void *)0, (void *)UINT16_MAX };
+	/* maximum size for pointers aligned by consecutive powers of 2 */
+	const uint64_t region_sizes_16[NUM_REGIONS] = {
+		MAX_16BIT_REGION,
+		MAX_16BIT_REGION * 2,
+		MAX_16BIT_REGION * 4,
+		MAX_16BIT_REGION * 8,
+	};
+	const uint64_t region_sizes_32[NUM_REGIONS] = {
+		MAX_32BIT_REGION,
+		MAX_32BIT_REGION * 2,
+		MAX_32BIT_REGION * 4,
+		MAX_32BIT_REGION * 8,
+	};
+
+	/* main test compresses and decompresses arrays of pointers
+	 * and compares the array before and after to verify that
+	 * pointers are successfully decompressed
+	 */
+
+	for (j = 0; j < NUM_REGIONS; j++) {
+		for (k = 0; k < NUM_BASES; k++) {
+			for (n = 1; n < MAX_PTRS; n++) {
+				ret |= test_ptr_compress_params(
+					bases[k],
+					region_sizes_16[j],
+					j /* exponent of alignment */,
+					n,
+					false
+				);
+				ret |= test_ptr_compress_params(
+					bases[k],
+					region_sizes_32[j],
+					j /* exponent of alignment */,
+					n,
+					true
+				);
+				if (ret != 0)
+					return ret;
+			}
+		}
+	}
+
+	/* verify helper macro computations */
+
+	n = RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(0);
+	TEST_ASSERT_EQUAL(n, 1,
+			"RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(1);
+	TEST_ASSERT_EQUAL(n, 1,
+			"RTE_PTR_COMPREtopSS_BITS_REQUIRED_TO_STORE_VALUE "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(MAX_16BIT_REGION);
+	TEST_ASSERT_EQUAL(n, 16,
+			"RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(MAX_32BIT_REGION);
+	TEST_ASSERT_EQUAL(n, 32,
+			"RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(0);
+	TEST_ASSERT_EQUAL(n, 0,
+			"RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(1);
+	TEST_ASSERT_EQUAL(n, 0,
+			"RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(2);
+	TEST_ASSERT_EQUAL(n, 1,
+			"RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(3);
+	TEST_ASSERT_EQUAL(n, 0,
+			"RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(4);
+	TEST_ASSERT_EQUAL(n, 2,
+			"RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT(MAX_16BIT_REGION, 1);
+	TEST_ASSERT_EQUAL(ret, 1,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT(MAX_16BIT_REGION + 1, 1);
+	TEST_ASSERT_EQUAL(ret, 0,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT(MAX_16BIT_REGION + 1, 2);
+	TEST_ASSERT_EQUAL(ret, 1,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT(MAX_32BIT_REGION, 1);
+	TEST_ASSERT_EQUAL(ret, 1,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT(MAX_32BIT_REGION + 1, 1);
+	TEST_ASSERT_EQUAL(ret, 0,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT(MAX_32BIT_REGION + 1, 2);
+	TEST_ASSERT_EQUAL(ret, 1,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT "
+			"macro computation incorrect\n");
+
+	return 0;
+}
+
+REGISTER_FAST_TEST(ptr_compress_autotest, true, true, test_ptr_compress);
--
2.25.1

[PATCH v13 0/6] add pointer compression API

[Paul Szczepanek]

This patchset is proposing adding a new header only library
with utility functions that allow compression of arrays of pointers.

Since this is a header only library a patch needed to be added to amend
the build system to allow adding libraries without source files.

When passing caches full of pointers between threads, memory containing
the pointers is copied multiple times which is especially costly between
cores. A compression method will allow us to shrink the memory size
copied.

The compression takes advantage of the fact that pointers are usually
located in a limited memory region. We can compress them by converting them
to offsets from a base memory address.

Offsets can be stored in fewer bytes (dictated by the memory region size
and alignment of the pointer). For example: an 8 byte aligned pointer
which is part of a 32GB memory pool can be stored in 4 bytes. The API is
very generic and does not assume mempool pointers, any pointer can be
passed in.

Compression is based on few and fast operations and especially with vector
instructions leveraged creates minimal overhead.

The API accepts and returns arrays because the overhead means it only is
worth it when done in bulk.

Test is added that shows potential performance gain from compression. In
this test an array of pointers is passed through a ring between two cores.
It shows the gain which is dependent on the bulk operation size. In this
synthetic test run on ampere altra a substantial (up to 25%) performance
gain is seen if done in bulk size larger than 32. At 32 it breaks even and
lower sizes create a small (less than 5%) slowdown due to overhead.

In a more realistic mock application running the l3 forwarding dpdk
example that works in pipeline mode on two cores this translated into a
~5% throughput increase on an ampere altra.

v2:
* addressed review comments (style, explanations and typos)
* lowered bulk iterations closer to original numbers to keep runtime short
* fixed pointer size warning on 32-bit arch
v3:
* added 16-bit versions of compression functions and tests
* added documentation of these new utility functions in the EAL guide
v4:
* added unit test
* fix bug in NEON implementation of 32-bit decompress
v5:
* disable NEON and SVE implementation on AARCH32 due to wrong pointer size
v6:
* added example usage to commit message of the initial commit
v7:
* rebase to remove clashing mailmap changes
v8:
* put ptr compress into its own library
* add depends-on tag
* remove copyright bumps
* typos
v9
* added MAINTAINERS entries, release notes, doc indexes etc.
* added patch for build system to allow header only library
v10
* fixed problem with meson build adding shared deps to static deps
v11
* added mempool functions to get information about memory range and
alignment
* added tests for the new mempool functions
* added macros to help find the parameters for compression functions
* minor improvement in the SVE compression code
* amended documentation to reflect these changes
v12
* added doxygen and prefixes to macros
* use rte_bitops for clz and ctz
* added unit tests to verify macros
* fixed incorrect letter case in docs
v13
* added contiguous parameter to rte_mempool_get_mem_range
* made rte_mempool_get_mem_range parameters optional

Paul Szczepanek (6):
  lib: allow libraries with no sources
  mempool: add functions to get extra mempool info
  ptr_compress: add pointer compression library
  test: add pointer compress tests to ring perf test
  docs: add pointer compression guide
  test: add unit test for ptr compression

 MAINTAINERS                                |   6 +
 app/test/meson.build                       |  21 +-
 app/test/test_mempool.c                    |  71 +++++
 app/test/test_ptr_compress.c               | 193 +++++++++++
 app/test/test_ring.h                       |  94 ++++++
 app/test/test_ring_perf.c                  | 352 ++++++++++++++-------
 doc/api/doxy-api-index.md                  |   1 +
 doc/api/doxy-api.conf.in                   |   1 +
 doc/guides/prog_guide/index.rst            |   1 +
 doc/guides/prog_guide/ptr_compress_lib.rst | 160 ++++++++++
 doc/guides/rel_notes/release_24_07.rst     |   5 +
 lib/mempool/rte_mempool.c                  |  48 +++
 lib/mempool/rte_mempool.h                  |  41 +++
 lib/mempool/version.map                    |   3 +
 lib/meson.build                            |  17 +
 lib/ptr_compress/meson.build               |   4 +
 lib/ptr_compress/rte_ptr_compress.h        | 324 +++++++++++++++++++
 17 files changed, 1210 insertions(+), 132 deletions(-)
 create mode 100644 app/test/test_ptr_compress.c
 create mode 100644 doc/guides/prog_guide/ptr_compress_lib.rst
 create mode 100644 lib/ptr_compress/meson.build
 create mode 100644 lib/ptr_compress/rte_ptr_compress.h

--
2.25.1

[PATCH v13 1/6] lib: allow libraries with no sources

[Paul Szczepanek]

Allow header only libraries.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
 lib/meson.build | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/lib/meson.build b/lib/meson.build
index 179a272932..7c90602bf5 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -222,6 +222,22 @@ foreach l:libraries
     includes += include_directories(l)
     dpdk_includes += include_directories(l)

+    # special case for header only libraries
+    if sources.length() == 0
+        shared_dep = declare_dependency(include_directories: includes,
+                dependencies: shared_deps)
+        static_dep = declare_dependency(include_directories: includes,
+                dependencies: static_deps)
+        set_variable('shared_rte_' + name, shared_dep)
+        set_variable('static_rte_' + name, static_dep)
+        dpdk_shared_lib_deps += shared_dep
+        dpdk_static_lib_deps += static_dep
+        if developer_mode
+            message('lib/@0@: Defining dependency "@1@"'.format(l, name))
+        endif
+        continue
+    endif
+
     if developer_mode and is_windows and use_function_versioning
         message('@0@: Function versioning is not supported by Windows.'.format(name))
     endif
--
2.25.1

[PATCH v13 2/6] mempool: add functions to get extra mempool info

[Paul Szczepanek]

Add two functions:
- rte_mempool_get_mem_range - get virtual memory range
of the objects in the mempool,
- rte_mempool_get_obj_alignment - get alignment of
objects in the mempool.

Add two tests that test these new functions.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
---
 app/test/test_mempool.c   | 71 +++++++++++++++++++++++++++++++++++++++
 lib/mempool/rte_mempool.c | 48 ++++++++++++++++++++++++++
 lib/mempool/rte_mempool.h | 41 ++++++++++++++++++++++
 lib/mempool/version.map   |  3 ++
 4 files changed, 163 insertions(+)

diff --git a/app/test/test_mempool.c b/app/test/test_mempool.c
index ad7ebd6363..f32d4a3bb9 100644
--- a/app/test/test_mempool.c
+++ b/app/test/test_mempool.c
@@ -843,12 +843,17 @@ test_mempool(void)
 	int ret = -1;
 	uint32_t nb_objs = 0;
 	uint32_t nb_mem_chunks = 0;
+	void *start = NULL;
+	size_t length = 0;
+	size_t alignment = 0;
+	bool ret_bool = false;
 	struct rte_mempool *mp_cache = NULL;
 	struct rte_mempool *mp_nocache = NULL;
 	struct rte_mempool *mp_stack_anon = NULL;
 	struct rte_mempool *mp_stack_mempool_iter = NULL;
 	struct rte_mempool *mp_stack = NULL;
 	struct rte_mempool *default_pool = NULL;
+	struct rte_mempool *mp_alignment = NULL;
 	struct mp_data cb_arg = {
 		.ret = -1
 	};
@@ -967,6 +972,71 @@ test_mempool(void)
 	}
 	rte_mempool_obj_iter(default_pool, my_obj_init, NULL);

+	if (rte_mempool_get_mem_range(default_pool, &start, &length, NULL)) {
+		printf("cannot get mem range from default mempool\n");
+		GOTO_ERR(ret, err);
+	}
+
+	if (rte_mempool_get_mem_range(NULL, NULL, NULL, NULL) != -EINVAL) {
+		printf("rte_mempool_get_mem_range failed to return -EINVAL "
+				"when passed invalid arguments\n");
+		GOTO_ERR(ret, err);
+	}
+
+	if (start == NULL || length < (MEMPOOL_SIZE * MEMPOOL_ELT_SIZE)) {
+		printf("mem range of default mempool is invalid\n");
+		GOTO_ERR(ret, err);
+	}
+
+	/* by default mempool objects are aligned by RTE_MEMPOOL_ALIGN */
+	alignment = rte_mempool_get_obj_alignment(default_pool);
+	if (alignment != RTE_MEMPOOL_ALIGN) {
+		printf("rte_mempool_get_obj_alignment returned wrong value, "
+				"expected %zu, returned %zu\n",
+				(size_t)RTE_MEMPOOL_ALIGN, alignment);
+		GOTO_ERR(ret, err);
+	}
+
+	/* create a mempool with a RTE_MEMPOOL_F_NO_CACHE_ALIGN flag */
+	mp_alignment = rte_mempool_create("test_alignment",
+		1, 8, /* the small size guarantees single memory chunk */
+		0, 0, NULL, NULL, my_obj_init, NULL,
+		SOCKET_ID_ANY, RTE_MEMPOOL_F_NO_CACHE_ALIGN);
+
+	if (mp_alignment == NULL) {
+		printf("cannot allocate mempool with "
+				"RTE_MEMPOOL_F_NO_CACHE_ALIGN flag\n");
+		GOTO_ERR(ret, err);
+	}
+
+	/* mempool was created with RTE_MEMPOOL_F_NO_CACHE_ALIGN
+	 * and minimum alignment is expected which is sizeof(uint64_t)
+	 */
+	alignment = rte_mempool_get_obj_alignment(mp_alignment);
+	if (alignment != sizeof(uint64_t)) {
+		printf("rte_mempool_get_obj_alignment returned wrong value, "
+				"expected %zu, returned %zu\n",
+				(size_t)sizeof(uint64_t), alignment);
+		GOTO_ERR(ret, err);
+	}
+
+	alignment = rte_mempool_get_obj_alignment(NULL);
+	if (alignment != 0) {
+		printf("rte_mempool_get_obj_alignment failed to return 0 for "
+				" an invalid mempool\n");
+		GOTO_ERR(ret, err);
+	}
+
+	if (rte_mempool_get_mem_range(mp_alignment, NULL, NULL, &ret_bool)) {
+		printf("cannot get mem range from mempool\n");
+		GOTO_ERR(ret, err);
+	}
+
+	if (!ret_bool) {
+		printf("mempool not contiguous\n");
+		GOTO_ERR(ret, err);
+	}
+
 	/* retrieve the mempool from its name */
 	if (rte_mempool_lookup("test_nocache") != mp_nocache) {
 		printf("Cannot lookup mempool from its name\n");
@@ -1039,6 +1109,7 @@ test_mempool(void)
 	rte_mempool_free(mp_stack_mempool_iter);
 	rte_mempool_free(mp_stack);
 	rte_mempool_free(default_pool);
+	rte_mempool_free(mp_alignment);

 	return ret;
 }
diff --git a/lib/mempool/rte_mempool.c b/lib/mempool/rte_mempool.c
index 12390a2c81..b2551572ed 100644
--- a/lib/mempool/rte_mempool.c
+++ b/lib/mempool/rte_mempool.c
@@ -1386,6 +1386,54 @@ void rte_mempool_walk(void (*func)(struct rte_mempool *, void *),
 	rte_mcfg_mempool_read_unlock();
 }

+int rte_mempool_get_mem_range(struct rte_mempool *mp,
+		void **mem_range_start, size_t *mem_range_length,
+		bool *contiguous)
+{
+	if (mp == NULL)
+		return -EINVAL;
+
+	void *address_low = (void *)UINTPTR_MAX;
+	void *address_high = 0;
+	size_t address_diff = 0;
+	size_t mem_total_size = 0;
+	struct rte_mempool_memhdr *hdr;
+
+	/* go through memory chunks and find the lowest and highest addresses */
+	STAILQ_FOREACH(hdr, &mp->mem_list, next) {
+		if (address_low > hdr->addr)
+			address_low = hdr->addr;
+		if (address_high < RTE_PTR_ADD(hdr->addr, hdr->len))
+			address_high = RTE_PTR_ADD(hdr->addr, hdr->len);
+		mem_total_size += hdr->len;
+	}
+
+	/* check if mempool was not populated yet (no memory chunks) */
+	if (address_low == (void *)UINTPTR_MAX)
+		return -EINVAL;
+
+	address_diff = (size_t)RTE_PTR_DIFF(address_high, address_low);
+	if (mem_range_start != NULL)
+		*mem_range_start = address_low;
+	if (mem_range_length != NULL)
+		*mem_range_length = address_diff;
+	if (contiguous != NULL)
+		*contiguous = (mem_total_size == address_diff) ? true : false;
+
+	return 0;
+}
+
+size_t rte_mempool_get_obj_alignment(struct rte_mempool *mp)
+{
+	if (mp == NULL)
+		return 0;
+
+	if (mp->flags & RTE_MEMPOOL_F_NO_CACHE_ALIGN)
+		return sizeof(uint64_t);
+	else
+		return RTE_MEMPOOL_ALIGN;
+}
+
 struct mempool_callback_data {
 	TAILQ_ENTRY(mempool_callback_data) callbacks;
 	rte_mempool_event_callback *func;
diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h
index 23fd5c8465..8a97814b39 100644
--- a/lib/mempool/rte_mempool.h
+++ b/lib/mempool/rte_mempool.h
@@ -1917,6 +1917,47 @@ uint32_t rte_mempool_calc_obj_size(uint32_t elt_size, uint32_t flags,
 void rte_mempool_walk(void (*func)(struct rte_mempool *, void *arg),
 		      void *arg);

+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Get information about the memory range used by the mempool.
+ *
+ * @param[in] mp
+ *   Pointer to an initialized mempool.
+ * @param[out] mem_range_start
+ *   Returns lowest address in mempool. May be NULL.
+ * @param[out] mem_range_length
+ *   Returns the length of the memory range containing all the
+ *   virtual addresses in the memory pool. May be NULL.
+ * @param[out] contiguous
+ *   Returns true if virtual addresses in the memory allocated for the
+ *   mempool are contiguous. May be NULL.
+ * @return
+ *   0 on success, -EINVAL if mempool is not valid.
+ *
+ **/
+__rte_experimental
+int rte_mempool_get_mem_range(struct rte_mempool *mp,
+		void **mem_range_start, size_t *mem_range_length,
+		bool *contiguous);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Return object alignment.
+ *
+ * @param[in] mp
+ *   Pointer to a mempool.
+ *
+ * @return
+ *   Object alignment if mp is valid. 0 if mp is NULL.
+ *
+ **/
+__rte_experimental
+size_t rte_mempool_get_obj_alignment(struct rte_mempool *mp);
+
 /**
  * @internal Get page size used for mempool object allocation.
  * This function is internal to mempool library and mempool drivers.
diff --git a/lib/mempool/version.map b/lib/mempool/version.map
index 473277400c..02df634b2a 100644
--- a/lib/mempool/version.map
+++ b/lib/mempool/version.map
@@ -50,6 +50,9 @@ EXPERIMENTAL {
 	__rte_mempool_trace_get_contig_blocks;
 	__rte_mempool_trace_default_cache;
 	__rte_mempool_trace_cache_flush;
+	# added in 24.07
+	rte_mempool_get_mem_range;
+	rte_mempool_get_obj_alignment;
 };

 INTERNAL {
--
2.25.1

RE: [PATCH v13 2/6] mempool: add functions to get extra mempool info

[Morten Brørup]

> From: Paul Szczepanek [mailto:paul.szczepanek@arm.com]
> Sent: Thursday, 30 May 2024 11.41
> 
> Add two functions:
> - rte_mempool_get_mem_range - get virtual memory range
> of the objects in the mempool,
> - rte_mempool_get_obj_alignment - get alignment of
> objects in the mempool.
> 
> Add two tests that test these new functions.
> 
> Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
> Reviewed-by: Nathan Brown <nathan.brown@arm.com>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
> ---

LGTM.

With or without the suggested changes below,

Reviewed-by: Morten Brørup <mb@smartsharesystems.com>


> @@ -843,12 +843,17 @@ test_mempool(void)
>  	int ret = -1;
>  	uint32_t nb_objs = 0;
>  	uint32_t nb_mem_chunks = 0;
> +	void *start = NULL;
> +	size_t length = 0;
> +	size_t alignment = 0;
> +	bool ret_bool = false;

Suggestion, for consistency:
ret_bool -> contiguous



> +int rte_mempool_get_mem_range(struct rte_mempool *mp,
> +		void **mem_range_start, size_t *mem_range_length,
> +		bool *contiguous)

Suggestion, for consistency:
contiguous -> mem_range_contiguous

Also in the header file.

> +{
> +	if (mp == NULL)
> +		return -EINVAL;
> +
> +	void *address_low = (void *)UINTPTR_MAX;
> +	void *address_high = 0;
> +	size_t address_diff = 0;
> +	size_t mem_total_size = 0;

Suggestion, for consistency:
mem_total_size -> total_size

RE: [PATCH v13 2/6] mempool: add functions to get extra mempool info

[Konstantin Ananyev]

> Add two functions:
> - rte_mempool_get_mem_range - get virtual memory range
> of the objects in the mempool,
> - rte_mempool_get_obj_alignment - get alignment of
> objects in the mempool.
> 
> Add two tests that test these new functions.

LGTM in general, few nits/suggestions below.
 
> 
> Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
> Reviewed-by: Nathan Brown <nathan.brown@arm.com>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
> ---
>  app/test/test_mempool.c   | 71 +++++++++++++++++++++++++++++++++++++++
>  lib/mempool/rte_mempool.c | 48 ++++++++++++++++++++++++++
>  lib/mempool/rte_mempool.h | 41 ++++++++++++++++++++++
>  lib/mempool/version.map   |  3 ++
>  4 files changed, 163 insertions(+)
> 
> diff --git a/app/test/test_mempool.c b/app/test/test_mempool.c
> index ad7ebd6363..f32d4a3bb9 100644
> --- a/app/test/test_mempool.c
> +++ b/app/test/test_mempool.c
> @@ -843,12 +843,17 @@ test_mempool(void)
>  	int ret = -1;
>  	uint32_t nb_objs = 0;
>  	uint32_t nb_mem_chunks = 0;
> +	void *start = NULL;
> +	size_t length = 0;
> +	size_t alignment = 0;
> +	bool ret_bool = false;
>  	struct rte_mempool *mp_cache = NULL;
>  	struct rte_mempool *mp_nocache = NULL;
>  	struct rte_mempool *mp_stack_anon = NULL;
>  	struct rte_mempool *mp_stack_mempool_iter = NULL;
>  	struct rte_mempool *mp_stack = NULL;
>  	struct rte_mempool *default_pool = NULL;
> +	struct rte_mempool *mp_alignment = NULL;
>  	struct mp_data cb_arg = {
>  		.ret = -1
>  	};
> @@ -967,6 +972,71 @@ test_mempool(void)
>  	}
>  	rte_mempool_obj_iter(default_pool, my_obj_init, NULL);
> 
> +	if (rte_mempool_get_mem_range(default_pool, &start, &length, NULL)) {
> +		printf("cannot get mem range from default mempool\n");
> +		GOTO_ERR(ret, err);
> +	}
> +
> +	if (rte_mempool_get_mem_range(NULL, NULL, NULL, NULL) != -EINVAL) {
> +		printf("rte_mempool_get_mem_range failed to return -EINVAL "
> +				"when passed invalid arguments\n");
> +		GOTO_ERR(ret, err);
> +	}
> +
> +	if (start == NULL || length < (MEMPOOL_SIZE * MEMPOOL_ELT_SIZE)) {
> +		printf("mem range of default mempool is invalid\n");
> +		GOTO_ERR(ret, err);
> +	}
> +
> +	/* by default mempool objects are aligned by RTE_MEMPOOL_ALIGN */
> +	alignment = rte_mempool_get_obj_alignment(default_pool);
> +	if (alignment != RTE_MEMPOOL_ALIGN) {
> +		printf("rte_mempool_get_obj_alignment returned wrong value, "
> +				"expected %zu, returned %zu\n",
> +				(size_t)RTE_MEMPOOL_ALIGN, alignment);
> +		GOTO_ERR(ret, err);
> +	}
> +
> +	/* create a mempool with a RTE_MEMPOOL_F_NO_CACHE_ALIGN flag */
> +	mp_alignment = rte_mempool_create("test_alignment",
> +		1, 8, /* the small size guarantees single memory chunk */
> +		0, 0, NULL, NULL, my_obj_init, NULL,
> +		SOCKET_ID_ANY, RTE_MEMPOOL_F_NO_CACHE_ALIGN);
> +
> +	if (mp_alignment == NULL) {
> +		printf("cannot allocate mempool with "
> +				"RTE_MEMPOOL_F_NO_CACHE_ALIGN flag\n");
> +		GOTO_ERR(ret, err);
> +	}
> +
> +	/* mempool was created with RTE_MEMPOOL_F_NO_CACHE_ALIGN
> +	 * and minimum alignment is expected which is sizeof(uint64_t)
> +	 */
> +	alignment = rte_mempool_get_obj_alignment(mp_alignment);
> +	if (alignment != sizeof(uint64_t)) {
> +		printf("rte_mempool_get_obj_alignment returned wrong value, "
> +				"expected %zu, returned %zu\n",
> +				(size_t)sizeof(uint64_t), alignment);
> +		GOTO_ERR(ret, err);
> +	}
> +
> +	alignment = rte_mempool_get_obj_alignment(NULL);
> +	if (alignment != 0) {
> +		printf("rte_mempool_get_obj_alignment failed to return 0 for "
> +				" an invalid mempool\n");
> +		GOTO_ERR(ret, err);
> +	}
> +
> +	if (rte_mempool_get_mem_range(mp_alignment, NULL, NULL, &ret_bool)) {
> +		printf("cannot get mem range from mempool\n");
> +		GOTO_ERR(ret, err);
> +	}
> +
> +	if (!ret_bool) {
> +		printf("mempool not contiguous\n");
> +		GOTO_ERR(ret, err);
> +	}
> +
>  	/* retrieve the mempool from its name */
>  	if (rte_mempool_lookup("test_nocache") != mp_nocache) {
>  		printf("Cannot lookup mempool from its name\n");
> @@ -1039,6 +1109,7 @@ test_mempool(void)
>  	rte_mempool_free(mp_stack_mempool_iter);
>  	rte_mempool_free(mp_stack);
>  	rte_mempool_free(default_pool);
> +	rte_mempool_free(mp_alignment);
> 
>  	return ret;
>  }
> diff --git a/lib/mempool/rte_mempool.c b/lib/mempool/rte_mempool.c
> index 12390a2c81..b2551572ed 100644
> --- a/lib/mempool/rte_mempool.c
> +++ b/lib/mempool/rte_mempool.c
> @@ -1386,6 +1386,54 @@ void rte_mempool_walk(void (*func)(struct rte_mempool *, void *),
>  	rte_mcfg_mempool_read_unlock();
>  }
> 
> +int rte_mempool_get_mem_range(struct rte_mempool *mp,

If possible: "const struct rte_mempool * mp".
 

> +		void **mem_range_start, size_t *mem_range_length,
> +		bool *contiguous)

Jusr as a suggestion: instead of having 3 different params, probably create an
aggregate struct for them:
struct rte_mempool_range {
    uintptr_t start;
    size_t length;
    bool contiguous;
 };
And then pass pointer to that struct as a parameter to fill

> +{
> +	if (mp == NULL)
> +		return -EINVAL;
> +
> +	void *address_low = (void *)UINTPTR_MAX;
> +	void *address_high = 0;
> +	size_t address_diff = 0;
> +	size_t mem_total_size = 0;
> +	struct rte_mempool_memhdr *hdr;
> +
> +	/* go through memory chunks and find the lowest and highest addresses */
> +	STAILQ_FOREACH(hdr, &mp->mem_list, next) {
> +		if (address_low > hdr->addr)
> +			address_low = hdr->addr;
> +		if (address_high < RTE_PTR_ADD(hdr->addr, hdr->len))
> +			address_high = RTE_PTR_ADD(hdr->addr, hdr->len);
> +		mem_total_size += hdr->len;
> +	}
> +
> +	/* check if mempool was not populated yet (no memory chunks) */
> +	if (address_low == (void *)UINTPTR_MAX)
> +		return -EINVAL;
> +
> +	address_diff = (size_t)RTE_PTR_DIFF(address_high, address_low);
> +	if (mem_range_start != NULL)
> +		*mem_range_start = address_low;
> +	if (mem_range_length != NULL)
> +		*mem_range_length = address_diff;
> +	if (contiguous != NULL)
> +		*contiguous = (mem_total_size == address_diff) ? true : false;
> +
> +	return 0;
> +}
> +
> +size_t rte_mempool_get_obj_alignment(struct rte_mempool *mp)

Again: const struct rte_mempool *mp;

> +{
> +	if (mp == NULL)
> +		return 0;
> +
> +	if (mp->flags & RTE_MEMPOOL_F_NO_CACHE_ALIGN)
> +		return sizeof(uint64_t);
> +	else
> +		return RTE_MEMPOOL_ALIGN;
> +}
> +
>  struct mempool_callback_data {
>  	TAILQ_ENTRY(mempool_callback_data) callbacks;
>  	rte_mempool_event_callback *func;
> diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h
> index 23fd5c8465..8a97814b39 100644
> --- a/lib/mempool/rte_mempool.h
> +++ b/lib/mempool/rte_mempool.h
> @@ -1917,6 +1917,47 @@ uint32_t rte_mempool_calc_obj_size(uint32_t elt_size, uint32_t flags,
>  void rte_mempool_walk(void (*func)(struct rte_mempool *, void *arg),
>  		      void *arg);
> 
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice.
> + *
> + * Get information about the memory range used by the mempool.
> + *
> + * @param[in] mp
> + *   Pointer to an initialized mempool.
> + * @param[out] mem_range_start
> + *   Returns lowest address in mempool. May be NULL.
> + * @param[out] mem_range_length
> + *   Returns the length of the memory range containing all the
> + *   virtual addresses in the memory pool. May be NULL.
> + * @param[out] contiguous
> + *   Returns true if virtual addresses in the memory allocated for the
> + *   mempool are contiguous. May be NULL.
> + * @return
> + *   0 on success, -EINVAL if mempool is not valid.
> + *
> + **/
> +__rte_experimental
> +int rte_mempool_get_mem_range(struct rte_mempool *mp,
> +		void **mem_range_start, size_t *mem_range_length,
> +		bool *contiguous);
> +
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice.
> + *
> + * Return object alignment.
> + *
> + * @param[in] mp
> + *   Pointer to a mempool.
> + *
> + * @return
> + *   Object alignment if mp is valid. 0 if mp is NULL.
> + *
> + **/
> +__rte_experimental
> +size_t rte_mempool_get_obj_alignment(struct rte_mempool *mp);
> +
>  /**
>   * @internal Get page size used for mempool object allocation.
>   * This function is internal to mempool library and mempool drivers.
> diff --git a/lib/mempool/version.map b/lib/mempool/version.map
> index 473277400c..02df634b2a 100644
> --- a/lib/mempool/version.map
> +++ b/lib/mempool/version.map
> @@ -50,6 +50,9 @@ EXPERIMENTAL {
>  	__rte_mempool_trace_get_contig_blocks;
>  	__rte_mempool_trace_default_cache;
>  	__rte_mempool_trace_cache_flush;
> +	# added in 24.07
> +	rte_mempool_get_mem_range;
> +	rte_mempool_get_obj_alignment;
>  };
> 
>  INTERNAL {
> --
> 2.25.1

Re: [PATCH v13 2/6] mempool: add functions to get extra mempool info

[Paul Szczepanek]

On 06/06/2024 13:28, Konstantin Ananyev wrote:
> 
> 
>> Add two functions:
>> - rte_mempool_get_mem_range - get virtual memory range
>> of the objects in the mempool,
>> - rte_mempool_get_obj_alignment - get alignment of
>> objects in the mempool.
>>
>> Add two tests that test these new functions.
> 
> LGTM in general, few nits/suggestions below.
>  

I have added your suggestions.
Can I please have an Acked by or Reviewed by from you?

[PATCH v13 3/6] ptr_compress: add pointer compression library

[Paul Szczepanek]

Add a new utility header for compressing pointers. The provided
functions can store pointers as 32-bit or 16-bit offsets.

The compression takes advantage of the fact that pointers are
usually located in a limited memory region (like a mempool).
We can compress them by converting them to offsets from a base
memory address. Offsets can be stored in fewer bytes (dictated
by the memory region size and alignment of the pointer).
For example: an 8 byte aligned pointer which is part of a 32GB
memory pool can be stored in 4 bytes.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
---
 MAINTAINERS                            |   4 +
 doc/api/doxy-api-index.md              |   1 +
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/rel_notes/release_24_07.rst |   5 +
 lib/meson.build                        |   1 +
 lib/ptr_compress/meson.build           |   4 +
 lib/ptr_compress/rte_ptr_compress.h    | 324 +++++++++++++++++++++++++
 7 files changed, 340 insertions(+)
 create mode 100644 lib/ptr_compress/meson.build
 create mode 100644 lib/ptr_compress/rte_ptr_compress.h

diff --git a/MAINTAINERS b/MAINTAINERS
index c9adff9846..27b2f03e6c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1694,6 +1694,10 @@ M: Chenbo Xia <chenbox@nvidia.com>
 M: Gaetan Rivet <grive@u256.net>
 F: lib/pci/

+Pointer Compression
+M: Paul Szczepanek <paul.szczepanek@arm.com>
+F: lib/ptr_compress/
+
 Power management
 M: Anatoly Burakov <anatoly.burakov@intel.com>
 M: David Hunt <david.hunt@intel.com>
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 8c1eb8fafa..f9283154f8 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -222,6 +222,7 @@ The public API headers are grouped by topics:
   [config file](@ref rte_cfgfile.h),
   [key/value args](@ref rte_kvargs.h),
   [argument parsing](@ref rte_argparse.h),
+  [ptr_compress](@ref rte_ptr_compress.h),
   [string](@ref rte_string_fns.h),
   [thread](@ref rte_thread.h)

diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index 27afec8b3b..a8823c046f 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -71,6 +71,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/pipeline \
                           @TOPDIR@/lib/port \
                           @TOPDIR@/lib/power \
+                          @TOPDIR@/lib/ptr_compress \
                           @TOPDIR@/lib/rawdev \
                           @TOPDIR@/lib/rcu \
                           @TOPDIR@/lib/regexdev \
diff --git a/doc/guides/rel_notes/release_24_07.rst b/doc/guides/rel_notes/release_24_07.rst
index a69f24cf99..4711792e61 100644
--- a/doc/guides/rel_notes/release_24_07.rst
+++ b/doc/guides/rel_notes/release_24_07.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================

+* **Introduced pointer compression library.**
+
+  Library provides functions to compress and decompress arrays of pointers
+  which can improve application performance under certain conditions.
+  Performance test was added to help users evaluate performance on their setup.

 Removed Items
 -------------
diff --git a/lib/meson.build b/lib/meson.build
index 7c90602bf5..63becee142 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -14,6 +14,7 @@ libraries = [
         'argparse',
         'telemetry', # basic info querying
         'eal', # everything depends on eal
+        'ptr_compress',
         'ring',
         'rcu', # rcu depends on ring
         'mempool',
diff --git a/lib/ptr_compress/meson.build b/lib/ptr_compress/meson.build
new file mode 100644
index 0000000000..e92706a45f
--- /dev/null
+++ b/lib/ptr_compress/meson.build
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+headers = files('rte_ptr_compress.h')
diff --git a/lib/ptr_compress/rte_ptr_compress.h b/lib/ptr_compress/rte_ptr_compress.h
new file mode 100644
index 0000000000..11586246a0
--- /dev/null
+++ b/lib/ptr_compress/rte_ptr_compress.h
@@ -0,0 +1,324 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2024 Arm Limited
+ */
+
+#ifndef RTE_PTR_COMPRESS_H
+#define RTE_PTR_COMPRESS_H
+
+/**
+ * @file
+ * Pointer compression and decompression functions.
+ *
+ * When passing arrays full of pointers between threads, memory containing
+ * the pointers is copied multiple times which is especially costly between
+ * cores. These functions allow us to compress the pointers.
+ *
+ * Compression takes advantage of the fact that pointers are usually located in
+ * a limited memory region. We compress them by converting them to offsets from
+ * a base memory address. Offsets can be stored in fewer bytes.
+ *
+ * The compression functions come in two varieties: 32-bit and 16-bit.
+ *
+ * To determine how many bits are needed to compress the pointer calculate
+ * the biggest offset possible (highest value pointer - base pointer)
+ * and shift the value right according to alignment (shift by exponent of the
+ * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by
+ * 3, etc.). The resulting value must fit in either 32 or 16 bits.
+ *
+ * For usage example and further explanation please see "Pointer Compression" in
+ * doc/guides/prog_guide/ptr_compress_lib.rst
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_vect.h>
+#include <rte_mempool.h>
+#include <rte_bitops.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Calculate how many bits are required to store a value within a given range.
+ * This is to help decide which pointer compression functions can be used to
+ * store pointers contained within a memory range.
+ *
+ * @param range
+ *   The size of the range the value belongs to.
+ * @return
+ *   Number of bits required to store a value.
+ **/
+#define RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(range) \
+	(((uint64_t)range) < 2 ? 1 : \
+		(sizeof(uint64_t) * CHAR_BIT - rte_clz64((uint64_t)range - 1)))
+
+/**
+ * Calculate how many bits in the address can be dropped without losing any
+ * information thanks to the alignment of the address.
+ *
+ * @param alignment
+ *   Memory alignment.
+ * @return
+ *   Size of shift allowed without dropping any information from the pointer.
+ **/
+#define RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(alignment) \
+	((alignment) == 0 ? 0 : rte_ctz64((uint64_t)alignment))
+
+/**
+ * Determine if rte_ptr_compress_16_shift can be used to compress pointers
+ * that contain addresses of memory objects whose memory is aligned by
+ * a given amount and contained in a given memory range.
+ *
+ * @param mem_range
+ *   The size of the memory region that contains the objects pointed to.
+ * @param obj_alignment
+ *   The alignment of objects pointed to.
+ * @return
+ *   1 if function can be used, 0 otherwise.
+ **/
+#define RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT(mem_range, obj_alignment) \
+	((RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(mem_range) - \
+	RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(obj_alignment)) <= 16 ? 1 : 0)
+
+/**
+ * Determine if rte_ptr_compress_32_shift can be used to compress pointers
+ * that contain addresses of memory objects whose memory is aligned by
+ * a given amount and contained in a given memory range.
+ *
+ * @param mem_range
+ *   The size of the memory region that contains the objects pointed to.
+ * @param obj_alignment
+ *   The alignment of objects pointed to.
+ * @return
+ *   1 if function can be used, 0 otherwise.
+ **/
+#define RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT(mem_range, obj_alignment) \
+	((RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(mem_range) - \
+	RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(obj_alignment)) <= 32 ? 1 : 0)
+
+/**
+ * Compress pointers into 32-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 32 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 32GB of the base pointer. If there is no such alignment guarantee they
+ * must be within 4GB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_32_shift(void *ptr_base, void **src_table,
+		uint32_t *dest_table, size_t n, uint8_t bit_shift)
+{
+	size_t i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	do {
+		svbool_t pg = svwhilelt_b64(i, n);
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1w(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+	} while (i < n);
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	/* right shift is done by left shifting by negative int */
+	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	const size_t n_even = n & ~0x1;
+	for (; i < n_even; i += 2) {
+		v_ptr_table = vld1q_u64((const uint64_t *)src_table + i);
+		v_ptr_table = vsubq_u64(v_ptr_table, v_ptr_base);
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		vst1_u32(dest_table + i, vqmovn_u64(v_ptr_table));
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT32_MAX);
+		dest_table[i] = (uint32_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 32-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_32_shift(void *ptr_base, uint32_t *src_table,
+		void **dest_table, size_t n, uint8_t bit_shift)
+{
+	size_t i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	do {
+		svbool_t pg = svwhilelt_b64(i, n);
+		v_ptr_table = svld1uw_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+	} while (i < n);
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	int64x2_t v_shift = vdupq_n_s64(bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	const size_t n_even = n & ~0x1;
+	for (; i < n_even; i += 2) {
+		v_ptr_table = vmovl_u32(vld1_u32(src_table + i));
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		v_ptr_table = vaddq_u64(v_ptr_table, v_ptr_base);
+		vst1q_u64((uint64_t *)dest_table + i, v_ptr_table);
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+/**
+ * Compress pointers into 16-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 16 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 256KB of the base pointer. If there is no such alignment guarantee
+ * they must be within 64KB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_16_shift(void *ptr_base, void **src_table,
+		uint16_t *dest_table, size_t n, uint8_t bit_shift)
+{
+
+	size_t i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	do {
+		svbool_t pg = svwhilelt_b64(i, n);
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1h(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+	} while (i < n);
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT16_MAX);
+		dest_table[i] = (uint16_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 16-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_16_shift(void *ptr_base, uint16_t *src_table,
+		void **dest_table, size_t n, uint8_t bit_shift)
+{
+	size_t i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	do {
+		svbool_t pg = svwhilelt_b64(i, n);
+		v_ptr_table = svld1uh_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+	} while (i < n);
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_PTR_COMPRESS_H */
--
2.25.1

[PATCH v13 4/6] test: add pointer compress tests to ring perf test

[Paul Szczepanek]

Add a test that runs a zero copy burst enqueue and dequeue on a ring
of raw pointers and compressed pointers at different burst sizes to
showcase performance benefits of newly added pointer compression APIs.

Refactored threading code to pass more parameters to threads to
reuse existing code. Added more bulk sizes to showcase their effects
on compression. Adjusted loop iteration numbers to take into account
bulk sizes to keep runtime constant (instead of number of operations).

Adjusted old printfs to match new ones which have aligned numbers.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
---
 app/test/meson.build      |  20 +--
 app/test/test_ring.h      |  94 ++++++++++
 app/test/test_ring_perf.c | 352 +++++++++++++++++++++++++-------------
 3 files changed, 334 insertions(+), 132 deletions(-)

diff --git a/app/test/meson.build b/app/test/meson.build
index 7d909039ae..df8cc00730 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -156,16 +156,16 @@ source_file_deps = {
 #    'test_resource.c': [],
     'test_rib.c': ['net', 'rib'],
     'test_rib6.c': ['net', 'rib'],
-    'test_ring.c': [],
-    'test_ring_hts_stress.c': [],
-    'test_ring_mpmc_stress.c': [],
-    'test_ring_mt_peek_stress.c': [],
-    'test_ring_mt_peek_stress_zc.c': [],
-    'test_ring_perf.c': [],
-    'test_ring_rts_stress.c': [],
-    'test_ring_st_peek_stress.c': [],
-    'test_ring_st_peek_stress_zc.c': [],
-    'test_ring_stress.c': [],
+    'test_ring.c': ['ptr_compress'],
+    'test_ring_hts_stress.c': ['ptr_compress'],
+    'test_ring_mpmc_stress.c': ['ptr_compress'],
+    'test_ring_mt_peek_stress.c': ['ptr_compress'],
+    'test_ring_mt_peek_stress_zc.c': ['ptr_compress'],
+    'test_ring_perf.c': ['ptr_compress'],
+    'test_ring_rts_stress.c': ['ptr_compress'],
+    'test_ring_st_peek_stress.c': ['ptr_compress'],
+    'test_ring_st_peek_stress_zc.c': ['ptr_compress'],
+    'test_ring_stress.c': ['ptr_compress'],
     'test_rwlock.c': [],
     'test_sched.c': ['net', 'sched'],
     'test_security.c': ['net', 'security'],
diff --git a/app/test/test_ring.h b/app/test/test_ring.h
index 45c263f3ff..9e97c5e3e7 100644
--- a/app/test/test_ring.h
+++ b/app/test/test_ring.h
@@ -5,6 +5,8 @@
 #include <rte_malloc.h>
 #include <rte_ring.h>
 #include <rte_ring_elem.h>
+#include <rte_memcpy.h>
+#include <rte_ptr_compress.h>

 /* API type to call
  * rte_ring_<sp/mp or sc/mc>_enqueue_<bulk/burst>
@@ -25,6 +27,10 @@
 #define TEST_RING_ELEM_BULK 16
 #define TEST_RING_ELEM_BURST 32

+#define TEST_RING_ELEM_BURST_ZC 64
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 128
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 256
+
 #define TEST_RING_IGNORE_API_TYPE ~0U

 /* This function is placed here as it is required for both
@@ -101,6 +107,9 @@ static inline unsigned int
 test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -152,6 +161,47 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n,
 								NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy(zcd.ptr2,
+						(char *)obj + zcd.n1 * esize,
+						(ret - zcd.n1) * esize);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_16_shift(
+					0, obj, zcd.ptr1, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_16_shift(0,
+						obj + (zcd.n1 * 2),
+						zcd.ptr2,
+						(ret - zcd.n1) * 2, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_32_shift(0, obj, zcd.ptr1, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_32_shift(0, obj + zcd.n1,
+						zcd.ptr2, ret - zcd.n1, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
@@ -162,6 +212,9 @@ static inline unsigned int
 test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -213,6 +266,47 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mc_dequeue_burst_elem(r, obj, esize,
 								n, NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy((char *)obj + zcd.n1 * esize,
+						zcd.ptr2,
+						(ret - zcd.n1) * esize);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_16_shift(
+					0, zcd.ptr1, obj, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_16_shift(0, zcd.ptr2,
+						obj + zcd.n1,
+						(ret - zcd.n1) * 2,
+						3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_32_shift(0, zcd.ptr1, obj, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_32_shift(0, zcd.ptr2,
+						obj + zcd.n1, ret - zcd.n1, 3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c
index ee328614c1..57cd04a124 100644
--- a/app/test/test_ring_perf.c
+++ b/app/test/test_ring_perf.c
@@ -22,13 +22,13 @@

 #define RING_NAME "RING_PERF"
 #define RING_SIZE 4096
-#define MAX_BURST 32
+#define MAX_BURST 256

 /*
  * the sizes to enqueue and dequeue in testing
  * (marked volatile so they won't be seen as compile-time constants)
  */
-static const volatile unsigned bulk_sizes[] = { 8, 32 };
+static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128, 256 };

 struct lcore_pair {
 	unsigned c1, c2;
@@ -43,26 +43,30 @@ test_ring_print_test_string(unsigned int api_type, int esize,
 	if (esize == -1)
 		printf("legacy APIs");
 	else
-		printf("elem APIs: element size %dB", esize);
+		printf("elem APIs (size:%2dB)", esize);

 	if (api_type == TEST_RING_IGNORE_API_TYPE)
 		return;

 	if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF)
-		printf(": default enqueue/dequeue: ");
+		printf(" - default enqueue/dequeue");
 	else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC)
-		printf(": SP/SC: ");
+		printf(" - SP/SC");
 	else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC)
-		printf(": MP/MC: ");
+		printf(" - MP/MC");

 	if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE)
-		printf("single: ");
+		printf(" - single - ");
 	else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK)
-		printf("bulk (size: %u): ", bsz);
+		printf(" - bulk (n:%-3u) - ", bsz);
 	else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST)
-		printf("burst (size: %u): ", bsz);
+		printf(" - burst (n:%-3u) - ", bsz);
+	else if ((api_type & (TEST_RING_ELEM_BURST_ZC |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0)
+		printf(" - burst zero copy (n:%-3u) - ", bsz);

-	printf("%.2F\n", value);
+	printf("cycles per elem: %.3F\n", value);
 }

 /**** Functions to analyse our core mask to get cores for different tests ***/
@@ -144,28 +148,33 @@ static void
 test_empty_dequeue(struct rte_ring *r, const int esize,
 			const unsigned int api_type)
 {
-	const unsigned int iter_shift = 26;
+	const unsigned int iter_shift = 29;
 	const unsigned int iterations = 1 << iter_shift;
 	unsigned int i = 0;
 	void *burst[MAX_BURST];

+	const unsigned int bulk_iterations = iterations / bulk_sizes[0];
 	const uint64_t start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	for (i = 0; i < bulk_iterations; i++)
 		test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type);
 	const uint64_t end = rte_rdtsc();

 	test_ring_print_test_string(api_type, esize, bulk_sizes[0],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 }

-/*
- * for the separate enqueue and dequeue threads they take in one param
- * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc
- */
-struct thread_params {
+/* describes the ring used by the enqueue and dequeue thread */
+struct ring_params {
 	struct rte_ring *r;
-	unsigned size;        /* input value, the burst size */
-	double spsc, mpmc;    /* output value, the single or multi timings */
+	unsigned int elem_size;
+	unsigned int bulk_sizes_i; /* index into bulk_size array */
+	unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */
+};
+
+/* Used to specify enqueue and dequeue ring operations and their results */
+struct thread_params {
+	struct ring_params *ring_params;
+	double *results; /* result array size must be equal to bulk_sizes */
 };

 /*
@@ -174,16 +183,15 @@ struct thread_params {
  * flag == 1 -> dequeue
  */
 static __rte_always_inline int
-enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
-	struct thread_params *p)
+enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 15;
+	const unsigned int iter_shift = 22;
 	const unsigned int iterations = 1 << iter_shift;
-	struct rte_ring *r = p->r;
-	unsigned int bsize = p->size;
 	unsigned int i;
 	void *burst = NULL;
+	unsigned int n_remaining;
+	const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i];

 #ifdef RTE_USE_C11_MEM_MODEL
 	if (rte_atomic_fetch_add_explicit(&lcore_count, 1, rte_memory_order_relaxed) + 1 != 2)
@@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 		while(lcore_count != 2)
 			rte_pause();

-	burst = test_ring_calloc(MAX_BURST, esize);
+	burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size);
 	if (burst == NULL)
 		return -1;

 	const uint64_t sp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	const unsigned int bulk_iterations = iterations / bulk_n;
+	for (i = 0; i < bulk_iterations; i++) {
+		n_remaining = bulk_n;
 		do {
 			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_enqueue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_dequeue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			if (ret == 0)
 				rte_pause();
-		} while (!ret);
+			else
+				n_remaining -= ret;
+		} while (n_remaining > 0);
+	}
 	const uint64_t sp_end = rte_rdtsc();

-	const uint64_t mp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
-		do {
-			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			if (ret == 0)
-				rte_pause();
-		} while (!ret);
-	const uint64_t mp_end = rte_rdtsc();
+	p->results[p->ring_params->bulk_sizes_i] =
+			((double)sp_end - sp_start) / iterations;

-	p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize);
-	p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize);
 	return 0;
 }

@@ -243,15 +245,7 @@ enqueue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(0, -1, params);
-}
-
-static int
-enqueue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(0, 16, params);
+	return enqueue_dequeue_bulk_helper(0, params);
 }

 /*
@@ -263,15 +257,7 @@ dequeue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(1, -1, params);
-}
-
-static int
-dequeue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(1, 16, params);
+	return enqueue_dequeue_bulk_helper(1, params);
 }

 /*
@@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p)
  * used to measure ring perf between hyperthreads, cores and sockets.
  */
 static int
-run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize)
+run_on_core_pair(struct lcore_pair *cores,
+		struct thread_params *param1, struct thread_params *param2)
 {
-	lcore_function_t *f1, *f2;
-	struct thread_params param1 = {0}, param2 = {0};
 	unsigned i;
-
-	if (esize == -1) {
-		f1 = enqueue_bulk;
-		f2 = dequeue_bulk;
-	} else {
-		f1 = enqueue_bulk_16B;
-		f2 = dequeue_bulk_16B;
-	}
+	struct ring_params *ring_params = param1->ring_params;

 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		lcore_count = 0;
-		param1.size = param2.size = bulk_sizes[i];
-		param1.r = param2.r = r;
+		ring_params->bulk_sizes_i = i;
 		if (cores->c1 == rte_get_main_lcore()) {
-			rte_eal_remote_launch(f2, &param2, cores->c2);
-			f1(&param1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
+			enqueue_bulk(param1);
 			rte_eal_wait_lcore(cores->c2);
 		} else {
-			rte_eal_remote_launch(f1, &param1, cores->c1);
-			rte_eal_remote_launch(f2, &param2, cores->c2);
+			rte_eal_remote_launch(enqueue_bulk, param1, cores->c1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
 			if (rte_eal_wait_lcore(cores->c1) < 0)
 				return -1;
 			if (rte_eal_wait_lcore(cores->c2) < 0)
 				return -1;
 		}
 		test_ring_print_test_string(
-			TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.spsc + param2.spsc);
-		test_ring_print_test_string(
-			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.mpmc + param2.mpmc);
+				ring_params->ring_flags,
+				ring_params->elem_size,
+				bulk_sizes[i],
+				param1->results[i] + param2->results[i]);
 	}

 	return 0;
@@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 	uint64_t hz = rte_get_timer_hz();
 	uint64_t lcount = 0;
 	const unsigned int lcore = rte_lcore_id();
-	struct thread_params *params = p;
+	struct ring_params *ring_params = p->ring_params;
 	void *burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -347,9 +323,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize)

 	begin = rte_get_timer_cycles();
 	while (time_diff < hz * TIME_MS / 1000) {
-		test_ring_enqueue(params->r, burst, esize, params->size,
+		test_ring_enqueue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
-		test_ring_dequeue(params->r, burst, esize, params->size,
+		test_ring_dequeue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 		lcount++;
 		time_diff = rte_get_timer_cycles() - begin;
@@ -381,7 +359,8 @@ static int
 run_on_all_cores(struct rte_ring *r, const int esize)
 {
 	uint64_t total;
-	struct thread_params param;
+	struct ring_params ring_params = {0};
+	struct thread_params params = { .ring_params = &ring_params };
 	lcore_function_t *lcore_f;
 	unsigned int i, c;

@@ -390,21 +369,20 @@ run_on_all_cores(struct rte_ring *r, const int esize)
 	else
 		lcore_f = load_loop_fn_16B;

-	memset(&param, 0, sizeof(struct thread_params));
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		total = 0;
 		printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]);
-		param.size = bulk_sizes[i];
-		param.r = r;
+		params.ring_params->bulk_sizes_i = i;
+		params.ring_params->r = r;

 		/* clear synchro and start workers */
 		rte_atomic_store_explicit(&synchro, 0, rte_memory_order_relaxed);
-		if (rte_eal_mp_remote_launch(lcore_f, &param, SKIP_MAIN) < 0)
+		if (rte_eal_mp_remote_launch(lcore_f, &params, SKIP_MAIN) < 0)
 			return -1;

 		/* start synchro and launch test on main */
 		rte_atomic_store_explicit(&synchro, 1, rte_memory_order_relaxed);
-		lcore_f(&param);
+		lcore_f(&params);

 		rte_eal_mp_wait_lcore();

@@ -463,9 +441,9 @@ static int
 test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	const unsigned int api_type)
 {
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 26;
 	const unsigned int iterations = 1 << iter_shift;
-	unsigned int sz, i = 0;
+	unsigned int sz, i;
 	void **burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -473,17 +451,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 		return -1;

 	for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) {
+		const unsigned int n = iterations / bulk_sizes[sz];
 		const uint64_t start = rte_rdtsc();
-		for (i = 0; i < iterations; i++) {
+		for (i = 0; i < n; i++) {
 			test_ring_enqueue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 			test_ring_dequeue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 		}
 		const uint64_t end = rte_rdtsc();

 		test_ring_print_test_string(api_type, esize, bulk_sizes[sz],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 	}

 	rte_free(burst);
@@ -491,12 +470,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	return 0;
 }

+static __rte_always_inline int
+test_ring_perf_esize_run_on_two_cores(
+		struct thread_params *param1, struct thread_params *param2)
+{
+	struct lcore_pair cores;
+
+	if (get_two_hyperthreads(&cores) == 0) {
+		printf("\n### Testing using two hyperthreads ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing using two physical cores ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_sockets(&cores) == 0) {
+		printf("\n### Testing using two NUMA nodes ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	return 0;
+}
+
 /* Run all tests for a given element size */
 static __rte_always_inline int
 test_ring_perf_esize(const int esize)
 {
-	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
+	double results_enq[RTE_DIM(bulk_sizes)];
+	double results_deq[RTE_DIM(bulk_sizes)];
+	struct ring_params ring_params = {
+			.elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results_enq };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results_deq };

 	/*
 	 * Performance test for legacy/_elem APIs
@@ -536,22 +546,13 @@ test_ring_perf_esize(const int esize)
 	test_empty_dequeue(r, esize,
 			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);

-	if (get_two_hyperthreads(&cores) == 0) {
-		printf("\n### Testing using two hyperthreads ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.r = r;

-	if (get_two_cores(&cores) == 0) {
-		printf("\n### Testing using two physical cores ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
-	if (get_two_sockets(&cores) == 0) {
-		printf("\n### Testing using two NUMA nodes ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
+
+	ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);

 	printf("\n### Testing using all worker nodes ###\n");
 	if (run_on_all_cores(r, esize) < 0)
@@ -567,6 +568,109 @@ test_ring_perf_esize(const int esize)
 	return -1;
 }

+
+static __rte_always_inline int
+test_ring_perf_compression(void)
+{
+	double results1[RTE_DIM(bulk_sizes)];
+	double results2[RTE_DIM(bulk_sizes)];
+	double results1_comp[2][RTE_DIM(bulk_sizes)];
+	double results2_comp[2][RTE_DIM(bulk_sizes)];
+
+	struct lcore_pair cores;
+	int ret = -1;
+	unsigned int i, j;
+	struct ring_params ring_params = { .elem_size = sizeof(void *) };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results1 };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results2 };
+
+	printf("\n### Testing compression gain ###");
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(void *),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy ###\n");
+		ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	/* rings allow only multiples of 4 as sizes,
+	 * we allocate size 4 despite only using 2 bytes
+	 * and use half of RING_SIZE as the number of elements
+	 */
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE / 2, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[0];
+	param2.results = results2_comp[0];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (16b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[1];
+	param2.results = results2_comp[1];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (32b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	for (j = 0; j < 2; j++) {
+		printf("\n### Potential gain from compression (%d-bit offsets) "
+		"###\n", (j + 1) * 16);
+		for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
+			const double result = results1[i] + results2[i];
+			const double result_comp = results1_comp[j][i] +
+				results2_comp[j][i];
+			const double gain = 100 - (result_comp / result) * 100;
+
+			printf("Gain of %5.1F%% for burst of %-3u elems\n",
+					gain, bulk_sizes[i]);
+		}
+	}
+
+	return ret;
+}
+
 static int
 test_ring_perf(void)
 {
@@ -577,6 +681,10 @@ test_ring_perf(void)
 	if (test_ring_perf_esize(16) == -1)
 		return -1;

+	/* Test for performance gain of compression */
+	if (test_ring_perf_compression() == -1)
+		return -1;
+
 	return 0;
 }

--
2.25.1

[PATCH v13 5/6] docs: add pointer compression guide

[Paul Szczepanek]

Documentation added in the prog guide for the new
utility functions for pointer compression
showing example code and potential usecases.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
---
 MAINTAINERS                                |   1 +
 doc/guides/prog_guide/index.rst            |   1 +
 doc/guides/prog_guide/ptr_compress_lib.rst | 160 +++++++++++++++++++++
 3 files changed, 162 insertions(+)
 create mode 100644 doc/guides/prog_guide/ptr_compress_lib.rst

diff --git a/MAINTAINERS b/MAINTAINERS
index 27b2f03e6c..ed50121bd2 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1697,6 +1697,7 @@ F: lib/pci/
 Pointer Compression
 M: Paul Szczepanek <paul.szczepanek@arm.com>
 F: lib/ptr_compress/
+F: doc/guides/prog_guide/ptr_compress_lib.rst

 Power management
 M: Anatoly Burakov <anatoly.burakov@intel.com>
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index d09d958e6c..6366849eb0 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -73,6 +73,7 @@ Programmer's Guide
     telemetry_lib
     bpf_lib
     graph_lib
+    ptr_compress_lib
     build-sdk-meson
     meson_ut
     build_app
diff --git a/doc/guides/prog_guide/ptr_compress_lib.rst b/doc/guides/prog_guide/ptr_compress_lib.rst
new file mode 100644
index 0000000000..49e94e6c4e
--- /dev/null
+++ b/doc/guides/prog_guide/ptr_compress_lib.rst
@@ -0,0 +1,160 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2024 Arm Limited.
+
+Pointer Compression Library
+===========================
+
+Use ``rte_ptr_compress_16_shift()`` and ``rte_ptr_decompress_16_shift()`` to
+compress and decompress pointers into 16-bit offsets.
+Use ``rte_ptr_compress_32_shift()`` and ``rte_ptr_decompress_32_shift()`` to
+compress and decompress pointers into 32-bit offsets.
+
+Compression takes advantage of the fact that pointers are usually located in a
+limited memory region (like a mempool). By converting them to offsets from a
+base memory address they can be stored in fewer bytes. How many bytes are needed
+to store the offset is dictated by the memory region size and alignment of
+objects the pointers point to.
+
+For example, a pointer which is part of a 4GB memory pool can be stored as 32
+bit offset. If the pointer points to memory that is 8 bytes aligned then 3 bits
+can be dropped from the offset and a 32GB memory pool can now fit in 32 bits.
+
+For performance reasons these requirements are not enforced programmatically.
+The programmer is responsible for ensuring that the combination of distance
+from the base pointer and memory alignment allow for storing of the offset in
+the number of bits indicated by the function name (16 or 32). Start of mempool
+memory would be a good candidate for the base pointer. Otherwise any pointer
+that precedes all pointers, is close enough and has the same alignment as the
+pointers being compressed will work.
+
+Macros present in the rte_ptr_compress.h header may be used to evaluate whether
+compression is possible:
+
+*   RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE
+
+*   RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT
+
+*   RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT
+
+*   RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT
+
+These will help you calculate compression parameters and whether these are
+legal for particular compression function.
+
+If using an rte_mempool you can get the parameters you need to use in the
+compression macros and functions by using ``rte_mempool_get_mem_range()``
+and ``rte_mempool_get_obj_alignment()``.
+
+.. note::
+
+    Performance gains depend on the batch size of pointers and CPU capabilities
+    such as vector extensions. It's important to measure the performance
+    increase on target hardware. A test called ``ring_perf_autotest`` in
+    ``dpdk-test`` can provide the measurements.
+
+Example usage
+~~~~~~~~~~~~~
+
+In this example we send pointers between two cores through a ring. While this
+is a realistic use case the code is simplified for demonstration purposes and
+does not have error handling.
+
+.. code-block:: c
+
+    #include <rte_launch.h>
+    #include <rte_ring.h>
+    #include <rte_ring_elem.h>
+    #include <rte_ptr_compress.h>
+
+    #define ITEMS_ARRAY_SIZE (1024)
+    #define BATCH_SIZE (128)
+    #define ALIGN_EXPONENT (3)
+    #define ITEM_ALIGN (1<<ALIGN_EXPONENT)
+    #define CORE_SEND (1)
+    #define CORE_RECV (2)
+
+    struct item {
+      alignas(ITEM_ALIGN) int a;
+    };
+
+    static struct item items[ITEMS_ARRAY_SIZE] = {0};
+    static struct rte_ring *ring = NULL;
+
+    static int
+    send_compressed(void *args)
+    {
+      struct item *ptrs_send[BATCH_SIZE] = {0};
+      unsigned int n_send = 0;
+      struct rte_ring_zc_data zcd = {0};
+
+      /* in this example we only fill the ptrs_send once and reuse */
+      for (;n_send < BATCH_SIZE; n_send++)
+        ptrs_send[n_send] = &items[n_send];
+
+      for(;;) {
+        n_send = rte_ring_enqueue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        /* compress ptrs_send into offsets */
+        rte_ptr_compress_32_shift(items, /* base pointer */
+          ptrs_send, /* source array to be compressed */
+          zcd.ptr1, /* destination array to store offsets */
+          zcd.n1, /* how many pointers to compress */
+          ALIGN_EXPONENT /* how many bits can we drop from the offset */);
+
+        if (zcd.ptr2 != NULL)
+          rte_ptr_compress_32_shift(items, ptrs_send + zcd.n1,
+            zcd.ptr2, n_send - zcd.n1, ALIGN_EXPONENT);
+
+        rte_ring_enqueue_zc_finish(ring, n_send);
+      }
+      return 1;
+    }
+
+    static int
+    recv_compressed(void *args)
+    {
+      struct item *ptrs_recv[BATCH_SIZE] = {0};
+      unsigned int n_recv;
+      struct rte_ring_zc_data zcd = {0};
+
+      for(;;) {
+        /* receive compressed pointers from the ring */
+        n_recv = rte_ring_dequeue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        rte_ptr_decompress_32_shift(items, /* base pointer */
+          zcd.ptr1, /* source array to decompress */
+          ptrs_recv, /* destination array to store pointers */
+          zcd.n1, /* how many pointers to decompress */
+          ALIGN_EXPONENT /* how many bits were dropped from the offset */);
+
+        /* handle the potential secondary buffer (caused by ring boundary) */
+        if (zcd.ptr2 != NULL)
+          rte_ptr_decompress_32_shift(items,
+            zcd.ptr2,
+            ptrs_recv + zcd.n1,
+            n_recv - zcd.n1,
+            ALIGN_EXPONENT);
+
+        rte_ring_dequeue_zc_finish(ring, n_recv);
+
+        /* ptrs_recv contains what ptrs_send contained in the other thread */
+        /* (...) */
+      }
+      return 1;
+    }
+
+    void
+    compression_example(void)
+    {
+      ring = rte_ring_create_elem(
+        "COMPR_PTRS", sizeof(uint32_t),
+        1024, rte_socket_id(),
+        RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+      rte_eal_remote_launch(send_compressed, NULL, CORE_SEND);
+      rte_eal_remote_launch(recv_compressed, NULL, CORE_RECV);
+
+      for(;;) {}
+    }
--
2.25.1

[PATCH v13 6/6] test: add unit test for ptr compression

[Paul Szczepanek]

Test compresses and decompresses pointers with various combinations
of memory regions and alignments and verifies the pointers are
recovered correctly. Additionally tests helper macros perform
calculations correctly.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
---
 MAINTAINERS                  |   1 +
 app/test/meson.build         |   1 +
 app/test/test_ptr_compress.c | 193 +++++++++++++++++++++++++++++++++++
 3 files changed, 195 insertions(+)
 create mode 100644 app/test/test_ptr_compress.c

diff --git a/MAINTAINERS b/MAINTAINERS
index ed50121bd2..2565ef5f4b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1697,6 +1697,7 @@ F: lib/pci/
 Pointer Compression
 M: Paul Szczepanek <paul.szczepanek@arm.com>
 F: lib/ptr_compress/
+F: app/test/test_ptr_compress.c
 F: doc/guides/prog_guide/ptr_compress_lib.rst

 Power management
diff --git a/app/test/meson.build b/app/test/meson.build
index df8cc00730..e29258e6ec 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -144,6 +144,7 @@ source_file_deps = {
     'test_power_intel_uncore.c': ['power'],
     'test_power_kvm_vm.c': ['power'],
     'test_prefetch.c': [],
+    'test_ptr_compress.c': ['ptr_compress'],
     'test_rand_perf.c': [],
     'test_rawdev.c': ['rawdev', 'bus_vdev'],
     'test_rcu_qsbr.c': ['rcu', 'hash'],
diff --git a/app/test/test_ptr_compress.c b/app/test/test_ptr_compress.c
new file mode 100644
index 0000000000..ab33815974
--- /dev/null
+++ b/app/test/test_ptr_compress.c
@@ -0,0 +1,193 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2024 Arm Limited
+ */
+
+#include "test.h"
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_ptr_compress.h>
+
+#define MAX_ALIGN_EXPONENT 3
+#define MAX_PTRS 16
+#define NUM_BASES 2
+#define NUM_REGIONS 4
+#define MAX_32BIT_REGION ((uint64_t)UINT32_MAX + 1)
+#define MAX_16BIT_REGION (UINT16_MAX + 1)
+
+static int
+test_ptr_compress_params(
+	void *base,
+	uint64_t mem_sz,
+	unsigned int align_exp,
+	unsigned int num_ptrs,
+	bool use_32_bit)
+{
+	unsigned int i;
+	unsigned int align = 1 << align_exp;
+	void *ptrs[MAX_PTRS] = {0};
+	void *ptrs_out[MAX_PTRS] = {0};
+	uint32_t offsets32[MAX_PTRS] = {0};
+	uint16_t offsets16[MAX_PTRS] = {0};
+
+	for (i = 0; i < num_ptrs; i++) {
+		/* make pointers point at memory in steps of align */
+		/* alternate steps from the start and end of memory region */
+		if ((i & 1) == 1)
+			ptrs[i] = (char *)base + mem_sz - i * align;
+		else
+			ptrs[i] = (char *)base + i * align;
+	}
+
+	if (use_32_bit) {
+		rte_ptr_compress_32_shift(
+				base, ptrs, offsets32, num_ptrs, align_exp);
+		rte_ptr_decompress_32_shift(base, offsets32, ptrs_out, num_ptrs,
+				align_exp);
+	} else {
+		rte_ptr_compress_16_shift(
+				base, ptrs, offsets16, num_ptrs, align_exp);
+		rte_ptr_decompress_16_shift(base, offsets16, ptrs_out, num_ptrs,
+				align_exp);
+	}
+
+	TEST_ASSERT_BUFFERS_ARE_EQUAL(ptrs, ptrs_out, sizeof(void *) * num_ptrs,
+		"Decompressed pointers corrupted\nbase pointer: %p, "
+		"memory region size: %" PRIu64 ", alignment exponent: %u, "
+		"num of pointers: %u, using %s offsets",
+		base, mem_sz, align_exp, num_ptrs,
+		use_32_bit ? "32-bit" : "16-bit");
+
+	return 0;
+}
+
+static int
+test_ptr_compress(void)
+{
+	unsigned int j, k, n;
+	int ret = 0;
+	/* the test is run with multiple memory regions and base addresses */
+	void * const bases[NUM_BASES] = { (void *)0, (void *)UINT16_MAX };
+	/* maximum size for pointers aligned by consecutive powers of 2 */
+	const uint64_t region_sizes_16[NUM_REGIONS] = {
+		MAX_16BIT_REGION,
+		MAX_16BIT_REGION * 2,
+		MAX_16BIT_REGION * 4,
+		MAX_16BIT_REGION * 8,
+	};
+	const uint64_t region_sizes_32[NUM_REGIONS] = {
+		MAX_32BIT_REGION,
+		MAX_32BIT_REGION * 2,
+		MAX_32BIT_REGION * 4,
+		MAX_32BIT_REGION * 8,
+	};
+
+	/* main test compresses and decompresses arrays of pointers
+	 * and compares the array before and after to verify that
+	 * pointers are successfully decompressed
+	 */
+
+	for (j = 0; j < NUM_REGIONS; j++) {
+		for (k = 0; k < NUM_BASES; k++) {
+			for (n = 1; n < MAX_PTRS; n++) {
+				ret |= test_ptr_compress_params(
+					bases[k],
+					region_sizes_16[j],
+					j /* exponent of alignment */,
+					n,
+					false
+				);
+				ret |= test_ptr_compress_params(
+					bases[k],
+					region_sizes_32[j],
+					j /* exponent of alignment */,
+					n,
+					true
+				);
+				if (ret != 0)
+					return ret;
+			}
+		}
+	}
+
+	/* verify helper macro computations */
+
+	n = RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(0);
+	TEST_ASSERT_EQUAL(n, 1,
+			"RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(1);
+	TEST_ASSERT_EQUAL(n, 1,
+			"RTE_PTR_COMPREtopSS_BITS_REQUIRED_TO_STORE_VALUE "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(MAX_16BIT_REGION);
+	TEST_ASSERT_EQUAL(n, 16,
+			"RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(MAX_32BIT_REGION);
+	TEST_ASSERT_EQUAL(n, 32,
+			"RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(0);
+	TEST_ASSERT_EQUAL(n, 0,
+			"RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(1);
+	TEST_ASSERT_EQUAL(n, 0,
+			"RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(2);
+	TEST_ASSERT_EQUAL(n, 1,
+			"RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(3);
+	TEST_ASSERT_EQUAL(n, 0,
+			"RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(4);
+	TEST_ASSERT_EQUAL(n, 2,
+			"RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT(MAX_16BIT_REGION, 1);
+	TEST_ASSERT_EQUAL(ret, 1,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT(MAX_16BIT_REGION + 1, 1);
+	TEST_ASSERT_EQUAL(ret, 0,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT(MAX_16BIT_REGION + 1, 2);
+	TEST_ASSERT_EQUAL(ret, 1,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT(MAX_32BIT_REGION, 1);
+	TEST_ASSERT_EQUAL(ret, 1,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT(MAX_32BIT_REGION + 1, 1);
+	TEST_ASSERT_EQUAL(ret, 0,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT(MAX_32BIT_REGION + 1, 2);
+	TEST_ASSERT_EQUAL(ret, 1,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT "
+			"macro computation incorrect\n");
+
+	return 0;
+}
+
+REGISTER_FAST_TEST(ptr_compress_autotest, true, true, test_ptr_compress);
--
2.25.1

Re: [PATCH v13 6/6] test: add unit test for ptr compression

[Paul Szczepanek]

Recheck-request: github-robot

Re: [PATCH v13 6/6] test: add unit test for ptr compression

[Paul Szczepanek]

Recheck-request: iol-unit-amd64-testing

Re: [PATCH v13 0/6] add pointer compression API

[Paul Szczepanek]

Recheck-request: github-robot

Re: [PATCH v13 0/6] add pointer compression API

[Paul Szczepanek]

Recheck-request: iol-unit-amd64-testing

[RFC 2/2] test: add pointer compress tests to ring perf test

[Paul Szczepanek]

Add a test that runs a zero copy burst enqueue and dequeue on a ring
of raw pointers and compressed pointers at different burst sizes to
showcase performance benefits of newly added pointer compression APIs

To reuse existing code, some refactoring was done to pass more
parameters to test threads. Additionally more bulk sizes were
added to showcase their effects on compression. To keep runtime
reasoanble iterations where adjusted to take into account bulk sizes.

Old printfs are adjusted to match new ones which have aligned numbers.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 app/test/test_ring.h      |  59 ++++++-
 app/test/test_ring_perf.c | 324 +++++++++++++++++++++++---------------
 2 files changed, 259 insertions(+), 124 deletions(-)

diff --git a/app/test/test_ring.h b/app/test/test_ring.h
index 45c263f3ff..e8b7525c23 100644
--- a/app/test/test_ring.h
+++ b/app/test/test_ring.h
@@ -1,10 +1,12 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */
 
 #include <rte_malloc.h>
 #include <rte_ring.h>
 #include <rte_ring_elem.h>
+#include <rte_memcpy.h>
+#include <rte_ptr_compress.h>
 
 /* API type to call
  * rte_ring_<sp/mp or sc/mc>_enqueue_<bulk/burst>
@@ -25,6 +27,9 @@
 #define TEST_RING_ELEM_BULK 16
 #define TEST_RING_ELEM_BURST 32
 
+#define TEST_RING_ELEM_BURST_ZC 64
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 128
+
 #define TEST_RING_IGNORE_API_TYPE ~0U
 
 /* This function is placed here as it is required for both
@@ -101,6 +106,9 @@ static inline unsigned int
 test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -152,6 +160,29 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n,
 								NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy(zcd.ptr2,
+						(char *)obj + zcd.n1 * esize,
+						(ret - zcd.n1) * esize);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_32(0, obj, zcd.ptr1, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_32(0, obj + zcd.n1,
+						zcd.ptr2, ret - zcd.n1, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
@@ -162,6 +193,9 @@ static inline unsigned int
 test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -213,6 +247,29 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mc_dequeue_burst_elem(r, obj, esize,
 								n, NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy((char *)obj + zcd.n1 * esize,
+						zcd.ptr2,
+						(ret - zcd.n1) * esize);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_32(0, zcd.ptr1, obj, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_32(0, zcd.ptr2,
+						obj + zcd.n1, ret - zcd.n1, 3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c
index d7c5a4c30b..3c9dc24447 100644
--- a/app/test/test_ring_perf.c
+++ b/app/test/test_ring_perf.c
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2010-2014 Intel Corporation
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */
 
 
@@ -21,14 +21,15 @@
  */
 
 #define RING_NAME "RING_PERF"
+#define RING_NAME_COMPRESS "RING_COMP"
 #define RING_SIZE 4096
-#define MAX_BURST 32
+#define MAX_BURST 128
 
 /*
  * the sizes to enqueue and dequeue in testing
  * (marked volatile so they won't be seen as compile-time constants)
  */
-static const volatile unsigned bulk_sizes[] = { 8, 32 };
+static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128 };
 
 struct lcore_pair {
 	unsigned c1, c2;
@@ -43,26 +44,29 @@ test_ring_print_test_string(unsigned int api_type, int esize,
 	if (esize == -1)
 		printf("legacy APIs");
 	else
-		printf("elem APIs: element size %dB", esize);
+		printf("elem APIs (size:%2dB)", esize);
 
 	if (api_type == TEST_RING_IGNORE_API_TYPE)
 		return;
 
 	if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF)
-		printf(": default enqueue/dequeue: ");
+		printf(" - default enqueue/dequeue");
 	else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC)
-		printf(": SP/SC: ");
+		printf(" - SP/SC");
 	else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC)
-		printf(": MP/MC: ");
+		printf(" - MP/MC");
 
 	if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE)
-		printf("single: ");
+		printf(" - single - ");
 	else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK)
-		printf("bulk (size: %u): ", bsz);
+		printf(" - bulk (n:%-3u) - ", bsz);
 	else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST)
-		printf("burst (size: %u): ", bsz);
+		printf(" - burst (n:%-3u) - ", bsz);
+	else if ((api_type & (TEST_RING_ELEM_BURST_ZC |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0)
+		printf(" - burst zero copy (n:%-3u) - ", bsz);
 
-	printf("%.2F\n", value);
+	printf("cycles per elem: %.3F\n", value);
 }
 
 /**** Functions to analyse our core mask to get cores for different tests ***/
@@ -144,28 +148,33 @@ static void
 test_empty_dequeue(struct rte_ring *r, const int esize,
 			const unsigned int api_type)
 {
-	const unsigned int iter_shift = 26;
+	const unsigned int iter_shift = 30;
 	const unsigned int iterations = 1 << iter_shift;
 	unsigned int i = 0;
 	void *burst[MAX_BURST];
 
+	const unsigned int bulk_iterations = iterations / bulk_sizes[0];
 	const uint64_t start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	for (i = 0; i < bulk_iterations; i++)
 		test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type);
 	const uint64_t end = rte_rdtsc();
 
 	test_ring_print_test_string(api_type, esize, bulk_sizes[0],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 }
 
-/*
- * for the separate enqueue and dequeue threads they take in one param
- * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc
- */
-struct thread_params {
+/* describes the ring used by the enqueue and dequeue thread */
+struct ring_params {
 	struct rte_ring *r;
-	unsigned size;        /* input value, the burst size */
-	double spsc, mpmc;    /* output value, the single or multi timings */
+	unsigned int elem_size;
+	unsigned int bulk_sizes_i; /* index into bulk_size array */
+	unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */
+};
+
+/* Used do specify enqueue and dequeue ring operations and their results */
+struct thread_params {
+	struct ring_params *ring_params;
+	double *results; /* result array size must be equal to bulk_sizes */
 };
 
 /*
@@ -174,16 +183,15 @@ struct thread_params {
  * flag == 1 -> dequeue
  */
 static __rte_always_inline int
-enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
-	struct thread_params *p)
+enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 15;
+	const unsigned int iter_shift = 30;
 	const unsigned int iterations = 1 << iter_shift;
-	struct rte_ring *r = p->r;
-	unsigned int bsize = p->size;
 	unsigned int i;
 	void *burst = NULL;
+	unsigned int n_remaining;
+	const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i];
 
 #ifdef RTE_USE_C11_MEM_MODEL
 	if (__atomic_fetch_add(&lcore_count, 1, __ATOMIC_RELAXED) + 1 != 2)
@@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 		while(lcore_count != 2)
 			rte_pause();
 
-	burst = test_ring_calloc(MAX_BURST, esize);
+	burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size);
 	if (burst == NULL)
 		return -1;
 
 	const uint64_t sp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	const unsigned int bulk_iterations = iterations / bulk_n;
+	for (i = 0; i < bulk_iterations; i++) {
+		n_remaining = bulk_n;
 		do {
 			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_enqueue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_dequeue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			if (ret == 0)
 				rte_pause();
-		} while (!ret);
+			else
+				n_remaining -= ret;
+		} while (n_remaining > 0);
+	}
 	const uint64_t sp_end = rte_rdtsc();
 
-	const uint64_t mp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
-		do {
-			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			if (ret == 0)
-				rte_pause();
-		} while (!ret);
-	const uint64_t mp_end = rte_rdtsc();
+	p->results[p->ring_params->bulk_sizes_i] =
+			((double)sp_end - sp_start) / iterations;
 
-	p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize);
-	p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize);
 	return 0;
 }
 
@@ -243,15 +245,7 @@ enqueue_bulk(void *p)
 {
 	struct thread_params *params = p;
 
-	return enqueue_dequeue_bulk_helper(0, -1, params);
-}
-
-static int
-enqueue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(0, 16, params);
+	return enqueue_dequeue_bulk_helper(0, params);
 }
 
 /*
@@ -263,15 +257,7 @@ dequeue_bulk(void *p)
 {
 	struct thread_params *params = p;
 
-	return enqueue_dequeue_bulk_helper(1, -1, params);
-}
-
-static int
-dequeue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(1, 16, params);
+	return enqueue_dequeue_bulk_helper(1, params);
 }
 
 /*
@@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p)
  * used to measure ring perf between hyperthreads, cores and sockets.
  */
 static int
-run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize)
+run_on_core_pair(struct lcore_pair *cores,
+		struct thread_params *param1, struct thread_params *param2)
 {
-	lcore_function_t *f1, *f2;
-	struct thread_params param1 = {0}, param2 = {0};
 	unsigned i;
-
-	if (esize == -1) {
-		f1 = enqueue_bulk;
-		f2 = dequeue_bulk;
-	} else {
-		f1 = enqueue_bulk_16B;
-		f2 = dequeue_bulk_16B;
-	}
+	struct ring_params *ring_params = param1->ring_params;
 
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		lcore_count = 0;
-		param1.size = param2.size = bulk_sizes[i];
-		param1.r = param2.r = r;
+		ring_params->bulk_sizes_i = i;
 		if (cores->c1 == rte_get_main_lcore()) {
-			rte_eal_remote_launch(f2, &param2, cores->c2);
-			f1(&param1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
+			enqueue_bulk(param1);
 			rte_eal_wait_lcore(cores->c2);
 		} else {
-			rte_eal_remote_launch(f1, &param1, cores->c1);
-			rte_eal_remote_launch(f2, &param2, cores->c2);
+			rte_eal_remote_launch(enqueue_bulk, param1, cores->c1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
 			if (rte_eal_wait_lcore(cores->c1) < 0)
 				return -1;
 			if (rte_eal_wait_lcore(cores->c2) < 0)
 				return -1;
 		}
 		test_ring_print_test_string(
-			TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.spsc + param2.spsc);
-		test_ring_print_test_string(
-			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.mpmc + param2.mpmc);
+				ring_params->ring_flags,
+				ring_params->elem_size,
+				bulk_sizes[i],
+				param1->results[i] + param2->results[i]);
 	}
 
 	return 0;
@@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 	uint64_t hz = rte_get_timer_hz();
 	uint64_t lcount = 0;
 	const unsigned int lcore = rte_lcore_id();
-	struct thread_params *params = p;
+	struct ring_params *ring_params = p->ring_params;
 	void *burst = NULL;
 
 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -346,9 +322,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 
 	begin = rte_get_timer_cycles();
 	while (time_diff < hz * TIME_MS / 1000) {
-		test_ring_enqueue(params->r, burst, esize, params->size,
+		test_ring_enqueue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
-		test_ring_dequeue(params->r, burst, esize, params->size,
+		test_ring_dequeue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 		lcount++;
 		time_diff = rte_get_timer_cycles() - begin;
@@ -380,7 +358,8 @@ static int
 run_on_all_cores(struct rte_ring *r, const int esize)
 {
 	uint64_t total;
-	struct thread_params param;
+	struct ring_params ring_params = {0};
+	struct thread_params params = { .ring_params = &ring_params };
 	lcore_function_t *lcore_f;
 	unsigned int i, c;
 
@@ -389,21 +368,20 @@ run_on_all_cores(struct rte_ring *r, const int esize)
 	else
 		lcore_f = load_loop_fn_16B;
 
-	memset(&param, 0, sizeof(struct thread_params));
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		total = 0;
 		printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]);
-		param.size = bulk_sizes[i];
-		param.r = r;
+		params.ring_params->bulk_sizes_i = i;
+		params.ring_params->r = r;
 
 		/* clear synchro and start workers */
 		__atomic_store_n(&synchro, 0, __ATOMIC_RELAXED);
-		if (rte_eal_mp_remote_launch(lcore_f, &param, SKIP_MAIN) < 0)
+		if (rte_eal_mp_remote_launch(lcore_f, &params, SKIP_MAIN) < 0)
 			return -1;
 
 		/* start synchro and launch test on main */
 		__atomic_store_n(&synchro, 1, __ATOMIC_RELAXED);
-		lcore_f(&param);
+		lcore_f(&params);
 
 		rte_eal_mp_wait_lcore();
 
@@ -462,9 +440,9 @@ static int
 test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	const unsigned int api_type)
 {
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 27;
 	const unsigned int iterations = 1 << iter_shift;
-	unsigned int sz, i = 0;
+	unsigned int sz, i;
 	void **burst = NULL;
 
 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -472,17 +450,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 		return -1;
 
 	for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) {
+		const unsigned int n = iterations / bulk_sizes[sz];
 		const uint64_t start = rte_rdtsc();
-		for (i = 0; i < iterations; i++) {
+		for (i = 0; i < n; i++) {
 			test_ring_enqueue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 			test_ring_dequeue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 		}
 		const uint64_t end = rte_rdtsc();
 
 		test_ring_print_test_string(api_type, esize, bulk_sizes[sz],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 	}
 
 	rte_free(burst);
@@ -490,12 +469,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	return 0;
 }
 
+static __rte_always_inline int
+test_ring_perf_esize_run_on_two_cores(
+		struct thread_params *param1, struct thread_params *param2)
+{
+	struct lcore_pair cores;
+
+	if (get_two_hyperthreads(&cores) == 0) {
+		printf("\n### Testing using two hyperthreads ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing using two physical cores ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_sockets(&cores) == 0) {
+		printf("\n### Testing using two NUMA nodes ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	return 0;
+}
+
 /* Run all tests for a given element size */
 static __rte_always_inline int
 test_ring_perf_esize(const int esize)
 {
-	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
+	double results_enq[RTE_DIM(bulk_sizes)];
+	double results_deq[RTE_DIM(bulk_sizes)];
+	struct ring_params ring_params = {
+			.elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results_enq };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results_deq };
 
 	/*
 	 * Performance test for legacy/_elem APIs
@@ -535,22 +545,13 @@ test_ring_perf_esize(const int esize)
 	test_empty_dequeue(r, esize,
 			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 
-	if (get_two_hyperthreads(&cores) == 0) {
-		printf("\n### Testing using two hyperthreads ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.r = r;
 
-	if (get_two_cores(&cores) == 0) {
-		printf("\n### Testing using two physical cores ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
-	if (get_two_sockets(&cores) == 0) {
-		printf("\n### Testing using two NUMA nodes ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
+
+	ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
 
 	printf("\n### Testing using all worker nodes ###\n");
 	if (run_on_all_cores(r, esize) < 0)
@@ -566,6 +567,79 @@ test_ring_perf_esize(const int esize)
 	return -1;
 }
 
+
+static __rte_always_inline int
+test_ring_perf_compression(void)
+{
+	double results1[RTE_DIM(bulk_sizes)];
+	double results2[RTE_DIM(bulk_sizes)];
+	double results1_comp[RTE_DIM(bulk_sizes)];
+	double results2_comp[RTE_DIM(bulk_sizes)];
+
+	struct lcore_pair cores;
+	int ret = -1;
+	unsigned int i;
+	struct ring_params ring_params = { .elem_size = sizeof(void *) };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results1 };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results2 };
+
+	printf("\n### Testing compression gain ###");
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(void *),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy ###\n");
+		ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME_COMPRESS, sizeof(uint32_t),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	/* swap out the result arrays for the compression ones */
+	param1.results = results1_comp;
+	param2.results = results2_comp;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	printf("\n\n### Potential gain from compression ###\n");
+	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
+		const double result = results1[i] + results2[i];
+		const double result_comp = results1_comp[i] + results1_comp[i];
+		const double gain = 100 - (result_comp / result) * 100;
+
+		printf("Gain of %5.1F%% for burst of %-3u elems\n",
+				gain, bulk_sizes[i]);
+	}
+
+	return ret;
+}
+
 static int
 test_ring_perf(void)
 {
@@ -576,6 +650,10 @@ test_ring_perf(void)
 	if (test_ring_perf_esize(16) == -1)
 		return -1;
 
+	/* Test for performance gain of compression */
+	if (test_ring_perf_compression() == -1)
+		return -1;
+
 	return 0;
 }
 
-- 
2.25.1

Re: [RFC 2/2] test: add pointer compress tests to ring perf test

[Thomas Monjalon]

27/09/2023 17:08, Paul Szczepanek:
> Add a test that runs a zero copy burst enqueue and dequeue on a ring
> of raw pointers and compressed pointers at different burst sizes to
> showcase performance benefits of newly added pointer compression APIs
> 
> To reuse existing code, some refactoring was done to pass more
> parameters to test threads. Additionally more bulk sizes were
> added to showcase their effects on compression. To keep runtime
> reasoanble iterations where adjusted to take into account bulk sizes.

typo: reasonable

How much time does it take before/after?

[PATCH v14 0/6] add pointer compression API

[Paul Szczepanek]

This patchset is proposing adding a new header only library
with utility functions that allow compression of arrays of pointers.

Since this is a header only library a patch needed to be added to amend
the build system to allow adding libraries without source files.

When passing caches full of pointers between threads, memory containing
the pointers is copied multiple times which is especially costly between
cores. A compression method will allow us to shrink the memory size
copied.

The compression takes advantage of the fact that pointers are usually
located in a limited memory region. We can compress them by converting them
to offsets from a base memory address.

Offsets can be stored in fewer bytes (dictated by the memory region size
and alignment of the pointer). For example: an 8 byte aligned pointer
which is part of a 32GB memory pool can be stored in 4 bytes. The API is
very generic and does not assume mempool pointers, any pointer can be
passed in.

Compression is based on few and fast operations and especially with vector
instructions leveraged creates minimal overhead.

The API accepts and returns arrays because the overhead means it only is
worth it when done in bulk.

Test is added that shows potential performance gain from compression. In
this test an array of pointers is passed through a ring between two cores.
It shows the gain which is dependent on the bulk operation size. In this
synthetic test run on ampere altra a substantial (up to 25%) performance
gain is seen if done in bulk size larger than 32. At 32 it breaks even and
lower sizes create a small (less than 5%) slowdown due to overhead.

In a more realistic mock application running the l3 forwarding dpdk
example that works in pipeline mode on two cores this translated into a
~5% throughput increase on an ampere altra.

v2:
* addressed review comments (style, explanations and typos)
* lowered bulk iterations closer to original numbers to keep runtime short
* fixed pointer size warning on 32-bit arch
v3:
* added 16-bit versions of compression functions and tests
* added documentation of these new utility functions in the EAL guide
v4:
* added unit test
* fix bug in NEON implementation of 32-bit decompress
v5:
* disable NEON and SVE implementation on AARCH32 due to wrong pointer size
v6:
* added example usage to commit message of the initial commit
v7:
* rebase to remove clashing mailmap changes
v8:
* put ptr compress into its own library
* add depends-on tag
* remove copyright bumps
* typos
v9
* added MAINTAINERS entries, release notes, doc indexes etc.
* added patch for build system to allow header only library
v10
* fixed problem with meson build adding shared deps to static deps
v11
* added mempool functions to get information about memory range and
alignment
* added tests for the new mempool functions
* added macros to help find the parameters for compression functions
* minor improvement in the SVE compression code
* amended documentation to reflect these changes
v12
* added doxygen and prefixes to macros
* use rte_bitops for clz and ctz
* added unit tests to verify macros
* fixed incorrect letter case in docs
v13
* added contiguous parameter to rte_mempool_get_mem_range
* made rte_mempool_get_mem_range parameters optional
v14
* encapsulated parameters of rte_mempool_get_mem_range in a struct
* added consts to function parameters where appropriate

Paul Szczepanek (6):
  lib: allow libraries with no sources
  mempool: add functions to get extra mempool info
  ptr_compress: add pointer compression library
  test: add pointer compress tests to ring perf test
  docs: add pointer compression guide
  test: add unit test for ptr compression

 MAINTAINERS                                |   6 +
 app/test/meson.build                       |  21 +-
 app/test/test_mempool.c                    |  70 ++++
 app/test/test_ptr_compress.c               | 193 +++++++++++
 app/test/test_ring.h                       |  94 ++++++
 app/test/test_ring_perf.c                  | 352 ++++++++++++++-------
 doc/api/doxy-api-index.md                  |   1 +
 doc/api/doxy-api.conf.in                   |   1 +
 doc/guides/prog_guide/index.rst            |   1 +
 doc/guides/prog_guide/ptr_compress_lib.rst | 160 ++++++++++
 doc/guides/rel_notes/release_24_07.rst     |   5 +
 lib/mempool/rte_mempool.c                  |  45 +++
 lib/mempool/rte_mempool.h                  |  47 +++
 lib/mempool/version.map                    |   3 +
 lib/meson.build                            |  17 +
 lib/ptr_compress/meson.build               |   4 +
 lib/ptr_compress/rte_ptr_compress.h        | 324 +++++++++++++++++++
 17 files changed, 1212 insertions(+), 132 deletions(-)
 create mode 100644 app/test/test_ptr_compress.c
 create mode 100644 doc/guides/prog_guide/ptr_compress_lib.rst
 create mode 100644 lib/ptr_compress/meson.build
 create mode 100644 lib/ptr_compress/rte_ptr_compress.h

--
2.25.1

[PATCH v14 1/6] lib: allow libraries with no sources

[Paul Szczepanek]

Allow header only libraries.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
 lib/meson.build | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/lib/meson.build b/lib/meson.build
index 179a272932..7c90602bf5 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -222,6 +222,22 @@ foreach l:libraries
     includes += include_directories(l)
     dpdk_includes += include_directories(l)

+    # special case for header only libraries
+    if sources.length() == 0
+        shared_dep = declare_dependency(include_directories: includes,
+                dependencies: shared_deps)
+        static_dep = declare_dependency(include_directories: includes,
+                dependencies: static_deps)
+        set_variable('shared_rte_' + name, shared_dep)
+        set_variable('static_rte_' + name, static_dep)
+        dpdk_shared_lib_deps += shared_dep
+        dpdk_static_lib_deps += static_dep
+        if developer_mode
+            message('lib/@0@: Defining dependency "@1@"'.format(l, name))
+        endif
+        continue
+    endif
+
     if developer_mode and is_windows and use_function_versioning
         message('@0@: Function versioning is not supported by Windows.'.format(name))
     endif
--
2.25.1

[PATCH v14 2/6] mempool: add functions to get extra mempool info

[Paul Szczepanek]

Add two functions:
- rte_mempool_get_mem_range - get virtual memory range
of the objects in the mempool,
- rte_mempool_get_obj_alignment - get alignment of
objects in the mempool.

Add two tests that test these new functions.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Morten Brørup <mb@smartsharesystems.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
---
 app/test/test_mempool.c   | 70 +++++++++++++++++++++++++++++++++++++++
 lib/mempool/rte_mempool.c | 45 +++++++++++++++++++++++++
 lib/mempool/rte_mempool.h | 47 ++++++++++++++++++++++++++
 lib/mempool/version.map   |  3 ++
 4 files changed, 165 insertions(+)

diff --git a/app/test/test_mempool.c b/app/test/test_mempool.c
index ad7ebd6363..3f7ba5872d 100644
--- a/app/test/test_mempool.c
+++ b/app/test/test_mempool.c
@@ -843,16 +843,19 @@ test_mempool(void)
 	int ret = -1;
 	uint32_t nb_objs = 0;
 	uint32_t nb_mem_chunks = 0;
+	size_t alignment = 0;
 	struct rte_mempool *mp_cache = NULL;
 	struct rte_mempool *mp_nocache = NULL;
 	struct rte_mempool *mp_stack_anon = NULL;
 	struct rte_mempool *mp_stack_mempool_iter = NULL;
 	struct rte_mempool *mp_stack = NULL;
 	struct rte_mempool *default_pool = NULL;
+	struct rte_mempool *mp_alignment = NULL;
 	struct mp_data cb_arg = {
 		.ret = -1
 	};
 	const char *default_pool_ops = rte_mbuf_best_mempool_ops();
+	struct rte_mempool_mem_range_info mem_range = { 0 };

 	/* create a mempool (without cache) */
 	mp_nocache = rte_mempool_create("test_nocache", MEMPOOL_SIZE,
@@ -967,6 +970,72 @@ test_mempool(void)
 	}
 	rte_mempool_obj_iter(default_pool, my_obj_init, NULL);

+	if (rte_mempool_get_mem_range(default_pool, &mem_range)) {
+		printf("cannot get mem range from default mempool\n");
+		GOTO_ERR(ret, err);
+	}
+
+	if (rte_mempool_get_mem_range(NULL, NULL) != -EINVAL) {
+		printf("rte_mempool_get_mem_range failed to return -EINVAL "
+				"when passed invalid arguments\n");
+		GOTO_ERR(ret, err);
+	}
+
+	if (mem_range.start == NULL || mem_range.length <
+			(MEMPOOL_SIZE * MEMPOOL_ELT_SIZE)) {
+		printf("mem range of default mempool is invalid\n");
+		GOTO_ERR(ret, err);
+	}
+
+	/* by default mempool objects are aligned by RTE_MEMPOOL_ALIGN */
+	alignment = rte_mempool_get_obj_alignment(default_pool);
+	if (alignment != RTE_MEMPOOL_ALIGN) {
+		printf("rte_mempool_get_obj_alignment returned wrong value, "
+				"expected %zu, returned %zu\n",
+				(size_t)RTE_MEMPOOL_ALIGN, alignment);
+		GOTO_ERR(ret, err);
+	}
+
+	/* create a mempool with a RTE_MEMPOOL_F_NO_CACHE_ALIGN flag */
+	mp_alignment = rte_mempool_create("test_alignment",
+		1, 8, /* the small size guarantees single memory chunk */
+		0, 0, NULL, NULL, my_obj_init, NULL,
+		SOCKET_ID_ANY, RTE_MEMPOOL_F_NO_CACHE_ALIGN);
+
+	if (mp_alignment == NULL) {
+		printf("cannot allocate mempool with "
+				"RTE_MEMPOOL_F_NO_CACHE_ALIGN flag\n");
+		GOTO_ERR(ret, err);
+	}
+
+	/* mempool was created with RTE_MEMPOOL_F_NO_CACHE_ALIGN
+	 * and minimum alignment is expected which is sizeof(uint64_t)
+	 */
+	alignment = rte_mempool_get_obj_alignment(mp_alignment);
+	if (alignment != sizeof(uint64_t)) {
+		printf("rte_mempool_get_obj_alignment returned wrong value, "
+				"expected %zu, returned %zu\n",
+				(size_t)sizeof(uint64_t), alignment);
+		GOTO_ERR(ret, err);
+	}
+
+	alignment = rte_mempool_get_obj_alignment(NULL);
+	if (alignment != 0) {
+		printf("rte_mempool_get_obj_alignment failed to return 0 for "
+				" an invalid mempool\n");
+		GOTO_ERR(ret, err);
+	}
+
+	if (rte_mempool_get_mem_range(mp_alignment, &mem_range)) {
+		printf("cannot get mem range from mempool\n");
+		GOTO_ERR(ret, err);
+	}
+
+	if (!mem_range.is_contiguous) {
+		printf("mempool not contiguous\n");
+		GOTO_ERR(ret, err);
+	}
+
 	/* retrieve the mempool from its name */
 	if (rte_mempool_lookup("test_nocache") != mp_nocache) {
 		printf("Cannot lookup mempool from its name\n");
@@ -1039,6 +1108,7 @@ test_mempool(void)
 	rte_mempool_free(mp_stack_mempool_iter);
 	rte_mempool_free(mp_stack);
 	rte_mempool_free(default_pool);
+	rte_mempool_free(mp_alignment);

 	return ret;
 }
diff --git a/lib/mempool/rte_mempool.c b/lib/mempool/rte_mempool.c
index 12390a2c81..54d2804151 100644
--- a/lib/mempool/rte_mempool.c
+++ b/lib/mempool/rte_mempool.c
@@ -1386,6 +1386,51 @@ void rte_mempool_walk(void (*func)(struct rte_mempool *, void *),
 	rte_mcfg_mempool_read_unlock();
 }

+int rte_mempool_get_mem_range(const struct rte_mempool *mp,
+		struct rte_mempool_mem_range_info *mem_range)
+{
+	if (mp == NULL || mem_range == NULL)
+		return -EINVAL;
+
+	void *address_low = (void *)UINTPTR_MAX;
+	void *address_high = 0;
+	size_t address_diff = 0;
+	size_t total_size = 0;
+	struct rte_mempool_memhdr *hdr;
+
+	/* go through memory chunks and find the lowest and highest addresses */
+	STAILQ_FOREACH(hdr, &mp->mem_list, next) {
+		if (address_low > hdr->addr)
+			address_low = hdr->addr;
+		if (address_high < RTE_PTR_ADD(hdr->addr, hdr->len))
+			address_high = RTE_PTR_ADD(hdr->addr, hdr->len);
+		total_size += hdr->len;
+	}
+
+	/* check if mempool was not populated yet (no memory chunks) */
+	if (address_low == (void *)UINTPTR_MAX)
+		return -EINVAL;
+
+	address_diff = (size_t)RTE_PTR_DIFF(address_high, address_low);
+
+	mem_range->start = address_low;
+	mem_range->length = address_diff;
+	mem_range->is_contiguous = (total_size == address_diff) ? true : false;
+
+	return 0;
+}
+
+size_t rte_mempool_get_obj_alignment(const struct rte_mempool *mp)
+{
+	if (mp == NULL)
+		return 0;
+
+	if (mp->flags & RTE_MEMPOOL_F_NO_CACHE_ALIGN)
+		return sizeof(uint64_t);
+	else
+		return RTE_MEMPOOL_ALIGN;
+}
+
 struct mempool_callback_data {
 	TAILQ_ENTRY(mempool_callback_data) callbacks;
 	rte_mempool_event_callback *func;
diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h
index 23fd5c8465..990491a7a7 100644
--- a/lib/mempool/rte_mempool.h
+++ b/lib/mempool/rte_mempool.h
@@ -1917,6 +1917,53 @@ uint32_t rte_mempool_calc_obj_size(uint32_t elt_size, uint32_t flags,
 void rte_mempool_walk(void (*func)(struct rte_mempool *, void *arg),
 		      void *arg);

+/**
+ * A structure used to retrieve information about the memory range
+ * of the mempool.
+ */
+struct rte_mempool_mem_range_info {
+	/** Start of the memory range used by mempool objects */
+	void *start;
+	/** Length of the memory range used by mempool objects */
+	size_t length;
+	/** Are all memory addresses used by mempool objects contiguous */
+	bool is_contiguous;
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Get information about the memory range used by the mempool.
+ *
+ * @param[in] mp
+ *   Pointer to an initialized mempool.
+ * @param[out] mem_range
+ *   Pointer to struct which is used to return lowest address,
+ *   length of the memory range containing all the addresses,
+ *   and whether these addresses are contiguous.
+ * @return
+ *   0 on success, -EINVAL if mempool is not valid or mem_range is NULL.
+ **/
+__rte_experimental
+int rte_mempool_get_mem_range(const struct rte_mempool *mp,
+		struct rte_mempool_mem_range_info *mem_range);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Return object alignment.
+ *
+ * @param[in] mp
+ *   Pointer to a mempool.
+ * @return
+ *   Object alignment if mp is valid. 0 if mp is NULL.
+ *
+ **/
+__rte_experimental
+size_t rte_mempool_get_obj_alignment(const struct rte_mempool *mp);
+
 /**
  * @internal Get page size used for mempool object allocation.
  * This function is internal to mempool library and mempool drivers.
diff --git a/lib/mempool/version.map b/lib/mempool/version.map
index 473277400c..02df634b2a 100644
--- a/lib/mempool/version.map
+++ b/lib/mempool/version.map
@@ -50,6 +50,9 @@ EXPERIMENTAL {
 	__rte_mempool_trace_get_contig_blocks;
 	__rte_mempool_trace_default_cache;
 	__rte_mempool_trace_cache_flush;
+	# added in 24.07
+	rte_mempool_get_mem_range;
+	rte_mempool_get_obj_alignment;
 };

 INTERNAL {
--
2.25.1

RE: [PATCH v14 2/6] mempool: add functions to get extra mempool info

[Konstantin Ananyev]

> Add two functions:
> - rte_mempool_get_mem_range - get virtual memory range
> of the objects in the mempool,
> - rte_mempool_get_obj_alignment - get alignment of
> objects in the mempool.
> 
> Add two tests that test these new functions.
> 
> Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
> Reviewed-by: Nathan Brown <nathan.brown@arm.com>
> Reviewed-by: Morten Brørup <mb@smartsharesystems.com>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
> ---
>  app/test/test_mempool.c   | 70 +++++++++++++++++++++++++++++++++++++++
>  lib/mempool/rte_mempool.c | 45 +++++++++++++++++++++++++
>  lib/mempool/rte_mempool.h | 47 ++++++++++++++++++++++++++
>  lib/mempool/version.map   |  3 ++
>  4 files changed, 165 insertions(+)
> 
> diff --git a/app/test/test_mempool.c b/app/test/test_mempool.c
> index ad7ebd6363..3f7ba5872d 100644
> --- a/app/test/test_mempool.c
> +++ b/app/test/test_mempool.c
> @@ -843,16 +843,19 @@ test_mempool(void)
>  	int ret = -1;
>  	uint32_t nb_objs = 0;
>  	uint32_t nb_mem_chunks = 0;
> +	size_t alignment = 0;
>  	struct rte_mempool *mp_cache = NULL;
>  	struct rte_mempool *mp_nocache = NULL;
>  	struct rte_mempool *mp_stack_anon = NULL;
>  	struct rte_mempool *mp_stack_mempool_iter = NULL;
>  	struct rte_mempool *mp_stack = NULL;
>  	struct rte_mempool *default_pool = NULL;
> +	struct rte_mempool *mp_alignment = NULL;
>  	struct mp_data cb_arg = {
>  		.ret = -1
>  	};
>  	const char *default_pool_ops = rte_mbuf_best_mempool_ops();
> +	struct rte_mempool_mem_range_info mem_range = { 0 };
> 
>  	/* create a mempool (without cache) */
>  	mp_nocache = rte_mempool_create("test_nocache", MEMPOOL_SIZE,
> @@ -967,6 +970,72 @@ test_mempool(void)
>  	}
>  	rte_mempool_obj_iter(default_pool, my_obj_init, NULL);
> 
> +	if (rte_mempool_get_mem_range(default_pool, &mem_range)) {
> +		printf("cannot get mem range from default mempool\n");
> +		GOTO_ERR(ret, err);
> +	}
> +
> +	if (rte_mempool_get_mem_range(NULL, NULL) != -EINVAL) {
> +		printf("rte_mempool_get_mem_range failed to return -EINVAL "
> +				"when passed invalid arguments\n");
> +		GOTO_ERR(ret, err);
> +	}
> +
> +	if (mem_range.start == NULL || mem_range.length <
> +			(MEMPOOL_SIZE * MEMPOOL_ELT_SIZE)) {
> +		printf("mem range of default mempool is invalid\n");
> +		GOTO_ERR(ret, err);
> +	}
> +
> +	/* by default mempool objects are aligned by RTE_MEMPOOL_ALIGN */
> +	alignment = rte_mempool_get_obj_alignment(default_pool);
> +	if (alignment != RTE_MEMPOOL_ALIGN) {
> +		printf("rte_mempool_get_obj_alignment returned wrong value, "
> +				"expected %zu, returned %zu\n",
> +				(size_t)RTE_MEMPOOL_ALIGN, alignment);
> +		GOTO_ERR(ret, err);
> +	}
> +
> +	/* create a mempool with a RTE_MEMPOOL_F_NO_CACHE_ALIGN flag */
> +	mp_alignment = rte_mempool_create("test_alignment",
> +		1, 8, /* the small size guarantees single memory chunk */
> +		0, 0, NULL, NULL, my_obj_init, NULL,
> +		SOCKET_ID_ANY, RTE_MEMPOOL_F_NO_CACHE_ALIGN);
> +
> +	if (mp_alignment == NULL) {
> +		printf("cannot allocate mempool with "
> +				"RTE_MEMPOOL_F_NO_CACHE_ALIGN flag\n");
> +		GOTO_ERR(ret, err);
> +	}
> +
> +	/* mempool was created with RTE_MEMPOOL_F_NO_CACHE_ALIGN
> +	 * and minimum alignment is expected which is sizeof(uint64_t)
> +	 */
> +	alignment = rte_mempool_get_obj_alignment(mp_alignment);
> +	if (alignment != sizeof(uint64_t)) {
> +		printf("rte_mempool_get_obj_alignment returned wrong value, "
> +				"expected %zu, returned %zu\n",
> +				(size_t)sizeof(uint64_t), alignment);
> +		GOTO_ERR(ret, err);
> +	}
> +
> +	alignment = rte_mempool_get_obj_alignment(NULL);
> +	if (alignment != 0) {
> +		printf("rte_mempool_get_obj_alignment failed to return 0 for "
> +				" an invalid mempool\n");
> +		GOTO_ERR(ret, err);
> +	}
> +
> +	if (rte_mempool_get_mem_range(mp_alignment, &mem_range)) {
> +		printf("cannot get mem range from mempool\n");
> +		GOTO_ERR(ret, err);
> +	}
> +
> +	if (!mem_range.is_contiguous) {
> +		printf("mempool not contiguous\n");
> +		GOTO_ERR(ret, err);
> +	}
> +
>  	/* retrieve the mempool from its name */
>  	if (rte_mempool_lookup("test_nocache") != mp_nocache) {
>  		printf("Cannot lookup mempool from its name\n");
> @@ -1039,6 +1108,7 @@ test_mempool(void)
>  	rte_mempool_free(mp_stack_mempool_iter);
>  	rte_mempool_free(mp_stack);
>  	rte_mempool_free(default_pool);
> +	rte_mempool_free(mp_alignment);
> 
>  	return ret;
>  }
> diff --git a/lib/mempool/rte_mempool.c b/lib/mempool/rte_mempool.c
> index 12390a2c81..54d2804151 100644
> --- a/lib/mempool/rte_mempool.c
> +++ b/lib/mempool/rte_mempool.c
> @@ -1386,6 +1386,51 @@ void rte_mempool_walk(void (*func)(struct rte_mempool *, void *),
>  	rte_mcfg_mempool_read_unlock();
>  }
> 
> +int rte_mempool_get_mem_range(const struct rte_mempool *mp,
> +		struct rte_mempool_mem_range_info *mem_range)
> +{

To follow DPDK coding style guide:
int
rte_mempool_get_mem_range(...
same for the second function. 
 

> +	if (mp == NULL || mem_range == NULL)
> +		return -EINVAL;

Again, to follow DPDK coding style:
Variables should be declared at the start of a block of code rather than in the middle.
I.E - move that input params check after local vars definition block. 

With that changes in place:
Acked-by: Konstantin Ananyev <konstantin.ananyev@huawei.com>

> +	void *address_low = (void *)UINTPTR_MAX;
> +	void *address_high = 0;
> +	size_t address_diff = 0;
> +	size_t total_size = 0;
> +	struct rte_mempool_memhdr *hdr;
> +
> +	/* go through memory chunks and find the lowest and highest addresses */
> +	STAILQ_FOREACH(hdr, &mp->mem_list, next) {
> +		if (address_low > hdr->addr)
> +			address_low = hdr->addr;
> +		if (address_high < RTE_PTR_ADD(hdr->addr, hdr->len))
> +			address_high = RTE_PTR_ADD(hdr->addr, hdr->len);
> +		total_size += hdr->len;
> +	}
> +
> +	/* check if mempool was not populated yet (no memory chunks) */
> +	if (address_low == (void *)UINTPTR_MAX)
> +		return -EINVAL;
> +
> +	address_diff = (size_t)RTE_PTR_DIFF(address_high, address_low);
> +
> +	mem_range->start = address_low;
> +	mem_range->length = address_diff;
> +	mem_range->is_contiguous = (total_size == address_diff) ? true : false;
> +
> +	return 0;
> +}
> +
> +size_t rte_mempool_get_obj_alignment(const struct rte_mempool *mp)
> +{
> +	if (mp == NULL)
> +		return 0;

In case of mp==NULL, would it be better to return negative error code (-EINVAL or so)?
In that case the function should be changed to return ssize_t though.

> +
> +	if (mp->flags & RTE_MEMPOOL_F_NO_CACHE_ALIGN)
> +		return sizeof(uint64_t);
> +	else
> +		return RTE_MEMPOOL_ALIGN;
> +}
> +
>  struct mempool_callback_data {
>  	TAILQ_ENTRY(mempool_callback_data) callbacks;
>  	rte_mempool_event_callback *func;
> diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h
> index 23fd5c8465..990491a7a7 100644
> --- a/lib/mempool/rte_mempool.h
> +++ b/lib/mempool/rte_mempool.h
> @@ -1917,6 +1917,53 @@ uint32_t rte_mempool_calc_obj_size(uint32_t elt_size, uint32_t flags,
>  void rte_mempool_walk(void (*func)(struct rte_mempool *, void *arg),
>  		      void *arg);
> 
> +/**
> + * A structure used to retrieve information about the memory range
> + * of the mempool.
> + */
> +struct rte_mempool_mem_range_info {
> +	/** Start of the memory range used by mempool objects */
> +	void *start;
> +	/** Length of the memory range used by mempool objects */
> +	size_t length;
> +	/** Are all memory addresses used by mempool objects contiguous */
> +	bool is_contiguous;
> +};
> +
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice.
> + *
> + * Get information about the memory range used by the mempool.
> + *
> + * @param[in] mp
> + *   Pointer to an initialized mempool.
> + * @param[out] mem_range
> + *   Pointer to struct which is used to return lowest address,
> + *   length of the memory range containing all the addresses,
> + *   and whether these addresses are contiguous.
> + * @return
> + *   0 on success, -EINVAL if mempool is not valid or mem_range is NULL.
> + **/
> +__rte_experimental
> +int rte_mempool_get_mem_range(const struct rte_mempool *mp,
> +		struct rte_mempool_mem_range_info *mem_range);
> +
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice.
> + *
> + * Return object alignment.
> + *
> + * @param[in] mp
> + *   Pointer to a mempool.
> + * @return
> + *   Object alignment if mp is valid. 0 if mp is NULL.
> + *
> + **/
> +__rte_experimental
> +size_t rte_mempool_get_obj_alignment(const struct rte_mempool *mp);
> +
>  /**
>   * @internal Get page size used for mempool object allocation.
>   * This function is internal to mempool library and mempool drivers.
> diff --git a/lib/mempool/version.map b/lib/mempool/version.map
> index 473277400c..02df634b2a 100644
> --- a/lib/mempool/version.map
> +++ b/lib/mempool/version.map
> @@ -50,6 +50,9 @@ EXPERIMENTAL {
>  	__rte_mempool_trace_get_contig_blocks;
>  	__rte_mempool_trace_default_cache;
>  	__rte_mempool_trace_cache_flush;
> +	# added in 24.07
> +	rte_mempool_get_mem_range;
> +	rte_mempool_get_obj_alignment;
>  };
> 
>  INTERNAL {
> --
> 2.25.1

Re: [PATCH v14 2/6] mempool: add functions to get extra mempool info

[Paul Szczepanek]

On 10/06/2024 15:24, Konstantin Ananyev wrote:
[snip]
> 
> With that changes in place:
> Acked-by: Konstantin Ananyev <konstantin.ananyev@huawei.com>
> 

I have applied your comments in v15.

[snip]

>> +
>> +size_t rte_mempool_get_obj_alignment(const struct rte_mempool *mp)
>> +{
>> +	if (mp == NULL)
>> +		return 0;
> 
> In case of mp==NULL, would it be better to return negative error code (-EINVAL or so)?
> In that case the function should be changed to return ssize_t though.
> 

I have chosen size_t for portability and ease of use as that is the type
most likely expected by the caller and because there are no other error
modes expected in the future that would require distinguishing between
them. I hope this is an acceptable solution and have kept it in v15.

[snip]

[PATCH v14 3/6] ptr_compress: add pointer compression library

[Paul Szczepanek]

Add a new utility header for compressing pointers. The provided
functions can store pointers as 32-bit or 16-bit offsets.

The compression takes advantage of the fact that pointers are
usually located in a limited memory region (like a mempool).
We can compress them by converting them to offsets from a base
memory address. Offsets can be stored in fewer bytes (dictated
by the memory region size and alignment of the pointer).
For example: an 8 byte aligned pointer which is part of a 32GB
memory pool can be stored in 4 bytes.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
---
 MAINTAINERS                            |   4 +
 doc/api/doxy-api-index.md              |   1 +
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/rel_notes/release_24_07.rst |   5 +
 lib/meson.build                        |   1 +
 lib/ptr_compress/meson.build           |   4 +
 lib/ptr_compress/rte_ptr_compress.h    | 324 +++++++++++++++++++++++++
 7 files changed, 340 insertions(+)
 create mode 100644 lib/ptr_compress/meson.build
 create mode 100644 lib/ptr_compress/rte_ptr_compress.h

diff --git a/MAINTAINERS b/MAINTAINERS
index c9adff9846..27b2f03e6c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1694,6 +1694,10 @@ M: Chenbo Xia <chenbox@nvidia.com>
 M: Gaetan Rivet <grive@u256.net>
 F: lib/pci/

+Pointer Compression
+M: Paul Szczepanek <paul.szczepanek@arm.com>
+F: lib/ptr_compress/
+
 Power management
 M: Anatoly Burakov <anatoly.burakov@intel.com>
 M: David Hunt <david.hunt@intel.com>
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 8c1eb8fafa..f9283154f8 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -222,6 +222,7 @@ The public API headers are grouped by topics:
   [config file](@ref rte_cfgfile.h),
   [key/value args](@ref rte_kvargs.h),
   [argument parsing](@ref rte_argparse.h),
+  [ptr_compress](@ref rte_ptr_compress.h),
   [string](@ref rte_string_fns.h),
   [thread](@ref rte_thread.h)

diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index 27afec8b3b..a8823c046f 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -71,6 +71,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/pipeline \
                           @TOPDIR@/lib/port \
                           @TOPDIR@/lib/power \
+                          @TOPDIR@/lib/ptr_compress \
                           @TOPDIR@/lib/rawdev \
                           @TOPDIR@/lib/rcu \
                           @TOPDIR@/lib/regexdev \
diff --git a/doc/guides/rel_notes/release_24_07.rst b/doc/guides/rel_notes/release_24_07.rst
index a69f24cf99..4711792e61 100644
--- a/doc/guides/rel_notes/release_24_07.rst
+++ b/doc/guides/rel_notes/release_24_07.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================

+* **Introduced pointer compression library.**
+
+  Library provides functions to compress and decompress arrays of pointers
+  which can improve application performance under certain conditions.
+  Performance test was added to help users evaluate performance on their setup.

 Removed Items
 -------------
diff --git a/lib/meson.build b/lib/meson.build
index 7c90602bf5..63becee142 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -14,6 +14,7 @@ libraries = [
         'argparse',
         'telemetry', # basic info querying
         'eal', # everything depends on eal
+        'ptr_compress',
         'ring',
         'rcu', # rcu depends on ring
         'mempool',
diff --git a/lib/ptr_compress/meson.build b/lib/ptr_compress/meson.build
new file mode 100644
index 0000000000..e92706a45f
--- /dev/null
+++ b/lib/ptr_compress/meson.build
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+headers = files('rte_ptr_compress.h')
diff --git a/lib/ptr_compress/rte_ptr_compress.h b/lib/ptr_compress/rte_ptr_compress.h
new file mode 100644
index 0000000000..bf9cfb0661
--- /dev/null
+++ b/lib/ptr_compress/rte_ptr_compress.h
@@ -0,0 +1,324 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2024 Arm Limited
+ */
+
+#ifndef RTE_PTR_COMPRESS_H
+#define RTE_PTR_COMPRESS_H
+
+/**
+ * @file
+ * Pointer compression and decompression functions.
+ *
+ * When passing arrays full of pointers between threads, memory containing
+ * the pointers is copied multiple times which is especially costly between
+ * cores. These functions allow us to compress the pointers.
+ *
+ * Compression takes advantage of the fact that pointers are usually located in
+ * a limited memory region. We compress them by converting them to offsets from
+ * a base memory address. Offsets can be stored in fewer bytes.
+ *
+ * The compression functions come in two varieties: 32-bit and 16-bit.
+ *
+ * To determine how many bits are needed to compress the pointer calculate
+ * the biggest offset possible (highest value pointer - base pointer)
+ * and shift the value right according to alignment (shift by exponent of the
+ * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by
+ * 3, etc.). The resulting value must fit in either 32 or 16 bits.
+ *
+ * For usage example and further explanation please see "Pointer Compression" in
+ * doc/guides/prog_guide/ptr_compress_lib.rst
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_vect.h>
+#include <rte_mempool.h>
+#include <rte_bitops.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Calculate how many bits are required to store a value within a given range.
+ * This is to help decide which pointer compression functions can be used to
+ * store pointers contained within a memory range.
+ *
+ * @param range
+ *   The size of the range the value belongs to.
+ * @return
+ *   Number of bits required to store a value.
+ **/
+#define RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(range) \
+	(((uint64_t)range) < 2 ? 1 : \
+		(sizeof(uint64_t) * CHAR_BIT - rte_clz64((uint64_t)range - 1)))
+
+/**
+ * Calculate how many bits in the address can be dropped without losing any
+ * information thanks to the alignment of the address.
+ *
+ * @param alignment
+ *   Memory alignment.
+ * @return
+ *   Size of shift allowed without dropping any information from the pointer.
+ **/
+#define RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(alignment) \
+	((alignment) == 0 ? 0 : rte_ctz64((uint64_t)alignment))
+
+/**
+ * Determine if rte_ptr_compress_16_shift can be used to compress pointers
+ * that contain addresses of memory objects whose memory is aligned by
+ * a given amount and contained in a given memory range.
+ *
+ * @param mem_range
+ *   The size of the memory region that contains the objects pointed to.
+ * @param obj_alignment
+ *   The alignment of objects pointed to.
+ * @return
+ *   1 if function can be used, 0 otherwise.
+ **/
+#define RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT(mem_range, obj_alignment) \
+	((RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(mem_range) - \
+	RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(obj_alignment)) <= 16 ? 1 : 0)
+
+/**
+ * Determine if rte_ptr_compress_32_shift can be used to compress pointers
+ * that contain addresses of memory objects whose memory is aligned by
+ * a given amount and contained in a given memory range.
+ *
+ * @param mem_range
+ *   The size of the memory region that contains the objects pointed to.
+ * @param obj_alignment
+ *   The alignment of objects pointed to.
+ * @return
+ *   1 if function can be used, 0 otherwise.
+ **/
+#define RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT(mem_range, obj_alignment) \
+	((RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(mem_range) - \
+	RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(obj_alignment)) <= 32 ? 1 : 0)
+
+/**
+ * Compress pointers into 32-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 32 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 32GB of the base pointer. If there is no such alignment guarantee they
+ * must be within 4GB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_32_shift(void *ptr_base, void * const *src_table,
+		uint32_t *dest_table, size_t n, uint8_t bit_shift)
+{
+	size_t i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	do {
+		svbool_t pg = svwhilelt_b64(i, n);
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1w(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+	} while (i < n);
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	/* right shift is done by left shifting by negative int */
+	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	const size_t n_even = n & ~0x1;
+	for (; i < n_even; i += 2) {
+		v_ptr_table = vld1q_u64((const uint64_t *)src_table + i);
+		v_ptr_table = vsubq_u64(v_ptr_table, v_ptr_base);
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		vst1_u32(dest_table + i, vqmovn_u64(v_ptr_table));
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT32_MAX);
+		dest_table[i] = (uint32_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 32-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_32_shift(void *ptr_base, uint32_t const *src_table,
+		void **dest_table, size_t n, uint8_t bit_shift)
+{
+	size_t i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	do {
+		svbool_t pg = svwhilelt_b64(i, n);
+		v_ptr_table = svld1uw_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+	} while (i < n);
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	int64x2_t v_shift = vdupq_n_s64(bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	const size_t n_even = n & ~0x1;
+	for (; i < n_even; i += 2) {
+		v_ptr_table = vmovl_u32(vld1_u32(src_table + i));
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		v_ptr_table = vaddq_u64(v_ptr_table, v_ptr_base);
+		vst1q_u64((uint64_t *)dest_table + i, v_ptr_table);
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+/**
+ * Compress pointers into 16-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 16 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 256KB of the base pointer. If there is no such alignment guarantee
+ * they must be within 64KB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_16_shift(void *ptr_base, void * const *src_table,
+		uint16_t *dest_table, size_t n, uint8_t bit_shift)
+{
+
+	size_t i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	do {
+		svbool_t pg = svwhilelt_b64(i, n);
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1h(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+	} while (i < n);
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT16_MAX);
+		dest_table[i] = (uint16_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 16-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_16_shift(void *ptr_base, uint16_t const *src_table,
+		void **dest_table, size_t n, uint8_t bit_shift)
+{
+	size_t i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	do {
+		svbool_t pg = svwhilelt_b64(i, n);
+		v_ptr_table = svld1uh_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+	} while (i < n);
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_PTR_COMPRESS_H */
--
2.25.1

Re: [PATCH v14 3/6] ptr_compress: add pointer compression library

[David Marchand]

Hello,

On Fri, Jun 7, 2024 at 5:10 PM Paul Szczepanek <paul.szczepanek@arm.com> wrote:
>
> Add a new utility header for compressing pointers. The provided
> functions can store pointers as 32-bit or 16-bit offsets.
>
> The compression takes advantage of the fact that pointers are
> usually located in a limited memory region (like a mempool).
> We can compress them by converting them to offsets from a base
> memory address. Offsets can be stored in fewer bytes (dictated
> by the memory region size and alignment of the pointer).
> For example: an 8 byte aligned pointer which is part of a 32GB
> memory pool can be stored in 4 bytes.
>
> Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> Reviewed-by: Nathan Brown <nathan.brown@arm.com>
> Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
> ---
>  MAINTAINERS                            |   4 +
>  doc/api/doxy-api-index.md              |   1 +
>  doc/api/doxy-api.conf.in               |   1 +
>  doc/guides/rel_notes/release_24_07.rst |   5 +
>  lib/meson.build                        |   1 +
>  lib/ptr_compress/meson.build           |   4 +
>  lib/ptr_compress/rte_ptr_compress.h    | 324 +++++++++++++++++++++++++
>  7 files changed, 340 insertions(+)
>  create mode 100644 lib/ptr_compress/meson.build
>  create mode 100644 lib/ptr_compress/rte_ptr_compress.h
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index c9adff9846..27b2f03e6c 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1694,6 +1694,10 @@ M: Chenbo Xia <chenbox@nvidia.com>
>  M: Gaetan Rivet <grive@u256.net>
>  F: lib/pci/
>
> +Pointer Compression
> +M: Paul Szczepanek <paul.szczepanek@arm.com>
> +F: lib/ptr_compress/
> +
>  Power management
>  M: Anatoly Burakov <anatoly.burakov@intel.com>
>  M: David Hunt <david.hunt@intel.com>
> diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
> index 8c1eb8fafa..f9283154f8 100644
> --- a/doc/api/doxy-api-index.md
> +++ b/doc/api/doxy-api-index.md
> @@ -222,6 +222,7 @@ The public API headers are grouped by topics:
>    [config file](@ref rte_cfgfile.h),
>    [key/value args](@ref rte_kvargs.h),
>    [argument parsing](@ref rte_argparse.h),
> +  [ptr_compress](@ref rte_ptr_compress.h),
>    [string](@ref rte_string_fns.h),
>    [thread](@ref rte_thread.h)
>
> diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
> index 27afec8b3b..a8823c046f 100644
> --- a/doc/api/doxy-api.conf.in
> +++ b/doc/api/doxy-api.conf.in
> @@ -71,6 +71,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
>                            @TOPDIR@/lib/pipeline \
>                            @TOPDIR@/lib/port \
>                            @TOPDIR@/lib/power \
> +                          @TOPDIR@/lib/ptr_compress \
>                            @TOPDIR@/lib/rawdev \
>                            @TOPDIR@/lib/rcu \
>                            @TOPDIR@/lib/regexdev \
> diff --git a/doc/guides/rel_notes/release_24_07.rst b/doc/guides/rel_notes/release_24_07.rst
> index a69f24cf99..4711792e61 100644
> --- a/doc/guides/rel_notes/release_24_07.rst
> +++ b/doc/guides/rel_notes/release_24_07.rst
> @@ -55,6 +55,11 @@ New Features
>       Also, make sure to start the actual text at the margin.
>       =======================================================
>
> +* **Introduced pointer compression library.**
> +
> +  Library provides functions to compress and decompress arrays of pointers
> +  which can improve application performance under certain conditions.
> +  Performance test was added to help users evaluate performance on their setup.

Please, double empty line before a new section in the release notes.


>
>  Removed Items
>  -------------
> diff --git a/lib/meson.build b/lib/meson.build
> index 7c90602bf5..63becee142 100644
> --- a/lib/meson.build
> +++ b/lib/meson.build
> @@ -14,6 +14,7 @@ libraries = [
>          'argparse',
>          'telemetry', # basic info querying
>          'eal', # everything depends on eal
> +        'ptr_compress',
>          'ring',
>          'rcu', # rcu depends on ring
>          'mempool',
> diff --git a/lib/ptr_compress/meson.build b/lib/ptr_compress/meson.build
> new file mode 100644
> index 0000000000..e92706a45f
> --- /dev/null
> +++ b/lib/ptr_compress/meson.build
> @@ -0,0 +1,4 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2024 Arm Limited
> +
> +headers = files('rte_ptr_compress.h')
> diff --git a/lib/ptr_compress/rte_ptr_compress.h b/lib/ptr_compress/rte_ptr_compress.h
> new file mode 100644
> index 0000000000..bf9cfb0661
> --- /dev/null
> +++ b/lib/ptr_compress/rte_ptr_compress.h
> @@ -0,0 +1,324 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2024 Arm Limited
> + */
> +
> +#ifndef RTE_PTR_COMPRESS_H
> +#define RTE_PTR_COMPRESS_H
> +
> +/**
> + * @file
> + * Pointer compression and decompression functions.
> + *
> + * When passing arrays full of pointers between threads, memory containing
> + * the pointers is copied multiple times which is especially costly between
> + * cores. These functions allow us to compress the pointers.
> + *
> + * Compression takes advantage of the fact that pointers are usually located in
> + * a limited memory region. We compress them by converting them to offsets from
> + * a base memory address. Offsets can be stored in fewer bytes.
> + *
> + * The compression functions come in two varieties: 32-bit and 16-bit.
> + *
> + * To determine how many bits are needed to compress the pointer calculate

Please add a ,
... to compress the pointer,  calculate ...

> + * the biggest offset possible (highest value pointer - base pointer)
> + * and shift the value right according to alignment (shift by exponent of the
> + * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by
> + * 3, etc.). The resulting value must fit in either 32 or 16 bits.
> + *
> + * For usage example and further explanation please see "Pointer Compression" in
> + * doc/guides/prog_guide/ptr_compress_lib.rst

I don't like refering to a path in the sources: the rendered doc won't
link to the actual documentation page, and this path may get incorrect
in the future if for some reason the doc is renamed or moved.
The simpler is to state "please see this library documentation
programming guide".


> + */
> +
> +#include <stdint.h>
> +#include <inttypes.h>
> +
> +#include <rte_branch_prediction.h>
> +#include <rte_common.h>
> +#include <rte_debug.h>
> +#include <rte_vect.h>
> +#include <rte_mempool.h>

It is unneeded.
Please, don't create a dependency to the mempool library.


> +#include <rte_bitops.h>
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +/**
> + * Calculate how many bits are required to store a value within a given range.
> + * This is to help decide which pointer compression functions can be used to
> + * store pointers contained within a memory range.
> + *
> + * @param range
> + *   The size of the range the value belongs to.

In my mind, a range takes the form of a set of two symbols.

Here, iiuc, what is needed/requested as an input is the number of
entries of a (contiguous) space.
So the naming is at least strange.

> + * @return
> + *   Number of bits required to store a value.
> + **/
> +#define RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(range) \
> +       (((uint64_t)range) < 2 ? 1 : \
> +               (sizeof(uint64_t) * CHAR_BIT - rte_clz64((uint64_t)range - 1)))

But now, I don't see why we need to expose this macro (and its sister
RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT) at all.
Can you clarify the usecase?


> +
> +/**
> + * Calculate how many bits in the address can be dropped without losing any
> + * information thanks to the alignment of the address.
> + *
> + * @param alignment
> + *   Memory alignment.
> + * @return
> + *   Size of shift allowed without dropping any information from the pointer.
> + **/
> +#define RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(alignment) \
> +       ((alignment) == 0 ? 0 : rte_ctz64((uint64_t)alignment))
> +
> +/**
> + * Determine if rte_ptr_compress_16_shift can be used to compress pointers
> + * that contain addresses of memory objects whose memory is aligned by
> + * a given amount and contained in a given memory range.
> + *
> + * @param mem_range
> + *   The size of the memory region that contains the objects pointed to.

Again, can you rephrase and not use the term "range"?
Expected input here is a number of pointers / elements in a space.


> + * @param obj_alignment
> + *   The alignment of objects pointed to.
> + * @return
> + *   1 if function can be used, 0 otherwise.
> + **/
> +#define RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT(mem_range, obj_alignment) \
> +       ((RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(mem_range) - \
> +       RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(obj_alignment)) <= 16 ? 1 : 0)
> +
> +/**
> + * Determine if rte_ptr_compress_32_shift can be used to compress pointers
> + * that contain addresses of memory objects whose memory is aligned by
> + * a given amount and contained in a given memory range.
> + *
> + * @param mem_range
> + *   The size of the memory region that contains the objects pointed to.
> + * @param obj_alignment
> + *   The alignment of objects pointed to.
> + * @return
> + *   1 if function can be used, 0 otherwise.
> + **/
> +#define RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT(mem_range, obj_alignment) \
> +       ((RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(mem_range) - \
> +       RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(obj_alignment)) <= 32 ? 1 : 0)

RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT and
RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT do the same job.
The only thing that differs is the size of the compressed space, so
maybe this size could be passed as an input.

No strong opinion though.


> +
> +/**
> + * Compress pointers into 32-bit offsets from base pointer.
> + *
> + * @note It is programmer's responsibility to ensure the resulting offsets fit
> + * into 32 bits. Alignment of the structures pointed to by the pointers allows
> + * us to drop bits from the offsets. This is controlled by the bit_shift
> + * parameter. This means that if structures are aligned by 8 bytes they must be
> + * within 32GB of the base pointer. If there is no such alignment guarantee they
> + * must be within 4GB.
> + *
> + * @param ptr_base
> + *   A pointer used to calculate offsets of pointers in src_table.
> + * @param src_table
> + *   A pointer to an array of pointers.
> + * @param dest_table
> + *   A pointer to an array of compressed pointers returned by this function.
> + * @param n
> + *   The number of objects to compress, must be strictly positive.
> + * @param bit_shift
> + *   Byte alignment of memory pointed to by the pointers allows for
> + *   bits to be dropped from the offset and hence widen the memory region that
> + *   can be covered. This controls how many bits are right shifted.
> + **/
> +static __rte_always_inline void
> +rte_ptr_compress_32_shift(void *ptr_base, void * const *src_table,
> +               uint32_t *dest_table, size_t n, uint8_t bit_shift)
> +{
> +       size_t i = 0;
> +#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32

Note: DPDK usually split per architecture implementation in separate
header files, but I guess this is okay in the current form with #ifdef
for now..

[snip]


-- 
David Marchand

RE: [PATCH v14 3/6] ptr_compress: add pointer compression library

[Morten Brørup]

> From: David Marchand [mailto:david.marchand@redhat.com]
> Sent: Monday, 10 June 2024 17.19
> 
> Hello,
> 
> On Fri, Jun 7, 2024 at 5:10 PM Paul Szczepanek <paul.szczepanek@arm.com>
> wrote:
> >
> > Add a new utility header for compressing pointers. The provided
> > functions can store pointers as 32-bit or 16-bit offsets.
> >

[...]

> RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT and
> RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT do the same job.
> The only thing that differs is the size of the compressed space, so
> maybe this size could be passed as an input.
> 
> No strong opinion though.

Conceptually I consider "16_shift" and "32_shift" unique compression algorithm names, like "zip", "rar", "gzip", "bz2", "xz" etc.

The function/macro naming convention of the library is:
rte_ptr_[de]compress_<function/macro><algorithm>

So I prefer keeping the two separate macros (one per algorithm), rather than combining them into one with the size as input parameter.

Re: [PATCH v14 3/6] ptr_compress: add pointer compression library

[Paul Szczepanek]

On 10/06/2024 16:18, David Marchand wrote:
> Hello,
> 
> On Fri, Jun 7, 2024 at 5:10 PM Paul Szczepanek <paul.szczepanek@arm.com> wrote:
>>
>> Add a new utility header for compressing pointers. The provided
>> functions can store pointers as 32-bit or 16-bit offsets.
>>
>> The compression takes advantage of the fact that pointers are
>> usually located in a limited memory region (like a mempool).
>> We can compress them by converting them to offsets from a base
>> memory address. Offsets can be stored in fewer bytes (dictated
>> by the memory region size and alignment of the pointer).
>> For example: an 8 byte aligned pointer which is part of a 32GB
>> memory pool can be stored in 4 bytes.
>>
>> Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
>> Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
>> Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
>> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
>> Reviewed-by: Nathan Brown <nathan.brown@arm.com>
>> Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
>> Acked-by: Morten Brørup <mb@smartsharesystems.com>
>> ---
>>  MAINTAINERS                            |   4 +
>>  doc/api/doxy-api-index.md              |   1 +
>>  doc/api/doxy-api.conf.in               |   1 +
>>  doc/guides/rel_notes/release_24_07.rst |   5 +
>>  lib/meson.build                        |   1 +
>>  lib/ptr_compress/meson.build           |   4 +
>>  lib/ptr_compress/rte_ptr_compress.h    | 324 +++++++++++++++++++++++++
>>  7 files changed, 340 insertions(+)
>>  create mode 100644 lib/ptr_compress/meson.build
>>  create mode 100644 lib/ptr_compress/rte_ptr_compress.h
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index c9adff9846..27b2f03e6c 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -1694,6 +1694,10 @@ M: Chenbo Xia <chenbox@nvidia.com>
>>  M: Gaetan Rivet <grive@u256.net>
>>  F: lib/pci/
>>
>> +Pointer Compression
>> +M: Paul Szczepanek <paul.szczepanek@arm.com>
>> +F: lib/ptr_compress/
>> +
>>  Power management
>>  M: Anatoly Burakov <anatoly.burakov@intel.com>
>>  M: David Hunt <david.hunt@intel.com>
>> diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
>> index 8c1eb8fafa..f9283154f8 100644
>> --- a/doc/api/doxy-api-index.md
>> +++ b/doc/api/doxy-api-index.md
>> @@ -222,6 +222,7 @@ The public API headers are grouped by topics:
>>    [config file](@ref rte_cfgfile.h),
>>    [key/value args](@ref rte_kvargs.h),
>>    [argument parsing](@ref rte_argparse.h),
>> +  [ptr_compress](@ref rte_ptr_compress.h),
>>    [string](@ref rte_string_fns.h),
>>    [thread](@ref rte_thread.h)
>>
>> diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
>> index 27afec8b3b..a8823c046f 100644
>> --- a/doc/api/doxy-api.conf.in
>> +++ b/doc/api/doxy-api.conf.in
>> @@ -71,6 +71,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
>>                            @TOPDIR@/lib/pipeline \
>>                            @TOPDIR@/lib/port \
>>                            @TOPDIR@/lib/power \
>> +                          @TOPDIR@/lib/ptr_compress \
>>                            @TOPDIR@/lib/rawdev \
>>                            @TOPDIR@/lib/rcu \
>>                            @TOPDIR@/lib/regexdev \
>> diff --git a/doc/guides/rel_notes/release_24_07.rst b/doc/guides/rel_notes/release_24_07.rst
>> index a69f24cf99..4711792e61 100644
>> --- a/doc/guides/rel_notes/release_24_07.rst
>> +++ b/doc/guides/rel_notes/release_24_07.rst
>> @@ -55,6 +55,11 @@ New Features
>>       Also, make sure to start the actual text at the margin.
>>       =======================================================
>>
>> +* **Introduced pointer compression library.**
>> +
>> +  Library provides functions to compress and decompress arrays of pointers
>> +  which can improve application performance under certain conditions.
>> +  Performance test was added to help users evaluate performance on their setup.
> 
> Please, double empty line before a new section in the release notes.
> 
> 
>>
>>  Removed Items
>>  -------------
>> diff --git a/lib/meson.build b/lib/meson.build
>> index 7c90602bf5..63becee142 100644
>> --- a/lib/meson.build
>> +++ b/lib/meson.build
>> @@ -14,6 +14,7 @@ libraries = [
>>          'argparse',
>>          'telemetry', # basic info querying
>>          'eal', # everything depends on eal
>> +        'ptr_compress',
>>          'ring',
>>          'rcu', # rcu depends on ring
>>          'mempool',
>> diff --git a/lib/ptr_compress/meson.build b/lib/ptr_compress/meson.build
>> new file mode 100644
>> index 0000000000..e92706a45f
>> --- /dev/null
>> +++ b/lib/ptr_compress/meson.build
>> @@ -0,0 +1,4 @@
>> +# SPDX-License-Identifier: BSD-3-Clause
>> +# Copyright(c) 2024 Arm Limited
>> +
>> +headers = files('rte_ptr_compress.h')
>> diff --git a/lib/ptr_compress/rte_ptr_compress.h b/lib/ptr_compress/rte_ptr_compress.h
>> new file mode 100644
>> index 0000000000..bf9cfb0661
>> --- /dev/null
>> +++ b/lib/ptr_compress/rte_ptr_compress.h
>> @@ -0,0 +1,324 @@
>> +/* SPDX-License-Identifier: BSD-3-Clause
>> + * Copyright(c) 2024 Arm Limited
>> + */
>> +
>> +#ifndef RTE_PTR_COMPRESS_H
>> +#define RTE_PTR_COMPRESS_H
>> +
>> +/**
>> + * @file
>> + * Pointer compression and decompression functions.
>> + *
>> + * When passing arrays full of pointers between threads, memory containing
>> + * the pointers is copied multiple times which is especially costly between
>> + * cores. These functions allow us to compress the pointers.
>> + *
>> + * Compression takes advantage of the fact that pointers are usually located in
>> + * a limited memory region. We compress them by converting them to offsets from
>> + * a base memory address. Offsets can be stored in fewer bytes.
>> + *
>> + * The compression functions come in two varieties: 32-bit and 16-bit.
>> + *
>> + * To determine how many bits are needed to compress the pointer calculate
> 
> Please add a ,
> ... to compress the pointer,  calculate ...
> 
>> + * the biggest offset possible (highest value pointer - base pointer)
>> + * and shift the value right according to alignment (shift by exponent of the
>> + * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by
>> + * 3, etc.). The resulting value must fit in either 32 or 16 bits.
>> + *
>> + * For usage example and further explanation please see "Pointer Compression" in
>> + * doc/guides/prog_guide/ptr_compress_lib.rst
> 
> I don't like refering to a path in the sources: the rendered doc won't
> link to the actual documentation page, and this path may get incorrect
> in the future if for some reason the doc is renamed or moved.
> The simpler is to state "please see this library documentation
> programming guide".
> 
> 
>> + */
>> +
>> +#include <stdint.h>
>> +#include <inttypes.h>
>> +
>> +#include <rte_branch_prediction.h>
>> +#include <rte_common.h>
>> +#include <rte_debug.h>
>> +#include <rte_vect.h>
>> +#include <rte_mempool.h>
> 
> It is unneeded.
> Please, don't create a dependency to the mempool library.
> 
> 

I have addressed all of the above in v15.

>> +#include <rte_bitops.h>
>> +
>> +#ifdef __cplusplus
>> +extern "C" {
>> +#endif
>> +
>> +/**
>> + * Calculate how many bits are required to store a value within a given range.
>> + * This is to help decide which pointer compression functions can be used to
>> + * store pointers contained within a memory range.
>> + *
>> + * @param range
>> + *   The size of the range the value belongs to.
> 
> In my mind, a range takes the form of a set of two symbols.
> 
> Here, iiuc, what is needed/requested as an input is the number of
> entries of a (contiguous) space.
> So the naming is at least strange.
> 
>> + * @return
>> + *   Number of bits required to store a value.
>> + **/
>> +#define RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(range) \
>> +       (((uint64_t)range) < 2 ? 1 : \
>> +               (sizeof(uint64_t) * CHAR_BIT - rte_clz64((uint64_t)range - 1)))
> 
> But now, I don't see why we need to expose this macro (and its sister
> RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT) at all.
> Can you clarify the usecase?
> 
> 
>> +
>> +/**
>> + * Calculate how many bits in the address can be dropped without losing any
>> + * information thanks to the alignment of the address.
>> + *
>> + * @param alignment
>> + *   Memory alignment.
>> + * @return
>> + *   Size of shift allowed without dropping any information from the pointer.
>> + **/
>> +#define RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(alignment) \
>> +       ((alignment) == 0 ? 0 : rte_ctz64((uint64_t)alignment))
>> +
>> +/**
>> + * Determine if rte_ptr_compress_16_shift can be used to compress pointers
>> + * that contain addresses of memory objects whose memory is aligned by
>> + * a given amount and contained in a given memory range.
>> + *
>> + * @param mem_range
>> + *   The size of the memory region that contains the objects pointed to.
> 
> Again, can you rephrase and not use the term "range"?
> Expected input here is a number of pointers / elements in a space.
> 
>

I hope I addressed all of the comments above about the use of range in
macros in v15. I have renamed them to better reflect their use. The
functions don't care about number of elements but instead about the
memory ranges that the pointers are constrained to - hence the range in
names originally. I agree that range name implies two values, I am now
calling it a mem_length to make it unambiguous. The parameter is the
length of the memory region.

All the macros exposed are useful - they allow you to easily determine
the params to use in the compression functions.

>> + * @param obj_alignment
>> + *   The alignment of objects pointed to.
>> + * @return
>> + *   1 if function can be used, 0 otherwise.
>> + **/
>> +#define RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT(mem_range, obj_alignment) \
>> +       ((RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(mem_range) - \
>> +       RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(obj_alignment)) <= 16 ? 1 : 0)
>> +
>> +/**
>> + * Determine if rte_ptr_compress_32_shift can be used to compress pointers
>> + * that contain addresses of memory objects whose memory is aligned by
>> + * a given amount and contained in a given memory range.
>> + *
>> + * @param mem_range
>> + *   The size of the memory region that contains the objects pointed to.
>> + * @param obj_alignment
>> + *   The alignment of objects pointed to.
>> + * @return
>> + *   1 if function can be used, 0 otherwise.
>> + **/
>> +#define RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT(mem_range, obj_alignment) \
>> +       ((RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(mem_range) - \
>> +       RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(obj_alignment)) <= 32 ? 1 : 0)
> 
> RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT and
> RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT do the same job.
> The only thing that differs is the size of the compressed space, so
> maybe this size could be passed as an input.
> 
> No strong opinion though.
> 
> 

They are different and depending on your architecture you might not want
to use 16-bit even if you can fit the pointers in 16 bits due to
performance. I am keeping them separate.

>> +
>> +/**
>> + * Compress pointers into 32-bit offsets from base pointer.
>> + *
>> + * @note It is programmer's responsibility to ensure the resulting offsets fit
>> + * into 32 bits. Alignment of the structures pointed to by the pointers allows
>> + * us to drop bits from the offsets. This is controlled by the bit_shift
>> + * parameter. This means that if structures are aligned by 8 bytes they must be
>> + * within 32GB of the base pointer. If there is no such alignment guarantee they
>> + * must be within 4GB.
>> + *
>> + * @param ptr_base
>> + *   A pointer used to calculate offsets of pointers in src_table.
>> + * @param src_table
>> + *   A pointer to an array of pointers.
>> + * @param dest_table
>> + *   A pointer to an array of compressed pointers returned by this function.
>> + * @param n
>> + *   The number of objects to compress, must be strictly positive.
>> + * @param bit_shift
>> + *   Byte alignment of memory pointed to by the pointers allows for
>> + *   bits to be dropped from the offset and hence widen the memory region that
>> + *   can be covered. This controls how many bits are right shifted.
>> + **/
>> +static __rte_always_inline void
>> +rte_ptr_compress_32_shift(void *ptr_base, void * const *src_table,
>> +               uint32_t *dest_table, size_t n, uint8_t bit_shift)
>> +{
>> +       size_t i = 0;
>> +#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
> 
> Note: DPDK usually split per architecture implementation in separate
> header files, but I guess this is okay in the current form with #ifdef
> for now..
> 
> [snip]
> 
> 

[PATCH v14 4/6] test: add pointer compress tests to ring perf test

[Paul Szczepanek]

Add a test that runs a zero copy burst enqueue and dequeue on a ring
of raw pointers and compressed pointers at different burst sizes to
showcase performance benefits of newly added pointer compression APIs.

Refactored threading code to pass more parameters to threads to
reuse existing code. Added more bulk sizes to showcase their effects
on compression. Adjusted loop iteration numbers to take into account
bulk sizes to keep runtime constant (instead of number of operations).

Adjusted old printfs to match new ones which have aligned numbers.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
---
 app/test/meson.build      |  20 +--
 app/test/test_ring.h      |  94 ++++++++++
 app/test/test_ring_perf.c | 352 +++++++++++++++++++++++++-------------
 3 files changed, 334 insertions(+), 132 deletions(-)

diff --git a/app/test/meson.build b/app/test/meson.build
index 7d909039ae..df8cc00730 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -156,16 +156,16 @@ source_file_deps = {
 #    'test_resource.c': [],
     'test_rib.c': ['net', 'rib'],
     'test_rib6.c': ['net', 'rib'],
-    'test_ring.c': [],
-    'test_ring_hts_stress.c': [],
-    'test_ring_mpmc_stress.c': [],
-    'test_ring_mt_peek_stress.c': [],
-    'test_ring_mt_peek_stress_zc.c': [],
-    'test_ring_perf.c': [],
-    'test_ring_rts_stress.c': [],
-    'test_ring_st_peek_stress.c': [],
-    'test_ring_st_peek_stress_zc.c': [],
-    'test_ring_stress.c': [],
+    'test_ring.c': ['ptr_compress'],
+    'test_ring_hts_stress.c': ['ptr_compress'],
+    'test_ring_mpmc_stress.c': ['ptr_compress'],
+    'test_ring_mt_peek_stress.c': ['ptr_compress'],
+    'test_ring_mt_peek_stress_zc.c': ['ptr_compress'],
+    'test_ring_perf.c': ['ptr_compress'],
+    'test_ring_rts_stress.c': ['ptr_compress'],
+    'test_ring_st_peek_stress.c': ['ptr_compress'],
+    'test_ring_st_peek_stress_zc.c': ['ptr_compress'],
+    'test_ring_stress.c': ['ptr_compress'],
     'test_rwlock.c': [],
     'test_sched.c': ['net', 'sched'],
     'test_security.c': ['net', 'security'],
diff --git a/app/test/test_ring.h b/app/test/test_ring.h
index 45c263f3ff..9e97c5e3e7 100644
--- a/app/test/test_ring.h
+++ b/app/test/test_ring.h
@@ -5,6 +5,8 @@
 #include <rte_malloc.h>
 #include <rte_ring.h>
 #include <rte_ring_elem.h>
+#include <rte_memcpy.h>
+#include <rte_ptr_compress.h>

 /* API type to call
  * rte_ring_<sp/mp or sc/mc>_enqueue_<bulk/burst>
@@ -25,6 +27,10 @@
 #define TEST_RING_ELEM_BULK 16
 #define TEST_RING_ELEM_BURST 32

+#define TEST_RING_ELEM_BURST_ZC 64
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 128
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 256
+
 #define TEST_RING_IGNORE_API_TYPE ~0U

 /* This function is placed here as it is required for both
@@ -101,6 +107,9 @@ static inline unsigned int
 test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -152,6 +161,47 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n,
 								NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy(zcd.ptr2,
+						(char *)obj + zcd.n1 * esize,
+						(ret - zcd.n1) * esize);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_16_shift(
+					0, obj, zcd.ptr1, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_16_shift(0,
+						obj + (zcd.n1 * 2),
+						zcd.ptr2,
+						(ret - zcd.n1) * 2, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_32_shift(0, obj, zcd.ptr1, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_32_shift(0, obj + zcd.n1,
+						zcd.ptr2, ret - zcd.n1, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
@@ -162,6 +212,9 @@ static inline unsigned int
 test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -213,6 +266,47 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mc_dequeue_burst_elem(r, obj, esize,
 								n, NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy((char *)obj + zcd.n1 * esize,
+						zcd.ptr2,
+						(ret - zcd.n1) * esize);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_16_shift(
+					0, zcd.ptr1, obj, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_16_shift(0, zcd.ptr2,
+						obj + zcd.n1,
+						(ret - zcd.n1) * 2,
+						3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_32_shift(0, zcd.ptr1, obj, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_32_shift(0, zcd.ptr2,
+						obj + zcd.n1, ret - zcd.n1, 3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c
index ee328614c1..57cd04a124 100644
--- a/app/test/test_ring_perf.c
+++ b/app/test/test_ring_perf.c
@@ -22,13 +22,13 @@

 #define RING_NAME "RING_PERF"
 #define RING_SIZE 4096
-#define MAX_BURST 32
+#define MAX_BURST 256

 /*
  * the sizes to enqueue and dequeue in testing
  * (marked volatile so they won't be seen as compile-time constants)
  */
-static const volatile unsigned bulk_sizes[] = { 8, 32 };
+static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128, 256 };

 struct lcore_pair {
 	unsigned c1, c2;
@@ -43,26 +43,30 @@ test_ring_print_test_string(unsigned int api_type, int esize,
 	if (esize == -1)
 		printf("legacy APIs");
 	else
-		printf("elem APIs: element size %dB", esize);
+		printf("elem APIs (size:%2dB)", esize);

 	if (api_type == TEST_RING_IGNORE_API_TYPE)
 		return;

 	if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF)
-		printf(": default enqueue/dequeue: ");
+		printf(" - default enqueue/dequeue");
 	else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC)
-		printf(": SP/SC: ");
+		printf(" - SP/SC");
 	else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC)
-		printf(": MP/MC: ");
+		printf(" - MP/MC");

 	if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE)
-		printf("single: ");
+		printf(" - single - ");
 	else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK)
-		printf("bulk (size: %u): ", bsz);
+		printf(" - bulk (n:%-3u) - ", bsz);
 	else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST)
-		printf("burst (size: %u): ", bsz);
+		printf(" - burst (n:%-3u) - ", bsz);
+	else if ((api_type & (TEST_RING_ELEM_BURST_ZC |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0)
+		printf(" - burst zero copy (n:%-3u) - ", bsz);

-	printf("%.2F\n", value);
+	printf("cycles per elem: %.3F\n", value);
 }

 /**** Functions to analyse our core mask to get cores for different tests ***/
@@ -144,28 +148,33 @@ static void
 test_empty_dequeue(struct rte_ring *r, const int esize,
 			const unsigned int api_type)
 {
-	const unsigned int iter_shift = 26;
+	const unsigned int iter_shift = 29;
 	const unsigned int iterations = 1 << iter_shift;
 	unsigned int i = 0;
 	void *burst[MAX_BURST];

+	const unsigned int bulk_iterations = iterations / bulk_sizes[0];
 	const uint64_t start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	for (i = 0; i < bulk_iterations; i++)
 		test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type);
 	const uint64_t end = rte_rdtsc();

 	test_ring_print_test_string(api_type, esize, bulk_sizes[0],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 }

-/*
- * for the separate enqueue and dequeue threads they take in one param
- * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc
- */
-struct thread_params {
+/* describes the ring used by the enqueue and dequeue thread */
+struct ring_params {
 	struct rte_ring *r;
-	unsigned size;        /* input value, the burst size */
-	double spsc, mpmc;    /* output value, the single or multi timings */
+	unsigned int elem_size;
+	unsigned int bulk_sizes_i; /* index into bulk_size array */
+	unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */
+};
+
+/* Used to specify enqueue and dequeue ring operations and their results */
+struct thread_params {
+	struct ring_params *ring_params;
+	double *results; /* result array size must be equal to bulk_sizes */
 };

 /*
@@ -174,16 +183,15 @@ struct thread_params {
  * flag == 1 -> dequeue
  */
 static __rte_always_inline int
-enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
-	struct thread_params *p)
+enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 15;
+	const unsigned int iter_shift = 22;
 	const unsigned int iterations = 1 << iter_shift;
-	struct rte_ring *r = p->r;
-	unsigned int bsize = p->size;
 	unsigned int i;
 	void *burst = NULL;
+	unsigned int n_remaining;
+	const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i];

 #ifdef RTE_USE_C11_MEM_MODEL
 	if (rte_atomic_fetch_add_explicit(&lcore_count, 1, rte_memory_order_relaxed) + 1 != 2)
@@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 		while(lcore_count != 2)
 			rte_pause();

-	burst = test_ring_calloc(MAX_BURST, esize);
+	burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size);
 	if (burst == NULL)
 		return -1;

 	const uint64_t sp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	const unsigned int bulk_iterations = iterations / bulk_n;
+	for (i = 0; i < bulk_iterations; i++) {
+		n_remaining = bulk_n;
 		do {
 			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_enqueue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_dequeue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			if (ret == 0)
 				rte_pause();
-		} while (!ret);
+			else
+				n_remaining -= ret;
+		} while (n_remaining > 0);
+	}
 	const uint64_t sp_end = rte_rdtsc();

-	const uint64_t mp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
-		do {
-			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			if (ret == 0)
-				rte_pause();
-		} while (!ret);
-	const uint64_t mp_end = rte_rdtsc();
+	p->results[p->ring_params->bulk_sizes_i] =
+			((double)sp_end - sp_start) / iterations;

-	p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize);
-	p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize);
 	return 0;
 }

@@ -243,15 +245,7 @@ enqueue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(0, -1, params);
-}
-
-static int
-enqueue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(0, 16, params);
+	return enqueue_dequeue_bulk_helper(0, params);
 }

 /*
@@ -263,15 +257,7 @@ dequeue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(1, -1, params);
-}
-
-static int
-dequeue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(1, 16, params);
+	return enqueue_dequeue_bulk_helper(1, params);
 }

 /*
@@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p)
  * used to measure ring perf between hyperthreads, cores and sockets.
  */
 static int
-run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize)
+run_on_core_pair(struct lcore_pair *cores,
+		struct thread_params *param1, struct thread_params *param2)
 {
-	lcore_function_t *f1, *f2;
-	struct thread_params param1 = {0}, param2 = {0};
 	unsigned i;
-
-	if (esize == -1) {
-		f1 = enqueue_bulk;
-		f2 = dequeue_bulk;
-	} else {
-		f1 = enqueue_bulk_16B;
-		f2 = dequeue_bulk_16B;
-	}
+	struct ring_params *ring_params = param1->ring_params;

 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		lcore_count = 0;
-		param1.size = param2.size = bulk_sizes[i];
-		param1.r = param2.r = r;
+		ring_params->bulk_sizes_i = i;
 		if (cores->c1 == rte_get_main_lcore()) {
-			rte_eal_remote_launch(f2, &param2, cores->c2);
-			f1(&param1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
+			enqueue_bulk(param1);
 			rte_eal_wait_lcore(cores->c2);
 		} else {
-			rte_eal_remote_launch(f1, &param1, cores->c1);
-			rte_eal_remote_launch(f2, &param2, cores->c2);
+			rte_eal_remote_launch(enqueue_bulk, param1, cores->c1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
 			if (rte_eal_wait_lcore(cores->c1) < 0)
 				return -1;
 			if (rte_eal_wait_lcore(cores->c2) < 0)
 				return -1;
 		}
 		test_ring_print_test_string(
-			TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.spsc + param2.spsc);
-		test_ring_print_test_string(
-			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.mpmc + param2.mpmc);
+				ring_params->ring_flags,
+				ring_params->elem_size,
+				bulk_sizes[i],
+				param1->results[i] + param2->results[i]);
 	}

 	return 0;
@@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 	uint64_t hz = rte_get_timer_hz();
 	uint64_t lcount = 0;
 	const unsigned int lcore = rte_lcore_id();
-	struct thread_params *params = p;
+	struct ring_params *ring_params = p->ring_params;
 	void *burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -347,9 +323,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize)

 	begin = rte_get_timer_cycles();
 	while (time_diff < hz * TIME_MS / 1000) {
-		test_ring_enqueue(params->r, burst, esize, params->size,
+		test_ring_enqueue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
-		test_ring_dequeue(params->r, burst, esize, params->size,
+		test_ring_dequeue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 		lcount++;
 		time_diff = rte_get_timer_cycles() - begin;
@@ -381,7 +359,8 @@ static int
 run_on_all_cores(struct rte_ring *r, const int esize)
 {
 	uint64_t total;
-	struct thread_params param;
+	struct ring_params ring_params = {0};
+	struct thread_params params = { .ring_params = &ring_params };
 	lcore_function_t *lcore_f;
 	unsigned int i, c;

@@ -390,21 +369,20 @@ run_on_all_cores(struct rte_ring *r, const int esize)
 	else
 		lcore_f = load_loop_fn_16B;

-	memset(&param, 0, sizeof(struct thread_params));
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		total = 0;
 		printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]);
-		param.size = bulk_sizes[i];
-		param.r = r;
+		params.ring_params->bulk_sizes_i = i;
+		params.ring_params->r = r;

 		/* clear synchro and start workers */
 		rte_atomic_store_explicit(&synchro, 0, rte_memory_order_relaxed);
-		if (rte_eal_mp_remote_launch(lcore_f, &param, SKIP_MAIN) < 0)
+		if (rte_eal_mp_remote_launch(lcore_f, &params, SKIP_MAIN) < 0)
 			return -1;

 		/* start synchro and launch test on main */
 		rte_atomic_store_explicit(&synchro, 1, rte_memory_order_relaxed);
-		lcore_f(&param);
+		lcore_f(&params);

 		rte_eal_mp_wait_lcore();

@@ -463,9 +441,9 @@ static int
 test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	const unsigned int api_type)
 {
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 26;
 	const unsigned int iterations = 1 << iter_shift;
-	unsigned int sz, i = 0;
+	unsigned int sz, i;
 	void **burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -473,17 +451,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 		return -1;

 	for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) {
+		const unsigned int n = iterations / bulk_sizes[sz];
 		const uint64_t start = rte_rdtsc();
-		for (i = 0; i < iterations; i++) {
+		for (i = 0; i < n; i++) {
 			test_ring_enqueue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 			test_ring_dequeue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 		}
 		const uint64_t end = rte_rdtsc();

 		test_ring_print_test_string(api_type, esize, bulk_sizes[sz],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 	}

 	rte_free(burst);
@@ -491,12 +470,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	return 0;
 }

+static __rte_always_inline int
+test_ring_perf_esize_run_on_two_cores(
+		struct thread_params *param1, struct thread_params *param2)
+{
+	struct lcore_pair cores;
+
+	if (get_two_hyperthreads(&cores) == 0) {
+		printf("\n### Testing using two hyperthreads ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing using two physical cores ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_sockets(&cores) == 0) {
+		printf("\n### Testing using two NUMA nodes ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	return 0;
+}
+
 /* Run all tests for a given element size */
 static __rte_always_inline int
 test_ring_perf_esize(const int esize)
 {
-	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
+	double results_enq[RTE_DIM(bulk_sizes)];
+	double results_deq[RTE_DIM(bulk_sizes)];
+	struct ring_params ring_params = {
+			.elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results_enq };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results_deq };

 	/*
 	 * Performance test for legacy/_elem APIs
@@ -536,22 +546,13 @@ test_ring_perf_esize(const int esize)
 	test_empty_dequeue(r, esize,
 			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);

-	if (get_two_hyperthreads(&cores) == 0) {
-		printf("\n### Testing using two hyperthreads ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.r = r;

-	if (get_two_cores(&cores) == 0) {
-		printf("\n### Testing using two physical cores ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
-	if (get_two_sockets(&cores) == 0) {
-		printf("\n### Testing using two NUMA nodes ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
+
+	ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);

 	printf("\n### Testing using all worker nodes ###\n");
 	if (run_on_all_cores(r, esize) < 0)
@@ -567,6 +568,109 @@ test_ring_perf_esize(const int esize)
 	return -1;
 }

+
+static __rte_always_inline int
+test_ring_perf_compression(void)
+{
+	double results1[RTE_DIM(bulk_sizes)];
+	double results2[RTE_DIM(bulk_sizes)];
+	double results1_comp[2][RTE_DIM(bulk_sizes)];
+	double results2_comp[2][RTE_DIM(bulk_sizes)];
+
+	struct lcore_pair cores;
+	int ret = -1;
+	unsigned int i, j;
+	struct ring_params ring_params = { .elem_size = sizeof(void *) };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results1 };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results2 };
+
+	printf("\n### Testing compression gain ###");
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(void *),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy ###\n");
+		ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	/* rings allow only multiples of 4 as sizes,
+	 * we allocate size 4 despite only using 2 bytes
+	 * and use half of RING_SIZE as the number of elements
+	 */
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE / 2, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[0];
+	param2.results = results2_comp[0];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (16b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[1];
+	param2.results = results2_comp[1];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (32b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	for (j = 0; j < 2; j++) {
+		printf("\n### Potential gain from compression (%d-bit offsets) "
+		"###\n", (j + 1) * 16);
+		for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
+			const double result = results1[i] + results2[i];
+			const double result_comp = results1_comp[j][i] +
+				results2_comp[j][i];
+			const double gain = 100 - (result_comp / result) * 100;
+
+			printf("Gain of %5.1F%% for burst of %-3u elems\n",
+					gain, bulk_sizes[i]);
+		}
+	}
+
+	return ret;
+}
+
 static int
 test_ring_perf(void)
 {
@@ -577,6 +681,10 @@ test_ring_perf(void)
 	if (test_ring_perf_esize(16) == -1)
 		return -1;

+	/* Test for performance gain of compression */
+	if (test_ring_perf_compression() == -1)
+		return -1;
+
 	return 0;
 }

--
2.25.1

[PATCH v14 5/6] docs: add pointer compression guide

[Paul Szczepanek]

Documentation added in the prog guide for the new
utility functions for pointer compression
showing example code and potential usecases.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
---
 MAINTAINERS                                |   1 +
 doc/guides/prog_guide/index.rst            |   1 +
 doc/guides/prog_guide/ptr_compress_lib.rst | 160 +++++++++++++++++++++
 3 files changed, 162 insertions(+)
 create mode 100644 doc/guides/prog_guide/ptr_compress_lib.rst

diff --git a/MAINTAINERS b/MAINTAINERS
index 27b2f03e6c..ed50121bd2 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1697,6 +1697,7 @@ F: lib/pci/
 Pointer Compression
 M: Paul Szczepanek <paul.szczepanek@arm.com>
 F: lib/ptr_compress/
+F: doc/guides/prog_guide/ptr_compress_lib.rst

 Power management
 M: Anatoly Burakov <anatoly.burakov@intel.com>
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index d09d958e6c..6366849eb0 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -73,6 +73,7 @@ Programmer's Guide
     telemetry_lib
     bpf_lib
     graph_lib
+    ptr_compress_lib
     build-sdk-meson
     meson_ut
     build_app
diff --git a/doc/guides/prog_guide/ptr_compress_lib.rst b/doc/guides/prog_guide/ptr_compress_lib.rst
new file mode 100644
index 0000000000..49e94e6c4e
--- /dev/null
+++ b/doc/guides/prog_guide/ptr_compress_lib.rst
@@ -0,0 +1,160 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2024 Arm Limited.
+
+Pointer Compression Library
+===========================
+
+Use ``rte_ptr_compress_16_shift()`` and ``rte_ptr_decompress_16_shift()`` to
+compress and decompress pointers into 16-bit offsets.
+Use ``rte_ptr_compress_32_shift()`` and ``rte_ptr_decompress_32_shift()`` to
+compress and decompress pointers into 32-bit offsets.
+
+Compression takes advantage of the fact that pointers are usually located in a
+limited memory region (like a mempool). By converting them to offsets from a
+base memory address they can be stored in fewer bytes. How many bytes are needed
+to store the offset is dictated by the memory region size and alignment of
+objects the pointers point to.
+
+For example, a pointer which is part of a 4GB memory pool can be stored as 32
+bit offset. If the pointer points to memory that is 8 bytes aligned then 3 bits
+can be dropped from the offset and a 32GB memory pool can now fit in 32 bits.
+
+For performance reasons these requirements are not enforced programmatically.
+The programmer is responsible for ensuring that the combination of distance
+from the base pointer and memory alignment allow for storing of the offset in
+the number of bits indicated by the function name (16 or 32). Start of mempool
+memory would be a good candidate for the base pointer. Otherwise any pointer
+that precedes all pointers, is close enough and has the same alignment as the
+pointers being compressed will work.
+
+Macros present in the rte_ptr_compress.h header may be used to evaluate whether
+compression is possible:
+
+*   RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE
+
+*   RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT
+
+*   RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT
+
+*   RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT
+
+These will help you calculate compression parameters and whether these are
+legal for particular compression function.
+
+If using an rte_mempool you can get the parameters you need to use in the
+compression macros and functions by using ``rte_mempool_get_mem_range()``
+and ``rte_mempool_get_obj_alignment()``.
+
+.. note::
+
+    Performance gains depend on the batch size of pointers and CPU capabilities
+    such as vector extensions. It's important to measure the performance
+    increase on target hardware. A test called ``ring_perf_autotest`` in
+    ``dpdk-test`` can provide the measurements.
+
+Example usage
+~~~~~~~~~~~~~
+
+In this example we send pointers between two cores through a ring. While this
+is a realistic use case the code is simplified for demonstration purposes and
+does not have error handling.
+
+.. code-block:: c
+
+    #include <rte_launch.h>
+    #include <rte_ring.h>
+    #include <rte_ring_elem.h>
+    #include <rte_ptr_compress.h>
+
+    #define ITEMS_ARRAY_SIZE (1024)
+    #define BATCH_SIZE (128)
+    #define ALIGN_EXPONENT (3)
+    #define ITEM_ALIGN (1<<ALIGN_EXPONENT)
+    #define CORE_SEND (1)
+    #define CORE_RECV (2)
+
+    struct item {
+      alignas(ITEM_ALIGN) int a;
+    };
+
+    static struct item items[ITEMS_ARRAY_SIZE] = {0};
+    static struct rte_ring *ring = NULL;
+
+    static int
+    send_compressed(void *args)
+    {
+      struct item *ptrs_send[BATCH_SIZE] = {0};
+      unsigned int n_send = 0;
+      struct rte_ring_zc_data zcd = {0};
+
+      /* in this example we only fill the ptrs_send once and reuse */
+      for (;n_send < BATCH_SIZE; n_send++)
+        ptrs_send[n_send] = &items[n_send];
+
+      for(;;) {
+        n_send = rte_ring_enqueue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        /* compress ptrs_send into offsets */
+        rte_ptr_compress_32_shift(items, /* base pointer */
+          ptrs_send, /* source array to be compressed */
+          zcd.ptr1, /* destination array to store offsets */
+          zcd.n1, /* how many pointers to compress */
+          ALIGN_EXPONENT /* how many bits can we drop from the offset */);
+
+        if (zcd.ptr2 != NULL)
+          rte_ptr_compress_32_shift(items, ptrs_send + zcd.n1,
+            zcd.ptr2, n_send - zcd.n1, ALIGN_EXPONENT);
+
+        rte_ring_enqueue_zc_finish(ring, n_send);
+      }
+      return 1;
+    }
+
+    static int
+    recv_compressed(void *args)
+    {
+      struct item *ptrs_recv[BATCH_SIZE] = {0};
+      unsigned int n_recv;
+      struct rte_ring_zc_data zcd = {0};
+
+      for(;;) {
+        /* receive compressed pointers from the ring */
+        n_recv = rte_ring_dequeue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        rte_ptr_decompress_32_shift(items, /* base pointer */
+          zcd.ptr1, /* source array to decompress */
+          ptrs_recv, /* destination array to store pointers */
+          zcd.n1, /* how many pointers to decompress */
+          ALIGN_EXPONENT /* how many bits were dropped from the offset */);
+
+        /* handle the potential secondary buffer (caused by ring boundary) */
+        if (zcd.ptr2 != NULL)
+          rte_ptr_decompress_32_shift(items,
+            zcd.ptr2,
+            ptrs_recv + zcd.n1,
+            n_recv - zcd.n1,
+            ALIGN_EXPONENT);
+
+        rte_ring_dequeue_zc_finish(ring, n_recv);
+
+        /* ptrs_recv contains what ptrs_send contained in the other thread */
+        /* (...) */
+      }
+      return 1;
+    }
+
+    void
+    compression_example(void)
+    {
+      ring = rte_ring_create_elem(
+        "COMPR_PTRS", sizeof(uint32_t),
+        1024, rte_socket_id(),
+        RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+      rte_eal_remote_launch(send_compressed, NULL, CORE_SEND);
+      rte_eal_remote_launch(recv_compressed, NULL, CORE_RECV);
+
+      for(;;) {}
+    }
--
2.25.1

[PATCH v14 6/6] test: add unit test for ptr compression

[Paul Szczepanek]

Test compresses and decompresses pointers with various combinations
of memory regions and alignments and verifies the pointers are
recovered correctly. Additionally tests helper macros perform
calculations correctly.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
---
 MAINTAINERS                  |   1 +
 app/test/meson.build         |   1 +
 app/test/test_ptr_compress.c | 193 +++++++++++++++++++++++++++++++++++
 3 files changed, 195 insertions(+)
 create mode 100644 app/test/test_ptr_compress.c

diff --git a/MAINTAINERS b/MAINTAINERS
index ed50121bd2..2565ef5f4b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1697,6 +1697,7 @@ F: lib/pci/
 Pointer Compression
 M: Paul Szczepanek <paul.szczepanek@arm.com>
 F: lib/ptr_compress/
+F: app/test/test_ptr_compress.c
 F: doc/guides/prog_guide/ptr_compress_lib.rst

 Power management
diff --git a/app/test/meson.build b/app/test/meson.build
index df8cc00730..e29258e6ec 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -144,6 +144,7 @@ source_file_deps = {
     'test_power_intel_uncore.c': ['power'],
     'test_power_kvm_vm.c': ['power'],
     'test_prefetch.c': [],
+    'test_ptr_compress.c': ['ptr_compress'],
     'test_rand_perf.c': [],
     'test_rawdev.c': ['rawdev', 'bus_vdev'],
     'test_rcu_qsbr.c': ['rcu', 'hash'],
diff --git a/app/test/test_ptr_compress.c b/app/test/test_ptr_compress.c
new file mode 100644
index 0000000000..ab33815974
--- /dev/null
+++ b/app/test/test_ptr_compress.c
@@ -0,0 +1,193 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2024 Arm Limited
+ */
+
+#include "test.h"
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_ptr_compress.h>
+
+#define MAX_ALIGN_EXPONENT 3
+#define MAX_PTRS 16
+#define NUM_BASES 2
+#define NUM_REGIONS 4
+#define MAX_32BIT_REGION ((uint64_t)UINT32_MAX + 1)
+#define MAX_16BIT_REGION (UINT16_MAX + 1)
+
+static int
+test_ptr_compress_params(
+	void *base,
+	uint64_t mem_sz,
+	unsigned int align_exp,
+	unsigned int num_ptrs,
+	bool use_32_bit)
+{
+	unsigned int i;
+	unsigned int align = 1 << align_exp;
+	void *ptrs[MAX_PTRS] = {0};
+	void *ptrs_out[MAX_PTRS] = {0};
+	uint32_t offsets32[MAX_PTRS] = {0};
+	uint16_t offsets16[MAX_PTRS] = {0};
+
+	for (i = 0; i < num_ptrs; i++) {
+		/* make pointers point at memory in steps of align */
+		/* alternate steps from the start and end of memory region */
+		if ((i & 1) == 1)
+			ptrs[i] = (char *)base + mem_sz - i * align;
+		else
+			ptrs[i] = (char *)base + i * align;
+	}
+
+	if (use_32_bit) {
+		rte_ptr_compress_32_shift(
+				base, ptrs, offsets32, num_ptrs, align_exp);
+		rte_ptr_decompress_32_shift(base, offsets32, ptrs_out, num_ptrs,
+				align_exp);
+	} else {
+		rte_ptr_compress_16_shift(
+				base, ptrs, offsets16, num_ptrs, align_exp);
+		rte_ptr_decompress_16_shift(base, offsets16, ptrs_out, num_ptrs,
+				align_exp);
+	}
+
+	TEST_ASSERT_BUFFERS_ARE_EQUAL(ptrs, ptrs_out, sizeof(void *) * num_ptrs,
+		"Decompressed pointers corrupted\nbase pointer: %p, "
+		"memory region size: %" PRIu64 ", alignment exponent: %u, "
+		"num of pointers: %u, using %s offsets",
+		base, mem_sz, align_exp, num_ptrs,
+		use_32_bit ? "32-bit" : "16-bit");
+
+	return 0;
+}
+
+static int
+test_ptr_compress(void)
+{
+	unsigned int j, k, n;
+	int ret = 0;
+	/* the test is run with multiple memory regions and base addresses */
+	void * const bases[NUM_BASES] = { (void *)0, (void *)UINT16_MAX };
+	/* maximum size for pointers aligned by consecutive powers of 2 */
+	const uint64_t region_sizes_16[NUM_REGIONS] = {
+		MAX_16BIT_REGION,
+		MAX_16BIT_REGION * 2,
+		MAX_16BIT_REGION * 4,
+		MAX_16BIT_REGION * 8,
+	};
+	const uint64_t region_sizes_32[NUM_REGIONS] = {
+		MAX_32BIT_REGION,
+		MAX_32BIT_REGION * 2,
+		MAX_32BIT_REGION * 4,
+		MAX_32BIT_REGION * 8,
+	};
+
+	/* main test compresses and decompresses arrays of pointers
+	 * and compares the array before and after to verify that
+	 * pointers are successfully decompressed
+	 */
+
+	for (j = 0; j < NUM_REGIONS; j++) {
+		for (k = 0; k < NUM_BASES; k++) {
+			for (n = 1; n < MAX_PTRS; n++) {
+				ret |= test_ptr_compress_params(
+					bases[k],
+					region_sizes_16[j],
+					j /* exponent of alignment */,
+					n,
+					false
+				);
+				ret |= test_ptr_compress_params(
+					bases[k],
+					region_sizes_32[j],
+					j /* exponent of alignment */,
+					n,
+					true
+				);
+				if (ret != 0)
+					return ret;
+			}
+		}
+	}
+
+	/* verify helper macro computations */
+
+	n = RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(0);
+	TEST_ASSERT_EQUAL(n, 1,
+			"RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(1);
+	TEST_ASSERT_EQUAL(n, 1,
+			"RTE_PTR_COMPREtopSS_BITS_REQUIRED_TO_STORE_VALUE "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(MAX_16BIT_REGION);
+	TEST_ASSERT_EQUAL(n, 16,
+			"RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE(MAX_32BIT_REGION);
+	TEST_ASSERT_EQUAL(n, 32,
+			"RTE_PTR_COMPRESS_BITS_REQUIRED_TO_STORE_VALUE_IN_RANGE "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(0);
+	TEST_ASSERT_EQUAL(n, 0,
+			"RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(1);
+	TEST_ASSERT_EQUAL(n, 0,
+			"RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(2);
+	TEST_ASSERT_EQUAL(n, 1,
+			"RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(3);
+	TEST_ASSERT_EQUAL(n, 0,
+			"RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(4);
+	TEST_ASSERT_EQUAL(n, 2,
+			"RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT(MAX_16BIT_REGION, 1);
+	TEST_ASSERT_EQUAL(ret, 1,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT(MAX_16BIT_REGION + 1, 1);
+	TEST_ASSERT_EQUAL(ret, 0,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT(MAX_16BIT_REGION + 1, 2);
+	TEST_ASSERT_EQUAL(ret, 1,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT(MAX_32BIT_REGION, 1);
+	TEST_ASSERT_EQUAL(ret, 1,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT(MAX_32BIT_REGION + 1, 1);
+	TEST_ASSERT_EQUAL(ret, 0,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT(MAX_32BIT_REGION + 1, 2);
+	TEST_ASSERT_EQUAL(ret, 1,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT "
+			"macro computation incorrect\n");
+
+	return 0;
+}
+
+REGISTER_FAST_TEST(ptr_compress_autotest, true, true, test_ptr_compress);
--
2.25.1

[PATCH v15 0/6] add pointer compression API

[Paul Szczepanek]

This patchset is proposing adding a new header only library
with utility functions that allow compression of arrays of pointers.

Since this is a header only library a patch needed to be added to amend
the build system to allow adding libraries without source files.

When passing caches full of pointers between threads, memory containing
the pointers is copied multiple times which is especially costly between
cores. A compression method will allow us to shrink the memory size
copied.

The compression takes advantage of the fact that pointers are usually
located in a limited memory region. We can compress them by converting them
to offsets from a base memory address.

Offsets can be stored in fewer bytes (dictated by the memory region size
and alignment of the pointer). For example: an 8 byte aligned pointer
which is part of a 32GB memory pool can be stored in 4 bytes. The API is
very generic and does not assume mempool pointers, any pointer can be
passed in.

Compression is based on few and fast operations and especially with vector
instructions leveraged creates minimal overhead.

The API accepts and returns arrays because the overhead means it only is
worth it when done in bulk.

Test is added that shows potential performance gain from compression. In
this test an array of pointers is passed through a ring between two cores.
It shows the gain which is dependent on the bulk operation size. In this
synthetic test run on ampere altra a substantial (up to 25%) performance
gain is seen if done in bulk size larger than 32. At 32 it breaks even and
lower sizes create a small (less than 5%) slowdown due to overhead.

In a more realistic mock application running the l3 forwarding dpdk
example that works in pipeline mode on two cores this translated into a
~5% throughput increase on an ampere altra.

v2:
* addressed review comments (style, explanations and typos)
* lowered bulk iterations closer to original numbers to keep runtime short
* fixed pointer size warning on 32-bit arch
v3:
* added 16-bit versions of compression functions and tests
* added documentation of these new utility functions in the EAL guide
v4:
* added unit test
* fix bug in NEON implementation of 32-bit decompress
v5:
* disable NEON and SVE implementation on AARCH32 due to wrong pointer size
v6:
* added example usage to commit message of the initial commit
v7:
* rebase to remove clashing mailmap changes
v8:
* put ptr compress into its own library
* add depends-on tag
* remove copyright bumps
* typos
v9
* added MAINTAINERS entries, release notes, doc indexes etc.
* added patch for build system to allow header only library
v10
* fixed problem with meson build adding shared deps to static deps
v11
* added mempool functions to get information about memory range and
alignment
* added tests for the new mempool functions
* added macros to help find the parameters for compression functions
* minor improvement in the SVE compression code
* amended documentation to reflect these changes
v12
* added doxygen and prefixes to macros
* use rte_bitops for clz and ctz
* added unit tests to verify macros
* fixed incorrect letter case in docs
v13
* added contiguous parameter to rte_mempool_get_mem_range
* made rte_mempool_get_mem_range parameters optional
v14
* encapsulated parameters of rte_mempool_get_mem_range in a struct
* added consts to function parameters where appropriate
v15
* fix whitespace in rel notes
* move parameter check to after variable declaration
* change the naming of the range variable in macros
* improve doxygen

Paul Szczepanek (6):
  lib: allow libraries with no sources
  mempool: add functions to get extra mempool info
  ptr_compress: add pointer compression library
  test: add pointer compress tests to ring perf test
  docs: add pointer compression guide
  test: add unit test for ptr compression

 MAINTAINERS                                |   6 +
 app/test/meson.build                       |  21 +-
 app/test/test_mempool.c                    |  70 ++++
 app/test/test_ptr_compress.c               | 193 +++++++++++
 app/test/test_ring.h                       |  94 ++++++
 app/test/test_ring_perf.c                  | 352 ++++++++++++++-------
 doc/api/doxy-api-index.md                  |   1 +
 doc/api/doxy-api.conf.in                   |   1 +
 doc/guides/prog_guide/index.rst            |   1 +
 doc/guides/prog_guide/ptr_compress_lib.rst | 160 ++++++++++
 doc/guides/rel_notes/release_24_07.rst     |   7 +
 lib/mempool/rte_mempool.c                  |  45 +++
 lib/mempool/rte_mempool.h                  |  49 +++
 lib/mempool/version.map                    |   3 +
 lib/meson.build                            |  17 +
 lib/ptr_compress/meson.build               |   4 +
 lib/ptr_compress/rte_ptr_compress.h        | 325 +++++++++++++++++++
 17 files changed, 1217 insertions(+), 132 deletions(-)
 create mode 100644 app/test/test_ptr_compress.c
 create mode 100644 doc/guides/prog_guide/ptr_compress_lib.rst
 create mode 100644 lib/ptr_compress/meson.build
 create mode 100644 lib/ptr_compress/rte_ptr_compress.h

--
2.25.1

[PATCH v15 1/6] lib: allow libraries with no sources

[Paul Szczepanek]

Allow header only libraries.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
 lib/meson.build | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/lib/meson.build b/lib/meson.build
index 179a272932..7c90602bf5 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -222,6 +222,22 @@ foreach l:libraries
     includes += include_directories(l)
     dpdk_includes += include_directories(l)

+    # special case for header only libraries
+    if sources.length() == 0
+        shared_dep = declare_dependency(include_directories: includes,
+                dependencies: shared_deps)
+        static_dep = declare_dependency(include_directories: includes,
+                dependencies: static_deps)
+        set_variable('shared_rte_' + name, shared_dep)
+        set_variable('static_rte_' + name, static_dep)
+        dpdk_shared_lib_deps += shared_dep
+        dpdk_static_lib_deps += static_dep
+        if developer_mode
+            message('lib/@0@: Defining dependency "@1@"'.format(l, name))
+        endif
+        continue
+    endif
+
     if developer_mode and is_windows and use_function_versioning
         message('@0@: Function versioning is not supported by Windows.'.format(name))
     endif
--
2.25.1

[PATCH v15 2/6] mempool: add functions to get extra mempool info

[Paul Szczepanek]

Add two functions:
- rte_mempool_get_mem_range - get virtual memory range
of the objects in the mempool,
- rte_mempool_get_obj_alignment - get alignment of
objects in the mempool.

Add two tests that test these new functions.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Acked-by: Konstantin Ananyev <konstantin.v.ananyev@yandex.ru>
---
 app/test/test_mempool.c   | 70 +++++++++++++++++++++++++++++++++++++++
 lib/mempool/rte_mempool.c | 45 +++++++++++++++++++++++++
 lib/mempool/rte_mempool.h | 49 +++++++++++++++++++++++++++
 lib/mempool/version.map   |  3 ++
 4 files changed, 167 insertions(+)

diff --git a/app/test/test_mempool.c b/app/test/test_mempool.c
index ad7ebd6363..3f7ba5872d 100644
--- a/app/test/test_mempool.c
+++ b/app/test/test_mempool.c
@@ -843,16 +843,19 @@ test_mempool(void)
 	int ret = -1;
 	uint32_t nb_objs = 0;
 	uint32_t nb_mem_chunks = 0;
+	size_t alignment = 0;
 	struct rte_mempool *mp_cache = NULL;
 	struct rte_mempool *mp_nocache = NULL;
 	struct rte_mempool *mp_stack_anon = NULL;
 	struct rte_mempool *mp_stack_mempool_iter = NULL;
 	struct rte_mempool *mp_stack = NULL;
 	struct rte_mempool *default_pool = NULL;
+	struct rte_mempool *mp_alignment = NULL;
 	struct mp_data cb_arg = {
 		.ret = -1
 	};
 	const char *default_pool_ops = rte_mbuf_best_mempool_ops();
+	struct rte_mempool_mem_range_info mem_range = { 0 };

 	/* create a mempool (without cache) */
 	mp_nocache = rte_mempool_create("test_nocache", MEMPOOL_SIZE,
@@ -967,6 +970,72 @@ test_mempool(void)
 	}
 	rte_mempool_obj_iter(default_pool, my_obj_init, NULL);

+	if (rte_mempool_get_mem_range(default_pool, &mem_range)) {
+		printf("cannot get mem range from default mempool\n");
+		GOTO_ERR(ret, err);
+	}
+
+	if (rte_mempool_get_mem_range(NULL, NULL) != -EINVAL) {
+		printf("rte_mempool_get_mem_range failed to return -EINVAL "
+				"when passed invalid arguments\n");
+		GOTO_ERR(ret, err);
+	}
+
+	if (mem_range.start == NULL || mem_range.length <
+			(MEMPOOL_SIZE * MEMPOOL_ELT_SIZE)) {
+		printf("mem range of default mempool is invalid\n");
+		GOTO_ERR(ret, err);
+	}
+
+	/* by default mempool objects are aligned by RTE_MEMPOOL_ALIGN */
+	alignment = rte_mempool_get_obj_alignment(default_pool);
+	if (alignment != RTE_MEMPOOL_ALIGN) {
+		printf("rte_mempool_get_obj_alignment returned wrong value, "
+				"expected %zu, returned %zu\n",
+				(size_t)RTE_MEMPOOL_ALIGN, alignment);
+		GOTO_ERR(ret, err);
+	}
+
+	/* create a mempool with a RTE_MEMPOOL_F_NO_CACHE_ALIGN flag */
+	mp_alignment = rte_mempool_create("test_alignment",
+		1, 8, /* the small size guarantees single memory chunk */
+		0, 0, NULL, NULL, my_obj_init, NULL,
+		SOCKET_ID_ANY, RTE_MEMPOOL_F_NO_CACHE_ALIGN);
+
+	if (mp_alignment == NULL) {
+		printf("cannot allocate mempool with "
+				"RTE_MEMPOOL_F_NO_CACHE_ALIGN flag\n");
+		GOTO_ERR(ret, err);
+	}
+
+	/* mempool was created with RTE_MEMPOOL_F_NO_CACHE_ALIGN
+	 * and minimum alignment is expected which is sizeof(uint64_t)
+	 */
+	alignment = rte_mempool_get_obj_alignment(mp_alignment);
+	if (alignment != sizeof(uint64_t)) {
+		printf("rte_mempool_get_obj_alignment returned wrong value, "
+				"expected %zu, returned %zu\n",
+				(size_t)sizeof(uint64_t), alignment);
+		GOTO_ERR(ret, err);
+	}
+
+	alignment = rte_mempool_get_obj_alignment(NULL);
+	if (alignment != 0) {
+		printf("rte_mempool_get_obj_alignment failed to return 0 for "
+				" an invalid mempool\n");
+		GOTO_ERR(ret, err);
+	}
+
+	if (rte_mempool_get_mem_range(mp_alignment, &mem_range)) {
+		printf("cannot get mem range from mempool\n");
+		GOTO_ERR(ret, err);
+	}
+
+	if (!mem_range.is_contiguous) {
+		printf("mempool not contiguous\n");
+		GOTO_ERR(ret, err);
+	}
+
 	/* retrieve the mempool from its name */
 	if (rte_mempool_lookup("test_nocache") != mp_nocache) {
 		printf("Cannot lookup mempool from its name\n");
@@ -1039,6 +1108,7 @@ test_mempool(void)
 	rte_mempool_free(mp_stack_mempool_iter);
 	rte_mempool_free(mp_stack);
 	rte_mempool_free(default_pool);
+	rte_mempool_free(mp_alignment);

 	return ret;
 }
diff --git a/lib/mempool/rte_mempool.c b/lib/mempool/rte_mempool.c
index 12390a2c81..71100bff4f 100644
--- a/lib/mempool/rte_mempool.c
+++ b/lib/mempool/rte_mempool.c
@@ -1386,6 +1386,51 @@ void rte_mempool_walk(void (*func)(struct rte_mempool *, void *),
 	rte_mcfg_mempool_read_unlock();
 }

+int rte_mempool_get_mem_range(const struct rte_mempool *mp,
+		struct rte_mempool_mem_range_info *mem_range)
+{
+	void *address_low = (void *)UINTPTR_MAX;
+	void *address_high = 0;
+	size_t address_diff = 0;
+	size_t total_size = 0;
+	struct rte_mempool_memhdr *hdr;
+
+	if (mp == NULL || mem_range == NULL)
+		return -EINVAL;
+
+	/* go through memory chunks and find the lowest and highest addresses */
+	STAILQ_FOREACH(hdr, &mp->mem_list, next) {
+		if (address_low > hdr->addr)
+			address_low = hdr->addr;
+		if (address_high < RTE_PTR_ADD(hdr->addr, hdr->len))
+			address_high = RTE_PTR_ADD(hdr->addr, hdr->len);
+		total_size += hdr->len;
+	}
+
+	/* check if mempool was not populated yet (no memory chunks) */
+	if (address_low == (void *)UINTPTR_MAX)
+		return -EINVAL;
+
+	address_diff = (size_t)RTE_PTR_DIFF(address_high, address_low);
+
+	mem_range->start = address_low;
+	mem_range->length = address_diff;
+	mem_range->is_contiguous = (total_size == address_diff) ? true : false;
+
+	return 0;
+}
+
+size_t rte_mempool_get_obj_alignment(const struct rte_mempool *mp)
+{
+	if (mp == NULL)
+		return 0;
+
+	if (mp->flags & RTE_MEMPOOL_F_NO_CACHE_ALIGN)
+		return sizeof(uint64_t);
+	else
+		return RTE_MEMPOOL_ALIGN;
+}
+
 struct mempool_callback_data {
 	TAILQ_ENTRY(mempool_callback_data) callbacks;
 	rte_mempool_event_callback *func;
diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h
index 23fd5c8465..e11282763d 100644
--- a/lib/mempool/rte_mempool.h
+++ b/lib/mempool/rte_mempool.h
@@ -1917,6 +1917,55 @@ uint32_t rte_mempool_calc_obj_size(uint32_t elt_size, uint32_t flags,
 void rte_mempool_walk(void (*func)(struct rte_mempool *, void *arg),
 		      void *arg);

+/**
+ * A structure used to retrieve information about the memory range
+ * of the mempool.
+ */
+struct rte_mempool_mem_range_info {
+	/** Start of the memory range used by mempool objects */
+	void *start;
+	/** Length of the memory range used by mempool objects */
+	size_t length;
+	/** Are all memory addresses used by mempool objects contiguous */
+	bool is_contiguous;
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Get information about the memory range used by the mempool.
+ *
+ * @param[in] mp
+ *   Pointer to an initialized mempool.
+ * @param[out] mem_range
+ *   Pointer to struct which is used to return lowest address,
+ *   length of the memory range containing all the addresses,
+ *   and whether these addresses are contiguous.
+ * @return
+ *   0 on success, -EINVAL if mempool is not valid or mem_range is NULL.
+ **/
+__rte_experimental
+int
+rte_mempool_get_mem_range(const struct rte_mempool *mp,
+		struct rte_mempool_mem_range_info *mem_range);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Return object alignment.
+ *
+ * @param[in] mp
+ *   Pointer to a mempool.
+ * @return
+ *   Object alignment if mp is valid. 0 if mp is NULL.
+ *
+ **/
+__rte_experimental
+size_t
+rte_mempool_get_obj_alignment(const struct rte_mempool *mp);
+
 /**
  * @internal Get page size used for mempool object allocation.
  * This function is internal to mempool library and mempool drivers.
diff --git a/lib/mempool/version.map b/lib/mempool/version.map
index 473277400c..02df634b2a 100644
--- a/lib/mempool/version.map
+++ b/lib/mempool/version.map
@@ -50,6 +50,9 @@ EXPERIMENTAL {
 	__rte_mempool_trace_get_contig_blocks;
 	__rte_mempool_trace_default_cache;
 	__rte_mempool_trace_cache_flush;
+	# added in 24.07
+	rte_mempool_get_mem_range;
+	rte_mempool_get_obj_alignment;
 };

 INTERNAL {
--
2.25.1

[PATCH v15 3/6] ptr_compress: add pointer compression library

[Paul Szczepanek]

Add a new utility header for compressing pointers. The provided
functions can store pointers as 32-bit or 16-bit offsets.

The compression takes advantage of the fact that pointers are
usually located in a limited memory region (like a mempool).
We can compress them by converting them to offsets from a base
memory address. Offsets can be stored in fewer bytes (dictated
by the memory region size and alignment of the pointer).
For example: an 8 byte aligned pointer which is part of a 32GB
memory pool can be stored in 4 bytes.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
---
 MAINTAINERS                            |   4 +
 doc/api/doxy-api-index.md              |   1 +
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/rel_notes/release_24_07.rst |   5 +
 lib/meson.build                        |   1 +
 lib/ptr_compress/meson.build           |   4 +
 lib/ptr_compress/rte_ptr_compress.h    | 325 +++++++++++++++++++++++++
 7 files changed, 341 insertions(+)
 create mode 100644 lib/ptr_compress/meson.build
 create mode 100644 lib/ptr_compress/rte_ptr_compress.h

diff --git a/MAINTAINERS b/MAINTAINERS
index c9adff9846..27b2f03e6c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1694,6 +1694,10 @@ M: Chenbo Xia <chenbox@nvidia.com>
 M: Gaetan Rivet <grive@u256.net>
 F: lib/pci/

+Pointer Compression
+M: Paul Szczepanek <paul.szczepanek@arm.com>
+F: lib/ptr_compress/
+
 Power management
 M: Anatoly Burakov <anatoly.burakov@intel.com>
 M: David Hunt <david.hunt@intel.com>
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 8c1eb8fafa..f9283154f8 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -222,6 +222,7 @@ The public API headers are grouped by topics:
   [config file](@ref rte_cfgfile.h),
   [key/value args](@ref rte_kvargs.h),
   [argument parsing](@ref rte_argparse.h),
+  [ptr_compress](@ref rte_ptr_compress.h),
   [string](@ref rte_string_fns.h),
   [thread](@ref rte_thread.h)

diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index 27afec8b3b..a8823c046f 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -71,6 +71,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/pipeline \
                           @TOPDIR@/lib/port \
                           @TOPDIR@/lib/power \
+                          @TOPDIR@/lib/ptr_compress \
                           @TOPDIR@/lib/rawdev \
                           @TOPDIR@/lib/rcu \
                           @TOPDIR@/lib/regexdev \
diff --git a/doc/guides/rel_notes/release_24_07.rst b/doc/guides/rel_notes/release_24_07.rst
index a69f24cf99..4711792e61 100644
--- a/doc/guides/rel_notes/release_24_07.rst
+++ b/doc/guides/rel_notes/release_24_07.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================

+* **Introduced pointer compression library.**
+
+  Library provides functions to compress and decompress arrays of pointers
+  which can improve application performance under certain conditions.
+  Performance test was added to help users evaluate performance on their setup.

 Removed Items
 -------------
diff --git a/lib/meson.build b/lib/meson.build
index 7c90602bf5..63becee142 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -14,6 +14,7 @@ libraries = [
         'argparse',
         'telemetry', # basic info querying
         'eal', # everything depends on eal
+        'ptr_compress',
         'ring',
         'rcu', # rcu depends on ring
         'mempool',
diff --git a/lib/ptr_compress/meson.build b/lib/ptr_compress/meson.build
new file mode 100644
index 0000000000..e92706a45f
--- /dev/null
+++ b/lib/ptr_compress/meson.build
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+headers = files('rte_ptr_compress.h')
diff --git a/lib/ptr_compress/rte_ptr_compress.h b/lib/ptr_compress/rte_ptr_compress.h
new file mode 100644
index 0000000000..b9ab17b2db
--- /dev/null
+++ b/lib/ptr_compress/rte_ptr_compress.h
@@ -0,0 +1,325 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2024 Arm Limited
+ */
+
+#ifndef RTE_PTR_COMPRESS_H
+#define RTE_PTR_COMPRESS_H
+
+/**
+ * @file
+ * Pointer compression and decompression functions.
+ *
+ * When passing arrays full of pointers between threads, memory containing
+ * the pointers is copied multiple times which is especially costly between
+ * cores. These functions allow us to compress the pointers.
+ *
+ * Compression takes advantage of the fact that pointers are usually located in
+ * a limited memory region. We compress them by converting them to offsets from
+ * a base memory address. Offsets can be stored in fewer bytes.
+ *
+ * The compression functions come in two varieties: 32-bit and 16-bit.
+ *
+ * To determine how many bits are needed to compress the pointer, calculate
+ * the biggest offset possible (highest value pointer - base pointer)
+ * and shift the value right according to alignment (shift by exponent of the
+ * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by
+ * 3, etc.). The resulting value must fit in either 32 or 16 bits. You may
+ * use the macros provided in this file to do it programmatically.
+ *
+ * For usage example and further explanation please see this library's
+ * documentation in the programming guide.
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_vect.h>
+#include <rte_bitops.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Calculate how many bits are required to store pointers within a given memory
+ * region as offsets. This can help decide which pointer compression functions
+ * can be used.
+ *
+ * @param mem_length
+ *   Length of the memory region the pointers are constrained to.
+ * @return
+ *   Number of bits required to store a value.
+ **/
+#define RTE_PTR_COMPRESS_BITS_NEEDED_FOR_POINTER_WITHIN_RANGE(mem_length) \
+	(((uint64_t)mem_length) < 2 ? 1 : \
+		(sizeof(uint64_t) * CHAR_BIT - \
+		 rte_clz64((uint64_t)mem_length - 1)))
+
+/**
+ * Calculate how many bits in the address can be dropped without losing any
+ * information thanks to the alignment of the address.
+ *
+ * @param alignment
+ *   Memory alignment.
+ * @return
+ *   Size of shift allowed without dropping any information from the pointer.
+ **/
+#define RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(alignment) \
+	((alignment) == 0 ? 0 : rte_ctz64((uint64_t)alignment))
+
+/**
+ * Determine if rte_ptr_compress_16_shift can be used to compress pointers
+ * that contain addresses of memory objects whose memory is aligned by
+ * a given amount and contained in a given memory region.
+ *
+ * @param mem_length
+ *   The length of the memory region that contains the objects pointed to.
+ * @param obj_alignment
+ *   The alignment of objects pointed to.
+ * @return
+ *   1 if function can be used, 0 otherwise.
+ **/
+#define RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT(mem_length, obj_alignment) \
+	((RTE_PTR_COMPRESS_BITS_NEEDED_FOR_POINTER_WITHIN_RANGE(mem_length) - \
+	RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(obj_alignment)) <= 16 ? 1 : 0)
+
+/**
+ * Determine if rte_ptr_compress_32_shift can be used to compress pointers
+ * that contain addresses of memory objects whose memory is aligned by
+ * a given amount and contained in a given memory region.
+ *
+ * @param mem_length
+ *   The length of the memory region that contains the objects pointed to.
+ * @param obj_alignment
+ *   The alignment of objects pointed to.
+ * @return
+ *   1 if function can be used, 0 otherwise.
+ **/
+#define RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT(mem_length, obj_alignment) \
+	((RTE_PTR_COMPRESS_BITS_NEEDED_FOR_POINTER_WITHIN_RANGE(mem_length) - \
+	RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(obj_alignment)) <= 32 ? 1 : 0)
+
+/**
+ * Compress pointers into 32-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 32 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 32GB of the base pointer. If there is no such alignment guarantee they
+ * must be within 4GB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_32_shift(void *ptr_base, void * const *src_table,
+		uint32_t *dest_table, size_t n, uint8_t bit_shift)
+{
+	size_t i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	do {
+		svbool_t pg = svwhilelt_b64(i, n);
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1w(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+	} while (i < n);
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	/* right shift is done by left shifting by negative int */
+	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	const size_t n_even = n & ~0x1;
+	for (; i < n_even; i += 2) {
+		v_ptr_table = vld1q_u64((const uint64_t *)src_table + i);
+		v_ptr_table = vsubq_u64(v_ptr_table, v_ptr_base);
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		vst1_u32(dest_table + i, vqmovn_u64(v_ptr_table));
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT32_MAX);
+		dest_table[i] = (uint32_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 32-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_32_shift(void *ptr_base, uint32_t const *src_table,
+		void **dest_table, size_t n, uint8_t bit_shift)
+{
+	size_t i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	do {
+		svbool_t pg = svwhilelt_b64(i, n);
+		v_ptr_table = svld1uw_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+	} while (i < n);
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	int64x2_t v_shift = vdupq_n_s64(bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	const size_t n_even = n & ~0x1;
+	for (; i < n_even; i += 2) {
+		v_ptr_table = vmovl_u32(vld1_u32(src_table + i));
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		v_ptr_table = vaddq_u64(v_ptr_table, v_ptr_base);
+		vst1q_u64((uint64_t *)dest_table + i, v_ptr_table);
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+/**
+ * Compress pointers into 16-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 16 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 256KB of the base pointer. If there is no such alignment guarantee
+ * they must be within 64KB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_16_shift(void *ptr_base, void * const *src_table,
+		uint16_t *dest_table, size_t n, uint8_t bit_shift)
+{
+
+	size_t i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	do {
+		svbool_t pg = svwhilelt_b64(i, n);
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1h(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+	} while (i < n);
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT16_MAX);
+		dest_table[i] = (uint16_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 16-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_16_shift(void *ptr_base, uint16_t const *src_table,
+		void **dest_table, size_t n, uint8_t bit_shift)
+{
+	size_t i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	do {
+		svbool_t pg = svwhilelt_b64(i, n);
+		v_ptr_table = svld1uh_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+	} while (i < n);
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_PTR_COMPRESS_H */
--
2.25.1

[PATCH v15 4/6] test: add pointer compress tests to ring perf test

[Paul Szczepanek]

Add a test that runs a zero copy burst enqueue and dequeue on a ring
of raw pointers and compressed pointers at different burst sizes to
showcase performance benefits of newly added pointer compression APIs.

Refactored threading code to pass more parameters to threads to
reuse existing code. Added more bulk sizes to showcase their effects
on compression. Adjusted loop iteration numbers to take into account
bulk sizes to keep runtime constant (instead of number of operations).

Adjusted old printfs to match new ones which have aligned numbers.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
---
 app/test/meson.build      |  20 +--
 app/test/test_ring.h      |  94 ++++++++++
 app/test/test_ring_perf.c | 352 +++++++++++++++++++++++++-------------
 3 files changed, 334 insertions(+), 132 deletions(-)

diff --git a/app/test/meson.build b/app/test/meson.build
index 7d909039ae..df8cc00730 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -156,16 +156,16 @@ source_file_deps = {
 #    'test_resource.c': [],
     'test_rib.c': ['net', 'rib'],
     'test_rib6.c': ['net', 'rib'],
-    'test_ring.c': [],
-    'test_ring_hts_stress.c': [],
-    'test_ring_mpmc_stress.c': [],
-    'test_ring_mt_peek_stress.c': [],
-    'test_ring_mt_peek_stress_zc.c': [],
-    'test_ring_perf.c': [],
-    'test_ring_rts_stress.c': [],
-    'test_ring_st_peek_stress.c': [],
-    'test_ring_st_peek_stress_zc.c': [],
-    'test_ring_stress.c': [],
+    'test_ring.c': ['ptr_compress'],
+    'test_ring_hts_stress.c': ['ptr_compress'],
+    'test_ring_mpmc_stress.c': ['ptr_compress'],
+    'test_ring_mt_peek_stress.c': ['ptr_compress'],
+    'test_ring_mt_peek_stress_zc.c': ['ptr_compress'],
+    'test_ring_perf.c': ['ptr_compress'],
+    'test_ring_rts_stress.c': ['ptr_compress'],
+    'test_ring_st_peek_stress.c': ['ptr_compress'],
+    'test_ring_st_peek_stress_zc.c': ['ptr_compress'],
+    'test_ring_stress.c': ['ptr_compress'],
     'test_rwlock.c': [],
     'test_sched.c': ['net', 'sched'],
     'test_security.c': ['net', 'security'],
diff --git a/app/test/test_ring.h b/app/test/test_ring.h
index 45c263f3ff..9e97c5e3e7 100644
--- a/app/test/test_ring.h
+++ b/app/test/test_ring.h
@@ -5,6 +5,8 @@
 #include <rte_malloc.h>
 #include <rte_ring.h>
 #include <rte_ring_elem.h>
+#include <rte_memcpy.h>
+#include <rte_ptr_compress.h>

 /* API type to call
  * rte_ring_<sp/mp or sc/mc>_enqueue_<bulk/burst>
@@ -25,6 +27,10 @@
 #define TEST_RING_ELEM_BULK 16
 #define TEST_RING_ELEM_BURST 32

+#define TEST_RING_ELEM_BURST_ZC 64
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 128
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 256
+
 #define TEST_RING_IGNORE_API_TYPE ~0U

 /* This function is placed here as it is required for both
@@ -101,6 +107,9 @@ static inline unsigned int
 test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -152,6 +161,47 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n,
 								NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy(zcd.ptr2,
+						(char *)obj + zcd.n1 * esize,
+						(ret - zcd.n1) * esize);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_16_shift(
+					0, obj, zcd.ptr1, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_16_shift(0,
+						obj + (zcd.n1 * 2),
+						zcd.ptr2,
+						(ret - zcd.n1) * 2, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_32_shift(0, obj, zcd.ptr1, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_32_shift(0, obj + zcd.n1,
+						zcd.ptr2, ret - zcd.n1, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
@@ -162,6 +212,9 @@ static inline unsigned int
 test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -213,6 +266,47 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mc_dequeue_burst_elem(r, obj, esize,
 								n, NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy((char *)obj + zcd.n1 * esize,
+						zcd.ptr2,
+						(ret - zcd.n1) * esize);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_16_shift(
+					0, zcd.ptr1, obj, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_16_shift(0, zcd.ptr2,
+						obj + zcd.n1,
+						(ret - zcd.n1) * 2,
+						3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_32_shift(0, zcd.ptr1, obj, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_32_shift(0, zcd.ptr2,
+						obj + zcd.n1, ret - zcd.n1, 3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c
index ee328614c1..57cd04a124 100644
--- a/app/test/test_ring_perf.c
+++ b/app/test/test_ring_perf.c
@@ -22,13 +22,13 @@

 #define RING_NAME "RING_PERF"
 #define RING_SIZE 4096
-#define MAX_BURST 32
+#define MAX_BURST 256

 /*
  * the sizes to enqueue and dequeue in testing
  * (marked volatile so they won't be seen as compile-time constants)
  */
-static const volatile unsigned bulk_sizes[] = { 8, 32 };
+static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128, 256 };

 struct lcore_pair {
 	unsigned c1, c2;
@@ -43,26 +43,30 @@ test_ring_print_test_string(unsigned int api_type, int esize,
 	if (esize == -1)
 		printf("legacy APIs");
 	else
-		printf("elem APIs: element size %dB", esize);
+		printf("elem APIs (size:%2dB)", esize);

 	if (api_type == TEST_RING_IGNORE_API_TYPE)
 		return;

 	if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF)
-		printf(": default enqueue/dequeue: ");
+		printf(" - default enqueue/dequeue");
 	else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC)
-		printf(": SP/SC: ");
+		printf(" - SP/SC");
 	else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC)
-		printf(": MP/MC: ");
+		printf(" - MP/MC");

 	if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE)
-		printf("single: ");
+		printf(" - single - ");
 	else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK)
-		printf("bulk (size: %u): ", bsz);
+		printf(" - bulk (n:%-3u) - ", bsz);
 	else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST)
-		printf("burst (size: %u): ", bsz);
+		printf(" - burst (n:%-3u) - ", bsz);
+	else if ((api_type & (TEST_RING_ELEM_BURST_ZC |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0)
+		printf(" - burst zero copy (n:%-3u) - ", bsz);

-	printf("%.2F\n", value);
+	printf("cycles per elem: %.3F\n", value);
 }

 /**** Functions to analyse our core mask to get cores for different tests ***/
@@ -144,28 +148,33 @@ static void
 test_empty_dequeue(struct rte_ring *r, const int esize,
 			const unsigned int api_type)
 {
-	const unsigned int iter_shift = 26;
+	const unsigned int iter_shift = 29;
 	const unsigned int iterations = 1 << iter_shift;
 	unsigned int i = 0;
 	void *burst[MAX_BURST];

+	const unsigned int bulk_iterations = iterations / bulk_sizes[0];
 	const uint64_t start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	for (i = 0; i < bulk_iterations; i++)
 		test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type);
 	const uint64_t end = rte_rdtsc();

 	test_ring_print_test_string(api_type, esize, bulk_sizes[0],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 }

-/*
- * for the separate enqueue and dequeue threads they take in one param
- * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc
- */
-struct thread_params {
+/* describes the ring used by the enqueue and dequeue thread */
+struct ring_params {
 	struct rte_ring *r;
-	unsigned size;        /* input value, the burst size */
-	double spsc, mpmc;    /* output value, the single or multi timings */
+	unsigned int elem_size;
+	unsigned int bulk_sizes_i; /* index into bulk_size array */
+	unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */
+};
+
+/* Used to specify enqueue and dequeue ring operations and their results */
+struct thread_params {
+	struct ring_params *ring_params;
+	double *results; /* result array size must be equal to bulk_sizes */
 };

 /*
@@ -174,16 +183,15 @@ struct thread_params {
  * flag == 1 -> dequeue
  */
 static __rte_always_inline int
-enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
-	struct thread_params *p)
+enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 15;
+	const unsigned int iter_shift = 22;
 	const unsigned int iterations = 1 << iter_shift;
-	struct rte_ring *r = p->r;
-	unsigned int bsize = p->size;
 	unsigned int i;
 	void *burst = NULL;
+	unsigned int n_remaining;
+	const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i];

 #ifdef RTE_USE_C11_MEM_MODEL
 	if (rte_atomic_fetch_add_explicit(&lcore_count, 1, rte_memory_order_relaxed) + 1 != 2)
@@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 		while(lcore_count != 2)
 			rte_pause();

-	burst = test_ring_calloc(MAX_BURST, esize);
+	burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size);
 	if (burst == NULL)
 		return -1;

 	const uint64_t sp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	const unsigned int bulk_iterations = iterations / bulk_n;
+	for (i = 0; i < bulk_iterations; i++) {
+		n_remaining = bulk_n;
 		do {
 			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_enqueue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_dequeue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			if (ret == 0)
 				rte_pause();
-		} while (!ret);
+			else
+				n_remaining -= ret;
+		} while (n_remaining > 0);
+	}
 	const uint64_t sp_end = rte_rdtsc();

-	const uint64_t mp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
-		do {
-			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			if (ret == 0)
-				rte_pause();
-		} while (!ret);
-	const uint64_t mp_end = rte_rdtsc();
+	p->results[p->ring_params->bulk_sizes_i] =
+			((double)sp_end - sp_start) / iterations;

-	p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize);
-	p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize);
 	return 0;
 }

@@ -243,15 +245,7 @@ enqueue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(0, -1, params);
-}
-
-static int
-enqueue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(0, 16, params);
+	return enqueue_dequeue_bulk_helper(0, params);
 }

 /*
@@ -263,15 +257,7 @@ dequeue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(1, -1, params);
-}
-
-static int
-dequeue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(1, 16, params);
+	return enqueue_dequeue_bulk_helper(1, params);
 }

 /*
@@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p)
  * used to measure ring perf between hyperthreads, cores and sockets.
  */
 static int
-run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize)
+run_on_core_pair(struct lcore_pair *cores,
+		struct thread_params *param1, struct thread_params *param2)
 {
-	lcore_function_t *f1, *f2;
-	struct thread_params param1 = {0}, param2 = {0};
 	unsigned i;
-
-	if (esize == -1) {
-		f1 = enqueue_bulk;
-		f2 = dequeue_bulk;
-	} else {
-		f1 = enqueue_bulk_16B;
-		f2 = dequeue_bulk_16B;
-	}
+	struct ring_params *ring_params = param1->ring_params;

 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		lcore_count = 0;
-		param1.size = param2.size = bulk_sizes[i];
-		param1.r = param2.r = r;
+		ring_params->bulk_sizes_i = i;
 		if (cores->c1 == rte_get_main_lcore()) {
-			rte_eal_remote_launch(f2, &param2, cores->c2);
-			f1(&param1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
+			enqueue_bulk(param1);
 			rte_eal_wait_lcore(cores->c2);
 		} else {
-			rte_eal_remote_launch(f1, &param1, cores->c1);
-			rte_eal_remote_launch(f2, &param2, cores->c2);
+			rte_eal_remote_launch(enqueue_bulk, param1, cores->c1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
 			if (rte_eal_wait_lcore(cores->c1) < 0)
 				return -1;
 			if (rte_eal_wait_lcore(cores->c2) < 0)
 				return -1;
 		}
 		test_ring_print_test_string(
-			TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.spsc + param2.spsc);
-		test_ring_print_test_string(
-			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.mpmc + param2.mpmc);
+				ring_params->ring_flags,
+				ring_params->elem_size,
+				bulk_sizes[i],
+				param1->results[i] + param2->results[i]);
 	}

 	return 0;
@@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 	uint64_t hz = rte_get_timer_hz();
 	uint64_t lcount = 0;
 	const unsigned int lcore = rte_lcore_id();
-	struct thread_params *params = p;
+	struct ring_params *ring_params = p->ring_params;
 	void *burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -347,9 +323,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize)

 	begin = rte_get_timer_cycles();
 	while (time_diff < hz * TIME_MS / 1000) {
-		test_ring_enqueue(params->r, burst, esize, params->size,
+		test_ring_enqueue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
-		test_ring_dequeue(params->r, burst, esize, params->size,
+		test_ring_dequeue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 		lcount++;
 		time_diff = rte_get_timer_cycles() - begin;
@@ -381,7 +359,8 @@ static int
 run_on_all_cores(struct rte_ring *r, const int esize)
 {
 	uint64_t total;
-	struct thread_params param;
+	struct ring_params ring_params = {0};
+	struct thread_params params = { .ring_params = &ring_params };
 	lcore_function_t *lcore_f;
 	unsigned int i, c;

@@ -390,21 +369,20 @@ run_on_all_cores(struct rte_ring *r, const int esize)
 	else
 		lcore_f = load_loop_fn_16B;

-	memset(&param, 0, sizeof(struct thread_params));
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		total = 0;
 		printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]);
-		param.size = bulk_sizes[i];
-		param.r = r;
+		params.ring_params->bulk_sizes_i = i;
+		params.ring_params->r = r;

 		/* clear synchro and start workers */
 		rte_atomic_store_explicit(&synchro, 0, rte_memory_order_relaxed);
-		if (rte_eal_mp_remote_launch(lcore_f, &param, SKIP_MAIN) < 0)
+		if (rte_eal_mp_remote_launch(lcore_f, &params, SKIP_MAIN) < 0)
 			return -1;

 		/* start synchro and launch test on main */
 		rte_atomic_store_explicit(&synchro, 1, rte_memory_order_relaxed);
-		lcore_f(&param);
+		lcore_f(&params);

 		rte_eal_mp_wait_lcore();

@@ -463,9 +441,9 @@ static int
 test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	const unsigned int api_type)
 {
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 26;
 	const unsigned int iterations = 1 << iter_shift;
-	unsigned int sz, i = 0;
+	unsigned int sz, i;
 	void **burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -473,17 +451,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 		return -1;

 	for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) {
+		const unsigned int n = iterations / bulk_sizes[sz];
 		const uint64_t start = rte_rdtsc();
-		for (i = 0; i < iterations; i++) {
+		for (i = 0; i < n; i++) {
 			test_ring_enqueue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 			test_ring_dequeue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 		}
 		const uint64_t end = rte_rdtsc();

 		test_ring_print_test_string(api_type, esize, bulk_sizes[sz],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 	}

 	rte_free(burst);
@@ -491,12 +470,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	return 0;
 }

+static __rte_always_inline int
+test_ring_perf_esize_run_on_two_cores(
+		struct thread_params *param1, struct thread_params *param2)
+{
+	struct lcore_pair cores;
+
+	if (get_two_hyperthreads(&cores) == 0) {
+		printf("\n### Testing using two hyperthreads ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing using two physical cores ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_sockets(&cores) == 0) {
+		printf("\n### Testing using two NUMA nodes ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	return 0;
+}
+
 /* Run all tests for a given element size */
 static __rte_always_inline int
 test_ring_perf_esize(const int esize)
 {
-	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
+	double results_enq[RTE_DIM(bulk_sizes)];
+	double results_deq[RTE_DIM(bulk_sizes)];
+	struct ring_params ring_params = {
+			.elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results_enq };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results_deq };

 	/*
 	 * Performance test for legacy/_elem APIs
@@ -536,22 +546,13 @@ test_ring_perf_esize(const int esize)
 	test_empty_dequeue(r, esize,
 			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);

-	if (get_two_hyperthreads(&cores) == 0) {
-		printf("\n### Testing using two hyperthreads ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.r = r;

-	if (get_two_cores(&cores) == 0) {
-		printf("\n### Testing using two physical cores ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
-	if (get_two_sockets(&cores) == 0) {
-		printf("\n### Testing using two NUMA nodes ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
+
+	ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);

 	printf("\n### Testing using all worker nodes ###\n");
 	if (run_on_all_cores(r, esize) < 0)
@@ -567,6 +568,109 @@ test_ring_perf_esize(const int esize)
 	return -1;
 }

+
+static __rte_always_inline int
+test_ring_perf_compression(void)
+{
+	double results1[RTE_DIM(bulk_sizes)];
+	double results2[RTE_DIM(bulk_sizes)];
+	double results1_comp[2][RTE_DIM(bulk_sizes)];
+	double results2_comp[2][RTE_DIM(bulk_sizes)];
+
+	struct lcore_pair cores;
+	int ret = -1;
+	unsigned int i, j;
+	struct ring_params ring_params = { .elem_size = sizeof(void *) };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results1 };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results2 };
+
+	printf("\n### Testing compression gain ###");
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(void *),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy ###\n");
+		ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	/* rings allow only multiples of 4 as sizes,
+	 * we allocate size 4 despite only using 2 bytes
+	 * and use half of RING_SIZE as the number of elements
+	 */
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE / 2, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[0];
+	param2.results = results2_comp[0];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (16b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[1];
+	param2.results = results2_comp[1];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (32b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	for (j = 0; j < 2; j++) {
+		printf("\n### Potential gain from compression (%d-bit offsets) "
+		"###\n", (j + 1) * 16);
+		for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
+			const double result = results1[i] + results2[i];
+			const double result_comp = results1_comp[j][i] +
+				results2_comp[j][i];
+			const double gain = 100 - (result_comp / result) * 100;
+
+			printf("Gain of %5.1F%% for burst of %-3u elems\n",
+					gain, bulk_sizes[i]);
+		}
+	}
+
+	return ret;
+}
+
 static int
 test_ring_perf(void)
 {
@@ -577,6 +681,10 @@ test_ring_perf(void)
 	if (test_ring_perf_esize(16) == -1)
 		return -1;

+	/* Test for performance gain of compression */
+	if (test_ring_perf_compression() == -1)
+		return -1;
+
 	return 0;
 }

--
2.25.1

[PATCH v15 5/6] docs: add pointer compression guide

[Paul Szczepanek]

Documentation added in the prog guide for the new
utility functions for pointer compression
showing example code and potential usecases.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
---
 MAINTAINERS                                |   1 +
 doc/guides/prog_guide/index.rst            |   1 +
 doc/guides/prog_guide/ptr_compress_lib.rst | 160 +++++++++++++++++++++
 doc/guides/rel_notes/release_24_07.rst     |   2 +
 4 files changed, 164 insertions(+)
 create mode 100644 doc/guides/prog_guide/ptr_compress_lib.rst

diff --git a/MAINTAINERS b/MAINTAINERS
index 27b2f03e6c..ed50121bd2 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1697,6 +1697,7 @@ F: lib/pci/
 Pointer Compression
 M: Paul Szczepanek <paul.szczepanek@arm.com>
 F: lib/ptr_compress/
+F: doc/guides/prog_guide/ptr_compress_lib.rst

 Power management
 M: Anatoly Burakov <anatoly.burakov@intel.com>
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index d09d958e6c..6366849eb0 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -73,6 +73,7 @@ Programmer's Guide
     telemetry_lib
     bpf_lib
     graph_lib
+    ptr_compress_lib
     build-sdk-meson
     meson_ut
     build_app
diff --git a/doc/guides/prog_guide/ptr_compress_lib.rst b/doc/guides/prog_guide/ptr_compress_lib.rst
new file mode 100644
index 0000000000..5edbc35e56
--- /dev/null
+++ b/doc/guides/prog_guide/ptr_compress_lib.rst
@@ -0,0 +1,160 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2024 Arm Limited.
+
+Pointer Compression Library
+===========================
+
+Use ``rte_ptr_compress_16_shift()`` and ``rte_ptr_decompress_16_shift()`` to
+compress and decompress pointers into 16-bit offsets.
+Use ``rte_ptr_compress_32_shift()`` and ``rte_ptr_decompress_32_shift()`` to
+compress and decompress pointers into 32-bit offsets.
+
+Compression takes advantage of the fact that pointers are usually located in a
+limited memory region (like a mempool). By converting them to offsets from a
+base memory address they can be stored in fewer bytes. How many bytes are needed
+to store the offset is dictated by the memory region size and alignment of
+objects the pointers point to.
+
+For example, a pointer which is part of a 4GB memory pool can be stored as 32
+bit offset. If the pointer points to memory that is 8 bytes aligned then 3 bits
+can be dropped from the offset and a 32GB memory pool can now fit in 32 bits.
+
+For performance reasons these requirements are not enforced programmatically.
+The programmer is responsible for ensuring that the combination of distance
+from the base pointer and memory alignment allow for storing of the offset in
+the number of bits indicated by the function name (16 or 32). Start of mempool
+memory would be a good candidate for the base pointer. Otherwise any pointer
+that precedes all pointers, is close enough and has the same alignment as the
+pointers being compressed will work.
+
+Macros present in the rte_ptr_compress.h header may be used to evaluate whether
+compression is possible:
+
+*   RTE_PTR_COMPRESS_BITS_NEEDED_FOR_POINTER_WITHIN_RANGE
+
+*   RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT
+
+*   RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT
+
+*   RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT
+
+These will help you calculate compression parameters and whether these are
+legal for particular compression function.
+
+If using an rte_mempool you can get the parameters you need to use in the
+compression macros and functions by using ``rte_mempool_get_mem_range()``
+and ``rte_mempool_get_obj_alignment()``.
+
+.. note::
+
+    Performance gains depend on the batch size of pointers and CPU capabilities
+    such as vector extensions. It's important to measure the performance
+    increase on target hardware. A test called ``ring_perf_autotest`` in
+    ``dpdk-test`` can provide the measurements.
+
+Example usage
+~~~~~~~~~~~~~
+
+In this example we send pointers between two cores through a ring. While this
+is a realistic use case the code is simplified for demonstration purposes and
+does not have error handling.
+
+.. code-block:: c
+
+    #include <rte_launch.h>
+    #include <rte_ring.h>
+    #include <rte_ring_elem.h>
+    #include <rte_ptr_compress.h>
+
+    #define ITEMS_ARRAY_SIZE (1024)
+    #define BATCH_SIZE (128)
+    #define ALIGN_EXPONENT (3)
+    #define ITEM_ALIGN (1<<ALIGN_EXPONENT)
+    #define CORE_SEND (1)
+    #define CORE_RECV (2)
+
+    struct item {
+      alignas(ITEM_ALIGN) int a;
+    };
+
+    static struct item items[ITEMS_ARRAY_SIZE] = {0};
+    static struct rte_ring *ring = NULL;
+
+    static int
+    send_compressed(void *args)
+    {
+      struct item *ptrs_send[BATCH_SIZE] = {0};
+      unsigned int n_send = 0;
+      struct rte_ring_zc_data zcd = {0};
+
+      /* in this example we only fill the ptrs_send once and reuse */
+      for (;n_send < BATCH_SIZE; n_send++)
+        ptrs_send[n_send] = &items[n_send];
+
+      for(;;) {
+        n_send = rte_ring_enqueue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        /* compress ptrs_send into offsets */
+        rte_ptr_compress_32_shift(items, /* base pointer */
+          ptrs_send, /* source array to be compressed */
+          zcd.ptr1, /* destination array to store offsets */
+          zcd.n1, /* how many pointers to compress */
+          ALIGN_EXPONENT /* how many bits can we drop from the offset */);
+
+        if (zcd.ptr2 != NULL)
+          rte_ptr_compress_32_shift(items, ptrs_send + zcd.n1,
+            zcd.ptr2, n_send - zcd.n1, ALIGN_EXPONENT);
+
+        rte_ring_enqueue_zc_finish(ring, n_send);
+      }
+      return 1;
+    }
+
+    static int
+    recv_compressed(void *args)
+    {
+      struct item *ptrs_recv[BATCH_SIZE] = {0};
+      unsigned int n_recv;
+      struct rte_ring_zc_data zcd = {0};
+
+      for(;;) {
+        /* receive compressed pointers from the ring */
+        n_recv = rte_ring_dequeue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        rte_ptr_decompress_32_shift(items, /* base pointer */
+          zcd.ptr1, /* source array to decompress */
+          ptrs_recv, /* destination array to store pointers */
+          zcd.n1, /* how many pointers to decompress */
+          ALIGN_EXPONENT /* how many bits were dropped from the offset */);
+
+        /* handle the potential secondary buffer (caused by ring boundary) */
+        if (zcd.ptr2 != NULL)
+          rte_ptr_decompress_32_shift(items,
+            zcd.ptr2,
+            ptrs_recv + zcd.n1,
+            n_recv - zcd.n1,
+            ALIGN_EXPONENT);
+
+        rte_ring_dequeue_zc_finish(ring, n_recv);
+
+        /* ptrs_recv contains what ptrs_send contained in the other thread */
+        /* (...) */
+      }
+      return 1;
+    }
+
+    void
+    compression_example(void)
+    {
+      ring = rte_ring_create_elem(
+        "COMPR_PTRS", sizeof(uint32_t),
+        1024, rte_socket_id(),
+        RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+      rte_eal_remote_launch(send_compressed, NULL, CORE_SEND);
+      rte_eal_remote_launch(recv_compressed, NULL, CORE_RECV);
+
+      for(;;) {}
+    }
diff --git a/doc/guides/rel_notes/release_24_07.rst b/doc/guides/rel_notes/release_24_07.rst
index 4711792e61..eb4bdec6dc 100644
--- a/doc/guides/rel_notes/release_24_07.rst
+++ b/doc/guides/rel_notes/release_24_07.rst
@@ -55,12 +55,14 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================

+
 * **Introduced pointer compression library.**

   Library provides functions to compress and decompress arrays of pointers
   which can improve application performance under certain conditions.
   Performance test was added to help users evaluate performance on their setup.

+
 Removed Items
 -------------

--
2.25.1

[PATCH v15 6/6] test: add unit test for ptr compression

[Paul Szczepanek]

Test compresses and decompresses pointers with various combinations
of memory regions and alignments and verifies the pointers are
recovered correctly. Additionally tests helper macros perform
calculations correctly.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
---
 MAINTAINERS                  |   1 +
 app/test/meson.build         |   1 +
 app/test/test_ptr_compress.c | 193 +++++++++++++++++++++++++++++++++++
 3 files changed, 195 insertions(+)
 create mode 100644 app/test/test_ptr_compress.c

diff --git a/MAINTAINERS b/MAINTAINERS
index ed50121bd2..2565ef5f4b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1697,6 +1697,7 @@ F: lib/pci/
 Pointer Compression
 M: Paul Szczepanek <paul.szczepanek@arm.com>
 F: lib/ptr_compress/
+F: app/test/test_ptr_compress.c
 F: doc/guides/prog_guide/ptr_compress_lib.rst

 Power management
diff --git a/app/test/meson.build b/app/test/meson.build
index df8cc00730..e29258e6ec 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -144,6 +144,7 @@ source_file_deps = {
     'test_power_intel_uncore.c': ['power'],
     'test_power_kvm_vm.c': ['power'],
     'test_prefetch.c': [],
+    'test_ptr_compress.c': ['ptr_compress'],
     'test_rand_perf.c': [],
     'test_rawdev.c': ['rawdev', 'bus_vdev'],
     'test_rcu_qsbr.c': ['rcu', 'hash'],
diff --git a/app/test/test_ptr_compress.c b/app/test/test_ptr_compress.c
new file mode 100644
index 0000000000..807b19eaf6
--- /dev/null
+++ b/app/test/test_ptr_compress.c
@@ -0,0 +1,193 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2024 Arm Limited
+ */
+
+#include "test.h"
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_ptr_compress.h>
+
+#define MAX_ALIGN_EXPONENT 3
+#define MAX_PTRS 16
+#define NUM_BASES 2
+#define NUM_REGIONS 4
+#define MAX_32BIT_REGION ((uint64_t)UINT32_MAX + 1)
+#define MAX_16BIT_REGION (UINT16_MAX + 1)
+
+static int
+test_ptr_compress_params(
+	void *base,
+	uint64_t mem_sz,
+	unsigned int align_exp,
+	unsigned int num_ptrs,
+	bool use_32_bit)
+{
+	unsigned int i;
+	unsigned int align = 1 << align_exp;
+	void *ptrs[MAX_PTRS] = {0};
+	void *ptrs_out[MAX_PTRS] = {0};
+	uint32_t offsets32[MAX_PTRS] = {0};
+	uint16_t offsets16[MAX_PTRS] = {0};
+
+	for (i = 0; i < num_ptrs; i++) {
+		/* make pointers point at memory in steps of align */
+		/* alternate steps from the start and end of memory region */
+		if ((i & 1) == 1)
+			ptrs[i] = (char *)base + mem_sz - i * align;
+		else
+			ptrs[i] = (char *)base + i * align;
+	}
+
+	if (use_32_bit) {
+		rte_ptr_compress_32_shift(
+				base, ptrs, offsets32, num_ptrs, align_exp);
+		rte_ptr_decompress_32_shift(base, offsets32, ptrs_out, num_ptrs,
+				align_exp);
+	} else {
+		rte_ptr_compress_16_shift(
+				base, ptrs, offsets16, num_ptrs, align_exp);
+		rte_ptr_decompress_16_shift(base, offsets16, ptrs_out, num_ptrs,
+				align_exp);
+	}
+
+	TEST_ASSERT_BUFFERS_ARE_EQUAL(ptrs, ptrs_out, sizeof(void *) * num_ptrs,
+		"Decompressed pointers corrupted\nbase pointer: %p, "
+		"memory region size: %" PRIu64 ", alignment exponent: %u, "
+		"num of pointers: %u, using %s offsets",
+		base, mem_sz, align_exp, num_ptrs,
+		use_32_bit ? "32-bit" : "16-bit");
+
+	return 0;
+}
+
+static int
+test_ptr_compress(void)
+{
+	unsigned int j, k, n;
+	int ret = 0;
+	/* the test is run with multiple memory regions and base addresses */
+	void * const bases[NUM_BASES] = { (void *)0, (void *)UINT16_MAX };
+	/* maximum size for pointers aligned by consecutive powers of 2 */
+	const uint64_t region_sizes_16[NUM_REGIONS] = {
+		MAX_16BIT_REGION,
+		MAX_16BIT_REGION * 2,
+		MAX_16BIT_REGION * 4,
+		MAX_16BIT_REGION * 8,
+	};
+	const uint64_t region_sizes_32[NUM_REGIONS] = {
+		MAX_32BIT_REGION,
+		MAX_32BIT_REGION * 2,
+		MAX_32BIT_REGION * 4,
+		MAX_32BIT_REGION * 8,
+	};
+
+	/* main test compresses and decompresses arrays of pointers
+	 * and compares the array before and after to verify that
+	 * pointers are successfully decompressed
+	 */
+
+	for (j = 0; j < NUM_REGIONS; j++) {
+		for (k = 0; k < NUM_BASES; k++) {
+			for (n = 1; n < MAX_PTRS; n++) {
+				ret |= test_ptr_compress_params(
+					bases[k],
+					region_sizes_16[j],
+					j /* exponent of alignment */,
+					n,
+					false
+				);
+				ret |= test_ptr_compress_params(
+					bases[k],
+					region_sizes_32[j],
+					j /* exponent of alignment */,
+					n,
+					true
+				);
+				if (ret != 0)
+					return ret;
+			}
+		}
+	}
+
+	/* verify helper macro computations */
+
+	n = RTE_PTR_COMPRESS_BITS_NEEDED_FOR_POINTER_WITHIN_RANGE(0);
+	TEST_ASSERT_EQUAL(n, 1,
+			"RTE_PTR_COMPRESS_BITS_NEEDED_FOR_POINTER_WITHIN_RANGE "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BITS_NEEDED_FOR_POINTER_WITHIN_RANGE(1);
+	TEST_ASSERT_EQUAL(n, 1,
+			"RTE_PTR_COMPREtopSS_BITS_REQUIRED_TO_STORE_VALUE "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BITS_NEEDED_FOR_POINTER_WITHIN_RANGE(MAX_16BIT_REGION);
+	TEST_ASSERT_EQUAL(n, 16,
+			"RTE_PTR_COMPRESS_BITS_NEEDED_FOR_POINTER_WITHIN_RANGE "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BITS_NEEDED_FOR_POINTER_WITHIN_RANGE(MAX_32BIT_REGION);
+	TEST_ASSERT_EQUAL(n, 32,
+			"RTE_PTR_COMPRESS_BITS_NEEDED_FOR_POINTER_WITHIN_RANGE "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(0);
+	TEST_ASSERT_EQUAL(n, 0,
+			"RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(1);
+	TEST_ASSERT_EQUAL(n, 0,
+			"RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(2);
+	TEST_ASSERT_EQUAL(n, 1,
+			"RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(3);
+	TEST_ASSERT_EQUAL(n, 0,
+			"RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT "
+			"macro computation incorrect\n");
+
+	n = RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT(4);
+	TEST_ASSERT_EQUAL(n, 2,
+			"RTE_PTR_COMPRESS_BIT_SHIFT_FROM_ALIGNMENT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT(MAX_16BIT_REGION, 1);
+	TEST_ASSERT_EQUAL(ret, 1,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT(MAX_16BIT_REGION + 1, 1);
+	TEST_ASSERT_EQUAL(ret, 0,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT(MAX_16BIT_REGION + 1, 2);
+	TEST_ASSERT_EQUAL(ret, 1,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_16_SHIFT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT(MAX_32BIT_REGION, 1);
+	TEST_ASSERT_EQUAL(ret, 1,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT(MAX_32BIT_REGION + 1, 1);
+	TEST_ASSERT_EQUAL(ret, 0,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT "
+			"macro computation incorrect\n");
+
+	ret = RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT(MAX_32BIT_REGION + 1, 2);
+	TEST_ASSERT_EQUAL(ret, 1,
+			"RTE_PTR_COMPRESS_CAN_COMPRESS_32_SHIFT "
+			"macro computation incorrect\n");
+
+	return 0;
+}
+
+REGISTER_FAST_TEST(ptr_compress_autotest, true, true, test_ptr_compress);
--
2.25.1

Re: [PATCH v15 0/6] add pointer compression API

[David Marchand]

On Tue, Jun 11, 2024 at 3:00 PM Paul Szczepanek <paul.szczepanek@arm.com> wrote:
>
> This patchset is proposing adding a new header only library
> with utility functions that allow compression of arrays of pointers.
>
> Since this is a header only library a patch needed to be added to amend
> the build system to allow adding libraries without source files.
>
> When passing caches full of pointers between threads, memory containing
> the pointers is copied multiple times which is especially costly between
> cores. A compression method will allow us to shrink the memory size
> copied.
>
> The compression takes advantage of the fact that pointers are usually
> located in a limited memory region. We can compress them by converting them
> to offsets from a base memory address.
>
> Offsets can be stored in fewer bytes (dictated by the memory region size
> and alignment of the pointer). For example: an 8 byte aligned pointer
> which is part of a 32GB memory pool can be stored in 4 bytes. The API is
> very generic and does not assume mempool pointers, any pointer can be
> passed in.
>
> Compression is based on few and fast operations and especially with vector
> instructions leveraged creates minimal overhead.
>
> The API accepts and returns arrays because the overhead means it only is
> worth it when done in bulk.
>
> Test is added that shows potential performance gain from compression. In
> this test an array of pointers is passed through a ring between two cores.
> It shows the gain which is dependent on the bulk operation size. In this
> synthetic test run on ampere altra a substantial (up to 25%) performance
> gain is seen if done in bulk size larger than 32. At 32 it breaks even and
> lower sizes create a small (less than 5%) slowdown due to overhead.
>
> In a more realistic mock application running the l3 forwarding dpdk
> example that works in pipeline mode on two cores this translated into a
> ~5% throughput increase on an ampere altra.
>
> Paul Szczepanek (6):
>   lib: allow libraries with no sources
>   mempool: add functions to get extra mempool info
>   ptr_compress: add pointer compression library
>   test: add pointer compress tests to ring perf test
>   docs: add pointer compression guide
>   test: add unit test for ptr compression
>
>  MAINTAINERS                                |   6 +
>  app/test/meson.build                       |  21 +-
>  app/test/test_mempool.c                    |  70 ++++
>  app/test/test_ptr_compress.c               | 193 +++++++++++
>  app/test/test_ring.h                       |  94 ++++++
>  app/test/test_ring_perf.c                  | 352 ++++++++++++++-------
>  doc/api/doxy-api-index.md                  |   1 +
>  doc/api/doxy-api.conf.in                   |   1 +
>  doc/guides/prog_guide/index.rst            |   1 +
>  doc/guides/prog_guide/ptr_compress_lib.rst | 160 ++++++++++
>  doc/guides/rel_notes/release_24_07.rst     |   7 +
>  lib/mempool/rte_mempool.c                  |  45 +++
>  lib/mempool/rte_mempool.h                  |  49 +++
>  lib/mempool/version.map                    |   3 +
>  lib/meson.build                            |  17 +
>  lib/ptr_compress/meson.build               |   4 +
>  lib/ptr_compress/rte_ptr_compress.h        | 325 +++++++++++++++++++
>  17 files changed, 1217 insertions(+), 132 deletions(-)
>  create mode 100644 app/test/test_ptr_compress.c
>  create mode 100644 doc/guides/prog_guide/ptr_compress_lib.rst
>  create mode 100644 lib/ptr_compress/meson.build
>  create mode 100644 lib/ptr_compress/rte_ptr_compress.h

I reorganised the series, adjusted the documentation and added some
release notes for mempool, as agreed offlist.

Even if this library only contains a header, with no tie to other
public DPDK API, this library should be optional.
If no objection, please work on this change for -rc2,

Series applied, thanks Paul.


-- 
David Marchand

Re: [PATCH v15 0/6] add pointer compression API

[David Marchand]

Hello Paul,

On Fri, Jun 14, 2024 at 12:28 PM David Marchand
<david.marchand@redhat.com> wrote:
> Even if this library only contains a header, with no tie to other
> public DPDK API, this library should be optional.
> If no objection, please work on this change for -rc2,

We have a build error on armv7 (tested in OBS).
https://build.opensuse.org/public/build/home:bluca:dpdk/openSUSE_Factory_ARM/armv7l/dpdk/_log

[  395s] [1730/2037] cc -Iapp/dpdk-test.p -Iapp -I../app -Ilib/cmdline
-I../lib/cmdline -I. -I.. -Iconfig -I../config -Ilib/eal/include
-I../lib/eal/include -Ilib/eal/linux/include
-I../lib/eal/linux/include -Ilib/eal/arm/include
-I../lib/eal/arm/include -Ilib/eal/common -I../lib/eal/common
-Ilib/eal -I../lib/eal -Ilib/kvargs -I../lib/kvargs -Ilib/log
-I../lib/log -Ilib/metrics -I../lib/metrics -Ilib/telemetry
-I../lib/telemetry -Ilib/net -I../lib/net -Ilib/mbuf -I../lib/mbuf
-Ilib/mempool -I../lib/mempool -Ilib/ring -I../lib/ring
-Idrivers/net/ring -I../drivers/net/ring -Ilib/ethdev -I../lib/ethdev
-Ilib/meter -I../lib/meter -Idrivers/bus/pci -I../drivers/bus/pci
-I../drivers/bus/pci/linux -Ilib/pci -I../lib/pci -Idrivers/bus/vdev
-I../drivers/bus/vdev -Ilib/acl -I../lib/acl -Ilib/argparse
-I../lib/argparse -Ilib/hash -I../lib/hash -Ilib/rcu -I../lib/rcu
-Ilib/bitratestats -I../lib/bitratestats -Ilib/bpf -I../lib/bpf
-Ilib/compressdev -I../lib/compressdev -Ilib/cryptodev
-I../lib/cryptodev -Ilib/security -I../lib/security -Ilib/dispatcher
-I../lib/dispatcher -Ilib/eventdev -I../lib/eventdev -Ilib/timer
-I../lib/timer -Ilib/dmadev -I../lib/dmadev -Ilib/distributor
-I../lib/distributor -Ilib/efd -I../lib/efd -Ilib/fib -I../lib/fib
-Ilib/rib -I../lib/rib -Ilib/table -I../lib/table -Ilib/port
-I../lib/port -Ilib/sched -I../lib/sched -Ilib/ip_frag
-I../lib/ip_frag -Ilib/lpm -I../lib/lpm -Ilib/graph -I../lib/graph
-Ilib/pcapng -I../lib/pcapng -Ilib/ipsec -I../lib/ipsec
-Ilib/latencystats -I../lib/latencystats -Idrivers/net/bonding
-I../drivers/net/bonding -Ilib/member -I../lib/member -Ilib/pdcp
-I../lib/pdcp -Ilib/reorder -I../lib/reorder -Ilib/pdump
-I../lib/pdump -Ilib/power -I../lib/power -Ilib/ptr_compress
-I../lib/ptr_compress -Ilib/rawdev -I../lib/rawdev -Ilib/stack
-I../lib/stack -Ilib/pipeline -I../lib/pipeline
-Idrivers/crypto/scheduler -I../drivers/crypto/scheduler
-I/usr/include/libnl3 -I/usr/include/dbus-1.0
-I/usr/lib/dbus-1.0/include -fdiagnostics-color=always
-D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -std=c11 -O3
-include rte_config.h -Wcast-qual -Wdeprecated -Wformat
-Wmissing-declarations -Wmissing-prototypes -Wnested-externs
-Wold-style-definition -Wpointer-arith -Wsign-compare
-Wstrict-prototypes -Wundef -Wwrite-strings
-Wno-address-of-packed-member -Wno-packed-not-aligned
-Wno-missing-field-initializers -Wno-zero-length-bounds
-Wno-pointer-to-int-cast -D_GNU_SOURCE -fcommon -Werror -march=armv7-a
-mfpu=neon -DALLOW_EXPERIMENTAL_API -Wno-format-truncation
-fno-strict-aliasing -DALLOW_INTERNAL_API -MD -MQ
app/dpdk-test.p/test_test_ptr_compress.c.o -MF
app/dpdk-test.p/test_test_ptr_compress.c.o.d -o
app/dpdk-test.p/test_test_ptr_compress.c.o -c
../app/test/test_ptr_compress.c
[  395s] FAILED: app/dpdk-test.p/test_test_ptr_compress.c.o
[  395s] cc -Iapp/dpdk-test.p -Iapp -I../app -Ilib/cmdline
-I../lib/cmdline -I. -I.. -Iconfig -I../config -Ilib/eal/include
-I../lib/eal/include -Ilib/eal/linux/include
-I../lib/eal/linux/include -Ilib/eal/arm/include
-I../lib/eal/arm/include -Ilib/eal/common -I../lib/eal/common
-Ilib/eal -I../lib/eal -Ilib/kvargs -I../lib/kvargs -Ilib/log
-I../lib/log -Ilib/metrics -I../lib/metrics -Ilib/telemetry
-I../lib/telemetry -Ilib/net -I../lib/net -Ilib/mbuf -I../lib/mbuf
-Ilib/mempool -I../lib/mempool -Ilib/ring -I../lib/ring
-Idrivers/net/ring -I../drivers/net/ring -Ilib/ethdev -I../lib/ethdev
-Ilib/meter -I../lib/meter -Idrivers/bus/pci -I../drivers/bus/pci
-I../drivers/bus/pci/linux -Ilib/pci -I../lib/pci -Idrivers/bus/vdev
-I../drivers/bus/vdev -Ilib/acl -I../lib/acl -Ilib/argparse
-I../lib/argparse -Ilib/hash -I../lib/hash -Ilib/rcu -I../lib/rcu
-Ilib/bitratestats -I../lib/bitratestats -Ilib/bpf -I../lib/bpf
-Ilib/compressdev -I../lib/compressdev -Ilib/cryptodev
-I../lib/cryptodev -Ilib/security -I../lib/security -Ilib/dispatcher
-I../lib/dispatcher -Ilib/eventdev -I../lib/eventdev -Ilib/timer
-I../lib/timer -Ilib/dmadev -I../lib/dmadev -Ilib/distributor
-I../lib/distributor -Ilib/efd -I../lib/efd -Ilib/fib -I../lib/fib
-Ilib/rib -I../lib/rib -Ilib/table -I../lib/table -Ilib/port
-I../lib/port -Ilib/sched -I../lib/sched -Ilib/ip_frag
-I../lib/ip_frag -Ilib/lpm -I../lib/lpm -Ilib/graph -I../lib/graph
-Ilib/pcapng -I../lib/pcapng -Ilib/ipsec -I../lib/ipsec
-Ilib/latencystats -I../lib/latencystats -Idrivers/net/bonding
-I../drivers/net/bonding -Ilib/member -I../lib/member -Ilib/pdcp
-I../lib/pdcp -Ilib/reorder -I../lib/reorder -Ilib/pdump
-I../lib/pdump -Ilib/power -I../lib/power -Ilib/ptr_compress
-I../lib/ptr_compress -Ilib/rawdev -I../lib/rawdev -Ilib/stack
-I../lib/stack -Ilib/pipeline -I../lib/pipeline
-Idrivers/crypto/scheduler -I../drivers/crypto/scheduler
-I/usr/include/libnl3 -I/usr/include/dbus-1.0
-I/usr/lib/dbus-1.0/include -fdiagnostics-color=always
-D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -std=c11 -O3
-include rte_config.h -Wcast-qual -Wdeprecated -Wformat
-Wmissing-declarations -Wmissing-prototypes -Wnested-externs
-Wold-style-definition -Wpointer-arith -Wsign-compare
-Wstrict-prototypes -Wundef -Wwrite-strings
-Wno-address-of-packed-member -Wno-packed-not-aligned
-Wno-missing-field-initializers -Wno-zero-length-bounds
-Wno-pointer-to-int-cast -D_GNU_SOURCE -fcommon -Werror -march=armv7-a
-mfpu=neon -DALLOW_EXPERIMENTAL_API -Wno-format-truncation
-fno-strict-aliasing -DALLOW_INTERNAL_API -MD -MQ
app/dpdk-test.p/test_test_ptr_compress.c.o -MF
app/dpdk-test.p/test_test_ptr_compress.c.o.d -o
app/dpdk-test.p/test_test_ptr_compress.c.o -c
../app/test/test_ptr_compress.c
[  395s] In file included from ../app/test/test.h:14,
[  395s]                  from ../app/test/test_ptr_compress.c:5:
[  395s] ../lib/ptr_compress/rte_ptr_compress.h: In function
‘rte_ptr_decompress_32_shift’:
[  395s] ../lib/eal/include/rte_common.h:404:30: error: cast to
pointer from integer of different size [-Werror=int-to-pointer-cast]
[  395s]   404 | #define RTE_PTR_ADD(ptr, x) ((void*)((uintptr_t)(ptr) + (x)))
[  395s]       |                              ^
[  395s] ../lib/ptr_compress/rte_ptr_compress.h:219:33: note: in
expansion of macro ‘RTE_PTR_ADD’
[  395s]   219 |                 dest_table[i] = RTE_PTR_ADD(ptr_base,
ptr_diff);
[  395s]       |                                 ^~~~~~~~~~~
[  395s] cc1: all warnings being treated as errors

Please have a look.
Thanks.


-- 
David Marchand

Re: [PATCH v15 0/6] add pointer compression API

[Paul Szczepanek]

On 17/06/2024 11:02, David Marchand wrote:
> Hello Paul,
> 
> On Fri, Jun 14, 2024 at 12:28 PM David Marchand
> <david.marchand@redhat.com> wrote:
>> Even if this library only contains a header, with no tie to other
>> public DPDK API, this library should be optional.
>> If no objection, please work on this change for -rc2,
> 
> We have a build error on armv7 (tested in OBS).
> https://build.opensuse.org/public/build/home:bluca:dpdk/openSUSE_Factory_ARM/armv7l/dpdk/_log
> 
> [  395s] [1730/2037] cc -Iapp/dpdk-test.p -Iapp -I../app -Ilib/cmdline
> -I../lib/cmdline -I. -I.. -Iconfig -I../config -Ilib/eal/include
> -I../lib/eal/include -Ilib/eal/linux/include
> -I../lib/eal/linux/include -Ilib/eal/arm/include
> -I../lib/eal/arm/include -Ilib/eal/common -I../lib/eal/common
> -Ilib/eal -I../lib/eal -Ilib/kvargs -I../lib/kvargs -Ilib/log
> -I../lib/log -Ilib/metrics -I../lib/metrics -Ilib/telemetry
> -I../lib/telemetry -Ilib/net -I../lib/net -Ilib/mbuf -I../lib/mbuf
> -Ilib/mempool -I../lib/mempool -Ilib/ring -I../lib/ring
> -Idrivers/net/ring -I../drivers/net/ring -Ilib/ethdev -I../lib/ethdev
> -Ilib/meter -I../lib/meter -Idrivers/bus/pci -I../drivers/bus/pci
> -I../drivers/bus/pci/linux -Ilib/pci -I../lib/pci -Idrivers/bus/vdev
> -I../drivers/bus/vdev -Ilib/acl -I../lib/acl -Ilib/argparse
> -I../lib/argparse -Ilib/hash -I../lib/hash -Ilib/rcu -I../lib/rcu
> -Ilib/bitratestats -I../lib/bitratestats -Ilib/bpf -I../lib/bpf
> -Ilib/compressdev -I../lib/compressdev -Ilib/cryptodev
> -I../lib/cryptodev -Ilib/security -I../lib/security -Ilib/dispatcher
> -I../lib/dispatcher -Ilib/eventdev -I../lib/eventdev -Ilib/timer
> -I../lib/timer -Ilib/dmadev -I../lib/dmadev -Ilib/distributor
> -I../lib/distributor -Ilib/efd -I../lib/efd -Ilib/fib -I../lib/fib
> -Ilib/rib -I../lib/rib -Ilib/table -I../lib/table -Ilib/port
> -I../lib/port -Ilib/sched -I../lib/sched -Ilib/ip_frag
> -I../lib/ip_frag -Ilib/lpm -I../lib/lpm -Ilib/graph -I../lib/graph
> -Ilib/pcapng -I../lib/pcapng -Ilib/ipsec -I../lib/ipsec
> -Ilib/latencystats -I../lib/latencystats -Idrivers/net/bonding
> -I../drivers/net/bonding -Ilib/member -I../lib/member -Ilib/pdcp
> -I../lib/pdcp -Ilib/reorder -I../lib/reorder -Ilib/pdump
> -I../lib/pdump -Ilib/power -I../lib/power -Ilib/ptr_compress
> -I../lib/ptr_compress -Ilib/rawdev -I../lib/rawdev -Ilib/stack
> -I../lib/stack -Ilib/pipeline -I../lib/pipeline
> -Idrivers/crypto/scheduler -I../drivers/crypto/scheduler
> -I/usr/include/libnl3 -I/usr/include/dbus-1.0
> -I/usr/lib/dbus-1.0/include -fdiagnostics-color=always
> -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -std=c11 -O3
> -include rte_config.h -Wcast-qual -Wdeprecated -Wformat
> -Wmissing-declarations -Wmissing-prototypes -Wnested-externs
> -Wold-style-definition -Wpointer-arith -Wsign-compare
> -Wstrict-prototypes -Wundef -Wwrite-strings
> -Wno-address-of-packed-member -Wno-packed-not-aligned
> -Wno-missing-field-initializers -Wno-zero-length-bounds
> -Wno-pointer-to-int-cast -D_GNU_SOURCE -fcommon -Werror -march=armv7-a
> -mfpu=neon -DALLOW_EXPERIMENTAL_API -Wno-format-truncation
> -fno-strict-aliasing -DALLOW_INTERNAL_API -MD -MQ
> app/dpdk-test.p/test_test_ptr_compress.c.o -MF
> app/dpdk-test.p/test_test_ptr_compress.c.o.d -o
> app/dpdk-test.p/test_test_ptr_compress.c.o -c
> ../app/test/test_ptr_compress.c
> [  395s] FAILED: app/dpdk-test.p/test_test_ptr_compress.c.o
> [  395s] cc -Iapp/dpdk-test.p -Iapp -I../app -Ilib/cmdline
> -I../lib/cmdline -I. -I.. -Iconfig -I../config -Ilib/eal/include
> -I../lib/eal/include -Ilib/eal/linux/include
> -I../lib/eal/linux/include -Ilib/eal/arm/include
> -I../lib/eal/arm/include -Ilib/eal/common -I../lib/eal/common
> -Ilib/eal -I../lib/eal -Ilib/kvargs -I../lib/kvargs -Ilib/log
> -I../lib/log -Ilib/metrics -I../lib/metrics -Ilib/telemetry
> -I../lib/telemetry -Ilib/net -I../lib/net -Ilib/mbuf -I../lib/mbuf
> -Ilib/mempool -I../lib/mempool -Ilib/ring -I../lib/ring
> -Idrivers/net/ring -I../drivers/net/ring -Ilib/ethdev -I../lib/ethdev
> -Ilib/meter -I../lib/meter -Idrivers/bus/pci -I../drivers/bus/pci
> -I../drivers/bus/pci/linux -Ilib/pci -I../lib/pci -Idrivers/bus/vdev
> -I../drivers/bus/vdev -Ilib/acl -I../lib/acl -Ilib/argparse
> -I../lib/argparse -Ilib/hash -I../lib/hash -Ilib/rcu -I../lib/rcu
> -Ilib/bitratestats -I../lib/bitratestats -Ilib/bpf -I../lib/bpf
> -Ilib/compressdev -I../lib/compressdev -Ilib/cryptodev
> -I../lib/cryptodev -Ilib/security -I../lib/security -Ilib/dispatcher
> -I../lib/dispatcher -Ilib/eventdev -I../lib/eventdev -Ilib/timer
> -I../lib/timer -Ilib/dmadev -I../lib/dmadev -Ilib/distributor
> -I../lib/distributor -Ilib/efd -I../lib/efd -Ilib/fib -I../lib/fib
> -Ilib/rib -I../lib/rib -Ilib/table -I../lib/table -Ilib/port
> -I../lib/port -Ilib/sched -I../lib/sched -Ilib/ip_frag
> -I../lib/ip_frag -Ilib/lpm -I../lib/lpm -Ilib/graph -I../lib/graph
> -Ilib/pcapng -I../lib/pcapng -Ilib/ipsec -I../lib/ipsec
> -Ilib/latencystats -I../lib/latencystats -Idrivers/net/bonding
> -I../drivers/net/bonding -Ilib/member -I../lib/member -Ilib/pdcp
> -I../lib/pdcp -Ilib/reorder -I../lib/reorder -Ilib/pdump
> -I../lib/pdump -Ilib/power -I../lib/power -Ilib/ptr_compress
> -I../lib/ptr_compress -Ilib/rawdev -I../lib/rawdev -Ilib/stack
> -I../lib/stack -Ilib/pipeline -I../lib/pipeline
> -Idrivers/crypto/scheduler -I../drivers/crypto/scheduler
> -I/usr/include/libnl3 -I/usr/include/dbus-1.0
> -I/usr/lib/dbus-1.0/include -fdiagnostics-color=always
> -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -std=c11 -O3
> -include rte_config.h -Wcast-qual -Wdeprecated -Wformat
> -Wmissing-declarations -Wmissing-prototypes -Wnested-externs
> -Wold-style-definition -Wpointer-arith -Wsign-compare
> -Wstrict-prototypes -Wundef -Wwrite-strings
> -Wno-address-of-packed-member -Wno-packed-not-aligned
> -Wno-missing-field-initializers -Wno-zero-length-bounds
> -Wno-pointer-to-int-cast -D_GNU_SOURCE -fcommon -Werror -march=armv7-a
> -mfpu=neon -DALLOW_EXPERIMENTAL_API -Wno-format-truncation
> -fno-strict-aliasing -DALLOW_INTERNAL_API -MD -MQ
> app/dpdk-test.p/test_test_ptr_compress.c.o -MF
> app/dpdk-test.p/test_test_ptr_compress.c.o.d -o
> app/dpdk-test.p/test_test_ptr_compress.c.o -c
> ../app/test/test_ptr_compress.c
> [  395s] In file included from ../app/test/test.h:14,
> [  395s]                  from ../app/test/test_ptr_compress.c:5:
> [  395s] ../lib/ptr_compress/rte_ptr_compress.h: In function
> ‘rte_ptr_decompress_32_shift’:
> [  395s] ../lib/eal/include/rte_common.h:404:30: error: cast to
> pointer from integer of different size [-Werror=int-to-pointer-cast]
> [  395s]   404 | #define RTE_PTR_ADD(ptr, x) ((void*)((uintptr_t)(ptr) + (x)))
> [  395s]       |                              ^
> [  395s] ../lib/ptr_compress/rte_ptr_compress.h:219:33: note: in
> expansion of macro ‘RTE_PTR_ADD’
> [  395s]   219 |                 dest_table[i] = RTE_PTR_ADD(ptr_base,
> ptr_diff);
> [  395s]       |                                 ^~~~~~~~~~~
> [  395s] cc1: all warnings being treated as errors
> 
> Please have a look.
> Thanks.
> 

This will solve it:

diff --git a/lib/ptr_compress/rte_ptr_compress.h
b/lib/ptr_compress/rte_ptr_compress.h
index b9ab17b2db..a779f21fda 100644
--- a/lib/ptr_compress/rte_ptr_compress.h
+++ b/lib/ptr_compress/rte_ptr_compress.h
@@ -141,7 +141,7 @@ rte_ptr_compress_32_shift(void *ptr_base, void *
const *src_table,
                i += svcntd();
        } while (i < n);
 #elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
-       uint64_t ptr_diff;
+       uintptr_t ptr_diff;
        uint64x2_t v_ptr_table;
        /* right shift is done by left shifting by negative int */
        int64x2_t v_shift = vdupq_n_s64(-bit_shift);
@@ -202,7 +202,7 @@ rte_ptr_decompress_32_shift(void *ptr_base, uint32_t
const *src_table,
                i += svcntd();
        } while (i < n);
 #elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
-       uint64_t ptr_diff;
+       uintptr_t ptr_diff;
        uint64x2_t v_ptr_table;
        int64x2_t v_shift = vdupq_n_s64(bit_shift);
        uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
@@ -215,7 +215,7 @@ rte_ptr_decompress_32_shift(void *ptr_base, uint32_t
const *src_table,
        }
        /* process leftover single item in case of odd number of n */
        if (unlikely(n & 0x1)) {
-               ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+               ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
                dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
        }
 #else

Do you want me to make a separate patch?

Re: [PATCH v15 0/6] add pointer compression API

[David Marchand]

On Mon, Jun 17, 2024 at 3:47 PM Paul Szczepanek <paul.szczepanek@arm.com> wrote:
> This will solve it:
>
> diff --git a/lib/ptr_compress/rte_ptr_compress.h
> b/lib/ptr_compress/rte_ptr_compress.h
> index b9ab17b2db..a779f21fda 100644
> --- a/lib/ptr_compress/rte_ptr_compress.h
> +++ b/lib/ptr_compress/rte_ptr_compress.h
> @@ -141,7 +141,7 @@ rte_ptr_compress_32_shift(void *ptr_base, void *
> const *src_table,
>                 i += svcntd();
>         } while (i < n);
>  #elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
> -       uint64_t ptr_diff;
> +       uintptr_t ptr_diff;
>         uint64x2_t v_ptr_table;
>         /* right shift is done by left shifting by negative int */
>         int64x2_t v_shift = vdupq_n_s64(-bit_shift);
> @@ -202,7 +202,7 @@ rte_ptr_decompress_32_shift(void *ptr_base, uint32_t
> const *src_table,
>                 i += svcntd();
>         } while (i < n);
>  #elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
> -       uint64_t ptr_diff;
> +       uintptr_t ptr_diff;
>         uint64x2_t v_ptr_table;
>         int64x2_t v_shift = vdupq_n_s64(bit_shift);
>         uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
> @@ -215,7 +215,7 @@ rte_ptr_decompress_32_shift(void *ptr_base, uint32_t
> const *src_table,
>         }
>         /* process leftover single item in case of odd number of n */
>         if (unlikely(n & 0x1)) {
> -               ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
> +               ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
>                 dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
>         }
>  #else
>
> Do you want me to make a separate patch?

Yes please.


-- 
David Marchand