DPDK patches and discussions
 help / color / mirror / Atom feed
* [RFC 0/2] add pointer compression API
@ 2023-09-27 15:08 Paul Szczepanek
  2023-09-27 15:08 ` [RFC 1/2] eal: add pointer compression functions Paul Szczepanek
  2023-09-27 15:08 ` [RFC 2/2] test: add pointer compress tests to ring perf test Paul Szczepanek
  0 siblings, 2 replies; 29+ messages in thread
From: Paul Szczepanek @ 2023-09-27 15:08 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek

This patchset is proposing adding a new EAL header with utility functions
that allow compression of arrays of pointers.

When passing caches full of pointers between threads, memory containing
the pointers is copied multiple times which is especially costly between
cores. A compression method will allow us to shrink the memory size
copied.

The compression takes advantage of the fact that pointers are usually
located in a limited memory region (like a mempool). We can compress them
by converting them to offsets from a base memory address.

Offsets can be stored in fewer bytes (dictated by the memory region size
and alignment of the pointer). For example: an 8 byte aligned pointer
which is part of a 32GB memory pool can be stored in 4 bytes. The API is
very generic and does not assume mempool pointers, any pointer can be
passed in.

Compression is based on few and fast operations and especially with vector
instructions leveraged creates minimal overhead.

The API accepts and returns arrays because the overhead means it only is
worth it when done in bulk.

Test is added that shows potential performance gain from compression. In
this test an array of pointers is passed through a ring between two cores.
It shows the gain which is dependent on the bulk operation size. In this
synthetic test run on ampere altra a substantial (up to 25%) performance
gain is seen if done in bulk size larger than 32. At 32 it breaks even and
lower sizes create a small (less than 5%) slowdown due to overhead.

In a more realistic mock application running the l3 forwarding dpdk
example that works in pipeline mode this translated into a ~5% throughput
increase on an ampere altra.

Paul Szczepanek (2):
  eal: add pointer compression functions
  test: add pointer compress tests to ring perf test

 .mailmap                           |   1 +
 app/test/test_ring.h               |  59 +++++-
 app/test/test_ring_perf.c          | 324 ++++++++++++++++++-----------
 lib/eal/include/meson.build        |   1 +
 lib/eal/include/rte_ptr_compress.h | 158 ++++++++++++++
 5 files changed, 419 insertions(+), 124 deletions(-)
 create mode 100644 lib/eal/include/rte_ptr_compress.h

-- 
2.25.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [RFC 1/2] eal: add pointer compression functions
  2023-09-27 15:08 [RFC 0/2] add pointer compression API Paul Szczepanek
@ 2023-09-27 15:08 ` Paul Szczepanek
  2023-10-09 15:54   ` Thomas Monjalon
                     ` (4 more replies)
  2023-09-27 15:08 ` [RFC 2/2] test: add pointer compress tests to ring perf test Paul Szczepanek
  1 sibling, 5 replies; 29+ messages in thread
From: Paul Szczepanek @ 2023-09-27 15:08 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Honnappa Nagarahalli, Kamalakshitha Aligeri

Add a new utility header for compressing pointers. Pointers are
compressed by taking advantage of their locality. Instead of
storing the full address only an offset from a known base is stored.

The provided functions can store pointers in 32bit offsets.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .mailmap                           |   1 +
 lib/eal/include/meson.build        |   1 +
 lib/eal/include/rte_ptr_compress.h | 158 +++++++++++++++++++++++++++++
 3 files changed, 160 insertions(+)
 create mode 100644 lib/eal/include/rte_ptr_compress.h

diff --git a/.mailmap b/.mailmap
index 864d33ee46..3f0c9d32f5 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1058,6 +1058,7 @@ Paul Greenwalt <paul.greenwalt@intel.com>
 Paulis Gributs <paulis.gributs@intel.com>
 Paul Luse <paul.e.luse@intel.com>
 Paul M Stillwell Jr <paul.m.stillwell.jr@intel.com>
+Paul Szczepanek <paul.szczepanek@arm.com>
 Pavan Kumar Linga <pavan.kumar.linga@intel.com>
 Pavan Nikhilesh <pbhagavatula@marvell.com> <pbhagavatula@caviumnetworks.com>
 Pavel Belous <pavel.belous@aquantia.com>
diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index a0463efac7..60b056ef96 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -35,6 +35,7 @@ headers += files(
         'rte_pci_dev_feature_defs.h',
         'rte_pci_dev_features.h',
         'rte_per_lcore.h',
+	'rte_ptr_compress.h',
         'rte_pflock.h',
         'rte_random.h',
         'rte_reciprocal.h',
diff --git a/lib/eal/include/rte_ptr_compress.h b/lib/eal/include/rte_ptr_compress.h
new file mode 100644
index 0000000000..6498587c0b
--- /dev/null
+++ b/lib/eal/include/rte_ptr_compress.h
@@ -0,0 +1,158 @@
+/* SPDX-License-Identifier: BSD-shift-Clause
+ * Copyright(c) 2023 Arm Limited
+ */
+
+#ifndef _RTE_PTR_COMPRESS_H_
+#define _RTE_PTR_COMPRESS_H_
+
+/**
+ * @file
+ * RTE pointer compression and decompression.
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_vect.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Compress pointers into 32 bit offsets from base pointer.
+ *
+ * @note Offsets from the base pointer must fit within 32bits. Alignment allows
+ * us to drop bits from the offsets - this means that for pointers aligned by
+ * 8 bytes they must be within 32GB of the base pointer. Unaligned pointers
+ * must be within 4GB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_32(void *ptr_base, void **src_table,
+		uint32_t *dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_src_table;
+	svuint64_t v_dest_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_src_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_dest_table = svsub_x(pg, v_src_table, (uint64_t)ptr_base);
+		v_dest_table = svlsr_x(pg, v_dest_table, bit_shift);
+		svst1w(pg, &dest_table[i], v_dest_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON
+	uint64_t ptr_diff;
+	uint64x2_t v_src_table;
+	uint64x2_t v_dest_table;
+	/* right shift is done by left shifting by negative int */
+	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_src_table = vld1q_u64((const uint64_t *)src_table + i);
+		v_dest_table = vsubq_u64(v_src_table, v_ptr_base);
+		v_dest_table = vshlq_u64(v_dest_table, v_shift);
+		vst1_u32(dest_table + i, vqmovn_u64(v_dest_table));
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
+	}
+#else
+	uint64_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		/* save extra bits that are redundant due to alignment */
+		ptr_diff = ptr_diff >> bit_shift;
+		/* make sure no truncation will happen when casting */
+		RTE_ASSERT(ptr_diff <= UINT32_MAX);
+		dest_table[i] = (uint32_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 32 bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_32(void *ptr_base, uint32_t *src_table,
+		void **dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_src_table;
+	svuint64_t v_dest_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_src_table = svld1uw_u64(pg, &src_table[i]);
+		v_src_table = svlsl_x(pg, v_src_table, bit_shift);
+		v_dest_table = svadd_x(pg, v_src_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_dest_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON
+	uint64_t ptr_diff;
+	uint64x2_t v_src_table;
+	uint64x2_t v_dest_table;
+	int64x2_t v_shift = vdupq_n_s64(bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_src_table = vmovl_u32(vld1_u32(src_table + i));
+		v_src_table = vshlq_u64(v_dest_table, v_shift);
+		v_dest_table = vaddq_u64(v_src_table, v_ptr_base);
+		vst1q_u64((uint64_t *)dest_table + i, v_dest_table);
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#else
+	uint64_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PTR_COMPRESS_H_ */
-- 
2.25.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [RFC 2/2] test: add pointer compress tests to ring perf test
  2023-09-27 15:08 [RFC 0/2] add pointer compression API Paul Szczepanek
  2023-09-27 15:08 ` [RFC 1/2] eal: add pointer compression functions Paul Szczepanek
@ 2023-09-27 15:08 ` Paul Szczepanek
  2023-10-09 15:48   ` Thomas Monjalon
  1 sibling, 1 reply; 29+ messages in thread
From: Paul Szczepanek @ 2023-09-27 15:08 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Honnappa Nagarahalli

Add a test that runs a zero copy burst enqueue and dequeue on a ring
of raw pointers and compressed pointers at different burst sizes to
showcase performance benefits of newly added pointer compression APIs

To reuse existing code, some refactoring was done to pass more
parameters to test threads. Additionally more bulk sizes were
added to showcase their effects on compression. To keep runtime
reasoanble iterations where adjusted to take into account bulk sizes.

Old printfs are adjusted to match new ones which have aligned numbers.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 app/test/test_ring.h      |  59 ++++++-
 app/test/test_ring_perf.c | 324 +++++++++++++++++++++++---------------
 2 files changed, 259 insertions(+), 124 deletions(-)

diff --git a/app/test/test_ring.h b/app/test/test_ring.h
index 45c263f3ff..e8b7525c23 100644
--- a/app/test/test_ring.h
+++ b/app/test/test_ring.h
@@ -1,10 +1,12 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */
 
 #include <rte_malloc.h>
 #include <rte_ring.h>
 #include <rte_ring_elem.h>
+#include <rte_memcpy.h>
+#include <rte_ptr_compress.h>
 
 /* API type to call
  * rte_ring_<sp/mp or sc/mc>_enqueue_<bulk/burst>
@@ -25,6 +27,9 @@
 #define TEST_RING_ELEM_BULK 16
 #define TEST_RING_ELEM_BURST 32
 
+#define TEST_RING_ELEM_BURST_ZC 64
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 128
+
 #define TEST_RING_IGNORE_API_TYPE ~0U
 
 /* This function is placed here as it is required for both
@@ -101,6 +106,9 @@ static inline unsigned int
 test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -152,6 +160,29 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n,
 								NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy(zcd.ptr2,
+						(char *)obj + zcd.n1 * esize,
+						(ret - zcd.n1) * esize);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_32(0, obj, zcd.ptr1, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_32(0, obj + zcd.n1,
+						zcd.ptr2, ret - zcd.n1, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
@@ -162,6 +193,9 @@ static inline unsigned int
 test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -213,6 +247,29 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mc_dequeue_burst_elem(r, obj, esize,
 								n, NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy((char *)obj + zcd.n1 * esize,
+						zcd.ptr2,
+						(ret - zcd.n1) * esize);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_32(0, zcd.ptr1, obj, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_32(0, zcd.ptr2,
+						obj + zcd.n1, ret - zcd.n1, 3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c
index d7c5a4c30b..3c9dc24447 100644
--- a/app/test/test_ring_perf.c
+++ b/app/test/test_ring_perf.c
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2010-2014 Intel Corporation
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */
 
 
@@ -21,14 +21,15 @@
  */
 
 #define RING_NAME "RING_PERF"
+#define RING_NAME_COMPRESS "RING_COMP"
 #define RING_SIZE 4096
-#define MAX_BURST 32
+#define MAX_BURST 128
 
 /*
  * the sizes to enqueue and dequeue in testing
  * (marked volatile so they won't be seen as compile-time constants)
  */
-static const volatile unsigned bulk_sizes[] = { 8, 32 };
+static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128 };
 
 struct lcore_pair {
 	unsigned c1, c2;
@@ -43,26 +44,29 @@ test_ring_print_test_string(unsigned int api_type, int esize,
 	if (esize == -1)
 		printf("legacy APIs");
 	else
-		printf("elem APIs: element size %dB", esize);
+		printf("elem APIs (size:%2dB)", esize);
 
 	if (api_type == TEST_RING_IGNORE_API_TYPE)
 		return;
 
 	if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF)
-		printf(": default enqueue/dequeue: ");
+		printf(" - default enqueue/dequeue");
 	else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC)
-		printf(": SP/SC: ");
+		printf(" - SP/SC");
 	else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC)
-		printf(": MP/MC: ");
+		printf(" - MP/MC");
 
 	if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE)
-		printf("single: ");
+		printf(" - single - ");
 	else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK)
-		printf("bulk (size: %u): ", bsz);
+		printf(" - bulk (n:%-3u) - ", bsz);
 	else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST)
-		printf("burst (size: %u): ", bsz);
+		printf(" - burst (n:%-3u) - ", bsz);
+	else if ((api_type & (TEST_RING_ELEM_BURST_ZC |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0)
+		printf(" - burst zero copy (n:%-3u) - ", bsz);
 
-	printf("%.2F\n", value);
+	printf("cycles per elem: %.3F\n", value);
 }
 
 /**** Functions to analyse our core mask to get cores for different tests ***/
@@ -144,28 +148,33 @@ static void
 test_empty_dequeue(struct rte_ring *r, const int esize,
 			const unsigned int api_type)
 {
-	const unsigned int iter_shift = 26;
+	const unsigned int iter_shift = 30;
 	const unsigned int iterations = 1 << iter_shift;
 	unsigned int i = 0;
 	void *burst[MAX_BURST];
 
+	const unsigned int bulk_iterations = iterations / bulk_sizes[0];
 	const uint64_t start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	for (i = 0; i < bulk_iterations; i++)
 		test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type);
 	const uint64_t end = rte_rdtsc();
 
 	test_ring_print_test_string(api_type, esize, bulk_sizes[0],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 }
 
-/*
- * for the separate enqueue and dequeue threads they take in one param
- * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc
- */
-struct thread_params {
+/* describes the ring used by the enqueue and dequeue thread */
+struct ring_params {
 	struct rte_ring *r;
-	unsigned size;        /* input value, the burst size */
-	double spsc, mpmc;    /* output value, the single or multi timings */
+	unsigned int elem_size;
+	unsigned int bulk_sizes_i; /* index into bulk_size array */
+	unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */
+};
+
+/* Used do specify enqueue and dequeue ring operations and their results */
+struct thread_params {
+	struct ring_params *ring_params;
+	double *results; /* result array size must be equal to bulk_sizes */
 };
 
 /*
@@ -174,16 +183,15 @@ struct thread_params {
  * flag == 1 -> dequeue
  */
 static __rte_always_inline int
-enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
-	struct thread_params *p)
+enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 15;
+	const unsigned int iter_shift = 30;
 	const unsigned int iterations = 1 << iter_shift;
-	struct rte_ring *r = p->r;
-	unsigned int bsize = p->size;
 	unsigned int i;
 	void *burst = NULL;
+	unsigned int n_remaining;
+	const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i];
 
 #ifdef RTE_USE_C11_MEM_MODEL
 	if (__atomic_fetch_add(&lcore_count, 1, __ATOMIC_RELAXED) + 1 != 2)
@@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 		while(lcore_count != 2)
 			rte_pause();
 
-	burst = test_ring_calloc(MAX_BURST, esize);
+	burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size);
 	if (burst == NULL)
 		return -1;
 
 	const uint64_t sp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	const unsigned int bulk_iterations = iterations / bulk_n;
+	for (i = 0; i < bulk_iterations; i++) {
+		n_remaining = bulk_n;
 		do {
 			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_enqueue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_dequeue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			if (ret == 0)
 				rte_pause();
-		} while (!ret);
+			else
+				n_remaining -= ret;
+		} while (n_remaining > 0);
+	}
 	const uint64_t sp_end = rte_rdtsc();
 
-	const uint64_t mp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
-		do {
-			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			if (ret == 0)
-				rte_pause();
-		} while (!ret);
-	const uint64_t mp_end = rte_rdtsc();
+	p->results[p->ring_params->bulk_sizes_i] =
+			((double)sp_end - sp_start) / iterations;
 
-	p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize);
-	p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize);
 	return 0;
 }
 
@@ -243,15 +245,7 @@ enqueue_bulk(void *p)
 {
 	struct thread_params *params = p;
 
-	return enqueue_dequeue_bulk_helper(0, -1, params);
-}
-
-static int
-enqueue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(0, 16, params);
+	return enqueue_dequeue_bulk_helper(0, params);
 }
 
 /*
@@ -263,15 +257,7 @@ dequeue_bulk(void *p)
 {
 	struct thread_params *params = p;
 
-	return enqueue_dequeue_bulk_helper(1, -1, params);
-}
-
-static int
-dequeue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(1, 16, params);
+	return enqueue_dequeue_bulk_helper(1, params);
 }
 
 /*
@@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p)
  * used to measure ring perf between hyperthreads, cores and sockets.
  */
 static int
-run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize)
+run_on_core_pair(struct lcore_pair *cores,
+		struct thread_params *param1, struct thread_params *param2)
 {
-	lcore_function_t *f1, *f2;
-	struct thread_params param1 = {0}, param2 = {0};
 	unsigned i;
-
-	if (esize == -1) {
-		f1 = enqueue_bulk;
-		f2 = dequeue_bulk;
-	} else {
-		f1 = enqueue_bulk_16B;
-		f2 = dequeue_bulk_16B;
-	}
+	struct ring_params *ring_params = param1->ring_params;
 
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		lcore_count = 0;
-		param1.size = param2.size = bulk_sizes[i];
-		param1.r = param2.r = r;
+		ring_params->bulk_sizes_i = i;
 		if (cores->c1 == rte_get_main_lcore()) {
-			rte_eal_remote_launch(f2, &param2, cores->c2);
-			f1(&param1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
+			enqueue_bulk(param1);
 			rte_eal_wait_lcore(cores->c2);
 		} else {
-			rte_eal_remote_launch(f1, &param1, cores->c1);
-			rte_eal_remote_launch(f2, &param2, cores->c2);
+			rte_eal_remote_launch(enqueue_bulk, param1, cores->c1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
 			if (rte_eal_wait_lcore(cores->c1) < 0)
 				return -1;
 			if (rte_eal_wait_lcore(cores->c2) < 0)
 				return -1;
 		}
 		test_ring_print_test_string(
-			TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.spsc + param2.spsc);
-		test_ring_print_test_string(
-			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.mpmc + param2.mpmc);
+				ring_params->ring_flags,
+				ring_params->elem_size,
+				bulk_sizes[i],
+				param1->results[i] + param2->results[i]);
 	}
 
 	return 0;
@@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 	uint64_t hz = rte_get_timer_hz();
 	uint64_t lcount = 0;
 	const unsigned int lcore = rte_lcore_id();
-	struct thread_params *params = p;
+	struct ring_params *ring_params = p->ring_params;
 	void *burst = NULL;
 
 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -346,9 +322,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 
 	begin = rte_get_timer_cycles();
 	while (time_diff < hz * TIME_MS / 1000) {
-		test_ring_enqueue(params->r, burst, esize, params->size,
+		test_ring_enqueue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
-		test_ring_dequeue(params->r, burst, esize, params->size,
+		test_ring_dequeue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 		lcount++;
 		time_diff = rte_get_timer_cycles() - begin;
@@ -380,7 +358,8 @@ static int
 run_on_all_cores(struct rte_ring *r, const int esize)
 {
 	uint64_t total;
-	struct thread_params param;
+	struct ring_params ring_params = {0};
+	struct thread_params params = { .ring_params = &ring_params };
 	lcore_function_t *lcore_f;
 	unsigned int i, c;
 
@@ -389,21 +368,20 @@ run_on_all_cores(struct rte_ring *r, const int esize)
 	else
 		lcore_f = load_loop_fn_16B;
 
-	memset(&param, 0, sizeof(struct thread_params));
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		total = 0;
 		printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]);
-		param.size = bulk_sizes[i];
-		param.r = r;
+		params.ring_params->bulk_sizes_i = i;
+		params.ring_params->r = r;
 
 		/* clear synchro and start workers */
 		__atomic_store_n(&synchro, 0, __ATOMIC_RELAXED);
-		if (rte_eal_mp_remote_launch(lcore_f, &param, SKIP_MAIN) < 0)
+		if (rte_eal_mp_remote_launch(lcore_f, &params, SKIP_MAIN) < 0)
 			return -1;
 
 		/* start synchro and launch test on main */
 		__atomic_store_n(&synchro, 1, __ATOMIC_RELAXED);
-		lcore_f(&param);
+		lcore_f(&params);
 
 		rte_eal_mp_wait_lcore();
 
@@ -462,9 +440,9 @@ static int
 test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	const unsigned int api_type)
 {
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 27;
 	const unsigned int iterations = 1 << iter_shift;
-	unsigned int sz, i = 0;
+	unsigned int sz, i;
 	void **burst = NULL;
 
 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -472,17 +450,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 		return -1;
 
 	for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) {
+		const unsigned int n = iterations / bulk_sizes[sz];
 		const uint64_t start = rte_rdtsc();
-		for (i = 0; i < iterations; i++) {
+		for (i = 0; i < n; i++) {
 			test_ring_enqueue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 			test_ring_dequeue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 		}
 		const uint64_t end = rte_rdtsc();
 
 		test_ring_print_test_string(api_type, esize, bulk_sizes[sz],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 	}
 
 	rte_free(burst);
@@ -490,12 +469,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	return 0;
 }
 
+static __rte_always_inline int
+test_ring_perf_esize_run_on_two_cores(
+		struct thread_params *param1, struct thread_params *param2)
+{
+	struct lcore_pair cores;
+
+	if (get_two_hyperthreads(&cores) == 0) {
+		printf("\n### Testing using two hyperthreads ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing using two physical cores ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_sockets(&cores) == 0) {
+		printf("\n### Testing using two NUMA nodes ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	return 0;
+}
+
 /* Run all tests for a given element size */
 static __rte_always_inline int
 test_ring_perf_esize(const int esize)
 {
-	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
+	double results_enq[RTE_DIM(bulk_sizes)];
+	double results_deq[RTE_DIM(bulk_sizes)];
+	struct ring_params ring_params = {
+			.elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results_enq };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results_deq };
 
 	/*
 	 * Performance test for legacy/_elem APIs
@@ -535,22 +545,13 @@ test_ring_perf_esize(const int esize)
 	test_empty_dequeue(r, esize,
 			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 
-	if (get_two_hyperthreads(&cores) == 0) {
-		printf("\n### Testing using two hyperthreads ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.r = r;
 
-	if (get_two_cores(&cores) == 0) {
-		printf("\n### Testing using two physical cores ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
-	if (get_two_sockets(&cores) == 0) {
-		printf("\n### Testing using two NUMA nodes ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
+
+	ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
 
 	printf("\n### Testing using all worker nodes ###\n");
 	if (run_on_all_cores(r, esize) < 0)
@@ -566,6 +567,79 @@ test_ring_perf_esize(const int esize)
 	return -1;
 }
 
+
+static __rte_always_inline int
+test_ring_perf_compression(void)
+{
+	double results1[RTE_DIM(bulk_sizes)];
+	double results2[RTE_DIM(bulk_sizes)];
+	double results1_comp[RTE_DIM(bulk_sizes)];
+	double results2_comp[RTE_DIM(bulk_sizes)];
+
+	struct lcore_pair cores;
+	int ret = -1;
+	unsigned int i;
+	struct ring_params ring_params = { .elem_size = sizeof(void *) };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results1 };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results2 };
+
+	printf("\n### Testing compression gain ###");
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(void *),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy ###\n");
+		ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME_COMPRESS, sizeof(uint32_t),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	/* swap out the result arrays for the compression ones */
+	param1.results = results1_comp;
+	param2.results = results2_comp;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	printf("\n\n### Potential gain from compression ###\n");
+	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
+		const double result = results1[i] + results2[i];
+		const double result_comp = results1_comp[i] + results1_comp[i];
+		const double gain = 100 - (result_comp / result) * 100;
+
+		printf("Gain of %5.1F%% for burst of %-3u elems\n",
+				gain, bulk_sizes[i]);
+	}
+
+	return ret;
+}
+
 static int
 test_ring_perf(void)
 {
@@ -576,6 +650,10 @@ test_ring_perf(void)
 	if (test_ring_perf_esize(16) == -1)
 		return -1;
 
+	/* Test for performance gain of compression */
+	if (test_ring_perf_compression() == -1)
+		return -1;
+
 	return 0;
 }
 
-- 
2.25.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [RFC 2/2] test: add pointer compress tests to ring perf test
  2023-09-27 15:08 ` [RFC 2/2] test: add pointer compress tests to ring perf test Paul Szczepanek
@ 2023-10-09 15:48   ` Thomas Monjalon
  0 siblings, 0 replies; 29+ messages in thread
From: Thomas Monjalon @ 2023-10-09 15:48 UTC (permalink / raw)
  To: Paul Szczepanek; +Cc: dev, Honnappa Nagarahalli

27/09/2023 17:08, Paul Szczepanek:
> Add a test that runs a zero copy burst enqueue and dequeue on a ring
> of raw pointers and compressed pointers at different burst sizes to
> showcase performance benefits of newly added pointer compression APIs
> 
> To reuse existing code, some refactoring was done to pass more
> parameters to test threads. Additionally more bulk sizes were
> added to showcase their effects on compression. To keep runtime
> reasoanble iterations where adjusted to take into account bulk sizes.

typo: reasonable

How much time does it take before/after?




^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [RFC 1/2] eal: add pointer compression functions
  2023-09-27 15:08 ` [RFC 1/2] eal: add pointer compression functions Paul Szczepanek
@ 2023-10-09 15:54   ` Thomas Monjalon
  2023-10-11 13:36     ` Honnappa Nagarahalli
  2023-10-11 12:43   ` [RFC v2 0/2] add pointer compression API Paul Szczepanek
                     ` (3 subsequent siblings)
  4 siblings, 1 reply; 29+ messages in thread
From: Thomas Monjalon @ 2023-10-09 15:54 UTC (permalink / raw)
  To: Paul Szczepanek; +Cc: dev, Honnappa Nagarahalli, Kamalakshitha Aligeri

27/09/2023 17:08, Paul Szczepanek:
> Add a new utility header for compressing pointers. Pointers are
> compressed by taking advantage of their locality. Instead of
> storing the full address only an offset from a known base is stored.

You probably need to insert some explanations from the cover letter.

> The provided functions can store pointers in 32bit offsets.
> 
> Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
[...]
> --- a/lib/eal/include/meson.build
> +++ b/lib/eal/include/meson.build
> @@ -35,6 +35,7 @@ headers += files(
>          'rte_pci_dev_feature_defs.h',
>          'rte_pci_dev_features.h',
>          'rte_per_lcore.h',
> +	'rte_ptr_compress.h',
>          'rte_pflock.h',
>          'rte_random.h',
>          'rte_reciprocal.h',

Did you try to sort alphabetically? failed :)

> +#ifndef _RTE_PTR_COMPRESS_H_
> +#define _RTE_PTR_COMPRESS_H_

No need extra underscores.

> +
> +/**
> + * @file
> + * RTE pointer compression and decompression.

RTE has no mean here I think.

> + */
> +
> +#include <stdint.h>
> +#include <inttypes.h>
> +
> +#include <rte_branch_prediction.h>
> +#include <rte_common.h>
> +#include <rte_debug.h>
> +#include <rte_vect.h>
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +/**
> + * Compress pointers into 32 bit offsets from base pointer.

I think it should be "32-bit".

> + *
> + * @note Offsets from the base pointer must fit within 32bits. Alignment allows
> + * us to drop bits from the offsets - this means that for pointers aligned by
> + * 8 bytes they must be within 32GB of the base pointer. Unaligned pointers
> + * must be within 4GB.

Not clear what is "alignment".
> + *
> + * @param ptr_base
> + *   A pointer used to calculate offsets of pointers in src_table.
> + * @param src_table
> + *   A pointer to an array of pointers.
> + * @param dest_table
> + *   A pointer to an array of compressed pointers returned by this function.
> + * @param n
> + *   The number of objects to compress, must be strictly positive.
> + * @param bit_shift
> + *   Byte alignment of memory pointed to by the pointers allows for
> + *   bits to be dropped from the offset and hence widen the memory region that
> + *   can be covered. This controls how many bits are right shifted.
> + **/
> +static __rte_always_inline void
> +rte_ptr_compress_32(void *ptr_base, void **src_table,
> +		uint32_t *dest_table, unsigned int n, unsigned int bit_shift)
> +{
> +	unsigned int i = 0;
> +#if defined RTE_HAS_SVE_ACLE
> +	svuint64_t v_src_table;
> +	svuint64_t v_dest_table;
> +	svbool_t pg = svwhilelt_b64(i, n);
> +	do {
> +		v_src_table = svld1_u64(pg, (uint64_t *)src_table + i);
> +		v_dest_table = svsub_x(pg, v_src_table, (uint64_t)ptr_base);
> +		v_dest_table = svlsr_x(pg, v_dest_table, bit_shift);
> +		svst1w(pg, &dest_table[i], v_dest_table);
> +		i += svcntd();
> +		pg = svwhilelt_b64(i, n);
> +	} while (svptest_any(svptrue_b64(), pg));
> +#elif defined __ARM_NEON
> +	uint64_t ptr_diff;
> +	uint64x2_t v_src_table;
> +	uint64x2_t v_dest_table;
> +	/* right shift is done by left shifting by negative int */
> +	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
> +	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
> +	for (; i < (n & ~0x1); i += 2) {
> +		v_src_table = vld1q_u64((const uint64_t *)src_table + i);
> +		v_dest_table = vsubq_u64(v_src_table, v_ptr_base);
> +		v_dest_table = vshlq_u64(v_dest_table, v_shift);
> +		vst1_u32(dest_table + i, vqmovn_u64(v_dest_table));
> +	}
> +	/* process leftover single item in case of odd number of n */
> +	if (unlikely(n & 0x1)) {
> +		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
> +		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
> +	}
> +#else
> +	uint64_t ptr_diff;
> +	for (; i < n; i++) {
> +		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
> +		/* save extra bits that are redundant due to alignment */
> +		ptr_diff = ptr_diff >> bit_shift;
> +		/* make sure no truncation will happen when casting */
> +		RTE_ASSERT(ptr_diff <= UINT32_MAX);
> +		dest_table[i] = (uint32_t) ptr_diff;
> +	}
> +#endif
> +}

I see it is providing some per-CPU optimizations,
so it is in favor of having it in DPDK.
Other than that, it looks very generic, so it is questionable to have in DPDK.




^ permalink raw reply	[flat|nested] 29+ messages in thread

* [RFC v2 0/2]  add pointer compression API
  2023-09-27 15:08 ` [RFC 1/2] eal: add pointer compression functions Paul Szczepanek
  2023-10-09 15:54   ` Thomas Monjalon
@ 2023-10-11 12:43   ` Paul Szczepanek
  2023-10-11 12:43     ` [RFC v2 1/2] eal: add pointer compression functions Paul Szczepanek
  2023-10-11 12:43     ` [RFC v2 2/2] test: add pointer compress tests to ring perf test Paul Szczepanek
  2023-10-31 18:10   ` [PATCH v3 0/3] add pointer compression API Paul Szczepanek
                     ` (2 subsequent siblings)
  4 siblings, 2 replies; 29+ messages in thread
From: Paul Szczepanek @ 2023-10-11 12:43 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek

This patchset is proposing adding a new EAL header with utility functions
that allow compression of arrays of pointers.

When passing caches full of pointers between threads, memory containing
the pointers is copied multiple times which is especially costly between
cores. A compression method will allow us to shrink the memory size
copied.

The compression takes advantage of the fact that pointers are usually
located in a limited memory region (like a mempool). We can compress them
by converting them to offsets from a base memory address.

Offsets can be stored in fewer bytes (dictated by the memory region size
and alignment of the pointer). For example: an 8 byte aligned pointer
which is part of a 32GB memory pool can be stored in 4 bytes. The API is
very generic and does not assume mempool pointers, any pointer can be
passed in.

Compression is based on few and fast operations and especially with vector
instructions leveraged creates minimal overhead.

The API accepts and returns arrays because the overhead means it only is
worth it when done in bulk.

Test is added that shows potential performance gain from compression. In
this test an array of pointers is passed through a ring between two cores.
It shows the gain which is dependent on the bulk operation size. In this
synthetic test run on ampere altra a substantial (up to 25%) performance
gain is seen if done in bulk size larger than 32. At 32 it breaks even and
lower sizes create a small (less than 5%) slowdown due to overhead.

In a more realistic mock application running the l3 forwarding dpdk
example that works in pipeline mode this translated into a ~5% throughput
increase on an ampere altra.

v2:
* addressed review comments (style, explanations and typos)
* lowered bulk iterations closer to original numbers to keep runtime short
* fixed pointer size warning on 32-bit arch

Paul Szczepanek (2):
  eal: add pointer compression functions
  test: add pointer compress tests to ring perf test

 .mailmap                           |   1 +
 app/test/test_ring.h               |  59 +++++-
 app/test/test_ring_perf.c          | 324 ++++++++++++++++++-----------
 lib/eal/include/meson.build        |   1 +
 lib/eal/include/rte_ptr_compress.h | 160 ++++++++++++++
 5 files changed, 421 insertions(+), 124 deletions(-)
 create mode 100644 lib/eal/include/rte_ptr_compress.h

-- 
2.25.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [RFC v2 1/2] eal: add pointer compression functions
  2023-10-11 12:43   ` [RFC v2 0/2] add pointer compression API Paul Szczepanek
@ 2023-10-11 12:43     ` Paul Szczepanek
  2023-10-11 12:43     ` [RFC v2 2/2] test: add pointer compress tests to ring perf test Paul Szczepanek
  1 sibling, 0 replies; 29+ messages in thread
From: Paul Szczepanek @ 2023-10-11 12:43 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Honnappa Nagarahalli, Kamalakshitha Aligeri

Add a new utility header for compressing pointers. The provided
functions can store pointers in 32-bit offsets.

The compression takes advantage of the fact that pointers are
usually located in a limited memory region (like a mempool).
We can compress them by converting them to offsets from a base
memory address. Offsets can be stored in fewer bytes (dictated
by the memory region size and alignment of the pointer).
For example: an 8 byte aligned pointer which is part of a 32GB
memory pool can be stored in 4 bytes.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .mailmap                           |   1 +
 lib/eal/include/meson.build        |   1 +
 lib/eal/include/rte_ptr_compress.h | 160 +++++++++++++++++++++++++++++
 3 files changed, 162 insertions(+)
 create mode 100644 lib/eal/include/rte_ptr_compress.h

diff --git a/.mailmap b/.mailmap
index 864d33ee46..3f0c9d32f5 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1058,6 +1058,7 @@ Paul Greenwalt <paul.greenwalt@intel.com>
 Paulis Gributs <paulis.gributs@intel.com>
 Paul Luse <paul.e.luse@intel.com>
 Paul M Stillwell Jr <paul.m.stillwell.jr@intel.com>
+Paul Szczepanek <paul.szczepanek@arm.com>
 Pavan Kumar Linga <pavan.kumar.linga@intel.com>
 Pavan Nikhilesh <pbhagavatula@marvell.com> <pbhagavatula@caviumnetworks.com>
 Pavel Belous <pavel.belous@aquantia.com>
diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index a0463efac7..17d8373648 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -36,6 +36,7 @@ headers += files(
         'rte_pci_dev_features.h',
         'rte_per_lcore.h',
         'rte_pflock.h',
+	'rte_ptr_compress.h',
         'rte_random.h',
         'rte_reciprocal.h',
         'rte_seqcount.h',
diff --git a/lib/eal/include/rte_ptr_compress.h b/lib/eal/include/rte_ptr_compress.h
new file mode 100644
index 0000000000..73bde22973
--- /dev/null
+++ b/lib/eal/include/rte_ptr_compress.h
@@ -0,0 +1,160 @@
+/* SPDX-License-Identifier: BSD-shift-Clause
+ * Copyright(c) 2023 Arm Limited
+ */
+
+#ifndef RTE_PTR_COMPRESS_H
+#define RTE_PTR_COMPRESS_H
+
+/**
+ * @file
+ * Pointer compression and decompression.
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_vect.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Compress pointers into 32-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 32 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 32GB of the base pointer. If there is no such alignment guarantee they
+ * must be within 4GB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_32(void *ptr_base, void **src_table,
+		uint32_t *dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_src_table;
+	svuint64_t v_dest_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_src_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_dest_table = svsub_x(pg, v_src_table, (uint64_t)ptr_base);
+		v_dest_table = svlsr_x(pg, v_dest_table, bit_shift);
+		svst1w(pg, &dest_table[i], v_dest_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON
+	uint64_t ptr_diff;
+	uint64x2_t v_src_table;
+	uint64x2_t v_dest_table;
+	/* right shift is done by left shifting by negative int */
+	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_src_table = vld1q_u64((const uint64_t *)src_table + i);
+		v_dest_table = vsubq_u64(v_src_table, v_ptr_base);
+		v_dest_table = vshlq_u64(v_dest_table, v_shift);
+		vst1_u32(dest_table + i, vqmovn_u64(v_dest_table));
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		/* save extra bits that are redundant due to alignment */
+		ptr_diff = ptr_diff >> bit_shift;
+		/* make sure no truncation will happen when casting */
+		RTE_ASSERT(ptr_diff <= UINT32_MAX);
+		dest_table[i] = (uint32_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 32-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_32(void *ptr_base, uint32_t *src_table,
+		void **dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_src_table;
+	svuint64_t v_dest_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_src_table = svld1uw_u64(pg, &src_table[i]);
+		v_src_table = svlsl_x(pg, v_src_table, bit_shift);
+		v_dest_table = svadd_x(pg, v_src_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_dest_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON
+	uint64_t ptr_diff;
+	uint64x2_t v_src_table;
+	uint64x2_t v_dest_table;
+	int64x2_t v_shift = vdupq_n_s64(bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_src_table = vmovl_u32(vld1_u32(src_table + i));
+		v_src_table = vshlq_u64(v_dest_table, v_shift);
+		v_dest_table = vaddq_u64(v_src_table, v_ptr_base);
+		vst1q_u64((uint64_t *)dest_table + i, v_dest_table);
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_PTR_COMPRESS_H */
-- 
2.25.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [RFC v2 2/2] test: add pointer compress tests to ring perf test
  2023-10-11 12:43   ` [RFC v2 0/2] add pointer compression API Paul Szczepanek
  2023-10-11 12:43     ` [RFC v2 1/2] eal: add pointer compression functions Paul Szczepanek
@ 2023-10-11 12:43     ` Paul Szczepanek
  1 sibling, 0 replies; 29+ messages in thread
From: Paul Szczepanek @ 2023-10-11 12:43 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Honnappa Nagarahalli

Add a test that runs a zero copy burst enqueue and dequeue on a ring
of raw pointers and compressed pointers at different burst sizes to
showcase performance benefits of newly added pointer compression APIs.

Refactored threading code to pass more parameters to threads to
reuse existing code. Added more bulk sizes to showcase their effects
on compression. Adjusted loop iteration numbers to take into account
bulk sizes to keep runtime constant (instead of number of operations).

Adjusted old printfs to match new ones which have aligned numbers.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 app/test/test_ring.h      |  59 ++++++-
 app/test/test_ring_perf.c | 324 +++++++++++++++++++++++---------------
 2 files changed, 259 insertions(+), 124 deletions(-)

diff --git a/app/test/test_ring.h b/app/test/test_ring.h
index 45c263f3ff..e8b7525c23 100644
--- a/app/test/test_ring.h
+++ b/app/test/test_ring.h
@@ -1,10 +1,12 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */
 
 #include <rte_malloc.h>
 #include <rte_ring.h>
 #include <rte_ring_elem.h>
+#include <rte_memcpy.h>
+#include <rte_ptr_compress.h>
 
 /* API type to call
  * rte_ring_<sp/mp or sc/mc>_enqueue_<bulk/burst>
@@ -25,6 +27,9 @@
 #define TEST_RING_ELEM_BULK 16
 #define TEST_RING_ELEM_BURST 32
 
+#define TEST_RING_ELEM_BURST_ZC 64
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 128
+
 #define TEST_RING_IGNORE_API_TYPE ~0U
 
 /* This function is placed here as it is required for both
@@ -101,6 +106,9 @@ static inline unsigned int
 test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -152,6 +160,29 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n,
 								NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy(zcd.ptr2,
+						(char *)obj + zcd.n1 * esize,
+						(ret - zcd.n1) * esize);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_32(0, obj, zcd.ptr1, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_32(0, obj + zcd.n1,
+						zcd.ptr2, ret - zcd.n1, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
@@ -162,6 +193,9 @@ static inline unsigned int
 test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -213,6 +247,29 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mc_dequeue_burst_elem(r, obj, esize,
 								n, NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy((char *)obj + zcd.n1 * esize,
+						zcd.ptr2,
+						(ret - zcd.n1) * esize);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_32(0, zcd.ptr1, obj, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_32(0, zcd.ptr2,
+						obj + zcd.n1, ret - zcd.n1, 3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c
index d7c5a4c30b..3c9dc24447 100644
--- a/app/test/test_ring_perf.c
+++ b/app/test/test_ring_perf.c
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2010-2014 Intel Corporation
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */
 
 
@@ -21,14 +21,15 @@
  */
 
 #define RING_NAME "RING_PERF"
+#define RING_NAME_COMPRESS "RING_COMP"
 #define RING_SIZE 4096
-#define MAX_BURST 32
+#define MAX_BURST 128
 
 /*
  * the sizes to enqueue and dequeue in testing
  * (marked volatile so they won't be seen as compile-time constants)
  */
-static const volatile unsigned bulk_sizes[] = { 8, 32 };
+static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128 };
 
 struct lcore_pair {
 	unsigned c1, c2;
@@ -43,26 +44,29 @@ test_ring_print_test_string(unsigned int api_type, int esize,
 	if (esize == -1)
 		printf("legacy APIs");
 	else
-		printf("elem APIs: element size %dB", esize);
+		printf("elem APIs (size:%2dB)", esize);
 
 	if (api_type == TEST_RING_IGNORE_API_TYPE)
 		return;
 
 	if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF)
-		printf(": default enqueue/dequeue: ");
+		printf(" - default enqueue/dequeue");
 	else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC)
-		printf(": SP/SC: ");
+		printf(" - SP/SC");
 	else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC)
-		printf(": MP/MC: ");
+		printf(" - MP/MC");
 
 	if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE)
-		printf("single: ");
+		printf(" - single - ");
 	else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK)
-		printf("bulk (size: %u): ", bsz);
+		printf(" - bulk (n:%-3u) - ", bsz);
 	else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST)
-		printf("burst (size: %u): ", bsz);
+		printf(" - burst (n:%-3u) - ", bsz);
+	else if ((api_type & (TEST_RING_ELEM_BURST_ZC |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0)
+		printf(" - burst zero copy (n:%-3u) - ", bsz);
 
-	printf("%.2F\n", value);
+	printf("cycles per elem: %.3F\n", value);
 }
 
 /**** Functions to analyse our core mask to get cores for different tests ***/
@@ -144,28 +148,33 @@ static void
 test_empty_dequeue(struct rte_ring *r, const int esize,
 			const unsigned int api_type)
 {
-	const unsigned int iter_shift = 26;
+	const unsigned int iter_shift = 30;
 	const unsigned int iterations = 1 << iter_shift;
 	unsigned int i = 0;
 	void *burst[MAX_BURST];
 
+	const unsigned int bulk_iterations = iterations / bulk_sizes[0];
 	const uint64_t start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	for (i = 0; i < bulk_iterations; i++)
 		test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type);
 	const uint64_t end = rte_rdtsc();
 
 	test_ring_print_test_string(api_type, esize, bulk_sizes[0],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 }
 
-/*
- * for the separate enqueue and dequeue threads they take in one param
- * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc
- */
-struct thread_params {
+/* describes the ring used by the enqueue and dequeue thread */
+struct ring_params {
 	struct rte_ring *r;
-	unsigned size;        /* input value, the burst size */
-	double spsc, mpmc;    /* output value, the single or multi timings */
+	unsigned int elem_size;
+	unsigned int bulk_sizes_i; /* index into bulk_size array */
+	unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */
+};
+
+/* Used do specify enqueue and dequeue ring operations and their results */
+struct thread_params {
+	struct ring_params *ring_params;
+	double *results; /* result array size must be equal to bulk_sizes */
 };
 
 /*
@@ -174,16 +183,15 @@ struct thread_params {
  * flag == 1 -> dequeue
  */
 static __rte_always_inline int
-enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
-	struct thread_params *p)
+enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 15;
+	const unsigned int iter_shift = 30;
 	const unsigned int iterations = 1 << iter_shift;
-	struct rte_ring *r = p->r;
-	unsigned int bsize = p->size;
 	unsigned int i;
 	void *burst = NULL;
+	unsigned int n_remaining;
+	const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i];
 
 #ifdef RTE_USE_C11_MEM_MODEL
 	if (__atomic_fetch_add(&lcore_count, 1, __ATOMIC_RELAXED) + 1 != 2)
@@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 		while(lcore_count != 2)
 			rte_pause();
 
-	burst = test_ring_calloc(MAX_BURST, esize);
+	burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size);
 	if (burst == NULL)
 		return -1;
 
 	const uint64_t sp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	const unsigned int bulk_iterations = iterations / bulk_n;
+	for (i = 0; i < bulk_iterations; i++) {
+		n_remaining = bulk_n;
 		do {
 			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_enqueue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_dequeue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			if (ret == 0)
 				rte_pause();
-		} while (!ret);
+			else
+				n_remaining -= ret;
+		} while (n_remaining > 0);
+	}
 	const uint64_t sp_end = rte_rdtsc();
 
-	const uint64_t mp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
-		do {
-			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			if (ret == 0)
-				rte_pause();
-		} while (!ret);
-	const uint64_t mp_end = rte_rdtsc();
+	p->results[p->ring_params->bulk_sizes_i] =
+			((double)sp_end - sp_start) / iterations;
 
-	p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize);
-	p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize);
 	return 0;
 }
 
@@ -243,15 +245,7 @@ enqueue_bulk(void *p)
 {
 	struct thread_params *params = p;
 
-	return enqueue_dequeue_bulk_helper(0, -1, params);
-}
-
-static int
-enqueue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(0, 16, params);
+	return enqueue_dequeue_bulk_helper(0, params);
 }
 
 /*
@@ -263,15 +257,7 @@ dequeue_bulk(void *p)
 {
 	struct thread_params *params = p;
 
-	return enqueue_dequeue_bulk_helper(1, -1, params);
-}
-
-static int
-dequeue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(1, 16, params);
+	return enqueue_dequeue_bulk_helper(1, params);
 }
 
 /*
@@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p)
  * used to measure ring perf between hyperthreads, cores and sockets.
  */
 static int
-run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize)
+run_on_core_pair(struct lcore_pair *cores,
+		struct thread_params *param1, struct thread_params *param2)
 {
-	lcore_function_t *f1, *f2;
-	struct thread_params param1 = {0}, param2 = {0};
 	unsigned i;
-
-	if (esize == -1) {
-		f1 = enqueue_bulk;
-		f2 = dequeue_bulk;
-	} else {
-		f1 = enqueue_bulk_16B;
-		f2 = dequeue_bulk_16B;
-	}
+	struct ring_params *ring_params = param1->ring_params;
 
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		lcore_count = 0;
-		param1.size = param2.size = bulk_sizes[i];
-		param1.r = param2.r = r;
+		ring_params->bulk_sizes_i = i;
 		if (cores->c1 == rte_get_main_lcore()) {
-			rte_eal_remote_launch(f2, &param2, cores->c2);
-			f1(&param1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
+			enqueue_bulk(param1);
 			rte_eal_wait_lcore(cores->c2);
 		} else {
-			rte_eal_remote_launch(f1, &param1, cores->c1);
-			rte_eal_remote_launch(f2, &param2, cores->c2);
+			rte_eal_remote_launch(enqueue_bulk, param1, cores->c1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
 			if (rte_eal_wait_lcore(cores->c1) < 0)
 				return -1;
 			if (rte_eal_wait_lcore(cores->c2) < 0)
 				return -1;
 		}
 		test_ring_print_test_string(
-			TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.spsc + param2.spsc);
-		test_ring_print_test_string(
-			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.mpmc + param2.mpmc);
+				ring_params->ring_flags,
+				ring_params->elem_size,
+				bulk_sizes[i],
+				param1->results[i] + param2->results[i]);
 	}
 
 	return 0;
@@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 	uint64_t hz = rte_get_timer_hz();
 	uint64_t lcount = 0;
 	const unsigned int lcore = rte_lcore_id();
-	struct thread_params *params = p;
+	struct ring_params *ring_params = p->ring_params;
 	void *burst = NULL;
 
 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -346,9 +322,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 
 	begin = rte_get_timer_cycles();
 	while (time_diff < hz * TIME_MS / 1000) {
-		test_ring_enqueue(params->r, burst, esize, params->size,
+		test_ring_enqueue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
-		test_ring_dequeue(params->r, burst, esize, params->size,
+		test_ring_dequeue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 		lcount++;
 		time_diff = rte_get_timer_cycles() - begin;
@@ -380,7 +358,8 @@ static int
 run_on_all_cores(struct rte_ring *r, const int esize)
 {
 	uint64_t total;
-	struct thread_params param;
+	struct ring_params ring_params = {0};
+	struct thread_params params = { .ring_params = &ring_params };
 	lcore_function_t *lcore_f;
 	unsigned int i, c;
 
@@ -389,21 +368,20 @@ run_on_all_cores(struct rte_ring *r, const int esize)
 	else
 		lcore_f = load_loop_fn_16B;
 
-	memset(&param, 0, sizeof(struct thread_params));
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		total = 0;
 		printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]);
-		param.size = bulk_sizes[i];
-		param.r = r;
+		params.ring_params->bulk_sizes_i = i;
+		params.ring_params->r = r;
 
 		/* clear synchro and start workers */
 		__atomic_store_n(&synchro, 0, __ATOMIC_RELAXED);
-		if (rte_eal_mp_remote_launch(lcore_f, &param, SKIP_MAIN) < 0)
+		if (rte_eal_mp_remote_launch(lcore_f, &params, SKIP_MAIN) < 0)
 			return -1;
 
 		/* start synchro and launch test on main */
 		__atomic_store_n(&synchro, 1, __ATOMIC_RELAXED);
-		lcore_f(&param);
+		lcore_f(&params);
 
 		rte_eal_mp_wait_lcore();
 
@@ -462,9 +440,9 @@ static int
 test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	const unsigned int api_type)
 {
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 27;
 	const unsigned int iterations = 1 << iter_shift;
-	unsigned int sz, i = 0;
+	unsigned int sz, i;
 	void **burst = NULL;
 
 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -472,17 +450,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 		return -1;
 
 	for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) {
+		const unsigned int n = iterations / bulk_sizes[sz];
 		const uint64_t start = rte_rdtsc();
-		for (i = 0; i < iterations; i++) {
+		for (i = 0; i < n; i++) {
 			test_ring_enqueue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 			test_ring_dequeue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 		}
 		const uint64_t end = rte_rdtsc();
 
 		test_ring_print_test_string(api_type, esize, bulk_sizes[sz],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 	}
 
 	rte_free(burst);
@@ -490,12 +469,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	return 0;
 }
 
+static __rte_always_inline int
+test_ring_perf_esize_run_on_two_cores(
+		struct thread_params *param1, struct thread_params *param2)
+{
+	struct lcore_pair cores;
+
+	if (get_two_hyperthreads(&cores) == 0) {
+		printf("\n### Testing using two hyperthreads ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing using two physical cores ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_sockets(&cores) == 0) {
+		printf("\n### Testing using two NUMA nodes ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	return 0;
+}
+
 /* Run all tests for a given element size */
 static __rte_always_inline int
 test_ring_perf_esize(const int esize)
 {
-	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
+	double results_enq[RTE_DIM(bulk_sizes)];
+	double results_deq[RTE_DIM(bulk_sizes)];
+	struct ring_params ring_params = {
+			.elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results_enq };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results_deq };
 
 	/*
 	 * Performance test for legacy/_elem APIs
@@ -535,22 +545,13 @@ test_ring_perf_esize(const int esize)
 	test_empty_dequeue(r, esize,
 			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 
-	if (get_two_hyperthreads(&cores) == 0) {
-		printf("\n### Testing using two hyperthreads ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.r = r;
 
-	if (get_two_cores(&cores) == 0) {
-		printf("\n### Testing using two physical cores ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
-	if (get_two_sockets(&cores) == 0) {
-		printf("\n### Testing using two NUMA nodes ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
+
+	ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
 
 	printf("\n### Testing using all worker nodes ###\n");
 	if (run_on_all_cores(r, esize) < 0)
@@ -566,6 +567,79 @@ test_ring_perf_esize(const int esize)
 	return -1;
 }
 
+
+static __rte_always_inline int
+test_ring_perf_compression(void)
+{
+	double results1[RTE_DIM(bulk_sizes)];
+	double results2[RTE_DIM(bulk_sizes)];
+	double results1_comp[RTE_DIM(bulk_sizes)];
+	double results2_comp[RTE_DIM(bulk_sizes)];
+
+	struct lcore_pair cores;
+	int ret = -1;
+	unsigned int i;
+	struct ring_params ring_params = { .elem_size = sizeof(void *) };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results1 };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results2 };
+
+	printf("\n### Testing compression gain ###");
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(void *),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy ###\n");
+		ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME_COMPRESS, sizeof(uint32_t),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	/* swap out the result arrays for the compression ones */
+	param1.results = results1_comp;
+	param2.results = results2_comp;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	printf("\n\n### Potential gain from compression ###\n");
+	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
+		const double result = results1[i] + results2[i];
+		const double result_comp = results1_comp[i] + results1_comp[i];
+		const double gain = 100 - (result_comp / result) * 100;
+
+		printf("Gain of %5.1F%% for burst of %-3u elems\n",
+				gain, bulk_sizes[i]);
+	}
+
+	return ret;
+}
+
 static int
 test_ring_perf(void)
 {
@@ -576,6 +650,10 @@ test_ring_perf(void)
 	if (test_ring_perf_esize(16) == -1)
 		return -1;
 
+	/* Test for performance gain of compression */
+	if (test_ring_perf_compression() == -1)
+		return -1;
+
 	return 0;
 }
 
-- 
2.25.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* RE: [RFC 1/2] eal: add pointer compression functions
  2023-10-09 15:54   ` Thomas Monjalon
@ 2023-10-11 13:36     ` Honnappa Nagarahalli
  2023-10-11 16:43       ` Paul Szczepanek
  0 siblings, 1 reply; 29+ messages in thread
From: Honnappa Nagarahalli @ 2023-10-11 13:36 UTC (permalink / raw)
  To: thomas, Paul Szczepanek; +Cc: dev, Kamalakshitha Aligeri, nd



> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Monday, October 9, 2023 10:54 AM
> To: Paul Szczepanek <Paul.Szczepanek@arm.com>
> Cc: dev@dpdk.org; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>;
> Kamalakshitha Aligeri <Kamalakshitha.Aligeri@arm.com>
> Subject: Re: [RFC 1/2] eal: add pointer compression functions
> 
> 27/09/2023 17:08, Paul Szczepanek:
> > Add a new utility header for compressing pointers. Pointers are
> > compressed by taking advantage of their locality. Instead of storing
> > the full address only an offset from a known base is stored.
> 
> You probably need to insert some explanations from the cover letter.
> 
> > The provided functions can store pointers in 32bit offsets.
> >
> > Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> > Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
> > Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
> > Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> [...]
> > --- a/lib/eal/include/meson.build
> > +++ b/lib/eal/include/meson.build
> > @@ -35,6 +35,7 @@ headers += files(
> >          'rte_pci_dev_feature_defs.h',
> >          'rte_pci_dev_features.h',
> >          'rte_per_lcore.h',
> > +	'rte_ptr_compress.h',
> >          'rte_pflock.h',
> >          'rte_random.h',
> >          'rte_reciprocal.h',
> 
> Did you try to sort alphabetically? failed :)
> 
> > +#ifndef _RTE_PTR_COMPRESS_H_
> > +#define _RTE_PTR_COMPRESS_H_
> 
> No need extra underscores.
> 
> > +
> > +/**
> > + * @file
> > + * RTE pointer compression and decompression.
> 
> RTE has no mean here I think.
> 
> > + */
> > +
> > +#include <stdint.h>
> > +#include <inttypes.h>
> > +
> > +#include <rte_branch_prediction.h>
> > +#include <rte_common.h>
> > +#include <rte_debug.h>
> > +#include <rte_vect.h>
> > +
> > +#ifdef __cplusplus
> > +extern "C" {
> > +#endif
> > +
> > +/**
> > + * Compress pointers into 32 bit offsets from base pointer.
> 
> I think it should be "32-bit".
> 
> > + *
> > + * @note Offsets from the base pointer must fit within 32bits.
> > + Alignment allows
> > + * us to drop bits from the offsets - this means that for pointers
> > + aligned by
> > + * 8 bytes they must be within 32GB of the base pointer. Unaligned
> > + pointers
> > + * must be within 4GB.
> 
> Not clear what is "alignment".
> > + *
> > + * @param ptr_base
> > + *   A pointer used to calculate offsets of pointers in src_table.
> > + * @param src_table
> > + *   A pointer to an array of pointers.
> > + * @param dest_table
> > + *   A pointer to an array of compressed pointers returned by this function.
> > + * @param n
> > + *   The number of objects to compress, must be strictly positive.
> > + * @param bit_shift
> > + *   Byte alignment of memory pointed to by the pointers allows for
> > + *   bits to be dropped from the offset and hence widen the memory region
> that
> > + *   can be covered. This controls how many bits are right shifted.
> > + **/
> > +static __rte_always_inline void
> > +rte_ptr_compress_32(void *ptr_base, void **src_table,
> > +		uint32_t *dest_table, unsigned int n, unsigned int bit_shift) {
> > +	unsigned int i = 0;
> > +#if defined RTE_HAS_SVE_ACLE
> > +	svuint64_t v_src_table;
> > +	svuint64_t v_dest_table;
> > +	svbool_t pg = svwhilelt_b64(i, n);
> > +	do {
> > +		v_src_table = svld1_u64(pg, (uint64_t *)src_table + i);
> > +		v_dest_table = svsub_x(pg, v_src_table, (uint64_t)ptr_base);
> > +		v_dest_table = svlsr_x(pg, v_dest_table, bit_shift);
> > +		svst1w(pg, &dest_table[i], v_dest_table);
> > +		i += svcntd();
> > +		pg = svwhilelt_b64(i, n);
> > +	} while (svptest_any(svptrue_b64(), pg)); #elif defined __ARM_NEON
> > +	uint64_t ptr_diff;
> > +	uint64x2_t v_src_table;
> > +	uint64x2_t v_dest_table;
> > +	/* right shift is done by left shifting by negative int */
> > +	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
> > +	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
> > +	for (; i < (n & ~0x1); i += 2) {
> > +		v_src_table = vld1q_u64((const uint64_t *)src_table + i);
> > +		v_dest_table = vsubq_u64(v_src_table, v_ptr_base);
> > +		v_dest_table = vshlq_u64(v_dest_table, v_shift);
> > +		vst1_u32(dest_table + i, vqmovn_u64(v_dest_table));
> > +	}
> > +	/* process leftover single item in case of odd number of n */
> > +	if (unlikely(n & 0x1)) {
> > +		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
> > +		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
> > +	}
> > +#else
> > +	uint64_t ptr_diff;
> > +	for (; i < n; i++) {
> > +		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
> > +		/* save extra bits that are redundant due to alignment */
> > +		ptr_diff = ptr_diff >> bit_shift;
> > +		/* make sure no truncation will happen when casting */
> > +		RTE_ASSERT(ptr_diff <= UINT32_MAX);
> > +		dest_table[i] = (uint32_t) ptr_diff;
> > +	}
> > +#endif
> > +}
> 
> I see it is providing some per-CPU optimizations, so it is in favor of having it in
> DPDK.
> Other than that, it looks very generic, so it is questionable to have in DPDK.
We had it done for mbuf pointers. But then, we thought it could be generic.

Right now the API results in 32b indices. We could make it generic further by allowing for 16b indices. 8b indices does not make sense.
> 
> 


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [RFC 1/2] eal: add pointer compression functions
  2023-10-11 13:36     ` Honnappa Nagarahalli
@ 2023-10-11 16:43       ` Paul Szczepanek
  0 siblings, 0 replies; 29+ messages in thread
From: Paul Szczepanek @ 2023-10-11 16:43 UTC (permalink / raw)
  To: Honnappa Nagarahalli, thomas; +Cc: nd, dev, Kamalakshitha Aligeri

On 11/10/2023 14:36, Honnappa Nagarahalli wrote:
>> -----Original Message-----
>> From: Thomas Monjalon <thomas@monjalon.net>
>> Sent: Monday, October 9, 2023 10:54 AM
>> To: Paul Szczepanek <Paul.Szczepanek@arm.com>
>> Cc: dev@dpdk.org; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>;
>> Kamalakshitha Aligeri <Kamalakshitha.Aligeri@arm.com>
>> Subject: Re: [RFC 1/2] eal: add pointer compression functions
[...]
>> I see it is providing some per-CPU optimizations, so it is in favor of having it in
>> DPDK.
>> Other than that, it looks very generic, so it is questionable to have in DPDK.
> We had it done for mbuf pointers. But then, we thought it could be generic.
>
> Right now the API results in 32b indices. We could make it generic further by allowing for 16b indices. 8b indices does not make sense.

To add to this, this being generic is I think a good thing.

I think it belongs in DPDK as it will make it easy for other 
architectures to add their versions and maintain the abstraction.


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v3 0/3] add pointer compression API
  2023-09-27 15:08 ` [RFC 1/2] eal: add pointer compression functions Paul Szczepanek
  2023-10-09 15:54   ` Thomas Monjalon
  2023-10-11 12:43   ` [RFC v2 0/2] add pointer compression API Paul Szczepanek
@ 2023-10-31 18:10   ` Paul Szczepanek
  2023-10-31 18:10     ` [PATCH v3 1/3] eal: add pointer compression functions Paul Szczepanek
                       ` (3 more replies)
  2023-11-01 12:46   ` [PATCH v4 0/4] " Paul Szczepanek
  2023-11-01 18:12   ` [PATCH v5 0/4] add pointer compression API Paul Szczepanek
  4 siblings, 4 replies; 29+ messages in thread
From: Paul Szczepanek @ 2023-10-31 18:10 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek

This patchset is proposing adding a new EAL header with utility functions
that allow compression of arrays of pointers.

When passing caches full of pointers between threads, memory containing
the pointers is copied multiple times which is especially costly between
cores. A compression method will allow us to shrink the memory size
copied.

The compression takes advantage of the fact that pointers are usually
located in a limited memory region (like a mempool). We can compress them
by converting them to offsets from a base memory address.

Offsets can be stored in fewer bytes (dictated by the memory region size
and alignment of the pointer). For example: an 8 byte aligned pointer
which is part of a 32GB memory pool can be stored in 4 bytes. The API is
very generic and does not assume mempool pointers, any pointer can be
passed in.

Compression is based on few and fast operations and especially with vector
instructions leveraged creates minimal overhead.

The API accepts and returns arrays because the overhead means it only is
worth it when done in bulk.

Test is added that shows potential performance gain from compression. In
this test an array of pointers is passed through a ring between two cores.
It shows the gain which is dependent on the bulk operation size. In this
synthetic test run on ampere altra a substantial (up to 25%) performance
gain is seen if done in bulk size larger than 32. At 32 it breaks even and
lower sizes create a small (less than 5%) slowdown due to overhead.

In a more realistic mock application running the l3 forwarding dpdk
example that works in pipeline mode this translated into a ~5% throughput
increase on an ampere altra.

v2:
* addressed review comments (style, explanations and typos)
* lowered bulk iterations closer to original numbers to keep runtime short
* fixed pointer size warning on 32-bit arch
v3:
* added 16 bit versions of compression functions and tests
* added documentation of these new utility functions in the EAL guide

Paul Szczepanek (3):
  eal: add pointer compression functions
  test: add pointer compress tests to ring perf test
  docs: add pointer compression to the EAL guide

 .mailmap                                      |   1 +
 app/test/test_ring.h                          |  94 ++++-
 app/test/test_ring_perf.c                     | 354 ++++++++++++------
 .../prog_guide/env_abstraction_layer.rst      | 142 +++++++
 lib/eal/include/meson.build                   |   1 +
 lib/eal/include/rte_ptr_compress.h            | 272 ++++++++++++++
 6 files changed, 740 insertions(+), 124 deletions(-)
 create mode 100644 lib/eal/include/rte_ptr_compress.h

--
2.25.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v3 1/3] eal: add pointer compression functions
  2023-10-31 18:10   ` [PATCH v3 0/3] add pointer compression API Paul Szczepanek
@ 2023-10-31 18:10     ` Paul Szczepanek
  2023-10-31 18:10     ` [PATCH v3 2/3] test: add pointer compress tests to ring perf test Paul Szczepanek
                       ` (2 subsequent siblings)
  3 siblings, 0 replies; 29+ messages in thread
From: Paul Szczepanek @ 2023-10-31 18:10 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Honnappa Nagarahalli, Kamalakshitha Aligeri

Add a new utility header for compressing pointers. The provided
functions can store pointers in 32-bit offsets.

The compression takes advantage of the fact that pointers are
usually located in a limited memory region (like a mempool).
We can compress them by converting them to offsets from a base
memory address. Offsets can be stored in fewer bytes (dictated
by the memory region size and alignment of the pointer).
For example: an 8 byte aligned pointer which is part of a 32GB
memory pool can be stored in 4 bytes.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .mailmap                           |   1 +
 lib/eal/include/meson.build        |   1 +
 lib/eal/include/rte_ptr_compress.h | 272 +++++++++++++++++++++++++++++
 3 files changed, 274 insertions(+)
 create mode 100644 lib/eal/include/rte_ptr_compress.h

diff --git a/.mailmap b/.mailmap
index 864d33ee46..3f0c9d32f5 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1058,6 +1058,7 @@ Paul Greenwalt <paul.greenwalt@intel.com>
 Paulis Gributs <paulis.gributs@intel.com>
 Paul Luse <paul.e.luse@intel.com>
 Paul M Stillwell Jr <paul.m.stillwell.jr@intel.com>
+Paul Szczepanek <paul.szczepanek@arm.com>
 Pavan Kumar Linga <pavan.kumar.linga@intel.com>
 Pavan Nikhilesh <pbhagavatula@marvell.com> <pbhagavatula@caviumnetworks.com>
 Pavel Belous <pavel.belous@aquantia.com>
diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index a0463efac7..17d8373648 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -36,6 +36,7 @@ headers += files(
         'rte_pci_dev_features.h',
         'rte_per_lcore.h',
         'rte_pflock.h',
+	'rte_ptr_compress.h',
         'rte_random.h',
         'rte_reciprocal.h',
         'rte_seqcount.h',
diff --git a/lib/eal/include/rte_ptr_compress.h b/lib/eal/include/rte_ptr_compress.h
new file mode 100644
index 0000000000..ceb4662c14
--- /dev/null
+++ b/lib/eal/include/rte_ptr_compress.h
@@ -0,0 +1,272 @@
+/* SPDX-License-Identifier: BSD-shift-Clause
+ * Copyright(c) 2023 Arm Limited
+ */
+
+#ifndef RTE_PTR_COMPRESS_H
+#define RTE_PTR_COMPRESS_H
+
+/**
+ * @file
+ * Pointer compression and decompression functions.
+ *
+ * When passing arrays full of pointers between threads, memory containing
+ * the pointers is copied multiple times which is especially costly between
+ * cores. These functions allow us to compress the pointers.
+ *
+ * Compression takes advantage of the fact that pointers are usually located in
+ * a limited memory region (like a mempool). We compress them by converting them
+ * to offsets from a base memory address. Offsets can be stored in fewer bytes.
+ *
+ * The compression functions come in two varieties: 32-bit and 16-bit.
+ *
+ * To determine how many bits are needed to compress the pointer calculate
+ * the biggest offset possible (highest value pointer - base pointer)
+ * and shift the value right according to alignment (shift by exponent of the
+ * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by
+ * 3, etc.). The resulting value must fit in either 32 or 16 bits.
+ *
+ * For usage example and further explanation please see "Pointer Compression" in
+ * doc/guides/prog_guide/env_abstraction_layer.rst
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_vect.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Compress pointers into 32-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 32 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 32GB of the base pointer. If there is no such alignment guarantee they
+ * must be within 4GB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_32(void *ptr_base, void **src_table,
+		uint32_t *dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_src_table;
+	svuint64_t v_dest_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_src_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_dest_table = svsub_x(pg, v_src_table, (uint64_t)ptr_base);
+		v_dest_table = svlsr_x(pg, v_dest_table, bit_shift);
+		svst1w(pg, &dest_table[i], v_dest_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON
+	uint64_t ptr_diff;
+	uint64x2_t v_src_table;
+	uint64x2_t v_dest_table;
+	/* right shift is done by left shifting by negative int */
+	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_src_table = vld1q_u64((const uint64_t *)src_table + i);
+		v_dest_table = vsubq_u64(v_src_table, v_ptr_base);
+		v_dest_table = vshlq_u64(v_dest_table, v_shift);
+		vst1_u32(dest_table + i, vqmovn_u64(v_dest_table));
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT32_MAX);
+		dest_table[i] = (uint32_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 32-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_32(void *ptr_base, uint32_t *src_table,
+		void **dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_src_table;
+	svuint64_t v_dest_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_src_table = svld1uw_u64(pg, &src_table[i]);
+		v_src_table = svlsl_x(pg, v_src_table, bit_shift);
+		v_dest_table = svadd_x(pg, v_src_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_dest_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON
+	uint64_t ptr_diff;
+	uint64x2_t v_src_table;
+	uint64x2_t v_dest_table;
+	int64x2_t v_shift = vdupq_n_s64(bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_src_table = vmovl_u32(vld1_u32(src_table + i));
+		v_src_table = vshlq_u64(v_dest_table, v_shift);
+		v_dest_table = vaddq_u64(v_src_table, v_ptr_base);
+		vst1q_u64((uint64_t *)dest_table + i, v_dest_table);
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+/**
+ * Compress pointers into 16-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 16 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 256KB of the base pointer. If there is no such alignment guarantee
+ * they must be within 64KB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_16(void *ptr_base, void **src_table,
+		uint16_t *dest_table, unsigned int n, unsigned int bit_shift)
+{
+
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_src_table;
+	svuint64_t v_dest_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_src_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_dest_table = svsub_x(pg, v_src_table, (uint64_t)ptr_base);
+		v_dest_table = svlsr_x(pg, v_dest_table, bit_shift);
+		svst1h(pg, &dest_table[i], v_dest_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT16_MAX);
+		dest_table[i] = (uint16_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 16-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_16(void *ptr_base, uint16_t *src_table,
+		void **dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_src_table;
+	svuint64_t v_dest_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_src_table = svld1uh_u64(pg, &src_table[i]);
+		v_src_table = svlsl_x(pg, v_src_table, bit_shift);
+		v_dest_table = svadd_x(pg, v_src_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_dest_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_PTR_COMPRESS_H */
--
2.25.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v3 2/3] test: add pointer compress tests to ring perf test
  2023-10-31 18:10   ` [PATCH v3 0/3] add pointer compression API Paul Szczepanek
  2023-10-31 18:10     ` [PATCH v3 1/3] eal: add pointer compression functions Paul Szczepanek
@ 2023-10-31 18:10     ` Paul Szczepanek
  2023-10-31 18:10     ` [PATCH v3 3/3] docs: add pointer compression to the EAL guide Paul Szczepanek
  2023-11-01  7:42     ` [PATCH v3 0/3] add pointer compression API Morten Brørup
  3 siblings, 0 replies; 29+ messages in thread
From: Paul Szczepanek @ 2023-10-31 18:10 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Honnappa Nagarahalli

Add a test that runs a zero copy burst enqueue and dequeue on a ring
of raw pointers and compressed pointers at different burst sizes to
showcase performance benefits of newly added pointer compression APIs.

Refactored threading code to pass more parameters to threads to
reuse existing code. Added more bulk sizes to showcase their effects
on compression. Adjusted loop iteration numbers to take into account
bulk sizes to keep runtime constant (instead of number of operations).

Adjusted old printfs to match new ones which have aligned numbers.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 app/test/test_ring.h      |  94 +++++++++-
 app/test/test_ring_perf.c | 354 +++++++++++++++++++++++++-------------
 2 files changed, 324 insertions(+), 124 deletions(-)

diff --git a/app/test/test_ring.h b/app/test/test_ring.h
index 45c263f3ff..3b00f2465d 100644
--- a/app/test/test_ring.h
+++ b/app/test/test_ring.h
@@ -1,10 +1,12 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */

 #include <rte_malloc.h>
 #include <rte_ring.h>
 #include <rte_ring_elem.h>
+#include <rte_memcpy.h>
+#include <rte_ptr_compress.h>

 /* API type to call
  * rte_ring_<sp/mp or sc/mc>_enqueue_<bulk/burst>
@@ -25,6 +27,10 @@
 #define TEST_RING_ELEM_BULK 16
 #define TEST_RING_ELEM_BURST 32

+#define TEST_RING_ELEM_BURST_ZC 64
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 128
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 256
+
 #define TEST_RING_IGNORE_API_TYPE ~0U

 /* This function is placed here as it is required for both
@@ -101,6 +107,9 @@ static inline unsigned int
 test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -152,6 +161,46 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n,
 								NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy(zcd.ptr2,
+						(char *)obj + zcd.n1 * esize,
+						(ret - zcd.n1) * esize);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_16(0, obj, zcd.ptr1, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_16(0,
+						obj + (zcd.n1 * 2),
+						zcd.ptr2,
+						(ret - zcd.n1) * 2, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_32(0, obj, zcd.ptr1, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_32(0, obj + zcd.n1,
+						zcd.ptr2, ret - zcd.n1, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
@@ -162,6 +211,9 @@ static inline unsigned int
 test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -213,6 +265,46 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mc_dequeue_burst_elem(r, obj, esize,
 								n, NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy((char *)obj + zcd.n1 * esize,
+						zcd.ptr2,
+						(ret - zcd.n1) * esize);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_16(0, zcd.ptr1, obj, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_16(0, zcd.ptr2,
+						obj + zcd.n1,
+						(ret - zcd.n1) * 2,
+						3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_32(0, zcd.ptr1, obj, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_32(0, zcd.ptr2,
+						obj + zcd.n1, ret - zcd.n1, 3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c
index d7c5a4c30b..8392d0c781 100644
--- a/app/test/test_ring_perf.c
+++ b/app/test/test_ring_perf.c
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2010-2014 Intel Corporation
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */


@@ -22,13 +22,13 @@

 #define RING_NAME "RING_PERF"
 #define RING_SIZE 4096
-#define MAX_BURST 32
+#define MAX_BURST 256

 /*
  * the sizes to enqueue and dequeue in testing
  * (marked volatile so they won't be seen as compile-time constants)
  */
-static const volatile unsigned bulk_sizes[] = { 8, 32 };
+static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128, 256 };

 struct lcore_pair {
 	unsigned c1, c2;
@@ -43,26 +43,30 @@ test_ring_print_test_string(unsigned int api_type, int esize,
 	if (esize == -1)
 		printf("legacy APIs");
 	else
-		printf("elem APIs: element size %dB", esize);
+		printf("elem APIs (size:%2dB)", esize);

 	if (api_type == TEST_RING_IGNORE_API_TYPE)
 		return;

 	if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF)
-		printf(": default enqueue/dequeue: ");
+		printf(" - default enqueue/dequeue");
 	else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC)
-		printf(": SP/SC: ");
+		printf(" - SP/SC");
 	else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC)
-		printf(": MP/MC: ");
+		printf(" - MP/MC");

 	if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE)
-		printf("single: ");
+		printf(" - single - ");
 	else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK)
-		printf("bulk (size: %u): ", bsz);
+		printf(" - bulk (n:%-3u) - ", bsz);
 	else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST)
-		printf("burst (size: %u): ", bsz);
+		printf(" - burst (n:%-3u) - ", bsz);
+	else if ((api_type & (TEST_RING_ELEM_BURST_ZC |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0)
+		printf(" - burst zero copy (n:%-3u) - ", bsz);

-	printf("%.2F\n", value);
+	printf("cycles per elem: %.3F\n", value);
 }

 /**** Functions to analyse our core mask to get cores for different tests ***/
@@ -144,28 +148,33 @@ static void
 test_empty_dequeue(struct rte_ring *r, const int esize,
 			const unsigned int api_type)
 {
-	const unsigned int iter_shift = 26;
+	const unsigned int iter_shift = 29;
 	const unsigned int iterations = 1 << iter_shift;
 	unsigned int i = 0;
 	void *burst[MAX_BURST];

+	const unsigned int bulk_iterations = iterations / bulk_sizes[0];
 	const uint64_t start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	for (i = 0; i < bulk_iterations; i++)
 		test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type);
 	const uint64_t end = rte_rdtsc();

 	test_ring_print_test_string(api_type, esize, bulk_sizes[0],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 }

-/*
- * for the separate enqueue and dequeue threads they take in one param
- * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc
- */
-struct thread_params {
+/* describes the ring used by the enqueue and dequeue thread */
+struct ring_params {
 	struct rte_ring *r;
-	unsigned size;        /* input value, the burst size */
-	double spsc, mpmc;    /* output value, the single or multi timings */
+	unsigned int elem_size;
+	unsigned int bulk_sizes_i; /* index into bulk_size array */
+	unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */
+};
+
+/* Used do specify enqueue and dequeue ring operations and their results */
+struct thread_params {
+	struct ring_params *ring_params;
+	double *results; /* result array size must be equal to bulk_sizes */
 };

 /*
@@ -174,16 +183,15 @@ struct thread_params {
  * flag == 1 -> dequeue
  */
 static __rte_always_inline int
-enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
-	struct thread_params *p)
+enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 15;
+	const unsigned int iter_shift = 22;
 	const unsigned int iterations = 1 << iter_shift;
-	struct rte_ring *r = p->r;
-	unsigned int bsize = p->size;
 	unsigned int i;
 	void *burst = NULL;
+	unsigned int n_remaining;
+	const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i];

 #ifdef RTE_USE_C11_MEM_MODEL
 	if (__atomic_fetch_add(&lcore_count, 1, __ATOMIC_RELAXED) + 1 != 2)
@@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 		while(lcore_count != 2)
 			rte_pause();

-	burst = test_ring_calloc(MAX_BURST, esize);
+	burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size);
 	if (burst == NULL)
 		return -1;

 	const uint64_t sp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	const unsigned int bulk_iterations = iterations / bulk_n;
+	for (i = 0; i < bulk_iterations; i++) {
+		n_remaining = bulk_n;
 		do {
 			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_enqueue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_dequeue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			if (ret == 0)
 				rte_pause();
-		} while (!ret);
+			else
+				n_remaining -= ret;
+		} while (n_remaining > 0);
+	}
 	const uint64_t sp_end = rte_rdtsc();

-	const uint64_t mp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
-		do {
-			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			if (ret == 0)
-				rte_pause();
-		} while (!ret);
-	const uint64_t mp_end = rte_rdtsc();
+	p->results[p->ring_params->bulk_sizes_i] =
+			((double)sp_end - sp_start) / iterations;

-	p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize);
-	p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize);
 	return 0;
 }

@@ -243,15 +245,7 @@ enqueue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(0, -1, params);
-}
-
-static int
-enqueue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(0, 16, params);
+	return enqueue_dequeue_bulk_helper(0, params);
 }

 /*
@@ -263,15 +257,7 @@ dequeue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(1, -1, params);
-}
-
-static int
-dequeue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(1, 16, params);
+	return enqueue_dequeue_bulk_helper(1, params);
 }

 /*
@@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p)
  * used to measure ring perf between hyperthreads, cores and sockets.
  */
 static int
-run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize)
+run_on_core_pair(struct lcore_pair *cores,
+		struct thread_params *param1, struct thread_params *param2)
 {
-	lcore_function_t *f1, *f2;
-	struct thread_params param1 = {0}, param2 = {0};
 	unsigned i;
-
-	if (esize == -1) {
-		f1 = enqueue_bulk;
-		f2 = dequeue_bulk;
-	} else {
-		f1 = enqueue_bulk_16B;
-		f2 = dequeue_bulk_16B;
-	}
+	struct ring_params *ring_params = param1->ring_params;

 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		lcore_count = 0;
-		param1.size = param2.size = bulk_sizes[i];
-		param1.r = param2.r = r;
+		ring_params->bulk_sizes_i = i;
 		if (cores->c1 == rte_get_main_lcore()) {
-			rte_eal_remote_launch(f2, &param2, cores->c2);
-			f1(&param1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
+			enqueue_bulk(param1);
 			rte_eal_wait_lcore(cores->c2);
 		} else {
-			rte_eal_remote_launch(f1, &param1, cores->c1);
-			rte_eal_remote_launch(f2, &param2, cores->c2);
+			rte_eal_remote_launch(enqueue_bulk, param1, cores->c1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
 			if (rte_eal_wait_lcore(cores->c1) < 0)
 				return -1;
 			if (rte_eal_wait_lcore(cores->c2) < 0)
 				return -1;
 		}
 		test_ring_print_test_string(
-			TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.spsc + param2.spsc);
-		test_ring_print_test_string(
-			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.mpmc + param2.mpmc);
+				ring_params->ring_flags,
+				ring_params->elem_size,
+				bulk_sizes[i],
+				param1->results[i] + param2->results[i]);
 	}

 	return 0;
@@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 	uint64_t hz = rte_get_timer_hz();
 	uint64_t lcount = 0;
 	const unsigned int lcore = rte_lcore_id();
-	struct thread_params *params = p;
+	struct ring_params *ring_params = p->ring_params;
 	void *burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -346,9 +322,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize)

 	begin = rte_get_timer_cycles();
 	while (time_diff < hz * TIME_MS / 1000) {
-		test_ring_enqueue(params->r, burst, esize, params->size,
+		test_ring_enqueue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
-		test_ring_dequeue(params->r, burst, esize, params->size,
+		test_ring_dequeue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 		lcount++;
 		time_diff = rte_get_timer_cycles() - begin;
@@ -380,7 +358,8 @@ static int
 run_on_all_cores(struct rte_ring *r, const int esize)
 {
 	uint64_t total;
-	struct thread_params param;
+	struct ring_params ring_params = {0};
+	struct thread_params params = { .ring_params = &ring_params };
 	lcore_function_t *lcore_f;
 	unsigned int i, c;

@@ -389,21 +368,20 @@ run_on_all_cores(struct rte_ring *r, const int esize)
 	else
 		lcore_f = load_loop_fn_16B;

-	memset(&param, 0, sizeof(struct thread_params));
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		total = 0;
 		printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]);
-		param.size = bulk_sizes[i];
-		param.r = r;
+		params.ring_params->bulk_sizes_i = i;
+		params.ring_params->r = r;

 		/* clear synchro and start workers */
 		__atomic_store_n(&synchro, 0, __ATOMIC_RELAXED);
-		if (rte_eal_mp_remote_launch(lcore_f, &param, SKIP_MAIN) < 0)
+		if (rte_eal_mp_remote_launch(lcore_f, &params, SKIP_MAIN) < 0)
 			return -1;

 		/* start synchro and launch test on main */
 		__atomic_store_n(&synchro, 1, __ATOMIC_RELAXED);
-		lcore_f(&param);
+		lcore_f(&params);

 		rte_eal_mp_wait_lcore();

@@ -462,9 +440,9 @@ static int
 test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	const unsigned int api_type)
 {
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 26;
 	const unsigned int iterations = 1 << iter_shift;
-	unsigned int sz, i = 0;
+	unsigned int sz, i;
 	void **burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -472,17 +450,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 		return -1;

 	for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) {
+		const unsigned int n = iterations / bulk_sizes[sz];
 		const uint64_t start = rte_rdtsc();
-		for (i = 0; i < iterations; i++) {
+		for (i = 0; i < n; i++) {
 			test_ring_enqueue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 			test_ring_dequeue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 		}
 		const uint64_t end = rte_rdtsc();

 		test_ring_print_test_string(api_type, esize, bulk_sizes[sz],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 	}

 	rte_free(burst);
@@ -490,12 +469,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	return 0;
 }

+static __rte_always_inline int
+test_ring_perf_esize_run_on_two_cores(
+		struct thread_params *param1, struct thread_params *param2)
+{
+	struct lcore_pair cores;
+
+	if (get_two_hyperthreads(&cores) == 0) {
+		printf("\n### Testing using two hyperthreads ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing using two physical cores ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_sockets(&cores) == 0) {
+		printf("\n### Testing using two NUMA nodes ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	return 0;
+}
+
 /* Run all tests for a given element size */
 static __rte_always_inline int
 test_ring_perf_esize(const int esize)
 {
-	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
+	double results_enq[RTE_DIM(bulk_sizes)];
+	double results_deq[RTE_DIM(bulk_sizes)];
+	struct ring_params ring_params = {
+			.elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results_enq };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results_deq };

 	/*
 	 * Performance test for legacy/_elem APIs
@@ -535,22 +545,13 @@ test_ring_perf_esize(const int esize)
 	test_empty_dequeue(r, esize,
 			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);

-	if (get_two_hyperthreads(&cores) == 0) {
-		printf("\n### Testing using two hyperthreads ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.r = r;

-	if (get_two_cores(&cores) == 0) {
-		printf("\n### Testing using two physical cores ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
-	if (get_two_sockets(&cores) == 0) {
-		printf("\n### Testing using two NUMA nodes ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
+
+	ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);

 	printf("\n### Testing using all worker nodes ###\n");
 	if (run_on_all_cores(r, esize) < 0)
@@ -566,6 +567,109 @@ test_ring_perf_esize(const int esize)
 	return -1;
 }

+
+static __rte_always_inline int
+test_ring_perf_compression(void)
+{
+	double results1[RTE_DIM(bulk_sizes)];
+	double results2[RTE_DIM(bulk_sizes)];
+	double results1_comp[2][RTE_DIM(bulk_sizes)];
+	double results2_comp[2][RTE_DIM(bulk_sizes)];
+
+	struct lcore_pair cores;
+	int ret = -1;
+	unsigned int i, j;
+	struct ring_params ring_params = { .elem_size = sizeof(void *) };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results1 };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results2 };
+
+	printf("\n### Testing compression gain ###");
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(void *),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy ###\n");
+		ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	/* rings allow only multiples of 4 as sizes,
+	 * we allocate size 4 despite only using 2 bytes
+	 * and use half of RING_SIZE as the number of elements
+	 */
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE / 2, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[0];
+	param2.results = results2_comp[0];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (16b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[1];
+	param2.results = results2_comp[1];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (32b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	for (j = 0; j < 2; j++) {
+		printf("\n### Potential gain from compression (%d-bit offsets) "
+		"###\n", (j + 1) * 16);
+		for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
+			const double result = results1[i] + results2[i];
+			const double result_comp = results1_comp[j][i] +
+				results2_comp[j][i];
+			const double gain = 100 - (result_comp / result) * 100;
+
+			printf("Gain of %5.1F%% for burst of %-3u elems\n",
+					gain, bulk_sizes[i]);
+		}
+	}
+
+	return ret;
+}
+
 static int
 test_ring_perf(void)
 {
@@ -576,6 +680,10 @@ test_ring_perf(void)
 	if (test_ring_perf_esize(16) == -1)
 		return -1;

+	/* Test for performance gain of compression */
+	if (test_ring_perf_compression() == -1)
+		return -1;
+
 	return 0;
 }

--
2.25.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v3 3/3] docs: add pointer compression to the EAL guide
  2023-10-31 18:10   ` [PATCH v3 0/3] add pointer compression API Paul Szczepanek
  2023-10-31 18:10     ` [PATCH v3 1/3] eal: add pointer compression functions Paul Szczepanek
  2023-10-31 18:10     ` [PATCH v3 2/3] test: add pointer compress tests to ring perf test Paul Szczepanek
@ 2023-10-31 18:10     ` Paul Szczepanek
  2023-11-01  7:42     ` [PATCH v3 0/3] add pointer compression API Morten Brørup
  3 siblings, 0 replies; 29+ messages in thread
From: Paul Szczepanek @ 2023-10-31 18:10 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Honnappa Nagarahalli

Documentation added in the EAL guide for the new
utility functions for pointer compression
showing example code and potential usecases

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .../prog_guide/env_abstraction_layer.rst      | 142 ++++++++++++++++++
 1 file changed, 142 insertions(+)

diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 89014789de..88cf1f16e2 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -1192,3 +1192,145 @@ will not be deallocated.

 Any successful deallocation event will trigger a callback, for which user
 applications and other DPDK subsystems can register.
+
+.. _pointer_compression:
+
+Pointer Compression
+-------------------
+
+Use ``rte_ptr_compress_16()`` and ``rte_ptr_decompress_16()`` to compress and
+decompress pointers into 16-bit offsets. Use ``rte_ptr_compress_32()`` and
+``rte_ptr_decompress_32()`` to compress and decompress pointers into 32-bit
+offsets.
+
+Compression takes advantage of the fact that pointers are usually located in a
+limited memory region (like a mempool). By converting them to offsets from a
+base memory address they can be stored in fewer bytes. How many bytes are needed
+to store the offset is dictated by the memory region size and alignment of
+objects the pointers point to.
+
+For example, a pointer which is part of a 4GB memory pool can be stored as 32
+bit offset. If the pointer points to memory that is 8 bytes aligned then 3 bits
+can be dropped from the offset and a 32GB memory pool can now fit in 32 bits.
+
+For performance reasons these requirements are not enforced programmatically.
+The programmer is responsible for ensuring that the combination of distance
+from the base pointer and memory alignment allow for storing of the offset in
+the number of bits indicated by the function name (16 or 32). Start of mempool
+memory would be a good candidate for the base pointer. Otherwise any pointer
+that precedes all pointers, is close enough and has the same alignment as the
+pointers being compressed will work.
+
+.. note::
+
+    Performance gains depend on the batch size of pointers and CPU capabilities
+    such as vector extensions. It's important to measure the performance
+    increase on target hardware. A test called ``ring_perf_autotest`` in
+    ``dpdk-test`` can provide the measurements.
+
+Example usage
+~~~~~~~~~~~~~
+
+In this example we send pointers between two cores through a ring. While this
+is a realistic use case the code is simplified for demonstration purposes and
+does not have error handling.
+
+.. code-block:: c
+
+    #include <rte_launch.h>
+    #include <rte_ring.h>
+    #include <rte_ring_elem.h>
+    #include <rte_ptr_compress.h>
+
+    #define ITEMS_ARRAY_SIZE (1024)
+    #define BATCH_SIZE (128)
+    #define ALIGN_EXPONENT (3)
+    #define ITEM_ALIGN (1<<ALIGN_EXPONENT)
+    #define CORE_SEND (1)
+    #define CORE_RECV (2)
+
+    struct item {
+      int a;
+    } __rte_aligned(ITEM_ALIGN);
+
+    static struct item items[ITEMS_ARRAY_SIZE] = {0};
+    static struct rte_ring *ring = NULL;
+
+    static int
+    send_compressed(void *args)
+    {
+      struct item *ptrs_send[BATCH_SIZE] = {0};
+      unsigned int n_send = 0;
+      struct rte_ring_zc_data zcd = {0};
+
+      /* in this toy example we only fill the ptrs_send once and re-use */
+      for (;n_send < BATCH_SIZE; n_send++)
+        ptrs_send[n_send] = &items[n_send];
+
+      for(;;) {
+        n_send = rte_ring_enqueue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        /* compress ptrs_send into offsets */
+        rte_ptr_compress_32(items, /* base pointer */
+          ptrs_send, /* source array to be compressed */
+          zcd.ptr1, /* destination array to store offsets */
+          zcd.n1, /* how many pointers to compress */
+          ALIGN_EXPONENT /* how many bits can we drop from the offset */);
+
+        if (zcd.ptr2 != NULL)
+          rte_ptr_compress_32(items, ptrs_send + zcd.n1,
+            zcd.ptr2, n_send - zcd.n1, ALIGN_EXPONENT);
+
+        rte_ring_enqueue_zc_finish(ring, n_send);
+      }
+      return 1;
+    }
+
+    static int
+    recv_compressed(void *args)
+    {
+      struct item *ptrs_recv[BATCH_SIZE] = {0};
+      unsigned int n_recv;
+      struct rte_ring_zc_data zcd = {0};
+
+      for(;;) {
+        /* receive compressed pointers from the ring */
+        n_recv = rte_ring_dequeue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        rte_ptr_decompress_32(items, /* base pointer */
+          zcd.ptr1, /* source array to decompress */
+          ptrs_recv, /* destination array to store pointers */
+          zcd.n1, /* how many pointers to decompress */
+          ALIGN_EXPONENT /* how many bits were dropped from the offset */);
+
+        /* handle the potential secondary buffer (caused by ring boundary) */
+        if (zcd.ptr2 != NULL)
+          rte_ptr_decompress_32(items,
+            zcd.ptr2,
+            ptrs_recv + zcd.n1,
+            n_recv - zcd.n1,
+            ALIGN_EXPONENT);
+
+        rte_ring_dequeue_zc_finish(ring, n_recv);
+
+        /* ptrs_recv contains what ptrs_send contained in the other thread */
+        /* (...) */
+      }
+      return 1;
+    }
+
+    void
+    compression_example(void)
+    {
+      ring = rte_ring_create_elem(
+        "COMPR_PTRS", sizeof(uint32_t),
+        1024, rte_socket_id(),
+        RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+      rte_eal_remote_launch(send_compressed, NULL, CORE_SEND);
+      rte_eal_remote_launch(recv_compressed, NULL, CORE_RECV);
+
+      for(;;) {}
+    }
--
2.25.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* RE: [PATCH v3 0/3] add pointer compression API
  2023-10-31 18:10   ` [PATCH v3 0/3] add pointer compression API Paul Szczepanek
                       ` (2 preceding siblings ...)
  2023-10-31 18:10     ` [PATCH v3 3/3] docs: add pointer compression to the EAL guide Paul Szczepanek
@ 2023-11-01  7:42     ` Morten Brørup
  2023-11-01 12:52       ` Paul Szczepanek
  3 siblings, 1 reply; 29+ messages in thread
From: Morten Brørup @ 2023-11-01  7:42 UTC (permalink / raw)
  To: Paul Szczepanek, dev

> From: Paul Szczepanek [mailto:paul.szczepanek@arm.com]
> Sent: Tuesday, 31 October 2023 19.11

[...]

> Test is added that shows potential performance gain from compression.
> In
> this test an array of pointers is passed through a ring between two
> cores.
> It shows the gain which is dependent on the bulk operation size. In
> this
> synthetic test run on ampere altra a substantial (up to 25%)
> performance
> gain is seen if done in bulk size larger than 32. At 32 it breaks even
> and
> lower sizes create a small (less than 5%) slowdown due to overhead.
> 
> In a more realistic mock application running the l3 forwarding dpdk
> example that works in pipeline mode this translated into a ~5%
> throughput
> increase on an ampere altra.

What was the bulk size in this test?

And were the pipeline stages running on the same lcore or individual lcores per pipeline stage?


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v4 0/4] add pointer compression API
  2023-09-27 15:08 ` [RFC 1/2] eal: add pointer compression functions Paul Szczepanek
                     ` (2 preceding siblings ...)
  2023-10-31 18:10   ` [PATCH v3 0/3] add pointer compression API Paul Szczepanek
@ 2023-11-01 12:46   ` Paul Szczepanek
  2023-11-01 12:46     ` [PATCH v4 1/4] eal: add pointer compression functions Paul Szczepanek
                       ` (3 more replies)
  2023-11-01 18:12   ` [PATCH v5 0/4] add pointer compression API Paul Szczepanek
  4 siblings, 4 replies; 29+ messages in thread
From: Paul Szczepanek @ 2023-11-01 12:46 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek

This patchset is proposing adding a new EAL header with utility functions
that allow compression of arrays of pointers.

When passing caches full of pointers between threads, memory containing
the pointers is copied multiple times which is especially costly between
cores. A compression method will allow us to shrink the memory size
copied.

The compression takes advantage of the fact that pointers are usually
located in a limited memory region (like a mempool). We can compress them
by converting them to offsets from a base memory address.

Offsets can be stored in fewer bytes (dictated by the memory region size
and alignment of the pointer). For example: an 8 byte aligned pointer
which is part of a 32GB memory pool can be stored in 4 bytes. The API is
very generic and does not assume mempool pointers, any pointer can be
passed in.

Compression is based on few and fast operations and especially with vector
instructions leveraged creates minimal overhead.

The API accepts and returns arrays because the overhead means it only is
worth it when done in bulk.

Test is added that shows potential performance gain from compression. In
this test an array of pointers is passed through a ring between two cores.
It shows the gain which is dependent on the bulk operation size. In this
synthetic test run on ampere altra a substantial (up to 25%) performance
gain is seen if done in bulk size larger than 32. At 32 it breaks even and
lower sizes create a small (less than 5%) slowdown due to overhead.

In a more realistic mock application running the l3 forwarding dpdk
example that works in pipeline mode on two cores this translated into a
~5% throughput increase on an ampere altra.

v2:
* addressed review comments (style, explanations and typos)
* lowered bulk iterations closer to original numbers to keep runtime short
* fixed pointer size warning on 32-bit arch
v3:
* added 16-bit versions of compression functions and tests
* added documentation of these new utility functions in the EAL guide
v4:
* added unit test
* fix bug in NEON implementation of 32-bit decompress

Paul Szczepanek (4):
  eal: add pointer compression functions
  test: add pointer compress tests to ring perf test
  docs: add pointer compression to the EAL guide
  test: add unit test for ptr compression

 .mailmap                                      |   1 +
 app/test/meson.build                          |   1 +
 app/test/test_eal_ptr_compress.c              | 108 ++++++
 app/test/test_ring.h                          |  94 ++++-
 app/test/test_ring_perf.c                     | 354 ++++++++++++------
 .../prog_guide/env_abstraction_layer.rst      | 142 +++++++
 lib/eal/include/meson.build                   |   1 +
 lib/eal/include/rte_ptr_compress.h            | 266 +++++++++++++
 8 files changed, 843 insertions(+), 124 deletions(-)
 create mode 100644 app/test/test_eal_ptr_compress.c
 create mode 100644 lib/eal/include/rte_ptr_compress.h

--
2.25.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v4 1/4] eal: add pointer compression functions
  2023-11-01 12:46   ` [PATCH v4 0/4] " Paul Szczepanek
@ 2023-11-01 12:46     ` Paul Szczepanek
  2023-11-01 12:46     ` [PATCH v4 2/4] test: add pointer compress tests to ring perf test Paul Szczepanek
                       ` (2 subsequent siblings)
  3 siblings, 0 replies; 29+ messages in thread
From: Paul Szczepanek @ 2023-11-01 12:46 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Honnappa Nagarahalli, Kamalakshitha Aligeri

Add a new utility header for compressing pointers. The provided
functions can store pointers in 32-bit offsets.

The compression takes advantage of the fact that pointers are
usually located in a limited memory region (like a mempool).
We can compress them by converting them to offsets from a base
memory address. Offsets can be stored in fewer bytes (dictated
by the memory region size and alignment of the pointer).
For example: an 8 byte aligned pointer which is part of a 32GB
memory pool can be stored in 4 bytes.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .mailmap                           |   1 +
 lib/eal/include/meson.build        |   1 +
 lib/eal/include/rte_ptr_compress.h | 266 +++++++++++++++++++++++++++++
 3 files changed, 268 insertions(+)
 create mode 100644 lib/eal/include/rte_ptr_compress.h

diff --git a/.mailmap b/.mailmap
index 864d33ee46..3f0c9d32f5 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1058,6 +1058,7 @@ Paul Greenwalt <paul.greenwalt@intel.com>
 Paulis Gributs <paulis.gributs@intel.com>
 Paul Luse <paul.e.luse@intel.com>
 Paul M Stillwell Jr <paul.m.stillwell.jr@intel.com>
+Paul Szczepanek <paul.szczepanek@arm.com>
 Pavan Kumar Linga <pavan.kumar.linga@intel.com>
 Pavan Nikhilesh <pbhagavatula@marvell.com> <pbhagavatula@caviumnetworks.com>
 Pavel Belous <pavel.belous@aquantia.com>
diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index a0463efac7..17d8373648 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -36,6 +36,7 @@ headers += files(
         'rte_pci_dev_features.h',
         'rte_per_lcore.h',
         'rte_pflock.h',
+	'rte_ptr_compress.h',
         'rte_random.h',
         'rte_reciprocal.h',
         'rte_seqcount.h',
diff --git a/lib/eal/include/rte_ptr_compress.h b/lib/eal/include/rte_ptr_compress.h
new file mode 100644
index 0000000000..6697385113
--- /dev/null
+++ b/lib/eal/include/rte_ptr_compress.h
@@ -0,0 +1,266 @@
+/* SPDX-License-Identifier: BSD-shift-Clause
+ * Copyright(c) 2023 Arm Limited
+ */
+
+#ifndef RTE_PTR_COMPRESS_H
+#define RTE_PTR_COMPRESS_H
+
+/**
+ * @file
+ * Pointer compression and decompression functions.
+ *
+ * When passing arrays full of pointers between threads, memory containing
+ * the pointers is copied multiple times which is especially costly between
+ * cores. These functions allow us to compress the pointers.
+ *
+ * Compression takes advantage of the fact that pointers are usually located in
+ * a limited memory region (like a mempool). We compress them by converting them
+ * to offsets from a base memory address. Offsets can be stored in fewer bytes.
+ *
+ * The compression functions come in two varieties: 32-bit and 16-bit.
+ *
+ * To determine how many bits are needed to compress the pointer calculate
+ * the biggest offset possible (highest value pointer - base pointer)
+ * and shift the value right according to alignment (shift by exponent of the
+ * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by
+ * 3, etc.). The resulting value must fit in either 32 or 16 bits.
+ *
+ * For usage example and further explanation please see "Pointer Compression" in
+ * doc/guides/prog_guide/env_abstraction_layer.rst
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_vect.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Compress pointers into 32-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 32 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 32GB of the base pointer. If there is no such alignment guarantee they
+ * must be within 4GB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_32(void *ptr_base, void **src_table,
+		uint32_t *dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1w(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	/* right shift is done by left shifting by negative int */
+	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_ptr_table = vld1q_u64((const uint64_t *)src_table + i);
+		v_ptr_table = vsubq_u64(v_ptr_table, v_ptr_base);
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		vst1_u32(dest_table + i, vqmovn_u64(v_ptr_table));
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT32_MAX);
+		dest_table[i] = (uint32_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 32-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_32(void *ptr_base, uint32_t *src_table,
+		void **dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1uw_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	int64x2_t v_shift = vdupq_n_s64(bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_ptr_table = vmovl_u32(vld1_u32(src_table + i));
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		v_ptr_table = vaddq_u64(v_ptr_table, v_ptr_base);
+		vst1q_u64((uint64_t *)dest_table + i, v_ptr_table);
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+/**
+ * Compress pointers into 16-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 16 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 256KB of the base pointer. If there is no such alignment guarantee
+ * they must be within 64KB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_16(void *ptr_base, void **src_table,
+		uint16_t *dest_table, unsigned int n, unsigned int bit_shift)
+{
+
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1h(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT16_MAX);
+		dest_table[i] = (uint16_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 16-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_16(void *ptr_base, uint16_t *src_table,
+		void **dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1uh_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_PTR_COMPRESS_H */
--
2.25.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v4 2/4] test: add pointer compress tests to ring perf test
  2023-11-01 12:46   ` [PATCH v4 0/4] " Paul Szczepanek
  2023-11-01 12:46     ` [PATCH v4 1/4] eal: add pointer compression functions Paul Szczepanek
@ 2023-11-01 12:46     ` Paul Szczepanek
  2023-11-01 12:46     ` [PATCH v4 3/4] docs: add pointer compression to the EAL guide Paul Szczepanek
  2023-11-01 12:46     ` [PATCH v4 4/4] test: add unit test for ptr compression Paul Szczepanek
  3 siblings, 0 replies; 29+ messages in thread
From: Paul Szczepanek @ 2023-11-01 12:46 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Honnappa Nagarahalli

Add a test that runs a zero copy burst enqueue and dequeue on a ring
of raw pointers and compressed pointers at different burst sizes to
showcase performance benefits of newly added pointer compression APIs.

Refactored threading code to pass more parameters to threads to
reuse existing code. Added more bulk sizes to showcase their effects
on compression. Adjusted loop iteration numbers to take into account
bulk sizes to keep runtime constant (instead of number of operations).

Adjusted old printfs to match new ones which have aligned numbers.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 app/test/test_ring.h      |  94 +++++++++-
 app/test/test_ring_perf.c | 354 +++++++++++++++++++++++++-------------
 2 files changed, 324 insertions(+), 124 deletions(-)

diff --git a/app/test/test_ring.h b/app/test/test_ring.h
index 45c263f3ff..3b00f2465d 100644
--- a/app/test/test_ring.h
+++ b/app/test/test_ring.h
@@ -1,10 +1,12 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */

 #include <rte_malloc.h>
 #include <rte_ring.h>
 #include <rte_ring_elem.h>
+#include <rte_memcpy.h>
+#include <rte_ptr_compress.h>

 /* API type to call
  * rte_ring_<sp/mp or sc/mc>_enqueue_<bulk/burst>
@@ -25,6 +27,10 @@
 #define TEST_RING_ELEM_BULK 16
 #define TEST_RING_ELEM_BURST 32

+#define TEST_RING_ELEM_BURST_ZC 64
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 128
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 256
+
 #define TEST_RING_IGNORE_API_TYPE ~0U

 /* This function is placed here as it is required for both
@@ -101,6 +107,9 @@ static inline unsigned int
 test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -152,6 +161,46 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n,
 								NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy(zcd.ptr2,
+						(char *)obj + zcd.n1 * esize,
+						(ret - zcd.n1) * esize);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_16(0, obj, zcd.ptr1, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_16(0,
+						obj + (zcd.n1 * 2),
+						zcd.ptr2,
+						(ret - zcd.n1) * 2, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_32(0, obj, zcd.ptr1, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_32(0, obj + zcd.n1,
+						zcd.ptr2, ret - zcd.n1, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
@@ -162,6 +211,9 @@ static inline unsigned int
 test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -213,6 +265,46 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mc_dequeue_burst_elem(r, obj, esize,
 								n, NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy((char *)obj + zcd.n1 * esize,
+						zcd.ptr2,
+						(ret - zcd.n1) * esize);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_16(0, zcd.ptr1, obj, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_16(0, zcd.ptr2,
+						obj + zcd.n1,
+						(ret - zcd.n1) * 2,
+						3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_32(0, zcd.ptr1, obj, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_32(0, zcd.ptr2,
+						obj + zcd.n1, ret - zcd.n1, 3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c
index d7c5a4c30b..8392d0c781 100644
--- a/app/test/test_ring_perf.c
+++ b/app/test/test_ring_perf.c
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2010-2014 Intel Corporation
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */


@@ -22,13 +22,13 @@

 #define RING_NAME "RING_PERF"
 #define RING_SIZE 4096
-#define MAX_BURST 32
+#define MAX_BURST 256

 /*
  * the sizes to enqueue and dequeue in testing
  * (marked volatile so they won't be seen as compile-time constants)
  */
-static const volatile unsigned bulk_sizes[] = { 8, 32 };
+static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128, 256 };

 struct lcore_pair {
 	unsigned c1, c2;
@@ -43,26 +43,30 @@ test_ring_print_test_string(unsigned int api_type, int esize,
 	if (esize == -1)
 		printf("legacy APIs");
 	else
-		printf("elem APIs: element size %dB", esize);
+		printf("elem APIs (size:%2dB)", esize);

 	if (api_type == TEST_RING_IGNORE_API_TYPE)
 		return;

 	if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF)
-		printf(": default enqueue/dequeue: ");
+		printf(" - default enqueue/dequeue");
 	else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC)
-		printf(": SP/SC: ");
+		printf(" - SP/SC");
 	else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC)
-		printf(": MP/MC: ");
+		printf(" - MP/MC");

 	if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE)
-		printf("single: ");
+		printf(" - single - ");
 	else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK)
-		printf("bulk (size: %u): ", bsz);
+		printf(" - bulk (n:%-3u) - ", bsz);
 	else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST)
-		printf("burst (size: %u): ", bsz);
+		printf(" - burst (n:%-3u) - ", bsz);
+	else if ((api_type & (TEST_RING_ELEM_BURST_ZC |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0)
+		printf(" - burst zero copy (n:%-3u) - ", bsz);

-	printf("%.2F\n", value);
+	printf("cycles per elem: %.3F\n", value);
 }

 /**** Functions to analyse our core mask to get cores for different tests ***/
@@ -144,28 +148,33 @@ static void
 test_empty_dequeue(struct rte_ring *r, const int esize,
 			const unsigned int api_type)
 {
-	const unsigned int iter_shift = 26;
+	const unsigned int iter_shift = 29;
 	const unsigned int iterations = 1 << iter_shift;
 	unsigned int i = 0;
 	void *burst[MAX_BURST];

+	const unsigned int bulk_iterations = iterations / bulk_sizes[0];
 	const uint64_t start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	for (i = 0; i < bulk_iterations; i++)
 		test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type);
 	const uint64_t end = rte_rdtsc();

 	test_ring_print_test_string(api_type, esize, bulk_sizes[0],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 }

-/*
- * for the separate enqueue and dequeue threads they take in one param
- * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc
- */
-struct thread_params {
+/* describes the ring used by the enqueue and dequeue thread */
+struct ring_params {
 	struct rte_ring *r;
-	unsigned size;        /* input value, the burst size */
-	double spsc, mpmc;    /* output value, the single or multi timings */
+	unsigned int elem_size;
+	unsigned int bulk_sizes_i; /* index into bulk_size array */
+	unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */
+};
+
+/* Used do specify enqueue and dequeue ring operations and their results */
+struct thread_params {
+	struct ring_params *ring_params;
+	double *results; /* result array size must be equal to bulk_sizes */
 };

 /*
@@ -174,16 +183,15 @@ struct thread_params {
  * flag == 1 -> dequeue
  */
 static __rte_always_inline int
-enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
-	struct thread_params *p)
+enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 15;
+	const unsigned int iter_shift = 22;
 	const unsigned int iterations = 1 << iter_shift;
-	struct rte_ring *r = p->r;
-	unsigned int bsize = p->size;
 	unsigned int i;
 	void *burst = NULL;
+	unsigned int n_remaining;
+	const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i];

 #ifdef RTE_USE_C11_MEM_MODEL
 	if (__atomic_fetch_add(&lcore_count, 1, __ATOMIC_RELAXED) + 1 != 2)
@@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 		while(lcore_count != 2)
 			rte_pause();

-	burst = test_ring_calloc(MAX_BURST, esize);
+	burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size);
 	if (burst == NULL)
 		return -1;

 	const uint64_t sp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	const unsigned int bulk_iterations = iterations / bulk_n;
+	for (i = 0; i < bulk_iterations; i++) {
+		n_remaining = bulk_n;
 		do {
 			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_enqueue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_dequeue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			if (ret == 0)
 				rte_pause();
-		} while (!ret);
+			else
+				n_remaining -= ret;
+		} while (n_remaining > 0);
+	}
 	const uint64_t sp_end = rte_rdtsc();

-	const uint64_t mp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
-		do {
-			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			if (ret == 0)
-				rte_pause();
-		} while (!ret);
-	const uint64_t mp_end = rte_rdtsc();
+	p->results[p->ring_params->bulk_sizes_i] =
+			((double)sp_end - sp_start) / iterations;

-	p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize);
-	p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize);
 	return 0;
 }

@@ -243,15 +245,7 @@ enqueue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(0, -1, params);
-}
-
-static int
-enqueue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(0, 16, params);
+	return enqueue_dequeue_bulk_helper(0, params);
 }

 /*
@@ -263,15 +257,7 @@ dequeue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(1, -1, params);
-}
-
-static int
-dequeue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(1, 16, params);
+	return enqueue_dequeue_bulk_helper(1, params);
 }

 /*
@@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p)
  * used to measure ring perf between hyperthreads, cores and sockets.
  */
 static int
-run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize)
+run_on_core_pair(struct lcore_pair *cores,
+		struct thread_params *param1, struct thread_params *param2)
 {
-	lcore_function_t *f1, *f2;
-	struct thread_params param1 = {0}, param2 = {0};
 	unsigned i;
-
-	if (esize == -1) {
-		f1 = enqueue_bulk;
-		f2 = dequeue_bulk;
-	} else {
-		f1 = enqueue_bulk_16B;
-		f2 = dequeue_bulk_16B;
-	}
+	struct ring_params *ring_params = param1->ring_params;

 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		lcore_count = 0;
-		param1.size = param2.size = bulk_sizes[i];
-		param1.r = param2.r = r;
+		ring_params->bulk_sizes_i = i;
 		if (cores->c1 == rte_get_main_lcore()) {
-			rte_eal_remote_launch(f2, &param2, cores->c2);
-			f1(&param1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
+			enqueue_bulk(param1);
 			rte_eal_wait_lcore(cores->c2);
 		} else {
-			rte_eal_remote_launch(f1, &param1, cores->c1);
-			rte_eal_remote_launch(f2, &param2, cores->c2);
+			rte_eal_remote_launch(enqueue_bulk, param1, cores->c1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
 			if (rte_eal_wait_lcore(cores->c1) < 0)
 				return -1;
 			if (rte_eal_wait_lcore(cores->c2) < 0)
 				return -1;
 		}
 		test_ring_print_test_string(
-			TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.spsc + param2.spsc);
-		test_ring_print_test_string(
-			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.mpmc + param2.mpmc);
+				ring_params->ring_flags,
+				ring_params->elem_size,
+				bulk_sizes[i],
+				param1->results[i] + param2->results[i]);
 	}

 	return 0;
@@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 	uint64_t hz = rte_get_timer_hz();
 	uint64_t lcount = 0;
 	const unsigned int lcore = rte_lcore_id();
-	struct thread_params *params = p;
+	struct ring_params *ring_params = p->ring_params;
 	void *burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -346,9 +322,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize)

 	begin = rte_get_timer_cycles();
 	while (time_diff < hz * TIME_MS / 1000) {
-		test_ring_enqueue(params->r, burst, esize, params->size,
+		test_ring_enqueue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
-		test_ring_dequeue(params->r, burst, esize, params->size,
+		test_ring_dequeue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 		lcount++;
 		time_diff = rte_get_timer_cycles() - begin;
@@ -380,7 +358,8 @@ static int
 run_on_all_cores(struct rte_ring *r, const int esize)
 {
 	uint64_t total;
-	struct thread_params param;
+	struct ring_params ring_params = {0};
+	struct thread_params params = { .ring_params = &ring_params };
 	lcore_function_t *lcore_f;
 	unsigned int i, c;

@@ -389,21 +368,20 @@ run_on_all_cores(struct rte_ring *r, const int esize)
 	else
 		lcore_f = load_loop_fn_16B;

-	memset(&param, 0, sizeof(struct thread_params));
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		total = 0;
 		printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]);
-		param.size = bulk_sizes[i];
-		param.r = r;
+		params.ring_params->bulk_sizes_i = i;
+		params.ring_params->r = r;

 		/* clear synchro and start workers */
 		__atomic_store_n(&synchro, 0, __ATOMIC_RELAXED);
-		if (rte_eal_mp_remote_launch(lcore_f, &param, SKIP_MAIN) < 0)
+		if (rte_eal_mp_remote_launch(lcore_f, &params, SKIP_MAIN) < 0)
 			return -1;

 		/* start synchro and launch test on main */
 		__atomic_store_n(&synchro, 1, __ATOMIC_RELAXED);
-		lcore_f(&param);
+		lcore_f(&params);

 		rte_eal_mp_wait_lcore();

@@ -462,9 +440,9 @@ static int
 test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	const unsigned int api_type)
 {
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 26;
 	const unsigned int iterations = 1 << iter_shift;
-	unsigned int sz, i = 0;
+	unsigned int sz, i;
 	void **burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -472,17 +450,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 		return -1;

 	for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) {
+		const unsigned int n = iterations / bulk_sizes[sz];
 		const uint64_t start = rte_rdtsc();
-		for (i = 0; i < iterations; i++) {
+		for (i = 0; i < n; i++) {
 			test_ring_enqueue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 			test_ring_dequeue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 		}
 		const uint64_t end = rte_rdtsc();

 		test_ring_print_test_string(api_type, esize, bulk_sizes[sz],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 	}

 	rte_free(burst);
@@ -490,12 +469,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	return 0;
 }

+static __rte_always_inline int
+test_ring_perf_esize_run_on_two_cores(
+		struct thread_params *param1, struct thread_params *param2)
+{
+	struct lcore_pair cores;
+
+	if (get_two_hyperthreads(&cores) == 0) {
+		printf("\n### Testing using two hyperthreads ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing using two physical cores ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_sockets(&cores) == 0) {
+		printf("\n### Testing using two NUMA nodes ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	return 0;
+}
+
 /* Run all tests for a given element size */
 static __rte_always_inline int
 test_ring_perf_esize(const int esize)
 {
-	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
+	double results_enq[RTE_DIM(bulk_sizes)];
+	double results_deq[RTE_DIM(bulk_sizes)];
+	struct ring_params ring_params = {
+			.elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results_enq };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results_deq };

 	/*
 	 * Performance test for legacy/_elem APIs
@@ -535,22 +545,13 @@ test_ring_perf_esize(const int esize)
 	test_empty_dequeue(r, esize,
 			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);

-	if (get_two_hyperthreads(&cores) == 0) {
-		printf("\n### Testing using two hyperthreads ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.r = r;

-	if (get_two_cores(&cores) == 0) {
-		printf("\n### Testing using two physical cores ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
-	if (get_two_sockets(&cores) == 0) {
-		printf("\n### Testing using two NUMA nodes ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
+
+	ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);

 	printf("\n### Testing using all worker nodes ###\n");
 	if (run_on_all_cores(r, esize) < 0)
@@ -566,6 +567,109 @@ test_ring_perf_esize(const int esize)
 	return -1;
 }

+
+static __rte_always_inline int
+test_ring_perf_compression(void)
+{
+	double results1[RTE_DIM(bulk_sizes)];
+	double results2[RTE_DIM(bulk_sizes)];
+	double results1_comp[2][RTE_DIM(bulk_sizes)];
+	double results2_comp[2][RTE_DIM(bulk_sizes)];
+
+	struct lcore_pair cores;
+	int ret = -1;
+	unsigned int i, j;
+	struct ring_params ring_params = { .elem_size = sizeof(void *) };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results1 };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results2 };
+
+	printf("\n### Testing compression gain ###");
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(void *),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy ###\n");
+		ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	/* rings allow only multiples of 4 as sizes,
+	 * we allocate size 4 despite only using 2 bytes
+	 * and use half of RING_SIZE as the number of elements
+	 */
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE / 2, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[0];
+	param2.results = results2_comp[0];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (16b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[1];
+	param2.results = results2_comp[1];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (32b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	for (j = 0; j < 2; j++) {
+		printf("\n### Potential gain from compression (%d-bit offsets) "
+		"###\n", (j + 1) * 16);
+		for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
+			const double result = results1[i] + results2[i];
+			const double result_comp = results1_comp[j][i] +
+				results2_comp[j][i];
+			const double gain = 100 - (result_comp / result) * 100;
+
+			printf("Gain of %5.1F%% for burst of %-3u elems\n",
+					gain, bulk_sizes[i]);
+		}
+	}
+
+	return ret;
+}
+
 static int
 test_ring_perf(void)
 {
@@ -576,6 +680,10 @@ test_ring_perf(void)
 	if (test_ring_perf_esize(16) == -1)
 		return -1;

+	/* Test for performance gain of compression */
+	if (test_ring_perf_compression() == -1)
+		return -1;
+
 	return 0;
 }

--
2.25.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v4 3/4] docs: add pointer compression to the EAL guide
  2023-11-01 12:46   ` [PATCH v4 0/4] " Paul Szczepanek
  2023-11-01 12:46     ` [PATCH v4 1/4] eal: add pointer compression functions Paul Szczepanek
  2023-11-01 12:46     ` [PATCH v4 2/4] test: add pointer compress tests to ring perf test Paul Szczepanek
@ 2023-11-01 12:46     ` Paul Szczepanek
  2023-11-01 12:46     ` [PATCH v4 4/4] test: add unit test for ptr compression Paul Szczepanek
  3 siblings, 0 replies; 29+ messages in thread
From: Paul Szczepanek @ 2023-11-01 12:46 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Honnappa Nagarahalli

Documentation added in the EAL guide for the new
utility functions for pointer compression
showing example code and potential usecases

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .../prog_guide/env_abstraction_layer.rst      | 142 ++++++++++++++++++
 1 file changed, 142 insertions(+)

diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 89014789de..cc56784e3d 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -1192,3 +1192,145 @@ will not be deallocated.

 Any successful deallocation event will trigger a callback, for which user
 applications and other DPDK subsystems can register.
+
+.. _pointer_compression:
+
+Pointer Compression
+-------------------
+
+Use ``rte_ptr_compress_16()`` and ``rte_ptr_decompress_16()`` to compress and
+decompress pointers into 16-bit offsets. Use ``rte_ptr_compress_32()`` and
+``rte_ptr_decompress_32()`` to compress and decompress pointers into 32-bit
+offsets.
+
+Compression takes advantage of the fact that pointers are usually located in a
+limited memory region (like a mempool). By converting them to offsets from a
+base memory address they can be stored in fewer bytes. How many bytes are needed
+to store the offset is dictated by the memory region size and alignment of
+objects the pointers point to.
+
+For example, a pointer which is part of a 4GB memory pool can be stored as 32
+bit offset. If the pointer points to memory that is 8 bytes aligned then 3 bits
+can be dropped from the offset and a 32GB memory pool can now fit in 32 bits.
+
+For performance reasons these requirements are not enforced programmatically.
+The programmer is responsible for ensuring that the combination of distance
+from the base pointer and memory alignment allow for storing of the offset in
+the number of bits indicated by the function name (16 or 32). Start of mempool
+memory would be a good candidate for the base pointer. Otherwise any pointer
+that precedes all pointers, is close enough and has the same alignment as the
+pointers being compressed will work.
+
+.. note::
+
+    Performance gains depend on the batch size of pointers and CPU capabilities
+    such as vector extensions. It's important to measure the performance
+    increase on target hardware. A test called ``ring_perf_autotest`` in
+    ``dpdk-test`` can provide the measurements.
+
+Example usage
+~~~~~~~~~~~~~
+
+In this example we send pointers between two cores through a ring. While this
+is a realistic use case the code is simplified for demonstration purposes and
+does not have error handling.
+
+.. code-block:: c
+
+    #include <rte_launch.h>
+    #include <rte_ring.h>
+    #include <rte_ring_elem.h>
+    #include <rte_ptr_compress.h>
+
+    #define ITEMS_ARRAY_SIZE (1024)
+    #define BATCH_SIZE (128)
+    #define ALIGN_EXPONENT (3)
+    #define ITEM_ALIGN (1<<ALIGN_EXPONENT)
+    #define CORE_SEND (1)
+    #define CORE_RECV (2)
+
+    struct item {
+      int a;
+    } __rte_aligned(ITEM_ALIGN);
+
+    static struct item items[ITEMS_ARRAY_SIZE] = {0};
+    static struct rte_ring *ring = NULL;
+
+    static int
+    send_compressed(void *args)
+    {
+      struct item *ptrs_send[BATCH_SIZE] = {0};
+      unsigned int n_send = 0;
+      struct rte_ring_zc_data zcd = {0};
+
+      /* in this example we only fill the ptrs_send once and reuse */
+      for (;n_send < BATCH_SIZE; n_send++)
+        ptrs_send[n_send] = &items[n_send];
+
+      for(;;) {
+        n_send = rte_ring_enqueue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        /* compress ptrs_send into offsets */
+        rte_ptr_compress_32(items, /* base pointer */
+          ptrs_send, /* source array to be compressed */
+          zcd.ptr1, /* destination array to store offsets */
+          zcd.n1, /* how many pointers to compress */
+          ALIGN_EXPONENT /* how many bits can we drop from the offset */);
+
+        if (zcd.ptr2 != NULL)
+          rte_ptr_compress_32(items, ptrs_send + zcd.n1,
+            zcd.ptr2, n_send - zcd.n1, ALIGN_EXPONENT);
+
+        rte_ring_enqueue_zc_finish(ring, n_send);
+      }
+      return 1;
+    }
+
+    static int
+    recv_compressed(void *args)
+    {
+      struct item *ptrs_recv[BATCH_SIZE] = {0};
+      unsigned int n_recv;
+      struct rte_ring_zc_data zcd = {0};
+
+      for(;;) {
+        /* receive compressed pointers from the ring */
+        n_recv = rte_ring_dequeue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        rte_ptr_decompress_32(items, /* base pointer */
+          zcd.ptr1, /* source array to decompress */
+          ptrs_recv, /* destination array to store pointers */
+          zcd.n1, /* how many pointers to decompress */
+          ALIGN_EXPONENT /* how many bits were dropped from the offset */);
+
+        /* handle the potential secondary buffer (caused by ring boundary) */
+        if (zcd.ptr2 != NULL)
+          rte_ptr_decompress_32(items,
+            zcd.ptr2,
+            ptrs_recv + zcd.n1,
+            n_recv - zcd.n1,
+            ALIGN_EXPONENT);
+
+        rte_ring_dequeue_zc_finish(ring, n_recv);
+
+        /* ptrs_recv contains what ptrs_send contained in the other thread */
+        /* (...) */
+      }
+      return 1;
+    }
+
+    void
+    compression_example(void)
+    {
+      ring = rte_ring_create_elem(
+        "COMPR_PTRS", sizeof(uint32_t),
+        1024, rte_socket_id(),
+        RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+      rte_eal_remote_launch(send_compressed, NULL, CORE_SEND);
+      rte_eal_remote_launch(recv_compressed, NULL, CORE_RECV);
+
+      for(;;) {}
+    }
--
2.25.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v4 4/4] test: add unit test for ptr compression
  2023-11-01 12:46   ` [PATCH v4 0/4] " Paul Szczepanek
                       ` (2 preceding siblings ...)
  2023-11-01 12:46     ` [PATCH v4 3/4] docs: add pointer compression to the EAL guide Paul Szczepanek
@ 2023-11-01 12:46     ` Paul Szczepanek
  3 siblings, 0 replies; 29+ messages in thread
From: Paul Szczepanek @ 2023-11-01 12:46 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek

Test compresses and decompresses pointers with various combinations
of memory regions and alignments and verifies the pointers are
recovered correctly.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 app/test/meson.build             |   1 +
 app/test/test_eal_ptr_compress.c | 108 +++++++++++++++++++++++++++++++
 2 files changed, 109 insertions(+)
 create mode 100644 app/test/test_eal_ptr_compress.c

diff --git a/app/test/meson.build b/app/test/meson.build
index 05bae9216d..753de4bbd3 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -61,6 +61,7 @@ source_file_deps = {
     'test_dmadev_api.c': ['dmadev'],
     'test_eal_flags.c': [],
     'test_eal_fs.c': [],
+    'test_eal_ptr_compress.c': [],
     'test_efd.c': ['efd', 'net'],
     'test_efd_perf.c': ['efd', 'hash'],
     'test_errno.c': [],
diff --git a/app/test/test_eal_ptr_compress.c b/app/test/test_eal_ptr_compress.c
new file mode 100644
index 0000000000..c1c9a98be7
--- /dev/null
+++ b/app/test/test_eal_ptr_compress.c
@@ -0,0 +1,108 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2014 Intel Corporation
+ */
+
+#include "test.h"
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_ptr_compress.h>
+
+#define MAX_ALIGN_EXPONENT 3
+#define PTRS_SIZE 16
+#define NUM_BASES 2
+#define NUM_REGIONS 4
+#define MAX_32BIT_REGION ((uint64_t)UINT32_MAX + 1)
+#define MAX_16BIT_REGION (UINT16_MAX + 1)
+
+static int
+test_eal_ptr_compress_params(
+	void *base,
+	uint64_t mem_sz,
+	unsigned int align_exp,
+	unsigned int num_ptrs,
+	bool use_32_bit)
+{
+	unsigned int i;
+	unsigned int align = 1 << align_exp;
+	void *ptrs[PTRS_SIZE] = {0};
+	void *ptrs_out[PTRS_SIZE] = {0};
+	uint32_t offsets32[PTRS_SIZE] = {0};
+	uint16_t offsets16[PTRS_SIZE] = {0};
+
+	for (i = 0; i < num_ptrs; i++) {
+		/* make pointers point at memory in steps of align */
+		/* alternate steps from the start and end of memory region */
+		if ((i & 1) == 1)
+			ptrs[i] = (char *)base + mem_sz - i * align;
+		else
+			ptrs[i] = (char *)base + i * align;
+	}
+
+	if (use_32_bit) {
+		rte_ptr_compress_32(base, ptrs, offsets32, num_ptrs, align_exp);
+		rte_ptr_decompress_32(base, offsets32, ptrs_out, num_ptrs,
+				align_exp);
+	} else {
+		rte_ptr_compress_16(base, ptrs, offsets16, num_ptrs, align_exp);
+		rte_ptr_decompress_16(base, offsets16, ptrs_out, num_ptrs,
+				align_exp);
+	}
+
+	TEST_ASSERT_BUFFERS_ARE_EQUAL(ptrs, ptrs_out, sizeof(void *) * num_ptrs,
+		"Decompressed pointers corrupted\nbase pointer: %p, "
+		"memory region size: %" PRIu64 ", alignment exponent: %u, "
+		"num of pointers: %u, using %s offsets",
+		base, mem_sz, align_exp, num_ptrs,
+		use_32_bit ? "32-bit" : "16-bit");
+
+	return 0;
+}
+
+static int
+test_eal_ptr_compress(void)
+{
+	unsigned int j, k, n;
+	int ret = 0;
+	void * const bases[NUM_BASES] = { (void *)0, (void *)UINT16_MAX };
+	/* maximum size for pointers aligned by consecutive powers of 2 */
+	const uint64_t region_sizes_16[NUM_REGIONS] = {
+		MAX_16BIT_REGION,
+		MAX_16BIT_REGION * 2,
+		MAX_16BIT_REGION * 4,
+		MAX_16BIT_REGION * 8,
+	};
+	const uint64_t region_sizes_32[NUM_REGIONS] = {
+		MAX_32BIT_REGION,
+		MAX_32BIT_REGION * 2,
+		MAX_32BIT_REGION * 4,
+		MAX_32BIT_REGION * 8,
+	};
+
+	for (j = 0; j < NUM_REGIONS; j++) {
+		for (k = 0; k < NUM_BASES; k++) {
+			for (n = 1; n < PTRS_SIZE; n++) {
+				ret |= test_eal_ptr_compress_params(
+					bases[k],
+					region_sizes_16[j],
+					j /* exponent of alignment */,
+					n,
+					false
+				);
+				ret |= test_eal_ptr_compress_params(
+					bases[k],
+					region_sizes_32[j],
+					j /* exponent of alignment */,
+					n,
+					true
+				);
+				if (ret != 0)
+					return ret;
+			}
+		}
+	}
+
+	return ret;
+}
+
+REGISTER_FAST_TEST(eal_ptr_compress_autotest, true, true, test_eal_ptr_compress);
--
2.25.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v3 0/3] add pointer compression API
  2023-11-01  7:42     ` [PATCH v3 0/3] add pointer compression API Morten Brørup
@ 2023-11-01 12:52       ` Paul Szczepanek
  0 siblings, 0 replies; 29+ messages in thread
From: Paul Szczepanek @ 2023-11-01 12:52 UTC (permalink / raw)
  To: Morten Brørup, dev; +Cc: nd

On 01/11/2023 07:42, Morten Brørup wrote:
>> From: Paul Szczepanek [mailto:paul.szczepanek@arm.com]
>> Sent: Tuesday, 31 October 2023 19.11
> [...]
>
>> In a more realistic mock application running the l3 forwarding dpdk
>> example that works in pipeline mode this translated into a ~5%
>> throughput
>> increase on an ampere altra.
> What was the bulk size in this test?
>
> And were the pipeline stages running on the same lcore or individual lcores per pipeline stage?
>

The pipeline mode was run on separate cores and used 128 as the bulk size.


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v5 0/4] add pointer compression API
  2023-09-27 15:08 ` [RFC 1/2] eal: add pointer compression functions Paul Szczepanek
                     ` (3 preceding siblings ...)
  2023-11-01 12:46   ` [PATCH v4 0/4] " Paul Szczepanek
@ 2023-11-01 18:12   ` Paul Szczepanek
  2023-11-01 18:12     ` [PATCH v5 1/4] eal: add pointer compression functions Paul Szczepanek
                       ` (4 more replies)
  4 siblings, 5 replies; 29+ messages in thread
From: Paul Szczepanek @ 2023-11-01 18:12 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek

This patchset is proposing adding a new EAL header with utility functions
that allow compression of arrays of pointers.

When passing caches full of pointers between threads, memory containing
the pointers is copied multiple times which is especially costly between
cores. A compression method will allow us to shrink the memory size
copied.

The compression takes advantage of the fact that pointers are usually
located in a limited memory region (like a mempool). We can compress them
by converting them to offsets from a base memory address.

Offsets can be stored in fewer bytes (dictated by the memory region size
and alignment of the pointer). For example: an 8 byte aligned pointer
which is part of a 32GB memory pool can be stored in 4 bytes. The API is
very generic and does not assume mempool pointers, any pointer can be
passed in.

Compression is based on few and fast operations and especially with vector
instructions leveraged creates minimal overhead.

The API accepts and returns arrays because the overhead means it only is
worth it when done in bulk.

Test is added that shows potential performance gain from compression. In
this test an array of pointers is passed through a ring between two cores.
It shows the gain which is dependent on the bulk operation size. In this
synthetic test run on ampere altra a substantial (up to 25%) performance
gain is seen if done in bulk size larger than 32. At 32 it breaks even and
lower sizes create a small (less than 5%) slowdown due to overhead.

In a more realistic mock application running the l3 forwarding dpdk
example that works in pipeline mode on two cores this translated into a
~5% throughput increase on an ampere altra.

v2:
* addressed review comments (style, explanations and typos)
* lowered bulk iterations closer to original numbers to keep runtime short
* fixed pointer size warning on 32-bit arch
v3:
* added 16-bit versions of compression functions and tests
* added documentation of these new utility functions in the EAL guide
v4:
* added unit test
* fix bug in NEON implementation of 32-bit decompress
v5:
* disable NEON and SVE implementation on AARCH32 due to wrong pointer size

Paul Szczepanek (4):
  eal: add pointer compression functions
  test: add pointer compress tests to ring perf test
  docs: add pointer compression to the EAL guide
  test: add unit test for ptr compression

 .mailmap                                      |   1 +
 app/test/meson.build                          |   1 +
 app/test/test_eal_ptr_compress.c              | 108 ++++++
 app/test/test_ring.h                          |  94 ++++-
 app/test/test_ring_perf.c                     | 354 ++++++++++++------
 .../prog_guide/env_abstraction_layer.rst      | 142 +++++++
 lib/eal/include/meson.build                   |   1 +
 lib/eal/include/rte_ptr_compress.h            | 266 +++++++++++++
 8 files changed, 843 insertions(+), 124 deletions(-)
 create mode 100644 app/test/test_eal_ptr_compress.c
 create mode 100644 lib/eal/include/rte_ptr_compress.h

--
2.25.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v5 1/4] eal: add pointer compression functions
  2023-11-01 18:12   ` [PATCH v5 0/4] add pointer compression API Paul Szczepanek
@ 2023-11-01 18:12     ` Paul Szczepanek
  2024-02-11 15:32       ` Konstantin Ananyev
  2023-11-01 18:12     ` [PATCH v5 2/4] test: add pointer compress tests to ring perf test Paul Szczepanek
                       ` (3 subsequent siblings)
  4 siblings, 1 reply; 29+ messages in thread
From: Paul Szczepanek @ 2023-11-01 18:12 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Honnappa Nagarahalli, Kamalakshitha Aligeri

Add a new utility header for compressing pointers. The provided
functions can store pointers in 32-bit offsets.

The compression takes advantage of the fact that pointers are
usually located in a limited memory region (like a mempool).
We can compress them by converting them to offsets from a base
memory address. Offsets can be stored in fewer bytes (dictated
by the memory region size and alignment of the pointer).
For example: an 8 byte aligned pointer which is part of a 32GB
memory pool can be stored in 4 bytes.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .mailmap                           |   1 +
 lib/eal/include/meson.build        |   1 +
 lib/eal/include/rte_ptr_compress.h | 266 +++++++++++++++++++++++++++++
 3 files changed, 268 insertions(+)
 create mode 100644 lib/eal/include/rte_ptr_compress.h

diff --git a/.mailmap b/.mailmap
index 3f5bab26a8..004751d27a 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1069,6 +1069,7 @@ Paul Greenwalt <paul.greenwalt@intel.com>
 Paulis Gributs <paulis.gributs@intel.com>
 Paul Luse <paul.e.luse@intel.com>
 Paul M Stillwell Jr <paul.m.stillwell.jr@intel.com>
+Paul Szczepanek <paul.szczepanek@arm.com>
 Pavan Kumar Linga <pavan.kumar.linga@intel.com>
 Pavan Nikhilesh <pbhagavatula@marvell.com> <pbhagavatula@caviumnetworks.com>
 Pavel Belous <pavel.belous@aquantia.com>
diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index e94b056d46..ce2c733633 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -36,6 +36,7 @@ headers += files(
         'rte_pci_dev_features.h',
         'rte_per_lcore.h',
         'rte_pflock.h',
+	'rte_ptr_compress.h',
         'rte_random.h',
         'rte_reciprocal.h',
         'rte_seqcount.h',
diff --git a/lib/eal/include/rte_ptr_compress.h b/lib/eal/include/rte_ptr_compress.h
new file mode 100644
index 0000000000..47a72e4213
--- /dev/null
+++ b/lib/eal/include/rte_ptr_compress.h
@@ -0,0 +1,266 @@
+/* SPDX-License-Identifier: BSD-shift-Clause
+ * Copyright(c) 2023 Arm Limited
+ */
+
+#ifndef RTE_PTR_COMPRESS_H
+#define RTE_PTR_COMPRESS_H
+
+/**
+ * @file
+ * Pointer compression and decompression functions.
+ *
+ * When passing arrays full of pointers between threads, memory containing
+ * the pointers is copied multiple times which is especially costly between
+ * cores. These functions allow us to compress the pointers.
+ *
+ * Compression takes advantage of the fact that pointers are usually located in
+ * a limited memory region (like a mempool). We compress them by converting them
+ * to offsets from a base memory address. Offsets can be stored in fewer bytes.
+ *
+ * The compression functions come in two varieties: 32-bit and 16-bit.
+ *
+ * To determine how many bits are needed to compress the pointer calculate
+ * the biggest offset possible (highest value pointer - base pointer)
+ * and shift the value right according to alignment (shift by exponent of the
+ * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by
+ * 3, etc.). The resulting value must fit in either 32 or 16 bits.
+ *
+ * For usage example and further explanation please see "Pointer Compression" in
+ * doc/guides/prog_guide/env_abstraction_layer.rst
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_vect.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Compress pointers into 32-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 32 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 32GB of the base pointer. If there is no such alignment guarantee they
+ * must be within 4GB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_32(void *ptr_base, void **src_table,
+		uint32_t *dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1w(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	/* right shift is done by left shifting by negative int */
+	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_ptr_table = vld1q_u64((const uint64_t *)src_table + i);
+		v_ptr_table = vsubq_u64(v_ptr_table, v_ptr_base);
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		vst1_u32(dest_table + i, vqmovn_u64(v_ptr_table));
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT32_MAX);
+		dest_table[i] = (uint32_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 32-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_32(void *ptr_base, uint32_t *src_table,
+		void **dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1uw_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
+	uint64_t ptr_diff;
+	uint64x2_t v_ptr_table;
+	int64x2_t v_shift = vdupq_n_s64(bit_shift);
+	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
+	for (; i < (n & ~0x1); i += 2) {
+		v_ptr_table = vmovl_u32(vld1_u32(src_table + i));
+		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
+		v_ptr_table = vaddq_u64(v_ptr_table, v_ptr_base);
+		vst1q_u64((uint64_t *)dest_table + i, v_ptr_table);
+	}
+	/* process leftover single item in case of odd number of n */
+	if (unlikely(n & 0x1)) {
+		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+/**
+ * Compress pointers into 16-bit offsets from base pointer.
+ *
+ * @note It is programmer's responsibility to ensure the resulting offsets fit
+ * into 16 bits. Alignment of the structures pointed to by the pointers allows
+ * us to drop bits from the offsets. This is controlled by the bit_shift
+ * parameter. This means that if structures are aligned by 8 bytes they must be
+ * within 256KB of the base pointer. If there is no such alignment guarantee
+ * they must be within 64KB.
+ *
+ * @param ptr_base
+ *   A pointer used to calculate offsets of pointers in src_table.
+ * @param src_table
+ *   A pointer to an array of pointers.
+ * @param dest_table
+ *   A pointer to an array of compressed pointers returned by this function.
+ * @param n
+ *   The number of objects to compress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are right shifted.
+ **/
+static __rte_always_inline void
+rte_ptr_compress_16(void *ptr_base, void **src_table,
+		uint16_t *dest_table, unsigned int n, unsigned int bit_shift)
+{
+
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
+		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
+		svst1h(pg, &dest_table[i], v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
+		ptr_diff = ptr_diff >> bit_shift;
+		RTE_ASSERT(ptr_diff <= UINT16_MAX);
+		dest_table[i] = (uint16_t) ptr_diff;
+	}
+#endif
+}
+
+/**
+ * Decompress pointers from 16-bit offsets from base pointer.
+ *
+ * @param ptr_base
+ *   A pointer which was used to calculate offsets in src_table.
+ * @param src_table
+ *   A pointer to an array to compressed pointers.
+ * @param dest_table
+ *   A pointer to an array of decompressed pointers returned by this function.
+ * @param n
+ *   The number of objects to decompress, must be strictly positive.
+ * @param bit_shift
+ *   Byte alignment of memory pointed to by the pointers allows for
+ *   bits to be dropped from the offset and hence widen the memory region that
+ *   can be covered. This controls how many bits are left shifted when pointers
+ *   are recovered from the offsets.
+ **/
+static __rte_always_inline void
+rte_ptr_decompress_16(void *ptr_base, uint16_t *src_table,
+		void **dest_table, unsigned int n, unsigned int bit_shift)
+{
+	unsigned int i = 0;
+#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
+	svuint64_t v_ptr_table;
+	svbool_t pg = svwhilelt_b64(i, n);
+	do {
+		v_ptr_table = svld1uh_u64(pg, &src_table[i]);
+		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
+		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
+		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
+		i += svcntd();
+		pg = svwhilelt_b64(i, n);
+	} while (svptest_any(svptrue_b64(), pg));
+#else
+	uintptr_t ptr_diff;
+	for (; i < n; i++) {
+		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
+		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
+	}
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_PTR_COMPRESS_H */
--
2.25.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v5 2/4] test: add pointer compress tests to ring perf test
  2023-11-01 18:12   ` [PATCH v5 0/4] add pointer compression API Paul Szczepanek
  2023-11-01 18:12     ` [PATCH v5 1/4] eal: add pointer compression functions Paul Szczepanek
@ 2023-11-01 18:12     ` Paul Szczepanek
  2023-11-01 18:13     ` [PATCH v5 3/4] docs: add pointer compression to the EAL guide Paul Szczepanek
                       ` (2 subsequent siblings)
  4 siblings, 0 replies; 29+ messages in thread
From: Paul Szczepanek @ 2023-11-01 18:12 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Honnappa Nagarahalli

Add a test that runs a zero copy burst enqueue and dequeue on a ring
of raw pointers and compressed pointers at different burst sizes to
showcase performance benefits of newly added pointer compression APIs.

Refactored threading code to pass more parameters to threads to
reuse existing code. Added more bulk sizes to showcase their effects
on compression. Adjusted loop iteration numbers to take into account
bulk sizes to keep runtime constant (instead of number of operations).

Adjusted old printfs to match new ones which have aligned numbers.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 app/test/test_ring.h      |  94 +++++++++-
 app/test/test_ring_perf.c | 354 +++++++++++++++++++++++++-------------
 2 files changed, 324 insertions(+), 124 deletions(-)

diff --git a/app/test/test_ring.h b/app/test/test_ring.h
index 45c263f3ff..3b00f2465d 100644
--- a/app/test/test_ring.h
+++ b/app/test/test_ring.h
@@ -1,10 +1,12 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */

 #include <rte_malloc.h>
 #include <rte_ring.h>
 #include <rte_ring_elem.h>
+#include <rte_memcpy.h>
+#include <rte_ptr_compress.h>

 /* API type to call
  * rte_ring_<sp/mp or sc/mc>_enqueue_<bulk/burst>
@@ -25,6 +27,10 @@
 #define TEST_RING_ELEM_BULK 16
 #define TEST_RING_ELEM_BURST 32

+#define TEST_RING_ELEM_BURST_ZC 64
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 128
+#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 256
+
 #define TEST_RING_IGNORE_API_TYPE ~0U

 /* This function is placed here as it is required for both
@@ -101,6 +107,9 @@ static inline unsigned int
 test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -152,6 +161,46 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n,
 								NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy(zcd.ptr2,
+						(char *)obj + zcd.n1 * esize,
+						(ret - zcd.n1) * esize);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_16(0, obj, zcd.ptr1, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_16(0,
+						obj + (zcd.n1 * 2),
+						zcd.ptr2,
+						(ret - zcd.n1) * 2, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_enqueue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_compress_32(0, obj, zcd.ptr1, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_compress_32(0, obj + zcd.n1,
+						zcd.ptr2, ret - zcd.n1, 3);
+			rte_ring_enqueue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
@@ -162,6 +211,9 @@ static inline unsigned int
 test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 			unsigned int api_type)
 {
+	unsigned int ret;
+	struct rte_ring_zc_data zcd = {0};
+
 	/* Legacy queue APIs? */
 	if (esize == -1)
 		switch (api_type) {
@@ -213,6 +265,46 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n,
 		case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST):
 			return rte_ring_mc_dequeue_burst_elem(r, obj, esize,
 								n, NULL);
+		case (TEST_RING_ELEM_BURST_ZC):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, esize, n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_memcpy((char *)obj + zcd.n1 * esize,
+						zcd.ptr2,
+						(ret - zcd.n1) * esize);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16):
+			/* rings cannot store uint16_t so we use a uint32_t
+			 * and half the requested number of elements
+			 * and compensate by doubling the returned numbers
+			 */
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n / 2, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_16(0, zcd.ptr1, obj, zcd.n1 * 2, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_16(0, zcd.ptr2,
+						obj + zcd.n1,
+						(ret - zcd.n1) * 2,
+						3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret * 2;
+		case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32):
+			ret = rte_ring_dequeue_zc_burst_elem_start(
+					r, sizeof(uint32_t), n, &zcd, NULL);
+			if (unlikely(ret == 0))
+				return 0;
+			rte_ptr_decompress_32(0, zcd.ptr1, obj, zcd.n1, 3);
+			if (unlikely(zcd.ptr2 != NULL))
+				rte_ptr_decompress_32(0, zcd.ptr2,
+						obj + zcd.n1, ret - zcd.n1, 3);
+			rte_ring_dequeue_zc_finish(r, ret);
+			return ret;
 		default:
 			printf("Invalid API type\n");
 			return 0;
diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c
index d7c5a4c30b..8392d0c781 100644
--- a/app/test/test_ring_perf.c
+++ b/app/test/test_ring_perf.c
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2010-2014 Intel Corporation
- * Copyright(c) 2019 Arm Limited
+ * Copyright(c) 2019-2023 Arm Limited
  */


@@ -22,13 +22,13 @@

 #define RING_NAME "RING_PERF"
 #define RING_SIZE 4096
-#define MAX_BURST 32
+#define MAX_BURST 256

 /*
  * the sizes to enqueue and dequeue in testing
  * (marked volatile so they won't be seen as compile-time constants)
  */
-static const volatile unsigned bulk_sizes[] = { 8, 32 };
+static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128, 256 };

 struct lcore_pair {
 	unsigned c1, c2;
@@ -43,26 +43,30 @@ test_ring_print_test_string(unsigned int api_type, int esize,
 	if (esize == -1)
 		printf("legacy APIs");
 	else
-		printf("elem APIs: element size %dB", esize);
+		printf("elem APIs (size:%2dB)", esize);

 	if (api_type == TEST_RING_IGNORE_API_TYPE)
 		return;

 	if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF)
-		printf(": default enqueue/dequeue: ");
+		printf(" - default enqueue/dequeue");
 	else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC)
-		printf(": SP/SC: ");
+		printf(" - SP/SC");
 	else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC)
-		printf(": MP/MC: ");
+		printf(" - MP/MC");

 	if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE)
-		printf("single: ");
+		printf(" - single - ");
 	else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK)
-		printf("bulk (size: %u): ", bsz);
+		printf(" - bulk (n:%-3u) - ", bsz);
 	else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST)
-		printf("burst (size: %u): ", bsz);
+		printf(" - burst (n:%-3u) - ", bsz);
+	else if ((api_type & (TEST_RING_ELEM_BURST_ZC |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 |
+			TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0)
+		printf(" - burst zero copy (n:%-3u) - ", bsz);

-	printf("%.2F\n", value);
+	printf("cycles per elem: %.3F\n", value);
 }

 /**** Functions to analyse our core mask to get cores for different tests ***/
@@ -144,28 +148,33 @@ static void
 test_empty_dequeue(struct rte_ring *r, const int esize,
 			const unsigned int api_type)
 {
-	const unsigned int iter_shift = 26;
+	const unsigned int iter_shift = 29;
 	const unsigned int iterations = 1 << iter_shift;
 	unsigned int i = 0;
 	void *burst[MAX_BURST];

+	const unsigned int bulk_iterations = iterations / bulk_sizes[0];
 	const uint64_t start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	for (i = 0; i < bulk_iterations; i++)
 		test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type);
 	const uint64_t end = rte_rdtsc();

 	test_ring_print_test_string(api_type, esize, bulk_sizes[0],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 }

-/*
- * for the separate enqueue and dequeue threads they take in one param
- * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc
- */
-struct thread_params {
+/* describes the ring used by the enqueue and dequeue thread */
+struct ring_params {
 	struct rte_ring *r;
-	unsigned size;        /* input value, the burst size */
-	double spsc, mpmc;    /* output value, the single or multi timings */
+	unsigned int elem_size;
+	unsigned int bulk_sizes_i; /* index into bulk_size array */
+	unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */
+};
+
+/* Used do specify enqueue and dequeue ring operations and their results */
+struct thread_params {
+	struct ring_params *ring_params;
+	double *results; /* result array size must be equal to bulk_sizes */
 };

 /*
@@ -174,16 +183,15 @@ struct thread_params {
  * flag == 1 -> dequeue
  */
 static __rte_always_inline int
-enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
-	struct thread_params *p)
+enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 15;
+	const unsigned int iter_shift = 22;
 	const unsigned int iterations = 1 << iter_shift;
-	struct rte_ring *r = p->r;
-	unsigned int bsize = p->size;
 	unsigned int i;
 	void *burst = NULL;
+	unsigned int n_remaining;
+	const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i];

 #ifdef RTE_USE_C11_MEM_MODEL
 	if (__atomic_fetch_add(&lcore_count, 1, __ATOMIC_RELAXED) + 1 != 2)
@@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 		while(lcore_count != 2)
 			rte_pause();

-	burst = test_ring_calloc(MAX_BURST, esize);
+	burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size);
 	if (burst == NULL)
 		return -1;

 	const uint64_t sp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
+	const unsigned int bulk_iterations = iterations / bulk_n;
+	for (i = 0; i < bulk_iterations; i++) {
+		n_remaining = bulk_n;
 		do {
 			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_enqueue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_SPSC |
-						TEST_RING_ELEM_BULK);
+				ret = test_ring_dequeue(p->ring_params->r,
+						burst,
+						p->ring_params->elem_size,
+						n_remaining,
+						p->ring_params->ring_flags);
 			if (ret == 0)
 				rte_pause();
-		} while (!ret);
+			else
+				n_remaining -= ret;
+		} while (n_remaining > 0);
+	}
 	const uint64_t sp_end = rte_rdtsc();

-	const uint64_t mp_start = rte_rdtsc();
-	for (i = 0; i < iterations; i++)
-		do {
-			if (flag == 0)
-				ret = test_ring_enqueue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			else if (flag == 1)
-				ret = test_ring_dequeue(r, burst, esize, bsize,
-						TEST_RING_THREAD_MPMC |
-						TEST_RING_ELEM_BULK);
-			if (ret == 0)
-				rte_pause();
-		} while (!ret);
-	const uint64_t mp_end = rte_rdtsc();
+	p->results[p->ring_params->bulk_sizes_i] =
+			((double)sp_end - sp_start) / iterations;

-	p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize);
-	p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize);
 	return 0;
 }

@@ -243,15 +245,7 @@ enqueue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(0, -1, params);
-}
-
-static int
-enqueue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(0, 16, params);
+	return enqueue_dequeue_bulk_helper(0, params);
 }

 /*
@@ -263,15 +257,7 @@ dequeue_bulk(void *p)
 {
 	struct thread_params *params = p;

-	return enqueue_dequeue_bulk_helper(1, -1, params);
-}
-
-static int
-dequeue_bulk_16B(void *p)
-{
-	struct thread_params *params = p;
-
-	return enqueue_dequeue_bulk_helper(1, 16, params);
+	return enqueue_dequeue_bulk_helper(1, params);
 }

 /*
@@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p)
  * used to measure ring perf between hyperthreads, cores and sockets.
  */
 static int
-run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize)
+run_on_core_pair(struct lcore_pair *cores,
+		struct thread_params *param1, struct thread_params *param2)
 {
-	lcore_function_t *f1, *f2;
-	struct thread_params param1 = {0}, param2 = {0};
 	unsigned i;
-
-	if (esize == -1) {
-		f1 = enqueue_bulk;
-		f2 = dequeue_bulk;
-	} else {
-		f1 = enqueue_bulk_16B;
-		f2 = dequeue_bulk_16B;
-	}
+	struct ring_params *ring_params = param1->ring_params;

 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		lcore_count = 0;
-		param1.size = param2.size = bulk_sizes[i];
-		param1.r = param2.r = r;
+		ring_params->bulk_sizes_i = i;
 		if (cores->c1 == rte_get_main_lcore()) {
-			rte_eal_remote_launch(f2, &param2, cores->c2);
-			f1(&param1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
+			enqueue_bulk(param1);
 			rte_eal_wait_lcore(cores->c2);
 		} else {
-			rte_eal_remote_launch(f1, &param1, cores->c1);
-			rte_eal_remote_launch(f2, &param2, cores->c2);
+			rte_eal_remote_launch(enqueue_bulk, param1, cores->c1);
+			rte_eal_remote_launch(dequeue_bulk, param2, cores->c2);
 			if (rte_eal_wait_lcore(cores->c1) < 0)
 				return -1;
 			if (rte_eal_wait_lcore(cores->c2) < 0)
 				return -1;
 		}
 		test_ring_print_test_string(
-			TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.spsc + param2.spsc);
-		test_ring_print_test_string(
-			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK,
-			esize, bulk_sizes[i], param1.mpmc + param2.mpmc);
+				ring_params->ring_flags,
+				ring_params->elem_size,
+				bulk_sizes[i],
+				param1->results[i] + param2->results[i]);
 	}

 	return 0;
@@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize)
 	uint64_t hz = rte_get_timer_hz();
 	uint64_t lcount = 0;
 	const unsigned int lcore = rte_lcore_id();
-	struct thread_params *params = p;
+	struct ring_params *ring_params = p->ring_params;
 	void *burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -346,9 +322,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize)

 	begin = rte_get_timer_cycles();
 	while (time_diff < hz * TIME_MS / 1000) {
-		test_ring_enqueue(params->r, burst, esize, params->size,
+		test_ring_enqueue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
-		test_ring_dequeue(params->r, burst, esize, params->size,
+		test_ring_dequeue(ring_params->r, burst, esize,
+				ring_params->elem_size,
 				TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);
 		lcount++;
 		time_diff = rte_get_timer_cycles() - begin;
@@ -380,7 +358,8 @@ static int
 run_on_all_cores(struct rte_ring *r, const int esize)
 {
 	uint64_t total;
-	struct thread_params param;
+	struct ring_params ring_params = {0};
+	struct thread_params params = { .ring_params = &ring_params };
 	lcore_function_t *lcore_f;
 	unsigned int i, c;

@@ -389,21 +368,20 @@ run_on_all_cores(struct rte_ring *r, const int esize)
 	else
 		lcore_f = load_loop_fn_16B;

-	memset(&param, 0, sizeof(struct thread_params));
 	for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
 		total = 0;
 		printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]);
-		param.size = bulk_sizes[i];
-		param.r = r;
+		params.ring_params->bulk_sizes_i = i;
+		params.ring_params->r = r;

 		/* clear synchro and start workers */
 		__atomic_store_n(&synchro, 0, __ATOMIC_RELAXED);
-		if (rte_eal_mp_remote_launch(lcore_f, &param, SKIP_MAIN) < 0)
+		if (rte_eal_mp_remote_launch(lcore_f, &params, SKIP_MAIN) < 0)
 			return -1;

 		/* start synchro and launch test on main */
 		__atomic_store_n(&synchro, 1, __ATOMIC_RELAXED);
-		lcore_f(&param);
+		lcore_f(&params);

 		rte_eal_mp_wait_lcore();

@@ -462,9 +440,9 @@ static int
 test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	const unsigned int api_type)
 {
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 26;
 	const unsigned int iterations = 1 << iter_shift;
-	unsigned int sz, i = 0;
+	unsigned int sz, i;
 	void **burst = NULL;

 	burst = test_ring_calloc(MAX_BURST, esize);
@@ -472,17 +450,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 		return -1;

 	for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) {
+		const unsigned int n = iterations / bulk_sizes[sz];
 		const uint64_t start = rte_rdtsc();
-		for (i = 0; i < iterations; i++) {
+		for (i = 0; i < n; i++) {
 			test_ring_enqueue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 			test_ring_dequeue(r, burst, esize, bulk_sizes[sz],
-						api_type);
+					api_type);
 		}
 		const uint64_t end = rte_rdtsc();

 		test_ring_print_test_string(api_type, esize, bulk_sizes[sz],
-					((double)(end - start)) / iterations);
+					((double)end - start) / iterations);
 	}

 	rte_free(burst);
@@ -490,12 +469,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize,
 	return 0;
 }

+static __rte_always_inline int
+test_ring_perf_esize_run_on_two_cores(
+		struct thread_params *param1, struct thread_params *param2)
+{
+	struct lcore_pair cores;
+
+	if (get_two_hyperthreads(&cores) == 0) {
+		printf("\n### Testing using two hyperthreads ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing using two physical cores ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	if (get_two_sockets(&cores) == 0) {
+		printf("\n### Testing using two NUMA nodes ###\n");
+		if (run_on_core_pair(&cores, param1, param2) < 0)
+			return -1;
+	}
+	return 0;
+}
+
 /* Run all tests for a given element size */
 static __rte_always_inline int
 test_ring_perf_esize(const int esize)
 {
-	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
+	double results_enq[RTE_DIM(bulk_sizes)];
+	double results_deq[RTE_DIM(bulk_sizes)];
+	struct ring_params ring_params = {
+			.elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results_enq };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results_deq };

 	/*
 	 * Performance test for legacy/_elem APIs
@@ -535,22 +545,13 @@ test_ring_perf_esize(const int esize)
 	test_empty_dequeue(r, esize,
 			TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK);

-	if (get_two_hyperthreads(&cores) == 0) {
-		printf("\n### Testing using two hyperthreads ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.r = r;

-	if (get_two_cores(&cores) == 0) {
-		printf("\n### Testing using two physical cores ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
-	if (get_two_sockets(&cores) == 0) {
-		printf("\n### Testing using two NUMA nodes ###\n");
-		if (run_on_core_pair(&cores, r, esize) < 0)
-			goto test_fail;
-	}
+	ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);
+
+	ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK;
+	test_ring_perf_esize_run_on_two_cores(&param1, &param2);

 	printf("\n### Testing using all worker nodes ###\n");
 	if (run_on_all_cores(r, esize) < 0)
@@ -566,6 +567,109 @@ test_ring_perf_esize(const int esize)
 	return -1;
 }

+
+static __rte_always_inline int
+test_ring_perf_compression(void)
+{
+	double results1[RTE_DIM(bulk_sizes)];
+	double results2[RTE_DIM(bulk_sizes)];
+	double results1_comp[2][RTE_DIM(bulk_sizes)];
+	double results2_comp[2][RTE_DIM(bulk_sizes)];
+
+	struct lcore_pair cores;
+	int ret = -1;
+	unsigned int i, j;
+	struct ring_params ring_params = { .elem_size = sizeof(void *) };
+	struct thread_params param1 = {
+			.ring_params = &ring_params, .results = results1 };
+	struct thread_params param2 = {
+			.ring_params = &ring_params, .results = results2 };
+
+	printf("\n### Testing compression gain ###");
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(void *),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy ###\n");
+		ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	/* rings allow only multiples of 4 as sizes,
+	 * we allocate size 4 despite only using 2 bytes
+	 * and use half of RING_SIZE as the number of elements
+	 */
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE / 2, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[0];
+	param2.results = results2_comp[0];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (16b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	if (ret != 0)
+		return ret;
+
+	ring_params.r = rte_ring_create_elem(
+			RING_NAME, sizeof(uint32_t),
+			RING_SIZE, rte_socket_id(),
+			RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+	if (ring_params.r == NULL)
+		return -1;
+
+	param1.results = results1_comp[1];
+	param2.results = results2_comp[1];
+
+	if (get_two_cores(&cores) == 0) {
+		printf("\n### Testing zero copy with compression (32b) ###\n");
+		ring_params.ring_flags =
+				TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32;
+		ret = run_on_core_pair(&cores, &param1, &param2);
+	}
+
+	rte_ring_free(ring_params.r);
+
+	for (j = 0; j < 2; j++) {
+		printf("\n### Potential gain from compression (%d-bit offsets) "
+		"###\n", (j + 1) * 16);
+		for (i = 0; i < RTE_DIM(bulk_sizes); i++) {
+			const double result = results1[i] + results2[i];
+			const double result_comp = results1_comp[j][i] +
+				results2_comp[j][i];
+			const double gain = 100 - (result_comp / result) * 100;
+
+			printf("Gain of %5.1F%% for burst of %-3u elems\n",
+					gain, bulk_sizes[i]);
+		}
+	}
+
+	return ret;
+}
+
 static int
 test_ring_perf(void)
 {
@@ -576,6 +680,10 @@ test_ring_perf(void)
 	if (test_ring_perf_esize(16) == -1)
 		return -1;

+	/* Test for performance gain of compression */
+	if (test_ring_perf_compression() == -1)
+		return -1;
+
 	return 0;
 }

--
2.25.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v5 3/4] docs: add pointer compression to the EAL guide
  2023-11-01 18:12   ` [PATCH v5 0/4] add pointer compression API Paul Szczepanek
  2023-11-01 18:12     ` [PATCH v5 1/4] eal: add pointer compression functions Paul Szczepanek
  2023-11-01 18:12     ` [PATCH v5 2/4] test: add pointer compress tests to ring perf test Paul Szczepanek
@ 2023-11-01 18:13     ` Paul Szczepanek
  2023-11-01 18:13     ` [PATCH v5 4/4] test: add unit test for ptr compression Paul Szczepanek
  2024-02-22  8:15     ` [PATCH v5 0/4] add pointer compression API Paul Szczepanek
  4 siblings, 0 replies; 29+ messages in thread
From: Paul Szczepanek @ 2023-11-01 18:13 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Honnappa Nagarahalli

Documentation added in the EAL guide for the new
utility functions for pointer compression
showing example code and potential usecases.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .../prog_guide/env_abstraction_layer.rst      | 142 ++++++++++++++++++
 1 file changed, 142 insertions(+)

diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 6debf54efb..f04d032442 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -1192,3 +1192,145 @@ will not be deallocated.

 Any successful deallocation event will trigger a callback, for which user
 applications and other DPDK subsystems can register.
+
+.. _pointer_compression:
+
+Pointer Compression
+-------------------
+
+Use ``rte_ptr_compress_16()`` and ``rte_ptr_decompress_16()`` to compress and
+decompress pointers into 16-bit offsets. Use ``rte_ptr_compress_32()`` and
+``rte_ptr_decompress_32()`` to compress and decompress pointers into 32-bit
+offsets.
+
+Compression takes advantage of the fact that pointers are usually located in a
+limited memory region (like a mempool). By converting them to offsets from a
+base memory address they can be stored in fewer bytes. How many bytes are needed
+to store the offset is dictated by the memory region size and alignment of
+objects the pointers point to.
+
+For example, a pointer which is part of a 4GB memory pool can be stored as 32
+bit offset. If the pointer points to memory that is 8 bytes aligned then 3 bits
+can be dropped from the offset and a 32GB memory pool can now fit in 32 bits.
+
+For performance reasons these requirements are not enforced programmatically.
+The programmer is responsible for ensuring that the combination of distance
+from the base pointer and memory alignment allow for storing of the offset in
+the number of bits indicated by the function name (16 or 32). Start of mempool
+memory would be a good candidate for the base pointer. Otherwise any pointer
+that precedes all pointers, is close enough and has the same alignment as the
+pointers being compressed will work.
+
+.. note::
+
+    Performance gains depend on the batch size of pointers and CPU capabilities
+    such as vector extensions. It's important to measure the performance
+    increase on target hardware. A test called ``ring_perf_autotest`` in
+    ``dpdk-test`` can provide the measurements.
+
+Example usage
+~~~~~~~~~~~~~
+
+In this example we send pointers between two cores through a ring. While this
+is a realistic use case the code is simplified for demonstration purposes and
+does not have error handling.
+
+.. code-block:: c
+
+    #include <rte_launch.h>
+    #include <rte_ring.h>
+    #include <rte_ring_elem.h>
+    #include <rte_ptr_compress.h>
+
+    #define ITEMS_ARRAY_SIZE (1024)
+    #define BATCH_SIZE (128)
+    #define ALIGN_EXPONENT (3)
+    #define ITEM_ALIGN (1<<ALIGN_EXPONENT)
+    #define CORE_SEND (1)
+    #define CORE_RECV (2)
+
+    struct item {
+      int a;
+    } __rte_aligned(ITEM_ALIGN);
+
+    static struct item items[ITEMS_ARRAY_SIZE] = {0};
+    static struct rte_ring *ring = NULL;
+
+    static int
+    send_compressed(void *args)
+    {
+      struct item *ptrs_send[BATCH_SIZE] = {0};
+      unsigned int n_send = 0;
+      struct rte_ring_zc_data zcd = {0};
+
+      /* in this example we only fill the ptrs_send once and reuse */
+      for (;n_send < BATCH_SIZE; n_send++)
+        ptrs_send[n_send] = &items[n_send];
+
+      for(;;) {
+        n_send = rte_ring_enqueue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        /* compress ptrs_send into offsets */
+        rte_ptr_compress_32(items, /* base pointer */
+          ptrs_send, /* source array to be compressed */
+          zcd.ptr1, /* destination array to store offsets */
+          zcd.n1, /* how many pointers to compress */
+          ALIGN_EXPONENT /* how many bits can we drop from the offset */);
+
+        if (zcd.ptr2 != NULL)
+          rte_ptr_compress_32(items, ptrs_send + zcd.n1,
+            zcd.ptr2, n_send - zcd.n1, ALIGN_EXPONENT);
+
+        rte_ring_enqueue_zc_finish(ring, n_send);
+      }
+      return 1;
+    }
+
+    static int
+    recv_compressed(void *args)
+    {
+      struct item *ptrs_recv[BATCH_SIZE] = {0};
+      unsigned int n_recv;
+      struct rte_ring_zc_data zcd = {0};
+
+      for(;;) {
+        /* receive compressed pointers from the ring */
+        n_recv = rte_ring_dequeue_zc_burst_elem_start(
+          ring, sizeof(uint32_t), BATCH_SIZE, &zcd, NULL);
+
+        rte_ptr_decompress_32(items, /* base pointer */
+          zcd.ptr1, /* source array to decompress */
+          ptrs_recv, /* destination array to store pointers */
+          zcd.n1, /* how many pointers to decompress */
+          ALIGN_EXPONENT /* how many bits were dropped from the offset */);
+
+        /* handle the potential secondary buffer (caused by ring boundary) */
+        if (zcd.ptr2 != NULL)
+          rte_ptr_decompress_32(items,
+            zcd.ptr2,
+            ptrs_recv + zcd.n1,
+            n_recv - zcd.n1,
+            ALIGN_EXPONENT);
+
+        rte_ring_dequeue_zc_finish(ring, n_recv);
+
+        /* ptrs_recv contains what ptrs_send contained in the other thread */
+        /* (...) */
+      }
+      return 1;
+    }
+
+    void
+    compression_example(void)
+    {
+      ring = rte_ring_create_elem(
+        "COMPR_PTRS", sizeof(uint32_t),
+        1024, rte_socket_id(),
+        RING_F_SP_ENQ | RING_F_SC_DEQ);
+
+      rte_eal_remote_launch(send_compressed, NULL, CORE_SEND);
+      rte_eal_remote_launch(recv_compressed, NULL, CORE_RECV);
+
+      for(;;) {}
+    }
--
2.25.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v5 4/4] test: add unit test for ptr compression
  2023-11-01 18:12   ` [PATCH v5 0/4] add pointer compression API Paul Szczepanek
                       ` (2 preceding siblings ...)
  2023-11-01 18:13     ` [PATCH v5 3/4] docs: add pointer compression to the EAL guide Paul Szczepanek
@ 2023-11-01 18:13     ` Paul Szczepanek
  2024-02-22  8:15     ` [PATCH v5 0/4] add pointer compression API Paul Szczepanek
  4 siblings, 0 replies; 29+ messages in thread
From: Paul Szczepanek @ 2023-11-01 18:13 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek

Test compresses and decompresses pointers with various combinations
of memory regions and alignments and verifies the pointers are
recovered correctly.

Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 app/test/meson.build             |   1 +
 app/test/test_eal_ptr_compress.c | 108 +++++++++++++++++++++++++++++++
 2 files changed, 109 insertions(+)
 create mode 100644 app/test/test_eal_ptr_compress.c

diff --git a/app/test/meson.build b/app/test/meson.build
index 4183d66b0e..3e172b154d 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -66,6 +66,7 @@ source_file_deps = {
     'test_dmadev_api.c': ['dmadev'],
     'test_eal_flags.c': [],
     'test_eal_fs.c': [],
+    'test_eal_ptr_compress.c': [],
     'test_efd.c': ['efd', 'net'],
     'test_efd_perf.c': ['efd', 'hash'],
     'test_errno.c': [],
diff --git a/app/test/test_eal_ptr_compress.c b/app/test/test_eal_ptr_compress.c
new file mode 100644
index 0000000000..c1c9a98be7
--- /dev/null
+++ b/app/test/test_eal_ptr_compress.c
@@ -0,0 +1,108 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2014 Intel Corporation
+ */
+
+#include "test.h"
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_ptr_compress.h>
+
+#define MAX_ALIGN_EXPONENT 3
+#define PTRS_SIZE 16
+#define NUM_BASES 2
+#define NUM_REGIONS 4
+#define MAX_32BIT_REGION ((uint64_t)UINT32_MAX + 1)
+#define MAX_16BIT_REGION (UINT16_MAX + 1)
+
+static int
+test_eal_ptr_compress_params(
+	void *base,
+	uint64_t mem_sz,
+	unsigned int align_exp,
+	unsigned int num_ptrs,
+	bool use_32_bit)
+{
+	unsigned int i;
+	unsigned int align = 1 << align_exp;
+	void *ptrs[PTRS_SIZE] = {0};
+	void *ptrs_out[PTRS_SIZE] = {0};
+	uint32_t offsets32[PTRS_SIZE] = {0};
+	uint16_t offsets16[PTRS_SIZE] = {0};
+
+	for (i = 0; i < num_ptrs; i++) {
+		/* make pointers point at memory in steps of align */
+		/* alternate steps from the start and end of memory region */
+		if ((i & 1) == 1)
+			ptrs[i] = (char *)base + mem_sz - i * align;
+		else
+			ptrs[i] = (char *)base + i * align;
+	}
+
+	if (use_32_bit) {
+		rte_ptr_compress_32(base, ptrs, offsets32, num_ptrs, align_exp);
+		rte_ptr_decompress_32(base, offsets32, ptrs_out, num_ptrs,
+				align_exp);
+	} else {
+		rte_ptr_compress_16(base, ptrs, offsets16, num_ptrs, align_exp);
+		rte_ptr_decompress_16(base, offsets16, ptrs_out, num_ptrs,
+				align_exp);
+	}
+
+	TEST_ASSERT_BUFFERS_ARE_EQUAL(ptrs, ptrs_out, sizeof(void *) * num_ptrs,
+		"Decompressed pointers corrupted\nbase pointer: %p, "
+		"memory region size: %" PRIu64 ", alignment exponent: %u, "
+		"num of pointers: %u, using %s offsets",
+		base, mem_sz, align_exp, num_ptrs,
+		use_32_bit ? "32-bit" : "16-bit");
+
+	return 0;
+}
+
+static int
+test_eal_ptr_compress(void)
+{
+	unsigned int j, k, n;
+	int ret = 0;
+	void * const bases[NUM_BASES] = { (void *)0, (void *)UINT16_MAX };
+	/* maximum size for pointers aligned by consecutive powers of 2 */
+	const uint64_t region_sizes_16[NUM_REGIONS] = {
+		MAX_16BIT_REGION,
+		MAX_16BIT_REGION * 2,
+		MAX_16BIT_REGION * 4,
+		MAX_16BIT_REGION * 8,
+	};
+	const uint64_t region_sizes_32[NUM_REGIONS] = {
+		MAX_32BIT_REGION,
+		MAX_32BIT_REGION * 2,
+		MAX_32BIT_REGION * 4,
+		MAX_32BIT_REGION * 8,
+	};
+
+	for (j = 0; j < NUM_REGIONS; j++) {
+		for (k = 0; k < NUM_BASES; k++) {
+			for (n = 1; n < PTRS_SIZE; n++) {
+				ret |= test_eal_ptr_compress_params(
+					bases[k],
+					region_sizes_16[j],
+					j /* exponent of alignment */,
+					n,
+					false
+				);
+				ret |= test_eal_ptr_compress_params(
+					bases[k],
+					region_sizes_32[j],
+					j /* exponent of alignment */,
+					n,
+					true
+				);
+				if (ret != 0)
+					return ret;
+			}
+		}
+	}
+
+	return ret;
+}
+
+REGISTER_FAST_TEST(eal_ptr_compress_autotest, true, true, test_eal_ptr_compress);
--
2.25.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v5 1/4] eal: add pointer compression functions
  2023-11-01 18:12     ` [PATCH v5 1/4] eal: add pointer compression functions Paul Szczepanek
@ 2024-02-11 15:32       ` Konstantin Ananyev
  0 siblings, 0 replies; 29+ messages in thread
From: Konstantin Ananyev @ 2024-02-11 15:32 UTC (permalink / raw)
  To: Paul Szczepanek, dev; +Cc: Honnappa Nagarahalli, Kamalakshitha Aligeri


> Add a new utility header for compressing pointers. The provided
> functions can store pointers in 32-bit offsets.
> 
> The compression takes advantage of the fact that pointers are
> usually located in a limited memory region (like a mempool).
> We can compress them by converting them to offsets from a base
> memory address. Offsets can be stored in fewer bytes (dictated
> by the memory region size and alignment of the pointer).
> For example: an 8 byte aligned pointer which is part of a 32GB
> memory pool can be stored in 4 bytes.
> 
> Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>


 From one side the code itself is very small and straightforward,
from other side - it is not clear to me what is intended usage for it
within DPDK and it's applianances?
Konstantin


> ---
>   .mailmap                           |   1 +
>   lib/eal/include/meson.build        |   1 +
>   lib/eal/include/rte_ptr_compress.h | 266 +++++++++++++++++++++++++++++
>   3 files changed, 268 insertions(+)
>   create mode 100644 lib/eal/include/rte_ptr_compress.h
> 
> diff --git a/.mailmap b/.mailmap
> index 3f5bab26a8..004751d27a 100644
> --- a/.mailmap
> +++ b/.mailmap
> @@ -1069,6 +1069,7 @@ Paul Greenwalt <paul.greenwalt@intel.com>
>   Paulis Gributs <paulis.gributs@intel.com>
>   Paul Luse <paul.e.luse@intel.com>
>   Paul M Stillwell Jr <paul.m.stillwell.jr@intel.com>
> +Paul Szczepanek <paul.szczepanek@arm.com>
>   Pavan Kumar Linga <pavan.kumar.linga@intel.com>
>   Pavan Nikhilesh <pbhagavatula@marvell.com> <pbhagavatula@caviumnetworks.com>
>   Pavel Belous <pavel.belous@aquantia.com>
> diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
> index e94b056d46..ce2c733633 100644
> --- a/lib/eal/include/meson.build
> +++ b/lib/eal/include/meson.build
> @@ -36,6 +36,7 @@ headers += files(
>           'rte_pci_dev_features.h',
>           'rte_per_lcore.h',
>           'rte_pflock.h',
> +	'rte_ptr_compress.h',
>           'rte_random.h',
>           'rte_reciprocal.h',
>           'rte_seqcount.h',
> diff --git a/lib/eal/include/rte_ptr_compress.h b/lib/eal/include/rte_ptr_compress.h
> new file mode 100644
> index 0000000000..47a72e4213
> --- /dev/null
> +++ b/lib/eal/include/rte_ptr_compress.h
> @@ -0,0 +1,266 @@
> +/* SPDX-License-Identifier: BSD-shift-Clause
> + * Copyright(c) 2023 Arm Limited
> + */
> +
> +#ifndef RTE_PTR_COMPRESS_H
> +#define RTE_PTR_COMPRESS_H
> +
> +/**
> + * @file
> + * Pointer compression and decompression functions.
> + *
> + * When passing arrays full of pointers between threads, memory containing
> + * the pointers is copied multiple times which is especially costly between
> + * cores. These functions allow us to compress the pointers.
> + *
> + * Compression takes advantage of the fact that pointers are usually located in
> + * a limited memory region (like a mempool). We compress them by converting them
> + * to offsets from a base memory address. Offsets can be stored in fewer bytes.
> + *
> + * The compression functions come in two varieties: 32-bit and 16-bit.
> + *
> + * To determine how many bits are needed to compress the pointer calculate
> + * the biggest offset possible (highest value pointer - base pointer)
> + * and shift the value right according to alignment (shift by exponent of the
> + * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by
> + * 3, etc.). The resulting value must fit in either 32 or 16 bits.
> + *
> + * For usage example and further explanation please see "Pointer Compression" in
> + * doc/guides/prog_guide/env_abstraction_layer.rst
> + */
> +
> +#include <stdint.h>
> +#include <inttypes.h>
> +
> +#include <rte_branch_prediction.h>
> +#include <rte_common.h>
> +#include <rte_debug.h>
> +#include <rte_vect.h>
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +/**
> + * Compress pointers into 32-bit offsets from base pointer.
> + *
> + * @note It is programmer's responsibility to ensure the resulting offsets fit
> + * into 32 bits. Alignment of the structures pointed to by the pointers allows
> + * us to drop bits from the offsets. This is controlled by the bit_shift
> + * parameter. This means that if structures are aligned by 8 bytes they must be
> + * within 32GB of the base pointer. If there is no such alignment guarantee they
> + * must be within 4GB.
> + *
> + * @param ptr_base
> + *   A pointer used to calculate offsets of pointers in src_table.
> + * @param src_table
> + *   A pointer to an array of pointers.
> + * @param dest_table
> + *   A pointer to an array of compressed pointers returned by this function.
> + * @param n
> + *   The number of objects to compress, must be strictly positive.
> + * @param bit_shift
> + *   Byte alignment of memory pointed to by the pointers allows for
> + *   bits to be dropped from the offset and hence widen the memory region that
> + *   can be covered. This controls how many bits are right shifted.
> + **/
> +static __rte_always_inline void
> +rte_ptr_compress_32(void *ptr_base, void **src_table,
> +		uint32_t *dest_table, unsigned int n, unsigned int bit_shift)
> +{
> +	unsigned int i = 0;
> +#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
> +	svuint64_t v_ptr_table;
> +	svbool_t pg = svwhilelt_b64(i, n);
> +	do {
> +		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
> +		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
> +		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
> +		svst1w(pg, &dest_table[i], v_ptr_table);
> +		i += svcntd();
> +		pg = svwhilelt_b64(i, n);
> +	} while (svptest_any(svptrue_b64(), pg));
> +#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
> +	uint64_t ptr_diff;
> +	uint64x2_t v_ptr_table;
> +	/* right shift is done by left shifting by negative int */
> +	int64x2_t v_shift = vdupq_n_s64(-bit_shift);
> +	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
> +	for (; i < (n & ~0x1); i += 2) {
> +		v_ptr_table = vld1q_u64((const uint64_t *)src_table + i);
> +		v_ptr_table = vsubq_u64(v_ptr_table, v_ptr_base);
> +		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
> +		vst1_u32(dest_table + i, vqmovn_u64(v_ptr_table));
> +	}
> +	/* process leftover single item in case of odd number of n */
> +	if (unlikely(n & 0x1)) {
> +		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
> +		dest_table[i] = (uint32_t) (ptr_diff >> bit_shift);
> +	}
> +#else
> +	uintptr_t ptr_diff;
> +	for (; i < n; i++) {
> +		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
> +		ptr_diff = ptr_diff >> bit_shift;
> +		RTE_ASSERT(ptr_diff <= UINT32_MAX);
> +		dest_table[i] = (uint32_t) ptr_diff;
> +	}
> +#endif
> +}
> +
> +/**
> + * Decompress pointers from 32-bit offsets from base pointer.
> + *
> + * @param ptr_base
> + *   A pointer which was used to calculate offsets in src_table.
> + * @param src_table
> + *   A pointer to an array to compressed pointers.
> + * @param dest_table
> + *   A pointer to an array of decompressed pointers returned by this function.
> + * @param n
> + *   The number of objects to decompress, must be strictly positive.
> + * @param bit_shift
> + *   Byte alignment of memory pointed to by the pointers allows for
> + *   bits to be dropped from the offset and hence widen the memory region that
> + *   can be covered. This controls how many bits are left shifted when pointers
> + *   are recovered from the offsets.
> + **/
> +static __rte_always_inline void
> +rte_ptr_decompress_32(void *ptr_base, uint32_t *src_table,
> +		void **dest_table, unsigned int n, unsigned int bit_shift)
> +{
> +	unsigned int i = 0;
> +#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
> +	svuint64_t v_ptr_table;
> +	svbool_t pg = svwhilelt_b64(i, n);
> +	do {
> +		v_ptr_table = svld1uw_u64(pg, &src_table[i]);
> +		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
> +		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
> +		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
> +		i += svcntd();
> +		pg = svwhilelt_b64(i, n);
> +	} while (svptest_any(svptrue_b64(), pg));
> +#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32
> +	uint64_t ptr_diff;
> +	uint64x2_t v_ptr_table;
> +	int64x2_t v_shift = vdupq_n_s64(bit_shift);
> +	uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base);
> +	for (; i < (n & ~0x1); i += 2) {
> +		v_ptr_table = vmovl_u32(vld1_u32(src_table + i));
> +		v_ptr_table = vshlq_u64(v_ptr_table, v_shift);
> +		v_ptr_table = vaddq_u64(v_ptr_table, v_ptr_base);
> +		vst1q_u64((uint64_t *)dest_table + i, v_ptr_table);
> +	}
> +	/* process leftover single item in case of odd number of n */
> +	if (unlikely(n & 0x1)) {
> +		ptr_diff = ((uint64_t) src_table[i]) << bit_shift;
> +		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
> +	}
> +#else
> +	uintptr_t ptr_diff;
> +	for (; i < n; i++) {
> +		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
> +		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
> +	}
> +#endif
> +}
> +
> +/**
> + * Compress pointers into 16-bit offsets from base pointer.
> + *
> + * @note It is programmer's responsibility to ensure the resulting offsets fit
> + * into 16 bits. Alignment of the structures pointed to by the pointers allows
> + * us to drop bits from the offsets. This is controlled by the bit_shift
> + * parameter. This means that if structures are aligned by 8 bytes they must be
> + * within 256KB of the base pointer. If there is no such alignment guarantee
> + * they must be within 64KB.
> + *
> + * @param ptr_base
> + *   A pointer used to calculate offsets of pointers in src_table.
> + * @param src_table
> + *   A pointer to an array of pointers.
> + * @param dest_table
> + *   A pointer to an array of compressed pointers returned by this function.
> + * @param n
> + *   The number of objects to compress, must be strictly positive.
> + * @param bit_shift
> + *   Byte alignment of memory pointed to by the pointers allows for
> + *   bits to be dropped from the offset and hence widen the memory region that
> + *   can be covered. This controls how many bits are right shifted.
> + **/
> +static __rte_always_inline void
> +rte_ptr_compress_16(void *ptr_base, void **src_table,
> +		uint16_t *dest_table, unsigned int n, unsigned int bit_shift)
> +{
> +
> +	unsigned int i = 0;
> +#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
> +	svuint64_t v_ptr_table;
> +	svbool_t pg = svwhilelt_b64(i, n);
> +	do {
> +		v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i);
> +		v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base);
> +		v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift);
> +		svst1h(pg, &dest_table[i], v_ptr_table);
> +		i += svcntd();
> +		pg = svwhilelt_b64(i, n);
> +	} while (svptest_any(svptrue_b64(), pg));
> +#else
> +	uintptr_t ptr_diff;
> +	for (; i < n; i++) {
> +		ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base);
> +		ptr_diff = ptr_diff >> bit_shift;
> +		RTE_ASSERT(ptr_diff <= UINT16_MAX);
> +		dest_table[i] = (uint16_t) ptr_diff;
> +	}
> +#endif
> +}
> +
> +/**
> + * Decompress pointers from 16-bit offsets from base pointer.
> + *
> + * @param ptr_base
> + *   A pointer which was used to calculate offsets in src_table.
> + * @param src_table
> + *   A pointer to an array to compressed pointers.
> + * @param dest_table
> + *   A pointer to an array of decompressed pointers returned by this function.
> + * @param n
> + *   The number of objects to decompress, must be strictly positive.
> + * @param bit_shift
> + *   Byte alignment of memory pointed to by the pointers allows for
> + *   bits to be dropped from the offset and hence widen the memory region that
> + *   can be covered. This controls how many bits are left shifted when pointers
> + *   are recovered from the offsets.
> + **/
> +static __rte_always_inline void
> +rte_ptr_decompress_16(void *ptr_base, uint16_t *src_table,
> +		void **dest_table, unsigned int n, unsigned int bit_shift)
> +{
> +	unsigned int i = 0;
> +#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32
> +	svuint64_t v_ptr_table;
> +	svbool_t pg = svwhilelt_b64(i, n);
> +	do {
> +		v_ptr_table = svld1uh_u64(pg, &src_table[i]);
> +		v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift);
> +		v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base);
> +		svst1(pg, (uint64_t *)dest_table + i, v_ptr_table);
> +		i += svcntd();
> +		pg = svwhilelt_b64(i, n);
> +	} while (svptest_any(svptrue_b64(), pg));
> +#else
> +	uintptr_t ptr_diff;
> +	for (; i < n; i++) {
> +		ptr_diff = ((uintptr_t) src_table[i]) << bit_shift;
> +		dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff);
> +	}
> +#endif
> +}
> +
> +#ifdef __cplusplus
> +}
> +#endif
> +
> +#endif /* RTE_PTR_COMPRESS_H */
> --
> 2.25.1
> 


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v5 0/4] add pointer compression API
  2023-11-01 18:12   ` [PATCH v5 0/4] add pointer compression API Paul Szczepanek
                       ` (3 preceding siblings ...)
  2023-11-01 18:13     ` [PATCH v5 4/4] test: add unit test for ptr compression Paul Szczepanek
@ 2024-02-22  8:15     ` Paul Szczepanek
  2024-02-22 16:16       ` Konstantin Ananyev
  4 siblings, 1 reply; 29+ messages in thread
From: Paul Szczepanek @ 2024-02-22  8:15 UTC (permalink / raw)
  To: dev, konstantin.v.ananyev; +Cc: nd

For some reason your email is not visible to me, even though it's in the 
archive.

On 02/11/202416:32,Konstantin Ananyev konstantin.v.ananyev  wrote:

> From one side the code itself is very small and straightforward, > from other side - it is not clear to me what is intended usage for it
> within DPDK and it's applianances?
> Konstantin

The intended usage is explained in the cover email (see below) and demonstrated
in the test supplied in the following patch - when sending arrays of pointers
between cores as it happens in a forwarding example.

On 01/11/2023 18:12, Paul Szczepanek wrote:

> This patchset is proposing adding a new EAL header with utility functions
> that allow compression of arrays of pointers.
>
> When passing caches full of pointers between threads, memory containing
> the pointers is copied multiple times which is especially costly between
> cores. A compression method will allow us to shrink the memory size
> copied.
>
> The compression takes advantage of the fact that pointers are usually
> located in a limited memory region (like a mempool). We can compress them
> by converting them to offsets from a base memory address.
>
> Offsets can be stored in fewer bytes (dictated by the memory region size
> and alignment of the pointer). For example: an 8 byte aligned pointer
> which is part of a 32GB memory pool can be stored in 4 bytes. The API is
> very generic and does not assume mempool pointers, any pointer can be
> passed in.
>
> Compression is based on few and fast operations and especially with vector
> instructions leveraged creates minimal overhead.
>
> The API accepts and returns arrays because the overhead means it only is
> worth it when done in bulk.
>
> Test is added that shows potential performance gain from compression. In
> this test an array of pointers is passed through a ring between two cores.
> It shows the gain which is dependent on the bulk operation size. In this
> synthetic test run on ampere altra a substantial (up to 25%) performance
> gain is seen if done in bulk size larger than 32. At 32 it breaks even and
> lower sizes create a small (less than 5%) slowdown due to overhead.
>
> In a more realistic mock application running the l3 forwarding dpdk
> example that works in pipeline mode on two cores this translated into a
> ~5% throughput increase on an ampere altra.
>
> v2:
> * addressed review comments (style, explanations and typos)
> * lowered bulk iterations closer to original numbers to keep runtime short
> * fixed pointer size warning on 32-bit arch
> v3:
> * added 16-bit versions of compression functions and tests
> * added documentation of these new utility functions in the EAL guide
> v4:
> * added unit test
> * fix bug in NEON implementation of 32-bit decompress
> v5:
> * disable NEON and SVE implementation on AARCH32 due to wrong pointer size
>
> Paul Szczepanek (4):
>    eal: add pointer compression functions
>    test: add pointer compress tests to ring perf test
>    docs: add pointer compression to the EAL guide
>    test: add unit test for ptr compression
>
>   .mailmap                                      |   1 +
>   app/test/meson.build                          |   1 +
>   app/test/test_eal_ptr_compress.c              | 108 ++++++
>   app/test/test_ring.h                          |  94 ++++-
>   app/test/test_ring_perf.c                     | 354 ++++++++++++------
>   .../prog_guide/env_abstraction_layer.rst      | 142 +++++++
>   lib/eal/include/meson.build                   |   1 +
>   lib/eal/include/rte_ptr_compress.h            | 266 +++++++++++++
>   8 files changed, 843 insertions(+), 124 deletions(-)
>   create mode 100644 app/test/test_eal_ptr_compress.c
>   create mode 100644 lib/eal/include/rte_ptr_compress.h
>
> --
> 2.25.1
>

^ permalink raw reply	[flat|nested] 29+ messages in thread

* RE: [PATCH v5 0/4] add pointer compression API
  2024-02-22  8:15     ` [PATCH v5 0/4] add pointer compression API Paul Szczepanek
@ 2024-02-22 16:16       ` Konstantin Ananyev
  0 siblings, 0 replies; 29+ messages in thread
From: Konstantin Ananyev @ 2024-02-22 16:16 UTC (permalink / raw)
  To: Paul Szczepanek, dev, konstantin.v.ananyev


> For some reason your email is not visible to me, even though it's in the
> archive.

No worries.

> 
> On 02/11/202416:32,Konstantin Ananyev konstantin.v.ananyev  wrote:
> 
> > From one side the code itself is very small and straightforward, > from other side - it is not clear to me what is intended usage for it
> > within DPDK and it's applianances?
> > Konstantin
> 
> The intended usage is explained in the cover email (see below) and demonstrated
> in the test supplied in the following patch - when sending arrays of pointers
> between cores as it happens in a forwarding example.

Yes, I saw that. The thing is that test is a 'synthetic' one.
My question was about how do you expect people to use it in more realistic scenarios?
Let say user has a bunch of mbuf pointers, possibly from different mempools.
How he can use this API: how to deduce the base pointer for all of them and what to
do if it can't be done?
 
> On 01/11/2023 18:12, Paul Szczepanek wrote:
> 
> > This patchset is proposing adding a new EAL header with utility functions
> > that allow compression of arrays of pointers.
> >
> > When passing caches full of pointers between threads, memory containing
> > the pointers is copied multiple times which is especially costly between
> > cores. A compression method will allow us to shrink the memory size
> > copied.
> >
> > The compression takes advantage of the fact that pointers are usually
> > located in a limited memory region (like a mempool). We can compress them
> > by converting them to offsets from a base memory address.
> >
> > Offsets can be stored in fewer bytes (dictated by the memory region size
> > and alignment of the pointer). For example: an 8 byte aligned pointer
> > which is part of a 32GB memory pool can be stored in 4 bytes. The API is
> > very generic and does not assume mempool pointers, any pointer can be
> > passed in.
> >
> > Compression is based on few and fast operations and especially with vector
> > instructions leveraged creates minimal overhead.
> >
> > The API accepts and returns arrays because the overhead means it only is
> > worth it when done in bulk.
> >
> > Test is added that shows potential performance gain from compression. In
> > this test an array of pointers is passed through a ring between two cores.
> > It shows the gain which is dependent on the bulk operation size. In this
> > synthetic test run on ampere altra a substantial (up to 25%) performance
> > gain is seen if done in bulk size larger than 32. At 32 it breaks even and
> > lower sizes create a small (less than 5%) slowdown due to overhead.
> >
> > In a more realistic mock application running the l3 forwarding dpdk
> > example that works in pipeline mode on two cores this translated into a
> > ~5% throughput increase on an ampere altra.
> >
> > v2:
> > * addressed review comments (style, explanations and typos)
> > * lowered bulk iterations closer to original numbers to keep runtime short
> > * fixed pointer size warning on 32-bit arch
> > v3:
> > * added 16-bit versions of compression functions and tests
> > * added documentation of these new utility functions in the EAL guide
> > v4:
> > * added unit test
> > * fix bug in NEON implementation of 32-bit decompress
> > v5:
> > * disable NEON and SVE implementation on AARCH32 due to wrong pointer size
> >
> > Paul Szczepanek (4):
> >    eal: add pointer compression functions
> >    test: add pointer compress tests to ring perf test
> >    docs: add pointer compression to the EAL guide
> >    test: add unit test for ptr compression
> >
> >   .mailmap                                      |   1 +
> >   app/test/meson.build                          |   1 +
> >   app/test/test_eal_ptr_compress.c              | 108 ++++++
> >   app/test/test_ring.h                          |  94 ++++-
> >   app/test/test_ring_perf.c                     | 354 ++++++++++++------
> >   .../prog_guide/env_abstraction_layer.rst      | 142 +++++++
> >   lib/eal/include/meson.build                   |   1 +
> >   lib/eal/include/rte_ptr_compress.h            | 266 +++++++++++++
> >   8 files changed, 843 insertions(+), 124 deletions(-)
> >   create mode 100644 app/test/test_eal_ptr_compress.c
> >   create mode 100644 lib/eal/include/rte_ptr_compress.h
> >
> > --
> > 2.25.1
> >

^ permalink raw reply	[flat|nested] 29+ messages in thread

end of thread, other threads:[~2024-02-22 16:16 UTC | newest]

Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-09-27 15:08 [RFC 0/2] add pointer compression API Paul Szczepanek
2023-09-27 15:08 ` [RFC 1/2] eal: add pointer compression functions Paul Szczepanek
2023-10-09 15:54   ` Thomas Monjalon
2023-10-11 13:36     ` Honnappa Nagarahalli
2023-10-11 16:43       ` Paul Szczepanek
2023-10-11 12:43   ` [RFC v2 0/2] add pointer compression API Paul Szczepanek
2023-10-11 12:43     ` [RFC v2 1/2] eal: add pointer compression functions Paul Szczepanek
2023-10-11 12:43     ` [RFC v2 2/2] test: add pointer compress tests to ring perf test Paul Szczepanek
2023-10-31 18:10   ` [PATCH v3 0/3] add pointer compression API Paul Szczepanek
2023-10-31 18:10     ` [PATCH v3 1/3] eal: add pointer compression functions Paul Szczepanek
2023-10-31 18:10     ` [PATCH v3 2/3] test: add pointer compress tests to ring perf test Paul Szczepanek
2023-10-31 18:10     ` [PATCH v3 3/3] docs: add pointer compression to the EAL guide Paul Szczepanek
2023-11-01  7:42     ` [PATCH v3 0/3] add pointer compression API Morten Brørup
2023-11-01 12:52       ` Paul Szczepanek
2023-11-01 12:46   ` [PATCH v4 0/4] " Paul Szczepanek
2023-11-01 12:46     ` [PATCH v4 1/4] eal: add pointer compression functions Paul Szczepanek
2023-11-01 12:46     ` [PATCH v4 2/4] test: add pointer compress tests to ring perf test Paul Szczepanek
2023-11-01 12:46     ` [PATCH v4 3/4] docs: add pointer compression to the EAL guide Paul Szczepanek
2023-11-01 12:46     ` [PATCH v4 4/4] test: add unit test for ptr compression Paul Szczepanek
2023-11-01 18:12   ` [PATCH v5 0/4] add pointer compression API Paul Szczepanek
2023-11-01 18:12     ` [PATCH v5 1/4] eal: add pointer compression functions Paul Szczepanek
2024-02-11 15:32       ` Konstantin Ananyev
2023-11-01 18:12     ` [PATCH v5 2/4] test: add pointer compress tests to ring perf test Paul Szczepanek
2023-11-01 18:13     ` [PATCH v5 3/4] docs: add pointer compression to the EAL guide Paul Szczepanek
2023-11-01 18:13     ` [PATCH v5 4/4] test: add unit test for ptr compression Paul Szczepanek
2024-02-22  8:15     ` [PATCH v5 0/4] add pointer compression API Paul Szczepanek
2024-02-22 16:16       ` Konstantin Ananyev
2023-09-27 15:08 ` [RFC 2/2] test: add pointer compress tests to ring perf test Paul Szczepanek
2023-10-09 15:48   ` Thomas Monjalon

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).