DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] [PATCH v2 0/4] DPDK memcpy optimization
@ 2015-01-29  2:38 Zhihong Wang
  2015-01-29  2:38 ` [dpdk-dev] [PATCH v2 1/4] app/test: Disabled VTA for memcpy test in app/test/Makefile Zhihong Wang
                   ` (6 more replies)
  0 siblings, 7 replies; 12+ messages in thread
From: Zhihong Wang @ 2015-01-29  2:38 UTC (permalink / raw)
  To: dev

This patch set optimizes memcpy for DPDK for both SSE and AVX platforms.
It also extends memcpy test coverage with unaligned cases and more test points.

Optimization techniques are summarized below:

1. Utilize full cache bandwidth

2. Enforce aligned stores

3. Apply load address alignment based on architecture features

4. Make load/store address available as early as possible

5. General optimization techniques like inlining, branch reducing, prefetch pattern access

--------------
Changes in v2:

1. Reduced constant test cases in app/test/test_memcpy_perf.c for fast build

2. Modified macro definition for better code readability & safety

Zhihong Wang (4):
  app/test: Disabled VTA for memcpy test in app/test/Makefile
  app/test: Removed unnecessary test cases in app/test/test_memcpy.c
  app/test: Extended test coverage in app/test/test_memcpy_perf.c
  lib/librte_eal: Optimized memcpy in arch/x86/rte_memcpy.h for both SSE
    and AVX platforms

 app/test/Makefile                                  |   6 +
 app/test/test_memcpy.c                             |  52 +-
 app/test/test_memcpy_perf.c                        | 220 ++++---
 .../common/include/arch/x86/rte_memcpy.h           | 680 +++++++++++++++------
 4 files changed, 654 insertions(+), 304 deletions(-)

-- 
1.9.3

^ permalink raw reply	[flat|nested] 12+ messages in thread

* [dpdk-dev] [PATCH v2 1/4] app/test: Disabled VTA for memcpy test in app/test/Makefile
  2015-01-29  2:38 [dpdk-dev] [PATCH v2 0/4] DPDK memcpy optimization Zhihong Wang
@ 2015-01-29  2:38 ` Zhihong Wang
  2015-01-29  2:38 ` [dpdk-dev] [PATCH v2 2/4] app/test: Removed unnecessary test cases in app/test/test_memcpy.c Zhihong Wang
                   ` (5 subsequent siblings)
  6 siblings, 0 replies; 12+ messages in thread
From: Zhihong Wang @ 2015-01-29  2:38 UTC (permalink / raw)
  To: dev

VTA is for debugging only, it increases compile time and binary size, especially when there're a lot of inlines.
So disable it since memcpy test contains a lot of inline calls.

Signed-off-by: Zhihong Wang <zhihong.wang@intel.com>
---
 app/test/Makefile | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/app/test/Makefile b/app/test/Makefile
index 4311f96..94dbadf 100644
--- a/app/test/Makefile
+++ b/app/test/Makefile
@@ -143,6 +143,12 @@ CFLAGS_test_kni.o += -Wno-deprecated-declarations
 endif
 CFLAGS += -D_GNU_SOURCE
 
+# Disable VTA for memcpy test
+ifeq ($(CC), gcc)
+CFLAGS_test_memcpy.o += -fno-var-tracking-assignments
+CFLAGS_test_memcpy_perf.o += -fno-var-tracking-assignments
+endif
+
 # this application needs libraries first
 DEPDIRS-y += lib
 
-- 
1.9.3

^ permalink raw reply	[flat|nested] 12+ messages in thread

* [dpdk-dev] [PATCH v2 2/4] app/test: Removed unnecessary test cases in app/test/test_memcpy.c
  2015-01-29  2:38 [dpdk-dev] [PATCH v2 0/4] DPDK memcpy optimization Zhihong Wang
  2015-01-29  2:38 ` [dpdk-dev] [PATCH v2 1/4] app/test: Disabled VTA for memcpy test in app/test/Makefile Zhihong Wang
@ 2015-01-29  2:38 ` Zhihong Wang
  2015-01-29  2:38 ` [dpdk-dev] [PATCH v2 3/4] app/test: Extended test coverage in app/test/test_memcpy_perf.c Zhihong Wang
                   ` (4 subsequent siblings)
  6 siblings, 0 replies; 12+ messages in thread
From: Zhihong Wang @ 2015-01-29  2:38 UTC (permalink / raw)
  To: dev

Removed unnecessary test cases for base move functions since the function "func_test" covers them all.

Signed-off-by: Zhihong Wang <zhihong.wang@intel.com>
---
 app/test/test_memcpy.c | 52 +-------------------------------------------------
 1 file changed, 1 insertion(+), 51 deletions(-)

diff --git a/app/test/test_memcpy.c b/app/test/test_memcpy.c
index 56b8e1e..b2bb4e0 100644
--- a/app/test/test_memcpy.c
+++ b/app/test/test_memcpy.c
@@ -78,56 +78,9 @@ static size_t buf_sizes[TEST_VALUE_RANGE];
 #define TEST_BATCH_SIZE         100
 
 /* Data is aligned on this many bytes (power of 2) */
-#define ALIGNMENT_UNIT          16
+#define ALIGNMENT_UNIT          32
 
 
-
-/* Structure with base memcpy func pointer, and number of bytes it copies */
-struct base_memcpy_func {
-	void (*func)(uint8_t *dst, const uint8_t *src);
-	unsigned size;
-};
-
-/* To create base_memcpy_func structure entries */
-#define BASE_FUNC(n) {rte_mov##n, n}
-
-/* Max number of bytes that can be copies with a "base" memcpy functions */
-#define MAX_BASE_FUNC_SIZE 256
-
-/*
- * Test the "base" memcpy functions, that a copy fixed number of bytes.
- */
-static int
-base_func_test(void)
-{
-	const struct base_memcpy_func base_memcpy_funcs[6] = {
-		BASE_FUNC(16),
-		BASE_FUNC(32),
-		BASE_FUNC(48),
-		BASE_FUNC(64),
-		BASE_FUNC(128),
-		BASE_FUNC(256),
-	};
-	unsigned i, j;
-	unsigned num_funcs = sizeof(base_memcpy_funcs) / sizeof(base_memcpy_funcs[0]);
-	uint8_t dst[MAX_BASE_FUNC_SIZE];
-	uint8_t src[MAX_BASE_FUNC_SIZE];
-
-	for (i = 0; i < num_funcs; i++) {
-		unsigned size = base_memcpy_funcs[i].size;
-		for (j = 0; j < size; j++) {
-			dst[j] = 0;
-			src[j] = (uint8_t) rte_rand();
-		}
-		base_memcpy_funcs[i].func(dst, src);
-		for (j = 0; j < size; j++)
-			if (dst[j] != src[j])
-				return -1;
-	}
-
-	return 0;
-}
-
 /*
  * Create two buffers, and initialise one with random values. These are copied
  * to the second buffer and then compared to see if the copy was successful.
@@ -218,9 +171,6 @@ test_memcpy(void)
 	ret = func_test();
 	if (ret != 0)
 		return -1;
-	ret = base_func_test();
-	if (ret != 0)
-		return -1;
 	return 0;
 }
 
-- 
1.9.3

^ permalink raw reply	[flat|nested] 12+ messages in thread

* [dpdk-dev] [PATCH v2 3/4] app/test: Extended test coverage in app/test/test_memcpy_perf.c
  2015-01-29  2:38 [dpdk-dev] [PATCH v2 0/4] DPDK memcpy optimization Zhihong Wang
  2015-01-29  2:38 ` [dpdk-dev] [PATCH v2 1/4] app/test: Disabled VTA for memcpy test in app/test/Makefile Zhihong Wang
  2015-01-29  2:38 ` [dpdk-dev] [PATCH v2 2/4] app/test: Removed unnecessary test cases in app/test/test_memcpy.c Zhihong Wang
@ 2015-01-29  2:38 ` Zhihong Wang
  2015-01-29  2:38 ` [dpdk-dev] [PATCH v2 4/4] lib/librte_eal: Optimized memcpy in arch/x86/rte_memcpy.h for both SSE and AVX platforms Zhihong Wang
                   ` (3 subsequent siblings)
  6 siblings, 0 replies; 12+ messages in thread
From: Zhihong Wang @ 2015-01-29  2:38 UTC (permalink / raw)
  To: dev

Main code changes:

1. Added more typical data points for a thorough performance test

2. Added unaligned test cases since it's common in DPDK usage

Signed-off-by: Zhihong Wang <zhihong.wang@intel.com>
---
 app/test/test_memcpy_perf.c | 220 +++++++++++++++++++++++++++-----------------
 1 file changed, 138 insertions(+), 82 deletions(-)

diff --git a/app/test/test_memcpy_perf.c b/app/test/test_memcpy_perf.c
index 7809610..754828e 100644
--- a/app/test/test_memcpy_perf.c
+++ b/app/test/test_memcpy_perf.c
@@ -54,9 +54,10 @@
 /* List of buffer sizes to test */
 #if TEST_VALUE_RANGE == 0
 static size_t buf_sizes[] = {
-	0, 1, 7, 8, 9, 15, 16, 17, 31, 32, 33, 63, 64, 65, 127, 128, 129, 255,
-	256, 257, 320, 384, 511, 512, 513, 1023, 1024, 1025, 1518, 1522, 1600,
-	2048, 3072, 4096, 5120, 6144, 7168, 8192
+	1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 15, 16, 17, 31, 32, 33, 63, 64, 65, 127, 128,
+	129, 191, 192, 193, 255, 256, 257, 319, 320, 321, 383, 384, 385, 447, 448,
+	449, 511, 512, 513, 767, 768, 769, 1023, 1024, 1025, 1518, 1522, 1536, 1600,
+	2048, 2560, 3072, 3584, 4096, 4608, 5120, 5632, 6144, 6656, 7168, 7680, 8192
 };
 /* MUST be as large as largest packet size above */
 #define SMALL_BUFFER_SIZE       8192
@@ -78,7 +79,7 @@ static size_t buf_sizes[TEST_VALUE_RANGE];
 #define TEST_BATCH_SIZE         100
 
 /* Data is aligned on this many bytes (power of 2) */
-#define ALIGNMENT_UNIT          16
+#define ALIGNMENT_UNIT          32
 
 /*
  * Pointers used in performance tests. The two large buffers are for uncached
@@ -94,19 +95,19 @@ init_buffers(void)
 {
 	unsigned i;
 
-	large_buf_read = rte_malloc("memcpy", LARGE_BUFFER_SIZE, ALIGNMENT_UNIT);
+	large_buf_read = rte_malloc("memcpy", LARGE_BUFFER_SIZE + ALIGNMENT_UNIT, ALIGNMENT_UNIT);
 	if (large_buf_read == NULL)
 		goto error_large_buf_read;
 
-	large_buf_write = rte_malloc("memcpy", LARGE_BUFFER_SIZE, ALIGNMENT_UNIT);
+	large_buf_write = rte_malloc("memcpy", LARGE_BUFFER_SIZE + ALIGNMENT_UNIT, ALIGNMENT_UNIT);
 	if (large_buf_write == NULL)
 		goto error_large_buf_write;
 
-	small_buf_read = rte_malloc("memcpy", SMALL_BUFFER_SIZE, ALIGNMENT_UNIT);
+	small_buf_read = rte_malloc("memcpy", SMALL_BUFFER_SIZE + ALIGNMENT_UNIT, ALIGNMENT_UNIT);
 	if (small_buf_read == NULL)
 		goto error_small_buf_read;
 
-	small_buf_write = rte_malloc("memcpy", SMALL_BUFFER_SIZE, ALIGNMENT_UNIT);
+	small_buf_write = rte_malloc("memcpy", SMALL_BUFFER_SIZE + ALIGNMENT_UNIT, ALIGNMENT_UNIT);
 	if (small_buf_write == NULL)
 		goto error_small_buf_write;
 
@@ -140,25 +141,25 @@ free_buffers(void)
 
 /*
  * Get a random offset into large array, with enough space needed to perform
- * max copy size. Offset is aligned.
+ * max copy size. Offset is aligned, uoffset is used for unalignment setting.
  */
 static inline size_t
-get_rand_offset(void)
+get_rand_offset(size_t uoffset)
 {
-	return ((rte_rand() % (LARGE_BUFFER_SIZE - SMALL_BUFFER_SIZE)) &
-	                ~(ALIGNMENT_UNIT - 1));
+	return (((rte_rand() % (LARGE_BUFFER_SIZE - SMALL_BUFFER_SIZE)) &
+			~(ALIGNMENT_UNIT - 1)) + uoffset);
 }
 
 /* Fill in source and destination addresses. */
 static inline void
-fill_addr_arrays(size_t *dst_addr, int is_dst_cached,
-		size_t *src_addr, int is_src_cached)
+fill_addr_arrays(size_t *dst_addr, int is_dst_cached, size_t dst_uoffset,
+				 size_t *src_addr, int is_src_cached, size_t src_uoffset)
 {
 	unsigned int i;
 
 	for (i = 0; i < TEST_BATCH_SIZE; i++) {
-		dst_addr[i] = (is_dst_cached) ? 0 : get_rand_offset();
-		src_addr[i] = (is_src_cached) ? 0 : get_rand_offset();
+		dst_addr[i] = (is_dst_cached) ? dst_uoffset : get_rand_offset(dst_uoffset);
+		src_addr[i] = (is_src_cached) ? src_uoffset : get_rand_offset(src_uoffset);
 	}
 }
 
@@ -169,16 +170,17 @@ fill_addr_arrays(size_t *dst_addr, int is_dst_cached,
  */
 static void
 do_uncached_write(uint8_t *dst, int is_dst_cached,
-		const uint8_t *src, int is_src_cached, size_t size)
+				  const uint8_t *src, int is_src_cached, size_t size)
 {
 	unsigned i, j;
 	size_t dst_addrs[TEST_BATCH_SIZE], src_addrs[TEST_BATCH_SIZE];
 
 	for (i = 0; i < (TEST_ITERATIONS / TEST_BATCH_SIZE); i++) {
-		fill_addr_arrays(dst_addrs, is_dst_cached,
-			 src_addrs, is_src_cached);
-		for (j = 0; j < TEST_BATCH_SIZE; j++)
+		fill_addr_arrays(dst_addrs, is_dst_cached, 0,
+						 src_addrs, is_src_cached, 0);
+		for (j = 0; j < TEST_BATCH_SIZE; j++) {
 			rte_memcpy(dst+dst_addrs[j], src+src_addrs[j], size);
+		}
 	}
 }
 
@@ -186,52 +188,111 @@ do_uncached_write(uint8_t *dst, int is_dst_cached,
  * Run a single memcpy performance test. This is a macro to ensure that if
  * the "size" parameter is a constant it won't be converted to a variable.
  */
-#define SINGLE_PERF_TEST(dst, is_dst_cached, src, is_src_cached, size) do {   \
-	unsigned int iter, t;                                                 \
-	size_t dst_addrs[TEST_BATCH_SIZE], src_addrs[TEST_BATCH_SIZE];        \
-	uint64_t start_time, total_time = 0;                                  \
-	uint64_t total_time2 = 0;                                             \
-	for (iter = 0; iter < (TEST_ITERATIONS / TEST_BATCH_SIZE); iter++) {  \
-		fill_addr_arrays(dst_addrs, is_dst_cached,                    \
-		                 src_addrs, is_src_cached);                   \
-		start_time = rte_rdtsc();                                     \
-		for (t = 0; t < TEST_BATCH_SIZE; t++)                         \
-			rte_memcpy(dst+dst_addrs[t], src+src_addrs[t], size); \
-		total_time += rte_rdtsc() - start_time;                       \
-	}                                                                     \
-	for (iter = 0; iter < (TEST_ITERATIONS / TEST_BATCH_SIZE); iter++) {  \
-		fill_addr_arrays(dst_addrs, is_dst_cached,                    \
-		                 src_addrs, is_src_cached);                   \
-		start_time = rte_rdtsc();                                     \
-		for (t = 0; t < TEST_BATCH_SIZE; t++)                         \
-			memcpy(dst+dst_addrs[t], src+src_addrs[t], size);     \
-		total_time2 += rte_rdtsc() - start_time;                      \
-	}                                                                     \
-	printf("%8.0f -",  (double)total_time /TEST_ITERATIONS);              \
-	printf("%5.0f",  (double)total_time2 / TEST_ITERATIONS);              \
+#define SINGLE_PERF_TEST(dst, is_dst_cached, dst_uoffset,                   \
+                         src, is_src_cached, src_uoffset, size)             \
+do {                                                                        \
+    unsigned int iter, t;                                                   \
+    size_t dst_addrs[TEST_BATCH_SIZE], src_addrs[TEST_BATCH_SIZE];          \
+    uint64_t start_time, total_time = 0;                                    \
+    uint64_t total_time2 = 0;                                               \
+    for (iter = 0; iter < (TEST_ITERATIONS / TEST_BATCH_SIZE); iter++) {    \
+        fill_addr_arrays(dst_addrs, is_dst_cached, dst_uoffset,             \
+                         src_addrs, is_src_cached, src_uoffset);            \
+        start_time = rte_rdtsc();                                           \
+        for (t = 0; t < TEST_BATCH_SIZE; t++)                               \
+            rte_memcpy(dst+dst_addrs[t], src+src_addrs[t], size);           \
+        total_time += rte_rdtsc() - start_time;                             \
+    }                                                                       \
+    for (iter = 0; iter < (TEST_ITERATIONS / TEST_BATCH_SIZE); iter++) {    \
+        fill_addr_arrays(dst_addrs, is_dst_cached, dst_uoffset,             \
+                         src_addrs, is_src_cached, src_uoffset);            \
+        start_time = rte_rdtsc();                                           \
+        for (t = 0; t < TEST_BATCH_SIZE; t++)                               \
+            memcpy(dst+dst_addrs[t], src+src_addrs[t], size);               \
+        total_time2 += rte_rdtsc() - start_time;                            \
+    }                                                                       \
+    printf("%8.0f -",  (double)total_time /TEST_ITERATIONS);                \
+    printf("%5.0f",  (double)total_time2 / TEST_ITERATIONS);                \
 } while (0)
 
-/* Run memcpy() tests for each cached/uncached permutation. */
-#define ALL_PERF_TESTS_FOR_SIZE(n) do {                             \
-	if (__builtin_constant_p(n))                                \
-		printf("\nC%6u", (unsigned)n);                      \
-	else                                                        \
-		printf("\n%7u", (unsigned)n);                       \
-	SINGLE_PERF_TEST(small_buf_write, 1, small_buf_read, 1, n); \
-	SINGLE_PERF_TEST(large_buf_write, 0, small_buf_read, 1, n); \
-	SINGLE_PERF_TEST(small_buf_write, 1, large_buf_read, 0, n); \
-	SINGLE_PERF_TEST(large_buf_write, 0, large_buf_read, 0, n); \
+/* Run aligned memcpy tests for each cached/uncached permutation */
+#define ALL_PERF_TESTS_FOR_SIZE(n)                                       \
+do {                                                                     \
+    if (__builtin_constant_p(n))                                         \
+        printf("\nC%6u", (unsigned)n);                                   \
+    else                                                                 \
+        printf("\n%7u", (unsigned)n);                                    \
+    SINGLE_PERF_TEST(small_buf_write, 1, 0, small_buf_read, 1, 0, n);    \
+    SINGLE_PERF_TEST(large_buf_write, 0, 0, small_buf_read, 1, 0, n);    \
+    SINGLE_PERF_TEST(small_buf_write, 1, 0, large_buf_read, 0, 0, n);    \
+    SINGLE_PERF_TEST(large_buf_write, 0, 0, large_buf_read, 0, 0, n);    \
 } while (0)
 
-/*
- * Run performance tests for a number of different sizes and cached/uncached
- * permutations.
- */
+/* Run unaligned memcpy tests for each cached/uncached permutation */
+#define ALL_PERF_TESTS_FOR_SIZE_UNALIGNED(n)                             \
+do {                                                                     \
+    if (__builtin_constant_p(n))                                         \
+        printf("\nC%6u", (unsigned)n);                                   \
+    else                                                                 \
+        printf("\n%7u", (unsigned)n);                                    \
+    SINGLE_PERF_TEST(small_buf_write, 1, 1, small_buf_read, 1, 5, n);    \
+    SINGLE_PERF_TEST(large_buf_write, 0, 1, small_buf_read, 1, 5, n);    \
+    SINGLE_PERF_TEST(small_buf_write, 1, 1, large_buf_read, 0, 5, n);    \
+    SINGLE_PERF_TEST(large_buf_write, 0, 1, large_buf_read, 0, 5, n);    \
+} while (0)
+
+/* Run memcpy tests for constant length */
+#define ALL_PERF_TEST_FOR_CONSTANT                                      \
+do {                                                                    \
+    TEST_CONSTANT(6U); TEST_CONSTANT(64U); TEST_CONSTANT(128U);         \
+    TEST_CONSTANT(192U); TEST_CONSTANT(256U); TEST_CONSTANT(512U);      \
+    TEST_CONSTANT(768U); TEST_CONSTANT(1024U); TEST_CONSTANT(1536U);    \
+} while (0)
+
+/* Run all memcpy tests for aligned constant cases */
+static inline void
+perf_test_constant_aligned(void)
+{
+#define TEST_CONSTANT ALL_PERF_TESTS_FOR_SIZE
+	ALL_PERF_TEST_FOR_CONSTANT;
+#undef TEST_CONSTANT
+}
+
+/* Run all memcpy tests for unaligned constant cases */
+static inline void
+perf_test_constant_unaligned(void)
+{
+#define TEST_CONSTANT ALL_PERF_TESTS_FOR_SIZE_UNALIGNED
+	ALL_PERF_TEST_FOR_CONSTANT;
+#undef TEST_CONSTANT
+}
+
+/* Run all memcpy tests for aligned variable cases */
+static inline void
+perf_test_variable_aligned(void)
+{
+	unsigned n = sizeof(buf_sizes) / sizeof(buf_sizes[0]);
+	unsigned i;
+	for (i = 0; i < n; i++) {
+		ALL_PERF_TESTS_FOR_SIZE((size_t)buf_sizes[i]);
+	}
+}
+
+/* Run all memcpy tests for unaligned variable cases */
+static inline void
+perf_test_variable_unaligned(void)
+{
+	unsigned n = sizeof(buf_sizes) / sizeof(buf_sizes[0]);
+	unsigned i;
+	for (i = 0; i < n; i++) {
+		ALL_PERF_TESTS_FOR_SIZE_UNALIGNED((size_t)buf_sizes[i]);
+	}
+}
+
+/* Run all memcpy tests */
 static int
 perf_test(void)
 {
-	const unsigned num_buf_sizes = sizeof(buf_sizes) / sizeof(buf_sizes[0]);
-	unsigned i;
 	int ret;
 
 	ret = init_buffers();
@@ -239,7 +300,8 @@ perf_test(void)
 		return ret;
 
 #if TEST_VALUE_RANGE != 0
-	/* Setup buf_sizes array, if required */
+	/* Set up buf_sizes array, if required */
+	unsigned i;
 	for (i = 0; i < TEST_VALUE_RANGE; i++)
 		buf_sizes[i] = i;
 #endif
@@ -248,28 +310,23 @@ perf_test(void)
 	do_uncached_write(large_buf_write, 0, small_buf_read, 1, SMALL_BUFFER_SIZE);
 
 	printf("\n** rte_memcpy() - memcpy perf. tests (C = compile-time constant) **\n"
-	       "======= ============== ============== ============== ==============\n"
-	       "   Size Cache to cache   Cache to mem   Mem to cache     Mem to mem\n"
-	       "(bytes)        (ticks)        (ticks)        (ticks)        (ticks)\n"
-	       "------- -------------- -------------- -------------- --------------");
-
-	/* Do tests where size is a variable */
-	for (i = 0; i < num_buf_sizes; i++) {
-		ALL_PERF_TESTS_FOR_SIZE((size_t)buf_sizes[i]);
-	}
+		   "======= ============== ============== ============== ==============\n"
+		   "   Size Cache to cache   Cache to mem   Mem to cache     Mem to mem\n"
+		   "(bytes)        (ticks)        (ticks)        (ticks)        (ticks)\n"
+		   "------- -------------- -------------- -------------- --------------");
+
+	printf("\n========================== %2dB aligned ============================", ALIGNMENT_UNIT);
+	/* Do aligned tests where size is a variable */
+	perf_test_variable_aligned();
 	printf("\n------- -------------- -------------- -------------- --------------");
-	/* Do tests where size is a compile-time constant */
-	ALL_PERF_TESTS_FOR_SIZE(63U);
-	ALL_PERF_TESTS_FOR_SIZE(64U);
-	ALL_PERF_TESTS_FOR_SIZE(65U);
-	ALL_PERF_TESTS_FOR_SIZE(255U);
-	ALL_PERF_TESTS_FOR_SIZE(256U);
-	ALL_PERF_TESTS_FOR_SIZE(257U);
-	ALL_PERF_TESTS_FOR_SIZE(1023U);
-	ALL_PERF_TESTS_FOR_SIZE(1024U);
-	ALL_PERF_TESTS_FOR_SIZE(1025U);
-	ALL_PERF_TESTS_FOR_SIZE(1518U);
-
+	/* Do aligned tests where size is a compile-time constant */
+	perf_test_constant_aligned();
+	printf("\n=========================== Unaligned =============================");
+	/* Do unaligned tests where size is a variable */
+	perf_test_variable_unaligned();
+	printf("\n------- -------------- -------------- -------------- --------------");
+	/* Do unaligned tests where size is a compile-time constant */
+	perf_test_constant_unaligned();
 	printf("\n======= ============== ============== ============== ==============\n\n");
 
 	free_buffers();
@@ -277,7 +334,6 @@ perf_test(void)
 	return 0;
 }
 
-
 static int
 test_memcpy_perf(void)
 {
-- 
1.9.3

^ permalink raw reply	[flat|nested] 12+ messages in thread

* [dpdk-dev] [PATCH v2 4/4] lib/librte_eal: Optimized memcpy in arch/x86/rte_memcpy.h for both SSE and AVX platforms
  2015-01-29  2:38 [dpdk-dev] [PATCH v2 0/4] DPDK memcpy optimization Zhihong Wang
                   ` (2 preceding siblings ...)
  2015-01-29  2:38 ` [dpdk-dev] [PATCH v2 3/4] app/test: Extended test coverage in app/test/test_memcpy_perf.c Zhihong Wang
@ 2015-01-29  2:38 ` Zhihong Wang
  2015-01-29 15:17   ` Ananyev, Konstantin
  2015-01-29  6:16 ` [dpdk-dev] [PATCH v2 0/4] DPDK memcpy optimization Fu, JingguoX
                   ` (2 subsequent siblings)
  6 siblings, 1 reply; 12+ messages in thread
From: Zhihong Wang @ 2015-01-29  2:38 UTC (permalink / raw)
  To: dev

Main code changes:

1. Differentiate architectural features based on CPU flags

    a. Implement separated move functions for SSE/AVX/AVX2 to make full utilization of cache bandwidth

    b. Implement separated copy flow specifically optimized for target architecture

2. Rewrite the memcpy function "rte_memcpy"

    a. Add store aligning

    b. Add load aligning based on architectural features

    c. Put block copy loop into inline move functions for better control of instruction order

    d. Eliminate unnecessary MOVs

3. Rewrite the inline move functions

    a. Add move functions for unaligned load cases

    b. Change instruction order in copy loops for better pipeline utilization

    c. Use intrinsics instead of assembly code

4. Remove slow glibc call for constant copies

Signed-off-by: Zhihong Wang <zhihong.wang@intel.com>
---
 .../common/include/arch/x86/rte_memcpy.h           | 680 +++++++++++++++------
 1 file changed, 509 insertions(+), 171 deletions(-)

diff --git a/lib/librte_eal/common/include/arch/x86/rte_memcpy.h b/lib/librte_eal/common/include/arch/x86/rte_memcpy.h
index fb9eba8..7b2d382 100644
--- a/lib/librte_eal/common/include/arch/x86/rte_memcpy.h
+++ b/lib/librte_eal/common/include/arch/x86/rte_memcpy.h
@@ -34,166 +34,189 @@
 #ifndef _RTE_MEMCPY_X86_64_H_
 #define _RTE_MEMCPY_X86_64_H_
 
+/**
+ * @file
+ *
+ * Functions for SSE/AVX/AVX2 implementation of memcpy().
+ */
+
+#include <stdio.h>
 #include <stdint.h>
 #include <string.h>
-#include <emmintrin.h>
+#include <x86intrin.h>
 
 #ifdef __cplusplus
 extern "C" {
 #endif
 
-#include "generic/rte_memcpy.h"
+/**
+ * Copy bytes from one location to another. The locations must not overlap.
+ *
+ * @note This is implemented as a macro, so it's address should not be taken
+ * and care is needed as parameter expressions may be evaluated multiple times.
+ *
+ * @param dst
+ *   Pointer to the destination of the data.
+ * @param src
+ *   Pointer to the source data.
+ * @param n
+ *   Number of bytes to copy.
+ * @return
+ *   Pointer to the destination data.
+ */
+static inline void *
+rte_memcpy(void *dst, const void *src, size_t n) __attribute__((always_inline));
 
-#ifdef __INTEL_COMPILER
-#pragma warning(disable:593) /* Stop unused variable warning (reg_a etc). */
-#endif
+#ifdef RTE_MACHINE_CPUFLAG_AVX2
 
+/**
+ * AVX2 implementation below
+ */
+
+/**
+ * Copy 16 bytes from one location to another,
+ * locations should not overlap.
+ */
 static inline void
 rte_mov16(uint8_t *dst, const uint8_t *src)
 {
-	__m128i reg_a;
-	asm volatile (
-		"movdqu (%[src]), %[reg_a]\n\t"
-		"movdqu %[reg_a], (%[dst])\n\t"
-		: [reg_a] "=x" (reg_a)
-		: [src] "r" (src),
-		  [dst] "r"(dst)
-		: "memory"
-	);
+	__m128i xmm0;
+
+	xmm0 = _mm_loadu_si128((const __m128i *)src);
+	_mm_storeu_si128((__m128i *)dst, xmm0);
 }
 
+/**
+ * Copy 32 bytes from one location to another,
+ * locations should not overlap.
+ */
 static inline void
 rte_mov32(uint8_t *dst, const uint8_t *src)
 {
-	__m128i reg_a, reg_b;
-	asm volatile (
-		"movdqu (%[src]), %[reg_a]\n\t"
-		"movdqu 16(%[src]), %[reg_b]\n\t"
-		"movdqu %[reg_a], (%[dst])\n\t"
-		"movdqu %[reg_b], 16(%[dst])\n\t"
-		: [reg_a] "=x" (reg_a),
-		  [reg_b] "=x" (reg_b)
-		: [src] "r" (src),
-		  [dst] "r"(dst)
-		: "memory"
-	);
-}
+	__m256i ymm0;
 
-static inline void
-rte_mov48(uint8_t *dst, const uint8_t *src)
-{
-	__m128i reg_a, reg_b, reg_c;
-	asm volatile (
-		"movdqu (%[src]), %[reg_a]\n\t"
-		"movdqu 16(%[src]), %[reg_b]\n\t"
-		"movdqu 32(%[src]), %[reg_c]\n\t"
-		"movdqu %[reg_a], (%[dst])\n\t"
-		"movdqu %[reg_b], 16(%[dst])\n\t"
-		"movdqu %[reg_c], 32(%[dst])\n\t"
-		: [reg_a] "=x" (reg_a),
-		  [reg_b] "=x" (reg_b),
-		  [reg_c] "=x" (reg_c)
-		: [src] "r" (src),
-		  [dst] "r"(dst)
-		: "memory"
-	);
+	ymm0 = _mm256_loadu_si256((const __m256i *)src);
+	_mm256_storeu_si256((__m256i *)dst, ymm0);
 }
 
+/**
+ * Copy 64 bytes from one location to another,
+ * locations should not overlap.
+ */
 static inline void
 rte_mov64(uint8_t *dst, const uint8_t *src)
 {
-	__m128i reg_a, reg_b, reg_c, reg_d;
-	asm volatile (
-		"movdqu (%[src]), %[reg_a]\n\t"
-		"movdqu 16(%[src]), %[reg_b]\n\t"
-		"movdqu 32(%[src]), %[reg_c]\n\t"
-		"movdqu 48(%[src]), %[reg_d]\n\t"
-		"movdqu %[reg_a], (%[dst])\n\t"
-		"movdqu %[reg_b], 16(%[dst])\n\t"
-		"movdqu %[reg_c], 32(%[dst])\n\t"
-		"movdqu %[reg_d], 48(%[dst])\n\t"
-		: [reg_a] "=x" (reg_a),
-		  [reg_b] "=x" (reg_b),
-		  [reg_c] "=x" (reg_c),
-		  [reg_d] "=x" (reg_d)
-		: [src] "r" (src),
-		  [dst] "r"(dst)
-		: "memory"
-	);
+	rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32);
+	rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32);
 }
 
+/**
+ * Copy 128 bytes from one location to another,
+ * locations should not overlap.
+ */
 static inline void
 rte_mov128(uint8_t *dst, const uint8_t *src)
 {
-	__m128i reg_a, reg_b, reg_c, reg_d, reg_e, reg_f, reg_g, reg_h;
-	asm volatile (
-		"movdqu (%[src]), %[reg_a]\n\t"
-		"movdqu 16(%[src]), %[reg_b]\n\t"
-		"movdqu 32(%[src]), %[reg_c]\n\t"
-		"movdqu 48(%[src]), %[reg_d]\n\t"
-		"movdqu 64(%[src]), %[reg_e]\n\t"
-		"movdqu 80(%[src]), %[reg_f]\n\t"
-		"movdqu 96(%[src]), %[reg_g]\n\t"
-		"movdqu 112(%[src]), %[reg_h]\n\t"
-		"movdqu %[reg_a], (%[dst])\n\t"
-		"movdqu %[reg_b], 16(%[dst])\n\t"
-		"movdqu %[reg_c], 32(%[dst])\n\t"
-		"movdqu %[reg_d], 48(%[dst])\n\t"
-		"movdqu %[reg_e], 64(%[dst])\n\t"
-		"movdqu %[reg_f], 80(%[dst])\n\t"
-		"movdqu %[reg_g], 96(%[dst])\n\t"
-		"movdqu %[reg_h], 112(%[dst])\n\t"
-		: [reg_a] "=x" (reg_a),
-		  [reg_b] "=x" (reg_b),
-		  [reg_c] "=x" (reg_c),
-		  [reg_d] "=x" (reg_d),
-		  [reg_e] "=x" (reg_e),
-		  [reg_f] "=x" (reg_f),
-		  [reg_g] "=x" (reg_g),
-		  [reg_h] "=x" (reg_h)
-		: [src] "r" (src),
-		  [dst] "r"(dst)
-		: "memory"
-	);
+	rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32);
+	rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32);
+	rte_mov32((uint8_t *)dst + 2 * 32, (const uint8_t *)src + 2 * 32);
+	rte_mov32((uint8_t *)dst + 3 * 32, (const uint8_t *)src + 3 * 32);
 }
 
-#ifdef __INTEL_COMPILER
-#pragma warning(enable:593)
-#endif
-
+/**
+ * Copy 256 bytes from one location to another,
+ * locations should not overlap.
+ */
 static inline void
 rte_mov256(uint8_t *dst, const uint8_t *src)
 {
-	rte_mov128(dst, src);
-	rte_mov128(dst + 128, src + 128);
+	rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32);
+	rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32);
+	rte_mov32((uint8_t *)dst + 2 * 32, (const uint8_t *)src + 2 * 32);
+	rte_mov32((uint8_t *)dst + 3 * 32, (const uint8_t *)src + 3 * 32);
+	rte_mov32((uint8_t *)dst + 4 * 32, (const uint8_t *)src + 4 * 32);
+	rte_mov32((uint8_t *)dst + 5 * 32, (const uint8_t *)src + 5 * 32);
+	rte_mov32((uint8_t *)dst + 6 * 32, (const uint8_t *)src + 6 * 32);
+	rte_mov32((uint8_t *)dst + 7 * 32, (const uint8_t *)src + 7 * 32);
 }
 
-#define rte_memcpy(dst, src, n)              \
-	({ (__builtin_constant_p(n)) ?       \
-	memcpy((dst), (src), (n)) :          \
-	rte_memcpy_func((dst), (src), (n)); })
+/**
+ * Copy 64-byte blocks from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov64blocks(uint8_t *dst, const uint8_t *src, size_t n)
+{
+	__m256i ymm0, ymm1;
+
+	while (n >= 64) {
+		ymm0 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 0 * 32));
+		n -= 64;
+		ymm1 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 1 * 32));
+		src = (const uint8_t *)src + 64;
+		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 0 * 32), ymm0);
+		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 1 * 32), ymm1);
+		dst = (uint8_t *)dst + 64;
+	}
+}
+
+/**
+ * Copy 256-byte blocks from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov256blocks(uint8_t *dst, const uint8_t *src, size_t n)
+{
+	__m256i ymm0, ymm1, ymm2, ymm3, ymm4, ymm5, ymm6, ymm7;
+
+	while (n >= 256) {
+		ymm0 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 0 * 32));
+		n -= 256;
+		ymm1 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 1 * 32));
+		ymm2 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 2 * 32));
+		ymm3 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 3 * 32));
+		ymm4 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 4 * 32));
+		ymm5 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 5 * 32));
+		ymm6 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 6 * 32));
+		ymm7 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 7 * 32));
+		src = (const uint8_t *)src + 256;
+		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 0 * 32), ymm0);
+		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 1 * 32), ymm1);
+		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 2 * 32), ymm2);
+		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 3 * 32), ymm3);
+		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 4 * 32), ymm4);
+		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 5 * 32), ymm5);
+		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 6 * 32), ymm6);
+		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 7 * 32), ymm7);
+		dst = (uint8_t *)dst + 256;
+	}
+}
 
 static inline void *
-rte_memcpy_func(void *dst, const void *src, size_t n)
+rte_memcpy(void *dst, const void *src, size_t n)
 {
 	void *ret = dst;
+	int dstofss;
+	int bits;
 
-	/* We can't copy < 16 bytes using XMM registers so do it manually. */
+	/**
+	 * Copy less than 16 bytes
+	 */
 	if (n < 16) {
 		if (n & 0x01) {
 			*(uint8_t *)dst = *(const uint8_t *)src;
-			dst = (uint8_t *)dst + 1;
 			src = (const uint8_t *)src + 1;
+			dst = (uint8_t *)dst + 1;
 		}
 		if (n & 0x02) {
 			*(uint16_t *)dst = *(const uint16_t *)src;
-			dst = (uint16_t *)dst + 1;
 			src = (const uint16_t *)src + 1;
+			dst = (uint16_t *)dst + 1;
 		}
 		if (n & 0x04) {
 			*(uint32_t *)dst = *(const uint32_t *)src;
-			dst = (uint32_t *)dst + 1;
 			src = (const uint32_t *)src + 1;
+			dst = (uint32_t *)dst + 1;
 		}
 		if (n & 0x08) {
 			*(uint64_t *)dst = *(const uint64_t *)src;
@@ -201,95 +224,410 @@ rte_memcpy_func(void *dst, const void *src, size_t n)
 		return ret;
 	}
 
-	/* Special fast cases for <= 128 bytes */
+	/**
+	 * Fast way when copy size doesn't exceed 512 bytes
+	 */
 	if (n <= 32) {
 		rte_mov16((uint8_t *)dst, (const uint8_t *)src);
 		rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n);
 		return ret;
 	}
-
 	if (n <= 64) {
 		rte_mov32((uint8_t *)dst, (const uint8_t *)src);
 		rte_mov32((uint8_t *)dst - 32 + n, (const uint8_t *)src - 32 + n);
 		return ret;
 	}
-
-	if (n <= 128) {
-		rte_mov64((uint8_t *)dst, (const uint8_t *)src);
-		rte_mov64((uint8_t *)dst - 64 + n, (const uint8_t *)src - 64 + n);
+	if (n <= 512) {
+		if (n >= 256) {
+			n -= 256;
+			rte_mov256((uint8_t *)dst, (const uint8_t *)src);
+			src = (const uint8_t *)src + 256;
+			dst = (uint8_t *)dst + 256;
+		}
+		if (n >= 128) {
+			n -= 128;
+			rte_mov128((uint8_t *)dst, (const uint8_t *)src);
+			src = (const uint8_t *)src + 128;
+			dst = (uint8_t *)dst + 128;
+		}
+		if (n >= 64) {
+			n -= 64;
+			rte_mov64((uint8_t *)dst, (const uint8_t *)src);
+			src = (const uint8_t *)src + 64;
+			dst = (uint8_t *)dst + 64;
+		}
+COPY_BLOCK_64_BACK31:
+		if (n > 32) {
+			rte_mov32((uint8_t *)dst, (const uint8_t *)src);
+			rte_mov32((uint8_t *)dst - 32 + n, (const uint8_t *)src - 32 + n);
+			return ret;
+		}
+		if (n > 0) {
+			rte_mov32((uint8_t *)dst - 32 + n, (const uint8_t *)src - 32 + n);
+		}
 		return ret;
 	}
 
-	/*
-	 * For large copies > 128 bytes. This combination of 256, 64 and 16 byte
-	 * copies was found to be faster than doing 128 and 32 byte copies as
-	 * well.
+	/**
+	 * Make store aligned when copy size exceeds 512 bytes
 	 */
-	for ( ; n >= 256; n -= 256) {
-		rte_mov256((uint8_t *)dst, (const uint8_t *)src);
-		dst = (uint8_t *)dst + 256;
-		src = (const uint8_t *)src + 256;
+	dstofss = 32 - (int)((long long)(void *)dst & 0x1F);
+	n -= dstofss;
+	rte_mov32((uint8_t *)dst, (const uint8_t *)src);
+	src = (const uint8_t *)src + dstofss;
+	dst = (uint8_t *)dst + dstofss;
+
+	/**
+	 * Copy 256-byte blocks.
+	 * Use copy block function for better instruction order control,
+	 * which is important when load is unaligned.
+	 */
+	rte_mov256blocks((uint8_t *)dst, (const uint8_t *)src, n);
+	bits = n;
+	n = n & 255;
+	bits -= n;
+	src = (const uint8_t *)src + bits;
+	dst = (uint8_t *)dst + bits;
+
+	/**
+	 * Copy 64-byte blocks.
+	 * Use copy block function for better instruction order control,
+	 * which is important when load is unaligned.
+	 */
+	if (n >= 64) {
+		rte_mov64blocks((uint8_t *)dst, (const uint8_t *)src, n);
+		bits = n;
+		n = n & 63;
+		bits -= n;
+		src = (const uint8_t *)src + bits;
+		dst = (uint8_t *)dst + bits;
 	}
 
-	/*
-	 * We split the remaining bytes (which will be less than 256) into
-	 * 64byte (2^6) chunks.
-	 * Using incrementing integers in the case labels of a switch statement
-	 * enourages the compiler to use a jump table. To get incrementing
-	 * integers, we shift the 2 relevant bits to the LSB position to first
-	 * get decrementing integers, and then subtract.
+	/**
+	 * Copy whatever left
 	 */
-	switch (3 - (n >> 6)) {
-	case 0x00:
-		rte_mov64((uint8_t *)dst, (const uint8_t *)src);
-		n -= 64;
-		dst = (uint8_t *)dst + 64;
-		src = (const uint8_t *)src + 64;      /* fallthrough */
-	case 0x01:
-		rte_mov64((uint8_t *)dst, (const uint8_t *)src);
-		n -= 64;
-		dst = (uint8_t *)dst + 64;
-		src = (const uint8_t *)src + 64;      /* fallthrough */
-	case 0x02:
-		rte_mov64((uint8_t *)dst, (const uint8_t *)src);
-		n -= 64;
-		dst = (uint8_t *)dst + 64;
-		src = (const uint8_t *)src + 64;      /* fallthrough */
-	default:
-		;
+	goto COPY_BLOCK_64_BACK31;
+}
+
+#else /* RTE_MACHINE_CPUFLAG_AVX2 */
+
+/**
+ * SSE & AVX implementation below
+ */
+
+/**
+ * Copy 16 bytes from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov16(uint8_t *dst, const uint8_t *src)
+{
+	__m128i xmm0;
+
+	xmm0 = _mm_loadu_si128((const __m128i *)(const __m128i *)src);
+	_mm_storeu_si128((__m128i *)dst, xmm0);
+}
+
+/**
+ * Copy 32 bytes from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov32(uint8_t *dst, const uint8_t *src)
+{
+	rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
+	rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
+}
+
+/**
+ * Copy 64 bytes from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov64(uint8_t *dst, const uint8_t *src)
+{
+	rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
+	rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
+	rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16);
+	rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16);
+}
+
+/**
+ * Copy 128 bytes from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov128(uint8_t *dst, const uint8_t *src)
+{
+	rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
+	rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
+	rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16);
+	rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16);
+	rte_mov16((uint8_t *)dst + 4 * 16, (const uint8_t *)src + 4 * 16);
+	rte_mov16((uint8_t *)dst + 5 * 16, (const uint8_t *)src + 5 * 16);
+	rte_mov16((uint8_t *)dst + 6 * 16, (const uint8_t *)src + 6 * 16);
+	rte_mov16((uint8_t *)dst + 7 * 16, (const uint8_t *)src + 7 * 16);
+}
+
+/**
+ * Copy 256 bytes from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov256(uint8_t *dst, const uint8_t *src)
+{
+	rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
+	rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
+	rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16);
+	rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16);
+	rte_mov16((uint8_t *)dst + 4 * 16, (const uint8_t *)src + 4 * 16);
+	rte_mov16((uint8_t *)dst + 5 * 16, (const uint8_t *)src + 5 * 16);
+	rte_mov16((uint8_t *)dst + 6 * 16, (const uint8_t *)src + 6 * 16);
+	rte_mov16((uint8_t *)dst + 7 * 16, (const uint8_t *)src + 7 * 16);
+	rte_mov16((uint8_t *)dst + 8 * 16, (const uint8_t *)src + 8 * 16);
+	rte_mov16((uint8_t *)dst + 9 * 16, (const uint8_t *)src + 9 * 16);
+	rte_mov16((uint8_t *)dst + 10 * 16, (const uint8_t *)src + 10 * 16);
+	rte_mov16((uint8_t *)dst + 11 * 16, (const uint8_t *)src + 11 * 16);
+	rte_mov16((uint8_t *)dst + 12 * 16, (const uint8_t *)src + 12 * 16);
+	rte_mov16((uint8_t *)dst + 13 * 16, (const uint8_t *)src + 13 * 16);
+	rte_mov16((uint8_t *)dst + 14 * 16, (const uint8_t *)src + 14 * 16);
+	rte_mov16((uint8_t *)dst + 15 * 16, (const uint8_t *)src + 15 * 16);
+}
+
+/**
+ * Macro for copying unaligned block from one location to another with constant load offset,
+ * 47 bytes leftover maximum,
+ * locations should not overlap.
+ * Requirements:
+ * - Store is aligned
+ * - Load offset is <offset>, which must be immediate value within [1, 15]
+ * - For <src>, make sure <offset> bit backwards & <16 - offset> bit forwards are available for loading
+ * - <dst>, <src>, <len> must be variables
+ * - __m128i <xmm0> ~ <xmm8> must be pre-defined
+ */
+#define MOVEUNALIGNED_LEFT47_IMM(dst, src, len, offset)                                                     \
+({                                                                                                          \
+    int tmp;                                                                                                \
+    while (len >= 128 + 16 - offset) {                                                                      \
+        xmm0 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 0 * 16));                  \
+        len -= 128;                                                                                         \
+        xmm1 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 1 * 16));                  \
+        xmm2 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 2 * 16));                  \
+        xmm3 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 3 * 16));                  \
+        xmm4 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 4 * 16));                  \
+        xmm5 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 5 * 16));                  \
+        xmm6 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 6 * 16));                  \
+        xmm7 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 7 * 16));                  \
+        xmm8 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 8 * 16));                  \
+        src = (const uint8_t *)src + 128;                                                                   \
+        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16), _mm_alignr_epi8(xmm1, xmm0, offset));        \
+        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16), _mm_alignr_epi8(xmm2, xmm1, offset));        \
+        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 2 * 16), _mm_alignr_epi8(xmm3, xmm2, offset));        \
+        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 3 * 16), _mm_alignr_epi8(xmm4, xmm3, offset));        \
+        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 4 * 16), _mm_alignr_epi8(xmm5, xmm4, offset));        \
+        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 5 * 16), _mm_alignr_epi8(xmm6, xmm5, offset));        \
+        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 6 * 16), _mm_alignr_epi8(xmm7, xmm6, offset));        \
+        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 7 * 16), _mm_alignr_epi8(xmm8, xmm7, offset));        \
+        dst = (uint8_t *)dst + 128;                                                                         \
+    }                                                                                                       \
+    tmp = len;                                                                                              \
+    len = ((len - 16 + offset) & 127) + 16 - offset;                                                        \
+    tmp -= len;                                                                                             \
+    src = (const uint8_t *)src + tmp;                                                                       \
+    dst = (uint8_t *)dst + tmp;                                                                             \
+    if (len >= 32 + 16 - offset) {                                                                          \
+        while (len >= 32 + 16 - offset) {                                                                   \
+            xmm0 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 0 * 16));              \
+            len -= 32;                                                                                      \
+            xmm1 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 1 * 16));              \
+            xmm2 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 2 * 16));              \
+            src = (const uint8_t *)src + 32;                                                                \
+            _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16), _mm_alignr_epi8(xmm1, xmm0, offset));    \
+            _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16), _mm_alignr_epi8(xmm2, xmm1, offset));    \
+            dst = (uint8_t *)dst + 32;                                                                      \
+        }                                                                                                   \
+        tmp = len;                                                                                          \
+        len = ((len - 16 + offset) & 31) + 16 - offset;                                                     \
+        tmp -= len;                                                                                         \
+        src = (const uint8_t *)src + tmp;                                                                   \
+        dst = (uint8_t *)dst + tmp;                                                                         \
+    }                                                                                                       \
+})
+
+/**
+ * Macro for copying unaligned block from one location to another,
+ * 47 bytes leftover maximum,
+ * locations should not overlap.
+ * Use switch here because the aligning instruction requires immediate value for shift count.
+ * Requirements:
+ * - Store is aligned
+ * - Load offset is <offset>, which must be within [1, 15]
+ * - For <src>, make sure <offset> bit backwards & <16 - offset> bit forwards are available for loading
+ * - <dst>, <src>, <len> must be variables
+ * - __m128i <xmm0> ~ <xmm8> used in MOVEUNALIGNED_LEFT47_IMM must be pre-defined
+ */
+#define MOVEUNALIGNED_LEFT47(dst, src, len, offset)                   \
+({                                                                    \
+    switch (offset) {                                                 \
+    case 0x01: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x01); break;    \
+    case 0x02: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x02); break;    \
+    case 0x03: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x03); break;    \
+    case 0x04: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x04); break;    \
+    case 0x05: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x05); break;    \
+    case 0x06: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x06); break;    \
+    case 0x07: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x07); break;    \
+    case 0x08: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x08); break;    \
+    case 0x09: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x09); break;    \
+    case 0x0A: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0A); break;    \
+    case 0x0B: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0B); break;    \
+    case 0x0C: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0C); break;    \
+    case 0x0D: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0D); break;    \
+    case 0x0E: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0E); break;    \
+    case 0x0F: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0F); break;    \
+    default:;                                                         \
+    }                                                                 \
+})
+
+static inline void *
+rte_memcpy(void *dst, const void *src, size_t n)
+{
+	__m128i xmm0, xmm1, xmm2, xmm3, xmm4, xmm5, xmm6, xmm7, xmm8;
+	void *ret = dst;
+	int dstofss;
+	int srcofs;
+
+	/**
+	 * Copy less than 16 bytes
+	 */
+	if (n < 16) {
+		if (n & 0x01) {
+			*(uint8_t *)dst = *(const uint8_t *)src;
+			src = (const uint8_t *)src + 1;
+			dst = (uint8_t *)dst + 1;
+		}
+		if (n & 0x02) {
+			*(uint16_t *)dst = *(const uint16_t *)src;
+			src = (const uint16_t *)src + 1;
+			dst = (uint16_t *)dst + 1;
+		}
+		if (n & 0x04) {
+			*(uint32_t *)dst = *(const uint32_t *)src;
+			src = (const uint32_t *)src + 1;
+			dst = (uint32_t *)dst + 1;
+		}
+		if (n & 0x08) {
+			*(uint64_t *)dst = *(const uint64_t *)src;
+		}
+		return ret;
 	}
 
-	/*
-	 * We split the remaining bytes (which will be less than 64) into
-	 * 16byte (2^4) chunks, using the same switch structure as above.
+	/**
+	 * Fast way when copy size doesn't exceed 512 bytes
 	 */
-	switch (3 - (n >> 4)) {
-	case 0x00:
-		rte_mov16((uint8_t *)dst, (const uint8_t *)src);
-		n -= 16;
-		dst = (uint8_t *)dst + 16;
-		src = (const uint8_t *)src + 16;      /* fallthrough */
-	case 0x01:
-		rte_mov16((uint8_t *)dst, (const uint8_t *)src);
-		n -= 16;
-		dst = (uint8_t *)dst + 16;
-		src = (const uint8_t *)src + 16;      /* fallthrough */
-	case 0x02:
+	if (n <= 32) {
 		rte_mov16((uint8_t *)dst, (const uint8_t *)src);
-		n -= 16;
-		dst = (uint8_t *)dst + 16;
-		src = (const uint8_t *)src + 16;      /* fallthrough */
-	default:
-		;
+		rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n);
+		return ret;
 	}
-
-	/* Copy any remaining bytes, without going beyond end of buffers */
-	if (n != 0) {
+	if (n <= 48) {
+		rte_mov32((uint8_t *)dst, (const uint8_t *)src);
+		rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n);
+		return ret;
+	}
+	if (n <= 64) {
+		rte_mov32((uint8_t *)dst, (const uint8_t *)src);
+		rte_mov16((uint8_t *)dst + 32, (const uint8_t *)src + 32);
 		rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n);
+		return ret;
+	}
+	if (n <= 128) {
+		goto COPY_BLOCK_128_BACK15;
 	}
-	return ret;
+	if (n <= 512) {
+		if (n >= 256) {
+			n -= 256;
+			rte_mov128((uint8_t *)dst, (const uint8_t *)src);
+			rte_mov128((uint8_t *)dst + 128, (const uint8_t *)src + 128);
+			src = (const uint8_t *)src + 256;
+			dst = (uint8_t *)dst + 256;
+		}
+COPY_BLOCK_255_BACK15:
+		if (n >= 128) {
+			n -= 128;
+			rte_mov128((uint8_t *)dst, (const uint8_t *)src);
+			src = (const uint8_t *)src + 128;
+			dst = (uint8_t *)dst + 128;
+		}
+COPY_BLOCK_128_BACK15:
+		if (n >= 64) {
+			n -= 64;
+			rte_mov64((uint8_t *)dst, (const uint8_t *)src);
+			src = (const uint8_t *)src + 64;
+			dst = (uint8_t *)dst + 64;
+		}
+COPY_BLOCK_64_BACK15:
+		if (n >= 32) {
+			n -= 32;
+			rte_mov32((uint8_t *)dst, (const uint8_t *)src);
+			src = (const uint8_t *)src + 32;
+			dst = (uint8_t *)dst + 32;
+		}
+		if (n > 16) {
+			rte_mov16((uint8_t *)dst, (const uint8_t *)src);
+			rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n);
+			return ret;
+		}
+		if (n > 0) {
+			rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n);
+		}
+		return ret;
+	}
+
+	/**
+	 * Make store aligned when copy size exceeds 512 bytes,
+	 * and make sure the first 15 bytes are copied, because
+	 * unaligned copy functions require up to 15 bytes
+	 * backwards access.
+	 */
+	dstofss = 16 - (int)((long long)(void *)dst & 0x0F) + 16;
+	n -= dstofss;
+	rte_mov32((uint8_t *)dst, (const uint8_t *)src);
+	src = (const uint8_t *)src + dstofss;
+	dst = (uint8_t *)dst + dstofss;
+	srcofs = (int)((long long)(const void *)src & 0x0F);
+
+	/**
+	 * For aligned copy
+	 */
+	if (srcofs == 0) {
+		/**
+		 * Copy 256-byte blocks
+		 */
+		for (; n >= 256; n -= 256) {
+			rte_mov256((uint8_t *)dst, (const uint8_t *)src);
+			dst = (uint8_t *)dst + 256;
+			src = (const uint8_t *)src + 256;
+		}
+
+		/**
+		 * Copy whatever left
+		 */
+		goto COPY_BLOCK_255_BACK15;
+	}
+
+	/**
+	 * For copy with unaligned load
+	 */
+	MOVEUNALIGNED_LEFT47(dst, src, n, srcofs);
+
+	/**
+	 * Copy whatever left
+	 */
+	goto COPY_BLOCK_64_BACK15;
 }
 
+#endif /* RTE_MACHINE_CPUFLAG_AVX2 */
+
 #ifdef __cplusplus
 }
 #endif
-- 
1.9.3

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [dpdk-dev] [PATCH v2 0/4] DPDK memcpy optimization
  2015-01-29  2:38 [dpdk-dev] [PATCH v2 0/4] DPDK memcpy optimization Zhihong Wang
                   ` (3 preceding siblings ...)
  2015-01-29  2:38 ` [dpdk-dev] [PATCH v2 4/4] lib/librte_eal: Optimized memcpy in arch/x86/rte_memcpy.h for both SSE and AVX platforms Zhihong Wang
@ 2015-01-29  6:16 ` Fu, JingguoX
  2015-02-10  3:06 ` Liang, Cunming
  2015-02-16 15:57 ` De Lara Guarch, Pablo
  6 siblings, 0 replies; 12+ messages in thread
From: Fu, JingguoX @ 2015-01-29  6:16 UTC (permalink / raw)
  To: Wang, Zhihong, dev

Basic Information

        Patch name        DPDK memcpy optimization v2
        Brief description about test purpose    Verify memory copy and memory copy performance cases on variety OS
        Test Flag         Tested-by
        Tester name       jingguox.fu at intel.com

        Test Tool Chain information     N/A
	  Commit ID	88fa98a60b34812bfed92e5b2706fcf7e1cbcbc8
        Test Result Summary     Total 6 cases, 6 passed, 0 failed
        
Test environment

        -   Environment 1:
            OS: Ubuntu12.04 3.2.0-23-generic X86_64
            GCC: gcc version 4.6.3
            CPU: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
            NIC: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ [8086:10fb] (rev 01)

        -   Environment 2: 
            OS: Ubuntu14.04 3.13.0-24-generic
            GCC: gcc version 4.8.2
            CPU: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
            NIC: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ [8086:10fb] (rev 01)

            Environment 3:
            OS: Fedora18 3.6.10-4.fc18.x86_64
            GCC: gcc version 4.7.2 20121109
            CPU: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
            NIC: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ [8086:10fb] (rev 01)

Detailed Testing information

        Test Case - name  test_memcpy
        Test Case - Description 
                  Create two buffers, and initialise one with random values. These are copied 
                  to the second buffer and then compared to see if the copy was successful. The 
                  bytes outside the copied area are also checked to make sure they were not changed.
        Test Case -test sample/application
                  test application in app/test
        Test Case -command / instruction
                  # ./app/test/test -n 1 -c ffff
                  #RTE>> memcpy_autotest
        Test Case - expected
                  #RTE>> Test	OK
        Test Result- PASSED

        Test Case - name  test_memcpy_perf
        Test Case - Description
                  a number of different sizes and cached/uncached permutations
        Test Case -test sample/application
                  test application in app/test
        Test Case -command / instruction
                  # ./app/test/test -n 1 -c ffff
                  #RTE>> memcpy_perf_autotest
        Test Case - expected
                  #RTE>> Test	OK
        Test Result- PASSED


-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zhihong Wang
Sent: Thursday, January 29, 2015 10:39
To: dev@dpdk.org
Subject: [dpdk-dev] [PATCH v2 0/4] DPDK memcpy optimization

This patch set optimizes memcpy for DPDK for both SSE and AVX platforms.
It also extends memcpy test coverage with unaligned cases and more test points.

Optimization techniques are summarized below:

1. Utilize full cache bandwidth

2. Enforce aligned stores

3. Apply load address alignment based on architecture features

4. Make load/store address available as early as possible

5. General optimization techniques like inlining, branch reducing, prefetch pattern access

--------------
Changes in v2:

1. Reduced constant test cases in app/test/test_memcpy_perf.c for fast build

2. Modified macro definition for better code readability & safety

Zhihong Wang (4):
  app/test: Disabled VTA for memcpy test in app/test/Makefile
  app/test: Removed unnecessary test cases in app/test/test_memcpy.c
  app/test: Extended test coverage in app/test/test_memcpy_perf.c
  lib/librte_eal: Optimized memcpy in arch/x86/rte_memcpy.h for both SSE
    and AVX platforms

 app/test/Makefile                                  |   6 +
 app/test/test_memcpy.c                             |  52 +-
 app/test/test_memcpy_perf.c                        | 220 ++++---
 .../common/include/arch/x86/rte_memcpy.h           | 680 +++++++++++++++------
 4 files changed, 654 insertions(+), 304 deletions(-)

-- 
1.9.3

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [dpdk-dev] [PATCH v2 4/4] lib/librte_eal: Optimized memcpy in arch/x86/rte_memcpy.h for both SSE and AVX platforms
  2015-01-29  2:38 ` [dpdk-dev] [PATCH v2 4/4] lib/librte_eal: Optimized memcpy in arch/x86/rte_memcpy.h for both SSE and AVX platforms Zhihong Wang
@ 2015-01-29 15:17   ` Ananyev, Konstantin
  2015-01-30  5:57     ` Wang, Zhihong
  0 siblings, 1 reply; 12+ messages in thread
From: Ananyev, Konstantin @ 2015-01-29 15:17 UTC (permalink / raw)
  To: Wang, Zhihong, dev

Hi Zhihong,

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zhihong Wang
> Sent: Thursday, January 29, 2015 2:39 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] [PATCH v2 4/4] lib/librte_eal: Optimized memcpy in arch/x86/rte_memcpy.h for both SSE and AVX platforms
> 
> Main code changes:
> 
> 1. Differentiate architectural features based on CPU flags
> 
>     a. Implement separated move functions for SSE/AVX/AVX2 to make full utilization of cache bandwidth
> 
>     b. Implement separated copy flow specifically optimized for target architecture
> 
> 2. Rewrite the memcpy function "rte_memcpy"
> 
>     a. Add store aligning
> 
>     b. Add load aligning based on architectural features
> 
>     c. Put block copy loop into inline move functions for better control of instruction order
> 
>     d. Eliminate unnecessary MOVs
> 
> 3. Rewrite the inline move functions
> 
>     a. Add move functions for unaligned load cases
> 
>     b. Change instruction order in copy loops for better pipeline utilization
> 
>     c. Use intrinsics instead of assembly code
> 
> 4. Remove slow glibc call for constant copies
> 
> Signed-off-by: Zhihong Wang <zhihong.wang@intel.com>
> ---
>  .../common/include/arch/x86/rte_memcpy.h           | 680 +++++++++++++++------
>  1 file changed, 509 insertions(+), 171 deletions(-)
> 
> diff --git a/lib/librte_eal/common/include/arch/x86/rte_memcpy.h b/lib/librte_eal/common/include/arch/x86/rte_memcpy.h
> index fb9eba8..7b2d382 100644
> --- a/lib/librte_eal/common/include/arch/x86/rte_memcpy.h
> +++ b/lib/librte_eal/common/include/arch/x86/rte_memcpy.h
> @@ -34,166 +34,189 @@
>  #ifndef _RTE_MEMCPY_X86_64_H_
>  #define _RTE_MEMCPY_X86_64_H_
> 
> +/**
> + * @file
> + *
> + * Functions for SSE/AVX/AVX2 implementation of memcpy().
> + */
> +
> +#include <stdio.h>
>  #include <stdint.h>
>  #include <string.h>
> -#include <emmintrin.h>
> +#include <x86intrin.h>
> 
>  #ifdef __cplusplus
>  extern "C" {
>  #endif
> 
> -#include "generic/rte_memcpy.h"
> +/**
> + * Copy bytes from one location to another. The locations must not overlap.
> + *
> + * @note This is implemented as a macro, so it's address should not be taken
> + * and care is needed as parameter expressions may be evaluated multiple times.
> + *
> + * @param dst
> + *   Pointer to the destination of the data.
> + * @param src
> + *   Pointer to the source data.
> + * @param n
> + *   Number of bytes to copy.
> + * @return
> + *   Pointer to the destination data.
> + */
> +static inline void *
> +rte_memcpy(void *dst, const void *src, size_t n) __attribute__((always_inline));
> 
> -#ifdef __INTEL_COMPILER
> -#pragma warning(disable:593) /* Stop unused variable warning (reg_a etc). */
> -#endif
> +#ifdef RTE_MACHINE_CPUFLAG_AVX2
> 
> +/**
> + * AVX2 implementation below
> + */
> +
> +/**
> + * Copy 16 bytes from one location to another,
> + * locations should not overlap.
> + */
>  static inline void
>  rte_mov16(uint8_t *dst, const uint8_t *src)
>  {
> -	__m128i reg_a;
> -	asm volatile (
> -		"movdqu (%[src]), %[reg_a]\n\t"
> -		"movdqu %[reg_a], (%[dst])\n\t"
> -		: [reg_a] "=x" (reg_a)
> -		: [src] "r" (src),
> -		  [dst] "r"(dst)
> -		: "memory"
> -	);
> +	__m128i xmm0;
> +
> +	xmm0 = _mm_loadu_si128((const __m128i *)src);
> +	_mm_storeu_si128((__m128i *)dst, xmm0);
>  }
> 
> +/**
> + * Copy 32 bytes from one location to another,
> + * locations should not overlap.
> + */
>  static inline void
>  rte_mov32(uint8_t *dst, const uint8_t *src)
>  {
> -	__m128i reg_a, reg_b;
> -	asm volatile (
> -		"movdqu (%[src]), %[reg_a]\n\t"
> -		"movdqu 16(%[src]), %[reg_b]\n\t"
> -		"movdqu %[reg_a], (%[dst])\n\t"
> -		"movdqu %[reg_b], 16(%[dst])\n\t"
> -		: [reg_a] "=x" (reg_a),
> -		  [reg_b] "=x" (reg_b)
> -		: [src] "r" (src),
> -		  [dst] "r"(dst)
> -		: "memory"
> -	);
> -}
> +	__m256i ymm0;
> 
> -static inline void
> -rte_mov48(uint8_t *dst, const uint8_t *src)
> -{
> -	__m128i reg_a, reg_b, reg_c;
> -	asm volatile (
> -		"movdqu (%[src]), %[reg_a]\n\t"
> -		"movdqu 16(%[src]), %[reg_b]\n\t"
> -		"movdqu 32(%[src]), %[reg_c]\n\t"
> -		"movdqu %[reg_a], (%[dst])\n\t"
> -		"movdqu %[reg_b], 16(%[dst])\n\t"
> -		"movdqu %[reg_c], 32(%[dst])\n\t"
> -		: [reg_a] "=x" (reg_a),
> -		  [reg_b] "=x" (reg_b),
> -		  [reg_c] "=x" (reg_c)
> -		: [src] "r" (src),
> -		  [dst] "r"(dst)
> -		: "memory"
> -	);
> +	ymm0 = _mm256_loadu_si256((const __m256i *)src);
> +	_mm256_storeu_si256((__m256i *)dst, ymm0);
>  }
> 
> +/**
> + * Copy 64 bytes from one location to another,
> + * locations should not overlap.
> + */
>  static inline void
>  rte_mov64(uint8_t *dst, const uint8_t *src)
>  {
> -	__m128i reg_a, reg_b, reg_c, reg_d;
> -	asm volatile (
> -		"movdqu (%[src]), %[reg_a]\n\t"
> -		"movdqu 16(%[src]), %[reg_b]\n\t"
> -		"movdqu 32(%[src]), %[reg_c]\n\t"
> -		"movdqu 48(%[src]), %[reg_d]\n\t"
> -		"movdqu %[reg_a], (%[dst])\n\t"
> -		"movdqu %[reg_b], 16(%[dst])\n\t"
> -		"movdqu %[reg_c], 32(%[dst])\n\t"
> -		"movdqu %[reg_d], 48(%[dst])\n\t"
> -		: [reg_a] "=x" (reg_a),
> -		  [reg_b] "=x" (reg_b),
> -		  [reg_c] "=x" (reg_c),
> -		  [reg_d] "=x" (reg_d)
> -		: [src] "r" (src),
> -		  [dst] "r"(dst)
> -		: "memory"
> -	);
> +	rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32);
> +	rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32);
>  }
> 
> +/**
> + * Copy 128 bytes from one location to another,
> + * locations should not overlap.
> + */
>  static inline void
>  rte_mov128(uint8_t *dst, const uint8_t *src)
>  {
> -	__m128i reg_a, reg_b, reg_c, reg_d, reg_e, reg_f, reg_g, reg_h;
> -	asm volatile (
> -		"movdqu (%[src]), %[reg_a]\n\t"
> -		"movdqu 16(%[src]), %[reg_b]\n\t"
> -		"movdqu 32(%[src]), %[reg_c]\n\t"
> -		"movdqu 48(%[src]), %[reg_d]\n\t"
> -		"movdqu 64(%[src]), %[reg_e]\n\t"
> -		"movdqu 80(%[src]), %[reg_f]\n\t"
> -		"movdqu 96(%[src]), %[reg_g]\n\t"
> -		"movdqu 112(%[src]), %[reg_h]\n\t"
> -		"movdqu %[reg_a], (%[dst])\n\t"
> -		"movdqu %[reg_b], 16(%[dst])\n\t"
> -		"movdqu %[reg_c], 32(%[dst])\n\t"
> -		"movdqu %[reg_d], 48(%[dst])\n\t"
> -		"movdqu %[reg_e], 64(%[dst])\n\t"
> -		"movdqu %[reg_f], 80(%[dst])\n\t"
> -		"movdqu %[reg_g], 96(%[dst])\n\t"
> -		"movdqu %[reg_h], 112(%[dst])\n\t"
> -		: [reg_a] "=x" (reg_a),
> -		  [reg_b] "=x" (reg_b),
> -		  [reg_c] "=x" (reg_c),
> -		  [reg_d] "=x" (reg_d),
> -		  [reg_e] "=x" (reg_e),
> -		  [reg_f] "=x" (reg_f),
> -		  [reg_g] "=x" (reg_g),
> -		  [reg_h] "=x" (reg_h)
> -		: [src] "r" (src),
> -		  [dst] "r"(dst)
> -		: "memory"
> -	);
> +	rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32);
> +	rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32);
> +	rte_mov32((uint8_t *)dst + 2 * 32, (const uint8_t *)src + 2 * 32);
> +	rte_mov32((uint8_t *)dst + 3 * 32, (const uint8_t *)src + 3 * 32);
>  }
> 
> -#ifdef __INTEL_COMPILER
> -#pragma warning(enable:593)
> -#endif
> -
> +/**
> + * Copy 256 bytes from one location to another,
> + * locations should not overlap.
> + */
>  static inline void
>  rte_mov256(uint8_t *dst, const uint8_t *src)
>  {
> -	rte_mov128(dst, src);
> -	rte_mov128(dst + 128, src + 128);
> +	rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32);
> +	rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32);
> +	rte_mov32((uint8_t *)dst + 2 * 32, (const uint8_t *)src + 2 * 32);
> +	rte_mov32((uint8_t *)dst + 3 * 32, (const uint8_t *)src + 3 * 32);
> +	rte_mov32((uint8_t *)dst + 4 * 32, (const uint8_t *)src + 4 * 32);
> +	rte_mov32((uint8_t *)dst + 5 * 32, (const uint8_t *)src + 5 * 32);
> +	rte_mov32((uint8_t *)dst + 6 * 32, (const uint8_t *)src + 6 * 32);
> +	rte_mov32((uint8_t *)dst + 7 * 32, (const uint8_t *)src + 7 * 32);
>  }
> 
> -#define rte_memcpy(dst, src, n)              \
> -	({ (__builtin_constant_p(n)) ?       \
> -	memcpy((dst), (src), (n)) :          \
> -	rte_memcpy_func((dst), (src), (n)); })
> +/**
> + * Copy 64-byte blocks from one location to another,
> + * locations should not overlap.
> + */
> +static inline void
> +rte_mov64blocks(uint8_t *dst, const uint8_t *src, size_t n)
> +{
> +	__m256i ymm0, ymm1;
> +
> +	while (n >= 64) {
> +		ymm0 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 0 * 32));
> +		n -= 64;
> +		ymm1 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 1 * 32));
> +		src = (const uint8_t *)src + 64;
> +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 0 * 32), ymm0);
> +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 1 * 32), ymm1);
> +		dst = (uint8_t *)dst + 64;
> +	}
> +}
> +
> +/**
> + * Copy 256-byte blocks from one location to another,
> + * locations should not overlap.
> + */
> +static inline void
> +rte_mov256blocks(uint8_t *dst, const uint8_t *src, size_t n)
> +{
> +	__m256i ymm0, ymm1, ymm2, ymm3, ymm4, ymm5, ymm6, ymm7;
> +
> +	while (n >= 256) {
> +		ymm0 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 0 * 32));
> +		n -= 256;
> +		ymm1 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 1 * 32));
> +		ymm2 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 2 * 32));
> +		ymm3 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 3 * 32));
> +		ymm4 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 4 * 32));
> +		ymm5 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 5 * 32));
> +		ymm6 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 6 * 32));
> +		ymm7 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 7 * 32));
> +		src = (const uint8_t *)src + 256;
> +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 0 * 32), ymm0);
> +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 1 * 32), ymm1);
> +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 2 * 32), ymm2);
> +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 3 * 32), ymm3);
> +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 4 * 32), ymm4);
> +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 5 * 32), ymm5);
> +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 6 * 32), ymm6);
> +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 7 * 32), ymm7);
> +		dst = (uint8_t *)dst + 256;
> +	}
> +}
> 
>  static inline void *
> -rte_memcpy_func(void *dst, const void *src, size_t n)
> +rte_memcpy(void *dst, const void *src, size_t n)
>  {
>  	void *ret = dst;
> +	int dstofss;
> +	int bits;
> 
> -	/* We can't copy < 16 bytes using XMM registers so do it manually. */
> +	/**
> +	 * Copy less than 16 bytes
> +	 */
>  	if (n < 16) {
>  		if (n & 0x01) {
>  			*(uint8_t *)dst = *(const uint8_t *)src;
> -			dst = (uint8_t *)dst + 1;
>  			src = (const uint8_t *)src + 1;
> +			dst = (uint8_t *)dst + 1;
>  		}
>  		if (n & 0x02) {
>  			*(uint16_t *)dst = *(const uint16_t *)src;
> -			dst = (uint16_t *)dst + 1;
>  			src = (const uint16_t *)src + 1;
> +			dst = (uint16_t *)dst + 1;
>  		}
>  		if (n & 0x04) {
>  			*(uint32_t *)dst = *(const uint32_t *)src;
> -			dst = (uint32_t *)dst + 1;
>  			src = (const uint32_t *)src + 1;
> +			dst = (uint32_t *)dst + 1;
>  		}
>  		if (n & 0x08) {
>  			*(uint64_t *)dst = *(const uint64_t *)src;
> @@ -201,95 +224,410 @@ rte_memcpy_func(void *dst, const void *src, size_t n)
>  		return ret;
>  	}
> 
> -	/* Special fast cases for <= 128 bytes */
> +	/**
> +	 * Fast way when copy size doesn't exceed 512 bytes
> +	 */
>  	if (n <= 32) {
>  		rte_mov16((uint8_t *)dst, (const uint8_t *)src);
>  		rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n);
>  		return ret;
>  	}
> -
>  	if (n <= 64) {
>  		rte_mov32((uint8_t *)dst, (const uint8_t *)src);
>  		rte_mov32((uint8_t *)dst - 32 + n, (const uint8_t *)src - 32 + n);
>  		return ret;
>  	}
> -
> -	if (n <= 128) {
> -		rte_mov64((uint8_t *)dst, (const uint8_t *)src);
> -		rte_mov64((uint8_t *)dst - 64 + n, (const uint8_t *)src - 64 + n);
> +	if (n <= 512) {
> +		if (n >= 256) {
> +			n -= 256;
> +			rte_mov256((uint8_t *)dst, (const uint8_t *)src);
> +			src = (const uint8_t *)src + 256;
> +			dst = (uint8_t *)dst + 256;
> +		}
> +		if (n >= 128) {
> +			n -= 128;
> +			rte_mov128((uint8_t *)dst, (const uint8_t *)src);
> +			src = (const uint8_t *)src + 128;
> +			dst = (uint8_t *)dst + 128;
> +		}
> +		if (n >= 64) {
> +			n -= 64;
> +			rte_mov64((uint8_t *)dst, (const uint8_t *)src);
> +			src = (const uint8_t *)src + 64;
> +			dst = (uint8_t *)dst + 64;
> +		}
> +COPY_BLOCK_64_BACK31:
> +		if (n > 32) {
> +			rte_mov32((uint8_t *)dst, (const uint8_t *)src);
> +			rte_mov32((uint8_t *)dst - 32 + n, (const uint8_t *)src - 32 + n);
> +			return ret;
> +		}
> +		if (n > 0) {
> +			rte_mov32((uint8_t *)dst - 32 + n, (const uint8_t *)src - 32 + n);
> +		}
>  		return ret;
>  	}
> 
> -	/*
> -	 * For large copies > 128 bytes. This combination of 256, 64 and 16 byte
> -	 * copies was found to be faster than doing 128 and 32 byte copies as
> -	 * well.
> +	/**
> +	 * Make store aligned when copy size exceeds 512 bytes
>  	 */
> -	for ( ; n >= 256; n -= 256) {
> -		rte_mov256((uint8_t *)dst, (const uint8_t *)src);
> -		dst = (uint8_t *)dst + 256;
> -		src = (const uint8_t *)src + 256;
> +	dstofss = 32 - (int)((long long)(void *)dst & 0x1F);
> +	n -= dstofss;
> +	rte_mov32((uint8_t *)dst, (const uint8_t *)src);
> +	src = (const uint8_t *)src + dstofss;
> +	dst = (uint8_t *)dst + dstofss;
> +
> +	/**
> +	 * Copy 256-byte blocks.
> +	 * Use copy block function for better instruction order control,
> +	 * which is important when load is unaligned.
> +	 */
> +	rte_mov256blocks((uint8_t *)dst, (const uint8_t *)src, n);
> +	bits = n;
> +	n = n & 255;
> +	bits -= n;
> +	src = (const uint8_t *)src + bits;
> +	dst = (uint8_t *)dst + bits;
> +
> +	/**
> +	 * Copy 64-byte blocks.
> +	 * Use copy block function for better instruction order control,
> +	 * which is important when load is unaligned.
> +	 */
> +	if (n >= 64) {
> +		rte_mov64blocks((uint8_t *)dst, (const uint8_t *)src, n);
> +		bits = n;
> +		n = n & 63;
> +		bits -= n;
> +		src = (const uint8_t *)src + bits;
> +		dst = (uint8_t *)dst + bits;
>  	}
> 
> -	/*
> -	 * We split the remaining bytes (which will be less than 256) into
> -	 * 64byte (2^6) chunks.
> -	 * Using incrementing integers in the case labels of a switch statement
> -	 * enourages the compiler to use a jump table. To get incrementing
> -	 * integers, we shift the 2 relevant bits to the LSB position to first
> -	 * get decrementing integers, and then subtract.
> +	/**
> +	 * Copy whatever left
>  	 */
> -	switch (3 - (n >> 6)) {
> -	case 0x00:
> -		rte_mov64((uint8_t *)dst, (const uint8_t *)src);
> -		n -= 64;
> -		dst = (uint8_t *)dst + 64;
> -		src = (const uint8_t *)src + 64;      /* fallthrough */
> -	case 0x01:
> -		rte_mov64((uint8_t *)dst, (const uint8_t *)src);
> -		n -= 64;
> -		dst = (uint8_t *)dst + 64;
> -		src = (const uint8_t *)src + 64;      /* fallthrough */
> -	case 0x02:
> -		rte_mov64((uint8_t *)dst, (const uint8_t *)src);
> -		n -= 64;
> -		dst = (uint8_t *)dst + 64;
> -		src = (const uint8_t *)src + 64;      /* fallthrough */
> -	default:
> -		;
> +	goto COPY_BLOCK_64_BACK31;
> +}
> +
> +#else /* RTE_MACHINE_CPUFLAG_AVX2 */
> +
> +/**
> + * SSE & AVX implementation below
> + */
> +
> +/**
> + * Copy 16 bytes from one location to another,
> + * locations should not overlap.
> + */
> +static inline void
> +rte_mov16(uint8_t *dst, const uint8_t *src)
> +{
> +	__m128i xmm0;
> +
> +	xmm0 = _mm_loadu_si128((const __m128i *)(const __m128i *)src);
> +	_mm_storeu_si128((__m128i *)dst, xmm0);
> +}
> +
> +/**
> + * Copy 32 bytes from one location to another,
> + * locations should not overlap.
> + */
> +static inline void
> +rte_mov32(uint8_t *dst, const uint8_t *src)
> +{
> +	rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
> +	rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
> +}
> +
> +/**
> + * Copy 64 bytes from one location to another,
> + * locations should not overlap.
> + */
> +static inline void
> +rte_mov64(uint8_t *dst, const uint8_t *src)
> +{
> +	rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
> +	rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
> +	rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16);
> +	rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16);
> +}
> +
> +/**
> + * Copy 128 bytes from one location to another,
> + * locations should not overlap.
> + */
> +static inline void
> +rte_mov128(uint8_t *dst, const uint8_t *src)
> +{
> +	rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
> +	rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
> +	rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16);
> +	rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16);
> +	rte_mov16((uint8_t *)dst + 4 * 16, (const uint8_t *)src + 4 * 16);
> +	rte_mov16((uint8_t *)dst + 5 * 16, (const uint8_t *)src + 5 * 16);
> +	rte_mov16((uint8_t *)dst + 6 * 16, (const uint8_t *)src + 6 * 16);
> +	rte_mov16((uint8_t *)dst + 7 * 16, (const uint8_t *)src + 7 * 16);
> +}
> +
> +/**
> + * Copy 256 bytes from one location to another,
> + * locations should not overlap.
> + */
> +static inline void
> +rte_mov256(uint8_t *dst, const uint8_t *src)
> +{
> +	rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
> +	rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
> +	rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16);
> +	rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16);
> +	rte_mov16((uint8_t *)dst + 4 * 16, (const uint8_t *)src + 4 * 16);
> +	rte_mov16((uint8_t *)dst + 5 * 16, (const uint8_t *)src + 5 * 16);
> +	rte_mov16((uint8_t *)dst + 6 * 16, (const uint8_t *)src + 6 * 16);
> +	rte_mov16((uint8_t *)dst + 7 * 16, (const uint8_t *)src + 7 * 16);
> +	rte_mov16((uint8_t *)dst + 8 * 16, (const uint8_t *)src + 8 * 16);
> +	rte_mov16((uint8_t *)dst + 9 * 16, (const uint8_t *)src + 9 * 16);
> +	rte_mov16((uint8_t *)dst + 10 * 16, (const uint8_t *)src + 10 * 16);
> +	rte_mov16((uint8_t *)dst + 11 * 16, (const uint8_t *)src + 11 * 16);
> +	rte_mov16((uint8_t *)dst + 12 * 16, (const uint8_t *)src + 12 * 16);
> +	rte_mov16((uint8_t *)dst + 13 * 16, (const uint8_t *)src + 13 * 16);
> +	rte_mov16((uint8_t *)dst + 14 * 16, (const uint8_t *)src + 14 * 16);
> +	rte_mov16((uint8_t *)dst + 15 * 16, (const uint8_t *)src + 15 * 16);
> +}
> +
> +/**
> + * Macro for copying unaligned block from one location to another with constant load offset,
> + * 47 bytes leftover maximum,
> + * locations should not overlap.
> + * Requirements:
> + * - Store is aligned
> + * - Load offset is <offset>, which must be immediate value within [1, 15]
> + * - For <src>, make sure <offset> bit backwards & <16 - offset> bit forwards are available for loading
> + * - <dst>, <src>, <len> must be variables
> + * - __m128i <xmm0> ~ <xmm8> must be pre-defined
> + */
> +#define MOVEUNALIGNED_LEFT47_IMM(dst, src, len, offset)                                                     \
> +({                                                                                                          \
> +    int tmp;                                                                                                \
> +    while (len >= 128 + 16 - offset) {                                                                      \
> +        xmm0 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 0 * 16));                  \
> +        len -= 128;                                                                                         \
> +        xmm1 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 1 * 16));                  \
> +        xmm2 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 2 * 16));                  \
> +        xmm3 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 3 * 16));                  \
> +        xmm4 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 4 * 16));                  \
> +        xmm5 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 5 * 16));                  \
> +        xmm6 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 6 * 16));                  \
> +        xmm7 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 7 * 16));                  \
> +        xmm8 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 8 * 16));                  \
> +        src = (const uint8_t *)src + 128;                                                                   \
> +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16), _mm_alignr_epi8(xmm1, xmm0, offset));        \
> +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16), _mm_alignr_epi8(xmm2, xmm1, offset));        \
> +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 2 * 16), _mm_alignr_epi8(xmm3, xmm2, offset));        \
> +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 3 * 16), _mm_alignr_epi8(xmm4, xmm3, offset));        \
> +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 4 * 16), _mm_alignr_epi8(xmm5, xmm4, offset));        \
> +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 5 * 16), _mm_alignr_epi8(xmm6, xmm5, offset));        \
> +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 6 * 16), _mm_alignr_epi8(xmm7, xmm6, offset));        \
> +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 7 * 16), _mm_alignr_epi8(xmm8, xmm7, offset));        \
> +        dst = (uint8_t *)dst + 128;                                                                         \
> +    }                                                                                                       \
> +    tmp = len;                                                                                              \
> +    len = ((len - 16 + offset) & 127) + 16 - offset;                                                        \
> +    tmp -= len;                                                                                             \
> +    src = (const uint8_t *)src + tmp;                                                                       \
> +    dst = (uint8_t *)dst + tmp;                                                                             \
> +    if (len >= 32 + 16 - offset) {                                                                          \
> +        while (len >= 32 + 16 - offset) {                                                                   \
> +            xmm0 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 0 * 16));              \
> +            len -= 32;                                                                                      \
> +            xmm1 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 1 * 16));              \
> +            xmm2 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 2 * 16));              \
> +            src = (const uint8_t *)src + 32;                                                                \
> +            _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16), _mm_alignr_epi8(xmm1, xmm0, offset));    \
> +            _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16), _mm_alignr_epi8(xmm2, xmm1, offset));    \
> +            dst = (uint8_t *)dst + 32;                                                                      \
> +        }                                                                                                   \
> +        tmp = len;                                                                                          \
> +        len = ((len - 16 + offset) & 31) + 16 - offset;                                                     \
> +        tmp -= len;                                                                                         \
> +        src = (const uint8_t *)src + tmp;                                                                   \
> +        dst = (uint8_t *)dst + tmp;                                                                         \
> +    }                                                                                                       \
> +})
> +
> +/**
> + * Macro for copying unaligned block from one location to another,
> + * 47 bytes leftover maximum,
> + * locations should not overlap.
> + * Use switch here because the aligning instruction requires immediate value for shift count.
> + * Requirements:
> + * - Store is aligned
> + * - Load offset is <offset>, which must be within [1, 15]
> + * - For <src>, make sure <offset> bit backwards & <16 - offset> bit forwards are available for loading
> + * - <dst>, <src>, <len> must be variables
> + * - __m128i <xmm0> ~ <xmm8> used in MOVEUNALIGNED_LEFT47_IMM must be pre-defined
> + */
> +#define MOVEUNALIGNED_LEFT47(dst, src, len, offset)                   \
> +({                                                                    \
> +    switch (offset) {                                                 \
> +    case 0x01: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x01); break;    \
> +    case 0x02: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x02); break;    \
> +    case 0x03: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x03); break;    \
> +    case 0x04: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x04); break;    \
> +    case 0x05: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x05); break;    \
> +    case 0x06: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x06); break;    \
> +    case 0x07: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x07); break;    \
> +    case 0x08: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x08); break;    \
> +    case 0x09: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x09); break;    \
> +    case 0x0A: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0A); break;    \
> +    case 0x0B: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0B); break;    \
> +    case 0x0C: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0C); break;    \
> +    case 0x0D: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0D); break;    \
> +    case 0x0E: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0E); break;    \
> +    case 0x0F: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0F); break;    \
> +    default:;                                                         \
> +    }                                                                 \
> +})

We probably didn't understand each other properly.
My thought was to do something like:

#define MOVEUNALIGNED_128(offset)	do { \
 _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16), _mm_alignr_epi8(xmm1, xmm0, offset));        \
 _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16), _mm_alignr_epi8(xmm2, xmm1, offset));        \
 _mm_storeu_si128((__m128i *)((uint8_t *)dst + 2 * 16), _mm_alignr_epi8(xmm3, xmm2, offset));        \
 _mm_storeu_si128((__m128i *)((uint8_t *)dst + 3 * 16), _mm_alignr_epi8(xmm4, xmm3, offset));        \
 _mm_storeu_si128((__m128i *)((uint8_t *)dst + 4 * 16), _mm_alignr_epi8(xmm5, xmm4, offset));        \
 _mm_storeu_si128((__m128i *)((uint8_t *)dst + 5 * 16), _mm_alignr_epi8(xmm6, xmm5, offset));        \
 _mm_storeu_si128((__m128i *)((uint8_t *)dst + 6 * 16), _mm_alignr_epi8(xmm7, xmm6, offset));        \
 _mm_storeu_si128((__m128i *)((uint8_t *)dst + 7 * 16), _mm_alignr_epi8(xmm8, xmm7, offset));       \    
} while(0);


Then at MOVEUNALIGNED_LEFT47_IMM:
....
xmm7 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 7 * 16));                  \
xmm8 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 8 * 16));                  \
src = (const uint8_t *)src + 128;

switch(offset) {
    case 0x01: MOVEUNALIGNED_128(1); break;  
    ...
    case 0x0f: MOVEUNALIGNED_128(f); break;
}
...
dst = (uint8_t *)dst + 128

An then in rte_memcpy() you don't need a switch for MOVEUNALIGNED_LEFT47_IMM itself.
Thought it might help to generate smaller code for rte_ememcpy() while keeping performance reasonably high.

Konstantin


> +
> +static inline void *
> +rte_memcpy(void *dst, const void *src, size_t n)
> +{
> +	__m128i xmm0, xmm1, xmm2, xmm3, xmm4, xmm5, xmm6, xmm7, xmm8;
> +	void *ret = dst;
> +	int dstofss;
> +	int srcofs;
> +
> +	/**
> +	 * Copy less than 16 bytes
> +	 */
> +	if (n < 16) {
> +		if (n & 0x01) {
> +			*(uint8_t *)dst = *(const uint8_t *)src;
> +			src = (const uint8_t *)src + 1;
> +			dst = (uint8_t *)dst + 1;
> +		}
> +		if (n & 0x02) {
> +			*(uint16_t *)dst = *(const uint16_t *)src;
> +			src = (const uint16_t *)src + 1;
> +			dst = (uint16_t *)dst + 1;
> +		}
> +		if (n & 0x04) {
> +			*(uint32_t *)dst = *(const uint32_t *)src;
> +			src = (const uint32_t *)src + 1;
> +			dst = (uint32_t *)dst + 1;
> +		}
> +		if (n & 0x08) {
> +			*(uint64_t *)dst = *(const uint64_t *)src;
> +		}
> +		return ret;
>  	}
> 
> -	/*
> -	 * We split the remaining bytes (which will be less than 64) into
> -	 * 16byte (2^4) chunks, using the same switch structure as above.
> +	/**
> +	 * Fast way when copy size doesn't exceed 512 bytes
>  	 */
> -	switch (3 - (n >> 4)) {
> -	case 0x00:
> -		rte_mov16((uint8_t *)dst, (const uint8_t *)src);
> -		n -= 16;
> -		dst = (uint8_t *)dst + 16;
> -		src = (const uint8_t *)src + 16;      /* fallthrough */
> -	case 0x01:
> -		rte_mov16((uint8_t *)dst, (const uint8_t *)src);
> -		n -= 16;
> -		dst = (uint8_t *)dst + 16;
> -		src = (const uint8_t *)src + 16;      /* fallthrough */
> -	case 0x02:
> +	if (n <= 32) {
>  		rte_mov16((uint8_t *)dst, (const uint8_t *)src);
> -		n -= 16;
> -		dst = (uint8_t *)dst + 16;
> -		src = (const uint8_t *)src + 16;      /* fallthrough */
> -	default:
> -		;
> +		rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n);
> +		return ret;
>  	}
> -
> -	/* Copy any remaining bytes, without going beyond end of buffers */
> -	if (n != 0) {
> +	if (n <= 48) {
> +		rte_mov32((uint8_t *)dst, (const uint8_t *)src);
> +		rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n);
> +		return ret;
> +	}
> +	if (n <= 64) {
> +		rte_mov32((uint8_t *)dst, (const uint8_t *)src);
> +		rte_mov16((uint8_t *)dst + 32, (const uint8_t *)src + 32);
>  		rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n);
> +		return ret;
> +	}
> +	if (n <= 128) {
> +		goto COPY_BLOCK_128_BACK15;
>  	}
> -	return ret;
> +	if (n <= 512) {
> +		if (n >= 256) {
> +			n -= 256;
> +			rte_mov128((uint8_t *)dst, (const uint8_t *)src);
> +			rte_mov128((uint8_t *)dst + 128, (const uint8_t *)src + 128);
> +			src = (const uint8_t *)src + 256;
> +			dst = (uint8_t *)dst + 256;
> +		}
> +COPY_BLOCK_255_BACK15:
> +		if (n >= 128) {
> +			n -= 128;
> +			rte_mov128((uint8_t *)dst, (const uint8_t *)src);
> +			src = (const uint8_t *)src + 128;
> +			dst = (uint8_t *)dst + 128;
> +		}
> +COPY_BLOCK_128_BACK15:
> +		if (n >= 64) {
> +			n -= 64;
> +			rte_mov64((uint8_t *)dst, (const uint8_t *)src);
> +			src = (const uint8_t *)src + 64;
> +			dst = (uint8_t *)dst + 64;
> +		}
> +COPY_BLOCK_64_BACK15:
> +		if (n >= 32) {
> +			n -= 32;
> +			rte_mov32((uint8_t *)dst, (const uint8_t *)src);
> +			src = (const uint8_t *)src + 32;
> +			dst = (uint8_t *)dst + 32;
> +		}
> +		if (n > 16) {
> +			rte_mov16((uint8_t *)dst, (const uint8_t *)src);
> +			rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n);
> +			return ret;
> +		}
> +		if (n > 0) {
> +			rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n);
> +		}
> +		return ret;
> +	}
> +
> +	/**
> +	 * Make store aligned when copy size exceeds 512 bytes,
> +	 * and make sure the first 15 bytes are copied, because
> +	 * unaligned copy functions require up to 15 bytes
> +	 * backwards access.
> +	 */
> +	dstofss = 16 - (int)((long long)(void *)dst & 0x0F) + 16;
> +	n -= dstofss;
> +	rte_mov32((uint8_t *)dst, (const uint8_t *)src);
> +	src = (const uint8_t *)src + dstofss;
> +	dst = (uint8_t *)dst + dstofss;
> +	srcofs = (int)((long long)(const void *)src & 0x0F);
> +
> +	/**
> +	 * For aligned copy
> +	 */
> +	if (srcofs == 0) {
> +		/**
> +		 * Copy 256-byte blocks
> +		 */
> +		for (; n >= 256; n -= 256) {
> +			rte_mov256((uint8_t *)dst, (const uint8_t *)src);
> +			dst = (uint8_t *)dst + 256;
> +			src = (const uint8_t *)src + 256;
> +		}
> +
> +		/**
> +		 * Copy whatever left
> +		 */
> +		goto COPY_BLOCK_255_BACK15;
> +	}
> +
> +	/**
> +	 * For copy with unaligned load
> +	 */
> +	MOVEUNALIGNED_LEFT47(dst, src, n, srcofs);
> +
> +	/**
> +	 * Copy whatever left
> +	 */
> +	goto COPY_BLOCK_64_BACK15;
>  }
> 
> +#endif /* RTE_MACHINE_CPUFLAG_AVX2 */
> +
>  #ifdef __cplusplus
>  }
>  #endif
> --
> 1.9.3

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [dpdk-dev] [PATCH v2 4/4] lib/librte_eal: Optimized memcpy in arch/x86/rte_memcpy.h for both SSE and AVX platforms
  2015-01-29 15:17   ` Ananyev, Konstantin
@ 2015-01-30  5:57     ` Wang, Zhihong
  2015-01-30 10:44       ` Ananyev, Konstantin
  0 siblings, 1 reply; 12+ messages in thread
From: Wang, Zhihong @ 2015-01-30  5:57 UTC (permalink / raw)
  To: Ananyev, Konstantin, dev

Hey Konstantin,

This method does reduce code size but lead to significant performance drop.
I think we need to keep the original code.


Thanks
Zhihong (John)


> -----Original Message-----
> From: Ananyev, Konstantin
> Sent: Thursday, January 29, 2015 11:18 PM
> To: Wang, Zhihong; dev@dpdk.org
> Subject: RE: [dpdk-dev] [PATCH v2 4/4] lib/librte_eal: Optimized memcpy in
> arch/x86/rte_memcpy.h for both SSE and AVX platforms
> 
> Hi Zhihong,
> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zhihong Wang
> > Sent: Thursday, January 29, 2015 2:39 AM
> > To: dev@dpdk.org
> > Subject: [dpdk-dev] [PATCH v2 4/4] lib/librte_eal: Optimized memcpy in
> > arch/x86/rte_memcpy.h for both SSE and AVX platforms
> >
> > Main code changes:
> >
> > 1. Differentiate architectural features based on CPU flags
> >
> >     a. Implement separated move functions for SSE/AVX/AVX2 to make
> > full utilization of cache bandwidth
> >
> >     b. Implement separated copy flow specifically optimized for target
> > architecture
> >
> > 2. Rewrite the memcpy function "rte_memcpy"
> >
> >     a. Add store aligning
> >
> >     b. Add load aligning based on architectural features
> >
> >     c. Put block copy loop into inline move functions for better
> > control of instruction order
> >
> >     d. Eliminate unnecessary MOVs
> >
> > 3. Rewrite the inline move functions
> >
> >     a. Add move functions for unaligned load cases
> >
> >     b. Change instruction order in copy loops for better pipeline
> > utilization
> >
> >     c. Use intrinsics instead of assembly code
> >
> > 4. Remove slow glibc call for constant copies
> >
> > Signed-off-by: Zhihong Wang <zhihong.wang@intel.com>
> > ---
> >  .../common/include/arch/x86/rte_memcpy.h           | 680
> +++++++++++++++------
> >  1 file changed, 509 insertions(+), 171 deletions(-)
> >
> > diff --git a/lib/librte_eal/common/include/arch/x86/rte_memcpy.h
> > b/lib/librte_eal/common/include/arch/x86/rte_memcpy.h
> > index fb9eba8..7b2d382 100644
> > --- a/lib/librte_eal/common/include/arch/x86/rte_memcpy.h
> > +++ b/lib/librte_eal/common/include/arch/x86/rte_memcpy.h
> > @@ -34,166 +34,189 @@
> >  #ifndef _RTE_MEMCPY_X86_64_H_
> >  #define _RTE_MEMCPY_X86_64_H_
> >
> > +/**
> > + * @file
> > + *
> > + * Functions for SSE/AVX/AVX2 implementation of memcpy().
> > + */
> > +
> > +#include <stdio.h>
> >  #include <stdint.h>
> >  #include <string.h>
> > -#include <emmintrin.h>
> > +#include <x86intrin.h>
> >
> >  #ifdef __cplusplus
> >  extern "C" {
> >  #endif
> >
> > -#include "generic/rte_memcpy.h"
> > +/**
> > + * Copy bytes from one location to another. The locations must not
> overlap.
> > + *
> > + * @note This is implemented as a macro, so it's address should not
> > +be taken
> > + * and care is needed as parameter expressions may be evaluated
> multiple times.
> > + *
> > + * @param dst
> > + *   Pointer to the destination of the data.
> > + * @param src
> > + *   Pointer to the source data.
> > + * @param n
> > + *   Number of bytes to copy.
> > + * @return
> > + *   Pointer to the destination data.
> > + */
> > +static inline void *
> > +rte_memcpy(void *dst, const void *src, size_t n)
> > +__attribute__((always_inline));
> >
> > -#ifdef __INTEL_COMPILER
> > -#pragma warning(disable:593) /* Stop unused variable warning (reg_a
> > etc). */ -#endif
> > +#ifdef RTE_MACHINE_CPUFLAG_AVX2
> >
> > +/**
> > + * AVX2 implementation below
> > + */
> > +
> > +/**
> > + * Copy 16 bytes from one location to another,
> > + * locations should not overlap.
> > + */
> >  static inline void
> >  rte_mov16(uint8_t *dst, const uint8_t *src)  {
> > -	__m128i reg_a;
> > -	asm volatile (
> > -		"movdqu (%[src]), %[reg_a]\n\t"
> > -		"movdqu %[reg_a], (%[dst])\n\t"
> > -		: [reg_a] "=x" (reg_a)
> > -		: [src] "r" (src),
> > -		  [dst] "r"(dst)
> > -		: "memory"
> > -	);
> > +	__m128i xmm0;
> > +
> > +	xmm0 = _mm_loadu_si128((const __m128i *)src);
> > +	_mm_storeu_si128((__m128i *)dst, xmm0);
> >  }
> >
> > +/**
> > + * Copy 32 bytes from one location to another,
> > + * locations should not overlap.
> > + */
> >  static inline void
> >  rte_mov32(uint8_t *dst, const uint8_t *src)  {
> > -	__m128i reg_a, reg_b;
> > -	asm volatile (
> > -		"movdqu (%[src]), %[reg_a]\n\t"
> > -		"movdqu 16(%[src]), %[reg_b]\n\t"
> > -		"movdqu %[reg_a], (%[dst])\n\t"
> > -		"movdqu %[reg_b], 16(%[dst])\n\t"
> > -		: [reg_a] "=x" (reg_a),
> > -		  [reg_b] "=x" (reg_b)
> > -		: [src] "r" (src),
> > -		  [dst] "r"(dst)
> > -		: "memory"
> > -	);
> > -}
> > +	__m256i ymm0;
> >
> > -static inline void
> > -rte_mov48(uint8_t *dst, const uint8_t *src) -{
> > -	__m128i reg_a, reg_b, reg_c;
> > -	asm volatile (
> > -		"movdqu (%[src]), %[reg_a]\n\t"
> > -		"movdqu 16(%[src]), %[reg_b]\n\t"
> > -		"movdqu 32(%[src]), %[reg_c]\n\t"
> > -		"movdqu %[reg_a], (%[dst])\n\t"
> > -		"movdqu %[reg_b], 16(%[dst])\n\t"
> > -		"movdqu %[reg_c], 32(%[dst])\n\t"
> > -		: [reg_a] "=x" (reg_a),
> > -		  [reg_b] "=x" (reg_b),
> > -		  [reg_c] "=x" (reg_c)
> > -		: [src] "r" (src),
> > -		  [dst] "r"(dst)
> > -		: "memory"
> > -	);
> > +	ymm0 = _mm256_loadu_si256((const __m256i *)src);
> > +	_mm256_storeu_si256((__m256i *)dst, ymm0);
> >  }
> >
> > +/**
> > + * Copy 64 bytes from one location to another,
> > + * locations should not overlap.
> > + */
> >  static inline void
> >  rte_mov64(uint8_t *dst, const uint8_t *src)  {
> > -	__m128i reg_a, reg_b, reg_c, reg_d;
> > -	asm volatile (
> > -		"movdqu (%[src]), %[reg_a]\n\t"
> > -		"movdqu 16(%[src]), %[reg_b]\n\t"
> > -		"movdqu 32(%[src]), %[reg_c]\n\t"
> > -		"movdqu 48(%[src]), %[reg_d]\n\t"
> > -		"movdqu %[reg_a], (%[dst])\n\t"
> > -		"movdqu %[reg_b], 16(%[dst])\n\t"
> > -		"movdqu %[reg_c], 32(%[dst])\n\t"
> > -		"movdqu %[reg_d], 48(%[dst])\n\t"
> > -		: [reg_a] "=x" (reg_a),
> > -		  [reg_b] "=x" (reg_b),
> > -		  [reg_c] "=x" (reg_c),
> > -		  [reg_d] "=x" (reg_d)
> > -		: [src] "r" (src),
> > -		  [dst] "r"(dst)
> > -		: "memory"
> > -	);
> > +	rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32);
> > +	rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32);
> >  }
> >
> > +/**
> > + * Copy 128 bytes from one location to another,
> > + * locations should not overlap.
> > + */
> >  static inline void
> >  rte_mov128(uint8_t *dst, const uint8_t *src)  {
> > -	__m128i reg_a, reg_b, reg_c, reg_d, reg_e, reg_f, reg_g, reg_h;
> > -	asm volatile (
> > -		"movdqu (%[src]), %[reg_a]\n\t"
> > -		"movdqu 16(%[src]), %[reg_b]\n\t"
> > -		"movdqu 32(%[src]), %[reg_c]\n\t"
> > -		"movdqu 48(%[src]), %[reg_d]\n\t"
> > -		"movdqu 64(%[src]), %[reg_e]\n\t"
> > -		"movdqu 80(%[src]), %[reg_f]\n\t"
> > -		"movdqu 96(%[src]), %[reg_g]\n\t"
> > -		"movdqu 112(%[src]), %[reg_h]\n\t"
> > -		"movdqu %[reg_a], (%[dst])\n\t"
> > -		"movdqu %[reg_b], 16(%[dst])\n\t"
> > -		"movdqu %[reg_c], 32(%[dst])\n\t"
> > -		"movdqu %[reg_d], 48(%[dst])\n\t"
> > -		"movdqu %[reg_e], 64(%[dst])\n\t"
> > -		"movdqu %[reg_f], 80(%[dst])\n\t"
> > -		"movdqu %[reg_g], 96(%[dst])\n\t"
> > -		"movdqu %[reg_h], 112(%[dst])\n\t"
> > -		: [reg_a] "=x" (reg_a),
> > -		  [reg_b] "=x" (reg_b),
> > -		  [reg_c] "=x" (reg_c),
> > -		  [reg_d] "=x" (reg_d),
> > -		  [reg_e] "=x" (reg_e),
> > -		  [reg_f] "=x" (reg_f),
> > -		  [reg_g] "=x" (reg_g),
> > -		  [reg_h] "=x" (reg_h)
> > -		: [src] "r" (src),
> > -		  [dst] "r"(dst)
> > -		: "memory"
> > -	);
> > +	rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32);
> > +	rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32);
> > +	rte_mov32((uint8_t *)dst + 2 * 32, (const uint8_t *)src + 2 * 32);
> > +	rte_mov32((uint8_t *)dst + 3 * 32, (const uint8_t *)src + 3 * 32);
> >  }
> >
> > -#ifdef __INTEL_COMPILER
> > -#pragma warning(enable:593)
> > -#endif
> > -
> > +/**
> > + * Copy 256 bytes from one location to another,
> > + * locations should not overlap.
> > + */
> >  static inline void
> >  rte_mov256(uint8_t *dst, const uint8_t *src)  {
> > -	rte_mov128(dst, src);
> > -	rte_mov128(dst + 128, src + 128);
> > +	rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32);
> > +	rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32);
> > +	rte_mov32((uint8_t *)dst + 2 * 32, (const uint8_t *)src + 2 * 32);
> > +	rte_mov32((uint8_t *)dst + 3 * 32, (const uint8_t *)src + 3 * 32);
> > +	rte_mov32((uint8_t *)dst + 4 * 32, (const uint8_t *)src + 4 * 32);
> > +	rte_mov32((uint8_t *)dst + 5 * 32, (const uint8_t *)src + 5 * 32);
> > +	rte_mov32((uint8_t *)dst + 6 * 32, (const uint8_t *)src + 6 * 32);
> > +	rte_mov32((uint8_t *)dst + 7 * 32, (const uint8_t *)src + 7 * 32);
> >  }
> >
> > -#define rte_memcpy(dst, src, n)              \
> > -	({ (__builtin_constant_p(n)) ?       \
> > -	memcpy((dst), (src), (n)) :          \
> > -	rte_memcpy_func((dst), (src), (n)); })
> > +/**
> > + * Copy 64-byte blocks from one location to another,
> > + * locations should not overlap.
> > + */
> > +static inline void
> > +rte_mov64blocks(uint8_t *dst, const uint8_t *src, size_t n) {
> > +	__m256i ymm0, ymm1;
> > +
> > +	while (n >= 64) {
> > +		ymm0 = _mm256_loadu_si256((const __m256i *)((const
> uint8_t *)src + 0 * 32));
> > +		n -= 64;
> > +		ymm1 = _mm256_loadu_si256((const __m256i *)((const
> uint8_t *)src + 1 * 32));
> > +		src = (const uint8_t *)src + 64;
> > +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 0 * 32),
> ymm0);
> > +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 1 * 32),
> ymm1);
> > +		dst = (uint8_t *)dst + 64;
> > +	}
> > +}
> > +
> > +/**
> > + * Copy 256-byte blocks from one location to another,
> > + * locations should not overlap.
> > + */
> > +static inline void
> > +rte_mov256blocks(uint8_t *dst, const uint8_t *src, size_t n) {
> > +	__m256i ymm0, ymm1, ymm2, ymm3, ymm4, ymm5, ymm6, ymm7;
> > +
> > +	while (n >= 256) {
> > +		ymm0 = _mm256_loadu_si256((const __m256i *)((const
> uint8_t *)src + 0 * 32));
> > +		n -= 256;
> > +		ymm1 = _mm256_loadu_si256((const __m256i *)((const
> uint8_t *)src + 1 * 32));
> > +		ymm2 = _mm256_loadu_si256((const __m256i *)((const
> uint8_t *)src + 2 * 32));
> > +		ymm3 = _mm256_loadu_si256((const __m256i *)((const
> uint8_t *)src + 3 * 32));
> > +		ymm4 = _mm256_loadu_si256((const __m256i *)((const
> uint8_t *)src + 4 * 32));
> > +		ymm5 = _mm256_loadu_si256((const __m256i *)((const
> uint8_t *)src + 5 * 32));
> > +		ymm6 = _mm256_loadu_si256((const __m256i *)((const
> uint8_t *)src + 6 * 32));
> > +		ymm7 = _mm256_loadu_si256((const __m256i *)((const
> uint8_t *)src + 7 * 32));
> > +		src = (const uint8_t *)src + 256;
> > +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 0 * 32),
> ymm0);
> > +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 1 * 32),
> ymm1);
> > +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 2 * 32),
> ymm2);
> > +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 3 * 32),
> ymm3);
> > +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 4 * 32),
> ymm4);
> > +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 5 * 32),
> ymm5);
> > +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 6 * 32),
> ymm6);
> > +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 7 * 32),
> ymm7);
> > +		dst = (uint8_t *)dst + 256;
> > +	}
> > +}
> >
> >  static inline void *
> > -rte_memcpy_func(void *dst, const void *src, size_t n)
> > +rte_memcpy(void *dst, const void *src, size_t n)
> >  {
> >  	void *ret = dst;
> > +	int dstofss;
> > +	int bits;
> >
> > -	/* We can't copy < 16 bytes using XMM registers so do it manually. */
> > +	/**
> > +	 * Copy less than 16 bytes
> > +	 */
> >  	if (n < 16) {
> >  		if (n & 0x01) {
> >  			*(uint8_t *)dst = *(const uint8_t *)src;
> > -			dst = (uint8_t *)dst + 1;
> >  			src = (const uint8_t *)src + 1;
> > +			dst = (uint8_t *)dst + 1;
> >  		}
> >  		if (n & 0x02) {
> >  			*(uint16_t *)dst = *(const uint16_t *)src;
> > -			dst = (uint16_t *)dst + 1;
> >  			src = (const uint16_t *)src + 1;
> > +			dst = (uint16_t *)dst + 1;
> >  		}
> >  		if (n & 0x04) {
> >  			*(uint32_t *)dst = *(const uint32_t *)src;
> > -			dst = (uint32_t *)dst + 1;
> >  			src = (const uint32_t *)src + 1;
> > +			dst = (uint32_t *)dst + 1;
> >  		}
> >  		if (n & 0x08) {
> >  			*(uint64_t *)dst = *(const uint64_t *)src; @@ -201,95
> +224,410 @@
> > rte_memcpy_func(void *dst, const void *src, size_t n)
> >  		return ret;
> >  	}
> >
> > -	/* Special fast cases for <= 128 bytes */
> > +	/**
> > +	 * Fast way when copy size doesn't exceed 512 bytes
> > +	 */
> >  	if (n <= 32) {
> >  		rte_mov16((uint8_t *)dst, (const uint8_t *)src);
> >  		rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 +
> n);
> >  		return ret;
> >  	}
> > -
> >  	if (n <= 64) {
> >  		rte_mov32((uint8_t *)dst, (const uint8_t *)src);
> >  		rte_mov32((uint8_t *)dst - 32 + n, (const uint8_t *)src - 32 +
> n);
> >  		return ret;
> >  	}
> > -
> > -	if (n <= 128) {
> > -		rte_mov64((uint8_t *)dst, (const uint8_t *)src);
> > -		rte_mov64((uint8_t *)dst - 64 + n, (const uint8_t *)src - 64 +
> n);
> > +	if (n <= 512) {
> > +		if (n >= 256) {
> > +			n -= 256;
> > +			rte_mov256((uint8_t *)dst, (const uint8_t *)src);
> > +			src = (const uint8_t *)src + 256;
> > +			dst = (uint8_t *)dst + 256;
> > +		}
> > +		if (n >= 128) {
> > +			n -= 128;
> > +			rte_mov128((uint8_t *)dst, (const uint8_t *)src);
> > +			src = (const uint8_t *)src + 128;
> > +			dst = (uint8_t *)dst + 128;
> > +		}
> > +		if (n >= 64) {
> > +			n -= 64;
> > +			rte_mov64((uint8_t *)dst, (const uint8_t *)src);
> > +			src = (const uint8_t *)src + 64;
> > +			dst = (uint8_t *)dst + 64;
> > +		}
> > +COPY_BLOCK_64_BACK31:
> > +		if (n > 32) {
> > +			rte_mov32((uint8_t *)dst, (const uint8_t *)src);
> > +			rte_mov32((uint8_t *)dst - 32 + n, (const uint8_t
> *)src - 32 + n);
> > +			return ret;
> > +		}
> > +		if (n > 0) {
> > +			rte_mov32((uint8_t *)dst - 32 + n, (const uint8_t
> *)src - 32 + n);
> > +		}
> >  		return ret;
> >  	}
> >
> > -	/*
> > -	 * For large copies > 128 bytes. This combination of 256, 64 and 16
> byte
> > -	 * copies was found to be faster than doing 128 and 32 byte copies as
> > -	 * well.
> > +	/**
> > +	 * Make store aligned when copy size exceeds 512 bytes
> >  	 */
> > -	for ( ; n >= 256; n -= 256) {
> > -		rte_mov256((uint8_t *)dst, (const uint8_t *)src);
> > -		dst = (uint8_t *)dst + 256;
> > -		src = (const uint8_t *)src + 256;
> > +	dstofss = 32 - (int)((long long)(void *)dst & 0x1F);
> > +	n -= dstofss;
> > +	rte_mov32((uint8_t *)dst, (const uint8_t *)src);
> > +	src = (const uint8_t *)src + dstofss;
> > +	dst = (uint8_t *)dst + dstofss;
> > +
> > +	/**
> > +	 * Copy 256-byte blocks.
> > +	 * Use copy block function for better instruction order control,
> > +	 * which is important when load is unaligned.
> > +	 */
> > +	rte_mov256blocks((uint8_t *)dst, (const uint8_t *)src, n);
> > +	bits = n;
> > +	n = n & 255;
> > +	bits -= n;
> > +	src = (const uint8_t *)src + bits;
> > +	dst = (uint8_t *)dst + bits;
> > +
> > +	/**
> > +	 * Copy 64-byte blocks.
> > +	 * Use copy block function for better instruction order control,
> > +	 * which is important when load is unaligned.
> > +	 */
> > +	if (n >= 64) {
> > +		rte_mov64blocks((uint8_t *)dst, (const uint8_t *)src, n);
> > +		bits = n;
> > +		n = n & 63;
> > +		bits -= n;
> > +		src = (const uint8_t *)src + bits;
> > +		dst = (uint8_t *)dst + bits;
> >  	}
> >
> > -	/*
> > -	 * We split the remaining bytes (which will be less than 256) into
> > -	 * 64byte (2^6) chunks.
> > -	 * Using incrementing integers in the case labels of a switch
> statement
> > -	 * enourages the compiler to use a jump table. To get incrementing
> > -	 * integers, we shift the 2 relevant bits to the LSB position to first
> > -	 * get decrementing integers, and then subtract.
> > +	/**
> > +	 * Copy whatever left
> >  	 */
> > -	switch (3 - (n >> 6)) {
> > -	case 0x00:
> > -		rte_mov64((uint8_t *)dst, (const uint8_t *)src);
> > -		n -= 64;
> > -		dst = (uint8_t *)dst + 64;
> > -		src = (const uint8_t *)src + 64;      /* fallthrough */
> > -	case 0x01:
> > -		rte_mov64((uint8_t *)dst, (const uint8_t *)src);
> > -		n -= 64;
> > -		dst = (uint8_t *)dst + 64;
> > -		src = (const uint8_t *)src + 64;      /* fallthrough */
> > -	case 0x02:
> > -		rte_mov64((uint8_t *)dst, (const uint8_t *)src);
> > -		n -= 64;
> > -		dst = (uint8_t *)dst + 64;
> > -		src = (const uint8_t *)src + 64;      /* fallthrough */
> > -	default:
> > -		;
> > +	goto COPY_BLOCK_64_BACK31;
> > +}
> > +
> > +#else /* RTE_MACHINE_CPUFLAG_AVX2 */
> > +
> > +/**
> > + * SSE & AVX implementation below
> > + */
> > +
> > +/**
> > + * Copy 16 bytes from one location to another,
> > + * locations should not overlap.
> > + */
> > +static inline void
> > +rte_mov16(uint8_t *dst, const uint8_t *src) {
> > +	__m128i xmm0;
> > +
> > +	xmm0 = _mm_loadu_si128((const __m128i *)(const __m128i *)src);
> > +	_mm_storeu_si128((__m128i *)dst, xmm0); }
> > +
> > +/**
> > + * Copy 32 bytes from one location to another,
> > + * locations should not overlap.
> > + */
> > +static inline void
> > +rte_mov32(uint8_t *dst, const uint8_t *src) {
> > +	rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
> > +	rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16); }
> > +
> > +/**
> > + * Copy 64 bytes from one location to another,
> > + * locations should not overlap.
> > + */
> > +static inline void
> > +rte_mov64(uint8_t *dst, const uint8_t *src) {
> > +	rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
> > +	rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
> > +	rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16);
> > +	rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16); }
> > +
> > +/**
> > + * Copy 128 bytes from one location to another,
> > + * locations should not overlap.
> > + */
> > +static inline void
> > +rte_mov128(uint8_t *dst, const uint8_t *src) {
> > +	rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
> > +	rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
> > +	rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16);
> > +	rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16);
> > +	rte_mov16((uint8_t *)dst + 4 * 16, (const uint8_t *)src + 4 * 16);
> > +	rte_mov16((uint8_t *)dst + 5 * 16, (const uint8_t *)src + 5 * 16);
> > +	rte_mov16((uint8_t *)dst + 6 * 16, (const uint8_t *)src + 6 * 16);
> > +	rte_mov16((uint8_t *)dst + 7 * 16, (const uint8_t *)src + 7 * 16); }
> > +
> > +/**
> > + * Copy 256 bytes from one location to another,
> > + * locations should not overlap.
> > + */
> > +static inline void
> > +rte_mov256(uint8_t *dst, const uint8_t *src) {
> > +	rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
> > +	rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
> > +	rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16);
> > +	rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16);
> > +	rte_mov16((uint8_t *)dst + 4 * 16, (const uint8_t *)src + 4 * 16);
> > +	rte_mov16((uint8_t *)dst + 5 * 16, (const uint8_t *)src + 5 * 16);
> > +	rte_mov16((uint8_t *)dst + 6 * 16, (const uint8_t *)src + 6 * 16);
> > +	rte_mov16((uint8_t *)dst + 7 * 16, (const uint8_t *)src + 7 * 16);
> > +	rte_mov16((uint8_t *)dst + 8 * 16, (const uint8_t *)src + 8 * 16);
> > +	rte_mov16((uint8_t *)dst + 9 * 16, (const uint8_t *)src + 9 * 16);
> > +	rte_mov16((uint8_t *)dst + 10 * 16, (const uint8_t *)src + 10 * 16);
> > +	rte_mov16((uint8_t *)dst + 11 * 16, (const uint8_t *)src + 11 * 16);
> > +	rte_mov16((uint8_t *)dst + 12 * 16, (const uint8_t *)src + 12 * 16);
> > +	rte_mov16((uint8_t *)dst + 13 * 16, (const uint8_t *)src + 13 * 16);
> > +	rte_mov16((uint8_t *)dst + 14 * 16, (const uint8_t *)src + 14 * 16);
> > +	rte_mov16((uint8_t *)dst + 15 * 16, (const uint8_t *)src + 15 * 16);
> > +}
> > +
> > +/**
> > + * Macro for copying unaligned block from one location to another
> > +with constant load offset,
> > + * 47 bytes leftover maximum,
> > + * locations should not overlap.
> > + * Requirements:
> > + * - Store is aligned
> > + * - Load offset is <offset>, which must be immediate value within
> > +[1, 15]
> > + * - For <src>, make sure <offset> bit backwards & <16 - offset> bit
> > +forwards are available for loading
> > + * - <dst>, <src>, <len> must be variables
> > + * - __m128i <xmm0> ~ <xmm8> must be pre-defined  */
> > +#define MOVEUNALIGNED_LEFT47_IMM(dst, src, len, offset)
> \
> > +({                                                                                                          \
> > +    int tmp;                                                                                                \
> > +    while (len >= 128 + 16 - offset) {                                                                      \
> > +        xmm0 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> offset + 0 * 16));                  \
> > +        len -= 128;                                                                                         \
> > +        xmm1 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> offset + 1 * 16));                  \
> > +        xmm2 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> offset + 2 * 16));                  \
> > +        xmm3 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> offset + 3 * 16));                  \
> > +        xmm4 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> offset + 4 * 16));                  \
> > +        xmm5 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> offset + 5 * 16));                  \
> > +        xmm6 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> offset + 6 * 16));                  \
> > +        xmm7 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> offset + 7 * 16));                  \
> > +        xmm8 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> offset + 8 * 16));                  \
> > +        src = (const uint8_t *)src + 128;                                                                   \
> > +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16),
> _mm_alignr_epi8(xmm1, xmm0, offset));        \
> > +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16),
> _mm_alignr_epi8(xmm2, xmm1, offset));        \
> > +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 2 * 16),
> _mm_alignr_epi8(xmm3, xmm2, offset));        \
> > +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 3 * 16),
> _mm_alignr_epi8(xmm4, xmm3, offset));        \
> > +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 4 * 16),
> _mm_alignr_epi8(xmm5, xmm4, offset));        \
> > +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 5 * 16),
> _mm_alignr_epi8(xmm6, xmm5, offset));        \
> > +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 6 * 16),
> _mm_alignr_epi8(xmm7, xmm6, offset));        \
> > +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 7 * 16),
> _mm_alignr_epi8(xmm8, xmm7, offset));        \
> > +        dst = (uint8_t *)dst + 128;                                                                         \
> > +    }                                                                                                       \
> > +    tmp = len;                                                                                              \
> > +    len = ((len - 16 + offset) & 127) + 16 - offset;                                                        \
> > +    tmp -= len;                                                                                             \
> > +    src = (const uint8_t *)src + tmp;                                                                       \
> > +    dst = (uint8_t *)dst + tmp;                                                                             \
> > +    if (len >= 32 + 16 - offset) {                                                                          \
> > +        while (len >= 32 + 16 - offset) {                                                                   \
> > +            xmm0 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> offset + 0 * 16));              \
> > +            len -= 32;                                                                                      \
> > +            xmm1 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> offset + 1 * 16));              \
> > +            xmm2 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> offset + 2 * 16));              \
> > +            src = (const uint8_t *)src + 32;                                                                \
> > +            _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16),
> _mm_alignr_epi8(xmm1, xmm0, offset));    \
> > +            _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16),
> _mm_alignr_epi8(xmm2, xmm1, offset));    \
> > +            dst = (uint8_t *)dst + 32;                                                                      \
> > +        }                                                                                                   \
> > +        tmp = len;                                                                                          \
> > +        len = ((len - 16 + offset) & 31) + 16 - offset;                                                     \
> > +        tmp -= len;                                                                                         \
> > +        src = (const uint8_t *)src + tmp;                                                                   \
> > +        dst = (uint8_t *)dst + tmp;                                                                         \
> > +    }                                                                                                       \
> > +})
> > +
> > +/**
> > + * Macro for copying unaligned block from one location to another,
> > + * 47 bytes leftover maximum,
> > + * locations should not overlap.
> > + * Use switch here because the aligning instruction requires immediate
> value for shift count.
> > + * Requirements:
> > + * - Store is aligned
> > + * - Load offset is <offset>, which must be within [1, 15]
> > + * - For <src>, make sure <offset> bit backwards & <16 - offset> bit
> > +forwards are available for loading
> > + * - <dst>, <src>, <len> must be variables
> > + * - __m128i <xmm0> ~ <xmm8> used in MOVEUNALIGNED_LEFT47_IMM
> must be
> > +pre-defined  */
> > +#define MOVEUNALIGNED_LEFT47(dst, src, len, offset)                   \
> > +({                                                                    \
> > +    switch (offset) {                                                 \
> > +    case 0x01: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x01); break;    \
> > +    case 0x02: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x02); break;    \
> > +    case 0x03: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x03); break;    \
> > +    case 0x04: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x04); break;    \
> > +    case 0x05: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x05); break;    \
> > +    case 0x06: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x06); break;    \
> > +    case 0x07: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x07); break;    \
> > +    case 0x08: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x08); break;    \
> > +    case 0x09: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x09); break;    \
> > +    case 0x0A: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0A); break;    \
> > +    case 0x0B: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0B); break;    \
> > +    case 0x0C: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0C); break;    \
> > +    case 0x0D: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0D); break;    \
> > +    case 0x0E: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0E); break;    \
> > +    case 0x0F: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0F); break;    \
> > +    default:;                                                         \
> > +    }                                                                 \
> > +})
> 
> We probably didn't understand each other properly.
> My thought was to do something like:
> 
> #define MOVEUNALIGNED_128(offset)	do { \
>  _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16),
> _mm_alignr_epi8(xmm1, xmm0, offset));        \
>  _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16),
> _mm_alignr_epi8(xmm2, xmm1, offset));        \
>  _mm_storeu_si128((__m128i *)((uint8_t *)dst + 2 * 16),
> _mm_alignr_epi8(xmm3, xmm2, offset));        \
>  _mm_storeu_si128((__m128i *)((uint8_t *)dst + 3 * 16),
> _mm_alignr_epi8(xmm4, xmm3, offset));        \
>  _mm_storeu_si128((__m128i *)((uint8_t *)dst + 4 * 16),
> _mm_alignr_epi8(xmm5, xmm4, offset));        \
>  _mm_storeu_si128((__m128i *)((uint8_t *)dst + 5 * 16),
> _mm_alignr_epi8(xmm6, xmm5, offset));        \
>  _mm_storeu_si128((__m128i *)((uint8_t *)dst + 6 * 16),
> _mm_alignr_epi8(xmm7, xmm6, offset));        \
>  _mm_storeu_si128((__m128i *)((uint8_t *)dst + 7 * 16),
> _mm_alignr_epi8(xmm8, xmm7, offset));       \
> } while(0);
> 
> 
> Then at MOVEUNALIGNED_LEFT47_IMM:
> ....
> xmm7 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 7
> * 16));                  \
> xmm8 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 8
> * 16));                  \
> src = (const uint8_t *)src + 128;
> 
> switch(offset) {
>     case 0x01: MOVEUNALIGNED_128(1); break;
>     ...
>     case 0x0f: MOVEUNALIGNED_128(f); break; } ...
> dst = (uint8_t *)dst + 128
> 
> An then in rte_memcpy() you don't need a switch for
> MOVEUNALIGNED_LEFT47_IMM itself.
> Thought it might help to generate smaller code for rte_ememcpy() while
> keeping performance reasonably high.
> 
> Konstantin
> 
> 
> > +
> > +static inline void *
> > +rte_memcpy(void *dst, const void *src, size_t n) {
> > +	__m128i xmm0, xmm1, xmm2, xmm3, xmm4, xmm5, xmm6, xmm7,
> xmm8;
> > +	void *ret = dst;
> > +	int dstofss;
> > +	int srcofs;
> > +
> > +	/**
> > +	 * Copy less than 16 bytes
> > +	 */
> > +	if (n < 16) {
> > +		if (n & 0x01) {
> > +			*(uint8_t *)dst = *(const uint8_t *)src;
> > +			src = (const uint8_t *)src + 1;
> > +			dst = (uint8_t *)dst + 1;
> > +		}
> > +		if (n & 0x02) {
> > +			*(uint16_t *)dst = *(const uint16_t *)src;
> > +			src = (const uint16_t *)src + 1;
> > +			dst = (uint16_t *)dst + 1;
> > +		}
> > +		if (n & 0x04) {
> > +			*(uint32_t *)dst = *(const uint32_t *)src;
> > +			src = (const uint32_t *)src + 1;
> > +			dst = (uint32_t *)dst + 1;
> > +		}
> > +		if (n & 0x08) {
> > +			*(uint64_t *)dst = *(const uint64_t *)src;
> > +		}
> > +		return ret;
> >  	}
> >
> > -	/*
> > -	 * We split the remaining bytes (which will be less than 64) into
> > -	 * 16byte (2^4) chunks, using the same switch structure as above.
> > +	/**
> > +	 * Fast way when copy size doesn't exceed 512 bytes
> >  	 */
> > -	switch (3 - (n >> 4)) {
> > -	case 0x00:
> > -		rte_mov16((uint8_t *)dst, (const uint8_t *)src);
> > -		n -= 16;
> > -		dst = (uint8_t *)dst + 16;
> > -		src = (const uint8_t *)src + 16;      /* fallthrough */
> > -	case 0x01:
> > -		rte_mov16((uint8_t *)dst, (const uint8_t *)src);
> > -		n -= 16;
> > -		dst = (uint8_t *)dst + 16;
> > -		src = (const uint8_t *)src + 16;      /* fallthrough */
> > -	case 0x02:
> > +	if (n <= 32) {
> >  		rte_mov16((uint8_t *)dst, (const uint8_t *)src);
> > -		n -= 16;
> > -		dst = (uint8_t *)dst + 16;
> > -		src = (const uint8_t *)src + 16;      /* fallthrough */
> > -	default:
> > -		;
> > +		rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 +
> n);
> > +		return ret;
> >  	}
> > -
> > -	/* Copy any remaining bytes, without going beyond end of buffers
> */
> > -	if (n != 0) {
> > +	if (n <= 48) {
> > +		rte_mov32((uint8_t *)dst, (const uint8_t *)src);
> > +		rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 +
> n);
> > +		return ret;
> > +	}
> > +	if (n <= 64) {
> > +		rte_mov32((uint8_t *)dst, (const uint8_t *)src);
> > +		rte_mov16((uint8_t *)dst + 32, (const uint8_t *)src + 32);
> >  		rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 +
> n);
> > +		return ret;
> > +	}
> > +	if (n <= 128) {
> > +		goto COPY_BLOCK_128_BACK15;
> >  	}
> > -	return ret;
> > +	if (n <= 512) {
> > +		if (n >= 256) {
> > +			n -= 256;
> > +			rte_mov128((uint8_t *)dst, (const uint8_t *)src);
> > +			rte_mov128((uint8_t *)dst + 128, (const uint8_t *)src
> + 128);
> > +			src = (const uint8_t *)src + 256;
> > +			dst = (uint8_t *)dst + 256;
> > +		}
> > +COPY_BLOCK_255_BACK15:
> > +		if (n >= 128) {
> > +			n -= 128;
> > +			rte_mov128((uint8_t *)dst, (const uint8_t *)src);
> > +			src = (const uint8_t *)src + 128;
> > +			dst = (uint8_t *)dst + 128;
> > +		}
> > +COPY_BLOCK_128_BACK15:
> > +		if (n >= 64) {
> > +			n -= 64;
> > +			rte_mov64((uint8_t *)dst, (const uint8_t *)src);
> > +			src = (const uint8_t *)src + 64;
> > +			dst = (uint8_t *)dst + 64;
> > +		}
> > +COPY_BLOCK_64_BACK15:
> > +		if (n >= 32) {
> > +			n -= 32;
> > +			rte_mov32((uint8_t *)dst, (const uint8_t *)src);
> > +			src = (const uint8_t *)src + 32;
> > +			dst = (uint8_t *)dst + 32;
> > +		}
> > +		if (n > 16) {
> > +			rte_mov16((uint8_t *)dst, (const uint8_t *)src);
> > +			rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t
> *)src - 16 + n);
> > +			return ret;
> > +		}
> > +		if (n > 0) {
> > +			rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t
> *)src - 16 + n);
> > +		}
> > +		return ret;
> > +	}
> > +
> > +	/**
> > +	 * Make store aligned when copy size exceeds 512 bytes,
> > +	 * and make sure the first 15 bytes are copied, because
> > +	 * unaligned copy functions require up to 15 bytes
> > +	 * backwards access.
> > +	 */
> > +	dstofss = 16 - (int)((long long)(void *)dst & 0x0F) + 16;
> > +	n -= dstofss;
> > +	rte_mov32((uint8_t *)dst, (const uint8_t *)src);
> > +	src = (const uint8_t *)src + dstofss;
> > +	dst = (uint8_t *)dst + dstofss;
> > +	srcofs = (int)((long long)(const void *)src & 0x0F);
> > +
> > +	/**
> > +	 * For aligned copy
> > +	 */
> > +	if (srcofs == 0) {
> > +		/**
> > +		 * Copy 256-byte blocks
> > +		 */
> > +		for (; n >= 256; n -= 256) {
> > +			rte_mov256((uint8_t *)dst, (const uint8_t *)src);
> > +			dst = (uint8_t *)dst + 256;
> > +			src = (const uint8_t *)src + 256;
> > +		}
> > +
> > +		/**
> > +		 * Copy whatever left
> > +		 */
> > +		goto COPY_BLOCK_255_BACK15;
> > +	}
> > +
> > +	/**
> > +	 * For copy with unaligned load
> > +	 */
> > +	MOVEUNALIGNED_LEFT47(dst, src, n, srcofs);
> > +
> > +	/**
> > +	 * Copy whatever left
> > +	 */
> > +	goto COPY_BLOCK_64_BACK15;
> >  }
> >
> > +#endif /* RTE_MACHINE_CPUFLAG_AVX2 */
> > +
> >  #ifdef __cplusplus
> >  }
> >  #endif
> > --
> > 1.9.3

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [dpdk-dev] [PATCH v2 4/4] lib/librte_eal: Optimized memcpy in arch/x86/rte_memcpy.h for both SSE and AVX platforms
  2015-01-30  5:57     ` Wang, Zhihong
@ 2015-01-30 10:44       ` Ananyev, Konstantin
  0 siblings, 0 replies; 12+ messages in thread
From: Ananyev, Konstantin @ 2015-01-30 10:44 UTC (permalink / raw)
  To: Wang, Zhihong, dev

Hey Zhihong,

> -----Original Message-----
> From: Wang, Zhihong
> Sent: Friday, January 30, 2015 5:57 AM
> To: Ananyev, Konstantin; dev@dpdk.org
> Subject: RE: [dpdk-dev] [PATCH v2 4/4] lib/librte_eal: Optimized memcpy in arch/x86/rte_memcpy.h for both SSE and AVX platforms
> 
> Hey Konstantin,
> 
> This method does reduce code size but lead to significant performance drop.
> I think we need to keep the original code.

Sure, no point to make it slower.
Thanks for trying it anyway.
Konstantin

> 
> 
> Thanks
> Zhihong (John)
> 
> 
> > -----Original Message-----
> > From: Ananyev, Konstantin
> > Sent: Thursday, January 29, 2015 11:18 PM
> > To: Wang, Zhihong; dev@dpdk.org
> > Subject: RE: [dpdk-dev] [PATCH v2 4/4] lib/librte_eal: Optimized memcpy in
> > arch/x86/rte_memcpy.h for both SSE and AVX platforms
> >
> > Hi Zhihong,
> >
> > > -----Original Message-----
> > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zhihong Wang
> > > Sent: Thursday, January 29, 2015 2:39 AM
> > > To: dev@dpdk.org
> > > Subject: [dpdk-dev] [PATCH v2 4/4] lib/librte_eal: Optimized memcpy in
> > > arch/x86/rte_memcpy.h for both SSE and AVX platforms
> > >
> > > Main code changes:
> > >
> > > 1. Differentiate architectural features based on CPU flags
> > >
> > >     a. Implement separated move functions for SSE/AVX/AVX2 to make
> > > full utilization of cache bandwidth
> > >
> > >     b. Implement separated copy flow specifically optimized for target
> > > architecture
> > >
> > > 2. Rewrite the memcpy function "rte_memcpy"
> > >
> > >     a. Add store aligning
> > >
> > >     b. Add load aligning based on architectural features
> > >
> > >     c. Put block copy loop into inline move functions for better
> > > control of instruction order
> > >
> > >     d. Eliminate unnecessary MOVs
> > >
> > > 3. Rewrite the inline move functions
> > >
> > >     a. Add move functions for unaligned load cases
> > >
> > >     b. Change instruction order in copy loops for better pipeline
> > > utilization
> > >
> > >     c. Use intrinsics instead of assembly code
> > >
> > > 4. Remove slow glibc call for constant copies
> > >
> > > Signed-off-by: Zhihong Wang <zhihong.wang@intel.com>
> > > ---
> > >  .../common/include/arch/x86/rte_memcpy.h           | 680
> > +++++++++++++++------
> > >  1 file changed, 509 insertions(+), 171 deletions(-)
> > >
> > > diff --git a/lib/librte_eal/common/include/arch/x86/rte_memcpy.h
> > > b/lib/librte_eal/common/include/arch/x86/rte_memcpy.h
> > > index fb9eba8..7b2d382 100644
> > > --- a/lib/librte_eal/common/include/arch/x86/rte_memcpy.h
> > > +++ b/lib/librte_eal/common/include/arch/x86/rte_memcpy.h
> > > @@ -34,166 +34,189 @@
> > >  #ifndef _RTE_MEMCPY_X86_64_H_
> > >  #define _RTE_MEMCPY_X86_64_H_
> > >
> > > +/**
> > > + * @file
> > > + *
> > > + * Functions for SSE/AVX/AVX2 implementation of memcpy().
> > > + */
> > > +
> > > +#include <stdio.h>
> > >  #include <stdint.h>
> > >  #include <string.h>
> > > -#include <emmintrin.h>
> > > +#include <x86intrin.h>
> > >
> > >  #ifdef __cplusplus
> > >  extern "C" {
> > >  #endif
> > >
> > > -#include "generic/rte_memcpy.h"
> > > +/**
> > > + * Copy bytes from one location to another. The locations must not
> > overlap.
> > > + *
> > > + * @note This is implemented as a macro, so it's address should not
> > > +be taken
> > > + * and care is needed as parameter expressions may be evaluated
> > multiple times.
> > > + *
> > > + * @param dst
> > > + *   Pointer to the destination of the data.
> > > + * @param src
> > > + *   Pointer to the source data.
> > > + * @param n
> > > + *   Number of bytes to copy.
> > > + * @return
> > > + *   Pointer to the destination data.
> > > + */
> > > +static inline void *
> > > +rte_memcpy(void *dst, const void *src, size_t n)
> > > +__attribute__((always_inline));
> > >
> > > -#ifdef __INTEL_COMPILER
> > > -#pragma warning(disable:593) /* Stop unused variable warning (reg_a
> > > etc). */ -#endif
> > > +#ifdef RTE_MACHINE_CPUFLAG_AVX2
> > >
> > > +/**
> > > + * AVX2 implementation below
> > > + */
> > > +
> > > +/**
> > > + * Copy 16 bytes from one location to another,
> > > + * locations should not overlap.
> > > + */
> > >  static inline void
> > >  rte_mov16(uint8_t *dst, const uint8_t *src)  {
> > > -	__m128i reg_a;
> > > -	asm volatile (
> > > -		"movdqu (%[src]), %[reg_a]\n\t"
> > > -		"movdqu %[reg_a], (%[dst])\n\t"
> > > -		: [reg_a] "=x" (reg_a)
> > > -		: [src] "r" (src),
> > > -		  [dst] "r"(dst)
> > > -		: "memory"
> > > -	);
> > > +	__m128i xmm0;
> > > +
> > > +	xmm0 = _mm_loadu_si128((const __m128i *)src);
> > > +	_mm_storeu_si128((__m128i *)dst, xmm0);
> > >  }
> > >
> > > +/**
> > > + * Copy 32 bytes from one location to another,
> > > + * locations should not overlap.
> > > + */
> > >  static inline void
> > >  rte_mov32(uint8_t *dst, const uint8_t *src)  {
> > > -	__m128i reg_a, reg_b;
> > > -	asm volatile (
> > > -		"movdqu (%[src]), %[reg_a]\n\t"
> > > -		"movdqu 16(%[src]), %[reg_b]\n\t"
> > > -		"movdqu %[reg_a], (%[dst])\n\t"
> > > -		"movdqu %[reg_b], 16(%[dst])\n\t"
> > > -		: [reg_a] "=x" (reg_a),
> > > -		  [reg_b] "=x" (reg_b)
> > > -		: [src] "r" (src),
> > > -		  [dst] "r"(dst)
> > > -		: "memory"
> > > -	);
> > > -}
> > > +	__m256i ymm0;
> > >
> > > -static inline void
> > > -rte_mov48(uint8_t *dst, const uint8_t *src) -{
> > > -	__m128i reg_a, reg_b, reg_c;
> > > -	asm volatile (
> > > -		"movdqu (%[src]), %[reg_a]\n\t"
> > > -		"movdqu 16(%[src]), %[reg_b]\n\t"
> > > -		"movdqu 32(%[src]), %[reg_c]\n\t"
> > > -		"movdqu %[reg_a], (%[dst])\n\t"
> > > -		"movdqu %[reg_b], 16(%[dst])\n\t"
> > > -		"movdqu %[reg_c], 32(%[dst])\n\t"
> > > -		: [reg_a] "=x" (reg_a),
> > > -		  [reg_b] "=x" (reg_b),
> > > -		  [reg_c] "=x" (reg_c)
> > > -		: [src] "r" (src),
> > > -		  [dst] "r"(dst)
> > > -		: "memory"
> > > -	);
> > > +	ymm0 = _mm256_loadu_si256((const __m256i *)src);
> > > +	_mm256_storeu_si256((__m256i *)dst, ymm0);
> > >  }
> > >
> > > +/**
> > > + * Copy 64 bytes from one location to another,
> > > + * locations should not overlap.
> > > + */
> > >  static inline void
> > >  rte_mov64(uint8_t *dst, const uint8_t *src)  {
> > > -	__m128i reg_a, reg_b, reg_c, reg_d;
> > > -	asm volatile (
> > > -		"movdqu (%[src]), %[reg_a]\n\t"
> > > -		"movdqu 16(%[src]), %[reg_b]\n\t"
> > > -		"movdqu 32(%[src]), %[reg_c]\n\t"
> > > -		"movdqu 48(%[src]), %[reg_d]\n\t"
> > > -		"movdqu %[reg_a], (%[dst])\n\t"
> > > -		"movdqu %[reg_b], 16(%[dst])\n\t"
> > > -		"movdqu %[reg_c], 32(%[dst])\n\t"
> > > -		"movdqu %[reg_d], 48(%[dst])\n\t"
> > > -		: [reg_a] "=x" (reg_a),
> > > -		  [reg_b] "=x" (reg_b),
> > > -		  [reg_c] "=x" (reg_c),
> > > -		  [reg_d] "=x" (reg_d)
> > > -		: [src] "r" (src),
> > > -		  [dst] "r"(dst)
> > > -		: "memory"
> > > -	);
> > > +	rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32);
> > > +	rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32);
> > >  }
> > >
> > > +/**
> > > + * Copy 128 bytes from one location to another,
> > > + * locations should not overlap.
> > > + */
> > >  static inline void
> > >  rte_mov128(uint8_t *dst, const uint8_t *src)  {
> > > -	__m128i reg_a, reg_b, reg_c, reg_d, reg_e, reg_f, reg_g, reg_h;
> > > -	asm volatile (
> > > -		"movdqu (%[src]), %[reg_a]\n\t"
> > > -		"movdqu 16(%[src]), %[reg_b]\n\t"
> > > -		"movdqu 32(%[src]), %[reg_c]\n\t"
> > > -		"movdqu 48(%[src]), %[reg_d]\n\t"
> > > -		"movdqu 64(%[src]), %[reg_e]\n\t"
> > > -		"movdqu 80(%[src]), %[reg_f]\n\t"
> > > -		"movdqu 96(%[src]), %[reg_g]\n\t"
> > > -		"movdqu 112(%[src]), %[reg_h]\n\t"
> > > -		"movdqu %[reg_a], (%[dst])\n\t"
> > > -		"movdqu %[reg_b], 16(%[dst])\n\t"
> > > -		"movdqu %[reg_c], 32(%[dst])\n\t"
> > > -		"movdqu %[reg_d], 48(%[dst])\n\t"
> > > -		"movdqu %[reg_e], 64(%[dst])\n\t"
> > > -		"movdqu %[reg_f], 80(%[dst])\n\t"
> > > -		"movdqu %[reg_g], 96(%[dst])\n\t"
> > > -		"movdqu %[reg_h], 112(%[dst])\n\t"
> > > -		: [reg_a] "=x" (reg_a),
> > > -		  [reg_b] "=x" (reg_b),
> > > -		  [reg_c] "=x" (reg_c),
> > > -		  [reg_d] "=x" (reg_d),
> > > -		  [reg_e] "=x" (reg_e),
> > > -		  [reg_f] "=x" (reg_f),
> > > -		  [reg_g] "=x" (reg_g),
> > > -		  [reg_h] "=x" (reg_h)
> > > -		: [src] "r" (src),
> > > -		  [dst] "r"(dst)
> > > -		: "memory"
> > > -	);
> > > +	rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32);
> > > +	rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32);
> > > +	rte_mov32((uint8_t *)dst + 2 * 32, (const uint8_t *)src + 2 * 32);
> > > +	rte_mov32((uint8_t *)dst + 3 * 32, (const uint8_t *)src + 3 * 32);
> > >  }
> > >
> > > -#ifdef __INTEL_COMPILER
> > > -#pragma warning(enable:593)
> > > -#endif
> > > -
> > > +/**
> > > + * Copy 256 bytes from one location to another,
> > > + * locations should not overlap.
> > > + */
> > >  static inline void
> > >  rte_mov256(uint8_t *dst, const uint8_t *src)  {
> > > -	rte_mov128(dst, src);
> > > -	rte_mov128(dst + 128, src + 128);
> > > +	rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32);
> > > +	rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32);
> > > +	rte_mov32((uint8_t *)dst + 2 * 32, (const uint8_t *)src + 2 * 32);
> > > +	rte_mov32((uint8_t *)dst + 3 * 32, (const uint8_t *)src + 3 * 32);
> > > +	rte_mov32((uint8_t *)dst + 4 * 32, (const uint8_t *)src + 4 * 32);
> > > +	rte_mov32((uint8_t *)dst + 5 * 32, (const uint8_t *)src + 5 * 32);
> > > +	rte_mov32((uint8_t *)dst + 6 * 32, (const uint8_t *)src + 6 * 32);
> > > +	rte_mov32((uint8_t *)dst + 7 * 32, (const uint8_t *)src + 7 * 32);
> > >  }
> > >
> > > -#define rte_memcpy(dst, src, n)              \
> > > -	({ (__builtin_constant_p(n)) ?       \
> > > -	memcpy((dst), (src), (n)) :          \
> > > -	rte_memcpy_func((dst), (src), (n)); })
> > > +/**
> > > + * Copy 64-byte blocks from one location to another,
> > > + * locations should not overlap.
> > > + */
> > > +static inline void
> > > +rte_mov64blocks(uint8_t *dst, const uint8_t *src, size_t n) {
> > > +	__m256i ymm0, ymm1;
> > > +
> > > +	while (n >= 64) {
> > > +		ymm0 = _mm256_loadu_si256((const __m256i *)((const
> > uint8_t *)src + 0 * 32));
> > > +		n -= 64;
> > > +		ymm1 = _mm256_loadu_si256((const __m256i *)((const
> > uint8_t *)src + 1 * 32));
> > > +		src = (const uint8_t *)src + 64;
> > > +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 0 * 32),
> > ymm0);
> > > +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 1 * 32),
> > ymm1);
> > > +		dst = (uint8_t *)dst + 64;
> > > +	}
> > > +}
> > > +
> > > +/**
> > > + * Copy 256-byte blocks from one location to another,
> > > + * locations should not overlap.
> > > + */
> > > +static inline void
> > > +rte_mov256blocks(uint8_t *dst, const uint8_t *src, size_t n) {
> > > +	__m256i ymm0, ymm1, ymm2, ymm3, ymm4, ymm5, ymm6, ymm7;
> > > +
> > > +	while (n >= 256) {
> > > +		ymm0 = _mm256_loadu_si256((const __m256i *)((const
> > uint8_t *)src + 0 * 32));
> > > +		n -= 256;
> > > +		ymm1 = _mm256_loadu_si256((const __m256i *)((const
> > uint8_t *)src + 1 * 32));
> > > +		ymm2 = _mm256_loadu_si256((const __m256i *)((const
> > uint8_t *)src + 2 * 32));
> > > +		ymm3 = _mm256_loadu_si256((const __m256i *)((const
> > uint8_t *)src + 3 * 32));
> > > +		ymm4 = _mm256_loadu_si256((const __m256i *)((const
> > uint8_t *)src + 4 * 32));
> > > +		ymm5 = _mm256_loadu_si256((const __m256i *)((const
> > uint8_t *)src + 5 * 32));
> > > +		ymm6 = _mm256_loadu_si256((const __m256i *)((const
> > uint8_t *)src + 6 * 32));
> > > +		ymm7 = _mm256_loadu_si256((const __m256i *)((const
> > uint8_t *)src + 7 * 32));
> > > +		src = (const uint8_t *)src + 256;
> > > +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 0 * 32),
> > ymm0);
> > > +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 1 * 32),
> > ymm1);
> > > +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 2 * 32),
> > ymm2);
> > > +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 3 * 32),
> > ymm3);
> > > +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 4 * 32),
> > ymm4);
> > > +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 5 * 32),
> > ymm5);
> > > +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 6 * 32),
> > ymm6);
> > > +		_mm256_storeu_si256((__m256i *)((uint8_t *)dst + 7 * 32),
> > ymm7);
> > > +		dst = (uint8_t *)dst + 256;
> > > +	}
> > > +}
> > >
> > >  static inline void *
> > > -rte_memcpy_func(void *dst, const void *src, size_t n)
> > > +rte_memcpy(void *dst, const void *src, size_t n)
> > >  {
> > >  	void *ret = dst;
> > > +	int dstofss;
> > > +	int bits;
> > >
> > > -	/* We can't copy < 16 bytes using XMM registers so do it manually. */
> > > +	/**
> > > +	 * Copy less than 16 bytes
> > > +	 */
> > >  	if (n < 16) {
> > >  		if (n & 0x01) {
> > >  			*(uint8_t *)dst = *(const uint8_t *)src;
> > > -			dst = (uint8_t *)dst + 1;
> > >  			src = (const uint8_t *)src + 1;
> > > +			dst = (uint8_t *)dst + 1;
> > >  		}
> > >  		if (n & 0x02) {
> > >  			*(uint16_t *)dst = *(const uint16_t *)src;
> > > -			dst = (uint16_t *)dst + 1;
> > >  			src = (const uint16_t *)src + 1;
> > > +			dst = (uint16_t *)dst + 1;
> > >  		}
> > >  		if (n & 0x04) {
> > >  			*(uint32_t *)dst = *(const uint32_t *)src;
> > > -			dst = (uint32_t *)dst + 1;
> > >  			src = (const uint32_t *)src + 1;
> > > +			dst = (uint32_t *)dst + 1;
> > >  		}
> > >  		if (n & 0x08) {
> > >  			*(uint64_t *)dst = *(const uint64_t *)src; @@ -201,95
> > +224,410 @@
> > > rte_memcpy_func(void *dst, const void *src, size_t n)
> > >  		return ret;
> > >  	}
> > >
> > > -	/* Special fast cases for <= 128 bytes */
> > > +	/**
> > > +	 * Fast way when copy size doesn't exceed 512 bytes
> > > +	 */
> > >  	if (n <= 32) {
> > >  		rte_mov16((uint8_t *)dst, (const uint8_t *)src);
> > >  		rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 +
> > n);
> > >  		return ret;
> > >  	}
> > > -
> > >  	if (n <= 64) {
> > >  		rte_mov32((uint8_t *)dst, (const uint8_t *)src);
> > >  		rte_mov32((uint8_t *)dst - 32 + n, (const uint8_t *)src - 32 +
> > n);
> > >  		return ret;
> > >  	}
> > > -
> > > -	if (n <= 128) {
> > > -		rte_mov64((uint8_t *)dst, (const uint8_t *)src);
> > > -		rte_mov64((uint8_t *)dst - 64 + n, (const uint8_t *)src - 64 +
> > n);
> > > +	if (n <= 512) {
> > > +		if (n >= 256) {
> > > +			n -= 256;
> > > +			rte_mov256((uint8_t *)dst, (const uint8_t *)src);
> > > +			src = (const uint8_t *)src + 256;
> > > +			dst = (uint8_t *)dst + 256;
> > > +		}
> > > +		if (n >= 128) {
> > > +			n -= 128;
> > > +			rte_mov128((uint8_t *)dst, (const uint8_t *)src);
> > > +			src = (const uint8_t *)src + 128;
> > > +			dst = (uint8_t *)dst + 128;
> > > +		}
> > > +		if (n >= 64) {
> > > +			n -= 64;
> > > +			rte_mov64((uint8_t *)dst, (const uint8_t *)src);
> > > +			src = (const uint8_t *)src + 64;
> > > +			dst = (uint8_t *)dst + 64;
> > > +		}
> > > +COPY_BLOCK_64_BACK31:
> > > +		if (n > 32) {
> > > +			rte_mov32((uint8_t *)dst, (const uint8_t *)src);
> > > +			rte_mov32((uint8_t *)dst - 32 + n, (const uint8_t
> > *)src - 32 + n);
> > > +			return ret;
> > > +		}
> > > +		if (n > 0) {
> > > +			rte_mov32((uint8_t *)dst - 32 + n, (const uint8_t
> > *)src - 32 + n);
> > > +		}
> > >  		return ret;
> > >  	}
> > >
> > > -	/*
> > > -	 * For large copies > 128 bytes. This combination of 256, 64 and 16
> > byte
> > > -	 * copies was found to be faster than doing 128 and 32 byte copies as
> > > -	 * well.
> > > +	/**
> > > +	 * Make store aligned when copy size exceeds 512 bytes
> > >  	 */
> > > -	for ( ; n >= 256; n -= 256) {
> > > -		rte_mov256((uint8_t *)dst, (const uint8_t *)src);
> > > -		dst = (uint8_t *)dst + 256;
> > > -		src = (const uint8_t *)src + 256;
> > > +	dstofss = 32 - (int)((long long)(void *)dst & 0x1F);
> > > +	n -= dstofss;
> > > +	rte_mov32((uint8_t *)dst, (const uint8_t *)src);
> > > +	src = (const uint8_t *)src + dstofss;
> > > +	dst = (uint8_t *)dst + dstofss;
> > > +
> > > +	/**
> > > +	 * Copy 256-byte blocks.
> > > +	 * Use copy block function for better instruction order control,
> > > +	 * which is important when load is unaligned.
> > > +	 */
> > > +	rte_mov256blocks((uint8_t *)dst, (const uint8_t *)src, n);
> > > +	bits = n;
> > > +	n = n & 255;
> > > +	bits -= n;
> > > +	src = (const uint8_t *)src + bits;
> > > +	dst = (uint8_t *)dst + bits;
> > > +
> > > +	/**
> > > +	 * Copy 64-byte blocks.
> > > +	 * Use copy block function for better instruction order control,
> > > +	 * which is important when load is unaligned.
> > > +	 */
> > > +	if (n >= 64) {
> > > +		rte_mov64blocks((uint8_t *)dst, (const uint8_t *)src, n);
> > > +		bits = n;
> > > +		n = n & 63;
> > > +		bits -= n;
> > > +		src = (const uint8_t *)src + bits;
> > > +		dst = (uint8_t *)dst + bits;
> > >  	}
> > >
> > > -	/*
> > > -	 * We split the remaining bytes (which will be less than 256) into
> > > -	 * 64byte (2^6) chunks.
> > > -	 * Using incrementing integers in the case labels of a switch
> > statement
> > > -	 * enourages the compiler to use a jump table. To get incrementing
> > > -	 * integers, we shift the 2 relevant bits to the LSB position to first
> > > -	 * get decrementing integers, and then subtract.
> > > +	/**
> > > +	 * Copy whatever left
> > >  	 */
> > > -	switch (3 - (n >> 6)) {
> > > -	case 0x00:
> > > -		rte_mov64((uint8_t *)dst, (const uint8_t *)src);
> > > -		n -= 64;
> > > -		dst = (uint8_t *)dst + 64;
> > > -		src = (const uint8_t *)src + 64;      /* fallthrough */
> > > -	case 0x01:
> > > -		rte_mov64((uint8_t *)dst, (const uint8_t *)src);
> > > -		n -= 64;
> > > -		dst = (uint8_t *)dst + 64;
> > > -		src = (const uint8_t *)src + 64;      /* fallthrough */
> > > -	case 0x02:
> > > -		rte_mov64((uint8_t *)dst, (const uint8_t *)src);
> > > -		n -= 64;
> > > -		dst = (uint8_t *)dst + 64;
> > > -		src = (const uint8_t *)src + 64;      /* fallthrough */
> > > -	default:
> > > -		;
> > > +	goto COPY_BLOCK_64_BACK31;
> > > +}
> > > +
> > > +#else /* RTE_MACHINE_CPUFLAG_AVX2 */
> > > +
> > > +/**
> > > + * SSE & AVX implementation below
> > > + */
> > > +
> > > +/**
> > > + * Copy 16 bytes from one location to another,
> > > + * locations should not overlap.
> > > + */
> > > +static inline void
> > > +rte_mov16(uint8_t *dst, const uint8_t *src) {
> > > +	__m128i xmm0;
> > > +
> > > +	xmm0 = _mm_loadu_si128((const __m128i *)(const __m128i *)src);
> > > +	_mm_storeu_si128((__m128i *)dst, xmm0); }
> > > +
> > > +/**
> > > + * Copy 32 bytes from one location to another,
> > > + * locations should not overlap.
> > > + */
> > > +static inline void
> > > +rte_mov32(uint8_t *dst, const uint8_t *src) {
> > > +	rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
> > > +	rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16); }
> > > +
> > > +/**
> > > + * Copy 64 bytes from one location to another,
> > > + * locations should not overlap.
> > > + */
> > > +static inline void
> > > +rte_mov64(uint8_t *dst, const uint8_t *src) {
> > > +	rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
> > > +	rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
> > > +	rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16);
> > > +	rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16); }
> > > +
> > > +/**
> > > + * Copy 128 bytes from one location to another,
> > > + * locations should not overlap.
> > > + */
> > > +static inline void
> > > +rte_mov128(uint8_t *dst, const uint8_t *src) {
> > > +	rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
> > > +	rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
> > > +	rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16);
> > > +	rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16);
> > > +	rte_mov16((uint8_t *)dst + 4 * 16, (const uint8_t *)src + 4 * 16);
> > > +	rte_mov16((uint8_t *)dst + 5 * 16, (const uint8_t *)src + 5 * 16);
> > > +	rte_mov16((uint8_t *)dst + 6 * 16, (const uint8_t *)src + 6 * 16);
> > > +	rte_mov16((uint8_t *)dst + 7 * 16, (const uint8_t *)src + 7 * 16); }
> > > +
> > > +/**
> > > + * Copy 256 bytes from one location to another,
> > > + * locations should not overlap.
> > > + */
> > > +static inline void
> > > +rte_mov256(uint8_t *dst, const uint8_t *src) {
> > > +	rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
> > > +	rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
> > > +	rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16);
> > > +	rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16);
> > > +	rte_mov16((uint8_t *)dst + 4 * 16, (const uint8_t *)src + 4 * 16);
> > > +	rte_mov16((uint8_t *)dst + 5 * 16, (const uint8_t *)src + 5 * 16);
> > > +	rte_mov16((uint8_t *)dst + 6 * 16, (const uint8_t *)src + 6 * 16);
> > > +	rte_mov16((uint8_t *)dst + 7 * 16, (const uint8_t *)src + 7 * 16);
> > > +	rte_mov16((uint8_t *)dst + 8 * 16, (const uint8_t *)src + 8 * 16);
> > > +	rte_mov16((uint8_t *)dst + 9 * 16, (const uint8_t *)src + 9 * 16);
> > > +	rte_mov16((uint8_t *)dst + 10 * 16, (const uint8_t *)src + 10 * 16);
> > > +	rte_mov16((uint8_t *)dst + 11 * 16, (const uint8_t *)src + 11 * 16);
> > > +	rte_mov16((uint8_t *)dst + 12 * 16, (const uint8_t *)src + 12 * 16);
> > > +	rte_mov16((uint8_t *)dst + 13 * 16, (const uint8_t *)src + 13 * 16);
> > > +	rte_mov16((uint8_t *)dst + 14 * 16, (const uint8_t *)src + 14 * 16);
> > > +	rte_mov16((uint8_t *)dst + 15 * 16, (const uint8_t *)src + 15 * 16);
> > > +}
> > > +
> > > +/**
> > > + * Macro for copying unaligned block from one location to another
> > > +with constant load offset,
> > > + * 47 bytes leftover maximum,
> > > + * locations should not overlap.
> > > + * Requirements:
> > > + * - Store is aligned
> > > + * - Load offset is <offset>, which must be immediate value within
> > > +[1, 15]
> > > + * - For <src>, make sure <offset> bit backwards & <16 - offset> bit
> > > +forwards are available for loading
> > > + * - <dst>, <src>, <len> must be variables
> > > + * - __m128i <xmm0> ~ <xmm8> must be pre-defined  */
> > > +#define MOVEUNALIGNED_LEFT47_IMM(dst, src, len, offset)
> > \
> > > +({                                                                                                          \
> > > +    int tmp;                                                                                                \
> > > +    while (len >= 128 + 16 - offset) {                                                                      \
> > > +        xmm0 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> > offset + 0 * 16));                  \
> > > +        len -= 128;                                                                                         \
> > > +        xmm1 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> > offset + 1 * 16));                  \
> > > +        xmm2 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> > offset + 2 * 16));                  \
> > > +        xmm3 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> > offset + 3 * 16));                  \
> > > +        xmm4 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> > offset + 4 * 16));                  \
> > > +        xmm5 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> > offset + 5 * 16));                  \
> > > +        xmm6 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> > offset + 6 * 16));                  \
> > > +        xmm7 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> > offset + 7 * 16));                  \
> > > +        xmm8 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> > offset + 8 * 16));                  \
> > > +        src = (const uint8_t *)src + 128;                                                                   \
> > > +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16),
> > _mm_alignr_epi8(xmm1, xmm0, offset));        \
> > > +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16),
> > _mm_alignr_epi8(xmm2, xmm1, offset));        \
> > > +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 2 * 16),
> > _mm_alignr_epi8(xmm3, xmm2, offset));        \
> > > +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 3 * 16),
> > _mm_alignr_epi8(xmm4, xmm3, offset));        \
> > > +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 4 * 16),
> > _mm_alignr_epi8(xmm5, xmm4, offset));        \
> > > +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 5 * 16),
> > _mm_alignr_epi8(xmm6, xmm5, offset));        \
> > > +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 6 * 16),
> > _mm_alignr_epi8(xmm7, xmm6, offset));        \
> > > +        _mm_storeu_si128((__m128i *)((uint8_t *)dst + 7 * 16),
> > _mm_alignr_epi8(xmm8, xmm7, offset));        \
> > > +        dst = (uint8_t *)dst + 128;                                                                         \
> > > +    }                                                                                                       \
> > > +    tmp = len;                                                                                              \
> > > +    len = ((len - 16 + offset) & 127) + 16 - offset;                                                        \
> > > +    tmp -= len;                                                                                             \
> > > +    src = (const uint8_t *)src + tmp;                                                                       \
> > > +    dst = (uint8_t *)dst + tmp;                                                                             \
> > > +    if (len >= 32 + 16 - offset) {                                                                          \
> > > +        while (len >= 32 + 16 - offset) {                                                                   \
> > > +            xmm0 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> > offset + 0 * 16));              \
> > > +            len -= 32;                                                                                      \
> > > +            xmm1 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> > offset + 1 * 16));              \
> > > +            xmm2 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src -
> > offset + 2 * 16));              \
> > > +            src = (const uint8_t *)src + 32;                                                                \
> > > +            _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16),
> > _mm_alignr_epi8(xmm1, xmm0, offset));    \
> > > +            _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16),
> > _mm_alignr_epi8(xmm2, xmm1, offset));    \
> > > +            dst = (uint8_t *)dst + 32;                                                                      \
> > > +        }                                                                                                   \
> > > +        tmp = len;                                                                                          \
> > > +        len = ((len - 16 + offset) & 31) + 16 - offset;                                                     \
> > > +        tmp -= len;                                                                                         \
> > > +        src = (const uint8_t *)src + tmp;                                                                   \
> > > +        dst = (uint8_t *)dst + tmp;                                                                         \
> > > +    }                                                                                                       \
> > > +})
> > > +
> > > +/**
> > > + * Macro for copying unaligned block from one location to another,
> > > + * 47 bytes leftover maximum,
> > > + * locations should not overlap.
> > > + * Use switch here because the aligning instruction requires immediate
> > value for shift count.
> > > + * Requirements:
> > > + * - Store is aligned
> > > + * - Load offset is <offset>, which must be within [1, 15]
> > > + * - For <src>, make sure <offset> bit backwards & <16 - offset> bit
> > > +forwards are available for loading
> > > + * - <dst>, <src>, <len> must be variables
> > > + * - __m128i <xmm0> ~ <xmm8> used in MOVEUNALIGNED_LEFT47_IMM
> > must be
> > > +pre-defined  */
> > > +#define MOVEUNALIGNED_LEFT47(dst, src, len, offset)                   \
> > > +({                                                                    \
> > > +    switch (offset) {                                                 \
> > > +    case 0x01: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x01); break;    \
> > > +    case 0x02: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x02); break;    \
> > > +    case 0x03: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x03); break;    \
> > > +    case 0x04: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x04); break;    \
> > > +    case 0x05: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x05); break;    \
> > > +    case 0x06: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x06); break;    \
> > > +    case 0x07: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x07); break;    \
> > > +    case 0x08: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x08); break;    \
> > > +    case 0x09: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x09); break;    \
> > > +    case 0x0A: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0A); break;    \
> > > +    case 0x0B: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0B); break;    \
> > > +    case 0x0C: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0C); break;    \
> > > +    case 0x0D: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0D); break;    \
> > > +    case 0x0E: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0E); break;    \
> > > +    case 0x0F: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0F); break;    \
> > > +    default:;                                                         \
> > > +    }                                                                 \
> > > +})
> >
> > We probably didn't understand each other properly.
> > My thought was to do something like:
> >
> > #define MOVEUNALIGNED_128(offset)	do { \
> >  _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16),
> > _mm_alignr_epi8(xmm1, xmm0, offset));        \
> >  _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16),
> > _mm_alignr_epi8(xmm2, xmm1, offset));        \
> >  _mm_storeu_si128((__m128i *)((uint8_t *)dst + 2 * 16),
> > _mm_alignr_epi8(xmm3, xmm2, offset));        \
> >  _mm_storeu_si128((__m128i *)((uint8_t *)dst + 3 * 16),
> > _mm_alignr_epi8(xmm4, xmm3, offset));        \
> >  _mm_storeu_si128((__m128i *)((uint8_t *)dst + 4 * 16),
> > _mm_alignr_epi8(xmm5, xmm4, offset));        \
> >  _mm_storeu_si128((__m128i *)((uint8_t *)dst + 5 * 16),
> > _mm_alignr_epi8(xmm6, xmm5, offset));        \
> >  _mm_storeu_si128((__m128i *)((uint8_t *)dst + 6 * 16),
> > _mm_alignr_epi8(xmm7, xmm6, offset));        \
> >  _mm_storeu_si128((__m128i *)((uint8_t *)dst + 7 * 16),
> > _mm_alignr_epi8(xmm8, xmm7, offset));       \
> > } while(0);
> >
> >
> > Then at MOVEUNALIGNED_LEFT47_IMM:
> > ....
> > xmm7 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 7
> > * 16));                  \
> > xmm8 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 8
> > * 16));                  \
> > src = (const uint8_t *)src + 128;
> >
> > switch(offset) {
> >     case 0x01: MOVEUNALIGNED_128(1); break;
> >     ...
> >     case 0x0f: MOVEUNALIGNED_128(f); break; } ...
> > dst = (uint8_t *)dst + 128
> >
> > An then in rte_memcpy() you don't need a switch for
> > MOVEUNALIGNED_LEFT47_IMM itself.
> > Thought it might help to generate smaller code for rte_ememcpy() while
> > keeping performance reasonably high.
> >
> > Konstantin
> >
> >
> > > +
> > > +static inline void *
> > > +rte_memcpy(void *dst, const void *src, size_t n) {
> > > +	__m128i xmm0, xmm1, xmm2, xmm3, xmm4, xmm5, xmm6, xmm7,
> > xmm8;
> > > +	void *ret = dst;
> > > +	int dstofss;
> > > +	int srcofs;
> > > +
> > > +	/**
> > > +	 * Copy less than 16 bytes
> > > +	 */
> > > +	if (n < 16) {
> > > +		if (n & 0x01) {
> > > +			*(uint8_t *)dst = *(const uint8_t *)src;
> > > +			src = (const uint8_t *)src + 1;
> > > +			dst = (uint8_t *)dst + 1;
> > > +		}
> > > +		if (n & 0x02) {
> > > +			*(uint16_t *)dst = *(const uint16_t *)src;
> > > +			src = (const uint16_t *)src + 1;
> > > +			dst = (uint16_t *)dst + 1;
> > > +		}
> > > +		if (n & 0x04) {
> > > +			*(uint32_t *)dst = *(const uint32_t *)src;
> > > +			src = (const uint32_t *)src + 1;
> > > +			dst = (uint32_t *)dst + 1;
> > > +		}
> > > +		if (n & 0x08) {
> > > +			*(uint64_t *)dst = *(const uint64_t *)src;
> > > +		}
> > > +		return ret;
> > >  	}
> > >
> > > -	/*
> > > -	 * We split the remaining bytes (which will be less than 64) into
> > > -	 * 16byte (2^4) chunks, using the same switch structure as above.
> > > +	/**
> > > +	 * Fast way when copy size doesn't exceed 512 bytes
> > >  	 */
> > > -	switch (3 - (n >> 4)) {
> > > -	case 0x00:
> > > -		rte_mov16((uint8_t *)dst, (const uint8_t *)src);
> > > -		n -= 16;
> > > -		dst = (uint8_t *)dst + 16;
> > > -		src = (const uint8_t *)src + 16;      /* fallthrough */
> > > -	case 0x01:
> > > -		rte_mov16((uint8_t *)dst, (const uint8_t *)src);
> > > -		n -= 16;
> > > -		dst = (uint8_t *)dst + 16;
> > > -		src = (const uint8_t *)src + 16;      /* fallthrough */
> > > -	case 0x02:
> > > +	if (n <= 32) {
> > >  		rte_mov16((uint8_t *)dst, (const uint8_t *)src);
> > > -		n -= 16;
> > > -		dst = (uint8_t *)dst + 16;
> > > -		src = (const uint8_t *)src + 16;      /* fallthrough */
> > > -	default:
> > > -		;
> > > +		rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 +
> > n);
> > > +		return ret;
> > >  	}
> > > -
> > > -	/* Copy any remaining bytes, without going beyond end of buffers
> > */
> > > -	if (n != 0) {
> > > +	if (n <= 48) {
> > > +		rte_mov32((uint8_t *)dst, (const uint8_t *)src);
> > > +		rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 +
> > n);
> > > +		return ret;
> > > +	}
> > > +	if (n <= 64) {
> > > +		rte_mov32((uint8_t *)dst, (const uint8_t *)src);
> > > +		rte_mov16((uint8_t *)dst + 32, (const uint8_t *)src + 32);
> > >  		rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 +
> > n);
> > > +		return ret;
> > > +	}
> > > +	if (n <= 128) {
> > > +		goto COPY_BLOCK_128_BACK15;
> > >  	}
> > > -	return ret;
> > > +	if (n <= 512) {
> > > +		if (n >= 256) {
> > > +			n -= 256;
> > > +			rte_mov128((uint8_t *)dst, (const uint8_t *)src);
> > > +			rte_mov128((uint8_t *)dst + 128, (const uint8_t *)src
> > + 128);
> > > +			src = (const uint8_t *)src + 256;
> > > +			dst = (uint8_t *)dst + 256;
> > > +		}
> > > +COPY_BLOCK_255_BACK15:
> > > +		if (n >= 128) {
> > > +			n -= 128;
> > > +			rte_mov128((uint8_t *)dst, (const uint8_t *)src);
> > > +			src = (const uint8_t *)src + 128;
> > > +			dst = (uint8_t *)dst + 128;
> > > +		}
> > > +COPY_BLOCK_128_BACK15:
> > > +		if (n >= 64) {
> > > +			n -= 64;
> > > +			rte_mov64((uint8_t *)dst, (const uint8_t *)src);
> > > +			src = (const uint8_t *)src + 64;
> > > +			dst = (uint8_t *)dst + 64;
> > > +		}
> > > +COPY_BLOCK_64_BACK15:
> > > +		if (n >= 32) {
> > > +			n -= 32;
> > > +			rte_mov32((uint8_t *)dst, (const uint8_t *)src);
> > > +			src = (const uint8_t *)src + 32;
> > > +			dst = (uint8_t *)dst + 32;
> > > +		}
> > > +		if (n > 16) {
> > > +			rte_mov16((uint8_t *)dst, (const uint8_t *)src);
> > > +			rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t
> > *)src - 16 + n);
> > > +			return ret;
> > > +		}
> > > +		if (n > 0) {
> > > +			rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t
> > *)src - 16 + n);
> > > +		}
> > > +		return ret;
> > > +	}
> > > +
> > > +	/**
> > > +	 * Make store aligned when copy size exceeds 512 bytes,
> > > +	 * and make sure the first 15 bytes are copied, because
> > > +	 * unaligned copy functions require up to 15 bytes
> > > +	 * backwards access.
> > > +	 */
> > > +	dstofss = 16 - (int)((long long)(void *)dst & 0x0F) + 16;
> > > +	n -= dstofss;
> > > +	rte_mov32((uint8_t *)dst, (const uint8_t *)src);
> > > +	src = (const uint8_t *)src + dstofss;
> > > +	dst = (uint8_t *)dst + dstofss;
> > > +	srcofs = (int)((long long)(const void *)src & 0x0F);
> > > +
> > > +	/**
> > > +	 * For aligned copy
> > > +	 */
> > > +	if (srcofs == 0) {
> > > +		/**
> > > +		 * Copy 256-byte blocks
> > > +		 */
> > > +		for (; n >= 256; n -= 256) {
> > > +			rte_mov256((uint8_t *)dst, (const uint8_t *)src);
> > > +			dst = (uint8_t *)dst + 256;
> > > +			src = (const uint8_t *)src + 256;
> > > +		}
> > > +
> > > +		/**
> > > +		 * Copy whatever left
> > > +		 */
> > > +		goto COPY_BLOCK_255_BACK15;
> > > +	}
> > > +
> > > +	/**
> > > +	 * For copy with unaligned load
> > > +	 */
> > > +	MOVEUNALIGNED_LEFT47(dst, src, n, srcofs);
> > > +
> > > +	/**
> > > +	 * Copy whatever left
> > > +	 */
> > > +	goto COPY_BLOCK_64_BACK15;
> > >  }
> > >
> > > +#endif /* RTE_MACHINE_CPUFLAG_AVX2 */
> > > +
> > >  #ifdef __cplusplus
> > >  }
> > >  #endif
> > > --
> > > 1.9.3

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [dpdk-dev] [PATCH v2 0/4] DPDK memcpy optimization
  2015-01-29  2:38 [dpdk-dev] [PATCH v2 0/4] DPDK memcpy optimization Zhihong Wang
                   ` (4 preceding siblings ...)
  2015-01-29  6:16 ` [dpdk-dev] [PATCH v2 0/4] DPDK memcpy optimization Fu, JingguoX
@ 2015-02-10  3:06 ` Liang, Cunming
  2015-02-16 15:57 ` De Lara Guarch, Pablo
  6 siblings, 0 replies; 12+ messages in thread
From: Liang, Cunming @ 2015-02-10  3:06 UTC (permalink / raw)
  To: Wang, Zhihong, dev



> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zhihong Wang
> Sent: Thursday, January 29, 2015 10:39 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] [PATCH v2 0/4] DPDK memcpy optimization
> 
> This patch set optimizes memcpy for DPDK for both SSE and AVX platforms.
> It also extends memcpy test coverage with unaligned cases and more test points.
> 
> Optimization techniques are summarized below:
> 
> 1. Utilize full cache bandwidth
> 
> 2. Enforce aligned stores
> 
> 3. Apply load address alignment based on architecture features
> 
> 4. Make load/store address available as early as possible
> 
> 5. General optimization techniques like inlining, branch reducing, prefetch
> pattern access
> 
> --------------
> Changes in v2:
> 
> 1. Reduced constant test cases in app/test/test_memcpy_perf.c for fast build
> 
> 2. Modified macro definition for better code readability & safety
> 
> Zhihong Wang (4):
>   app/test: Disabled VTA for memcpy test in app/test/Makefile
>   app/test: Removed unnecessary test cases in app/test/test_memcpy.c
>   app/test: Extended test coverage in app/test/test_memcpy_perf.c
>   lib/librte_eal: Optimized memcpy in arch/x86/rte_memcpy.h for both SSE
>     and AVX platforms
> 
>  app/test/Makefile                                  |   6 +
>  app/test/test_memcpy.c                             |  52 +-
>  app/test/test_memcpy_perf.c                        | 220 ++++---
>  .../common/include/arch/x86/rte_memcpy.h           | 680 +++++++++++++++-----
> -
>  4 files changed, 654 insertions(+), 304 deletions(-)
> 
> --
> 1.9.3 

Acked-by:  Cunming Liang <cunming.liang@intel.com>

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [dpdk-dev] [PATCH v2 0/4] DPDK memcpy optimization
  2015-01-29  2:38 [dpdk-dev] [PATCH v2 0/4] DPDK memcpy optimization Zhihong Wang
                   ` (5 preceding siblings ...)
  2015-02-10  3:06 ` Liang, Cunming
@ 2015-02-16 15:57 ` De Lara Guarch, Pablo
  2015-02-25 10:46   ` Thomas Monjalon
  6 siblings, 1 reply; 12+ messages in thread
From: De Lara Guarch, Pablo @ 2015-02-16 15:57 UTC (permalink / raw)
  To: Wang, Zhihong, dev



> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zhihong Wang
> Sent: Thursday, January 29, 2015 2:39 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] [PATCH v2 0/4] DPDK memcpy optimization
> 
> This patch set optimizes memcpy for DPDK for both SSE and AVX platforms.
> It also extends memcpy test coverage with unaligned cases and more test
> points.
> 
> Optimization techniques are summarized below:
> 
> 1. Utilize full cache bandwidth
> 
> 2. Enforce aligned stores
> 
> 3. Apply load address alignment based on architecture features
> 
> 4. Make load/store address available as early as possible
> 
> 5. General optimization techniques like inlining, branch reducing, prefetch
> pattern access
> 
> --------------
> Changes in v2:
> 
> 1. Reduced constant test cases in app/test/test_memcpy_perf.c for fast
> build
> 
> 2. Modified macro definition for better code readability & safety
> 
> Zhihong Wang (4):
>   app/test: Disabled VTA for memcpy test in app/test/Makefile
>   app/test: Removed unnecessary test cases in app/test/test_memcpy.c
>   app/test: Extended test coverage in app/test/test_memcpy_perf.c
>   lib/librte_eal: Optimized memcpy in arch/x86/rte_memcpy.h for both SSE
>     and AVX platforms
> 
>  app/test/Makefile                                  |   6 +
>  app/test/test_memcpy.c                             |  52 +-
>  app/test/test_memcpy_perf.c                        | 220 ++++---
>  .../common/include/arch/x86/rte_memcpy.h           | 680
> +++++++++++++++------
>  4 files changed, 654 insertions(+), 304 deletions(-)
> 
> --
> 1.9.3

Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [dpdk-dev] [PATCH v2 0/4] DPDK memcpy optimization
  2015-02-16 15:57 ` De Lara Guarch, Pablo
@ 2015-02-25 10:46   ` Thomas Monjalon
  0 siblings, 0 replies; 12+ messages in thread
From: Thomas Monjalon @ 2015-02-25 10:46 UTC (permalink / raw)
  To: Wang, Zhihong, konstantin.ananyev; +Cc: dev

> > This patch set optimizes memcpy for DPDK for both SSE and AVX platforms.
> > It also extends memcpy test coverage with unaligned cases and more test
> > points.
> > 
> > Optimization techniques are summarized below:
> > 
> > 1. Utilize full cache bandwidth
> > 
> > 2. Enforce aligned stores
> > 
> > 3. Apply load address alignment based on architecture features
> > 
> > 4. Make load/store address available as early as possible
> > 
> > 5. General optimization techniques like inlining, branch reducing, prefetch
> > pattern access
> > 
> > --------------
> > Changes in v2:
> > 
> > 1. Reduced constant test cases in app/test/test_memcpy_perf.c for fast
> > build
> > 
> > 2. Modified macro definition for better code readability & safety
> > 
> > Zhihong Wang (4):
> >   app/test: Disabled VTA for memcpy test in app/test/Makefile
> >   app/test: Removed unnecessary test cases in app/test/test_memcpy.c
> >   app/test: Extended test coverage in app/test/test_memcpy_perf.c
> >   lib/librte_eal: Optimized memcpy in arch/x86/rte_memcpy.h for both SSE
> >     and AVX platforms
> 
> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>

Applied, thanks for the great work!

Note: we are still looking for a maintainer of x86 EAL.

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2015-02-25 10:47 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-01-29  2:38 [dpdk-dev] [PATCH v2 0/4] DPDK memcpy optimization Zhihong Wang
2015-01-29  2:38 ` [dpdk-dev] [PATCH v2 1/4] app/test: Disabled VTA for memcpy test in app/test/Makefile Zhihong Wang
2015-01-29  2:38 ` [dpdk-dev] [PATCH v2 2/4] app/test: Removed unnecessary test cases in app/test/test_memcpy.c Zhihong Wang
2015-01-29  2:38 ` [dpdk-dev] [PATCH v2 3/4] app/test: Extended test coverage in app/test/test_memcpy_perf.c Zhihong Wang
2015-01-29  2:38 ` [dpdk-dev] [PATCH v2 4/4] lib/librte_eal: Optimized memcpy in arch/x86/rte_memcpy.h for both SSE and AVX platforms Zhihong Wang
2015-01-29 15:17   ` Ananyev, Konstantin
2015-01-30  5:57     ` Wang, Zhihong
2015-01-30 10:44       ` Ananyev, Konstantin
2015-01-29  6:16 ` [dpdk-dev] [PATCH v2 0/4] DPDK memcpy optimization Fu, JingguoX
2015-02-10  3:06 ` Liang, Cunming
2015-02-16 15:57 ` De Lara Guarch, Pablo
2015-02-25 10:46   ` Thomas Monjalon

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).