From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id B3127255 for ; Fri, 30 Jan 2015 11:44:20 +0100 (CET) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga103.fm.intel.com with ESMTP; 30 Jan 2015 02:38:01 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.09,491,1418112000"; d="scan'208";a="659138106" Received: from irsmsx152.ger.corp.intel.com ([163.33.192.66]) by fmsmga001.fm.intel.com with ESMTP; 30 Jan 2015 02:44:18 -0800 Received: from irsmsx105.ger.corp.intel.com ([169.254.7.81]) by IRSMSX152.ger.corp.intel.com ([169.254.6.43]) with mapi id 14.03.0195.001; Fri, 30 Jan 2015 10:44:16 +0000 From: "Ananyev, Konstantin" To: "Wang, Zhihong" , "dev@dpdk.org" Thread-Topic: [dpdk-dev] [PATCH v2 4/4] lib/librte_eal: Optimized memcpy in arch/x86/rte_memcpy.h for both SSE and AVX platforms Thread-Index: AQHQO2zhVuBnoe9lV0q9rFfXffl6zZzXMYTQgAD5z0CAAE5I4A== Date: Fri, 30 Jan 2015 10:44:15 +0000 Message-ID: <2601191342CEEE43887BDE71AB977258213E1CC1@irsmsx105.ger.corp.intel.com> References: <1422499127-11689-1-git-send-email-zhihong.wang@intel.com> <1422499127-11689-5-git-send-email-zhihong.wang@intel.com> <2601191342CEEE43887BDE71AB977258213E085B@irsmsx105.ger.corp.intel.com> In-Reply-To: Accept-Language: en-IE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [163.33.239.181] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v2 4/4] lib/librte_eal: Optimized memcpy in arch/x86/rte_memcpy.h for both SSE and AVX platforms X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jan 2015 10:44:22 -0000 Hey Zhihong, > -----Original Message----- > From: Wang, Zhihong > Sent: Friday, January 30, 2015 5:57 AM > To: Ananyev, Konstantin; dev@dpdk.org > Subject: RE: [dpdk-dev] [PATCH v2 4/4] lib/librte_eal: Optimized memcpy i= n arch/x86/rte_memcpy.h for both SSE and AVX platforms >=20 > Hey Konstantin, >=20 > This method does reduce code size but lead to significant performance dro= p. > I think we need to keep the original code. Sure, no point to make it slower. Thanks for trying it anyway. Konstantin >=20 >=20 > Thanks > Zhihong (John) >=20 >=20 > > -----Original Message----- > > From: Ananyev, Konstantin > > Sent: Thursday, January 29, 2015 11:18 PM > > To: Wang, Zhihong; dev@dpdk.org > > Subject: RE: [dpdk-dev] [PATCH v2 4/4] lib/librte_eal: Optimized memcpy= in > > arch/x86/rte_memcpy.h for both SSE and AVX platforms > > > > Hi Zhihong, > > > > > -----Original Message----- > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zhihong Wang > > > Sent: Thursday, January 29, 2015 2:39 AM > > > To: dev@dpdk.org > > > Subject: [dpdk-dev] [PATCH v2 4/4] lib/librte_eal: Optimized memcpy i= n > > > arch/x86/rte_memcpy.h for both SSE and AVX platforms > > > > > > Main code changes: > > > > > > 1. Differentiate architectural features based on CPU flags > > > > > > a. Implement separated move functions for SSE/AVX/AVX2 to make > > > full utilization of cache bandwidth > > > > > > b. Implement separated copy flow specifically optimized for targe= t > > > architecture > > > > > > 2. Rewrite the memcpy function "rte_memcpy" > > > > > > a. Add store aligning > > > > > > b. Add load aligning based on architectural features > > > > > > c. Put block copy loop into inline move functions for better > > > control of instruction order > > > > > > d. Eliminate unnecessary MOVs > > > > > > 3. Rewrite the inline move functions > > > > > > a. Add move functions for unaligned load cases > > > > > > b. Change instruction order in copy loops for better pipeline > > > utilization > > > > > > c. Use intrinsics instead of assembly code > > > > > > 4. Remove slow glibc call for constant copies > > > > > > Signed-off-by: Zhihong Wang > > > --- > > > .../common/include/arch/x86/rte_memcpy.h | 680 > > +++++++++++++++------ > > > 1 file changed, 509 insertions(+), 171 deletions(-) > > > > > > diff --git a/lib/librte_eal/common/include/arch/x86/rte_memcpy.h > > > b/lib/librte_eal/common/include/arch/x86/rte_memcpy.h > > > index fb9eba8..7b2d382 100644 > > > --- a/lib/librte_eal/common/include/arch/x86/rte_memcpy.h > > > +++ b/lib/librte_eal/common/include/arch/x86/rte_memcpy.h > > > @@ -34,166 +34,189 @@ > > > #ifndef _RTE_MEMCPY_X86_64_H_ > > > #define _RTE_MEMCPY_X86_64_H_ > > > > > > +/** > > > + * @file > > > + * > > > + * Functions for SSE/AVX/AVX2 implementation of memcpy(). > > > + */ > > > + > > > +#include > > > #include > > > #include > > > -#include > > > +#include > > > > > > #ifdef __cplusplus > > > extern "C" { > > > #endif > > > > > > -#include "generic/rte_memcpy.h" > > > +/** > > > + * Copy bytes from one location to another. The locations must not > > overlap. > > > + * > > > + * @note This is implemented as a macro, so it's address should not > > > +be taken > > > + * and care is needed as parameter expressions may be evaluated > > multiple times. > > > + * > > > + * @param dst > > > + * Pointer to the destination of the data. > > > + * @param src > > > + * Pointer to the source data. > > > + * @param n > > > + * Number of bytes to copy. > > > + * @return > > > + * Pointer to the destination data. > > > + */ > > > +static inline void * > > > +rte_memcpy(void *dst, const void *src, size_t n) > > > +__attribute__((always_inline)); > > > > > > -#ifdef __INTEL_COMPILER > > > -#pragma warning(disable:593) /* Stop unused variable warning (reg_a > > > etc). */ -#endif > > > +#ifdef RTE_MACHINE_CPUFLAG_AVX2 > > > > > > +/** > > > + * AVX2 implementation below > > > + */ > > > + > > > +/** > > > + * Copy 16 bytes from one location to another, > > > + * locations should not overlap. > > > + */ > > > static inline void > > > rte_mov16(uint8_t *dst, const uint8_t *src) { > > > - __m128i reg_a; > > > - asm volatile ( > > > - "movdqu (%[src]), %[reg_a]\n\t" > > > - "movdqu %[reg_a], (%[dst])\n\t" > > > - : [reg_a] "=3Dx" (reg_a) > > > - : [src] "r" (src), > > > - [dst] "r"(dst) > > > - : "memory" > > > - ); > > > + __m128i xmm0; > > > + > > > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > > > + _mm_storeu_si128((__m128i *)dst, xmm0); > > > } > > > > > > +/** > > > + * Copy 32 bytes from one location to another, > > > + * locations should not overlap. > > > + */ > > > static inline void > > > rte_mov32(uint8_t *dst, const uint8_t *src) { > > > - __m128i reg_a, reg_b; > > > - asm volatile ( > > > - "movdqu (%[src]), %[reg_a]\n\t" > > > - "movdqu 16(%[src]), %[reg_b]\n\t" > > > - "movdqu %[reg_a], (%[dst])\n\t" > > > - "movdqu %[reg_b], 16(%[dst])\n\t" > > > - : [reg_a] "=3Dx" (reg_a), > > > - [reg_b] "=3Dx" (reg_b) > > > - : [src] "r" (src), > > > - [dst] "r"(dst) > > > - : "memory" > > > - ); > > > -} > > > + __m256i ymm0; > > > > > > -static inline void > > > -rte_mov48(uint8_t *dst, const uint8_t *src) -{ > > > - __m128i reg_a, reg_b, reg_c; > > > - asm volatile ( > > > - "movdqu (%[src]), %[reg_a]\n\t" > > > - "movdqu 16(%[src]), %[reg_b]\n\t" > > > - "movdqu 32(%[src]), %[reg_c]\n\t" > > > - "movdqu %[reg_a], (%[dst])\n\t" > > > - "movdqu %[reg_b], 16(%[dst])\n\t" > > > - "movdqu %[reg_c], 32(%[dst])\n\t" > > > - : [reg_a] "=3Dx" (reg_a), > > > - [reg_b] "=3Dx" (reg_b), > > > - [reg_c] "=3Dx" (reg_c) > > > - : [src] "r" (src), > > > - [dst] "r"(dst) > > > - : "memory" > > > - ); > > > + ymm0 =3D _mm256_loadu_si256((const __m256i *)src); > > > + _mm256_storeu_si256((__m256i *)dst, ymm0); > > > } > > > > > > +/** > > > + * Copy 64 bytes from one location to another, > > > + * locations should not overlap. > > > + */ > > > static inline void > > > rte_mov64(uint8_t *dst, const uint8_t *src) { > > > - __m128i reg_a, reg_b, reg_c, reg_d; > > > - asm volatile ( > > > - "movdqu (%[src]), %[reg_a]\n\t" > > > - "movdqu 16(%[src]), %[reg_b]\n\t" > > > - "movdqu 32(%[src]), %[reg_c]\n\t" > > > - "movdqu 48(%[src]), %[reg_d]\n\t" > > > - "movdqu %[reg_a], (%[dst])\n\t" > > > - "movdqu %[reg_b], 16(%[dst])\n\t" > > > - "movdqu %[reg_c], 32(%[dst])\n\t" > > > - "movdqu %[reg_d], 48(%[dst])\n\t" > > > - : [reg_a] "=3Dx" (reg_a), > > > - [reg_b] "=3Dx" (reg_b), > > > - [reg_c] "=3Dx" (reg_c), > > > - [reg_d] "=3Dx" (reg_d) > > > - : [src] "r" (src), > > > - [dst] "r"(dst) > > > - : "memory" > > > - ); > > > + rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32); > > > + rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32); > > > } > > > > > > +/** > > > + * Copy 128 bytes from one location to another, > > > + * locations should not overlap. > > > + */ > > > static inline void > > > rte_mov128(uint8_t *dst, const uint8_t *src) { > > > - __m128i reg_a, reg_b, reg_c, reg_d, reg_e, reg_f, reg_g, reg_h; > > > - asm volatile ( > > > - "movdqu (%[src]), %[reg_a]\n\t" > > > - "movdqu 16(%[src]), %[reg_b]\n\t" > > > - "movdqu 32(%[src]), %[reg_c]\n\t" > > > - "movdqu 48(%[src]), %[reg_d]\n\t" > > > - "movdqu 64(%[src]), %[reg_e]\n\t" > > > - "movdqu 80(%[src]), %[reg_f]\n\t" > > > - "movdqu 96(%[src]), %[reg_g]\n\t" > > > - "movdqu 112(%[src]), %[reg_h]\n\t" > > > - "movdqu %[reg_a], (%[dst])\n\t" > > > - "movdqu %[reg_b], 16(%[dst])\n\t" > > > - "movdqu %[reg_c], 32(%[dst])\n\t" > > > - "movdqu %[reg_d], 48(%[dst])\n\t" > > > - "movdqu %[reg_e], 64(%[dst])\n\t" > > > - "movdqu %[reg_f], 80(%[dst])\n\t" > > > - "movdqu %[reg_g], 96(%[dst])\n\t" > > > - "movdqu %[reg_h], 112(%[dst])\n\t" > > > - : [reg_a] "=3Dx" (reg_a), > > > - [reg_b] "=3Dx" (reg_b), > > > - [reg_c] "=3Dx" (reg_c), > > > - [reg_d] "=3Dx" (reg_d), > > > - [reg_e] "=3Dx" (reg_e), > > > - [reg_f] "=3Dx" (reg_f), > > > - [reg_g] "=3Dx" (reg_g), > > > - [reg_h] "=3Dx" (reg_h) > > > - : [src] "r" (src), > > > - [dst] "r"(dst) > > > - : "memory" > > > - ); > > > + rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32); > > > + rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32); > > > + rte_mov32((uint8_t *)dst + 2 * 32, (const uint8_t *)src + 2 * 32); > > > + rte_mov32((uint8_t *)dst + 3 * 32, (const uint8_t *)src + 3 * 32); > > > } > > > > > > -#ifdef __INTEL_COMPILER > > > -#pragma warning(enable:593) > > > -#endif > > > - > > > +/** > > > + * Copy 256 bytes from one location to another, > > > + * locations should not overlap. > > > + */ > > > static inline void > > > rte_mov256(uint8_t *dst, const uint8_t *src) { > > > - rte_mov128(dst, src); > > > - rte_mov128(dst + 128, src + 128); > > > + rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32); > > > + rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32); > > > + rte_mov32((uint8_t *)dst + 2 * 32, (const uint8_t *)src + 2 * 32); > > > + rte_mov32((uint8_t *)dst + 3 * 32, (const uint8_t *)src + 3 * 32); > > > + rte_mov32((uint8_t *)dst + 4 * 32, (const uint8_t *)src + 4 * 32); > > > + rte_mov32((uint8_t *)dst + 5 * 32, (const uint8_t *)src + 5 * 32); > > > + rte_mov32((uint8_t *)dst + 6 * 32, (const uint8_t *)src + 6 * 32); > > > + rte_mov32((uint8_t *)dst + 7 * 32, (const uint8_t *)src + 7 * 32); > > > } > > > > > > -#define rte_memcpy(dst, src, n) \ > > > - ({ (__builtin_constant_p(n)) ? \ > > > - memcpy((dst), (src), (n)) : \ > > > - rte_memcpy_func((dst), (src), (n)); }) > > > +/** > > > + * Copy 64-byte blocks from one location to another, > > > + * locations should not overlap. > > > + */ > > > +static inline void > > > +rte_mov64blocks(uint8_t *dst, const uint8_t *src, size_t n) { > > > + __m256i ymm0, ymm1; > > > + > > > + while (n >=3D 64) { > > > + ymm0 =3D _mm256_loadu_si256((const __m256i *)((const > > uint8_t *)src + 0 * 32)); > > > + n -=3D 64; > > > + ymm1 =3D _mm256_loadu_si256((const __m256i *)((const > > uint8_t *)src + 1 * 32)); > > > + src =3D (const uint8_t *)src + 64; > > > + _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 0 * 32), > > ymm0); > > > + _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 1 * 32), > > ymm1); > > > + dst =3D (uint8_t *)dst + 64; > > > + } > > > +} > > > + > > > +/** > > > + * Copy 256-byte blocks from one location to another, > > > + * locations should not overlap. > > > + */ > > > +static inline void > > > +rte_mov256blocks(uint8_t *dst, const uint8_t *src, size_t n) { > > > + __m256i ymm0, ymm1, ymm2, ymm3, ymm4, ymm5, ymm6, ymm7; > > > + > > > + while (n >=3D 256) { > > > + ymm0 =3D _mm256_loadu_si256((const __m256i *)((const > > uint8_t *)src + 0 * 32)); > > > + n -=3D 256; > > > + ymm1 =3D _mm256_loadu_si256((const __m256i *)((const > > uint8_t *)src + 1 * 32)); > > > + ymm2 =3D _mm256_loadu_si256((const __m256i *)((const > > uint8_t *)src + 2 * 32)); > > > + ymm3 =3D _mm256_loadu_si256((const __m256i *)((const > > uint8_t *)src + 3 * 32)); > > > + ymm4 =3D _mm256_loadu_si256((const __m256i *)((const > > uint8_t *)src + 4 * 32)); > > > + ymm5 =3D _mm256_loadu_si256((const __m256i *)((const > > uint8_t *)src + 5 * 32)); > > > + ymm6 =3D _mm256_loadu_si256((const __m256i *)((const > > uint8_t *)src + 6 * 32)); > > > + ymm7 =3D _mm256_loadu_si256((const __m256i *)((const > > uint8_t *)src + 7 * 32)); > > > + src =3D (const uint8_t *)src + 256; > > > + _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 0 * 32), > > ymm0); > > > + _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 1 * 32), > > ymm1); > > > + _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 2 * 32), > > ymm2); > > > + _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 3 * 32), > > ymm3); > > > + _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 4 * 32), > > ymm4); > > > + _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 5 * 32), > > ymm5); > > > + _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 6 * 32), > > ymm6); > > > + _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 7 * 32), > > ymm7); > > > + dst =3D (uint8_t *)dst + 256; > > > + } > > > +} > > > > > > static inline void * > > > -rte_memcpy_func(void *dst, const void *src, size_t n) > > > +rte_memcpy(void *dst, const void *src, size_t n) > > > { > > > void *ret =3D dst; > > > + int dstofss; > > > + int bits; > > > > > > - /* We can't copy < 16 bytes using XMM registers so do it manually. = */ > > > + /** > > > + * Copy less than 16 bytes > > > + */ > > > if (n < 16) { > > > if (n & 0x01) { > > > *(uint8_t *)dst =3D *(const uint8_t *)src; > > > - dst =3D (uint8_t *)dst + 1; > > > src =3D (const uint8_t *)src + 1; > > > + dst =3D (uint8_t *)dst + 1; > > > } > > > if (n & 0x02) { > > > *(uint16_t *)dst =3D *(const uint16_t *)src; > > > - dst =3D (uint16_t *)dst + 1; > > > src =3D (const uint16_t *)src + 1; > > > + dst =3D (uint16_t *)dst + 1; > > > } > > > if (n & 0x04) { > > > *(uint32_t *)dst =3D *(const uint32_t *)src; > > > - dst =3D (uint32_t *)dst + 1; > > > src =3D (const uint32_t *)src + 1; > > > + dst =3D (uint32_t *)dst + 1; > > > } > > > if (n & 0x08) { > > > *(uint64_t *)dst =3D *(const uint64_t *)src; @@ -201,95 > > +224,410 @@ > > > rte_memcpy_func(void *dst, const void *src, size_t n) > > > return ret; > > > } > > > > > > - /* Special fast cases for <=3D 128 bytes */ > > > + /** > > > + * Fast way when copy size doesn't exceed 512 bytes > > > + */ > > > if (n <=3D 32) { > > > rte_mov16((uint8_t *)dst, (const uint8_t *)src); > > > rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + > > n); > > > return ret; > > > } > > > - > > > if (n <=3D 64) { > > > rte_mov32((uint8_t *)dst, (const uint8_t *)src); > > > rte_mov32((uint8_t *)dst - 32 + n, (const uint8_t *)src - 32 + > > n); > > > return ret; > > > } > > > - > > > - if (n <=3D 128) { > > > - rte_mov64((uint8_t *)dst, (const uint8_t *)src); > > > - rte_mov64((uint8_t *)dst - 64 + n, (const uint8_t *)src - 64 + > > n); > > > + if (n <=3D 512) { > > > + if (n >=3D 256) { > > > + n -=3D 256; > > > + rte_mov256((uint8_t *)dst, (const uint8_t *)src); > > > + src =3D (const uint8_t *)src + 256; > > > + dst =3D (uint8_t *)dst + 256; > > > + } > > > + if (n >=3D 128) { > > > + n -=3D 128; > > > + rte_mov128((uint8_t *)dst, (const uint8_t *)src); > > > + src =3D (const uint8_t *)src + 128; > > > + dst =3D (uint8_t *)dst + 128; > > > + } > > > + if (n >=3D 64) { > > > + n -=3D 64; > > > + rte_mov64((uint8_t *)dst, (const uint8_t *)src); > > > + src =3D (const uint8_t *)src + 64; > > > + dst =3D (uint8_t *)dst + 64; > > > + } > > > +COPY_BLOCK_64_BACK31: > > > + if (n > 32) { > > > + rte_mov32((uint8_t *)dst, (const uint8_t *)src); > > > + rte_mov32((uint8_t *)dst - 32 + n, (const uint8_t > > *)src - 32 + n); > > > + return ret; > > > + } > > > + if (n > 0) { > > > + rte_mov32((uint8_t *)dst - 32 + n, (const uint8_t > > *)src - 32 + n); > > > + } > > > return ret; > > > } > > > > > > - /* > > > - * For large copies > 128 bytes. This combination of 256, 64 and 16 > > byte > > > - * copies was found to be faster than doing 128 and 32 byte copies = as > > > - * well. > > > + /** > > > + * Make store aligned when copy size exceeds 512 bytes > > > */ > > > - for ( ; n >=3D 256; n -=3D 256) { > > > - rte_mov256((uint8_t *)dst, (const uint8_t *)src); > > > - dst =3D (uint8_t *)dst + 256; > > > - src =3D (const uint8_t *)src + 256; > > > + dstofss =3D 32 - (int)((long long)(void *)dst & 0x1F); > > > + n -=3D dstofss; > > > + rte_mov32((uint8_t *)dst, (const uint8_t *)src); > > > + src =3D (const uint8_t *)src + dstofss; > > > + dst =3D (uint8_t *)dst + dstofss; > > > + > > > + /** > > > + * Copy 256-byte blocks. > > > + * Use copy block function for better instruction order control, > > > + * which is important when load is unaligned. > > > + */ > > > + rte_mov256blocks((uint8_t *)dst, (const uint8_t *)src, n); > > > + bits =3D n; > > > + n =3D n & 255; > > > + bits -=3D n; > > > + src =3D (const uint8_t *)src + bits; > > > + dst =3D (uint8_t *)dst + bits; > > > + > > > + /** > > > + * Copy 64-byte blocks. > > > + * Use copy block function for better instruction order control, > > > + * which is important when load is unaligned. > > > + */ > > > + if (n >=3D 64) { > > > + rte_mov64blocks((uint8_t *)dst, (const uint8_t *)src, n); > > > + bits =3D n; > > > + n =3D n & 63; > > > + bits -=3D n; > > > + src =3D (const uint8_t *)src + bits; > > > + dst =3D (uint8_t *)dst + bits; > > > } > > > > > > - /* > > > - * We split the remaining bytes (which will be less than 256) into > > > - * 64byte (2^6) chunks. > > > - * Using incrementing integers in the case labels of a switch > > statement > > > - * enourages the compiler to use a jump table. To get incrementing > > > - * integers, we shift the 2 relevant bits to the LSB position to fi= rst > > > - * get decrementing integers, and then subtract. > > > + /** > > > + * Copy whatever left > > > */ > > > - switch (3 - (n >> 6)) { > > > - case 0x00: > > > - rte_mov64((uint8_t *)dst, (const uint8_t *)src); > > > - n -=3D 64; > > > - dst =3D (uint8_t *)dst + 64; > > > - src =3D (const uint8_t *)src + 64; /* fallthrough */ > > > - case 0x01: > > > - rte_mov64((uint8_t *)dst, (const uint8_t *)src); > > > - n -=3D 64; > > > - dst =3D (uint8_t *)dst + 64; > > > - src =3D (const uint8_t *)src + 64; /* fallthrough */ > > > - case 0x02: > > > - rte_mov64((uint8_t *)dst, (const uint8_t *)src); > > > - n -=3D 64; > > > - dst =3D (uint8_t *)dst + 64; > > > - src =3D (const uint8_t *)src + 64; /* fallthrough */ > > > - default: > > > - ; > > > + goto COPY_BLOCK_64_BACK31; > > > +} > > > + > > > +#else /* RTE_MACHINE_CPUFLAG_AVX2 */ > > > + > > > +/** > > > + * SSE & AVX implementation below > > > + */ > > > + > > > +/** > > > + * Copy 16 bytes from one location to another, > > > + * locations should not overlap. > > > + */ > > > +static inline void > > > +rte_mov16(uint8_t *dst, const uint8_t *src) { > > > + __m128i xmm0; > > > + > > > + xmm0 =3D _mm_loadu_si128((const __m128i *)(const __m128i *)src); > > > + _mm_storeu_si128((__m128i *)dst, xmm0); } > > > + > > > +/** > > > + * Copy 32 bytes from one location to another, > > > + * locations should not overlap. > > > + */ > > > +static inline void > > > +rte_mov32(uint8_t *dst, const uint8_t *src) { > > > + rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16); > > > + rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16); = } > > > + > > > +/** > > > + * Copy 64 bytes from one location to another, > > > + * locations should not overlap. > > > + */ > > > +static inline void > > > +rte_mov64(uint8_t *dst, const uint8_t *src) { > > > + rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16); > > > + rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16); > > > + rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16); > > > + rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16); = } > > > + > > > +/** > > > + * Copy 128 bytes from one location to another, > > > + * locations should not overlap. > > > + */ > > > +static inline void > > > +rte_mov128(uint8_t *dst, const uint8_t *src) { > > > + rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16); > > > + rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16); > > > + rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16); > > > + rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16); > > > + rte_mov16((uint8_t *)dst + 4 * 16, (const uint8_t *)src + 4 * 16); > > > + rte_mov16((uint8_t *)dst + 5 * 16, (const uint8_t *)src + 5 * 16); > > > + rte_mov16((uint8_t *)dst + 6 * 16, (const uint8_t *)src + 6 * 16); > > > + rte_mov16((uint8_t *)dst + 7 * 16, (const uint8_t *)src + 7 * 16); = } > > > + > > > +/** > > > + * Copy 256 bytes from one location to another, > > > + * locations should not overlap. > > > + */ > > > +static inline void > > > +rte_mov256(uint8_t *dst, const uint8_t *src) { > > > + rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16); > > > + rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16); > > > + rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16); > > > + rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16); > > > + rte_mov16((uint8_t *)dst + 4 * 16, (const uint8_t *)src + 4 * 16); > > > + rte_mov16((uint8_t *)dst + 5 * 16, (const uint8_t *)src + 5 * 16); > > > + rte_mov16((uint8_t *)dst + 6 * 16, (const uint8_t *)src + 6 * 16); > > > + rte_mov16((uint8_t *)dst + 7 * 16, (const uint8_t *)src + 7 * 16); > > > + rte_mov16((uint8_t *)dst + 8 * 16, (const uint8_t *)src + 8 * 16); > > > + rte_mov16((uint8_t *)dst + 9 * 16, (const uint8_t *)src + 9 * 16); > > > + rte_mov16((uint8_t *)dst + 10 * 16, (const uint8_t *)src + 10 * 16)= ; > > > + rte_mov16((uint8_t *)dst + 11 * 16, (const uint8_t *)src + 11 * 16)= ; > > > + rte_mov16((uint8_t *)dst + 12 * 16, (const uint8_t *)src + 12 * 16)= ; > > > + rte_mov16((uint8_t *)dst + 13 * 16, (const uint8_t *)src + 13 * 16)= ; > > > + rte_mov16((uint8_t *)dst + 14 * 16, (const uint8_t *)src + 14 * 16)= ; > > > + rte_mov16((uint8_t *)dst + 15 * 16, (const uint8_t *)src + 15 * 16)= ; > > > +} > > > + > > > +/** > > > + * Macro for copying unaligned block from one location to another > > > +with constant load offset, > > > + * 47 bytes leftover maximum, > > > + * locations should not overlap. > > > + * Requirements: > > > + * - Store is aligned > > > + * - Load offset is , which must be immediate value within > > > +[1, 15] > > > + * - For , make sure bit backwards & <16 - offset> bit > > > +forwards are available for loading > > > + * - , , must be variables > > > + * - __m128i ~ must be pre-defined */ > > > +#define MOVEUNALIGNED_LEFT47_IMM(dst, src, len, offset) > > \ > > > +({ = \ > > > + int tmp; = \ > > > + while (len >=3D 128 + 16 - offset) { = \ > > > + xmm0 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)= src - > > offset + 0 * 16)); \ > > > + len -=3D 128; = \ > > > + xmm1 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)= src - > > offset + 1 * 16)); \ > > > + xmm2 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)= src - > > offset + 2 * 16)); \ > > > + xmm3 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)= src - > > offset + 3 * 16)); \ > > > + xmm4 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)= src - > > offset + 4 * 16)); \ > > > + xmm5 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)= src - > > offset + 5 * 16)); \ > > > + xmm6 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)= src - > > offset + 6 * 16)); \ > > > + xmm7 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)= src - > > offset + 7 * 16)); \ > > > + xmm8 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)= src - > > offset + 8 * 16)); \ > > > + src =3D (const uint8_t *)src + 128; = \ > > > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16), > > _mm_alignr_epi8(xmm1, xmm0, offset)); \ > > > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16), > > _mm_alignr_epi8(xmm2, xmm1, offset)); \ > > > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 2 * 16), > > _mm_alignr_epi8(xmm3, xmm2, offset)); \ > > > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 3 * 16), > > _mm_alignr_epi8(xmm4, xmm3, offset)); \ > > > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 4 * 16), > > _mm_alignr_epi8(xmm5, xmm4, offset)); \ > > > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 5 * 16), > > _mm_alignr_epi8(xmm6, xmm5, offset)); \ > > > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 6 * 16), > > _mm_alignr_epi8(xmm7, xmm6, offset)); \ > > > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 7 * 16), > > _mm_alignr_epi8(xmm8, xmm7, offset)); \ > > > + dst =3D (uint8_t *)dst + 128; = \ > > > + } = \ > > > + tmp =3D len; = \ > > > + len =3D ((len - 16 + offset) & 127) + 16 - offset; = \ > > > + tmp -=3D len; = \ > > > + src =3D (const uint8_t *)src + tmp; = \ > > > + dst =3D (uint8_t *)dst + tmp; = \ > > > + if (len >=3D 32 + 16 - offset) { = \ > > > + while (len >=3D 32 + 16 - offset) { = \ > > > + xmm0 =3D _mm_loadu_si128((const __m128i *)((const uint8_= t *)src - > > offset + 0 * 16)); \ > > > + len -=3D 32; = \ > > > + xmm1 =3D _mm_loadu_si128((const __m128i *)((const uint8_= t *)src - > > offset + 1 * 16)); \ > > > + xmm2 =3D _mm_loadu_si128((const __m128i *)((const uint8_= t *)src - > > offset + 2 * 16)); \ > > > + src =3D (const uint8_t *)src + 32; = \ > > > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16), > > _mm_alignr_epi8(xmm1, xmm0, offset)); \ > > > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16), > > _mm_alignr_epi8(xmm2, xmm1, offset)); \ > > > + dst =3D (uint8_t *)dst + 32; = \ > > > + } = \ > > > + tmp =3D len; = \ > > > + len =3D ((len - 16 + offset) & 31) + 16 - offset; = \ > > > + tmp -=3D len; = \ > > > + src =3D (const uint8_t *)src + tmp; = \ > > > + dst =3D (uint8_t *)dst + tmp; = \ > > > + } = \ > > > +}) > > > + > > > +/** > > > + * Macro for copying unaligned block from one location to another, > > > + * 47 bytes leftover maximum, > > > + * locations should not overlap. > > > + * Use switch here because the aligning instruction requires immedia= te > > value for shift count. > > > + * Requirements: > > > + * - Store is aligned > > > + * - Load offset is , which must be within [1, 15] > > > + * - For , make sure bit backwards & <16 - offset> bit > > > +forwards are available for loading > > > + * - , , must be variables > > > + * - __m128i ~ used in MOVEUNALIGNED_LEFT47_IMM > > must be > > > +pre-defined */ > > > +#define MOVEUNALIGNED_LEFT47(dst, src, len, offset) = \ > > > +({ = \ > > > + switch (offset) { = \ > > > + case 0x01: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x01); break; = \ > > > + case 0x02: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x02); break; = \ > > > + case 0x03: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x03); break; = \ > > > + case 0x04: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x04); break; = \ > > > + case 0x05: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x05); break; = \ > > > + case 0x06: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x06); break; = \ > > > + case 0x07: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x07); break; = \ > > > + case 0x08: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x08); break; = \ > > > + case 0x09: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x09); break; = \ > > > + case 0x0A: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0A); break; = \ > > > + case 0x0B: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0B); break; = \ > > > + case 0x0C: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0C); break; = \ > > > + case 0x0D: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0D); break; = \ > > > + case 0x0E: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0E); break; = \ > > > + case 0x0F: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0F); break; = \ > > > + default:; = \ > > > + } = \ > > > +}) > > > > We probably didn't understand each other properly. > > My thought was to do something like: > > > > #define MOVEUNALIGNED_128(offset) do { \ > > _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16), > > _mm_alignr_epi8(xmm1, xmm0, offset)); \ > > _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16), > > _mm_alignr_epi8(xmm2, xmm1, offset)); \ > > _mm_storeu_si128((__m128i *)((uint8_t *)dst + 2 * 16), > > _mm_alignr_epi8(xmm3, xmm2, offset)); \ > > _mm_storeu_si128((__m128i *)((uint8_t *)dst + 3 * 16), > > _mm_alignr_epi8(xmm4, xmm3, offset)); \ > > _mm_storeu_si128((__m128i *)((uint8_t *)dst + 4 * 16), > > _mm_alignr_epi8(xmm5, xmm4, offset)); \ > > _mm_storeu_si128((__m128i *)((uint8_t *)dst + 5 * 16), > > _mm_alignr_epi8(xmm6, xmm5, offset)); \ > > _mm_storeu_si128((__m128i *)((uint8_t *)dst + 6 * 16), > > _mm_alignr_epi8(xmm7, xmm6, offset)); \ > > _mm_storeu_si128((__m128i *)((uint8_t *)dst + 7 * 16), > > _mm_alignr_epi8(xmm8, xmm7, offset)); \ > > } while(0); > > > > > > Then at MOVEUNALIGNED_LEFT47_IMM: > > .... > > xmm7 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offse= t + 7 > > * 16)); \ > > xmm8 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offse= t + 8 > > * 16)); \ > > src =3D (const uint8_t *)src + 128; > > > > switch(offset) { > > case 0x01: MOVEUNALIGNED_128(1); break; > > ... > > case 0x0f: MOVEUNALIGNED_128(f); break; } ... > > dst =3D (uint8_t *)dst + 128 > > > > An then in rte_memcpy() you don't need a switch for > > MOVEUNALIGNED_LEFT47_IMM itself. > > Thought it might help to generate smaller code for rte_ememcpy() while > > keeping performance reasonably high. > > > > Konstantin > > > > > > > + > > > +static inline void * > > > +rte_memcpy(void *dst, const void *src, size_t n) { > > > + __m128i xmm0, xmm1, xmm2, xmm3, xmm4, xmm5, xmm6, xmm7, > > xmm8; > > > + void *ret =3D dst; > > > + int dstofss; > > > + int srcofs; > > > + > > > + /** > > > + * Copy less than 16 bytes > > > + */ > > > + if (n < 16) { > > > + if (n & 0x01) { > > > + *(uint8_t *)dst =3D *(const uint8_t *)src; > > > + src =3D (const uint8_t *)src + 1; > > > + dst =3D (uint8_t *)dst + 1; > > > + } > > > + if (n & 0x02) { > > > + *(uint16_t *)dst =3D *(const uint16_t *)src; > > > + src =3D (const uint16_t *)src + 1; > > > + dst =3D (uint16_t *)dst + 1; > > > + } > > > + if (n & 0x04) { > > > + *(uint32_t *)dst =3D *(const uint32_t *)src; > > > + src =3D (const uint32_t *)src + 1; > > > + dst =3D (uint32_t *)dst + 1; > > > + } > > > + if (n & 0x08) { > > > + *(uint64_t *)dst =3D *(const uint64_t *)src; > > > + } > > > + return ret; > > > } > > > > > > - /* > > > - * We split the remaining bytes (which will be less than 64) into > > > - * 16byte (2^4) chunks, using the same switch structure as above. > > > + /** > > > + * Fast way when copy size doesn't exceed 512 bytes > > > */ > > > - switch (3 - (n >> 4)) { > > > - case 0x00: > > > - rte_mov16((uint8_t *)dst, (const uint8_t *)src); > > > - n -=3D 16; > > > - dst =3D (uint8_t *)dst + 16; > > > - src =3D (const uint8_t *)src + 16; /* fallthrough */ > > > - case 0x01: > > > - rte_mov16((uint8_t *)dst, (const uint8_t *)src); > > > - n -=3D 16; > > > - dst =3D (uint8_t *)dst + 16; > > > - src =3D (const uint8_t *)src + 16; /* fallthrough */ > > > - case 0x02: > > > + if (n <=3D 32) { > > > rte_mov16((uint8_t *)dst, (const uint8_t *)src); > > > - n -=3D 16; > > > - dst =3D (uint8_t *)dst + 16; > > > - src =3D (const uint8_t *)src + 16; /* fallthrough */ > > > - default: > > > - ; > > > + rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + > > n); > > > + return ret; > > > } > > > - > > > - /* Copy any remaining bytes, without going beyond end of buffers > > */ > > > - if (n !=3D 0) { > > > + if (n <=3D 48) { > > > + rte_mov32((uint8_t *)dst, (const uint8_t *)src); > > > + rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + > > n); > > > + return ret; > > > + } > > > + if (n <=3D 64) { > > > + rte_mov32((uint8_t *)dst, (const uint8_t *)src); > > > + rte_mov16((uint8_t *)dst + 32, (const uint8_t *)src + 32); > > > rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + > > n); > > > + return ret; > > > + } > > > + if (n <=3D 128) { > > > + goto COPY_BLOCK_128_BACK15; > > > } > > > - return ret; > > > + if (n <=3D 512) { > > > + if (n >=3D 256) { > > > + n -=3D 256; > > > + rte_mov128((uint8_t *)dst, (const uint8_t *)src); > > > + rte_mov128((uint8_t *)dst + 128, (const uint8_t *)src > > + 128); > > > + src =3D (const uint8_t *)src + 256; > > > + dst =3D (uint8_t *)dst + 256; > > > + } > > > +COPY_BLOCK_255_BACK15: > > > + if (n >=3D 128) { > > > + n -=3D 128; > > > + rte_mov128((uint8_t *)dst, (const uint8_t *)src); > > > + src =3D (const uint8_t *)src + 128; > > > + dst =3D (uint8_t *)dst + 128; > > > + } > > > +COPY_BLOCK_128_BACK15: > > > + if (n >=3D 64) { > > > + n -=3D 64; > > > + rte_mov64((uint8_t *)dst, (const uint8_t *)src); > > > + src =3D (const uint8_t *)src + 64; > > > + dst =3D (uint8_t *)dst + 64; > > > + } > > > +COPY_BLOCK_64_BACK15: > > > + if (n >=3D 32) { > > > + n -=3D 32; > > > + rte_mov32((uint8_t *)dst, (const uint8_t *)src); > > > + src =3D (const uint8_t *)src + 32; > > > + dst =3D (uint8_t *)dst + 32; > > > + } > > > + if (n > 16) { > > > + rte_mov16((uint8_t *)dst, (const uint8_t *)src); > > > + rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t > > *)src - 16 + n); > > > + return ret; > > > + } > > > + if (n > 0) { > > > + rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t > > *)src - 16 + n); > > > + } > > > + return ret; > > > + } > > > + > > > + /** > > > + * Make store aligned when copy size exceeds 512 bytes, > > > + * and make sure the first 15 bytes are copied, because > > > + * unaligned copy functions require up to 15 bytes > > > + * backwards access. > > > + */ > > > + dstofss =3D 16 - (int)((long long)(void *)dst & 0x0F) + 16; > > > + n -=3D dstofss; > > > + rte_mov32((uint8_t *)dst, (const uint8_t *)src); > > > + src =3D (const uint8_t *)src + dstofss; > > > + dst =3D (uint8_t *)dst + dstofss; > > > + srcofs =3D (int)((long long)(const void *)src & 0x0F); > > > + > > > + /** > > > + * For aligned copy > > > + */ > > > + if (srcofs =3D=3D 0) { > > > + /** > > > + * Copy 256-byte blocks > > > + */ > > > + for (; n >=3D 256; n -=3D 256) { > > > + rte_mov256((uint8_t *)dst, (const uint8_t *)src); > > > + dst =3D (uint8_t *)dst + 256; > > > + src =3D (const uint8_t *)src + 256; > > > + } > > > + > > > + /** > > > + * Copy whatever left > > > + */ > > > + goto COPY_BLOCK_255_BACK15; > > > + } > > > + > > > + /** > > > + * For copy with unaligned load > > > + */ > > > + MOVEUNALIGNED_LEFT47(dst, src, n, srcofs); > > > + > > > + /** > > > + * Copy whatever left > > > + */ > > > + goto COPY_BLOCK_64_BACK15; > > > } > > > > > > +#endif /* RTE_MACHINE_CPUFLAG_AVX2 */ > > > + > > > #ifdef __cplusplus > > > } > > > #endif > > > -- > > > 1.9.3