From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 8C5841B26C for ; Mon, 2 Oct 2017 18:39:05 +0200 (CEST) Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 02 Oct 2017 09:39:04 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.42,470,1500966000"; d="scan'208";a="157912354" Received: from irsmsx101.ger.corp.intel.com ([163.33.3.153]) by fmsmga005.fm.intel.com with ESMTP; 02 Oct 2017 09:39:02 -0700 Received: from irsmsx103.ger.corp.intel.com ([169.254.3.49]) by IRSMSX101.ger.corp.intel.com ([169.254.1.22]) with mapi id 14.03.0319.002; Mon, 2 Oct 2017 17:39:01 +0100 From: "Ananyev, Konstantin" To: "Li, Xiaoyun" , "Richardson, Bruce" CC: "Lu, Wenzhuo" , "Zhang, Helin" , "dev@dpdk.org" Thread-Topic: [PATCH v4 1/3] eal/x86: run-time dispatch over memcpy Thread-Index: AQHTO5m/ev7m6EWN7Eqz5p6v1EkKxKLQwURA Date: Mon, 2 Oct 2017 16:39:01 +0000 Message-ID: <2601191342CEEE43887BDE71AB9772585FAA2F94@IRSMSX103.ger.corp.intel.com> References: <1506411689-94690-1-git-send-email-xiaoyun.li@intel.com> <1506960796-71620-1-git-send-email-xiaoyun.li@intel.com> <1506960796-71620-2-git-send-email-xiaoyun.li@intel.com> In-Reply-To: <1506960796-71620-2-git-send-email-xiaoyun.li@intel.com> Accept-Language: en-IE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiYjBkMzkyYzgtZDVjOS00OTkwLTg3YjItMjU2MGFlYjNhNzRmIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX0lDIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE2LjUuOS4zIiwiVHJ1c3RlZExhYmVsSGFzaCI6ImRBZ25yQmdWQWl5YnFpcjNQazJaOXRxVTNhTGJCUXcxRUtGMUZtWHpxc0U9In0= x-ctpclassification: CTP_IC dlp-product: dlpe-windows dlp-version: 11.0.0.116 dlp-reaction: no-action x-originating-ip: [163.33.239.180] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v4 1/3] eal/x86: run-time dispatch over memcpy X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 16:39:07 -0000 > -----Original Message----- > From: Li, Xiaoyun > Sent: Monday, October 2, 2017 5:13 PM > To: Ananyev, Konstantin ; Richardson, Bruce= > Cc: Lu, Wenzhuo ; Zhang, Helin ; dev@dpdk.org; Li, Xiaoyun > Subject: [PATCH v4 1/3] eal/x86: run-time dispatch over memcpy >=20 > This patch dynamically selects functions of memcpy at run-time based > on CPU flags that current machine supports. This patch uses function > pointers which are bind to the relative functions at constrctor time. > In addition, AVX512 instructions set would be compiled only if users > config it enabled and the compiler supports it. >=20 > Signed-off-by: Xiaoyun Li > --- > v2 > * Use gcc function multi-versioning to avoid compilation issues. > * Add macros for AVX512 and AVX2. Only if users enable AVX512 and the > compiler supports it, the AVX512 codes would be compiled. Only if the > compiler supports AVX2, the AVX2 codes would be compiled. >=20 > v3 > * Reduce function calls via only keep rte_memcpy_xxx. > * Add conditions that when copy size is small, use inline code path. > Otherwise, use dynamic code path. > * To support attribute target, clang version must be greater than 3.7. > Otherwise, would choose SSE/AVX code path, the same as before. > * Move two mocro functions to the top of the code since they would be > used in inline SSE/AVX and dynamic SSE/AVX codes. >=20 > v4 > * Modify rte_memcpy.h to several .c files and modify makefiles to compile > AVX2 and AVX512 files. Could you explain to me why instead of reusing existing rte_memcpy() code to generate _sse/_avx2/ax512f flavors you keep pushing changes with 3 separ= ate implementations? Obviously that is much more expensive in terms of maintenance and doesn't l= ook like feasible solution to me. Is existing rte_memcpy() implementation is not good enough in terms of func= tionality and/or performance? If so, can you outline these problems and try to fix them first. Konstantin >=20 > lib/librte_eal/bsdapp/eal/Makefile | 17 + > .../common/include/arch/x86/rte_memcpy.c | 59 ++ > .../common/include/arch/x86/rte_memcpy.h | 861 +--------------= ---- > .../common/include/arch/x86/rte_memcpy_avx2.c | 291 +++++++ > .../common/include/arch/x86/rte_memcpy_avx512f.c | 316 +++++++ > .../common/include/arch/x86/rte_memcpy_internal.h | 909 +++++++++++++++= ++++++ > .../common/include/arch/x86/rte_memcpy_sse.c | 585 +++++++++++++ > lib/librte_eal/linuxapp/eal/Makefile | 17 + > mk/rte.cpuflags.mk | 14 + > 9 files changed, 2223 insertions(+), 846 deletions(-) > create mode 100644 lib/librte_eal/common/include/arch/x86/rte_memcpy.c > create mode 100644 lib/librte_eal/common/include/arch/x86/rte_memcpy_avx= 2.c > create mode 100644 lib/librte_eal/common/include/arch/x86/rte_memcpy_avx= 512f.c > create mode 100644 lib/librte_eal/common/include/arch/x86/rte_memcpy_int= ernal.h > create mode 100644 lib/librte_eal/common/include/arch/x86/rte_memcpy_sse= .c >=20 > diff --git a/lib/librte_eal/bsdapp/eal/Makefile b/lib/librte_eal/bsdapp/e= al/Makefile > index 005019e..27023c6 100644 > --- a/lib/librte_eal/bsdapp/eal/Makefile > +++ b/lib/librte_eal/bsdapp/eal/Makefile > @@ -36,6 +36,7 @@ LIB =3D librte_eal.a > ARCH_DIR ?=3D $(RTE_ARCH) > VPATH +=3D $(RTE_SDK)/lib/librte_eal/common > VPATH +=3D $(RTE_SDK)/lib/librte_eal/common/arch/$(ARCH_DIR) > +VPATH +=3D $(RTE_SDK)/lib/librte_eal/common/include/arch/$(ARCH_DIR) >=20 > CFLAGS +=3D -I$(SRCDIR)/include > CFLAGS +=3D -I$(RTE_SDK)/lib/librte_eal/common > @@ -93,6 +94,22 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) +=3D rte_service.c > SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) +=3D rte_cpuflags.c > SRCS-$(CONFIG_RTE_ARCH_X86) +=3D rte_spinlock.c >=20 > +# for run-time dispatch of memcpy > +SRCS-$(CONFIG_RTE_ARCH_X86) +=3D rte_memcpy.c > +SRCS-$(CONFIG_RTE_ARCH_X86) +=3D rte_memcpy_sse.c > + > +# if the compiler supports AVX512, add avx512 file > +ifneq ($(filter $(MACHINE_CFLAGS),CC_SUPPORT_AVX512F),) > +SRCS-$(CONFIG_RTE_ARCH_X86) +=3D rte_memcpy_avx512f.c > +CFLAGS_rte_memcpy_avx512f.o +=3D -mavx512f > +endif > + > +# if the compiler supports AVX2, add avx2 file > +ifneq ($(filter $(MACHINE_CFLAGS),CC_SUPPORT_AVX2),) > +SRCS-$(CONFIG_RTE_ARCH_X86) +=3D rte_memcpy_avx2.c > +CFLAGS_rte_memcpy_avx2.o +=3D -mavx2 > +endif > + > CFLAGS_eal_common_cpuflags.o :=3D $(CPUFLAGS_LIST) >=20 > CFLAGS_eal.o :=3D -D_GNU_SOURCE > diff --git a/lib/librte_eal/common/include/arch/x86/rte_memcpy.c b/lib/li= brte_eal/common/include/arch/x86/rte_memcpy.c > new file mode 100644 > index 0000000..74ae702 > --- /dev/null > +++ b/lib/librte_eal/common/include/arch/x86/rte_memcpy.c > @@ -0,0 +1,59 @@ > +/*- > + * BSD LICENSE > + * > + * Copyright(c) 2010-2017 Intel Corporation. All rights reserved. > + * All rights reserved. > + * > + * Redistribution and use in source and binary forms, with or without > + * modification, are permitted provided that the following conditions > + * are met: > + * > + * * Redistributions of source code must retain the above copyright > + * notice, this list of conditions and the following disclaimer. > + * * Redistributions in binary form must reproduce the above copyrig= ht > + * notice, this list of conditions and the following disclaimer in > + * the documentation and/or other materials provided with the > + * distribution. > + * * Neither the name of Intel Corporation nor the names of its > + * contributors may be used to endorse or promote products derived > + * from this software without specific prior written permission. > + * > + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS > + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT > + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS F= OR > + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGH= T > + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTA= L, > + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT > + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF US= E, > + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON A= NY > + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT > + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE U= SE > + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE= . > + */ > + > +#include > +#include > +#include > + > +void *(*rte_memcpy_ptr)(void *dst, const void *src, size_t n) =3D NULL; > + > +static void __attribute__((constructor)) > +rte_memcpy_init(void) > +{ > +#ifdef CC_SUPPORT_AVX512F > + if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F)) { > + rte_memcpy_ptr =3D rte_memcpy_avx512f; > + RTE_LOG(DEBUG, EAL, "AVX512 memcpy is using!\n"); > + return; > + } > +#endif > +#ifdef CC_SUPPORT_AVX2 > + if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2)) { > + rte_memcpy_ptr =3D rte_memcpy_avx2; > + RTE_LOG(DEBUG, EAL, "AVX2 memcpy is using!\n"); > + return; > + } > +#endif > + rte_memcpy_ptr =3D rte_memcpy_sse; > + RTE_LOG(DEBUG, EAL, "Default SSE/AVX memcpy is using!\n"); > +} > diff --git a/lib/librte_eal/common/include/arch/x86/rte_memcpy.h b/lib/li= brte_eal/common/include/arch/x86/rte_memcpy.h > index 74c280c..460dcdb 100644 > --- a/lib/librte_eal/common/include/arch/x86/rte_memcpy.h > +++ b/lib/librte_eal/common/include/arch/x86/rte_memcpy.h > @@ -1,7 +1,7 @@ > /*- > * BSD LICENSE > * > - * Copyright(c) 2010-2014 Intel Corporation. All rights reserved. > + * Copyright(c) 2010-2017 Intel Corporation. All rights reserved. > * All rights reserved. > * > * Redistribution and use in source and binary forms, with or without > @@ -34,867 +34,36 @@ > #ifndef _RTE_MEMCPY_X86_64_H_ > #define _RTE_MEMCPY_X86_64_H_ >=20 > -/** > - * @file > - * > - * Functions for SSE/AVX/AVX2/AVX512 implementation of memcpy(). > - */ > - > -#include > -#include > -#include > -#include > -#include > +#include >=20 > #ifdef __cplusplus > extern "C" { > #endif >=20 > -/** > - * Copy bytes from one location to another. The locations must not overl= ap. > - * > - * @note This is implemented as a macro, so it's address should not be t= aken > - * and care is needed as parameter expressions may be evaluated multiple= times. > - * > - * @param dst > - * Pointer to the destination of the data. > - * @param src > - * Pointer to the source data. > - * @param n > - * Number of bytes to copy. > - * @return > - * Pointer to the destination data. > - */ > -static __rte_always_inline void * > -rte_memcpy(void *dst, const void *src, size_t n); > - > -#ifdef RTE_MACHINE_CPUFLAG_AVX512F > +#define RTE_X86_MEMCPY_THRESH 128 >=20 > -#define ALIGNMENT_MASK 0x3F > +extern void * > +(*rte_memcpy_ptr)(void *dst, const void *src, size_t n); >=20 > /** > - * AVX512 implementation below > + * Different implementations of memcpy. > */ > +extern void* > +rte_memcpy_avx512f(void *dst, const void *src, size_t n); >=20 > -/** > - * Copy 16 bytes from one location to another, > - * locations should not overlap. > - */ > -static inline void > -rte_mov16(uint8_t *dst, const uint8_t *src) > -{ > - __m128i xmm0; > - > - xmm0 =3D _mm_loadu_si128((const __m128i *)src); > - _mm_storeu_si128((__m128i *)dst, xmm0); > -} > - > -/** > - * Copy 32 bytes from one location to another, > - * locations should not overlap. > - */ > -static inline void > -rte_mov32(uint8_t *dst, const uint8_t *src) > -{ > - __m256i ymm0; > +extern void * > +rte_memcpy_avx2(void *dst, const void *src, size_t n); >=20 > - ymm0 =3D _mm256_loadu_si256((const __m256i *)src); > - _mm256_storeu_si256((__m256i *)dst, ymm0); > -} > - > -/** > - * Copy 64 bytes from one location to another, > - * locations should not overlap. > - */ > -static inline void > -rte_mov64(uint8_t *dst, const uint8_t *src) > -{ > - __m512i zmm0; > - > - zmm0 =3D _mm512_loadu_si512((const void *)src); > - _mm512_storeu_si512((void *)dst, zmm0); > -} > - > -/** > - * Copy 128 bytes from one location to another, > - * locations should not overlap. > - */ > -static inline void > -rte_mov128(uint8_t *dst, const uint8_t *src) > -{ > - rte_mov64(dst + 0 * 64, src + 0 * 64); > - rte_mov64(dst + 1 * 64, src + 1 * 64); > -} > - > -/** > - * Copy 256 bytes from one location to another, > - * locations should not overlap. > - */ > -static inline void > -rte_mov256(uint8_t *dst, const uint8_t *src) > -{ > - rte_mov64(dst + 0 * 64, src + 0 * 64); > - rte_mov64(dst + 1 * 64, src + 1 * 64); > - rte_mov64(dst + 2 * 64, src + 2 * 64); > - rte_mov64(dst + 3 * 64, src + 3 * 64); > -} > - > -/** > - * Copy 128-byte blocks from one location to another, > - * locations should not overlap. > - */ > -static inline void > -rte_mov128blocks(uint8_t *dst, const uint8_t *src, size_t n) > -{ > - __m512i zmm0, zmm1; > - > - while (n >=3D 128) { > - zmm0 =3D _mm512_loadu_si512((const void *)(src + 0 * 64)); > - n -=3D 128; > - zmm1 =3D _mm512_loadu_si512((const void *)(src + 1 * 64)); > - src =3D src + 128; > - _mm512_storeu_si512((void *)(dst + 0 * 64), zmm0); > - _mm512_storeu_si512((void *)(dst + 1 * 64), zmm1); > - dst =3D dst + 128; > - } > -} > - > -/** > - * Copy 512-byte blocks from one location to another, > - * locations should not overlap. > - */ > -static inline void > -rte_mov512blocks(uint8_t *dst, const uint8_t *src, size_t n) > -{ > - __m512i zmm0, zmm1, zmm2, zmm3, zmm4, zmm5, zmm6, zmm7; > - > - while (n >=3D 512) { > - zmm0 =3D _mm512_loadu_si512((const void *)(src + 0 * 64)); > - n -=3D 512; > - zmm1 =3D _mm512_loadu_si512((const void *)(src + 1 * 64)); > - zmm2 =3D _mm512_loadu_si512((const void *)(src + 2 * 64)); > - zmm3 =3D _mm512_loadu_si512((const void *)(src + 3 * 64)); > - zmm4 =3D _mm512_loadu_si512((const void *)(src + 4 * 64)); > - zmm5 =3D _mm512_loadu_si512((const void *)(src + 5 * 64)); > - zmm6 =3D _mm512_loadu_si512((const void *)(src + 6 * 64)); > - zmm7 =3D _mm512_loadu_si512((const void *)(src + 7 * 64)); > - src =3D src + 512; > - _mm512_storeu_si512((void *)(dst + 0 * 64), zmm0); > - _mm512_storeu_si512((void *)(dst + 1 * 64), zmm1); > - _mm512_storeu_si512((void *)(dst + 2 * 64), zmm2); > - _mm512_storeu_si512((void *)(dst + 3 * 64), zmm3); > - _mm512_storeu_si512((void *)(dst + 4 * 64), zmm4); > - _mm512_storeu_si512((void *)(dst + 5 * 64), zmm5); > - _mm512_storeu_si512((void *)(dst + 6 * 64), zmm6); > - _mm512_storeu_si512((void *)(dst + 7 * 64), zmm7); > - dst =3D dst + 512; > - } > -} > - > -static inline void * > -rte_memcpy_generic(void *dst, const void *src, size_t n) > -{ > - uintptr_t dstu =3D (uintptr_t)dst; > - uintptr_t srcu =3D (uintptr_t)src; > - void *ret =3D dst; > - size_t dstofss; > - size_t bits; > - > - /** > - * Copy less than 16 bytes > - */ > - if (n < 16) { > - if (n & 0x01) { > - *(uint8_t *)dstu =3D *(const uint8_t *)srcu; > - srcu =3D (uintptr_t)((const uint8_t *)srcu + 1); > - dstu =3D (uintptr_t)((uint8_t *)dstu + 1); > - } > - if (n & 0x02) { > - *(uint16_t *)dstu =3D *(const uint16_t *)srcu; > - srcu =3D (uintptr_t)((const uint16_t *)srcu + 1); > - dstu =3D (uintptr_t)((uint16_t *)dstu + 1); > - } > - if (n & 0x04) { > - *(uint32_t *)dstu =3D *(const uint32_t *)srcu; > - srcu =3D (uintptr_t)((const uint32_t *)srcu + 1); > - dstu =3D (uintptr_t)((uint32_t *)dstu + 1); > - } > - if (n & 0x08) > - *(uint64_t *)dstu =3D *(const uint64_t *)srcu; > - return ret; > - } > - > - /** > - * Fast way when copy size doesn't exceed 512 bytes > - */ > - if (n <=3D 32) { > - rte_mov16((uint8_t *)dst, (const uint8_t *)src); > - rte_mov16((uint8_t *)dst - 16 + n, > - (const uint8_t *)src - 16 + n); > - return ret; > - } > - if (n <=3D 64) { > - rte_mov32((uint8_t *)dst, (const uint8_t *)src); > - rte_mov32((uint8_t *)dst - 32 + n, > - (const uint8_t *)src - 32 + n); > - return ret; > - } > - if (n <=3D 512) { > - if (n >=3D 256) { > - n -=3D 256; > - rte_mov256((uint8_t *)dst, (const uint8_t *)src); > - src =3D (const uint8_t *)src + 256; > - dst =3D (uint8_t *)dst + 256; > - } > - if (n >=3D 128) { > - n -=3D 128; > - rte_mov128((uint8_t *)dst, (const uint8_t *)src); > - src =3D (const uint8_t *)src + 128; > - dst =3D (uint8_t *)dst + 128; > - } > -COPY_BLOCK_128_BACK63: > - if (n > 64) { > - rte_mov64((uint8_t *)dst, (const uint8_t *)src); > - rte_mov64((uint8_t *)dst - 64 + n, > - (const uint8_t *)src - 64 + n); > - return ret; > - } > - if (n > 0) > - rte_mov64((uint8_t *)dst - 64 + n, > - (const uint8_t *)src - 64 + n); > - return ret; > - } > - > - /** > - * Make store aligned when copy size exceeds 512 bytes > - */ > - dstofss =3D ((uintptr_t)dst & 0x3F); > - if (dstofss > 0) { > - dstofss =3D 64 - dstofss; > - n -=3D dstofss; > - rte_mov64((uint8_t *)dst, (const uint8_t *)src); > - src =3D (const uint8_t *)src + dstofss; > - dst =3D (uint8_t *)dst + dstofss; > - } > - > - /** > - * Copy 512-byte blocks. > - * Use copy block function for better instruction order control, > - * which is important when load is unaligned. > - */ > - rte_mov512blocks((uint8_t *)dst, (const uint8_t *)src, n); > - bits =3D n; > - n =3D n & 511; > - bits -=3D n; > - src =3D (const uint8_t *)src + bits; > - dst =3D (uint8_t *)dst + bits; > - > - /** > - * Copy 128-byte blocks. > - * Use copy block function for better instruction order control, > - * which is important when load is unaligned. > - */ > - if (n >=3D 128) { > - rte_mov128blocks((uint8_t *)dst, (const uint8_t *)src, n); > - bits =3D n; > - n =3D n & 127; > - bits -=3D n; > - src =3D (const uint8_t *)src + bits; > - dst =3D (uint8_t *)dst + bits; > - } > - > - /** > - * Copy whatever left > - */ > - goto COPY_BLOCK_128_BACK63; > -} > - > -#elif defined RTE_MACHINE_CPUFLAG_AVX2 > - > -#define ALIGNMENT_MASK 0x1F > - > -/** > - * AVX2 implementation below > - */ > - > -/** > - * Copy 16 bytes from one location to another, > - * locations should not overlap. > - */ > -static inline void > -rte_mov16(uint8_t *dst, const uint8_t *src) > -{ > - __m128i xmm0; > - > - xmm0 =3D _mm_loadu_si128((const __m128i *)src); > - _mm_storeu_si128((__m128i *)dst, xmm0); > -} > - > -/** > - * Copy 32 bytes from one location to another, > - * locations should not overlap. > - */ > -static inline void > -rte_mov32(uint8_t *dst, const uint8_t *src) > -{ > - __m256i ymm0; > - > - ymm0 =3D _mm256_loadu_si256((const __m256i *)src); > - _mm256_storeu_si256((__m256i *)dst, ymm0); > -} > - > -/** > - * Copy 64 bytes from one location to another, > - * locations should not overlap. > - */ > -static inline void > -rte_mov64(uint8_t *dst, const uint8_t *src) > -{ > - rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32); > - rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32); > -} > - > -/** > - * Copy 128 bytes from one location to another, > - * locations should not overlap. > - */ > -static inline void > -rte_mov128(uint8_t *dst, const uint8_t *src) > -{ > - rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32); > - rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32); > - rte_mov32((uint8_t *)dst + 2 * 32, (const uint8_t *)src + 2 * 32); > - rte_mov32((uint8_t *)dst + 3 * 32, (const uint8_t *)src + 3 * 32); > -} > - > -/** > - * Copy 128-byte blocks from one location to another, > - * locations should not overlap. > - */ > -static inline void > -rte_mov128blocks(uint8_t *dst, const uint8_t *src, size_t n) > -{ > - __m256i ymm0, ymm1, ymm2, ymm3; > - > - while (n >=3D 128) { > - ymm0 =3D _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 0= * 32)); > - n -=3D 128; > - ymm1 =3D _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 1= * 32)); > - ymm2 =3D _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 2= * 32)); > - ymm3 =3D _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 3= * 32)); > - src =3D (const uint8_t *)src + 128; > - _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 0 * 32), ymm0); > - _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 1 * 32), ymm1); > - _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 2 * 32), ymm2); > - _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 3 * 32), ymm3); > - dst =3D (uint8_t *)dst + 128; > - } > -} > - > -static inline void * > -rte_memcpy_generic(void *dst, const void *src, size_t n) > -{ > - uintptr_t dstu =3D (uintptr_t)dst; > - uintptr_t srcu =3D (uintptr_t)src; > - void *ret =3D dst; > - size_t dstofss; > - size_t bits; > - > - /** > - * Copy less than 16 bytes > - */ > - if (n < 16) { > - if (n & 0x01) { > - *(uint8_t *)dstu =3D *(const uint8_t *)srcu; > - srcu =3D (uintptr_t)((const uint8_t *)srcu + 1); > - dstu =3D (uintptr_t)((uint8_t *)dstu + 1); > - } > - if (n & 0x02) { > - *(uint16_t *)dstu =3D *(const uint16_t *)srcu; > - srcu =3D (uintptr_t)((const uint16_t *)srcu + 1); > - dstu =3D (uintptr_t)((uint16_t *)dstu + 1); > - } > - if (n & 0x04) { > - *(uint32_t *)dstu =3D *(const uint32_t *)srcu; > - srcu =3D (uintptr_t)((const uint32_t *)srcu + 1); > - dstu =3D (uintptr_t)((uint32_t *)dstu + 1); > - } > - if (n & 0x08) { > - *(uint64_t *)dstu =3D *(const uint64_t *)srcu; > - } > - return ret; > - } > - > - /** > - * Fast way when copy size doesn't exceed 256 bytes > - */ > - if (n <=3D 32) { > - rte_mov16((uint8_t *)dst, (const uint8_t *)src); > - rte_mov16((uint8_t *)dst - 16 + n, > - (const uint8_t *)src - 16 + n); > - return ret; > - } > - if (n <=3D 48) { > - rte_mov16((uint8_t *)dst, (const uint8_t *)src); > - rte_mov16((uint8_t *)dst + 16, (const uint8_t *)src + 16); > - rte_mov16((uint8_t *)dst - 16 + n, > - (const uint8_t *)src - 16 + n); > - return ret; > - } > - if (n <=3D 64) { > - rte_mov32((uint8_t *)dst, (const uint8_t *)src); > - rte_mov32((uint8_t *)dst - 32 + n, > - (const uint8_t *)src - 32 + n); > - return ret; > - } > - if (n <=3D 256) { > - if (n >=3D 128) { > - n -=3D 128; > - rte_mov128((uint8_t *)dst, (const uint8_t *)src); > - src =3D (const uint8_t *)src + 128; > - dst =3D (uint8_t *)dst + 128; > - } > -COPY_BLOCK_128_BACK31: > - if (n >=3D 64) { > - n -=3D 64; > - rte_mov64((uint8_t *)dst, (const uint8_t *)src); > - src =3D (const uint8_t *)src + 64; > - dst =3D (uint8_t *)dst + 64; > - } > - if (n > 32) { > - rte_mov32((uint8_t *)dst, (const uint8_t *)src); > - rte_mov32((uint8_t *)dst - 32 + n, > - (const uint8_t *)src - 32 + n); > - return ret; > - } > - if (n > 0) { > - rte_mov32((uint8_t *)dst - 32 + n, > - (const uint8_t *)src - 32 + n); > - } > - return ret; > - } > - > - /** > - * Make store aligned when copy size exceeds 256 bytes > - */ > - dstofss =3D (uintptr_t)dst & 0x1F; > - if (dstofss > 0) { > - dstofss =3D 32 - dstofss; > - n -=3D dstofss; > - rte_mov32((uint8_t *)dst, (const uint8_t *)src); > - src =3D (const uint8_t *)src + dstofss; > - dst =3D (uint8_t *)dst + dstofss; > - } > - > - /** > - * Copy 128-byte blocks > - */ > - rte_mov128blocks((uint8_t *)dst, (const uint8_t *)src, n); > - bits =3D n; > - n =3D n & 127; > - bits -=3D n; > - src =3D (const uint8_t *)src + bits; > - dst =3D (uint8_t *)dst + bits; > - > - /** > - * Copy whatever left > - */ > - goto COPY_BLOCK_128_BACK31; > -} > - > -#else /* RTE_MACHINE_CPUFLAG */ > - > -#define ALIGNMENT_MASK 0x0F > - > -/** > - * SSE & AVX implementation below > - */ > - > -/** > - * Copy 16 bytes from one location to another, > - * locations should not overlap. > - */ > -static inline void > -rte_mov16(uint8_t *dst, const uint8_t *src) > -{ > - __m128i xmm0; > - > - xmm0 =3D _mm_loadu_si128((const __m128i *)(const __m128i *)src); > - _mm_storeu_si128((__m128i *)dst, xmm0); > -} > - > -/** > - * Copy 32 bytes from one location to another, > - * locations should not overlap. > - */ > -static inline void > -rte_mov32(uint8_t *dst, const uint8_t *src) > -{ > - rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16); > - rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16); > -} > - > -/** > - * Copy 64 bytes from one location to another, > - * locations should not overlap. > - */ > -static inline void > -rte_mov64(uint8_t *dst, const uint8_t *src) > -{ > - rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16); > - rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16); > - rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16); > - rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16); > -} > - > -/** > - * Copy 128 bytes from one location to another, > - * locations should not overlap. > - */ > -static inline void > -rte_mov128(uint8_t *dst, const uint8_t *src) > -{ > - rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16); > - rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16); > - rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16); > - rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16); > - rte_mov16((uint8_t *)dst + 4 * 16, (const uint8_t *)src + 4 * 16); > - rte_mov16((uint8_t *)dst + 5 * 16, (const uint8_t *)src + 5 * 16); > - rte_mov16((uint8_t *)dst + 6 * 16, (const uint8_t *)src + 6 * 16); > - rte_mov16((uint8_t *)dst + 7 * 16, (const uint8_t *)src + 7 * 16); > -} > - > -/** > - * Copy 256 bytes from one location to another, > - * locations should not overlap. > - */ > -static inline void > -rte_mov256(uint8_t *dst, const uint8_t *src) > -{ > - rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16); > - rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16); > - rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16); > - rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16); > - rte_mov16((uint8_t *)dst + 4 * 16, (const uint8_t *)src + 4 * 16); > - rte_mov16((uint8_t *)dst + 5 * 16, (const uint8_t *)src + 5 * 16); > - rte_mov16((uint8_t *)dst + 6 * 16, (const uint8_t *)src + 6 * 16); > - rte_mov16((uint8_t *)dst + 7 * 16, (const uint8_t *)src + 7 * 16); > - rte_mov16((uint8_t *)dst + 8 * 16, (const uint8_t *)src + 8 * 16); > - rte_mov16((uint8_t *)dst + 9 * 16, (const uint8_t *)src + 9 * 16); > - rte_mov16((uint8_t *)dst + 10 * 16, (const uint8_t *)src + 10 * 16); > - rte_mov16((uint8_t *)dst + 11 * 16, (const uint8_t *)src + 11 * 16); > - rte_mov16((uint8_t *)dst + 12 * 16, (const uint8_t *)src + 12 * 16); > - rte_mov16((uint8_t *)dst + 13 * 16, (const uint8_t *)src + 13 * 16); > - rte_mov16((uint8_t *)dst + 14 * 16, (const uint8_t *)src + 14 * 16); > - rte_mov16((uint8_t *)dst + 15 * 16, (const uint8_t *)src + 15 * 16); > -} > - > -/** > - * Macro for copying unaligned block from one location to another with c= onstant load offset, > - * 47 bytes leftover maximum, > - * locations should not overlap. > - * Requirements: > - * - Store is aligned > - * - Load offset is , which must be immediate value within [1, 1= 5] > - * - For , make sure bit backwards & <16 - offset> bit for= wards are available for loading > - * - , , must be variables > - * - __m128i ~ must be pre-defined > - */ > -#define MOVEUNALIGNED_LEFT47_IMM(dst, src, len, offset) = \ > -__extension__ ({ = \ > - int tmp; = \ > - while (len >=3D 128 + 16 - offset) { = \ > - xmm0 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 0 * 16)); \ > - len -=3D 128; = \ > - xmm1 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 1 * 16)); \ > - xmm2 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 2 * 16)); \ > - xmm3 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 3 * 16)); \ > - xmm4 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 4 * 16)); \ > - xmm5 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 5 * 16)); \ > - xmm6 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 6 * 16)); \ > - xmm7 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 7 * 16)); \ > - xmm8 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 8 * 16)); \ > - src =3D (const uint8_t *)src + 128; = \ > - _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16), _mm_align= r_epi8(xmm1, xmm0, offset)); \ > - _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16), _mm_align= r_epi8(xmm2, xmm1, offset)); \ > - _mm_storeu_si128((__m128i *)((uint8_t *)dst + 2 * 16), _mm_align= r_epi8(xmm3, xmm2, offset)); \ > - _mm_storeu_si128((__m128i *)((uint8_t *)dst + 3 * 16), _mm_align= r_epi8(xmm4, xmm3, offset)); \ > - _mm_storeu_si128((__m128i *)((uint8_t *)dst + 4 * 16), _mm_align= r_epi8(xmm5, xmm4, offset)); \ > - _mm_storeu_si128((__m128i *)((uint8_t *)dst + 5 * 16), _mm_align= r_epi8(xmm6, xmm5, offset)); \ > - _mm_storeu_si128((__m128i *)((uint8_t *)dst + 6 * 16), _mm_align= r_epi8(xmm7, xmm6, offset)); \ > - _mm_storeu_si128((__m128i *)((uint8_t *)dst + 7 * 16), _mm_align= r_epi8(xmm8, xmm7, offset)); \ > - dst =3D (uint8_t *)dst + 128; = \ > - } = \ > - tmp =3D len; = \ > - len =3D ((len - 16 + offset) & 127) + 16 - offset; = \ > - tmp -=3D len; = \ > - src =3D (const uint8_t *)src + tmp; = \ > - dst =3D (uint8_t *)dst + tmp; = \ > - if (len >=3D 32 + 16 - offset) { = \ > - while (len >=3D 32 + 16 - offset) { = \ > - xmm0 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)= src - offset + 0 * 16)); \ > - len -=3D 32; = \ > - xmm1 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)= src - offset + 1 * 16)); \ > - xmm2 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)= src - offset + 2 * 16)); \ > - src =3D (const uint8_t *)src + 32; = \ > - _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16), _mm_a= lignr_epi8(xmm1, xmm0, offset)); \ > - _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16), _mm_a= lignr_epi8(xmm2, xmm1, offset)); \ > - dst =3D (uint8_t *)dst + 32; = \ > - } = \ > - tmp =3D len; = \ > - len =3D ((len - 16 + offset) & 31) + 16 - offset; = \ > - tmp -=3D len; = \ > - src =3D (const uint8_t *)src + tmp; = \ > - dst =3D (uint8_t *)dst + tmp; = \ > - } = \ > -}) > - > -/** > - * Macro for copying unaligned block from one location to another, > - * 47 bytes leftover maximum, > - * locations should not overlap. > - * Use switch here because the aligning instruction requires immediate v= alue for shift count. > - * Requirements: > - * - Store is aligned > - * - Load offset is , which must be within [1, 15] > - * - For , make sure bit backwards & <16 - offset> bit for= wards are available for loading > - * - , , must be variables > - * - __m128i ~ used in MOVEUNALIGNED_LEFT47_IMM must be pr= e-defined > - */ > -#define MOVEUNALIGNED_LEFT47(dst, src, len, offset) \ > -__extension__ ({ \ > - switch (offset) { \ > - case 0x01: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x01); break; \ > - case 0x02: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x02); break; \ > - case 0x03: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x03); break; \ > - case 0x04: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x04); break; \ > - case 0x05: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x05); break; \ > - case 0x06: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x06); break; \ > - case 0x07: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x07); break; \ > - case 0x08: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x08); break; \ > - case 0x09: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x09); break; \ > - case 0x0A: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0A); break; \ > - case 0x0B: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0B); break; \ > - case 0x0C: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0C); break; \ > - case 0x0D: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0D); break; \ > - case 0x0E: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0E); break; \ > - case 0x0F: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0F); break; \ > - default:; \ > - } \ > -}) > - > -static inline void * > -rte_memcpy_generic(void *dst, const void *src, size_t n) > -{ > - __m128i xmm0, xmm1, xmm2, xmm3, xmm4, xmm5, xmm6, xmm7, xmm8; > - uintptr_t dstu =3D (uintptr_t)dst; > - uintptr_t srcu =3D (uintptr_t)src; > - void *ret =3D dst; > - size_t dstofss; > - size_t srcofs; > - > - /** > - * Copy less than 16 bytes > - */ > - if (n < 16) { > - if (n & 0x01) { > - *(uint8_t *)dstu =3D *(const uint8_t *)srcu; > - srcu =3D (uintptr_t)((const uint8_t *)srcu + 1); > - dstu =3D (uintptr_t)((uint8_t *)dstu + 1); > - } > - if (n & 0x02) { > - *(uint16_t *)dstu =3D *(const uint16_t *)srcu; > - srcu =3D (uintptr_t)((const uint16_t *)srcu + 1); > - dstu =3D (uintptr_t)((uint16_t *)dstu + 1); > - } > - if (n & 0x04) { > - *(uint32_t *)dstu =3D *(const uint32_t *)srcu; > - srcu =3D (uintptr_t)((const uint32_t *)srcu + 1); > - dstu =3D (uintptr_t)((uint32_t *)dstu + 1); > - } > - if (n & 0x08) { > - *(uint64_t *)dstu =3D *(const uint64_t *)srcu; > - } > - return ret; > - } > - > - /** > - * Fast way when copy size doesn't exceed 512 bytes > - */ > - if (n <=3D 32) { > - rte_mov16((uint8_t *)dst, (const uint8_t *)src); > - rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n); > - return ret; > - } > - if (n <=3D 48) { > - rte_mov32((uint8_t *)dst, (const uint8_t *)src); > - rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n); > - return ret; > - } > - if (n <=3D 64) { > - rte_mov32((uint8_t *)dst, (const uint8_t *)src); > - rte_mov16((uint8_t *)dst + 32, (const uint8_t *)src + 32); > - rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n); > - return ret; > - } > - if (n <=3D 128) { > - goto COPY_BLOCK_128_BACK15; > - } > - if (n <=3D 512) { > - if (n >=3D 256) { > - n -=3D 256; > - rte_mov128((uint8_t *)dst, (const uint8_t *)src); > - rte_mov128((uint8_t *)dst + 128, (const uint8_t *)src + 128); > - src =3D (const uint8_t *)src + 256; > - dst =3D (uint8_t *)dst + 256; > - } > -COPY_BLOCK_255_BACK15: > - if (n >=3D 128) { > - n -=3D 128; > - rte_mov128((uint8_t *)dst, (const uint8_t *)src); > - src =3D (const uint8_t *)src + 128; > - dst =3D (uint8_t *)dst + 128; > - } > -COPY_BLOCK_128_BACK15: > - if (n >=3D 64) { > - n -=3D 64; > - rte_mov64((uint8_t *)dst, (const uint8_t *)src); > - src =3D (const uint8_t *)src + 64; > - dst =3D (uint8_t *)dst + 64; > - } > -COPY_BLOCK_64_BACK15: > - if (n >=3D 32) { > - n -=3D 32; > - rte_mov32((uint8_t *)dst, (const uint8_t *)src); > - src =3D (const uint8_t *)src + 32; > - dst =3D (uint8_t *)dst + 32; > - } > - if (n > 16) { > - rte_mov16((uint8_t *)dst, (const uint8_t *)src); > - rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n); > - return ret; > - } > - if (n > 0) { > - rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n); > - } > - return ret; > - } > - > - /** > - * Make store aligned when copy size exceeds 512 bytes, > - * and make sure the first 15 bytes are copied, because > - * unaligned copy functions require up to 15 bytes > - * backwards access. > - */ > - dstofss =3D (uintptr_t)dst & 0x0F; > - if (dstofss > 0) { > - dstofss =3D 16 - dstofss + 16; > - n -=3D dstofss; > - rte_mov32((uint8_t *)dst, (const uint8_t *)src); > - src =3D (const uint8_t *)src + dstofss; > - dst =3D (uint8_t *)dst + dstofss; > - } > - srcofs =3D ((uintptr_t)src & 0x0F); > - > - /** > - * For aligned copy > - */ > - if (srcofs =3D=3D 0) { > - /** > - * Copy 256-byte blocks > - */ > - for (; n >=3D 256; n -=3D 256) { > - rte_mov256((uint8_t *)dst, (const uint8_t *)src); > - dst =3D (uint8_t *)dst + 256; > - src =3D (const uint8_t *)src + 256; > - } > - > - /** > - * Copy whatever left > - */ > - goto COPY_BLOCK_255_BACK15; > - } > - > - /** > - * For copy with unaligned load > - */ > - MOVEUNALIGNED_LEFT47(dst, src, n, srcofs); > - > - /** > - * Copy whatever left > - */ > - goto COPY_BLOCK_64_BACK15; > -} > - > -#endif /* RTE_MACHINE_CPUFLAG */ > - > -static inline void * > -rte_memcpy_aligned(void *dst, const void *src, size_t n) > -{ > - void *ret =3D dst; > - > - /* Copy size <=3D 16 bytes */ > - if (n < 16) { > - if (n & 0x01) { > - *(uint8_t *)dst =3D *(const uint8_t *)src; > - src =3D (const uint8_t *)src + 1; > - dst =3D (uint8_t *)dst + 1; > - } > - if (n & 0x02) { > - *(uint16_t *)dst =3D *(const uint16_t *)src; > - src =3D (const uint16_t *)src + 1; > - dst =3D (uint16_t *)dst + 1; > - } > - if (n & 0x04) { > - *(uint32_t *)dst =3D *(const uint32_t *)src; > - src =3D (const uint32_t *)src + 1; > - dst =3D (uint32_t *)dst + 1; > - } > - if (n & 0x08) > - *(uint64_t *)dst =3D *(const uint64_t *)src; > - > - return ret; > - } > - > - /* Copy 16 <=3D size <=3D 32 bytes */ > - if (n <=3D 32) { > - rte_mov16((uint8_t *)dst, (const uint8_t *)src); > - rte_mov16((uint8_t *)dst - 16 + n, > - (const uint8_t *)src - 16 + n); > - > - return ret; > - } > - > - /* Copy 32 < size <=3D 64 bytes */ > - if (n <=3D 64) { > - rte_mov32((uint8_t *)dst, (const uint8_t *)src); > - rte_mov32((uint8_t *)dst - 32 + n, > - (const uint8_t *)src - 32 + n); > - > - return ret; > - } > - > - /* Copy 64 bytes blocks */ > - for (; n >=3D 64; n -=3D 64) { > - rte_mov64((uint8_t *)dst, (const uint8_t *)src); > - dst =3D (uint8_t *)dst + 64; > - src =3D (const uint8_t *)src + 64; > - } > - > - /* Copy whatever left */ > - rte_mov64((uint8_t *)dst - 64 + n, > - (const uint8_t *)src - 64 + n); > - > - return ret; > -} > +extern void * > +rte_memcpy_sse(void *dst, const void *src, size_t n); >=20 > static inline void * > rte_memcpy(void *dst, const void *src, size_t n) > { > - if (!(((uintptr_t)dst | (uintptr_t)src) & ALIGNMENT_MASK)) > - return rte_memcpy_aligned(dst, src, n); > + if (n <=3D RTE_X86_MEMCPY_THRESH) > + return rte_memcpy_internal(dst, src, n); > else > - return rte_memcpy_generic(dst, src, n); > + return (*rte_memcpy_ptr)(dst, src, n); > } >=20 > #ifdef __cplusplus > diff --git a/lib/librte_eal/common/include/arch/x86/rte_memcpy_avx2.c b/l= ib/librte_eal/common/include/arch/x86/rte_memcpy_avx2.c > new file mode 100644 > index 0000000..c83351a > --- /dev/null > +++ b/lib/librte_eal/common/include/arch/x86/rte_memcpy_avx2.c > @@ -0,0 +1,291 @@ > +/*- > + * BSD LICENSE > + * > + * Copyright(c) 2010-2017 Intel Corporation. All rights reserved. > + * All rights reserved. > + * > + * Redistribution and use in source and binary forms, with or without > + * modification, are permitted provided that the following conditions > + * are met: > + * > + * * Redistributions of source code must retain the above copyright > + * notice, this list of conditions and the following disclaimer. > + * * Redistributions in binary form must reproduce the above copyrig= ht > + * notice, this list of conditions and the following disclaimer in > + * the documentation and/or other materials provided with the > + * distribution. > + * * Neither the name of Intel Corporation nor the names of its > + * contributors may be used to endorse or promote products derived > + * from this software without specific prior written permission. > + * > + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS > + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT > + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS F= OR > + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGH= T > + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTA= L, > + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT > + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF US= E, > + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON A= NY > + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT > + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE U= SE > + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE= . > + */ > + > +#include > + > +#ifndef CC_SUPPORT_AVX2 > +#error CC_SUPPORT_AVX2 not defined > +#endif > + > +void * > +rte_memcpy_avx2(void *dst, const void *src, size_t n) > +{ > + if (!(((uintptr_t)dst | (uintptr_t)src) & 0x1F)) { > + void *ret =3D dst; > + > + /* Copy size <=3D 16 bytes */ > + if (n < 16) { > + if (n & 0x01) { > + *(uint8_t *)dst =3D *(const uint8_t *)src; > + src =3D (const uint8_t *)src + 1; > + dst =3D (uint8_t *)dst + 1; > + } > + if (n & 0x02) { > + *(uint16_t *)dst =3D *(const uint16_t *)src; > + src =3D (const uint16_t *)src + 1; > + dst =3D (uint16_t *)dst + 1; > + } > + if (n & 0x04) { > + *(uint32_t *)dst =3D *(const uint32_t *)src; > + src =3D (const uint32_t *)src + 1; > + dst =3D (uint32_t *)dst + 1; > + } > + if (n & 0x08) > + *(uint64_t *)dst =3D *(const uint64_t *)src; > + > + return ret; > + } > + > + /* Copy 16 <=3D size <=3D 32 bytes */ > + if (n <=3D 32) { > + __m128i xmm0, xmm1; > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src - 16 + n)); > + _mm_storeu_si128((__m128i *)dst, xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst - 16 + n), xmm1); > + > + return ret; > + } > + > + /* Copy 32 < size <=3D 64 bytes */ > + if (n <=3D 64) { > + __m256i ymm0, ymm1; > + ymm0 =3D _mm256_loadu_si256((const __m256i *)src); > + ymm1 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src - 32 + n)); > + _mm256_storeu_si256((__m256i *)dst, ymm0); > + _mm256_storeu_si256((__m256i *) > + ((uint8_t *)dst - 32 + n), ymm1); > + > + return ret; > + } > + > + /* Copy 64 bytes blocks */ > + for (; n >=3D 64; n -=3D 64) { > + __m256i ymm0, ymm1; > + ymm0 =3D _mm256_loadu_si256((const __m256i *)src); > + ymm1 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src + 32)); > + _mm256_storeu_si256((__m256i *)dst, ymm0); > + _mm256_storeu_si256((__m256i *) > + ((uint8_t *)dst + 32), ymm1); > + dst =3D (uint8_t *)dst + 64; > + src =3D (const uint8_t *)src + 64; > + } > + > + /* Copy whatever left */ > + __m256i ymm0, ymm1; > + ymm0 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src - 64 + n)); > + ymm1 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src - 32 + n)); > + _mm256_storeu_si256((__m256i *)((uint8_t *)dst - 64 + n), ymm0); > + _mm256_storeu_si256((__m256i *)((uint8_t *)dst - 32 + n), ymm1); > + > + return ret; > + } else { > + uintptr_t dstu =3D (uintptr_t)dst; > + uintptr_t srcu =3D (uintptr_t)src; > + void *ret =3D dst; > + size_t dstofss; > + size_t bits; > + > + /** > + * Copy less than 16 bytes > + */ > + if (n < 16) { > + if (n & 0x01) { > + *(uint8_t *)dstu =3D *(const uint8_t *)srcu; > + srcu =3D (uintptr_t)((const uint8_t *)srcu + 1); > + dstu =3D (uintptr_t)((uint8_t *)dstu + 1); > + } > + if (n & 0x02) { > + *(uint16_t *)dstu =3D *(const uint16_t *)srcu; > + srcu =3D (uintptr_t)((const uint16_t *)srcu + 1); > + dstu =3D (uintptr_t)((uint16_t *)dstu + 1); > + } > + if (n & 0x04) { > + *(uint32_t *)dstu =3D *(const uint32_t *)srcu; > + srcu =3D (uintptr_t)((const uint32_t *)srcu + 1); > + dstu =3D (uintptr_t)((uint32_t *)dstu + 1); > + } > + if (n & 0x08) > + *(uint64_t *)dstu =3D *(const uint64_t *)srcu; > + return ret; > + } > + > + /** > + * Fast way when copy size doesn't exceed 256 bytes > + */ > + if (n <=3D 32) { > + __m128i xmm0, xmm1; > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src - 16 + n)); > + _mm_storeu_si128((__m128i *)dst, xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst - 16 + n), xmm1); > + return ret; > + } > + if (n <=3D 48) { > + __m128i xmm0, xmm1, xmm2; > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 16)); > + xmm2 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src - 16 + n)); > + _mm_storeu_si128((__m128i *)dst, xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 16), xmm1); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst - 16 + n), xmm2); > + return ret; > + } > + if (n <=3D 64) { > + __m256i ymm0, ymm1; > + ymm0 =3D _mm256_loadu_si256((const __m256i *)src); > + ymm1 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src - 32 + n)); > + _mm256_storeu_si256((__m256i *)dst, ymm0); > + _mm256_storeu_si256((__m256i *) > + ((uint8_t *)dst - 32 + n), ymm1); > + return ret; > + } > + if (n <=3D 256) { > + if (n >=3D 128) { > + n -=3D 128; > + __m256i ymm0, ymm1, ymm2, ymm3; > + ymm0 =3D _mm256_loadu_si256((const __m256i *)src); > + ymm1 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src + 32)); > + ymm2 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src + 2*32)); > + ymm3 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src + 3*32)); > + _mm256_storeu_si256((__m256i *)dst, ymm0); > + _mm256_storeu_si256((__m256i *) > + ((uint8_t *)dst + 32), ymm1); > + _mm256_storeu_si256((__m256i *) > + ((uint8_t *)dst + 2*32), ymm2); > + _mm256_storeu_si256((__m256i *) > + ((uint8_t *)dst + 3*32), ymm3); > + src =3D (const uint8_t *)src + 128; > + dst =3D (uint8_t *)dst + 128; > + } > +COPY_BLOCK_128_BACK31: > + if (n >=3D 64) { > + n -=3D 64; > + __m256i ymm0, ymm1; > + ymm0 =3D _mm256_loadu_si256((const __m256i *)src); > + ymm1 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src + 32)); > + _mm256_storeu_si256((__m256i *)dst, ymm0); > + _mm256_storeu_si256((__m256i *) > + ((uint8_t *)dst + 32), ymm1); > + src =3D (const uint8_t *)src + 64; > + dst =3D (uint8_t *)dst + 64; > + } > + if (n > 32) { > + __m256i ymm0, ymm1; > + ymm0 =3D _mm256_loadu_si256((const __m256i *)src); > + ymm1 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src - 32 + n)); > + _mm256_storeu_si256((__m256i *)dst, ymm0); > + _mm256_storeu_si256((__m256i *) > + ((uint8_t *)dst - 32 + n), ymm1); > + return ret; > + } > + if (n > 0) { > + __m256i ymm0; > + ymm0 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src - 32 + n)); > + _mm256_storeu_si256((__m256i *) > + ((uint8_t *)dst - 32 + n), ymm0); > + } > + return ret; > + } > + > + /** > + * Make store aligned when copy size exceeds 256 bytes > + */ > + dstofss =3D (uintptr_t)dst & 0x1F; > + if (dstofss > 0) { > + dstofss =3D 32 - dstofss; > + n -=3D dstofss; > + __m256i ymm0; > + ymm0 =3D _mm256_loadu_si256((const __m256i *)src); > + _mm256_storeu_si256((__m256i *)dst, ymm0); > + src =3D (const uint8_t *)src + dstofss; > + dst =3D (uint8_t *)dst + dstofss; > + } > + > + /** > + * Copy 128-byte blocks > + */ > + __m256i ymm0, ymm1, ymm2, ymm3; > + > + while (n >=3D 128) { > + ymm0 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src + 0 * 32)); > + n -=3D 128; > + ymm1 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src + 1 * 32)); > + ymm2 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src + 2 * 32)); > + ymm3 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src + 3 * 32)); > + src =3D (const uint8_t *)src + 128; > + _mm256_storeu_si256((__m256i *) > + ((uint8_t *)dst + 0 * 32), ymm0); > + _mm256_storeu_si256((__m256i *) > + ((uint8_t *)dst + 1 * 32), ymm1); > + _mm256_storeu_si256((__m256i *) > + ((uint8_t *)dst + 2 * 32), ymm2); > + _mm256_storeu_si256((__m256i *) > + ((uint8_t *)dst + 3 * 32), ymm3); > + dst =3D (uint8_t *)dst + 128; > + } > + bits =3D n; > + n =3D n & 127; > + bits -=3D n; > + src =3D (const uint8_t *)src + bits; > + dst =3D (uint8_t *)dst + bits; > + > + /** > + * Copy whatever left > + */ > + goto COPY_BLOCK_128_BACK31; > + } > +} > diff --git a/lib/librte_eal/common/include/arch/x86/rte_memcpy_avx512f.c > b/lib/librte_eal/common/include/arch/x86/rte_memcpy_avx512f.c > new file mode 100644 > index 0000000..c8a9d20 > --- /dev/null > +++ b/lib/librte_eal/common/include/arch/x86/rte_memcpy_avx512f.c > @@ -0,0 +1,316 @@ > +/*- > + * BSD LICENSE > + * > + * Copyright(c) 2010-2017 Intel Corporation. All rights reserved. > + * All rights reserved. > + * > + * Redistribution and use in source and binary forms, with or without > + * modification, are permitted provided that the following conditions > + * are met: > + * > + * * Redistributions of source code must retain the above copyright > + * notice, this list of conditions and the following disclaimer. > + * * Redistributions in binary form must reproduce the above copyrig= ht > + * notice, this list of conditions and the following disclaimer in > + * the documentation and/or other materials provided with the > + * distribution. > + * * Neither the name of Intel Corporation nor the names of its > + * contributors may be used to endorse or promote products derived > + * from this software without specific prior written permission. > + * > + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS > + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT > + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS F= OR > + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGH= T > + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTA= L, > + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT > + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF US= E, > + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON A= NY > + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT > + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE U= SE > + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE= . > + */ > + > +#include > + > +#ifndef CC_SUPPORT_AVX512F > +#error CC_SUPPORT_AVX512F not defined > +#endif > + > +void * > +rte_memcpy_avx512f(void *dst, const void *src, size_t n) > +{ > + if (!(((uintptr_t)dst | (uintptr_t)src) & 0x3F)) { > + void *ret =3D dst; > + > + /* Copy size <=3D 16 bytes */ > + if (n < 16) { > + if (n & 0x01) { > + *(uint8_t *)dst =3D *(const uint8_t *)src; > + src =3D (const uint8_t *)src + 1; > + dst =3D (uint8_t *)dst + 1; > + } > + if (n & 0x02) { > + *(uint16_t *)dst =3D *(const uint16_t *)src; > + src =3D (const uint16_t *)src + 1; > + dst =3D (uint16_t *)dst + 1; > + } > + if (n & 0x04) { > + *(uint32_t *)dst =3D *(const uint32_t *)src; > + src =3D (const uint32_t *)src + 1; > + dst =3D (uint32_t *)dst + 1; > + } > + if (n & 0x08) > + *(uint64_t *)dst =3D *(const uint64_t *)src; > + > + return ret; > + } > + > + /* Copy 16 <=3D size <=3D 32 bytes */ > + if (n <=3D 32) { > + __m128i xmm0, xmm1; > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src - 16 + n)); > + _mm_storeu_si128((__m128i *)dst, xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst - 16 + n), xmm1); > + > + return ret; > + } > + > + /* Copy 32 < size <=3D 64 bytes */ > + if (n <=3D 64) { > + __m256i ymm0, ymm1; > + ymm0 =3D _mm256_loadu_si256((const __m256i *)src); > + ymm1 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src - 32 + n)); > + _mm256_storeu_si256((__m256i *)dst, ymm0); > + _mm256_storeu_si256((__m256i *) > + ((uint8_t *)dst - 32 + n), ymm1); > + > + return ret; > + } > + > + /* Copy 64 bytes blocks */ > + for (; n >=3D 64; n -=3D 64) { > + __m512i zmm0; > + zmm0 =3D _mm512_loadu_si512((const void *)src); > + _mm512_storeu_si512((void *)dst, zmm0); > + dst =3D (uint8_t *)dst + 64; > + src =3D (const uint8_t *)src + 64; > + } > + > + /* Copy whatever left */ > + __m512i zmm0; > + zmm0 =3D _mm512_loadu_si512((const void *) > + ((const uint8_t *)src - 64 + n)); > + _mm512_storeu_si512((void *)((uint8_t *)dst - 64 + n), zmm0); > + > + return ret; > + } else { > + uintptr_t dstu =3D (uintptr_t)dst; > + uintptr_t srcu =3D (uintptr_t)src; > + void *ret =3D dst; > + size_t dstofss; > + size_t bits; > + > + /** > + * Copy less than 16 bytes > + */ > + if (n < 16) { > + if (n & 0x01) { > + *(uint8_t *)dstu =3D *(const uint8_t *)srcu; > + srcu =3D (uintptr_t)((const uint8_t *)srcu + 1); > + dstu =3D (uintptr_t)((uint8_t *)dstu + 1); > + } > + if (n & 0x02) { > + *(uint16_t *)dstu =3D *(const uint16_t *)srcu; > + srcu =3D (uintptr_t)((const uint16_t *)srcu + 1); > + dstu =3D (uintptr_t)((uint16_t *)dstu + 1); > + } > + if (n & 0x04) { > + *(uint32_t *)dstu =3D *(const uint32_t *)srcu; > + srcu =3D (uintptr_t)((const uint32_t *)srcu + 1); > + dstu =3D (uintptr_t)((uint32_t *)dstu + 1); > + } > + if (n & 0x08) > + *(uint64_t *)dstu =3D *(const uint64_t *)srcu; > + return ret; > + } > + > + /** > + * Fast way when copy size doesn't exceed 512 bytes > + */ > + if (n <=3D 32) { > + __m128i xmm0, xmm1; > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src - 16 + n)); > + _mm_storeu_si128((__m128i *)dst, xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst - 16 + n), xmm1); > + return ret; > + } > + if (n <=3D 64) { > + __m256i ymm0, ymm1; > + ymm0 =3D _mm256_loadu_si256((const __m256i *)src); > + ymm1 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src - 32 + n)); > + _mm256_storeu_si256((__m256i *)dst, ymm0); > + _mm256_storeu_si256((__m256i *) > + ((uint8_t *)dst - 32 + n), ymm1); > + return ret; > + } > + if (n <=3D 512) { > + if (n >=3D 256) { > + n -=3D 256; > + __m512i zmm0, zmm1, zmm2, zmm3; > + zmm0 =3D _mm512_loadu_si512((const void *)src); > + zmm1 =3D _mm512_loadu_si512((const void *) > + ((const uint8_t *)src + 64)); > + zmm2 =3D _mm512_loadu_si512((const void *) > + ((const uint8_t *)src + 2*64)); > + zmm3 =3D _mm512_loadu_si512((const void *) > + ((const uint8_t *)src + 3*64)); > + _mm512_storeu_si512((void *)dst, zmm0); > + _mm512_storeu_si512((void *) > + ((uint8_t *)dst + 64), zmm1); > + _mm512_storeu_si512((void *) > + ((uint8_t *)dst + 2*64), zmm2); > + _mm512_storeu_si512((void *) > + ((uint8_t *)dst + 3*64), zmm3); > + src =3D (const uint8_t *)src + 256; > + dst =3D (uint8_t *)dst + 256; > + } > + if (n >=3D 128) { > + n -=3D 128; > + __m512i zmm0, zmm1; > + zmm0 =3D _mm512_loadu_si512((const void *)src); > + zmm1 =3D _mm512_loadu_si512((const void *) > + ((const uint8_t *)src + 64)); > + _mm512_storeu_si512((void *)dst, zmm0); > + _mm512_storeu_si512((void *) > + ((uint8_t *)dst + 64), zmm1); > + src =3D (const uint8_t *)src + 128; > + dst =3D (uint8_t *)dst + 128; > + } > +COPY_BLOCK_128_BACK63: > + if (n > 64) { > + __m512i zmm0, zmm1; > + zmm0 =3D _mm512_loadu_si512((const void *)src); > + zmm1 =3D _mm512_loadu_si512((const void *) > + ((const uint8_t *)src - 64 + n)); > + _mm512_storeu_si512((void *)dst, zmm0); > + _mm512_storeu_si512((void *) > + ((uint8_t *)dst - 64 + n), zmm1); > + return ret; > + } > + if (n > 0) { > + __m512i zmm0; > + zmm0 =3D _mm512_loadu_si512((const void *) > + ((const uint8_t *)src - 64 + n)); > + _mm512_storeu_si512((void *) > + ((uint8_t *)dst - 64 + n), zmm0); > + } > + return ret; > + } > + > + /** > + * Make store aligned when copy size exceeds 512 bytes > + */ > + dstofss =3D ((uintptr_t)dst & 0x3F); > + if (dstofss > 0) { > + dstofss =3D 64 - dstofss; > + n -=3D dstofss; > + __m512i zmm0; > + zmm0 =3D _mm512_loadu_si512((const void *)src); > + _mm512_storeu_si512((void *)dst, zmm0); > + src =3D (const uint8_t *)src + dstofss; > + dst =3D (uint8_t *)dst + dstofss; > + } > + > + /** > + * Copy 512-byte blocks. > + * Use copy block function for better instruction order control, > + * which is important when load is unaligned. > + */ > + __m512i zmm0, zmm1, zmm2, zmm3, zmm4, zmm5, zmm6, zmm7; > + > + while (n >=3D 512) { > + zmm0 =3D _mm512_loadu_si512((const void *) > + ((const uint8_t *)src + 0 * 64)); > + n -=3D 512; > + zmm1 =3D _mm512_loadu_si512((const void *) > + ((const uint8_t *)src + 1 * 64)); > + zmm2 =3D _mm512_loadu_si512((const void *) > + ((const uint8_t *)src + 2 * 64)); > + zmm3 =3D _mm512_loadu_si512((const void *) > + ((const uint8_t *)src + 3 * 64)); > + zmm4 =3D _mm512_loadu_si512((const void *) > + ((const uint8_t *)src + 4 * 64)); > + zmm5 =3D _mm512_loadu_si512((const void *) > + ((const uint8_t *)src + 5 * 64)); > + zmm6 =3D _mm512_loadu_si512((const void *) > + ((const uint8_t *)src + 6 * 64)); > + zmm7 =3D _mm512_loadu_si512((const void *) > + ((const uint8_t *)src + 7 * 64)); > + src =3D (const uint8_t *)src + 512; > + _mm512_storeu_si512((void *) > + ((uint8_t *)dst + 0 * 64), zmm0); > + _mm512_storeu_si512((void *) > + ((uint8_t *)dst + 1 * 64), zmm1); > + _mm512_storeu_si512((void *) > + ((uint8_t *)dst + 2 * 64), zmm2); > + _mm512_storeu_si512((void *) > + ((uint8_t *)dst + 3 * 64), zmm3); > + _mm512_storeu_si512((void *) > + ((uint8_t *)dst + 4 * 64), zmm4); > + _mm512_storeu_si512((void *) > + ((uint8_t *)dst + 5 * 64), zmm5); > + _mm512_storeu_si512((void *) > + ((uint8_t *)dst + 6 * 64), zmm6); > + _mm512_storeu_si512((void *) > + ((uint8_t *)dst + 7 * 64), zmm7); > + dst =3D (uint8_t *)dst + 512; > + } > + bits =3D n; > + n =3D n & 511; > + bits -=3D n; > + src =3D (const uint8_t *)src + bits; > + dst =3D (uint8_t *)dst + bits; > + > + /** > + * Copy 128-byte blocks. > + * Use copy block function for better instruction order control, > + * which is important when load is unaligned. > + */ > + if (n >=3D 128) { > + __m512i zmm0, zmm1; > + > + while (n >=3D 128) { > + zmm0 =3D _mm512_loadu_si512((const void *) > + ((const uint8_t *)src + 0 * 64)); > + n -=3D 128; > + zmm1 =3D _mm512_loadu_si512((const void *) > + ((const uint8_t *)src + 1 * 64)); > + src =3D (const uint8_t *)src + 128; > + _mm512_storeu_si512((void *) > + ((uint8_t *)dst + 0 * 64), zmm0); > + _mm512_storeu_si512((void *) > + ((uint8_t *)dst + 1 * 64), zmm1); > + dst =3D (uint8_t *)dst + 128; > + } > + bits =3D n; > + n =3D n & 127; > + bits -=3D n; > + src =3D (const uint8_t *)src + bits; > + dst =3D (uint8_t *)dst + bits; > + } > + > + /** > + * Copy whatever left > + */ > + goto COPY_BLOCK_128_BACK63; > + } > +} > diff --git a/lib/librte_eal/common/include/arch/x86/rte_memcpy_internal.h > b/lib/librte_eal/common/include/arch/x86/rte_memcpy_internal.h > new file mode 100644 > index 0000000..d17fb5b > --- /dev/null > +++ b/lib/librte_eal/common/include/arch/x86/rte_memcpy_internal.h > @@ -0,0 +1,909 @@ > +/*- > + * BSD LICENSE > + * > + * Copyright(c) 2010-2014 Intel Corporation. All rights reserved. > + * All rights reserved. > + * > + * Redistribution and use in source and binary forms, with or without > + * modification, are permitted provided that the following conditions > + * are met: > + * > + * * Redistributions of source code must retain the above copyright > + * notice, this list of conditions and the following disclaimer. > + * * Redistributions in binary form must reproduce the above copyrig= ht > + * notice, this list of conditions and the following disclaimer in > + * the documentation and/or other materials provided with the > + * distribution. > + * * Neither the name of Intel Corporation nor the names of its > + * contributors may be used to endorse or promote products derived > + * from this software without specific prior written permission. > + * > + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS > + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT > + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS F= OR > + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGH= T > + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTA= L, > + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT > + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF US= E, > + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON A= NY > + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT > + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE U= SE > + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE= . > + */ > + > +#ifndef _RTE_MEMCPY_INTERNAL_X86_64_H_ > +#define _RTE_MEMCPY_INTERNAL_X86_64_H_ > + > +/** > + * @file > + * > + * Functions for SSE/AVX/AVX2/AVX512 implementation of memcpy(). > + */ > + > +#include > +#include > +#include > +#include > +#include > + > +#ifdef __cplusplus > +extern "C" { > +#endif > + > +/** > + * Copy bytes from one location to another. The locations must not overl= ap. > + * > + * @note This is implemented as a macro, so it's address should not be t= aken > + * and care is needed as parameter expressions may be evaluated multiple= times. > + * > + * @param dst > + * Pointer to the destination of the data. > + * @param src > + * Pointer to the source data. > + * @param n > + * Number of bytes to copy. > + * @return > + * Pointer to the destination data. > + */ > + > +#ifdef RTE_MACHINE_CPUFLAG_AVX512F > + > +#define ALIGNMENT_MASK 0x3F > + > +/** > + * AVX512 implementation below > + */ > + > +/** > + * Copy 16 bytes from one location to another, > + * locations should not overlap. > + */ > +static inline void > +rte_mov16(uint8_t *dst, const uint8_t *src) > +{ > + __m128i xmm0; > + > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > + _mm_storeu_si128((__m128i *)dst, xmm0); > +} > + > +/** > + * Copy 32 bytes from one location to another, > + * locations should not overlap. > + */ > +static inline void > +rte_mov32(uint8_t *dst, const uint8_t *src) > +{ > + __m256i ymm0; > + > + ymm0 =3D _mm256_loadu_si256((const __m256i *)src); > + _mm256_storeu_si256((__m256i *)dst, ymm0); > +} > + > +/** > + * Copy 64 bytes from one location to another, > + * locations should not overlap. > + */ > +static inline void > +rte_mov64(uint8_t *dst, const uint8_t *src) > +{ > + __m512i zmm0; > + > + zmm0 =3D _mm512_loadu_si512((const void *)src); > + _mm512_storeu_si512((void *)dst, zmm0); > +} > + > +/** > + * Copy 128 bytes from one location to another, > + * locations should not overlap. > + */ > +static inline void > +rte_mov128(uint8_t *dst, const uint8_t *src) > +{ > + rte_mov64(dst + 0 * 64, src + 0 * 64); > + rte_mov64(dst + 1 * 64, src + 1 * 64); > +} > + > +/** > + * Copy 256 bytes from one location to another, > + * locations should not overlap. > + */ > +static inline void > +rte_mov256(uint8_t *dst, const uint8_t *src) > +{ > + rte_mov64(dst + 0 * 64, src + 0 * 64); > + rte_mov64(dst + 1 * 64, src + 1 * 64); > + rte_mov64(dst + 2 * 64, src + 2 * 64); > + rte_mov64(dst + 3 * 64, src + 3 * 64); > +} > + > +/** > + * Copy 128-byte blocks from one location to another, > + * locations should not overlap. > + */ > +static inline void > +rte_mov128blocks(uint8_t *dst, const uint8_t *src, size_t n) > +{ > + __m512i zmm0, zmm1; > + > + while (n >=3D 128) { > + zmm0 =3D _mm512_loadu_si512((const void *)(src + 0 * 64)); > + n -=3D 128; > + zmm1 =3D _mm512_loadu_si512((const void *)(src + 1 * 64)); > + src =3D src + 128; > + _mm512_storeu_si512((void *)(dst + 0 * 64), zmm0); > + _mm512_storeu_si512((void *)(dst + 1 * 64), zmm1); > + dst =3D dst + 128; > + } > +} > + > +/** > + * Copy 512-byte blocks from one location to another, > + * locations should not overlap. > + */ > +static inline void > +rte_mov512blocks(uint8_t *dst, const uint8_t *src, size_t n) > +{ > + __m512i zmm0, zmm1, zmm2, zmm3, zmm4, zmm5, zmm6, zmm7; > + > + while (n >=3D 512) { > + zmm0 =3D _mm512_loadu_si512((const void *)(src + 0 * 64)); > + n -=3D 512; > + zmm1 =3D _mm512_loadu_si512((const void *)(src + 1 * 64)); > + zmm2 =3D _mm512_loadu_si512((const void *)(src + 2 * 64)); > + zmm3 =3D _mm512_loadu_si512((const void *)(src + 3 * 64)); > + zmm4 =3D _mm512_loadu_si512((const void *)(src + 4 * 64)); > + zmm5 =3D _mm512_loadu_si512((const void *)(src + 5 * 64)); > + zmm6 =3D _mm512_loadu_si512((const void *)(src + 6 * 64)); > + zmm7 =3D _mm512_loadu_si512((const void *)(src + 7 * 64)); > + src =3D src + 512; > + _mm512_storeu_si512((void *)(dst + 0 * 64), zmm0); > + _mm512_storeu_si512((void *)(dst + 1 * 64), zmm1); > + _mm512_storeu_si512((void *)(dst + 2 * 64), zmm2); > + _mm512_storeu_si512((void *)(dst + 3 * 64), zmm3); > + _mm512_storeu_si512((void *)(dst + 4 * 64), zmm4); > + _mm512_storeu_si512((void *)(dst + 5 * 64), zmm5); > + _mm512_storeu_si512((void *)(dst + 6 * 64), zmm6); > + _mm512_storeu_si512((void *)(dst + 7 * 64), zmm7); > + dst =3D dst + 512; > + } > +} > + > +static inline void * > +rte_memcpy_generic(void *dst, const void *src, size_t n) > +{ > + uintptr_t dstu =3D (uintptr_t)dst; > + uintptr_t srcu =3D (uintptr_t)src; > + void *ret =3D dst; > + size_t dstofss; > + size_t bits; > + > + /** > + * Copy less than 16 bytes > + */ > + if (n < 16) { > + if (n & 0x01) { > + *(uint8_t *)dstu =3D *(const uint8_t *)srcu; > + srcu =3D (uintptr_t)((const uint8_t *)srcu + 1); > + dstu =3D (uintptr_t)((uint8_t *)dstu + 1); > + } > + if (n & 0x02) { > + *(uint16_t *)dstu =3D *(const uint16_t *)srcu; > + srcu =3D (uintptr_t)((const uint16_t *)srcu + 1); > + dstu =3D (uintptr_t)((uint16_t *)dstu + 1); > + } > + if (n & 0x04) { > + *(uint32_t *)dstu =3D *(const uint32_t *)srcu; > + srcu =3D (uintptr_t)((const uint32_t *)srcu + 1); > + dstu =3D (uintptr_t)((uint32_t *)dstu + 1); > + } > + if (n & 0x08) > + *(uint64_t *)dstu =3D *(const uint64_t *)srcu; > + return ret; > + } > + > + /** > + * Fast way when copy size doesn't exceed 512 bytes > + */ > + if (n <=3D 32) { > + rte_mov16((uint8_t *)dst, (const uint8_t *)src); > + rte_mov16((uint8_t *)dst - 16 + n, > + (const uint8_t *)src - 16 + n); > + return ret; > + } > + if (n <=3D 64) { > + rte_mov32((uint8_t *)dst, (const uint8_t *)src); > + rte_mov32((uint8_t *)dst - 32 + n, > + (const uint8_t *)src - 32 + n); > + return ret; > + } > + if (n <=3D 512) { > + if (n >=3D 256) { > + n -=3D 256; > + rte_mov256((uint8_t *)dst, (const uint8_t *)src); > + src =3D (const uint8_t *)src + 256; > + dst =3D (uint8_t *)dst + 256; > + } > + if (n >=3D 128) { > + n -=3D 128; > + rte_mov128((uint8_t *)dst, (const uint8_t *)src); > + src =3D (const uint8_t *)src + 128; > + dst =3D (uint8_t *)dst + 128; > + } > +COPY_BLOCK_128_BACK63: > + if (n > 64) { > + rte_mov64((uint8_t *)dst, (const uint8_t *)src); > + rte_mov64((uint8_t *)dst - 64 + n, > + (const uint8_t *)src - 64 + n); > + return ret; > + } > + if (n > 0) > + rte_mov64((uint8_t *)dst - 64 + n, > + (const uint8_t *)src - 64 + n); > + return ret; > + } > + > + /** > + * Make store aligned when copy size exceeds 512 bytes > + */ > + dstofss =3D ((uintptr_t)dst & 0x3F); > + if (dstofss > 0) { > + dstofss =3D 64 - dstofss; > + n -=3D dstofss; > + rte_mov64((uint8_t *)dst, (const uint8_t *)src); > + src =3D (const uint8_t *)src + dstofss; > + dst =3D (uint8_t *)dst + dstofss; > + } > + > + /** > + * Copy 512-byte blocks. > + * Use copy block function for better instruction order control, > + * which is important when load is unaligned. > + */ > + rte_mov512blocks((uint8_t *)dst, (const uint8_t *)src, n); > + bits =3D n; > + n =3D n & 511; > + bits -=3D n; > + src =3D (const uint8_t *)src + bits; > + dst =3D (uint8_t *)dst + bits; > + > + /** > + * Copy 128-byte blocks. > + * Use copy block function for better instruction order control, > + * which is important when load is unaligned. > + */ > + if (n >=3D 128) { > + rte_mov128blocks((uint8_t *)dst, (const uint8_t *)src, n); > + bits =3D n; > + n =3D n & 127; > + bits -=3D n; > + src =3D (const uint8_t *)src + bits; > + dst =3D (uint8_t *)dst + bits; > + } > + > + /** > + * Copy whatever left > + */ > + goto COPY_BLOCK_128_BACK63; > +} > + > +#elif defined RTE_MACHINE_CPUFLAG_AVX2 > + > +#define ALIGNMENT_MASK 0x1F > + > +/** > + * AVX2 implementation below > + */ > + > +/** > + * Copy 16 bytes from one location to another, > + * locations should not overlap. > + */ > +static inline void > +rte_mov16(uint8_t *dst, const uint8_t *src) > +{ > + __m128i xmm0; > + > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > + _mm_storeu_si128((__m128i *)dst, xmm0); > +} > + > +/** > + * Copy 32 bytes from one location to another, > + * locations should not overlap. > + */ > +static inline void > +rte_mov32(uint8_t *dst, const uint8_t *src) > +{ > + __m256i ymm0; > + > + ymm0 =3D _mm256_loadu_si256((const __m256i *)src); > + _mm256_storeu_si256((__m256i *)dst, ymm0); > +} > + > +/** > + * Copy 64 bytes from one location to another, > + * locations should not overlap. > + */ > +static inline void > +rte_mov64(uint8_t *dst, const uint8_t *src) > +{ > + rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32); > + rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32); > +} > + > +/** > + * Copy 128 bytes from one location to another, > + * locations should not overlap. > + */ > +static inline void > +rte_mov128(uint8_t *dst, const uint8_t *src) > +{ > + rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32); > + rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32); > + rte_mov32((uint8_t *)dst + 2 * 32, (const uint8_t *)src + 2 * 32); > + rte_mov32((uint8_t *)dst + 3 * 32, (const uint8_t *)src + 3 * 32); > +} > + > +/** > + * Copy 128-byte blocks from one location to another, > + * locations should not overlap. > + */ > +static inline void > +rte_mov128blocks(uint8_t *dst, const uint8_t *src, size_t n) > +{ > + __m256i ymm0, ymm1, ymm2, ymm3; > + > + while (n >=3D 128) { > + ymm0 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src + 0 * 32)); > + n -=3D 128; > + ymm1 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src + 1 * 32)); > + ymm2 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src + 2 * 32)); > + ymm3 =3D _mm256_loadu_si256((const __m256i *) > + ((const uint8_t *)src + 3 * 32)); > + src =3D (const uint8_t *)src + 128; > + _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 0 * 32), ymm0); > + _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 1 * 32), ymm1); > + _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 2 * 32), ymm2); > + _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 3 * 32), ymm3); > + dst =3D (uint8_t *)dst + 128; > + } > +} > + > +static inline void * > +rte_memcpy_generic(void *dst, const void *src, size_t n) > +{ > + uintptr_t dstu =3D (uintptr_t)dst; > + uintptr_t srcu =3D (uintptr_t)src; > + void *ret =3D dst; > + size_t dstofss; > + size_t bits; > + > + /** > + * Copy less than 16 bytes > + */ > + if (n < 16) { > + if (n & 0x01) { > + *(uint8_t *)dstu =3D *(const uint8_t *)srcu; > + srcu =3D (uintptr_t)((const uint8_t *)srcu + 1); > + dstu =3D (uintptr_t)((uint8_t *)dstu + 1); > + } > + if (n & 0x02) { > + *(uint16_t *)dstu =3D *(const uint16_t *)srcu; > + srcu =3D (uintptr_t)((const uint16_t *)srcu + 1); > + dstu =3D (uintptr_t)((uint16_t *)dstu + 1); > + } > + if (n & 0x04) { > + *(uint32_t *)dstu =3D *(const uint32_t *)srcu; > + srcu =3D (uintptr_t)((const uint32_t *)srcu + 1); > + dstu =3D (uintptr_t)((uint32_t *)dstu + 1); > + } > + if (n & 0x08) > + *(uint64_t *)dstu =3D *(const uint64_t *)srcu; > + return ret; > + } > + > + /** > + * Fast way when copy size doesn't exceed 256 bytes > + */ > + if (n <=3D 32) { > + rte_mov16((uint8_t *)dst, (const uint8_t *)src); > + rte_mov16((uint8_t *)dst - 16 + n, > + (const uint8_t *)src - 16 + n); > + return ret; > + } > + if (n <=3D 48) { > + rte_mov16((uint8_t *)dst, (const uint8_t *)src); > + rte_mov16((uint8_t *)dst + 16, (const uint8_t *)src + 16); > + rte_mov16((uint8_t *)dst - 16 + n, > + (const uint8_t *)src - 16 + n); > + return ret; > + } > + if (n <=3D 64) { > + rte_mov32((uint8_t *)dst, (const uint8_t *)src); > + rte_mov32((uint8_t *)dst - 32 + n, > + (const uint8_t *)src - 32 + n); > + return ret; > + } > + if (n <=3D 256) { > + if (n >=3D 128) { > + n -=3D 128; > + rte_mov128((uint8_t *)dst, (const uint8_t *)src); > + src =3D (const uint8_t *)src + 128; > + dst =3D (uint8_t *)dst + 128; > + } > +COPY_BLOCK_128_BACK31: > + if (n >=3D 64) { > + n -=3D 64; > + rte_mov64((uint8_t *)dst, (const uint8_t *)src); > + src =3D (const uint8_t *)src + 64; > + dst =3D (uint8_t *)dst + 64; > + } > + if (n > 32) { > + rte_mov32((uint8_t *)dst, (const uint8_t *)src); > + rte_mov32((uint8_t *)dst - 32 + n, > + (const uint8_t *)src - 32 + n); > + return ret; > + } > + if (n > 0) { > + rte_mov32((uint8_t *)dst - 32 + n, > + (const uint8_t *)src - 32 + n); > + } > + return ret; > + } > + > + /** > + * Make store aligned when copy size exceeds 256 bytes > + */ > + dstofss =3D (uintptr_t)dst & 0x1F; > + if (dstofss > 0) { > + dstofss =3D 32 - dstofss; > + n -=3D dstofss; > + rte_mov32((uint8_t *)dst, (const uint8_t *)src); > + src =3D (const uint8_t *)src + dstofss; > + dst =3D (uint8_t *)dst + dstofss; > + } > + > + /** > + * Copy 128-byte blocks > + */ > + rte_mov128blocks((uint8_t *)dst, (const uint8_t *)src, n); > + bits =3D n; > + n =3D n & 127; > + bits -=3D n; > + src =3D (const uint8_t *)src + bits; > + dst =3D (uint8_t *)dst + bits; > + > + /** > + * Copy whatever left > + */ > + goto COPY_BLOCK_128_BACK31; > +} > + > +#else /* RTE_MACHINE_CPUFLAG */ > + > +#define ALIGNMENT_MASK 0x0F > + > +/** > + * SSE & AVX implementation below > + */ > + > +/** > + * Copy 16 bytes from one location to another, > + * locations should not overlap. > + */ > +static inline void > +rte_mov16(uint8_t *dst, const uint8_t *src) > +{ > + __m128i xmm0; > + > + xmm0 =3D _mm_loadu_si128((const __m128i *)(const __m128i *)src); > + _mm_storeu_si128((__m128i *)dst, xmm0); > +} > + > +/** > + * Copy 32 bytes from one location to another, > + * locations should not overlap. > + */ > +static inline void > +rte_mov32(uint8_t *dst, const uint8_t *src) > +{ > + rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16); > + rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16); > +} > + > +/** > + * Copy 64 bytes from one location to another, > + * locations should not overlap. > + */ > +static inline void > +rte_mov64(uint8_t *dst, const uint8_t *src) > +{ > + rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16); > + rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16); > + rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16); > + rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16); > +} > + > +/** > + * Copy 128 bytes from one location to another, > + * locations should not overlap. > + */ > +static inline void > +rte_mov128(uint8_t *dst, const uint8_t *src) > +{ > + rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16); > + rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16); > + rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16); > + rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16); > + rte_mov16((uint8_t *)dst + 4 * 16, (const uint8_t *)src + 4 * 16); > + rte_mov16((uint8_t *)dst + 5 * 16, (const uint8_t *)src + 5 * 16); > + rte_mov16((uint8_t *)dst + 6 * 16, (const uint8_t *)src + 6 * 16); > + rte_mov16((uint8_t *)dst + 7 * 16, (const uint8_t *)src + 7 * 16); > +} > + > +/** > + * Copy 256 bytes from one location to another, > + * locations should not overlap. > + */ > +static inline void > +rte_mov256(uint8_t *dst, const uint8_t *src) > +{ > + rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16); > + rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16); > + rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16); > + rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16); > + rte_mov16((uint8_t *)dst + 4 * 16, (const uint8_t *)src + 4 * 16); > + rte_mov16((uint8_t *)dst + 5 * 16, (const uint8_t *)src + 5 * 16); > + rte_mov16((uint8_t *)dst + 6 * 16, (const uint8_t *)src + 6 * 16); > + rte_mov16((uint8_t *)dst + 7 * 16, (const uint8_t *)src + 7 * 16); > + rte_mov16((uint8_t *)dst + 8 * 16, (const uint8_t *)src + 8 * 16); > + rte_mov16((uint8_t *)dst + 9 * 16, (const uint8_t *)src + 9 * 16); > + rte_mov16((uint8_t *)dst + 10 * 16, (const uint8_t *)src + 10 * 16); > + rte_mov16((uint8_t *)dst + 11 * 16, (const uint8_t *)src + 11 * 16); > + rte_mov16((uint8_t *)dst + 12 * 16, (const uint8_t *)src + 12 * 16); > + rte_mov16((uint8_t *)dst + 13 * 16, (const uint8_t *)src + 13 * 16); > + rte_mov16((uint8_t *)dst + 14 * 16, (const uint8_t *)src + 14 * 16); > + rte_mov16((uint8_t *)dst + 15 * 16, (const uint8_t *)src + 15 * 16); > +} > + > +/** > + * Macro for copying unaligned block from one location to another with c= onstant load offset, > + * 47 bytes leftover maximum, > + * locations should not overlap. > + * Requirements: > + * - Store is aligned > + * - Load offset is , which must be immediate value within [1, 1= 5] > + * - For , make sure bit backwards & <16 - offset> bit for= wards are available for loading > + * - , , must be variables > + * - __m128i ~ must be pre-defined > + */ > +#define MOVEUNALIGNED_LEFT47_IMM(dst, src, len, offset) = \ > +__extension__ ({ = \ > + int tmp; = \ > + while (len >=3D 128 + 16 - offset) { = \ > + xmm0 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 0 * 16)); \ > + len -=3D 128; = \ > + xmm1 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 1 * 16)); \ > + xmm2 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 2 * 16)); \ > + xmm3 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 3 * 16)); \ > + xmm4 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 4 * 16)); \ > + xmm5 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 5 * 16)); \ > + xmm6 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 6 * 16)); \ > + xmm7 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 7 * 16)); \ > + xmm8 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 8 * 16)); \ > + src =3D (const uint8_t *)src + 128; = \ > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16), _mm_align= r_epi8(xmm1, xmm0, offset)); \ > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16), _mm_align= r_epi8(xmm2, xmm1, offset)); \ > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 2 * 16), _mm_align= r_epi8(xmm3, xmm2, offset)); \ > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 3 * 16), _mm_align= r_epi8(xmm4, xmm3, offset)); \ > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 4 * 16), _mm_align= r_epi8(xmm5, xmm4, offset)); \ > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 5 * 16), _mm_align= r_epi8(xmm6, xmm5, offset)); \ > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 6 * 16), _mm_align= r_epi8(xmm7, xmm6, offset)); \ > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 7 * 16), _mm_align= r_epi8(xmm8, xmm7, offset)); \ > + dst =3D (uint8_t *)dst + 128; = \ > + } = \ > + tmp =3D len; = \ > + len =3D ((len - 16 + offset) & 127) + 16 - offset; = \ > + tmp -=3D len; = \ > + src =3D (const uint8_t *)src + tmp; = \ > + dst =3D (uint8_t *)dst + tmp; = \ > + if (len >=3D 32 + 16 - offset) { = \ > + while (len >=3D 32 + 16 - offset) { = \ > + xmm0 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)= src - offset + 0 * 16)); \ > + len -=3D 32; = \ > + xmm1 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)= src - offset + 1 * 16)); \ > + xmm2 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)= src - offset + 2 * 16)); \ > + src =3D (const uint8_t *)src + 32; = \ > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16), _mm_a= lignr_epi8(xmm1, xmm0, offset)); \ > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16), _mm_a= lignr_epi8(xmm2, xmm1, offset)); \ > + dst =3D (uint8_t *)dst + 32; = \ > + } = \ > + tmp =3D len; = \ > + len =3D ((len - 16 + offset) & 31) + 16 - offset; = \ > + tmp -=3D len; = \ > + src =3D (const uint8_t *)src + tmp; = \ > + dst =3D (uint8_t *)dst + tmp; = \ > + } = \ > +}) > + > +/** > + * Macro for copying unaligned block from one location to another, > + * 47 bytes leftover maximum, > + * locations should not overlap. > + * Use switch here because the aligning instruction requires immediate v= alue for shift count. > + * Requirements: > + * - Store is aligned > + * - Load offset is , which must be within [1, 15] > + * - For , make sure bit backwards & <16 - offset> bit for= wards are available for loading > + * - , , must be variables > + * - __m128i ~ used in MOVEUNALIGNED_LEFT47_IMM must be pr= e-defined > + */ > +#define MOVEUNALIGNED_LEFT47(dst, src, len, offset) \ > +__extension__ ({ \ > + switch (offset) { \ > + case 0x01: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x01); break; \ > + case 0x02: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x02); break; \ > + case 0x03: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x03); break; \ > + case 0x04: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x04); break; \ > + case 0x05: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x05); break; \ > + case 0x06: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x06); break; \ > + case 0x07: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x07); break; \ > + case 0x08: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x08); break; \ > + case 0x09: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x09); break; \ > + case 0x0A: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0A); break; \ > + case 0x0B: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0B); break; \ > + case 0x0C: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0C); break; \ > + case 0x0D: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0D); break; \ > + case 0x0E: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0E); break; \ > + case 0x0F: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0F); break; \ > + default:; \ > + } \ > +}) > + > +static inline void * > +rte_memcpy_generic(void *dst, const void *src, size_t n) > +{ > + __m128i xmm0, xmm1, xmm2, xmm3, xmm4, xmm5, xmm6, xmm7, xmm8; > + uintptr_t dstu =3D (uintptr_t)dst; > + uintptr_t srcu =3D (uintptr_t)src; > + void *ret =3D dst; > + size_t dstofss; > + size_t srcofs; > + > + /** > + * Copy less than 16 bytes > + */ > + if (n < 16) { > + if (n & 0x01) { > + *(uint8_t *)dstu =3D *(const uint8_t *)srcu; > + srcu =3D (uintptr_t)((const uint8_t *)srcu + 1); > + dstu =3D (uintptr_t)((uint8_t *)dstu + 1); > + } > + if (n & 0x02) { > + *(uint16_t *)dstu =3D *(const uint16_t *)srcu; > + srcu =3D (uintptr_t)((const uint16_t *)srcu + 1); > + dstu =3D (uintptr_t)((uint16_t *)dstu + 1); > + } > + if (n & 0x04) { > + *(uint32_t *)dstu =3D *(const uint32_t *)srcu; > + srcu =3D (uintptr_t)((const uint32_t *)srcu + 1); > + dstu =3D (uintptr_t)((uint32_t *)dstu + 1); > + } > + if (n & 0x08) > + *(uint64_t *)dstu =3D *(const uint64_t *)srcu; > + return ret; > + } > + > + /** > + * Fast way when copy size doesn't exceed 512 bytes > + */ > + if (n <=3D 32) { > + rte_mov16((uint8_t *)dst, (const uint8_t *)src); > + rte_mov16((uint8_t *)dst - 16 + n, > + (const uint8_t *)src - 16 + n); > + return ret; > + } > + if (n <=3D 48) { > + rte_mov32((uint8_t *)dst, (const uint8_t *)src); > + rte_mov16((uint8_t *)dst - 16 + n, > + (const uint8_t *)src - 16 + n); > + return ret; > + } > + if (n <=3D 64) { > + rte_mov32((uint8_t *)dst, (const uint8_t *)src); > + rte_mov16((uint8_t *)dst + 32, (const uint8_t *)src + 32); > + rte_mov16((uint8_t *)dst - 16 + n, > + (const uint8_t *)src - 16 + n); > + return ret; > + } > + if (n <=3D 128) > + goto COPY_BLOCK_128_BACK15; > + if (n <=3D 512) { > + if (n >=3D 256) { > + n -=3D 256; > + rte_mov128((uint8_t *)dst, (const uint8_t *)src); > + rte_mov128((uint8_t *)dst + 128, > + (const uint8_t *)src + 128); > + src =3D (const uint8_t *)src + 256; > + dst =3D (uint8_t *)dst + 256; > + } > +COPY_BLOCK_255_BACK15: > + if (n >=3D 128) { > + n -=3D 128; > + rte_mov128((uint8_t *)dst, (const uint8_t *)src); > + src =3D (const uint8_t *)src + 128; > + dst =3D (uint8_t *)dst + 128; > + } > +COPY_BLOCK_128_BACK15: > + if (n >=3D 64) { > + n -=3D 64; > + rte_mov64((uint8_t *)dst, (const uint8_t *)src); > + src =3D (const uint8_t *)src + 64; > + dst =3D (uint8_t *)dst + 64; > + } > +COPY_BLOCK_64_BACK15: > + if (n >=3D 32) { > + n -=3D 32; > + rte_mov32((uint8_t *)dst, (const uint8_t *)src); > + src =3D (const uint8_t *)src + 32; > + dst =3D (uint8_t *)dst + 32; > + } > + if (n > 16) { > + rte_mov16((uint8_t *)dst, (const uint8_t *)src); > + rte_mov16((uint8_t *)dst - 16 + n, > + (const uint8_t *)src - 16 + n); > + return ret; > + } > + if (n > 0) { > + rte_mov16((uint8_t *)dst - 16 + n, > + (const uint8_t *)src - 16 + n); > + } > + return ret; > + } > + > + /** > + * Make store aligned when copy size exceeds 512 bytes, > + * and make sure the first 15 bytes are copied, because > + * unaligned copy functions require up to 15 bytes > + * backwards access. > + */ > + dstofss =3D (uintptr_t)dst & 0x0F; > + if (dstofss > 0) { > + dstofss =3D 16 - dstofss + 16; > + n -=3D dstofss; > + rte_mov32((uint8_t *)dst, (const uint8_t *)src); > + src =3D (const uint8_t *)src + dstofss; > + dst =3D (uint8_t *)dst + dstofss; > + } > + srcofs =3D ((uintptr_t)src & 0x0F); > + > + /** > + * For aligned copy > + */ > + if (srcofs =3D=3D 0) { > + /** > + * Copy 256-byte blocks > + */ > + for (; n >=3D 256; n -=3D 256) { > + rte_mov256((uint8_t *)dst, (const uint8_t *)src); > + dst =3D (uint8_t *)dst + 256; > + src =3D (const uint8_t *)src + 256; > + } > + > + /** > + * Copy whatever left > + */ > + goto COPY_BLOCK_255_BACK15; > + } > + > + /** > + * For copy with unaligned load > + */ > + MOVEUNALIGNED_LEFT47(dst, src, n, srcofs); > + > + /** > + * Copy whatever left > + */ > + goto COPY_BLOCK_64_BACK15; > +} > + > +#endif /* RTE_MACHINE_CPUFLAG */ > + > +static inline void * > +rte_memcpy_aligned(void *dst, const void *src, size_t n) > +{ > + void *ret =3D dst; > + > + /* Copy size <=3D 16 bytes */ > + if (n < 16) { > + if (n & 0x01) { > + *(uint8_t *)dst =3D *(const uint8_t *)src; > + src =3D (const uint8_t *)src + 1; > + dst =3D (uint8_t *)dst + 1; > + } > + if (n & 0x02) { > + *(uint16_t *)dst =3D *(const uint16_t *)src; > + src =3D (const uint16_t *)src + 1; > + dst =3D (uint16_t *)dst + 1; > + } > + if (n & 0x04) { > + *(uint32_t *)dst =3D *(const uint32_t *)src; > + src =3D (const uint32_t *)src + 1; > + dst =3D (uint32_t *)dst + 1; > + } > + if (n & 0x08) > + *(uint64_t *)dst =3D *(const uint64_t *)src; > + > + return ret; > + } > + > + /* Copy 16 <=3D size <=3D 32 bytes */ > + if (n <=3D 32) { > + rte_mov16((uint8_t *)dst, (const uint8_t *)src); > + rte_mov16((uint8_t *)dst - 16 + n, > + (const uint8_t *)src - 16 + n); > + > + return ret; > + } > + > + /* Copy 32 < size <=3D 64 bytes */ > + if (n <=3D 64) { > + rte_mov32((uint8_t *)dst, (const uint8_t *)src); > + rte_mov32((uint8_t *)dst - 32 + n, > + (const uint8_t *)src - 32 + n); > + > + return ret; > + } > + > + /* Copy 64 bytes blocks */ > + for (; n >=3D 64; n -=3D 64) { > + rte_mov64((uint8_t *)dst, (const uint8_t *)src); > + dst =3D (uint8_t *)dst + 64; > + src =3D (const uint8_t *)src + 64; > + } > + > + /* Copy whatever left */ > + rte_mov64((uint8_t *)dst - 64 + n, > + (const uint8_t *)src - 64 + n); > + > + return ret; > +} > + > +static inline void * > +rte_memcpy_internal(void *dst, const void *src, size_t n) > +{ > + if (!(((uintptr_t)dst | (uintptr_t)src) & ALIGNMENT_MASK)) > + return rte_memcpy_aligned(dst, src, n); > + else > + return rte_memcpy_generic(dst, src, n); > +} > + > +#ifdef __cplusplus > +} > +#endif > + > +#endif /* _RTE_MEMCPY_INTERNAL_X86_64_H_ */ > diff --git a/lib/librte_eal/common/include/arch/x86/rte_memcpy_sse.c b/li= b/librte_eal/common/include/arch/x86/rte_memcpy_sse.c > new file mode 100644 > index 0000000..2532696 > --- /dev/null > +++ b/lib/librte_eal/common/include/arch/x86/rte_memcpy_sse.c > @@ -0,0 +1,585 @@ > +/*- > + * BSD LICENSE > + * > + * Copyright(c) 2010-2017 Intel Corporation. All rights reserved. > + * All rights reserved. > + * > + * Redistribution and use in source and binary forms, with or without > + * modification, are permitted provided that the following conditions > + * are met: > + * > + * * Redistributions of source code must retain the above copyright > + * notice, this list of conditions and the following disclaimer. > + * * Redistributions in binary form must reproduce the above copyrig= ht > + * notice, this list of conditions and the following disclaimer in > + * the documentation and/or other materials provided with the > + * distribution. > + * * Neither the name of Intel Corporation nor the names of its > + * contributors may be used to endorse or promote products derived > + * from this software without specific prior written permission. > + * > + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS > + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT > + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS F= OR > + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGH= T > + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTA= L, > + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT > + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF US= E, > + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON A= NY > + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT > + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE U= SE > + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE= . > + */ > + > +#include > + > +/** > + * Macro for copying unaligned block from one location to another with c= onstant load offset, > + * 47 bytes leftover maximum, > + * locations should not overlap. > + * Requirements: > + * - Store is aligned > + * - Load offset is , which must be immediate value within [1, 1= 5] > + * - For , make sure bit backwards & <16 - offset> bit for= wards are available for loading > + * - , , must be variables > + * - __m128i ~ must be pre-defined > + */ > +#define MOVEUNALIGNED_LEFT47_IMM(dst, src, len, offset) = \ > +__extension__ ({ = \ > + int tmp; = \ > + while (len >=3D 128 + 16 - offset) { = \ > + xmm0 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 0 * 16)); \ > + len -=3D 128; = \ > + xmm1 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 1 * 16)); \ > + xmm2 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 2 * 16)); \ > + xmm3 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 3 * 16)); \ > + xmm4 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 4 * 16)); \ > + xmm5 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 5 * 16)); \ > + xmm6 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 6 * 16)); \ > + xmm7 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 7 * 16)); \ > + xmm8 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)src = - offset + 8 * 16)); \ > + src =3D (const uint8_t *)src + 128; = \ > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16), _mm_align= r_epi8(xmm1, xmm0, offset)); \ > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16), _mm_align= r_epi8(xmm2, xmm1, offset)); \ > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 2 * 16), _mm_align= r_epi8(xmm3, xmm2, offset)); \ > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 3 * 16), _mm_align= r_epi8(xmm4, xmm3, offset)); \ > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 4 * 16), _mm_align= r_epi8(xmm5, xmm4, offset)); \ > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 5 * 16), _mm_align= r_epi8(xmm6, xmm5, offset)); \ > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 6 * 16), _mm_align= r_epi8(xmm7, xmm6, offset)); \ > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 7 * 16), _mm_align= r_epi8(xmm8, xmm7, offset)); \ > + dst =3D (uint8_t *)dst + 128; = \ > + } = \ > + tmp =3D len; = \ > + len =3D ((len - 16 + offset) & 127) + 16 - offset; = \ > + tmp -=3D len; = \ > + src =3D (const uint8_t *)src + tmp; = \ > + dst =3D (uint8_t *)dst + tmp; = \ > + if (len >=3D 32 + 16 - offset) { = \ > + while (len >=3D 32 + 16 - offset) { = \ > + xmm0 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)= src - offset + 0 * 16)); \ > + len -=3D 32; = \ > + xmm1 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)= src - offset + 1 * 16)); \ > + xmm2 =3D _mm_loadu_si128((const __m128i *)((const uint8_t *)= src - offset + 2 * 16)); \ > + src =3D (const uint8_t *)src + 32; = \ > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16), _mm_a= lignr_epi8(xmm1, xmm0, offset)); \ > + _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16), _mm_a= lignr_epi8(xmm2, xmm1, offset)); \ > + dst =3D (uint8_t *)dst + 32; = \ > + } = \ > + tmp =3D len; = \ > + len =3D ((len - 16 + offset) & 31) + 16 - offset; = \ > + tmp -=3D len; = \ > + src =3D (const uint8_t *)src + tmp; = \ > + dst =3D (uint8_t *)dst + tmp; = \ > + } = \ > +}) > + > +/** > + * Macro for copying unaligned block from one location to another, > + * 47 bytes leftover maximum, > + * locations should not overlap. > + * Use switch here because the aligning instruction requires immediate v= alue for shift count. > + * Requirements: > + * - Store is aligned > + * - Load offset is , which must be within [1, 15] > + * - For , make sure bit backwards & <16 - offset> bit for= wards are available for loading > + * - , , must be variables > + * - __m128i ~ used in MOVEUNALIGNED_LEFT47_IMM must be pr= e-defined > + */ > +#define MOVEUNALIGNED_LEFT47(dst, src, len, offset) \ > +__extension__ ({ \ > + switch (offset) { \ > + case 0x01: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x01); break; \ > + case 0x02: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x02); break; \ > + case 0x03: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x03); break; \ > + case 0x04: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x04); break; \ > + case 0x05: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x05); break; \ > + case 0x06: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x06); break; \ > + case 0x07: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x07); break; \ > + case 0x08: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x08); break; \ > + case 0x09: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x09); break; \ > + case 0x0A: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0A); break; \ > + case 0x0B: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0B); break; \ > + case 0x0C: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0C); break; \ > + case 0x0D: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0D); break; \ > + case 0x0E: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0E); break; \ > + case 0x0F: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0F); break; \ > + default:; \ > + } \ > +}) > + > +void * > +rte_memcpy_sse(void *dst, const void *src, size_t n) > +{ > + if (!(((uintptr_t)dst | (uintptr_t)src) & 0x0F)) { > + void *ret =3D dst; > + > + /* Copy size <=3D 16 bytes */ > + if (n < 16) { > + if (n & 0x01) { > + *(uint8_t *)dst =3D *(const uint8_t *)src; > + src =3D (const uint8_t *)src + 1; > + dst =3D (uint8_t *)dst + 1; > + } > + if (n & 0x02) { > + *(uint16_t *)dst =3D *(const uint16_t *)src; > + src =3D (const uint16_t *)src + 1; > + dst =3D (uint16_t *)dst + 1; > + } > + if (n & 0x04) { > + *(uint32_t *)dst =3D *(const uint32_t *)src; > + src =3D (const uint32_t *)src + 1; > + dst =3D (uint32_t *)dst + 1; > + } > + if (n & 0x08) > + *(uint64_t *)dst =3D *(const uint64_t *)src; > + > + return ret; > + } > + > + /* Copy 16 <=3D size <=3D 32 bytes */ > + if (n <=3D 32) { > + __m128i xmm0, xmm1; > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src - 16 + n)); > + _mm_storeu_si128((__m128i *)dst, xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst - 16 + n), xmm1); > + > + return ret; > + } > + > + /* Copy 32 < size <=3D 64 bytes */ > + if (n <=3D 64) { > + __m128i xmm0, xmm1, xmm2, xmm3; > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 16)); > + xmm2 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src - 32 + n)); > + xmm3 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src - 16 + n)); > + _mm_storeu_si128((__m128i *)dst, xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 16), xmm1); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst - 32 + n), xmm2); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst - 16 + n), xmm3); > + > + return ret; > + } > + > + /* Copy 64 bytes blocks */ > + for (; n >=3D 64; n -=3D 64) { > + __m128i xmm0, xmm1, xmm2, xmm3; > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 16)); > + xmm2 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 2*16)); > + xmm3 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 3*16)); > + _mm_storeu_si128((__m128i *)dst, xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 16), xmm1); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 2*16), xmm2); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 3*16), xmm3); > + dst =3D (uint8_t *)dst + 64; > + src =3D (const uint8_t *)src + 64; > + } > + > + /* Copy whatever left */ > + __m128i xmm0, xmm1, xmm2, xmm3; > + xmm0 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src - 64 + n)); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src - 48 + n)); > + xmm2 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src - 32 + n)); > + xmm3 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src - 16 + n)); > + _mm_storeu_si128((__m128i *)((uint8_t *)dst - 64 + n), xmm0); > + _mm_storeu_si128((__m128i *)((uint8_t *)dst - 48 + n), xmm1); > + _mm_storeu_si128((__m128i *)((uint8_t *)dst - 32 + n), xmm2); > + _mm_storeu_si128((__m128i *)((uint8_t *)dst - 16 + n), xmm3); > + > + return ret; > + } else { > + __m128i xmm0, xmm1, xmm2, xmm3, xmm4, xmm5, xmm6, xmm7, xmm8; > + uintptr_t dstu =3D (uintptr_t)dst; > + uintptr_t srcu =3D (uintptr_t)src; > + void *ret =3D dst; > + size_t dstofss; > + size_t srcofs; > + > + /** > + * Copy less than 16 bytes > + */ > + if (n < 16) { > + if (n & 0x01) { > + *(uint8_t *)dstu =3D *(const uint8_t *)srcu; > + srcu =3D (uintptr_t)((const uint8_t *)srcu + 1); > + dstu =3D (uintptr_t)((uint8_t *)dstu + 1); > + } > + if (n & 0x02) { > + *(uint16_t *)dstu =3D *(const uint16_t *)srcu; > + srcu =3D (uintptr_t)((const uint16_t *)srcu + 1); > + dstu =3D (uintptr_t)((uint16_t *)dstu + 1); > + } > + if (n & 0x04) { > + *(uint32_t *)dstu =3D *(const uint32_t *)srcu; > + srcu =3D (uintptr_t)((const uint32_t *)srcu + 1); > + dstu =3D (uintptr_t)((uint32_t *)dstu + 1); > + } > + if (n & 0x08) > + *(uint64_t *)dstu =3D *(const uint64_t *)srcu; > + return ret; > + } > + > + /** > + * Fast way when copy size doesn't exceed 512 bytes > + */ > + if (n <=3D 32) { > + __m128i xmm0, xmm1; > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src - 16 + n)); > + _mm_storeu_si128((__m128i *)dst, xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst - 16 + n), xmm1); > + return ret; > + } > + if (n <=3D 48) { > + __m128i xmm0, xmm1, xmm2; > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 16)); > + xmm2 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src - 16 + n)); > + _mm_storeu_si128((__m128i *)dst, xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 16), xmm1); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst - 16 + n), xmm2); > + return ret; > + } > + if (n <=3D 64) { > + __m128i xmm0, xmm1, xmm2, xmm3; > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 16)); > + xmm2 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 32)); > + xmm3 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src - 16 + n)); > + _mm_storeu_si128((__m128i *)dst, xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 16), xmm1); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 32), xmm2); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst - 16 + n), xmm3); > + return ret; > + } > + if (n <=3D 128) > + goto COPY_BLOCK_128_BACK15; > + if (n <=3D 512) { > + if (n >=3D 256) { > + n -=3D 256; > + __m128i xmm0, xmm1; > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 16)); > + _mm_storeu_si128((__m128i *)dst, xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 16), xmm1); > + xmm0 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 2*16)); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 3*16)); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 2*16), xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 3*16), xmm1); > + xmm0 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 4*16)); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 5*16)); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 4*16), xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 5*16), xmm1); > + xmm0 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 6*16)); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 7*16)); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 6*16), xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 7*16), xmm1); > + > + xmm0 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 128)); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 128 + 16)); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 128), xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 128 + 16), xmm1); > + xmm0 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 128 + 2*16)); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 128 + 3*16)); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 128 + 2*16), xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 128 + 3*16), xmm1); > + xmm0 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 128 + 4*16)); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 128 + 5*16)); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 128 + 4*16), xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 128 + 5*16), xmm1); > + xmm0 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 128 + 6*16)); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 128 + 7*16)); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 128 + 6*16), xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 128 + 7*16), xmm1); > + src =3D (const uint8_t *)src + 256; > + dst =3D (uint8_t *)dst + 256; > + } > +COPY_BLOCK_255_BACK15: > + if (n >=3D 128) { > + n -=3D 128; > + __m128i xmm0, xmm1; > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 16)); > + _mm_storeu_si128((__m128i *)dst, xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 16), xmm1); > + xmm0 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 2*16)); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 3*16)); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 2*16), xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 3*16), xmm1); > + xmm0 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 4*16)); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 5*16)); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 4*16), xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 5*16), xmm1); > + xmm0 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 6*16)); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 7*16)); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 6*16), xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 7*16), xmm1); > + src =3D (const uint8_t *)src + 128; > + dst =3D (uint8_t *)dst + 128; > + } > +COPY_BLOCK_128_BACK15: > + if (n >=3D 64) { > + n -=3D 64; > + __m128i xmm0, xmm1; > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 16)); > + _mm_storeu_si128((__m128i *)dst, xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 16), xmm1); > + > + xmm0 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 2*16)); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 3*16)); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 2*16), xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 3*16), xmm1); > + src =3D (const uint8_t *)src + 64; > + dst =3D (uint8_t *)dst + 64; > + } > +COPY_BLOCK_64_BACK15: > + if (n >=3D 32) { > + n -=3D 32; > + __m128i xmm0, xmm1; > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 16)); > + _mm_storeu_si128((__m128i *)dst, xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 16), xmm1); > + src =3D (const uint8_t *)src + 32; > + dst =3D (uint8_t *)dst + 32; > + } > + if (n > 16) { > + __m128i xmm0, xmm1; > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src - 16 + n)); > + _mm_storeu_si128((__m128i *)dst, xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst - 16 + n), xmm1); > + return ret; > + } > + if (n > 0) { > + __m128i xmm0; > + xmm0 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src - 16 + n)); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst - 16 + n), xmm0); > + } > + return ret; > + } > + > + /** > + * Make store aligned when copy size exceeds 512 bytes, > + * and make sure the first 15 bytes are copied, because > + * unaligned copy functions require up to 15 bytes > + * backwards access. > + */ > + dstofss =3D (uintptr_t)dst & 0x0F; > + if (dstofss > 0) { > + dstofss =3D 16 - dstofss + 16; > + n -=3D dstofss; > + __m128i xmm0, xmm1; > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 16)); > + _mm_storeu_si128((__m128i *)dst, xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 16), xmm1); > + src =3D (const uint8_t *)src + dstofss; > + dst =3D (uint8_t *)dst + dstofss; > + } > + srcofs =3D ((uintptr_t)src & 0x0F); > + > + /** > + * For aligned copy > + */ > + if (srcofs =3D=3D 0) { > + /** > + * Copy 256-byte blocks > + */ > + for (; n >=3D 256; n -=3D 256) { > + __m128i xmm0, xmm1; > + xmm0 =3D _mm_loadu_si128((const __m128i *)src); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 16)); > + _mm_storeu_si128((__m128i *)dst, xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 16), xmm1); > + xmm0 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 2*16)); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 3*16)); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 2*16), xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 3*16), xmm1); > + xmm0 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 4*16)); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 5*16)); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 4*16), xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 5*16), xmm1); > + xmm0 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 6*16)); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 7*16)); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 6*16), xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 7*16), xmm1); > + > + xmm0 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 8*16)); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 9*16)); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 8*16), xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 9*16), xmm1); > + xmm0 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 10*16)); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 11*16)); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 10*16), xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 11*16), xmm1); > + xmm0 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 12*16)); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 13*16)); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 12*16), xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 13*16), xmm1); > + xmm0 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 14*16)); > + xmm1 =3D _mm_loadu_si128((const __m128i *) > + ((const uint8_t *)src + 15*16)); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 14*16), xmm0); > + _mm_storeu_si128((__m128i *) > + ((uint8_t *)dst + 15*16), xmm1); > + dst =3D (uint8_t *)dst + 256; > + src =3D (const uint8_t *)src + 256; > + } > + > + /** > + * Copy whatever left > + */ > + goto COPY_BLOCK_255_BACK15; > + } > + > + /** > + * For copy with unaligned load > + */ > + MOVEUNALIGNED_LEFT47(dst, src, n, srcofs); > + > + /** > + * Copy whatever left > + */ > + goto COPY_BLOCK_64_BACK15; > + } > +} > diff --git a/lib/librte_eal/linuxapp/eal/Makefile b/lib/librte_eal/linuxa= pp/eal/Makefile > index 90bca4d..88d3298 100644 > --- a/lib/librte_eal/linuxapp/eal/Makefile > +++ b/lib/librte_eal/linuxapp/eal/Makefile > @@ -40,6 +40,7 @@ VPATH +=3D $(RTE_SDK)/lib/librte_eal/common/arch/$(ARCH= _DIR) > LIBABIVER :=3D 5 >=20 > VPATH +=3D $(RTE_SDK)/lib/librte_eal/common > +VPATH +=3D $(RTE_SDK)/lib/librte_eal/common/include/arch/$(ARCH_DIR) >=20 > CFLAGS +=3D -I$(SRCDIR)/include > CFLAGS +=3D -I$(RTE_SDK)/lib/librte_eal/common > @@ -105,6 +106,22 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) +=3D rte_servic= e.c > SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) +=3D rte_cpuflags.c > SRCS-$(CONFIG_RTE_ARCH_X86) +=3D rte_spinlock.c >=20 > +# for run-time dispatch of memcpy > +SRCS-$(CONFIG_RTE_ARCH_X86) +=3D rte_memcpy.c > +SRCS-$(CONFIG_RTE_ARCH_X86) +=3D rte_memcpy_sse.c > + > +# if the compiler supports AVX512, add avx512 file > +ifneq ($(filter $(MACHINE_CFLAGS),CC_SUPPORT_AVX512F),) > +SRCS-$(CONFIG_RTE_ARCH_X86) +=3D rte_memcpy_avx512f.c > +CFLAGS_rte_memcpy_avx512f.o +=3D -mavx512f > +endif > + > +# if the compiler supports AVX2, add avx2 file > +ifneq ($(filter $(MACHINE_CFLAGS),CC_SUPPORT_AVX2),) > +SRCS-$(CONFIG_RTE_ARCH_X86) +=3D rte_memcpy_avx2.c > +CFLAGS_rte_memcpy_avx2.o +=3D -mavx2 > +endif > + > CFLAGS_eal_common_cpuflags.o :=3D $(CPUFLAGS_LIST) >=20 > CFLAGS_eal.o :=3D -D_GNU_SOURCE > diff --git a/mk/rte.cpuflags.mk b/mk/rte.cpuflags.mk > index a813c91..8a7a1e7 100644 > --- a/mk/rte.cpuflags.mk > +++ b/mk/rte.cpuflags.mk > @@ -134,6 +134,20 @@ endif >=20 > MACHINE_CFLAGS +=3D $(addprefix -DRTE_MACHINE_CPUFLAG_,$(CPUFLAGS)) >=20 > +# Check if the compiler suppoerts AVX512 > +CC_SUPPORT_AVX512F :=3D $(shell $(CC) -mavx512f -dM -E - < /dev/null 2>&= 1 | grep -q AVX512 && echo 1) > +ifeq ($(CC_SUPPORT_AVX512F),1) > +ifeq ($(CONFIG_RTE_ENABLE_AVX512),y) > +MACHINE_CFLAGS +=3D -DCC_SUPPORT_AVX512F > +endif > +endif > + > +# Check if the compiler supports AVX2 > +CC_SUPPORT_AVX2 :=3D $(shell $(CC) -mavx2 -dM -E - < /dev/null 2>&1 | gr= ep -q AVX2 && echo 1) > +ifeq ($(CC_SUPPORT_AVX2),1) > +MACHINE_CFLAGS +=3D -DCC_SUPPORT_AVX2 > +endif > + > # To strip whitespace > comma:=3D , > empty:=3D > -- > 2.7.4