From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B6131A0547; Wed, 27 Oct 2021 09:28:31 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2360B410F4; Wed, 27 Oct 2021 09:28:31 +0200 (CEST) Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) by mails.dpdk.org (Postfix) with ESMTP id 74E0C410F4 for ; Wed, 27 Oct 2021 09:28:29 +0200 (CEST) Received: by mail-pj1-f45.google.com with SMTP id np13so1380104pjb.4 for ; Wed, 27 Oct 2021 00:28:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vvdntech-in.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lIwaTUT3T8oFg2keIOqoUpFkjTSp9e77BH/m6hSIZg8=; b=B/Yp6MijaDQSYDs3Oe+Msro9Fe1KY28NySLCFzhKUNuDIx6eCKAJ0+cLnXW2Zym8pm KP+K25ncPZ29BgvMg4E5RwC71fDZR4qFJ7I71E1yXvcXs6EGBdHcSTtFJoAllHB7Ef+g ZlIErr2MGq+axzh82AHRtJlEmazF3BuCiU8u6rZysC+O+gILXtJTotoRg/JMeh1n+K+t kzCi/BK1mRB8a7vCLICrzBT50roXR+SMWRndWvnIir/hs5Dy3qEAB3+vcq/scpvK5nUJ ChBQJSjYU26k22xJLTzy4k16I2lBl45QA65yGBOE+8Lrb1ndTYCG0wUZ0Kwa9v1DMVN4 32uA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lIwaTUT3T8oFg2keIOqoUpFkjTSp9e77BH/m6hSIZg8=; b=RoAFImFcFUNTz7IBSEhi+Jt07zwJfrV7ViqzDYDfoxSEnzHBUb4ypI6OKWYE6n1cU6 r/+QSsKhlnQiA53KL45BELIeiYzt1V6ERzePmoEMJAx02+ILcx97SaruHnjMxkGoLy6y /zzQqn0jDj59upx5UdRWnp6/ApXf1GSpjUGi9WYl6tBkI5FuVT5JWzhrHptaIBvwjoKF 9fm5ZpmcoEdhkDgBLVSxsytPjHr2c6UNgsCTi6C1SE4yN8Ehkpue6wkfTWsIGG69D30O 5PIK4/j+9QHgpIXcse6Cm6vLKRGKJt0f57zgKgndCixJ51AI2bf8lviqpdyUseHuUV3k QEYw== X-Gm-Message-State: AOAM532yWIlbJWYIWDoZSUmLMLufPe7is2yWJRPnBe0IcKVdoeuWu6up 523mdE9Zr7SH8QHKAKaAkasmRFlb0ypNM2jx X-Google-Smtp-Source: ABdhPJzDDy98rMDFyJ3+2Q4Xzfn2AfFvUsAe6d6GIMI8CVhew5Pjlezur54NYAYtaC+CuubMsr2Ffw== X-Received: by 2002:a17:90b:3910:: with SMTP id ob16mr4063367pjb.234.1635319708449; Wed, 27 Oct 2021 00:28:28 -0700 (PDT) Received: from 470--5GDC--BLR.blore.vvdntech.com ([106.51.39.131]) by smtp.gmail.com with ESMTPSA id f4sm21087651pgn.93.2021.10.27.00.28.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Oct 2021 00:28:28 -0700 (PDT) From: Aman Kumar To: dev@dpdk.org Cc: thomas@monjalon.net, viacheslavo@nvidia.com, anatoly.burakov@intel.com, keesang.song@amd.com, aman.kumar@vvdntech.in, jerinjacobk@gmail.com, konstantin.ananyev@intel.com, bruce.richardson@intel.com Date: Wed, 27 Oct 2021 12:58:10 +0530 Message-Id: <20211027072810.257795-2-aman.kumar@vvdntech.in> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211027072810.257795-1-aman.kumar@vvdntech.in> References: <20211026155645.246783-1-aman.kumar@vvdntech.in> <20211027072810.257795-1-aman.kumar@vvdntech.in> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH v4 2/2] lib/eal: add temporal store memcpy support for AMD platform X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch provides a rte_memcpy* call with temporal stores. Use -Dcpu_instruction_set=znverX with build to enable this API. Signed-off-by: Aman Kumar --- config/x86/meson.build | 2 + lib/eal/x86/include/rte_memcpy.h | 114 +++++++++++++++++++++++++++++++ 2 files changed, 116 insertions(+) diff --git a/config/x86/meson.build b/config/x86/meson.build index 21cda6fd33..56dae4aca7 100644 --- a/config/x86/meson.build +++ b/config/x86/meson.build @@ -78,6 +78,8 @@ if get_option('cpu_instruction_set') == 'znver1' dpdk_conf.set('RTE_MAX_LCORE', 256) elif get_option('cpu_instruction_set') == 'znver2' dpdk_conf.set('RTE_MAX_LCORE', 512) + dpdk_conf.set('RTE_MEMCPY_AMDEPYC', 1) elif get_option('cpu_instruction_set') == 'znver3' dpdk_conf.set('RTE_MAX_LCORE', 512) + dpdk_conf.set('RTE_MEMCPY_AMDEPYC', 1) endif diff --git a/lib/eal/x86/include/rte_memcpy.h b/lib/eal/x86/include/rte_memcpy.h index 1b6c6e585f..8fe7822cb4 100644 --- a/lib/eal/x86/include/rte_memcpy.h +++ b/lib/eal/x86/include/rte_memcpy.h @@ -376,6 +376,120 @@ rte_mov128blocks(uint8_t *dst, const uint8_t *src, size_t n) } } +#if defined RTE_MEMCPY_AMDEPYC + +/** + * Copy 16 bytes from one location to another, + * with temporal stores + */ +static __rte_always_inline void +rte_copy16_ts(uint8_t *dst, uint8_t *src) +{ + __m128i var128; + + var128 = _mm_stream_load_si128((__m128i *)src); + _mm_storeu_si128((__m128i *)dst, var128); +} + +/** + * Copy 32 bytes from one location to another, + * with temporal stores + */ +static __rte_always_inline void +rte_copy32_ts(uint8_t *dst, uint8_t *src) +{ + __m256i ymm0; + + ymm0 = _mm256_stream_load_si256((const __m256i *)src); + _mm256_storeu_si256((__m256i *)dst, ymm0); +} + +/** + * Copy 64 bytes from one location to another, + * with temporal stores + */ +static __rte_always_inline void +rte_copy64_ts(uint8_t *dst, uint8_t *src) +{ + rte_copy32_ts(dst + 0 * 32, src + 0 * 32); + rte_copy32_ts(dst + 1 * 32, src + 1 * 32); +} + +/** + * Copy 128 bytes from one location to another, + * with temporal stores + */ +static __rte_always_inline void +rte_copy128_ts(uint8_t *dst, uint8_t *src) +{ + rte_copy32_ts(dst + 0 * 32, src + 0 * 32); + rte_copy32_ts(dst + 1 * 32, src + 1 * 32); + rte_copy32_ts(dst + 2 * 32, src + 2 * 32); + rte_copy32_ts(dst + 3 * 32, src + 3 * 32); +} + +/** + * Copy len bytes from one location to another, + * with temporal stores 16B aligned + */ +static __rte_always_inline void * +rte_memcpy_aligned_tstore16_generic(void *dst, void *src, int len) +{ + void *dest = dst; + + while (len >= 128) { + rte_copy128_ts((uint8_t *)dst, (uint8_t *)src); + dst = (uint8_t *)dst + 128; + src = (uint8_t *)src + 128; + len -= 128; + } + while (len >= 64) { + rte_copy64_ts((uint8_t *)dst, (uint8_t *)src); + dst = (uint8_t *)dst + 64; + src = (uint8_t *)src + 64; + len -= 64; + } + while (len >= 32) { + rte_copy32_ts((uint8_t *)dst, (uint8_t *)src); + dst = (uint8_t *)dst + 32; + src = (uint8_t *)src + 32; + len -= 32; + } + if (len >= 16) { + rte_copy16_ts((uint8_t *)dst, (uint8_t *)src); + dst = (uint8_t *)dst + 16; + src = (uint8_t *)src + 16; + len -= 16; + } + if (len >= 8) { + *(uint64_t *)dst = *(const uint64_t *)src; + dst = (uint8_t *)dst + 8; + src = (uint8_t *)src + 8; + len -= 8; + } + if (len >= 4) { + *(uint32_t *)dst = *(const uint32_t *)src; + dst = (uint8_t *)dst + 4; + src = (uint8_t *)src + 4; + len -= 4; + } + if (len != 0) { + dst = (uint8_t *)dst - (4 - len); + src = (uint8_t *)src - (4 - len); + *(uint32_t *)dst = *(const uint32_t *)src; + } + + return dest; +} + +static __rte_always_inline void * +rte_memcpy_aligned_tstore16(void *dst, void *src, int len) +{ + return rte_memcpy_aligned_tstore16_generic(dst, src, len); +} + +#endif /* RTE_MEMCPY_AMDEPYC */ + static __rte_always_inline void * rte_memcpy_generic(void *dst, const void *src, size_t n) { -- 2.25.1