From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2on0068.outbound.protection.outlook.com [207.46.100.68]) by dpdk.org (Postfix) with ESMTP id D38A38E9F for ; Mon, 23 Nov 2015 19:46:42 +0100 (CET) Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jerin.Jacob@caviumnetworks.com; Received: from localhost.caveonetworks.com (122.167.53.209) by BY2PR0701MB1975.namprd07.prod.outlook.com (10.163.155.21) with Microsoft SMTP Server (TLS) id 15.1.331.20; Mon, 23 Nov 2015 18:46:39 +0000 From: Jerin Jacob To: Date: Tue, 24 Nov 2015 00:15:36 +0530 Message-ID: <1448304338-22767-3-git-send-email-jerin.jacob@caviumnetworks.com> X-Mailer: git-send-email 2.1.0 In-Reply-To: <1448304338-22767-1-git-send-email-jerin.jacob@caviumnetworks.com> References: <1448304338-22767-1-git-send-email-jerin.jacob@caviumnetworks.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [122.167.53.209] X-ClientProxiedBy: MAXPR01CA0022.INDPRD01.PROD.OUTLOOK.COM (25.164.147.29) To BY2PR0701MB1975.namprd07.prod.outlook.com (25.163.155.21) X-Microsoft-Exchange-Diagnostics: 1; BY2PR0701MB1975; 2:1xtZiEh/Obhz6zYn5yclY8D3nxJw6ZuHdV1GdOuEEiu9VC69LwG0N4bdu1Hx1hXlboGrihDZRyn1djllejPbBKXE1Z5WzQQ7mUSLiksiBbEWUEwnDjRAX+5K+C74KOaAe53Z6l4kmDu1BdsyLJpdXg==; 3:aBEzUFsZw6zoFFg1bmXB0dFx2IjlvZnscyFpM4gmFyzTwObc2CQOzultr0peBwym28EtLyisTcRdMXc2PascG5zDjg1z4XeKpV6rw/U4jzGKLvWhRYEHv04uxllFUaQq; 25:vwGYP4Uyi5P05FeljfUP7va1QeLhVssdiOGpby2F/S7RIjaR1Gmshf7Keual8tZQN2dklM+R727w5AtmupAUzWjlVakmQZfwKyN8TOBgq0gD45r4s5U4YjwPdyUip/WeU3Zy9FUCZ2gEMf/Yf8b0SzAbXp+35wRHxYjwCr4tDu2a0Qhv1SjIOKOo1BF/R0MmuwJUStBNIKc3ioFA97Q4u3OsZ5v6IM6i4KIWj5vuOpYs/aAue+0q7YAInHfQ5yZFZ8DpBS51oXyFogq1kUQIEA== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BY2PR0701MB1975; X-Microsoft-Exchange-Diagnostics: 1; BY2PR0701MB1975; 20:PIAD8YJUifXi05mDKArvUcMn7kqyC4ISCkhSfA9BoTnZhNJ/cBi6lp532sK0TE6ayW32+ft0zn9HdSPP5nkmWjmui7R/aY1TY8587QY5RVdPRDVWS1Oo06y+uFUoNcmVtObvzMjMXfJQjQACESR4OFm5uU+giM6QCefNsPUvCZ5oruuD1wYOV7MYi+ZKr1nDlqsvdi6jyc3eVR6ZWDh78QhDuIMSzW3GSnEJFbmm6F9wGtU2YeI1tXiyqAgvzQe0D8E519hYARtJgtrEocFVg2HpJYYp+bY34Dfb39G2uHwtKmJyinqAWfCmOPYE25G+aL/vagMjMOUN7ma82q2SmQl/3nqSjnIbw1WzvrLg6BuvpuQBTCjPHwdUMKxfuiXzqYbDVt0dGjaQUWon6+/y/OcYW6CLu3x4q13ewOfYkwSkDPVcAXpDkj708FVq4rUocnBD89zxEvsMQRxvsJDeawMxkA2pRpSzry9NTMG26dmv9ek6MduMm8NoQIWNLGz7XLyvaHwEZBfQ79wkO3IuG188iUTe3BRqBxGYlbpyDuD8z7RLURrdlb2Iwh/Hrl/OvmNgpz9GMVeDg7S3K4Ps6D7Pn5G3jX3PIE8GgfOiF6M= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(236414709691187); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(5005006)(520078)(8121501046)(3002001)(10201501046); SRVR:BY2PR0701MB1975; BCL:0; PCL:0; RULEID:; SRVR:BY2PR0701MB1975; X-Microsoft-Exchange-Diagnostics: 1; BY2PR0701MB1975; 4:omW0+48eu+8NCgwujMYBYY3RDgUouAxE+lvKkqhTYTKUQbG1yvCL1FFyUe4vZRCgBwPGmc43Frmalkfz520Qp2mD3Q0cx4oA9+bU18npRRDFGDr9/NzqA1Il61kB6TjFXkzSOgazr6+M7aHdqH4ZbxkUR10jGJI9eAIEMYKsLjz7cE7C+8IKdk/h1cKWSpC+ZrrHHmf0BiWtrhDh9Gd/05x5oqnJujcYXbZKz0+a5v27+e2bgxZxLuS70WwzTEn+qpvkrClcgFR4zFivhN94Kb/1WRxu2oUUTjGtqUWl93vBXa5XQP0C/sznPKGy5Zc5Ia3XlU5qtpYs/V0aj1sV+38kTrc5JXE4+xO4JDyZd6Bgx5DaqUVC+tVx/142bONdQhYX/NXcDw9khh4CMqAz/R4dN7hCGtv+Ri3zlzTVLE4KEu9e0PtmnGQDKyWehHlv X-Forefront-PRVS: 07697999E6 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(6069001)(199003)(189002)(107886002)(92566002)(69596002)(48376002)(5008740100001)(50986999)(101416001)(19580405001)(47776003)(42186005)(19580395003)(76506005)(97736004)(586003)(4001430100002)(66066001)(6116002)(3846002)(106356001)(40100003)(5004730100002)(36756003)(50226001)(33646002)(87976001)(86362001)(5007970100001)(50466002)(81156007)(77096005)(2950100001)(122386002)(5001960100002)(5001920100001)(110136002)(189998001)(2351001)(76176999)(5003940100001)(105586002)(229853001)(53416004)(7099028); DIR:OUT; SFP:1101; SCL:1; SRVR:BY2PR0701MB1975; H:localhost.caveonetworks.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: caviumnetworks.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BY2PR0701MB1975; 23:xzXjAmRj97z4LLLtiMM0t0T4jMeILODomCCzyGF?= =?us-ascii?Q?TS0GletPjDbiqszV8G+++iFI4KqMofDCKzYtAnZZJLPu0VrXnDYR4Q7x1KVJ?= =?us-ascii?Q?lfIjzVILUCLrvaF9fxsUcuv2lLBU/mt0vD+2HcxALwkxJ+sYkUNHNx1nBFNO?= =?us-ascii?Q?x90kMjvQcCL6IyBQ3o9895lp3g+8bXca6wrJiZ3ilnF6WLUYLm1DMMtQHGcB?= =?us-ascii?Q?z57qdg7M/1FjLAPACueMvfLGxk3yvaEzbaC2mhzo5ZkEvIWVvlNW9w0txYq9?= =?us-ascii?Q?gNi5jWIUKWOpN/nTPQ0DynHItPuJ28J7Otx6xHsqQ89qQn2eTTM08B/+r3p7?= =?us-ascii?Q?vCd06FuEOP3SS0bALC4Nn921IbQi2HjBhnUom9dHvYiq/y/rJjMYvXVR8ts4?= =?us-ascii?Q?VAmfJi1o+1kqKJHMN4+TA8+ZfC/314ky70VTKDfJU5HRkRCsY5iLQlzT4CcC?= =?us-ascii?Q?iu5myuJsq6+gjiG+ELjUNPaIHPjIM/wycN/GR8kloRIs0cu7QVz0RrTGlDrc?= =?us-ascii?Q?h3KzlSL75yz8rHN0N0QAXL2Ir0Cyz/x1YMoEf/cKRQonoQovusqM52ZqO1DB?= =?us-ascii?Q?WfJr/QUqV06sW5qZN5yjITTfqrA0Xzk7/WEKN6DeZwIr/RHkacuxipe+Ywt+?= =?us-ascii?Q?xYjOhQQObap+yZnq9PXdTTS/dz+DLntqL26OmUgpCVVst3O/cMUcR3AVMEDu?= =?us-ascii?Q?Wn8LdaBoWXlMRnBtyPcxjABXGKEFaUmPnhAhJoHn8iL1G/C6Qy/wSXPA/VEh?= =?us-ascii?Q?qMLBSeSBg6k6g0JUttUXdHQXf81vVOWHqQW0FTVBjzKvPZy07Ce3ZCBKrm2g?= =?us-ascii?Q?nR0Ph43ViANl474ii02kiqI8ijAhFOrewUxnxE0LtqmOrC9W1dINfFeTwocX?= =?us-ascii?Q?yXRa0J2sCklrMfGczKzvKU9HExZkAS7RTwrYqOZgkVTSp8KDcjgCMlxoi9ec?= =?us-ascii?Q?guYX8WfAGopn+J8BFCFurVPsejMaofmhGjK/vSc/SbnAfMZPm+DGh70qDSIz?= =?us-ascii?Q?YjFRLsfTUW3qNO0y+b3I5VPrZE9ZpSWn0Y0AP8wXhVAK16MxPXW3N5O2Ncjs?= =?us-ascii?Q?H0Srk41v5MoLn/At1s7vW3ECQSMl3oXlcODtFMJnpkM+OG59Ckddyxb/4nRg?= =?us-ascii?Q?aSDDZKgwLiiZ0U5tnVtd+EHOB/O5atLBtNiQK/N/dKKotltEf3k74lF10sxp?= =?us-ascii?Q?tz3FfELhn/uCMl/Rt9JUYKYmzhpXfXenWuwSkZL9RPy7G94OcaDeR32Cyrby?= =?us-ascii?Q?Ph88VVieCF/HJ0HzdtUA=3D?= X-Microsoft-Exchange-Diagnostics: 1; BY2PR0701MB1975; 5:68+HiaMi2ROl2sgRjjdkhq6B6A8ieW23yMzP3s01TGFdvjSQvEQc1Ni6KgtdeIFy1L9E43e0Z2lUIZVnHv4hyHbHbmsLnHAkz5rGtcDrzNnD5iCKYpumggZ0reh1NbPmjYKMb9ly7/uJqXiLcJfD1Q==; 24:VtkfoUMTASvSWVlMIqVP2BE40PveTp9jyY20CwoXbtkpDUhvK2eFX251jzeU+ly4BYODkUPDAFXlYftMmj0CU1Zbc+TmRxq5M4BGy1Z1z9I= SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Nov 2015 18:46:39.1575 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR0701MB1975 Subject: [dpdk-dev] [PATCH 2/4] hash: implement rte_hash_crc_* based on armv8-a CRC32 instructions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Nov 2015 18:46:43 -0000 armv8-a has optional CRC32 extension, march=armv8-a+crc enables code generation for the ARMv8-A architecture together with the optional CRC32 extensions. added RTE_MACHINE_CPUFLAG_CRC32 to detect the availability of CRC32 extension in compile time. At run-time, The RTE_CPUFLAG_CRC32 can be used to find the availability. armv8-a+crc target support added in GCC 4.9, Used inline assembly and emulated __ARM_FEATURE_CRC32 to work with tool-chain < 4.9 Signed-off-by: Jerin Jacob --- app/test/test_hash.c | 7 ++ lib/librte_hash/Makefile | 3 + lib/librte_hash/rte_crc_arm64.h | 151 +++++++++++++++++++++++++++++++ lib/librte_hash/rte_hash_crc.h | 7 ++ mk/machine/armv8a/rte.vars.mk | 2 +- mk/machine/thunderx/rte.vars.mk | 2 +- mk/rte.cpuflags.mk | 4 + mk/toolchain/gcc/rte.toolchain-compat.mk | 6 +- 8 files changed, 179 insertions(+), 3 deletions(-) create mode 100644 lib/librte_hash/rte_crc_arm64.h diff --git a/app/test/test_hash.c b/app/test/test_hash.c index 4f2509d..2f3d884 100644 --- a/app/test/test_hash.c +++ b/app/test/test_hash.c @@ -217,6 +217,13 @@ test_crc32_hash_alg_equiv(void) printf("Failed checking CRC32_SW against CRC32_SSE42_x64\n"); break; } + + /* Check against 8-byte-operand ARM64 CRC32 if available */ + rte_hash_crc_set_alg(CRC32_ARM64); + if (hash_val != rte_hash_crc(data64, data_len, init_val)) { + printf("Failed checking CRC32_SW against CRC32_ARM64\n"); + break; + } } /* Resetting to best available algorithm */ diff --git a/lib/librte_hash/Makefile b/lib/librte_hash/Makefile index 7902c2b..bb1ea99 100644 --- a/lib/librte_hash/Makefile +++ b/lib/librte_hash/Makefile @@ -48,6 +48,9 @@ SRCS-$(CONFIG_RTE_LIBRTE_HASH) += rte_fbk_hash.c # install this header file SYMLINK-$(CONFIG_RTE_LIBRTE_HASH)-include := rte_hash.h SYMLINK-$(CONFIG_RTE_LIBRTE_HASH)-include += rte_hash_crc.h +ifeq ($(CONFIG_RTE_ARCH_ARM64),y) +SYMLINK-$(CONFIG_RTE_LIBRTE_HASH)-include += rte_crc_arm64.h +endif SYMLINK-$(CONFIG_RTE_LIBRTE_HASH)-include += rte_jhash.h SYMLINK-$(CONFIG_RTE_LIBRTE_HASH)-include += rte_thash.h SYMLINK-$(CONFIG_RTE_LIBRTE_HASH)-include += rte_fbk_hash.h diff --git a/lib/librte_hash/rte_crc_arm64.h b/lib/librte_hash/rte_crc_arm64.h new file mode 100644 index 0000000..02e26bc --- /dev/null +++ b/lib/librte_hash/rte_crc_arm64.h @@ -0,0 +1,151 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2015 Cavium networks. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Cavium networks nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef _RTE_CRC_ARM64_H_ +#define _RTE_CRC_ARM64_H_ + +/** + * @file + * + * RTE CRC arm64 Hash + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include +#include +#include +#include + +static inline uint32_t +crc32c_arm64_u32(uint32_t data, uint32_t init_val) +{ + asm(".arch armv8-a+crc"); + __asm__ volatile( + "crc32cw %w[crc], %w[crc], %w[value]" + : [crc] "+r" (init_val) + : [value] "r" (data)); + return init_val; +} + +static inline uint32_t +crc32c_arm64_u64(uint64_t data, uint32_t init_val) +{ + asm(".arch armv8-a+crc"); + __asm__ volatile( + "crc32cx %w[crc], %w[crc], %x[value]" + : [crc] "+r" (init_val) + : [value] "r" (data)); + return init_val; +} + +/** + * Allow or disallow use of arm64 SIMD instrinsics for CRC32 hash + * calculation. + * + * @param alg + * An OR of following flags: + * - (CRC32_SW) Don't use arm64 crc intrinsics + * - (CRC32_ARM64) Use ARMv8 CRC intrinsic if available + * + */ +static inline void +rte_hash_crc_set_alg(uint8_t alg) +{ + switch (alg) { + case CRC32_ARM64: + if (!rte_cpu_get_flag_enabled(RTE_CPUFLAG_CRC32)) + alg = CRC32_SW; + case CRC32_SW: + crc32_alg = alg; + default: + break; + } +} + +/* Setting the best available algorithm */ +static inline void __attribute__((constructor)) +rte_hash_crc_init_alg(void) +{ + rte_hash_crc_set_alg(CRC32_ARM64); +} + +/** + * Use single crc32 instruction to perform a hash on a 4 byte value. + * Fall back to software crc32 implementation in case arm64 crc intrinsics is + * not supported + * + * @param data + * Data to perform hash on. + * @param init_val + * Value to initialise hash generator. + * @return + * 32bit calculated hash value. + */ +static inline uint32_t +rte_hash_crc_4byte(uint32_t data, uint32_t init_val) +{ + if (likely(crc32_alg & CRC32_ARM64)) + return crc32c_arm64_u32(data, init_val); + + return crc32c_1word(data, init_val); +} + +/** + * Use single crc32 instruction to perform a hash on a 8 byte value. + * Fall back to software crc32 implementation in case arm64 crc intrinsics is + * not supported + * + * @param data + * Data to perform hash on. + * @param init_val + * Value to initialise hash generator. + * @return + * 32bit calculated hash value. + */ +static inline uint32_t +rte_hash_crc_8byte(uint64_t data, uint32_t init_val) +{ + if (likely(crc32_alg == CRC32_ARM64)) + return crc32c_arm64_u64(data, init_val); + + return crc32c_2words(data, init_val); +} + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_CRC_ARM64_H_ */ diff --git a/lib/librte_hash/rte_hash_crc.h b/lib/librte_hash/rte_hash_crc.h index 1f6f5bf..78a34b7 100644 --- a/lib/librte_hash/rte_hash_crc.h +++ b/lib/librte_hash/rte_hash_crc.h @@ -407,9 +407,14 @@ crc32c_sse42_u64(uint64_t data, uint64_t init_val) #define CRC32_SSE42 (1U << 1) #define CRC32_x64 (1U << 2) #define CRC32_SSE42_x64 (CRC32_x64|CRC32_SSE42) +#define CRC32_ARM64 (1U << 3) static uint8_t crc32_alg = CRC32_SW; +#if defined(RTE_ARCH_ARM64) +#include "rte_crc_arm64.h" +#else + /** * Allow or disallow use of SSE4.2 instrinsics for CRC32 hash * calculation. @@ -498,6 +503,8 @@ rte_hash_crc_8byte(uint64_t data, uint32_t init_val) return crc32c_2words(data, init_val); } +#endif + /** * Calculate CRC32 hash on user-supplied byte array. * diff --git a/mk/machine/armv8a/rte.vars.mk b/mk/machine/armv8a/rte.vars.mk index bdf8c6b..8c018a4 100644 --- a/mk/machine/armv8a/rte.vars.mk +++ b/mk/machine/armv8a/rte.vars.mk @@ -55,4 +55,4 @@ # CPU_LDFLAGS = # CPU_ASFLAGS = -MACHINE_CFLAGS += -march=armv8-a -DRTE_CACHE_LINE_SIZE=64 +MACHINE_CFLAGS += -march=armv8-a+crc -DRTE_CACHE_LINE_SIZE=64 diff --git a/mk/machine/thunderx/rte.vars.mk b/mk/machine/thunderx/rte.vars.mk index e49f9e1..0bb6b3d 100644 --- a/mk/machine/thunderx/rte.vars.mk +++ b/mk/machine/thunderx/rte.vars.mk @@ -55,4 +55,4 @@ CROSS ?= aarch64-thunderx-linux-gnu- # CPU_LDFLAGS = # CPU_ASFLAGS = -MACHINE_CFLAGS += -march=armv8-a -mcpu=thunderx -DRTE_CACHE_LINE_SIZE=128 +MACHINE_CFLAGS += -march=armv8-a+crc -mcpu=thunderx -DRTE_CACHE_LINE_SIZE=128 diff --git a/mk/rte.cpuflags.mk b/mk/rte.cpuflags.mk index bec7bdd..0a340a9 100644 --- a/mk/rte.cpuflags.mk +++ b/mk/rte.cpuflags.mk @@ -111,6 +111,10 @@ ifneq ($(filter $(AUTO_CPUFLAGS),__ARM_NEON_FP),) CPUFLAGS += NEON endif +ifneq ($(filter $(AUTO_CPUFLAGS),__ARM_FEATURE_CRC32),) +CPUFLAGS += CRC32 +endif + MACHINE_CFLAGS += $(addprefix -DRTE_MACHINE_CPUFLAG_,$(CPUFLAGS)) diff --git a/mk/toolchain/gcc/rte.toolchain-compat.mk b/mk/toolchain/gcc/rte.toolchain-compat.mk index 61bb5b7..e144216 100644 --- a/mk/toolchain/gcc/rte.toolchain-compat.mk +++ b/mk/toolchain/gcc/rte.toolchain-compat.mk @@ -54,7 +54,11 @@ else # GCC 4.5.x - added support for atom # GCC 4.6.x - added support for corei7, corei7-avx # GCC 4.7.x - added support for fsgsbase, rdrnd, f16c, core-avx-i, core-avx2 - +# GCC 4.9.x - added support for armv8-a+crc +# + ifeq ($(shell test $(GCC_VERSION) -le 49 && echo 1), 1) + MACHINE_CFLAGS := $(patsubst -march=armv8-a+crc,-march=armv8-a+crc -D__ARM_FEATURE_CRC32=1,$(MACHINE_CFLAGS)) + endif ifeq ($(shell test $(GCC_VERSION) -le 47 && echo 1), 1) MACHINE_CFLAGS := $(patsubst -march=core-avx-i,-march=corei7-avx,$(MACHINE_CFLAGS)) MACHINE_CFLAGS := $(patsubst -march=core-avx2,-march=core-avx2,$(MACHINE_CFLAGS)) -- 2.1.0