From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 059E3A04DC; Mon, 19 Oct 2020 15:56:25 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 098A9E2C0; Mon, 19 Oct 2020 15:51:14 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by dpdk.org (Postfix) with ESMTP id 70EFFE2C0 for ; Mon, 19 Oct 2020 15:51:12 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1603115470; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DrMMO9q0l0zsFcoui7pBdT0Q31AIND9SdkxTeb7WfOg=; b=MBapKjPdgHEBRb5YYGSksnbinTULRtXZuCKLXY+OfZbBeepHPehh0AdANKlBWT7VWPIekN etOalSeNOLfTb1jhngjOA8/QFQQUr3DTqgO2qYxHXQ2SyYSRQE27PJ5cMnmiXtQ1b6Ttot lU0PldJYcLsjbsSKLIsVyM/peynJ7MU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-175-0Ra3QuIcNoa1sSp5rtVKYw-1; Mon, 19 Oct 2020 09:51:04 -0400 X-MC-Unique: 0Ra3QuIcNoa1sSp5rtVKYw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 101B68B8D34; Mon, 19 Oct 2020 13:49:59 +0000 (UTC) Received: from dmarchan.remote.csb (unknown [10.40.195.116]) by smtp.corp.redhat.com (Postfix) with ESMTP id 75E2555775; Mon, 19 Oct 2020 13:49:56 +0000 (UTC) From: David Marchand To: dev@dpdk.org Cc: viktorin@rehivetech.com, ruifeng.wang@arm.com, jerinj@marvell.com, drc@linux.vnet.ibm.com, bruce.richardson@intel.com, konstantin.ananyev@intel.com, ciara.power@intel.com, Jasvinder Singh , Olivier Matz Date: Mon, 19 Oct 2020 15:48:56 +0200 Message-Id: <20201019134858.32507-17-david.marchand@redhat.com> In-Reply-To: <20201019134858.32507-1-david.marchand@redhat.com> References: <20201019134858.32507-1-david.marchand@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=david.marchand@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="US-ASCII" Subject: [dpdk-dev] [PATCH v10 16/18] net: check max SIMD bitwidth X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Ciara Power When choosing a vector path to take, an extra condition must be satisfied to ensure the max SIMD bitwidth allows for the CPU enabled path. The vector path was initially chosen in RTE_INIT, however this is no longer suitable as we cannot check the max SIMD bitwidth at that time. Default handlers are now chosen on initialisation, these default handlers are used the first time the crc calc is called, and they set the suitable handlers to be used going forward. Suggested-by: Jasvinder Singh Suggested-by: Olivier Matz Signed-off-by: Ciara Power Acked-by: Jasvinder Singh --- v7: Removed unnecessary log variable. v6: - Moved log variable and macro to c file instead of public header. - Added the max_simd_bitwidth condition check to the recently added handler helper functions. - Modified default handlers to follow the approach of the set alg function. v4: - Added default handlers to be set at RTE_INIT time, rather than choosing scalar handlers. - Modified logging. - Updated enum name. v3: - Moved choosing vector paths out of RTE_INIT. - Moved checking max_simd_bitwidth into the set_alg function. --- lib/librte_net/rte_net_crc.c | 117 +++++++++++++++++++++++++---------- 1 file changed, 86 insertions(+), 31 deletions(-) diff --git a/lib/librte_net/rte_net_crc.c b/lib/librte_net/rte_net_crc.c index 32a3665908..1fe58f7c75 100644 --- a/lib/librte_net/rte_net_crc.c +++ b/lib/librte_net/rte_net_crc.c @@ -9,6 +9,9 @@ #include #include #include +#include +#include +#include #include "net_crc.h" @@ -22,6 +25,12 @@ static uint32_t crc32_eth_lut[CRC_LUT_SIZE]; static uint32_t crc16_ccitt_lut[CRC_LUT_SIZE]; +static uint32_t +rte_crc16_ccitt_default_handler(const uint8_t *data, uint32_t data_len); + +static uint32_t +rte_crc32_eth_default_handler(const uint8_t *data, uint32_t data_len); + static uint32_t rte_crc16_ccitt_handler(const uint8_t *data, uint32_t data_len); @@ -31,7 +40,12 @@ rte_crc32_eth_handler(const uint8_t *data, uint32_t data_len); typedef uint32_t (*rte_net_crc_handler)(const uint8_t *data, uint32_t data_len); -static const rte_net_crc_handler *handlers; +static rte_net_crc_handler handlers_default[] = { + [RTE_NET_CRC16_CCITT] = rte_crc16_ccitt_default_handler, + [RTE_NET_CRC32_ETH] = rte_crc32_eth_default_handler, +}; + +static const rte_net_crc_handler *handlers = handlers_default; static const rte_net_crc_handler handlers_scalar[] = { [RTE_NET_CRC16_CCITT] = rte_crc16_ccitt_handler, @@ -56,6 +70,14 @@ static const rte_net_crc_handler handlers_neon[] = { }; #endif +static uint16_t max_simd_bitwidth; + +#define NET_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ## level, libnet_logtype, "%s(): " fmt "\n", \ + __func__, ## args) + +RTE_LOG_REGISTER(libnet_logtype, lib.net, INFO); + /* Scalar handling */ /** @@ -155,22 +177,21 @@ static const rte_net_crc_handler * avx512_vpclmulqdq_get_handlers(void) { #ifdef CC_X86_64_AVX512_VPCLMULQDQ_SUPPORT - if (AVX512_VPCLMULQDQ_CPU_SUPPORTED) + if (AVX512_VPCLMULQDQ_CPU_SUPPORTED && + max_simd_bitwidth >= RTE_VECT_SIMD_512) return handlers_avx512; #endif + NET_LOG(INFO, "Requirements not met, can't use AVX512\n"); return NULL; } -static uint8_t +static void avx512_vpclmulqdq_init(void) { #ifdef CC_X86_64_AVX512_VPCLMULQDQ_SUPPORT - if (AVX512_VPCLMULQDQ_CPU_SUPPORTED) { + if (AVX512_VPCLMULQDQ_CPU_SUPPORTED) rte_net_crc_avx512_init(); - return 1; - } #endif - return 0; } /* SSE4.2/PCLMULQDQ handling */ @@ -182,22 +203,21 @@ static const rte_net_crc_handler * sse42_pclmulqdq_get_handlers(void) { #ifdef CC_X86_64_SSE42_PCLMULQDQ_SUPPORT - if (SSE42_PCLMULQDQ_CPU_SUPPORTED) + if (SSE42_PCLMULQDQ_CPU_SUPPORTED && + max_simd_bitwidth >= RTE_VECT_SIMD_128) return handlers_sse42; #endif + NET_LOG(INFO, "Requirements not met, can't use SSE\n"); return NULL; } -static uint8_t +static void sse42_pclmulqdq_init(void) { #ifdef CC_X86_64_SSE42_PCLMULQDQ_SUPPORT - if (SSE42_PCLMULQDQ_CPU_SUPPORTED) { + if (SSE42_PCLMULQDQ_CPU_SUPPORTED) rte_net_crc_sse42_init(); - return 1; - } #endif - return 0; } /* NEON/PMULL handling */ @@ -209,22 +229,63 @@ static const rte_net_crc_handler * neon_pmull_get_handlers(void) { #ifdef CC_ARM64_NEON_PMULL_SUPPORT - if (NEON_PMULL_CPU_SUPPORTED) + if (NEON_PMULL_CPU_SUPPORTED && + max_simd_bitwidth >= RTE_VECT_SIMD_128) return handlers_neon; #endif + NET_LOG(INFO, "Requirements not met, can't use NEON\n"); return NULL; } -static uint8_t +static void neon_pmull_init(void) { #ifdef CC_ARM64_NEON_PMULL_SUPPORT - if (NEON_PMULL_CPU_SUPPORTED) { + if (NEON_PMULL_CPU_SUPPORTED) rte_net_crc_neon_init(); - return 1; - } #endif - return 0; +} + +/* Default handling */ + +static uint32_t +rte_crc16_ccitt_default_handler(const uint8_t *data, uint32_t data_len) +{ + handlers = NULL; + if (max_simd_bitwidth == 0) + max_simd_bitwidth = rte_vect_get_max_simd_bitwidth(); + + handlers = avx512_vpclmulqdq_get_handlers(); + if (handlers != NULL) + return handlers[RTE_NET_CRC16_CCITT](data, data_len); + handlers = sse42_pclmulqdq_get_handlers(); + if (handlers != NULL) + return handlers[RTE_NET_CRC16_CCITT](data, data_len); + handlers = neon_pmull_get_handlers(); + if (handlers != NULL) + return handlers[RTE_NET_CRC16_CCITT](data, data_len); + handlers = handlers_scalar; + return handlers[RTE_NET_CRC16_CCITT](data, data_len); +} + +static uint32_t +rte_crc32_eth_default_handler(const uint8_t *data, uint32_t data_len) +{ + handlers = NULL; + if (max_simd_bitwidth == 0) + max_simd_bitwidth = rte_vect_get_max_simd_bitwidth(); + + handlers = avx512_vpclmulqdq_get_handlers(); + if (handlers != NULL) + return handlers[RTE_NET_CRC32_ETH](data, data_len); + handlers = sse42_pclmulqdq_get_handlers(); + if (handlers != NULL) + return handlers[RTE_NET_CRC32_ETH](data, data_len); + handlers = neon_pmull_get_handlers(); + if (handlers != NULL) + return handlers[RTE_NET_CRC32_ETH](data, data_len); + handlers = handlers_scalar; + return handlers[RTE_NET_CRC32_ETH](data, data_len); } /* Public API */ @@ -233,6 +294,8 @@ void rte_net_crc_set_alg(enum rte_net_crc_alg alg) { handlers = NULL; + if (max_simd_bitwidth == 0) + max_simd_bitwidth = rte_vect_get_max_simd_bitwidth(); switch (alg) { case RTE_NET_CRC_AVX512: @@ -270,19 +333,11 @@ rte_net_crc_calc(const void *data, return ret; } -/* Select highest available crc algorithm as default one */ +/* Call initialisation helpers for all crc algorithm handlers */ RTE_INIT(rte_net_crc_init) { - enum rte_net_crc_alg alg = RTE_NET_CRC_SCALAR; - rte_net_crc_scalar_init(); - - if (sse42_pclmulqdq_init()) - alg = RTE_NET_CRC_SSE42; - if (avx512_vpclmulqdq_init()) - alg = RTE_NET_CRC_AVX512; - if (neon_pmull_init()) - alg = RTE_NET_CRC_NEON; - - rte_net_crc_set_alg(alg); + sse42_pclmulqdq_init(); + avx512_vpclmulqdq_init(); + neon_pmull_init(); } -- 2.23.0