From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D69B6A04FF for ; Tue, 24 May 2022 12:11:16 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B225B400D6; Tue, 24 May 2022 12:11:16 +0200 (CEST) Received: from m12-17.163.com (m12-17.163.com [220.181.12.17]) by mails.dpdk.org (Postfix) with ESMTP id 4723D400D6 for ; Tue, 24 May 2022 12:11:13 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:Subject:Date:Message-Id; bh=S2Ne7rJqgx4nfJ5yC9 +wStUd/GIvT2MdzCkPXqIW9CY=; b=PoPdme8lMxacYmwfzp77l9BIAkEyAWz64t +DkXw4x9aXgbTjINKeZHglufOvMI3qa1i8qiwC/hjk9+gdE0PjqTtY6YzmLi3i1U YR03SuQjnXktWUea4k28SYaO21PB9U6KxYj6zaYXYME1KtFo3t3lqc+OjqICh/nB 07EBQwqpM= Received: from localhost.localdomain (unknown [119.3.119.18]) by smtp13 (Coremail) with SMTP id EcCowABXxaE4r4xiC3S5EA--.62630S2; Tue, 24 May 2022 18:11:11 +0800 (CST) From: Xing Zhang To: zhangxing_neurs@163.com Cc: stable@dpdk.org Subject: [PATCH] net/mlx5: add 128B padding of Rx completion entry Date: Tue, 24 May 2022 03:10:15 -0700 Message-Id: <1653387015-12320-1-git-send-email-zhangxing_neurs@163.com> X-Mailer: git-send-email 2.7.4 X-CM-TRANSID: EcCowABXxaE4r4xiC3S5EA--.62630S2 X-Coremail-Antispam: 1Uf129KBjvJXoWxuw4UCw45CF4DGr45ArWfuFg_yoWxAF4xpF 47CFyav345JF93Gwn7uw4Sgr45GwsayFW2kr9xA3yrZrnIyFyUAry8KFyavrWUWr98AF9I vF43WasruF45J3DanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x0zR_9NsUUUUU= X-Originating-IP: [119.3.119.18] X-CM-SenderInfo: x2kd0w50lqwsxqhx22i6rwjhhfrp/xS2BdB8Lk1gi2jhkggAAsN X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org From: "Xing Zhang zhangxing_neurs@163.com" When using Verbs flow engine("dv_flow_en"=0)and compile macro contain HAVE_IBV_MLX5_MOD_CQE_128B_PAD, vendor cap lag is setted to MLX5DV_CQ_INIT_ATTR_FLAGS_CQE_PAD.But if the device is not support the capability, this will cause CQ creation failure.So use padded CQE when system cache-line size is 128B and the device has the capability of CQE padding. The patch that CQE padding device argument should not be deleted. Fixes: 4a7f979af28e ("net/mlx5: remove CQE padding device argument") Cc: stable@dpdk.org Signed-off-by: Xing Zhang --- doc/guides/nics/mlx5.rst | 19 ++++++++++++++++++- drivers/net/mlx5/linux/mlx5_os.c | 5 +++++ drivers/net/mlx5/linux/mlx5_verbs.c | 2 +- drivers/net/mlx5/mlx5.c | 13 ++++++++++++- drivers/net/mlx5/mlx5.h | 2 ++ 5 files changed, 38 insertions(+), 3 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index a0b9284..bd11e5e 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -14,7 +14,6 @@ ConnectX-6 Lx**, **Mellanox BlueField** and **Mellanox BlueField-2** families of 10/25/40/50/100/200 Gb/s adapters as well as their virtual functions (VF) in SR-IOV context. - Design ------ @@ -580,6 +579,24 @@ for an additional list of options shared with other mlx5 drivers. - POWER9 and ARMv8 with ConnectX-4 Lx, ConnectX-5, ConnectX-6, ConnectX-6 Dx, ConnectX-6 Lx, BlueField and BlueField-2. +- ``rxq_cqe_pad_en`` parameter [int] + + A nonzero value enables 128B padding of CQE on RX side. The size of CQE + is aligned with the size of a cacheline of the core. If cacheline size is + 128B, the CQE size is configured to be 128B even though the device writes + only 64B data on the cacheline. This is to avoid unnecessary cache + invalidation by device's two consecutive writes on to one cacheline. + However in some architecture, it is more beneficial to update entire + cacheline with padding the rest 64B rather than striding because + read-modify-write could drop performance a lot. On the other hand, + writing extra data will consume more PCIe bandwidth and could also drop + the maximum throughput. It is recommended to empirically set this + parameter. Disabled by default. + + Supported on: + + - CPU having 128B cacheline with ConnectX-5 and BlueField. + - ``rxq_pkt_pad_en`` parameter [int] A nonzero value enables padding Rx packet to the size of cacheline on PCI diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index a821153..ef26d94 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -298,6 +298,11 @@ mlx5_os_capabilities_prepare(struct mlx5_dev_ctx_shared *sh) DRV_LOG(DEBUG, "Device supports Multi-Packet RQ."); } #endif +#ifdef HAVE_IBV_MLX5_MOD_CQE_128B_PAD + /* Whether device supports 128B Rx CQE padding. */ + sh->dev_cap.cqe_pad = RTE_CACHE_LINE_SIZE == 128 && + (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_CQE_128B_PAD); +#endif #ifdef HAVE_IBV_DEVICE_TUNNEL_SUPPORT if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS) { sh->dev_cap.tunnel_en = dv_attr.tunnel_offloads_caps & diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c index b6ba21c..4ced94a 100644 --- a/drivers/net/mlx5/linux/mlx5_verbs.c +++ b/drivers/net/mlx5/linux/mlx5_verbs.c @@ -200,7 +200,7 @@ mlx5_rxq_ibv_cq_create(struct mlx5_rxq_priv *rxq) priv->dev_data->port_id); } #ifdef HAVE_IBV_MLX5_MOD_CQE_128B_PAD - if (RTE_CACHE_LINE_SIZE == 128) { + if (priv->config.cqe_pad) { cq_attr.mlx5.comp_mask |= MLX5DV_CQ_INIT_ATTR_MASK_FLAGS; cq_attr.mlx5.flags |= MLX5DV_CQ_INIT_ATTR_FLAGS_CQE_PAD; } diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 72b1e35..51c92b2 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -46,6 +46,9 @@ /* Device parameter to enable RX completion queue compression. */ #define MLX5_RXQ_CQE_COMP_EN "rxq_cqe_comp_en" +/* Device parameter to enable RX completion entry padding to 128B. */ +#define MLX5_RXQ_CQE_PAD_EN "rxq_cqe_pad_en" + /* Device parameter to enable padding Rx packet to cacheline size. */ #define MLX5_RXQ_PKT_PAD_EN "rxq_pkt_pad_en" @@ -2176,7 +2179,9 @@ mlx5_port_args_check_handler(const char *key, const char *val, void *opaque) } config->cqe_comp = !!tmp; config->cqe_comp_fmt = tmp; - } else if (strcmp(MLX5_RXQ_PKT_PAD_EN, key) == 0) { + } else if (strcmp(MLX5_RXQ_CQE_PAD_EN, key) == 0) { + config->cqe_pad = !!tmp; + } else if (strcmp(MLX5_RXQ_PKT_PAD_EN, key) == 0) { config->hw_padding = !!tmp; } else if (strcmp(MLX5_RX_MPRQ_EN, key) == 0) { config->mprq.enabled = !!tmp; @@ -2352,6 +2357,12 @@ mlx5_port_args_config(struct mlx5_priv *priv, struct mlx5_kvargs_ctrl *mkvlist, } DRV_LOG(DEBUG, "Rx CQE compression is %ssupported.", config->cqe_comp ? "" : "not "); + if (config->cqe_pad && !dev_cap->cqe_pad) { + DRV_LOG(WARNING, "Rx CQE padding isn't supported."); + config->cqe_pad = 0; + } else if (config->cqe_pad) { + DRV_LOG(INFO, "Rx CQE padding is enabled."); + } if ((config->std_delay_drop || config->hp_delay_drop) && !dev_cap->rq_delay_drop_en) { config->std_delay_drop = 0; diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 23a28f6..fa41b44 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -140,6 +140,7 @@ struct mlx5_dev_cap { uint32_t txpp_en:1; /* Tx packet pacing is supported. */ uint32_t mpls_en:1; /* MPLS over GRE/UDP is supported. */ uint32_t cqe_comp:1; /* CQE compression is supported. */ + uint32_t cqe_pad:1; /* CQE padding is supported. */ uint32_t hw_csum:1; /* Checksum offload is supported. */ uint32_t hw_padding:1; /* End alignment padding is supported. */ uint32_t dest_tir:1; /* Whether advanced DR API is available. */ @@ -264,6 +265,7 @@ struct mlx5_port_config { unsigned int hw_vlan_insert:1; /* VLAN insertion in WQE is supported. */ unsigned int hw_padding:1; /* End alignment padding is supported. */ unsigned int cqe_comp:1; /* CQE compression is enabled. */ + unsigned int cqe_pad:1; /* CQE padding is enabled. */ unsigned int cqe_comp_fmt:3; /* CQE compression format. */ unsigned int rx_vec_en:1; /* Rx vector is enabled. */ unsigned int std_delay_drop:1; /* Enable standard Rxq delay drop. */ -- 2.7.4