From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8F2E2A0C4B; Fri, 5 Nov 2021 15:28:39 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E9A6C41223; Fri, 5 Nov 2021 15:28:36 +0100 (CET) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2059.outbound.protection.outlook.com [40.107.243.59]) by mails.dpdk.org (Postfix) with ESMTP id AA4F14113D for ; Fri, 5 Nov 2021 15:28:34 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mdSHs9MatAsxvxQizFZlDJ1BhzacWcgVq1kmLjpMkfA0BUyXZ7fX6fRDctfb2RZWlQ6i1hSvobLJayDF4mHpAXmV72hMvFYBijGWT4HDpMMG/EGYsj5qydRDJqqoLS/OeCB45a8/7Xk/69fpCreR9gtJTCAthQQqn/+wBsqqj2o7AvORa6priuHi10q1axLJ+j3+Jx2Yfgim2A3ARBRINAMnwuEVc7jARNW2bp33F6v69oTlsuic1nsUBn0zxmzZsk42s8MkcmiSIju3CLVWzCEjnGreevIsnlehyZA1EXXMdf7evZgWxh+Gtr8sXafM9XUSEb4X4PuSyhTtfRwzjA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=uom121oGCr19mLlJG5jvsWRSHzVFnX0fKv2/Ek160uM=; b=Tj7crsNpsUL7YJm2LTkWt+/ak+GVZjQc8hWAd+ikWEgyVEGGNaRQ6Zebb2lYGHnXz9A8iXVB5NDr15Ekmzfa/X4CGyjbgMeWeyTSy6+0yNpSKSoJHJ09xIoQlV30LxROrsMoamolKsgLOv4VbAcRu2xWNR75AsQ2LvzA59s3kd8zzmnXU2TN29QV78A30sXYG8a2Sr6d9mkBXLUXG7iN5WpF/E3YXd2Tw9hkHDXbjdFQNLHuMuIu9AbfdWJbncrlk3/nrIAytpItcoloGHdPpPOFdtaO3+SBbMPwtajb4ZOpVrHn03kq7XPXVGvPT0nsRy0d614GO8bzgx6WtGkmSg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=monjalon.net smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=uom121oGCr19mLlJG5jvsWRSHzVFnX0fKv2/Ek160uM=; b=juHTlcSQrw3P/7OcqsRZQMXH2MJofm9Tw+xudNxV2BkLm1FKS2op9ZD0QNaRXRa35jhfsNa0myakS5DCtrj1YZs1utHaQSJQxY7Z7cIhY3OIFN+K0SELUfX/Wg622NNdkSCdVZhoADnYaeXv0mTm0awmGsGYIZBWG0rOOJUqmll2oykMWWQyK2Cen8HXGft2hfVPfmRoX+NgfUlpN+fXbge4FstGFgvVLUdsEC2/ZSY/9V7i1EekZETxto7xw7dy+cv89LaHAi69rr+y51ayfRLR/sKSnd7eIM7+xxUgjooUe/t35bV13wcU6GB5VK/qaWI/zGGDIYpQIYBxu9xkLw== Received: from MW4PR03CA0040.namprd03.prod.outlook.com (2603:10b6:303:8e::15) by MN2PR12MB4376.namprd12.prod.outlook.com (2603:10b6:208:26c::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.11; Fri, 5 Nov 2021 14:28:32 +0000 Received: from CO1NAM11FT068.eop-nam11.prod.protection.outlook.com (2603:10b6:303:8e:cafe::ea) by MW4PR03CA0040.outlook.office365.com (2603:10b6:303:8e::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.11 via Frontend Transport; Fri, 5 Nov 2021 14:28:32 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; monjalon.net; dkim=none (message not signed) header.d=none;monjalon.net; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by CO1NAM11FT068.mail.protection.outlook.com (10.13.175.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4669.10 via Frontend Transport; Fri, 5 Nov 2021 14:28:32 +0000 Received: from nvidia.com (172.20.187.5) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Fri, 5 Nov 2021 14:28:30 +0000 From: Bing Zhao To: , CC: , , , Date: Fri, 5 Nov 2021 16:28:13 +0200 Message-ID: <20211105142814.183067-2-bingz@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211105142814.183067-1-bingz@nvidia.com> References: <20211104112644.17278-1-bingz@nvidia.com> <20211105142814.183067-1-bingz@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.187.5] X-ClientProxiedBy: HQMAIL101.nvidia.com (172.20.187.10) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 118a03b5-e7ec-4ee8-7e9d-08d9a0688b53 X-MS-TrafficTypeDiagnostic: MN2PR12MB4376: X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7691; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: V+b7SY0ipnnlowcIXtPC0NnlnGiwbUZhFiMFIE1QjJ2ZtBKUpZdZrQn4W7fruM00MNIujKXKGArfcXNym/WqEmJ8XYG477IJ/JpSszujaqn3bxKF6E1NkAem7dndjqkslNNxBU8ftJ8GVUf9s6LVsjjxt9bSV8SY5HRFnx7xsmjhyjUSMEzK+f2hWp79XoGlC57AC0mPCFJLOZCG/eebMchRwYtqXwPprZXoQrHLp2DEtjFzVXZj6qsp5AgyEM5p+sYR1bsonTaDM6hvfW32y5myb+bZWEsKjKlf88cysyo7P/bTQygVKsbc9oR8XndTcdgSkfXtRw96nwe9db0SXcT8jeLki5e0do7gv3XmndZkIThz4Xu5mwHE6jg3WlbLie+9bfDTPrxRikr+ixluudOeDRxq//5deo5KA2i0BL1HuxbHy/EgfDwhkkvB/RrK0GRO8ggb1nIykeZf+1uwNWnJIzwcs+svziINJvW02le44eqkoc4AD9CTR5TlE09xlF++4aKL4o6nMm2VEr6E9ollnZ7FTBJI8Lm6GnVsBw8IPzKRm5hHfPMEZSnTKvPSR3WiOGUjfIvfzMPVEx5s6K5HiosaMTO5kXj3Fz17XAjsHPQQXVUQv+bOrneAImTQhclO2fwsBpzhMMu3c0lcR/bBAY9RObflh3VwYviwPsHz81OOU+tidV2fnkikDn0sgIoSpiSMOHTYkt4o8mEaMqobYzSV4/3Pf1BoMbbV9bM= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(36860700001)(107886003)(7636003)(426003)(6666004)(356005)(86362001)(82310400003)(8676002)(316002)(8936002)(30864003)(4326008)(6286002)(70206006)(508600001)(1076003)(70586007)(2616005)(83380400001)(7696005)(6636002)(36756003)(336012)(26005)(186003)(47076005)(54906003)(36906005)(110136005)(5660300002)(2906002)(55016002)(16526019)(309714004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Nov 2021 14:28:32.2944 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 118a03b5-e7ec-4ee8-7e9d-08d9a0688b53 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT068.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4376 Subject: [dpdk-dev] [PATCH v6 1/2] net/mlx5: add support for Rx queue delay drop X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" For the Ethernet RQs, if there all receiving descriptors are exhausted, the packets being received will be dropped. This behavior prevents slow or malicious software entities at the host from affecting the network. While for hairpin cases, even if there is no software involved during the packet forwarding from Rx to Tx side, some hiccup in the hardware or back pressure from Tx side may still cause the descriptors to be exhausted. In certain scenarios it may be preferred to configure the device to avoid such packet drops, assuming the posting of descriptors will resume shortly. To support this, a new devarg "delay_drop" is introduced. By default, the delay drop is enabled for hairpin Rx queues and disabled for standard Rx queues. This value is used as a bit mask: - bit 0: enablement of standard Rx queue - bit 1: enablement of hairpin Rx queue And this attribute will be applied to all Rx queues of a device. The "rq_delay_drop" capability in the HCA_CAP is checked before creating any queue. If the hardware capabilities do not support this delay drop, all the Rx queues will still be created without this attribute, and the devarg setting will be ignored even if it is specified explicitly. A warning log is used to notify the application when this occurs. Signed-off-by: Bing Zhao Acked-by: Viacheslav Ovsiienko --- doc/guides/nics/mlx5.rst | 27 ++++++++++++++++++++++++++ doc/guides/rel_notes/release_21_11.rst | 1 + drivers/common/mlx5/mlx5_devx_cmds.c | 1 + drivers/common/mlx5/mlx5_devx_cmds.h | 1 + drivers/net/mlx5/linux/mlx5_os.c | 11 +++++++++++ drivers/net/mlx5/mlx5.c | 7 +++++++ drivers/net/mlx5/mlx5.h | 9 +++++++++ drivers/net/mlx5/mlx5_devx.c | 5 +++++ drivers/net/mlx5/mlx5_rx.h | 1 + 9 files changed, 63 insertions(+) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 824971d89a..82dda457c0 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -114,6 +114,7 @@ Features - Sub-Function representors. - Sub-Function. - Shared Rx queue. +- Rx queue delay drop. Limitations @@ -608,6 +609,32 @@ Driver options - POWER8 and ARMv8 with ConnectX-4 Lx, ConnectX-5, ConnectX-6, ConnectX-6 Dx, ConnectX-6 Lx, BlueField and BlueField-2. +- ``delay_drop`` parameter [int] + + Bitmask value for the Rx queue delay drop attribute. Bit 0 is used for the + standard Rx queue and bit 1 is used for the hairpin Rx queue. By default, the + delay drop is disabled for all Rx queues. It will be ignored if the port does + not support the attribute even if it is enabled explicitly. + + The packets being received will not be dropped immediately when the WQEs are + exhausted in a Rx queue with delay drop enabled. + + A timeout value is set in the driver to control the waiting time before + dropping a packet. Once the timer is expired, the delay drop will be + deactivated for all the Rx queues with this feature enable. To re-activeate + it, a rearming is needed and it is part of the kernel driver starting from + OFED 5.5. + + To enable / disable the delay drop rearming, the private flag ``dropless_rq`` + can be set and queried via ethtool: + + - ethtool --set-priv-flags dropless_rq on (/ off) + - ethtool --show-priv-flags + + The configuration flag is global per PF and can only be set on the PF, once + it is on, all the VFs', SFs' and representors' Rx queues will share the timer + and rearming. + - ``mprq_en`` parameter [int] A nonzero value enables configuring Multi-Packet Rx queues. Rx queue is diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index 92180bb4bd..9556aa8bd9 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -192,6 +192,7 @@ New Features * Added implicit mempool registration to avoid data path hiccups (opt-out). * Added NIC offloads for the PMD on Windows (TSO, VLAN strip, CRC keep). * Added socket direct mode bonding support. + * Added delay drop support for Rx queue. * **Updated Solarflare network PMD.** diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c index fca1470be7..49db07facc 100644 --- a/drivers/common/mlx5/mlx5_devx_cmds.c +++ b/drivers/common/mlx5/mlx5_devx_cmds.c @@ -965,6 +965,7 @@ mlx5_devx_cmd_query_hca_attr(void *ctx, attr->ct_offload = !!(MLX5_GET64(cmd_hca_cap, hcattr, general_obj_types) & MLX5_GENERAL_OBJ_TYPES_CAP_CONN_TRACK_OFFLOAD); + attr->rq_delay_drop = MLX5_GET(cmd_hca_cap, hcattr, rq_delay_drop); if (attr->qos.sup) { hcattr = mlx5_devx_get_hca_cap(ctx, in, out, &rc, MLX5_GET_HCA_CAP_OP_MOD_QOS_CAP | diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h index 344cd7bbf3..447f76f1f9 100644 --- a/drivers/common/mlx5/mlx5_devx_cmds.h +++ b/drivers/common/mlx5/mlx5_devx_cmds.h @@ -178,6 +178,7 @@ struct mlx5_hca_attr { uint32_t swp_csum:1; uint32_t swp_lso:1; uint32_t lro_max_msg_sz_mode:2; + uint32_t rq_delay_drop:1; uint32_t lro_timer_supported_periods[MLX5_LRO_NUM_SUPP_PERIODS]; uint16_t lro_min_mss_size; uint32_t flex_parser_protocols; diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index e0304b685e..de880ee4c9 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1508,6 +1508,15 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, goto error; #endif } + if (config->std_delay_drop || config->hp_delay_drop) { + if (!config->hca_attr.rq_delay_drop) { + config->std_delay_drop = 0; + config->hp_delay_drop = 0; + DRV_LOG(WARNING, + "dev_port-%u: Rxq delay drop is not supported", + priv->dev_port); + } + } if (sh->devx) { uint32_t reg[MLX5_ST_SZ_DW(register_mtutc)]; @@ -2077,6 +2086,8 @@ mlx5_os_config_default(struct mlx5_dev_config *config) config->decap_en = 1; config->log_hp_size = MLX5_ARG_UNSET; config->allow_duplicate_pattern = 1; + config->std_delay_drop = 0; + config->hp_delay_drop = 0; } /** diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 8614b8ffdd..9c8d1cc76f 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -183,6 +183,9 @@ /* Device parameter to configure implicit registration of mempool memory. */ #define MLX5_MR_MEMPOOL_REG_EN "mr_mempool_reg_en" +/* Device parameter to configure the delay drop when creating Rxqs. */ +#define MLX5_DELAY_DROP "delay_drop" + /* Shared memory between primary and secondary processes. */ struct mlx5_shared_data *mlx5_shared_data; @@ -2091,6 +2094,9 @@ mlx5_args_check(const char *key, const char *val, void *opaque) config->decap_en = !!tmp; } else if (strcmp(MLX5_ALLOW_DUPLICATE_PATTERN, key) == 0) { config->allow_duplicate_pattern = !!tmp; + } else if (strcmp(MLX5_DELAY_DROP, key) == 0) { + config->std_delay_drop = tmp & MLX5_DELAY_DROP_STANDARD; + config->hp_delay_drop = tmp & MLX5_DELAY_DROP_HAIRPIN; } else { DRV_LOG(WARNING, "%s: unknown parameter", key); rte_errno = EINVAL; @@ -2153,6 +2159,7 @@ mlx5_args(struct mlx5_dev_config *config, struct rte_devargs *devargs) MLX5_DECAP_EN, MLX5_ALLOW_DUPLICATE_PATTERN, MLX5_MR_MEMPOOL_REG_EN, + MLX5_DELAY_DROP, NULL, }; struct rte_kvargs *kvlist; diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 51f4578838..b2022f3300 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -99,6 +99,13 @@ enum mlx5_flow_type { MLX5_FLOW_TYPE_MAXI, }; +/* The mode of delay drop for Rx queues. */ +enum mlx5_delay_drop_mode { + MLX5_DELAY_DROP_NONE = 0, /* All disabled. */ + MLX5_DELAY_DROP_STANDARD = RTE_BIT32(0), /* Standard queues enable. */ + MLX5_DELAY_DROP_HAIRPIN = RTE_BIT32(1), /* Hairpin queues enable. */ +}; + /* Hlist and list callback context. */ struct mlx5_flow_cb_ctx { struct rte_eth_dev *dev; @@ -264,6 +271,8 @@ struct mlx5_dev_config { unsigned int dv_miss_info:1; /* restore packet after partial hw miss */ unsigned int allow_duplicate_pattern:1; /* Allow/Prevent the duplicate rules pattern. */ + unsigned int std_delay_drop:1; /* Enable standard Rxq delay drop. */ + unsigned int hp_delay_drop:1; /* Enable hairpin Rxq delay drop. */ struct { unsigned int enabled:1; /* Whether MPRQ is enabled. */ unsigned int stride_num_n; /* Number of strides. */ diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index a9f9f4af70..e46f79124d 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -277,6 +277,7 @@ mlx5_rxq_create_devx_rq_resources(struct mlx5_rxq_priv *rxq) MLX5_WQ_END_PAD_MODE_NONE; rq_attr.wq_attr.pd = cdev->pdn; rq_attr.counter_set_id = priv->counter_set_id; + rq_attr.delay_drop_en = rxq_data->delay_drop; rq_attr.user_index = rte_cpu_to_be_16(priv->dev_data->port_id); if (rxq_data->shared) /* Create RMP based RQ. */ rxq->devx_rq.rmp = &rxq_ctrl->obj->devx_rmp; @@ -439,6 +440,8 @@ mlx5_rxq_obj_hairpin_new(struct mlx5_rxq_priv *rxq) attr.wq_attr.log_hairpin_data_sz - MLX5_HAIRPIN_QUEUE_STRIDE; attr.counter_set_id = priv->counter_set_id; + rxq_ctrl->rxq.delay_drop = priv->config.hp_delay_drop; + attr.delay_drop_en = priv->config.hp_delay_drop; tmpl->rq = mlx5_devx_cmd_create_rq(priv->sh->cdev->ctx, &attr, rxq_ctrl->socket); if (!tmpl->rq) { @@ -496,6 +499,7 @@ mlx5_rxq_devx_obj_new(struct mlx5_rxq_priv *rxq) DRV_LOG(ERR, "Failed to create CQ."); goto error; } + rxq_data->delay_drop = priv->config.std_delay_drop; /* Create RQ using DevX API. */ ret = mlx5_rxq_create_devx_rq_resources(rxq); if (ret) { @@ -941,6 +945,7 @@ mlx5_rxq_devx_obj_drop_create(struct rte_eth_dev *dev) dev->data->port_id); goto error; } + rxq_ctrl->rxq.delay_drop = 0; /* Create RQ using DevX API. */ ret = mlx5_rxq_create_devx_rq_resources(rxq); if (ret != 0) { diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h index eda6eca8de..3b797e577a 100644 --- a/drivers/net/mlx5/mlx5_rx.h +++ b/drivers/net/mlx5/mlx5_rx.h @@ -97,6 +97,7 @@ struct mlx5_rxq_data { unsigned int dynf_meta:1; /* Dynamic metadata is configured. */ unsigned int mcqe_format:3; /* CQE compression format. */ unsigned int shared:1; /* Shared RXQ. */ + unsigned int delay_drop:1; /* Enable delay drop. */ volatile uint32_t *rq_db; volatile uint32_t *cq_db; uint16_t port_id; -- 2.27.0