From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2978545599; Fri, 5 Jul 2024 15:06:47 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6E10041109; Fri, 5 Jul 2024 15:06:46 +0200 (CEST) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2049.outbound.protection.outlook.com [40.107.223.49]) by mails.dpdk.org (Postfix) with ESMTP id D99B940FDE; Fri, 5 Jul 2024 15:06:43 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=N61VdbgebR+f3NWKMdTj6SdYb8OGbFuwNkZ3u/0bJJUt210YXgi+Z3oyHyeVrPDZf5Jj/+zINJ7vH16AI8hQQiJnvetGTrWpIzoPPxtEmFiYDQctaaQJRboRsXtcXiKpL/bpRmpBbkzeQ+OskP9Ekwrq30jnU7gaemStY07A/u7TR9GafY6Fy3uYSw+AmqNZgz8IuvrDa1rLV/9018/qLVFBORKblRTDtKYf15l1NsPtqkHvZ418YiXyIZVplRK2087OkxDPTSa4jCLpNeko6NzCC46uc8KPVNJKeHuqU14OlcXiS+AApSZw/RLb6PzI/Tb7RMppF7BHkxXL0uMFMQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=8ZzxIWCQlwlM7sueDeX3D5XL0Midf4amM9GrB6QyyFA=; b=BtDtXi5bwBixoiF2jGrtequpBIHG3J1wM+Cw3qfECxmmLOgtQX/14J5JGQcksn69WFS66Qjd45ITcpUtnxTWAo7e/z4xU7+olltMnQ3F70R9hAdUNU9ub8AjdibcBSr001qWa0J6gy0hPpKvmnQnq+5l/CEvDJ5exyfGMLTTn0ToH5Uy5pudQLJCVT7tyczZEH53KNiUyHRVHcAHQwf5M0M4TgoTuKIxHhUUrDdVEjhdqaCPbH4blvCI9NCH5d4Rdgwgls3WCO2tQ+jvPN06TxEivKqHq0GihGH3ynABTMFYNik5XMFmAA64z9NtGSxFj94PBkRLKxGyLiIKK28Sew== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8ZzxIWCQlwlM7sueDeX3D5XL0Midf4amM9GrB6QyyFA=; b=iBxcnc64Xw6aWwXT3pWzU9AXekhD6UsPhB91ysfgttyzxHQIJ+uajxZcb+tyXcFhD/Pt4vnb1J67AkVIpnFcNLZyk5ZF0IzmXwVKzup/XpsAiU2ISZzT28PnK/ALgGmxHXayg95uSlQ8JDSovMNkENbSbRoKXe3SFBNrz354EfJFP2RuedgqPEEP4wIC9U1Ec4tIqVzlfCA/JE8OMsCL+3MZM5CgYUhfrrh406M2IcBRYLYzRLSvrizAd3x2KONsaHgnIpFGEXVafvgMT/f07KINy+YvJ+gOVmv4U3YTVvRUWZ9+6RXO11Q6cjrwwlM0c25HmiUmg5jaEsF4idxbmw== Received: from SJ0PR03CA0097.namprd03.prod.outlook.com (2603:10b6:a03:333::12) by CY5PR12MB6322.namprd12.prod.outlook.com (2603:10b6:930:21::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7741.29; Fri, 5 Jul 2024 13:06:39 +0000 Received: from SJ1PEPF00002311.namprd03.prod.outlook.com (2603:10b6:a03:333:cafe::ff) by SJ0PR03CA0097.outlook.office365.com (2603:10b6:a03:333::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7741.29 via Frontend Transport; Fri, 5 Jul 2024 13:06:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by SJ1PEPF00002311.mail.protection.outlook.com (10.167.242.165) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7741.18 via Frontend Transport; Fri, 5 Jul 2024 13:06:39 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 5 Jul 2024 06:06:20 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 5 Jul 2024 06:06:17 -0700 From: Jiawei Wang To: , , Dariusz Sosnowski , Ori Kam , Suanming Mou , Matan Azrad , Alexander Kozyrev CC: , , Subject: [PATCH] net/mlx5: fix data access race condition for shared Rx queue Date: Fri, 5 Jul 2024 16:05:46 +0300 Message-ID: <20240705130546.1506-1-jiaweiw@nvidia.com> X-Mailer: git-send-email 2.18.1 MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF00002311:EE_|CY5PR12MB6322:EE_ X-MS-Office365-Filtering-Correlation-Id: 36df0577-e636-43f7-1ce7-08dc9cf34f02 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|36860700013|82310400026|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Ky4ny0191qVeA6N40vgC/p6SueYWsVcfdAJSa2cFMNmt4dwfHuaEYW+rjGBO?= =?us-ascii?Q?ZlcB9qwR349QzMLs1P9GXDJ1JnTANLDTT9osr4kfZPsZgRgkqTeS1AHKkkIX?= =?us-ascii?Q?zHKeSttk9FLYph8mSYWydL8iW8LRcZXF3r4rOAbEX14VRJeXx6ey/dfqFZvu?= =?us-ascii?Q?FUDSWexGQg6UmBovHxcS1k8W4D6SJvZXpTWqg+rgMhHBDJJYDZ1YI3PKlfJ+?= =?us-ascii?Q?G7DckZh9OOvBqPnO2uEA+xEqNPlj5xdR/gjziLpr3AaGANceCF0nsmbxrHHF?= =?us-ascii?Q?Vn9/yFlScyCvg+U3Oabbyy34SiEAySd5yBVvBYBi4hGec0H130vfIp6tWby6?= =?us-ascii?Q?+pJHA2zZ9tLVy2BSUZX9BoZ/qNFaQAUIgbGf8H3r/C6PZz7Po4e095JZlMnG?= =?us-ascii?Q?Msr32hHY1BLHZwBncI+0m+mn83ZTa8dkMFSs85m6m0jADhw6Jxpq8AzPIYks?= =?us-ascii?Q?QzTsp3EpfLgrwZ4B8pBSoVMR2OItvD9xO5VcqP2xL7jejN3OuJIhkQ09kSBk?= =?us-ascii?Q?KG2EoXapN2Gbd8DJYjTwJtileECEOnUS0c4kRk4WHc0JMyDxvIun0nr2GE+w?= =?us-ascii?Q?0teNXq6+F0O3ZhNitwD2HRq+Eae5t11I7SAE9MiexbUnN6Rj7+KhW9i9P/pX?= =?us-ascii?Q?RYO9c66E56Q7DJDnsQYFC43ENPsmgLqPMD4IoPxihq6h1JRrlAK5YGRNT1m4?= =?us-ascii?Q?u0iZ2wqxg1DYyEUTk22FcAKMq/TKmOE4DvDXJ2fWZv4EEGIV/40u5EvZT46L?= =?us-ascii?Q?WE0pr3uE3uJOqnVBnt9n/8Wamo5+7RkR+5iPYOi1YHgW+HARtmASVYKyFpVi?= =?us-ascii?Q?xzRlxcpkethuI3xBzaqPSCW8hjdMzS8v/l+4uXBBo5ZFeje2mIeKv4gK778o?= =?us-ascii?Q?fVQ0bN9kPT/8p+2e3BJ/TNmkiFTyIJ4U1WShCVdXQHlnv18nRJdP5UbSYJ84?= =?us-ascii?Q?06J8KdpfahOC/IaI1ParW+8R0Y7FZGNDIyhsk+IXUn/gLuvT8Agx0+JYtzhD?= =?us-ascii?Q?nOb8rtP6VbdtJ7rv92fvG4mQl4pqu4yy9sQzUXWEdVjBvFm83aDDMJjKOJSE?= =?us-ascii?Q?NkIQRZBzhMAjBFLFBuUasktrwtYucvQKkXRN8Hb64fK5LDO5MD8Lw94tNnuu?= =?us-ascii?Q?v9Qf1FfmLzB6S2uNeT1QlKnj8ScIWKVM4rWDwwlu1aH5x+aFjnMPrsUXVxaE?= =?us-ascii?Q?5YMYWW3/695VdHGjZtgFDcK40KLoi6AqV2J4vNHbPtJJCTfFi1KFpqPXo9Mu?= =?us-ascii?Q?Ueth4SAgRE/cAtvqdL5IHk2RkOg8wX/HoygKqZXEZp/gcXa7zn3Wc+UbXWO/?= =?us-ascii?Q?N5N2WWhnhvgc5gkah/WJAWgQViK30nLbcYFrakJKxt+jPz63/3D7lSoeeWyv?= =?us-ascii?Q?L9XcAbiAASi/3yQFZhoEd1Z2Pmy6KTDz4PpOVoh5vz32UEDJ8Kvt7Aa6KvOX?= =?us-ascii?Q?z3/KtlxO8YfEvuA8/+7cESHotgKoyNtW?= X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230040)(36860700013)(82310400026)(1800799024)(376014); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jul 2024 13:06:39.5979 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 36df0577-e636-43f7-1ce7-08dc9cf34f02 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF00002311.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6322 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The rxq_data resources were shared for shared Rx queue with the same group and queue ID. The cq_ci:24 of rxq_data was unalignment with other fields in the one 32-bit data, like the dynf_meta and delay_drop. 32bit: xxxx xxxI IIII IIII IIII IIII IIII IIIx ^ .... .... .... .... ...^ | cq_ci | The issue is that while the control thread updates the dynf_meta:1 or delay_drop:1 value during port start, another data thread updates the cq_ci at the same time, it causes the bytes race condition with different thread, and cq_ci value may be overwritten and updated the abnormal value into HW CQ DB. This patch separates the cq_ci from the configuration data spaces, and adds checking for delay_drop and dynf_meta if shared Rx queue if started. Fixes: 02a6195cbe ("net/mlx5: support enhanced CQE compression in Rx burst") Cc: stable@dpdk.org Signed-off-by: Jiawei Wang Acked-by: Bing Zhao Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5_devx.c | 3 ++- drivers/net/mlx5/mlx5_flow.c | 24 +++++++++++++----------- drivers/net/mlx5/mlx5_rx.h | 4 ++-- 3 files changed, 17 insertions(+), 14 deletions(-) diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index 7db271acb4..8ebe784000 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -684,7 +684,8 @@ mlx5_rxq_devx_obj_new(struct mlx5_rxq_priv *rxq) DRV_LOG(ERR, "Failed to create CQ."); goto error; } - rxq_data->delay_drop = priv->config.std_delay_drop; + if (!rxq_data->shared || !rxq_ctrl->started) + rxq_data->delay_drop = priv->config.std_delay_drop; /* Create RQ using DevX API. */ ret = mlx5_rxq_create_devx_rq_resources(rxq); if (ret) { diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c index 5b3f2b9119..72fb3a55ba 100644 --- a/drivers/net/mlx5/mlx5_flow.c +++ b/drivers/net/mlx5/mlx5_flow.c @@ -1853,18 +1853,20 @@ mlx5_flow_rxq_dynf_set(struct rte_eth_dev *dev) if (rxq == NULL || rxq->ctrl == NULL) continue; data = &rxq->ctrl->rxq; - if (!rte_flow_dynf_metadata_avail()) { - data->dynf_meta = 0; - data->flow_meta_mask = 0; - data->flow_meta_offset = -1; - data->flow_meta_port_mask = 0; - } else { - data->dynf_meta = 1; - data->flow_meta_mask = rte_flow_dynf_metadata_mask; - data->flow_meta_offset = rte_flow_dynf_metadata_offs; - data->flow_meta_port_mask = priv->sh->dv_meta_mask; + if (!data->shared || !rxq->ctrl->started) { + if (!rte_flow_dynf_metadata_avail()) { + data->dynf_meta = 0; + data->flow_meta_mask = 0; + data->flow_meta_offset = -1; + data->flow_meta_port_mask = 0; + } else { + data->dynf_meta = 1; + data->flow_meta_mask = rte_flow_dynf_metadata_mask; + data->flow_meta_offset = rte_flow_dynf_metadata_offs; + data->flow_meta_port_mask = priv->sh->dv_meta_mask; + } + data->mark_flag = mark_flag; } - data->mark_flag = mark_flag; } } diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h index 1485556d89..7d144921ab 100644 --- a/drivers/net/mlx5/mlx5_rx.h +++ b/drivers/net/mlx5/mlx5_rx.h @@ -101,14 +101,14 @@ struct __rte_cache_aligned mlx5_rxq_data { unsigned int shared:1; /* Shared RXQ. */ unsigned int delay_drop:1; /* Enable delay drop. */ unsigned int cqe_comp_layout:1; /* CQE Compression Layout*/ - unsigned int cq_ci:24; + uint16_t port_id; volatile uint32_t *rq_db; volatile uint32_t *cq_db; - uint16_t port_id; uint32_t elts_ci; uint32_t rq_ci; uint16_t consumed_strd; /* Number of consumed strides in WQE. */ uint32_t rq_pi; + uint32_t cq_ci:24; uint16_t rq_repl_thresh; /* Threshold for buffer replenishment. */ uint32_t byte_mask; union { -- 2.18.1