From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id F0E9345C0B; Tue, 29 Oct 2024 14:44:06 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B87F242E7D; Tue, 29 Oct 2024 14:44:02 +0100 (CET) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2059.outbound.protection.outlook.com [40.107.223.59]) by mails.dpdk.org (Postfix) with ESMTP id 34FB442E8A for ; Tue, 29 Oct 2024 14:43:57 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gEAs/S3iw+WJC9Rcug4X1Ergxax+Zjw+jmSCg4Mm6wF0FMfK8Ia0M8ZIm3dxMDx8sg0CI1Yy99wTDnZ6xZ1wWVPDSdDx8sQkGMxnrgLQdLqcI5TEHZ6G1i1Vxtk2giEEu6lya1jxQGnryu6ufLhByWIo87TZuUIHCqVVs2pEodeK8npq6qbbaW4ka2jmFHdOZUrXsvcwixM6CZlHzk7WMIdGhv5OWqK6aulQzqpQTLYStB9HxbtK2FXIXQcnp5+b7H+lzXUc7LHpa7INadVNFBHp3n04UF5S1+m9D7dKm3yEPNGfDr5a6475xa0DkrODsCiKtsTZ37JRu2R906F/DQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=49O9zv8g7dvC9BXMRP5lRvb/M+RuAlYm6A58LttWoJw=; b=qepx04V0K0Cil4t8UgVPIZTq9yQX1gm7bcOebGibjnHGXHGxqCHATXdI3SZR/ZC3cTM1AllENUOkM1GUijDf2qlwCiaG//LaIT3GbofgzIP8rU8gGw8AatGX+KLxUvp4+bZ8cu2Rw471isaC1KUG3Q4E2VIyla0yZhWrhunEEMvfr0oqbUgYyDAzIpedXeqkdFNH6YxinPpw3TwqqdmN7+XiwmRSuWa0Whdc++8ixO5TH64tgDCQ+HCvdMX+GeWC6SNl3v5NB9AdJy8L7igud1ilyb6opGHTNIzpxEDX2lm0SkC75B81t8A7Es+52Lt0D7DV8NyKcTPcPrPodUnaHg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=monjalon.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=49O9zv8g7dvC9BXMRP5lRvb/M+RuAlYm6A58LttWoJw=; b=hCN5NLKtzu7mKoSFZ/nKFzEROO8BITzmf0fYM8A71t6iCBvP0JLWLjr5DIOXKaiUlIdas2Y3FqAQri+udvYik1hMoK1iuouRAWAvedovTHlbXy96FOtFq9TbI7nvSTy9pnGS2HZY/WpBeWylcHHc1m/j+w2JUq+dNXiEapgnpW8MiZ/CCmm0qqlVsmd8UVmNcBIA3nHE8i8gc7XZxXT6lCTr0m7VNNJr0a3MOj9camYoXf5vwrFwC0NXq13UvJuU3jTBIOhLbhUG1D2yA2bGOa/cA92u2E6vCJ9DyrYS1NmiRxZT+m+F43Jz1APSb1BVi2ozHG7819G0X8rxhAn6vg== Received: from BYAPR01CA0072.prod.exchangelabs.com (2603:10b6:a03:94::49) by CY5PR12MB6106.namprd12.prod.outlook.com (2603:10b6:930:29::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8093.25; Tue, 29 Oct 2024 13:43:51 +0000 Received: from SJ1PEPF0000231D.namprd03.prod.outlook.com (2603:10b6:a03:94:cafe::42) by BYAPR01CA0072.outlook.office365.com (2603:10b6:a03:94::49) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.34 via Frontend Transport; Tue, 29 Oct 2024 13:43:54 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by SJ1PEPF0000231D.mail.protection.outlook.com (10.167.242.234) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8114.16 via Frontend Transport; Tue, 29 Oct 2024 13:43:50 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 29 Oct 2024 06:43:24 -0700 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 29 Oct 2024 06:43:21 -0700 From: "Minggang Li(Gavin)" To: , , , , Dariusz Sosnowski , Bing Zhao , Suanming Mou CC: , , Rongwei Liu Subject: [PATCH V3 3/7] net/mlx5: add new devargs to control probe optimization Date: Tue, 29 Oct 2024 15:42:52 +0200 Message-ID: <20241029134256.874767-4-gavinl@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241029134256.874767-1-gavinl@nvidia.com> References: <20241028091822.860660-8-gavinl@nvidia.com> <20241029134256.874767-1-gavinl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF0000231D:EE_|CY5PR12MB6106:EE_ X-MS-Office365-Filtering-Correlation-Id: 133a8cd2-5a8c-4897-aadd-08dcf81fb8fb X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|82310400026|36860700013|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?sBkQ/JNHLewuy9VPldA01/70Hck8oNSTyBMviooGTIEuUqMqYE/N/5YFCQXn?= =?us-ascii?Q?ehrQaGvtzxr7NkeCfwu25qZiQCIqwi4z3kpPXbtzNWGK3INJI49/+G18/jMH?= =?us-ascii?Q?UFCnResAHRUxY+8QDkliP6ShBqE+cMLCIGd3x3jy+PSjt+kwrdTFJtZhDHTv?= =?us-ascii?Q?DuqnJI0aFRtBBRk4jx7vs3VKuZeacZFWPXkcZI4pTgsBRdN/IkWCVksZeX5r?= =?us-ascii?Q?xV72D/cNCqXme9xmvwjOEya0ggB/YsAzLCRuhgLIiPzzB8UXP3CoZlNVSWg0?= =?us-ascii?Q?j0CWWwk38FOILWMODi2q6QJAeW8zf4lBryFfrMsoMSB54paBjmqHjvxWmYSE?= =?us-ascii?Q?A5aiz4dQ6LBaNyF/aGS/uLP7qmfkuIKA7hn4oDtgJEfnv07lIvbVZx5ScAZM?= =?us-ascii?Q?oK6lBoxCd6Z6F73yTb/27+8mZh+pLuirsMZskn9ZXK+d7UmBftOJNroXSwhS?= =?us-ascii?Q?KPlnINGCNOhPy0tdpYZixbWFiACUazx1i1nUEt5I/d+i6O4aA++LEtjJpE3w?= =?us-ascii?Q?tsJ2Fy4uWEBe6N39V8aWNp8beGAhdpxJ3vgU9dvk8SAhhMkks0jmdfV+Uadw?= =?us-ascii?Q?ic31qwBRL1aA/zO+bC3DuzU1i/KkigJPmvbFB4LXB8aqxxJd7u3PU1JWpdrd?= =?us-ascii?Q?7jzGAlE5F6Hx+1qCxVJJJMrJ//JiXE+Cx8tquHmwtNE9nKERhYK3JKL99z/P?= =?us-ascii?Q?if3GI6ZVf45hiHhNjbIuyoXVDst24OIialFMwDhwhnxOltd5vjEvivGmpjlA?= =?us-ascii?Q?GhVUMJZMo8UfffLyTNfa8uQTovriPrAVRDp1WOaOkdDwjh/ZWvmA7a1BoSki?= =?us-ascii?Q?y/xsvdJmMI4f8us2wNs+ZBvhnUvC9cZ8dCTet6EDx2a8OUbpBm+NvYkoZAKe?= =?us-ascii?Q?CG7zWuaIgymZcAMtPoa28tYGwz6zseoUrEdWaMYaiGk5vkW8MB2AMrSXdu71?= =?us-ascii?Q?vHayjveKJBP810+hoAN2OzrT+kK8kvawk2VW+FYhKSSnfBjX7wkUonVgfuNB?= =?us-ascii?Q?UnoZgWQbGLDgtVI6+/cVkrUge5Jk8V4merQtJPk7WqMaFG7PAzrpQL0ioH9q?= =?us-ascii?Q?2GGEoqVSRU9DV0cztBw3R7hAdXP6GoGlZO9tchYAlH/DUKzkqKlGq8Fac5J0?= =?us-ascii?Q?2AhzvcX/+IN7qKRwGz2VIQLThJwSFOiKaEJhlLQZTOi7W+NuSktosj70uTDQ?= =?us-ascii?Q?BzTXX1WQpypRVgVT968Po+Q1Y4hbOEOky5jLp8UHJcucuno77/KDsa2KJfcv?= =?us-ascii?Q?Aof+i88xhrKyaQ2iXGUCrGJJE3t9/nR7bhtzdO4SMmJZtJnjqo57AYbgswmc?= =?us-ascii?Q?u4nf6LUUhogxG8s3xz1ZxNi8dOobLFldVFVu9fXQgXb0pRbpkqRAePFHW/9K?= =?us-ascii?Q?k2GFfJFexuRBXpO4RXbegAW4Fzybes7F2/t6/0ZtCFHxIi/dZA=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230040)(82310400026)(36860700013)(1800799024)(376014); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Oct 2024 13:43:50.9693 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 133a8cd2-5a8c-4897-aadd-08dcf81fb8fb X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF0000231D.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6106 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Rongwei Liu Add a new devarg probe_opt_en to control probe optimization in PMD. By default, the value is 0 and no behavior changed. Signed-off-by: Rongwei Liu Acked-by: Viacheslav Ovsiienko --- doc/guides/nics/mlx5.rst | 7 +++++++ drivers/common/mlx5/linux/mlx5_nl.c | 12 ++++++++---- drivers/common/mlx5/mlx5_common.c | 15 +++++++++++++++ drivers/common/mlx5/mlx5_common.h | 2 ++ drivers/net/mlx5/linux/mlx5_ethdev_os.c | 5 ++++- drivers/net/mlx5/linux/mlx5_os.c | 2 +- 6 files changed, 37 insertions(+), 6 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index f82e2d75de..981401a9f2 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -1436,6 +1436,13 @@ for an additional list of options shared with other mlx5 drivers. By default, the PMD will set this value to 1. +- ``probe_opt_en`` parameter [int] + + A non-zero value optimizes the probe process, especially for large scale. + PMD will hold the IB device information internally and reuse it. + + By default, the PMD will set this value to 0. + - ``lacp_by_user`` parameter [int] A nonzero value enables the control of LACP traffic by the user application. diff --git a/drivers/common/mlx5/linux/mlx5_nl.c b/drivers/common/mlx5/linux/mlx5_nl.c index e98073aafe..745e443f8f 100644 --- a/drivers/common/mlx5/linux/mlx5_nl.c +++ b/drivers/common/mlx5/linux/mlx5_nl.c @@ -1148,7 +1148,7 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex, struct mlx5_dev_info .flags = 0, }; - if (!strcmp(name, dev_info->ibname)) { + if (dev_info->probe_opt && !strcmp(name, dev_info->ibname)) { if (dev_info->port_info && pindex <= dev_info->port_num && dev_info->port_info[pindex].valid) { if (!dev_info->port_info[pindex].ifindex) @@ -1161,7 +1161,7 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex, struct mlx5_dev_info ret = mlx5_nl_port_info(nl, pindex, &data); - if (!strcmp(dev_info->ibname, name)) { + if (dev_info->probe_opt && !strcmp(dev_info->ibname, name)) { if ((!ret || ret == -ENODEV) && dev_info->port_info && pindex <= dev_info->port_num) { if (!ret) @@ -1201,7 +1201,8 @@ mlx5_nl_port_state(int nl, const char *name, uint32_t pindex, struct mlx5_dev_in .ibindex = UINT32_MAX, }; - if (dev_info && !strcmp(name, dev_info->ibname) && dev_info->port_num) + if (dev_info && dev_info->probe_opt && + !strcmp(name, dev_info->ibname) && dev_info->port_num) data.ibindex = dev_info->ibindex; if (mlx5_nl_port_info(nl, pindex, &data) < 0) return -rte_errno; @@ -1244,7 +1245,8 @@ mlx5_nl_portnum(int nl, const char *name, struct mlx5_dev_info *dev_info) uint32_t sn = MLX5_NL_SN_GENERATE; int ret, size; - if (dev_info->port_num && !strcmp(name, dev_info->ibname)) + if (dev_info->probe_opt && dev_info->port_num && + !strcmp(name, dev_info->ibname)) return dev_info->port_num; ret = mlx5_nl_send(nl, &req, sn); @@ -1263,6 +1265,8 @@ mlx5_nl_portnum(int nl, const char *name, struct mlx5_dev_info *dev_info) rte_errno = EINVAL; return 0; } + if (!dev_info->probe_opt) + return data.portnum; MLX5_ASSERT(!strlen(dev_info->ibname)); dev_info->port_num = data.portnum; dev_info->ibindex = data.ibindex; diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 0aaae91c31..9abae4a374 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -40,6 +40,9 @@ uint8_t haswell_broadwell_cpu; /* The default memory allocator used in PMD. */ #define MLX5_SYS_MEM_EN "sys_mem_en" +/* Probe optimization in PMD. */ +#define MLX5_PROBE_OPT "probe_opt_en" + /* * Device parameter to force doorbell register mapping * to non-cached region eliminating the extra write memory barrier. @@ -295,6 +298,8 @@ mlx5_common_args_check_handler(const char *key, const char *val, void *opaque) config->device_fd = tmp; } else if (strcmp(key, MLX5_PD_HANDLE) == 0) { config->pd_handle = tmp; + } else if (strcmp(key, MLX5_PROBE_OPT) == 0) { + config->probe_opt = !!tmp; } return 0; } @@ -324,6 +329,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist, MLX5_MR_MEMPOOL_REG_EN, MLX5_DEVICE_FD, MLX5_PD_HANDLE, + MLX5_PROBE_OPT, NULL, }; int ret = 0; @@ -332,6 +338,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist, config->mr_ext_memseg_en = 1; config->mr_mempool_reg_en = 1; config->sys_mem_en = 0; + config->probe_opt = 0; config->dbnc = MLX5_ARG_UNSET; config->device_fd = MLX5_ARG_UNSET; config->pd_handle = MLX5_ARG_UNSET; @@ -351,6 +358,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist, DRV_LOG(DEBUG, "mr_ext_memseg_en is %u.", config->mr_ext_memseg_en); DRV_LOG(DEBUG, "mr_mempool_reg_en is %u.", config->mr_mempool_reg_en); DRV_LOG(DEBUG, "sys_mem_en is %u.", config->sys_mem_en); + DRV_LOG(DEBUG, "probe_opt_en is %u.", config->probe_opt); DRV_LOG(DEBUG, "Send Queue doorbell mapping parameter is %d.", config->dbnc); return ret; @@ -791,6 +799,7 @@ mlx5_common_dev_create(struct rte_device *eal_dev, uint32_t classes, if (TAILQ_EMPTY(&devices_list)) rte_mem_event_callback_register("MLX5_MEM_EVENT_CB", mlx5_mr_mem_event_cb, NULL); + cdev->dev_info.probe_opt = cdev->config.probe_opt; exit: pthread_mutex_lock(&devices_list_lock); TAILQ_INSERT_HEAD(&devices_list, cdev, next); @@ -880,6 +889,12 @@ mlx5_common_probe_again_args_validate(struct mlx5_common_device *cdev, cdev->dev->name); goto error; } + if (cdev->config.probe_opt != config->probe_opt) { + DRV_LOG(ERR, "\"" MLX5_PROBE_OPT"\" " + "configuration mismatch for device %s.", + cdev->dev->name); + goto error; + } if (cdev->config.dbnc != config->dbnc) { DRV_LOG(ERR, "\"" MLX5_SQ_DB_NC "\" " "configuration mismatch for device %s.", diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 6cb40f54dd..f1b59d6f07 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -183,6 +183,7 @@ struct mlx5_dev_info { uint32_t port_num; uint32_t ibindex; char ibname[MLX5_FS_NAME_MAX]; + uint8_t probe_opt; struct mlx5_port_nl_info *port_info; }; @@ -525,6 +526,7 @@ struct mlx5_common_dev_config { int pd_handle; /* Protection Domain handle for importation. */ unsigned int devx:1; /* Whether devx interface is available or not. */ unsigned int sys_mem_en:1; /* The default memory allocator. */ + unsigned int probe_opt:1; /* Optimize probing . */ unsigned int mr_mempool_reg_en:1; /* Allow/prevent implicit mempool memory registration. */ unsigned int mr_ext_memseg_en:1; diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c index 08ac6dd939..88d3c57c6e 100644 --- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c +++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c @@ -691,6 +691,8 @@ mlx5_handle_port_info_update(struct mlx5_dev_info *dev_info, uint32_t if_index, if (dev_info->port_num <= 1 || dev_info->port_info == NULL) return; + DRV_LOG(DEBUG, "IB device %s ifindex %u received netlink event %u", + dev_info->ibname, if_index, msg_type); for (i = 1; i <= dev_info->port_num; i++) { if (!dev_info->port_info[i].valid) continue; @@ -734,7 +736,8 @@ mlx5_dev_interrupt_nl_cb(struct nlmsghdr *hdr, void *cb_arg) if (mlx5_nl_parse_link_status_update(hdr, &if_index) < 0) return; - mlx5_handle_port_info_update(&sh->cdev->dev_info, if_index, hdr->nlmsg_type); + if (sh->cdev->config.probe_opt && sh->cdev->dev_info.port_num > 1) + mlx5_handle_port_info_update(&sh->cdev->dev_info, if_index, hdr->nlmsg_type); for (i = 0; i < sh->max_port; i++) { struct mlx5_dev_shared_port *port = &sh->port[i]; diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index 8f6e584154..695936f634 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -2340,7 +2340,7 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, while (ret-- > 0) { struct rte_pci_addr pci_addr; - if (cdev->dev_info.port_num) { + if (cdev->config.probe_opt && cdev->dev_info.port_num) { if (strcmp(ibv_list[ret]->name, cdev->dev_info.ibname)) { DRV_LOG(INFO, "Unmatched caching device \"%s\" \"%s\"", cdev->dev_info.ibname, ibv_list[ret]->name); -- 2.34.1