From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3F8F245B4C; Tue, 29 Oct 2024 15:32:39 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0939F42E87; Tue, 29 Oct 2024 15:32:31 +0100 (CET) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2064.outbound.protection.outlook.com [40.107.244.64]) by mails.dpdk.org (Postfix) with ESMTP id D28A542E98 for ; Tue, 29 Oct 2024 15:32:19 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=CAgml57BECvbuyLsDp31Fkm9UGmJALR/+4F5my114WFCwYQrQIR9PjAv6RhqSbFzHJi5vClivgEqpm2uzdVYmQpmRUDX+mceprngLxuOv+PhAjz5ivxVU2Ei6yxS4MzE9nYaxKQBaMIW7hQdWsrlPWLiluX4Rt9OzCweBEg48DvvfsjvuScMGDu1zbVEwcbgMPgF7dkDAzvD+8CAOOjJ/+94Rs9htnzRhg5RxZH7XsymS+EL+WHNLHUacChsmI9V7t0mEDjhCM/UEsUcjoGFx3SQJFXoeJOJJ4p2XoMC0aTaq+oJ5HTy5b5NqJxwTdrjtl+sRobwZZW3a+6FnHPuig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=49O9zv8g7dvC9BXMRP5lRvb/M+RuAlYm6A58LttWoJw=; b=NmmGSRmppo+iVHLCv2Q2LSLjiF5EMyx+0lRshcpSxxeMOC36/q4BrGBEv5SwIkm0/QkcU/oFa0JTdlHySGKcMTrroqhMX4mkTVAWZsu2GRB0VgP7GQBXcH0JyhjFaWFEBAZEpYDsBujHBIfX7O5u8KdBCY2S690WzG9S+LDfQqTh3EmiKNLmK2rIqE1sXM9YS/q2MpUq5H6JQR1HqqFPU1Js5jmYfSeNts/cH0ytJevqRuq7iT495amnt+EdBLj7HMW+ZJAAtePYA4Hnp/KQPFWXXmM8GqIXlNYSRLoS8CAwA5ZmCuGTFNQcdvNrdpC+ZF7g0A/6DexhvYou1lA1sw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=monjalon.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=49O9zv8g7dvC9BXMRP5lRvb/M+RuAlYm6A58LttWoJw=; b=QBW3Mjw6ar1UGriq11AYOnJYrnExKwWs4Fv8FEXm7vqglbUVgjJ1QJml7pZsA804dLtSJE8b9LZZH+KKtt3RWGyP+SkePDOFMmziTNot35DQaM3jYCtogRq6W3qNswWwj7gUKMiXXq1XTIH5M9gt+g0DjL96/ndsQfoolo4L/3u3fzmpotGjBAyHYTRbGneqI68VQ7RkEUyNXGemrW5B5wRXWr7FMH6rBhdhwOhsh7+0geyJ0kdGM6ZXiDyi7higMFFspLniJ6YlNyQ2f3fXN6m3BvkZhn8cw9L3vWQnNZ1kF++7bzBOCVbTzIXoRFtfO+fw4QGNqIrOM6cCZSZthg== Received: from CH2PR15CA0015.namprd15.prod.outlook.com (2603:10b6:610:51::25) by PH8PR12MB6987.namprd12.prod.outlook.com (2603:10b6:510:1be::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8093.24; Tue, 29 Oct 2024 14:32:14 +0000 Received: from CH2PEPF00000144.namprd02.prod.outlook.com (2603:10b6:610:51:cafe::ec) by CH2PR15CA0015.outlook.office365.com (2603:10b6:610:51::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8093.25 via Frontend Transport; Tue, 29 Oct 2024 14:32:12 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CH2PEPF00000144.mail.protection.outlook.com (10.167.244.101) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8114.16 via Frontend Transport; Tue, 29 Oct 2024 14:32:12 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 29 Oct 2024 07:31:43 -0700 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 29 Oct 2024 07:31:39 -0700 From: "Minggang Li(Gavin)" To: , , , , Dariusz Sosnowski , Bing Zhao , Suanming Mou CC: , , Rongwei Liu Subject: [PATCH V3 3/7] net/mlx5: add new devargs to control probe optimization Date: Tue, 29 Oct 2024 16:31:14 +0200 Message-ID: <20241029143118.875214-4-gavinl@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241029143118.875214-1-gavinl@nvidia.com> References: <20241029134256.874767-8-gavinl@nvidia.com> <20241029143118.875214-1-gavinl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000144:EE_|PH8PR12MB6987:EE_ X-MS-Office365-Filtering-Correlation-Id: 6d963e49-a84f-4a39-4e7a-08dcf8267a46 X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|82310400026|36860700013|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?BXmp9CfOhvZR7PELryL5vdFoPsK5MTwKrwqMa9zfyO9fNbhAcu3OPIYd1YRi?= =?us-ascii?Q?eATI/big3EGWtHle3JlkoLZZfsTJV1/3NxgI0BQDhnBlrhLUQ9MVB3kjZBBJ?= =?us-ascii?Q?72bu15wOXXxjp2T+UAS47kyiLQq73py9zH8AZTNBOJyv7wSits5xgxbtsOLE?= =?us-ascii?Q?PtX0JRVv6Uu3UqlU0ERNwrWQNrIxnpWsC22C6LoHcuiyTffJtxDn7BVU5qPf?= =?us-ascii?Q?y/2joPVIDdWKilnudVNMCjpGtEwc3qONmBUxDaVMk7N5KIcSjhGGVcTdlisv?= =?us-ascii?Q?q/4gzedOySiApgapsRIL8fiA+J8m1ogruZA2C43AdYgvB4CEBB0kqKKPJll6?= =?us-ascii?Q?OghrZWcwJ8eLkMtB5OOsiuZbvEared+U1Y/aThN9gUsmWCj8VyF9VVmsyHc3?= =?us-ascii?Q?da+fnQ/T3/AM445vObwKKCd1Oe369h9QsF3FNBJaeqtSLNK7T7heE5L1MrU+?= =?us-ascii?Q?0f0nYIWW7+W0BKznuXIRWhr8aQz39nwyo0pmxJq8MFLXDIKqAg8Q8XkM32oJ?= =?us-ascii?Q?qQDXbRW1ztvTr6vcAWfkn2L/OIU76IL7TIep2A0fDYbIkhBpPMeDlrg08tgi?= =?us-ascii?Q?uETdKoxyPV8SO9H90ww8Z8WGFsp78/cYlEaEvNQMflKs9HSivdxFe40ORd6o?= =?us-ascii?Q?lWfa+klYKm6kdIHcWg2JYBvBJfdWubWWvfza4h4MdwVhQvVUlsjBXpzyy7n/?= =?us-ascii?Q?OjmHVslTqELUxNQt2mWnCteecSynNfT8TtBIN/g2fowmWGDXvahmbYb/Gk3n?= =?us-ascii?Q?3XVhhXP3g+8mzkxtMHK/lwDwONd7xpMd1zBA/OatYqH9feMQK27X5nrfMQ5e?= =?us-ascii?Q?eql5E5hDB6Zc1ZeU6r1sKqf1isH9quxJigSQ70U3evFeIaFHdf9gI7hdIYV6?= =?us-ascii?Q?QGa42bj1TiG9Ugx5m8EF4ltdDyGQXV9wYF1jZMZBwpL/ZfjyWDDINO018hnV?= =?us-ascii?Q?DxYILDyOL5qNd3BAAZ+iAc7BQFFVhoKLf2xYOvd35m0LVh6ou/+ExwXUBIyK?= =?us-ascii?Q?hpEy7Nx/fBMPsdyzdmi8kDt11CCvwBNMwWuaSFgZwfd6bTi9+ukKOQrtEKhi?= =?us-ascii?Q?tfo6b+9SJc26ZvXXNKUB40c4JdT+NnwUHqKvIgg1pfl0nntMQwSMUETAu97t?= =?us-ascii?Q?918PWa+oB6zKvD7FIQnwnqpcvlLQ7bOzWEk4aDdDIWJpJQOUWHA3+rXEUZ9B?= =?us-ascii?Q?rdhQFXva7xbQMKJvL6GAOj1YtGoTMoGDGYp2Iqowx9N6qqSdVzK9YzYO2Bj7?= =?us-ascii?Q?sCSDLRttVq7e2fJPI+qQF05wOAk8NOMx8zb5XlGO6w7vcWFkCsdY+9kGOxkh?= =?us-ascii?Q?N6d9gWM2DMsguMr2Ib1YlFGsLyhjnVjmPMUNskfti+cQENhTJt7ukz9oFqTq?= =?us-ascii?Q?dVP+RICQWD45Ynn4gjvS3en5pw6eIrBK1KAZhRA0SZimMcB3XQ=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230040)(82310400026)(36860700013)(1800799024)(376014); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Oct 2024 14:32:12.2078 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6d963e49-a84f-4a39-4e7a-08dcf8267a46 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000144.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB6987 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Rongwei Liu Add a new devarg probe_opt_en to control probe optimization in PMD. By default, the value is 0 and no behavior changed. Signed-off-by: Rongwei Liu Acked-by: Viacheslav Ovsiienko --- doc/guides/nics/mlx5.rst | 7 +++++++ drivers/common/mlx5/linux/mlx5_nl.c | 12 ++++++++---- drivers/common/mlx5/mlx5_common.c | 15 +++++++++++++++ drivers/common/mlx5/mlx5_common.h | 2 ++ drivers/net/mlx5/linux/mlx5_ethdev_os.c | 5 ++++- drivers/net/mlx5/linux/mlx5_os.c | 2 +- 6 files changed, 37 insertions(+), 6 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index f82e2d75de..981401a9f2 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -1436,6 +1436,13 @@ for an additional list of options shared with other mlx5 drivers. By default, the PMD will set this value to 1. +- ``probe_opt_en`` parameter [int] + + A non-zero value optimizes the probe process, especially for large scale. + PMD will hold the IB device information internally and reuse it. + + By default, the PMD will set this value to 0. + - ``lacp_by_user`` parameter [int] A nonzero value enables the control of LACP traffic by the user application. diff --git a/drivers/common/mlx5/linux/mlx5_nl.c b/drivers/common/mlx5/linux/mlx5_nl.c index e98073aafe..745e443f8f 100644 --- a/drivers/common/mlx5/linux/mlx5_nl.c +++ b/drivers/common/mlx5/linux/mlx5_nl.c @@ -1148,7 +1148,7 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex, struct mlx5_dev_info .flags = 0, }; - if (!strcmp(name, dev_info->ibname)) { + if (dev_info->probe_opt && !strcmp(name, dev_info->ibname)) { if (dev_info->port_info && pindex <= dev_info->port_num && dev_info->port_info[pindex].valid) { if (!dev_info->port_info[pindex].ifindex) @@ -1161,7 +1161,7 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex, struct mlx5_dev_info ret = mlx5_nl_port_info(nl, pindex, &data); - if (!strcmp(dev_info->ibname, name)) { + if (dev_info->probe_opt && !strcmp(dev_info->ibname, name)) { if ((!ret || ret == -ENODEV) && dev_info->port_info && pindex <= dev_info->port_num) { if (!ret) @@ -1201,7 +1201,8 @@ mlx5_nl_port_state(int nl, const char *name, uint32_t pindex, struct mlx5_dev_in .ibindex = UINT32_MAX, }; - if (dev_info && !strcmp(name, dev_info->ibname) && dev_info->port_num) + if (dev_info && dev_info->probe_opt && + !strcmp(name, dev_info->ibname) && dev_info->port_num) data.ibindex = dev_info->ibindex; if (mlx5_nl_port_info(nl, pindex, &data) < 0) return -rte_errno; @@ -1244,7 +1245,8 @@ mlx5_nl_portnum(int nl, const char *name, struct mlx5_dev_info *dev_info) uint32_t sn = MLX5_NL_SN_GENERATE; int ret, size; - if (dev_info->port_num && !strcmp(name, dev_info->ibname)) + if (dev_info->probe_opt && dev_info->port_num && + !strcmp(name, dev_info->ibname)) return dev_info->port_num; ret = mlx5_nl_send(nl, &req, sn); @@ -1263,6 +1265,8 @@ mlx5_nl_portnum(int nl, const char *name, struct mlx5_dev_info *dev_info) rte_errno = EINVAL; return 0; } + if (!dev_info->probe_opt) + return data.portnum; MLX5_ASSERT(!strlen(dev_info->ibname)); dev_info->port_num = data.portnum; dev_info->ibindex = data.ibindex; diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 0aaae91c31..9abae4a374 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -40,6 +40,9 @@ uint8_t haswell_broadwell_cpu; /* The default memory allocator used in PMD. */ #define MLX5_SYS_MEM_EN "sys_mem_en" +/* Probe optimization in PMD. */ +#define MLX5_PROBE_OPT "probe_opt_en" + /* * Device parameter to force doorbell register mapping * to non-cached region eliminating the extra write memory barrier. @@ -295,6 +298,8 @@ mlx5_common_args_check_handler(const char *key, const char *val, void *opaque) config->device_fd = tmp; } else if (strcmp(key, MLX5_PD_HANDLE) == 0) { config->pd_handle = tmp; + } else if (strcmp(key, MLX5_PROBE_OPT) == 0) { + config->probe_opt = !!tmp; } return 0; } @@ -324,6 +329,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist, MLX5_MR_MEMPOOL_REG_EN, MLX5_DEVICE_FD, MLX5_PD_HANDLE, + MLX5_PROBE_OPT, NULL, }; int ret = 0; @@ -332,6 +338,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist, config->mr_ext_memseg_en = 1; config->mr_mempool_reg_en = 1; config->sys_mem_en = 0; + config->probe_opt = 0; config->dbnc = MLX5_ARG_UNSET; config->device_fd = MLX5_ARG_UNSET; config->pd_handle = MLX5_ARG_UNSET; @@ -351,6 +358,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist, DRV_LOG(DEBUG, "mr_ext_memseg_en is %u.", config->mr_ext_memseg_en); DRV_LOG(DEBUG, "mr_mempool_reg_en is %u.", config->mr_mempool_reg_en); DRV_LOG(DEBUG, "sys_mem_en is %u.", config->sys_mem_en); + DRV_LOG(DEBUG, "probe_opt_en is %u.", config->probe_opt); DRV_LOG(DEBUG, "Send Queue doorbell mapping parameter is %d.", config->dbnc); return ret; @@ -791,6 +799,7 @@ mlx5_common_dev_create(struct rte_device *eal_dev, uint32_t classes, if (TAILQ_EMPTY(&devices_list)) rte_mem_event_callback_register("MLX5_MEM_EVENT_CB", mlx5_mr_mem_event_cb, NULL); + cdev->dev_info.probe_opt = cdev->config.probe_opt; exit: pthread_mutex_lock(&devices_list_lock); TAILQ_INSERT_HEAD(&devices_list, cdev, next); @@ -880,6 +889,12 @@ mlx5_common_probe_again_args_validate(struct mlx5_common_device *cdev, cdev->dev->name); goto error; } + if (cdev->config.probe_opt != config->probe_opt) { + DRV_LOG(ERR, "\"" MLX5_PROBE_OPT"\" " + "configuration mismatch for device %s.", + cdev->dev->name); + goto error; + } if (cdev->config.dbnc != config->dbnc) { DRV_LOG(ERR, "\"" MLX5_SQ_DB_NC "\" " "configuration mismatch for device %s.", diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 6cb40f54dd..f1b59d6f07 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -183,6 +183,7 @@ struct mlx5_dev_info { uint32_t port_num; uint32_t ibindex; char ibname[MLX5_FS_NAME_MAX]; + uint8_t probe_opt; struct mlx5_port_nl_info *port_info; }; @@ -525,6 +526,7 @@ struct mlx5_common_dev_config { int pd_handle; /* Protection Domain handle for importation. */ unsigned int devx:1; /* Whether devx interface is available or not. */ unsigned int sys_mem_en:1; /* The default memory allocator. */ + unsigned int probe_opt:1; /* Optimize probing . */ unsigned int mr_mempool_reg_en:1; /* Allow/prevent implicit mempool memory registration. */ unsigned int mr_ext_memseg_en:1; diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c index 08ac6dd939..88d3c57c6e 100644 --- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c +++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c @@ -691,6 +691,8 @@ mlx5_handle_port_info_update(struct mlx5_dev_info *dev_info, uint32_t if_index, if (dev_info->port_num <= 1 || dev_info->port_info == NULL) return; + DRV_LOG(DEBUG, "IB device %s ifindex %u received netlink event %u", + dev_info->ibname, if_index, msg_type); for (i = 1; i <= dev_info->port_num; i++) { if (!dev_info->port_info[i].valid) continue; @@ -734,7 +736,8 @@ mlx5_dev_interrupt_nl_cb(struct nlmsghdr *hdr, void *cb_arg) if (mlx5_nl_parse_link_status_update(hdr, &if_index) < 0) return; - mlx5_handle_port_info_update(&sh->cdev->dev_info, if_index, hdr->nlmsg_type); + if (sh->cdev->config.probe_opt && sh->cdev->dev_info.port_num > 1) + mlx5_handle_port_info_update(&sh->cdev->dev_info, if_index, hdr->nlmsg_type); for (i = 0; i < sh->max_port; i++) { struct mlx5_dev_shared_port *port = &sh->port[i]; diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index 8f6e584154..695936f634 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -2340,7 +2340,7 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, while (ret-- > 0) { struct rte_pci_addr pci_addr; - if (cdev->dev_info.port_num) { + if (cdev->config.probe_opt && cdev->dev_info.port_num) { if (strcmp(ibv_list[ret]->name, cdev->dev_info.ibname)) { DRV_LOG(INFO, "Unmatched caching device \"%s\" \"%s\"", cdev->dev_info.ibname, ibv_list[ret]->name); -- 2.34.1