From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8A8DD45F20; Mon, 23 Dec 2024 11:12:15 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A303140B9B; Mon, 23 Dec 2024 11:11:55 +0100 (CET) Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2049.outbound.protection.outlook.com [40.107.96.49]) by mails.dpdk.org (Postfix) with ESMTP id 1F59940A7D for ; Mon, 23 Dec 2024 11:11:51 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=yvba6W0MhgwjjkYSUsv24j5pQsSOk2Vpp6uXGUT4+Qzs/hYJNcwEifUAFKt6K8/BXOBoWtmELZErhGI/ElUQu3aA9WdLFxOHgIJEPCMUUfDZYRV3eVFyQPDhNNdpwCUeES2e2wGiften+wYWgfptxeDQp6vfylhGguJ49AvchvHo6LBzA6D4alIVTkkowqgyRfGSTHzHK0IB8nyGal63lfUvJf3xaESb2esUEZ5MuZIX643GzcQoB1ibG0fNUNcSxtHdGDJxLFDq0knm2DVA5bIz8NWnI4CSfgmPwC9RUlcrAfxsulA9NAe0VzC7P3WvkVWRIQ3EWXU+oVA7QTBreQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=leyIDJxYF/QvlzQ7qLCeY4UrU9wGqbnA1YNAbu/fGTA=; b=PmnfmQISwpv4SC5R3G+OxxKXD8G2DYr/7j17r1zbYhx0bjv/pg21rPOsE/kfQAvFy3mpOdBv51qqV5EtaYWcFASY1iu57eMnNln1gklvmyU/IMruo1uqbhBPWakZ7Bv7ajg0RR1908BgHJWqlozLNevcArvFa2q8jjEPXK5qbkhZCgeym8PVVlm2otNwyVvjI5AAwaxkuaMTROCafdmwVAlihRCs8XxP54U0H2pTNE3oQCL/NTAPbcV0+RmdX6dXaYz9T1FH23FnpA1ANPLH0LIeRihAOTG01qd9EIinR0Yg/FAstgL/Gl1BzbnGGMlNpmvz21zHt92NSfpoM8fnUw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=monjalon.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=leyIDJxYF/QvlzQ7qLCeY4UrU9wGqbnA1YNAbu/fGTA=; b=gs7hfg1NaVlNi6t/dIyPWpG6UEG+k2l1q7iNQPmksTypFGaPNcFaEUZIHG1hQhyhwsjLST2gfxSNGeoJTixmNb+p4OyHhH++hW70xc9miMDnSpsDiGCMR8F/RoZR3iQPIkoVAx3adh4IXdNM46xc5d64vCgpS4lQl3DwUa5Zt7gdRA/ZcoRCx44bVCpYzFR1JWLJTMx2UNFUldduSELJER4xWJh7x4c19s76zPst2miY5hZ0bjYhuvXyQFjAoydw5sXnzGPstUr4s75XMiO2emblWmx8sBabCgc3y21w4w8QKHmGLl9eStgn63aXPsB9cjqeJecI+7crkdfkytX3OQ== Received: from SJ0PR03CA0100.namprd03.prod.outlook.com (2603:10b6:a03:333::15) by PH7PR12MB6810.namprd12.prod.outlook.com (2603:10b6:510:1b4::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.20; Mon, 23 Dec 2024 10:11:40 +0000 Received: from SJ1PEPF0000231B.namprd03.prod.outlook.com (2603:10b6:a03:333:cafe::f) by SJ0PR03CA0100.outlook.office365.com (2603:10b6:a03:333::15) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8272.17 via Frontend Transport; Mon, 23 Dec 2024 10:11:40 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by SJ1PEPF0000231B.mail.protection.outlook.com (10.167.242.232) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8293.12 via Frontend Transport; Mon, 23 Dec 2024 10:11:40 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Mon, 23 Dec 2024 02:11:28 -0800 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Mon, 23 Dec 2024 02:11:25 -0800 From: "Minggang Li(Gavin)" To: , , , , Dariusz Sosnowski , Bing Zhao , Suanming Mou CC: , , Rongwei Liu Subject: [PATCH 3/7] net/mlx5: add new devargs to control probe optimization Date: Mon, 23 Dec 2024 12:10:57 +0200 Message-ID: <20241223101101.677449-4-gavinl@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241223101101.677449-1-gavinl@nvidia.com> References: <20241223101101.677449-1-gavinl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF0000231B:EE_|PH7PR12MB6810:EE_ X-MS-Office365-Filtering-Correlation-Id: 097daf49-e349-4a24-bb61-08dd233a3181 X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|376014|1800799024|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?al5QChlMOEYiuT9wKuUXIubWLmVpAHdQD63IjAriEcQ8JfMfPHXwCK73Zae6?= =?us-ascii?Q?G0UFOJHXCv5Gjqp8niqTWpC6lqx5SpBPYeeNd1OzJYSFc2+PaQJGs1Mu4VBD?= =?us-ascii?Q?7shpAhWj6oevEMWaArm0W7CvbOQIegHvNLIDD/5LGLhSIh+tprDCtOqzvSKx?= =?us-ascii?Q?VH7MBJnxmMnaFF65HCmKhl2I/6/Zlo8+26b2i1Oj1xk5KA24oT3aNDf8HAi/?= =?us-ascii?Q?QSQI6HRTgdyzQ8MH/bMIZe7GWdBlfQucx0F9artyi+isjzuGTrHndpvHzYzM?= =?us-ascii?Q?QXqXBubV4C8rh60/uReRT8XMvjGCdfi6/tzT05fdTqa1FaxjcpE8fK6bq4dK?= =?us-ascii?Q?9MOq3f3aw6COrT9rtNjcuWfyAtAl3xdgw2iN0/9TvpGyRkuuw5N5z/5R8muE?= =?us-ascii?Q?G5g8RheLcRY0QGe1pA56n4S0HzJk3xnuCdfQzJHBeovz1g1w0z0gwQRG8Eg2?= =?us-ascii?Q?Q9sgU8k7/VlWElBEjla7bAnhnSz+XvpUB+a/sSQW0lm77wMCjnzoGYpAOO9F?= =?us-ascii?Q?ctGgFNxuESGSbOSaAFlRsnRFlJuR9ytJsYEUMJsszyiTc63yT4B+8+mPDELn?= =?us-ascii?Q?ll+tYv9ALRTbnYUtCAjhBygKhvtZ8NHPjLWOX7CPAaaZnPSDQH4w1ngpkqUk?= =?us-ascii?Q?ngIjFYPkMa2n7L6wzuETyD9wVcB56pDQKxE5MuK9kPQ/owHH577HT7KXWp22?= =?us-ascii?Q?HYx4tdDrhId9f6zKzEhPCLngKriT40FcF0MRXq8SAojPsbe7dXIoKzx7AQy+?= =?us-ascii?Q?fK8xBNuE3KJ7qEQyqK7gyQGYCR2JUGEYSJf0c4qV2GHdvCS1IZt0pHDWBIFd?= =?us-ascii?Q?q6tNjr1hVTbD6gQIN3OOh0UxSGjRtoTksiFRUSXYCmkGaXo/85j3WcVEtkh6?= =?us-ascii?Q?pb0o9PFqm245pI4numlo1v69ZNieXM7Bc6b4CIdY91OV9B/fmMIc2DJ3W4c7?= =?us-ascii?Q?9BqVjXSLRXYL14m3+I5+2Kk2rwMNV+pjCcyHe/GNAb4k2xzce6uog40RH1H9?= =?us-ascii?Q?YJ9QA0Y6wKifGmAOZvzzN7+I931kQBQRIbWIUintuEweSOEUL2quuNlZFxcF?= =?us-ascii?Q?61WaVvnvCx5b9pMcMC0xsMHfSctoxBnPaOjY1Xa03rNRsZdkEB5vDClzfV66?= =?us-ascii?Q?KwVT8/BCdLIngtdcJgpikW5f9HFnMwqRXZO9+Exo+yaqqATMQmoGxEtAl899?= =?us-ascii?Q?mg/39jN2DGDkwRH9EylbFAkOKaA6OiaszXy55zFmRhV6pHYdpx8AtIFIPli4?= =?us-ascii?Q?STMYIJkdPk5A4mP52M8Cu6NeHLmjWFiwETMX21PUkFYSSJM5b1b/pUeZeiem?= =?us-ascii?Q?hDBGL8fCsdRuvMhHas8WXynLm1kfz+UjLijqnd8P/XttfSUYe6y/lOaXuMKY?= =?us-ascii?Q?UnqbwdQvAB73LrkKSa3fLCcyM6WC6ob1TWLS0rKTTllAzu4/9V5Sl1TmNm/H?= =?us-ascii?Q?IUoG8s55yILf1WzTt/Ez3VSkRoUnHl91obKUyGhPy882gtFWk5tEmA=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230040)(376014)(1800799024)(36860700013)(82310400026); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Dec 2024 10:11:40.1826 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 097daf49-e349-4a24-bb61-08dd233a3181 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF0000231B.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6810 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Rongwei Liu Add a new devarg probe_opt_en to control probe optimization in PMD. By default, the value is 0 and no behavior changed. Signed-off-by: Rongwei Liu Acked-by: Viacheslav Ovsiienko --- doc/guides/nics/mlx5.rst | 7 +++++++ drivers/common/mlx5/linux/mlx5_nl.c | 12 ++++++++---- drivers/common/mlx5/mlx5_common.c | 15 +++++++++++++++ drivers/common/mlx5/mlx5_common.h | 2 ++ drivers/net/mlx5/linux/mlx5_ethdev_os.c | 5 ++++- drivers/net/mlx5/linux/mlx5_os.c | 2 +- 6 files changed, 37 insertions(+), 6 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 30c40f767b..3bc8495e7a 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -1517,6 +1517,13 @@ for an additional list of options shared with other mlx5 drivers. By default, the PMD will set this value to 1. +- ``probe_opt_en`` parameter [int] + + A non-zero value optimizes the probe process, especially for large scale. + PMD will hold the IB device information internally and reuse it. + + By default, the PMD will set this value to 0. + - ``lacp_by_user`` parameter [int] A nonzero value enables the control of LACP traffic by the user application. diff --git a/drivers/common/mlx5/linux/mlx5_nl.c b/drivers/common/mlx5/linux/mlx5_nl.c index e98073aafe..745e443f8f 100644 --- a/drivers/common/mlx5/linux/mlx5_nl.c +++ b/drivers/common/mlx5/linux/mlx5_nl.c @@ -1148,7 +1148,7 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex, struct mlx5_dev_info .flags = 0, }; - if (!strcmp(name, dev_info->ibname)) { + if (dev_info->probe_opt && !strcmp(name, dev_info->ibname)) { if (dev_info->port_info && pindex <= dev_info->port_num && dev_info->port_info[pindex].valid) { if (!dev_info->port_info[pindex].ifindex) @@ -1161,7 +1161,7 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex, struct mlx5_dev_info ret = mlx5_nl_port_info(nl, pindex, &data); - if (!strcmp(dev_info->ibname, name)) { + if (dev_info->probe_opt && !strcmp(dev_info->ibname, name)) { if ((!ret || ret == -ENODEV) && dev_info->port_info && pindex <= dev_info->port_num) { if (!ret) @@ -1201,7 +1201,8 @@ mlx5_nl_port_state(int nl, const char *name, uint32_t pindex, struct mlx5_dev_in .ibindex = UINT32_MAX, }; - if (dev_info && !strcmp(name, dev_info->ibname) && dev_info->port_num) + if (dev_info && dev_info->probe_opt && + !strcmp(name, dev_info->ibname) && dev_info->port_num) data.ibindex = dev_info->ibindex; if (mlx5_nl_port_info(nl, pindex, &data) < 0) return -rte_errno; @@ -1244,7 +1245,8 @@ mlx5_nl_portnum(int nl, const char *name, struct mlx5_dev_info *dev_info) uint32_t sn = MLX5_NL_SN_GENERATE; int ret, size; - if (dev_info->port_num && !strcmp(name, dev_info->ibname)) + if (dev_info->probe_opt && dev_info->port_num && + !strcmp(name, dev_info->ibname)) return dev_info->port_num; ret = mlx5_nl_send(nl, &req, sn); @@ -1263,6 +1265,8 @@ mlx5_nl_portnum(int nl, const char *name, struct mlx5_dev_info *dev_info) rte_errno = EINVAL; return 0; } + if (!dev_info->probe_opt) + return data.portnum; MLX5_ASSERT(!strlen(dev_info->ibname)); dev_info->port_num = data.portnum; dev_info->ibindex = data.ibindex; diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 0aaae91c31..9abae4a374 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -40,6 +40,9 @@ uint8_t haswell_broadwell_cpu; /* The default memory allocator used in PMD. */ #define MLX5_SYS_MEM_EN "sys_mem_en" +/* Probe optimization in PMD. */ +#define MLX5_PROBE_OPT "probe_opt_en" + /* * Device parameter to force doorbell register mapping * to non-cached region eliminating the extra write memory barrier. @@ -295,6 +298,8 @@ mlx5_common_args_check_handler(const char *key, const char *val, void *opaque) config->device_fd = tmp; } else if (strcmp(key, MLX5_PD_HANDLE) == 0) { config->pd_handle = tmp; + } else if (strcmp(key, MLX5_PROBE_OPT) == 0) { + config->probe_opt = !!tmp; } return 0; } @@ -324,6 +329,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist, MLX5_MR_MEMPOOL_REG_EN, MLX5_DEVICE_FD, MLX5_PD_HANDLE, + MLX5_PROBE_OPT, NULL, }; int ret = 0; @@ -332,6 +338,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist, config->mr_ext_memseg_en = 1; config->mr_mempool_reg_en = 1; config->sys_mem_en = 0; + config->probe_opt = 0; config->dbnc = MLX5_ARG_UNSET; config->device_fd = MLX5_ARG_UNSET; config->pd_handle = MLX5_ARG_UNSET; @@ -351,6 +358,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist, DRV_LOG(DEBUG, "mr_ext_memseg_en is %u.", config->mr_ext_memseg_en); DRV_LOG(DEBUG, "mr_mempool_reg_en is %u.", config->mr_mempool_reg_en); DRV_LOG(DEBUG, "sys_mem_en is %u.", config->sys_mem_en); + DRV_LOG(DEBUG, "probe_opt_en is %u.", config->probe_opt); DRV_LOG(DEBUG, "Send Queue doorbell mapping parameter is %d.", config->dbnc); return ret; @@ -791,6 +799,7 @@ mlx5_common_dev_create(struct rte_device *eal_dev, uint32_t classes, if (TAILQ_EMPTY(&devices_list)) rte_mem_event_callback_register("MLX5_MEM_EVENT_CB", mlx5_mr_mem_event_cb, NULL); + cdev->dev_info.probe_opt = cdev->config.probe_opt; exit: pthread_mutex_lock(&devices_list_lock); TAILQ_INSERT_HEAD(&devices_list, cdev, next); @@ -880,6 +889,12 @@ mlx5_common_probe_again_args_validate(struct mlx5_common_device *cdev, cdev->dev->name); goto error; } + if (cdev->config.probe_opt != config->probe_opt) { + DRV_LOG(ERR, "\"" MLX5_PROBE_OPT"\" " + "configuration mismatch for device %s.", + cdev->dev->name); + goto error; + } if (cdev->config.dbnc != config->dbnc) { DRV_LOG(ERR, "\"" MLX5_SQ_DB_NC "\" " "configuration mismatch for device %s.", diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 6cb40f54dd..f1b59d6f07 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -183,6 +183,7 @@ struct mlx5_dev_info { uint32_t port_num; uint32_t ibindex; char ibname[MLX5_FS_NAME_MAX]; + uint8_t probe_opt; struct mlx5_port_nl_info *port_info; }; @@ -525,6 +526,7 @@ struct mlx5_common_dev_config { int pd_handle; /* Protection Domain handle for importation. */ unsigned int devx:1; /* Whether devx interface is available or not. */ unsigned int sys_mem_en:1; /* The default memory allocator. */ + unsigned int probe_opt:1; /* Optimize probing . */ unsigned int mr_mempool_reg_en:1; /* Allow/prevent implicit mempool memory registration. */ unsigned int mr_ext_memseg_en:1; diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c index 08ac6dd939..88d3c57c6e 100644 --- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c +++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c @@ -691,6 +691,8 @@ mlx5_handle_port_info_update(struct mlx5_dev_info *dev_info, uint32_t if_index, if (dev_info->port_num <= 1 || dev_info->port_info == NULL) return; + DRV_LOG(DEBUG, "IB device %s ifindex %u received netlink event %u", + dev_info->ibname, if_index, msg_type); for (i = 1; i <= dev_info->port_num; i++) { if (!dev_info->port_info[i].valid) continue; @@ -734,7 +736,8 @@ mlx5_dev_interrupt_nl_cb(struct nlmsghdr *hdr, void *cb_arg) if (mlx5_nl_parse_link_status_update(hdr, &if_index) < 0) return; - mlx5_handle_port_info_update(&sh->cdev->dev_info, if_index, hdr->nlmsg_type); + if (sh->cdev->config.probe_opt && sh->cdev->dev_info.port_num > 1) + mlx5_handle_port_info_update(&sh->cdev->dev_info, if_index, hdr->nlmsg_type); for (i = 0; i < sh->max_port; i++) { struct mlx5_dev_shared_port *port = &sh->port[i]; diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index 8f6e584154..695936f634 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -2340,7 +2340,7 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, while (ret-- > 0) { struct rte_pci_addr pci_addr; - if (cdev->dev_info.port_num) { + if (cdev->config.probe_opt && cdev->dev_info.port_num) { if (strcmp(ibv_list[ret]->name, cdev->dev_info.ibname)) { DRV_LOG(INFO, "Unmatched caching device \"%s\" \"%s\"", cdev->dev_info.ibname, ibv_list[ret]->name); -- 2.34.1