From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2649945B50; Wed, 16 Oct 2024 10:39:14 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1B9F440663; Wed, 16 Oct 2024 10:39:03 +0200 (CEST) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2048.outbound.protection.outlook.com [40.107.243.48]) by mails.dpdk.org (Postfix) with ESMTP id 3964140615 for ; Wed, 16 Oct 2024 10:38:59 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=MHNhbVSzr9DiyGPWwB81LeFjAVz9+Idd9/tV0izesLwR+Wd14HNu5hYQwoHho8vfglpvZYOXRoPtL4y4KyVZ9x+jQJpBKeXYjZABd2bwDcHp8++F2BKC/DNRE4DE+KICi1i2at9gZZlvPLc0Ffax873rqRZYXNXYPb7ydN9K2NAQs678/qO+AR0uafAiekNbq8et33gL6nGheyRGmxfKNBkDf98q3SKMWLBplAfyzCLUWkJDP4M1hDJLvHo7jndupxozm8OVBGs1gb/cWFiSPLHNtSKFEvfCn5h3zGs7kVYoRRjz6JsxNSzw9AlCbJrqkf2mcZlvHonfLmbui3P7Iw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LQXO+Ht9n27CACo3V4qPJzy3iTr7M9qU2rNQnAnN8J4=; b=QCe9PtKzHa4HvwowOepWQDE587JCpG7z7e3/vPybIDttb4Gg6IZrbDd5pO8w/tPgt8qfknn0oQkuBRLn1eofyJutH3oJS63ZXmb2QamUv4Py6omp+3Z8bVE1/Pj2RD0mTa1ZlOgR4uTq+Oi/cS1hhRpAnIZQZTrs+sKXPBWEXdgxaCX+QFROri2PkSxzB2diUmaPSq60/3ufbFDu43tV8tqvr9M+bD/5v3bnntYcSwPHURplofQcaeMEXqWMvU7WAsU00336Azt+Q4s6l3i9dZSGEeEeexj5aEjRBYmNXYblyLsBiBpYcpbyWN6eh0ga5a8I+AIxTlqrxXWf7nR2XA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=monjalon.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LQXO+Ht9n27CACo3V4qPJzy3iTr7M9qU2rNQnAnN8J4=; b=GQ3ReCC2ptXWCNyKfH+DQ1uAlZM1vTKfN7TcE1oSNXHXrYqNssRkQ17yWjWcG+MfCtg5LHETY62YElOhbsb4M1Vrcx1eh6pgsDb1fPnkRvX0lDlyTqXHGGTFPMIPMWgpvVJHhoIYFv5utyoCbhpYQJ0IjSHH4pdgZwVhKEWpWxYtoLHwtbTzaeD+cUekw5NBOJTuhuD3+z4Fn37/v5IGlR7rllUZdWHSCjBCiUpiQwCwDwwcPUPcfc656ahtq1OtMnl9gxh/qjnTOTdQBA1+whohlS+NU1rg6vZF7sG7ODg8ODi2RB/GbODmzRcYoLQv86r3u2Uk2InQElpNefuKgA== Received: from MW4PR03CA0138.namprd03.prod.outlook.com (2603:10b6:303:8c::23) by MW4PR12MB6756.namprd12.prod.outlook.com (2603:10b6:303:1e9::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.25; Wed, 16 Oct 2024 08:38:55 +0000 Received: from MWH0EPF000989E6.namprd02.prod.outlook.com (2603:10b6:303:8c:cafe::29) by MW4PR03CA0138.outlook.office365.com (2603:10b6:303:8c::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.27 via Frontend Transport; Wed, 16 Oct 2024 08:38:55 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MWH0EPF000989E6.mail.protection.outlook.com (10.167.241.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8069.17 via Frontend Transport; Wed, 16 Oct 2024 08:38:55 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Wed, 16 Oct 2024 01:38:46 -0700 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Wed, 16 Oct 2024 01:38:43 -0700 From: "Minggang Li(Gavin)" To: , , , , Dariusz Sosnowski , Bing Zhao , Suanming Mou CC: , , Rongwei Liu Subject: [PATCH V1 3/7] net/mlx5: add new devargs to control probe optimization Date: Wed, 16 Oct 2024 11:38:14 +0300 Message-ID: <20241016083818.662020-4-gavinl@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241016083818.662020-1-gavinl@nvidia.com> References: <20241016083818.662020-1-gavinl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MWH0EPF000989E6:EE_|MW4PR12MB6756:EE_ X-MS-Office365-Filtering-Correlation-Id: 03fac9a4-c595-45da-d7ab-08dcedbdf894 X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|376014|1800799024|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?t7HAG9R7xtK0Bnrf1F+0qZ0JhiNq2212TJUbbd9H6EgWnAWu0Qv/Kd5lGFcu?= =?us-ascii?Q?nHmW78cwjzMN6Uy9jKA3JbJWsEscdvshL3Gal16AJTuCZL31QEeyJpKwt4J/?= =?us-ascii?Q?cPssPpOO40/ZMLZQk3aurN1Woc1Ru63DDIxQnx38ndzwGsAf68LN1a5b1YzB?= =?us-ascii?Q?5noYD1oTwiPS3wAYlfc0uFCOBtmeNMMxEz8SwN563B6P30TQgf66N6PFhQ/x?= =?us-ascii?Q?X4tKf7zZI0Fk1+CuFi1uduxY1eNGDpEnCiLVt5b/fR5f7PK/bFZly2f1oDkj?= =?us-ascii?Q?1oJdS1tNGTF/HZ2jqoN6n86H26mrbS1wB+UYYoLzzlYodY9XvVsp24x4uhvY?= =?us-ascii?Q?M1L0JD4ZAmJBcjNTIR+cxz+2q1hGJsCGVphjoCA4lgSNHuRpOSVo4UW0zE8q?= =?us-ascii?Q?CTTn3yRdGauOBVjIgrn9RbJ0y7evCAnI9LHdMbEHKj7gG/YPKtKCO4IUPx/k?= =?us-ascii?Q?Jyx57xs/fGff7VQbMMylnmA/zBKzmJ/s+6Wy5ouyliXpqB05eai/riTQ/+gu?= =?us-ascii?Q?PqN86eAeDGjE+H7nAbwuvojG8L+GWtxPES2A3MFfkYHsH0qYAxrXXoSrjL1u?= =?us-ascii?Q?BEhAU3hH+PcoYjKaeigVCrUrDxj0TEZ13wFWiEMjK8m3p6e+GniJda/ZB77s?= =?us-ascii?Q?chcC4vtmHQXquBB3NC2xU4Zt3VoVZxr+Nl6DMZRGdOgaTft2/hRETWk4QeHy?= =?us-ascii?Q?gpddOiyu4U8VUT6cxEGt9uuobFxEjnTrB+W/c9ypT1ukuZKFDqB+SDTC5rAP?= =?us-ascii?Q?9eXMVk2befzxKNoTEJd+Y2raPFWIEk7nE80Qqo9vP6SIFGSSXRHk8ku4qujp?= =?us-ascii?Q?57yAgCyst1E2zf3471TRxIuFNLxqbFkFvF+XbzJgC6hLPsHXRqhpr1bW/jtR?= =?us-ascii?Q?e552rE2Qe2dzmUyFz10NAMi5Cgzst4eAdsOFdGjvsJfT9I8rH5nJC0BF55gy?= =?us-ascii?Q?NPgO3xreaNrN6iFWIFUcDh7iQbtpJIJoq1wcnF3FVbtFg6im7wwDn1xioTy4?= =?us-ascii?Q?ipJm03UtSH6EhrImGi+dPphO9587yWcjwOtaHKM5qGO1F7I3Do/Kt17juIuA?= =?us-ascii?Q?eM2evWEuvyiVOjEjD0KQOFzkfvpDhI4/O936jPE7Hyw15c56lLdh3gRpn5qJ?= =?us-ascii?Q?CgGKIUopo3LN3rhccs7ovAK1+RLbQ3tS1VMomfPrnNGBwmoK0gv1ZZHCFOYt?= =?us-ascii?Q?BaMGT/YCMXW3LB8L1Q5tjg2mIPHu1QTxr7FfRjCQlf27HErtZqQgvtvpPyMf?= =?us-ascii?Q?NdAxZeJmTat4ZQ7OW+7RGVmD62Nkw4RvGGQlGM3t4krwDx0NsYvDXWIThFdA?= =?us-ascii?Q?FmKt8wM6HoslEN+93joaV9o8RC4c6j2zmSBMsRZb/PXiKEaJteXmw19EX1wi?= =?us-ascii?Q?NlNmoYE=3D?= X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230040)(376014)(1800799024)(36860700013)(82310400026); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Oct 2024 08:38:55.4092 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 03fac9a4-c595-45da-d7ab-08dcedbdf894 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MWH0EPF000989E6.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB6756 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Rongwei Liu Add a new devarg probe_opt_en to control probe optimization in PMD. By default, the value is 0 and no behavior changed. Signed-off-by: Rongwei Liu Acked-by: Viacheslav Ovsiienko --- doc/guides/nics/mlx5.rst | 7 +++++++ drivers/common/mlx5/linux/mlx5_nl.c | 12 ++++++++---- drivers/common/mlx5/mlx5_common.c | 15 +++++++++++++++ drivers/common/mlx5/mlx5_common.h | 2 ++ drivers/net/mlx5/linux/mlx5_ethdev_os.c | 5 ++++- drivers/net/mlx5/linux/mlx5_os.c | 2 +- 6 files changed, 37 insertions(+), 6 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 1dccdaad50..b4a4e57cde 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -1436,6 +1436,13 @@ for an additional list of options shared with other mlx5 drivers. By default, the PMD will set this value to 1. +- ``probe_opt_en`` parameter [int] + + A non-zero value optimizes the probe process, especially for large scale. + PMD will hold the IB device information internally and reuse it. + + By default, the PMD will set this value to 0. + - ``lacp_by_user`` parameter [int] A nonzero value enables the control of LACP traffic by the user application. diff --git a/drivers/common/mlx5/linux/mlx5_nl.c b/drivers/common/mlx5/linux/mlx5_nl.c index e98073aafe..745e443f8f 100644 --- a/drivers/common/mlx5/linux/mlx5_nl.c +++ b/drivers/common/mlx5/linux/mlx5_nl.c @@ -1148,7 +1148,7 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex, struct mlx5_dev_info .flags = 0, }; - if (!strcmp(name, dev_info->ibname)) { + if (dev_info->probe_opt && !strcmp(name, dev_info->ibname)) { if (dev_info->port_info && pindex <= dev_info->port_num && dev_info->port_info[pindex].valid) { if (!dev_info->port_info[pindex].ifindex) @@ -1161,7 +1161,7 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex, struct mlx5_dev_info ret = mlx5_nl_port_info(nl, pindex, &data); - if (!strcmp(dev_info->ibname, name)) { + if (dev_info->probe_opt && !strcmp(dev_info->ibname, name)) { if ((!ret || ret == -ENODEV) && dev_info->port_info && pindex <= dev_info->port_num) { if (!ret) @@ -1201,7 +1201,8 @@ mlx5_nl_port_state(int nl, const char *name, uint32_t pindex, struct mlx5_dev_in .ibindex = UINT32_MAX, }; - if (dev_info && !strcmp(name, dev_info->ibname) && dev_info->port_num) + if (dev_info && dev_info->probe_opt && + !strcmp(name, dev_info->ibname) && dev_info->port_num) data.ibindex = dev_info->ibindex; if (mlx5_nl_port_info(nl, pindex, &data) < 0) return -rte_errno; @@ -1244,7 +1245,8 @@ mlx5_nl_portnum(int nl, const char *name, struct mlx5_dev_info *dev_info) uint32_t sn = MLX5_NL_SN_GENERATE; int ret, size; - if (dev_info->port_num && !strcmp(name, dev_info->ibname)) + if (dev_info->probe_opt && dev_info->port_num && + !strcmp(name, dev_info->ibname)) return dev_info->port_num; ret = mlx5_nl_send(nl, &req, sn); @@ -1263,6 +1265,8 @@ mlx5_nl_portnum(int nl, const char *name, struct mlx5_dev_info *dev_info) rte_errno = EINVAL; return 0; } + if (!dev_info->probe_opt) + return data.portnum; MLX5_ASSERT(!strlen(dev_info->ibname)); dev_info->port_num = data.portnum; dev_info->ibindex = data.ibindex; diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 0aaae91c31..9abae4a374 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -40,6 +40,9 @@ uint8_t haswell_broadwell_cpu; /* The default memory allocator used in PMD. */ #define MLX5_SYS_MEM_EN "sys_mem_en" +/* Probe optimization in PMD. */ +#define MLX5_PROBE_OPT "probe_opt_en" + /* * Device parameter to force doorbell register mapping * to non-cached region eliminating the extra write memory barrier. @@ -295,6 +298,8 @@ mlx5_common_args_check_handler(const char *key, const char *val, void *opaque) config->device_fd = tmp; } else if (strcmp(key, MLX5_PD_HANDLE) == 0) { config->pd_handle = tmp; + } else if (strcmp(key, MLX5_PROBE_OPT) == 0) { + config->probe_opt = !!tmp; } return 0; } @@ -324,6 +329,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist, MLX5_MR_MEMPOOL_REG_EN, MLX5_DEVICE_FD, MLX5_PD_HANDLE, + MLX5_PROBE_OPT, NULL, }; int ret = 0; @@ -332,6 +338,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist, config->mr_ext_memseg_en = 1; config->mr_mempool_reg_en = 1; config->sys_mem_en = 0; + config->probe_opt = 0; config->dbnc = MLX5_ARG_UNSET; config->device_fd = MLX5_ARG_UNSET; config->pd_handle = MLX5_ARG_UNSET; @@ -351,6 +358,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist, DRV_LOG(DEBUG, "mr_ext_memseg_en is %u.", config->mr_ext_memseg_en); DRV_LOG(DEBUG, "mr_mempool_reg_en is %u.", config->mr_mempool_reg_en); DRV_LOG(DEBUG, "sys_mem_en is %u.", config->sys_mem_en); + DRV_LOG(DEBUG, "probe_opt_en is %u.", config->probe_opt); DRV_LOG(DEBUG, "Send Queue doorbell mapping parameter is %d.", config->dbnc); return ret; @@ -791,6 +799,7 @@ mlx5_common_dev_create(struct rte_device *eal_dev, uint32_t classes, if (TAILQ_EMPTY(&devices_list)) rte_mem_event_callback_register("MLX5_MEM_EVENT_CB", mlx5_mr_mem_event_cb, NULL); + cdev->dev_info.probe_opt = cdev->config.probe_opt; exit: pthread_mutex_lock(&devices_list_lock); TAILQ_INSERT_HEAD(&devices_list, cdev, next); @@ -880,6 +889,12 @@ mlx5_common_probe_again_args_validate(struct mlx5_common_device *cdev, cdev->dev->name); goto error; } + if (cdev->config.probe_opt != config->probe_opt) { + DRV_LOG(ERR, "\"" MLX5_PROBE_OPT"\" " + "configuration mismatch for device %s.", + cdev->dev->name); + goto error; + } if (cdev->config.dbnc != config->dbnc) { DRV_LOG(ERR, "\"" MLX5_SQ_DB_NC "\" " "configuration mismatch for device %s.", diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 6cb40f54dd..f1b59d6f07 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -183,6 +183,7 @@ struct mlx5_dev_info { uint32_t port_num; uint32_t ibindex; char ibname[MLX5_FS_NAME_MAX]; + uint8_t probe_opt; struct mlx5_port_nl_info *port_info; }; @@ -525,6 +526,7 @@ struct mlx5_common_dev_config { int pd_handle; /* Protection Domain handle for importation. */ unsigned int devx:1; /* Whether devx interface is available or not. */ unsigned int sys_mem_en:1; /* The default memory allocator. */ + unsigned int probe_opt:1; /* Optimize probing . */ unsigned int mr_mempool_reg_en:1; /* Allow/prevent implicit mempool memory registration. */ unsigned int mr_ext_memseg_en:1; diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c index 08ac6dd939..88d3c57c6e 100644 --- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c +++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c @@ -691,6 +691,8 @@ mlx5_handle_port_info_update(struct mlx5_dev_info *dev_info, uint32_t if_index, if (dev_info->port_num <= 1 || dev_info->port_info == NULL) return; + DRV_LOG(DEBUG, "IB device %s ifindex %u received netlink event %u", + dev_info->ibname, if_index, msg_type); for (i = 1; i <= dev_info->port_num; i++) { if (!dev_info->port_info[i].valid) continue; @@ -734,7 +736,8 @@ mlx5_dev_interrupt_nl_cb(struct nlmsghdr *hdr, void *cb_arg) if (mlx5_nl_parse_link_status_update(hdr, &if_index) < 0) return; - mlx5_handle_port_info_update(&sh->cdev->dev_info, if_index, hdr->nlmsg_type); + if (sh->cdev->config.probe_opt && sh->cdev->dev_info.port_num > 1) + mlx5_handle_port_info_update(&sh->cdev->dev_info, if_index, hdr->nlmsg_type); for (i = 0; i < sh->max_port; i++) { struct mlx5_dev_shared_port *port = &sh->port[i]; diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index dcf1ff917b..a408790d1e 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -2335,7 +2335,7 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, while (ret-- > 0) { struct rte_pci_addr pci_addr; - if (cdev->dev_info.port_num) { + if (cdev->config.probe_opt && cdev->dev_info.port_num) { if (strcmp(ibv_list[ret]->name, cdev->dev_info.ibname)) { DRV_LOG(INFO, "Unmatched caching device \"%s\" \"%s\"", cdev->dev_info.ibname, ibv_list[ret]->name); -- 2.34.1