From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AE0AF45BFD; Mon, 28 Oct 2024 10:19:57 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 97A3540DF6; Mon, 28 Oct 2024 10:19:45 +0100 (CET) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2084.outbound.protection.outlook.com [40.107.237.84]) by mails.dpdk.org (Postfix) with ESMTP id C910040E17 for ; Mon, 28 Oct 2024 10:19:43 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gjWRZ82eN7B1EKOln0UsdhDWSbfxLbrAoN5fQ+8/PO02wtmCsVMSN1EbkqjWT9VmLDTsT2fy4lNOoEWcAdWPjeiW2O0wWzDFasnCWNEl6f0AfMMWsJ7zOsyY0s/jB2xECNhVpNRj+yVXPbalogczqIQN62zh44KXHpayYUE+UJbBcfUmm/Zq1vEXuDsGCqxrZ+of+4bVRkf9UnuiibWh1PUlwlxXSBTmefu/6J6pGIdjBsSGKQ9tVfZej07leqhk8gEQ40sCt9XSlMu8JSzP6UP15S7BYh9fV2lta6ZM3ijv89nloeoxlF5rY9tJ+B4gd+SdH3zFNwrvLbSXDIgy9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=5fvyO3a8goK6peUGOdieUke6we/yNe9nKOxnwghIvwA=; b=lLWenWvPjXE2oUYlkytynexEB85+ApsLw8F5CxomGW1w7XwnwfL0uzRVd065aEU5wYAN5ScpogP+W/7joR8glumgsN8PDWenic89+XxFYBrpY2PaQMtDjtnAYdJx1uA6lTLM1AlI1lSa3dSFPDSsd6FuHbY7ys5Pl2QP77z/xWC2fhY5zUGU3jkSrSegKXL0nq+CabNtIXTiFQTqMr+bExEWpc3xG2eNeTmGyfhjr2wzwvZ4QCZ74lV04KJ3tI/8u9lcYqC8PO+5QRPwh8ENV4/maQC6/Wvo/qMOMWq7QrxzsM2EOIgC1EP7X/1w40MLbIPQ2R8SBQpMxLhmv91ucg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=monjalon.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5fvyO3a8goK6peUGOdieUke6we/yNe9nKOxnwghIvwA=; b=pS04SNGUgp1fQ/Z+eRxhrOcZnt/6Dalzpmo6TBO4Z4ds2X/asLxoIzqNjr0dep2ai1ue86ugrFiwC1ndwqlJt1tYpYfkAobRdJW7hQdMQyi22ybPinapzGPisAyRaAVsO03NHHQVGMFjSlLQIKm6CdxhjykIltpyYEDHruQ5Rus4jvstmIrPwahGrFF2cAFNH1W28TEcwsmjh5o/QBnYZ4qVpPb6gksU6gObbvr7/7FwhuyrhqNEN7a1iy6YUv9vg8ZAYM52A24Uag60PAUXlBr6cD56oMu3tIojBAb3pz+f7FqZ2NF4FTWx64ZaCgoivXyFGi46COr1T9XbtyQeiw== Received: from CH0PR13CA0033.namprd13.prod.outlook.com (2603:10b6:610:b2::8) by PH7PR12MB7968.namprd12.prod.outlook.com (2603:10b6:510:272::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8093.23; Mon, 28 Oct 2024 09:19:02 +0000 Received: from CH2PEPF0000013D.namprd02.prod.outlook.com (2603:10b6:610:b2:cafe::fa) by CH0PR13CA0033.outlook.office365.com (2603:10b6:610:b2::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8114.14 via Frontend Transport; Mon, 28 Oct 2024 09:19:02 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CH2PEPF0000013D.mail.protection.outlook.com (10.167.244.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8114.16 via Frontend Transport; Mon, 28 Oct 2024 09:19:01 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Mon, 28 Oct 2024 02:18:51 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Mon, 28 Oct 2024 02:18:47 -0700 From: "Minggang Li(Gavin)" To: , , , , Dariusz Sosnowski , Bing Zhao , Suanming Mou CC: , , Rongwei Liu Subject: [PATCH V2 3/7] net/mlx5: add new devargs to control probe optimization Date: Mon, 28 Oct 2024 11:18:18 +0200 Message-ID: <20241028091822.860660-4-gavinl@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241028091822.860660-1-gavinl@nvidia.com> References: <20241016083818.662020-8-gavinl@nvidia.com> <20241028091822.860660-1-gavinl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF0000013D:EE_|PH7PR12MB7968:EE_ X-MS-Office365-Filtering-Correlation-Id: f526ed33-cda5-4def-bffe-08dcf7318fee X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|1800799024|36860700013|376014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?vFxVr5XV9wOpgEKsy6/OmRYsYjOFaDM895Lpr7fSDPBcwOT0JqXhQdPwBZef?= =?us-ascii?Q?xydBpZC1GRRIY06vmgKg2Jfn25PxGVyKCzq5gMa9Rqu6Fe/UCq9tiyRYuygn?= =?us-ascii?Q?IOYTtb9yn0r9d49d93Ip6YJp/g6GzQRLHnSGcBDYrJ4Mk28CBIb7A0EmeJN6?= =?us-ascii?Q?pyt3QGkHCNs2F9JDFB6JFhvxY2T7o7bm7N1mIZ91d1zew2u+Bel/Fm54AaQ5?= =?us-ascii?Q?1kBPk1yLRoSPPTNzcuCE+5TvryPurbb8XN2ns34AKpB3twoio6hpF/2K2N6s?= =?us-ascii?Q?Iqd+7oWlVukyuh2FcOfI09btMS0BSVLHiYCqmoGlhncvJ+hKlWnAXG1OR3uk?= =?us-ascii?Q?ALL3LkULZ9UeehaA3exDueqdouZ+0zCP5ff2AWKncPb9au7w8I6ZA69W8LxL?= =?us-ascii?Q?J7LJdkzT00gfHtOZvGXWTMf9clUJzeNXMvNWo14hjne1H4KXchLRsl0m7nc+?= =?us-ascii?Q?7kPiUFB1Vo2U8ab4jxzw9dj0BpXgBpOJi7DqCOruY9H72rRHQs4DyRItSWQe?= =?us-ascii?Q?WyahK6fjY28YlWgytSxkUe4/ve0DRveJ6gCAGp53ZRucZ99E4EZa1I4NWjdx?= =?us-ascii?Q?gpOSGj0HgD8gced9JThHDX5eHOuGBZnvcW+zlGJJKTsSGgoLKIuIJmFwlr9q?= =?us-ascii?Q?+/V2k+4I/4U13azt7LmDlnGSbuP3u1Octh33fE9bT+LfxzeCdBFkXbngaqMJ?= =?us-ascii?Q?CHi4el07D6NWYMbvVokNVOsQbihqZ9P27UFi7YWp317/NgPuK7eM0Hs+/qe7?= =?us-ascii?Q?wicFGyV4xNrovBaeWR6pzf4btwlRkYORN4cDHj1FfGXQ1z8fqUwdAw3VEftl?= =?us-ascii?Q?0DogW3iCkhimmNkdkrxYIur+KKC94umektrt1aWGTNlEb4rZmZs5RmzJbyFZ?= =?us-ascii?Q?ghPOhgD10R9pUgmcV36vesEmDJbbEMgw0G1dE9xAqdhWt5a0Rt1QW4LxOjdz?= =?us-ascii?Q?r4DQiON7OoNZMrhmjFIHizBH313D9t/WsuYQ5iqruBGWLF6K0K/dYuY7870S?= =?us-ascii?Q?fEIyBiUr5+p0rVWPctRE5eNEND4JJj+TZLdBzs4rg5V+4ssPU73M/oxKaGKV?= =?us-ascii?Q?mNci3jkkive92a7PTv7ODa7ycDvRxqxIoisgJeeLWSsxTdpbFJYrkvOF7upp?= =?us-ascii?Q?lhO5TrcMXRY6v6DWpPlsndCW8buV3j1mS/20k9/H5cunCL3YROhiv1m15d3B?= =?us-ascii?Q?SS9sI1F9o1puTLwIr8GE3U0gJo+RRdc+qLGTKZyw7g+Ng57T9VAS9Qe6AI4Z?= =?us-ascii?Q?DRPuuBrMvHV1mBERas9+/HUlZgDCE+4XcZQIjGJtCzAUgPzZ7OdU+cgGJW7M?= =?us-ascii?Q?pfvujaZYFTXZ0PnAewpVEN4NFbKVBD5RObT8YagNWIBV6v+Hgjrr6K4OIXbb?= =?us-ascii?Q?aQTHUzpUx1yWIRZVIPVPaNJO6Ni+xdvj+L2/bZet5lNyhPWAMg=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230040)(1800799024)(36860700013)(376014)(82310400026); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Oct 2024 09:19:01.8295 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f526ed33-cda5-4def-bffe-08dcf7318fee X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF0000013D.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7968 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Rongwei Liu Add a new devarg probe_opt_en to control probe optimization in PMD. By default, the value is 0 and no behavior changed. Signed-off-by: Rongwei Liu Acked-by: Viacheslav Ovsiienko --- doc/guides/nics/mlx5.rst | 7 +++++++ drivers/common/mlx5/linux/mlx5_nl.c | 12 ++++++++---- drivers/common/mlx5/mlx5_common.c | 15 +++++++++++++++ drivers/common/mlx5/mlx5_common.h | 2 ++ drivers/net/mlx5/linux/mlx5_ethdev_os.c | 5 ++++- drivers/net/mlx5/linux/mlx5_os.c | 2 +- 6 files changed, 37 insertions(+), 6 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index f82e2d75de..981401a9f2 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -1436,6 +1436,13 @@ for an additional list of options shared with other mlx5 drivers. By default, the PMD will set this value to 1. +- ``probe_opt_en`` parameter [int] + + A non-zero value optimizes the probe process, especially for large scale. + PMD will hold the IB device information internally and reuse it. + + By default, the PMD will set this value to 0. + - ``lacp_by_user`` parameter [int] A nonzero value enables the control of LACP traffic by the user application. diff --git a/drivers/common/mlx5/linux/mlx5_nl.c b/drivers/common/mlx5/linux/mlx5_nl.c index e98073aafe..745e443f8f 100644 --- a/drivers/common/mlx5/linux/mlx5_nl.c +++ b/drivers/common/mlx5/linux/mlx5_nl.c @@ -1148,7 +1148,7 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex, struct mlx5_dev_info .flags = 0, }; - if (!strcmp(name, dev_info->ibname)) { + if (dev_info->probe_opt && !strcmp(name, dev_info->ibname)) { if (dev_info->port_info && pindex <= dev_info->port_num && dev_info->port_info[pindex].valid) { if (!dev_info->port_info[pindex].ifindex) @@ -1161,7 +1161,7 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex, struct mlx5_dev_info ret = mlx5_nl_port_info(nl, pindex, &data); - if (!strcmp(dev_info->ibname, name)) { + if (dev_info->probe_opt && !strcmp(dev_info->ibname, name)) { if ((!ret || ret == -ENODEV) && dev_info->port_info && pindex <= dev_info->port_num) { if (!ret) @@ -1201,7 +1201,8 @@ mlx5_nl_port_state(int nl, const char *name, uint32_t pindex, struct mlx5_dev_in .ibindex = UINT32_MAX, }; - if (dev_info && !strcmp(name, dev_info->ibname) && dev_info->port_num) + if (dev_info && dev_info->probe_opt && + !strcmp(name, dev_info->ibname) && dev_info->port_num) data.ibindex = dev_info->ibindex; if (mlx5_nl_port_info(nl, pindex, &data) < 0) return -rte_errno; @@ -1244,7 +1245,8 @@ mlx5_nl_portnum(int nl, const char *name, struct mlx5_dev_info *dev_info) uint32_t sn = MLX5_NL_SN_GENERATE; int ret, size; - if (dev_info->port_num && !strcmp(name, dev_info->ibname)) + if (dev_info->probe_opt && dev_info->port_num && + !strcmp(name, dev_info->ibname)) return dev_info->port_num; ret = mlx5_nl_send(nl, &req, sn); @@ -1263,6 +1265,8 @@ mlx5_nl_portnum(int nl, const char *name, struct mlx5_dev_info *dev_info) rte_errno = EINVAL; return 0; } + if (!dev_info->probe_opt) + return data.portnum; MLX5_ASSERT(!strlen(dev_info->ibname)); dev_info->port_num = data.portnum; dev_info->ibindex = data.ibindex; diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 0aaae91c31..9abae4a374 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -40,6 +40,9 @@ uint8_t haswell_broadwell_cpu; /* The default memory allocator used in PMD. */ #define MLX5_SYS_MEM_EN "sys_mem_en" +/* Probe optimization in PMD. */ +#define MLX5_PROBE_OPT "probe_opt_en" + /* * Device parameter to force doorbell register mapping * to non-cached region eliminating the extra write memory barrier. @@ -295,6 +298,8 @@ mlx5_common_args_check_handler(const char *key, const char *val, void *opaque) config->device_fd = tmp; } else if (strcmp(key, MLX5_PD_HANDLE) == 0) { config->pd_handle = tmp; + } else if (strcmp(key, MLX5_PROBE_OPT) == 0) { + config->probe_opt = !!tmp; } return 0; } @@ -324,6 +329,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist, MLX5_MR_MEMPOOL_REG_EN, MLX5_DEVICE_FD, MLX5_PD_HANDLE, + MLX5_PROBE_OPT, NULL, }; int ret = 0; @@ -332,6 +338,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist, config->mr_ext_memseg_en = 1; config->mr_mempool_reg_en = 1; config->sys_mem_en = 0; + config->probe_opt = 0; config->dbnc = MLX5_ARG_UNSET; config->device_fd = MLX5_ARG_UNSET; config->pd_handle = MLX5_ARG_UNSET; @@ -351,6 +358,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist, DRV_LOG(DEBUG, "mr_ext_memseg_en is %u.", config->mr_ext_memseg_en); DRV_LOG(DEBUG, "mr_mempool_reg_en is %u.", config->mr_mempool_reg_en); DRV_LOG(DEBUG, "sys_mem_en is %u.", config->sys_mem_en); + DRV_LOG(DEBUG, "probe_opt_en is %u.", config->probe_opt); DRV_LOG(DEBUG, "Send Queue doorbell mapping parameter is %d.", config->dbnc); return ret; @@ -791,6 +799,7 @@ mlx5_common_dev_create(struct rte_device *eal_dev, uint32_t classes, if (TAILQ_EMPTY(&devices_list)) rte_mem_event_callback_register("MLX5_MEM_EVENT_CB", mlx5_mr_mem_event_cb, NULL); + cdev->dev_info.probe_opt = cdev->config.probe_opt; exit: pthread_mutex_lock(&devices_list_lock); TAILQ_INSERT_HEAD(&devices_list, cdev, next); @@ -880,6 +889,12 @@ mlx5_common_probe_again_args_validate(struct mlx5_common_device *cdev, cdev->dev->name); goto error; } + if (cdev->config.probe_opt != config->probe_opt) { + DRV_LOG(ERR, "\"" MLX5_PROBE_OPT"\" " + "configuration mismatch for device %s.", + cdev->dev->name); + goto error; + } if (cdev->config.dbnc != config->dbnc) { DRV_LOG(ERR, "\"" MLX5_SQ_DB_NC "\" " "configuration mismatch for device %s.", diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 6cb40f54dd..f1b59d6f07 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -183,6 +183,7 @@ struct mlx5_dev_info { uint32_t port_num; uint32_t ibindex; char ibname[MLX5_FS_NAME_MAX]; + uint8_t probe_opt; struct mlx5_port_nl_info *port_info; }; @@ -525,6 +526,7 @@ struct mlx5_common_dev_config { int pd_handle; /* Protection Domain handle for importation. */ unsigned int devx:1; /* Whether devx interface is available or not. */ unsigned int sys_mem_en:1; /* The default memory allocator. */ + unsigned int probe_opt:1; /* Optimize probing . */ unsigned int mr_mempool_reg_en:1; /* Allow/prevent implicit mempool memory registration. */ unsigned int mr_ext_memseg_en:1; diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c index 08ac6dd939..88d3c57c6e 100644 --- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c +++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c @@ -691,6 +691,8 @@ mlx5_handle_port_info_update(struct mlx5_dev_info *dev_info, uint32_t if_index, if (dev_info->port_num <= 1 || dev_info->port_info == NULL) return; + DRV_LOG(DEBUG, "IB device %s ifindex %u received netlink event %u", + dev_info->ibname, if_index, msg_type); for (i = 1; i <= dev_info->port_num; i++) { if (!dev_info->port_info[i].valid) continue; @@ -734,7 +736,8 @@ mlx5_dev_interrupt_nl_cb(struct nlmsghdr *hdr, void *cb_arg) if (mlx5_nl_parse_link_status_update(hdr, &if_index) < 0) return; - mlx5_handle_port_info_update(&sh->cdev->dev_info, if_index, hdr->nlmsg_type); + if (sh->cdev->config.probe_opt && sh->cdev->dev_info.port_num > 1) + mlx5_handle_port_info_update(&sh->cdev->dev_info, if_index, hdr->nlmsg_type); for (i = 0; i < sh->max_port; i++) { struct mlx5_dev_shared_port *port = &sh->port[i]; diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index e7277d1e43..4c2caa21e9 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -2338,7 +2338,7 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, while (ret-- > 0) { struct rte_pci_addr pci_addr; - if (cdev->dev_info.port_num) { + if (cdev->config.probe_opt && cdev->dev_info.port_num) { if (strcmp(ibv_list[ret]->name, cdev->dev_info.ibname)) { DRV_LOG(INFO, "Unmatched caching device \"%s\" \"%s\"", cdev->dev_info.ibname, ibv_list[ret]->name); -- 2.34.1