From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 41F8043E57 for ; Sat, 13 Apr 2024 14:55:50 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3A4C940294; Sat, 13 Apr 2024 14:55:50 +0200 (CEST) Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2045.outbound.protection.outlook.com [40.107.93.45]) by mails.dpdk.org (Postfix) with ESMTP id 69AF740294 for ; Sat, 13 Apr 2024 14:55:48 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CLAuF64p0q0gcR87tycPfNuISHwXYOYpbCTVs0h/1KtsNu04oXDeVkrIb9XIvZDcQamSsw4azp7BWQZX3sEjIClzKu9mQgneMWNgl2i4Fl2XWerpOgr86oZCd3TWRegp3aDmtBS3bdTg5d2SFFQTVUg43FFdqbJy6053xSqgzCfH2l29kqRRPAEyP51vr+VCBL1qG1RpeBW5fXTSI0UNZhcT2oYujmdVGJ42mkftm0zOfP17WOnVNHcR/ZNgXn0XQLhhEUHnGGLtwFRReB3WUzrjMADSYAu90o202uNbudExSDN5moNsoAX0kYCmq+O1xrhHZlb9RsE/D2qPxj8b1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jJajkReg1SuaYqOMX/4T/5YGIeWdNRXjsZkCO42eIOQ=; b=OC8PknB2s+ixR3iiUOq/yrcdrrKMuEUD8sykB/qGoTe65A0TL6wR94Js9hb1VSx30+musDFCRhOZhwuCPrzZT0SltWKTWUBpxm/B0X9jh8KpRFg+C79XlW7VJR61NHtp0mbH6AzwP3tqXtAcbANcwKpUvhbgbD5m8dRk+UfOpnGZ6If0BohMZq2ovVLkRfP7u701E3WsPn8r49gsrgzfk3n8ZVF/2QarLZcV46r6B3oLOxNZToIZ6KWXNy7uq6zNNyIwiztAFblb8jh1Ohg+hAPftwN7Wp1LMUxx8mzv4+svT0ouXNiXIM6YTpk4ZMarsMz4rnnjO7cOSApVbKSnpA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jJajkReg1SuaYqOMX/4T/5YGIeWdNRXjsZkCO42eIOQ=; b=SAhqycN2EFs5LnPvnO0h0mI82uhgfglT/IEKjV2MNIGHlQR8Ar9fSQxN/fvDLKrxdotq72CdwvEXe9R9qlnCAQr2NmWW1vL3VspU2qTrX/KdWT4hCCD83eBsnCVtI4KJK3j0wLFm4TRzbfp9Z/89TW2aEeTzrN2nnAPjtaYz6EploXfD7OZyoTs65E01qnkOBjDqvl0hHO77zwkDl1u+rzUWrRvy2DT7JAoH5xX9luvJVVmatz8wV+GuD3vIUHz+9XeaEslKCE1lgtDgcPgzG/pNH2hNwOVHqNM0ygAZh09TuaRaTB/fRBbkrCVPMzlQqw2oeneCuGWwNBF6al5DHg== Received: from DS7PR03CA0141.namprd03.prod.outlook.com (2603:10b6:5:3b4::26) by DS0PR12MB9423.namprd12.prod.outlook.com (2603:10b6:8:192::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.55; Sat, 13 Apr 2024 12:55:45 +0000 Received: from DS3PEPF0000C37C.namprd04.prod.outlook.com (2603:10b6:5:3b4:cafe::71) by DS7PR03CA0141.outlook.office365.com (2603:10b6:5:3b4::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7472.26 via Frontend Transport; Sat, 13 Apr 2024 12:55:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DS3PEPF0000C37C.mail.protection.outlook.com (10.167.23.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7452.22 via Frontend Transport; Sat, 13 Apr 2024 12:55:45 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Sat, 13 Apr 2024 05:55:26 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1258.12; Sat, 13 Apr 2024 05:55:24 -0700 From: Xueming Li To: Bing Zhao CC: Viacheslav Ovsiienko , dpdk stable Subject: patch 'net/mlx5: fix counters map in bonding mode' has been queued to stable release 23.11.1 Date: Sat, 13 Apr 2024 20:49:00 +0800 Message-ID: <20240413125005.725659-60-xuemingl@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240413125005.725659-1-xuemingl@nvidia.com> References: <20240305094757.439387-1-xuemingl@nvidia.com> <20240413125005.725659-1-xuemingl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF0000C37C:EE_|DS0PR12MB9423:EE_ X-MS-Office365-Filtering-Correlation-Id: 96fde5a5-6847-4d6f-529d-08dc5bb908c5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: JDM1qEc+UpSGJeDNTPD21seg7odIg4nyreIAVdsvTKyEU5/SQqHvRSLo79O3c5SMichyfP64580yHw+sn9Nz27XdEQqmyP+Tl6s2PK3Da73jZqAT0Hf4SeL+vabM2qRQ269cNPec4NgBvSX9F0WaunbkxYWlX6gvfar1vAglfIS9xdLa3nlx3HLKvrzkW7yW9dMA17lfxRBMZ9KHq5Gwz+Be/WxPM32uV/RWbrKbgu3DiqV03zeVzKqVkYklwcK6kJGkc3krdT1xQSosJzKemK5alQY3qREfpvUGntMV1llaXMzw5ROqI+yEtcI5cfVx1ec1bP7eARUuM1n2HTGGxOAAshiRv6mYtvQSVSeeDujtxa0rBET07fclenmjgh+flr5UA5qOWtDJIYdCoY/7uoawZ6Mh3sikDdUsDMmEmTWs1vKd8/gxrYfgkepOL5s4AqdM1biHMZkII4BVtLJWLQyq3MJHtDf0MIFe4kxCLgD2JHcywcmlCJZXEHuOemHX9EPMrociWL1yt22B7vYKjpCc+VcPohNO2D7QGagdCCNoCiGl0+9EhRNYwwLA3RJsnAaE72AAni91ST1U+JoXtCKAnJw7rEk8yBb24qLRNbfJiEUejRDaQZUJpS2tIcDgSxFbvUb0R/q7ZnNxQlkGHlG+iu144gCqiu6GmagP6hnaXLohN/88h1U04lhTMFFxEY13vBtzVzhwUpN1IdOgosR1QOcJEi5jsnCJGbqioXiYkogcG1WsCoTC/IHj4yIB X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230031)(1800799015)(82310400014)(376005)(36860700004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Apr 2024 12:55:45.2432 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 96fde5a5-6847-4d6f-529d-08dc5bb908c5 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF0000C37C.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB9423 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Hi, FYI, your patch has been queued to stable release 23.11.1 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 04/15/24. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Queued patches are on a temporary branch at: https://git.dpdk.org/dpdk-stable/log/?h=23.11-staging This queued commit can be viewed at: https://git.dpdk.org/dpdk-stable/commit/?h=23.11-staging&id=0c31d1220ffaff4742154c3c957ab5305e5f5c3a Thanks. Xueming Li --- >From 0c31d1220ffaff4742154c3c957ab5305e5f5c3a Mon Sep 17 00:00:00 2001 From: Bing Zhao Date: Thu, 29 Feb 2024 11:34:56 +0200 Subject: [PATCH] net/mlx5: fix counters map in bonding mode Cc: Xueming Li [ upstream commit a687c3e658c2d889052089af8340bc0b9299c856 ] In the HW-LAG mode, there is only one mlx5 IB device with 2 ETH interfaces. In theory, the settings on both ports should be the same. But in the real life, some inconsistent settings may be done by the user and the PMD is not aware of this. In the previous implementation, the xstats map was generated from the information fetched on the 1st port of a bonding interface. If the 2nd port had a different settings, the number and the order of the counters may differ from that of the 1st one. The ioctl() call may corrupt the user buffers (copy_to_user) and cause a crash. The commit will change the map between the driver counters to the PMD user counters. 1. Switch the inner and outer loop to speed up the initialization time AMAP - since there will be >300 counters returned from the driver. 2. Generate an unique map for both ports in LAG mode. a. Scan the 1st port and find the supported counters' strings, then add to the map. b. In bonding, scan the 2nd port and find the strings. If one is already in the map, use the index. Or append to the next free slot. c. Append the device counters that needs to be fetched via sysfs or Devx command. This kind of counter(s) is unique per IB device. After querying the statistics from the driver, the value will be read from the proper offset in the "struct ethtool_stats" and then added into the output array based on the map information. In bonding mode, the statistics from both ports will be accumulated if the counters are valid on both ports. Compared to the system call or Devx command, the overhead introduced by the extra index comparison is light and should not cause a significant degradation. The application should ensure that the port settings should not be changed out of the DPDK application dynamically in most cases. Or else the change cannot be notified and the counters map might not be valid when the number doesn't change but the counters set had changed. A device restart will help to re-initialize the map from scrath. Fixes: 7ed15acdcd69 ("net/mlx5: improve xstats of bonding port") Signed-off-by: Bing Zhao Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/linux/mlx5_ethdev_os.c | 249 +++++++++++++++------- drivers/net/mlx5/mlx5.h | 15 +- drivers/net/mlx5/mlx5_stats.c | 58 +++-- drivers/net/mlx5/windows/mlx5_ethdev_os.c | 22 +- 4 files changed, 242 insertions(+), 102 deletions(-) diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c index dd5a0c546d..0ee8c58ba7 100644 --- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c +++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c @@ -1286,13 +1286,16 @@ _mlx5_os_read_dev_counters(struct rte_eth_dev *dev, int pf, uint64_t *stats) struct mlx5_xstats_ctrl *xstats_ctrl = &priv->xstats_ctrl; unsigned int i; struct ifreq ifr; - unsigned int stats_sz = xstats_ctrl->stats_n * sizeof(uint64_t); + unsigned int max_stats_n = RTE_MAX(xstats_ctrl->stats_n, xstats_ctrl->stats_n_2nd); + unsigned int stats_sz = max_stats_n * sizeof(uint64_t); unsigned char et_stat_buf[sizeof(struct ethtool_stats) + stats_sz]; struct ethtool_stats *et_stats = (struct ethtool_stats *)et_stat_buf; int ret; + uint16_t i_idx, o_idx; et_stats->cmd = ETHTOOL_GSTATS; - et_stats->n_stats = xstats_ctrl->stats_n; + /* Pass the maximum value, the driver may ignore this. */ + et_stats->n_stats = max_stats_n; ifr.ifr_data = (caddr_t)et_stats; if (pf >= 0) ret = mlx5_ifreq_by_ifname(priv->sh->bond.ports[pf].ifname, @@ -1305,21 +1308,34 @@ _mlx5_os_read_dev_counters(struct rte_eth_dev *dev, int pf, uint64_t *stats) dev->data->port_id); return ret; } - for (i = 0; i != xstats_ctrl->mlx5_stats_n; ++i) { - if (xstats_ctrl->info[i].dev) - continue; - stats[i] += (uint64_t) - et_stats->data[xstats_ctrl->dev_table_idx[i]]; + if (pf <= 0) { + for (i = 0; i != xstats_ctrl->mlx5_stats_n; i++) { + i_idx = xstats_ctrl->dev_table_idx[i]; + if (i_idx == UINT16_MAX || xstats_ctrl->info[i].dev) + continue; + o_idx = xstats_ctrl->xstats_o_idx[i]; + stats[o_idx] += (uint64_t)et_stats->data[i_idx]; + } + } else { + for (i = 0; i != xstats_ctrl->mlx5_stats_n; i++) { + i_idx = xstats_ctrl->dev_table_idx_2nd[i]; + if (i_idx == UINT16_MAX) + continue; + o_idx = xstats_ctrl->xstats_o_idx_2nd[i]; + stats[o_idx] += (uint64_t)et_stats->data[i_idx]; + } } return 0; } -/** +/* * Read device counters. * * @param dev * Pointer to Ethernet device. - * @param[out] stats + * @param bond_master + * Indicate if the device is a bond master. + * @param stats * Counters table output buffer. * * @return @@ -1327,7 +1343,7 @@ _mlx5_os_read_dev_counters(struct rte_eth_dev *dev, int pf, uint64_t *stats) * rte_errno is set. */ int -mlx5_os_read_dev_counters(struct rte_eth_dev *dev, uint64_t *stats) +mlx5_os_read_dev_counters(struct rte_eth_dev *dev, bool bond_master, uint64_t *stats) { struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_xstats_ctrl *xstats_ctrl = &priv->xstats_ctrl; @@ -1335,7 +1351,7 @@ mlx5_os_read_dev_counters(struct rte_eth_dev *dev, uint64_t *stats) memset(stats, 0, sizeof(*stats) * xstats_ctrl->mlx5_stats_n); /* Read ifreq counters. */ - if (priv->master && priv->pf_bond >= 0) { + if (bond_master) { /* Sum xstats from bonding device member ports. */ for (i = 0; i < priv->sh->bond.n_port; i++) { ret = _mlx5_os_read_dev_counters(dev, i, stats); @@ -1347,13 +1363,17 @@ mlx5_os_read_dev_counters(struct rte_eth_dev *dev, uint64_t *stats) if (ret) return ret; } - /* Read IB counters. */ - for (i = 0; i != xstats_ctrl->mlx5_stats_n; ++i) { + /* + * Read IB counters. + * The counters are unique per IB device but not per net IF. + * In bonding mode, getting the stats name only from 1 port is enough. + */ + for (i = 0; i != xstats_ctrl->mlx5_stats_n; i++) { if (!xstats_ctrl->info[i].dev) continue; /* return last xstats counter if fail to read. */ if (mlx5_os_read_dev_stat(priv, xstats_ctrl->info[i].ctr_name, - &stats[i]) == 0) + &stats[i]) == 0) xstats_ctrl->xstats[i] = stats[i]; else stats[i] = xstats_ctrl->xstats[i]; @@ -1361,18 +1381,24 @@ mlx5_os_read_dev_counters(struct rte_eth_dev *dev, uint64_t *stats) return ret; } -/** +/* * Query the number of statistics provided by ETHTOOL. * * @param dev * Pointer to Ethernet device. + * @param bond_master + * Indicate if the device is a bond master. + * @param n_stats + * Pointer to number of stats to store. + * @param n_stats_sec + * Pointer to number of stats to store for the 2nd port of the bond. * * @return - * Number of statistics on success, negative errno value otherwise and - * rte_errno is set. + * 0 on success, negative errno value otherwise and rte_errno is set. */ int -mlx5_os_get_stats_n(struct rte_eth_dev *dev) +mlx5_os_get_stats_n(struct rte_eth_dev *dev, bool bond_master, + uint16_t *n_stats, uint16_t *n_stats_sec) { struct mlx5_priv *priv = dev->data->dev_private; struct ethtool_drvinfo drvinfo; @@ -1381,18 +1407,34 @@ mlx5_os_get_stats_n(struct rte_eth_dev *dev) drvinfo.cmd = ETHTOOL_GDRVINFO; ifr.ifr_data = (caddr_t)&drvinfo; - if (priv->master && priv->pf_bond >= 0) - /* Bonding PF. */ + /* Bonding PFs. */ + if (bond_master) { ret = mlx5_ifreq_by_ifname(priv->sh->bond.ports[0].ifname, SIOCETHTOOL, &ifr); - else + if (ret) { + DRV_LOG(WARNING, "bonding port %u unable to query number of" + " statistics for the 1st slave, %d", PORT_ID(priv), ret); + return ret; + } + *n_stats = drvinfo.n_stats; + ret = mlx5_ifreq_by_ifname(priv->sh->bond.ports[1].ifname, + SIOCETHTOOL, &ifr); + if (ret) { + DRV_LOG(WARNING, "bonding port %u unable to query number of" + " statistics for the 2nd slave, %d", PORT_ID(priv), ret); + return ret; + } + *n_stats_sec = drvinfo.n_stats; + } else { ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); - if (ret) { - DRV_LOG(WARNING, "port %u unable to query number of statistics", - dev->data->port_id); - return ret; + if (ret) { + DRV_LOG(WARNING, "port %u unable to query number of statistics", + PORT_ID(priv)); + return ret; + } + *n_stats = drvinfo.n_stats; } - return drvinfo.n_stats; + return 0; } static const struct mlx5_counter_ctrl mlx5_counters_init[] = { @@ -1578,6 +1620,101 @@ static const struct mlx5_counter_ctrl mlx5_counters_init[] = { static const unsigned int xstats_n = RTE_DIM(mlx5_counters_init); +static int +mlx5_os_get_stats_strings(struct rte_eth_dev *dev, bool bond_master, + struct ethtool_gstrings *strings, + uint32_t stats_n, uint32_t stats_n_2nd) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_xstats_ctrl *xstats_ctrl = &priv->xstats_ctrl; + struct ifreq ifr; + int ret; + uint32_t i, j, idx; + + /* Ensure no out of bounds access before. */ + MLX5_ASSERT(xstats_n <= MLX5_MAX_XSTATS); + strings->cmd = ETHTOOL_GSTRINGS; + strings->string_set = ETH_SS_STATS; + strings->len = stats_n; + ifr.ifr_data = (caddr_t)strings; + if (bond_master) + ret = mlx5_ifreq_by_ifname(priv->sh->bond.ports[0].ifname, + SIOCETHTOOL, &ifr); + else + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + if (ret) { + DRV_LOG(WARNING, "port %u unable to get statistic names with %d", + PORT_ID(priv), ret); + return ret; + } + /* Reorganize the orders to reduce the iterations. */ + for (j = 0; j < xstats_n; j++) { + xstats_ctrl->dev_table_idx[j] = UINT16_MAX; + for (i = 0; i < stats_n; i++) { + const char *curr_string = + (const char *)&strings->data[i * ETH_GSTRING_LEN]; + + if (!strcmp(mlx5_counters_init[j].ctr_name, curr_string)) { + idx = xstats_ctrl->mlx5_stats_n++; + xstats_ctrl->dev_table_idx[j] = i; + xstats_ctrl->xstats_o_idx[j] = idx; + xstats_ctrl->info[idx] = mlx5_counters_init[j]; + } + } + } + if (!bond_master) { + /* Add dev counters, unique per IB device. */ + for (j = 0; j != xstats_n; j++) { + if (mlx5_counters_init[j].dev) { + idx = xstats_ctrl->mlx5_stats_n++; + xstats_ctrl->info[idx] = mlx5_counters_init[j]; + xstats_ctrl->hw_stats[idx] = 0; + } + } + return 0; + } + + strings->len = stats_n_2nd; + ret = mlx5_ifreq_by_ifname(priv->sh->bond.ports[1].ifname, + SIOCETHTOOL, &ifr); + if (ret) { + DRV_LOG(WARNING, "port %u unable to get statistic names for 2nd slave with %d", + PORT_ID(priv), ret); + return ret; + } + /* The 2nd slave port may have a different strings set, based on the configuration. */ + for (j = 0; j != xstats_n; j++) { + xstats_ctrl->dev_table_idx_2nd[j] = UINT16_MAX; + for (i = 0; i != stats_n_2nd; i++) { + const char *curr_string = + (const char *)&strings->data[i * ETH_GSTRING_LEN]; + + if (!strcmp(mlx5_counters_init[j].ctr_name, curr_string)) { + xstats_ctrl->dev_table_idx_2nd[j] = i; + if (xstats_ctrl->dev_table_idx[j] != UINT16_MAX) { + /* Already mapped in the 1st slave port. */ + idx = xstats_ctrl->xstats_o_idx[j]; + xstats_ctrl->xstats_o_idx_2nd[j] = idx; + } else { + /* Append the new items to the end of the map. */ + idx = xstats_ctrl->mlx5_stats_n++; + xstats_ctrl->xstats_o_idx_2nd[j] = idx; + xstats_ctrl->info[idx] = mlx5_counters_init[j]; + } + } + } + } + /* Dev counters are always at the last now. */ + for (j = 0; j != xstats_n; j++) { + if (mlx5_counters_init[j].dev) { + idx = xstats_ctrl->mlx5_stats_n++; + xstats_ctrl->info[idx] = mlx5_counters_init[j]; + xstats_ctrl->hw_stats[idx] = 0; + } + } + return 0; +} + /** * Init the structures to read device counters. * @@ -1590,76 +1727,44 @@ mlx5_os_stats_init(struct rte_eth_dev *dev) struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_xstats_ctrl *xstats_ctrl = &priv->xstats_ctrl; struct mlx5_stats_ctrl *stats_ctrl = &priv->stats_ctrl; - unsigned int i; - unsigned int j; - struct ifreq ifr; struct ethtool_gstrings *strings = NULL; - unsigned int dev_stats_n; + uint16_t dev_stats_n = 0; + uint16_t dev_stats_n_2nd = 0; + unsigned int max_stats_n; unsigned int str_sz; int ret; + bool bond_master = (priv->master && priv->pf_bond >= 0); /* So that it won't aggregate for each init. */ xstats_ctrl->mlx5_stats_n = 0; - ret = mlx5_os_get_stats_n(dev); + ret = mlx5_os_get_stats_n(dev, bond_master, &dev_stats_n, &dev_stats_n_2nd); if (ret < 0) { DRV_LOG(WARNING, "port %u no extended statistics available", dev->data->port_id); return; } - dev_stats_n = ret; + max_stats_n = RTE_MAX(dev_stats_n, dev_stats_n_2nd); /* Allocate memory to grab stat names and values. */ - str_sz = dev_stats_n * ETH_GSTRING_LEN; + str_sz = max_stats_n * ETH_GSTRING_LEN; strings = (struct ethtool_gstrings *) mlx5_malloc(0, str_sz + sizeof(struct ethtool_gstrings), 0, SOCKET_ID_ANY); if (!strings) { DRV_LOG(WARNING, "port %u unable to allocate memory for xstats", - dev->data->port_id); + dev->data->port_id); return; } - strings->cmd = ETHTOOL_GSTRINGS; - strings->string_set = ETH_SS_STATS; - strings->len = dev_stats_n; - ifr.ifr_data = (caddr_t)strings; - if (priv->master && priv->pf_bond >= 0) - /* Bonding master. */ - ret = mlx5_ifreq_by_ifname(priv->sh->bond.ports[0].ifname, - SIOCETHTOOL, &ifr); - else - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); - if (ret) { - DRV_LOG(WARNING, "port %u unable to get statistic names", + ret = mlx5_os_get_stats_strings(dev, bond_master, strings, + dev_stats_n, dev_stats_n_2nd); + if (ret < 0) { + DRV_LOG(WARNING, "port %u failed to get the stats strings", dev->data->port_id); goto free; } - for (i = 0; i != dev_stats_n; ++i) { - const char *curr_string = (const char *) - &strings->data[i * ETH_GSTRING_LEN]; - - for (j = 0; j != xstats_n; ++j) { - if (!strcmp(mlx5_counters_init[j].ctr_name, - curr_string)) { - unsigned int idx = xstats_ctrl->mlx5_stats_n++; - - xstats_ctrl->dev_table_idx[idx] = i; - xstats_ctrl->info[idx] = mlx5_counters_init[j]; - break; - } - } - } - /* Add dev counters. */ - MLX5_ASSERT(xstats_ctrl->mlx5_stats_n <= MLX5_MAX_XSTATS); - for (i = 0; i != xstats_n; ++i) { - if (mlx5_counters_init[i].dev) { - unsigned int idx = xstats_ctrl->mlx5_stats_n++; - - xstats_ctrl->info[idx] = mlx5_counters_init[i]; - xstats_ctrl->hw_stats[idx] = 0; - } - } xstats_ctrl->stats_n = dev_stats_n; + xstats_ctrl->stats_n_2nd = dev_stats_n_2nd; /* Copy to base at first time. */ - ret = mlx5_os_read_dev_counters(dev, xstats_ctrl->base); + ret = mlx5_os_read_dev_counters(dev, bond_master, xstats_ctrl->base); if (ret) DRV_LOG(ERR, "port %u cannot read device counters: %s", dev->data->port_id, strerror(rte_errno)); diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 263ebead7f..153374802a 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -263,14 +263,22 @@ struct mlx5_counter_ctrl { struct mlx5_xstats_ctrl { /* Number of device stats. */ uint16_t stats_n; + /* Number of device stats, for the 2nd port in bond. */ + uint16_t stats_n_2nd; /* Number of device stats identified by PMD. */ - uint16_t mlx5_stats_n; + uint16_t mlx5_stats_n; /* Index in the device counters table. */ uint16_t dev_table_idx[MLX5_MAX_XSTATS]; + /* Index in the output table. */ + uint16_t xstats_o_idx[MLX5_MAX_XSTATS]; uint64_t base[MLX5_MAX_XSTATS]; uint64_t xstats[MLX5_MAX_XSTATS]; uint64_t hw_stats[MLX5_MAX_XSTATS]; struct mlx5_counter_ctrl info[MLX5_MAX_XSTATS]; + /* Index in the device counters table, for the 2nd port in bond. */ + uint16_t dev_table_idx_2nd[MLX5_MAX_XSTATS]; + /* Index in the output table, for the 2nd port in bond. */ + uint16_t xstats_o_idx_2nd[MLX5_MAX_XSTATS]; }; struct mlx5_stats_ctrl { @@ -2131,8 +2139,9 @@ int mlx5_get_module_eeprom(struct rte_eth_dev *dev, struct rte_dev_eeprom_info *info); int mlx5_os_read_dev_stat(struct mlx5_priv *priv, const char *ctr_name, uint64_t *stat); -int mlx5_os_read_dev_counters(struct rte_eth_dev *dev, uint64_t *stats); -int mlx5_os_get_stats_n(struct rte_eth_dev *dev); +int mlx5_os_read_dev_counters(struct rte_eth_dev *dev, bool bond_master, uint64_t *stats); +int mlx5_os_get_stats_n(struct rte_eth_dev *dev, bool bond_master, + uint16_t *n_stats, uint16_t *n_stats_sec); void mlx5_os_stats_init(struct rte_eth_dev *dev); int mlx5_get_flag_dropless_rq(struct rte_eth_dev *dev); diff --git a/drivers/net/mlx5/mlx5_stats.c b/drivers/net/mlx5/mlx5_stats.c index 615e1d073d..f4ac58e2f9 100644 --- a/drivers/net/mlx5/mlx5_stats.c +++ b/drivers/net/mlx5/mlx5_stats.c @@ -39,24 +39,36 @@ mlx5_xstats_get(struct rte_eth_dev *dev, struct rte_eth_xstat *stats, unsigned int n) { struct mlx5_priv *priv = dev->data->dev_private; - unsigned int i; - uint64_t counters[n]; + uint64_t counters[MLX5_MAX_XSTATS]; struct mlx5_xstats_ctrl *xstats_ctrl = &priv->xstats_ctrl; + unsigned int i; + uint16_t stats_n = 0; + uint16_t stats_n_2nd = 0; uint16_t mlx5_stats_n = xstats_ctrl->mlx5_stats_n; + bool bond_master = (priv->master && priv->pf_bond >= 0); if (n >= mlx5_stats_n && stats) { - int stats_n; int ret; - stats_n = mlx5_os_get_stats_n(dev); - if (stats_n < 0) - return stats_n; - if (xstats_ctrl->stats_n != stats_n) + ret = mlx5_os_get_stats_n(dev, bond_master, &stats_n, &stats_n_2nd); + if (ret < 0) + return ret; + /* + * The number of statistics fetched via "ETH_SS_STATS" may vary because + * of the port configuration each time. This is also true between 2 + * ports. There might be a case that the numbers are the same even if + * configurations are different. + * It is not recommended to change the configuration without using + * RTE API. The port(traffic) restart may trigger another initialization + * to make sure the map are correct. + */ + if (xstats_ctrl->stats_n != stats_n || + (bond_master && xstats_ctrl->stats_n_2nd != stats_n_2nd)) mlx5_os_stats_init(dev); - ret = mlx5_os_read_dev_counters(dev, counters); - if (ret) + ret = mlx5_os_read_dev_counters(dev, bond_master, counters); + if (ret < 0) return ret; - for (i = 0; i != mlx5_stats_n; ++i) { + for (i = 0; i != mlx5_stats_n; i++) { stats[i].id = i; if (xstats_ctrl->info[i].dev) { uint64_t wrap_n; @@ -225,30 +237,32 @@ mlx5_xstats_reset(struct rte_eth_dev *dev) { struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_xstats_ctrl *xstats_ctrl = &priv->xstats_ctrl; - int stats_n; unsigned int i; uint64_t *counters; int ret; + uint16_t stats_n = 0; + uint16_t stats_n_2nd = 0; + bool bond_master = (priv->master && priv->pf_bond >= 0); - stats_n = mlx5_os_get_stats_n(dev); - if (stats_n < 0) { + ret = mlx5_os_get_stats_n(dev, bond_master, &stats_n, &stats_n_2nd); + if (ret < 0) { DRV_LOG(ERR, "port %u cannot get stats: %s", dev->data->port_id, - strerror(-stats_n)); - return stats_n; + strerror(-ret)); + return ret; } - if (xstats_ctrl->stats_n != stats_n) + if (xstats_ctrl->stats_n != stats_n || + (bond_master && xstats_ctrl->stats_n_2nd != stats_n_2nd)) mlx5_os_stats_init(dev); - counters = mlx5_malloc(MLX5_MEM_SYS, sizeof(*counters) * - xstats_ctrl->mlx5_stats_n, 0, - SOCKET_ID_ANY); + /* Considering to use stack directly. */ + counters = mlx5_malloc(MLX5_MEM_SYS, sizeof(*counters) * xstats_ctrl->mlx5_stats_n, + 0, SOCKET_ID_ANY); if (!counters) { - DRV_LOG(WARNING, "port %u unable to allocate memory for xstats " - "counters", + DRV_LOG(WARNING, "port %u unable to allocate memory for xstats counters", dev->data->port_id); rte_errno = ENOMEM; return -rte_errno; } - ret = mlx5_os_read_dev_counters(dev, counters); + ret = mlx5_os_read_dev_counters(dev, bond_master, counters); if (ret) { DRV_LOG(ERR, "port %u cannot read device counters: %s", dev->data->port_id, strerror(rte_errno)); diff --git a/drivers/net/mlx5/windows/mlx5_ethdev_os.c b/drivers/net/mlx5/windows/mlx5_ethdev_os.c index a31e1b5494..49f750be68 100644 --- a/drivers/net/mlx5/windows/mlx5_ethdev_os.c +++ b/drivers/net/mlx5/windows/mlx5_ethdev_os.c @@ -178,20 +178,29 @@ mlx5_dev_set_flow_ctrl(struct rte_eth_dev *dev, struct rte_eth_fc_conf *fc_conf) return -ENOTSUP; } -/** +/* * Query the number of statistics provided by ETHTOOL. * * @param dev * Pointer to Ethernet device. + * @param bond_master + * Indicate if the device is a bond master. + * @param n_stats + * Pointer to number of stats to store. + * @param n_stats_sec + * Pointer to number of stats to store for the 2nd port of the bond. * * @return - * Number of statistics on success, negative errno value otherwise and - * rte_errno is set. + * 0 on success, negative errno value otherwise and rte_errno is set. */ int -mlx5_os_get_stats_n(struct rte_eth_dev *dev) +mlx5_os_get_stats_n(struct rte_eth_dev *dev, bool bond_master, + uint16_t *n_stats, uint16_t *n_stats_sec) { RTE_SET_USED(dev); + RTE_SET_USED(bond_master); + RTE_SET_USED(n_stats); + RTE_SET_USED(n_stats_sec); return -ENOTSUP; } @@ -221,6 +230,8 @@ mlx5_os_stats_init(struct rte_eth_dev *dev) * * @param dev * Pointer to Ethernet device. + * @param bond_master + * Indicate if the device is a bond master. * @param[out] stats * Counters table output buffer. * @@ -229,9 +240,10 @@ mlx5_os_stats_init(struct rte_eth_dev *dev) * rte_errno is set. */ int -mlx5_os_read_dev_counters(struct rte_eth_dev *dev, uint64_t *stats) +mlx5_os_read_dev_counters(struct rte_eth_dev *dev, bool bond_master, uint64_t *stats) { RTE_SET_USED(dev); + RTE_SET_USED(bond_master); RTE_SET_USED(stats); return -ENOTSUP; } -- 2.34.1 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2024-04-13 20:43:06.901405245 +0800 +++ 0060-net-mlx5-fix-counters-map-in-bonding-mode.patch 2024-04-13 20:43:05.007753918 +0800 @@ -1 +1 @@ -From a687c3e658c2d889052089af8340bc0b9299c856 Mon Sep 17 00:00:00 2001 +From 0c31d1220ffaff4742154c3c957ab5305e5f5c3a Mon Sep 17 00:00:00 2001 @@ -4,0 +5,3 @@ +Cc: Xueming Li + +[ upstream commit a687c3e658c2d889052089af8340bc0b9299c856 ] @@ -50 +52,0 @@ -Cc: stable@dpdk.org @@ -62 +64 @@ -index 92c47a3b3d..eb47c284ec 100644 +index dd5a0c546d..0ee8c58ba7 100644 @@ -237 +239 @@ -@@ -1615,6 +1657,101 @@ static const struct mlx5_counter_ctrl mlx5_counters_init[] = { +@@ -1578,6 +1620,101 @@ static const struct mlx5_counter_ctrl mlx5_counters_init[] = { @@ -339 +341 @@ -@@ -1627,76 +1764,44 @@ mlx5_os_stats_init(struct rte_eth_dev *dev) +@@ -1590,76 +1727,44 @@ mlx5_os_stats_init(struct rte_eth_dev *dev) @@ -431 +433 @@ -index f11a0181b8..fb3df76cac 100644 +index 263ebead7f..153374802a 100644 @@ -458 +460 @@ -@@ -2182,8 +2190,9 @@ int mlx5_get_module_eeprom(struct rte_eth_dev *dev, +@@ -2131,8 +2139,9 @@ int mlx5_get_module_eeprom(struct rte_eth_dev *dev,