From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 218EA45AF1; Wed, 9 Oct 2024 13:41:31 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0B06F4279F; Wed, 9 Oct 2024 13:41:19 +0200 (CEST) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2082.outbound.protection.outlook.com [40.107.220.82]) by mails.dpdk.org (Postfix) with ESMTP id 28FAA42796 for ; Wed, 9 Oct 2024 13:41:18 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=b6opb6svltoOacH7znI8i5HbhA+QF/23dRr7SJCYyPsGAT9OtH9aS2MPBaAXPkyM9zvYSbi5nsJXDBrww7W73wku4HZouNWJ20fvELulHYkzWZgfSgMj8mPQBz4dHssPDkQGnK7T5fFlsunlV6nUhVmQ5BZzuPd0uyAw2ErsPBzejaQEU8QA1l9d948p1X8fcyadUeIwYFp7VAaoM2dEHQp1eFytn6En+fr+koTJzaJQOuh/aYGMwcuL+nwayYhcYiednOUt70HZjCBqULlH8+l+VbltnqDRatGIJfMU/Wj15mBtWk6M8wlSWSzOaRKm40S34QQ46Sm7j0fVKcEcfA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=sXRokpnVGUGt40cBVHsH7CzHrrjfRlUAOTTiyURARq8=; b=NNEUF8nSLb7MX47twXFTwuxxflRz50yvzy/cRmM3FoAD8wBg66JOjAZfJrjpBXpv8m4iSzSDlXcbxYeKCJAQOvXOHnj5GtYQdwIzbMkcIOAl+DQjf2XzoSeQzMJXIdYwJniFIsVeOpyEbDY16ImDyJd2ahtgKrOZFYykabRluFtXX+b8EWCYW1W5v8fsfRnkpH1hlPFbgU3ke4skukWv2XsOTPStqYOF3AZ2WuxiQ2qnbLA2T3Hs5thFV0g2Oj27hQ6dTrdvSvJxySHeoY0mXwgEtLRNfiihAmrxPtBdM0q/ijsvvdM8NKFRk6aHTMWr0ucH+Irz7xQZ5E1T5+JpvQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sXRokpnVGUGt40cBVHsH7CzHrrjfRlUAOTTiyURARq8=; b=mAAFe74THHjrfir8h9j+8DdzE/HxapySfSI3L7KPi/zuk3zwhqMIbdJ9O3bTW0Dff5/fmAHqt92qiyfv3RGG9NPcj9oRITB8M2P7Naa3ZIMQWM6n8eDqzaKIF1aYhZQYYUoya4/v8qUmU13mYn3G/0iHA7GRQkTYbvJ/k3P0x4WC0ElSqINx40fiHo+w4A3pSJYz2yLWaJRcV5MtySAiPyGNP4v4lKpIFSI7r6+gWzgpAfcNrDAfd6VvuXNHzBgEWxfe+8tTKgDhrrPgRZ0x8RZMSYZaQIH91+CfeniZUWvQZZBWOHziUKp5wb5P3tAB/Sip5Mi57z7l9IoLNMUWIQ== Received: from CH5PR04CA0018.namprd04.prod.outlook.com (2603:10b6:610:1f4::29) by PH8PR12MB6700.namprd12.prod.outlook.com (2603:10b6:510:1cf::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8026.22; Wed, 9 Oct 2024 11:41:12 +0000 Received: from DS3PEPF000099DF.namprd04.prod.outlook.com (2603:10b6:610:1f4:cafe::73) by CH5PR04CA0018.outlook.office365.com (2603:10b6:610:1f4::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.17 via Frontend Transport; Wed, 9 Oct 2024 11:41:11 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by DS3PEPF000099DF.mail.protection.outlook.com (10.167.17.202) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.13 via Frontend Transport; Wed, 9 Oct 2024 11:41:11 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Wed, 9 Oct 2024 04:41:01 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Wed, 9 Oct 2024 04:40:59 -0700 From: Viacheslav Ovsiienko To: CC: , , , Subject: [PATCH 4/4] net/mlx5: update dump script to show incomplete records Date: Wed, 9 Oct 2024 14:40:28 +0300 Message-ID: <20241009114028.973284-5-viacheslavo@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241009114028.973284-1-viacheslavo@nvidia.com> References: <20241009114028.973284-1-viacheslavo@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099DF:EE_|PH8PR12MB6700:EE_ X-MS-Office365-Filtering-Correlation-Id: 8fdd71cf-ae70-4e99-09c6-08dce8574643 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|1800799024|36860700013|376014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?MlOHwlRwdrON9Jsv9nbc8j0VkmhDJRggcaxZmefJ270kpQziPnrX75xnjlI+?= =?us-ascii?Q?tE6k31xycT2PbAvJ+yMS9STlabRqxDc2YxxsITYYKxFB5Rwa0XIyGRo2Dy6P?= =?us-ascii?Q?LdykhIDXc0iFYEK1edYb4xIWobnSv0bbE9Mk7Rp2XCEH+qRuWB/39lY/nyte?= =?us-ascii?Q?QjZBadmf+wt4e7y/8hlVq4rRvvU96OuKLfg6gu/n7QsJMwxSWZqMgOM/olpd?= =?us-ascii?Q?fav1BBUUpSbcI5hRIiwRnXL/wNK98D629QnR4pmfvLTSN5UN5mfX7v1Xfm66?= =?us-ascii?Q?3BjRcR0vdo72mEpMvMVzdlZEMX771sIMwqaOh/4RZyozAs/vn1Eq6hCBQ6Sb?= =?us-ascii?Q?XMy4+5F7iryIQF4I7aXxQDTe0soAAWlPTLb4cr1Zp9zDdDuKkTvcODCeZQv+?= =?us-ascii?Q?aQh7p01EQ8iO3GX/pvc5WmIBoQ998+C1sxKKJIirYjal3Ob4TcQarBlm+pGE?= =?us-ascii?Q?4B+oAiW5iepakw3Ceik3qlCy1PVCAF3wK4x2iktqVHNvWLxQaxe7ltSu1W7Z?= =?us-ascii?Q?c/zjs0qE2jZgiOAPHScIXKUs8m9xcUAVlz3r/Hwk1uF6vsadlRy+S1X248wA?= =?us-ascii?Q?gorb9aKJGgZH6urUrO7o09atc6ZZIEjpNxR+RNRTrwLuI2VJLtsKqK8Ol/3Q?= =?us-ascii?Q?rag4E9gV3VSPsjFf0QoInGrIhv1AOQ2D/4BgwzTILc4LHF2ZPhpUoS5l+lU9?= =?us-ascii?Q?h3rzd+wkEW2ZazKpvzmkuam4skfUKR04MnSYlckhx/y9KNzu/rgrPFl//cZP?= =?us-ascii?Q?P2nJ//HkV8y/LqHu1znDzFsAofnqJnTXAR+EfxCU4Sb3uTU6h8zpcXSgxXHL?= =?us-ascii?Q?X+lZR91Eryg+PzuXCinWkj3CwtZByoCFdiI6YD5BS0uc9Hd5QCuBvl34XWLn?= =?us-ascii?Q?5cWF608YmKpanMTFKs0NcaNJA7LXn0PZHFln6NCunlozQh3CiWK4Sl0IbaoM?= =?us-ascii?Q?qu8P5Rdx6UdsKxu/lyPapQE8Mh8OIuj6oc8lwhNeRDHCItxGeet/pTW7vO+2?= =?us-ascii?Q?zciJd6j/VL86AJTaqV4jS8tVZAqqGoDfYYTbMAW6ulgfNNuzLccx+TltLP6z?= =?us-ascii?Q?jCY3mLfoDXU96sW1TxttolSjPZSXttMT8qXTi7LeuBd3iEnEPk4EYDkTnrVo?= =?us-ascii?Q?JDkALXiqAtQecvPqPE9MsfulaRxAJZEvcxoYYbhgJRsfXxr6IZXAyuKgii7i?= =?us-ascii?Q?LOTrymOj+Nx5iWyh8ho/Rf9b/S+QSAqownjnC4Yqqr4iabGmv7UEyeCN8MAW?= =?us-ascii?Q?8nT/OzAmmXamXW3NfBts1mbaIa/P4yPpmIm5HZ5FOjSscPaKIPoSd/Dhe1PN?= =?us-ascii?Q?/Zq5Nio5zUPig9fga48ubA0mxJMd9iehRY/NPB2RW8DvA5Y4npdA/OEYH5az?= =?us-ascii?Q?verelXJ4T+lB5R81BpaWgDS9oMg+?= X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230040)(1800799024)(36860700013)(376014)(82310400026); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2024 11:41:11.6958 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8fdd71cf-ae70-4e99-09c6-08dce8574643 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099DF.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB6700 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org If the trace dump is stored at the moment when there is some incomplete Tx transfers - WQE is pushed but hardware did not sent the completions yet - this incomplete was not dumped by the script mlx5_trace. For some cases (for example, if queue was stuck) the valuable debug information was lost. To improve the dump fullness the following optional script arguments are added: -v [level] - provides the raw dump of the object record of the specified level (0 - bursts, 1 - WQEs, 2+ - mbufs) -a - dumps all bursts, including incomplete ones Signed-off-by: Viacheslav Ovsiienko --- drivers/net/mlx5/tools/mlx5_trace.py | 82 ++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 23 deletions(-) --- doc/guides/nics/mlx5.rst | 6 +++ drivers/net/mlx5/tools/mlx5_trace.py | 73 ++++++++++++++++++++-------- 2 files changed, 59 insertions(+), 20 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 1dccdaad50..f82e2d75de 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -2360,6 +2360,12 @@ Steps to enable Tx datapath tracing: The parameter of the script is the trace data folder. + The optional parameter ``-a`` forces to dump incomplete bursts. + + The optional parameter ``-v [level]`` forces to dump raw records data + for the specified level and below. Level 0 dumps bursts, level 1 dumps WQEs, + level 2 dumps mbufs. + .. code-block:: console mlx5_trace.py /var/log/rte-2023-01-23-AM-11-52-39 diff --git a/drivers/net/mlx5/tools/mlx5_trace.py b/drivers/net/mlx5/tools/mlx5_trace.py index 5eb634a490..96eb82082f 100755 --- a/drivers/net/mlx5/tools/mlx5_trace.py +++ b/drivers/net/mlx5/tools/mlx5_trace.py @@ -21,10 +21,13 @@ def __init__(self): self.wait_burst = [] # waiting for completion self.pq_id = 0 - def log(self): + def log(self, all): """Log all queue bursts""" for txb in self.done_burst: txb.log() + if all == True: + for txb in self.wait_burst: + txb.log() class MlxMbuf: @@ -147,24 +150,26 @@ def __init__(self): self.tx_qlst = {} # active Tx queues per port/queue self.tx_wlst = {} # wait timestamp list per CPU - def run(self, msg_it): + def run(self, msg_it, verbose): """Run over gathered tracing data and build database""" for msg in msg_it: if not isinstance(msg, bt2._EventMessageConst): continue event = msg.event if event.name.startswith(PFX_TX): - do_tx(msg, self) + do_tx(msg, self, verbose) # Handling of other log event cathegories can be added here + if verbose: + print("*** End of raw data dump ***") - def log(self): + def log(self, all): """Log gathered trace database""" for pq_id in self.tx_qlst: queue = self.tx_qlst.get(pq_id) - queue.log() + queue.log(all) -def do_tx_entry(msg, trace): +def do_tx_entry(msg, trace, verbose): """Handle entry Tx busrt""" event = msg.event cpu_id = event["cpu_id"] @@ -172,6 +177,10 @@ def do_tx_entry(msg, trace): if burst is not None: # continue existing burst after WAIT return + if verbose > 0: + print("%u:%X tx_entry(real_time=%u, port_id=%u, queue_id=%u)" % + (msg.default_clock_snapshot.ns_from_origin, cpu_id, + event["real_time"], event["port_id"], event["queue_id"])) # allocate the new burst and append to the queue burst = MlxBurst() burst.call_ts = event["real_time"] @@ -189,10 +198,14 @@ def do_tx_entry(msg, trace): queue.wait_burst.append(burst) -def do_tx_exit(msg, trace): +def do_tx_exit(msg, trace, verbose): """Handle exit Tx busrt""" event = msg.event cpu_id = event["cpu_id"] + if verbose > 0: + print("%u:%X tx_exit(real_time=%u, nb_sent=%u, nb_req=%u)" % + (msg.default_clock_snapshot.ns_from_origin, cpu_id, + event["real_time"], event["nb_sent"], event["nb_req"])) burst = trace.tx_blst.get(cpu_id) if burst is None: return @@ -204,10 +217,14 @@ def do_tx_exit(msg, trace): trace.tx_blst.pop(cpu_id) -def do_tx_wqe(msg, trace): +def do_tx_wqe(msg, trace, verbose): """Handle WQE record""" event = msg.event cpu_id = event["cpu_id"] + if verbose > 1: + print("%u:%X tx_wqe(real_time=%u, opcode=%08X)" % + (msg.default_clock_snapshot.ns_from_origin, cpu_id, + event["real_time"], event["opcode"])) burst = trace.tx_blst.get(cpu_id) if burst is None: return @@ -221,17 +238,24 @@ def do_tx_wqe(msg, trace): burst.wqes.append(wqe) -def do_tx_wait(msg, trace): +def do_tx_wait(msg, trace, verbose): """Handle WAIT record""" event = msg.event cpu_id = event["cpu_id"] + if verbose > 1: + print("%u:%X tx_wait(ts=%u)" % + (msg.default_clock_snapshot.ns_from_origin, cpu_id, event["ts"])) trace.tx_wlst[cpu_id] = event["ts"] -def do_tx_push(msg, trace): +def do_tx_push(msg, trace, verbose): """Handle WQE push event""" event = msg.event cpu_id = event["cpu_id"] + if verbose > 2: + print("%u:%X tx_push(mbuf=%X, pkt_len=%u, nb_segs=%u, wqe_id=%04X)" % + (msg.default_clock_snapshot.ns_from_origin, cpu_id, event["mbuf"], + event["mbuf_pkt_len"], event["mbuf_nb_segs"], event["wqe_id"])) burst = trace.tx_blst.get(cpu_id) if burst is None: return @@ -246,10 +270,15 @@ def do_tx_push(msg, trace): wqe.mbuf.append(mbuf) -def do_tx_complete(msg, trace): +def do_tx_complete(msg, trace, verbose): """Handle send completion event""" event = msg.event pq_id = event["port_id"] << 16 | event["queue_id"] + if verbose > 1: + cpu_id = event["cpu_id"] + print("%u:%X tx_complete(port_id=%u, queue_id=%u, ts=%u, wqe_id=%04X)" % + (msg.default_clock_snapshot.ns_from_origin, cpu_id, + event["port_id"], event["queue_id"], event["ts"], event["wqe_id"])) queue = trace.tx_qlst.get(pq_id) if queue is None: return @@ -274,21 +303,21 @@ def do_tx_complete(msg, trace): queue.wait_burst = queue.wait_burst[rmv:] -def do_tx(msg, trace): +def do_tx(msg, trace, verbose): """Handle Tx related records""" name = msg.event.name[PFX_TX_LEN:] if name == "entry": - do_tx_entry(msg, trace) + do_tx_entry(msg, trace, verbose) elif name == "exit": - do_tx_exit(msg, trace) + do_tx_exit(msg, trace, verbose) elif name == "wqe": - do_tx_wqe(msg, trace) + do_tx_wqe(msg, trace, verbose) elif name == "wait": - do_tx_wait(msg, trace) + do_tx_wait(msg, trace, verbose) elif name == "push": - do_tx_push(msg, trace) + do_tx_push(msg, trace, verbose) elif name == "complete": - do_tx_complete(msg, trace) + do_tx_complete(msg, trace, verbose) else: print("Error: unrecognized Tx event name: %s" % msg.event.name, file=sys.stderr) raise ValueError() @@ -299,12 +328,16 @@ def main() -> int: try: parser = argparse.ArgumentParser() parser.add_argument("path", nargs=1, type=str, help="input trace folder") + parser.add_argument("-a", "--all", nargs="?", default=False, const=True, + help="show all the bursts, including incomplete ones") + parser.add_argument("-v", "--verbose", type=int, nargs="?", default=0, const=2, + help="show all the records below specified level") args = parser.parse_args() mlx_tr = MlxTrace() msg_it = bt2.TraceCollectionMessageIterator(args.path) - mlx_tr.run(msg_it) - mlx_tr.log() + mlx_tr.run(msg_it, args.verbose) + mlx_tr.log(args.all) return 0 except ValueError: return -1 -- 2.34.1