From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E31BB45B36; Mon, 14 Oct 2024 10:06:05 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9B10D40661; Mon, 14 Oct 2024 10:06:00 +0200 (CEST) Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam04on2075.outbound.protection.outlook.com [40.107.101.75]) by mails.dpdk.org (Postfix) with ESMTP id CFA274060F for ; Mon, 14 Oct 2024 10:05:58 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=j20NY6DtKVgSHqG9W/ONxRkqEI+OKcHOFnVSFQCjeybRBqOPWma/cJBEWwJUq2cEYwGrry+ffM3Qi1h40Ogg/iEKEX171xSuaqUKTQ8JxMOAeGyz3hHggBkP5X5Fsc8YlU+2YXhcMDCkaQx6N+VffChb9gJgdOvzmuj1BTfp9l49GGxeUrmaaDNNvEuO+2SJMCMnJfp7DOl238gWYE33On2ySH+EwG41sC42t07n949XOAv7IFfcCs1IdX6RQ1RGZiJEPyXncJA8ySfpJwAc5klXjSpRqG3zH8DagSoIIHm49M4dAQHjGXTocF977G/QPstn+lfXDpM1d9j9EAymog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=nxv2jSJwNSV6o+vJHABgQGykPkt6z47z5UE3vRm+oKs=; b=ZO0bshdsxfmMLaxyafLF6Vau4b+sUkgquCO45N+DkTz36kt/yAUv7Mb05v12ic749SQAi+sgD97G6FELNPBrqVQyI6uVPvXc4f3UeK8IiT/4eohljh3H9WPGzFPUaFeb+9W/tycWGvwPA3/mzO0EpXwqf6ke9s9RV6zvhxqGPzhScsfscNIvH40ytle4vCVjYQPXOOcw19pefPc7ALLkloX4+z/R9kKB6czCmxt44NlHTXMuT4+MhOW/pAHuYQGVSy/z/zWL+1w9O5ByiPeygZePXCVSXZ1GUMEJyc0FMn8FJRTwdxabyFfmpZ1JFB00bA3rEL+p8Ly8vqiWHHpg7A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nxv2jSJwNSV6o+vJHABgQGykPkt6z47z5UE3vRm+oKs=; b=h0Ll+BXcyKUedCvMGoRVGcWjFmo/pscVWisjfF4YlPV7P+vKKJBzTY+ES5hqUX2RxMP1ampHF4/8s0o32tR096KFdYzXHTjichXu6xAXnFN2KDdx6ShgiGYhdkti4FEjVzHwTQ6JrU9d40VDh2zWYnEid7C0NcKyKjMPAbkEKhNX9Y8PmaERieKjb1wx8ZY1IjwENcGSju2i2/JEckqJR49Onsxfwr+UnXzj2PneYTtz7lVWiiZ8c2TngGLugp14vyyQtn6nfc1vaE6w+ckWeffyIh30x87bug4Kv5YfEDbUcF1j7yTr0r2OlC1QfwtjvHBuu69NTYHUrK5i1Cup3Q== Received: from MN2PR02CA0032.namprd02.prod.outlook.com (2603:10b6:208:fc::45) by SJ2PR12MB7895.namprd12.prod.outlook.com (2603:10b6:a03:4c6::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.24; Mon, 14 Oct 2024 08:05:53 +0000 Received: from BL02EPF00021F6F.namprd02.prod.outlook.com (2603:10b6:208:fc:cafe::3a) by MN2PR02CA0032.outlook.office365.com (2603:10b6:208:fc::45) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.26 via Frontend Transport; Mon, 14 Oct 2024 08:05:52 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BL02EPF00021F6F.mail.protection.outlook.com (10.167.249.11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.13 via Frontend Transport; Mon, 14 Oct 2024 08:05:52 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Mon, 14 Oct 2024 01:05:34 -0700 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Mon, 14 Oct 2024 01:05:32 -0700 From: Viacheslav Ovsiienko To: CC: , , , Subject: [PATCH v2 4/4] net/mlx5: update dump script to show incomplete records Date: Mon, 14 Oct 2024 11:04:34 +0300 Message-ID: <20241014080434.1211629-5-viacheslavo@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241014080434.1211629-1-viacheslavo@nvidia.com> References: <20241009114028.973284-1-viacheslavo@nvidia.com> <20241014080434.1211629-1-viacheslavo@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF00021F6F:EE_|SJ2PR12MB7895:EE_ X-MS-Office365-Filtering-Correlation-Id: 4b6f6e7e-a160-4297-3a4e-08dcec2705e9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|82310400026|376014|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?7a5DWb/VaBDaKDkVuKUv8N7OUb+9OqQqibXaescXIXAIRqrsXRPsn6TlOEpm?= =?us-ascii?Q?Cnl+gSsSv/saVWeDAGKSqlxGhWs1IpCqYm39tkytstuNtNvBAzgjLR8u7MKf?= =?us-ascii?Q?hndS2gSJSowQKv2oSVk1XT07sGbYVmOFGCuPFuqOhuvbFrkxWFoHzldiUhLU?= =?us-ascii?Q?iYPclv34B/D9mRZCVLC8468TLxzjP2gut1o8UZrq3ikKuXDLaEwB0iFj9fdn?= =?us-ascii?Q?yUsTmf/iwXqTEE6DREmPCojxikp4PWuSyn3DIJRVSFE5bg/uZQKihEHMi19n?= =?us-ascii?Q?4dypSP3RTfVdG3n23Q4mHNz1u9eIdPIRI3Qg5jOKzXyH01rKyHTggG0fPO8i?= =?us-ascii?Q?qitolqdh0Mw0NO4xtDnNH3eir4sNEqJqHZrYaBnQg2qjTUOPoAPv/z1lVQJt?= =?us-ascii?Q?7p/4UeavqhBRi6iapIgG+YfyZALpcf3MSuxs8T4P2LjS/4AB+4OZQ8BfzPK/?= =?us-ascii?Q?uMADfGE9MMTFg4a/n6NCFMmC11jexj9sZnDq5NQTCeJUwnMeSc6v5JxIzMht?= =?us-ascii?Q?P3b09mPKBQSVzRywtksRV1LvgIv0rRlmHXCXD4dEhRYt4zIXjYsriKMd7OUm?= =?us-ascii?Q?O8tUvgnjnyEERpE7hJincb4XtENJqqzRqt45FsTEiVoq6LdrA3WHG92ICHes?= =?us-ascii?Q?K7dqtgJRl1wLbSb7RsQroZekZklHEMycUoXjdKZW6woKGaBWip3h4fVBZPl2?= =?us-ascii?Q?a6XXmyo61+OKkEEnFpZ45vP9ixarnuUGao5HhVeU7v2FXo3gVIn0tTzAxI15?= =?us-ascii?Q?+8OE1l1ZwjJpyCbTiPIDdp8CiF5bAVEgTA9ZWJkyvoscDMGzlctbU+plyrxd?= =?us-ascii?Q?SWFQpxMO/StB/6MUlbkjYJZ+7oiUIA8tgk59cPocfJg1uRcjkglGMEF1zX5F?= =?us-ascii?Q?UhOzZpLgn3nbHvFNzY6ZT4qUA7PVocOd5We09F/d5SmdhTol71+TD8doWAib?= =?us-ascii?Q?2RhKhLKTVZcvVZnhzyf8nbB8oNH7J6IjrXfMMg7g/nsKdHajOvuMZxp34DRo?= =?us-ascii?Q?pG6BnqILABp6YxwFwYpkobZh2Qf1Dl7biN2m8BhGK3jKS6z/4j/WO3GuMCEW?= =?us-ascii?Q?ur7UTANqL8jgneM+TQ436UUHX8EfQfwCqwBxPz+Ue6EewtmqGxvOlazttEmb?= =?us-ascii?Q?IxuJAqy2JJPmBDLCngzbAZqTvoTNptrxWhADBHHSglufRbYLrW6wjIqBuuF9?= =?us-ascii?Q?YxZPdNU2m87k5l/I3543NjlYddA0ir2uy7qi7lDBiuBO5T8ka+WfC+T2kTaR?= =?us-ascii?Q?vyCmIrpZbXfvi+ix8xouC9b+yg7sCJWiB7Zv3xktpDJfkutVl8TH6A3a1sze?= =?us-ascii?Q?oLluMeWp/d/ozOdSg0jO5c479n0plAD4QU8dSfsm0GPIBqaMf9XdA+afKjJs?= =?us-ascii?Q?EnTWdaWKgLF/oiWU+mQYERkrLvdd?= X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230040)(82310400026)(376014)(36860700013)(1800799024); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Oct 2024 08:05:52.4787 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4b6f6e7e-a160-4297-3a4e-08dcec2705e9 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF00021F6F.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB7895 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org If the trace dump is stored at the moment when there is some incomplete Tx transfers - WQE is pushed but hardware did not sent the completions yet - this incomplete was not dumped by the script mlx5_trace. For some cases (for example, if queue was stuck) the valuable debug information was lost. To improve the dump fullness the following optional script arguments are added: -v [level] - provides the raw dump of the object record of the specified level (0 - bursts, 1 - WQEs, 2+ - mbufs) -a - dumps all bursts, including incomplete ones Signed-off-by: Viacheslav Ovsiienko Acked-by: Dariusz Sosnowski --- drivers/net/mlx5/tools/mlx5_trace.py | 82 ++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 23 deletions(-) --- doc/guides/nics/mlx5.rst | 6 +++ drivers/net/mlx5/tools/mlx5_trace.py | 73 ++++++++++++++++++++-------- 2 files changed, 59 insertions(+), 20 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 1dccdaad50..f82e2d75de 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -2360,6 +2360,12 @@ Steps to enable Tx datapath tracing: The parameter of the script is the trace data folder. + The optional parameter ``-a`` forces to dump incomplete bursts. + + The optional parameter ``-v [level]`` forces to dump raw records data + for the specified level and below. Level 0 dumps bursts, level 1 dumps WQEs, + level 2 dumps mbufs. + .. code-block:: console mlx5_trace.py /var/log/rte-2023-01-23-AM-11-52-39 diff --git a/drivers/net/mlx5/tools/mlx5_trace.py b/drivers/net/mlx5/tools/mlx5_trace.py index 5eb634a490..96eb82082f 100755 --- a/drivers/net/mlx5/tools/mlx5_trace.py +++ b/drivers/net/mlx5/tools/mlx5_trace.py @@ -21,10 +21,13 @@ def __init__(self): self.wait_burst = [] # waiting for completion self.pq_id = 0 - def log(self): + def log(self, all): """Log all queue bursts""" for txb in self.done_burst: txb.log() + if all == True: + for txb in self.wait_burst: + txb.log() class MlxMbuf: @@ -147,24 +150,26 @@ def __init__(self): self.tx_qlst = {} # active Tx queues per port/queue self.tx_wlst = {} # wait timestamp list per CPU - def run(self, msg_it): + def run(self, msg_it, verbose): """Run over gathered tracing data and build database""" for msg in msg_it: if not isinstance(msg, bt2._EventMessageConst): continue event = msg.event if event.name.startswith(PFX_TX): - do_tx(msg, self) + do_tx(msg, self, verbose) # Handling of other log event cathegories can be added here + if verbose: + print("*** End of raw data dump ***") - def log(self): + def log(self, all): """Log gathered trace database""" for pq_id in self.tx_qlst: queue = self.tx_qlst.get(pq_id) - queue.log() + queue.log(all) -def do_tx_entry(msg, trace): +def do_tx_entry(msg, trace, verbose): """Handle entry Tx busrt""" event = msg.event cpu_id = event["cpu_id"] @@ -172,6 +177,10 @@ def do_tx_entry(msg, trace): if burst is not None: # continue existing burst after WAIT return + if verbose > 0: + print("%u:%X tx_entry(real_time=%u, port_id=%u, queue_id=%u)" % + (msg.default_clock_snapshot.ns_from_origin, cpu_id, + event["real_time"], event["port_id"], event["queue_id"])) # allocate the new burst and append to the queue burst = MlxBurst() burst.call_ts = event["real_time"] @@ -189,10 +198,14 @@ def do_tx_entry(msg, trace): queue.wait_burst.append(burst) -def do_tx_exit(msg, trace): +def do_tx_exit(msg, trace, verbose): """Handle exit Tx busrt""" event = msg.event cpu_id = event["cpu_id"] + if verbose > 0: + print("%u:%X tx_exit(real_time=%u, nb_sent=%u, nb_req=%u)" % + (msg.default_clock_snapshot.ns_from_origin, cpu_id, + event["real_time"], event["nb_sent"], event["nb_req"])) burst = trace.tx_blst.get(cpu_id) if burst is None: return @@ -204,10 +217,14 @@ def do_tx_exit(msg, trace): trace.tx_blst.pop(cpu_id) -def do_tx_wqe(msg, trace): +def do_tx_wqe(msg, trace, verbose): """Handle WQE record""" event = msg.event cpu_id = event["cpu_id"] + if verbose > 1: + print("%u:%X tx_wqe(real_time=%u, opcode=%08X)" % + (msg.default_clock_snapshot.ns_from_origin, cpu_id, + event["real_time"], event["opcode"])) burst = trace.tx_blst.get(cpu_id) if burst is None: return @@ -221,17 +238,24 @@ def do_tx_wqe(msg, trace): burst.wqes.append(wqe) -def do_tx_wait(msg, trace): +def do_tx_wait(msg, trace, verbose): """Handle WAIT record""" event = msg.event cpu_id = event["cpu_id"] + if verbose > 1: + print("%u:%X tx_wait(ts=%u)" % + (msg.default_clock_snapshot.ns_from_origin, cpu_id, event["ts"])) trace.tx_wlst[cpu_id] = event["ts"] -def do_tx_push(msg, trace): +def do_tx_push(msg, trace, verbose): """Handle WQE push event""" event = msg.event cpu_id = event["cpu_id"] + if verbose > 2: + print("%u:%X tx_push(mbuf=%X, pkt_len=%u, nb_segs=%u, wqe_id=%04X)" % + (msg.default_clock_snapshot.ns_from_origin, cpu_id, event["mbuf"], + event["mbuf_pkt_len"], event["mbuf_nb_segs"], event["wqe_id"])) burst = trace.tx_blst.get(cpu_id) if burst is None: return @@ -246,10 +270,15 @@ def do_tx_push(msg, trace): wqe.mbuf.append(mbuf) -def do_tx_complete(msg, trace): +def do_tx_complete(msg, trace, verbose): """Handle send completion event""" event = msg.event pq_id = event["port_id"] << 16 | event["queue_id"] + if verbose > 1: + cpu_id = event["cpu_id"] + print("%u:%X tx_complete(port_id=%u, queue_id=%u, ts=%u, wqe_id=%04X)" % + (msg.default_clock_snapshot.ns_from_origin, cpu_id, + event["port_id"], event["queue_id"], event["ts"], event["wqe_id"])) queue = trace.tx_qlst.get(pq_id) if queue is None: return @@ -274,21 +303,21 @@ def do_tx_complete(msg, trace): queue.wait_burst = queue.wait_burst[rmv:] -def do_tx(msg, trace): +def do_tx(msg, trace, verbose): """Handle Tx related records""" name = msg.event.name[PFX_TX_LEN:] if name == "entry": - do_tx_entry(msg, trace) + do_tx_entry(msg, trace, verbose) elif name == "exit": - do_tx_exit(msg, trace) + do_tx_exit(msg, trace, verbose) elif name == "wqe": - do_tx_wqe(msg, trace) + do_tx_wqe(msg, trace, verbose) elif name == "wait": - do_tx_wait(msg, trace) + do_tx_wait(msg, trace, verbose) elif name == "push": - do_tx_push(msg, trace) + do_tx_push(msg, trace, verbose) elif name == "complete": - do_tx_complete(msg, trace) + do_tx_complete(msg, trace, verbose) else: print("Error: unrecognized Tx event name: %s" % msg.event.name, file=sys.stderr) raise ValueError() @@ -299,12 +328,16 @@ def main() -> int: try: parser = argparse.ArgumentParser() parser.add_argument("path", nargs=1, type=str, help="input trace folder") + parser.add_argument("-a", "--all", nargs="?", default=False, const=True, + help="show all the bursts, including incomplete ones") + parser.add_argument("-v", "--verbose", type=int, nargs="?", default=0, const=2, + help="show all the records below specified level") args = parser.parse_args() mlx_tr = MlxTrace() msg_it = bt2.TraceCollectionMessageIterator(args.path) - mlx_tr.run(msg_it) - mlx_tr.log() + mlx_tr.run(msg_it, args.verbose) + mlx_tr.log(args.all) return 0 except ValueError: return -1 -- 2.34.1