From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3FA5E42DD9; Wed, 5 Jul 2023 17:32:22 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 82D6842D44; Wed, 5 Jul 2023 17:32:05 +0200 (CEST) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2064.outbound.protection.outlook.com [40.107.220.64]) by mails.dpdk.org (Postfix) with ESMTP id 8795C42D2F for ; Wed, 5 Jul 2023 17:32:01 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eGRZ0pBjN4LyHfHPlBxFQuoI/eNAIuLoNanLZgRJwhJ1WMCWywpFUZolQS6yQCoPOv0/GUjl3jeZH65Agrpsa4EW5I9L3NAg6p1hCJtJBtwWq7wXkJDxPX7Jd7XQ1/LqukJ1GAOjMLuRKfvvFU+KV/JcR50YewsgwnGQGVGUtQ5cJYyUoTczDM8zUqUzbuYmJHOfkFLLz0wnANOBJoc+d1DvZft4dQg5mwqSzdApa0TETyV0fF3J/+d96wCek3TS6uiJbnPJ7CcZ9Cq7cooaAqNJOurr7oHGCuNmCvTNyimP0gUvuU+YfQh7RDwwdPJwGvKTxzx4K0QBQzLUZYV7jQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/DsapU1Z/lqzk/PAxwff7QOfmmrMm5Xv0ofwjQ6zr/I=; b=XMJoXZyCLPEQqTTMvL9IY3gufedEGula4dA202DeKg1j+ocrDmGgfTivRaPvmoUCIYgC1KIOK5qlnD1HzNG34ji3zffBDgHaj4GfM7KszcfmcgLOYQFpCbsrf/k5ZiYJtCCO5W8VhyJgb2AMXGFbOQJNCR8kZPU4Rr9l15KqL1zYzhTstnlaAddT9hoGzVsiNNiOp7SSEC1yyLIjMb/dPbDc7TS8ZQ1zW9fam2OUBRguAlQaxpWFcugV0bay29IU9VksUUHlYN/fp8qHzQWF9BBuxNGGKT6ZbhkejL1uofjCy7Q04SMMRj3zPmlJ1kL2tjoWPEESJaeydIBJOFw8Uw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/DsapU1Z/lqzk/PAxwff7QOfmmrMm5Xv0ofwjQ6zr/I=; b=sJXjQf3L4dAHYjzCsl2UHJEokATAY+dhULdsfCEtVW4p/ekylsYjC3MA5e4JYeo2Y6kQbZ9hrTgcY7HvCEOw5L/pOi6su4r2up1BYenMdedSh4VMZQYiEdgpaUw83bt329O3ryxy9+bpuaDFXhJz5yab+d6X/vML9JTUSq7elQ7OV1ChHABRAZG6jTCIEI7UTAzjqc+gRyr+rV1DnmNDbBldzeCzGoxYQHQUQ8h55VprWcLG7CJXSaD8/mG10Lyi9HpciKycP6lTm8hEzwszhvObVV/9kNQSShPICSZh9Jc0+zQ/xticzVZug1YJhAoRuhamzB3XcRCc+4HtSqqa6w== Received: from SJ0PR03CA0089.namprd03.prod.outlook.com (2603:10b6:a03:331::34) by PH7PR12MB6693.namprd12.prod.outlook.com (2603:10b6:510:1b0::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6565.17; Wed, 5 Jul 2023 15:31:59 +0000 Received: from DM6NAM11FT098.eop-nam11.prod.protection.outlook.com (2603:10b6:a03:331:cafe::79) by SJ0PR03CA0089.outlook.office365.com (2603:10b6:a03:331::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6565.18 via Frontend Transport; Wed, 5 Jul 2023 15:31:59 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DM6NAM11FT098.mail.protection.outlook.com (10.13.173.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.45 via Frontend Transport; Wed, 5 Jul 2023 15:31:58 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Wed, 5 Jul 2023 08:31:44 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Wed, 5 Jul 2023 08:31:43 -0700 From: Viacheslav Ovsiienko To: CC: , Subject: [PATCH v5 3/4] net/mlx5: add Tx datapath trace analyzing script Date: Wed, 5 Jul 2023 18:31:24 +0300 Message-ID: <20230705153125.4657-4-viacheslavo@nvidia.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20230705153125.4657-1-viacheslavo@nvidia.com> References: <20230420100803.494-1-viacheslavo@nvidia.com> <20230705153125.4657-1-viacheslavo@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT098:EE_|PH7PR12MB6693:EE_ X-MS-Office365-Filtering-Correlation-Id: 3148950a-f6c4-4076-4b4c-08db7d6cf8f3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: gkTz3uz2Xm1HH/N/Yr2Khejl6ptYhjlsKePmN9xlkEzl4nCKlNCajD4XItM+V9lMSsusqwGISon8a2O0NmuaBwyUIMeUb40YZqzAzK8+UL+UuM4Za2YWdpd9y/JlZ8s8Zskx9+GMey3oa5CQw7czsuNLRCModjh64Zr4N2qQnnmhMle6L1F8i88bAfgfzIcj94KuF7rGU0o50wb3UJjHNxE0HKAmm5QKCw68nQtrJfkXZRFDBSpCFE0TFsZxTaK+k2ldZq6fydiVafHPR56LW6UCuIs4FxeyDG2dROo1VTo9qJ77ASRbFqtJ60X0jXdcLU2ZEkwnRnpruAAGggt9pbxgU5twtu47/BBtWkwVj5YMam4QfHB8hy9EZcyHvA+doUm9VokNpSSp78yXvpozs4oCYnMub6II0aaJ3xwaqXEplZ5Y18BS0UvgKhbAof0QBOkdzO/n/wmtllCBRq1mizZ22Kv5yrA4YAxWCJaHWKCcTohXlWbYD/RuZ8tgIsfMcz9eYd3XuCo1XCEJMMga78r65FnVOk2F9vusAepdNJTKk2Y0Bs54FVu0lB2lgSaPU6fDoPQQqF4wfCgjWEfdpfKnCrVweMuUA6n7Satj7iXEJB8Jhco+33uxR+2flFVwRyA2jvZ1RPCnqx5tQR0C+kvvlGDuxfJFQudQI0rkrXjP7fohOQXdApQ0o0p4GjQ+aLPTKMlEMJNCgQbBNRtIaSXqJlPB8E/OgLgLoEGUSojRQ6LixjSF8kM0XMeUS+3F X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(136003)(346002)(376002)(39860400002)(396003)(451199021)(36840700001)(40470700004)(46966006)(5660300002)(36860700001)(8936002)(8676002)(41300700001)(316002)(47076005)(2906002)(336012)(426003)(83380400001)(2616005)(82740400003)(55016003)(6286002)(40480700001)(16526019)(70206006)(6916009)(4326008)(1076003)(356005)(7636003)(70586007)(40460700003)(82310400005)(86362001)(186003)(54906003)(26005)(107886003)(478600001)(6666004)(7696005)(36756003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jul 2023 15:31:58.8128 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3148950a-f6c4-4076-4b4c-08db7d6cf8f3 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT098.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6693 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The Python script is intended to analyze mlx5 PMD datapath traces and report: - tx_burst routine timings - how packets are pushed to WQEs - how packet sending is completed with timings Signed-off-by: Viacheslav Ovsiienko --- drivers/net/mlx5/tools/mlx5_trace.py | 307 +++++++++++++++++++++++++++ 1 file changed, 307 insertions(+) create mode 100755 drivers/net/mlx5/tools/mlx5_trace.py diff --git a/drivers/net/mlx5/tools/mlx5_trace.py b/drivers/net/mlx5/tools/mlx5_trace.py new file mode 100755 index 0000000000..8c1fd0a350 --- /dev/null +++ b/drivers/net/mlx5/tools/mlx5_trace.py @@ -0,0 +1,307 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2023 NVIDIA Corporation & Affiliates + +""" +Analyzing the mlx5 PMD datapath tracings +""" +import sys +import argparse +import bt2 + +PFX_TX = "pmd.net.mlx5.tx." +PFX_TX_LEN = len(PFX_TX) + + +class MlxQueue: + """Queue container object""" + + def __init__(self): + self.done_burst = [] # completed bursts + self.wait_burst = [] # waiting for completion + self.pq_id = 0 + + def log(self): + """Log all queue bursts""" + for txb in self.done_burst: + txb.log() + + +class MlxMbuf: + """Packet mbufs container object""" + + def __init__(self): + self.wqe = 0 # wqe id + self.ptr = None # first packet mbuf pointer + self.len = 0 # packet data length + self.nseg = 0 # number of segments + + def log(self): + """Log mbuf""" + out_txt = " %X: %u" % (self.ptr, self.len) + if self.nseg != 1: + out_txt += " (%d segs)" % self.nseg + print(out_txt) + + +class MlxWqe: + """WQE container object""" + + def __init__(self): + self.mbuf = [] # list of mbufs in WQE + self.wait_ts = 0 # preceding wait/push timestamp + self.comp_ts = 0 # send/recv completion timestamp + self.opcode = 0 + + def log(self): + """Log WQE""" + wqe_id = (self.opcode >> 8) & 0xFFFF + wqe_op = self.opcode & 0xFF + out_txt = " %04X: " % wqe_id + if wqe_op == 0xF: + out_txt += "WAIT" + elif wqe_op == 0x29: + out_txt += "EMPW" + elif wqe_op == 0xE: + out_txt += "TSO " + elif wqe_op == 0xA: + out_txt += "SEND" + else: + out_txt += "0x%02X" % wqe_op + if self.comp_ts != 0: + out_txt += " (%d, %d)" % (self.wait_ts, self.comp_ts - self.wait_ts) + else: + out_txt += " (%d)" % self.wait_ts + print(out_txt) + for mbuf in self.mbuf: + mbuf.log() + + def comp(self, wqe_id, wqe_ts): + """Return 0 if WQE in not completedLog WQE""" + if self.comp_ts != 0: + return 1 + cur_id = (self.opcode >> 8) & 0xFFFF + if cur_id > wqe_id: + cur_id -= wqe_id + if cur_id <= 0x8000: + return 0 + else: + cur_id = wqe_id - cur_id + if cur_id >= 0x8000: + return 0 + self.comp_ts = wqe_ts + return 1 + + +class MlxBurst: + """Packet burst container object""" + + def __init__(self): + self.wqes = [] # issued burst WQEs + self.done = 0 # number of sent/recv packets + self.req = 0 # requested number of packets + self.call_ts = 0 # burst routine invocation + self.done_ts = 0 # burst routine done + self.queue = None + + def log(self): + """Log burst""" + port = self.queue.pq_id >> 16 + queue = self.queue.pq_id & 0xFFFF + if self.req == 0: + print( + "%u: tx(p=%u, q=%u, %u/%u pkts (incomplete)" + % (self.call_ts, port, queue, self.done, self.req) + ) + else: + print( + "%u: tx(p=%u, q=%u, %u/%u pkts in %u" + % ( + self.call_ts, + port, + queue, + self.done, + self.req, + self.done_ts - self.call_ts, + ) + ) + for wqe in self.wqes: + wqe.log() + + def comp(self, wqe_id, wqe_ts): + """Return 0 if not all of WQEs in burst completed""" + wlen = len(self.wqes) + if wlen == 0: + return 0 + for wqe in self.wqes: + if wqe.comp(wqe_id, wqe_ts) == 0: + return 0 + return 1 + + +class MlxTrace: + """Trace representing object""" + + def __init__(self): + self.tx_blst = {} # current Tx bursts per CPU + self.tx_qlst = {} # active Tx queues per port/queue + self.tx_wlst = {} # wait timestamp list per CPU + + def run(self, msg_it): + """Run over gathered tracing data and build database""" + for msg in msg_it: + if not isinstance(msg, bt2._EventMessageConst): + continue + event = msg.event + if event.name.startswith(PFX_TX): + do_tx(msg, self) + # Handling of other log event cathegories can be added here + + def log(self): + """Log gathered trace database""" + for pq_id in self.tx_qlst: + queue = self.tx_qlst.get(pq_id) + queue.log() + + +def do_tx_entry(msg, trace): + """Handle entry Tx busrt""" + event = msg.event + cpu_id = event["cpu_id"] + burst = trace.tx_blst.get(cpu_id) + if burst is not None: + # continue existing burst after WAIT + return + # allocate the new burst and append to the queue + burst = MlxBurst() + burst.call_ts = msg.default_clock_snapshot.ns_from_origin + trace.tx_blst[cpu_id] = burst + pq_id = event["port_id"] << 16 | event["queue_id"] + queue = trace.tx_qlst.get(pq_id) + if queue is None: + # queue does not exist - allocate the new one + queue = MlxQueue() + queue.pq_id = pq_id + trace.tx_qlst[pq_id] = queue + burst.queue = queue + queue.wait_burst.append(burst) + + +def do_tx_exit(msg, trace): + """Handle exit Tx busrt""" + event = msg.event + cpu_id = event["cpu_id"] + burst = trace.tx_blst.get(cpu_id) + if burst is None: + return + burst.done_ts = msg.default_clock_snapshot.ns_from_origin + burst.req = event["nb_req"] + burst.done = event["nb_sent"] + trace.tx_blst.pop(cpu_id) + + +def do_tx_wqe(msg, trace): + """Handle WQE record""" + event = msg.event + cpu_id = event["cpu_id"] + burst = trace.tx_blst.get(cpu_id) + if burst is None: + return + wqe = MlxWqe() + wqe.wait_ts = trace.tx_wlst.get(cpu_id) + if wqe.wait_ts is None: + wqe.wait_ts = msg.default_clock_snapshot.ns_from_origin + wqe.opcode = event["opcode"] + burst.wqes.append(wqe) + + +def do_tx_wait(msg, trace): + """Handle WAIT record""" + event = msg.event + cpu_id = event["cpu_id"] + trace.tx_wlst[cpu_id] = event["ts"] + + +def do_tx_push(msg, trace): + """Handle WQE push event""" + event = msg.event + cpu_id = event["cpu_id"] + burst = trace.tx_blst.get(cpu_id) + if burst is None: + return + if not burst.wqes: + return + wqe = burst.wqes[-1] + mbuf = MlxMbuf() + mbuf.wqe = event["wqe_id"] + mbuf.ptr = event["mbuf"] + mbuf.len = event["mbuf_pkt_len"] + mbuf.nseg = event["mbuf_nb_segs"] + wqe.mbuf.append(mbuf) + + +def do_tx_complete(msg, trace): + """Handle send completion event""" + event = msg.event + pq_id = event["port_id"] << 16 | event["queue_id"] + queue = trace.tx_qlst.get(pq_id) + if queue is None: + return + qlen = len(queue.wait_burst) + if qlen == 0: + return + wqe_id = event["wqe_id"] + wqe_ts = event["ts"] + rmv = 0 + while rmv < qlen: + burst = queue.wait_burst[rmv] + if burst.comp(wqe_id, wqe_ts) == 0: + break + rmv += 1 + # mode completed burst to done list + if rmv != 0: + idx = 0 + while idx < rmv: + queue.done_burst.append(burst) + idx += 1 + del queue.wait_burst[0:rmv] + + +def do_tx(msg, trace): + """Handle Tx related records""" + name = msg.event.name[PFX_TX_LEN:] + if name == "entry": + do_tx_entry(msg, trace) + elif name == "exit": + do_tx_exit(msg, trace) + elif name == "wqe": + do_tx_wqe(msg, trace) + elif name == "wait": + do_tx_wait(msg, trace) + elif name == "push": + do_tx_push(msg, trace) + elif name == "complete": + do_tx_complete(msg, trace) + else: + print("Error: unrecognized Tx event name: %s" % msg.event.name, file=sys.stderr) + raise ValueError() + + +def main() -> int: + """Script entry point""" + try: + parser = argparse.ArgumentParser() + parser.add_argument("path", nargs=1, type=str, help="input trace folder") + args = parser.parse_args() + + mlx_tr = MlxTrace() + msg_it = bt2.TraceCollectionMessageIterator(args.path) + mlx_tr.run(msg_it) + mlx_tr.log() + return 0 + except ValueError: + return -1 + + +if __name__ == "__main__": + sys.exit(main()) -- 2.18.1