From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DCC4C42D71; Tue, 27 Jun 2023 13:24:18 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 70B6140F18; Tue, 27 Jun 2023 13:24:18 +0200 (CEST) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2067.outbound.protection.outlook.com [40.107.220.67]) by mails.dpdk.org (Postfix) with ESMTP id 47F2940EE1 for ; Tue, 27 Jun 2023 13:24:17 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bLWSaALai/lbEJz/l0AbZwUK65od+e9ZPYjWJkWtIim+OeKYLdf48irnovvGiOt7O3dNSDhcI6cwblbPBIpllQhJ0lVQ4gM8HbT3sXg77f8e5D5oBarubWwCWMCQoJilFkuKjC/28YbqB2/p9Gfk0kujJckdm4iM1RhLpiiZjKt7BlAY+UzJ8aYA3a3XaGoqnl3XFjMaCVGx+AxLUmlkqVPF0BUBEccgP6wY56YA0bX2bd5qjeqbzfJehtotlDd1gzTctK/FoFyYT+t6BbzWmJL7ShuIP4urm8PUVQkkCCavES3Tn6CF+hBdqApCI7RaoAxwK1cuy/SH2/ZBNRjTOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=B3vJSsPHXG3cT1jDapbHg69HwLbQXI73QGl/sr56CzY=; b=dfvPpQmNxlLUrkZfftuvwe/rWRminHoZWL+z8frDxQv7Y91cGV+iaqybgIlnIOI7UvWBBU2jWMC3yNZH7+t9qUY4F/nyFbU1/NGgpWgWBM+3+xodlc+37qp+R9z756Wg04yOwuYQ9WGppvhBYN27vjIYbZdynWjcyRtHNiUGeaph4leKsHogndp3vNncRmQk3FmQXh/rN57tYK2bKxL6A9g1uI60BELd6PgoPZn1dPleDtzDi8JzuSHXwVMcIZodiOPs1iUZMcR7301ca6IsIq6ggvaSqBNXqetfC6velk7TMLADnWHVGUjMixU38yWAOH3YLwUXFgxrYBsFj+aOfA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=B3vJSsPHXG3cT1jDapbHg69HwLbQXI73QGl/sr56CzY=; b=bV2Hq1eHTVNEq3trdXKoKMQApH3PLUCUbz1xlmcjAAev8Bc55dTJAsiomSkVG07k4+5qEcKqDOoWCmrcZ5dA9u79pn6vxF5MvIYvikrnpEFdr2seDu6TmP0utTbYKXhGdSLEnJjtB/matrkSEtC7fB2tnGgBq2kWUSIves1VqRyWih6PgSrENFYrJtNsYghJSRCkPGY8QyVJpdPgMEzO+ds4R88gyG/qTY56IOuc+s7OvR8RLz3FWRoUoC4bX9TIgxaL+hgCghzcnR6kdH3cNtHLbjIlLZ7maVVMiOvpjRQV1IrzfPjmtoEFFa1pFKsqFcUSYAhxEAWAKMcI1LGZmw== Received: from DM6PR12MB3753.namprd12.prod.outlook.com (2603:10b6:5:1c7::18) by PH7PR12MB6444.namprd12.prod.outlook.com (2603:10b6:510:1f8::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.23; Tue, 27 Jun 2023 11:24:15 +0000 Received: from DM6PR12MB3753.namprd12.prod.outlook.com ([fe80::ab87:3bac:b8e:a23d]) by DM6PR12MB3753.namprd12.prod.outlook.com ([fe80::ab87:3bac:b8e:a23d%3]) with mapi id 15.20.6521.023; Tue, 27 Jun 2023 11:24:14 +0000 From: Slava Ovsiienko To: "NBU-Contact-Thomas Monjalon (EXTERNAL)" CC: "dev@dpdk.org" , Raslan Darawsheh , "rjarry@redhat.com" , "jerinj@marvell.com" Subject: RE: [PATCH v2 0/5] net/mlx5: introduce Tx datapath tracing Thread-Topic: [PATCH v2 0/5] net/mlx5: introduce Tx datapath tracing Thread-Index: AQHZnhhx0BkXf4lbDE6wy39ujPmB+q+ToU+AgApEF4CAALCMwA== Date: Tue, 27 Jun 2023 11:24:14 +0000 Message-ID: References: <20230420100803.494-1-viacheslavo@nvidia.com> <20230613165845.19109-1-viacheslavo@nvidia.com> <2676545.uZKlY2gecq@thomas> In-Reply-To: <2676545.uZKlY2gecq@thomas> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: DM6PR12MB3753:EE_|PH7PR12MB6444:EE_ x-ms-office365-filtering-correlation-id: b70ab2fb-7b37-4fb0-e36e-08db770109dd x-ld-processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: PpFnIW/Hj1aMskhDri0n7fFmepFNp8RhXY4DFrdCXQgCwlG59aG90vnq/9A2LLQBuwR5zUfRS8ykjREtWfRY7mLKVMuZWKQm1BI7CQGmPdeR99lzpxPFR3IvgSruQVDX3NQ1PXuBAt5rQd++2aZs5cNwwvmOrYxsve351ZFDeIc324p1jJXlYdB0zaDBhe2l03rV21ctlay7JRXKcdjJ7ZtJvWkqwk9E7vhgHwSovBz+YH+fIM9yQAg9CCD8rBvhnnZ6UD9ApfEkuAan0CXNb0+AP1OraAeiBZuzNZNEK7B/X7uo+CJJXKSw1j9E42w6M9e3i31f777T69u6TLJFpUGSkWChnFYG18QGroQ23IG4d3QIVASQ8AZJITuBVV0gzT5WcNWxTS1+3VB0C/ik/0grHsxSie3JsEjDQIFmB2LsNemlfAW9MQBsJ5I1vxALGodw0KaflwCKQc+7zoXk5RwMREtiF3kc4cHEti4LdobeCfQw0StSmgbxPnF0hot87Qf4IdG162iuDZSXh3WHqByxqx7tiefZ/m6VIOKCjvt4XddyxQ2Oka8ORtZpbooEoFarNKho6IRFc4RYiLLRYgzVQI9jM3jlqbBZLtKzvsiDg9sQbofxqQGjVfmGcWlhXzRqLDmqIJQSsRm1VQoQC+mkahEkRICYN088M+BbgcU= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR12MB3753.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(4636009)(366004)(346002)(376002)(136003)(396003)(39860400002)(451199021)(122000001)(38100700002)(38070700005)(33656002)(86362001)(54906003)(966005)(9686003)(71200400001)(7696005)(66476007)(41300700001)(66946007)(64756008)(316002)(55016003)(66556008)(83380400001)(8676002)(8936002)(76116006)(66446008)(53546011)(26005)(4326008)(6916009)(6506007)(186003)(2906002)(478600001)(5660300002)(52536014); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?dv75rJJhUda4jGow2fMrB9ayDqXeRsED8OltHa7oFS/XjXD4T6aNMEMhfuiC?= =?us-ascii?Q?O/X4c5Ks3thnY5Dm3n3ksSqUJBAo5QMvlpWuVmyz10pfYm5VLBU/BkfJykP6?= =?us-ascii?Q?y64//Ipmd/FThjtGsuuMUe42No51ZdvY/bJcJC6/3d/DSz/9W+sqCnE8NWRr?= =?us-ascii?Q?SZQLU4F+Iwe5oCjtu7m9k+Wa8jvqUAGo8Ob0fZDc+F6BKm3Jy2EF0Z25pKPt?= =?us-ascii?Q?3Ouu204TnWoG4Ts3LwY3d9DFQpvxvv9ZfIdPUkxCX7GfEEFbtQCdINAjYHik?= =?us-ascii?Q?raLUAeNo06JIa/lHwbafADhCZMrZLYdnEbAFeXbvKSt7adOnYEaUM6QMcd0+?= =?us-ascii?Q?8y91iduQll+nV4suyqDME4Bo2eOCaVXFCjfxAUcx76+m7Qdxy9CIczcf9Ker?= =?us-ascii?Q?XgfDt525DXF01qDuSxZXNwj5gVUDXBOx1L83jvhUGzGZlVozUH6vJGUkVhqg?= =?us-ascii?Q?uph/skxSBOquZq82dFTu5/S22mQ3bLTBkWA9Y6fictxyDhjwa2kBG85ccxIF?= =?us-ascii?Q?uswuOPmn78dTDR618S3NHBFHMAWvPEIJ5SGWKYQ93KYTFutc37MtOvpJBMIg?= =?us-ascii?Q?sPpHHad03ZQJY9/0OzYaKgiJDV6pTMw6paBnshFa+HitkfU5gQTtdapTH77b?= =?us-ascii?Q?eVsQq2GKV9ym0FaapH7emcaFrZ42/kZ9qBhVbpaYiJdN/uVpigQudWmN6Fb8?= =?us-ascii?Q?f6fm6M7hRb11pRpp++87Vf/9JLPW25tS7yEf4ooun7BiiErLc17nTPoGTGnw?= =?us-ascii?Q?jOPX1LsjVQ2+5J5r3euMbmm7fnXTZGlZc/znZy2f6N0oii5pRc3PcO2I/+3i?= =?us-ascii?Q?UeZmWcoQUlmW0otvMvf3+/EMqab1clYkgpljdAXwe4WTpJh0GBSeXB/Xln4o?= =?us-ascii?Q?kilvxZ0q4R1TZRpCg44+lnqDNEY55wEPRHg1XrT96aXynWoud7U5vo3/y46x?= =?us-ascii?Q?8YNxYNoLBYGHaLy6xqVRfpSDAC5k2oM5G0QAj9TtfK6Azj2Gim6C+GJTL6u2?= =?us-ascii?Q?sIMq6oPYorlzgZAId1R84+DEbyWgclh6aY0SsXQ+tBcfe8MpSnFLA/6WHf8e?= =?us-ascii?Q?8KQs4EnsuY2hnLfmLTCV8+kpYp0QpS4XYQ8S7SAVAN9kbURNDaLXoF+DfhVL?= =?us-ascii?Q?x1SNh03JW6kHfqHiypL4sXHPuPgLQnmK6Q6GbRFWV9wUIZmicOlt2G5SVBBd?= =?us-ascii?Q?LpfA8JoHVLmkPQ+F6pMppMoRAqQF3EJXFwfWPCDR9FbPbQ5C3pH5kRDZ/+kc?= =?us-ascii?Q?KJbkwjeOElGAvS4wkbzuU+ZwBCrn+FEg7Dj5GZI3w7JpKGpTgOUkMGZP3075?= =?us-ascii?Q?CUyefrEJAo56FtOqJo6VkxUW42jaDO8Vrg3tJ2VQqKrWuls5XFoEI41uHI+L?= =?us-ascii?Q?CUSDrMjuEzain9ppMjtY0VhBBKrgcUCd/OM0CIFbB7om+INfW7MUpBinUZII?= =?us-ascii?Q?wQ/cqVkW6ipX0tb4tNs5RB5tj068LznOh07Y/z5Ak2t9NXeMZJEY4luvNU62?= =?us-ascii?Q?acWP8SGuOC6T1lu9RC7YDIcIrQc48nGbwK9Bo3P2nctw9sG+5kDKWEMyDP/J?= =?us-ascii?Q?EFdGqsJgK1eMASVXbblahQZAQSahFQMoDva7NaK2?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3753.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: b70ab2fb-7b37-4fb0-e36e-08db770109dd X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Jun 2023 11:24:14.6695 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: gMls+VEwYkwWioWJoM9MaaE64tWPB5tjgcnxTm4NPQE730WP2t4d331AsxBBymAT08Lpban8TRWNpZacOKkktQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6444 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Thomas Monjalon > Sent: Tuesday, June 27, 2023 3:46 AM > To: Slava Ovsiienko > Cc: dev@dpdk.org; Raslan Darawsheh ; > rjarry@redhat.com; jerinj@marvell.com > Subject: Re: [PATCH v2 0/5] net/mlx5: introduce Tx datapath tracing >=20 > 20/06/2023 14:00, Raslan Darawsheh: > > Hi, > > > > > -----Original Message----- > > > From: Viacheslav Ovsiienko > > > Sent: Tuesday, June 13, 2023 7:59 PM > > > To: dev@dpdk.org > > > Subject: [PATCH v2 0/5] net/mlx5: introduce Tx datapath tracing > > > > > > The mlx5 provides the send scheduling on specific moment of time, > > > and for the related kind of applications it would be extremely > > > useful to have extra debug information - when and how packets were > > > scheduled and when the actual sending was completed by the NIC > > > hardware (it helps application to track the internal delay issues). > > > > > > Because the DPDK tx datapath API does not suppose getting any > > > feedback from the driver and the feature looks like to be mlx5 > > > specific, it seems to be reasonable to engage exisiting DPDK datapath > tracing capability. > > > > > > The work cycle is supposed to be: > > > - compile appplication with enabled tracing > > > - run application with EAL parameters configuring the tracing in ml= x5 > > > Tx datapath > > > - store the dump file with gathered tracing information > > > - run analyzing scrypt (in Python) to combine related events (packe= t > > > firing and completion) and see the data in human-readable view > > > > > > Below is the detailed instruction "how to" with mlx5 NIC to gather > > > all the debug data including the full timings information. > > > > > > > > > 1. Build DPDK application with enabled datapath tracing > > > > > > The meson option should be specified: > > > --enable_trace_fp=3Dtrue > > > > > > The c_args shoudl be specified: > > > -DALLOW_EXPERIMENTAL_API > > > > > > The DPDK configuration examples: > > > > > > meson configure --buildtype=3Ddebug -Denable_trace_fp=3Dtrue > > > -Dc_args=3D'-DRTE_LIBRTE_MLX5_DEBUG -DRTE_ENABLE_ASSERT - > > > DALLOW_EXPERIMENTAL_API' build > > > > > > meson configure --buildtype=3Ddebug -Denable_trace_fp=3Dtrue > > > -Dc_args=3D'-DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API' > > > build > > > > > > meson configure --buildtype=3Drelease -Denable_trace_fp=3Dtrue > > > -Dc_args=3D'-DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API' > > > build > > > > > > meson configure --buildtype=3Drelease -Denable_trace_fp=3Dtrue > > > -Dc_args=3D'-DALLOW_EXPERIMENTAL_API' build > > > > > > > > > 2. Configuring the NIC > > > > > > If the sending completion timings are important the NIC should be > > > configured to provide realtime timestamps, the > > > REAL_TIME_CLOCK_ENABLE NV settings parameter should be configured > to > > > TRUE, for example with command (and with following FW/driver reset): > > > > > > sudo mlxconfig -d /dev/mst/mt4125_pciconf0 s > > > REAL_TIME_CLOCK_ENABLE=3D1 > > > > > > > > > 3. Run DPDK application to gather the traces > > > > > > EAL parameters controlling trace capability in runtime > > > > > > --trace=3Dpmd.net.mlx5.tx - the regular expression enabling the tra= cepoints > > > with matching names at least "pmd.net.mlx= 5.tx" > > > must be enabled to gather all events need= ed > > > to analyze mlx5 Tx datapath and its timin= gs. > > > By default all tracepoints are disabled. > > > > > > --trace-dir=3D/var/log - trace storing directory > > > > > > --trace-bufsz=3DB|K|M - optional, trace data buffer = size > > > per thread. The default is 1MB= . > > > > > > --trace-mode=3Doverwrite|discard - optional, selects trace data bu= ffer > mode. > > > > > > > > > 4. Installing or Building Babeltrace2 Package > > > > > > The gathered trace data can be analyzed with a developed Python scrip= t. > > > To parse the trace, the data script uses the Babeltrace2 library. > > > The package should be either installed or built from source code as > > > shown below: > > > > > > git clone https://github.com/efficios/babeltrace.git > > > cd babeltrace > > > ./bootstrap > > > ./configure -help > > > ./configure --disable-api-doc --disable-man-pages > > > --disable-python-bindings-doc --enbale-python-plugins > > > --enable-python-binding > > > > > > 5. Running the Analyzing Script > > > > > > The analyzing script is located in the folder: > > > ./drivers/net/mlx5/tools It requires Python3.6, Babeltrace2 packages > > > and it takes the only parameter of trace data file. For example: > > > > > > ./mlx5_trace.py /var/log/rte-2023-01-23-AM-11-52-39 > > > > > > > > > 6. Interpreting the Script Output Data > > > > > > All the timings are given in nanoseconds. > > > The list of Tx (and coming Rx) bursts per port/queue is presented in > > > the output. > > > Each list element contains the list of built WQEs with specific > > > opcodes, and each WQE contains the list of the encompassed packets to > send. >=20 > This information should be in the documentation. OK, should we make this cover-letter part of mlx5.rst? >=20 > I think we should request a review of the Python script from people famil= iar > with tracing and from people more familiar with Python scripting for user > tools. Would be very helpful, could you recommend/ask someone? With best regards, Slava >=20