From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BD000A00C2; Sun, 17 Jul 2022 05:10:21 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 89F4440684; Sun, 17 Jul 2022 05:10:20 +0200 (CEST) Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-eopbgr70041.outbound.protection.outlook.com [40.107.7.41]) by mails.dpdk.org (Postfix) with ESMTP id E8DEE40156 for ; Sun, 17 Jul 2022 05:10:18 +0200 (CEST) ARC-Seal: i=2; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=pass; b=aXwwZ5DcTFnoZFVkuFOdlADXMueewshMf6FXRZQ2M6358vhbgU0gdc2HhkK5y+UEBo6uNB9PENlogsZsmOncDHvbew3pMTgeH2EiwqPzg7zYdLUs75US4u6bpzSYThpSTA7LgeXtTyQUQRcuGF6HBKlzZUUBlzpFDQcpzgldVJ5ZOX9Jymnkud7f899XzAbtNHaqIVSqw3Zbl/uKZzVMyUtxw4mHB0lF108qW+qx1EMP4J+uNpl6RE1eUsEN+w2lwvY4+yjxOYU45zNyExFZTt+8P3iwPH1R/PiPfCn2ZCMvoj17NJW5t359WWKTXAmKX8jF6kMI/pqKAsKvmVc87w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=EL4QpqeaCXseURZUGgtnNLczVcCyvWEE8eZsXDBhWBQ=; b=Ys92bxCuirJvgWDVu/ihbI4a0di2o3xSaw4wJHELQx70kPWsRGZD8Yh3+ZjULjycwkSJ+sVJpeuu+eAoHnAaxqe5H3vCJeS/+bBwlaTNeQbFT6/XVA1YGco5YLjGXrER9dVWMX3c/tE70Gvx/96vb7wKAtI2N62exdnv/jyjIrUK2DdpZn94pfJ7QZKVaAD82UMW3es7l21o5uuHW+WVSUAOTMkVvt2xp+f9sSzkkvR4twabgip/F1xKzkTkZy0lWCAQMY8pIyPj1CBN7tpy1916jgzwygjAnmVaQbxb65VxPPg6AiVyD0d5olTXOhhQQZyRcBAkSP4NdGjKvhScKg== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 63.35.35.123) smtp.rcpttodomain=dpdk.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dkim=[1,1,header.d=arm.com] dmarc=[1,1,header.from=arm.com]) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EL4QpqeaCXseURZUGgtnNLczVcCyvWEE8eZsXDBhWBQ=; b=AyZ7v5w5t8mS5HVpAycRXa0pbCKMGbVqMo82xVzA1Xv85Aeg1CaZcCRzSGX2uv00X8G8tZdN8PAJ2tkpTNtEN4grMr2xJnerLcWQ+JeWZCt9nU4PwRlTlrPxa7vBQ5xpwXb1T5VcOUU4G5sVFFN+KN7mc45qBWGEYKAXG46qaUo= Received: from FR3P281CA0100.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a1::18) by DBBPR08MB4712.eurprd08.prod.outlook.com (2603:10a6:10:f4::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5438.20; Sun, 17 Jul 2022 03:10:15 +0000 Received: from VE1EUR03FT040.eop-EUR03.prod.protection.outlook.com (2603:10a6:d10:a1:cafe::63) by FR3P281CA0100.outlook.office365.com (2603:10a6:d10:a1::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5458.10 via Frontend Transport; Sun, 17 Jul 2022 03:10:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by VE1EUR03FT040.mail.protection.outlook.com (10.152.18.210) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5438.12 via Frontend Transport; Sun, 17 Jul 2022 03:10:14 +0000 Received: ("Tessian outbound cc6a8ab50b6b:v123"); Sun, 17 Jul 2022 03:10:12 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: 28b5aa8f16d13a22 X-CR-MTA-TID: 64aa7808 Received: from e5d19c10f3e5.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 760D55E8-1825-499A-BAFA-CC138343ADA1.1; Sun, 17 Jul 2022 03:10:05 +0000 Received: from EUR02-VE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id e5d19c10f3e5.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Sun, 17 Jul 2022 03:10:05 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=X8KDU6ew7JWM7+s8WFu7nvjOOFV54XJrYC8ZXeJGTAQFGvqZaf0B5ZuQLr+YR8JPXEo8cJGxfx7Zc+L1OHqfmMDbFDzhW7fL2BwP3akq5l0/sqR2/WVa+0pSljZHTwwj25UikhuX6RfbxTeUuddtmUxI7zfOMjoqlKgxRf+jv+37pPxEsQikQh0/LN+fQDPjgEd5Ob+LwyhuwDl9CKus0h9q7MSeE5IIU/hw3dYqFWNAVmvtS3Czbn+5jGlzI9bHyysOYPm/SeilwI5fqRvUwr9Ep2Ovfb3CcJGLE2qkGXGDsR/19g2/xwdDZDgZtc1173cxwmN2gplPM7D3ng+bmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=EL4QpqeaCXseURZUGgtnNLczVcCyvWEE8eZsXDBhWBQ=; b=HNrVZSliMUwnWIXf//5F5gx0V6KAXYU3j8eOILCNTUJaFxGn5dVcpcsBBHzXab4bOoEPMWCYsH/7R4uH3KpwwlW97yj6L0KT/PG3gYHmWrJkLj5VlnuckOKT3GDlDjBC30aS9ap9Em7Fpwj+AAdFCp59tiG2XHMfJyci0YSIA63Y3kq0vvGOpyhWMWlCt19Xz00NfD/r1gEBy6ZAPCDsX+BS2RHtj6frCIYG8n9l41kKlNb1B51PkHggHg7a+EEaZWkARXoz4YUC/iy8ibw2yqlHKqBwtIZ8NSWBvgT2IzooY0Ww7UU+nEWN2qi35UxTsCfplPW/BfjeeDi5+ZnrzQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EL4QpqeaCXseURZUGgtnNLczVcCyvWEE8eZsXDBhWBQ=; b=AyZ7v5w5t8mS5HVpAycRXa0pbCKMGbVqMo82xVzA1Xv85Aeg1CaZcCRzSGX2uv00X8G8tZdN8PAJ2tkpTNtEN4grMr2xJnerLcWQ+JeWZCt9nU4PwRlTlrPxa7vBQ5xpwXb1T5VcOUU4G5sVFFN+KN7mc45qBWGEYKAXG46qaUo= Received: from DBAPR08MB5814.eurprd08.prod.outlook.com (2603:10a6:10:1b1::6) by PA4PR08MB6285.eurprd08.prod.outlook.com (2603:10a6:102:e0::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5438.12; Sun, 17 Jul 2022 03:10:02 +0000 Received: from DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::1c7f:6a8d:b518:f972]) by DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::1c7f:6a8d:b518:f972%3]) with mapi id 15.20.5438.022; Sun, 17 Jul 2022 03:10:02 +0000 From: Honnappa Nagarahalli To: =?iso-8859-1?Q?Morten_Br=F8rup?= , Anatoly Burakov , "dev@dpdk.org" , Bruce Richardson , Nicolas Chautru , Fan Zhang , Ashish Gupta , Akhil Goyal , David Hunt , Chengwen Feng , Kevin Laatz , Ray Kinsella , "thomas@monjalon.net" , Ferruh Yigit , Andrew Rybchenko , "jerinj@marvell.com" , Sachin Saxena , "hemant.agrawal@nxp.com" , Ori Kam , Konstantin Ananyev CC: Conor Walsh , nd , nd Subject: RE: [PATCH v1 1/2] eal: add lcore busyness telemetry Thread-Topic: [PATCH v1 1/2] eal: add lcore busyness telemetry Thread-Index: AdiYTJmZ7BVKfv4oQhKijV2m/y3LPAASDVxwAD1op6A= Date: Sun, 17 Jul 2022 03:10:02 +0000 Message-ID: References: <24c49429394294cfbf0d9c506b205029bac77c8b.1657890378.git.anatoly.burakov@intel.com> <98CBD80474FA8B44BF855DF32C47DC35D871CC@smartserver.smartshare.dk> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D871CC@smartserver.smartshare.dk> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: D2CE8B81E3306D47AACBEDE879AD6CD4.0 Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-MS-Office365-Filtering-Correlation-Id: 819d3dd3-36f9-4293-8779-08da67a1de51 x-ms-traffictypediagnostic: PA4PR08MB6285:EE_|VE1EUR03FT040:EE_|DBBPR08MB4712:EE_ x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr x-checkrecipientrouted: true nodisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: 18bmT83CSF1QX5jvEbfbnq0vMI6xAjd6Nz3sgf6YL9iNgcMPc4VsLvsQ88G4sPTSTdtgntk2NlXDpFdzC8Ac0wGfoZkiV/Xz0pKVYLBFVxO8d3BA9u9jJ0I/CFZLDYMNc18ROxCmWWoDR4NfXIpqDJaFgFgQKfu2lARCZ41kXadBfFySunya9DPa8MYPr28PRximQrdn3BcvlSpvlKcEACcKE0UbY9T3nprqoToLH+ZzmgsrrWhGzb6a5NgiA5n5xYFz0sxcL1qN9rqw2nboc6BHa3XgRRq76qe1HpolCA/qNOaxf1Ax+I6cbxMWZigLr9SZIH0smxjE7sAgLUqucNiFUPw8eEfqofMLt2IMPpuBIkOO1hZtPGLEdzyQJhOAhtAtHTVymVcAzxEp+DHkDt+mVGn+F6IyhI4PvlXFKMYn81ZYXD3r9sRoEzV4ajPjnBGi4TqZkeZhrEIV93C3RX3megkBOkaFAbH4sxbqcnw0BMMg/XJF6rYYmVHuCL61FDvULwFGhjzz7JlldI10RkbbRcQTmYUMCKJwLYJld/6TgtcPAmaYDrYFyoMkfwlsVJpbokcshILTl/IbYoqeByK77DWuCEfb+SRuokcTKHl/3ks1mOGLPhtx420NBzdmnyHS9JaisnQiKW+HvdERi3ixqNoyxymnWFMBk885ZDTbr2YLhXmOKnVg0TO3o2hXUKClmXC7IvHpMnmxdSRyi8bWSsYz7o3IFgAr5QGqFctbvyHT8ve4zkEqFpw6nogAy7WMIE7773UfpOcLTP7zGcPAwaLhBwVMNgKxKAvHdERzhdXc+8i8L5ogw056YQ9Vj3R0Ig1SK0hl+pxl6+rwXw== X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DBAPR08MB5814.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(4636009)(136003)(346002)(39850400004)(366004)(376002)(396003)(2906002)(41300700001)(186003)(6506007)(9686003)(33656002)(26005)(7696005)(38100700002)(83380400001)(122000001)(38070700005)(921005)(76116006)(55016003)(86362001)(66946007)(66556008)(66476007)(66446008)(64756008)(8676002)(4326008)(71200400001)(5660300002)(478600001)(8936002)(52536014)(110136005)(54906003)(7416002)(316002); DIR:OUT; SFP:1101; Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: PA4PR08MB6285 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: VE1EUR03FT040.eop-EUR03.prod.protection.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 760c01e2-b8e0-4bf2-eb02-08da67a1d729 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 0v4BGJe7RhKgOKwfomCxFYHWi1WbvgtPkglb6bbDD6ncUYO4pON8Yqt3ETeAQ4GUubzSo0HBeZZN52d+3EyPARiykCm1Ckur4viWRHRMWZ4ExOrNLDncTYij98rlQQfb7WuMAmemxaBFPt3CEbO0Shx8Pbsjit1X4hNbrADJWns9Sydg9vpUowh2wf/PkQJ/rR8mZHjtmAlQ6TE/x5i8u9gGdyXpT+Q3ytXoki6FeDq+PeI5ZYI3eYF1yOFDuFAM3v8WZSDzt0/KSemLt6BsvExKb7k3QSlZqQ9ve8ugYHfCwwtL1FU0Ll3b/oirQdS1sxsIcvdGu+tL4/PKr+Gx/GuiHdhV8RiYUO0sC8VfzkpB2U8zSS5TO1aSCW8xc7tgcmZ6QQfcq8aMX+IF0d1gqZqEeo4Aqs1FMUbdaOrU/AeahoXoBvdab6IRHTNo8MBXdLC8Plp5e2yu3lIh405ePTGD1mrv9X6WUre4QkYNGonTru6i9QU/2goFs0jLKMTwzKQU/wkXeevy93Ss2nfDnnnKlvcZm/RyENDw3xXjLFyoip7CPaf0+IjTHlhUfTwwtH/7xC/EHayBBttTesQJ6HhKtpW6vd33f/QKBb4xEZ+oOw1ydGuwNpRyKjViAdIXdggbJ3pXiNECRh2T//OPx8JWe+dLghiVltpvhs8wyZquA75oh6wSI4Hqy2TTtXRY5UnDeKulMm7AaqD5mS9+Wdf3bdxr1/qdZuXIPnM6nCEPATQmG/KTAa74BHna5FgjsLOis/RQlFWZBOTW5mJQornLlwMwVO92dQxTgScSNLPhsDmVU4b+DcLLtuS+T1cO X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(13230016)(4636009)(346002)(396003)(136003)(376002)(39850400004)(46966006)(36840700001)(5660300002)(9686003)(41300700001)(478600001)(6506007)(26005)(82310400005)(7696005)(316002)(54906003)(110136005)(36860700001)(86362001)(336012)(83380400001)(81166007)(47076005)(2906002)(52536014)(33656002)(4326008)(8676002)(8936002)(70206006)(82740400003)(921005)(186003)(356005)(40480700001)(55016003)(70586007); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jul 2022 03:10:14.2533 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 819d3dd3-36f9-4293-8779-08da67a1de51 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: VE1EUR03FT040.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR08MB4712 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > Subject: RE: [PATCH v1 1/2] eal: add lcore busyness telemetry >=20 > > From: Anatoly Burakov [mailto:anatoly.burakov@intel.com] > > Sent: Friday, 15 July 2022 15.13 > > > > Currently, there is no way to measure lcore busyness in a passive way, > > without any modifications to the application. This patch adds a new > > EAL API that will be able to passively track core busyness. > > > > The busyness is calculated by relying on the fact that most DPDK API's > > will poll for packets. >=20 > This is an "alternative fact"! Only run-to-completion applications polls = for RX. > Pipelined applications do not poll for packets in every pipeline stage. I guess you meant, poll for packets from NIC. They still need to receive pa= ckets from queues. We could do a similar thing for rte_ring APIs. >=20 > > Empty polls can be counted as "idle", while non-empty polls can be > > counted as busy. To measure lcore busyness, we simply call the > > telemetry timestamping function with the number of polls a particular > > code section has processed, and count the number of cycles we've spent > > processing empty bursts. The more empty bursts we encounter, the less > > cycles we spend in "busy" state, and the less core busyness will be > > reported. > > > > In order for all of the above to work without modifications to the > > application, the library code needs to be instrumented with calls to > > the lcore telemetry busyness timestamping function. The following > > parts of DPDK are instrumented with lcore telemetry calls: > > > > - All major driver API's: > > - ethdev > > - cryptodev > > - compressdev > > - regexdev > > - bbdev > > - rawdev > > - eventdev > > - dmadev > > - Some additional libraries: > > - ring > > - distributor > > > > To avoid performance impact from having lcore telemetry support, a > > global variable is exported by EAL, and a call to timestamping > > function is wrapped into a macro, so that whenever telemetry is > > disabled, it only takes one additional branch and no function calls > > are performed. It is also possible to disable it at compile time by > > commenting out RTE_LCORE_BUSYNESS from build config. >=20 > Since all of this can be completely disabled at build time, and thus has = exactly > zero performance impact, I will not object to this patch. >=20 > > > > This patch also adds a telemetry endpoint to report lcore busyness, as > > well as telemetry endpoints to enable/disable lcore telemetry. > > > > Signed-off-by: Kevin Laatz > > Signed-off-by: Conor Walsh > > Signed-off-by: David Hunt > > Signed-off-by: Anatoly Burakov > > --- > > > > Notes: > > We did a couple of quick smoke tests to see if this patch causes > > any performance > > degradation, and it seemed to have none that we could measure. > > Telemetry can be > > disabled at compile time via a config option, while at runtime it > > can be > > disabled, seemingly at a cost of one additional branch. > > > > That said, our benchmarking efforts were admittedly not very > > rigorous, so > > comments welcome! >=20 > This patch does not reflect lcore business, it reflects some sort of ingr= ess > activity level. >=20 > All the considerations regarding non-intrusiveness and low overhead are > good, but everything in this patch needs to be renamed to reflect what it= truly > does, so it is clear that pipelined applications cannot use this telemetr= y for > measuring lcore business (except on the ingress pipeline stage). >=20 > It's a shame that so much effort clearly has gone into this patch, and no= one > stopped to consider pipelined applications. :-(