From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4A5C7A0543; Thu, 22 Sep 2022 19:20:02 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CE64C40156; Thu, 22 Sep 2022 19:20:01 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mails.dpdk.org (Postfix) with ESMTP id 2F94D400D7 for ; Thu, 22 Sep 2022 19:19:59 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663867200; x=1695403200; h=message-id:date:subject:to:cc:references:from: in-reply-to:content-transfer-encoding:mime-version; bh=s4bsAO7F6xMvwdDgt3PZPqXsq47/k1D/XIXLLWHMYPk=; b=Ik3KFEeK7DLe/zToB8SUvJ69knOtkAKxt2YRSd8zHp4hmCcU+BnCktu4 gXrbVyDrtQ6rv4Jj1v1poEtJ/RU1OOwajupBAFePNPscNQ5e9Jahns0kj uWzDl5tbI+6P+HJqhGo1sz+j8BxlCiJun7dXOxmfKAwz4STn5oTeYR4YD TMGp/9+uLkB4qBC8NaAkilIdAD3LTRzSHywFWGI28o27tzOouBd2GyXb/ VZ9g+utPeDSiyacmroHijEvDD83lxFFwMTKDeCoE9nS/Wf8qY2c81rCNE p2LkOQmebihZvAWBGE8ErEGsNDg8/NoFUB4EXU5j9EjYnnFp+cl/qAmcL A==; X-IronPort-AV: E=McAfee;i="6500,9779,10478"; a="280090882" X-IronPort-AV: E=Sophos;i="5.93,337,1654585200"; d="scan'208";a="280090882" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Sep 2022 10:14:21 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,337,1654585200"; d="scan'208";a="653058436" Received: from fmsmsx601.amr.corp.intel.com ([10.18.126.81]) by orsmga001.jf.intel.com with ESMTP; 22 Sep 2022 10:14:20 -0700 Received: from fmsmsx607.amr.corp.intel.com (10.18.126.87) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Thu, 22 Sep 2022 10:14:19 -0700 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx607.amr.corp.intel.com (10.18.126.87) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31 via Frontend Transport; Thu, 22 Sep 2022 10:14:19 -0700 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (104.47.57.49) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2375.31; Thu, 22 Sep 2022 10:14:18 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FBm2KyWPDHoohYkfaN1oNLOBJIfzIQL1Nk3rOAWErHkraxrH/XBlNH1JIHkvYdPiNUq8d7+e9GNSHuiPVF31B2+SwPeMBw9rFNE+v2HFJNqMniqn6RCEcs90gp8U7Ky+N8Ld/DXLrDK7vwShj8gjbA/z3fyt2PBHLTY6x+ToMlyK/+MtMOGng7bwR1e/wu4/WjFz73M40mPVBsMSEmMmDV/S8FVE8M8PqEiFa6igacnSIOhP+5xXGlJjtVmQJ2jyzgReuly9skSc3S57J9KX96m+KEgCWxNUFqReueqOP0agEM/WQ9ldILTt/GNOH1VRePMpeDjq6JtV6FiKOY1p6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UtTKafq86MGWc9z2MpDguMTUKJ6K+4BoIHqHD2rVy/k=; b=mF8pP0giq1Bkz+kszb3wAOF0/SCS5Iryavi6bJhnkoGk5VduV2lgq/FrxZGp04SOD0q/AO0cmiH7HooA4l7axFySytjv15OL3uv7mOkhwR6oUX5ueohmsvmNXYQROYV++Y/x4WaZX0Dh32SSCqpJ5Vr8afuucpwa7IKDGPoGFzjk4ru3rJRoasifo371FQzV3TtGBA91srO8F4Ln4u2grAj3pJ46CYlC9LovrVKZ/guL712b38KkKp3vSAWjQA1fxIeKFKdqu6t1EGBeCAauhk1IX97y3yr+Nl2Z+u4xFUFF5FOldCopNNpIqYxvoIjpvDTYoZUvgdtbzQNS3YQ1zA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from MW4PR11MB5872.namprd11.prod.outlook.com (2603:10b6:303:169::14) by CO1PR11MB4978.namprd11.prod.outlook.com (2603:10b6:303:91::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5654.17; Thu, 22 Sep 2022 17:14:16 +0000 Received: from MW4PR11MB5872.namprd11.prod.outlook.com ([fe80::c17d:f1c9:e958:b5e]) by MW4PR11MB5872.namprd11.prod.outlook.com ([fe80::c17d:f1c9:e958:b5e%6]) with mapi id 15.20.5632.016; Thu, 22 Sep 2022 17:14:16 +0000 Message-ID: <4af4b5d6-57a9-39a4-2197-a2acdc57156b@intel.com> Date: Thu, 22 Sep 2022 18:14:07 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.3.0 Subject: Re: [PATCH v7 1/4] eal: add lcore poll busyness telemetry Content-Language: en-US To: Konstantin Ananyev , "dev@dpdk.org" CC: "anatoly.burakov@intel.com" , Conor Walsh , David Hunt , Bruce Richardson , Nicolas Chautru , Fan Zhang , Ashish Gupta , Akhil Goyal , Fengchengwen , "Ray Kinsella" , Thomas Monjalon , Ferruh Yigit , Andrew Rybchenko , Jerin Jacob , Sachin Saxena , Hemant Agrawal , Ori Kam , "Honnappa Nagarahalli" , Konstantin Ananyev References: <24c49429394294cfbf0d9c506b205029bac77c8b.1657890378.git.anatoly.burakov@intel.com> <20220914092929.1159773-1-kevin.laatz@intel.com> <20220914092929.1159773-2-kevin.laatz@intel.com> <9a6fec15f9684d21bb4730596cceacff@huawei.com> From: Kevin Laatz In-Reply-To: <9a6fec15f9684d21bb4730596cceacff@huawei.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: DB8PR04CA0003.eurprd04.prod.outlook.com (2603:10a6:10:110::13) To MW4PR11MB5872.namprd11.prod.outlook.com (2603:10b6:303:169::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MW4PR11MB5872:EE_|CO1PR11MB4978:EE_ X-MS-Office365-Filtering-Correlation-Id: 3e50d455-d0ad-4852-970c-08da9cbde0d3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: zCMvHil6CuAt0Xb3oDi6h/y/kpIxIpkZiLlyM0QlGDQWxRPFyJAJx/yuwEa7LCrR5wDi5dt/cSJma/HIZcMkmqNZP6BzOFMJPXXlZ0DDP/dvv/dblifym+mq5A2l0mPT2nc9DAo3RX/cesV+TeOj5/RtZYZvphQXHoy62/xPyOn4sV0w/ZaREwFuzcmvcHKMe+s56T71oeEg451S3cRytx5r3Hj9euiGFExT36znGP9NqBk31bnzaot3zwMl8iyRpslEZOMu/6DubBkrTMpO1J0nOR5EyFPXpYY+jpNP8lh3+73boOGudL4Pv3RkFJzX/4UUi5cyvzwKmqxLNnXtC8rpMzfxacQEprqf68i8kIe5/hHoqi/4aCKp4ZDdw/938LTR181VZXgd8XJAMlA5HbJrtXSOwPCrzMIqz7iUIg1H1vyXVJtV8sZ/uOtcFJ2YiwXOxWLRIy787vvjDO0Waoe/Xq1pUBX2eAbwWuDSa+jXnS+evnRnR48wGcwwrcplYJtPqlTaSjvR8R5kbygrF1NwxtH0a3kXSTI44GJb9RKU1GeuygSA1e7XPjjyvKN42V1KcmhfWCvNjY3+vo+2M2hlniGBoPgtjeta1h8dh8ve7n3nXo+hyPo1yeceWdFkcofQPYWaYPeHOPM92FDntM4jqUBmDNc78ijIDYBBf4VRMjTf7jBxmggVIU0QAlMb78OugcP0E+le2i7/+rasosRVJIbr4793O+LA0jcRAalCwqEcGOdf5Lxbpd7g8K89MREOhWYOrkbujKFeXLU0YP7Odm1amupy66aj6WrvVfc= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MW4PR11MB5872.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(136003)(376002)(396003)(346002)(366004)(39860400002)(451199015)(31686004)(316002)(54906003)(44832011)(82960400001)(110136005)(8936002)(5660300002)(6486002)(478600001)(4326008)(38100700002)(41300700001)(36756003)(86362001)(66946007)(53546011)(2906002)(66476007)(7416002)(26005)(2616005)(186003)(66556008)(8676002)(83380400001)(6512007)(6506007)(31696002)(6666004)(43740500002)(45980500001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TVFrYk94bG1TTmV0UTk2cXR5akszSndCRWVQVVJmVURLdDlTcmwvKys3dFl3?= =?utf-8?B?eUVMK1ZVczQrL212Q2RHWDJiUWJtakpORk1VdnJHbmhCZFlaR2JLQUhNcUIz?= =?utf-8?B?Z1J2YU9QZWFwaXZRUHhrOFlYWEZMMzZjSUM3OHRXbUtKRFVnbkNDcHBRb1Nj?= =?utf-8?B?bG53WENOaStIMU1sT3VobS9YNnNma2dod2tEWitHd2FwYmlwZ2E3OXMrY3NV?= =?utf-8?B?Y1dMZlg0eGtNUDJObFlOUWhNUzA5dXU3bVgxV2o2bU8zOEdXMWpqbzN5ZUdP?= =?utf-8?B?SklqUG5uQnk3QVAvOEdmbTJEMzBndWI1K1FMeGhrK0dHM01PeHVIRHEwUkdz?= =?utf-8?B?TmpyMkltUXFlazU2K3U4ZHV6dnR6UU5pVnBtd0srK1RINkErZ3BsaTZOblhC?= =?utf-8?B?dThOR1RJblh5SmFnc0FrOTJXWngrak5Wa1hLcWhZc1NTdUZxQXU0VW42OFZM?= =?utf-8?B?QXk5NjRvL0NJMitEWnJyaVNEV253cHpERiswSHZHWnA4VDJJQkxrbS90U0du?= =?utf-8?B?Y2h1UzhTS1hPZVdIbnFwTzdhVTFIN2E2RTBPSWQzb3pnWFZOTXQvblZIdGFm?= =?utf-8?B?RlZEa0wwY2xndlc3Y0JrUWNMcWNkMjB2Q01FMGVmaThZa3M0NTdoMzhuQjlB?= =?utf-8?B?NlBpUHVoOEtlMWtmSkR4VzRZMVlGZlo2d2lRU2cyMldZNDU2eDkwMXhpOU9k?= =?utf-8?B?QVF3MFJMWHpoZUdqb1R5eXU1eGw4UG5XNGx4eWRmazRpWTVQbTV6RG1kRWIz?= =?utf-8?B?OUxHbEswNGNPZllsWVQzZ3BPb0p1VWROeW44alpEai9jNlIrL3ZhelYvbFBB?= =?utf-8?B?OUdHZkx2YnorM1lvMVFVRUVQTERoNDQ3L09wQ2NPcXl2eFo3Z2pCOUROMUU5?= =?utf-8?B?ajhJdU1aL3dCbTFmdUxtNnUzdTFnNjZ6OU1sMUJmenIzejYwb3RQR3FpNnVN?= =?utf-8?B?VzU1TVpoL2E3MHR3Q3A5UkduUTVLdEgzNVphaXY3aDJkeHBZeDlNODc5bHVk?= =?utf-8?B?RTdrZUU0eUc1SzYvTEJOUm5xQkNFYjRtZzhBMVNXejlFRzl6L1pXQ3FKcXhv?= =?utf-8?B?RDRlRWJUSnI2RTdWOUxnNzBQVU0yMkZCaTZnTHJZV1VaU3hyMzg1Z3NNL3BJ?= =?utf-8?B?Q0VpWFVzR0x0ekFhREltVXQzNWVZblFoc0Nnc1F0eVRUVjNWOEhlK2txanBB?= =?utf-8?B?Y3dKWVBNZjFpT3JpaUV6Qk8wamNFaDNTMHJZMTlWRkNXQVM1Y1NoMk1EeS9o?= =?utf-8?B?dnhwV0NYeEdkWE1INmVXRS9FMVlwUjZYejk2OEd2ek90Z1ZBSmNKRHdxTUhu?= =?utf-8?B?RHFsNFBDa3MzQko5aFVyN1pmOWlUa3hKclFIMXhFamdIcml5anlTYmJvWnBJ?= =?utf-8?B?NWFCVHRuenZRbFhKMmtNQjlLL1lIdFpscitGSDdtb3JyTHFCRTBsU2Jnajlt?= =?utf-8?B?Rkg1dnhUV0wxa3daQ1pGRmo1TjN3NHZWc09RT1VuTFM5VVV0VFdmb1JzZkhu?= =?utf-8?B?SmExQ3gzOE5wRE1OTk1SbE1KWktSY1hrbHdVYW5kai9Rci94YVZZblRHQ29M?= =?utf-8?B?R1NOSStkR3BoNnpIbW9pTDFpT2N5NUxoR0dIOUJITTVndXgzOFFuNWNYbzJ5?= =?utf-8?B?dituQUtMY3QvVk1Hbkx5R3NXQnkzRjNGUUhYaDBYY2RZdG9JVlRseG9wdmdz?= =?utf-8?B?ZTlEdEF4RnlOTWdDa1VVNEZhbm0yK0E5OUt6MjZxd2ZIQmN4SkJtcTBLbTNQ?= =?utf-8?B?b3ZOblNoTFB2L2dsRS85VXBnbmcwa09lSEZQSnQyQWtYZUkwSWNVR0hCUng0?= =?utf-8?B?K0dNZWpTM3NIK05PUTNnazR2V0xXeWlTTzBxbDhNaEszTmQzRUpPNXpaU3lv?= =?utf-8?B?RGk2emtObnNobjJONUNFSmFoWHlIRmpZcUJFcjNuYzBTY3hiN3JTQXFJbTI0?= =?utf-8?B?VW9YVWw1cDRjVHBPNFpXem5nc1BiSWV4K0JxYndvdTN3NktnWHk5Sk1KZzhy?= =?utf-8?B?T1ZBV2ZTTWJiejZOYWIvWDhKdTdicnRiMyt1WXJPM1J4NmF0THNnQ2NlNHVU?= =?utf-8?B?ak5Ib2xTL2lTZE1vUEw5aWtnb1Ird1hyQnVucThCcGNYWHZrVTh3elhSSHZV?= =?utf-8?B?bysyS1dLYWw5SXNOV0s2NU0vRngvV24yU2s0Mis4VDhWRHh0UE1YVVBIek1C?= =?utf-8?B?QVE9PQ==?= X-MS-Exchange-CrossTenant-Network-Message-Id: 3e50d455-d0ad-4852-970c-08da9cbde0d3 X-MS-Exchange-CrossTenant-AuthSource: MW4PR11MB5872.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2022 17:14:16.5904 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: nL5uMhmluO7matNuydBB7Vidj22/xEkmk0w73byQbHQ8iMnoH+wWlRhADdU8UMbe3omKNGBsk1Q3i/egdtqYVA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR11MB4978 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 19/09/2022 11:19, Konstantin Ananyev wrote: > Hi everyone, > >> From: Anatoly Burakov >> >> Currently, there is no way to measure lcore poll busyness in a passive way, >> without any modifications to the application. This patch adds a new EAL API >> that will be able to passively track core polling busyness. >> >> The poll busyness is calculated by relying on the fact that most DPDK API's >> will poll for work (packets, completions, eventdev events, etc). Empty >> polls can be counted as "idle", while non-empty polls can be counted as >> busy. To measure lcore poll busyness, we simply call the telemetry >> timestamping function with the number of polls a particular code section >> has processed, and count the number of cycles we've spent processing empty >> bursts. The more empty bursts we encounter, the less cycles we spend in >> "busy" state, and the less core poll busyness will be reported. >> >> In order for all of the above to work without modifications to the >> application, the library code needs to be instrumented with calls to the >> lcore telemetry busyness timestamping function. The following parts of DPDK >> are instrumented with lcore poll busyness timestamping calls: >> >> - All major driver API's: >> - ethdev >> - cryptodev >> - compressdev >> - regexdev >> - bbdev >> - rawdev >> - eventdev >> - dmadev >> - Some additional libraries: >> - ring >> - distributor >> >> To avoid performance impact from having lcore telemetry support, a global >> variable is exported by EAL, and a call to timestamping function is wrapped >> into a macro, so that whenever telemetry is disabled, it only takes one >> additional branch and no function calls are performed. It is disabled at >> compile time by default. >> >> This patch also adds a telemetry endpoint to report lcore poll busyness, as >> well as telemetry endpoints to enable/disable lcore telemetry. A >> documentation entry has been added to the howto guides to explain the usage >> of the new telemetry endpoints and API. > As was already mentioned by other reviewers, it would be much better > to let application itself decide when it is idle and when it is busy. > With current approach even for constant polling run-to-completion model there > are plenty of opportunities to get things wrong and provide misleading statistics. > My special concern - inserting it into ring dequeue code. > Ring is used for various different things, not only pass packets between threads (mempool, etc.). > Blindly assuming that ring dequeue returns empty means idle cycles seams wrong to me. > Which make me wonder should we really hard-code these calls into DPDK core functions? > If you still like to introduce such stats, might be better to implement it via callback mechanism. > As I remember nearly all our drivers (net, crypto, etc.) do support it. > That way our generic code will remain unaffected, plus user will have ability to enable/disable > it on a per device basis. Thanks for your feedback, Konstantin. You are right in saying that this approach won't be 100% suitable for all use-cases, but should be suitable for the majority of applications. It's worth keeping in mind that this feature is compile-time disabled by default, so there is no impact to any application/user that does not wish to use this, for example applications where this type of busyness is not useful, or for applications that already use other mechanisms to report similar telemetry. However, the upside for applications that do wish to use this is that there are no code changes required (for the most part), the feature simply needs to be enabled at compile-time via the meson option. In scenarios where contextual awareness of the application is needed in order to report more accurate "busyness", the "RTE_LCORE_POLL_BUSYNESS_TIMESTAMP(n)" macro can be used to mark sections of code as "busy" or "idle". This way, the application can assume control of determining the poll busyness of its lcores while leveraging the telemetry hooks adding in this patchset. We did initially consider implementing this via callbacks, however we found this approach to have 2 main drawbacks: 1. Application changes are required for all applications wanting to report this telemetry - rather than the majority getting it for free. 2. Ring does not have callback support, meaning pipelined applications could not report lcore poll busyness telemetry with this approach. Eventdev is another driver which would be completely missed with this approach. BR, Kevin