From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 09E08A0032; Fri, 15 Jul 2022 15:35:54 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A309140A87; Fri, 15 Jul 2022 15:35:53 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id C9DE340696 for ; Fri, 15 Jul 2022 15:35:51 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657892152; x=1689428152; h=message-id:date:subject:from:to:cc:references: in-reply-to:content-transfer-encoding:mime-version; bh=L17SHB2ACeO3Jz5LvgtQ2koOJ/1fs2XXqZ6yYgfdqRY=; b=YuRJv6Bw81qf5TkbAqFkpyGFoI63Yf+dK1yOPlD+hpe7TziC5EL4KofJ qv4QRlz2pwBNVWkXAhFTKUKCWOyn9hBfWCsgjXxY0HcjUkKrqvxb/BcJS JJ8lr5fFH1mNxlGgt/5xJUqxzQxBG00IV/Jlw94Y1L14l1O8GmCQMbWqp ubz6Hm5lC6ODLEMlFBYLpUOhqPr9NWbCZcgW8TxUlcCHw83S4Ns7fXOGb v3b+a42fqtRPpFAUBGMrAQ7lITduFA2KYwUqMjYtkTleAuwMbnCgZtcIM GEwAZwS5jOCxJDzCbI8nePInnI0uo5E50MiaqcjzcPcZFX/s82QomQHY3 g==; X-IronPort-AV: E=McAfee;i="6400,9594,10408"; a="285819562" X-IronPort-AV: E=Sophos;i="5.92,274,1650956400"; d="scan'208";a="285819562" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Jul 2022 06:35:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,274,1650956400"; d="scan'208";a="546657627" Received: from orsmsx604.amr.corp.intel.com ([10.22.229.17]) by orsmga003.jf.intel.com with ESMTP; 15 Jul 2022 06:35:50 -0700 Received: from orsmsx608.amr.corp.intel.com (10.22.229.21) by ORSMSX604.amr.corp.intel.com (10.22.229.17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27; Fri, 15 Jul 2022 06:35:49 -0700 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX608.amr.corp.intel.com (10.22.229.21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27; Fri, 15 Jul 2022 06:35:49 -0700 Received: from ORSEDG601.ED.cps.intel.com (10.7.248.6) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27 via Frontend Transport; Fri, 15 Jul 2022 06:35:49 -0700 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.102) by edgegateway.intel.com (134.134.137.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2308.27; Fri, 15 Jul 2022 06:35:49 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HVn1GaC0o8yy8kxqBLFNxAH6RalnfPt4rvbZDFatE2m2U0Lxl9+59WTSg5Gw2v30A0D+285AJfH3O7afG/k9Se0DCOB4LzVZG8uzuxlzzaBIOv16ze9G56AyvTG6Dw1ckfxOqd2yO2lXe29hZbnyzQuUEiZUGitn5goUMRaURRxhmcT7UeT8TCA1v/WlgA8htOkxtlIf4EJgd/bhfU+HpVJP62S03Ktd944n0CJmzsvE+Ov0h+yPDOVZYATJXloER+U2EkZsUSu9PVBjbssHQp1CFyPIv/rTsAznq1J+RGD7rnjv19f81K6KX+ABoX1EfhWVCL9J0FcghL0B8GL0sg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=5f4JA3+udMa216+beJAF45G2I+HE8fJzIDTxrp2w6Uc=; b=cDdCZdVMN1BudSbXvweCwR7Hu0J/baaH4iYolMoU53iV958kXnaaHfg+BhNXnlqkYU/N8F57kkXLdK6BQOWnNrqDTHyNY0cvaS3bTqd9j3JEMO1Lyt0F2inw7ewjXxEWD7XHQy9M8m/NaJFgAGO67FiC8yxudjJFWy5RIR8Gypdknvm3Y1XILYvyW2nCozIdC1dngTDoP2eomMBJq/mbkIUBqZ1CFQh+NgNeEhgUgGVs5bPHmTGNr2XkNVxYrk3Z7PFHxIIsJCwJCvUt9gjfKAVEXvaR6YNniVSTERku7lBfPIzqsAIPe6a5HmNWfNJc6gheF9FINDnxGJqXSRMdrg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from BN6PR11MB1251.namprd11.prod.outlook.com (2603:10b6:404:48::10) by DM6PR11MB3883.namprd11.prod.outlook.com (2603:10b6:5:19f::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5438.17; Fri, 15 Jul 2022 13:35:47 +0000 Received: from BN6PR11MB1251.namprd11.prod.outlook.com ([fe80::128:8fb:9d0:1a9f]) by BN6PR11MB1251.namprd11.prod.outlook.com ([fe80::128:8fb:9d0:1a9f%8]) with mapi id 15.20.5417.029; Fri, 15 Jul 2022 13:35:47 +0000 Message-ID: Date: Fri, 15 Jul 2022 14:35:41 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0 Thunderbird/91.11.0 Subject: Re: [PATCH v1 1/2] eal: add lcore busyness telemetry Content-Language: en-US From: "Burakov, Anatoly" To: , Bruce Richardson , "Nicolas Chautru" , Fan Zhang , Ashish Gupta , Akhil Goyal , David Hunt , Chengwen Feng , Kevin Laatz , Ray Kinsella , "Thomas Monjalon" , Ferruh Yigit , Andrew Rybchenko , Jerin Jacob , Sachin Saxena , "Hemant Agrawal" , Ori Kam , "Honnappa Nagarahalli" , Konstantin Ananyev CC: Conor Walsh References: <24c49429394294cfbf0d9c506b205029bac77c8b.1657890378.git.anatoly.burakov@intel.com> In-Reply-To: <24c49429394294cfbf0d9c506b205029bac77c8b.1657890378.git.anatoly.burakov@intel.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: LO6P123CA0019.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:313::13) To BN6PR11MB1251.namprd11.prod.outlook.com (2603:10b6:404:48::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ae1d568d-c417-49ff-34bf-08da6666eca2 X-MS-TrafficTypeDiagnostic: DM6PR11MB3883:EE_ X-LD-Processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: st5yrqL4BM1M5YXo5hbUOvcWoXtcFhUk9eNzeAvDzxyXHsoJJTPMiO38LvZUAIc1uSfgNOSbLpTB7lsD9/mddaeT+ZT/G1WPMbsxeAotdhBP2vFboPAGXn2SxVlOP+qEr9XNuxx5yJfvLS465MNW9UjIocsKf09DyDmM+ya1azfo5KPMM06/KvAerVSL7ApG4DympToBeH77W0XMoSEWNDm6WvdfkcQ+3VIawi8gOiYY22Wia5GUjwK2Tdy1arJZ5eNgEc4GQK2hLVZijxMbljb7vH1cfFCOnkOLfBVgRebHOjOtphxciP99AMwz99ddF/gb2HI0xNmJ1KEocd6mEbPBDI+cMUgU1flW4+X4/xdzAHkruRluujiTvge3WDLDfcJBJ0onv2tAbgoMTp92J32EiaQ3DzDHBq/E5ljPQJ8ln2iqXzfl7qGoP1JrLcG+6XZRtSn/ta5FI2RqFYzP249Mnl4DlVOaAPjclhtC501E0YA3d2MrokCyQqhZx9aJNXhIXR2pNB8a5prlTDYWV2yhfQz5vuwF8MePUkZsEtBz+9RKBhDfWDe58vbAzdynWyLnwKDWQCOqMeXMN4udnMAHxVQ6xm47QfzSCmFaMA0ctRrLe0Aia5jHatJMR8HFu8nsRhPpWFpXX0ofc0KjP4MgiDx3lNFrIWB2F45z3TZUE5mSkh7r+3yxuXpIO+kvFgKGM7OP6sTR51VtdYuRQHZ6mtJYSmB6otZ8IdLQQlT1/XWOEhwEjkgyN//fbls+vGCu2BK466r9P9yjZwHc9v+ZPPbsN5aoRArYx9Ym4hfyEYOqGCEAZiiHgfzY4ZpcgZYkyybOp9P/eScze76QDbezIabvnk4M5YNpDBcIZe5XAcSgXTy8RYfpjbNS7Vqw X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN6PR11MB1251.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(39860400002)(376002)(346002)(366004)(136003)(396003)(86362001)(82960400001)(31696002)(921005)(7416002)(5660300002)(6512007)(107886003)(2616005)(83380400001)(36756003)(38100700002)(186003)(31686004)(110136005)(478600001)(41300700001)(316002)(6666004)(53546011)(26005)(6506007)(8676002)(4326008)(66556008)(66946007)(66476007)(8936002)(2906002)(6486002)(43740500002)(45980500001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?WWdsb0RxWUFwRFhJalBNSVN0SWErQW9DOXVIQ3FoRVhwbGxLTGF0SDNjd2Jq?= =?utf-8?B?c3ZGbDZ2YUFTSm1vRm95SVVpSWNrY3NrdFdDdFIwNUxrajBQQVZrZVgreXNZ?= =?utf-8?B?NWZhMFBGQW1PZkxxd1IwQlp4dFMwbXBYU1Znc2JrVnhSRzhjZHhpV1NLRy9p?= =?utf-8?B?RjMxTVEwa1NxSU9EVFhHOVBESnRIN2VmZytIVkJtVFBiYXgyZlpHT3pTNFN1?= =?utf-8?B?NHZxVC9NZFc5MVlTekpxSDBTL0pyZDJhSGFrUHZTamlNN2xHclRKdmJFNkU2?= =?utf-8?B?TlJjR1JDSHNnSTJ0M1kreEtZM2tqMTJoWEtPMkNkcGdJVGFIbWhQYys4K2kv?= =?utf-8?B?aGxUd1FMaW1hYWozcTJnWUY3bHNTUXRVTHlPS0o4aUxRSStsS1BJT0JaL2xD?= =?utf-8?B?UjBlenF5MWxYSUppM1R3R1phVmpNZ3Rxc2NLdWVQNkpFM1BRV08wSTNXSjl0?= =?utf-8?B?NGlhM2d3bmNIbENuejV4WWdiUlpBV0lSNFhLRk16eGo2SFpLN0xpKzFBNjlB?= =?utf-8?B?RkQxSWovemtleVBobW5HSTFYWUxtQ0VQcW11VnVmYjR0M0ZzQ1pPQzFEZ3Js?= =?utf-8?B?RXRaSHlYZmM1R3lXazdlV3JtRHh1STZ3dldZZzlEaW1nc09Ka0ZtSXlYTFc5?= =?utf-8?B?REV2RFJXZDBQU1BLL1lDYTdSZU0zMEpERElTZk1oOTcvWGswc0JPc1ZraWh1?= =?utf-8?B?ak5HMWJuemd3Y2ZDWkpERFVOWFl6TDNTWDNoQU9OZDhMMXljY3ZkdjdySXR5?= =?utf-8?B?ZEg3WUJDMklpUnNLc0dzM0lQNHJBeXFPOURyZVgwYmdlMGZtM1NpTUplSktE?= =?utf-8?B?VjlpSlVzaElyRkZuNVBiTzdrTTk1L0U4Njdwd3BFaDdHaFV3ZVJiT2xUSkow?= =?utf-8?B?eDJiR2FUQ0xybUw2Yi9qbUI3UzZNY0I4UW1xNE9aZ0JhSzRGbkM2WGNabDNj?= =?utf-8?B?dUlMTkxUd0R3N0lkSkhWMXF0ZVhJS0ZNa0NMSVVHckxPSGg4OXQ0dEtCRHV3?= =?utf-8?B?L1Y0aC9SREgvdEhsdUhUZnpBZjZibVNydzUvTC9rcEcvZERSazJyQ01VcjBP?= =?utf-8?B?V0htUHJHclppSU9LOURBTG9VdWlNcVV3SUZ3RTBhbzIvL0hiSUxvRXZEVEph?= =?utf-8?B?OGVJeXpzYnY4QVpkTmVBVDdsbzUwakNYQ1NMMEowUUpvVTAvTitrVm5Ea1k5?= =?utf-8?B?aituTUZWZFBPVFVpY3gzNmRWbUloNGwrZ01DWGxkMDYweGxJbG51Uld4RzNq?= =?utf-8?B?cldZQkQ5V0NMVXhTUDBhRHdsTmtVZkNOck5UZ01Uc0QyaThZOVd5TDV4Zm1q?= =?utf-8?B?SGg5ci9GVldCZkRnVDFaVERxeXBHcEo3R1FiZjdBaXh0WGQxOHhDV3hQd1RF?= =?utf-8?B?M25QVExMdGlpaXRpcCs0amd2N1hLQmkwT1ZLSkczcWtXSlV2dU1SUjVGcUxk?= =?utf-8?B?V1ozZUFra3JaQ000MEdMNzFpSEJjdit6c21vZGwzNlpMOGJIL2tYZlVsRDFI?= =?utf-8?B?S095LzdDemtGSzVrYzFTL3RxVGNneHJDeGhPMTdZQ2VSUnMwa2ozd0xzTk15?= =?utf-8?B?b0Q4cjdudFlzZkVBRnlNZTdrNnZCWXFUdllnbzBWWnNTYWhoa3FZdUhpY0Q2?= =?utf-8?B?N1JEeExHT0dVeENsSjJRY3RkanR2Mm51bGZvd0Y4U2ZyZUpaUnhlOTVIVDgx?= =?utf-8?B?T2RmUVpDYllaS21aTVZwdE5WeHczYlIvR2FXazUyZkRVSEZoRmhWeUgzWE1l?= =?utf-8?B?czNBcWU5Y0hsZ2FtQm1pRmdoNGpveTk2cThLcHZ0Qm9ZTDNoYkd3UHBWUllI?= =?utf-8?B?Z3MzaHM1VVpzVElZcGlKVEkvQXQxbXBoN09iaWoyYW1VbE1jTzhzbGwwakRN?= =?utf-8?B?SjhuTjFIU1EzcmErTUkxeUpFdFdhNnlRZVdiK0UyVWxvQ0IxR2wzLzV3MElT?= =?utf-8?B?SWRnWExQcjZ5YnVwWjY4dDI5M3ZoYzJnbFFKMThkaW8xOFZabllMUkt2NTU4?= =?utf-8?B?Q05wWTJlamh4VkFHUHRnUWxWZ2FFN3I5WlhrUzZNekdiZFYyT3A2WWtrcHNO?= =?utf-8?B?T25SWm9TNlgxYmF4R3NteUNNVDJOOEQxMStKM25YUjNMQ1hSRGtsSGVqUTFr?= =?utf-8?B?SUROMzM2REhzcWxySEdQdlNFMkY2ZTBrSEFweHJNdHdYQ0lVT1JnbGw3VnB3?= =?utf-8?B?OEE9PQ==?= X-MS-Exchange-CrossTenant-Network-Message-Id: ae1d568d-c417-49ff-34bf-08da6666eca2 X-MS-Exchange-CrossTenant-AuthSource: BN6PR11MB1251.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2022 13:35:47.1753 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XkigseAJG7gSOTFNuFxpV+zrghlPZA99fiw14olQsutg+06Mov4bi+OKcdlGONQ5wxpq0hAAX9gDVKKxkm5uMMS6m5spEq1X3U/SOJ4SBKU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB3883 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 15-Jul-22 2:12 PM, Anatoly Burakov wrote: > Currently, there is no way to measure lcore busyness in a passive way, > without any modifications to the application. This patch adds a new EAL > API that will be able to passively track core busyness. > > The busyness is calculated by relying on the fact that most DPDK API's > will poll for packets. Empty polls can be counted as "idle", while > non-empty polls can be counted as busy. To measure lcore busyness, we > simply call the telemetry timestamping function with the number of polls > a particular code section has processed, and count the number of cycles > we've spent processing empty bursts. The more empty bursts we encounter, > the less cycles we spend in "busy" state, and the less core busyness > will be reported. > > In order for all of the above to work without modifications to the > application, the library code needs to be instrumented with calls to > the lcore telemetry busyness timestamping function. The following parts > of DPDK are instrumented with lcore telemetry calls: > > - All major driver API's: > - ethdev > - cryptodev > - compressdev > - regexdev > - bbdev > - rawdev > - eventdev > - dmadev > - Some additional libraries: > - ring > - distributor > > To avoid performance impact from having lcore telemetry support, a > global variable is exported by EAL, and a call to timestamping function > is wrapped into a macro, so that whenever telemetry is disabled, it only > takes one additional branch and no function calls are performed. It is > also possible to disable it at compile time by commenting out > RTE_LCORE_BUSYNESS from build config. > > This patch also adds a telemetry endpoint to report lcore busyness, as > well as telemetry endpoints to enable/disable lcore telemetry. > > Signed-off-by: Kevin Laatz > Signed-off-by: Conor Walsh > Signed-off-by: David Hunt > Signed-off-by: Anatoly Burakov > --- > > Notes: > We did a couple of quick smoke tests to see if this patch causes any performance > degradation, and it seemed to have none that we could measure. Telemetry can be > disabled at compile time via a config option, while at runtime it can be > disabled, seemingly at a cost of one additional branch. > > That said, our benchmarking efforts were admittedly not very rigorous, so > comments welcome! > > config/rte_config.h | 2 + > lib/bbdev/rte_bbdev.h | 17 +- > lib/compressdev/rte_compressdev.c | 2 + > lib/cryptodev/rte_cryptodev.h | 2 + > lib/distributor/rte_distributor.c | 21 +- > lib/distributor/rte_distributor_single.c | 14 +- > lib/dmadev/rte_dmadev.h | 15 +- > lib/eal/common/eal_common_lcore_telemetry.c | 274 ++++++++++++++++++++ > lib/eal/common/meson.build | 1 + > lib/eal/include/rte_lcore.h | 80 ++++++ > lib/eal/meson.build | 3 + > lib/eal/version.map | 7 + > lib/ethdev/rte_ethdev.h | 2 + > lib/eventdev/rte_eventdev.h | 10 +- > lib/rawdev/rte_rawdev.c | 5 +- > lib/regexdev/rte_regexdev.h | 5 +- > lib/ring/rte_ring_elem_pvt.h | 1 + > 17 files changed, 437 insertions(+), 24 deletions(-) > create mode 100644 lib/eal/common/eal_common_lcore_telemetry.c > > diff --git a/config/rte_config.h b/config/rte_config.h > index 46549cb062..583cb6f7a5 100644 > --- a/config/rte_config.h > +++ b/config/rte_config.h > @@ -39,6 +39,8 @@ > #define RTE_LOG_DP_LEVEL RTE_LOG_INFO > #define RTE_BACKTRACE 1 > #define RTE_MAX_VFIO_CONTAINERS 64 > +#define RTE_LCORE_BUSYNESS 1 > +#define RTE_LCORE_BUSYNESS_PERIOD 4000000ULL One possible improvement here would be to specify period in microseconds, and use rte_get_tsc_hz() to adjust the telemetry period. This would require adding code to EAL init, because we can't use that API until EAL has called `rte_eal_timer_init()`, but it would make the telemetry period CPU frequency independent. -- Thanks, Anatoly