From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4E24F42A50; Wed, 3 May 2023 17:34:39 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CF95241144; Wed, 3 May 2023 17:34:38 +0200 (CEST) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by mails.dpdk.org (Postfix) with ESMTP id 09BCD410F9 for ; Wed, 3 May 2023 17:34:35 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683128076; x=1714664076; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=yQ5GeZy1EuG49bJYD0e9Qs75azJe+DzDEZcFPvir5gg=; b=fa+LZ59se94MW74QxEMGyw5+PwWrE2WudCp/24LEelnu3CF1Zm/4xemk lJW5DG4Flbu8DdFyPxs+Y47benxjkOtk2D1i+K/uYDrOjZe6wWdYdm9aU IhDMFeolIjw2lkZ8/O9XpTJcMNEigiMpueSjH1zIGxV/qb2ynb92qqz0k hGZNCMLcdBBJ5tgz8ejuAiW2NpwyNmAXhpx3yYRq+Emx1PdQA1WWxQl47 dk5GZWrNiPynXFGBGfRSrvcnsyMkTk6LiZS0dtI34BYxeH72WdBPN4jGd gPFHFSC9B7G3tdiPQGBbswAe1zpDf2tSyIUmbH6rdcY0sqhNMySlnWO8i w==; X-IronPort-AV: E=McAfee;i="6600,9927,10699"; a="376750636" X-IronPort-AV: E=Sophos;i="5.99,247,1677571200"; d="scan'208";a="376750636" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 May 2023 08:34:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10699"; a="729377774" X-IronPort-AV: E=Sophos;i="5.99,247,1677571200"; d="scan'208";a="729377774" Received: from orsmsx602.amr.corp.intel.com ([10.22.229.15]) by orsmga001.jf.intel.com with ESMTP; 03 May 2023 08:31:50 -0700 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX602.amr.corp.intel.com (10.22.229.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Wed, 3 May 2023 08:31:49 -0700 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Wed, 3 May 2023 08:31:49 -0700 Received: from ORSEDG602.ED.cps.intel.com (10.7.248.7) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23 via Frontend Transport; Wed, 3 May 2023 08:31:49 -0700 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (104.47.66.45) by edgegateway.intel.com (134.134.137.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.23; Wed, 3 May 2023 08:31:49 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Sx0N3+LJSaEr45EmzBFgKa4I1bkbD14n/OYyNwqrSqaHdvSv9E1j4uZonK4d+AGUeKROaNmc/bBJTw1V4pcSH4MR2tiZXOsnGqTvsGOreFgipOmyPrO78bGBTEv16Lo9CIFtqUn26Dy+/1IW1Uq8nvbsfRuAO0GWnXog9g+UM7GBSdHslIz6f27I68+HyiyHIGRfRlqLgaVZwO0NdMe4DsKODpKnzqUml0GFcgDZwa3whcgn6BHGLNKzvhdzkuIGi1FA4GvOsePldN+tsDKNiNlx2AUG+HhzjKKCnOdTd43VdymVWgvKO0qg3OWSxZIhs3940YZj0z7tENohZART0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yQ5GeZy1EuG49bJYD0e9Qs75azJe+DzDEZcFPvir5gg=; b=OY9IQSnIdPBEnv2rByTQLx+DMVZXtPaVk/92a7WCMzNyNU/byEa1uwD/HmI5+0vhfppgPxZpwzfDRaoQpyXVhEag8N8j9w1EsD5zudDqn0O+sDCJ6HiUKkiVSKrj51XGSZ+oLrvp85O3WMzEz4Y6Ay5K3++hTtGabOOGx8xfgEy5jSEXU7YU1N74kcywr/AhyMqZ0TIu2oT1xrWou/rfUplGVJu12uZ2N0srZKUwMVQYeV/zVjWTq6yBb/LcvKr5l+RuqzQHfJoRhLOKQxGFV9c2NPcwR+NPLCLFFmj+FSeQDWdl4byN/8+0SI46WSlbc2cwmWMYguED3xRL3JrAnw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from SJ0PR11MB5615.namprd11.prod.outlook.com (2603:10b6:a03:305::17) by SN7PR11MB7511.namprd11.prod.outlook.com (2603:10b6:806:347::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6363.22; Wed, 3 May 2023 15:31:47 +0000 Received: from SJ0PR11MB5615.namprd11.prod.outlook.com ([fe80::bf38:f8ca:62e8:e31a]) by SJ0PR11MB5615.namprd11.prod.outlook.com ([fe80::bf38:f8ca:62e8:e31a%7]) with mapi id 15.20.6363.022; Wed, 3 May 2023 15:31:47 +0000 From: "Coyle, David" To: =?iso-8859-1?Q?Morten_Br=F8rup?= , "dev@dpdk.org" CC: "honnappa.nagarahalli@arm.com" , "konstantin.v.ananyev@yandex.ru" , "Sexton, Rory" Subject: RE: [RFC PATCH] ring: adding TPAUSE instruction to ring dequeue Thread-Topic: [RFC PATCH] ring: adding TPAUSE instruction to ring dequeue Thread-Index: AQHZfbPnG2/msW+xn0Gzh27kyZT0UK9Ii/+AgAAP7eA= Date: Wed, 3 May 2023 15:31:47 +0000 Message-ID: References: <20230503113840.11010-1-david.coyle@intel.com> <98CBD80474FA8B44BF855DF32C47DC35D878DD@smartserver.smartshare.dk> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D878DD@smartserver.smartshare.dk> Accept-Language: en-IE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SJ0PR11MB5615:EE_|SN7PR11MB7511:EE_ x-ms-office365-filtering-correlation-id: 4daf7952-e366-4216-5c6a-08db4beb8239 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: bpCG7VQgFf8luiJyRywaszfG5m1M/2/5/hIWnrJgUHEzYgKhw7emd/iMjcOQU/XYeGvIgYf3QtFQ1hQbbneP5Mc3zLVzCfkuk+WZgDJG8TxjHrMb/eJXtPAsDzIZfNFqjO/Zoyt3BLpEkfhQ7burbT8SS8DxbdsdctpUbxB/UfOvcQBfM56h57nSE2GhO8v1vKBriDnXuriPpxFq8p78QIUGbLepHs4lUROZiK4CZyrKRucBWqUMXmc1EC//QFWS+faYaMW9xszaHSx3G/KHjeKrhynhj4EiCGlW9Fph3eTMK2PPLZXOuquKwl78zAX8Qw+lN5rtdEmNv0BctlDPOD7UHDdaA53gBQtH4p5bLlRygEtgZ1CyJWcZ7E5sCEAQdo6rRr3noOejaVvZCl8S6f1Icj92BOov4EBuxv1oxuM2UF6f+zrPxV5ZOchsstB847cJ+imkR8bLWSBGIj5ubwN4UNXk0MVEGGcVZ4F8qSKAis26QdTlzkS5egBZnOUn6hCzS1YiM+6i4pzbUkL+otW9lnk34D2AT3tCSg32Vn+saxSggS5QIiOgh3GKA0RgghT2SyFIp3LMSnK4u6Q9jIgLEjyG4HjLJOQpxObV65Sikl758K7gVzzTMZYK+psV0yBFOdIbBwJ+dap+JBARRA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SJ0PR11MB5615.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(39860400002)(376002)(396003)(346002)(136003)(366004)(451199021)(41300700001)(66574015)(83380400001)(316002)(8936002)(8676002)(5660300002)(4326008)(52536014)(66946007)(76116006)(66446008)(66556008)(66476007)(64756008)(55016003)(186003)(122000001)(9686003)(26005)(6506007)(38100700002)(82960400001)(7696005)(107886003)(54906003)(38070700005)(966005)(86362001)(71200400001)(2906002)(478600001)(33656002)(110136005)(458404004); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?J6hh57umm9RrBtbR7xjAeM88CahE9QLj5mRc9iaVBotAiRZ8r8Qr6Fbc8T?= =?iso-8859-1?Q?yw/JN4yYH6oD66FJjFPQoOaYswHg29xIQWRaC/0vDU3+shYsv9pVJDdhlN?= =?iso-8859-1?Q?Pqo724s4199BYGB5YH1D4pYe2BnEtZITizCqJ73ikmLCBj2DwlqrHlgnFt?= =?iso-8859-1?Q?CQZq6OUhrGV8cQuFoczCwQ6N1D8IIkZwceFP+46oz19/ZOAWumxcbPcLvn?= =?iso-8859-1?Q?lsAWj/jE/DEjR18rKCEJBElM3yAj1wQmv7ffpHYdvtSt1SRPi/5fkHxJi/?= =?iso-8859-1?Q?iPlLth17GY0d/kuCdz/ZgB94M5/JgfjS5EVIDLfjVHti9hTQBaH9EUcRuY?= =?iso-8859-1?Q?rmVAs2GupAqwd9EtMcecnP1i0sISgBuQtHykHy4ZtT7k6qo1SQ8ffyD0Hl?= =?iso-8859-1?Q?xM6K61LoC5XMn2i/cvnSkOSOxoEN3oxJ4NkKuEdWsL04FNTkdurM2epgVH?= =?iso-8859-1?Q?hLjFwzYbOxSXDhItvEnL5ELhVTa/rs4jSS9ME6ApcuvHRSghXnev5xBX2E?= =?iso-8859-1?Q?laapfFP0hkn5wm2DBQrAhHIrPir6ZGHA4zl3V+JBE9S+yg36b7FmuLxbQy?= =?iso-8859-1?Q?kHFrctgRa6yYiOz0CO8pbxukQdtZ6BknKtsrwiLIaOniRidvkjd6omfM99?= =?iso-8859-1?Q?mdukKufSDFFtfri8gGiyEBpETeqsFK0yEr3mzHns4HvfuyHQ6VLXCw6Hjk?= =?iso-8859-1?Q?RqHJwg+crIboC934AZhDlh4pnPENaZTiCJZeCuVWqPDloZrmyD9pKv+pok?= =?iso-8859-1?Q?/URZiWMRgktKVxxhB9d0Q3MIirOBxhqZ+LzlyxeV2kX2gPI6UEkikMxcvD?= =?iso-8859-1?Q?1ixbCRUsCN2BYtCrDtXc3RMgnGIhAH0kDoVWllB1oUNil1rwnKQYEVyAke?= =?iso-8859-1?Q?HwabiRmPgGJXgihfeM7EM65S3qGb0SxNIstX/NlTwusUo9+0rcYBHERtD4?= =?iso-8859-1?Q?zqSNb6sQJL5LnOfin4V8DQhhdXUuBd/BbKu+BdcEaGQK5hpxB65smAWnSH?= =?iso-8859-1?Q?EeOsf32IcCCaA6xHZDgYlluKtSO7W8q5sZqTXdPUM4xWopuGlLfKvIx31K?= =?iso-8859-1?Q?dWAeZkMrrEEzkQRnEqxujCD6TjOF/Wu870FC5J9ze9lUeaL4xFr5bdpwo+?= =?iso-8859-1?Q?WQgNatXcdpBd9JoOwwWlJScjpJY58VK4aw1mObkWA0GvTosefwKov4/oQg?= =?iso-8859-1?Q?YXvCFtChOSxthf4oqsbxjgYAb1REr3ry8fO/ovFaQ3uhp0PSRiInBCXLNy?= =?iso-8859-1?Q?WGyHw4WmHJnbn+e/qu8em3KJWeJkkLzM20zOvtpdiud1oZIoi4tg7hFf87?= =?iso-8859-1?Q?cCwfS5CMZu2jWUwCjWxXAUrAoT0haawhnE+rv0uWV18TOAXMFUCw1ocNGl?= =?iso-8859-1?Q?HgL1kCveEj1y2NTJ5A0Qfd8COr7zzoP+JLmjFaqiNzWAW1y1Mz7i6mcO8g?= =?iso-8859-1?Q?4OdpeleFnDy5oYypTtX86jS343kUwPhfNjZdZFxd4l+ZlGYU0fZqlX+LIn?= =?iso-8859-1?Q?DoGUshIiOSKMiXXkTxXg5+LIY7Uwa7iDPmd/O8PXMioQvGHxWuolfEJzVr?= =?iso-8859-1?Q?y/5UqwgRyXcXA09SDDHRGaV9Sm0z7b1ydoPO3FUYiLoH7bevdI0hhw+H28?= =?iso-8859-1?Q?eFEaViLrdlaDkbwmZVbfSqc6xnkSG7DJHC?= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SJ0PR11MB5615.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4daf7952-e366-4216-5c6a-08db4beb8239 X-MS-Exchange-CrossTenant-originalarrivaltime: 03 May 2023 15:31:47.7084 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 44oeHP15t0dgyHgsQ+YX6hdRwriKckfJV7Z/44g9AisGehdoYdVK3IYcZtlj6nGHi2ARB5BgxbSBuLlfyS+wZg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB7511 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Morten > -----Original Message----- > From: Morten Br=F8rup >=20 > > From: David Coyle [mailto:david.coyle@intel.com] > > Sent: Wednesday, 3 May 2023 13.39 > > > > This is NOT for upstreaming. This is being submitted to allow early > > comparison testing with the preferred solution, which will add TAPUSE > > power management support to the ring library through the addition of > > callbacks. Initial stages of the preferred solution are available at > > http://dpdk.org/patch/125454. > > > > This patch adds functionality directly to rte_ring_dequeue functions > > to monitor the empty reads of the ring. When a configurable number of > > empty reads is reached, a TPAUSE instruction is triggered by using > > rte_power_pause() on supported architectures. rte_pause() is used on > > other architectures. The functionality can be included or excluded at > > compilation time using the RTE_RING_PMGMT flag. If included, the new > > API can be used to enable/disable the feature on a per-ring basis. > > Other related settings can also be configured using the API. >=20 > I don't understand why DPDK developers keep spending time on trying to > invent methods to determine application busyness based on entry/exit > points in a variety of libraries, when the application is in a much bette= r > position to determine busyness. All of these "busyness measuring" library > extensions have their own specific assumptions and weird limitations. >=20 > I do understand that the goal is power saving, which certainly is relevan= t! I > only criticize the measuring methods. >=20 > For reference, we implemented something very simple in our application > framework: > 1. When each pipeline stage has completed a burst, it reports if it was b= usy or > not. > 2. If the pipeline busyness is low, we take a nap to save some power. >=20 > And here is the magic twist to this simple algorithm: > 3. A pipeline stage is not considered busy unless it processed a full bur= st, and > is ready to process more packets immediately. This interpretation of > busyness has a significant impact on the percentage of time spent napping > during the low-traffic hours. >=20 > This algorithm was very quickly implemented. It might not be perfect, and= we > do intend to improve it (also to determine CPU Utilization on a scale tha= t the > end user can translate to a linear interpretation of how busy the system = is). > But I seriously doubt that any of the proposed "busyness measuring" libra= ry > extensions are any better. >=20 > So: The application knows better, please spend your precious time on > something useful instead. >=20 > @David, my outburst is not directed at you specifically. Generally, I do > appreciate experimenting as a good way of obtaining knowledge. So thank > you for sharing your experiments with this audience! >=20 > PS: If cruft can be disabled at build time, I generally don't oppose to i= t. [DC] Appreciate that feedback, and it is certainly another way of looking a= t and tackling the problem that we are ultimately trying to solve (i.e power saving) The problem however is that we work with a large number of ISVs and operato= rs, each with their own workload architecture and implementation. That means we would have to work individually with each of these to integrate this type o= f pipeline-stage-busyness algorithm into their applications. And as these applications are usually commercial, non-open-source applications, that cou= ld prove to be very difficult. Also most ISVs and operators don't want to have to worry about changing the= ir application, especially their fast-path dataplane, in order to get power savings. They prefer for it to just happen without them caring about the fi= ner details. For these reasons, consolidating the busyness algorithms down into the DPDK libraries and PMDs is currently the preferred solution. As you say though, = the libraries and PMDs may not be in the best position to determine the busynes= s of the pipeline, but it provides a good balance between achieving power sav= ings and ease of adoption. It's also worth calling out again that this patch is only to allow early testing by some customers of the benefit of adding TPAUSE support to the ri= ng library. We don't intend on this patch being upstreamed. The preferred long= er term solution is to use callbacks from the ring library to initiate the pau= se (either via the DPDK power management API or through functions that an ISV may write themselves). This is mentioned in the commit message. Also, the pipeline stage busyness algorithm that you have added to your pipeline - have you ever considered implementing this into DPDK as a generi= c type library. This could certainly be of benefit to other DPDK application developers, and having this mechanism in DPDK could again ease the adoption and realisation of power savings for others. I understand though if this is= your own secret sauce and you want to keep it like that :) David