From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3878A42A5F; Thu, 4 May 2023 18:13:21 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B611741144; Thu, 4 May 2023 18:13:20 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id 8610D410DC for ; Thu, 4 May 2023 18:13:18 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683216798; x=1714752798; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=H4xHZupOGk2ROgjXktYfDmpFeSt6aN1BDuBsKlnbimY=; b=PIQyovdreONLShSG0248tyA5FOiiuMWzGYWIJIFXHgpFo8QXRte9cNg0 abkHL/MMEh/x6UqjiPbxs78LI6YfEhxGkP6+ZoGhNDuWtwx73XAfaGxlg qKWIQvpxrW113ixTXh4+IzVimAJ7JDB03ED1qy/5dbs2ssOrGuxd5G5Yt fqYGCrDlWd9feijLREaFw798HaRxIaNWKArBYHJOz7bOLNR1GShWzcEJ5 M28aTaCjhiKWSEQIZtn/7WldIulLjdzBpShZzkCn1OSxxN5yfIBPCbP6X uW7PVxXc7U0THHRFj5gY8mtEjod9YWMl4ocY+bKef1rxm5YJgsqSTORYh g==; X-IronPort-AV: E=McAfee;i="6600,9927,10700"; a="435286230" X-IronPort-AV: E=Sophos;i="5.99,249,1677571200"; d="scan'208";a="435286230" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 May 2023 09:11:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10700"; a="821238122" X-IronPort-AV: E=Sophos;i="5.99,249,1677571200"; d="scan'208";a="821238122" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by orsmga004.jf.intel.com with ESMTP; 04 May 2023 09:11:34 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Thu, 4 May 2023 09:11:33 -0700 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23 via Frontend Transport; Thu, 4 May 2023 09:11:33 -0700 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (104.47.58.169) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.23; Thu, 4 May 2023 09:11:33 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=c6FReMPCakIMuIAt/k6mIYA1J8QC0PKhYtkIrCaZ5wCWAH7z96hE3Q2tRyYWTXN19e0UdJ55KT3+tu1md0EB9cWxESt8j8tEa7NvMsOWKRjiw1YJ/xPOgMx1pATCdLw8NLIRmAuthgTLMInF3/BLICM3sNYabkBeEeu9z1UsHtq3LihjkaaAUjFE4+rwNVOxurody+IdPi/JrTq1GZbzQiVtmvAqmWLzLaDranOK+C/QsBx+OimgUc+hcJ24G7oCJu7OUYyKsig28iIyizMYOqib6s50tao8oX1lrHNHJDuUP4nsdug25HjDlymFhzdu6MoaOBS1OBioiVzXaqGr6w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=H4xHZupOGk2ROgjXktYfDmpFeSt6aN1BDuBsKlnbimY=; b=TVYnh4nFCSt1AR0r5bDLWFvw5iFBKFxxiheezrbUREkwF6LMUleK0x4Z0p+zZoK2I+GUSdRI9EJ9uWHcd0urbR8pZ/Lgg6SUmSJf6E+6THHHwNkrJEM9nppCamldqvlGXQbAKpGShgTSNCvQNzj1VXZY+23eMx57lkDm2yViJxXKTjNtGUOxIBDjV6eaTcGjknIYKqx38c7XPrrkYIR49CB6Q7Stcj4DuzHZChTlVU3YGsizjTc9fn15xhOEUX8EggLpTZNk91YG8Q3VVGZiXFQJZEg50yU/SGEzp/d/tbMBpRFFaLR3jwYdFuT2g0tp7WY72TGNPdFmOWMU5r0Ecg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from CO6PR11MB5604.namprd11.prod.outlook.com (2603:10b6:303:138::10) by BL1PR11MB5556.namprd11.prod.outlook.com (2603:10b6:208:314::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6363.26; Thu, 4 May 2023 16:11:31 +0000 Received: from CO6PR11MB5604.namprd11.prod.outlook.com ([fe80::a934:f9b4:6a15:255b]) by CO6PR11MB5604.namprd11.prod.outlook.com ([fe80::a934:f9b4:6a15:255b%7]) with mapi id 15.20.6363.022; Thu, 4 May 2023 16:11:31 +0000 From: "Coyle, David" To: =?iso-8859-1?Q?Morten_Br=F8rup?= , "dev@dpdk.org" CC: "honnappa.nagarahalli@arm.com" , "konstantin.v.ananyev@yandex.ru" , "Sexton, Rory" Subject: RE: [RFC PATCH] ring: adding TPAUSE instruction to ring dequeue Thread-Topic: [RFC PATCH] ring: adding TPAUSE instruction to ring dequeue Thread-Index: AQHZfbPnG2/msW+xn0Gzh27kyZT0UK9Ii/+AgAAP7eCAAFKdUIABMRpw Date: Thu, 4 May 2023 16:11:31 +0000 Message-ID: References: <20230503113840.11010-1-david.coyle@intel.com> <98CBD80474FA8B44BF855DF32C47DC35D878DD@smartserver.smartshare.dk> <98CBD80474FA8B44BF855DF32C47DC35D878E1@smartserver.smartshare.dk> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D878E1@smartserver.smartshare.dk> Accept-Language: en-IE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CO6PR11MB5604:EE_|BL1PR11MB5556:EE_ x-ms-office365-filtering-correlation-id: 37aaa995-28bf-49dc-e28c-08db4cba3936 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: ml3PK6Soih91E2Jg7UKaSViHk8blMvXjFABMN84DKeggp3FOCI4jZ/aAYjn8qDDBC0SCh96JLX3IrNn8c3h9tSorZJSfcAPjDaK2uYJ2EDZMuqxfMamEspjIwYhKyijJI14ipPzjq+H/irWHY84zxKHBPmej+V1/P/DX2e7cJa5RCEWiw2jKzoC7I+20haYdLZO6xKw8w8TtRno4bySrJ9yKVwM6vFhm5TmCHX7m//hnOiuWyf9cj1X7NATD5ees2l+eaAr96eIay2+epkIY0NrC3nn+5RTRQBB02dO0d8vKAw1PgK7LYw6b7roRGGaD2uzu/3ggiBDstU1wlh8W2Bf2PlqEeISmOKCKezsOpSG3VUN7ANBqsRtek+W9C1uZOkKConKCSgzWo9duEfat4Mi/V3EiCAaSSoZ+zMAtOcCaUAkSRCJWIkUUjLs9lxqIx2+2YqM4lv7r+Gp99DjNIqziTk9j0JdqiWGmWB170/1vVNfcLVZzr5qb1IVzoCfIRQPiS7ljb0HZMqDEJhEmNkvtV8hDRvASu7JESghWSeBwaNZtn1g24CHHLHR4k0sSnIZ7d6ShJzijpnSTUz0ul7OZJGkLnTCsmoOmBbQxDshY3+/g6Ks1sszrUvwEIZ7l x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CO6PR11MB5604.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(346002)(376002)(366004)(39860400002)(396003)(136003)(451199021)(38070700005)(41300700001)(6506007)(9686003)(26005)(478600001)(71200400001)(4326008)(55016003)(86362001)(7696005)(33656002)(66476007)(66446008)(66556008)(64756008)(66946007)(76116006)(38100700002)(52536014)(107886003)(316002)(186003)(5660300002)(54906003)(110136005)(83380400001)(2906002)(66574015)(82960400001)(122000001)(8936002)(8676002); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?vS8EDw0W0AhCmsZlyOH2QF0bDMz8OEMfl3pItiuE+3UTFK507VpILSUqks?= =?iso-8859-1?Q?lcN7EOSjndRfEyJ31f5V8PuudvWvzv7lN+iniASs9bIUoHD1MIdkQD4RJx?= =?iso-8859-1?Q?RBcRknOuXNLaa1QDGyan9bokip5DHM/sDLjvqcblSbJq4PB8MYMPYc7Uzs?= =?iso-8859-1?Q?seyZ2mi2lAnZIo0YHUi+cuv57btJjI3bvoHs5taudeFciJcNfD5m44MO/C?= =?iso-8859-1?Q?9qqv8gZLzyjiNzMxnKWqgf7Z/JwEDwKKzL1jT9KagGt1ggiQ/VRJiyHO0I?= =?iso-8859-1?Q?q30R5KiGSVz0GLE7cGEj8zG0QoGpVFCXlHYceOtLC6/10EttA+6Afln0aB?= =?iso-8859-1?Q?OSxVPBwMvZGeFbjXRzcurY6jjyTlYiZSdJw3k6oT0N0DnR04uxcG21f6Of?= =?iso-8859-1?Q?9JTEQ0LLrYmsRM4dolXbLxiuT28grDZv6lHnqhSJYZzey5c8hl4QnOw0AD?= =?iso-8859-1?Q?/2dEBjvJLb/bSOHTH4pctFQZjFw1a3s/BVm5rAq4mYrt+WNaT7OZY5+9cZ?= =?iso-8859-1?Q?7Hnp/kI1lkmZVYm0PrHRjq2LIy+040jNGnKjdvpfuTfuvFwhwp5SCCwUIc?= =?iso-8859-1?Q?piQMIv5hufPLgToW1dBWlc6BkDIJvOhG8qf3qi87cvwak4iN/lPxSQAksr?= =?iso-8859-1?Q?7fQSX+dGgPIoMhLOx9QVViQw69OdV112/ShRCVAKm3aVcL3FRRv9AbH1CL?= =?iso-8859-1?Q?dUZQGWYYkC1CC0NhPDGWP2YaLFk19O78F9pxga5vjDOZiDb4d5XMDOdeeZ?= =?iso-8859-1?Q?TVutPFP5OjGeLGWLlkajWJb8Vprocid2g+vCfqbpnpwI7YQ3yrsbXiREYB?= =?iso-8859-1?Q?IzZf1e8H0IwEE/nwZ0Sqmcj6Te+1Y1oSyJaiRiqTvV47ZuLe8+/xz/0hXa?= =?iso-8859-1?Q?KHet1e0H6jb+67gCTVTAD49XbYRi4RzILaQy7//BEVXUIRz7QyFtv7+zZL?= =?iso-8859-1?Q?Gy7q/Hhx32cuWBesy7fxs9/BFDyJR1B+qcBAMN5krF3pIBYwj0R3V4SkoU?= =?iso-8859-1?Q?PuG/qAfc+jORVOi8RrCVUFQOEoJ5JP2pNU/PnBCkrwiAuAFAyjKRdtSrhM?= =?iso-8859-1?Q?fabfiVzF8MjwBTXPquqQjYey2NfmcC9m957+dhLXJiXx75H2Uo/1XUE7QW?= =?iso-8859-1?Q?trxPUN7EkgDfCKRgZLr8nw6EFeFInL3Eno/qVPnsvUcWck9J0uq390eXRr?= =?iso-8859-1?Q?l1DgX8Q8NcGYhuCz/uCOSBVnztF9qdAibIbEnQwPmX2agykzIfvrJ4GuZH?= =?iso-8859-1?Q?PY2tmvU49Ap6JvJP++IYd11lpyt7hjwx7eTe5qAGLS1UTUQeG/EaU6LFOk?= =?iso-8859-1?Q?wRTl7fJMC3xszUufuyej9e9XAm86YJl/DeQWWwubVj0PMWHO9LmISCTmxp?= =?iso-8859-1?Q?rGCie8mtBav/fTkareMMtZuTnczNKUxINbZj0pKOhcurvxQXXqJIP9ECdy?= =?iso-8859-1?Q?gXrAQUhhABSB1stWI0XB0xTcPbN0zALCsUN8T8fEpgmtF6Xkh8HECdB4Ui?= =?iso-8859-1?Q?tGcFjqJ6ltCkU1QTKaLDLALFIo03qAI9Bj0CdF+EYijGcGFaz55avuO+/0?= =?iso-8859-1?Q?ZIPSzY1MCZFhQldITfP1VlgWRs+nosvClEG27iSZgon2iTsIA52oCQm5It?= =?iso-8859-1?Q?QOuhYgYvw52xoVYi03e/1QgDCyo9vj9am6?= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CO6PR11MB5604.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 37aaa995-28bf-49dc-e28c-08db4cba3936 X-MS-Exchange-CrossTenant-originalarrivaltime: 04 May 2023 16:11:31.0283 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: srLIQdcDNryzLyPVXv5508hsSpbC0TMcRdgBCgbPNxP7JQN9tTc1XhsiGcpjg3FdhzJyffR0eE2DjNuHmmIfIw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR11MB5556 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Morten > -----Original Message----- > From: Morten Br=F8rup >=20 > Power saving is important for the environment (to save the planet and all > that), so everyone should contribute, if they have a good solution. So ev= en if > our algorithm had a significant degree of innovation, we would probably > choose to make it public anyway. Open sourcing it also makes it possible = for > chip vendors like Intel to fine tune it more than we can ourselves, which= also > comes back to benefit us. All products need some sort of power saving in = to > stay competitive, but power saving algorithms is not an area we want to > pursue for competitive purposes in our products. >=20 > Our algorithm is too simple to make a library at this point, but I have b= een > thinking about how we can make it a generic library when it has matured > some more. I will take your information about the many customers' need to > have it invisibly injected into consideration in this regard. >=20 > Our current algorithm works like this: >=20 > while (running) { > int more =3D 0; > more +=3D stage1(); > more +=3D stage2(); > more +=3D stage3(); > if (!more) sleep(); > } >=20 > Each pipeline stage only returns 1 if it processed a full burst. Furtherm= ore, if a > pipeline stage processed a full burst, but happens to know that no more d= ata > is readily available for it, it returns 0 instead. >=20 > Obviously, the sleep() duration must be short enough to avoid that the NI= C > RX descriptor rings overflow before the ingress pipeline stage is service= d > again. >=20 > Changing the algorithm to "more" (1 =3D more work expected by the pipelin= e > stage) from "busy" (1 =3D some work done by the pipeline stage) has the > consequence that sleep() is called more often, which has the follow-on > consequence that the ingress stage is called less often, and thus more of= ten > has a full burst to process. >=20 > We know from our in-house profiler that processing a full burst provides > *much* higher execution efficiency (cycles/packet) than processing a few > packets. This is public knowledge - after all, this is the whole point of= DPDK's > vector packet processing design! Nonetheless, it might surprise some peop= le > how much the efficiency (cycles/packet) increases when processing a full > burst compared to processing just a few packets. I will leave it up to th= e > readers to make their own experiments. :-) >=20 > Our initial "busy" algorithm behaved like this: > Process a few packets (at low efficiency), don't sleep, Process a few pac= kets > (at low efficiency), don't sleep, Process a few packets (at low efficienc= y), > don't sleep, Process a few packets (at low efficiency), don't sleep, Proc= ess a > few packets (at low efficiency), don't sleep, Process a few packets (at l= ow > efficiency), don't sleep, Process a few packets (at low efficiency), don'= t > sleep, Process a few packets (at low efficiency), don't sleep, No packets= to > process (we are lucky this time!), sleep briefly, Repeat. >=20 > So we switched to our "more" algorithm, which behaves like this: > Process a few packets (at low efficiency), sleep briefly, Process a full = burst of > packets (at high efficiency), don't sleep, Repeat. >=20 > Instead of processing e.g. 8 small bursts per sleep, we now process only = 2 > bursts per sleep. And the big of the two bursts is processed at higher > efficiency. >=20 > We can improve this algorithm in some areas... >=20 > E.g. some of our pipeline stages also know that they are not going to do > anymore work for the next X amount of nanoseconds; but we don't use that > information in our power management algorithm yet. The sleep duration > could depend on this. >=20 > Also, we don't use the CPU power management states yet. I assume that > doing some work for 20 us at half clock speed is more power conserving th= an > doing the same work at full speed for 10 us and then sleeping for 10 us. > That's another potential improvement. >=20 >=20 > What we need in generic a power management helper library are functions > to feed it with the application's perception of how much work is being do= ne, > and functions to tell if we can sleep and/or if we should change the powe= r > management states of the individual CPU cores. >=20 > Such a unified power management helper (or "busyness") library could > perhaps also be fed with data directly from the drivers and libraries to > support the customer use cases you described. [DC] Thank you for that detailed description, very interesting. There may well be merit in upstreaming such an algorithm as a library once it has matured as you said. Configuration could include specifying what a "full burst" actually is. Different stages of a pipeline may also have different definit= ions of busyness, so that may also need to considered: - Some stages may perform an operation (e.g. running an acl rule check) on = a burst of packets and then it is complete - Other stages may be more asynchronous in nature e.g. enqueuing and dequeuing to/from a crypto device or a QoS scheduler. The dequeue might not dequeue any packets on a particular call of the dequeue API, but there may still be packets waiting inside the crypto device or scheduler. Those w= aiting packets would also need to be taken into account so as not to sleep for too= long. Using such an API would require a workload developer to update their datapa= th to report the pipeline stage busyness to the algorithm, but if those calls = are kept to a minimum, then that shouldn't be too much of a problem Thanks, David