From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id C1231A00B8; Mon, 28 Oct 2019 06:04:34 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id BB1171BEFF; Mon, 28 Oct 2019 06:04:33 +0100 (CET) Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-eopbgr70089.outbound.protection.outlook.com [40.107.7.89]) by dpdk.org (Postfix) with ESMTP id D9D661BEFA for ; Mon, 28 Oct 2019 06:04:31 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0u5j/MfDl43xMatZl0eZUjciD8kwWLAR7qKexNG9iJA=; b=sp5ixRjOXcbTLD0Osc18AwOObBicJR5yawrzN0Rpvi5HABGVipta9jsKoSWkmsF+0wSxdMiEO3Y0GEHWUtbTmvOHxRo3Vrx9SzFwjOj7M708lIxhHLa7e6otlERkBz/rTLlvSl8PTRq3eJXUgDttHc4XYm5rikPdJyaw9FOADP0= Received: from DB6PR0801CA0065.eurprd08.prod.outlook.com (2603:10a6:4:2b::33) by VI1PR0802MB2304.eurprd08.prod.outlook.com (2603:10a6:800:9f::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2387.24; Mon, 28 Oct 2019 05:04:29 +0000 Received: from DB5EUR03FT037.eop-EUR03.prod.protection.outlook.com (2a01:111:f400:7e0a::209) by DB6PR0801CA0065.outlook.office365.com (2603:10a6:4:2b::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2387.20 via Frontend Transport; Mon, 28 Oct 2019 05:04:29 +0000 Authentication-Results: spf=fail (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=none action=none header.from=arm.com; Received-SPF: Fail (protection.outlook.com: domain of arm.com does not designate 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DB5EUR03FT037.mail.protection.outlook.com (10.152.20.215) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2387.20 via Frontend Transport; Mon, 28 Oct 2019 05:04:29 +0000 Received: ("Tessian outbound 6481c7fa5a3c:v33"); Mon, 28 Oct 2019 05:04:28 +0000 X-CR-MTA-TID: 64aa7808 Received: from 488bffe26c95.2 (cr-mta-lb-1.cr-mta-net [104.47.6.53]) by 64aa7808-outbound-1.mta.getcheckrecipient.com id FD9CAA75-6C62-454A-AA71-0365DE821633.1; Mon, 28 Oct 2019 05:04:23 +0000 Received: from EUR02-VE1-obe.outbound.protection.outlook.com (mail-ve1eur02lp2053.outbound.protection.outlook.com [104.47.6.53]) by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 488bffe26c95.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384); Mon, 28 Oct 2019 05:04:23 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kY8qh3S7TnC/vUePHIPIiJY2zZ4JjkAn7VfHNjxPQY9NA7nDb+wCORM69Ysg62zbBs0zwI5vapC1TKCxKmOnBcCBPvVRMoohdt2owFW56FkdCnfLscF8f4UQbyZgel2TwdasUwVWz2nLmAgQHI8Z0CN9XzSYylcxV8DxBlE9ufTNO+XG8GRY7037b5V06rPjywwNMNfhp8HWeFdBeQtbrS31Otjlmm7KJ5wnvXU8zv0P1XoYHmYY2OCABIs+Lf38E66xeUWQUWlCLd+crriy6ktTJM8vy1pbDdQii8tJuSts/ZN7+Ux5Tyswdg++PplbrBJvy7zCgWp/w+Djttq9uQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0u5j/MfDl43xMatZl0eZUjciD8kwWLAR7qKexNG9iJA=; b=Vg5mM7WgWzxlF6Iawm6z08ERQ+XWArOvYhsgy9SbjBw/ZPk8x6h1tnft5j+JKYF/a8MbLsOF3bcRDAsFtASHhFMvaK/6M4QJ00Ihx6Wc+7sQr1UzrVlNi+9bV9AA1dPKXw1traenesk6mwf/GQwmQdLkUFbuVUVBHBiabPAeR6ygSY7L+u3+HHEUcZ/wMzQ/qGCmDq/w8ctgAd9MC+U4sXkC4z0BG4c4m7fC1vDJwys/sECwt2od9cIfi61TDA7ORb5LaJ20O6abjvtZc6oq5Cv4scrPzo44vVg2PwlFK9BDJZmpdFWfL4ffli7kGC1A/oK+L0YHW6NRsKgdEYCFfQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0u5j/MfDl43xMatZl0eZUjciD8kwWLAR7qKexNG9iJA=; b=sp5ixRjOXcbTLD0Osc18AwOObBicJR5yawrzN0Rpvi5HABGVipta9jsKoSWkmsF+0wSxdMiEO3Y0GEHWUtbTmvOHxRo3Vrx9SzFwjOj7M708lIxhHLa7e6otlERkBz/rTLlvSl8PTRq3eJXUgDttHc4XYm5rikPdJyaw9FOADP0= Received: from VI1PR08MB5376.eurprd08.prod.outlook.com (10.255.196.79) by VI1PR08MB4399.eurprd08.prod.outlook.com (20.179.28.141) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2387.22; Mon, 28 Oct 2019 05:04:21 +0000 Received: from VI1PR08MB5376.eurprd08.prod.outlook.com ([fe80::709e:c2c9:139d:9df3]) by VI1PR08MB5376.eurprd08.prod.outlook.com ([fe80::709e:c2c9:139d:9df3%4]) with mapi id 15.20.2387.025; Mon, 28 Oct 2019 05:04:20 +0000 From: "Gavin Hu (Arm Technology China)" To: "Ananyev, Konstantin" , "dev@dpdk.org" CC: nd , "david.marchand@redhat.com" , "thomas@monjalon.net" , "stephen@networkplumber.org" , "hemant.agrawal@nxp.com" , "jerinj@marvell.com" , "pbhagavatula@marvell.com" , Honnappa Nagarahalli , "Ruifeng Wang (Arm Technology China)" , "Phil Yang (Arm Technology China)" , Steve Capper , nd Thread-Topic: [PATCH v11 2/5] eal: add the APIs to wait until equal Thread-Index: AQHVjRSraVC8zTCyy0+QZ+8fGLw49qdvf8HQ Date: Mon, 28 Oct 2019 05:04:20 +0000 Message-ID: References: <1561911676-37718-1-git-send-email-gavin.hu@arm.com> <1572180765-49767-3-git-send-email-gavin.hu@arm.com> <2601191342CEEE43887BDE71AB97725801A8C710F4@IRSMSX104.ger.corp.intel.com> In-Reply-To: <2601191342CEEE43887BDE71AB97725801A8C710F4@IRSMSX104.ger.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 5d20dbbc-2728-4ed8-a6c7-8edfa1c690ae.0 x-checkrecipientchecked: true Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=Gavin.Hu@arm.com; x-originating-ip: [113.29.88.7] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 48eeecdd-ce15-4f15-2caa-08d75b645026 X-MS-TrafficTypeDiagnostic: VI1PR08MB4399:|VI1PR08MB4399:|VI1PR0802MB2304: x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true x-ms-oob-tlc-oobclassifiers: OLM:8882;OLM:8882; x-forefront-prvs: 0204F0BDE2 X-Forefront-Antispam-Report-Untrusted: SFV:NSPM; SFS:(10009020)(4636009)(136003)(346002)(396003)(366004)(376002)(39850400004)(199004)(189003)(13464003)(229853002)(6116002)(3846002)(33656002)(6246003)(76176011)(14444005)(256004)(14454004)(102836004)(71200400001)(71190400001)(446003)(186003)(6506007)(55236004)(26005)(53546011)(305945005)(486006)(2906002)(11346002)(86362001)(7736002)(99286004)(476003)(5660300002)(66476007)(66446008)(64756008)(6436002)(4326008)(66556008)(66946007)(52536014)(76116006)(7696005)(66066001)(30864003)(81156014)(81166006)(8676002)(478600001)(54906003)(25786009)(110136005)(316002)(8936002)(55016002)(9686003)(2501003)(74316002)(21314003); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR08MB4399; H:VI1PR08MB5376.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: rJ0b7bV/sgDQ6ispMXNf5ucsQl+wOIV2FfjLukir8fCvI8Us0IrcFsJiTJzu02g0qiVZ+op8xRafXsYNkyyyXjQJ/eQqUU2SLMM7Y4NIuvuXG4Jb8TYqrCVGYpI7Qoe73mP7n8YFY8PhuhwPgwBkvMvcOuZXkM4qzpi8peeo34IwAnr2X15dLn+Sln4+eYBQKvdJF8kVo34hdSbEvBbDMnzsk5crPOsMtIr6jV5lKKprqE6FAxWxxs9h5CJHGn+qf8EvSft/GlnBy2/7Mx7DJk2zyHEBCZFh35dF77fuP0Gmc+j3z9CTF6QC1upnsMn9t+iL6y54dlkQWKTECPARn21dwsAk1/NzQt5Vk6/jfYOHbtI0mrR15y62hfgAyT+6GSGAjKs+kEeH/2Ehx0ez90m0IdOy3msy8ziefCMFYBuLDvl/WQjl9amXW4zBKnf6 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR08MB4399 Original-Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Gavin.Hu@arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DB5EUR03FT037.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; IPV:CAL; SCL:-1; CTRY:IE; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(346002)(396003)(376002)(39850400004)(136003)(1110001)(339900001)(199004)(189003)(13464003)(26826003)(76130400001)(70206006)(356004)(316002)(52536014)(110136005)(54906003)(26005)(50466002)(25786009)(478600001)(22756006)(2906002)(99286004)(33656002)(6506007)(5660300002)(30864003)(53546011)(70586007)(446003)(6116002)(476003)(9686003)(4326008)(105606002)(86362001)(76176011)(55016002)(14454004)(3846002)(14444005)(186003)(305945005)(7736002)(6246003)(66066001)(486006)(2501003)(8936002)(102836004)(8746002)(81166006)(336012)(81156014)(97756001)(8676002)(46406003)(23726003)(74316002)(47776003)(126002)(229853002)(7696005)(11346002)(21314003); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR0802MB2304; H:64aa7808-outbound-1.mta.getcheckrecipient.com; FPR:; SPF:Fail; LANG:en; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; A:1; MX:1; X-MS-Office365-Filtering-Correlation-Id-Prvs: 1b2133a9-b14a-46fd-8bc9-08d75b644ae3 NoDisclaimer: True X-Forefront-PRVS: 0204F0BDE2 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: n1UEYZTprS+LyW2cZUfdlbB+i3vZlpxTeSuSKAGAr798RZWo1NSSLgpWP7ZMibCmZQvy8cc4mnOdLc5qpDGh2c23AnqXogoIEs/4WqC9a1Rh+38UW6ISef9434JTfrhI6JCYXaH+DRxbOCtPIioQ8yhQr1Swr67VL2hlVkz34IFRbuUviVw2Q6s8G39QxK5S23mSVVWz2SBHUCKjfzVaBXmCglXqf05+PTI9BrgJyjBgKH2KRJaU6ajdqY4W7bbzwkq7klJIlaCeYxeWYoKe4Y0nkLatZI7ilMoRXyAE21gmU/PEXhqyijbP3SbQB5pax1+/ZP+5qm5ly3a8md7LP+s+P/O5dE8fMmskX4pIIts4QUHXckymfsnrc49ycYQZ34lwO6Tuu7/W7h+JdFE2oIDtpgaQy/kOsN6kXARv9Yge7xd2ifzksxR5jjnT5sde X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Oct 2019 05:04:29.5220 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 48eeecdd-ce15-4f15-2caa-08d75b645026 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0802MB2304 Subject: Re: [dpdk-dev] [PATCH v11 2/5] eal: add the APIs to wait until equal X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Konstantin, > -----Original Message----- > From: Ananyev, Konstantin > Sent: Monday, October 28, 2019 6:20 AM > To: Gavin Hu (Arm Technology China) ; dev@dpdk.org > Cc: nd ; david.marchand@redhat.com; thomas@monjalon.net; > stephen@networkplumber.org; hemant.agrawal@nxp.com; > jerinj@marvell.com; pbhagavatula@marvell.com; Honnappa Nagarahalli > ; Ruifeng Wang (Arm Technology China) > ; Phil Yang (Arm Technology China) > ; Steve Capper > Subject: RE: [PATCH v11 2/5] eal: add the APIs to wait until equal >=20 >=20 > > The rte_wait_until_equal_xx APIs abstract the functionality of > > 'polling for a memory location to become equal to a given value'. > > > > Add the RTE_ARM_USE_WFE configuration entry for aarch64, disabled > > by default. When it is enabled, the above APIs will call WFE instructio= n > > to save CPU cycles and power. > > > > From a VM, when calling this API on aarch64, it may trap in and out to > > release vCPUs whereas cause high exit latency. Since kernel 4.18.20 an > > adaptive trapping mechanism is introduced to balance the latency and > > workload. > > > > Signed-off-by: Gavin Hu > > Reviewed-by: Ruifeng Wang > > Reviewed-by: Steve Capper > > Reviewed-by: Ola Liljedahl > > Reviewed-by: Honnappa Nagarahalli > > Reviewed-by: Phil Yang > > Acked-by: Pavan Nikhilesh > > Acked-by: Jerin Jacob > > --- > > config/arm/meson.build | 1 + > > config/common_base | 5 + > > .../common/include/arch/arm/rte_pause_64.h | 188 > +++++++++++++++++++++ > > lib/librte_eal/common/include/generic/rte_pause.h | 99 +++++++++++ > > 4 files changed, 293 insertions(+) > > > > diff --git a/config/arm/meson.build b/config/arm/meson.build > > index 979018e..b4b4cac 100644 > > --- a/config/arm/meson.build > > +++ b/config/arm/meson.build > > @@ -26,6 +26,7 @@ flags_common_default =3D [ > > ['RTE_LIBRTE_AVP_PMD', false], > > > > ['RTE_SCHED_VECTOR', false], > > + ['RTE_ARM_USE_WFE', false], > > ] > > > > flags_generic =3D [ > > diff --git a/config/common_base b/config/common_base > > index e843a21..c812156 100644 > > --- a/config/common_base > > +++ b/config/common_base > > @@ -111,6 +111,11 @@ CONFIG_RTE_MAX_VFIO_CONTAINERS=3D64 > > CONFIG_RTE_MALLOC_DEBUG=3Dn > > CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=3Dn > > CONFIG_RTE_USE_LIBBSD=3Dn > > +# Use WFE instructions to implement the rte_wait_for_equal_xxx APIs, > > +# calling these APIs put the cores in low power state while waiting > > +# for the memory address to become equal to the expected value. > > +# This is supported only by aarch64. > > +CONFIG_RTE_ARM_USE_WFE=3Dn > > > > # > > # Recognize/ignore the AVX/AVX512 CPU flags for performance/power > testing. > > diff --git a/lib/librte_eal/common/include/arch/arm/rte_pause_64.h > b/lib/librte_eal/common/include/arch/arm/rte_pause_64.h > > index 93895d3..1680d7a 100644 > > --- a/lib/librte_eal/common/include/arch/arm/rte_pause_64.h > > +++ b/lib/librte_eal/common/include/arch/arm/rte_pause_64.h > > @@ -1,5 +1,6 @@ > > /* SPDX-License-Identifier: BSD-3-Clause > > * Copyright(c) 2017 Cavium, Inc > > + * Copyright(c) 2019 Arm Limited > > */ > > > > #ifndef _RTE_PAUSE_ARM64_H_ > > @@ -10,6 +11,11 @@ extern "C" { > > #endif > > > > #include > > + > > +#ifdef RTE_ARM_USE_WFE > > +#define RTE_WAIT_UNTIL_EQUAL_ARCH_DEFINED > > +#endif > > + > > #include "generic/rte_pause.h" > > > > static inline void rte_pause(void) > > @@ -17,6 +23,188 @@ static inline void rte_pause(void) > > asm volatile("yield" ::: "memory"); > > } > > > > +/** > > + * Send an event to quit WFE. > > + */ > > +static inline void rte_sevl(void); > > + > > +/** > > + * Put processor into low power WFE(Wait For Event) state > > + */ > > +static inline void rte_wfe(void); > > + > > +#ifdef RTE_ARM_USE_WFE > > +static inline void rte_sevl(void) > > +{ > > + asm volatile("sevl" : : : "memory"); > > +} > > + > > +static inline void rte_wfe(void) > > +{ > > + asm volatile("wfe" : : : "memory"); > > +} > > +#else > > +static inline void rte_sevl(void) > > +{ > > +} > > +static inline void rte_wfe(void) > > +{ > > + rte_pause(); > > +} > > +#endif > > + > > +/** > > + * @warning > > + * @b EXPERIMENTAL: this API may change, or be removed, without prior > notice > > + * > > + * Atomic exclusive load from addr, it returns the 16-bit content of *= addr > > + * while making it 'monitored',when it is written by someone else, the > > + * 'monitored' state is cleared and a event is generated implicitly to= exit > > + * WFE. > > + * > > + * @param addr > > + * A pointer to the memory location. > > + * @param memorder > > + * The valid memory order variants are __ATOMIC_ACQUIRE and > __ATOMIC_RELAXED. > > + * These map to C++11 memory orders with the same names, see the > C++11 standard > > + * the GCC wiki on atomic synchronization for detailed definitions. > > + */ > > +static __rte_always_inline uint16_t > > +rte_atomic_load_ex_16(volatile uint16_t *addr, int memorder); > > + > > +/** > > + * @warning > > + * @b EXPERIMENTAL: this API may change, or be removed, without prior > notice > > + * > > + * Atomic exclusive load from addr, it returns the 32-bit content of *= addr > > + * while making it 'monitored',when it is written by someone else, the > > + * 'monitored' state is cleared and a event is generated implicitly to= exit > > + * WFE. > > + * > > + * @param addr > > + * A pointer to the memory location. > > + * @param memorder > > + * The valid memory order variants are __ATOMIC_ACQUIRE and > __ATOMIC_RELAXED. > > + * These map to C++11 memory orders with the same names, see the > C++11 standard > > + * the GCC wiki on atomic synchronization for detailed definitions. > > + */ > > +static __rte_always_inline uint32_t > > +rte_atomic_load_ex_32(volatile uint32_t *addr, int memorder); > > + > > +/** > > + * @warning > > + * @b EXPERIMENTAL: this API may change, or be removed, without prior > notice > > + * > > + * Atomic exclusive load from addr, it returns the 64-bit content of *= addr > > + * while making it 'monitored',when it is written by someone else, the > > + * 'monitored' state is cleared and a event is generated implicitly to= exit > > + * WFE. > > + * > > + * @param addr > > + * A pointer to the memory location. > > + * @param memorder > > + * The valid memory order variants are __ATOMIC_ACQUIRE and > __ATOMIC_RELAXED. > > + * These map to C++11 memory orders with the same names, see the > C++11 standard > > + * the GCC wiki on atomic synchronization for detailed definitions. > > + */ > > +static __rte_always_inline uint64_t > > +rte_atomic_load_ex_64(volatile uint64_t *addr, int memorder); > > + > > +static __rte_always_inline uint16_t > > +rte_atomic_load_ex_16(volatile uint16_t *addr, int memorder) > > +{ > > + uint16_t tmp; > > + assert((memorder =3D=3D __ATOMIC_ACQUIRE) > > + || (memorder =3D=3D __ATOMIC_RELAXED)); > > + if (memorder =3D=3D __ATOMIC_ACQUIRE) > > + asm volatile("ldaxrh %w[tmp], [%x[addr]]" > > + : [tmp] "=3D&r" (tmp) > > + : [addr] "r"(addr) > > + : "memory"); > > + else if (memorder =3D=3D __ATOMIC_RELAXED) > > + asm volatile("ldxrh %w[tmp], [%x[addr]]" > > + : [tmp] "=3D&r" (tmp) > > + : [addr] "r"(addr) > > + : "memory"); > > + return tmp; > > +} > > + > > +static __rte_always_inline uint32_t > > +rte_atomic_load_ex_32(volatile uint32_t *addr, int memorder) > > +{ > > + uint32_t tmp; > > + assert((memorder =3D=3D __ATOMIC_ACQUIRE) > > + || (memorder =3D=3D __ATOMIC_RELAXED)); > > + if (memorder =3D=3D __ATOMIC_ACQUIRE) > > + asm volatile("ldaxr %w[tmp], [%x[addr]]" > > + : [tmp] "=3D&r" (tmp) > > + : [addr] "r"(addr) > > + : "memory"); > > + else if (memorder =3D=3D __ATOMIC_RELAXED) > > + asm volatile("ldxr %w[tmp], [%x[addr]]" > > + : [tmp] "=3D&r" (tmp) > > + : [addr] "r"(addr) > > + : "memory"); > > + return tmp; > > +} > > + > > +static __rte_always_inline uint64_t > > +rte_atomic_load_ex_64(volatile uint64_t *addr, int memorder) > > +{ > > + uint64_t tmp; > > + assert((memorder =3D=3D __ATOMIC_ACQUIRE) > > + || (memorder =3D=3D __ATOMIC_RELAXED)); > > + if (memorder =3D=3D __ATOMIC_ACQUIRE) > > + asm volatile("ldaxr %x[tmp], [%x[addr]]" > > + : [tmp] "=3D&r" (tmp) > > + : [addr] "r"(addr) > > + : "memory"); > > + else if (memorder =3D=3D __ATOMIC_RELAXED) > > + asm volatile("ldxr %x[tmp], [%x[addr]]" > > + : [tmp] "=3D&r" (tmp) > > + : [addr] "r"(addr) > > + : "memory"); > > + return tmp; > > +} > > + > > +#ifdef RTE_WAIT_UNTIL_EQUAL_ARCH_DEFINED > > +static __rte_always_inline void > > +rte_wait_until_equal_16(volatile uint16_t *addr, uint16_t expected, > > +int memorder) > > +{ > > + if (__atomic_load_n(addr, memorder) !=3D expected) { > > + rte_sevl(); > > + do { > > + rte_wfe(); > > + } while (rte_atomic_load_ex_16(addr, memorder) !=3D > expected); > > + } > > +} > > + > > +static __rte_always_inline void > > +rte_wait_until_equal_32(volatile uint32_t *addr, uint32_t expected, > > +int memorder) > > +{ > > + if (__atomic_load_n(addr, memorder) !=3D expected) { > > + rte_sevl(); > > + do { > > + rte_wfe(); > > + } while (__atomic_load_n(addr, memorder) !=3D expected); >=20 > Here and in _64, shouldn't it be: > rte_atomic_load_ex_.. Thanks for spotting this error, David also spotted it. Sorry for that.=20 >=20 > > + } > > +} > > + > > +static __rte_always_inline void > > +rte_wait_until_equal_64(volatile uint64_t *addr, uint64_t expected, > > +int memorder) > > +{ > > + if (__atomic_load_n(addr, memorder) !=3D expected) { > > + rte_sevl(); > > + do { > > + rte_wfe(); > > + } while (__atomic_load_n(addr, memorder) !=3D expected); > > + } > > +} > > +#endif > > + > > #ifdef __cplusplus > > } > > #endif > > diff --git a/lib/librte_eal/common/include/generic/rte_pause.h > b/lib/librte_eal/common/include/generic/rte_pause.h > > index 52bd4db..9d42e32 100644 > > --- a/lib/librte_eal/common/include/generic/rte_pause.h > > +++ b/lib/librte_eal/common/include/generic/rte_pause.h > > @@ -1,5 +1,6 @@ > > /* SPDX-License-Identifier: BSD-3-Clause > > * Copyright(c) 2017 Cavium, Inc > > + * Copyright(c) 2019 Arm Limited > > */ > > > > #ifndef _RTE_PAUSE_H_ > > @@ -12,6 +13,12 @@ > > * > > */ > > > > +#include > > +#include > > +#include > > +#include > > +#include > > + > > /** > > * Pause CPU execution for a short while > > * > > @@ -20,4 +27,96 @@ > > */ > > static inline void rte_pause(void); > > > > +/** > > + * @warning > > + * @b EXPERIMENTAL: this API may change, or be removed, without prior > notice > > + * > > + * Wait for *addr to be updated with a 16-bit expected value, with a r= elaxed > > + * memory ordering model meaning the loads around this API can be > reordered. > > + * > > + * @param addr > > + * A pointer to the memory location. > > + * @param expected > > + * A 16-bit expected value to be in the memory location. > > + * @param memorder > > + * Two different memory orders that can be specified: > > + * __ATOMIC_ACQUIRE and __ATOMIC_RELAXED. These map to > > + * C++11 memory orders with the same names, see the C++11 standard or > > + * the GCC wiki on atomic synchronization for detailed definition. > > + */ > > +__rte_experimental > > +static __rte_always_inline void > > +rte_wait_until_equal_16(volatile uint16_t *addr, uint16_t expected, > > +int memorder); > > + > > +/** > > + * @warning > > + * @b EXPERIMENTAL: this API may change, or be removed, without prior > notice > > + * > > + * Wait for *addr to be updated with a 32-bit expected value, with a r= elaxed > > + * memory ordering model meaning the loads around this API can be > reordered. > > + * > > + * @param addr > > + * A pointer to the memory location. > > + * @param expected > > + * A 32-bit expected value to be in the memory location. > > + * @param memorder > > + * Two different memory orders that can be specified: > > + * __ATOMIC_ACQUIRE and __ATOMIC_RELAXED. These map to > > + * C++11 memory orders with the same names, see the C++11 standard or > > + * the GCC wiki on atomic synchronization for detailed definition. > > + */ > > +__rte_experimental > > +static __rte_always_inline void > > +rte_wait_until_equal_32(volatile uint32_t *addr, uint32_t expected, > > +int memorder); > > + > > +/** > > + * @warning > > + * @b EXPERIMENTAL: this API may change, or be removed, without prior > notice > > + * > > + * Wait for *addr to be updated with a 64-bit expected value, with a r= elaxed > > + * memory ordering model meaning the loads around this API can be > reordered. > > + * > > + * @param addr > > + * A pointer to the memory location. > > + * @param expected > > + * A 64-bit expected value to be in the memory location. > > + * @param memorder > > + * Two different memory orders that can be specified: > > + * __ATOMIC_ACQUIRE and __ATOMIC_RELAXED. These map to > > + * C++11 memory orders with the same names, see the C++11 standard or > > + * the GCC wiki on atomic synchronization for detailed definition. > > + */ > > +__rte_experimental > > +static __rte_always_inline void > > +rte_wait_until_equal_64(volatile uint64_t *addr, uint64_t expected, > > +int memorder); > > + > > +#ifndef RTE_WAIT_UNTIL_EQUAL_ARCH_DEFINED > > +static __rte_always_inline void > > +rte_wait_until_equal_16(volatile uint16_t *addr, uint16_t expected, > > +int memorder) > > +{ > > + while (__atomic_load_n(addr, memorder) !=3D expected) > > + rte_pause(); > > +} > > + > > +static __rte_always_inline void > > +rte_wait_until_equal_32(volatile uint32_t *addr, uint32_t expected, > > +int memorder) > > +{ > > + while (__atomic_load_n(addr, memorder) !=3D expected) > > + rte_pause(); > > +} > > + > > +static __rte_always_inline void > > +rte_wait_until_equal_64(volatile uint64_t *addr, uint64_t expected, > > +int memorder) > > +{ > > + while (__atomic_load_n(addr, memorder) !=3D expected) > > + rte_pause(); > > +} > > +#endif > > + > > #endif /* _RTE_PAUSE_H_ */ > > -- > > 2.7.4