From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3359AA04B4; Fri, 8 Nov 2019 17:39:04 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 151911C230; Fri, 8 Nov 2019 17:39:03 +0100 (CET) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by dpdk.org (Postfix) with ESMTP id A39551BFC4 for ; Fri, 8 Nov 2019 17:39:00 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga107.fm.intel.com with ESMTP; 08 Nov 2019 08:38:59 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.68,282,1569308400"; d="scan'208";a="233685765" Received: from irsmsx154.ger.corp.intel.com ([163.33.192.96]) by fmsmga002.fm.intel.com with ESMTP; 08 Nov 2019 08:38:58 -0800 Received: from irsmsx156.ger.corp.intel.com (10.108.20.68) by IRSMSX154.ger.corp.intel.com (163.33.192.96) with Microsoft SMTP Server (TLS) id 14.3.439.0; Fri, 8 Nov 2019 16:38:57 +0000 Received: from irsmsx104.ger.corp.intel.com ([169.254.5.252]) by IRSMSX156.ger.corp.intel.com ([169.254.3.227]) with mapi id 14.03.0439.000; Fri, 8 Nov 2019 16:38:57 +0000 From: "Ananyev, Konstantin" To: David Marchand , "dev@dpdk.org" CC: "nd@arm.com" , Gavin Hu , Thomas Monjalon , "Mcnamara, John" , "Kovacevic, Marko" , Jerin Jacob , Jan Viktorin Thread-Topic: [PATCH v13 2/5] eal: add the APIs to wait until equal Thread-Index: AQHVlbNd73fGjCsb40qrF5PYyLu+BKeBeVbA Date: Fri, 8 Nov 2019 16:38:56 +0000 Message-ID: <2601191342CEEE43887BDE71AB97725801A8C833FA@IRSMSX104.ger.corp.intel.com> References: <1561911676-37718-1-git-send-email-gavin.hu@arm.com> <1573162528-16230-1-git-send-email-david.marchand@redhat.com> <1573162528-16230-3-git-send-email-david.marchand@redhat.com> In-Reply-To: <1573162528-16230-3-git-send-email-david.marchand@redhat.com> Accept-Language: en-IE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiZjFhNzJjZDEtZDQwZS00YmY4LWEwNjEtOGU1NzUwMmE1MmRkIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiXC9iMENvNmdQc3RFczBDU1VjQ0hJOHU3YjYwK2Rsdk9lRzRNUzZVNTFZM0JLMWw5VzdSMUxBNUFhUzRLcUpGNmYifQ== x-ctpclassification: CTP_NT dlp-product: dlpe-windows dlp-version: 11.2.0.6 dlp-reaction: no-action x-originating-ip: [163.33.239.182] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v13 2/5] eal: add the APIs to wait until equal X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi David, > From: Gavin Hu >=20 > The rte_wait_until_equal_xx APIs abstract the functionality of > 'polling for a memory location to become equal to a given value'. >=20 > Add the RTE_ARM_USE_WFE configuration entry for aarch64, disabled > by default. When it is enabled, the above APIs will call WFE instruction > to save CPU cycles and power. >=20 > From a VM, when calling this API on aarch64, it may trap in and out to > release vCPUs whereas cause high exit latency. Since kernel 4.18.20 an > adaptive trapping mechanism is introduced to balance the latency and > workload. >=20 > Signed-off-by: Gavin Hu > Reviewed-by: Ruifeng Wang > Reviewed-by: Steve Capper > Reviewed-by: Ola Liljedahl > Reviewed-by: Honnappa Nagarahalli > Reviewed-by: Phil Yang > Acked-by: Pavan Nikhilesh > Acked-by: Jerin Jacob > Acked-by: Konstantin Ananyev > Signed-off-by: David Marchand > --- > Changelog since v12: > - added release notes update, > - fixed function prototypes indent, > - reimplemented the arm implementation without exposing internal inline > functions, > - added asserts in generic implementation, >=20 > --- > config/arm/meson.build | 1 + > config/common_base | 5 + > doc/guides/rel_notes/release_19_11.rst | 5 + > .../common/include/arch/arm/rte_pause_64.h | 133 +++++++++++++++= ++++++ > lib/librte_eal/common/include/generic/rte_pause.h | 105 +++++++++++++++= + > 5 files changed, 249 insertions(+) >=20 > diff --git a/config/arm/meson.build b/config/arm/meson.build > index 46dff3a..ea47425 100644 > --- a/config/arm/meson.build > +++ b/config/arm/meson.build > @@ -26,6 +26,7 @@ flags_common_default =3D [ > ['RTE_LIBRTE_AVP_PMD', false], >=20 > ['RTE_SCHED_VECTOR', false], > + ['RTE_ARM_USE_WFE', false], > ] >=20 > flags_generic =3D [ > diff --git a/config/common_base b/config/common_base > index 1858598..bb1b1ed 100644 > --- a/config/common_base > +++ b/config/common_base > @@ -110,6 +110,11 @@ CONFIG_RTE_MAX_VFIO_CONTAINERS=3D64 > CONFIG_RTE_MALLOC_DEBUG=3Dn > CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=3Dn > CONFIG_RTE_USE_LIBBSD=3Dn > +# Use WFE instructions to implement the rte_wait_for_equal_xxx APIs, > +# calling these APIs put the cores in low power state while waiting > +# for the memory address to become equal to the expected value. > +# This is supported only by aarch64. > +CONFIG_RTE_ARM_USE_WFE=3Dn >=20 > # > # Recognize/ignore the AVX/AVX512 CPU flags for performance/power testin= g. > diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_note= s/release_19_11.rst > index fe11b4b..af5f2c5 100644 > --- a/doc/guides/rel_notes/release_19_11.rst > +++ b/doc/guides/rel_notes/release_19_11.rst > @@ -65,6 +65,11 @@ New Features >=20 > The lock-free stack implementation is enabled for aarch64 platforms. >=20 > +* **Added Wait Until Equal API.** > + > + A new API has been added to wait for a memory location to be updated w= ith a > + 16-bit, 32-bit, 64-bit value. > + > * **Changed mempool allocation behaviour.** >=20 > Objects are no longer across pages by default. > diff --git a/lib/librte_eal/common/include/arch/arm/rte_pause_64.h b/lib/= librte_eal/common/include/arch/arm/rte_pause_64.h > index 93895d3..e87d10b 100644 > --- a/lib/librte_eal/common/include/arch/arm/rte_pause_64.h > +++ b/lib/librte_eal/common/include/arch/arm/rte_pause_64.h > @@ -1,5 +1,6 @@ > /* SPDX-License-Identifier: BSD-3-Clause > * Copyright(c) 2017 Cavium, Inc > + * Copyright(c) 2019 Arm Limited > */ >=20 > #ifndef _RTE_PAUSE_ARM64_H_ > @@ -10,6 +11,11 @@ extern "C" { > #endif >=20 > #include > + > +#ifdef RTE_ARM_USE_WFE > +#define RTE_WAIT_UNTIL_EQUAL_ARCH_DEFINED > +#endif > + > #include "generic/rte_pause.h" >=20 > static inline void rte_pause(void) > @@ -17,6 +23,133 @@ static inline void rte_pause(void) > asm volatile("yield" ::: "memory"); > } >=20 > +#ifdef RTE_WAIT_UNTIL_EQUAL_ARCH_DEFINED > + > +/* Send an event to quit WFE. */ > +#define __SEVL() { asm volatile("sevl" : : : "memory"); } > + > +/* Put processor into low power WFE(Wait For Event) state. */ > +#define __WFE() { asm volatile("wfe" : : : "memory"); } > + > +static __rte_always_inline void > +rte_wait_until_equal_16(volatile uint16_t *addr, uint16_t expected, > + int memorder) > +{ > + uint16_t value; > + > + assert(memorder =3D=3D __ATOMIC_ACQUIRE || memorder =3D=3D __ATOMIC_REL= AXED); > + > + /* > + * Atomic exclusive load from addr, it returns the 16-bit content of > + * *addr while making it 'monitored',when it is written by someone > + * else, the 'monitored' state is cleared and a event is generated > + * implicitly to exit WFE. > + */ > +#define __LOAD_EXC_16(src, dst, memorder) { \ > + if (memorder =3D=3D __ATOMIC_RELAXED) { \ > + asm volatile("ldxrh %w[tmp], [%x[addr]]" \ > + : [tmp] "=3D&r" (dst) \ > + : [addr] "r"(src) \ > + : "memory"); \ > + } else { \ > + asm volatile("ldaxrh %w[tmp], [%x[addr]]" \ > + : [tmp] "=3D&r" (dst) \ > + : [addr] "r"(src) \ > + : "memory"); \ > + } } > + > + __LOAD_EXC_16(addr, value, memorder) > + if (value !=3D expected) { > + __SEVL() > + do { > + __WFE() > + __LOAD_EXC_16(addr, value, memorder) > + } while (value !=3D expected); > + } > +#undef __LOAD_EXC_16 > +} > + > +static __rte_always_inline void > +rte_wait_until_equal_32(volatile uint32_t *addr, uint32_t expected, > + int memorder) > +{ > + uint32_t value; > + > + assert(memorder =3D=3D __ATOMIC_ACQUIRE || memorder =3D=3D __ATOMIC_REL= AXED); > + > + /* > + * Atomic exclusive load from addr, it returns the 32-bit content of > + * *addr while making it 'monitored',when it is written by someone > + * else, the 'monitored' state is cleared and a event is generated > + * implicitly to exit WFE. > + */ > +#define __LOAD_EXC_32(src, dst, memorder) { \ > + if (memorder =3D=3D __ATOMIC_RELAXED) { \ > + asm volatile("ldxr %w[tmp], [%x[addr]]" \ > + : [tmp] "=3D&r" (dst) \ > + : [addr] "r"(src) \ > + : "memory"); \ > + } else { \ > + asm volatile("ldaxr %w[tmp], [%x[addr]]" \ > + : [tmp] "=3D&r" (dst) \ > + : [addr] "r"(src) \ > + : "memory"); \ > + } } > + > + __LOAD_EXC_32(addr, value, memorder) > + if (value !=3D expected) { > + __SEVL() > + do { > + __WFE() > + __LOAD_EXC_32(addr, value, memorder) > + } while (value !=3D expected); > + } > +#undef __LOAD_EXC_32 > +} > + > +static __rte_always_inline void > +rte_wait_until_equal_64(volatile uint64_t *addr, uint64_t expected, > + int memorder) > +{ > + uint64_t value; > + > + assert(memorder =3D=3D __ATOMIC_ACQUIRE || memorder =3D=3D __ATOMIC_REL= AXED); > + > + /* > + * Atomic exclusive load from addr, it returns the 64-bit content of > + * *addr while making it 'monitored',when it is written by someone > + * else, the 'monitored' state is cleared and a event is generated > + * implicitly to exit WFE. > + */ > +#define __LOAD_EXC_64(src, dst, memorder) { \ > + if (memorder =3D=3D __ATOMIC_RELAXED) { \ > + asm volatile("ldxr %x[tmp], [%x[addr]]" \ > + : [tmp] "=3D&r" (dst) \ > + : [addr] "r"(src) \ > + : "memory"); \ > + } else { \ > + asm volatile("ldaxr %x[tmp], [%x[addr]]" \ > + : [tmp] "=3D&r" (dst) \ > + : [addr] "r"(src) \ > + : "memory"); \ > + } } > + > + __LOAD_EXC_64(addr, value, memorder) > + if (value !=3D expected) { > + __SEVL() > + do { > + __WFE() > + __LOAD_EXC_64(addr, value, memorder) > + } while (value !=3D expected); > + } > +} > +#undef __LOAD_EXC_64 > + > +#undef __SEVL > +#undef __WFE Personally I don't see how these define/undef are better then inline functi= ons... Again I suppose they might be re-used in future some other ARM specific low= -level primitvies. My preference would be to keep them as inline function - much cleaner code. But as I don't develop for/use, I wouldn't insist and let you and arm guys = to decide. Konstantin=20 > + > +#endif > + > #ifdef __cplusplus > } > #endif > diff --git a/lib/librte_eal/common/include/generic/rte_pause.h b/lib/libr= te_eal/common/include/generic/rte_pause.h > index 52bd4db..7422785 100644 > --- a/lib/librte_eal/common/include/generic/rte_pause.h > +++ b/lib/librte_eal/common/include/generic/rte_pause.h > @@ -1,5 +1,6 @@ > /* SPDX-License-Identifier: BSD-3-Clause > * Copyright(c) 2017 Cavium, Inc > + * Copyright(c) 2019 Arm Limited > */ >=20 > #ifndef _RTE_PAUSE_H_ > @@ -12,6 +13,12 @@ > * > */ >=20 > +#include > +#include > +#include > +#include > +#include > + > /** > * Pause CPU execution for a short while > * > @@ -20,4 +27,102 @@ > */ > static inline void rte_pause(void); >=20 > +/** > + * @warning > + * @b EXPERIMENTAL: this API may change, or be removed, without prior no= tice > + * > + * Wait for *addr to be updated with a 16-bit expected value, with a rel= axed > + * memory ordering model meaning the loads around this API can be reorde= red. > + * > + * @param addr > + * A pointer to the memory location. > + * @param expected > + * A 16-bit expected value to be in the memory location. > + * @param memorder > + * Two different memory orders that can be specified: > + * __ATOMIC_ACQUIRE and __ATOMIC_RELAXED. These map to > + * C++11 memory orders with the same names, see the C++11 standard or > + * the GCC wiki on atomic synchronization for detailed definition. > + */ > +__rte_experimental > +static __rte_always_inline void > +rte_wait_until_equal_16(volatile uint16_t *addr, uint16_t expected, > + int memorder); > + > +/** > + * @warning > + * @b EXPERIMENTAL: this API may change, or be removed, without prior no= tice > + * > + * Wait for *addr to be updated with a 32-bit expected value, with a rel= axed > + * memory ordering model meaning the loads around this API can be reorde= red. > + * > + * @param addr > + * A pointer to the memory location. > + * @param expected > + * A 32-bit expected value to be in the memory location. > + * @param memorder > + * Two different memory orders that can be specified: > + * __ATOMIC_ACQUIRE and __ATOMIC_RELAXED. These map to > + * C++11 memory orders with the same names, see the C++11 standard or > + * the GCC wiki on atomic synchronization for detailed definition. > + */ > +__rte_experimental > +static __rte_always_inline void > +rte_wait_until_equal_32(volatile uint32_t *addr, uint32_t expected, > + int memorder); > + > +/** > + * @warning > + * @b EXPERIMENTAL: this API may change, or be removed, without prior no= tice > + * > + * Wait for *addr to be updated with a 64-bit expected value, with a rel= axed > + * memory ordering model meaning the loads around this API can be reorde= red. > + * > + * @param addr > + * A pointer to the memory location. > + * @param expected > + * A 64-bit expected value to be in the memory location. > + * @param memorder > + * Two different memory orders that can be specified: > + * __ATOMIC_ACQUIRE and __ATOMIC_RELAXED. These map to > + * C++11 memory orders with the same names, see the C++11 standard or > + * the GCC wiki on atomic synchronization for detailed definition. > + */ > +__rte_experimental > +static __rte_always_inline void > +rte_wait_until_equal_64(volatile uint64_t *addr, uint64_t expected, > + int memorder); > + > +#ifndef RTE_WAIT_UNTIL_EQUAL_ARCH_DEFINED > +static __rte_always_inline void > +rte_wait_until_equal_16(volatile uint16_t *addr, uint16_t expected, > + int memorder) > +{ > + assert(memorder =3D=3D __ATOMIC_ACQUIRE || memorder =3D=3D __ATOMIC_REL= AXED); > + > + while (__atomic_load_n(addr, memorder) !=3D expected) > + rte_pause(); > +} > + > +static __rte_always_inline void > +rte_wait_until_equal_32(volatile uint32_t *addr, uint32_t expected, > + int memorder) > +{ > + assert(memorder =3D=3D __ATOMIC_ACQUIRE || memorder =3D=3D __ATOMIC_REL= AXED); > + > + while (__atomic_load_n(addr, memorder) !=3D expected) > + rte_pause(); > +} > + > +static __rte_always_inline void > +rte_wait_until_equal_64(volatile uint64_t *addr, uint64_t expected, > + int memorder) > +{ > + assert(memorder =3D=3D __ATOMIC_ACQUIRE || memorder =3D=3D __ATOMIC_REL= AXED); > + > + while (__atomic_load_n(addr, memorder) !=3D expected) > + rte_pause(); > +} > +#endif > + > #endif /* _RTE_PAUSE_H_ */ > -- > 1.8.3.1