From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id C2688A2EEB for ; Sat, 14 Sep 2019 17:00:37 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 68CD61E917; Sat, 14 Sep 2019 17:00:22 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 4F3781E912 for ; Sat, 14 Sep 2019 17:00:20 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CDC3428; Sat, 14 Sep 2019 08:00:19 -0700 (PDT) Received: from net-arm-thunderx2-01.test.ast.arm.com (net-arm-thunderx2-01.shanghai.arm.com [10.169.40.40]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id A4F503F67D; Sat, 14 Sep 2019 08:00:17 -0700 (PDT) From: Gavin Hu To: dev@dpdk.org Cc: nd@arm.com, thomas@monjalon.net, stephen@networkplumber.org, hemant.agrawal@nxp.com, jerinj@marvell.com, pbhagavatula@marvell.com, Honnappa.Nagarahalli@arm.com, ruifeng.wang@arm.com, phil.yang@arm.com, steve.capper@arm.com Date: Sat, 14 Sep 2019 22:59:51 +0800 Message-Id: <1568473196-34972-3-git-send-email-gavin.hu@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1568473196-34972-1-git-send-email-gavin.hu@arm.com> References: <1568473196-34972-1-git-send-email-gavin.hu@arm.com> In-Reply-To: <1561911676-37718-1-git-send-email-gavin.hu@arm.com> References: <1561911676-37718-1-git-send-email-gavin.hu@arm.com> Subject: [dpdk-dev] [PATCH v6 2/7] eal: add the APIs to wait until equal X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The rte_wait_until_equal_xx APIs abstract the functionality of 'polling for a memory location to become equal to a given value'. Add the RTE_ARM_USE_WFE configuration entry for aarch64, disabled by default. When it is enabled, the above APIs will call WFE instruction to save CPU cycles and power. Signed-off-by: Gavin Hu Reviewed-by: Ruifeng Wang Reviewed-by: Steve Capper Reviewed-by: Ola Liljedahl Reviewed-by: Honnappa Nagarahalli Reviewed-by: Phil Yang Acked-by: Pavan Nikhilesh --- config/arm/meson.build | 1 + config/common_base | 5 + .../common/include/arch/arm/rte_pause_64.h | 30 ++++++ lib/librte_eal/common/include/generic/rte_pause.h | 103 +++++++++++++++++++++ 4 files changed, 139 insertions(+) diff --git a/config/arm/meson.build b/config/arm/meson.build index 979018e..b4b4cac 100644 --- a/config/arm/meson.build +++ b/config/arm/meson.build @@ -26,6 +26,7 @@ flags_common_default = [ ['RTE_LIBRTE_AVP_PMD', false], ['RTE_SCHED_VECTOR', false], + ['RTE_ARM_USE_WFE', false], ] flags_generic = [ diff --git a/config/common_base b/config/common_base index 8ef75c2..8861713 100644 --- a/config/common_base +++ b/config/common_base @@ -111,6 +111,11 @@ CONFIG_RTE_MAX_VFIO_CONTAINERS=64 CONFIG_RTE_MALLOC_DEBUG=n CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n CONFIG_RTE_USE_LIBBSD=n +# Use WFE instructions to implement the rte_wait_for_equal_xxx APIs, +# calling these APIs put the cores in low power state while waiting +# for the memory address to become equal to the expected value. +# This is supported only by aarch64. +CONFIG_RTE_ARM_USE_WFE=n # # Recognize/ignore the AVX/AVX512 CPU flags for performance/power testing. diff --git a/lib/librte_eal/common/include/arch/arm/rte_pause_64.h b/lib/librte_eal/common/include/arch/arm/rte_pause_64.h index 93895d3..dabde17 100644 --- a/lib/librte_eal/common/include/arch/arm/rte_pause_64.h +++ b/lib/librte_eal/common/include/arch/arm/rte_pause_64.h @@ -1,5 +1,6 @@ /* SPDX-License-Identifier: BSD-3-Clause * Copyright(c) 2017 Cavium, Inc + * Copyright(c) 2019 Arm Limited */ #ifndef _RTE_PAUSE_ARM64_H_ @@ -17,6 +18,35 @@ static inline void rte_pause(void) asm volatile("yield" ::: "memory"); } +#ifdef RTE_ARM_USE_WFE +#define __WAIT_UNTIL_EQUAL(name, asm_op, wide, type) \ +static __rte_always_inline void \ +rte_wait_until_equal_##name(volatile type * addr, type expected) \ +{ \ + type tmp; \ + asm volatile( \ + #asm_op " %" #wide "[tmp], %[addr]\n" \ + "cmp %" #wide "[tmp], %" #wide "[expected]\n" \ + "b.eq 2f\n" \ + "sevl\n" \ + "1: wfe\n" \ + #asm_op " %" #wide "[tmp], %[addr]\n" \ + "cmp %" #wide "[tmp], %" #wide "[expected]\n" \ + "bne 1b\n" \ + "2:\n" \ + : [tmp] "=&r" (tmp) \ + : [addr] "Q"(*addr), [expected] "r"(expected) \ + : "cc", "memory"); \ +} +/* Wait for *addr to be updated with expected value */ +__WAIT_UNTIL_EQUAL(relaxed_16, ldxrh, w, uint16_t) +__WAIT_UNTIL_EQUAL(acquire_16, ldaxrh, w, uint16_t) +__WAIT_UNTIL_EQUAL(relaxed_32, ldxr, w, uint32_t) +__WAIT_UNTIL_EQUAL(acquire_32, ldaxr, w, uint32_t) +__WAIT_UNTIL_EQUAL(relaxed_64, ldxr, x, uint64_t) +__WAIT_UNTIL_EQUAL(acquire_64, ldaxr, x, uint64_t) +#endif + #ifdef __cplusplus } #endif diff --git a/lib/librte_eal/common/include/generic/rte_pause.h b/lib/librte_eal/common/include/generic/rte_pause.h index 52bd4db..772a2ab 100644 --- a/lib/librte_eal/common/include/generic/rte_pause.h +++ b/lib/librte_eal/common/include/generic/rte_pause.h @@ -1,5 +1,6 @@ /* SPDX-License-Identifier: BSD-3-Clause * Copyright(c) 2017 Cavium, Inc + * Copyright(c) 2019 Arm Limited */ #ifndef _RTE_PAUSE_H_ @@ -12,6 +13,10 @@ * */ +#include +#include +#include + /** * Pause CPU execution for a short while * @@ -20,4 +25,102 @@ */ static inline void rte_pause(void); +/** + * Wait for *addr to be updated with a 16-bit expected value, with a relaxed memory + * ordering model meaning the loads around this API can be reordered. + * + * @param addr + * A pointer to the memory location. + * @param expected + * A 16-bit expected value to be in the memory location. + */ +__rte_always_inline +static void +rte_wait_until_equal_relaxed_16(volatile uint16_t *addr, uint16_t expected); + +/** + * Wait for *addr to be updated with a 32-bit expected value, with a relaxed memory + * ordering model meaning the loads around this API can be reordered. + * + * @param addr + * A pointer to the memory location. + * @param expected + * A 32-bit expected value to be in the memory location. + */ +__rte_always_inline +static void +rte_wait_until_equal_relaxed_32(volatile uint32_t *addr, uint32_t expected); + +/** + * Wait for *addr to be updated with a 64-bit expected value, with a relaxed memory + * ordering model meaning the loads around this API can be reordered. + * + * @param addr + * A pointer to the memory location. + * @param expected + * A 64-bit expected value to be in the memory location. + */ +__rte_always_inline +static void +rte_wait_until_equal_relaxed_64(volatile uint64_t *addr, uint64_t expected); + +/** + * Wait for *addr to be updated with a 16-bit expected value, with an acquire memory + * ordering model meaning the loads after this API can't be observed before this API. + * + * @param addr + * A pointer to the memory location. + * @param expected + * A 16-bit expected value to be in the memory location. + */ +__rte_always_inline +static void +rte_wait_until_equal_acquire_16(volatile uint16_t *addr, uint16_t expected); + +/** + * Wait for *addr to be updated with a 32-bit expected value, with an acquire memory + * ordering model meaning the loads after this API can't be observed before this API. + * + * @param addr + * A pointer to the memory location. + * @param expected + * A 32-bit expected value to be in the memory location. + */ +__rte_always_inline +static void +rte_wait_until_equal_acquire_32(volatile uint32_t *addr, uint32_t expected); + +/** + * Wait for *addr to be updated with a 64-bit expected value, with an acquire memory + * ordering model meaning the loads after this API can't be observed before this API. + * + * @param addr + * A pointer to the memory location. + * @param expected + * A 64-bit expected value to be in the memory location. + */ +__rte_always_inline +static void +rte_wait_until_equal_acquire_64(volatile uint64_t *addr, uint64_t expected); + +#if !defined(RTE_ARM_USE_WFE) +#define __WAIT_UNTIL_EQUAL(op_name, size, type, memorder) \ +__rte_always_inline \ +static void \ +rte_wait_until_equal_##op_name##_##size(volatile type *addr, \ + type expected) \ +{ \ + while (__atomic_load_n(addr, memorder) != expected) \ + rte_pause(); \ +} + +/* Wait for *addr to be updated with expected value */ +__WAIT_UNTIL_EQUAL(relaxed, 16, uint16_t, __ATOMIC_RELAXED) +__WAIT_UNTIL_EQUAL(acquire, 16, uint16_t, __ATOMIC_ACQUIRE) +__WAIT_UNTIL_EQUAL(relaxed, 32, uint32_t, __ATOMIC_RELAXED) +__WAIT_UNTIL_EQUAL(acquire, 32, uint32_t, __ATOMIC_ACQUIRE) +__WAIT_UNTIL_EQUAL(relaxed, 64, uint64_t, __ATOMIC_RELAXED) +__WAIT_UNTIL_EQUAL(acquire, 64, uint64_t, __ATOMIC_ACQUIRE) +#endif /* RTE_ARM_USE_WFE */ + #endif /* _RTE_PAUSE_H_ */ -- 2.7.4