From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2FFF5A328D for ; Tue, 22 Oct 2019 12:17:28 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 020458F96; Tue, 22 Oct 2019 12:17:27 +0200 (CEST) Received: from us-smtp-1.mimecast.com (us-smtp-1.mimecast.com [205.139.110.61]) by dpdk.org (Postfix) with ESMTP id C17EF5B3E for ; Tue, 22 Oct 2019 12:17:26 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1571739446; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RGt1kdUGKU2Pr3SHjMKQhvy+/SUgE+OcujSJaVgjzVQ=; b=W4qDbKfTXEWR26isCNwns5ifmyny52bUhF0Xzjl5MlmTHMIGPPg85u8VIyPAyW+gF2bB7Y 7o1RxnUtEU/0edeggx61SjABLYa/dEgD1dqy7RHGYho1KetWKPJYoKRA54S8rfSwYhI4zL msMMZvdfwr9NABHp8QHnuk0h9aaueHs= Received: from mail-io1-f71.google.com (mail-io1-f71.google.com [209.85.166.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-4-JBRKKSD5Na-HkTn_AsmGlQ-1; Tue, 22 Oct 2019 06:17:19 -0400 Received: by mail-io1-f71.google.com with SMTP id g15so19884791ioc.0 for ; Tue, 22 Oct 2019 03:17:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=5mcLG3d4ps88HX3BaldKDqvPYe9UP8xxauZkA9NsP2Q=; b=DzsjjKrc6WO1yGTdoaJSQqEK6dmqjI1xfqBqshBnl6dQuwSPNjHgPbt9G677FFqkQF VBPVbXmxrPd1n0dGVEqFBNsZVRqGTG6nfZ2lk0sftHbPLZYxTkv5Jp327+4tB6I8qNlc y+NO2L00Cmtfa8cCdwzZ9JQPy8en71qlfkNra1Q1Vp162Gr7vDi+V0ivbV+8X3GpQofQ DU8W/fcU7g/65LgnWM96UqyZGJCVAABnklYLscQcTmDmbTsqJ6qv4jv42+F342dJhiWU 0hK/qx/6dxs4JF0aagTIVWDbchtnfGJOyY0g4PyUOIfQYoDN2Eyn+VgQdZ/DzpWp7Huh l1lA== X-Gm-Message-State: APjAAAVGf2uCLfylnDxUkBU6yWDg5PohHV6eqFdp4Bxeg5zqwSKyFhiL sUUPeyARdlGLkYNL8cHUI5twQkJNoKPIqWBTubTp0JHeeD4IPM3a7ldQuS4xsyhtOL2Lkp95Ck2 qnKti6aA/TweaVJqrgEM= X-Received: by 2002:a05:6602:21d8:: with SMTP id c24mr2783411ioc.30.1571739439240; Tue, 22 Oct 2019 03:17:19 -0700 (PDT) X-Google-Smtp-Source: APXvYqwWrTjpTIHUd605jmNnveeQOqomQZYjB/U31C0FcBzLTRxqrUGdYuHCeShVdIVxXziBIKvT4ylzRkf49VCEbzA= X-Received: by 2002:a05:6602:21d8:: with SMTP id c24mr2783386ioc.30.1571739438819; Tue, 22 Oct 2019 03:17:18 -0700 (PDT) MIME-Version: 1.0 References: <1561911676-37718-1-git-send-email-gavin.hu@arm.com> <1571651279-51857-1-git-send-email-gavin.hu@arm.com> <1571651279-51857-3-git-send-email-gavin.hu@arm.com> <2601191342CEEE43887BDE71AB97725801A8C6DDF9@IRSMSX104.ger.corp.intel.com> In-Reply-To: <2601191342CEEE43887BDE71AB97725801A8C6DDF9@IRSMSX104.ger.corp.intel.com> From: David Marchand Date: Tue, 22 Oct 2019 12:17:07 +0200 Message-ID: To: "Ananyev, Konstantin" Cc: Gavin Hu , dev , nd , Thomas Monjalon , Stephen Hemminger , Hemant Agrawal , Jerin Jacob Kollanukkaran , Pavan Nikhilesh , Honnappa Nagarahalli , "Ruifeng Wang (Arm Technology China)" , Phil Yang , Steve Capper X-MC-Unique: JBRKKSD5Na-HkTn_AsmGlQ-1 X-Mimecast-Spam-Score: 0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-dev] [PATCH v8 2/6] eal: add the APIs to wait until equal X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Tue, Oct 22, 2019 at 11:37 AM Ananyev, Konstantin wrote: > > > > The rte_wait_until_equal_xx APIs abstract the functionality of > > > 'polling for a memory location to become equal to a given value'. > > > > > > Add the RTE_ARM_USE_WFE configuration entry for aarch64, disabled > > > by default. When it is enabled, the above APIs will call WFE instruct= ion > > > to save CPU cycles and power. > > > > > > From a VM, when calling this API on aarch64, it may trap in and out t= o > > > release vCPUs whereas cause high exit latency. Since kernel 4.18.20 a= n > > > adaptive trapping mechanism is introduced to balance the latency and > > > workload. > > > > > > Signed-off-by: Gavin Hu > > > Reviewed-by: Ruifeng Wang > > > Reviewed-by: Steve Capper > > > Reviewed-by: Ola Liljedahl > > > Reviewed-by: Honnappa Nagarahalli > > > Reviewed-by: Phil Yang > > > Acked-by: Pavan Nikhilesh > > > Acked-by: Jerin Jacob > > > --- > > > config/arm/meson.build | 1 + > > > config/common_base | 5 ++ > > > .../common/include/arch/arm/rte_pause_64.h | 26 ++++++ > > > lib/librte_eal/common/include/generic/rte_pause.h | 93 ++++++++++++= ++++++++++ > > > 4 files changed, 125 insertions(+) > > > > > > diff --git a/config/arm/meson.build b/config/arm/meson.build > > > index 979018e..b4b4cac 100644 > > > --- a/config/arm/meson.build > > > +++ b/config/arm/meson.build > > > @@ -26,6 +26,7 @@ flags_common_default =3D [ > > > ['RTE_LIBRTE_AVP_PMD', false], > > > > > > ['RTE_SCHED_VECTOR', false], > > > + ['RTE_ARM_USE_WFE', false], > > > ] > > > > > > flags_generic =3D [ > > > diff --git a/config/common_base b/config/common_base > > > index e843a21..c812156 100644 > > > --- a/config/common_base > > > +++ b/config/common_base > > > @@ -111,6 +111,11 @@ CONFIG_RTE_MAX_VFIO_CONTAINERS=3D64 > > > CONFIG_RTE_MALLOC_DEBUG=3Dn > > > CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=3Dn > > > CONFIG_RTE_USE_LIBBSD=3Dn > > > +# Use WFE instructions to implement the rte_wait_for_equal_xxx APIs, > > > +# calling these APIs put the cores in low power state while waiting > > > +# for the memory address to become equal to the expected value. > > > +# This is supported only by aarch64. > > > +CONFIG_RTE_ARM_USE_WFE=3Dn > > > > > > # > > > # Recognize/ignore the AVX/AVX512 CPU flags for performance/power te= sting. > > > diff --git a/lib/librte_eal/common/include/arch/arm/rte_pause_64.h b/= lib/librte_eal/common/include/arch/arm/rte_pause_64.h > > > index 93895d3..eb8f73e 100644 > > > --- a/lib/librte_eal/common/include/arch/arm/rte_pause_64.h > > > +++ b/lib/librte_eal/common/include/arch/arm/rte_pause_64.h > > > @@ -1,5 +1,6 @@ > > > /* SPDX-License-Identifier: BSD-3-Clause > > > * Copyright(c) 2017 Cavium, Inc > > > + * Copyright(c) 2019 Arm Limited > > > */ > > > > Before including generic/rte_pause.h, put a check like: > > > > #ifdef RTE_ARM_USE_WFE > > #define RTE_ARCH_HAS_WFE > > #endif > > > > #include "generic/rte_pause.h" > > > > > > > > #ifndef _RTE_PAUSE_ARM64_H_ > > > @@ -17,6 +18,31 @@ static inline void rte_pause(void) > > > asm volatile("yield" ::: "memory"); > > > } > > > > > > +#ifdef RTE_ARM_USE_WFE > > > +#define sev() { asm volatile("sev" : : : "memory") } > > > +#define wfe() { asm volatile("wfe" : : : "memory") } > > > + > > > +#define __WAIT_UNTIL_EQUAL(type, size, addr, expected, memorder) \ > > > +__rte_experimental \ > > > > The experimental tag is unnecessary here. > > We only need it in the function prototype (in the generic header). > > > > > +static __rte_always_inline void = \ > > > +rte_wait_until_equal_##size(volatile type * addr, type expected,\ > > > +int memorder) \ > > > +{ \ > > > + if (__atomic_load_n(addr, memorder) !=3D expected) { \ > > > + sev(); = \ > > > + do { = \ > > > + wfe(); = \ > > > + } while (__atomic_load_n(addr, memorder) !=3D expecte= d); \ > > > + } = \ > > > +} > > > +__WAIT_UNTIL_EQUAL(uint16_t, 16, addr, expected, memorder) > > > +__WAIT_UNTIL_EQUAL(uint32_t, 32, addr, expected, memorder) > > > +__WAIT_UNTIL_EQUAL(uint64_t, 64, addr, expected, memorder) > > > + > > > +#undef __WAIT_UNTIL_EQUAL > > Might be instead of defining/undefining these macros, just > define explicitly these 3 functions? > Now they are really small, so I think it would be an easier/cleaner way. > Yes, a bit of code duplication, but as I said they are really small now. > Same thought about generic version. I don't really like those macros defining inlines either. I am fine with this little duplication, so +1 from me. > > > > > Missing #undef on sev and wfe macros. > > Actually should we undefine them? > Or should we add rte_ prefix (which is needed anyway I suppose) > and have them always defined? > Might be you can reuse them in other arm specific places too (spinlock, r= wlock, etc.) > Actually probably it is possible to make them either emiting a proper ins= tructions or NOP, > then you'll need RTE_ARM_USE_WFE only around these macros. Interesting idea, but only if it gets used in this series. I don't want to see stuff that will not be used later. > > I.E > > #ifdef RTE_ARM_USE_WFE > #define rte_sev() { asm volatile("sev" : : : "memory") } > #define rte_wfe() { asm volatile("wfe" : : : "memory") } > #else > static inline void rte_sev(void) > { > } > static inline void rte_wfe(void) > { > rte_pause(); > } > #endif > > And then just one common version of _wait_ functios: > > static __rte_always_inline void \ > rte_wait_until_equal_32(volatile type * addr, type expected, int memorder= ) > { \ > if (__atomic_load_n(addr, memorder) !=3D expected) { > rte_sev(); > do { > rte_wfe(); > } while (__atomic_load_n(addr, memorder) !=3D expec= ted); > } > } > > [snip] > > Here, change this check to: > > > > #ifndef RTE_ARCH_HAS_WFE > > > > > > > +#ifndef RTE_ARM_USE_WFE > > Might be something arch neutral in name here? > RTE_WAIT_UNTIL_EQUAL_ARCH_DEFINED or so? Yes, better than my suggestion. Gavin, I noticed that you added a #ifndef ARM in the generic rte_spinlock.h header later in this series. Please, can you think of a way to avoid this? Maybe applying the same pattern to the spinlock? Thanks. -- David Marchand