[dpdk-dev] 回复: [PATCH v8 1/5] eal: add new definitions for wait scheme

[Feifei Wang <Feifei.Wang2@arm.com>]

> -----邮件原件-----
> 发件人: David Marchand <david.marchand@redhat.com>
> 发送时间: Sunday, October 31, 2021 4:39 PM
> 收件人: Feifei Wang <Feifei.Wang2@arm.com>
> 抄送: Ruifeng Wang <Ruifeng.Wang@arm.com>; dev <dev@dpdk.org>; nd
> <nd@arm.com>; Jerin Jacob <jerinjacobk@gmail.com>; Stephen Hemminger
> <stephen@networkplumber.org>; thomas@monjalon.net; Mattias Rönnblom
> <mattias.ronnblom@ericsson.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>
> 主题: Re: [PATCH v8 1/5] eal: add new definitions for wait scheme
> 
> On Fri, Oct 29, 2021 at 10:20 AM Feifei Wang <feifei.wang2@arm.com>
> wrote:
> >
> > Introduce macros as generic interface for address monitoring.
> 
> The main point of this patch is to add a new generic helper.
[Feifei] Thanks for the comments, I will change this commit message.
> 
> 
> >
> > Add '__LOAD_EXC_128' for size of 128. For different size, encapsulate
> > '__LOAD_EXC_16', '__LOAD_EXC_32', '__LOAD_EXC_64' and
> '__LOAD_EXC_128'
> > into a new macro '__LOAD_EXC'.
> 
> ARM macros are just a result of introducing this new helper as a macro.
> I would not mention them.
[Feifei] Ok, I will delete it.
> 
> 
> >
> > Furthermore, to prevent compilation warning in arm:
> > ----------------------------------------------
> > 'warning: implicit declaration of function ...'
> > ----------------------------------------------
> > Delete 'undef' constructions for '__LOAD_EXC_xx', '__SEVL' and '__WFE'.
> > And add ‘__RTE_ARM’ for these macros to fix the namespace.
> > This is because original macros are undefine at the end of the file.
> > If new macro 'rte_wait_event' calls them in other files, they will be
> > seen as 'not defined'.
> 
> 
> About this new helper, it's rather confusing:
> - it is a macro, should be in capital letters,
> - "rte_wait_event(addr, mask, cond, expected)" waits until "*addr & mask
> cond expected" becomes false. I find this confusing. I would invert the
> condition.
> - so far, we had rte_wait_until_* helpers, rte_wait_event seems like a step
> backward as it seems to talk about the ARM stuff (wfe),
[Feifei] So if I understand correctly, we need to avoid using 'wait_event' as name.

> - the masking part is artificial in some cases, at least let's avoid using a too
> generic name, we can decide to add a non-masked helper later.
[Feifei] Ok, I will change this name to match the mask.
> 
> For those reasons, I'd prefer we have something like:
> 
> /*
>  * Wait until *addr & mask makes the condition true. With a relaxed memory
>  * ordering model, the loads around this helper can be reordered.
>  *
>  * @param addr
>  *  A pointer to the memory location.
>  * @param mask
>  *  A mask of *addr bits in interest.
>  * @param cond
>  *  A symbol representing the condition.
>  * @param expected
>  *  An expected value to be in the memory location.
>  * @param memorder
>  *  Two different memory orders that can be specified:
>  *  __ATOMIC_ACQUIRE and __ATOMIC_RELAXED. These map to
>  *  C++11 memory orders with the same names, see the C++11 standard or
>  *  the GCC wiki on atomic synchronization for detailed definition.
>  */
> #define RTE_WAIT_UNTIL_MASKED(addr, mask, cond, expected, memorder)
>            \
> do {
>            \
>         RTE_BUILD_BUG_ON(memorder != __ATOMIC_ACQUIRE &&
>            \
>                 memorder != __ATOMIC_RELAXED);
>            \
>         typeof(*(addr)) expected_value = expected;
>            \
>         while (!((__atomic_load_n(addr, memorder) & (mask)) cond
> expected_value)) \
>                 rte_pause();
>            \
> } while (0)
> 
> 
> Comments below.
> 
> >
> > Signed-off-by: Feifei Wang <feifei.wang2@arm.com>
> > Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
> > Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> > ---
> >  lib/eal/arm/include/rte_pause_64.h  | 202
> > +++++++++++++++++-----------  lib/eal/include/generic/rte_pause.h |
> > 28 ++++
> >  2 files changed, 154 insertions(+), 76 deletions(-)
> >
> > diff --git a/lib/eal/arm/include/rte_pause_64.h
> > b/lib/eal/arm/include/rte_pause_64.h
> > index e87d10b8cc..783c6aae87 100644
> > --- a/lib/eal/arm/include/rte_pause_64.h
> > +++ b/lib/eal/arm/include/rte_pause_64.h
> 
> [snip]
> 
> > +/*
> > + * Atomic exclusive load from addr, it returns the 64-bit content of
> > + * *addr while making it 'monitored', when it is written by someone
> > + * else, the 'monitored' state is cleared and an event is generated
> > + * implicitly to exit WFE.
> > + */
> > +#define __RTE_ARM_LOAD_EXC_64(src, dst, memorder) {      \
> > +       if (memorder == __ATOMIC_RELAXED) {              \
> > +               asm volatile("ldxr %x[tmp], [%x[addr]]"  \
> > +                       : [tmp] "=&r" (dst)              \
> > +                       : [addr] "r" (src)               \
> > +                       : "memory");                     \
> > +       } else {                                         \
> > +               asm volatile("ldaxr %x[tmp], [%x[addr]]" \
> > +                       : [tmp] "=&r" (dst)              \
> > +                       : [addr] "r" (src)               \
> > +                       : "memory");                     \
> > +       } }
> > +
> > +/*
> > + * Atomic exclusive load from addr, it returns the 128-bit content of
> > + * *addr while making it 'monitored', when it is written by someone
> > + * else, the 'monitored' state is cleared and an event is generated
> > + * implicitly to exit WFE.
> > + */
> > +#define __RTE_ARM_LOAD_EXC_128(src, dst, memorder) {                    \
> > +       volatile rte_int128_t *dst_128 = (volatile rte_int128_t
> > +*)&dst; \
> 
> dst needs some () protection => &(dst)
[Feifei] dst is from internal defined variable 'value' in
'rte_wait_event', we can ensure it is just a simple variable,
so we do not need to add '()'
> Is volatile necessary?
[Feifei] This volatile is for the case that our parameter 'addr' is volatile.
And we use 'typeof' for 'value', so the value will be defined as 'volatile':
+ typeof(*(addr)) value
So if there is now 'volatile' here, compiler will be report warning:
'volatile is  discard' .                                        
> 
> 
> > +       if (memorder == __ATOMIC_RELAXED) {                             \
> > +               asm volatile("ldxp %x[tmp0], %x[tmp1], [%x[addr]]"      \
> > +                       : [tmp0] "=&r" (dst_128->val[0]),               \
> > +                         [tmp1] "=&r" (dst_128->val[1])                \
> > +                       : [addr] "r" (src)                              \
> > +                       : "memory");                                    \
> > +       } else {                                                        \
> > +               asm volatile("ldaxp %x[tmp0], %x[tmp1], [%x[addr]]"     \
> > +                       : [tmp0] "=&r" (dst_128->val[0]),               \
> > +                         [tmp1] "=&r" (dst_128->val[1])                \
> > +                       : [addr] "r" (src)                              \
> > +                       : "memory");                                    \
> > +       } }                                                             \
> > +
> > +#define __RTE_ARM_LOAD_EXC(src, dst, memorder, size) {          \
> > +       RTE_BUILD_BUG_ON(size != 16 && size != 32 && size != 64 \
> > +                               && size != 128);                \
> 
> Indent should be one tab (idem in other places of this patch).
> Double tab is when we have line continuation in tests.
[Feifei] Ok.
> 
> 
> > +       if (size == 16)                                         \
> > +               __RTE_ARM_LOAD_EXC_16(src, dst, memorder)       \
> > +       else if (size == 32)                                    \
> > +               __RTE_ARM_LOAD_EXC_32(src, dst, memorder)       \
> > +       else if (size == 64)                                    \
> > +               __RTE_ARM_LOAD_EXC_64(src, dst, memorder)       \
> > +       else if (size == 128)                                   \
> > +               __RTE_ARM_LOAD_EXC_128(src, dst, memorder)      \
> > +}
> > +
> 
> [snip]
> 
> > -#undef __LOAD_EXC_64
> >
> > -#undef __SEVL
> > -#undef __WFE
> > +#define rte_wait_event(addr, mask, cond, expected, memorder)              \
> > +do {                                                                      \
> > +       RTE_BUILD_BUG_ON(!__builtin_constant_p(memorder));                \
> 
> Is this check on memorder being constant necessary?
> We have a build bug on, right after, would it not catch non constant cases?
I think this can firstly check whether memorder has been assigned or NULL.
> 
> > +       RTE_BUILD_BUG_ON(memorder != __ATOMIC_ACQUIRE &&
> \
> > +                               memorder != __ATOMIC_RELAXED);            \
> > +       const uint32_t size = sizeof(*(addr)) << 3;                       \
> > +       typeof(*(addr)) expected_value = (expected);                      \
> 
> No need for () around expected.
[Feifei] expected and addr are macro arguments, and we cannot
know what form users will define them, so in order to avoid un-predicted
side-effects with operands associativity, It is necessary to add them.
Please see the discussion with Konstantin:
http://patches.dpdk.org/project/dpdk/patch/20211020084523.1309177-2-feifei.wang2@arm.com/
> 
> 
> > +       typeof(*(addr)) value;                                            \
> > +       __RTE_ARM_LOAD_EXC((addr), value, memorder, size)                 \
> 
> No need for () around addr.
> 
> 
> > +       if ((value & (mask)) cond expected_value) {                       \
> > +               __RTE_ARM_SEVL()                                          \
> > +               do {                                                      \
> > +                       __RTE_ARM_WFE()                                   \
> > +                       __RTE_ARM_LOAD_EXC((addr), value, memorder,
> > + size) \
> 
> Idem.
> 
> 
> > +               } while ((value & (mask)) cond expected_value);           \
> > +       }                                                                 \
> > +} while (0)
> >
> >  #endif
> >
> > diff --git a/lib/eal/include/generic/rte_pause.h
> > b/lib/eal/include/generic/rte_pause.h
> > index 668ee4a184..d0c5b5a415 100644
> > --- a/lib/eal/include/generic/rte_pause.h
> > +++ b/lib/eal/include/generic/rte_pause.h
> > @@ -111,6 +111,34 @@ rte_wait_until_equal_64(volatile uint64_t *addr,
> uint64_t expected,
> >         while (__atomic_load_n(addr, memorder) != expected)
> >                 rte_pause();
> >  }
> 
> With this patch, ARM header goes though a conversion of assert() to
> compilation checks (build bug on).
> I don't see a reason not to do the same in generic header.
> 
> As a result of this conversion, #include <assert.h> then can be removed.
> Though it triggers build failure on following files (afaics) who were implictly
> relying on this inclusion:
> drivers/net/ark/ark_ddm.c
> drivers/net/ark/ark_udm.c
> drivers/net/ice/ice_fdir_filter.c
> drivers/net/ionic/ionic_rxtx.c
> drivers/net/mlx4/mlx4_txq.c
[Feifei]You are right, and  we can put this change in another patch series. 
> 
> 
> --
> David Marchand