From: Ruifeng Wang <Ruifeng.Wang@arm.com>
To: Jerin Jacob <jerinjacobk@gmail.com>
Cc: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>,
"dev@dpdk.org" <dev@dpdk.org>,
"jerinj@marvell.com" <jerinj@marvell.com>,
"hemant.agrawal@nxp.com" <hemant.agrawal@nxp.com>,
"Ajit Khaparde (ajit.khaparde@broadcom.com)"
<ajit.khaparde@broadcom.com>,
"igorch@amazon.com" <igorch@amazon.com>,
"thomas@monjalon.net" <thomas@monjalon.net>,
"viacheslavo@mellanox.com" <viacheslavo@mellanox.com>,
"arybchenko@solarflare.com" <arybchenko@solarflare.com>,
nd <nd@arm.com>, nd <nd@arm.com>
Subject: Re: [dpdk-dev] [RFC] eal: adjust barriers for IO on Armv8-a
Date: Tue, 12 May 2020 08:02:29 +0000 [thread overview]
Message-ID: <HE1PR0801MB202574D9E0E14ADB480460099EBE0@HE1PR0801MB2025.eurprd08.prod.outlook.com> (raw)
In-Reply-To: <CALBAE1PAPyfXc_4NogVxkw9HFc3hkei=P=917gCs56G2utH-+g@mail.gmail.com>
> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Tuesday, May 12, 2020 2:42 PM
> To: Ruifeng Wang <Ruifeng.Wang@arm.com>
> Cc: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>;
> dev@dpdk.org; jerinj@marvell.com; hemant.agrawal@nxp.com; Ajit
> Khaparde (ajit.khaparde@broadcom.com) <ajit.khaparde@broadcom.com>;
> igorch@amazon.com; thomas@monjalon.net; viacheslavo@mellanox.com;
> arybchenko@solarflare.com; nd <nd@arm.com>
> Subject: Re: [dpdk-dev] [RFC] eal: adjust barriers for IO on Armv8-a
>
> On Tue, May 12, 2020 at 11:48 AM Ruifeng Wang <Ruifeng.Wang@arm.com>
> wrote:
> >
> >
> > > -----Original Message-----
> > > From: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> > > Sent: Tuesday, May 12, 2020 2:07 AM
> > > To: dev@dpdk.org; jerinj@marvell.com; hemant.agrawal@nxp.com; Ajit
> > > Khaparde (ajit.khaparde@broadcom.com)
> <ajit.khaparde@broadcom.com>;
> > > igorch@amazon.com; thomas@monjalon.net;
> viacheslavo@mellanox.com;
> > > arybchenko@solarflare.com; Honnappa Nagarahalli
> > > <Honnappa.Nagarahalli@arm.com>
> > > Cc: Ruifeng Wang <Ruifeng.Wang@arm.com>; nd <nd@arm.com>
> > > Subject: [RFC] eal: adjust barriers for IO on Armv8-a
> > >
> > > Change the barrier APIs for IO to reflect that Armv8-a is
> > > other-multi-copy atomicity memory model.
> > >
> > > Armv8-a memory model has been strengthened to require
> > > other-multi-copy atomicity. This property requires memory accesses
> > > from an observer to become visible to all other observers
> > > simultaneously [3]. This means
> > >
> > > a) A write arriving at an endpoint shared between multiple CPUs is
> > > visible to all CPUs
> > > b) A write that is visible to all CPUs is also visible to all other
> > > observers in the shareability domain
> > >
> > > This allows for using cheaper DMB instructions in the place of DSB
> > > for devices that are visible to all CPUs (i.e. devices that DPDK caters to).
> > >
> > > Please refer to [1], [2] and [3] for more information.
> > >
> > > [1]
> > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/c
> > > ommit/?i d=22ec71615d824f4f11d38d0e55a88d8956b7e45f
> > > [2] https://www.youtube.com/watch?v=i6DayghhA8Q
> > > [3] https://www.cl.cam.ac.uk/~pes20/armv8-mca/
> > >
> > > Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> > > ---
> > > lib/librte_eal/arm/include/rte_atomic_64.h | 10 +++++-----
> > > 1 file changed, 5 insertions(+), 5 deletions(-)
> > >
> > > diff --git a/lib/librte_eal/arm/include/rte_atomic_64.h
> > > b/lib/librte_eal/arm/include/rte_atomic_64.h
> > > index 7b7099cdc..e406411bb 100644
> > > --- a/lib/librte_eal/arm/include/rte_atomic_64.h
> > > +++ b/lib/librte_eal/arm/include/rte_atomic_64.h
> > > @@ -19,11 +19,11 @@ extern "C" {
> > > #include <rte_compat.h>
> > > #include <rte_debug.h>
> > >
> > > -#define rte_mb() asm volatile("dsb sy" : : : "memory")
> > > +#define rte_mb() asm volatile("dmb osh" : : : "memory")
> > >
> > > -#define rte_wmb() asm volatile("dsb st" : : : "memory")
> > > +#define rte_wmb() asm volatile("dmb oshst" : : : "memory")
> > >
> > > -#define rte_rmb() asm volatile("dsb ld" : : : "memory")
> > > +#define rte_rmb() asm volatile("dmb oshld" : : : "memory")
> > >
> > > #define rte_smp_mb() asm volatile("dmb ish" : : : "memory")
> > >
> > > @@ -37,9 +37,9 @@ extern "C" {
> > >
> > > #define rte_io_rmb() rte_rmb()
> > >
> > > -#define rte_cio_wmb() asm volatile("dmb oshst" : : : "memory")
> > > +#define rte_cio_wmb() rte_wmb()
> > >
> > > -#define rte_cio_rmb() asm volatile("dmb oshld" : : : "memory")
> > > +#define rte_cio_rmb() rte_rmb()
> > >
> > > /*------------------------ 128 bit atomic operations
> > > -------------------------*/
> > >
> > > --
> > > 2.17.1
> >
> > This change showed about 7% performance gain in testpmd single core
> NDR test.
>
> I am trying to understand this patch wrt DPDK current usage model?
>
> 1) Is performance improvement due to the fact that the PMD that you are
> using it for testing suppose to use existing rte_cio_* but it was using
> rte_[rw]mb?
This is part of the reason. There are also cases where rte_io_* was used and can be relaxed.
Such as: http://patches.dpdk.org/patch/68162/
> 2) In my understanding :
> a) CPU to CPU barrier requirements are addressed by rte_smp_*
> b) CPU to DMA/Device barrier requirements are addressed by rte_cio_*
> c) CPU to ANY(CPU or Device) are addressed by rte_[rw]mb
>
> If (c) is true then we are violating the DPDK spec with change. Right?
Developers are still required to use correct barrier APIs for different use cases.
I think this change mitigates performance penalty when non optimal barrier is used.
> This change will not be required if fastpath (CPU to Device) is using rte_cio_*.
> Right?
See 1). Correct usage of rte_cio_* is not the whole.
For some other use cases, such as barrier between accesses of different memory types, we can also use lighter barrier 'dmb'.
>
>
>
> > Tested-by: Ruifeng Wang <ruifeng.wang@arm.com>
> >
next prev parent reply other threads:[~2020-05-12 8:02 UTC|newest]
Thread overview: 70+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-02-13 12:38 [dpdk-dev] [PATCH RFC v1 0/6] barrier fix and optimization for mlx5 on aarch64 Gavin Hu
2020-02-13 12:38 ` [dpdk-dev] [PATCH RFC v1 1/6] net/mlx5: relax the barrier for UAR write Gavin Hu
2020-02-13 12:38 ` [dpdk-dev] [PATCH RFC v1 2/6] net/mlx5: use cio barrier before the BF WQE Gavin Hu
2020-02-13 12:38 ` [dpdk-dev] [PATCH RFC v1 3/6] net/mlx5: add missing barrier Gavin Hu
2020-02-13 12:38 ` [dpdk-dev] [PATCH RFC v1 4/6] net/mlx5: add descriptive comment for a barrier Gavin Hu
2020-02-13 12:38 ` [dpdk-dev] [PATCH RFC v1 5/6] net/mlx5: non-cacheable mapping defaulted for aarch64 Gavin Hu
2020-02-13 12:38 ` [dpdk-dev] [PATCH RFC v1 6/6] net/mlx5: relaxed ordering for multi-packet RQ buffer refcnt Gavin Hu
2020-04-10 16:41 ` [dpdk-dev] [PATCH RFC v2 0/7] introduce new barrier class and use it for mlx5 PMD Gavin Hu
2020-04-10 17:20 ` Andrew Rybchenko
2020-04-11 3:46 ` Gavin Hu
2020-04-13 9:51 ` Andrew Rybchenko
2020-04-13 16:46 ` Gavin Hu
2020-05-11 18:06 ` [dpdk-dev] [RFC] eal: adjust barriers for IO on Armv8-a Honnappa Nagarahalli
2020-05-12 6:18 ` Ruifeng Wang
2020-05-12 6:42 ` Jerin Jacob
2020-05-12 8:02 ` Ruifeng Wang [this message]
2020-05-12 8:28 ` Jerin Jacob
2020-05-12 21:44 ` Honnappa Nagarahalli
2020-05-13 14:49 ` Jerin Jacob
2020-05-14 1:02 ` Honnappa Nagarahalli
2020-06-27 19:12 ` [dpdk-dev] [PATCH v2] " Honnappa Nagarahalli
2020-06-27 19:25 ` Honnappa Nagarahalli
2020-06-30 5:13 ` Jerin Jacob
2020-07-03 18:57 ` [dpdk-dev] [PATCH v3 1/3] " Honnappa Nagarahalli
2020-07-03 18:57 ` [dpdk-dev] [PATCH v3 2/3] doc: update armv8-a IO barrier changes Honnappa Nagarahalli
2020-07-05 0:57 ` Jerin Jacob
2020-07-03 18:57 ` [dpdk-dev] [PATCH v3 3/3] doc: update deprecation of CIO barrier APIs Honnappa Nagarahalli
2020-07-05 0:57 ` Jerin Jacob
2020-07-07 20:19 ` Ajit Khaparde
2020-07-08 11:05 ` Ananyev, Konstantin
2020-07-06 23:43 ` [dpdk-dev] [PATCH v4 1/3] eal: adjust barriers for IO on Armv8-a Honnappa Nagarahalli
2020-07-06 23:43 ` [dpdk-dev] [PATCH v4 2/3] doc: update armv8-a IO barrier changes Honnappa Nagarahalli
2020-07-07 8:36 ` David Marchand
2020-07-07 18:37 ` Honnappa Nagarahalli
2020-07-06 23:43 ` [dpdk-dev] [PATCH v4 3/3] doc: update deprecation of CIO barrier APIs Honnappa Nagarahalli
2020-07-07 8:39 ` David Marchand
2020-07-07 20:14 ` David Christensen
2020-07-08 11:49 ` David Marchand
2020-04-10 16:41 ` [dpdk-dev] [PATCH RFC v2 1/7] eal: introduce new class of barriers for DMA use cases Gavin Hu
2020-04-10 16:41 ` [dpdk-dev] [PATCH RFC v2 2/7] net/mlx5: dmb for immediate doorbell ring on aarch64 Gavin Hu
2020-04-10 16:41 ` [dpdk-dev] [PATCH RFC v2 3/7] net/mlx5: relax barrier to order UAR writes " Gavin Hu
2020-04-10 16:41 ` [dpdk-dev] [PATCH RFC v2 4/7] net/mlx5: relax barrier for aarch64 Gavin Hu
2020-04-10 16:41 ` [dpdk-dev] [PATCH RFC v2 5/7] net/mlx5: add descriptive comment for a barrier Gavin Hu
2020-04-10 16:41 ` [dpdk-dev] [PATCH RFC v2 6/7] net/mlx5: relax ordering for multi-packet RQ buffer refcnt Gavin Hu
2020-06-23 8:26 ` [dpdk-dev] [PATCH v3] net/mlx5: relaxed " Phil Yang
2020-07-13 3:02 ` Phil Yang
2020-07-20 23:21 ` Alexander Kozyrev
2020-07-21 1:55 ` Phil Yang
2020-07-21 3:58 ` Alexander Kozyrev
2020-07-21 4:03 ` Honnappa Nagarahalli
2020-07-21 4:11 ` Alexander Kozyrev
2020-07-22 12:06 ` Phil Yang
2020-07-23 4:47 ` Honnappa Nagarahalli
2020-07-23 6:11 ` Phil Yang
2020-07-23 16:53 ` Alexander Kozyrev
2020-07-27 14:52 ` Phil Yang
2020-08-06 2:43 ` Alexander Kozyrev
2020-08-11 5:20 ` Honnappa Nagarahalli
2020-09-02 21:52 ` Alexander Kozyrev
2020-09-03 2:55 ` Phil Yang
2020-09-09 13:29 ` Alexander Kozyrev
2020-09-10 1:34 ` Honnappa Nagarahalli
2020-09-03 2:53 ` [dpdk-dev] [PATCH v4] " Phil Yang
2020-09-10 1:30 ` Honnappa Nagarahalli
2020-09-10 1:36 ` Alexander Kozyrev
2020-09-29 15:22 ` Phil Yang
2020-09-30 12:44 ` Slava Ovsiienko
2020-09-30 12:52 ` Raslan Darawsheh
2020-09-30 13:57 ` Raslan Darawsheh
2020-04-10 16:41 ` [dpdk-dev] [PATCH RFC v2 7/7] doc: clarify one configuration in mlx5 guide Gavin Hu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=HE1PR0801MB202574D9E0E14ADB480460099EBE0@HE1PR0801MB2025.eurprd08.prod.outlook.com \
--to=ruifeng.wang@arm.com \
--cc=Honnappa.Nagarahalli@arm.com \
--cc=ajit.khaparde@broadcom.com \
--cc=arybchenko@solarflare.com \
--cc=dev@dpdk.org \
--cc=hemant.agrawal@nxp.com \
--cc=igorch@amazon.com \
--cc=jerinj@marvell.com \
--cc=jerinjacobk@gmail.com \
--cc=nd@arm.com \
--cc=thomas@monjalon.net \
--cc=viacheslavo@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).