DPDK patches and discussions
 help / color / mirror / Atom feed
From: Shahaf Shuler <shahafs@mellanox.com>
To: Thomas Monjalon <thomas@monjalon.net>,
	Dekel Peled <dekelp@mellanox.com>,
	 Chao Zhu <chaozhu@linux.vnet.ibm.com>
Cc: Yongseok Koh <yskoh@mellanox.com>, "dev@dpdk.org" <dev@dpdk.org>,
	Ori Kam <orika@mellanox.com>, "stable@dpdk.org" <stable@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH] eal/ppc: remove fix of memory barrier for IBM POWER
Date: Tue, 19 Mar 2019 19:42:21 +0000	[thread overview]
Message-ID: <AM0PR0502MB37957B54AFF7FF8C0629CDFCC3400@AM0PR0502MB3795.eurprd05.prod.outlook.com> (raw)
Message-ID: <20190319194221.Zzey_1Mkdhvv3N-xaaD9g_MZCM8ZfEE-9iLMugXRWgY@z> (raw)
In-Reply-To: <1789153.zrlSK8XYcq@xps>

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="UTF-8", Size: 6291 bytes --]

Tuesday, March 19, 2019 1:15 PM, Thomas Monjalon:
> Subject: Re: [PATCH] eal/ppc: remove fix of memory barrier for IBM POWER
> 
> Guys, please let's avoid top-post.
> 
> You are both not replying to each other:
> 
> 1/ Dekel mentioned the IBM doc but Chao did not argue about the lack of IO
> protection with lwsync.
> We assume that rte_mb should protect any access including IO.
> 
> 2/ Chao asked about the semantic of the barrier used in mlx5 code, but Dekel
> did not reply about the semantic: are we protecting IO or general memory
> access?

In mlx5 code we want to sync between two different writes:
1. write to system memory (RAM)
2. write to MMIO memory (device)

We need #1 to be visible on host memory before #2 is committed to NIC.
We want to have a single type of barrier which will translate to the correct assembly based on the system arch, and in addition we want it light-weight as possible.

So far, when not running on power, we used the rte_wmb for that. On x86 and ARM systems it provided the needed guarantees.  
It is also mentioned in the barrier doxygen on ARM arch:
"
Write memory barrier.                                            
                                                                 
Guarantees that the STORE operations generated before the barrier
occur before the STORE operations generated after.
"

It doesn't restrict to store to system memory only. 
w/ power is on somewhat different and in fact rte_mb is required. It obviously miss the point of those barrier if we will need to use a different barrier based on the system arch. 

We need to align the definition of the different barriers in DPDK:
1. need a clear documentation of each. this should be global and not part of the specific implementation on each arch. 
2. either modify ppc rte_wmb to match ARM and x86 ones or to define a new type of barrier which will sync between both I/O and stores to systems memory. 

> 
> 
> 19/03/2019 11:05, Dekel Peled:
> > Hi,
> >
> > For ppc, rte_io_mb() is defined as rte_mb(), which is defined as asm sync.
> > According to comments in arch/ppc_64/rte_atomic.h, rte_wmb() and
> rte_rmb() are the same as rte_mb(), for store and load respectively.
> > My patch propose to define rte_wmb() and rte_rmb() as asm sync, like
> rte_mb(), since using lwsync is incorrect for them.
> >
> > Regards,
> > Dekel
> >
> > > -----Original Message-----
> > > From: Chao Zhu <chaozhu@linux.vnet.ibm.com>
> > > Sent: Tuesday, March 19, 2019 5:24 AM
> > > To: Dekel Peled <dekelp@mellanox.com>
> > > Cc: Yongseok Koh <yskoh@mellanox.com>; Shahaf Shuler
> > > <shahafs@mellanox.com>; dev@dpdk.org; Ori Kam
> <orika@mellanox.com>;
> > > Thomas Monjalon <thomas@monjalon.net>; stable@dpdk.org
> > > Subject: RE: [PATCH] eal/ppc: remove fix of memory barrier for IBM
> > > POWER
> > >
> > > Dekel£¬
> > >
> > > To control the memory order for device memory, I think you should
> > > use
> > > rte_io_mb() instead of rte_mb(). This will generate correct result.
> > > rte_wmb() is used for system memory.
> > >
> > > > -----Original Message-----
> > > > From: Dekel Peled <dekelp@mellanox.com>
> > > > Sent: Monday, March 18, 2019 8:58 PM
> > > > To: chaozhu@linux.vnet.ibm.com
> > > > Cc: yskoh@mellanox.com; shahafs@mellanox.com; dev@dpdk.org;
> > > > orika@mellanox.com; thomas@monjalon.net; dekelp@mellanox.com;
> > > > stable@dpdk.org
> > > > Subject: [PATCH] eal/ppc: remove fix of memory barrier for IBM
> > > > POWER
> > > >
> > > > From previous patch description: "to improve performance on PPC64,
> > > > use light weight sync instruction instead of sync instruction."
> > > >
> > > > Excerpt from IBM doc [1], section "Memory barrier instructions":
> > > > "The second form of the sync instruction is light-weight sync, or lwsync.
> > > > This form is used to control ordering for storage accesses to
> > > > system memory only. It does not create a memory barrier for
> > > > accesses to device
> > > memory."
> > > >
> > > > This patch removes the use of lwsync, so calls to rte_wmb() and
> > > > rte_rmb() will provide correct memory barrier to ensure order of
> > > > accesses to system memory and device memory.
> > > >
> > > > [1]
> > > >
> > >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fww
> > > w
> > > .
> > > >
> > >
> ibm.com%2Fdeveloperworks%2Fsystems%2Farticles%2Fpowerpc.html&amp
> > > ;data=
> > > >
> > >
> 02%7C01%7Cdekelp%40mellanox.com%7C381426b6b9d042f776fa08d6ac1a5d
> > > c5%7Ca
> > > >
> > >
> 652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636885626593364016&am
> > > p;sdata
> > > >
> > >
> =wFYTcFX2A%2BMdtQMgtojTAtUOzqds7U5pypNS%2F2SoXUM%3D&amp;re
> > > served=0
> > > >
> > > > Fixes: d23a6bd04d72 ("eal/ppc: fix memory barrier for IBM POWER")
> > > > Cc: stable@dpdk.org
> > > >
> > > > Signed-off-by: Dekel Peled <dekelp@mellanox.com>
> > > > ---
> > > >  lib/librte_eal/common/include/arch/ppc_64/rte_atomic.h | 8
> > > > --------
> > > >  1 file changed, 8 deletions(-)
> > > >
> > > > diff --git
> > > > a/lib/librte_eal/common/include/arch/ppc_64/rte_atomic.h
> > > > b/lib/librte_eal/common/include/arch/ppc_64/rte_atomic.h
> > > > index ce38350..797381c 100644
> > > > --- a/lib/librte_eal/common/include/arch/ppc_64/rte_atomic.h
> > > > +++ b/lib/librte_eal/common/include/arch/ppc_64/rte_atomic.h
> > > > @@ -63,11 +63,7 @@
> > > >   * Guarantees that the STORE operations generated before the barrier
> > > >   * occur before the STORE operations generated after.
> > > >   */
> > > > -#ifdef RTE_ARCH_64
> > > > -#define	rte_wmb() asm volatile("lwsync" : : : "memory")
> > > > -#else
> > > >  #define	rte_wmb() asm volatile("sync" : : : "memory")
> > > > -#endif
> > > >
> > > >  /**
> > > >   * Read memory barrier.
> > > > @@ -75,11 +71,7 @@
> > > >   * Guarantees that the LOAD operations generated before the barrier
> > > >   * occur before the LOAD operations generated after.
> > > >   */
> > > > -#ifdef RTE_ARCH_64
> > > > -#define	rte_rmb() asm volatile("lwsync" : : : "memory")
> > > > -#else
> > > >  #define	rte_rmb() asm volatile("sync" : : : "memory")
> > > > -#endif
> > > >
> > > >  #define rte_smp_mb() rte_mb()
> 
> 


  parent reply	other threads:[~2019-03-19 19:42 UTC|newest]

Thread overview: 42+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-03-18 12:58 Dekel Peled
2019-03-18 12:58 ` Dekel Peled
2019-03-19  3:24 ` Chao Zhu
2019-03-19  3:24   ` Chao Zhu
2019-03-19 10:05   ` Dekel Peled
2019-03-19 10:05     ` Dekel Peled
2019-03-19 11:14     ` Thomas Monjalon
2019-03-19 11:14       ` Thomas Monjalon
2019-03-19 19:42       ` Shahaf Shuler [this message]
2019-03-19 19:42         ` Shahaf Shuler
2019-03-19 20:45         ` Thomas Monjalon
2019-03-19 20:45           ` Thomas Monjalon
2019-03-20 22:40           ` Pradeep Satyanarayana
2019-03-20 22:40             ` Pradeep Satyanarayana
2019-03-21  8:49             ` Shahaf Shuler
2019-03-21  8:49               ` Shahaf Shuler
2019-03-22  1:40               ` Pradeep Satyanarayana
2019-03-22  1:40                 ` Pradeep Satyanarayana
2019-03-22  8:49                 ` Thomas Monjalon
2019-03-22  8:49                   ` Thomas Monjalon
2019-03-22 15:30                   ` Pradeep Satyanarayana
2019-03-22 15:30                     ` Pradeep Satyanarayana
2019-03-22 17:51                     ` Thomas Monjalon
2019-03-22 17:51                       ` Thomas Monjalon
2019-03-22 22:57                       ` Pradeep Satyanarayana
2019-03-22 22:57                         ` Pradeep Satyanarayana
2019-03-24  6:37                         ` Shahaf Shuler
2019-03-24  6:37                           ` Shahaf Shuler
2019-03-24 17:37                           ` Pradeep Satyanarayana
2019-03-24 17:37                             ` Pradeep Satyanarayana
2019-03-26  9:15                             ` Dekel Peled
2019-03-26  9:15                               ` Dekel Peled
2019-03-27  9:19                               ` Thomas Monjalon
2019-03-27  9:19                                 ` Thomas Monjalon
2019-03-27 23:50                                 ` Pradeep Satyanarayana
2019-03-27 23:50                                   ` Pradeep Satyanarayana
     [not found]                                 ` <OF456B0ECC.006EF7E7-ON882583CA.00827A75-882583CA.0082F7BE@LocalDomain>
2019-03-28 17:51                                   ` Pradeep Satyanarayana
2019-03-28 17:51                                     ` Pradeep Satyanarayana
2019-03-28 17:56                                     ` Thomas Monjalon
2019-03-28 17:56                                       ` Thomas Monjalon
2019-03-28 22:50 ` [dpdk-dev] [dpdk-stable] " Thomas Monjalon
2019-03-28 22:50   ` Thomas Monjalon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=AM0PR0502MB37957B54AFF7FF8C0629CDFCC3400@AM0PR0502MB3795.eurprd05.prod.outlook.com \
    --to=shahafs@mellanox.com \
    --cc=chaozhu@linux.vnet.ibm.com \
    --cc=dekelp@mellanox.com \
    --cc=dev@dpdk.org \
    --cc=orika@mellanox.com \
    --cc=stable@dpdk.org \
    --cc=thomas@monjalon.net \
    --cc=yskoh@mellanox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).