From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from dpdk.org (dpdk.org [92.243.14.124])
by dpdk.space (Postfix) with ESMTP id 78319A00E6
for <public@inbox.dpdk.org>; Tue, 19 Mar 2019 20:42:24 +0100 (CET)
Received: from [92.243.14.124] (localhost [127.0.0.1])
by dpdk.org (Postfix) with ESMTP id 3141A1B19;
Tue, 19 Mar 2019 20:42:24 +0100 (CET)
Received: from EUR01-DB5-obe.outbound.protection.outlook.com
(mail-eopbgr150051.outbound.protection.outlook.com [40.107.15.51])
by dpdk.org (Postfix) with ESMTP id 6D84311A4;
Tue, 19 Mar 2019 20:42:23 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com;
s=selector1;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=OIWS9ReyPcqQpH20C9K6CSSxhbbt7Mw9ZnP3R8BLgSs=;
b=qPNaqaKz0qecoLRT5g6Jdm87PqLSRhIsJq2332vnlkmljEYD/YNVb4UMbraNDdx+kux6LCBRkMje2Xc8h6t8xeOg6fiCIzD9Gj15ACypPyu/mOxbnAnT4MY8Z7F2ZaLBfXEl4Z8KkMcwvdwKo3uNmhWeUOEqR4+C//lwzPFBO4w=
Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com (52.133.45.150) by
AM0SPR01MB010.eurprd05.prod.outlook.com (52.133.36.143) with Microsoft SMTP
Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
15.20.1709.14; Tue, 19 Mar 2019 19:42:22 +0000
Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com
([fe80::84f3:7e92:7a51:1003]) by AM0PR0502MB3795.eurprd05.prod.outlook.com
([fe80::84f3:7e92:7a51:1003%2]) with mapi id 15.20.1709.015; Tue, 19 Mar 2019
19:42:22 +0000
From: Shahaf Shuler <shahafs@mellanox.com>
To: Thomas Monjalon <thomas@monjalon.net>, Dekel Peled <dekelp@mellanox.com>,
Chao Zhu <chaozhu@linux.vnet.ibm.com>
CC: Yongseok Koh <yskoh@mellanox.com>, "dev@dpdk.org" <dev@dpdk.org>, Ori Kam
<orika@mellanox.com>, "stable@dpdk.org" <stable@dpdk.org>
Thread-Topic: [PATCH] eal/ppc: remove fix of memory barrier for IBM POWER
Thread-Index: AQHU3js5UXrzwqXJNk26I7vX4kWVFKYSzYSAgACJ6eA=
Date: Tue, 19 Mar 2019 19:42:21 +0000
Message-ID:
<AM0PR0502MB37957B54AFF7FF8C0629CDFCC3400@AM0PR0502MB3795.eurprd05.prod.outlook.com>
References: <1552913893-43407-1-git-send-email-dekelp@mellanox.com>
<001d01d4de03$378f18a0$a6ad49e0$@linux.vnet.ibm.com>
<VI1PR05MB42242D09494D874AC58CFC88B6400@VI1PR05MB4224.eurprd05.prod.outlook.com>
<1789153.zrlSK8XYcq@xps>
In-Reply-To: <1789153.zrlSK8XYcq@xps>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
authentication-results: spf=none (sender IP is )
smtp.mailfrom=shahafs@mellanox.com;
x-originating-ip: [31.154.10.105]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: d2c1244e-a12c-43a4-fd9c-08d6aca30152
x-ms-office365-filtering-ht: Tenant
x-microsoft-antispam: BCL:0; PCL:0;
RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020);
SRVR:AM0SPR01MB010;
x-ms-traffictypediagnostic: AM0SPR01MB010:
x-ms-exchange-purlcount: 1
x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr
x-microsoft-antispam-prvs: <AM0SPR01MB010E17410F8285840318236C3400@AM0SPR01MB010.eurprd05.prod.outlook.com>
x-forefront-prvs: 0981815F2F
x-forefront-antispam-report: SFV:NSPM;
SFS:(10009020)(136003)(396003)(376002)(346002)(39860400002)(366004)(189003)(199004)(13464003)(6436002)(105586002)(106356001)(25786009)(229853002)(74316002)(9686003)(7736002)(14444005)(81166006)(256004)(6306002)(8676002)(81156014)(55016002)(305945005)(66066001)(4326008)(486006)(33656002)(14454004)(93886005)(966005)(6246003)(478600001)(99286004)(71190400001)(3846002)(6116002)(71200400001)(97736004)(52536014)(8936002)(316002)(45080400002)(68736007)(186003)(54906003)(26005)(6506007)(2906002)(53546011)(5660300002)(110136005)(102836004)(76176011)(11346002)(446003)(53936002)(7696005)(476003)(86362001);
DIR:OUT; SFP:1101; SCL:1; SRVR:AM0SPR01MB010;
H:AM0PR0502MB3795.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en;
PTR:InfoNoRecords; MX:1; A:1;
received-spf: None (protection.outlook.com: mellanox.com does not designate
permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: lCcc6dAxVTaoovXH3XsocQvTPDuZtwZHglKEPRHQhTkxTkDOEPCLZbZOfK/mAvuCRCZ1+Ejl3zK1pt9QwrXcexvNkk93Df2yan0sAB3QSCrRj/Rc3cEdGgYHlWPr4B1otP+Ot77LgxDbBnjfoUHeogNrlj1FRPn8Xtp9J8ewIVegR7f+7SUpJla77NzWN6YzLT2TUFGM+Rke+ttPmJJ/lwZZaEf0X/kdI0RigoCaiYj3Qvr4Z0BqNSQTrjQ1YwXyjITugvty6mfcdEmlbEQjYsqoScVN/asGbDMEmPDUgH1NgByF+6BOTiwwdZsdWozL30k4Y3RtKb2m18SNqeER5KlvliXnCIezhaaDyoBd916yz4+j9zpG55O3iNjrmUhsdd3UsqJaQ1QcVD23mAdAq9168vy/hQqCWP2gQNDR270=
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: Mellanox.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d2c1244e-a12c-43a4-fd9c-08d6aca30152
X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Mar 2019 19:42:21.9915 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0SPR01MB010
Subject: Re: [dpdk-dev] [PATCH] eal/ppc: remove fix of memory barrier for
IBM POWER
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>
Message-ID: <20190319194221.Zzey_1Mkdhvv3N-xaaD9g_MZCM8ZfEE-9iLMugXRWgY@z>
Tuesday, March 19, 2019 1:15 PM, Thomas Monjalon:
> Subject: Re: [PATCH] eal/ppc: remove fix of memory barrier for IBM POWER
>=20
> Guys, please let's avoid top-post.
>=20
> You are both not replying to each other:
>=20
> 1/ Dekel mentioned the IBM doc but Chao did not argue about the lack of I=
O
> protection with lwsync.
> We assume that rte_mb should protect any access including IO.
>=20
> 2/ Chao asked about the semantic of the barrier used in mlx5 code, but De=
kel
> did not reply about the semantic: are we protecting IO or general memory
> access?
In mlx5 code we want to sync between two different writes:
1. write to system memory (RAM)
2. write to MMIO memory (device)
We need #1 to be visible on host memory before #2 is committed to NIC.
We want to have a single type of barrier which will translate to the correc=
t assembly based on the system arch, and in addition we want it light-weigh=
t as possible.
So far, when not running on power, we used the rte_wmb for that. On x86 and=
ARM systems it provided the needed guarantees. =20
It is also mentioned in the barrier doxygen on ARM arch:
"
Write memory barrier. =20
=20
Guarantees that the STORE operations generated before the barrier
occur before the STORE operations generated after.
"
It doesn't restrict to store to system memory only.=20
w/ power is on somewhat different and in fact rte_mb is required. It obviou=
sly miss the point of those barrier if we will need to use a different barr=
ier based on the system arch.=20
We need to align the definition of the different barriers in DPDK:
1. need a clear documentation of each. this should be global and not part o=
f the specific implementation on each arch.=20
2. either modify ppc rte_wmb to match ARM and x86 ones or to define a new t=
ype of barrier which will sync between both I/O and stores to systems memor=
y.=20
>=20
>=20
> 19/03/2019 11:05, Dekel Peled:
> > Hi,
> >
> > For ppc, rte_io_mb() is defined as rte_mb(), which is defined as asm sy=
nc.
> > According to comments in arch/ppc_64/rte_atomic.h, rte_wmb() and
> rte_rmb() are the same as rte_mb(), for store and load respectively.
> > My patch propose to define rte_wmb() and rte_rmb() as asm sync, like
> rte_mb(), since using lwsync is incorrect for them.
> >
> > Regards,
> > Dekel
> >
> > > -----Original Message-----
> > > From: Chao Zhu <chaozhu@linux.vnet.ibm.com>
> > > Sent: Tuesday, March 19, 2019 5:24 AM
> > > To: Dekel Peled <dekelp@mellanox.com>
> > > Cc: Yongseok Koh <yskoh@mellanox.com>; Shahaf Shuler
> > > <shahafs@mellanox.com>; dev@dpdk.org; Ori Kam
> <orika@mellanox.com>;
> > > Thomas Monjalon <thomas@monjalon.net>; stable@dpdk.org
> > > Subject: RE: [PATCH] eal/ppc: remove fix of memory barrier for IBM
> > > POWER
> > >
> > > Dekel=A3=AC
> > >
> > > To control the memory order for device memory, I think you should
> > > use
> > > rte_io_mb() instead of rte_mb(). This will generate correct result.
> > > rte_wmb() is used for system memory.
> > >
> > > > -----Original Message-----
> > > > From: Dekel Peled <dekelp@mellanox.com>
> > > > Sent: Monday, March 18, 2019 8:58 PM
> > > > To: chaozhu@linux.vnet.ibm.com
> > > > Cc: yskoh@mellanox.com; shahafs@mellanox.com; dev@dpdk.org;
> > > > orika@mellanox.com; thomas@monjalon.net; dekelp@mellanox.com;
> > > > stable@dpdk.org
> > > > Subject: [PATCH] eal/ppc: remove fix of memory barrier for IBM
> > > > POWER
> > > >
> > > > From previous patch description: "to improve performance on PPC64,
> > > > use light weight sync instruction instead of sync instruction."
> > > >
> > > > Excerpt from IBM doc [1], section "Memory barrier instructions":
> > > > "The second form of the sync instruction is light-weight sync, or l=
wsync.
> > > > This form is used to control ordering for storage accesses to
> > > > system memory only. It does not create a memory barrier for
> > > > accesses to device
> > > memory."
> > > >
> > > > This patch removes the use of lwsync, so calls to rte_wmb() and
> > > > rte_rmb() will provide correct memory barrier to ensure order of
> > > > accesses to system memory and device memory.
> > > >
> > > > [1]
> > > >
> > >
> https://eur03.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fww
> > > w
> > > .
> > > >
> > >
> ibm.com%2Fdeveloperworks%2Fsystems%2Farticles%2Fpowerpc.html&
> > > ;data=3D
> > > >
> > >
> 02%7C01%7Cdekelp%40mellanox.com%7C381426b6b9d042f776fa08d6ac1a5d
> > > c5%7Ca
> > > >
> > >
> 652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636885626593364016&am
> > > p;sdata
> > > >
> > >
> =3DwFYTcFX2A%2BMdtQMgtojTAtUOzqds7U5pypNS%2F2SoXUM%3D&re
> > > served=3D0
> > > >
> > > > Fixes: d23a6bd04d72 ("eal/ppc: fix memory barrier for IBM POWER")
> > > > Cc: stable@dpdk.org
> > > >
> > > > Signed-off-by: Dekel Peled <dekelp@mellanox.com>
> > > > ---
> > > > lib/librte_eal/common/include/arch/ppc_64/rte_atomic.h | 8
> > > > --------
> > > > 1 file changed, 8 deletions(-)
> > > >
> > > > diff --git
> > > > a/lib/librte_eal/common/include/arch/ppc_64/rte_atomic.h
> > > > b/lib/librte_eal/common/include/arch/ppc_64/rte_atomic.h
> > > > index ce38350..797381c 100644
> > > > --- a/lib/librte_eal/common/include/arch/ppc_64/rte_atomic.h
> > > > +++ b/lib/librte_eal/common/include/arch/ppc_64/rte_atomic.h
> > > > @@ -63,11 +63,7 @@
> > > > * Guarantees that the STORE operations generated before the barri=
er
> > > > * occur before the STORE operations generated after.
> > > > */
> > > > -#ifdef RTE_ARCH_64
> > > > -#define rte_wmb() asm volatile("lwsync" : : : "memory")
> > > > -#else
> > > > #define rte_wmb() asm volatile("sync" : : : "memory")
> > > > -#endif
> > > >
> > > > /**
> > > > * Read memory barrier.
> > > > @@ -75,11 +71,7 @@
> > > > * Guarantees that the LOAD operations generated before the barrie=
r
> > > > * occur before the LOAD operations generated after.
> > > > */
> > > > -#ifdef RTE_ARCH_64
> > > > -#define rte_rmb() asm volatile("lwsync" : : : "memory")
> > > > -#else
> > > > #define rte_rmb() asm volatile("sync" : : : "memory")
> > > > -#endif
> > > >
> > > > #define rte_smp_mb() rte_mb()
>=20
>=20