From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <shahafs@mellanox.com> Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-eopbgr150055.outbound.protection.outlook.com [40.107.15.55]) by dpdk.org (Postfix) with ESMTP id A8F031B44F; Thu, 21 Mar 2019 09:49:43 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=m7jXR4ouH4YVAglXs3BMVj1gHMA/ja8KESMbBd7EGQc=; b=c100MMSm2rVio43654QJuAmLy+H7XUs8oheYo1ioYHuoFK31HG4vWCbwja1XXwkik4zE62krGcxgSZ9K0WjK7zTGX6YTTPFJP0YiLiQEkudh5r8e7Rfn5H3APJMhKbfMpvnTuQU3WmoJpNb6UFYFwU0iZoTdjpEvNF4cdSm17Zc= Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com (52.133.45.150) by AM0PR0502MB3761.eurprd05.prod.outlook.com (52.133.50.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1730.15; Thu, 21 Mar 2019 08:49:40 +0000 Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::84f3:7e92:7a51:1003]) by AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::84f3:7e92:7a51:1003%2]) with mapi id 15.20.1730.013; Thu, 21 Mar 2019 08:49:40 +0000 From: Shahaf Shuler <shahafs@mellanox.com> To: "pradeep@us.ibm.com" <pradeep@us.ibm.com>, Thomas Monjalon <thomas@monjalon.net> CC: Chao Zhu <chaozhu@linux.vnet.ibm.com>, Dekel Peled <dekelp@mellanox.com>, "dev@dpdk.org" <dev@dpdk.org>, Ori Kam <orika@mellanox.com>, "stable@dpdk.org" <stable@dpdk.org>, Yongseok Koh <yskoh@mellanox.com>, David Christensen <drc@ibm.com>, David Wilder <wilder@us.ibm.com> Thread-Topic: [PATCH] eal/ppc: remove fix of memory barrier for IBM POWER Thread-Index: AQHU3js5UXrzwqXJNk26I7vX4kWVFKYSzYSAgACJ6eCAABVbgIABsqAAgACpvJA= Date: Thu, 21 Mar 2019 08:49:39 +0000 Message-ID: <AM0PR0502MB3795291036D9019239391AF4C3420@AM0PR0502MB3795.eurprd05.prod.outlook.com> References: <1552913893-43407-1-git-send-email-dekelp@mellanox.com> <1789153.zrlSK8XYcq@xps> <AM0PR0502MB37957B54AFF7FF8C0629CDFCC3400@AM0PR0502MB3795.eurprd05.prod.outlook.com> <12440555.pBuX5BjkXC@xps> <OF129065AB.1B264FB9-ON882583C3.00792656-882583C3.007C9172@notes.na.collabserv.com> In-Reply-To: <OF129065AB.1B264FB9-ON882583C3.00792656-882583C3.007C9172@notes.na.collabserv.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=shahafs@mellanox.com; x-originating-ip: [31.154.10.105] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 6d7bfa07-7614-432f-5f13-08d6adda27b7 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020); SRVR:AM0PR0502MB3761; x-ms-traffictypediagnostic: AM0PR0502MB3761: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: <AM0PR0502MB376194B9F58B319ACA9A0E67C3420@AM0PR0502MB3761.eurprd05.prod.outlook.com> x-forefront-prvs: 0983EAD6B2 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39860400002)(366004)(346002)(136003)(376002)(396003)(52314003)(199004)(189003)(14454004)(66066001)(256004)(478600001)(74316002)(7736002)(33656002)(229853002)(446003)(6116002)(186003)(11346002)(790700001)(54906003)(3846002)(110136005)(99286004)(486006)(93886005)(476003)(6436002)(316002)(76176011)(8676002)(7696005)(71190400001)(105586002)(71200400001)(106356001)(6506007)(25786009)(2906002)(86362001)(97736004)(102836004)(26005)(9686003)(4326008)(8936002)(68736007)(81156014)(6306002)(55016002)(52536014)(5660300002)(53936002)(6246003)(2501003)(81166006)(54896002); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR0502MB3761; H:AM0PR0502MB3795.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:3; MX:3; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: Zm/SbkAURk1m6M0ysFP7FfrJ7hB6G45YLx4gkmUZWGV8DkT6H1OFs6MolYWj3TtPqHq19VgI/zcfncoJY3dUJOGQ7a1C7JSGDp05jmOkgBDRR9OaHY6xGY1xEQL+PHGAA4gE5LeIOxA3m/JWaicX23tWdnFs5vlPZEJkBT1FAvsVczfUPnh9ubS4LrvwmoUds/CwsrhSYMMlhRDy+ALHlcWyubNMEs3FeRxQ/n8pV7gQwTqmAyB0iMkkznzP1viISbGwgN0WsDldKXLd6szpvYGESXCHQEqk0cLnNXKJT1VL7zSQtnIGiq6yAQ4WGZXBmRFEuwlFjPgVRK/QF9V+cF7NYafeLsjWGrOfFJT1BlITKeP5L/PaKMumFnmy4HHyAl86gDZB89t2sV/ZJGQn2RXLmTYj5HuTh64603tkC60= MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6d7bfa07-7614-432f-5f13-08d6adda27b7 X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Mar 2019 08:49:39.9207 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR0502MB3761 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH] eal/ppc: remove fix of memory barrier for IBM POWER X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> X-List-Received-Date: Thu, 21 Mar 2019 08:49:43 -0000 Pradeep Satyanarayana <pradeep@us.ibm.com> wrote on Thu 3/21/2019 12:41 AM: >Thomas Monjalon <thomas@monjalon.net> wrote on 03/19/2019 01:45:01 PM: > >> From: Thomas Monjalon <thomas@monjalon.net> >> To: Shahaf Shuler <shahafs@mellanox.com> >> Cc: Dekel Peled <dekelp@mellanox.com>, Chao Zhu >> <chaozhu@linux.vnet.ibm.com>, Yongseok Koh <yskoh@mellanox.com>, >> "dev@dpdk.org" <dev@dpdk.org>, Ori Kam <orika@mellanox.com>, >> "stable@dpdk.org" <stable@dpdk.org>, pradeep@us.ibm.com >> Date: 03/19/2019 01:45 PM >> Subject: Re: [PATCH] eal/ppc: remove fix of memory barrier for IBM POWER >> [...] >> > >> > So far, when not running on power, we used the rte_wmb for that. >> On x86 and ARM systems it provided the needed guarantees. >> > It is also mentioned in the barrier doxygen on ARM arch: >> > " >> > Write memory barrier. >> > >> > Guarantees that the STORE operations generated before the barrier >> > occur before the STORE operations generated after. >> > " >> > >> > It doesn't restrict to store to system memory only. >> > w/ power is on somewhat different and in fact rte_mb is required. >> It obviously miss the point of those barrier if we will need to use >> a different barrier based on the system arch. >> > >> > We need to align the definition of the different barriers in DPDK: >> > 1. need a clear documentation of each. this should be global and >> not part of the specific implementation on each arch. > >A single approach may not work for all architectures. Power is different >from others, so we need to be able to accommodate that. More comments belo= w. it don't get this claim. It is ok to have some differences between the different arch, but here you = implement a well-defined barrier - rte_wmb. if you see a need we can discuss to define a **new** barrier which sync STO= RE only to system memory, and will be able to utilize the lwsync command. > >> >> The global definition is in lib/librte_eal/common/include/generic/rte_at= omic.h >> >> There are some copy/paste in Arm32 and PPC that I will remove. >> >> > 2. either modify ppc rte_wmb to match ARM and x86 ones or to >> define a new type of barrier which will sync between both I/O and >> stores to systems memory. >> >> The basic memory barrier of DPDK does not mention >> a difference between I/O and system memory. > >In the case of Power, sync will cater to both I/O and system memory. Howev= er, that >is too big a hammer in all cases. rte_wmb requires such sync. you propose to have the wrong barrier in favor = of performance. to mitigate this you can take my suggestion above and define a new, more li= ghtweight one. > >> It is not explicit (yet) but I assume it is protecting both. >> So, in my opinion, we need to make it explicit in the doc, >> and fix the PPC implementation to comply with this definition. >> >> Anyway, I don't see any significant effort from IBM to move from >> the alpha support stage to a real Open Source support. >> PS: sending a mail every two months, to promise improvements, is not eno= ugh! > [...] > >We should retain lwsync, should not be removed as discussed in here: > >http://mails.dpdk.org/archives/dev/2019-March/126746.html i don't agree. it is very clear the rte_wmb implementation in power is broken and we need = to fix this right away before other customers will hit the same issue. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <dev-bounces@dpdk.org> Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id C69E3A00E6 for <public@inbox.dpdk.org>; Thu, 21 Mar 2019 09:49:44 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 841071B450; Thu, 21 Mar 2019 09:49:44 +0100 (CET) Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-eopbgr150055.outbound.protection.outlook.com [40.107.15.55]) by dpdk.org (Postfix) with ESMTP id A8F031B44F; Thu, 21 Mar 2019 09:49:43 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=m7jXR4ouH4YVAglXs3BMVj1gHMA/ja8KESMbBd7EGQc=; b=c100MMSm2rVio43654QJuAmLy+H7XUs8oheYo1ioYHuoFK31HG4vWCbwja1XXwkik4zE62krGcxgSZ9K0WjK7zTGX6YTTPFJP0YiLiQEkudh5r8e7Rfn5H3APJMhKbfMpvnTuQU3WmoJpNb6UFYFwU0iZoTdjpEvNF4cdSm17Zc= Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com (52.133.45.150) by AM0PR0502MB3761.eurprd05.prod.outlook.com (52.133.50.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1730.15; Thu, 21 Mar 2019 08:49:40 +0000 Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::84f3:7e92:7a51:1003]) by AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::84f3:7e92:7a51:1003%2]) with mapi id 15.20.1730.013; Thu, 21 Mar 2019 08:49:40 +0000 From: Shahaf Shuler <shahafs@mellanox.com> To: "pradeep@us.ibm.com" <pradeep@us.ibm.com>, Thomas Monjalon <thomas@monjalon.net> CC: Chao Zhu <chaozhu@linux.vnet.ibm.com>, Dekel Peled <dekelp@mellanox.com>, "dev@dpdk.org" <dev@dpdk.org>, Ori Kam <orika@mellanox.com>, "stable@dpdk.org" <stable@dpdk.org>, Yongseok Koh <yskoh@mellanox.com>, David Christensen <drc@ibm.com>, David Wilder <wilder@us.ibm.com> Thread-Topic: [PATCH] eal/ppc: remove fix of memory barrier for IBM POWER Thread-Index: AQHU3js5UXrzwqXJNk26I7vX4kWVFKYSzYSAgACJ6eCAABVbgIABsqAAgACpvJA= Date: Thu, 21 Mar 2019 08:49:39 +0000 Message-ID: <AM0PR0502MB3795291036D9019239391AF4C3420@AM0PR0502MB3795.eurprd05.prod.outlook.com> References: <1552913893-43407-1-git-send-email-dekelp@mellanox.com> <1789153.zrlSK8XYcq@xps> <AM0PR0502MB37957B54AFF7FF8C0629CDFCC3400@AM0PR0502MB3795.eurprd05.prod.outlook.com> <12440555.pBuX5BjkXC@xps> <OF129065AB.1B264FB9-ON882583C3.00792656-882583C3.007C9172@notes.na.collabserv.com> In-Reply-To: <OF129065AB.1B264FB9-ON882583C3.00792656-882583C3.007C9172@notes.na.collabserv.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=shahafs@mellanox.com; x-originating-ip: [31.154.10.105] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 6d7bfa07-7614-432f-5f13-08d6adda27b7 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020); SRVR:AM0PR0502MB3761; x-ms-traffictypediagnostic: AM0PR0502MB3761: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: <AM0PR0502MB376194B9F58B319ACA9A0E67C3420@AM0PR0502MB3761.eurprd05.prod.outlook.com> x-forefront-prvs: 0983EAD6B2 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39860400002)(366004)(346002)(136003)(376002)(396003)(52314003)(199004)(189003)(14454004)(66066001)(256004)(478600001)(74316002)(7736002)(33656002)(229853002)(446003)(6116002)(186003)(11346002)(790700001)(54906003)(3846002)(110136005)(99286004)(486006)(93886005)(476003)(6436002)(316002)(76176011)(8676002)(7696005)(71190400001)(105586002)(71200400001)(106356001)(6506007)(25786009)(2906002)(86362001)(97736004)(102836004)(26005)(9686003)(4326008)(8936002)(68736007)(81156014)(6306002)(55016002)(52536014)(5660300002)(53936002)(6246003)(2501003)(81166006)(54896002); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR0502MB3761; H:AM0PR0502MB3795.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:3; MX:3; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: Zm/SbkAURk1m6M0ysFP7FfrJ7hB6G45YLx4gkmUZWGV8DkT6H1OFs6MolYWj3TtPqHq19VgI/zcfncoJY3dUJOGQ7a1C7JSGDp05jmOkgBDRR9OaHY6xGY1xEQL+PHGAA4gE5LeIOxA3m/JWaicX23tWdnFs5vlPZEJkBT1FAvsVczfUPnh9ubS4LrvwmoUds/CwsrhSYMMlhRDy+ALHlcWyubNMEs3FeRxQ/n8pV7gQwTqmAyB0iMkkznzP1viISbGwgN0WsDldKXLd6szpvYGESXCHQEqk0cLnNXKJT1VL7zSQtnIGiq6yAQ4WGZXBmRFEuwlFjPgVRK/QF9V+cF7NYafeLsjWGrOfFJT1BlITKeP5L/PaKMumFnmy4HHyAl86gDZB89t2sV/ZJGQn2RXLmTYj5HuTh64603tkC60= MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6d7bfa07-7614-432f-5f13-08d6adda27b7 X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Mar 2019 08:49:39.9207 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR0502MB3761 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH] eal/ppc: remove fix of memory barrier for IBM POWER X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org Sender: "dev" <dev-bounces@dpdk.org> Message-ID: <20190321084939.U7XXB0bSwR1Sq4AgHLZ8Xgk8zjpwR-DLbXFW3tZjJdI@z> Pradeep Satyanarayana <pradeep@us.ibm.com> wrote on Thu 3/21/2019 12:41 AM: >Thomas Monjalon <thomas@monjalon.net> wrote on 03/19/2019 01:45:01 PM: > >> From: Thomas Monjalon <thomas@monjalon.net> >> To: Shahaf Shuler <shahafs@mellanox.com> >> Cc: Dekel Peled <dekelp@mellanox.com>, Chao Zhu >> <chaozhu@linux.vnet.ibm.com>, Yongseok Koh <yskoh@mellanox.com>, >> "dev@dpdk.org" <dev@dpdk.org>, Ori Kam <orika@mellanox.com>, >> "stable@dpdk.org" <stable@dpdk.org>, pradeep@us.ibm.com >> Date: 03/19/2019 01:45 PM >> Subject: Re: [PATCH] eal/ppc: remove fix of memory barrier for IBM POWER >> [...] >> > >> > So far, when not running on power, we used the rte_wmb for that. >> On x86 and ARM systems it provided the needed guarantees. >> > It is also mentioned in the barrier doxygen on ARM arch: >> > " >> > Write memory barrier. >> > >> > Guarantees that the STORE operations generated before the barrier >> > occur before the STORE operations generated after. >> > " >> > >> > It doesn't restrict to store to system memory only. >> > w/ power is on somewhat different and in fact rte_mb is required. >> It obviously miss the point of those barrier if we will need to use >> a different barrier based on the system arch. >> > >> > We need to align the definition of the different barriers in DPDK: >> > 1. need a clear documentation of each. this should be global and >> not part of the specific implementation on each arch. > >A single approach may not work for all architectures. Power is different >from others, so we need to be able to accommodate that. More comments belo= w. it don't get this claim. It is ok to have some differences between the different arch, but here you = implement a well-defined barrier - rte_wmb. if you see a need we can discuss to define a **new** barrier which sync STO= RE only to system memory, and will be able to utilize the lwsync command. > >> >> The global definition is in lib/librte_eal/common/include/generic/rte_at= omic.h >> >> There are some copy/paste in Arm32 and PPC that I will remove. >> >> > 2. either modify ppc rte_wmb to match ARM and x86 ones or to >> define a new type of barrier which will sync between both I/O and >> stores to systems memory. >> >> The basic memory barrier of DPDK does not mention >> a difference between I/O and system memory. > >In the case of Power, sync will cater to both I/O and system memory. Howev= er, that >is too big a hammer in all cases. rte_wmb requires such sync. you propose to have the wrong barrier in favor = of performance. to mitigate this you can take my suggestion above and define a new, more li= ghtweight one. > >> It is not explicit (yet) but I assume it is protecting both. >> So, in my opinion, we need to make it explicit in the doc, >> and fix the PPC implementation to comply with this definition. >> >> Anyway, I don't see any significant effort from IBM to move from >> the alpha support stage to a real Open Source support. >> PS: sending a mail every two months, to promise improvements, is not eno= ugh! > [...] > >We should retain lwsync, should not be removed as discussed in here: > >http://mails.dpdk.org/archives/dev/2019-March/126746.html i don't agree. it is very clear the rte_wmb implementation in power is broken and we need = to fix this right away before other customers will hit the same issue.