From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR02-AM5-obe.outbound.protection.outlook.com (mail-eopbgr00051.outbound.protection.outlook.com [40.107.0.51]) by dpdk.org (Postfix) with ESMTP id B9341A3 for ; Mon, 18 Mar 2019 10:37:33 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=puXjyCNLalPhciNvA2mkRUofUqiMQkefZ4aV/vcDhd8=; b=wMluYoeofasgDxZGujpxkHePLji6n6gEThimD6XAvAc2dYNuaqZ66vz6A60IH1FYTa6tMta9VSUMN5iQAmBvN8CZZFxhadrFff98shiViqAb0IgFr2W7bydoc4SGFjKLabSQYouDR4vrhU4EdnpSwDpXLHFBy/C6A/NlfXUqQKA= Received: from VI1PR05MB4224.eurprd05.prod.outlook.com (52.133.12.13) by VI1PR05MB4670.eurprd05.prod.outlook.com (20.176.3.155) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1709.13; Mon, 18 Mar 2019 09:37:31 +0000 Received: from VI1PR05MB4224.eurprd05.prod.outlook.com ([fe80::bcb0:ed58:d76:cac]) by VI1PR05MB4224.eurprd05.prod.outlook.com ([fe80::bcb0:ed58:d76:cac%4]) with mapi id 15.20.1709.015; Mon, 18 Mar 2019 09:37:31 +0000 From: Dekel Peled To: Chao Zhu CC: "dev@dpdk.org" , "pradeep@us.ibm.com" , Yongseok Koh , Ori Kam , Thomas Monjalon , 'David Wilder' , Shahaf Shuler , Olga Shern Thread-Topic: issue with memory sync on P9 Thread-Index: AdTdbgiThOK2Bh+aQQ22M3oBoYmqUQ== Date: Mon, 18 Mar 2019 09:37:31 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=dekelp@mellanox.com; x-originating-ip: [193.47.165.251] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 4ad226dd-8c98-4442-a353-08d6ab8557d9 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020); SRVR:VI1PR05MB4670; x-ms-traffictypediagnostic: VI1PR05MB4670: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: x-forefront-prvs: 098076C36C x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(979002)(346002)(376002)(136003)(39860400002)(396003)(366004)(189003)(199004)(13464003)(2906002)(33656002)(486006)(6506007)(53546011)(102836004)(7696005)(99286004)(97736004)(476003)(6916009)(8936002)(105586002)(106356001)(107886003)(53936002)(68736007)(14454004)(6436002)(25786009)(316002)(54906003)(66066001)(478600001)(4326008)(55016002)(9686003)(186003)(81166006)(81156014)(14444005)(256004)(74316002)(3846002)(7736002)(52536014)(8676002)(6116002)(305945005)(86362001)(26005)(5660300002)(71190400001)(71200400001)(969003)(989001)(999001)(1009001)(1019001); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR05MB4670; H:VI1PR05MB4224.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: 7+QpwqO2WJArvokZxqZs+kw5OVg4d2K4QI8EIkrshNP9G4L9rQpa1Yid6LUKfws2Pz4jZpNc024wOAXjN56GCxS7zXXSGgGgBscHNqo8NFg/hm63rPhFfeD+fj8zNSaQjCao4rSW29y8WD80/5+Y60QSdWXjEx2inZ1LvcxWYhXFRbRQ023UYLgMubeBA6LjJx/ERyILDzdA5MVMswPIrau+a+DzUZWTRSZ9Boyk1c97YJWIixPLkUiaJA/r2LGwa1PAydhKkydr7mTqpxWk7UoRh3XKlQzm4JVVGwTVi8BaU1oxN6JRyQC468HR1SxL+16QQYfxCVXdHXS1IETbJV7OJ8YgDFxaIWLp28/mmFotiWfkk+q1ikFGdWjxoko+46vGvI/G5LNUSYfHSpOw8mOMWHl/TDSpEWpbYn/1SbI= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4ad226dd-8c98-4442-a353-08d6ab8557d9 X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Mar 2019 09:37:31.0263 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR05MB4670 Subject: [dpdk-dev] issue with memory sync on P9 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Mar 2019 09:37:33 -0000 Hi Chao Zhu, We have recently encountered runtime issues when running DPDK 18.11 (and la= ter) MLX5 PMD on P9 setup. Using rte_wmb() was found to be insufficient in specific location, memory s= ync problem occurred regardless. Using rte_mb() in this specific location fixed the issue. --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -706,7 +706,7 @@ mlx5_tx_dbrec_cond_wmb(struct mlx5_txq_data *txq, volat= ile struct mlx5_wqe *wqe, rte_cio_wmb(); *txq->qp_db =3D rte_cpu_to_be_32(txq->wqe_ci); /* Ensure ordering between DB record and BF copy. */ - rte_wmb(); + rte_mb(); mlx5_uar_write64_relaxed(*src, dst, txq->uar_lock); if (cond) rte_wmb(); I see the change in rte_wmb() implementation comes from your patch: commit d23a6bd04d728226c99d6995c8bd65d49d7e61d1) Author: Chao Zhu Date: Fri Jul 15 10:30:19 2016 +0800 eal/ppc: fix memory barrier for IBM POWER I ran some performance tests and got the same results with and without this= patch. Do you have different findings? Will you approve removing this patch? Regards, Dekel > -----Original Message----- > From: dev On Behalf Of Chao Zhu > Sent: Wednesday, February 20, 2019 10:07 AM > To: Thomas Monjalon ; 'David Wilder' > > Cc: dev@dpdk.org; pradeep@us.ibm.com > Subject: Re: [dpdk-dev] [dpdk-stable] [PATCH v3] mem: Fix anonymous > mapping on Power9. >=20 > +Pradeep >=20 > > -----Original Message----- > > From: Thomas Monjalon > > Sent: Monday, February 18, 2019 11:47 PM > > To: David Wilder > > Cc: dev@dpdk.org; chaozhu@linux.vnet.ibm.com > > Subject: Re: [dpdk-stable] [PATCH v3] mem: Fix anonymous mapping on > > Power9. > > > > Hi, > > > > Long time no talk about the IBM Power support status. > > > > 16/11/2018 18:35, David Wilder: > > > --- a/doc/guides/rel_notes/release_18_11.rst > > > +++ b/doc/guides/rel_notes/release_18_11.rst > > > +* IBM Power8 is not supported by this release of DPDK. IBM Power9 > > > +is > > supported. > > > > The config file is defconfig_ppc_64-power8-linuxapp-gcc. > > Given that only Power9 is supported, should we rename it? > > > > Is there more changes required to make Power9 working with the latest > DPDK? > > >=20 >=20 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id 61B87A0662 for ; Mon, 18 Mar 2019 10:37:35 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 2CF97152A; Mon, 18 Mar 2019 10:37:35 +0100 (CET) Received: from EUR02-AM5-obe.outbound.protection.outlook.com (mail-eopbgr00051.outbound.protection.outlook.com [40.107.0.51]) by dpdk.org (Postfix) with ESMTP id B9341A3 for ; Mon, 18 Mar 2019 10:37:33 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=puXjyCNLalPhciNvA2mkRUofUqiMQkefZ4aV/vcDhd8=; b=wMluYoeofasgDxZGujpxkHePLji6n6gEThimD6XAvAc2dYNuaqZ66vz6A60IH1FYTa6tMta9VSUMN5iQAmBvN8CZZFxhadrFff98shiViqAb0IgFr2W7bydoc4SGFjKLabSQYouDR4vrhU4EdnpSwDpXLHFBy/C6A/NlfXUqQKA= Received: from VI1PR05MB4224.eurprd05.prod.outlook.com (52.133.12.13) by VI1PR05MB4670.eurprd05.prod.outlook.com (20.176.3.155) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1709.13; Mon, 18 Mar 2019 09:37:31 +0000 Received: from VI1PR05MB4224.eurprd05.prod.outlook.com ([fe80::bcb0:ed58:d76:cac]) by VI1PR05MB4224.eurprd05.prod.outlook.com ([fe80::bcb0:ed58:d76:cac%4]) with mapi id 15.20.1709.015; Mon, 18 Mar 2019 09:37:31 +0000 From: Dekel Peled To: Chao Zhu CC: "dev@dpdk.org" , "pradeep@us.ibm.com" , Yongseok Koh , Ori Kam , Thomas Monjalon , 'David Wilder' , Shahaf Shuler , Olga Shern Thread-Topic: issue with memory sync on P9 Thread-Index: AdTdbgiThOK2Bh+aQQ22M3oBoYmqUQ== Date: Mon, 18 Mar 2019 09:37:31 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=dekelp@mellanox.com; x-originating-ip: [193.47.165.251] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 4ad226dd-8c98-4442-a353-08d6ab8557d9 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020); SRVR:VI1PR05MB4670; x-ms-traffictypediagnostic: VI1PR05MB4670: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: x-forefront-prvs: 098076C36C x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(979002)(346002)(376002)(136003)(39860400002)(396003)(366004)(189003)(199004)(13464003)(2906002)(33656002)(486006)(6506007)(53546011)(102836004)(7696005)(99286004)(97736004)(476003)(6916009)(8936002)(105586002)(106356001)(107886003)(53936002)(68736007)(14454004)(6436002)(25786009)(316002)(54906003)(66066001)(478600001)(4326008)(55016002)(9686003)(186003)(81166006)(81156014)(14444005)(256004)(74316002)(3846002)(7736002)(52536014)(8676002)(6116002)(305945005)(86362001)(26005)(5660300002)(71190400001)(71200400001)(969003)(989001)(999001)(1009001)(1019001); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR05MB4670; H:VI1PR05MB4224.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: 7+QpwqO2WJArvokZxqZs+kw5OVg4d2K4QI8EIkrshNP9G4L9rQpa1Yid6LUKfws2Pz4jZpNc024wOAXjN56GCxS7zXXSGgGgBscHNqo8NFg/hm63rPhFfeD+fj8zNSaQjCao4rSW29y8WD80/5+Y60QSdWXjEx2inZ1LvcxWYhXFRbRQ023UYLgMubeBA6LjJx/ERyILDzdA5MVMswPIrau+a+DzUZWTRSZ9Boyk1c97YJWIixPLkUiaJA/r2LGwa1PAydhKkydr7mTqpxWk7UoRh3XKlQzm4JVVGwTVi8BaU1oxN6JRyQC468HR1SxL+16QQYfxCVXdHXS1IETbJV7OJ8YgDFxaIWLp28/mmFotiWfkk+q1ikFGdWjxoko+46vGvI/G5LNUSYfHSpOw8mOMWHl/TDSpEWpbYn/1SbI= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4ad226dd-8c98-4442-a353-08d6ab8557d9 X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Mar 2019 09:37:31.0263 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR05MB4670 Subject: [dpdk-dev] issue with memory sync on P9 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190318093731.Y9vYr-LAczeVIuOrsdE2dz6I1A7E8szinXBm0dsy68E@z> Hi Chao Zhu, We have recently encountered runtime issues when running DPDK 18.11 (and la= ter) MLX5 PMD on P9 setup. Using rte_wmb() was found to be insufficient in specific location, memory s= ync problem occurred regardless. Using rte_mb() in this specific location fixed the issue. --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -706,7 +706,7 @@ mlx5_tx_dbrec_cond_wmb(struct mlx5_txq_data *txq, volat= ile struct mlx5_wqe *wqe, rte_cio_wmb(); *txq->qp_db =3D rte_cpu_to_be_32(txq->wqe_ci); /* Ensure ordering between DB record and BF copy. */ - rte_wmb(); + rte_mb(); mlx5_uar_write64_relaxed(*src, dst, txq->uar_lock); if (cond) rte_wmb(); I see the change in rte_wmb() implementation comes from your patch: commit d23a6bd04d728226c99d6995c8bd65d49d7e61d1) Author: Chao Zhu Date: Fri Jul 15 10:30:19 2016 +0800 eal/ppc: fix memory barrier for IBM POWER I ran some performance tests and got the same results with and without this= patch. Do you have different findings? Will you approve removing this patch? Regards, Dekel > -----Original Message----- > From: dev On Behalf Of Chao Zhu > Sent: Wednesday, February 20, 2019 10:07 AM > To: Thomas Monjalon ; 'David Wilder' > > Cc: dev@dpdk.org; pradeep@us.ibm.com > Subject: Re: [dpdk-dev] [dpdk-stable] [PATCH v3] mem: Fix anonymous > mapping on Power9. >=20 > +Pradeep >=20 > > -----Original Message----- > > From: Thomas Monjalon > > Sent: Monday, February 18, 2019 11:47 PM > > To: David Wilder > > Cc: dev@dpdk.org; chaozhu@linux.vnet.ibm.com > > Subject: Re: [dpdk-stable] [PATCH v3] mem: Fix anonymous mapping on > > Power9. > > > > Hi, > > > > Long time no talk about the IBM Power support status. > > > > 16/11/2018 18:35, David Wilder: > > > --- a/doc/guides/rel_notes/release_18_11.rst > > > +++ b/doc/guides/rel_notes/release_18_11.rst > > > +* IBM Power8 is not supported by this release of DPDK. IBM Power9 > > > +is > > supported. > > > > The config file is defconfig_ppc_64-power8-linuxapp-gcc. > > Given that only Power9 is supported, should we rename it? > > > > Is there more changes required to make Power9 working with the latest > DPDK? > > >=20 >=20