From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-eopbgr60067.outbound.protection.outlook.com [40.107.6.67]) by dpdk.org (Postfix) with ESMTP id DAE7C4D27; Wed, 27 Mar 2019 23:21:20 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=RMxISgQDVsyYpRczs5BvrfWy8qSULVFnGxveRpY+Poo=; b=sOwbmj7MN/DYYjQbujkEs8OJqYRdt884kX7CaQtLZ5Xnw62s9V86DcIYlykTzfbFMgfMtIKUmWnkqNDPZHNJoHMi0JXWURC9xBsbJt4ZKSKpdwQuF8rRQLKOaKoI7kxHuBB8Ocmy6B4hKaS4wWhm8/gGP6md0ST1b6JQb9EtI9c= Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) by DB3PR0502MB4057.eurprd05.prod.outlook.com (52.134.67.18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1730.18; Wed, 27 Mar 2019 22:21:19 +0000 Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::6072:43be:7c2d:103a]) by DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::6072:43be:7c2d:103a%3]) with mapi id 15.20.1750.014; Wed, 27 Mar 2019 22:21:19 +0000 From: Yongseok Koh To: Shahaf Shuler CC: dev , dpdk stable , Kevin Traynor Thread-Topic: [dpdk-dev] [PATCH] net/mlx5: revert mbuf address calculation for x86 Thread-Index: AQHU5JNxdyjF17+s/0O8TnJoI7UoB6YgDaUA Date: Wed, 27 Mar 2019 22:21:19 +0000 Message-ID: References: <20190325191310.20594-1-yskoh@mellanox.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; x-originating-ip: [209.116.155.178] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 55f8b248-05d3-4ef5-a600-08d6b3028937 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(5600127)(711020)(4605104)(4618075)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(2017052603328)(7153060)(7193020); SRVR:DB3PR0502MB4057; x-ms-traffictypediagnostic: DB3PR0502MB4057: x-microsoft-antispam-prvs: x-forefront-prvs: 0989A7979C x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(136003)(396003)(39860400002)(346002)(376002)(366004)(189003)(199004)(78124002)(66066001)(14454004)(105586002)(81156014)(81166006)(99286004)(5660300002)(106356001)(33656002)(6636002)(7736002)(53936002)(478600001)(3846002)(6116002)(36756003)(6512007)(68736007)(316002)(71200400001)(37006003)(71190400001)(76176011)(54906003)(6506007)(83716004)(2906002)(53546011)(25786009)(102836004)(8936002)(4326008)(6486002)(486006)(97736004)(14444005)(6862004)(26005)(6436002)(186003)(6246003)(82746002)(86362001)(8676002)(11346002)(2616005)(256004)(476003)(446003)(305945005)(229853002); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB4057; H:DB3PR0502MB3980.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: r3wu38IR4hhvCoiVlbDSjVaikrETk9S5u2GFX+J+6cBj7MmpSEW4DFBWC9ysNzsdTLk9hVhMgGxKchNkgq0EiFAi20ie9Zb40A9vxQu73sQIAsiRK6lNc7dVJJyo6wtIR8Y8+tSc+0kUoedz8X3pAyHyZnD8xu4bKoQtjEDcLY/OO6Nfp8ekK/1pomQomD2Vm7ZhCm5BHj8xhdKF4G0UJ+1h0xw0si4pMCidnYTWvj5kV+RvxEIFqiqL01WsgLrHzKQN7peHhUGDEIi/Crw5M66Q7BMza7/a6f5Rq4mlCjkEKodiHLs9/EjhCiAhlXeMqJCJgbSspyiv3tra8hm7wABP75Cuxd+LsMeluiisZdYeo009vGiVhgROwSMAOcjy0agaQDigPWda+HqRvISzrQtYWyQBEdcnCAwIWsCJTuk= Content-Type: text/plain; charset="us-ascii" Content-ID: <3C36B99B86E2DC48AC582530817FB58F@eurprd05.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 55f8b248-05d3-4ef5-a600-08d6b3028937 X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Mar 2019 22:21:19.1134 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB4057 Subject: Re: [dpdk-dev] [PATCH] net/mlx5: revert mbuf address calculation for x86 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Mar 2019 22:21:21 -0000 > On Mar 27, 2019, at 4:51 AM, Kevin Traynor wrote: >=20 > On 25/03/2019 19:13, Yongseok Koh wrote: >> When replenishing mbufs on Rx, buffer address (mbuf->buf_addr) should be >> loaded. non-x86 processors (mostly RISC such as ARM and Power) are more >> vulnerable to load stall. For x86, reducing the number of instructions >> seems to matter most. >>=20 >> For x86, this is simply a load but for other architectures, it is >> calculated from the address of mbuf structure by rte_mbuf_buf_addr() >> without having to load the first cacheline of the mbuf. >>=20 >=20 > Hi Yongseok, >=20 >> Fixes: 12d468a62bc1 ("net/mlx5: fix instruction hotspot on replenishing = Rx buffer") >=20 > A similar backport was just added into 18.11.1-RC2, should it be > reverted? I'm not keen to put another fix for it in for 18.11.1 at this > stage, I think it can be part of 18.11.2. WDYT? I spoke with Kevin and we decided to drop the old fix. I have also dropped it from 17.11.6-rc1. This new fix will be merged to 18.11.2. I'll merge it to 17.11.6 (or 17.11.7) if it is merged in the master. thanks, Yongseok >> Cc: stable@dpdk.org >>=20 >> Signed-off-by: Yongseok Koh >> --- >> drivers/net/mlx5/mlx5_rxtx_vec.h | 14 +++++++++++++- >> 1 file changed, 13 insertions(+), 1 deletion(-) >>=20 >> diff --git a/drivers/net/mlx5/mlx5_rxtx_vec.h b/drivers/net/mlx5/mlx5_rx= tx_vec.h >> index 5df8e291e6..4220b08dd2 100644 >> --- a/drivers/net/mlx5/mlx5_rxtx_vec.h >> +++ b/drivers/net/mlx5/mlx5_rxtx_vec.h >> @@ -102,9 +102,21 @@ mlx5_rx_replenish_bulk_mbuf(struct mlx5_rxq_data *r= xq, uint16_t n) >> return; >> } >> for (i =3D 0; i < n; ++i) { >> - void *buf_addr =3D rte_mbuf_buf_addr(elts[i], rxq->mp); >> + void *buf_addr; >>=20 >> + /* >> + * Load the virtual address for Rx WQE. non-x86 processors >> + * (mostly RISC such as ARM and Power) are more vulnerable to >> + * load stall. For x86, reducing the number of instructions >> + * seems to matter most. >> + */ >> +#ifdef RTE_ARCH_X86_64 >> + buf_addr =3D elts[i]->buf_addr; >> + assert(buf_addr =3D=3D rte_mbuf_buf_addr(elts[i], rxq->mp)); >> +#else >> + buf_addr =3D rte_mbuf_buf_addr(elts[i], rxq->mp); >> assert(buf_addr =3D=3D elts[i]->buf_addr); >> +#endif >> wq[i].addr =3D rte_cpu_to_be_64((uintptr_t)buf_addr + >> RTE_PKTMBUF_HEADROOM); >> /* If there's only one MR, no need to replace LKey in WQE. */ >>=20 >=20 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id 2A48CA05D3 for ; Wed, 27 Mar 2019 23:21:23 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 09A7A4F9B; Wed, 27 Mar 2019 23:21:22 +0100 (CET) Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-eopbgr60067.outbound.protection.outlook.com [40.107.6.67]) by dpdk.org (Postfix) with ESMTP id DAE7C4D27; Wed, 27 Mar 2019 23:21:20 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=RMxISgQDVsyYpRczs5BvrfWy8qSULVFnGxveRpY+Poo=; b=sOwbmj7MN/DYYjQbujkEs8OJqYRdt884kX7CaQtLZ5Xnw62s9V86DcIYlykTzfbFMgfMtIKUmWnkqNDPZHNJoHMi0JXWURC9xBsbJt4ZKSKpdwQuF8rRQLKOaKoI7kxHuBB8Ocmy6B4hKaS4wWhm8/gGP6md0ST1b6JQb9EtI9c= Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) by DB3PR0502MB4057.eurprd05.prod.outlook.com (52.134.67.18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1730.18; Wed, 27 Mar 2019 22:21:19 +0000 Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::6072:43be:7c2d:103a]) by DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::6072:43be:7c2d:103a%3]) with mapi id 15.20.1750.014; Wed, 27 Mar 2019 22:21:19 +0000 From: Yongseok Koh To: Shahaf Shuler CC: dev , dpdk stable , Kevin Traynor Thread-Topic: [dpdk-dev] [PATCH] net/mlx5: revert mbuf address calculation for x86 Thread-Index: AQHU5JNxdyjF17+s/0O8TnJoI7UoB6YgDaUA Date: Wed, 27 Mar 2019 22:21:19 +0000 Message-ID: References: <20190325191310.20594-1-yskoh@mellanox.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; x-originating-ip: [209.116.155.178] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 55f8b248-05d3-4ef5-a600-08d6b3028937 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(5600127)(711020)(4605104)(4618075)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(2017052603328)(7153060)(7193020); SRVR:DB3PR0502MB4057; x-ms-traffictypediagnostic: DB3PR0502MB4057: x-microsoft-antispam-prvs: x-forefront-prvs: 0989A7979C x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(136003)(396003)(39860400002)(346002)(376002)(366004)(189003)(199004)(78124002)(66066001)(14454004)(105586002)(81156014)(81166006)(99286004)(5660300002)(106356001)(33656002)(6636002)(7736002)(53936002)(478600001)(3846002)(6116002)(36756003)(6512007)(68736007)(316002)(71200400001)(37006003)(71190400001)(76176011)(54906003)(6506007)(83716004)(2906002)(53546011)(25786009)(102836004)(8936002)(4326008)(6486002)(486006)(97736004)(14444005)(6862004)(26005)(6436002)(186003)(6246003)(82746002)(86362001)(8676002)(11346002)(2616005)(256004)(476003)(446003)(305945005)(229853002); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB4057; H:DB3PR0502MB3980.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: r3wu38IR4hhvCoiVlbDSjVaikrETk9S5u2GFX+J+6cBj7MmpSEW4DFBWC9ysNzsdTLk9hVhMgGxKchNkgq0EiFAi20ie9Zb40A9vxQu73sQIAsiRK6lNc7dVJJyo6wtIR8Y8+tSc+0kUoedz8X3pAyHyZnD8xu4bKoQtjEDcLY/OO6Nfp8ekK/1pomQomD2Vm7ZhCm5BHj8xhdKF4G0UJ+1h0xw0si4pMCidnYTWvj5kV+RvxEIFqiqL01WsgLrHzKQN7peHhUGDEIi/Crw5M66Q7BMza7/a6f5Rq4mlCjkEKodiHLs9/EjhCiAhlXeMqJCJgbSspyiv3tra8hm7wABP75Cuxd+LsMeluiisZdYeo009vGiVhgROwSMAOcjy0agaQDigPWda+HqRvISzrQtYWyQBEdcnCAwIWsCJTuk= Content-Type: text/plain; charset="UTF-8" Content-ID: <3C36B99B86E2DC48AC582530817FB58F@eurprd05.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 55f8b248-05d3-4ef5-a600-08d6b3028937 X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Mar 2019 22:21:19.1134 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB4057 Subject: Re: [dpdk-dev] [PATCH] net/mlx5: revert mbuf address calculation for x86 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190327222119.YjylCm9fa0SVc-TYi4tkRfadHDnKua03wUrUshA2Lfo@z> > On Mar 27, 2019, at 4:51 AM, Kevin Traynor wrote: >=20 > On 25/03/2019 19:13, Yongseok Koh wrote: >> When replenishing mbufs on Rx, buffer address (mbuf->buf_addr) should be >> loaded. non-x86 processors (mostly RISC such as ARM and Power) are more >> vulnerable to load stall. For x86, reducing the number of instructions >> seems to matter most. >>=20 >> For x86, this is simply a load but for other architectures, it is >> calculated from the address of mbuf structure by rte_mbuf_buf_addr() >> without having to load the first cacheline of the mbuf. >>=20 >=20 > Hi Yongseok, >=20 >> Fixes: 12d468a62bc1 ("net/mlx5: fix instruction hotspot on replenishing = Rx buffer") >=20 > A similar backport was just added into 18.11.1-RC2, should it be > reverted? I'm not keen to put another fix for it in for 18.11.1 at this > stage, I think it can be part of 18.11.2. WDYT? I spoke with Kevin and we decided to drop the old fix. I have also dropped it from 17.11.6-rc1. This new fix will be merged to 18.11.2. I'll merge it to 17.11.6 (or 17.11.7) if it is merged in the master. thanks, Yongseok >> Cc: stable@dpdk.org >>=20 >> Signed-off-by: Yongseok Koh >> --- >> drivers/net/mlx5/mlx5_rxtx_vec.h | 14 +++++++++++++- >> 1 file changed, 13 insertions(+), 1 deletion(-) >>=20 >> diff --git a/drivers/net/mlx5/mlx5_rxtx_vec.h b/drivers/net/mlx5/mlx5_rx= tx_vec.h >> index 5df8e291e6..4220b08dd2 100644 >> --- a/drivers/net/mlx5/mlx5_rxtx_vec.h >> +++ b/drivers/net/mlx5/mlx5_rxtx_vec.h >> @@ -102,9 +102,21 @@ mlx5_rx_replenish_bulk_mbuf(struct mlx5_rxq_data *r= xq, uint16_t n) >> return; >> } >> for (i =3D 0; i < n; ++i) { >> - void *buf_addr =3D rte_mbuf_buf_addr(elts[i], rxq->mp); >> + void *buf_addr; >>=20 >> + /* >> + * Load the virtual address for Rx WQE. non-x86 processors >> + * (mostly RISC such as ARM and Power) are more vulnerable to >> + * load stall. For x86, reducing the number of instructions >> + * seems to matter most. >> + */ >> +#ifdef RTE_ARCH_X86_64 >> + buf_addr =3D elts[i]->buf_addr; >> + assert(buf_addr =3D=3D rte_mbuf_buf_addr(elts[i], rxq->mp)); >> +#else >> + buf_addr =3D rte_mbuf_buf_addr(elts[i], rxq->mp); >> assert(buf_addr =3D=3D elts[i]->buf_addr); >> +#endif >> wq[i].addr =3D rte_cpu_to_be_64((uintptr_t)buf_addr + >> RTE_PKTMBUF_HEADROOM); >> /* If there's only one MR, no need to replace LKey in WQE. */ >>=20 >=20