From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 46C24A0C3F; Fri, 2 Jul 2021 23:52:09 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BD12841402; Fri, 2 Jul 2021 23:52:08 +0200 (CEST) Received: from EUR02-VE1-obe.outbound.protection.outlook.com (mail-eopbgr20076.outbound.protection.outlook.com [40.107.2.76]) by mails.dpdk.org (Postfix) with ESMTP id 19C5141363; Fri, 2 Jul 2021 23:52:08 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IhAu6z3NXVlkvAaYeXLwKaj4WR8uJ5epJ7Lm7MP2rkw=; b=IvuOzKSZE6LBFEPhIKZaSdRWjo7EaFUK7ED/Mb22tGbNqQv7nqo/JmZZAWXKmF3+3C57Ya0iQaKuZN62t+L9AcvqfLl6W7tx8B78NVA2Al86zY7I+oEfr4L5HlSqiIrDN8kyQKiCqxg88AS6mf5dx4vVorpjITVcGV7drVrrHDw= Received: from AM6PR04CA0071.eurprd04.prod.outlook.com (2603:10a6:20b:f0::48) by AM9PR08MB6049.eurprd08.prod.outlook.com (2603:10a6:20b:2de::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4287.27; Fri, 2 Jul 2021 21:52:05 +0000 Received: from AM5EUR03FT008.eop-EUR03.prod.protection.outlook.com (2603:10a6:20b:f0:cafe::46) by AM6PR04CA0071.outlook.office365.com (2603:10a6:20b:f0::48) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4287.23 via Frontend Transport; Fri, 2 Jul 2021 21:52:05 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM5EUR03FT008.mail.protection.outlook.com (10.152.16.123) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4287.22 via Frontend Transport; Fri, 2 Jul 2021 21:52:05 +0000 Received: ("Tessian outbound 80741586f868:v97"); Fri, 02 Jul 2021 21:52:05 +0000 X-CR-MTA-TID: 64aa7808 Received: from b18de12f3e29.3 by 64aa7808-outbound-1.mta.getcheckrecipient.com id D712BBD3-BB2A-44E0-8C31-35E21BCB5CE9.1; Fri, 02 Jul 2021 21:51:54 +0000 Received: from EUR03-AM5-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id b18de12f3e29.3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Fri, 02 Jul 2021 21:51:54 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ma5SFopgKNDr4Fe4opr8+C2mNgLW95MC/tUoWS2hXpjXEpMcyxDumIi9e5pHr2XHpaE0HIHCCTc1YgrjpmaDb/GRGb3pTq/Er8vGTeuWqtQB7bOtgmwujNfGLOuaBM79WxzKxsdL65oOymD9hEmVSTFPzUAubdRErGRwtZyHRqYrAOnXzO6EjkjpErj1c1F/owaRM2ek3XUQ4mvd2LbIU8s4BtQW8o4q7ahQb4QiuFN/Jz8/DIjBf6PwQHEgjfjANeDE6OoFVbEZaOvNKB0qq/N7kQa9dexvCjGgtSJBKCpv4CUq6P7gH0seaF7xN0aMaWsmJz8yymDEjemzR8YyNg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IhAu6z3NXVlkvAaYeXLwKaj4WR8uJ5epJ7Lm7MP2rkw=; b=DMIxOGt2sgbqRvJLoHdSdK4J9rZJQBbUSLY7aOs4l8AsosKD1iFMOELdyGLWb+abyH8mXx3ByR0GI4C0x9nDqxHfvqEFgm89bNe5kc0cNxagEnrO2qrRuU1Ib65X3j1fcOqlyybheF50KhQHCx2auiaiH/zgNRvx4koycDCMwZyelb5pNgWw5OrHG8l38auBU6M9PqXcYPSBQVJybC7PDvxCoktK/uEt22kdvTELzpe/RnMIto85VFnJactYmO2ya+qmO75uccFPtlgpzw0kTpX8y4lTcQf8N9rqDU3ndOy+78qebV0GPn3nU5dR6D+v3i94w2aSODAAE7bwmPuz2w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IhAu6z3NXVlkvAaYeXLwKaj4WR8uJ5epJ7Lm7MP2rkw=; b=IvuOzKSZE6LBFEPhIKZaSdRWjo7EaFUK7ED/Mb22tGbNqQv7nqo/JmZZAWXKmF3+3C57Ya0iQaKuZN62t+L9AcvqfLl6W7tx8B78NVA2Al86zY7I+oEfr4L5HlSqiIrDN8kyQKiCqxg88AS6mf5dx4vVorpjITVcGV7drVrrHDw= Received: from DBAPR08MB5814.eurprd08.prod.outlook.com (2603:10a6:10:1b1::6) by DBBPR08MB4444.eurprd08.prod.outlook.com (2603:10a6:10:c4::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4287.27; Fri, 2 Jul 2021 21:51:53 +0000 Received: from DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::f15f:821c:74c5:2482]) by DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::f15f:821c:74c5:2482%2]) with mapi id 15.20.4287.031; Fri, 2 Jul 2021 21:51:53 +0000 From: Honnappa Nagarahalli To: Slava Ovsiienko , "dev@dpdk.org" , Matan Azrad , Shahaf Shuler CC: Ruifeng Wang , Matan Azrad , "stable@dpdk.org" , nd , Honnappa Nagarahalli , nd Thread-Topic: [PATCH] net/mlx5: remove unwanted barrier Thread-Index: AQHXWvQGk/qeEMXKyEOAmwoZnR6qUKsuKwNQgACS0ZA= Date: Fri, 2 Jul 2021 21:51:52 +0000 Message-ID: References: <20210606164948.35997-1-honnappa.nagarahalli@arm.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: A5676A30B04AA1479089CEAA5497C550.0 x-checkrecipientchecked: true Authentication-Results-Original: nvidia.com; dkim=none (message not signed) header.d=none;nvidia.com; dmarc=none action=none header.from=arm.com; x-originating-ip: [107.77.221.47] x-ms-publictraffictype: Email X-MS-Office365-Filtering-Correlation-Id: 863380db-83b7-48b2-58bb-08d93da3a20a x-ms-traffictypediagnostic: DBBPR08MB4444:|AM9PR08MB6049: x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:9508;OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: hY3ILjA8KqSM42Y782neLEOROn8Jq5wOC+a4HKWOTk7NSGsrRwcEyzM4Nb1+qiCfltqPaoQhCu0amUenHEi+ezYZeqd2DaI7sAoZKDSNWftOv7wKm9SrOMeCPsGVXYCZPLAcsgeB3CIi+ameomh37jOdZzHHf+CMVQ2oIi5lf8cIzfY6yJnKKwnDcelRVYQFAMwld55crQa6KIeKjDh4Ji0SrBbS/eGtvswO6mfHtVtcVjMo+Pd/IcCXAlH8HaHpmXQUu6Axwc2d0tqOFkEFWgv+hRdLoRNwcjINM3GT9wB2obaa13D1G52BFum1r3WbG7NuiNgRSJQrZckBvYRkwaq8xvs4slD+OZdfMnBO8zkKSdoLDdK4tTQ7hPSHiCwz+lB3cWUXfxwzAlcgqbr36tKBgO8aiA4P+uTnlob5fR5gA7BOaXQPQHrLH5tOh+PMjn5HCNfUNxBX6RyoGHwWbXN/t4VeAzhChD4Vi//La5qVf9tI78Ipjve8eSop9AJtfpnojlFXq7C4d1AZnZ6mWUKaCR9ut3CG6BMDilMTRIh+YuQaxZkX0YO7r9IwIPnF1bI8rK2n7kYa9G+NvWFC6B8zCWHpZmQQmvARmhu1NyMlFE8vtKs+yKK2Axfe1XcvblS/iKTRTnAutYueMjHzxXTemhfYeozHq0lULSfaGwZ6Yt1Tf0VuqFuzXubash9EwwJaqrhnD5wSZ9rjrhDEcQ== X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DBAPR08MB5814.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(376002)(346002)(136003)(366004)(39850400004)(396003)(55016002)(316002)(7696005)(8676002)(2906002)(110136005)(52536014)(71200400001)(8936002)(86362001)(5660300002)(54906003)(38100700002)(64756008)(66446008)(66476007)(66556008)(76116006)(122000001)(478600001)(66946007)(186003)(26005)(9686003)(83380400001)(53546011)(33656002)(4326008)(6506007)(26123001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?M8vaarYeimfPsXMJCGrtGKm6O2W7yQ1XO2oJnYVkzFElj6YKcNJTf6OP3KyY?= =?us-ascii?Q?BWJrRk6WqJfdI5aA5DLacVbqAksZnVdYvCPW1+BWrQymK5ZnmXf73DRc8EFp?= =?us-ascii?Q?fdkC0QCYa9+I1jNIBxeGjx8UbrOEge3ggJt8W5/e+Px6SNgP2QxKF6EyxL95?= =?us-ascii?Q?29PciPpgpI7b60CUgg6m0d8ha6N1mga7xg5BAXXrpkx+cpW9vStEXEXfgvTz?= =?us-ascii?Q?jTDAtAhpD3KdCwXXUndZbjdKMYsdaZHbH7JdLjckIDpdfBLCXAKKJQGWl/qH?= =?us-ascii?Q?7MAocLA7ZV/+2yZ5UPA7kAAkapOXrhbSZXTZ/izDdgu3hFzWfgKRHRoK3Dty?= =?us-ascii?Q?i5ZpgNz//CtZ/T6Em0eqOnSIkjrH/InzkTg/OK0Hks0UQvnVOxyN/Y5m35mr?= =?us-ascii?Q?WptGPWZ86Ff2jz/jwLqI4eH51DXD6+8dNTdnTSaZYhjyt13XVjFFp8HHPTTr?= =?us-ascii?Q?1Oun2K2AFldIGnY00kGy7IQZ96eta13S4/ZhY3KpoYk2zGkARda+77PcpF6H?= =?us-ascii?Q?XxeNYbMNbXY5l1CDlbqcGMjdkRqAKdoQFb+Sx8JAkzeoyccXMVIqG+JeBTq4?= =?us-ascii?Q?OknUaaFA5ODdEViXAioA6O61dsHPlkCreUBIAAiqtsfBQG0ZIP4hQ8vj+3zW?= =?us-ascii?Q?elVbhOuvP5AMs5w9ee+pa95UmT73zRp5mgyHzirNwN8bDIQkcJn99F+qhXpY?= =?us-ascii?Q?TRt0EmPbQja/oBFm2ZKnXMEHJpX5LQBMWCvPlxSVPkI0NqTV8Wv++QhdK3O1?= =?us-ascii?Q?rbr8dv9L894dUALwn+y3Mm3uZ5angmscXdVcj2CHK/tW9CBOlCAsxjsckLTb?= =?us-ascii?Q?ioMzhcTdcX6CqDXmuiRf8ANZTaekYk5Devt6SbOyeSrZ5TAqYqiW5tJ6SDN/?= =?us-ascii?Q?aa1qyxtMx1cLKGkSWDfcCqOvSVKll0gPVnqFM/NQuFH3UhHkaSa4dnVsC8DG?= =?us-ascii?Q?3oJMy/E3ZLMK4rnMGmGJVwvi9+QBnd8q84rBFVUA/PTUXhhv0DIR60/1t4Fr?= =?us-ascii?Q?q1mi98tngbiyrFgbahwFpJKjF309jT19TrqARaB9q+NF0L9+UiB7rp+8t78B?= =?us-ascii?Q?cKcT2effFvK3kM/ePve2jkmue8Wjfxl9UVYCoL0PZQ+zYVbG7CHCxDbQQurV?= =?us-ascii?Q?7YydhYYHRoGps4IloGK1/TXCQ0F3htP7dkMJeoaf1Km8/PDHGHY2jTUPMjeZ?= =?us-ascii?Q?0lA3zhJgwNPV2Dx4FacFSExGREQGhCmdCsQVfmTF5BCzyxQt2LdZnka4irG7?= =?us-ascii?Q?LeshU4J26ybn3IW21+xGhFS1xhuxORiAYAsPAMgF9/dBYtw3cOyitVAtBtna?= =?us-ascii?Q?pSCzdcQ7AAkWHlbbNrhGJZTE?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR08MB4444 Original-Authentication-Results: nvidia.com; dkim=none (message not signed) header.d=none;nvidia.com; dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM5EUR03FT008.eop-EUR03.prod.protection.outlook.com X-MS-Office365-Filtering-Correlation-Id-Prvs: d2054296-5c9a-474e-1505-08d93da39a7b X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: SDwBOtgZUqU/Tr1+6r1uNQ7wqyn/n/9v1dsKm7Xa5AsNjkP/VznUCeOTkjmRYtCaK0lOvU6jS+qXCa4XRLEf3d8L1UceA1WPM+T9Mk+vCoZ/IKBnq7FbJ0c2MsvHkYzGlF+QtE3nu6uNHdKGeM4FRTA0ZvF8xJnubTCXwnHHOT6mrhJACdpAbVlyBg88p6p9QzkiyxQsRVCQQVtB2G1VnbMPrr4pJ9hMqIZWCBpQvz5qdR8tK7xPkUCOBjNjSTnHJ645LPx6zAVWEkRy+SiBButmJWdTeCNxGrryWDdN3tML0Y1uAx5rvg+JOgtGkXIdkFPPcAe5rFtOrnD/ChwpHWKuh+wclE7WvtgtFq5Cug3h5SNtJ9l873TE7L1+VcqnFffRDvpNnDJwo6UX6duTix4sr0WnQCBZKpFKl7VMs7udz2d3hq3viNqMdSsMvGA3FTvhLs+X2KlGYy69srcqZdBb+9QoWA5WT8Ws6V0bA1wFh8ZVjL5EN9UYt9lH4eB0Tj5jVFRwrEiDEmPailSIzO3xicLqziKyWrPPgLPHPjkyyeQ414kRO3HLWRXZhns1JvXIFXRka0Xhgher39KFGeZPElurovnscenV6F2/MyoRkhiaTlmFdNfhKoTNodVFXfw+BrdRt292NsiRPjwIKS5GELoYXRtwgc9+6ZaQnjwysQGOYkQQqLSlx+UgOm/FZvNqRBjhsOfDyuXbG+RFTkjJfZLKY7Yu8xCJ1w9u14Y= X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(4636009)(376002)(396003)(39850400004)(136003)(346002)(46966006)(36840700001)(55016002)(316002)(47076005)(7696005)(8676002)(54906003)(110136005)(36860700001)(2906002)(8936002)(336012)(86362001)(5660300002)(81166007)(82740400003)(52536014)(70586007)(478600001)(70206006)(186003)(26005)(82310400003)(9686003)(356005)(53546011)(450100002)(33656002)(83380400001)(4326008)(6506007)(26123001); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jul 2021 21:52:05.6386 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 863380db-83b7-48b2-58bb-08d93da3a20a X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: AM5EUR03FT008.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM9PR08MB6049 Subject: Re: [dpdk-dev] [PATCH] net/mlx5: remove unwanted barrier X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Slava, Thanks for taking the pain to explain this (this is similar to DD bits in = Intel i40e PMD). Agree, that for ensuring the ordering of the loads between= op_own and the rest of the fields we need a barrier. For Arm architecture,= we can use the barrier for inner sharable domain (rte_smp_rmb()) which is = less costly than the outer sharable domain (rte_io_rmb()). I will re-spin t= his patch. Thanks, Honnappa > -----Original Message----- > From: Slava Ovsiienko > Sent: Thursday, July 1, 2021 7:53 AM > To: Honnappa Nagarahalli ; > dev@dpdk.org; Matan Azrad ; Shahaf Shuler > > Cc: Ruifeng Wang ; Matan Azrad > ; stable@dpdk.org > Subject: RE: [PATCH] net/mlx5: remove unwanted barrier >=20 > Hi, Honnappa >=20 > The rte_io_rmb() was inserted not to prevent the extra access to cqe- > >op_own (the volatile qualifier is quite enough, if we had some doubts, w= e > would insert rte_compiler_barrier), but the real intention of io_rmw was = to > isolate cqe->op_own loads on hardware level. >=20 > cqe points to the Completion Queue Entry (CQE), that is mapped to the > memory that is continuously being updated by the device (NIC). CQE is 64B > size structure and op_own is located at the structure end, and is updated= by > HW in last order, after the entire CQE is completely written to the host > memory. >=20 > After detecting by cqe_check() the CQE is owned by software (hardware > completed operation) the PMD starts touching other CQE fields, i.e. the n= ext > load transactions from CQE are triggered. > And we must make sure these loads happen in correct order, only if cqe- > >op_own load was completed and valid ownership flags were seen, i.e. - do > not allow speculative reads with possible incorrect values fetched). >=20 > Just hypothetical case (I agree in advance - it is very unlikely, but is = not > impossible :)): >=20 > 1. owner =3D cqe->op_own - load A triggered 2. some code is being specula= tively > executed, no barrier 3. length =3D cqe->length - load B triggered 4. Let'= s suppose > CPU reordered A and B, ie order of loads: B, A 5. In memory/CPU cache we > have old CQE, owned by HW 6. B load gets the old length value (invalid) 7= . > Hardware writes the new CQE and CPU cache is invalidated 8. A load gets t= he > CQE is owned by SW and the invalid results of load B will be used by PMD >=20 > Hence, I would consider the patch as risky, and as one that is extremely = hard > to be covered completely with tests - we should test for race conditions = on > multiple architectures, on many CPU models, PCIe root complexes, etc. >=20 > With best regards, > Slava >=20 > > -----Original Message----- > > From: Honnappa Nagarahalli > > Sent: Sunday, June 6, 2021 19:50 > > To: dev@dpdk.org; honnappa.nagarahalli@arm.com; Matan Azrad > > ; Shahaf Shuler ; Slava > > Ovsiienko > > Cc: ruifeng.wang@arm.com; Matan Azrad ; > > stable@dpdk.org > > Subject: [PATCH] net/mlx5: remove unwanted barrier > > > > The IO barrier is not required as cqe->op_own is read once. The checks > > done on the local variable and the memory is not read again. > > > > Fixes: 88c0733535d6 ("net/mlx5: extend Rx completion with error > > handling") > > Cc: matan@mellanox.com > > Cc: stable@dpdk.org > > > > Signed-off-by: Honnappa Nagarahalli > > Reviewed-by: Ruifeng Wang > > --- > > drivers/common/mlx5/mlx5_common.h | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/drivers/common/mlx5/mlx5_common.h > > b/drivers/common/mlx5/mlx5_common.h > > index 5028a05b49..a4c29f51f1 100644 > > --- a/drivers/common/mlx5/mlx5_common.h > > +++ b/drivers/common/mlx5/mlx5_common.h > > @@ -195,7 +195,7 @@ check_cqe(volatile struct mlx5_cqe *cqe, const > > uint16_t cqes_n, > > > > if (unlikely((op_owner !=3D (!!(idx))) || (op_code =3D=3D > > MLX5_CQE_INVALID))) > > return MLX5_CQE_STATUS_HW_OWN; > > - rte_io_rmb(); > > + > > if (unlikely(op_code =3D=3D MLX5_CQE_RESP_ERR || > > op_code =3D=3D MLX5_CQE_REQ_ERR)) > > return MLX5_CQE_STATUS_ERR; > > -- > > 2.17.1