From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1EB4CA0471 for ; Mon, 9 Sep 2019 12:12:32 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E65FC1EB58; Mon, 9 Sep 2019 12:12:31 +0200 (CEST) Received: from EUR04-VI1-obe.outbound.protection.outlook.com (mail-eopbgr80054.outbound.protection.outlook.com [40.107.8.54]) by dpdk.org (Postfix) with ESMTP id 2D7B91C1ED; Mon, 9 Sep 2019 12:12:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=76/sNeoxWPUGFf0LrAGyejSqxLy+mhitKZJ+8lnvNic=; b=pALMo4YZ64nJqGULtqUos/ryOl3GxUthRcbn+8yLqqWtWp8EuuXy2Vm29mK+b9f02247omYZH5hjLkb0VGL8G0Hmd93AQWxr2nG/aLUzExOttk5umTS7+1VJyFQgpwxsxxzh2wbGhwUdBG5YRtfen2s7rcUvgn0bT0FksiqjTaA= Received: from VE1PR08CA0026.eurprd08.prod.outlook.com (2603:10a6:803:104::39) by VI1PR08MB4031.eurprd08.prod.outlook.com (2603:10a6:803:e2::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2241.18; Mon, 9 Sep 2019 10:12:28 +0000 Received: from AM5EUR03FT028.eop-EUR03.prod.protection.outlook.com (2a01:111:f400:7e08::203) by VE1PR08CA0026.outlook.office365.com (2603:10a6:803:104::39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2241.14 via Frontend Transport; Mon, 9 Sep 2019 10:12:27 +0000 Authentication-Results: spf=temperror (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=temperror action=none header.from=arm.com; Received-SPF: TempError (protection.outlook.com: error in processing during lookup of arm.com: DNS Timeout) Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM5EUR03FT028.mail.protection.outlook.com (10.152.16.118) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2241.14 via Frontend Transport; Mon, 9 Sep 2019 10:12:26 +0000 Received: ("Tessian outbound 32e58c90c37b:v28"); Mon, 09 Sep 2019 10:12:24 +0000 X-CR-MTA-TID: 64aa7808 Received: from d8c488484004.2 (ip-172-16-0-2.eu-west-1.compute.internal [104.47.0.53]) by 64aa7808-outbound-1.mta.getcheckrecipient.com id 75D705A8-8E4C-4901-A744-FCBCD40130E3.1; Mon, 09 Sep 2019 10:12:19 +0000 Received: from EUR01-HE1-obe.outbound.protection.outlook.com (mail-he1eur01lp2053.outbound.protection.outlook.com [104.47.0.53]) by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id d8c488484004.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Mon, 09 Sep 2019 10:12:19 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FhrP1Kct+TMEbmVpejq0tCt1d2BiX4g6VKX2oLr2TDWxu/Hfa/5+fOiHxKj7xSVRXZG0HhKAPH6rnlirR1HYfozbd16IUV4/12akcd6wG9XEnd37KOEUWXlnOg6ki4qv0tfEx1hlSXS5XXQtKJylJCsBZonpJN4OzTg4soTtGK5ye21S+MhqJXcwg0GJGF4+jgLZZGnMm6A4YQEWqyb2F387jRG6jvyoF7CeisHB0AnUiZUwgXPU9vTS8X7KOw8fl8JZKEcLIWQcdvRXr5jzisyHZ7xomu14Lpl1bNfBjZnwJ/bTKfo0IVr4wP6Ibcz3jK6+5A1dLDB7vt6+tATHJg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=76/sNeoxWPUGFf0LrAGyejSqxLy+mhitKZJ+8lnvNic=; b=l4BvH9IQIzpBrRq1DO6O6XvzhlmVu5cI8hhNLy+e3zftwd53ITkdtJQOycK7qY1NNH7LREoD6KZIvgS5ghGy62Hwd07sM5J7vFVTOtZiJxrhpFpW1FntyrBqzxozrN7wocCsWXqPo9d4vZ3nax8nEaYbrefmbUx3C6M1vqrKw6RDGcB3kA5/CTOjY5Vc5BVCu04eiqv5uOkfYdAi5yN1ZcA5Vj2skTAjYNX6mBAaf9dwzAzDUv+hGNc8etOYgk5gp+Yji6K6LeJY8YAro+XEzXQxhMJiGB25Zeu2KAx1Os2ue/sjpwCu8pnHphuwp03IbxbigFG036ZHz/Oyi07CAA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=76/sNeoxWPUGFf0LrAGyejSqxLy+mhitKZJ+8lnvNic=; b=pALMo4YZ64nJqGULtqUos/ryOl3GxUthRcbn+8yLqqWtWp8EuuXy2Vm29mK+b9f02247omYZH5hjLkb0VGL8G0Hmd93AQWxr2nG/aLUzExOttk5umTS7+1VJyFQgpwxsxxzh2wbGhwUdBG5YRtfen2s7rcUvgn0bT0FksiqjTaA= Received: from VE1PR08MB4640.eurprd08.prod.outlook.com (10.255.27.75) by VE1PR08MB4895.eurprd08.prod.outlook.com (10.255.113.212) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2220.21; Mon, 9 Sep 2019 10:12:15 +0000 Received: from VE1PR08MB4640.eurprd08.prod.outlook.com ([fe80::2d67:163d:b192:4b9]) by VE1PR08MB4640.eurprd08.prod.outlook.com ([fe80::2d67:163d:b192:4b9%7]) with mapi id 15.20.2220.022; Mon, 9 Sep 2019 10:12:15 +0000 From: "Phil Yang (Arm Technology China)" To: Slava Ovsiienko , "yskoh@mellanox.com" , Matan Azrad , =?iso-8859-1?Q?N=E9lio_Laranjeiro?= , "dev@dpdk.org" CC: "thomas@monjalon.net" , "jerinj@marvell.com" , Honnappa Nagarahalli , "Gavin Hu (Arm Technology China)" , nd , "stable@dpdk.org" , Steve Capper , nd Thread-Topic: [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on aarch64 Thread-Index: AQHVY+MlefwZT1B+GUKGfvGmUb8pcqceJ2yQgABtcQCABHeRsA== Date: Mon, 9 Sep 2019 10:12:15 +0000 Message-ID: References: <1567680908-31210-1-git-send-email-phil.yang@arm.com> <1567680908-31210-2-git-send-email-phil.yang@arm.com> In-Reply-To: Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 8d6344d1-7523-4e17-84bf-c3a11417d248.0 x-checkrecipientchecked: true Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=Phil.Yang@arm.com; x-originating-ip: [113.29.88.7] x-ms-publictraffictype: Email X-MS-Office365-Filtering-Correlation-Id: ecbecaa1-3dec-4738-ba54-08d7350e3713 X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam-Untrusted: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(7168020)(4627221)(201703031133081)(201702281549075)(8990200)(5600166)(711020)(4605104)(1401327)(4618075)(2017052603328)(7193020); SRVR:VE1PR08MB4895; X-MS-TrafficTypeDiagnostic: VE1PR08MB4895:|VE1PR08MB4895:|VI1PR08MB4031: X-MS-Exchange-PUrlCount: 1 x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true x-ms-oob-tlc-oobclassifiers: OLM:10000;OLM:10000; x-forefront-prvs: 01559F388D X-Forefront-Antispam-Report-Untrusted: SFV:NSPM; SFS:(10009020)(4636009)(346002)(39860400002)(136003)(376002)(396003)(366004)(13464003)(189003)(199004)(966005)(71200400001)(6436002)(476003)(7696005)(14454004)(74316002)(86362001)(9686003)(4326008)(2501003)(2906002)(110136005)(6246003)(76176011)(5660300002)(66066001)(229853002)(81156014)(8676002)(3846002)(305945005)(6116002)(7736002)(6506007)(76116006)(486006)(186003)(53546011)(6306002)(26005)(446003)(66946007)(102836004)(8936002)(11346002)(54906003)(55016002)(256004)(52536014)(33656002)(25786009)(14444005)(45080400002)(478600001)(66476007)(66556008)(53936002)(66574012)(64756008)(71190400001)(66446008)(316002)(55236004)(81166006)(99286004); DIR:OUT; SFP:1101; SCL:1; SRVR:VE1PR08MB4895; H:VE1PR08MB4640.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Message-Info-Original: aMCQVzTiH2zCziUbxzlJ9MxAJk/on9vMFasmjd/TzUdKiRlM8QtX1wlo+jtuKJZtx4tzBYs1oJ36xJvKNuI4ac8ajSfTq+p7vyHYL3/StzXSULSEtA6nJykDezJ8IjAOgBcn1+WLzNY4nU5dPq43kvBDdPxYltlyn73fzzF9+bULGWLVJA2ZEx5u1Y9plSanngrrD1fn3ZhV4XmOObEmpBjQc3RpQLVo07hnCBQQ+kazTIAV8LfsjMqgCuUeizNB9Y2m1bnjegQEHCkv2nQN+NnvtezDDEIWKXT9algzbdAfY6oRcF5fRKIF5hUTkvcEOD+GX76wrBlwLKa7ugzmBgU36ftoke68VAXQLbhj6lAQLPPon/u0RxVSf5vtGedTq1A/TaQMQr8Z5XecIQBV0FdLX3UvFPyVnl6M6i0Z4Yw= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VE1PR08MB4895 Original-Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Phil.Yang@arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM5EUR03FT028.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; IPV:CAL; SCL:-1; CTRY:IE; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(346002)(136003)(376002)(39860400002)(396003)(2980300002)(189003)(199004)(13464003)(54906003)(25786009)(316002)(8936002)(8746002)(478600001)(26826003)(966005)(2906002)(11346002)(26005)(186003)(3846002)(486006)(6116002)(126002)(23756003)(305945005)(55016002)(4326008)(6246003)(110136005)(229853002)(450100002)(50466002)(45080400002)(476003)(356004)(9686003)(33656002)(74316002)(6306002)(36906005)(53546011)(86362001)(52536014)(47776003)(6506007)(2501003)(66574012)(446003)(5660300002)(66066001)(81166006)(81156014)(22756006)(7736002)(336012)(63350400001)(63370400001)(14444005)(102836004)(99286004)(14454004)(70206006)(70586007)(8676002)(7696005)(76176011)(76130400001); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR08MB4031; H:64aa7808-outbound-1.mta.getcheckrecipient.com; FPR:; SPF:TempError; LANG:en; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; MX:1; A:1; X-MS-Office365-Filtering-Correlation-Id-Prvs: c30b7c51-8969-4ab4-5915-08d7350e30b9 X-Microsoft-Antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600166)(710020)(711020)(4605104)(1401327)(4618075)(2017052603328)(7193020); SRVR:VI1PR08MB4031; NoDisclaimer: True X-Forefront-PRVS: 01559F388D X-Microsoft-Antispam-Message-Info: H585o2eJreLY5u+ydjSKTsKiB7kpBL6O7aOlkvqoj/VzOFubwAHEcXS4goo2JGLrMTxVxNBSzqYtZYFmBRFze1m6o4sAUilsGULKQGCQYcoDxxklUr6nmgXsb+lTKBQxTr8AvBvIzDxKvncPfSgUoWZp1MnY5uk/UTiDVjsCW7pfFyJFAWOY9RMgC/9uLqoCFUDNmBiEXNZDPF9/GkQ/FW10+1Fat8+/0xlnPqedPLgidDZ3PJqH6cv+Xxyp3pnqJl4aoK3ctSG0OsDbQKUYou8LMOIfK8QY1ugLP6ECdDekk9F6AL4mqgwFG6RjX01V2YWe1deROQSPP+XzE/IcBtoicmuaiVxUwio1EqkFwYdRcEhb48rzwFPpqygMPdO6hN1BrBGFgz10uisue4c53A00QkQ1KVc+dL2466d6I60= X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Sep 2019 10:12:26.4107 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ecbecaa1-3dec-4738-ba54-08d7350e3713 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR08MB4031 Subject: Re: [dpdk-dev] [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on aarch64 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: Slava Ovsiienko > Sent: Friday, September 6, 2019 8:27 PM > To: Phil Yang (Arm Technology China) ; > yskoh@mellanox.com; Matan Azrad ; N=E9lio > Laranjeiro ; dev@dpdk.org > Cc: thomas@monjalon.net; jerinj@marvell.com; Honnappa Nagarahalli > ; Gavin Hu (Arm Technology China) > ; nd ; stable@dpdk.org; nd > > Subject: RE: [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on > aarch64 >=20 > Hi, Phil >=20 > Thanks for explanations, please, see below. >=20 > > -----Original Message----- > > From: Phil Yang (Arm Technology China) > > Sent: Friday, September 6, 2019 10:20 > > To: Slava Ovsiienko ; Yongseok Koh > > ; Matan Azrad ; N=E9lio > > Laranjeiro ; dev@dpdk.org > > Cc: Thomas Monjalon ; jerinj@marvell.com; > > Honnappa Nagarahalli ; Gavin Hu (Arm > > Technology China) ; nd ; > > stable@dpdk.org; nd > > Subject: RE: [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization o= n > > aarch64 > > > > Hi, Slava > > > > Thanks for your comments. > > > > > -----Original Message----- > > > From: Slava Ovsiienko > > > Sent: Thursday, September 5, 2019 8:12 PM > > > To: Phil Yang (Arm Technology China) ; > > > yskoh@mellanox.com; Matan Azrad ; N=E9lio > > Laranjeiro > > > ; dev@dpdk.org > > > Cc: thomas@monjalon.net; jerinj@marvell.com; Honnappa Nagarahalli > > > ; Gavin Hu (Arm Technology China) > > > ; nd ; stable@dpdk.org > > > Subject: RE: [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization > > > on > > > aarch64 > > > > > > Hi, Phil > > > > > > This point is in datapath and performance is very critical. > > > The rte_cio_wmb() may take a lot of CPU cycles, waiting till all > > > previous writes become visible for all external (relating to core) ag= ents. > > > The Tx CQE doorbelling does not need any writes to other locations to > > > be completed, > > > > In my understanding, the PMD needs to wait till all txq fields update i= s > > completed then ring the doorbell for HW. > > Before the Tx CQE doorbelling, it will update the producer index of wor= k > > queue in Tx queue descriptor (at line 2037). >=20 > txq->wqe_pi is exclusively software field, not related to HW directly. > We should not wait for write completions to this one (assuming the tx_bur= st() > must be called with strict affinity settings and core can't be changed). Understood, I really appreciate your explanation. Could you please review the 1/2 patch in this series? All your comments are= welcomed. >=20 > There may be some concern about reading from "last_cqe->wqe_counter" > at the line 2037. The compiler barrier was implemented to guarantee this > read is issued before doorbell write. >=20 > As for possible reordering these operations (read index from CQE at 2037 = and > write to CQ doorbell register at 2046): >=20 > a) read is performed from already cached area (we touched > this CQE performing ownership check very recently) so it is quite unlikel= y > to be completed after the doorbell write Yes. The "last_cqe->we_counter " is cached. However, it might not guarantee= the cached data is valid when CPU issues the read operation. Because mlx5_cqe is in the coherent memory and is shared with the HW, so up= dating of any other filed in CQE will invalid the whole cache line. In that case, it is possible to complete the doorbell write before the wqe_= counter read completed.=20 So we might need a rte_cio_rmb() here, right? >=20 > b) The only risk to read wrong data is the case of CQE overwriting by HW > with CQ buffer overflow. We create the CQ ring buffer with some extra > space, > so completions which are "in-flight" can't overwrite the CQE is being rea= d. >=20 > The new completion request may be issued by setting flags in WQE > descriptors > and following SQ doorbell write, which is already prepended by wmb. > (in mlx5_tx_dbrec_cond_wmb(), line 4733). So, it seems there is no chance > for CQE to be overwritten. >=20 > > The compiler barrier cannot guarantee the ordering of these operations.= So > > use the explicit HW fence to achieve that. > > > > As same as the HW Tx doorbell in vectorized Tx burst routine, it uses a= write > > memory barrier to enforce the register update visible to HW immediately= . > > Section 32.5.2 in > > > https://eur03.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fdoc.d > > > pdk.org%2Fguides%2Fnics%2Fmlx5.html&data=3D02%7C01%7Cviacheslav > > o%40mellanox.com%7C76938b08a9f145c4a0dd08d7329ab932%7Ca652971 > > c7d2e4d9ba6a4d149256f461b%7C0%7C1%7C637033512428674501&sd > > > ata=3D8tdVjY0%2FHOUFo1%2BeHiuqkPadSS%2FHLeo4b97gdgEHgME%3D&am > > p;reserved=3D0 >=20 > This is quite different case. PMD build descriptors (WQEs) in the memory = and > must > guarantee these data are visible for external agents before SQ (sending > queue, > not completion queue) doorbelling. Now there are no vectorized Tx routine= s > (since 19.08), > but, of course, we still have the "true" write memory barrier (in > mlx5_tx_dbrec_cond_wmb) > for this case. >=20 > > > > > the only concern is not to reorder/merge the writes to the same > > > doorbell register of the same sending queue in the tx_burst() interna= l > > sending loop/subsequent calls. > > > > > > As far as I know - the writes to the same location should not be > > > reordered by any arch (may be merged if memory settings allow this, i= t > > > is not critical for CQE doorbell), could you, please, explain why we > > > need explicit hardware fence before CQE doorbell update? Do you think > > > doorbell write might be rearranged with previously reads from the rin= g > > buffer? > > > > > > WBR, > > > Slava > > > > > > > -----Original Message----- > > > > From: Phil Yang > > > > Sent: Thursday, September 5, 2019 13:55 > > > > To: Yongseok Koh ; Slava Ovsiienko > > > > ; Matan Azrad ; > > > N=E9lio > > > > Laranjeiro ; dev@dpdk.org > > > > Cc: Thomas Monjalon ; jerinj@marvell.com; > > > > Honnappa.Nagarahalli@arm.com; gavin.hu@arm.com; nd@arm.com; > > > > stable@dpdk.org > > > > Subject: [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization o= n > > > > aarch64 > > > > > > > > For the weaker memory model processors, the compiler barrier is not > > > > sufficient to guarantee the coherent memory update be observed by > > > > I/O device. It needs the coherent I/O memory barrier to enforce the > > > > ordering > > > of > > > > Tx completion queue doorbell operation. > > > > > > > > Fixes: da1df1ccabad ("net/mlx5: fix completion queue drain loop") > > > > Cc: stable@dpdk.org > > > > > > > > Suggested-by: Gavin Hu > > > > Signed-off-by: Phil Yang > > > > Reviewed-by: Gavin Hu > > > > --- > > > > drivers/net/mlx5/mlx5_rxtx.c | 2 +- > > > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > > > > > diff --git a/drivers/net/mlx5/mlx5_rxtx.c > > > > b/drivers/net/mlx5/mlx5_rxtx.c index 4c01187..c11148b 100644 > > > > --- a/drivers/net/mlx5/mlx5_rxtx.c > > > > +++ b/drivers/net/mlx5/mlx5_rxtx.c > > > > @@ -2042,7 +2042,7 @@ mlx5_tx_comp_flush(struct mlx5_txq_data > > > > *restrict txq, > > > > } else { > > > > return; > > > > } > > > > - rte_compiler_barrier(); > > > > + rte_cio_wmb(); > > > > *txq->cq_db =3D rte_cpu_to_be_32(txq->cq_ci); > > > > if (likely(tail !=3D txq->elts_tail)) { > > > > mlx5_tx_free_elts(txq, tail, olx); > > > > -- > > > > 2.7.4