From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 37B77A04F8 for ; Tue, 24 Dec 2019 16:46:37 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E47DAF72; Tue, 24 Dec 2019 16:46:36 +0100 (CET) Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-eopbgr60073.outbound.protection.outlook.com [40.107.6.73]) by dpdk.org (Postfix) with ESMTP id E6B5BF72; Tue, 24 Dec 2019 16:46:35 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TDyGTIRAK7kxO32HA9uWoEZ0jYhlunec5TZRlxhjV4k=; b=H+3i8ZZp8f882of3FKRFD4RtX8uFgO3qu5jMT70Q5z4XXvftgNwUJpRdL/wttyufAQGJVnXRH1PGILq5aIIGGBF0HYJnoSDPyjUrpH3cfLQjNT9axqf5ys1iJpJO6JVAkElI3466wSbHxjX/x0V7kTk+BQhvAdR1S+eXlZqe2Zo= Received: from AM4PR08CA0069.eurprd08.prod.outlook.com (2603:10a6:205:2::40) by AM0PR08MB5092.eurprd08.prod.outlook.com (2603:10a6:208:15f::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2581.11; Tue, 24 Dec 2019 15:46:34 +0000 Received: from DB5EUR03FT054.eop-EUR03.prod.protection.outlook.com (2a01:111:f400:7e0a::205) by AM4PR08CA0069.outlook.office365.com (2603:10a6:205:2::40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2559.14 via Frontend Transport; Tue, 24 Dec 2019 15:46:34 +0000 Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=bestguesspass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DB5EUR03FT054.mail.protection.outlook.com (10.152.20.248) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2559.14 via Frontend Transport; Tue, 24 Dec 2019 15:46:33 +0000 Received: ("Tessian outbound 1da651c29646:v40"); Tue, 24 Dec 2019 15:46:32 +0000 X-CR-MTA-TID: 64aa7808 Received: from 6e9db96413b0.2 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 78431738-0D7C-4741-B904-7FCB85F9D6D4.1; Tue, 24 Dec 2019 15:46:27 +0000 Received: from EUR02-HE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 6e9db96413b0.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Tue, 24 Dec 2019 15:46:27 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ItvqBmVs+iy/3uWEYkSMrT+iuaXTeAiy5AM/0DM0KQbS0MoiJtYd0CE2Is9+1KWHHQ/BffMMUd3BPmKA/iTYiD5uN3bxoppkaOWOt0HHltmJz/EHencqUTFS497ZLQQ5N3BnelG5wHT4xK2U/kbvsu4FCH0t3GKTYfEgux2K1+DaXgd7TXDjLDcn7C9nyBaKCgqB/57hN06rJKnmT2iF5uSmHpp9kloGsY51+fJxeA8f0dw/gPAk7b1nTv/NmZF37QcKS9iJloGJq29XLrNU9+sk52gpggZCpjtRUghKOfzz0AsZndepY3gO/woCE7QJzRZbbN1D8YG7ZvdBCtvAdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TDyGTIRAK7kxO32HA9uWoEZ0jYhlunec5TZRlxhjV4k=; b=bnLEEodN8pxAxvMQlA2E6n2k1tHrZVpXoNnbchzvc/J8J7kI3Z5eLqAt/9xMuHkAxeIVcMAOsvv35PqS3BEyGKX9XQUH36KMrQtVGszNGlYkzyB6CLixd8eKXaa+igJIhsCO2H0fnX2rQw7FBOgZNoLxyr8+jqOgx4KRlSgx+7b5K51Zh+Hk6qwvfh8OSyu8CA5te7YG/Uwu/+QtPlmde0lOBm2+G8i+UbRZKE5Pcop90Wdxa/uADnK9rcaMjx0e5Mg+nJopsFgjwxMEA+LvgQt7Y+/HV0RqUzgLD75M6bodOmvIpPDy3Gta+YThxt6JuvI8Tm6YC2/8fnfYHOf0cw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TDyGTIRAK7kxO32HA9uWoEZ0jYhlunec5TZRlxhjV4k=; b=H+3i8ZZp8f882of3FKRFD4RtX8uFgO3qu5jMT70Q5z4XXvftgNwUJpRdL/wttyufAQGJVnXRH1PGILq5aIIGGBF0HYJnoSDPyjUrpH3cfLQjNT9axqf5ys1iJpJO6JVAkElI3466wSbHxjX/x0V7kTk+BQhvAdR1S+eXlZqe2Zo= Received: from VI1PR08MB5376.eurprd08.prod.outlook.com (10.255.196.79) by VI1PR08MB3727.eurprd08.prod.outlook.com (20.178.13.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2559.15; Tue, 24 Dec 2019 15:46:20 +0000 Received: from VI1PR08MB5376.eurprd08.prod.outlook.com ([fe80::dde8:216f:6a0b:8cfb]) by VI1PR08MB5376.eurprd08.prod.outlook.com ([fe80::dde8:216f:6a0b:8cfb%7]) with mapi id 15.20.2559.017; Tue, 24 Dec 2019 15:46:20 +0000 From: Gavin Hu To: "Li, Xiaoyun" , "Wu, Jingjing" CC: "dev@dpdk.org" , "Maslekar, Omkar" , "stable@dpdk.org" , nd , "jerinj@marvell.com" , Honnappa Nagarahalli , "Richardson, Bruce" , nd , nd Thread-Topic: [dpdk-dev] [PATCH v2] raw/ntb: fix write memory barrier issue Thread-Index: AQHVs7Tni9/XHz+bM0yLOm9Kl+X1h6e8lRjwgArPOoCAAAdocIAAFYwAgAHo/EA= Date: Tue, 24 Dec 2019 15:46:20 +0000 Message-ID: References: <20191204151916.12607-1-xiaoyun.li@intel.com> <20191216015854.28725-1-xiaoyun.li@intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 425bf1e9-3905-4d31-a0d1-06232fe1d395.0 x-checkrecipientchecked: true Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=Gavin.Hu@arm.com; x-originating-ip: [180.161.88.205] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: c4303ec1-b8ea-4c03-ac6d-08d788887400 X-MS-TrafficTypeDiagnostic: VI1PR08MB3727:|VI1PR08MB3727:|AM0PR08MB5092: x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true x-ms-oob-tlc-oobclassifiers: OLM:7219;OLM:7219; x-forefront-prvs: 0261CCEEDF X-Forefront-Antispam-Report-Untrusted: SFV:NSPM; SFS:(10009020)(4636009)(396003)(39860400002)(346002)(136003)(376002)(366004)(189003)(199004)(13464003)(478600001)(71200400001)(5660300002)(9686003)(76116006)(26005)(81166006)(53546011)(8936002)(33656002)(6506007)(7696005)(8676002)(81156014)(966005)(66946007)(86362001)(66446008)(55016002)(186003)(52536014)(66476007)(66556008)(64756008)(54906003)(4326008)(110136005)(316002)(2906002); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR08MB3727; H:VI1PR08MB5376.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: +hKz0oiFhlntyxkPU5EA3hHhbRo1jcxD3kQ618PD97oW2gP0vkUzJ1mTtD7S55DpPSVbLlkqUp7zSv5xI4pkwCCV5LoYCngLILwfgHG9sLAvVuAc7Q+yMM2a3Wb10Aq3oaVL+9szJy+oEPF5Dj2tFWe/urW+ujknamv5hUYg6L8/XAXf4Mblw+VErYxhoNZgTwoR5XbyLPJ0jPDHBahmaJIALPySdyuQqO9WRkN0Oi/bt21fJE3xVibWJQHbQ8hzROKY6ax8zHUGaM+vOyn+scr3YLthxqHOpvTCEJLK/gjzLw0CbNu94u2P1xcHT8HxNs5eDhGrYKERwOHEsOe0/D2zO+OdVsK4t7Y0OuXbUX03E7MxNj//sIq9ZexCzSKw9GgEGXV9E+dUoMXnkkFpFzZG0+ZoePRLlihjl2iVXiEY2d7pG2p4em1/xy2m0Mnte11IeXJMA/LB8Tj9d9t6ErDjD+2DVyENxo5X4pwCzaH0zI+xvq/O2BjXSvNanZobPGdZwO9weH/x827CIOMPV+VdLQjT+xYuowy91fOHnsM= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR08MB3727 Original-Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Gavin.Hu@arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DB5EUR03FT054.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; IPV:CAL; SCL:-1; CTRY:IE; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(396003)(376002)(39860400002)(136003)(346002)(199004)(13464003)(189003)(2906002)(450100002)(4326008)(54906003)(5660300002)(356004)(52536014)(26005)(53546011)(110136005)(7696005)(316002)(6506007)(26826003)(186003)(336012)(966005)(478600001)(33656002)(86362001)(9686003)(55016002)(81156014)(81166006)(8936002)(76130400001)(70586007)(8676002)(70206006); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR08MB5092; H:64aa7808-outbound-1.mta.getcheckrecipient.com; FPR:; SPF:Pass; LANG:en; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; A:1; MX:1; X-MS-Office365-Filtering-Correlation-Id-Prvs: cf671d97-563a-49b3-5481-08d788886c09 NoDisclaimer: True X-Forefront-PRVS: 0261CCEEDF X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4XHDQQQB+MDwrf7WisdPcLwN22ADVL6vtrgH4TudM+lyRp8BzvLX+Bi8KZFcHgM1OiPgrsTIzDVkP1oZ/zv5sGT6iqhvHRb2c+I69S1ZwTJlxjMJd48BwAKO/qu/ZyyMfVVFti5YNEeG1yd5surWuilhRr9hhxIV+I1mFZAGrzU8vCnS3ZDU9gKhWMUj5ScUJ/oRnV9OoH1rKkCk8O52xfpgCIFtWPdpmPmAtWsnDFSf890J/k+EoJx1yB8ydg26MMkt0AzCMCEcmLHER9Qm2p/1eQ3thF4+Bv073yjjMmXIidsyrP1ZNPSkjCJt/YKuKGLF3KDI1Dd3lvdm+Xj50zfd1sktJWCvVYX0mgtL5xpDCjdwsm1yjbjjFL06j946YFbdY7N7O9VQInlrQLF8cuCRffy9L0/kFElG2peddxffm4uSbv/Y5vyJlcciPWkHorv76Zbj/n2Zlp1RctBkHbS05kZaiFEiL4m5ZUlT+l3RRcY3Ug7FWPoOx8qv/3enphOcekKY7/DHPA3V6HjCTU8BYR/9CBTxhJZTUkcbsLs= X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Dec 2019 15:46:33.8624 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c4303ec1-b8ea-4c03-ac6d-08d788887400 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR08MB5092 Subject: Re: [dpdk-stable] [dpdk-dev] [PATCH v2] raw/ntb: fix write memory barrier issue X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Sender: "stable" Hi Xiaoyun, > -----Original Message----- > From: Li, Xiaoyun > Sent: Monday, December 23, 2019 5:36 PM > To: Gavin Hu ; Wu, Jingjing > Cc: dev@dpdk.org; Maslekar, Omkar ; > stable@dpdk.org; nd ; jerinj@marvell.com; Honnappa > Nagarahalli ; Richardson, Bruce > ; nd > Subject: RE: [dpdk-dev] [PATCH v2] raw/ntb: fix write memory barrier issu= e >=20 > Hi >=20 > Still, stability and correctness are much more important than performance= . > As I said, with WC can benefit more than 20X perf. Comparing to this bene= fit, > the difference between rte_wmb and rte_io_wmb is not that important. > And in my test, the performance is not that bad with rte_wmb especially w= ith > large packets which are the normal use cases. I agree 'sfence' is the correct barrier for WC, as it is a weak memory mode= l. Rte_io on x86 is a compiler barrier, that is not strong enough. =20 >=20 > BTW, I've searched linux kernel codes and don't see any NTB device on arm > platform. > So I don't think you need to consider the perf hurt to arm. Limited the discussion to NTB, I am fine, and since WC in not used for any = other NICs, that's ok. >=20 > Best Regards > Xiaoyun Li >=20 > > -----Original Message----- > > From: Gavin Hu [mailto:Gavin.Hu@arm.com] > > Sent: Monday, December 23, 2019 16:38 > > To: Li, Xiaoyun ; Wu, Jingjing > > > Cc: dev@dpdk.org; Maslekar, Omkar ; > > stable@dpdk.org; nd ; jerinj@marvell.com; Honnappa > > Nagarahalli ; Richardson, Bruce > > ; nd > > Subject: RE: [dpdk-dev] [PATCH v2] raw/ntb: fix write memory barrier is= sue > > > > Hi Xiaoyun, > > > > > -----Original Message----- > > > From: Li, Xiaoyun > > > Sent: Monday, December 23, 2019 3:52 PM > > > To: Gavin Hu ; Wu, Jingjing > > > Cc: dev@dpdk.org; Maslekar, Omkar ; > > > stable@dpdk.org; nd > > > Subject: RE: [dpdk-dev] [PATCH v2] raw/ntb: fix write memory barrier > > > issue > > > > > > Hi > > > I reconsidered and retested about this issue. > > > I still need to use rte_wmb instead of using rte_io_wmb. > > > > > > Because to achieve high performance, ntb needs to turn on WC(write > > > combining) feature. The perf difference with and without WC enabled i= s > > > more than 20X. > > > And when WC enabled, rte_io_wmb cannot make sure the instructions > are > > > in order only rte_wmb can make sure that. > > > > > > And in my retest, when sending 64 bytes packets, using rte_io_wmb wil= l > > > cause out-of-order issue and cause memory corruption on rx side. > > > And using rte_wmb is fine. > > That's true, as it is declared as 'write combine' region, even x86 is k= nown as > > strong ordered, it is the interconnect or PCI RC may do the reordering,= 'write > > combine', 'write coalescing', which caused this problem. > > IMO, rte_io_*mb barriers on x86 should be promoted to stronger is WC is > > involved(but that will sap performance for non-WC memories?). > > > https://code.dpdk.org/dpdk/latest/source/lib/librte_eal/common/include/ar > ch/ > > x86/rte_atomic.h#L78 > > > > Using rte_wmb will hurt performance for aarch64 also, as pci device > memory > > accesses to a single device are strongly ordered therefore the stronges= t > > rte_wmb is not necessary. > > > So I can only use v1 patch and suspend v2 patch in patchwork. > > > > > > Best Regards > > > Xiaoyun Li > > > > > > > -----Original Message----- > > > > From: Gavin Hu (Arm Technology China) [mailto:Gavin.Hu@arm.com] > > > > Sent: Monday, December 16, 2019 18:50 > > > > To: Li, Xiaoyun ; Wu, Jingjing > > > > > > > Cc: dev@dpdk.org; Maslekar, Omkar ; > > > > stable@dpdk.org; nd > > > > Subject: RE: [dpdk-dev] [PATCH v2] raw/ntb: fix write memory barrie= r > > > issue > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: dev On Behalf Of Xiaoyun Li > > > > > Sent: Monday, December 16, 2019 9:59 AM > > > > > To: jingjing.wu@intel.com > > > > > Cc: dev@dpdk.org; omkar.maslekar@intel.com; Xiaoyun Li > > > > > ; stable@dpdk.org > > > > > Subject: [dpdk-dev] [PATCH v2] raw/ntb: fix write memory barrier > > > > > issue > > > > > > > > > > All buffers and ring info should be written before tail register = update. > > > > > This patch relocates the write memory barrier before updating tai= l > > > > > register to avoid potential issues. > > > > > > > > > > Fixes: 11b5c7daf019 ("raw/ntb: add enqueue and dequeue functions"= ) > > > > > Cc: stable@dpdk.org > > > > > > > > > > Signed-off-by: Xiaoyun Li > > > > > --- > > > > > v2: > > > > > * Replaced rte_wmb with rte_io_wmb since rte_io_wmb is enough. > > > > > --- > > > > > drivers/raw/ntb/ntb.c | 4 ++-- > > > > > 1 file changed, 2 insertions(+), 2 deletions(-) > > > > > > > > > > diff --git a/drivers/raw/ntb/ntb.c b/drivers/raw/ntb/ntb.c index > > > > > ad7f6abfd..c7de86f36 100644 > > > > > --- a/drivers/raw/ntb/ntb.c > > > > > +++ b/drivers/raw/ntb/ntb.c > > > > > @@ -683,8 +683,8 @@ ntb_enqueue_bufs(struct rte_rawdev *dev, > > > > > sizeof(struct ntb_used) * nb1); > > > > > rte_memcpy(txq->tx_used_ring, tx_used + nb1, > > > > > sizeof(struct ntb_used) * nb2); > > > > > + rte_io_wmb(); > > > > As both txq->tx_used_ring and *txq->used_cnt are physically reside > > > > in the > > > PCI > > > > device side, rte_io_wmb is correct to ensure the ordering. > > > > > > > > > *txq->used_cnt =3D txq->last_used; > > > > > - rte_wmb(); > > > > > > > > > > /* update queue stats */ > > > > > hw->ntb_xstats[NTB_TX_BYTES_ID + off] +=3D bytes; @@ - > > > 789,8 > > > > +789,8 @@ > > > > > ntb_dequeue_bufs(struct rte_rawdev *dev, > > > > > sizeof(struct ntb_desc) * nb1); > > > > > rte_memcpy(rxq->rx_desc_ring, rx_desc + nb1, > > > > > sizeof(struct ntb_desc) * nb2); > > > > > + rte_io_wmb(); > > > > > *rxq->avail_cnt =3D rxq->last_avail; > > > > > - rte_wmb(); > > > > > > > > > > /* update queue stats */ > > > > > off =3D NTB_XSTATS_NUM * ((size_t)context + 1); > > > > > -- > > > > > 2.17.1 > > > > > > > > Reviewed-by: Gavin Hu