From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 102D3A04B3 for ; Mon, 23 Dec 2019 09:38:24 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B040A1BE51; Mon, 23 Dec 2019 09:38:23 +0100 (CET) Received: from EUR04-VI1-obe.outbound.protection.outlook.com (mail-eopbgr80054.outbound.protection.outlook.com [40.107.8.54]) by dpdk.org (Postfix) with ESMTP id ED0852BA3; Mon, 23 Dec 2019 09:38:19 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2vsWHq/qKAanX8Q9iFqGdrv3CeEDTbWm9kyzm7rArQo=; b=ocIluts2IdzWgiS7F/KpOgicrMZfOHwunMShmbGxMBbjO3xNSVeivlpFODxyWbCSL41QvWYA+XwVMuHCKm3veteiEG5SW3CrhDpA6DiQ5wnzbkg+QIlNy4S4/o9bwo1nrIhhD7/03bSaex/yCPnr+4uUx7+TGckVMQj0E62Dplo= Received: from VI1PR08CA0185.eurprd08.prod.outlook.com (2603:10a6:800:d2::15) by DBBPR08MB4298.eurprd08.prod.outlook.com (2603:10a6:10:cb::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2559.17; Mon, 23 Dec 2019 08:38:18 +0000 Received: from AM5EUR03FT059.eop-EUR03.prod.protection.outlook.com (2a01:111:f400:7e08::202) by VI1PR08CA0185.outlook.office365.com (2603:10a6:800:d2::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2559.14 via Frontend Transport; Mon, 23 Dec 2019 08:38:18 +0000 Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=bestguesspass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM5EUR03FT059.mail.protection.outlook.com (10.152.17.193) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2559.14 via Frontend Transport; Mon, 23 Dec 2019 08:38:18 +0000 Received: ("Tessian outbound 1da651c29646:v40"); Mon, 23 Dec 2019 08:38:17 +0000 X-CR-MTA-TID: 64aa7808 Received: from e62005be5581.2 by 64aa7808-outbound-1.mta.getcheckrecipient.com id BBC7FD58-D2D0-4F6F-8129-1AC1F3F7570F.1; Mon, 23 Dec 2019 08:38:12 +0000 Received: from EUR02-HE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id e62005be5581.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Mon, 23 Dec 2019 08:38:12 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=aeThU1De318BYzeQLq60Lb24r5u5uvjX4GxqPusqNWN9vNZMt3lDxA6aHg04oThMqMcV88UnmtpGzHWnpOEu5wU2U7oV+JPfke/V8VoWpWcjJ5mhtDQH/C21BD000l4ArVSTqMf1CcgFqsSM3XDXIWWjonlva0LM8eYDXl+2+n/nbvYmXpZxlXag+udAKFHr8ikcW72UCedcQ7Cte2wTwMo0lcQZFWfuAb9N1J4PXWPwe3XOXUcpLKfs4eS1slCoQAb0CZOmKsplKodpiQToZD7+0V7oZR/4oZKgA0bOrWXT0Mnc0Z43DXm7BRhZmX1jSdt2+md9lGzZe/mIG0Se7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2vsWHq/qKAanX8Q9iFqGdrv3CeEDTbWm9kyzm7rArQo=; b=UdUfboecJ2ERkkVyoQCZq+vFPwEbG7mI5HEnXPyUqNa9XDPuLyUG+Nd23xEOKrfbwSjMrY/11+OF+YLDrT2TfT4opcrKg6zmIzuCG2YbRnjLXeecKMKxk30MiIPfVsRpFSQLSYHDP+BBp3dr7jEId0225CWPswnuqV8+hERYZ6pCPFjFb7RrRbtTsSakxLVlddRDfbJikCOc9BDrNOku10u+SnyD4OhuIcckHQ2C/1o/fR/W+7l0USZt10sy5AB40dml6OlAycvAkL+Po7tzQU5fsEcHgizc3lJX0RebprOzv8Fl/4ogsGctIdv65f195a3yQsdWKE9eeatDj2SvVQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2vsWHq/qKAanX8Q9iFqGdrv3CeEDTbWm9kyzm7rArQo=; b=ocIluts2IdzWgiS7F/KpOgicrMZfOHwunMShmbGxMBbjO3xNSVeivlpFODxyWbCSL41QvWYA+XwVMuHCKm3veteiEG5SW3CrhDpA6DiQ5wnzbkg+QIlNy4S4/o9bwo1nrIhhD7/03bSaex/yCPnr+4uUx7+TGckVMQj0E62Dplo= Received: from VI1PR08MB5376.eurprd08.prod.outlook.com (10.255.196.79) by VI1PR08MB2896.eurprd08.prod.outlook.com (10.170.239.152) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2559.14; Mon, 23 Dec 2019 08:38:09 +0000 Received: from VI1PR08MB5376.eurprd08.prod.outlook.com ([fe80::dde8:216f:6a0b:8cfb]) by VI1PR08MB5376.eurprd08.prod.outlook.com ([fe80::dde8:216f:6a0b:8cfb%7]) with mapi id 15.20.2559.017; Mon, 23 Dec 2019 08:38:09 +0000 From: Gavin Hu To: "Li, Xiaoyun" , "Wu, Jingjing" CC: "dev@dpdk.org" , "Maslekar, Omkar" , "stable@dpdk.org" , nd , "jerinj@marvell.com" , Honnappa Nagarahalli , "bruce.richardson@intel.com" , nd Thread-Topic: [dpdk-dev] [PATCH v2] raw/ntb: fix write memory barrier issue Thread-Index: AQHVs7Tni9/XHz+bM0yLOm9Kl+X1h6e8lRjwgArPOoCAAAdocA== Date: Mon, 23 Dec 2019 08:38:08 +0000 Message-ID: References: <20191204151916.12607-1-xiaoyun.li@intel.com> <20191216015854.28725-1-xiaoyun.li@intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 7e99563b-4c76-4211-9926-9409110dbe8f.0 x-checkrecipientchecked: true Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=Gavin.Hu@arm.com; x-originating-ip: [113.29.88.7] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: d685cfa6-5ced-4459-85d3-08d7878375b6 X-MS-TrafficTypeDiagnostic: VI1PR08MB2896:|VI1PR08MB2896:|DBBPR08MB4298: x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true x-ms-oob-tlc-oobclassifiers: OLM:6790;OLM:6790; x-forefront-prvs: 0260457E99 X-Forefront-Antispam-Report-Untrusted: SFV:NSPM; SFS:(10009020)(4636009)(366004)(136003)(39860400002)(376002)(396003)(346002)(199004)(189003)(13464003)(966005)(52536014)(9686003)(26005)(64756008)(186003)(66946007)(33656002)(66446008)(66476007)(66556008)(76116006)(55016002)(110136005)(81156014)(54906003)(81166006)(53546011)(6506007)(5660300002)(316002)(8676002)(7696005)(2906002)(86362001)(55236004)(8936002)(71200400001)(4326008)(478600001); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR08MB2896; H:VI1PR08MB5376.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: n3/rOhOrNHUA0rH7H0sWQ+4k0bv9bo0m5XzaYlM6Xn5CQdHhfPTitQKBlZOy8kq+kPH4Iai5G7I4vzPlJ/VRthPjB1qlcpPi23RV41GYuKwRq8BaHBz6HBOP1VnEzZVK3YdW5pkn/5lgRyTHSt7H//kQNxnXugJ0quyyr9wWorSgD5rmFIR4S4IYpF8OrK9VI+MnJpDN4MEJWFAzTBQbhZ4jdk4LczcpCwTJbKGy/4VNH0lFaYyBvaYFJ/RBtkXJXvxyI0qOWz3+XHmxHb6f+hWYE1A/3HMhHHpyRMgEIfvZRsTy3lcHCeBmeq/OWShFHR9twRS3a90yHSqk6/yVlySkagcZD5gQZz6FvKRJqiOyPsG06yw8bO8z2sZ7746B4Z9wiNcx0vBk2MTD2HT+POidhxGmMM/aUaOljQgg67tBbw+DzKTXW5KPeOF3oGdb2PscBRMnYH04Oh5Ru0IRGkpibXn2dlTbaWyY/eAOIFCwIqur2/+hGDeVRi+zy3nsH/y8CtOSQWwWmrIR0PXVuEholODqpYwxH9CwbFxzgNE= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR08MB2896 Original-Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Gavin.Hu@arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM5EUR03FT059.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; IPV:CAL; SCL:-1; CTRY:IE; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(39860400002)(136003)(396003)(346002)(376002)(199004)(189003)(13464003)(5660300002)(450100002)(86362001)(110136005)(70206006)(54906003)(356004)(9686003)(55016002)(33656002)(70586007)(336012)(52536014)(8936002)(4326008)(8676002)(81166006)(81156014)(76130400001)(316002)(966005)(26826003)(2906002)(26005)(7696005)(36906005)(53546011)(186003)(6506007)(478600001); DIR:OUT; SFP:1101; SCL:1; SRVR:DBBPR08MB4298; H:64aa7808-outbound-1.mta.getcheckrecipient.com; FPR:; SPF:Pass; LANG:en; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; A:1; MX:1; X-MS-Office365-Filtering-Correlation-Id-Prvs: d3196a2c-e0be-45dc-3230-08d78783705e NoDisclaimer: True X-Forefront-PRVS: 0260457E99 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: oRWVWdaAdTfcncD4AiyiH8w6Xm4P271vBViTmrtb4lMc9TJG8FaqjhHY7sl1ZeyCrqtEzCQd28KwipGIZ4ky2D4MUgONykaYiRT2Hjajsx3UdOaT3nPM/mWGpElo66LZ+lIpNp2vVthVpmDwF4HnghdzN7XZY/t/HsTOCkIbqcMLv/o4oE+cUSZxIqJKTl3NaZqYciLYBP6I9/V5zsz0gma4/vMF7rUF4YYLJg5XlwZ5TklStabOwKr9Wj02G/XZAY4LtEkCj+xgZIxRYzkAh6zfTYFPSBQFnvyfB+puRQZcW8LXQNCqBV9qD2TLxB2jzQpkanR9g/6r0Bpev0zQfnuocdJjDTNZFQbN6l+o86C9ItBSYYGCwonFzOslRN0UAbt+tI+RSM+BYyrg2MHvZVlFOd6ssd37ptfaLxr6DwnQmrnK+QFY/hbevAT9VJE7+x/nzqLQ3LiCVIlNdqF0/zD4FeCTV7+ESti8NSbgwYHLmlzC+vqVYwDvifWgvOlRcbvaibhwCS8nj1fJk26q91tgL9nBSyFykJdfeMaJwKE= X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Dec 2019 08:38:18.0765 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d685cfa6-5ced-4459-85d3-08d7878375b6 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR08MB4298 Subject: Re: [dpdk-stable] [dpdk-dev] [PATCH v2] raw/ntb: fix write memory barrier issue X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Sender: "stable" Hi Xiaoyun, > -----Original Message----- > From: Li, Xiaoyun > Sent: Monday, December 23, 2019 3:52 PM > To: Gavin Hu ; Wu, Jingjing > Cc: dev@dpdk.org; Maslekar, Omkar ; > stable@dpdk.org; nd > Subject: RE: [dpdk-dev] [PATCH v2] raw/ntb: fix write memory barrier issu= e >=20 > Hi > I reconsidered and retested about this issue. > I still need to use rte_wmb instead of using rte_io_wmb. >=20 > Because to achieve high performance, ntb needs to turn on WC(write > combining) feature. The perf difference with and without WC enabled is > more than 20X. > And when WC enabled, rte_io_wmb cannot make sure the instructions are > in order only rte_wmb can make sure that. >=20 > And in my retest, when sending 64 bytes packets, using rte_io_wmb will > cause out-of-order issue and cause memory corruption on rx side. > And using rte_wmb is fine. That's true, as it is declared as 'write combine' region, even x86 is known= as strong ordered, it is the interconnect or PCI RC may do the reordering,= 'write combine', 'write coalescing', which caused this problem. IMO, rte_io_*mb barriers on x86 should be promoted to stronger is WC is inv= olved(but that will sap performance for non-WC memories?).=20 https://code.dpdk.org/dpdk/latest/source/lib/librte_eal/common/include/arch= /x86/rte_atomic.h#L78=20 Using rte_wmb will hurt performance for aarch64 also, as pci device memory = accesses to a single device are strongly ordered therefore the strongest rt= e_wmb is not necessary. =20 > So I can only use v1 patch and suspend v2 patch in patchwork. >=20 > Best Regards > Xiaoyun Li >=20 > > -----Original Message----- > > From: Gavin Hu (Arm Technology China) [mailto:Gavin.Hu@arm.com] > > Sent: Monday, December 16, 2019 18:50 > > To: Li, Xiaoyun ; Wu, Jingjing > > > Cc: dev@dpdk.org; Maslekar, Omkar ; > > stable@dpdk.org; nd > > Subject: RE: [dpdk-dev] [PATCH v2] raw/ntb: fix write memory barrier > issue > > > > > > > > > -----Original Message----- > > > From: dev On Behalf Of Xiaoyun Li > > > Sent: Monday, December 16, 2019 9:59 AM > > > To: jingjing.wu@intel.com > > > Cc: dev@dpdk.org; omkar.maslekar@intel.com; Xiaoyun Li > > > ; stable@dpdk.org > > > Subject: [dpdk-dev] [PATCH v2] raw/ntb: fix write memory barrier issu= e > > > > > > All buffers and ring info should be written before tail register upda= te. > > > This patch relocates the write memory barrier before updating tail > > > register to avoid potential issues. > > > > > > Fixes: 11b5c7daf019 ("raw/ntb: add enqueue and dequeue functions") > > > Cc: stable@dpdk.org > > > > > > Signed-off-by: Xiaoyun Li > > > --- > > > v2: > > > * Replaced rte_wmb with rte_io_wmb since rte_io_wmb is enough. > > > --- > > > drivers/raw/ntb/ntb.c | 4 ++-- > > > 1 file changed, 2 insertions(+), 2 deletions(-) > > > > > > diff --git a/drivers/raw/ntb/ntb.c b/drivers/raw/ntb/ntb.c index > > > ad7f6abfd..c7de86f36 100644 > > > --- a/drivers/raw/ntb/ntb.c > > > +++ b/drivers/raw/ntb/ntb.c > > > @@ -683,8 +683,8 @@ ntb_enqueue_bufs(struct rte_rawdev *dev, > > > sizeof(struct ntb_used) * nb1); > > > rte_memcpy(txq->tx_used_ring, tx_used + nb1, > > > sizeof(struct ntb_used) * nb2); > > > + rte_io_wmb(); > > As both txq->tx_used_ring and *txq->used_cnt are physically reside in t= he > PCI > > device side, rte_io_wmb is correct to ensure the ordering. > > > > > *txq->used_cnt =3D txq->last_used; > > > - rte_wmb(); > > > > > > /* update queue stats */ > > > hw->ntb_xstats[NTB_TX_BYTES_ID + off] +=3D bytes; @@ - > 789,8 > > +789,8 @@ > > > ntb_dequeue_bufs(struct rte_rawdev *dev, > > > sizeof(struct ntb_desc) * nb1); > > > rte_memcpy(rxq->rx_desc_ring, rx_desc + nb1, > > > sizeof(struct ntb_desc) * nb2); > > > + rte_io_wmb(); > > > *rxq->avail_cnt =3D rxq->last_avail; > > > - rte_wmb(); > > > > > > /* update queue stats */ > > > off =3D NTB_XSTATS_NUM * ((size_t)context + 1); > > > -- > > > 2.17.1 > > > > Reviewed-by: Gavin Hu