From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 93E35A2EEB for ; Thu, 12 Sep 2019 10:21:20 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0B9071BEE4; Thu, 12 Sep 2019 10:21:19 +0200 (CEST) Received: from EUR02-AM5-obe.outbound.protection.outlook.com (mail-eopbgr00078.outbound.protection.outlook.com [40.107.0.78]) by dpdk.org (Postfix) with ESMTP id 625FC2C08 for ; Thu, 12 Sep 2019 10:21:17 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=y7oPOqC2cGqHWqEDLyq/B445iw0fmRLv2ZSL/kQ3OcI=; b=WMCgTxvyuLuvEMpTU+DvBpOWfzokFwOWrJRvxvSbkEXjiluIQnhU03Wnm1wBDlzV6H5o9zWdK1Wv9M3ViVpVOJ/o+Rez1N6Ke0COTZyRr98cT5qvbXOvnHciIqFr6uAArcIfc6HedxPMfJhbYPw8NZRlbi4SrYzBkiK02ahPDsc= Received: from VI1PR08CA0158.eurprd08.prod.outlook.com (2603:10a6:800:d1::12) by AM0PR08MB4195.eurprd08.prod.outlook.com (2603:10a6:208:131::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2263.17; Thu, 12 Sep 2019 08:21:15 +0000 Received: from AM5EUR03FT028.eop-EUR03.prod.protection.outlook.com (2a01:111:f400:7e08::206) by VI1PR08CA0158.outlook.office365.com (2603:10a6:800:d1::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2263.13 via Frontend Transport; Thu, 12 Sep 2019 08:21:15 +0000 Authentication-Results: spf=temperror (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=none action=none header.from=arm.com; Received-SPF: TempError (protection.outlook.com: error in processing during lookup of arm.com: DNS Timeout) Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM5EUR03FT028.mail.protection.outlook.com (10.152.16.118) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2263.14 via Frontend Transport; Thu, 12 Sep 2019 08:21:13 +0000 Received: ("Tessian outbound 09ff1df8404a:v30"); Thu, 12 Sep 2019 08:21:11 +0000 X-CR-MTA-TID: 64aa7808 Received: from a26761fca7ae.2 (cr-mta-lb-1.cr-mta-net [104.47.8.55]) by 64aa7808-outbound-1.mta.getcheckrecipient.com id 9B0241DF-DE09-4480-B76F-1D41B798AACA.1; Thu, 12 Sep 2019 08:21:06 +0000 Received: from EUR03-AM5-obe.outbound.protection.outlook.com (mail-am5eur03lp2055.outbound.protection.outlook.com [104.47.8.55]) by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id a26761fca7ae.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Thu, 12 Sep 2019 08:21:06 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=W+eL086TB5AZv0yyffP29cUDa5Qk4Y1D7hJChIGm/+zwpPaTiwfuZL/q29gMTOmW3h/SYNrJGBfLAML84z3khCS85IqWwL5z8+QlSFp2OjEYbejys30Q/ODeO8nUed2UVuprRzEPe312EKC/8Uo8bcP2ybT9IxcjfotarMAArveCeuMo2NTEDgDu/h+S2+r4+hu9ieYkGgj2VCuwclOVgRJKTZdHkNsISQ8etgytmxD1lBDf0Ns4FzQILfM+wiJbc+ewyrfyYV8s3/YkxzKT/9Sx4HvNO0HGWXQZmSTbU4/SbtSAAEapXw7LroxQywXTWzi2wxUWAy4DhrqsMM2DDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=y7oPOqC2cGqHWqEDLyq/B445iw0fmRLv2ZSL/kQ3OcI=; b=KZ/MHJuDJBXRc8lsYs8GHHx/b5rJ+exFPwf0bEprk0tao75HnFE2jD3si/iMmjHU2BHigMuA3cln8z0LKPotJ8bBklkircihqYLMomnS5I37PLk9WVa4H4sNl7jnmKBkoNmXfALh3/dspQxu/JvofdL1i/PQJuixhmXaOxfsOQ/1by94TYmN3NOBiLS4cnCkkjseqCyvJdooljXpO7WiBBQ7C/0wdFj4W22Avm/9UK1nlFGYECU9fRWHGnSojnnUkm9Cfb1LGT6U9HXNOJGQ4fxCMcyV/BiKxzdkfqYcD+auERMYePhMKa/AxBqjlKmlSWNKoKVyQJpqB20wmFQ8Ew== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=y7oPOqC2cGqHWqEDLyq/B445iw0fmRLv2ZSL/kQ3OcI=; b=WMCgTxvyuLuvEMpTU+DvBpOWfzokFwOWrJRvxvSbkEXjiluIQnhU03Wnm1wBDlzV6H5o9zWdK1Wv9M3ViVpVOJ/o+Rez1N6Ke0COTZyRr98cT5qvbXOvnHciIqFr6uAArcIfc6HedxPMfJhbYPw8NZRlbi4SrYzBkiK02ahPDsc= Received: from VI1PR08MB5376.eurprd08.prod.outlook.com (52.133.244.200) by VI1PR08MB2974.eurprd08.prod.outlook.com (52.133.14.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2263.15; Thu, 12 Sep 2019 08:21:04 +0000 Received: from VI1PR08MB5376.eurprd08.prod.outlook.com ([fe80::c09a:49d3:eab2:f707]) by VI1PR08MB5376.eurprd08.prod.outlook.com ([fe80::c09a:49d3:eab2:f707%3]) with mapi id 15.20.2241.022; Thu, 12 Sep 2019 08:21:04 +0000 From: "Gavin Hu (Arm Technology China)" To: Bruce Richardson CC: "Liu, Yong" , "Wang, Yinan" , Maxime Coquelin , "Joyce Kong (Arm Technology China)" , "dev@dpdk.org" , nd , "Bie, Tiwei" , "Wang, Zhihong" , "amorenoz@redhat.com" , "Wang, Xiao W" , "jfreimann@redhat.com" , Honnappa Nagarahalli , Steve Capper Thread-Topic: [dpdk-dev] [PATCH v3 1/2] virtio: one way barrier for packed vring desc avail flags Thread-Index: AQHVZu8jWs4VS3EgzUyEHCO+eMCv8acjH7eAgAEpTQCAAFrvcIABIlAAgAAMECCAADRqAIAAHiDAgAAdUgCAAXVloA== Date: Thu, 12 Sep 2019 08:21:04 +0000 Message-ID: References: <1568020491-52462-1-git-send-email-joyce.kong@arm.com> <1568020491-52462-2-git-send-email-joyce.kong@arm.com> <86228AFD5BCD8E4EBFD2B90117B5E81E633A0F03@SHSMSX103.ccr.corp.intel.com> <86228AFD5BCD8E4EBFD2B90117B5E81E633A180D@SHSMSX103.ccr.corp.intel.com> <20190911100232.GA1908@bricha3-MOBL.ger.corp.intel.com> In-Reply-To: <20190911100232.GA1908@bricha3-MOBL.ger.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: e6b967cd-a2c1-4b54-91b2-971945e72949.0 x-checkrecipientchecked: true Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=Gavin.Hu@arm.com; x-originating-ip: [113.29.88.7] x-ms-publictraffictype: Email X-MS-Office365-Filtering-Correlation-Id: 9707986c-2d1e-47cc-5591-08d7375a2cca X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam-Untrusted: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(7168020)(4627221)(201703031133081)(201702281549075)(8990200)(5600166)(711020)(4605104)(1401327)(4618075)(2017052603328)(7193020); SRVR:VI1PR08MB2974; X-MS-TrafficTypeDiagnostic: VI1PR08MB2974:|VI1PR08MB2974:|AM0PR08MB4195: x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true x-ms-oob-tlc-oobclassifiers: OLM:8882;OLM:8882; x-forefront-prvs: 01583E185C X-Forefront-Antispam-Report-Untrusted: SFV:NSPM; SFS:(10009020)(4636009)(396003)(346002)(376002)(39860400002)(366004)(136003)(13464003)(199004)(189003)(14454004)(5660300002)(8936002)(6246003)(53546011)(186003)(102836004)(55236004)(486006)(25786009)(55016002)(9686003)(26005)(33656002)(7696005)(305945005)(7736002)(76176011)(99286004)(476003)(74316002)(52536014)(66446008)(64756008)(76116006)(66476007)(66556008)(14444005)(256004)(66946007)(6116002)(316002)(86362001)(71200400001)(7416002)(71190400001)(229853002)(66066001)(6436002)(6916009)(6506007)(8676002)(508600001)(81166006)(81156014)(53936002)(4326008)(11346002)(446003)(3846002)(2906002)(54906003); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR08MB2974; H:VI1PR08MB5376.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Message-Info-Original: ZZQWrK/wXYmShNXQbEJHxyS5vCOyT64rcQVkMF49gtszcbOXioM2EFAMvSlEzvsMHTaDDIyzXP1cQaRIpRqkRBdSWbh8PtlvLkTWBHlElCxtGLZ1bChI6FnDEmewk1mFLn1vhs4M7B1+t1D7oUBoCTW5ADp6nann4lrzmRnRpsUSjNxMytq4CLATZf1VRqGMkScbBAdSOaxgDv2B0H4W8l+XXYOgWBZX/57EbgMcprWzEGTDwmkn+ZJk9PIKvsZMj0lfxSq1nsT0lwHIDW/SJClC0n4SzD12AEX3kJXXh+3cqwboyLquSx7jVhXprhpiE0KGLf+VS5QOkbt0blZBVW2Fk77tcFS17VYa8jud03OpDyUuxAex6fFrOnPnY6SKm/XKx+OG3M+7zggYj/4iUfJ7+y6rpnlpnDeUquv3nDE= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR08MB2974 Original-Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Gavin.Hu@arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM5EUR03FT028.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; IPV:CAL; SCL:-1; CTRY:IE; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(136003)(346002)(39860400002)(396003)(376002)(13464003)(189003)(199004)(46406003)(23726003)(229853002)(25786009)(6116002)(3846002)(97756001)(6862004)(9686003)(22756006)(6246003)(126002)(476003)(55016002)(14444005)(4326008)(5660300002)(50466002)(86362001)(356004)(99286004)(47776003)(336012)(36906005)(316002)(54906003)(11346002)(446003)(7736002)(63350400001)(102836004)(53546011)(7696005)(6506007)(14454004)(76176011)(186003)(74316002)(26005)(486006)(305945005)(26826003)(33656002)(508600001)(81156014)(8746002)(8936002)(2906002)(8676002)(81166006)(52536014)(76130400001)(70586007)(70206006)(66066001); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR08MB4195; H:64aa7808-outbound-1.mta.getcheckrecipient.com; FPR:; SPF:TempError; LANG:en; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; MX:1; A:1; X-MS-Office365-Filtering-Correlation-Id-Prvs: 167e76b2-30e2-4701-607f-08d7375a278d X-Microsoft-Antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600166)(710020)(711020)(4605104)(1401327)(4618075)(2017052603328)(7193020); SRVR:AM0PR08MB4195; NoDisclaimer: True X-Forefront-PRVS: 01583E185C X-Microsoft-Antispam-Message-Info: wJd0y610RQ5bcDvva1gQzkQxg0lXIlD2t/cabOTsJSU+KPgSYeu/M4HyUYY/CDIGYRPhfBhcAVzkGUyoN3yzBWlut91uaZ8UAPTMvb707Fyq0OhYKrLx1ONzwBxHhWHLBtOCShHa1H33bOOZzUnQV1WHMQpcKSL9OgavVQadFGepvg1JCJJ0OVXpQfVx3eGyZS8wEHoBaHFgFt8kPkEcgdAx7nPYugCccCFfbhmhU9kdDIWj0RzOpDOAviQh+SNwtK7ZIBzPtz2FZuPm6k2k9oAHEZH5fw7SgJbxK32osZ4e3rPECP9s+AZ8t+NfF6yprr9vJyxnuPhzT2/Ri+YIqUYoEtQZtGwaiAD+pN7FZ0F675zAzsSmLXGmMgLysfds+H0RRQSaIyIY4R5L6XipFeM6tcQEZ4iuCn1FOKP5kzM= X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Sep 2019 08:21:13.3696 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9707986c-2d1e-47cc-5591-08d7375a2cca X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR08MB4195 Subject: Re: [dpdk-dev] [PATCH v3 1/2] virtio: one way barrier for packed vring desc avail flags X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Bruce, > -----Original Message----- > From: Bruce Richardson > Sent: Wednesday, September 11, 2019 6:03 PM > To: Gavin Hu (Arm Technology China) > Cc: Liu, Yong ; Wang, Yinan ; > Maxime Coquelin ; Joyce Kong (Arm > Technology China) ; dev@dpdk.org; nd > ; Bie, Tiwei ; Wang, Zhihong > ; amorenoz@redhat.com; Wang, Xiao W > ; jfreimann@redhat.com; Honnappa Nagarahalli > ; Steve Capper > Subject: Re: [dpdk-dev] [PATCH v3 1/2] virtio: one way barrier for packed= vring > desc avail flags >=20 > On Wed, Sep 11, 2019 at 08:32:16AM +0000, Gavin Hu (Arm Technology China) > wrote: > > Thanks Marvin, my inline comments. > > > > > -----Original Message----- > > > From: Liu, Yong > > > Sent: Wednesday, September 11, 2019 2:30 PM > > > To: Gavin Hu (Arm Technology China) ; Wang, Yinan > > > ; Maxime Coquelin > ; > > > Joyce Kong (Arm Technology China) ; > dev@dpdk.org > > > Cc: nd ; Bie, Tiwei ; Wang, Zhihong > > > ; amorenoz@redhat.com; Wang, Xiao W > > > ; jfreimann@redhat.com; Honnappa Nagarahalli > > > ; Steve Capper > > > > Subject: RE: [dpdk-dev] [PATCH v3 1/2] virtio: one way barrier for pa= cked > vring > > > desc avail flags > > > > > > Thanks Gavin, my answers are inline. > > > > > > > -----Original Message----- > > > > From: Gavin Hu (Arm Technology China) [mailto:Gavin.Hu@arm.com] > > > > Sent: Wednesday, September 11, 2019 11:35 AM > > > > To: Liu, Yong ; Wang, Yinan > ; > > > > Maxime Coquelin ; Joyce Kong (Arm > > > Technology > > > > China) ; dev@dpdk.org > > > > Cc: nd ; Bie, Tiwei ; Wang, > Zhihong > > > > ; amorenoz@redhat.com; Wang, Xiao W > > > > ; jfreimann@redhat.com; Honnappa > Nagarahalli > > > > ; Steve Capper > > > > > Subject: RE: [dpdk-dev] [PATCH v3 1/2] virtio: one way barrier for = packed > > > > vring desc avail flags > > > > > > > > Hi Marvin, > > > > > > > > Thanks for your answers, one more question for x86: > > > > 1. For CIO memory alone or MMIO memory(eg PCI BAR) alone, the > compiler > > > > barrier is enough to keep ordering, that's why both rte_io_mb and > > > > rte_cio_mb are defined as compiler barriers, right? > > > > > > Yes, that's right for x86. > > > > > > > 2. How about the ordering of interleaved CIO and MMIO accesses, for > > > example, > > > > a young store to MMIO can be reordered before an older store to CIO= ? > CIO > > > > may be faster than devices, but store buffers or caching may cause = the > CIO > > > > update not visible to the device(in a common doorbell case)? > > > > > > > > > > There's always one kind of cache coherent engine in x86 uncore sub- > system. > > > When CIO write instruction was retried, data will be in CPU LLC. > > > When device doing inbound read, request will go to cache engine first= and > > > then check memory state and retrieve latest value. > > I understand your words that the cache coherent engine is working like = a > hub/coordinator/arbiter for all the accesses to three types of memory: 1 = - > normal memory, 2 - CIO memory, 3 - MMIO memory, and the ordering > behaviors are no different? > > Then in what scenarios mfence/sfence/lfence should be used? Maybe just > mfence is enough to keep orderings of store/load(which is the only one mi= ght > reordered on x86)? > > > >=20 > The fence types needed will depend on the memory types used, for example, > any memory mapped as write-combining will have different behaviour and > need different fences to the regular write-back memory we are most famili= ar > with. For the situations we deal with in DPDK, for regular memory writes > and MMIO writes, reads won't be reordered with other reads, and writes > won't be reordered with other writes, so therefore, as you point out, the > mfence instruction is only rarely needed, and barriers to prevent compile= r > reordering are sufficient in nearly all cases. Thanks for your explanation about the barriers on x86, it is really helpful= for us to optimize PMDs for aarch64 by using less restrictive barriers while not b= reaking x86 platforms. >=20 > /Bruce