From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 627A3A2EEB for ; Wed, 11 Sep 2019 12:02:43 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 4C16C1E8B6; Wed, 11 Sep 2019 12:02:42 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by dpdk.org (Postfix) with ESMTP id D79B31E8B3 for ; Wed, 11 Sep 2019 12:02:39 +0200 (CEST) X-Amp-Result: UNSCANNABLE X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga106.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Sep 2019 03:02:38 -0700 X-IronPort-AV: E=Sophos;i="5.64,492,1559545200"; d="scan'208";a="175604225" Received: from bricha3-mobl.ger.corp.intel.com ([10.237.221.46]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Sep 2019 03:02:35 -0700 Date: Wed, 11 Sep 2019 11:02:32 +0100 From: Bruce Richardson To: "Gavin Hu (Arm Technology China)" Cc: "Liu, Yong" , "Wang, Yinan" , Maxime Coquelin , "Joyce Kong (Arm Technology China)" , "dev@dpdk.org" , nd , "Bie, Tiwei" , "Wang, Zhihong" , "amorenoz@redhat.com" , "Wang, Xiao W" , "jfreimann@redhat.com" , Honnappa Nagarahalli , Steve Capper Message-ID: <20190911100232.GA1908@bricha3-MOBL.ger.corp.intel.com> References: <1568020491-52462-1-git-send-email-joyce.kong@arm.com> <1568020491-52462-2-git-send-email-joyce.kong@arm.com> <86228AFD5BCD8E4EBFD2B90117B5E81E633A0F03@SHSMSX103.ccr.corp.intel.com> <86228AFD5BCD8E4EBFD2B90117B5E81E633A180D@SHSMSX103.ccr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.11.4 (2019-03-13) Subject: Re: [dpdk-dev] [PATCH v3 1/2] virtio: one way barrier for packed vring desc avail flags X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Wed, Sep 11, 2019 at 08:32:16AM +0000, Gavin Hu (Arm Technology China) wrote: > Thanks Marvin, my inline comments. > > > -----Original Message----- > > From: Liu, Yong > > Sent: Wednesday, September 11, 2019 2:30 PM > > To: Gavin Hu (Arm Technology China) ; Wang, Yinan > > ; Maxime Coquelin ; > > Joyce Kong (Arm Technology China) ; dev@dpdk.org > > Cc: nd ; Bie, Tiwei ; Wang, Zhihong > > ; amorenoz@redhat.com; Wang, Xiao W > > ; jfreimann@redhat.com; Honnappa Nagarahalli > > ; Steve Capper > > Subject: RE: [dpdk-dev] [PATCH v3 1/2] virtio: one way barrier for packed vring > > desc avail flags > > > > Thanks Gavin, my answers are inline. > > > > > -----Original Message----- > > > From: Gavin Hu (Arm Technology China) [mailto:Gavin.Hu@arm.com] > > > Sent: Wednesday, September 11, 2019 11:35 AM > > > To: Liu, Yong ; Wang, Yinan ; > > > Maxime Coquelin ; Joyce Kong (Arm > > Technology > > > China) ; dev@dpdk.org > > > Cc: nd ; Bie, Tiwei ; Wang, Zhihong > > > ; amorenoz@redhat.com; Wang, Xiao W > > > ; jfreimann@redhat.com; Honnappa Nagarahalli > > > ; Steve Capper > > > Subject: RE: [dpdk-dev] [PATCH v3 1/2] virtio: one way barrier for packed > > > vring desc avail flags > > > > > > Hi Marvin, > > > > > > Thanks for your answers, one more question for x86: > > > 1. For CIO memory alone or MMIO memory(eg PCI BAR) alone, the compiler > > > barrier is enough to keep ordering, that's why both rte_io_mb and > > > rte_cio_mb are defined as compiler barriers, right? > > > > Yes, that's right for x86. > > > > > 2. How about the ordering of interleaved CIO and MMIO accesses, for > > example, > > > a young store to MMIO can be reordered before an older store to CIO? CIO > > > may be faster than devices, but store buffers or caching may cause the CIO > > > update not visible to the device(in a common doorbell case)? > > > > > > > There's always one kind of cache coherent engine in x86 uncore sub-system. > > When CIO write instruction was retried, data will be in CPU LLC. > > When device doing inbound read, request will go to cache engine first and > > then check memory state and retrieve latest value. > I understand your words that the cache coherent engine is working like a hub/coordinator/arbiter for all the accesses to three types of memory: 1 - normal memory, 2 - CIO memory, 3 - MMIO memory, and the ordering behaviors are no different? > Then in what scenarios mfence/sfence/lfence should be used? Maybe just mfence is enough to keep orderings of store/load(which is the only one might reordered on x86)? > > The fence types needed will depend on the memory types used, for example, any memory mapped as write-combining will have different behaviour and need different fences to the regular write-back memory we are most familiar with. For the situations we deal with in DPDK, for regular memory writes and MMIO writes, reads won't be reordered with other reads, and writes won't be reordered with other writes, so therefore, as you point out, the mfence instruction is only rarely needed, and barriers to prevent compiler reordering are sufficient in nearly all cases. /Bruce