From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id 4144B2E41 for ; Wed, 8 Apr 2015 17:15:21 +0200 (CEST) Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga101.jf.intel.com with ESMTP; 08 Apr 2015 08:15:20 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.11,545,1422950400"; d="scan'208";a="677119897" Received: from kmsmsx152.gar.corp.intel.com ([172.21.73.87]) by orsmga001.jf.intel.com with ESMTP; 08 Apr 2015 08:15:20 -0700 Received: from shsmsx102.ccr.corp.intel.com (10.239.4.154) by KMSMSX152.gar.corp.intel.com (172.21.73.87) with Microsoft SMTP Server (TLS) id 14.3.224.2; Wed, 8 Apr 2015 23:15:17 +0800 Received: from shsmsx101.ccr.corp.intel.com ([169.254.1.24]) by shsmsx102.ccr.corp.intel.com ([169.254.2.223]) with mapi id 14.03.0224.002; Wed, 8 Apr 2015 23:15:16 +0800 From: "Xie, Huawei" To: Luke Gorrie , "snabb-devel@googlegroups.com" Thread-Topic: [dpdk-dev] [snabb-devel] Re: memory barriers in virtq.lua? Thread-Index: AdByDtEa7ppE54PwQlqb9NruVYVs9w== Date: Wed, 8 Apr 2015 15:15:16 +0000 Message-ID: References: <20150127160126.GA10651@redhat.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "dev@dpdk.org" , VirtualOpenSystems Technical Team , "Michael S. Tsirkin" , "virtualization@lists.linux-foundation.org" Subject: Re: [dpdk-dev] [snabb-devel] Re: memory barriers in virtq.lua? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Apr 2015 15:15:21 -0000 On 4/7/2015 10:23 PM, Luke Gorrie wrote:=0A= > Hi Michael,=0A= >=0A= > I'm writing to follow up the previous discussion about memory barriers in= =0A= > virtio-net device implementations, and Cc'ing the DPDK list because I=0A= > believe this is relevant to them too.=0A= >=0A= > First, thanks again for getting in touch and reviewing our code.=0A= >=0A= > I have now found a missed case where we *do* require a hardware memory=0A= > barrier on x86 in our vhost/virtio-net device. That is when checking the= =0A= > interrupt suppression flag after updating used->idx. This is needed becau= se=0A= > x86 can reorder the write to used->idx after the read from avail->flags,= =0A= > and that causes the guest to see a stale value of used->idx after it=0A= > toggles interrupt suppression.=0A= luke:=0A= 1. host read the flag. 2 guest toggles the flag 3.guest checks the used.=0A= 4. host update used.=0A= Is this your case?=0A= =0A= >=0A= > If I may spell out my mental model, for the sake of being corrected and/o= r=0A= > as an example of how third party developers are reading and interpreting= =0A= > the Virtio-net spec:=0A= >=0A= > Relating this to Virtio 1.0, the most relevant section is 3.2.1 (Supplyin= g=0A= > Buffers to the Device) which calls for two "suitable memory barriers". Th= e=0A= > spec talks about these from the driver perspective, but they are both=0A= > relevant to the device side too.=0A= >=0A= > The first barrier (write to descriptor table before write to used->idx) i= s=0A= > implicit on x86 because writes by the same core are not reordered. This= =0A= > means that no explicit hardware barrier is needed. (A compiler barrier ma= y=0A= > be needed, however.)=0A= >=0A= > The second memory barrier (write to used->idx before reading avail->flags= )=0A= > is not implicit on x86 because stores are reordered after loads. So an=0A= > explicit hardware memory barrier is needed.=0A= >=0A= > I hope that is a correct assessment of the situation. (Forgive my=0A= > x86centricity, I am sure that seems very foreign to kernel hackers.)=0A= >=0A= > If this assessment is correct then the DPDK developers might also want to= =0A= > review librte_vhost/vhost_rxtx.c and consider adding a hardware memory=0A= > barrier between writing used->idx and reading avail->flags.=0A= >=0A= > Cheers,=0A= > -Luke=0A= >=0A= > P.S. I notice that the Linux virtio-net driver does not seem to tolerate= =0A= > spurious interrupts, even though the Virtio 1.0 spec requires this=0A= > ("must"). On 3.13.11-ckt15 I see them trigger an "irq nobody cared" kerne= l=0A= > log message and then the irq is disabled. If that sounds suspicious I can= =0A= > supply more information.=0A= >=0A= >=0A= =0A=