From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f180.google.com (mail-wi0-f180.google.com [209.85.212.180]) by dpdk.org (Postfix) with ESMTP id 32379377A for ; Tue, 7 Apr 2015 16:22:43 +0200 (CEST) Received: by wiun10 with SMTP id n10so20815933wiu.1 for ; Tue, 07 Apr 2015 07:22:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=9yvxm2q81LktIDWMWKn5b6eYKHegHRfnHYQkpD+o7BM=; b=iMm9dAytZEh70braafseVBDDpyJDs1OYGTZeSvI5WRDBJK8so4DdIqofecmCXoG2P8 A1wkA0+V2Rsfh03riXaw8+2HRd8XMApiHTFuhr7jQcbOY3Y33LBakNAbfA3fyytG3wdP GnTI9p+QcIhAmm986wCttzegRU20XPJ3J/KZSd0O0dmcfjTmdb2Xa4BKpgeChFD/EAOM JFE+ce+O0j/5D9r3JTZoo+EFqdy+IZXu/lhPbABugtZn3+oiGNnXcickQIfNeGgwBSV8 4Ge6TmhhyEdTQJ2RM9+Nsim8etnQKvg+pldRqOrS/kOs8nJtvYAxhAbRSX4FbBWHKHPx argw== MIME-Version: 1.0 X-Received: by 10.181.29.170 with SMTP id jx10mr5101528wid.29.1428416563033; Tue, 07 Apr 2015 07:22:43 -0700 (PDT) Sender: lukego@gmail.com Received: by 10.27.217.3 with HTTP; Tue, 7 Apr 2015 07:22:42 -0700 (PDT) In-Reply-To: References: <20150127160126.GA10651@redhat.com> Date: Tue, 7 Apr 2015 16:22:42 +0200 X-Google-Sender-Auth: 2vSeJmhyr_YaOq1Fe8J551fmEmw Message-ID: From: Luke Gorrie To: "snabb-devel@googlegroups.com" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Cc: "dev@dpdk.org" , virtualization@lists.linux-foundation.org, VirtualOpenSystems Technical Team , "Michael S. Tsirkin" Subject: Re: [dpdk-dev] [snabb-devel] Re: memory barriers in virtq.lua? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Apr 2015 14:22:43 -0000 Hi Michael, I'm writing to follow up the previous discussion about memory barriers in virtio-net device implementations, and Cc'ing the DPDK list because I believe this is relevant to them too. First, thanks again for getting in touch and reviewing our code. I have now found a missed case where we *do* require a hardware memory barrier on x86 in our vhost/virtio-net device. That is when checking the interrupt suppression flag after updating used->idx. This is needed because x86 can reorder the write to used->idx after the read from avail->flags, and that causes the guest to see a stale value of used->idx after it toggles interrupt suppression. If I may spell out my mental model, for the sake of being corrected and/or as an example of how third party developers are reading and interpreting the Virtio-net spec: Relating this to Virtio 1.0, the most relevant section is 3.2.1 (Supplying Buffers to the Device) which calls for two "suitable memory barriers". The spec talks about these from the driver perspective, but they are both relevant to the device side too. The first barrier (write to descriptor table before write to used->idx) is implicit on x86 because writes by the same core are not reordered. This means that no explicit hardware barrier is needed. (A compiler barrier may be needed, however.) The second memory barrier (write to used->idx before reading avail->flags) is not implicit on x86 because stores are reordered after loads. So an explicit hardware memory barrier is needed. I hope that is a correct assessment of the situation. (Forgive my x86centricity, I am sure that seems very foreign to kernel hackers.) If this assessment is correct then the DPDK developers might also want to review librte_vhost/vhost_rxtx.c and consider adding a hardware memory barrier between writing used->idx and reading avail->flags. Cheers, -Luke P.S. I notice that the Linux virtio-net driver does not seem to tolerate spurious interrupts, even though the Virtio 1.0 spec requires this ("must"). On 3.13.11-ckt15 I see them trigger an "irq nobody cared" kernel log message and then the irq is disabled. If that sounds suspicious I can supply more information.