DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Michael S. Tsirkin" <mst@redhat.com>
To: Luke Gorrie <luke@snabb.co>
Cc: "dev@dpdk.org" <dev@dpdk.org>,
	"snabb-devel@googlegroups.com" <snabb-devel@googlegroups.com>,
	VirtualOpenSystems Technical Team <tech@virtualopensystems.com>,
	virtualization@lists.linux-foundation.org
Subject: Re: [dpdk-dev] [snabb-devel] Re: memory barriers in virtq.lua?
Date: Tue, 7 Apr 2015 17:30:59 +0200	[thread overview]
Message-ID: <20150407172849-mutt-send-email-mst@redhat.com> (raw)
In-Reply-To: <CAA2XHbcU0tV0NrBXT6oh6LOz7sKm9P8jqD1=T-ZgTahSVM_qwQ@mail.gmail.com>

On Tue, Apr 07, 2015 at 04:22:42PM +0200, Luke Gorrie wrote:
> Hi Michael,
> 
> I'm writing to follow up the previous discussion about memory barriers in
> virtio-net device implementations, and Cc'ing the DPDK list because I believe
> this is relevant to them too.
> 
> First, thanks again for getting in touch and reviewing our code.
> 
> I have now found a missed case where we *do* require a hardware memory barrier
> on x86 in our vhost/virtio-net device. That is when checking the interrupt
> suppression flag after updating used->idx. This is needed because x86 can
> reorder the write to used->idx after the read from avail->flags, and that
> causes the guest to see a stale value of used->idx after it toggles interrupt
> suppression.
> 
> If I may spell out my mental model, for the sake of being corrected and/or as
> an example of how third party developers are reading and interpreting the
> Virtio-net spec:
> 
> Relating this to Virtio 1.0, the most relevant section is 3.2.1 (Supplying
> Buffers to the Device) which calls for two "suitable memory barriers". The spec
> talks about these from the driver perspective, but they are both relevant to
> the device side too.
> 
> The first barrier (write to descriptor table before write to used->idx) is
> implicit on x86 because writes by the same core are not reordered. This means
> that no explicit hardware barrier is needed. (A compiler barrier may be needed,
> however.)
> 
> The second memory barrier (write to used->idx before reading avail->flags) is
> not implicit on x86 because stores are reordered after loads. So an explicit
> hardware memory barrier is needed.
> 
> I hope that is a correct assessment of the situation. (Forgive my
> x86centricity, I am sure that seems very foreign to kernel hackers.)
> 
> If this assessment is correct then the DPDK developers might also want to
> review librte_vhost/vhost_rxtx.c and consider adding a hardware memory barrier
> between writing used->idx and reading avail->flags.
> 
> Cheers,
> -Luke

I agree, this looks like a bug in dpdk.

> P.S. I notice that the Linux virtio-net driver does not seem to tolerate
> spurious interrupts, even though the Virtio 1.0 spec requires this ("must"). On
> 3.13.11-ckt15 I see them trigger an "irq nobody cared" kernel log message and
> then the irq is disabled. If that sounds suspicious I can supply more
> information.
> 
>

More information might be useful, yes.

Just guessing from the available info:

I think you refer to this:
        The driver MUST handle spurious interrupts from the device.

The intent is to be able to handle some spurious interrupts once in a
while.  AFAIK linux triggers the message if it gets a huge number of
spurious interrupts for an extended period of time.
For example, this will trigger if the device does not clear interrupt
line after interrupt register read.
 

  reply	other threads:[~2015-04-07 15:31 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <20150127160126.GA10651@redhat.com>
     [not found] ` <CADDJ2=M6hwFwooXqUjUc9+JxjW1sVYvKhY9dBavrmMUrej6Ysw@mail.gmail.com>
2015-04-07 14:22   ` Luke Gorrie
2015-04-07 15:30     ` Michael S. Tsirkin [this message]
2015-04-08  3:40       ` Luke Gorrie
2015-04-08 15:15     ` Xie, Huawei
2015-04-09  3:12       ` Luke Gorrie
2015-04-09 15:00         ` Xie, Huawei

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20150407172849-mutt-send-email-mst@redhat.com \
    --to=mst@redhat.com \
    --cc=dev@dpdk.org \
    --cc=luke@snabb.co \
    --cc=snabb-devel@googlegroups.com \
    --cc=tech@virtualopensystems.com \
    --cc=virtualization@lists.linux-foundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).