From: Yuanhan Liu <yuanhan.liu@linux.intel.com>
To: Ilya Maximets <i.maximets@samsung.com>
Cc: dev@dpdk.org, Huawei Xie <huawei.xie@intel.com>,
Dyasly Sergey <s.dyasly@samsung.com>,
Heetae Ahn <heetae82.ahn@samsung.com>,
Jianfeng Tan <jianfeng.tan@intel.com>,
Stephen Hemminger <stephen@networkplumber.org>,
Thomas Monjalon <thomas.monjalon@6wind.com>,
Rich Lane <rich.lane@bigswitch.com>
Subject: Re: [dpdk-dev] [PATCH] vhost: fix segfault on bad descriptor address.
Date: Wed, 13 Jul 2016 16:47:32 +0800 [thread overview]
Message-ID: <20160713084732.GH26521@yliu-dev.sh.intel.com> (raw)
In-Reply-To: <5785EEEF.3080400@samsung.com>
On Wed, Jul 13, 2016 at 10:34:07AM +0300, Ilya Maximets wrote:
> On 12.07.2016 08:53, Ilya Maximets wrote:
> > On 12.07.2016 05:43, Yuanhan Liu wrote:
> >> On Mon, Jul 11, 2016 at 02:47:56PM +0300, Ilya Maximets wrote:
> >>> On 11.07.2016 14:05, Yuanhan Liu wrote:
> >>>> On Mon, Jul 11, 2016 at 12:50:24PM +0300, Ilya Maximets wrote:
> >>>>> On 11.07.2016 11:38, Yuanhan Liu wrote:
> >>>>>> On Sun, Jul 10, 2016 at 09:17:31PM +0800, Yuanhan Liu wrote:
> >>>>>>> On Fri, Jul 08, 2016 at 02:48:56PM +0300, Ilya Maximets wrote:
> >>>>>>>>
> >>>>>>>> Another point is that crash constantly happens on queue_id=3 (second RX queue) in
> >>>>>>>> my scenario. It is newly allocated virtqueue while reconfiguration from rxq=1 to
> >>>>>>>> rxq=2.
> >>>>>>>
> >>>>>>> That's a valuable message: what's your DPDK HEAD commit while triggering
> >>>>>>> this issue?
> >>>>>
> >>>>> fbfd99551ca3 ("mbuf: add raw allocation function")
> >>>>>
> >>>>>>
> >>>>>> I guess I have understood what goes wrong in you case.
> >>>>>>
> >>>>>> I would guess that your vhost has 2 queues (here I mean queue-pairs,
> >>>>>> including one Tx and Rx queue; below usage is the same) configured,
> >>>>>> so does to your QEMU. However, you just enabled 1 queue while starting
> >>>>>> testpmd inside the guest, and you want to enable 2 queues by running
> >>>>>> following testpmd commands:
> >>>>>>
> >>>>>> stop
> >>>>>> port stop all
> >>>>>> port config all rxq 2
> >>>>>> port config all txq 2
> >>>>>> port start all
> >>>>>>
> >>>>>> Badly, that won't work for current virtio PMD implementation, and what's
> >>>>>> worse, it triggers a vhost crash, the one you saw.
> >>>>>>
> >>>>>> Here is how it comes. Since you just enabled 1 queue while starting
> >>>>>> testpmd, it will setup 1 queue only, meaning only one queue's **valid**
> >>>>>> information will be sent to vhost. You might see SET_VRING_ADDR
> >>>>>> (and related vhost messages) for the other queue as well, but they
> >>>>>> are just the dummy messages: they don't include any valid/real
> >>>>>> information about the 2nd queue: the driver don't setup it after all.
> >>>>>>
> >>>>>> So far, so good. It became broken when you run above commands. Those
> >>>>>> commands do setup for the 2nd queue, however, they failed to trigger
> >>>>>> the QEMU virtio device to start the vhost-user negotiation, meaning
> >>>>>> no SET_VRING_ADDR will be sent for the 2nd queue, leaving vhost
> >>>>>> untold and not updated.
> >>>>>>
> >>>>>> What's worse, above commands trigger the QEMU to send SET_VRING_ENABLE
> >>>>>> messages, to enable all the vrings. And since the vrings for the 2nd
> >>>>>> queue are not properly configured, the crash happens.
> >>>>>
> >>>>> Hmm, why 2nd queue works properly with my fix to vhost in this case?
> >>>>
> >>>> Hmm, really? You are sure that data flows in your 2nd queue after those
> >>>> commands? From what I know is that your patch just avoid a crash, but
> >>>> does not fix it.
> >>>
> >>> Oh, sorry. Yes, it doesn't work. With my patch applied I have a QEMU hang.
> >>
> >> The crash actually could be avoided by commit 0823c1cb0a73 ("vhost:
> >> workaround stale vring base"), accidentally. That's why I asked you
> >> above the HEAD commit you were using.
> >
> > Thanks for pointing this. I'll check it.
>
> I've checked my DPDK HEAD with above commit backported. Yes, it helps to
> avoid vhost crash in my scenario. As expected, after reconfiguration new
> virtqueue doesn't work, QEMU hangs sometimes.
> >>>>>> So maybe we should do virtio reset on port start?
> >>>>>
> >>>>> I guess it was removed by this patch:
> >>>>> a85786dc816f ("virtio: fix states handling during initialization").
> >>>>
> >>>> Seems yes.
> >>
> >> Actually, we should not do that: do reset on port start. The right fix
> >> should be allocating MAX queues virtio device supports (2 here). This
> >> would allow us changing the queue number dynamically.
> >
> > Yes, I agree that this is the right way to fix this issue.
> >
> >> But this doesn't sound a simple fix; it involves many code changes, due
> >> to it was not designed this way before. Therefore, we will not fix it
> >> in this release, due to it's too late. Let's fix it in the next release
> >> instead. For the crash issue, it will not happen with the latest HEAD.
> >> Though it's accident fix, I think we are fine here.
>
> This scenario fixed somehow, I agree. But this patch still needed to protect
> vhost from untrusted VM, from malicious or buggy virtio application.
> Maybe we could change the commit-message and resend this patch as a
> security enhancement? What do you think?
Indeed, but I'm a bit concerned about the performance regression found
by Rich, yet I am not quite sure why it happens, though Rich claimed
that it seems to be a problem related to compiler.
--yliu
next prev parent reply other threads:[~2016-07-13 8:44 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-05-20 12:50 Ilya Maximets
2016-05-23 10:57 ` Yuanhan Liu
2016-05-23 11:04 ` Ilya Maximets
2016-05-30 11:05 ` Ilya Maximets
2016-05-30 14:25 ` Yuanhan Liu
2016-05-31 9:12 ` Ilya Maximets
2016-05-30 12:00 ` Tan, Jianfeng
2016-05-30 12:24 ` Ilya Maximets
2016-05-31 6:53 ` Tan, Jianfeng
2016-05-31 9:10 ` Ilya Maximets
2016-05-31 22:06 ` Rich Lane
2016-06-02 10:46 ` Ilya Maximets
2016-06-02 16:22 ` Rich Lane
2016-06-03 6:01 ` Ilya Maximets
2016-07-01 7:35 ` Yuanhan Liu
2016-07-06 11:19 ` Ilya Maximets
2016-07-06 12:24 ` Yuanhan Liu
2016-07-08 11:48 ` Ilya Maximets
2016-07-10 13:17 ` Yuanhan Liu
2016-07-11 8:38 ` Yuanhan Liu
2016-07-11 9:50 ` Ilya Maximets
2016-07-11 11:05 ` Yuanhan Liu
2016-07-11 11:47 ` Ilya Maximets
2016-07-12 2:43 ` Yuanhan Liu
2016-07-12 5:53 ` Ilya Maximets
2016-07-13 7:34 ` Ilya Maximets
2016-07-13 8:47 ` Yuanhan Liu [this message]
2016-07-13 15:54 ` Rich Lane
2016-07-14 1:42 ` Yuanhan Liu
2016-07-14 4:38 ` Ilya Maximets
2016-07-14 8:18 ` [dpdk-dev] [PATCH v2] " Ilya Maximets
2016-07-15 6:17 ` Yuanhan Liu
2016-07-15 7:23 ` Ilya Maximets
2016-07-15 8:40 ` Yuanhan Liu
2016-07-15 11:15 ` [dpdk-dev] [PATCH v3 0/2] " Ilya Maximets
2016-07-15 11:15 ` [dpdk-dev] [PATCH v3 1/2] vhost: fix using of bad return value on mergeable enqueue Ilya Maximets
2016-07-15 11:15 ` [dpdk-dev] [PATCH v3 2/2] vhost: do sanity check for ring descriptor address Ilya Maximets
2016-07-15 12:14 ` [dpdk-dev] [PATCH v3 0/2] vhost: fix segfault on bad " Yuanhan Liu
2016-07-15 19:37 ` Thomas Monjalon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160713084732.GH26521@yliu-dev.sh.intel.com \
--to=yuanhan.liu@linux.intel.com \
--cc=dev@dpdk.org \
--cc=heetae82.ahn@samsung.com \
--cc=huawei.xie@intel.com \
--cc=i.maximets@samsung.com \
--cc=jianfeng.tan@intel.com \
--cc=rich.lane@bigswitch.com \
--cc=s.dyasly@samsung.com \
--cc=stephen@networkplumber.org \
--cc=thomas.monjalon@6wind.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).