From: Ilya Maximets <i.maximets@samsung.com>
To: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Cc: dev@dpdk.org, Huawei Xie <huawei.xie@intel.com>,
Dyasly Sergey <s.dyasly@samsung.com>,
Heetae Ahn <heetae82.ahn@samsung.com>,
Jianfeng Tan <jianfeng.tan@intel.com>,
Stephen Hemminger <stephen@networkplumber.org>,
Thomas Monjalon <thomas.monjalon@6wind.com>
Subject: Re: [dpdk-dev] [PATCH] vhost: fix segfault on bad descriptor address.
Date: Tue, 12 Jul 2016 08:53:16 +0300 [thread overview]
Message-ID: <578485CC.8070809@samsung.com> (raw)
In-Reply-To: <20160712024305.GB26521@yliu-dev.sh.intel.com>
On 12.07.2016 05:43, Yuanhan Liu wrote:
> On Mon, Jul 11, 2016 at 02:47:56PM +0300, Ilya Maximets wrote:
>> On 11.07.2016 14:05, Yuanhan Liu wrote:
>>> On Mon, Jul 11, 2016 at 12:50:24PM +0300, Ilya Maximets wrote:
>>>> On 11.07.2016 11:38, Yuanhan Liu wrote:
>>>>> On Sun, Jul 10, 2016 at 09:17:31PM +0800, Yuanhan Liu wrote:
>>>>>> On Fri, Jul 08, 2016 at 02:48:56PM +0300, Ilya Maximets wrote:
>>>>>>>
>>>>>>> Another point is that crash constantly happens on queue_id=3 (second RX queue) in
>>>>>>> my scenario. It is newly allocated virtqueue while reconfiguration from rxq=1 to
>>>>>>> rxq=2.
>>>>>>
>>>>>> That's a valuable message: what's your DPDK HEAD commit while triggering
>>>>>> this issue?
>>>>
>>>> fbfd99551ca3 ("mbuf: add raw allocation function")
>>>>
>>>>>
>>>>> I guess I have understood what goes wrong in you case.
>>>>>
>>>>> I would guess that your vhost has 2 queues (here I mean queue-pairs,
>>>>> including one Tx and Rx queue; below usage is the same) configured,
>>>>> so does to your QEMU. However, you just enabled 1 queue while starting
>>>>> testpmd inside the guest, and you want to enable 2 queues by running
>>>>> following testpmd commands:
>>>>>
>>>>> stop
>>>>> port stop all
>>>>> port config all rxq 2
>>>>> port config all txq 2
>>>>> port start all
>>>>>
>>>>> Badly, that won't work for current virtio PMD implementation, and what's
>>>>> worse, it triggers a vhost crash, the one you saw.
>>>>>
>>>>> Here is how it comes. Since you just enabled 1 queue while starting
>>>>> testpmd, it will setup 1 queue only, meaning only one queue's **valid**
>>>>> information will be sent to vhost. You might see SET_VRING_ADDR
>>>>> (and related vhost messages) for the other queue as well, but they
>>>>> are just the dummy messages: they don't include any valid/real
>>>>> information about the 2nd queue: the driver don't setup it after all.
>>>>>
>>>>> So far, so good. It became broken when you run above commands. Those
>>>>> commands do setup for the 2nd queue, however, they failed to trigger
>>>>> the QEMU virtio device to start the vhost-user negotiation, meaning
>>>>> no SET_VRING_ADDR will be sent for the 2nd queue, leaving vhost
>>>>> untold and not updated.
>>>>>
>>>>> What's worse, above commands trigger the QEMU to send SET_VRING_ENABLE
>>>>> messages, to enable all the vrings. And since the vrings for the 2nd
>>>>> queue are not properly configured, the crash happens.
>>>>
>>>> Hmm, why 2nd queue works properly with my fix to vhost in this case?
>>>
>>> Hmm, really? You are sure that data flows in your 2nd queue after those
>>> commands? From what I know is that your patch just avoid a crash, but
>>> does not fix it.
>>
>> Oh, sorry. Yes, it doesn't work. With my patch applied I have a QEMU hang.
>
> The crash actually could be avoided by commit 0823c1cb0a73 ("vhost:
> workaround stale vring base"), accidentally. That's why I asked you
> above the HEAD commit you were using.
Thanks for pointing this. I'll check it.
>>>>> So maybe we should do virtio reset on port start?
>>>>
>>>> I guess it was removed by this patch:
>>>> a85786dc816f ("virtio: fix states handling during initialization").
>>>
>>> Seems yes.
>
> Actually, we should not do that: do reset on port start. The right fix
> should be allocating MAX queues virtio device supports (2 here). This
> would allow us changing the queue number dynamically.
Yes, I agree that this is the right way to fix this issue.
> But this doesn't sound a simple fix; it involves many code changes, due
> to it was not designed this way before. Therefore, we will not fix it
> in this release, due to it's too late. Let's fix it in the next release
> instead. For the crash issue, it will not happen with the latest HEAD.
> Though it's accident fix, I think we are fine here.
>
> --yliu
>
>
next prev parent reply other threads:[~2016-07-12 5:53 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-05-20 12:50 Ilya Maximets
2016-05-23 10:57 ` Yuanhan Liu
2016-05-23 11:04 ` Ilya Maximets
2016-05-30 11:05 ` Ilya Maximets
2016-05-30 14:25 ` Yuanhan Liu
2016-05-31 9:12 ` Ilya Maximets
2016-05-30 12:00 ` Tan, Jianfeng
2016-05-30 12:24 ` Ilya Maximets
2016-05-31 6:53 ` Tan, Jianfeng
2016-05-31 9:10 ` Ilya Maximets
2016-05-31 22:06 ` Rich Lane
2016-06-02 10:46 ` Ilya Maximets
2016-06-02 16:22 ` Rich Lane
2016-06-03 6:01 ` Ilya Maximets
2016-07-01 7:35 ` Yuanhan Liu
2016-07-06 11:19 ` Ilya Maximets
2016-07-06 12:24 ` Yuanhan Liu
2016-07-08 11:48 ` Ilya Maximets
2016-07-10 13:17 ` Yuanhan Liu
2016-07-11 8:38 ` Yuanhan Liu
2016-07-11 9:50 ` Ilya Maximets
2016-07-11 11:05 ` Yuanhan Liu
2016-07-11 11:47 ` Ilya Maximets
2016-07-12 2:43 ` Yuanhan Liu
2016-07-12 5:53 ` Ilya Maximets [this message]
2016-07-13 7:34 ` Ilya Maximets
2016-07-13 8:47 ` Yuanhan Liu
2016-07-13 15:54 ` Rich Lane
2016-07-14 1:42 ` Yuanhan Liu
2016-07-14 4:38 ` Ilya Maximets
2016-07-14 8:18 ` [dpdk-dev] [PATCH v2] " Ilya Maximets
2016-07-15 6:17 ` Yuanhan Liu
2016-07-15 7:23 ` Ilya Maximets
2016-07-15 8:40 ` Yuanhan Liu
2016-07-15 11:15 ` [dpdk-dev] [PATCH v3 0/2] " Ilya Maximets
2016-07-15 11:15 ` [dpdk-dev] [PATCH v3 1/2] vhost: fix using of bad return value on mergeable enqueue Ilya Maximets
2016-07-15 11:15 ` [dpdk-dev] [PATCH v3 2/2] vhost: do sanity check for ring descriptor address Ilya Maximets
2016-07-15 12:14 ` [dpdk-dev] [PATCH v3 0/2] vhost: fix segfault on bad " Yuanhan Liu
2016-07-15 19:37 ` Thomas Monjalon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=578485CC.8070809@samsung.com \
--to=i.maximets@samsung.com \
--cc=dev@dpdk.org \
--cc=heetae82.ahn@samsung.com \
--cc=huawei.xie@intel.com \
--cc=jianfeng.tan@intel.com \
--cc=s.dyasly@samsung.com \
--cc=stephen@networkplumber.org \
--cc=thomas.monjalon@6wind.com \
--cc=yuanhan.liu@linux.intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).