DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Tan, Jianfeng" <jianfeng.tan@intel.com>
To: Maxime Coquelin <maxime.coquelin@redhat.com>,
	Victor Kaplansky <vkaplans@redhat.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>,
	"stable@dpdk.org" <stable@dpdk.org>,
	"Yang, Yi Y" <yi.y.yang@intel.com>,
	"Harris, James R" <james.r.harris@intel.com>,
	"Yang, Ziye" <ziye.yang@intel.com>,
	"Liu, Changpeng" <changpeng.liu@intel.com>,
	"Wodkowski, PawelX" <pawelx.wodkowski@intel.com>,
	"Stojaczyk, DariuszX" <dariuszx.stojaczyk@intel.com>,
	Yuanhan Liu <yliu@fridaylinux.org>
Subject: Re: [dpdk-dev] [PATCH] vhost: fix segfault as handle set_mem_table message
Date: Thu, 29 Mar 2018 20:59:45 +0800	[thread overview]
Message-ID: <1aac21ef-e566-857f-3ccd-a9e5be7b3b37@intel.com> (raw)
In-Reply-To: <1c5d3010-c7a9-a2b1-909e-dd43cde24af1@redhat.com>


On 3/29/2018 3:35 PM, Maxime Coquelin wrote:
>
>
> On 03/29/2018 09:01 AM, Tan, Jianfeng wrote:
>> Hi Maxime and Victor,
>>
>>> -----Original Message-----
>>> From: Maxime Coquelin [mailto:maxime.coquelin@redhat.com]
>>> Sent: Tuesday, December 5, 2017 10:28 PM
>>> To: Yuanhan Liu; Tan, Jianfeng; Victor Kaplansky
>>> Cc: dev@dpdk.org; stable@dpdk.org; Yang, Yi Y
>>> Subject: Re: [PATCH] vhost: fix segfault as handle set_mem_table 
>>> message
>>>
>>>
>>>
>>> On 12/05/2017 03:19 PM, Yuanhan Liu wrote:
>>>> On Tue, Nov 28, 2017 at 01:09:29PM +0100, Maxime Coquelin wrote:
>>>>>
>>>>>
>>>>> On 11/15/2017 12:41 PM, Jianfeng Tan wrote:
>>>>>> In a running VM, operations (like device attach/detach) will
>>>>>> trigger the QEMU to resend set_mem_table to vhost-user backend.
>>>>>>
>>>>>> DPDK vhost-user handles this message rudely by unmap all existing
>>>>>> regions and map new ones. This might lead to segfault if there
>>>>>> is pmd thread just trying to touch those unmapped memory regions.
>>>>>>
>>>>>> But for most cases, except VM memory hotplug,
>>>
>>> FYI, Victor is working on implementing a lock-less protection mechanism
>>> to prevent crashes in such cases. It is intended first to protect
>>> log_base in case of multiqueue + live-migration, but would solve thi
>>> issue too.
>>
>> Bring this issue back for discussion.
>>
>> Reported by SPDK guys, even with per-queue lock, they could still run 
>> into crash as of memory hot plug or unplug.
>> In detail, you add the lock in an internal struct, vhost_queue which 
>> is, unfortunately, not visible to the external datapath, like 
>> vhost-scsi in SPDK.
>
> Yes, I agree current solution is not enough
>
>> The memory hot plug and unplug would be the main issue from SPDK side 
>> so far. For this specific issue, I think we can continue this patch 
>> to filter out the changed regions, and keep unchanged regions not 
>> remapped.

How do you think that we move forward on this specific memory issue? I 
think it can be parallel with the general mechanism below.

>>
>> But I know that the per-vq lock is not only to resolve the memory 
>> table issue, some other vhost-user messages could also lead to 
>> problems? If yes, shall we take a step back, to think about how to 
>> solve this kind of issues for backends, like vhost-scsi.
>
> Right, any message that can change the device or virtqueue states can be
> problematic.
>
>> Thoughts?
>
> In another thread, 

Apologize for starting another thread.

> SPDK people proposed to destroy and re-create the
> device for every message. I think this is not acceptable.

It sounds a little overkill and error-prone in my first impression.

>
> I proposed an alternative that I think would work:
> - external backends & applications implements the .vring_state_changed()
>   callback. On disable it stops processing the rings and only return
>   once all descriptor buffers are processed. On enable, they resume the
>   rings processing.

OK, this gives a chance for external backends & applications to sync 
(lock or event) with polling threads, just like how destroy_device() works.

> - In vhost lib, we implement vhost_vring_pause and vhost_vring_resume
>   functions. In pause function, we save device state (enable or
>   disable) in a variable, and if the ring is enabled at device level it
>   calls .vring_state_changed() with disabled state. In resume, it checks
>   the vring state in the device and call .vring_state_changed() with
>   enable state if it was enabled in device state.

>
> So, I think that would work but I hadn't had a clear reply from SPDK
> people proving it wouldn't.
>
> They asked we revert Victor's patch, but I don't see the need as it does
> not hurt SPDK (but doesn't protect anything for them I agree), while it
> really fixes real issues with internal Net backend.

I agree with you. As SPDK has no chance to call the lock, it can not be 
affected. I think what people (including myself) are really against is 
adding lock in DPDK PMD. But we did not find a better way so far.

>
> What do you think of my proposal? Do you see other alternative?


Sounds a feasible way. Let's wait and see how SPDK guys think.

Thanks,
Jianfeng

>
> Thanks,
> Maxime
>
>> Thanks,
>> Jianfeng

      parent reply	other threads:[~2018-03-29 12:59 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-11-15 11:41 Jianfeng Tan
2017-11-28 12:09 ` Maxime Coquelin
2017-12-05 14:19   ` Yuanhan Liu
2017-12-05 14:28     ` Maxime Coquelin
2018-03-29  7:01       ` Tan, Jianfeng
2018-03-29  7:35         ` Maxime Coquelin
2018-03-29 12:57           ` Wodkowski, PawelX
2018-03-29 13:20             ` Tan, Jianfeng
2018-03-29 16:37             ` Maxime Coquelin
2018-03-29 18:09               ` Wodkowski, PawelX
2018-03-29 12:59           ` Tan, Jianfeng [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1aac21ef-e566-857f-3ccd-a9e5be7b3b37@intel.com \
    --to=jianfeng.tan@intel.com \
    --cc=changpeng.liu@intel.com \
    --cc=dariuszx.stojaczyk@intel.com \
    --cc=dev@dpdk.org \
    --cc=james.r.harris@intel.com \
    --cc=maxime.coquelin@redhat.com \
    --cc=pawelx.wodkowski@intel.com \
    --cc=stable@dpdk.org \
    --cc=vkaplans@redhat.com \
    --cc=yi.y.yang@intel.com \
    --cc=yliu@fridaylinux.org \
    --cc=ziye.yang@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).