From: "Tan, Jianfeng" <jianfeng.tan@intel.com>
To: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Cc: dev@dpdk.org, zhihong.wang@intel.com, lining18@jd.com
Subject: Re: [dpdk-dev] [PATCH 2/3] net/virtio_user: fix wrong sequence of messages
Date: Fri, 9 Sep 2016 11:59:18 +0800 [thread overview]
Message-ID: <11291eac-cbbe-7080-3d35-bdacb5341d89@intel.com> (raw)
In-Reply-To: <20160908121818.GO23158@yliu-dev.sh.intel.com>
On 9/8/2016 8:18 PM, Yuanhan Liu wrote:
> On Thu, Sep 08, 2016 at 04:53:22PM +0800, Tan, Jianfeng wrote:
>>
>> On 9/6/2016 4:20 PM, Yuanhan Liu wrote:
>>> On Tue, Sep 06, 2016 at 03:54:30PM +0800, Tan, Jianfeng wrote:
>>>> Hi Yuanhan,
>>>>
>>>>
>>>> On 9/6/2016 2:42 PM, Yuanhan Liu wrote:
>>>>> On Fri, Aug 05, 2016 at 11:36:42AM +0000, Jianfeng Tan wrote:
>>>>>> When virtio_user is used with VPP's native vhost user, it cannot
>>>>>> send/receive any packets.
>>>>>>
>>>>>> The root cause is that vpp-vhost-user translates the message
>>>>>> VHOST_USER_SET_FEATURES as puting this device into init state,
>>>>>> aka, zero all related structures. However, previous code
>>>>>> puts this message at last in the whole initialization process,
>>>>>> which leads to all previous information are zeroed.
>>>>>>
>>>>>> To fix this issue, we rearrange the sequence of those messages.
>>>>>> - step 0, send VHOST_USER_SET_VRING_CALL so that vhost allocates
>>>>>> virtqueue structures;
>>>>> Yes, it is. However, it's not that right to do that (you see there is
>>>>> a FIXME in vhost_user_set_vring_call()).
>>>> I suppose you are specifying vhost_set_vring_call().
>>> Oh, I was talking about the new code: I have renamed it to
>>> vhost_user_set_vring_call :)
>>>
>>>>> That means it need be fixed: we should not rely on fact that it's the
>>>>> first per-vring message we will get in the current QEMU implementation
>>>>> as the truth.
>>>>>
>>>>> That also means, naming a function like virtio_user_create_queue() based
>>>>> on above behaviour is wrong.
>>>> It's actually a good catch. After a light thought, I think in DPDK vhost, we
>>>> may need to create those virtqueues once unix socket gets connected, just
>>>> like in vhost-net, virtqueues are created on char file open. Right?
>>> There is a difference: for vhost-net and tap mode, IIRC, it knows how
>>> many queues before doing setup.
>> No, from linux/drivers/vhost/net.c:vhost_net_open(), we can see that
>> virtqueues are allocated according to VHOST_NET_VQ_MAX.
> Well, if you took a closer look, you will find VHOST_NET_VQ_MAX is
> defined to 2. That means it allocates a queue-pair only.
>
> And FYI, the MQ support for vhost-net is actually done in the tap
> driver, but not in vhost-net driver. That results to the MQ
> implementation is a bit different between vhost-net and vhost-user.
>
> Put simply, in vhost-net, one queue-pair has a backend fd associated
> with it. Vhost requests for different queue-pair are sent by different
> fd. That also means the vhost-net doesn't even need be aware of the
> MQ stuff.
>
> However, for vhost-user implementation, all queue-pairs share one
> socket fd. All requests all also sent over the single socket fd,
> thus the backend (DPDK vhost) has to figure out how many queue
> pairs are actually enabled: and we detect it by reading the
> vring index of SET_VRING_CALL message; it's not quite right though.
Aha, right, nice analysis.
>
>> How about reconsidering previous suggestion to allocate vq once connection
>> is established?
> That's too much, because DPDK claims to support up to 0x8000
> queue-pairs. Don't even to say that the vhost_virtqueue struct
> was way too big before: it even holds an array of buf_vec with
> size 256.
Another mistake of my memory, I was remember it wrongly as only 8 VQs
are supported.
One thing not related, provided that VHOST_MAX_QUEUE_PAIRS equals to
0x8000, struct vhost_virtqueue *virtqueue[VHOST_MAX_QUEUE_PAIRS * 2]
spends 4MB for each virtio device, which could be a refined.
>
>> Never mind, above fix on the vhost side will not take effect on existing
>> vpp-vhost implementations.
> Actually, I was talking about the DPDK vhost implementation :)
This patch is talking about vpp's native vhost implementation, not
dpdk-vhost, and not the way vpp uses dpdk-vhost.
>
>> We still need to fix it in the virtio side.
> Yes, we could fix it in our side, even though VPP is broken.
OK, let's back to this patch.
>
>>> While for vhost-user, it doesn't. That
>>> means, we have to allocate and setup virtqueues reactively: just like
>>> what we have done in vhost_set_vring_call(). What doesn't look perfect
>>> is it assume SET_VRING_CALL is the first per-vring message we will get.
>> Yes, depending on the assumption that SET_VRING_CALL is the first per-vring
>> message, looks like a bad implementation. As Stephen has suggested, it's
>> more like a bug in vpp. If we treat it like that way, we will fix nothing
>> here.
>>
>>
>>>>>> - step 1, send VHOST_USER_SET_FEATURES to confirm the features;
>>>>>> - step 2, send VHOST_USER_SET_MEM_TABLE to share mem regions;
>>>>>> - step 3, send VHOST_USER_SET_VRING_NUM, VHOST_USER_SET_VRING_BASE,
>>>>>> VHOST_USER_SET_VRING_ADDR, VHOST_USER_SET_VRING_KICK for each
>>>>>> queue;
>>>>>> - ...
>>>>>>
>>>>>> Fixes: 37a7eb2ae816 ("net/virtio-user: add device emulation layer")
>>>>>>
>>>>>> Reported-by: Zhihong Wang <zhihong.wang@intel.com>
>>>>>> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
>>>>>> ---
>>>>>> drivers/net/virtio/virtio_user/virtio_user_dev.c | 120 ++++++++++++++---------
>>>>>> 1 file changed, 72 insertions(+), 48 deletions(-)
>>>>> That's too much of code for a bug fix. I'm wondering how about just
>>>>> moving VHOST_USER_GET_PROTOCOL_FEATURES ahead, to the begining of
>>>>> virtio_user_start_device()? It should fix this issue.
>>>> Why does VHOST_USER_GET_PROTOCOL_FEATURES care? Do you mean shifting
>>>> VHOST_USER_SET_FEATURES earlier?
>>> Oops, right, I meant SET_FEATURES. Sorry about confusion introduced by
>>> the silly auto-completion.
>> Still not working. VPP needs SET_VRING_CALL to create vq firstly.
> Didn't get it. In the proposal, SET_FEATURES is sent before every other
> messages, thus it should not cause the issue you described in this patch.
OK. Let me try to explain. We take three vhost implementations into
consideration: dpdk-2.2-vhost, dpdk-master-vhost, vpp-native-vhost.
If set_feature before set_vring_call, dpdk-2.2-vhost will fail: inside
set_feature handler, assigning header length to VQs which will be
created in set_vring_call handler.
So we need to keep set_vring_call firstly. Then set_feature needs to be
sent before any other msgs, this is what vpp-native-vhost requires. In
all, the sequence is like this:
1. set_vring_call,
2. set_feature,
3. other msgs
> Besides, haven't we already sent SET_VRING_CALL before other messages
> (well, execept the SET_FEATURES and SET_MEM_TABLE message)?
Yes, set_vring_call is already in the first place, but we need to plugin
set_feature between set_vring_call and other msgs. Previously,
set_vring_call and other msgs are together.
Thanks,
Jianfeng
> That's still
> not enough for vpp's native vhost-user implementation?
> --yliu
next prev parent reply other threads:[~2016-09-09 3:59 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-08-05 11:36 [dpdk-dev] [PATCH 0/3] fix virtio_user issues Jianfeng Tan
2016-08-05 11:36 ` [dpdk-dev] [PATCH 1/3] net/virtio_user: fix queue pair not enabled Jianfeng Tan
2016-09-06 6:30 ` Yuanhan Liu
2016-08-05 11:36 ` [dpdk-dev] [PATCH 2/3] net/virtio_user: fix wrong sequence of messages Jianfeng Tan
2016-08-05 16:36 ` Stephen Hemminger
2016-08-08 1:19 ` Tan, Jianfeng
2016-09-06 6:42 ` Yuanhan Liu
2016-09-06 7:54 ` Tan, Jianfeng
2016-09-06 8:20 ` Yuanhan Liu
2016-09-08 8:53 ` Tan, Jianfeng
2016-09-08 12:18 ` Yuanhan Liu
2016-09-09 3:59 ` Tan, Jianfeng [this message]
2016-09-09 4:19 ` Yuanhan Liu
2016-09-09 5:50 ` Tan, Jianfeng
2016-09-09 6:03 ` Yuanhan Liu
2016-09-09 6:24 ` Tan, Jianfeng
2016-09-09 6:31 ` Yuanhan Liu
2016-08-05 11:36 ` [dpdk-dev] [PATCH 3/3] net/virtio_user: fix dev not freed after init error Jianfeng Tan
2016-08-05 16:34 ` Stephen Hemminger
2016-08-08 1:07 ` Tan, Jianfeng
2016-08-29 7:01 ` [dpdk-dev] [PATCH 0/3] fix virtio_user issues Christian Ehrhardt
2016-09-27 19:11 ` [dpdk-dev] [PATCH v2 " Jianfeng Tan
2016-09-27 19:11 ` [dpdk-dev] [PATCH v2 1/3] net/virtio_user: fix queue pair not enabled Jianfeng Tan
2016-09-28 0:05 ` Yuanhan Liu
2016-09-27 19:11 ` [dpdk-dev] [PATCH v2 2/3] net/virtio_user: fix wrong sequence of messages Jianfeng Tan
2016-09-27 19:11 ` [dpdk-dev] [PATCH v2 3/3] net/virtio_user: fix dev not freed after init error Jianfeng Tan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=11291eac-cbbe-7080-3d35-bdacb5341d89@intel.com \
--to=jianfeng.tan@intel.com \
--cc=dev@dpdk.org \
--cc=lining18@jd.com \
--cc=yuanhan.liu@linux.intel.com \
--cc=zhihong.wang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).