From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by dpdk.org (Postfix) with ESMTP id E24C72B88 for ; Fri, 9 Sep 2016 06:19:21 +0200 (CEST) Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga105.jf.intel.com with ESMTP; 08 Sep 2016 21:19:20 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.30,304,1470726000"; d="scan'208";a="876712161" Received: from yliu-dev.sh.intel.com (HELO yliu-dev) ([10.239.67.162]) by orsmga003.jf.intel.com with ESMTP; 08 Sep 2016 21:19:21 -0700 Date: Fri, 9 Sep 2016 12:19:49 +0800 From: Yuanhan Liu To: "Tan, Jianfeng" Cc: dev@dpdk.org, zhihong.wang@intel.com, lining18@jd.com Message-ID: <20160909041949.GP23158@yliu-dev.sh.intel.com> References: <1470397003-5782-1-git-send-email-jianfeng.tan@intel.com> <1470397003-5782-3-git-send-email-jianfeng.tan@intel.com> <20160906064246.GQ30752@yliu-dev.sh.intel.com> <20160906082011.GA23158@yliu-dev.sh.intel.com> <53e1d78c-e3b0-e16c-992d-e45ed1b31a4f@intel.com> <20160908121818.GO23158@yliu-dev.sh.intel.com> <11291eac-cbbe-7080-3d35-bdacb5341d89@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <11291eac-cbbe-7080-3d35-bdacb5341d89@intel.com> User-Agent: Mutt/1.5.23 (2014-03-12) Subject: Re: [dpdk-dev] [PATCH 2/3] net/virtio_user: fix wrong sequence of messages X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Sep 2016 04:19:22 -0000 On Fri, Sep 09, 2016 at 11:59:18AM +0800, Tan, Jianfeng wrote: > It's actually a good catch. After a light thought, I think in DPDK vhost, we > may need to create those virtqueues once unix socket gets connected, just > like in vhost-net, virtqueues are created on char file open. Right? > > There is a difference: for vhost-net and tap mode, IIRC, it knows how > many queues before doing setup. > > No, from linux/drivers/vhost/net.c:vhost_net_open(), we can see that > virtqueues are allocated according to VHOST_NET_VQ_MAX. > > Well, if you took a closer look, you will find VHOST_NET_VQ_MAX is > defined to 2. That means it allocates a queue-pair only. > > And FYI, the MQ support for vhost-net is actually done in the tap > driver, but not in vhost-net driver. That results to the MQ > implementation is a bit different between vhost-net and vhost-user. > > Put simply, in vhost-net, one queue-pair has a backend fd associated > with it. Vhost requests for different queue-pair are sent by different > fd. That also means the vhost-net doesn't even need be aware of the > MQ stuff. > > However, for vhost-user implementation, all queue-pairs share one > socket fd. All requests all also sent over the single socket fd, > thus the backend (DPDK vhost) has to figure out how many queue > pairs are actually enabled: and we detect it by reading the > vring index of SET_VRING_CALL message; it's not quite right though. > > > Aha, right, nice analysis. Just some rough memory in my mind ;-) > > > > How about reconsidering previous suggestion to allocate vq once connection > is established? > > That's too much, because DPDK claims to support up to 0x8000 > queue-pairs. Don't even to say that the vhost_virtqueue struct > was way too big before: it even holds an array of buf_vec with > size 256. > > > Another mistake of my memory, I was remember it wrongly as only 8 VQs are > supported. > One thing not related, provided that VHOST_MAX_QUEUE_PAIRS equals to 0x8000, > struct vhost_virtqueue  *virtqueue[VHOST_MAX_QUEUE_PAIRS * 2] spends 4MB for > each virtio device, which could be a refined. Yes, we could allocate a small array first, and then re-allocate if necessary. > > > > Never mind, above fix on the vhost side will not take effect on existing > vpp-vhost implementations. > > Actually, I was talking about the DPDK vhost implementation :) > > > This patch is talking about vpp's native vhost implementation, not dpdk-vhost, > and not the way vpp uses dpdk-vhost. Yes, I know. What I meant is there was a "workaround" in DPDK vhost implementation, and since you bring this issue on the table again, it's a chance to think about how can we fix it. A rough idea come to my mind is we could check all the per-vring message at the begining of vhost_user_msg_handler() and allocate related vq when necessary (when it's the first vring message we got). Yeah, I know it's a bit ugly, but it at least gets rid of that "not-that-true" assumption. > > Still not working. VPP needs SET_VRING_CALL to create vq firstly. > > Didn't get it. In the proposal, SET_FEATURES is sent before every other > messages, thus it should not cause the issue you described in this patch. > > > OK. Let me try to explain. We take three vhost implementations into > consideration: dpdk-2.2-vhost, dpdk-master-vhost, vpp-native-vhost. > > If set_feature before set_vring_call, dpdk-2.2-vhost will fail: inside > set_feature handler, assigning header length to VQs which will be created in > set_vring_call handler. Oh, right. That was an in-correct implementation. > So we need to keep set_vring_call firstly. > Then set_feature needs to be sent > before any other msgs, this is what vpp-native-vhost requires. In all, the > sequence is like this: > 1. set_vring_call, > 2. set_feature, > 3. other msgs > > > Besides, haven't we already sent SET_VRING_CALL before other messages > (well, execept the SET_FEATURES and SET_MEM_TABLE message)? > > > Yes, set_vring_call is already in the first place, but we need to plugin > set_feature between set_vring_call and other msgs. Previously, set_vring_call > and other msgs are together. Okay. Another thing I noticed is that virtio-user lacks some feature negotiations, like GET_FEATURES and GET_PROTOCOL_FEATURES. I think you might need add them back somewhen? --yliu