From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 7AD553208 for ; Fri, 27 Feb 2015 10:43:04 +0100 (CET) Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga102.fm.intel.com with ESMTP; 27 Feb 2015 01:43:02 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.09,658,1418112000"; d="scan'208";a="691555563" Received: from kmsmsx153.gar.corp.intel.com ([172.21.73.88]) by orsmga002.jf.intel.com with ESMTP; 27 Feb 2015 01:43:02 -0800 Received: from shsmsx104.ccr.corp.intel.com (10.239.110.15) by KMSMSX153.gar.corp.intel.com (172.21.73.88) with Microsoft SMTP Server (TLS) id 14.3.195.1; Fri, 27 Feb 2015 17:43:00 +0800 Received: from shsmsx101.ccr.corp.intel.com ([169.254.1.192]) by SHSMSX104.ccr.corp.intel.com ([169.254.5.161]) with mapi id 14.03.0195.001; Fri, 27 Feb 2015 17:42:53 +0800 From: "Xie, Huawei" To: "Czesnowicz, Przemyslaw" , "dev@dpdk.org" Thread-Topic: [PATCH v3 09/11] lib/librte_vhost: vhost user support Thread-Index: AdBSccGQktR/nxkjQ3S/PoOFekiUAg== Date: Fri, 27 Feb 2015 09:42:53 +0000 Message-ID: References: <1423717649-11818-1-git-send-email-huawei.xie@intel.com> <1424712993-73818-1-git-send-email-przemyslaw.czesnowicz@intel.com> <1424712993-73818-10-git-send-email-przemyslaw.czesnowicz@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v3 09/11] lib/librte_vhost: vhost user support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Feb 2015 09:43:05 -0000 On 2/24/2015 1:36 AM, Czesnowicz, Przemyslaw wrote:=0A= > From: "Xie, Huawei" =0A= >=0A= > In rte_vhost_driver_register(), vhost unix domain socket listener fd is c= reated=0A= > and added to polled(based on select) fdset.=0A= >=0A= > In rte_vhost_driver_session_start(), fds in the fdset are checked for=0A= > processing. If there is new connection from qemu, connection fd accepted = is=0A= > added to polled fdset. The listener and connection fds in the fdset are= =0A= > then both checked. When there is message on the connection fd, its=0A= > callback vserver_message_handler is called to process vhost-user messages= .=0A= >=0A= > To support identifying which virtio is from which guest VM, we could call= =0A= > rte_vhost_driver_register with different socket path. Virtio devices from= =0A= > same VM will connect to VM specific socket. The socket path information i= s=0A= > stored in the virtio_net structure.=0A= >=0A= > Signed-off-by: Huawei Xie =0A= > Signed-off-by: Przemyslaw Czesnowicz =0A= > ---=0A= > lib/librte_vhost/Makefile | 8 +-=0A= > lib/librte_vhost/rte_virtio_net.h | 2 +=0A= > lib/librte_vhost/vhost-net.h | 4 +-=0A= > lib/librte_vhost/vhost_user/vhost-net-user.c | 457 ++++++++++++++++++++= ++++++=0A= > lib/librte_vhost/vhost_user/vhost-net-user.h | 106 ++++++=0A= > lib/librte_vhost/vhost_user/virtio-net-user.c | 314 ++++++++++++++++++= =0A= > lib/librte_vhost/vhost_user/virtio-net-user.h | 49 +++=0A= > lib/librte_vhost/virtio-net.c | 15 +-=0A= > 8 files changed, 948 insertions(+), 7 deletions(-)=0A= > create mode 100644 lib/librte_vhost/vhost_user/vhost-net-user.c=0A= > create mode 100644 lib/librte_vhost/vhost_user/vhost-net-user.h=0A= > create mode 100644 lib/librte_vhost/vhost_user/virtio-net-user.c=0A= > create mode 100644 lib/librte_vhost/vhost_user/virtio-net-user.h=0A= >=0A= > diff --git a/lib/librte_vhost/Makefile b/lib/librte_vhost/Makefile=0A= > index 298e4f8..cac943e 100644=0A= > --- a/lib/librte_vhost/Makefile=0A= > +++ b/lib/librte_vhost/Makefile=0A= =0A= cut lines...=0A= =0A= > +=0A= > +int=0A= > +user_set_mem_table(struct vhost_device_ctx ctx, struct VhostUserMsg *pms= g)=0A= > +{=0A= > + struct VhostUserMemory memory =3D pmsg->payload.memory;=0A= > + struct virtio_memory_regions *pregion;=0A= > + uint64_t mapped_address, mapped_size;=0A= > + struct virtio_net *dev;=0A= > + unsigned int idx =3D 0;=0A= > + struct orig_region_map *pregion_orig;=0A= > + uint64_t alignment;=0A= > +=0A= > + /* unmap old memory regions one by one*/=0A= > + dev =3D get_device(ctx);=0A= > + if (dev =3D=3D NULL)=0A= > + return -1;=0A= > +=0A= > + if (dev->mem) {=0A= > + free_mem_region(dev);=0A= > + free(dev->mem);=0A= > + dev->mem =3D NULL;=0A= > + }=0A= > +=0A= VHOST_SET_MEM_TABLE is sent and only sent to vhost backend in=0A= vhost_dev_start.=0A= In FC21, we don't receive VHOST_GET_VRING_BASE to stop the vhost=0A= device(refer to later comment below). Need root cause the change of=0A= guest virtio driver.=0A= For that case, we could remove the vhost device from data plane(if it is=0A= there) when we receive this message.=0A= Will submit a fix.=0A= So far everything works fine but apparently we need clear spec when to=0A= stop and start vhost device.=0A= =0A= > + dev->mem =3D calloc(1,=0A= cut lines....=0A= =0A= +}=0A= +=0A= +=0A= +/*=0A= + * In vhost-user, when we receive kick message, will test whether virtio= =0A= + * device is ready for packet processing.=0A= + */=0A= +void=0A= +user_set_vring_kick(struct vhost_device_ctx ctx, struct VhostUserMsg *pmsg= )=0A= +{=0A= + struct vhost_vring_file file;=0A= + struct virtio_net *dev =3D get_device(ctx);=0A= +=0A= + file.index =3D pmsg->payload.u64 & VHOST_USER_VRING_IDX_MASK;=0A= + if (pmsg->payload.u64 & VHOST_USER_VRING_NOFD_MASK)=0A= + file.fd =3D -1;=0A= + else=0A= + file.fd =3D pmsg->fds[0];=0A= + RTE_LOG(INFO, VHOST_CONFIG,=0A= + "vring kick idx:%d file:%d\n", file.index, file.fd);=0A= + ops->set_vring_kick(ctx, &file);=0A= +=0A= + if (virtio_is_ready(dev) &&=0A= + !(dev->flags & VIRTIO_DEV_RUNNING))=0A= + notify_ops->new_device(dev);=0A= +}=0A= =0A= When vhost device is started, qemu will call=0A= vhost_net_start_one->vhost_dev_start->vhost_virtqueue_start.=0A= 1. vhost_dev_start send VHOST_SET_MEM_TABLE to vhost backend.=0A= 2. kick fd is sent and only sent to vhost backend in=0A= vhost_virtqueue_start for each queue, through VHOST_SET_VRING_KICK message.= =0A= 2. vhost_net_start_one will send VHOST_NET_SET_BACKEND to vhost=0A= backend if type is TAP.=0A= For vhost-user, as we don't receive SET_BACKEND message, it makes sense=0A= to use VHOST_SET_VRING_KICK as the start signal for vhost device.=0A= =0A= > +=0A= > +/*=0A= > + * when virtio is stopped, qemu will send us the GET_VRING_BASE message.= =0A= > + */=0A= > +int=0A= > +user_get_vring_base(struct vhost_device_ctx ctx,=0A= > + struct vhost_vring_state *state)=0A= > +{=0A= > + struct virtio_net *dev =3D get_device(ctx);=0A= > +=0A= > + /* We have to stop the queue (virtio) if it is running. */=0A= > + if (dev->flags & VIRTIO_DEV_RUNNING)=0A= > + notify_ops->destroy_device(dev);=0A= > +=0A= > + /* Here we are safe to get the last used index */=0A= > + ops->get_vring_base(ctx, state->index, state);=0A= > +=0A= > + RTE_LOG(INFO, VHOST_CONFIG,=0A= > + "vring base idx:%d file:%d\n", state->index, state->num);=0A= > + /*=0A= > + * Based on current qemu vhost-user implementation, this message is=0A= > + * sent and only sent in vhost_vring_stop.=0A= > + * TODO: cleanup the vring, it isn't usable since here.=0A= > + */=0A= > + if (((int)dev->virtqueue[VIRTIO_RXQ]->callfd) >=3D 0) {=0A= > + close(dev->virtqueue[VIRTIO_RXQ]->callfd);=0A= > + dev->virtqueue[VIRTIO_RXQ]->callfd =3D (eventfd_t)-1;=0A= > + }=0A= > + if (((int)dev->virtqueue[VIRTIO_TXQ]->callfd) >=3D 0) {=0A= > + close(dev->virtqueue[VIRTIO_TXQ]->callfd);=0A= > + dev->virtqueue[VIRTIO_TXQ]->callfd =3D (eventfd_t)-1;=0A= > + }=0A= > +=0A= > + return 0;=0A= > +}=0A= =0A= When vhost device is to be stopped(mostly when virtio device status=0A= register is written), vhost_net_stop_one->vhost_dev_stop-> is called.=0A= 1. VHOST_NET_SET_BACKEND message is sent to vhost backend if type is TAP.= =0A= 2. VHOST_GET_VRING_BASE is sent in vhost_virtqueue_stop for each virt queue= .=0A= It makes sense to use VHOST_GET_VRING_BASE as the stop signal for vhost=0A= device.=0A= =0A= > +=0A= > +void=0A= > +user_destroy_device(struct vhost_device_ctx ctx)=0A= > +{=0A= > + struct virtio_net *dev =3D get_device(ctx);=0A= > +=0A= > + if (dev && (dev->flags & VIRTIO_DEV_RUNNING))=0A= > + notify_ops->destroy_device(dev);=0A= > +=0A= > + if (dev && dev->mem) {=0A= > + free_mem_region(dev);=0A= > + free(dev->mem);=0A= > + dev->mem =3D NULL;=0A= > + }=0A= > +}=0A= > diff --git a/lib/librte_vhost/vhost_user/virtio-net-user.h b/lib/librte_v= host/vhost_user/virtio-net-user.h=0A= > new file mode 100644=0A= > index 0000000..df24860=0A= > --- /dev/null=0A= cut lines...=0A=