From: "Xie, Huawei" <huawei.xie@intel.com>
To: "Czesnowicz, Przemyslaw" <przemyslaw.czesnowicz@intel.com>,
"dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH v3 09/11] lib/librte_vhost: vhost user support
Date: Fri, 27 Feb 2015 09:42:53 +0000 [thread overview]
Message-ID: <C37D651A908B024F974696C65296B57B0F3A6C5F@SHSMSX101.ccr.corp.intel.com> (raw)
In-Reply-To: <1424712993-73818-10-git-send-email-przemyslaw.czesnowicz@intel.com>
On 2/24/2015 1:36 AM, Czesnowicz, Przemyslaw wrote:
> From: "Xie, Huawei" <huawei.xie@intel.com>
>
> In rte_vhost_driver_register(), vhost unix domain socket listener fd is created
> and added to polled(based on select) fdset.
>
> In rte_vhost_driver_session_start(), fds in the fdset are checked for
> processing. If there is new connection from qemu, connection fd accepted is
> added to polled fdset. The listener and connection fds in the fdset are
> then both checked. When there is message on the connection fd, its
> callback vserver_message_handler is called to process vhost-user messages.
>
> To support identifying which virtio is from which guest VM, we could call
> rte_vhost_driver_register with different socket path. Virtio devices from
> same VM will connect to VM specific socket. The socket path information is
> stored in the virtio_net structure.
>
> Signed-off-by: Huawei Xie <huawei.xie@intel.com>
> Signed-off-by: Przemyslaw Czesnowicz <przemyslaw.czesnowicz@intel.com>
> ---
> lib/librte_vhost/Makefile | 8 +-
> lib/librte_vhost/rte_virtio_net.h | 2 +
> lib/librte_vhost/vhost-net.h | 4 +-
> lib/librte_vhost/vhost_user/vhost-net-user.c | 457 ++++++++++++++++++++++++++
> lib/librte_vhost/vhost_user/vhost-net-user.h | 106 ++++++
> lib/librte_vhost/vhost_user/virtio-net-user.c | 314 ++++++++++++++++++
> lib/librte_vhost/vhost_user/virtio-net-user.h | 49 +++
> lib/librte_vhost/virtio-net.c | 15 +-
> 8 files changed, 948 insertions(+), 7 deletions(-)
> create mode 100644 lib/librte_vhost/vhost_user/vhost-net-user.c
> create mode 100644 lib/librte_vhost/vhost_user/vhost-net-user.h
> create mode 100644 lib/librte_vhost/vhost_user/virtio-net-user.c
> create mode 100644 lib/librte_vhost/vhost_user/virtio-net-user.h
>
> diff --git a/lib/librte_vhost/Makefile b/lib/librte_vhost/Makefile
> index 298e4f8..cac943e 100644
> --- a/lib/librte_vhost/Makefile
> +++ b/lib/librte_vhost/Makefile
cut lines...
> +
> +int
> +user_set_mem_table(struct vhost_device_ctx ctx, struct VhostUserMsg *pmsg)
> +{
> + struct VhostUserMemory memory = pmsg->payload.memory;
> + struct virtio_memory_regions *pregion;
> + uint64_t mapped_address, mapped_size;
> + struct virtio_net *dev;
> + unsigned int idx = 0;
> + struct orig_region_map *pregion_orig;
> + uint64_t alignment;
> +
> + /* unmap old memory regions one by one*/
> + dev = get_device(ctx);
> + if (dev == NULL)
> + return -1;
> +
> + if (dev->mem) {
> + free_mem_region(dev);
> + free(dev->mem);
> + dev->mem = NULL;
> + }
> +
VHOST_SET_MEM_TABLE is sent and only sent to vhost backend in
vhost_dev_start.
In FC21, we don't receive VHOST_GET_VRING_BASE to stop the vhost
device(refer to later comment below). Need root cause the change of
guest virtio driver.
For that case, we could remove the vhost device from data plane(if it is
there) when we receive this message.
Will submit a fix.
So far everything works fine but apparently we need clear spec when to
stop and start vhost device.
> + dev->mem = calloc(1,
cut lines....
+}
+
+
+/*
+ * In vhost-user, when we receive kick message, will test whether virtio
+ * device is ready for packet processing.
+ */
+void
+user_set_vring_kick(struct vhost_device_ctx ctx, struct VhostUserMsg *pmsg)
+{
+ struct vhost_vring_file file;
+ struct virtio_net *dev = get_device(ctx);
+
+ file.index = pmsg->payload.u64 & VHOST_USER_VRING_IDX_MASK;
+ if (pmsg->payload.u64 & VHOST_USER_VRING_NOFD_MASK)
+ file.fd = -1;
+ else
+ file.fd = pmsg->fds[0];
+ RTE_LOG(INFO, VHOST_CONFIG,
+ "vring kick idx:%d file:%d\n", file.index, file.fd);
+ ops->set_vring_kick(ctx, &file);
+
+ if (virtio_is_ready(dev) &&
+ !(dev->flags & VIRTIO_DEV_RUNNING))
+ notify_ops->new_device(dev);
+}
When vhost device is started, qemu will call
vhost_net_start_one->vhost_dev_start->vhost_virtqueue_start.
1. vhost_dev_start send VHOST_SET_MEM_TABLE to vhost backend.
2. kick fd is sent and only sent to vhost backend in
vhost_virtqueue_start for each queue, through VHOST_SET_VRING_KICK message.
2. vhost_net_start_one will send VHOST_NET_SET_BACKEND to vhost
backend if type is TAP.
For vhost-user, as we don't receive SET_BACKEND message, it makes sense
to use VHOST_SET_VRING_KICK as the start signal for vhost device.
> +
> +/*
> + * when virtio is stopped, qemu will send us the GET_VRING_BASE message.
> + */
> +int
> +user_get_vring_base(struct vhost_device_ctx ctx,
> + struct vhost_vring_state *state)
> +{
> + struct virtio_net *dev = get_device(ctx);
> +
> + /* We have to stop the queue (virtio) if it is running. */
> + if (dev->flags & VIRTIO_DEV_RUNNING)
> + notify_ops->destroy_device(dev);
> +
> + /* Here we are safe to get the last used index */
> + ops->get_vring_base(ctx, state->index, state);
> +
> + RTE_LOG(INFO, VHOST_CONFIG,
> + "vring base idx:%d file:%d\n", state->index, state->num);
> + /*
> + * Based on current qemu vhost-user implementation, this message is
> + * sent and only sent in vhost_vring_stop.
> + * TODO: cleanup the vring, it isn't usable since here.
> + */
> + if (((int)dev->virtqueue[VIRTIO_RXQ]->callfd) >= 0) {
> + close(dev->virtqueue[VIRTIO_RXQ]->callfd);
> + dev->virtqueue[VIRTIO_RXQ]->callfd = (eventfd_t)-1;
> + }
> + if (((int)dev->virtqueue[VIRTIO_TXQ]->callfd) >= 0) {
> + close(dev->virtqueue[VIRTIO_TXQ]->callfd);
> + dev->virtqueue[VIRTIO_TXQ]->callfd = (eventfd_t)-1;
> + }
> +
> + return 0;
> +}
When vhost device is to be stopped(mostly when virtio device status
register is written), vhost_net_stop_one->vhost_dev_stop-> is called.
1. VHOST_NET_SET_BACKEND message is sent to vhost backend if type is TAP.
2. VHOST_GET_VRING_BASE is sent in vhost_virtqueue_stop for each virt queue.
It makes sense to use VHOST_GET_VRING_BASE as the stop signal for vhost
device.
> +
> +void
> +user_destroy_device(struct vhost_device_ctx ctx)
> +{
> + struct virtio_net *dev = get_device(ctx);
> +
> + if (dev && (dev->flags & VIRTIO_DEV_RUNNING))
> + notify_ops->destroy_device(dev);
> +
> + if (dev && dev->mem) {
> + free_mem_region(dev);
> + free(dev->mem);
> + dev->mem = NULL;
> + }
> +}
> diff --git a/lib/librte_vhost/vhost_user/virtio-net-user.h b/lib/librte_vhost/vhost_user/virtio-net-user.h
> new file mode 100644
> index 0000000..df24860
> --- /dev/null
cut lines...
next prev parent reply other threads:[~2015-02-27 9:43 UTC|newest]
Thread overview: 41+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-02-12 5:07 [dpdk-dev] [PATCH v2 00/11] qemu vhost-user support Huawei Xie
2015-02-12 5:07 ` [dpdk-dev] [PATCH v2 01/11] lib/librte_vhost: enable VIRTIO_NET_F_CTRL_RX VIRTIO_NET_F_CTRL_RX is dependant on VIRTIO_NET_F_CTRL_VQ. Observed that virtio-net driver in guest would crash with only CTRL_RX enabled Huawei Xie
2015-02-16 8:15 ` Tetsuya Mukawa
2015-02-12 5:07 ` [dpdk-dev] [PATCH v2 02/11] lib/librte_vhost: create vhost_cuse directory and move vhost-net-cdev.c into vhost_cuse Huawei Xie
2015-02-12 5:07 ` [dpdk-dev] [PATCH v2 03/11] lib/librte_vhost: rename vhost-net-cdev.h to vhost-net.h Huawei Xie
2015-02-12 5:07 ` [dpdk-dev] [PATCH v2 04/11] lib/librte_vhost: move fd copying(from qemu process into vhost process) to eventfd_copy.c Huawei Xie
2015-02-12 5:07 ` [dpdk-dev] [PATCH v2 05/11] lib/librte_vhost: copy host_memory_map from virtio-net.c to a new file virtio-net-cdev.c Huawei Xie
2015-02-12 5:07 ` [dpdk-dev] [PATCH v2 06/11] lib/librte_vhost: make host_memory_map a more generic function Huawei Xie
2015-02-12 5:07 ` [dpdk-dev] [PATCH v2 07/11] lib/librte_vhost: implement cuse_set_memory_table Huawei Xie
2015-02-12 5:07 ` [dpdk-dev] [PATCH v2 08/11] lib/librte_vhost: add select based event driven processing Huawei Xie
2015-02-16 17:10 ` Ananyev, Konstantin
2015-02-12 5:07 ` [dpdk-dev] [PATCH v2 09/11] lib/librte_vhost: vhost user support Huawei Xie
2015-02-12 8:26 ` Linhaifeng
2015-02-12 9:28 ` Xie, Huawei
2015-02-12 10:19 ` Linhaifeng
2015-02-12 5:07 ` [dpdk-dev] [PATCH v2 10/11] lib/librte_vhost: support dev->ifname for vhost-user Huawei Xie
2015-02-12 5:07 ` [dpdk-dev] [PATCH v2 11/11] lib/librte_vhost: support dynamically registering vhost server Huawei Xie
2015-02-16 8:17 ` Tetsuya Mukawa
2015-02-16 17:11 ` Ananyev, Konstantin
2015-02-12 5:26 ` [dpdk-dev] [PATCH v2 00/11] qemu vhost-user support Xie, Huawei
2015-02-16 8:19 ` Tetsuya Mukawa
2015-02-22 18:20 ` Thomas Monjalon
2015-02-23 13:53 ` Czesnowicz, Przemyslaw
2015-02-23 14:08 ` Thomas Monjalon
2015-02-23 14:15 ` Czesnowicz, Przemyslaw
2015-02-23 2:50 ` Tetsuya Mukawa
2015-02-23 17:36 ` [dpdk-dev] [PATCH v3 " Przemyslaw Czesnowicz
2015-02-23 17:36 ` [dpdk-dev] [PATCH v3 01/11] lib/librte_vhost: enable VIRTIO_NET_F_CTRL_RX VIRTIO_NET_F_CTRL_RX is dependant on VIRTIO_NET_F_CTRL_VQ. Observed that virtio-net driver in guest would crash with only CTRL_RX enabled Przemyslaw Czesnowicz
2015-02-23 17:36 ` [dpdk-dev] [PATCH v3 02/11] lib/librte_vhost: create vhost_cuse directory and move vhost-net-cdev.c into vhost_cuse Przemyslaw Czesnowicz
2015-02-23 17:36 ` [dpdk-dev] [PATCH v3 03/11] lib/librte_vhost: rename vhost-net-cdev.h to vhost-net.h Przemyslaw Czesnowicz
2015-02-23 17:36 ` [dpdk-dev] [PATCH v3 04/11] lib/librte_vhost: move fd copying(from qemu process into vhost process) to eventfd_copy.c Przemyslaw Czesnowicz
2015-02-23 17:36 ` [dpdk-dev] [PATCH v3 05/11] lib/librte_vhost: copy host_memory_map from virtio-net.c to a new file virtio-net-cdev.c Przemyslaw Czesnowicz
2015-02-23 17:36 ` [dpdk-dev] [PATCH v3 06/11] lib/librte_vhost: make host_memory_map a more generic function Przemyslaw Czesnowicz
2015-02-23 17:36 ` [dpdk-dev] [PATCH v3 07/11] lib/librte_vhost: implement cuse_set_memory_table Przemyslaw Czesnowicz
2015-02-23 17:36 ` [dpdk-dev] [PATCH v3 08/11] lib/librte_vhost: add select based event driven processing Przemyslaw Czesnowicz
2015-02-23 17:36 ` [dpdk-dev] [PATCH v3 09/11] lib/librte_vhost: vhost user support Przemyslaw Czesnowicz
2015-02-27 9:42 ` Xie, Huawei [this message]
2015-02-23 17:36 ` [dpdk-dev] [PATCH v3 10/11] lib/librte_vhost: support dev->ifname for vhost-user Przemyslaw Czesnowicz
2015-02-23 17:36 ` [dpdk-dev] [PATCH v3 11/11] lib/librte_vhost: support dynamically registering vhost server Przemyslaw Czesnowicz
2015-02-24 0:41 ` [dpdk-dev] [PATCH v3 00/11] qemu vhost-user support Thomas Monjalon
2015-02-25 5:55 ` Xie, Huawei
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=C37D651A908B024F974696C65296B57B0F3A6C5F@SHSMSX101.ccr.corp.intel.com \
--to=huawei.xie@intel.com \
--cc=dev@dpdk.org \
--cc=przemyslaw.czesnowicz@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).