From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 7E5481B7D8 for ; Wed, 4 Apr 2018 05:47:40 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 03 Apr 2018 20:47:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.48,404,1517904000"; d="scan'208";a="217452389" Received: from fmsmsx107.amr.corp.intel.com ([10.18.124.205]) by fmsmga005.fm.intel.com with ESMTP; 03 Apr 2018 20:47:37 -0700 Received: from FMSMSX109.amr.corp.intel.com (10.18.116.9) by fmsmsx107.amr.corp.intel.com (10.18.124.205) with Microsoft SMTP Server (TLS) id 14.3.319.2; Tue, 3 Apr 2018 20:47:37 -0700 Received: from shsmsx104.ccr.corp.intel.com (10.239.4.70) by fmsmsx109.amr.corp.intel.com (10.18.116.9) with Microsoft SMTP Server (TLS) id 14.3.319.2; Tue, 3 Apr 2018 20:47:36 -0700 Received: from shsmsx103.ccr.corp.intel.com ([169.254.4.241]) by SHSMSX104.ccr.corp.intel.com ([169.254.5.43]) with mapi id 14.03.0319.002; Wed, 4 Apr 2018 11:47:34 +0800 From: "Tan, Jianfeng" To: "Yang, Zhiyong" , "dev@dpdk.org" CC: "maxime.coquelin@redhat.com" , "thomas@monjalon.net" , "Wang, Zhihong" , "Bie, Tiwei" Thread-Topic: [PATCH v4 1/1] net/virtio-user: add support for server mode Thread-Index: AQHTy0YlxmzhmdCr70KRSjAJf9NK1qPvJusAgABHTwCAAIfWsA== Date: Wed, 4 Apr 2018 03:47:34 +0000 Message-ID: References: <20180321030343.64399-1-zhiyong.yang@intel.com> <20180403122009.52876-1-zhiyong.yang@intel.com> <20180403122009.52876-2-zhiyong.yang@intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v4 1/1] net/virtio-user: add support for server mode X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Apr 2018 03:47:41 -0000 > -----Original Message----- > From: Yang, Zhiyong > Sent: Wednesday, April 4, 2018 11:32 AM > To: Tan, Jianfeng; dev@dpdk.org > Cc: maxime.coquelin@redhat.com; thomas@monjalon.net; Wang, Zhihong; > Bie, Tiwei > Subject: RE: [PATCH v4 1/1] net/virtio-user: add support for server mode >=20 > Hi Jiafeng, >=20 > Thanks for your review and comments firstly. > Reply inline. >=20 > > -----Original Message----- > > From: Tan, Jianfeng > > Sent: Tuesday, April 3, 2018 11:16 PM > > To: Yang, Zhiyong ; dev@dpdk.org > > Cc: maxime.coquelin@redhat.com; thomas@monjalon.net; Wang, Zhihong > > ; Bie, Tiwei > > Subject: Re: [PATCH v4 1/1] net/virtio-user: add support for server mod= e > > > > > > > > On 4/3/2018 8:20 PM, zhiyong.yang@intel.com wrote: > > > virtio-user adds support for server mode in this patch. > > > > > > Client mode vhost-user startup firstly, server mode virtio-user > > > startups and creates the socket file to exchange vhost messages. > > > > > > If the connection is broken, client mode vhost-user can support to > > > reconnect virtio-user. > > > > > > Server mode virtio-user supports many times' vhost-user reconnections > > > with the same parameter configurations. > > > > > > Release note is updated in the patch. > > > > With current implementation, we have to enable LSC; or no chance to > accept > > the coming connection. We shall point this out. > > > Ok. >=20 > > And if possible, split this patch into multiple patches. > > > How to split? I think it can be split into three: - One for new dev parameter. - One for server socket setup. - One for LSC handling. But it's not a big patch anyway, I'm OK if you think better to keep in one = patch. >=20 > > > > > > Signed-off-by: Zhiyong Yang > > > --- > > > doc/guides/rel_notes/release_18_05.rst | 6 ++ > > > drivers/net/virtio/virtio_user/vhost_user.c | 64 ++++++++++++= -- > > > drivers/net/virtio/virtio_user/virtio_user_dev.c | 45 +++++++--- > > > drivers/net/virtio/virtio_user/virtio_user_dev.h | 4 + > > > drivers/net/virtio/virtio_user_ethdev.c | 103 > > +++++++++++++++++++++-- > > > 5 files changed, 194 insertions(+), 28 deletions(-) > > > > > > diff --git a/doc/guides/rel_notes/release_18_05.rst > > > b/doc/guides/rel_notes/release_18_05.rst > > > index 9cc77f893..f8897b2e9 100644 > > > --- a/doc/guides/rel_notes/release_18_05.rst > > > +++ b/doc/guides/rel_notes/release_18_05.rst > > > @@ -58,6 +58,12 @@ New Features > > > * Added support for NVGRE, VXLAN and GENEVE filters in flow API. > > > * Added support for DROP action in flow API. > > > > > > +* **Added support for virtio-user server mode.** > > > + In a container environment if the vhost-user backend restarts, > > > +there's no way > > > + for it to reconnect to virtio-user. To address this, support for > > > +server mode > > > + is added. In this mode the socket file is created by virtio-user, > > > +which the > > > + backend connects to. This means that if the backend restarts, it > > > +can reconnect > > > + to virtio-user and continue communications. > > > > > > API Changes > > > ----------- > > > diff --git a/drivers/net/virtio/virtio_user/vhost_user.c > > > b/drivers/net/virtio/virtio_user/vhost_user.c > > > index 91c6449bb..1b3401d4f 100644 > > > --- a/drivers/net/virtio/virtio_user/vhost_user.c > > > +++ b/drivers/net/virtio/virtio_user/vhost_user.c > > > @@ -378,6 +378,50 @@ vhost_user_sock(struct virtio_user_dev *dev, > > > return 0; > > > } > > > > > > +static void > > > +virtio_user_set_block(int fd, bool enabled) > > > > This is only used once, no need to abstract it into a function? >=20 > Ok. >=20 > > > > > +{ > > > + int f; > > > + > > > + f =3D fcntl(fd, F_GETFL); > > > + if (enabled) > > > + fcntl(fd, F_SETFL, f & ~O_NONBLOCK); > > > + else > > > + fcntl(fd, F_SETFL, f | O_NONBLOCK); } > > > + > > > +#define MAX_VIRTIO_USER_BACKLOG 128 > > > > We only allow one connection from vhost-user, so how about just make > the > > backlog queue length as 1? > > > > > +static int > > > +virtio_user_start_server(struct virtio_user_dev *dev, struct > > > +sockaddr_un *un) { > > > + int ret; > > > + int fd =3D dev->listenfd; > > > + int connectfd; > > > + > > > + ret =3D bind(fd, (struct sockaddr *)un, sizeof(*un)); > > > + if (ret < 0) { > > > + PMD_DRV_LOG(ERR, "failed to bind to %s: %s; remove it and > > try again\n", > > > + dev->path, strerror(errno)); > > > + goto err; > > > + } > > > + ret =3D listen(fd, MAX_VIRTIO_USER_BACKLOG); > > > + if (ret < 0) > > > + goto err; > > > + > > > + connectfd =3D accept(fd, NULL, NULL); > > > + if (connectfd >=3D 0) > > > + dev->connected =3D true; > > > + else > > > + goto err; > > > > if (connectfd < 0) > > goto err; > > > > dev->connected =3D true; > > dev->vhostfd =3D connectfd; > > ... >=20 > Ok. >=20 > > > > > + > > > + dev->vhostfd =3D connectfd; > > > + virtio_user_set_block(connectfd, true); > > > + return 0; > > > +err: > > > + close(dev->listenfd); > > > + return -1; > > > +} > > > + > > > /** > > > * Set up environment to talk with a vhost user backend. > > > * > > > @@ -390,6 +434,7 @@ vhost_user_setup(struct virtio_user_dev *dev) > > > { > > > int fd; > > > int flag; > > > + int ret =3D 0; > > > struct sockaddr_un un; > > > > > > fd =3D socket(AF_UNIX, SOCK_STREAM, 0); @@ -405,14 +450,21 @@ > > > vhost_user_setup(struct virtio_user_dev *dev) > > > memset(&un, 0, sizeof(un)); > > > un.sun_family =3D AF_UNIX; > > > snprintf(un.sun_path, sizeof(un.sun_path), "%s", dev->path); > > > - if (connect(fd, (struct sockaddr *)&un, sizeof(un)) < 0) { > > > - PMD_DRV_LOG(ERR, "connect error, %s", strerror(errno)); > > > - close(fd); > > > - return -1; > > > + > > > + if (dev->is_server) { > > > + dev->listenfd =3D fd; > > > + ret =3D virtio_user_start_server(dev, &un); > > > + } else { > > > + dev->vhostfd =3D fd; > > > + if (connect(fd, (struct sockaddr *)&un, sizeof(un)) < 0) { > > > + PMD_DRV_LOG(ERR, "connect error, %s", > > strerror(errno)); > > > + close(fd); > > > + return -1; > > > + } > > > + dev->connected =3D true; > > > } > > > > > > - dev->vhostfd =3D fd; > > > - return 0; > > > + return ret; > > > } > > > > > > static int > > > diff --git a/drivers/net/virtio/virtio_user/virtio_user_dev.c > > > b/drivers/net/virtio/virtio_user/virtio_user_dev.c > > > index f90fee9e5..dd9fa9bdf 100644 > > > --- a/drivers/net/virtio/virtio_user/virtio_user_dev.c > > > +++ b/drivers/net/virtio/virtio_user/virtio_user_dev.c > > > @@ -142,6 +142,9 @@ int virtio_user_stop_device(struct > virtio_user_dev > > *dev) > > > { > > > uint32_t i; > > > > > > + if (!dev->connected) > > > + return -1; > > > + > > > for (i =3D 0; i < dev->max_queue_pairs; ++i) > > > dev->ops->enable_qp(dev, i, 0); > > > > > > @@ -267,21 +270,27 @@ virtio_user_dev_setup(struct virtio_user_dev > > *dev) > > > dev->vhostfds =3D NULL; > > > dev->tapfds =3D NULL; > > > > Add a check here: > > if (dev->is_server && !is_vhost_user_by_type(dev->path)) > > return error; >=20 > Ok. >=20 > > > > > > > > - if (is_vhost_user_by_type(dev->path)) { > > > - dev->ops =3D &ops_user; > > > + if (dev->is_server) { > > > + dev->ops =3D &ops_user;/* server mode only supports vhost > > user*/ > > > } else { > > > - dev->ops =3D &ops_kernel; > > > - > > > - dev->vhostfds =3D malloc(dev->max_queue_pairs * > > sizeof(int)); > > > - dev->tapfds =3D malloc(dev->max_queue_pairs * sizeof(int)); > > > - if (!dev->vhostfds || !dev->tapfds) { > > > - PMD_INIT_LOG(ERR, "Failed to malloc"); > > > - return -1; > > > - } > > > - > > > - for (q =3D 0; q < dev->max_queue_pairs; ++q) { > > > - dev->vhostfds[q] =3D -1; > > > - dev->tapfds[q] =3D -1; > > > + if (is_vhost_user_by_type(dev->path)) { > > > + dev->ops =3D &ops_user; > > > + } else { > > > + dev->ops =3D &ops_kernel; > > > + > > > + dev->vhostfds =3D malloc(dev->max_queue_pairs * > > > + sizeof(int)); > > > + dev->tapfds =3D malloc(dev->max_queue_pairs * > > > + sizeof(int)); > > > + if (!dev->vhostfds || !dev->tapfds) { > > > + PMD_INIT_LOG(ERR, "Failed to malloc"); > > > + return -1; > > > + } > > > + > > > + for (q =3D 0; q < dev->max_queue_pairs; ++q) { > > > + dev->vhostfds[q] =3D -1; > > > + dev->tapfds[q] =3D -1; > > > + } > > > } > > > } > > > > > > @@ -388,6 +397,11 @@ virtio_user_dev_uninit(struct virtio_user_dev > > > *dev) > > > > > > close(dev->vhostfd); > > > > > > + if (dev->is_server && dev->listenfd >=3D 0) { > > > + close(dev->listenfd); > > > + dev->listenfd =3D -1; > > > + } > > > + dev->connected =3D false; > > > if (dev->vhostfds) { > > > for (i =3D 0; i < dev->max_queue_pairs; ++i) > > > close(dev->vhostfds[i]); > > > @@ -396,6 +410,9 @@ virtio_user_dev_uninit(struct virtio_user_dev > *dev) > > > } > > > > > > free(dev->ifname); > > > + > > > + if (dev->is_server) > > > + unlink(dev->path); > > > } > > > > > > static uint8_t > > > diff --git a/drivers/net/virtio/virtio_user/virtio_user_dev.h > > > b/drivers/net/virtio/virtio_user/virtio_user_dev.h > > > index 64467b4f9..68056720d 100644 > > > --- a/drivers/net/virtio/virtio_user/virtio_user_dev.h > > > +++ b/drivers/net/virtio/virtio_user/virtio_user_dev.h > > > @@ -6,6 +6,7 @@ > > > #define _VIRTIO_USER_DEV_H > > > > > > #include > > > +#include > > > #include "../virtio_pci.h" > > > #include "../virtio_ring.h" > > > #include "vhost.h" > > > @@ -13,6 +14,9 @@ > > > struct virtio_user_dev { > > > /* for vhost_user backend */ > > > int vhostfd; > > > + int listenfd; /* listening fd */ > > > + bool connected; /* connection status */ > > > > Seems not necessary to add this field; as the connection status can be > > deduced from (vhostfd>=3D0) >=20 > Ok, remove it. >=20 > > > > > + bool is_server; /* server or client mode */ > > > > > > /* for vhost_kernel backend */ > > > char *ifname; > > > diff --git a/drivers/net/virtio/virtio_user_ethdev.c > > > b/drivers/net/virtio/virtio_user_ethdev.c > > > index 263649006..5b8c8e291 100644 > > > --- a/drivers/net/virtio/virtio_user_ethdev.c > > > +++ b/drivers/net/virtio/virtio_user_ethdev.c > > > @@ -24,15 +24,76 @@ > > > #define virtio_user_get_dev(hw) \ > > > ((struct virtio_user_dev *)(hw)->virtio_user_dev) > > > > > > +static void > > > +virtio_user_server_reconnection(struct virtio_user_dev *dev) > > > > s/reconnection/reconnect? > > >=20 > Ok, Good Suggestion. >=20 > > > +{ > > > + int ret; > > > + int flag; > > > + int connectfd; > > > + struct rte_eth_dev *eth_dev =3D &rte_eth_devices[dev->port_id]; > > > + > > > + if (dev->connected) > > > + return; > > > + > > > + connectfd =3D accept(dev->listenfd, NULL, NULL); > > > + if (connectfd < 0) > > > + return; > > > + > > > + dev->vhostfd =3D connectfd; > > > + flag =3D fcntl(connectfd, F_GETFD); > > > + fcntl(connectfd, F_SETFL, flag & ~O_NONBLOCK); > > > + > > > + ret =3D virtio_user_start_device(dev); > > > + if (ret < 0) > > > + return; > > > + > > > + if (eth_dev->data->dev_flags & RTE_ETH_DEV_INTR_LSC) { > > > + if (rte_intr_disable(eth_dev->intr_handle) < 0) { > > > + PMD_DRV_LOG(ERR, "interrupt disable failed"); > > > + return; > > > + } > > > + rte_intr_callback_unregister(eth_dev->intr_handle, > > > + virtio_interrupt_handler, > > > + eth_dev); > > > + eth_dev->intr_handle->fd =3D connectfd; > > > + rte_intr_callback_register(eth_dev->intr_handle, > > > + virtio_interrupt_handler, eth_dev); > > > + > > > + if (rte_intr_enable(eth_dev->intr_handle) < 0) { > > > + PMD_DRV_LOG(ERR, "interrupt enable failed"); > > > + return; > > > + } > > > + } > > > + dev->connected =3D true; > > > + PMD_INIT_LOG(NOTICE, "server mode virtio-user reconnection > > > +succeeds!"); } > > > + > > > static void > > > virtio_user_delayed_handler(void *param) > > > { > > > struct virtio_hw *hw =3D (struct virtio_hw *)param; > > > - struct rte_eth_dev *dev =3D &rte_eth_devices[hw->port_id]; > > > + struct rte_eth_dev *eth_dev =3D &rte_eth_devices[hw->port_id]; > > > + struct virtio_user_dev *dev =3D virtio_user_get_dev(hw); > > > > > > - rte_intr_callback_unregister(dev->intr_handle, > > > - virtio_interrupt_handler, > > > - dev); > > > + if (rte_intr_disable(eth_dev->intr_handle) < 0) { > > > + PMD_DRV_LOG(ERR, "interrupt disable failed"); > > > + return; > > > + } > > > + rte_intr_callback_unregister(eth_dev->intr_handle, > > > + virtio_interrupt_handler, eth_dev); > > > + if (dev->is_server) { > > > + if (dev->vhostfd >=3D 0) { > > > + close(dev->vhostfd); > > > + dev->vhostfd =3D -1; > > > + } > > > + eth_dev->intr_handle->fd =3D dev->listenfd; > > > + rte_intr_callback_register(eth_dev->intr_handle, > > > + virtio_interrupt_handler, eth_dev); > > > + if (rte_intr_enable(eth_dev->intr_handle) < 0) { > > > + PMD_DRV_LOG(ERR, "interrupt enable failed"); > > > + return; > > > + } > > > + } > > > } > > > > > > static void > > > @@ -65,8 +126,7 @@ virtio_user_read_dev_config(struct virtio_hw *hw, > > size_t offset, > > > r =3D recv(dev->vhostfd, buf, 128, MSG_PEEK); > > > > As server mode and the connection is not connected, vhostfd is -1 now, > then > > r < 0 and errno is EBADF, how could it go into server handling in the "= else if" > > block? > > >=20 > I think I don't catch you. When server mode, if vhostfd =3D -1, > if (dev->vhostfd >=3D 0) is false, so, the code will goto to check else = if (dev->is_server) > to handle server mode. I'm sorry, I overlooked the " if (dev->vhostfd >=3D 0)", you are right! Thanks, Jianfeng >=20 > > > if (r =3D=3D 0 || (r < 0 && errno !=3D EAGAIN)) { > > > dev->status &=3D (~VIRTIO_NET_S_LINK_UP); > > > - PMD_DRV_LOG(ERR, "virtio-user port %u is > > down", > > > - hw->port_id); > > > + > > > /* Only client mode is available now. Once > > the > > > > Can you also correct this note as we support server mode now? >=20 > Ok. > > > > > * connection is broken, it can never be up > > > * again. Besides, this function could be called > > @@ -74,9 > > > +134,14 @@ virtio_user_read_dev_config(struct virtio_hw *hw, size_t > > offset, > > > * callback cannot be unregistered here, set > > an > > > * alarm to do it. > > > */ > > > - rte_eal_alarm_set(1, > > > + if (dev->connected) { > > > + dev->connected =3D false; > > > + PMD_DRV_LOG(ERR, "virtio-user > > port %u is down", > > > + hw->port_id); > > > + rte_eal_alarm_set(1, > > > > > virtio_user_delayed_handler, > > > (void *)hw); > > > + } > > > } else { > > > dev->status |=3D VIRTIO_NET_S_LINK_UP; > > > } > > > @@ -85,7 +150,10 @@ virtio_user_read_dev_config(struct virtio_hw > *hw, > > size_t offset, > > > PMD_DRV_LOG(ERR, "error clearing > > O_NONBLOCK flag"); > > > return; > > > } > > > - } > > > + > > > + } else if (dev->is_server) > > > + virtio_user_server_reconnection(dev); > > > + > > > *(uint16_t *)dst =3D dev->status; > > > } > > > > > > @@ -278,12 +346,15 @@ static const char *valid_args[] =3D { > > > VIRTIO_USER_ARG_QUEUE_SIZE, > > > #define VIRTIO_USER_ARG_INTERFACE_NAME "iface" > > > VIRTIO_USER_ARG_INTERFACE_NAME, > > > +#define VIRTIO_USER_ARG_SERVER_MODE "server" > > > + VIRTIO_USER_ARG_SERVER_MODE, > > > NULL > > > }; > > > > > > #define VIRTIO_USER_DEF_CQ_EN 0 > > > #define VIRTIO_USER_DEF_Q_NUM 1 > > > #define VIRTIO_USER_DEF_Q_SZ 256 > > > +#define VIRTIO_USER_DEF_SERVER_MODE 0 > > > > > > static int > > > get_string_arg(const char *key __rte_unused, @@ -378,10 +449,12 @@ > > > virtio_user_pmd_probe(struct rte_vdev_device *dev) > > > uint64_t queues =3D VIRTIO_USER_DEF_Q_NUM; > > > uint64_t cq =3D VIRTIO_USER_DEF_CQ_EN; > > > uint64_t queue_size =3D VIRTIO_USER_DEF_Q_SZ; > > > + uint64_t server_mode =3D VIRTIO_USER_DEF_SERVER_MODE; > > > char *path =3D NULL; > > > char *ifname =3D NULL; > > > char *mac_addr =3D NULL; > > > int ret =3D -1; > > > + struct virtio_user_dev *vu_dev =3D NULL; > > > > > > kvlist =3D rte_kvargs_parse(rte_vdev_device_args(dev), valid_args)= ; > > > if (!kvlist) { > > > @@ -445,6 +518,15 @@ virtio_user_pmd_probe(struct rte_vdev_device > > *dev) > > > } > > > } > > > > > > + if (rte_kvargs_count(kvlist, VIRTIO_USER_ARG_SERVER_MODE) =3D=3D 1) > > { > > > + if (rte_kvargs_process(kvlist, > > VIRTIO_USER_ARG_SERVER_MODE, > > > + &get_integer_arg, &server_mode) < 0) { > > > + PMD_INIT_LOG(ERR, "error to parse %s", > > > + VIRTIO_USER_ARG_SERVER_MODE); > > > + goto end; > > > + } > > > + } > > > + > > > if (rte_kvargs_count(kvlist, VIRTIO_USER_ARG_CQ_NUM) =3D=3D 1) { > > > if (rte_kvargs_process(kvlist, VIRTIO_USER_ARG_CQ_NUM, > > > &get_integer_arg, &cq) < 0) { @@ -476,6 > > +558,11 @@ > > > virtio_user_pmd_probe(struct rte_vdev_device *dev) > > > } > > > > > > hw =3D eth_dev->data->dev_private; > > > + vu_dev =3D virtio_user_get_dev(hw); > > > + if (server_mode =3D=3D 1) > > > + vu_dev->is_server =3D true; > > > + else > > > + vu_dev->is_server =3D false; > > > if (virtio_user_dev_init(hw->virtio_user_dev, path, queues, > > cq, > > > queue_size, mac_addr, &ifname) < 0) { > > > PMD_INIT_LOG(ERR, "virtio_user_dev_init fails");