From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mailout1.w1.samsung.com (mailout1.w1.samsung.com [210.118.77.11]) by dpdk.org (Postfix) with ESMTP id D0D3337AC for ; Thu, 21 Jul 2016 15:19:42 +0200 (CEST) Received: from eucpsbgm1.samsung.com (unknown [203.254.199.244]) by mailout1.w1.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTP id <0OAO00B332CTW9B0@mailout1.w1.samsung.com> for dev@dpdk.org; Thu, 21 Jul 2016 14:19:41 +0100 (BST) X-AuditID: cbfec7f4-f796c6d000001486-fd-5790cbed53ca Received: from eusync4.samsung.com ( [203.254.199.214]) by eucpsbgm1.samsung.com (EUCPMTA) with SMTP id AF.E6.05254.DEBC0975; Thu, 21 Jul 2016 14:19:41 +0100 (BST) Received: from imaximets.rnd.samsung.ru ([106.109.129.180]) by eusync4.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTPA id <0OAO00BSX2CPVNB0@eusync4.samsung.com>; Thu, 21 Jul 2016 14:19:41 +0100 (BST) From: Ilya Maximets To: dev@dpdk.org, Huawei Xie , Yuanhan Liu Cc: Dyasly Sergey , Heetae Ahn , Thomas Monjalon , Ilya Maximets Date: Thu, 21 Jul 2016 16:19:35 +0300 Message-id: <1469107175-1216-1-git-send-email-i.maximets@samsung.com> X-Mailer: git-send-email 2.7.4 In-reply-to: <1469089275-15209-1-git-send-email-i.maximets@samsung.com> References: <1469089275-15209-1-git-send-email-i.maximets@samsung.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprELMWRmVeSWpSXmKPExsVy+t/xa7pvT08IN1hzgM3i3aftTBbTPt9m t2ifeZbJ4kr7T3aLybOlLL5sms5mcX3CBVYHdo+L/XcYPX4tWMrqsXjPSyaPeScDPfq2rGIM YI3isklJzcksSy3St0vgylj4/QRTwVW1iuPTHrE2ME6U72Lk5JAQMJFY2rybFcIWk7hwbz1b FyMXh5DAUkaJzS/+soMkhARamSTubzcAsdkEdCROrT7CCGKLCCRIHNn/mxWkgVlgFaPE9ceT gBo4OIQFrCReH08EqWERUJV4t+IPC4jNK+Aq8eHif3aIZXISN891MoPYnALuEqd63jFB7HKT WP3rEMsERt4FjAyrGEVTS5MLipPScw31ihNzi0vz0vWS83M3MUJC6ssOxsXHrA4xCnAwKvHw JqzsDxdiTSwrrsw9xCjBwawkwrvs2IRwId6UxMqq1KL8+KLSnNTiQ4zSHCxK4rxzd70PERJI TyxJzU5NLUgtgskycXBKNTBuWWO/5vaD1Zy/m3oE9JYn7tq2VDzf+fqGmNnPBeKnu90zLtvJ d/Koya+O6StrHNdJJfhuL2Xj2dG8fdnuyA/5k0+t/fhNTe7LkiUhZ6pXzmEVmmLKIFq0PXYz m/mt5ycu8aTWavicX6i8rKZgwxW+Fb6nLUIN9p1I+5azKdlorfw5R9470+L/KrEUZyQaajEX FScCAP2b4qclAgAA Subject: [dpdk-dev] [PATCH v3] vhost: fix connect hang in client mode X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jul 2016 13:19:42 -0000 If something abnormal happened to QEMU, 'connect()' can block calling thread (e.g. main thread of OVS) forever or for a really long time. This can break whole application or block the reconnection thread. Example with OVS: ovs_rcu(urcu2)|WARN|blocked 512000 ms waiting for main to quiesce (gdb) bt #0 connect () from /lib64/libpthread.so.0 #1 vhost_user_create_client (vsocket=0xa816e0) #2 rte_vhost_driver_register #3 netdev_dpdk_vhost_user_construct #4 netdev_open (name=0xa664b0 "vhost1") [...] #11 main Fix that by setting non-blocking mode for client sockets for connection. Fixes: 64ab701c3d1e ("vhost: add vhost-user client mode") Signed-off-by: Ilya Maximets --- This was reproduced with current QEMU master branch (commit 1ecfb24da987b862f) + patch-set "vhost-user reconnect fixes" (https://lists.nongnu.org/archive/html/qemu-devel/2016-07/msg01547.html). OVS was patched to support client mode: http://openvswitch.org/pipermail/dev/2016-July/074972.html Following script forces QEMU to fail to initialize vhost because disconnection occures while device not fully configured: while true do ovs-vsctl set Interface vhost1 ofport_request=125 ovs-vsctl set Interface vhost1 ofport_request=126 done As a result: QEMU still works, network interface broken and OVS main thread stalled inside 'connect()'. Version 3: * Removed unnecessary 'getsockopt()' check. Version 2: * EINPROGRESS not checked. EISCONN checked instead on the next iteration of reconnection loop. lib/librte_vhost/vhost_user/vhost-net-user.c | 52 +++++++++++++++++++++++++--- 1 file changed, 48 insertions(+), 4 deletions(-) diff --git a/lib/librte_vhost/vhost_user/vhost-net-user.c b/lib/librte_vhost/vhost_user/vhost-net-user.c index 62c5ec3..b35594d 100644 --- a/lib/librte_vhost/vhost_user/vhost-net-user.c +++ b/lib/librte_vhost/vhost_user/vhost-net-user.c @@ -43,6 +43,7 @@ #include #include #include +#include #include #include @@ -449,6 +450,14 @@ create_unix_socket(const char *path, struct sockaddr_un *un, bool is_server) RTE_LOG(INFO, VHOST_CONFIG, "vhost-user %s: socket created, fd: %d\n", is_server ? "server" : "client", fd); + if (!is_server && fcntl(fd, F_SETFL, O_NONBLOCK)) { + RTE_LOG(ERR, VHOST_CONFIG, + "vhost-user: can't set nonblocking mode for socket, fd: " + "%d (%s)\n", fd, strerror(errno)); + close(fd); + return -1; + } + memset(un, 0, sizeof(*un)); un->sun_family = AF_UNIX; strncpy(un->sun_path, path, sizeof(un->sun_path)); @@ -516,9 +525,33 @@ struct vhost_user_reconnect_list { static struct vhost_user_reconnect_list reconn_list; static pthread_t reconn_tid; +static int +vhost_user_connect_nonblock(int fd, struct sockaddr *un, size_t sz) +{ + int ret, flags; + + ret = connect(fd, un, sz); + if (ret < 0 && errno != EISCONN) + return -1; + + flags = fcntl(fd, F_GETFL, 0); + if (flags < 0) { + RTE_LOG(ERR, VHOST_CONFIG, + "can't get flags for connfd %d\n", fd); + return -2; + } + if ((flags & O_NONBLOCK) && fcntl(fd, F_SETFL, flags & ~O_NONBLOCK)) { + RTE_LOG(ERR, VHOST_CONFIG, + "can't disable nonblocking on fd %d\n", fd); + return -2; + } + return 0; +} + static void * vhost_user_client_reconnect(void *arg __rte_unused) { + int ret; struct vhost_user_reconnect *reconn, *next; while (1) { @@ -532,13 +565,23 @@ vhost_user_client_reconnect(void *arg __rte_unused) reconn != NULL; reconn = next) { next = TAILQ_NEXT(reconn, next); - if (connect(reconn->fd, (struct sockaddr *)&reconn->un, - sizeof(reconn->un)) < 0) + ret = vhost_user_connect_nonblock(reconn->fd, + (struct sockaddr *)&reconn->un, + sizeof(reconn->un)); + if (ret == -2) { + close(reconn->fd); + RTE_LOG(ERR, VHOST_CONFIG, + "reconnection for fd %d failed\n", + reconn->fd); + goto remove_fd; + } + if (ret == -1) continue; RTE_LOG(INFO, VHOST_CONFIG, "%s: connected\n", reconn->vsocket->path); vhost_user_add_connection(reconn->fd, reconn->vsocket); +remove_fd: TAILQ_REMOVE(&reconn_list.head, reconn, next); free(reconn); } @@ -579,7 +622,8 @@ vhost_user_create_client(struct vhost_user_socket *vsocket) if (fd < 0) return -1; - ret = connect(fd, (struct sockaddr *)&un, sizeof(un)); + ret = vhost_user_connect_nonblock(fd, (struct sockaddr *)&un, + sizeof(un)); if (ret == 0) { vhost_user_add_connection(fd, vsocket); return 0; @@ -589,7 +633,7 @@ vhost_user_create_client(struct vhost_user_socket *vsocket) "failed to connect to %s: %s\n", path, strerror(errno)); - if (!vsocket->reconnect) { + if (ret == -2 || !vsocket->reconnect) { close(fd); return -1; } -- 2.7.4