From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mailout3.w1.samsung.com (mailout3.w1.samsung.com [210.118.77.13]) by dpdk.org (Postfix) with ESMTP id AC0C13979 for ; Thu, 21 Jul 2016 10:21:44 +0200 (CEST) Received: from eucpsbgm1.samsung.com (unknown [203.254.199.244]) by mailout3.w1.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTP id <0OAN004RKOK6BT30@mailout3.w1.samsung.com> for dev@dpdk.org; Thu, 21 Jul 2016 09:21:42 +0100 (BST) X-AuditID: cbfec7f4-f796c6d000001486-39-5790861696e1 Received: from eusync4.samsung.com ( [203.254.199.214]) by eucpsbgm1.samsung.com (EUCPMTA) with SMTP id 48.E5.05254.61680975; Thu, 21 Jul 2016 09:21:42 +0100 (BST) Received: from imaximets.rnd.samsung.ru ([106.109.129.180]) by eusync4.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTPA id <0OAN00DKBOK0UX30@eusync4.samsung.com>; Thu, 21 Jul 2016 09:21:42 +0100 (BST) From: Ilya Maximets To: dev@dpdk.org, Huawei Xie , Yuanhan Liu Cc: Dyasly Sergey , Heetae Ahn , Thomas Monjalon , Ilya Maximets Date: Thu, 21 Jul 2016 11:21:15 +0300 Message-id: <1469089275-15209-1-git-send-email-i.maximets@samsung.com> X-Mailer: git-send-email 2.7.4 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrFJMWRmVeSWpSXmKPExsVy+t/xa7pibRPCDfatMrB492k7k8W0z7fZ LdpnnmWyuNL+k91i8mwpiy+bprNZXJ9wgdWB3eNi/x1Gj18LlrJ6LN7zkslj3slAj74tqxgD WKO4bFJSczLLUov07RK4Mnp2X2ArOKhZ8ezaE/YGxvmKXYycHBICJhIPbnxng7DFJC7cWw9k c3EICSxllNj8dwoLSEJIoJVJYt7KUhCbTUBH4tTqI4wgtohAgsSR/b9ZQRqYBVYxSlx/PIkd JCEsYC7Rf/cd0CQODhYBVYkLywtBwrwCbhK9F86yQiyTk7h5rpN5AiP3AkaGVYyiqaXJBcVJ 6bmGesWJucWleel6yfm5mxghIfJlB+PiY1aHGAU4GJV4eBNW9ocLsSaWFVfmHmKU4GBWEuH9 3zIhXIg3JbGyKrUoP76oNCe1+BCjNAeLkjjv3F3vQ4QE0hNLUrNTUwtSi2CyTBycUg2M3lnN bcWtjQ/Vzj8S3RJ79uqBdcwiGRmFkq8lTTacW3dTcZYwl+6cyBPMyja2WUuOPdyoJu4Z/Plz ouPRKzWHuGdrinAmCrIESE1c0O2j3zPNXXiJQpZbreONgh0lN64+PH60V2jrnygV13UX/Je+ MFWKYrMNMDiboX0sNyLp2lP3nQvyjlQosRRnJBpqMRcVJwIA9Wh62A0CAAA= Subject: [dpdk-dev] [PATCH] vhost: fix connect hang in client mode X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jul 2016 08:21:44 -0000 If something abnormal happened to QEMU, 'connect()' can block calling thread (e.g. main thread of OVS) forever or for a really long time. This can break whole application or block the reconnection thread. Example with OVS: ovs_rcu(urcu2)|WARN|blocked 512000 ms waiting for main to quiesce (gdb) bt #0 connect () from /lib64/libpthread.so.0 #1 vhost_user_create_client (vsocket=0xa816e0) #2 rte_vhost_driver_register #3 netdev_dpdk_vhost_user_construct #4 netdev_open (name=0xa664b0 "vhost1") [...] #11 main Fix that by setting non-blocking mode for client sockets for connection. Fixes: 64ab701c3d1e ("vhost: add vhost-user client mode") Signed-off-by: Ilya Maximets --- This was reproduced with current QEMU master branch (commit 1ecfb24da987b862f) + patch-set "vhost-user reconnect fixes" (https://lists.nongnu.org/archive/html/qemu-devel/2016-07/msg01547.html). OVS was patched to support client mode: http://openvswitch.org/pipermail/dev/2016-July/074972.html Following script forces QEMU to fail to initialize vhost because disconnection occures while device not fully configured: while true do ovs-vsctl set Interface vhost1 ofport_request=125 ovs-vsctl set Interface vhost1 ofport_request=126 done As a result: QEMU still works, network interface broken and OVS main thread stalled inside 'connect()'. lib/librte_vhost/vhost_user/vhost-net-user.c | 77 ++++++++++++++++++++++++++-- 1 file changed, 73 insertions(+), 4 deletions(-) diff --git a/lib/librte_vhost/vhost_user/vhost-net-user.c b/lib/librte_vhost/vhost_user/vhost-net-user.c index f0f92f8..08a5f6b 100644 --- a/lib/librte_vhost/vhost_user/vhost-net-user.c +++ b/lib/librte_vhost/vhost_user/vhost-net-user.c @@ -43,6 +43,7 @@ #include #include #include +#include #include #include @@ -449,6 +450,14 @@ create_unix_socket(const char *path, struct sockaddr_un *un, bool is_server) RTE_LOG(INFO, VHOST_CONFIG, "vhost-user %s: socket created, fd: %d\n", is_server ? "server" : "client", fd); + if (!is_server && fcntl(fd, F_SETFL, O_NONBLOCK)) { + RTE_LOG(ERR, VHOST_CONFIG, + "vhost-user: can't set nonblocking mode for socket, fd: " + "%d (%s)\n", fd, strerror(errno)); + close(fd); + return -1; + } + memset(un, 0, sizeof(*un)); un->sun_family = AF_UNIX; strncpy(un->sun_path, path, sizeof(un->sun_path)); @@ -516,9 +525,58 @@ struct vhost_user_reconnect_list { static struct vhost_user_reconnect_list reconn_list; static pthread_t reconn_tid; +static int +vhost_user_connect_nonblock(int fd, struct sockaddr *un, size_t sz) +{ + int ret, flags, so_error; + socklen_t len = sizeof(so_error); + fd_set fdset; + /* 5 second timeout */ + struct timeval tv = {.tv_sec = 5, .tv_usec = 0}; + + errno = EINVAL; + + ret = connect(fd, un, sz); + if (ret == -1 && errno != EINPROGRESS) + return -1; + if (ret == 0) + goto connected; + + FD_ZERO(&fdset); + FD_SET(fd, &fdset); + + ret = select(fd + 1, NULL, &fdset, NULL, &tv); + if (!ret) + errno = ETIMEDOUT; + if (ret != 1) + return -1; + + ret = getsockopt(fd, SOL_SOCKET, SO_ERROR, &so_error, &len); + if (ret < 0 || so_error) { + if (!ret) + errno = so_error; + return -1; + } + +connected: + flags = fcntl(fd, F_GETFL, 0); + if (flags < 0) { + RTE_LOG(ERR, VHOST_CONFIG, + "can't get flags for connfd %d\n", fd); + return -2; + } + if ((flags & O_NONBLOCK) && fcntl(fd, F_SETFL, flags & ~O_NONBLOCK)) { + RTE_LOG(ERR, VHOST_CONFIG, + "can't disable nonblocking on fd %d\n", fd); + return -2; + } + return 0; +} + static void * vhost_user_client_reconnect(void *arg __rte_unused) { + int ret; struct vhost_user_reconnect *reconn, *next; while (1) { @@ -532,13 +590,23 @@ vhost_user_client_reconnect(void *arg __rte_unused) reconn != NULL; reconn = next) { next = TAILQ_NEXT(reconn, next); - if (connect(reconn->fd, (struct sockaddr *)&reconn->un, - sizeof(reconn->un)) < 0) + ret = vhost_user_connect_nonblock(reconn->fd, + (struct sockaddr *)&reconn->un, + sizeof(reconn->un)); + if (ret == -2) { + close(reconn->fd); + RTE_LOG(ERR, VHOST_CONFIG, + "reconnection for fd %d failed\n", + reconn->fd); + goto remove_fd; + } + if (ret == -1) continue; RTE_LOG(INFO, VHOST_CONFIG, "%s: connected\n", reconn->vsocket->path); vhost_user_add_connection(reconn->fd, reconn->vsocket); +remove_fd: TAILQ_REMOVE(&reconn_list.head, reconn, next); free(reconn); } @@ -579,7 +647,8 @@ vhost_user_create_client(struct vhost_user_socket *vsocket) if (fd < 0) return -1; - ret = connect(fd, (struct sockaddr *)&un, sizeof(un)); + ret = vhost_user_connect_nonblock(fd, (struct sockaddr *)&un, + sizeof(un)); if (ret == 0) { vhost_user_add_connection(fd, vsocket); return 0; @@ -589,7 +658,7 @@ vhost_user_create_client(struct vhost_user_socket *vsocket) "failed to connect to %s: %s\n", path, strerror(errno)); - if (!vsocket->reconnect) { + if (ret == -2 || !vsocket->reconnect) { close(fd); return -1; } -- 2.7.4