From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-it0-f65.google.com (mail-it0-f65.google.com [209.85.214.65]) by dpdk.org (Postfix) with ESMTP id A0C9A5F1B for ; Wed, 28 Mar 2018 07:49:38 +0200 (CEST) Received: by mail-it0-f65.google.com with SMTP id z143-v6so18282607itc.0 for ; Tue, 27 Mar 2018 22:49:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=K5pI422NUhqoKaPaNU1gRdixJuuSXdpwzfE6netu+XE=; b=Vv0hQvU8DLSLyafV9gqIQDJ9dCywoLVsAto33Gtc7wiLYuXBXoyplGwwNhWzuuMoX6 HrbiisuUfSBdO2N3aMhHU8lP8hTDMm+SNUAwWv3wzDB+6LEbxafFGYg4Q9yQdMgGckK/ idWj7z37m9rrf1HHkmq5Plr0jUs6vNBRhMdislwaoH++Xk12UmwFBFZlLZL1Ci+RbNhs xepI4jaI90tGAqxIin+ENDpH8wY0bcYMx9679PSH/0TFt4kNrtDt646FufnIFsOegj2U I52C/4N8RX96z5O9RTHDn/+EiFzWCOGIf6Ry3V5NvlyVnMuu9fgwrt+gxy2T4THTz0tO GkPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=K5pI422NUhqoKaPaNU1gRdixJuuSXdpwzfE6netu+XE=; b=GXA92vMXDpn0kgOG/UqboZeFiVVs6wBVOvb6FPgszhODNfRUqI3NAgloD29nvEOTym FLT3N4g1FOygYA/9sZK/5aw4450qSPLJuxlGrTSAHVuwHXqFugHoNHyhvp5fQlsKJwfH 6hwjpfckku+GFJjfao3UWYvq2NC2TPAr902aVJquEwyMWWgj7GN73sIbAshxja4EILXI io458/CdMF3fb8NWqJEnTfMeaK4siNolxkUES6MTaXZC82HCv3Tkf/uckCF8x7TqFXDa TcDSWE6Z8T4sEwv0vjPUcYBJeQ5/skWlV8HrrKMqeckiLJjSGKVidS1MRK8PrrvH94yN jX0w== X-Gm-Message-State: AElRT7EIEStYMEDTq+a0n2jv0jP9sD0EQmNN7Jag5M2mnJKN4DatTOhU qrF/GORnA9+Qo4dn+E+8kqbDrU97 X-Google-Smtp-Source: AIpwx48R6MG+0OhJOMz2f72NS4kMVQDvfybvT2S3LFohnX1IhBz4+YwjOZiCncM50bHQa5SpOH/v+w== X-Received: by 2002:a24:b541:: with SMTP id j1-v6mr2135529iti.128.1522216178021; Tue, 27 Mar 2018 22:49:38 -0700 (PDT) Received: from local.opencloud.tech.localdomain ([52.229.160.192]) by smtp.gmail.com with ESMTPSA id 40sm1903044iol.39.2018.03.27.22.49.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 27 Mar 2018 22:49:37 -0700 (PDT) From: xiangxia.m.yue@gmail.com To: jianfeng.tan@intel.com Cc: dev@dpdk.org, Tonghao Zhang Date: Tue, 27 Mar 2018 22:49:25 -0700 Message-Id: <1522216165-19666-3-git-send-email-xiangxia.m.yue@gmail.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1522216165-19666-1-git-send-email-xiangxia.m.yue@gmail.com> References: <1522216165-19666-1-git-send-email-xiangxia.m.yue@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH 2/2] vhost: add pipe event for optimizing negotiating X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2018 05:49:39 -0000 From: Tonghao Zhang When vhost—user connects qemu successfully, dpdk will call the vhost_user_add_connection to add unix socket fd to poll. And fdset_add only set the socket fd to a fdentry while poll may sleep now. In a general case, this is no problem. But if we use hot update for vhost-user, most downtime of VMs network is 750+ms. This patch adds pipe event, so after connections are ok, dpdk rebuild the poll immediately. With this patch, the most downtime is 20~30ms. Signed-off-by: Tonghao Zhang --- lib/librte_vhost/fd_man.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++ lib/librte_vhost/fd_man.h | 16 ++++++++++++++++ lib/librte_vhost/socket.c | 14 ++++++++++++++ 3 files changed, 79 insertions(+) diff --git a/lib/librte_vhost/fd_man.c b/lib/librte_vhost/fd_man.c index 181711c..7716757 100644 --- a/lib/librte_vhost/fd_man.c +++ b/lib/librte_vhost/fd_man.c @@ -15,6 +15,7 @@ #include #include "fd_man.h" +#include "vhost.h" #define FDPOLLERR (POLLERR | POLLHUP | POLLNVAL) @@ -272,3 +273,51 @@ return NULL; } + +static void +fdset_pipe_read_cb(int readfd, void *dat __rte_unused, + int *remove __rte_unused) +{ + char charbuf[16]; + read(readfd, charbuf, sizeof(charbuf)); +} + +void +fdset_pipe_uninit(struct fdset *fdset) +{ + fdset_del(fdset, fdset->u.readfd); + close(fdset->u.readfd); + close(fdset->u.writefd); +} + +int +fdset_pipe_init(struct fdset *fdset) +{ + int ret; + + if (pipe(fdset->u.pipefd) < 0) { + RTE_LOG(ERR, VHOST_CONFIG, + "failed to create pipe for vhost fdset\n"); + return -1; + } + + ret = fdset_add(fdset, fdset->u.readfd, + fdset_pipe_read_cb, NULL, NULL); + + if (ret < 0) { + RTE_LOG(ERR, VHOST_CONFIG, + "failed to add pipe readfd %d into vhost server fdset\n", + fdset->u.readfd); + + fdset_pipe_uninit(fdset); + return -1; + } + + return 0; +} + +void +fdset_pipe_notify(struct fdset *fdset) +{ + write(fdset->u.writefd, "1", 1); +} diff --git a/lib/librte_vhost/fd_man.h b/lib/librte_vhost/fd_man.h index 3a9276c..76a42fb 100644 --- a/lib/librte_vhost/fd_man.h +++ b/lib/librte_vhost/fd_man.h @@ -25,6 +25,16 @@ struct fdset { struct fdentry fd[MAX_FDS]; pthread_mutex_t fd_mutex; int num; /* current fd number of this fdset */ + + union pipefds { + struct { + int pipefd[2]; + }; + struct { + int readfd; + int writefd; + }; + } u; }; @@ -37,4 +47,10 @@ int fdset_add(struct fdset *pfdset, int fd, void *fdset_event_dispatch(void *arg); +int fdset_pipe_init(struct fdset *fdset); + +void fdset_pipe_uninit(struct fdset *fdset); + +void fdset_pipe_notify(struct fdset *fdset); + #endif diff --git a/lib/librte_vhost/socket.c b/lib/librte_vhost/socket.c index 95bed78..795239c 100644 --- a/lib/librte_vhost/socket.c +++ b/lib/librte_vhost/socket.c @@ -231,6 +231,8 @@ struct vhost_user { pthread_mutex_lock(&vsocket->conn_mutex); TAILQ_INSERT_TAIL(&vsocket->conn_list, conn, next); pthread_mutex_unlock(&vsocket->conn_mutex); + + fdset_pipe_notify(&vhost_user.fdset); return; err: @@ -829,11 +831,23 @@ struct vhost_device_ops const * return -1; if (fdset_tid == 0) { + /** + * create a pipe which will be waited by poll and notified to + * rebuild the wait list of poll. + */ + if (fdset_pipe_init(&vhost_user.fdset) < 0) { + RTE_LOG(ERR, VHOST_CONFIG, + "failed to create pipe for vhost fdset\n"); + return -1; + } + int ret = pthread_create(&fdset_tid, NULL, fdset_event_dispatch, &vhost_user.fdset); if (ret != 0) { RTE_LOG(ERR, VHOST_CONFIG, "failed to create fdset handling thread"); + + fdset_pipe_uninit(&vhost_user.fdset); return -1; } else { snprintf(thread_name, RTE_MAX_THREAD_NAME_LEN, -- 1.8.3.1