From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by dpdk.org (Postfix) with ESMTP id E5C101B4FF for ; Fri, 23 Nov 2018 11:28:53 +0100 (CET) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 52ED830821C5; Fri, 23 Nov 2018 10:28:53 +0000 (UTC) Received: from ktraynor.remote.csb (unknown [10.36.118.7]) by smtp.corp.redhat.com (Postfix) with ESMTP id 05FC2413C; Fri, 23 Nov 2018 10:28:51 +0000 (UTC) From: Kevin Traynor To: Darek Stojaczyk Cc: Anatoly Burakov , dpdk stable Date: Fri, 23 Nov 2018 10:26:14 +0000 Message-Id: <20181123102713.17309-10-ktraynor@redhat.com> In-Reply-To: <20181123102713.17309-1-ktraynor@redhat.com> References: <20181123102713.17309-1-ktraynor@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.47]); Fri, 23 Nov 2018 10:28:53 +0000 (UTC) Subject: [dpdk-stable] patch 'vfio: share default container in multi-process' has been queued to stable release 18.08.1 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 10:28:54 -0000 Hi, FYI, your patch has been queued to stable release 18.08.1 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 11/29/18. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. If the code is different (ie: not only metadata diffs), due for example to a change in context or macro names, please double check it. Thanks. Kevin Traynor --- >>From 239dadd17803875567072a6efda6c5a1783b9b61 Mon Sep 17 00:00:00 2001 From: Darek Stojaczyk Date: Wed, 3 Oct 2018 14:39:25 +0200 Subject: [PATCH] vfio: share default container in multi-process [ upstream commit 6bcb7c95fe14544bca9de1a698e4553533192197 ] So far each process in MP used to have a separate container and relied on the primary process to register all memsegs. Mapping external memory via rte_vfio_container_dma_map() in secondary processes was broken, because the default (process-local) container had no groups bound. There was even no way to bind any groups to it, because the container fd was deeply encapsulated within EAL. This patch introduces a new SOCKET_REQ_DEFAULT_CONTAINER message type for MP synchronization, makes all processes within a MP party use a single default container, and hence fixes rte_vfio_container_dma_map() for secondary processes. >>From what I checked this behavior was always the same, but started to be invalid/insufficient once mapping external memory was allowed. While here, fix up the comment on rte_vfio_get_container_fd(). This function always opens a new container, never reuses an old one. Fixes: 73a639085938 ("vfio: allow to map other memory regions") Signed-off-by: Darek Stojaczyk Reviewed-by: Anatoly Burakov --- lib/librte_eal/common/include/rte_vfio.h | 2 +- lib/librte_eal/linuxapp/eal/eal_vfio.c | 49 ++++++++++++++++++- lib/librte_eal/linuxapp/eal/eal_vfio.h | 4 ++ .../linuxapp/eal/eal_vfio_mp_sync.c | 11 +++++ 4 files changed, 64 insertions(+), 2 deletions(-) diff --git a/lib/librte_eal/common/include/rte_vfio.h b/lib/librte_eal/common/include/rte_vfio.h index 5ca13fcce..1064426eb 100644 --- a/lib/librte_eal/common/include/rte_vfio.h +++ b/lib/librte_eal/common/include/rte_vfio.h @@ -228,5 +228,5 @@ rte_vfio_get_group_num(const char *sysfs_base, /** - * Open VFIO container fd or get an existing one + * Open a new VFIO container fd * * This function is only relevant to linux and will return diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c b/lib/librte_eal/linuxapp/eal/eal_vfio.c index 3fd5be79f..c4d80a994 100644 --- a/lib/librte_eal/linuxapp/eal/eal_vfio.c +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c @@ -897,5 +897,13 @@ rte_vfio_enable(const char *modname) } - default_vfio_cfg->vfio_container_fd = rte_vfio_get_container_fd(); + if (internal_config.process_type == RTE_PROC_PRIMARY) { + /* open a new container */ + default_vfio_cfg->vfio_container_fd = + rte_vfio_get_container_fd(); + } else { + /* get the default container from the primary process */ + default_vfio_cfg->vfio_container_fd = + vfio_get_default_container_fd(); + } /* check if we have VFIO driver enabled */ @@ -917,4 +925,43 @@ rte_vfio_is_enabled(const char *modname) } +int +vfio_get_default_container_fd(void) +{ + struct rte_mp_msg mp_req, *mp_rep; + struct rte_mp_reply mp_reply; + struct timespec ts = {.tv_sec = 5, .tv_nsec = 0}; + struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param; + + if (default_vfio_cfg->vfio_enabled) + return default_vfio_cfg->vfio_container_fd; + + if (internal_config.process_type == RTE_PROC_PRIMARY) { + /* if we were secondary process we would try requesting + * container fd from the primary, but we're the primary + * process so just exit here + */ + return -1; + } + + p->req = SOCKET_REQ_DEFAULT_CONTAINER; + strcpy(mp_req.name, EAL_VFIO_MP); + mp_req.len_param = sizeof(*p); + mp_req.num_fds = 0; + + if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 && + mp_reply.nb_received == 1) { + mp_rep = &mp_reply.msgs[0]; + p = (struct vfio_mp_param *)mp_rep->param; + if (p->result == SOCKET_OK && mp_rep->num_fds == 1) { + free(mp_reply.msgs); + return mp_rep->fds[0]; + } + free(mp_reply.msgs); + } + + RTE_LOG(ERR, EAL, " cannot request default container fd\n"); + return -1; +} + const struct vfio_iommu_type * vfio_set_iommu_type(int vfio_container_fd) diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.h b/lib/librte_eal/linuxapp/eal/eal_vfio.h index 68d4750a5..63ae115c3 100644 --- a/lib/librte_eal/linuxapp/eal/eal_vfio.h +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.h @@ -116,4 +116,7 @@ struct vfio_iommu_type { }; +/* get the vfio container that devices are bound to by default */ +int vfio_get_default_container_fd(void); + /* pick IOMMU type. returns a pointer to vfio_iommu_type or NULL for error */ const struct vfio_iommu_type * @@ -130,4 +133,5 @@ int vfio_mp_sync_setup(void); #define SOCKET_REQ_CONTAINER 0x100 #define SOCKET_REQ_GROUP 0x200 +#define SOCKET_REQ_DEFAULT_CONTAINER 0x400 #define SOCKET_OK 0x0 #define SOCKET_NO_FD 0x1 diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c b/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c index 680a24aae..a1e8c834f 100644 --- a/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c +++ b/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c @@ -67,4 +67,15 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer) } break; + case SOCKET_REQ_DEFAULT_CONTAINER: + r->req = SOCKET_REQ_DEFAULT_CONTAINER; + fd = vfio_get_default_container_fd(); + if (fd < 0) + r->result = SOCKET_ERR; + else { + r->result = SOCKET_OK; + reply.num_fds = 1; + reply.fds[0] = fd; + } + break; default: RTE_LOG(ERR, EAL, "vfio received invalid message!\n"); -- 2.19.0 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2018-11-23 10:22:54.506035172 +0000 +++ 0010-vfio-share-default-container-in-multi-process.patch 2018-11-23 10:22:54.000000000 +0000 @@ -1,8 +1,10 @@ -From 6bcb7c95fe14544bca9de1a698e4553533192197 Mon Sep 17 00:00:00 2001 +From 239dadd17803875567072a6efda6c5a1783b9b61 Mon Sep 17 00:00:00 2001 From: Darek Stojaczyk Date: Wed, 3 Oct 2018 14:39:25 +0200 Subject: [PATCH] vfio: share default container in multi-process +[ upstream commit 6bcb7c95fe14544bca9de1a698e4553533192197 ] + So far each process in MP used to have a separate container and relied on the primary process to register all memsegs. @@ -26,7 +28,6 @@ an old one. Fixes: 73a639085938 ("vfio: allow to map other memory regions") -Cc: stable@dpdk.org Signed-off-by: Darek Stojaczyk Reviewed-by: Anatoly Burakov @@ -38,10 +39,10 @@ 4 files changed, 64 insertions(+), 2 deletions(-) diff --git a/lib/librte_eal/common/include/rte_vfio.h b/lib/librte_eal/common/include/rte_vfio.h -index 7d66438b5..cae96fab9 100644 +index 5ca13fcce..1064426eb 100644 --- a/lib/librte_eal/common/include/rte_vfio.h +++ b/lib/librte_eal/common/include/rte_vfio.h -@@ -257,5 +257,5 @@ rte_vfio_get_group_num(const char *sysfs_base, +@@ -228,5 +228,5 @@ rte_vfio_get_group_num(const char *sysfs_base, /** - * Open VFIO container fd or get an existing one @@ -49,10 +50,10 @@ * * This function is only relevant to linux and will return diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c b/lib/librte_eal/linuxapp/eal/eal_vfio.c -index 13a8b1811..0516b1597 100644 +index 3fd5be79f..c4d80a994 100644 --- a/lib/librte_eal/linuxapp/eal/eal_vfio.c +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c -@@ -911,5 +911,13 @@ rte_vfio_enable(const char *modname) +@@ -897,5 +897,13 @@ rte_vfio_enable(const char *modname) } - default_vfio_cfg->vfio_container_fd = rte_vfio_get_container_fd(); @@ -67,7 +68,7 @@ + } /* check if we have VFIO driver enabled */ -@@ -931,4 +939,43 @@ rte_vfio_is_enabled(const char *modname) +@@ -917,4 +925,43 @@ rte_vfio_is_enabled(const char *modname) } +int