From: "Wang, Xiao W" <xiao.w.wang@intel.com>
To: "Burakov, Anatoly" <anatoly.burakov@intel.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>,
"maxime.coquelin@redhat.com" <maxime.coquelin@redhat.com>,
"Wang, Zhihong" <zhihong.wang@intel.com>,
"Bie, Tiwei" <tiwei.bie@intel.com>,
"Tan, Jianfeng" <jianfeng.tan@intel.com>,
"Liang, Cunming" <cunming.liang@intel.com>,
"Daly, Dan" <dan.daly@intel.com>,
"thomas@monjalon.net" <thomas@monjalon.net>,
"Chen, Junjie J" <junjie.j.chen@intel.com>,
"Yigit, Ferruh" <ferruh.yigit@intel.com>
Subject: Re: [dpdk-dev] [PATCH v7 2/5] vfio: add multi container support
Date: Mon, 16 Apr 2018 12:44:13 +0000 [thread overview]
Message-ID: <B7F2E978279D1D49A3034B7786DACF406F899823@SHSMSX101.ccr.corp.intel.com> (raw)
In-Reply-To: <acf5f0de-3480-a7b7-0a14-0500120471f8@intel.com>
Hi Anatoly,
> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Monday, April 16, 2018 6:03 PM
> To: Wang, Xiao W <xiao.w.wang@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>
> Cc: dev@dpdk.org; maxime.coquelin@redhat.com; Wang, Zhihong
> <zhihong.wang@intel.com>; Bie, Tiwei <tiwei.bie@intel.com>; Tan, Jianfeng
> <jianfeng.tan@intel.com>; Liang, Cunming <cunming.liang@intel.com>; Daly,
> Dan <dan.daly@intel.com>; thomas@monjalon.net; Chen, Junjie J
> <junjie.j.chen@intel.com>
> Subject: Re: [PATCH v7 2/5] vfio: add multi container support
>
> On 15-Apr-18 4:33 PM, Xiao Wang wrote:
> > This patch adds APIs to support container create/destroy and device
> > bind/unbind with a container. It also provides API for IOMMU programing
> > on a specified container.
> >
> > A driver could use "rte_vfio_create_container" helper to create a
>
> ^^ wrong API name in commit message :)
Thanks for the catch. Will fix it.
>
> > new container from eal, use "rte_vfio_bind_group" to bind a device
> > to the newly created container. During rte_vfio_setup_device the
> > container bound with the device will be used for IOMMU setup.
> >
> > Signed-off-by: Junjie Chen <junjie.j.chen@intel.com>
> > Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
> > Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> > Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
> > ---
> > lib/librte_eal/bsdapp/eal/eal.c | 52 +++++
> > lib/librte_eal/common/include/rte_vfio.h | 119 ++++++++++++
> > lib/librte_eal/linuxapp/eal/eal_vfio.c | 316
> +++++++++++++++++++++++++++++++
> > lib/librte_eal/rte_eal_version.map | 6 +
> > 4 files changed, 493 insertions(+)
> >
> > diff --git a/lib/librte_eal/bsdapp/eal/eal.c b/lib/librte_eal/bsdapp/eal/eal.c
> > index 727adc5d2..c5106d0d6 100644
> > --- a/lib/librte_eal/bsdapp/eal/eal.c
> > +++ b/lib/librte_eal/bsdapp/eal/eal.c
> > @@ -769,6 +769,14 @@ int rte_vfio_noiommu_is_enabled(void);
> > int rte_vfio_clear_group(int vfio_group_fd);
> > int rte_vfio_dma_map(uint64_t vaddr, uint64_t iova, uint64_t len);
> > int rte_vfio_dma_unmap(uint64_t vaddr, uint64_t iova, uint64_t len);
> > +int rte_vfio_container_create(void);
> > +int rte_vfio_container_destroy(int container_fd);
> > +int rte_vfio_bind_group(int container_fd, int iommu_group_no);
> > +int rte_vfio_unbind_group(int container_fd, int iommu_group_no);
>
> Maybe have these under "container" too? e.g.
> rte_vfio_container_group_bind/unbind? Seems like it would be more
> consistent that way - anything to do with custom containers would be
> under rte_vfio_container_* namespace.
Agree.
>
> > +int rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
> > + uint64_t iova, uint64_t len);
> > +int rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr,
> > + uint64_t iova, uint64_t len);
> >
> > int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
> > __rte_unused const char *dev_addr,
> > @@ -818,3 +826,47 @@ rte_vfio_dma_unmap(uint64_t __rte_unused vaddr,
> uint64_t __rte_unused iova,
> > {
> > return -1;
> > }
> > +
>
> <...>
>
> > diff --git a/lib/librte_eal/common/include/rte_vfio.h
> b/lib/librte_eal/common/include/rte_vfio.h
> > index d26ab01cb..0c1509b29 100644
> > --- a/lib/librte_eal/common/include/rte_vfio.h
> > +++ b/lib/librte_eal/common/include/rte_vfio.h
> > @@ -168,6 +168,125 @@ rte_vfio_dma_map(uint64_t vaddr, uint64_t iova,
> uint64_t len);
> > int __rte_experimental
> > rte_vfio_dma_unmap(uint64_t vaddr, uint64_t iova, uint64_t len);
> >
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change, or be removed, without prior
> notice
> > + *
> > + * Create a new container for device binding.
>
> I would add a note that any newly allocated DPDK memory will not be
> mapped into these containers by default.
Will add it.
>
> > + *
> > + * @return
> > + * the container fd if successful
> > + * <0 if failed
> > + */
> > +int __rte_experimental
> > +rte_vfio_container_create(void);
> > +
>
> <...>
>
> > + * 0 if successful
> > + * <0 if failed
> > + */
> > +int __rte_experimental
> > +rte_vfio_unbind_group(int container_fd, int iommu_group_no);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change, or be removed, without prior
> notice
> > + *
> > + * Perform dma mapping for devices in a conainer.
>
> Here and in other places: "dma" should be DMA, and typo: "conainer" :)
>
> I think you should also add a note to the original API (not this one,
> but the old one) that DMA maps done via that API will only apply to
> default container and will not apply to any of the containers created
> via container_create(). IOW, documentation should make it clear that if
> you use this functionality, you're on your own and you have to manage
> your own DMA mappings for any containers you create.
OK, will add note to clearly describe it.
>
> > + *
> > + * @param container_fd
> > + * the specified container fd
> > + *
> > + * @param vaddr
> > + * Starting virtual address of memory to be mapped.
> > + *
>
> <...>
>
> > +
> > +int __rte_experimental
> > +rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t
> iova,
> > + uint64_t len)
> > +{
> > + struct user_mem_map *new_map;
> > + struct vfio_config *vfio_cfg;
> > + struct user_mem_maps *user_mem_maps;
> > + int ret = 0;
> > +
> > + if (len == 0) {
> > + rte_errno = EINVAL;
> > + return -1;
> > + }
> > +
> > + vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
> > + if (vfio_cfg == NULL) {
> > + RTE_LOG(ERR, EAL, "Invalid container fd\n");
> > + return -1;
> > + }
> > +
> > + user_mem_maps = &vfio_cfg->mem_maps;
> > + rte_spinlock_recursive_lock(&user_mem_maps->lock);
> > + if (user_mem_maps->n_maps == VFIO_MAX_USER_MEM_MAPS) {
> > + RTE_LOG(ERR, EAL, "No more space for user mem maps\n");
> > + rte_errno = ENOMEM;
> > + ret = -1;
> > + goto out;
> > + }
> > + /* map the entry */
> > + if (vfio_dma_mem_map(vfio_cfg, vaddr, iova, len, 1)) {
> > + /* technically, this will fail if there are currently no devices
> > + * plugged in, even if a device were added later, this mapping
> > + * might have succeeded. however, since we cannot verify if
> this
> > + * is a valid mapping without having a device attached,
> consider
> > + * this to be unsupported, because we can't just store any old
> > + * mapping and pollute list of active mappings willy-nilly.
> > + */
> > + RTE_LOG(ERR, EAL, "Couldn't map new region for DMA\n");
> > + ret = -1;
> > + goto out;
> > + }
> > + /* create new user mem map entry */
> > + new_map = &user_mem_maps->maps[user_mem_maps->n_maps++];
> > + new_map->addr = vaddr;
> > + new_map->iova = iova;
> > + new_map->len = len;
> > +
> > + compact_user_maps(user_mem_maps);
> > +out:
> > + rte_spinlock_recursive_unlock(&user_mem_maps->lock);
> > + return ret;
>
> Please correct me if i'm wrong, but it looks like you've just duplicated
> the code for rte_vfio_dma_map() here and made a few small changes. It
> would be better if you moved most of this into a static function (e.g.
> static int container_dma_map(vfio_cfg, vaddr, iova, len)) and called it
> with either default vfio_cfg from rte_vfio_dma_map, or found vfio_cfg
> from rte_vfio_container_dma_map. Same applies to function below.
Agree, will do it in v8.
BRs,
Xiao
>
> > +}
> > +
> > +int __rte_experimental
> > +rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t
> iova,
> > + uint64_t len)
> > +{
> > + struct user_mem_map *map, *new_map = NULL;
> > + struct vfio_config *vfio_cfg;
> > + struct user_mem_maps *user_mem_maps;
> > + int ret = 0;
> > +
> > + if (len == 0) {
> > + rte_errno = EINVAL;
> > + return -1;
> > + }
> > +
>
> <...>
>
> --
> Thanks,
> Anatoly
next prev parent reply other threads:[~2018-04-16 12:44 UTC|newest]
Thread overview: 98+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-03-09 23:08 [dpdk-dev] [PATCH 0/3] add ifcvf driver Xiao Wang
2018-03-09 23:08 ` [dpdk-dev] [PATCH 1/3] eal/vfio: add support for multiple container Xiao Wang
2018-03-14 12:08 ` Burakov, Anatoly
2018-03-15 16:49 ` Wang, Xiao W
2018-03-09 23:08 ` [dpdk-dev] [PATCH 2/3] bus/pci: expose sysfs parsing API Xiao Wang
2018-03-14 11:19 ` Burakov, Anatoly
2018-03-14 13:30 ` Gaëtan Rivet
2018-03-15 16:49 ` Wang, Xiao W
2018-03-15 17:19 ` Gaëtan Rivet
2018-03-19 1:31 ` Wang, Xiao W
2018-03-21 13:21 ` [dpdk-dev] [PATCH v2 0/3] add ifcvf driver Xiao Wang
2018-03-21 13:21 ` [dpdk-dev] [PATCH v2 1/3] eal/vfio: add support for multiple container Xiao Wang
2018-03-21 20:32 ` Thomas Monjalon
2018-03-21 21:37 ` Gaëtan Rivet
2018-03-22 3:00 ` Wang, Xiao W
2018-03-21 13:21 ` [dpdk-dev] [PATCH v2 2/3] bus/pci: expose sysfs parsing API Xiao Wang
2018-03-21 20:44 ` Thomas Monjalon
2018-03-22 2:46 ` Wang, Xiao W
2018-03-21 13:21 ` [dpdk-dev] [PATCH v2 3/3] net/ifcvf: add ifcvf driver Xiao Wang
2018-03-21 20:52 ` Thomas Monjalon
2018-03-23 10:39 ` Wang, Xiao W
2018-03-21 20:57 ` Maxime Coquelin
2018-03-23 10:37 ` Wang, Xiao W
2018-03-22 8:51 ` Ferruh Yigit
2018-03-22 17:23 ` Wang, Xiao W
2018-03-31 2:29 ` [dpdk-dev] [PATCH v3 0/3] add ifcvf vdpa driver Xiao Wang
2018-03-31 2:29 ` [dpdk-dev] [PATCH v3 1/4] eal/vfio: add support for multiple container Xiao Wang
2018-03-31 11:06 ` Maxime Coquelin
2018-03-31 2:29 ` [dpdk-dev] [PATCH v3 2/4] net/virtio: skip device probe in vdpa mode Xiao Wang
2018-03-31 11:13 ` Maxime Coquelin
2018-03-31 13:16 ` Thomas Monjalon
2018-04-02 4:08 ` Wang, Xiao W
2018-03-31 2:29 ` [dpdk-dev] [PATCH v3 3/4] net/ifcvf: add ifcvf vdpa driver Xiao Wang
2018-03-31 11:26 ` Maxime Coquelin
2018-04-03 9:38 ` Wang, Xiao W
2018-04-04 14:40 ` [dpdk-dev] [PATCH v4 0/4] " Xiao Wang
2018-04-04 14:40 ` [dpdk-dev] [PATCH v4 1/4] eal/vfio: add multiple container support Xiao Wang
2018-04-05 18:06 ` [dpdk-dev] [PATCH v5 0/4] add ifcvf vdpa driver Xiao Wang
2018-04-05 18:06 ` [dpdk-dev] [PATCH v5 1/4] eal/vfio: add multiple container support Xiao Wang
2018-04-05 18:06 ` [dpdk-dev] [PATCH v5 2/4] net/virtio: skip device probe in vdpa mode Xiao Wang
2018-04-11 18:58 ` Ferruh Yigit
2018-04-05 18:07 ` [dpdk-dev] [PATCH v5 3/4] net/ifcvf: add ifcvf vdpa driver Xiao Wang
2018-04-11 18:58 ` Ferruh Yigit
2018-04-12 7:19 ` [dpdk-dev] [PATCH v6 0/4] " Xiao Wang
2018-04-12 7:19 ` [dpdk-dev] [PATCH v6 1/4] eal/vfio: add multiple container support Xiao Wang
2018-04-12 14:03 ` Burakov, Anatoly
2018-04-12 16:07 ` Wang, Xiao W
2018-04-12 16:24 ` Burakov, Anatoly
2018-04-13 9:18 ` Wang, Xiao W
2018-04-15 15:33 ` [dpdk-dev] [PATCH v7 0/5] add ifcvf vdpa driver Xiao Wang
2018-04-15 15:33 ` [dpdk-dev] [PATCH v7 1/5] vfio: extend data structure for multi container Xiao Wang
2018-04-16 10:02 ` Burakov, Anatoly
2018-04-16 12:22 ` Wang, Xiao W
2018-04-16 15:34 ` [dpdk-dev] [PATCH v8 0/5] add ifcvf vdpa driver Xiao Wang
2018-04-16 15:34 ` [dpdk-dev] [PATCH v8 1/5] vfio: extend data structure for multi container Xiao Wang
2018-04-16 15:56 ` Burakov, Anatoly
2018-04-16 15:34 ` [dpdk-dev] [PATCH v8 2/5] vfio: add multi container support Xiao Wang
2018-04-16 15:58 ` Burakov, Anatoly
2018-04-17 7:06 ` [dpdk-dev] [PATCH v9 0/5] add ifcvf vdpa driver Xiao Wang
2018-04-17 7:06 ` [dpdk-dev] [PATCH v9 1/5] vfio: extend data structure for multi container Xiao Wang
2018-04-17 7:06 ` [dpdk-dev] [PATCH v9 2/5] vfio: add multi container support Xiao Wang
2018-04-17 7:06 ` [dpdk-dev] [PATCH v9 3/5] net/virtio: skip device probe in vdpa mode Xiao Wang
2018-04-17 7:06 ` [dpdk-dev] [PATCH v9 4/5] net/ifcvf: add ifcvf vdpa driver Xiao Wang
2018-04-17 7:06 ` [dpdk-dev] [PATCH v9 5/5] doc: add ifcvf driver document and release note Xiao Wang
2018-04-17 11:13 ` [dpdk-dev] [PATCH v9 0/5] add ifcvf vdpa driver Ferruh Yigit
2018-04-16 15:34 ` [dpdk-dev] [PATCH v8 3/5] net/virtio: skip device probe in vdpa mode Xiao Wang
2018-04-16 15:34 ` [dpdk-dev] [PATCH v8 4/5] net/ifcvf: add ifcvf vdpa driver Xiao Wang
2018-04-16 15:34 ` [dpdk-dev] [PATCH v8 5/5] doc: add ifcvf driver document and release note Xiao Wang
2018-04-16 16:36 ` [dpdk-dev] [PATCH v8 0/5] add ifcvf vdpa driver Ferruh Yigit
2018-04-16 18:07 ` Thomas Monjalon
2018-04-17 5:36 ` Wang, Xiao W
2018-04-15 15:33 ` [dpdk-dev] [PATCH v7 2/5] vfio: add multi container support Xiao Wang
2018-04-16 10:03 ` Burakov, Anatoly
2018-04-16 12:44 ` Wang, Xiao W [this message]
2018-04-15 15:33 ` [dpdk-dev] [PATCH v7 3/5] net/virtio: skip device probe in vdpa mode Xiao Wang
2018-04-15 15:33 ` [dpdk-dev] [PATCH v7 4/5] net/ifcvf: add ifcvf vdpa driver Xiao Wang
2018-04-15 15:33 ` [dpdk-dev] [PATCH v7 5/5] doc: add ifcvf driver document and release note Xiao Wang
2018-04-12 7:19 ` [dpdk-dev] [PATCH v6 2/4] net/virtio: skip device probe in vdpa mode Xiao Wang
2018-04-12 7:19 ` [dpdk-dev] [PATCH v6 3/4] net/ifcvf: add ifcvf vdpa driver Xiao Wang
2018-04-12 7:19 ` [dpdk-dev] [PATCH v6 4/4] doc: add ifcvf driver document and release note Xiao Wang
2018-04-05 18:07 ` [dpdk-dev] [PATCH v5 " Xiao Wang
2018-04-11 18:59 ` [dpdk-dev] [PATCH v5 0/4] add ifcvf vdpa driver Ferruh Yigit
2018-04-12 5:47 ` Wang, Xiao W
2018-04-04 14:40 ` [dpdk-dev] [PATCH v4 2/4] net/virtio: skip device probe in vdpa mode Xiao Wang
2018-04-04 14:40 ` [dpdk-dev] [PATCH v4 3/4] net/ifcvf: add ifcvf vdpa driver Xiao Wang
2018-04-04 14:40 ` [dpdk-dev] [PATCH v4 4/4] doc: add ifcvf driver document and release note Xiao Wang
2018-03-31 2:29 ` [dpdk-dev] [PATCH v3 4/4] net/ifcvf: add " Xiao Wang
2018-03-31 11:28 ` Maxime Coquelin
2018-03-09 23:08 ` [dpdk-dev] [PATCH 3/3] net/ifcvf: add ifcvf driver Xiao Wang
2018-03-10 18:23 ` [dpdk-dev] [PATCH 0/3] " Maxime Coquelin
2018-03-15 16:49 ` Wang, Xiao W
2018-03-21 20:47 ` Maxime Coquelin
2018-03-23 10:27 ` Wang, Xiao W
2018-03-25 9:51 ` Maxime Coquelin
2018-03-26 9:05 ` Wang, Xiao W
2018-03-26 13:29 ` Maxime Coquelin
2018-03-27 4:40 ` Wang, Xiao W
2018-03-27 5:09 ` Maxime Coquelin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=B7F2E978279D1D49A3034B7786DACF406F899823@SHSMSX101.ccr.corp.intel.com \
--to=xiao.w.wang@intel.com \
--cc=anatoly.burakov@intel.com \
--cc=cunming.liang@intel.com \
--cc=dan.daly@intel.com \
--cc=dev@dpdk.org \
--cc=ferruh.yigit@intel.com \
--cc=jianfeng.tan@intel.com \
--cc=junjie.j.chen@intel.com \
--cc=maxime.coquelin@redhat.com \
--cc=thomas@monjalon.net \
--cc=tiwei.bie@intel.com \
--cc=zhihong.wang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).