From: Nithin Dabilpuram <ndabilpuram@marvell.com> To: <anatoly.burakov@intel.com>, David Christensen <drc@linux.vnet.ibm.com>, <david.marchand@redhat.com> Cc: <jerinj@marvell.com>, <dev@dpdk.org>, Nithin Dabilpuram <ndabilpuram@marvell.com>, <stable@dpdk.org> Subject: [dpdk-stable] [PATCH v6 1/4] vfio: revert changes for map contiguous areas in one go Date: Fri, 18 Dec 2020 00:36:01 +0530 Message-ID: <20201217190604.29803-2-ndabilpuram@marvell.com> (raw) In-Reply-To: <20201217190604.29803-1-ndabilpuram@marvell.com> In order to save DMA entries limited by kernel both for externel memory and hugepage memory, an attempt was made to map physically contiguous memory in one go. This cannot be done as VFIO IOMMU type1 does not support partially unmapping a previously mapped memory region while Heap can request for multi page mapping and partial unmapping. Hence for going back to old method of mapping/unmapping at memseg granularity, this commit reverts commit d1c7c0cdf7ba ("vfio: map contiguous areas in one go") Also add documentation on what module parameter needs to be used to increase the per-container dma map limit for VFIO. Fixes: d1c7c0cdf7ba ("vfio: map contiguous areas in one go") Cc: anatoly.burakov@intel.com Cc: stable@dpdk.org Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com> Acked-by: David Christensen <drc@linux.vnet.ibm.com> --- doc/guides/linux_gsg/linux_drivers.rst | 10 ++++++ lib/librte_eal/linux/eal_vfio.c | 59 +++++----------------------------- 2 files changed, 18 insertions(+), 51 deletions(-) diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst index 90635a4..9a662a7 100644 --- a/doc/guides/linux_gsg/linux_drivers.rst +++ b/doc/guides/linux_gsg/linux_drivers.rst @@ -25,6 +25,16 @@ To make use of VFIO, the ``vfio-pci`` module must be loaded: VFIO kernel is usually present by default in all distributions, however please consult your distributions documentation to make sure that is the case. +For DMA mapping of either external memory or hugepages, VFIO interface is used. +VFIO does not support partial unmap of once mapped memory. Hence DPDK's memory is +mapped in hugepage granularity or system page granularity. Number of DMA +mappings is limited by kernel with user locked memory limit of a process(rlimit) +for system/hugepage memory. Another per-container overall limit applicable both +for external memory and system memory was added in kernel 5.1 defined by +VFIO module parameter ``dma_entry_limit`` with a default value of 64K. +When application is out of DMA entries, these limits need to be adjusted to +increase the allowed limit. + Since Linux version 5.7, the ``vfio-pci`` module supports the creation of virtual functions. After the PF is bound to ``vfio-pci`` module, diff --git a/lib/librte_eal/linux/eal_vfio.c b/lib/librte_eal/linux/eal_vfio.c index 0500824..64b134d 100644 --- a/lib/librte_eal/linux/eal_vfio.c +++ b/lib/librte_eal/linux/eal_vfio.c @@ -517,11 +517,9 @@ static void vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len, void *arg __rte_unused) { - rte_iova_t iova_start, iova_expected; struct rte_memseg_list *msl; struct rte_memseg *ms; size_t cur_len = 0; - uint64_t va_start; msl = rte_mem_virt2memseg_list(addr); @@ -539,63 +537,22 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len, /* memsegs are contiguous in memory */ ms = rte_mem_virt2memseg(addr, msl); - - /* - * This memory is not guaranteed to be contiguous, but it still could - * be, or it could have some small contiguous chunks. Since the number - * of VFIO mappings is limited, and VFIO appears to not concatenate - * adjacent mappings, we have to do this ourselves. - * - * So, find contiguous chunks, then map them. - */ - va_start = ms->addr_64; - iova_start = iova_expected = ms->iova; while (cur_len < len) { - bool new_contig_area = ms->iova != iova_expected; - bool last_seg = (len - cur_len) == ms->len; - bool skip_last = false; - - /* only do mappings when current contiguous area ends */ - if (new_contig_area) { - if (type == RTE_MEM_EVENT_ALLOC) - vfio_dma_mem_map(default_vfio_cfg, va_start, - iova_start, - iova_expected - iova_start, 1); - else - vfio_dma_mem_map(default_vfio_cfg, va_start, - iova_start, - iova_expected - iova_start, 0); - va_start = ms->addr_64; - iova_start = ms->iova; - } /* some memory segments may have invalid IOVA */ if (ms->iova == RTE_BAD_IOVA) { RTE_LOG(DEBUG, EAL, "Memory segment at %p has bad IOVA, skipping\n", ms->addr); - skip_last = true; + goto next; } - iova_expected = ms->iova + ms->len; + if (type == RTE_MEM_EVENT_ALLOC) + vfio_dma_mem_map(default_vfio_cfg, ms->addr_64, + ms->iova, ms->len, 1); + else + vfio_dma_mem_map(default_vfio_cfg, ms->addr_64, + ms->iova, ms->len, 0); +next: cur_len += ms->len; ++ms; - - /* - * don't count previous segment, and don't attempt to - * dereference a potentially invalid pointer. - */ - if (skip_last && !last_seg) { - iova_expected = iova_start = ms->iova; - va_start = ms->addr_64; - } else if (!skip_last && last_seg) { - /* this is the last segment and we're not skipping */ - if (type == RTE_MEM_EVENT_ALLOC) - vfio_dma_mem_map(default_vfio_cfg, va_start, - iova_start, - iova_expected - iova_start, 1); - else - vfio_dma_mem_map(default_vfio_cfg, va_start, - iova_start, - iova_expected - iova_start, 0); - } } } -- 2.8.4
next prev parent reply other threads:[~2020-12-17 19:06 UTC|newest] Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top [not found] <20201012081106.10610-1-ndabilpuram@marvell.com> 2020-10-12 8:11 ` [dpdk-stable] [PATCH 2/2] vfio: fix partial DMA unmapping for VFIO type1 Nithin Dabilpuram 2020-10-14 15:07 ` Burakov, Anatoly 2020-10-15 6:09 ` [dpdk-stable] [EXT] " Nithin Dabilpuram 2020-10-15 10:00 ` Burakov, Anatoly 2020-10-15 11:38 ` Nithin Dabilpuram 2020-10-15 11:50 ` Nithin Dabilpuram 2020-10-15 11:57 ` [dpdk-stable] [dpdk-dev] " Nithin Dabilpuram 2020-10-15 15:10 ` Burakov, Anatoly 2020-10-16 7:10 ` Nithin Dabilpuram 2020-10-17 16:14 ` Burakov, Anatoly 2020-10-19 9:43 ` Nithin Dabilpuram 2020-10-22 12:13 ` Nithin Dabilpuram 2020-10-28 13:04 ` Burakov, Anatoly 2020-10-28 14:17 ` Nithin Dabilpuram 2020-10-28 16:07 ` Burakov, Anatoly 2020-10-28 16:31 ` Nithin Dabilpuram [not found] ` <20201105090423.11954-1-ndabilpuram@marvell.com> 2020-11-05 9:04 ` [dpdk-stable] [PATCH v2 1/3] vfio: revert changes for map contiguous areas in one go Nithin Dabilpuram 2020-11-05 9:04 ` [dpdk-stable] [PATCH v2 2/3] vfio: fix DMA mapping granularity for type1 iova as va Nithin Dabilpuram 2020-11-10 14:04 ` Burakov, Anatoly 2020-11-10 14:22 ` [dpdk-stable] [dpdk-dev] " Burakov, Anatoly 2020-11-10 14:17 ` [dpdk-stable] " Burakov, Anatoly 2020-11-11 5:08 ` [dpdk-stable] [dpdk-dev] " Nithin Dabilpuram 2020-11-11 10:00 ` Burakov, Anatoly [not found] ` <20201201193302.28131-1-ndabilpuram@marvell.com> 2020-12-01 19:32 ` [dpdk-stable] [PATCH v3 1/4] vfio: revert changes for map contiguous areas in one go Nithin Dabilpuram 2020-12-01 19:33 ` [dpdk-stable] [PATCH v3 2/4] vfio: fix DMA mapping granularity for type1 IOVA as VA Nithin Dabilpuram [not found] ` <20201202054647.3449-1-ndabilpuram@marvell.com> 2020-12-02 5:46 ` [dpdk-stable] [PATCH v4 1/4] vfio: revert changes for map contiguous areas in one go Nithin Dabilpuram 2020-12-02 18:36 ` David Christensen 2020-12-02 5:46 ` [dpdk-stable] [PATCH v4 2/4] vfio: fix DMA mapping granularity for type1 IOVA as VA Nithin Dabilpuram 2020-12-02 18:38 ` David Christensen [not found] ` <20201214081935.23577-1-ndabilpuram@marvell.com> 2020-12-14 8:19 ` [dpdk-stable] [PATCH v5 1/4] vfio: revert changes for map contiguous areas in one go Nithin Dabilpuram 2020-12-14 8:19 ` [dpdk-stable] [PATCH v5 2/4] vfio: fix DMA mapping granularity for type1 IOVA as VA Nithin Dabilpuram [not found] ` <20201217190604.29803-1-ndabilpuram@marvell.com> 2020-12-17 19:06 ` Nithin Dabilpuram [this message] 2020-12-17 19:06 ` [dpdk-stable] [PATCH v6 " Nithin Dabilpuram [not found] ` <20210112173923.30320-1-ndabilpuram@marvell.com> 2021-01-12 17:39 ` [dpdk-stable] [PATCH v7 1/3] vfio: revert changes for map contiguous areas in one go Nithin Dabilpuram 2021-01-12 17:39 ` [dpdk-stable] [PATCH v7 2/3] vfio: fix DMA mapping granularity for type1 IOVA as VA Nithin Dabilpuram [not found] ` <20210115073243.7025-1-ndabilpuram@marvell.com> 2021-01-15 7:32 ` [dpdk-stable] [PATCH v8 1/3] vfio: revert changes for map contiguous areas in one go Nithin Dabilpuram 2021-01-15 7:32 ` [dpdk-stable] [PATCH v8 2/3] vfio: fix DMA mapping granularity for type1 IOVA as VA Nithin Dabilpuram 2021-01-15 7:32 ` [dpdk-stable] [PATCH v8 3/3] test: change external memory test to use system page sz Nithin Dabilpuram 2021-02-11 11:21 ` Burakov, Anatoly
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20201217190604.29803-2-ndabilpuram@marvell.com \ --to=ndabilpuram@marvell.com \ --cc=anatoly.burakov@intel.com \ --cc=david.marchand@redhat.com \ --cc=dev@dpdk.org \ --cc=drc@linux.vnet.ibm.com \ --cc=jerinj@marvell.com \ --cc=stable@dpdk.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
patches for DPDK stable branches This inbox may be cloned and mirrored by anyone: git clone --mirror http://inbox.dpdk.org/stable/0 stable/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 stable stable/ http://inbox.dpdk.org/stable \ stable@dpdk.org public-inbox-index stable Example config snippet for mirrors. Newsgroup available over NNTP: nntp://inbox.dpdk.org/inbox.dpdk.stable AGPL code for this site: git clone https://public-inbox.org/public-inbox.git