From: David Christensen <drc@linux.vnet.ibm.com>
To: Nithin Dabilpuram <ndabilpuram@marvell.com>,
anatoly.burakov@intel.com, david.marchand@redhat.com
Cc: jerinj@marvell.com, dev@dpdk.org, stable@dpdk.org
Subject: Re: [dpdk-stable] [PATCH v4 1/4] vfio: revert changes for map contiguous areas in one go
Date: Wed, 2 Dec 2020 10:36:34 -0800 [thread overview]
Message-ID: <7a79bbe4-2402-9f6e-4101-360f14b4e599@linux.vnet.ibm.com> (raw)
In-Reply-To: <20201202054647.3449-2-ndabilpuram@marvell.com>
On 12/1/20 9:46 PM, Nithin Dabilpuram wrote:
> In order to save DMA entries limited by kernel both for externel
> memory and hugepage memory, an attempt was made to map physically
> contiguous memory in one go. This cannot be done as VFIO IOMMU type1
> does not support partially unmapping a previously mapped memory
> region while Heap can request for multi page mapping and
> partial unmapping.
> Hence for going back to old method of mapping/unmapping at
> memseg granularity, this commit reverts
> commit d1c7c0cdf7ba ("vfio: map contiguous areas in one go")
>
> Also add documentation on what module parameter needs to be used
> to increase the per-container dma map limit for VFIO.
>
> Fixes: d1c7c0cdf7ba ("vfio: map contiguous areas in one go")
> Cc: anatoly.burakov@intel.com
> Cc: stable@dpdk.org
>
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> ---
> doc/guides/linux_gsg/linux_drivers.rst | 10 ++++++
> lib/librte_eal/linux/eal_vfio.c | 59 +++++-----------------------------
> 2 files changed, 18 insertions(+), 51 deletions(-)
>
> diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
> index 90635a4..9a662a7 100644
> --- a/doc/guides/linux_gsg/linux_drivers.rst
> +++ b/doc/guides/linux_gsg/linux_drivers.rst
> @@ -25,6 +25,16 @@ To make use of VFIO, the ``vfio-pci`` module must be loaded:
> VFIO kernel is usually present by default in all distributions,
> however please consult your distributions documentation to make sure that is the case.
>
> +For DMA mapping of either external memory or hugepages, VFIO interface is used.
> +VFIO does not support partial unmap of once mapped memory. Hence DPDK's memory is
> +mapped in hugepage granularity or system page granularity. Number of DMA
> +mappings is limited by kernel with user locked memory limit of a process(rlimit)
> +for system/hugepage memory. Another per-container overall limit applicable both
> +for external memory and system memory was added in kernel 5.1 defined by
> +VFIO module parameter ``dma_entry_limit`` with a default value of 64K.
> +When application is out of DMA entries, these limits need to be adjusted to
> +increase the allowed limit.
> +
> Since Linux version 5.7,
> the ``vfio-pci`` module supports the creation of virtual functions.
> After the PF is bound to ``vfio-pci`` module,
> diff --git a/lib/librte_eal/linux/eal_vfio.c b/lib/librte_eal/linux/eal_vfio.c
> index 0500824..64b134d 100644
> --- a/lib/librte_eal/linux/eal_vfio.c
> +++ b/lib/librte_eal/linux/eal_vfio.c
> @@ -517,11 +517,9 @@ static void
> vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
> void *arg __rte_unused)
> {
> - rte_iova_t iova_start, iova_expected;
> struct rte_memseg_list *msl;
> struct rte_memseg *ms;
> size_t cur_len = 0;
> - uint64_t va_start;
>
> msl = rte_mem_virt2memseg_list(addr);
>
> @@ -539,63 +537,22 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
>
> /* memsegs are contiguous in memory */
> ms = rte_mem_virt2memseg(addr, msl);
> -
> - /*
> - * This memory is not guaranteed to be contiguous, but it still could
> - * be, or it could have some small contiguous chunks. Since the number
> - * of VFIO mappings is limited, and VFIO appears to not concatenate
> - * adjacent mappings, we have to do this ourselves.
> - *
> - * So, find contiguous chunks, then map them.
> - */
> - va_start = ms->addr_64;
> - iova_start = iova_expected = ms->iova;
> while (cur_len < len) {
> - bool new_contig_area = ms->iova != iova_expected;
> - bool last_seg = (len - cur_len) == ms->len;
> - bool skip_last = false;
> -
> - /* only do mappings when current contiguous area ends */
> - if (new_contig_area) {
> - if (type == RTE_MEM_EVENT_ALLOC)
> - vfio_dma_mem_map(default_vfio_cfg, va_start,
> - iova_start,
> - iova_expected - iova_start, 1);
> - else
> - vfio_dma_mem_map(default_vfio_cfg, va_start,
> - iova_start,
> - iova_expected - iova_start, 0);
> - va_start = ms->addr_64;
> - iova_start = ms->iova;
> - }
> /* some memory segments may have invalid IOVA */
> if (ms->iova == RTE_BAD_IOVA) {
> RTE_LOG(DEBUG, EAL, "Memory segment at %p has bad IOVA, skipping\n",
> ms->addr);
> - skip_last = true;
> + goto next;
> }
> - iova_expected = ms->iova + ms->len;
> + if (type == RTE_MEM_EVENT_ALLOC)
> + vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
> + ms->iova, ms->len, 1);
> + else
> + vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
> + ms->iova, ms->len, 0);
> +next:
> cur_len += ms->len;
> ++ms;
> -
> - /*
> - * don't count previous segment, and don't attempt to
> - * dereference a potentially invalid pointer.
> - */
> - if (skip_last && !last_seg) {
> - iova_expected = iova_start = ms->iova;
> - va_start = ms->addr_64;
> - } else if (!skip_last && last_seg) {
> - /* this is the last segment and we're not skipping */
> - if (type == RTE_MEM_EVENT_ALLOC)
> - vfio_dma_mem_map(default_vfio_cfg, va_start,
> - iova_start,
> - iova_expected - iova_start, 1);
> - else
> - vfio_dma_mem_map(default_vfio_cfg, va_start,
> - iova_start,
> - iova_expected - iova_start, 0);
> - }
> }
> }
>
Acked-by: David Christensen <drc@linux.vnet.ibm.com>
next prev parent reply other threads:[~2020-12-02 18:36 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20201012081106.10610-1-ndabilpuram@marvell.com>
2020-10-12 8:11 ` [dpdk-stable] [PATCH 2/2] vfio: fix partial DMA unmapping for VFIO type1 Nithin Dabilpuram
2020-10-14 15:07 ` Burakov, Anatoly
2020-10-15 6:09 ` [dpdk-stable] [EXT] " Nithin Dabilpuram
2020-10-15 10:00 ` Burakov, Anatoly
2020-10-15 11:38 ` Nithin Dabilpuram
2020-10-15 11:50 ` Nithin Dabilpuram
2020-10-15 11:57 ` [dpdk-stable] [dpdk-dev] " Nithin Dabilpuram
2020-10-15 15:10 ` Burakov, Anatoly
2020-10-16 7:10 ` Nithin Dabilpuram
2020-10-17 16:14 ` Burakov, Anatoly
2020-10-19 9:43 ` Nithin Dabilpuram
2020-10-22 12:13 ` Nithin Dabilpuram
2020-10-28 13:04 ` Burakov, Anatoly
2020-10-28 14:17 ` Nithin Dabilpuram
2020-10-28 16:07 ` Burakov, Anatoly
2020-10-28 16:31 ` Nithin Dabilpuram
[not found] ` <20201105090423.11954-1-ndabilpuram@marvell.com>
2020-11-05 9:04 ` [dpdk-stable] [PATCH v2 1/3] vfio: revert changes for map contiguous areas in one go Nithin Dabilpuram
2020-11-05 9:04 ` [dpdk-stable] [PATCH v2 2/3] vfio: fix DMA mapping granularity for type1 iova as va Nithin Dabilpuram
2020-11-10 14:04 ` Burakov, Anatoly
2020-11-10 14:22 ` [dpdk-stable] [dpdk-dev] " Burakov, Anatoly
2020-11-10 14:17 ` [dpdk-stable] " Burakov, Anatoly
2020-11-11 5:08 ` [dpdk-stable] [dpdk-dev] " Nithin Dabilpuram
2020-11-11 10:00 ` Burakov, Anatoly
[not found] ` <20201201193302.28131-1-ndabilpuram@marvell.com>
2020-12-01 19:32 ` [dpdk-stable] [PATCH v3 1/4] vfio: revert changes for map contiguous areas in one go Nithin Dabilpuram
2020-12-01 19:33 ` [dpdk-stable] [PATCH v3 2/4] vfio: fix DMA mapping granularity for type1 IOVA as VA Nithin Dabilpuram
[not found] ` <20201202054647.3449-1-ndabilpuram@marvell.com>
2020-12-02 5:46 ` [dpdk-stable] [PATCH v4 1/4] vfio: revert changes for map contiguous areas in one go Nithin Dabilpuram
2020-12-02 18:36 ` David Christensen [this message]
2020-12-02 5:46 ` [dpdk-stable] [PATCH v4 2/4] vfio: fix DMA mapping granularity for type1 IOVA as VA Nithin Dabilpuram
2020-12-02 18:38 ` David Christensen
[not found] ` <20201214081935.23577-1-ndabilpuram@marvell.com>
2020-12-14 8:19 ` [dpdk-stable] [PATCH v5 1/4] vfio: revert changes for map contiguous areas in one go Nithin Dabilpuram
2020-12-14 8:19 ` [dpdk-stable] [PATCH v5 2/4] vfio: fix DMA mapping granularity for type1 IOVA as VA Nithin Dabilpuram
[not found] ` <20201217190604.29803-1-ndabilpuram@marvell.com>
2020-12-17 19:06 ` [dpdk-stable] [PATCH v6 1/4] vfio: revert changes for map contiguous areas in one go Nithin Dabilpuram
2020-12-17 19:06 ` [dpdk-stable] [PATCH v6 2/4] vfio: fix DMA mapping granularity for type1 IOVA as VA Nithin Dabilpuram
[not found] ` <20210112173923.30320-1-ndabilpuram@marvell.com>
2021-01-12 17:39 ` [dpdk-stable] [PATCH v7 1/3] vfio: revert changes for map contiguous areas in one go Nithin Dabilpuram
2021-01-12 17:39 ` [dpdk-stable] [PATCH v7 2/3] vfio: fix DMA mapping granularity for type1 IOVA as VA Nithin Dabilpuram
[not found] ` <20210115073243.7025-1-ndabilpuram@marvell.com>
2021-01-15 7:32 ` [dpdk-stable] [PATCH v8 1/3] vfio: revert changes for map contiguous areas in one go Nithin Dabilpuram
2021-03-05 7:50 ` David Marchand
2021-03-05 13:54 ` Burakov, Anatoly
2021-03-05 15:50 ` [dpdk-stable] [dpdk-dev] " Nithin Dabilpuram
2021-04-01 11:27 ` Burakov, Anatoly
2021-01-15 7:32 ` [dpdk-stable] [PATCH v8 2/3] vfio: fix DMA mapping granularity for type1 IOVA as VA Nithin Dabilpuram
2021-01-15 7:32 ` [dpdk-stable] [PATCH v8 3/3] test: change external memory test to use system page sz Nithin Dabilpuram
2021-02-11 11:21 ` Burakov, Anatoly
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=7a79bbe4-2402-9f6e-4101-360f14b4e599@linux.vnet.ibm.com \
--to=drc@linux.vnet.ibm.com \
--cc=anatoly.burakov@intel.com \
--cc=david.marchand@redhat.com \
--cc=dev@dpdk.org \
--cc=jerinj@marvell.com \
--cc=ndabilpuram@marvell.com \
--cc=stable@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).