DPDK patches and discussions
 help / color / mirror / Atom feed
From: David Christensen <drc@linux.vnet.ibm.com>
To: Nithin Dabilpuram <ndabilpuram@marvell.com>,
	anatoly.burakov@intel.com, david.marchand@redhat.com
Cc: jerinj@marvell.com, dev@dpdk.org, stable@dpdk.org
Subject: Re: [dpdk-dev] [PATCH v4 1/4] vfio: revert changes for map contiguous areas in one go
Date: Wed, 2 Dec 2020 10:36:34 -0800	[thread overview]
Message-ID: <7a79bbe4-2402-9f6e-4101-360f14b4e599@linux.vnet.ibm.com> (raw)
In-Reply-To: <20201202054647.3449-2-ndabilpuram@marvell.com>



On 12/1/20 9:46 PM, Nithin Dabilpuram wrote:
> In order to save DMA entries limited by kernel both for externel
> memory and hugepage memory, an attempt was made to map physically
> contiguous memory in one go. This cannot be done as VFIO IOMMU type1
> does not support partially unmapping a previously mapped memory
> region while Heap can request for multi page mapping and
> partial unmapping.
> Hence for going back to old method of mapping/unmapping at
> memseg granularity, this commit reverts
> commit d1c7c0cdf7ba ("vfio: map contiguous areas in one go")
> 
> Also add documentation on what module parameter needs to be used
> to increase the per-container dma map limit for VFIO.
> 
> Fixes: d1c7c0cdf7ba ("vfio: map contiguous areas in one go")
> Cc: anatoly.burakov@intel.com
> Cc: stable@dpdk.org
> 
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> ---
>   doc/guides/linux_gsg/linux_drivers.rst | 10 ++++++
>   lib/librte_eal/linux/eal_vfio.c        | 59 +++++-----------------------------
>   2 files changed, 18 insertions(+), 51 deletions(-)
> 
> diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
> index 90635a4..9a662a7 100644
> --- a/doc/guides/linux_gsg/linux_drivers.rst
> +++ b/doc/guides/linux_gsg/linux_drivers.rst
> @@ -25,6 +25,16 @@ To make use of VFIO, the ``vfio-pci`` module must be loaded:
>   VFIO kernel is usually present by default in all distributions,
>   however please consult your distributions documentation to make sure that is the case.
> 
> +For DMA mapping of either external memory or hugepages, VFIO interface is used.
> +VFIO does not support partial unmap of once mapped memory. Hence DPDK's memory is
> +mapped in hugepage granularity or system page granularity. Number of DMA
> +mappings is limited by kernel with user locked memory limit of a process(rlimit)
> +for system/hugepage memory. Another per-container overall limit applicable both
> +for external memory and system memory was added in kernel 5.1 defined by
> +VFIO module parameter ``dma_entry_limit`` with a default value of 64K.
> +When application is out of DMA entries, these limits need to be adjusted to
> +increase the allowed limit.
> +
>   Since Linux version 5.7,
>   the ``vfio-pci`` module supports the creation of virtual functions.
>   After the PF is bound to ``vfio-pci`` module,
> diff --git a/lib/librte_eal/linux/eal_vfio.c b/lib/librte_eal/linux/eal_vfio.c
> index 0500824..64b134d 100644
> --- a/lib/librte_eal/linux/eal_vfio.c
> +++ b/lib/librte_eal/linux/eal_vfio.c
> @@ -517,11 +517,9 @@ static void
>   vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
>   		void *arg __rte_unused)
>   {
> -	rte_iova_t iova_start, iova_expected;
>   	struct rte_memseg_list *msl;
>   	struct rte_memseg *ms;
>   	size_t cur_len = 0;
> -	uint64_t va_start;
> 
>   	msl = rte_mem_virt2memseg_list(addr);
> 
> @@ -539,63 +537,22 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
> 
>   	/* memsegs are contiguous in memory */
>   	ms = rte_mem_virt2memseg(addr, msl);
> -
> -	/*
> -	 * This memory is not guaranteed to be contiguous, but it still could
> -	 * be, or it could have some small contiguous chunks. Since the number
> -	 * of VFIO mappings is limited, and VFIO appears to not concatenate
> -	 * adjacent mappings, we have to do this ourselves.
> -	 *
> -	 * So, find contiguous chunks, then map them.
> -	 */
> -	va_start = ms->addr_64;
> -	iova_start = iova_expected = ms->iova;
>   	while (cur_len < len) {
> -		bool new_contig_area = ms->iova != iova_expected;
> -		bool last_seg = (len - cur_len) == ms->len;
> -		bool skip_last = false;
> -
> -		/* only do mappings when current contiguous area ends */
> -		if (new_contig_area) {
> -			if (type == RTE_MEM_EVENT_ALLOC)
> -				vfio_dma_mem_map(default_vfio_cfg, va_start,
> -						iova_start,
> -						iova_expected - iova_start, 1);
> -			else
> -				vfio_dma_mem_map(default_vfio_cfg, va_start,
> -						iova_start,
> -						iova_expected - iova_start, 0);
> -			va_start = ms->addr_64;
> -			iova_start = ms->iova;
> -		}
>   		/* some memory segments may have invalid IOVA */
>   		if (ms->iova == RTE_BAD_IOVA) {
>   			RTE_LOG(DEBUG, EAL, "Memory segment at %p has bad IOVA, skipping\n",
>   					ms->addr);
> -			skip_last = true;
> +			goto next;
>   		}
> -		iova_expected = ms->iova + ms->len;
> +		if (type == RTE_MEM_EVENT_ALLOC)
> +			vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
> +					ms->iova, ms->len, 1);
> +		else
> +			vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
> +					ms->iova, ms->len, 0);
> +next:
>   		cur_len += ms->len;
>   		++ms;
> -
> -		/*
> -		 * don't count previous segment, and don't attempt to
> -		 * dereference a potentially invalid pointer.
> -		 */
> -		if (skip_last && !last_seg) {
> -			iova_expected = iova_start = ms->iova;
> -			va_start = ms->addr_64;
> -		} else if (!skip_last && last_seg) {
> -			/* this is the last segment and we're not skipping */
> -			if (type == RTE_MEM_EVENT_ALLOC)
> -				vfio_dma_mem_map(default_vfio_cfg, va_start,
> -						iova_start,
> -						iova_expected - iova_start, 1);
> -			else
> -				vfio_dma_mem_map(default_vfio_cfg, va_start,
> -						iova_start,
> -						iova_expected - iova_start, 0);
> -		}
>   	}
>   }
> 

Acked-by: David Christensen <drc@linux.vnet.ibm.com>

  reply	other threads:[~2020-12-02 18:36 UTC|newest]

Thread overview: 81+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-10-12  8:11 [dpdk-dev] [PATCH 0/2] fix issue with partial DMA unmap Nithin Dabilpuram
2020-10-12  8:11 ` [dpdk-dev] [PATCH 1/2] test: add test case to validate VFIO DMA map/unmap Nithin Dabilpuram
2020-10-14 14:39   ` Burakov, Anatoly
2020-10-15  9:54     ` [dpdk-dev] [EXT] " Nithin Dabilpuram
2020-10-12  8:11 ` [dpdk-dev] [PATCH 2/2] vfio: fix partial DMA unmapping for VFIO type1 Nithin Dabilpuram
2020-10-14 15:07   ` Burakov, Anatoly
2020-10-15  6:09     ` [dpdk-dev] [EXT] " Nithin Dabilpuram
2020-10-15 10:00       ` Burakov, Anatoly
2020-10-15 11:38         ` Nithin Dabilpuram
2020-10-15 11:50         ` Nithin Dabilpuram
2020-10-15 11:57         ` Nithin Dabilpuram
2020-10-15 15:10           ` Burakov, Anatoly
2020-10-16  7:10             ` Nithin Dabilpuram
2020-10-17 16:14               ` Burakov, Anatoly
2020-10-19  9:43                 ` Nithin Dabilpuram
2020-10-22 12:13                   ` Nithin Dabilpuram
2020-10-28 13:04                     ` Burakov, Anatoly
2020-10-28 14:17                       ` Nithin Dabilpuram
2020-10-28 16:07                         ` Burakov, Anatoly
2020-10-28 16:31                           ` Nithin Dabilpuram
2020-11-05  9:04 ` [dpdk-dev] [PATCH v2 0/3] fix issue with partial DMA unmap Nithin Dabilpuram
2020-11-05  9:04   ` [dpdk-dev] [PATCH v2 1/3] vfio: revert changes for map contiguous areas in one go Nithin Dabilpuram
2020-11-05  9:04   ` [dpdk-dev] [PATCH v2 2/3] vfio: fix DMA mapping granularity for type1 iova as va Nithin Dabilpuram
2020-11-10 14:04     ` Burakov, Anatoly
2020-11-10 14:22       ` Burakov, Anatoly
2020-11-10 14:17     ` Burakov, Anatoly
2020-11-11  5:08       ` Nithin Dabilpuram
2020-11-11 10:00         ` Burakov, Anatoly
2020-11-05  9:04   ` [dpdk-dev] [PATCH v2 3/3] test: add test case to validate VFIO DMA map/unmap Nithin Dabilpuram
2020-12-01 19:32 ` [dpdk-dev] [PATCH v3 0/4] fix issue with partial DMA unmap Nithin Dabilpuram
2020-12-01 19:32   ` [dpdk-dev] [PATCH v3 1/4] vfio: revert changes for map contiguous areas in one go Nithin Dabilpuram
2020-12-01 19:33   ` [dpdk-dev] [PATCH v3 2/4] vfio: fix DMA mapping granularity for type1 IOVA as VA Nithin Dabilpuram
2020-12-01 19:33   ` [dpdk-dev] [PATCH v3 3/4] test: add test case to validate VFIO DMA map/unmap Nithin Dabilpuram
2020-12-01 19:33   ` [dpdk-dev] [PATCH v3 4/4] test: change external memory test to use system page sz Nithin Dabilpuram
2020-12-01 23:23     ` David Christensen
2020-12-02  5:40       ` Nithin Dabilpuram
2020-12-02  5:46 ` [dpdk-dev] [PATCH v4 0/4] fix issue with partial DMA unmap Nithin Dabilpuram
2020-12-02  5:46   ` [dpdk-dev] [PATCH v4 1/4] vfio: revert changes for map contiguous areas in one go Nithin Dabilpuram
2020-12-02 18:36     ` David Christensen [this message]
2020-12-02  5:46   ` [dpdk-dev] [PATCH v4 2/4] vfio: fix DMA mapping granularity for type1 IOVA as VA Nithin Dabilpuram
2020-12-02 18:38     ` David Christensen
2020-12-02  5:46   ` [dpdk-dev] [PATCH v4 3/4] test: add test case to validate VFIO DMA map/unmap Nithin Dabilpuram
2020-12-02 19:23     ` David Christensen
2020-12-03  7:14       ` Nithin Dabilpuram
2020-12-14  8:24         ` Nithin Dabilpuram
2020-12-02  5:46   ` [dpdk-dev] [PATCH v4 4/4] test: change external memory test to use system page sz Nithin Dabilpuram
2020-12-14  8:19 ` [dpdk-dev] [PATCH v5 0/4] fix issue with partial DMA unmap Nithin Dabilpuram
2020-12-14  8:19   ` [dpdk-dev] [PATCH v5 1/4] vfio: revert changes for map contiguous areas in one go Nithin Dabilpuram
2020-12-14  8:19   ` [dpdk-dev] [PATCH v5 2/4] vfio: fix DMA mapping granularity for type1 IOVA as VA Nithin Dabilpuram
2020-12-14  8:19   ` [dpdk-dev] [PATCH v5 3/4] test: add test case to validate VFIO DMA map/unmap Nithin Dabilpuram
2020-12-14  8:19   ` [dpdk-dev] [PATCH v5 4/4] test: change external memory test to use system page sz Nithin Dabilpuram
2020-12-17 19:06 ` [dpdk-dev] [PATCH v6 0/4] fix issue with partial DMA unmap Nithin Dabilpuram
2020-12-17 19:06   ` [dpdk-dev] [PATCH v6 1/4] vfio: revert changes for map contiguous areas in one go Nithin Dabilpuram
2020-12-17 19:06   ` [dpdk-dev] [PATCH v6 2/4] vfio: fix DMA mapping granularity for type1 IOVA as VA Nithin Dabilpuram
2020-12-17 19:06   ` [dpdk-dev] [PATCH v6 3/4] test: add test case to validate VFIO DMA map/unmap Nithin Dabilpuram
2020-12-17 19:10     ` Nithin Dabilpuram
2021-01-05 19:33       ` David Christensen
2021-01-06  8:40         ` Nithin Dabilpuram
2021-01-06 21:20           ` David Christensen
2020-12-17 19:06   ` [dpdk-dev] [PATCH v6 4/4] test: change external memory test to use system page sz Nithin Dabilpuram
2020-12-23  5:13   ` [dpdk-dev] [PATCH v6 0/4] fix issue with partial DMA unmap Nithin Dabilpuram
2021-01-04 22:29     ` David Christensen
2021-01-12 17:39 ` [dpdk-dev] [PATCH v7 0/3] " Nithin Dabilpuram
2021-01-12 17:39   ` [dpdk-dev] [PATCH v7 1/3] vfio: revert changes for map contiguous areas in one go Nithin Dabilpuram
2021-01-12 17:39   ` [dpdk-dev] [PATCH v7 2/3] vfio: fix DMA mapping granularity for type1 IOVA as VA Nithin Dabilpuram
2021-01-12 17:39   ` [dpdk-dev] [PATCH v7 3/3] test: change external memory test to use system page sz Nithin Dabilpuram
2021-01-14 16:30     ` David Marchand
2021-01-15  6:57       ` Nithin Dabilpuram
2021-01-15  7:32 ` [dpdk-dev] [PATCH v8 0/3] fix issue with partial DMA unmap Nithin Dabilpuram
2021-01-15  7:32   ` [dpdk-dev] [PATCH v8 1/3] vfio: revert changes for map contiguous areas in one go Nithin Dabilpuram
2021-03-05  7:50     ` David Marchand
2021-03-05 13:54       ` Burakov, Anatoly
2021-03-05 15:50         ` Nithin Dabilpuram
2021-04-01 11:27           ` Burakov, Anatoly
2021-01-15  7:32   ` [dpdk-dev] [PATCH v8 2/3] vfio: fix DMA mapping granularity for type1 IOVA as VA Nithin Dabilpuram
2021-01-15  7:32   ` [dpdk-dev] [PATCH v8 3/3] test: change external memory test to use system page sz Nithin Dabilpuram
2021-02-11 11:21     ` Burakov, Anatoly
2021-02-16 13:14   ` [dpdk-dev] [PATCH v8 0/3] fix issue with partial DMA unmap Burakov, Anatoly
2021-02-22  9:41     ` Nithin Dabilpuram
2021-02-22 10:06       ` David Marchand
2021-03-01 12:13   ` David Marchand

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=7a79bbe4-2402-9f6e-4101-360f14b4e599@linux.vnet.ibm.com \
    --to=drc@linux.vnet.ibm.com \
    --cc=anatoly.burakov@intel.com \
    --cc=david.marchand@redhat.com \
    --cc=dev@dpdk.org \
    --cc=jerinj@marvell.com \
    --cc=ndabilpuram@marvell.com \
    --cc=stable@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).