From: "Gaëtan Rivet" <gaetan.rivet@6wind.com>
To: Shahaf Shuler <shahafs@mellanox.com>
Cc: anatoly.burakov@intel.com, yskoh@mellanox.com,
thomas@monjalon.net, ferruh.yigit@intel.com,
nhorman@tuxdriver.com, dev@dpdk.org
Subject: Re: [dpdk-dev] [PATCH v4 0/6] introduce DMA memory mapping for external memory
Date: Mon, 11 Mar 2019 10:27:38 +0100 [thread overview]
Message-ID: <20190311092738.qztd3j6sujbz64m4@bidouze.vm.6wind.com> (raw)
In-Reply-To: <cover.1552206210.git.shahafs@mellanox.com>
Hello Shahaf,
Thanks for taking my remarks into account. You can add my acked-by if
you need to the series (not really for mlx5 PMD but you get the idea).
BR,
On Sun, Mar 10, 2019 at 10:27:57AM +0200, Shahaf Shuler wrote:
> The DPDK APIs expose 3 different modes to work with memory used for DMA:
>
> 1. Use the DPDK owned memory (backed by the DPDK provided hugepages).
> This memory is allocated by the DPDK libraries, included in the DPDK
> memory system (memseg lists) and automatically DMA mapped by the DPDK
> layers.
>
> 2. Use memory allocated by the user and register to the DPDK memory
> systems. Upon registration of memory, the DPDK layers will DMA map it
> to all needed devices. After registration, allocation of this memory
> will be done with rte_*malloc APIs.
>
> 3. Use memory allocated by the user and not registered to the DPDK memory
> system. This is for users who wants to have tight control on this
> memory (e.g. avoid the rte_malloc header).
> The user should create a memory, register it through rte_extmem_register
> API, and call DMA map function in order to register such memory to
> the different devices.
>
> The scope of the patch focus on #3 above.
>
> Currently the only way to map external memory is through VFIO
> (rte_vfio_dma_map). While VFIO is common, there are other vendors
> which use different ways to map memory (e.g. Mellanox and NXP).
>
> The work in this patch moves the DMA mapping to vendor agnostic APIs.
> Device level DMA map and unmap APIs were added. Implementation of those
> APIs was done currently only for PCI devices.
>
> For PCI bus devices, the pci driver can expose its own map and unmap
> functions to be used for the mapping. In case the driver doesn't provide
> any, the memory will be mapped, if possible, to IOMMU through VFIO APIs.
>
> Application usage with those APIs is quite simple:
> * allocate memory
> * call rte_extmem_register on the memory chunk.
> * take a device, and query its rte_device.
> * call the device specific mapping function for this device.
>
> Future work will deprecate the rte_vfio_dma_map and rte_vfio_dma_unmap
> APIs, leaving the rte device APIs as the preferred option for the user.
>
> On v4:
> - Changed rte_dev_dma_map errno to ENOTSUP in case bus doesn't support
> DMA map API.
>
> On v3:
> - Fixed compilation issue on freebsd.
> - Fixed forgotten rte_bus_dma_map to rte_dev_dma_map.
> - Removed __rte_deprecated from vfio function till the time the rte_dev_dma_map
> will be non-experimental.
> - Changed error return value to always be -1, with proper errno.
> - Used rte_mem_virt2memseg_list instead of rte_mem_virt2memseg to verify
> memory is registered.
> - Added above check also on dma_unmap calls.
> - Added note in the API the memory must be registered in advance.
> - Added debug log to report the case memory mapping to vfio was skipped.
>
> On v2:
> - Added warn in release notes about the API change in vfio.
> - Moved function doc to prototype declaration.
> - Used dma_map and dma_unmap instead of map and unmap.
> - Used RTE_VFIO_DEFAULT_CONTAINER_FD instead of -1 fixed value.
> - Moved bus function to eal_common_dev.c and renamed them properly.
> - Changed eth device iterator to use RTE_DEV_FOREACH.
> - Enforced memory is registered with rte_extmem_* prior to mapping.
> - Used EEXIST as the only possible return value from type1 vfio IOMMU mapping.
>
> [1] https://patches.dpdk.org/patch/47796/
>
> Shahaf Shuler (6):
> vfio: allow DMA map of memory for the default vfio fd
> vfio: don't fail to DMA map if memory is already mapped
> bus: introduce device level DMA memory mapping
> net/mlx5: refactor external memory registration
> net/mlx5: support PCI device DMA map and unmap
> doc: deprecation notice for VFIO DMA map APIs
>
> doc/guides/prog_guide/env_abstraction_layer.rst | 2 +-
> doc/guides/rel_notes/deprecation.rst | 4 +
> doc/guides/rel_notes/release_19_05.rst | 3 +
> drivers/bus/pci/pci_common.c | 48 ++++
> drivers/bus/pci/rte_bus_pci.h | 40 ++++
> drivers/net/mlx5/mlx5.c | 2 +
> drivers/net/mlx5/mlx5_mr.c | 225 ++++++++++++++++---
> drivers/net/mlx5/mlx5_rxtx.h | 5 +
> lib/librte_eal/common/eal_common_dev.c | 34 +++
> lib/librte_eal/common/include/rte_bus.h | 44 ++++
> lib/librte_eal/common/include/rte_dev.h | 47 ++++
> lib/librte_eal/common/include/rte_vfio.h | 8 +-
> lib/librte_eal/linuxapp/eal/eal_vfio.c | 42 +++-
> lib/librte_eal/rte_eal_version.map | 2 +
> 14 files changed, 468 insertions(+), 38 deletions(-)
>
> --
> 2.12.0
>
--
Gaëtan Rivet
6WIND
next prev parent reply other threads:[~2019-03-11 9:27 UTC|newest]
Thread overview: 79+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-02-13 9:10 [dpdk-dev] [PATCH " Shahaf Shuler
2019-02-13 9:10 ` [dpdk-dev] [PATCH 1/6] vfio: allow DMA map of memory for the default vfio fd Shahaf Shuler
2019-02-13 9:45 ` Gaëtan Rivet
2019-02-13 11:38 ` Gaëtan Rivet
2019-02-13 15:23 ` Shahaf Shuler
2019-02-13 14:41 ` Burakov, Anatoly
2019-02-13 9:10 ` [dpdk-dev] [PATCH 2/6] vfio: don't fail to DMA map if memory is already mapped Shahaf Shuler
2019-02-13 9:58 ` Gaëtan Rivet
2019-02-13 19:52 ` Shahaf Shuler
2019-02-13 9:10 ` [dpdk-dev] [PATCH 3/6] bus: introduce DMA memory mapping for external memory Shahaf Shuler
2019-02-13 11:17 ` Gaëtan Rivet
2019-02-13 19:07 ` Shahaf Shuler
2019-02-14 14:00 ` Gaëtan Rivet
2019-02-17 6:23 ` Shahaf Shuler
2019-02-13 9:10 ` [dpdk-dev] [PATCH 4/6] net/mlx5: refactor external memory registration Shahaf Shuler
2019-02-13 9:10 ` [dpdk-dev] [PATCH 5/6] net/mlx5: support PCI device DMA map and unmap Shahaf Shuler
2019-02-13 11:35 ` Gaëtan Rivet
2019-02-13 11:44 ` Gaëtan Rivet
2019-02-13 19:11 ` Shahaf Shuler
2019-02-14 10:21 ` Gaëtan Rivet
2019-02-21 9:21 ` Shahaf Shuler
2019-02-13 9:10 ` [dpdk-dev] [PATCH 6/6] doc: deprecate VFIO DMA map APIs Shahaf Shuler
2019-02-13 11:43 ` [dpdk-dev] [PATCH 0/6] introduce DMA memory mapping for external memory Alejandro Lucero
2019-02-13 19:24 ` Shahaf Shuler
2019-02-14 10:19 ` Burakov, Anatoly
2019-02-14 13:28 ` Shahaf Shuler
2019-02-14 16:19 ` Burakov, Anatoly
2019-02-17 6:18 ` Shahaf Shuler
2019-02-18 12:21 ` Burakov, Anatoly
2019-02-14 12:22 ` Alejandro Lucero
2019-02-14 12:27 ` Alejandro Lucero
2019-02-14 13:41 ` Shahaf Shuler
2019-02-14 16:43 ` Burakov, Anatoly
2019-02-21 14:50 ` [dpdk-dev] [PATCH v2 " Shahaf Shuler
2019-03-05 13:59 ` [dpdk-dev] [PATCH v3 " Shahaf Shuler
2019-03-10 8:27 ` [dpdk-dev] [PATCH v4 " Shahaf Shuler
2019-03-10 8:27 ` [dpdk-dev] [PATCH v4 1/6] vfio: allow DMA map of memory for the default vfio fd Shahaf Shuler
2019-03-30 0:23 ` Thomas Monjalon
2019-03-30 0:23 ` Thomas Monjalon
2019-03-30 14:29 ` Thomas Monjalon
2019-03-30 14:29 ` Thomas Monjalon
2019-03-10 8:27 ` [dpdk-dev] [PATCH v4 2/6] vfio: don't fail to DMA map if memory is already mapped Shahaf Shuler
2019-03-10 8:28 ` [dpdk-dev] [PATCH v4 3/6] bus: introduce device level DMA memory mapping Shahaf Shuler
2019-03-11 10:19 ` Burakov, Anatoly
2019-03-13 9:56 ` Thomas Monjalon
2019-03-13 11:12 ` Shahaf Shuler
2019-03-13 11:19 ` Thomas Monjalon
2019-03-13 11:47 ` Burakov, Anatoly
2019-03-30 14:36 ` Thomas Monjalon
2019-03-30 14:36 ` Thomas Monjalon
2019-03-10 8:28 ` [dpdk-dev] [PATCH v4 4/6] net/mlx5: refactor external memory registration Shahaf Shuler
2019-03-10 8:28 ` [dpdk-dev] [PATCH v4 5/6] net/mlx5: support PCI device DMA map and unmap Shahaf Shuler
2019-03-10 8:28 ` [dpdk-dev] [PATCH v4 6/6] doc: deprecation notice for VFIO DMA map APIs Shahaf Shuler
2019-03-11 10:20 ` Burakov, Anatoly
2019-03-11 17:35 ` Rami Rosen
2019-10-01 15:20 ` David Marchand
2019-10-02 4:53 ` Shahaf Shuler
2019-10-02 7:51 ` David Marchand
2019-03-11 9:27 ` Gaëtan Rivet [this message]
2019-03-30 14:40 ` [dpdk-dev] [PATCH v4 0/6] introduce DMA memory mapping for external memory Thomas Monjalon
2019-03-30 14:40 ` Thomas Monjalon
2019-03-05 13:59 ` [dpdk-dev] [PATCH v3 1/6] vfio: allow DMA map of memory for the default vfio fd Shahaf Shuler
2019-03-05 13:59 ` [dpdk-dev] [PATCH v3 2/6] vfio: don't fail to DMA map if memory is already mapped Shahaf Shuler
2019-03-05 13:59 ` [dpdk-dev] [PATCH v3 3/6] bus: introduce device level DMA memory mapping Shahaf Shuler
2019-03-05 16:35 ` Burakov, Anatoly
2019-03-05 13:59 ` [dpdk-dev] [PATCH v3 4/6] net/mlx5: refactor external memory registration Shahaf Shuler
2019-03-05 13:59 ` [dpdk-dev] [PATCH v3 5/6] net/mlx5: support PCI device DMA map and unmap Shahaf Shuler
2019-03-05 13:59 ` [dpdk-dev] [PATCH v3 6/6] doc: deprecation notice for VFIO DMA map APIs Shahaf Shuler
2019-02-21 14:50 ` [dpdk-dev] [PATCH v2 1/6] vfio: allow DMA map of memory for the default vfio fd Shahaf Shuler
2019-02-28 11:56 ` Burakov, Anatoly
2019-02-21 14:50 ` [dpdk-dev] [PATCH v2 2/6] vfio: don't fail to DMA map if memory is already mapped Shahaf Shuler
2019-02-28 11:58 ` Burakov, Anatoly
2019-02-21 14:50 ` [dpdk-dev] [PATCH v2 3/6] bus: introduce device level DMA memory mapping Shahaf Shuler
2019-02-28 12:14 ` Burakov, Anatoly
2019-02-28 14:41 ` Burakov, Anatoly
2019-02-21 14:50 ` [dpdk-dev] [PATCH v2 4/6] net/mlx5: refactor external memory registration Shahaf Shuler
2019-02-21 14:50 ` [dpdk-dev] [PATCH v2 5/6] net/mlx5: support PCI device DMA map and unmap Shahaf Shuler
2019-02-21 14:50 ` [dpdk-dev] [PATCH v2 6/6] doc: deprecate VFIO DMA map APIs Shahaf Shuler
2019-02-21 15:50 ` David Marchand
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190311092738.qztd3j6sujbz64m4@bidouze.vm.6wind.com \
--to=gaetan.rivet@6wind.com \
--cc=anatoly.burakov@intel.com \
--cc=dev@dpdk.org \
--cc=ferruh.yigit@intel.com \
--cc=nhorman@tuxdriver.com \
--cc=shahafs@mellanox.com \
--cc=thomas@monjalon.net \
--cc=yskoh@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).