[dpdk-dev] [PATCH 0/4] Add external contiguous memory support to Virtio-user

[dpdk-dev] [PATCH 0/4] Add external contiguous memory support to Virtio-user

[Maxime Coquelin]

Somme applications/frameworks using DPDK allocate their own memory
for its buffers. This is the case of Seastar, and also VPP.

It is currently not possible to use Virtio-user PMD with these
applications because Virtio-user requires file descriptor for this
memory to be shared with the vhost-user backend (vDPA in our case [0]).

This series introduces a new API in EAL to register external memory
that is contiguous in both the VA and the IOVA space.

It also enables VDEVs to implement the DMA map/unmap callback, and
implement it in Virtio-user PMD.

Please note that extmem unit tests are missing. I'm working on it
and it will be in v2. Just share this v1 for early reviews.

Maxime Coquelin (4):
  eal: add new API to register contiguous external memory
  eal: allow getting memory segment FD with external memory
  bus/vdev: add DMA mapping supprt
  net/virtio: add DMA mapping callback to virtio-user

 drivers/bus/vdev/rte_bus_vdev.h               | 24 +++++
 drivers/bus/vdev/vdev.c                       | 50 +++++++++++
 .../net/virtio/virtio_user/virtio_user_dev.c  | 10 +--
 .../net/virtio/virtio_user/virtio_user_dev.h  |  3 +
 drivers/net/virtio/virtio_user_ethdev.c       | 58 +++++++++++++
 lib/librte_eal/common/eal_common_memory.c     | 87 ++++++++++++++++---
 lib/librte_eal/common/include/rte_memory.h    | 46 ++++++++++
 lib/librte_eal/common/malloc_heap.c           | 17 +++-
 lib/librte_eal/common/malloc_heap.h           |  2 +-
 lib/librte_eal/common/rte_malloc.c            |  2 +-
 lib/librte_eal/rte_eal_version.map            |  3 +
 11 files changed, 278 insertions(+), 24 deletions(-)

-- 
2.21.0

[dpdk-dev] [PATCH 1/4] eal: add new API to register contiguous external memory

[Maxime Coquelin]

This new API allows to pass a file descriptor while
registering external and contiguous memory in the IOVA space.

This is required for using Virtio-user PMD with application
using external memory for the mbuf's buffers, like Seastar
or VPP.

FD is only attached to the segments if single_file_segment
option is enabled.

Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 lib/librte_eal/common/eal_common_memory.c  | 75 +++++++++++++++++++++-
 lib/librte_eal/common/include/rte_memory.h | 46 +++++++++++++
 lib/librte_eal/common/malloc_heap.c        | 17 ++++-
 lib/librte_eal/common/malloc_heap.h        |  2 +-
 lib/librte_eal/common/rte_malloc.c         |  2 +-
 lib/librte_eal/rte_eal_version.map         |  3 +
 6 files changed, 141 insertions(+), 4 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_memory.c b/lib/librte_eal/common/eal_common_memory.c
index 4a9cc1f19a..7a4b371828 100644
--- a/lib/librte_eal/common/eal_common_memory.c
+++ b/lib/librte_eal/common/eal_common_memory.c
@@ -772,6 +772,79 @@ rte_memseg_get_fd_offset(const struct rte_memseg *ms, size_t *offset)
 	return ret;
 }
 
+int
+rte_extmem_register_contig(void *va_addr, size_t len, rte_iova_t iova_addr,
+		size_t page_sz, int fd)
+{
+	struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;
+	rte_iova_t *iova_addrs = NULL;
+	unsigned int socket_id, n, i;
+	int ret = 0;
+
+	if (va_addr == NULL || page_sz == 0 || len == 0 ||
+			!rte_is_power_of_2(page_sz) ||
+			RTE_ALIGN(len, page_sz) != len ||
+			((len % page_sz) != 0  ||
+			!rte_is_aligned(va_addr, page_sz))) {
+		rte_errno = EINVAL;
+		return -1;
+	}
+
+	n = len / page_sz;
+	if (iova_addr != 0) {
+		iova_addrs = malloc(n * sizeof(*iova_addrs));
+		if (iova_addrs == NULL) {
+			rte_errno = -ENOMEM;
+			return -1;
+		}
+
+		for (i = 0; i < n; i++)
+			iova_addrs[i] = iova_addr + n * page_sz;
+
+	}
+
+
+	if (fd >= 0 && !internal_config.single_file_segments) {
+		RTE_LOG(INFO, EAL, "FD won't be attached to the external memory," \
+				" requires single file segments\n");
+		fd = -1;
+	}
+	rte_mcfg_mem_write_lock();
+
+	/* make sure the segment doesn't already exist */
+	if (malloc_heap_find_external_seg(va_addr, len) != NULL) {
+		rte_errno = EEXIST;
+		ret = -1;
+		goto unlock;
+	}
+
+	/* get next available socket ID */
+	socket_id = mcfg->next_socket_id;
+	if (socket_id > INT32_MAX) {
+		RTE_LOG(ERR, EAL, "Cannot assign new socket ID's\n");
+		rte_errno = ENOSPC;
+		ret = -1;
+		goto unlock;
+	}
+
+	/* we can create a new memseg */
+	if (malloc_heap_create_external_seg(va_addr, iova_addrs, n,
+			page_sz, "extmem_contig", socket_id, fd) == NULL) {
+		ret = -1;
+		goto unlock;
+	}
+
+	/* memseg list successfully created - increment next socket ID */
+	mcfg->next_socket_id++;
+unlock:
+	rte_mcfg_mem_write_unlock();
+
+	if (iova_addrs != NULL)
+		free(iova_addrs);
+
+	return ret;
+}
+
 int
 rte_extmem_register(void *va_addr, size_t len, rte_iova_t iova_addrs[],
 		unsigned int n_pages, size_t page_sz)
@@ -809,7 +882,7 @@ rte_extmem_register(void *va_addr, size_t len, rte_iova_t iova_addrs[],
 	/* we can create a new memseg */
 	n = len / page_sz;
 	if (malloc_heap_create_external_seg(va_addr, iova_addrs, n,
-			page_sz, "extmem", socket_id) == NULL) {
+			page_sz, "extmem", socket_id, -1) == NULL) {
 		ret = -1;
 		goto unlock;
 	}
diff --git a/lib/librte_eal/common/include/rte_memory.h b/lib/librte_eal/common/include/rte_memory.h
index 3d8d0bd697..e274e47e5e 100644
--- a/lib/librte_eal/common/include/rte_memory.h
+++ b/lib/librte_eal/common/include/rte_memory.h
@@ -451,6 +451,10 @@ rte_memseg_get_fd_offset_thread_unsafe(const struct rte_memseg *ms,
  *   is NULL.
  * @param page_sz
  *   Page size of the underlying memory
+ * @param fd
+ *   File descriptor for the external memory region registered. Must be set to
+ *   -1 if no FD, and ignored if single-segment isn't not used or if iova
+ *   aren't contiguous (iova_addrs != NULL).
  *
  * @return
  *   - 0 on success
@@ -461,6 +465,48 @@ rte_memseg_get_fd_offset_thread_unsafe(const struct rte_memseg *ms,
  */
 __rte_experimental
 int
+rte_extmem_register_contig(void *va_addr, size_t len, rte_iova_t iova_addr,
+		size_t page_sz, int fd);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Register external contiguous memory chunk with DPDK.
+ *
+ * @note Using this API is mutually exclusive with ``rte_malloc`` family of
+ *   API's.
+ *
+ * @note This API will not perform any DMA mapping. It is expected that user
+ *   will do that themselves.
+ *
+ * @note Before accessing this memory in other processes, it needs to be
+ *   attached in each of those processes by calling ``rte_extmem_attach`` in
+ *   each other process.
+ *
+ * @param va_addr
+ *   Start of virtual area to register. Must be aligned by ``page_sz``.
+ * @param len
+ *   Length of virtual area to register. Must be aligned by ``page_sz``.
+ * @param iova_addr
+ *   IOVA address for the contiguous memory chunck. Can be 0, in which case
+ *   page IOVA addresses will be set to RTE_BAD_IOVA.
+ * @param page_sz
+ *   Page size of the underlying memory
+ * @param fd
+ *   File descriptor for the external memory region registered. Must be set to
+ *   -1 if no FD, and ignored if single-segment isn't not used.
+ *
+ * @return
+ *   - 0 on success
+ *   - -1 in case of error, with rte_errno set to one of the following:
+ *     EINVAL - one of the parameters was invalid
+ *     EEXIST - memory chunk is already registered
+ *     ENOSPC - no more space in internal config to store a new memory chunk
+ *     ENOMEM - failed to allocate pages IOVA addresses
+ */
+__rte_experimental
+int
 rte_extmem_register(void *va_addr, size_t len, rte_iova_t iova_addrs[],
 		unsigned int n_pages, size_t page_sz);
 
diff --git a/lib/librte_eal/common/malloc_heap.c b/lib/librte_eal/common/malloc_heap.c
index 842eb9de75..229ab6563c 100644
--- a/lib/librte_eal/common/malloc_heap.c
+++ b/lib/librte_eal/common/malloc_heap.c
@@ -1096,7 +1096,7 @@ destroy_elem(struct malloc_elem *elem, size_t len)
 struct rte_memseg_list *
 malloc_heap_create_external_seg(void *va_addr, rte_iova_t iova_addrs[],
 		unsigned int n_pages, size_t page_sz, const char *seg_name,
-		unsigned int socket_id)
+		unsigned int socket_id, int fd)
 {
 	struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;
 	char fbarray_name[RTE_FBARRAY_NAME_LEN];
@@ -1153,6 +1153,13 @@ malloc_heap_create_external_seg(void *va_addr, rte_iova_t iova_addrs[],
 	msl->version = 0;
 	msl->external = 1;
 
+	if (fd >= 0) {
+		int list_idx = msl - mcfg->memsegs;
+
+		if (eal_memalloc_set_seg_list_fd(list_idx, fd))
+			RTE_LOG(ERR, EAL, "Failed to set segment list FD\n");
+	}
+
 	return msl;
 }
 
@@ -1202,10 +1209,18 @@ malloc_heap_find_external_seg(void *va_addr, size_t len)
 int
 malloc_heap_destroy_external_seg(struct rte_memseg_list *msl)
 {
+	struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;
+	int list_idx;
+
 	/* destroy the fbarray backing this memory */
 	if (rte_fbarray_destroy(&msl->memseg_arr) < 0)
 		return -1;
 
+	list_idx = msl - mcfg->memsegs;
+	if (eal_memalloc_set_seg_list_fd(list_idx, -1))
+		RTE_LOG(ERR, EAL, "Failed to reset segment list FD\n");
+
+
 	/* reset the memseg list */
 	memset(msl, 0, sizeof(*msl));
 
diff --git a/lib/librte_eal/common/malloc_heap.h b/lib/librte_eal/common/malloc_heap.h
index 772736b53f..438ce908de 100644
--- a/lib/librte_eal/common/malloc_heap.h
+++ b/lib/librte_eal/common/malloc_heap.h
@@ -65,7 +65,7 @@ malloc_heap_destroy(struct malloc_heap *heap);
 struct rte_memseg_list *
 malloc_heap_create_external_seg(void *va_addr, rte_iova_t iova_addrs[],
 		unsigned int n_pages, size_t page_sz, const char *seg_name,
-		unsigned int socket_id);
+		unsigned int socket_id, int fd);
 
 struct rte_memseg_list *
 malloc_heap_find_external_seg(void *va_addr, size_t len);
diff --git a/lib/librte_eal/common/rte_malloc.c b/lib/librte_eal/common/rte_malloc.c
index d6026a2b17..aa19b0517f 100644
--- a/lib/librte_eal/common/rte_malloc.c
+++ b/lib/librte_eal/common/rte_malloc.c
@@ -389,7 +389,7 @@ rte_malloc_heap_memory_add(const char *heap_name, void *va_addr, size_t len,
 	n = len / page_sz;
 
 	msl = malloc_heap_create_external_seg(va_addr, iova_addrs, n, page_sz,
-			heap_name, heap->socket_id);
+			heap_name, heap->socket_id, -1);
 	if (msl == NULL) {
 		ret = -1;
 		goto unlock;
diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
index e38d02530c..ddcb4b0512 100644
--- a/lib/librte_eal/rte_eal_version.map
+++ b/lib/librte_eal/rte_eal_version.map
@@ -332,4 +332,7 @@ EXPERIMENTAL {
 	# added in 19.11
 	rte_log_get_stream;
 	rte_mcfg_get_single_file_segments;
+
+	# added in 20.02
+	rte_extmem_register_contig;
 };
-- 
2.21.0

[dpdk-dev] [PATCH 2/4] eal: allow getting memory segment FD with external memory

[Maxime Coquelin]

Now that it is possible to associate a file descriptor when
registering external and contiguous memory, let's allow it.

If no FD is available for the memory segment, ENOENT error
is returned.

Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 lib/librte_eal/common/eal_common_memory.c | 12 ------------
 1 file changed, 12 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_memory.c b/lib/librte_eal/common/eal_common_memory.c
index 7a4b371828..cc660f5b85 100644
--- a/lib/librte_eal/common/eal_common_memory.c
+++ b/lib/librte_eal/common/eal_common_memory.c
@@ -691,12 +691,6 @@ rte_memseg_get_fd_thread_unsafe(const struct rte_memseg *ms)
 		return -1;
 	}
 
-	/* segment fd API is not supported for external segments */
-	if (msl->external) {
-		rte_errno = ENOTSUP;
-		return -1;
-	}
-
 	ret = eal_memalloc_get_seg_fd(msl_idx, seg_idx);
 	if (ret < 0) {
 		rte_errno = -ret;
@@ -746,12 +740,6 @@ rte_memseg_get_fd_offset_thread_unsafe(const struct rte_memseg *ms,
 		return -1;
 	}
 
-	/* segment fd API is not supported for external segments */
-	if (msl->external) {
-		rte_errno = ENOTSUP;
-		return -1;
-	}
-
 	ret = eal_memalloc_get_seg_fd_offset(msl_idx, seg_idx, offset);
 	if (ret < 0) {
 		rte_errno = -ret;
-- 
2.21.0

[dpdk-dev] [PATCH 3/4] bus/vdev: add DMA mapping supprt

[Maxime Coquelin]

This patch enables VDEVs to implement their own DMA map
and unmap callbacks.

It will be used by Virtio-user PMD to support external
memory registered by applications like Seastar or VPP.

Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 drivers/bus/vdev/rte_bus_vdev.h | 24 ++++++++++++++++
 drivers/bus/vdev/vdev.c         | 50 +++++++++++++++++++++++++++++++++
 2 files changed, 74 insertions(+)

diff --git a/drivers/bus/vdev/rte_bus_vdev.h b/drivers/bus/vdev/rte_bus_vdev.h
index 2bc46530c9..6f920cb064 100644
--- a/drivers/bus/vdev/rte_bus_vdev.h
+++ b/drivers/bus/vdev/rte_bus_vdev.h
@@ -53,6 +53,18 @@ rte_vdev_device_args(const struct rte_vdev_device *dev)
 /** Double linked list of virtual device drivers. */
 TAILQ_HEAD(vdev_driver_list, rte_vdev_driver);
 
+/**
+ * Virtual device DMA map function.
+ */
+typedef int (rte_vdev_dma_map_t)(struct rte_vdev_device *dev, void *addr,
+			    uint64_t iova, size_t len);
+
+/**
+ * Virtual device DMA unmap function.
+ */
+typedef int (rte_vdev_dma_unmap_t)(struct rte_vdev_device *dev, void *addr,
+			    uint64_t iova, size_t len);
+
 /**
  * Probe function called for each virtual device driver once.
  */
@@ -69,10 +81,22 @@ typedef int (rte_vdev_remove_t)(struct rte_vdev_device *dev);
 struct rte_vdev_driver {
 	TAILQ_ENTRY(rte_vdev_driver) next; /**< Next in list. */
 	struct rte_driver driver;      /**< Inherited general driver. */
+	rte_vdev_dma_map_t *dma_map;   /**<Virtual device DMA map function. */
+	rte_vdev_dma_unmap_t *dma_unmap;   /**<Virtual device DMA unmap function. */
 	rte_vdev_probe_t *probe;       /**< Virtual device probe function. */
 	rte_vdev_remove_t *remove;     /**< Virtual device remove function. */
 };
 
+/**
+ * @internal
+ * Helper macro for drivers that need to convert to struct rte_vdev_driver.
+ */
+#define RTE_DRV_TO_VDRV(ptr) \
+	container_of(ptr, struct rte_vdev_driver, driver)
+
+#define RTE_DRV_TO_VDRV_CONST(ptr) \
+	container_of(ptr, const struct rte_vdev_driver, driver)
+
 /**
  * Register a virtual device driver.
  *
diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
index a89ea23530..8bdca23448 100644
--- a/drivers/bus/vdev/vdev.c
+++ b/drivers/bus/vdev/vdev.c
@@ -546,6 +546,54 @@ vdev_unplug(struct rte_device *dev)
 	return rte_vdev_uninit(dev->name);
 }
 
+static int
+vdev_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+	struct rte_vdev_device *vdev = RTE_DEV_TO_VDEV(dev);
+	const struct rte_vdev_driver *vdrv;
+
+	if (!dev || !dev->driver) {
+		rte_errno = EINVAL;
+		return -1;
+	}
+
+	vdrv = RTE_DRV_TO_VDRV_CONST(dev->driver);
+	if (!vdrv) {
+		rte_errno = EINVAL;
+		return -1;
+	}
+
+	if (vdrv->dma_map)
+		return vdrv->dma_map(vdev, addr, iova, len);
+
+	rte_errno = ENOTSUP;
+	return -1;
+}
+
+static int
+vdev_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+	struct rte_vdev_device *vdev = RTE_DEV_TO_VDEV(dev);
+	const struct rte_vdev_driver *vdrv;
+
+	if (!dev || !dev->driver) {
+		rte_errno = EINVAL;
+		return -1;
+	}
+
+	vdrv = RTE_DRV_TO_VDRV_CONST(dev->driver);
+	if (!vdrv) {
+		rte_errno = EINVAL;
+		return -1;
+	}
+
+	if (vdrv->dma_map)
+		return vdrv->dma_unmap(vdev, addr, iova, len);
+
+	rte_errno = ENOTSUP;
+	return -1;
+}
+
 static struct rte_bus rte_vdev_bus = {
 	.scan = vdev_scan,
 	.probe = vdev_probe,
@@ -553,6 +601,8 @@ static struct rte_bus rte_vdev_bus = {
 	.plug = vdev_plug,
 	.unplug = vdev_unplug,
 	.parse = vdev_parse,
+	.dma_map = vdev_dma_map,
+	.dma_unmap = vdev_dma_unmap,
 	.dev_iterate = rte_vdev_dev_iterate,
 };
 
-- 
2.21.0

[dpdk-dev] [PATCH 4/4] net/virtio: add DMA mapping callback to virtio-user

[Maxime Coquelin]

Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 .../net/virtio/virtio_user/virtio_user_dev.c  | 10 +---
 .../net/virtio/virtio_user/virtio_user_dev.h  |  3 +
 drivers/net/virtio/virtio_user_ethdev.c       | 58 +++++++++++++++++++
 3 files changed, 63 insertions(+), 8 deletions(-)

diff --git a/drivers/net/virtio/virtio_user/virtio_user_dev.c b/drivers/net/virtio/virtio_user/virtio_user_dev.c
index ea016e85d8..c72a9543cd 100644
--- a/drivers/net/virtio/virtio_user/virtio_user_dev.c
+++ b/drivers/net/virtio/virtio_user/virtio_user_dev.c
@@ -313,21 +313,15 @@ virtio_user_fill_intr_handle(struct virtio_user_dev *dev)
 	return 0;
 }
 
-static void
+void
 virtio_user_mem_event_cb(enum rte_mem_event type __rte_unused,
-			 const void *addr,
+			 const void *addr __rte_unused,
 			 size_t len __rte_unused,
 			 void *arg)
 {
 	struct virtio_user_dev *dev = arg;
-	struct rte_memseg_list *msl;
 	uint16_t i;
 
-	/* ignore externally allocated memory */
-	msl = rte_mem_virt2memseg_list(addr);
-	if (msl->external)
-		return;
-
 	pthread_mutex_lock(&dev->mutex);
 
 	if (dev->started == false)
diff --git a/drivers/net/virtio/virtio_user/virtio_user_dev.h b/drivers/net/virtio/virtio_user/virtio_user_dev.h
index ad86837717..5a147c74f0 100644
--- a/drivers/net/virtio/virtio_user/virtio_user_dev.h
+++ b/drivers/net/virtio/virtio_user/virtio_user_dev.h
@@ -67,4 +67,7 @@ void virtio_user_handle_cq(struct virtio_user_dev *dev, uint16_t queue_idx);
 void virtio_user_handle_cq_packed(struct virtio_user_dev *dev,
 				  uint16_t queue_idx);
 uint8_t virtio_user_handle_mq(struct virtio_user_dev *dev, uint16_t q_pairs);
+void
+virtio_user_mem_event_cb(enum rte_mem_event type,
+			const void *addr, size_t len , void *arg);
 #endif
diff --git a/drivers/net/virtio/virtio_user_ethdev.c b/drivers/net/virtio/virtio_user_ethdev.c
index 3fc1725736..fd3ce2dfab 100644
--- a/drivers/net/virtio/virtio_user_ethdev.c
+++ b/drivers/net/virtio/virtio_user_ethdev.c
@@ -385,6 +385,62 @@ const struct virtio_pci_ops virtio_user_ops = {
 	.notify_queue	= virtio_user_notify_queue,
 };
 
+static int
+virtio_user_dma_map(struct rte_vdev_device *vdev, void *addr,
+		uint64_t iova __rte_unused, size_t len)
+{
+	const char *name;
+	struct rte_eth_dev *eth_dev;
+	struct virtio_hw *hw;
+	struct virtio_user_dev *dev;
+
+	if (!vdev)
+		return -EINVAL;
+
+	name = rte_vdev_device_name(vdev);
+	eth_dev = rte_eth_dev_allocated(name);
+	if (!eth_dev)
+		return -ENODEV;
+
+	hw = eth_dev->data->dev_private;
+	dev = hw->virtio_user_dev;
+
+	virtio_user_mem_event_cb(RTE_MEM_EVENT_ALLOC,
+						 addr,
+						 len,
+						 (void *)dev);
+
+	return 0;
+}
+
+static int
+virtio_user_dma_unmap(struct rte_vdev_device *vdev, void *addr,
+		uint64_t iova __rte_unused, size_t len)
+{
+	const char *name;
+	struct rte_eth_dev *eth_dev;
+	struct virtio_hw *hw;
+	struct virtio_user_dev *dev;
+
+	if (!vdev)
+		return -EINVAL;
+
+	name = rte_vdev_device_name(vdev);
+	eth_dev = rte_eth_dev_allocated(name);
+	if (!eth_dev)
+		return -ENODEV;
+
+	hw = eth_dev->data->dev_private;
+	dev = hw->virtio_user_dev;
+
+	virtio_user_mem_event_cb(RTE_MEM_EVENT_FREE,
+						 addr,
+						 len,
+						 (void *)dev);
+
+	return 0;
+}
+
 static const char *valid_args[] = {
 #define VIRTIO_USER_ARG_QUEUES_NUM     "queues"
 	VIRTIO_USER_ARG_QUEUES_NUM,
@@ -717,6 +773,8 @@ virtio_user_pmd_remove(struct rte_vdev_device *vdev)
 }
 
 static struct rte_vdev_driver virtio_user_driver = {
+	.dma_map = virtio_user_dma_map,
+	.dma_unmap = virtio_user_dma_unmap,
 	.probe = virtio_user_pmd_probe,
 	.remove = virtio_user_pmd_remove,
 };
-- 
2.21.0

Re: [dpdk-dev] [PATCH 0/4] Add external contiguous memory support to Virtio-user

[Maxime Coquelin]

Hi Anatoly,

Any feedback on the series?
This is important for vDPA support, to be compliant with containeriez
application using external memory.

Thanks,
Maxime

On 12/13/19 3:13 PM, Maxime Coquelin wrote:
> Somme applications/frameworks using DPDK allocate their own memory
> for its buffers. This is the case of Seastar, and also VPP.
> 
> It is currently not possible to use Virtio-user PMD with these
> applications because Virtio-user requires file descriptor for this
> memory to be shared with the vhost-user backend (vDPA in our case [0]).
> 
> This series introduces a new API in EAL to register external memory
> that is contiguous in both the VA and the IOVA space.
> 
> It also enables VDEVs to implement the DMA map/unmap callback, and
> implement it in Virtio-user PMD.
> 
> Please note that extmem unit tests are missing. I'm working on it
> and it will be in v2. Just share this v1 for early reviews.
> 
> Maxime Coquelin (4):
>   eal: add new API to register contiguous external memory
>   eal: allow getting memory segment FD with external memory
>   bus/vdev: add DMA mapping supprt
>   net/virtio: add DMA mapping callback to virtio-user
> 
>  drivers/bus/vdev/rte_bus_vdev.h               | 24 +++++
>  drivers/bus/vdev/vdev.c                       | 50 +++++++++++
>  .../net/virtio/virtio_user/virtio_user_dev.c  | 10 +--
>  .../net/virtio/virtio_user/virtio_user_dev.h  |  3 +
>  drivers/net/virtio/virtio_user_ethdev.c       | 58 +++++++++++++
>  lib/librte_eal/common/eal_common_memory.c     | 87 ++++++++++++++++---
>  lib/librte_eal/common/include/rte_memory.h    | 46 ++++++++++
>  lib/librte_eal/common/malloc_heap.c           | 17 +++-
>  lib/librte_eal/common/malloc_heap.h           |  2 +-
>  lib/librte_eal/common/rte_malloc.c            |  2 +-
>  lib/librte_eal/rte_eal_version.map            |  3 +
>  11 files changed, 278 insertions(+), 24 deletions(-)
> 

Re: [dpdk-dev] [PATCH 1/4] eal: add new API to register contiguous external memory

[Burakov, Anatoly]

On 13-Dec-19 2:13 PM, Maxime Coquelin wrote:
> This new API allows to pass a file descriptor while
> registering external and contiguous memory in the IOVA space.
> 
> This is required for using Virtio-user PMD with application
> using external memory for the mbuf's buffers, like Seastar
> or VPP.
> 
> FD is only attached to the segments if single_file_segment
> option is enabled.
>  > Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Hi Maxime,

We've discussed this privately already, but i'll send feedback anyway 
just so that it's visible.

> ---
>   lib/librte_eal/common/eal_common_memory.c  | 75 +++++++++++++++++++++-
>   lib/librte_eal/common/include/rte_memory.h | 46 +++++++++++++
>   lib/librte_eal/common/malloc_heap.c        | 17 ++++-
>   lib/librte_eal/common/malloc_heap.h        |  2 +-
>   lib/librte_eal/common/rte_malloc.c         |  2 +-
>   lib/librte_eal/rte_eal_version.map         |  3 +
>   6 files changed, 141 insertions(+), 4 deletions(-)
> 
> diff --git a/lib/librte_eal/common/eal_common_memory.c b/lib/librte_eal/common/eal_common_memory.c
> index 4a9cc1f19a..7a4b371828 100644
> --- a/lib/librte_eal/common/eal_common_memory.c
> +++ b/lib/librte_eal/common/eal_common_memory.c
> @@ -772,6 +772,79 @@ rte_memseg_get_fd_offset(const struct rte_memseg *ms, size_t *offset)
>   	return ret;
>   }
>   
> +int
> +rte_extmem_register_contig(void *va_addr, size_t len, rte_iova_t iova_addr,
> +		size_t page_sz, int fd)
> +{

I don't see why do you have to link IOVA-contiguousness with setting an 
fd - a chunk of memory backed by an fd does not necessarily have to be 
IOVA-contiguous.

In fact, registering IOVA-contiguous memory is already possible - you 
just specify the IOVA table such that each successive IOVA is contiguous 
to the preceding one.

So, IMO, the 'fd' part should be decoupled from registering.

Also, we support two modes of operation (fd per segment and fd per 
segment list), it would be nice to have both.

For example, consider the following:

// set fd for an entire segment list, addressed by VA and len.
// must match a memory chunk that was registered earlier
rte_extmem_set_seg_fd(..., int fd);

// set per-segment fd's for the segment list, addressed by VA and len.
// must match a memory chunk that was registered earlier, and length of
// fd array must add up to length of the chunk
rte_extmem_set_seg_fd_list(..., int *fd, int len);

This would essentially accomplish what you're trying to do by just 
adding the missing piece required.

Question: do we want to be able to set segment fd's for externally 
allocated by internally managed memory? E.g. something that was added 
via malloc_heap_add_memory rather than extmem_register? I would guess 
so, so maybe it shouldn't be under rte_extmem namespace.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH 0/4] Add external contiguous memory support to Virtio-user

[David Marchand]

On Fri, Dec 13, 2019 at 3:13 PM Maxime Coquelin
<maxime.coquelin@redhat.com> wrote:
>
> Somme applications/frameworks using DPDK allocate their own memory
> for its buffers. This is the case of Seastar, and also VPP.
>
> It is currently not possible to use Virtio-user PMD with these
> applications because Virtio-user requires file descriptor for this
> memory to be shared with the vhost-user backend (vDPA in our case [0]).
>
> This series introduces a new API in EAL to register external memory
> that is contiguous in both the VA and the IOVA space.
>
> It also enables VDEVs to implement the DMA map/unmap callback, and
> implement it in Virtio-user PMD.
>
> Please note that extmem unit tests are missing. I'm working on it
> and it will be in v2. Just share this v1 for early reviews.
>
> Maxime Coquelin (4):
>   eal: add new API to register contiguous external memory
>   eal: allow getting memory segment FD with external memory
>   bus/vdev: add DMA mapping supprt
>   net/virtio: add DMA mapping callback to virtio-user
>
>  drivers/bus/vdev/rte_bus_vdev.h               | 24 +++++
>  drivers/bus/vdev/vdev.c                       | 50 +++++++++++
>  .../net/virtio/virtio_user/virtio_user_dev.c  | 10 +--
>  .../net/virtio/virtio_user/virtio_user_dev.h  |  3 +
>  drivers/net/virtio/virtio_user_ethdev.c       | 58 +++++++++++++
>  lib/librte_eal/common/eal_common_memory.c     | 87 ++++++++++++++++---
>  lib/librte_eal/common/include/rte_memory.h    | 46 ++++++++++
>  lib/librte_eal/common/malloc_heap.c           | 17 +++-
>  lib/librte_eal/common/malloc_heap.h           |  2 +-
>  lib/librte_eal/common/rte_malloc.c            |  2 +-
>  lib/librte_eal/rte_eal_version.map            |  3 +
>  11 files changed, 278 insertions(+), 24 deletions(-)

I am not sure this feature is still needed as there was no activity
for a long time.

There was some feedback from Anatoly and the series needs a rebase in any case.
I marked the series as Changes requested so it is clear it can't be
considered for merging in its current state.

Thanks.

-- 
David Marchand