[RFC PATCH] dmadev: offload to free source buffer

[RFC PATCH] dmadev: offload to free source buffer

[Amit Prakash Shukla]

This changeset adds support in DMA library to free source DMA buffer by
hardware. On a supported hardware, application can pass on the mempool
information as part of vchan config when the DMA transfer direction is
configured as RTE_DMA_DIR_MEM_TO_DEV.

Signed-off-by: Amit Prakash Shukla <amitprakashs@marvell.com>
---
 lib/dmadev/rte_dmadev.h | 45 +++++++++++++++++++++++++++++------------
 1 file changed, 32 insertions(+), 13 deletions(-)

diff --git a/lib/dmadev/rte_dmadev.h b/lib/dmadev/rte_dmadev.h
index e61d71959e..98539e5830 100644
--- a/lib/dmadev/rte_dmadev.h
+++ b/lib/dmadev/rte_dmadev.h
@@ -241,26 +241,26 @@ int16_t rte_dma_next_dev(int16_t start_dev_id);
  * @see struct rte_dma_info::dev_capa
  */
 /** Support memory-to-memory transfer */
-#define RTE_DMA_CAPA_MEM_TO_MEM		RTE_BIT64(0)
+#define RTE_DMA_CAPA_MEM_TO_MEM				RTE_BIT64(0)
 /** Support memory-to-device transfer. */
-#define RTE_DMA_CAPA_MEM_TO_DEV		RTE_BIT64(1)
+#define RTE_DMA_CAPA_MEM_TO_DEV				RTE_BIT64(1)
 /** Support device-to-memory transfer. */
-#define RTE_DMA_CAPA_DEV_TO_MEM		RTE_BIT64(2)
+#define RTE_DMA_CAPA_DEV_TO_MEM				RTE_BIT64(2)
 /** Support device-to-device transfer. */
-#define RTE_DMA_CAPA_DEV_TO_DEV		RTE_BIT64(3)
+#define RTE_DMA_CAPA_DEV_TO_DEV				RTE_BIT64(3)
 /** Support SVA which could use VA as DMA address.
  * If device support SVA then application could pass any VA address like memory
  * from rte_malloc(), rte_memzone(), malloc, stack memory.
  * If device don't support SVA, then application should pass IOVA address which
  * from rte_malloc(), rte_memzone().
  */
-#define RTE_DMA_CAPA_SVA                RTE_BIT64(4)
+#define RTE_DMA_CAPA_SVA				RTE_BIT64(4)
 /** Support work in silent mode.
  * In this mode, application don't required to invoke rte_dma_completed*()
  * API.
  * @see struct rte_dma_conf::silent_mode
  */
-#define RTE_DMA_CAPA_SILENT             RTE_BIT64(5)
+#define RTE_DMA_CAPA_SILENT				RTE_BIT64(5)
 /** Supports error handling
  *
  * With this bit set, invalid input addresses will be reported as operation failures
@@ -268,16 +268,18 @@ int16_t rte_dma_next_dev(int16_t start_dev_id);
  * Without this bit set, invalid data is not handled by either HW or driver, so user
  * must ensure that all memory addresses are valid and accessible by HW.
  */
-#define RTE_DMA_CAPA_HANDLES_ERRORS	RTE_BIT64(6)
+#define RTE_DMA_CAPA_HANDLES_ERRORS			RTE_BIT64(6)
 /** Support copy operation.
  * This capability start with index of 32, so that it could leave gap between
  * normal capability and ops capability.
  */
-#define RTE_DMA_CAPA_OPS_COPY           RTE_BIT64(32)
+#define RTE_DMA_CAPA_OPS_COPY				RTE_BIT64(32)
 /** Support scatter-gather list copy operation. */
-#define RTE_DMA_CAPA_OPS_COPY_SG	RTE_BIT64(33)
+#define RTE_DMA_CAPA_OPS_COPY_SG			RTE_BIT64(33)
 /** Support fill operation. */
-#define RTE_DMA_CAPA_OPS_FILL		RTE_BIT64(34)
+#define RTE_DMA_CAPA_OPS_FILL				RTE_BIT64(34)
+/** Support for source buffer free for mem to dev transfer. */
+#define RTE_DMA_CAPA_MEM_TO_DEV_SOURCE_BUFFER_FREE	RTE_BIT64(35)
 /**@}*/
 
 /**
@@ -582,6 +584,16 @@ struct rte_dma_vchan_conf {
 	 * @see struct rte_dma_port_param
 	 */
 	struct rte_dma_port_param dst_port;
+	/** mempool from which source buffer is allocated. mempool info is used
+	 * for freeing source buffer by hardware when configured direction is
+	 * RTE_DMA_DIR_MEM_TO_DEV. To free the source buffer by hardware,
+	 * RTE_DMA_OP_FLAG_FREE_SBUF must be set while calling rte_dma_copy and
+	 * rte_dma_copy_sg().
+	 *
+	 * @see RTE_DMA_OP_FLAG_FREE_SBUF
+	 */
+	struct rte_mempool *mem_to_dev_src_buf_pool;
+
 };
 
 /**
@@ -808,17 +820,24 @@ struct rte_dma_sge {
  * If the specify DMA HW works in-order (it means it has default fence between
  * operations), this flag could be NOP.
  */
-#define RTE_DMA_OP_FLAG_FENCE   RTE_BIT64(0)
+#define RTE_DMA_OP_FLAG_FENCE      RTE_BIT64(0)
 /** Submit flag.
  * It means the operation with this flag must issue doorbell to hardware after
  * enqueued jobs.
  */
-#define RTE_DMA_OP_FLAG_SUBMIT  RTE_BIT64(1)
+#define RTE_DMA_OP_FLAG_SUBMIT     RTE_BIT64(1)
 /** Write data to low level cache hint.
  * Used for performance optimization, this is just a hint, and there is no
  * capability bit for this, driver should not return error if this flag was set.
  */
-#define RTE_DMA_OP_FLAG_LLC     RTE_BIT64(2)
+#define RTE_DMA_OP_FLAG_LLC        RTE_BIT64(2)
+/** Mem to dev source buffer free flag.
+ * Used for freeing source DMA buffer by hardware when the transfer direction is
+ * configured as RTE_DMA_DIR_MEM_TO_DEV.
+ *
+ * @see struct rte_dma_vchan_conf::mem_to_dev_src_buf_pool
+ */
+#define RTE_DMA_OP_FLAG_FREE_SBUF  RTE_BIT64(3)
 /**@}*/
 
 /**
-- 
2.25.1

RE: [RFC PATCH] dmadev: offload to free source buffer

[Morten Brørup]

> From: Amit Prakash Shukla [mailto:amitprakashs@marvell.com]
> Sent: Wednesday, 9 August 2023 08.09
> 
> This changeset adds support in DMA library to free source DMA buffer by
> hardware. On a supported hardware, application can pass on the mempool
> information as part of vchan config when the DMA transfer direction is
> configured as RTE_DMA_DIR_MEM_TO_DEV.

Isn't the DMA source buffer a memory area, and what needs to be freed is the mbuf holding the memory area, i.e. two different pointers?

I like the concept. Something similar might also be useful for RTE_DMA_DIR_MEM_TO_MEM, e.g. packet capture. Although such a use case might require decrementing the mbuf refcount instead of freeing the mbuf directly to the mempool.

PS: It has been a while since I looked at the DMA library, so ignore my comments if I got this wrong.

RE: [RFC PATCH] dmadev: offload to free source buffer

[Amit Prakash Shukla]

Hi Morten,

Please find my reply in-line.

Thanks,
Amit Shukla

> -----Original Message-----
> From: Morten Brørup <mb@smartsharesystems.com>
> Sent: Wednesday, August 9, 2023 2:37 PM
> To: Amit Prakash Shukla <amitprakashs@marvell.com>; Chengwen Feng
> <fengchengwen@huawei.com>; Kevin Laatz <kevin.laatz@intel.com>; Bruce
> Richardson <bruce.richardson@intel.com>
> Cc: dev@dpdk.org; Jerin Jacob Kollanukkaran <jerinj@marvell.com>;
> conor.walsh@intel.com; Vamsi Krishna Attunuru <vattunuru@marvell.com>;
> g.singh@nxp.com; sachin.saxena@oss.nxp.com; hemant.agrawal@nxp.com;
> cheng1.jiang@intel.com; Nithin Kumar Dabilpuram
> <ndabilpuram@marvell.com>; Anoob Joseph <anoobj@marvell.com>
> Subject: [EXT] RE: [RFC PATCH] dmadev: offload to free source buffer
> 
> External Email
> 
> ----------------------------------------------------------------------
> > From: Amit Prakash Shukla [mailto:amitprakashs@marvell.com]
> > Sent: Wednesday, 9 August 2023 08.09
> >
> > This changeset adds support in DMA library to free source DMA buffer
> > by hardware. On a supported hardware, application can pass on the
> > mempool information as part of vchan config when the DMA transfer
> > direction is configured as RTE_DMA_DIR_MEM_TO_DEV.
> 
> Isn't the DMA source buffer a memory area, and what needs to be freed is
> the mbuf holding the memory area, i.e. two different pointers?
No, it is same pointer. Assume mbuf created via mempool, mempool needs to be given via vchan config and iova passed to rte_dma_copy/rte_dma_copy_sg's can be any address in mbuf area of given mempool element.
For example, mempool element size is S. dequeued buff from mempool is at X. Any address in (X, X+S) can be given as iova to rte_dma_copy.

> 
> I like the concept. Something similar might also be useful for
> RTE_DMA_DIR_MEM_TO_MEM, e.g. packet capture. Although such a use
> case might require decrementing the mbuf refcount instead of freeing the
> mbuf directly to the mempool.
This operation is not supported in our hardware. It can be implemented in future if any hardware supports it.

> 
> PS: It has been a while since I looked at the DMA library, so ignore my
> comments if I got this wrong.

RE: [RFC PATCH] dmadev: offload to free source buffer

[Morten Brørup]

> From: Amit Prakash Shukla [mailto:amitprakashs@marvell.com]
> Sent: Wednesday, 9 August 2023 16.27
> 
> > From: Morten Brørup <mb@smartsharesystems.com>
> > Sent: Wednesday, August 9, 2023 2:37 PM
> >
> > > From: Amit Prakash Shukla [mailto:amitprakashs@marvell.com]
> > > Sent: Wednesday, 9 August 2023 08.09
> > >
> > > This changeset adds support in DMA library to free source DMA buffer
> > > by hardware. On a supported hardware, application can pass on the
> > > mempool information as part of vchan config when the DMA transfer
> > > direction is configured as RTE_DMA_DIR_MEM_TO_DEV.
> >
> > Isn't the DMA source buffer a memory area, and what needs to be freed
> is
> > the mbuf holding the memory area, i.e. two different pointers?
> No, it is same pointer. Assume mbuf created via mempool, mempool needs to
> be given via vchan config and iova passed to
> rte_dma_copy/rte_dma_copy_sg's can be any address in mbuf area of given
> mempool element.
> For example, mempool element size is S. dequeued buff from mempool is at
> X. Any address in (X, X+S) can be given as iova to rte_dma_copy.

So the DMA library determines the pointer to the mbuf (in the given mempool) by looking at the iova passed to rte_dma_copy/rte_dma_copy_sg, and then calls rte_mempool_put with that pointer?

> 
> >
> > I like the concept. Something similar might also be useful for
> > RTE_DMA_DIR_MEM_TO_MEM, e.g. packet capture. Although such a use
> > case might require decrementing the mbuf refcount instead of freeing
> the
> > mbuf directly to the mempool.
> This operation is not supported in our hardware. It can be implemented in
> future if any hardware supports it.

OK, I didn't expect that - just floating the idea. :-)

> 
> >
> > PS: It has been a while since I looked at the DMA library, so ignore my
> > comments if I got this wrong.

RE: [RFC PATCH] dmadev: offload to free source buffer

[Amit Prakash Shukla]

> -----Original Message-----
> From: Morten Brørup <mb@smartsharesystems.com>
> Sent: Wednesday, August 9, 2023 8:19 PM
> To: Amit Prakash Shukla <amitprakashs@marvell.com>; Chengwen Feng
> <fengchengwen@huawei.com>; Kevin Laatz <kevin.laatz@intel.com>; Bruce
> Richardson <bruce.richardson@intel.com>
> Cc: dev@dpdk.org; Jerin Jacob Kollanukkaran <jerinj@marvell.com>;
> conor.walsh@intel.com; Vamsi Krishna Attunuru <vattunuru@marvell.com>;
> g.singh@nxp.com; sachin.saxena@oss.nxp.com; hemant.agrawal@nxp.com;
> cheng1.jiang@intel.com; Nithin Kumar Dabilpuram
> <ndabilpuram@marvell.com>; Anoob Joseph <anoobj@marvell.com>
> Subject: [EXT] RE: [RFC PATCH] dmadev: offload to free source buffer
> 
> External Email
> 
> ----------------------------------------------------------------------
> > From: Amit Prakash Shukla [mailto:amitprakashs@marvell.com]
> > Sent: Wednesday, 9 August 2023 16.27
> >
> > > From: Morten Brørup <mb@smartsharesystems.com>
> > > Sent: Wednesday, August 9, 2023 2:37 PM
> > >
> > > > From: Amit Prakash Shukla [mailto:amitprakashs@marvell.com]
> > > > Sent: Wednesday, 9 August 2023 08.09
> > > >
> > > > This changeset adds support in DMA library to free source DMA
> > > > buffer by hardware. On a supported hardware, application can pass
> > > > on the mempool information as part of vchan config when the DMA
> > > > transfer direction is configured as RTE_DMA_DIR_MEM_TO_DEV.
> > >
> > > Isn't the DMA source buffer a memory area, and what needs to be
> > > freed
> > is
> > > the mbuf holding the memory area, i.e. two different pointers?
> > No, it is same pointer. Assume mbuf created via mempool, mempool needs
> > to be given via vchan config and iova passed to
> > rte_dma_copy/rte_dma_copy_sg's can be any address in mbuf area of
> > given mempool element.
> > For example, mempool element size is S. dequeued buff from mempool is
> > at X. Any address in (X, X+S) can be given as iova to rte_dma_copy.
> 
> So the DMA library determines the pointer to the mbuf (in the given
> mempool) by looking at the iova passed to rte_dma_copy/rte_dma_copy_sg,
> and then calls rte_mempool_put with that pointer?

No. DMA hardware would determine the pointer to the mbuf using iova address and mempool. Hardware will free the buffer, on completion of data transfer.

> 
> >
> > >
> > > I like the concept. Something similar might also be useful for
> > > RTE_DMA_DIR_MEM_TO_MEM, e.g. packet capture. Although such a use
> > > case might require decrementing the mbuf refcount instead of freeing
> > the
> > > mbuf directly to the mempool.
> > This operation is not supported in our hardware. It can be implemented
> > in future if any hardware supports it.
> 
> OK, I didn't expect that - just floating the idea. :-)
> 
> >
> > >
> > > PS: It has been a while since I looked at the DMA library, so ignore
> > > my comments if I got this wrong.

RE: [RFC PATCH] dmadev: offload to free source buffer

[Morten Brørup]

> From: Amit Prakash Shukla [mailto:amitprakashs@marvell.com]
> Sent: Wednesday, 9 August 2023 20.12
> 
> > From: Morten Brørup <mb@smartsharesystems.com>
> > Sent: Wednesday, August 9, 2023 8:19 PM
> >
> > > From: Amit Prakash Shukla [mailto:amitprakashs@marvell.com]
> > > Sent: Wednesday, 9 August 2023 16.27
> > >
> > > > From: Morten Brørup <mb@smartsharesystems.com>
> > > > Sent: Wednesday, August 9, 2023 2:37 PM
> > > >
> > > > > From: Amit Prakash Shukla [mailto:amitprakashs@marvell.com]
> > > > > Sent: Wednesday, 9 August 2023 08.09
> > > > >
> > > > > This changeset adds support in DMA library to free source DMA
> > > > > buffer by hardware. On a supported hardware, application can pass
> > > > > on the mempool information as part of vchan config when the DMA
> > > > > transfer direction is configured as RTE_DMA_DIR_MEM_TO_DEV.
> > > >
> > > > Isn't the DMA source buffer a memory area, and what needs to be
> > > > freed
> > > is
> > > > the mbuf holding the memory area, i.e. two different pointers?
> > > No, it is same pointer. Assume mbuf created via mempool, mempool needs
> > > to be given via vchan config and iova passed to
> > > rte_dma_copy/rte_dma_copy_sg's can be any address in mbuf area of
> > > given mempool element.
> > > For example, mempool element size is S. dequeued buff from mempool is
> > > at X. Any address in (X, X+S) can be given as iova to rte_dma_copy.
> >
> > So the DMA library determines the pointer to the mbuf (in the given
> > mempool) by looking at the iova passed to rte_dma_copy/rte_dma_copy_sg,
> > and then calls rte_mempool_put with that pointer?
> 
> No. DMA hardware would determine the pointer to the mbuf using iova address
> and mempool. Hardware will free the buffer, on completion of data transfer.

OK. If there are any requirements to the mempool, it needs to be documented in the source code comments. E.g. does it work with mempools where the mempool driver is an MP_RTS/MC_RTS ring, or a stack?

> 
> >
> > >
> > > >
> > > > I like the concept. Something similar might also be useful for
> > > > RTE_DMA_DIR_MEM_TO_MEM, e.g. packet capture. Although such a use
> > > > case might require decrementing the mbuf refcount instead of freeing
> > > the
> > > > mbuf directly to the mempool.
> > > This operation is not supported in our hardware. It can be implemented
> > > in future if any hardware supports it.
> >
> > OK, I didn't expect that - just floating the idea. :-)
> >
> > >
> > > >
> > > > PS: It has been a while since I looked at the DMA library, so ignore
> > > > my comments if I got this wrong.

RE: [RFC PATCH] dmadev: offload to free source buffer

[Amit Prakash Shukla]

> -----Original Message-----
> From: Morten Brørup <mb@smartsharesystems.com>
> Sent: Thursday, August 10, 2023 3:03 PM
> To: Amit Prakash Shukla <amitprakashs@marvell.com>; Chengwen Feng
> <fengchengwen@huawei.com>; Kevin Laatz <kevin.laatz@intel.com>; Bruce
> Richardson <bruce.richardson@intel.com>
> Cc: dev@dpdk.org; Jerin Jacob Kollanukkaran <jerinj@marvell.com>;
> conor.walsh@intel.com; Vamsi Krishna Attunuru <vattunuru@marvell.com>;
> g.singh@nxp.com; sachin.saxena@oss.nxp.com; hemant.agrawal@nxp.com;
> cheng1.jiang@intel.com; Nithin Kumar Dabilpuram
> <ndabilpuram@marvell.com>; Anoob Joseph <anoobj@marvell.com>
> Subject: [EXT] RE: [RFC PATCH] dmadev: offload to free source buffer
> 
> External Email
> 
> ----------------------------------------------------------------------
> > From: Amit Prakash Shukla [mailto:amitprakashs@marvell.com]
> > Sent: Wednesday, 9 August 2023 20.12
> >
> > > From: Morten Brørup <mb@smartsharesystems.com>
> > > Sent: Wednesday, August 9, 2023 8:19 PM
> > >
> > > > From: Amit Prakash Shukla [mailto:amitprakashs@marvell.com]
> > > > Sent: Wednesday, 9 August 2023 16.27
> > > >
> > > > > From: Morten Brørup <mb@smartsharesystems.com>
> > > > > Sent: Wednesday, August 9, 2023 2:37 PM
> > > > >
> > > > > > From: Amit Prakash Shukla [mailto:amitprakashs@marvell.com]
> > > > > > Sent: Wednesday, 9 August 2023 08.09
> > > > > >
> > > > > > This changeset adds support in DMA library to free source DMA
> > > > > > buffer by hardware. On a supported hardware, application can
> > > > > > pass on the mempool information as part of vchan config when
> > > > > > the DMA transfer direction is configured as
> RTE_DMA_DIR_MEM_TO_DEV.
> > > > >
> > > > > Isn't the DMA source buffer a memory area, and what needs to be
> > > > > freed
> > > > is
> > > > > the mbuf holding the memory area, i.e. two different pointers?
> > > > No, it is same pointer. Assume mbuf created via mempool, mempool
> > > > needs to be given via vchan config and iova passed to
> > > > rte_dma_copy/rte_dma_copy_sg's can be any address in mbuf area of
> > > > given mempool element.
> > > > For example, mempool element size is S. dequeued buff from
> mempool
> > > > is at X. Any address in (X, X+S) can be given as iova to rte_dma_copy.
> > >
> > > So the DMA library determines the pointer to the mbuf (in the given
> > > mempool) by looking at the iova passed to
> > > rte_dma_copy/rte_dma_copy_sg, and then calls rte_mempool_put with
> that pointer?
> >
> > No. DMA hardware would determine the pointer to the mbuf using iova
> > address and mempool. Hardware will free the buffer, on completion of
> data transfer.
> 
> OK. If there are any requirements to the mempool, it needs to be
> documented in the source code comments. E.g. does it work with mempools
> where the mempool driver is an MP_RTS/MC_RTS ring, or a stack?

I think adding a comment, related to type of supported mempool, in dma library code might not be needed as it is driver implementation dependent. Call to dev->dev_ops->vchan_setup for the driver shall check and return error for unsupported type of mempool.

> 
> >
> > >
> > > >
> > > > >
> > > > > I like the concept. Something similar might also be useful for
> > > > > RTE_DMA_DIR_MEM_TO_MEM, e.g. packet capture. Although such a
> use
> > > > > case might require decrementing the mbuf refcount instead of
> > > > > freeing
> > > > the
> > > > > mbuf directly to the mempool.
> > > > This operation is not supported in our hardware. It can be
> > > > implemented in future if any hardware supports it.
> > >
> > > OK, I didn't expect that - just floating the idea. :-)
> > >
> > > >
> > > > >
> > > > > PS: It has been a while since I looked at the DMA library, so
> > > > > ignore my comments if I got this wrong.

RE: [RFC PATCH] dmadev: offload to free source buffer

[Morten Brørup]

> From: Amit Prakash Shukla [mailto:amitprakashs@marvell.com]
> Sent: Thursday, 10 August 2023 12.28
> 
> > From: Morten Brørup <mb@smartsharesystems.com>
> > Sent: Thursday, August 10, 2023 3:03 PM
> >
> > > From: Amit Prakash Shukla [mailto:amitprakashs@marvell.com]
> > > Sent: Wednesday, 9 August 2023 20.12
> > >
> > > > From: Morten Brørup <mb@smartsharesystems.com>
> > > > Sent: Wednesday, August 9, 2023 8:19 PM
> > > >
> > > > > From: Amit Prakash Shukla [mailto:amitprakashs@marvell.com]
> > > > > Sent: Wednesday, 9 August 2023 16.27
> > > > >
> > > > > > From: Morten Brørup <mb@smartsharesystems.com>
> > > > > > Sent: Wednesday, August 9, 2023 2:37 PM
> > > > > >
> > > > > > > From: Amit Prakash Shukla [mailto:amitprakashs@marvell.com]
> > > > > > > Sent: Wednesday, 9 August 2023 08.09
> > > > > > >
> > > > > > > This changeset adds support in DMA library to free source DMA
> > > > > > > buffer by hardware. On a supported hardware, application can
> > > > > > > pass on the mempool information as part of vchan config when
> > > > > > > the DMA transfer direction is configured as
> > RTE_DMA_DIR_MEM_TO_DEV.
> > > > > >
> > > > > > Isn't the DMA source buffer a memory area, and what needs to be
> > > > > > freed
> > > > > is
> > > > > > the mbuf holding the memory area, i.e. two different pointers?
> > > > > No, it is same pointer. Assume mbuf created via mempool, mempool
> > > > > needs to be given via vchan config and iova passed to
> > > > > rte_dma_copy/rte_dma_copy_sg's can be any address in mbuf area of
> > > > > given mempool element.
> > > > > For example, mempool element size is S. dequeued buff from
> > mempool
> > > > > is at X. Any address in (X, X+S) can be given as iova to
> rte_dma_copy.
> > > >
> > > > So the DMA library determines the pointer to the mbuf (in the given
> > > > mempool) by looking at the iova passed to
> > > > rte_dma_copy/rte_dma_copy_sg, and then calls rte_mempool_put with
> > that pointer?
> > >
> > > No. DMA hardware would determine the pointer to the mbuf using iova
> > > address and mempool. Hardware will free the buffer, on completion of
> > data transfer.
> >
> > OK. If there are any requirements to the mempool, it needs to be
> > documented in the source code comments. E.g. does it work with mempools
> > where the mempool driver is an MP_RTS/MC_RTS ring, or a stack?
> 
> I think adding a comment, related to type of supported mempool, in dma
> library code might not be needed as it is driver implementation dependent.
> Call to dev->dev_ops->vchan_setup for the driver shall check and return
> error for unsupported type of mempool.

Makes sense. But I still think that it needs to be mentioned that RTE_DMA_CAPA_MEM_TO_DEV_SOURCE_BUFFER_FREE has some limitations, and doesn't mean that any type of mempool can be used.

I suggest you add a note to the description of the new "struct rte_mempool *mem_to_dev_src_buf_pool" field in the rte_dma_vchan_conf structure, such as:

Note: If the mempool is not supported by the DMA driver, rte_dma_vchan_setup() will fail.

You should also mention it with the description of RTE_DMA_CAPA_MEM_TO_DEV_SOURCE_BUFFER_FREE flag, such as:

Note: Even though the DMA driver has this capability, it may not support all mempool drivers. If the mempool is not supported by the DMA driver, rte_dma_vchan_setup() will fail.


> 
> >
> > >
> > > >
> > > > >
> > > > > >
> > > > > > I like the concept. Something similar might also be useful for
> > > > > > RTE_DMA_DIR_MEM_TO_MEM, e.g. packet capture. Although such a
> > use
> > > > > > case might require decrementing the mbuf refcount instead of
> > > > > > freeing
> > > > > the
> > > > > > mbuf directly to the mempool.
> > > > > This operation is not supported in our hardware. It can be
> > > > > implemented in future if any hardware supports it.
> > > >
> > > > OK, I didn't expect that - just floating the idea. :-)
> > > >
> > > > >
> > > > > >
> > > > > > PS: It has been a while since I looked at the DMA library, so
> > > > > > ignore my comments if I got this wrong.

RE: [RFC PATCH] dmadev: offload to free source buffer

[Amit Prakash Shukla]

> > > >
> > > > No. DMA hardware would determine the pointer to the mbuf using
> > > > iova address and mempool. Hardware will free the buffer, on
> > > > completion of
> > > data transfer.
> > >
> > > OK. If there are any requirements to the mempool, it needs to be
> > > documented in the source code comments. E.g. does it work with
> > > mempools where the mempool driver is an MP_RTS/MC_RTS ring, or a
> stack?
> >
> > I think adding a comment, related to type of supported mempool, in dma
> > library code might not be needed as it is driver implementation dependent.
> > Call to dev->dev_ops->vchan_setup for the driver shall check and
> > return error for unsupported type of mempool.
> 
> Makes sense. But I still think that it needs to be mentioned that
> RTE_DMA_CAPA_MEM_TO_DEV_SOURCE_BUFFER_FREE has some
> limitations, and doesn't mean that any type of mempool can be used.
> 
> I suggest you add a note to the description of the new "struct rte_mempool
> *mem_to_dev_src_buf_pool" field in the rte_dma_vchan_conf structure,
> such as:
> 
> Note: If the mempool is not supported by the DMA driver,
> rte_dma_vchan_setup() will fail.
> 
> You should also mention it with the description of
> RTE_DMA_CAPA_MEM_TO_DEV_SOURCE_BUFFER_FREE flag, such as:
> 
> Note: Even though the DMA driver has this capability, it may not support all
> mempool drivers. If the mempool is not supported by the DMA driver,
> rte_dma_vchan_setup() will fail.

Sure, I will add a note in next version of the patch.

Thanks.

[RFC PATCH v2] dmadev: offload to free source buffer

[Amit Prakash Shukla]

This changeset adds support in DMA library to free source DMA buffer by
hardware. On a supported hardware, application can pass on the mempool
information as part of vchan config when the DMA transfer direction is
configured as RTE_DMA_DIR_MEM_TO_DEV.

Signed-off-by: Amit Prakash Shukla <amitprakashs@marvell.com>
---
v2:
- Added note related to mempool.

 lib/dmadev/rte_dmadev.h | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/lib/dmadev/rte_dmadev.h b/lib/dmadev/rte_dmadev.h
index e61d71959e..f4879d9fd8 100644
--- a/lib/dmadev/rte_dmadev.h
+++ b/lib/dmadev/rte_dmadev.h
@@ -278,6 +278,13 @@ int16_t rte_dma_next_dev(int16_t start_dev_id);
 #define RTE_DMA_CAPA_OPS_COPY_SG	RTE_BIT64(33)
 /** Support fill operation. */
 #define RTE_DMA_CAPA_OPS_FILL		RTE_BIT64(34)
+/** Support for source buffer free for mem to dev transfer.
+ *
+ * @note Even though the DMA driver has this capability, it may not support all
+ * mempool drivers. If the mempool is not supported by the DMA driver,
+ * rte_dma_vchan_setup() will fail.
+ **/
+#define RTE_DMA_CAPA_MEM_TO_DEV_SOURCE_BUFFER_FREE	RTE_BIT64(35)
 /**@}*/
 
 /**
@@ -582,6 +589,19 @@ struct rte_dma_vchan_conf {
 	 * @see struct rte_dma_port_param
 	 */
 	struct rte_dma_port_param dst_port;
+	/** mempool from which source buffer is allocated. mempool info is used
+	 * for freeing source buffer by hardware when configured direction is
+	 * RTE_DMA_DIR_MEM_TO_DEV. To free the source buffer by hardware,
+	 * RTE_DMA_OP_FLAG_FREE_SBUF must be set while calling rte_dma_copy and
+	 * rte_dma_copy_sg().
+	 *
+	 * @note If the mempool is not supported by the DMA driver,
+	 * rte_dma_vchan_setup() will fail.
+	 *
+	 * @see RTE_DMA_OP_FLAG_FREE_SBUF
+	 */
+	struct rte_mempool *mem_to_dev_src_buf_pool;
+
 };
 
 /**
@@ -819,6 +839,13 @@ struct rte_dma_sge {
  * capability bit for this, driver should not return error if this flag was set.
  */
 #define RTE_DMA_OP_FLAG_LLC     RTE_BIT64(2)
+/** Mem to dev source buffer free flag.
+ * Used for freeing source DMA buffer by hardware when the transfer direction is
+ * configured as RTE_DMA_DIR_MEM_TO_DEV.
+ *
+ * @see struct rte_dma_vchan_conf::mem_to_dev_src_buf_pool
+ */
+#define RTE_DMA_OP_FLAG_FREE_SBUF	RTE_BIT64(3)
 /**@}*/
 
 /**
-- 
2.25.1

RE: [RFC PATCH v2] dmadev: offload to free source buffer

[Morten Brørup]

> From: Amit Prakash Shukla [mailto:amitprakashs@marvell.com]
> Sent: Thursday, 10 August 2023 18.53
> 
> This changeset adds support in DMA library to free source DMA buffer by
> hardware. On a supported hardware, application can pass on the mempool
> information as part of vchan config when the DMA transfer direction is
> configured as RTE_DMA_DIR_MEM_TO_DEV.
> 
> Signed-off-by: Amit Prakash Shukla <amitprakashs@marvell.com>
> ---
> v2:
> - Added note related to mempool.
> 

This seems useful. With the v2 additions,

Acked-by: Morten Brørup <mb@smartsharesystems.com>