Re: [dpdk-dev] [PATCH v5 09/16] dma/idxd: add data-path job submission functions

[Bruce Richardson <bruce.richardson@intel.com>]

On Fri, Sep 17, 2021 at 03:24:30PM +0000, Kevin Laatz wrote:
> Add data path functions for enqueuing and submitting operations to DSA
> devices.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Signed-off-by: Kevin Laatz <kevin.laatz@intel.com>
> Reviewed-by: Conor Walsh <conor.walsh@intel.com>
> ---
>  doc/guides/dmadevs/idxd.rst      |  64 +++++++++++++++
>  drivers/dma/idxd/idxd_common.c   | 136 +++++++++++++++++++++++++++++++
>  drivers/dma/idxd/idxd_internal.h |   5 ++
>  drivers/dma/idxd/meson.build     |   1 +
>  4 files changed, 206 insertions(+)
> 
> diff --git a/doc/guides/dmadevs/idxd.rst b/doc/guides/dmadevs/idxd.rst
> index a603c5dd22..7835461a22 100644
> --- a/doc/guides/dmadevs/idxd.rst
> +++ b/doc/guides/dmadevs/idxd.rst
> @@ -153,3 +153,67 @@ The following code shows how the device is configured in
>  
>  Once configured, the device can then be made ready for use by calling the
>  ``rte_dma_start()`` API.
> +
> +Performing Data Copies
> +~~~~~~~~~~~~~~~~~~~~~~~
> +
> +To perform data copies using IDXD dmadev devices, descriptors should be enqueued
> +using the ``rte_dma_copy()`` API. The HW can be triggered to perform the copy
> +in two ways, either via a ``RTE_DMA_OP_FLAG_SUBMIT`` flag or by calling
> +``rte_dma_submit()``. Once copies have been completed, the completion will
> +be reported back when the application calls ``rte_dma_completed()`` or
> +``rte_dma_completed_status()``. The latter will also report the status of each
> +completed operation.
> +
> +The ``rte_dma_copy()`` function enqueues a single copy to the device ring for
> +copying at a later point. The parameters to that function include the IOVA addresses
> +of both the source and destination buffers, as well as the length of the copy.
> +
> +The ``rte_dma_copy()`` function enqueues a copy operation on the device ring.
> +If the ``RTE_DMA_OP_FLAG_SUBMIT`` flag is set when calling ``rte_dma_copy()``,
> +the device hardware will be informed of the elements. Alternatively, if the flag
> +is not set, the application needs to call the ``rte_dma_submit()`` function to
> +notify the device hardware. Once the device hardware is informed of the elements
> +enqueued on the ring, the device will begin to process them. It is expected
> +that, for efficiency reasons, a burst of operations will be enqueued to the
> +device via multiple enqueue calls between calls to the ``rte_dma_submit()``
> +function.
> +
> +The following code demonstrates how to enqueue a burst of copies to the
> +device and start the hardware processing of them:
> +
> +.. code-block:: C
> +
> +   struct rte_mbuf *srcs[COMP_BURST_SZ], *dsts[COMP_BURST_SZ];
> +   unsigned int i;
> +
> +   for (i = 0; i < RTE_DIM(srcs); i++) {
> +      uint64_t *src_data;
> +
> +      srcs[i] = rte_pktmbuf_alloc(pool);
> +      dsts[i] = rte_pktmbuf_alloc(pool);
> +      src_data = rte_pktmbuf_mtod(srcs[i], uint64_t *);
> +      if (srcs[i] == NULL || dsts[i] == NULL) {
> +         PRINT_ERR("Error allocating buffers\n");
> +         return -1;
> +      }
> +
> +      for (j = 0; j < COPY_LEN/sizeof(uint64_t); j++)
> +         src_data[j] = rte_rand();
> +
> +      if (rte_dma_copy(dev_id, vchan, srcs[i]->buf_iova + srcs[i]->data_off,
> +            dsts[i]->buf_iova + dsts[i]->data_off, COPY_LEN, 0) < 0) {
> +         PRINT_ERR("Error with rte_dma_copy for buffer %u\n", i);
> +         return -1;
> +      }
> +   }
> +   rte_dma_submit(dev_id, vchan);
> +

I think this code block is larger than necessary, because it shows buffer
allocation and initialization rather than just the basics of copy() and
submit() APIs. Furthermore, rather than calling out the generic API use in
the idxd-specific docs, can we just include a reference to the dmadev
documentation?

/Bruce