Re: [dpdk-dev] [PATCH v4 06/11] dma/ioat: add data path job submission functions

[Bruce Richardson <bruce.richardson@intel.com>]

On Fri, Sep 17, 2021 at 03:42:22PM +0000, Conor Walsh wrote:
> Add data path functions for enqueuing and submitting operations to
> IOAT devices.
> 
> Signed-off-by: Conor Walsh <conor.walsh@intel.com>
> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com>
> ---
>  doc/guides/dmadevs/ioat.rst    | 54 ++++++++++++++++++++
>  drivers/dma/ioat/ioat_dmadev.c | 92 ++++++++++++++++++++++++++++++++++
>  2 files changed, 146 insertions(+)
> 
> diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst
> index a64d67bf89..2464207e20 100644
> --- a/doc/guides/dmadevs/ioat.rst
> +++ b/doc/guides/dmadevs/ioat.rst
> @@ -89,3 +89,57 @@ The following code shows how the device is configured in ``test_dmadev.c``:
>  
>  Once configured, the device can then be made ready for use by calling the
>  ``rte_dma_start()`` API.
> +
> +Performing Data Copies
> +~~~~~~~~~~~~~~~~~~~~~~~
> +
> +To perform data copies using IOAT dmadev devices, the functions
> +``rte_dma_copy()`` and ``rte_dma_submit()`` should be used. Alternatively
> +``rte_dma_copy()`` can be called with the ``RTE_DMA_OP_FLAG_SUBMIT`` flag
> +set.
> +
> +The ``rte_dma_copy()`` function enqueues a single copy to the
> +device ring for copying at a later point. The parameters to the function
> +include the device ID of the desired device, the virtual DMA channel required
> +(always 0 for IOAT), the IOVA addresses of both the source and destination
> +buffers, the length of the data to be copied and any operation flags. The
> +function will return the index of the enqueued job which can be use to
> +track that operation.
> +
> +While the ``rte_dma_copy()`` function enqueues a copy operation on the device
> +ring, the copy will not actually be performed until after the application calls
> +the ``rte_dma_submit()`` function. This function informs the device hardware
> +of the elements enqueued on the ring, and the device will begin to process them.
> +It is expected that, for efficiency reasons, a burst of operations will be
> +enqueued to the device via multiple enqueue calls between calls to the
> +``rte_dma_submit()`` function. If desired you can pass the
> +``RTE_DMA_OP_FLAG_SUBMIT`` flag when calling ``rte_dma_copy()`` and this will
> +tell the device to perform the enqueued operation and any unperformed operations
> +before it. The ``RTE_DMA_OP_FLAG_SUBMIT`` flag can be passed instead of calling
> +the ``rte_dma_submit()`` function for example on the last enqueue of the burst.
> +
> +The following code from demonstrates how to enqueue a burst of copies to the
> +device and start the hardware processing of them:
> +
> +.. code-block:: C
> +
> +   for (i = 0; i < BURST_SIZE; i++) {
> +      if (rte_dma_copy(dev_id, vchan, rte_mbuf_data_iova(srcs[i]),
> +            rte_mbuf_data_iova(dsts[i]), COPY_LEN, 0) < 0) {
> +         PRINT_ERR("Error with rte_dma_copy for buffer %u\n", i);
> +         return -1;
> +      }
> +   }
> +   if (rte_dma_submit(dev_id, vchan) < 0) {
> +      PRINT_ERR("Error with performing operations\n", i);
> +      return -1;
> +   }
> +
> +Filling an Area of Memory
> +~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The driver also has support for the ``fill`` operation, where an area
> +of memory is overwritten, or filled, with a short pattern of data.
> +Fill operations can be performed in much the same was as copy operations
> +described above, just using the ``rte_dma_fill()`` function rather
> +than the ``rte_dma_copy()`` function.

Similar to the feedback on the idxd driver, I think we need to see how much
of this text is already present in the generic dmadev documentation and
re-use or reference that. If it's not present, then these patches should
add it to the common doc, not a separate driver-specific doc.

/Bruce