Re: [dpdk-dev] [RFC PATCH 20.02] mbuf: hint PMD not to inline packet

[Jerin Jacob <jerinjacobk@gmail.com>]

On Thu, Oct 17, 2019 at 12:57 PM Shahaf Shuler <shahafs@mellanox.com> wrote:
>
> Some PMDs inline the mbuf data buffer directly to device. This is in
> order to save the overhead of the PCI headers involved when the device
> DMA read the buffer pointer. For some devices it is essential in order
> to reach the pick BW.
>
> However, there are cases where such inlining is in-efficient. For example
> when the data buffer resides on other device memory (like GPU or storage
> device). attempt to inline such buffer will result in high PCI overhead
> for reading and copying the data from the remote device.

Some questions to understand the use case
# Is this use case where CPU, local DRAM, NW card and GPU memory
connected on the coherent bus
# Assuming the CPU needs to touch the buffer prior to Tx, In that
case, it will be useful?
# How the application knows, The data buffer is in GPU memory in order
to use this flag efficiently?
# Just an random thought, Does it help, if we create two different
mempools one from local DRAM
and one from GPU memory so that the application can work transparently.





>
> To support a mixed traffic pattern (some buffers from local DRAM, some
> buffers from other devices) with high BW, a hint flag is introduced in
> the mbuf.
> Application will hint the PMD whether or not it should try to inline the
> given mbuf data buffer. PMD should do best effort to act upon this
> request.
>
> Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
> ---
>  lib/librte_mbuf/rte_mbuf.h | 9 +++++++++
>  1 file changed, 9 insertions(+)
>
> diff --git a/lib/librte_mbuf/rte_mbuf.h b/lib/librte_mbuf/rte_mbuf.h
> index 98225ec80b..5934532b7f 100644
> --- a/lib/librte_mbuf/rte_mbuf.h
> +++ b/lib/librte_mbuf/rte_mbuf.h
> @@ -203,6 +203,15 @@ extern "C" {
>  /* add new TX flags here */
>
>  /**
> + * Hint to PMD to not inline the mbuf data buffer to device
> + * rather let the device use its DMA engine to fetch the data with the
> + * provided pointer.
> + *
> + * This flag is a only a hint. PMD should enforce it as best effort.
> + */
> +#define PKT_TX_DONT_INLINE_HINT (1ULL << 39)
> +
> +/**
>   * Indicate that the metadata field in the mbuf is in use.
>   */
>  #define PKT_TX_METADATA        (1ULL << 40)
> --
> 2.12.0
>