Re: [dpdk-dev] [PATCH v5 0/5] mbuf: detach mbuf with pinned external buffer

[Slava Ovsiienko <viacheslavo@mellanox.com>]

The unit test (as part of  test_mbuf application) will be provided as separated patch.

With best regards, Slava

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Viacheslav Ovsiienko
> Sent: Monday, January 20, 2020 19:23
> To: dev@dpdk.org
> Cc: Matan Azrad <matan@mellanox.com>; Raslan Darawsheh
> <rasland@mellanox.com>; Ori Kam <orika@mellanox.com>; Shahaf Shuler
> <shahafs@mellanox.com>; olivier.matz@6wind.com;
> stephen@networkplumber.org; thomas@mellanox.net
> Subject: [dpdk-dev] [PATCH v5 0/5] mbuf: detach mbuf with pinned external
> buffer
> 
> Today's pktmbuf pool contains only mbufs with no external buffers.
> This means data buffer for the mbuf should be placed right after the mbuf
> structure (+ the private data when enabled).
> 
> On some cases, the application would want to have the buffers allocated from
> a different device in the platform. This is in order to do zero copy for the
> packet directly to the device memory. Examples for such devices can be GPU
> or storage device. For such cases the native pktmbuf pool does not fit since
> each mbuf would need to point to external buffer.
> 
> To support above, the pktmbuf pool will be populated with mbuf pointing to
> the device buffers using the mbuf external buffer feature.
> The PMD will populate its receive queues with those buffer, so that every
> packet received will be scattered directly to the device memory.
> on the other direction, embedding the buffer pointer to the transmit queues
> of the NIC, will make the DMA to fetch device memory using peer to peer
> communication.
> 
> Such mbuf with external buffer should be handled with care when mbuf is
> freed. Mainly The external buffer should not be detached, so that it can be
> reused for the next packet receive.
> 
> This patch introduce a new flag on the rte_pktmbuf_pool_private structure to
> specify this mempool is for mbuf with pinned external buffer. Upon detach
> this flag is validated and buffer is not detached.
> A new mempool create wrapper is also introduced to help application to
> create and populate such mempool.
> 
> Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> 
> RFC:
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpatches.
> dpdk.org%2Fpatch%2F63077&amp;data=02%7C01%7Cviacheslavo%40mellano
> x.com%7Cefff8bba40804fd99d1808d79dcd7ac7%7Ca652971c7d2e4d9ba6a4d1
> 49256f461b%7C0%7C0%7C637151378168487564&amp;sdata=eukDFAJo1IuVkx
> OPcxh7fVbMlOWtdVA7jN%2FxKtt9wgg%3D&amp;reserved=0
> v1: -
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpatches.
> dpdk.org%2Fcover%2F64424&amp;data=02%7C01%7Cviacheslavo%40mellanox
> .com%7Cefff8bba40804fd99d1808d79dcd7ac7%7Ca652971c7d2e4d9ba6a4d14
> 9256f461b%7C0%7C0%7C637151378168487564&amp;sdata=WXZ22vksi%2FkEa
> KRrc4d6X%2F24Da4PJXBABotz6O8SPTs%3D&amp;reserved=0
> v2: - fix rte_experimantal issue on comment addressing
>     - rte_mbuf_has_pinned_extbuf return type is uint32_t
>     - fix Power9 compilation issue
> v3: - fix "#include <stdbool.h> leftover
> v4: -
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches
> .dpdk.org%2Fcover%2F64809%2F&amp;data=02%7C01%7Cviacheslavo%40mell
> anox.com%7Cefff8bba40804fd99d1808d79dcd7ac7%7Ca652971c7d2e4d9ba6a
> 4d149256f461b%7C0%7C0%7C637151378168487564&amp;sdata=5tOR2I9DUD
> nLYANlB05lf2qFYuCpVARUMYP4PJbz2ac%3D&amp;reserved=0
>     - introduce rte_pktmbuf_priv_flags
>     - support cloning pinned mbufs as for regular mbufs
>       with external buffers
>     - address the minor comments
> v5: - update rte_pktmbuf_prefree_seg
>     - rename __rte_pktmbuf_extbuf_detach
>     - addressing comment
>     - fix typos
> 
> Viacheslav Ovsiienko (5):
>   mbuf: introduce routine to get private mbuf pool flags
>   mbuf: detach mbuf with pinned external buffer
>   mbuf: create packet pool with external memory buffers
>   app/testpmd: add mempool with external data buffers
>   net/mlx5: allow use allocated mbuf with external buffer
> 
>  app/test-pmd/config.c                    |   2 +
>  app/test-pmd/flowgen.c                   |   3 +-
>  app/test-pmd/parameters.c                |   2 +
>  app/test-pmd/testpmd.c                   |  81 +++++++++++++
>  app/test-pmd/testpmd.h                   |   4 +-
>  app/test-pmd/txonly.c                    |   3 +-
>  drivers/net/mlx5/mlx5_rxq.c              |   7 +-
>  drivers/net/mlx5/mlx5_rxtx.c             |   2 +-
>  drivers/net/mlx5/mlx5_rxtx.h             |   2 +-
>  drivers/net/mlx5/mlx5_rxtx_vec.h         |  14 +--
>  drivers/net/mlx5/mlx5_rxtx_vec_altivec.h |   5 +-
>  drivers/net/mlx5/mlx5_rxtx_vec_neon.h    |  29 ++---
>  drivers/net/mlx5/mlx5_rxtx_vec_sse.h     |   2 +-
>  lib/librte_mbuf/rte_mbuf.c               | 198
> ++++++++++++++++++++++++++++++-
>  lib/librte_mbuf/rte_mbuf.h               | 183 ++++++++++++++++++++++++++--
>  lib/librte_mbuf/rte_mbuf_version.map     |   1 +
>  16 files changed, 492 insertions(+), 46 deletions(-)
> 
> --
> 1.8.3.1