[dpdk-dev] [PATCH v11 0/6] ethdev: introduce Rx buffer split

[Viacheslav Ovsiienko <viacheslavo@nvidia.com>]

The DPDK datapath in the transmit direction is very flexible.
An application can build the multi-segment packet and manages
almost all data aspects - the memory pools where segments
are allocated from, the segment lengths, the memory attributes
like external buffers, registered for DMA, etc.

In the receiving direction, the datapath is much less flexible,
an application can only specify the memory pool to configure the
receiving queue and nothing more. In order to extend receiving
datapath capabilities it is proposed to add the way to provide
extended information how to split the packets being received.

The new offload flag RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT in device
capabilities is introduced to present the way for PMD to report to
application about supporting Rx packet split to configurable
segments. Prior invoking the rte_eth_rx_queue_setup() routine
application should check RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT flag.

The following structure is introduced to specify the Rx packet
segment for RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT offload:

struct rte_eth_rxseg_split {

    struct rte_mempool *mp; /* memory pools to allocate segment from */
    uint16_t length; /* segment maximal data length,
		       	configures "split point" */
    uint16_t offset; /* data offset from beginning
		       	of mbuf data buffer */
    uint32_t reserved; /* reserved field */
};

The segment descriptions are added to the rte_eth_rxconf structure:
   rx_seg - pointer the array of segment descriptions, each element
             describes the memory pool, maximal data length, initial
             data offset from the beginning of data buffer in mbuf.
	     This array allows to specify the different settings for
	     each segment in individual fashion.
   rx_nseg - number of elements in the array

If the extended segment descriptions is provided with these new
fields the mp parameter of the rte_eth_rx_queue_setup must be
specified as NULL to avoid ambiguity.

There are two options to specify Rx buffer configuration:
- mp is not NULL, rx_conf.rx_seg is NULL, rx_conf.rx_nseg is zero,
  it is compatible configuration, follows existing implementation,
  provides single pool and no description for segment sizes
  and offsets.
- mp is NULL, rx_conf.rx_seg is not NULL, rx_conf.rx_nseg is not
  zero, it provides the extended configuration, individually for
  each segment.

f the Rx queue is configured with new settings the packets being
received will be split into multiple segments pushed to the mbufs
with specified attributes. The PMD will split the received packets
into multiple segments according to the specification in the
description array.

For example, let's suppose we configured the Rx queue with the
following segments:
    seg0 - pool0, len0=14B, off0=2
    seg1 - pool1, len1=20B, off1=128B
    seg2 - pool2, len2=20B, off2=0B
    seg3 - pool3, len3=512B, off3=0B

The packet 46 bytes long will look like the following:
    seg0 - 14B long @ RTE_PKTMBUF_HEADROOM + 2 in mbuf from pool0
    seg1 - 20B long @ 128 in mbuf from pool1
    seg2 - 12B long @ 0 in mbuf from pool2

The packet 1500 bytes long will look like the following:
    seg0 - 14B @ RTE_PKTMBUF_HEADROOM + 2 in mbuf from pool0
    seg1 - 20B @ 128 in mbuf from pool1
    seg2 - 20B @ 0 in mbuf from pool2
    seg3 - 512B @ 0 in mbuf from pool3
    seg4 - 512B @ 0 in mbuf from pool3
    seg5 - 422B @ 0 in mbuf from pool3

The offload RTE_ETH_RX_OFFLOAD_SCATTER must be present and
configured to support new buffer split feature (if rx_nseg
is greater than one).

The split limitations imposed by underlying PMD is reported
in the new introduced rte_eth_dev_info->rx_seg_capa field.

The new approach would allow splitting the ingress packets into
multiple parts pushed to the memory with different attributes.
For example, the packet headers can be pushed to the embedded
data buffers within mbufs and the application data into
the external buffers attached to mbufs allocated from the
different memory pools. The memory attributes for the split
parts may differ either - for example the application data
may be pushed into the external memory located on the dedicated
physical device, say GPU or NVMe. This would improve the DPDK
receiving datapath flexibility with preserving compatibility
with existing API.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---

v1: http://patches.dpdk.org/patch/79594/
v2: http://patches.dpdk.org/patch/79893/
    - add feature support to mlx5 PMD

v3: http://patches.dpdk.org/patch/80389/
    - rte_eth_rx_queue_setup_ex is renamed to rte_eth_rxseg_queue_setup
    - DEV_RX_OFFLOAD_BUFFER_SPLIT is renamed to RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT
    - commit message update
    - documentaion provided
    - release notes update
    - minor bug fixes in testpmd related part

v4: http://patches.dpdk.org/patch/80401/
    - common part of rx_queue_setup/rxseg_queue_setup

v5: http://patches.dpdk.org/patch/80609/
    - refactored to approach of providing split configuration
      in the rte_eth_rxconf structure instead of introducing
      the new API routine
    - added support for rxoffs command to testpmd to
      provide segment offsets for complete testing of split
      configurations
    - patchset is split into two parts - PMD part will
      be presented as separate series

v6: http://patches.dpdk.org/patch/80785/
    - wordy comments rephrased
    - typos fixed
    - rte_eth_rx_queue_setup configuration check isolated
      for two main options
    - the rest of comments addressed

v7: http://patches.dpdk.org/patch/80978/
    - description as union of feature dedicated segment split description
    - the split limitations reported in the rte_eth_dev_info->rx_seg_capa
    - comments made less wordy

v8: - bitfield instead of uint8;
    - named union to avoid warinings;
    - typos fixed


v9: http://patches.dpdk.org/patch/81080/
    - simplified rte_eth_rxseg union
    - moved out wordy comment from rte_eth_rxconf
    - rte_eth_rxseg_capa uint32 bitfield and naming
    - added experimental tags

v10: - missed in mailing list, see v11 for changes

v11: - documentation cleanup (removed rxq_info_get mention)
     - max_nseg moved from the recored and converted to uint16_t;
     - reserved uint16_t is added to rte_eth_rxseg_capa;

---

Viacheslav Ovsiienko (6):
  ethdev: introduce Rx buffer split
  app/testpmd: add multiple pools per core creation
  app/testpmd: add buffer split offload configuration
  app/testpmd: add rxpkts commands and parameters
  app/testpmd: add rxoffs commands and parameters
  app/testpmd: add extended Rx queue setup

 app/test-pmd/bpf_cmd.c                      |   4 +-
 app/test-pmd/cmdline.c                      | 151 +++++++++++++++++++++++----
 app/test-pmd/config.c                       | 107 ++++++++++++++++++-
 app/test-pmd/parameters.c                   |  54 ++++++++--
 app/test-pmd/testpmd.c                      | 121 ++++++++++++++++------
 app/test-pmd/testpmd.h                      |  44 ++++++--
 doc/guides/nics/features.rst                |  14 +++
 doc/guides/rel_notes/deprecation.rst        |   5 -
 doc/guides/rel_notes/release_20_11.rst      |   9 ++
 doc/guides/testpmd_app_ug/run_app.rst       |  22 +++-
 doc/guides/testpmd_app_ug/testpmd_funcs.rst |  36 ++++++-
 lib/librte_ethdev/rte_ethdev.c              | 153 +++++++++++++++++++++++-----
 lib/librte_ethdev/rte_ethdev.h              |  90 +++++++++++++++-
 13 files changed, 707 insertions(+), 103 deletions(-)

-- 
1.8.3.1