Re: [PATCH v2 0/8] ethdev: introduce hairpin memory capabilities

[Thomas Monjalon <thomas@monjalon.net>]

06/10/2022 13:00, Dariusz Sosnowski:
> The hairpin queues are used to transmit packets received on the wire, back to the wire.
> How hairpin queues are implemented and configured is decided internally by the PMD and
> applications have no control over the configuration of Rx and Tx hairpin queues.
> This patchset addresses that by:
> 
> - Extending hairpin queue capabilities reported by PMDs.
> - Exposing new configuration options for Rx and Tx hairpin queues.
> 
> Main goal of this patchset is to allow applications to provide configuration hints
> regarding memory placement of hairpin queues.
> These hints specify whether buffers of hairpin queues should be placed in host memory
> or in dedicated device memory.
> 
> For example, in context of NVIDIA Connect-X and BlueField devices,
> this distinction is important for several reasons:
> 
> - By default, data buffers and packet descriptors are placed in device memory region
>   which is shared with other resources (e.g. flow rules).
>   This results in memory contention on the device,
>   which may lead to degraded performance under heavy load.
> - Placing hairpin queues in dedicated device memory can decrease latency of hairpinned traffic,
>   since hairpin queue processing will not be memory starved by other operations.
>   Side effect of this memory configuration is that it leaves less memory for other resources,
>   possibly causing memory contention in non-hairpin traffic.
> - Placing hairpin queues in host memory can increase throughput of hairpinned
>   traffic at the cost of increasing latency.
>   Each packet processed by hairpin queues will incur additional PCI transactions (increase in latency),
>   but memory contention on the device is avoided.
> 
> Depending on the workload and whether throughput or latency has a higher priority for developers,
> it would be beneficial if developers could choose the best hairpin configuration for their use case.
> 
> To address that, this patchset adds the following configuration options (in rte_eth_hairpin_conf struct):
> 
> - use_locked_device_memory - If set, PMD will allocate specialized on-device memory for the queue.
> - use_rte_memory - If set, PMD will use DPDK-managed memory for the queue.
> - force_memory - If set, PMD will be forced to use provided memory configuration.
>   If no appropriate resources are available, the queue allocation will fail.
>   If unset and no appropriate resources are available, PMD will fallback to its default behavior.
> 
> Implementing support for these flags is optional and applications should be allowed to not set any of these new flags.
> This will result in default memory configuration provided by the PMD.
> Application developers should consult the PMD documentation in that case.
> 
> These changes were originally proposed in http://patches.dpdk.org/project/dpdk/patch/20220811120530.191683-1-dsosnowski@nvidia.com/.
> 
> Dariusz Sosnowski (8):
>   ethdev: introduce hairpin memory capabilities
>   common/mlx5: add hairpin SQ buffer type capabilities
>   common/mlx5: add hairpin RQ buffer type capabilities
>   net/mlx5: allow hairpin Tx queue in RTE memory
>   net/mlx5: allow hairpin Rx queue in locked memory
>   doc: add notes for hairpin to mlx5 documentation
>   app/testpmd: add hairpin queues memory modes
>   app/flow-perf: add hairpin queue memory config

Doc squashed in mlx5 commits.
Applied, thanks.