* [PATCH 0/5] refactore mlx5 guides
@ 2022-02-22 12:48 Michael Baum
2022-02-22 12:48 ` [PATCH 1/5] doc: remove obsolete explanations from mlx5 guide Michael Baum
` (6 more replies)
0 siblings, 7 replies; 17+ messages in thread
From: Michael Baum @ 2022-02-22 12:48 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Raslan Darawsheh, Viacheslav Ovsiienko
Recently [1] all the drivers running over the mlx5 device started
sharing some objects.
This raised the need to create documentation for the common driver that
would contain information relevant to all drivers and in particular:
- Tutorial for configuration of the device.
- Instruction for using Mellanox ofed.
- devargs of the common driver.
This series adds such documentation and reference to it as parts of the
documentation of the other drivers.
[1]
https://patchwork.dpdk.org/project/dpdk/cover/20211019205602.3188203-1-michaelba@nvidia.com/
Michael Baum (5):
doc: remove obsolete explanations from mlx5 guide
doc: replace broken links in mlx5 guide
doc: correct name of BlueField-2 in mlx5 guide
doc: add shared guide for mlx5 PMDs
doc: update doorbell mapping parameter name in mlx5 guide
doc/guides/compressdevs/mlx5.rst | 40 +-
doc/guides/cryptodevs/mlx5.rst | 38 +-
doc/guides/nics/mlx4.rst | 4 +-
doc/guides/nics/mlx5.rst | 567 +-----------------
doc/guides/platform/index.rst | 1 +
doc/guides/platform/mlx5.rst | 641 +++++++++++++++++++++
doc/guides/regexdevs/mlx5.rst | 44 +-
doc/guides/vdpadevs/mlx5.rst | 87 +--
drivers/common/mlx5/linux/mlx5_common_os.c | 2 +-
drivers/common/mlx5/mlx5_common.c | 31 +-
drivers/common/mlx5/mlx5_common_defs.h | 8 +-
drivers/net/mlx5/linux/mlx5_verbs.c | 2 +-
drivers/net/mlx5/mlx5_devx.c | 2 +-
13 files changed, 747 insertions(+), 720 deletions(-)
create mode 100644 doc/guides/platform/mlx5.rst
--
2.25.1
^ permalink raw reply [flat|nested] 17+ messages in thread* [PATCH 1/5] doc: remove obsolete explanations from mlx5 guide 2022-02-22 12:48 [PATCH 0/5] refactore mlx5 guides Michael Baum @ 2022-02-22 12:48 ` Michael Baum 2022-02-22 12:48 ` [PATCH 2/5] doc: replace broken links in " Michael Baum ` (5 subsequent siblings) 6 siblings, 0 replies; 17+ messages in thread From: Michael Baum @ 2022-02-22 12:48 UTC (permalink / raw) To: dev; +Cc: Matan Azrad, Raslan Darawsheh, Viacheslav Ovsiienko, stable Vectorized routines were removed in result of Tx datapath refactoring, and devarg keys documentation was updated. However, more updating should have been done. In environment variables doc, there was explanation according to vectorized Tx which isn't relevant anymore. This patch removes this irrelevant explanation. Fixes: a6bd4911ad93 ("net/mlx5: remove Tx implementation") Cc: stable@dpdk.org Signed-off-by: Michael Baum <michaelba@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> --- doc/guides/nics/mlx5.rst | 9 --------- 1 file changed, 9 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index c3cc0c0f41..3f5c8bb303 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -557,15 +557,6 @@ Environment variables The register would be flushed to HW usually when the write-combining buffer becomes full, but it depends on CPU design. - Except for vectorized Tx burst routines, a write memory barrier is enforced - after updating the register so that the update can be immediately visible to - HW. - - When vectorized Tx burst is called, the barrier is set only if the burst size - is not aligned to MLX5_VPMD_TX_MAX_BURST. However, setting this environmental - variable will bring better latency even though the maximum throughput can - slightly decline. - Run-time configuration ~~~~~~~~~~~~~~~~~~~~~~ -- 2.25.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 2/5] doc: replace broken links in mlx5 guide 2022-02-22 12:48 [PATCH 0/5] refactore mlx5 guides Michael Baum 2022-02-22 12:48 ` [PATCH 1/5] doc: remove obsolete explanations from mlx5 guide Michael Baum @ 2022-02-22 12:48 ` Michael Baum 2022-02-22 12:48 ` [PATCH 3/5] doc: correct name of BlueField-2 " Michael Baum ` (4 subsequent siblings) 6 siblings, 0 replies; 17+ messages in thread From: Michael Baum @ 2022-02-22 12:48 UTC (permalink / raw) To: dev; +Cc: Matan Azrad, Raslan Darawsheh, Viacheslav Ovsiienko Update links in both mlx4 and mlx5 doc. Signed-off-by: Michael Baum <michaelba@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> --- doc/guides/nics/mlx4.rst | 4 ++-- doc/guides/nics/mlx5.rst | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/guides/nics/mlx4.rst b/doc/guides/nics/mlx4.rst index c81105730d..66493a1157 100644 --- a/doc/guides/nics/mlx4.rst +++ b/doc/guides/nics/mlx4.rst @@ -14,7 +14,7 @@ the `Mellanox website <http://www.mellanox.com>`_. Help is also provided by the `Mellanox community <http://community.mellanox.com/welcome>`_. There is also a `section dedicated to this poll mode driver -<http://www.mellanox.com/page/products_dyn?product_family=209&mtag=pmd_for_dpdk>`_. +<https://developer.nvidia.com/networking/dpdk>`_. Implementation details @@ -219,7 +219,7 @@ Mellanox OFED as a fallback - `Mellanox OFED`_ version: **4.4, 4.5, 4.6**. - firmware version: **2.42.5000** and above. -.. _`Mellanox OFED`: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers +.. _`Mellanox OFED`: https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ .. note:: diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 3f5c8bb303..c9defd0c60 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -19,7 +19,7 @@ Information and documentation about these adapters can be found on the `Mellanox community <http://community.mellanox.com/welcome>`__. There is also a `section dedicated to this poll mode driver -<http://www.mellanox.com/page/products_dyn?product_family=209&mtag=pmd_for_dpdk>`__. +<https://developer.nvidia.com/networking/dpdk>`_. Design @@ -1376,9 +1376,9 @@ managers on most distributions, this PMD requires Ethernet extensions that may not be supported at the moment (this is a work in progress). `Mellanox OFED -<http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux>`__ and +<https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/>`__ and `Mellanox EN -<http://www.mellanox.com/page/products_dyn?product_family=27&mtag=linux>`__ +<https://network.nvidia.com/products/ethernet-drivers/linux/mlnx_en/>`__ include the necessary support and should be used in the meantime. For DPDK, only libibverbs, libmlx5, mlnx-ofed-kernel packages and firmware updates are required from that distribution. -- 2.25.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 3/5] doc: correct name of BlueField-2 in mlx5 guide 2022-02-22 12:48 [PATCH 0/5] refactore mlx5 guides Michael Baum 2022-02-22 12:48 ` [PATCH 1/5] doc: remove obsolete explanations from mlx5 guide Michael Baum 2022-02-22 12:48 ` [PATCH 2/5] doc: replace broken links in " Michael Baum @ 2022-02-22 12:48 ` Michael Baum 2022-02-22 12:48 ` [PATCH 4/5] doc: add shared guide for mlx5 PMDs Michael Baum ` (3 subsequent siblings) 6 siblings, 0 replies; 17+ messages in thread From: Michael Baum @ 2022-02-22 12:48 UTC (permalink / raw) To: dev; +Cc: Matan Azrad, Raslan Darawsheh, Viacheslav Ovsiienko Update "BlueField 2" -> "BlueField-2" in mlx5 docs. Signed-off-by: Michael Baum <michaelba@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> --- doc/guides/compressdevs/mlx5.rst | 6 +++--- doc/guides/cryptodevs/mlx5.rst | 6 +++--- doc/guides/nics/mlx5.rst | 2 +- doc/guides/regexdevs/mlx5.rst | 8 ++++---- 4 files changed, 11 insertions(+), 11 deletions(-) diff --git a/doc/guides/compressdevs/mlx5.rst b/doc/guides/compressdevs/mlx5.rst index a4e17f65b3..7f2d6bdfff 100644 --- a/doc/guides/compressdevs/mlx5.rst +++ b/doc/guides/compressdevs/mlx5.rst @@ -7,7 +7,7 @@ MLX5 compress driver ==================== The MLX5 compress driver library -(**librte_compress_mlx5**) provides support for **Mellanox BlueField 2** +(**librte_compress_mlx5**) provides support for **Mellanox BlueField-2** families of 25/50/100/200 Gb/s adapters. Design @@ -21,7 +21,7 @@ These engines are part of the ARM complex of the BlueField chip, and as such they do not use NIC related resources (e.g. RX/TX bandwidth). They do share the same PCI and memory bandwidth. -So, using the BlueField device (starting from BlueField 2), the compress +So, using the BlueField device (starting from BlueField-2), the compress class operations can be supported in parallel to the net, vDPA and RegEx class operations. @@ -95,7 +95,7 @@ Driver options Supported NICs -------------- -* Mellanox\ |reg| BlueField 2 SmartNIC +* Mellanox\ |reg| BlueField-2 SmartNIC Prerequisites ------------- diff --git a/doc/guides/cryptodevs/mlx5.rst b/doc/guides/cryptodevs/mlx5.rst index e86a6205e8..9936556cc9 100644 --- a/doc/guides/cryptodevs/mlx5.rst +++ b/doc/guides/cryptodevs/mlx5.rst @@ -88,7 +88,7 @@ The mlxreg dedicated tool should be used as follows: should not be specified. All the device ports should set it in order to move to operational mode. - For BlueField 2, the internal ports in the ARM system should also be set. + For BlueField-2, the internal ports in the ARM system should also be set. - Query CRYPTO_OPERATIONAL register to make sure the device is in Operational mode. @@ -142,7 +142,7 @@ Supported NICs -------------- * Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G) -* Mellanox\ |reg| BlueField 2 SmartNIC +* Mellanox\ |reg| BlueField-2 SmartNIC * Mellanox\ |reg| ConnectX\ |reg|-6 Dx @@ -162,7 +162,7 @@ FW Prerequisites ~~~~~~~~~~~~~~~~ - xx.31.0328 for ConnectX-6. -- xx.32.0108 for ConnectX-6 Dx and BlueField 2. +- xx.32.0108 for ConnectX-6 Dx and BlueField-2. Linux Prerequisites ~~~~~~~~~~~~~~~~~~~ diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index c9defd0c60..b7ad55561c 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -253,7 +253,7 @@ Limitations - Flex item: - - Hardware support: BlueField 2. + - Hardware support: BlueField-2. - Flex item is supported on PF only. - Hardware limits ``header_length_mask_width`` up to 6 bits. - Firmware supports 8 global sample fields. diff --git a/doc/guides/regexdevs/mlx5.rst b/doc/guides/regexdevs/mlx5.rst index b2bf0afd01..fc2cacba60 100644 --- a/doc/guides/regexdevs/mlx5.rst +++ b/doc/guides/regexdevs/mlx5.rst @@ -7,7 +7,7 @@ MLX5 RegEx driver ================= The MLX5 RegEx (Regular Expression) driver library -(**librte_regex_mlx5**) provides support for **Mellanox BlueField 2** +(**librte_regex_mlx5**) provides support for **Mellanox BlueField-2** families of 25/50/100/200 Gb/s adapters. Design @@ -43,13 +43,13 @@ Features Supported NICs -------------- -* Mellanox\ |reg| BlueField 2 SmartNIC +* Mellanox\ |reg| BlueField-2 SmartNIC Prerequisites ------------- -- BlueField 2 running Mellanox supported kernel. -- Enable the RegEx capabilities using system call from the BlueField 2. +- BlueField-2 running Mellanox supported kernel. +- Enable the RegEx capabilities using system call from the BlueField-2. - Official support is not yet released. Limitations -- 2.25.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 4/5] doc: add shared guide for mlx5 PMDs 2022-02-22 12:48 [PATCH 0/5] refactore mlx5 guides Michael Baum ` (2 preceding siblings ...) 2022-02-22 12:48 ` [PATCH 3/5] doc: correct name of BlueField-2 " Michael Baum @ 2022-02-22 12:48 ` Michael Baum 2022-02-22 12:48 ` [PATCH 5/5] doc: update doorbell mapping parameter name in mlx5 guide Michael Baum ` (2 subsequent siblings) 6 siblings, 0 replies; 17+ messages in thread From: Michael Baum @ 2022-02-22 12:48 UTC (permalink / raw) To: dev; +Cc: Matan Azrad, Raslan Darawsheh, Viacheslav Ovsiienko Adds new documentation for MLX5 common driver that contains: - Its features list (doesn't exist for now). - Its devargs description. - Device configuration information and tutorial. - Quick Start Guide for Mellanox OFED/EN. Move into this doc all shared information from other MLX5 PMD docs and add them reference to new common doc. Signed-off-by: Michael Baum <michaelba@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> --- doc/guides/compressdevs/mlx5.rst | 34 +- doc/guides/cryptodevs/mlx5.rst | 32 +- doc/guides/nics/mlx5.rst | 522 ++------------------------- doc/guides/platform/index.rst | 1 + doc/guides/platform/mlx5.rst | 602 +++++++++++++++++++++++++++++++ doc/guides/regexdevs/mlx5.rst | 36 +- doc/guides/vdpadevs/mlx5.rst | 87 +---- 7 files changed, 662 insertions(+), 652 deletions(-) create mode 100644 doc/guides/platform/mlx5.rst diff --git a/doc/guides/compressdevs/mlx5.rst b/doc/guides/compressdevs/mlx5.rst index 7f2d6bdfff..273251e088 100644 --- a/doc/guides/compressdevs/mlx5.rst +++ b/doc/guides/compressdevs/mlx5.rst @@ -3,10 +3,10 @@ .. include:: <isonum.txt> -MLX5 compress driver +MLX5 Compress Driver ==================== -The MLX5 compress driver library +The mlx5 compress driver library (**librte_compress_mlx5**) provides support for **Mellanox BlueField-2** families of 25/50/100/200 Gb/s adapters. @@ -25,30 +25,7 @@ So, using the BlueField device (starting from BlueField-2), the compress class operations can be supported in parallel to the net, vDPA and RegEx class operations. -For security reasons and robustness, this driver only deals with virtual -memory addresses. The way resources allocations are handled by the kernel, -combined with hardware specifications that allow to handle virtual memory -addresses directly, ensure that DPDK applications cannot access random -physical memory (or memory that does not belong to the current process). - -The PMD uses libibverbs and libmlx5 to access the device firmware -or directly the hardware components. -There are different levels of objects and bypassing abilities -to get the best performances: - -- Verbs is a complete high-level generic API. -- Direct Verbs is a device-specific API. -- DevX allows to access firmware objects. - -Enabling librte_compress_mlx5 causes DPDK applications to be linked against -libibverbs. - -Mellanox mlx5 PCI device can be probed by number of different PCI devices, -for example net / vDPA / RegEx. To select the compress PMD ``class=compress`` -should be specified as device parameter. The compress device can be probed and -used with other Mellanox classes, by adding more options in the class. -For example: ``class=net:compress`` will probe both the net PMD and the compress -PMD. +See :doc:`../../platform/mlx5` guide for more design details. Features -------- @@ -85,6 +62,9 @@ Limitations Driver options -------------- +Please refer to :ref:`mlx5 common options <mlx5_common_driver_options>` +for an additional list of options shared with other mlx5 drivers. + - ``log-block-size`` parameter [int] Log of the Huffman block size in the Deflate algorithm. @@ -101,4 +81,4 @@ Prerequisites ------------- - Mellanox OFED version: **5.2** - see :doc:`../../nics/mlx5` guide for more Mellanox OFED details. + See :ref:`mlx5 common prerequisites <mlx5_linux_prerequisites>` for more details. diff --git a/doc/guides/cryptodevs/mlx5.rst b/doc/guides/cryptodevs/mlx5.rst index 9936556cc9..ef47aa65dd 100644 --- a/doc/guides/cryptodevs/mlx5.rst +++ b/doc/guides/cryptodevs/mlx5.rst @@ -28,23 +28,12 @@ when the MKEY is configured to perform crypto operations. The encryption does not require text to be aligned to the AES block size (128b). -For security reasons and to increase robustness, this driver only deals with virtual -memory addresses. The way resources allocations are handled by the kernel, -combined with hardware specifications that allow handling virtual memory -addresses directly, ensure that DPDK applications cannot access random -physical memory (or memory that does not belong to the current process). +See :doc:`../../platform/mlx5` guide for more design details. -The PMD uses ``libibverbs`` and ``libmlx5`` to access the device firmware -or to access the hardware components directly. -There are different levels of objects and bypassing abilities. -To get the best performances: - -- Verbs is a complete high-level generic API (Linux only). -- Direct Verbs is a device-specific API (Linux only). -- DevX allows to access firmware objects. +Configuration +------------- -Enabling ``librte_crypto_mlx5`` causes DPDK applications -to be linked against libibverbs on Linux OS. +See the :ref:`mlx5 common configuration <mlx5_common_env>`. In order to move the device to crypto operational mode, credential and KEK (Key Encrypting Key) should be set as the first step. @@ -109,10 +98,8 @@ The mlxreg dedicated tool should be used as follows: Driver options -------------- -- ``class`` parameter [string] - - Select the class of the driver that should probe the device. - `crypto` for the mlx5 crypto driver. +Please refer to :ref:`mlx5 common options <mlx5_common_driver_options>` +for an additional list of options shared with other mlx5 drivers. - ``wcs_file`` parameter [string] - mandatory @@ -168,13 +155,12 @@ Linux Prerequisites ~~~~~~~~~~~~~~~~~~~ - Mellanox OFED version: **5.3**. - see :doc:`../../nics/mlx5` guide for more Mellanox OFED details. - - Compilation can be done also with rdma-core v15+. - see :doc:`../../nics/mlx5` guide for more rdma-core details. + + See :ref:`mlx5 common prerequisites <mlx5_linux_prerequisites>` for more details. Windows Prerequisites ~~~~~~~~~~~~~~~~~~~~~ - Mellanox WINOF-2 version: **2.60** or higher. - see :doc:`../../nics/mlx5` guide for more Mellanox WINOF-2 details. + See :ref:`mlx5 common prerequisites <mlx5_windows_prerequisites>` for more details. diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index b7ad55561c..8eb67efcf3 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -4,23 +4,16 @@ .. include:: <isonum.txt> -MLX5 poll mode driver -===================== +MLX5 Ethernet Poll Mode Driver +============================== -The MLX5 poll mode driver library (**librte_net_mlx5**) provides support +The mlx5 Ethernet poll mode driver library (**librte_net_mlx5**) provides support for **Mellanox ConnectX-4**, **Mellanox ConnectX-4 Lx** , **Mellanox ConnectX-5**, **Mellanox ConnectX-6**, **Mellanox ConnectX-6 Dx**, **Mellanox ConnectX-6 Lx**, **Mellanox BlueField** and **Mellanox BlueField-2** families of 10/25/40/50/100/200 Gb/s adapters as well as their virtual functions (VF) in SR-IOV context. -Information and documentation about these adapters can be found on the -`Mellanox website <http://www.mellanox.com>`__. Help is also provided by the -`Mellanox community <http://community.mellanox.com/welcome>`__. - -There is also a `section dedicated to this poll mode driver -<https://developer.nvidia.com/networking/dpdk>`_. - Design ------ @@ -29,12 +22,6 @@ Besides its dependency on libibverbs (that implies libmlx5 and associated kernel support), librte_net_mlx5 relies heavily on system calls for control operations such as querying/updating the MTU and flow control parameters. -For security reasons and robustness, this driver only deals with virtual -memory addresses. The way resources allocations are handled by the kernel, -combined with hardware specifications that allow to handle virtual memory -addresses directly, ensure that DPDK applications cannot access random -physical memory (or memory that does not belong to the current process). - This capability allows the PMD to coexist with kernel network interfaces which remain functional, although they stop receiving unicast packets as long as they share the same MAC address. @@ -42,18 +29,7 @@ This means legacy linux control tools (for example: ethtool, ifconfig and more) can operate on the same network interfaces that owned by the DPDK application. -The PMD can use libibverbs and libmlx5 to access the device firmware -or directly the hardware components. -There are different levels of objects and bypassing abilities -to get the best performances: - -- Verbs is a complete high-level generic API -- Direct Verbs is a device-specific API -- DevX allows to access firmware objects -- Direct Rules manages flow steering at low-level hardware layer - -Enabling librte_net_mlx5 causes DPDK applications to be linked against -libibverbs. +See :doc:`../../platform/mlx5` guide for more design details. Features -------- @@ -522,75 +498,31 @@ Extended statistics can be queried using ``rte_eth_xstats_get()``. The extended Finally per-flow statistics can by queried using ``rte_flow_query`` when attaching a count action for specific flow. The flow counter counts the number of packets received successfully by the port and match the specific flow. -Configuration -------------- - -Compilation options -~~~~~~~~~~~~~~~~~~~ - -The ibverbs libraries can be linked with this PMD in a number of ways, -configured by the ``ibverbs_link`` build option: - -- ``shared`` (default): the PMD depends on some .so files. - -- ``dlopen``: Split the dependencies glue in a separate library - loaded when needed by dlopen. - It make dependencies on libibverbs and libmlx4 optional, - and has no performance impact. - -- ``static``: Embed static flavor of the dependencies libibverbs and libmlx4 - in the PMD shared library or the executable static binary. -Environment variables -~~~~~~~~~~~~~~~~~~~~~ +Compilation +----------- -- ``MLX5_GLUE_PATH`` +See :ref:`mlx5 common compilation <mlx5_common_compilation>`. - A list of directories in which to search for the rdma-core "glue" plug-in, - separated by colons or semi-colons. -- ``MLX5_SHUT_UP_BF`` +Configuration +------------- - Configures HW Tx doorbell register as IO-mapped. +Environment Configuration +~~~~~~~~~~~~~~~~~~~~~~~~~ - By default, the HW Tx doorbell is configured as a write-combining register. - The register would be flushed to HW usually when the write-combining buffer - becomes full, but it depends on CPU design. +See :ref:`mlx5 common configuration <mlx5_common_env>`. -Run-time configuration +Firmware configuration ~~~~~~~~~~~~~~~~~~~~~~ -- librte_net_mlx5 brings kernel network interfaces up during initialization - because it is affected by their state. Forcing them down prevents packets - reception. - -- **ethtool** operations on related kernel interfaces also affect the PMD. - -Run as non-root -^^^^^^^^^^^^^^^ - -In order to run as a non-root user, -some capabilities must be granted to the application:: - - setcap cap_sys_admin,cap_net_admin,cap_net_raw,cap_ipc_lock+ep <dpdk-app> - -Below are the reasons of the need for each capability: - -``cap_sys_admin`` - When using physical addresses (PA mode), with Linux >= 4.0, - for access to ``/proc/self/pagemap``. - -``cap_net_admin`` - For device configuration. - -``cap_net_raw`` - For raw ethernet queue allocation through kernel driver. - -``cap_ipc_lock`` - For DMA memory pinning. +See :ref:`mlx5_firmware_config` guide. Driver options -^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~ + +Please refer to :ref:`mlx5 common options <mlx5_common_driver_options>` +for an additional list of options shared with other mlx5 drivers. - ``rxq_cqe_comp_en`` parameter [int] @@ -1054,30 +986,6 @@ Driver options Disabled by default (set to 0). -- ``mr_ext_memseg_en`` parameter [int] - - A nonzero value enables extending memseg when registering DMA memory. If - enabled, the number of entries in MR (Memory Region) lookup table on datapath - is minimized and it benefits performance. On the other hand, it worsens memory - utilization because registered memory is pinned by kernel driver. Even if a - page in the extended chunk is freed, that doesn't become reusable until the - entire memory is freed. - - Enabled by default. - -- ``mr_mempool_reg_en`` parameter [int] - - A nonzero value enables implicit registration of DMA memory of all mempools - except those having ``RTE_MEMPOOL_F_NON_IO``. This flag is set automatically - for mempools populated with non-contiguous objects or those without IOVA. - The effect is that when a packet from a mempool is transmitted, - its memory is already registered for DMA in the PMD and no registration - will happen on the data path. The tradeoff is extra work on the creation - of each mempool and increased HW resource use if some mempools - are not used with MLX5 devices. - - Enabled by default. - - ``representor`` parameter [list] This parameter can be used to instantiate DPDK Ethernet devices from @@ -1148,13 +1056,6 @@ Driver options By default, the PMD will set this value to 0. -- ``sys_mem_en`` parameter [int] - - A non-zero value enables the PMD memory management allocating memory - from system by default, without explicit rte memory flag. - - By default, the PMD will set this value to 0. - - ``decap_en`` parameter [int] Some devices do not support FCS (frame checksum) scattering for @@ -1178,253 +1079,6 @@ Driver options By default, the PMD will set this value to 1. -.. _mlx5_firmware_config: - -Firmware configuration -~~~~~~~~~~~~~~~~~~~~~~ - -Firmware features can be configured as key/value pairs. - -The command to set a value is:: - - mlxconfig -d <device> set <key>=<value> - -The command to query a value is:: - - mlxconfig -d <device> query | grep <key> - -The device name for the command ``mlxconfig`` can be either the PCI address, -or the mst device name found with:: - - mst status - -Below are some firmware configurations listed. - -- link type:: - - LINK_TYPE_P1 - LINK_TYPE_P2 - value: 1=Infiniband 2=Ethernet 3=VPI(auto-sense) - -- enable SR-IOV:: - - SRIOV_EN=1 - -- maximum number of SR-IOV virtual functions:: - - NUM_OF_VFS=<max> - -- enable DevX (required by Direct Rules and other features):: - - UCTX_EN=1 - -- aggressive CQE zipping:: - - CQE_COMPRESSION=1 - -- L3 VXLAN and VXLAN-GPE destination UDP port:: - - IP_OVER_VXLAN_EN=1 - IP_OVER_VXLAN_PORT=<udp dport> - -- enable VXLAN-GPE tunnel flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=0 - or - FLEX_PARSER_PROFILE_ENABLE=2 - -- enable IP-in-IP tunnel flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=0 - -- enable MPLS flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=1 - -- enable ICMP(code/type/identifier/sequence number) / ICMP6(code/type) fields matching:: - - FLEX_PARSER_PROFILE_ENABLE=2 - -- enable Geneve flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=0 - or - FLEX_PARSER_PROFILE_ENABLE=1 - -- enable Geneve TLV option flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=0 - -- enable GTP flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=3 - -- enable eCPRI flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=4 - PROG_PARSE_GRAPH=1 - -- enable dynamic flex parser for flex item:: - - FLEX_PARSER_PROFILE_ENABLE=4 - PROG_PARSE_GRAPH=1 - -- enable realtime timestamp format:: - - REAL_TIME_CLOCK_ENABLE=1 - -Linux Prerequisites -------------------- - -This driver relies on external libraries and kernel drivers for resources -allocations and initialization. The following dependencies are not part of -DPDK and must be installed separately: - -- **libibverbs** - - User space Verbs framework used by librte_net_mlx5. This library provides - a generic interface between the kernel and low-level user space drivers - such as libmlx5. - - It allows slow and privileged operations (context initialization, hardware - resources allocations) to be managed by the kernel and fast operations to - never leave user space. - -- **libmlx5** - - Low-level user space driver library for Mellanox - ConnectX-4/ConnectX-5/ConnectX-6/BlueField devices, it is automatically loaded - by libibverbs. - - This library basically implements send/receive calls to the hardware - queues. - -- **Kernel modules** - - They provide the kernel-side Verbs API and low level device drivers that - manage actual hardware initialization and resources sharing with user - space processes. - - Unlike most other PMDs, these modules must remain loaded and bound to - their devices: - - - mlx5_core: hardware driver managing Mellanox - ConnectX-4/ConnectX-5/ConnectX-6/BlueField devices and related Ethernet kernel - network devices. - - mlx5_ib: InfiniBand device driver. - - ib_uverbs: user space driver for Verbs (entry point for libibverbs). - -- **Firmware update** - - Mellanox OFED/EN releases include firmware updates for - ConnectX-4/ConnectX-5/ConnectX-6/BlueField adapters. - - Because each release provides new features, these updates must be applied to - match the kernel modules and libraries they come with. - -.. note:: - - Both libraries are BSD and GPL licensed. Linux kernel modules are GPL - licensed. - -Installation -~~~~~~~~~~~~ - -Either RDMA Core library with a recent enough Linux kernel release -(recommended) or Mellanox OFED/EN, which provides compatibility with older -releases. - -RDMA Core with Linux Kernel -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -- Minimal kernel version : v4.14 or the most recent 4.14-rc (see `Linux installation documentation`_) -- Minimal rdma-core version: v15+ commit 0c5f5765213a ("Merge pull request #227 from yishaih/tm") - (see `RDMA Core installation documentation`_) -- When building for i686 use: - - - rdma-core version 18.0 or above built with 32bit support. - - Kernel version 4.14.41 or above. - -- Starting with rdma-core v21, static libraries can be built:: - - cd build - CFLAGS=-fPIC cmake -DIN_PLACE=1 -DENABLE_STATIC=1 -GNinja .. - ninja - -.. _`Linux installation documentation`: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/plain/Documentation/admin-guide/README.rst -.. _`RDMA Core installation documentation`: https://raw.githubusercontent.com/linux-rdma/rdma-core/master/README.md - - -Mellanox OFED/EN -^^^^^^^^^^^^^^^^ - -- Mellanox OFED version: **4.5** and above / - Mellanox EN version: **4.5** and above -- firmware version: - - - ConnectX-4: **12.21.1000** and above. - - ConnectX-4 Lx: **14.21.1000** and above. - - ConnectX-5: **16.21.1000** and above. - - ConnectX-5 Ex: **16.21.1000** and above. - - ConnectX-6: **20.27.0090** and above. - - ConnectX-6 Dx: **22.27.0090** and above. - - BlueField: **18.25.1010** and above. - -While these libraries and kernel modules are available on OpenFabrics -Alliance's `website <https://www.openfabrics.org/>`__ and provided by package -managers on most distributions, this PMD requires Ethernet extensions that -may not be supported at the moment (this is a work in progress). - -`Mellanox OFED -<https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/>`__ and -`Mellanox EN -<https://network.nvidia.com/products/ethernet-drivers/linux/mlnx_en/>`__ -include the necessary support and should be used in the meantime. For DPDK, -only libibverbs, libmlx5, mlnx-ofed-kernel packages and firmware updates are -required from that distribution. - -.. note:: - - Several versions of Mellanox OFED/EN are available. Installing the version - this DPDK release was developed and tested against is strongly - recommended. Please check the `linux prerequisites`_. - -Windows Prerequisites ---------------------- - -This driver relies on external libraries and kernel drivers for resources -allocations and initialization. The dependencies in the following sub-sections -are not part of DPDK, and must be installed separately. - -Compilation Prerequisites -~~~~~~~~~~~~~~~~~~~~~~~~~ - -DevX SDK installation -^^^^^^^^^^^^^^^^^^^^^ - -The DevX SDK must be installed on the machine building the Windows PMD. -Additional information can be found at -`How to Integrate Windows DevX in Your Development Environment -<https://docs.mellanox.com/display/winof2v250/RShim+Drivers+and+Usage#RShimDriversandUsage-DevXInterface>`__. - -Runtime Prerequisites -~~~~~~~~~~~~~~~~~~~~~ - -WinOF2 version 2.60 or higher must be installed on the machine. - -WinOF2 installation -^^^^^^^^^^^^^^^^^^^ - -The driver can be downloaded from the following site: -`WINOF2 -<https://www.mellanox.com/products/adapter-software/ethernet/windows/winof-2>`__ - -DevX Enablement -^^^^^^^^^^^^^^^ - -DevX for Windows must be enabled in the Windows registry. -The keys ``DevxEnabled`` and ``DevxFsRules`` must be set. -Additional information can be found in the WinOF2 user manual. Supported NICs -------------- @@ -1470,149 +1124,21 @@ Below are detailed device names: * Mellanox\ |reg| ConnectX\ |reg|-6 Dx EN 200G MCX623105AN-VDAT (1x200G) * Mellanox\ |reg| ConnectX\ |reg|-6 Lx EN 25G MCX631102AN-ADAT (2x25G) -Quick Start Guide on OFED/EN ----------------------------- - -1. Download latest Mellanox OFED/EN. For more info check the `linux prerequisites`_. - - -2. Install the required libraries and kernel modules either by installing - only the required set, or by installing the entire Mellanox OFED/EN:: - - ./mlnxofedinstall --upstream-libs --dpdk - -3. Verify the firmware is the correct one:: - - ibv_devinfo - -4. Verify all ports links are set to Ethernet:: - - mlxconfig -d <mst device> query | grep LINK_TYPE - LINK_TYPE_P1 ETH(2) - LINK_TYPE_P2 ETH(2) - - Link types may have to be configured to Ethernet:: - mlxconfig -d <mst device> set LINK_TYPE_P1/2=1/2/3 +Sub-Function +------------ - * LINK_TYPE_P1=<1|2|3> , 1=Infiniband 2=Ethernet 3=VPI(auto-sense) - - For hypervisors, verify SR-IOV is enabled on the NIC:: - - mlxconfig -d <mst device> query | grep SRIOV_EN - SRIOV_EN True(1) - - If needed, configure SR-IOV:: - - mlxconfig -d <mst device> set SRIOV_EN=1 NUM_OF_VFS=16 - mlxfwreset -d <mst device> reset - -5. Restart the driver:: - - /etc/init.d/openibd restart - - or:: - - service openibd restart - - If link type was changed, firmware must be reset as well:: - - mlxfwreset -d <mst device> reset - - For hypervisors, after reset write the sysfs number of virtual functions - needed for the PF. - - To dynamically instantiate a given number of virtual functions (VFs):: - - echo [num_vfs] > /sys/class/infiniband/mlx5_0/device/sriov_numvfs - -6. Install DPDK and you are ready to go. - See :doc:`compilation instructions <../linux_gsg/build_dpdk>`. - -Enable switchdev mode ---------------------- - -Switchdev mode is a mode in E-Switch, that binds between representor and VF or SF. -Representor is a port in DPDK that is connected to a VF or SF in such a way -that assuming there are no offload flows, each packet that is sent from the VF or SF -will be received by the corresponding representor. While each packet that is or SF -sent to a representor will be received by the VF or SF. -This is very useful in case of SRIOV mode, where the first packet that is sent -by the VF or SF will be received by the DPDK application which will decide if this -flow should be offloaded to the E-Switch. After offloading the flow packet -that the VF or SF that are matching the flow will not be received any more by -the DPDK application. - -1. Enable SRIOV mode:: - - mlxconfig -d <mst device> set SRIOV_EN=true - -2. Configure the max number of VFs:: - - mlxconfig -d <mst device> set NUM_OF_VFS=<num of vfs> - -3. Reset the FW:: - - mlxfwreset -d <mst device> reset - -3. Configure the actual number of VFs:: - - echo <num of vfs > /sys/class/net/<net device>/device/sriov_numvfs - -4. Unbind the device (can be rebind after the switchdev mode):: - - echo -n "<device pci address" > /sys/bus/pci/drivers/mlx5_core/unbind - -5. Enable switchdev mode:: - - echo switchdev > /sys/class/net/<net device>/compat/devlink/mode - -Sub-Function support --------------------- - -Sub-Function is a portion of the PCI device, a SF netdev has its own -dedicated queues (txq, rxq). -A SF shares PCI level resources with other SFs and/or with its parent PCI function. - -0. Requirement:: - - OFED version >= 5.4-0.3.3.0 - -1. Configure SF feature:: - - # Run mlxconfig on both PFs on host and ECPFs on BlueField. - mlxconfig -d <mst device> set PER_PF_NUM_SF=1 PF_TOTAL_SF=252 PF_SF_BAR_SIZE=12 - -2. Enable switchdev mode:: - - mlxdevm dev eswitch set pci/<DBDF> mode switchdev - -3. Add SF port:: - - mlxdevm port add pci/<DBDF> flavour pcisf pfnum 0 sfnum <sfnum> - - Get SFID from output: pci/<DBDF>/<SFID> - -4. Modify MAC address:: - - mlxdevm port function set pci/<DBDF>/<SFID> hw_addr <MAC> - -5. Activate SF port:: - - mlxdevm port function set pci/<DBDF>/<ID> state active - -6. Devargs to probe SF device:: - - auxiliary:mlx5_core.sf.<num>,dv_flow_en=1 +See :ref:`mlx5_sub_function`. Sub-Function representor support --------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A SF netdev supports E-Switch representation offload similar to PF and VF representors. Use <sfnum> to probe SF representor:: - testpmd> port attach <PCI_BDF>,representor=sf<sfnum>,dv_flow_en=1 + testpmd> port attach <PCI_BDF>,representor=sf<sfnum>,dv_flow_en=1 + Performance tuning ------------------ diff --git a/doc/guides/platform/index.rst b/doc/guides/platform/index.rst index 2ff91a6018..b1ba11df73 100644 --- a/doc/guides/platform/index.rst +++ b/doc/guides/platform/index.rst @@ -14,4 +14,5 @@ The following are platform specific guides and setup information. cnxk dpaa dpaa2 + mlx5 octeontx diff --git a/doc/guides/platform/mlx5.rst b/doc/guides/platform/mlx5.rst new file mode 100644 index 0000000000..0fd5e6604d --- /dev/null +++ b/doc/guides/platform/mlx5.rst @@ -0,0 +1,602 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2022 6WIND S.A. + Copyright (c) 2022 NVIDIA Corporation & Affiliates + +.. include:: <isonum.txt> + +MLX5 Common Driver +================== + +The mlx5 common driver library (**librte_common_mlx5**) provides support for +**Mellanox ConnectX-4**, **Mellanox ConnectX-4 Lx**, **Mellanox ConnectX-5**, +**Mellanox ConnectX-6**, **Mellanox ConnectX-6 Dx**, **Mellanox ConnectX-6 Lx**, +**Mellanox BlueField** and **Mellanox BlueField-2** families of +10/25/40/50/100/200 Gb/s adapters. + +Information and documentation for these adapters can be found on the +`NVIDIA website <https://www.nvidia.com/en-us/networking/>`_. +Help is also provided by the +`Mellanox community <http://community.mellanox.com/welcome>`_. +In addition, there is a `web section dedicated to the Poll Mode Driver +<https://developer.nvidia.com/networking/dpdk>`_. + + +Design +------ + +For security reasons and to enhance robustness, +this driver only handles virtual memory addresses. +The way resources allocations are handled by the kernel, +combined with hardware specifications that allow handling virtual memory addresses directly, +ensure that DPDK applications cannot access random physical memory +(or memory that does not belong to the current process). + +There are different levels of objects and bypassing abilities +which are used to get the best performance: + +- **Verbs** is a complete high-level generic API +- **Direct Verbs** is a device-specific API +- **DevX** allows accessing firmware objects +- **Direct Rules** manages flow steering at the low-level hardware layer + +On Linux, above interfaces are provided by linking with `libibverbs` and `libmlx5`. +See :ref:`mlx5_linux_prerequisites` for installation. + +On Windows, DevX is the only requirement from the above list. +See :ref:`mlx5_windows_prerequisites` for DevX SDK package installation. + + +.. _mlx5_classes: + +Classes +------- + +One mlx5 device can be probed by a number of different PMDs. +To select a specific PMD, its name should be specified as a device parameter +(e.g. ``0000:08:00.1,class=eth``). + +In order to allow probing by multiple PMDs, +several classes may be listed separated by a colon. +For example: ``class=crypto:regex`` will probe both Crypto and RegEx PMDs. + + +Supported Classes +~~~~~~~~~~~~~~~~~ + +- ``class=compress`` for :doc:`../../compressdevs/mlx5`. +- ``class=crypto`` for :doc:`../../cryptodevs/mlx5`. +- ``class=eth`` for :doc:`../../nics/mlx5`. +- ``class=regex`` for :doc:`../../regexdevs/mlx5`. +- ``class=vdpa`` for :doc:`../../vdpadevs/mlx5`. + +By default, the mlx5 device will be probed by the ``eth`` PMD. + + +Limitations +~~~~~~~~~~~ + +- ``eth`` and ``vdpa`` PMDs cannot be probed at the same time. + All other combinations are possible. + +- On Windows, only ``eth`` and ``crypto`` are supported. + + +.. _mlx5_common_compilation: + +Compilation Prerequisites +------------------------- + +.. _mlx5_linux_prerequisites: + +Linux Prerequisites +~~~~~~~~~~~~~~~~~~~ + +This driver relies on external libraries and kernel drivers for resources +allocations and initialization. +The following dependencies are not part of DPDK and must be installed separately: + +- **libibverbs** + + User space Verbs framework used by ``librte_common_mlx5``. + This library provides a generic interface between the kernel + and low-level user space drivers such as ``libmlx5``. + + It allows slow and privileged operations (context initialization, + hardware resources allocations) to be managed by the kernel + and fast operations to never leave user space. + +- **libmlx5** + + Low-level user space driver library for Mellanox devices, + it is automatically loaded by ``libibverbs``. + + This library basically implements send/receive calls to the hardware queues. + +- **Kernel modules** + + They provide the kernel-side Verbs API and low level device drivers + that manage actual hardware initialization + and resources sharing with user-space processes. + + Unlike most other PMDs, these modules must remain loaded and bound to + their devices: + + - ``mlx5_core``: hardware driver managing Mellanox devices + and related Ethernet kernel network devices. + - ``mlx5_ib``: InfiniBand device driver. + - ``ib_uverbs``: user space driver for Verbs (entry point for ``libibverbs``). + +- **Firmware update** + + Mellanox OFED/EN releases include firmware updates. + + Because each release provides new features, these updates must be applied to + match the kernel modules and libraries they come with. + +Libraries and kernel modules can be provided either by the Linux distribution, +or by installing Mellanox OFED/EN which provides compatibility with older kernels. + + +Upstream Dependencies +^^^^^^^^^^^^^^^^^^^^^ + +The mlx5 kernel modules are part of upstream Linux. +The minimal supported kernel version is 4.14. +For 32-bit, version 4.14.41 or above is required. + +The libraries `libibverbs` and `libmlx5` are part of ``rdma-core``. +It is packaged by most of Linux distributions. +The minimal supported rdma-core version is 16. +For 32-bit, version 18 or above is required. + +The rdma-core sources can be downloaded at +https://github.com/linux-rdma/rdma-core + +It is possible to build rdma-core as static libraries starting with version 21:: + + cd build + CFLAGS=-fPIC cmake -DIN_PLACE=1 -DENABLE_STATIC=1 -GNinja .. + ninja + + +Mellanox OFED/EN +^^^^^^^^^^^^^^^^ + +The kernel modules and libraries are packaged with other tools +in Mellanox OFED or Mellanox EN. +The minimal supported versions are: + +- Mellanox OFED version: **4.5** and above. +- Mellanox EN version: **4.5** and above. +- Firmware version: + + - ConnectX-4: **12.21.1000** and above. + - ConnectX-4 Lx: **14.21.1000** and above. + - ConnectX-5: **16.21.1000** and above. + - ConnectX-5 Ex: **16.21.1000** and above. + - ConnectX-6: **20.27.0090** and above. + - ConnectX-6 Dx: **22.27.0090** and above. + - BlueField: **18.25.1010** and above. + - BlueField-2: **24.28.1002** and above. + +The firmware, the libraries libibverbs, libmlx5, and mlnx-ofed-kernel modules +are packaged in `Mellanox OFED +<https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/>`_. +After downloading, it can be installed with this command:: + + ./mlnxofedinstall --dpdk + +`Mellanox EN +<https://network.nvidia.com/products/ethernet-drivers/linux/mlnx_en/>`_ +is a smaller package including what is needed for DPDK. +After downloading, it can be installed with this command:: + + ./install --dpdk + +After installing, the firmware version can be checked:: + + ibv_devinfo + +.. note:: + + Several versions of Mellanox OFED/EN are available. Installing the version + this DPDK release was developed and tested against is strongly recommended. + Please check the "Tested Platforms" section in the :doc:`../../rel_notes/index`. + + +.. _mlx5_windows_prerequisites: + +Windows Prerequisites +~~~~~~~~~~~~~~~~~~~~~ + +The mlx5 PMDs rely on external libraries and kernel drivers +for resource allocation and initialization. + + +DevX SDK Installation +^^^^^^^^^^^^^^^^^^^^^ + +The DevX SDK must be installed on the machine building the Windows PMD. +Additional information can be found at +`How to Integrate Windows DevX in Your Development Environment +<https://docs.nvidia.com/networking/display/winof2v260/RShim+Drivers+and+Usage#RShimDriversandUsage-DevXInterface>`_. +The minimal supported WinOF2 version is 2.60. + + +Compilation Options +------------------- + +Compilation on Linux +~~~~~~~~~~~~~~~~~~~~ + +The ibverbs libraries can be linked with this PMD in a number of ways, +configured by the ``ibverbs_link`` build option: + +``shared`` (default) + The PMD depends on some .so files. + +``dlopen`` + Split the dependencies glue in a separate library + loaded when needed by dlopen (see ``MLX5_GLUE_PATH``). + It makes dependencies on libibverbs and libmlx5 optional, + and has no performance impact. + +``static`` + Embed static flavor of the dependencies libibverbs and libmlx5 + in the PMD shared library or the executable static binary. + + +Compilation on Windows +~~~~~~~~~~~~~~~~~~~~~~ + +The DevX SDK location must be set through two environment variables: + +``DEVX_LIB_PATH`` + path to the DevX lib file. + +``DEVX_INC_PATH`` + path to the DevX header files. + + +.. _mlx5_common_env: + +Environment Configuration +------------------------- + +Linux Environment +~~~~~~~~~~~~~~~~~ + +The kernel network interfaces are brought up during initialization. +Forcing them down prevents packets reception. + +The ethtool operations on the kernel interfaces may also affect the PMD. + +Some runtime behaviours may be configured through environment variables. + +``MLX5_GLUE_PATH`` + If built with ``ibverbs_link=dlopen``, + list of directories in which to search for the rdma-core "glue" plug-in, + separated by colons or semi-colons. + +``MLX5_SHUT_UP_BF`` + If Verbs is used (DevX disabled), + HW queue doorbell register mapping. + The value 0 means non-cached IO mapping, + while 1 is a regular memory mapping. + + With regular memory mapping, the register is flushed to HW + usually when the write-combining buffer becomes full, + but it depends on CPU design. + + +Port Link with OFED/EN +^^^^^^^^^^^^^^^^^^^^^^ + +Ports links must be set to Ethernet:: + + mlxconfig -d <mst device> query | grep LINK_TYPE + LINK_TYPE_P1 ETH(2) + LINK_TYPE_P2 ETH(2) + + mlxconfig -d <mst device> set LINK_TYPE_P1/2=1/2/3 + +Link type values are: + +* ``1`` Infiniband +* ``2`` Ethernet +* ``3`` VPI (auto-sense) + +If link type was changed, firmware must be reset as well:: + + mlxfwreset -d <mst device> reset + + +.. _mlx5_vf: + +SR-IOV Virtual Function with OFED/EN +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +SR-IOV must be enabled on the NIC. +It can be checked in the following command:: + + mlxconfig -d <mst device> query | grep SRIOV_EN + SRIOV_EN True(1) + +If needed, configure SR-IOV:: + + mlxconfig -d <mst device> set SRIOV_EN=1 NUM_OF_VFS=16 + mlxfwreset -d <mst device> reset + +After doing the change, restart the driver:: + + /etc/init.d/openibd restart + +or:: + + service openibd restart + +Then the virtual functions can be instantiated:: + + echo [num_vfs] > /sys/class/infiniband/mlx5_0/device/sriov_numvfs + + +.. _mlx5_sub_function: + +Sub-Function with OFED/EN +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Sub-Function is a portion of the PCI device, +it has its own dedicated queues. +An SF shares PCI-level resources with other SFs and/or with its parent PCI function. + +0. Requirement:: + + OFED version >= 5.4-0.3.3.0 + +1. Configure SF feature:: + + # Run mlxconfig on both PFs on host and ECPFs on BlueField. + mlxconfig -d <mst device> set PER_PF_NUM_SF=1 PF_TOTAL_SF=252 PF_SF_BAR_SIZE=12 + +2. Enable switchdev mode:: + + mlxdevm dev eswitch set pci/<DBDF> mode switchdev + +3. Add SF port:: + + mlxdevm port add pci/<DBDF> flavour pcisf pfnum 0 sfnum <sfnum> + + Get SFID from output: pci/<DBDF>/<SFID> + +4. Modify MAC address:: + + mlxdevm port function set pci/<DBDF>/<SFID> hw_addr <MAC> + +5. Activate SF port:: + + mlxdevm port function set pci/<DBDF>/<ID> state active + +6. Devargs to probe SF device:: + + auxiliary:mlx5_core.sf.<num>,class=eth:regex + + +Enable Switchdev Mode +^^^^^^^^^^^^^^^^^^^^^ + +Switchdev mode is a mode in E-Switch, that binds between representor and VF or SF. +Representor is a port in DPDK that is connected to a VF or SF in such a way +that assuming there are no offload flows, each packet that is sent from the VF or SF +will be received by the corresponding representor. +While each packet that is sent to a representor will be received by the VF or SF. + +After :ref:`configuring VF <mlx5_vf>`, the device must be unbound:: + + printf "<device pci address>" > /sys/bus/pci/drivers/mlx5_core/unbind + +Then switchdev mode is enabled:: + + echo switchdev > /sys/class/net/<net device>/compat/devlink/mode + +The device can be bound again at this point. + + +Run as Non-Root +^^^^^^^^^^^^^^^ + +In order to run as a non-root user, +some capabilities must be granted to the application:: + + setcap cap_sys_admin,cap_net_admin,cap_net_raw,cap_ipc_lock+ep <dpdk-app> + +Below are the reasons for the need of each capability: + +``cap_sys_admin`` + When using physical addresses (PA mode), with Linux >= 4.0, + for access to ``/proc/self/pagemap``. + +``cap_net_admin`` + For device configuration. + +``cap_net_raw`` + For raw ethernet queue allocation through kernel driver. + +``cap_ipc_lock`` + For DMA memory pinning. + + +Windows Environment +~~~~~~~~~~~~~~~~~~~ + +WinOF2 version 2.60 or higher must be installed on the machine. + + +WinOF2 Installation +^^^^^^^^^^^^^^^^^^^ + +The driver can be downloaded from the following site: `WINOF2 +<https://network.nvidia.com/products/adapter-software/ethernet/windows/winof-2/>`_. + + +DevX Enablement +^^^^^^^^^^^^^^^ + +DevX for Windows must be enabled in the Windows registry. +The keys ``DevxEnabled`` and ``DevxFsRules`` must be set. +Additional information can be found in the WinOF2 user manual. + + +.. _mlx5_firmware_config: + +Firmware Configuration +~~~~~~~~~~~~~~~~~~~~~~ + +Firmware features can be configured as key/value pairs. + +The command to set a value is:: + + mlxconfig -d <device> set <key>=<value> + +The command to query a value is:: + + mlxconfig -d <device> query <key> + +The device name for the command ``mlxconfig`` can be either the PCI address, +or the mst device name found with:: + + mst status + +Below are some firmware configurations listed. + +- link type:: + + LINK_TYPE_P1 + LINK_TYPE_P2 + value: 1=Infiniband 2=Ethernet 3=VPI(auto-sense) + +- enable SR-IOV:: + + SRIOV_EN=1 + +- the maximum number of SR-IOV virtual functions:: + + NUM_OF_VFS=<max> + +- enable DevX (required by Direct Rules and other features):: + + UCTX_EN=1 + +- aggressive CQE zipping:: + + CQE_COMPRESSION=1 + +- L3 VXLAN and VXLAN-GPE destination UDP port:: + + IP_OVER_VXLAN_EN=1 + IP_OVER_VXLAN_PORT=<udp dport> + +- enable VXLAN-GPE tunnel flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=0 + or + FLEX_PARSER_PROFILE_ENABLE=2 + +- enable IP-in-IP tunnel flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=0 + +- enable MPLS flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=1 + +- enable ICMP(code/type/identifier/sequence number) / ICMP6(code/type) fields matching:: + + FLEX_PARSER_PROFILE_ENABLE=2 + +- enable Geneve flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=0 + or + FLEX_PARSER_PROFILE_ENABLE=1 + +- enable Geneve TLV option flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=0 + +- enable GTP flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=3 + +- enable eCPRI flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=4 + PROG_PARSE_GRAPH=1 + +- enable dynamic flex parser for flex item:: + + FLEX_PARSER_PROFILE_ENABLE=4 + PROG_PARSE_GRAPH=1 + +- enable realtime timestamp format:: + + REAL_TIME_CLOCK_ENABLE=1 + + +.. _mlx5_common_driver_options: + +Device Arguments +---------------- + +The driver can be configured per device. +A single argument list can be used for a device managed by multiple PMDs. +The parameters must be passed through the EAL option ``-a``, +as examples below: + +- PCI device:: + + -a 0000:03:00.2,class=eth:regex,mr_mempool_reg_en=0 + +- Auxiliary SF:: + + -a auxiliary:mlx5_core.sf.2,class=compress,mr_ext_memseg_en=0 + +Each device class PMD has its own list of specific arguments, +and below are the arguments supported by the common mlx5 layer. + +- ``class`` parameter [string] + + Select the classes of the drivers that should probe the device. + See :ref:`mlx5_classes` for more explanation and details. + + The default value is ``eth``. + +- ``mr_ext_memseg_en`` parameter [int] + + A nonzero value enables extending memseg when registering DMA memory. If + enabled, the number of entries in MR (Memory Region) lookup table on datapath + is minimized and it benefits performance. On the other hand, it worsens memory + utilization because registered memory is pinned by kernel driver. Even if a + page in the extended chunk is freed, that doesn't become reusable until the + entire memory is freed. + + Enabled by default. + +- ``mr_mempool_reg_en`` parameter [int] + + A nonzero value enables implicit registration of DMA memory of all mempools + except those having ``RTE_MEMPOOL_F_NON_IO``. This flag is set automatically + for mempools populated with non-contiguous objects or those without IOVA. + The effect is that when a packet from a mempool is transmitted, + its memory is already registered for DMA in the PMD and no registration + will happen on the data path. The tradeoff is extra work on the creation + of each mempool and increased HW resource use if some mempools + are not used with MLX5 devices. + + Enabled by default. + +- ``sys_mem_en`` parameter [int] + + A non-zero value enables the PMD memory management allocating memory + from system by default, without explicit rte memory flag. + + By default, the PMD will set this value to 0. diff --git a/doc/guides/regexdevs/mlx5.rst b/doc/guides/regexdevs/mlx5.rst index fc2cacba60..83280bda40 100644 --- a/doc/guides/regexdevs/mlx5.rst +++ b/doc/guides/regexdevs/mlx5.rst @@ -3,10 +3,10 @@ .. include:: <isonum.txt> -MLX5 RegEx driver +MLX5 RegEx Driver ================= -The MLX5 RegEx (Regular Expression) driver library +The mlx5 RegEx (Regular Expression) driver library (**librte_regex_mlx5**) provides support for **Mellanox BlueField-2** families of 25/50/100/200 Gb/s adapters. @@ -17,29 +17,21 @@ This PMD is configuring the RegEx HW engine. For the PMD to work, the application must supply a precompiled rule file in rof2 format. -The PMD uses libibverbs and libmlx5 to access the device firmware -or directly the hardware components. -There are different levels of objects and bypassing abilities -to get the best performances: - -- Verbs is a complete high-level generic API -- Direct Verbs is a device-specific API -- DevX allows to access firmware objects - -Enabling librte_regex_mlx5 causes DPDK applications to be linked against -libibverbs. - -Mellanox mlx5 pci device can be probed by number of different pci devices, -for example net / vDPA / RegEx. To select the RegEx PMD ``class=regex`` should -be specified as device parameter. The RegEx device can be probed and used with -other Mellanox devices, by adding more options in the class. -For example: ``class=net:regex`` will probe both the net PMD and the RegEx PMD. +See :doc:`../../platform/mlx5` guide for more design details. Features -------- - Multi segments mbuf support. +Configuration +------------- + +See :ref:`mlx5 common compilation <mlx5_common_compilation>`, +:ref:`mlx5 firmware configuration <mlx5_firmware_config>`, +and :ref:`mlx5 common driver options <mlx5_common_driver_options>`. + + Supported NICs -------------- @@ -52,12 +44,8 @@ Prerequisites - Enable the RegEx capabilities using system call from the BlueField-2. - Official support is not yet released. + Limitations ----------- - The firmware version must be greater than XX.31.0364 - -Run-time configuration -~~~~~~~~~~~~~~~~~~~~~~ - -- **ethtool** operations on related kernel interfaces also affect the PMD. diff --git a/doc/guides/vdpadevs/mlx5.rst b/doc/guides/vdpadevs/mlx5.rst index 30f0b62eb4..acb791032a 100644 --- a/doc/guides/vdpadevs/mlx5.rst +++ b/doc/guides/vdpadevs/mlx5.rst @@ -3,10 +3,10 @@ .. include:: <isonum.txt> -MLX5 vDPA driver +MLX5 vDPA Driver ================ -The MLX5 vDPA (vhost data path acceleration) driver library +The mlx5 vDPA (vhost data path acceleration) driver library (**librte_vdpa_mlx5**) provides support for **Mellanox ConnectX-6**, **Mellanox ConnectX-6 Dx** and **Mellanox BlueField** families of 10/25/40/50/100/200 Gb/s adapters as well as their virtual functions (VF) in @@ -17,33 +17,8 @@ SR-IOV context. This driver is enabled automatically when using "meson" build system which will detect dependencies. - -Design ------- - -For security reasons and robustness, this driver only deals with virtual -memory addresses. The way resources allocations are handled by the kernel, -combined with hardware specifications that allow to handle virtual memory -addresses directly, ensure that DPDK applications cannot access random -physical memory (or memory that does not belong to the current process). - -The PMD can use libibverbs and libmlx5 to access the device firmware -or directly the hardware components. -There are different levels of objects and bypassing abilities -to get the best performances: - -- Verbs is a complete high-level generic API -- Direct Verbs is a device-specific API -- DevX allows to access firmware objects -- Direct Rules manages flow steering at low-level hardware layer - -Enabling librte_vdpa_mlx5 causes DPDK applications to be linked against -libibverbs. - -A Mellanox mlx5 PCI device can be probed by either net/mlx5 driver or vdpa/mlx5 -driver but not in parallel. Hence, the user should decide the driver by the -``class`` parameter in the device argument list. -By default, the mlx5 device will be probed by the net/mlx5 driver. +See :doc:`../../platform/mlx5` guide for design details, +and which PMDs can be combined with vDPA PMD. Supported NICs -------------- @@ -58,52 +33,16 @@ Prerequisites ------------- - Mellanox OFED version: **5.0** - see :doc:`../../nics/mlx5` guide for more Mellanox OFED details. - -Compilation option -~~~~~~~~~~~~~~~~~~ - -The meson option ``ibverbs_link`` is **shared** by default, -but can be configured to have the following values: - -- ``dlopen`` - - Build PMD with additional code to make it loadable without hard - dependencies on **libibverbs** nor **libmlx5**, which may not be installed - on the target system. - - In this mode, their presence is still required for it to run properly, - however their absence won't prevent a DPDK application from starting (with - DPDK shared build disabled) and they won't show up as missing with ``ldd(1)``. - - It works by moving these dependencies to a purpose-built rdma-core "glue" - plug-in which must be installed in a directory whose name is based - on ``RTE_EAL_PMD_PATH`` suffixed with ``-glue``. - - This option has no performance impact. - -- ``static`` - - Embed static flavor of the dependencies **libibverbs** and **libmlx5** - in the PMD shared library or the executable static binary. - -.. note:: - - Default armv8a configuration of meson build sets ``RTE_CACHE_LINE_SIZE`` - to 128 then brings performance degradation. + See :ref:`mlx5 common prerequisites <mlx5_linux_prerequisites>` for more details. Run-time configuration ~~~~~~~~~~~~~~~~~~~~~~ -- **ethtool** operations on related kernel interfaces also affect the PMD. - Driver options ^^^^^^^^^^^^^^ -- ``class`` parameter [string] - - Select the class of the driver that should probe the device. - `vdpa` for the mlx5 vDPA driver. +Please refer to :ref:`mlx5 common options <mlx5_common_driver_options>` +for an additional list of options shared with other mlx5 drivers. - ``event_mode`` parameter [int] @@ -163,18 +102,6 @@ Driver options - 0, HW default. -Devargs example -^^^^^^^^^^^^^^^ - -- PCI devargs:: - - -a 0000:03:00.2,class=vdpa - -- Auxiliary devargs:: - - -a auxiliary:mlx5_core.sf.2,class=vdpa - - Error handling ^^^^^^^^^^^^^^ -- 2.25.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 5/5] doc: update doorbell mapping parameter name in mlx5 guide 2022-02-22 12:48 [PATCH 0/5] refactore mlx5 guides Michael Baum ` (3 preceding siblings ...) 2022-02-22 12:48 ` [PATCH 4/5] doc: add shared guide for mlx5 PMDs Michael Baum @ 2022-02-22 12:48 ` Michael Baum 2022-02-22 14:43 ` [PATCH 0/5] refactore mlx5 guides Raslan Darawsheh 2022-02-23 13:48 ` [PATCH v2 0/5] Refactor " Michael Baum 6 siblings, 0 replies; 17+ messages in thread From: Michael Baum @ 2022-02-22 12:48 UTC (permalink / raw) To: dev; +Cc: Matan Azrad, Raslan Darawsheh, Viacheslav Ovsiienko The "tx_db_nc" devarg forces doorbell register mapping to non-cached region eliminating the extra write memory barrier. This argument was used in creating the UAR for Tx and thus affected its performance. Recently [1] its use has been extended to all UAR creation in all mlx5 drivers, and now its name is no longer so accurate. This patch changes its name to "sq_db_nc" to suit any send queue that uses it. The old name will still work for backward compatibility. [1] commit 5dfa003db53f ("common/mlx5: fix post doorbell barrier") Signed-off-by: Michael Baum <michaelba@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> --- doc/guides/nics/mlx5.rst | 34 ++----------------- doc/guides/platform/mlx5.rst | 39 ++++++++++++++++++++++ drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- drivers/common/mlx5/mlx5_common.c | 31 ++++++++++++----- drivers/common/mlx5/mlx5_common_defs.h | 8 ++--- drivers/net/mlx5/linux/mlx5_verbs.c | 2 +- drivers/net/mlx5/mlx5_devx.c | 2 +- 7 files changed, 72 insertions(+), 46 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 8eb67efcf3..968aac01d2 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -814,37 +814,9 @@ for an additional list of options shared with other mlx5 drivers. - ``tx_db_nc`` parameter [int] - The rdma core library can map doorbell register in two ways, depending on the - environment variable "MLX5_SHUT_UP_BF": - - - As regular cached memory (usually with write combining attribute), if the - variable is either missing or set to zero. - - As non-cached memory, if the variable is present and set to not "0" value. - - The type of mapping may slightly affect the Tx performance, the optimal choice - is strongly relied on the host architecture and should be deduced practically. - - If ``tx_db_nc`` is set to zero, the doorbell is forced to be mapped to regular - memory (with write combining), the PMD will perform the extra write memory barrier - after writing to doorbell, it might increase the needed CPU clocks per packet - to send, but latency might be improved. - - If ``tx_db_nc`` is set to one, the doorbell is forced to be mapped to non - cached memory, the PMD will not perform the extra write memory barrier - after writing to doorbell, on some architectures it might improve the - performance. - - If ``tx_db_nc`` is set to two, the doorbell is forced to be mapped to regular - memory, the PMD will use heuristics to decide whether write memory barrier - should be performed. For bursts with size multiple of recommended one (64 pkts) - it is supposed the next burst is coming and no need to issue the extra memory - barrier (it is supposed to be issued in the next coming burst, at least after - descriptor writing). It might increase latency (on some hosts till next - packets transmit) and should be used with care. - - If ``tx_db_nc`` is omitted or set to zero, the preset (if any) environment - variable "MLX5_SHUT_UP_BF" value is used. If there is no "MLX5_SHUT_UP_BF", - the default ``tx_db_nc`` value is zero for ARM64 hosts and one for others. + This parameter name is deprecated and ignored. + The new name for this parameter is ``sq_db_nc``. + See :ref:`common driver options <mlx5_common_driver_options>`. - ``tx_pp`` parameter [int] diff --git a/doc/guides/platform/mlx5.rst b/doc/guides/platform/mlx5.rst index 0fd5e6604d..d073c213ca 100644 --- a/doc/guides/platform/mlx5.rst +++ b/doc/guides/platform/mlx5.rst @@ -600,3 +600,42 @@ and below are the arguments supported by the common mlx5 layer. from system by default, without explicit rte memory flag. By default, the PMD will set this value to 0. + +- ``sq_db_nc`` parameter [int] + + The rdma core library can map doorbell register in two ways, + depending on the environment variable "MLX5_SHUT_UP_BF": + + - As regular cached memory (usually with write combining attribute), + if the variable is either missing or set to zero. + - As non-cached memory, if the variable is present and set to not "0" value. + + The same doorbell mapping approach is implemented directly by PMD + in UAR generation for queues created with DevX. + + The type of mapping may slightly affect the send queue performance, + the optimal choice strongly relied on the host architecture + and should be deduced practically. + + If ``sq_db_nc`` is set to zero, the doorbell is forced to be mapped to + regular memory (with write combining), the PMD will perform the extra write + memory barrier after writing to doorbell, it might increase the needed CPU + clocks per packet to send, but latency might be improved. + + If ``sq_db_nc`` is set to one, the doorbell is forced to be mapped to non + cached memory, the PMD will not perform the extra write memory barrier after + writing to doorbell, on some architectures it might improve the performance. + + If ``sq_db_nc`` is set to two, the doorbell is forced to be mapped to + regular memory, the PMD will use heuristics to decide whether a write memory + barrier should be performed. For bursts with size multiple of recommended one + (64 pkts) it is supposed the next burst is coming and no need to issue the + extra memory barrier (it is supposed to be issued in the next coming burst, + at least after descriptor writing). It might increase latency (on some hosts + till the next packets transmit) and should be used with care. + The PMD uses heuristics only for Tx queue, for other semd queues the doorbell + is forced to be mapped to regular memory as same as ``sq_db_nc`` is set to 0. + + If ``sq_db_nc`` is omitted, the preset (if any) environment variable + "MLX5_SHUT_UP_BF" value is used. If there is no "MLX5_SHUT_UP_BF", the + default ``sq_db_nc`` value is zero for ARM64 hosts and one for others. \ No newline at end of file diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c index 0d3e24e04e..a752d79e8e 100644 --- a/drivers/common/mlx5/linux/mlx5_common_os.c +++ b/drivers/common/mlx5/linux/mlx5_common_os.c @@ -630,7 +630,7 @@ mlx5_config_doorbell_mapping_env(int dbnc) setenv(MLX5_SHUT_UP_BF, MLX5_SHUT_UP_BF_DEFAULT, 1); else setenv(MLX5_SHUT_UP_BF, - dbnc == MLX5_TXDB_NCACHED ? "1" : "0", 1); + dbnc == MLX5_SQ_DB_NCACHED ? "1" : "0", 1); return value; } diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 96906d3f39..8cf391df13 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -35,10 +35,17 @@ uint8_t haswell_broadwell_cpu; /* * Device parameter to force doorbell register mapping - * to non-cahed region eliminating the extra write memory barrier. + * to non-cached region eliminating the extra write memory barrier. + * Deprecated, ignored (Name changed to sq_db_nc). */ #define MLX5_TX_DB_NC "tx_db_nc" +/* + * Device parameter to force doorbell register mapping + * to non-cached region eliminating the extra write memory barrier. + */ +#define MLX5_SQ_DB_NC "sq_db_nc" + /* In case this is an x86_64 intel processor to check if * we should use relaxed ordering. */ @@ -255,11 +262,17 @@ mlx5_common_args_check_handler(const char *key, const char *val, void *opaque) DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val); return -rte_errno; } - if (strcmp(key, MLX5_TX_DB_NC) == 0) { - if (tmp != MLX5_TXDB_CACHED && - tmp != MLX5_TXDB_NCACHED && - tmp != MLX5_TXDB_HEURISTIC) { - DRV_LOG(ERR, "Invalid Tx doorbell mapping parameter."); + if (strcmp(key, MLX5_TX_DB_NC) == 0) + DRV_LOG(WARNING, + "%s: deprecated parameter, converted to queue_db_nc", + key); + if (strcmp(key, MLX5_SQ_DB_NC) == 0 || + strcmp(key, MLX5_TX_DB_NC) == 0) { + if (tmp != MLX5_SQ_DB_CACHED && + tmp != MLX5_SQ_DB_NCACHED && + tmp != MLX5_SQ_DB_HEURISTIC) { + DRV_LOG(ERR, + "Invalid Send Queue doorbell mapping parameter."); rte_errno = EINVAL; return -rte_errno; } @@ -293,6 +306,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist, RTE_DEVARGS_KEY_CLASS, MLX5_DRIVER_KEY, MLX5_TX_DB_NC, + MLX5_SQ_DB_NC, MLX5_MR_EXT_MEMSEG_EN, MLX5_SYS_MEM_EN, MLX5_MR_MEMPOOL_REG_EN, @@ -317,7 +331,8 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist, DRV_LOG(DEBUG, "mr_ext_memseg_en is %u.", config->mr_ext_memseg_en); DRV_LOG(DEBUG, "mr_mempool_reg_en is %u.", config->mr_mempool_reg_en); DRV_LOG(DEBUG, "sys_mem_en is %u.", config->sys_mem_en); - DRV_LOG(DEBUG, "Tx doorbell mapping parameter is %d.", config->dbnc); + DRV_LOG(DEBUG, "Send Queue doorbell mapping parameter is %d.", + config->dbnc); return ret; } @@ -1231,7 +1246,7 @@ mlx5_devx_alloc_uar(struct mlx5_common_device *cdev) for (retry = 0; retry < MLX5_ALLOC_UAR_RETRY; ++retry) { #ifdef MLX5DV_UAR_ALLOC_TYPE_NC /* Control the mapping type according to the settings. */ - uar_mapping = (cdev->config.dbnc == MLX5_TXDB_NCACHED) ? + uar_mapping = (cdev->config.dbnc == MLX5_SQ_DB_NCACHED) ? MLX5DV_UAR_ALLOC_TYPE_NC : MLX5DV_UAR_ALLOC_TYPE_BF; #else /* diff --git a/drivers/common/mlx5/mlx5_common_defs.h b/drivers/common/mlx5/mlx5_common_defs.h index ca80cd8d29..68b700dc0b 100644 --- a/drivers/common/mlx5/mlx5_common_defs.h +++ b/drivers/common/mlx5/mlx5_common_defs.h @@ -34,10 +34,10 @@ /* Default PMD specific parameter value. */ #define MLX5_ARG_UNSET (-1) -/* MLX5_TX_DB_NC supported values. */ -#define MLX5_TXDB_CACHED 0 -#define MLX5_TXDB_NCACHED 1 -#define MLX5_TXDB_HEURISTIC 2 +/* MLX5_SQ_DB_NC supported values. */ +#define MLX5_SQ_DB_CACHED 0 +#define MLX5_SQ_DB_NCACHED 1 +#define MLX5_SQ_DB_HEURISTIC 2 /* Fields of memory mapping type in offset parameter of mmap() */ #define MLX5_UAR_MMAP_CMD_SHIFT 8 diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c index 73c44138de..dfbc5a1e08 100644 --- a/drivers/net/mlx5/linux/mlx5_verbs.c +++ b/drivers/net/mlx5/linux/mlx5_verbs.c @@ -926,7 +926,7 @@ mlx5_txq_ibv_uar_init(struct mlx5_txq_ctrl *txq_ctrl, void *bf_reg) DRV_LOG(ERR, "Failed to get mem page size"); rte_errno = ENOMEM; } - txq->db_heu = priv->sh->cdev->config.dbnc == MLX5_TXDB_HEURISTIC; + txq->db_heu = priv->sh->cdev->config.dbnc == MLX5_SQ_DB_HEURISTIC; txq->db_nc = mlx5_db_map_type_get(uar_mmap_offset, page_size); ppriv->uar_table[txq->idx].db = bf_reg; #ifndef RTE_ARCH_64 diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index e57787cfec..f18b18b1a2 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -1326,7 +1326,7 @@ mlx5_txq_devx_obj_new(struct rte_eth_dev *dev, uint16_t idx) txq_data->qp_db = &txq_obj->sq_obj.db_rec[MLX5_SND_DBR]; *txq_data->qp_db = 0; txq_data->qp_num_8s = txq_obj->sq_obj.sq->id << 8; - txq_data->db_heu = sh->cdev->config.dbnc == MLX5_TXDB_HEURISTIC; + txq_data->db_heu = sh->cdev->config.dbnc == MLX5_SQ_DB_HEURISTIC; txq_data->db_nc = sh->tx_uar.dbnc; /* Change Send Queue state to Ready-to-Send. */ ret = mlx5_txq_devx_modify(txq_obj, MLX5_TXQ_MOD_RST2RDY, 0); -- 2.25.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: [PATCH 0/5] refactore mlx5 guides 2022-02-22 12:48 [PATCH 0/5] refactore mlx5 guides Michael Baum ` (4 preceding siblings ...) 2022-02-22 12:48 ` [PATCH 5/5] doc: update doorbell mapping parameter name in mlx5 guide Michael Baum @ 2022-02-22 14:43 ` Raslan Darawsheh 2022-02-23 13:48 ` [PATCH v2 0/5] Refactor " Michael Baum 6 siblings, 0 replies; 17+ messages in thread From: Raslan Darawsheh @ 2022-02-22 14:43 UTC (permalink / raw) To: Michael Baum, dev; +Cc: Matan Azrad, Slava Ovsiienko Hi, > -----Original Message----- > From: Michael Baum <michaelba@nvidia.com> > Sent: Tuesday, February 22, 2022 2:48 PM > To: dev@dpdk.org > Cc: Matan Azrad <matan@nvidia.com>; Raslan Darawsheh > <rasland@nvidia.com>; Slava Ovsiienko <viacheslavo@nvidia.com> > Subject: [PATCH 0/5] refactore mlx5 guides > > Recently [1] all the drivers running over the mlx5 device started sharing some > objects. > This raised the need to create documentation for the common driver that > would contain information relevant to all drivers and in particular: > > - Tutorial for configuration of the device. > - Instruction for using Mellanox ofed. > - devargs of the common driver. > > This series adds such documentation and reference to it as parts of the > documentation of the other drivers. > > [1] > https://patchwork.dpdk.org/project/dpdk/cover/20211019205602.3188203- > 1-michaelba@nvidia.com/ > > Michael Baum (5): > doc: remove obsolete explanations from mlx5 guide > doc: replace broken links in mlx5 guide > doc: correct name of BlueField-2 in mlx5 guide > doc: add shared guide for mlx5 PMDs > doc: update doorbell mapping parameter name in mlx5 guide > > doc/guides/compressdevs/mlx5.rst | 40 +- > doc/guides/cryptodevs/mlx5.rst | 38 +- > doc/guides/nics/mlx4.rst | 4 +- > doc/guides/nics/mlx5.rst | 567 +----------------- > doc/guides/platform/index.rst | 1 + > doc/guides/platform/mlx5.rst | 641 +++++++++++++++++++++ > doc/guides/regexdevs/mlx5.rst | 44 +- > doc/guides/vdpadevs/mlx5.rst | 87 +-- > drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- > drivers/common/mlx5/mlx5_common.c | 31 +- > drivers/common/mlx5/mlx5_common_defs.h | 8 +- > drivers/net/mlx5/linux/mlx5_verbs.c | 2 +- > drivers/net/mlx5/mlx5_devx.c | 2 +- > 13 files changed, 747 insertions(+), 720 deletions(-) create mode 100644 > doc/guides/platform/mlx5.rst > > -- > 2.25.1 For the series, Reviewed-by: Raslan Darawsheh <rasland@nvidia.com> Kindest regards, Raslan Darawsheh ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH v2 0/5] Refactor mlx5 guides 2022-02-22 12:48 [PATCH 0/5] refactore mlx5 guides Michael Baum ` (5 preceding siblings ...) 2022-02-22 14:43 ` [PATCH 0/5] refactore mlx5 guides Raslan Darawsheh @ 2022-02-23 13:48 ` Michael Baum 2022-02-23 13:48 ` [PATCH v2 1/5] doc: remove obsolete explanations from mlx5 guide Michael Baum ` (5 more replies) 6 siblings, 6 replies; 17+ messages in thread From: Michael Baum @ 2022-02-23 13:48 UTC (permalink / raw) To: dev; +Cc: Matan Azrad, Raslan Darawsheh, Viacheslav Ovsiienko Recently [1] all the drivers running over the mlx5 device started sharing some objects. This raised the need to create documentation for the common driver that would contain information relevant to all drivers and in particular: - Tutorial for configuration of the device. - Instruction for using Mellanox ofed. - devargs of the common driver. This series adds such documentation and reference to it as parts of the documentation of the other drivers. [1] https://patchwork.dpdk.org/project/dpdk/cover/20211019205602.3188203-1-michaelba@nvidia.com/ v2: Rebase Michael Baum (5): doc: remove obsolete explanations from mlx5 guide doc: replace broken links in mlx5 guide doc: correct name of BlueField-2 in mlx5 guide doc: add shared guide for mlx5 PMDs doc: update doorbell mapping parameter name in mlx5 guide doc/guides/compressdevs/mlx5.rst | 40 +- doc/guides/cryptodevs/mlx5.rst | 38 +- doc/guides/nics/mlx4.rst | 4 +- doc/guides/nics/mlx5.rst | 567 +----------------- doc/guides/platform/index.rst | 1 + doc/guides/platform/mlx5.rst | 641 +++++++++++++++++++++ doc/guides/regexdevs/mlx5.rst | 44 +- doc/guides/vdpadevs/mlx5.rst | 87 +-- drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- drivers/common/mlx5/mlx5_common.c | 31 +- drivers/common/mlx5/mlx5_common_defs.h | 8 +- drivers/net/mlx5/linux/mlx5_verbs.c | 2 +- drivers/net/mlx5/mlx5_devx.c | 2 +- 13 files changed, 747 insertions(+), 720 deletions(-) create mode 100644 doc/guides/platform/mlx5.rst -- 2.25.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH v2 1/5] doc: remove obsolete explanations from mlx5 guide 2022-02-23 13:48 ` [PATCH v2 0/5] Refactor " Michael Baum @ 2022-02-23 13:48 ` Michael Baum 2022-02-23 13:48 ` [PATCH v2 2/5] doc: replace broken links in " Michael Baum ` (4 subsequent siblings) 5 siblings, 0 replies; 17+ messages in thread From: Michael Baum @ 2022-02-23 13:48 UTC (permalink / raw) To: dev; +Cc: Matan Azrad, Raslan Darawsheh, Viacheslav Ovsiienko, stable Vectorized routines were removed in result of Tx datapath refactoring, and devarg keys documentation was updated. However, more updating should have been done. In environment variables doc, there was explanation according to vectorized Tx which isn't relevant anymore. This patch removes this irrelevant explanation. Fixes: a6bd4911ad93 ("net/mlx5: remove Tx implementation") Cc: stable@dpdk.org Signed-off-by: Michael Baum <michaelba@nvidia.com> Reviewed-by: Raslan Darawsheh <rasland@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> --- doc/guides/nics/mlx5.rst | 9 --------- 1 file changed, 9 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 6494f4ae39..c21df81717 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -557,15 +557,6 @@ Environment variables The register would be flushed to HW usually when the write-combining buffer becomes full, but it depends on CPU design. - Except for vectorized Tx burst routines, a write memory barrier is enforced - after updating the register so that the update can be immediately visible to - HW. - - When vectorized Tx burst is called, the barrier is set only if the burst size - is not aligned to MLX5_VPMD_TX_MAX_BURST. However, setting this environmental - variable will bring better latency even though the maximum throughput can - slightly decline. - Run-time configuration ~~~~~~~~~~~~~~~~~~~~~~ -- 2.25.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH v2 2/5] doc: replace broken links in mlx5 guide 2022-02-23 13:48 ` [PATCH v2 0/5] Refactor " Michael Baum 2022-02-23 13:48 ` [PATCH v2 1/5] doc: remove obsolete explanations from mlx5 guide Michael Baum @ 2022-02-23 13:48 ` Michael Baum 2022-02-23 13:48 ` [PATCH v2 3/5] doc: correct name of BlueField-2 " Michael Baum ` (3 subsequent siblings) 5 siblings, 0 replies; 17+ messages in thread From: Michael Baum @ 2022-02-23 13:48 UTC (permalink / raw) To: dev; +Cc: Matan Azrad, Raslan Darawsheh, Viacheslav Ovsiienko Update links in both mlx4 and mlx5 doc. Signed-off-by: Michael Baum <michaelba@nvidia.com> Reviewed-by: Raslan Darawsheh <rasland@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> --- doc/guides/nics/mlx4.rst | 4 ++-- doc/guides/nics/mlx5.rst | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/guides/nics/mlx4.rst b/doc/guides/nics/mlx4.rst index c81105730d..66493a1157 100644 --- a/doc/guides/nics/mlx4.rst +++ b/doc/guides/nics/mlx4.rst @@ -14,7 +14,7 @@ the `Mellanox website <http://www.mellanox.com>`_. Help is also provided by the `Mellanox community <http://community.mellanox.com/welcome>`_. There is also a `section dedicated to this poll mode driver -<http://www.mellanox.com/page/products_dyn?product_family=209&mtag=pmd_for_dpdk>`_. +<https://developer.nvidia.com/networking/dpdk>`_. Implementation details @@ -219,7 +219,7 @@ Mellanox OFED as a fallback - `Mellanox OFED`_ version: **4.4, 4.5, 4.6**. - firmware version: **2.42.5000** and above. -.. _`Mellanox OFED`: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers +.. _`Mellanox OFED`: https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ .. note:: diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index c21df81717..a15f1620ff 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -19,7 +19,7 @@ Information and documentation about these adapters can be found on the `Mellanox community <http://community.mellanox.com/welcome>`__. There is also a `section dedicated to this poll mode driver -<http://www.mellanox.com/page/products_dyn?product_family=209&mtag=pmd_for_dpdk>`__. +<https://developer.nvidia.com/networking/dpdk>`_. Design @@ -1381,9 +1381,9 @@ managers on most distributions, this PMD requires Ethernet extensions that may not be supported at the moment (this is a work in progress). `Mellanox OFED -<http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux>`__ and +<https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/>`__ and `Mellanox EN -<http://www.mellanox.com/page/products_dyn?product_family=27&mtag=linux>`__ +<https://network.nvidia.com/products/ethernet-drivers/linux/mlnx_en/>`__ include the necessary support and should be used in the meantime. For DPDK, only libibverbs, libmlx5, mlnx-ofed-kernel packages and firmware updates are required from that distribution. -- 2.25.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH v2 3/5] doc: correct name of BlueField-2 in mlx5 guide 2022-02-23 13:48 ` [PATCH v2 0/5] Refactor " Michael Baum 2022-02-23 13:48 ` [PATCH v2 1/5] doc: remove obsolete explanations from mlx5 guide Michael Baum 2022-02-23 13:48 ` [PATCH v2 2/5] doc: replace broken links in " Michael Baum @ 2022-02-23 13:48 ` Michael Baum 2022-02-23 13:48 ` [PATCH v2 4/5] doc: add shared guide for mlx5 PMDs Michael Baum ` (2 subsequent siblings) 5 siblings, 0 replies; 17+ messages in thread From: Michael Baum @ 2022-02-23 13:48 UTC (permalink / raw) To: dev; +Cc: Matan Azrad, Raslan Darawsheh, Viacheslav Ovsiienko Update "BlueField 2" -> "BlueField-2" in mlx5 docs. Signed-off-by: Michael Baum <michaelba@nvidia.com> Reviewed-by: Raslan Darawsheh <rasland@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> --- doc/guides/compressdevs/mlx5.rst | 6 +++--- doc/guides/cryptodevs/mlx5.rst | 6 +++--- doc/guides/nics/mlx5.rst | 2 +- doc/guides/regexdevs/mlx5.rst | 8 ++++---- 4 files changed, 11 insertions(+), 11 deletions(-) diff --git a/doc/guides/compressdevs/mlx5.rst b/doc/guides/compressdevs/mlx5.rst index a4e17f65b3..7f2d6bdfff 100644 --- a/doc/guides/compressdevs/mlx5.rst +++ b/doc/guides/compressdevs/mlx5.rst @@ -7,7 +7,7 @@ MLX5 compress driver ==================== The MLX5 compress driver library -(**librte_compress_mlx5**) provides support for **Mellanox BlueField 2** +(**librte_compress_mlx5**) provides support for **Mellanox BlueField-2** families of 25/50/100/200 Gb/s adapters. Design @@ -21,7 +21,7 @@ These engines are part of the ARM complex of the BlueField chip, and as such they do not use NIC related resources (e.g. RX/TX bandwidth). They do share the same PCI and memory bandwidth. -So, using the BlueField device (starting from BlueField 2), the compress +So, using the BlueField device (starting from BlueField-2), the compress class operations can be supported in parallel to the net, vDPA and RegEx class operations. @@ -95,7 +95,7 @@ Driver options Supported NICs -------------- -* Mellanox\ |reg| BlueField 2 SmartNIC +* Mellanox\ |reg| BlueField-2 SmartNIC Prerequisites ------------- diff --git a/doc/guides/cryptodevs/mlx5.rst b/doc/guides/cryptodevs/mlx5.rst index e86a6205e8..9936556cc9 100644 --- a/doc/guides/cryptodevs/mlx5.rst +++ b/doc/guides/cryptodevs/mlx5.rst @@ -88,7 +88,7 @@ The mlxreg dedicated tool should be used as follows: should not be specified. All the device ports should set it in order to move to operational mode. - For BlueField 2, the internal ports in the ARM system should also be set. + For BlueField-2, the internal ports in the ARM system should also be set. - Query CRYPTO_OPERATIONAL register to make sure the device is in Operational mode. @@ -142,7 +142,7 @@ Supported NICs -------------- * Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G) -* Mellanox\ |reg| BlueField 2 SmartNIC +* Mellanox\ |reg| BlueField-2 SmartNIC * Mellanox\ |reg| ConnectX\ |reg|-6 Dx @@ -162,7 +162,7 @@ FW Prerequisites ~~~~~~~~~~~~~~~~ - xx.31.0328 for ConnectX-6. -- xx.32.0108 for ConnectX-6 Dx and BlueField 2. +- xx.32.0108 for ConnectX-6 Dx and BlueField-2. Linux Prerequisites ~~~~~~~~~~~~~~~~~~~ diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index a15f1620ff..9e10a68d6f 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -253,7 +253,7 @@ Limitations - Flex item: - - Hardware support: BlueField 2. + - Hardware support: BlueField-2. - Flex item is supported on PF only. - Hardware limits ``header_length_mask_width`` up to 6 bits. - Firmware supports 8 global sample fields. diff --git a/doc/guides/regexdevs/mlx5.rst b/doc/guides/regexdevs/mlx5.rst index b2bf0afd01..fc2cacba60 100644 --- a/doc/guides/regexdevs/mlx5.rst +++ b/doc/guides/regexdevs/mlx5.rst @@ -7,7 +7,7 @@ MLX5 RegEx driver ================= The MLX5 RegEx (Regular Expression) driver library -(**librte_regex_mlx5**) provides support for **Mellanox BlueField 2** +(**librte_regex_mlx5**) provides support for **Mellanox BlueField-2** families of 25/50/100/200 Gb/s adapters. Design @@ -43,13 +43,13 @@ Features Supported NICs -------------- -* Mellanox\ |reg| BlueField 2 SmartNIC +* Mellanox\ |reg| BlueField-2 SmartNIC Prerequisites ------------- -- BlueField 2 running Mellanox supported kernel. -- Enable the RegEx capabilities using system call from the BlueField 2. +- BlueField-2 running Mellanox supported kernel. +- Enable the RegEx capabilities using system call from the BlueField-2. - Official support is not yet released. Limitations -- 2.25.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH v2 4/5] doc: add shared guide for mlx5 PMDs 2022-02-23 13:48 ` [PATCH v2 0/5] Refactor " Michael Baum ` (2 preceding siblings ...) 2022-02-23 13:48 ` [PATCH v2 3/5] doc: correct name of BlueField-2 " Michael Baum @ 2022-02-23 13:48 ` Michael Baum 2022-02-23 13:48 ` [PATCH v2 5/5] doc: update doorbell mapping parameter name in mlx5 guide Michael Baum 2022-02-23 14:35 ` [PATCH v2 0/5] Refactor mlx5 guides Raslan Darawsheh 5 siblings, 0 replies; 17+ messages in thread From: Michael Baum @ 2022-02-23 13:48 UTC (permalink / raw) To: dev; +Cc: Matan Azrad, Raslan Darawsheh, Viacheslav Ovsiienko Adds new documentation for MLX5 common driver that contains: - Its features list (doesn't exist for now). - Its devargs description. - Device configuration information and tutorial. - Quick Start Guide for Mellanox OFED/EN. Move into this doc all shared information from other MLX5 PMD docs and add them reference to new common doc. Signed-off-by: Michael Baum <michaelba@nvidia.com> Reviewed-by: Raslan Darawsheh <rasland@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> --- doc/guides/compressdevs/mlx5.rst | 34 +- doc/guides/cryptodevs/mlx5.rst | 32 +- doc/guides/nics/mlx5.rst | 522 ++------------------------- doc/guides/platform/index.rst | 1 + doc/guides/platform/mlx5.rst | 602 +++++++++++++++++++++++++++++++ doc/guides/regexdevs/mlx5.rst | 36 +- doc/guides/vdpadevs/mlx5.rst | 87 +---- 7 files changed, 662 insertions(+), 652 deletions(-) create mode 100644 doc/guides/platform/mlx5.rst diff --git a/doc/guides/compressdevs/mlx5.rst b/doc/guides/compressdevs/mlx5.rst index 7f2d6bdfff..273251e088 100644 --- a/doc/guides/compressdevs/mlx5.rst +++ b/doc/guides/compressdevs/mlx5.rst @@ -3,10 +3,10 @@ .. include:: <isonum.txt> -MLX5 compress driver +MLX5 Compress Driver ==================== -The MLX5 compress driver library +The mlx5 compress driver library (**librte_compress_mlx5**) provides support for **Mellanox BlueField-2** families of 25/50/100/200 Gb/s adapters. @@ -25,30 +25,7 @@ So, using the BlueField device (starting from BlueField-2), the compress class operations can be supported in parallel to the net, vDPA and RegEx class operations. -For security reasons and robustness, this driver only deals with virtual -memory addresses. The way resources allocations are handled by the kernel, -combined with hardware specifications that allow to handle virtual memory -addresses directly, ensure that DPDK applications cannot access random -physical memory (or memory that does not belong to the current process). - -The PMD uses libibverbs and libmlx5 to access the device firmware -or directly the hardware components. -There are different levels of objects and bypassing abilities -to get the best performances: - -- Verbs is a complete high-level generic API. -- Direct Verbs is a device-specific API. -- DevX allows to access firmware objects. - -Enabling librte_compress_mlx5 causes DPDK applications to be linked against -libibverbs. - -Mellanox mlx5 PCI device can be probed by number of different PCI devices, -for example net / vDPA / RegEx. To select the compress PMD ``class=compress`` -should be specified as device parameter. The compress device can be probed and -used with other Mellanox classes, by adding more options in the class. -For example: ``class=net:compress`` will probe both the net PMD and the compress -PMD. +See :doc:`../../platform/mlx5` guide for more design details. Features -------- @@ -85,6 +62,9 @@ Limitations Driver options -------------- +Please refer to :ref:`mlx5 common options <mlx5_common_driver_options>` +for an additional list of options shared with other mlx5 drivers. + - ``log-block-size`` parameter [int] Log of the Huffman block size in the Deflate algorithm. @@ -101,4 +81,4 @@ Prerequisites ------------- - Mellanox OFED version: **5.2** - see :doc:`../../nics/mlx5` guide for more Mellanox OFED details. + See :ref:`mlx5 common prerequisites <mlx5_linux_prerequisites>` for more details. diff --git a/doc/guides/cryptodevs/mlx5.rst b/doc/guides/cryptodevs/mlx5.rst index 9936556cc9..ef47aa65dd 100644 --- a/doc/guides/cryptodevs/mlx5.rst +++ b/doc/guides/cryptodevs/mlx5.rst @@ -28,23 +28,12 @@ when the MKEY is configured to perform crypto operations. The encryption does not require text to be aligned to the AES block size (128b). -For security reasons and to increase robustness, this driver only deals with virtual -memory addresses. The way resources allocations are handled by the kernel, -combined with hardware specifications that allow handling virtual memory -addresses directly, ensure that DPDK applications cannot access random -physical memory (or memory that does not belong to the current process). +See :doc:`../../platform/mlx5` guide for more design details. -The PMD uses ``libibverbs`` and ``libmlx5`` to access the device firmware -or to access the hardware components directly. -There are different levels of objects and bypassing abilities. -To get the best performances: - -- Verbs is a complete high-level generic API (Linux only). -- Direct Verbs is a device-specific API (Linux only). -- DevX allows to access firmware objects. +Configuration +------------- -Enabling ``librte_crypto_mlx5`` causes DPDK applications -to be linked against libibverbs on Linux OS. +See the :ref:`mlx5 common configuration <mlx5_common_env>`. In order to move the device to crypto operational mode, credential and KEK (Key Encrypting Key) should be set as the first step. @@ -109,10 +98,8 @@ The mlxreg dedicated tool should be used as follows: Driver options -------------- -- ``class`` parameter [string] - - Select the class of the driver that should probe the device. - `crypto` for the mlx5 crypto driver. +Please refer to :ref:`mlx5 common options <mlx5_common_driver_options>` +for an additional list of options shared with other mlx5 drivers. - ``wcs_file`` parameter [string] - mandatory @@ -168,13 +155,12 @@ Linux Prerequisites ~~~~~~~~~~~~~~~~~~~ - Mellanox OFED version: **5.3**. - see :doc:`../../nics/mlx5` guide for more Mellanox OFED details. - - Compilation can be done also with rdma-core v15+. - see :doc:`../../nics/mlx5` guide for more rdma-core details. + + See :ref:`mlx5 common prerequisites <mlx5_linux_prerequisites>` for more details. Windows Prerequisites ~~~~~~~~~~~~~~~~~~~~~ - Mellanox WINOF-2 version: **2.60** or higher. - see :doc:`../../nics/mlx5` guide for more Mellanox WINOF-2 details. + See :ref:`mlx5 common prerequisites <mlx5_windows_prerequisites>` for more details. diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 9e10a68d6f..f94ed90ef0 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -4,23 +4,16 @@ .. include:: <isonum.txt> -MLX5 poll mode driver -===================== +MLX5 Ethernet Poll Mode Driver +============================== -The MLX5 poll mode driver library (**librte_net_mlx5**) provides support +The mlx5 Ethernet poll mode driver library (**librte_net_mlx5**) provides support for **Mellanox ConnectX-4**, **Mellanox ConnectX-4 Lx** , **Mellanox ConnectX-5**, **Mellanox ConnectX-6**, **Mellanox ConnectX-6 Dx**, **Mellanox ConnectX-6 Lx**, **Mellanox BlueField** and **Mellanox BlueField-2** families of 10/25/40/50/100/200 Gb/s adapters as well as their virtual functions (VF) in SR-IOV context. -Information and documentation about these adapters can be found on the -`Mellanox website <http://www.mellanox.com>`__. Help is also provided by the -`Mellanox community <http://community.mellanox.com/welcome>`__. - -There is also a `section dedicated to this poll mode driver -<https://developer.nvidia.com/networking/dpdk>`_. - Design ------ @@ -29,12 +22,6 @@ Besides its dependency on libibverbs (that implies libmlx5 and associated kernel support), librte_net_mlx5 relies heavily on system calls for control operations such as querying/updating the MTU and flow control parameters. -For security reasons and robustness, this driver only deals with virtual -memory addresses. The way resources allocations are handled by the kernel, -combined with hardware specifications that allow to handle virtual memory -addresses directly, ensure that DPDK applications cannot access random -physical memory (or memory that does not belong to the current process). - This capability allows the PMD to coexist with kernel network interfaces which remain functional, although they stop receiving unicast packets as long as they share the same MAC address. @@ -42,18 +29,7 @@ This means legacy linux control tools (for example: ethtool, ifconfig and more) can operate on the same network interfaces that owned by the DPDK application. -The PMD can use libibverbs and libmlx5 to access the device firmware -or directly the hardware components. -There are different levels of objects and bypassing abilities -to get the best performances: - -- Verbs is a complete high-level generic API -- Direct Verbs is a device-specific API -- DevX allows to access firmware objects -- Direct Rules manages flow steering at low-level hardware layer - -Enabling librte_net_mlx5 causes DPDK applications to be linked against -libibverbs. +See :doc:`../../platform/mlx5` guide for more design details. Features -------- @@ -522,75 +498,31 @@ Extended statistics can be queried using ``rte_eth_xstats_get()``. The extended Finally per-flow statistics can by queried using ``rte_flow_query`` when attaching a count action for specific flow. The flow counter counts the number of packets received successfully by the port and match the specific flow. -Configuration -------------- - -Compilation options -~~~~~~~~~~~~~~~~~~~ - -The ibverbs libraries can be linked with this PMD in a number of ways, -configured by the ``ibverbs_link`` build option: - -- ``shared`` (default): the PMD depends on some .so files. - -- ``dlopen``: Split the dependencies glue in a separate library - loaded when needed by dlopen. - It make dependencies on libibverbs and libmlx4 optional, - and has no performance impact. - -- ``static``: Embed static flavor of the dependencies libibverbs and libmlx4 - in the PMD shared library or the executable static binary. -Environment variables -~~~~~~~~~~~~~~~~~~~~~ +Compilation +----------- -- ``MLX5_GLUE_PATH`` +See :ref:`mlx5 common compilation <mlx5_common_compilation>`. - A list of directories in which to search for the rdma-core "glue" plug-in, - separated by colons or semi-colons. -- ``MLX5_SHUT_UP_BF`` +Configuration +------------- - Configures HW Tx doorbell register as IO-mapped. +Environment Configuration +~~~~~~~~~~~~~~~~~~~~~~~~~ - By default, the HW Tx doorbell is configured as a write-combining register. - The register would be flushed to HW usually when the write-combining buffer - becomes full, but it depends on CPU design. +See :ref:`mlx5 common configuration <mlx5_common_env>`. -Run-time configuration +Firmware configuration ~~~~~~~~~~~~~~~~~~~~~~ -- librte_net_mlx5 brings kernel network interfaces up during initialization - because it is affected by their state. Forcing them down prevents packets - reception. - -- **ethtool** operations on related kernel interfaces also affect the PMD. - -Run as non-root -^^^^^^^^^^^^^^^ - -In order to run as a non-root user, -some capabilities must be granted to the application:: - - setcap cap_sys_admin,cap_net_admin,cap_net_raw,cap_ipc_lock+ep <dpdk-app> - -Below are the reasons of the need for each capability: - -``cap_sys_admin`` - When using physical addresses (PA mode), with Linux >= 4.0, - for access to ``/proc/self/pagemap``. - -``cap_net_admin`` - For device configuration. - -``cap_net_raw`` - For raw ethernet queue allocation through kernel driver. - -``cap_ipc_lock`` - For DMA memory pinning. +See :ref:`mlx5_firmware_config` guide. Driver options -^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~ + +Please refer to :ref:`mlx5 common options <mlx5_common_driver_options>` +for an additional list of options shared with other mlx5 drivers. - ``rxq_cqe_comp_en`` parameter [int] @@ -1059,30 +991,6 @@ Driver options Disabled by default (set to 0). -- ``mr_ext_memseg_en`` parameter [int] - - A nonzero value enables extending memseg when registering DMA memory. If - enabled, the number of entries in MR (Memory Region) lookup table on datapath - is minimized and it benefits performance. On the other hand, it worsens memory - utilization because registered memory is pinned by kernel driver. Even if a - page in the extended chunk is freed, that doesn't become reusable until the - entire memory is freed. - - Enabled by default. - -- ``mr_mempool_reg_en`` parameter [int] - - A nonzero value enables implicit registration of DMA memory of all mempools - except those having ``RTE_MEMPOOL_F_NON_IO``. This flag is set automatically - for mempools populated with non-contiguous objects or those without IOVA. - The effect is that when a packet from a mempool is transmitted, - its memory is already registered for DMA in the PMD and no registration - will happen on the data path. The tradeoff is extra work on the creation - of each mempool and increased HW resource use if some mempools - are not used with MLX5 devices. - - Enabled by default. - - ``representor`` parameter [list] This parameter can be used to instantiate DPDK Ethernet devices from @@ -1153,13 +1061,6 @@ Driver options By default, the PMD will set this value to 0. -- ``sys_mem_en`` parameter [int] - - A non-zero value enables the PMD memory management allocating memory - from system by default, without explicit rte memory flag. - - By default, the PMD will set this value to 0. - - ``decap_en`` parameter [int] Some devices do not support FCS (frame checksum) scattering for @@ -1183,253 +1084,6 @@ Driver options By default, the PMD will set this value to 1. -.. _mlx5_firmware_config: - -Firmware configuration -~~~~~~~~~~~~~~~~~~~~~~ - -Firmware features can be configured as key/value pairs. - -The command to set a value is:: - - mlxconfig -d <device> set <key>=<value> - -The command to query a value is:: - - mlxconfig -d <device> query | grep <key> - -The device name for the command ``mlxconfig`` can be either the PCI address, -or the mst device name found with:: - - mst status - -Below are some firmware configurations listed. - -- link type:: - - LINK_TYPE_P1 - LINK_TYPE_P2 - value: 1=Infiniband 2=Ethernet 3=VPI(auto-sense) - -- enable SR-IOV:: - - SRIOV_EN=1 - -- maximum number of SR-IOV virtual functions:: - - NUM_OF_VFS=<max> - -- enable DevX (required by Direct Rules and other features):: - - UCTX_EN=1 - -- aggressive CQE zipping:: - - CQE_COMPRESSION=1 - -- L3 VXLAN and VXLAN-GPE destination UDP port:: - - IP_OVER_VXLAN_EN=1 - IP_OVER_VXLAN_PORT=<udp dport> - -- enable VXLAN-GPE tunnel flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=0 - or - FLEX_PARSER_PROFILE_ENABLE=2 - -- enable IP-in-IP tunnel flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=0 - -- enable MPLS flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=1 - -- enable ICMP(code/type/identifier/sequence number) / ICMP6(code/type) fields matching:: - - FLEX_PARSER_PROFILE_ENABLE=2 - -- enable Geneve flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=0 - or - FLEX_PARSER_PROFILE_ENABLE=1 - -- enable Geneve TLV option flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=0 - -- enable GTP flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=3 - -- enable eCPRI flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=4 - PROG_PARSE_GRAPH=1 - -- enable dynamic flex parser for flex item:: - - FLEX_PARSER_PROFILE_ENABLE=4 - PROG_PARSE_GRAPH=1 - -- enable realtime timestamp format:: - - REAL_TIME_CLOCK_ENABLE=1 - -Linux Prerequisites -------------------- - -This driver relies on external libraries and kernel drivers for resources -allocations and initialization. The following dependencies are not part of -DPDK and must be installed separately: - -- **libibverbs** - - User space Verbs framework used by librte_net_mlx5. This library provides - a generic interface between the kernel and low-level user space drivers - such as libmlx5. - - It allows slow and privileged operations (context initialization, hardware - resources allocations) to be managed by the kernel and fast operations to - never leave user space. - -- **libmlx5** - - Low-level user space driver library for Mellanox - ConnectX-4/ConnectX-5/ConnectX-6/BlueField devices, it is automatically loaded - by libibverbs. - - This library basically implements send/receive calls to the hardware - queues. - -- **Kernel modules** - - They provide the kernel-side Verbs API and low level device drivers that - manage actual hardware initialization and resources sharing with user - space processes. - - Unlike most other PMDs, these modules must remain loaded and bound to - their devices: - - - mlx5_core: hardware driver managing Mellanox - ConnectX-4/ConnectX-5/ConnectX-6/BlueField devices and related Ethernet kernel - network devices. - - mlx5_ib: InfiniBand device driver. - - ib_uverbs: user space driver for Verbs (entry point for libibverbs). - -- **Firmware update** - - Mellanox OFED/EN releases include firmware updates for - ConnectX-4/ConnectX-5/ConnectX-6/BlueField adapters. - - Because each release provides new features, these updates must be applied to - match the kernel modules and libraries they come with. - -.. note:: - - Both libraries are BSD and GPL licensed. Linux kernel modules are GPL - licensed. - -Installation -~~~~~~~~~~~~ - -Either RDMA Core library with a recent enough Linux kernel release -(recommended) or Mellanox OFED/EN, which provides compatibility with older -releases. - -RDMA Core with Linux Kernel -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -- Minimal kernel version : v4.14 or the most recent 4.14-rc (see `Linux installation documentation`_) -- Minimal rdma-core version: v15+ commit 0c5f5765213a ("Merge pull request #227 from yishaih/tm") - (see `RDMA Core installation documentation`_) -- When building for i686 use: - - - rdma-core version 18.0 or above built with 32bit support. - - Kernel version 4.14.41 or above. - -- Starting with rdma-core v21, static libraries can be built:: - - cd build - CFLAGS=-fPIC cmake -DIN_PLACE=1 -DENABLE_STATIC=1 -GNinja .. - ninja - -.. _`Linux installation documentation`: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/plain/Documentation/admin-guide/README.rst -.. _`RDMA Core installation documentation`: https://raw.githubusercontent.com/linux-rdma/rdma-core/master/README.md - - -Mellanox OFED/EN -^^^^^^^^^^^^^^^^ - -- Mellanox OFED version: **4.5** and above / - Mellanox EN version: **4.5** and above -- firmware version: - - - ConnectX-4: **12.21.1000** and above. - - ConnectX-4 Lx: **14.21.1000** and above. - - ConnectX-5: **16.21.1000** and above. - - ConnectX-5 Ex: **16.21.1000** and above. - - ConnectX-6: **20.27.0090** and above. - - ConnectX-6 Dx: **22.27.0090** and above. - - BlueField: **18.25.1010** and above. - -While these libraries and kernel modules are available on OpenFabrics -Alliance's `website <https://www.openfabrics.org/>`__ and provided by package -managers on most distributions, this PMD requires Ethernet extensions that -may not be supported at the moment (this is a work in progress). - -`Mellanox OFED -<https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/>`__ and -`Mellanox EN -<https://network.nvidia.com/products/ethernet-drivers/linux/mlnx_en/>`__ -include the necessary support and should be used in the meantime. For DPDK, -only libibverbs, libmlx5, mlnx-ofed-kernel packages and firmware updates are -required from that distribution. - -.. note:: - - Several versions of Mellanox OFED/EN are available. Installing the version - this DPDK release was developed and tested against is strongly - recommended. Please check the `linux prerequisites`_. - -Windows Prerequisites ---------------------- - -This driver relies on external libraries and kernel drivers for resources -allocations and initialization. The dependencies in the following sub-sections -are not part of DPDK, and must be installed separately. - -Compilation Prerequisites -~~~~~~~~~~~~~~~~~~~~~~~~~ - -DevX SDK installation -^^^^^^^^^^^^^^^^^^^^^ - -The DevX SDK must be installed on the machine building the Windows PMD. -Additional information can be found at -`How to Integrate Windows DevX in Your Development Environment -<https://docs.mellanox.com/display/winof2v250/RShim+Drivers+and+Usage#RShimDriversandUsage-DevXInterface>`__. - -Runtime Prerequisites -~~~~~~~~~~~~~~~~~~~~~ - -WinOF2 version 2.60 or higher must be installed on the machine. - -WinOF2 installation -^^^^^^^^^^^^^^^^^^^ - -The driver can be downloaded from the following site: -`WINOF2 -<https://www.mellanox.com/products/adapter-software/ethernet/windows/winof-2>`__ - -DevX Enablement -^^^^^^^^^^^^^^^ - -DevX for Windows must be enabled in the Windows registry. -The keys ``DevxEnabled`` and ``DevxFsRules`` must be set. -Additional information can be found in the WinOF2 user manual. Supported NICs -------------- @@ -1475,149 +1129,21 @@ Below are detailed device names: * Mellanox\ |reg| ConnectX\ |reg|-6 Dx EN 200G MCX623105AN-VDAT (1x200G) * Mellanox\ |reg| ConnectX\ |reg|-6 Lx EN 25G MCX631102AN-ADAT (2x25G) -Quick Start Guide on OFED/EN ----------------------------- - -1. Download latest Mellanox OFED/EN. For more info check the `linux prerequisites`_. - - -2. Install the required libraries and kernel modules either by installing - only the required set, or by installing the entire Mellanox OFED/EN:: - - ./mlnxofedinstall --upstream-libs --dpdk - -3. Verify the firmware is the correct one:: - - ibv_devinfo - -4. Verify all ports links are set to Ethernet:: - - mlxconfig -d <mst device> query | grep LINK_TYPE - LINK_TYPE_P1 ETH(2) - LINK_TYPE_P2 ETH(2) - - Link types may have to be configured to Ethernet:: - mlxconfig -d <mst device> set LINK_TYPE_P1/2=1/2/3 +Sub-Function +------------ - * LINK_TYPE_P1=<1|2|3> , 1=Infiniband 2=Ethernet 3=VPI(auto-sense) - - For hypervisors, verify SR-IOV is enabled on the NIC:: - - mlxconfig -d <mst device> query | grep SRIOV_EN - SRIOV_EN True(1) - - If needed, configure SR-IOV:: - - mlxconfig -d <mst device> set SRIOV_EN=1 NUM_OF_VFS=16 - mlxfwreset -d <mst device> reset - -5. Restart the driver:: - - /etc/init.d/openibd restart - - or:: - - service openibd restart - - If link type was changed, firmware must be reset as well:: - - mlxfwreset -d <mst device> reset - - For hypervisors, after reset write the sysfs number of virtual functions - needed for the PF. - - To dynamically instantiate a given number of virtual functions (VFs):: - - echo [num_vfs] > /sys/class/infiniband/mlx5_0/device/sriov_numvfs - -6. Install DPDK and you are ready to go. - See :doc:`compilation instructions <../linux_gsg/build_dpdk>`. - -Enable switchdev mode ---------------------- - -Switchdev mode is a mode in E-Switch, that binds between representor and VF or SF. -Representor is a port in DPDK that is connected to a VF or SF in such a way -that assuming there are no offload flows, each packet that is sent from the VF or SF -will be received by the corresponding representor. While each packet that is or SF -sent to a representor will be received by the VF or SF. -This is very useful in case of SRIOV mode, where the first packet that is sent -by the VF or SF will be received by the DPDK application which will decide if this -flow should be offloaded to the E-Switch. After offloading the flow packet -that the VF or SF that are matching the flow will not be received any more by -the DPDK application. - -1. Enable SRIOV mode:: - - mlxconfig -d <mst device> set SRIOV_EN=true - -2. Configure the max number of VFs:: - - mlxconfig -d <mst device> set NUM_OF_VFS=<num of vfs> - -3. Reset the FW:: - - mlxfwreset -d <mst device> reset - -3. Configure the actual number of VFs:: - - echo <num of vfs > /sys/class/net/<net device>/device/sriov_numvfs - -4. Unbind the device (can be rebind after the switchdev mode):: - - echo -n "<device pci address" > /sys/bus/pci/drivers/mlx5_core/unbind - -5. Enable switchdev mode:: - - echo switchdev > /sys/class/net/<net device>/compat/devlink/mode - -Sub-Function support --------------------- - -Sub-Function is a portion of the PCI device, a SF netdev has its own -dedicated queues (txq, rxq). -A SF shares PCI level resources with other SFs and/or with its parent PCI function. - -0. Requirement:: - - OFED version >= 5.4-0.3.3.0 - -1. Configure SF feature:: - - # Run mlxconfig on both PFs on host and ECPFs on BlueField. - mlxconfig -d <mst device> set PER_PF_NUM_SF=1 PF_TOTAL_SF=252 PF_SF_BAR_SIZE=12 - -2. Enable switchdev mode:: - - mlxdevm dev eswitch set pci/<DBDF> mode switchdev - -3. Add SF port:: - - mlxdevm port add pci/<DBDF> flavour pcisf pfnum 0 sfnum <sfnum> - - Get SFID from output: pci/<DBDF>/<SFID> - -4. Modify MAC address:: - - mlxdevm port function set pci/<DBDF>/<SFID> hw_addr <MAC> - -5. Activate SF port:: - - mlxdevm port function set pci/<DBDF>/<ID> state active - -6. Devargs to probe SF device:: - - auxiliary:mlx5_core.sf.<num>,dv_flow_en=1 +See :ref:`mlx5_sub_function`. Sub-Function representor support --------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A SF netdev supports E-Switch representation offload similar to PF and VF representors. Use <sfnum> to probe SF representor:: - testpmd> port attach <PCI_BDF>,representor=sf<sfnum>,dv_flow_en=1 + testpmd> port attach <PCI_BDF>,representor=sf<sfnum>,dv_flow_en=1 + Performance tuning ------------------ diff --git a/doc/guides/platform/index.rst b/doc/guides/platform/index.rst index 2ff91a6018..b1ba11df73 100644 --- a/doc/guides/platform/index.rst +++ b/doc/guides/platform/index.rst @@ -14,4 +14,5 @@ The following are platform specific guides and setup information. cnxk dpaa dpaa2 + mlx5 octeontx diff --git a/doc/guides/platform/mlx5.rst b/doc/guides/platform/mlx5.rst new file mode 100644 index 0000000000..0fd5e6604d --- /dev/null +++ b/doc/guides/platform/mlx5.rst @@ -0,0 +1,602 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2022 6WIND S.A. + Copyright (c) 2022 NVIDIA Corporation & Affiliates + +.. include:: <isonum.txt> + +MLX5 Common Driver +================== + +The mlx5 common driver library (**librte_common_mlx5**) provides support for +**Mellanox ConnectX-4**, **Mellanox ConnectX-4 Lx**, **Mellanox ConnectX-5**, +**Mellanox ConnectX-6**, **Mellanox ConnectX-6 Dx**, **Mellanox ConnectX-6 Lx**, +**Mellanox BlueField** and **Mellanox BlueField-2** families of +10/25/40/50/100/200 Gb/s adapters. + +Information and documentation for these adapters can be found on the +`NVIDIA website <https://www.nvidia.com/en-us/networking/>`_. +Help is also provided by the +`Mellanox community <http://community.mellanox.com/welcome>`_. +In addition, there is a `web section dedicated to the Poll Mode Driver +<https://developer.nvidia.com/networking/dpdk>`_. + + +Design +------ + +For security reasons and to enhance robustness, +this driver only handles virtual memory addresses. +The way resources allocations are handled by the kernel, +combined with hardware specifications that allow handling virtual memory addresses directly, +ensure that DPDK applications cannot access random physical memory +(or memory that does not belong to the current process). + +There are different levels of objects and bypassing abilities +which are used to get the best performance: + +- **Verbs** is a complete high-level generic API +- **Direct Verbs** is a device-specific API +- **DevX** allows accessing firmware objects +- **Direct Rules** manages flow steering at the low-level hardware layer + +On Linux, above interfaces are provided by linking with `libibverbs` and `libmlx5`. +See :ref:`mlx5_linux_prerequisites` for installation. + +On Windows, DevX is the only requirement from the above list. +See :ref:`mlx5_windows_prerequisites` for DevX SDK package installation. + + +.. _mlx5_classes: + +Classes +------- + +One mlx5 device can be probed by a number of different PMDs. +To select a specific PMD, its name should be specified as a device parameter +(e.g. ``0000:08:00.1,class=eth``). + +In order to allow probing by multiple PMDs, +several classes may be listed separated by a colon. +For example: ``class=crypto:regex`` will probe both Crypto and RegEx PMDs. + + +Supported Classes +~~~~~~~~~~~~~~~~~ + +- ``class=compress`` for :doc:`../../compressdevs/mlx5`. +- ``class=crypto`` for :doc:`../../cryptodevs/mlx5`. +- ``class=eth`` for :doc:`../../nics/mlx5`. +- ``class=regex`` for :doc:`../../regexdevs/mlx5`. +- ``class=vdpa`` for :doc:`../../vdpadevs/mlx5`. + +By default, the mlx5 device will be probed by the ``eth`` PMD. + + +Limitations +~~~~~~~~~~~ + +- ``eth`` and ``vdpa`` PMDs cannot be probed at the same time. + All other combinations are possible. + +- On Windows, only ``eth`` and ``crypto`` are supported. + + +.. _mlx5_common_compilation: + +Compilation Prerequisites +------------------------- + +.. _mlx5_linux_prerequisites: + +Linux Prerequisites +~~~~~~~~~~~~~~~~~~~ + +This driver relies on external libraries and kernel drivers for resources +allocations and initialization. +The following dependencies are not part of DPDK and must be installed separately: + +- **libibverbs** + + User space Verbs framework used by ``librte_common_mlx5``. + This library provides a generic interface between the kernel + and low-level user space drivers such as ``libmlx5``. + + It allows slow and privileged operations (context initialization, + hardware resources allocations) to be managed by the kernel + and fast operations to never leave user space. + +- **libmlx5** + + Low-level user space driver library for Mellanox devices, + it is automatically loaded by ``libibverbs``. + + This library basically implements send/receive calls to the hardware queues. + +- **Kernel modules** + + They provide the kernel-side Verbs API and low level device drivers + that manage actual hardware initialization + and resources sharing with user-space processes. + + Unlike most other PMDs, these modules must remain loaded and bound to + their devices: + + - ``mlx5_core``: hardware driver managing Mellanox devices + and related Ethernet kernel network devices. + - ``mlx5_ib``: InfiniBand device driver. + - ``ib_uverbs``: user space driver for Verbs (entry point for ``libibverbs``). + +- **Firmware update** + + Mellanox OFED/EN releases include firmware updates. + + Because each release provides new features, these updates must be applied to + match the kernel modules and libraries they come with. + +Libraries and kernel modules can be provided either by the Linux distribution, +or by installing Mellanox OFED/EN which provides compatibility with older kernels. + + +Upstream Dependencies +^^^^^^^^^^^^^^^^^^^^^ + +The mlx5 kernel modules are part of upstream Linux. +The minimal supported kernel version is 4.14. +For 32-bit, version 4.14.41 or above is required. + +The libraries `libibverbs` and `libmlx5` are part of ``rdma-core``. +It is packaged by most of Linux distributions. +The minimal supported rdma-core version is 16. +For 32-bit, version 18 or above is required. + +The rdma-core sources can be downloaded at +https://github.com/linux-rdma/rdma-core + +It is possible to build rdma-core as static libraries starting with version 21:: + + cd build + CFLAGS=-fPIC cmake -DIN_PLACE=1 -DENABLE_STATIC=1 -GNinja .. + ninja + + +Mellanox OFED/EN +^^^^^^^^^^^^^^^^ + +The kernel modules and libraries are packaged with other tools +in Mellanox OFED or Mellanox EN. +The minimal supported versions are: + +- Mellanox OFED version: **4.5** and above. +- Mellanox EN version: **4.5** and above. +- Firmware version: + + - ConnectX-4: **12.21.1000** and above. + - ConnectX-4 Lx: **14.21.1000** and above. + - ConnectX-5: **16.21.1000** and above. + - ConnectX-5 Ex: **16.21.1000** and above. + - ConnectX-6: **20.27.0090** and above. + - ConnectX-6 Dx: **22.27.0090** and above. + - BlueField: **18.25.1010** and above. + - BlueField-2: **24.28.1002** and above. + +The firmware, the libraries libibverbs, libmlx5, and mlnx-ofed-kernel modules +are packaged in `Mellanox OFED +<https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/>`_. +After downloading, it can be installed with this command:: + + ./mlnxofedinstall --dpdk + +`Mellanox EN +<https://network.nvidia.com/products/ethernet-drivers/linux/mlnx_en/>`_ +is a smaller package including what is needed for DPDK. +After downloading, it can be installed with this command:: + + ./install --dpdk + +After installing, the firmware version can be checked:: + + ibv_devinfo + +.. note:: + + Several versions of Mellanox OFED/EN are available. Installing the version + this DPDK release was developed and tested against is strongly recommended. + Please check the "Tested Platforms" section in the :doc:`../../rel_notes/index`. + + +.. _mlx5_windows_prerequisites: + +Windows Prerequisites +~~~~~~~~~~~~~~~~~~~~~ + +The mlx5 PMDs rely on external libraries and kernel drivers +for resource allocation and initialization. + + +DevX SDK Installation +^^^^^^^^^^^^^^^^^^^^^ + +The DevX SDK must be installed on the machine building the Windows PMD. +Additional information can be found at +`How to Integrate Windows DevX in Your Development Environment +<https://docs.nvidia.com/networking/display/winof2v260/RShim+Drivers+and+Usage#RShimDriversandUsage-DevXInterface>`_. +The minimal supported WinOF2 version is 2.60. + + +Compilation Options +------------------- + +Compilation on Linux +~~~~~~~~~~~~~~~~~~~~ + +The ibverbs libraries can be linked with this PMD in a number of ways, +configured by the ``ibverbs_link`` build option: + +``shared`` (default) + The PMD depends on some .so files. + +``dlopen`` + Split the dependencies glue in a separate library + loaded when needed by dlopen (see ``MLX5_GLUE_PATH``). + It makes dependencies on libibverbs and libmlx5 optional, + and has no performance impact. + +``static`` + Embed static flavor of the dependencies libibverbs and libmlx5 + in the PMD shared library or the executable static binary. + + +Compilation on Windows +~~~~~~~~~~~~~~~~~~~~~~ + +The DevX SDK location must be set through two environment variables: + +``DEVX_LIB_PATH`` + path to the DevX lib file. + +``DEVX_INC_PATH`` + path to the DevX header files. + + +.. _mlx5_common_env: + +Environment Configuration +------------------------- + +Linux Environment +~~~~~~~~~~~~~~~~~ + +The kernel network interfaces are brought up during initialization. +Forcing them down prevents packets reception. + +The ethtool operations on the kernel interfaces may also affect the PMD. + +Some runtime behaviours may be configured through environment variables. + +``MLX5_GLUE_PATH`` + If built with ``ibverbs_link=dlopen``, + list of directories in which to search for the rdma-core "glue" plug-in, + separated by colons or semi-colons. + +``MLX5_SHUT_UP_BF`` + If Verbs is used (DevX disabled), + HW queue doorbell register mapping. + The value 0 means non-cached IO mapping, + while 1 is a regular memory mapping. + + With regular memory mapping, the register is flushed to HW + usually when the write-combining buffer becomes full, + but it depends on CPU design. + + +Port Link with OFED/EN +^^^^^^^^^^^^^^^^^^^^^^ + +Ports links must be set to Ethernet:: + + mlxconfig -d <mst device> query | grep LINK_TYPE + LINK_TYPE_P1 ETH(2) + LINK_TYPE_P2 ETH(2) + + mlxconfig -d <mst device> set LINK_TYPE_P1/2=1/2/3 + +Link type values are: + +* ``1`` Infiniband +* ``2`` Ethernet +* ``3`` VPI (auto-sense) + +If link type was changed, firmware must be reset as well:: + + mlxfwreset -d <mst device> reset + + +.. _mlx5_vf: + +SR-IOV Virtual Function with OFED/EN +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +SR-IOV must be enabled on the NIC. +It can be checked in the following command:: + + mlxconfig -d <mst device> query | grep SRIOV_EN + SRIOV_EN True(1) + +If needed, configure SR-IOV:: + + mlxconfig -d <mst device> set SRIOV_EN=1 NUM_OF_VFS=16 + mlxfwreset -d <mst device> reset + +After doing the change, restart the driver:: + + /etc/init.d/openibd restart + +or:: + + service openibd restart + +Then the virtual functions can be instantiated:: + + echo [num_vfs] > /sys/class/infiniband/mlx5_0/device/sriov_numvfs + + +.. _mlx5_sub_function: + +Sub-Function with OFED/EN +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Sub-Function is a portion of the PCI device, +it has its own dedicated queues. +An SF shares PCI-level resources with other SFs and/or with its parent PCI function. + +0. Requirement:: + + OFED version >= 5.4-0.3.3.0 + +1. Configure SF feature:: + + # Run mlxconfig on both PFs on host and ECPFs on BlueField. + mlxconfig -d <mst device> set PER_PF_NUM_SF=1 PF_TOTAL_SF=252 PF_SF_BAR_SIZE=12 + +2. Enable switchdev mode:: + + mlxdevm dev eswitch set pci/<DBDF> mode switchdev + +3. Add SF port:: + + mlxdevm port add pci/<DBDF> flavour pcisf pfnum 0 sfnum <sfnum> + + Get SFID from output: pci/<DBDF>/<SFID> + +4. Modify MAC address:: + + mlxdevm port function set pci/<DBDF>/<SFID> hw_addr <MAC> + +5. Activate SF port:: + + mlxdevm port function set pci/<DBDF>/<ID> state active + +6. Devargs to probe SF device:: + + auxiliary:mlx5_core.sf.<num>,class=eth:regex + + +Enable Switchdev Mode +^^^^^^^^^^^^^^^^^^^^^ + +Switchdev mode is a mode in E-Switch, that binds between representor and VF or SF. +Representor is a port in DPDK that is connected to a VF or SF in such a way +that assuming there are no offload flows, each packet that is sent from the VF or SF +will be received by the corresponding representor. +While each packet that is sent to a representor will be received by the VF or SF. + +After :ref:`configuring VF <mlx5_vf>`, the device must be unbound:: + + printf "<device pci address>" > /sys/bus/pci/drivers/mlx5_core/unbind + +Then switchdev mode is enabled:: + + echo switchdev > /sys/class/net/<net device>/compat/devlink/mode + +The device can be bound again at this point. + + +Run as Non-Root +^^^^^^^^^^^^^^^ + +In order to run as a non-root user, +some capabilities must be granted to the application:: + + setcap cap_sys_admin,cap_net_admin,cap_net_raw,cap_ipc_lock+ep <dpdk-app> + +Below are the reasons for the need of each capability: + +``cap_sys_admin`` + When using physical addresses (PA mode), with Linux >= 4.0, + for access to ``/proc/self/pagemap``. + +``cap_net_admin`` + For device configuration. + +``cap_net_raw`` + For raw ethernet queue allocation through kernel driver. + +``cap_ipc_lock`` + For DMA memory pinning. + + +Windows Environment +~~~~~~~~~~~~~~~~~~~ + +WinOF2 version 2.60 or higher must be installed on the machine. + + +WinOF2 Installation +^^^^^^^^^^^^^^^^^^^ + +The driver can be downloaded from the following site: `WINOF2 +<https://network.nvidia.com/products/adapter-software/ethernet/windows/winof-2/>`_. + + +DevX Enablement +^^^^^^^^^^^^^^^ + +DevX for Windows must be enabled in the Windows registry. +The keys ``DevxEnabled`` and ``DevxFsRules`` must be set. +Additional information can be found in the WinOF2 user manual. + + +.. _mlx5_firmware_config: + +Firmware Configuration +~~~~~~~~~~~~~~~~~~~~~~ + +Firmware features can be configured as key/value pairs. + +The command to set a value is:: + + mlxconfig -d <device> set <key>=<value> + +The command to query a value is:: + + mlxconfig -d <device> query <key> + +The device name for the command ``mlxconfig`` can be either the PCI address, +or the mst device name found with:: + + mst status + +Below are some firmware configurations listed. + +- link type:: + + LINK_TYPE_P1 + LINK_TYPE_P2 + value: 1=Infiniband 2=Ethernet 3=VPI(auto-sense) + +- enable SR-IOV:: + + SRIOV_EN=1 + +- the maximum number of SR-IOV virtual functions:: + + NUM_OF_VFS=<max> + +- enable DevX (required by Direct Rules and other features):: + + UCTX_EN=1 + +- aggressive CQE zipping:: + + CQE_COMPRESSION=1 + +- L3 VXLAN and VXLAN-GPE destination UDP port:: + + IP_OVER_VXLAN_EN=1 + IP_OVER_VXLAN_PORT=<udp dport> + +- enable VXLAN-GPE tunnel flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=0 + or + FLEX_PARSER_PROFILE_ENABLE=2 + +- enable IP-in-IP tunnel flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=0 + +- enable MPLS flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=1 + +- enable ICMP(code/type/identifier/sequence number) / ICMP6(code/type) fields matching:: + + FLEX_PARSER_PROFILE_ENABLE=2 + +- enable Geneve flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=0 + or + FLEX_PARSER_PROFILE_ENABLE=1 + +- enable Geneve TLV option flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=0 + +- enable GTP flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=3 + +- enable eCPRI flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=4 + PROG_PARSE_GRAPH=1 + +- enable dynamic flex parser for flex item:: + + FLEX_PARSER_PROFILE_ENABLE=4 + PROG_PARSE_GRAPH=1 + +- enable realtime timestamp format:: + + REAL_TIME_CLOCK_ENABLE=1 + + +.. _mlx5_common_driver_options: + +Device Arguments +---------------- + +The driver can be configured per device. +A single argument list can be used for a device managed by multiple PMDs. +The parameters must be passed through the EAL option ``-a``, +as examples below: + +- PCI device:: + + -a 0000:03:00.2,class=eth:regex,mr_mempool_reg_en=0 + +- Auxiliary SF:: + + -a auxiliary:mlx5_core.sf.2,class=compress,mr_ext_memseg_en=0 + +Each device class PMD has its own list of specific arguments, +and below are the arguments supported by the common mlx5 layer. + +- ``class`` parameter [string] + + Select the classes of the drivers that should probe the device. + See :ref:`mlx5_classes` for more explanation and details. + + The default value is ``eth``. + +- ``mr_ext_memseg_en`` parameter [int] + + A nonzero value enables extending memseg when registering DMA memory. If + enabled, the number of entries in MR (Memory Region) lookup table on datapath + is minimized and it benefits performance. On the other hand, it worsens memory + utilization because registered memory is pinned by kernel driver. Even if a + page in the extended chunk is freed, that doesn't become reusable until the + entire memory is freed. + + Enabled by default. + +- ``mr_mempool_reg_en`` parameter [int] + + A nonzero value enables implicit registration of DMA memory of all mempools + except those having ``RTE_MEMPOOL_F_NON_IO``. This flag is set automatically + for mempools populated with non-contiguous objects or those without IOVA. + The effect is that when a packet from a mempool is transmitted, + its memory is already registered for DMA in the PMD and no registration + will happen on the data path. The tradeoff is extra work on the creation + of each mempool and increased HW resource use if some mempools + are not used with MLX5 devices. + + Enabled by default. + +- ``sys_mem_en`` parameter [int] + + A non-zero value enables the PMD memory management allocating memory + from system by default, without explicit rte memory flag. + + By default, the PMD will set this value to 0. diff --git a/doc/guides/regexdevs/mlx5.rst b/doc/guides/regexdevs/mlx5.rst index fc2cacba60..83280bda40 100644 --- a/doc/guides/regexdevs/mlx5.rst +++ b/doc/guides/regexdevs/mlx5.rst @@ -3,10 +3,10 @@ .. include:: <isonum.txt> -MLX5 RegEx driver +MLX5 RegEx Driver ================= -The MLX5 RegEx (Regular Expression) driver library +The mlx5 RegEx (Regular Expression) driver library (**librte_regex_mlx5**) provides support for **Mellanox BlueField-2** families of 25/50/100/200 Gb/s adapters. @@ -17,29 +17,21 @@ This PMD is configuring the RegEx HW engine. For the PMD to work, the application must supply a precompiled rule file in rof2 format. -The PMD uses libibverbs and libmlx5 to access the device firmware -or directly the hardware components. -There are different levels of objects and bypassing abilities -to get the best performances: - -- Verbs is a complete high-level generic API -- Direct Verbs is a device-specific API -- DevX allows to access firmware objects - -Enabling librte_regex_mlx5 causes DPDK applications to be linked against -libibverbs. - -Mellanox mlx5 pci device can be probed by number of different pci devices, -for example net / vDPA / RegEx. To select the RegEx PMD ``class=regex`` should -be specified as device parameter. The RegEx device can be probed and used with -other Mellanox devices, by adding more options in the class. -For example: ``class=net:regex`` will probe both the net PMD and the RegEx PMD. +See :doc:`../../platform/mlx5` guide for more design details. Features -------- - Multi segments mbuf support. +Configuration +------------- + +See :ref:`mlx5 common compilation <mlx5_common_compilation>`, +:ref:`mlx5 firmware configuration <mlx5_firmware_config>`, +and :ref:`mlx5 common driver options <mlx5_common_driver_options>`. + + Supported NICs -------------- @@ -52,12 +44,8 @@ Prerequisites - Enable the RegEx capabilities using system call from the BlueField-2. - Official support is not yet released. + Limitations ----------- - The firmware version must be greater than XX.31.0364 - -Run-time configuration -~~~~~~~~~~~~~~~~~~~~~~ - -- **ethtool** operations on related kernel interfaces also affect the PMD. diff --git a/doc/guides/vdpadevs/mlx5.rst b/doc/guides/vdpadevs/mlx5.rst index 30f0b62eb4..acb791032a 100644 --- a/doc/guides/vdpadevs/mlx5.rst +++ b/doc/guides/vdpadevs/mlx5.rst @@ -3,10 +3,10 @@ .. include:: <isonum.txt> -MLX5 vDPA driver +MLX5 vDPA Driver ================ -The MLX5 vDPA (vhost data path acceleration) driver library +The mlx5 vDPA (vhost data path acceleration) driver library (**librte_vdpa_mlx5**) provides support for **Mellanox ConnectX-6**, **Mellanox ConnectX-6 Dx** and **Mellanox BlueField** families of 10/25/40/50/100/200 Gb/s adapters as well as their virtual functions (VF) in @@ -17,33 +17,8 @@ SR-IOV context. This driver is enabled automatically when using "meson" build system which will detect dependencies. - -Design ------- - -For security reasons and robustness, this driver only deals with virtual -memory addresses. The way resources allocations are handled by the kernel, -combined with hardware specifications that allow to handle virtual memory -addresses directly, ensure that DPDK applications cannot access random -physical memory (or memory that does not belong to the current process). - -The PMD can use libibverbs and libmlx5 to access the device firmware -or directly the hardware components. -There are different levels of objects and bypassing abilities -to get the best performances: - -- Verbs is a complete high-level generic API -- Direct Verbs is a device-specific API -- DevX allows to access firmware objects -- Direct Rules manages flow steering at low-level hardware layer - -Enabling librte_vdpa_mlx5 causes DPDK applications to be linked against -libibverbs. - -A Mellanox mlx5 PCI device can be probed by either net/mlx5 driver or vdpa/mlx5 -driver but not in parallel. Hence, the user should decide the driver by the -``class`` parameter in the device argument list. -By default, the mlx5 device will be probed by the net/mlx5 driver. +See :doc:`../../platform/mlx5` guide for design details, +and which PMDs can be combined with vDPA PMD. Supported NICs -------------- @@ -58,52 +33,16 @@ Prerequisites ------------- - Mellanox OFED version: **5.0** - see :doc:`../../nics/mlx5` guide for more Mellanox OFED details. - -Compilation option -~~~~~~~~~~~~~~~~~~ - -The meson option ``ibverbs_link`` is **shared** by default, -but can be configured to have the following values: - -- ``dlopen`` - - Build PMD with additional code to make it loadable without hard - dependencies on **libibverbs** nor **libmlx5**, which may not be installed - on the target system. - - In this mode, their presence is still required for it to run properly, - however their absence won't prevent a DPDK application from starting (with - DPDK shared build disabled) and they won't show up as missing with ``ldd(1)``. - - It works by moving these dependencies to a purpose-built rdma-core "glue" - plug-in which must be installed in a directory whose name is based - on ``RTE_EAL_PMD_PATH`` suffixed with ``-glue``. - - This option has no performance impact. - -- ``static`` - - Embed static flavor of the dependencies **libibverbs** and **libmlx5** - in the PMD shared library or the executable static binary. - -.. note:: - - Default armv8a configuration of meson build sets ``RTE_CACHE_LINE_SIZE`` - to 128 then brings performance degradation. + See :ref:`mlx5 common prerequisites <mlx5_linux_prerequisites>` for more details. Run-time configuration ~~~~~~~~~~~~~~~~~~~~~~ -- **ethtool** operations on related kernel interfaces also affect the PMD. - Driver options ^^^^^^^^^^^^^^ -- ``class`` parameter [string] - - Select the class of the driver that should probe the device. - `vdpa` for the mlx5 vDPA driver. +Please refer to :ref:`mlx5 common options <mlx5_common_driver_options>` +for an additional list of options shared with other mlx5 drivers. - ``event_mode`` parameter [int] @@ -163,18 +102,6 @@ Driver options - 0, HW default. -Devargs example -^^^^^^^^^^^^^^^ - -- PCI devargs:: - - -a 0000:03:00.2,class=vdpa - -- Auxiliary devargs:: - - -a auxiliary:mlx5_core.sf.2,class=vdpa - - Error handling ^^^^^^^^^^^^^^ -- 2.25.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH v2 5/5] doc: update doorbell mapping parameter name in mlx5 guide 2022-02-23 13:48 ` [PATCH v2 0/5] Refactor " Michael Baum ` (3 preceding siblings ...) 2022-02-23 13:48 ` [PATCH v2 4/5] doc: add shared guide for mlx5 PMDs Michael Baum @ 2022-02-23 13:48 ` Michael Baum 2022-02-23 21:48 ` Ferruh Yigit 2022-02-23 14:35 ` [PATCH v2 0/5] Refactor mlx5 guides Raslan Darawsheh 5 siblings, 1 reply; 17+ messages in thread From: Michael Baum @ 2022-02-23 13:48 UTC (permalink / raw) To: dev; +Cc: Matan Azrad, Raslan Darawsheh, Viacheslav Ovsiienko The "tx_db_nc" devarg forces doorbell register mapping to non-cached region eliminating the extra write memory barrier. This argument was used in creating the UAR for Tx and thus affected its performance. Recently [1] its use has been extended to all UAR creation in all mlx5 drivers, and now its name is no longer so accurate. This patch changes its name to "sq_db_nc" to suit any send queue that uses it. The old name will still work for backward compatibility. [1] commit 5dfa003db53f ("common/mlx5: fix post doorbell barrier") Signed-off-by: Michael Baum <michaelba@nvidia.com> Reviewed-by: Raslan Darawsheh <rasland@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> --- doc/guides/nics/mlx5.rst | 34 ++----------------- doc/guides/platform/mlx5.rst | 39 ++++++++++++++++++++++ drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- drivers/common/mlx5/mlx5_common.c | 31 ++++++++++++----- drivers/common/mlx5/mlx5_common_defs.h | 8 ++--- drivers/net/mlx5/linux/mlx5_verbs.c | 2 +- drivers/net/mlx5/mlx5_devx.c | 2 +- 7 files changed, 72 insertions(+), 46 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index f94ed90ef0..8956cd1dd8 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -814,37 +814,9 @@ for an additional list of options shared with other mlx5 drivers. - ``tx_db_nc`` parameter [int] - The rdma core library can map doorbell register in two ways, depending on the - environment variable "MLX5_SHUT_UP_BF": - - - As regular cached memory (usually with write combining attribute), if the - variable is either missing or set to zero. - - As non-cached memory, if the variable is present and set to not "0" value. - - The type of mapping may slightly affect the Tx performance, the optimal choice - is strongly relied on the host architecture and should be deduced practically. - - If ``tx_db_nc`` is set to zero, the doorbell is forced to be mapped to regular - memory (with write combining), the PMD will perform the extra write memory barrier - after writing to doorbell, it might increase the needed CPU clocks per packet - to send, but latency might be improved. - - If ``tx_db_nc`` is set to one, the doorbell is forced to be mapped to non - cached memory, the PMD will not perform the extra write memory barrier - after writing to doorbell, on some architectures it might improve the - performance. - - If ``tx_db_nc`` is set to two, the doorbell is forced to be mapped to regular - memory, the PMD will use heuristics to decide whether write memory barrier - should be performed. For bursts with size multiple of recommended one (64 pkts) - it is supposed the next burst is coming and no need to issue the extra memory - barrier (it is supposed to be issued in the next coming burst, at least after - descriptor writing). It might increase latency (on some hosts till next - packets transmit) and should be used with care. - - If ``tx_db_nc`` is omitted or set to zero, the preset (if any) environment - variable "MLX5_SHUT_UP_BF" value is used. If there is no "MLX5_SHUT_UP_BF", - the default ``tx_db_nc`` value is zero for ARM64 hosts and one for others. + This parameter name is deprecated and ignored. + The new name for this parameter is ``sq_db_nc``. + See :ref:`common driver options <mlx5_common_driver_options>`. - ``tx_pp`` parameter [int] diff --git a/doc/guides/platform/mlx5.rst b/doc/guides/platform/mlx5.rst index 0fd5e6604d..d073c213ca 100644 --- a/doc/guides/platform/mlx5.rst +++ b/doc/guides/platform/mlx5.rst @@ -600,3 +600,42 @@ and below are the arguments supported by the common mlx5 layer. from system by default, without explicit rte memory flag. By default, the PMD will set this value to 0. + +- ``sq_db_nc`` parameter [int] + + The rdma core library can map doorbell register in two ways, + depending on the environment variable "MLX5_SHUT_UP_BF": + + - As regular cached memory (usually with write combining attribute), + if the variable is either missing or set to zero. + - As non-cached memory, if the variable is present and set to not "0" value. + + The same doorbell mapping approach is implemented directly by PMD + in UAR generation for queues created with DevX. + + The type of mapping may slightly affect the send queue performance, + the optimal choice strongly relied on the host architecture + and should be deduced practically. + + If ``sq_db_nc`` is set to zero, the doorbell is forced to be mapped to + regular memory (with write combining), the PMD will perform the extra write + memory barrier after writing to doorbell, it might increase the needed CPU + clocks per packet to send, but latency might be improved. + + If ``sq_db_nc`` is set to one, the doorbell is forced to be mapped to non + cached memory, the PMD will not perform the extra write memory barrier after + writing to doorbell, on some architectures it might improve the performance. + + If ``sq_db_nc`` is set to two, the doorbell is forced to be mapped to + regular memory, the PMD will use heuristics to decide whether a write memory + barrier should be performed. For bursts with size multiple of recommended one + (64 pkts) it is supposed the next burst is coming and no need to issue the + extra memory barrier (it is supposed to be issued in the next coming burst, + at least after descriptor writing). It might increase latency (on some hosts + till the next packets transmit) and should be used with care. + The PMD uses heuristics only for Tx queue, for other semd queues the doorbell + is forced to be mapped to regular memory as same as ``sq_db_nc`` is set to 0. + + If ``sq_db_nc`` is omitted, the preset (if any) environment variable + "MLX5_SHUT_UP_BF" value is used. If there is no "MLX5_SHUT_UP_BF", the + default ``sq_db_nc`` value is zero for ARM64 hosts and one for others. \ No newline at end of file diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c index 0d3e24e04e..a752d79e8e 100644 --- a/drivers/common/mlx5/linux/mlx5_common_os.c +++ b/drivers/common/mlx5/linux/mlx5_common_os.c @@ -630,7 +630,7 @@ mlx5_config_doorbell_mapping_env(int dbnc) setenv(MLX5_SHUT_UP_BF, MLX5_SHUT_UP_BF_DEFAULT, 1); else setenv(MLX5_SHUT_UP_BF, - dbnc == MLX5_TXDB_NCACHED ? "1" : "0", 1); + dbnc == MLX5_SQ_DB_NCACHED ? "1" : "0", 1); return value; } diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 96906d3f39..8cf391df13 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -35,10 +35,17 @@ uint8_t haswell_broadwell_cpu; /* * Device parameter to force doorbell register mapping - * to non-cahed region eliminating the extra write memory barrier. + * to non-cached region eliminating the extra write memory barrier. + * Deprecated, ignored (Name changed to sq_db_nc). */ #define MLX5_TX_DB_NC "tx_db_nc" +/* + * Device parameter to force doorbell register mapping + * to non-cached region eliminating the extra write memory barrier. + */ +#define MLX5_SQ_DB_NC "sq_db_nc" + /* In case this is an x86_64 intel processor to check if * we should use relaxed ordering. */ @@ -255,11 +262,17 @@ mlx5_common_args_check_handler(const char *key, const char *val, void *opaque) DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val); return -rte_errno; } - if (strcmp(key, MLX5_TX_DB_NC) == 0) { - if (tmp != MLX5_TXDB_CACHED && - tmp != MLX5_TXDB_NCACHED && - tmp != MLX5_TXDB_HEURISTIC) { - DRV_LOG(ERR, "Invalid Tx doorbell mapping parameter."); + if (strcmp(key, MLX5_TX_DB_NC) == 0) + DRV_LOG(WARNING, + "%s: deprecated parameter, converted to queue_db_nc", + key); + if (strcmp(key, MLX5_SQ_DB_NC) == 0 || + strcmp(key, MLX5_TX_DB_NC) == 0) { + if (tmp != MLX5_SQ_DB_CACHED && + tmp != MLX5_SQ_DB_NCACHED && + tmp != MLX5_SQ_DB_HEURISTIC) { + DRV_LOG(ERR, + "Invalid Send Queue doorbell mapping parameter."); rte_errno = EINVAL; return -rte_errno; } @@ -293,6 +306,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist, RTE_DEVARGS_KEY_CLASS, MLX5_DRIVER_KEY, MLX5_TX_DB_NC, + MLX5_SQ_DB_NC, MLX5_MR_EXT_MEMSEG_EN, MLX5_SYS_MEM_EN, MLX5_MR_MEMPOOL_REG_EN, @@ -317,7 +331,8 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist, DRV_LOG(DEBUG, "mr_ext_memseg_en is %u.", config->mr_ext_memseg_en); DRV_LOG(DEBUG, "mr_mempool_reg_en is %u.", config->mr_mempool_reg_en); DRV_LOG(DEBUG, "sys_mem_en is %u.", config->sys_mem_en); - DRV_LOG(DEBUG, "Tx doorbell mapping parameter is %d.", config->dbnc); + DRV_LOG(DEBUG, "Send Queue doorbell mapping parameter is %d.", + config->dbnc); return ret; } @@ -1231,7 +1246,7 @@ mlx5_devx_alloc_uar(struct mlx5_common_device *cdev) for (retry = 0; retry < MLX5_ALLOC_UAR_RETRY; ++retry) { #ifdef MLX5DV_UAR_ALLOC_TYPE_NC /* Control the mapping type according to the settings. */ - uar_mapping = (cdev->config.dbnc == MLX5_TXDB_NCACHED) ? + uar_mapping = (cdev->config.dbnc == MLX5_SQ_DB_NCACHED) ? MLX5DV_UAR_ALLOC_TYPE_NC : MLX5DV_UAR_ALLOC_TYPE_BF; #else /* diff --git a/drivers/common/mlx5/mlx5_common_defs.h b/drivers/common/mlx5/mlx5_common_defs.h index ca80cd8d29..68b700dc0b 100644 --- a/drivers/common/mlx5/mlx5_common_defs.h +++ b/drivers/common/mlx5/mlx5_common_defs.h @@ -34,10 +34,10 @@ /* Default PMD specific parameter value. */ #define MLX5_ARG_UNSET (-1) -/* MLX5_TX_DB_NC supported values. */ -#define MLX5_TXDB_CACHED 0 -#define MLX5_TXDB_NCACHED 1 -#define MLX5_TXDB_HEURISTIC 2 +/* MLX5_SQ_DB_NC supported values. */ +#define MLX5_SQ_DB_CACHED 0 +#define MLX5_SQ_DB_NCACHED 1 +#define MLX5_SQ_DB_HEURISTIC 2 /* Fields of memory mapping type in offset parameter of mmap() */ #define MLX5_UAR_MMAP_CMD_SHIFT 8 diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c index 331c61d3c5..b6ba21c216 100644 --- a/drivers/net/mlx5/linux/mlx5_verbs.c +++ b/drivers/net/mlx5/linux/mlx5_verbs.c @@ -926,7 +926,7 @@ mlx5_txq_ibv_uar_init(struct mlx5_txq_ctrl *txq_ctrl, void *bf_reg) DRV_LOG(ERR, "Failed to get mem page size"); rte_errno = ENOMEM; } - txq->db_heu = priv->sh->cdev->config.dbnc == MLX5_TXDB_HEURISTIC; + txq->db_heu = priv->sh->cdev->config.dbnc == MLX5_SQ_DB_HEURISTIC; txq->db_nc = mlx5_db_map_type_get(uar_mmap_offset, page_size); ppriv->uar_table[txq->idx].db = bf_reg; #ifndef RTE_ARCH_64 diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index e178b799fa..a9b8c2a1b7 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -1326,7 +1326,7 @@ mlx5_txq_devx_obj_new(struct rte_eth_dev *dev, uint16_t idx) txq_data->qp_db = &txq_obj->sq_obj.db_rec[MLX5_SND_DBR]; *txq_data->qp_db = 0; txq_data->qp_num_8s = txq_obj->sq_obj.sq->id << 8; - txq_data->db_heu = sh->cdev->config.dbnc == MLX5_TXDB_HEURISTIC; + txq_data->db_heu = sh->cdev->config.dbnc == MLX5_SQ_DB_HEURISTIC; txq_data->db_nc = sh->tx_uar.dbnc; txq_data->wait_on_time = !!(!sh->config.tx_pp && sh->cdev->config.hca_attr.wait_on_time); -- 2.25.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH v2 5/5] doc: update doorbell mapping parameter name in mlx5 guide 2022-02-23 13:48 ` [PATCH v2 5/5] doc: update doorbell mapping parameter name in mlx5 guide Michael Baum @ 2022-02-23 21:48 ` Ferruh Yigit 2022-02-25 0:01 ` Michael Baum 0 siblings, 1 reply; 17+ messages in thread From: Ferruh Yigit @ 2022-02-23 21:48 UTC (permalink / raw) To: Michael Baum, dev; +Cc: Matan Azrad, Raslan Darawsheh, Viacheslav Ovsiienko On 2/23/2022 1:48 PM, Michael Baum wrote: > The "tx_db_nc" devarg forces doorbell register mapping to non-cached > region eliminating the extra write memory barrier. This argument was > used in creating the UAR for Tx and thus affected its performance. > > Recently [1] its use has been extended to all UAR creation in all mlx5 > drivers, and now its name is no longer so accurate. > > This patch changes its name to "sq_db_nc" to suit any send queue that > uses it. The old name will still work for backward compatibility. > > [1] commit 5dfa003db53f ("common/mlx5: fix post doorbell barrier") > > Signed-off-by: Michael Baum<michaelba@nvidia.com> > Reviewed-by: Raslan Darawsheh<rasland@nvidia.com> > Acked-by: Viacheslav Ovsiienko<viacheslavo@nvidia.com> > --- > doc/guides/nics/mlx5.rst | 34 ++----------------- > doc/guides/platform/mlx5.rst | 39 ++++++++++++++++++++++ > drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- > drivers/common/mlx5/mlx5_common.c | 31 ++++++++++++----- > drivers/common/mlx5/mlx5_common_defs.h | 8 ++--- > drivers/net/mlx5/linux/mlx5_verbs.c | 2 +- > drivers/net/mlx5/mlx5_devx.c | 2 +- > 7 files changed, 72 insertions(+), 46 deletions(-) Hi Michael, This is not a document patch to update the parameter name, patch actually changes the parameter name in the code and documents it, updating it as: net/mlx5: update doorbell mapping parameter name ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: [PATCH v2 5/5] doc: update doorbell mapping parameter name in mlx5 guide 2022-02-23 21:48 ` Ferruh Yigit @ 2022-02-25 0:01 ` Michael Baum 2022-02-25 10:21 ` Ferruh Yigit 0 siblings, 1 reply; 17+ messages in thread From: Michael Baum @ 2022-02-25 0:01 UTC (permalink / raw) To: Ferruh Yigit, dev; +Cc: Matan Azrad, Raslan Darawsheh, Slava Ovsiienko On 2/23/2022 11:48 PM, Ferruh Yigit wrote: > On 2/23/2022 1:48 PM, Michael Baum wrote: > > The "tx_db_nc" devarg forces doorbell register mapping to non-cached > > region eliminating the extra write memory barrier. This argument was > > used in creating the UAR for Tx and thus affected its performance. > > > > Recently [1] its use has been extended to all UAR creation in all mlx5 > > drivers, and now its name is no longer so accurate. > > > > This patch changes its name to "sq_db_nc" to suit any send queue that > > uses it. The old name will still work for backward compatibility. > > > > [1] commit 5dfa003db53f ("common/mlx5: fix post doorbell barrier") > > > > Signed-off-by: Michael Baum<michaelba@nvidia.com> > > Reviewed-by: Raslan Darawsheh<rasland@nvidia.com> > > Acked-by: Viacheslav Ovsiienko<viacheslavo@nvidia.com> > > --- > > doc/guides/nics/mlx5.rst | 34 ++----------------- > > doc/guides/platform/mlx5.rst | 39 ++++++++++++++++++++++ > > drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- > > drivers/common/mlx5/mlx5_common.c | 31 ++++++++++++----- > > drivers/common/mlx5/mlx5_common_defs.h | 8 ++--- > > drivers/net/mlx5/linux/mlx5_verbs.c | 2 +- > > drivers/net/mlx5/mlx5_devx.c | 2 +- > > 7 files changed, 72 insertions(+), 46 deletions(-) > > Hi Michael, > > This is not a document patch to update the parameter name, patch actually > changes the parameter name in the code and documents it, updating it as: > net/mlx5: update doorbell mapping parameter name Hi Ferruh, Your argument makes sense to me, but most of updates was in MLX5 common library so I think it is better to write: common/mlx5: update doorbell mapping parameter name ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH v2 5/5] doc: update doorbell mapping parameter name in mlx5 guide 2022-02-25 0:01 ` Michael Baum @ 2022-02-25 10:21 ` Ferruh Yigit 0 siblings, 0 replies; 17+ messages in thread From: Ferruh Yigit @ 2022-02-25 10:21 UTC (permalink / raw) To: Michael Baum, dev; +Cc: Matan Azrad, Raslan Darawsheh, Slava Ovsiienko On 2/25/2022 12:01 AM, Michael Baum wrote: > > On 2/23/2022 11:48 PM, Ferruh Yigit wrote: >> On 2/23/2022 1:48 PM, Michael Baum wrote: >>> The "tx_db_nc" devarg forces doorbell register mapping to non-cached >>> region eliminating the extra write memory barrier. This argument was >>> used in creating the UAR for Tx and thus affected its performance. >>> >>> Recently [1] its use has been extended to all UAR creation in all mlx5 >>> drivers, and now its name is no longer so accurate. >>> >>> This patch changes its name to "sq_db_nc" to suit any send queue that >>> uses it. The old name will still work for backward compatibility. >>> >>> [1] commit 5dfa003db53f ("common/mlx5: fix post doorbell barrier") >>> >>> Signed-off-by: Michael Baum<michaelba@nvidia.com> >>> Reviewed-by: Raslan Darawsheh<rasland@nvidia.com> >>> Acked-by: Viacheslav Ovsiienko<viacheslavo@nvidia.com> >>> --- >>> doc/guides/nics/mlx5.rst | 34 ++----------------- >>> doc/guides/platform/mlx5.rst | 39 ++++++++++++++++++++++ >>> drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- >>> drivers/common/mlx5/mlx5_common.c | 31 ++++++++++++----- >>> drivers/common/mlx5/mlx5_common_defs.h | 8 ++--- >>> drivers/net/mlx5/linux/mlx5_verbs.c | 2 +- >>> drivers/net/mlx5/mlx5_devx.c | 2 +- >>> 7 files changed, 72 insertions(+), 46 deletions(-) >> >> Hi Michael, >> >> This is not a document patch to update the parameter name, patch actually >> changes the parameter name in the code and documents it, updating it as: >> net/mlx5: update doorbell mapping parameter name > > Hi Ferruh, > > Your argument makes sense to me, but most of updates was in MLX5 common library so I think it is better to write: > common/mlx5: update doorbell mapping parameter name ack, updated accordingly. ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: [PATCH v2 0/5] Refactor mlx5 guides 2022-02-23 13:48 ` [PATCH v2 0/5] Refactor " Michael Baum ` (4 preceding siblings ...) 2022-02-23 13:48 ` [PATCH v2 5/5] doc: update doorbell mapping parameter name in mlx5 guide Michael Baum @ 2022-02-23 14:35 ` Raslan Darawsheh 5 siblings, 0 replies; 17+ messages in thread From: Raslan Darawsheh @ 2022-02-23 14:35 UTC (permalink / raw) To: Michael Baum, dev; +Cc: Matan Azrad, Slava Ovsiienko Hi, > -----Original Message----- > From: Michael Baum <michaelba@nvidia.com> > Sent: Wednesday, February 23, 2022 3:48 PM > To: dev@dpdk.org > Cc: Matan Azrad <matan@nvidia.com>; Raslan Darawsheh > <rasland@nvidia.com>; Slava Ovsiienko <viacheslavo@nvidia.com> > Subject: [PATCH v2 0/5] Refactor mlx5 guides > > Recently [1] all the drivers running over the mlx5 device started sharing some > objects. > This raised the need to create documentation for the common driver that > would contain information relevant to all drivers and in particular: > > - Tutorial for configuration of the device. > - Instruction for using Mellanox ofed. > - devargs of the common driver. > > This series adds such documentation and reference to it as parts of the > documentation of the other drivers. > > [1] > https://patchwork.dpdk.org/project/dpdk/cover/20211019205602.3188203- > 1-michaelba@nvidia.com/ > > v2: Rebase > > > Michael Baum (5): > doc: remove obsolete explanations from mlx5 guide > doc: replace broken links in mlx5 guide > doc: correct name of BlueField-2 in mlx5 guide > doc: add shared guide for mlx5 PMDs > doc: update doorbell mapping parameter name in mlx5 guide > > doc/guides/compressdevs/mlx5.rst | 40 +- > doc/guides/cryptodevs/mlx5.rst | 38 +- > doc/guides/nics/mlx4.rst | 4 +- > doc/guides/nics/mlx5.rst | 567 +----------------- > doc/guides/platform/index.rst | 1 + > doc/guides/platform/mlx5.rst | 641 +++++++++++++++++++++ > doc/guides/regexdevs/mlx5.rst | 44 +- > doc/guides/vdpadevs/mlx5.rst | 87 +-- > drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- > drivers/common/mlx5/mlx5_common.c | 31 +- > drivers/common/mlx5/mlx5_common_defs.h | 8 +- > drivers/net/mlx5/linux/mlx5_verbs.c | 2 +- > drivers/net/mlx5/mlx5_devx.c | 2 +- > 13 files changed, 747 insertions(+), 720 deletions(-) create mode 100644 > doc/guides/platform/mlx5.rst > > -- > 2.25.1 Series applied to next-net-mlx, Kindest regards, Raslan Darawsheh ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2022-02-25 10:21 UTC | newest] Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-02-22 12:48 [PATCH 0/5] refactore mlx5 guides Michael Baum 2022-02-22 12:48 ` [PATCH 1/5] doc: remove obsolete explanations from mlx5 guide Michael Baum 2022-02-22 12:48 ` [PATCH 2/5] doc: replace broken links in " Michael Baum 2022-02-22 12:48 ` [PATCH 3/5] doc: correct name of BlueField-2 " Michael Baum 2022-02-22 12:48 ` [PATCH 4/5] doc: add shared guide for mlx5 PMDs Michael Baum 2022-02-22 12:48 ` [PATCH 5/5] doc: update doorbell mapping parameter name in mlx5 guide Michael Baum 2022-02-22 14:43 ` [PATCH 0/5] refactore mlx5 guides Raslan Darawsheh 2022-02-23 13:48 ` [PATCH v2 0/5] Refactor " Michael Baum 2022-02-23 13:48 ` [PATCH v2 1/5] doc: remove obsolete explanations from mlx5 guide Michael Baum 2022-02-23 13:48 ` [PATCH v2 2/5] doc: replace broken links in " Michael Baum 2022-02-23 13:48 ` [PATCH v2 3/5] doc: correct name of BlueField-2 " Michael Baum 2022-02-23 13:48 ` [PATCH v2 4/5] doc: add shared guide for mlx5 PMDs Michael Baum 2022-02-23 13:48 ` [PATCH v2 5/5] doc: update doorbell mapping parameter name in mlx5 guide Michael Baum 2022-02-23 21:48 ` Ferruh Yigit 2022-02-25 0:01 ` Michael Baum 2022-02-25 10:21 ` Ferruh Yigit 2022-02-23 14:35 ` [PATCH v2 0/5] Refactor mlx5 guides Raslan Darawsheh
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).