From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp-4.sys.kth.se (smtp-4.sys.kth.se [130.237.48.193]) by dpdk.org (Postfix) with ESMTP id 98A6C2BB8 for ; Tue, 14 May 2019 07:47:28 +0200 (CEST) Received: from smtp-4.sys.kth.se (localhost.localdomain [127.0.0.1]) by smtp-4.sys.kth.se (Postfix) with ESMTP id 3AEFF29AE; Tue, 14 May 2019 07:47:28 +0200 (CEST) X-Virus-Scanned: by amavisd-new at kth.se Received: from smtp-4.sys.kth.se ([127.0.0.1]) by smtp-4.sys.kth.se (smtp-4.sys.kth.se [127.0.0.1]) (amavisd-new, port 10024) with LMTP id HCI2BHObTpbg; Tue, 14 May 2019 07:47:26 +0200 (CEST) Received: from exdb01.ug.kth.se (exdb01.ug.kth.se [192.168.32.111]) by smtp-4.sys.kth.se (Postfix) with ESMTPS id C41B729E2; Tue, 14 May 2019 07:47:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kth.se; s=default; t=1557812846; bh=+0W77fsnyh7aoUAZdGaKTLgoyKC6fV9BN0gJuemvfKE=; h=From:To:CC:Subject:Date:References:In-Reply-To; b=fX/HyhFoLKGCt0/eCltdW04eQ/hAW2JVSVpH/eSAaiSa3ZDGIqAQ1ItzDMJx8lZum kGeHoEsfwY2pEraavQp8qoBbdauE4lSgO9VRU9q3DzncwDzQjtD5GK4367vBZBJye8 fLo1NuV+tDqbNeS6X8fVWVjs0swfb3qMR8c/gwbM= Received: from exdb04.ug.kth.se (192.168.32.114) by exdb01.ug.kth.se (192.168.32.111) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 14 May 2019 07:47:25 +0200 Received: from exdb05.ug.kth.se (192.168.32.115) by exdb04.ug.kth.se (192.168.32.114) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 14 May 2019 07:47:24 +0200 Received: from exdb05.ug.kth.se ([192.168.32.115]) by exdb05.ug.kth.se ([192.168.32.115]) with mapi id 15.00.1473.003; Tue, 14 May 2019 07:47:24 +0200 From: Tom Barbette To: Ori Kam CC: "yskoh@mellanox.com" , "shahafs@mellanox.com" , "matan@mellanox.com" , "thomas@monjalon.net" , "dev@dpdk.org" Thread-Topic: [dpdk-dev] [PATCH v3] doc: fix update release notes for Mellanox drivers Thread-Index: AQHVCYVn0AmdCNjIoEO5tiVtsSCmT6ZqHfcu Date: Tue, 14 May 2019 05:47:24 +0000 Message-ID: <836d61b3-19e1-42cf-82bd-f7051970b2e7@email.android.com> References: <1557685859-121772-1-git-send-email-orika@mellanox.com>, <1557747662-188493-1-git-send-email-orika@mellanox.com> In-Reply-To: <1557747662-188493-1-git-send-email-orika@mellanox.com> Accept-Language: fr-FR, sv-SE, en-US Content-Language: fr-FR X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH v3] doc: fix update release notes for Mellanox drivers X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 May 2019 05:47:29 -0000 Hi all, I still find it a little bit unclear about direct rules. And therefore, how= to enable this large sclale mode. It is only available for eswitch ? Therefore PF rules will still be slow ? = How can I be sure to be in direct mode ? At some point someone mentioned performance of using a second group would b= e much faster than updating rules in the main group. I think having that s= omewhere in the doc, or internal Mellanox guide would be useful. Last point, is the requirement really OFED 4.6.2 for direct rules ? Because= it's still not publicly available. Thanks, Tom Le 13 mai 2019 14:14, Ori Kam a =E9crit : This patch adds some missing features to Mellanox drivers release notes. It also updates the mlx5/mlx4 documentations. Fixes: d85b204b5dba ("doc: update release notes for Mellanox drivers") Cc: yskoh@mellanox.com Signed-off-by: Ori Kam Acked-by: Shahaf Shuler --- v3: * Address ML comments. V2: * Fix checkpatch issues. --- doc/guides/nics/mlx4.rst | 2 +- doc/guides/nics/mlx5.rst | 184 +++++++++++++++++++++++++----= ---- doc/guides/rel_notes/release_19_05.rst | 7 +- 3 files changed, 148 insertions(+), 45 deletions(-) diff --git a/doc/guides/nics/mlx4.rst b/doc/guides/nics/mlx4.rst index f6d7a16..5c6bbde 100644 --- a/doc/guides/nics/mlx4.rst +++ b/doc/guides/nics/mlx4.rst @@ -253,7 +253,7 @@ thanks to these environment variables: Mellanox OFED as a fallback ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- `Mellanox OFED`_ version: **4.4, 4.5**. +- `Mellanox OFED`_ version: **4.4, 4.5, 4.6**. - firmware version: **2.42.5000** and above. .. _`Mellanox OFED`: http://www.mellanox.com/page/products_dyn?product_fam= ily=3D26&mtag=3Dlinux_sw_drivers diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 325e9f6..9540657 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -7,7 +7,7 @@ MLX5 poll mode driver The MLX5 poll mode driver library (**librte_pmd_mlx5**) provides support for **Mellanox ConnectX-4**, **Mellanox ConnectX-4 Lx** , **Mellanox -ConnectX-5**, **Mellanox ConnectX-6** and **Mellanox Bluefield** families +ConnectX-5**, **Mellanox ConnectX-6** and **Mellanox BlueField** families of 10/25/40/50/100/200 Gb/s adapters as well as their virtual functions (V= F) in SR-IOV context. @@ -62,8 +62,8 @@ Features - RX VLAN stripping. - TX VLAN insertion. - RX CRC stripping configuration. -- Promiscuous mode. -- Multicast promiscuous mode. +- Promiscuous mode on PF and VF. +- Multicast promiscuous mode on PF and VF. - Hardware checksum offloads. - Flow director (RTE_FDIR_MODE_PERFECT, RTE_FDIR_MODE_PERFECT_MAC_VLAN and RTE_ETH_FDIR_REJECT). @@ -78,6 +78,10 @@ Features - Rx HW timestamp. - Tunnel types: VXLAN, L3 VXLAN, VXLAN-GPE, GRE, MPLSoGRE, MPLSoUDP. - Tunnel HW offloads: packet type, inner/outer RSS, IP and UDP checksum ve= rification. +- Nic HW offloads: encapsulation (vxlan, gre, mplsoudp, mplsogre), NAT, ro= uting, TTL + increment/decrement, count, drop, mark. For details please see :ref:`Sup= ported hardware offloads using rte_flow API`. +- Flow insertion rate of more then million flows per second, when using Di= rect Rules. +- Support for multiple rte_flow groups. Limitations ----------- @@ -112,8 +116,6 @@ Limitations is set to multi-packet send or Enhanced multi-packet send. Otherwise it = must have less than 50 segments. -- Count action for RTE flow is **only supported in Mellanox OFED**. - - Flows with a VXLAN Network Identifier equal (or ends to be equal) to 0 are not supported. @@ -147,30 +149,16 @@ Limitations To receive IPv6 Multicast messages on VM, explicitly set the relevant MAC address using rte_eth_dev_mac_addr_add() API. -- E-Switch VXLAN tunnel is not supported together with outer VLAN. - -- E-Switch Flows with VNI pattern must include the VXLAN decapsulation act= ion. - -- E-Switch VXLAN decapsulation Flow: +- E-Switch decapsulation Flow: - can be applied to PF port only. - must specify VF port action (packet redirection from PF to VF). - - must specify tunnel outer UDP local (destination) port, wildcards not = allowed. - - must specify tunnel outer VNI, wildcards not allowed. - - must specify tunnel outer local (destination) IPv4 or IPv6 address, w= ildcards not allowed. - - optionally may specify tunnel outer remote (source) IPv4 or IPv6, wild= cards or group IPs allowed. - optionally may specify tunnel inner source and destination MAC address= es. -- E-Switch VXLAN encapsulation Flow: +- E-Switch encapsulation Flow: - can be applied to VF ports only. - must specify PF port action (packet redirection from VF to PF). - - must specify the VXLAN item with tunnel outer parameters. - - must specify the tunnel outer VNI in the VXLAN item. - - must specify the tunnel outer remote (destination) UDP port in the VXL= AN item. - - must specify the tunnel outer local (source) IPv4 or IPv6 in the , thi= s address will locally (with scope link) assigned to the outer network inte= rface, wildcards not allowed. - - must specify the tunnel outer remote (destination) IPv4 or IPv6 in the= VXLAN item, group IPs allowed. - - must specify the tunnel outer destination MAC address in the VXLAN ite= m, this address will be used to create neigh rule. Statistics ---------- @@ -227,7 +215,7 @@ These options can be modified in the ``.config`` file. .. note:: - For Bluefield, target should be set to ``arm64-bluefield-linux-gcc``. T= his + For BlueField, target should be set to ``arm64-bluefield-linux-gcc``. T= his will enable ``CONFIG_RTE_LIBRTE_MLX5_PMD`` and set ``RTE_CACHE_LINE_SIZ= E`` to 64. Default armv8a configuration of make build and meson build set it t= o 128 then brings performance degradation. @@ -277,8 +265,8 @@ Run-time configuration Supported on: - - x86_64 with ConnectX-4, ConnectX-4 LX, ConnectX-5, ConnectX-6 and Blue= field. - - POWER8 and ARMv8 with ConnectX-4 LX, ConnectX-5, ConnectX-6 and Bluefi= eld. + - x86_64 with ConnectX-4, ConnectX-4 LX, ConnectX-5, ConnectX-6 and Blue= Field. + - POWER9 and ARMv8 with ConnectX-4 LX, ConnectX-5, ConnectX-6 and BlueFi= eld. - ``rxq_cqe_pad_en`` parameter [int] @@ -296,7 +284,7 @@ Run-time configuration Supported on: - - CPU having 128B cacheline with ConnectX-5 and Bluefield. + - CPU having 128B cacheline with ConnectX-5 and BlueField. - ``rxq_pkt_pad_en`` parameter [int] @@ -308,8 +296,8 @@ Run-time configuration Supported on: - - x86_64 with ConnectX-4, ConnectX-4 LX, ConnectX-5, ConnectX-6 and Blue= field. - - POWER8 and ARMv8 with ConnectX-4 LX, ConnectX-5, ConnectX-6 and Bluefi= eld. + - x86_64 with ConnectX-4, ConnectX-4 LX, ConnectX-5, ConnectX-6 and Blue= Field. + - POWER8 and ARMv8 with ConnectX-4 LX, ConnectX-5, ConnectX-6 and BlueFi= eld. - ``mprq_en`` parameter [int] @@ -375,13 +363,13 @@ Run-time configuration This option should be used in combination with ``txq_inline`` above. - On ConnectX-4, ConnectX-4 LX, ConnectX-5, ConnectX-6 and Bluefield witho= ut + On ConnectX-4, ConnectX-4 LX, ConnectX-5, ConnectX-6 and BlueField witho= ut Enhanced MPW: - Disabled by default. - In case ``txq_inline`` is set recommendation is 4. - On ConnectX-5, ConnectX-6 and Bluefield with Enhanced MPW: + On ConnectX-5, ConnectX-6 and BlueField with Enhanced MPW: - Set to 8 by default. @@ -395,14 +383,14 @@ Run-time configuration - Set to 8 by default on ARMv8. - Set to 4 by default otherwise. - On Bluefield + On BlueField - Set to 16 by default. - ``txq_mpw_en`` parameter [int] A nonzero value enables multi-packet send (MPS) for ConnectX-4 Lx and - enhanced multi-packet send (Enhanced MPS) for ConnectX-5, ConnectX-6 and= Bluefield. + enhanced multi-packet send (Enhanced MPS) for ConnectX-5, ConnectX-6 and= BlueField. MPS allows the TX burst function to pack up multiple packets in a single descriptor session in order to save PCI bandwidth and improve performance at the cost of a slightly higher CPU usage. When @@ -417,13 +405,13 @@ Run-time configuration DEV_TX_OFFLOAD_VXLAN_TNL_TSO, DEV_TX_OFFLOAD_GRE_TNL_TSO, DEV_TX_OFFLOAD= _VLAN_INSERT``. When those offloads are requested the MPS send function will not be used= . - It is currently only supported on the ConnectX-4 Lx, ConnectX-5, Connect= X-6 and Bluefield + It is currently only supported on the ConnectX-4 Lx, ConnectX-5, Connect= X-6 and BlueField families of adapters. On ConnectX-4 Lx the MPW is considered un-secure hence disabled by defau= lt. Users which enable the MPW should be aware that application which provid= es incorrect mbuf descriptors in the Tx burst can lead to serious errors in the host = including, on some cases, NIC to get stuck. - On ConnectX-5, ConnectX-6 and Bluefield the MPW is secure and enabled by= default. + On ConnectX-5, ConnectX-6 and BlueField the MPW is secure and enabled by= default. - ``txq_mpw_hdr_dseg_en`` parameter [int] @@ -443,14 +431,14 @@ Run-time configuration - ``tx_vec_en`` parameter [int] - A nonzero value enables Tx vector on ConnectX-5, ConnectX-6 and Bluefiel= d NICs if the number of + A nonzero value enables Tx vector on ConnectX-5, ConnectX-6 and BlueFiel= d NICs if the number of global Tx queues on the port is less than ``txqs_max_vec``. This option cannot be used with certain offloads such as ``DEV_TX_OFFLOA= D_TCP_TSO, DEV_TX_OFFLOAD_VXLAN_TNL_TSO, DEV_TX_OFFLOAD_GRE_TNL_TSO, DEV_TX_OFFLOAD= _VLAN_INSERT``. When those offloads are requested the MPS send function will not be used= . - Enabled by default on ConnectX-5, ConnectX-6 and Bluefield. + Enabled by default on ConnectX-5, ConnectX-6 and BlueField. - ``rx_vec_en`` parameter [int] @@ -480,10 +468,15 @@ Run-time configuration A nonzero value enables the DV flow steering assuming it is supported by the driver. - The DV flow steering is not supported on switchdev mode. Disabled by default. +- ``dv_esw_en`` parameter [int] + + A nonzero value enables E-Switch using Direct Rules. + + Enabled by default if supported. + - ``mr_ext_memseg_en`` parameter [int] A nonzero value enables extending memseg when registering DMA memory. If @@ -545,7 +538,7 @@ DPDK and must be installed separately: - **libmlx5** Low-level user space driver library for Mellanox - ConnectX-4/ConnectX-5/ConnectX-6/Bluefield devices, it is automatically = loaded + ConnectX-4/ConnectX-5/ConnectX-6/BlueField devices, it is automatically = loaded by libibverbs. This library basically implements send/receive calls to the hardware @@ -567,7 +560,7 @@ DPDK and must be installed separately: their devices: - mlx5_core: hardware driver managing Mellanox - ConnectX-4/ConnectX-5/ConnectX-6/Bluefield devices and related Etherne= t kernel + ConnectX-4/ConnectX-5/ConnectX-6/BlueField devices and related Etherne= t kernel network devices. - mlx5_ib: InifiniBand device driver. - ib_uverbs: user space driver for Verbs (entry point for libibverbs). @@ -575,7 +568,7 @@ DPDK and must be installed separately: - **Firmware update** Mellanox OFED/EN releases include firmware updates for - ConnectX-4/ConnectX-5/ConnectX-6/Bluefield adapters. + ConnectX-4/ConnectX-5/ConnectX-6/BlueField adapters. Because each release provides new features, these updates must be applie= d to match the kernel modules and libraries they come with. @@ -622,7 +615,8 @@ thanks to these environment variables: Mellanox OFED/EN ^^^^^^^^^^^^^^^^ -- Mellanox OFED version: **4.4, 4.5** / Mellanox EN version: **4.5** +- Mellanox OFED version: ** 4.5, 4.6** / + Mellanox EN version: **4.5, 4.6** - firmware version: - ConnectX-4: **12.21.1000** and above. @@ -630,7 +624,7 @@ Mellanox OFED/EN - ConnectX-5: **16.21.1000** and above. - ConnectX-5 Ex: **16.21.1000** and above. - ConnectX-6: **20.99.5374** and above. - - Bluefield: **18.99.3950** and above. + - BlueField: **18.25.1010** and above. While these libraries and kernel modules are available on OpenFabrics Alliance's `website `__ and provided by pack= age @@ -766,6 +760,56 @@ Quick Start Guide on OFED/EN 6. Compile DPDK and you are ready to go. See instructions on :ref:`Development Kit Build System ` +Enable switchdev mode +--------------------- + +Switchdev mode is a mode in E-Switch, that binds between representor and V= F. +Representor is a port in DPDK that is connected to a VF in such a way +that assuming there are no offload flows, each packet that is sent from th= e VF +will be received by the corresponding representor. While each packet that = is +sent to a representor will be received by the VF. +This is very useful in case of SRIOV mode, where the first packet that is = sent +by the VF will be received by the DPDK application which will decide if th= is +flow should be offloaded to the E-Switch. After offloading the flow packet +that the VF that are matching the flow will not be received any more by +the DPDK application. + +1. Enable SRIOV mode: + + .. code-block:: console + + mlxconfig -d set SRIOV_EN=3Dtrue + +2. Configure the max number of VFs: + + .. code-block:: console + + mlxconfig -d set NUM_OF_VFS=3D + +3. Reset the FW: + + .. code-block:: console + + mlxfwreset -d reset + +3. Configure the actual number of VFs: + + .. code-block:: console + + echo /sys/class/net//device/sriov_numvfs + +4. Unbind the device (can be rebind after the switchdev mode): + + .. code-block:: console + + echo -n " /sys/bus/pci/drivers/mlx5_core/unb= ind + +5. Enbale switchdev mode: + + .. code-block:: console + + echo switchdev > /sys/class/net//compat/devlink/mode + Performance tuning ------------------ @@ -842,6 +886,62 @@ Performance tuning - Configure per-lcore cache when creating Mempools for packet buffer. - Refrain from dynamically allocating/freeing memory in run-time. +Supported hardware offloads using rte_flow API +---------------------------------------------- + +.. _Supported hardware offloads using rte_flow API: + +.. table:: Supported hardware offloads using rte_flow API + + +-----------------------+-----------------+-----------------+ + | Offload | E-Switch | NIC | + | | | | + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+ + | Count | | DPDK 19.05 | | DPDK 19.02 | + | | | OFED 4.6 | | OFED 4.6 | + | | | RDMA-CORE V24 | | RDMA-CORE V23 | + | | | ConnectX-5 | | ConnectX-5 | + +-----------------------+-----------------+-----------------+ + | Drop / Queue / RSS | | DPDK 19.05 | | DPDK 18.11 | + | | | OFED 4.6 | | OFED 4.5 | + | | | RDMA-CORE V24 | | RDMA-CORE V23 | + | | | ConnectX-5 | | ConnectX-4 | + +-----------------------+-----------------+-----------------+ + | Encapsulation | | DPDK 19.05 | | DPDK 19.02 | + | (VXLAN / NVGRE / RAW) | | OFED 4.6.2 | | OFED 4.6 | + | | | RDMA-CORE V24 | | RDMA-CORE V23 | + | | | ConnectX-5 | | ConnectX-5 | + +-----------------------+-----------------+-----------------+ + | Header rewrite | | DPDK 19.05 | | DPDK 19.02 | + | (set_ipv4_src / | | OFED 4.6.2 | | OFED 4.6.2 | + | set_ipv4_dst / | | RDMA-CORE V24 | | RDMA-CORE V23 | + | set_ipv6_src / | | ConnectX-5 | | ConnectX-5 | + | set_ipv6_dst / | | | + | set_tp_src / | | | + | set_tp_dst / | | | + | dec_ttl / | | | + | set_ttl / | | | + | set_mac_src / | | | + | set_mac_dst) | | | + +-----------------------+-----------------+-----------------+ + | Jump | | DPDK 19.05 | | DPDK 19.02 | + | | | OFED 4.6.2 | | OFED 4.6.2 | + | | | RDMA-CORE V24 | | N/A | + | | | ConnectX-5 | | ConnectX-5 | + +-----------------------+-----------------+-----------------+ + | Mark / Flag | | DPDK 19.05 | | DPDK 18.11 | + | | | OFED 4.6 | | OFED 4.5 | + | | | RDMA-CORE V24 | | RDMA-CORE V23 | + | | | ConnectX-5 | | ConnectX-4 | + +-----------------------+-----------------+-----------------+ + | Port ID | | DPDK 19.05 | | N/A | + | | | OFED 4.6 | | N/A | + | | | RDMA-CORE V24 | | N/A | + | | | ConnectX-5 | | N/A | + +-----------------------+-----------------+-----------------+ + +* Minimum version for each component and nic. + Notes for testpmd ----------------- @@ -863,7 +963,7 @@ Usage example ------------- This section demonstrates how to launch **testpmd** with Mellanox -ConnectX-4/ConnectX-5/ConnectX-6/Bluefield devices managed by librte_pmd_m= lx5. +ConnectX-4/ConnectX-5/ConnectX-6/BlueField devices managed by librte_pmd_m= lx5. #. Load the kernel modules: diff --git a/doc/guides/rel_notes/release_19_05.rst b/doc/guides/rel_notes/= release_19_05.rst index 4e0eed5..ce90f65 100644 --- a/doc/guides/rel_notes/release_19_05.rst +++ b/doc/guides/rel_notes/release_19_05.rst @@ -148,10 +148,13 @@ New Features * Added support for multiport InfiniBand device. * Added control of excessive memory pinning by kernel. * Added support of DMA memory registration by secondary process. - * Added Direct Rule support in Direct Verbs flow driver. * Added support of per-process device registers, reserving identical VA= space is not needed anymore. - * Added E-Switch support in Direct Verbs flow driver. + * Added support for jump action for both E-Switch and Nic. + * Added Support for multiple rte_flow groups in Nic steering. + * Flow engine re-design to support large scale deployments. this includ= es: + * Support millions of offloaded flow rules. + * Fast flow insertion and deletion up to 1M flow update per second. * **Renamed avf to iavf.** -- 1.8.3.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id 922F9A00E6 for ; Tue, 14 May 2019 07:47:31 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E28B62BCE; Tue, 14 May 2019 07:47:30 +0200 (CEST) Received: from smtp-4.sys.kth.se (smtp-4.sys.kth.se [130.237.48.193]) by dpdk.org (Postfix) with ESMTP id 98A6C2BB8 for ; Tue, 14 May 2019 07:47:28 +0200 (CEST) Received: from smtp-4.sys.kth.se (localhost.localdomain [127.0.0.1]) by smtp-4.sys.kth.se (Postfix) with ESMTP id 3AEFF29AE; Tue, 14 May 2019 07:47:28 +0200 (CEST) X-Virus-Scanned: by amavisd-new at kth.se Received: from smtp-4.sys.kth.se ([127.0.0.1]) by smtp-4.sys.kth.se (smtp-4.sys.kth.se [127.0.0.1]) (amavisd-new, port 10024) with LMTP id HCI2BHObTpbg; Tue, 14 May 2019 07:47:26 +0200 (CEST) Received: from exdb01.ug.kth.se (exdb01.ug.kth.se [192.168.32.111]) by smtp-4.sys.kth.se (Postfix) with ESMTPS id C41B729E2; Tue, 14 May 2019 07:47:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kth.se; s=default; t=1557812846; bh=+0W77fsnyh7aoUAZdGaKTLgoyKC6fV9BN0gJuemvfKE=; h=From:To:CC:Subject:Date:References:In-Reply-To; b=fX/HyhFoLKGCt0/eCltdW04eQ/hAW2JVSVpH/eSAaiSa3ZDGIqAQ1ItzDMJx8lZum kGeHoEsfwY2pEraavQp8qoBbdauE4lSgO9VRU9q3DzncwDzQjtD5GK4367vBZBJye8 fLo1NuV+tDqbNeS6X8fVWVjs0swfb3qMR8c/gwbM= Received: from exdb04.ug.kth.se (192.168.32.114) by exdb01.ug.kth.se (192.168.32.111) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 14 May 2019 07:47:25 +0200 Received: from exdb05.ug.kth.se (192.168.32.115) by exdb04.ug.kth.se (192.168.32.114) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 14 May 2019 07:47:24 +0200 Received: from exdb05.ug.kth.se ([192.168.32.115]) by exdb05.ug.kth.se ([192.168.32.115]) with mapi id 15.00.1473.003; Tue, 14 May 2019 07:47:24 +0200 From: Tom Barbette To: Ori Kam CC: "yskoh@mellanox.com" , "shahafs@mellanox.com" , "matan@mellanox.com" , "thomas@monjalon.net" , "dev@dpdk.org" Thread-Topic: [dpdk-dev] [PATCH v3] doc: fix update release notes for Mellanox drivers Thread-Index: AQHVCYVn0AmdCNjIoEO5tiVtsSCmT6ZqHfcu Date: Tue, 14 May 2019 05:47:24 +0000 Message-ID: <836d61b3-19e1-42cf-82bd-f7051970b2e7@email.android.com> References: <1557685859-121772-1-git-send-email-orika@mellanox.com>, <1557747662-188493-1-git-send-email-orika@mellanox.com> In-Reply-To: <1557747662-188493-1-git-send-email-orika@mellanox.com> Accept-Language: fr-FR, sv-SE, en-US Content-Language: fr-FR X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH v3] doc: fix update release notes for Mellanox drivers X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190514054724.VpYeiStJL1Us87M0nWWMf644SKjh0uDGoMouk_mOlqo@z> Hi all, I still find it a little bit unclear about direct rules. And therefore, how= to enable this large sclale mode. It is only available for eswitch ? Therefore PF rules will still be slow ? = How can I be sure to be in direct mode ? At some point someone mentioned performance of using a second group would b= e much faster than updating rules in the main group. I think having that s= omewhere in the doc, or internal Mellanox guide would be useful. Last point, is the requirement really OFED 4.6.2 for direct rules ? Because= it's still not publicly available. Thanks, Tom Le 13 mai 2019 14:14, Ori Kam a =E9crit : This patch adds some missing features to Mellanox drivers release notes. It also updates the mlx5/mlx4 documentations. Fixes: d85b204b5dba ("doc: update release notes for Mellanox drivers") Cc: yskoh@mellanox.com Signed-off-by: Ori Kam Acked-by: Shahaf Shuler --- v3: * Address ML comments. V2: * Fix checkpatch issues. --- doc/guides/nics/mlx4.rst | 2 +- doc/guides/nics/mlx5.rst | 184 +++++++++++++++++++++++++----= ---- doc/guides/rel_notes/release_19_05.rst | 7 +- 3 files changed, 148 insertions(+), 45 deletions(-) diff --git a/doc/guides/nics/mlx4.rst b/doc/guides/nics/mlx4.rst index f6d7a16..5c6bbde 100644 --- a/doc/guides/nics/mlx4.rst +++ b/doc/guides/nics/mlx4.rst @@ -253,7 +253,7 @@ thanks to these environment variables: Mellanox OFED as a fallback ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- `Mellanox OFED`_ version: **4.4, 4.5**. +- `Mellanox OFED`_ version: **4.4, 4.5, 4.6**. - firmware version: **2.42.5000** and above. .. _`Mellanox OFED`: http://www.mellanox.com/page/products_dyn?product_fam= ily=3D26&mtag=3Dlinux_sw_drivers diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 325e9f6..9540657 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -7,7 +7,7 @@ MLX5 poll mode driver The MLX5 poll mode driver library (**librte_pmd_mlx5**) provides support for **Mellanox ConnectX-4**, **Mellanox ConnectX-4 Lx** , **Mellanox -ConnectX-5**, **Mellanox ConnectX-6** and **Mellanox Bluefield** families +ConnectX-5**, **Mellanox ConnectX-6** and **Mellanox BlueField** families of 10/25/40/50/100/200 Gb/s adapters as well as their virtual functions (V= F) in SR-IOV context. @@ -62,8 +62,8 @@ Features - RX VLAN stripping. - TX VLAN insertion. - RX CRC stripping configuration. -- Promiscuous mode. -- Multicast promiscuous mode. +- Promiscuous mode on PF and VF. +- Multicast promiscuous mode on PF and VF. - Hardware checksum offloads. - Flow director (RTE_FDIR_MODE_PERFECT, RTE_FDIR_MODE_PERFECT_MAC_VLAN and RTE_ETH_FDIR_REJECT). @@ -78,6 +78,10 @@ Features - Rx HW timestamp. - Tunnel types: VXLAN, L3 VXLAN, VXLAN-GPE, GRE, MPLSoGRE, MPLSoUDP. - Tunnel HW offloads: packet type, inner/outer RSS, IP and UDP checksum ve= rification. +- Nic HW offloads: encapsulation (vxlan, gre, mplsoudp, mplsogre), NAT, ro= uting, TTL + increment/decrement, count, drop, mark. For details please see :ref:`Sup= ported hardware offloads using rte_flow API`. +- Flow insertion rate of more then million flows per second, when using Di= rect Rules. +- Support for multiple rte_flow groups. Limitations ----------- @@ -112,8 +116,6 @@ Limitations is set to multi-packet send or Enhanced multi-packet send. Otherwise it = must have less than 50 segments. -- Count action for RTE flow is **only supported in Mellanox OFED**. - - Flows with a VXLAN Network Identifier equal (or ends to be equal) to 0 are not supported. @@ -147,30 +149,16 @@ Limitations To receive IPv6 Multicast messages on VM, explicitly set the relevant MAC address using rte_eth_dev_mac_addr_add() API. -- E-Switch VXLAN tunnel is not supported together with outer VLAN. - -- E-Switch Flows with VNI pattern must include the VXLAN decapsulation act= ion. - -- E-Switch VXLAN decapsulation Flow: +- E-Switch decapsulation Flow: - can be applied to PF port only. - must specify VF port action (packet redirection from PF to VF). - - must specify tunnel outer UDP local (destination) port, wildcards not = allowed. - - must specify tunnel outer VNI, wildcards not allowed. - - must specify tunnel outer local (destination) IPv4 or IPv6 address, w= ildcards not allowed. - - optionally may specify tunnel outer remote (source) IPv4 or IPv6, wild= cards or group IPs allowed. - optionally may specify tunnel inner source and destination MAC address= es. -- E-Switch VXLAN encapsulation Flow: +- E-Switch encapsulation Flow: - can be applied to VF ports only. - must specify PF port action (packet redirection from VF to PF). - - must specify the VXLAN item with tunnel outer parameters. - - must specify the tunnel outer VNI in the VXLAN item. - - must specify the tunnel outer remote (destination) UDP port in the VXL= AN item. - - must specify the tunnel outer local (source) IPv4 or IPv6 in the , thi= s address will locally (with scope link) assigned to the outer network inte= rface, wildcards not allowed. - - must specify the tunnel outer remote (destination) IPv4 or IPv6 in the= VXLAN item, group IPs allowed. - - must specify the tunnel outer destination MAC address in the VXLAN ite= m, this address will be used to create neigh rule. Statistics ---------- @@ -227,7 +215,7 @@ These options can be modified in the ``.config`` file. .. note:: - For Bluefield, target should be set to ``arm64-bluefield-linux-gcc``. T= his + For BlueField, target should be set to ``arm64-bluefield-linux-gcc``. T= his will enable ``CONFIG_RTE_LIBRTE_MLX5_PMD`` and set ``RTE_CACHE_LINE_SIZ= E`` to 64. Default armv8a configuration of make build and meson build set it t= o 128 then brings performance degradation. @@ -277,8 +265,8 @@ Run-time configuration Supported on: - - x86_64 with ConnectX-4, ConnectX-4 LX, ConnectX-5, ConnectX-6 and Blue= field. - - POWER8 and ARMv8 with ConnectX-4 LX, ConnectX-5, ConnectX-6 and Bluefi= eld. + - x86_64 with ConnectX-4, ConnectX-4 LX, ConnectX-5, ConnectX-6 and Blue= Field. + - POWER9 and ARMv8 with ConnectX-4 LX, ConnectX-5, ConnectX-6 and BlueFi= eld. - ``rxq_cqe_pad_en`` parameter [int] @@ -296,7 +284,7 @@ Run-time configuration Supported on: - - CPU having 128B cacheline with ConnectX-5 and Bluefield. + - CPU having 128B cacheline with ConnectX-5 and BlueField. - ``rxq_pkt_pad_en`` parameter [int] @@ -308,8 +296,8 @@ Run-time configuration Supported on: - - x86_64 with ConnectX-4, ConnectX-4 LX, ConnectX-5, ConnectX-6 and Blue= field. - - POWER8 and ARMv8 with ConnectX-4 LX, ConnectX-5, ConnectX-6 and Bluefi= eld. + - x86_64 with ConnectX-4, ConnectX-4 LX, ConnectX-5, ConnectX-6 and Blue= Field. + - POWER8 and ARMv8 with ConnectX-4 LX, ConnectX-5, ConnectX-6 and BlueFi= eld. - ``mprq_en`` parameter [int] @@ -375,13 +363,13 @@ Run-time configuration This option should be used in combination with ``txq_inline`` above. - On ConnectX-4, ConnectX-4 LX, ConnectX-5, ConnectX-6 and Bluefield witho= ut + On ConnectX-4, ConnectX-4 LX, ConnectX-5, ConnectX-6 and BlueField witho= ut Enhanced MPW: - Disabled by default. - In case ``txq_inline`` is set recommendation is 4. - On ConnectX-5, ConnectX-6 and Bluefield with Enhanced MPW: + On ConnectX-5, ConnectX-6 and BlueField with Enhanced MPW: - Set to 8 by default. @@ -395,14 +383,14 @@ Run-time configuration - Set to 8 by default on ARMv8. - Set to 4 by default otherwise. - On Bluefield + On BlueField - Set to 16 by default. - ``txq_mpw_en`` parameter [int] A nonzero value enables multi-packet send (MPS) for ConnectX-4 Lx and - enhanced multi-packet send (Enhanced MPS) for ConnectX-5, ConnectX-6 and= Bluefield. + enhanced multi-packet send (Enhanced MPS) for ConnectX-5, ConnectX-6 and= BlueField. MPS allows the TX burst function to pack up multiple packets in a single descriptor session in order to save PCI bandwidth and improve performance at the cost of a slightly higher CPU usage. When @@ -417,13 +405,13 @@ Run-time configuration DEV_TX_OFFLOAD_VXLAN_TNL_TSO, DEV_TX_OFFLOAD_GRE_TNL_TSO, DEV_TX_OFFLOAD= _VLAN_INSERT``. When those offloads are requested the MPS send function will not be used= . - It is currently only supported on the ConnectX-4 Lx, ConnectX-5, Connect= X-6 and Bluefield + It is currently only supported on the ConnectX-4 Lx, ConnectX-5, Connect= X-6 and BlueField families of adapters. On ConnectX-4 Lx the MPW is considered un-secure hence disabled by defau= lt. Users which enable the MPW should be aware that application which provid= es incorrect mbuf descriptors in the Tx burst can lead to serious errors in the host = including, on some cases, NIC to get stuck. - On ConnectX-5, ConnectX-6 and Bluefield the MPW is secure and enabled by= default. + On ConnectX-5, ConnectX-6 and BlueField the MPW is secure and enabled by= default. - ``txq_mpw_hdr_dseg_en`` parameter [int] @@ -443,14 +431,14 @@ Run-time configuration - ``tx_vec_en`` parameter [int] - A nonzero value enables Tx vector on ConnectX-5, ConnectX-6 and Bluefiel= d NICs if the number of + A nonzero value enables Tx vector on ConnectX-5, ConnectX-6 and BlueFiel= d NICs if the number of global Tx queues on the port is less than ``txqs_max_vec``. This option cannot be used with certain offloads such as ``DEV_TX_OFFLOA= D_TCP_TSO, DEV_TX_OFFLOAD_VXLAN_TNL_TSO, DEV_TX_OFFLOAD_GRE_TNL_TSO, DEV_TX_OFFLOAD= _VLAN_INSERT``. When those offloads are requested the MPS send function will not be used= . - Enabled by default on ConnectX-5, ConnectX-6 and Bluefield. + Enabled by default on ConnectX-5, ConnectX-6 and BlueField. - ``rx_vec_en`` parameter [int] @@ -480,10 +468,15 @@ Run-time configuration A nonzero value enables the DV flow steering assuming it is supported by the driver. - The DV flow steering is not supported on switchdev mode. Disabled by default. +- ``dv_esw_en`` parameter [int] + + A nonzero value enables E-Switch using Direct Rules. + + Enabled by default if supported. + - ``mr_ext_memseg_en`` parameter [int] A nonzero value enables extending memseg when registering DMA memory. If @@ -545,7 +538,7 @@ DPDK and must be installed separately: - **libmlx5** Low-level user space driver library for Mellanox - ConnectX-4/ConnectX-5/ConnectX-6/Bluefield devices, it is automatically = loaded + ConnectX-4/ConnectX-5/ConnectX-6/BlueField devices, it is automatically = loaded by libibverbs. This library basically implements send/receive calls to the hardware @@ -567,7 +560,7 @@ DPDK and must be installed separately: their devices: - mlx5_core: hardware driver managing Mellanox - ConnectX-4/ConnectX-5/ConnectX-6/Bluefield devices and related Etherne= t kernel + ConnectX-4/ConnectX-5/ConnectX-6/BlueField devices and related Etherne= t kernel network devices. - mlx5_ib: InifiniBand device driver. - ib_uverbs: user space driver for Verbs (entry point for libibverbs). @@ -575,7 +568,7 @@ DPDK and must be installed separately: - **Firmware update** Mellanox OFED/EN releases include firmware updates for - ConnectX-4/ConnectX-5/ConnectX-6/Bluefield adapters. + ConnectX-4/ConnectX-5/ConnectX-6/BlueField adapters. Because each release provides new features, these updates must be applie= d to match the kernel modules and libraries they come with. @@ -622,7 +615,8 @@ thanks to these environment variables: Mellanox OFED/EN ^^^^^^^^^^^^^^^^ -- Mellanox OFED version: **4.4, 4.5** / Mellanox EN version: **4.5** +- Mellanox OFED version: ** 4.5, 4.6** / + Mellanox EN version: **4.5, 4.6** - firmware version: - ConnectX-4: **12.21.1000** and above. @@ -630,7 +624,7 @@ Mellanox OFED/EN - ConnectX-5: **16.21.1000** and above. - ConnectX-5 Ex: **16.21.1000** and above. - ConnectX-6: **20.99.5374** and above. - - Bluefield: **18.99.3950** and above. + - BlueField: **18.25.1010** and above. While these libraries and kernel modules are available on OpenFabrics Alliance's `website `__ and provided by pack= age @@ -766,6 +760,56 @@ Quick Start Guide on OFED/EN 6. Compile DPDK and you are ready to go. See instructions on :ref:`Development Kit Build System ` +Enable switchdev mode +--------------------- + +Switchdev mode is a mode in E-Switch, that binds between representor and V= F. +Representor is a port in DPDK that is connected to a VF in such a way +that assuming there are no offload flows, each packet that is sent from th= e VF +will be received by the corresponding representor. While each packet that = is +sent to a representor will be received by the VF. +This is very useful in case of SRIOV mode, where the first packet that is = sent +by the VF will be received by the DPDK application which will decide if th= is +flow should be offloaded to the E-Switch. After offloading the flow packet +that the VF that are matching the flow will not be received any more by +the DPDK application. + +1. Enable SRIOV mode: + + .. code-block:: console + + mlxconfig -d set SRIOV_EN=3Dtrue + +2. Configure the max number of VFs: + + .. code-block:: console + + mlxconfig -d set NUM_OF_VFS=3D + +3. Reset the FW: + + .. code-block:: console + + mlxfwreset -d reset + +3. Configure the actual number of VFs: + + .. code-block:: console + + echo /sys/class/net//device/sriov_numvfs + +4. Unbind the device (can be rebind after the switchdev mode): + + .. code-block:: console + + echo -n " /sys/bus/pci/drivers/mlx5_core/unb= ind + +5. Enbale switchdev mode: + + .. code-block:: console + + echo switchdev > /sys/class/net//compat/devlink/mode + Performance tuning ------------------ @@ -842,6 +886,62 @@ Performance tuning - Configure per-lcore cache when creating Mempools for packet buffer. - Refrain from dynamically allocating/freeing memory in run-time. +Supported hardware offloads using rte_flow API +---------------------------------------------- + +.. _Supported hardware offloads using rte_flow API: + +.. table:: Supported hardware offloads using rte_flow API + + +-----------------------+-----------------+-----------------+ + | Offload | E-Switch | NIC | + | | | | + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+ + | Count | | DPDK 19.05 | | DPDK 19.02 | + | | | OFED 4.6 | | OFED 4.6 | + | | | RDMA-CORE V24 | | RDMA-CORE V23 | + | | | ConnectX-5 | | ConnectX-5 | + +-----------------------+-----------------+-----------------+ + | Drop / Queue / RSS | | DPDK 19.05 | | DPDK 18.11 | + | | | OFED 4.6 | | OFED 4.5 | + | | | RDMA-CORE V24 | | RDMA-CORE V23 | + | | | ConnectX-5 | | ConnectX-4 | + +-----------------------+-----------------+-----------------+ + | Encapsulation | | DPDK 19.05 | | DPDK 19.02 | + | (VXLAN / NVGRE / RAW) | | OFED 4.6.2 | | OFED 4.6 | + | | | RDMA-CORE V24 | | RDMA-CORE V23 | + | | | ConnectX-5 | | ConnectX-5 | + +-----------------------+-----------------+-----------------+ + | Header rewrite | | DPDK 19.05 | | DPDK 19.02 | + | (set_ipv4_src / | | OFED 4.6.2 | | OFED 4.6.2 | + | set_ipv4_dst / | | RDMA-CORE V24 | | RDMA-CORE V23 | + | set_ipv6_src / | | ConnectX-5 | | ConnectX-5 | + | set_ipv6_dst / | | | + | set_tp_src / | | | + | set_tp_dst / | | | + | dec_ttl / | | | + | set_ttl / | | | + | set_mac_src / | | | + | set_mac_dst) | | | + +-----------------------+-----------------+-----------------+ + | Jump | | DPDK 19.05 | | DPDK 19.02 | + | | | OFED 4.6.2 | | OFED 4.6.2 | + | | | RDMA-CORE V24 | | N/A | + | | | ConnectX-5 | | ConnectX-5 | + +-----------------------+-----------------+-----------------+ + | Mark / Flag | | DPDK 19.05 | | DPDK 18.11 | + | | | OFED 4.6 | | OFED 4.5 | + | | | RDMA-CORE V24 | | RDMA-CORE V23 | + | | | ConnectX-5 | | ConnectX-4 | + +-----------------------+-----------------+-----------------+ + | Port ID | | DPDK 19.05 | | N/A | + | | | OFED 4.6 | | N/A | + | | | RDMA-CORE V24 | | N/A | + | | | ConnectX-5 | | N/A | + +-----------------------+-----------------+-----------------+ + +* Minimum version for each component and nic. + Notes for testpmd ----------------- @@ -863,7 +963,7 @@ Usage example ------------- This section demonstrates how to launch **testpmd** with Mellanox -ConnectX-4/ConnectX-5/ConnectX-6/Bluefield devices managed by librte_pmd_m= lx5. +ConnectX-4/ConnectX-5/ConnectX-6/BlueField devices managed by librte_pmd_m= lx5. #. Load the kernel modules: diff --git a/doc/guides/rel_notes/release_19_05.rst b/doc/guides/rel_notes/= release_19_05.rst index 4e0eed5..ce90f65 100644 --- a/doc/guides/rel_notes/release_19_05.rst +++ b/doc/guides/rel_notes/release_19_05.rst @@ -148,10 +148,13 @@ New Features * Added support for multiport InfiniBand device. * Added control of excessive memory pinning by kernel. * Added support of DMA memory registration by secondary process. - * Added Direct Rule support in Direct Verbs flow driver. * Added support of per-process device registers, reserving identical VA= space is not needed anymore. - * Added E-Switch support in Direct Verbs flow driver. + * Added support for jump action for both E-Switch and Nic. + * Added Support for multiple rte_flow groups in Nic steering. + * Flow engine re-design to support large scale deployments. this includ= es: + * Support millions of offloaded flow rules. + * Fast flow insertion and deletion up to 1M flow update per second. * **Renamed avf to iavf.** -- 1.8.3.1