Re: [PATCH] doc: update NFP documentation with Corigine information

["Niklas Söderlund" <niklas.soderlund@corigine.com>]

Hello Ferruh,

Thanks for your feedback.

On 2023-02-15 13:37:05 +0000, Ferruh Yigit wrote:
> On 2/3/2023 8:08 AM, Chaoyong He wrote:
> > From: Walter Heymans <walter.heymans@corigine.com>
> > 
> > The NFP PMD documentation is updated to include information about
> > Corigine and their new vendor device ID.
> > 
> > Outdated information regarding the use of the PMD is also updated.
> > 
> > While making major changes to the document, the maximum number of
> > characters per line is updated to 80 characters to improve the
> > readability in raw format.
> > 
> 
> There are three groups of changes done to documentation as explained in
> three paragraphs above.
> 
> To help review, is it possible to separate this patch into three
> patches? Later they can be squashed and merged as a single patch.
> But as it is, easy to miss content changes among formatting changes.
> 
> (You can include simple grammar updates (that doesn't change either
> content or Corigine related information) to formatting update patch)

We will break this patch in to three as you suggest, address the 
comments below and post a v2.

> 
> 
> > Signed-off-by: Walter Heymans <walter.heymans@corigine.com>
> > Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
> > Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> > ---
> >  doc/guides/nics/nfp.rst | 168 +++++++++++++++++++++-------------------
> >  1 file changed, 90 insertions(+), 78 deletions(-)
> > 
> > diff --git a/doc/guides/nics/nfp.rst b/doc/guides/nics/nfp.rst
> > index a085d7d9ae..6fea280411 100644
> > --- a/doc/guides/nics/nfp.rst
> > +++ b/doc/guides/nics/nfp.rst
> > @@ -1,35 +1,34 @@
> >  ..  SPDX-License-Identifier: BSD-3-Clause
> >      Copyright(c) 2015-2017 Netronome Systems, Inc. All rights reserved.
> > -    All rights reserved.
> > +    Copyright(c) 2021 Corigine, Inc. All rights reserved.
> >  
> >  NFP poll mode driver library
> >  ============================
> >  
> > -Netronome's sixth generation of flow processors pack 216 programmable
> > -cores and over 100 hardware accelerators that uniquely combine packet,
> > -flow, security and content processing in a single device that scales
> > +Netronome and Corigine's sixth generation of flow processors pack 216
> > +programmable cores and over 100 hardware accelerators that uniquely combine
> > +packet, flow, security and content processing in a single device that scales
> >  up to 400-Gb/s.
> >  
> > -This document explains how to use DPDK with the Netronome Poll Mode
> > -Driver (PMD) supporting Netronome's Network Flow Processor 6xxx
> > -(NFP-6xxx), Netronome's Network Flow Processor 4xxx (NFP-4xxx) and
> > -Netronome's Network Flow Processor 38xx (NFP-38xx).
> > +This document explains how to use DPDK with the Network Flow Processor (NFP)
> > +Poll Mode Driver (PMD) supporting Netronome and Corigine's NFP-6xxx, NFP-4xxx
> > +and NFP-38xx product lines.
> >  
> > -NFP is a SRIOV capable device and the PMD supports the physical
> > -function (PF) and the virtual functions (VFs).
> > +NFP is a SR-IOV capable device and the PMD supports the physical function (PF)
> > +and the virtual functions (VFs).
> >  
> >  Dependencies
> >  ------------
> >  
> > -Before using the Netronome's DPDK PMD some NFP configuration,
> > -which is not related to DPDK, is required. The system requires
> > -installation of **Netronome's BSP (Board Support Package)** along
> > -with a specific NFP firmware application. Netronome's NSP ABI
> > -version should be 0.20 or higher.
> > +Before using the NFP DPDK PMD some NFP configuration, which is not related to
> > +DPDK, is required. The system requires installation of
> > +**NFP-BSP (Board Support Package)** along with a specific NFP firmware
> > +application. The NSP ABI version should be 0.20 or higher.
> >  
> > -If you have a NFP device you should already have the code and
> > -documentation for this configuration. Contact
> > -**support@netronome.com** to obtain the latest available firmware.
> > +If you have a NFP device you should already have the documentation to perform
> > +this configuration. Contact **support@netronome.com** (for Netronome products)
> > +or **smartnic-support@corigine.com** (for Corigine products) to obtain the
> > +latest available firmware.
> >  
> >  The NFP Linux netdev kernel driver for VFs has been a part of the
> >  vanilla kernel since kernel version 4.5, and support for the PF
> > @@ -44,11 +43,11 @@ Linux kernel driver.
> >  Building the software
> >  ---------------------
> >  
> > -Netronome's PMD code is provided in the **drivers/net/nfp** directory.
> > -Although NFP PMD has Netronome´s BSP dependencies, it is possible to
> > -compile it along with other DPDK PMDs even if no BSP was installed previously.
> > -Of course, a DPDK app will require such a BSP installed for using the
> > -NFP PMD, along with a specific NFP firmware application.
> > +The NFP PMD code is provided in the **drivers/net/nfp** directory. Although
> > +NFP PMD has BSP dependencies, it is possible to compile it along with other
> > +DPDK PMDs even if no BSP was installed previously. Of course, a DPDK app will
> > +require such a BSP installed for using the NFP PMD, along with a specific NFP
> > +firmware application.
> >  
> >  Once the DPDK is built all the DPDK apps and examples include support for
> >  the NFP PMD.
> > @@ -57,27 +56,20 @@ the NFP PMD.
> >  Driver compilation and testing
> >  ------------------------------
> >  
> > -Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
> > -for details.
> > +Refer to the document
> > +:ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` for details.
> >  
> >  Using the PF
> >  ------------
> >  
> > -NFP PMD supports using the NFP PF as another DPDK port, but it does not
> > -have any functionality for controlling VFs. In fact, it is not possible to use
> > -the PMD with the VFs if the PF is being used by DPDK, that is, with the NFP PF
> > -bound to ``igb_uio`` or ``vfio-pci`` kernel drivers. Future DPDK versions will
> > -have a PMD able to work with the PF and VFs at the same time and with the PF
> > -implementing VF management along with other PF-only functionalities/offloads.
> > -
> 
> Why this paragraph is removed? Is it because it is not correct anymore,
> or just because of document organization change.
> 
> >  The PMD PF has extra work to do which will delay the DPDK app initialization
> > -like uploading the firmware and configure the Link state properly when starting or
> > -stopping a PF port. Since DPDK 18.05 the firmware upload happens when
> > +like uploading the firmware and configure the Link state properly when starting
> > +or stopping a PF port. Since DPDK 18.05 the firmware upload happens when
> >  a PF is initialized, which was not always true with older DPDK versions.
> >  
> > -Depending on the Netronome product installed in the system, firmware files
> > -should be available under ``/lib/firmware/netronome``. DPDK PMD supporting the
> > -PF looks for a firmware file in this order:
> > +Depending on the product installed in the system, firmware files should be
> > +available under ``/lib/firmware/netronome``. DPDK PMD supporting the PF looks
> > +for a firmware file in this order:
> >  
> >  	1) First try to find a firmware image specific for this device using the
> >  	   NFP serial number:
> > @@ -92,18 +84,21 @@ PF looks for a firmware file in this order:
> >  
> >  		nic_AMDA0099-0001_2x25.nffw
> >  
> > -Netronome's software packages install firmware files under ``/lib/firmware/netronome``
> > -to support all the Netronome's SmartNICs and different firmware applications.
> > -This is usually done using file names based on SmartNIC type and media and with a
> > -directory per firmware application. Options 1 and 2 for firmware filenames allow
> > -more than one SmartNIC, same type of SmartNIC or different ones, and to upload a
> > -different firmware to each SmartNIC.
> > +Netronome and Corigine's software packages install firmware files under
> > +``/lib/firmware/netronome`` to support all the SmartNICs and different firmware
> > +applications. This is usually done using file names based on SmartNIC type and
> > +media and with a directory per firmware application. Options 1 and 2 for
> > +firmware filenames allow more than one SmartNIC, same type of SmartNIC or
> > +different ones, and to upload a different firmware to each SmartNIC.
> >  
> >     .. Note::
> > -      Currently the NFP PMD supports using the PF with Agilio Firmware with NFD3
> > -      and Agilio Firmware with NFDk. See https://help.netronome.com/support/solutions
> > +      Currently the NFP PMD supports using the PF with Agilio Firmware with
> > +      NFD3 and Agilio Firmware with NFDk. See
> > +      `Netronome Support <https://help.netronome.com/support/solutions>`_.
> >        for more information on the various firmwares supported by the Netronome
> > -      Agilio CX smartNIC.
> > +      Agilio SmartNICs range, or
> > +      `Corigine Support <https://www.corigine.com/productsOverviewList-30.html>`_.
> > +      for more information about Corigine's range.
> >  
> >  PF multiport support
> >  --------------------
> > @@ -118,7 +113,7 @@ this particular configuration requires the PMD to create ports in a special way,
> >  although once they are created, DPDK apps should be able to use them as normal
> >  PCI ports.
> >  
> > -NFP ports belonging to same PF can be seen inside PMD initialization with a
> > +NFP ports belonging to the same PF can be seen inside PMD initialization with a
> >  suffix added to the PCI ID: wwww:xx:yy.z_portn. For example, a PF with PCI ID
> >  0000:03:00.0 and four ports is seen by the PMD code as:
> >  
> > @@ -137,50 +132,67 @@ suffix added to the PCI ID: wwww:xx:yy.z_portn. For example, a PF with PCI ID
> >  PF multiprocess support
> >  -----------------------
> >  
> > -Due to how the driver needs to access the NFP through a CPP interface, which implies
> > -to use specific registers inside the chip, the number of secondary processes with PF
> > -ports is limited to only one.
> > +Due to how the driver needs to access the NFP through a CPP interface, which
> > +implies to use specific registers inside the chip, the number of secondary
> > +processes with PF ports is limited to only one.
> >  
> > -This limitation will be solved in future versions but having basic multiprocess support
> > -is important for allowing development and debugging through the PF using a secondary
> > -process which will create a CPP bridge for user space tools accessing the NFP.
> > +This limitation will be solved in future versions, but having basic
> > +multiprocess support is important for allowing development and debugging
> > +through the PF using a secondary process, which will create a CPP bridge
> > +for user space tools accessing the NFP.
> >  
> >  
> >  System configuration
> >  --------------------
> >  
> >  #. **Enable SR-IOV on the NFP device:** The current NFP PMD supports the PF and
> > -   the VFs on a NFP device. However, it is not possible to work with both at the
> > -   same time because the VFs require the PF being bound to the NFP PF Linux
> > -   netdev driver.  Make sure you are working with a kernel with NFP PF support or
> > -   get the drivers from the above Github repository and follow the instructions
> > -   for building and installing it.
> > +   the VFs on a NFP device. However, it is not possible to work with both at
> > +   the same time when using the netdev NFP Linux netdev driver.
> 
> Old and new text doesn't say same thing.
> Old one says: "For DPDK to support VF, PF needs to bound to kernel driver.:
> 
> Is this changed, or just wording mistake?
> 
> 
> >     It is possible
> > +   to bind the PF to the ``vfio-pci`` kernel module, and create VFs afterwards.
> > +   This requires loading the ``vfio-pci`` module with the following parameters:
> > +
> > +   .. code-block:: console
> > +
> > +      modprobe vfio-pci enable_sriov=1 disable_idle_d3=1
> > +
> > +   VFs need to be enabled before they can be used with the PMD. Before enabling
> > +   the VFs it is useful to obtain information about the current NFP PCI device
> > +   detected by the system. This can be done on Netronome SmartNICs using:
> > +
> > +   .. code-block:: console
> > +
> > +      lspci -d 19ee:
> >  
> 
> What I understand is, to support VF by DPDK two things are required:
> 1) Ability to create VFs, this can be done both by using device's kernel
> driver or 'vfio-pci'
> 2) PF driver should support managing VFs.
> 
> Above lines document about item (1) and how 'vfio-pci' is used for it.
> 
> But old documentation mentions about item (2) is missing, why that part
> removed, isn't it valid anymore? I mean is "PF -> kernel, VF -> DPDK"
> combination supported now?
> 
> 
> > -   VFs need to be enabled before they can be used with the PMD.
> > -   Before enabling the VFs it is useful to obtain information about the
> > -   current NFP PCI device detected by the system:
> > +   and on Corigine SmartNICs using:
> >  
> >     .. code-block:: console
> >  
> > -      lspci -d19ee:
> > +      lspci -d 1da8:
> >  
> > -   Now, for example, configure two virtual functions on a NFP-6xxx device
> > +   Now, for example, to configure two virtual functions on a NFP device
> >     whose PCI system identity is "0000:03:00.0":
> >  
> >     .. code-block:: console
> >  
> >        echo 2 > /sys/bus/pci/devices/0000:03:00.0/sriov_numvfs
> >  
> > -   The result of this command may be shown using lspci again:
> > +   The result of this command may be shown using lspci again on Netronome
> > +   SmartNICs:
> > +
> > +   .. code-block:: console
> > +
> > +      lspci -d 19ee: -k
> > +
> > +   and on Corigine SmartNICs:
> >  
> >     .. code-block:: console
> >  
> > -      lspci -d19ee: -k
> > +      lspci -d 1da8: -k
> >  
> >     Two new PCI devices should appear in the output of the above command. The
> > -   -k option shows the device driver, if any, that devices are bound to.
> > -   Depending on the modules loaded at this point the new PCI devices may be
> > -   bound to nfp_netvf driver.
> > +   -k option shows the device driver, if any, that the devices are bound to.
> > +   Depending on the modules loaded, at this point the new PCI devices may be
> > +   bound to the ``nfp`` kernel driver or ``vfio-pci``.
> >  
> >  
> >  Flow offload
> > @@ -193,13 +205,13 @@ The flower firmware application requires the PMD running two services:
> >  
> >  	* PF vNIC service: handling the feedback traffic.
> >  	* ctrl vNIC service: communicate between PMD and firmware through
> > -	  control message.
> > +	  control messages.
> >  
> >  To achieve the offload of flow, the representor ports are exposed to OVS.
> > -The flower firmware application support representor port for VF and physical
> > -port. There will always exist a representor port for each physical port,
> > -and the number of the representor port for VF is specified by the user through
> > -parameter.
> > +The flower firmware application supports VF, PF, and physical port representor
> > +ports. 
> 
> Again old document and new one is not saying same thing, is it intentional?
> 
> Old one says: "Having representor ports for both VF and PF is supported."
> 
> New one says: "FW supports representor port, VF and PF."
> 
> > There will always exist a representor port for a PF and each physical
> > +port. The number of the representor ports for VFs are specified by the user
> > +through a parameter.
> >  
> >  In the Rx direction, the flower firmware application will prepend the input
> >  port information into metadata for each packet which can't offloaded. The PF
> > @@ -207,12 +219,12 @@ vNIC service will keep polling packets from the firmware, and multiplex them
> >  to the corresponding representor port.
> >  
> >  In the Tx direction, the representor port will prepend the output port
> > -information into metadata for each packet, and then send it to firmware through
> > -PF vNIC.
> > +information into metadata for each packet, and then send it to the firmware
> > +through the PF vNIC.
> >  
> > -The ctrl vNIC service handling various control message, like the creation and
> > -configuration of representor port, the pattern and action of flow rules, the
> > -statistics of flow rules, and so on.
> > +The ctrl vNIC service handles various control messages, for example, the
> > +creation and configuration of a representor port, the pattern and action of
> > +flow rules, the statistics of flow rules, etc.
> >  
> >  Metadata Format
> >  ---------------
> 

-- 
Kind Regards,
Niklas Söderlund