[dpdk-dev] [PATCH v9 8/9] nfp: adding nic guide

[Alejandro Lucero <alejandro.lucero@netronome.com>]

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@netronome.com>
---
 doc/guides/nics/index.rst |    1 +
 doc/guides/nics/nfp.rst   |  265 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 266 insertions(+)
 create mode 100644 doc/guides/nics/nfp.rst

diff --git a/doc/guides/nics/index.rst b/doc/guides/nics/index.rst
index 0a0b724..7bf2938 100644
--- a/doc/guides/nics/index.rst
+++ b/doc/guides/nics/index.rst
@@ -46,6 +46,7 @@ Network Interface Controller Drivers
     intel_vf
     mlx4
     mlx5
+    nfp
     szedata2
     virtio
     vmxnet3
diff --git a/doc/guides/nics/nfp.rst b/doc/guides/nics/nfp.rst
new file mode 100644
index 0000000..55ba64d
--- /dev/null
+++ b/doc/guides/nics/nfp.rst
@@ -0,0 +1,265 @@
+..  BSD LICENSE
+    Copyright(c) 2015 Netronome Systems, Inc. All rights reserved.
+    All rights reserved.
+
+    Redistribution and use in source and binary forms, with or without
+    modification, are permitted provided that the following conditions
+    are met:
+
+    * Redistributions of source code must retain the above copyright
+    notice, this list of conditions and the following disclaimer.
+    * Redistributions in binary form must reproduce the above copyright
+    notice, this list of conditions and the following disclaimer in
+    the documentation and/or other materials provided with the
+    distribution.
+    * Neither the name of Intel Corporation nor the names of its
+    contributors may be used to endorse or promote products derived
+    from this software without specific prior written permission.
+
+    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+NFP poll mode driver library
+============================
+
+Netronome's sixth generation of flow processors pack 216 programmable
+cores and over 100 hardware accelerators that uniquely combine packet,
+flow, security and content processing in a single device that scales
+up to 400 Gbps.
+
+This document explains how to use DPDK with the Netronome Poll Mode
+Driver (PMD) supporting Netronome's Network Flow Processor 6xxx
+(NFP-6xxx).
+
+Currently the driver supports virtual functions (VFs) only.
+
+Dependencies
+------------
+
+Before using the Netronome's DPDK PMD some NFP-6xxx configuration,
+which is not related to DPDK, is required. The system requires
+installation of **Netronome's BSP (Board Support Package)** which includes
+Linux drivers, programs and libraries.
+
+If you have a NFP-6xxx device you should already have the code and
+documentation for doing this configuration. Contact
+**support@netronome.com** to obtain the latest available firmware.
+
+The NFP Linux kernel drivers (including the required PF driver for the
+NFP) are available on Github at
+**https://github.com/Netronome/nfp-drv-kmods** along with build
+instructions.
+
+DPDK runs in userspace and PMDs uses the Linux kernel UIO interface to
+allow access to physical devices from userspace. The NFP PMD requires
+a separate UIO driver, **nfp_uio**, to perform correct
+initialization. This driver is part of Netronome´s BSP and it is
+equivalent to Intel's igb_uio driver.
+
+Building the software
+---------------------
+
+Netronome's PMD code is provided in the **drivers/net/nfp** directory.
+Because Netronome´s BSP dependencies the driver is disabled by default
+in DPDK build using **common_linuxapp configuration** file. Enabling the
+driver or if you use another configuration file and want to have NFP
+support, this variable is needed:
+
+- **CONFIG_RTE_LIBRTE_NFP_PMD=y**
+
+Once DPDK is built all the DPDK apps and examples include support for
+the NFP PMD.
+
+
+System configuration
+--------------------
+
+Using the NFP PMD is not different to using other PMDs. Usual steps are:
+
+#. **Configure hugepages:** All major Linux distributions have the hugepages
+   functionality enabled by default. By default this allows the system uses for
+   working with transparent hugepages. But in this case some hugepages need to
+   be created/reserved for use with the DPDK through the hugetlbfs file system.
+   First the virtual file system need to be mounted:
+
+   .. code-block:: console
+
+      mount -t hugetlbfs none /mnt/hugetlbfs
+
+   The command uses the common mount point for this file system and it needs to
+   be created if necessary.
+
+   Configuring hugepages is performed via sysfs:
+
+   .. code-block:: console
+
+      /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
+
+   This sysfs file is used to specify the number of hugepages to reserve.
+   For example:
+
+   .. code-block:: console
+
+      echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
+
+   This will reserve 2GB of memory using 1024 2MB hugepages. The file may be
+   read to see if the operation was performed correctly:
+
+   .. code-block:: console
+
+      cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
+
+   The number of unused hugepages may also be inspected.
+
+   Before executing the DPDK app it should match the value of nr_hugepages.
+
+   .. code-block:: console
+
+      cat /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages
+
+   The hugepages reservation should be performed at system initialisation and
+   it is usual to use a kernel parameter for configuration. If the reservation
+   is attempted on a busy system it will likely fail. Reserving memory for
+   hugepages may be done adding the following to the grub kernel command line:
+
+   .. code-block:: console
+
+      default_hugepagesz=1M hugepagesz=2M hugepages=1024
+
+   This will reserve 2GBytes of memory using 2Mbytes huge pages.
+
+   Finally, for a NUMA system the allocation needs to be made on the correct
+   NUMA node. In a DPDK app there is a master core which will (usually) perform
+   memory allocation. It is important that some of the hugepages are reserved
+   on the NUMA memory node where the network device is attached. This is because
+   of a restriction in DPDK by which TX and RX descriptors rings must be created
+   on the master code.
+
+   Per-node allocation of hugepages may be inspected and controlled using sysfs.
+   For example:
+
+   .. code-block:: console
+
+      cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
+
+   For a NUMA system there will be a specific hugepage directory per node
+   allowing control of hugepage reservation. A common problem may occur when
+   hugepages reservation is performed after the system has been working for
+   some time. Configuration using the global sysfs hugepage interface will
+   succeed but the per-node allocations may be unsatisfactory.
+
+   The number of hugepages that need to be reserved depends on how the app uses
+   TX and RX descriptors, and packets mbufs.
+
+#. **Enable SR-IOV on the NFP-6xxx device:** The current NFP PMD works with
+   Virtual Functions (VFs) on a NFP device. Make sure that one of the Physical
+   Function (PF) drivers from the above Github repository is installed and
+   loaded.
+
+   Virtual Functions need to be enabled before they can be used with the PMD.
+   Before enabling the VFs it is useful to obtain information about the
+   current NFP PCI device detected by the system:
+
+   .. code-block:: console
+
+      lspci -d19ee:
+
+   Now, for example, configure two virtual functions on a NFP-6xxx device
+   whose PCI system identity is "0000:03:00.0":
+
+   .. code-block:: console
+
+      echo 2 > /sys/bus/pci/devices/0000:03:00.0/sriov_numvfs
+
+   The result of this command may be shown using lspci again:
+
+   .. code-block:: console
+
+      lspci -d19ee: -k
+
+   Two new PCI devices should appear in the output of the above command. The
+   -k option shows the device driver, if any, that devices are bound to.
+   Depending on the modules loaded at this point the new PCI devices may be
+   bound to nfp_netvf driver.
+
+#. **To install the uio kernel module (manually):** All major Linux
+   distributions have support for this kernel module so it is straightforward
+   to install it:
+
+   .. code-block:: console
+
+      modprobe uio
+
+   The module should now be listed by the lsmod command.
+
+#. **To install the nfp_uio kernel module (manually):** This module supports
+   NFP-6xxx devices through the UIO interface.
+
+   This module is part of Netronome´s BSP and it should be available when the
+   BSP is installed.
+
+   .. code-block:: console
+
+      modprobe nfp_uio.ko
+
+   The module should now be listed by the lsmod command.
+
+   Depending on which NFP modules are loaded, nfp_uio may be automatically
+   bound to the NFP PCI devices by the system. Otherwise the binding needs
+   to be done explicitly. This is the case when nfp_netvf, the Linux kernel
+   driver for NFP VFs, was loaded when VFs were created. As described later
+   in this document this configuration may also be performed using scripts
+   provided by the Netronome´s BSP.
+
+   First the device needs to be unbound, for example from the nfp_netvf
+   driver:
+
+   .. code-block:: console
+
+      echo 0000:03:08.0 > /sys/bus/pci/devices/0000:03:08.0/driver/unbind
+
+      lspci -d19ee: -k
+
+   The output of lspci should now show that 0000:03:08.0 is not bound to
+   any driver.
+
+   The next step is to add the NFP PCI ID to the NFP UIO driver:
+
+   .. code-block:: console
+
+      echo 19ee 6003 > /sys/bus/pci/drivers/nfp_uio/new_id
+
+   And then to bind the device to the nfp_uio driver:
+
+   .. code-block:: console
+
+      echo 0000:03:08.0 > /sys/bus/pci/drivers/nfp_uio/bind
+
+      lspci -d19ee: -k
+
+   lspci should show that device bound to nfp_uio driver.
+
+#. **Using tools from Netronome´s BSP to install and bind modules:** DPDK provides
+   scripts which are useful for installing the UIO modules and for binding the
+   right device to those modules avoiding doing so manually. However, these scripts
+   have not support for Netronome´s UIO driver. Along with drivers, the BSP installs
+   those DPDK scripts slightly modified with support for Netronome´s UIO driver.
+
+   Those specific scripts can be found in Netronome´s BSP installation directory.
+   Refer to BSP documentation for more information.
+
+   * **setup.sh**
+   * **dpdk_nic_bind.py**
+
+   Configuration may be performed by running setup.sh which invokes
+   dpdk_nic_bind.py as needed. Executing setup.sh will display a menu of
+   configuration options.
-- 
1.7.9.5