From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ubuntu (host217-39-174-19.in-addr.btopenworld.com [217.39.174.19]) by dpdk.org (Postfix) with SMTP id 00DFE5699 for ; Fri, 16 Oct 2015 12:45:11 +0200 (CEST) Received: by ubuntu (Postfix, from userid 5466) id 9F800E92E5; Fri, 16 Oct 2015 11:45:24 +0100 (BST) From: "Alejandro.Lucero" To: dev@dpdk.org Date: Fri, 16 Oct 2015 11:45:23 +0100 Message-Id: <1444992324-5504-4-git-send-email-alejandro.lucero@netronome.com> X-Mailer: git-send-email 1.7.9.5 In-Reply-To: <1444992324-5504-1-git-send-email-alejandro.lucero@netronome.com> References: <1444992324-5504-1-git-send-email-alejandro.lucero@netronome.com> Subject: [dpdk-dev] [PATCH v3 3/4] doc: add netronome nfp6000 guide X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Oct 2015 10:45:12 -0000 From: "Alejandro.Lucero" Signed-off-by: Alejandro.Lucero Signed-off-by: Rolf.Neugebauer --- doc/guides/nics/index.rst | 1 + doc/guides/nics/nfp.rst | 270 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 271 insertions(+) create mode 100644 doc/guides/nics/nfp.rst diff --git a/doc/guides/nics/index.rst b/doc/guides/nics/index.rst index d1a92f8..596ff88 100644 --- a/doc/guides/nics/index.rst +++ b/doc/guides/nics/index.rst @@ -48,6 +48,7 @@ Network Interface Controller Drivers virtio vmxnet3 pcap_ring + nfp **Figures** diff --git a/doc/guides/nics/nfp.rst b/doc/guides/nics/nfp.rst new file mode 100644 index 0000000..57b34c6 --- /dev/null +++ b/doc/guides/nics/nfp.rst @@ -0,0 +1,270 @@ +.. BSD LICENSE + Copyright(c) 2015 Netronome Systems, Inc. All rights reserved. + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + * Neither the name of Intel Corporation nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +NFP poll mode driver library +============================ + +Netronome's sixth generation of flow processors pack 216 programmable +cores and over 100 hardware accelerators that uniquely combine packet, +flow, security and content processing in a single device that scales +up to 400 Gbps. + +This document explains how to use DPDK with the Netronome Poll Mode +Driver (PMD) supporting Netronome's Network Flow Processor 6xxx +(NFP-6xxx). + +Currently the driver supports virtual functions (VFs) only. + +Dependencies +------------ + +Before using the Netronome's DPDK PMD some NFP-6xxx configuration, +which is not related to DPDK, is required. The system requires +installation of **Netronome's BSP (Board Support Package)** which includes +Linux drivers, programs and libraries. + +If you have a NFP-6xxx device you should already have the code and +documentation for doing this configuration. Contact +**support@netronome.com** to obtain the latest available firmware. + +The NFP Linux kernel drivers (including the required PF driver for the +NFP) are available on Github at +**https://github.com/Netronome/nfp-drv-kmods** along with build +instructions. + +DPDK runs in userspace and PMDs uses the Linux kernel UIO interface to +allow access to physical devices from userspace. The NFP PMD requires +a separate UIO driver, **nfp_uio**, to perform correct +initialization. This driver is part of the DPDK source tree and is +equivalent to Intel's igb_uio driver. + +Building the software +--------------------- + +Netronome's PMD code is provided in the **drivers/net/nfp** directory and +nfp_uio is present in the **lib/librte_eal/linuxapp/nfp_uio** directory. Both +are part of the DPDK build if the **common_linuxapp configuration** file is +used. If you use another configuration file and want to have NFP support +just add: + +- **CONFIG_RTE_EAL_NFP_UIO=y** + +- **CONFIG_RTE_LIBRTE_NFP_PMD=y** + +Once DPDK is built all the DPDK apps and examples include support for +the NFP PMD. The nfp_uio.ko module will be at build/kmods directory or +at the directory specified when building DPDK. + + +System configuration +-------------------- + +Using the NFP PMD is not different to using other PMDs. Usual steps are: + +#. **Configure hugepages:** All major Linux distributions have the hugepages + functionality enabled by default. By default this allows the system uses for + working with transparent hugepages. But in this case some hugepages need to + be created/reserved for use with the DPDK through the hugetlbfs file system. + First the virtual file system need to be mounted: + + .. code-block:: console + + mount -t hugetlbfs none /mnt/hugetlbfs + + The command uses the common mount point for this file system and it needs to + be created if necessary. + + Configuring hugepages is performed via sysfs: + + .. code-block:: console + + /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages + + This sysfs file is used to specify the number of hugepages to reserve. + For example: + + .. code-block:: console + + echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages + + This will reserve 2GB of memory using 1024 2MB hugepages. The file may be + read to see if the operation was performed correctly: + + .. code-block:: console + + cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages + + The number of unused hugepages may also be inspected. + + Before executing the DPDK app it should match the value of nr_hugepages. + + .. code-block:: console + + cat /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages + + The hugepages reservation should be performed at system initialisation and + it is usual to use a kernel parameter for configuration. If the reservation + is attempted on a busy system it will likely fail. Reserving memory for + hugepages may be done adding the following to the grub kernel command line: + + .. code-block:: console + + default_hugepagesz=1M hugepagesz=2M hugepages=1024 + + This will reserve 2GBytes of memory using 2Mbytes huge pages. + + Finally, for a NUMA system the allocation needs to be made on the correct + NUMA node. In a DPDK app there is a master core which will (usually) perform + memory allocation. It is important that some of the hugepages are reserved + on the NUMA memory node where the network device is attached. This is because + of a restriction in DPDK by which TX and RX descriptors rings must be created + on the master code. + + Per-node allocation of hugepages may be inspected and controlled using sysfs. + For example: + + .. code-block:: console + + cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages + + For a NUMA system there will be a specific hugepage directory per node + allowing control of hugepage reservation. A common problem may occur when + hugepages reservation is performed after the system has been working for + some time. Configuration using the global sysfs hugepage interface will + succeed but the per-node allocations may be unsatisfactory. + + The number of hugepages that need to be reserved depends on how the app uses + TX and RX descriptors, and packets mbufs. + +#. **Enable SR-IOV on the NFP-6xxx device:** The current NFP PMD works with + Virtual Functions (VFs) on a NFP device. Make sure that one of the Physical + Function (PF) drivers from the above Github repository is installed and + loaded. + + Virtual Functions need to be enabled before they can be used with the PMD. + Before enabling the VFs it is useful to obtain information about the + current NFP PCI device detected by the system: + + .. code-block:: console + + lspci -d19ee: + + Now, for example, configure two virtual functions on a NFP-6xxx device + whose PCI system identity is "0000:03:00.0": + + .. code-block:: console + + echo 2 > /sys/bus/pci/devices/0000:03:00.0/sriov_numvfs + + The result of this command may be shown using lspci again: + + .. code-block:: console + + lspci -d19ee: -k + + Two new PCI devices should appear in the output of the above command. The + -k option shows the device driver, if any, that devices are bound to. + Depending on the modules loaded at this point the new PCI devices may be + bound to nfp_netvf driver. + +#. **To install the uio kernel module (manually):** All major Linux + distributions have support for this kernel module so it is straightforward + to install it: + + .. code-block:: console + + modprobe uio + + The module should now be listed by the lsmod command. + +#. **To install the nfp_uio kernel module (manually):** This module supports + NFP-6xxx devices through the UIO interface. + + After compilation the module should be in the build kmod directory or + wherever the build directory is configured. + + .. code-block:: console + + cd build/kmod + insmod ./nfp_uio.ko + + The module should now be listed by the lsmod command. + + Depending on which NFP modules are loaded, nfp_uio may be automatically + bound to the NFP PCI devices by the system. Otherwise the binding needs + to be done explicitly. This is the case when nfp_netvf, the Linux kernel + driver for NFP VFs, was loaded when VFs were created. As described later + in this document this configuration may also be performed using scripts + provided by DPDK. + + First the device needs to be unbound, for example from the nfp_netvf + driver: + + .. code-block:: console + + echo 0000:03:08.0 > /sys/bus/pci/devices/0000:03:08.0/driver/unbind + + lspci -d19ee: -k + + The output of lspci should now show that 0000:03:08.0 is not bound to + any driver. + + The next step is to add the NFP PCI ID to the NFP UIO driver: + + .. code-block:: console + + echo 19ee 6003 > /sys/bus/pci/drivers/nfp_uio/new_id + + And then to bind the device to the nfp_uio driver: + + .. code-block:: console + + echo 0000:03:08.0 > /sys/bus/pci/drivers/nfp_uio/bind + + lspci -d19ee: -k + + lspci should show that device bound to nfp_uio driver. + +#. **Using tools from DPDK source to install and bind modules:** DPDK provides + scripts which are useful for installing the UIO modules and for binding the + right device to those modules avoiding doing so manually. + + In the tools directory of the DPDK source there are two scripts: + + * **setup.sh** + * **dpdk_nic_bind.py** + + Configuration may be performed by running setup.sh which invokes + dpdk_nic_bind.py as needed. Executing setup.sh will display a menu of + configuration options. + + + + -- 1.7.9.5