From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6926C43274; Fri, 3 Nov 2023 05:03:00 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 625B842DC3; Fri, 3 Nov 2023 05:02:28 +0100 (CET) Received: from mail-yb1-f176.google.com (mail-yb1-f176.google.com [209.85.219.176]) by mails.dpdk.org (Postfix) with ESMTP id 25A4640265 for ; Fri, 3 Nov 2023 05:02:25 +0100 (CET) Received: by mail-yb1-f176.google.com with SMTP id 3f1490d57ef6-da34f90f6e3so417998276.0 for ; Thu, 02 Nov 2023 21:02:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=youngcopy-com.20230601.gappssmtp.com; s=20230601; t=1698984144; x=1699588944; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=g5ZtCG/8bPQDFKgDw4WTwkpSDegAxVEqQCkogrr0iZM=; b=hP2mlnHMzYOLwlmjuTAafhxs7IMf9obw1S6qAmbN/cSnuqdlNzL5VuCwj2vt/oz0Ac Lp0cF4oVED3zzz6/0p+VDwC6pA+dNINo0qnXUoHK0H/YsklqDq+NW40DWjMOqZDNtFH4 oX88yaxuCAUgkwFJV2OWz8nqMaNXBoJ63a0X5Qw8dih5uDdc9z0W5C4wx0zNuDHTxz1F NVzKfm0VBEUocucAvBXVkf1aLIg9t+cJBzTD8gXBmmKayyVj74hfT3iOSRfAeRde43PT ywfMCHIgG9sS6MQ5LVk02YcUZxo11XUrC71iJAD4lIZSWZEEBXu/m646lGhtSRJAY25I 16Kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698984144; x=1699588944; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=g5ZtCG/8bPQDFKgDw4WTwkpSDegAxVEqQCkogrr0iZM=; b=G/5t3AokV4YN3FkdakZYA+nJuQddP5gQTMwzzqyw4V1mhO89Ie0q59W5yhlLgNGxTO fafcS3odHWzuQc9JZws7gIwAQbp5qB/wiI5857B2ywosBE5PJIAABAa32gwq8G22cpt1 DWO21bWJhvu1j33cW/wGWfFaZ4AMdqjIDo3AloFxhNnsC8LRuMIxVFsn2d2YtQevuY4o HOh2YhifQ2f8KsAwd6JOKSoUpQkOJ5PMGMk5zXv6XcWtUbv5peADiQqlkqWRgi9BFLVQ 0aBeM1d0njw8lLT8n9zkQovOck3yvuoJP3eTi288//3R4+KJBkOKrDzSnDSWk8qxmr5K 15jA== X-Gm-Message-State: AOJu0YyTU0TFzcYbSeSwXGvN2AKMIB2UmcHKeAF34mITcui16EBEUBwY FVRJE5qIoRJPrXHbuIl1kNWozJxyVLebBTUUJnU= X-Google-Smtp-Source: AGHT+IHcuo4AkNzKssegQI//Z1tOT3ACZi9hN96eBUL/j3/VdVxBz8vyCVXXrbMrFJ+HC64jOm7Rcg== X-Received: by 2002:a05:690c:4803:b0:5b3:1c41:733f with SMTP id hc3-20020a05690c480300b005b31c41733fmr10237007ywb.5.1698984143873; Thu, 02 Nov 2023 21:02:23 -0700 (PDT) Received: from localhost.localdomain ([2600:1700:20c0:a560:c076:581e:323d:c06e]) by smtp.gmail.com with ESMTPSA id x64-20020a817c43000000b0059b1f6d1959sm493958ywc.101.2023.11.02.21.02.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Nov 2023 21:02:23 -0700 (PDT) From: David Young To: dev@dpdk.org Cc: Bruce Richardson , Aaron Conole , David Young Subject: [PATCH v3 5/7] Section 5: Appendix Date: Fri, 3 Nov 2023 00:01:51 -0400 Message-ID: <20231103040202.2849-6-dave@youngcopy.com> X-Mailer: git-send-email 2.41.0.windows.1 In-Reply-To: <20231103040202.2849-1-dave@youngcopy.com> References: <20231103040202.2849-1-dave@youngcopy.com> MIME-Version: 1.0 Content-Type: text/plain; charset=y Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --- .../appendix/cross_compile_dpdk.rst | 37 +++ .../appendix/dpdk_meson_build_options.rst | 57 ++++ .../getting_started_guide/appendix/index.rst | 17 + .../running_dpdk_apps_without_root.rst | 36 +++ .../appendix/vfio_advanced.rst | 295 ++++++++++++++++++ 5 files changed, 442 insertions(+) create mode 100644 doc/guides/getting_started_guide/appendix/cross_compile_dpdk.rst create mode 100644 doc/guides/getting_started_guide/appendix/dpdk_meson_build_options.rst create mode 100644 doc/guides/getting_started_guide/appendix/index.rst create mode 100644 doc/guides/getting_started_guide/appendix/running_dpdk_apps_without_root.rst create mode 100644 doc/guides/getting_started_guide/appendix/vfio_advanced.rst diff --git a/doc/guides/getting_started_guide/appendix/cross_compile_dpdk.rst b/doc/guides/getting_started_guide/appendix/cross_compile_dpdk.rst new file mode 100644 index 0000000000..3e4efe23a4 --- /dev/null +++ b/doc/guides/getting_started_guide/appendix/cross_compile_dpdk.rst @@ -0,0 +1,37 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2025 Intel Corporation. + +.. _cross_compile_dpdk: + +Cross-compiling DPDK for Different Architectures on Linux +========================================================= + +Cross-compiling DPDK for different architectures follows a similar process. Here are the general steps: + +1. **Get Compiler and Libraries**: Obtain the cross-compiler toolchain and any required libraries specific to the target architecture. + +2. **Build Using Cross-File**: Use Meson to set up the build with a cross-file specific to the target architecture, and then build with Ninja. + +Prerequisites +------------- + +- NUMA Library (if required) +- Meson and Ninja +- pkg-config for the target architecture +- Specific GNU or LLVM/Clang toolchain for the target architecture + +Cross-Compiling DPDK +-------------------- + +1. **Set Up the Cross Toolchain**: Download and extract the toolchain for the target architecture. Add it to the PATH. + +2. **Compile Any Required Libraries**: Compile libraries like NUMA if required. + +3. **Cross-Compile DPDK with Meson**: + + .. code-block:: bash + + meson setup cross-build --cross-file + ninja -C cross-build + +Refer to the specific sections for ARM64, LoongArch, and RISC-V for detailed instructions and architecture-specific considerations. \ No newline at end of file diff --git a/doc/guides/getting_started_guide/appendix/dpdk_meson_build_options.rst b/doc/guides/getting_started_guide/appendix/dpdk_meson_build_options.rst new file mode 100644 index 0000000000..6669f98371 --- /dev/null +++ b/doc/guides/getting_started_guide/appendix/dpdk_meson_build_options.rst @@ -0,0 +1,57 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2025 Intel Corporation. + +.. _dpdk_meson_build_options: + +DPDK Meson Build Configuration Options +====================================== + +DPDK provides a number of build configuration options that can be adjusted using the Meson build system. These options can be listed by running ``meson configure`` inside a configured build +folder. + +Changing the Build Type +----------------------- + +To change the build type from the default "release" to a regular "debug" build, +you can either: + +- Pass ``-Dbuildtype=debug`` or ``--buildtype=debug`` to meson when configuring the build folder initially. +- Run ``meson configure -Dbuildtype=debug`` inside the build folder after the initial meson run. + +Platform Options +---------------- + +The "platform" option specifies a set of configuration parameters that will be used. +The valid values are: + +- ``-Dplatform=native`` will tailor the configuration to the build machine. +- ``-Dplatform=generic`` will use configuration that works on all machines of the same architecture as the build machine. +- ``-Dplatform=`` will use configuration optimized for a particular SoC. + +Consult the "socs" dictionary in ``config/arm/meson.build`` to see which SoCs are supported. + +Overriding Platform Parameters +------------------------------ + +The values determined by the platform parameter may be overwritten. For example, +to set the ``max_lcores`` value to 256, you can either: + +- Pass ``-Dmax_lcores=256`` to meson when configuring the build folder initially. +- Run ``meson configure -Dmax_lcores=256`` inside the build folder after the initial meson run. + +Building Sample Applications +---------------------------- + +Some of the DPDK sample applications in the examples directory can be automatically built as +part of a meson build. To do so, pass a comma-separated list of the examples to build to the +``-Dexamples`` meson option as below:: + + meson setup -Dexamples=l2fwd,l3fwd build + +There is also a special value "all" to request that all example applications whose dependencies +are met on the current system are built. When ``-Dexamples=all`` is set as a meson option, +meson will check each example application to see if it can be built, and add all which can be +built to the list of tasks in the ninja build configuration file. + +For a complete list of options, run ``meson configure`` inside your configured build +folder. \ No newline at end of file diff --git a/doc/guides/getting_started_guide/appendix/index.rst b/doc/guides/getting_started_guide/appendix/index.rst new file mode 100644 index 0000000000..23bb1fcf78 --- /dev/null +++ b/doc/guides/getting_started_guide/appendix/index.rst @@ -0,0 +1,17 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2025 Intel Corporation. + +.. _appendix: + +Appendix +======== + +This section covers specific guides related to DPDK. + +.. toctree:: + :maxdepth: 2 + + dpdk_meson_build_options + running_dpdk_apps_without_root + vfio_advanced + cross_compile_dpdk \ No newline at end of file diff --git a/doc/guides/getting_started_guide/appendix/running_dpdk_apps_without_root.rst b/doc/guides/getting_started_guide/appendix/running_dpdk_apps_without_root.rst new file mode 100644 index 0000000000..9f214bbdc8 --- /dev/null +++ b/doc/guides/getting_started_guide/appendix/running_dpdk_apps_without_root.rst @@ -0,0 +1,36 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2025 Intel Corporation. + +.. _running_dpdk_apps_without_root: + +Running DPDK Applications Without Root Privileges +================================================= + +Although applications using the DPDK use network ports and other hardware resources +directly, with a number of small permission adjustments, +it is possible to run these applications as a user other than “root”. +To do so, the ownership, or permissions, on the following file system objects should be +adjusted so the user account being used to run the DPDK application has +access to them: + +Linux +----- + +1. **Create a DPDK User Group**: Create a new user group for DPDK and add the desired user to this group. + +2. **Set Up Hugepages**: Configure hugepages for the user. + +3. **Bind the NIC to a User-Space Driver**: Use the DPDK tool ``dpdk-devbind.py`` to bind the NIC to a user-space driver like ``vfio-pci`` or ``igb_uio``. + +4. **Set Permissions for UIO/VFIO Devices**: Change the ownership and permissions of the UIO or VFIO devices to allow access by the DPDK user group. + +5. **Run the DPDK Application**: Run the desired DPDK application as the user who has been added to the DPDK group. + +FreeBSD +------- + +- The userspace-io device files in ``/dev``, for example, ``/dev/uio0``, ``/dev/uio1``, and so on +- The userspace contiguous memory device: ``/dev/contigmem`` + + +Refer to the `DPDK Release Notes `_ for supported applications. \ No newline at end of file diff --git a/doc/guides/getting_started_guide/appendix/vfio_advanced.rst b/doc/guides/getting_started_guide/appendix/vfio_advanced.rst new file mode 100644 index 0000000000..1cdb138eb7 --- /dev/null +++ b/doc/guides/getting_started_guide/appendix/vfio_advanced.rst @@ -0,0 +1,295 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2025 Intel Corporation. + +.. _vfio_advanced: + +.. |reg| unicode:: U+000AE + +VFIO Advanced +============= + + +.. contents:: Table of Contents + :local: + +.. _vfio_no_iommu_mode: + +VFIO no-IOMMU mode +------------------ + +If there is no IOMMU available on the system, VFIO can still be used, +but it has to be loaded with an additional module parameter: + +.. code-block:: console + + modprobe vfio enable_unsafe_noiommu_mode=1 + +Alternatively, one can also enable this option in an already loaded kernel module: + +.. code-block:: console + + echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode + +After that, VFIO can be used with hardware devices as usual. + +.. note:: + + It may be required to unload all VFIO related-modules before probing + the module again with ``enable_unsafe_noiommu_mode=1`` parameter. + +.. warning:: + + Since no-IOMMU mode forgoes IOMMU protection, it is inherently unsafe. + That said, it does make it possible for the user + to keep the degree of device access and programming that VFIO has, + in situations where IOMMU is not available. + +VFIO Memory Mapping Limits +-------------------------- + +For DMA mapping of either external memory or hugepages, VFIO interface is used. +VFIO does not support partial unmap of once mapped memory. Hence DPDK's memory is +mapped in hugepage granularity or system page granularity. Number of DMA +mappings is limited by kernel with user locked memory limit of a process (rlimit) +for system/hugepage memory. Another per-container overall limit applicable both +for external memory and system memory was added in kernel 5.1 defined by +VFIO module parameter ``dma_entry_limit`` with a default value of 64K. +When application is out of DMA entries, these limits need to be adjusted to +increase the allowed limit. + +Creating Virtual Functions using vfio-pci +----------------------------------------- + +Since Linux version 5.7, +the ``vfio-pci`` module supports the creation of virtual functions. +After the PF is bound to ``vfio-pci`` module, +the user can create the VFs using the ``sysfs`` interface, +and these VFs will be bound to ``vfio-pci`` module automatically. + +When the PF is bound to ``vfio-pci``, +by default it will have a randomly generated VF token. +For security reasons, this token is write only, +so the user cannot read it from the kernel directly. +To access the VFs, the user needs to create a new token, +and use it to initialize both VF and PF devices. +The tokens are in UUID format, +so any UUID generation tool can be used to create a new token. + +This VF token can be passed to DPDK by using EAL parameter ``--vfio-vf-token``. +The token will be used for all PF and VF ports within the application. + +#. Generate the VF token by uuid command + + .. code-block:: console + + 14d63f20-8445-11ea-8900-1f9ce7d5650d + +#. Load the ``vfio-pci`` module with ``enable_sriov`` parameter set + + .. code-block:: console + + sudo modprobe vfio-pci enable_sriov=1 + + Alternatively, pass the ``enable_sriov`` parameter through the ``sysfs`` if the module is already loaded or is built-in: + + .. code-block:: console + + echo 1 | sudo tee /sys/module/vfio_pci/parameters/enable_sriov + +#. Bind the PCI devices to ``vfio-pci`` driver + + .. code-block:: console + + ./usertools/dpdk-devbind.py -b vfio-pci 0000:86:00.0 + +#. Create the desired number of VF devices + + .. code-block:: console + + echo 2 > /sys/bus/pci/devices/0000:86:00.0/sriov_numvfs + +#. Start the DPDK application that will manage the PF device + + .. code-block:: console + + /app/dpdk-testpmd -l 22-25 -n 4 -a 86:00.0 \ + --vfio-vf-token=14d63f20-8445-11ea-8900-1f9ce7d5650d --file-prefix=pf -- -i + +#. Start the DPDK application that will manage the VF device + + .. code-block:: console + + /app/dpdk-testpmd -l 26-29 -n 4 -a 86:02.0 \ + --vfio-vf-token=14d63f20-8445-11ea-8900-1f9ce7d5650d --file-prefix=vf0 -- -i + +.. note:: + + Linux versions earlier than version 5.7 do not support the creation of + virtual functions within the VFIO framework. + +Troubleshooting VFIO +-------------------- + +In certain situations, using ``dpdk-devbind.py`` script +to bind a device to VFIO driver may fail. +The first place to check is the kernel messages: + +.. code-block:: console + + dmesg | tail + ... + [ 1297.875090] vfio-pci: probe of 0000:31:00.0 failed with error -22 + ... + +In most cases, the ``error -22`` indicates that the VFIO subsystem +could not be enabled because there is no IOMMU support. + +To check whether the kernel has been booted with correct parameters, +one can check the kernel command-line: + +.. code-block:: console + + cat /proc/cmdline + +Please refer to earlier sections on how to configure kernel parameters +correctly for your system. + +If the kernel is configured correctly, one also has to make sure that +the BIOS configuration has virtualization features (such as Intel\ |reg| VT-d). +There is no standard way to check if the platform is configured correctly, +so please check with your platform documentation to see if it has such features, +and how to enable them. + +In certain distributions, default kernel configuration is such that +the no-IOMMU mode is disabled altogether at compile time. +This can be checked in the boot configuration of your system: + +.. code-block:: console + + cat /boot/config-$(uname -r) | grep NOIOMMU + # CONFIG_VFIO_NOIOMMU is not set + +If ``CONFIG_VFIO_NOIOMMU`` is not enabled in the kernel configuration, +VFIO driver will not support the no-IOMMU mode, +and other alternatives (such as UIO drivers) will have to be used. + +VFIO Platform +------------- + +VFIO Platform is a kernel driver that extends capabilities of VFIO +by adding support for platform devices that reside behind an IOMMU. +Linux usually learns about platform devices directly from device tree +during boot-up phase, +unlike for example, PCI devices which have necessary information built-in. + +To make use of VFIO platform, the ``vfio-platform`` module must be loaded first: + +.. code-block:: console + + sudo modprobe vfio-platform + +.. note:: + + By default ``vfio-platform`` assumes that platform device has dedicated reset driver. + If such driver is missing or device does not require one, + this option can be turned off by setting ``reset_required=0`` module parameter. + +Afterwards platform device needs to be bound to ``vfio-platform``. +This is standard procedure requiring two steps. +First ``driver_override``, which is available inside platform device directory, +needs to be set to ``vfio-platform``: + +.. code-block:: console + + sudo echo vfio-platform > /sys/bus/platform/devices/DEV/driver_override + +Next ``DEV`` device must be bound to ``vfio-platform`` driver: + +.. code-block:: console + + sudo echo DEV > /sys/bus/platform/drivers/vfio-platform/bind + +On application startup, DPDK platform bus driver scans ``/sys/bus/platform/devices`` +searching for devices that have ``driver`` symbolic link +pointing to ``vfio-platform`` driver. +Finally, scanned devices are matched against available PMDs. +Matching is successful if either PMD name or PMD alias matches kernel driver name +or PMD name matches platform device name, all in that order. + +VFIO Platform depends on ARM/ARM64 and is usually enabled on distributions +running on these systems. +Consult your distributions documentation to make sure that is the case. + +Bifurcated Driver +----------------- + +PMDs which use the bifurcated driver co-exists with the device kernel driver. +On such model the NIC is controlled by the kernel, while the data +path is performed by the PMD directly on top of the device. + +Such model has the following benefits: + + - It is secure and robust, as the memory management and isolation + is done by the kernel. + - It enables the user to use legacy linux tools such as ``ethtool`` or + ``ifconfig`` while running DPDK application on the same network ports. + - It enables the DPDK application to filter only part of the traffic, + while the rest will be directed and handled by the kernel driver. + The flow bifurcation is performed by the NIC hardware. + As an example, using :ref:`flow_isolated_mode` allows to choose + strictly what is received in DPDK. + +More about the bifurcated driver can be found in +NVIDIA `bifurcated PMD +`_ presentation. + +UIO +--- + +.. warning:: + + Using UIO drivers is inherently unsafe due to this method lacking IOMMU protection, + and can only be done by root user. + +In situations where using VFIO is not an option, there are alternative drivers one can +use. +In many cases, the standard ``uio_pci_generic`` module included in the Linux kernel +can be used as a substitute for VFIO. This module can be loaded using the command: + +.. code-block:: console + + sudo modprobe uio_pci_generic + +.. note:: + + ``uio_pci_generic`` module doesn't support the creation of virtual functions. + +As an alternative to the ``uio_pci_generic``, there is the ``igb_uio`` module +which can be found in the repository `dpdk-kmods `_. +It can be loaded as shown below: + +.. code-block:: console + + sudo modprobe uio + sudo insmod igb_uio.ko + +.. note:: + + For some devices which lack support for legacy interrupts, e.g. virtual function + (VF) devices, the ``igb_uio`` module may be needed in place of ``uio_pci_generic``. + +.. note:: + + If UEFI secure boot is enabled, + the Linux kernel may disallow the use of UIO on the system. + Therefore, devices for use by DPDK should be bound to the ``vfio-pci`` kernel module + rather than any UIO-based module. + For more details see :ref:`linux_gsg_binding_kernel` below. + +.. note:: + + If the devices used for DPDK are bound to a UIO-based kernel module, + please make sure that the IOMMU is disabled or is in passthrough mode. + One can add ``intel_iommu=off`` or ``amd_iommu=off`` or ``intel_iommu=on iommu=pt`` + in GRUB command line on x86_64 systems, + or add ``iommu.passthrough=1`` on aarch64 systems. \ No newline at end of file -- 2.41.0.windows.1