[PATCH v4 5/6] Section 5: Appendix

[David Young <dave@youngcopy.com>]

Added new guide on hugepages for different architectures

Added hugepages_different_architectures to index

-Content Streamlining: Removed initial detailed instructions on setting up DPDK
for non-root users.
Emphasized that running DPDK as non-root on Linux requires IOMMU support through vfio.
-Linux Section Updates: Simplified the section to focus on:
Adjusting permissions for VFIO entries in /dev and hugepage mount directories.
Running DPDK applications as a user in the DPDK group.
-FreeBSD Section Updates:
Streamlined content to specify adjusting permissions for:
Userspace-io device files in /dev, such as /dev/uio0, /dev/uio1, etc.
The userspace contiguous memory device: /dev/contigmem.
-Removed Sections: Detailed steps for creating user groups, binding NICs,
and setting up hugepages were removed.
-Overall Impact: The page now focuses more on the prerequisites for
running DPDK applications without root privileges, specifically for Linux and FreeBSD,
with less emphasis on detailed procedural steps.

-Added a reference tag `.. _vfio_platform:` at the start of the VFIO Platform section.
-Inserted a reference tag `.. _bifurcated_driver:` at the beginning of the Bifurcated Driver section.
-Appended a reference tag `.. _uio:` to the UIO section.
---
 .../appendix/cross_compile_dpdk.rst           |  37 +++
 .../appendix/dpdk_meson_build_options.rst     |  57 ++++
 .../hugepages_different_architectures.rst     |  56 ++++
 .../getting_started_guide/appendix/index.rst  |  18 ++
 .../running_dpdk_apps_without_root.rst        |  24 ++
 .../appendix/vfio_advanced.rst                | 301 ++++++++++++++++++
 6 files changed, 493 insertions(+)
 create mode 100644 doc/guides/getting_started_guide/appendix/cross_compile_dpdk.rst
 create mode 100644 doc/guides/getting_started_guide/appendix/dpdk_meson_build_options.rst
 create mode 100644 doc/guides/getting_started_guide/appendix/hugepages_different_architectures.rst
 create mode 100644 doc/guides/getting_started_guide/appendix/index.rst
 create mode 100644 doc/guides/getting_started_guide/appendix/running_dpdk_apps_without_root.rst
 create mode 100644 doc/guides/getting_started_guide/appendix/vfio_advanced.rst

diff --git a/doc/guides/getting_started_guide/appendix/cross_compile_dpdk.rst b/doc/guides/getting_started_guide/appendix/cross_compile_dpdk.rst
new file mode 100644
index 0000000000..3e4efe23a4
--- /dev/null
+++ b/doc/guides/getting_started_guide/appendix/cross_compile_dpdk.rst
@@ -0,0 +1,37 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2010-2025 Intel Corporation.
+
+.. _cross_compile_dpdk:
+
+Cross-compiling DPDK for Different Architectures on Linux
+=========================================================
+
+Cross-compiling DPDK for different architectures follows a similar process. Here are the general steps:
+
+1. **Get Compiler and Libraries**: Obtain the cross-compiler toolchain and any required libraries specific to the target architecture.
+
+2. **Build Using Cross-File**: Use Meson to set up the build with a cross-file specific to the target architecture, and then build with Ninja.
+
+Prerequisites
+-------------
+
+- NUMA Library (if required)
+- Meson and Ninja
+- pkg-config for the target architecture
+- Specific GNU or LLVM/Clang toolchain for the target architecture
+
+Cross-Compiling DPDK
+--------------------
+
+1. **Set Up the Cross Toolchain**: Download and extract the toolchain for the target architecture. Add it to the PATH.
+
+2. **Compile Any Required Libraries**: Compile libraries like NUMA if required.
+
+3. **Cross-Compile DPDK with Meson**:
+
+   .. code-block:: bash
+
+      meson setup cross-build --cross-file <target_machine_configuration>
+      ninja -C cross-build
+
+Refer to the specific sections for ARM64, LoongArch, and RISC-V for detailed instructions and architecture-specific considerations.
\ No newline at end of file
diff --git a/doc/guides/getting_started_guide/appendix/dpdk_meson_build_options.rst b/doc/guides/getting_started_guide/appendix/dpdk_meson_build_options.rst
new file mode 100644
index 0000000000..6669f98371
--- /dev/null
+++ b/doc/guides/getting_started_guide/appendix/dpdk_meson_build_options.rst
@@ -0,0 +1,57 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2010-2025 Intel Corporation.
+
+.. _dpdk_meson_build_options:
+
+DPDK Meson Build Configuration Options
+======================================
+
+DPDK provides a number of build configuration options that can be adjusted using the Meson build system. These options can be listed by running ``meson configure`` inside a configured build
+folder.
+
+Changing the Build Type
+-----------------------
+
+To change the build type from the default "release" to a regular "debug" build,
+you can either:
+
+- Pass ``-Dbuildtype=debug`` or ``--buildtype=debug`` to meson when configuring the build folder initially.
+- Run ``meson configure -Dbuildtype=debug`` inside the build folder after the initial meson run.
+
+Platform Options
+----------------
+
+The "platform" option specifies a set of configuration parameters that will be used. 
+The valid values are:
+
+- ``-Dplatform=native`` will tailor the configuration to the build machine.
+- ``-Dplatform=generic`` will use configuration that works on all machines of the same architecture as the build machine.
+- ``-Dplatform=<SoC>`` will use configuration optimized for a particular SoC.
+
+Consult the "socs" dictionary in ``config/arm/meson.build`` to see which SoCs are supported.
+
+Overriding Platform Parameters
+------------------------------
+
+The values determined by the platform parameter may be overwritten. For example,
+to set the ``max_lcores`` value to 256, you can either:
+
+- Pass ``-Dmax_lcores=256`` to meson when configuring the build folder initially.
+- Run ``meson configure -Dmax_lcores=256`` inside the build folder after the initial meson run.
+
+Building Sample Applications
+----------------------------
+
+Some of the DPDK sample applications in the examples directory can be automatically built as
+part of a meson build. To do so, pass a comma-separated list of the examples to build to the
+``-Dexamples`` meson option as below::
+
+    meson setup -Dexamples=l2fwd,l3fwd build
+
+There is also a special value "all" to request that all example applications whose dependencies
+are met on the current system are built. When ``-Dexamples=all`` is set as a meson option,
+meson will check each example application to see if it can be built, and add all which can be
+built to the list of tasks in the ninja build configuration file.
+
+For a complete list of options, run ``meson configure`` inside your configured build
+folder.
\ No newline at end of file
diff --git a/doc/guides/getting_started_guide/appendix/hugepages_different_architectures.rst b/doc/guides/getting_started_guide/appendix/hugepages_different_architectures.rst
new file mode 100644
index 0000000000..a93904fc49
--- /dev/null
+++ b/doc/guides/getting_started_guide/appendix/hugepages_different_architectures.rst
@@ -0,0 +1,56 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2010-2025 Intel Corporation.
+
+.. _hugepages_different_architectures:
+
+Hugepages Configuration for Multiple Architectures
+==================================================
+
+This section outlines the steps for configuring hugepages of various sizes on different architectures, an important aspect for optimizing DPDK performance.
+
+Hugepages on x86 Architecture
+-----------------------------
+
+**2MB and 1G Hugepages**
+
+- *2MB hugepages* are commonly used on x86.
+- *1G hugepages* can improve performance for large-memory applications.
+
+**Configuring 1G Hugepages**
+
+.. code-block:: bash
+
+    # Example GRUB configuration for 1G hugepages
+    GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=4"
+
+Update GRUB and reboot after making these changes.
+
+Hugepages on ARM Architecture
+-----------------------------
+
+ARM supports a range of hugepage sizes, such as 64KB, 512KB, and 2MB.
+
+**Example Configuration**
+
+.. code-block:: bash
+
+    # Setting 2MB hugepages on ARM
+    echo N > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
+
+Replace 'N' with the number of pages needed.
+
+Other Architectures
+-------------------
+
+Refer to architecture-specific documentation for hugepage configurations on platforms like PowerPC or MIPS.
+
+Boot-Time Reservation of Hugepages
+----------------------------------
+
+Boot-time reservation is essential for large hugepage sizes. Modify the boot loader, such as GRUB, for this purpose:
+
+.. code-block:: bash
+
+    GRUB_CMDLINE_LINUX="hugepagesz=2M hugepages=512"
+
+Regenerate the GRUB config and reboot your system.
diff --git a/doc/guides/getting_started_guide/appendix/index.rst b/doc/guides/getting_started_guide/appendix/index.rst
new file mode 100644
index 0000000000..b92e3afb5a
--- /dev/null
+++ b/doc/guides/getting_started_guide/appendix/index.rst
@@ -0,0 +1,18 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2010-2025 Intel Corporation.
+
+.. _appendix:
+
+Appendix
+========
+
+This section covers specific guides related to DPDK.
+
+.. toctree::
+    :maxdepth: 2
+
+    dpdk_meson_build_options
+    hugepages_different_architectures
+    running_dpdk_apps_without_root
+    vfio_advanced
+    cross_compile_dpdk
diff --git a/doc/guides/getting_started_guide/appendix/running_dpdk_apps_without_root.rst b/doc/guides/getting_started_guide/appendix/running_dpdk_apps_without_root.rst
new file mode 100644
index 0000000000..50cead324e
--- /dev/null
+++ b/doc/guides/getting_started_guide/appendix/running_dpdk_apps_without_root.rst
@@ -0,0 +1,24 @@
+.. _running_dpdk_apps_without_root:
+
+Running DPDK Applications Without Root Privileges
+=================================================
+
+It's important to note that running DPDK as non-root on Linux requires IOMMU support through vfio.
+
+Linux
+-----
+To run DPDK applications without root privileges on Linux, follow these steps:
+
+1. **Adjust Permissions for Specific Files and Directories**:
+
+   - VFIO entries in ``/dev``, such as ``/dev/vfio/<id>``, where ``<id>`` is the VFIO group to which a device used by DPDK belongs.
+   - The hugepage mount directory, typically ``/dev/hugepages``, or any alternative mount point configured by the user, e.g., ``/mnt/huge``, ``/mnt/huge_1G``.
+
+2. **Run the DPDK Application**: Execute the desired DPDK application as the user who has been added to the DPDK group.
+
+FreeBSD
+-------
+Adjust the permissions of the following files to run DPDK applications as a non-root user:
+
+- The userspace-io device files in ``/dev``, for example, ``/dev/uio0``, ``/dev/uio1``, and so on.
+- The userspace contiguous memory device: ``/dev/contigmem``.
diff --git a/doc/guides/getting_started_guide/appendix/vfio_advanced.rst b/doc/guides/getting_started_guide/appendix/vfio_advanced.rst
new file mode 100644
index 0000000000..35d9e783bf
--- /dev/null
+++ b/doc/guides/getting_started_guide/appendix/vfio_advanced.rst
@@ -0,0 +1,301 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2010-2025 Intel Corporation.
+
+.. _vfio_advanced:
+
+.. |reg| unicode:: U+000AE
+
+VFIO Advanced
+=============
+
+
+.. contents:: Table of Contents
+   :local:
+
+.. _vfio_no_iommu_mode:
+
+VFIO no-IOMMU mode
+------------------
+
+If there is no IOMMU available on the system, VFIO can still be used,
+but it has to be loaded with an additional module parameter:
+
+.. code-block:: console
+
+   modprobe vfio enable_unsafe_noiommu_mode=1
+
+Alternatively, one can also enable this option in an already loaded kernel module:
+
+.. code-block:: console
+
+   echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
+
+After that, VFIO can be used with hardware devices as usual.
+
+.. note::
+
+   It may be required to unload all VFIO related-modules before probing
+   the module again with ``enable_unsafe_noiommu_mode=1`` parameter.
+
+.. warning::
+
+   Since no-IOMMU mode    forgoes IOMMU protection, it is inherently unsafe.
+   That said, it does make it possible for the user
+   to keep the degree of device access and programming that VFIO has,
+   in situations where IOMMU is not available.
+
+VFIO Memory Mapping Limits
+--------------------------
+
+For DMA mapping of either external memory or hugepages, VFIO interface is used.
+VFIO does not support partial unmap of once mapped memory. Hence DPDK's memory is
+mapped in hugepage granularity or system page granularity. Number of DMA
+mappings is limited by kernel with user locked memory limit of a process (rlimit)
+for system/hugepage memory. Another per-container overall limit applicable both
+for external memory and system memory was added in kernel 5.1 defined by
+VFIO module parameter ``dma_entry_limit`` with a default value of 64K.
+When application is out of DMA entries, these limits need to be adjusted to
+increase the allowed limit.
+
+Creating Virtual Functions using vfio-pci
+-----------------------------------------
+
+Since Linux version 5.7,
+the ``vfio-pci`` module supports the creation of virtual functions.
+After the PF is bound to ``vfio-pci`` module,
+the user can create the VFs using the ``sysfs`` interface,
+and these VFs will be bound to ``vfio-pci`` module automatically.
+
+When the PF is bound to ``vfio-pci``,
+by default it will have a randomly generated VF token.
+For security reasons, this token is write only,
+so the user cannot read it from the kernel directly.
+To access the VFs, the user needs to create a new token,
+and use it to initialize both VF and PF devices.
+The tokens are in UUID format,
+so any UUID generation tool can be used to create a new token.
+
+This VF token can be passed to DPDK by using EAL parameter ``--vfio-vf-token``.
+The token will be used for all PF and VF ports within the application.
+
+#. Generate the VF token by uuid command
+
+   .. code-block:: console
+
+      14d63f20-8445-11ea-8900-1f9ce7d5650d
+
+#. Load the ``vfio-pci`` module with ``enable_sriov`` parameter set
+
+   .. code-block:: console
+
+      sudo modprobe vfio-pci enable_sriov=1
+
+   Alternatively, pass the ``enable_sriov`` parameter through the ``sysfs`` if the    module is already loaded or is built-in:
+
+   .. code-block:: console
+
+      echo 1 | sudo tee /sys/module/vfio_pci/parameters/enable_sriov
+
+#. Bind the PCI devices to ``vfio-pci`` driver
+
+   .. code-block:: console
+
+      ./usertools/dpdk-devbind.py -b vfio-pci 0000:86:00.0
+
+#. Create the desired number of VF devices
+
+   .. code-block:: console
+
+      echo 2 > /sys/bus/pci/devices/0000:86:00.0/sriov_numvfs
+
+#. Start the DPDK application that will manage the PF device
+
+   .. code-block:: console
+
+      <build_dir>/app/dpdk-testpmd -l 22-25 -n 4 -a 86:00.0 \
+      --vfio-vf-token=14d63f20-8445-11ea-8900-1f9ce7d5650d --file-prefix=pf -- -i
+
+#. Start the DPDK application that will manage the VF device
+
+   .. code-block:: console
+
+      <build_dir>/app/dpdk-testpmd -l 26-29 -n 4 -a 86:02.0 \
+      --vfio-vf-token=14d63f20-8445-11ea-8900-1f9ce7d5650d --file-prefix=vf0 -- -i
+
+.. note::
+
+   Linux versions earlier than version 5.7 do not support the creation of
+   virtual functions within the VFIO framework.
+
+Troubleshooting VFIO
+--------------------
+
+In certain situations, using ``dpdk-devbind.py`` script
+to bind a device to VFIO driver may fail.
+The first place to check is the kernel messages:
+
+.. code-block:: console
+
+   dmesg | tail
+   ...
+   [ 1297.875090] vfio-pci: probe of 0000:31:00.0 failed with error -22
+   ...
+
+In most cases, the ``error -22`` indicates that the VFIO subsystem
+could not be enabled because there is no IOMMU support.
+
+To check whether the kernel has been booted with correct parameters,
+one can check the kernel command-line:
+
+.. code-block:: console
+
+   cat /proc/cmdline
+
+Please refer to earlier sections on how to configure kernel parameters
+correctly for your system.
+
+If the kernel is configured correctly, one also has to make sure that
+the BIOS configuration has virtualization features (such as Intel\ |reg| VT-d).
+There is no standard way to check if the platform is configured correctly,
+so please check with your platform documentation to see if it has such features,
+and how to enable them.
+
+In certain distributions, default kernel configuration is such that
+the no-IOMMU mode is disabled altogether at compile time.
+This can be checked in the boot configuration of your system:
+
+.. code-block:: console
+
+   cat /boot/config-$(uname -r) | grep NOIOMMU
+   # CONFIG_VFIO_NOIOMMU is not set
+
+If ``CONFIG_VFIO_NOIOMMU`` is not enabled in the kernel configuration,
+VFIO driver will not support the no-IOMMU mode,
+and other alternatives (such as UIO drivers) will have to be used.
+
+.. _vfio_platform:
+
+VFIO Platform
+-------------
+
+VFIO Platform is a kernel driver that extends capabilities of VFIO
+by adding support for platform devices that reside behind an IOMMU.
+Linux usually learns about platform devices directly from device tree
+during boot-up phase,
+unlike for example, PCI devices which have necessary information built-in.
+
+To make use of VFIO platform, the ``vfio-platform`` module must be loaded first:
+
+.. code-block:: console
+
+   sudo modprobe vfio-platform
+
+.. note::
+
+   By default ``vfio-platform`` assumes that platform device has dedicated reset driver.
+   If such driver is missing or device does not require one,
+   this option can be turned off by setting ``reset_required=0`` module parameter.
+
+Afterwards platform device needs to be bound to ``vfio-platform``.
+This is standard procedure requiring two steps.
+First ``driver_override``, which is available inside platform device directory,
+needs to be set to ``vfio-platform``:
+
+.. code-block:: console
+
+   sudo echo vfio-platform > /sys/bus/platform/devices/DEV/driver_override
+
+Next ``DEV`` device must be bound to ``vfio-platform`` driver:
+
+.. code-block:: console
+
+   sudo echo DEV > /sys/bus/platform/drivers/vfio-platform/bind
+
+On application startup, DPDK platform bus driver scans ``/sys/bus/platform/devices``
+searching for devices that have ``driver`` symbolic link
+pointing to ``vfio-platform`` driver.
+Finally, scanned devices are matched against available PMDs.
+Matching is    successful if either PMD name or PMD alias matches kernel driver name
+or PMD name matches platform device name, all in that order.
+
+VFIO Platform depends on ARM/ARM64 and is usually enabled on distributions
+running on these systems.
+Consult your distributions documentation to make sure that is the case.
+
+.. _bifurcated_driver:
+
+Bifurcated Driver
+-----------------
+
+PMDs which use the bifurcated driver co-exists with the device kernel driver.
+On such model the NIC is controlled by the kernel, while the data
+path is performed by the PMD directly on top of the device.
+
+Such model has the following benefits:
+
+ - It is secure and robust, as the memory management and isolation
+   is done by the kernel.
+ - It enables the user to use legacy linux tools such as ``ethtool`` or
+   ``ifconfig`` while running DPDK application on the same network ports.
+ - It enables the DPDK application to filter only part of the traffic,
+   while the rest will be directed and handled by the kernel driver.
+   The flow bifurcation is performed by the NIC hardware.
+   As an example, using :ref:`flow_isolated_mode` allows to choose
+   strictly what is received in DPDK.
+
+More about the bifurcated driver can be found in
+NVIDIA `bifurcated PMD
+<https://www.dpdk.org/wp-content/uploads/sites/35/2016/10/Day02-Session04-RonyEfraim-Userspace2016.pdf>`_ presentation.
+
+.. _uio:
+
+UIO
+---
+
+.. warning::
+
+   Using UIO drivers is inherently unsafe due to this method lacking IOMMU protection,
+   and can only be done by root user.
+
+In situations where using VFIO is not an option, there are alternative drivers one can
+use.
+In many cases, the standard ``uio_pci_generic`` module included in the Linux kernel
+can be used as a substitute for VFIO. This module can be loaded using the command:
+
+.. code-block:: console
+
+   sudo modprobe uio_pci_generic
+
+.. note::
+
+   ``uio_pci_generic`` module doesn't support the creation of virtual functions.
+
+As an alternative to the ``uio_pci_generic``, there is the ``igb_uio`` module
+which can be found in the repository `dpdk-kmods <http://git.dpdk.org/dpdk-kmods>`_.
+It can be loaded as shown below:
+
+.. code-block:: console
+
+   sudo modprobe uio
+   sudo insmod igb_uio.ko
+
+.. note::
+
+    For some devices which lack support for legacy interrupts, e.g. virtual function
+    (VF) devices, the ``igb_uio`` module may be needed in place of ``uio_pci_generic``.
+
+.. note::
+
+   If UEFI secure boot is enabled,
+   the Linux kernel may disallow the use of UIO on the system.
+   Therefore, devices for use by DPDK should be bound to the ``vfio-pci`` kernel module
+   rather than any UIO-based module.
+   For more details see :ref:`linux_gsg_binding_kernel` below.
+
+.. note::
+
+   If the devices used for DPDK are bound to a UIO-based kernel module,
+   please make sure that the IOMMU is disabled or is in passthrough mode.
+   One can add ``intel_iommu=off`` or ``amd_iommu=off`` or ``intel_iommu=on iommu=pt``
+   in GRUB command line on x86_64 systems,
+   or add ``iommu.passthrough=1`` on aarch64 systems.
-- 
2.41.0.windows.1