[PATCH 0/9] Improve linux drivers GSG section

[PATCH 0/9] Improve linux drivers GSG section

[Bruce Richardson]

This patchset reworks large parts of the linux drivers section of the
Linux Getting Started Guide, so as to make it clearer and easier to
read. The overall flow of the page is adjusted so that more important
information is nearer the top, and so that all information on VFIO is
consolidated into a single section. In terms of the content of the
sections, there are some minor changes to adjust what is a warning vs
just a "note" in the doc, and there are some updates to increase
emphasis on VFIO ovs UIO.

Including stable on CC for the set, as all patches may be candidates for
backport to help improve docs for older releases too.

Bruce Richardson (9):
  doc/linux_gsg: replace special characters for (R) symbol
  doc/linux_gsg: fix missing note on UIO module
  doc/linux_gsg: make UIO safety warning more visible
  doc/linux_gsg: move section on binding drivers up the page
  doc/linux_gsg: emphasise VFIO over UIO-based modules
  doc/linux_gsg: split VFIO section into subsections
  doc/linux_gsg: move UIO section to the end
  doc/linux_gsg: consolidate all VFIO content
  doc/linux_gsg: change informational warnings to notes

 doc/guides/linux_gsg/enable_func.rst   |  16 +-
 doc/guides/linux_gsg/linux_drivers.rst | 396 +++++++++++++------------
 doc/guides/linux_gsg/sys_reqs.rst      |  10 +-
 3 files changed, 223 insertions(+), 199 deletions(-)

--
2.32.0

[PATCH 1/9] doc/linux_gsg: replace special characters for (R) symbol

[Bruce Richardson]

Some IDEs, such as eclipse, complained on save about the use of special
characters in the (R) symbol in linux GSG doc. We can replace those with
the equivalent "|reg|" text, and including isonum.txt.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/linux_gsg/enable_func.rst   | 16 +++++++++-------
 doc/guides/linux_gsg/linux_drivers.rst |  6 ++++--
 doc/guides/linux_gsg/sys_reqs.rst      | 10 ++++++----
 3 files changed, 19 insertions(+), 13 deletions(-)

diff --git a/doc/guides/linux_gsg/enable_func.rst b/doc/guides/linux_gsg/enable_func.rst
index 7bd6b03f10..b5c4c652d7 100644
--- a/doc/guides/linux_gsg/enable_func.rst
+++ b/doc/guides/linux_gsg/enable_func.rst
@@ -1,6 +1,8 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright(c) 2010-2014 Intel Corporation.
 
+.. include:: <isonum.txt>
+
 .. _Enabling_Additional_Functionality:
 
 Enabling Additional Functionality
@@ -119,15 +121,15 @@ system objects' permissions should be adjusted:
 Power Management and Power Saving Functionality
 -----------------------------------------------
 
-Enhanced Intel SpeedStep® Technology must be enabled in the platform BIOS if the power management feature of DPDK is to be used.
+Enhanced Intel SpeedStep\ |reg| Technology must be enabled in the platform BIOS if the power management feature of DPDK is to be used.
 Otherwise, the sys file folder ``/sys/devices/system/cpu/cpu0/cpufreq`` will not exist, and the CPU frequency- based power management cannot be used.
 Consult the relevant BIOS documentation to determine how these settings can be accessed.
 
-For example, on some Intel reference platform BIOS variants, the path to Enhanced Intel SpeedStep® Technology is::
+For example, on some Intel reference platform BIOS variants, the path to Enhanced Intel SpeedStep\ |reg| Technology is::
 
    Advanced
      -> Processor Configuration
-     -> Enhanced Intel SpeedStep® Tech
+     -> Enhanced Intel SpeedStep\ |reg| Tech
 
 In addition, C3 and C6 should be enabled as well for power management. The path of C3 and C6 on the same platform BIOS is::
 
@@ -165,10 +167,10 @@ It should be loaded using the insmod command::
 
    See the "Kernel NIC Interface Sample Application" chapter in the *DPDK Sample Applications User Guide* for more details.
 
-Using Linux IOMMU Pass-Through to Run DPDK with Intel® VT-d
------------------------------------------------------------
+Using Linux IOMMU Pass-Through to Run DPDK with Intel\ |reg| VT-d
+------------------------------------------------------------------
 
-To enable Intel® VT-d in a Linux kernel, a number of kernel configuration options must be set. These include:
+To enable Intel\ |reg| VT-d in a Linux kernel, a number of kernel configuration options must be set. These include:
 
 *   ``IOMMU_SUPPORT``
 
@@ -176,7 +178,7 @@ To enable Intel® VT-d in a Linux kernel, a number of kernel configuration optio
 
 *   ``INTEL_IOMMU``
 
-In addition, to run the DPDK with Intel® VT-d, the ``iommu=pt`` kernel parameter must be used when using ``igb_uio`` driver.
+In addition, to run the DPDK with Intel\ |reg| VT-d, the ``iommu=pt`` kernel parameter must be used when using ``igb_uio`` driver.
 This results in pass-through of the DMAR (DMA Remapping) lookup in the host.
 Also, if ``INTEL_IOMMU_DEFAULT_ON`` is not set in the kernel, the ``intel_iommu=on`` kernel parameter must be used too.
 This ensures that the Intel IOMMU is being initialized as expected.
diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
index 2dd711bb37..56564286ac 100644
--- a/doc/guides/linux_gsg/linux_drivers.rst
+++ b/doc/guides/linux_gsg/linux_drivers.rst
@@ -3,6 +3,8 @@
     Copyright 2017 Mellanox Technologies, Ltd
     All rights reserved.
 
+.. include:: <isonum.txt>
+
 .. _linux_gsg_linux_drivers:
 
 Linux Drivers
@@ -99,7 +101,7 @@ The token will be used for all PF and VF ports within the application.
 
 To make use of full VFIO functionality,
 both kernel and BIOS must support and be configured
-to use IO virtualization (such as Intel® VT-d).
+to use IO virtualization (such as Intel\ |reg| VT-d).
 
 .. note::
 
@@ -335,7 +337,7 @@ Please refer to earlier sections on how to configure kernel parameters
 correctly for your system.
 
 If the kernel is configured correctly, one also has to make sure that
-the BIOS configuration has virtualization features (such as Intel® VT-d).
+the BIOS configuration has virtualization features (such as Intel\ |reg| VT-d).
 There is no standard way to check if the platform is configured correctly,
 so please check with your platform documentation to see if it has such features,
 and how to enable them.
diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
index d95a78d156..9dccd54d9c 100644
--- a/doc/guides/linux_gsg/sys_reqs.rst
+++ b/doc/guides/linux_gsg/sys_reqs.rst
@@ -1,6 +1,8 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright(c) 2010-2014 Intel Corporation.
 
+.. include:: <isonum.txt>
+
 System Requirements
 ===================
 
@@ -8,8 +10,8 @@ This chapter describes the packages required to compile the DPDK.
 
 .. note::
 
-    If the DPDK is being used on an Intel® Communications Chipset 89xx Series platform,
-    please consult the *Intel® Communications Chipset 89xx Series Software for Linux Getting Started Guide*.
+    If the DPDK is being used on an Intel\ |reg| Communications Chipset 89xx Series platform,
+    please consult the *Intel\ |reg| Communications Chipset 89xx Series Software for Linux Getting Started Guide*.
 
 BIOS Setting Prerequisite on x86
 --------------------------------
@@ -72,10 +74,10 @@ Compilation of the DPDK
 
 **Optional Tools:**
 
-*   Intel® C++ Compiler (icc). For installation, additional libraries may be required.
+*   Intel\ |reg| C++ Compiler (icc). For installation, additional libraries may be required.
     See the icc Installation Guide found in the Documentation directory under the compiler installation.
 
-*   IBM® Advance ToolChain for Powerlinux. This is a set of open source development tools and runtime libraries
+*   IBM\ |reg| Advance ToolChain for Powerlinux. This is a set of open source development tools and runtime libraries
     which allows users to take leading edge advantage of IBM's latest POWER hardware features on Linux. To install
     it, see the IBM official installation document.
 
-- 
2.32.0

[PATCH 2/9] doc/linux_gsg: fix missing note on UIO module

[Bruce Richardson]

The docs on binding drivers was updated as part of the removal of the
igb_uio module from the main DPDK repo. As part of that update, a note
about uio_pci_generic requiring legacy interrupts was removed, but
should have been kept.

Fixes: 56bb5841fd06 ("kernel/linux: remove igb_uio")
Cc: thomas@monjalon.net
Cc: stable@dpdk.org

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/linux_gsg/linux_drivers.rst | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
index 56564286ac..75af2f01e1 100644
--- a/doc/guides/linux_gsg/linux_drivers.rst
+++ b/doc/guides/linux_gsg/linux_drivers.rst
@@ -174,6 +174,11 @@ It can be loaded as shown below:
    sudo modprobe uio
    sudo insmod igb_uio.ko
 
+.. note::
+
+    For some devices which lack support for legacy interrupts, e.g. virtual function
+    (VF) devices, the ``igb_uio`` module may be needed in place of ``uio_pci_generic``.
+
 .. note::
 
    If UEFI secure boot is enabled,
-- 
2.32.0

[PATCH 3/9] doc/linux_gsg: make UIO safety warning more visible

[Bruce Richardson]

The GSG has a note warning that use of UIO is inherently unsafe due to
lack of IOMMU protection. However, this was only flagged as a "NOTE",
meaning it could easily be missed. Changing the rst tag from "note" to
"warning" and moving it to the top of the UIO subsection makes this a
lot more visible to users.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/linux_gsg/linux_drivers.rst | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
index 75af2f01e1..2cdacf0961 100644
--- a/doc/guides/linux_gsg/linux_drivers.rst
+++ b/doc/guides/linux_gsg/linux_drivers.rst
@@ -153,6 +153,11 @@ After that, VFIO can be used with hardware devices as usual.
 UIO
 ---

+.. warning::
+
+   Using UIO drivers is inherently unsafe due to this method lacking IOMMU protection,
+   and can only be done by root user.
+
 In situations where using VFIO is not an option, there are alternative drivers one can use.
 In many cases, the standard ``uio_pci_generic`` module included in the Linux kernel
 can be used as a substitute for VFIO. This module can be loaded using the command:
@@ -195,11 +200,6 @@ It can be loaded as shown below:
    in GRUB command line on x86_64 systems,
    or add ``iommu.passthrough=1`` on aarch64 systems.

-.. note::
-
-   Using UIO drivers is inherently unsafe due to this method lacking IOMMU protection,
-   and can only be done by root user.
-
 .. _bifurcated_driver:

 Bifurcated Driver
--
2.32.0

[PATCH 4/9] doc/linux_gsg: move section on binding drivers up the page

[Bruce Richardson]

While the details of VFIO and UIO may be of interest to some, most users
of the doc are likely primarily interested in how to bind their devices
to the kernel driver and then move on to running the app. Therefore, the
most important part of the "Linux Drivers" section of the GSG is the
subsection on "Binding and Unbinding", so put that first.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/linux_gsg/linux_drivers.rst | 179 +++++++++++++------------
 1 file changed, 90 insertions(+), 89 deletions(-)

diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
index 2cdacf0961..83f28d86fe 100644
--- a/doc/guides/linux_gsg/linux_drivers.rst
+++ b/doc/guides/linux_gsg/linux_drivers.rst
@@ -14,6 +14,96 @@ Different PMDs may require different kernel drivers in order to work properly.
 Depending on the PMD being used, a corresponding kernel driver should be loaded,
 and network ports should be bound to that driver.
 
+.. _linux_gsg_binding_kernel:
+
+Binding and Unbinding Network Ports to/from the Kernel Modules
+--------------------------------------------------------------
+
+.. note::
+
+   PMDs which use the bifurcated driver should not be unbound from their kernel drivers.
+   This section is for PMDs which use the UIO or VFIO drivers.
+   See :ref:`bifurcated_driver` section for more details.
+
+As of release 1.4, DPDK applications no longer automatically unbind all supported network ports from the kernel driver in use.
+Instead, in case the PMD being used use the VFIO or UIO drivers,
+all ports that are to be used by a DPDK application must be bound to
+the ``vfio-pci``, ``uio_pci_generic``, or ``igb_uio`` module
+before the application is run.
+For such PMDs, any network ports under Linux* control will be ignored and cannot be used by the application.
+
+To bind ports to the ``vfio-pci``, ``uio_pci_generic`` or ``igb_uio`` module
+for DPDK use, or to return ports to Linux control,
+a utility script called ``dpdk-devbind.py`` is provided in the ``usertools`` subdirectory.
+This utility can be used to provide a view of the current state of the network ports on the system,
+and to bind and unbind those ports from the different kernel modules,
+including the VFIO and UIO modules.
+The following are some examples of how the script can be used.
+A full description of the script and its parameters can be obtained
+by calling the script with the ``--help`` or ``--usage`` options.
+Note that the UIO or VFIO kernel modules to be used,
+should be loaded into the kernel before running the ``dpdk-devbind.py`` script.
+
+.. warning::
+
+   Due to the way VFIO works, there are certain limitations
+   to which devices can be used with VFIO.
+   Mainly it comes down to how IOMMU groups work.
+   Any Virtual Function device can usually be used with VFIO on its own,
+   but physical devices may require either all ports bound to VFIO,
+   or some of them bound to VFIO while others not being bound to anything at all.
+
+   If your device is behind a PCI-to-PCI bridge,
+   the bridge will then be part of the IOMMU group in which your device is in.
+   Therefore, the bridge driver should also be unbound from the bridge PCI device
+   for VFIO to work with devices behind the bridge.
+
+.. warning::
+
+   While any user can run the ``dpdk-devbind.py`` script
+   to view the status of the network ports,
+   binding or unbinding network ports requires root privileges.
+
+To see the status of all network ports on the system:
+
+.. code-block:: console
+
+    ./usertools/dpdk-devbind.py --status
+
+    Network devices using DPDK-compatible driver
+    ============================================
+    0000:82:00.0 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe
+    0000:82:00.1 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe
+
+    Network devices using kernel driver
+    ===================================
+    0000:04:00.0 'I350 1-GbE NIC' if=em0  drv=igb unused=uio_pci_generic *Active*
+    0000:04:00.1 'I350 1-GbE NIC' if=eth1 drv=igb unused=uio_pci_generic
+    0000:04:00.2 'I350 1-GbE NIC' if=eth2 drv=igb unused=uio_pci_generic
+    0000:04:00.3 'I350 1-GbE NIC' if=eth3 drv=igb unused=uio_pci_generic
+
+    Other network devices
+    =====================
+    <none>
+
+To bind device ``eth1``,``04:00.1``, to the ``uio_pci_generic`` driver:
+
+.. code-block:: console
+
+    ./usertools/dpdk-devbind.py --bind=uio_pci_generic 04:00.1
+
+or, alternatively,
+
+.. code-block:: console
+
+    ./usertools/dpdk-devbind.py --bind=uio_pci_generic eth1
+
+To restore device ``82:00.0`` to its original kernel binding:
+
+.. code-block:: console
+
+    ./usertools/dpdk-devbind.py --bind=ixgbe 82:00.0
+
 VFIO
 ----
 
@@ -225,95 +315,6 @@ More about the bifurcated driver can be found in
 `Mellanox Bifurcated DPDK PMD
 <https://www.dpdk.org/wp-content/uploads/sites/35/2016/10/Day02-Session04-RonyEfraim-Userspace2016.pdf>`__.
 
-.. _linux_gsg_binding_kernel:
-
-Binding and Unbinding Network Ports to/from the Kernel Modules
---------------------------------------------------------------
-
-.. note::
-
-   PMDs which use the bifurcated driver should not be unbound from their kernel drivers.
-   This section is for PMDs which use the UIO or VFIO drivers.
-
-As of release 1.4, DPDK applications no longer automatically unbind all supported network ports from the kernel driver in use.
-Instead, in case the PMD being used use the VFIO or UIO drivers,
-all ports that are to be used by a DPDK application must be bound to
-the ``vfio-pci``, ``uio_pci_generic``, or ``igb_uio`` module
-before the application is run.
-For such PMDs, any network ports under Linux* control will be ignored and cannot be used by the application.
-
-To bind ports to the ``vfio-pci``, ``uio_pci_generic`` or ``igb_uio`` module
-for DPDK use, or to return ports to Linux control,
-a utility script called ``dpdk-devbind.py`` is provided in the ``usertools`` subdirectory.
-This utility can be used to provide a view of the current state of the network ports on the system,
-and to bind and unbind those ports from the different kernel modules,
-including the VFIO and UIO modules.
-The following are some examples of how the script can be used.
-A full description of the script and its parameters can be obtained
-by calling the script with the ``--help`` or ``--usage`` options.
-Note that the UIO or VFIO kernel modules to be used,
-should be loaded into the kernel before running the ``dpdk-devbind.py`` script.
-
-.. warning::
-
-   Due to the way VFIO works, there are certain limitations
-   to which devices can be used with VFIO.
-   Mainly it comes down to how IOMMU groups work.
-   Any Virtual Function device can usually be used with VFIO on its own,
-   but physical devices may require either all ports bound to VFIO,
-   or some of them bound to VFIO while others not being bound to anything at all.
-
-   If your device is behind a PCI-to-PCI bridge,
-   the bridge will then be part of the IOMMU group in which your device is in.
-   Therefore, the bridge driver should also be unbound from the bridge PCI device
-   for VFIO to work with devices behind the bridge.
-
-.. warning::
-
-   While any user can run the ``dpdk-devbind.py`` script
-   to view the status of the network ports,
-   binding or unbinding network ports requires root privileges.
-
-To see the status of all network ports on the system:
-
-.. code-block:: console
-
-    ./usertools/dpdk-devbind.py --status
-
-    Network devices using DPDK-compatible driver
-    ============================================
-    0000:82:00.0 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe
-    0000:82:00.1 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe
-
-    Network devices using kernel driver
-    ===================================
-    0000:04:00.0 'I350 1-GbE NIC' if=em0  drv=igb unused=uio_pci_generic *Active*
-    0000:04:00.1 'I350 1-GbE NIC' if=eth1 drv=igb unused=uio_pci_generic
-    0000:04:00.2 'I350 1-GbE NIC' if=eth2 drv=igb unused=uio_pci_generic
-    0000:04:00.3 'I350 1-GbE NIC' if=eth3 drv=igb unused=uio_pci_generic
-
-    Other network devices
-    =====================
-    <none>
-
-To bind device ``eth1``,``04:00.1``, to the ``uio_pci_generic`` driver:
-
-.. code-block:: console
-
-    ./usertools/dpdk-devbind.py --bind=uio_pci_generic 04:00.1
-
-or, alternatively,
-
-.. code-block:: console
-
-    ./usertools/dpdk-devbind.py --bind=uio_pci_generic eth1
-
-To restore device ``82:00.0`` to its original kernel binding:
-
-.. code-block:: console
-
-    ./usertools/dpdk-devbind.py --bind=ixgbe 82:00.0
-
 Troubleshooting VFIO
 --------------------
 
-- 
2.32.0

[PATCH 5/9] doc/linux_gsg: emphasise VFIO over UIO-based modules

[Bruce Richardson]

VFIO is to be strongly preferred over uio-based modules, so update our
text and examples to only refer to vfio, giving an initial reference at
the start to uio as a fallback option.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/linux_gsg/linux_drivers.rst | 47 +++++++++++++++-----------
 1 file changed, 28 insertions(+), 19 deletions(-)

diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
index 83f28d86fe..af7e3c4fbc 100644
--- a/doc/guides/linux_gsg/linux_drivers.rst
+++ b/doc/guides/linux_gsg/linux_drivers.rst
@@ -25,14 +25,18 @@ Binding and Unbinding Network Ports to/from the Kernel Modules
    This section is for PMDs which use the UIO or VFIO drivers.
    See :ref:`bifurcated_driver` section for more details.
 
-As of release 1.4, DPDK applications no longer automatically unbind all supported network ports from the kernel driver in use.
-Instead, in case the PMD being used use the VFIO or UIO drivers,
-all ports that are to be used by a DPDK application must be bound to
-the ``vfio-pci``, ``uio_pci_generic``, or ``igb_uio`` module
-before the application is run.
-For such PMDs, any network ports under Linux* control will be ignored and cannot be used by the application.
-
-To bind ports to the ``vfio-pci``, ``uio_pci_generic`` or ``igb_uio`` module
+.. note::
+
+   It is recommended that ``vfio-pci`` be used as the kernel module for DPDK-bound ports in all cases.
+   If an IOMMU is unavailable, the ``vfio-pci`` can be used in :ref:`no-iommu<vfio_noiommu>` mode.
+   If, for some reason, vfio is unavailable, then UIO-based modules, ``igb_uio`` and ``uio_pci_generic`` may be used.
+   See section :ref:`uio` for details.
+
+Most devices require that the hardware to be used by DPDK be unbound from the kernel driver it uses,
+and instead be bound to the ``vfio-pci`` kernel module before the application is run.
+For such PMDs, any network ports or other hardware under Linux* control will be ignored and cannot be used by the application.
+
+To bind ports to the ``vfio-pci`` module
 for DPDK use, or to return ports to Linux control,
 a utility script called ``dpdk-devbind.py`` is provided in the ``usertools`` subdirectory.
 This utility can be used to provide a view of the current state of the network ports on the system,
@@ -72,37 +76,38 @@ To see the status of all network ports on the system:
 
     Network devices using DPDK-compatible driver
     ============================================
-    0000:82:00.0 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe
-    0000:82:00.1 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe
+    0000:82:00.0 '82599EB 10-GbE NIC' drv=vfio-pci unused=ixgbe
+    0000:82:00.1 '82599EB 10-GbE NIC' drv=vfio-pci unused=ixgbe
 
     Network devices using kernel driver
     ===================================
-    0000:04:00.0 'I350 1-GbE NIC' if=em0  drv=igb unused=uio_pci_generic *Active*
-    0000:04:00.1 'I350 1-GbE NIC' if=eth1 drv=igb unused=uio_pci_generic
-    0000:04:00.2 'I350 1-GbE NIC' if=eth2 drv=igb unused=uio_pci_generic
-    0000:04:00.3 'I350 1-GbE NIC' if=eth3 drv=igb unused=uio_pci_generic
+    0000:04:00.0 'I350 1-GbE NIC' if=em0  drv=igb unused=vfio-pci *Active*
+    0000:04:00.1 'I350 1-GbE NIC' if=eth1 drv=igb unused=vfio-pci
+    0000:04:00.2 'I350 1-GbE NIC' if=eth2 drv=igb unused=vfio-pci
+    0000:04:00.3 'I350 1-GbE NIC' if=eth3 drv=igb unused=vfio-pci
 
     Other network devices
     =====================
     <none>
 
-To bind device ``eth1``,``04:00.1``, to the ``uio_pci_generic`` driver:
+To bind device ``eth1``,``04:00.1``, to the ``vfio-pci`` driver:
 
 .. code-block:: console
 
-    ./usertools/dpdk-devbind.py --bind=uio_pci_generic 04:00.1
+    ./usertools/dpdk-devbind.py --bind=vfio-pci 04:00.1
 
 or, alternatively,
 
 .. code-block:: console
 
-    ./usertools/dpdk-devbind.py --bind=uio_pci_generic eth1
+    ./usertools/dpdk-devbind.py --bind=vfio-pci eth1
 
-To restore device ``82:00.0`` to its original kernel binding:
+When specifying device ids, wildcards can be used for the final part of the address.
+To restore device ``82:00.0`` and ``82:00.1`` to their original kernel binding:
 
 .. code-block:: console
 
-    ./usertools/dpdk-devbind.py --bind=ixgbe 82:00.0
+    ./usertools/dpdk-devbind.py --bind=ixgbe 82:00.*
 
 VFIO
 ----
@@ -210,6 +215,8 @@ to use IO virtualization (such as Intel\ |reg| VT-d).
 For proper operation of VFIO when running DPDK applications as a non-privileged user, correct permissions should also be set up.
 For more information, please refer to :ref:`Running_Without_Root_Privileges`.
 
+.. _vfio_noiommu:
+
 VFIO no-IOMMU mode
 ------------------
 
@@ -240,6 +247,8 @@ After that, VFIO can be used with hardware devices as usual.
    to keep the degree of device access and programming that VFIO has,
    in situations where IOMMU is not available.
 
+.. _uio:
+
 UIO
 ---
 
-- 
2.32.0

[PATCH 6/9] doc/linux_gsg: split VFIO section into subsections

[Bruce Richardson]

The VFIO section of the page about linux drivers was rather long and
unstructured. This can be improved by splitting it up into subsections,
to cover the specifics of memory limits and creating VFs. When moving
the various text notes into the relevant subsections, we can drop the
note about kernels earlier than 3.6, since DPDK no longer supports
kernels that old.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/linux_gsg/linux_drivers.rst | 35 ++++++++++++++------------
 1 file changed, 19 insertions(+), 16 deletions(-)

diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
index af7e3c4fbc..e29f79e385 100644
--- a/doc/guides/linux_gsg/linux_drivers.rst
+++ b/doc/guides/linux_gsg/linux_drivers.rst
@@ -122,6 +122,22 @@ To make use of VFIO, the ``vfio-pci`` module must be loaded:
 VFIO kernel is usually present by default in all distributions,
 however please consult your distributions documentation to make sure that is the case.
 
+To make use of full VFIO functionality,
+both kernel and BIOS must support and be configured
+to use IO virtualization (such as Intel\ |reg| VT-d).
+
+.. note::
+
+   In most cases, specifying "iommu=on" as kernel parameter should be enough to
+   configure the Linux kernel to use IOMMU.
+
+For proper operation of VFIO when running DPDK applications as a non-privileged user, correct permissions should also be set up.
+For more information, please refer to :ref:`Running_Without_Root_Privileges`.
+
+
+VFIO Memory Mapping Limits
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 For DMA mapping of either external memory or hugepages, VFIO interface is used.
 VFIO does not support partial unmap of once mapped memory. Hence DPDK's memory is
 mapped in hugepage granularity or system page granularity. Number of DMA
@@ -132,6 +148,9 @@ VFIO module parameter ``dma_entry_limit`` with a default value of 64K.
 When application is out of DMA entries, these limits need to be adjusted to
 increase the allowed limit.
 
+Creating Virtual Functions using vfio-pci
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 Since Linux version 5.7,
 the ``vfio-pci`` module supports the creation of virtual functions.
 After the PF is bound to ``vfio-pci`` module,
@@ -194,27 +213,11 @@ The token will be used for all PF and VF ports within the application.
       <build_dir>/app/dpdk-testpmd -l 26-29 -n 4 -a 86:02.0 \
       --vfio-vf-token=14d63f20-8445-11ea-8900-1f9ce7d5650d --file-prefix=vf0 -- -i
 
-To make use of full VFIO functionality,
-both kernel and BIOS must support and be configured
-to use IO virtualization (such as Intel\ |reg| VT-d).
-
-.. note::
-
-   Linux versions earlier than version 3.6 do not support VFIO.
-
 .. note::
 
    Linux versions earlier than version 5.7 do not support the creation of
    virtual functions within the VFIO framework.
 
-.. note::
-
-   In most cases, specifying "iommu=on" as kernel parameter should be enough to
-   configure the Linux kernel to use IOMMU.
-
-For proper operation of VFIO when running DPDK applications as a non-privileged user, correct permissions should also be set up.
-For more information, please refer to :ref:`Running_Without_Root_Privileges`.
-
 .. _vfio_noiommu:
 
 VFIO no-IOMMU mode
-- 
2.32.0

[PATCH 7/9] doc/linux_gsg: move UIO section to the end

[Bruce Richardson]

To further de-emphasise UIO over the alternatives, we can move the UIO
section of the drivers page to the end of the document, giving more
prominence to VFIO and bifurcated drivers.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/linux_gsg/linux_drivers.rst | 104 ++++++++++++-------------
 1 file changed, 52 insertions(+), 52 deletions(-)

diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
index e29f79e385..b2be442d27 100644
--- a/doc/guides/linux_gsg/linux_drivers.rst
+++ b/doc/guides/linux_gsg/linux_drivers.rst
@@ -250,58 +250,6 @@ After that, VFIO can be used with hardware devices as usual.
    to keep the degree of device access and programming that VFIO has,
    in situations where IOMMU is not available.
 
-.. _uio:
-
-UIO
----
-
-.. warning::
-
-   Using UIO drivers is inherently unsafe due to this method lacking IOMMU protection,
-   and can only be done by root user.
-
-In situations where using VFIO is not an option, there are alternative drivers one can use.
-In many cases, the standard ``uio_pci_generic`` module included in the Linux kernel
-can be used as a substitute for VFIO. This module can be loaded using the command:
-
-.. code-block:: console
-
-   sudo modprobe uio_pci_generic
-
-.. note::
-
-   ``uio_pci_generic`` module doesn't support the creation of virtual functions.
-
-As an alternative to the ``uio_pci_generic``, there is the ``igb_uio`` module
-which can be found in the repository `dpdk-kmods <http://git.dpdk.org/dpdk-kmods>`_.
-It can be loaded as shown below:
-
-.. code-block:: console
-
-   sudo modprobe uio
-   sudo insmod igb_uio.ko
-
-.. note::
-
-    For some devices which lack support for legacy interrupts, e.g. virtual function
-    (VF) devices, the ``igb_uio`` module may be needed in place of ``uio_pci_generic``.
-
-.. note::
-
-   If UEFI secure boot is enabled,
-   the Linux kernel may disallow the use of UIO on the system.
-   Therefore, devices for use by DPDK should be bound to the ``vfio-pci`` kernel module
-   rather than any UIO-based module.
-   For more details see :ref:`linux_gsg_binding_kernel` below.
-
-.. note::
-
-   If the devices used for DPDK are bound to the ``uio_pci_generic`` kernel module,
-   please make sure that the IOMMU is disabled or is in passthrough mode.
-   One can add ``intel_iommu=off`` or ``amd_iommu=off`` or ``intel_iommu=on iommu=pt``
-   in GRUB command line on x86_64 systems,
-   or add ``iommu.passthrough=1`` on aarch64 systems.
-
 .. _bifurcated_driver:
 
 Bifurcated Driver
@@ -372,3 +320,55 @@ This can be checked in the boot configuration of your system:
 If ``CONFIG_VFIO_NOIOMMU`` is not enabled in the kernel configuration,
 VFIO driver will not support the no-IOMMU mode,
 and other alternatives (such as UIO drivers) will have to be used.
+
+.. _uio:
+
+UIO
+---
+
+.. warning::
+
+   Using UIO drivers is inherently unsafe due to this method lacking IOMMU protection,
+   and can only be done by root user.
+
+In situations where using VFIO is not an option, there are alternative drivers one can use.
+In many cases, the standard ``uio_pci_generic`` module included in the Linux kernel
+can be used as a substitute for VFIO. This module can be loaded using the command:
+
+.. code-block:: console
+
+   sudo modprobe uio_pci_generic
+
+.. note::
+
+   ``uio_pci_generic`` module doesn't support the creation of virtual functions.
+
+As an alternative to the ``uio_pci_generic``, there is the ``igb_uio`` module
+which can be found in the repository `dpdk-kmods <http://git.dpdk.org/dpdk-kmods>`_.
+It can be loaded as shown below:
+
+.. code-block:: console
+
+   sudo modprobe uio
+   sudo insmod igb_uio.ko
+
+.. note::
+
+    For some devices which lack support for legacy interrupts, e.g. virtual function
+    (VF) devices, the ``igb_uio`` module may be needed in place of ``uio_pci_generic``.
+
+.. note::
+
+   If UEFI secure boot is enabled,
+   the Linux kernel may disallow the use of UIO on the system.
+   Therefore, devices for use by DPDK should be bound to the ``vfio-pci`` kernel module
+   rather than any UIO-based module.
+   For more details see :ref:`linux_gsg_binding_kernel` below.
+
+.. note::
+
+   If the devices used for DPDK are bound to the ``uio_pci_generic`` kernel module,
+   please make sure that the IOMMU is disabled or is in passthrough mode.
+   One can add ``intel_iommu=off`` or ``amd_iommu=off`` or ``intel_iommu=on iommu=pt``
+   in GRUB command line on x86_64 systems,
+   or add ``iommu.passthrough=1`` on aarch64 systems.
-- 
2.32.0

[PATCH 8/9] doc/linux_gsg: consolidate all VFIO content

[Bruce Richardson]

Rather than having separate sections for VFIO and VFIO no-iommu mode, as
well as a separate section further down the document on troubleshooting
VFIO, we can consolidate all these as subsections into a primary VFIO
section. This section starts with the basics of VFIO use, then covers
no-iommu mode, before moving on to the more advanced topics such as
creating VFs and ending with the troubleshooting subsection.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/linux_gsg/linux_drivers.rst | 116 ++++++++++++-------------
 1 file changed, 58 insertions(+), 58 deletions(-)

diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
index b2be442d27..ef91999dd9 100644
--- a/doc/guides/linux_gsg/linux_drivers.rst
+++ b/doc/guides/linux_gsg/linux_drivers.rst
@@ -135,6 +135,38 @@ For proper operation of VFIO when running DPDK applications as a non-privileged
 For more information, please refer to :ref:`Running_Without_Root_Privileges`.
 
 
+.. _vfio_noiommu:
+
+VFIO no-IOMMU mode
+~~~~~~~~~~~~~~~~~~
+
+If there is no IOMMU available on the system, VFIO can still be used,
+but it has to be loaded with an additional module parameter:
+
+.. code-block:: console
+
+   modprobe vfio enable_unsafe_noiommu_mode=1
+
+Alternatively, one can also enable this option in an already loaded kernel module:
+
+.. code-block:: console
+
+   echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
+
+After that, VFIO can be used with hardware devices as usual.
+
+.. note::
+
+   It may be required to unload all VFIO related-modules before probing
+   the module again with ``enable_unsafe_noiommu_mode=1`` parameter.
+
+.. warning::
+
+   Since no-IOMMU mode forgoes IOMMU protection, it is inherently unsafe.
+   That said, it does make it possible for the user
+   to keep the degree of device access and programming that VFIO has,
+   in situations where IOMMU is not available.
+
 VFIO Memory Mapping Limits
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -218,65 +250,8 @@ The token will be used for all PF and VF ports within the application.
    Linux versions earlier than version 5.7 do not support the creation of
    virtual functions within the VFIO framework.
 
-.. _vfio_noiommu:
-
-VFIO no-IOMMU mode
-------------------
-
-If there is no IOMMU available on the system, VFIO can still be used,
-but it has to be loaded with an additional module parameter:
-
-.. code-block:: console
-
-   modprobe vfio enable_unsafe_noiommu_mode=1
-
-Alternatively, one can also enable this option in an already loaded kernel module:
-
-.. code-block:: console
-
-   echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
-
-After that, VFIO can be used with hardware devices as usual.
-
-.. note::
-
-   It may be required to unload all VFIO related-modules before probing
-   the module again with ``enable_unsafe_noiommu_mode=1`` parameter.
-
-.. warning::
-
-   Since no-IOMMU mode forgoes IOMMU protection, it is inherently unsafe.
-   That said, it does make it possible for the user
-   to keep the degree of device access and programming that VFIO has,
-   in situations where IOMMU is not available.
-
-.. _bifurcated_driver:
-
-Bifurcated Driver
------------------
-
-PMDs which use the bifurcated driver co-exists with the device kernel driver.
-On such model the NIC is controlled by the kernel, while the data
-path is performed by the PMD directly on top of the device.
-
-Such model has the following benefits:
-
- - It is secure and robust, as the memory management and isolation
-   is done by the kernel.
- - It enables the user to use legacy linux tools such as ``ethtool`` or
-   ``ifconfig`` while running DPDK application on the same network ports.
- - It enables the DPDK application to filter only part of the traffic,
-   while the rest will be directed and handled by the kernel driver.
-   The flow bifurcation is performed by the NIC hardware.
-   As an example, using :ref:`flow_isolated_mode` allows to choose
-   strictly what is received in DPDK.
-
-More about the bifurcated driver can be found in
-`Mellanox Bifurcated DPDK PMD
-<https://www.dpdk.org/wp-content/uploads/sites/35/2016/10/Day02-Session04-RonyEfraim-Userspace2016.pdf>`__.
-
 Troubleshooting VFIO
---------------------
+~~~~~~~~~~~~~~~~~~~~
 
 In certain situations, using ``dpdk-devbind.py`` script
 to bind a device to VFIO driver may fail.
@@ -321,6 +296,31 @@ If ``CONFIG_VFIO_NOIOMMU`` is not enabled in the kernel configuration,
 VFIO driver will not support the no-IOMMU mode,
 and other alternatives (such as UIO drivers) will have to be used.
 
+.. _bifurcated_driver:
+
+Bifurcated Driver
+-----------------
+
+PMDs which use the bifurcated driver co-exists with the device kernel driver.
+On such model the NIC is controlled by the kernel, while the data
+path is performed by the PMD directly on top of the device.
+
+Such model has the following benefits:
+
+ - It is secure and robust, as the memory management and isolation
+   is done by the kernel.
+ - It enables the user to use legacy linux tools such as ``ethtool`` or
+   ``ifconfig`` while running DPDK application on the same network ports.
+ - It enables the DPDK application to filter only part of the traffic,
+   while the rest will be directed and handled by the kernel driver.
+   The flow bifurcation is performed by the NIC hardware.
+   As an example, using :ref:`flow_isolated_mode` allows to choose
+   strictly what is received in DPDK.
+
+More about the bifurcated driver can be found in
+`Mellanox Bifurcated DPDK PMD
+<https://www.dpdk.org/wp-content/uploads/sites/35/2016/10/Day02-Session04-RonyEfraim-Userspace2016.pdf>`__.
+
 .. _uio:
 
 UIO
-- 
2.32.0

[PATCH 9/9] doc/linux_gsg: change informational warnings to notes

[Bruce Richardson]

There are two warnings in the VFIO section about limitations of VFIO and
limitations on who can bind/unbind devices. Since these don't actually
describe any unsafe conditions, and are more informational, we can
change these to notes. This also helps emphasise the other warnings in
the documents which flag genuine security concerns.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/linux_gsg/linux_drivers.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
index ef91999dd9..03cf264a0a 100644
--- a/doc/guides/linux_gsg/linux_drivers.rst
+++ b/doc/guides/linux_gsg/linux_drivers.rst
@@ -48,7 +48,7 @@ by calling the script with the ``--help`` or ``--usage`` options.
 Note that the UIO or VFIO kernel modules to be used,
 should be loaded into the kernel before running the ``dpdk-devbind.py`` script.
 
-.. warning::
+.. note::
 
    Due to the way VFIO works, there are certain limitations
    to which devices can be used with VFIO.
@@ -62,7 +62,7 @@ should be loaded into the kernel before running the ``dpdk-devbind.py`` script.
    Therefore, the bridge driver should also be unbound from the bridge PCI device
    for VFIO to work with devices behind the bridge.
 
-.. warning::
+.. note::
 
    While any user can run the ``dpdk-devbind.py`` script
    to view the status of the network ports,
-- 
2.32.0

[PATCH v2 1/9] doc: replace special characters for (R) symbol in Linux GSG

[Bruce Richardson]

Some IDEs, such as eclipse, complained on save about the use of special
characters in the (R) symbol in linux GSG doc. We can replace those with
the equivalent "|reg|" text, and including isonum.txt.

Cc: stable@dpdk.org

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/linux_gsg/enable_func.rst   | 8 +++++---
 doc/guides/linux_gsg/linux_drivers.rst | 6 ++++--
 doc/guides/linux_gsg/sys_reqs.rst      | 6 ++++--
 3 files changed, 13 insertions(+), 7 deletions(-)

diff --git a/doc/guides/linux_gsg/enable_func.rst b/doc/guides/linux_gsg/enable_func.rst
index 44c3b05130..1df3ab0255 100644
--- a/doc/guides/linux_gsg/enable_func.rst
+++ b/doc/guides/linux_gsg/enable_func.rst
@@ -1,6 +1,8 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright(c) 2010-2014 Intel Corporation.

+.. include:: <isonum.txt>
+
 .. _Enabling_Additional_Functionality:

 Enabling Additional Functionality
@@ -66,15 +68,15 @@ system objects' permissions should be adjusted:
 Power Management and Power Saving Functionality
 -----------------------------------------------

-Enhanced Intel SpeedStep® Technology must be enabled in the platform BIOS if the power management feature of DPDK is to be used.
+Enhanced Intel SpeedStep\ |reg| Technology must be enabled in the platform BIOS if the power management feature of DPDK is to be used.
 Otherwise, the sys file folder ``/sys/devices/system/cpu/cpu0/cpufreq`` will not exist, and the CPU frequency- based power management cannot be used.
 Consult the relevant BIOS documentation to determine how these settings can be accessed.

-For example, on some Intel reference platform BIOS variants, the path to Enhanced Intel SpeedStep® Technology is::
+For example, on some Intel reference platform BIOS variants, the path to Enhanced Intel SpeedStep\ |reg| Technology is::

    Advanced
      -> Processor Configuration
-     -> Enhanced Intel SpeedStep® Tech
+     -> Enhanced Intel SpeedStep\ |reg| Tech

 In addition, C3 and C6 should be enabled as well for power management. The path of C3 and C6 on the same platform BIOS is::

diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
index 006a9df83e..ef6fec10d7 100644
--- a/doc/guides/linux_gsg/linux_drivers.rst
+++ b/doc/guides/linux_gsg/linux_drivers.rst
@@ -3,6 +3,8 @@
     Copyright 2017 Mellanox Technologies, Ltd
     All rights reserved.

+.. include:: <isonum.txt>
+
 .. _linux_gsg_linux_drivers:

 Linux Drivers
@@ -99,7 +101,7 @@ The token will be used for all PF and VF ports within the application.

 To make use of full VFIO functionality,
 both kernel and BIOS must support and be configured
-to use IO virtualization (such as Intel® VT-d).
+to use IO virtualization (such as Intel\ |reg| VT-d).

 .. note::

@@ -335,7 +337,7 @@ Please refer to earlier sections on how to configure kernel parameters
 correctly for your system.

 If the kernel is configured correctly, one also has to make sure that
-the BIOS configuration has virtualization features (such as Intel® VT-d).
+the BIOS configuration has virtualization features (such as Intel\ |reg| VT-d).
 There is no standard way to check if the platform is configured correctly,
 so please check with your platform documentation to see if it has such features,
 and how to enable them.
diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
index b2eacac6dc..08d45898f0 100644
--- a/doc/guides/linux_gsg/sys_reqs.rst
+++ b/doc/guides/linux_gsg/sys_reqs.rst
@@ -1,6 +1,8 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright(c) 2010-2014 Intel Corporation.

+.. include:: <isonum.txt>
+
 System Requirements
 ===================

@@ -68,10 +70,10 @@ Compilation of the DPDK

 **Optional Tools:**

-*   Intel® C++ Compiler (icc). For installation, additional libraries may be required.
+*   Intel\ |reg| C++ Compiler (icc). For installation, additional libraries may be required.
     See the icc Installation Guide found in the Documentation directory under the compiler installation.

-*   IBM® Advance ToolChain for Powerlinux. This is a set of open source development tools and runtime libraries
+*   IBM\ |reg| Advance ToolChain for Powerlinux. This is a set of open source development tools and runtime libraries
     which allows users to take leading edge advantage of IBM's latest POWER hardware features on Linux. To install
     it, see the IBM official installation document.

--
2.32.0

[PATCH v2 2/9] doc: fix missing note on UIO module in Linux GSG

[Bruce Richardson]

The docs on binding drivers was updated as part of the removal of the
igb_uio module from the main DPDK repo. As part of that update, a note
about uio_pci_generic requiring legacy interrupts was removed, but
should have been kept.

Fixes: 56bb5841fd06 ("kernel/linux: remove igb_uio")
Cc: thomas@monjalon.net
Cc: stable@dpdk.org

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/linux_gsg/linux_drivers.rst | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
index ef6fec10d7..bd983b4d81 100644
--- a/doc/guides/linux_gsg/linux_drivers.rst
+++ b/doc/guides/linux_gsg/linux_drivers.rst
@@ -174,6 +174,11 @@ It can be loaded as shown below:
    sudo modprobe uio
    sudo insmod igb_uio.ko
 
+.. note::
+
+    For some devices which lack support for legacy interrupts, e.g. virtual function
+    (VF) devices, the ``igb_uio`` module may be needed in place of ``uio_pci_generic``.
+
 .. note::
 
    If UEFI secure boot is enabled,
-- 
2.32.0

[PATCH v2 3/9] doc: make UIO safety warning more visible in Linux GSG

[Bruce Richardson]

The GSG has a note warning that use of UIO is inherently unsafe due to
lack of IOMMU protection. However, this was only flagged as a "NOTE",
meaning it could easily be missed. Changing the rst tag from "note" to
"warning" and moving it to the top of the UIO subsection makes this a
lot more visible to users.

Cc: stable@dpdk.org

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/linux_gsg/linux_drivers.rst | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
index bd983b4d81..f0a274df6a 100644
--- a/doc/guides/linux_gsg/linux_drivers.rst
+++ b/doc/guides/linux_gsg/linux_drivers.rst
@@ -153,6 +153,11 @@ After that, VFIO can be used with hardware devices as usual.
 UIO
 ---

+.. warning::
+
+   Using UIO drivers is inherently unsafe due to this method lacking IOMMU protection,
+   and can only be done by root user.
+
 In situations where using VFIO is not an option, there are alternative drivers one can use.
 In many cases, the standard ``uio_pci_generic`` module included in the Linux kernel
 can be used as a substitute for VFIO. This module can be loaded using the command:
@@ -195,11 +200,6 @@ It can be loaded as shown below:
    in GRUB command line on x86_64 systems,
    or add ``iommu.passthrough=1`` on aarch64 systems.

-.. note::
-
-   Using UIO drivers is inherently unsafe due to this method lacking IOMMU protection,
-   and can only be done by root user.
-
 .. _bifurcated_driver:

 Bifurcated Driver
--
2.32.0

[PATCH v2 4/9] doc: move section on binding drivers up the page in GSG

[Bruce Richardson]

While the details of VFIO and UIO may be of interest to some, most users
of the doc are likely primarily interested in how to bind their devices
to the kernel driver and then move on to running the app. Therefore, the
most important part of the "Linux Drivers" section of the GSG is the
subsection on "Binding and Unbinding", so put that first.

Cc: stable@dpdk.org

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/linux_gsg/linux_drivers.rst | 179 +++++++++++++------------
 1 file changed, 90 insertions(+), 89 deletions(-)

diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
index f0a274df6a..b93feae7a1 100644
--- a/doc/guides/linux_gsg/linux_drivers.rst
+++ b/doc/guides/linux_gsg/linux_drivers.rst
@@ -14,6 +14,96 @@ Different PMDs may require different kernel drivers in order to work properly.
 Depending on the PMD being used, a corresponding kernel driver should be loaded,
 and network ports should be bound to that driver.

+.. _linux_gsg_binding_kernel:
+
+Binding and Unbinding Network Ports to/from the Kernel Modules
+--------------------------------------------------------------
+
+.. note::
+
+   PMDs which use the bifurcated driver should not be unbound from their kernel drivers.
+   This section is for PMDs which use the UIO or VFIO drivers.
+   See :ref:`bifurcated_driver` section for more details.
+
+As of release 1.4, DPDK applications no longer automatically unbind all supported network ports from the kernel driver in use.
+Instead, in case the PMD being used use the VFIO or UIO drivers,
+all ports that are to be used by a DPDK application must be bound to
+the ``vfio-pci``, ``uio_pci_generic``, or ``igb_uio`` module
+before the application is run.
+For such PMDs, any network ports under Linux* control will be ignored and cannot be used by the application.
+
+To bind ports to the ``vfio-pci``, ``uio_pci_generic`` or ``igb_uio`` module
+for DPDK use, or to return ports to Linux control,
+a utility script called ``dpdk-devbind.py`` is provided in the ``usertools`` subdirectory.
+This utility can be used to provide a view of the current state of the network ports on the system,
+and to bind and unbind those ports from the different kernel modules,
+including the VFIO and UIO modules.
+The following are some examples of how the script can be used.
+A full description of the script and its parameters can be obtained
+by calling the script with the ``--help`` or ``--usage`` options.
+Note that the UIO or VFIO kernel modules to be used,
+should be loaded into the kernel before running the ``dpdk-devbind.py`` script.
+
+.. warning::
+
+   Due to the way VFIO works, there are certain limitations
+   to which devices can be used with VFIO.
+   Mainly it comes down to how IOMMU groups work.
+   Any Virtual Function device can usually be used with VFIO on its own,
+   but physical devices may require either all ports bound to VFIO,
+   or some of them bound to VFIO while others not being bound to anything at all.
+
+   If your device is behind a PCI-to-PCI bridge,
+   the bridge will then be part of the IOMMU group in which your device is in.
+   Therefore, the bridge driver should also be unbound from the bridge PCI device
+   for VFIO to work with devices behind the bridge.
+
+.. warning::
+
+   While any user can run the ``dpdk-devbind.py`` script
+   to view the status of the network ports,
+   binding or unbinding network ports requires root privileges.
+
+To see the status of all network ports on the system:
+
+.. code-block:: console
+
+    ./usertools/dpdk-devbind.py --status
+
+    Network devices using DPDK-compatible driver
+    ============================================
+    0000:82:00.0 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe
+    0000:82:00.1 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe
+
+    Network devices using kernel driver
+    ===================================
+    0000:04:00.0 'I350 1-GbE NIC' if=em0  drv=igb unused=uio_pci_generic *Active*
+    0000:04:00.1 'I350 1-GbE NIC' if=eth1 drv=igb unused=uio_pci_generic
+    0000:04:00.2 'I350 1-GbE NIC' if=eth2 drv=igb unused=uio_pci_generic
+    0000:04:00.3 'I350 1-GbE NIC' if=eth3 drv=igb unused=uio_pci_generic
+
+    Other network devices
+    =====================
+    <none>
+
+To bind device ``eth1``,``04:00.1``, to the ``uio_pci_generic`` driver:
+
+.. code-block:: console
+
+    ./usertools/dpdk-devbind.py --bind=uio_pci_generic 04:00.1
+
+or, alternatively,
+
+.. code-block:: console
+
+    ./usertools/dpdk-devbind.py --bind=uio_pci_generic eth1
+
+To restore device ``82:00.0`` to its original kernel binding:
+
+.. code-block:: console
+
+    ./usertools/dpdk-devbind.py --bind=ixgbe 82:00.0
+
 VFIO
 ----

@@ -225,95 +315,6 @@ More about the bifurcated driver can be found in
 `Mellanox Bifurcated DPDK PMD
 <https://www.dpdk.org/wp-content/uploads/sites/35/2016/10/Day02-Session04-RonyEfraim-Userspace2016.pdf>`__.

-.. _linux_gsg_binding_kernel:
-
-Binding and Unbinding Network Ports to/from the Kernel Modules
---------------------------------------------------------------
-
-.. note::
-
-   PMDs which use the bifurcated driver should not be unbound from their kernel drivers.
-   This section is for PMDs which use the UIO or VFIO drivers.
-
-As of release 1.4, DPDK applications no longer automatically unbind all supported network ports from the kernel driver in use.
-Instead, in case the PMD being used use the VFIO or UIO drivers,
-all ports that are to be used by a DPDK application must be bound to
-the ``vfio-pci``, ``uio_pci_generic``, or ``igb_uio`` module
-before the application is run.
-For such PMDs, any network ports under Linux* control will be ignored and cannot be used by the application.
-
-To bind ports to the ``vfio-pci``, ``uio_pci_generic`` or ``igb_uio`` module
-for DPDK use, or to return ports to Linux control,
-a utility script called ``dpdk-devbind.py`` is provided in the ``usertools`` subdirectory.
-This utility can be used to provide a view of the current state of the network ports on the system,
-and to bind and unbind those ports from the different kernel modules,
-including the VFIO and UIO modules.
-The following are some examples of how the script can be used.
-A full description of the script and its parameters can be obtained
-by calling the script with the ``--help`` or ``--usage`` options.
-Note that the UIO or VFIO kernel modules to be used,
-should be loaded into the kernel before running the ``dpdk-devbind.py`` script.
-
-.. warning::
-
-   Due to the way VFIO works, there are certain limitations
-   to which devices can be used with VFIO.
-   Mainly it comes down to how IOMMU groups work.
-   Any Virtual Function device can usually be used with VFIO on its own,
-   but physical devices may require either all ports bound to VFIO,
-   or some of them bound to VFIO while others not being bound to anything at all.
-
-   If your device is behind a PCI-to-PCI bridge,
-   the bridge will then be part of the IOMMU group in which your device is in.
-   Therefore, the bridge driver should also be unbound from the bridge PCI device
-   for VFIO to work with devices behind the bridge.
-
-.. warning::
-
-   While any user can run the ``dpdk-devbind.py`` script
-   to view the status of the network ports,
-   binding or unbinding network ports requires root privileges.
-
-To see the status of all network ports on the system:
-
-.. code-block:: console
-
-    ./usertools/dpdk-devbind.py --status
-
-    Network devices using DPDK-compatible driver
-    ============================================
-    0000:82:00.0 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe
-    0000:82:00.1 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe
-
-    Network devices using kernel driver
-    ===================================
-    0000:04:00.0 'I350 1-GbE NIC' if=em0  drv=igb unused=uio_pci_generic *Active*
-    0000:04:00.1 'I350 1-GbE NIC' if=eth1 drv=igb unused=uio_pci_generic
-    0000:04:00.2 'I350 1-GbE NIC' if=eth2 drv=igb unused=uio_pci_generic
-    0000:04:00.3 'I350 1-GbE NIC' if=eth3 drv=igb unused=uio_pci_generic
-
-    Other network devices
-    =====================
-    <none>
-
-To bind device ``eth1``,``04:00.1``, to the ``uio_pci_generic`` driver:
-
-.. code-block:: console
-
-    ./usertools/dpdk-devbind.py --bind=uio_pci_generic 04:00.1
-
-or, alternatively,
-
-.. code-block:: console
-
-    ./usertools/dpdk-devbind.py --bind=uio_pci_generic eth1
-
-To restore device ``82:00.0`` to its original kernel binding:
-
-.. code-block:: console
-
-    ./usertools/dpdk-devbind.py --bind=ixgbe 82:00.0
-
 Troubleshooting VFIO
 --------------------

--
2.32.0

[PATCH v2 5/9] doc: emphasise VFIO over UIO-based modules in GSG

[Bruce Richardson]

VFIO is to be strongly preferred over uio-based modules, so update our
text and examples to only refer to vfio, giving an initial reference at
the start to uio as a fallback option.

Cc: stable@dpdk.org

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/linux_gsg/linux_drivers.rst | 47 +++++++++++++++-----------
 1 file changed, 28 insertions(+), 19 deletions(-)

diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
index b93feae7a1..bd649db929 100644
--- a/doc/guides/linux_gsg/linux_drivers.rst
+++ b/doc/guides/linux_gsg/linux_drivers.rst
@@ -25,14 +25,18 @@ Binding and Unbinding Network Ports to/from the Kernel Modules
    This section is for PMDs which use the UIO or VFIO drivers.
    See :ref:`bifurcated_driver` section for more details.

-As of release 1.4, DPDK applications no longer automatically unbind all supported network ports from the kernel driver in use.
-Instead, in case the PMD being used use the VFIO or UIO drivers,
-all ports that are to be used by a DPDK application must be bound to
-the ``vfio-pci``, ``uio_pci_generic``, or ``igb_uio`` module
-before the application is run.
-For such PMDs, any network ports under Linux* control will be ignored and cannot be used by the application.
-
-To bind ports to the ``vfio-pci``, ``uio_pci_generic`` or ``igb_uio`` module
+.. note::
+
+   It is recommended that ``vfio-pci`` be used as the kernel module for DPDK-bound ports in all cases.
+   If an IOMMU is unavailable, the ``vfio-pci`` can be used in :ref:`no-iommu<vfio_noiommu>` mode.
+   If, for some reason, vfio is unavailable, then UIO-based modules, ``igb_uio`` and ``uio_pci_generic`` may be used.
+   See section :ref:`uio` for details.
+
+Most devices require that the hardware to be used by DPDK be unbound from the kernel driver it uses,
+and instead be bound to the ``vfio-pci`` kernel module before the application is run.
+For such PMDs, any network ports or other hardware under Linux* control will be ignored and cannot be used by the application.
+
+To bind ports to the ``vfio-pci`` module
 for DPDK use, or to return ports to Linux control,
 a utility script called ``dpdk-devbind.py`` is provided in the ``usertools`` subdirectory.
 This utility can be used to provide a view of the current state of the network ports on the system,
@@ -72,37 +76,38 @@ To see the status of all network ports on the system:

     Network devices using DPDK-compatible driver
     ============================================
-    0000:82:00.0 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe
-    0000:82:00.1 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe
+    0000:82:00.0 '82599EB 10-GbE NIC' drv=vfio-pci unused=ixgbe
+    0000:82:00.1 '82599EB 10-GbE NIC' drv=vfio-pci unused=ixgbe

     Network devices using kernel driver
     ===================================
-    0000:04:00.0 'I350 1-GbE NIC' if=em0  drv=igb unused=uio_pci_generic *Active*
-    0000:04:00.1 'I350 1-GbE NIC' if=eth1 drv=igb unused=uio_pci_generic
-    0000:04:00.2 'I350 1-GbE NIC' if=eth2 drv=igb unused=uio_pci_generic
-    0000:04:00.3 'I350 1-GbE NIC' if=eth3 drv=igb unused=uio_pci_generic
+    0000:04:00.0 'I350 1-GbE NIC' if=em0  drv=igb unused=vfio-pci *Active*
+    0000:04:00.1 'I350 1-GbE NIC' if=eth1 drv=igb unused=vfio-pci
+    0000:04:00.2 'I350 1-GbE NIC' if=eth2 drv=igb unused=vfio-pci
+    0000:04:00.3 'I350 1-GbE NIC' if=eth3 drv=igb unused=vfio-pci

     Other network devices
     =====================
     <none>

-To bind device ``eth1``,``04:00.1``, to the ``uio_pci_generic`` driver:
+To bind device ``eth1``,``04:00.1``, to the ``vfio-pci`` driver:

 .. code-block:: console

-    ./usertools/dpdk-devbind.py --bind=uio_pci_generic 04:00.1
+    ./usertools/dpdk-devbind.py --bind=vfio-pci 04:00.1

 or, alternatively,

 .. code-block:: console

-    ./usertools/dpdk-devbind.py --bind=uio_pci_generic eth1
+    ./usertools/dpdk-devbind.py --bind=vfio-pci eth1

-To restore device ``82:00.0`` to its original kernel binding:
+When specifying device ids, wildcards can be used for the final part of the address.
+To restore device ``82:00.0`` and ``82:00.1`` to their original kernel binding:

 .. code-block:: console

-    ./usertools/dpdk-devbind.py --bind=ixgbe 82:00.0
+    ./usertools/dpdk-devbind.py --bind=ixgbe 82:00.*

 VFIO
 ----
@@ -210,6 +215,8 @@ to use IO virtualization (such as Intel\ |reg| VT-d).
 For proper operation of VFIO when running DPDK applications as a non-privileged user, correct permissions should also be set up.
 For more information, please refer to :ref:`Running_Without_Root_Privileges`.

+.. _vfio_noiommu:
+
 VFIO no-IOMMU mode
 ------------------

@@ -240,6 +247,8 @@ After that, VFIO can be used with hardware devices as usual.
    to keep the degree of device access and programming that VFIO has,
    in situations where IOMMU is not available.

+.. _uio:
+
 UIO
 ---

--
2.32.0

[PATCH v2 6/9] doc: split GSG VFIO section into subsections

[Bruce Richardson]

The VFIO section of the page about linux drivers was rather long and
unstructured. This can be improved by splitting it up into subsections,
to cover the specifics of memory limits and creating VFs. When moving
the various text notes into the relevant subsections, we can drop the
note about kernels earlier than 3.6, since DPDK no longer supports
kernels that old.

Cc: stable@dpdk.org

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/linux_gsg/linux_drivers.rst | 35 ++++++++++++++------------
 1 file changed, 19 insertions(+), 16 deletions(-)

diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
index bd649db929..f9c24e9456 100644
--- a/doc/guides/linux_gsg/linux_drivers.rst
+++ b/doc/guides/linux_gsg/linux_drivers.rst
@@ -122,6 +122,22 @@ To make use of VFIO, the ``vfio-pci`` module must be loaded:
 VFIO kernel is usually present by default in all distributions,
 however please consult your distributions documentation to make sure that is the case.

+To make use of full VFIO functionality,
+both kernel and BIOS must support and be configured
+to use IO virtualization (such as Intel\ |reg| VT-d).
+
+.. note::
+
+   In most cases, specifying "iommu=on" as kernel parameter should be enough to
+   configure the Linux kernel to use IOMMU.
+
+For proper operation of VFIO when running DPDK applications as a non-privileged user, correct permissions should also be set up.
+For more information, please refer to :ref:`Running_Without_Root_Privileges`.
+
+
+VFIO Memory Mapping Limits
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 For DMA mapping of either external memory or hugepages, VFIO interface is used.
 VFIO does not support partial unmap of once mapped memory. Hence DPDK's memory is
 mapped in hugepage granularity or system page granularity. Number of DMA
@@ -132,6 +148,9 @@ VFIO module parameter ``dma_entry_limit`` with a default value of 64K.
 When application is out of DMA entries, these limits need to be adjusted to
 increase the allowed limit.

+Creating Virtual Functions using vfio-pci
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 Since Linux version 5.7,
 the ``vfio-pci`` module supports the creation of virtual functions.
 After the PF is bound to ``vfio-pci`` module,
@@ -194,27 +213,11 @@ The token will be used for all PF and VF ports within the application.
       <build_dir>/app/dpdk-testpmd -l 26-29 -n 4 -a 86:02.0 \
       --vfio-vf-token=14d63f20-8445-11ea-8900-1f9ce7d5650d --file-prefix=vf0 -- -i

-To make use of full VFIO functionality,
-both kernel and BIOS must support and be configured
-to use IO virtualization (such as Intel\ |reg| VT-d).
-
-.. note::
-
-   Linux versions earlier than version 3.6 do not support VFIO.
-
 .. note::

    Linux versions earlier than version 5.7 do not support the creation of
    virtual functions within the VFIO framework.

-.. note::
-
-   In most cases, specifying "iommu=on" as kernel parameter should be enough to
-   configure the Linux kernel to use IOMMU.
-
-For proper operation of VFIO when running DPDK applications as a non-privileged user, correct permissions should also be set up.
-For more information, please refer to :ref:`Running_Without_Root_Privileges`.
-
 .. _vfio_noiommu:

 VFIO no-IOMMU mode
--
2.32.0

[PATCH v2 7/9] doc: move GSG section on UIO to the end of drivers page

[Bruce Richardson]

To further de-emphasise UIO over the alternatives, we can move the UIO
section of the drivers page to the end of the document, giving more
prominence to VFIO and bifurcated drivers.

Cc: stable@dpdk.org

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/linux_gsg/linux_drivers.rst | 104 ++++++++++++-------------
 1 file changed, 52 insertions(+), 52 deletions(-)

diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
index f9c24e9456..6ff8586940 100644
--- a/doc/guides/linux_gsg/linux_drivers.rst
+++ b/doc/guides/linux_gsg/linux_drivers.rst
@@ -250,58 +250,6 @@ After that, VFIO can be used with hardware devices as usual.
    to keep the degree of device access and programming that VFIO has,
    in situations where IOMMU is not available.

-.. _uio:
-
-UIO
----
-
-.. warning::
-
-   Using UIO drivers is inherently unsafe due to this method lacking IOMMU protection,
-   and can only be done by root user.
-
-In situations where using VFIO is not an option, there are alternative drivers one can use.
-In many cases, the standard ``uio_pci_generic`` module included in the Linux kernel
-can be used as a substitute for VFIO. This module can be loaded using the command:
-
-.. code-block:: console
-
-   sudo modprobe uio_pci_generic
-
-.. note::
-
-   ``uio_pci_generic`` module doesn't support the creation of virtual functions.
-
-As an alternative to the ``uio_pci_generic``, there is the ``igb_uio`` module
-which can be found in the repository `dpdk-kmods <http://git.dpdk.org/dpdk-kmods>`_.
-It can be loaded as shown below:
-
-.. code-block:: console
-
-   sudo modprobe uio
-   sudo insmod igb_uio.ko
-
-.. note::
-
-    For some devices which lack support for legacy interrupts, e.g. virtual function
-    (VF) devices, the ``igb_uio`` module may be needed in place of ``uio_pci_generic``.
-
-.. note::
-
-   If UEFI secure boot is enabled,
-   the Linux kernel may disallow the use of UIO on the system.
-   Therefore, devices for use by DPDK should be bound to the ``vfio-pci`` kernel module
-   rather than any UIO-based module.
-   For more details see :ref:`linux_gsg_binding_kernel` below.
-
-.. note::
-
-   If the devices used for DPDK are bound to a UIO-based kernel module,
-   please make sure that the IOMMU is disabled or is in passthrough mode.
-   One can add ``intel_iommu=off`` or ``amd_iommu=off`` or ``intel_iommu=on iommu=pt``
-   in GRUB command line on x86_64 systems,
-   or add ``iommu.passthrough=1`` on aarch64 systems.
-
 .. _bifurcated_driver:

 Bifurcated Driver
@@ -372,3 +320,55 @@ This can be checked in the boot configuration of your system:
 If ``CONFIG_VFIO_NOIOMMU`` is not enabled in the kernel configuration,
 VFIO driver will not support the no-IOMMU mode,
 and other alternatives (such as UIO drivers) will have to be used.
+
+.. _uio:
+
+UIO
+---
+
+.. warning::
+
+   Using UIO drivers is inherently unsafe due to this method lacking IOMMU protection,
+   and can only be done by root user.
+
+In situations where using VFIO is not an option, there are alternative drivers one can use.
+In many cases, the standard ``uio_pci_generic`` module included in the Linux kernel
+can be used as a substitute for VFIO. This module can be loaded using the command:
+
+.. code-block:: console
+
+   sudo modprobe uio_pci_generic
+
+.. note::
+
+   ``uio_pci_generic`` module doesn't support the creation of virtual functions.
+
+As an alternative to the ``uio_pci_generic``, there is the ``igb_uio`` module
+which can be found in the repository `dpdk-kmods <http://git.dpdk.org/dpdk-kmods>`_.
+It can be loaded as shown below:
+
+.. code-block:: console
+
+   sudo modprobe uio
+   sudo insmod igb_uio.ko
+
+.. note::
+
+    For some devices which lack support for legacy interrupts, e.g. virtual function
+    (VF) devices, the ``igb_uio`` module may be needed in place of ``uio_pci_generic``.
+
+.. note::
+
+   If UEFI secure boot is enabled,
+   the Linux kernel may disallow the use of UIO on the system.
+   Therefore, devices for use by DPDK should be bound to the ``vfio-pci`` kernel module
+   rather than any UIO-based module.
+   For more details see :ref:`linux_gsg_binding_kernel` below.
+
+.. note::
+
+   If the devices used for DPDK are bound to a UIO-based kernel module,
+   please make sure that the IOMMU is disabled or is in passthrough mode.
+   One can add ``intel_iommu=off`` or ``amd_iommu=off`` or ``intel_iommu=on iommu=pt``
+   in GRUB command line on x86_64 systems,
+   or add ``iommu.passthrough=1`` on aarch64 systems.
--
2.32.0

[PATCH v2 8/9] doc: consolidate all VFIO content on GSG driver page

[Bruce Richardson]

Rather than having separate sections for VFIO and VFIO no-iommu mode, as
well as a separate section further down the document on troubleshooting
VFIO, we can consolidate all these as subsections into a primary VFIO
section. This section starts with the basics of VFIO use, then covers
no-iommu mode, before moving on to the more advanced topics such as
creating VFs and ending with the troubleshooting subsection.

Cc: stable@dpdk.org

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/linux_gsg/linux_drivers.rst | 116 ++++++++++++-------------
 1 file changed, 58 insertions(+), 58 deletions(-)

diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
index 6ff8586940..8320db44d9 100644
--- a/doc/guides/linux_gsg/linux_drivers.rst
+++ b/doc/guides/linux_gsg/linux_drivers.rst
@@ -135,6 +135,38 @@ For proper operation of VFIO when running DPDK applications as a non-privileged
 For more information, please refer to :ref:`Running_Without_Root_Privileges`.


+.. _vfio_noiommu:
+
+VFIO no-IOMMU mode
+~~~~~~~~~~~~~~~~~~
+
+If there is no IOMMU available on the system, VFIO can still be used,
+but it has to be loaded with an additional module parameter:
+
+.. code-block:: console
+
+   modprobe vfio enable_unsafe_noiommu_mode=1
+
+Alternatively, one can also enable this option in an already loaded kernel module:
+
+.. code-block:: console
+
+   echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
+
+After that, VFIO can be used with hardware devices as usual.
+
+.. note::
+
+   It may be required to unload all VFIO related-modules before probing
+   the module again with ``enable_unsafe_noiommu_mode=1`` parameter.
+
+.. warning::
+
+   Since no-IOMMU mode forgoes IOMMU protection, it is inherently unsafe.
+   That said, it does make it possible for the user
+   to keep the degree of device access and programming that VFIO has,
+   in situations where IOMMU is not available.
+
 VFIO Memory Mapping Limits
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~

@@ -218,65 +250,8 @@ The token will be used for all PF and VF ports within the application.
    Linux versions earlier than version 5.7 do not support the creation of
    virtual functions within the VFIO framework.

-.. _vfio_noiommu:
-
-VFIO no-IOMMU mode
-------------------
-
-If there is no IOMMU available on the system, VFIO can still be used,
-but it has to be loaded with an additional module parameter:
-
-.. code-block:: console
-
-   modprobe vfio enable_unsafe_noiommu_mode=1
-
-Alternatively, one can also enable this option in an already loaded kernel module:
-
-.. code-block:: console
-
-   echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
-
-After that, VFIO can be used with hardware devices as usual.
-
-.. note::
-
-   It may be required to unload all VFIO related-modules before probing
-   the module again with ``enable_unsafe_noiommu_mode=1`` parameter.
-
-.. warning::
-
-   Since no-IOMMU mode forgoes IOMMU protection, it is inherently unsafe.
-   That said, it does make it possible for the user
-   to keep the degree of device access and programming that VFIO has,
-   in situations where IOMMU is not available.
-
-.. _bifurcated_driver:
-
-Bifurcated Driver
------------------
-
-PMDs which use the bifurcated driver co-exists with the device kernel driver.
-On such model the NIC is controlled by the kernel, while the data
-path is performed by the PMD directly on top of the device.
-
-Such model has the following benefits:
-
- - It is secure and robust, as the memory management and isolation
-   is done by the kernel.
- - It enables the user to use legacy linux tools such as ``ethtool`` or
-   ``ifconfig`` while running DPDK application on the same network ports.
- - It enables the DPDK application to filter only part of the traffic,
-   while the rest will be directed and handled by the kernel driver.
-   The flow bifurcation is performed by the NIC hardware.
-   As an example, using :ref:`flow_isolated_mode` allows to choose
-   strictly what is received in DPDK.
-
-More about the bifurcated driver can be found in
-`Mellanox Bifurcated DPDK PMD
-<https://www.dpdk.org/wp-content/uploads/sites/35/2016/10/Day02-Session04-RonyEfraim-Userspace2016.pdf>`__.
-
 Troubleshooting VFIO
---------------------
+~~~~~~~~~~~~~~~~~~~~

 In certain situations, using ``dpdk-devbind.py`` script
 to bind a device to VFIO driver may fail.
@@ -321,6 +296,31 @@ If ``CONFIG_VFIO_NOIOMMU`` is not enabled in the kernel configuration,
 VFIO driver will not support the no-IOMMU mode,
 and other alternatives (such as UIO drivers) will have to be used.

+.. _bifurcated_driver:
+
+Bifurcated Driver
+-----------------
+
+PMDs which use the bifurcated driver co-exists with the device kernel driver.
+On such model the NIC is controlled by the kernel, while the data
+path is performed by the PMD directly on top of the device.
+
+Such model has the following benefits:
+
+ - It is secure and robust, as the memory management and isolation
+   is done by the kernel.
+ - It enables the user to use legacy linux tools such as ``ethtool`` or
+   ``ifconfig`` while running DPDK application on the same network ports.
+ - It enables the DPDK application to filter only part of the traffic,
+   while the rest will be directed and handled by the kernel driver.
+   The flow bifurcation is performed by the NIC hardware.
+   As an example, using :ref:`flow_isolated_mode` allows to choose
+   strictly what is received in DPDK.
+
+More about the bifurcated driver can be found in
+`Mellanox Bifurcated DPDK PMD
+<https://www.dpdk.org/wp-content/uploads/sites/35/2016/10/Day02-Session04-RonyEfraim-Userspace2016.pdf>`__.
+
 .. _uio:

 UIO
--
2.32.0

[PATCH v2 9/9] doc: change informational warnings to notes in Linux GSG

[Bruce Richardson]

There are two warnings in the VFIO section about limitations of VFIO and
limitations on who can bind/unbind devices. Since these don't actually
describe any unsafe conditions, and are more informational, we can
change these to notes. This also helps emphasise the other warnings in
the documents which flag genuine security concerns.

Cc: stable@dpdk.org

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/linux_gsg/linux_drivers.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
index 8320db44d9..2e4c80ebd3 100644
--- a/doc/guides/linux_gsg/linux_drivers.rst
+++ b/doc/guides/linux_gsg/linux_drivers.rst
@@ -48,7 +48,7 @@ by calling the script with the ``--help`` or ``--usage`` options.
 Note that the UIO or VFIO kernel modules to be used,
 should be loaded into the kernel before running the ``dpdk-devbind.py`` script.

-.. warning::
+.. note::

    Due to the way VFIO works, there are certain limitations
    to which devices can be used with VFIO.
@@ -62,7 +62,7 @@ should be loaded into the kernel before running the ``dpdk-devbind.py`` script.
    Therefore, the bridge driver should also be unbound from the bridge PCI device
    for VFIO to work with devices behind the bridge.

-.. warning::
+.. note::

    While any user can run the ``dpdk-devbind.py`` script
    to view the status of the network ports,
--
2.32.0