[PATCH 0/4] Improve documentation for running as non-root

[PATCH 0/4] Improve documentation for running as non-root

[Dmitry Kozlyuk]

Dmitry Kozlyuk (4):
  usertools: add option to select hugetlbfs directory
  usertools: add option to change mount point owner
  doc: give specific instructions for running as non-root
  doc: update instructions for running as non-root for MLX5

 doc/guides/linux_gsg/enable_func.rst          | 53 ++++++++++++++++---
 doc/guides/platform/mlx5.rst                  | 22 ++++----
 .../prog_guide/env_abstraction_layer.rst      |  2 +
 usertools/dpdk-hugepages.py                   | 19 +++++--
 4 files changed, 75 insertions(+), 21 deletions(-)

-- 
2.25.1

[PATCH 1/4] usertools: add option to select hugetlbfs directory

[Dmitry Kozlyuk]

dpdk-hugepages.py had /dev/hugepages hardcoded as the mount point.
It may be desirable to setup hugepage directory at another path,
for example, when using hugepages of multiple sizes in different
directories or when granting different permissions to mount points.
Add --directory/-d option to the script.

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
---
 usertools/dpdk-hugepages.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/usertools/dpdk-hugepages.py b/usertools/dpdk-hugepages.py
index 4fdb199744..8bab086a2f 100755
--- a/usertools/dpdk-hugepages.py
+++ b/usertools/dpdk-hugepages.py
@@ -228,6 +228,12 @@ def main():
         '-u',
         action='store_true',
         help='unmount the system huge page directory')
+    parser.add_argument(
+        '--directory',
+        '-d',
+        metavar='DIR',
+        default=HUGE_MOUNT,
+        help='mount point')
     parser.add_argument(
         '--node', '-n', help='select numa node to reserve pages on')
     parser.add_argument(
@@ -262,7 +268,7 @@ def main():
     if args.clear:
         clear_pages()
     if args.unmount:
-        umount_huge(HUGE_MOUNT)
+        umount_huge(args.directory)
 
     if args.reserve:
         reserve_kb = get_memsize(args.reserve)
@@ -273,7 +279,7 @@ def main():
         reserve_pages(
             int(reserve_kb / pagesize_kb), pagesize_kb, node=args.node)
     if args.mount:
-        mount_huge(pagesize_kb, HUGE_MOUNT)
+        mount_huge(pagesize_kb, args.directory)
     if args.show:
         show_pages()
         print()
-- 
2.25.1

[PATCH 2/4] usertools: add option to change mount point owner

[Dmitry Kozlyuk]

Per mount(8), the previous owner and mode of the mount point
become invisible as long as this filesystem remains mounted.
Because dpdk-hugepages.py must be run as root,
the new owner would be root.
This is undesirable if the hugepage directory is being set up
by the administrator for an unprivileged user.
Add --owner/-o option to set the owner after mounting.
The benefit over calling chown directly after the script
is that the user does not need to care about this detail
of mount command operation.

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
---
 usertools/dpdk-hugepages.py | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/usertools/dpdk-hugepages.py b/usertools/dpdk-hugepages.py
index 8bab086a2f..1e01ac3d9f 100755
--- a/usertools/dpdk-hugepages.py
+++ b/usertools/dpdk-hugepages.py
@@ -170,7 +170,7 @@ def get_mountpoints():
     return mounted
 
 
-def mount_huge(pagesize, mountpoint):
+def mount_huge(pagesize, mountpoint, owner):
     '''Mount the huge TLB file system'''
     if mountpoint in get_mountpoints():
         print(mountpoint, "already mounted")
@@ -180,6 +180,8 @@ def mount_huge(pagesize, mountpoint):
         cmd += ' -o pagesize={}'.format(pagesize * 1024)
     cmd += ' nodev ' + mountpoint
     os.system(cmd)
+    if owner:
+        os.system('chown {} {}'.format(owner, mountpoint))
 
 
 def umount_huge(mountpoint):
@@ -234,6 +236,11 @@ def main():
         metavar='DIR',
         default=HUGE_MOUNT,
         help='mount point')
+    parser.add_argument(
+        '--owner',
+        '-o',
+        metavar='USER:GROUP',
+        help='change the mounted directory owner')
     parser.add_argument(
         '--node', '-n', help='select numa node to reserve pages on')
     parser.add_argument(
@@ -279,7 +286,7 @@ def main():
         reserve_pages(
             int(reserve_kb / pagesize_kb), pagesize_kb, node=args.node)
     if args.mount:
-        mount_huge(pagesize_kb, args.directory)
+        mount_huge(pagesize_kb, args.directory, args.owner)
     if args.show:
         show_pages()
         print()
-- 
2.25.1

Re: [PATCH 2/4] usertools: add option to change mount point owner

[Stephen Hemminger]

On Wed, 8 Jun 2022 02:49:47 +0300
Dmitry Kozlyuk <dkozlyuk@nvidia.com> wrote:

>  
> -def mount_huge(pagesize, mountpoint):
> +def mount_huge(pagesize, mountpoint, owner):
>      '''Mount the huge TLB file system'''
>      if mountpoint in get_mountpoints():
>          print(mountpoint, "already mounted")
> @@ -180,6 +180,8 @@ def mount_huge(pagesize, mountpoint):
>          cmd += ' -o pagesize={}'.format(pagesize * 1024)
>      cmd += ' nodev ' + mountpoint
>      os.system(cmd)
> +    if owner:
> +        os.system('chown {} {}'.format(owner, mountpoint))
>  

The hugetlb filesystem type already supports set owner.
Please use that instead of an additional chown command.

https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt



If the user applications are going to request huge pages using mmap system
call, then it is required that system administrator mount a file system of
type hugetlbfs:

  mount -t hugetlbfs \
	-o uid=<value>,gid=<value>,mode=<value>,pagesize=<value>,size=<value>,\
	min_size=<value>,nr_inodes=<value> none /mnt/huge

[PATCH 3/4] doc: give specific instructions for running as non-root

[Dmitry Kozlyuk]

The guide to run DPDK applications as non-root in Linux
did not provide specific instructions to configure the required access
and did not explain why each bit is needed.
The latter is important because running as non-root
is one of the ways to tighten security and grant minimal permissions.

Cc: stable@dpdk.org

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
---
 doc/guides/linux_gsg/enable_func.rst          | 53 ++++++++++++++++---
 .../prog_guide/env_abstraction_layer.rst      |  2 +
 2 files changed, 49 insertions(+), 6 deletions(-)

diff --git a/doc/guides/linux_gsg/enable_func.rst b/doc/guides/linux_gsg/enable_func.rst
index 1df3ab0255..c6975ce8bf 100644
--- a/doc/guides/linux_gsg/enable_func.rst
+++ b/doc/guides/linux_gsg/enable_func.rst
@@ -13,13 +13,46 @@ Enabling Additional Functionality
 Running DPDK Applications Without Root Privileges
 -------------------------------------------------
 
-In order to run DPDK as non-root, the following Linux filesystem objects'
-permissions should be adjusted to ensure that the Linux account being used to
-run the DPDK application has access to them:
+The following sections describe generic requirements and configuration
+for running DPDK applications as non-root.
+There may be additional requirements documented for some drivers.
 
-*   All directories which serve as hugepage mount points, for example, ``/dev/hugepages``
+Hugepages
+~~~~~~~~~
 
-*   If the HPET is to be used,  ``/dev/hpet``
+Hugepages must be reserved as root before runing the application as non-root,
+for example::
+
+  sudo dpdk-hugepages.py --reserve 1G
+
+If multi-process is not required, running with ``--in-memory``
+bypasses the need to access hugepage mount point and files within it.
+Otherwise, hugepage directory must be made accessible
+for writing to the unprivileged user, for example::
+
+  export HUGEDIR=$HOME/huge-1G
+  mkdir -p $HUGEDIR
+  sudo dpdk-hugepages.py --mount --directory $HUGEDIR --owner `id -u`:`id -g`
+
+If the driver requires using physical addresses (PA),
+the executable file must be granted additional capabilities:
+
+* ``SYS_ADMIN`` to read ``/proc/self/pagemaps``
+* ``IPC_LOCK`` to lock hugepages in memory
+
+.. code-block:: console
+
+   setcap cap_ipc_lock,cap_sys_admin+ep <executable>
+
+If physical addresses are not accessible,
+the following message will appear during EAL initialization::
+
+  EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
+
+It is harmless in case PA are not needed.
+
+Resource Limits
+~~~~~~~~~~~~~~~
 
 When running as non-root user, there may be some additional resource limits
 that are imposed by the system. Specifically, the following resource limits may
@@ -34,7 +67,15 @@ need to be adjusted in order to ensure normal DPDK operation:
 The above limits can usually be adjusted by editing
 ``/etc/security/limits.conf`` file, and rebooting.
 
-Additionally, depending on which kernel driver is in use, the relevant
+See `Hugepage Mapping <hugepage_mapping>`_
+secton to learn how these limits affect EAL.
+
+Device Control
+~~~~~~~~~~~~~~
+
+If the HPET is to be used, ``/dev/hpet`` permissions must be adjusted.
+
+Depending on which kernel driver is in use, the relevant
 resources also should be accessible by the user running the DPDK application.
 
 For ``vfio-pci`` kernel driver, the following Linux file system objects'
diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 5f0748fba1..70fa099d30 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -228,6 +228,8 @@ Normally, these options do not need to be changed.
     can later be mapped into that preallocated VA space (if dynamic memory mode
     is enabled), and can optionally be mapped into it at startup.
 
+.. _hugepage_mapping:
+
 Hugepage Mapping
 ^^^^^^^^^^^^^^^^
 
-- 
2.25.1

Re: [PATCH 3/4] doc: give specific instructions for running as non-root

[Stephen Hemminger]

On Wed, 8 Jun 2022 02:49:48 +0300
Dmitry Kozlyuk <dkozlyuk@nvidia.com> wrote:

> The guide to run DPDK applications as non-root in Linux
> did not provide specific instructions to configure the required access
> and did not explain why each bit is needed.
> The latter is important because running as non-root
> is one of the ways to tighten security and grant minimal permissions.
> 
> Cc: stable@dpdk.org
> 
> Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>

If running with multiple containers it is often better to have OS
take care of mounting huge pages.

https://github.com/systemd/systemd/blob/main/units/dev-hugepages.mount

And a good way for managing multiple applications using hugepages
is to mount device with group permissions and add supplementary
group to each container.

[PATCH 4/4] doc: update instructions for running as non-root for MLX5

[Dmitry Kozlyuk]

Reference the common guide for generic setup.
Remove excessive capabilities from the recommended list.

Cc: stable@dpdk.org

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
---
 doc/guides/platform/mlx5.rst | 22 ++++++++++------------
 1 file changed, 10 insertions(+), 12 deletions(-)

diff --git a/doc/guides/platform/mlx5.rst b/doc/guides/platform/mlx5.rst
index 64a4c5e76e..956a72fadf 100644
--- a/doc/guides/platform/mlx5.rst
+++ b/doc/guides/platform/mlx5.rst
@@ -404,25 +404,23 @@ The device can be bound again at this point.
 Run as Non-Root
 ^^^^^^^^^^^^^^^
 
-In order to run as a non-root user,
-some capabilities must be granted to the application::
+Hugepage and resource limit setup is documented
+in the :ref:`common Linux guide <Running_Without_Root_Privileges>`.
+This PMD does not require physical addresses,
+so capability configuration is not needed to access hugepages.
+Note that physical addresses may be required by other drivers.
 
-   setcap cap_sys_admin,cap_net_admin,cap_net_raw,cap_ipc_lock+ep <dpdk-app>
+Additional capabilities must be granted to the application::
 
-Below are the reasons for the need of each capability:
-
-``cap_sys_admin``
-   When using physical addresses (PA mode), with Linux >= 4.0,
-   for access to ``/proc/self/pagemap``.
+   setcap cap_net_raw,cap_net_admin,cap_sys_rawio+ep <executable>
 
-``cap_net_admin``
-   For device configuration.
+Below are the reasons for the need of each capability:
 
 ``cap_net_raw``
    For raw ethernet queue allocation through kernel driver.
 
-``cap_ipc_lock``
-   For DMA memory pinning.
+``cap_net_admin``
+   For device configuration, like setting link status or MTU.
 
 
 Windows Environment
-- 
2.25.1

Re: [PATCH 4/4] doc: update instructions for running as non-root for MLX5

[Stephen Hemminger]

On Wed, 8 Jun 2022 02:49:49 +0300
Dmitry Kozlyuk <dkozlyuk@nvidia.com> wrote:

> Reference the common guide for generic setup.
> Remove excessive capabilities from the recommended list.
> 
> Cc: stable@dpdk.org
> 
> Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
> ---
>  doc/guides/platform/mlx5.rst | 22 ++++++++++------------
>  1 file changed, 10 insertions(+), 12 deletions(-)


This change needs additional changes to make it correct English grammar.

> diff --git a/doc/guides/platform/mlx5.rst b/doc/guides/platform/mlx5.rst
> index 64a4c5e76e..956a72fadf 100644
> --- a/doc/guides/platform/mlx5.rst
> +++ b/doc/guides/platform/mlx5.rst
> @@ -404,25 +404,23 @@ The device can be bound again at this point.
>  Run as Non-Root
>  ^^^^^^^^^^^^^^^
>  
> -In order to run as a non-root user,
> -some capabilities must be granted to the application::
> +Hugepage and resource limit setup is documented

Subject is plural so verb must be plural => are documented

> +in the :ref:`common Linux guide <Running_Without_Root_Privileges>`.
> +This PMD does not require physical addresses,
> +so capability configuration is not needed to access hugepages.

In technical writing "therefore" is preferred over "so"
and you need a preposition. Please reword something like:

"This PMD does can operate without direct physical memory and hugepages
are not required."

Often applications will keep using hugepages (makes them NIC independent)
and in that case they would still need permissions.

> +Note that physical addresses may be required by other drivers.
>  
> -   setcap cap_sys_admin,cap_net_admin,cap_net_raw,cap_ipc_lock+ep <dpdk-app>
> +Additional capabilities must be granted to the application::
>  
> -Below are the reasons for the need of each capability:
> -
> -``cap_sys_admin``
> -   When using physical addresses (PA mode), with Linux >= 4.0,
> -   for access to ``/proc/self/pagemap``.
> +   setcap cap_net_raw,cap_net_admin,cap_sys_rawio+ep <executable>
>  
> -``cap_net_admin``
> -   For device configuration.
> +Below are the reasons for the need of each capability:
>  
>  ``cap_net_raw``
>     For raw ethernet queue allocation through kernel driver.
>  
> -``cap_ipc_lock``
> -   For DMA memory pinning.
> +``cap_net_admin``
> +   For device configuration, like setting link status or MTU.
>  

The most common usage for running as non-root is some container system.
In that case capabilities are managed by the container service (ie systemd, docker, etc)
and not done by setting filesystem capabilities.

RE: [PATCH 4/4] doc: update instructions for running as non-root for MLX5

[Dmitry Kozlyuk]

> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Wednesday, June 8, 2022 3:14 AM
> [...]
> This change needs additional changes to make it correct English
> grammar.

Thank you for the useful comments to this and other patches.
I hope you don't mind that I took some sentences almost verbatim for v2.

> [...]
> > +This PMD does not require physical addresses,
> > +so capability configuration is not needed to access hugepages.
> 
> In technical writing "therefore" is preferred over "so"
> and you need a preposition. Please reword something like:
> 
> "This PMD does can operate without direct physical memory and
> hugepages
> are not required."
> 
> Often applications will keep using hugepages (makes them NIC
> independent)
> and in that case they would still need permissions.

MLX5 PMD uses hugepages but does not use physical addresses
unlike most other HW drivers.
I tried to make it more clear in v2.

[PATCH v2 0/4] Improve documentation for running as non-root

[Dmitry Kozlyuk]

v1: Address comments by Stephen Hemminger:
    - Use hugetlbfs options instead of chown.
    - Suggest using OS or container runtime where possible.
    - Improve wording.
    Add SYS_RAWIO requirement for legacy virtio.
    Explain SYS_RAWIO requirement for MLX5 PMD.

Dmitry Kozlyuk (4):
  usertools: add option to select hugetlbfs directory
  usertools: add option to change mount point owner
  doc: give specific instructions for running as non-root
  doc: update instructions for running as non-root for MLX5

 doc/guides/linux_gsg/enable_func.rst          | 67 +++++++++++++++++--
 doc/guides/platform/mlx5.rst                  | 31 +++++----
 .../prog_guide/env_abstraction_layer.rst      |  2 +
 usertools/dpdk-hugepages.py                   | 20 +++++-
 4 files changed, 98 insertions(+), 22 deletions(-)

-- 
2.25.1

[PATCH v2 1/4] usertools: add option to select hugetlbfs directory

[Dmitry Kozlyuk]

dpdk-hugepages.py had /dev/hugepages hardcoded as the mount point.
It may be desirable to setup hugepage directory at another path,
for example, when using hugepages of multiple sizes in different
directories or when granting different permissions to mount points.
Add --directory/-d option to the script.

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
---
 usertools/dpdk-hugepages.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/usertools/dpdk-hugepages.py b/usertools/dpdk-hugepages.py
index 4fdb199744..8bab086a2f 100755
--- a/usertools/dpdk-hugepages.py
+++ b/usertools/dpdk-hugepages.py
@@ -228,6 +228,12 @@ def main():
         '-u',
         action='store_true',
         help='unmount the system huge page directory')
+    parser.add_argument(
+        '--directory',
+        '-d',
+        metavar='DIR',
+        default=HUGE_MOUNT,
+        help='mount point')
     parser.add_argument(
         '--node', '-n', help='select numa node to reserve pages on')
     parser.add_argument(
@@ -262,7 +268,7 @@ def main():
     if args.clear:
         clear_pages()
     if args.unmount:
-        umount_huge(HUGE_MOUNT)
+        umount_huge(args.directory)
 
     if args.reserve:
         reserve_kb = get_memsize(args.reserve)
@@ -273,7 +279,7 @@ def main():
         reserve_pages(
             int(reserve_kb / pagesize_kb), pagesize_kb, node=args.node)
     if args.mount:
-        mount_huge(pagesize_kb, HUGE_MOUNT)
+        mount_huge(pagesize_kb, args.directory)
     if args.show:
         show_pages()
         print()
-- 
2.25.1

Re: [PATCH v2 1/4] usertools: add option to select hugetlbfs directory

[Bruce Richardson]

On Fri, Jun 17, 2022 at 02:25:05PM +0300, Dmitry Kozlyuk wrote:
> dpdk-hugepages.py had /dev/hugepages hardcoded as the mount point.
> It may be desirable to setup hugepage directory at another path,
> for example, when using hugepages of multiple sizes in different
> directories or when granting different permissions to mount points.
> Add --directory/-d option to the script.
> 
> Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>

Acked-by: Bruce Richardson <bruce.richardson@intel.com>

[PATCH v2 2/4] usertools: add option to change mount point owner

[Dmitry Kozlyuk]

Per mount(8), the previous owner and mode of the mount point
become invisible as long as this filesystem remains mounted.
Because dpdk-hugepages.py must be run as root,
the new owner would be root.
This is undesirable if the hugepage directory is being set up
by the administrator for an unprivileged user.
HugeTLB filesystem has options to set the mount point owner.
Add --owner/-o option to apply this option when mounting.
The benefit of performing this in dpdk-hugepages.py
is that the user does not need to care about this detail
of mount command operation.

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
---
 usertools/dpdk-hugepages.py | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/usertools/dpdk-hugepages.py b/usertools/dpdk-hugepages.py
index 8bab086a2f..5120518bcb 100755
--- a/usertools/dpdk-hugepages.py
+++ b/usertools/dpdk-hugepages.py
@@ -170,7 +170,7 @@ def get_mountpoints():
     return mounted
 
 
-def mount_huge(pagesize, mountpoint):
+def mount_huge(pagesize, mountpoint, owner):
     '''Mount the huge TLB file system'''
     if mountpoint in get_mountpoints():
         print(mountpoint, "already mounted")
@@ -178,6 +178,9 @@ def mount_huge(pagesize, mountpoint):
     cmd = "mount -t hugetlbfs"
     if pagesize:
         cmd += ' -o pagesize={}'.format(pagesize * 1024)
+    if owner:
+        uid, gid = owner.split(':', maxsplit=1)
+        cmd += ' -o uid={},gid={}'.format(uid, gid)
     cmd += ' nodev ' + mountpoint
     os.system(cmd)
 
@@ -234,6 +237,11 @@ def main():
         metavar='DIR',
         default=HUGE_MOUNT,
         help='mount point')
+    parser.add_argument(
+        '--owner',
+        '-o',
+        metavar='USER:GROUP',
+        help='change the mounted directory owner')
     parser.add_argument(
         '--node', '-n', help='select numa node to reserve pages on')
     parser.add_argument(
@@ -279,7 +287,7 @@ def main():
         reserve_pages(
             int(reserve_kb / pagesize_kb), pagesize_kb, node=args.node)
     if args.mount:
-        mount_huge(pagesize_kb, args.directory)
+        mount_huge(pagesize_kb, args.directory, args.owner)
     if args.show:
         show_pages()
         print()
-- 
2.25.1

Re: [PATCH v2 2/4] usertools: add option to change mount point owner

[Bruce Richardson]

On Fri, Jun 17, 2022 at 02:25:06PM +0300, Dmitry Kozlyuk wrote:
> Per mount(8), the previous owner and mode of the mount point
> become invisible as long as this filesystem remains mounted.
> Because dpdk-hugepages.py must be run as root,
> the new owner would be root.
> This is undesirable if the hugepage directory is being set up
> by the administrator for an unprivileged user.
> HugeTLB filesystem has options to set the mount point owner.
> Add --owner/-o option to apply this option when mounting.
> The benefit of performing this in dpdk-hugepages.py
> is that the user does not need to care about this detail
> of mount command operation.
> 
> Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
> ---
>  usertools/dpdk-hugepages.py | 12 ++++++++++--
>  1 file changed, 10 insertions(+), 2 deletions(-)
> 
> diff --git a/usertools/dpdk-hugepages.py b/usertools/dpdk-hugepages.py
> index 8bab086a2f..5120518bcb 100755
> --- a/usertools/dpdk-hugepages.py
> +++ b/usertools/dpdk-hugepages.py
> @@ -170,7 +170,7 @@ def get_mountpoints():
>      return mounted
>  
>  
> -def mount_huge(pagesize, mountpoint):
> +def mount_huge(pagesize, mountpoint, owner):
>      '''Mount the huge TLB file system'''
>      if mountpoint in get_mountpoints():
>          print(mountpoint, "already mounted")
> @@ -178,6 +178,9 @@ def mount_huge(pagesize, mountpoint):
>      cmd = "mount -t hugetlbfs"
>      if pagesize:
>          cmd += ' -o pagesize={}'.format(pagesize * 1024)
> +    if owner:
> +        uid, gid = owner.split(':', maxsplit=1)
> +        cmd += ' -o uid={},gid={}'.format(uid, gid)

I'm not sure about forcing the user to always provide a "user:group" format
parameter. How about:

1. checking initially for the presence of a ":" and if not present just
   working with the uid parameter?
2. alternatively, what about adding in separate parameters for user or
   group, so they can be specified independently?

/Bruce

RE: [PATCH v2 2/4] usertools: add option to change mount point owner

[Dmitry Kozlyuk]

> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Friday, June 17, 2022 6:53 PM
> [...]
> > +    if owner:
> > +        uid, gid = owner.split(':', maxsplit=1)
> > +        cmd += ' -o uid={},gid={}'.format(uid, gid)
> 
> I'm not sure about forcing the user to always provide a "user:group"
> format parameter. How about:
> 
> 1. checking initially for the presence of a ":" and if not present
>    just working with the uid parameter?
> 2. alternatively, what about adding in separate parameters for user or
>    group, so they can be specified independently?

Explicit is better than implicit, especially in security configuration.
If you insist on changing, I prefer option 2
to clearly indicate what is configured and what is not.

[PATCH v2 3/4] doc: give specific instructions for running as non-root

[Dmitry Kozlyuk]

The guide to run DPDK applications as non-root in Linux
did not provide specific instructions to configure the required access
and did not explain why each bit is needed.
The latter is important because running as non-root
is one of the ways to tighten security and grant minimal permissions.

Cc: stable@dpdk.org

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
---
 doc/guides/linux_gsg/enable_func.rst          | 67 +++++++++++++++++--
 .../prog_guide/env_abstraction_layer.rst      |  2 +
 2 files changed, 63 insertions(+), 6 deletions(-)

diff --git a/doc/guides/linux_gsg/enable_func.rst b/doc/guides/linux_gsg/enable_func.rst
index 1df3ab0255..2f908e8b70 100644
--- a/doc/guides/linux_gsg/enable_func.rst
+++ b/doc/guides/linux_gsg/enable_func.rst
@@ -13,13 +13,58 @@ Enabling Additional Functionality
 Running DPDK Applications Without Root Privileges
 -------------------------------------------------
 
-In order to run DPDK as non-root, the following Linux filesystem objects'
-permissions should be adjusted to ensure that the Linux account being used to
-run the DPDK application has access to them:
+The following sections describe generic requirements and configuration
+for running DPDK applications as non-root.
+There may be additional requirements documented for some drivers.
 
-*   All directories which serve as hugepage mount points, for example, ``/dev/hugepages``
+Hugepages
+~~~~~~~~~
 
-*   If the HPET is to be used,  ``/dev/hpet``
+Hugepages must be reserved as root before runing the application as non-root,
+for example::
+
+  sudo dpdk-hugepages.py --reserve 1G
+
+If multi-process is not required, running with ``--in-memory``
+bypasses the need to access hugepage mount point and files within it.
+Otherwise, hugepage directory must be made accessible
+for writing to the unprivileged user.
+A good way for managing multiple applications using hugepages
+is to mount the filesystem with group permissions
+and add a supplementary group to each application or container.
+
+One option is to use the script provided by this project::
+
+  export HUGEDIR=$HOME/huge-1G
+  mkdir -p $HUGEDIR
+  sudo dpdk-hugepages.py --mount --directory $HUGEDIR --owner `id -u`:`id -g`
+
+In production environment, the OS can manage mount points
+(`systemd example <https://github.com/systemd/systemd/blob/main/units/dev-hugepages.mount>`_).
+
+The ``hugetlb`` filesystem has additional options to guarantee or limit
+the amount of memory that is possible to allocate using the mount point.
+Refer to the `documentation <https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt>`_.
+
+If the driver requires using physical addresses (PA),
+the executable file must be granted additional capabilities:
+
+* ``SYS_ADMIN`` to read ``/proc/self/pagemaps``
+* ``IPC_LOCK`` to lock hugepages in memory
+
+.. code-block:: console
+
+   setcap cap_ipc_lock,cap_sys_admin+ep <executable>
+
+If physical addresses are not accessible,
+the following message will appear during EAL initialization::
+
+  EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
+
+It is harmless in case PA are not needed.
+
+Resource Limits
+~~~~~~~~~~~~~~~
 
 When running as non-root user, there may be some additional resource limits
 that are imposed by the system. Specifically, the following resource limits may
@@ -34,7 +79,15 @@ need to be adjusted in order to ensure normal DPDK operation:
 The above limits can usually be adjusted by editing
 ``/etc/security/limits.conf`` file, and rebooting.
 
-Additionally, depending on which kernel driver is in use, the relevant
+See `Hugepage Mapping <hugepage_mapping>`_
+secton to learn how these limits affect EAL.
+
+Device Control
+~~~~~~~~~~~~~~
+
+If the HPET is to be used, ``/dev/hpet`` permissions must be adjusted.
+
+Depending on which kernel driver is in use, the relevant
 resources also should be accessible by the user running the DPDK application.
 
 For ``vfio-pci`` kernel driver, the following Linux file system objects'
@@ -64,6 +117,8 @@ system objects' permissions should be adjusted:
        /sys/class/uio/uio0/device/config
        /sys/class/uio/uio0/device/resource*
 
+For ``virtio`` PMD in legacy mode, ``SYS_RAWIO`` capability is required
+for ``iopl()`` call to enable access to PCI IO ports.
 
 Power Management and Power Saving Functionality
 -----------------------------------------------
diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 5f0748fba1..70fa099d30 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -228,6 +228,8 @@ Normally, these options do not need to be changed.
     can later be mapped into that preallocated VA space (if dynamic memory mode
     is enabled), and can optionally be mapped into it at startup.
 
+.. _hugepage_mapping:
+
 Hugepage Mapping
 ^^^^^^^^^^^^^^^^
 
-- 
2.25.1

Re: [PATCH v2 3/4] doc: give specific instructions for running as non-root

[Bruce Richardson]

On Fri, Jun 17, 2022 at 02:25:07PM +0300, Dmitry Kozlyuk wrote:
> The guide to run DPDK applications as non-root in Linux
> did not provide specific instructions to configure the required access
> and did not explain why each bit is needed.
> The latter is important because running as non-root
> is one of the ways to tighten security and grant minimal permissions.
> 
> Cc: stable@dpdk.org
> 
> Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>

Thanks for this, some good changes here. Comments inline below.

/Bruce

> ---
>  doc/guides/linux_gsg/enable_func.rst          | 67 +++++++++++++++++--
>  .../prog_guide/env_abstraction_layer.rst      |  2 +
>  2 files changed, 63 insertions(+), 6 deletions(-)
> 
> diff --git a/doc/guides/linux_gsg/enable_func.rst b/doc/guides/linux_gsg/enable_func.rst
> index 1df3ab0255..2f908e8b70 100644
> --- a/doc/guides/linux_gsg/enable_func.rst
> +++ b/doc/guides/linux_gsg/enable_func.rst
> @@ -13,13 +13,58 @@ Enabling Additional Functionality
>  Running DPDK Applications Without Root Privileges
>  -------------------------------------------------
>  
> -In order to run DPDK as non-root, the following Linux filesystem objects'
> -permissions should be adjusted to ensure that the Linux account being used to
> -run the DPDK application has access to them:
> +The following sections describe generic requirements and configuration
> +for running DPDK applications as non-root.
> +There may be additional requirements documented for some drivers.
>  
> -*   All directories which serve as hugepage mount points, for example, ``/dev/hugepages``
> +Hugepages
> +~~~~~~~~~
>  
> -*   If the HPET is to be used,  ``/dev/hpet``
> +Hugepages must be reserved as root before runing the application as non-root,
> +for example::
> +
> +  sudo dpdk-hugepages.py --reserve 1G
> +
> +If multi-process is not required, running with ``--in-memory``
> +bypasses the need to access hugepage mount point and files within it.
> +Otherwise, hugepage directory must be made accessible
> +for writing to the unprivileged user.
> +A good way for managing multiple applications using hugepages
> +is to mount the filesystem with group permissions
> +and add a supplementary group to each application or container.
> +
> +One option is to use the script provided by this project::
> +
> +  export HUGEDIR=$HOME/huge-1G
> +  mkdir -p $HUGEDIR
> +  sudo dpdk-hugepages.py --mount --directory $HUGEDIR --owner `id -u`:`id -g`
> +
> +In production environment, the OS can manage mount points
> +(`systemd example <https://github.com/systemd/systemd/blob/main/units/dev-hugepages.mount>`_).
> +
> +The ``hugetlb`` filesystem has additional options to guarantee or limit
> +the amount of memory that is possible to allocate using the mount point.
> +Refer to the `documentation <https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt>`_.
> +
> +If the driver requires using physical addresses (PA),
> +the executable file must be granted additional capabilities:
> +
> +* ``SYS_ADMIN`` to read ``/proc/self/pagemaps``
> +* ``IPC_LOCK`` to lock hugepages in memory

Are either of these necessary if using vfio-pci and VA mode? I have seen it
previously reported that IPC_LOCK is necessary for IOMMU memory mapping for
DMA - at least for docker containers - so I'd like it confirmed that we
don't need them in the in-memory case running on the host. If I get the
chance I'll try double-checking by testing myself.

> +
> +.. code-block:: console
> +
> +   setcap cap_ipc_lock,cap_sys_admin+ep <executable>
> +
> +If physical addresses are not accessible,
> +the following message will appear during EAL initialization::
> +
> +  EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
> +
> +It is harmless in case PA are not needed.
> +

While this is probably worth having in the doc, I think we should really
include a note here about using vfio-pci rather than uio and therefore not
needing physical addresses.

> +Resource Limits
> +~~~~~~~~~~~~~~~
>  
>  When running as non-root user, there may be some additional resource limits
>  that are imposed by the system. Specifically, the following resource limits may
> @@ -34,7 +79,15 @@ need to be adjusted in order to ensure normal DPDK operation:
>  The above limits can usually be adjusted by editing
>  ``/etc/security/limits.conf`` file, and rebooting.
>  
> -Additionally, depending on which kernel driver is in use, the relevant
> +See `Hugepage Mapping <hugepage_mapping>`_
> +secton to learn how these limits affect EAL.

Typo: s/secton/section/

> +
> +Device Control
> +~~~~~~~~~~~~~~
> +
> +If the HPET is to be used, ``/dev/hpet`` permissions must be adjusted.
> +

Given that HPET has been off by default for years, I think we can probably
remove this line. Anyone still using it likely already knows this.

> +Depending on which kernel driver is in use, the relevant
>  resources also should be accessible by the user running the DPDK application.
>  
>  For ``vfio-pci`` kernel driver, the following Linux file system objects'
> @@ -64,6 +117,8 @@ system objects' permissions should be adjusted:
>         /sys/class/uio/uio0/device/config
>         /sys/class/uio/uio0/device/resource*
>  

I think our minimum supported kernel version is now >4.0 so I believe this
uio section should be removed as it's only applicable for earlier kernel
versions.

> +For ``virtio`` PMD in legacy mode, ``SYS_RAWIO`` capability is required
> +for ``iopl()`` call to enable access to PCI IO ports.
>  

How "legacy" is legacy-mode? Is it still likely in widespread use that we
need this?

>  Power Management and Power Saving Functionality
>  -----------------------------------------------
> diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
> index 5f0748fba1..70fa099d30 100644
> --- a/doc/guides/prog_guide/env_abstraction_layer.rst
> +++ b/doc/guides/prog_guide/env_abstraction_layer.rst
> @@ -228,6 +228,8 @@ Normally, these options do not need to be changed.
>      can later be mapped into that preallocated VA space (if dynamic memory mode
>      is enabled), and can optionally be mapped into it at startup.
>  
> +.. _hugepage_mapping:
> +
>  Hugepage Mapping
>  ^^^^^^^^^^^^^^^^
>  
> -- 
> 2.25.1
> 

RE: [PATCH v2 3/4] doc: give specific instructions for running as non-root

[Dmitry Kozlyuk]

> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Friday, June 17, 2022 7:38 PM
> > [...]
> > +If the driver requires using physical addresses (PA),
> > +the executable file must be granted additional capabilities:
> > +
> > +* ``SYS_ADMIN`` to read ``/proc/self/pagemaps``
> > +* ``IPC_LOCK`` to lock hugepages in memory
> 
> Are either of these necessary if using vfio-pci and VA mode? I have
> seen it previously reported that IPC_LOCK is necessary for IOMMU
> memory mapping for DMA - at least for docker containers - so I'd
> like it confirmed that we don't need them in the in-memory case
> running on the host. If I get the chance I'll try double-checking
> by testing myself.

Sorry, I don't have a physical device using vfio-pci to check.
MLX5 that I have tested doesn't need these capabilities,
but it locks memory from the kernel side.
Note that --in-memory doesn't imply --iova-mode=va.

> 
> > +
> > +.. code-block:: console
> > +
> > +   setcap cap_ipc_lock,cap_sys_admin+ep <executable>
> > +
> > +If physical addresses are not accessible,
> > +the following message will appear during EAL initialization::
> > +
> > +  EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap:
> Permission denied
> > +
> > +It is harmless in case PA are not needed.
> > +
> 
> While this is probably worth having in the doc, I think we should
> really
> include a note here about using vfio-pci rather than uio and therefore
> not
> needing physical addresses.

A note won't harm. There are also non-PCI devices, though.

> > +For ``virtio`` PMD in legacy mode, ``SYS_RAWIO`` capability is
> required
> > +for ``iopl()`` call to enable access to PCI IO ports.
> >
> 
> How "legacy" is legacy-mode? Is it still likely in widespread use that
> we need this?

I don't really know.
The spec says that legacy support is optional
(2.2.3 Legacy Interface: A Note on Feature Bits) and it aims
to reduce the chance of a legacy driver attempting to drive the device
(4.1.2.1 Device Requirements: PCI Device Discovery).
OTOH, DPDK supports it and requirements must be documented.
I can add a line suggesting to use modern virtio,
but also don't mind removing this.

I'll address skipped comments in v3, thanks.

Re: [PATCH v2 3/4] doc: give specific instructions for running as non-root

[Bruce Richardson]

On Mon, Jun 20, 2022 at 06:10:37AM +0000, Dmitry Kozlyuk wrote:
> > From: Bruce Richardson <bruce.richardson@intel.com>
> > Sent: Friday, June 17, 2022 7:38 PM
> > > [...]
> > > +If the driver requires using physical addresses (PA),
> > > +the executable file must be granted additional capabilities:
> > > +
> > > +* ``SYS_ADMIN`` to read ``/proc/self/pagemaps``
> > > +* ``IPC_LOCK`` to lock hugepages in memory
> > 
> > Are either of these necessary if using vfio-pci and VA mode? I have
> > seen it previously reported that IPC_LOCK is necessary for IOMMU
> > memory mapping for DMA - at least for docker containers - so I'd
> > like it confirmed that we don't need them in the in-memory case
> > running on the host. If I get the chance I'll try double-checking
> > by testing myself.
> 
> Sorry, I don't have a physical device using vfio-pci to check.
> MLX5 that I have tested doesn't need these capabilities,
> but it locks memory from the kernel side.
> Note that --in-memory doesn't imply --iova-mode=va.
> 
> > 
> > > +
> > > +.. code-block:: console
> > > +
> > > +   setcap cap_ipc_lock,cap_sys_admin+ep <executable>
> > > +
> > > +If physical addresses are not accessible,
> > > +the following message will appear during EAL initialization::
> > > +
> > > +  EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap:
> > Permission denied
> > > +
> > > +It is harmless in case PA are not needed.
> > > +
> > 
> > While this is probably worth having in the doc, I think we should
> > really
> > include a note here about using vfio-pci rather than uio and therefore
> > not
> > needing physical addresses.
> 
> A note won't harm. There are also non-PCI devices, though.
> 
> > > +For ``virtio`` PMD in legacy mode, ``SYS_RAWIO`` capability is
> > required
> > > +for ``iopl()`` call to enable access to PCI IO ports.
> > >
> > 
> > How "legacy" is legacy-mode? Is it still likely in widespread use that
> > we need this?
> 
> I don't really know.
> The spec says that legacy support is optional
> (2.2.3 Legacy Interface: A Note on Feature Bits) and it aims
> to reduce the chance of a legacy driver attempting to drive the device
> (4.1.2.1 Device Requirements: PCI Device Discovery).
> OTOH, DPDK supports it and requirements must be documented.
> I can add a line suggesting to use modern virtio,
> but also don't mind removing this.
>

I suppose the main question for this legacy virtio bit is where it should
be documented, more than if it should be. Given this is a GSG, we should
try and avoid getting too deep into driver-specific issues, so I think we
should omit legacy virtio here, but have it docuemented in the relevant
virtio-specific doc. Does that seem reasonable?
 

RE: [PATCH v2 3/4] doc: give specific instructions for running as non-root

[Dmitry Kozlyuk]

> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Monday, June 20, 2022 11:38 AM
> [...]
> > > > +For ``virtio`` PMD in legacy mode, ``SYS_RAWIO`` capability is
> > > required
> > > > +for ``iopl()`` call to enable access to PCI IO ports.
> > > >
> > >
> > > How "legacy" is legacy-mode? Is it still likely in widespread use
> that
> > > we need this?
> >
> > I don't really know.
> > The spec says that legacy support is optional
> > (2.2.3 Legacy Interface: A Note on Feature Bits) and it aims
> > to reduce the chance of a legacy driver attempting to drive the
> device
> > (4.1.2.1 Device Requirements: PCI Device Discovery).
> > OTOH, DPDK supports it and requirements must be documented.
> > I can add a line suggesting to use modern virtio,
> > but also don't mind removing this.
> >
> 
> I suppose the main question for this legacy virtio bit
> is where it should be documented, more than if it should be.
> Given this is a GSG, we should try and avoid getting too deep
> into driver-specific issues, so I think we should omit legacy virtio here,
> but have it docuemented in the relevant virtio-specific doc.
> Does that seem reasonable?

Yes, moved to the virtio doc (it looks like it could use an update BTW).

I also chose to keep HPET line because there's an entire section on HPET
below in the document.

[PATCH v2 4/4] doc: update instructions for running as non-root for MLX5

[Dmitry Kozlyuk]

Reference the common guide for generic setup.
Remove excessive capabilities from the recommended list.

Cc: stable@dpdk.org

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
---
 doc/guides/platform/mlx5.rst | 31 ++++++++++++++++++-------------
 1 file changed, 18 insertions(+), 13 deletions(-)

diff --git a/doc/guides/platform/mlx5.rst b/doc/guides/platform/mlx5.rst
index 64a4c5e76e..18d38f3488 100644
--- a/doc/guides/platform/mlx5.rst
+++ b/doc/guides/platform/mlx5.rst
@@ -404,25 +404,30 @@ The device can be bound again at this point.
 Run as Non-Root
 ^^^^^^^^^^^^^^^
 
-In order to run as a non-root user,
-some capabilities must be granted to the application::
+Hugepage and resource limit setup are documented
+in the :ref:`common Linux guide <Running_Without_Root_Privileges>`.
+This PMD can operate without access to physical addresses,
+therefore it does not require ``SYS_ADMIN`` to access ``/proc/self/pagemaps``.
+Note that this requirement may still come from other drivers.
 
-   setcap cap_sys_admin,cap_net_admin,cap_net_raw,cap_ipc_lock+ep <dpdk-app>
+Below are additional capabilities that must be granted to the application
+with the reasons for the need of each capability:
 
-Below are the reasons for the need of each capability:
+``NET_RAW``
+   For raw Ethernet queue allocation through the kernel driver.
 
-``cap_sys_admin``
-   When using physical addresses (PA mode), with Linux >= 4.0,
-   for access to ``/proc/self/pagemap``.
+``NET_ADMIN``
+   For device configuration, like setting link status or MTU.
 
-``cap_net_admin``
-   For device configuration.
+``SYS_RAWIO``
+   For using group 1 and above (software steering) in Flow API.
 
-``cap_net_raw``
-   For raw ethernet queue allocation through kernel driver.
+They can be manually granted for a specific executable file::
 
-``cap_ipc_lock``
-   For DMA memory pinning.
+   setcap cap_net_raw,cap_net_admin,cap_sys_rawio+ep <executable>
+
+Alternatively, a service manager or a container runtime
+may configure the capabilities for a process.
 
 
 Windows Environment
-- 
2.25.1

[PATCH v3 0/5] Improve documentation for running as non-root

[Dmitry Kozlyuk]

v3: Address comments by Bruce Richardson:
    - Remove the section for using UIO with older kernels.
    - Move the note about legacy virtio to virtio doc.
    - Advertise vfio-pci usage to avoid physical addresses
      (still unclear whether and when it requires IPC_LOCK).
    - Fix typo.

v2: Address comments by Stephen Hemminger:
    - Use hugetlbfs options instead of chown.
    - Suggest using OS or container runtime where possible.
    - Improve wording.
    Add SYS_RAWIO requirement for legacy virtio.
    Explain SYS_RAWIO requirement for MLX5 PMD.

Dmitry Kozlyuk (5):
  usertools: add option to select hugetlbfs directory
  usertools: add option to change mount point owner
  doc: give specific instructions for running as non-root
  doc: update instructions for running as non-root for MLX5
  doc: add note about running virtio-legacy as non-root

 doc/guides/linux_gsg/enable_func.rst          | 89 +++++++++++++------
 doc/guides/nics/virtio.rst                    |  3 +
 doc/guides/platform/mlx5.rst                  | 31 ++++---
 .../prog_guide/env_abstraction_layer.rst      |  2 +
 usertools/dpdk-hugepages.py                   | 20 ++++-
 5 files changed, 102 insertions(+), 43 deletions(-)

-- 
2.25.1

[PATCH v3 1/5] usertools: add option to select hugetlbfs directory

[Dmitry Kozlyuk]

dpdk-hugepages.py had /dev/hugepages hardcoded as the mount point.
It may be desirable to setup hugepage directory at another path,
for example, when using hugepages of multiple sizes in different
directories or when granting different permissions to mount points.
Add --directory/-d option to the script.

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
---
 usertools/dpdk-hugepages.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/usertools/dpdk-hugepages.py b/usertools/dpdk-hugepages.py
index 4fdb199744..8bab086a2f 100755
--- a/usertools/dpdk-hugepages.py
+++ b/usertools/dpdk-hugepages.py
@@ -228,6 +228,12 @@ def main():
         '-u',
         action='store_true',
         help='unmount the system huge page directory')
+    parser.add_argument(
+        '--directory',
+        '-d',
+        metavar='DIR',
+        default=HUGE_MOUNT,
+        help='mount point')
     parser.add_argument(
         '--node', '-n', help='select numa node to reserve pages on')
     parser.add_argument(
@@ -262,7 +268,7 @@ def main():
     if args.clear:
         clear_pages()
     if args.unmount:
-        umount_huge(HUGE_MOUNT)
+        umount_huge(args.directory)
 
     if args.reserve:
         reserve_kb = get_memsize(args.reserve)
@@ -273,7 +279,7 @@ def main():
         reserve_pages(
             int(reserve_kb / pagesize_kb), pagesize_kb, node=args.node)
     if args.mount:
-        mount_huge(pagesize_kb, HUGE_MOUNT)
+        mount_huge(pagesize_kb, args.directory)
     if args.show:
         show_pages()
         print()
-- 
2.25.1

Re: [PATCH v3 1/5] usertools: add option to select hugetlbfs directory

[Bruce Richardson]

On Fri, Jun 24, 2022 at 11:48:13AM +0300, Dmitry Kozlyuk wrote:
> dpdk-hugepages.py had /dev/hugepages hardcoded as the mount point.
> It may be desirable to setup hugepage directory at another path,
> for example, when using hugepages of multiple sizes in different
> directories or when granting different permissions to mount points.
> Add --directory/-d option to the script.
> 
> Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
> ---

Acked-by: Bruce Richardson <bruce.richardson@intel.com>

[PATCH v3 2/5] usertools: add option to change mount point owner

[Dmitry Kozlyuk]

Per mount(8), the previous owner and mode of the mount point
become invisible as long as this filesystem remains mounted.
Because dpdk-hugepages.py must be run as root,
the new owner would be root.
This is undesirable if the hugepage directory is being set up
by the administrator for an unprivileged user.
HugeTLB filesystem has options to set the mount point owner.
Add --owner/-o option to apply this option when mounting.
The benefit of performing this in dpdk-hugepages.py
is that the user does not need to care about this detail
of mount command operation.

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
---
 usertools/dpdk-hugepages.py | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/usertools/dpdk-hugepages.py b/usertools/dpdk-hugepages.py
index 8bab086a2f..5120518bcb 100755
--- a/usertools/dpdk-hugepages.py
+++ b/usertools/dpdk-hugepages.py
@@ -170,7 +170,7 @@ def get_mountpoints():
     return mounted
 
 
-def mount_huge(pagesize, mountpoint):
+def mount_huge(pagesize, mountpoint, owner):
     '''Mount the huge TLB file system'''
     if mountpoint in get_mountpoints():
         print(mountpoint, "already mounted")
@@ -178,6 +178,9 @@ def mount_huge(pagesize, mountpoint):
     cmd = "mount -t hugetlbfs"
     if pagesize:
         cmd += ' -o pagesize={}'.format(pagesize * 1024)
+    if owner:
+        uid, gid = owner.split(':', maxsplit=1)
+        cmd += ' -o uid={},gid={}'.format(uid, gid)
     cmd += ' nodev ' + mountpoint
     os.system(cmd)
 
@@ -234,6 +237,11 @@ def main():
         metavar='DIR',
         default=HUGE_MOUNT,
         help='mount point')
+    parser.add_argument(
+        '--owner',
+        '-o',
+        metavar='USER:GROUP',
+        help='change the mounted directory owner')
     parser.add_argument(
         '--node', '-n', help='select numa node to reserve pages on')
     parser.add_argument(
@@ -279,7 +287,7 @@ def main():
         reserve_pages(
             int(reserve_kb / pagesize_kb), pagesize_kb, node=args.node)
     if args.mount:
-        mount_huge(pagesize_kb, args.directory)
+        mount_huge(pagesize_kb, args.directory, args.owner)
     if args.show:
         show_pages()
         print()
-- 
2.25.1

Re: [PATCH v3 2/5] usertools: add option to change mount point owner

[Bruce Richardson]

On Fri, Jun 24, 2022 at 11:48:14AM +0300, Dmitry Kozlyuk wrote:
> Per mount(8), the previous owner and mode of the mount point
> become invisible as long as this filesystem remains mounted.
> Because dpdk-hugepages.py must be run as root,
> the new owner would be root.
> This is undesirable if the hugepage directory is being set up
> by the administrator for an unprivileged user.
> HugeTLB filesystem has options to set the mount point owner.
> Add --owner/-o option to apply this option when mounting.
> The benefit of performing this in dpdk-hugepages.py
> is that the user does not need to care about this detail
> of mount command operation.
> 
> Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
> ---
>  usertools/dpdk-hugepages.py | 12 ++++++++++--
>  1 file changed, 10 insertions(+), 2 deletions(-)
> 
> diff --git a/usertools/dpdk-hugepages.py b/usertools/dpdk-hugepages.py
> index 8bab086a2f..5120518bcb 100755
> --- a/usertools/dpdk-hugepages.py
> +++ b/usertools/dpdk-hugepages.py
> @@ -170,7 +170,7 @@ def get_mountpoints():
>      return mounted
>  
>  
> -def mount_huge(pagesize, mountpoint):
> +def mount_huge(pagesize, mountpoint, owner):
>      '''Mount the huge TLB file system'''
>      if mountpoint in get_mountpoints():
>          print(mountpoint, "already mounted")
> @@ -178,6 +178,9 @@ def mount_huge(pagesize, mountpoint):
>      cmd = "mount -t hugetlbfs"
>      if pagesize:
>          cmd += ' -o pagesize={}'.format(pagesize * 1024)
> +    if owner:
> +        uid, gid = owner.split(':', maxsplit=1)
> +        cmd += ' -o uid={},gid={}'.format(uid, gid)

Still don't like this syntax and always setting both together. Can we
change the "owner" flag to separate "uid" and "gid" flags so they can be
set independently (and explicitly).

/Bruce

[PATCH v3 3/5] doc: give specific instructions for running as non-root

[Dmitry Kozlyuk]

The guide to run DPDK applications as non-root in Linux
did not provide specific instructions to configure the required access
and did not explain why each bit is needed.
The latter is important because running as non-root
is one of the ways to tighten security and grant minimal permissions.

Cc: stable@dpdk.org

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
---
 doc/guides/linux_gsg/enable_func.rst          | 89 +++++++++++++------
 .../prog_guide/env_abstraction_layer.rst      |  2 +
 2 files changed, 64 insertions(+), 27 deletions(-)

diff --git a/doc/guides/linux_gsg/enable_func.rst b/doc/guides/linux_gsg/enable_func.rst
index 1df3ab0255..0b57417c94 100644
--- a/doc/guides/linux_gsg/enable_func.rst
+++ b/doc/guides/linux_gsg/enable_func.rst
@@ -13,13 +13,63 @@ Enabling Additional Functionality
 Running DPDK Applications Without Root Privileges
 -------------------------------------------------
 
-In order to run DPDK as non-root, the following Linux filesystem objects'
-permissions should be adjusted to ensure that the Linux account being used to
-run the DPDK application has access to them:
+The following sections describe generic requirements and configuration
+for running DPDK applications as non-root.
+There may be additional requirements documented for some drivers.
 
-*   All directories which serve as hugepage mount points, for example, ``/dev/hugepages``
+Hugepages
+~~~~~~~~~
 
-*   If the HPET is to be used,  ``/dev/hpet``
+Hugepages must be reserved as root before running the application as non-root,
+for example::
+
+  sudo dpdk-hugepages.py --reserve 1G
+
+If multi-process is not required, running with ``--in-memory``
+bypasses the need to access hugepage mount point and files within it.
+Otherwise, hugepage directory must be made accessible
+for writing to the unprivileged user.
+A good way for managing multiple applications using hugepages
+is to mount the filesystem with group permissions
+and add a supplementary group to each application or container.
+
+One option is to use the script provided by this project::
+
+  export HUGEDIR=$HOME/huge-1G
+  mkdir -p $HUGEDIR
+  sudo dpdk-hugepages.py --mount --directory $HUGEDIR --owner `id -u`:`id -g`
+
+In production environment, the OS can manage mount points
+(`systemd example <https://github.com/systemd/systemd/blob/main/units/dev-hugepages.mount>`_).
+
+The ``hugetlb`` filesystem has additional options to guarantee or limit
+the amount of memory that is possible to allocate using the mount point.
+Refer to the `documentation <https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt>`_.
+
+If the driver requires using physical addresses (PA),
+the executable file must be granted additional capabilities:
+
+* ``SYS_ADMIN`` to read ``/proc/self/pagemaps``
+* ``IPC_LOCK`` to lock hugepages in memory
+
+.. code-block:: console
+
+   setcap cap_ipc_lock,cap_sys_admin+ep <executable>
+
+If physical addresses are not accessible,
+the following message will appear during EAL initialization::
+
+  EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
+
+It is harmless in case PA are not needed.
+
+.. note::
+
+   Using ``vfio-pci`` kernel driver, if applicable, can eliminate the need
+   for physical addresses and therefore reduce the permission requirements.
+
+Resource Limits
+~~~~~~~~~~~~~~~
 
 When running as non-root user, there may be some additional resource limits
 that are imposed by the system. Specifically, the following resource limits may
@@ -34,8 +84,13 @@ need to be adjusted in order to ensure normal DPDK operation:
 The above limits can usually be adjusted by editing
 ``/etc/security/limits.conf`` file, and rebooting.
 
-Additionally, depending on which kernel driver is in use, the relevant
-resources also should be accessible by the user running the DPDK application.
+See `Hugepage Mapping <hugepage_mapping>`_
+section to learn how these limits affect EAL.
+
+Device Control
+~~~~~~~~~~~~~~
+
+If the HPET is to be used, ``/dev/hpet`` permissions must be adjusted.
 
 For ``vfio-pci`` kernel driver, the following Linux file system objects'
 permissions should be adjusted:
@@ -45,26 +100,6 @@ permissions should be adjusted:
 * The directories under ``/dev/vfio`` that correspond to IOMMU group numbers of
   devices intended to be used by DPDK, for example, ``/dev/vfio/50``
 
-.. note::
-
-    The instructions below will allow running DPDK with ``igb_uio`` or
-    ``uio_pci_generic`` drivers as non-root with older Linux kernel versions.
-    However, since version 4.0, the kernel does not allow unprivileged processes
-    to read the physical address information from the pagemaps file, making it
-    impossible for those processes to be used by non-privileged users. In such
-    cases, using the VFIO driver is recommended.
-
-For ``igb_uio`` or ``uio_pci_generic`` kernel drivers, the following Linux file
-system objects' permissions should be adjusted:
-
-*   The userspace-io device files in  ``/dev``, for example,  ``/dev/uio0``, ``/dev/uio1``, and so on
-
-*   The userspace-io sysfs config and resource files, for example for ``uio0``::
-
-       /sys/class/uio/uio0/device/config
-       /sys/class/uio/uio0/device/resource*
-
-
 Power Management and Power Saving Functionality
 -----------------------------------------------
 
diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 5f0748fba1..70fa099d30 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -228,6 +228,8 @@ Normally, these options do not need to be changed.
     can later be mapped into that preallocated VA space (if dynamic memory mode
     is enabled), and can optionally be mapped into it at startup.
 
+.. _hugepage_mapping:
+
 Hugepage Mapping
 ^^^^^^^^^^^^^^^^
 
-- 
2.25.1

Re: [PATCH v3 3/5] doc: give specific instructions for running as non-root

[Bruce Richardson]

On Fri, Jun 24, 2022 at 11:48:15AM +0300, Dmitry Kozlyuk wrote:
> The guide to run DPDK applications as non-root in Linux
> did not provide specific instructions to configure the required access
> and did not explain why each bit is needed.
> The latter is important because running as non-root
> is one of the ways to tighten security and grant minimal permissions.
> 
> Cc: stable@dpdk.org
> 
> Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>

Good improvements. One small suggestion below, otherwise:

Acked-by: Bruce Richardson <bruce.richardson@intel.com>

> ---
>  doc/guides/linux_gsg/enable_func.rst          | 89 +++++++++++++------
>  .../prog_guide/env_abstraction_layer.rst      |  2 +
>  2 files changed, 64 insertions(+), 27 deletions(-)
> 
> diff --git a/doc/guides/linux_gsg/enable_func.rst b/doc/guides/linux_gsg/enable_func.rst
> index 1df3ab0255..0b57417c94 100644
> --- a/doc/guides/linux_gsg/enable_func.rst
> +++ b/doc/guides/linux_gsg/enable_func.rst
> @@ -13,13 +13,63 @@ Enabling Additional Functionality
>  Running DPDK Applications Without Root Privileges
>  -------------------------------------------------
>  
> -In order to run DPDK as non-root, the following Linux filesystem objects'
> -permissions should be adjusted to ensure that the Linux account being used to
> -run the DPDK application has access to them:
> +The following sections describe generic requirements and configuration
> +for running DPDK applications as non-root.
> +There may be additional requirements documented for some drivers.
>  
> -*   All directories which serve as hugepage mount points, for example, ``/dev/hugepages``
> +Hugepages
> +~~~~~~~~~
>  
> -*   If the HPET is to be used,  ``/dev/hpet``
> +Hugepages must be reserved as root before running the application as non-root,
> +for example::
> +
> +  sudo dpdk-hugepages.py --reserve 1G
> +
> +If multi-process is not required, running with ``--in-memory``
> +bypasses the need to access hugepage mount point and files within it.
> +Otherwise, hugepage directory must be made accessible
> +for writing to the unprivileged user.
> +A good way for managing multiple applications using hugepages
> +is to mount the filesystem with group permissions
> +and add a supplementary group to each application or container.
> +
> +One option is to use the script provided by this project::
> +
> +  export HUGEDIR=$HOME/huge-1G
> +  mkdir -p $HUGEDIR
> +  sudo dpdk-hugepages.py --mount --directory $HUGEDIR --owner `id -u`:`id -g`
> +
> +In production environment, the OS can manage mount points
> +(`systemd example <https://github.com/systemd/systemd/blob/main/units/dev-hugepages.mount>`_).
> +
> +The ``hugetlb`` filesystem has additional options to guarantee or limit
> +the amount of memory that is possible to allocate using the mount point.
> +Refer to the `documentation <https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt>`_.
> +
> +If the driver requires using physical addresses (PA),
> +the executable file must be granted additional capabilities:
> +
> +* ``SYS_ADMIN`` to read ``/proc/self/pagemaps``
> +* ``IPC_LOCK`` to lock hugepages in memory
> +
> +.. code-block:: console
> +
> +   setcap cap_ipc_lock,cap_sys_admin+ep <executable>
> +
> +If physical addresses are not accessible,
> +the following message will appear during EAL initialization::
> +
> +  EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
> +
> +It is harmless in case PA are not needed.
> +
> +.. note::
> +
> +   Using ``vfio-pci`` kernel driver, if applicable, can eliminate the need
> +   for physical addresses and therefore reduce the permission requirements.
> +

Can we move this note up a bit, to immediately after the paragraph about
requiring physical addresses. It's better to inform the user immediately if
a section is not relevant to them, than only telling them at the end once
they have already read it.

/Bruce

[PATCH v3 4/5] doc: update instructions for running as non-root for MLX5

[Dmitry Kozlyuk]

Reference the common guide for generic setup.
Remove excessive capabilities from the recommended list.

Cc: stable@dpdk.org

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
---
 doc/guides/platform/mlx5.rst | 31 ++++++++++++++++++-------------
 1 file changed, 18 insertions(+), 13 deletions(-)

diff --git a/doc/guides/platform/mlx5.rst b/doc/guides/platform/mlx5.rst
index 64a4c5e76e..18d38f3488 100644
--- a/doc/guides/platform/mlx5.rst
+++ b/doc/guides/platform/mlx5.rst
@@ -404,25 +404,30 @@ The device can be bound again at this point.
 Run as Non-Root
 ^^^^^^^^^^^^^^^
 
-In order to run as a non-root user,
-some capabilities must be granted to the application::
+Hugepage and resource limit setup are documented
+in the :ref:`common Linux guide <Running_Without_Root_Privileges>`.
+This PMD can operate without access to physical addresses,
+therefore it does not require ``SYS_ADMIN`` to access ``/proc/self/pagemaps``.
+Note that this requirement may still come from other drivers.
 
-   setcap cap_sys_admin,cap_net_admin,cap_net_raw,cap_ipc_lock+ep <dpdk-app>
+Below are additional capabilities that must be granted to the application
+with the reasons for the need of each capability:
 
-Below are the reasons for the need of each capability:
+``NET_RAW``
+   For raw Ethernet queue allocation through the kernel driver.
 
-``cap_sys_admin``
-   When using physical addresses (PA mode), with Linux >= 4.0,
-   for access to ``/proc/self/pagemap``.
+``NET_ADMIN``
+   For device configuration, like setting link status or MTU.
 
-``cap_net_admin``
-   For device configuration.
+``SYS_RAWIO``
+   For using group 1 and above (software steering) in Flow API.
 
-``cap_net_raw``
-   For raw ethernet queue allocation through kernel driver.
+They can be manually granted for a specific executable file::
 
-``cap_ipc_lock``
-   For DMA memory pinning.
+   setcap cap_net_raw,cap_net_admin,cap_sys_rawio+ep <executable>
+
+Alternatively, a service manager or a container runtime
+may configure the capabilities for a process.
 
 
 Windows Environment
-- 
2.25.1

[PATCH v3 5/5] doc: add note about running virtio-legacy as non-root

[Dmitry Kozlyuk]

The requirement of SYS_ADMIN capability in legacy virtio mode
was missing. Add it to the driver-specific page.

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
---
 doc/guides/nics/virtio.rst | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/doc/guides/nics/virtio.rst b/doc/guides/nics/virtio.rst
index 7c0ae2b3af..0e552b2701 100644
--- a/doc/guides/nics/virtio.rst
+++ b/doc/guides/nics/virtio.rst
@@ -86,6 +86,9 @@ The following prerequisites apply:
 *   Linux kernel with KVM module; vhost module loaded and ioeventfd supported.
     Qemu standard backend without vhost support isn't tested, and probably isn't supported.
 
+*   When using legacy interface, ``SYS_RAWIO`` capability is required
+    for ``iopl()`` call to enable access to PCI IO ports.
+
 Virtio with kni vhost Back End
 ------------------------------
 
-- 
2.25.1

[PATCH v4 0/5] Improve documentation for running as non-root

[Dmitry Kozlyuk]

v4: Same,
    - Split dpdk-hugepages.py option --owner/-o
      into --user/-U and --group/-G.
    - Move vfio-pci note to the beginning of section.

v3: Address comments by Bruce Richardson:
    - Remove the section for using UIO with older kernels.
    - Move the note about legacy virtio to virtio doc.
    - Advertise vfio-pci usage to avoid physical addresses
      (still unclear whether and when it requires IPC_LOCK).
    - Fix typo.

v2: Address comments by Stephen Hemminger:
    - Use hugetlbfs options instead of chown.
    - Suggest using OS or container runtime where possible.
    - Improve wording.
    Add SYS_RAWIO requirement for legacy virtio.
    Explain SYS_RAWIO requirement for MLX5 PMD.

Dmitry Kozlyuk (5):
  usertools: add option to select hugetlbfs directory
  usertools: add options to change mount point owner
  doc: give specific instructions for running as non-root
  doc: update instructions for running as non-root for MLX5
  doc: add note about running virtio-legacy as non-root

 doc/guides/linux_gsg/enable_func.rst          | 90 +++++++++++++------
 doc/guides/nics/virtio.rst                    |  3 +
 doc/guides/platform/mlx5.rst                  | 31 ++++---
 .../prog_guide/env_abstraction_layer.rst      |  2 +
 usertools/dpdk-hugepages.py                   | 26 +++++-
 5 files changed, 109 insertions(+), 43 deletions(-)

-- 
2.25.1

[PATCH v4 1/5] usertools: add option to select hugetlbfs directory

[Dmitry Kozlyuk]

dpdk-hugepages.py had /dev/hugepages hardcoded as the mount point.
It may be desirable to setup hugepage directory at another path,
for example, when using hugepages of multiple sizes in different
directories or when granting different permissions to mount points.
Add --directory/-d option to the script.

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
 usertools/dpdk-hugepages.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/usertools/dpdk-hugepages.py b/usertools/dpdk-hugepages.py
index 4fdb199744..8bab086a2f 100755
--- a/usertools/dpdk-hugepages.py
+++ b/usertools/dpdk-hugepages.py
@@ -228,6 +228,12 @@ def main():
         '-u',
         action='store_true',
         help='unmount the system huge page directory')
+    parser.add_argument(
+        '--directory',
+        '-d',
+        metavar='DIR',
+        default=HUGE_MOUNT,
+        help='mount point')
     parser.add_argument(
         '--node', '-n', help='select numa node to reserve pages on')
     parser.add_argument(
@@ -262,7 +268,7 @@ def main():
     if args.clear:
         clear_pages()
     if args.unmount:
-        umount_huge(HUGE_MOUNT)
+        umount_huge(args.directory)
 
     if args.reserve:
         reserve_kb = get_memsize(args.reserve)
@@ -273,7 +279,7 @@ def main():
         reserve_pages(
             int(reserve_kb / pagesize_kb), pagesize_kb, node=args.node)
     if args.mount:
-        mount_huge(pagesize_kb, HUGE_MOUNT)
+        mount_huge(pagesize_kb, args.directory)
     if args.show:
         show_pages()
         print()
-- 
2.25.1

[PATCH v4 2/5] usertools: add options to change mount point owner

[Dmitry Kozlyuk]

Per mount(8), the previous owner and mode of the mount point
become invisible as long as this filesystem remains mounted.
Because dpdk-hugepages.py must be run as root,
the new owner would be root.
This is undesirable if the hugepage directory is being set up
by the administrator for an unprivileged user.
HugeTLB filesystem has options to set the mount point owner.
Add --user/-U and --group/-G options to apply this when mounting.
The benefit of performing this in dpdk-hugepages.py
is that the user does not need to care about this detail
of mount command operation.

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
---
Option --unmount/-u was already taken.

 usertools/dpdk-hugepages.py | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/usertools/dpdk-hugepages.py b/usertools/dpdk-hugepages.py
index 8bab086a2f..a22d504d3a 100755
--- a/usertools/dpdk-hugepages.py
+++ b/usertools/dpdk-hugepages.py
@@ -170,7 +170,7 @@ def get_mountpoints():
     return mounted
 
 
-def mount_huge(pagesize, mountpoint):
+def mount_huge(pagesize, mountpoint, user, group):
     '''Mount the huge TLB file system'''
     if mountpoint in get_mountpoints():
         print(mountpoint, "already mounted")
@@ -178,6 +178,10 @@ def mount_huge(pagesize, mountpoint):
     cmd = "mount -t hugetlbfs"
     if pagesize:
         cmd += ' -o pagesize={}'.format(pagesize * 1024)
+    if user:
+        cmd += ' -o uid=' + user
+    if group:
+        cmd += ' -o gid=' + group
     cmd += ' nodev ' + mountpoint
     os.system(cmd)
 
@@ -234,6 +238,16 @@ def main():
         metavar='DIR',
         default=HUGE_MOUNT,
         help='mount point')
+    parser.add_argument(
+        '--user',
+        '-U',
+        metavar='UID',
+        help='set the mounted directory owner user')
+    parser.add_argument(
+        '--group',
+        '-G',
+        metavar='GID',
+        help='set the mounted directory owner group')
     parser.add_argument(
         '--node', '-n', help='select numa node to reserve pages on')
     parser.add_argument(
@@ -279,7 +293,7 @@ def main():
         reserve_pages(
             int(reserve_kb / pagesize_kb), pagesize_kb, node=args.node)
     if args.mount:
-        mount_huge(pagesize_kb, args.directory)
+        mount_huge(pagesize_kb, args.directory, args.user, args.group)
     if args.show:
         show_pages()
         print()
-- 
2.25.1

Re: [PATCH v4 2/5] usertools: add options to change mount point owner

[Bruce Richardson]

On Fri, Jun 24, 2022 at 04:19:53PM +0300, Dmitry Kozlyuk wrote:
> Per mount(8), the previous owner and mode of the mount point
> become invisible as long as this filesystem remains mounted.
> Because dpdk-hugepages.py must be run as root,
> the new owner would be root.
> This is undesirable if the hugepage directory is being set up
> by the administrator for an unprivileged user.
> HugeTLB filesystem has options to set the mount point owner.
> Add --user/-U and --group/-G options to apply this when mounting.
> The benefit of performing this in dpdk-hugepages.py
> is that the user does not need to care about this detail
> of mount command operation.
> 
> Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
> ---
Acked-by: Bruce Richardson <bruce.richardson@intel.com>

[PATCH v4 3/5] doc: give specific instructions for running as non-root

[Dmitry Kozlyuk]

The guide to run DPDK applications as non-root in Linux
did not provide specific instructions to configure the required access
and did not explain why each bit is needed.
The latter is important because running as non-root
is one of the ways to tighten security and grant minimal permissions.

Cc: stable@dpdk.org

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/linux_gsg/enable_func.rst          | 90 +++++++++++++------
 .../prog_guide/env_abstraction_layer.rst      |  2 +
 2 files changed, 65 insertions(+), 27 deletions(-)

diff --git a/doc/guides/linux_gsg/enable_func.rst b/doc/guides/linux_gsg/enable_func.rst
index 1df3ab0255..b15bfb2f9f 100644
--- a/doc/guides/linux_gsg/enable_func.rst
+++ b/doc/guides/linux_gsg/enable_func.rst
@@ -13,13 +13,64 @@ Enabling Additional Functionality
 Running DPDK Applications Without Root Privileges
 -------------------------------------------------
 
-In order to run DPDK as non-root, the following Linux filesystem objects'
-permissions should be adjusted to ensure that the Linux account being used to
-run the DPDK application has access to them:
+The following sections describe generic requirements and configuration
+for running DPDK applications as non-root.
+There may be additional requirements documented for some drivers.
 
-*   All directories which serve as hugepage mount points, for example, ``/dev/hugepages``
+Hugepages
+~~~~~~~~~
 
-*   If the HPET is to be used,  ``/dev/hpet``
+Hugepages must be reserved as root before running the application as non-root,
+for example::
+
+  sudo dpdk-hugepages.py --reserve 1G
+
+If multi-process is not required, running with ``--in-memory``
+bypasses the need to access hugepage mount point and files within it.
+Otherwise, hugepage directory must be made accessible
+for writing to the unprivileged user.
+A good way for managing multiple applications using hugepages
+is to mount the filesystem with group permissions
+and add a supplementary group to each application or container.
+
+One option is to use the script provided by this project::
+
+  export HUGEDIR=$HOME/huge-1G
+  mkdir -p $HUGEDIR
+  sudo dpdk-hugepages.py --mount --directory $HUGEDIR --user `id -u` --group `id -g`
+
+In production environment, the OS can manage mount points
+(`systemd example <https://github.com/systemd/systemd/blob/main/units/dev-hugepages.mount>`_).
+
+The ``hugetlb`` filesystem has additional options to guarantee or limit
+the amount of memory that is possible to allocate using the mount point.
+Refer to the `documentation <https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt>`_.
+
+.. note::
+
+   Using ``vfio-pci`` kernel driver, if applicable, can eliminate the need
+   for physical addresses and therefore eliminate the permission requirements
+   described below.
+
+If the driver requires using physical addresses (PA),
+the executable file must be granted additional capabilities:
+
+* ``SYS_ADMIN`` to read ``/proc/self/pagemaps``
+* ``IPC_LOCK`` to lock hugepages in memory
+
+.. code-block:: console
+
+   setcap cap_ipc_lock,cap_sys_admin+ep <executable>
+
+If physical addresses are not accessible,
+the following message will appear during EAL initialization::
+
+  EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
+
+It is harmless in case PA are not needed.
+
+Resource Limits
+~~~~~~~~~~~~~~~
 
 When running as non-root user, there may be some additional resource limits
 that are imposed by the system. Specifically, the following resource limits may
@@ -34,8 +85,13 @@ need to be adjusted in order to ensure normal DPDK operation:
 The above limits can usually be adjusted by editing
 ``/etc/security/limits.conf`` file, and rebooting.
 
-Additionally, depending on which kernel driver is in use, the relevant
-resources also should be accessible by the user running the DPDK application.
+See `Hugepage Mapping <hugepage_mapping>`_
+section to learn how these limits affect EAL.
+
+Device Control
+~~~~~~~~~~~~~~
+
+If the HPET is to be used, ``/dev/hpet`` permissions must be adjusted.
 
 For ``vfio-pci`` kernel driver, the following Linux file system objects'
 permissions should be adjusted:
@@ -45,26 +101,6 @@ permissions should be adjusted:
 * The directories under ``/dev/vfio`` that correspond to IOMMU group numbers of
   devices intended to be used by DPDK, for example, ``/dev/vfio/50``
 
-.. note::
-
-    The instructions below will allow running DPDK with ``igb_uio`` or
-    ``uio_pci_generic`` drivers as non-root with older Linux kernel versions.
-    However, since version 4.0, the kernel does not allow unprivileged processes
-    to read the physical address information from the pagemaps file, making it
-    impossible for those processes to be used by non-privileged users. In such
-    cases, using the VFIO driver is recommended.
-
-For ``igb_uio`` or ``uio_pci_generic`` kernel drivers, the following Linux file
-system objects' permissions should be adjusted:
-
-*   The userspace-io device files in  ``/dev``, for example,  ``/dev/uio0``, ``/dev/uio1``, and so on
-
-*   The userspace-io sysfs config and resource files, for example for ``uio0``::
-
-       /sys/class/uio/uio0/device/config
-       /sys/class/uio/uio0/device/resource*
-
-
 Power Management and Power Saving Functionality
 -----------------------------------------------
 
diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 5f0748fba1..70fa099d30 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -228,6 +228,8 @@ Normally, these options do not need to be changed.
     can later be mapped into that preallocated VA space (if dynamic memory mode
     is enabled), and can optionally be mapped into it at startup.
 
+.. _hugepage_mapping:
+
 Hugepage Mapping
 ^^^^^^^^^^^^^^^^
 
-- 
2.25.1

[PATCH v4 4/5] doc: update instructions for running as non-root for MLX5

[Dmitry Kozlyuk]

Reference the common guide for generic setup.
Remove excessive capabilities from the recommended list.

Cc: stable@dpdk.org

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
---
 doc/guides/platform/mlx5.rst | 31 ++++++++++++++++++-------------
 1 file changed, 18 insertions(+), 13 deletions(-)

diff --git a/doc/guides/platform/mlx5.rst b/doc/guides/platform/mlx5.rst
index 64a4c5e76e..18d38f3488 100644
--- a/doc/guides/platform/mlx5.rst
+++ b/doc/guides/platform/mlx5.rst
@@ -404,25 +404,30 @@ The device can be bound again at this point.
 Run as Non-Root
 ^^^^^^^^^^^^^^^
 
-In order to run as a non-root user,
-some capabilities must be granted to the application::
+Hugepage and resource limit setup are documented
+in the :ref:`common Linux guide <Running_Without_Root_Privileges>`.
+This PMD can operate without access to physical addresses,
+therefore it does not require ``SYS_ADMIN`` to access ``/proc/self/pagemaps``.
+Note that this requirement may still come from other drivers.
 
-   setcap cap_sys_admin,cap_net_admin,cap_net_raw,cap_ipc_lock+ep <dpdk-app>
+Below are additional capabilities that must be granted to the application
+with the reasons for the need of each capability:
 
-Below are the reasons for the need of each capability:
+``NET_RAW``
+   For raw Ethernet queue allocation through the kernel driver.
 
-``cap_sys_admin``
-   When using physical addresses (PA mode), with Linux >= 4.0,
-   for access to ``/proc/self/pagemap``.
+``NET_ADMIN``
+   For device configuration, like setting link status or MTU.
 
-``cap_net_admin``
-   For device configuration.
+``SYS_RAWIO``
+   For using group 1 and above (software steering) in Flow API.
 
-``cap_net_raw``
-   For raw ethernet queue allocation through kernel driver.
+They can be manually granted for a specific executable file::
 
-``cap_ipc_lock``
-   For DMA memory pinning.
+   setcap cap_net_raw,cap_net_admin,cap_sys_rawio+ep <executable>
+
+Alternatively, a service manager or a container runtime
+may configure the capabilities for a process.
 
 
 Windows Environment
-- 
2.25.1

[PATCH v4 5/5] doc: add note about running virtio-legacy as non-root

[Dmitry Kozlyuk]

The requirement of SYS_ADMIN capability in legacy virtio mode
was missing. Add it to the driver-specific page.

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
---
 doc/guides/nics/virtio.rst | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/doc/guides/nics/virtio.rst b/doc/guides/nics/virtio.rst
index 7c0ae2b3af..0e552b2701 100644
--- a/doc/guides/nics/virtio.rst
+++ b/doc/guides/nics/virtio.rst
@@ -86,6 +86,9 @@ The following prerequisites apply:
 *   Linux kernel with KVM module; vhost module loaded and ioeventfd supported.
     Qemu standard backend without vhost support isn't tested, and probably isn't supported.
 
+*   When using legacy interface, ``SYS_RAWIO`` capability is required
+    for ``iopl()`` call to enable access to PCI IO ports.
+
 Virtio with kni vhost Back End
 ------------------------------
 
-- 
2.25.1

Re: [PATCH v4 0/5] Improve documentation for running as non-root

[Thomas Monjalon]

> Dmitry Kozlyuk (5):
>   usertools: add option to select hugetlbfs directory
>   usertools: add options to change mount point owner
>   doc: give specific instructions for running as non-root
>   doc: update instructions for running as non-root for MLX5
>   doc: add note about running virtio-legacy as non-root

Applied, thanks.