Re: [dpdk-dev] [PATCH v2 2/2] doc/linux_gsg: update information on using hugepages

[Bruce Richardson <bruce.richardson@intel.com>]

On Tue, Aug 25, 2020 at 01:17:49PM +0100, Anatoly Burakov wrote:
> Current information regarding hugepage usage is a little out of date.
> Update it to include information on in-memory mode, as well as on
> default mountpoints provided by systemd.
> 
> Cc: stable@dpdk.org
> 
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> ---
> 
> Notes:
>     v2:
>     - Reworked the description
>     - Put runtime reservation first, and boot time as an alternative
>     - Clarified wording and fixed typos
>     - Mentioned that some kernel versions not supporting reserving 1G pages
> 
>  doc/guides/linux_gsg/sys_reqs.rst | 71 ++++++++++++++++++++-----------
>  1 file changed, 45 insertions(+), 26 deletions(-)
> 
> diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
> index a124656bcb..8782d05579 100644
> --- a/doc/guides/linux_gsg/sys_reqs.rst
> +++ b/doc/guides/linux_gsg/sys_reqs.rst
> @@ -155,8 +155,35 @@ Without hugepages, high TLB miss rates would occur with the standard 4k page siz
>  Reserving Hugepages for DPDK Use
>  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>  
> -The allocation of hugepages should be done at boot time or as soon as possible after system boot
> -to prevent memory from being fragmented in physical memory.
> +The reservation of hugepages can be performed at run time. This is done by
> +echoing the number of hugepages required to a ``nr_hugepages`` file in the
> +``/sys/kernel/`` directory corresponding to a specific page size (in
> +Kilobytes). For a single-node system, the command to use is as follows
> +(assuming that 1024 of 2MB pages are required)::
> +
> +    echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
> +
> +On a NUMA machine, the above command will usually divide the number of hugepages
> +equally across all NUMA nodes (assuming there is enough memory on all NUMA
> +nodes). However, pages can also be reserved explicitly on individual NUMA
> +nodes using a ``nr_hugepages`` file in the ``/sys/devices/`` directory::
> +
> +    echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
> +    echo 1024 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
> +
> +.. note::
> +
> +    Some kernel versions may not allow reserving 1 GB hugepages at run time, so
> +    reserving them at boot time may be the only option. Please see below for
> +    instructions.
> +
> +**Alternative:**
> +
> +In the general case, reserving hugepages at run time is perfectly fine, but in
> +use cases where having lots of physically contiguous memory is required, it is
> +preferable to reserve hugepages at boot time, as that will help in preventing
> +physical memory from becoming heavily fragmented.
> +
>  To reserve hugepages at boot time, a parameter is passed to the Linux kernel on the kernel command line.
>  
>  For 2 MB pages, just pass the hugepages option to the kernel. For example, to reserve 1024 pages of 2 MB, use::
> @@ -185,35 +212,27 @@ the number of hugepages reserved at boot time is generally divided equally betwe
>  
>  See the Documentation/admin-guide/kernel-parameters.txt file in your Linux source tree for further details of these and other kernel options.
>  
> -**Alternative:**
> -
> -For 2 MB pages, there is also the option of allocating hugepages after the system has booted.
> -This is done by echoing the number of hugepages required to a nr_hugepages file in the ``/sys/devices/`` directory.
> -For a single-node system, the command to use is as follows (assuming that 1024 pages are required)::
> -
> -    echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
> -
> -On a NUMA machine, pages should be allocated explicitly on separate nodes::
> -
> -    echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
> -    echo 1024 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
> -
> -.. note::
> -
> -    For 1G pages, it is not possible to reserve the hugepage memory after the system has booted.
> -
>  Using Hugepages with the DPDK
>  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>  
> -Once the hugepage memory is reserved, to make the memory available for DPDK use, perform the following steps::
> +If secondary process support is not required, DPDK is able to use hugepages
> +without any configuration by using "in-memory" mode. Please see
> +:ref:`linux_eal_parameters` for more details.
> +
> +If secondary process support is required, mount points for hugepages need to be
> +created. On modern Linux distributions, a default mount point for hugepages is provided
> +by the system and is located at ``/dev/hugepages``. This mount point will use the
> +default hugepage size set by the kernel parameters as described above.
> +
> +However, in order to use hugepage sizes other than default, it is necessary to
> +manually create mount points for hugepage sizes that are not provided by the
> +system (e.g. 1GB pages).

This reads a bit strangely, as it implies that the hugepage sizes are not
provided by the system, but I believe the intention is to say that the
mount points are not provided by the system, correct? Perhaps look to
reword.

> +
> +To make the hugepages of size 1GB available for DPDK use, perform the following steps::
>  
>      mkdir /mnt/huge
> -    mount -t hugetlbfs nodev /mnt/huge
> +    mount -t hugetlbfs pagesize=1GB /mnt/huge
>  
>  The mount point can be made permanent across reboots, by adding the following line to the ``/etc/fstab`` file::
>  
> -    nodev /mnt/huge hugetlbfs defaults 0 0
> -
> -For 1GB pages, the page size must be specified as a mount option::
> -
> -    nodev /mnt/huge_1GB hugetlbfs pagesize=1GB 0 0
> +    nodev /mnt/huge hugetlbfs pagesize=1GB 0 0
> -- 
> 2.17.1

Apart from the one note above, LGTM:

Acked-by: Bruce Richardson <bruce.richardson@intel.com>