From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0157AA04B1; Tue, 25 Aug 2020 15:57:51 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 43C8D2C18; Tue, 25 Aug 2020 15:57:41 +0200 (CEST) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id 7FC5823D; Tue, 25 Aug 2020 15:57:38 +0200 (CEST) IronPort-SDR: G2mm7xGdYK1KINTARUfa+kOZ0R2XEsz5U2GSshaLLgwrdpHDGMKraXNfHEhlvf1EYpnCCG4iru FkvufDkGjMgw== X-IronPort-AV: E=McAfee;i="6000,8403,9723"; a="156101582" X-IronPort-AV: E=Sophos;i="5.76,352,1592895600"; d="scan'208";a="156101582" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Aug 2020 06:57:37 -0700 IronPort-SDR: hhPzCDRzZdtG9BMW9g718T0KLO8PwWc47xzOZQZeegb9cjw+/9bzDqwIynUob9aat1KjvB3rKE gXlpCKTPaW6w== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.76,352,1592895600"; d="scan'208";a="443637061" Received: from silpixa00399498.ir.intel.com (HELO silpixa00399498.ger.corp.intel.com) ([10.237.222.52]) by orsmga004.jf.intel.com with ESMTP; 25 Aug 2020 06:57:36 -0700 From: Anatoly Burakov To: dev@dpdk.org Cc: John McNamara , Marko Kovacevic , ferruh.yigit@intel.com, bruce.richardson@intel.com, padraig.j.connolly@intel.com, stable@dpdk.org Date: Tue, 25 Aug 2020 14:57:35 +0100 Message-Id: <6f93d916bbceb67e09083a2be05f23a572a0d614.1598363848.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <242515b3b0d4ac57ee86cada96af90fb78e14997.1598363848.git.anatoly.burakov@intel.com> References: <242515b3b0d4ac57ee86cada96af90fb78e14997.1598363848.git.anatoly.burakov@intel.com> In-Reply-To: <196e97d2802cf2250577aaa113b9093b0beadb3d.1598357863.git.anatoly.burakov@intel.com> References: <196e97d2802cf2250577aaa113b9093b0beadb3d.1598357863.git.anatoly.burakov@intel.com> Subject: [dpdk-dev] [PATCH v3 2/2] doc/linux_gsg: update information on using hugepages X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Current information regarding hugepage usage is a little out of date. Update it to include information on in-memory mode, as well as on default mountpoints provided by systemd. Cc: stable@dpdk.org Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v3: - Clarified wording around non-default hugepage sizes v2: - Reworked the description - Put runtime reservation first, and boot time as an alternative - Clarified wording and fixed typos - Mentioned that some kernel versions not supporting reserving 1G pages doc/guides/linux_gsg/sys_reqs.rst | 70 +++++++++++++++++++------------ 1 file changed, 44 insertions(+), 26 deletions(-) diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst index a124656bcb..587f9e85e5 100644 --- a/doc/guides/linux_gsg/sys_reqs.rst +++ b/doc/guides/linux_gsg/sys_reqs.rst @@ -155,8 +155,35 @@ Without hugepages, high TLB miss rates would occur with the standard 4k page siz Reserving Hugepages for DPDK Use ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The allocation of hugepages should be done at boot time or as soon as possible after system boot -to prevent memory from being fragmented in physical memory. +The reservation of hugepages can be performed at run time. This is done by +echoing the number of hugepages required to a ``nr_hugepages`` file in the +``/sys/kernel/`` directory corresponding to a specific page size (in +Kilobytes). For a single-node system, the command to use is as follows +(assuming that 1024 of 2MB pages are required):: + + echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages + +On a NUMA machine, the above command will usually divide the number of hugepages +equally across all NUMA nodes (assuming there is enough memory on all NUMA +nodes). However, pages can also be reserved explicitly on individual NUMA +nodes using a ``nr_hugepages`` file in the ``/sys/devices/`` directory:: + + echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages + echo 1024 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages + +.. note:: + + Some kernel versions may not allow reserving 1 GB hugepages at run time, so + reserving them at boot time may be the only option. Please see below for + instructions. + +**Alternative:** + +In the general case, reserving hugepages at run time is perfectly fine, but in +use cases where having lots of physically contiguous memory is required, it is +preferable to reserve hugepages at boot time, as that will help in preventing +physical memory from becoming heavily fragmented. + To reserve hugepages at boot time, a parameter is passed to the Linux kernel on the kernel command line. For 2 MB pages, just pass the hugepages option to the kernel. For example, to reserve 1024 pages of 2 MB, use:: @@ -185,35 +212,26 @@ the number of hugepages reserved at boot time is generally divided equally betwe See the Documentation/admin-guide/kernel-parameters.txt file in your Linux source tree for further details of these and other kernel options. -**Alternative:** - -For 2 MB pages, there is also the option of allocating hugepages after the system has booted. -This is done by echoing the number of hugepages required to a nr_hugepages file in the ``/sys/devices/`` directory. -For a single-node system, the command to use is as follows (assuming that 1024 pages are required):: - - echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages - -On a NUMA machine, pages should be allocated explicitly on separate nodes:: - - echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages - echo 1024 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages - -.. note:: - - For 1G pages, it is not possible to reserve the hugepage memory after the system has booted. - Using Hugepages with the DPDK ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Once the hugepage memory is reserved, to make the memory available for DPDK use, perform the following steps:: +If secondary process support is not required, DPDK is able to use hugepages +without any configuration by using "in-memory" mode. Please see +:ref:`linux_eal_parameters` for more details. + +If secondary process support is required, mount points for hugepages need to be +created. On modern Linux distributions, a default mount point for hugepages is provided +by the system and is located at ``/dev/hugepages``. This mount point will use the +default hugepage size set by the kernel parameters as described above. + +However, in order to use hugepage sizes other than the default, it is necessary +to manually create mount points for those hugepage sizes (e.g. 1GB pages). + +To make the hugepages of size 1GB available for DPDK use, perform the following steps:: mkdir /mnt/huge - mount -t hugetlbfs nodev /mnt/huge + mount -t hugetlbfs pagesize=1GB /mnt/huge The mount point can be made permanent across reboots, by adding the following line to the ``/etc/fstab`` file:: - nodev /mnt/huge hugetlbfs defaults 0 0 - -For 1GB pages, the page size must be specified as a mount option:: - - nodev /mnt/huge_1GB hugetlbfs pagesize=1GB 0 0 + nodev /mnt/huge hugetlbfs pagesize=1GB 0 0 -- 2.17.1