DPDK patches and discussions
 help / color / mirror / Atom feed
* [PATCH v5 1/2] dts: Change hugepage runtime config to 2MB Exclusively
@ 2024-04-30 18:45 Nicholas Pratte
  2024-04-30 18:45 ` [PATCH v5 2/2] dts: Change hugepage 'amount' to a different term Nicholas Pratte
                   ` (2 more replies)
  0 siblings, 3 replies; 45+ messages in thread
From: Nicholas Pratte @ 2024-04-30 18:45 UTC (permalink / raw)
  To: jspewock@iol.unh.edumb@smartsharesystems.combruce.richardson,
	yoan.picchi, juraj.linkes, paul.szczepanek, wathsala.vithanage,
	thomas, Honnappa.Nagarahalli, probb
  Cc: dev, Nicholas Pratte, Jeremy Spewock

The previous implementation configures and allocates hugepage sizes
based on a system default. This can lead to two problems: overallocation of
hugepages (which may crash the remote host), and configuration of hugepage
sizes that are not recommended during runtime. This new implementation
allows only 2MB hugepage allocation during runtime; any other unique
hugepage size must be configured by the end-user for initializing DTS.

If the amount of 2MB hugepages requested exceeds the amount of 2MB
hugepages already configured on the system, then the system will remount
hugepages to cover the difference. If the amount of hugepages requested is
either less than or equal to the amount already configured on the system,
then nothing is done.

Bugzilla ID: 1370
Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
---

v4:
 * dts.rst punctuation/grammar corrections and 2mb exclusivity
   justifications included in documentation
---
 doc/guides/tools/dts.rst                     |  6 ++++-
 dts/conf.yaml                                |  4 ++--
 dts/framework/config/__init__.py             |  4 ++--
 dts/framework/config/conf_yaml_schema.json   |  6 ++---
 dts/framework/config/types.py                |  2 +-
 dts/framework/testbed_model/linux_session.py | 24 +++++++++++---------
 dts/framework/testbed_model/node.py          |  4 +++-
 dts/framework/testbed_model/os_session.py    |  7 +++++-
 8 files changed, 35 insertions(+), 22 deletions(-)

diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index 47b218b2c6..40c3ddafb6 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -131,7 +131,11 @@ There are two areas that need to be set up on a System Under Test:
 
      You may specify the optional hugepage configuration in the DTS config file.
      If you do, DTS will take care of configuring hugepages,
-     overwriting your current SUT hugepage configuration.
+     overwriting your current SUT hugepage configuration. Configuration of hugepages via DTS
+     allows only for allocation of 2MB hugepages, as doing so prevents accidental/over
+     allocation of hugepage sizes not recommended during runtime due to
+     contiguous memory space requirements. Thus, if you require hugepage
+     sizes not equal to 2MB, then this configuration must be done outside of the DTS framework.
 
    * System under test configuration
 
diff --git a/dts/conf.yaml b/dts/conf.yaml
index 8068345dd5..56c3ae6f4c 100644
--- a/dts/conf.yaml
+++ b/dts/conf.yaml
@@ -35,7 +35,7 @@ nodes:
     lcores: "" # use all the available logical cores
     use_first_core: false # tells DPDK to use any physical core
     memory_channels: 4 # tells DPDK to use 4 memory channels
-    hugepages:  # optional; if removed, will use system hugepage configuration
+    hugepages_2mb: # optional; if removed, will use system hugepage configuration
         amount: 256
         force_first_numa: false
     ports:
@@ -71,7 +71,7 @@ nodes:
         os_driver: rdma
         peer_node: "SUT 1"
         peer_pci: "0000:00:08.1"
-    hugepages:  # optional; if removed, will use system hugepage configuration
+    hugepages_2mb: # optional; if removed, will use system hugepage configuration
         amount: 256
         force_first_numa: false
     traffic_generator:
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 4cb5c74059..b6f820e39e 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -255,8 +255,8 @@ def from_dict(
             Either an SUT or TG configuration instance.
         """
         hugepage_config = None
-        if "hugepages" in d:
-            hugepage_config_dict = d["hugepages"]
+        if "hugepages_2mb" in d:
+            hugepage_config_dict = d["hugepages_2mb"]
             if "force_first_numa" not in hugepage_config_dict:
                 hugepage_config_dict["force_first_numa"] = False
             hugepage_config = HugepageConfiguration(**hugepage_config_dict)
diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index 4731f4511d..f4d7199523 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -146,7 +146,7 @@
         "compiler"
       ]
     },
-    "hugepages": {
+    "hugepages_2mb": {
       "type": "object",
       "description": "Optional hugepage configuration. If not specified, hugepages won't be configured and DTS will use system configuration.",
       "properties": {
@@ -253,8 +253,8 @@
             "type": "integer",
             "description": "How many memory channels to use. Optional, defaults to 1."
           },
-          "hugepages": {
-            "$ref": "#/definitions/hugepages"
+          "hugepages_2mb": {
+            "$ref": "#/definitions/hugepages_2mb"
           },
           "ports": {
             "type": "array",
diff --git a/dts/framework/config/types.py b/dts/framework/config/types.py
index 1927910d88..016e0c3dbd 100644
--- a/dts/framework/config/types.py
+++ b/dts/framework/config/types.py
@@ -46,7 +46,7 @@ class NodeConfigDict(TypedDict):
     """Allowed keys and values."""
 
     #:
-    hugepages: HugepageConfigurationDict
+    hugepages_2mb: HugepageConfigurationDict
     #:
     name: str
     #:
diff --git a/dts/framework/testbed_model/linux_session.py b/dts/framework/testbed_model/linux_session.py
index 5d24030c3d..d0f7cfa77c 100644
--- a/dts/framework/testbed_model/linux_session.py
+++ b/dts/framework/testbed_model/linux_session.py
@@ -15,7 +15,7 @@
 
 from typing_extensions import NotRequired
 
-from framework.exception import RemoteCommandExecutionError
+from framework.exception import ConfigurationError, RemoteCommandExecutionError
 from framework.utils import expand_range
 
 from .cpu import LogicalCore
@@ -84,14 +84,20 @@ def get_dpdk_file_prefix(self, dpdk_prefix: str) -> str:
         """Overrides :meth:`~.os_session.OSSession.get_dpdk_file_prefix`."""
         return dpdk_prefix
 
-    def setup_hugepages(self, hugepage_count: int, force_first_numa: bool) -> None:
+    def setup_hugepages(
+        self, hugepage_count: int, hugepage_size: int, force_first_numa: bool
+    ) -> None:
         """Overrides :meth:`~.os_session.OSSession.setup_hugepages`."""
         self._logger.info("Getting Hugepage information.")
-        hugepage_size = self._get_hugepage_size()
-        hugepages_total = self._get_hugepages_total()
+        hugepages_total = self._get_hugepages_total(hugepage_size)
+        if (
+            f"hugepages-{hugepage_size}kB"
+            not in self.send_command("ls /sys/kernel/mm/hugepages").stdout
+        ):
+            raise ConfigurationError("hugepage size not supported by operating system")
         self._numa_nodes = self._get_numa_nodes()
 
-        if force_first_numa or hugepages_total != hugepage_count:
+        if force_first_numa or hugepages_total < hugepage_count:
             # when forcing numa, we need to clear existing hugepages regardless
             # of size, so they can be moved to the first numa node
             self._configure_huge_pages(hugepage_count, hugepage_size, force_first_numa)
@@ -99,13 +105,9 @@ def setup_hugepages(self, hugepage_count: int, force_first_numa: bool) -> None:
             self._logger.info("Hugepages already configured.")
         self._mount_huge_pages()
 
-    def _get_hugepage_size(self) -> int:
-        hugepage_size = self.send_command("awk '/Hugepagesize/ {print $2}' /proc/meminfo").stdout
-        return int(hugepage_size)
-
-    def _get_hugepages_total(self) -> int:
+    def _get_hugepages_total(self, hugepage_size: int) -> int:
         hugepages_total = self.send_command(
-            "awk '/HugePages_Total/ { print $2 }' /proc/meminfo"
+            f"cat /sys/kernel/mm/hugepages/hugepages-{hugepage_size}kB/nr_hugepages"
         ).stdout
         return int(hugepages_total)
 
diff --git a/dts/framework/testbed_model/node.py b/dts/framework/testbed_model/node.py
index 74061f6262..3f2a727c3b 100644
--- a/dts/framework/testbed_model/node.py
+++ b/dts/framework/testbed_model/node.py
@@ -266,7 +266,9 @@ def _setup_hugepages(self) -> None:
         """
         if self.config.hugepages:
             self.main_session.setup_hugepages(
-                self.config.hugepages.amount, self.config.hugepages.force_first_numa
+                self.config.hugepages.amount,
+                self.main_session.hugepage_size,
+                self.config.hugepages.force_first_numa,
             )
 
     def configure_port_state(self, port: Port, enable: bool = True) -> None:
diff --git a/dts/framework/testbed_model/os_session.py b/dts/framework/testbed_model/os_session.py
index d5bf7e0401..19dcf5f963 100644
--- a/dts/framework/testbed_model/os_session.py
+++ b/dts/framework/testbed_model/os_session.py
@@ -64,6 +64,7 @@ class OSSession(ABC):
     _logger: DTSLogger
     remote_session: RemoteSession
     interactive_session: InteractiveRemoteSession
+    hugepage_size: int
 
     def __init__(
         self,
@@ -80,6 +81,7 @@ def __init__(
             name: The name of the session.
             logger: The logger instance this session will use.
         """
+        self.hugepage_size = 2048
         self._config = node_config
         self.name = name
         self._logger = logger
@@ -345,7 +347,9 @@ def get_dpdk_file_prefix(self, dpdk_prefix: str) -> str:
         """
 
     @abstractmethod
-    def setup_hugepages(self, hugepage_count: int, force_first_numa: bool) -> None:
+    def setup_hugepages(
+        self, hugepage_count: int, hugepage_size: int, force_first_numa: bool
+    ) -> None:
         """Configure hugepages on the node.
 
         Get the node's Hugepage Size, configure the specified count of hugepages
@@ -353,6 +357,7 @@ def setup_hugepages(self, hugepage_count: int, force_first_numa: bool) -> None:
 
         Args:
             hugepage_count: Configure this many hugepages.
+            hugepage_size: Configure hugepages of this size.
             force_first_numa:  If :data:`True`, configure hugepages just on the first numa node.
         """
 
-- 
2.44.0


^ permalink raw reply	[flat|nested] 45+ messages in thread
* [PATCH v9 0/2] Methodology change for hugepage configuration
@ 2024-06-10 15:20 Nicholas Pratte
  2024-06-20  2:06 ` Thomas Monjalon
  0 siblings, 1 reply; 45+ messages in thread
From: Nicholas Pratte @ 2024-06-10 15:20 UTC (permalink / raw)
  To: luca.vizzarro, jspewock, paul.szczepanek, Honnappa.Nagarahalli,
	yoan.picchi, juraj.linkes, probb, bruce.richardson
  Cc: dev, Nicholas Pratte

In order to prevent accidental misconfiguration of hugepages at runtime,
the following changes are made to only allow for configuration of 2MB
hugepages within the DTS config.yaml. In the previous implementation, a
default hugepage size was selected via the size listed in /proc/meminfo.
The problem with this implementation is that, assuming the end-user has
made prior modifications to the system, /proc/meminfo may default to
hugepage sizes that are not recommended to be configured at runtime
(i.e. 1GB hugepages). This can lead to two problems: overallocation of
hugepages (which may crash the remote host) configuration of hugepages
sizes that are not recommended during runtime. In this new implementation,
we stipulate that any runtime hugepage configuration size that is not 2MB
is considered an outlier. If the end-user would like to configure either
1GB hugepages or any unique hugepage size outside of 2MB, then they should
make these configurations either at startup (in the case of 1GB hugepages)
or runtime outside of DTS configuration (if a user would like hugepages
that are not 2MB). In either case, the expectation is that, if wish to
use hugepage sizes that are not 2MB, you will make these changes outside
and prior to the initialization of DTS.

The end-user has two options: remove the option for hugepage
configuration in the conf.yaml, or keep the option and specify the
amount of 2MB hugepages desired. In the case of the former, then we assume
that hugepages are already configured prior to DTS initialization. In
the latter case, the user must define the amount of 2MB hugepages to be
configured at runtime. If the amount of 2MB hugepages requested exceeds
the amount of 2MB hugepages already configured on the system, then the
system will remount hugepages to cover the difference. If the amount of
hugepages requested is either greater than or equal to the amount
already configured on the system, then nothing is done.

Nicholas Pratte (2):
  dts: Change hugepage runtime config to 2MB Exclusively
  dts: Change hugepage 'amount' to a different term

 doc/guides/tools/dts.rst                     | 14 ++++++----
 dts/conf.yaml                                |  8 +++---
 dts/framework/config/__init__.py             |  8 +++---
 dts/framework/config/conf_yaml_schema.json   | 12 ++++-----
 dts/framework/config/types.py                |  4 +--
 dts/framework/testbed_model/linux_session.py | 28 ++++++++++----------
 dts/framework/testbed_model/node.py          |  4 ++-
 dts/framework/testbed_model/os_session.py    |  9 ++++---
 8 files changed, 48 insertions(+), 39 deletions(-)

-- 
2.44.0


^ permalink raw reply	[flat|nested] 45+ messages in thread

end of thread, other threads:[~2024-06-20  2:06 UTC | newest]

Thread overview: 45+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-04-30 18:45 [PATCH v5 1/2] dts: Change hugepage runtime config to 2MB Exclusively Nicholas Pratte
2024-04-30 18:45 ` [PATCH v5 2/2] dts: Change hugepage 'amount' to a different term Nicholas Pratte
2024-05-02 11:55   ` Juraj Linkeš
2024-05-07 11:15   ` Luca Vizzarro
2024-05-07 12:05   ` Bruce Richardson
2024-05-07 12:43     ` Luca Vizzarro
2024-05-07 13:00       ` Bruce Richardson
2024-05-13 10:06         ` Juraj Linkeš
2024-05-15 15:12           ` Nicholas Pratte
2024-04-30 18:45 ` [PATCH v5 0/2] Methodology change for hugepage configuration Nicholas Pratte
2024-05-07 11:16   ` Luca Vizzarro
2024-05-07 16:37   ` [PATCH v6 " Nicholas Pratte
2024-05-07 16:37     ` [PATCH v6 1/2] dts: Change hugepage runtime config to 2MB Exclusively Nicholas Pratte
2024-05-07 16:37     ` [PATCH v6 2/2] dts: Change hugepage 'amount' to a different term Nicholas Pratte
2024-05-07 17:44   ` [PATCH v6 0/2] Methodology change for hugepage configuration Nicholas Pratte
2024-05-07 17:44     ` [PATCH v6 1/2] dts: Change hugepage runtime config to 2MB Exclusively Nicholas Pratte
2024-05-07 17:44     ` [PATCH v6 2/2] dts: Change hugepage 'amount' to a different term Nicholas Pratte
2024-05-07 17:46     ` [PATCH v6 0/2] Methodology change for hugepage configuration Nicholas Pratte
2024-05-13  9:53     ` Juraj Linkeš
2024-05-15 14:50       ` Nicholas Pratte
2024-05-29 10:30         ` Juraj Linkeš
2024-05-30 17:58   ` [PATCH v7 " Nicholas Pratte
2024-05-30 17:58     ` [PATCH v7 1/2] dts: Change hugepage runtime config to 2MB Exclusively Nicholas Pratte
2024-05-30 17:58     ` [PATCH v7 2/2] dts: Change hugepage 'amount' to a different term Nicholas Pratte
2024-05-30 18:37   ` [PATCH v8 0/2] Methodology change for hugepage configuration Nicholas Pratte
2024-05-30 19:37   ` Nicholas Pratte
2024-05-30 19:37     ` [PATCH v8 1/2] dts: Change hugepage runtime config to 2MB Exclusively Nicholas Pratte
2024-05-31  6:52       ` Juraj Linkeš
2024-05-31 11:37       ` Luca Vizzarro
2024-05-30 19:38     ` [PATCH v8 2/2] dts: Change hugepage 'amount' to a different term Nicholas Pratte
2024-05-31  6:53       ` Juraj Linkeš
2024-05-31 11:37       ` Luca Vizzarro
2024-06-07 14:31   ` [PATCH v9 0/2] Methodology change for hugepage configuration Nicholas Pratte
2024-06-07 14:31   ` [PATCH v9 1/2] dts: Change hugepage runtime config to 2MB Exclusively Nicholas Pratte
2024-06-07 14:31   ` [PATCH v9 2/2] dts: Change hugepage 'amount' to a different term Nicholas Pratte
2024-06-07 14:59   ` [PATCH v9 0/2] Methodology change for hugepage configuration Nicholas Pratte
2024-06-07 14:59     ` [PATCH v9 1/2] dts: Change hugepage runtime config to 2MB Exclusively Nicholas Pratte
2024-06-10  7:38       ` Juraj Linkeš
2024-06-17 14:59       ` Luca Vizzarro
2024-06-07 14:59     ` [PATCH v9 2/2] dts: Change hugepage 'amount' to a different term Nicholas Pratte
2024-06-10  7:38       ` Juraj Linkeš
2024-06-17 14:59       ` Luca Vizzarro
2024-05-02 11:55 ` [PATCH v5 1/2] dts: Change hugepage runtime config to 2MB Exclusively Juraj Linkeš
2024-06-10 15:20 [PATCH v9 0/2] Methodology change for hugepage configuration Nicholas Pratte
2024-06-20  2:06 ` Thomas Monjalon

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).