From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1EE2944183; Fri, 7 Jun 2024 16:59:40 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CD3B242D6B; Fri, 7 Jun 2024 16:59:34 +0200 (CEST) Received: from mail-qv1-f43.google.com (mail-qv1-f43.google.com [209.85.219.43]) by mails.dpdk.org (Postfix) with ESMTP id 38CDC42D95 for ; Fri, 7 Jun 2024 16:59:33 +0200 (CEST) Received: by mail-qv1-f43.google.com with SMTP id 6a1803df08f44-6af2aff35b9so1926996d6.3 for ; Fri, 07 Jun 2024 07:59:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1717772372; x=1718377172; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Aa/5NX4WFWKQwsaEwpxBKqNnsK9djJN8Cb7edMhCbM4=; b=ftxB4xrTQm8SoVkScF1xsjWa0i9BjIBdc5q8sihOviOB3FclWyPH65mMyDNg0TbYaF RA4vpBV/0zgF7oq8F1p0feY5ordXUGU1Fa/fyg8gfLLaVjt4owPhhcFQpI51KRUA7Itu +LFFtAonRYrM268WWLi2GCmZrmm0LS363zuu4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717772372; x=1718377172; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Aa/5NX4WFWKQwsaEwpxBKqNnsK9djJN8Cb7edMhCbM4=; b=w8jAT2VTku4dhZsvVtVOtnEuduW9xhwOtGxlQGdP61R7eBvORQu7m+DwTJYRDeKN5+ bErhdOma6ipgw6MrJ6Sno+vvh0ZR+BLj8txJSirM+MYGqbfWUI9yvXK3P0RgY1Tqi2Xo ZMMTbor3ebGuXBrOcuDgIQ+f6qLQHwYgjz97KmgoFCPnmcHNeEpJh7OgiePBGAXvukKw aPLmOj6UMncCSv2Qj+SYRcoJa2RCw0lhQ395Em8firVy32PoffgUNjDya7TYXt4XpaoI PSTDQxG5GkjOlrdgSOaC+QWeHVov2+86WbJQjEYpGjJHIhZUuFjM1f7VlDOt8RU/TcT1 edMA== X-Gm-Message-State: AOJu0YyHVAV2AwhDGfCoLXc5PCpnhli77aWMvXKeMpccqRi2g0lfGg8N 8h6f5uaENM7Ybk0RAxnq1WEND57YdsV75z2RdsymjuapQVyPYZvQ5stqQodmOwM= X-Google-Smtp-Source: AGHT+IGz8lkeSjyPdE2cHgiMzg40t/0zJ99QlA4a3S0kXF26qnSSC5Z/7AchjKseqxKQX90FmznHQw== X-Received: by 2002:a05:620a:4496:b0:794:f9fc:4b0d with SMTP id af79cd13be357-7953c5c87e0mr272234185a.6.1717772372381; Fri, 07 Jun 2024 07:59:32 -0700 (PDT) Received: from localhost.unh.edu ([2606:4100:3880:1271:e2f8:4ec3:8bf3:864c]) by smtp.gmail.com with ESMTPSA id af79cd13be357-795330b2217sm174751485a.87.2024.06.07.07.59.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Jun 2024 07:59:30 -0700 (PDT) From: Nicholas Pratte To: yoan.picchi@foss.arm.com, paul.szczepanek@arm.com, luca.vizzarro@arm.com, Honnappa.Nagarahalli@arm.com, jspewock@iol.unh.edu, bruce.richardson@intel.com, juraj.linkes@pantheon.tech, probb@iol.unh.edu Cc: dev@dpdk.org, Nicholas Pratte Subject: [PATCH v9 1/2] dts: Change hugepage runtime config to 2MB Exclusively Date: Fri, 7 Jun 2024 10:59:23 -0400 Message-ID: <20240607145925.27566-2-npratte@iol.unh.edu> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240607145925.27566-1-npratte@iol.unh.edu> References: <20240430184533.29247-4-npratte@iol.unh.edu> <20240607145925.27566-1-npratte@iol.unh.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The previous implementation configures and allocates hugepage sizes based on a system default. This can lead to two problems: overallocation of hugepages (which may crash the remote host), and configuration of hugepage sizes that are not recommended during runtime. This new implementation allows only 2MB hugepage allocation during runtime; any other unique hugepage size must be configured by the end-user for initializing DTS. If the amount of 2MB hugepages requested exceeds the amount of 2MB hugepages already configured on the system, then the system will remount hugepages to cover the difference. If the amount of hugepages requested is either less than or equal to the amount already configured on the system, then nothing is done. Bugzilla ID: 1370 Signed-off-by: Nicholas Pratte Reviewed-by: Jeremy Spewock --- v4: * dts.rst punctuation/grammar corrections and 2mb exclusivity justifications included in documentation changes made to documentation for hugepages --- doc/guides/tools/dts.rst | 14 ++++++++---- dts/conf.yaml | 4 ++-- dts/framework/config/__init__.py | 4 ++-- dts/framework/config/conf_yaml_schema.json | 6 ++--- dts/framework/config/types.py | 2 +- dts/framework/testbed_model/linux_session.py | 24 +++++++++++--------- dts/framework/testbed_model/node.py | 4 +++- dts/framework/testbed_model/os_session.py | 7 +++++- 8 files changed, 39 insertions(+), 26 deletions(-) diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst index 47b218b2c6..6e6a7e1f73 100644 --- a/doc/guides/tools/dts.rst +++ b/doc/guides/tools/dts.rst @@ -131,7 +131,11 @@ There are two areas that need to be set up on a System Under Test: You may specify the optional hugepage configuration in the DTS config file. If you do, DTS will take care of configuring hugepages, - overwriting your current SUT hugepage configuration. + overwriting your current SUT hugepage configuration. Configuration of hugepages via DTS + allows only for allocation of 2MB hugepages, as doing so prevents accidental/over + allocation of hugepage sizes not recommended during runtime due to + contiguous memory space requirements. Thus, if you require hugepage + sizes not equal to 2MB, then this configuration must be done outside of the DTS framework. * System under test configuration @@ -453,11 +457,11 @@ _`Build target` **Example**: ``ccache`` ==================== ================================================================= -_`hugepages` - *mapping* – hugepages described as: +_`hugepages_2mb` + *mapping* – hugepages_2mb described as: ==================== ================================================================ - ``amount`` *integer* – The amount of hugepages to configure. + ``number_of`` *integer* – The number of 2MB hugepages to configure. Hugepage size will be the system default. ``force_first_numa`` (*optional*, defaults to ``false``) – If ``true``, it forces the @@ -570,7 +574,7 @@ involved in the testing. These can be defined with the following mappings: | | | | | The number of the memory channels to use. | +-----------------------+---------------------------------------------------------------------------------------+ - | ``hugepages`` | (*optional*) See `hugepages`_. If unset, hugepages won't be configured | + | ``hugepages_2mb`` | (*optional*) See `hugepages`_. If unset, 2MB hugepages won't be configured | | | | | | in favour of the system configuration. | +-----------------------+---------------------------------------------------------------------------------------+ diff --git a/dts/conf.yaml b/dts/conf.yaml index 8068345dd5..56c3ae6f4c 100644 --- a/dts/conf.yaml +++ b/dts/conf.yaml @@ -35,7 +35,7 @@ nodes: lcores: "" # use all the available logical cores use_first_core: false # tells DPDK to use any physical core memory_channels: 4 # tells DPDK to use 4 memory channels - hugepages: # optional; if removed, will use system hugepage configuration + hugepages_2mb: # optional; if removed, will use system hugepage configuration amount: 256 force_first_numa: false ports: @@ -71,7 +71,7 @@ nodes: os_driver: rdma peer_node: "SUT 1" peer_pci: "0000:00:08.1" - hugepages: # optional; if removed, will use system hugepage configuration + hugepages_2mb: # optional; if removed, will use system hugepage configuration amount: 256 force_first_numa: false traffic_generator: diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py index 4cb5c74059..b6f820e39e 100644 --- a/dts/framework/config/__init__.py +++ b/dts/framework/config/__init__.py @@ -255,8 +255,8 @@ def from_dict( Either an SUT or TG configuration instance. """ hugepage_config = None - if "hugepages" in d: - hugepage_config_dict = d["hugepages"] + if "hugepages_2mb" in d: + hugepage_config_dict = d["hugepages_2mb"] if "force_first_numa" not in hugepage_config_dict: hugepage_config_dict["force_first_numa"] = False hugepage_config = HugepageConfiguration(**hugepage_config_dict) diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json index 4731f4511d..f4d7199523 100644 --- a/dts/framework/config/conf_yaml_schema.json +++ b/dts/framework/config/conf_yaml_schema.json @@ -146,7 +146,7 @@ "compiler" ] }, - "hugepages": { + "hugepages_2mb": { "type": "object", "description": "Optional hugepage configuration. If not specified, hugepages won't be configured and DTS will use system configuration.", "properties": { @@ -253,8 +253,8 @@ "type": "integer", "description": "How many memory channels to use. Optional, defaults to 1." }, - "hugepages": { - "$ref": "#/definitions/hugepages" + "hugepages_2mb": { + "$ref": "#/definitions/hugepages_2mb" }, "ports": { "type": "array", diff --git a/dts/framework/config/types.py b/dts/framework/config/types.py index 1927910d88..016e0c3dbd 100644 --- a/dts/framework/config/types.py +++ b/dts/framework/config/types.py @@ -46,7 +46,7 @@ class NodeConfigDict(TypedDict): """Allowed keys and values.""" #: - hugepages: HugepageConfigurationDict + hugepages_2mb: HugepageConfigurationDict #: name: str #: diff --git a/dts/framework/testbed_model/linux_session.py b/dts/framework/testbed_model/linux_session.py index 5d24030c3d..d0f7cfa77c 100644 --- a/dts/framework/testbed_model/linux_session.py +++ b/dts/framework/testbed_model/linux_session.py @@ -15,7 +15,7 @@ from typing_extensions import NotRequired -from framework.exception import RemoteCommandExecutionError +from framework.exception import ConfigurationError, RemoteCommandExecutionError from framework.utils import expand_range from .cpu import LogicalCore @@ -84,14 +84,20 @@ def get_dpdk_file_prefix(self, dpdk_prefix: str) -> str: """Overrides :meth:`~.os_session.OSSession.get_dpdk_file_prefix`.""" return dpdk_prefix - def setup_hugepages(self, hugepage_count: int, force_first_numa: bool) -> None: + def setup_hugepages( + self, hugepage_count: int, hugepage_size: int, force_first_numa: bool + ) -> None: """Overrides :meth:`~.os_session.OSSession.setup_hugepages`.""" self._logger.info("Getting Hugepage information.") - hugepage_size = self._get_hugepage_size() - hugepages_total = self._get_hugepages_total() + hugepages_total = self._get_hugepages_total(hugepage_size) + if ( + f"hugepages-{hugepage_size}kB" + not in self.send_command("ls /sys/kernel/mm/hugepages").stdout + ): + raise ConfigurationError("hugepage size not supported by operating system") self._numa_nodes = self._get_numa_nodes() - if force_first_numa or hugepages_total != hugepage_count: + if force_first_numa or hugepages_total < hugepage_count: # when forcing numa, we need to clear existing hugepages regardless # of size, so they can be moved to the first numa node self._configure_huge_pages(hugepage_count, hugepage_size, force_first_numa) @@ -99,13 +105,9 @@ def setup_hugepages(self, hugepage_count: int, force_first_numa: bool) -> None: self._logger.info("Hugepages already configured.") self._mount_huge_pages() - def _get_hugepage_size(self) -> int: - hugepage_size = self.send_command("awk '/Hugepagesize/ {print $2}' /proc/meminfo").stdout - return int(hugepage_size) - - def _get_hugepages_total(self) -> int: + def _get_hugepages_total(self, hugepage_size: int) -> int: hugepages_total = self.send_command( - "awk '/HugePages_Total/ { print $2 }' /proc/meminfo" + f"cat /sys/kernel/mm/hugepages/hugepages-{hugepage_size}kB/nr_hugepages" ).stdout return int(hugepages_total) diff --git a/dts/framework/testbed_model/node.py b/dts/framework/testbed_model/node.py index 74061f6262..3f2a727c3b 100644 --- a/dts/framework/testbed_model/node.py +++ b/dts/framework/testbed_model/node.py @@ -266,7 +266,9 @@ def _setup_hugepages(self) -> None: """ if self.config.hugepages: self.main_session.setup_hugepages( - self.config.hugepages.amount, self.config.hugepages.force_first_numa + self.config.hugepages.amount, + self.main_session.hugepage_size, + self.config.hugepages.force_first_numa, ) def configure_port_state(self, port: Port, enable: bool = True) -> None: diff --git a/dts/framework/testbed_model/os_session.py b/dts/framework/testbed_model/os_session.py index d5bf7e0401..19dcf5f963 100644 --- a/dts/framework/testbed_model/os_session.py +++ b/dts/framework/testbed_model/os_session.py @@ -64,6 +64,7 @@ class OSSession(ABC): _logger: DTSLogger remote_session: RemoteSession interactive_session: InteractiveRemoteSession + hugepage_size: int def __init__( self, @@ -80,6 +81,7 @@ def __init__( name: The name of the session. logger: The logger instance this session will use. """ + self.hugepage_size = 2048 self._config = node_config self.name = name self._logger = logger @@ -345,7 +347,9 @@ def get_dpdk_file_prefix(self, dpdk_prefix: str) -> str: """ @abstractmethod - def setup_hugepages(self, hugepage_count: int, force_first_numa: bool) -> None: + def setup_hugepages( + self, hugepage_count: int, hugepage_size: int, force_first_numa: bool + ) -> None: """Configure hugepages on the node. Get the node's Hugepage Size, configure the specified count of hugepages @@ -353,6 +357,7 @@ def setup_hugepages(self, hugepage_count: int, force_first_numa: bool) -> None: Args: hugepage_count: Configure this many hugepages. + hugepage_size: Configure hugepages of this size. force_first_numa: If :data:`True`, configure hugepages just on the first numa node. """ -- 2.44.0