From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C4F7B43E87; Tue, 16 Apr 2024 20:26:35 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AB8C640268; Tue, 16 Apr 2024 20:26:35 +0200 (CEST) Received: from mail-lj1-f172.google.com (mail-lj1-f172.google.com [209.85.208.172]) by mails.dpdk.org (Postfix) with ESMTP id 56C9140262 for ; Tue, 16 Apr 2024 20:26:34 +0200 (CEST) Received: by mail-lj1-f172.google.com with SMTP id 38308e7fff4ca-2da4e54549fso2607701fa.1 for ; Tue, 16 Apr 2024 11:26:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1713291994; x=1713896794; darn=dpdk.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=uiNqkIZkSK1faR+tkZ59ai3pRrwHJGxdmDnNDfLMco8=; b=cfaw6ZL3HNA0Wwt2Pc1ZtvVf48ESaNlv2csb1/QMtmZDN7bbOAce6yZ2Rr/Dt8WTda JjKMYm0Al/2B2Hc5p0UIh2olGNCpVv6jmyRR5FBbegFEf6jaT9l7K3znvVfmA1Ivmcqu DV68e6QLW3D5SkDd3n5dkwN3z2vqlqIqAJOrY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713291994; x=1713896794; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uiNqkIZkSK1faR+tkZ59ai3pRrwHJGxdmDnNDfLMco8=; b=bMRyidgDVrwxfkAxadiMnuKrwfJ0RCH/dvJzRt4Q5HiDu0S5yjeMeiZRixLisF3aDY BCjq0TVFn4BbVUPQCqqfLR9GLdL6J8z+eQObRQULj+4cDzZ6My4whp7UnqEB/0EvhxWt zz/CbPnl73xopI3wgiES1cdjzVqvPx3dqKe1HTcwxadNkabPFf+iB30qOQSgbDgU6I0X AwfFz6ONzWR8FNJ7Xxz+Ddpa7NUreKVdederqW4hC/Zdyn/0ngaQOb6L2k8hUh28V/n2 oSjgnvl5yrwcf57mU61I0F5F5/soMUPgsPEWgTpWLfROrNRVquEcXMs3XiMwiQ1LyvPN fkDw== X-Gm-Message-State: AOJu0YxA6GSYZU+0/dM4ub+kUsgVEKBZrVwwoytRRY3hPGSKWb/4k8a3 /qoIF8kTrgmZaEKtsSjhdsDmf8kakk9UOelGdkZL8MKtGTVX+q3o7fLHhMWZK8ldNQwpxVxonX6 m3c7/gyNSFdub+sj4A7QKAsPpB/gmIvJ0FXHXoQ== X-Google-Smtp-Source: AGHT+IH0Mlu4GYILdVhlbC2bqk8ctO4DpBFRjYcZPhCm7wPjmaRAdL5gkY9W0LOrM42zwTNpyef4CcTTuGu/BeeC77M= X-Received: by 2002:a2e:81c3:0:b0:2d8:80a5:76d4 with SMTP id s3-20020a2e81c3000000b002d880a576d4mr8503340ljg.2.1713291993621; Tue, 16 Apr 2024 11:26:33 -0700 (PDT) MIME-Version: 1.0 References: <20240404153106.19047-1-npratte@iol.unh.edu> <20240416181840.11539-1-npratte@iol.unh.edu> In-Reply-To: <20240416181840.11539-1-npratte@iol.unh.edu> From: Nicholas Pratte Date: Tue, 16 Apr 2024 14:26:22 -0400 Message-ID: Subject: Re: [PATCH v3] dts: Change hugepage runtime config to 2MB Exclusively To: paul.szczepanek@arm.com, wathsala.vithanage@arm.com, bruce.richardson@intel.com, juraj.linkes@pantheon.tech, yoan.picchi@foss.arm.com, mb@smartsharesystems.com, jspewock@iol.unh.edu, Honnappa.Nagarahalli@arm.com, thomas@monjalon.net, probb@iol.unh.edu Cc: dev@dpdk.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org I like the changes suggested by Morten, and much such modifications accordingly. Given that the hugepage size will likely be determined by arch going forward, looking ahead, when considering patch 1360 and ongoing config changes, I decided to create a property method in the Node object that can be modified to our liking in the future (credit to Jeremy for this idea). This solution is designed in such a way that, when arch is discovered and removed from the config, hugepage sizes can be determined right after discovering the arch (this is just to save me time in the future when working on patch 1360). I also opted to throw the size variable back into the methods under linux_session.py (per Morten's suggestions), but given the implementation Jeremy and I worked on, there is no need to have the get_hugepage_size() method in linux_session.py as we likely won't need it going forward, so this has been removed. On Tue, Apr 16, 2024 at 2:18=E2=80=AFPM Nicholas Pratte wrote: > > The previous implementation configures and allocates hugepage sizes > based on a system default. This can lead to two problems: overallocation = of > hugepages (which may crash the remote host), and configuration of hugepag= e > sizes that are not recommended during runtime. This new implementation > allows only 2MB hugepage allocation during runtime; any other unique > hugepage size must be configured by the end-user for initializing DTS. > > If the amount of 2MB hugepages requested exceeds the amount of 2MB > hugepages already configured on the system, then the system will remount > hugepages to cover the difference. If the amount of hugepages requested i= s > either less than or equal to the amount already configured on the system, > then nothing is done. > > Bugzilla ID: 1370 > Signed-off-by: Nicholas Pratte > Reviewed-by: Jeremy Spewock > --- > doc/guides/tools/dts.rst | 7 +++++- > dts/conf.yaml | 4 ++-- > dts/framework/config/__init__.py | 4 ++-- > dts/framework/config/conf_yaml_schema.json | 6 ++--- > dts/framework/config/types.py | 2 +- > dts/framework/testbed_model/linux_session.py | 24 +++++++++++--------- > dts/framework/testbed_model/node.py | 8 ++++++- > dts/framework/testbed_model/os_session.py | 4 +++- > 8 files changed, 37 insertions(+), 22 deletions(-) > > diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst > index 47b218b2c6..1103db0faa 100644 > --- a/doc/guides/tools/dts.rst > +++ b/doc/guides/tools/dts.rst > @@ -131,7 +131,12 @@ There are two areas that need to be set up on a Syst= em Under Test: > > You may specify the optional hugepage configuration in the DTS conf= ig file. > If you do, DTS will take care of configuring hugepages, > - overwriting your current SUT hugepage configuration. > + overwriting your current SUT hugepage configuration. Configuration = of hugepages via DTS > + allows only for configuration of 2MB hugepages. Thus, if your needs= require hugepage > + sizes not equal to 2MB, then you these configurations must be done = outside of the DTS > + framework; moreover, if you do not desire the use of 2MB hugepages,= and instead perfer > + other sizes (e.g 1GB), then we assume that hugepages have been manu= ally configued before > + runtime. > > * System under test configuration > > diff --git a/dts/conf.yaml b/dts/conf.yaml > index 8068345dd5..56c3ae6f4c 100644 > --- a/dts/conf.yaml > +++ b/dts/conf.yaml > @@ -35,7 +35,7 @@ nodes: > lcores: "" # use all the available logical cores > use_first_core: false # tells DPDK to use any physical core > memory_channels: 4 # tells DPDK to use 4 memory channels > - hugepages: # optional; if removed, will use system hugepage configu= ration > + hugepages_2mb: # optional; if removed, will use system hugepage conf= iguration > amount: 256 > force_first_numa: false > ports: > @@ -71,7 +71,7 @@ nodes: > os_driver: rdma > peer_node: "SUT 1" > peer_pci: "0000:00:08.1" > - hugepages: # optional; if removed, will use system hugepage configu= ration > + hugepages_2mb: # optional; if removed, will use system hugepage conf= iguration > amount: 256 > force_first_numa: false > traffic_generator: > diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__in= it__.py > index 4cb5c74059..b6f820e39e 100644 > --- a/dts/framework/config/__init__.py > +++ b/dts/framework/config/__init__.py > @@ -255,8 +255,8 @@ def from_dict( > Either an SUT or TG configuration instance. > """ > hugepage_config =3D None > - if "hugepages" in d: > - hugepage_config_dict =3D d["hugepages"] > + if "hugepages_2mb" in d: > + hugepage_config_dict =3D d["hugepages_2mb"] > if "force_first_numa" not in hugepage_config_dict: > hugepage_config_dict["force_first_numa"] =3D False > hugepage_config =3D HugepageConfiguration(**hugepage_config_= dict) > diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/c= onfig/conf_yaml_schema.json > index 4731f4511d..f4d7199523 100644 > --- a/dts/framework/config/conf_yaml_schema.json > +++ b/dts/framework/config/conf_yaml_schema.json > @@ -146,7 +146,7 @@ > "compiler" > ] > }, > - "hugepages": { > + "hugepages_2mb": { > "type": "object", > "description": "Optional hugepage configuration. If not specified,= hugepages won't be configured and DTS will use system configuration.", > "properties": { > @@ -253,8 +253,8 @@ > "type": "integer", > "description": "How many memory channels to use. Optional, d= efaults to 1." > }, > - "hugepages": { > - "$ref": "#/definitions/hugepages" > + "hugepages_2mb": { > + "$ref": "#/definitions/hugepages_2mb" > }, > "ports": { > "type": "array", > diff --git a/dts/framework/config/types.py b/dts/framework/config/types.p= y > index 1927910d88..016e0c3dbd 100644 > --- a/dts/framework/config/types.py > +++ b/dts/framework/config/types.py > @@ -46,7 +46,7 @@ class NodeConfigDict(TypedDict): > """Allowed keys and values.""" > > #: > - hugepages: HugepageConfigurationDict > + hugepages_2mb: HugepageConfigurationDict > #: > name: str > #: > diff --git a/dts/framework/testbed_model/linux_session.py b/dts/framework= /testbed_model/linux_session.py > index 5d24030c3d..d0f7cfa77c 100644 > --- a/dts/framework/testbed_model/linux_session.py > +++ b/dts/framework/testbed_model/linux_session.py > @@ -15,7 +15,7 @@ > > from typing_extensions import NotRequired > > -from framework.exception import RemoteCommandExecutionError > +from framework.exception import ConfigurationError, RemoteCommandExecuti= onError > from framework.utils import expand_range > > from .cpu import LogicalCore > @@ -84,14 +84,20 @@ def get_dpdk_file_prefix(self, dpdk_prefix: str) -> s= tr: > """Overrides :meth:`~.os_session.OSSession.get_dpdk_file_prefix`= .""" > return dpdk_prefix > > - def setup_hugepages(self, hugepage_count: int, force_first_numa: boo= l) -> None: > + def setup_hugepages( > + self, hugepage_count: int, hugepage_size: int, force_first_numa:= bool > + ) -> None: > """Overrides :meth:`~.os_session.OSSession.setup_hugepages`.""" > self._logger.info("Getting Hugepage information.") > - hugepage_size =3D self._get_hugepage_size() > - hugepages_total =3D self._get_hugepages_total() > + hugepages_total =3D self._get_hugepages_total(hugepage_size) > + if ( > + f"hugepages-{hugepage_size}kB" > + not in self.send_command("ls /sys/kernel/mm/hugepages").stdo= ut > + ): > + raise ConfigurationError("hugepage size not supported by ope= rating system") > self._numa_nodes =3D self._get_numa_nodes() > > - if force_first_numa or hugepages_total !=3D hugepage_count: > + if force_first_numa or hugepages_total < hugepage_count: > # when forcing numa, we need to clear existing hugepages reg= ardless > # of size, so they can be moved to the first numa node > self._configure_huge_pages(hugepage_count, hugepage_size, fo= rce_first_numa) > @@ -99,13 +105,9 @@ def setup_hugepages(self, hugepage_count: int, force_= first_numa: bool) -> None: > self._logger.info("Hugepages already configured.") > self._mount_huge_pages() > > - def _get_hugepage_size(self) -> int: > - hugepage_size =3D self.send_command("awk '/Hugepagesize/ {print = $2}' /proc/meminfo").stdout > - return int(hugepage_size) > - > - def _get_hugepages_total(self) -> int: > + def _get_hugepages_total(self, hugepage_size: int) -> int: > hugepages_total =3D self.send_command( > - "awk '/HugePages_Total/ { print $2 }' /proc/meminfo" > + f"cat /sys/kernel/mm/hugepages/hugepages-{hugepage_size}kB/n= r_hugepages" > ).stdout > return int(hugepages_total) > > diff --git a/dts/framework/testbed_model/node.py b/dts/framework/testbed_= model/node.py > index 74061f6262..056a031ca0 100644 > --- a/dts/framework/testbed_model/node.py > +++ b/dts/framework/testbed_model/node.py > @@ -97,6 +97,10 @@ def __init__(self, node_config: NodeConfiguration): > self.virtual_devices =3D [] > self._init_ports() > > + @property > + def _hugepage_default_size(self) -> int: > + return 2048 > + > def _init_ports(self) -> None: > self.ports =3D [Port(self.name, port_config) for port_config in = self.config.ports] > self.main_session.update_ports(self.ports) > @@ -266,7 +270,9 @@ def _setup_hugepages(self) -> None: > """ > if self.config.hugepages: > self.main_session.setup_hugepages( > - self.config.hugepages.amount, self.config.hugepages.forc= e_first_numa > + self.config.hugepages.amount, > + self._hugepage_default_size, > + self.config.hugepages.force_first_numa, > ) > > def configure_port_state(self, port: Port, enable: bool =3D True) ->= None: > diff --git a/dts/framework/testbed_model/os_session.py b/dts/framework/te= stbed_model/os_session.py > index d5bf7e0401..69f9483f05 100644 > --- a/dts/framework/testbed_model/os_session.py > +++ b/dts/framework/testbed_model/os_session.py > @@ -345,7 +345,9 @@ def get_dpdk_file_prefix(self, dpdk_prefix: str) -> s= tr: > """ > > @abstractmethod > - def setup_hugepages(self, hugepage_count: int, force_first_numa: boo= l) -> None: > + def setup_hugepages( > + self, hugepage_count: int, hugepage_size: int, force_first_numa:= bool > + ) -> None: > """Configure hugepages on the node. > > Get the node's Hugepage Size, configure the specified count of h= ugepages > -- > 2.44.0 >