From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DB22A43C12; Fri, 1 Mar 2024 18:42:02 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CA20841611; Fri, 1 Mar 2024 18:42:02 +0100 (CET) Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by mails.dpdk.org (Postfix) with ESMTP id 6A59D402CD for ; Fri, 1 Mar 2024 18:42:00 +0100 (CET) Received: by mail-pj1-f51.google.com with SMTP id 98e67ed59e1d1-299b92948a6so1636373a91.3 for ; Fri, 01 Mar 2024 09:42:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1709314919; x=1709919719; darn=dpdk.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=h2EsiVgGBg6guJ0z9Dri2PvAlZLivHlGELqZb61AJg8=; b=fYL/scmQRe0eSXOkY+e/ezu/BgopVkqGgdzSXBvMa2cLlXQC+FjJ1oZcd7kCe3DAP+ AvTV2doaXefIyFn0EC+9f9+h2/fdlCY6SPGP4LG+N7UfpoNODGxB1vLDtHP5D0ssfpjX 4LZENKCci8HVzDFHn5cFvvaS7V7H0BsHrf5gc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709314919; x=1709919719; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=h2EsiVgGBg6guJ0z9Dri2PvAlZLivHlGELqZb61AJg8=; b=L+e/3eB14OIdGhp93wt838cX5uUcPn2IxBfwDGIureN53MbE2Hn7UPLSyUj/BOLsX4 tgYMcJJNyRXqXPr497Cy9GO7HnGnR0OfcMiBohJXpUMAKU022GKO2QpghaDLv08bFAuF jusybkCV8EGxmEMhTWvUGa8aKpg4yP2wp0OZ3CJkcEsUVaVgKVHntQzgRFtYRG++cp7j 5Nf/ux0ZgBn/glLgE9KYzJqcIE/sy3Q7I9Tz/7/lTn3BwfMKjYYFdE0qG14AHQztvZKe DNFPe5JS8brW9sHb3CZ0sSUU2nPe2s0ltZrB5VSa+cVz4jdgv85v2Qdbr+l49ZvqX2QS cDuw== X-Forwarded-Encrypted: i=1; AJvYcCXnKTjLuyYp6YrDPqkgoYI4m7JoNs84CYpmGbadr+HReD/yZ6lf9NsteJ8oxetahznlgnWSfJQHIJJRtr8= X-Gm-Message-State: AOJu0Ywwgni/QAC71ezA6JfudFYczTaNWLYHw5uxwtPiGEz+/UCLDjv/ yIGuQqdoZNQxJ0MJmsClyJwREO0Kn0yLhVhlGBnl4MpRA/UpEr9O7ibDiIJs6apcAfC6XPNaEiW Urw0SpZ78iO8HKRJVB1wkpQrgh/BHcOImuCrXmQ== X-Google-Smtp-Source: AGHT+IELZ04aJws9uP3M9a/az2JRAZKWVHNlTy/4z4IklbN8tFkUIRvP/wYaFxD2GkPsBtXqPtS2wZLGxMyM1pXFNNw= X-Received: by 2002:a17:90a:f3c1:b0:299:464c:c9fc with SMTP id ha1-20020a17090af3c100b00299464cc9fcmr1915068pjb.45.1709314919400; Fri, 01 Mar 2024 09:41:59 -0800 (PST) MIME-Version: 1.0 References: <20231220103331.60888-1-juraj.linkes@pantheon.tech> <20240301105522.79870-1-juraj.linkes@pantheon.tech> <20240301105522.79870-8-juraj.linkes@pantheon.tech> In-Reply-To: <20240301105522.79870-8-juraj.linkes@pantheon.tech> From: Jeremy Spewock Date: Fri, 1 Mar 2024 12:41:48 -0500 Message-ID: Subject: Re: [PATCH v4 7/7] dts: improve test suite and case filtering To: =?UTF-8?Q?Juraj_Linke=C5=A1?= Cc: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, probb@iol.unh.edu, paul.szczepanek@arm.com, Luca.Vizzarro@arm.com, npratte@iol.unh.edu, dev@dpdk.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org For the whole series: Reviewed-by: Jeremy Spewock On Fri, Mar 1, 2024 at 5:55=E2=80=AFAM Juraj Linke=C5=A1 wrote: > > The two places where we specify which test suite and test cases to run > are complimentary and not that intuitive to use. A unified way provides > a better user experience. > > The syntax in test run configuration file has not changed, but the > environment variable and the command line arguments was changed to match > the config file syntax. This required changes in the settings module > which greatly simplified the parsing of the environment variables while > retaining the same functionality. > > Signed-off-by: Juraj Linke=C5=A1 > --- > doc/guides/tools/dts.rst | 14 ++- > dts/framework/config/__init__.py | 12 +- > dts/framework/runner.py | 18 +-- > dts/framework/settings.py | 187 ++++++++++++++----------------- > dts/framework/test_suite.py | 2 +- > 5 files changed, 114 insertions(+), 119 deletions(-) > > diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst > index f686ca487c..d1c3c2af7a 100644 > --- a/doc/guides/tools/dts.rst > +++ b/doc/guides/tools/dts.rst > @@ -215,28 +215,30 @@ DTS is run with ``main.py`` located in the ``dts`` = directory after entering Poet > .. code-block:: console > > (dts-py3.10) $ ./main.py --help > - usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_= DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_= TIMEOUT] [--test-cases TEST_CASES] [--re-run RE_RUN] > + usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_= DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_= TIMEOUT] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run RE_RUN] > > Run DPDK test suites. All options may be specified with the environme= nt variables provided in brackets. Command line arguments have higher prior= ity. > > options: > -h, --help show this help message and exit > --config-file CONFIG_FILE > - [DTS_CFG_FILE] configuration file that describe= s the test cases, SUTs and targets. (default: ./conf.yaml) > + [DTS_CFG_FILE] The configuration file that desc= ribes the test cases, SUTs and targets. (default: ./conf.yaml) > --output-dir OUTPUT_DIR, --output OUTPUT_DIR > [DTS_OUTPUT_DIR] Output directory where DTS log= s and results are saved. (default: output) > -t TIMEOUT, --timeout TIMEOUT > [DTS_TIMEOUT] The default timeout for all DTS o= perations except for compiling DPDK. (default: 15) > -v, --verbose [DTS_VERBOSE] Specify to enable verbose output,= logging all messages to the console. (default: False) > - -s, --skip-setup [DTS_SKIP_SETUP] Specify to skip all setup step= s on SUT and TG nodes. (default: None) > + -s, --skip-setup [DTS_SKIP_SETUP] Specify to skip all setup step= s on SUT and TG nodes. (default: False) > --tarball TARBALL, --snapshot TARBALL, --git-ref TARBALL > [DTS_DPDK_TARBALL] Path to DPDK source code tar= ball or a git commit ID, tag ID or tree ID to test. To test local changes, = first commit them, then use the commit ID with this option. (default: dpdk.= tar.xz) > --compile-timeout COMPILE_TIMEOUT > [DTS_COMPILE_TIMEOUT] The timeout for compiling= DPDK. (default: 1200) > - --test-cases TEST_CASES > - [DTS_TESTCASES] Comma-separated list of test ca= ses to execute. Unknown test cases will be silently ignored. (default: ) > + --test-suite TEST_SUITE [TEST_CASES ...] > + [DTS_TEST_SUITES] A list containing a test suit= e with test cases. The first parameter is the test suite name, and the rest= are test case names, which are optional. May be specified multiple times. = To specify multiple test suites in the environment > + variable, join the lists with a comma. Examples= : --test-suite suite case case --test-suite suite case ... | DTS_TEST_SUITE= S=3D'suite case case, suite case, ...' | --test-suite suite --test-suite su= ite case ... | DTS_TEST_SUITES=3D'suite, suite case, ...' > + (default: []) > --re-run RE_RUN, --re_run RE_RUN > - [DTS_RERUN] Re-run each test case the specified= number of times if a test failure occurs (default: 0) > + [DTS_RERUN] Re-run each test case the specified= number of times if a test failure occurs. (default: 0) > > > The brackets contain the names of environment variables that set the sam= e thing. > diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__in= it__.py > index c6a93b3b89..4cb5c74059 100644 > --- a/dts/framework/config/__init__.py > +++ b/dts/framework/config/__init__.py > @@ -35,9 +35,9 @@ > > import json > import os.path > -import pathlib > from dataclasses import dataclass, fields > from enum import auto, unique > +from pathlib import Path > from typing import Union > > import warlock # type: ignore[import] > @@ -53,7 +53,6 @@ > TrafficGeneratorConfigDict, > ) > from framework.exception import ConfigurationError > -from framework.settings import SETTINGS > from framework.utils import StrEnum > > > @@ -571,7 +570,7 @@ def from_dict(d: ConfigurationDict) -> "Configuration= ": > return Configuration(executions=3Dexecutions) > > > -def load_config() -> Configuration: > +def load_config(config_file_path: Path) -> Configuration: > """Load DTS test run configuration from a file. > > Load the YAML test run configuration file > @@ -581,13 +580,16 @@ def load_config() -> Configuration: > The YAML test run configuration file is specified in the :option:`--= config-file` command line > argument or the :envvar:`DTS_CFG_FILE` environment variable. > > + Args: > + config_file_path: The path to the YAML test run configuration fi= le. > + > Returns: > The parsed test run configuration. > """ > - with open(SETTINGS.config_file_path, "r") as f: > + with open(config_file_path, "r") as f: > config_data =3D yaml.safe_load(f) > > - schema_path =3D os.path.join(pathlib.Path(__file__).parent.resolve()= , "conf_yaml_schema.json") > + schema_path =3D os.path.join(Path(__file__).parent.resolve(), "conf_= yaml_schema.json") > > with open(schema_path, "r") as f: > schema =3D json.load(f) > diff --git a/dts/framework/runner.py b/dts/framework/runner.py > index dfee8ebd7c..db8e3ba96b 100644 > --- a/dts/framework/runner.py > +++ b/dts/framework/runner.py > @@ -24,7 +24,7 @@ > import sys > from pathlib import Path > from types import MethodType > -from typing import Iterable > +from typing import Iterable, Sequence > > from .config import ( > BuildTargetConfiguration, > @@ -83,7 +83,7 @@ class DTSRunner: > > def __init__(self): > """Initialize the instance with configuration, logger, result an= d string constants.""" > - self._configuration =3D load_config() > + self._configuration =3D load_config(SETTINGS.config_file_path) > self._logger =3D get_dts_logger() > if not os.path.exists(SETTINGS.output_dir): > os.makedirs(SETTINGS.output_dir) > @@ -129,7 +129,7 @@ def run(self): > #. Execution teardown > > The test cases are filtered according to the specification in th= e test run configuration and > - the :option:`--test-cases` command line argument or > + the :option:`--test-suite` command line argument or > the :envvar:`DTS_TESTCASES` environment variable. > """ > sut_nodes: dict[str, SutNode] =3D {} > @@ -147,7 +147,9 @@ def run(self): > ) > execution_result =3D self._result.add_execution(executio= n) > # we don't want to modify the original config, so create= a copy > - execution_test_suites =3D list(execution.test_suites) > + execution_test_suites =3D list( > + SETTINGS.test_suites if SETTINGS.test_suites else ex= ecution.test_suites > + ) > if not execution.skip_smoke_tests: > execution_test_suites[:0] =3D [TestSuiteConfig.from_= dict("smoke_tests")] > try: > @@ -226,7 +228,7 @@ def _get_test_suites_with_cases( > test_suite_class =3D self._get_test_suite_class(test_suite_c= onfig.test_suite) > test_cases =3D [] > func_test_cases, perf_test_cases =3D self._filter_test_cases= ( > - test_suite_class, set(test_suite_config.test_cases + SET= TINGS.test_cases) > + test_suite_class, test_suite_config.test_cases > ) > if func: > test_cases.extend(func_test_cases) > @@ -302,7 +304,7 @@ def is_test_suite(object) -> bool: > ) > > def _filter_test_cases( > - self, test_suite_class: type[TestSuite], test_cases_to_run: set[= str] > + self, test_suite_class: type[TestSuite], test_cases_to_run: Sequ= ence[str] > ) -> tuple[list[MethodType], list[MethodType]]: > """Filter `test_cases_to_run` from `test_suite_class`. > > @@ -331,7 +333,9 @@ def _filter_test_cases( > (name, method) for name, method in name_method_tuples if= name in test_cases_to_run > ] > if len(name_method_tuples) < len(test_cases_to_run): > - missing_test_cases =3D test_cases_to_run - {name for nam= e, _ in name_method_tuples} > + missing_test_cases =3D set(test_cases_to_run) - { > + name for name, _ in name_method_tuples > + } > raise ConfigurationError( > f"Test cases {missing_test_cases} not found among me= thods " > f"of {test_suite_class.__name__}." > diff --git a/dts/framework/settings.py b/dts/framework/settings.py > index 2b8bfbe0ed..688e8679a7 100644 > --- a/dts/framework/settings.py > +++ b/dts/framework/settings.py > @@ -48,10 +48,11 @@ > > The path to a DPDK tarball, git commit ID, tag ID or tree ID to test= . > > -.. option:: --test-cases > -.. envvar:: DTS_TESTCASES > +.. option:: --test-suite > +.. envvar:: DTS_TEST_SUITES > > - A comma-separated list of test cases to execute. Unknown test cases = will be silently ignored. > + A test suite with test cases which may be specified multiple tim= es. > + In the environment variable, the suites are joined with a comma. > > .. option:: --re-run, --re_run > .. envvar:: DTS_RERUN > @@ -71,83 +72,13 @@ > > import argparse > import os > -from collections.abc import Callable, Iterable, Sequence > from dataclasses import dataclass, field > from pathlib import Path > -from typing import Any, TypeVar > +from typing import Any > > +from .config import TestSuiteConfig > from .utils import DPDKGitTarball > > -_T =3D TypeVar("_T") > - > - > -def _env_arg(env_var: str) -> Any: > - """A helper method augmenting the argparse Action with environment v= ariables. > - > - If the supplied environment variable is defined, then the default va= lue > - of the argument is modified. This satisfies the priority order of > - command line argument > environment variable > default value. > - > - Arguments with no values (flags) should be defined using the const k= eyword argument > - (True or False). When the argument is specified, it will be set to c= onst, if not specified, > - the default will be stored (possibly modified by the corresponding e= nvironment variable). > - > - Other arguments work the same as default argparse arguments, that is= using > - the default 'store' action. > - > - Returns: > - The modified argparse.Action. > - """ > - > - class _EnvironmentArgument(argparse.Action): > - def __init__( > - self, > - option_strings: Sequence[str], > - dest: str, > - nargs: str | int | None =3D None, > - const: bool | None =3D None, > - default: Any =3D None, > - type: Callable[[str], _T | argparse.FileType | None] =3D Non= e, > - choices: Iterable[_T] | None =3D None, > - required: bool =3D False, > - help: str | None =3D None, > - metavar: str | tuple[str, ...] | None =3D None, > - ) -> None: > - env_var_value =3D os.environ.get(env_var) > - default =3D env_var_value or default > - if const is not None: > - nargs =3D 0 > - default =3D const if env_var_value else default > - type =3D None > - choices =3D None > - metavar =3D None > - super(_EnvironmentArgument, self).__init__( > - option_strings, > - dest, > - nargs=3Dnargs, > - const=3Dconst, > - default=3Ddefault, > - type=3Dtype, > - choices=3Dchoices, > - required=3Drequired, > - help=3Dhelp, > - metavar=3Dmetavar, > - ) > - > - def __call__( > - self, > - parser: argparse.ArgumentParser, > - namespace: argparse.Namespace, > - values: Any, > - option_string: str =3D None, > - ) -> None: > - if self.const is not None: > - setattr(namespace, self.dest, self.const) > - else: > - setattr(namespace, self.dest, values) > - > - return _EnvironmentArgument > - > > @dataclass(slots=3DTrue) > class Settings: > @@ -171,7 +102,7 @@ class Settings: > #: > compile_timeout: float =3D 1200 > #: > - test_cases: list[str] =3D field(default_factory=3Dlist) > + test_suites: list[TestSuiteConfig] =3D field(default_factory=3Dlist) > #: > re_run: int =3D 0 > > @@ -180,6 +111,31 @@ class Settings: > > > def _get_parser() -> argparse.ArgumentParser: > + """Create the argument parser for DTS. > + > + Command line options take precedence over environment variables, whi= ch in turn take precedence > + over default values. > + > + Returns: > + argparse.ArgumentParser: The configured argument parser with def= ined options. > + """ > + > + def env_arg(env_var: str, default: Any) -> Any: > + """A helper function augmenting the argparse with environment va= riables. > + > + If the supplied environment variable is defined, then the defaul= t value > + of the argument is modified. This satisfies the priority order o= f > + command line argument > environment variable > default value. > + > + Args: > + env_var: Environment variable name. > + default: Default value. > + > + Returns: > + Environment variable or default value. > + """ > + return os.environ.get(env_var) or default > + > parser =3D argparse.ArgumentParser( > description=3D"Run DPDK test suites. All options may be specifie= d with the environment " > "variables provided in brackets. Command line arguments have hig= her priority.", > @@ -188,25 +144,23 @@ def _get_parser() -> argparse.ArgumentParser: > > parser.add_argument( > "--config-file", > - action=3D_env_arg("DTS_CFG_FILE"), > - default=3DSETTINGS.config_file_path, > + default=3Denv_arg("DTS_CFG_FILE", SETTINGS.config_file_path), > type=3DPath, > - help=3D"[DTS_CFG_FILE] configuration file that describes the tes= t cases, SUTs and targets.", > + help=3D"[DTS_CFG_FILE] The configuration file that describes the= test cases, " > + "SUTs and targets.", > ) > > parser.add_argument( > "--output-dir", > "--output", > - action=3D_env_arg("DTS_OUTPUT_DIR"), > - default=3DSETTINGS.output_dir, > + default=3Denv_arg("DTS_OUTPUT_DIR", SETTINGS.output_dir), > help=3D"[DTS_OUTPUT_DIR] Output directory where DTS logs and res= ults are saved.", > ) > > parser.add_argument( > "-t", > "--timeout", > - action=3D_env_arg("DTS_TIMEOUT"), > - default=3DSETTINGS.timeout, > + default=3Denv_arg("DTS_TIMEOUT", SETTINGS.timeout), > type=3Dfloat, > help=3D"[DTS_TIMEOUT] The default timeout for all DTS operations= except for compiling DPDK.", > ) > @@ -214,9 +168,8 @@ def _get_parser() -> argparse.ArgumentParser: > parser.add_argument( > "-v", > "--verbose", > - action=3D_env_arg("DTS_VERBOSE"), > - default=3DSETTINGS.verbose, > - const=3DTrue, > + action=3D"store_true", > + default=3Denv_arg("DTS_VERBOSE", SETTINGS.verbose), > help=3D"[DTS_VERBOSE] Specify to enable verbose output, logging = all messages " > "to the console.", > ) > @@ -224,8 +177,8 @@ def _get_parser() -> argparse.ArgumentParser: > parser.add_argument( > "-s", > "--skip-setup", > - action=3D_env_arg("DTS_SKIP_SETUP"), > - const=3DTrue, > + action=3D"store_true", > + default=3Denv_arg("DTS_SKIP_SETUP", SETTINGS.skip_setup), > help=3D"[DTS_SKIP_SETUP] Specify to skip all setup steps on SUT = and TG nodes.", > ) > > @@ -233,8 +186,7 @@ def _get_parser() -> argparse.ArgumentParser: > "--tarball", > "--snapshot", > "--git-ref", > - action=3D_env_arg("DTS_DPDK_TARBALL"), > - default=3DSETTINGS.dpdk_tarball_path, > + default=3Denv_arg("DTS_DPDK_TARBALL", SETTINGS.dpdk_tarball_path= ), > type=3DPath, > help=3D"[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a= git commit ID, " > "tag ID or tree ID to test. To test local changes, first commit = them, " > @@ -243,36 +195,71 @@ def _get_parser() -> argparse.ArgumentParser: > > parser.add_argument( > "--compile-timeout", > - action=3D_env_arg("DTS_COMPILE_TIMEOUT"), > - default=3DSETTINGS.compile_timeout, > + default=3Denv_arg("DTS_COMPILE_TIMEOUT", SETTINGS.compile_timeou= t), > type=3Dfloat, > help=3D"[DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK.", > ) > > parser.add_argument( > - "--test-cases", > - action=3D_env_arg("DTS_TESTCASES"), > - default=3D"", > - help=3D"[DTS_TESTCASES] Comma-separated list of test cases to ex= ecute.", > + "--test-suite", > + action=3D"append", > + nargs=3D"+", > + metavar=3D("TEST_SUITE", "TEST_CASES"), > + default=3Denv_arg("DTS_TEST_SUITES", SETTINGS.test_suites), > + help=3D"[DTS_TEST_SUITES] A list containing a test suite with te= st cases. " > + "The first parameter is the test suite name, and the rest are te= st case names, " > + "which are optional. May be specified multiple times. To specify= multiple test suites in " > + "the environment variable, join the lists with a comma. " > + "Examples: " > + "--test-suite suite case case --test-suite suite case ... | " > + "DTS_TEST_SUITES=3D'suite case case, suite case, ...' | " > + "--test-suite suite --test-suite suite case ... | " > + "DTS_TEST_SUITES=3D'suite, suite case, ...'", > ) > > parser.add_argument( > "--re-run", > "--re_run", > - action=3D_env_arg("DTS_RERUN"), > - default=3DSETTINGS.re_run, > + default=3Denv_arg("DTS_RERUN", SETTINGS.re_run), > type=3Dint, > help=3D"[DTS_RERUN] Re-run each test case the specified number o= f times " > - "if a test failure occurs", > + "if a test failure occurs.", > ) > > return parser > > > +def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteC= onfig]: > + """Process the given argument to a list of :class:`TestSuiteConfig` = to execute. > + > + Args: > + args: The arguments to process. The args is a string from an env= ironment variable > + or a list of from the user input containing tests suites w= ith tests cases, > + each of which is a list of [test_suite, test_case, test_ca= se, ...]. > + > + Returns: > + A list of test suite configurations to execute. > + """ > + if isinstance(args, str): > + # Environment variable in the form of "suite case case, suite ca= se, suite, ..." > + args =3D [suite_with_cases.split() for suite_with_cases in args.= split(",")] > + > + test_suites_to_run =3D [] > + for suite_with_cases in args: > + test_suites_to_run.append( > + TestSuiteConfig(test_suite=3Dsuite_with_cases[0], test_cases= =3Dsuite_with_cases[1:]) > + ) > + > + return test_suites_to_run > + > + > def get_settings() -> Settings: > """Create new settings with inputs from the user. > > The inputs are taken from the command line and from environment vari= ables. > + > + Returns: > + The new settings object. > """ > parsed_args =3D _get_parser().parse_args() > return Settings( > @@ -287,6 +274,6 @@ def get_settings() -> Settings: > else Path(parsed_args.tarball) > ), > compile_timeout=3Dparsed_args.compile_timeout, > - test_cases=3D(parsed_args.test_cases.split(",") if parsed_args.t= est_cases else []), > + test_suites=3D_process_test_suites(parsed_args.test_suite), > re_run=3Dparsed_args.re_run, > ) > diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py > index 365f80e21a..1957ea7328 100644 > --- a/dts/framework/test_suite.py > +++ b/dts/framework/test_suite.py > @@ -40,7 +40,7 @@ class TestSuite(object): > and functional test cases (all other test cases). > > By default, all test cases will be executed. A list of testcase name= s may be specified > - in the YAML test run configuration file and in the :option:`--test-c= ases` command line argument > + in the YAML test run configuration file and in the :option:`--test-s= uite` command line argument > or in the :envvar:`DTS_TESTCASES` environment variable to filter whi= ch test cases to run. > The union of both lists will be used. Any unknown test cases from th= e latter lists > will be silently ignored. > -- > 2.34.1 >