From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7A021459FB; Fri, 27 Sep 2024 19:45:42 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 10D2B4028C; Fri, 27 Sep 2024 19:45:42 +0200 (CEST) Received: from mail-pg1-f172.google.com (mail-pg1-f172.google.com [209.85.215.172]) by mails.dpdk.org (Postfix) with ESMTP id 92DB14025D for ; Fri, 27 Sep 2024 19:45:40 +0200 (CEST) Received: by mail-pg1-f172.google.com with SMTP id 41be03b00d2f7-7e6cbf6cd1dso1769607a12.3 for ; Fri, 27 Sep 2024 10:45:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1727459139; x=1728063939; darn=dpdk.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=2NvjhzYMhvpuAUOyQJ1JWYd64b0RqbI8bUOaKTnWFso=; b=EOfn+8aNcBMdq/i5OSXxoeGZfhQeaiMn8l6fLdRIhzQ5LsdwWQo2IjPWV+5ldaVN+j f5hMm8ofSms2VAoX57A08MOMOzF5BqCqlbwtBfa1+SLFTwqjIZpxSQvAf0gBk/udji6J +i6XnGndmvyw1PiVLOBtGJxeQq4qnye5Kd1aU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727459139; x=1728063939; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2NvjhzYMhvpuAUOyQJ1JWYd64b0RqbI8bUOaKTnWFso=; b=xDYs2RoQvJZPdO8ns3u3QeToCIsNZzKT1+WxpS5Wa8vVOn0bwN2pY7pW75XF6Q/uVQ WSxPXNS02W/V3uTfpN0XeZwdXyGCJ2LL+n9/3JrhJFao0Qe3YHZPPP1clBSukRHp4aCR EdeNoSZ7DnpwODy6drj8HXhEZIHGhSpl2KCrEk6PbrrzCnhD1CdedXD5pt5X9GEmVIAT yzZ36K/esl89sFq9+UyETs2g9IsC+4k4qb9DA0otxwEcP+bfmwVkAr0iGQmrKnMC0+VK uCnURdSfmgpCKBTmBAP9eZywULU0k8sgprp8xfB9uPzuva6Lsn6532D0gWIlurfXM5GU ofzQ== X-Gm-Message-State: AOJu0YxhBo4qw8PG68iXgwLnQKZGxTY0u6Rc0G0hMmCAvpe30kqBErOe ZMkUFc4AN3IgcfPGh31CT7xvuDXR9gMBMUXR7HbmLA/oi9n+hg0hbfieHIPeT4iz0RdZ0VXyh1j bxug2rrQLfZb8Wy20yvMN+/QTOCAN4eicqoHXsg== X-Google-Smtp-Source: AGHT+IFfQ/teSIShpxuL5kzlnLejivB7pCTnc9RiLtd68nKKpoFovfPuP00ax+1aBqOFwhh4Knc5JM/PX42GEmjeQoQ= X-Received: by 2002:a17:90a:e2c4:b0:2d8:7804:b3a with SMTP id 98e67ed59e1d1-2e0b8ea775amr4217293a91.26.1727459139258; Fri, 27 Sep 2024 10:45:39 -0700 (PDT) MIME-Version: 1.0 References: <20240906161355.701688-1-luca.vizzarro@arm.com> In-Reply-To: <20240906161355.701688-1-luca.vizzarro@arm.com> From: Jeremy Spewock Date: Fri, 27 Sep 2024 13:45:27 -0400 Message-ID: Subject: Re: [PATCH] dts: add per-test-suite configuration To: Luca Vizzarro Cc: dev@dpdk.org, Honnappa Nagarahalli , Paul Szczepanek , Alex Chapman , =?UTF-8?Q?Juraj_Linke=C5=A1?= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Luca, I apologize for not giving this patch more time/attention. I think this patch is worth having a more in-depth discussion about since it does add quite a bit more complexity, but the benefit of that is some valuable simplicity. I know we discussed in the recent team meeting about whether or not to have the test suite configurations in conf.yaml or make ones specifically for each individual test suite, and I think that it could really go either way, but that them being in conf.yaml is potentially slightly better in my opinion as it helps keep things uniform, but obviously you get into trouble there if there is a lot of configuration and the file gets long. On Fri, Sep 6, 2024 at 12:14=E2=80=AFPM Luca Vizzarro wrote: > > Allow test suites to be configured individually. Moreover enable them to > implement their own custom configuration. > > This solution adds some new complexity to DTS, which is generated source > code. In order to ensure strong typing, the test suites and their custom Generated source code is definitely something new and interesting. I think the way you did it makes sense and it's well written, but the idea of generated source code scares me slightly. It leads to the sort of additional requirement when you are writing test suites to regenerate the code if you have a custom config, and seems a little harder to maintain (although, I doubt much maintenance will really be needed on it, so maybe that's not a good point). It makes me wonder if the very dumb approach of test suites just each getting a yaml file that matches the same of the test suite (and maybe ends with _conf) and then importing that into a TypedDict like we do with conf.yaml would be worth it to save on this complexity. It would still be simple enough for a test suite developer to just throw a yaml file into a directory (maybe conf/ ?) and have the framework auto-magically consume that and match it to a TypedDict. I guess there would be some extra rules in place with the dumb approach like the name of the config file and maybe the name of the TypedDict, but it still feels like it would save on complexity in the long run. Of course, that's not to say complexity is bad if it makes things easier, but it begs the question of how much easier is doing it this way versus making a yaml file and a matching TypedDict. Again, something good to discuss in a meeting. > configurations need to be linked in the main configuration class. > Unfortunately, this is not feasible during runtime as it will incur in > circular dependencies. Generating the links appear to be the most > straightforward approach. > > This commit also brings a new major change to the configuration schema. > Test suites are no longer defined as a list of strings, like: > > test_suites: > - hello_world > - pmd_buffer_scatter > > but as mapping of mappings or strings: > > test_suites: > hello_world: {} # any custom fields or test cases can be set here > pmd_buffer_scatter: all # "all" defines all the test cases, or > # they can individually be set separated > # by a space > > Not defining the `test_cases` field in the configuration is equivalent > to `all`, therefore the definitions for either test suite above are > also equivalent. Making these mappings in general however that also allow you to specify which test cases to run I really like. It doesn't seem completely relevant to the suite-wide configurations though, maybe it would make sense to be in a different patch? I get why it's all in one though since it they are handled in the same places. > > Creating the __init__.py file under the tests folder, allows it to be > picked up as a package. This is a mypy requirement to import the tests > from within the framework. > > Bugzilla ID: 1375 > > Signed-off-by: Luca Vizzarro > Reviewed-by: Paul Szczepanek > Reviewed-by: Alex Chapman I didn't really have any comments on this implementation overall however. I didn't fully understand everything (mostly due to my lack of pydantic understanding) so I probably would have had to read it over a few more times to completely understand it, but in general I think this seems well written. Great work Luca. > --- > Depends-on: series-32823 ("dts: Pydantic configuration") > > Hello, > > sending in a solution for the per-test-suite configuration issue. > This one took some thinking but I have given most of the motivations in > the commit body already. The docs are somewhat lacking but hopefully they > should be automatically tackled by the API docs generation. > > > Best, > Luca > --- > doc/guides/tools/dts.rst | 39 ++++-- > dts/conf.yaml | 4 +- > dts/framework/config/__init__.py | 98 ++------------- > dts/framework/config/conf_yaml_schema.json | 94 +++++++++++--- > dts/framework/config/generated.py | 40 ++++++ > dts/framework/config/test_suite.py | 140 +++++++++++++++++++++ > dts/framework/runner.py | 59 +++++++-- > dts/framework/settings.py | 29 +++-- > dts/framework/test_result.py | 12 +- > dts/framework/test_suite.py | 32 ++++- > dts/generate-schema.py | 4 +- > dts/generate-test-mappings.py | 132 +++++++++++++++++++ > dts/tests/TestSuite_hello_world.py | 12 +- > dts/tests/__init__.py | 7 ++ > 14 files changed, 539 insertions(+), 163 deletions(-) > create mode 100644 dts/framework/config/generated.py > create mode 100644 dts/framework/config/test_suite.py > create mode 100755 dts/generate-test-mappings.py > create mode 100644 dts/tests/__init__.py > > diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst > index 317bd0ff99..66681543cd 100644 > --- a/doc/guides/tools/dts.rst > +++ b/doc/guides/tools/dts.rst > @@ -391,6 +391,26 @@ There are four types of methods that comprise a test= suite: > should be implemented in the ``SutNode`` class (and the underlying cl= asses that ``SutNode`` uses) > and used by the test suite via the ``sut_node`` field. > > +The test suites can also implement their own custom configuration fields= . This can be achieved by > +creating a new test suite config file which inherits from ``TestSuiteCon= fig`` defined in > +``dts/framework/config/test_suite.py``. So that this new custom configur= ation class is used, the > +test suite class must override the ``config`` attribute annotation with = your new class, for example:: > + > +.. code:: python > + class CustomConfig(TestSuiteConfig): > + my_custom_field: int =3D 10 > + > + class TestMyNewTestSuite(TestSuite): > + config: CustomConfig > + > +Finally, the test suites and the custom configuration files need to link= ed in the global configuration. > +This can be easily achieved by running the ``dts/generate-test-mappings.= py``, e.g.: > + > +.. code-block:: console > + > + $ poetry shell > + (dts-py3.10) $ ./generate-test-mappings.py > + > > .. _dts_dev_tools: > > @@ -510,18 +530,13 @@ _`Network port` > ``peer_pci`` *string* =E2=80=93 the PCI address of the peer= node port. **Example**: ``000a:01:00.1`` > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D > > -_`Test suite` > - *string* =E2=80=93 name of the test suite to run. **Examples**: ``hel= lo_world``, ``os_udp`` > - > -_`Test target` > - *mapping* =E2=80=93 selects specific test cases to run from a test su= ite. Mapping is described as follows: > - > - =3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D > - ``suite`` See `Test suite`_ > - ``cases`` (*optional*) *sequence* of *string* =E2=80=93 list of the s= elected test cases in the test suite to run. > +_`Test suites` > + *mapping* =E2=80=93 selects the test suites to run. Each mapping key = corresponds to the test suite name. > > - Unknown test cases will be silently ignored. > - =3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D > + The value of the mapping can either "all" to select all the test case= s in that test suite, the test > + cases names divided by a space. Or it can be another mapping to set a= ny custom fields for the test suite. > + In the case of a mapping, all the test cases are selected by default.= In order to manually select test > + cases, the ``test_cases`` field can be set with a list of strings, ea= ch entry being a test case name. > > > Properties > @@ -542,7 +557,7 @@ involved in the testing. These can be defined with th= e following mappings: > +----------------------------+---------------------------------------= ----------------------------+ > | ``func`` | *boolean* =E2=80=93 Enable functional = testing. | > +----------------------------+---------------------------------------= ----------------------------+ > - | ``test_suites`` | *sequence* of **one of** `Test suite`_= **or** `Test target`_ | > + | ``test_suites`` | See `Test suites`_ = | > +----------------------------+---------------------------------------= ----------------------------+ > | ``skip_smoke_tests`` | (*optional*) *boolean* =E2=80=93 Allow= s you to skip smoke testing | > | | if ``true``. = | > diff --git a/dts/conf.yaml b/dts/conf.yaml > index 7d95016e68..c44bef604c 100644 > --- a/dts/conf.yaml > +++ b/dts/conf.yaml > @@ -15,8 +15,8 @@ test_runs: > func: true # enable functional testing > skip_smoke_tests: false # optional > test_suites: # the following test suites will be run in their entire= ty > - - hello_world > - - os_udp > + hello_world: all > + os_udp: all > # The machine running the DPDK test executable > system_under_test_node: > node_name: "SUT 1" > diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__in= it__.py > index 013c529829..d39e0823bd 100644 > --- a/dts/framework/config/__init__.py > +++ b/dts/framework/config/__init__.py > @@ -35,10 +35,12 @@ > and makes it thread safe should we ever want to move in that direc= tion. > """ > > +# pylama:ignore=3DW0611 > + > from enum import Enum, auto, unique > from functools import cached_property > from pathlib import Path > -from typing import TYPE_CHECKING, Annotated, Any, Literal, NamedTuple, P= rotocol > +from typing import Annotated, Literal, NamedTuple, Protocol > > import yaml > from pydantic import ( > @@ -50,15 +52,14 @@ > field_validator, > model_validator, > ) > -from pydantic.config import JsonDict > from pydantic.dataclasses import dataclass > from typing_extensions import Self > > from framework.exception import ConfigurationError > from framework.utils import StrEnum > > -if TYPE_CHECKING: > - from framework.test_suite import TestSuiteSpec > +from .generated import CUSTOM_CONFIG_TYPES, TestSuitesConfigs > +from .test_suite import TestSuiteConfig > > > @unique > @@ -289,7 +290,7 @@ class NodeInfo: > kernel_version: str > > > -@dataclass(slots=3DTrue, frozen=3DTrue, kw_only=3DTrue, config=3DConfigD= ict(extra=3D"forbid")) > +@dataclass(frozen=3DTrue, kw_only=3DTrue, config=3DConfigDict(extra=3D"f= orbid")) > class BuildTargetConfiguration: > """DPDK build configuration. > > @@ -329,89 +330,6 @@ class BuildTargetInfo: > compiler_version: str > > > -def make_parsable_schema(schema: JsonDict): > - """Updates a model's JSON schema to make a string representation a v= alid alternative. > - > - This utility function is required to be used with models that can be= represented and validated > - as a string instead of an object mapping. Normally the generated JSO= N schema will just show > - the object mapping. This function wraps the mapping under an anyOf p= roperty sequenced with a > - string type. > - > - This function is a valid `Callable` for the `json_schema_extra` attr= ibute of > - `~pydantic.config.ConfigDict`. > - """ > - inner_schema =3D schema.copy() > - del inner_schema["title"] > - > - title =3D schema.get("title") > - description =3D schema.get("description") > - > - schema.clear() > - > - schema["title"] =3D title > - schema["description"] =3D description > - schema["anyOf"] =3D [inner_schema, {"type": "string"}] > - > - > -@dataclass( > - frozen=3DTrue, > - config=3DConfigDict(extra=3D"forbid", json_schema_extra=3Dmake_parsa= ble_schema), > -) > -class TestSuiteConfig: > - """Test suite configuration. > - > - Information about a single test suite to be executed. It can be repr= esented and validated as a > - string type in the form of: ``TEST_SUITE [TEST_CASE, ...]``, in the = configuration file. > - > - Attributes: > - test_suite: The name of the test suite module without the starti= ng ``TestSuite_``. > - test_cases: The names of test cases from this test suite to exec= ute. > - If empty, all test cases will be executed. > - """ > - > - test_suite_name: str =3D Field( > - title=3D"Test suite name", > - description=3D"The identifying name of the test suite.", > - alias=3D"test_suite", > - ) > - test_cases_names: list[str] =3D Field( > - default_factory=3Dlist, > - title=3D"Test cases by name", > - description=3D"The identifying name of the test cases of the tes= t suite.", > - alias=3D"test_cases", > - ) > - > - @cached_property > - def test_suite_spec(self) -> "TestSuiteSpec": > - """The specification of the requested test suite.""" > - from framework.test_suite import find_by_name > - > - test_suite_spec =3D find_by_name(self.test_suite_name) > - assert test_suite_spec is not None, f"{self.test_suite_name} is = not a valid test suite name" > - return test_suite_spec > - > - @model_validator(mode=3D"before") > - @classmethod > - def convert_from_string(cls, data: Any) -> Any: > - """Convert the string representation into a valid mapping.""" > - if isinstance(data, str): > - [test_suite, *test_cases] =3D data.split() > - return dict(test_suite=3Dtest_suite, test_cases=3Dtest_cases= ) > - return data > - > - @model_validator(mode=3D"after") > - def validate_names(self) -> Self: > - """Validate the supplied test suite and test cases names.""" > - available_test_cases =3D map(lambda t: t.name, self.test_suite_s= pec.test_cases) > - for requested_test_case in self.test_cases_names: > - assert requested_test_case in available_test_cases, ( > - f"{requested_test_case} is not a valid test case " > - f"for test suite {self.test_suite_name}" > - ) > - > - return self > - > - > @dataclass(slots=3DTrue, frozen=3DTrue, kw_only=3DTrue, config=3DConfigD= ict(extra=3D"forbid")) > class TestRunSUTNodeConfiguration: > """The SUT node configuration of a test run. > @@ -446,7 +364,7 @@ class TestRunConfiguration: > perf: bool =3D Field(description=3D"Enable performance testing.") > func: bool =3D Field(description=3D"Enable functional testing.") > skip_smoke_tests: bool =3D False > - test_suites: list[TestSuiteConfig] =3D Field(min_length=3D1) > + test_suites: TestSuitesConfigs > system_under_test_node: TestRunSUTNodeConfiguration > traffic_generator_node: str > > @@ -581,7 +499,7 @@ def load_config(config_file_path: Path) -> Configurat= ion: > config_data =3D yaml.safe_load(f) > > try: > - ConfigurationType.json_schema() > + TestSuitesConfigs.fix_custom_config_annotations() > return ConfigurationType.validate_python(config_data) > except ValidationError as e: > raise ConfigurationError("failed to load the supplied configurat= ion") from e > diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/c= onfig/conf_yaml_schema.json > index 1cf1bb098a..1434cdfad3 100644 > --- a/dts/framework/config/conf_yaml_schema.json > +++ b/dts/framework/config/conf_yaml_schema.json > @@ -66,6 +66,33 @@ > "title": "Compiler", > "type": "string" > }, > + "HelloWorldConfig": { > + "anyOf": [ > + { > + "additionalProperties": false, > + "properties": { > + "test_cases": { > + "items": { > + "type": "string" > + }, > + "title": "Test Cases", > + "type": "array" > + }, > + "timeout": { > + "default": 50, > + "title": "Timeout", > + "type": "integer" > + } > + }, > + "type": "object" > + }, > + { > + "type": "string" > + } > + ], > + "description": "Example custom configuration for the `TestHelloWor= ld` test suite.", > + "title": "HelloWorldConfig" > + }, > "HugepageConfiguration": { > "additionalProperties": false, > "description": "The hugepage configuration of :class:`~framework.t= estbed_model.node.Node`\\s.\n\nAttributes:\n number_of: The number of hu= gepages to allocate.\n force_first_numa: If :data:`True`, the hugepages = will be configured on the first NUMA node.", > @@ -373,14 +400,6 @@ > "title": "Func", > "type": "boolean" > }, > - "test_suites": { > - "items": { > - "$ref": "#/$defs/TestSuiteConfig" > - }, > - "minItems": 1, > - "title": "Test Suites", > - "type": "array" > - }, > "build_targets": { > "items": { > "$ref": "#/$defs/BuildTargetConfiguration" > @@ -393,6 +412,9 @@ > "title": "Skip Smoke Tests", > "type": "boolean" > }, > + "test_suites": { > + "$ref": "#/$defs/TestSuitesConfigs" > + }, > "system_under_test_node": { > "$ref": "#/$defs/TestRunSUTNodeConfiguration" > }, > @@ -404,8 +426,8 @@ > "required": [ > "perf", > "func", > - "test_suites", > "build_targets", > + "test_suites", > "system_under_test_node", > "traffic_generator_node" > ], > @@ -439,31 +461,63 @@ > { > "additionalProperties": false, > "properties": { > - "test_suite": { > - "description": "The identifying name of the test suite.", > - "title": "Test suite name", > - "type": "string" > - }, > "test_cases": { > - "description": "The identifying name of the test cases of = the test suite.", > "items": { > "type": "string" > }, > - "title": "Test cases by name", > + "title": "Test Cases", > "type": "array" > } > }, > - "required": [ > - "test_suite" > - ], > "type": "object" > }, > { > "type": "string" > } > ], > - "description": "Test suite configuration.\n\nInformation about a s= ingle test suite to be executed. It can be represented and validated as a\n= string type in the form of: ``TEST_SUITE [TEST_CASE, ...]``, in the configu= ration file.\n\nAttributes:\n test_suite: The name of the test suite mod= ule without the starting ``TestSuite_``.\n test_cases: The names of test= cases from this test suite to execute.\n If empty, all test cases w= ill be executed.", > + "description": "Test suite configuration base model.\n\nBy default= the configuration of a generic test suite does not contain any attributes.= Any test\nsuite should inherit this class to create their own custom confi= guration. Finally override the\ntype of the :attr:`~TestSuite.config` to us= e the newly created one.\n\nAttributes:\n test_cases_names: The names of= test cases from this test suite to execute. If empty, all\n test ca= ses will be executed.", > "title": "TestSuiteConfig" > + }, > + "TestSuitesConfigs": { > + "additionalProperties": false, > + "description": "Configuration mapping class to select and configur= e the test suites.\n\nBefore using this class, the custom configuration typ= e annotations need to be fixed.\nTo do so, you need to call the `fix_custom= _config_annotations` method.", > + "properties": { > + "hello_world": { > + "anyOf": [ > + { > + "$ref": "#/$defs/HelloWorldConfig" > + }, > + { > + "type": "null" > + } > + ], > + "default": null > + }, > + "os_udp": { > + "anyOf": [ > + { > + "$ref": "#/$defs/TestSuiteConfig" > + }, > + { > + "type": "null" > + } > + ], > + "default": null > + }, > + "pmd_buffer_scatter": { > + "anyOf": [ > + { > + "$ref": "#/$defs/TestSuiteConfig" > + }, > + { > + "type": "null" > + } > + ], > + "default": null > + } > + }, > + "title": "TestSuitesConfigs", > + "type": "object" > } > }, > "description": "DTS testbed and test configuration.\n\nAttributes:\n = test_runs: Test run configurations.\n nodes: Node configurations.", > diff --git a/dts/framework/config/generated.py b/dts/framework/config/gen= erated.py > new file mode 100644 > index 0000000000..d42ce93a51 > --- /dev/null > +++ b/dts/framework/config/generated.py > @@ -0,0 +1,40 @@ > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright(c) 2024 The DPDK contributors > +# This file is automatically generated by generate-test-mappings.py. > +# Do NOT modify this file manually. > + > +"""Generated file containing the links between the test suites and the c= onfiguration.""" > + > +from typing import TYPE_CHECKING, Optional > + > +from framework.config.test_suite import BaseTestSuitesConfigs, TestSuite= Config > + > +if TYPE_CHECKING: > + from tests.TestSuite_hello_world import HelloWorldConfig > + > + > +CUSTOM_CONFIG_TYPES: dict[str, type[TestSuiteConfig]] =3D {} > + > + > +class TestSuitesConfigs(BaseTestSuitesConfigs): > + """Configuration mapping class to select and configure the test suit= es. > + > + Before using this class, the custom configuration type annotations n= eed to be fixed. > + To do so, you need to call the `fix_custom_config_annotations` metho= d. > + """ > + > + hello_world: Optional["HelloWorldConfig"] =3D None > + os_udp: Optional[TestSuiteConfig] =3D None > + pmd_buffer_scatter: Optional[TestSuiteConfig] =3D None > + > + @classmethod > + def fix_custom_config_annotations(cls): > + """Fixes the custom config types annotations. > + > + Moreover it also fills `CUSTOM_CONFIG_TYPES` with all the custom= config types. > + """ > + from tests.TestSuite_hello_world import HelloWorldConfig > + > + CUSTOM_CONFIG_TYPES["hello_world"] =3D HelloWorldConfig > + > + cls.model_rebuild() > diff --git a/dts/framework/config/test_suite.py b/dts/framework/config/te= st_suite.py > new file mode 100644 > index 0000000000..0c1ddf9d95 > --- /dev/null > +++ b/dts/framework/config/test_suite.py > @@ -0,0 +1,140 @@ > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright(c) 2024 Arm Limited > + > +"""Test suites configuration module. > + > +Test suites can inherit :class:`TestSuiteConfig` to create their own cus= tom configuration. > +By doing so, the test suite class must also override the annotation of t= he field > +`~framework.test_suite.TestSuite.config` to use their custom configurati= on type. > +""" > + > +from typing import TYPE_CHECKING, Any, Iterable > + > +from pydantic import BaseModel, Field, ValidationInfo, field_validator, = model_validator > +from pydantic.config import JsonDict > +from typing_extensions import Self > + > +if TYPE_CHECKING: > + from framework.test_suite import TestSuiteSpec > + > + > +def make_parsable_schema(schema: JsonDict): > + """Updates a model's JSON schema to make a string representation a v= alid alternative. > + > + This utility function is required to be used with models that can be= represented and validated > + as a string instead of an object mapping. Normally the generated JSO= N schema will just show > + the object mapping. This function wraps the mapping under an anyOf p= roperty sequenced with a > + string type. > + > + This function is a valid `Callable` for the `json_schema_extra` attr= ibute of > + `~pydantic.config.ConfigDict`. > + """ > + inner_schema =3D schema.copy() > + > + fields_to_preserve =3D ["title", "description"] > + extracted_fields =3D {k: v for k in fields_to_preserve if (v :=3D in= ner_schema.get(k))} > + for field in extracted_fields: > + del inner_schema[field] > + > + schema.clear() > + schema.update(extracted_fields) > + schema["anyOf"] =3D [inner_schema, {"type": "string"}] > + > + > +class TestSuiteConfig(BaseModel, extra=3D"forbid", json_schema_extra=3Dm= ake_parsable_schema): > + """Test suite configuration base model. > + > + By default the configuration of a generic test suite does not contai= n any attributes. Any test > + suite should inherit this class to create their own custom configura= tion. Finally override the > + type of the :attr:`~TestSuite.config` to use the newly created one. > + > + Attributes: > + test_cases_names: The names of test cases from this test suite t= o execute. If empty, all > + test cases will be executed. > + """ > + > + _test_suite_spec: "TestSuiteSpec" > + > + test_cases_names: list[str] =3D Field(default_factory=3Dlist, alias= =3D"test_cases") > + > + @property > + def test_suite_name(self) -> str: > + """The name of the test suite module without the starting ``Test= Suite_``.""" > + return self._test_suite_spec.name > + > + @property > + def test_suite_spec(self) -> "TestSuiteSpec": > + """The specification of the requested test suite.""" > + return self._test_suite_spec > + > + @model_validator(mode=3D"before") > + @classmethod > + def convert_from_string(cls, data: Any) -> Any: > + """Validator which allows to select a test suite by string inste= ad of a mapping.""" > + if isinstance(data, str): > + test_cases =3D [] if data =3D=3D "all" else data.split() > + return dict(test_cases=3Dtest_cases) > + return data > + > + @classmethod > + def make(cls, test_suite_name: str, *test_cases_names: str, **kwargs= ) -> Self: > + """Make a configuration for the requested test suite. > + > + Args: > + test_suite_name: The name of the test suite. > + test_cases_names: The test cases to select, if empty all are= selected. > + **kwargs: Any other configuration field. > + > + Raises: > + AssertionError: If the requested test suite or test cases do= not exist. > + ValidationError: If the configuration fields were not filled= correctly. > + """ > + from framework.test_suite import find_by_name > + > + test_suite_spec =3D find_by_name(test_suite_name) > + assert test_suite_spec is not None, f"Could not find test suite = '{test_suite_name}'." > + test_suite_spec.validate_test_cases(test_cases_names) > + > + config =3D cls.model_validate({"test_cases": test_cases_names, *= *kwargs}) > + config._test_suite_spec =3D test_suite_spec > + return config > + > + > +class BaseTestSuitesConfigs(BaseModel, extra=3D"forbid"): > + """Base class for test suites configs.""" > + > + def __contains__(self, key) -> bool: > + """Check if the provided test suite name has been selected and/o= r configured.""" > + return key in self.model_fields_set > + > + def __getitem__(self, key) -> TestSuiteConfig: > + """Get test suite configuration.""" > + return self.__getattribute__(key) > + > + def get_configs(self) -> Iterable[TestSuiteConfig]: > + """Get all the test suite configurations.""" > + return map(lambda t: self[t], self.model_fields_set) > + > + @classmethod > + def available_test_suites(cls) -> Iterable[str]: > + """List all the available test suites.""" > + return cls.model_fields.keys() > + > + @field_validator("*") > + @classmethod > + def validate_test_suite_config( > + cls, config: type[TestSuiteConfig], info: ValidationInfo > + ) -> type[TestSuiteConfig]: > + """Validate the provided test cases and link the test suite spec= to the configuration.""" > + from framework.test_suite import find_by_name > + > + test_suite_name =3D info.field_name > + assert test_suite_name is not None > + > + test_suite_spec =3D find_by_name(test_suite_name) > + assert test_suite_spec is not None > + > + config._test_suite_spec =3D test_suite_spec > + > + test_suite_spec.validate_test_cases(config.test_cases_names) > + return config > diff --git a/dts/framework/runner.py b/dts/framework/runner.py > index 00b63cc292..bc7aa1555c 100644 > --- a/dts/framework/runner.py > +++ b/dts/framework/runner.py > @@ -24,10 +24,13 @@ > from types import FunctionType > from typing import Iterable > > +from pydantic import ValidationError > + > from framework.testbed_model.sut_node import SutNode > from framework.testbed_model.tg_node import TGNode > > from .config import ( > + CUSTOM_CONFIG_TYPES, > BuildTargetConfiguration, > Configuration, > SutNodeConfiguration, > @@ -36,7 +39,12 @@ > TGNodeConfiguration, > load_config, > ) > -from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCase= VerifyError > +from .exception import ( > + BlockingTestSuiteError, > + ConfigurationError, > + SSHTimeoutError, > + TestCaseVerifyError, > +) > from .logger import DTSLogger, DtsStage, get_dts_logger > from .settings import SETTINGS > from .test_result import ( > @@ -142,12 +150,7 @@ def run(self) -> None: > self._logger.set_stage(DtsStage.test_run_setup) > self._logger.info(f"Running test run with SUT '{sut_node= _config.name}'.") > test_run_result =3D self._result.add_test_run(test_run_c= onfig) > - # we don't want to modify the original config, so create= a copy > - test_run_test_suites =3D list( > - SETTINGS.test_suites if SETTINGS.test_suites else te= st_run_config.test_suites > - ) > - if not test_run_config.skip_smoke_tests: > - test_run_test_suites[:0] =3D [TestSuiteConfig("smoke= _tests")] > + test_run_test_suites =3D self._prepare_test_suites(test_= run_config) > try: > test_suites_with_cases =3D self._get_test_suites_wit= h_cases( > test_run_test_suites, test_run_config.func, test= _run_config.perf > @@ -204,6 +207,40 @@ def _check_dts_python_version(self) -> None: > ) > self._logger.warning("Please use Python >=3D 3.10 instead.") > > + def _prepare_test_suites(self, test_run_config: TestRunConfiguration= ) -> list[TestSuiteConfig]: > + if SETTINGS.test_suites: > + test_suites_configs =3D [] > + for selected_test_suite, selected_test_cases in SETTINGS.tes= t_suites: > + if selected_test_suite in test_run_config.test_suites: > + config =3D test_run_config.test_suites[selected_test= _suite].model_copy() > + config.test_cases_names =3D selected_test_cases > + else: > + try: > + config =3D CUSTOM_CONFIG_TYPES[selected_test_sui= te].make( > + selected_test_suite, *selected_test_cases > + ) > + except AssertionError as e: > + raise ConfigurationError( > + "Invalid test cases were selected " > + f"for test suite {selected_test_suite}." > + ) from e > + except ValidationError as e: > + raise ConfigurationError( > + f"Test suite {selected_test_suite} needs to = be explicitly configured " > + "in order to be selected." > + ) from e > + test_suites_configs.append(config) > + else: > + # we don't want to modify the original config, so create a c= opy > + test_suites_configs =3D [ > + config.model_copy() for config in test_run_config.test_s= uites.get_configs() > + ] > + > + if not test_run_config.skip_smoke_tests: > + test_suites_configs[:0] =3D [TestSuiteConfig.make("smoke_tes= ts")] > + > + return test_suites_configs > + > def _get_test_suites_with_cases( > self, > test_suite_configs: list[TestSuiteConfig], > @@ -245,7 +282,9 @@ def _get_test_suites_with_cases( > > test_suites_with_cases.append( > TestSuiteWithCases( > - test_suite_class=3Dtest_suite_class, test_cases=3Dse= lected_test_cases > + test_suite_class=3Dtest_suite_class, > + test_cases=3Dselected_test_cases, > + config=3Dtest_suite_config, > ) > ) > return test_suites_with_cases > @@ -466,7 +505,9 @@ def _run_test_suite( > self._logger.set_stage( > DtsStage.test_suite_setup, Path(SETTINGS.output_dir, test_su= ite_name) > ) > - test_suite =3D test_suite_with_cases.test_suite_class(sut_node, = tg_node) > + test_suite =3D test_suite_with_cases.test_suite_class( > + sut_node, tg_node, test_suite_with_cases.config > + ) > try: > self._logger.info(f"Starting test suite setup: {test_suite_n= ame}") > test_suite.set_up_suite() > diff --git a/dts/framework/settings.py b/dts/framework/settings.py > index 2e8dedef4f..063f282edf 100644 > --- a/dts/framework/settings.py > +++ b/dts/framework/settings.py > @@ -85,9 +85,7 @@ > from pathlib import Path > from typing import Callable > > -from pydantic import ValidationError > - > -from .config import TestSuiteConfig > +from .config import TestSuitesConfigs > from .exception import ConfigurationError > from .utils import DPDKGitTarball, get_commit_id > > @@ -114,7 +112,7 @@ class Settings: > #: > compile_timeout: float =3D 1200 > #: > - test_suites: list[TestSuiteConfig] =3D field(default_factory=3Dlist) > + test_suites: list[tuple[str, list[str]]] =3D field(default_factory= =3Dlist) > #: > re_run: int =3D 0 > > @@ -382,7 +380,7 @@ def _get_parser() -> _DTSArgumentParser: > > def _process_test_suites( > parser: _DTSArgumentParser, args: list[list[str]] > -) -> list[TestSuiteConfig]: > +) -> list[tuple[str, list[str]]]: > """Process the given argument to a list of :class:`TestSuiteConfig` = to execute. > > Args: > @@ -398,16 +396,17 @@ def _process_test_suites( > # Environment variable in the form of "SUITE1 CASE1 CASE2, SUITE= 2 CASE1, SUITE3, ..." > args =3D [suite_with_cases.split() for suite_with_cases in args[= 0][0].split(",")] > > - try: > - return [TestSuiteConfig(test_suite, test_cases) for [test_suite,= *test_cases] in args] > - except ValidationError as e: > - print( > - "An error has occurred while validating the test suites supp= lied in the " > - f"{'environment variable' if action else 'arguments'}:", > - file=3Dsys.stderr, > - ) > - print(e, file=3Dsys.stderr) > - sys.exit(1) > + available_test_suites =3D TestSuitesConfigs.available_test_suites() > + for test_suite_name, *_ in args: > + if test_suite_name not in available_test_suites: > + print( > + f"The test suite {test_suite_name} supplied in the " > + f"{'environment variable' if action else 'arguments'} is= invalid.", > + file=3Dsys.stderr, > + ) > + sys.exit(1) > + > + return [(test_suite, test_cases) for test_suite, *test_cases in args= ] > > > def get_settings() -> Settings: > diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py > index 5694a2482b..6c10c1e40a 100644 > --- a/dts/framework/test_result.py > +++ b/dts/framework/test_result.py > @@ -64,17 +64,7 @@ class is to hold a subset of test cases (which could b= e all test cases) because > > test_suite_class: type[TestSuite] > test_cases: list[FunctionType] > - > - def create_config(self) -> TestSuiteConfig: > - """Generate a :class:`TestSuiteConfig` from the stored test suit= e with test cases. > - > - Returns: > - The :class:`TestSuiteConfig` representation. > - """ > - return TestSuiteConfig( > - test_suite=3Dself.test_suite_class.__name__, > - test_cases=3D[test_case.__name__ for test_case in self.test_= cases], > - ) > + config: TestSuiteConfig > > > class Result(Enum): > diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py > index 972968b036..78e1b4c49a 100644 > --- a/dts/framework/test_suite.py > +++ b/dts/framework/test_suite.py > @@ -23,7 +23,7 @@ > from ipaddress import IPv4Interface, IPv6Interface, ip_interface > from pkgutil import iter_modules > from types import FunctionType, ModuleType > -from typing import ClassVar, NamedTuple, Union > +from typing import ClassVar, Iterable, NamedTuple, Union, get_type_hints > > from pydantic.alias_generators import to_pascal > from scapy.layers.inet import IP # type: ignore[import-untyped] > @@ -31,6 +31,7 @@ > from scapy.packet import Packet, Padding # type: ignore[import-untyped] > from typing_extensions import Self > > +from framework.config import TestSuiteConfig > from framework.testbed_model.port import Port, PortLink > from framework.testbed_model.sut_node import SutNode > from framework.testbed_model.tg_node import TGNode > @@ -38,7 +39,7 @@ > PacketFilteringConfig, > ) > > -from .exception import TestCaseVerifyError > +from .exception import InternalError, TestCaseVerifyError > from .logger import DTSLogger, get_dts_logger > from .utils import get_packet_summaries > > @@ -78,6 +79,7 @@ class TestSuite: > #: Whether the test suite is blocking. A failure of a blocking test = suite > #: will block the execution of all subsequent test suites in the cur= rent build target. > is_blocking: ClassVar[bool] =3D False > + config: TestSuiteConfig > _logger: DTSLogger > _port_links: list[PortLink] > _sut_port_ingress: Port > @@ -93,6 +95,7 @@ def __init__( > self, > sut_node: SutNode, > tg_node: TGNode, > + config: TestSuiteConfig, > ): > """Initialize the test suite testbed information and basic confi= guration. > > @@ -102,9 +105,11 @@ def __init__( > Args: > sut_node: The SUT node where the test suite will run. > tg_node: The TG node where the test suite will run. > + config: The test suite configuration. > """ > self.sut_node =3D sut_node > self.tg_node =3D tg_node > + self.config =3D config > self._logger =3D get_dts_logger(self.__class__.__name__) > self._port_links =3D [] > self._process_links() > @@ -467,6 +472,21 @@ def is_test_suite(obj) -> bool: > > raise Exception("class not found in eligible test module") > > + @cached_property > + def config_type(self) -> type[TestSuiteConfig]: > + """A reference to the test suite's configuration type.""" > + fields =3D get_type_hints(self.class_type) > + config_type =3D fields.get("config") > + if config_type is None: > + raise InternalError( > + "Test suite class {self.class_name} is missing the `conf= ig` attribute." > + ) > + if not issubclass(config_type, TestSuiteConfig): > + raise InternalError( > + f"Test suite class {self.class_name} has an invalid conf= iguration type assigned." > + ) > + return config_type > + > @cached_property > def test_cases(self) -> list[TestCase]: > """A list of all the available test cases.""" > @@ -533,6 +553,14 @@ def discover_all( > > return test_suites > > + def validate_test_cases(self, test_cases_names: Iterable[str]) -> No= ne: > + """Validate if the supplied test cases exist in the test suite."= "" > + available_test_cases =3D map(lambda t: t.name, self.test_cases) > + for requested_test_case in test_cases_names: > + assert ( > + requested_test_case in available_test_cases > + ), f"{requested_test_case} is not a valid test case for test= suite {self.name}." > + > > AVAILABLE_TEST_SUITES: list[TestSuiteSpec] =3D TestSuiteSpec.discover_al= l() > """Constant to store all the available, discovered and imported test sui= tes. > diff --git a/dts/generate-schema.py b/dts/generate-schema.py > index b41d28492f..d24a67f68e 100755 > --- a/dts/generate-schema.py > +++ b/dts/generate-schema.py > @@ -9,7 +9,7 @@ > > from pydantic.json_schema import GenerateJsonSchema > > -from framework.config import ConfigurationType > +from framework.config import ConfigurationType, TestSuitesConfigs > > DTS_DIR =3D os.path.dirname(os.path.realpath(__file__)) > RELATIVE_PATH_TO_SCHEMA =3D "framework/config/conf_yaml_schema.json" > @@ -26,6 +26,8 @@ def generate(self, schema, mode=3D"validation"): > > > try: > + TestSuitesConfigs.fix_custom_config_annotations() > + > path =3D os.path.join(DTS_DIR, RELATIVE_PATH_TO_SCHEMA) > > with open(path, "w") as schema_file: > diff --git a/dts/generate-test-mappings.py b/dts/generate-test-mappings.p= y > new file mode 100755 > index 0000000000..d076ad0afe > --- /dev/null > +++ b/dts/generate-test-mappings.py > @@ -0,0 +1,132 @@ > +#!/usr/bin/env python3 > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright(c) 2024 Arm Limited > + > +"""Test Suites to Configuration mappings generation script.""" > + > +import os > +from collections import defaultdict > +from textwrap import indent > +from typing import Iterable > + > +from framework.config.test_suite import BaseTestSuitesConfigs, TestSuite= Config > +from framework.exception import InternalError > +from framework.test_suite import AVAILABLE_TEST_SUITES, TestSuiteSpec > + > +DTS_DIR =3D os.path.dirname(os.path.realpath(__file__)) > +SCRIPT_FILE_NAME =3D os.path.basename(__file__) > + > +FRAMEWORK_IMPORTS =3D [BaseTestSuitesConfigs, TestSuiteConfig] > + > +RELATIVE_PATH_TO_GENERATED_FILE =3D "framework/config/generated.py" > +SMOKE_TESTS_SUITE_NAME =3D "smoke_tests" > +CUSTOM_CONFIG_TYPES_VAR_NAME =3D "CUSTOM_CONFIG_TYPES" > +CUSTOM_CONFIG_LINKING_FUNCTION_NAME =3D "fix_custom_config_annotations" > +CUSTOM_CONFIG_LINKING_FUNCTION_DOCSTRING =3D [ > + '"""Fixes the custom config types annotations.', > + "", > + f"Moreover it also fills `{CUSTOM_CONFIG_TYPES_VAR_NAME}` with all t= he custom config types.", > + '"""', > +] > +TEST_SUITES_CONFIG_CLASS_NAME =3D "TestSuitesConfigs" > +TEST_SUITES_CONFIG_CLASS_DOCSTRING =3D [ > + '"""Configuration mapping class to select and configure the test sui= tes.', > + "", > + "Before using this class, the custom configuration type annotations = need to be fixed.", > + f"To do so, you need to call the `{CUSTOM_CONFIG_LINKING_FUNCTION_NA= ME}` method.", > + '"""', > +] > + > + > +GENERATED_FILE_HEADER =3D f"""# SPDX-License-Identifier: BSD-3-Clause > +# Copyright(c) 2024 The DPDK contributors > +# This file is automatically generated by {SCRIPT_FILE_NAME}. > +# Do NOT modify this file manually. > + > +\"\"\"Generated file containing the links between the test suites and th= e configuration.\"\"\" > +""" > + > + > +def join(lines: Iterable[str]) -> str: > + """Join list of strings into text lines.""" > + return "\n".join(lines) > + > + > +def join_and_indent(lines: Iterable[str], indentation_level=3D1, indenta= tion_spaces=3D4) -> str: > + """Join list of strings into indented text lines.""" > + return "\n".join([indent(line, " " * indentation_level * indentation= _spaces) for line in lines]) > + > + > +def format_attributes_types(test_suite_spec: TestSuiteSpec): > + """Format the config type into the respective configuration class fi= eld attribute type.""" > + config_type =3D test_suite_spec.config_type.__name__ > + if config_type !=3D TestSuiteConfig.__name__: > + config_type =3D f'"{config_type}"' > + return f"Optional[{config_type}]" > + > + > +try: > + framework_imports: dict[str, list[str]] =3D defaultdict(list) > + for _import in FRAMEWORK_IMPORTS: > + framework_imports[_import.__module__].append(_import.__name__) > + formatted_framework_imports =3D sorted( > + [ > + f"from {module} import {', '.join(sorted(imports))}" > + for module, imports in framework_imports.items() > + ] > + ) > + > + test_suites =3D [ > + test_suite_spec > + for test_suite_spec in AVAILABLE_TEST_SUITES > + if test_suite_spec.name !=3D SMOKE_TESTS_SUITE_NAME > + ] > + > + custom_configs =3D [t for t in test_suites if t.config_type is not T= estSuiteConfig] > + > + custom_config_imports =3D [ > + f"from {t.module_type.__name__} import {t.config_type.__name__}"= for t in custom_configs > + ] > + > + test_suites_attributes =3D [f"{t.name}: {format_attributes_types(t)}= =3D None" for t in test_suites] > + > + custom_config_assignments =3D [ > + f'{CUSTOM_CONFIG_TYPES_VAR_NAME}["{t.name}"] =3D {t.config_type.= __name__}' > + for t in custom_configs > + ] > + > + generated_file_contents =3D f"""{GENERATED_FILE_HEADER} > +from typing import TYPE_CHECKING, Optional > + > +{join(formatted_framework_imports)} > + > +if TYPE_CHECKING: > +{join_and_indent(custom_config_imports)} > + > + > +{CUSTOM_CONFIG_TYPES_VAR_NAME}: dict[str, type[{TestSuiteConfig.__name__= }]] =3D {'{}'} > + > + > +class {TEST_SUITES_CONFIG_CLASS_NAME}({BaseTestSuitesConfigs.__name__}): > +{join_and_indent(TEST_SUITES_CONFIG_CLASS_DOCSTRING)} > + > +{join_and_indent(test_suites_attributes)} > + > + @classmethod > + def {CUSTOM_CONFIG_LINKING_FUNCTION_NAME}(cls): > +{join_and_indent(CUSTOM_CONFIG_LINKING_FUNCTION_DOCSTRING, indentation_l= evel=3D2)} > +{join_and_indent(custom_config_imports, indentation_level=3D2)} > + > +{join_and_indent(custom_config_assignments, indentation_level=3D2)} > + > + cls.model_rebuild() > +""" > + > + path =3D os.path.join(DTS_DIR, RELATIVE_PATH_TO_GENERATED_FILE) > + > + with open(path, "w") as generated_file: > + generated_file.write(generated_file_contents) > + > + print("Test suites to configuration mappings generated successfully!= ") > +except Exception as e: > + raise InternalError("Failed to generate test suites to configuration= mappings.") from e > diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hel= lo_world.py > index d958f99030..1723c123bc 100644 > --- a/dts/tests/TestSuite_hello_world.py > +++ b/dts/tests/TestSuite_hello_world.py > @@ -7,6 +7,7 @@ > No other EAL parameters apart from cores are used. > """ > > +from framework.config import TestSuiteConfig > from framework.remote_session.dpdk_shell import compute_eal_params > from framework.test_suite import TestSuite > from framework.testbed_model.cpu import ( > @@ -16,9 +17,18 @@ > ) > > > +class HelloWorldConfig(TestSuiteConfig): > + """Example custom configuration for the `TestHelloWorld` test suite.= """ > + > + #: Timeout for the DPDK apps > + timeout: int =3D 50 > + > + > class TestHelloWorld(TestSuite): > """DPDK hello world app test suite.""" > > + config: HelloWorldConfig > + > def set_up_suite(self) -> None: > """Set up the test suite. > > @@ -59,7 +69,7 @@ def test_hello_world_all_cores(self) -> None: > eal_para =3D compute_eal_params( > self.sut_node, lcore_filter_specifier=3DLogicalCoreList(self= .sut_node.lcores) > ) > - result =3D self.sut_node.run_dpdk_app(self.app_helloworld_path, = eal_para, 50) > + result =3D self.sut_node.run_dpdk_app(self.app_helloworld_path, = eal_para, self.config.timeout) > for lcore in self.sut_node.lcores: > self.verify( > f"hello from core {int(lcore)}" in result.stdout, > diff --git a/dts/tests/__init__.py b/dts/tests/__init__.py > new file mode 100644 > index 0000000000..a300eb26fc > --- /dev/null > +++ b/dts/tests/__init__.py > @@ -0,0 +1,7 @@ > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright(c) 2024 Arm Limited > + > +"""Test suites. > + > +This package contains all the available test suites in DTS. > +""" > -- > 2.34.1 >