[PATCH] dts: add per-test-suite configuration

DPDK patches and discussions
 help / color / mirror / Atom feed

* [PATCH] dts: add per-test-suite configuration
@ 2024-09-06 16:13 Luca Vizzarro
  2024-09-27 17:45 ` Jeremy Spewock
                   ` (2 more replies)
  0 siblings, 3 replies; 17+ messages in thread
From: Luca Vizzarro @ 2024-09-06 16:13 UTC (permalink / raw)
  To: dev
  Cc: Honnappa Nagarahalli, Luca Vizzarro, Paul Szczepanek,
	Alex Chapman, Juraj Linkeš

Allow test suites to be configured individually. Moreover enable them to
implement their own custom configuration.

This solution adds some new complexity to DTS, which is generated source
code. In order to ensure strong typing, the test suites and their custom
configurations need to be linked in the main configuration class.
Unfortunately, this is not feasible during runtime as it will incur in
circular dependencies. Generating the links appear to be the most
straightforward approach.

This commit also brings a new major change to the configuration schema.
Test suites are no longer defined as a list of strings, like:

    test_suites:
    - hello_world
    - pmd_buffer_scatter

but as mapping of mappings or strings:

    test_suites:
      hello_world: {} # any custom fields or test cases can be set here
      pmd_buffer_scatter: all # "all" defines all the test cases, or
                              # they can individually be set separated
                              # by a space

Not defining the `test_cases` field in the configuration is equivalent
to `all`, therefore the definitions for either test suite above are
also equivalent.

Creating the __init__.py file under the tests folder, allows it to be
picked up as a package. This is a mypy requirement to import the tests
from within the framework.

Bugzilla ID: 1375

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Alex Chapman <alex.chapman@arm.com>
---
Depends-on: series-32823 ("dts: Pydantic configuration")

Hello,

sending in a solution for the per-test-suite configuration issue.
This one took some thinking but I have given most of the motivations in
the commit body already. The docs are somewhat lacking but hopefully they
should be automatically tackled by the API docs generation.


Best,
Luca
---
 doc/guides/tools/dts.rst                   |  39 ++++--
 dts/conf.yaml                              |   4 +-
 dts/framework/config/__init__.py           |  98 ++-------------
 dts/framework/config/conf_yaml_schema.json |  94 +++++++++++---
 dts/framework/config/generated.py          |  40 ++++++
 dts/framework/config/test_suite.py         | 140 +++++++++++++++++++++
 dts/framework/runner.py                    |  59 +++++++--
 dts/framework/settings.py                  |  29 +++--
 dts/framework/test_result.py               |  12 +-
 dts/framework/test_suite.py                |  32 ++++-
 dts/generate-schema.py                     |   4 +-
 dts/generate-test-mappings.py              | 132 +++++++++++++++++++
 dts/tests/TestSuite_hello_world.py         |  12 +-
 dts/tests/__init__.py                      |   7 ++
 14 files changed, 539 insertions(+), 163 deletions(-)
 create mode 100644 dts/framework/config/generated.py
 create mode 100644 dts/framework/config/test_suite.py
 create mode 100755 dts/generate-test-mappings.py
 create mode 100644 dts/tests/__init__.py

diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index 317bd0ff99..66681543cd 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -391,6 +391,26 @@ There are four types of methods that comprise a test suite:
    should be implemented in the ``SutNode`` class (and the underlying classes that ``SutNode`` uses)
    and used by the test suite via the ``sut_node`` field.
 
+The test suites can also implement their own custom configuration fields. This can be achieved by
+creating a new test suite config file which inherits from ``TestSuiteConfig`` defined in
+``dts/framework/config/test_suite.py``. So that this new custom configuration class is used, the
+test suite class must override the ``config`` attribute annotation with your new class, for example::
+
+.. code:: python
+   class CustomConfig(TestSuiteConfig):
+      my_custom_field: int = 10
+
+   class TestMyNewTestSuite(TestSuite):
+      config: CustomConfig
+
+Finally, the test suites and the custom configuration files need to linked in the global configuration.
+This can be easily achieved by running the ``dts/generate-test-mappings.py``, e.g.:
+
+.. code-block:: console
+
+   $ poetry shell
+   (dts-py3.10) $ ./generate-test-mappings.py
+
 
 .. _dts_dev_tools:
 
@@ -510,18 +530,13 @@ _`Network port`
    ``peer_pci``           *string* – the PCI address of the peer node port. **Example**: ``000a:01:00.1``
    ====================== =================================================================================
 
-_`Test suite`
-   *string* – name of the test suite to run. **Examples**: ``hello_world``, ``os_udp``
-
-_`Test target`
-   *mapping* – selects specific test cases to run from a test suite. Mapping is described as follows:
-
-   ========= ===============================================================================================
-   ``suite`` See `Test suite`_
-   ``cases`` (*optional*) *sequence* of *string* – list of the selected test cases in the test suite to run.
+_`Test suites`
+   *mapping* – selects the test suites to run. Each mapping key corresponds to the test suite name.
 
-             Unknown test cases will be silently ignored.
-   ========= ===============================================================================================
+   The value of the mapping can either "all" to select all the test cases in that test suite, the test
+   cases names divided by a space. Or it can be another mapping to set any custom fields for the test suite.
+   In the case of a mapping, all the test cases are selected by default. In order to manually select test
+   cases, the ``test_cases`` field can be set with a list of strings, each entry being a test case name.
 
 
 Properties
@@ -542,7 +557,7 @@ involved in the testing. These can be defined with the following mappings:
    +----------------------------+-------------------------------------------------------------------+
    | ``func``                   | *boolean* – Enable functional testing.                            |
    +----------------------------+-------------------------------------------------------------------+
-   | ``test_suites``            | *sequence* of **one of** `Test suite`_ **or** `Test target`_      |
+   | ``test_suites``            | See `Test suites`_                                                |
    +----------------------------+-------------------------------------------------------------------+
    | ``skip_smoke_tests``       | (*optional*) *boolean* – Allows you to skip smoke testing         |
    |                            | if ``true``.                                                      |
diff --git a/dts/conf.yaml b/dts/conf.yaml
index 7d95016e68..c44bef604c 100644
--- a/dts/conf.yaml
+++ b/dts/conf.yaml
@@ -15,8 +15,8 @@ test_runs:
     func: true # enable functional testing
     skip_smoke_tests: false # optional
     test_suites: # the following test suites will be run in their entirety
-      - hello_world
-      - os_udp
+      hello_world: all
+      os_udp: all
     # The machine running the DPDK test executable
     system_under_test_node:
       node_name: "SUT 1"
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 013c529829..d39e0823bd 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -35,10 +35,12 @@
       and makes it thread safe should we ever want to move in that direction.
 """
 
+# pylama:ignore=W0611
+
 from enum import Enum, auto, unique
 from functools import cached_property
 from pathlib import Path
-from typing import TYPE_CHECKING, Annotated, Any, Literal, NamedTuple, Protocol
+from typing import Annotated, Literal, NamedTuple, Protocol
 
 import yaml
 from pydantic import (
@@ -50,15 +52,14 @@
     field_validator,
     model_validator,
 )
-from pydantic.config import JsonDict
 from pydantic.dataclasses import dataclass
 from typing_extensions import Self
 
 from framework.exception import ConfigurationError
 from framework.utils import StrEnum
 
-if TYPE_CHECKING:
-    from framework.test_suite import TestSuiteSpec
+from .generated import CUSTOM_CONFIG_TYPES, TestSuitesConfigs
+from .test_suite import TestSuiteConfig
 
 
 @unique
@@ -289,7 +290,7 @@ class NodeInfo:
     kernel_version: str
 
 
-@dataclass(slots=True, frozen=True, kw_only=True, config=ConfigDict(extra="forbid"))
+@dataclass(frozen=True, kw_only=True, config=ConfigDict(extra="forbid"))
 class BuildTargetConfiguration:
     """DPDK build configuration.
 
@@ -329,89 +330,6 @@ class BuildTargetInfo:
     compiler_version: str
 
 
-def make_parsable_schema(schema: JsonDict):
-    """Updates a model's JSON schema to make a string representation a valid alternative.
-
-    This utility function is required to be used with models that can be represented and validated
-    as a string instead of an object mapping. Normally the generated JSON schema will just show
-    the object mapping. This function wraps the mapping under an anyOf property sequenced with a
-    string type.
-
-    This function is a valid `Callable` for the `json_schema_extra` attribute of
-    `~pydantic.config.ConfigDict`.
-    """
-    inner_schema = schema.copy()
-    del inner_schema["title"]
-
-    title = schema.get("title")
-    description = schema.get("description")
-
-    schema.clear()
-
-    schema["title"] = title
-    schema["description"] = description
-    schema["anyOf"] = [inner_schema, {"type": "string"}]
-
-
-@dataclass(
-    frozen=True,
-    config=ConfigDict(extra="forbid", json_schema_extra=make_parsable_schema),
-)
-class TestSuiteConfig:
-    """Test suite configuration.
-
-    Information about a single test suite to be executed. It can be represented and validated as a
-    string type in the form of: ``TEST_SUITE [TEST_CASE, ...]``, in the configuration file.
-
-    Attributes:
-        test_suite: The name of the test suite module without the starting ``TestSuite_``.
-        test_cases: The names of test cases from this test suite to execute.
-            If empty, all test cases will be executed.
-    """
-
-    test_suite_name: str = Field(
-        title="Test suite name",
-        description="The identifying name of the test suite.",
-        alias="test_suite",
-    )
-    test_cases_names: list[str] = Field(
-        default_factory=list,
-        title="Test cases by name",
-        description="The identifying name of the test cases of the test suite.",
-        alias="test_cases",
-    )
-
-    @cached_property
-    def test_suite_spec(self) -> "TestSuiteSpec":
-        """The specification of the requested test suite."""
-        from framework.test_suite import find_by_name
-
-        test_suite_spec = find_by_name(self.test_suite_name)
-        assert test_suite_spec is not None, f"{self.test_suite_name} is not a valid test suite name"
-        return test_suite_spec
-
-    @model_validator(mode="before")
-    @classmethod
-    def convert_from_string(cls, data: Any) -> Any:
-        """Convert the string representation into a valid mapping."""
-        if isinstance(data, str):
-            [test_suite, *test_cases] = data.split()
-            return dict(test_suite=test_suite, test_cases=test_cases)
-        return data
-
-    @model_validator(mode="after")
-    def validate_names(self) -> Self:
-        """Validate the supplied test suite and test cases names."""
-        available_test_cases = map(lambda t: t.name, self.test_suite_spec.test_cases)
-        for requested_test_case in self.test_cases_names:
-            assert requested_test_case in available_test_cases, (
-                f"{requested_test_case} is not a valid test case "
-                f"for test suite {self.test_suite_name}"
-            )
-
-        return self
-
-
 @dataclass(slots=True, frozen=True, kw_only=True, config=ConfigDict(extra="forbid"))
 class TestRunSUTNodeConfiguration:
     """The SUT node configuration of a test run.
@@ -446,7 +364,7 @@ class TestRunConfiguration:
     perf: bool = Field(description="Enable performance testing.")
     func: bool = Field(description="Enable functional testing.")
     skip_smoke_tests: bool = False
-    test_suites: list[TestSuiteConfig] = Field(min_length=1)
+    test_suites: TestSuitesConfigs
     system_under_test_node: TestRunSUTNodeConfiguration
     traffic_generator_node: str
 
@@ -581,7 +499,7 @@ def load_config(config_file_path: Path) -> Configuration:
         config_data = yaml.safe_load(f)
 
     try:
-        ConfigurationType.json_schema()
+        TestSuitesConfigs.fix_custom_config_annotations()
         return ConfigurationType.validate_python(config_data)
     except ValidationError as e:
         raise ConfigurationError("failed to load the supplied configuration") from e
diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index 1cf1bb098a..1434cdfad3 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -66,6 +66,33 @@
       "title": "Compiler",
       "type": "string"
     },
+    "HelloWorldConfig": {
+      "anyOf": [
+        {
+          "additionalProperties": false,
+          "properties": {
+            "test_cases": {
+              "items": {
+                "type": "string"
+              },
+              "title": "Test Cases",
+              "type": "array"
+            },
+            "timeout": {
+              "default": 50,
+              "title": "Timeout",
+              "type": "integer"
+            }
+          },
+          "type": "object"
+        },
+        {
+          "type": "string"
+        }
+      ],
+      "description": "Example custom configuration for the `TestHelloWorld` test suite.",
+      "title": "HelloWorldConfig"
+    },
     "HugepageConfiguration": {
       "additionalProperties": false,
       "description": "The hugepage configuration of :class:`~framework.testbed_model.node.Node`\\s.\n\nAttributes:\n    number_of: The number of hugepages to allocate.\n    force_first_numa: If :data:`True`, the hugepages will be configured on the first NUMA node.",
@@ -373,14 +400,6 @@
           "title": "Func",
           "type": "boolean"
         },
-        "test_suites": {
-          "items": {
-            "$ref": "#/$defs/TestSuiteConfig"
-          },
-          "minItems": 1,
-          "title": "Test Suites",
-          "type": "array"
-        },
         "build_targets": {
           "items": {
             "$ref": "#/$defs/BuildTargetConfiguration"
@@ -393,6 +412,9 @@
           "title": "Skip Smoke Tests",
           "type": "boolean"
         },
+        "test_suites": {
+          "$ref": "#/$defs/TestSuitesConfigs"
+        },
         "system_under_test_node": {
           "$ref": "#/$defs/TestRunSUTNodeConfiguration"
         },
@@ -404,8 +426,8 @@
       "required": [
         "perf",
         "func",
-        "test_suites",
         "build_targets",
+        "test_suites",
         "system_under_test_node",
         "traffic_generator_node"
       ],
@@ -439,31 +461,63 @@
         {
           "additionalProperties": false,
           "properties": {
-            "test_suite": {
-              "description": "The identifying name of the test suite.",
-              "title": "Test suite name",
-              "type": "string"
-            },
             "test_cases": {
-              "description": "The identifying name of the test cases of the test suite.",
               "items": {
                 "type": "string"
               },
-              "title": "Test cases by name",
+              "title": "Test Cases",
               "type": "array"
             }
           },
-          "required": [
-            "test_suite"
-          ],
           "type": "object"
         },
         {
           "type": "string"
         }
       ],
-      "description": "Test suite configuration.\n\nInformation about a single test suite to be executed. It can be represented and validated as a\nstring type in the form of: ``TEST_SUITE [TEST_CASE, ...]``, in the configuration file.\n\nAttributes:\n    test_suite: The name of the test suite module without the starting ``TestSuite_``.\n    test_cases: The names of test cases from this test suite to execute.\n        If empty, all test cases will be executed.",
+      "description": "Test suite configuration base model.\n\nBy default the configuration of a generic test suite does not contain any attributes. Any test\nsuite should inherit this class to create their own custom configuration. Finally override the\ntype of the :attr:`~TestSuite.config` to use the newly created one.\n\nAttributes:\n    test_cases_names: The names of test cases from this test suite to execute. If empty, all\n        test cases will be executed.",
       "title": "TestSuiteConfig"
+    },
+    "TestSuitesConfigs": {
+      "additionalProperties": false,
+      "description": "Configuration mapping class to select and configure the test suites.\n\nBefore using this class, the custom configuration type annotations need to be fixed.\nTo do so, you need to call the `fix_custom_config_annotations` method.",
+      "properties": {
+        "hello_world": {
+          "anyOf": [
+            {
+              "$ref": "#/$defs/HelloWorldConfig"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null
+        },
+        "os_udp": {
+          "anyOf": [
+            {
+              "$ref": "#/$defs/TestSuiteConfig"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null
+        },
+        "pmd_buffer_scatter": {
+          "anyOf": [
+            {
+              "$ref": "#/$defs/TestSuiteConfig"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null
+        }
+      },
+      "title": "TestSuitesConfigs",
+      "type": "object"
     }
   },
   "description": "DTS testbed and test configuration.\n\nAttributes:\n    test_runs: Test run configurations.\n    nodes: Node configurations.",
diff --git a/dts/framework/config/generated.py b/dts/framework/config/generated.py
new file mode 100644
index 0000000000..d42ce93a51
--- /dev/null
+++ b/dts/framework/config/generated.py
@@ -0,0 +1,40 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 The DPDK contributors
+# This file is automatically generated by generate-test-mappings.py.
+# Do NOT modify this file manually.
+
+"""Generated file containing the links between the test suites and the configuration."""
+
+from typing import TYPE_CHECKING, Optional
+
+from framework.config.test_suite import BaseTestSuitesConfigs, TestSuiteConfig
+
+if TYPE_CHECKING:
+    from tests.TestSuite_hello_world import HelloWorldConfig
+
+
+CUSTOM_CONFIG_TYPES: dict[str, type[TestSuiteConfig]] = {}
+
+
+class TestSuitesConfigs(BaseTestSuitesConfigs):
+    """Configuration mapping class to select and configure the test suites.
+
+    Before using this class, the custom configuration type annotations need to be fixed.
+    To do so, you need to call the `fix_custom_config_annotations` method.
+    """
+
+    hello_world: Optional["HelloWorldConfig"] = None
+    os_udp: Optional[TestSuiteConfig] = None
+    pmd_buffer_scatter: Optional[TestSuiteConfig] = None
+
+    @classmethod
+    def fix_custom_config_annotations(cls):
+        """Fixes the custom config types annotations.
+
+        Moreover it also fills `CUSTOM_CONFIG_TYPES` with all the custom config types.
+        """
+        from tests.TestSuite_hello_world import HelloWorldConfig
+
+        CUSTOM_CONFIG_TYPES["hello_world"] = HelloWorldConfig
+
+        cls.model_rebuild()
diff --git a/dts/framework/config/test_suite.py b/dts/framework/config/test_suite.py
new file mode 100644
index 0000000000..0c1ddf9d95
--- /dev/null
+++ b/dts/framework/config/test_suite.py
@@ -0,0 +1,140 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Test suites configuration module.
+
+Test suites can inherit :class:`TestSuiteConfig` to create their own custom configuration.
+By doing so, the test suite class must also override the annotation of the field
+`~framework.test_suite.TestSuite.config` to use their custom configuration type.
+"""
+
+from typing import TYPE_CHECKING, Any, Iterable
+
+from pydantic import BaseModel, Field, ValidationInfo, field_validator, model_validator
+from pydantic.config import JsonDict
+from typing_extensions import Self
+
+if TYPE_CHECKING:
+    from framework.test_suite import TestSuiteSpec
+
+
+def make_parsable_schema(schema: JsonDict):
+    """Updates a model's JSON schema to make a string representation a valid alternative.
+
+    This utility function is required to be used with models that can be represented and validated
+    as a string instead of an object mapping. Normally the generated JSON schema will just show
+    the object mapping. This function wraps the mapping under an anyOf property sequenced with a
+    string type.
+
+    This function is a valid `Callable` for the `json_schema_extra` attribute of
+    `~pydantic.config.ConfigDict`.
+    """
+    inner_schema = schema.copy()
+
+    fields_to_preserve = ["title", "description"]
+    extracted_fields = {k: v for k in fields_to_preserve if (v := inner_schema.get(k))}
+    for field in extracted_fields:
+        del inner_schema[field]
+
+    schema.clear()
+    schema.update(extracted_fields)
+    schema["anyOf"] = [inner_schema, {"type": "string"}]
+
+
+class TestSuiteConfig(BaseModel, extra="forbid", json_schema_extra=make_parsable_schema):
+    """Test suite configuration base model.
+
+    By default the configuration of a generic test suite does not contain any attributes. Any test
+    suite should inherit this class to create their own custom configuration. Finally override the
+    type of the :attr:`~TestSuite.config` to use the newly created one.
+
+    Attributes:
+        test_cases_names: The names of test cases from this test suite to execute. If empty, all
+            test cases will be executed.
+    """
+
+    _test_suite_spec: "TestSuiteSpec"
+
+    test_cases_names: list[str] = Field(default_factory=list, alias="test_cases")
+
+    @property
+    def test_suite_name(self) -> str:
+        """The name of the test suite module without the starting ``TestSuite_``."""
+        return self._test_suite_spec.name
+
+    @property
+    def test_suite_spec(self) -> "TestSuiteSpec":
+        """The specification of the requested test suite."""
+        return self._test_suite_spec
+
+    @model_validator(mode="before")
+    @classmethod
+    def convert_from_string(cls, data: Any) -> Any:
+        """Validator which allows to select a test suite by string instead of a mapping."""
+        if isinstance(data, str):
+            test_cases = [] if data == "all" else data.split()
+            return dict(test_cases=test_cases)
+        return data
+
+    @classmethod
+    def make(cls, test_suite_name: str, *test_cases_names: str, **kwargs) -> Self:
+        """Make a configuration for the requested test suite.
+
+        Args:
+            test_suite_name: The name of the test suite.
+            test_cases_names: The test cases to select, if empty all are selected.
+            **kwargs: Any other configuration field.
+
+        Raises:
+            AssertionError: If the requested test suite or test cases do not exist.
+            ValidationError: If the configuration fields were not filled correctly.
+        """
+        from framework.test_suite import find_by_name
+
+        test_suite_spec = find_by_name(test_suite_name)
+        assert test_suite_spec is not None, f"Could not find test suite '{test_suite_name}'."
+        test_suite_spec.validate_test_cases(test_cases_names)
+
+        config = cls.model_validate({"test_cases": test_cases_names, **kwargs})
+        config._test_suite_spec = test_suite_spec
+        return config
+
+
+class BaseTestSuitesConfigs(BaseModel, extra="forbid"):
+    """Base class for test suites configs."""
+
+    def __contains__(self, key) -> bool:
+        """Check if the provided test suite name has been selected and/or configured."""
+        return key in self.model_fields_set
+
+    def __getitem__(self, key) -> TestSuiteConfig:
+        """Get test suite configuration."""
+        return self.__getattribute__(key)
+
+    def get_configs(self) -> Iterable[TestSuiteConfig]:
+        """Get all the test suite configurations."""
+        return map(lambda t: self[t], self.model_fields_set)
+
+    @classmethod
+    def available_test_suites(cls) -> Iterable[str]:
+        """List all the available test suites."""
+        return cls.model_fields.keys()
+
+    @field_validator("*")
+    @classmethod
+    def validate_test_suite_config(
+        cls, config: type[TestSuiteConfig], info: ValidationInfo
+    ) -> type[TestSuiteConfig]:
+        """Validate the provided test cases and link the test suite spec to the configuration."""
+        from framework.test_suite import find_by_name
+
+        test_suite_name = info.field_name
+        assert test_suite_name is not None
+
+        test_suite_spec = find_by_name(test_suite_name)
+        assert test_suite_spec is not None
+
+        config._test_suite_spec = test_suite_spec
+
+        test_suite_spec.validate_test_cases(config.test_cases_names)
+        return config
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 00b63cc292..bc7aa1555c 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -24,10 +24,13 @@
 from types import FunctionType
 from typing import Iterable
 
+from pydantic import ValidationError
+
 from framework.testbed_model.sut_node import SutNode
 from framework.testbed_model.tg_node import TGNode
 
 from .config import (
+    CUSTOM_CONFIG_TYPES,
     BuildTargetConfiguration,
     Configuration,
     SutNodeConfiguration,
@@ -36,7 +39,12 @@
     TGNodeConfiguration,
     load_config,
 )
-from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError
+from .exception import (
+    BlockingTestSuiteError,
+    ConfigurationError,
+    SSHTimeoutError,
+    TestCaseVerifyError,
+)
 from .logger import DTSLogger, DtsStage, get_dts_logger
 from .settings import SETTINGS
 from .test_result import (
@@ -142,12 +150,7 @@ def run(self) -> None:
                 self._logger.set_stage(DtsStage.test_run_setup)
                 self._logger.info(f"Running test run with SUT '{sut_node_config.name}'.")
                 test_run_result = self._result.add_test_run(test_run_config)
-                # we don't want to modify the original config, so create a copy
-                test_run_test_suites = list(
-                    SETTINGS.test_suites if SETTINGS.test_suites else test_run_config.test_suites
-                )
-                if not test_run_config.skip_smoke_tests:
-                    test_run_test_suites[:0] = [TestSuiteConfig("smoke_tests")]
+                test_run_test_suites = self._prepare_test_suites(test_run_config)
                 try:
                     test_suites_with_cases = self._get_test_suites_with_cases(
                         test_run_test_suites, test_run_config.func, test_run_config.perf
@@ -204,6 +207,40 @@ def _check_dts_python_version(self) -> None:
             )
             self._logger.warning("Please use Python >= 3.10 instead.")
 
+    def _prepare_test_suites(self, test_run_config: TestRunConfiguration) -> list[TestSuiteConfig]:
+        if SETTINGS.test_suites:
+            test_suites_configs = []
+            for selected_test_suite, selected_test_cases in SETTINGS.test_suites:
+                if selected_test_suite in test_run_config.test_suites:
+                    config = test_run_config.test_suites[selected_test_suite].model_copy()
+                    config.test_cases_names = selected_test_cases
+                else:
+                    try:
+                        config = CUSTOM_CONFIG_TYPES[selected_test_suite].make(
+                            selected_test_suite, *selected_test_cases
+                        )
+                    except AssertionError as e:
+                        raise ConfigurationError(
+                            "Invalid test cases were selected "
+                            f"for test suite {selected_test_suite}."
+                        ) from e
+                    except ValidationError as e:
+                        raise ConfigurationError(
+                            f"Test suite {selected_test_suite} needs to be explicitly configured "
+                            "in order to be selected."
+                        ) from e
+                test_suites_configs.append(config)
+        else:
+            # we don't want to modify the original config, so create a copy
+            test_suites_configs = [
+                config.model_copy() for config in test_run_config.test_suites.get_configs()
+            ]
+
+        if not test_run_config.skip_smoke_tests:
+            test_suites_configs[:0] = [TestSuiteConfig.make("smoke_tests")]
+
+        return test_suites_configs
+
     def _get_test_suites_with_cases(
         self,
         test_suite_configs: list[TestSuiteConfig],
@@ -245,7 +282,9 @@ def _get_test_suites_with_cases(
 
             test_suites_with_cases.append(
                 TestSuiteWithCases(
-                    test_suite_class=test_suite_class, test_cases=selected_test_cases
+                    test_suite_class=test_suite_class,
+                    test_cases=selected_test_cases,
+                    config=test_suite_config,
                 )
             )
         return test_suites_with_cases
@@ -466,7 +505,9 @@ def _run_test_suite(
         self._logger.set_stage(
             DtsStage.test_suite_setup, Path(SETTINGS.output_dir, test_suite_name)
         )
-        test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node)
+        test_suite = test_suite_with_cases.test_suite_class(
+            sut_node, tg_node, test_suite_with_cases.config
+        )
         try:
             self._logger.info(f"Starting test suite setup: {test_suite_name}")
             test_suite.set_up_suite()
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 2e8dedef4f..063f282edf 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -85,9 +85,7 @@
 from pathlib import Path
 from typing import Callable
 
-from pydantic import ValidationError
-
-from .config import TestSuiteConfig
+from .config import TestSuitesConfigs
 from .exception import ConfigurationError
 from .utils import DPDKGitTarball, get_commit_id
 
@@ -114,7 +112,7 @@ class Settings:
     #:
     compile_timeout: float = 1200
     #:
-    test_suites: list[TestSuiteConfig] = field(default_factory=list)
+    test_suites: list[tuple[str, list[str]]] = field(default_factory=list)
     #:
     re_run: int = 0
 
@@ -382,7 +380,7 @@ def _get_parser() -> _DTSArgumentParser:
 
 def _process_test_suites(
     parser: _DTSArgumentParser, args: list[list[str]]
-) -> list[TestSuiteConfig]:
+) -> list[tuple[str, list[str]]]:
     """Process the given argument to a list of :class:`TestSuiteConfig` to execute.
 
     Args:
@@ -398,16 +396,17 @@ def _process_test_suites(
         # Environment variable in the form of "SUITE1 CASE1 CASE2, SUITE2 CASE1, SUITE3, ..."
         args = [suite_with_cases.split() for suite_with_cases in args[0][0].split(",")]
 
-    try:
-        return [TestSuiteConfig(test_suite, test_cases) for [test_suite, *test_cases] in args]
-    except ValidationError as e:
-        print(
-            "An error has occurred while validating the test suites supplied in the "
-            f"{'environment variable' if action else 'arguments'}:",
-            file=sys.stderr,
-        )
-        print(e, file=sys.stderr)
-        sys.exit(1)
+    available_test_suites = TestSuitesConfigs.available_test_suites()
+    for test_suite_name, *_ in args:
+        if test_suite_name not in available_test_suites:
+            print(
+                f"The test suite {test_suite_name} supplied in the "
+                f"{'environment variable' if action else 'arguments'} is invalid.",
+                file=sys.stderr,
+            )
+            sys.exit(1)
+
+    return [(test_suite, test_cases) for test_suite, *test_cases in args]
 
 
 def get_settings() -> Settings:
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 5694a2482b..6c10c1e40a 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -64,17 +64,7 @@ class is to hold a subset of test cases (which could be all test cases) because
 
     test_suite_class: type[TestSuite]
     test_cases: list[FunctionType]
-
-    def create_config(self) -> TestSuiteConfig:
-        """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
-
-        Returns:
-            The :class:`TestSuiteConfig` representation.
-        """
-        return TestSuiteConfig(
-            test_suite=self.test_suite_class.__name__,
-            test_cases=[test_case.__name__ for test_case in self.test_cases],
-        )
+    config: TestSuiteConfig
 
 
 class Result(Enum):
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 972968b036..78e1b4c49a 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -23,7 +23,7 @@
 from ipaddress import IPv4Interface, IPv6Interface, ip_interface
 from pkgutil import iter_modules
 from types import FunctionType, ModuleType
-from typing import ClassVar, NamedTuple, Union
+from typing import ClassVar, Iterable, NamedTuple, Union, get_type_hints
 
 from pydantic.alias_generators import to_pascal
 from scapy.layers.inet import IP  # type: ignore[import-untyped]
@@ -31,6 +31,7 @@
 from scapy.packet import Packet, Padding  # type: ignore[import-untyped]
 from typing_extensions import Self
 
+from framework.config import TestSuiteConfig
 from framework.testbed_model.port import Port, PortLink
 from framework.testbed_model.sut_node import SutNode
 from framework.testbed_model.tg_node import TGNode
@@ -38,7 +39,7 @@
     PacketFilteringConfig,
 )
 
-from .exception import TestCaseVerifyError
+from .exception import InternalError, TestCaseVerifyError
 from .logger import DTSLogger, get_dts_logger
 from .utils import get_packet_summaries
 
@@ -78,6 +79,7 @@ class TestSuite:
     #: Whether the test suite is blocking. A failure of a blocking test suite
     #: will block the execution of all subsequent test suites in the current build target.
     is_blocking: ClassVar[bool] = False
+    config: TestSuiteConfig
     _logger: DTSLogger
     _port_links: list[PortLink]
     _sut_port_ingress: Port
@@ -93,6 +95,7 @@ def __init__(
         self,
         sut_node: SutNode,
         tg_node: TGNode,
+        config: TestSuiteConfig,
     ):
         """Initialize the test suite testbed information and basic configuration.
 
@@ -102,9 +105,11 @@ def __init__(
         Args:
             sut_node: The SUT node where the test suite will run.
             tg_node: The TG node where the test suite will run.
+            config: The test suite configuration.
         """
         self.sut_node = sut_node
         self.tg_node = tg_node
+        self.config = config
         self._logger = get_dts_logger(self.__class__.__name__)
         self._port_links = []
         self._process_links()
@@ -467,6 +472,21 @@ def is_test_suite(obj) -> bool:
 
         raise Exception("class not found in eligible test module")
 
+    @cached_property
+    def config_type(self) -> type[TestSuiteConfig]:
+        """A reference to the test suite's configuration type."""
+        fields = get_type_hints(self.class_type)
+        config_type = fields.get("config")
+        if config_type is None:
+            raise InternalError(
+                "Test suite class {self.class_name} is missing the `config` attribute."
+            )
+        if not issubclass(config_type, TestSuiteConfig):
+            raise InternalError(
+                f"Test suite class {self.class_name} has an invalid configuration type assigned."
+            )
+        return config_type
+
     @cached_property
     def test_cases(self) -> list[TestCase]:
         """A list of all the available test cases."""
@@ -533,6 +553,14 @@ def discover_all(
 
         return test_suites
 
+    def validate_test_cases(self, test_cases_names: Iterable[str]) -> None:
+        """Validate if the supplied test cases exist in the test suite."""
+        available_test_cases = map(lambda t: t.name, self.test_cases)
+        for requested_test_case in test_cases_names:
+            assert (
+                requested_test_case in available_test_cases
+            ), f"{requested_test_case} is not a valid test case for test suite {self.name}."
+
 
 AVAILABLE_TEST_SUITES: list[TestSuiteSpec] = TestSuiteSpec.discover_all()
 """Constant to store all the available, discovered and imported test suites.
diff --git a/dts/generate-schema.py b/dts/generate-schema.py
index b41d28492f..d24a67f68e 100755
--- a/dts/generate-schema.py
+++ b/dts/generate-schema.py
@@ -9,7 +9,7 @@
 
 from pydantic.json_schema import GenerateJsonSchema
 
-from framework.config import ConfigurationType
+from framework.config import ConfigurationType, TestSuitesConfigs
 
 DTS_DIR = os.path.dirname(os.path.realpath(__file__))
 RELATIVE_PATH_TO_SCHEMA = "framework/config/conf_yaml_schema.json"
@@ -26,6 +26,8 @@ def generate(self, schema, mode="validation"):
 
 
 try:
+    TestSuitesConfigs.fix_custom_config_annotations()
+
     path = os.path.join(DTS_DIR, RELATIVE_PATH_TO_SCHEMA)
 
     with open(path, "w") as schema_file:
diff --git a/dts/generate-test-mappings.py b/dts/generate-test-mappings.py
new file mode 100755
index 0000000000..d076ad0afe
--- /dev/null
+++ b/dts/generate-test-mappings.py
@@ -0,0 +1,132 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Test Suites to Configuration mappings generation script."""
+
+import os
+from collections import defaultdict
+from textwrap import indent
+from typing import Iterable
+
+from framework.config.test_suite import BaseTestSuitesConfigs, TestSuiteConfig
+from framework.exception import InternalError
+from framework.test_suite import AVAILABLE_TEST_SUITES, TestSuiteSpec
+
+DTS_DIR = os.path.dirname(os.path.realpath(__file__))
+SCRIPT_FILE_NAME = os.path.basename(__file__)
+
+FRAMEWORK_IMPORTS = [BaseTestSuitesConfigs, TestSuiteConfig]
+
+RELATIVE_PATH_TO_GENERATED_FILE = "framework/config/generated.py"
+SMOKE_TESTS_SUITE_NAME = "smoke_tests"
+CUSTOM_CONFIG_TYPES_VAR_NAME = "CUSTOM_CONFIG_TYPES"
+CUSTOM_CONFIG_LINKING_FUNCTION_NAME = "fix_custom_config_annotations"
+CUSTOM_CONFIG_LINKING_FUNCTION_DOCSTRING = [
+    '"""Fixes the custom config types annotations.',
+    "",
+    f"Moreover it also fills `{CUSTOM_CONFIG_TYPES_VAR_NAME}` with all the custom config types.",
+    '"""',
+]
+TEST_SUITES_CONFIG_CLASS_NAME = "TestSuitesConfigs"
+TEST_SUITES_CONFIG_CLASS_DOCSTRING = [
+    '"""Configuration mapping class to select and configure the test suites.',
+    "",
+    "Before using this class, the custom configuration type annotations need to be fixed.",
+    f"To do so, you need to call the `{CUSTOM_CONFIG_LINKING_FUNCTION_NAME}` method.",
+    '"""',
+]
+
+
+GENERATED_FILE_HEADER = f"""# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 The DPDK contributors
+# This file is automatically generated by {SCRIPT_FILE_NAME}.
+# Do NOT modify this file manually.
+
+\"\"\"Generated file containing the links between the test suites and the configuration.\"\"\"
+"""
+
+
+def join(lines: Iterable[str]) -> str:
+    """Join list of strings into text lines."""
+    return "\n".join(lines)
+
+
+def join_and_indent(lines: Iterable[str], indentation_level=1, indentation_spaces=4) -> str:
+    """Join list of strings into indented text lines."""
+    return "\n".join([indent(line, " " * indentation_level * indentation_spaces) for line in lines])
+
+
+def format_attributes_types(test_suite_spec: TestSuiteSpec):
+    """Format the config type into the respective configuration class field attribute type."""
+    config_type = test_suite_spec.config_type.__name__
+    if config_type != TestSuiteConfig.__name__:
+        config_type = f'"{config_type}"'
+    return f"Optional[{config_type}]"
+
+
+try:
+    framework_imports: dict[str, list[str]] = defaultdict(list)
+    for _import in FRAMEWORK_IMPORTS:
+        framework_imports[_import.__module__].append(_import.__name__)
+    formatted_framework_imports = sorted(
+        [
+            f"from {module} import {', '.join(sorted(imports))}"
+            for module, imports in framework_imports.items()
+        ]
+    )
+
+    test_suites = [
+        test_suite_spec
+        for test_suite_spec in AVAILABLE_TEST_SUITES
+        if test_suite_spec.name != SMOKE_TESTS_SUITE_NAME
+    ]
+
+    custom_configs = [t for t in test_suites if t.config_type is not TestSuiteConfig]
+
+    custom_config_imports = [
+        f"from {t.module_type.__name__} import {t.config_type.__name__}" for t in custom_configs
+    ]
+
+    test_suites_attributes = [f"{t.name}: {format_attributes_types(t)} = None" for t in test_suites]
+
+    custom_config_assignments = [
+        f'{CUSTOM_CONFIG_TYPES_VAR_NAME}["{t.name}"] = {t.config_type.__name__}'
+        for t in custom_configs
+    ]
+
+    generated_file_contents = f"""{GENERATED_FILE_HEADER}
+from typing import TYPE_CHECKING, Optional
+
+{join(formatted_framework_imports)}
+
+if TYPE_CHECKING:
+{join_and_indent(custom_config_imports)}
+
+
+{CUSTOM_CONFIG_TYPES_VAR_NAME}: dict[str, type[{TestSuiteConfig.__name__}]] = {'{}'}
+
+
+class {TEST_SUITES_CONFIG_CLASS_NAME}({BaseTestSuitesConfigs.__name__}):
+{join_and_indent(TEST_SUITES_CONFIG_CLASS_DOCSTRING)}
+
+{join_and_indent(test_suites_attributes)}
+
+    @classmethod
+    def {CUSTOM_CONFIG_LINKING_FUNCTION_NAME}(cls):
+{join_and_indent(CUSTOM_CONFIG_LINKING_FUNCTION_DOCSTRING, indentation_level=2)}
+{join_and_indent(custom_config_imports, indentation_level=2)}
+
+{join_and_indent(custom_config_assignments, indentation_level=2)}
+
+        cls.model_rebuild()
+"""
+
+    path = os.path.join(DTS_DIR, RELATIVE_PATH_TO_GENERATED_FILE)
+
+    with open(path, "w") as generated_file:
+        generated_file.write(generated_file_contents)
+
+    print("Test suites to configuration mappings generated successfully!")
+except Exception as e:
+    raise InternalError("Failed to generate test suites to configuration mappings.") from e
diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
index d958f99030..1723c123bc 100644
--- a/dts/tests/TestSuite_hello_world.py
+++ b/dts/tests/TestSuite_hello_world.py
@@ -7,6 +7,7 @@
 No other EAL parameters apart from cores are used.
 """
 
+from framework.config import TestSuiteConfig
 from framework.remote_session.dpdk_shell import compute_eal_params
 from framework.test_suite import TestSuite
 from framework.testbed_model.cpu import (
@@ -16,9 +17,18 @@
 )
 
 
+class HelloWorldConfig(TestSuiteConfig):
+    """Example custom configuration for the `TestHelloWorld` test suite."""
+
+    #: Timeout for the DPDK apps
+    timeout: int = 50
+
+
 class TestHelloWorld(TestSuite):
     """DPDK hello world app test suite."""
 
+    config: HelloWorldConfig
+
     def set_up_suite(self) -> None:
         """Set up the test suite.
 
@@ -59,7 +69,7 @@ def test_hello_world_all_cores(self) -> None:
         eal_para = compute_eal_params(
             self.sut_node, lcore_filter_specifier=LogicalCoreList(self.sut_node.lcores)
         )
-        result = self.sut_node.run_dpdk_app(self.app_helloworld_path, eal_para, 50)
+        result = self.sut_node.run_dpdk_app(self.app_helloworld_path, eal_para, self.config.timeout)
         for lcore in self.sut_node.lcores:
             self.verify(
                 f"hello from core {int(lcore)}" in result.stdout,
diff --git a/dts/tests/__init__.py b/dts/tests/__init__.py
new file mode 100644
index 0000000000..a300eb26fc
--- /dev/null
+++ b/dts/tests/__init__.py
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Test suites.
+
+This package contains all the available test suites in DTS.
+"""
-- 
2.34.1


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH] dts: add per-test-suite configuration
  2024-09-06 16:13 [PATCH] dts: add per-test-suite configuration Luca Vizzarro
@ 2024-09-27 17:45 ` Jeremy Spewock
  2024-11-08 13:38 ` [PATCH v2 0/4] " Luca Vizzarro
  2024-11-08 13:45 ` [PATCH v3 0/4] dts: add per-test-suite configuration Luca Vizzarro
  2 siblings, 0 replies; 17+ messages in thread
From: Jeremy Spewock @ 2024-09-27 17:45 UTC (permalink / raw)
  To: Luca Vizzarro
  Cc: dev, Honnappa Nagarahalli, Paul Szczepanek, Alex Chapman,
	Juraj Linkeš

Hi Luca,

I apologize for not giving this patch more time/attention. I think
this patch is worth having a more in-depth discussion about since it
does add quite a bit more complexity, but the benefit of that is some
valuable simplicity. I know we discussed in the recent team meeting
about whether or not to have the test suite configurations in
conf.yaml or make ones specifically for each individual test suite,
and I think that it could really go either way, but that them being in
conf.yaml is potentially slightly better in my opinion as it helps
keep things uniform, but obviously you get into trouble there if there
is a lot of configuration and the file gets long.

On Fri, Sep 6, 2024 at 12:14 PM Luca Vizzarro <luca.vizzarro@arm.com> wrote:
>
> Allow test suites to be configured individually. Moreover enable them to
> implement their own custom configuration.
>
> This solution adds some new complexity to DTS, which is generated source
> code. In order to ensure strong typing, the test suites and their custom

Generated source code is definitely something new and interesting. I
think the way you did it makes sense and it's well written, but the
idea of generated source code scares me slightly. It leads to the sort
of additional requirement when you are writing test suites to
regenerate the code if you have a custom config, and seems a little
harder to maintain (although, I doubt much maintenance will really be
needed on it, so maybe that's not a good point). It makes me wonder if
the very dumb approach of test suites just each getting a yaml file
that matches the same of the test suite (and maybe ends with _conf)
and then importing that into a TypedDict like we do with conf.yaml
would be worth it to save on this complexity. It would still be simple
enough for a test suite developer to just throw a yaml file into a
directory (maybe conf/ ?) and have the framework auto-magically
consume that and match it to a TypedDict. I guess there would be some
extra rules in place with the dumb approach like the name of the
config file and maybe the name of the TypedDict, but it still feels
like it would save on complexity in the long run. Of course, that's
not to say complexity is bad if it makes things easier, but it begs
the question of how much easier is doing it this way versus making a
yaml file and a matching TypedDict.

Again, something good to discuss in a meeting.

> configurations need to be linked in the main configuration class.
> Unfortunately, this is not feasible during runtime as it will incur in
> circular dependencies. Generating the links appear to be the most
> straightforward approach.
>
> This commit also brings a new major change to the configuration schema.
> Test suites are no longer defined as a list of strings, like:
>
>     test_suites:
>     - hello_world
>     - pmd_buffer_scatter
>
> but as mapping of mappings or strings:
>
>     test_suites:
>       hello_world: {} # any custom fields or test cases can be set here
>       pmd_buffer_scatter: all # "all" defines all the test cases, or
>                               # they can individually be set separated
>                               # by a space
>
> Not defining the `test_cases` field in the configuration is equivalent
> to `all`, therefore the definitions for either test suite above are
> also equivalent.

Making these mappings in general however that also allow you to
specify which test cases to run I really like. It doesn't seem
completely relevant to the suite-wide configurations though, maybe it
would make sense to be in a different patch? I get why it's all in one
though since it they are handled in the same places.

>
> Creating the __init__.py file under the tests folder, allows it to be
> picked up as a package. This is a mypy requirement to import the tests
> from within the framework.
>
> Bugzilla ID: 1375
>
> Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
> Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Reviewed-by: Alex Chapman <alex.chapman@arm.com>

I didn't really have any comments on this implementation overall
however. I didn't fully understand everything (mostly due to my lack
of pydantic understanding) so I probably would have had to read it
over a few more times to completely understand it, but in general I
think this seems well written. Great work Luca.

> ---
> Depends-on: series-32823 ("dts: Pydantic configuration")
>
> Hello,
>
> sending in a solution for the per-test-suite configuration issue.
> This one took some thinking but I have given most of the motivations in
> the commit body already. The docs are somewhat lacking but hopefully they
> should be automatically tackled by the API docs generation.
>
>
> Best,
> Luca
> ---
>  doc/guides/tools/dts.rst                   |  39 ++++--
>  dts/conf.yaml                              |   4 +-
>  dts/framework/config/__init__.py           |  98 ++-------------
>  dts/framework/config/conf_yaml_schema.json |  94 +++++++++++---
>  dts/framework/config/generated.py          |  40 ++++++
>  dts/framework/config/test_suite.py         | 140 +++++++++++++++++++++
>  dts/framework/runner.py                    |  59 +++++++--
>  dts/framework/settings.py                  |  29 +++--
>  dts/framework/test_result.py               |  12 +-
>  dts/framework/test_suite.py                |  32 ++++-
>  dts/generate-schema.py                     |   4 +-
>  dts/generate-test-mappings.py              | 132 +++++++++++++++++++
>  dts/tests/TestSuite_hello_world.py         |  12 +-
>  dts/tests/__init__.py                      |   7 ++
>  14 files changed, 539 insertions(+), 163 deletions(-)
>  create mode 100644 dts/framework/config/generated.py
>  create mode 100644 dts/framework/config/test_suite.py
>  create mode 100755 dts/generate-test-mappings.py
>  create mode 100644 dts/tests/__init__.py
>
> diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
> index 317bd0ff99..66681543cd 100644
> --- a/doc/guides/tools/dts.rst
> +++ b/doc/guides/tools/dts.rst
> @@ -391,6 +391,26 @@ There are four types of methods that comprise a test suite:
>     should be implemented in the ``SutNode`` class (and the underlying classes that ``SutNode`` uses)
>     and used by the test suite via the ``sut_node`` field.
>
> +The test suites can also implement their own custom configuration fields. This can be achieved by
> +creating a new test suite config file which inherits from ``TestSuiteConfig`` defined in
> +``dts/framework/config/test_suite.py``. So that this new custom configuration class is used, the
> +test suite class must override the ``config`` attribute annotation with your new class, for example::
> +
> +.. code:: python
> +   class CustomConfig(TestSuiteConfig):
> +      my_custom_field: int = 10
> +
> +   class TestMyNewTestSuite(TestSuite):
> +      config: CustomConfig
> +
> +Finally, the test suites and the custom configuration files need to linked in the global configuration.
> +This can be easily achieved by running the ``dts/generate-test-mappings.py``, e.g.:
> +
> +.. code-block:: console
> +
> +   $ poetry shell
> +   (dts-py3.10) $ ./generate-test-mappings.py
> +
>
>  .. _dts_dev_tools:
>
> @@ -510,18 +530,13 @@ _`Network port`
>     ``peer_pci``           *string* – the PCI address of the peer node port. **Example**: ``000a:01:00.1``
>     ====================== =================================================================================
>
> -_`Test suite`
> -   *string* – name of the test suite to run. **Examples**: ``hello_world``, ``os_udp``
> -
> -_`Test target`
> -   *mapping* – selects specific test cases to run from a test suite. Mapping is described as follows:
> -
> -   ========= ===============================================================================================
> -   ``suite`` See `Test suite`_
> -   ``cases`` (*optional*) *sequence* of *string* – list of the selected test cases in the test suite to run.
> +_`Test suites`
> +   *mapping* – selects the test suites to run. Each mapping key corresponds to the test suite name.
>
> -             Unknown test cases will be silently ignored.
> -   ========= ===============================================================================================
> +   The value of the mapping can either "all" to select all the test cases in that test suite, the test
> +   cases names divided by a space. Or it can be another mapping to set any custom fields for the test suite.
> +   In the case of a mapping, all the test cases are selected by default. In order to manually select test
> +   cases, the ``test_cases`` field can be set with a list of strings, each entry being a test case name.
>
>
>  Properties
> @@ -542,7 +557,7 @@ involved in the testing. These can be defined with the following mappings:
>     +----------------------------+-------------------------------------------------------------------+
>     | ``func``                   | *boolean* – Enable functional testing.                            |
>     +----------------------------+-------------------------------------------------------------------+
> -   | ``test_suites``            | *sequence* of **one of** `Test suite`_ **or** `Test target`_      |
> +   | ``test_suites``            | See `Test suites`_                                                |
>     +----------------------------+-------------------------------------------------------------------+
>     | ``skip_smoke_tests``       | (*optional*) *boolean* – Allows you to skip smoke testing         |
>     |                            | if ``true``.                                                      |
> diff --git a/dts/conf.yaml b/dts/conf.yaml
> index 7d95016e68..c44bef604c 100644
> --- a/dts/conf.yaml
> +++ b/dts/conf.yaml
> @@ -15,8 +15,8 @@ test_runs:
>      func: true # enable functional testing
>      skip_smoke_tests: false # optional
>      test_suites: # the following test suites will be run in their entirety
> -      - hello_world
> -      - os_udp
> +      hello_world: all
> +      os_udp: all
>      # The machine running the DPDK test executable
>      system_under_test_node:
>        node_name: "SUT 1"
> diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
> index 013c529829..d39e0823bd 100644
> --- a/dts/framework/config/__init__.py
> +++ b/dts/framework/config/__init__.py
> @@ -35,10 +35,12 @@
>        and makes it thread safe should we ever want to move in that direction.
>  """
>
> +# pylama:ignore=W0611
> +
>  from enum import Enum, auto, unique
>  from functools import cached_property
>  from pathlib import Path
> -from typing import TYPE_CHECKING, Annotated, Any, Literal, NamedTuple, Protocol
> +from typing import Annotated, Literal, NamedTuple, Protocol
>
>  import yaml
>  from pydantic import (
> @@ -50,15 +52,14 @@
>      field_validator,
>      model_validator,
>  )
> -from pydantic.config import JsonDict
>  from pydantic.dataclasses import dataclass
>  from typing_extensions import Self
>
>  from framework.exception import ConfigurationError
>  from framework.utils import StrEnum
>
> -if TYPE_CHECKING:
> -    from framework.test_suite import TestSuiteSpec
> +from .generated import CUSTOM_CONFIG_TYPES, TestSuitesConfigs
> +from .test_suite import TestSuiteConfig
>
>
>  @unique
> @@ -289,7 +290,7 @@ class NodeInfo:
>      kernel_version: str
>
>
> -@dataclass(slots=True, frozen=True, kw_only=True, config=ConfigDict(extra="forbid"))
> +@dataclass(frozen=True, kw_only=True, config=ConfigDict(extra="forbid"))
>  class BuildTargetConfiguration:
>      """DPDK build configuration.
>
> @@ -329,89 +330,6 @@ class BuildTargetInfo:
>      compiler_version: str
>
>
> -def make_parsable_schema(schema: JsonDict):
> -    """Updates a model's JSON schema to make a string representation a valid alternative.
> -
> -    This utility function is required to be used with models that can be represented and validated
> -    as a string instead of an object mapping. Normally the generated JSON schema will just show
> -    the object mapping. This function wraps the mapping under an anyOf property sequenced with a
> -    string type.
> -
> -    This function is a valid `Callable` for the `json_schema_extra` attribute of
> -    `~pydantic.config.ConfigDict`.
> -    """
> -    inner_schema = schema.copy()
> -    del inner_schema["title"]
> -
> -    title = schema.get("title")
> -    description = schema.get("description")
> -
> -    schema.clear()
> -
> -    schema["title"] = title
> -    schema["description"] = description
> -    schema["anyOf"] = [inner_schema, {"type": "string"}]
> -
> -
> -@dataclass(
> -    frozen=True,
> -    config=ConfigDict(extra="forbid", json_schema_extra=make_parsable_schema),
> -)
> -class TestSuiteConfig:
> -    """Test suite configuration.
> -
> -    Information about a single test suite to be executed. It can be represented and validated as a
> -    string type in the form of: ``TEST_SUITE [TEST_CASE, ...]``, in the configuration file.
> -
> -    Attributes:
> -        test_suite: The name of the test suite module without the starting ``TestSuite_``.
> -        test_cases: The names of test cases from this test suite to execute.
> -            If empty, all test cases will be executed.
> -    """
> -
> -    test_suite_name: str = Field(
> -        title="Test suite name",
> -        description="The identifying name of the test suite.",
> -        alias="test_suite",
> -    )
> -    test_cases_names: list[str] = Field(
> -        default_factory=list,
> -        title="Test cases by name",
> -        description="The identifying name of the test cases of the test suite.",
> -        alias="test_cases",
> -    )
> -
> -    @cached_property
> -    def test_suite_spec(self) -> "TestSuiteSpec":
> -        """The specification of the requested test suite."""
> -        from framework.test_suite import find_by_name
> -
> -        test_suite_spec = find_by_name(self.test_suite_name)
> -        assert test_suite_spec is not None, f"{self.test_suite_name} is not a valid test suite name"
> -        return test_suite_spec
> -
> -    @model_validator(mode="before")
> -    @classmethod
> -    def convert_from_string(cls, data: Any) -> Any:
> -        """Convert the string representation into a valid mapping."""
> -        if isinstance(data, str):
> -            [test_suite, *test_cases] = data.split()
> -            return dict(test_suite=test_suite, test_cases=test_cases)
> -        return data
> -
> -    @model_validator(mode="after")
> -    def validate_names(self) -> Self:
> -        """Validate the supplied test suite and test cases names."""
> -        available_test_cases = map(lambda t: t.name, self.test_suite_spec.test_cases)
> -        for requested_test_case in self.test_cases_names:
> -            assert requested_test_case in available_test_cases, (
> -                f"{requested_test_case} is not a valid test case "
> -                f"for test suite {self.test_suite_name}"
> -            )
> -
> -        return self
> -
> -
>  @dataclass(slots=True, frozen=True, kw_only=True, config=ConfigDict(extra="forbid"))
>  class TestRunSUTNodeConfiguration:
>      """The SUT node configuration of a test run.
> @@ -446,7 +364,7 @@ class TestRunConfiguration:
>      perf: bool = Field(description="Enable performance testing.")
>      func: bool = Field(description="Enable functional testing.")
>      skip_smoke_tests: bool = False
> -    test_suites: list[TestSuiteConfig] = Field(min_length=1)
> +    test_suites: TestSuitesConfigs
>      system_under_test_node: TestRunSUTNodeConfiguration
>      traffic_generator_node: str
>
> @@ -581,7 +499,7 @@ def load_config(config_file_path: Path) -> Configuration:
>          config_data = yaml.safe_load(f)
>
>      try:
> -        ConfigurationType.json_schema()
> +        TestSuitesConfigs.fix_custom_config_annotations()
>          return ConfigurationType.validate_python(config_data)
>      except ValidationError as e:
>          raise ConfigurationError("failed to load the supplied configuration") from e
> diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
> index 1cf1bb098a..1434cdfad3 100644
> --- a/dts/framework/config/conf_yaml_schema.json
> +++ b/dts/framework/config/conf_yaml_schema.json
> @@ -66,6 +66,33 @@
>        "title": "Compiler",
>        "type": "string"
>      },
> +    "HelloWorldConfig": {
> +      "anyOf": [
> +        {
> +          "additionalProperties": false,
> +          "properties": {
> +            "test_cases": {
> +              "items": {
> +                "type": "string"
> +              },
> +              "title": "Test Cases",
> +              "type": "array"
> +            },
> +            "timeout": {
> +              "default": 50,
> +              "title": "Timeout",
> +              "type": "integer"
> +            }
> +          },
> +          "type": "object"
> +        },
> +        {
> +          "type": "string"
> +        }
> +      ],
> +      "description": "Example custom configuration for the `TestHelloWorld` test suite.",
> +      "title": "HelloWorldConfig"
> +    },
>      "HugepageConfiguration": {
>        "additionalProperties": false,
>        "description": "The hugepage configuration of :class:`~framework.testbed_model.node.Node`\\s.\n\nAttributes:\n    number_of: The number of hugepages to allocate.\n    force_first_numa: If :data:`True`, the hugepages will be configured on the first NUMA node.",
> @@ -373,14 +400,6 @@
>            "title": "Func",
>            "type": "boolean"
>          },
> -        "test_suites": {
> -          "items": {
> -            "$ref": "#/$defs/TestSuiteConfig"
> -          },
> -          "minItems": 1,
> -          "title": "Test Suites",
> -          "type": "array"
> -        },
>          "build_targets": {
>            "items": {
>              "$ref": "#/$defs/BuildTargetConfiguration"
> @@ -393,6 +412,9 @@
>            "title": "Skip Smoke Tests",
>            "type": "boolean"
>          },
> +        "test_suites": {
> +          "$ref": "#/$defs/TestSuitesConfigs"
> +        },
>          "system_under_test_node": {
>            "$ref": "#/$defs/TestRunSUTNodeConfiguration"
>          },
> @@ -404,8 +426,8 @@
>        "required": [
>          "perf",
>          "func",
> -        "test_suites",
>          "build_targets",
> +        "test_suites",
>          "system_under_test_node",
>          "traffic_generator_node"
>        ],
> @@ -439,31 +461,63 @@
>          {
>            "additionalProperties": false,
>            "properties": {
> -            "test_suite": {
> -              "description": "The identifying name of the test suite.",
> -              "title": "Test suite name",
> -              "type": "string"
> -            },
>              "test_cases": {
> -              "description": "The identifying name of the test cases of the test suite.",
>                "items": {
>                  "type": "string"
>                },
> -              "title": "Test cases by name",
> +              "title": "Test Cases",
>                "type": "array"
>              }
>            },
> -          "required": [
> -            "test_suite"
> -          ],
>            "type": "object"
>          },
>          {
>            "type": "string"
>          }
>        ],
> -      "description": "Test suite configuration.\n\nInformation about a single test suite to be executed. It can be represented and validated as a\nstring type in the form of: ``TEST_SUITE [TEST_CASE, ...]``, in the configuration file.\n\nAttributes:\n    test_suite: The name of the test suite module without the starting ``TestSuite_``.\n    test_cases: The names of test cases from this test suite to execute.\n        If empty, all test cases will be executed.",
> +      "description": "Test suite configuration base model.\n\nBy default the configuration of a generic test suite does not contain any attributes. Any test\nsuite should inherit this class to create their own custom configuration. Finally override the\ntype of the :attr:`~TestSuite.config` to use the newly created one.\n\nAttributes:\n    test_cases_names: The names of test cases from this test suite to execute. If empty, all\n        test cases will be executed.",
>        "title": "TestSuiteConfig"
> +    },
> +    "TestSuitesConfigs": {
> +      "additionalProperties": false,
> +      "description": "Configuration mapping class to select and configure the test suites.\n\nBefore using this class, the custom configuration type annotations need to be fixed.\nTo do so, you need to call the `fix_custom_config_annotations` method.",
> +      "properties": {
> +        "hello_world": {
> +          "anyOf": [
> +            {
> +              "$ref": "#/$defs/HelloWorldConfig"
> +            },
> +            {
> +              "type": "null"
> +            }
> +          ],
> +          "default": null
> +        },
> +        "os_udp": {
> +          "anyOf": [
> +            {
> +              "$ref": "#/$defs/TestSuiteConfig"
> +            },
> +            {
> +              "type": "null"
> +            }
> +          ],
> +          "default": null
> +        },
> +        "pmd_buffer_scatter": {
> +          "anyOf": [
> +            {
> +              "$ref": "#/$defs/TestSuiteConfig"
> +            },
> +            {
> +              "type": "null"
> +            }
> +          ],
> +          "default": null
> +        }
> +      },
> +      "title": "TestSuitesConfigs",
> +      "type": "object"
>      }
>    },
>    "description": "DTS testbed and test configuration.\n\nAttributes:\n    test_runs: Test run configurations.\n    nodes: Node configurations.",
> diff --git a/dts/framework/config/generated.py b/dts/framework/config/generated.py
> new file mode 100644
> index 0000000000..d42ce93a51
> --- /dev/null
> +++ b/dts/framework/config/generated.py
> @@ -0,0 +1,40 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2024 The DPDK contributors
> +# This file is automatically generated by generate-test-mappings.py.
> +# Do NOT modify this file manually.
> +
> +"""Generated file containing the links between the test suites and the configuration."""
> +
> +from typing import TYPE_CHECKING, Optional
> +
> +from framework.config.test_suite import BaseTestSuitesConfigs, TestSuiteConfig
> +
> +if TYPE_CHECKING:
> +    from tests.TestSuite_hello_world import HelloWorldConfig
> +
> +
> +CUSTOM_CONFIG_TYPES: dict[str, type[TestSuiteConfig]] = {}
> +
> +
> +class TestSuitesConfigs(BaseTestSuitesConfigs):
> +    """Configuration mapping class to select and configure the test suites.
> +
> +    Before using this class, the custom configuration type annotations need to be fixed.
> +    To do so, you need to call the `fix_custom_config_annotations` method.
> +    """
> +
> +    hello_world: Optional["HelloWorldConfig"] = None
> +    os_udp: Optional[TestSuiteConfig] = None
> +    pmd_buffer_scatter: Optional[TestSuiteConfig] = None
> +
> +    @classmethod
> +    def fix_custom_config_annotations(cls):
> +        """Fixes the custom config types annotations.
> +
> +        Moreover it also fills `CUSTOM_CONFIG_TYPES` with all the custom config types.
> +        """
> +        from tests.TestSuite_hello_world import HelloWorldConfig
> +
> +        CUSTOM_CONFIG_TYPES["hello_world"] = HelloWorldConfig
> +
> +        cls.model_rebuild()
> diff --git a/dts/framework/config/test_suite.py b/dts/framework/config/test_suite.py
> new file mode 100644
> index 0000000000..0c1ddf9d95
> --- /dev/null
> +++ b/dts/framework/config/test_suite.py
> @@ -0,0 +1,140 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2024 Arm Limited
> +
> +"""Test suites configuration module.
> +
> +Test suites can inherit :class:`TestSuiteConfig` to create their own custom configuration.
> +By doing so, the test suite class must also override the annotation of the field
> +`~framework.test_suite.TestSuite.config` to use their custom configuration type.
> +"""
> +
> +from typing import TYPE_CHECKING, Any, Iterable
> +
> +from pydantic import BaseModel, Field, ValidationInfo, field_validator, model_validator
> +from pydantic.config import JsonDict
> +from typing_extensions import Self
> +
> +if TYPE_CHECKING:
> +    from framework.test_suite import TestSuiteSpec
> +
> +
> +def make_parsable_schema(schema: JsonDict):
> +    """Updates a model's JSON schema to make a string representation a valid alternative.
> +
> +    This utility function is required to be used with models that can be represented and validated
> +    as a string instead of an object mapping. Normally the generated JSON schema will just show
> +    the object mapping. This function wraps the mapping under an anyOf property sequenced with a
> +    string type.
> +
> +    This function is a valid `Callable` for the `json_schema_extra` attribute of
> +    `~pydantic.config.ConfigDict`.
> +    """
> +    inner_schema = schema.copy()
> +
> +    fields_to_preserve = ["title", "description"]
> +    extracted_fields = {k: v for k in fields_to_preserve if (v := inner_schema.get(k))}
> +    for field in extracted_fields:
> +        del inner_schema[field]
> +
> +    schema.clear()
> +    schema.update(extracted_fields)
> +    schema["anyOf"] = [inner_schema, {"type": "string"}]
> +
> +
> +class TestSuiteConfig(BaseModel, extra="forbid", json_schema_extra=make_parsable_schema):
> +    """Test suite configuration base model.
> +
> +    By default the configuration of a generic test suite does not contain any attributes. Any test
> +    suite should inherit this class to create their own custom configuration. Finally override the
> +    type of the :attr:`~TestSuite.config` to use the newly created one.
> +
> +    Attributes:
> +        test_cases_names: The names of test cases from this test suite to execute. If empty, all
> +            test cases will be executed.
> +    """
> +
> +    _test_suite_spec: "TestSuiteSpec"
> +
> +    test_cases_names: list[str] = Field(default_factory=list, alias="test_cases")
> +
> +    @property
> +    def test_suite_name(self) -> str:
> +        """The name of the test suite module without the starting ``TestSuite_``."""
> +        return self._test_suite_spec.name
> +
> +    @property
> +    def test_suite_spec(self) -> "TestSuiteSpec":
> +        """The specification of the requested test suite."""
> +        return self._test_suite_spec
> +
> +    @model_validator(mode="before")
> +    @classmethod
> +    def convert_from_string(cls, data: Any) -> Any:
> +        """Validator which allows to select a test suite by string instead of a mapping."""
> +        if isinstance(data, str):
> +            test_cases = [] if data == "all" else data.split()
> +            return dict(test_cases=test_cases)
> +        return data
> +
> +    @classmethod
> +    def make(cls, test_suite_name: str, *test_cases_names: str, **kwargs) -> Self:
> +        """Make a configuration for the requested test suite.
> +
> +        Args:
> +            test_suite_name: The name of the test suite.
> +            test_cases_names: The test cases to select, if empty all are selected.
> +            **kwargs: Any other configuration field.
> +
> +        Raises:
> +            AssertionError: If the requested test suite or test cases do not exist.
> +            ValidationError: If the configuration fields were not filled correctly.
> +        """
> +        from framework.test_suite import find_by_name
> +
> +        test_suite_spec = find_by_name(test_suite_name)
> +        assert test_suite_spec is not None, f"Could not find test suite '{test_suite_name}'."
> +        test_suite_spec.validate_test_cases(test_cases_names)
> +
> +        config = cls.model_validate({"test_cases": test_cases_names, **kwargs})
> +        config._test_suite_spec = test_suite_spec
> +        return config
> +
> +
> +class BaseTestSuitesConfigs(BaseModel, extra="forbid"):
> +    """Base class for test suites configs."""
> +
> +    def __contains__(self, key) -> bool:
> +        """Check if the provided test suite name has been selected and/or configured."""
> +        return key in self.model_fields_set
> +
> +    def __getitem__(self, key) -> TestSuiteConfig:
> +        """Get test suite configuration."""
> +        return self.__getattribute__(key)
> +
> +    def get_configs(self) -> Iterable[TestSuiteConfig]:
> +        """Get all the test suite configurations."""
> +        return map(lambda t: self[t], self.model_fields_set)
> +
> +    @classmethod
> +    def available_test_suites(cls) -> Iterable[str]:
> +        """List all the available test suites."""
> +        return cls.model_fields.keys()
> +
> +    @field_validator("*")
> +    @classmethod
> +    def validate_test_suite_config(
> +        cls, config: type[TestSuiteConfig], info: ValidationInfo
> +    ) -> type[TestSuiteConfig]:
> +        """Validate the provided test cases and link the test suite spec to the configuration."""
> +        from framework.test_suite import find_by_name
> +
> +        test_suite_name = info.field_name
> +        assert test_suite_name is not None
> +
> +        test_suite_spec = find_by_name(test_suite_name)
> +        assert test_suite_spec is not None
> +
> +        config._test_suite_spec = test_suite_spec
> +
> +        test_suite_spec.validate_test_cases(config.test_cases_names)
> +        return config
> diff --git a/dts/framework/runner.py b/dts/framework/runner.py
> index 00b63cc292..bc7aa1555c 100644
> --- a/dts/framework/runner.py
> +++ b/dts/framework/runner.py
> @@ -24,10 +24,13 @@
>  from types import FunctionType
>  from typing import Iterable
>
> +from pydantic import ValidationError
> +
>  from framework.testbed_model.sut_node import SutNode
>  from framework.testbed_model.tg_node import TGNode
>
>  from .config import (
> +    CUSTOM_CONFIG_TYPES,
>      BuildTargetConfiguration,
>      Configuration,
>      SutNodeConfiguration,
> @@ -36,7 +39,12 @@
>      TGNodeConfiguration,
>      load_config,
>  )
> -from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError
> +from .exception import (
> +    BlockingTestSuiteError,
> +    ConfigurationError,
> +    SSHTimeoutError,
> +    TestCaseVerifyError,
> +)
>  from .logger import DTSLogger, DtsStage, get_dts_logger
>  from .settings import SETTINGS
>  from .test_result import (
> @@ -142,12 +150,7 @@ def run(self) -> None:
>                  self._logger.set_stage(DtsStage.test_run_setup)
>                  self._logger.info(f"Running test run with SUT '{sut_node_config.name}'.")
>                  test_run_result = self._result.add_test_run(test_run_config)
> -                # we don't want to modify the original config, so create a copy
> -                test_run_test_suites = list(
> -                    SETTINGS.test_suites if SETTINGS.test_suites else test_run_config.test_suites
> -                )
> -                if not test_run_config.skip_smoke_tests:
> -                    test_run_test_suites[:0] = [TestSuiteConfig("smoke_tests")]
> +                test_run_test_suites = self._prepare_test_suites(test_run_config)
>                  try:
>                      test_suites_with_cases = self._get_test_suites_with_cases(
>                          test_run_test_suites, test_run_config.func, test_run_config.perf
> @@ -204,6 +207,40 @@ def _check_dts_python_version(self) -> None:
>              )
>              self._logger.warning("Please use Python >= 3.10 instead.")
>
> +    def _prepare_test_suites(self, test_run_config: TestRunConfiguration) -> list[TestSuiteConfig]:
> +        if SETTINGS.test_suites:
> +            test_suites_configs = []
> +            for selected_test_suite, selected_test_cases in SETTINGS.test_suites:
> +                if selected_test_suite in test_run_config.test_suites:
> +                    config = test_run_config.test_suites[selected_test_suite].model_copy()
> +                    config.test_cases_names = selected_test_cases
> +                else:
> +                    try:
> +                        config = CUSTOM_CONFIG_TYPES[selected_test_suite].make(
> +                            selected_test_suite, *selected_test_cases
> +                        )
> +                    except AssertionError as e:
> +                        raise ConfigurationError(
> +                            "Invalid test cases were selected "
> +                            f"for test suite {selected_test_suite}."
> +                        ) from e
> +                    except ValidationError as e:
> +                        raise ConfigurationError(
> +                            f"Test suite {selected_test_suite} needs to be explicitly configured "
> +                            "in order to be selected."
> +                        ) from e
> +                test_suites_configs.append(config)
> +        else:
> +            # we don't want to modify the original config, so create a copy
> +            test_suites_configs = [
> +                config.model_copy() for config in test_run_config.test_suites.get_configs()
> +            ]
> +
> +        if not test_run_config.skip_smoke_tests:
> +            test_suites_configs[:0] = [TestSuiteConfig.make("smoke_tests")]
> +
> +        return test_suites_configs
> +
>      def _get_test_suites_with_cases(
>          self,
>          test_suite_configs: list[TestSuiteConfig],
> @@ -245,7 +282,9 @@ def _get_test_suites_with_cases(
>
>              test_suites_with_cases.append(
>                  TestSuiteWithCases(
> -                    test_suite_class=test_suite_class, test_cases=selected_test_cases
> +                    test_suite_class=test_suite_class,
> +                    test_cases=selected_test_cases,
> +                    config=test_suite_config,
>                  )
>              )
>          return test_suites_with_cases
> @@ -466,7 +505,9 @@ def _run_test_suite(
>          self._logger.set_stage(
>              DtsStage.test_suite_setup, Path(SETTINGS.output_dir, test_suite_name)
>          )
> -        test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node)
> +        test_suite = test_suite_with_cases.test_suite_class(
> +            sut_node, tg_node, test_suite_with_cases.config
> +        )
>          try:
>              self._logger.info(f"Starting test suite setup: {test_suite_name}")
>              test_suite.set_up_suite()
> diff --git a/dts/framework/settings.py b/dts/framework/settings.py
> index 2e8dedef4f..063f282edf 100644
> --- a/dts/framework/settings.py
> +++ b/dts/framework/settings.py
> @@ -85,9 +85,7 @@
>  from pathlib import Path
>  from typing import Callable
>
> -from pydantic import ValidationError
> -
> -from .config import TestSuiteConfig
> +from .config import TestSuitesConfigs
>  from .exception import ConfigurationError
>  from .utils import DPDKGitTarball, get_commit_id
>
> @@ -114,7 +112,7 @@ class Settings:
>      #:
>      compile_timeout: float = 1200
>      #:
> -    test_suites: list[TestSuiteConfig] = field(default_factory=list)
> +    test_suites: list[tuple[str, list[str]]] = field(default_factory=list)
>      #:
>      re_run: int = 0
>
> @@ -382,7 +380,7 @@ def _get_parser() -> _DTSArgumentParser:
>
>  def _process_test_suites(
>      parser: _DTSArgumentParser, args: list[list[str]]
> -) -> list[TestSuiteConfig]:
> +) -> list[tuple[str, list[str]]]:
>      """Process the given argument to a list of :class:`TestSuiteConfig` to execute.
>
>      Args:
> @@ -398,16 +396,17 @@ def _process_test_suites(
>          # Environment variable in the form of "SUITE1 CASE1 CASE2, SUITE2 CASE1, SUITE3, ..."
>          args = [suite_with_cases.split() for suite_with_cases in args[0][0].split(",")]
>
> -    try:
> -        return [TestSuiteConfig(test_suite, test_cases) for [test_suite, *test_cases] in args]
> -    except ValidationError as e:
> -        print(
> -            "An error has occurred while validating the test suites supplied in the "
> -            f"{'environment variable' if action else 'arguments'}:",
> -            file=sys.stderr,
> -        )
> -        print(e, file=sys.stderr)
> -        sys.exit(1)
> +    available_test_suites = TestSuitesConfigs.available_test_suites()
> +    for test_suite_name, *_ in args:
> +        if test_suite_name not in available_test_suites:
> +            print(
> +                f"The test suite {test_suite_name} supplied in the "
> +                f"{'environment variable' if action else 'arguments'} is invalid.",
> +                file=sys.stderr,
> +            )
> +            sys.exit(1)
> +
> +    return [(test_suite, test_cases) for test_suite, *test_cases in args]
>
>
>  def get_settings() -> Settings:
> diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
> index 5694a2482b..6c10c1e40a 100644
> --- a/dts/framework/test_result.py
> +++ b/dts/framework/test_result.py
> @@ -64,17 +64,7 @@ class is to hold a subset of test cases (which could be all test cases) because
>
>      test_suite_class: type[TestSuite]
>      test_cases: list[FunctionType]
> -
> -    def create_config(self) -> TestSuiteConfig:
> -        """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
> -
> -        Returns:
> -            The :class:`TestSuiteConfig` representation.
> -        """
> -        return TestSuiteConfig(
> -            test_suite=self.test_suite_class.__name__,
> -            test_cases=[test_case.__name__ for test_case in self.test_cases],
> -        )
> +    config: TestSuiteConfig
>
>
>  class Result(Enum):
> diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
> index 972968b036..78e1b4c49a 100644
> --- a/dts/framework/test_suite.py
> +++ b/dts/framework/test_suite.py
> @@ -23,7 +23,7 @@
>  from ipaddress import IPv4Interface, IPv6Interface, ip_interface
>  from pkgutil import iter_modules
>  from types import FunctionType, ModuleType
> -from typing import ClassVar, NamedTuple, Union
> +from typing import ClassVar, Iterable, NamedTuple, Union, get_type_hints
>
>  from pydantic.alias_generators import to_pascal
>  from scapy.layers.inet import IP  # type: ignore[import-untyped]
> @@ -31,6 +31,7 @@
>  from scapy.packet import Packet, Padding  # type: ignore[import-untyped]
>  from typing_extensions import Self
>
> +from framework.config import TestSuiteConfig
>  from framework.testbed_model.port import Port, PortLink
>  from framework.testbed_model.sut_node import SutNode
>  from framework.testbed_model.tg_node import TGNode
> @@ -38,7 +39,7 @@
>      PacketFilteringConfig,
>  )
>
> -from .exception import TestCaseVerifyError
> +from .exception import InternalError, TestCaseVerifyError
>  from .logger import DTSLogger, get_dts_logger
>  from .utils import get_packet_summaries
>
> @@ -78,6 +79,7 @@ class TestSuite:
>      #: Whether the test suite is blocking. A failure of a blocking test suite
>      #: will block the execution of all subsequent test suites in the current build target.
>      is_blocking: ClassVar[bool] = False
> +    config: TestSuiteConfig
>      _logger: DTSLogger
>      _port_links: list[PortLink]
>      _sut_port_ingress: Port
> @@ -93,6 +95,7 @@ def __init__(
>          self,
>          sut_node: SutNode,
>          tg_node: TGNode,
> +        config: TestSuiteConfig,
>      ):
>          """Initialize the test suite testbed information and basic configuration.
>
> @@ -102,9 +105,11 @@ def __init__(
>          Args:
>              sut_node: The SUT node where the test suite will run.
>              tg_node: The TG node where the test suite will run.
> +            config: The test suite configuration.
>          """
>          self.sut_node = sut_node
>          self.tg_node = tg_node
> +        self.config = config
>          self._logger = get_dts_logger(self.__class__.__name__)
>          self._port_links = []
>          self._process_links()
> @@ -467,6 +472,21 @@ def is_test_suite(obj) -> bool:
>
>          raise Exception("class not found in eligible test module")
>
> +    @cached_property
> +    def config_type(self) -> type[TestSuiteConfig]:
> +        """A reference to the test suite's configuration type."""
> +        fields = get_type_hints(self.class_type)
> +        config_type = fields.get("config")
> +        if config_type is None:
> +            raise InternalError(
> +                "Test suite class {self.class_name} is missing the `config` attribute."
> +            )
> +        if not issubclass(config_type, TestSuiteConfig):
> +            raise InternalError(
> +                f"Test suite class {self.class_name} has an invalid configuration type assigned."
> +            )
> +        return config_type
> +
>      @cached_property
>      def test_cases(self) -> list[TestCase]:
>          """A list of all the available test cases."""
> @@ -533,6 +553,14 @@ def discover_all(
>
>          return test_suites
>
> +    def validate_test_cases(self, test_cases_names: Iterable[str]) -> None:
> +        """Validate if the supplied test cases exist in the test suite."""
> +        available_test_cases = map(lambda t: t.name, self.test_cases)
> +        for requested_test_case in test_cases_names:
> +            assert (
> +                requested_test_case in available_test_cases
> +            ), f"{requested_test_case} is not a valid test case for test suite {self.name}."
> +
>
>  AVAILABLE_TEST_SUITES: list[TestSuiteSpec] = TestSuiteSpec.discover_all()
>  """Constant to store all the available, discovered and imported test suites.
> diff --git a/dts/generate-schema.py b/dts/generate-schema.py
> index b41d28492f..d24a67f68e 100755
> --- a/dts/generate-schema.py
> +++ b/dts/generate-schema.py
> @@ -9,7 +9,7 @@
>
>  from pydantic.json_schema import GenerateJsonSchema
>
> -from framework.config import ConfigurationType
> +from framework.config import ConfigurationType, TestSuitesConfigs
>
>  DTS_DIR = os.path.dirname(os.path.realpath(__file__))
>  RELATIVE_PATH_TO_SCHEMA = "framework/config/conf_yaml_schema.json"
> @@ -26,6 +26,8 @@ def generate(self, schema, mode="validation"):
>
>
>  try:
> +    TestSuitesConfigs.fix_custom_config_annotations()
> +
>      path = os.path.join(DTS_DIR, RELATIVE_PATH_TO_SCHEMA)
>
>      with open(path, "w") as schema_file:
> diff --git a/dts/generate-test-mappings.py b/dts/generate-test-mappings.py
> new file mode 100755
> index 0000000000..d076ad0afe
> --- /dev/null
> +++ b/dts/generate-test-mappings.py
> @@ -0,0 +1,132 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2024 Arm Limited
> +
> +"""Test Suites to Configuration mappings generation script."""
> +
> +import os
> +from collections import defaultdict
> +from textwrap import indent
> +from typing import Iterable
> +
> +from framework.config.test_suite import BaseTestSuitesConfigs, TestSuiteConfig
> +from framework.exception import InternalError
> +from framework.test_suite import AVAILABLE_TEST_SUITES, TestSuiteSpec
> +
> +DTS_DIR = os.path.dirname(os.path.realpath(__file__))
> +SCRIPT_FILE_NAME = os.path.basename(__file__)
> +
> +FRAMEWORK_IMPORTS = [BaseTestSuitesConfigs, TestSuiteConfig]
> +
> +RELATIVE_PATH_TO_GENERATED_FILE = "framework/config/generated.py"
> +SMOKE_TESTS_SUITE_NAME = "smoke_tests"
> +CUSTOM_CONFIG_TYPES_VAR_NAME = "CUSTOM_CONFIG_TYPES"
> +CUSTOM_CONFIG_LINKING_FUNCTION_NAME = "fix_custom_config_annotations"
> +CUSTOM_CONFIG_LINKING_FUNCTION_DOCSTRING = [
> +    '"""Fixes the custom config types annotations.',
> +    "",
> +    f"Moreover it also fills `{CUSTOM_CONFIG_TYPES_VAR_NAME}` with all the custom config types.",
> +    '"""',
> +]
> +TEST_SUITES_CONFIG_CLASS_NAME = "TestSuitesConfigs"
> +TEST_SUITES_CONFIG_CLASS_DOCSTRING = [
> +    '"""Configuration mapping class to select and configure the test suites.',
> +    "",
> +    "Before using this class, the custom configuration type annotations need to be fixed.",
> +    f"To do so, you need to call the `{CUSTOM_CONFIG_LINKING_FUNCTION_NAME}` method.",
> +    '"""',
> +]
> +
> +
> +GENERATED_FILE_HEADER = f"""# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2024 The DPDK contributors
> +# This file is automatically generated by {SCRIPT_FILE_NAME}.
> +# Do NOT modify this file manually.
> +
> +\"\"\"Generated file containing the links between the test suites and the configuration.\"\"\"
> +"""
> +
> +
> +def join(lines: Iterable[str]) -> str:
> +    """Join list of strings into text lines."""
> +    return "\n".join(lines)
> +
> +
> +def join_and_indent(lines: Iterable[str], indentation_level=1, indentation_spaces=4) -> str:
> +    """Join list of strings into indented text lines."""
> +    return "\n".join([indent(line, " " * indentation_level * indentation_spaces) for line in lines])
> +
> +
> +def format_attributes_types(test_suite_spec: TestSuiteSpec):
> +    """Format the config type into the respective configuration class field attribute type."""
> +    config_type = test_suite_spec.config_type.__name__
> +    if config_type != TestSuiteConfig.__name__:
> +        config_type = f'"{config_type}"'
> +    return f"Optional[{config_type}]"
> +
> +
> +try:
> +    framework_imports: dict[str, list[str]] = defaultdict(list)
> +    for _import in FRAMEWORK_IMPORTS:
> +        framework_imports[_import.__module__].append(_import.__name__)
> +    formatted_framework_imports = sorted(
> +        [
> +            f"from {module} import {', '.join(sorted(imports))}"
> +            for module, imports in framework_imports.items()
> +        ]
> +    )
> +
> +    test_suites = [
> +        test_suite_spec
> +        for test_suite_spec in AVAILABLE_TEST_SUITES
> +        if test_suite_spec.name != SMOKE_TESTS_SUITE_NAME
> +    ]
> +
> +    custom_configs = [t for t in test_suites if t.config_type is not TestSuiteConfig]
> +
> +    custom_config_imports = [
> +        f"from {t.module_type.__name__} import {t.config_type.__name__}" for t in custom_configs
> +    ]
> +
> +    test_suites_attributes = [f"{t.name}: {format_attributes_types(t)} = None" for t in test_suites]
> +
> +    custom_config_assignments = [
> +        f'{CUSTOM_CONFIG_TYPES_VAR_NAME}["{t.name}"] = {t.config_type.__name__}'
> +        for t in custom_configs
> +    ]
> +
> +    generated_file_contents = f"""{GENERATED_FILE_HEADER}
> +from typing import TYPE_CHECKING, Optional
> +
> +{join(formatted_framework_imports)}
> +
> +if TYPE_CHECKING:
> +{join_and_indent(custom_config_imports)}
> +
> +
> +{CUSTOM_CONFIG_TYPES_VAR_NAME}: dict[str, type[{TestSuiteConfig.__name__}]] = {'{}'}
> +
> +
> +class {TEST_SUITES_CONFIG_CLASS_NAME}({BaseTestSuitesConfigs.__name__}):
> +{join_and_indent(TEST_SUITES_CONFIG_CLASS_DOCSTRING)}
> +
> +{join_and_indent(test_suites_attributes)}
> +
> +    @classmethod
> +    def {CUSTOM_CONFIG_LINKING_FUNCTION_NAME}(cls):
> +{join_and_indent(CUSTOM_CONFIG_LINKING_FUNCTION_DOCSTRING, indentation_level=2)}
> +{join_and_indent(custom_config_imports, indentation_level=2)}
> +
> +{join_and_indent(custom_config_assignments, indentation_level=2)}
> +
> +        cls.model_rebuild()
> +"""
> +
> +    path = os.path.join(DTS_DIR, RELATIVE_PATH_TO_GENERATED_FILE)
> +
> +    with open(path, "w") as generated_file:
> +        generated_file.write(generated_file_contents)
> +
> +    print("Test suites to configuration mappings generated successfully!")
> +except Exception as e:
> +    raise InternalError("Failed to generate test suites to configuration mappings.") from e
> diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
> index d958f99030..1723c123bc 100644
> --- a/dts/tests/TestSuite_hello_world.py
> +++ b/dts/tests/TestSuite_hello_world.py
> @@ -7,6 +7,7 @@
>  No other EAL parameters apart from cores are used.
>  """
>
> +from framework.config import TestSuiteConfig
>  from framework.remote_session.dpdk_shell import compute_eal_params
>  from framework.test_suite import TestSuite
>  from framework.testbed_model.cpu import (
> @@ -16,9 +17,18 @@
>  )
>
>
> +class HelloWorldConfig(TestSuiteConfig):
> +    """Example custom configuration for the `TestHelloWorld` test suite."""
> +
> +    #: Timeout for the DPDK apps
> +    timeout: int = 50
> +
> +
>  class TestHelloWorld(TestSuite):
>      """DPDK hello world app test suite."""
>
> +    config: HelloWorldConfig
> +
>      def set_up_suite(self) -> None:
>          """Set up the test suite.
>
> @@ -59,7 +69,7 @@ def test_hello_world_all_cores(self) -> None:
>          eal_para = compute_eal_params(
>              self.sut_node, lcore_filter_specifier=LogicalCoreList(self.sut_node.lcores)
>          )
> -        result = self.sut_node.run_dpdk_app(self.app_helloworld_path, eal_para, 50)
> +        result = self.sut_node.run_dpdk_app(self.app_helloworld_path, eal_para, self.config.timeout)
>          for lcore in self.sut_node.lcores:
>              self.verify(
>                  f"hello from core {int(lcore)}" in result.stdout,
> diff --git a/dts/tests/__init__.py b/dts/tests/__init__.py
> new file mode 100644
> index 0000000000..a300eb26fc
> --- /dev/null
> +++ b/dts/tests/__init__.py
> @@ -0,0 +1,7 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2024 Arm Limited
> +
> +"""Test suites.
> +
> +This package contains all the available test suites in DTS.
> +"""
> --
> 2.34.1
>

^ permalink raw reply	[flat|nested] 17+ messages in thread

* [PATCH v2 0/4] dts: add per-test-suite configuration
  2024-09-06 16:13 [PATCH] dts: add per-test-suite configuration Luca Vizzarro
  2024-09-27 17:45 ` Jeremy Spewock
@ 2024-11-08 13:38 ` Luca Vizzarro
  2024-11-08 13:38   ` [PATCH v2 1/4] dts: add tests package to API docs Luca Vizzarro
                     ` (3 more replies)
  2024-11-08 13:45 ` [PATCH v3 0/4] dts: add per-test-suite configuration Luca Vizzarro
  2 siblings, 4 replies; 17+ messages in thread
From: Luca Vizzarro @ 2024-11-08 13:38 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Patrick Robb, Luca Vizzarro

Hi there,

sending a v2 for the per-test-suite configuration additions.

v2:
- rebase
- added test suites to doc pages
- doc fixes
- updated the custom config procedure to using tests.config
- simplified the generated file
- implemented the generator in dts-check-format

Best regards,
Luca Vizzarro

Depends-on: series-33871 ("dts: Pydantic configuration")

Luca Vizzarro (4):
  dts: add tests package to API docs
  dts: fix smoke tests docstring
  dts: add per-test-suite configuration
  dts: update autodoc sorting order

 devtools/dts-check-format.sh                  |  28 +++-
 devtools/dts-generate-tests-mappings.py       | 117 +++++++++++++
 doc/api/dts/framework.config.generated.rst    |   8 +
 doc/api/dts/framework.config.rst              |   7 +
 doc/api/dts/framework.config.test_suite.rst   |   8 +
 doc/api/dts/index.rst                         |   1 +
 doc/api/dts/tests.TestSuite_hello_world.rst   |   9 +
 doc/api/dts/tests.TestSuite_os_udp.rst        |   9 +
 .../tests.TestSuite_pmd_buffer_scatter.rst    |   9 +
 doc/api/dts/tests.TestSuite_smoke_tests.rst   |   9 +
 doc/api/dts/tests.TestSuite_vlan.rst          |  10 ++
 doc/api/dts/tests.config.rst                  |   9 +
 doc/api/dts/tests.rst                         |  20 +++
 doc/guides/conf.py                            |   1 +
 doc/guides/tools/dts.rst                      |  23 +++
 dts/conf.yaml                                 |   4 +-
 dts/framework/config/__init__.py              |  76 +--------
 dts/framework/config/generated.py             |  25 +++
 dts/framework/config/test_suite.py            | 154 ++++++++++++++++++
 dts/framework/runner.py                       |  69 ++++++--
 dts/framework/settings.py                     |  30 ++--
 dts/framework/test_result.py                  |  15 +-
 dts/framework/test_suite.py                   |  22 ++-
 dts/tests/TestSuite_hello_world.py            |   5 +-
 dts/tests/TestSuite_smoke_tests.py            |   2 -
 dts/tests/__init__.py                         |   7 +
 dts/tests/config.py                           |  20 +++
 27 files changed, 580 insertions(+), 117 deletions(-)
 create mode 100755 devtools/dts-generate-tests-mappings.py
 create mode 100644 doc/api/dts/framework.config.generated.rst
 create mode 100644 doc/api/dts/framework.config.test_suite.rst
 create mode 100644 doc/api/dts/tests.TestSuite_hello_world.rst
 create mode 100644 doc/api/dts/tests.TestSuite_os_udp.rst
 create mode 100644 doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
 create mode 100644 doc/api/dts/tests.TestSuite_smoke_tests.rst
 create mode 100644 doc/api/dts/tests.TestSuite_vlan.rst
 create mode 100644 doc/api/dts/tests.config.rst
 create mode 100644 doc/api/dts/tests.rst
 create mode 100644 dts/framework/config/generated.py
 create mode 100644 dts/framework/config/test_suite.py
 create mode 100644 dts/tests/__init__.py
 create mode 100644 dts/tests/config.py

-- 
2.43.0


^ permalink raw reply	[flat|nested] 17+ messages in thread

* [PATCH v2 1/4] dts: add tests package to API docs
  2024-11-08 13:38 ` [PATCH v2 0/4] " Luca Vizzarro
@ 2024-11-08 13:38   ` Luca Vizzarro
  2024-11-08 13:38   ` [PATCH v2 2/4] dts: fix smoke tests docstring Luca Vizzarro
                     ` (2 subsequent siblings)
  3 siblings, 0 replies; 17+ messages in thread
From: Luca Vizzarro @ 2024-11-08 13:38 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Patrick Robb, Luca Vizzarro

The test suites are not documented in the API as their respective
documentation files are missing.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 doc/api/dts/index.rst                         |  1 +
 doc/api/dts/tests.TestSuite_hello_world.rst   |  9 +++++++++
 doc/api/dts/tests.TestSuite_os_udp.rst        |  9 +++++++++
 .../tests.TestSuite_pmd_buffer_scatter.rst    |  9 +++++++++
 doc/api/dts/tests.TestSuite_smoke_tests.rst   |  9 +++++++++
 doc/api/dts/tests.TestSuite_vlan.rst          | 10 ++++++++++
 doc/api/dts/tests.rst                         | 19 +++++++++++++++++++
 7 files changed, 66 insertions(+)
 create mode 100644 doc/api/dts/tests.TestSuite_hello_world.rst
 create mode 100644 doc/api/dts/tests.TestSuite_os_udp.rst
 create mode 100644 doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
 create mode 100644 doc/api/dts/tests.TestSuite_smoke_tests.rst
 create mode 100644 doc/api/dts/tests.TestSuite_vlan.rst
 create mode 100644 doc/api/dts/tests.rst

diff --git a/doc/api/dts/index.rst b/doc/api/dts/index.rst
index 534512dc17..db05fd37d5 100644
--- a/doc/api/dts/index.rst
+++ b/doc/api/dts/index.rst
@@ -14,6 +14,7 @@ Packages
    :includehidden:
    :maxdepth: 1
 
+   tests
    framework.testbed_model
    framework.remote_session
    framework.params
diff --git a/doc/api/dts/tests.TestSuite_hello_world.rst b/doc/api/dts/tests.TestSuite_hello_world.rst
new file mode 100644
index 0000000000..f92786996f
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_hello_world.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+hello_world Test Suite
+======================
+
+.. automodule:: tests.TestSuite_hello_world
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.TestSuite_os_udp.rst b/doc/api/dts/tests.TestSuite_os_udp.rst
new file mode 100644
index 0000000000..121a653670
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_os_udp.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+os_udp Test Suite
+=================
+
+.. automodule:: tests.TestSuite_os_udp
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst b/doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
new file mode 100644
index 0000000000..b6571b24f5
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+pmd_buffer_scatter Test Suite
+=============================
+
+.. automodule:: tests.TestSuite_pmd_buffer_scatter
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.TestSuite_smoke_tests.rst b/doc/api/dts/tests.TestSuite_smoke_tests.rst
new file mode 100644
index 0000000000..5b11aa7f48
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_smoke_tests.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+smoke_tests Test Suite
+======================
+
+.. automodule:: tests.TestSuite_smoke_tests
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.TestSuite_vlan.rst b/doc/api/dts/tests.TestSuite_vlan.rst
new file mode 100644
index 0000000000..86a29538bb
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_vlan.rst
@@ -0,0 +1,10 @@
+
+.. SPDX-License-Identifier: BSD-3-Clause
+
+vlan Test Suite
+===============
+
+.. automodule:: tests.TestSuite_vlan
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.rst b/doc/api/dts/tests.rst
new file mode 100644
index 0000000000..0c136b4bb0
--- /dev/null
+++ b/doc/api/dts/tests.rst
@@ -0,0 +1,19 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+tests - Test Suites Package
+==============================
+
+.. automodule:: tests
+   :members:
+   :show-inheritance:
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   tests.TestSuite_hello_world
+   tests.TestSuite_os_udp
+   tests.TestSuite_pmd_buffer_scatter
+   tests.TestSuite_smoke_tests
+   tests.TestSuite_vlan
+
-- 
2.43.0


^ permalink raw reply	[flat|nested] 17+ messages in thread

* [PATCH v2 2/4] dts: fix smoke tests docstring
  2024-11-08 13:38 ` [PATCH v2 0/4] " Luca Vizzarro
  2024-11-08 13:38   ` [PATCH v2 1/4] dts: add tests package to API docs Luca Vizzarro
@ 2024-11-08 13:38   ` Luca Vizzarro
  2024-11-12 20:03     ` Dean Marx
  2024-11-08 13:38   ` [PATCH v2 3/4] dts: add per-test-suite configuration Luca Vizzarro
  2024-11-08 13:38   ` [PATCH v2 4/4] dts: update autodoc sorting order Luca Vizzarro
  3 siblings, 1 reply; 17+ messages in thread
From: Luca Vizzarro @ 2024-11-08 13:38 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Patrick Robb, Luca Vizzarro

Sphinx autodoc complains of is_blocking being duplicated. This is the
case as the parent already holds a docstring for this attribute. Remove
the duplication.

Fixes: 6ef07151aac4 ("dts: update docstrings")

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 dts/tests/TestSuite_smoke_tests.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
index bc3a2a6bf9..f564ac82b4 100644
--- a/dts/tests/TestSuite_smoke_tests.py
+++ b/dts/tests/TestSuite_smoke_tests.py
@@ -30,8 +30,6 @@ class TestSmokeTests(TestSuite):
     The infrastructure also needs to be tested, as that is also used by all other test suites.
 
     Attributes:
-        is_blocking: This test suite will block the execution of all other test suites
-            in the test run after it.
         nics_in_node: The NICs present on the SUT node.
     """
 
-- 
2.43.0


^ permalink raw reply	[flat|nested] 17+ messages in thread

* [PATCH v2 3/4] dts: add per-test-suite configuration
  2024-11-08 13:38 ` [PATCH v2 0/4] " Luca Vizzarro
  2024-11-08 13:38   ` [PATCH v2 1/4] dts: add tests package to API docs Luca Vizzarro
  2024-11-08 13:38   ` [PATCH v2 2/4] dts: fix smoke tests docstring Luca Vizzarro
@ 2024-11-08 13:38   ` Luca Vizzarro
  2024-11-08 13:38   ` [PATCH v2 4/4] dts: update autodoc sorting order Luca Vizzarro
  3 siblings, 0 replies; 17+ messages in thread
From: Luca Vizzarro @ 2024-11-08 13:38 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Patrick Robb, Luca Vizzarro

Allow test suites to be configured individually. Moreover enable them to
implement their own custom configuration.

This solution adds some new complexity to DTS, which is generated source
code. In order to ensure strong typing, the test suites and their custom
configurations need to be linked in the main configuration class.
Unfortunately, this is not feasible during runtime as it will incur in
circular dependencies. Generating the links appear to be the most
straightforward approach.

This commit also brings a new major change to the configuration schema.
Test suites are no longer defined as a list of strings, like:

    test_suites:
    - hello_world
    - pmd_buffer_scatter

but as mapping of mappings or strings:

    test_suites:
      hello_world: {} # any custom fields or test cases can be set here
      pmd_buffer_scatter: all # "all" defines all the test cases, or
                              # they can individually be set separated
                              # by a space

Not defining the `test_cases` field in the configuration is equivalent
to `all`, therefore the definitions for either test suite above are
also equivalent.

Creating the __init__.py file under the tests folder, allows it to be
picked up as a package. This is a mypy requirement to import the tests
from within the framework.

Bugzilla ID: 1375

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 devtools/dts-check-format.sh                |  28 +++-
 devtools/dts-generate-tests-mappings.py     | 117 +++++++++++++++
 doc/api/dts/framework.config.generated.rst  |   8 +
 doc/api/dts/framework.config.rst            |   7 +
 doc/api/dts/framework.config.test_suite.rst |   8 +
 doc/api/dts/tests.config.rst                |   9 ++
 doc/api/dts/tests.rst                       |   1 +
 doc/guides/tools/dts.rst                    |  23 +++
 dts/conf.yaml                               |   4 +-
 dts/framework/config/__init__.py            |  76 +---------
 dts/framework/config/generated.py           |  25 ++++
 dts/framework/config/test_suite.py          | 154 ++++++++++++++++++++
 dts/framework/runner.py                     |  69 +++++++--
 dts/framework/settings.py                   |  30 ++--
 dts/framework/test_result.py                |  15 +-
 dts/framework/test_suite.py                 |  22 ++-
 dts/tests/TestSuite_hello_world.py          |   5 +-
 dts/tests/__init__.py                       |   7 +
 dts/tests/config.py                         |  20 +++
 19 files changed, 513 insertions(+), 115 deletions(-)
 create mode 100755 devtools/dts-generate-tests-mappings.py
 create mode 100644 doc/api/dts/framework.config.generated.rst
 create mode 100644 doc/api/dts/framework.config.test_suite.rst
 create mode 100644 doc/api/dts/tests.config.rst
 create mode 100644 dts/framework/config/generated.py
 create mode 100644 dts/framework/config/test_suite.py
 create mode 100644 dts/tests/__init__.py
 create mode 100644 dts/tests/config.py

diff --git a/devtools/dts-check-format.sh b/devtools/dts-check-format.sh
index 3f43e17e88..7a440fc0cf 100755
--- a/devtools/dts-check-format.sh
+++ b/devtools/dts-check-format.sh
@@ -2,6 +2,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2022 University of New Hampshire
 # Copyright(c) 2023 PANTHEON.tech s.r.o.
+# Copyright(c) 2024 Arm Limited
 
 usage() {
 	echo "Usage: $(basename $0) [options] [directory]"
@@ -13,15 +14,19 @@ usage() {
 format=true
 lint=true
 typecheck=true
+generate=true
 
 # Comments after args serve as documentation; must be present
-while getopts "hflt" arg; do
+while getopts "hgflt" arg; do
 	case $arg in
 	h) # Display this message
-		echo 'Run formatting and linting programs for DTS.'
+		echo 'Run generating, formatting and linting programs for DTS.'
 		usage
 		exit 0
 		;;
+	g) # Don't run code generator
+		generate=false
+		;;
 	f) # Don't run formatters
 		format=false
 		;;
@@ -48,7 +53,22 @@ heading() {
 
 errors=0
 
+if $generate; then
+	heading "Generating test suites to configuration mappings"
+	if command -v python3 > /dev/null; then
+		../devtools/dts-generate-tests-mappings.py
+		errors=$((errors + $?))
+	else
+		echo "python3 not found, unable to run generator"
+		errros=$((errors + 1))
+	fi
+fi
+
 if $format; then
+	if $generate; then
+		echo
+	fi
+
 	if command -v git > /dev/null; then
 		if git rev-parse --is-inside-work-tree >&-; then
 			heading "Formatting in $directory/"
@@ -85,7 +105,7 @@ if $format; then
 fi
 
 if $lint; then
-	if $format; then
+	if $generate || $format; then
 		echo
 	fi
 	heading "Linting in $directory/"
@@ -99,7 +119,7 @@ if $lint; then
 fi
 
 if $typecheck; then
-	if $format || $lint; then
+	if $generate || $format || $lint; then
 		echo
 	fi
 	heading "Checking types in $directory/"
diff --git a/devtools/dts-generate-tests-mappings.py b/devtools/dts-generate-tests-mappings.py
new file mode 100755
index 0000000000..26ecc1018c
--- /dev/null
+++ b/devtools/dts-generate-tests-mappings.py
@@ -0,0 +1,117 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""DTS Test Suites to Configuration mappings generation script."""
+
+import os
+import sys
+from collections import defaultdict
+from pathlib import Path
+from textwrap import indent
+from typing import Iterable
+
+DTS_DIR = Path(__file__).parent.joinpath("..", "dts").resolve()
+SCRIPT_FILE_NAME = Path(__file__).relative_to(Path(__file__).parent.parent)
+
+sys.path.append(str(DTS_DIR))
+
+from framework.config.test_suite import BaseTestSuitesConfigs, TestSuiteConfig
+from framework.exception import InternalError
+from framework.test_suite import AVAILABLE_TEST_SUITES, TestSuiteSpec
+
+FRAMEWORK_IMPORTS = [BaseTestSuitesConfigs, TestSuiteConfig]
+
+RELATIVE_PATH_TO_GENERATED_FILE = "framework/config/generated.py"
+SMOKE_TESTS_SUITE_NAME = "smoke_tests"
+CUSTOM_CONFIG_TYPES_VAR_NAME = "CUSTOM_CONFIG_TYPES"
+CUSTOM_CONFIG_TYPES_VAR_DOCSTRING = [
+    "#: Mapping of test suites to their corresponding custom configuration objects if any."
+]
+TEST_SUITES_CONFIG_CLASS_NAME = "TestSuitesConfigs"
+TEST_SUITES_CONFIG_CLASS_DOCSTRING = [
+    '"""Configuration mapping class to select and configure the test suites."""',
+]
+
+
+GENERATED_FILE_HEADER = f"""# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 The DPDK contributors
+# This file is automatically generated by {SCRIPT_FILE_NAME}.
+# Do NOT modify this file manually.
+
+\"\"\"Generated file containing the links between the test suites and the configuration.\"\"\"
+"""
+
+
+def join(lines: Iterable[str]) -> str:
+    """Join list of strings into text lines."""
+    return "\n".join(lines)
+
+
+def join_and_indent(lines: Iterable[str], indentation_level=1, indentation_spaces=4) -> str:
+    """Join list of strings into indented text lines."""
+    return "\n".join([indent(line, " " * indentation_level * indentation_spaces) for line in lines])
+
+
+def format_attributes_types(test_suite_spec: TestSuiteSpec):
+    """Format the config type into the respective configuration class field attribute type."""
+    config_type = test_suite_spec.config_obj.__name__
+    return f"Optional[{config_type}]"
+
+
+try:
+    framework_imports: dict[str, list[str]] = defaultdict(list)
+    for _import in FRAMEWORK_IMPORTS:
+        framework_imports[_import.__module__].append(_import.__name__)
+    formatted_framework_imports = sorted(
+        [
+            f"from {module} import {', '.join(sorted(imports))}"
+            for module, imports in framework_imports.items()
+        ]
+    )
+
+    test_suites = [
+        test_suite_spec
+        for test_suite_spec in AVAILABLE_TEST_SUITES
+        if test_suite_spec.name != SMOKE_TESTS_SUITE_NAME
+    ]
+
+    custom_configs = [t for t in test_suites if t.config_obj is not TestSuiteConfig]
+
+    custom_config_imports = [
+        f"from {t.config_obj.__module__} import {t.config_obj.__name__}" for t in custom_configs
+    ]
+
+    test_suites_attributes = [f"{t.name}: {format_attributes_types(t)} = None" for t in test_suites]
+
+    custom_config_mappings = [f'"{t.name}": {t.config_obj.__name__},' for t in custom_configs]
+
+    generated_file_contents = f"""{GENERATED_FILE_HEADER}
+from typing import Optional
+
+{join(formatted_framework_imports)}
+
+{join(custom_config_imports)}
+
+{join(CUSTOM_CONFIG_TYPES_VAR_DOCSTRING)}
+{CUSTOM_CONFIG_TYPES_VAR_NAME}: dict[str, type[{TestSuiteConfig.__name__}]] = {'{'}
+{join_and_indent(custom_config_mappings)}
+{'}'}
+
+
+class {TEST_SUITES_CONFIG_CLASS_NAME}({BaseTestSuitesConfigs.__name__}):
+{join_and_indent(TEST_SUITES_CONFIG_CLASS_DOCSTRING)}
+
+{join_and_indent(test_suites_attributes)}
+"""
+
+    path = os.path.join(DTS_DIR, RELATIVE_PATH_TO_GENERATED_FILE)
+
+    with open(path, "w") as generated_file:
+        generated_file.write(generated_file_contents)
+
+    print("Test suites to configuration mappings generated successfully!")
+except Exception as e:
+    raise InternalError(
+        "Failed to generate test suites to configuration mappings."
+    ) from e
diff --git a/doc/api/dts/framework.config.generated.rst b/doc/api/dts/framework.config.generated.rst
new file mode 100644
index 0000000000..5dfa9342f0
--- /dev/null
+++ b/doc/api/dts/framework.config.generated.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+generated - Generated Test Suite Configurations
+===============================================
+
+.. automodule:: framework.config.generated
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/framework.config.rst b/doc/api/dts/framework.config.rst
index cc266276c1..217fe026c4 100644
--- a/doc/api/dts/framework.config.rst
+++ b/doc/api/dts/framework.config.rst
@@ -6,3 +6,10 @@ config - Configuration Package
 .. automodule:: framework.config
    :members:
    :show-inheritance:
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   framework.config.generated
+   framework.config.test_suite
diff --git a/doc/api/dts/framework.config.test_suite.rst b/doc/api/dts/framework.config.test_suite.rst
new file mode 100644
index 0000000000..d59dcf5d6e
--- /dev/null
+++ b/doc/api/dts/framework.config.test_suite.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+test_suite - Test Suite Configuration Definitions
+=================================================
+
+.. automodule:: framework.config.test_suite
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.config.rst b/doc/api/dts/tests.config.rst
new file mode 100644
index 0000000000..ce3d9df868
--- /dev/null
+++ b/doc/api/dts/tests.config.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+Test Suites Configurations
+==========================
+
+.. automodule:: tests.config
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.rst b/doc/api/dts/tests.rst
index 0c136b4bb0..7fc25f2123 100644
--- a/doc/api/dts/tests.rst
+++ b/doc/api/dts/tests.rst
@@ -11,6 +11,7 @@ tests - Test Suites Package
    :hidden:
    :maxdepth: 1
 
+   tests.config
    tests.TestSuite_hello_world
    tests.TestSuite_os_udp
    tests.TestSuite_pmd_buffer_scatter
diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index f4e297413d..4e63601b19 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -408,6 +408,29 @@ There are four types of methods that comprise a test suite:
    should be implemented in the ``SutNode`` class (and the underlying classes that ``SutNode`` uses)
    and used by the test suite via the ``sut_node`` field.
 
+The test suites can also implement their own custom configuration fields. This can be achieved by
+creating a new test suite config file which inherits from ``TestSuiteConfig`` defined in
+``dts/framework/config/test_suite.py``. So that this new custom configuration class is used, the
+test suite class must override the ``config`` attribute annotation with your new class, for example:
+
+.. code:: python
+
+   # place this under tests/config.py to avoid circular dependencies
+   class CustomConfig(TestSuiteConfig):
+      my_custom_field: int = 10
+
+   # place this under tests/TestSuite_my_new_test_suite.py
+   class TestMyNewTestSuite(TestSuite):
+      config: CustomConfig
+
+Finally, the test suites and the custom configuration files need to linked in the global configuration.
+This can be easily achieved by running the ``dts/generate-test-mappings.py``, e.g.:
+
+.. code-block:: console
+
+   $ poetry shell
+   (dts-py3.10) $ ./generate-test-mappings.py
+
 
 .. _dts_dev_tools:
 
diff --git a/dts/conf.yaml b/dts/conf.yaml
index 2496262854..377304dddf 100644
--- a/dts/conf.yaml
+++ b/dts/conf.yaml
@@ -28,8 +28,8 @@ test_runs:
     func: true # enable functional testing
     skip_smoke_tests: false # optional
     test_suites: # the following test suites will be run in their entirety
-      - hello_world
-      - os_udp
+      hello_world: all
+      os_udp: all
     # The machine running the DPDK test executable
     system_under_test_node:
       node_name: "SUT 1"
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 82113a6257..0ac7ab5c46 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -32,11 +32,13 @@
       and makes it thread safe should we ever want to move in that direction.
 """
 
+# pylama:ignore=W0611
+
 import tarfile
 from enum import Enum, auto, unique
 from functools import cached_property
 from pathlib import Path, PurePath
-from typing import TYPE_CHECKING, Annotated, Any, Literal, NamedTuple
+from typing import Annotated, Literal, NamedTuple
 
 import yaml
 from pydantic import (
@@ -52,8 +54,7 @@
 from framework.exception import ConfigurationError
 from framework.utils import REGEX_FOR_PCI_ADDRESS, StrEnum
 
-if TYPE_CHECKING:
-    from framework.test_suite import TestSuiteSpec
+from .generated import TestSuitesConfigs
 
 
 class FrozenModel(BaseModel):
@@ -382,69 +383,6 @@ class DPDKUncompiledBuildConfiguration(BaseDPDKBuildConfiguration):
 DPDKBuildConfiguration = DPDKPrecompiledBuildConfiguration | DPDKUncompiledBuildConfiguration
 
 
-class TestSuiteConfig(FrozenModel):
-    """Test suite configuration.
-
-    Information about a single test suite to be executed. This can also be represented as a string
-    instead of a mapping, example:
-
-    .. code:: yaml
-
-        test_runs:
-        - test_suites:
-            # As string representation:
-            - hello_world # test all of `hello_world`, or
-            - hello_world hello_world_single_core # test only `hello_world_single_core`
-            # or as model fields:
-            - test_suite: hello_world
-              test_cases: [hello_world_single_core] # without this field all test cases are run
-    """
-
-    #: The name of the test suite module without the starting ``TestSuite_``.
-    test_suite_name: str = Field(alias="test_suite")
-    #: The names of test cases from this test suite to execute. If empty, all test cases will be
-    #: executed.
-    test_cases_names: list[str] = Field(default_factory=list, alias="test_cases")
-
-    @cached_property
-    def test_suite_spec(self) -> "TestSuiteSpec":
-        """The specification of the requested test suite."""
-        from framework.test_suite import find_by_name
-
-        test_suite_spec = find_by_name(self.test_suite_name)
-        assert (
-            test_suite_spec is not None
-        ), f"{self.test_suite_name} is not a valid test suite module name."
-        return test_suite_spec
-
-    @model_validator(mode="before")
-    @classmethod
-    def convert_from_string(cls, data: Any) -> Any:
-        """Convert the string representation of the model into a valid mapping."""
-        if isinstance(data, str):
-            [test_suite, *test_cases] = data.split()
-            return dict(test_suite=test_suite, test_cases=test_cases)
-        return data
-
-    @model_validator(mode="after")
-    def validate_names(self) -> Self:
-        """Validate the supplied test suite and test cases names.
-
-        This validator relies on the cached property `test_suite_spec` to run for the first
-        time in this call, therefore triggering the assertions if needed.
-        """
-        available_test_cases = map(
-            lambda t: t.name, self.test_suite_spec.class_obj.get_test_cases()
-        )
-        for requested_test_case in self.test_cases_names:
-            assert requested_test_case in available_test_cases, (
-                f"{requested_test_case} is not a valid test case "
-                f"of test suite {self.test_suite_name}."
-            )
-
-        return self
-
-
 class TestRunSUTNodeConfiguration(FrozenModel):
     """The SUT node configuration of a test run."""
 
@@ -469,8 +407,8 @@ class TestRunConfiguration(FrozenModel):
     func: bool
     #: Whether to skip smoke tests.
     skip_smoke_tests: bool = False
-    #: The names of test suites and/or test cases to execute.
-    test_suites: list[TestSuiteConfig] = Field(min_length=1)
+    #: The test suites to be selected and/or configured.
+    test_suites: TestSuitesConfigs
     #: The SUT node configuration to use in this test run.
     system_under_test_node: TestRunSUTNodeConfiguration
     #: The TG node name to use in this test run.
@@ -602,6 +540,6 @@ def load_config(config_file_path: Path) -> Configuration:
         config_data = yaml.safe_load(f)
 
     try:
-        return Configuration.model_validate(config_data)
+        return Configuration.model_validate(config_data, context={})
     except ValidationError as e:
         raise ConfigurationError("failed to load the supplied configuration") from e
diff --git a/dts/framework/config/generated.py b/dts/framework/config/generated.py
new file mode 100644
index 0000000000..cc4a539987
--- /dev/null
+++ b/dts/framework/config/generated.py
@@ -0,0 +1,25 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 The DPDK contributors
+# This file is automatically generated by devtools/dts-generate-tests-mappings.py.
+# Do NOT modify this file manually.
+
+"""Generated file containing the links between the test suites and the configuration."""
+
+from typing import Optional
+
+from framework.config.test_suite import BaseTestSuitesConfigs, TestSuiteConfig
+from tests.config import HelloWorldConfig
+
+#: Mapping of test suites to their corresponding custom configuration objects if any.
+CUSTOM_CONFIG_TYPES: dict[str, type[TestSuiteConfig]] = {
+    "hello_world": HelloWorldConfig,
+}
+
+
+class TestSuitesConfigs(BaseTestSuitesConfigs):
+    """Configuration mapping class to select and configure the test suites."""
+
+    hello_world: Optional[HelloWorldConfig] = None
+    os_udp: Optional[TestSuiteConfig] = None
+    pmd_buffer_scatter: Optional[TestSuiteConfig] = None
+    vlan: Optional[TestSuiteConfig] = None
diff --git a/dts/framework/config/test_suite.py b/dts/framework/config/test_suite.py
new file mode 100644
index 0000000000..863052cbc1
--- /dev/null
+++ b/dts/framework/config/test_suite.py
@@ -0,0 +1,154 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Test suites configuration module.
+
+Test suites can inherit :class:`TestSuiteConfig` to create their own custom configuration.
+By doing so, the test suite class must also override the annotation of the field
+`~framework.test_suite.TestSuite.config` to use their custom configuration type.
+"""
+
+from functools import cached_property
+from typing import TYPE_CHECKING, Any, Iterable
+
+from pydantic import (
+    BaseModel,
+    ConfigDict,
+    Field,
+    ValidationInfo,
+    field_validator,
+    model_validator,
+)
+from typing_extensions import Self
+
+if TYPE_CHECKING:
+    from framework.test_suite import TestSuiteSpec
+
+
+class TestSuiteConfig(BaseModel):
+    """Test suite configuration base model.
+
+    By default the configuration of a generic test suite does not contain any attributes. Any test
+    suite should inherit this class to create their own custom configuration. Finally override the
+    type of the :attr:`~TestSuite.config` to use the newly created one.
+
+    If no custom fields require setting, this can also be represented as a string instead of
+    a mapping, example:
+
+    .. code:: yaml
+
+        test_runs:
+        - test_suites:
+            # As string representation:
+            hello_world: all # test all of `hello_world`, or
+            hello_world: hello_world_single_core # test only `hello_world_single_core`
+            # or as a mapping of the model's fields:
+            hello_world:
+              test_cases: [hello_world_single_core] # without this field all test cases are run
+
+    .. warning::
+
+        This class sets `protected_namespaces` to an empty tuple as a workaround for autodoc.
+        Due to autodoc loading this class first before any other child ones, it causes the Pydantic
+        fields in the protected namespace ``model_`` to be set on the parent. Leading any child
+        classes to inherit these protected fields as user-defined ones, finally triggering their
+        instances to complain about the presence of protected fields.
+
+        Because any class inheriting this class will therefore have protected namespaces disabled,
+        you won't be blocked to create fields starting with ``model_``. Nonetheless, you **must**
+        refrain from doing so as this is not the intended behavior.
+    """
+
+    model_config = ConfigDict(frozen=True, extra="forbid", protected_namespaces=())
+
+    #: The name of the test suite module without the starting ``TestSuite_``. This field **cannot**
+    #: be used in the configuration file. The name will be inherited from the mapping key instead.
+    test_suite_name: str
+    #: The names of test cases from this test suite to execute. If empty, all test cases will be
+    #: executed.
+    test_cases_names: list[str] = Field(default_factory=list, alias="test_cases")
+
+    @cached_property
+    def test_suite_spec(self) -> "TestSuiteSpec":
+        """The specification of the requested test suite."""
+        from framework.test_suite import find_by_name
+
+        test_suite_spec = find_by_name(self.test_suite_name)
+        assert (
+            test_suite_spec is not None
+        ), f"{self.test_suite_name} is not a valid test suite module name."
+        return test_suite_spec
+
+    @model_validator(mode="before")
+    @classmethod
+    def load_test_suite_name_from_context(cls, data: Any, info: ValidationInfo) -> dict:
+        """Load the test suite name from the validation context, if any."""
+        assert isinstance(data, dict), "The test suite configuration value is invalid."
+        name = data.get("test_suite_name")
+        # If the context is carrying the test suite name, then use it instead.
+        if info.context is not None and (test_suite_name := info.context.get("test_suite_name")):
+            assert not name, "The test suite name cannot be set manually."
+            data["test_suite_name"] = test_suite_name
+        return data
+
+    @model_validator(mode="before")
+    @classmethod
+    def convert_from_string(cls, data: Any) -> dict:
+        """Convert the string representation of the model into a valid mapping."""
+        if isinstance(data, str):
+            test_cases = [] if data == "all" else data.split()
+            return dict(test_cases=test_cases)
+        return data
+
+    @model_validator(mode="after")
+    def validate_names(self) -> Self:
+        """Validate the supplied test suite and test cases names.
+
+        This validator relies on the cached property `test_suite_spec` to run for the first
+        time in this call, therefore triggering the assertions if needed.
+        """
+        available_test_cases = map(
+            lambda t: t.name, self.test_suite_spec.class_obj.get_test_cases()
+        )
+        for requested_test_case in self.test_cases_names:
+            assert requested_test_case in available_test_cases, (
+                f"{requested_test_case} is not a valid test case "
+                f"of test suite {self.test_suite_name}."
+            )
+
+        return self
+
+
+class BaseTestSuitesConfigs(BaseModel):
+    """Base class for test suites configs."""
+
+    model_config = ConfigDict(frozen=True, extra="forbid")
+
+    def __contains__(self, key) -> bool:
+        """Check if the provided test suite name has been selected and/or configured."""
+        return key in self.model_fields_set
+
+    def __getitem__(self, key) -> TestSuiteConfig:
+        """Get test suite configuration."""
+        return self.__getattribute__(key)
+
+    def get_configs(self) -> Iterable[TestSuiteConfig]:
+        """Get all the test suite configurations."""
+        return map(lambda t: self[t], self.model_fields_set)
+
+    @classmethod
+    def available_test_suites(cls) -> Iterable[str]:
+        """List all the available test suites."""
+        return cls.model_fields.keys()
+
+    @field_validator("*", mode="before")
+    @classmethod
+    def pass_test_suite_name_to_config(cls, field_value: Any, info: ValidationInfo) -> Any:
+        """Before validating any :class:`TestSuiteConfig`, pass the test suite name via context."""
+        test_suite_name = info.field_name
+        assert test_suite_name is not None
+
+        assert info.context is not None, "A context dictionary is required to load test suites."
+        info.context.update({"test_suite_name": test_suite_name})
+
+        return field_value
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 5f5837a132..2ab8861f99 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -25,6 +25,8 @@
 from types import MethodType
 from typing import Iterable
 
+from pydantic import ValidationError
+
 from framework.testbed_model.capability import Capability, get_supported_capabilities
 from framework.testbed_model.sut_node import SutNode
 from framework.testbed_model.tg_node import TGNode
@@ -34,11 +36,17 @@
     DPDKPrecompiledBuildConfiguration,
     SutNodeConfiguration,
     TestRunConfiguration,
-    TestSuiteConfig,
     TGNodeConfiguration,
     load_config,
 )
-from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError
+from .config.generated import CUSTOM_CONFIG_TYPES
+from .config.test_suite import TestSuiteConfig
+from .exception import (
+    BlockingTestSuiteError,
+    ConfigurationError,
+    SSHTimeoutError,
+    TestCaseVerifyError,
+)
 from .logger import DTSLogger, DtsStage, get_dts_logger
 from .settings import SETTINGS
 from .test_result import (
@@ -141,12 +149,7 @@ def run(self) -> None:
                 self._logger.info(f"Running test run with SUT '{sut_node_config.name}'.")
                 self._init_random_seed(test_run_config)
                 test_run_result = self._result.add_test_run(test_run_config)
-                # we don't want to modify the original config, so create a copy
-                test_run_test_suites = list(
-                    SETTINGS.test_suites if SETTINGS.test_suites else test_run_config.test_suites
-                )
-                if not test_run_config.skip_smoke_tests:
-                    test_run_test_suites[:0] = [TestSuiteConfig(test_suite="smoke_tests")]
+                test_run_test_suites = self._prepare_test_suites(test_run_config)
                 try:
                     test_suites_with_cases = self._get_test_suites_with_cases(
                         test_run_test_suites, test_run_config.func, test_run_config.perf
@@ -203,6 +206,46 @@ def _check_dts_python_version(self) -> None:
             )
             self._logger.warning("Please use Python >= 3.10 instead.")
 
+    def _prepare_test_suites(self, test_run_config: TestRunConfiguration) -> list[TestSuiteConfig]:
+        if SETTINGS.test_suites:
+            test_suites_configs = []
+            for selected_test_suite, selected_test_cases in SETTINGS.test_suites:
+                if selected_test_suite in test_run_config.test_suites:
+                    config = test_run_config.test_suites[selected_test_suite].model_copy(
+                        update={"test_cases_names": selected_test_cases}
+                    )
+                else:
+                    try:
+                        config = CUSTOM_CONFIG_TYPES[selected_test_suite](
+                            test_suite_name=selected_test_suite, test_cases=selected_test_cases
+                        )
+                    except AssertionError as e:
+                        raise ConfigurationError(
+                            "Invalid test cases were selected "
+                            f"for test suite {selected_test_suite}."
+                        ) from e
+                    except ValidationError as e:
+                        raise ConfigurationError(
+                            f"Test suite {selected_test_suite} needs to be explicitly configured "
+                            "in order to be selected."
+                        ) from e
+                    except KeyError:
+                        # not a custom configuration
+                        config = TestSuiteConfig(
+                            test_suite_name=selected_test_suite, test_cases=selected_test_cases
+                        )
+                test_suites_configs.append(config)
+        else:
+            # we don't want to modify the original config, so create a copy
+            test_suites_configs = [
+                config.model_copy() for config in test_run_config.test_suites.get_configs()
+            ]
+
+        if not test_run_config.skip_smoke_tests:
+            test_suites_configs[:0] = [TestSuiteConfig(test_suite_name="smoke_tests")]
+
+        return test_suites_configs
+
     def _get_test_suites_with_cases(
         self,
         test_suite_configs: list[TestSuiteConfig],
@@ -236,7 +279,11 @@ def _get_test_suites_with_cases(
                 test_cases.extend(perf_test_cases)
 
             test_suites_with_cases.append(
-                TestSuiteWithCases(test_suite_class=test_suite_class, test_cases=test_cases)
+                TestSuiteWithCases(
+                    test_suite_class=test_suite_class,
+                    test_cases=test_cases,
+                    config=test_suite_config,
+                )
             )
         return test_suites_with_cases
 
@@ -453,7 +500,9 @@ def _run_test_suite(
         self._logger.set_stage(
             DtsStage.test_suite_setup, Path(SETTINGS.output_dir, test_suite_name)
         )
-        test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node, topology)
+        test_suite = test_suite_with_cases.test_suite_class(
+            sut_node, tg_node, topology, test_suite_with_cases.config
+        )
         try:
             self._logger.info(f"Starting test suite setup: {test_suite_name}")
             test_suite.set_up_suite()
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 5a8e6e5aee..f8783c4b59 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -107,7 +107,7 @@
     LocalDPDKTreeLocation,
     RemoteDPDKTarballLocation,
     RemoteDPDKTreeLocation,
-    TestSuiteConfig,
+    TestSuitesConfigs,
 )
 
 
@@ -133,7 +133,7 @@ class Settings:
     #:
     compile_timeout: float = 1200
     #:
-    test_suites: list[TestSuiteConfig] = field(default_factory=list)
+    test_suites: list[tuple[str, list[str]]] = field(default_factory=list)
     #:
     re_run: int = 0
     #:
@@ -508,7 +508,7 @@ def _process_dpdk_location(
 
 def _process_test_suites(
     parser: _DTSArgumentParser, args: list[list[str]]
-) -> list[TestSuiteConfig]:
+) -> list[tuple[str, list[str]]]:
     """Process the given argument to a list of :class:`TestSuiteConfig` to execute.
 
     Args:
@@ -524,19 +524,17 @@ def _process_test_suites(
         # Environment variable in the form of "SUITE1 CASE1 CASE2, SUITE2 CASE1, SUITE3, ..."
         args = [suite_with_cases.split() for suite_with_cases in args[0][0].split(",")]
 
-    try:
-        return [
-            TestSuiteConfig(test_suite=test_suite, test_cases=test_cases)
-            for [test_suite, *test_cases] in args
-        ]
-    except ValidationError as e:
-        print(
-            "An error has occurred while validating the test suites supplied in the "
-            f"{'environment variable' if action else 'arguments'}:",
-            file=sys.stderr,
-        )
-        print(e, file=sys.stderr)
-        sys.exit(1)
+    available_test_suites = TestSuitesConfigs.available_test_suites()
+    for test_suite_name, *_ in args:
+        if test_suite_name not in available_test_suites:
+            print(
+                f"The test suite {test_suite_name} supplied in the "
+                f"{'environment variable' if action else 'arguments'} is invalid.",
+                file=sys.stderr,
+            )
+            sys.exit(1)
+
+    return [(test_suite, test_cases) for test_suite, *test_cases in args]
 
 
 def get_settings() -> Settings:
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 6014d281b5..8c0c1bcfb3 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -30,7 +30,8 @@
 
 from framework.testbed_model.capability import Capability
 
-from .config import TestRunConfiguration, TestSuiteConfig
+from .config import TestRunConfiguration
+from .config.test_suite import TestSuiteConfig
 from .exception import DTSError, ErrorSeverity
 from .logger import DTSLogger
 from .settings import SETTINGS
@@ -59,23 +60,13 @@ class is to hold a subset of test cases (which could be all test cases) because
     test_suite_class: type[TestSuite]
     test_cases: list[type[TestCase]]
     required_capabilities: set[Capability] = field(default_factory=set, init=False)
+    config: TestSuiteConfig
 
     def __post_init__(self):
         """Gather the required capabilities of the test suite and all test cases."""
         for test_object in [self.test_suite_class] + self.test_cases:
             self.required_capabilities.update(test_object.required_capabilities)
 
-    def create_config(self) -> TestSuiteConfig:
-        """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
-
-        Returns:
-            The :class:`TestSuiteConfig` representation.
-        """
-        return TestSuiteConfig(
-            test_suite=self.test_suite_class.__name__,
-            test_cases=[test_case.__name__ for test_case in self.test_cases],
-        )
-
     def mark_skip_unsupported(self, supported_capabilities: set[Capability]) -> None:
         """Mark the test suite and test cases to be skipped.
 
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index fb5d646ce3..24cd0d38c1 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -24,13 +24,14 @@
 from ipaddress import IPv4Interface, IPv6Interface, ip_interface
 from pkgutil import iter_modules
 from types import ModuleType
-from typing import ClassVar, Protocol, TypeVar, Union, cast
+from typing import ClassVar, Protocol, TypeVar, Union, cast, get_type_hints
 
 from scapy.layers.inet import IP  # type: ignore[import-untyped]
 from scapy.layers.l2 import Ether  # type: ignore[import-untyped]
 from scapy.packet import Packet, Padding, raw  # type: ignore[import-untyped]
 from typing_extensions import Self
 
+from framework.config.test_suite import TestSuiteConfig
 from framework.testbed_model.capability import TestProtocol
 from framework.testbed_model.port import Port
 from framework.testbed_model.sut_node import SutNode
@@ -80,6 +81,7 @@ class TestSuite(TestProtocol):
     #: Whether the test suite is blocking. A failure of a blocking test suite
     #: will block the execution of all subsequent test suites in the current test run.
     is_blocking: ClassVar[bool] = False
+    config: TestSuiteConfig
     _logger: DTSLogger
     _sut_port_ingress: Port
     _sut_port_egress: Port
@@ -95,6 +97,7 @@ def __init__(
         sut_node: SutNode,
         tg_node: TGNode,
         topology: Topology,
+        config: TestSuiteConfig,
     ):
         """Initialize the test suite testbed information and basic configuration.
 
@@ -105,9 +108,11 @@ def __init__(
             sut_node: The SUT node where the test suite will run.
             tg_node: The TG node where the test suite will run.
             topology: The topology where the test suite will run.
+            config: The test suite configuration.
         """
         self.sut_node = sut_node
         self.tg_node = tg_node
+        self.config = config
         self._logger = get_dts_logger(self.__class__.__name__)
         self._tg_port_egress = topology.tg_port_egress
         self._sut_port_ingress = topology.sut_port_ingress
@@ -663,6 +668,21 @@ def is_test_suite(obj) -> bool:
             f"Expected class {self.class_name} not found in module {self.module_name}."
         )
 
+    @cached_property
+    def config_obj(self) -> type[TestSuiteConfig]:
+        """A reference to the test suite's configuration type."""
+        fields = get_type_hints(self.class_obj)
+        config_obj = fields.get("config")
+        if config_obj is None:
+            raise InternalError(
+                "Test suite class {self.class_name} is missing the `config` attribute."
+            )
+        if not issubclass(config_obj, TestSuiteConfig):
+            raise InternalError(
+                f"Test suite class {self.class_name} has an invalid configuration type assigned."
+            )
+        return config_obj
+
     @classmethod
     def discover_all(
         cls, package_name: str | None = None, module_prefix: str | None = None
diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
index 734f006026..f2998b968e 100644
--- a/dts/tests/TestSuite_hello_world.py
+++ b/dts/tests/TestSuite_hello_world.py
@@ -15,12 +15,15 @@
     LogicalCoreCountFilter,
     LogicalCoreList,
 )
+from tests.config import HelloWorldConfig
 
 
 @requires(topology_type=TopologyType.no_link)
 class TestHelloWorld(TestSuite):
     """DPDK hello world app test suite."""
 
+    config: HelloWorldConfig
+
     def set_up_suite(self) -> None:
         """Set up the test suite.
 
@@ -63,7 +66,7 @@ def hello_world_all_cores(self) -> None:
         eal_para = compute_eal_params(
             self.sut_node, lcore_filter_specifier=LogicalCoreList(self.sut_node.lcores)
         )
-        result = self.sut_node.run_dpdk_app(self.app_helloworld_path, eal_para, 50)
+        result = self.sut_node.run_dpdk_app(self.app_helloworld_path, eal_para, self.config.timeout)
         for lcore in self.sut_node.lcores:
             self.verify(
                 f"hello from core {int(lcore)}" in result.stdout,
diff --git a/dts/tests/__init__.py b/dts/tests/__init__.py
new file mode 100644
index 0000000000..a300eb26fc
--- /dev/null
+++ b/dts/tests/__init__.py
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Test suites.
+
+This package contains all the available test suites in DTS.
+"""
diff --git a/dts/tests/config.py b/dts/tests/config.py
new file mode 100644
index 0000000000..300ad3ef6a
--- /dev/null
+++ b/dts/tests/config.py
@@ -0,0 +1,20 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Module for test suites custom configurations.
+
+Any test suite that requires custom configuration fields should create a new config class inheriting
+:class:`~framework.config.test_suite.TestSuiteConfig`, while respecting the parents' frozen state.
+Any custom fields can be added in this class.
+
+The custom configuration classes can be stored in this module.
+"""
+
+from framework.config.test_suite import TestSuiteConfig
+
+
+class HelloWorldConfig(TestSuiteConfig):
+    """Example custom configuration for the `TestHelloWorld` test suite."""
+
+    #: Timeout for the DPDK apps.
+    timeout: int = 50
-- 
2.43.0


^ permalink raw reply	[flat|nested] 17+ messages in thread

* [PATCH v2 4/4] dts: update autodoc sorting order
  2024-11-08 13:38 ` [PATCH v2 0/4] " Luca Vizzarro
                     ` (2 preceding siblings ...)
  2024-11-08 13:38   ` [PATCH v2 3/4] dts: add per-test-suite configuration Luca Vizzarro
@ 2024-11-08 13:38   ` Luca Vizzarro
  3 siblings, 0 replies; 17+ messages in thread
From: Luca Vizzarro @ 2024-11-08 13:38 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Patrick Robb, Luca Vizzarro

The autodoc member sorting order default is set to alphabetical, which
translates to autodoc sorting every member in modules, classes etc.
This also brings some side effects, like sorting capabilities which
can't be compared and result in errors.

This change prevents autodoc from sorting, and keeping the order as the
developer intended it.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 doc/guides/conf.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/guides/conf.py b/doc/guides/conf.py
index 71fed45b3d..ed5bc5eb30 100644
--- a/doc/guides/conf.py
+++ b/doc/guides/conf.py
@@ -88,6 +88,7 @@
     autodoc_typehints = 'both'
     autodoc_typehints_format = 'short'
     autodoc_typehints_description_target = 'documented'
+    autodoc_member_order = 'bysource'
 
     # Intersphinx allows linking to external projects, such as Python docs.
     intersphinx_mapping = {'python': ('https://docs.python.org/3', None)}
-- 
2.43.0


^ permalink raw reply	[flat|nested] 17+ messages in thread

* [PATCH v3 0/4] dts: add per-test-suite configuration
  2024-09-06 16:13 [PATCH] dts: add per-test-suite configuration Luca Vizzarro
  2024-09-27 17:45 ` Jeremy Spewock
  2024-11-08 13:38 ` [PATCH v2 0/4] " Luca Vizzarro
@ 2024-11-08 13:45 ` Luca Vizzarro
  2024-11-08 13:45   ` [PATCH v3 1/4] dts: add tests package to API docs Luca Vizzarro
                     ` (3 more replies)
  2 siblings, 4 replies; 17+ messages in thread
From: Luca Vizzarro @ 2024-11-08 13:45 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Patrick Robb, Luca Vizzarro

Hi there,

sending a v3 for the per-test-suite configuration additions.

v3:
- fixed bug in dts-check-format
v2:
- rebase
- added test suites to doc pages
- doc fixes
- updated the custom config procedure to using tests.config
- simplified the generated file
- implemented the generator in dts-check-format

Best regards,
Luca Vizzarro

Depends-on: series-33871 ("dts: Pydantic configuration")

Luca Vizzarro (4):
  dts: add tests package to API docs
  dts: fix smoke tests docstring
  dts: add per-test-suite configuration
  dts: update autodoc sorting order

 devtools/dts-check-format.sh                  |  28 +++-
 devtools/dts-generate-tests-mappings.py       | 117 +++++++++++++
 doc/api/dts/framework.config.generated.rst    |   8 +
 doc/api/dts/framework.config.rst              |   7 +
 doc/api/dts/framework.config.test_suite.rst   |   8 +
 doc/api/dts/index.rst                         |   1 +
 doc/api/dts/tests.TestSuite_hello_world.rst   |   9 +
 doc/api/dts/tests.TestSuite_os_udp.rst        |   9 +
 .../tests.TestSuite_pmd_buffer_scatter.rst    |   9 +
 doc/api/dts/tests.TestSuite_smoke_tests.rst   |   9 +
 doc/api/dts/tests.TestSuite_vlan.rst          |  10 ++
 doc/api/dts/tests.config.rst                  |   9 +
 doc/api/dts/tests.rst                         |  20 +++
 doc/guides/conf.py                            |   1 +
 doc/guides/tools/dts.rst                      |  23 +++
 dts/conf.yaml                                 |   4 +-
 dts/framework/config/__init__.py              |  76 +--------
 dts/framework/config/generated.py             |  25 +++
 dts/framework/config/test_suite.py            | 154 ++++++++++++++++++
 dts/framework/runner.py                       |  69 ++++++--
 dts/framework/settings.py                     |  30 ++--
 dts/framework/test_result.py                  |  15 +-
 dts/framework/test_suite.py                   |  22 ++-
 dts/tests/TestSuite_hello_world.py            |   5 +-
 dts/tests/TestSuite_smoke_tests.py            |   2 -
 dts/tests/__init__.py                         |   7 +
 dts/tests/config.py                           |  20 +++
 27 files changed, 580 insertions(+), 117 deletions(-)
 create mode 100755 devtools/dts-generate-tests-mappings.py
 create mode 100644 doc/api/dts/framework.config.generated.rst
 create mode 100644 doc/api/dts/framework.config.test_suite.rst
 create mode 100644 doc/api/dts/tests.TestSuite_hello_world.rst
 create mode 100644 doc/api/dts/tests.TestSuite_os_udp.rst
 create mode 100644 doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
 create mode 100644 doc/api/dts/tests.TestSuite_smoke_tests.rst
 create mode 100644 doc/api/dts/tests.TestSuite_vlan.rst
 create mode 100644 doc/api/dts/tests.config.rst
 create mode 100644 doc/api/dts/tests.rst
 create mode 100644 dts/framework/config/generated.py
 create mode 100644 dts/framework/config/test_suite.py
 create mode 100644 dts/tests/__init__.py
 create mode 100644 dts/tests/config.py

-- 
2.43.0


^ permalink raw reply	[flat|nested] 17+ messages in thread

* [PATCH v3 1/4] dts: add tests package to API docs
  2024-11-08 13:45 ` [PATCH v3 0/4] dts: add per-test-suite configuration Luca Vizzarro
@ 2024-11-08 13:45   ` Luca Vizzarro
  2024-11-12 19:55     ` Dean Marx
  2024-11-08 13:45   ` [PATCH v3 2/4] dts: fix smoke tests docstring Luca Vizzarro
                     ` (2 subsequent siblings)
  3 siblings, 1 reply; 17+ messages in thread
From: Luca Vizzarro @ 2024-11-08 13:45 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Patrick Robb, Luca Vizzarro

The test suites are not documented in the API as their respective
documentation files are missing.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 doc/api/dts/index.rst                         |  1 +
 doc/api/dts/tests.TestSuite_hello_world.rst   |  9 +++++++++
 doc/api/dts/tests.TestSuite_os_udp.rst        |  9 +++++++++
 .../tests.TestSuite_pmd_buffer_scatter.rst    |  9 +++++++++
 doc/api/dts/tests.TestSuite_smoke_tests.rst   |  9 +++++++++
 doc/api/dts/tests.TestSuite_vlan.rst          | 10 ++++++++++
 doc/api/dts/tests.rst                         | 19 +++++++++++++++++++
 7 files changed, 66 insertions(+)
 create mode 100644 doc/api/dts/tests.TestSuite_hello_world.rst
 create mode 100644 doc/api/dts/tests.TestSuite_os_udp.rst
 create mode 100644 doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
 create mode 100644 doc/api/dts/tests.TestSuite_smoke_tests.rst
 create mode 100644 doc/api/dts/tests.TestSuite_vlan.rst
 create mode 100644 doc/api/dts/tests.rst

diff --git a/doc/api/dts/index.rst b/doc/api/dts/index.rst
index 534512dc17..db05fd37d5 100644
--- a/doc/api/dts/index.rst
+++ b/doc/api/dts/index.rst
@@ -14,6 +14,7 @@ Packages
    :includehidden:
    :maxdepth: 1
 
+   tests
    framework.testbed_model
    framework.remote_session
    framework.params
diff --git a/doc/api/dts/tests.TestSuite_hello_world.rst b/doc/api/dts/tests.TestSuite_hello_world.rst
new file mode 100644
index 0000000000..f92786996f
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_hello_world.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+hello_world Test Suite
+======================
+
+.. automodule:: tests.TestSuite_hello_world
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.TestSuite_os_udp.rst b/doc/api/dts/tests.TestSuite_os_udp.rst
new file mode 100644
index 0000000000..121a653670
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_os_udp.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+os_udp Test Suite
+=================
+
+.. automodule:: tests.TestSuite_os_udp
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst b/doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
new file mode 100644
index 0000000000..b6571b24f5
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+pmd_buffer_scatter Test Suite
+=============================
+
+.. automodule:: tests.TestSuite_pmd_buffer_scatter
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.TestSuite_smoke_tests.rst b/doc/api/dts/tests.TestSuite_smoke_tests.rst
new file mode 100644
index 0000000000..5b11aa7f48
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_smoke_tests.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+smoke_tests Test Suite
+======================
+
+.. automodule:: tests.TestSuite_smoke_tests
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.TestSuite_vlan.rst b/doc/api/dts/tests.TestSuite_vlan.rst
new file mode 100644
index 0000000000..86a29538bb
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_vlan.rst
@@ -0,0 +1,10 @@
+
+.. SPDX-License-Identifier: BSD-3-Clause
+
+vlan Test Suite
+===============
+
+.. automodule:: tests.TestSuite_vlan
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.rst b/doc/api/dts/tests.rst
new file mode 100644
index 0000000000..0c136b4bb0
--- /dev/null
+++ b/doc/api/dts/tests.rst
@@ -0,0 +1,19 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+tests - Test Suites Package
+==============================
+
+.. automodule:: tests
+   :members:
+   :show-inheritance:
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   tests.TestSuite_hello_world
+   tests.TestSuite_os_udp
+   tests.TestSuite_pmd_buffer_scatter
+   tests.TestSuite_smoke_tests
+   tests.TestSuite_vlan
+
-- 
2.43.0


^ permalink raw reply	[flat|nested] 17+ messages in thread

* [PATCH v3 2/4] dts: fix smoke tests docstring
  2024-11-08 13:45 ` [PATCH v3 0/4] dts: add per-test-suite configuration Luca Vizzarro
  2024-11-08 13:45   ` [PATCH v3 1/4] dts: add tests package to API docs Luca Vizzarro
@ 2024-11-08 13:45   ` Luca Vizzarro
  2024-11-08 13:45   ` [PATCH v3 3/4] dts: add per-test-suite configuration Luca Vizzarro
  2024-11-08 13:45   ` [PATCH v3 4/4] dts: update autodoc sorting order Luca Vizzarro
  3 siblings, 0 replies; 17+ messages in thread
From: Luca Vizzarro @ 2024-11-08 13:45 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Patrick Robb, Luca Vizzarro

Sphinx autodoc complains of is_blocking being duplicated. This is the
case as the parent already holds a docstring for this attribute. Remove
the duplication.

Fixes: 6ef07151aac4 ("dts: update docstrings")

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 dts/tests/TestSuite_smoke_tests.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
index bc3a2a6bf9..f564ac82b4 100644
--- a/dts/tests/TestSuite_smoke_tests.py
+++ b/dts/tests/TestSuite_smoke_tests.py
@@ -30,8 +30,6 @@ class TestSmokeTests(TestSuite):
     The infrastructure also needs to be tested, as that is also used by all other test suites.
 
     Attributes:
-        is_blocking: This test suite will block the execution of all other test suites
-            in the test run after it.
         nics_in_node: The NICs present on the SUT node.
     """
 
-- 
2.43.0


^ permalink raw reply	[flat|nested] 17+ messages in thread

* [PATCH v3 3/4] dts: add per-test-suite configuration
  2024-11-08 13:45 ` [PATCH v3 0/4] dts: add per-test-suite configuration Luca Vizzarro
  2024-11-08 13:45   ` [PATCH v3 1/4] dts: add tests package to API docs Luca Vizzarro
  2024-11-08 13:45   ` [PATCH v3 2/4] dts: fix smoke tests docstring Luca Vizzarro
@ 2024-11-08 13:45   ` Luca Vizzarro
  2024-11-13 21:44     ` Dean Marx
  2024-11-08 13:45   ` [PATCH v3 4/4] dts: update autodoc sorting order Luca Vizzarro
  3 siblings, 1 reply; 17+ messages in thread
From: Luca Vizzarro @ 2024-11-08 13:45 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Patrick Robb, Luca Vizzarro

Allow test suites to be configured individually. Moreover enable them to
implement their own custom configuration.

This solution adds some new complexity to DTS, which is generated source
code. In order to ensure strong typing, the test suites and their custom
configurations need to be linked in the main configuration class.
Unfortunately, this is not feasible during runtime as it will incur in
circular dependencies. Generating the links appear to be the most
straightforward approach.

This commit also brings a new major change to the configuration schema.
Test suites are no longer defined as a list of strings, like:

    test_suites:
    - hello_world
    - pmd_buffer_scatter

but as mapping of mappings or strings:

    test_suites:
      hello_world: {} # any custom fields or test cases can be set here
      pmd_buffer_scatter: all # "all" defines all the test cases, or
                              # they can individually be set separated
                              # by a space

Not defining the `test_cases` field in the configuration is equivalent
to `all`, therefore the definitions for either test suite above are
also equivalent.

Creating the __init__.py file under the tests folder, allows it to be
picked up as a package. This is a mypy requirement to import the tests
from within the framework.

Bugzilla ID: 1375

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 devtools/dts-check-format.sh                |  28 +++-
 devtools/dts-generate-tests-mappings.py     | 117 +++++++++++++++
 doc/api/dts/framework.config.generated.rst  |   8 +
 doc/api/dts/framework.config.rst            |   7 +
 doc/api/dts/framework.config.test_suite.rst |   8 +
 doc/api/dts/tests.config.rst                |   9 ++
 doc/api/dts/tests.rst                       |   1 +
 doc/guides/tools/dts.rst                    |  23 +++
 dts/conf.yaml                               |   4 +-
 dts/framework/config/__init__.py            |  76 +---------
 dts/framework/config/generated.py           |  25 ++++
 dts/framework/config/test_suite.py          | 154 ++++++++++++++++++++
 dts/framework/runner.py                     |  69 +++++++--
 dts/framework/settings.py                   |  30 ++--
 dts/framework/test_result.py                |  15 +-
 dts/framework/test_suite.py                 |  22 ++-
 dts/tests/TestSuite_hello_world.py          |   5 +-
 dts/tests/__init__.py                       |   7 +
 dts/tests/config.py                         |  20 +++
 19 files changed, 513 insertions(+), 115 deletions(-)
 create mode 100755 devtools/dts-generate-tests-mappings.py
 create mode 100644 doc/api/dts/framework.config.generated.rst
 create mode 100644 doc/api/dts/framework.config.test_suite.rst
 create mode 100644 doc/api/dts/tests.config.rst
 create mode 100644 dts/framework/config/generated.py
 create mode 100644 dts/framework/config/test_suite.py
 create mode 100644 dts/tests/__init__.py
 create mode 100644 dts/tests/config.py

diff --git a/devtools/dts-check-format.sh b/devtools/dts-check-format.sh
index 3f43e17e88..bad771fea3 100755
--- a/devtools/dts-check-format.sh
+++ b/devtools/dts-check-format.sh
@@ -2,6 +2,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2022 University of New Hampshire
 # Copyright(c) 2023 PANTHEON.tech s.r.o.
+# Copyright(c) 2024 Arm Limited
 
 usage() {
 	echo "Usage: $(basename $0) [options] [directory]"
@@ -13,15 +14,19 @@ usage() {
 format=true
 lint=true
 typecheck=true
+generate=true
 
 # Comments after args serve as documentation; must be present
-while getopts "hflt" arg; do
+while getopts "hgflt" arg; do
 	case $arg in
 	h) # Display this message
-		echo 'Run formatting and linting programs for DTS.'
+		echo 'Run generating, formatting and linting programs for DTS.'
 		usage
 		exit 0
 		;;
+	g) # Don't run code generator
+		generate=false
+		;;
 	f) # Don't run formatters
 		format=false
 		;;
@@ -48,7 +53,22 @@ heading() {
 
 errors=0
 
+if $generate; then
+	heading "Generating test suites to configuration mappings"
+	if command -v python3 > /dev/null; then
+		../devtools/dts-generate-tests-mappings.py
+		errors=$((errors + $?))
+	else
+		echo "python3 not found, unable to run generator"
+		errors=$((errors + 1))
+	fi
+fi
+
 if $format; then
+	if $generate; then
+		echo
+	fi
+
 	if command -v git > /dev/null; then
 		if git rev-parse --is-inside-work-tree >&-; then
 			heading "Formatting in $directory/"
@@ -85,7 +105,7 @@ if $format; then
 fi
 
 if $lint; then
-	if $format; then
+	if $generate || $format; then
 		echo
 	fi
 	heading "Linting in $directory/"
@@ -99,7 +119,7 @@ if $lint; then
 fi
 
 if $typecheck; then
-	if $format || $lint; then
+	if $generate || $format || $lint; then
 		echo
 	fi
 	heading "Checking types in $directory/"
diff --git a/devtools/dts-generate-tests-mappings.py b/devtools/dts-generate-tests-mappings.py
new file mode 100755
index 0000000000..26ecc1018c
--- /dev/null
+++ b/devtools/dts-generate-tests-mappings.py
@@ -0,0 +1,117 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""DTS Test Suites to Configuration mappings generation script."""
+
+import os
+import sys
+from collections import defaultdict
+from pathlib import Path
+from textwrap import indent
+from typing import Iterable
+
+DTS_DIR = Path(__file__).parent.joinpath("..", "dts").resolve()
+SCRIPT_FILE_NAME = Path(__file__).relative_to(Path(__file__).parent.parent)
+
+sys.path.append(str(DTS_DIR))
+
+from framework.config.test_suite import BaseTestSuitesConfigs, TestSuiteConfig
+from framework.exception import InternalError
+from framework.test_suite import AVAILABLE_TEST_SUITES, TestSuiteSpec
+
+FRAMEWORK_IMPORTS = [BaseTestSuitesConfigs, TestSuiteConfig]
+
+RELATIVE_PATH_TO_GENERATED_FILE = "framework/config/generated.py"
+SMOKE_TESTS_SUITE_NAME = "smoke_tests"
+CUSTOM_CONFIG_TYPES_VAR_NAME = "CUSTOM_CONFIG_TYPES"
+CUSTOM_CONFIG_TYPES_VAR_DOCSTRING = [
+    "#: Mapping of test suites to their corresponding custom configuration objects if any."
+]
+TEST_SUITES_CONFIG_CLASS_NAME = "TestSuitesConfigs"
+TEST_SUITES_CONFIG_CLASS_DOCSTRING = [
+    '"""Configuration mapping class to select and configure the test suites."""',
+]
+
+
+GENERATED_FILE_HEADER = f"""# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 The DPDK contributors
+# This file is automatically generated by {SCRIPT_FILE_NAME}.
+# Do NOT modify this file manually.
+
+\"\"\"Generated file containing the links between the test suites and the configuration.\"\"\"
+"""
+
+
+def join(lines: Iterable[str]) -> str:
+    """Join list of strings into text lines."""
+    return "\n".join(lines)
+
+
+def join_and_indent(lines: Iterable[str], indentation_level=1, indentation_spaces=4) -> str:
+    """Join list of strings into indented text lines."""
+    return "\n".join([indent(line, " " * indentation_level * indentation_spaces) for line in lines])
+
+
+def format_attributes_types(test_suite_spec: TestSuiteSpec):
+    """Format the config type into the respective configuration class field attribute type."""
+    config_type = test_suite_spec.config_obj.__name__
+    return f"Optional[{config_type}]"
+
+
+try:
+    framework_imports: dict[str, list[str]] = defaultdict(list)
+    for _import in FRAMEWORK_IMPORTS:
+        framework_imports[_import.__module__].append(_import.__name__)
+    formatted_framework_imports = sorted(
+        [
+            f"from {module} import {', '.join(sorted(imports))}"
+            for module, imports in framework_imports.items()
+        ]
+    )
+
+    test_suites = [
+        test_suite_spec
+        for test_suite_spec in AVAILABLE_TEST_SUITES
+        if test_suite_spec.name != SMOKE_TESTS_SUITE_NAME
+    ]
+
+    custom_configs = [t for t in test_suites if t.config_obj is not TestSuiteConfig]
+
+    custom_config_imports = [
+        f"from {t.config_obj.__module__} import {t.config_obj.__name__}" for t in custom_configs
+    ]
+
+    test_suites_attributes = [f"{t.name}: {format_attributes_types(t)} = None" for t in test_suites]
+
+    custom_config_mappings = [f'"{t.name}": {t.config_obj.__name__},' for t in custom_configs]
+
+    generated_file_contents = f"""{GENERATED_FILE_HEADER}
+from typing import Optional
+
+{join(formatted_framework_imports)}
+
+{join(custom_config_imports)}
+
+{join(CUSTOM_CONFIG_TYPES_VAR_DOCSTRING)}
+{CUSTOM_CONFIG_TYPES_VAR_NAME}: dict[str, type[{TestSuiteConfig.__name__}]] = {'{'}
+{join_and_indent(custom_config_mappings)}
+{'}'}
+
+
+class {TEST_SUITES_CONFIG_CLASS_NAME}({BaseTestSuitesConfigs.__name__}):
+{join_and_indent(TEST_SUITES_CONFIG_CLASS_DOCSTRING)}
+
+{join_and_indent(test_suites_attributes)}
+"""
+
+    path = os.path.join(DTS_DIR, RELATIVE_PATH_TO_GENERATED_FILE)
+
+    with open(path, "w") as generated_file:
+        generated_file.write(generated_file_contents)
+
+    print("Test suites to configuration mappings generated successfully!")
+except Exception as e:
+    raise InternalError(
+        "Failed to generate test suites to configuration mappings."
+    ) from e
diff --git a/doc/api/dts/framework.config.generated.rst b/doc/api/dts/framework.config.generated.rst
new file mode 100644
index 0000000000..5dfa9342f0
--- /dev/null
+++ b/doc/api/dts/framework.config.generated.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+generated - Generated Test Suite Configurations
+===============================================
+
+.. automodule:: framework.config.generated
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/framework.config.rst b/doc/api/dts/framework.config.rst
index cc266276c1..217fe026c4 100644
--- a/doc/api/dts/framework.config.rst
+++ b/doc/api/dts/framework.config.rst
@@ -6,3 +6,10 @@ config - Configuration Package
 .. automodule:: framework.config
    :members:
    :show-inheritance:
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   framework.config.generated
+   framework.config.test_suite
diff --git a/doc/api/dts/framework.config.test_suite.rst b/doc/api/dts/framework.config.test_suite.rst
new file mode 100644
index 0000000000..d59dcf5d6e
--- /dev/null
+++ b/doc/api/dts/framework.config.test_suite.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+test_suite - Test Suite Configuration Definitions
+=================================================
+
+.. automodule:: framework.config.test_suite
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.config.rst b/doc/api/dts/tests.config.rst
new file mode 100644
index 0000000000..ce3d9df868
--- /dev/null
+++ b/doc/api/dts/tests.config.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+Test Suites Configurations
+==========================
+
+.. automodule:: tests.config
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.rst b/doc/api/dts/tests.rst
index 0c136b4bb0..7fc25f2123 100644
--- a/doc/api/dts/tests.rst
+++ b/doc/api/dts/tests.rst
@@ -11,6 +11,7 @@ tests - Test Suites Package
    :hidden:
    :maxdepth: 1
 
+   tests.config
    tests.TestSuite_hello_world
    tests.TestSuite_os_udp
    tests.TestSuite_pmd_buffer_scatter
diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index f4e297413d..4e63601b19 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -408,6 +408,29 @@ There are four types of methods that comprise a test suite:
    should be implemented in the ``SutNode`` class (and the underlying classes that ``SutNode`` uses)
    and used by the test suite via the ``sut_node`` field.
 
+The test suites can also implement their own custom configuration fields. This can be achieved by
+creating a new test suite config file which inherits from ``TestSuiteConfig`` defined in
+``dts/framework/config/test_suite.py``. So that this new custom configuration class is used, the
+test suite class must override the ``config`` attribute annotation with your new class, for example:
+
+.. code:: python
+
+   # place this under tests/config.py to avoid circular dependencies
+   class CustomConfig(TestSuiteConfig):
+      my_custom_field: int = 10
+
+   # place this under tests/TestSuite_my_new_test_suite.py
+   class TestMyNewTestSuite(TestSuite):
+      config: CustomConfig
+
+Finally, the test suites and the custom configuration files need to linked in the global configuration.
+This can be easily achieved by running the ``dts/generate-test-mappings.py``, e.g.:
+
+.. code-block:: console
+
+   $ poetry shell
+   (dts-py3.10) $ ./generate-test-mappings.py
+
 
 .. _dts_dev_tools:
 
diff --git a/dts/conf.yaml b/dts/conf.yaml
index 2496262854..377304dddf 100644
--- a/dts/conf.yaml
+++ b/dts/conf.yaml
@@ -28,8 +28,8 @@ test_runs:
     func: true # enable functional testing
     skip_smoke_tests: false # optional
     test_suites: # the following test suites will be run in their entirety
-      - hello_world
-      - os_udp
+      hello_world: all
+      os_udp: all
     # The machine running the DPDK test executable
     system_under_test_node:
       node_name: "SUT 1"
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 82113a6257..0ac7ab5c46 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -32,11 +32,13 @@
       and makes it thread safe should we ever want to move in that direction.
 """
 
+# pylama:ignore=W0611
+
 import tarfile
 from enum import Enum, auto, unique
 from functools import cached_property
 from pathlib import Path, PurePath
-from typing import TYPE_CHECKING, Annotated, Any, Literal, NamedTuple
+from typing import Annotated, Literal, NamedTuple
 
 import yaml
 from pydantic import (
@@ -52,8 +54,7 @@
 from framework.exception import ConfigurationError
 from framework.utils import REGEX_FOR_PCI_ADDRESS, StrEnum
 
-if TYPE_CHECKING:
-    from framework.test_suite import TestSuiteSpec
+from .generated import TestSuitesConfigs
 
 
 class FrozenModel(BaseModel):
@@ -382,69 +383,6 @@ class DPDKUncompiledBuildConfiguration(BaseDPDKBuildConfiguration):
 DPDKBuildConfiguration = DPDKPrecompiledBuildConfiguration | DPDKUncompiledBuildConfiguration
 
 
-class TestSuiteConfig(FrozenModel):
-    """Test suite configuration.
-
-    Information about a single test suite to be executed. This can also be represented as a string
-    instead of a mapping, example:
-
-    .. code:: yaml
-
-        test_runs:
-        - test_suites:
-            # As string representation:
-            - hello_world # test all of `hello_world`, or
-            - hello_world hello_world_single_core # test only `hello_world_single_core`
-            # or as model fields:
-            - test_suite: hello_world
-              test_cases: [hello_world_single_core] # without this field all test cases are run
-    """
-
-    #: The name of the test suite module without the starting ``TestSuite_``.
-    test_suite_name: str = Field(alias="test_suite")
-    #: The names of test cases from this test suite to execute. If empty, all test cases will be
-    #: executed.
-    test_cases_names: list[str] = Field(default_factory=list, alias="test_cases")
-
-    @cached_property
-    def test_suite_spec(self) -> "TestSuiteSpec":
-        """The specification of the requested test suite."""
-        from framework.test_suite import find_by_name
-
-        test_suite_spec = find_by_name(self.test_suite_name)
-        assert (
-            test_suite_spec is not None
-        ), f"{self.test_suite_name} is not a valid test suite module name."
-        return test_suite_spec
-
-    @model_validator(mode="before")
-    @classmethod
-    def convert_from_string(cls, data: Any) -> Any:
-        """Convert the string representation of the model into a valid mapping."""
-        if isinstance(data, str):
-            [test_suite, *test_cases] = data.split()
-            return dict(test_suite=test_suite, test_cases=test_cases)
-        return data
-
-    @model_validator(mode="after")
-    def validate_names(self) -> Self:
-        """Validate the supplied test suite and test cases names.
-
-        This validator relies on the cached property `test_suite_spec` to run for the first
-        time in this call, therefore triggering the assertions if needed.
-        """
-        available_test_cases = map(
-            lambda t: t.name, self.test_suite_spec.class_obj.get_test_cases()
-        )
-        for requested_test_case in self.test_cases_names:
-            assert requested_test_case in available_test_cases, (
-                f"{requested_test_case} is not a valid test case "
-                f"of test suite {self.test_suite_name}."
-            )
-
-        return self
-
-
 class TestRunSUTNodeConfiguration(FrozenModel):
     """The SUT node configuration of a test run."""
 
@@ -469,8 +407,8 @@ class TestRunConfiguration(FrozenModel):
     func: bool
     #: Whether to skip smoke tests.
     skip_smoke_tests: bool = False
-    #: The names of test suites and/or test cases to execute.
-    test_suites: list[TestSuiteConfig] = Field(min_length=1)
+    #: The test suites to be selected and/or configured.
+    test_suites: TestSuitesConfigs
     #: The SUT node configuration to use in this test run.
     system_under_test_node: TestRunSUTNodeConfiguration
     #: The TG node name to use in this test run.
@@ -602,6 +540,6 @@ def load_config(config_file_path: Path) -> Configuration:
         config_data = yaml.safe_load(f)
 
     try:
-        return Configuration.model_validate(config_data)
+        return Configuration.model_validate(config_data, context={})
     except ValidationError as e:
         raise ConfigurationError("failed to load the supplied configuration") from e
diff --git a/dts/framework/config/generated.py b/dts/framework/config/generated.py
new file mode 100644
index 0000000000..cc4a539987
--- /dev/null
+++ b/dts/framework/config/generated.py
@@ -0,0 +1,25 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 The DPDK contributors
+# This file is automatically generated by devtools/dts-generate-tests-mappings.py.
+# Do NOT modify this file manually.
+
+"""Generated file containing the links between the test suites and the configuration."""
+
+from typing import Optional
+
+from framework.config.test_suite import BaseTestSuitesConfigs, TestSuiteConfig
+from tests.config import HelloWorldConfig
+
+#: Mapping of test suites to their corresponding custom configuration objects if any.
+CUSTOM_CONFIG_TYPES: dict[str, type[TestSuiteConfig]] = {
+    "hello_world": HelloWorldConfig,
+}
+
+
+class TestSuitesConfigs(BaseTestSuitesConfigs):
+    """Configuration mapping class to select and configure the test suites."""
+
+    hello_world: Optional[HelloWorldConfig] = None
+    os_udp: Optional[TestSuiteConfig] = None
+    pmd_buffer_scatter: Optional[TestSuiteConfig] = None
+    vlan: Optional[TestSuiteConfig] = None
diff --git a/dts/framework/config/test_suite.py b/dts/framework/config/test_suite.py
new file mode 100644
index 0000000000..863052cbc1
--- /dev/null
+++ b/dts/framework/config/test_suite.py
@@ -0,0 +1,154 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Test suites configuration module.
+
+Test suites can inherit :class:`TestSuiteConfig` to create their own custom configuration.
+By doing so, the test suite class must also override the annotation of the field
+`~framework.test_suite.TestSuite.config` to use their custom configuration type.
+"""
+
+from functools import cached_property
+from typing import TYPE_CHECKING, Any, Iterable
+
+from pydantic import (
+    BaseModel,
+    ConfigDict,
+    Field,
+    ValidationInfo,
+    field_validator,
+    model_validator,
+)
+from typing_extensions import Self
+
+if TYPE_CHECKING:
+    from framework.test_suite import TestSuiteSpec
+
+
+class TestSuiteConfig(BaseModel):
+    """Test suite configuration base model.
+
+    By default the configuration of a generic test suite does not contain any attributes. Any test
+    suite should inherit this class to create their own custom configuration. Finally override the
+    type of the :attr:`~TestSuite.config` to use the newly created one.
+
+    If no custom fields require setting, this can also be represented as a string instead of
+    a mapping, example:
+
+    .. code:: yaml
+
+        test_runs:
+        - test_suites:
+            # As string representation:
+            hello_world: all # test all of `hello_world`, or
+            hello_world: hello_world_single_core # test only `hello_world_single_core`
+            # or as a mapping of the model's fields:
+            hello_world:
+              test_cases: [hello_world_single_core] # without this field all test cases are run
+
+    .. warning::
+
+        This class sets `protected_namespaces` to an empty tuple as a workaround for autodoc.
+        Due to autodoc loading this class first before any other child ones, it causes the Pydantic
+        fields in the protected namespace ``model_`` to be set on the parent. Leading any child
+        classes to inherit these protected fields as user-defined ones, finally triggering their
+        instances to complain about the presence of protected fields.
+
+        Because any class inheriting this class will therefore have protected namespaces disabled,
+        you won't be blocked to create fields starting with ``model_``. Nonetheless, you **must**
+        refrain from doing so as this is not the intended behavior.
+    """
+
+    model_config = ConfigDict(frozen=True, extra="forbid", protected_namespaces=())
+
+    #: The name of the test suite module without the starting ``TestSuite_``. This field **cannot**
+    #: be used in the configuration file. The name will be inherited from the mapping key instead.
+    test_suite_name: str
+    #: The names of test cases from this test suite to execute. If empty, all test cases will be
+    #: executed.
+    test_cases_names: list[str] = Field(default_factory=list, alias="test_cases")
+
+    @cached_property
+    def test_suite_spec(self) -> "TestSuiteSpec":
+        """The specification of the requested test suite."""
+        from framework.test_suite import find_by_name
+
+        test_suite_spec = find_by_name(self.test_suite_name)
+        assert (
+            test_suite_spec is not None
+        ), f"{self.test_suite_name} is not a valid test suite module name."
+        return test_suite_spec
+
+    @model_validator(mode="before")
+    @classmethod
+    def load_test_suite_name_from_context(cls, data: Any, info: ValidationInfo) -> dict:
+        """Load the test suite name from the validation context, if any."""
+        assert isinstance(data, dict), "The test suite configuration value is invalid."
+        name = data.get("test_suite_name")
+        # If the context is carrying the test suite name, then use it instead.
+        if info.context is not None and (test_suite_name := info.context.get("test_suite_name")):
+            assert not name, "The test suite name cannot be set manually."
+            data["test_suite_name"] = test_suite_name
+        return data
+
+    @model_validator(mode="before")
+    @classmethod
+    def convert_from_string(cls, data: Any) -> dict:
+        """Convert the string representation of the model into a valid mapping."""
+        if isinstance(data, str):
+            test_cases = [] if data == "all" else data.split()
+            return dict(test_cases=test_cases)
+        return data
+
+    @model_validator(mode="after")
+    def validate_names(self) -> Self:
+        """Validate the supplied test suite and test cases names.
+
+        This validator relies on the cached property `test_suite_spec` to run for the first
+        time in this call, therefore triggering the assertions if needed.
+        """
+        available_test_cases = map(
+            lambda t: t.name, self.test_suite_spec.class_obj.get_test_cases()
+        )
+        for requested_test_case in self.test_cases_names:
+            assert requested_test_case in available_test_cases, (
+                f"{requested_test_case} is not a valid test case "
+                f"of test suite {self.test_suite_name}."
+            )
+
+        return self
+
+
+class BaseTestSuitesConfigs(BaseModel):
+    """Base class for test suites configs."""
+
+    model_config = ConfigDict(frozen=True, extra="forbid")
+
+    def __contains__(self, key) -> bool:
+        """Check if the provided test suite name has been selected and/or configured."""
+        return key in self.model_fields_set
+
+    def __getitem__(self, key) -> TestSuiteConfig:
+        """Get test suite configuration."""
+        return self.__getattribute__(key)
+
+    def get_configs(self) -> Iterable[TestSuiteConfig]:
+        """Get all the test suite configurations."""
+        return map(lambda t: self[t], self.model_fields_set)
+
+    @classmethod
+    def available_test_suites(cls) -> Iterable[str]:
+        """List all the available test suites."""
+        return cls.model_fields.keys()
+
+    @field_validator("*", mode="before")
+    @classmethod
+    def pass_test_suite_name_to_config(cls, field_value: Any, info: ValidationInfo) -> Any:
+        """Before validating any :class:`TestSuiteConfig`, pass the test suite name via context."""
+        test_suite_name = info.field_name
+        assert test_suite_name is not None
+
+        assert info.context is not None, "A context dictionary is required to load test suites."
+        info.context.update({"test_suite_name": test_suite_name})
+
+        return field_value
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 5f5837a132..2ab8861f99 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -25,6 +25,8 @@
 from types import MethodType
 from typing import Iterable
 
+from pydantic import ValidationError
+
 from framework.testbed_model.capability import Capability, get_supported_capabilities
 from framework.testbed_model.sut_node import SutNode
 from framework.testbed_model.tg_node import TGNode
@@ -34,11 +36,17 @@
     DPDKPrecompiledBuildConfiguration,
     SutNodeConfiguration,
     TestRunConfiguration,
-    TestSuiteConfig,
     TGNodeConfiguration,
     load_config,
 )
-from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError
+from .config.generated import CUSTOM_CONFIG_TYPES
+from .config.test_suite import TestSuiteConfig
+from .exception import (
+    BlockingTestSuiteError,
+    ConfigurationError,
+    SSHTimeoutError,
+    TestCaseVerifyError,
+)
 from .logger import DTSLogger, DtsStage, get_dts_logger
 from .settings import SETTINGS
 from .test_result import (
@@ -141,12 +149,7 @@ def run(self) -> None:
                 self._logger.info(f"Running test run with SUT '{sut_node_config.name}'.")
                 self._init_random_seed(test_run_config)
                 test_run_result = self._result.add_test_run(test_run_config)
-                # we don't want to modify the original config, so create a copy
-                test_run_test_suites = list(
-                    SETTINGS.test_suites if SETTINGS.test_suites else test_run_config.test_suites
-                )
-                if not test_run_config.skip_smoke_tests:
-                    test_run_test_suites[:0] = [TestSuiteConfig(test_suite="smoke_tests")]
+                test_run_test_suites = self._prepare_test_suites(test_run_config)
                 try:
                     test_suites_with_cases = self._get_test_suites_with_cases(
                         test_run_test_suites, test_run_config.func, test_run_config.perf
@@ -203,6 +206,46 @@ def _check_dts_python_version(self) -> None:
             )
             self._logger.warning("Please use Python >= 3.10 instead.")
 
+    def _prepare_test_suites(self, test_run_config: TestRunConfiguration) -> list[TestSuiteConfig]:
+        if SETTINGS.test_suites:
+            test_suites_configs = []
+            for selected_test_suite, selected_test_cases in SETTINGS.test_suites:
+                if selected_test_suite in test_run_config.test_suites:
+                    config = test_run_config.test_suites[selected_test_suite].model_copy(
+                        update={"test_cases_names": selected_test_cases}
+                    )
+                else:
+                    try:
+                        config = CUSTOM_CONFIG_TYPES[selected_test_suite](
+                            test_suite_name=selected_test_suite, test_cases=selected_test_cases
+                        )
+                    except AssertionError as e:
+                        raise ConfigurationError(
+                            "Invalid test cases were selected "
+                            f"for test suite {selected_test_suite}."
+                        ) from e
+                    except ValidationError as e:
+                        raise ConfigurationError(
+                            f"Test suite {selected_test_suite} needs to be explicitly configured "
+                            "in order to be selected."
+                        ) from e
+                    except KeyError:
+                        # not a custom configuration
+                        config = TestSuiteConfig(
+                            test_suite_name=selected_test_suite, test_cases=selected_test_cases
+                        )
+                test_suites_configs.append(config)
+        else:
+            # we don't want to modify the original config, so create a copy
+            test_suites_configs = [
+                config.model_copy() for config in test_run_config.test_suites.get_configs()
+            ]
+
+        if not test_run_config.skip_smoke_tests:
+            test_suites_configs[:0] = [TestSuiteConfig(test_suite_name="smoke_tests")]
+
+        return test_suites_configs
+
     def _get_test_suites_with_cases(
         self,
         test_suite_configs: list[TestSuiteConfig],
@@ -236,7 +279,11 @@ def _get_test_suites_with_cases(
                 test_cases.extend(perf_test_cases)
 
             test_suites_with_cases.append(
-                TestSuiteWithCases(test_suite_class=test_suite_class, test_cases=test_cases)
+                TestSuiteWithCases(
+                    test_suite_class=test_suite_class,
+                    test_cases=test_cases,
+                    config=test_suite_config,
+                )
             )
         return test_suites_with_cases
 
@@ -453,7 +500,9 @@ def _run_test_suite(
         self._logger.set_stage(
             DtsStage.test_suite_setup, Path(SETTINGS.output_dir, test_suite_name)
         )
-        test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node, topology)
+        test_suite = test_suite_with_cases.test_suite_class(
+            sut_node, tg_node, topology, test_suite_with_cases.config
+        )
         try:
             self._logger.info(f"Starting test suite setup: {test_suite_name}")
             test_suite.set_up_suite()
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 5a8e6e5aee..f8783c4b59 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -107,7 +107,7 @@
     LocalDPDKTreeLocation,
     RemoteDPDKTarballLocation,
     RemoteDPDKTreeLocation,
-    TestSuiteConfig,
+    TestSuitesConfigs,
 )
 
 
@@ -133,7 +133,7 @@ class Settings:
     #:
     compile_timeout: float = 1200
     #:
-    test_suites: list[TestSuiteConfig] = field(default_factory=list)
+    test_suites: list[tuple[str, list[str]]] = field(default_factory=list)
     #:
     re_run: int = 0
     #:
@@ -508,7 +508,7 @@ def _process_dpdk_location(
 
 def _process_test_suites(
     parser: _DTSArgumentParser, args: list[list[str]]
-) -> list[TestSuiteConfig]:
+) -> list[tuple[str, list[str]]]:
     """Process the given argument to a list of :class:`TestSuiteConfig` to execute.
 
     Args:
@@ -524,19 +524,17 @@ def _process_test_suites(
         # Environment variable in the form of "SUITE1 CASE1 CASE2, SUITE2 CASE1, SUITE3, ..."
         args = [suite_with_cases.split() for suite_with_cases in args[0][0].split(",")]
 
-    try:
-        return [
-            TestSuiteConfig(test_suite=test_suite, test_cases=test_cases)
-            for [test_suite, *test_cases] in args
-        ]
-    except ValidationError as e:
-        print(
-            "An error has occurred while validating the test suites supplied in the "
-            f"{'environment variable' if action else 'arguments'}:",
-            file=sys.stderr,
-        )
-        print(e, file=sys.stderr)
-        sys.exit(1)
+    available_test_suites = TestSuitesConfigs.available_test_suites()
+    for test_suite_name, *_ in args:
+        if test_suite_name not in available_test_suites:
+            print(
+                f"The test suite {test_suite_name} supplied in the "
+                f"{'environment variable' if action else 'arguments'} is invalid.",
+                file=sys.stderr,
+            )
+            sys.exit(1)
+
+    return [(test_suite, test_cases) for test_suite, *test_cases in args]
 
 
 def get_settings() -> Settings:
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 6014d281b5..8c0c1bcfb3 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -30,7 +30,8 @@
 
 from framework.testbed_model.capability import Capability
 
-from .config import TestRunConfiguration, TestSuiteConfig
+from .config import TestRunConfiguration
+from .config.test_suite import TestSuiteConfig
 from .exception import DTSError, ErrorSeverity
 from .logger import DTSLogger
 from .settings import SETTINGS
@@ -59,23 +60,13 @@ class is to hold a subset of test cases (which could be all test cases) because
     test_suite_class: type[TestSuite]
     test_cases: list[type[TestCase]]
     required_capabilities: set[Capability] = field(default_factory=set, init=False)
+    config: TestSuiteConfig
 
     def __post_init__(self):
         """Gather the required capabilities of the test suite and all test cases."""
         for test_object in [self.test_suite_class] + self.test_cases:
             self.required_capabilities.update(test_object.required_capabilities)
 
-    def create_config(self) -> TestSuiteConfig:
-        """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
-
-        Returns:
-            The :class:`TestSuiteConfig` representation.
-        """
-        return TestSuiteConfig(
-            test_suite=self.test_suite_class.__name__,
-            test_cases=[test_case.__name__ for test_case in self.test_cases],
-        )
-
     def mark_skip_unsupported(self, supported_capabilities: set[Capability]) -> None:
         """Mark the test suite and test cases to be skipped.
 
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index fb5d646ce3..24cd0d38c1 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -24,13 +24,14 @@
 from ipaddress import IPv4Interface, IPv6Interface, ip_interface
 from pkgutil import iter_modules
 from types import ModuleType
-from typing import ClassVar, Protocol, TypeVar, Union, cast
+from typing import ClassVar, Protocol, TypeVar, Union, cast, get_type_hints
 
 from scapy.layers.inet import IP  # type: ignore[import-untyped]
 from scapy.layers.l2 import Ether  # type: ignore[import-untyped]
 from scapy.packet import Packet, Padding, raw  # type: ignore[import-untyped]
 from typing_extensions import Self
 
+from framework.config.test_suite import TestSuiteConfig
 from framework.testbed_model.capability import TestProtocol
 from framework.testbed_model.port import Port
 from framework.testbed_model.sut_node import SutNode
@@ -80,6 +81,7 @@ class TestSuite(TestProtocol):
     #: Whether the test suite is blocking. A failure of a blocking test suite
     #: will block the execution of all subsequent test suites in the current test run.
     is_blocking: ClassVar[bool] = False
+    config: TestSuiteConfig
     _logger: DTSLogger
     _sut_port_ingress: Port
     _sut_port_egress: Port
@@ -95,6 +97,7 @@ def __init__(
         sut_node: SutNode,
         tg_node: TGNode,
         topology: Topology,
+        config: TestSuiteConfig,
     ):
         """Initialize the test suite testbed information and basic configuration.
 
@@ -105,9 +108,11 @@ def __init__(
             sut_node: The SUT node where the test suite will run.
             tg_node: The TG node where the test suite will run.
             topology: The topology where the test suite will run.
+            config: The test suite configuration.
         """
         self.sut_node = sut_node
         self.tg_node = tg_node
+        self.config = config
         self._logger = get_dts_logger(self.__class__.__name__)
         self._tg_port_egress = topology.tg_port_egress
         self._sut_port_ingress = topology.sut_port_ingress
@@ -663,6 +668,21 @@ def is_test_suite(obj) -> bool:
             f"Expected class {self.class_name} not found in module {self.module_name}."
         )
 
+    @cached_property
+    def config_obj(self) -> type[TestSuiteConfig]:
+        """A reference to the test suite's configuration type."""
+        fields = get_type_hints(self.class_obj)
+        config_obj = fields.get("config")
+        if config_obj is None:
+            raise InternalError(
+                "Test suite class {self.class_name} is missing the `config` attribute."
+            )
+        if not issubclass(config_obj, TestSuiteConfig):
+            raise InternalError(
+                f"Test suite class {self.class_name} has an invalid configuration type assigned."
+            )
+        return config_obj
+
     @classmethod
     def discover_all(
         cls, package_name: str | None = None, module_prefix: str | None = None
diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
index 734f006026..f2998b968e 100644
--- a/dts/tests/TestSuite_hello_world.py
+++ b/dts/tests/TestSuite_hello_world.py
@@ -15,12 +15,15 @@
     LogicalCoreCountFilter,
     LogicalCoreList,
 )
+from tests.config import HelloWorldConfig
 
 
 @requires(topology_type=TopologyType.no_link)
 class TestHelloWorld(TestSuite):
     """DPDK hello world app test suite."""
 
+    config: HelloWorldConfig
+
     def set_up_suite(self) -> None:
         """Set up the test suite.
 
@@ -63,7 +66,7 @@ def hello_world_all_cores(self) -> None:
         eal_para = compute_eal_params(
             self.sut_node, lcore_filter_specifier=LogicalCoreList(self.sut_node.lcores)
         )
-        result = self.sut_node.run_dpdk_app(self.app_helloworld_path, eal_para, 50)
+        result = self.sut_node.run_dpdk_app(self.app_helloworld_path, eal_para, self.config.timeout)
         for lcore in self.sut_node.lcores:
             self.verify(
                 f"hello from core {int(lcore)}" in result.stdout,
diff --git a/dts/tests/__init__.py b/dts/tests/__init__.py
new file mode 100644
index 0000000000..a300eb26fc
--- /dev/null
+++ b/dts/tests/__init__.py
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Test suites.
+
+This package contains all the available test suites in DTS.
+"""
diff --git a/dts/tests/config.py b/dts/tests/config.py
new file mode 100644
index 0000000000..300ad3ef6a
--- /dev/null
+++ b/dts/tests/config.py
@@ -0,0 +1,20 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Module for test suites custom configurations.
+
+Any test suite that requires custom configuration fields should create a new config class inheriting
+:class:`~framework.config.test_suite.TestSuiteConfig`, while respecting the parents' frozen state.
+Any custom fields can be added in this class.
+
+The custom configuration classes can be stored in this module.
+"""
+
+from framework.config.test_suite import TestSuiteConfig
+
+
+class HelloWorldConfig(TestSuiteConfig):
+    """Example custom configuration for the `TestHelloWorld` test suite."""
+
+    #: Timeout for the DPDK apps.
+    timeout: int = 50
-- 
2.43.0


^ permalink raw reply	[flat|nested] 17+ messages in thread

* [PATCH v3 4/4] dts: update autodoc sorting order
  2024-11-08 13:45 ` [PATCH v3 0/4] dts: add per-test-suite configuration Luca Vizzarro
                     ` (2 preceding siblings ...)
  2024-11-08 13:45   ` [PATCH v3 3/4] dts: add per-test-suite configuration Luca Vizzarro
@ 2024-11-08 13:45   ` Luca Vizzarro
  2024-11-12 20:04     ` Dean Marx
  3 siblings, 1 reply; 17+ messages in thread
From: Luca Vizzarro @ 2024-11-08 13:45 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Patrick Robb, Luca Vizzarro

The autodoc member sorting order default is set to alphabetical, which
translates to autodoc sorting every member in modules, classes etc.
This also brings some side effects, like sorting capabilities which
can't be compared and result in errors.

This change prevents autodoc from sorting, and keeping the order as the
developer intended it.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 doc/guides/conf.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/guides/conf.py b/doc/guides/conf.py
index 71fed45b3d..ed5bc5eb30 100644
--- a/doc/guides/conf.py
+++ b/doc/guides/conf.py
@@ -88,6 +88,7 @@
     autodoc_typehints = 'both'
     autodoc_typehints_format = 'short'
     autodoc_typehints_description_target = 'documented'
+    autodoc_member_order = 'bysource'
 
     # Intersphinx allows linking to external projects, such as Python docs.
     intersphinx_mapping = {'python': ('https://docs.python.org/3', None)}
-- 
2.43.0


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH v3 1/4] dts: add tests package to API docs
  2024-11-08 13:45   ` [PATCH v3 1/4] dts: add tests package to API docs Luca Vizzarro
@ 2024-11-12 19:55     ` Dean Marx
  0 siblings, 0 replies; 17+ messages in thread
From: Dean Marx @ 2024-11-12 19:55 UTC (permalink / raw)
  To: Luca Vizzarro; +Cc: dev, Paul Szczepanek, Patrick Robb

[-- Attachment #1: Type: text/plain, Size: 43 bytes --]

Reviewed-by: Dean Marx <dmarx@iol.unh.edu>

[-- Attachment #2: Type: text/html, Size: 109 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH v2 2/4] dts: fix smoke tests docstring
  2024-11-08 13:38   ` [PATCH v2 2/4] dts: fix smoke tests docstring Luca Vizzarro
@ 2024-11-12 20:03     ` Dean Marx
  0 siblings, 0 replies; 17+ messages in thread
From: Dean Marx @ 2024-11-12 20:03 UTC (permalink / raw)
  To: Luca Vizzarro; +Cc: dev, Paul Szczepanek, Patrick Robb

[-- Attachment #1: Type: text/plain, Size: 43 bytes --]

Reviewed-by: Dean Marx <dmarx@iol.unh.edu>

[-- Attachment #2: Type: text/html, Size: 109 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH v3 4/4] dts: update autodoc sorting order
  2024-11-08 13:45   ` [PATCH v3 4/4] dts: update autodoc sorting order Luca Vizzarro
@ 2024-11-12 20:04     ` Dean Marx
  0 siblings, 0 replies; 17+ messages in thread
From: Dean Marx @ 2024-11-12 20:04 UTC (permalink / raw)
  To: Luca Vizzarro; +Cc: dev, Paul Szczepanek, Patrick Robb

[-- Attachment #1: Type: text/plain, Size: 43 bytes --]

Reviewed-by: Dean Marx <dmarx@iol.unh.edu>

[-- Attachment #2: Type: text/html, Size: 109 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH v3 3/4] dts: add per-test-suite configuration
  2024-11-08 13:45   ` [PATCH v3 3/4] dts: add per-test-suite configuration Luca Vizzarro
@ 2024-11-13 21:44     ` Dean Marx
  2024-11-14 12:39       ` Luca Vizzarro
  0 siblings, 1 reply; 17+ messages in thread
From: Dean Marx @ 2024-11-13 21:44 UTC (permalink / raw)
  To: Luca Vizzarro; +Cc: dev, Paul Szczepanek, Patrick Robb

[-- Attachment #1: Type: text/plain, Size: 2735 bytes --]

>
>
> On Fri, Nov 8, 2024 at 8:38 AM Luca Vizzarro <luca.vizzarro@arm.com>
> wrote:
>
>> Allow test suites to be configured individually. Moreover enable them to
>> implement their own custom configuration.
>>
>> This solution adds some new complexity to DTS, which is generated source
>> code. In order to ensure strong typing, the test suites and their custom
>> configurations need to be linked in the main configuration class.
>> Unfortunately, this is not feasible during runtime as it will incur in
>> circular dependencies. Generating the links appear to be the most
>> straightforward approach.
>>
>> This commit also brings a new major change to the configuration schema.
>> Test suites are no longer defined as a list of strings, like:
>>
>>     test_suites:
>>     - hello_world
>>     - pmd_buffer_scatter
>>
>> but as mapping of mappings or strings:
>>
>>     test_suites:
>>       hello_world: {} # any custom fields or test cases can be set here
>>       pmd_buffer_scatter: all # "all" defines all the test cases, or
>>                               # they can individually be set separated
>>                               # by a space
>>
>> Not defining the `test_cases` field in the configuration is equivalent
>> to `all`, therefore the definitions for either test suite above are
>> also equivalent.
>>
>> Creating the __init__.py file under the tests folder, allows it to be
>> picked up as a package. This is a mypy requirement to import the tests
>> from within the framework.
>>
>> Bugzilla ID: 1375
>>
>> Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
>> Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
>>
>
> I like the idea of mapping the suite to specific test cases, and for the
> most part the custom configuration option as well. The only thing that I
> feel should be different is the way the code generation is documented, I
> think it might be worth providing an example within conf.yaml through a
> comment near the suites section, rather than just in the dts.rst file. It
> might be a little more clear where to create the custom config class as
> well.
>
>  <snip>
>
>> +class HelloWorldConfig(TestSuiteConfig):
>> +    """Example custom configuration for the `TestHelloWorld` test
>> suite."""
>> +
>> +    #: Timeout for the DPDK apps.
>> +    timeout: int = 50
>> --
>> 2.43.0
>>
>>
> Additionally, I was a bit confused by the custom config examples, do these
> fields (timeout, my_custom_field) actually affect the suite in any way as
> of this patch? Or is this just so that we can potentially add configuration
> options through this method in the future?
>
> Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
>

[-- Attachment #2: Type: text/html, Size: 3577 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH v3 3/4] dts: add per-test-suite configuration
  2024-11-13 21:44     ` Dean Marx
@ 2024-11-14 12:39       ` Luca Vizzarro
  0 siblings, 0 replies; 17+ messages in thread
From: Luca Vizzarro @ 2024-11-14 12:39 UTC (permalink / raw)
  To: Dean Marx; +Cc: dev, Paul Szczepanek, Patrick Robb

Hi Dean!

Thank you for your review!

On 13/11/2024 21:44, Dean Marx wrote:
> I like the idea of mapping the suite to specific test cases, and for
> the most part the custom configuration option as well. The only
> thing that I feel should be different is the way the code generation
> is documented, I think it might be worth providing an example within
> conf.yaml through a comment near the suites section, rather than
> just in the dts.rst file. It might be a little more clear where to
> create the custom config class as well.

So... this is a weird one. I've integrated the code generation under 
dts-check-format.sh, so nobody should really worry about it, as that 
should be run before submitting. If it's not, then the CI will 
eventually pick it up when dts-check-format will be in place.

>> +class HelloWorldConfig(TestSuiteConfig):
>> +    """Example custom configuration for the `TestHelloWorld`
>> test suite."""
>> +
>> +    #: Timeout for the DPDK apps.
>> +    timeout: int = 50
>> --
>> 2.43.0
> 
> Additionally, I was a bit confused by the custom config examples, do
> these fields (timeout, my_custom_field) actually affect the suite in
> any way as of this patch? Or is this just so that we can potentially
> add configuration options through this method in the future?

The hello_world suite doesn't really need this, I've just added it as an 
example. But yes test suites are affected. If you see the hello_world 
suite, it calls `self.config.timeout` which is a reference to this 
field. This class specifically is now wholly integrated in the 
configuration:

   test_runs:
   - test_suites:
       hello_world:  # this is a mapping to HelloWorldConfig
         # like before the behavior of this field is:
	# - if unset or empty list, all test cases are run
         # - if individual test cases are listed, those are run
         test_cases: [single_core] # field inherited from TestSuiteConfig
         timeout: 20               # optional field from HelloWorldConfig

What I've added for the simplicity is to be able to represent any 
TestSuiteConfig as a string instead of a mapping:

   test_runs:
   - test_suites:
       hello_world: all # special keyword which enables all test cases
       # or
       hello_world: single_core all_cores # a whitespace-delimited list 
of test suites

Mind that if the underlying test suite configuration has a mandatory 
field that has no default, then the string representation will fail and 
Pydantic will complain that the (required) field is missing.

Please let me know if I answered every query, and don't hesitate to ask 
for further clarifications.

Best,
Luca

^ permalink raw reply	[flat|nested] 17+ messages in thread

end of thread, other threads:[~2024-11-14 12:39 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-09-06 16:13 [PATCH] dts: add per-test-suite configuration Luca Vizzarro
2024-09-27 17:45 ` Jeremy Spewock
2024-11-08 13:38 ` [PATCH v2 0/4] " Luca Vizzarro
2024-11-08 13:38   ` [PATCH v2 1/4] dts: add tests package to API docs Luca Vizzarro
2024-11-08 13:38   ` [PATCH v2 2/4] dts: fix smoke tests docstring Luca Vizzarro
2024-11-12 20:03     ` Dean Marx
2024-11-08 13:38   ` [PATCH v2 3/4] dts: add per-test-suite configuration Luca Vizzarro
2024-11-08 13:38   ` [PATCH v2 4/4] dts: update autodoc sorting order Luca Vizzarro
2024-11-08 13:45 ` [PATCH v3 0/4] dts: add per-test-suite configuration Luca Vizzarro
2024-11-08 13:45   ` [PATCH v3 1/4] dts: add tests package to API docs Luca Vizzarro
2024-11-12 19:55     ` Dean Marx
2024-11-08 13:45   ` [PATCH v3 2/4] dts: fix smoke tests docstring Luca Vizzarro
2024-11-08 13:45   ` [PATCH v3 3/4] dts: add per-test-suite configuration Luca Vizzarro
2024-11-13 21:44     ` Dean Marx
2024-11-14 12:39       ` Luca Vizzarro
2024-11-08 13:45   ` [PATCH v3 4/4] dts: update autodoc sorting order Luca Vizzarro
2024-11-12 20:04     ` Dean Marx

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).