[PATCH] dts: add per-test-suite configuration

DPDK patches and discussions
 help / color / mirror / Atom feed

* [PATCH] dts: add per-test-suite configuration
@ 2024-09-06 16:13 Luca Vizzarro
  2024-09-27 17:45 ` Jeremy Spewock
                   ` (2 more replies)
  0 siblings, 3 replies; 27+ messages in thread
From: Luca Vizzarro @ 2024-09-06 16:13 UTC (permalink / raw)
  To: dev
  Cc: Honnappa Nagarahalli, Luca Vizzarro, Paul Szczepanek,
	Alex Chapman, Juraj Linkeš

Allow test suites to be configured individually. Moreover enable them to
implement their own custom configuration.

This solution adds some new complexity to DTS, which is generated source
code. In order to ensure strong typing, the test suites and their custom
configurations need to be linked in the main configuration class.
Unfortunately, this is not feasible during runtime as it will incur in
circular dependencies. Generating the links appear to be the most
straightforward approach.

This commit also brings a new major change to the configuration schema.
Test suites are no longer defined as a list of strings, like:

    test_suites:
    - hello_world
    - pmd_buffer_scatter

but as mapping of mappings or strings:

    test_suites:
      hello_world: {} # any custom fields or test cases can be set here
      pmd_buffer_scatter: all # "all" defines all the test cases, or
                              # they can individually be set separated
                              # by a space

Not defining the `test_cases` field in the configuration is equivalent
to `all`, therefore the definitions for either test suite above are
also equivalent.

Creating the __init__.py file under the tests folder, allows it to be
picked up as a package. This is a mypy requirement to import the tests
from within the framework.

Bugzilla ID: 1375

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Alex Chapman <alex.chapman@arm.com>
---
Depends-on: series-32823 ("dts: Pydantic configuration")

Hello,

sending in a solution for the per-test-suite configuration issue.
This one took some thinking but I have given most of the motivations in
the commit body already. The docs are somewhat lacking but hopefully they
should be automatically tackled by the API docs generation.


Best,
Luca
---
 doc/guides/tools/dts.rst                   |  39 ++++--
 dts/conf.yaml                              |   4 +-
 dts/framework/config/__init__.py           |  98 ++-------------
 dts/framework/config/conf_yaml_schema.json |  94 +++++++++++---
 dts/framework/config/generated.py          |  40 ++++++
 dts/framework/config/test_suite.py         | 140 +++++++++++++++++++++
 dts/framework/runner.py                    |  59 +++++++--
 dts/framework/settings.py                  |  29 +++--
 dts/framework/test_result.py               |  12 +-
 dts/framework/test_suite.py                |  32 ++++-
 dts/generate-schema.py                     |   4 +-
 dts/generate-test-mappings.py              | 132 +++++++++++++++++++
 dts/tests/TestSuite_hello_world.py         |  12 +-
 dts/tests/__init__.py                      |   7 ++
 14 files changed, 539 insertions(+), 163 deletions(-)
 create mode 100644 dts/framework/config/generated.py
 create mode 100644 dts/framework/config/test_suite.py
 create mode 100755 dts/generate-test-mappings.py
 create mode 100644 dts/tests/__init__.py

diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index 317bd0ff99..66681543cd 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -391,6 +391,26 @@ There are four types of methods that comprise a test suite:
    should be implemented in the ``SutNode`` class (and the underlying classes that ``SutNode`` uses)
    and used by the test suite via the ``sut_node`` field.
 
+The test suites can also implement their own custom configuration fields. This can be achieved by
+creating a new test suite config file which inherits from ``TestSuiteConfig`` defined in
+``dts/framework/config/test_suite.py``. So that this new custom configuration class is used, the
+test suite class must override the ``config`` attribute annotation with your new class, for example::
+
+.. code:: python
+   class CustomConfig(TestSuiteConfig):
+      my_custom_field: int = 10
+
+   class TestMyNewTestSuite(TestSuite):
+      config: CustomConfig
+
+Finally, the test suites and the custom configuration files need to linked in the global configuration.
+This can be easily achieved by running the ``dts/generate-test-mappings.py``, e.g.:
+
+.. code-block:: console
+
+   $ poetry shell
+   (dts-py3.10) $ ./generate-test-mappings.py
+
 
 .. _dts_dev_tools:
 
@@ -510,18 +530,13 @@ _`Network port`
    ``peer_pci``           *string* – the PCI address of the peer node port. **Example**: ``000a:01:00.1``
    ====================== =================================================================================
 
-_`Test suite`
-   *string* – name of the test suite to run. **Examples**: ``hello_world``, ``os_udp``
-
-_`Test target`
-   *mapping* – selects specific test cases to run from a test suite. Mapping is described as follows:
-
-   ========= ===============================================================================================
-   ``suite`` See `Test suite`_
-   ``cases`` (*optional*) *sequence* of *string* – list of the selected test cases in the test suite to run.
+_`Test suites`
+   *mapping* – selects the test suites to run. Each mapping key corresponds to the test suite name.
 
-             Unknown test cases will be silently ignored.
-   ========= ===============================================================================================
+   The value of the mapping can either "all" to select all the test cases in that test suite, the test
+   cases names divided by a space. Or it can be another mapping to set any custom fields for the test suite.
+   In the case of a mapping, all the test cases are selected by default. In order to manually select test
+   cases, the ``test_cases`` field can be set with a list of strings, each entry being a test case name.
 
 
 Properties
@@ -542,7 +557,7 @@ involved in the testing. These can be defined with the following mappings:
    +----------------------------+-------------------------------------------------------------------+
    | ``func``                   | *boolean* – Enable functional testing.                            |
    +----------------------------+-------------------------------------------------------------------+
-   | ``test_suites``            | *sequence* of **one of** `Test suite`_ **or** `Test target`_      |
+   | ``test_suites``            | See `Test suites`_                                                |
    +----------------------------+-------------------------------------------------------------------+
    | ``skip_smoke_tests``       | (*optional*) *boolean* – Allows you to skip smoke testing         |
    |                            | if ``true``.                                                      |
diff --git a/dts/conf.yaml b/dts/conf.yaml
index 7d95016e68..c44bef604c 100644
--- a/dts/conf.yaml
+++ b/dts/conf.yaml
@@ -15,8 +15,8 @@ test_runs:
     func: true # enable functional testing
     skip_smoke_tests: false # optional
     test_suites: # the following test suites will be run in their entirety
-      - hello_world
-      - os_udp
+      hello_world: all
+      os_udp: all
     # The machine running the DPDK test executable
     system_under_test_node:
       node_name: "SUT 1"
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 013c529829..d39e0823bd 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -35,10 +35,12 @@
       and makes it thread safe should we ever want to move in that direction.
 """
 
+# pylama:ignore=W0611
+
 from enum import Enum, auto, unique
 from functools import cached_property
 from pathlib import Path
-from typing import TYPE_CHECKING, Annotated, Any, Literal, NamedTuple, Protocol
+from typing import Annotated, Literal, NamedTuple, Protocol
 
 import yaml
 from pydantic import (
@@ -50,15 +52,14 @@
     field_validator,
     model_validator,
 )
-from pydantic.config import JsonDict
 from pydantic.dataclasses import dataclass
 from typing_extensions import Self
 
 from framework.exception import ConfigurationError
 from framework.utils import StrEnum
 
-if TYPE_CHECKING:
-    from framework.test_suite import TestSuiteSpec
+from .generated import CUSTOM_CONFIG_TYPES, TestSuitesConfigs
+from .test_suite import TestSuiteConfig
 
 
 @unique
@@ -289,7 +290,7 @@ class NodeInfo:
     kernel_version: str
 
 
-@dataclass(slots=True, frozen=True, kw_only=True, config=ConfigDict(extra="forbid"))
+@dataclass(frozen=True, kw_only=True, config=ConfigDict(extra="forbid"))
 class BuildTargetConfiguration:
     """DPDK build configuration.
 
@@ -329,89 +330,6 @@ class BuildTargetInfo:
     compiler_version: str
 
 
-def make_parsable_schema(schema: JsonDict):
-    """Updates a model's JSON schema to make a string representation a valid alternative.
-
-    This utility function is required to be used with models that can be represented and validated
-    as a string instead of an object mapping. Normally the generated JSON schema will just show
-    the object mapping. This function wraps the mapping under an anyOf property sequenced with a
-    string type.
-
-    This function is a valid `Callable` for the `json_schema_extra` attribute of
-    `~pydantic.config.ConfigDict`.
-    """
-    inner_schema = schema.copy()
-    del inner_schema["title"]
-
-    title = schema.get("title")
-    description = schema.get("description")
-
-    schema.clear()
-
-    schema["title"] = title
-    schema["description"] = description
-    schema["anyOf"] = [inner_schema, {"type": "string"}]
-
-
-@dataclass(
-    frozen=True,
-    config=ConfigDict(extra="forbid", json_schema_extra=make_parsable_schema),
-)
-class TestSuiteConfig:
-    """Test suite configuration.
-
-    Information about a single test suite to be executed. It can be represented and validated as a
-    string type in the form of: ``TEST_SUITE [TEST_CASE, ...]``, in the configuration file.
-
-    Attributes:
-        test_suite: The name of the test suite module without the starting ``TestSuite_``.
-        test_cases: The names of test cases from this test suite to execute.
-            If empty, all test cases will be executed.
-    """
-
-    test_suite_name: str = Field(
-        title="Test suite name",
-        description="The identifying name of the test suite.",
-        alias="test_suite",
-    )
-    test_cases_names: list[str] = Field(
-        default_factory=list,
-        title="Test cases by name",
-        description="The identifying name of the test cases of the test suite.",
-        alias="test_cases",
-    )
-
-    @cached_property
-    def test_suite_spec(self) -> "TestSuiteSpec":
-        """The specification of the requested test suite."""
-        from framework.test_suite import find_by_name
-
-        test_suite_spec = find_by_name(self.test_suite_name)
-        assert test_suite_spec is not None, f"{self.test_suite_name} is not a valid test suite name"
-        return test_suite_spec
-
-    @model_validator(mode="before")
-    @classmethod
-    def convert_from_string(cls, data: Any) -> Any:
-        """Convert the string representation into a valid mapping."""
-        if isinstance(data, str):
-            [test_suite, *test_cases] = data.split()
-            return dict(test_suite=test_suite, test_cases=test_cases)
-        return data
-
-    @model_validator(mode="after")
-    def validate_names(self) -> Self:
-        """Validate the supplied test suite and test cases names."""
-        available_test_cases = map(lambda t: t.name, self.test_suite_spec.test_cases)
-        for requested_test_case in self.test_cases_names:
-            assert requested_test_case in available_test_cases, (
-                f"{requested_test_case} is not a valid test case "
-                f"for test suite {self.test_suite_name}"
-            )
-
-        return self
-
-
 @dataclass(slots=True, frozen=True, kw_only=True, config=ConfigDict(extra="forbid"))
 class TestRunSUTNodeConfiguration:
     """The SUT node configuration of a test run.
@@ -446,7 +364,7 @@ class TestRunConfiguration:
     perf: bool = Field(description="Enable performance testing.")
     func: bool = Field(description="Enable functional testing.")
     skip_smoke_tests: bool = False
-    test_suites: list[TestSuiteConfig] = Field(min_length=1)
+    test_suites: TestSuitesConfigs
     system_under_test_node: TestRunSUTNodeConfiguration
     traffic_generator_node: str
 
@@ -581,7 +499,7 @@ def load_config(config_file_path: Path) -> Configuration:
         config_data = yaml.safe_load(f)
 
     try:
-        ConfigurationType.json_schema()
+        TestSuitesConfigs.fix_custom_config_annotations()
         return ConfigurationType.validate_python(config_data)
     except ValidationError as e:
         raise ConfigurationError("failed to load the supplied configuration") from e
diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index 1cf1bb098a..1434cdfad3 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -66,6 +66,33 @@
       "title": "Compiler",
       "type": "string"
     },
+    "HelloWorldConfig": {
+      "anyOf": [
+        {
+          "additionalProperties": false,
+          "properties": {
+            "test_cases": {
+              "items": {
+                "type": "string"
+              },
+              "title": "Test Cases",
+              "type": "array"
+            },
+            "timeout": {
+              "default": 50,
+              "title": "Timeout",
+              "type": "integer"
+            }
+          },
+          "type": "object"
+        },
+        {
+          "type": "string"
+        }
+      ],
+      "description": "Example custom configuration for the `TestHelloWorld` test suite.",
+      "title": "HelloWorldConfig"
+    },
     "HugepageConfiguration": {
       "additionalProperties": false,
       "description": "The hugepage configuration of :class:`~framework.testbed_model.node.Node`\\s.\n\nAttributes:\n    number_of: The number of hugepages to allocate.\n    force_first_numa: If :data:`True`, the hugepages will be configured on the first NUMA node.",
@@ -373,14 +400,6 @@
           "title": "Func",
           "type": "boolean"
         },
-        "test_suites": {
-          "items": {
-            "$ref": "#/$defs/TestSuiteConfig"
-          },
-          "minItems": 1,
-          "title": "Test Suites",
-          "type": "array"
-        },
         "build_targets": {
           "items": {
             "$ref": "#/$defs/BuildTargetConfiguration"
@@ -393,6 +412,9 @@
           "title": "Skip Smoke Tests",
           "type": "boolean"
         },
+        "test_suites": {
+          "$ref": "#/$defs/TestSuitesConfigs"
+        },
         "system_under_test_node": {
           "$ref": "#/$defs/TestRunSUTNodeConfiguration"
         },
@@ -404,8 +426,8 @@
       "required": [
         "perf",
         "func",
-        "test_suites",
         "build_targets",
+        "test_suites",
         "system_under_test_node",
         "traffic_generator_node"
       ],
@@ -439,31 +461,63 @@
         {
           "additionalProperties": false,
           "properties": {
-            "test_suite": {
-              "description": "The identifying name of the test suite.",
-              "title": "Test suite name",
-              "type": "string"
-            },
             "test_cases": {
-              "description": "The identifying name of the test cases of the test suite.",
               "items": {
                 "type": "string"
               },
-              "title": "Test cases by name",
+              "title": "Test Cases",
               "type": "array"
             }
           },
-          "required": [
-            "test_suite"
-          ],
           "type": "object"
         },
         {
           "type": "string"
         }
       ],
-      "description": "Test suite configuration.\n\nInformation about a single test suite to be executed. It can be represented and validated as a\nstring type in the form of: ``TEST_SUITE [TEST_CASE, ...]``, in the configuration file.\n\nAttributes:\n    test_suite: The name of the test suite module without the starting ``TestSuite_``.\n    test_cases: The names of test cases from this test suite to execute.\n        If empty, all test cases will be executed.",
+      "description": "Test suite configuration base model.\n\nBy default the configuration of a generic test suite does not contain any attributes. Any test\nsuite should inherit this class to create their own custom configuration. Finally override the\ntype of the :attr:`~TestSuite.config` to use the newly created one.\n\nAttributes:\n    test_cases_names: The names of test cases from this test suite to execute. If empty, all\n        test cases will be executed.",
       "title": "TestSuiteConfig"
+    },
+    "TestSuitesConfigs": {
+      "additionalProperties": false,
+      "description": "Configuration mapping class to select and configure the test suites.\n\nBefore using this class, the custom configuration type annotations need to be fixed.\nTo do so, you need to call the `fix_custom_config_annotations` method.",
+      "properties": {
+        "hello_world": {
+          "anyOf": [
+            {
+              "$ref": "#/$defs/HelloWorldConfig"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null
+        },
+        "os_udp": {
+          "anyOf": [
+            {
+              "$ref": "#/$defs/TestSuiteConfig"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null
+        },
+        "pmd_buffer_scatter": {
+          "anyOf": [
+            {
+              "$ref": "#/$defs/TestSuiteConfig"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null
+        }
+      },
+      "title": "TestSuitesConfigs",
+      "type": "object"
     }
   },
   "description": "DTS testbed and test configuration.\n\nAttributes:\n    test_runs: Test run configurations.\n    nodes: Node configurations.",
diff --git a/dts/framework/config/generated.py b/dts/framework/config/generated.py
new file mode 100644
index 0000000000..d42ce93a51
--- /dev/null
+++ b/dts/framework/config/generated.py
@@ -0,0 +1,40 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 The DPDK contributors
+# This file is automatically generated by generate-test-mappings.py.
+# Do NOT modify this file manually.
+
+"""Generated file containing the links between the test suites and the configuration."""
+
+from typing import TYPE_CHECKING, Optional
+
+from framework.config.test_suite import BaseTestSuitesConfigs, TestSuiteConfig
+
+if TYPE_CHECKING:
+    from tests.TestSuite_hello_world import HelloWorldConfig
+
+
+CUSTOM_CONFIG_TYPES: dict[str, type[TestSuiteConfig]] = {}
+
+
+class TestSuitesConfigs(BaseTestSuitesConfigs):
+    """Configuration mapping class to select and configure the test suites.
+
+    Before using this class, the custom configuration type annotations need to be fixed.
+    To do so, you need to call the `fix_custom_config_annotations` method.
+    """
+
+    hello_world: Optional["HelloWorldConfig"] = None
+    os_udp: Optional[TestSuiteConfig] = None
+    pmd_buffer_scatter: Optional[TestSuiteConfig] = None
+
+    @classmethod
+    def fix_custom_config_annotations(cls):
+        """Fixes the custom config types annotations.
+
+        Moreover it also fills `CUSTOM_CONFIG_TYPES` with all the custom config types.
+        """
+        from tests.TestSuite_hello_world import HelloWorldConfig
+
+        CUSTOM_CONFIG_TYPES["hello_world"] = HelloWorldConfig
+
+        cls.model_rebuild()
diff --git a/dts/framework/config/test_suite.py b/dts/framework/config/test_suite.py
new file mode 100644
index 0000000000..0c1ddf9d95
--- /dev/null
+++ b/dts/framework/config/test_suite.py
@@ -0,0 +1,140 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Test suites configuration module.
+
+Test suites can inherit :class:`TestSuiteConfig` to create their own custom configuration.
+By doing so, the test suite class must also override the annotation of the field
+`~framework.test_suite.TestSuite.config` to use their custom configuration type.
+"""
+
+from typing import TYPE_CHECKING, Any, Iterable
+
+from pydantic import BaseModel, Field, ValidationInfo, field_validator, model_validator
+from pydantic.config import JsonDict
+from typing_extensions import Self
+
+if TYPE_CHECKING:
+    from framework.test_suite import TestSuiteSpec
+
+
+def make_parsable_schema(schema: JsonDict):
+    """Updates a model's JSON schema to make a string representation a valid alternative.
+
+    This utility function is required to be used with models that can be represented and validated
+    as a string instead of an object mapping. Normally the generated JSON schema will just show
+    the object mapping. This function wraps the mapping under an anyOf property sequenced with a
+    string type.
+
+    This function is a valid `Callable` for the `json_schema_extra` attribute of
+    `~pydantic.config.ConfigDict`.
+    """
+    inner_schema = schema.copy()
+
+    fields_to_preserve = ["title", "description"]
+    extracted_fields = {k: v for k in fields_to_preserve if (v := inner_schema.get(k))}
+    for field in extracted_fields:
+        del inner_schema[field]
+
+    schema.clear()
+    schema.update(extracted_fields)
+    schema["anyOf"] = [inner_schema, {"type": "string"}]
+
+
+class TestSuiteConfig(BaseModel, extra="forbid", json_schema_extra=make_parsable_schema):
+    """Test suite configuration base model.
+
+    By default the configuration of a generic test suite does not contain any attributes. Any test
+    suite should inherit this class to create their own custom configuration. Finally override the
+    type of the :attr:`~TestSuite.config` to use the newly created one.
+
+    Attributes:
+        test_cases_names: The names of test cases from this test suite to execute. If empty, all
+            test cases will be executed.
+    """
+
+    _test_suite_spec: "TestSuiteSpec"
+
+    test_cases_names: list[str] = Field(default_factory=list, alias="test_cases")
+
+    @property
+    def test_suite_name(self) -> str:
+        """The name of the test suite module without the starting ``TestSuite_``."""
+        return self._test_suite_spec.name
+
+    @property
+    def test_suite_spec(self) -> "TestSuiteSpec":
+        """The specification of the requested test suite."""
+        return self._test_suite_spec
+
+    @model_validator(mode="before")
+    @classmethod
+    def convert_from_string(cls, data: Any) -> Any:
+        """Validator which allows to select a test suite by string instead of a mapping."""
+        if isinstance(data, str):
+            test_cases = [] if data == "all" else data.split()
+            return dict(test_cases=test_cases)
+        return data
+
+    @classmethod
+    def make(cls, test_suite_name: str, *test_cases_names: str, **kwargs) -> Self:
+        """Make a configuration for the requested test suite.
+
+        Args:
+            test_suite_name: The name of the test suite.
+            test_cases_names: The test cases to select, if empty all are selected.
+            **kwargs: Any other configuration field.
+
+        Raises:
+            AssertionError: If the requested test suite or test cases do not exist.
+            ValidationError: If the configuration fields were not filled correctly.
+        """
+        from framework.test_suite import find_by_name
+
+        test_suite_spec = find_by_name(test_suite_name)
+        assert test_suite_spec is not None, f"Could not find test suite '{test_suite_name}'."
+        test_suite_spec.validate_test_cases(test_cases_names)
+
+        config = cls.model_validate({"test_cases": test_cases_names, **kwargs})
+        config._test_suite_spec = test_suite_spec
+        return config
+
+
+class BaseTestSuitesConfigs(BaseModel, extra="forbid"):
+    """Base class for test suites configs."""
+
+    def __contains__(self, key) -> bool:
+        """Check if the provided test suite name has been selected and/or configured."""
+        return key in self.model_fields_set
+
+    def __getitem__(self, key) -> TestSuiteConfig:
+        """Get test suite configuration."""
+        return self.__getattribute__(key)
+
+    def get_configs(self) -> Iterable[TestSuiteConfig]:
+        """Get all the test suite configurations."""
+        return map(lambda t: self[t], self.model_fields_set)
+
+    @classmethod
+    def available_test_suites(cls) -> Iterable[str]:
+        """List all the available test suites."""
+        return cls.model_fields.keys()
+
+    @field_validator("*")
+    @classmethod
+    def validate_test_suite_config(
+        cls, config: type[TestSuiteConfig], info: ValidationInfo
+    ) -> type[TestSuiteConfig]:
+        """Validate the provided test cases and link the test suite spec to the configuration."""
+        from framework.test_suite import find_by_name
+
+        test_suite_name = info.field_name
+        assert test_suite_name is not None
+
+        test_suite_spec = find_by_name(test_suite_name)
+        assert test_suite_spec is not None
+
+        config._test_suite_spec = test_suite_spec
+
+        test_suite_spec.validate_test_cases(config.test_cases_names)
+        return config
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 00b63cc292..bc7aa1555c 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -24,10 +24,13 @@
 from types import FunctionType
 from typing import Iterable
 
+from pydantic import ValidationError
+
 from framework.testbed_model.sut_node import SutNode
 from framework.testbed_model.tg_node import TGNode
 
 from .config import (
+    CUSTOM_CONFIG_TYPES,
     BuildTargetConfiguration,
     Configuration,
     SutNodeConfiguration,
@@ -36,7 +39,12 @@
     TGNodeConfiguration,
     load_config,
 )
-from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError
+from .exception import (
+    BlockingTestSuiteError,
+    ConfigurationError,
+    SSHTimeoutError,
+    TestCaseVerifyError,
+)
 from .logger import DTSLogger, DtsStage, get_dts_logger
 from .settings import SETTINGS
 from .test_result import (
@@ -142,12 +150,7 @@ def run(self) -> None:
                 self._logger.set_stage(DtsStage.test_run_setup)
                 self._logger.info(f"Running test run with SUT '{sut_node_config.name}'.")
                 test_run_result = self._result.add_test_run(test_run_config)
-                # we don't want to modify the original config, so create a copy
-                test_run_test_suites = list(
-                    SETTINGS.test_suites if SETTINGS.test_suites else test_run_config.test_suites
-                )
-                if not test_run_config.skip_smoke_tests:
-                    test_run_test_suites[:0] = [TestSuiteConfig("smoke_tests")]
+                test_run_test_suites = self._prepare_test_suites(test_run_config)
                 try:
                     test_suites_with_cases = self._get_test_suites_with_cases(
                         test_run_test_suites, test_run_config.func, test_run_config.perf
@@ -204,6 +207,40 @@ def _check_dts_python_version(self) -> None:
             )
             self._logger.warning("Please use Python >= 3.10 instead.")
 
+    def _prepare_test_suites(self, test_run_config: TestRunConfiguration) -> list[TestSuiteConfig]:
+        if SETTINGS.test_suites:
+            test_suites_configs = []
+            for selected_test_suite, selected_test_cases in SETTINGS.test_suites:
+                if selected_test_suite in test_run_config.test_suites:
+                    config = test_run_config.test_suites[selected_test_suite].model_copy()
+                    config.test_cases_names = selected_test_cases
+                else:
+                    try:
+                        config = CUSTOM_CONFIG_TYPES[selected_test_suite].make(
+                            selected_test_suite, *selected_test_cases
+                        )
+                    except AssertionError as e:
+                        raise ConfigurationError(
+                            "Invalid test cases were selected "
+                            f"for test suite {selected_test_suite}."
+                        ) from e
+                    except ValidationError as e:
+                        raise ConfigurationError(
+                            f"Test suite {selected_test_suite} needs to be explicitly configured "
+                            "in order to be selected."
+                        ) from e
+                test_suites_configs.append(config)
+        else:
+            # we don't want to modify the original config, so create a copy
+            test_suites_configs = [
+                config.model_copy() for config in test_run_config.test_suites.get_configs()
+            ]
+
+        if not test_run_config.skip_smoke_tests:
+            test_suites_configs[:0] = [TestSuiteConfig.make("smoke_tests")]
+
+        return test_suites_configs
+
     def _get_test_suites_with_cases(
         self,
         test_suite_configs: list[TestSuiteConfig],
@@ -245,7 +282,9 @@ def _get_test_suites_with_cases(
 
             test_suites_with_cases.append(
                 TestSuiteWithCases(
-                    test_suite_class=test_suite_class, test_cases=selected_test_cases
+                    test_suite_class=test_suite_class,
+                    test_cases=selected_test_cases,
+                    config=test_suite_config,
                 )
             )
         return test_suites_with_cases
@@ -466,7 +505,9 @@ def _run_test_suite(
         self._logger.set_stage(
             DtsStage.test_suite_setup, Path(SETTINGS.output_dir, test_suite_name)
         )
-        test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node)
+        test_suite = test_suite_with_cases.test_suite_class(
+            sut_node, tg_node, test_suite_with_cases.config
+        )
         try:
             self._logger.info(f"Starting test suite setup: {test_suite_name}")
             test_suite.set_up_suite()
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 2e8dedef4f..063f282edf 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -85,9 +85,7 @@
 from pathlib import Path
 from typing import Callable
 
-from pydantic import ValidationError
-
-from .config import TestSuiteConfig
+from .config import TestSuitesConfigs
 from .exception import ConfigurationError
 from .utils import DPDKGitTarball, get_commit_id
 
@@ -114,7 +112,7 @@ class Settings:
     #:
     compile_timeout: float = 1200
     #:
-    test_suites: list[TestSuiteConfig] = field(default_factory=list)
+    test_suites: list[tuple[str, list[str]]] = field(default_factory=list)
     #:
     re_run: int = 0
 
@@ -382,7 +380,7 @@ def _get_parser() -> _DTSArgumentParser:
 
 def _process_test_suites(
     parser: _DTSArgumentParser, args: list[list[str]]
-) -> list[TestSuiteConfig]:
+) -> list[tuple[str, list[str]]]:
     """Process the given argument to a list of :class:`TestSuiteConfig` to execute.
 
     Args:
@@ -398,16 +396,17 @@ def _process_test_suites(
         # Environment variable in the form of "SUITE1 CASE1 CASE2, SUITE2 CASE1, SUITE3, ..."
         args = [suite_with_cases.split() for suite_with_cases in args[0][0].split(",")]
 
-    try:
-        return [TestSuiteConfig(test_suite, test_cases) for [test_suite, *test_cases] in args]
-    except ValidationError as e:
-        print(
-            "An error has occurred while validating the test suites supplied in the "
-            f"{'environment variable' if action else 'arguments'}:",
-            file=sys.stderr,
-        )
-        print(e, file=sys.stderr)
-        sys.exit(1)
+    available_test_suites = TestSuitesConfigs.available_test_suites()
+    for test_suite_name, *_ in args:
+        if test_suite_name not in available_test_suites:
+            print(
+                f"The test suite {test_suite_name} supplied in the "
+                f"{'environment variable' if action else 'arguments'} is invalid.",
+                file=sys.stderr,
+            )
+            sys.exit(1)
+
+    return [(test_suite, test_cases) for test_suite, *test_cases in args]
 
 
 def get_settings() -> Settings:
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 5694a2482b..6c10c1e40a 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -64,17 +64,7 @@ class is to hold a subset of test cases (which could be all test cases) because
 
     test_suite_class: type[TestSuite]
     test_cases: list[FunctionType]
-
-    def create_config(self) -> TestSuiteConfig:
-        """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
-
-        Returns:
-            The :class:`TestSuiteConfig` representation.
-        """
-        return TestSuiteConfig(
-            test_suite=self.test_suite_class.__name__,
-            test_cases=[test_case.__name__ for test_case in self.test_cases],
-        )
+    config: TestSuiteConfig
 
 
 class Result(Enum):
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 972968b036..78e1b4c49a 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -23,7 +23,7 @@
 from ipaddress import IPv4Interface, IPv6Interface, ip_interface
 from pkgutil import iter_modules
 from types import FunctionType, ModuleType
-from typing import ClassVar, NamedTuple, Union
+from typing import ClassVar, Iterable, NamedTuple, Union, get_type_hints
 
 from pydantic.alias_generators import to_pascal
 from scapy.layers.inet import IP  # type: ignore[import-untyped]
@@ -31,6 +31,7 @@
 from scapy.packet import Packet, Padding  # type: ignore[import-untyped]
 from typing_extensions import Self
 
+from framework.config import TestSuiteConfig
 from framework.testbed_model.port import Port, PortLink
 from framework.testbed_model.sut_node import SutNode
 from framework.testbed_model.tg_node import TGNode
@@ -38,7 +39,7 @@
     PacketFilteringConfig,
 )
 
-from .exception import TestCaseVerifyError
+from .exception import InternalError, TestCaseVerifyError
 from .logger import DTSLogger, get_dts_logger
 from .utils import get_packet_summaries
 
@@ -78,6 +79,7 @@ class TestSuite:
     #: Whether the test suite is blocking. A failure of a blocking test suite
     #: will block the execution of all subsequent test suites in the current build target.
     is_blocking: ClassVar[bool] = False
+    config: TestSuiteConfig
     _logger: DTSLogger
     _port_links: list[PortLink]
     _sut_port_ingress: Port
@@ -93,6 +95,7 @@ def __init__(
         self,
         sut_node: SutNode,
         tg_node: TGNode,
+        config: TestSuiteConfig,
     ):
         """Initialize the test suite testbed information and basic configuration.
 
@@ -102,9 +105,11 @@ def __init__(
         Args:
             sut_node: The SUT node where the test suite will run.
             tg_node: The TG node where the test suite will run.
+            config: The test suite configuration.
         """
         self.sut_node = sut_node
         self.tg_node = tg_node
+        self.config = config
         self._logger = get_dts_logger(self.__class__.__name__)
         self._port_links = []
         self._process_links()
@@ -467,6 +472,21 @@ def is_test_suite(obj) -> bool:
 
         raise Exception("class not found in eligible test module")
 
+    @cached_property
+    def config_type(self) -> type[TestSuiteConfig]:
+        """A reference to the test suite's configuration type."""
+        fields = get_type_hints(self.class_type)
+        config_type = fields.get("config")
+        if config_type is None:
+            raise InternalError(
+                "Test suite class {self.class_name} is missing the `config` attribute."
+            )
+        if not issubclass(config_type, TestSuiteConfig):
+            raise InternalError(
+                f"Test suite class {self.class_name} has an invalid configuration type assigned."
+            )
+        return config_type
+
     @cached_property
     def test_cases(self) -> list[TestCase]:
         """A list of all the available test cases."""
@@ -533,6 +553,14 @@ def discover_all(
 
         return test_suites
 
+    def validate_test_cases(self, test_cases_names: Iterable[str]) -> None:
+        """Validate if the supplied test cases exist in the test suite."""
+        available_test_cases = map(lambda t: t.name, self.test_cases)
+        for requested_test_case in test_cases_names:
+            assert (
+                requested_test_case in available_test_cases
+            ), f"{requested_test_case} is not a valid test case for test suite {self.name}."
+
 
 AVAILABLE_TEST_SUITES: list[TestSuiteSpec] = TestSuiteSpec.discover_all()
 """Constant to store all the available, discovered and imported test suites.
diff --git a/dts/generate-schema.py b/dts/generate-schema.py
index b41d28492f..d24a67f68e 100755
--- a/dts/generate-schema.py
+++ b/dts/generate-schema.py
@@ -9,7 +9,7 @@
 
 from pydantic.json_schema import GenerateJsonSchema
 
-from framework.config import ConfigurationType
+from framework.config import ConfigurationType, TestSuitesConfigs
 
 DTS_DIR = os.path.dirname(os.path.realpath(__file__))
 RELATIVE_PATH_TO_SCHEMA = "framework/config/conf_yaml_schema.json"
@@ -26,6 +26,8 @@ def generate(self, schema, mode="validation"):
 
 
 try:
+    TestSuitesConfigs.fix_custom_config_annotations()
+
     path = os.path.join(DTS_DIR, RELATIVE_PATH_TO_SCHEMA)
 
     with open(path, "w") as schema_file:
diff --git a/dts/generate-test-mappings.py b/dts/generate-test-mappings.py
new file mode 100755
index 0000000000..d076ad0afe
--- /dev/null
+++ b/dts/generate-test-mappings.py
@@ -0,0 +1,132 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Test Suites to Configuration mappings generation script."""
+
+import os
+from collections import defaultdict
+from textwrap import indent
+from typing import Iterable
+
+from framework.config.test_suite import BaseTestSuitesConfigs, TestSuiteConfig
+from framework.exception import InternalError
+from framework.test_suite import AVAILABLE_TEST_SUITES, TestSuiteSpec
+
+DTS_DIR = os.path.dirname(os.path.realpath(__file__))
+SCRIPT_FILE_NAME = os.path.basename(__file__)
+
+FRAMEWORK_IMPORTS = [BaseTestSuitesConfigs, TestSuiteConfig]
+
+RELATIVE_PATH_TO_GENERATED_FILE = "framework/config/generated.py"
+SMOKE_TESTS_SUITE_NAME = "smoke_tests"
+CUSTOM_CONFIG_TYPES_VAR_NAME = "CUSTOM_CONFIG_TYPES"
+CUSTOM_CONFIG_LINKING_FUNCTION_NAME = "fix_custom_config_annotations"
+CUSTOM_CONFIG_LINKING_FUNCTION_DOCSTRING = [
+    '"""Fixes the custom config types annotations.',
+    "",
+    f"Moreover it also fills `{CUSTOM_CONFIG_TYPES_VAR_NAME}` with all the custom config types.",
+    '"""',
+]
+TEST_SUITES_CONFIG_CLASS_NAME = "TestSuitesConfigs"
+TEST_SUITES_CONFIG_CLASS_DOCSTRING = [
+    '"""Configuration mapping class to select and configure the test suites.',
+    "",
+    "Before using this class, the custom configuration type annotations need to be fixed.",
+    f"To do so, you need to call the `{CUSTOM_CONFIG_LINKING_FUNCTION_NAME}` method.",
+    '"""',
+]
+
+
+GENERATED_FILE_HEADER = f"""# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 The DPDK contributors
+# This file is automatically generated by {SCRIPT_FILE_NAME}.
+# Do NOT modify this file manually.
+
+\"\"\"Generated file containing the links between the test suites and the configuration.\"\"\"
+"""
+
+
+def join(lines: Iterable[str]) -> str:
+    """Join list of strings into text lines."""
+    return "\n".join(lines)
+
+
+def join_and_indent(lines: Iterable[str], indentation_level=1, indentation_spaces=4) -> str:
+    """Join list of strings into indented text lines."""
+    return "\n".join([indent(line, " " * indentation_level * indentation_spaces) for line in lines])
+
+
+def format_attributes_types(test_suite_spec: TestSuiteSpec):
+    """Format the config type into the respective configuration class field attribute type."""
+    config_type = test_suite_spec.config_type.__name__
+    if config_type != TestSuiteConfig.__name__:
+        config_type = f'"{config_type}"'
+    return f"Optional[{config_type}]"
+
+
+try:
+    framework_imports: dict[str, list[str]] = defaultdict(list)
+    for _import in FRAMEWORK_IMPORTS:
+        framework_imports[_import.__module__].append(_import.__name__)
+    formatted_framework_imports = sorted(
+        [
+            f"from {module} import {', '.join(sorted(imports))}"
+            for module, imports in framework_imports.items()
+        ]
+    )
+
+    test_suites = [
+        test_suite_spec
+        for test_suite_spec in AVAILABLE_TEST_SUITES
+        if test_suite_spec.name != SMOKE_TESTS_SUITE_NAME
+    ]
+
+    custom_configs = [t for t in test_suites if t.config_type is not TestSuiteConfig]
+
+    custom_config_imports = [
+        f"from {t.module_type.__name__} import {t.config_type.__name__}" for t in custom_configs
+    ]
+
+    test_suites_attributes = [f"{t.name}: {format_attributes_types(t)} = None" for t in test_suites]
+
+    custom_config_assignments = [
+        f'{CUSTOM_CONFIG_TYPES_VAR_NAME}["{t.name}"] = {t.config_type.__name__}'
+        for t in custom_configs
+    ]
+
+    generated_file_contents = f"""{GENERATED_FILE_HEADER}
+from typing import TYPE_CHECKING, Optional
+
+{join(formatted_framework_imports)}
+
+if TYPE_CHECKING:
+{join_and_indent(custom_config_imports)}
+
+
+{CUSTOM_CONFIG_TYPES_VAR_NAME}: dict[str, type[{TestSuiteConfig.__name__}]] = {'{}'}
+
+
+class {TEST_SUITES_CONFIG_CLASS_NAME}({BaseTestSuitesConfigs.__name__}):
+{join_and_indent(TEST_SUITES_CONFIG_CLASS_DOCSTRING)}
+
+{join_and_indent(test_suites_attributes)}
+
+    @classmethod
+    def {CUSTOM_CONFIG_LINKING_FUNCTION_NAME}(cls):
+{join_and_indent(CUSTOM_CONFIG_LINKING_FUNCTION_DOCSTRING, indentation_level=2)}
+{join_and_indent(custom_config_imports, indentation_level=2)}
+
+{join_and_indent(custom_config_assignments, indentation_level=2)}
+
+        cls.model_rebuild()
+"""
+
+    path = os.path.join(DTS_DIR, RELATIVE_PATH_TO_GENERATED_FILE)
+
+    with open(path, "w") as generated_file:
+        generated_file.write(generated_file_contents)
+
+    print("Test suites to configuration mappings generated successfully!")
+except Exception as e:
+    raise InternalError("Failed to generate test suites to configuration mappings.") from e
diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
index d958f99030..1723c123bc 100644
--- a/dts/tests/TestSuite_hello_world.py
+++ b/dts/tests/TestSuite_hello_world.py
@@ -7,6 +7,7 @@
 No other EAL parameters apart from cores are used.
 """
 
+from framework.config import TestSuiteConfig
 from framework.remote_session.dpdk_shell import compute_eal_params
 from framework.test_suite import TestSuite
 from framework.testbed_model.cpu import (
@@ -16,9 +17,18 @@
 )
 
 
+class HelloWorldConfig(TestSuiteConfig):
+    """Example custom configuration for the `TestHelloWorld` test suite."""
+
+    #: Timeout for the DPDK apps
+    timeout: int = 50
+
+
 class TestHelloWorld(TestSuite):
     """DPDK hello world app test suite."""
 
+    config: HelloWorldConfig
+
     def set_up_suite(self) -> None:
         """Set up the test suite.
 
@@ -59,7 +69,7 @@ def test_hello_world_all_cores(self) -> None:
         eal_para = compute_eal_params(
             self.sut_node, lcore_filter_specifier=LogicalCoreList(self.sut_node.lcores)
         )
-        result = self.sut_node.run_dpdk_app(self.app_helloworld_path, eal_para, 50)
+        result = self.sut_node.run_dpdk_app(self.app_helloworld_path, eal_para, self.config.timeout)
         for lcore in self.sut_node.lcores:
             self.verify(
                 f"hello from core {int(lcore)}" in result.stdout,
diff --git a/dts/tests/__init__.py b/dts/tests/__init__.py
new file mode 100644
index 0000000000..a300eb26fc
--- /dev/null
+++ b/dts/tests/__init__.py
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Test suites.
+
+This package contains all the available test suites in DTS.
+"""
-- 
2.34.1


^ permalink raw reply	[flat|nested] 27+ messages in thread

* Re: [PATCH] dts: add per-test-suite configuration
  2024-09-06 16:13 [PATCH] dts: add per-test-suite configuration Luca Vizzarro
@ 2024-09-27 17:45 ` Jeremy Spewock
  2024-11-08 13:38 ` [PATCH v2 0/4] " Luca Vizzarro
  2024-11-08 13:45 ` [PATCH v3 0/4] dts: add per-test-suite configuration Luca Vizzarro
  2 siblings, 0 replies; 27+ messages in thread
From: Jeremy Spewock @ 2024-09-27 17:45 UTC (permalink / raw)
  To: Luca Vizzarro
  Cc: dev, Honnappa Nagarahalli, Paul Szczepanek, Alex Chapman,
	Juraj Linkeš

Hi Luca,

I apologize for not giving this patch more time/attention. I think
this patch is worth having a more in-depth discussion about since it
does add quite a bit more complexity, but the benefit of that is some
valuable simplicity. I know we discussed in the recent team meeting
about whether or not to have the test suite configurations in
conf.yaml or make ones specifically for each individual test suite,
and I think that it could really go either way, but that them being in
conf.yaml is potentially slightly better in my opinion as it helps
keep things uniform, but obviously you get into trouble there if there
is a lot of configuration and the file gets long.

On Fri, Sep 6, 2024 at 12:14 PM Luca Vizzarro <luca.vizzarro@arm.com> wrote:
>
> Allow test suites to be configured individually. Moreover enable them to
> implement their own custom configuration.
>
> This solution adds some new complexity to DTS, which is generated source
> code. In order to ensure strong typing, the test suites and their custom

Generated source code is definitely something new and interesting. I
think the way you did it makes sense and it's well written, but the
idea of generated source code scares me slightly. It leads to the sort
of additional requirement when you are writing test suites to
regenerate the code if you have a custom config, and seems a little
harder to maintain (although, I doubt much maintenance will really be
needed on it, so maybe that's not a good point). It makes me wonder if
the very dumb approach of test suites just each getting a yaml file
that matches the same of the test suite (and maybe ends with _conf)
and then importing that into a TypedDict like we do with conf.yaml
would be worth it to save on this complexity. It would still be simple
enough for a test suite developer to just throw a yaml file into a
directory (maybe conf/ ?) and have the framework auto-magically
consume that and match it to a TypedDict. I guess there would be some
extra rules in place with the dumb approach like the name of the
config file and maybe the name of the TypedDict, but it still feels
like it would save on complexity in the long run. Of course, that's
not to say complexity is bad if it makes things easier, but it begs
the question of how much easier is doing it this way versus making a
yaml file and a matching TypedDict.

Again, something good to discuss in a meeting.

> configurations need to be linked in the main configuration class.
> Unfortunately, this is not feasible during runtime as it will incur in
> circular dependencies. Generating the links appear to be the most
> straightforward approach.
>
> This commit also brings a new major change to the configuration schema.
> Test suites are no longer defined as a list of strings, like:
>
>     test_suites:
>     - hello_world
>     - pmd_buffer_scatter
>
> but as mapping of mappings or strings:
>
>     test_suites:
>       hello_world: {} # any custom fields or test cases can be set here
>       pmd_buffer_scatter: all # "all" defines all the test cases, or
>                               # they can individually be set separated
>                               # by a space
>
> Not defining the `test_cases` field in the configuration is equivalent
> to `all`, therefore the definitions for either test suite above are
> also equivalent.

Making these mappings in general however that also allow you to
specify which test cases to run I really like. It doesn't seem
completely relevant to the suite-wide configurations though, maybe it
would make sense to be in a different patch? I get why it's all in one
though since it they are handled in the same places.

>
> Creating the __init__.py file under the tests folder, allows it to be
> picked up as a package. This is a mypy requirement to import the tests
> from within the framework.
>
> Bugzilla ID: 1375
>
> Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
> Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Reviewed-by: Alex Chapman <alex.chapman@arm.com>

I didn't really have any comments on this implementation overall
however. I didn't fully understand everything (mostly due to my lack
of pydantic understanding) so I probably would have had to read it
over a few more times to completely understand it, but in general I
think this seems well written. Great work Luca.

> ---
> Depends-on: series-32823 ("dts: Pydantic configuration")
>
> Hello,
>
> sending in a solution for the per-test-suite configuration issue.
> This one took some thinking but I have given most of the motivations in
> the commit body already. The docs are somewhat lacking but hopefully they
> should be automatically tackled by the API docs generation.
>
>
> Best,
> Luca
> ---
>  doc/guides/tools/dts.rst                   |  39 ++++--
>  dts/conf.yaml                              |   4 +-
>  dts/framework/config/__init__.py           |  98 ++-------------
>  dts/framework/config/conf_yaml_schema.json |  94 +++++++++++---
>  dts/framework/config/generated.py          |  40 ++++++
>  dts/framework/config/test_suite.py         | 140 +++++++++++++++++++++
>  dts/framework/runner.py                    |  59 +++++++--
>  dts/framework/settings.py                  |  29 +++--
>  dts/framework/test_result.py               |  12 +-
>  dts/framework/test_suite.py                |  32 ++++-
>  dts/generate-schema.py                     |   4 +-
>  dts/generate-test-mappings.py              | 132 +++++++++++++++++++
>  dts/tests/TestSuite_hello_world.py         |  12 +-
>  dts/tests/__init__.py                      |   7 ++
>  14 files changed, 539 insertions(+), 163 deletions(-)
>  create mode 100644 dts/framework/config/generated.py
>  create mode 100644 dts/framework/config/test_suite.py
>  create mode 100755 dts/generate-test-mappings.py
>  create mode 100644 dts/tests/__init__.py
>
> diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
> index 317bd0ff99..66681543cd 100644
> --- a/doc/guides/tools/dts.rst
> +++ b/doc/guides/tools/dts.rst
> @@ -391,6 +391,26 @@ There are four types of methods that comprise a test suite:
>     should be implemented in the ``SutNode`` class (and the underlying classes that ``SutNode`` uses)
>     and used by the test suite via the ``sut_node`` field.
>
> +The test suites can also implement their own custom configuration fields. This can be achieved by
> +creating a new test suite config file which inherits from ``TestSuiteConfig`` defined in
> +``dts/framework/config/test_suite.py``. So that this new custom configuration class is used, the
> +test suite class must override the ``config`` attribute annotation with your new class, for example::
> +
> +.. code:: python
> +   class CustomConfig(TestSuiteConfig):
> +      my_custom_field: int = 10
> +
> +   class TestMyNewTestSuite(TestSuite):
> +      config: CustomConfig
> +
> +Finally, the test suites and the custom configuration files need to linked in the global configuration.
> +This can be easily achieved by running the ``dts/generate-test-mappings.py``, e.g.:
> +
> +.. code-block:: console
> +
> +   $ poetry shell
> +   (dts-py3.10) $ ./generate-test-mappings.py
> +
>
>  .. _dts_dev_tools:
>
> @@ -510,18 +530,13 @@ _`Network port`
>     ``peer_pci``           *string* – the PCI address of the peer node port. **Example**: ``000a:01:00.1``
>     ====================== =================================================================================
>
> -_`Test suite`
> -   *string* – name of the test suite to run. **Examples**: ``hello_world``, ``os_udp``
> -
> -_`Test target`
> -   *mapping* – selects specific test cases to run from a test suite. Mapping is described as follows:
> -
> -   ========= ===============================================================================================
> -   ``suite`` See `Test suite`_
> -   ``cases`` (*optional*) *sequence* of *string* – list of the selected test cases in the test suite to run.
> +_`Test suites`
> +   *mapping* – selects the test suites to run. Each mapping key corresponds to the test suite name.
>
> -             Unknown test cases will be silently ignored.
> -   ========= ===============================================================================================
> +   The value of the mapping can either "all" to select all the test cases in that test suite, the test
> +   cases names divided by a space. Or it can be another mapping to set any custom fields for the test suite.
> +   In the case of a mapping, all the test cases are selected by default. In order to manually select test
> +   cases, the ``test_cases`` field can be set with a list of strings, each entry being a test case name.
>
>
>  Properties
> @@ -542,7 +557,7 @@ involved in the testing. These can be defined with the following mappings:
>     +----------------------------+-------------------------------------------------------------------+
>     | ``func``                   | *boolean* – Enable functional testing.                            |
>     +----------------------------+-------------------------------------------------------------------+
> -   | ``test_suites``            | *sequence* of **one of** `Test suite`_ **or** `Test target`_      |
> +   | ``test_suites``            | See `Test suites`_                                                |
>     +----------------------------+-------------------------------------------------------------------+
>     | ``skip_smoke_tests``       | (*optional*) *boolean* – Allows you to skip smoke testing         |
>     |                            | if ``true``.                                                      |
> diff --git a/dts/conf.yaml b/dts/conf.yaml
> index 7d95016e68..c44bef604c 100644
> --- a/dts/conf.yaml
> +++ b/dts/conf.yaml
> @@ -15,8 +15,8 @@ test_runs:
>      func: true # enable functional testing
>      skip_smoke_tests: false # optional
>      test_suites: # the following test suites will be run in their entirety
> -      - hello_world
> -      - os_udp
> +      hello_world: all
> +      os_udp: all
>      # The machine running the DPDK test executable
>      system_under_test_node:
>        node_name: "SUT 1"
> diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
> index 013c529829..d39e0823bd 100644
> --- a/dts/framework/config/__init__.py
> +++ b/dts/framework/config/__init__.py
> @@ -35,10 +35,12 @@
>        and makes it thread safe should we ever want to move in that direction.
>  """
>
> +# pylama:ignore=W0611
> +
>  from enum import Enum, auto, unique
>  from functools import cached_property
>  from pathlib import Path
> -from typing import TYPE_CHECKING, Annotated, Any, Literal, NamedTuple, Protocol
> +from typing import Annotated, Literal, NamedTuple, Protocol
>
>  import yaml
>  from pydantic import (
> @@ -50,15 +52,14 @@
>      field_validator,
>      model_validator,
>  )
> -from pydantic.config import JsonDict
>  from pydantic.dataclasses import dataclass
>  from typing_extensions import Self
>
>  from framework.exception import ConfigurationError
>  from framework.utils import StrEnum
>
> -if TYPE_CHECKING:
> -    from framework.test_suite import TestSuiteSpec
> +from .generated import CUSTOM_CONFIG_TYPES, TestSuitesConfigs
> +from .test_suite import TestSuiteConfig
>
>
>  @unique
> @@ -289,7 +290,7 @@ class NodeInfo:
>      kernel_version: str
>
>
> -@dataclass(slots=True, frozen=True, kw_only=True, config=ConfigDict(extra="forbid"))
> +@dataclass(frozen=True, kw_only=True, config=ConfigDict(extra="forbid"))
>  class BuildTargetConfiguration:
>      """DPDK build configuration.
>
> @@ -329,89 +330,6 @@ class BuildTargetInfo:
>      compiler_version: str
>
>
> -def make_parsable_schema(schema: JsonDict):
> -    """Updates a model's JSON schema to make a string representation a valid alternative.
> -
> -    This utility function is required to be used with models that can be represented and validated
> -    as a string instead of an object mapping. Normally the generated JSON schema will just show
> -    the object mapping. This function wraps the mapping under an anyOf property sequenced with a
> -    string type.
> -
> -    This function is a valid `Callable` for the `json_schema_extra` attribute of
> -    `~pydantic.config.ConfigDict`.
> -    """
> -    inner_schema = schema.copy()
> -    del inner_schema["title"]
> -
> -    title = schema.get("title")
> -    description = schema.get("description")
> -
> -    schema.clear()
> -
> -    schema["title"] = title
> -    schema["description"] = description
> -    schema["anyOf"] = [inner_schema, {"type": "string"}]
> -
> -
> -@dataclass(
> -    frozen=True,
> -    config=ConfigDict(extra="forbid", json_schema_extra=make_parsable_schema),
> -)
> -class TestSuiteConfig:
> -    """Test suite configuration.
> -
> -    Information about a single test suite to be executed. It can be represented and validated as a
> -    string type in the form of: ``TEST_SUITE [TEST_CASE, ...]``, in the configuration file.
> -
> -    Attributes:
> -        test_suite: The name of the test suite module without the starting ``TestSuite_``.
> -        test_cases: The names of test cases from this test suite to execute.
> -            If empty, all test cases will be executed.
> -    """
> -
> -    test_suite_name: str = Field(
> -        title="Test suite name",
> -        description="The identifying name of the test suite.",
> -        alias="test_suite",
> -    )
> -    test_cases_names: list[str] = Field(
> -        default_factory=list,
> -        title="Test cases by name",
> -        description="The identifying name of the test cases of the test suite.",
> -        alias="test_cases",
> -    )
> -
> -    @cached_property
> -    def test_suite_spec(self) -> "TestSuiteSpec":
> -        """The specification of the requested test suite."""
> -        from framework.test_suite import find_by_name
> -
> -        test_suite_spec = find_by_name(self.test_suite_name)
> -        assert test_suite_spec is not None, f"{self.test_suite_name} is not a valid test suite name"
> -        return test_suite_spec
> -
> -    @model_validator(mode="before")
> -    @classmethod
> -    def convert_from_string(cls, data: Any) -> Any:
> -        """Convert the string representation into a valid mapping."""
> -        if isinstance(data, str):
> -            [test_suite, *test_cases] = data.split()
> -            return dict(test_suite=test_suite, test_cases=test_cases)
> -        return data
> -
> -    @model_validator(mode="after")
> -    def validate_names(self) -> Self:
> -        """Validate the supplied test suite and test cases names."""
> -        available_test_cases = map(lambda t: t.name, self.test_suite_spec.test_cases)
> -        for requested_test_case in self.test_cases_names:
> -            assert requested_test_case in available_test_cases, (
> -                f"{requested_test_case} is not a valid test case "
> -                f"for test suite {self.test_suite_name}"
> -            )
> -
> -        return self
> -
> -
>  @dataclass(slots=True, frozen=True, kw_only=True, config=ConfigDict(extra="forbid"))
>  class TestRunSUTNodeConfiguration:
>      """The SUT node configuration of a test run.
> @@ -446,7 +364,7 @@ class TestRunConfiguration:
>      perf: bool = Field(description="Enable performance testing.")
>      func: bool = Field(description="Enable functional testing.")
>      skip_smoke_tests: bool = False
> -    test_suites: list[TestSuiteConfig] = Field(min_length=1)
> +    test_suites: TestSuitesConfigs
>      system_under_test_node: TestRunSUTNodeConfiguration
>      traffic_generator_node: str
>
> @@ -581,7 +499,7 @@ def load_config(config_file_path: Path) -> Configuration:
>          config_data = yaml.safe_load(f)
>
>      try:
> -        ConfigurationType.json_schema()
> +        TestSuitesConfigs.fix_custom_config_annotations()
>          return ConfigurationType.validate_python(config_data)
>      except ValidationError as e:
>          raise ConfigurationError("failed to load the supplied configuration") from e
> diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
> index 1cf1bb098a..1434cdfad3 100644
> --- a/dts/framework/config/conf_yaml_schema.json
> +++ b/dts/framework/config/conf_yaml_schema.json
> @@ -66,6 +66,33 @@
>        "title": "Compiler",
>        "type": "string"
>      },
> +    "HelloWorldConfig": {
> +      "anyOf": [
> +        {
> +          "additionalProperties": false,
> +          "properties": {
> +            "test_cases": {
> +              "items": {
> +                "type": "string"
> +              },
> +              "title": "Test Cases",
> +              "type": "array"
> +            },
> +            "timeout": {
> +              "default": 50,
> +              "title": "Timeout",
> +              "type": "integer"
> +            }
> +          },
> +          "type": "object"
> +        },
> +        {
> +          "type": "string"
> +        }
> +      ],
> +      "description": "Example custom configuration for the `TestHelloWorld` test suite.",
> +      "title": "HelloWorldConfig"
> +    },
>      "HugepageConfiguration": {
>        "additionalProperties": false,
>        "description": "The hugepage configuration of :class:`~framework.testbed_model.node.Node`\\s.\n\nAttributes:\n    number_of: The number of hugepages to allocate.\n    force_first_numa: If :data:`True`, the hugepages will be configured on the first NUMA node.",
> @@ -373,14 +400,6 @@
>            "title": "Func",
>            "type": "boolean"
>          },
> -        "test_suites": {
> -          "items": {
> -            "$ref": "#/$defs/TestSuiteConfig"
> -          },
> -          "minItems": 1,
> -          "title": "Test Suites",
> -          "type": "array"
> -        },
>          "build_targets": {
>            "items": {
>              "$ref": "#/$defs/BuildTargetConfiguration"
> @@ -393,6 +412,9 @@
>            "title": "Skip Smoke Tests",
>            "type": "boolean"
>          },
> +        "test_suites": {
> +          "$ref": "#/$defs/TestSuitesConfigs"
> +        },
>          "system_under_test_node": {
>            "$ref": "#/$defs/TestRunSUTNodeConfiguration"
>          },
> @@ -404,8 +426,8 @@
>        "required": [
>          "perf",
>          "func",
> -        "test_suites",
>          "build_targets",
> +        "test_suites",
>          "system_under_test_node",
>          "traffic_generator_node"
>        ],
> @@ -439,31 +461,63 @@
>          {
>            "additionalProperties": false,
>            "properties": {
> -            "test_suite": {
> -              "description": "The identifying name of the test suite.",
> -              "title": "Test suite name",
> -              "type": "string"
> -            },
>              "test_cases": {
> -              "description": "The identifying name of the test cases of the test suite.",
>                "items": {
>                  "type": "string"
>                },
> -              "title": "Test cases by name",
> +              "title": "Test Cases",
>                "type": "array"
>              }
>            },
> -          "required": [
> -            "test_suite"
> -          ],
>            "type": "object"
>          },
>          {
>            "type": "string"
>          }
>        ],
> -      "description": "Test suite configuration.\n\nInformation about a single test suite to be executed. It can be represented and validated as a\nstring type in the form of: ``TEST_SUITE [TEST_CASE, ...]``, in the configuration file.\n\nAttributes:\n    test_suite: The name of the test suite module without the starting ``TestSuite_``.\n    test_cases: The names of test cases from this test suite to execute.\n        If empty, all test cases will be executed.",
> +      "description": "Test suite configuration base model.\n\nBy default the configuration of a generic test suite does not contain any attributes. Any test\nsuite should inherit this class to create their own custom configuration. Finally override the\ntype of the :attr:`~TestSuite.config` to use the newly created one.\n\nAttributes:\n    test_cases_names: The names of test cases from this test suite to execute. If empty, all\n        test cases will be executed.",
>        "title": "TestSuiteConfig"
> +    },
> +    "TestSuitesConfigs": {
> +      "additionalProperties": false,
> +      "description": "Configuration mapping class to select and configure the test suites.\n\nBefore using this class, the custom configuration type annotations need to be fixed.\nTo do so, you need to call the `fix_custom_config_annotations` method.",
> +      "properties": {
> +        "hello_world": {
> +          "anyOf": [
> +            {
> +              "$ref": "#/$defs/HelloWorldConfig"
> +            },
> +            {
> +              "type": "null"
> +            }
> +          ],
> +          "default": null
> +        },
> +        "os_udp": {
> +          "anyOf": [
> +            {
> +              "$ref": "#/$defs/TestSuiteConfig"
> +            },
> +            {
> +              "type": "null"
> +            }
> +          ],
> +          "default": null
> +        },
> +        "pmd_buffer_scatter": {
> +          "anyOf": [
> +            {
> +              "$ref": "#/$defs/TestSuiteConfig"
> +            },
> +            {
> +              "type": "null"
> +            }
> +          ],
> +          "default": null
> +        }
> +      },
> +      "title": "TestSuitesConfigs",
> +      "type": "object"
>      }
>    },
>    "description": "DTS testbed and test configuration.\n\nAttributes:\n    test_runs: Test run configurations.\n    nodes: Node configurations.",
> diff --git a/dts/framework/config/generated.py b/dts/framework/config/generated.py
> new file mode 100644
> index 0000000000..d42ce93a51
> --- /dev/null
> +++ b/dts/framework/config/generated.py
> @@ -0,0 +1,40 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2024 The DPDK contributors
> +# This file is automatically generated by generate-test-mappings.py.
> +# Do NOT modify this file manually.
> +
> +"""Generated file containing the links between the test suites and the configuration."""
> +
> +from typing import TYPE_CHECKING, Optional
> +
> +from framework.config.test_suite import BaseTestSuitesConfigs, TestSuiteConfig
> +
> +if TYPE_CHECKING:
> +    from tests.TestSuite_hello_world import HelloWorldConfig
> +
> +
> +CUSTOM_CONFIG_TYPES: dict[str, type[TestSuiteConfig]] = {}
> +
> +
> +class TestSuitesConfigs(BaseTestSuitesConfigs):
> +    """Configuration mapping class to select and configure the test suites.
> +
> +    Before using this class, the custom configuration type annotations need to be fixed.
> +    To do so, you need to call the `fix_custom_config_annotations` method.
> +    """
> +
> +    hello_world: Optional["HelloWorldConfig"] = None
> +    os_udp: Optional[TestSuiteConfig] = None
> +    pmd_buffer_scatter: Optional[TestSuiteConfig] = None
> +
> +    @classmethod
> +    def fix_custom_config_annotations(cls):
> +        """Fixes the custom config types annotations.
> +
> +        Moreover it also fills `CUSTOM_CONFIG_TYPES` with all the custom config types.
> +        """
> +        from tests.TestSuite_hello_world import HelloWorldConfig
> +
> +        CUSTOM_CONFIG_TYPES["hello_world"] = HelloWorldConfig
> +
> +        cls.model_rebuild()
> diff --git a/dts/framework/config/test_suite.py b/dts/framework/config/test_suite.py
> new file mode 100644
> index 0000000000..0c1ddf9d95
> --- /dev/null
> +++ b/dts/framework/config/test_suite.py
> @@ -0,0 +1,140 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2024 Arm Limited
> +
> +"""Test suites configuration module.
> +
> +Test suites can inherit :class:`TestSuiteConfig` to create their own custom configuration.
> +By doing so, the test suite class must also override the annotation of the field
> +`~framework.test_suite.TestSuite.config` to use their custom configuration type.
> +"""
> +
> +from typing import TYPE_CHECKING, Any, Iterable
> +
> +from pydantic import BaseModel, Field, ValidationInfo, field_validator, model_validator
> +from pydantic.config import JsonDict
> +from typing_extensions import Self
> +
> +if TYPE_CHECKING:
> +    from framework.test_suite import TestSuiteSpec
> +
> +
> +def make_parsable_schema(schema: JsonDict):
> +    """Updates a model's JSON schema to make a string representation a valid alternative.
> +
> +    This utility function is required to be used with models that can be represented and validated
> +    as a string instead of an object mapping. Normally the generated JSON schema will just show
> +    the object mapping. This function wraps the mapping under an anyOf property sequenced with a
> +    string type.
> +
> +    This function is a valid `Callable` for the `json_schema_extra` attribute of
> +    `~pydantic.config.ConfigDict`.
> +    """
> +    inner_schema = schema.copy()
> +
> +    fields_to_preserve = ["title", "description"]
> +    extracted_fields = {k: v for k in fields_to_preserve if (v := inner_schema.get(k))}
> +    for field in extracted_fields:
> +        del inner_schema[field]
> +
> +    schema.clear()
> +    schema.update(extracted_fields)
> +    schema["anyOf"] = [inner_schema, {"type": "string"}]
> +
> +
> +class TestSuiteConfig(BaseModel, extra="forbid", json_schema_extra=make_parsable_schema):
> +    """Test suite configuration base model.
> +
> +    By default the configuration of a generic test suite does not contain any attributes. Any test
> +    suite should inherit this class to create their own custom configuration. Finally override the
> +    type of the :attr:`~TestSuite.config` to use the newly created one.
> +
> +    Attributes:
> +        test_cases_names: The names of test cases from this test suite to execute. If empty, all
> +            test cases will be executed.
> +    """
> +
> +    _test_suite_spec: "TestSuiteSpec"
> +
> +    test_cases_names: list[str] = Field(default_factory=list, alias="test_cases")
> +
> +    @property
> +    def test_suite_name(self) -> str:
> +        """The name of the test suite module without the starting ``TestSuite_``."""
> +        return self._test_suite_spec.name
> +
> +    @property
> +    def test_suite_spec(self) -> "TestSuiteSpec":
> +        """The specification of the requested test suite."""
> +        return self._test_suite_spec
> +
> +    @model_validator(mode="before")
> +    @classmethod
> +    def convert_from_string(cls, data: Any) -> Any:
> +        """Validator which allows to select a test suite by string instead of a mapping."""
> +        if isinstance(data, str):
> +            test_cases = [] if data == "all" else data.split()
> +            return dict(test_cases=test_cases)
> +        return data
> +
> +    @classmethod
> +    def make(cls, test_suite_name: str, *test_cases_names: str, **kwargs) -> Self:
> +        """Make a configuration for the requested test suite.
> +
> +        Args:
> +            test_suite_name: The name of the test suite.
> +            test_cases_names: The test cases to select, if empty all are selected.
> +            **kwargs: Any other configuration field.
> +
> +        Raises:
> +            AssertionError: If the requested test suite or test cases do not exist.
> +            ValidationError: If the configuration fields were not filled correctly.
> +        """
> +        from framework.test_suite import find_by_name
> +
> +        test_suite_spec = find_by_name(test_suite_name)
> +        assert test_suite_spec is not None, f"Could not find test suite '{test_suite_name}'."
> +        test_suite_spec.validate_test_cases(test_cases_names)
> +
> +        config = cls.model_validate({"test_cases": test_cases_names, **kwargs})
> +        config._test_suite_spec = test_suite_spec
> +        return config
> +
> +
> +class BaseTestSuitesConfigs(BaseModel, extra="forbid"):
> +    """Base class for test suites configs."""
> +
> +    def __contains__(self, key) -> bool:
> +        """Check if the provided test suite name has been selected and/or configured."""
> +        return key in self.model_fields_set
> +
> +    def __getitem__(self, key) -> TestSuiteConfig:
> +        """Get test suite configuration."""
> +        return self.__getattribute__(key)
> +
> +    def get_configs(self) -> Iterable[TestSuiteConfig]:
> +        """Get all the test suite configurations."""
> +        return map(lambda t: self[t], self.model_fields_set)
> +
> +    @classmethod
> +    def available_test_suites(cls) -> Iterable[str]:
> +        """List all the available test suites."""
> +        return cls.model_fields.keys()
> +
> +    @field_validator("*")
> +    @classmethod
> +    def validate_test_suite_config(
> +        cls, config: type[TestSuiteConfig], info: ValidationInfo
> +    ) -> type[TestSuiteConfig]:
> +        """Validate the provided test cases and link the test suite spec to the configuration."""
> +        from framework.test_suite import find_by_name
> +
> +        test_suite_name = info.field_name
> +        assert test_suite_name is not None
> +
> +        test_suite_spec = find_by_name(test_suite_name)
> +        assert test_suite_spec is not None
> +
> +        config._test_suite_spec = test_suite_spec
> +
> +        test_suite_spec.validate_test_cases(config.test_cases_names)
> +        return config
> diff --git a/dts/framework/runner.py b/dts/framework/runner.py
> index 00b63cc292..bc7aa1555c 100644
> --- a/dts/framework/runner.py
> +++ b/dts/framework/runner.py
> @@ -24,10 +24,13 @@
>  from types import FunctionType
>  from typing import Iterable
>
> +from pydantic import ValidationError
> +
>  from framework.testbed_model.sut_node import SutNode
>  from framework.testbed_model.tg_node import TGNode
>
>  from .config import (
> +    CUSTOM_CONFIG_TYPES,
>      BuildTargetConfiguration,
>      Configuration,
>      SutNodeConfiguration,
> @@ -36,7 +39,12 @@
>      TGNodeConfiguration,
>      load_config,
>  )
> -from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError
> +from .exception import (
> +    BlockingTestSuiteError,
> +    ConfigurationError,
> +    SSHTimeoutError,
> +    TestCaseVerifyError,
> +)
>  from .logger import DTSLogger, DtsStage, get_dts_logger
>  from .settings import SETTINGS
>  from .test_result import (
> @@ -142,12 +150,7 @@ def run(self) -> None:
>                  self._logger.set_stage(DtsStage.test_run_setup)
>                  self._logger.info(f"Running test run with SUT '{sut_node_config.name}'.")
>                  test_run_result = self._result.add_test_run(test_run_config)
> -                # we don't want to modify the original config, so create a copy
> -                test_run_test_suites = list(
> -                    SETTINGS.test_suites if SETTINGS.test_suites else test_run_config.test_suites
> -                )
> -                if not test_run_config.skip_smoke_tests:
> -                    test_run_test_suites[:0] = [TestSuiteConfig("smoke_tests")]
> +                test_run_test_suites = self._prepare_test_suites(test_run_config)
>                  try:
>                      test_suites_with_cases = self._get_test_suites_with_cases(
>                          test_run_test_suites, test_run_config.func, test_run_config.perf
> @@ -204,6 +207,40 @@ def _check_dts_python_version(self) -> None:
>              )
>              self._logger.warning("Please use Python >= 3.10 instead.")
>
> +    def _prepare_test_suites(self, test_run_config: TestRunConfiguration) -> list[TestSuiteConfig]:
> +        if SETTINGS.test_suites:
> +            test_suites_configs = []
> +            for selected_test_suite, selected_test_cases in SETTINGS.test_suites:
> +                if selected_test_suite in test_run_config.test_suites:
> +                    config = test_run_config.test_suites[selected_test_suite].model_copy()
> +                    config.test_cases_names = selected_test_cases
> +                else:
> +                    try:
> +                        config = CUSTOM_CONFIG_TYPES[selected_test_suite].make(
> +                            selected_test_suite, *selected_test_cases
> +                        )
> +                    except AssertionError as e:
> +                        raise ConfigurationError(
> +                            "Invalid test cases were selected "
> +                            f"for test suite {selected_test_suite}."
> +                        ) from e
> +                    except ValidationError as e:
> +                        raise ConfigurationError(
> +                            f"Test suite {selected_test_suite} needs to be explicitly configured "
> +                            "in order to be selected."
> +                        ) from e
> +                test_suites_configs.append(config)
> +        else:
> +            # we don't want to modify the original config, so create a copy
> +            test_suites_configs = [
> +                config.model_copy() for config in test_run_config.test_suites.get_configs()
> +            ]
> +
> +        if not test_run_config.skip_smoke_tests:
> +            test_suites_configs[:0] = [TestSuiteConfig.make("smoke_tests")]
> +
> +        return test_suites_configs
> +
>      def _get_test_suites_with_cases(
>          self,
>          test_suite_configs: list[TestSuiteConfig],
> @@ -245,7 +282,9 @@ def _get_test_suites_with_cases(
>
>              test_suites_with_cases.append(
>                  TestSuiteWithCases(
> -                    test_suite_class=test_suite_class, test_cases=selected_test_cases
> +                    test_suite_class=test_suite_class,
> +                    test_cases=selected_test_cases,
> +                    config=test_suite_config,
>                  )
>              )
>          return test_suites_with_cases
> @@ -466,7 +505,9 @@ def _run_test_suite(
>          self._logger.set_stage(
>              DtsStage.test_suite_setup, Path(SETTINGS.output_dir, test_suite_name)
>          )
> -        test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node)
> +        test_suite = test_suite_with_cases.test_suite_class(
> +            sut_node, tg_node, test_suite_with_cases.config
> +        )
>          try:
>              self._logger.info(f"Starting test suite setup: {test_suite_name}")
>              test_suite.set_up_suite()
> diff --git a/dts/framework/settings.py b/dts/framework/settings.py
> index 2e8dedef4f..063f282edf 100644
> --- a/dts/framework/settings.py
> +++ b/dts/framework/settings.py
> @@ -85,9 +85,7 @@
>  from pathlib import Path
>  from typing import Callable
>
> -from pydantic import ValidationError
> -
> -from .config import TestSuiteConfig
> +from .config import TestSuitesConfigs
>  from .exception import ConfigurationError
>  from .utils import DPDKGitTarball, get_commit_id
>
> @@ -114,7 +112,7 @@ class Settings:
>      #:
>      compile_timeout: float = 1200
>      #:
> -    test_suites: list[TestSuiteConfig] = field(default_factory=list)
> +    test_suites: list[tuple[str, list[str]]] = field(default_factory=list)
>      #:
>      re_run: int = 0
>
> @@ -382,7 +380,7 @@ def _get_parser() -> _DTSArgumentParser:
>
>  def _process_test_suites(
>      parser: _DTSArgumentParser, args: list[list[str]]
> -) -> list[TestSuiteConfig]:
> +) -> list[tuple[str, list[str]]]:
>      """Process the given argument to a list of :class:`TestSuiteConfig` to execute.
>
>      Args:
> @@ -398,16 +396,17 @@ def _process_test_suites(
>          # Environment variable in the form of "SUITE1 CASE1 CASE2, SUITE2 CASE1, SUITE3, ..."
>          args = [suite_with_cases.split() for suite_with_cases in args[0][0].split(",")]
>
> -    try:
> -        return [TestSuiteConfig(test_suite, test_cases) for [test_suite, *test_cases] in args]
> -    except ValidationError as e:
> -        print(
> -            "An error has occurred while validating the test suites supplied in the "
> -            f"{'environment variable' if action else 'arguments'}:",
> -            file=sys.stderr,
> -        )
> -        print(e, file=sys.stderr)
> -        sys.exit(1)
> +    available_test_suites = TestSuitesConfigs.available_test_suites()
> +    for test_suite_name, *_ in args:
> +        if test_suite_name not in available_test_suites:
> +            print(
> +                f"The test suite {test_suite_name} supplied in the "
> +                f"{'environment variable' if action else 'arguments'} is invalid.",
> +                file=sys.stderr,
> +            )
> +            sys.exit(1)
> +
> +    return [(test_suite, test_cases) for test_suite, *test_cases in args]
>
>
>  def get_settings() -> Settings:
> diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
> index 5694a2482b..6c10c1e40a 100644
> --- a/dts/framework/test_result.py
> +++ b/dts/framework/test_result.py
> @@ -64,17 +64,7 @@ class is to hold a subset of test cases (which could be all test cases) because
>
>      test_suite_class: type[TestSuite]
>      test_cases: list[FunctionType]
> -
> -    def create_config(self) -> TestSuiteConfig:
> -        """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
> -
> -        Returns:
> -            The :class:`TestSuiteConfig` representation.
> -        """
> -        return TestSuiteConfig(
> -            test_suite=self.test_suite_class.__name__,
> -            test_cases=[test_case.__name__ for test_case in self.test_cases],
> -        )
> +    config: TestSuiteConfig
>
>
>  class Result(Enum):
> diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
> index 972968b036..78e1b4c49a 100644
> --- a/dts/framework/test_suite.py
> +++ b/dts/framework/test_suite.py
> @@ -23,7 +23,7 @@
>  from ipaddress import IPv4Interface, IPv6Interface, ip_interface
>  from pkgutil import iter_modules
>  from types import FunctionType, ModuleType
> -from typing import ClassVar, NamedTuple, Union
> +from typing import ClassVar, Iterable, NamedTuple, Union, get_type_hints
>
>  from pydantic.alias_generators import to_pascal
>  from scapy.layers.inet import IP  # type: ignore[import-untyped]
> @@ -31,6 +31,7 @@
>  from scapy.packet import Packet, Padding  # type: ignore[import-untyped]
>  from typing_extensions import Self
>
> +from framework.config import TestSuiteConfig
>  from framework.testbed_model.port import Port, PortLink
>  from framework.testbed_model.sut_node import SutNode
>  from framework.testbed_model.tg_node import TGNode
> @@ -38,7 +39,7 @@
>      PacketFilteringConfig,
>  )
>
> -from .exception import TestCaseVerifyError
> +from .exception import InternalError, TestCaseVerifyError
>  from .logger import DTSLogger, get_dts_logger
>  from .utils import get_packet_summaries
>
> @@ -78,6 +79,7 @@ class TestSuite:
>      #: Whether the test suite is blocking. A failure of a blocking test suite
>      #: will block the execution of all subsequent test suites in the current build target.
>      is_blocking: ClassVar[bool] = False
> +    config: TestSuiteConfig
>      _logger: DTSLogger
>      _port_links: list[PortLink]
>      _sut_port_ingress: Port
> @@ -93,6 +95,7 @@ def __init__(
>          self,
>          sut_node: SutNode,
>          tg_node: TGNode,
> +        config: TestSuiteConfig,
>      ):
>          """Initialize the test suite testbed information and basic configuration.
>
> @@ -102,9 +105,11 @@ def __init__(
>          Args:
>              sut_node: The SUT node where the test suite will run.
>              tg_node: The TG node where the test suite will run.
> +            config: The test suite configuration.
>          """
>          self.sut_node = sut_node
>          self.tg_node = tg_node
> +        self.config = config
>          self._logger = get_dts_logger(self.__class__.__name__)
>          self._port_links = []
>          self._process_links()
> @@ -467,6 +472,21 @@ def is_test_suite(obj) -> bool:
>
>          raise Exception("class not found in eligible test module")
>
> +    @cached_property
> +    def config_type(self) -> type[TestSuiteConfig]:
> +        """A reference to the test suite's configuration type."""
> +        fields = get_type_hints(self.class_type)
> +        config_type = fields.get("config")
> +        if config_type is None:
> +            raise InternalError(
> +                "Test suite class {self.class_name} is missing the `config` attribute."
> +            )
> +        if not issubclass(config_type, TestSuiteConfig):
> +            raise InternalError(
> +                f"Test suite class {self.class_name} has an invalid configuration type assigned."
> +            )
> +        return config_type
> +
>      @cached_property
>      def test_cases(self) -> list[TestCase]:
>          """A list of all the available test cases."""
> @@ -533,6 +553,14 @@ def discover_all(
>
>          return test_suites
>
> +    def validate_test_cases(self, test_cases_names: Iterable[str]) -> None:
> +        """Validate if the supplied test cases exist in the test suite."""
> +        available_test_cases = map(lambda t: t.name, self.test_cases)
> +        for requested_test_case in test_cases_names:
> +            assert (
> +                requested_test_case in available_test_cases
> +            ), f"{requested_test_case} is not a valid test case for test suite {self.name}."
> +
>
>  AVAILABLE_TEST_SUITES: list[TestSuiteSpec] = TestSuiteSpec.discover_all()
>  """Constant to store all the available, discovered and imported test suites.
> diff --git a/dts/generate-schema.py b/dts/generate-schema.py
> index b41d28492f..d24a67f68e 100755
> --- a/dts/generate-schema.py
> +++ b/dts/generate-schema.py
> @@ -9,7 +9,7 @@
>
>  from pydantic.json_schema import GenerateJsonSchema
>
> -from framework.config import ConfigurationType
> +from framework.config import ConfigurationType, TestSuitesConfigs
>
>  DTS_DIR = os.path.dirname(os.path.realpath(__file__))
>  RELATIVE_PATH_TO_SCHEMA = "framework/config/conf_yaml_schema.json"
> @@ -26,6 +26,8 @@ def generate(self, schema, mode="validation"):
>
>
>  try:
> +    TestSuitesConfigs.fix_custom_config_annotations()
> +
>      path = os.path.join(DTS_DIR, RELATIVE_PATH_TO_SCHEMA)
>
>      with open(path, "w") as schema_file:
> diff --git a/dts/generate-test-mappings.py b/dts/generate-test-mappings.py
> new file mode 100755
> index 0000000000..d076ad0afe
> --- /dev/null
> +++ b/dts/generate-test-mappings.py
> @@ -0,0 +1,132 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2024 Arm Limited
> +
> +"""Test Suites to Configuration mappings generation script."""
> +
> +import os
> +from collections import defaultdict
> +from textwrap import indent
> +from typing import Iterable
> +
> +from framework.config.test_suite import BaseTestSuitesConfigs, TestSuiteConfig
> +from framework.exception import InternalError
> +from framework.test_suite import AVAILABLE_TEST_SUITES, TestSuiteSpec
> +
> +DTS_DIR = os.path.dirname(os.path.realpath(__file__))
> +SCRIPT_FILE_NAME = os.path.basename(__file__)
> +
> +FRAMEWORK_IMPORTS = [BaseTestSuitesConfigs, TestSuiteConfig]
> +
> +RELATIVE_PATH_TO_GENERATED_FILE = "framework/config/generated.py"
> +SMOKE_TESTS_SUITE_NAME = "smoke_tests"
> +CUSTOM_CONFIG_TYPES_VAR_NAME = "CUSTOM_CONFIG_TYPES"
> +CUSTOM_CONFIG_LINKING_FUNCTION_NAME = "fix_custom_config_annotations"
> +CUSTOM_CONFIG_LINKING_FUNCTION_DOCSTRING = [
> +    '"""Fixes the custom config types annotations.',
> +    "",
> +    f"Moreover it also fills `{CUSTOM_CONFIG_TYPES_VAR_NAME}` with all the custom config types.",
> +    '"""',
> +]
> +TEST_SUITES_CONFIG_CLASS_NAME = "TestSuitesConfigs"
> +TEST_SUITES_CONFIG_CLASS_DOCSTRING = [
> +    '"""Configuration mapping class to select and configure the test suites.',
> +    "",
> +    "Before using this class, the custom configuration type annotations need to be fixed.",
> +    f"To do so, you need to call the `{CUSTOM_CONFIG_LINKING_FUNCTION_NAME}` method.",
> +    '"""',
> +]
> +
> +
> +GENERATED_FILE_HEADER = f"""# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2024 The DPDK contributors
> +# This file is automatically generated by {SCRIPT_FILE_NAME}.
> +# Do NOT modify this file manually.
> +
> +\"\"\"Generated file containing the links between the test suites and the configuration.\"\"\"
> +"""
> +
> +
> +def join(lines: Iterable[str]) -> str:
> +    """Join list of strings into text lines."""
> +    return "\n".join(lines)
> +
> +
> +def join_and_indent(lines: Iterable[str], indentation_level=1, indentation_spaces=4) -> str:
> +    """Join list of strings into indented text lines."""
> +    return "\n".join([indent(line, " " * indentation_level * indentation_spaces) for line in lines])
> +
> +
> +def format_attributes_types(test_suite_spec: TestSuiteSpec):
> +    """Format the config type into the respective configuration class field attribute type."""
> +    config_type = test_suite_spec.config_type.__name__
> +    if config_type != TestSuiteConfig.__name__:
> +        config_type = f'"{config_type}"'
> +    return f"Optional[{config_type}]"
> +
> +
> +try:
> +    framework_imports: dict[str, list[str]] = defaultdict(list)
> +    for _import in FRAMEWORK_IMPORTS:
> +        framework_imports[_import.__module__].append(_import.__name__)
> +    formatted_framework_imports = sorted(
> +        [
> +            f"from {module} import {', '.join(sorted(imports))}"
> +            for module, imports in framework_imports.items()
> +        ]
> +    )
> +
> +    test_suites = [
> +        test_suite_spec
> +        for test_suite_spec in AVAILABLE_TEST_SUITES
> +        if test_suite_spec.name != SMOKE_TESTS_SUITE_NAME
> +    ]
> +
> +    custom_configs = [t for t in test_suites if t.config_type is not TestSuiteConfig]
> +
> +    custom_config_imports = [
> +        f"from {t.module_type.__name__} import {t.config_type.__name__}" for t in custom_configs
> +    ]
> +
> +    test_suites_attributes = [f"{t.name}: {format_attributes_types(t)} = None" for t in test_suites]
> +
> +    custom_config_assignments = [
> +        f'{CUSTOM_CONFIG_TYPES_VAR_NAME}["{t.name}"] = {t.config_type.__name__}'
> +        for t in custom_configs
> +    ]
> +
> +    generated_file_contents = f"""{GENERATED_FILE_HEADER}
> +from typing import TYPE_CHECKING, Optional
> +
> +{join(formatted_framework_imports)}
> +
> +if TYPE_CHECKING:
> +{join_and_indent(custom_config_imports)}
> +
> +
> +{CUSTOM_CONFIG_TYPES_VAR_NAME}: dict[str, type[{TestSuiteConfig.__name__}]] = {'{}'}
> +
> +
> +class {TEST_SUITES_CONFIG_CLASS_NAME}({BaseTestSuitesConfigs.__name__}):
> +{join_and_indent(TEST_SUITES_CONFIG_CLASS_DOCSTRING)}
> +
> +{join_and_indent(test_suites_attributes)}
> +
> +    @classmethod
> +    def {CUSTOM_CONFIG_LINKING_FUNCTION_NAME}(cls):
> +{join_and_indent(CUSTOM_CONFIG_LINKING_FUNCTION_DOCSTRING, indentation_level=2)}
> +{join_and_indent(custom_config_imports, indentation_level=2)}
> +
> +{join_and_indent(custom_config_assignments, indentation_level=2)}
> +
> +        cls.model_rebuild()
> +"""
> +
> +    path = os.path.join(DTS_DIR, RELATIVE_PATH_TO_GENERATED_FILE)
> +
> +    with open(path, "w") as generated_file:
> +        generated_file.write(generated_file_contents)
> +
> +    print("Test suites to configuration mappings generated successfully!")
> +except Exception as e:
> +    raise InternalError("Failed to generate test suites to configuration mappings.") from e
> diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
> index d958f99030..1723c123bc 100644
> --- a/dts/tests/TestSuite_hello_world.py
> +++ b/dts/tests/TestSuite_hello_world.py
> @@ -7,6 +7,7 @@
>  No other EAL parameters apart from cores are used.
>  """
>
> +from framework.config import TestSuiteConfig
>  from framework.remote_session.dpdk_shell import compute_eal_params
>  from framework.test_suite import TestSuite
>  from framework.testbed_model.cpu import (
> @@ -16,9 +17,18 @@
>  )
>
>
> +class HelloWorldConfig(TestSuiteConfig):
> +    """Example custom configuration for the `TestHelloWorld` test suite."""
> +
> +    #: Timeout for the DPDK apps
> +    timeout: int = 50
> +
> +
>  class TestHelloWorld(TestSuite):
>      """DPDK hello world app test suite."""
>
> +    config: HelloWorldConfig
> +
>      def set_up_suite(self) -> None:
>          """Set up the test suite.
>
> @@ -59,7 +69,7 @@ def test_hello_world_all_cores(self) -> None:
>          eal_para = compute_eal_params(
>              self.sut_node, lcore_filter_specifier=LogicalCoreList(self.sut_node.lcores)
>          )
> -        result = self.sut_node.run_dpdk_app(self.app_helloworld_path, eal_para, 50)
> +        result = self.sut_node.run_dpdk_app(self.app_helloworld_path, eal_para, self.config.timeout)
>          for lcore in self.sut_node.lcores:
>              self.verify(
>                  f"hello from core {int(lcore)}" in result.stdout,
> diff --git a/dts/tests/__init__.py b/dts/tests/__init__.py
> new file mode 100644
> index 0000000000..a300eb26fc
> --- /dev/null
> +++ b/dts/tests/__init__.py
> @@ -0,0 +1,7 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2024 Arm Limited
> +
> +"""Test suites.
> +
> +This package contains all the available test suites in DTS.
> +"""
> --
> 2.34.1
>

^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH v2 0/4] dts: add per-test-suite configuration
  2024-09-06 16:13 [PATCH] dts: add per-test-suite configuration Luca Vizzarro
  2024-09-27 17:45 ` Jeremy Spewock
@ 2024-11-08 13:38 ` Luca Vizzarro
  2024-11-08 13:38   ` [PATCH v2 1/4] dts: add tests package to API docs Luca Vizzarro
                     ` (3 more replies)
  2024-11-08 13:45 ` [PATCH v3 0/4] dts: add per-test-suite configuration Luca Vizzarro
  2 siblings, 4 replies; 27+ messages in thread
From: Luca Vizzarro @ 2024-11-08 13:38 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Patrick Robb, Luca Vizzarro

Hi there,

sending a v2 for the per-test-suite configuration additions.

v2:
- rebase
- added test suites to doc pages
- doc fixes
- updated the custom config procedure to using tests.config
- simplified the generated file
- implemented the generator in dts-check-format

Best regards,
Luca Vizzarro

Depends-on: series-33871 ("dts: Pydantic configuration")

Luca Vizzarro (4):
  dts: add tests package to API docs
  dts: fix smoke tests docstring
  dts: add per-test-suite configuration
  dts: update autodoc sorting order

 devtools/dts-check-format.sh                  |  28 +++-
 devtools/dts-generate-tests-mappings.py       | 117 +++++++++++++
 doc/api/dts/framework.config.generated.rst    |   8 +
 doc/api/dts/framework.config.rst              |   7 +
 doc/api/dts/framework.config.test_suite.rst   |   8 +
 doc/api/dts/index.rst                         |   1 +
 doc/api/dts/tests.TestSuite_hello_world.rst   |   9 +
 doc/api/dts/tests.TestSuite_os_udp.rst        |   9 +
 .../tests.TestSuite_pmd_buffer_scatter.rst    |   9 +
 doc/api/dts/tests.TestSuite_smoke_tests.rst   |   9 +
 doc/api/dts/tests.TestSuite_vlan.rst          |  10 ++
 doc/api/dts/tests.config.rst                  |   9 +
 doc/api/dts/tests.rst                         |  20 +++
 doc/guides/conf.py                            |   1 +
 doc/guides/tools/dts.rst                      |  23 +++
 dts/conf.yaml                                 |   4 +-
 dts/framework/config/__init__.py              |  76 +--------
 dts/framework/config/generated.py             |  25 +++
 dts/framework/config/test_suite.py            | 154 ++++++++++++++++++
 dts/framework/runner.py                       |  69 ++++++--
 dts/framework/settings.py                     |  30 ++--
 dts/framework/test_result.py                  |  15 +-
 dts/framework/test_suite.py                   |  22 ++-
 dts/tests/TestSuite_hello_world.py            |   5 +-
 dts/tests/TestSuite_smoke_tests.py            |   2 -
 dts/tests/__init__.py                         |   7 +
 dts/tests/config.py                           |  20 +++
 27 files changed, 580 insertions(+), 117 deletions(-)
 create mode 100755 devtools/dts-generate-tests-mappings.py
 create mode 100644 doc/api/dts/framework.config.generated.rst
 create mode 100644 doc/api/dts/framework.config.test_suite.rst
 create mode 100644 doc/api/dts/tests.TestSuite_hello_world.rst
 create mode 100644 doc/api/dts/tests.TestSuite_os_udp.rst
 create mode 100644 doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
 create mode 100644 doc/api/dts/tests.TestSuite_smoke_tests.rst
 create mode 100644 doc/api/dts/tests.TestSuite_vlan.rst
 create mode 100644 doc/api/dts/tests.config.rst
 create mode 100644 doc/api/dts/tests.rst
 create mode 100644 dts/framework/config/generated.py
 create mode 100644 dts/framework/config/test_suite.py
 create mode 100644 dts/tests/__init__.py
 create mode 100644 dts/tests/config.py

-- 
2.43.0


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH v2 1/4] dts: add tests package to API docs
  2024-11-08 13:38 ` [PATCH v2 0/4] " Luca Vizzarro
@ 2024-11-08 13:38   ` Luca Vizzarro
  2024-11-08 13:38   ` [PATCH v2 2/4] dts: fix smoke tests docstring Luca Vizzarro
                     ` (2 subsequent siblings)
  3 siblings, 0 replies; 27+ messages in thread
From: Luca Vizzarro @ 2024-11-08 13:38 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Patrick Robb, Luca Vizzarro

The test suites are not documented in the API as their respective
documentation files are missing.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 doc/api/dts/index.rst                         |  1 +
 doc/api/dts/tests.TestSuite_hello_world.rst   |  9 +++++++++
 doc/api/dts/tests.TestSuite_os_udp.rst        |  9 +++++++++
 .../tests.TestSuite_pmd_buffer_scatter.rst    |  9 +++++++++
 doc/api/dts/tests.TestSuite_smoke_tests.rst   |  9 +++++++++
 doc/api/dts/tests.TestSuite_vlan.rst          | 10 ++++++++++
 doc/api/dts/tests.rst                         | 19 +++++++++++++++++++
 7 files changed, 66 insertions(+)
 create mode 100644 doc/api/dts/tests.TestSuite_hello_world.rst
 create mode 100644 doc/api/dts/tests.TestSuite_os_udp.rst
 create mode 100644 doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
 create mode 100644 doc/api/dts/tests.TestSuite_smoke_tests.rst
 create mode 100644 doc/api/dts/tests.TestSuite_vlan.rst
 create mode 100644 doc/api/dts/tests.rst

diff --git a/doc/api/dts/index.rst b/doc/api/dts/index.rst
index 534512dc17..db05fd37d5 100644
--- a/doc/api/dts/index.rst
+++ b/doc/api/dts/index.rst
@@ -14,6 +14,7 @@ Packages
    :includehidden:
    :maxdepth: 1
 
+   tests
    framework.testbed_model
    framework.remote_session
    framework.params
diff --git a/doc/api/dts/tests.TestSuite_hello_world.rst b/doc/api/dts/tests.TestSuite_hello_world.rst
new file mode 100644
index 0000000000..f92786996f
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_hello_world.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+hello_world Test Suite
+======================
+
+.. automodule:: tests.TestSuite_hello_world
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.TestSuite_os_udp.rst b/doc/api/dts/tests.TestSuite_os_udp.rst
new file mode 100644
index 0000000000..121a653670
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_os_udp.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+os_udp Test Suite
+=================
+
+.. automodule:: tests.TestSuite_os_udp
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst b/doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
new file mode 100644
index 0000000000..b6571b24f5
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+pmd_buffer_scatter Test Suite
+=============================
+
+.. automodule:: tests.TestSuite_pmd_buffer_scatter
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.TestSuite_smoke_tests.rst b/doc/api/dts/tests.TestSuite_smoke_tests.rst
new file mode 100644
index 0000000000..5b11aa7f48
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_smoke_tests.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+smoke_tests Test Suite
+======================
+
+.. automodule:: tests.TestSuite_smoke_tests
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.TestSuite_vlan.rst b/doc/api/dts/tests.TestSuite_vlan.rst
new file mode 100644
index 0000000000..86a29538bb
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_vlan.rst
@@ -0,0 +1,10 @@
+
+.. SPDX-License-Identifier: BSD-3-Clause
+
+vlan Test Suite
+===============
+
+.. automodule:: tests.TestSuite_vlan
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.rst b/doc/api/dts/tests.rst
new file mode 100644
index 0000000000..0c136b4bb0
--- /dev/null
+++ b/doc/api/dts/tests.rst
@@ -0,0 +1,19 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+tests - Test Suites Package
+==============================
+
+.. automodule:: tests
+   :members:
+   :show-inheritance:
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   tests.TestSuite_hello_world
+   tests.TestSuite_os_udp
+   tests.TestSuite_pmd_buffer_scatter
+   tests.TestSuite_smoke_tests
+   tests.TestSuite_vlan
+
-- 
2.43.0


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH v2 2/4] dts: fix smoke tests docstring
  2024-11-08 13:38 ` [PATCH v2 0/4] " Luca Vizzarro
  2024-11-08 13:38   ` [PATCH v2 1/4] dts: add tests package to API docs Luca Vizzarro
@ 2024-11-08 13:38   ` Luca Vizzarro
  2024-11-12 20:03     ` Dean Marx
  2024-11-08 13:38   ` [PATCH v2 3/4] dts: add per-test-suite configuration Luca Vizzarro
  2024-11-08 13:38   ` [PATCH v2 4/4] dts: update autodoc sorting order Luca Vizzarro
  3 siblings, 1 reply; 27+ messages in thread
From: Luca Vizzarro @ 2024-11-08 13:38 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Patrick Robb, Luca Vizzarro

Sphinx autodoc complains of is_blocking being duplicated. This is the
case as the parent already holds a docstring for this attribute. Remove
the duplication.

Fixes: 6ef07151aac4 ("dts: update docstrings")

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 dts/tests/TestSuite_smoke_tests.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
index bc3a2a6bf9..f564ac82b4 100644
--- a/dts/tests/TestSuite_smoke_tests.py
+++ b/dts/tests/TestSuite_smoke_tests.py
@@ -30,8 +30,6 @@ class TestSmokeTests(TestSuite):
     The infrastructure also needs to be tested, as that is also used by all other test suites.
 
     Attributes:
-        is_blocking: This test suite will block the execution of all other test suites
-            in the test run after it.
         nics_in_node: The NICs present on the SUT node.
     """
 
-- 
2.43.0


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH v2 3/4] dts: add per-test-suite configuration
  2024-11-08 13:38 ` [PATCH v2 0/4] " Luca Vizzarro
  2024-11-08 13:38   ` [PATCH v2 1/4] dts: add tests package to API docs Luca Vizzarro
  2024-11-08 13:38   ` [PATCH v2 2/4] dts: fix smoke tests docstring Luca Vizzarro
@ 2024-11-08 13:38   ` Luca Vizzarro
  2024-11-08 13:38   ` [PATCH v2 4/4] dts: update autodoc sorting order Luca Vizzarro
  3 siblings, 0 replies; 27+ messages in thread
From: Luca Vizzarro @ 2024-11-08 13:38 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Patrick Robb, Luca Vizzarro

Allow test suites to be configured individually. Moreover enable them to
implement their own custom configuration.

This solution adds some new complexity to DTS, which is generated source
code. In order to ensure strong typing, the test suites and their custom
configurations need to be linked in the main configuration class.
Unfortunately, this is not feasible during runtime as it will incur in
circular dependencies. Generating the links appear to be the most
straightforward approach.

This commit also brings a new major change to the configuration schema.
Test suites are no longer defined as a list of strings, like:

    test_suites:
    - hello_world
    - pmd_buffer_scatter

but as mapping of mappings or strings:

    test_suites:
      hello_world: {} # any custom fields or test cases can be set here
      pmd_buffer_scatter: all # "all" defines all the test cases, or
                              # they can individually be set separated
                              # by a space

Not defining the `test_cases` field in the configuration is equivalent
to `all`, therefore the definitions for either test suite above are
also equivalent.

Creating the __init__.py file under the tests folder, allows it to be
picked up as a package. This is a mypy requirement to import the tests
from within the framework.

Bugzilla ID: 1375

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 devtools/dts-check-format.sh                |  28 +++-
 devtools/dts-generate-tests-mappings.py     | 117 +++++++++++++++
 doc/api/dts/framework.config.generated.rst  |   8 +
 doc/api/dts/framework.config.rst            |   7 +
 doc/api/dts/framework.config.test_suite.rst |   8 +
 doc/api/dts/tests.config.rst                |   9 ++
 doc/api/dts/tests.rst                       |   1 +
 doc/guides/tools/dts.rst                    |  23 +++
 dts/conf.yaml                               |   4 +-
 dts/framework/config/__init__.py            |  76 +---------
 dts/framework/config/generated.py           |  25 ++++
 dts/framework/config/test_suite.py          | 154 ++++++++++++++++++++
 dts/framework/runner.py                     |  69 +++++++--
 dts/framework/settings.py                   |  30 ++--
 dts/framework/test_result.py                |  15 +-
 dts/framework/test_suite.py                 |  22 ++-
 dts/tests/TestSuite_hello_world.py          |   5 +-
 dts/tests/__init__.py                       |   7 +
 dts/tests/config.py                         |  20 +++
 19 files changed, 513 insertions(+), 115 deletions(-)
 create mode 100755 devtools/dts-generate-tests-mappings.py
 create mode 100644 doc/api/dts/framework.config.generated.rst
 create mode 100644 doc/api/dts/framework.config.test_suite.rst
 create mode 100644 doc/api/dts/tests.config.rst
 create mode 100644 dts/framework/config/generated.py
 create mode 100644 dts/framework/config/test_suite.py
 create mode 100644 dts/tests/__init__.py
 create mode 100644 dts/tests/config.py

diff --git a/devtools/dts-check-format.sh b/devtools/dts-check-format.sh
index 3f43e17e88..7a440fc0cf 100755
--- a/devtools/dts-check-format.sh
+++ b/devtools/dts-check-format.sh
@@ -2,6 +2,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2022 University of New Hampshire
 # Copyright(c) 2023 PANTHEON.tech s.r.o.
+# Copyright(c) 2024 Arm Limited
 
 usage() {
 	echo "Usage: $(basename $0) [options] [directory]"
@@ -13,15 +14,19 @@ usage() {
 format=true
 lint=true
 typecheck=true
+generate=true
 
 # Comments after args serve as documentation; must be present
-while getopts "hflt" arg; do
+while getopts "hgflt" arg; do
 	case $arg in
 	h) # Display this message
-		echo 'Run formatting and linting programs for DTS.'
+		echo 'Run generating, formatting and linting programs for DTS.'
 		usage
 		exit 0
 		;;
+	g) # Don't run code generator
+		generate=false
+		;;
 	f) # Don't run formatters
 		format=false
 		;;
@@ -48,7 +53,22 @@ heading() {
 
 errors=0
 
+if $generate; then
+	heading "Generating test suites to configuration mappings"
+	if command -v python3 > /dev/null; then
+		../devtools/dts-generate-tests-mappings.py
+		errors=$((errors + $?))
+	else
+		echo "python3 not found, unable to run generator"
+		errros=$((errors + 1))
+	fi
+fi
+
 if $format; then
+	if $generate; then
+		echo
+	fi
+
 	if command -v git > /dev/null; then
 		if git rev-parse --is-inside-work-tree >&-; then
 			heading "Formatting in $directory/"
@@ -85,7 +105,7 @@ if $format; then
 fi
 
 if $lint; then
-	if $format; then
+	if $generate || $format; then
 		echo
 	fi
 	heading "Linting in $directory/"
@@ -99,7 +119,7 @@ if $lint; then
 fi
 
 if $typecheck; then
-	if $format || $lint; then
+	if $generate || $format || $lint; then
 		echo
 	fi
 	heading "Checking types in $directory/"
diff --git a/devtools/dts-generate-tests-mappings.py b/devtools/dts-generate-tests-mappings.py
new file mode 100755
index 0000000000..26ecc1018c
--- /dev/null
+++ b/devtools/dts-generate-tests-mappings.py
@@ -0,0 +1,117 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""DTS Test Suites to Configuration mappings generation script."""
+
+import os
+import sys
+from collections import defaultdict
+from pathlib import Path
+from textwrap import indent
+from typing import Iterable
+
+DTS_DIR = Path(__file__).parent.joinpath("..", "dts").resolve()
+SCRIPT_FILE_NAME = Path(__file__).relative_to(Path(__file__).parent.parent)
+
+sys.path.append(str(DTS_DIR))
+
+from framework.config.test_suite import BaseTestSuitesConfigs, TestSuiteConfig
+from framework.exception import InternalError
+from framework.test_suite import AVAILABLE_TEST_SUITES, TestSuiteSpec
+
+FRAMEWORK_IMPORTS = [BaseTestSuitesConfigs, TestSuiteConfig]
+
+RELATIVE_PATH_TO_GENERATED_FILE = "framework/config/generated.py"
+SMOKE_TESTS_SUITE_NAME = "smoke_tests"
+CUSTOM_CONFIG_TYPES_VAR_NAME = "CUSTOM_CONFIG_TYPES"
+CUSTOM_CONFIG_TYPES_VAR_DOCSTRING = [
+    "#: Mapping of test suites to their corresponding custom configuration objects if any."
+]
+TEST_SUITES_CONFIG_CLASS_NAME = "TestSuitesConfigs"
+TEST_SUITES_CONFIG_CLASS_DOCSTRING = [
+    '"""Configuration mapping class to select and configure the test suites."""',
+]
+
+
+GENERATED_FILE_HEADER = f"""# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 The DPDK contributors
+# This file is automatically generated by {SCRIPT_FILE_NAME}.
+# Do NOT modify this file manually.
+
+\"\"\"Generated file containing the links between the test suites and the configuration.\"\"\"
+"""
+
+
+def join(lines: Iterable[str]) -> str:
+    """Join list of strings into text lines."""
+    return "\n".join(lines)
+
+
+def join_and_indent(lines: Iterable[str], indentation_level=1, indentation_spaces=4) -> str:
+    """Join list of strings into indented text lines."""
+    return "\n".join([indent(line, " " * indentation_level * indentation_spaces) for line in lines])
+
+
+def format_attributes_types(test_suite_spec: TestSuiteSpec):
+    """Format the config type into the respective configuration class field attribute type."""
+    config_type = test_suite_spec.config_obj.__name__
+    return f"Optional[{config_type}]"
+
+
+try:
+    framework_imports: dict[str, list[str]] = defaultdict(list)
+    for _import in FRAMEWORK_IMPORTS:
+        framework_imports[_import.__module__].append(_import.__name__)
+    formatted_framework_imports = sorted(
+        [
+            f"from {module} import {', '.join(sorted(imports))}"
+            for module, imports in framework_imports.items()
+        ]
+    )
+
+    test_suites = [
+        test_suite_spec
+        for test_suite_spec in AVAILABLE_TEST_SUITES
+        if test_suite_spec.name != SMOKE_TESTS_SUITE_NAME
+    ]
+
+    custom_configs = [t for t in test_suites if t.config_obj is not TestSuiteConfig]
+
+    custom_config_imports = [
+        f"from {t.config_obj.__module__} import {t.config_obj.__name__}" for t in custom_configs
+    ]
+
+    test_suites_attributes = [f"{t.name}: {format_attributes_types(t)} = None" for t in test_suites]
+
+    custom_config_mappings = [f'"{t.name}": {t.config_obj.__name__},' for t in custom_configs]
+
+    generated_file_contents = f"""{GENERATED_FILE_HEADER}
+from typing import Optional
+
+{join(formatted_framework_imports)}
+
+{join(custom_config_imports)}
+
+{join(CUSTOM_CONFIG_TYPES_VAR_DOCSTRING)}
+{CUSTOM_CONFIG_TYPES_VAR_NAME}: dict[str, type[{TestSuiteConfig.__name__}]] = {'{'}
+{join_and_indent(custom_config_mappings)}
+{'}'}
+
+
+class {TEST_SUITES_CONFIG_CLASS_NAME}({BaseTestSuitesConfigs.__name__}):
+{join_and_indent(TEST_SUITES_CONFIG_CLASS_DOCSTRING)}
+
+{join_and_indent(test_suites_attributes)}
+"""
+
+    path = os.path.join(DTS_DIR, RELATIVE_PATH_TO_GENERATED_FILE)
+
+    with open(path, "w") as generated_file:
+        generated_file.write(generated_file_contents)
+
+    print("Test suites to configuration mappings generated successfully!")
+except Exception as e:
+    raise InternalError(
+        "Failed to generate test suites to configuration mappings."
+    ) from e
diff --git a/doc/api/dts/framework.config.generated.rst b/doc/api/dts/framework.config.generated.rst
new file mode 100644
index 0000000000..5dfa9342f0
--- /dev/null
+++ b/doc/api/dts/framework.config.generated.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+generated - Generated Test Suite Configurations
+===============================================
+
+.. automodule:: framework.config.generated
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/framework.config.rst b/doc/api/dts/framework.config.rst
index cc266276c1..217fe026c4 100644
--- a/doc/api/dts/framework.config.rst
+++ b/doc/api/dts/framework.config.rst
@@ -6,3 +6,10 @@ config - Configuration Package
 .. automodule:: framework.config
    :members:
    :show-inheritance:
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   framework.config.generated
+   framework.config.test_suite
diff --git a/doc/api/dts/framework.config.test_suite.rst b/doc/api/dts/framework.config.test_suite.rst
new file mode 100644
index 0000000000..d59dcf5d6e
--- /dev/null
+++ b/doc/api/dts/framework.config.test_suite.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+test_suite - Test Suite Configuration Definitions
+=================================================
+
+.. automodule:: framework.config.test_suite
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.config.rst b/doc/api/dts/tests.config.rst
new file mode 100644
index 0000000000..ce3d9df868
--- /dev/null
+++ b/doc/api/dts/tests.config.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+Test Suites Configurations
+==========================
+
+.. automodule:: tests.config
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.rst b/doc/api/dts/tests.rst
index 0c136b4bb0..7fc25f2123 100644
--- a/doc/api/dts/tests.rst
+++ b/doc/api/dts/tests.rst
@@ -11,6 +11,7 @@ tests - Test Suites Package
    :hidden:
    :maxdepth: 1
 
+   tests.config
    tests.TestSuite_hello_world
    tests.TestSuite_os_udp
    tests.TestSuite_pmd_buffer_scatter
diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index f4e297413d..4e63601b19 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -408,6 +408,29 @@ There are four types of methods that comprise a test suite:
    should be implemented in the ``SutNode`` class (and the underlying classes that ``SutNode`` uses)
    and used by the test suite via the ``sut_node`` field.
 
+The test suites can also implement their own custom configuration fields. This can be achieved by
+creating a new test suite config file which inherits from ``TestSuiteConfig`` defined in
+``dts/framework/config/test_suite.py``. So that this new custom configuration class is used, the
+test suite class must override the ``config`` attribute annotation with your new class, for example:
+
+.. code:: python
+
+   # place this under tests/config.py to avoid circular dependencies
+   class CustomConfig(TestSuiteConfig):
+      my_custom_field: int = 10
+
+   # place this under tests/TestSuite_my_new_test_suite.py
+   class TestMyNewTestSuite(TestSuite):
+      config: CustomConfig
+
+Finally, the test suites and the custom configuration files need to linked in the global configuration.
+This can be easily achieved by running the ``dts/generate-test-mappings.py``, e.g.:
+
+.. code-block:: console
+
+   $ poetry shell
+   (dts-py3.10) $ ./generate-test-mappings.py
+
 
 .. _dts_dev_tools:
 
diff --git a/dts/conf.yaml b/dts/conf.yaml
index 2496262854..377304dddf 100644
--- a/dts/conf.yaml
+++ b/dts/conf.yaml
@@ -28,8 +28,8 @@ test_runs:
     func: true # enable functional testing
     skip_smoke_tests: false # optional
     test_suites: # the following test suites will be run in their entirety
-      - hello_world
-      - os_udp
+      hello_world: all
+      os_udp: all
     # The machine running the DPDK test executable
     system_under_test_node:
       node_name: "SUT 1"
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 82113a6257..0ac7ab5c46 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -32,11 +32,13 @@
       and makes it thread safe should we ever want to move in that direction.
 """
 
+# pylama:ignore=W0611
+
 import tarfile
 from enum import Enum, auto, unique
 from functools import cached_property
 from pathlib import Path, PurePath
-from typing import TYPE_CHECKING, Annotated, Any, Literal, NamedTuple
+from typing import Annotated, Literal, NamedTuple
 
 import yaml
 from pydantic import (
@@ -52,8 +54,7 @@
 from framework.exception import ConfigurationError
 from framework.utils import REGEX_FOR_PCI_ADDRESS, StrEnum
 
-if TYPE_CHECKING:
-    from framework.test_suite import TestSuiteSpec
+from .generated import TestSuitesConfigs
 
 
 class FrozenModel(BaseModel):
@@ -382,69 +383,6 @@ class DPDKUncompiledBuildConfiguration(BaseDPDKBuildConfiguration):
 DPDKBuildConfiguration = DPDKPrecompiledBuildConfiguration | DPDKUncompiledBuildConfiguration
 
 
-class TestSuiteConfig(FrozenModel):
-    """Test suite configuration.
-
-    Information about a single test suite to be executed. This can also be represented as a string
-    instead of a mapping, example:
-
-    .. code:: yaml
-
-        test_runs:
-        - test_suites:
-            # As string representation:
-            - hello_world # test all of `hello_world`, or
-            - hello_world hello_world_single_core # test only `hello_world_single_core`
-            # or as model fields:
-            - test_suite: hello_world
-              test_cases: [hello_world_single_core] # without this field all test cases are run
-    """
-
-    #: The name of the test suite module without the starting ``TestSuite_``.
-    test_suite_name: str = Field(alias="test_suite")
-    #: The names of test cases from this test suite to execute. If empty, all test cases will be
-    #: executed.
-    test_cases_names: list[str] = Field(default_factory=list, alias="test_cases")
-
-    @cached_property
-    def test_suite_spec(self) -> "TestSuiteSpec":
-        """The specification of the requested test suite."""
-        from framework.test_suite import find_by_name
-
-        test_suite_spec = find_by_name(self.test_suite_name)
-        assert (
-            test_suite_spec is not None
-        ), f"{self.test_suite_name} is not a valid test suite module name."
-        return test_suite_spec
-
-    @model_validator(mode="before")
-    @classmethod
-    def convert_from_string(cls, data: Any) -> Any:
-        """Convert the string representation of the model into a valid mapping."""
-        if isinstance(data, str):
-            [test_suite, *test_cases] = data.split()
-            return dict(test_suite=test_suite, test_cases=test_cases)
-        return data
-
-    @model_validator(mode="after")
-    def validate_names(self) -> Self:
-        """Validate the supplied test suite and test cases names.
-
-        This validator relies on the cached property `test_suite_spec` to run for the first
-        time in this call, therefore triggering the assertions if needed.
-        """
-        available_test_cases = map(
-            lambda t: t.name, self.test_suite_spec.class_obj.get_test_cases()
-        )
-        for requested_test_case in self.test_cases_names:
-            assert requested_test_case in available_test_cases, (
-                f"{requested_test_case} is not a valid test case "
-                f"of test suite {self.test_suite_name}."
-            )
-
-        return self
-
-
 class TestRunSUTNodeConfiguration(FrozenModel):
     """The SUT node configuration of a test run."""
 
@@ -469,8 +407,8 @@ class TestRunConfiguration(FrozenModel):
     func: bool
     #: Whether to skip smoke tests.
     skip_smoke_tests: bool = False
-    #: The names of test suites and/or test cases to execute.
-    test_suites: list[TestSuiteConfig] = Field(min_length=1)
+    #: The test suites to be selected and/or configured.
+    test_suites: TestSuitesConfigs
     #: The SUT node configuration to use in this test run.
     system_under_test_node: TestRunSUTNodeConfiguration
     #: The TG node name to use in this test run.
@@ -602,6 +540,6 @@ def load_config(config_file_path: Path) -> Configuration:
         config_data = yaml.safe_load(f)
 
     try:
-        return Configuration.model_validate(config_data)
+        return Configuration.model_validate(config_data, context={})
     except ValidationError as e:
         raise ConfigurationError("failed to load the supplied configuration") from e
diff --git a/dts/framework/config/generated.py b/dts/framework/config/generated.py
new file mode 100644
index 0000000000..cc4a539987
--- /dev/null
+++ b/dts/framework/config/generated.py
@@ -0,0 +1,25 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 The DPDK contributors
+# This file is automatically generated by devtools/dts-generate-tests-mappings.py.
+# Do NOT modify this file manually.
+
+"""Generated file containing the links between the test suites and the configuration."""
+
+from typing import Optional
+
+from framework.config.test_suite import BaseTestSuitesConfigs, TestSuiteConfig
+from tests.config import HelloWorldConfig
+
+#: Mapping of test suites to their corresponding custom configuration objects if any.
+CUSTOM_CONFIG_TYPES: dict[str, type[TestSuiteConfig]] = {
+    "hello_world": HelloWorldConfig,
+}
+
+
+class TestSuitesConfigs(BaseTestSuitesConfigs):
+    """Configuration mapping class to select and configure the test suites."""
+
+    hello_world: Optional[HelloWorldConfig] = None
+    os_udp: Optional[TestSuiteConfig] = None
+    pmd_buffer_scatter: Optional[TestSuiteConfig] = None
+    vlan: Optional[TestSuiteConfig] = None
diff --git a/dts/framework/config/test_suite.py b/dts/framework/config/test_suite.py
new file mode 100644
index 0000000000..863052cbc1
--- /dev/null
+++ b/dts/framework/config/test_suite.py
@@ -0,0 +1,154 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Test suites configuration module.
+
+Test suites can inherit :class:`TestSuiteConfig` to create their own custom configuration.
+By doing so, the test suite class must also override the annotation of the field
+`~framework.test_suite.TestSuite.config` to use their custom configuration type.
+"""
+
+from functools import cached_property
+from typing import TYPE_CHECKING, Any, Iterable
+
+from pydantic import (
+    BaseModel,
+    ConfigDict,
+    Field,
+    ValidationInfo,
+    field_validator,
+    model_validator,
+)
+from typing_extensions import Self
+
+if TYPE_CHECKING:
+    from framework.test_suite import TestSuiteSpec
+
+
+class TestSuiteConfig(BaseModel):
+    """Test suite configuration base model.
+
+    By default the configuration of a generic test suite does not contain any attributes. Any test
+    suite should inherit this class to create their own custom configuration. Finally override the
+    type of the :attr:`~TestSuite.config` to use the newly created one.
+
+    If no custom fields require setting, this can also be represented as a string instead of
+    a mapping, example:
+
+    .. code:: yaml
+
+        test_runs:
+        - test_suites:
+            # As string representation:
+            hello_world: all # test all of `hello_world`, or
+            hello_world: hello_world_single_core # test only `hello_world_single_core`
+            # or as a mapping of the model's fields:
+            hello_world:
+              test_cases: [hello_world_single_core] # without this field all test cases are run
+
+    .. warning::
+
+        This class sets `protected_namespaces` to an empty tuple as a workaround for autodoc.
+        Due to autodoc loading this class first before any other child ones, it causes the Pydantic
+        fields in the protected namespace ``model_`` to be set on the parent. Leading any child
+        classes to inherit these protected fields as user-defined ones, finally triggering their
+        instances to complain about the presence of protected fields.
+
+        Because any class inheriting this class will therefore have protected namespaces disabled,
+        you won't be blocked to create fields starting with ``model_``. Nonetheless, you **must**
+        refrain from doing so as this is not the intended behavior.
+    """
+
+    model_config = ConfigDict(frozen=True, extra="forbid", protected_namespaces=())
+
+    #: The name of the test suite module without the starting ``TestSuite_``. This field **cannot**
+    #: be used in the configuration file. The name will be inherited from the mapping key instead.
+    test_suite_name: str
+    #: The names of test cases from this test suite to execute. If empty, all test cases will be
+    #: executed.
+    test_cases_names: list[str] = Field(default_factory=list, alias="test_cases")
+
+    @cached_property
+    def test_suite_spec(self) -> "TestSuiteSpec":
+        """The specification of the requested test suite."""
+        from framework.test_suite import find_by_name
+
+        test_suite_spec = find_by_name(self.test_suite_name)
+        assert (
+            test_suite_spec is not None
+        ), f"{self.test_suite_name} is not a valid test suite module name."
+        return test_suite_spec
+
+    @model_validator(mode="before")
+    @classmethod
+    def load_test_suite_name_from_context(cls, data: Any, info: ValidationInfo) -> dict:
+        """Load the test suite name from the validation context, if any."""
+        assert isinstance(data, dict), "The test suite configuration value is invalid."
+        name = data.get("test_suite_name")
+        # If the context is carrying the test suite name, then use it instead.
+        if info.context is not None and (test_suite_name := info.context.get("test_suite_name")):
+            assert not name, "The test suite name cannot be set manually."
+            data["test_suite_name"] = test_suite_name
+        return data
+
+    @model_validator(mode="before")
+    @classmethod
+    def convert_from_string(cls, data: Any) -> dict:
+        """Convert the string representation of the model into a valid mapping."""
+        if isinstance(data, str):
+            test_cases = [] if data == "all" else data.split()
+            return dict(test_cases=test_cases)
+        return data
+
+    @model_validator(mode="after")
+    def validate_names(self) -> Self:
+        """Validate the supplied test suite and test cases names.
+
+        This validator relies on the cached property `test_suite_spec` to run for the first
+        time in this call, therefore triggering the assertions if needed.
+        """
+        available_test_cases = map(
+            lambda t: t.name, self.test_suite_spec.class_obj.get_test_cases()
+        )
+        for requested_test_case in self.test_cases_names:
+            assert requested_test_case in available_test_cases, (
+                f"{requested_test_case} is not a valid test case "
+                f"of test suite {self.test_suite_name}."
+            )
+
+        return self
+
+
+class BaseTestSuitesConfigs(BaseModel):
+    """Base class for test suites configs."""
+
+    model_config = ConfigDict(frozen=True, extra="forbid")
+
+    def __contains__(self, key) -> bool:
+        """Check if the provided test suite name has been selected and/or configured."""
+        return key in self.model_fields_set
+
+    def __getitem__(self, key) -> TestSuiteConfig:
+        """Get test suite configuration."""
+        return self.__getattribute__(key)
+
+    def get_configs(self) -> Iterable[TestSuiteConfig]:
+        """Get all the test suite configurations."""
+        return map(lambda t: self[t], self.model_fields_set)
+
+    @classmethod
+    def available_test_suites(cls) -> Iterable[str]:
+        """List all the available test suites."""
+        return cls.model_fields.keys()
+
+    @field_validator("*", mode="before")
+    @classmethod
+    def pass_test_suite_name_to_config(cls, field_value: Any, info: ValidationInfo) -> Any:
+        """Before validating any :class:`TestSuiteConfig`, pass the test suite name via context."""
+        test_suite_name = info.field_name
+        assert test_suite_name is not None
+
+        assert info.context is not None, "A context dictionary is required to load test suites."
+        info.context.update({"test_suite_name": test_suite_name})
+
+        return field_value
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 5f5837a132..2ab8861f99 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -25,6 +25,8 @@
 from types import MethodType
 from typing import Iterable
 
+from pydantic import ValidationError
+
 from framework.testbed_model.capability import Capability, get_supported_capabilities
 from framework.testbed_model.sut_node import SutNode
 from framework.testbed_model.tg_node import TGNode
@@ -34,11 +36,17 @@
     DPDKPrecompiledBuildConfiguration,
     SutNodeConfiguration,
     TestRunConfiguration,
-    TestSuiteConfig,
     TGNodeConfiguration,
     load_config,
 )
-from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError
+from .config.generated import CUSTOM_CONFIG_TYPES
+from .config.test_suite import TestSuiteConfig
+from .exception import (
+    BlockingTestSuiteError,
+    ConfigurationError,
+    SSHTimeoutError,
+    TestCaseVerifyError,
+)
 from .logger import DTSLogger, DtsStage, get_dts_logger
 from .settings import SETTINGS
 from .test_result import (
@@ -141,12 +149,7 @@ def run(self) -> None:
                 self._logger.info(f"Running test run with SUT '{sut_node_config.name}'.")
                 self._init_random_seed(test_run_config)
                 test_run_result = self._result.add_test_run(test_run_config)
-                # we don't want to modify the original config, so create a copy
-                test_run_test_suites = list(
-                    SETTINGS.test_suites if SETTINGS.test_suites else test_run_config.test_suites
-                )
-                if not test_run_config.skip_smoke_tests:
-                    test_run_test_suites[:0] = [TestSuiteConfig(test_suite="smoke_tests")]
+                test_run_test_suites = self._prepare_test_suites(test_run_config)
                 try:
                     test_suites_with_cases = self._get_test_suites_with_cases(
                         test_run_test_suites, test_run_config.func, test_run_config.perf
@@ -203,6 +206,46 @@ def _check_dts_python_version(self) -> None:
             )
             self._logger.warning("Please use Python >= 3.10 instead.")
 
+    def _prepare_test_suites(self, test_run_config: TestRunConfiguration) -> list[TestSuiteConfig]:
+        if SETTINGS.test_suites:
+            test_suites_configs = []
+            for selected_test_suite, selected_test_cases in SETTINGS.test_suites:
+                if selected_test_suite in test_run_config.test_suites:
+                    config = test_run_config.test_suites[selected_test_suite].model_copy(
+                        update={"test_cases_names": selected_test_cases}
+                    )
+                else:
+                    try:
+                        config = CUSTOM_CONFIG_TYPES[selected_test_suite](
+                            test_suite_name=selected_test_suite, test_cases=selected_test_cases
+                        )
+                    except AssertionError as e:
+                        raise ConfigurationError(
+                            "Invalid test cases were selected "
+                            f"for test suite {selected_test_suite}."
+                        ) from e
+                    except ValidationError as e:
+                        raise ConfigurationError(
+                            f"Test suite {selected_test_suite} needs to be explicitly configured "
+                            "in order to be selected."
+                        ) from e
+                    except KeyError:
+                        # not a custom configuration
+                        config = TestSuiteConfig(
+                            test_suite_name=selected_test_suite, test_cases=selected_test_cases
+                        )
+                test_suites_configs.append(config)
+        else:
+            # we don't want to modify the original config, so create a copy
+            test_suites_configs = [
+                config.model_copy() for config in test_run_config.test_suites.get_configs()
+            ]
+
+        if not test_run_config.skip_smoke_tests:
+            test_suites_configs[:0] = [TestSuiteConfig(test_suite_name="smoke_tests")]
+
+        return test_suites_configs
+
     def _get_test_suites_with_cases(
         self,
         test_suite_configs: list[TestSuiteConfig],
@@ -236,7 +279,11 @@ def _get_test_suites_with_cases(
                 test_cases.extend(perf_test_cases)
 
             test_suites_with_cases.append(
-                TestSuiteWithCases(test_suite_class=test_suite_class, test_cases=test_cases)
+                TestSuiteWithCases(
+                    test_suite_class=test_suite_class,
+                    test_cases=test_cases,
+                    config=test_suite_config,
+                )
             )
         return test_suites_with_cases
 
@@ -453,7 +500,9 @@ def _run_test_suite(
         self._logger.set_stage(
             DtsStage.test_suite_setup, Path(SETTINGS.output_dir, test_suite_name)
         )
-        test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node, topology)
+        test_suite = test_suite_with_cases.test_suite_class(
+            sut_node, tg_node, topology, test_suite_with_cases.config
+        )
         try:
             self._logger.info(f"Starting test suite setup: {test_suite_name}")
             test_suite.set_up_suite()
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 5a8e6e5aee..f8783c4b59 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -107,7 +107,7 @@
     LocalDPDKTreeLocation,
     RemoteDPDKTarballLocation,
     RemoteDPDKTreeLocation,
-    TestSuiteConfig,
+    TestSuitesConfigs,
 )
 
 
@@ -133,7 +133,7 @@ class Settings:
     #:
     compile_timeout: float = 1200
     #:
-    test_suites: list[TestSuiteConfig] = field(default_factory=list)
+    test_suites: list[tuple[str, list[str]]] = field(default_factory=list)
     #:
     re_run: int = 0
     #:
@@ -508,7 +508,7 @@ def _process_dpdk_location(
 
 def _process_test_suites(
     parser: _DTSArgumentParser, args: list[list[str]]
-) -> list[TestSuiteConfig]:
+) -> list[tuple[str, list[str]]]:
     """Process the given argument to a list of :class:`TestSuiteConfig` to execute.
 
     Args:
@@ -524,19 +524,17 @@ def _process_test_suites(
         # Environment variable in the form of "SUITE1 CASE1 CASE2, SUITE2 CASE1, SUITE3, ..."
         args = [suite_with_cases.split() for suite_with_cases in args[0][0].split(",")]
 
-    try:
-        return [
-            TestSuiteConfig(test_suite=test_suite, test_cases=test_cases)
-            for [test_suite, *test_cases] in args
-        ]
-    except ValidationError as e:
-        print(
-            "An error has occurred while validating the test suites supplied in the "
-            f"{'environment variable' if action else 'arguments'}:",
-            file=sys.stderr,
-        )
-        print(e, file=sys.stderr)
-        sys.exit(1)
+    available_test_suites = TestSuitesConfigs.available_test_suites()
+    for test_suite_name, *_ in args:
+        if test_suite_name not in available_test_suites:
+            print(
+                f"The test suite {test_suite_name} supplied in the "
+                f"{'environment variable' if action else 'arguments'} is invalid.",
+                file=sys.stderr,
+            )
+            sys.exit(1)
+
+    return [(test_suite, test_cases) for test_suite, *test_cases in args]
 
 
 def get_settings() -> Settings:
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 6014d281b5..8c0c1bcfb3 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -30,7 +30,8 @@
 
 from framework.testbed_model.capability import Capability
 
-from .config import TestRunConfiguration, TestSuiteConfig
+from .config import TestRunConfiguration
+from .config.test_suite import TestSuiteConfig
 from .exception import DTSError, ErrorSeverity
 from .logger import DTSLogger
 from .settings import SETTINGS
@@ -59,23 +60,13 @@ class is to hold a subset of test cases (which could be all test cases) because
     test_suite_class: type[TestSuite]
     test_cases: list[type[TestCase]]
     required_capabilities: set[Capability] = field(default_factory=set, init=False)
+    config: TestSuiteConfig
 
     def __post_init__(self):
         """Gather the required capabilities of the test suite and all test cases."""
         for test_object in [self.test_suite_class] + self.test_cases:
             self.required_capabilities.update(test_object.required_capabilities)
 
-    def create_config(self) -> TestSuiteConfig:
-        """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
-
-        Returns:
-            The :class:`TestSuiteConfig` representation.
-        """
-        return TestSuiteConfig(
-            test_suite=self.test_suite_class.__name__,
-            test_cases=[test_case.__name__ for test_case in self.test_cases],
-        )
-
     def mark_skip_unsupported(self, supported_capabilities: set[Capability]) -> None:
         """Mark the test suite and test cases to be skipped.
 
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index fb5d646ce3..24cd0d38c1 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -24,13 +24,14 @@
 from ipaddress import IPv4Interface, IPv6Interface, ip_interface
 from pkgutil import iter_modules
 from types import ModuleType
-from typing import ClassVar, Protocol, TypeVar, Union, cast
+from typing import ClassVar, Protocol, TypeVar, Union, cast, get_type_hints
 
 from scapy.layers.inet import IP  # type: ignore[import-untyped]
 from scapy.layers.l2 import Ether  # type: ignore[import-untyped]
 from scapy.packet import Packet, Padding, raw  # type: ignore[import-untyped]
 from typing_extensions import Self
 
+from framework.config.test_suite import TestSuiteConfig
 from framework.testbed_model.capability import TestProtocol
 from framework.testbed_model.port import Port
 from framework.testbed_model.sut_node import SutNode
@@ -80,6 +81,7 @@ class TestSuite(TestProtocol):
     #: Whether the test suite is blocking. A failure of a blocking test suite
     #: will block the execution of all subsequent test suites in the current test run.
     is_blocking: ClassVar[bool] = False
+    config: TestSuiteConfig
     _logger: DTSLogger
     _sut_port_ingress: Port
     _sut_port_egress: Port
@@ -95,6 +97,7 @@ def __init__(
         sut_node: SutNode,
         tg_node: TGNode,
         topology: Topology,
+        config: TestSuiteConfig,
     ):
         """Initialize the test suite testbed information and basic configuration.
 
@@ -105,9 +108,11 @@ def __init__(
             sut_node: The SUT node where the test suite will run.
             tg_node: The TG node where the test suite will run.
             topology: The topology where the test suite will run.
+            config: The test suite configuration.
         """
         self.sut_node = sut_node
         self.tg_node = tg_node
+        self.config = config
         self._logger = get_dts_logger(self.__class__.__name__)
         self._tg_port_egress = topology.tg_port_egress
         self._sut_port_ingress = topology.sut_port_ingress
@@ -663,6 +668,21 @@ def is_test_suite(obj) -> bool:
             f"Expected class {self.class_name} not found in module {self.module_name}."
         )
 
+    @cached_property
+    def config_obj(self) -> type[TestSuiteConfig]:
+        """A reference to the test suite's configuration type."""
+        fields = get_type_hints(self.class_obj)
+        config_obj = fields.get("config")
+        if config_obj is None:
+            raise InternalError(
+                "Test suite class {self.class_name} is missing the `config` attribute."
+            )
+        if not issubclass(config_obj, TestSuiteConfig):
+            raise InternalError(
+                f"Test suite class {self.class_name} has an invalid configuration type assigned."
+            )
+        return config_obj
+
     @classmethod
     def discover_all(
         cls, package_name: str | None = None, module_prefix: str | None = None
diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
index 734f006026..f2998b968e 100644
--- a/dts/tests/TestSuite_hello_world.py
+++ b/dts/tests/TestSuite_hello_world.py
@@ -15,12 +15,15 @@
     LogicalCoreCountFilter,
     LogicalCoreList,
 )
+from tests.config import HelloWorldConfig
 
 
 @requires(topology_type=TopologyType.no_link)
 class TestHelloWorld(TestSuite):
     """DPDK hello world app test suite."""
 
+    config: HelloWorldConfig
+
     def set_up_suite(self) -> None:
         """Set up the test suite.
 
@@ -63,7 +66,7 @@ def hello_world_all_cores(self) -> None:
         eal_para = compute_eal_params(
             self.sut_node, lcore_filter_specifier=LogicalCoreList(self.sut_node.lcores)
         )
-        result = self.sut_node.run_dpdk_app(self.app_helloworld_path, eal_para, 50)
+        result = self.sut_node.run_dpdk_app(self.app_helloworld_path, eal_para, self.config.timeout)
         for lcore in self.sut_node.lcores:
             self.verify(
                 f"hello from core {int(lcore)}" in result.stdout,
diff --git a/dts/tests/__init__.py b/dts/tests/__init__.py
new file mode 100644
index 0000000000..a300eb26fc
--- /dev/null
+++ b/dts/tests/__init__.py
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Test suites.
+
+This package contains all the available test suites in DTS.
+"""
diff --git a/dts/tests/config.py b/dts/tests/config.py
new file mode 100644
index 0000000000..300ad3ef6a
--- /dev/null
+++ b/dts/tests/config.py
@@ -0,0 +1,20 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Module for test suites custom configurations.
+
+Any test suite that requires custom configuration fields should create a new config class inheriting
+:class:`~framework.config.test_suite.TestSuiteConfig`, while respecting the parents' frozen state.
+Any custom fields can be added in this class.
+
+The custom configuration classes can be stored in this module.
+"""
+
+from framework.config.test_suite import TestSuiteConfig
+
+
+class HelloWorldConfig(TestSuiteConfig):
+    """Example custom configuration for the `TestHelloWorld` test suite."""
+
+    #: Timeout for the DPDK apps.
+    timeout: int = 50
-- 
2.43.0


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH v2 4/4] dts: update autodoc sorting order
  2024-11-08 13:38 ` [PATCH v2 0/4] " Luca Vizzarro
                     ` (2 preceding siblings ...)
  2024-11-08 13:38   ` [PATCH v2 3/4] dts: add per-test-suite configuration Luca Vizzarro
@ 2024-11-08 13:38   ` Luca Vizzarro
  3 siblings, 0 replies; 27+ messages in thread
From: Luca Vizzarro @ 2024-11-08 13:38 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Patrick Robb, Luca Vizzarro

The autodoc member sorting order default is set to alphabetical, which
translates to autodoc sorting every member in modules, classes etc.
This also brings some side effects, like sorting capabilities which
can't be compared and result in errors.

This change prevents autodoc from sorting, and keeping the order as the
developer intended it.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 doc/guides/conf.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/guides/conf.py b/doc/guides/conf.py
index 71fed45b3d..ed5bc5eb30 100644
--- a/doc/guides/conf.py
+++ b/doc/guides/conf.py
@@ -88,6 +88,7 @@
     autodoc_typehints = 'both'
     autodoc_typehints_format = 'short'
     autodoc_typehints_description_target = 'documented'
+    autodoc_member_order = 'bysource'
 
     # Intersphinx allows linking to external projects, such as Python docs.
     intersphinx_mapping = {'python': ('https://docs.python.org/3', None)}
-- 
2.43.0


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH v3 0/4] dts: add per-test-suite configuration
  2024-09-06 16:13 [PATCH] dts: add per-test-suite configuration Luca Vizzarro
  2024-09-27 17:45 ` Jeremy Spewock
  2024-11-08 13:38 ` [PATCH v2 0/4] " Luca Vizzarro
@ 2024-11-08 13:45 ` Luca Vizzarro
  2024-11-08 13:45   ` [PATCH v3 1/4] dts: add tests package to API docs Luca Vizzarro
                     ` (4 more replies)
  2 siblings, 5 replies; 27+ messages in thread
From: Luca Vizzarro @ 2024-11-08 13:45 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Patrick Robb, Luca Vizzarro

Hi there,

sending a v3 for the per-test-suite configuration additions.

v3:
- fixed bug in dts-check-format
v2:
- rebase
- added test suites to doc pages
- doc fixes
- updated the custom config procedure to using tests.config
- simplified the generated file
- implemented the generator in dts-check-format

Best regards,
Luca Vizzarro

Depends-on: series-33871 ("dts: Pydantic configuration")

Luca Vizzarro (4):
  dts: add tests package to API docs
  dts: fix smoke tests docstring
  dts: add per-test-suite configuration
  dts: update autodoc sorting order

 devtools/dts-check-format.sh                  |  28 +++-
 devtools/dts-generate-tests-mappings.py       | 117 +++++++++++++
 doc/api/dts/framework.config.generated.rst    |   8 +
 doc/api/dts/framework.config.rst              |   7 +
 doc/api/dts/framework.config.test_suite.rst   |   8 +
 doc/api/dts/index.rst                         |   1 +
 doc/api/dts/tests.TestSuite_hello_world.rst   |   9 +
 doc/api/dts/tests.TestSuite_os_udp.rst        |   9 +
 .../tests.TestSuite_pmd_buffer_scatter.rst    |   9 +
 doc/api/dts/tests.TestSuite_smoke_tests.rst   |   9 +
 doc/api/dts/tests.TestSuite_vlan.rst          |  10 ++
 doc/api/dts/tests.config.rst                  |   9 +
 doc/api/dts/tests.rst                         |  20 +++
 doc/guides/conf.py                            |   1 +
 doc/guides/tools/dts.rst                      |  23 +++
 dts/conf.yaml                                 |   4 +-
 dts/framework/config/__init__.py              |  76 +--------
 dts/framework/config/generated.py             |  25 +++
 dts/framework/config/test_suite.py            | 154 ++++++++++++++++++
 dts/framework/runner.py                       |  69 ++++++--
 dts/framework/settings.py                     |  30 ++--
 dts/framework/test_result.py                  |  15 +-
 dts/framework/test_suite.py                   |  22 ++-
 dts/tests/TestSuite_hello_world.py            |   5 +-
 dts/tests/TestSuite_smoke_tests.py            |   2 -
 dts/tests/__init__.py                         |   7 +
 dts/tests/config.py                           |  20 +++
 27 files changed, 580 insertions(+), 117 deletions(-)
 create mode 100755 devtools/dts-generate-tests-mappings.py
 create mode 100644 doc/api/dts/framework.config.generated.rst
 create mode 100644 doc/api/dts/framework.config.test_suite.rst
 create mode 100644 doc/api/dts/tests.TestSuite_hello_world.rst
 create mode 100644 doc/api/dts/tests.TestSuite_os_udp.rst
 create mode 100644 doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
 create mode 100644 doc/api/dts/tests.TestSuite_smoke_tests.rst
 create mode 100644 doc/api/dts/tests.TestSuite_vlan.rst
 create mode 100644 doc/api/dts/tests.config.rst
 create mode 100644 doc/api/dts/tests.rst
 create mode 100644 dts/framework/config/generated.py
 create mode 100644 dts/framework/config/test_suite.py
 create mode 100644 dts/tests/__init__.py
 create mode 100644 dts/tests/config.py

-- 
2.43.0


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH v3 1/4] dts: add tests package to API docs
  2024-11-08 13:45 ` [PATCH v3 0/4] dts: add per-test-suite configuration Luca Vizzarro
@ 2024-11-08 13:45   ` Luca Vizzarro
  2024-11-12 19:55     ` Dean Marx
  2024-11-08 13:45   ` [PATCH v3 2/4] dts: fix smoke tests docstring Luca Vizzarro
                     ` (3 subsequent siblings)
  4 siblings, 1 reply; 27+ messages in thread
From: Luca Vizzarro @ 2024-11-08 13:45 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Patrick Robb, Luca Vizzarro

The test suites are not documented in the API as their respective
documentation files are missing.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 doc/api/dts/index.rst                         |  1 +
 doc/api/dts/tests.TestSuite_hello_world.rst   |  9 +++++++++
 doc/api/dts/tests.TestSuite_os_udp.rst        |  9 +++++++++
 .../tests.TestSuite_pmd_buffer_scatter.rst    |  9 +++++++++
 doc/api/dts/tests.TestSuite_smoke_tests.rst   |  9 +++++++++
 doc/api/dts/tests.TestSuite_vlan.rst          | 10 ++++++++++
 doc/api/dts/tests.rst                         | 19 +++++++++++++++++++
 7 files changed, 66 insertions(+)
 create mode 100644 doc/api/dts/tests.TestSuite_hello_world.rst
 create mode 100644 doc/api/dts/tests.TestSuite_os_udp.rst
 create mode 100644 doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
 create mode 100644 doc/api/dts/tests.TestSuite_smoke_tests.rst
 create mode 100644 doc/api/dts/tests.TestSuite_vlan.rst
 create mode 100644 doc/api/dts/tests.rst

diff --git a/doc/api/dts/index.rst b/doc/api/dts/index.rst
index 534512dc17..db05fd37d5 100644
--- a/doc/api/dts/index.rst
+++ b/doc/api/dts/index.rst
@@ -14,6 +14,7 @@ Packages
    :includehidden:
    :maxdepth: 1
 
+   tests
    framework.testbed_model
    framework.remote_session
    framework.params
diff --git a/doc/api/dts/tests.TestSuite_hello_world.rst b/doc/api/dts/tests.TestSuite_hello_world.rst
new file mode 100644
index 0000000000..f92786996f
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_hello_world.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+hello_world Test Suite
+======================
+
+.. automodule:: tests.TestSuite_hello_world
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.TestSuite_os_udp.rst b/doc/api/dts/tests.TestSuite_os_udp.rst
new file mode 100644
index 0000000000..121a653670
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_os_udp.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+os_udp Test Suite
+=================
+
+.. automodule:: tests.TestSuite_os_udp
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst b/doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
new file mode 100644
index 0000000000..b6571b24f5
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+pmd_buffer_scatter Test Suite
+=============================
+
+.. automodule:: tests.TestSuite_pmd_buffer_scatter
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.TestSuite_smoke_tests.rst b/doc/api/dts/tests.TestSuite_smoke_tests.rst
new file mode 100644
index 0000000000..5b11aa7f48
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_smoke_tests.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+smoke_tests Test Suite
+======================
+
+.. automodule:: tests.TestSuite_smoke_tests
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.TestSuite_vlan.rst b/doc/api/dts/tests.TestSuite_vlan.rst
new file mode 100644
index 0000000000..86a29538bb
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_vlan.rst
@@ -0,0 +1,10 @@
+
+.. SPDX-License-Identifier: BSD-3-Clause
+
+vlan Test Suite
+===============
+
+.. automodule:: tests.TestSuite_vlan
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.rst b/doc/api/dts/tests.rst
new file mode 100644
index 0000000000..0c136b4bb0
--- /dev/null
+++ b/doc/api/dts/tests.rst
@@ -0,0 +1,19 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+tests - Test Suites Package
+==============================
+
+.. automodule:: tests
+   :members:
+   :show-inheritance:
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   tests.TestSuite_hello_world
+   tests.TestSuite_os_udp
+   tests.TestSuite_pmd_buffer_scatter
+   tests.TestSuite_smoke_tests
+   tests.TestSuite_vlan
+
-- 
2.43.0


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH v3 2/4] dts: fix smoke tests docstring
  2024-11-08 13:45 ` [PATCH v3 0/4] dts: add per-test-suite configuration Luca Vizzarro
  2024-11-08 13:45   ` [PATCH v3 1/4] dts: add tests package to API docs Luca Vizzarro
@ 2024-11-08 13:45   ` Luca Vizzarro
  2024-11-08 13:45   ` [PATCH v3 3/4] dts: add per-test-suite configuration Luca Vizzarro
                     ` (2 subsequent siblings)
  4 siblings, 0 replies; 27+ messages in thread
From: Luca Vizzarro @ 2024-11-08 13:45 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Patrick Robb, Luca Vizzarro

Sphinx autodoc complains of is_blocking being duplicated. This is the
case as the parent already holds a docstring for this attribute. Remove
the duplication.

Fixes: 6ef07151aac4 ("dts: update docstrings")

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 dts/tests/TestSuite_smoke_tests.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
index bc3a2a6bf9..f564ac82b4 100644
--- a/dts/tests/TestSuite_smoke_tests.py
+++ b/dts/tests/TestSuite_smoke_tests.py
@@ -30,8 +30,6 @@ class TestSmokeTests(TestSuite):
     The infrastructure also needs to be tested, as that is also used by all other test suites.
 
     Attributes:
-        is_blocking: This test suite will block the execution of all other test suites
-            in the test run after it.
         nics_in_node: The NICs present on the SUT node.
     """
 
-- 
2.43.0


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH v3 3/4] dts: add per-test-suite configuration
  2024-11-08 13:45 ` [PATCH v3 0/4] dts: add per-test-suite configuration Luca Vizzarro
  2024-11-08 13:45   ` [PATCH v3 1/4] dts: add tests package to API docs Luca Vizzarro
  2024-11-08 13:45   ` [PATCH v3 2/4] dts: fix smoke tests docstring Luca Vizzarro
@ 2024-11-08 13:45   ` Luca Vizzarro
  2024-11-13 21:44     ` Dean Marx
  2024-11-08 13:45   ` [PATCH v3 4/4] dts: update autodoc sorting order Luca Vizzarro
  2025-03-03 14:57   ` [PATCH v4 0/7] dts: add per-test-suite configuration Luca Vizzarro
  4 siblings, 1 reply; 27+ messages in thread
From: Luca Vizzarro @ 2024-11-08 13:45 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Patrick Robb, Luca Vizzarro

Allow test suites to be configured individually. Moreover enable them to
implement their own custom configuration.

This solution adds some new complexity to DTS, which is generated source
code. In order to ensure strong typing, the test suites and their custom
configurations need to be linked in the main configuration class.
Unfortunately, this is not feasible during runtime as it will incur in
circular dependencies. Generating the links appear to be the most
straightforward approach.

This commit also brings a new major change to the configuration schema.
Test suites are no longer defined as a list of strings, like:

    test_suites:
    - hello_world
    - pmd_buffer_scatter

but as mapping of mappings or strings:

    test_suites:
      hello_world: {} # any custom fields or test cases can be set here
      pmd_buffer_scatter: all # "all" defines all the test cases, or
                              # they can individually be set separated
                              # by a space

Not defining the `test_cases` field in the configuration is equivalent
to `all`, therefore the definitions for either test suite above are
also equivalent.

Creating the __init__.py file under the tests folder, allows it to be
picked up as a package. This is a mypy requirement to import the tests
from within the framework.

Bugzilla ID: 1375

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 devtools/dts-check-format.sh                |  28 +++-
 devtools/dts-generate-tests-mappings.py     | 117 +++++++++++++++
 doc/api/dts/framework.config.generated.rst  |   8 +
 doc/api/dts/framework.config.rst            |   7 +
 doc/api/dts/framework.config.test_suite.rst |   8 +
 doc/api/dts/tests.config.rst                |   9 ++
 doc/api/dts/tests.rst                       |   1 +
 doc/guides/tools/dts.rst                    |  23 +++
 dts/conf.yaml                               |   4 +-
 dts/framework/config/__init__.py            |  76 +---------
 dts/framework/config/generated.py           |  25 ++++
 dts/framework/config/test_suite.py          | 154 ++++++++++++++++++++
 dts/framework/runner.py                     |  69 +++++++--
 dts/framework/settings.py                   |  30 ++--
 dts/framework/test_result.py                |  15 +-
 dts/framework/test_suite.py                 |  22 ++-
 dts/tests/TestSuite_hello_world.py          |   5 +-
 dts/tests/__init__.py                       |   7 +
 dts/tests/config.py                         |  20 +++
 19 files changed, 513 insertions(+), 115 deletions(-)
 create mode 100755 devtools/dts-generate-tests-mappings.py
 create mode 100644 doc/api/dts/framework.config.generated.rst
 create mode 100644 doc/api/dts/framework.config.test_suite.rst
 create mode 100644 doc/api/dts/tests.config.rst
 create mode 100644 dts/framework/config/generated.py
 create mode 100644 dts/framework/config/test_suite.py
 create mode 100644 dts/tests/__init__.py
 create mode 100644 dts/tests/config.py

diff --git a/devtools/dts-check-format.sh b/devtools/dts-check-format.sh
index 3f43e17e88..bad771fea3 100755
--- a/devtools/dts-check-format.sh
+++ b/devtools/dts-check-format.sh
@@ -2,6 +2,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2022 University of New Hampshire
 # Copyright(c) 2023 PANTHEON.tech s.r.o.
+# Copyright(c) 2024 Arm Limited
 
 usage() {
 	echo "Usage: $(basename $0) [options] [directory]"
@@ -13,15 +14,19 @@ usage() {
 format=true
 lint=true
 typecheck=true
+generate=true
 
 # Comments after args serve as documentation; must be present
-while getopts "hflt" arg; do
+while getopts "hgflt" arg; do
 	case $arg in
 	h) # Display this message
-		echo 'Run formatting and linting programs for DTS.'
+		echo 'Run generating, formatting and linting programs for DTS.'
 		usage
 		exit 0
 		;;
+	g) # Don't run code generator
+		generate=false
+		;;
 	f) # Don't run formatters
 		format=false
 		;;
@@ -48,7 +53,22 @@ heading() {
 
 errors=0
 
+if $generate; then
+	heading "Generating test suites to configuration mappings"
+	if command -v python3 > /dev/null; then
+		../devtools/dts-generate-tests-mappings.py
+		errors=$((errors + $?))
+	else
+		echo "python3 not found, unable to run generator"
+		errors=$((errors + 1))
+	fi
+fi
+
 if $format; then
+	if $generate; then
+		echo
+	fi
+
 	if command -v git > /dev/null; then
 		if git rev-parse --is-inside-work-tree >&-; then
 			heading "Formatting in $directory/"
@@ -85,7 +105,7 @@ if $format; then
 fi
 
 if $lint; then
-	if $format; then
+	if $generate || $format; then
 		echo
 	fi
 	heading "Linting in $directory/"
@@ -99,7 +119,7 @@ if $lint; then
 fi
 
 if $typecheck; then
-	if $format || $lint; then
+	if $generate || $format || $lint; then
 		echo
 	fi
 	heading "Checking types in $directory/"
diff --git a/devtools/dts-generate-tests-mappings.py b/devtools/dts-generate-tests-mappings.py
new file mode 100755
index 0000000000..26ecc1018c
--- /dev/null
+++ b/devtools/dts-generate-tests-mappings.py
@@ -0,0 +1,117 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""DTS Test Suites to Configuration mappings generation script."""
+
+import os
+import sys
+from collections import defaultdict
+from pathlib import Path
+from textwrap import indent
+from typing import Iterable
+
+DTS_DIR = Path(__file__).parent.joinpath("..", "dts").resolve()
+SCRIPT_FILE_NAME = Path(__file__).relative_to(Path(__file__).parent.parent)
+
+sys.path.append(str(DTS_DIR))
+
+from framework.config.test_suite import BaseTestSuitesConfigs, TestSuiteConfig
+from framework.exception import InternalError
+from framework.test_suite import AVAILABLE_TEST_SUITES, TestSuiteSpec
+
+FRAMEWORK_IMPORTS = [BaseTestSuitesConfigs, TestSuiteConfig]
+
+RELATIVE_PATH_TO_GENERATED_FILE = "framework/config/generated.py"
+SMOKE_TESTS_SUITE_NAME = "smoke_tests"
+CUSTOM_CONFIG_TYPES_VAR_NAME = "CUSTOM_CONFIG_TYPES"
+CUSTOM_CONFIG_TYPES_VAR_DOCSTRING = [
+    "#: Mapping of test suites to their corresponding custom configuration objects if any."
+]
+TEST_SUITES_CONFIG_CLASS_NAME = "TestSuitesConfigs"
+TEST_SUITES_CONFIG_CLASS_DOCSTRING = [
+    '"""Configuration mapping class to select and configure the test suites."""',
+]
+
+
+GENERATED_FILE_HEADER = f"""# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 The DPDK contributors
+# This file is automatically generated by {SCRIPT_FILE_NAME}.
+# Do NOT modify this file manually.
+
+\"\"\"Generated file containing the links between the test suites and the configuration.\"\"\"
+"""
+
+
+def join(lines: Iterable[str]) -> str:
+    """Join list of strings into text lines."""
+    return "\n".join(lines)
+
+
+def join_and_indent(lines: Iterable[str], indentation_level=1, indentation_spaces=4) -> str:
+    """Join list of strings into indented text lines."""
+    return "\n".join([indent(line, " " * indentation_level * indentation_spaces) for line in lines])
+
+
+def format_attributes_types(test_suite_spec: TestSuiteSpec):
+    """Format the config type into the respective configuration class field attribute type."""
+    config_type = test_suite_spec.config_obj.__name__
+    return f"Optional[{config_type}]"
+
+
+try:
+    framework_imports: dict[str, list[str]] = defaultdict(list)
+    for _import in FRAMEWORK_IMPORTS:
+        framework_imports[_import.__module__].append(_import.__name__)
+    formatted_framework_imports = sorted(
+        [
+            f"from {module} import {', '.join(sorted(imports))}"
+            for module, imports in framework_imports.items()
+        ]
+    )
+
+    test_suites = [
+        test_suite_spec
+        for test_suite_spec in AVAILABLE_TEST_SUITES
+        if test_suite_spec.name != SMOKE_TESTS_SUITE_NAME
+    ]
+
+    custom_configs = [t for t in test_suites if t.config_obj is not TestSuiteConfig]
+
+    custom_config_imports = [
+        f"from {t.config_obj.__module__} import {t.config_obj.__name__}" for t in custom_configs
+    ]
+
+    test_suites_attributes = [f"{t.name}: {format_attributes_types(t)} = None" for t in test_suites]
+
+    custom_config_mappings = [f'"{t.name}": {t.config_obj.__name__},' for t in custom_configs]
+
+    generated_file_contents = f"""{GENERATED_FILE_HEADER}
+from typing import Optional
+
+{join(formatted_framework_imports)}
+
+{join(custom_config_imports)}
+
+{join(CUSTOM_CONFIG_TYPES_VAR_DOCSTRING)}
+{CUSTOM_CONFIG_TYPES_VAR_NAME}: dict[str, type[{TestSuiteConfig.__name__}]] = {'{'}
+{join_and_indent(custom_config_mappings)}
+{'}'}
+
+
+class {TEST_SUITES_CONFIG_CLASS_NAME}({BaseTestSuitesConfigs.__name__}):
+{join_and_indent(TEST_SUITES_CONFIG_CLASS_DOCSTRING)}
+
+{join_and_indent(test_suites_attributes)}
+"""
+
+    path = os.path.join(DTS_DIR, RELATIVE_PATH_TO_GENERATED_FILE)
+
+    with open(path, "w") as generated_file:
+        generated_file.write(generated_file_contents)
+
+    print("Test suites to configuration mappings generated successfully!")
+except Exception as e:
+    raise InternalError(
+        "Failed to generate test suites to configuration mappings."
+    ) from e
diff --git a/doc/api/dts/framework.config.generated.rst b/doc/api/dts/framework.config.generated.rst
new file mode 100644
index 0000000000..5dfa9342f0
--- /dev/null
+++ b/doc/api/dts/framework.config.generated.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+generated - Generated Test Suite Configurations
+===============================================
+
+.. automodule:: framework.config.generated
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/framework.config.rst b/doc/api/dts/framework.config.rst
index cc266276c1..217fe026c4 100644
--- a/doc/api/dts/framework.config.rst
+++ b/doc/api/dts/framework.config.rst
@@ -6,3 +6,10 @@ config - Configuration Package
 .. automodule:: framework.config
    :members:
    :show-inheritance:
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   framework.config.generated
+   framework.config.test_suite
diff --git a/doc/api/dts/framework.config.test_suite.rst b/doc/api/dts/framework.config.test_suite.rst
new file mode 100644
index 0000000000..d59dcf5d6e
--- /dev/null
+++ b/doc/api/dts/framework.config.test_suite.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+test_suite - Test Suite Configuration Definitions
+=================================================
+
+.. automodule:: framework.config.test_suite
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.config.rst b/doc/api/dts/tests.config.rst
new file mode 100644
index 0000000000..ce3d9df868
--- /dev/null
+++ b/doc/api/dts/tests.config.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+Test Suites Configurations
+==========================
+
+.. automodule:: tests.config
+   :members:
+   :show-inheritance:
+
diff --git a/doc/api/dts/tests.rst b/doc/api/dts/tests.rst
index 0c136b4bb0..7fc25f2123 100644
--- a/doc/api/dts/tests.rst
+++ b/doc/api/dts/tests.rst
@@ -11,6 +11,7 @@ tests - Test Suites Package
    :hidden:
    :maxdepth: 1
 
+   tests.config
    tests.TestSuite_hello_world
    tests.TestSuite_os_udp
    tests.TestSuite_pmd_buffer_scatter
diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index f4e297413d..4e63601b19 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -408,6 +408,29 @@ There are four types of methods that comprise a test suite:
    should be implemented in the ``SutNode`` class (and the underlying classes that ``SutNode`` uses)
    and used by the test suite via the ``sut_node`` field.
 
+The test suites can also implement their own custom configuration fields. This can be achieved by
+creating a new test suite config file which inherits from ``TestSuiteConfig`` defined in
+``dts/framework/config/test_suite.py``. So that this new custom configuration class is used, the
+test suite class must override the ``config`` attribute annotation with your new class, for example:
+
+.. code:: python
+
+   # place this under tests/config.py to avoid circular dependencies
+   class CustomConfig(TestSuiteConfig):
+      my_custom_field: int = 10
+
+   # place this under tests/TestSuite_my_new_test_suite.py
+   class TestMyNewTestSuite(TestSuite):
+      config: CustomConfig
+
+Finally, the test suites and the custom configuration files need to linked in the global configuration.
+This can be easily achieved by running the ``dts/generate-test-mappings.py``, e.g.:
+
+.. code-block:: console
+
+   $ poetry shell
+   (dts-py3.10) $ ./generate-test-mappings.py
+
 
 .. _dts_dev_tools:
 
diff --git a/dts/conf.yaml b/dts/conf.yaml
index 2496262854..377304dddf 100644
--- a/dts/conf.yaml
+++ b/dts/conf.yaml
@@ -28,8 +28,8 @@ test_runs:
     func: true # enable functional testing
     skip_smoke_tests: false # optional
     test_suites: # the following test suites will be run in their entirety
-      - hello_world
-      - os_udp
+      hello_world: all
+      os_udp: all
     # The machine running the DPDK test executable
     system_under_test_node:
       node_name: "SUT 1"
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 82113a6257..0ac7ab5c46 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -32,11 +32,13 @@
       and makes it thread safe should we ever want to move in that direction.
 """
 
+# pylama:ignore=W0611
+
 import tarfile
 from enum import Enum, auto, unique
 from functools import cached_property
 from pathlib import Path, PurePath
-from typing import TYPE_CHECKING, Annotated, Any, Literal, NamedTuple
+from typing import Annotated, Literal, NamedTuple
 
 import yaml
 from pydantic import (
@@ -52,8 +54,7 @@
 from framework.exception import ConfigurationError
 from framework.utils import REGEX_FOR_PCI_ADDRESS, StrEnum
 
-if TYPE_CHECKING:
-    from framework.test_suite import TestSuiteSpec
+from .generated import TestSuitesConfigs
 
 
 class FrozenModel(BaseModel):
@@ -382,69 +383,6 @@ class DPDKUncompiledBuildConfiguration(BaseDPDKBuildConfiguration):
 DPDKBuildConfiguration = DPDKPrecompiledBuildConfiguration | DPDKUncompiledBuildConfiguration
 
 
-class TestSuiteConfig(FrozenModel):
-    """Test suite configuration.
-
-    Information about a single test suite to be executed. This can also be represented as a string
-    instead of a mapping, example:
-
-    .. code:: yaml
-
-        test_runs:
-        - test_suites:
-            # As string representation:
-            - hello_world # test all of `hello_world`, or
-            - hello_world hello_world_single_core # test only `hello_world_single_core`
-            # or as model fields:
-            - test_suite: hello_world
-              test_cases: [hello_world_single_core] # without this field all test cases are run
-    """
-
-    #: The name of the test suite module without the starting ``TestSuite_``.
-    test_suite_name: str = Field(alias="test_suite")
-    #: The names of test cases from this test suite to execute. If empty, all test cases will be
-    #: executed.
-    test_cases_names: list[str] = Field(default_factory=list, alias="test_cases")
-
-    @cached_property
-    def test_suite_spec(self) -> "TestSuiteSpec":
-        """The specification of the requested test suite."""
-        from framework.test_suite import find_by_name
-
-        test_suite_spec = find_by_name(self.test_suite_name)
-        assert (
-            test_suite_spec is not None
-        ), f"{self.test_suite_name} is not a valid test suite module name."
-        return test_suite_spec
-
-    @model_validator(mode="before")
-    @classmethod
-    def convert_from_string(cls, data: Any) -> Any:
-        """Convert the string representation of the model into a valid mapping."""
-        if isinstance(data, str):
-            [test_suite, *test_cases] = data.split()
-            return dict(test_suite=test_suite, test_cases=test_cases)
-        return data
-
-    @model_validator(mode="after")
-    def validate_names(self) -> Self:
-        """Validate the supplied test suite and test cases names.
-
-        This validator relies on the cached property `test_suite_spec` to run for the first
-        time in this call, therefore triggering the assertions if needed.
-        """
-        available_test_cases = map(
-            lambda t: t.name, self.test_suite_spec.class_obj.get_test_cases()
-        )
-        for requested_test_case in self.test_cases_names:
-            assert requested_test_case in available_test_cases, (
-                f"{requested_test_case} is not a valid test case "
-                f"of test suite {self.test_suite_name}."
-            )
-
-        return self
-
-
 class TestRunSUTNodeConfiguration(FrozenModel):
     """The SUT node configuration of a test run."""
 
@@ -469,8 +407,8 @@ class TestRunConfiguration(FrozenModel):
     func: bool
     #: Whether to skip smoke tests.
     skip_smoke_tests: bool = False
-    #: The names of test suites and/or test cases to execute.
-    test_suites: list[TestSuiteConfig] = Field(min_length=1)
+    #: The test suites to be selected and/or configured.
+    test_suites: TestSuitesConfigs
     #: The SUT node configuration to use in this test run.
     system_under_test_node: TestRunSUTNodeConfiguration
     #: The TG node name to use in this test run.
@@ -602,6 +540,6 @@ def load_config(config_file_path: Path) -> Configuration:
         config_data = yaml.safe_load(f)
 
     try:
-        return Configuration.model_validate(config_data)
+        return Configuration.model_validate(config_data, context={})
     except ValidationError as e:
         raise ConfigurationError("failed to load the supplied configuration") from e
diff --git a/dts/framework/config/generated.py b/dts/framework/config/generated.py
new file mode 100644
index 0000000000..cc4a539987
--- /dev/null
+++ b/dts/framework/config/generated.py
@@ -0,0 +1,25 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 The DPDK contributors
+# This file is automatically generated by devtools/dts-generate-tests-mappings.py.
+# Do NOT modify this file manually.
+
+"""Generated file containing the links between the test suites and the configuration."""
+
+from typing import Optional
+
+from framework.config.test_suite import BaseTestSuitesConfigs, TestSuiteConfig
+from tests.config import HelloWorldConfig
+
+#: Mapping of test suites to their corresponding custom configuration objects if any.
+CUSTOM_CONFIG_TYPES: dict[str, type[TestSuiteConfig]] = {
+    "hello_world": HelloWorldConfig,
+}
+
+
+class TestSuitesConfigs(BaseTestSuitesConfigs):
+    """Configuration mapping class to select and configure the test suites."""
+
+    hello_world: Optional[HelloWorldConfig] = None
+    os_udp: Optional[TestSuiteConfig] = None
+    pmd_buffer_scatter: Optional[TestSuiteConfig] = None
+    vlan: Optional[TestSuiteConfig] = None
diff --git a/dts/framework/config/test_suite.py b/dts/framework/config/test_suite.py
new file mode 100644
index 0000000000..863052cbc1
--- /dev/null
+++ b/dts/framework/config/test_suite.py
@@ -0,0 +1,154 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Test suites configuration module.
+
+Test suites can inherit :class:`TestSuiteConfig` to create their own custom configuration.
+By doing so, the test suite class must also override the annotation of the field
+`~framework.test_suite.TestSuite.config` to use their custom configuration type.
+"""
+
+from functools import cached_property
+from typing import TYPE_CHECKING, Any, Iterable
+
+from pydantic import (
+    BaseModel,
+    ConfigDict,
+    Field,
+    ValidationInfo,
+    field_validator,
+    model_validator,
+)
+from typing_extensions import Self
+
+if TYPE_CHECKING:
+    from framework.test_suite import TestSuiteSpec
+
+
+class TestSuiteConfig(BaseModel):
+    """Test suite configuration base model.
+
+    By default the configuration of a generic test suite does not contain any attributes. Any test
+    suite should inherit this class to create their own custom configuration. Finally override the
+    type of the :attr:`~TestSuite.config` to use the newly created one.
+
+    If no custom fields require setting, this can also be represented as a string instead of
+    a mapping, example:
+
+    .. code:: yaml
+
+        test_runs:
+        - test_suites:
+            # As string representation:
+            hello_world: all # test all of `hello_world`, or
+            hello_world: hello_world_single_core # test only `hello_world_single_core`
+            # or as a mapping of the model's fields:
+            hello_world:
+              test_cases: [hello_world_single_core] # without this field all test cases are run
+
+    .. warning::
+
+        This class sets `protected_namespaces` to an empty tuple as a workaround for autodoc.
+        Due to autodoc loading this class first before any other child ones, it causes the Pydantic
+        fields in the protected namespace ``model_`` to be set on the parent. Leading any child
+        classes to inherit these protected fields as user-defined ones, finally triggering their
+        instances to complain about the presence of protected fields.
+
+        Because any class inheriting this class will therefore have protected namespaces disabled,
+        you won't be blocked to create fields starting with ``model_``. Nonetheless, you **must**
+        refrain from doing so as this is not the intended behavior.
+    """
+
+    model_config = ConfigDict(frozen=True, extra="forbid", protected_namespaces=())
+
+    #: The name of the test suite module without the starting ``TestSuite_``. This field **cannot**
+    #: be used in the configuration file. The name will be inherited from the mapping key instead.
+    test_suite_name: str
+    #: The names of test cases from this test suite to execute. If empty, all test cases will be
+    #: executed.
+    test_cases_names: list[str] = Field(default_factory=list, alias="test_cases")
+
+    @cached_property
+    def test_suite_spec(self) -> "TestSuiteSpec":
+        """The specification of the requested test suite."""
+        from framework.test_suite import find_by_name
+
+        test_suite_spec = find_by_name(self.test_suite_name)
+        assert (
+            test_suite_spec is not None
+        ), f"{self.test_suite_name} is not a valid test suite module name."
+        return test_suite_spec
+
+    @model_validator(mode="before")
+    @classmethod
+    def load_test_suite_name_from_context(cls, data: Any, info: ValidationInfo) -> dict:
+        """Load the test suite name from the validation context, if any."""
+        assert isinstance(data, dict), "The test suite configuration value is invalid."
+        name = data.get("test_suite_name")
+        # If the context is carrying the test suite name, then use it instead.
+        if info.context is not None and (test_suite_name := info.context.get("test_suite_name")):
+            assert not name, "The test suite name cannot be set manually."
+            data["test_suite_name"] = test_suite_name
+        return data
+
+    @model_validator(mode="before")
+    @classmethod
+    def convert_from_string(cls, data: Any) -> dict:
+        """Convert the string representation of the model into a valid mapping."""
+        if isinstance(data, str):
+            test_cases = [] if data == "all" else data.split()
+            return dict(test_cases=test_cases)
+        return data
+
+    @model_validator(mode="after")
+    def validate_names(self) -> Self:
+        """Validate the supplied test suite and test cases names.
+
+        This validator relies on the cached property `test_suite_spec` to run for the first
+        time in this call, therefore triggering the assertions if needed.
+        """
+        available_test_cases = map(
+            lambda t: t.name, self.test_suite_spec.class_obj.get_test_cases()
+        )
+        for requested_test_case in self.test_cases_names:
+            assert requested_test_case in available_test_cases, (
+                f"{requested_test_case} is not a valid test case "
+                f"of test suite {self.test_suite_name}."
+            )
+
+        return self
+
+
+class BaseTestSuitesConfigs(BaseModel):
+    """Base class for test suites configs."""
+
+    model_config = ConfigDict(frozen=True, extra="forbid")
+
+    def __contains__(self, key) -> bool:
+        """Check if the provided test suite name has been selected and/or configured."""
+        return key in self.model_fields_set
+
+    def __getitem__(self, key) -> TestSuiteConfig:
+        """Get test suite configuration."""
+        return self.__getattribute__(key)
+
+    def get_configs(self) -> Iterable[TestSuiteConfig]:
+        """Get all the test suite configurations."""
+        return map(lambda t: self[t], self.model_fields_set)
+
+    @classmethod
+    def available_test_suites(cls) -> Iterable[str]:
+        """List all the available test suites."""
+        return cls.model_fields.keys()
+
+    @field_validator("*", mode="before")
+    @classmethod
+    def pass_test_suite_name_to_config(cls, field_value: Any, info: ValidationInfo) -> Any:
+        """Before validating any :class:`TestSuiteConfig`, pass the test suite name via context."""
+        test_suite_name = info.field_name
+        assert test_suite_name is not None
+
+        assert info.context is not None, "A context dictionary is required to load test suites."
+        info.context.update({"test_suite_name": test_suite_name})
+
+        return field_value
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 5f5837a132..2ab8861f99 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -25,6 +25,8 @@
 from types import MethodType
 from typing import Iterable
 
+from pydantic import ValidationError
+
 from framework.testbed_model.capability import Capability, get_supported_capabilities
 from framework.testbed_model.sut_node import SutNode
 from framework.testbed_model.tg_node import TGNode
@@ -34,11 +36,17 @@
     DPDKPrecompiledBuildConfiguration,
     SutNodeConfiguration,
     TestRunConfiguration,
-    TestSuiteConfig,
     TGNodeConfiguration,
     load_config,
 )
-from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError
+from .config.generated import CUSTOM_CONFIG_TYPES
+from .config.test_suite import TestSuiteConfig
+from .exception import (
+    BlockingTestSuiteError,
+    ConfigurationError,
+    SSHTimeoutError,
+    TestCaseVerifyError,
+)
 from .logger import DTSLogger, DtsStage, get_dts_logger
 from .settings import SETTINGS
 from .test_result import (
@@ -141,12 +149,7 @@ def run(self) -> None:
                 self._logger.info(f"Running test run with SUT '{sut_node_config.name}'.")
                 self._init_random_seed(test_run_config)
                 test_run_result = self._result.add_test_run(test_run_config)
-                # we don't want to modify the original config, so create a copy
-                test_run_test_suites = list(
-                    SETTINGS.test_suites if SETTINGS.test_suites else test_run_config.test_suites
-                )
-                if not test_run_config.skip_smoke_tests:
-                    test_run_test_suites[:0] = [TestSuiteConfig(test_suite="smoke_tests")]
+                test_run_test_suites = self._prepare_test_suites(test_run_config)
                 try:
                     test_suites_with_cases = self._get_test_suites_with_cases(
                         test_run_test_suites, test_run_config.func, test_run_config.perf
@@ -203,6 +206,46 @@ def _check_dts_python_version(self) -> None:
             )
             self._logger.warning("Please use Python >= 3.10 instead.")
 
+    def _prepare_test_suites(self, test_run_config: TestRunConfiguration) -> list[TestSuiteConfig]:
+        if SETTINGS.test_suites:
+            test_suites_configs = []
+            for selected_test_suite, selected_test_cases in SETTINGS.test_suites:
+                if selected_test_suite in test_run_config.test_suites:
+                    config = test_run_config.test_suites[selected_test_suite].model_copy(
+                        update={"test_cases_names": selected_test_cases}
+                    )
+                else:
+                    try:
+                        config = CUSTOM_CONFIG_TYPES[selected_test_suite](
+                            test_suite_name=selected_test_suite, test_cases=selected_test_cases
+                        )
+                    except AssertionError as e:
+                        raise ConfigurationError(
+                            "Invalid test cases were selected "
+                            f"for test suite {selected_test_suite}."
+                        ) from e
+                    except ValidationError as e:
+                        raise ConfigurationError(
+                            f"Test suite {selected_test_suite} needs to be explicitly configured "
+                            "in order to be selected."
+                        ) from e
+                    except KeyError:
+                        # not a custom configuration
+                        config = TestSuiteConfig(
+                            test_suite_name=selected_test_suite, test_cases=selected_test_cases
+                        )
+                test_suites_configs.append(config)
+        else:
+            # we don't want to modify the original config, so create a copy
+            test_suites_configs = [
+                config.model_copy() for config in test_run_config.test_suites.get_configs()
+            ]
+
+        if not test_run_config.skip_smoke_tests:
+            test_suites_configs[:0] = [TestSuiteConfig(test_suite_name="smoke_tests")]
+
+        return test_suites_configs
+
     def _get_test_suites_with_cases(
         self,
         test_suite_configs: list[TestSuiteConfig],
@@ -236,7 +279,11 @@ def _get_test_suites_with_cases(
                 test_cases.extend(perf_test_cases)
 
             test_suites_with_cases.append(
-                TestSuiteWithCases(test_suite_class=test_suite_class, test_cases=test_cases)
+                TestSuiteWithCases(
+                    test_suite_class=test_suite_class,
+                    test_cases=test_cases,
+                    config=test_suite_config,
+                )
             )
         return test_suites_with_cases
 
@@ -453,7 +500,9 @@ def _run_test_suite(
         self._logger.set_stage(
             DtsStage.test_suite_setup, Path(SETTINGS.output_dir, test_suite_name)
         )
-        test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node, topology)
+        test_suite = test_suite_with_cases.test_suite_class(
+            sut_node, tg_node, topology, test_suite_with_cases.config
+        )
         try:
             self._logger.info(f"Starting test suite setup: {test_suite_name}")
             test_suite.set_up_suite()
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 5a8e6e5aee..f8783c4b59 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -107,7 +107,7 @@
     LocalDPDKTreeLocation,
     RemoteDPDKTarballLocation,
     RemoteDPDKTreeLocation,
-    TestSuiteConfig,
+    TestSuitesConfigs,
 )
 
 
@@ -133,7 +133,7 @@ class Settings:
     #:
     compile_timeout: float = 1200
     #:
-    test_suites: list[TestSuiteConfig] = field(default_factory=list)
+    test_suites: list[tuple[str, list[str]]] = field(default_factory=list)
     #:
     re_run: int = 0
     #:
@@ -508,7 +508,7 @@ def _process_dpdk_location(
 
 def _process_test_suites(
     parser: _DTSArgumentParser, args: list[list[str]]
-) -> list[TestSuiteConfig]:
+) -> list[tuple[str, list[str]]]:
     """Process the given argument to a list of :class:`TestSuiteConfig` to execute.
 
     Args:
@@ -524,19 +524,17 @@ def _process_test_suites(
         # Environment variable in the form of "SUITE1 CASE1 CASE2, SUITE2 CASE1, SUITE3, ..."
         args = [suite_with_cases.split() for suite_with_cases in args[0][0].split(",")]
 
-    try:
-        return [
-            TestSuiteConfig(test_suite=test_suite, test_cases=test_cases)
-            for [test_suite, *test_cases] in args
-        ]
-    except ValidationError as e:
-        print(
-            "An error has occurred while validating the test suites supplied in the "
-            f"{'environment variable' if action else 'arguments'}:",
-            file=sys.stderr,
-        )
-        print(e, file=sys.stderr)
-        sys.exit(1)
+    available_test_suites = TestSuitesConfigs.available_test_suites()
+    for test_suite_name, *_ in args:
+        if test_suite_name not in available_test_suites:
+            print(
+                f"The test suite {test_suite_name} supplied in the "
+                f"{'environment variable' if action else 'arguments'} is invalid.",
+                file=sys.stderr,
+            )
+            sys.exit(1)
+
+    return [(test_suite, test_cases) for test_suite, *test_cases in args]
 
 
 def get_settings() -> Settings:
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 6014d281b5..8c0c1bcfb3 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -30,7 +30,8 @@
 
 from framework.testbed_model.capability import Capability
 
-from .config import TestRunConfiguration, TestSuiteConfig
+from .config import TestRunConfiguration
+from .config.test_suite import TestSuiteConfig
 from .exception import DTSError, ErrorSeverity
 from .logger import DTSLogger
 from .settings import SETTINGS
@@ -59,23 +60,13 @@ class is to hold a subset of test cases (which could be all test cases) because
     test_suite_class: type[TestSuite]
     test_cases: list[type[TestCase]]
     required_capabilities: set[Capability] = field(default_factory=set, init=False)
+    config: TestSuiteConfig
 
     def __post_init__(self):
         """Gather the required capabilities of the test suite and all test cases."""
         for test_object in [self.test_suite_class] + self.test_cases:
             self.required_capabilities.update(test_object.required_capabilities)
 
-    def create_config(self) -> TestSuiteConfig:
-        """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
-
-        Returns:
-            The :class:`TestSuiteConfig` representation.
-        """
-        return TestSuiteConfig(
-            test_suite=self.test_suite_class.__name__,
-            test_cases=[test_case.__name__ for test_case in self.test_cases],
-        )
-
     def mark_skip_unsupported(self, supported_capabilities: set[Capability]) -> None:
         """Mark the test suite and test cases to be skipped.
 
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index fb5d646ce3..24cd0d38c1 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -24,13 +24,14 @@
 from ipaddress import IPv4Interface, IPv6Interface, ip_interface
 from pkgutil import iter_modules
 from types import ModuleType
-from typing import ClassVar, Protocol, TypeVar, Union, cast
+from typing import ClassVar, Protocol, TypeVar, Union, cast, get_type_hints
 
 from scapy.layers.inet import IP  # type: ignore[import-untyped]
 from scapy.layers.l2 import Ether  # type: ignore[import-untyped]
 from scapy.packet import Packet, Padding, raw  # type: ignore[import-untyped]
 from typing_extensions import Self
 
+from framework.config.test_suite import TestSuiteConfig
 from framework.testbed_model.capability import TestProtocol
 from framework.testbed_model.port import Port
 from framework.testbed_model.sut_node import SutNode
@@ -80,6 +81,7 @@ class TestSuite(TestProtocol):
     #: Whether the test suite is blocking. A failure of a blocking test suite
     #: will block the execution of all subsequent test suites in the current test run.
     is_blocking: ClassVar[bool] = False
+    config: TestSuiteConfig
     _logger: DTSLogger
     _sut_port_ingress: Port
     _sut_port_egress: Port
@@ -95,6 +97,7 @@ def __init__(
         sut_node: SutNode,
         tg_node: TGNode,
         topology: Topology,
+        config: TestSuiteConfig,
     ):
         """Initialize the test suite testbed information and basic configuration.
 
@@ -105,9 +108,11 @@ def __init__(
             sut_node: The SUT node where the test suite will run.
             tg_node: The TG node where the test suite will run.
             topology: The topology where the test suite will run.
+            config: The test suite configuration.
         """
         self.sut_node = sut_node
         self.tg_node = tg_node
+        self.config = config
         self._logger = get_dts_logger(self.__class__.__name__)
         self._tg_port_egress = topology.tg_port_egress
         self._sut_port_ingress = topology.sut_port_ingress
@@ -663,6 +668,21 @@ def is_test_suite(obj) -> bool:
             f"Expected class {self.class_name} not found in module {self.module_name}."
         )
 
+    @cached_property
+    def config_obj(self) -> type[TestSuiteConfig]:
+        """A reference to the test suite's configuration type."""
+        fields = get_type_hints(self.class_obj)
+        config_obj = fields.get("config")
+        if config_obj is None:
+            raise InternalError(
+                "Test suite class {self.class_name} is missing the `config` attribute."
+            )
+        if not issubclass(config_obj, TestSuiteConfig):
+            raise InternalError(
+                f"Test suite class {self.class_name} has an invalid configuration type assigned."
+            )
+        return config_obj
+
     @classmethod
     def discover_all(
         cls, package_name: str | None = None, module_prefix: str | None = None
diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
index 734f006026..f2998b968e 100644
--- a/dts/tests/TestSuite_hello_world.py
+++ b/dts/tests/TestSuite_hello_world.py
@@ -15,12 +15,15 @@
     LogicalCoreCountFilter,
     LogicalCoreList,
 )
+from tests.config import HelloWorldConfig
 
 
 @requires(topology_type=TopologyType.no_link)
 class TestHelloWorld(TestSuite):
     """DPDK hello world app test suite."""
 
+    config: HelloWorldConfig
+
     def set_up_suite(self) -> None:
         """Set up the test suite.
 
@@ -63,7 +66,7 @@ def hello_world_all_cores(self) -> None:
         eal_para = compute_eal_params(
             self.sut_node, lcore_filter_specifier=LogicalCoreList(self.sut_node.lcores)
         )
-        result = self.sut_node.run_dpdk_app(self.app_helloworld_path, eal_para, 50)
+        result = self.sut_node.run_dpdk_app(self.app_helloworld_path, eal_para, self.config.timeout)
         for lcore in self.sut_node.lcores:
             self.verify(
                 f"hello from core {int(lcore)}" in result.stdout,
diff --git a/dts/tests/__init__.py b/dts/tests/__init__.py
new file mode 100644
index 0000000000..a300eb26fc
--- /dev/null
+++ b/dts/tests/__init__.py
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Test suites.
+
+This package contains all the available test suites in DTS.
+"""
diff --git a/dts/tests/config.py b/dts/tests/config.py
new file mode 100644
index 0000000000..300ad3ef6a
--- /dev/null
+++ b/dts/tests/config.py
@@ -0,0 +1,20 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Module for test suites custom configurations.
+
+Any test suite that requires custom configuration fields should create a new config class inheriting
+:class:`~framework.config.test_suite.TestSuiteConfig`, while respecting the parents' frozen state.
+Any custom fields can be added in this class.
+
+The custom configuration classes can be stored in this module.
+"""
+
+from framework.config.test_suite import TestSuiteConfig
+
+
+class HelloWorldConfig(TestSuiteConfig):
+    """Example custom configuration for the `TestHelloWorld` test suite."""
+
+    #: Timeout for the DPDK apps.
+    timeout: int = 50
-- 
2.43.0


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH v3 4/4] dts: update autodoc sorting order
  2024-11-08 13:45 ` [PATCH v3 0/4] dts: add per-test-suite configuration Luca Vizzarro
                     ` (2 preceding siblings ...)
  2024-11-08 13:45   ` [PATCH v3 3/4] dts: add per-test-suite configuration Luca Vizzarro
@ 2024-11-08 13:45   ` Luca Vizzarro
  2024-11-12 20:04     ` Dean Marx
  2025-03-03 14:57   ` [PATCH v4 0/7] dts: add per-test-suite configuration Luca Vizzarro
  4 siblings, 1 reply; 27+ messages in thread
From: Luca Vizzarro @ 2024-11-08 13:45 UTC (permalink / raw)
  To: dev; +Cc: Paul Szczepanek, Patrick Robb, Luca Vizzarro

The autodoc member sorting order default is set to alphabetical, which
translates to autodoc sorting every member in modules, classes etc.
This also brings some side effects, like sorting capabilities which
can't be compared and result in errors.

This change prevents autodoc from sorting, and keeping the order as the
developer intended it.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 doc/guides/conf.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/guides/conf.py b/doc/guides/conf.py
index 71fed45b3d..ed5bc5eb30 100644
--- a/doc/guides/conf.py
+++ b/doc/guides/conf.py
@@ -88,6 +88,7 @@
     autodoc_typehints = 'both'
     autodoc_typehints_format = 'short'
     autodoc_typehints_description_target = 'documented'
+    autodoc_member_order = 'bysource'
 
     # Intersphinx allows linking to external projects, such as Python docs.
     intersphinx_mapping = {'python': ('https://docs.python.org/3', None)}
-- 
2.43.0


^ permalink raw reply	[flat|nested] 27+ messages in thread

* Re: [PATCH v3 1/4] dts: add tests package to API docs
  2024-11-08 13:45   ` [PATCH v3 1/4] dts: add tests package to API docs Luca Vizzarro
@ 2024-11-12 19:55     ` Dean Marx
  0 siblings, 0 replies; 27+ messages in thread
From: Dean Marx @ 2024-11-12 19:55 UTC (permalink / raw)
  To: Luca Vizzarro; +Cc: dev, Paul Szczepanek, Patrick Robb

[-- Attachment #1: Type: text/plain, Size: 43 bytes --]

Reviewed-by: Dean Marx <dmarx@iol.unh.edu>

[-- Attachment #2: Type: text/html, Size: 109 bytes --]

^ permalink raw reply	[flat|nested] 27+ messages in thread

* Re: [PATCH v2 2/4] dts: fix smoke tests docstring
  2024-11-08 13:38   ` [PATCH v2 2/4] dts: fix smoke tests docstring Luca Vizzarro
@ 2024-11-12 20:03     ` Dean Marx
  0 siblings, 0 replies; 27+ messages in thread
From: Dean Marx @ 2024-11-12 20:03 UTC (permalink / raw)
  To: Luca Vizzarro; +Cc: dev, Paul Szczepanek, Patrick Robb

[-- Attachment #1: Type: text/plain, Size: 43 bytes --]

Reviewed-by: Dean Marx <dmarx@iol.unh.edu>

[-- Attachment #2: Type: text/html, Size: 109 bytes --]

^ permalink raw reply	[flat|nested] 27+ messages in thread

* Re: [PATCH v3 4/4] dts: update autodoc sorting order
  2024-11-08 13:45   ` [PATCH v3 4/4] dts: update autodoc sorting order Luca Vizzarro
@ 2024-11-12 20:04     ` Dean Marx
  0 siblings, 0 replies; 27+ messages in thread
From: Dean Marx @ 2024-11-12 20:04 UTC (permalink / raw)
  To: Luca Vizzarro; +Cc: dev, Paul Szczepanek, Patrick Robb

[-- Attachment #1: Type: text/plain, Size: 43 bytes --]

Reviewed-by: Dean Marx <dmarx@iol.unh.edu>

[-- Attachment #2: Type: text/html, Size: 109 bytes --]

^ permalink raw reply	[flat|nested] 27+ messages in thread

* Re: [PATCH v3 3/4] dts: add per-test-suite configuration
  2024-11-08 13:45   ` [PATCH v3 3/4] dts: add per-test-suite configuration Luca Vizzarro
@ 2024-11-13 21:44     ` Dean Marx
  2024-11-14 12:39       ` Luca Vizzarro
  0 siblings, 1 reply; 27+ messages in thread
From: Dean Marx @ 2024-11-13 21:44 UTC (permalink / raw)
  To: Luca Vizzarro; +Cc: dev, Paul Szczepanek, Patrick Robb

[-- Attachment #1: Type: text/plain, Size: 2735 bytes --]

>
>
> On Fri, Nov 8, 2024 at 8:38 AM Luca Vizzarro <luca.vizzarro@arm.com>
> wrote:
>
>> Allow test suites to be configured individually. Moreover enable them to
>> implement their own custom configuration.
>>
>> This solution adds some new complexity to DTS, which is generated source
>> code. In order to ensure strong typing, the test suites and their custom
>> configurations need to be linked in the main configuration class.
>> Unfortunately, this is not feasible during runtime as it will incur in
>> circular dependencies. Generating the links appear to be the most
>> straightforward approach.
>>
>> This commit also brings a new major change to the configuration schema.
>> Test suites are no longer defined as a list of strings, like:
>>
>>     test_suites:
>>     - hello_world
>>     - pmd_buffer_scatter
>>
>> but as mapping of mappings or strings:
>>
>>     test_suites:
>>       hello_world: {} # any custom fields or test cases can be set here
>>       pmd_buffer_scatter: all # "all" defines all the test cases, or
>>                               # they can individually be set separated
>>                               # by a space
>>
>> Not defining the `test_cases` field in the configuration is equivalent
>> to `all`, therefore the definitions for either test suite above are
>> also equivalent.
>>
>> Creating the __init__.py file under the tests folder, allows it to be
>> picked up as a package. This is a mypy requirement to import the tests
>> from within the framework.
>>
>> Bugzilla ID: 1375
>>
>> Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
>> Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
>>
>
> I like the idea of mapping the suite to specific test cases, and for the
> most part the custom configuration option as well. The only thing that I
> feel should be different is the way the code generation is documented, I
> think it might be worth providing an example within conf.yaml through a
> comment near the suites section, rather than just in the dts.rst file. It
> might be a little more clear where to create the custom config class as
> well.
>
>  <snip>
>
>> +class HelloWorldConfig(TestSuiteConfig):
>> +    """Example custom configuration for the `TestHelloWorld` test
>> suite."""
>> +
>> +    #: Timeout for the DPDK apps.
>> +    timeout: int = 50
>> --
>> 2.43.0
>>
>>
> Additionally, I was a bit confused by the custom config examples, do these
> fields (timeout, my_custom_field) actually affect the suite in any way as
> of this patch? Or is this just so that we can potentially add configuration
> options through this method in the future?
>
> Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
>

[-- Attachment #2: Type: text/html, Size: 3577 bytes --]

^ permalink raw reply	[flat|nested] 27+ messages in thread

* Re: [PATCH v3 3/4] dts: add per-test-suite configuration
  2024-11-13 21:44     ` Dean Marx
@ 2024-11-14 12:39       ` Luca Vizzarro
  2024-11-19 16:31         ` Dean Marx
  0 siblings, 1 reply; 27+ messages in thread
From: Luca Vizzarro @ 2024-11-14 12:39 UTC (permalink / raw)
  To: Dean Marx; +Cc: dev, Paul Szczepanek, Patrick Robb

Hi Dean!

Thank you for your review!

On 13/11/2024 21:44, Dean Marx wrote:
> I like the idea of mapping the suite to specific test cases, and for
> the most part the custom configuration option as well. The only
> thing that I feel should be different is the way the code generation
> is documented, I think it might be worth providing an example within
> conf.yaml through a comment near the suites section, rather than
> just in the dts.rst file. It might be a little more clear where to
> create the custom config class as well.

So... this is a weird one. I've integrated the code generation under 
dts-check-format.sh, so nobody should really worry about it, as that 
should be run before submitting. If it's not, then the CI will 
eventually pick it up when dts-check-format will be in place.

>> +class HelloWorldConfig(TestSuiteConfig):
>> +    """Example custom configuration for the `TestHelloWorld`
>> test suite."""
>> +
>> +    #: Timeout for the DPDK apps.
>> +    timeout: int = 50
>> --
>> 2.43.0
> 
> Additionally, I was a bit confused by the custom config examples, do
> these fields (timeout, my_custom_field) actually affect the suite in
> any way as of this patch? Or is this just so that we can potentially
> add configuration options through this method in the future?

The hello_world suite doesn't really need this, I've just added it as an 
example. But yes test suites are affected. If you see the hello_world 
suite, it calls `self.config.timeout` which is a reference to this 
field. This class specifically is now wholly integrated in the 
configuration:

   test_runs:
   - test_suites:
       hello_world:  # this is a mapping to HelloWorldConfig
         # like before the behavior of this field is:
	# - if unset or empty list, all test cases are run
         # - if individual test cases are listed, those are run
         test_cases: [single_core] # field inherited from TestSuiteConfig
         timeout: 20               # optional field from HelloWorldConfig

What I've added for the simplicity is to be able to represent any 
TestSuiteConfig as a string instead of a mapping:

   test_runs:
   - test_suites:
       hello_world: all # special keyword which enables all test cases
       # or
       hello_world: single_core all_cores # a whitespace-delimited list 
of test suites

Mind that if the underlying test suite configuration has a mandatory 
field that has no default, then the string representation will fail and 
Pydantic will complain that the (required) field is missing.

Please let me know if I answered every query, and don't hesitate to ask 
for further clarifications.

Best,
Luca

^ permalink raw reply	[flat|nested] 27+ messages in thread

* Re: [PATCH v3 3/4] dts: add per-test-suite configuration
  2024-11-14 12:39       ` Luca Vizzarro
@ 2024-11-19 16:31         ` Dean Marx
  2024-11-19 16:41           ` Luca Vizzarro
  0 siblings, 1 reply; 27+ messages in thread
From: Dean Marx @ 2024-11-19 16:31 UTC (permalink / raw)
  To: Luca Vizzarro; +Cc: dev, Paul Szczepanek, Patrick Robb

> So... this is a weird one. I've integrated the code generation under
> dts-check-format.sh, so nobody should really worry about it, as that
> should be run before submitting. If it's not, then the CI will
> eventually pick it up when dts-check-format will be in place.

Right, and that definitely makes sense logically but I think maybe
even just throwing in a comment in conf.yaml similar to the one you
used below (timeout: 20               # optional field from
HelloWorldConfig) would be beneficial, even if just to show where the
customized fields should go. If you'd rather leave it as is though
that's totally fine with me, this is kind of a nitpick suggestion
anyways

> The hello_world suite doesn't really need this, I've just added it as an
> example. But yes test suites are affected. If you see the hello_world
> suite, it calls `self.config.timeout` which is a reference to this
> field. This class specifically is now wholly integrated in the
> configuration:

Okay that makes sense, thanks Luca!

^ permalink raw reply	[flat|nested] 27+ messages in thread

* Re: [PATCH v3 3/4] dts: add per-test-suite configuration
  2024-11-19 16:31         ` Dean Marx
@ 2024-11-19 16:41           ` Luca Vizzarro
  0 siblings, 0 replies; 27+ messages in thread
From: Luca Vizzarro @ 2024-11-19 16:41 UTC (permalink / raw)
  To: Dean Marx; +Cc: dev, Paul Szczepanek, Patrick Robb

On 19/11/2024 16:31, Dean Marx wrote:
>> So... this is a weird one. I've integrated the code generation under
>> dts-check-format.sh, so nobody should really worry about it, as that
>> should be run before submitting. If it's not, then the CI will
>> eventually pick it up when dts-check-format will be in place.
> 
> Right, and that definitely makes sense logically but I think maybe
> even just throwing in a comment in conf.yaml similar to the one you
> used below (timeout: 20               # optional field from
> HelloWorldConfig) would be beneficial, even if just to show where the
> customized fields should go. If you'd rather leave it as is though
> that's totally fine with me, this is kind of a nitpick suggestion
> anyways

Oh ok, yes it makes sense. It's a good shout. Although, this patchset 
may change how it will be once we start splitting the configuration file 
in multiple ones. I guess we'll see what happens then! This will 
probably stay on standby until we reach an agreement on what to do.

Thank you for your review!

Luca

^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH v4 0/7] dts: add per-test-suite configuration
  2024-11-08 13:45 ` [PATCH v3 0/4] dts: add per-test-suite configuration Luca Vizzarro
                     ` (3 preceding siblings ...)
  2024-11-08 13:45   ` [PATCH v3 4/4] dts: update autodoc sorting order Luca Vizzarro
@ 2025-03-03 14:57   ` Luca Vizzarro
  2025-03-03 14:57     ` [PATCH v4 1/7] dts: add tests package to API docs Luca Vizzarro
                       ` (6 more replies)
  4 siblings, 7 replies; 27+ messages in thread
From: Luca Vizzarro @ 2025-03-03 14:57 UTC (permalink / raw)
  To: dev; +Cc: Luca Vizzarro, Patrick Robb, Paul Szczepanek

From: Luca Vizzarro <luca.vizzarro@arm.com>

Hi there,

sending in v4 for the per-test-suite configuration patchset.

v4:
- rebased
- Changed implementation:
  - test suites configurations are now provided
    through a dedicated optional configuration file.
  - running multiple test runs hinders flexibility in loading
    configuration, therefore DTS now accepts only one test run.
  - the tests folder is no longer a package as this is no longer
    imported directly. The implementation now relies on the test
    suite globbing imports, then the configuration class is referenced
    through the annotation for TestSuite.config.
  - as a consequence of the above, a code generator is no longer needed.
- improved the way configuration errors are displayed.
- adding the test suites to the docs resulted in several warnings,
  minimally required fixes are added.

Best,
Luca

Luca Vizzarro (7):
  dts: add tests package to API docs
  dts: amend test suites docstring
  dts: fix smoke tests docstring
  dts: update autodoc sorting order
  dts: run only one test run per execution
  dts: add per-test-suite configuration
  dts: improve configuration errors

 doc/api/dts/index.rst                         |   1 +
 doc/api/dts/tests.TestSuite_blocklist.rst     |   8 ++
 .../dts/tests.TestSuite_checksum_offload.rst  |   8 ++
 doc/api/dts/tests.TestSuite_dual_vlan.rst     |   8 ++
 .../dts/tests.TestSuite_dynamic_config.rst    |   8 ++
 .../tests.TestSuite_dynamic_queue_conf.rst    |   8 ++
 doc/api/dts/tests.TestSuite_hello_world.rst   |   8 ++
 doc/api/dts/tests.TestSuite_l2fwd.rst         |   8 ++
 doc/api/dts/tests.TestSuite_mac_filter.rst    |   8 ++
 doc/api/dts/tests.TestSuite_mtu.rst           |   8 ++
 .../tests.TestSuite_pmd_buffer_scatter.rst    |   8 ++
 ...tSuite_port_restart_config_persistency.rst |   8 ++
 .../dts/tests.TestSuite_promisc_support.rst   |   8 ++
 .../dts/tests.TestSuite_queue_start_stop.rst  |   8 ++
 doc/api/dts/tests.TestSuite_smoke_tests.rst   |   8 ++
 doc/api/dts/tests.TestSuite_softnic.rst       |   8 ++
 doc/api/dts/tests.TestSuite_uni_pkt.rst       |   8 ++
 doc/api/dts/tests.TestSuite_vlan.rst          |   8 ++
 doc/api/dts/tests.rst                         |  11 ++
 doc/guides/conf.py                            |   1 +
 doc/guides/tools/dts.rst                      |  25 ++--
 dts/.gitignore                                |   2 +-
 dts/framework/config/__init__.py              | 112 ++++++++++-------
 dts/framework/config/test_run.py              |  53 ++++++--
 dts/framework/runner.py                       |  33 +++--
 dts/framework/settings.py                     |  33 +++--
 dts/framework/test_run.py                     |  23 ++--
 dts/framework/test_suite.py                   |  25 ++--
 dts/test_run.example.yaml                     |  43 +++++++
 dts/test_runs.example.yaml                    |  43 -------
 dts/tests/TestSuite_dynamic_config.py         |  10 +-
 dts/tests/TestSuite_hello_world.py            |  14 ++-
 dts/tests/TestSuite_mac_filter.py             |  42 +++----
 dts/tests/TestSuite_mtu.py                    | 114 +++++++++---------
 dts/tests/TestSuite_smoke_tests.py            |   2 -
 dts/tests_config.example.yaml                 |   2 +
 36 files changed, 495 insertions(+), 230 deletions(-)
 create mode 100644 doc/api/dts/tests.TestSuite_blocklist.rst
 create mode 100644 doc/api/dts/tests.TestSuite_checksum_offload.rst
 create mode 100644 doc/api/dts/tests.TestSuite_dual_vlan.rst
 create mode 100644 doc/api/dts/tests.TestSuite_dynamic_config.rst
 create mode 100644 doc/api/dts/tests.TestSuite_dynamic_queue_conf.rst
 create mode 100644 doc/api/dts/tests.TestSuite_hello_world.rst
 create mode 100644 doc/api/dts/tests.TestSuite_l2fwd.rst
 create mode 100644 doc/api/dts/tests.TestSuite_mac_filter.rst
 create mode 100644 doc/api/dts/tests.TestSuite_mtu.rst
 create mode 100644 doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
 create mode 100644 doc/api/dts/tests.TestSuite_port_restart_config_persistency.rst
 create mode 100644 doc/api/dts/tests.TestSuite_promisc_support.rst
 create mode 100644 doc/api/dts/tests.TestSuite_queue_start_stop.rst
 create mode 100644 doc/api/dts/tests.TestSuite_smoke_tests.rst
 create mode 100644 doc/api/dts/tests.TestSuite_softnic.rst
 create mode 100644 doc/api/dts/tests.TestSuite_uni_pkt.rst
 create mode 100644 doc/api/dts/tests.TestSuite_vlan.rst
 create mode 100644 doc/api/dts/tests.rst
 create mode 100644 dts/test_run.example.yaml
 delete mode 100644 dts/test_runs.example.yaml
 create mode 100644 dts/tests_config.example.yaml

-- 
2.43.0


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH v4 1/7] dts: add tests package to API docs
  2025-03-03 14:57   ` [PATCH v4 0/7] dts: add per-test-suite configuration Luca Vizzarro
@ 2025-03-03 14:57     ` Luca Vizzarro
  2025-03-03 14:57     ` [PATCH v4 2/7] dts: amend test suites docstring Luca Vizzarro
                       ` (5 subsequent siblings)
  6 siblings, 0 replies; 27+ messages in thread
From: Luca Vizzarro @ 2025-03-03 14:57 UTC (permalink / raw)
  To: dev; +Cc: Luca Vizzarro, Paul Szczepanek, Patrick Robb

From: Luca Vizzarro <luca.vizzarro@arm.com>

The test suites are not documented in the API as their respective
documentation files are missing.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 doc/api/dts/index.rst                                 |  1 +
 doc/api/dts/tests.TestSuite_blocklist.rst             |  8 ++++++++
 doc/api/dts/tests.TestSuite_checksum_offload.rst      |  8 ++++++++
 doc/api/dts/tests.TestSuite_dual_vlan.rst             |  8 ++++++++
 doc/api/dts/tests.TestSuite_dynamic_config.rst        |  8 ++++++++
 doc/api/dts/tests.TestSuite_dynamic_queue_conf.rst    |  8 ++++++++
 doc/api/dts/tests.TestSuite_hello_world.rst           |  8 ++++++++
 doc/api/dts/tests.TestSuite_l2fwd.rst                 |  8 ++++++++
 doc/api/dts/tests.TestSuite_mac_filter.rst            |  8 ++++++++
 doc/api/dts/tests.TestSuite_mtu.rst                   |  8 ++++++++
 doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst    |  8 ++++++++
 ...ests.TestSuite_port_restart_config_persistency.rst |  8 ++++++++
 doc/api/dts/tests.TestSuite_promisc_support.rst       |  8 ++++++++
 doc/api/dts/tests.TestSuite_queue_start_stop.rst      |  8 ++++++++
 doc/api/dts/tests.TestSuite_smoke_tests.rst           |  8 ++++++++
 doc/api/dts/tests.TestSuite_softnic.rst               |  8 ++++++++
 doc/api/dts/tests.TestSuite_uni_pkt.rst               |  8 ++++++++
 doc/api/dts/tests.TestSuite_vlan.rst                  |  8 ++++++++
 doc/api/dts/tests.rst                                 | 11 +++++++++++
 19 files changed, 148 insertions(+)
 create mode 100644 doc/api/dts/tests.TestSuite_blocklist.rst
 create mode 100644 doc/api/dts/tests.TestSuite_checksum_offload.rst
 create mode 100644 doc/api/dts/tests.TestSuite_dual_vlan.rst
 create mode 100644 doc/api/dts/tests.TestSuite_dynamic_config.rst
 create mode 100644 doc/api/dts/tests.TestSuite_dynamic_queue_conf.rst
 create mode 100644 doc/api/dts/tests.TestSuite_hello_world.rst
 create mode 100644 doc/api/dts/tests.TestSuite_l2fwd.rst
 create mode 100644 doc/api/dts/tests.TestSuite_mac_filter.rst
 create mode 100644 doc/api/dts/tests.TestSuite_mtu.rst
 create mode 100644 doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
 create mode 100644 doc/api/dts/tests.TestSuite_port_restart_config_persistency.rst
 create mode 100644 doc/api/dts/tests.TestSuite_promisc_support.rst
 create mode 100644 doc/api/dts/tests.TestSuite_queue_start_stop.rst
 create mode 100644 doc/api/dts/tests.TestSuite_smoke_tests.rst
 create mode 100644 doc/api/dts/tests.TestSuite_softnic.rst
 create mode 100644 doc/api/dts/tests.TestSuite_uni_pkt.rst
 create mode 100644 doc/api/dts/tests.TestSuite_vlan.rst
 create mode 100644 doc/api/dts/tests.rst

diff --git a/doc/api/dts/index.rst b/doc/api/dts/index.rst
index 33b05953d2..a11f395e11 100644
--- a/doc/api/dts/index.rst
+++ b/doc/api/dts/index.rst
@@ -14,6 +14,7 @@ Packages
    :includehidden:
    :maxdepth: 1
 
+   tests
    framework.testbed_model
    framework.remote_session
    framework.params
diff --git a/doc/api/dts/tests.TestSuite_blocklist.rst b/doc/api/dts/tests.TestSuite_blocklist.rst
new file mode 100644
index 0000000000..8d032786a0
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_blocklist.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+blocklist Test Suite
+====================
+
+.. automodule:: tests.TestSuite_blocklist
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.TestSuite_checksum_offload.rst b/doc/api/dts/tests.TestSuite_checksum_offload.rst
new file mode 100644
index 0000000000..8eea87f2e9
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_checksum_offload.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+checksum_offload Test Suite
+===========================
+
+.. automodule:: tests.TestSuite_checksum_offload
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.TestSuite_dual_vlan.rst b/doc/api/dts/tests.TestSuite_dual_vlan.rst
new file mode 100644
index 0000000000..22d8c7195f
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_dual_vlan.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+dual_vlan Test Suite
+====================
+
+.. automodule:: tests.TestSuite_dual_vlan
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.TestSuite_dynamic_config.rst b/doc/api/dts/tests.TestSuite_dynamic_config.rst
new file mode 100644
index 0000000000..268ef499b6
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_dynamic_config.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+dynamic_config Test Suite
+=========================
+
+.. automodule:: tests.TestSuite_dynamic_config
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.TestSuite_dynamic_queue_conf.rst b/doc/api/dts/tests.TestSuite_dynamic_queue_conf.rst
new file mode 100644
index 0000000000..c142b9dc40
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_dynamic_queue_conf.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+dynamic_queue_conf Test Suite
+=============================
+
+.. automodule:: tests.TestSuite_dynamic_queue_conf
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.TestSuite_hello_world.rst b/doc/api/dts/tests.TestSuite_hello_world.rst
new file mode 100644
index 0000000000..10db998987
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_hello_world.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+hello_world Test Suite
+======================
+
+.. automodule:: tests.TestSuite_hello_world
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.TestSuite_l2fwd.rst b/doc/api/dts/tests.TestSuite_l2fwd.rst
new file mode 100644
index 0000000000..5a0ca312e9
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_l2fwd.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+l2fwd Test Suite
+================
+
+.. automodule:: tests.TestSuite_l2fwd
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.TestSuite_mac_filter.rst b/doc/api/dts/tests.TestSuite_mac_filter.rst
new file mode 100644
index 0000000000..a65aaeac0b
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_mac_filter.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+mac_filter Test Suite
+=====================
+
+.. automodule:: tests.TestSuite_mac_filter
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.TestSuite_mtu.rst b/doc/api/dts/tests.TestSuite_mtu.rst
new file mode 100644
index 0000000000..6e8cb808da
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_mtu.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+mtu Test Suite
+==============
+
+.. automodule:: tests.TestSuite_mtu
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst b/doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
new file mode 100644
index 0000000000..cdf30fd879
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+pmd_buffer_scatter Test Suite
+=============================
+
+.. automodule:: tests.TestSuite_pmd_buffer_scatter
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.TestSuite_port_restart_config_persistency.rst b/doc/api/dts/tests.TestSuite_port_restart_config_persistency.rst
new file mode 100644
index 0000000000..2db4754a01
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_port_restart_config_persistency.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+port_restart_config_persistency Test Suite
+==========================================
+
+.. automodule:: tests.TestSuite_port_restart_config_persistency
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.TestSuite_promisc_support.rst b/doc/api/dts/tests.TestSuite_promisc_support.rst
new file mode 100644
index 0000000000..496ee01f2f
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_promisc_support.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+promisc_support Test Suite
+==========================
+
+.. automodule:: tests.TestSuite_promisc_support
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.TestSuite_queue_start_stop.rst b/doc/api/dts/tests.TestSuite_queue_start_stop.rst
new file mode 100644
index 0000000000..87121676fb
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_queue_start_stop.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+queue_start_stop Test Suite
+===========================
+
+.. automodule:: tests.TestSuite_queue_start_stop
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.TestSuite_smoke_tests.rst b/doc/api/dts/tests.TestSuite_smoke_tests.rst
new file mode 100644
index 0000000000..8c78c3025a
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_smoke_tests.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+smoke_tests Test Suite
+======================
+
+.. automodule:: tests.TestSuite_smoke_tests
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.TestSuite_softnic.rst b/doc/api/dts/tests.TestSuite_softnic.rst
new file mode 100644
index 0000000000..a091f265ea
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_softnic.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+softnic Test Suite
+==================
+
+.. automodule:: tests.TestSuite_softnic
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.TestSuite_uni_pkt.rst b/doc/api/dts/tests.TestSuite_uni_pkt.rst
new file mode 100644
index 0000000000..95c5a5a28b
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_uni_pkt.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+uni_pkt Test Suite
+==================
+
+.. automodule:: tests.TestSuite_uni_pkt
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.TestSuite_vlan.rst b/doc/api/dts/tests.TestSuite_vlan.rst
new file mode 100644
index 0000000000..dfbe6196fd
--- /dev/null
+++ b/doc/api/dts/tests.TestSuite_vlan.rst
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+vlan Test Suite
+===============
+
+.. automodule:: tests.TestSuite_vlan
+   :members:
+   :show-inheritance:
diff --git a/doc/api/dts/tests.rst b/doc/api/dts/tests.rst
new file mode 100644
index 0000000000..fdfa6c1f8d
--- /dev/null
+++ b/doc/api/dts/tests.rst
@@ -0,0 +1,11 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+
+Test Suites
+===========
+
+.. toctree::
+   :glob:
+   :hidden:
+   :maxdepth: 1
+
+   tests.*
-- 
2.43.0


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH v4 2/7] dts: amend test suites docstring
  2025-03-03 14:57   ` [PATCH v4 0/7] dts: add per-test-suite configuration Luca Vizzarro
  2025-03-03 14:57     ` [PATCH v4 1/7] dts: add tests package to API docs Luca Vizzarro
@ 2025-03-03 14:57     ` Luca Vizzarro
  2025-03-03 14:57     ` [PATCH v4 3/7] dts: fix smoke tests docstring Luca Vizzarro
                       ` (4 subsequent siblings)
  6 siblings, 0 replies; 27+ messages in thread
From: Luca Vizzarro @ 2025-03-03 14:57 UTC (permalink / raw)
  To: dev; +Cc: Luca Vizzarro, Paul Szczepanek, Patrick Robb

From: Luca Vizzarro <luca.vizzarro@arm.com>

Amend the test suites docstring to conform to valid rst.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 dts/tests/TestSuite_dynamic_config.py |  10 +--
 dts/tests/TestSuite_mac_filter.py     |  42 +++++-----
 dts/tests/TestSuite_mtu.py            | 114 +++++++++++++-------------
 3 files changed, 83 insertions(+), 83 deletions(-)

diff --git a/dts/tests/TestSuite_dynamic_config.py b/dts/tests/TestSuite_dynamic_config.py
index a4bee2e90b..1fce31a0b5 100644
--- a/dts/tests/TestSuite_dynamic_config.py
+++ b/dts/tests/TestSuite_dynamic_config.py
@@ -36,13 +36,13 @@ class TestDynamicConfig(TestSuite):
 
     1. Default mode: verify packets are received and forwarded.
     2. Disable promiscuous mode: verify that packets are received
-    only for the packet with destination address matching the port address.
+       only for the packet with destination address matching the port address.
     3. Disable promiscuous mode broadcast: verify that packets with destination
-    MAC address not matching the port are received and not forwarded, and verify
-    that broadcast packets are received and forwarded.
+       MAC address not matching the port are received and not forwarded, and verify
+       that broadcast packets are received and forwarded.
     4. Disable promiscuous mode multicast: verify that packets with destination
-    MAC address not matching the port are received and not forwarded, and verify
-    that multicast packets are received and forwarded.
+       MAC address not matching the port are received and not forwarded, and verify
+       that multicast packets are received and forwarded.
     """
 
     def send_packet_and_verify(self, should_receive: bool, mac_address: str) -> None:
diff --git a/dts/tests/TestSuite_mac_filter.py b/dts/tests/TestSuite_mac_filter.py
index e6c55d3ec6..9dbfec5da2 100644
--- a/dts/tests/TestSuite_mac_filter.py
+++ b/dts/tests/TestSuite_mac_filter.py
@@ -93,13 +93,13 @@ def test_add_remove_mac_addresses(self) -> None:
         the properties applied to the PMD at any given time.
 
         Test:
-            Start TestPMD without promiscuous mode.
-            Send a packet with the port's default mac address. (Should receive)
-            Send a packet with fake mac address. (Should not receive)
-            Add fake mac address to the PMD's address pool.
-            Send a packet with the fake mac address to the PMD. (Should receive)
-            Remove the fake mac address from the PMD's address pool.
-            Send a packet with the fake mac address to the PMD. (Should not receive)
+            * Start TestPMD without promiscuous mode.
+            * Send a packet with the port's default mac address. (Should receive)
+            * Send a packet with fake mac address. (Should not receive)
+            * Add fake mac address to the PMD's address pool.
+            * Send a packet with the fake mac address to the PMD. (Should receive)
+            * Remove the fake mac address from the PMD's address pool.
+            * Send a packet with the fake mac address to the PMD. (Should not receive)
         """
         with TestPmdShell() as testpmd:
             testpmd.set_promisc(0, enable=False)
@@ -127,15 +127,15 @@ def test_invalid_address(self) -> None:
         built-in hardware address, or exceed their address pools.
 
         Test:
-            Start TestPMD.
-            Attempt to add an invalid mac address. (Should fail)
-            Attempt to remove the device's hardware address with no additional addresses in the
-                address pool. (Should fail)
-            Add a fake mac address to the pool twice in succession. (Should not create any errors)
-            Attempt to remove the device's hardware address with other addresses in the address
-                pool. (Should fail)
-            Determine the device's mac address pool size, and fill the pool with fake addresses.
-            Attempt to add another fake mac address, overloading the address pool. (Should fail)
+            * Start TestPMD.
+            * Attempt to add an invalid mac address. (Should fail)
+            * Attempt to remove the device's hardware address with no additional addresses in the
+              address pool. (Should fail)
+            * Add a fake mac address to the pool twice in succession. (Should not create any errors)
+            * Attempt to remove the device's hardware address with other addresses in the address
+              pool. (Should fail)
+            * Determine the device's mac address pool size, and fill the pool with fake addresses.
+            * Attempt to add another fake mac address, overloading the address pool. (Should fail)
         """
         with TestPmdShell() as testpmd:
             testpmd.start()
@@ -185,11 +185,11 @@ def test_multicast_filter(self) -> None:
         to the PMD.
 
         Test:
-            Start TestPMD without promiscuous mode.
-            Add a fake multicast address to the PMD's multicast address pool.
-            Send a packet with the fake multicast address to the PMD. (Should receive)
-            Remove the fake multicast address from the PMDs multicast address filter.
-            Send a packet with the fake multicast address to the PMD. (Should not receive)
+            * Start TestPMD without promiscuous mode.
+            * Add a fake multicast address to the PMD's multicast address pool.
+            * Send a packet with the fake multicast address to the PMD. (Should receive)
+            * Remove the fake multicast address from the PMDs multicast address filter.
+            * Send a packet with the fake multicast address to the PMD. (Should not receive)
         """
         with TestPmdShell() as testpmd:
             testpmd.start()
diff --git a/dts/tests/TestSuite_mtu.py b/dts/tests/TestSuite_mtu.py
index 63e570ba03..af6ab88501 100644
--- a/dts/tests/TestSuite_mtu.py
+++ b/dts/tests/TestSuite_mtu.py
@@ -48,8 +48,7 @@ def set_up_suite(self) -> None:
         """Set up the test suite.
 
         Setup:
-            Set traffic generator MTU lengths to a size greater than scope of all
-            test cases.
+            Set traffic generator MTU lengths to a size greater than scope of all test cases.
         """
         self.topology.tg_port_egress.configure_mtu(JUMBO_MTU + 200)
         self.topology.tg_port_ingress.configure_mtu(JUMBO_MTU + 200)
@@ -89,12 +88,13 @@ def assess_mtu_boundary(self, testpmd_shell: TestPmdShell, mtu: int) -> None:
         will not.
 
         First, start testpmd and update the MTU. Then ensure the new value appears
-        on port info for all ports.
-        Next, start packet capturing and send 3 different lengths of packet and verify
-        they are handled correctly.
-            # 1. VENDOR_AGNOSTIC_PADDING units smaller than the MTU specified.
-            # 2. Equal to the MTU specified.
-            # 3. VENDOR_AGNOSTIC_PADDING units larger than the MTU specified (should be fragmented).
+        on port info for all ports. Next, start packet capturing and send 3 different lengths of
+        packet and verify they are handled correctly:
+
+        1. VENDOR_AGNOSTIC_PADDING units smaller than the MTU specified.
+        2. Equal to the MTU specified.
+        3. VENDOR_AGNOSTIC_PADDING units larger than the MTU specified (should be fragmented).
+
         Finally, stop packet capturing.
 
         Args:
@@ -126,35 +126,35 @@ def test_runtime_mtu_updating_and_forwarding(self) -> None:
         """Verify runtime MTU adjustments and assess packet forwarding behavior.
 
         Test:
-            Start TestPMD in a paired topology.
-            Set port MTU to 1500.
-            Send packets of 1491, 1500 and 1509 bytes.
-                Verify the first two packets are forwarded and the last is dropped.
-
-            Set port MTU to 2400.
-            Send packets of 1491, 1500 and 1509 bytes.
-                Verify all three packets are forwarded.
-            Send packets of 2391, 2400 and 2409 bytes.
-                Verify the first two packets are forwarded and the last is dropped.
-
-            Set port MTU to 4800.
-            Send packets of 1491, 1500 and 1509 bytes.
-                Verify all three packets are forwarded.
-            Send packets of 4791, 4800 and 4809 bytes.
-                Verify the first two packets are forwarded and the last is dropped.
-
-            Set port MTU to 9000.
-            Send packets of 1491, 1500 and 1509 bytes.
-                Verify all three packets are forwarded.
-            Send packets of 8991, 9000 and 9009 bytes.
-                Verify the first two packets are forwarded and the last is dropped.
+            * Start TestPMD in a paired topology.
+            * Set port MTU to 1500.
+            * Send packets of 1491, 1500 and 1509 bytes.
+                * Verify the first two packets are forwarded and the last is dropped.
+
+            * Set port MTU to 2400.
+            * Send packets of 1491, 1500 and 1509 bytes.
+                * Verify all three packets are forwarded.
+            * Send packets of 2391, 2400 and 2409 bytes.
+                * Verify the first two packets are forwarded and the last is dropped.
+
+            * Set port MTU to 4800.
+            * Send packets of 1491, 1500 and 1509 bytes.
+                * Verify all three packets are forwarded.
+            * Send packets of 4791, 4800 and 4809 bytes.
+                * Verify the first two packets are forwarded and the last is dropped.
+
+            * Set port MTU to 9000.
+            * Send packets of 1491, 1500 and 1509 bytes.
+                * Verify all three packets are forwarded.
+            * Send packets of 8991, 9000 and 9009 bytes.
+                * Verify the first two packets are forwarded and the last is dropped.
         Verify:
-            Verifies the successful forwarding of packets via a search for an inserted payload.
-            If the payload is found, the packet was transmitted successfully. Otherwise, the packet
-            is considered dropped.
+            * Verifies the successful forwarding of packets via a search for an inserted payload.
+              If the payload is found, the packet was transmitted successfully. Otherwise, the
+              packet is considered dropped.
 
-            Verify that standard MTU packets forward, in addition to packets within the limits of
-            an MTU size set during runtime.
+            * Verify that standard MTU packets forward, in addition to packets within the limits of
+              an MTU size set during runtime.
         """
         with TestPmdShell(tx_offloads=0x8000, mbuf_size=[JUMBO_MTU + 200]) as testpmd:
             testpmd.set_port_mtu_all(1500, verify=True)
@@ -185,16 +185,16 @@ def test_cli_mtu_forwarding_for_std_packets(self) -> None:
         """Assesses packet forwarding of standard MTU packets after pre-runtime MTU adjustments.
 
         Test:
-            Start TestPMD with MTU size of 1518 bytes, set pre-runtime.
-            Send packets of size 1491, 1500 and 1509 bytes.
-            Verify the first two packets are forwarded and the last is dropped.
+            * Start TestPMD with MTU size of 1518 bytes, set pre-runtime.
+            * Send packets of size 1491, 1500 and 1509 bytes.
+            * Verify the first two packets are forwarded and the last is dropped.
         Verify:
-            Verifies the successful forwarding of packets via a search for an inserted payload.
-            If the payload is found, the packet was transmitted successfully. Otherwise, the packet
-            is considered dropped.
+            * Verifies the successful forwarding of packets via a search for an inserted payload.
+              If the payload is found, the packet was transmitted successfully. Otherwise, the
+              packet is considered dropped.
 
-            Verify the first two packets are forwarded and the last is dropped after pre-runtime
-            MTU modification.
+            * Verify the first two packets are forwarded and the last is dropped after pre-runtime
+              MTU modification.
         """
         with TestPmdShell(
             tx_offloads=0x8000,
@@ -215,14 +215,14 @@ def test_cli_jumbo_forwarding_for_jumbo_mtu(self) -> None:
         """Assess packet forwarding of packets within the bounds of a pre-runtime MTU adjustment.
 
         Test:
-            Start TestPMD with MTU size of 9018 bytes, set pre-runtime.
-            Send packets of size 8991, 9000 and 1509 bytes.
+            * Start TestPMD with MTU size of 9018 bytes, set pre-runtime.
+            * Send packets of size 8991, 9000 and 1509 bytes.
         Verify:
-            Verifies the successful forwarding of packets via a search for an inserted payload.
-            If the payload is found, the packet was transmitted successfully. Otherwise, the packet
-            is considered dropped.
+            * Verifies the successful forwarding of packets via a search for an inserted payload.
+              If the payload is found, the packet was transmitted successfully. Otherwise, the
+              packet is considered dropped.
 
-            Verify that all packets are forwarded after pre-runtime MTU modification.
+            * Verify that all packets are forwarded after pre-runtime MTU modification.
         """
         with TestPmdShell(
             tx_offloads=0x8000,
@@ -241,16 +241,16 @@ def test_cli_mtu_std_packets_for_jumbo_mtu(self) -> None:
         """Assess boundary of jumbo MTU value set pre-runtime.
 
         Test:
-            Start TestPMD with MTU size of 9018 bytes, set pre-runtime.
-            Send a packets of size 8991, 9000 and 9009 bytes.
-            Verify the first two packets are forwarded and the last is dropped.
+            * Start TestPMD with MTU size of 9018 bytes, set pre-runtime.
+            * Send a packets of size 8991, 9000 and 9009 bytes.
+            * Verify the first two packets are forwarded and the last is dropped.
         Verify:
-            Verifies the successful forwarding of packets via a search for an inserted payload.
-            If the payload is found, the packet was transmitted successfully. Otherwise, the packet
-            is considered dropped.
+            * Verifies the successful forwarding of packets via a search for an inserted payload.
+              If the payload is found, the packet was transmitted successfully. Otherwise, the
+              packet is considered dropped.
 
-            Verify the first two packets are forwarded and the last is dropped after pre-runtime
-            MTU modification.
+            * Verify the first two packets are forwarded and the last is dropped after pre-runtime
+              MTU modification.
         """
         with TestPmdShell(
             tx_offloads=0x8000,
-- 
2.43.0


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH v4 3/7] dts: fix smoke tests docstring
  2025-03-03 14:57   ` [PATCH v4 0/7] dts: add per-test-suite configuration Luca Vizzarro
  2025-03-03 14:57     ` [PATCH v4 1/7] dts: add tests package to API docs Luca Vizzarro
  2025-03-03 14:57     ` [PATCH v4 2/7] dts: amend test suites docstring Luca Vizzarro
@ 2025-03-03 14:57     ` Luca Vizzarro
  2025-03-03 14:57     ` [PATCH v4 4/7] dts: update autodoc sorting order Luca Vizzarro
                       ` (3 subsequent siblings)
  6 siblings, 0 replies; 27+ messages in thread
From: Luca Vizzarro @ 2025-03-03 14:57 UTC (permalink / raw)
  To: dev; +Cc: Luca Vizzarro, Paul Szczepanek, Patrick Robb

From: Luca Vizzarro <luca.vizzarro@arm.com>

Sphinx autodoc complains of is_blocking being duplicated. This is the
case as the parent already holds a docstring for this attribute. Remove
the duplication.

Fixes: 6ef07151aac4 ("dts: update docstrings")

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 dts/tests/TestSuite_smoke_tests.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
index a8ea07595f..0474b82bd2 100644
--- a/dts/tests/TestSuite_smoke_tests.py
+++ b/dts/tests/TestSuite_smoke_tests.py
@@ -30,8 +30,6 @@ class TestSmokeTests(TestSuite):
     The infrastructure also needs to be tested, as that is also used by all other test suites.
 
     Attributes:
-        is_blocking: This test suite will block the execution of all other test suites
-            in the test run after it.
         nics_in_node: The NICs present on the SUT node.
     """
 
-- 
2.43.0


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH v4 4/7] dts: update autodoc sorting order
  2025-03-03 14:57   ` [PATCH v4 0/7] dts: add per-test-suite configuration Luca Vizzarro
                       ` (2 preceding siblings ...)
  2025-03-03 14:57     ` [PATCH v4 3/7] dts: fix smoke tests docstring Luca Vizzarro
@ 2025-03-03 14:57     ` Luca Vizzarro
  2025-03-03 14:57     ` [PATCH v4 5/7] dts: run only one test run per execution Luca Vizzarro
                       ` (2 subsequent siblings)
  6 siblings, 0 replies; 27+ messages in thread
From: Luca Vizzarro @ 2025-03-03 14:57 UTC (permalink / raw)
  To: dev; +Cc: Luca Vizzarro, Paul Szczepanek, Patrick Robb

From: Luca Vizzarro <luca.vizzarro@arm.com>

The autodoc member sorting order default is set to alphabetical, which
translates to autodoc sorting every member in modules, classes etc.
This also brings some side effects, like sorting capabilities which
can't be compared and result in errors.

This change prevents autodoc from sorting, and keeping the order as the
developer intended it.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 doc/guides/conf.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/guides/conf.py b/doc/guides/conf.py
index da139df1bd..565f5705d8 100644
--- a/doc/guides/conf.py
+++ b/doc/guides/conf.py
@@ -87,6 +87,7 @@
     autodoc_typehints = 'both'
     autodoc_typehints_format = 'short'
     autodoc_typehints_description_target = 'documented'
+    autodoc_member_order = 'bysource'
 
     # DTS docstring options.
     add_module_names = False
-- 
2.43.0


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH v4 5/7] dts: run only one test run per execution
  2025-03-03 14:57   ` [PATCH v4 0/7] dts: add per-test-suite configuration Luca Vizzarro
                       ` (3 preceding siblings ...)
  2025-03-03 14:57     ` [PATCH v4 4/7] dts: update autodoc sorting order Luca Vizzarro
@ 2025-03-03 14:57     ` Luca Vizzarro
  2025-03-03 14:57     ` [PATCH v4 6/7] dts: add per-test-suite configuration Luca Vizzarro
  2025-03-03 14:57     ` [PATCH v4 7/7] dts: improve configuration errors Luca Vizzarro
  6 siblings, 0 replies; 27+ messages in thread
From: Luca Vizzarro @ 2025-03-03 14:57 UTC (permalink / raw)
  To: dev; +Cc: Luca Vizzarro, Paul Szczepanek, Patrick Robb

From: Luca Vizzarro <luca.vizzarro@arm.com>

To aid configurability and flexibility both for the user and the
developer, make DTS run only one test run per execution.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 doc/guides/tools/dts.rst         | 18 ++++----
 dts/.gitignore                   |  2 +-
 dts/framework/config/__init__.py | 73 +++++++++++++++-----------------
 dts/framework/config/test_run.py | 15 +++----
 dts/framework/runner.py          | 14 +++---
 dts/framework/settings.py        | 16 +++----
 dts/test_run.example.yaml        | 43 +++++++++++++++++++
 dts/test_runs.example.yaml       | 43 -------------------
 dts/tests_config.example.yaml    |  0
 9 files changed, 107 insertions(+), 117 deletions(-)
 create mode 100644 dts/test_run.example.yaml
 delete mode 100644 dts/test_runs.example.yaml
 create mode 100644 dts/tests_config.example.yaml

diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index 097813e310..2affcf5d39 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -206,12 +206,12 @@ and then run the tests with the newly built binaries.
 Configuring DTS
 ~~~~~~~~~~~~~~~
 
-DTS configuration is split into nodes and test runs,
+DTS configuration is split into nodes and a test run,
 and must respect the model definitions
 as documented in the DTS API docs under the ``config`` page.
 The root of the configuration is represented by the ``Configuration`` model.
-By default, DTS will try to use the ``dts/test_runs.example.yaml``
-:ref:`config file <test_runs_configuration_example>`,
+By default, DTS will try to use the ``dts/test_run.example.yaml``
+:ref:`config file <test_run_configuration_example>`,
 and ``dts/nodes.example.yaml``
 :ref:`config file <nodes_configuration_example>`
 which are templates that illustrate what can be configured in DTS.
@@ -228,7 +228,7 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
 .. code-block:: console
 
    (dts-py3.10) $ ./main.py --help
-   usage: main.py [-h] [--test-runs-config-file FILE_PATH] [--nodes-config-file FILE_PATH] [--output-dir DIR_PATH] [-t SECONDS] [-v]
+   usage: main.py [-h] [--test-run-config-file FILE_PATH] [--nodes-config-file FILE_PATH] [--output-dir DIR_PATH] [-t SECONDS] [-v]
                   [--dpdk-tree DIR_PATH | --tarball FILE_PATH] [--remote-source] [--precompiled-build-dir DIR_NAME]
                   [--compile-timeout SECONDS] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run N_TIMES] [--random-seed NUMBER]
 
@@ -237,8 +237,8 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
 
    options:
      -h, --help            show this help message and exit
-     --test-runs-config-file FILE_PATH
-                           [DTS_TEST_RUNS_CFG_FILE] The configuration file that describes the test cases and DPDK build options. (default: test-runs.conf.yaml)
+     --test-run-config-file FILE_PATH
+                           [DTS_TEST_RUN_CFG_FILE] The configuration file that describes the test cases and DPDK build options. (default: test-run.conf.yaml)
      --nodes-config-file FILE_PATH
                            [DTS_NODES_CFG_FILE] The configuration file that describes the SUT and TG nodes. (default: nodes.conf.yaml)
      --output-dir DIR_PATH, --output DIR_PATH
@@ -486,12 +486,12 @@ And they both have two network ports which are physically connected to each othe
    This example assumes that you have setup SSH keys in both the system under test
    and traffic generator nodes.
 
-.. _test_runs_configuration_example:
+.. _test_run_configuration_example:
 
-``dts/test_runs.example.yaml``
+``dts/test_run.example.yaml``
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. literalinclude:: ../../../dts/test_runs.example.yaml
+.. literalinclude:: ../../../dts/test_run.example.yaml
    :language: yaml
    :start-at: # Define
 
diff --git a/dts/.gitignore b/dts/.gitignore
index d53a2f3b7e..96ee0b6ac4 100644
--- a/dts/.gitignore
+++ b/dts/.gitignore
@@ -1,4 +1,4 @@
 # default configuration files for DTS
 nodes.yaml
-test_runs.yaml
+test_run.yaml
 
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index c42eacb748..5495cfc5d5 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -13,7 +13,7 @@
 
 The configuration files are split in:
 
-    * A list of test run which are represented by :class:`~.test_run.TestRunConfiguration`
+    * The test run which is represented by :class:`~.test_run.TestRunConfiguration`
       defining what tests are going to be run and how DPDK will be built. It also references
       the testbed where these tests and DPDK are going to be run,
     * A list of the nodes of the testbed which ar represented by :class:`~.node.NodeConfiguration`.
@@ -40,16 +40,14 @@
 from .node import NodeConfiguration
 from .test_run import TestRunConfiguration
 
-TestRunsConfig = Annotated[list[TestRunConfiguration], Field(min_length=1)]
-
 NodesConfig = Annotated[list[NodeConfiguration], Field(min_length=1)]
 
 
 class Configuration(FrozenModel):
     """DTS testbed and test configuration."""
 
-    #: Test run configurations.
-    test_runs: TestRunsConfig
+    #: Test run configuration.
+    test_run: TestRunConfiguration
     #: Node configurations.
     nodes: NodesConfig
 
@@ -68,40 +66,36 @@ def validate_node_names(self) -> Self:
 
     @model_validator(mode="after")
     def validate_port_links(self) -> Self:
-        """Validate that all the test runs' port links are valid."""
+        """Validate that all of the test run's port links are valid."""
         existing_port_links: dict[tuple[str, str], Literal[False] | tuple[str, str]] = {
             (node.name, port.name): False for node in self.nodes for port in node.ports
         }
 
         defined_port_links = [
-            (test_run_idx, test_run, link_idx, link)
-            for test_run_idx, test_run in enumerate(self.test_runs)
-            for link_idx, link in enumerate(test_run.port_topology)
+            (link_idx, link) for link_idx, link in enumerate(self.test_run.port_topology)
         ]
-        for test_run_idx, test_run, link_idx, link in defined_port_links:
+        for link_idx, link in defined_port_links:
             sut_node_port_peer = existing_port_links.get(
-                (test_run.system_under_test_node, link.sut_port), None
-            )
-            assert sut_node_port_peer is not None, (
-                "Invalid SUT node port specified for link "
-                f"test_runs.{test_run_idx}.port_topology.{link_idx}."
+                (self.test_run.system_under_test_node, link.sut_port), None
             )
+            assert (
+                sut_node_port_peer is not None
+            ), f"Invalid SUT node port specified for link port_topology.{link_idx}."
 
             assert sut_node_port_peer is False or sut_node_port_peer == link.right, (
-                f"The SUT node port for link test_runs.{test_run_idx}.port_topology.{link_idx} is "
+                f"The SUT node port for link port_topology.{link_idx} is "
                 f"already linked to port {sut_node_port_peer[0]}.{sut_node_port_peer[1]}."
             )
 
             tg_node_port_peer = existing_port_links.get(
-                (test_run.traffic_generator_node, link.tg_port), None
-            )
-            assert tg_node_port_peer is not None, (
-                "Invalid TG node port specified for link "
-                f"test_runs.{test_run_idx}.port_topology.{link_idx}."
+                (self.test_run.traffic_generator_node, link.tg_port), None
             )
+            assert (
+                tg_node_port_peer is not None
+            ), f"Invalid TG node port specified for link port_topology.{link_idx}."
 
             assert tg_node_port_peer is False or sut_node_port_peer == link.left, (
-                f"The TG node port for link test_runs.{test_run_idx}.port_topology.{link_idx} is "
+                f"The TG node port for link port_topology.{link_idx} is "
                 f"already linked to port {tg_node_port_peer[0]}.{tg_node_port_peer[1]}."
             )
 
@@ -111,24 +105,21 @@ def validate_port_links(self) -> Self:
         return self
 
     @model_validator(mode="after")
-    def validate_test_runs_against_nodes(self) -> Self:
-        """Validate the test runs to nodes associations."""
-        for test_run_no, test_run in enumerate(self.test_runs):
-            sut_node_name = test_run.system_under_test_node
-            sut_node = next((n for n in self.nodes if n.name == sut_node_name), None)
-
-            assert sut_node is not None, (
-                f"Test run {test_run_no}.system_under_test_node "
-                f"({sut_node_name}) is not a valid node name."
-            )
+    def validate_test_run_against_nodes(self) -> Self:
+        """Validate the test run against the supplied nodes."""
+        sut_node_name = self.test_run.system_under_test_node
+        sut_node = next((n for n in self.nodes if n.name == sut_node_name), None)
 
-            tg_node_name = test_run.traffic_generator_node
-            tg_node = next((n for n in self.nodes if n.name == tg_node_name), None)
+        assert (
+            sut_node is not None
+        ), f"The system_under_test_node {sut_node_name} is not a valid node name."
 
-            assert tg_node is not None, (
-                f"Test run {test_run_no}.traffic_generator_name "
-                f"({tg_node_name}) is not a valid node name."
-            )
+        tg_node_name = self.test_run.traffic_generator_node
+        tg_node = next((n for n in self.nodes if n.name == tg_node_name), None)
+
+        assert (
+            tg_node is not None
+        ), f"The traffic_generator_name {tg_node_name} is not a valid node name."
 
         return self
 
@@ -160,10 +151,12 @@ def load_config(ctx: ValidationContext) -> Configuration:
     Raises:
         ConfigurationError: If the supplied configuration files are invalid.
     """
-    test_runs = _load_and_parse_model(ctx["settings"].test_runs_config_path, TestRunsConfig, ctx)
+    test_run = _load_and_parse_model(
+        ctx["settings"].test_run_config_path, TestRunConfiguration, ctx
+    )
     nodes = _load_and_parse_model(ctx["settings"].nodes_config_path, NodesConfig, ctx)
 
     try:
-        return Configuration.model_validate({"test_runs": test_runs, "nodes": nodes}, context=ctx)
+        return Configuration.model_validate({"test_run": test_run, "nodes": nodes}, context=ctx)
     except ValidationError as e:
         raise ConfigurationError("the configurations supplied are invalid") from e
diff --git a/dts/framework/config/test_run.py b/dts/framework/config/test_run.py
index 1b3045730d..c1e534e480 100644
--- a/dts/framework/config/test_run.py
+++ b/dts/framework/config/test_run.py
@@ -206,14 +206,13 @@ class TestSuiteConfig(FrozenModel):
 
     .. code:: yaml
 
-        test_runs:
-        - test_suites:
-            # As string representation:
-            - hello_world # test all of `hello_world`, or
-            - hello_world hello_world_single_core # test only `hello_world_single_core`
-            # or as model fields:
-            - test_suite: hello_world
-              test_cases: [hello_world_single_core] # without this field all test cases are run
+        test_suites:
+        # As string representation:
+        - hello_world # test all of `hello_world`, or
+        - hello_world hello_world_single_core # test only `hello_world_single_core`
+        # or as model fields:
+        - test_suite: hello_world
+            test_cases: [hello_world_single_core] # without this field all test cases are run
     """
 
     #: The name of the test suite module without the starting ``TestSuite_``.
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 801709a2aa..a0016d5d57 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -6,7 +6,7 @@
 
 """Test suite runner module.
 
-The module is responsible for preparing DTS and running the test runs.
+The module is responsible for preparing DTS and running the test run.
 """
 
 import os
@@ -47,8 +47,8 @@ def __init__(self):
     def run(self) -> None:
         """Run DTS.
 
-        Prepare all the nodes ahead of the test runs execution,
-        which are subsequently run as configured.
+        Prepare all the nodes ahead of the test run execution, which is subsequently run as
+        configured.
         """
         nodes: list[Node] = []
         try:
@@ -59,11 +59,9 @@ def run(self) -> None:
             for node_config in self._configuration.nodes:
                 nodes.append(Node(node_config))
 
-            # for all test run sections
-            for test_run_config in self._configuration.test_runs:
-                test_run_result = self._result.add_test_run(test_run_config)
-                test_run = TestRun(test_run_config, nodes, test_run_result)
-                test_run.spin()
+            test_run_result = self._result.add_test_run(self._configuration.test_run)
+            test_run = TestRun(self._configuration.test_run, nodes, test_run_result)
+            test_run.spin()
 
         except Exception as e:
             self._logger.exception("An unexpected error has occurred.")
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index cf82a7c18f..256afd5cfb 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -14,10 +14,10 @@
 
 The command line arguments along with the supported environment variables are:
 
-.. option:: --test-runs-config-file
-.. envvar:: DTS_TEST_RUNS_CFG_FILE
+.. option:: --test-run-config-file
+.. envvar:: DTS_TEST_RUN_CFG_FILE
 
-    The path to the YAML configuration file of the test runs.
+    The path to the YAML configuration file of the test run.
 
 .. option:: --nodes-config-file
 .. envvar:: DTS_NODES_CFG_FILE
@@ -125,7 +125,7 @@ class Settings:
     """
 
     #:
-    test_runs_config_path: Path = Path(__file__).parent.parent.joinpath("test_runs.yaml")
+    test_run_config_path: Path = Path(__file__).parent.parent.joinpath("test_run.yaml")
     #:
     nodes_config_path: Path = Path(__file__).parent.parent.joinpath("nodes.yaml")
     #:
@@ -323,14 +323,14 @@ def _get_parser() -> _DTSArgumentParser:
     )
 
     action = parser.add_argument(
-        "--test-runs-config-file",
-        default=SETTINGS.test_runs_config_path,
+        "--test-run-config-file",
+        default=SETTINGS.test_run_config_path,
         type=Path,
         help="The configuration file that describes the test cases and DPDK build options.",
         metavar="FILE_PATH",
-        dest="test_runs_config_path",
+        dest="test_run_config_path",
     )
-    _add_env_var_to_action(action, "TEST_RUNS_CFG_FILE")
+    _add_env_var_to_action(action, "TEST_RUN_CFG_FILE")
 
     action = parser.add_argument(
         "--nodes-config-file",
diff --git a/dts/test_run.example.yaml b/dts/test_run.example.yaml
new file mode 100644
index 0000000000..330a31bb18
--- /dev/null
+++ b/dts/test_run.example.yaml
@@ -0,0 +1,43 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2022-2023 The DPDK contributors
+# Copyright 2023 Arm Limited
+
+# Define the test run environment
+dpdk:
+  lcores: "" # use all available logical cores (Skips first core)
+  memory_channels: 4 # tells DPDK to use 4 memory channels
+  build:
+    dpdk_location:
+      # dpdk_tree: Commented out because `tarball` is defined.
+      tarball: dpdk-tarball.tar.xz
+      # Either `dpdk_tree` or `tarball` can be defined, but not both.
+      remote: false # Optional, defaults to false. If it's true, the `dpdk_tree` or `tarball`
+                    # is located on the SUT node, instead of the execution host.
+
+    # precompiled_build_dir: Commented out because `build_options` is defined.
+    build_options:
+      # the combination of the following two makes CC="ccache gcc"
+      compiler: gcc
+      compiler_wrapper: ccache # Optional.
+    # If `precompiled_build_dir` is defined, DPDK has been pre-built and the build directory is
+    # in a subdirectory of DPDK tree root directory. Otherwise, will be using the `build_options`
+    # to build the DPDK from source. Either `precompiled_build_dir` or `build_options` can be
+    # defined, but not both.
+traffic_generator:
+  type: SCAPY
+perf: false # disable performance testing
+func: true # enable functional testing
+skip_smoke_tests: false # optional
+# by removing the `test_suites` field, this test run will run every test suite available
+test_suites: # the following test suites will be run in their entirety
+  - hello_world
+vdevs: # optional; if removed, vdevs won't be used in the execution
+  - "crypto_openssl"
+# The machine running the DPDK test executable
+system_under_test_node: "SUT 1"
+# Traffic generator node to use for this execution environment
+traffic_generator_node: "TG 1"
+port_topology:
+  - sut.port-0 <-> tg.port-0  # explicit link. `sut` and `tg` are special identifiers that refer
+                              # to the respective test run's configured nodes.
+  - port-1 <-> port-1         # implicit link, left side is always SUT, right side is always TG.
diff --git a/dts/test_runs.example.yaml b/dts/test_runs.example.yaml
deleted file mode 100644
index 4f225d1b82..0000000000
--- a/dts/test_runs.example.yaml
+++ /dev/null
@@ -1,43 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright 2022-2023 The DPDK contributors
-# Copyright 2023 Arm Limited
-
-# Define one test run environment
-- dpdk:
-    lcores: "" # use all available logical cores (Skips first core)
-    memory_channels: 4 # tells DPDK to use 4 memory channels
-    build:
-      dpdk_location:
-        # dpdk_tree: Commented out because `tarball` is defined.
-        tarball: dpdk-tarball.tar.xz
-        # Either `dpdk_tree` or `tarball` can be defined, but not both.
-        remote: false # Optional, defaults to false. If it's true, the `dpdk_tree` or `tarball`
-                      # is located on the SUT node, instead of the execution host.
-
-      # precompiled_build_dir: Commented out because `build_options` is defined.
-      build_options:
-        # the combination of the following two makes CC="ccache gcc"
-        compiler: gcc
-        compiler_wrapper: ccache # Optional.
-      # If `precompiled_build_dir` is defined, DPDK has been pre-built and the build directory is
-      # in a subdirectory of DPDK tree root directory. Otherwise, will be using the `build_options`
-      # to build the DPDK from source. Either `precompiled_build_dir` or `build_options` can be
-      # defined, but not both.
-  traffic_generator:
-    type: SCAPY
-  perf: false # disable performance testing
-  func: true # enable functional testing
-  skip_smoke_tests: false # optional
-  # by removing the `test_suites` field, this test run will run every test suite available
-  test_suites: # the following test suites will be run in their entirety
-    - hello_world
-  vdevs: # optional; if removed, vdevs won't be used in the execution
-    - "crypto_openssl"
-  # The machine running the DPDK test executable
-  system_under_test_node: "SUT 1"
-  # Traffic generator node to use for this execution environment
-  traffic_generator_node: "TG 1"
-  port_topology:
-    - sut.port-0 <-> tg.port-0  # explicit link. `sut` and `tg` are special identifiers that refer
-                                # to the respective test run's configured nodes.
-    - port-1 <-> port-1         # implicit link, left side is always SUT, right side is always TG.
diff --git a/dts/tests_config.example.yaml b/dts/tests_config.example.yaml
new file mode 100644
index 0000000000..e69de29bb2
-- 
2.43.0


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH v4 6/7] dts: add per-test-suite configuration
  2025-03-03 14:57   ` [PATCH v4 0/7] dts: add per-test-suite configuration Luca Vizzarro
                       ` (4 preceding siblings ...)
  2025-03-03 14:57     ` [PATCH v4 5/7] dts: run only one test run per execution Luca Vizzarro
@ 2025-03-03 14:57     ` Luca Vizzarro
  2025-03-03 14:57     ` [PATCH v4 7/7] dts: improve configuration errors Luca Vizzarro
  6 siblings, 0 replies; 27+ messages in thread
From: Luca Vizzarro @ 2025-03-03 14:57 UTC (permalink / raw)
  To: dev; +Cc: Luca Vizzarro, Paul Szczepanek, Patrick Robb

From: Luca Vizzarro <luca.vizzarro@arm.com>

Allow test suites to be configured individually. Moreover enable them to
implement their own custom configuration.

Adds a new argument to the command line which enables the user to supply
a YAML file to configure all of the requested test suites. This argument
becomes mandatory only if any of the selected test suites' configuration
does not provide defaults for at least one field.

The configuration file is a simple mapping of test suite names to their
corresponding configurations. The validation object is created
dynamically, implementing only the selected test suites at runtime. This
is needed so that any test suites that require manual configuration,
won't complain if they are not selected.

Bugzilla ID: 1375

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 doc/guides/tools/dts.rst           |  9 ++++---
 dts/framework/config/__init__.py   | 39 +++++++++++++++++++++++++++---
 dts/framework/config/test_run.py   | 36 ++++++++++++++++++++++++---
 dts/framework/runner.py            |  7 +++++-
 dts/framework/settings.py          | 17 +++++++++++++
 dts/framework/test_run.py          | 23 ++++++++++++------
 dts/framework/test_suite.py        | 25 +++++++++++++------
 dts/tests/TestSuite_hello_world.py | 14 +++++++++--
 dts/tests_config.example.yaml      |  2 ++
 9 files changed, 144 insertions(+), 28 deletions(-)

diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index 2affcf5d39..fcc6d22036 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -228,9 +228,10 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
 .. code-block:: console
 
    (dts-py3.10) $ ./main.py --help
-   usage: main.py [-h] [--test-run-config-file FILE_PATH] [--nodes-config-file FILE_PATH] [--output-dir DIR_PATH] [-t SECONDS] [-v]
-                  [--dpdk-tree DIR_PATH | --tarball FILE_PATH] [--remote-source] [--precompiled-build-dir DIR_NAME]
-                  [--compile-timeout SECONDS] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run N_TIMES] [--random-seed NUMBER]
+   usage: main.py [-h] [--test-run-config-file FILE_PATH] [--nodes-config-file FILE_PATH] [--tests-config-file FILE_PATH]
+                  [--output-dir DIR_PATH] [-t SECONDS] [-v] [--dpdk-tree DIR_PATH | --tarball FILE_PATH] [--remote-source]
+                  [--precompiled-build-dir DIR_NAME] [--compile-timeout SECONDS] [--test-suite TEST_SUITE [TEST_CASES ...]]
+                  [--re-run N_TIMES] [--random-seed NUMBER]
 
    Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher
    priority.
@@ -241,6 +242,8 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
                            [DTS_TEST_RUN_CFG_FILE] The configuration file that describes the test cases and DPDK build options. (default: test-run.conf.yaml)
      --nodes-config-file FILE_PATH
                            [DTS_NODES_CFG_FILE] The configuration file that describes the SUT and TG nodes. (default: nodes.conf.yaml)
+     --tests-config-file FILE_PATH
+                           [DTS_TESTS_CFG_FILE] Configuration file used to override variable values inside specific test suites. (default: None)
      --output-dir DIR_PATH, --output DIR_PATH
                            [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
      -t SECONDS, --timeout SECONDS
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 5495cfc5d5..129e6f3222 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -17,6 +17,7 @@
       defining what tests are going to be run and how DPDK will be built. It also references
       the testbed where these tests and DPDK are going to be run,
     * A list of the nodes of the testbed which ar represented by :class:`~.node.NodeConfiguration`.
+    * A dictionary mapping test suite names to their corresponding configurations.
 
 The real-time information about testbed is supposed to be gathered at runtime.
 
@@ -27,8 +28,9 @@
       and makes it thread safe should we ever want to move in that direction.
 """
 
+import os
 from pathlib import Path
-from typing import Annotated, Any, Literal, TypeVar, cast
+from typing import TYPE_CHECKING, Annotated, Any, Literal, TypeVar, cast
 
 import yaml
 from pydantic import Field, TypeAdapter, ValidationError, model_validator
@@ -38,7 +40,11 @@
 
 from .common import FrozenModel, ValidationContext
 from .node import NodeConfiguration
-from .test_run import TestRunConfiguration
+from .test_run import TestRunConfiguration, create_test_suites_config_model
+
+# Import only if type checking or building docs, to prevent circular imports.
+if TYPE_CHECKING or os.environ.get("DTS_DOC_BUILD"):
+    from framework.test_suite import BaseConfig
 
 NodesConfig = Annotated[list[NodeConfiguration], Field(min_length=1)]
 
@@ -50,6 +56,8 @@ class Configuration(FrozenModel):
     test_run: TestRunConfiguration
     #: Node configurations.
     nodes: NodesConfig
+    #: Test suites custom configurations.
+    tests_config: dict[str, "BaseConfig"]
 
     @model_validator(mode="after")
     def validate_node_names(self) -> Self:
@@ -127,7 +135,7 @@ def validate_test_run_against_nodes(self) -> Self:
 T = TypeVar("T")
 
 
-def _load_and_parse_model(file_path: Path, model_type: T, ctx: ValidationContext) -> T:
+def _load_and_parse_model(file_path: Path, model_type: type[T], ctx: ValidationContext) -> T:
     with open(file_path) as f:
         try:
             data = yaml.safe_load(f)
@@ -154,9 +162,32 @@ def load_config(ctx: ValidationContext) -> Configuration:
     test_run = _load_and_parse_model(
         ctx["settings"].test_run_config_path, TestRunConfiguration, ctx
     )
+
+    TestSuitesConfiguration = create_test_suites_config_model(test_run.test_suites)
+    if ctx["settings"].tests_config_path:
+        tests_config = _load_and_parse_model(
+            ctx["settings"].tests_config_path,
+            TestSuitesConfiguration,
+            ctx,
+        )
+    else:
+        try:
+            tests_config = TestSuitesConfiguration()
+        except ValidationError as e:
+            raise ConfigurationError(
+                "A test suites' configuration file is required for the given test run. "
+                "The following selected test suites require manual configuration: "
+                + ", ".join(str(error["loc"][0]) for error in e.errors())
+            )
+
     nodes = _load_and_parse_model(ctx["settings"].nodes_config_path, NodesConfig, ctx)
 
     try:
-        return Configuration.model_validate({"test_run": test_run, "nodes": nodes}, context=ctx)
+        from framework.test_suite import BaseConfig as BaseConfig
+
+        Configuration.model_rebuild()
+        return Configuration.model_validate(
+            {"test_run": test_run, "nodes": nodes, "tests_config": dict(tests_config)}, context=ctx
+        )
     except ValidationError as e:
         raise ConfigurationError("the configurations supplied are invalid") from e
diff --git a/dts/framework/config/test_run.py b/dts/framework/config/test_run.py
index c1e534e480..688688e88e 100644
--- a/dts/framework/config/test_run.py
+++ b/dts/framework/config/test_run.py
@@ -18,7 +18,13 @@
 from pathlib import Path, PurePath
 from typing import Annotated, Any, Literal, NamedTuple
 
-from pydantic import Field, field_validator, model_validator
+from pydantic import (
+    BaseModel,
+    Field,
+    create_model,
+    field_validator,
+    model_validator,
+)
 from typing_extensions import TYPE_CHECKING, Self
 
 from framework.exception import InternalError
@@ -27,7 +33,7 @@
 from .common import FrozenModel, load_fields_from_settings
 
 if TYPE_CHECKING:
-    from framework.test_suite import TestCase, TestSuite, TestSuiteSpec
+    from framework.test_suite import BaseConfig, TestCase, TestSuite, TestSuiteSpec
 
 
 @unique
@@ -283,6 +289,27 @@ def fetch_all_test_suites() -> list[TestSuiteConfig]:
     ]
 
 
+def make_test_suite_config_field(config_obj: type["BaseConfig"]):
+    """Make a field for a test suite's configuration.
+
+    If the test suite's configuration has required fields, then make the field required. Otherwise
+    make it optional.
+    """
+    if any(f.is_required() for f in config_obj.model_fields.values()):
+        return config_obj, Field()
+    else:
+        return config_obj, Field(default_factory=config_obj)
+
+
+def create_test_suites_config_model(test_suites: Iterable[TestSuiteConfig]) -> type[BaseModel]:
+    """Create model for the test suites configuration."""
+    test_suites_kwargs = {
+        t.test_suite_name: make_test_suite_config_field(t.test_suite_spec.config_obj)
+        for t in test_suites
+    }
+    return create_model("TestSuitesConfiguration", **test_suites_kwargs)
+
+
 class LinkPortIdentifier(NamedTuple):
     """A tuple linking test run node type to port name."""
 
@@ -455,8 +482,8 @@ class TestRunConfiguration(FrozenModel):
     )
 
     def filter_tests(
-        self,
-    ) -> Iterable[tuple[type["TestSuite"], deque[type["TestCase"]]]]:
+        self, tests_config: dict[str, "BaseConfig"]
+    ) -> Iterable[tuple[type["TestSuite"], "BaseConfig", deque[type["TestCase"]]]]:
         """Filter test suites and cases selected for execution."""
         from framework.test_suite import TestCaseType
 
@@ -470,6 +497,7 @@ def filter_tests(
         return (
             (
                 t.test_suite_spec.class_obj,
+                tests_config[t.test_suite_name],
                 deque(
                     tt
                     for tt in t.test_cases
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index a0016d5d57..f822e8a8fc 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -60,7 +60,12 @@ def run(self) -> None:
                 nodes.append(Node(node_config))
 
             test_run_result = self._result.add_test_run(self._configuration.test_run)
-            test_run = TestRun(self._configuration.test_run, nodes, test_run_result)
+            test_run = TestRun(
+                self._configuration.test_run,
+                self._configuration.tests_config,
+                nodes,
+                test_run_result,
+            )
             test_run.spin()
 
         except Exception as e:
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 256afd5cfb..3f21615223 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -24,6 +24,11 @@
 
     The path to the YAML configuration file of the nodes.
 
+.. option:: --tests-config-file
+.. envvar:: DTS_TESTS_CFG_FILE
+
+    The path to the YAML configuration file of the test suites.
+
 .. option:: --output-dir, --output
 .. envvar:: DTS_OUTPUT_DIR
 
@@ -129,6 +134,8 @@ class Settings:
     #:
     nodes_config_path: Path = Path(__file__).parent.parent.joinpath("nodes.yaml")
     #:
+    tests_config_path: Path | None = None
+    #:
     output_dir: str = "output"
     #:
     timeout: float = 15
@@ -342,6 +349,16 @@ def _get_parser() -> _DTSArgumentParser:
     )
     _add_env_var_to_action(action, "NODES_CFG_FILE")
 
+    action = parser.add_argument(
+        "--tests-config-file",
+        default=SETTINGS.tests_config_path,
+        type=Path,
+        help="Configuration file used to override variable values inside specific test suites.",
+        metavar="FILE_PATH",
+        dest="tests_config_path",
+    )
+    _add_env_var_to_action(action, "TESTS_CFG_FILE")
+
     action = parser.add_argument(
         "--output-dir",
         "--output",
diff --git a/dts/framework/test_run.py b/dts/framework/test_run.py
index 2808d013f6..3f2eea693f 100644
--- a/dts/framework/test_run.py
+++ b/dts/framework/test_run.py
@@ -118,7 +118,7 @@
 from framework.remote_session.dpdk import DPDKBuildEnvironment, DPDKRuntimeEnvironment
 from framework.settings import SETTINGS
 from framework.test_result import BaseResult, Result, TestCaseResult, TestRunResult, TestSuiteResult
-from framework.test_suite import TestCase, TestSuite
+from framework.test_suite import BaseConfig, TestCase, TestSuite
 from framework.testbed_model.capability import (
     Capability,
     get_supported_capabilities,
@@ -128,7 +128,7 @@
 from framework.testbed_model.topology import PortLink, Topology
 from framework.testbed_model.traffic_generator import create_traffic_generator
 
-TestScenario = tuple[type[TestSuite], deque[type[TestCase]]]
+TestScenario = tuple[type[TestSuite], BaseConfig, deque[type[TestCase]]]
 
 
 class TestRun:
@@ -176,11 +176,18 @@ class TestRun:
     remaining_test_cases: deque[type[TestCase]]
     supported_capabilities: set[Capability]
 
-    def __init__(self, config: TestRunConfiguration, nodes: Iterable[Node], result: TestRunResult):
+    def __init__(
+        self,
+        config: TestRunConfiguration,
+        tests_config: dict[str, BaseConfig],
+        nodes: Iterable[Node],
+        result: TestRunResult,
+    ):
         """Test run constructor.
 
         Args:
             config: The test run's own configuration.
+            tests_config: The test run's test suites configurations.
             nodes: A reference to all the available nodes.
             result: A reference to the test run result object.
         """
@@ -201,7 +208,7 @@ def __init__(self, config: TestRunConfiguration, nodes: Iterable[Node], result:
 
         self.ctx = Context(sut_node, tg_node, topology, dpdk_runtime_env, traffic_generator)
         self.result = result
-        self.selected_tests = list(self.config.filter_tests())
+        self.selected_tests = list(self.config.filter_tests(tests_config))
         self.blocked = False
         self.remaining_tests = deque()
         self.remaining_test_cases = deque()
@@ -214,7 +221,7 @@ def required_capabilities(self) -> set[Capability]:
         """The capabilities required to run this test run in its totality."""
         caps = set()
 
-        for test_suite, test_cases in self.selected_tests:
+        for test_suite, _, test_cases in self.selected_tests:
             caps.update(test_suite.required_capabilities)
             for test_case in test_cases:
                 caps.update(test_case.required_capabilities)
@@ -371,8 +378,10 @@ def next(self) -> State | None:
         """Next state."""
         test_run = self.test_run
         try:
-            test_suite_class, test_run.remaining_test_cases = test_run.remaining_tests.popleft()
-            test_suite = test_suite_class()
+            test_suite_class, test_config, test_run.remaining_test_cases = (
+                test_run.remaining_tests.popleft()
+            )
+            test_suite = test_suite_class(test_config)
             test_suite_result = test_run.result.add_test_suite(test_suite.name)
 
             if test_run.blocked:
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 58da26adf0..e07c327b77 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -31,6 +31,7 @@
 from scapy.packet import Packet, Padding, raw
 from typing_extensions import Self
 
+from framework.config.common import FrozenModel
 from framework.testbed_model.capability import TestProtocol
 from framework.testbed_model.topology import Topology
 from framework.testbed_model.traffic_generator.capturing_traffic_generator import (
@@ -46,6 +47,10 @@
     from framework.context import Context
 
 
+class BaseConfig(FrozenModel):
+    """Base for a custom test suite configuration."""
+
+
 class TestSuite(TestProtocol):
     """The base class with building blocks needed by most test cases.
 
@@ -70,8 +75,13 @@ class TestSuite(TestProtocol):
 
     The test suite is aware of the testbed (the SUT and TG) it's running on. From this, it can
     properly choose the IP addresses and other configuration that must be tailored to the testbed.
+
+    Attributes:
+        config: The test suite configuration.
     """
 
+    config: BaseConfig
+
     #: Whether the test suite is blocking. A failure of a blocking test suite
     #: will block the execution of all subsequent test suites in the current test run.
     is_blocking: ClassVar[bool] = False
@@ -82,19 +92,15 @@ class TestSuite(TestProtocol):
     _tg_ip_address_ingress: Union[IPv4Interface, IPv6Interface]
     _tg_ip_address_egress: Union[IPv4Interface, IPv6Interface]
 
-    def __init__(self):
+    def __init__(self, config: BaseConfig):
         """Initialize the test suite testbed information and basic configuration.
 
-        Find links between ports and set up default IP addresses to be used when
-        configuring them.
-
         Args:
-            sut_node: The SUT node where the test suite will run.
-            tg_node: The TG node where the test suite will run.
-            topology: The topology where the test suite will run.
+            config: The test suite configuration.
         """
         from framework.context import get_ctx
 
+        self.config = config
         self._ctx = get_ctx()
         self._logger = get_dts_logger(self.__class__.__name__)
         self._sut_ip_address_ingress = ip_interface("192.168.100.2/24")
@@ -678,6 +684,11 @@ def is_test_suite(obj) -> bool:
             f"Expected class {self.class_name} not found in module {self.module_name}."
         )
 
+    @cached_property
+    def config_obj(self) -> type[BaseConfig]:
+        """A reference to the test suite's configuration class."""
+        return self.class_obj.__annotations__.get("config", BaseConfig)
+
     @classmethod
     def discover_all(
         cls, package_name: str | None = None, module_prefix: str | None = None
diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
index 141f2bc4c9..6c9ecc1177 100644
--- a/dts/tests/TestSuite_hello_world.py
+++ b/dts/tests/TestSuite_hello_world.py
@@ -1,5 +1,6 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2024 University of New Hampshire
+# Copyright(c) 2025 Arm Limited
 
 """DPDK Hello World test suite.
 
@@ -8,12 +9,21 @@
 """
 
 from framework.remote_session.testpmd_shell import TestPmdShell
-from framework.test_suite import TestSuite, func_test
+from framework.test_suite import BaseConfig, TestSuite, func_test
+
+
+class Config(BaseConfig):
+    """Example custom configuration."""
+
+    #: The hello world message to print.
+    msg: str = "Hello World!"
 
 
 class TestHelloWorld(TestSuite):
     """Hello World test suite. One test case, which starts and stops a testpmd session."""
 
+    config: Config
+
     @func_test
     def test_hello_world(self) -> None:
         """EAL confidence test.
@@ -25,4 +35,4 @@ def test_hello_world(self) -> None:
         """
         with TestPmdShell() as testpmd:
             testpmd.start()
-        self.log("Hello World!")
+        self.log(self.config.msg)
diff --git a/dts/tests_config.example.yaml b/dts/tests_config.example.yaml
index e69de29bb2..72af9da260 100644
--- a/dts/tests_config.example.yaml
+++ b/dts/tests_config.example.yaml
@@ -0,0 +1,2 @@
+hello_world:
+  msg: A custom hello world to you!
\ No newline at end of file
-- 
2.43.0


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH v4 7/7] dts: improve configuration errors
  2025-03-03 14:57   ` [PATCH v4 0/7] dts: add per-test-suite configuration Luca Vizzarro
                       ` (5 preceding siblings ...)
  2025-03-03 14:57     ` [PATCH v4 6/7] dts: add per-test-suite configuration Luca Vizzarro
@ 2025-03-03 14:57     ` Luca Vizzarro
  6 siblings, 0 replies; 27+ messages in thread
From: Luca Vizzarro @ 2025-03-03 14:57 UTC (permalink / raw)
  To: dev; +Cc: Luca Vizzarro, Paul Szczepanek, Patrick Robb

From: Luca Vizzarro <luca.vizzarro@arm.com>

Improve the way that configuration errors are displayed to the user.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 dts/framework/config/__init__.py |  4 ++--
 dts/framework/config/test_run.py |  2 +-
 dts/framework/runner.py          | 14 +++++++++++++-
 3 files changed, 16 insertions(+), 4 deletions(-)

diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 129e6f3222..1ec744d1d4 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -141,7 +141,7 @@ def _load_and_parse_model(file_path: Path, model_type: type[T], ctx: ValidationC
             data = yaml.safe_load(f)
             return TypeAdapter(model_type).validate_python(data, context=cast(dict[str, Any], ctx))
         except ValidationError as e:
-            msg = f"failed to load the configuration file {file_path}"
+            msg = f"Failed to load the configuration file {file_path}."
             raise ConfigurationError(msg) from e
 
 
@@ -190,4 +190,4 @@ def load_config(ctx: ValidationContext) -> Configuration:
             {"test_run": test_run, "nodes": nodes, "tests_config": dict(tests_config)}, context=ctx
         )
     except ValidationError as e:
-        raise ConfigurationError("the configurations supplied are invalid") from e
+        raise ConfigurationError("The configurations supplied are invalid.") from e
diff --git a/dts/framework/config/test_run.py b/dts/framework/config/test_run.py
index 688688e88e..06fe28143c 100644
--- a/dts/framework/config/test_run.py
+++ b/dts/framework/config/test_run.py
@@ -445,7 +445,7 @@ def use_first_core(self) -> bool:
 class DPDKConfiguration(DPDKRuntimeConfiguration):
     """The DPDK configuration needed to test."""
 
-    #: The DPDKD build configuration used to test.
+    #: The DPDK build configuration used to test.
     build: DPDKBuildConfiguration
 
 
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index f822e8a8fc..f20aa3576a 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -11,8 +11,10 @@
 
 import os
 import sys
+import textwrap
 
 from framework.config.common import ValidationContext
+from framework.exception import ConfigurationError
 from framework.test_run import TestRun
 from framework.testbed_model.node import Node
 
@@ -37,7 +39,17 @@ class DTSRunner:
 
     def __init__(self):
         """Initialize the instance with configuration, logger, result and string constants."""
-        self._configuration = load_config(ValidationContext(settings=SETTINGS))
+        try:
+            self._configuration = load_config(ValidationContext(settings=SETTINGS))
+        except ConfigurationError as e:
+            if e.__cause__:
+                print(f"{e} Reason:", file=sys.stderr)
+                print(file=sys.stderr)
+                print(textwrap.indent(str(e.__cause__), prefix=" " * 2), file=sys.stderr)
+            else:
+                print(e, file=sys.stderr)
+            sys.exit(e.severity)
+
         self._logger = get_dts_logger()
         if not os.path.exists(SETTINGS.output_dir):
             os.makedirs(SETTINGS.output_dir)
-- 
2.43.0


^ permalink raw reply	[flat|nested] 27+ messages in thread

end of thread, other threads:[~2025-03-03 14:58 UTC | newest]

Thread overview: 27+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-09-06 16:13 [PATCH] dts: add per-test-suite configuration Luca Vizzarro
2024-09-27 17:45 ` Jeremy Spewock
2024-11-08 13:38 ` [PATCH v2 0/4] " Luca Vizzarro
2024-11-08 13:38   ` [PATCH v2 1/4] dts: add tests package to API docs Luca Vizzarro
2024-11-08 13:38   ` [PATCH v2 2/4] dts: fix smoke tests docstring Luca Vizzarro
2024-11-12 20:03     ` Dean Marx
2024-11-08 13:38   ` [PATCH v2 3/4] dts: add per-test-suite configuration Luca Vizzarro
2024-11-08 13:38   ` [PATCH v2 4/4] dts: update autodoc sorting order Luca Vizzarro
2024-11-08 13:45 ` [PATCH v3 0/4] dts: add per-test-suite configuration Luca Vizzarro
2024-11-08 13:45   ` [PATCH v3 1/4] dts: add tests package to API docs Luca Vizzarro
2024-11-12 19:55     ` Dean Marx
2024-11-08 13:45   ` [PATCH v3 2/4] dts: fix smoke tests docstring Luca Vizzarro
2024-11-08 13:45   ` [PATCH v3 3/4] dts: add per-test-suite configuration Luca Vizzarro
2024-11-13 21:44     ` Dean Marx
2024-11-14 12:39       ` Luca Vizzarro
2024-11-19 16:31         ` Dean Marx
2024-11-19 16:41           ` Luca Vizzarro
2024-11-08 13:45   ` [PATCH v3 4/4] dts: update autodoc sorting order Luca Vizzarro
2024-11-12 20:04     ` Dean Marx
2025-03-03 14:57   ` [PATCH v4 0/7] dts: add per-test-suite configuration Luca Vizzarro
2025-03-03 14:57     ` [PATCH v4 1/7] dts: add tests package to API docs Luca Vizzarro
2025-03-03 14:57     ` [PATCH v4 2/7] dts: amend test suites docstring Luca Vizzarro
2025-03-03 14:57     ` [PATCH v4 3/7] dts: fix smoke tests docstring Luca Vizzarro
2025-03-03 14:57     ` [PATCH v4 4/7] dts: update autodoc sorting order Luca Vizzarro
2025-03-03 14:57     ` [PATCH v4 5/7] dts: run only one test run per execution Luca Vizzarro
2025-03-03 14:57     ` [PATCH v4 6/7] dts: add per-test-suite configuration Luca Vizzarro
2025-03-03 14:57     ` [PATCH v4 7/7] dts: improve configuration errors Luca Vizzarro

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).