DPDK patches and discussions
 help / color / mirror / Atom feed
* [PATCH] dts: add per-test-suite configuration
@ 2024-09-06 16:13 Luca Vizzarro
  2024-09-27 17:45 ` Jeremy Spewock
                   ` (2 more replies)
  0 siblings, 3 replies; 17+ messages in thread
From: Luca Vizzarro @ 2024-09-06 16:13 UTC (permalink / raw)
  To: dev
  Cc: Honnappa Nagarahalli, Luca Vizzarro, Paul Szczepanek,
	Alex Chapman, Juraj Linkeš

Allow test suites to be configured individually. Moreover enable them to
implement their own custom configuration.

This solution adds some new complexity to DTS, which is generated source
code. In order to ensure strong typing, the test suites and their custom
configurations need to be linked in the main configuration class.
Unfortunately, this is not feasible during runtime as it will incur in
circular dependencies. Generating the links appear to be the most
straightforward approach.

This commit also brings a new major change to the configuration schema.
Test suites are no longer defined as a list of strings, like:

    test_suites:
    - hello_world
    - pmd_buffer_scatter

but as mapping of mappings or strings:

    test_suites:
      hello_world: {} # any custom fields or test cases can be set here
      pmd_buffer_scatter: all # "all" defines all the test cases, or
                              # they can individually be set separated
                              # by a space

Not defining the `test_cases` field in the configuration is equivalent
to `all`, therefore the definitions for either test suite above are
also equivalent.

Creating the __init__.py file under the tests folder, allows it to be
picked up as a package. This is a mypy requirement to import the tests
from within the framework.

Bugzilla ID: 1375

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Alex Chapman <alex.chapman@arm.com>
---
Depends-on: series-32823 ("dts: Pydantic configuration")

Hello,

sending in a solution for the per-test-suite configuration issue.
This one took some thinking but I have given most of the motivations in
the commit body already. The docs are somewhat lacking but hopefully they
should be automatically tackled by the API docs generation.


Best,
Luca
---
 doc/guides/tools/dts.rst                   |  39 ++++--
 dts/conf.yaml                              |   4 +-
 dts/framework/config/__init__.py           |  98 ++-------------
 dts/framework/config/conf_yaml_schema.json |  94 +++++++++++---
 dts/framework/config/generated.py          |  40 ++++++
 dts/framework/config/test_suite.py         | 140 +++++++++++++++++++++
 dts/framework/runner.py                    |  59 +++++++--
 dts/framework/settings.py                  |  29 +++--
 dts/framework/test_result.py               |  12 +-
 dts/framework/test_suite.py                |  32 ++++-
 dts/generate-schema.py                     |   4 +-
 dts/generate-test-mappings.py              | 132 +++++++++++++++++++
 dts/tests/TestSuite_hello_world.py         |  12 +-
 dts/tests/__init__.py                      |   7 ++
 14 files changed, 539 insertions(+), 163 deletions(-)
 create mode 100644 dts/framework/config/generated.py
 create mode 100644 dts/framework/config/test_suite.py
 create mode 100755 dts/generate-test-mappings.py
 create mode 100644 dts/tests/__init__.py

diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index 317bd0ff99..66681543cd 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -391,6 +391,26 @@ There are four types of methods that comprise a test suite:
    should be implemented in the ``SutNode`` class (and the underlying classes that ``SutNode`` uses)
    and used by the test suite via the ``sut_node`` field.
 
+The test suites can also implement their own custom configuration fields. This can be achieved by
+creating a new test suite config file which inherits from ``TestSuiteConfig`` defined in
+``dts/framework/config/test_suite.py``. So that this new custom configuration class is used, the
+test suite class must override the ``config`` attribute annotation with your new class, for example::
+
+.. code:: python
+   class CustomConfig(TestSuiteConfig):
+      my_custom_field: int = 10
+
+   class TestMyNewTestSuite(TestSuite):
+      config: CustomConfig
+
+Finally, the test suites and the custom configuration files need to linked in the global configuration.
+This can be easily achieved by running the ``dts/generate-test-mappings.py``, e.g.:
+
+.. code-block:: console
+
+   $ poetry shell
+   (dts-py3.10) $ ./generate-test-mappings.py
+
 
 .. _dts_dev_tools:
 
@@ -510,18 +530,13 @@ _`Network port`
    ``peer_pci``           *string* – the PCI address of the peer node port. **Example**: ``000a:01:00.1``
    ====================== =================================================================================
 
-_`Test suite`
-   *string* – name of the test suite to run. **Examples**: ``hello_world``, ``os_udp``
-
-_`Test target`
-   *mapping* – selects specific test cases to run from a test suite. Mapping is described as follows:
-
-   ========= ===============================================================================================
-   ``suite`` See `Test suite`_
-   ``cases`` (*optional*) *sequence* of *string* – list of the selected test cases in the test suite to run.
+_`Test suites`
+   *mapping* – selects the test suites to run. Each mapping key corresponds to the test suite name.
 
-             Unknown test cases will be silently ignored.
-   ========= ===============================================================================================
+   The value of the mapping can either "all" to select all the test cases in that test suite, the test
+   cases names divided by a space. Or it can be another mapping to set any custom fields for the test suite.
+   In the case of a mapping, all the test cases are selected by default. In order to manually select test
+   cases, the ``test_cases`` field can be set with a list of strings, each entry being a test case name.
 
 
 Properties
@@ -542,7 +557,7 @@ involved in the testing. These can be defined with the following mappings:
    +----------------------------+-------------------------------------------------------------------+
    | ``func``                   | *boolean* – Enable functional testing.                            |
    +----------------------------+-------------------------------------------------------------------+
-   | ``test_suites``            | *sequence* of **one of** `Test suite`_ **or** `Test target`_      |
+   | ``test_suites``            | See `Test suites`_                                                |
    +----------------------------+-------------------------------------------------------------------+
    | ``skip_smoke_tests``       | (*optional*) *boolean* – Allows you to skip smoke testing         |
    |                            | if ``true``.                                                      |
diff --git a/dts/conf.yaml b/dts/conf.yaml
index 7d95016e68..c44bef604c 100644
--- a/dts/conf.yaml
+++ b/dts/conf.yaml
@@ -15,8 +15,8 @@ test_runs:
     func: true # enable functional testing
     skip_smoke_tests: false # optional
     test_suites: # the following test suites will be run in their entirety
-      - hello_world
-      - os_udp
+      hello_world: all
+      os_udp: all
     # The machine running the DPDK test executable
     system_under_test_node:
       node_name: "SUT 1"
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 013c529829..d39e0823bd 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -35,10 +35,12 @@
       and makes it thread safe should we ever want to move in that direction.
 """
 
+# pylama:ignore=W0611
+
 from enum import Enum, auto, unique
 from functools import cached_property
 from pathlib import Path
-from typing import TYPE_CHECKING, Annotated, Any, Literal, NamedTuple, Protocol
+from typing import Annotated, Literal, NamedTuple, Protocol
 
 import yaml
 from pydantic import (
@@ -50,15 +52,14 @@
     field_validator,
     model_validator,
 )
-from pydantic.config import JsonDict
 from pydantic.dataclasses import dataclass
 from typing_extensions import Self
 
 from framework.exception import ConfigurationError
 from framework.utils import StrEnum
 
-if TYPE_CHECKING:
-    from framework.test_suite import TestSuiteSpec
+from .generated import CUSTOM_CONFIG_TYPES, TestSuitesConfigs
+from .test_suite import TestSuiteConfig
 
 
 @unique
@@ -289,7 +290,7 @@ class NodeInfo:
     kernel_version: str
 
 
-@dataclass(slots=True, frozen=True, kw_only=True, config=ConfigDict(extra="forbid"))
+@dataclass(frozen=True, kw_only=True, config=ConfigDict(extra="forbid"))
 class BuildTargetConfiguration:
     """DPDK build configuration.
 
@@ -329,89 +330,6 @@ class BuildTargetInfo:
     compiler_version: str
 
 
-def make_parsable_schema(schema: JsonDict):
-    """Updates a model's JSON schema to make a string representation a valid alternative.
-
-    This utility function is required to be used with models that can be represented and validated
-    as a string instead of an object mapping. Normally the generated JSON schema will just show
-    the object mapping. This function wraps the mapping under an anyOf property sequenced with a
-    string type.
-
-    This function is a valid `Callable` for the `json_schema_extra` attribute of
-    `~pydantic.config.ConfigDict`.
-    """
-    inner_schema = schema.copy()
-    del inner_schema["title"]
-
-    title = schema.get("title")
-    description = schema.get("description")
-
-    schema.clear()
-
-    schema["title"] = title
-    schema["description"] = description
-    schema["anyOf"] = [inner_schema, {"type": "string"}]
-
-
-@dataclass(
-    frozen=True,
-    config=ConfigDict(extra="forbid", json_schema_extra=make_parsable_schema),
-)
-class TestSuiteConfig:
-    """Test suite configuration.
-
-    Information about a single test suite to be executed. It can be represented and validated as a
-    string type in the form of: ``TEST_SUITE [TEST_CASE, ...]``, in the configuration file.
-
-    Attributes:
-        test_suite: The name of the test suite module without the starting ``TestSuite_``.
-        test_cases: The names of test cases from this test suite to execute.
-            If empty, all test cases will be executed.
-    """
-
-    test_suite_name: str = Field(
-        title="Test suite name",
-        description="The identifying name of the test suite.",
-        alias="test_suite",
-    )
-    test_cases_names: list[str] = Field(
-        default_factory=list,
-        title="Test cases by name",
-        description="The identifying name of the test cases of the test suite.",
-        alias="test_cases",
-    )
-
-    @cached_property
-    def test_suite_spec(self) -> "TestSuiteSpec":
-        """The specification of the requested test suite."""
-        from framework.test_suite import find_by_name
-
-        test_suite_spec = find_by_name(self.test_suite_name)
-        assert test_suite_spec is not None, f"{self.test_suite_name} is not a valid test suite name"
-        return test_suite_spec
-
-    @model_validator(mode="before")
-    @classmethod
-    def convert_from_string(cls, data: Any) -> Any:
-        """Convert the string representation into a valid mapping."""
-        if isinstance(data, str):
-            [test_suite, *test_cases] = data.split()
-            return dict(test_suite=test_suite, test_cases=test_cases)
-        return data
-
-    @model_validator(mode="after")
-    def validate_names(self) -> Self:
-        """Validate the supplied test suite and test cases names."""
-        available_test_cases = map(lambda t: t.name, self.test_suite_spec.test_cases)
-        for requested_test_case in self.test_cases_names:
-            assert requested_test_case in available_test_cases, (
-                f"{requested_test_case} is not a valid test case "
-                f"for test suite {self.test_suite_name}"
-            )
-
-        return self
-
-
 @dataclass(slots=True, frozen=True, kw_only=True, config=ConfigDict(extra="forbid"))
 class TestRunSUTNodeConfiguration:
     """The SUT node configuration of a test run.
@@ -446,7 +364,7 @@ class TestRunConfiguration:
     perf: bool = Field(description="Enable performance testing.")
     func: bool = Field(description="Enable functional testing.")
     skip_smoke_tests: bool = False
-    test_suites: list[TestSuiteConfig] = Field(min_length=1)
+    test_suites: TestSuitesConfigs
     system_under_test_node: TestRunSUTNodeConfiguration
     traffic_generator_node: str
 
@@ -581,7 +499,7 @@ def load_config(config_file_path: Path) -> Configuration:
         config_data = yaml.safe_load(f)
 
     try:
-        ConfigurationType.json_schema()
+        TestSuitesConfigs.fix_custom_config_annotations()
         return ConfigurationType.validate_python(config_data)
     except ValidationError as e:
         raise ConfigurationError("failed to load the supplied configuration") from e
diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index 1cf1bb098a..1434cdfad3 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -66,6 +66,33 @@
       "title": "Compiler",
       "type": "string"
     },
+    "HelloWorldConfig": {
+      "anyOf": [
+        {
+          "additionalProperties": false,
+          "properties": {
+            "test_cases": {
+              "items": {
+                "type": "string"
+              },
+              "title": "Test Cases",
+              "type": "array"
+            },
+            "timeout": {
+              "default": 50,
+              "title": "Timeout",
+              "type": "integer"
+            }
+          },
+          "type": "object"
+        },
+        {
+          "type": "string"
+        }
+      ],
+      "description": "Example custom configuration for the `TestHelloWorld` test suite.",
+      "title": "HelloWorldConfig"
+    },
     "HugepageConfiguration": {
       "additionalProperties": false,
       "description": "The hugepage configuration of :class:`~framework.testbed_model.node.Node`\\s.\n\nAttributes:\n    number_of: The number of hugepages to allocate.\n    force_first_numa: If :data:`True`, the hugepages will be configured on the first NUMA node.",
@@ -373,14 +400,6 @@
           "title": "Func",
           "type": "boolean"
         },
-        "test_suites": {
-          "items": {
-            "$ref": "#/$defs/TestSuiteConfig"
-          },
-          "minItems": 1,
-          "title": "Test Suites",
-          "type": "array"
-        },
         "build_targets": {
           "items": {
             "$ref": "#/$defs/BuildTargetConfiguration"
@@ -393,6 +412,9 @@
           "title": "Skip Smoke Tests",
           "type": "boolean"
         },
+        "test_suites": {
+          "$ref": "#/$defs/TestSuitesConfigs"
+        },
         "system_under_test_node": {
           "$ref": "#/$defs/TestRunSUTNodeConfiguration"
         },
@@ -404,8 +426,8 @@
       "required": [
         "perf",
         "func",
-        "test_suites",
         "build_targets",
+        "test_suites",
         "system_under_test_node",
         "traffic_generator_node"
       ],
@@ -439,31 +461,63 @@
         {
           "additionalProperties": false,
           "properties": {
-            "test_suite": {
-              "description": "The identifying name of the test suite.",
-              "title": "Test suite name",
-              "type": "string"
-            },
             "test_cases": {
-              "description": "The identifying name of the test cases of the test suite.",
               "items": {
                 "type": "string"
               },
-              "title": "Test cases by name",
+              "title": "Test Cases",
               "type": "array"
             }
           },
-          "required": [
-            "test_suite"
-          ],
           "type": "object"
         },
         {
           "type": "string"
         }
       ],
-      "description": "Test suite configuration.\n\nInformation about a single test suite to be executed. It can be represented and validated as a\nstring type in the form of: ``TEST_SUITE [TEST_CASE, ...]``, in the configuration file.\n\nAttributes:\n    test_suite: The name of the test suite module without the starting ``TestSuite_``.\n    test_cases: The names of test cases from this test suite to execute.\n        If empty, all test cases will be executed.",
+      "description": "Test suite configuration base model.\n\nBy default the configuration of a generic test suite does not contain any attributes. Any test\nsuite should inherit this class to create their own custom configuration. Finally override the\ntype of the :attr:`~TestSuite.config` to use the newly created one.\n\nAttributes:\n    test_cases_names: The names of test cases from this test suite to execute. If empty, all\n        test cases will be executed.",
       "title": "TestSuiteConfig"
+    },
+    "TestSuitesConfigs": {
+      "additionalProperties": false,
+      "description": "Configuration mapping class to select and configure the test suites.\n\nBefore using this class, the custom configuration type annotations need to be fixed.\nTo do so, you need to call the `fix_custom_config_annotations` method.",
+      "properties": {
+        "hello_world": {
+          "anyOf": [
+            {
+              "$ref": "#/$defs/HelloWorldConfig"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null
+        },
+        "os_udp": {
+          "anyOf": [
+            {
+              "$ref": "#/$defs/TestSuiteConfig"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null
+        },
+        "pmd_buffer_scatter": {
+          "anyOf": [
+            {
+              "$ref": "#/$defs/TestSuiteConfig"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null
+        }
+      },
+      "title": "TestSuitesConfigs",
+      "type": "object"
     }
   },
   "description": "DTS testbed and test configuration.\n\nAttributes:\n    test_runs: Test run configurations.\n    nodes: Node configurations.",
diff --git a/dts/framework/config/generated.py b/dts/framework/config/generated.py
new file mode 100644
index 0000000000..d42ce93a51
--- /dev/null
+++ b/dts/framework/config/generated.py
@@ -0,0 +1,40 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 The DPDK contributors
+# This file is automatically generated by generate-test-mappings.py.
+# Do NOT modify this file manually.
+
+"""Generated file containing the links between the test suites and the configuration."""
+
+from typing import TYPE_CHECKING, Optional
+
+from framework.config.test_suite import BaseTestSuitesConfigs, TestSuiteConfig
+
+if TYPE_CHECKING:
+    from tests.TestSuite_hello_world import HelloWorldConfig
+
+
+CUSTOM_CONFIG_TYPES: dict[str, type[TestSuiteConfig]] = {}
+
+
+class TestSuitesConfigs(BaseTestSuitesConfigs):
+    """Configuration mapping class to select and configure the test suites.
+
+    Before using this class, the custom configuration type annotations need to be fixed.
+    To do so, you need to call the `fix_custom_config_annotations` method.
+    """
+
+    hello_world: Optional["HelloWorldConfig"] = None
+    os_udp: Optional[TestSuiteConfig] = None
+    pmd_buffer_scatter: Optional[TestSuiteConfig] = None
+
+    @classmethod
+    def fix_custom_config_annotations(cls):
+        """Fixes the custom config types annotations.
+
+        Moreover it also fills `CUSTOM_CONFIG_TYPES` with all the custom config types.
+        """
+        from tests.TestSuite_hello_world import HelloWorldConfig
+
+        CUSTOM_CONFIG_TYPES["hello_world"] = HelloWorldConfig
+
+        cls.model_rebuild()
diff --git a/dts/framework/config/test_suite.py b/dts/framework/config/test_suite.py
new file mode 100644
index 0000000000..0c1ddf9d95
--- /dev/null
+++ b/dts/framework/config/test_suite.py
@@ -0,0 +1,140 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Test suites configuration module.
+
+Test suites can inherit :class:`TestSuiteConfig` to create their own custom configuration.
+By doing so, the test suite class must also override the annotation of the field
+`~framework.test_suite.TestSuite.config` to use their custom configuration type.
+"""
+
+from typing import TYPE_CHECKING, Any, Iterable
+
+from pydantic import BaseModel, Field, ValidationInfo, field_validator, model_validator
+from pydantic.config import JsonDict
+from typing_extensions import Self
+
+if TYPE_CHECKING:
+    from framework.test_suite import TestSuiteSpec
+
+
+def make_parsable_schema(schema: JsonDict):
+    """Updates a model's JSON schema to make a string representation a valid alternative.
+
+    This utility function is required to be used with models that can be represented and validated
+    as a string instead of an object mapping. Normally the generated JSON schema will just show
+    the object mapping. This function wraps the mapping under an anyOf property sequenced with a
+    string type.
+
+    This function is a valid `Callable` for the `json_schema_extra` attribute of
+    `~pydantic.config.ConfigDict`.
+    """
+    inner_schema = schema.copy()
+
+    fields_to_preserve = ["title", "description"]
+    extracted_fields = {k: v for k in fields_to_preserve if (v := inner_schema.get(k))}
+    for field in extracted_fields:
+        del inner_schema[field]
+
+    schema.clear()
+    schema.update(extracted_fields)
+    schema["anyOf"] = [inner_schema, {"type": "string"}]
+
+
+class TestSuiteConfig(BaseModel, extra="forbid", json_schema_extra=make_parsable_schema):
+    """Test suite configuration base model.
+
+    By default the configuration of a generic test suite does not contain any attributes. Any test
+    suite should inherit this class to create their own custom configuration. Finally override the
+    type of the :attr:`~TestSuite.config` to use the newly created one.
+
+    Attributes:
+        test_cases_names: The names of test cases from this test suite to execute. If empty, all
+            test cases will be executed.
+    """
+
+    _test_suite_spec: "TestSuiteSpec"
+
+    test_cases_names: list[str] = Field(default_factory=list, alias="test_cases")
+
+    @property
+    def test_suite_name(self) -> str:
+        """The name of the test suite module without the starting ``TestSuite_``."""
+        return self._test_suite_spec.name
+
+    @property
+    def test_suite_spec(self) -> "TestSuiteSpec":
+        """The specification of the requested test suite."""
+        return self._test_suite_spec
+
+    @model_validator(mode="before")
+    @classmethod
+    def convert_from_string(cls, data: Any) -> Any:
+        """Validator which allows to select a test suite by string instead of a mapping."""
+        if isinstance(data, str):
+            test_cases = [] if data == "all" else data.split()
+            return dict(test_cases=test_cases)
+        return data
+
+    @classmethod
+    def make(cls, test_suite_name: str, *test_cases_names: str, **kwargs) -> Self:
+        """Make a configuration for the requested test suite.
+
+        Args:
+            test_suite_name: The name of the test suite.
+            test_cases_names: The test cases to select, if empty all are selected.
+            **kwargs: Any other configuration field.
+
+        Raises:
+            AssertionError: If the requested test suite or test cases do not exist.
+            ValidationError: If the configuration fields were not filled correctly.
+        """
+        from framework.test_suite import find_by_name
+
+        test_suite_spec = find_by_name(test_suite_name)
+        assert test_suite_spec is not None, f"Could not find test suite '{test_suite_name}'."
+        test_suite_spec.validate_test_cases(test_cases_names)
+
+        config = cls.model_validate({"test_cases": test_cases_names, **kwargs})
+        config._test_suite_spec = test_suite_spec
+        return config
+
+
+class BaseTestSuitesConfigs(BaseModel, extra="forbid"):
+    """Base class for test suites configs."""
+
+    def __contains__(self, key) -> bool:
+        """Check if the provided test suite name has been selected and/or configured."""
+        return key in self.model_fields_set
+
+    def __getitem__(self, key) -> TestSuiteConfig:
+        """Get test suite configuration."""
+        return self.__getattribute__(key)
+
+    def get_configs(self) -> Iterable[TestSuiteConfig]:
+        """Get all the test suite configurations."""
+        return map(lambda t: self[t], self.model_fields_set)
+
+    @classmethod
+    def available_test_suites(cls) -> Iterable[str]:
+        """List all the available test suites."""
+        return cls.model_fields.keys()
+
+    @field_validator("*")
+    @classmethod
+    def validate_test_suite_config(
+        cls, config: type[TestSuiteConfig], info: ValidationInfo
+    ) -> type[TestSuiteConfig]:
+        """Validate the provided test cases and link the test suite spec to the configuration."""
+        from framework.test_suite import find_by_name
+
+        test_suite_name = info.field_name
+        assert test_suite_name is not None
+
+        test_suite_spec = find_by_name(test_suite_name)
+        assert test_suite_spec is not None
+
+        config._test_suite_spec = test_suite_spec
+
+        test_suite_spec.validate_test_cases(config.test_cases_names)
+        return config
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 00b63cc292..bc7aa1555c 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -24,10 +24,13 @@
 from types import FunctionType
 from typing import Iterable
 
+from pydantic import ValidationError
+
 from framework.testbed_model.sut_node import SutNode
 from framework.testbed_model.tg_node import TGNode
 
 from .config import (
+    CUSTOM_CONFIG_TYPES,
     BuildTargetConfiguration,
     Configuration,
     SutNodeConfiguration,
@@ -36,7 +39,12 @@
     TGNodeConfiguration,
     load_config,
 )
-from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError
+from .exception import (
+    BlockingTestSuiteError,
+    ConfigurationError,
+    SSHTimeoutError,
+    TestCaseVerifyError,
+)
 from .logger import DTSLogger, DtsStage, get_dts_logger
 from .settings import SETTINGS
 from .test_result import (
@@ -142,12 +150,7 @@ def run(self) -> None:
                 self._logger.set_stage(DtsStage.test_run_setup)
                 self._logger.info(f"Running test run with SUT '{sut_node_config.name}'.")
                 test_run_result = self._result.add_test_run(test_run_config)
-                # we don't want to modify the original config, so create a copy
-                test_run_test_suites = list(
-                    SETTINGS.test_suites if SETTINGS.test_suites else test_run_config.test_suites
-                )
-                if not test_run_config.skip_smoke_tests:
-                    test_run_test_suites[:0] = [TestSuiteConfig("smoke_tests")]
+                test_run_test_suites = self._prepare_test_suites(test_run_config)
                 try:
                     test_suites_with_cases = self._get_test_suites_with_cases(
                         test_run_test_suites, test_run_config.func, test_run_config.perf
@@ -204,6 +207,40 @@ def _check_dts_python_version(self) -> None:
             )
             self._logger.warning("Please use Python >= 3.10 instead.")
 
+    def _prepare_test_suites(self, test_run_config: TestRunConfiguration) -> list[TestSuiteConfig]:
+        if SETTINGS.test_suites:
+            test_suites_configs = []
+            for selected_test_suite, selected_test_cases in SETTINGS.test_suites:
+                if selected_test_suite in test_run_config.test_suites:
+                    config = test_run_config.test_suites[selected_test_suite].model_copy()
+                    config.test_cases_names = selected_test_cases
+                else:
+                    try:
+                        config = CUSTOM_CONFIG_TYPES[selected_test_suite].make(
+                            selected_test_suite, *selected_test_cases
+                        )
+                    except AssertionError as e:
+                        raise ConfigurationError(
+                            "Invalid test cases were selected "
+                            f"for test suite {selected_test_suite}."
+                        ) from e
+                    except ValidationError as e:
+                        raise ConfigurationError(
+                            f"Test suite {selected_test_suite} needs to be explicitly configured "
+                            "in order to be selected."
+                        ) from e
+                test_suites_configs.append(config)
+        else:
+            # we don't want to modify the original config, so create a copy
+            test_suites_configs = [
+                config.model_copy() for config in test_run_config.test_suites.get_configs()
+            ]
+
+        if not test_run_config.skip_smoke_tests:
+            test_suites_configs[:0] = [TestSuiteConfig.make("smoke_tests")]
+
+        return test_suites_configs
+
     def _get_test_suites_with_cases(
         self,
         test_suite_configs: list[TestSuiteConfig],
@@ -245,7 +282,9 @@ def _get_test_suites_with_cases(
 
             test_suites_with_cases.append(
                 TestSuiteWithCases(
-                    test_suite_class=test_suite_class, test_cases=selected_test_cases
+                    test_suite_class=test_suite_class,
+                    test_cases=selected_test_cases,
+                    config=test_suite_config,
                 )
             )
         return test_suites_with_cases
@@ -466,7 +505,9 @@ def _run_test_suite(
         self._logger.set_stage(
             DtsStage.test_suite_setup, Path(SETTINGS.output_dir, test_suite_name)
         )
-        test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node)
+        test_suite = test_suite_with_cases.test_suite_class(
+            sut_node, tg_node, test_suite_with_cases.config
+        )
         try:
             self._logger.info(f"Starting test suite setup: {test_suite_name}")
             test_suite.set_up_suite()
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 2e8dedef4f..063f282edf 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -85,9 +85,7 @@
 from pathlib import Path
 from typing import Callable
 
-from pydantic import ValidationError
-
-from .config import TestSuiteConfig
+from .config import TestSuitesConfigs
 from .exception import ConfigurationError
 from .utils import DPDKGitTarball, get_commit_id
 
@@ -114,7 +112,7 @@ class Settings:
     #:
     compile_timeout: float = 1200
     #:
-    test_suites: list[TestSuiteConfig] = field(default_factory=list)
+    test_suites: list[tuple[str, list[str]]] = field(default_factory=list)
     #:
     re_run: int = 0
 
@@ -382,7 +380,7 @@ def _get_parser() -> _DTSArgumentParser:
 
 def _process_test_suites(
     parser: _DTSArgumentParser, args: list[list[str]]
-) -> list[TestSuiteConfig]:
+) -> list[tuple[str, list[str]]]:
     """Process the given argument to a list of :class:`TestSuiteConfig` to execute.
 
     Args:
@@ -398,16 +396,17 @@ def _process_test_suites(
         # Environment variable in the form of "SUITE1 CASE1 CASE2, SUITE2 CASE1, SUITE3, ..."
         args = [suite_with_cases.split() for suite_with_cases in args[0][0].split(",")]
 
-    try:
-        return [TestSuiteConfig(test_suite, test_cases) for [test_suite, *test_cases] in args]
-    except ValidationError as e:
-        print(
-            "An error has occurred while validating the test suites supplied in the "
-            f"{'environment variable' if action else 'arguments'}:",
-            file=sys.stderr,
-        )
-        print(e, file=sys.stderr)
-        sys.exit(1)
+    available_test_suites = TestSuitesConfigs.available_test_suites()
+    for test_suite_name, *_ in args:
+        if test_suite_name not in available_test_suites:
+            print(
+                f"The test suite {test_suite_name} supplied in the "
+                f"{'environment variable' if action else 'arguments'} is invalid.",
+                file=sys.stderr,
+            )
+            sys.exit(1)
+
+    return [(test_suite, test_cases) for test_suite, *test_cases in args]
 
 
 def get_settings() -> Settings:
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 5694a2482b..6c10c1e40a 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -64,17 +64,7 @@ class is to hold a subset of test cases (which could be all test cases) because
 
     test_suite_class: type[TestSuite]
     test_cases: list[FunctionType]
-
-    def create_config(self) -> TestSuiteConfig:
-        """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
-
-        Returns:
-            The :class:`TestSuiteConfig` representation.
-        """
-        return TestSuiteConfig(
-            test_suite=self.test_suite_class.__name__,
-            test_cases=[test_case.__name__ for test_case in self.test_cases],
-        )
+    config: TestSuiteConfig
 
 
 class Result(Enum):
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 972968b036..78e1b4c49a 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -23,7 +23,7 @@
 from ipaddress import IPv4Interface, IPv6Interface, ip_interface
 from pkgutil import iter_modules
 from types import FunctionType, ModuleType
-from typing import ClassVar, NamedTuple, Union
+from typing import ClassVar, Iterable, NamedTuple, Union, get_type_hints
 
 from pydantic.alias_generators import to_pascal
 from scapy.layers.inet import IP  # type: ignore[import-untyped]
@@ -31,6 +31,7 @@
 from scapy.packet import Packet, Padding  # type: ignore[import-untyped]
 from typing_extensions import Self
 
+from framework.config import TestSuiteConfig
 from framework.testbed_model.port import Port, PortLink
 from framework.testbed_model.sut_node import SutNode
 from framework.testbed_model.tg_node import TGNode
@@ -38,7 +39,7 @@
     PacketFilteringConfig,
 )
 
-from .exception import TestCaseVerifyError
+from .exception import InternalError, TestCaseVerifyError
 from .logger import DTSLogger, get_dts_logger
 from .utils import get_packet_summaries
 
@@ -78,6 +79,7 @@ class TestSuite:
     #: Whether the test suite is blocking. A failure of a blocking test suite
     #: will block the execution of all subsequent test suites in the current build target.
     is_blocking: ClassVar[bool] = False
+    config: TestSuiteConfig
     _logger: DTSLogger
     _port_links: list[PortLink]
     _sut_port_ingress: Port
@@ -93,6 +95,7 @@ def __init__(
         self,
         sut_node: SutNode,
         tg_node: TGNode,
+        config: TestSuiteConfig,
     ):
         """Initialize the test suite testbed information and basic configuration.
 
@@ -102,9 +105,11 @@ def __init__(
         Args:
             sut_node: The SUT node where the test suite will run.
             tg_node: The TG node where the test suite will run.
+            config: The test suite configuration.
         """
         self.sut_node = sut_node
         self.tg_node = tg_node
+        self.config = config
         self._logger = get_dts_logger(self.__class__.__name__)
         self._port_links = []
         self._process_links()
@@ -467,6 +472,21 @@ def is_test_suite(obj) -> bool:
 
         raise Exception("class not found in eligible test module")
 
+    @cached_property
+    def config_type(self) -> type[TestSuiteConfig]:
+        """A reference to the test suite's configuration type."""
+        fields = get_type_hints(self.class_type)
+        config_type = fields.get("config")
+        if config_type is None:
+            raise InternalError(
+                "Test suite class {self.class_name} is missing the `config` attribute."
+            )
+        if not issubclass(config_type, TestSuiteConfig):
+            raise InternalError(
+                f"Test suite class {self.class_name} has an invalid configuration type assigned."
+            )
+        return config_type
+
     @cached_property
     def test_cases(self) -> list[TestCase]:
         """A list of all the available test cases."""
@@ -533,6 +553,14 @@ def discover_all(
 
         return test_suites
 
+    def validate_test_cases(self, test_cases_names: Iterable[str]) -> None:
+        """Validate if the supplied test cases exist in the test suite."""
+        available_test_cases = map(lambda t: t.name, self.test_cases)
+        for requested_test_case in test_cases_names:
+            assert (
+                requested_test_case in available_test_cases
+            ), f"{requested_test_case} is not a valid test case for test suite {self.name}."
+
 
 AVAILABLE_TEST_SUITES: list[TestSuiteSpec] = TestSuiteSpec.discover_all()
 """Constant to store all the available, discovered and imported test suites.
diff --git a/dts/generate-schema.py b/dts/generate-schema.py
index b41d28492f..d24a67f68e 100755
--- a/dts/generate-schema.py
+++ b/dts/generate-schema.py
@@ -9,7 +9,7 @@
 
 from pydantic.json_schema import GenerateJsonSchema
 
-from framework.config import ConfigurationType
+from framework.config import ConfigurationType, TestSuitesConfigs
 
 DTS_DIR = os.path.dirname(os.path.realpath(__file__))
 RELATIVE_PATH_TO_SCHEMA = "framework/config/conf_yaml_schema.json"
@@ -26,6 +26,8 @@ def generate(self, schema, mode="validation"):
 
 
 try:
+    TestSuitesConfigs.fix_custom_config_annotations()
+
     path = os.path.join(DTS_DIR, RELATIVE_PATH_TO_SCHEMA)
 
     with open(path, "w") as schema_file:
diff --git a/dts/generate-test-mappings.py b/dts/generate-test-mappings.py
new file mode 100755
index 0000000000..d076ad0afe
--- /dev/null
+++ b/dts/generate-test-mappings.py
@@ -0,0 +1,132 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Test Suites to Configuration mappings generation script."""
+
+import os
+from collections import defaultdict
+from textwrap import indent
+from typing import Iterable
+
+from framework.config.test_suite import BaseTestSuitesConfigs, TestSuiteConfig
+from framework.exception import InternalError
+from framework.test_suite import AVAILABLE_TEST_SUITES, TestSuiteSpec
+
+DTS_DIR = os.path.dirname(os.path.realpath(__file__))
+SCRIPT_FILE_NAME = os.path.basename(__file__)
+
+FRAMEWORK_IMPORTS = [BaseTestSuitesConfigs, TestSuiteConfig]
+
+RELATIVE_PATH_TO_GENERATED_FILE = "framework/config/generated.py"
+SMOKE_TESTS_SUITE_NAME = "smoke_tests"
+CUSTOM_CONFIG_TYPES_VAR_NAME = "CUSTOM_CONFIG_TYPES"
+CUSTOM_CONFIG_LINKING_FUNCTION_NAME = "fix_custom_config_annotations"
+CUSTOM_CONFIG_LINKING_FUNCTION_DOCSTRING = [
+    '"""Fixes the custom config types annotations.',
+    "",
+    f"Moreover it also fills `{CUSTOM_CONFIG_TYPES_VAR_NAME}` with all the custom config types.",
+    '"""',
+]
+TEST_SUITES_CONFIG_CLASS_NAME = "TestSuitesConfigs"
+TEST_SUITES_CONFIG_CLASS_DOCSTRING = [
+    '"""Configuration mapping class to select and configure the test suites.',
+    "",
+    "Before using this class, the custom configuration type annotations need to be fixed.",
+    f"To do so, you need to call the `{CUSTOM_CONFIG_LINKING_FUNCTION_NAME}` method.",
+    '"""',
+]
+
+
+GENERATED_FILE_HEADER = f"""# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 The DPDK contributors
+# This file is automatically generated by {SCRIPT_FILE_NAME}.
+# Do NOT modify this file manually.
+
+\"\"\"Generated file containing the links between the test suites and the configuration.\"\"\"
+"""
+
+
+def join(lines: Iterable[str]) -> str:
+    """Join list of strings into text lines."""
+    return "\n".join(lines)
+
+
+def join_and_indent(lines: Iterable[str], indentation_level=1, indentation_spaces=4) -> str:
+    """Join list of strings into indented text lines."""
+    return "\n".join([indent(line, " " * indentation_level * indentation_spaces) for line in lines])
+
+
+def format_attributes_types(test_suite_spec: TestSuiteSpec):
+    """Format the config type into the respective configuration class field attribute type."""
+    config_type = test_suite_spec.config_type.__name__
+    if config_type != TestSuiteConfig.__name__:
+        config_type = f'"{config_type}"'
+    return f"Optional[{config_type}]"
+
+
+try:
+    framework_imports: dict[str, list[str]] = defaultdict(list)
+    for _import in FRAMEWORK_IMPORTS:
+        framework_imports[_import.__module__].append(_import.__name__)
+    formatted_framework_imports = sorted(
+        [
+            f"from {module} import {', '.join(sorted(imports))}"
+            for module, imports in framework_imports.items()
+        ]
+    )
+
+    test_suites = [
+        test_suite_spec
+        for test_suite_spec in AVAILABLE_TEST_SUITES
+        if test_suite_spec.name != SMOKE_TESTS_SUITE_NAME
+    ]
+
+    custom_configs = [t for t in test_suites if t.config_type is not TestSuiteConfig]
+
+    custom_config_imports = [
+        f"from {t.module_type.__name__} import {t.config_type.__name__}" for t in custom_configs
+    ]
+
+    test_suites_attributes = [f"{t.name}: {format_attributes_types(t)} = None" for t in test_suites]
+
+    custom_config_assignments = [
+        f'{CUSTOM_CONFIG_TYPES_VAR_NAME}["{t.name}"] = {t.config_type.__name__}'
+        for t in custom_configs
+    ]
+
+    generated_file_contents = f"""{GENERATED_FILE_HEADER}
+from typing import TYPE_CHECKING, Optional
+
+{join(formatted_framework_imports)}
+
+if TYPE_CHECKING:
+{join_and_indent(custom_config_imports)}
+
+
+{CUSTOM_CONFIG_TYPES_VAR_NAME}: dict[str, type[{TestSuiteConfig.__name__}]] = {'{}'}
+
+
+class {TEST_SUITES_CONFIG_CLASS_NAME}({BaseTestSuitesConfigs.__name__}):
+{join_and_indent(TEST_SUITES_CONFIG_CLASS_DOCSTRING)}
+
+{join_and_indent(test_suites_attributes)}
+
+    @classmethod
+    def {CUSTOM_CONFIG_LINKING_FUNCTION_NAME}(cls):
+{join_and_indent(CUSTOM_CONFIG_LINKING_FUNCTION_DOCSTRING, indentation_level=2)}
+{join_and_indent(custom_config_imports, indentation_level=2)}
+
+{join_and_indent(custom_config_assignments, indentation_level=2)}
+
+        cls.model_rebuild()
+"""
+
+    path = os.path.join(DTS_DIR, RELATIVE_PATH_TO_GENERATED_FILE)
+
+    with open(path, "w") as generated_file:
+        generated_file.write(generated_file_contents)
+
+    print("Test suites to configuration mappings generated successfully!")
+except Exception as e:
+    raise InternalError("Failed to generate test suites to configuration mappings.") from e
diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
index d958f99030..1723c123bc 100644
--- a/dts/tests/TestSuite_hello_world.py
+++ b/dts/tests/TestSuite_hello_world.py
@@ -7,6 +7,7 @@
 No other EAL parameters apart from cores are used.
 """
 
+from framework.config import TestSuiteConfig
 from framework.remote_session.dpdk_shell import compute_eal_params
 from framework.test_suite import TestSuite
 from framework.testbed_model.cpu import (
@@ -16,9 +17,18 @@
 )
 
 
+class HelloWorldConfig(TestSuiteConfig):
+    """Example custom configuration for the `TestHelloWorld` test suite."""
+
+    #: Timeout for the DPDK apps
+    timeout: int = 50
+
+
 class TestHelloWorld(TestSuite):
     """DPDK hello world app test suite."""
 
+    config: HelloWorldConfig
+
     def set_up_suite(self) -> None:
         """Set up the test suite.
 
@@ -59,7 +69,7 @@ def test_hello_world_all_cores(self) -> None:
         eal_para = compute_eal_params(
             self.sut_node, lcore_filter_specifier=LogicalCoreList(self.sut_node.lcores)
         )
-        result = self.sut_node.run_dpdk_app(self.app_helloworld_path, eal_para, 50)
+        result = self.sut_node.run_dpdk_app(self.app_helloworld_path, eal_para, self.config.timeout)
         for lcore in self.sut_node.lcores:
             self.verify(
                 f"hello from core {int(lcore)}" in result.stdout,
diff --git a/dts/tests/__init__.py b/dts/tests/__init__.py
new file mode 100644
index 0000000000..a300eb26fc
--- /dev/null
+++ b/dts/tests/__init__.py
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""Test suites.
+
+This package contains all the available test suites in DTS.
+"""
-- 
2.34.1


^ permalink raw reply	[flat|nested] 17+ messages in thread
end of thread, other threads:[~2024-11-14 12:39 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-09-06 16:13 [PATCH] dts: add per-test-suite configuration Luca Vizzarro
2024-09-27 17:45 ` Jeremy Spewock
2024-11-08 13:38 ` [PATCH v2 0/4] " Luca Vizzarro
2024-11-08 13:38   ` [PATCH v2 1/4] dts: add tests package to API docs Luca Vizzarro
2024-11-08 13:38   ` [PATCH v2 2/4] dts: fix smoke tests docstring Luca Vizzarro
2024-11-12 20:03     ` Dean Marx
2024-11-08 13:38   ` [PATCH v2 3/4] dts: add per-test-suite configuration Luca Vizzarro
2024-11-08 13:38   ` [PATCH v2 4/4] dts: update autodoc sorting order Luca Vizzarro
2024-11-08 13:45 ` [PATCH v3 0/4] dts: add per-test-suite configuration Luca Vizzarro
2024-11-08 13:45   ` [PATCH v3 1/4] dts: add tests package to API docs Luca Vizzarro
2024-11-12 19:55     ` Dean Marx
2024-11-08 13:45   ` [PATCH v3 2/4] dts: fix smoke tests docstring Luca Vizzarro
2024-11-08 13:45   ` [PATCH v3 3/4] dts: add per-test-suite configuration Luca Vizzarro
2024-11-13 21:44     ` Dean Marx
2024-11-14 12:39       ` Luca Vizzarro
2024-11-08 13:45   ` [PATCH v3 4/4] dts: update autodoc sorting order Luca Vizzarro
2024-11-12 20:04     ` Dean Marx

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).