On Fri, Nov 8, 2024 at 8:38 AM Luca Vizzarro <luca.vizzarro@arm.com> wrote:

Allow test suites to be configured individually. Moreover enable them to
implement their own custom configuration.

This solution adds some new complexity to DTS, which is generated source
code. In order to ensure strong typing, the test suites and their custom
configurations need to be linked in the main configuration class.
Unfortunately, this is not feasible during runtime as it will incur in
circular dependencies. Generating the links appear to be the most
straightforward approach.

This commit also brings a new major change to the configuration schema.
Test suites are no longer defined as a list of strings, like:

    test_suites:
    - hello_world
    - pmd_buffer_scatter

but as mapping of mappings or strings:

    test_suites:
      hello_world: {} # any custom fields or test cases can be set here
      pmd_buffer_scatter: all # "all" defines all the test cases, or
                              # they can individually be set separated
                              # by a space

Not defining the `test_cases` field in the configuration is equivalent
to `all`, therefore the definitions for either test suite above are
also equivalent.

Creating the __init__.py file under the tests folder, allows it to be
picked up as a package. This is a mypy requirement to import the tests
from within the framework.

Bugzilla ID: 1375

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>

I like the idea of mapping the suite to specific test cases, and for the most part the custom configuration option as well. The only thing that I feel should be different is the way the code generation is documented, I think it might be worth providing an example within conf.yaml through a comment near the suites section, rather than just in the dts.rst file. It might be a little more clear where to create the custom config class as well. 

 <snip>

+class HelloWorldConfig(TestSuiteConfig):
+    """Example custom configuration for the `TestHelloWorld` test suite."""
+
+    #: Timeout for the DPDK apps.
+    timeout: int = 50
--
2.43.0

Additionally, I was a bit confused by the custom config examples, do these fields (timeout, my_custom_field) actually affect the suite in any way as of this patch? Or is this just so that we can potentially add configuration options through this method in the future?

Reviewed-by: Dean Marx <dmarx@iol.unh.edu>