Re: [PATCH v3 3/4] dts: add per-test-suite configuration

[Dean Marx <dmarx@iol.unh.edu>]

[-- Attachment #1: Type: text/plain, Size: 2735 bytes --]

>
>
> On Fri, Nov 8, 2024 at 8:38 AM Luca Vizzarro <luca.vizzarro@arm.com>
> wrote:
>
>> Allow test suites to be configured individually. Moreover enable them to
>> implement their own custom configuration.
>>
>> This solution adds some new complexity to DTS, which is generated source
>> code. In order to ensure strong typing, the test suites and their custom
>> configurations need to be linked in the main configuration class.
>> Unfortunately, this is not feasible during runtime as it will incur in
>> circular dependencies. Generating the links appear to be the most
>> straightforward approach.
>>
>> This commit also brings a new major change to the configuration schema.
>> Test suites are no longer defined as a list of strings, like:
>>
>>     test_suites:
>>     - hello_world
>>     - pmd_buffer_scatter
>>
>> but as mapping of mappings or strings:
>>
>>     test_suites:
>>       hello_world: {} # any custom fields or test cases can be set here
>>       pmd_buffer_scatter: all # "all" defines all the test cases, or
>>                               # they can individually be set separated
>>                               # by a space
>>
>> Not defining the `test_cases` field in the configuration is equivalent
>> to `all`, therefore the definitions for either test suite above are
>> also equivalent.
>>
>> Creating the __init__.py file under the tests folder, allows it to be
>> picked up as a package. This is a mypy requirement to import the tests
>> from within the framework.
>>
>> Bugzilla ID: 1375
>>
>> Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
>> Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
>>
>
> I like the idea of mapping the suite to specific test cases, and for the
> most part the custom configuration option as well. The only thing that I
> feel should be different is the way the code generation is documented, I
> think it might be worth providing an example within conf.yaml through a
> comment near the suites section, rather than just in the dts.rst file. It
> might be a little more clear where to create the custom config class as
> well.
>
>  <snip>
>
>> +class HelloWorldConfig(TestSuiteConfig):
>> +    """Example custom configuration for the `TestHelloWorld` test
>> suite."""
>> +
>> +    #: Timeout for the DPDK apps.
>> +    timeout: int = 50
>> --
>> 2.43.0
>>
>>
> Additionally, I was a bit confused by the custom config examples, do these
> fields (timeout, my_custom_field) actually affect the suite in any way as
> of this patch? Or is this just so that we can potentially add configuration
> options through this method in the future?
>
> Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
>

[-- Attachment #2: Type: text/html, Size: 3577 bytes --]