Re: [PATCH v4 00/11] dts: add test skipping based on capabilities

["Juraj Linkeš" <juraj.linkes@pantheon.tech>]

On 24. 9. 2024 10:20, Juraj Linkeš wrote:
> Add an automated way to gather available capabilities of the tested
> hardware and skip test suites or cases which require capabilities that
> are not available.
> 
> This is done through two decorators:
> 1. The first marks a test suite method as test case. This populates the
>     default attributes of each test case.
> 2. The seconds adds the required capabilities to a test suite or case,
>     using the attributes from 1).
> 
> Two types of capabilities are added:
> 1. NIC capabilities. These are gathered once DPDK has been built because
>     we use testpmd for this. It's possible to add a function that will
>     add configuration before assessing capabilities associated with the
>     function. This is because some capabilities return different status
>     with different configuration present.
> 2. The topology capability. Each test case is marked as requiring a
>     default topology. The required topology of a test case (or a whole
>     test suite) may be change with the second decorator.
> 
> This is how it all works:
> 1. The required capabilities are first all gathered from all test suites
>     and test cases.
> 2. The list of required capabilities is divided into supported and
>     unsupported capabilities. In this step, the probing of hardware
>     and/or anything else that needs to happen to gauge whether a
>     capability is supported is done.
> 3. Each test suite and test case is then marked to be skipped if any of
>     their required capabilities are not supported.
> 
> v4:
> Rebased on next-dts.
> Adjusted the decorator for setting and reverting 9000 MTU to be
> parametrized.
> Added references to DPDK libraries and testpmd code to testpmd command
> parsing methods.
> Updated capability docstrings. Moved the capability decorator to
> NicCapability.
> 
> Juraj Linkeš (11):
>    dts: add the aenum dependency
>    dts: add test case decorators
>    dts: add mechanism to skip test cases or suites
>    dts: add support for simpler topologies
>    dts: add basic capability support
>    dts: add NIC capability support
>    dts: add NIC capabilities from show rxq info
>    dts: add topology capability
>    doc: add DTS capability doc sources
>    dts: add Rx offload capabilities
>    dts: add NIC capabilities from show port info
> 
>   .../framework.testbed_model.capability.rst    |   6 +
>   doc/api/dts/framework.testbed_model.rst       |   2 +
>   .../dts/framework.testbed_model.topology.rst  |   6 +
>   dts/framework/remote_session/testpmd_shell.py | 461 +++++++++++++++-
>   dts/framework/runner.py                       | 155 +++---
>   dts/framework/test_result.py                  | 119 ++--
>   dts/framework/test_suite.py                   | 161 +++++-
>   dts/framework/testbed_model/capability.py     | 507 ++++++++++++++++++
>   dts/framework/testbed_model/node.py           |   2 +-
>   dts/framework/testbed_model/port.py           |   4 +-
>   dts/framework/testbed_model/topology.py       | 127 +++++
>   dts/poetry.lock                               |  14 +-
>   dts/pyproject.toml                            |   1 +
>   dts/tests/TestSuite_hello_world.py            |  10 +-
>   dts/tests/TestSuite_os_udp.py                 |   3 +-
>   dts/tests/TestSuite_pmd_buffer_scatter.py     |  14 +-
>   dts/tests/TestSuite_smoke_tests.py            |   8 +-
>   17 files changed, 1447 insertions(+), 153 deletions(-)
>   create mode 100644 doc/api/dts/framework.testbed_model.capability.rst
>   create mode 100644 doc/api/dts/framework.testbed_model.topology.rst
>   create mode 100644 dts/framework/testbed_model/capability.py
>   create mode 100644 dts/framework/testbed_model/topology.py
> 
Applied to next-dts, thanks for the reviews.