[PATCH 0/2] dts: smoke test bugfixes

[PATCH 0/2] dts: smoke test bugfixes

[Patrick Robb]

There are two smoke test bugfixes which have come in from our 25.03 DTS
development and which should be resolved with this series.

1. The patchseries which we merged for 25.03 which added the testsuite
specific configs inadvertently broke enabling smoke tests, as we build
the tests config model using the test suite list coming in from the
parsed test_run.yaml, before smoke_tests have been added. Then, later in
the execution this produces a key error. I think that adding a
placeholder smoke tests test suite config is the most reasonable
solution to this. Users are unlikely to specify any test config for
smoke tests as they don't require any, although if they happen to do so,
I understand this change to be non-breaking (and I gave it a test run).

2. I am also seeing the smoke test which relies on regex on
dpdk-devbind.py status output is failing on at least one of our systems
currently. The fact that this broke is rather peculiar since regex
syntax is obviously stable, and my reading of the current regex is that
it "should" work. In any case, I simply changed the regex to a pattern
which is more permissive and still captures the driver in use, which is
what is needed for the testsuite.

Patrick Robb (2):
  dts: include smoke tests in tests config
  dts: update dpdk-devbind script regex

 dts/framework/config/test_run.py   | 3 +++
 dts/tests/TestSuite_smoke_tests.py | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)

-- 
2.48.1

[PATCH 1/2] dts: include smoke tests in tests config

[Patrick Robb]

The tests config is created by parsing the testrun config
earlier in the DTS execution than when the smoke_tests
testsuite is added. This situation causes a key error
on the tests_config dictionary when smoke_tests is enabled.
This commit creates a placeholder TestSuiteConfig for
smoke tests in the function which is responsible for
creating the tests config model, resolving the issue.

Fixes: 184d21f08db4 ("dts: add per-test-suite configuration")

Signed-off-by: Patrick Robb <probb@iol.unh.edu>
---
 dts/framework/config/test_run.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/dts/framework/config/test_run.py b/dts/framework/config/test_run.py
index 06fe28143c..9bbed10f33 100644
--- a/dts/framework/config/test_run.py
+++ b/dts/framework/config/test_run.py
@@ -303,6 +303,9 @@ def make_test_suite_config_field(config_obj: type["BaseConfig"]):
 
 def create_test_suites_config_model(test_suites: Iterable[TestSuiteConfig]) -> type[BaseModel]:
     """Create model for the test suites configuration."""
+
+    test_suites += [TestSuiteConfig(test_suite="smoke_tests")]
+
     test_suites_kwargs = {
         t.test_suite_name: make_test_suite_config_field(t.test_suite_spec.config_obj)
         for t in test_suites
-- 
2.48.1

[PATCH 2/2] dts: update dpdk-devbind script regex

[Patrick Robb]

The current regex used for matching against the
dpdk-devbind status output is no longer matching
correctly. This commit introduces a more permissive
pattern which will yield the device driver in use.

Signed-off-by: Patrick Robb <probb@iol.unh.edu>
---
 dts/tests/TestSuite_smoke_tests.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
index 352b92ce6c..9e4d06a1c4 100644
--- a/dts/tests/TestSuite_smoke_tests.py
+++ b/dts/tests/TestSuite_smoke_tests.py
@@ -135,7 +135,7 @@ def test_device_bound_to_driver(self) -> None:
             # with the address for the nic we are on in the loop and then captures the
             # name of the driver in a group
             devbind_info_for_nic = re.search(
-                rf"{nic.pci}[^\\n]*drv=([\\d\\w-]*) [^\\n]*",
+                rf"{nic.pci}.*drv=(\S+) [^\\n]*",
                 all_nics_in_dpdk_devbind,
             )
             self.verify(
-- 
2.48.1

Re: [PATCH 1/2] dts: include smoke tests in tests config

[Patrick Robb]

[-- Attachment #1: Type: text/plain, Size: 519 bytes --]

Thanks Luca for the info - there is also 1 other fix needed. I'll send a v2.

On Thu, Apr 3, 2025 at 9:45 AM Luca Vizzarro <Luca.Vizzarro@arm.com> wrote:

> Hi Patrick,
>
> this looks good, easy and quick fix. I am afraid that you can't append
> to an `Iterable` type though. I've also confirmed this with mypy:
>
>    Unsupported left operand type for + ("Iterable[TestSuiteConfig]")
>
> I guess the easiest fix is to just change the argument type to `list`
> from `Iterable`.
>
> Best,
> Luca
>
>

[-- Attachment #2: Type: text/html, Size: 872 bytes --]

Re: [PATCH 1/2] dts: include smoke tests in tests config

[Luca Vizzarro]

Hi Patrick,

this looks good, easy and quick fix. I am afraid that you can't append 
to an `Iterable` type though. I've also confirmed this with mypy:

   Unsupported left operand type for + ("Iterable[TestSuiteConfig]")

I guess the easiest fix is to just change the argument type to `list` 
from `Iterable`.

Best,
Luca

Re: [PATCH 2/2] dts: update dpdk-devbind script regex

[Luca Vizzarro]

Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>

[PATCH 0/4] dts: smoke test bugfixes

[Patrick Robb]

There are two smoke test bugfixes which have come in from our 25.03 DTS
development and which should be resolved with this series.

1. The patchseries which we merged for 25.03 which added the testsuite
specific configs inadvertently broke enabling smoke tests, as we build
the tests config model using the test suite list coming in from the
parsed test_run.yaml, before smoke_tests have been added. Then, later in
the execution this produces a key error. I think that adding a
placeholder smoke tests test suite config is the most reasonable
solution to this. Users are unlikely to specify any test config for
smoke tests as they don't require any, although if they happen to do so,
I understand this change to be non-breaking (and I gave it a test run).

2. I am also seeing the smoke test which relies on regex on
dpdk-devbind.py status output is failing on at least one of our systems
currently. The fact that this broke is rather peculiar since regex
syntax is obviously stable, and my reading of the current regex is that
it "should" work. In any case, I simply changed the regex to a pattern
which is more permissive and still captures the driver in use, which is
what is needed for the testsuite.

changes in v2:

The v2 patchseries adds an extension to the smoke test meson fast tests
timeout in order to accommodate for slower test times on older systems.

The v2 also disables smoke tests in the example testrun config file,
based on discussion at the previous DTS meeting. In some cases
there are esoteric reasons why some meson tests may fail, and they will
also extend the test run execution time. For these reasons we think that
smoke tests should be opt in by default instead of opt out by default.

Patrick Robb (4):
  dts: include smoke tests in tests config
  dts: update dpdk-devbind script regex
  dts: default smoke tests to off
  dts: extend meson fast tests timeout

 dts/framework/config/test_run.py   | 7 +++++--
 dts/test_run.example.yaml          | 2 +-
 dts/tests/TestSuite_smoke_tests.py | 4 ++--
 3 files changed, 8 insertions(+), 5 deletions(-)

-- 
2.48.1

[PATCH 1/4] dts: include smoke tests in tests config

[Patrick Robb]

The tests config is created by parsing the testrun config
earlier in the DTS execution than when the smoke_tests
testsuite is added. This situation causes a key error
on the tests_config dictionary when smoke_tests is enabled.
This commit creates a placeholder TestSuiteConfig for
smoke tests in the function which is responsible for
creating the tests config model, resolving the issue.

Fixes: 184d21f08db4 ("dts: add per-test-suite configuration")

Signed-off-by: Patrick Robb <probb@iol.unh.edu>
Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>
---
 dts/framework/config/test_run.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/dts/framework/config/test_run.py b/dts/framework/config/test_run.py
index 06fe28143c..b6e4099eeb 100644
--- a/dts/framework/config/test_run.py
+++ b/dts/framework/config/test_run.py
@@ -301,11 +301,14 @@ def make_test_suite_config_field(config_obj: type["BaseConfig"]):
         return config_obj, Field(default_factory=config_obj)
 
 
-def create_test_suites_config_model(test_suites: Iterable[TestSuiteConfig]) -> type[BaseModel]:
+def create_test_suites_config_model(test_suites: list[TestSuiteConfig]) -> type[BaseModel]:
     """Create model for the test suites configuration."""
+    complete_test_suites = [TestSuiteConfig(test_suite="smoke_tests")]
+    complete_test_suites += test_suites
+
     test_suites_kwargs = {
         t.test_suite_name: make_test_suite_config_field(t.test_suite_spec.config_obj)
-        for t in test_suites
+        for t in complete_test_suites
     }
     return create_model("TestSuitesConfiguration", **test_suites_kwargs)
 
-- 
2.48.1

[PATCH 2/4] dts: update dpdk-devbind script regex

[Patrick Robb]

The current regex used for matching against the
dpdk-devbind status output is no longer matching
correctly. This commit introduces a more permissive
pattern which will yield the device driver in use.

Signed-off-by: Patrick Robb <probb@iol.unh.edu>
Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>
---
 dts/tests/TestSuite_smoke_tests.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
index 352b92ce6c..9e4d06a1c4 100644
--- a/dts/tests/TestSuite_smoke_tests.py
+++ b/dts/tests/TestSuite_smoke_tests.py
@@ -135,7 +135,7 @@ def test_device_bound_to_driver(self) -> None:
             # with the address for the nic we are on in the loop and then captures the
             # name of the driver in a group
             devbind_info_for_nic = re.search(
-                rf"{nic.pci}[^\\n]*drv=([\\d\\w-]*) [^\\n]*",
+                rf"{nic.pci}.*drv=(\S+) [^\\n]*",
                 all_nics_in_dpdk_devbind,
             )
             self.verify(
-- 
2.48.1

[PATCH 3/4] dts: default smoke tests to off

[Patrick Robb]

Set smoke tests to false in the example config so that
users will have to consciously decide whether the
smoke tests will be run or not.

Signed-off-by: Patrick Robb <probb@iol.unh.edu>
---
 dts/test_run.example.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/dts/test_run.example.yaml b/dts/test_run.example.yaml
index 330a31bb18..1bc436eed1 100644
--- a/dts/test_run.example.yaml
+++ b/dts/test_run.example.yaml
@@ -27,7 +27,7 @@ traffic_generator:
   type: SCAPY
 perf: false # disable performance testing
 func: true # enable functional testing
-skip_smoke_tests: false # optional
+skip_smoke_tests: true # optional
 # by removing the `test_suites` field, this test run will run every test suite available
 test_suites: # the following test suites will be run in their entirety
   - hello_world
-- 
2.48.1

[PATCH 4/4] dts: extend meson fast tests timeout

[Patrick Robb]

The current timeout is insufficient for running
fast tests on some systems which are running DTS.
This timeout extension resolves the issue for the
systems in question.

Signed-off-by: Patrick Robb <probb@iol.unh.edu>
---
 dts/tests/TestSuite_smoke_tests.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
index 9e4d06a1c4..72901e59d5 100644
--- a/dts/tests/TestSuite_smoke_tests.py
+++ b/dts/tests/TestSuite_smoke_tests.py
@@ -59,7 +59,7 @@ def test_unit_tests(self) -> None:
             Run the ``fast-tests`` unit test suite through meson.
         """
         self.sut_node.main_session.send_command(
-            f"meson test -C {self.dpdk_build_dir_path} --suite fast-tests -t 60",
+            f"meson test -C {self.dpdk_build_dir_path} --suite fast-tests -t 120",
             480,
             verify=True,
             privileged=True,
-- 
2.48.1