* [RFC PATCH v1] dts: skip test cases based on capabilities
@ 2024-03-01 15:54 Juraj Linkeš
2024-04-11 8:48 ` [RFC PATCH v2] " Juraj Linkeš
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
0 siblings, 2 replies; 75+ messages in thread
From: Juraj Linkeš @ 2024-03-01 15:54 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte
Cc: dev, Juraj Linkeš
The devices under test may not support the capabilities required by
various test cases. Add support for:
1. Marking test suites and test cases with required capabilities,
2. Getting which required capabilities are supported by the device under
test,
3. And then skipping test suites and test cases if their required
capabilities are not supported by the device.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/remote_session/__init__.py | 2 +-
dts/framework/remote_session/testpmd_shell.py | 48 ++++++++++
dts/framework/runner.py | 46 ++++++++--
dts/framework/test_result.py | 90 +++++++++++++++----
dts/framework/test_suite.py | 25 ++++++
dts/framework/testbed_model/sut_node.py | 25 +++++-
dts/tests/TestSuite_hello_world.py | 4 +-
7 files changed, 209 insertions(+), 31 deletions(-)
diff --git a/dts/framework/remote_session/__init__.py b/dts/framework/remote_session/__init__.py
index 1910c81c3c..f18a9f2259 100644
--- a/dts/framework/remote_session/__init__.py
+++ b/dts/framework/remote_session/__init__.py
@@ -22,7 +22,7 @@
from .python_shell import PythonShell
from .remote_session import CommandResult, RemoteSession
from .ssh_session import SSHSession
-from .testpmd_shell import TestPmdShell
+from .testpmd_shell import NicCapability, TestPmdShell
def create_remote_session(
diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index 0184cc2e71..71379d54b8 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -15,6 +15,9 @@
testpmd_shell.close()
"""
+from collections.abc import MutableSet
+from enum import Enum
+from functools import partial
from pathlib import PurePath
from typing import Callable, ClassVar
@@ -82,3 +85,48 @@ def get_devices(self) -> list[TestPmdDevice]:
if "device name:" in line.lower():
dev_list.append(TestPmdDevice(line))
return dev_list
+
+ def get_capas_rxq(
+ self, supported_capabilities: MutableSet, unsupported_capabilities: MutableSet
+ ) -> None:
+ """Get all rxq capabilities and divide them into supported and unsupported.
+
+ Args:
+ supported_capabilities: A set where capabilities which are supported will be stored.
+ unsupported_capabilities: A set where capabilities which are
+ not supported will be stored.
+ """
+ self._logger.debug("Getting rxq capabilities.")
+ command = "show rxq info 0 0"
+ rxq_info = self.send_command(command)
+ for line in rxq_info.split("\n"):
+ bare_line = line.strip()
+ if bare_line.startswith("RX scattered packets:"):
+ if bare_line.endswith("on"):
+ supported_capabilities.add(NicCapability.scattered_rx)
+ else:
+ unsupported_capabilities.add(NicCapability.scattered_rx)
+
+ def close(self) -> None:
+ """Send the quit command to close testpmd."""
+ self.send_command("quit", "Bye...")
+ super().close()
+
+
+class NicCapability(Enum):
+ """A mapping between capability names and the associated :class:`TestPmdShell` methods.
+
+ The :class:`TestPmdShell` method executes the command that checks
+ whether the capability is supported.
+
+ The signature of each :class:`TestPmdShell` method must be::
+
+ fn(self, supported_capabilities: MutableSet, unsupported_capabilities: MutableSet) -> None
+
+ The function must execute the testpmd command from which the capability support can be obtained.
+ If multiple capabilities can be obtained from the same testpmd command, each should be obtained
+ in one function. These capabilities should then be added to `supported_capabilities` or
+ `unsupported_capabilities` based on their support.
+ """
+
+ scattered_rx = partial(TestPmdShell.get_capas_rxq)
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index db8e3ba96b..7407ea30b8 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -501,6 +501,12 @@ def _run_test_suites(
The method assumes the build target we're testing has already been built on the SUT node.
The current build target thus corresponds to the current DPDK build present on the SUT node.
+ Before running any suites, the method determines whether they should be skipped
+ by inspecting any required capabilities the test suite needs and comparing those
+ to capabilities supported by the tested NIC. If all capabilities are supported,
+ the suite is run. If all test cases in a test suite would be skipped, the whole test suite
+ is skipped (the setup and teardown is not run).
+
If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites
in the current build target won't be executed.
@@ -512,10 +518,30 @@ def _run_test_suites(
test_suites_with_cases: The test suites with test cases to run.
"""
end_build_target = False
+ required_capabilities = set()
+ supported_capabilities = set()
+ for test_suite_with_cases in test_suites_with_cases:
+ required_capabilities.update(test_suite_with_cases.req_capabilities)
+ self._logger.debug(f"Found required capabilities: {required_capabilities}")
+ if required_capabilities:
+ supported_capabilities = sut_node.get_supported_capabilities(required_capabilities)
for test_suite_with_cases in test_suites_with_cases:
test_suite_result = build_target_result.add_test_suite(test_suite_with_cases)
+ test_suite_with_cases.mark_skip(supported_capabilities)
try:
- self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases)
+ if test_suite_with_cases:
+ self._run_test_suite(
+ sut_node,
+ tg_node,
+ test_suite_result,
+ test_suite_with_cases,
+ )
+ else:
+ self._logger.info(
+ f"Test suite execution SKIPPED: "
+ f"{test_suite_with_cases.test_suite_class.__name__}"
+ )
+ test_suite_result.update_setup(Result.SKIP)
except BlockingTestSuiteError as e:
self._logger.exception(
f"An error occurred within {test_suite_with_cases.test_suite_class.__name__}. "
@@ -614,14 +640,18 @@ def _execute_test_suite(
test_case_result = test_suite_result.add_test_case(test_case_name)
all_attempts = SETTINGS.re_run + 1
attempt_nr = 1
- self._run_test_case(test_suite, test_case_method, test_case_result)
- while not test_case_result and attempt_nr < all_attempts:
- attempt_nr += 1
- self._logger.info(
- f"Re-running FAILED test case '{test_case_name}'. "
- f"Attempt number {attempt_nr} out of {all_attempts}."
- )
+ if TestSuiteWithCases.should_not_be_skipped(test_case_method):
self._run_test_case(test_suite, test_case_method, test_case_result)
+ while not test_case_result and attempt_nr < all_attempts:
+ attempt_nr += 1
+ self._logger.info(
+ f"Re-running FAILED test case '{test_case_name}'. "
+ f"Attempt number {attempt_nr} out of {all_attempts}."
+ )
+ self._run_test_case(test_suite, test_case_method, test_case_result)
+ else:
+ self._logger.info(f"Test case execution SKIPPED: {test_case_name}")
+ test_case_result.update_setup(Result.SKIP)
def _run_test_case(
self,
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 28f84fd793..26c58a8606 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -24,12 +24,14 @@
"""
import os.path
-from collections.abc import MutableSequence
-from dataclasses import dataclass
+from collections.abc import MutableSequence, MutableSet
+from dataclasses import dataclass, field
from enum import Enum, auto
from types import MethodType
from typing import Union
+from framework.remote_session import NicCapability
+
from .config import (
OS,
Architecture,
@@ -64,6 +66,14 @@ class is to hold a subset of test cases (which could be all test cases) because
test_suite_class: type[TestSuite]
test_cases: list[MethodType]
+ req_capabilities: set[NicCapability] = field(default_factory=set, init=False)
+
+ def __post_init__(self):
+ """Gather the required capabilities of the test suite and all test cases."""
+ for test_object in [self.test_suite_class] + self.test_cases:
+ test_object.skip = False
+ if hasattr(test_object, "req_capa"):
+ self.req_capabilities.update(test_object.req_capa)
def create_config(self) -> TestSuiteConfig:
"""Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
@@ -76,6 +86,47 @@ def create_config(self) -> TestSuiteConfig:
test_cases=[test_case.__name__ for test_case in self.test_cases],
)
+ def mark_skip(self, supported_capabilities: MutableSet[NicCapability]) -> None:
+ """Mark the test suite and test cases to be skipped.
+
+ The mark is applied is object to be skipped requires any capabilities and at least one of
+ them is not among `capabilities`.
+
+ Args:
+ supported_capabilities: The supported capabilities.
+ """
+ for test_object in [self.test_suite_class] + self.test_cases:
+ if set(getattr(test_object, "req_capa", [])) - supported_capabilities:
+ test_object.skip = True
+
+ @staticmethod
+ def should_not_be_skipped(test_object: Union[type[TestSuite] | MethodType]) -> bool:
+ """Figure out whether `test_object` should be skipped.
+
+ If `test_object` is a :class:`TestSuite`, its test cases are not checked,
+ only the class itself.
+
+ Args:
+ test_object: The test suite or test case to be inspected.
+
+ Returns:
+ :data:`True` if the test suite or test case should be skipped, :data:`False` otherwise.
+ """
+ return not getattr(test_object, "skip", False)
+
+ def __bool__(self) -> bool:
+ """The truth value is determined by whether the test suite should be run.
+
+ Returns:
+ :data:`False` if the test suite should be skipped, :data:`True` otherwise.
+ """
+ found_test_case_to_run = False
+ for test_case in self.test_cases:
+ if self.should_not_be_skipped(test_case):
+ found_test_case_to_run = True
+ break
+ return found_test_case_to_run and self.should_not_be_skipped(self.test_suite_class)
+
class Result(Enum):
"""The possible states that a setup, a teardown or a test case may end up in."""
@@ -170,12 +221,13 @@ def update_setup(self, result: Result, error: Exception | None = None) -> None:
self.setup_result.result = result
self.setup_result.error = error
- if result in [Result.BLOCK, Result.ERROR, Result.FAIL]:
- self.update_teardown(Result.BLOCK)
- self._block_result()
+ if result != Result.PASS:
+ result_to_mark = Result.BLOCK if result != Result.SKIP else Result.SKIP
+ self.update_teardown(result_to_mark)
+ self._mark_results(result_to_mark)
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed.
+ def _mark_results(self, result) -> None:
+ """Mark the result as well as the child result as `result`.
The blocking of child results should be done in overloaded methods.
"""
@@ -390,11 +442,11 @@ def add_sut_info(self, sut_info: NodeInfo) -> None:
self.sut_os_version = sut_info.os_version
self.sut_kernel_version = sut_info.kernel_version
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ def _mark_results(self, result) -> None:
+ """Mark the result as well as the child result as `result`."""
for build_target in self._config.build_targets:
child_result = self.add_build_target(build_target)
- child_result.update_setup(Result.BLOCK)
+ child_result.update_setup(result)
class BuildTargetResult(BaseResult):
@@ -464,11 +516,11 @@ def add_build_target_info(self, versions: BuildTargetInfo) -> None:
self.compiler_version = versions.compiler_version
self.dpdk_version = versions.dpdk_version
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ def _mark_results(self, result) -> None:
+ """Mark the result as well as the child result as `result`."""
for test_suite_with_cases in self._test_suites_with_cases:
child_result = self.add_test_suite(test_suite_with_cases)
- child_result.update_setup(Result.BLOCK)
+ child_result.update_setup(result)
class TestSuiteResult(BaseResult):
@@ -508,11 +560,11 @@ def add_test_case(self, test_case_name: str) -> "TestCaseResult":
self.child_results.append(result)
return result
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ def _mark_results(self, result) -> None:
+ """Mark the result as well as the child result as `result`."""
for test_case_method in self._test_suite_with_cases.test_cases:
child_result = self.add_test_case(test_case_method.__name__)
- child_result.update_setup(Result.BLOCK)
+ child_result.update_setup(result)
class TestCaseResult(BaseResult, FixtureResult):
@@ -566,9 +618,9 @@ def add_stats(self, statistics: "Statistics") -> None:
"""
statistics += self.result
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
- self.update(Result.BLOCK)
+ def _mark_results(self, result) -> None:
+ r"""Mark the result as `result`."""
+ self.update(result)
def __bool__(self) -> bool:
"""The test case passed only if setup, teardown and the test case itself passed."""
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 1957ea7328..1d53db092f 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -13,6 +13,7 @@
* Test case verification.
"""
+from collections.abc import Callable
from ipaddress import IPv4Interface, IPv6Interface, ip_interface
from typing import ClassVar, Union
@@ -20,6 +21,8 @@
from scapy.layers.l2 import Ether # type: ignore[import]
from scapy.packet import Packet, Padding # type: ignore[import]
+from framework.remote_session import NicCapability
+
from .exception import TestCaseVerifyError
from .logger import DTSLogger, get_dts_logger
from .testbed_model import Port, PortLink, SutNode, TGNode
@@ -61,6 +64,7 @@ class TestSuite(object):
#: Whether the test suite is blocking. A failure of a blocking test suite
#: will block the execution of all subsequent test suites in the current build target.
is_blocking: ClassVar[bool] = False
+ skip: bool
_logger: DTSLogger
_port_links: list[PortLink]
_sut_port_ingress: Port
@@ -88,6 +92,7 @@ def __init__(
"""
self.sut_node = sut_node
self.tg_node = tg_node
+ self.skip = False
self._logger = get_dts_logger(self.__class__.__name__)
self._port_links = []
self._process_links()
@@ -349,3 +354,23 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool:
if received_packet.src != expected_packet.src or received_packet.dst != expected_packet.dst:
return False
return True
+
+
+def requires(capability: NicCapability) -> Callable:
+ """A decorator that marks the decorated test case or test suite as one to be skipped.
+
+ Args:
+ The capability that's required by the decorated test case or test suite.
+
+ Returns:
+ The decorated function.
+ """
+
+ def add_req_capa(func) -> Callable:
+ if hasattr(func, "req_capa"):
+ func.req_capa.append(capability)
+ else:
+ func.req_capa = [capability]
+ return func
+
+ return add_req_capa
diff --git a/dts/framework/testbed_model/sut_node.py b/dts/framework/testbed_model/sut_node.py
index c4acea38d1..943836a051 100644
--- a/dts/framework/testbed_model/sut_node.py
+++ b/dts/framework/testbed_model/sut_node.py
@@ -15,7 +15,7 @@
import tarfile
import time
from pathlib import PurePath
-from typing import Type
+from typing import Iterable, Type
from framework.config import (
BuildTargetConfiguration,
@@ -23,7 +23,7 @@
NodeInfo,
SutNodeConfiguration,
)
-from framework.remote_session import CommandResult
+from framework.remote_session import CommandResult, NicCapability, TestPmdShell
from framework.settings import SETTINGS
from framework.utils import MesonArgs
@@ -223,6 +223,27 @@ def get_build_target_info(self) -> BuildTargetInfo:
def _guess_dpdk_remote_dir(self) -> PurePath:
return self.main_session.guess_dpdk_remote_dir(self._remote_tmp_dir)
+ def get_supported_capabilities(
+ self, capabilities: Iterable[NicCapability]
+ ) -> set[NicCapability]:
+ """Get the supported capabilities of the current NIC from `capabilities`.
+
+ Args:
+ capabilities: The capabilities to verify.
+
+ Returns:
+ The set of supported capabilities of the current NIC.
+ """
+ supported_capas: set[NicCapability] = set()
+ unsupported_capas: set[NicCapability] = set()
+ self._logger.debug(f"Checking which capabilities from {capabilities} NIC are supported.")
+ testpmd_shell = self.create_interactive_shell(TestPmdShell, privileged=True)
+ for capability in capabilities:
+ if capability not in supported_capas or capability not in unsupported_capas:
+ capability.value(testpmd_shell, supported_capas, unsupported_capas)
+ del testpmd_shell
+ return supported_capas
+
def _set_up_build_target(self, build_target_config: BuildTargetConfiguration) -> None:
"""Setup DPDK on the SUT node.
diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
index fd7ff1534d..31b1564582 100644
--- a/dts/tests/TestSuite_hello_world.py
+++ b/dts/tests/TestSuite_hello_world.py
@@ -7,7 +7,8 @@
No other EAL parameters apart from cores are used.
"""
-from framework.test_suite import TestSuite
+from framework.remote_session import NicCapability
+from framework.test_suite import TestSuite, requires
from framework.testbed_model import (
LogicalCoreCount,
LogicalCoreCountFilter,
@@ -26,6 +27,7 @@ def set_up_suite(self) -> None:
"""
self.app_helloworld_path = self.sut_node.build_dpdk_app("helloworld")
+ @requires(NicCapability.scattered_rx)
def test_hello_world_single_core(self) -> None:
"""Single core test case.
--
2.34.1
^ permalink raw reply [flat|nested] 75+ messages in thread
* [RFC PATCH v2] dts: skip test cases based on capabilities
2024-03-01 15:54 [RFC PATCH v1] dts: skip test cases based on capabilities Juraj Linkeš
@ 2024-04-11 8:48 ` Juraj Linkeš
2024-05-21 15:47 ` Luca Vizzarro
` (4 more replies)
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
1 sibling, 5 replies; 75+ messages in thread
From: Juraj Linkeš @ 2024-04-11 8:48 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte
Cc: dev, Juraj Linkeš
The devices under test may not support the capabilities required by
various test cases. Add support for:
1. Marking test suites and test cases with required capabilities,
2. Getting which required capabilities are supported by the device under
test,
3. And then skipping test suites and test cases if their required
capabilities are not supported by the device.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/remote_session/__init__.py | 2 +-
dts/framework/remote_session/testpmd_shell.py | 44 ++++++++-
dts/framework/runner.py | 46 ++++++++--
dts/framework/test_result.py | 90 +++++++++++++++----
dts/framework/test_suite.py | 25 ++++++
dts/framework/testbed_model/sut_node.py | 25 +++++-
dts/tests/TestSuite_hello_world.py | 4 +-
7 files changed, 204 insertions(+), 32 deletions(-)
diff --git a/dts/framework/remote_session/__init__.py b/dts/framework/remote_session/__init__.py
index 1910c81c3c..f18a9f2259 100644
--- a/dts/framework/remote_session/__init__.py
+++ b/dts/framework/remote_session/__init__.py
@@ -22,7 +22,7 @@
from .python_shell import PythonShell
from .remote_session import CommandResult, RemoteSession
from .ssh_session import SSHSession
-from .testpmd_shell import TestPmdShell
+from .testpmd_shell import NicCapability, TestPmdShell
def create_remote_session(
diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index cb2ab6bd00..f6783af621 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -16,7 +16,9 @@
"""
import time
-from enum import auto
+from collections.abc import MutableSet
+from enum import Enum, auto
+from functools import partial
from pathlib import PurePath
from typing import Callable, ClassVar
@@ -229,3 +231,43 @@ def close(self) -> None:
"""Overrides :meth:`~.interactive_shell.close`."""
self.send_command("quit", "")
return super().close()
+
+ def get_capas_rxq(
+ self, supported_capabilities: MutableSet, unsupported_capabilities: MutableSet
+ ) -> None:
+ """Get all rxq capabilities and divide them into supported and unsupported.
+
+ Args:
+ supported_capabilities: A set where capabilities which are supported will be stored.
+ unsupported_capabilities: A set where capabilities which are
+ not supported will be stored.
+ """
+ self._logger.debug("Getting rxq capabilities.")
+ command = "show rxq info 0 0"
+ rxq_info = self.send_command(command)
+ for line in rxq_info.split("\n"):
+ bare_line = line.strip()
+ if bare_line.startswith("RX scattered packets:"):
+ if bare_line.endswith("on"):
+ supported_capabilities.add(NicCapability.scattered_rx)
+ else:
+ unsupported_capabilities.add(NicCapability.scattered_rx)
+
+
+class NicCapability(Enum):
+ """A mapping between capability names and the associated :class:`TestPmdShell` methods.
+
+ The :class:`TestPmdShell` method executes the command that checks
+ whether the capability is supported.
+
+ The signature of each :class:`TestPmdShell` method must be::
+
+ fn(self, supported_capabilities: MutableSet, unsupported_capabilities: MutableSet) -> None
+
+ The function must execute the testpmd command from which the capability support can be obtained.
+ If multiple capabilities can be obtained from the same testpmd command, each should be obtained
+ in one function. These capabilities should then be added to `supported_capabilities` or
+ `unsupported_capabilities` based on their support.
+ """
+
+ scattered_rx = partial(TestPmdShell.get_capas_rxq)
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index db8e3ba96b..7407ea30b8 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -501,6 +501,12 @@ def _run_test_suites(
The method assumes the build target we're testing has already been built on the SUT node.
The current build target thus corresponds to the current DPDK build present on the SUT node.
+ Before running any suites, the method determines whether they should be skipped
+ by inspecting any required capabilities the test suite needs and comparing those
+ to capabilities supported by the tested NIC. If all capabilities are supported,
+ the suite is run. If all test cases in a test suite would be skipped, the whole test suite
+ is skipped (the setup and teardown is not run).
+
If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites
in the current build target won't be executed.
@@ -512,10 +518,30 @@ def _run_test_suites(
test_suites_with_cases: The test suites with test cases to run.
"""
end_build_target = False
+ required_capabilities = set()
+ supported_capabilities = set()
+ for test_suite_with_cases in test_suites_with_cases:
+ required_capabilities.update(test_suite_with_cases.req_capabilities)
+ self._logger.debug(f"Found required capabilities: {required_capabilities}")
+ if required_capabilities:
+ supported_capabilities = sut_node.get_supported_capabilities(required_capabilities)
for test_suite_with_cases in test_suites_with_cases:
test_suite_result = build_target_result.add_test_suite(test_suite_with_cases)
+ test_suite_with_cases.mark_skip(supported_capabilities)
try:
- self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases)
+ if test_suite_with_cases:
+ self._run_test_suite(
+ sut_node,
+ tg_node,
+ test_suite_result,
+ test_suite_with_cases,
+ )
+ else:
+ self._logger.info(
+ f"Test suite execution SKIPPED: "
+ f"{test_suite_with_cases.test_suite_class.__name__}"
+ )
+ test_suite_result.update_setup(Result.SKIP)
except BlockingTestSuiteError as e:
self._logger.exception(
f"An error occurred within {test_suite_with_cases.test_suite_class.__name__}. "
@@ -614,14 +640,18 @@ def _execute_test_suite(
test_case_result = test_suite_result.add_test_case(test_case_name)
all_attempts = SETTINGS.re_run + 1
attempt_nr = 1
- self._run_test_case(test_suite, test_case_method, test_case_result)
- while not test_case_result and attempt_nr < all_attempts:
- attempt_nr += 1
- self._logger.info(
- f"Re-running FAILED test case '{test_case_name}'. "
- f"Attempt number {attempt_nr} out of {all_attempts}."
- )
+ if TestSuiteWithCases.should_not_be_skipped(test_case_method):
self._run_test_case(test_suite, test_case_method, test_case_result)
+ while not test_case_result and attempt_nr < all_attempts:
+ attempt_nr += 1
+ self._logger.info(
+ f"Re-running FAILED test case '{test_case_name}'. "
+ f"Attempt number {attempt_nr} out of {all_attempts}."
+ )
+ self._run_test_case(test_suite, test_case_method, test_case_result)
+ else:
+ self._logger.info(f"Test case execution SKIPPED: {test_case_name}")
+ test_case_result.update_setup(Result.SKIP)
def _run_test_case(
self,
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 28f84fd793..26c58a8606 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -24,12 +24,14 @@
"""
import os.path
-from collections.abc import MutableSequence
-from dataclasses import dataclass
+from collections.abc import MutableSequence, MutableSet
+from dataclasses import dataclass, field
from enum import Enum, auto
from types import MethodType
from typing import Union
+from framework.remote_session import NicCapability
+
from .config import (
OS,
Architecture,
@@ -64,6 +66,14 @@ class is to hold a subset of test cases (which could be all test cases) because
test_suite_class: type[TestSuite]
test_cases: list[MethodType]
+ req_capabilities: set[NicCapability] = field(default_factory=set, init=False)
+
+ def __post_init__(self):
+ """Gather the required capabilities of the test suite and all test cases."""
+ for test_object in [self.test_suite_class] + self.test_cases:
+ test_object.skip = False
+ if hasattr(test_object, "req_capa"):
+ self.req_capabilities.update(test_object.req_capa)
def create_config(self) -> TestSuiteConfig:
"""Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
@@ -76,6 +86,47 @@ def create_config(self) -> TestSuiteConfig:
test_cases=[test_case.__name__ for test_case in self.test_cases],
)
+ def mark_skip(self, supported_capabilities: MutableSet[NicCapability]) -> None:
+ """Mark the test suite and test cases to be skipped.
+
+ The mark is applied is object to be skipped requires any capabilities and at least one of
+ them is not among `capabilities`.
+
+ Args:
+ supported_capabilities: The supported capabilities.
+ """
+ for test_object in [self.test_suite_class] + self.test_cases:
+ if set(getattr(test_object, "req_capa", [])) - supported_capabilities:
+ test_object.skip = True
+
+ @staticmethod
+ def should_not_be_skipped(test_object: Union[type[TestSuite] | MethodType]) -> bool:
+ """Figure out whether `test_object` should be skipped.
+
+ If `test_object` is a :class:`TestSuite`, its test cases are not checked,
+ only the class itself.
+
+ Args:
+ test_object: The test suite or test case to be inspected.
+
+ Returns:
+ :data:`True` if the test suite or test case should be skipped, :data:`False` otherwise.
+ """
+ return not getattr(test_object, "skip", False)
+
+ def __bool__(self) -> bool:
+ """The truth value is determined by whether the test suite should be run.
+
+ Returns:
+ :data:`False` if the test suite should be skipped, :data:`True` otherwise.
+ """
+ found_test_case_to_run = False
+ for test_case in self.test_cases:
+ if self.should_not_be_skipped(test_case):
+ found_test_case_to_run = True
+ break
+ return found_test_case_to_run and self.should_not_be_skipped(self.test_suite_class)
+
class Result(Enum):
"""The possible states that a setup, a teardown or a test case may end up in."""
@@ -170,12 +221,13 @@ def update_setup(self, result: Result, error: Exception | None = None) -> None:
self.setup_result.result = result
self.setup_result.error = error
- if result in [Result.BLOCK, Result.ERROR, Result.FAIL]:
- self.update_teardown(Result.BLOCK)
- self._block_result()
+ if result != Result.PASS:
+ result_to_mark = Result.BLOCK if result != Result.SKIP else Result.SKIP
+ self.update_teardown(result_to_mark)
+ self._mark_results(result_to_mark)
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed.
+ def _mark_results(self, result) -> None:
+ """Mark the result as well as the child result as `result`.
The blocking of child results should be done in overloaded methods.
"""
@@ -390,11 +442,11 @@ def add_sut_info(self, sut_info: NodeInfo) -> None:
self.sut_os_version = sut_info.os_version
self.sut_kernel_version = sut_info.kernel_version
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ def _mark_results(self, result) -> None:
+ """Mark the result as well as the child result as `result`."""
for build_target in self._config.build_targets:
child_result = self.add_build_target(build_target)
- child_result.update_setup(Result.BLOCK)
+ child_result.update_setup(result)
class BuildTargetResult(BaseResult):
@@ -464,11 +516,11 @@ def add_build_target_info(self, versions: BuildTargetInfo) -> None:
self.compiler_version = versions.compiler_version
self.dpdk_version = versions.dpdk_version
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ def _mark_results(self, result) -> None:
+ """Mark the result as well as the child result as `result`."""
for test_suite_with_cases in self._test_suites_with_cases:
child_result = self.add_test_suite(test_suite_with_cases)
- child_result.update_setup(Result.BLOCK)
+ child_result.update_setup(result)
class TestSuiteResult(BaseResult):
@@ -508,11 +560,11 @@ def add_test_case(self, test_case_name: str) -> "TestCaseResult":
self.child_results.append(result)
return result
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ def _mark_results(self, result) -> None:
+ """Mark the result as well as the child result as `result`."""
for test_case_method in self._test_suite_with_cases.test_cases:
child_result = self.add_test_case(test_case_method.__name__)
- child_result.update_setup(Result.BLOCK)
+ child_result.update_setup(result)
class TestCaseResult(BaseResult, FixtureResult):
@@ -566,9 +618,9 @@ def add_stats(self, statistics: "Statistics") -> None:
"""
statistics += self.result
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
- self.update(Result.BLOCK)
+ def _mark_results(self, result) -> None:
+ r"""Mark the result as `result`."""
+ self.update(result)
def __bool__(self) -> bool:
"""The test case passed only if setup, teardown and the test case itself passed."""
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 9c3b516002..07cdd294b9 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -13,6 +13,7 @@
* Test case verification.
"""
+from collections.abc import Callable
from ipaddress import IPv4Interface, IPv6Interface, ip_interface
from typing import ClassVar, Union
@@ -20,6 +21,8 @@
from scapy.layers.l2 import Ether # type: ignore[import]
from scapy.packet import Packet, Padding # type: ignore[import]
+from framework.remote_session import NicCapability
+
from .exception import TestCaseVerifyError
from .logger import DTSLogger, get_dts_logger
from .testbed_model import Port, PortLink, SutNode, TGNode
@@ -62,6 +65,7 @@ class TestSuite(object):
#: Whether the test suite is blocking. A failure of a blocking test suite
#: will block the execution of all subsequent test suites in the current build target.
is_blocking: ClassVar[bool] = False
+ skip: bool
_logger: DTSLogger
_port_links: list[PortLink]
_sut_port_ingress: Port
@@ -89,6 +93,7 @@ def __init__(
"""
self.sut_node = sut_node
self.tg_node = tg_node
+ self.skip = False
self._logger = get_dts_logger(self.__class__.__name__)
self._port_links = []
self._process_links()
@@ -360,3 +365,23 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool:
if received_packet.src != expected_packet.src or received_packet.dst != expected_packet.dst:
return False
return True
+
+
+def requires(capability: NicCapability) -> Callable:
+ """A decorator that marks the decorated test case or test suite as one to be skipped.
+
+ Args:
+ The capability that's required by the decorated test case or test suite.
+
+ Returns:
+ The decorated function.
+ """
+
+ def add_req_capa(func) -> Callable:
+ if hasattr(func, "req_capa"):
+ func.req_capa.append(capability)
+ else:
+ func.req_capa = [capability]
+ return func
+
+ return add_req_capa
diff --git a/dts/framework/testbed_model/sut_node.py b/dts/framework/testbed_model/sut_node.py
index 97aa26d419..1fb536735d 100644
--- a/dts/framework/testbed_model/sut_node.py
+++ b/dts/framework/testbed_model/sut_node.py
@@ -15,7 +15,7 @@
import tarfile
import time
from pathlib import PurePath
-from typing import Type
+from typing import Iterable, Type
from framework.config import (
BuildTargetConfiguration,
@@ -23,7 +23,7 @@
NodeInfo,
SutNodeConfiguration,
)
-from framework.remote_session import CommandResult
+from framework.remote_session import CommandResult, NicCapability, TestPmdShell
from framework.settings import SETTINGS
from framework.utils import MesonArgs
@@ -228,6 +228,27 @@ def get_build_target_info(self) -> BuildTargetInfo:
def _guess_dpdk_remote_dir(self) -> PurePath:
return self.main_session.guess_dpdk_remote_dir(self._remote_tmp_dir)
+ def get_supported_capabilities(
+ self, capabilities: Iterable[NicCapability]
+ ) -> set[NicCapability]:
+ """Get the supported capabilities of the current NIC from `capabilities`.
+
+ Args:
+ capabilities: The capabilities to verify.
+
+ Returns:
+ The set of supported capabilities of the current NIC.
+ """
+ supported_capas: set[NicCapability] = set()
+ unsupported_capas: set[NicCapability] = set()
+ self._logger.debug(f"Checking which capabilities from {capabilities} NIC are supported.")
+ testpmd_shell = self.create_interactive_shell(TestPmdShell, privileged=True)
+ for capability in capabilities:
+ if capability not in supported_capas or capability not in unsupported_capas:
+ capability.value(testpmd_shell, supported_capas, unsupported_capas)
+ del testpmd_shell
+ return supported_capas
+
def _set_up_build_target(self, build_target_config: BuildTargetConfiguration) -> None:
"""Setup DPDK on the SUT node.
diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
index fd7ff1534d..31b1564582 100644
--- a/dts/tests/TestSuite_hello_world.py
+++ b/dts/tests/TestSuite_hello_world.py
@@ -7,7 +7,8 @@
No other EAL parameters apart from cores are used.
"""
-from framework.test_suite import TestSuite
+from framework.remote_session import NicCapability
+from framework.test_suite import TestSuite, requires
from framework.testbed_model import (
LogicalCoreCount,
LogicalCoreCountFilter,
@@ -26,6 +27,7 @@ def set_up_suite(self) -> None:
"""
self.app_helloworld_path = self.sut_node.build_dpdk_app("helloworld")
+ @requires(NicCapability.scattered_rx)
def test_hello_world_single_core(self) -> None:
"""Single core test case.
--
2.34.1
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [RFC PATCH v2] dts: skip test cases based on capabilities
2024-04-11 8:48 ` [RFC PATCH v2] " Juraj Linkeš
@ 2024-05-21 15:47 ` Luca Vizzarro
2024-05-22 14:58 ` Luca Vizzarro
` (3 subsequent siblings)
4 siblings, 0 replies; 75+ messages in thread
From: Luca Vizzarro @ 2024-05-21 15:47 UTC (permalink / raw)
To: Juraj Linkeš,
thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
npratte
Cc: dev
On 11/04/2024 09:48, Juraj Linkeš wrote:
> + def get_capas_rxq(
> + self, supported_capabilities: MutableSet, unsupported_capabilities: MutableSet
> + ) -> None:
> + """Get all rxq capabilities and divide them into supported and unsupported.
> +
> + Args:
> + supported_capabilities: A set where capabilities which are supported will be stored.
> + unsupported_capabilities: A set where capabilities which are
> + not supported will be stored.
> + """
> + self._logger.debug("Getting rxq capabilities.")
> + command = "show rxq info 0 0"
> + rxq_info = self.send_command(command)
> + for line in rxq_info.split("\n"):
> + bare_line = line.strip()
> + if bare_line.startswith("RX scattered packets:"):
> + if bare_line.endswith("on"):
> + supported_capabilities.add(NicCapability.scattered_rx)
> + else:
> + unsupported_capabilities.add(NicCapability.scattered_rx)
It doesn't look like this works in normal condition. I've noticed that
this appears as "on" if I set --max-pkt-len=9000 on the E810-C.
Otherwise it's off... and with Jeremy's patch based on this, the
pmd_buffer_scatter test gets skipped when it's supported.
Apart from this, everything else seems to work as expected. I'll send a
review of the code as soon as possible.
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [RFC PATCH v2] dts: skip test cases based on capabilities
2024-04-11 8:48 ` [RFC PATCH v2] " Juraj Linkeš
2024-05-21 15:47 ` Luca Vizzarro
@ 2024-05-22 14:58 ` Luca Vizzarro
2024-06-07 13:13 ` Juraj Linkeš
2024-05-24 20:51 ` Nicholas Pratte
` (2 subsequent siblings)
4 siblings, 1 reply; 75+ messages in thread
From: Luca Vizzarro @ 2024-05-22 14:58 UTC (permalink / raw)
To: Juraj Linkeš,
thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
npratte
Cc: dev
Hi Juraj,
Here's my review. Excuse me for the unordinary format, but I thought
it would have just been easier to convey my suggestions through code.
Apart from the smaller suggestions, the most important one I think is
that we should make sure to enforce type checking (and hinting).
Overall I like your approach, but I think it'd be better to initialise
all the required variables per test case, so we can access them
directly without doing checks everytime. The easiest approach I can see
to do this though, is to decorate all the test cases, for example
through @test. It'd be a somewhat important change as it changes the
test writing API, but it brings some improvements while making the
system more resilient.
The comments in the code are part of the review and may refer to
either your code or mine. The diff is in working order, so you could
test the functionality if you wished.
Best regards,
Luca
---
diff --git a/dts/framework/remote_session/__init__.py b/dts/framework/remote_session/__init__.py
index f18a9f2259..d4dfed3a58 100644
--- a/dts/framework/remote_session/__init__.py
+++ b/dts/framework/remote_session/__init__.py
@@ -22,6 +22,9 @@
from .python_shell import PythonShell
from .remote_session import CommandResult, RemoteSession
from .ssh_session import SSHSession
+
+# in my testpmd params series these imports are removed as they promote eager module loading,
+# significantly increasing the chances of circular dependencies
from .testpmd_shell import NicCapability, TestPmdShell
diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index f6783af621..2b87e2e5c8 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -16,7 +16,6 @@
"""
import time
-from collections.abc import MutableSet
from enum import Enum, auto
from functools import partial
from pathlib import PurePath
@@ -232,9 +231,8 @@ def close(self) -> None:
self.send_command("quit", "")
return super().close()
- def get_capas_rxq(
- self, supported_capabilities: MutableSet, unsupported_capabilities: MutableSet
- ) -> None:
+ # the built-in `set` is a mutable set. Is there an advantage to using MutableSet?
+ def get_capas_rxq(self, supported_capabilities: set, unsupported_capabilities: set) -> None:
"""Get all rxq capabilities and divide them into supported and unsupported.
Args:
@@ -243,6 +241,7 @@ def get_capas_rxq(
not supported will be stored.
"""
self._logger.debug("Getting rxq capabilities.")
+ # this assumes that the used ports are all the same. Could this be of concern?
command = "show rxq info 0 0"
rxq_info = self.send_command(command)
for line in rxq_info.split("\n"):
@@ -270,4 +269,6 @@ class NicCapability(Enum):
`unsupported_capabilities` based on their support.
"""
+ # partial is just a high-order function that pre-fills the arguments... but we have no arguments
+ # here? Was this intentional?
scattered_rx = partial(TestPmdShell.get_capas_rxq)
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 7407ea30b8..db02735ee9 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -20,12 +20,13 @@
import importlib
import inspect
import os
-import re
import sys
from pathlib import Path
from types import MethodType
from typing import Iterable, Sequence
+from framework.remote_session.testpmd_shell import NicCapability
+
from .config import (
BuildTargetConfiguration,
Configuration,
@@ -50,7 +51,7 @@
TestSuiteResult,
TestSuiteWithCases,
)
-from .test_suite import TestSuite
+from .test_suite import TestCase, TestCaseType, TestSuite
from .testbed_model import SutNode, TGNode
@@ -305,7 +306,7 @@ def is_test_suite(object) -> bool:
def _filter_test_cases(
self, test_suite_class: type[TestSuite], test_cases_to_run: Sequence[str]
- ) -> tuple[list[MethodType], list[MethodType]]:
+ ) -> tuple[list[type[TestCase]], list[type[TestCase]]]:
"""Filter `test_cases_to_run` from `test_suite_class`.
There are two rounds of filtering if `test_cases_to_run` is not empty.
@@ -327,30 +328,28 @@ def _filter_test_cases(
"""
func_test_cases = []
perf_test_cases = []
- name_method_tuples = inspect.getmembers(test_suite_class, inspect.isfunction)
+ # By introducing the TestCase class this could be simplified.
+ # Also adding separation of concerns, delegating the auto discovery of test cases to the
+ # test suite class.
if test_cases_to_run:
- name_method_tuples = [
- (name, method) for name, method in name_method_tuples if name in test_cases_to_run
- ]
- if len(name_method_tuples) < len(test_cases_to_run):
+ test_cases = test_suite_class.get_test_cases(lambda t: t.__name__ in test_cases_to_run)
+ if len(test_cases) < len(test_cases_to_run):
missing_test_cases = set(test_cases_to_run) - {
- name for name, _ in name_method_tuples
+ test_case.__name__ for test_case in test_cases
}
raise ConfigurationError(
f"Test cases {missing_test_cases} not found among methods "
f"of {test_suite_class.__name__}."
)
+ else:
+ test_cases = test_suite_class.get_test_cases()
- for test_case_name, test_case_method in name_method_tuples:
- if re.match(self._func_test_case_regex, test_case_name):
- func_test_cases.append(test_case_method)
- elif re.match(self._perf_test_case_regex, test_case_name):
- perf_test_cases.append(test_case_method)
- elif test_cases_to_run:
- raise ConfigurationError(
- f"Method '{test_case_name}' matches neither "
- f"a functional nor a performance test case name."
- )
+ for test_case in test_cases:
+ match test_case.type:
+ case TestCaseType.PERFORMANCE:
+ perf_test_cases.append(test_case)
+ case TestCaseType.FUNCTIONAL:
+ func_test_cases.append(test_case)
return func_test_cases, perf_test_cases
@@ -489,6 +488,17 @@ def _run_build_target(
self._logger.exception("Build target teardown failed.")
build_target_result.update_teardown(Result.FAIL, e)
+ def _get_supported_required_capabilities(
+ self, sut_node: SutNode, test_suites_with_cases: Iterable[TestSuiteWithCases]
+ ) -> set[NicCapability]:
+ required_capabilities = set()
+ for test_suite_with_cases in test_suites_with_cases:
+ required_capabilities.update(test_suite_with_cases.req_capabilities)
+ self._logger.debug(f"Found required capabilities: {required_capabilities}")
+ if required_capabilities:
+ return sut_node.get_supported_capabilities(required_capabilities)
+ return set()
+
def _run_test_suites(
self,
sut_node: SutNode,
@@ -518,18 +528,17 @@ def _run_test_suites(
test_suites_with_cases: The test suites with test cases to run.
"""
end_build_target = False
- required_capabilities = set()
- supported_capabilities = set()
- for test_suite_with_cases in test_suites_with_cases:
- required_capabilities.update(test_suite_with_cases.req_capabilities)
- self._logger.debug(f"Found required capabilities: {required_capabilities}")
- if required_capabilities:
- supported_capabilities = sut_node.get_supported_capabilities(required_capabilities)
+ # extract this logic
+ supported_capabilities = self._get_supported_required_capabilities(
+ sut_node, test_suites_with_cases
+ )
for test_suite_with_cases in test_suites_with_cases:
test_suite_result = build_target_result.add_test_suite(test_suite_with_cases)
+ # not against this but perhaps something more explanatory like mark_skip_to_unsupported?
test_suite_with_cases.mark_skip(supported_capabilities)
try:
- if test_suite_with_cases:
+ # is_any_supported is more self-explanatory than an ambiguous boolean check
+ if test_suite_with_cases.is_any_supported():
self._run_test_suite(
sut_node,
tg_node,
@@ -619,7 +628,7 @@ def _run_test_suite(
def _execute_test_suite(
self,
test_suite: TestSuite,
- test_cases: Iterable[MethodType],
+ test_cases: Iterable[type[TestCase]],
test_suite_result: TestSuiteResult,
) -> None:
"""Execute all `test_cases` in `test_suite`.
@@ -640,7 +649,7 @@ def _execute_test_suite(
test_case_result = test_suite_result.add_test_case(test_case_name)
all_attempts = SETTINGS.re_run + 1
attempt_nr = 1
- if TestSuiteWithCases.should_not_be_skipped(test_case_method):
+ if not test_case_method.skip:
self._run_test_case(test_suite, test_case_method, test_case_result)
while not test_case_result and attempt_nr < all_attempts:
attempt_nr += 1
@@ -656,7 +665,7 @@ def _execute_test_suite(
def _run_test_case(
self,
test_suite: TestSuite,
- test_case_method: MethodType,
+ test_case_method: type[TestCase],
test_case_result: TestCaseResult,
) -> None:
"""Setup, execute and teardown `test_case_method` from `test_suite`.
@@ -702,7 +711,7 @@ def _run_test_case(
def _execute_test_case(
self,
test_suite: TestSuite,
- test_case_method: MethodType,
+ test_case_method: type[TestCase],
test_case_result: TestCaseResult,
) -> None:
"""Execute `test_case_method` from `test_suite`, record the result and handle failures.
@@ -716,7 +725,8 @@ def _execute_test_case(
test_case_name = test_case_method.__name__
try:
self._logger.info(f"Starting test case execution: {test_case_name}")
- test_case_method(test_suite)
+ # Explicit method binding is now required, otherwise mypy complains
+ MethodType(test_case_method, test_suite)()
test_case_result.update(Result.PASS)
self._logger.info(f"Test case execution PASSED: {test_case_name}")
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 26c58a8606..82b2dc17b8 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -24,10 +24,9 @@
"""
import os.path
-from collections.abc import MutableSequence, MutableSet
+from collections.abc import MutableSequence
from dataclasses import dataclass, field
from enum import Enum, auto
-from types import MethodType
from typing import Union
from framework.remote_session import NicCapability
@@ -46,7 +45,7 @@
from .exception import DTSError, ErrorSeverity
from .logger import DTSLogger
from .settings import SETTINGS
-from .test_suite import TestSuite
+from .test_suite import TestCase, TestSuite
@dataclass(slots=True, frozen=True)
@@ -65,7 +64,7 @@ class is to hold a subset of test cases (which could be all test cases) because
"""
test_suite_class: type[TestSuite]
- test_cases: list[MethodType]
+ test_cases: list[type[TestCase]]
req_capabilities: set[NicCapability] = field(default_factory=set, init=False)
def __post_init__(self):
@@ -86,7 +85,7 @@ def create_config(self) -> TestSuiteConfig:
test_cases=[test_case.__name__ for test_case in self.test_cases],
)
- def mark_skip(self, supported_capabilities: MutableSet[NicCapability]) -> None:
+ def mark_skip(self, supported_capabilities: set[NicCapability]) -> None:
"""Mark the test suite and test cases to be skipped.
The mark is applied is object to be skipped requires any capabilities and at least one of
@@ -95,26 +94,15 @@ def mark_skip(self, supported_capabilities: MutableSet[NicCapability]) -> None:
Args:
supported_capabilities: The supported capabilities.
"""
- for test_object in [self.test_suite_class] + self.test_cases:
- if set(getattr(test_object, "req_capa", [])) - supported_capabilities:
+ for test_object in [self.test_suite_class, *self.test_cases]:
+ # mypy picks up that both TestSuite and TestCase implement RequiresCapabilities and is
+ # happy. We could explicitly type hint the list for readability if we preferred.
+ if test_object.required_capabilities - supported_capabilities:
+ # the latest version of mypy complains about this in the original code because
+ # test_object can be distinct classes. The added protocol solves this
test_object.skip = True
- @staticmethod
- def should_not_be_skipped(test_object: Union[type[TestSuite] | MethodType]) -> bool:
- """Figure out whether `test_object` should be skipped.
-
- If `test_object` is a :class:`TestSuite`, its test cases are not checked,
- only the class itself.
-
- Args:
- test_object: The test suite or test case to be inspected.
-
- Returns:
- :data:`True` if the test suite or test case should be skipped, :data:`False` otherwise.
- """
- return not getattr(test_object, "skip", False)
-
- def __bool__(self) -> bool:
+ def is_any_supported(self) -> bool:
"""The truth value is determined by whether the test suite should be run.
Returns:
@@ -122,10 +110,12 @@ def __bool__(self) -> bool:
"""
found_test_case_to_run = False
for test_case in self.test_cases:
- if self.should_not_be_skipped(test_case):
+ # mypy and the TestCase decorators now ensure that the attributes always exist
+ # so the above static method is no longer needed
+ if not test_case.skip:
found_test_case_to_run = True
break
- return found_test_case_to_run and self.should_not_be_skipped(self.test_suite_class)
+ return found_test_case_to_run and not self.test_suite_class.skip
class Result(Enum):
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 07cdd294b9..d03f8db712 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -13,9 +13,10 @@
* Test case verification.
"""
-from collections.abc import Callable
+from enum import Enum, auto
+import inspect
from ipaddress import IPv4Interface, IPv6Interface, ip_interface
-from typing import ClassVar, Union
+from typing import Callable, ClassVar, Protocol, TypeVar, Union, cast
from scapy.layers.inet import IP # type: ignore[import]
from scapy.layers.l2 import Ether # type: ignore[import]
@@ -30,7 +31,14 @@
from .utils import get_packet_summaries
-class TestSuite(object):
+# a protocol/interface for common fields. The defaults are correctly picked up
+# by child concrete classes.
+class RequiresCapabilities(Protocol):
+ skip: ClassVar[bool] = False
+ required_capabilities: ClassVar[set[NicCapability]] = set()
+
+
+class TestSuite(RequiresCapabilities):
"""The base class with building blocks needed by most test cases.
* Test suite setup/cleanup methods to override,
@@ -65,7 +73,6 @@ class TestSuite(object):
#: Whether the test suite is blocking. A failure of a blocking test suite
#: will block the execution of all subsequent test suites in the current build target.
is_blocking: ClassVar[bool] = False
- skip: bool
_logger: DTSLogger
_port_links: list[PortLink]
_sut_port_ingress: Port
@@ -93,7 +100,7 @@ def __init__(
"""
self.sut_node = sut_node
self.tg_node = tg_node
- self.skip = False
+
self._logger = get_dts_logger(self.__class__.__name__)
self._port_links = []
self._process_links()
@@ -110,6 +117,23 @@ def __init__(
self._tg_ip_address_egress = ip_interface("192.168.100.3/24")
self._tg_ip_address_ingress = ip_interface("192.168.101.3/24")
+ # move the discovery of the test cases to TestSuite for separation of concerns
+ @classmethod
+ def get_test_cases(
+ cls, filter: Callable[[type["TestCase"]], bool] | None = None
+ ) -> set[type["TestCase"]]:
+ test_cases = set()
+ for _, method in inspect.getmembers(cls, inspect.isfunction):
+ # at runtime the function remains a function with custom attributes. An ininstance check
+ # unfortunately wouldn't work. Therefore force and test for the presence of TestCaseType
+ test_case = cast(type[TestCase], method)
+ try:
+ if isinstance(test_case.type, TestCaseType) and (not filter or filter(test_case)):
+ test_cases.add(test_case)
+ except AttributeError:
+ pass
+ return test_cases
+
def _process_links(self) -> None:
"""Construct links between SUT and TG ports."""
for sut_port in self.sut_node.ports:
@@ -367,8 +391,46 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool:
return True
-def requires(capability: NicCapability) -> Callable:
- """A decorator that marks the decorated test case or test suite as one to be skipped.
+# generic type for a method of an instance of TestSuite
+M = TypeVar("M", bound=Callable[[TestSuite], None])
+
+
+class TestCaseType(Enum):
+ FUNCTIONAL = auto()
+ PERFORMANCE = auto()
+
+
+# the protocol here is merely for static type checking and hinting.
+# the defaults don't get applied to functions like it does with inheriting
+# classes. The cast here is necessary as we are forcing mypy to understand
+# we want to treat the function as TestCase. When all the attributes are correctly
+# initialised this is fine.
+class TestCase(RequiresCapabilities, Protocol[M]):
+ type: ClassVar[TestCaseType]
+ __call__: M # necessary for mypy so that it can treat this class as the function it's shadowing
+
+ @staticmethod
+ def make_decorator(test_case_type: TestCaseType):
+ def _decorator(func: M) -> type[TestCase]:
+ test_case = cast(type[TestCase], func)
+ test_case.skip = False
+ test_case.required_capabilities = set()
+ test_case.type = test_case_type
+ return test_case
+
+ return _decorator
+
+
+# this now requires to tag all test cases with the following decorators and the advantage of:
+# - a cleaner auto discovery
+# - enforcing the TestCase type
+# - initialising all the required attributes for test cases
+test = TestCase.make_decorator(TestCaseType.FUNCTIONAL)
+perf_test = TestCase.make_decorator(TestCaseType.PERFORMANCE)
+
+
+def requires(*capabilities: NicCapability):
+ """A decorator that marks the decorated test case or test suite as skippable.
Args:
The capability that's required by the decorated test case or test suite.
@@ -377,11 +439,8 @@ def requires(capability: NicCapability) -> Callable:
The decorated function.
"""
- def add_req_capa(func) -> Callable:
- if hasattr(func, "req_capa"):
- func.req_capa.append(capability)
- else:
- func.req_capa = [capability]
- return func
+ def add_req_capa(test_case: type[TestCase]) -> type[TestCase]:
+ test_case.required_capabilities.update(capabilities)
+ return test_case
return add_req_capa
diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
index 31b1564582..61533665f8 100644
--- a/dts/tests/TestSuite_hello_world.py
+++ b/dts/tests/TestSuite_hello_world.py
@@ -8,7 +8,7 @@
"""
from framework.remote_session import NicCapability
-from framework.test_suite import TestSuite, requires
+from framework.test_suite import TestSuite, requires, test
from framework.testbed_model import (
LogicalCoreCount,
LogicalCoreCountFilter,
@@ -28,7 +28,8 @@ def set_up_suite(self) -> None:
self.app_helloworld_path = self.sut_node.build_dpdk_app("helloworld")
@requires(NicCapability.scattered_rx)
- def test_hello_world_single_core(self) -> None:
+ @test
+ def hello_world_single_core(self) -> None:
"""Single core test case.
Steps:
@@ -47,7 +48,8 @@ def test_hello_world_single_core(self) -> None:
f"helloworld didn't start on lcore{lcores[0]}",
)
- def test_hello_world_all_cores(self) -> None:
+ @test
+ def hello_world_all_cores(self) -> None:
"""All cores test case.
Steps:
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [RFC PATCH v2] dts: skip test cases based on capabilities
2024-04-11 8:48 ` [RFC PATCH v2] " Juraj Linkeš
2024-05-21 15:47 ` Luca Vizzarro
2024-05-22 14:58 ` Luca Vizzarro
@ 2024-05-24 20:51 ` Nicholas Pratte
2024-05-31 16:44 ` Luca Vizzarro
2024-06-03 14:40 ` Nicholas Pratte
4 siblings, 0 replies; 75+ messages in thread
From: Nicholas Pratte @ 2024-05-24 20:51 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, dev
I think this implementation is great, and I plan on testing it
properly with the jumbo frames suite that I am developing before
giving the final review. The only input that I could reasonably give
is a couple rewordings on the docstrings which I'll highlight below.
On Thu, Apr 11, 2024 at 4:48 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>
> The devices under test may not support the capabilities required by
> various test cases. Add support for:
> 1. Marking test suites and test cases with required capabilities,
> 2. Getting which required capabilities are supported by the device under
> test,
> 3. And then skipping test suites and test cases if their required
> capabilities are not supported by the device.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
> dts/framework/remote_session/__init__.py | 2 +-
> dts/framework/remote_session/testpmd_shell.py | 44 ++++++++-
> dts/framework/runner.py | 46 ++++++++--
> dts/framework/test_result.py | 90 +++++++++++++++----
> dts/framework/test_suite.py | 25 ++++++
> dts/framework/testbed_model/sut_node.py | 25 +++++-
> dts/tests/TestSuite_hello_world.py | 4 +-
> 7 files changed, 204 insertions(+), 32 deletions(-)
>
> diff --git a/dts/framework/remote_session/__init__.py b/dts/framework/remote_session/__init__.py
> index 1910c81c3c..f18a9f2259 100644
> --- a/dts/framework/remote_session/__init__.py
> +++ b/dts/framework/remote_session/__init__.py
> @@ -22,7 +22,7 @@
> from .python_shell import PythonShell
> from .remote_session import CommandResult, RemoteSession
> from .ssh_session import SSHSession
> -from .testpmd_shell import TestPmdShell
> +from .testpmd_shell import NicCapability, TestPmdShell
>
>
> def create_remote_session(
> diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
> index cb2ab6bd00..f6783af621 100644
> --- a/dts/framework/remote_session/testpmd_shell.py
> +++ b/dts/framework/remote_session/testpmd_shell.py
> @@ -16,7 +16,9 @@
> """
>
> import time
> -from enum import auto
> +from collections.abc import MutableSet
> +from enum import Enum, auto
> +from functools import partial
> from pathlib import PurePath
> from typing import Callable, ClassVar
>
> @@ -229,3 +231,43 @@ def close(self) -> None:
> """Overrides :meth:`~.interactive_shell.close`."""
> self.send_command("quit", "")
> return super().close()
> +
> + def get_capas_rxq(
> + self, supported_capabilities: MutableSet, unsupported_capabilities: MutableSet
> + ) -> None:
> + """Get all rxq capabilities and divide them into supported and unsupported.
> +
> + Args:
> + supported_capabilities: A set where capabilities which are supported will be stored.
> + unsupported_capabilities: A set where capabilities which are
> + not supported will be stored.
Maybe change the arg descriptions to something like "A set where
supported capabilities are stored" and "A set where unsupported
capabilities are stored."
> + """
> + self._logger.debug("Getting rxq capabilities.")
> + command = "show rxq info 0 0"
> + rxq_info = self.send_command(command)
> + for line in rxq_info.split("\n"):
> + bare_line = line.strip()
> + if bare_line.startswith("RX scattered packets:"):
> + if bare_line.endswith("on"):
> + supported_capabilities.add(NicCapability.scattered_rx)
> + else:
> + unsupported_capabilities.add(NicCapability.scattered_rx)
> +
> +
> +class NicCapability(Enum):
> + """A mapping between capability names and the associated :class:`TestPmdShell` methods.
> +
> + The :class:`TestPmdShell` method executes the command that checks
> + whether the capability is supported.
> +
> + The signature of each :class:`TestPmdShell` method must be::
> +
> + fn(self, supported_capabilities: MutableSet, unsupported_capabilities: MutableSet) -> None
> +
> + The function must execute the testpmd command from which the capability support can be obtained.
"Which capability supported can be obtained." I think there was tense
error here.
> + If multiple capabilities can be obtained from the same testpmd command, each should be obtained
> + in one function. These capabilities should then be added to `supported_capabilities` or
> + `unsupported_capabilities` based on their support.
> + """
> +
> + scattered_rx = partial(TestPmdShell.get_capas_rxq)
> diff --git a/dts/framework/runner.py b/dts/framework/runner.py
> index db8e3ba96b..7407ea30b8 100644
> --- a/dts/framework/runner.py
> +++ b/dts/framework/runner.py
> @@ -501,6 +501,12 @@ def _run_test_suites(
> The method assumes the build target we're testing has already been built on the SUT node.
> The current build target thus corresponds to the current DPDK build present on the SUT node.
>
> + Before running any suites, the method determines whether they should be skipped
> + by inspecting any required capabilities the test suite needs and comparing those
> + to capabilities supported by the tested NIC. If all capabilities are supported,
> + the suite is run. If all test cases in a test suite would be skipped, the whole test suite
> + is skipped (the setup and teardown is not run).
> +
> If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites
> in the current build target won't be executed.
>
> @@ -512,10 +518,30 @@ def _run_test_suites(
> test_suites_with_cases: The test suites with test cases to run.
> """
> end_build_target = False
> + required_capabilities = set()
> + supported_capabilities = set()
> + for test_suite_with_cases in test_suites_with_cases:
> + required_capabilities.update(test_suite_with_cases.req_capabilities)
> + self._logger.debug(f"Found required capabilities: {required_capabilities}")
> + if required_capabilities:
> + supported_capabilities = sut_node.get_supported_capabilities(required_capabilities)
> for test_suite_with_cases in test_suites_with_cases:
> test_suite_result = build_target_result.add_test_suite(test_suite_with_cases)
> + test_suite_with_cases.mark_skip(supported_capabilities)
> try:
> - self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases)
> + if test_suite_with_cases:
> + self._run_test_suite(
> + sut_node,
> + tg_node,
> + test_suite_result,
> + test_suite_with_cases,
> + )
> + else:
> + self._logger.info(
> + f"Test suite execution SKIPPED: "
> + f"{test_suite_with_cases.test_suite_class.__name__}"
> + )
> + test_suite_result.update_setup(Result.SKIP)
> except BlockingTestSuiteError as e:
> self._logger.exception(
> f"An error occurred within {test_suite_with_cases.test_suite_class.__name__}. "
> @@ -614,14 +640,18 @@ def _execute_test_suite(
> test_case_result = test_suite_result.add_test_case(test_case_name)
> all_attempts = SETTINGS.re_run + 1
> attempt_nr = 1
> - self._run_test_case(test_suite, test_case_method, test_case_result)
> - while not test_case_result and attempt_nr < all_attempts:
> - attempt_nr += 1
> - self._logger.info(
> - f"Re-running FAILED test case '{test_case_name}'. "
> - f"Attempt number {attempt_nr} out of {all_attempts}."
> - )
> + if TestSuiteWithCases.should_not_be_skipped(test_case_method):
> self._run_test_case(test_suite, test_case_method, test_case_result)
> + while not test_case_result and attempt_nr < all_attempts:
> + attempt_nr += 1
> + self._logger.info(
> + f"Re-running FAILED test case '{test_case_name}'. "
> + f"Attempt number {attempt_nr} out of {all_attempts}."
> + )
> + self._run_test_case(test_suite, test_case_method, test_case_result)
> + else:
> + self._logger.info(f"Test case execution SKIPPED: {test_case_name}")
> + test_case_result.update_setup(Result.SKIP)
>
> def _run_test_case(
> self,
> diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
> index 28f84fd793..26c58a8606 100644
> --- a/dts/framework/test_result.py
> +++ b/dts/framework/test_result.py
> @@ -24,12 +24,14 @@
> """
>
> import os.path
> -from collections.abc import MutableSequence
> -from dataclasses import dataclass
> +from collections.abc import MutableSequence, MutableSet
> +from dataclasses import dataclass, field
> from enum import Enum, auto
> from types import MethodType
> from typing import Union
>
> +from framework.remote_session import NicCapability
> +
> from .config import (
> OS,
> Architecture,
> @@ -64,6 +66,14 @@ class is to hold a subset of test cases (which could be all test cases) because
>
> test_suite_class: type[TestSuite]
> test_cases: list[MethodType]
> + req_capabilities: set[NicCapability] = field(default_factory=set, init=False)
> +
> + def __post_init__(self):
> + """Gather the required capabilities of the test suite and all test cases."""
> + for test_object in [self.test_suite_class] + self.test_cases:
> + test_object.skip = False
> + if hasattr(test_object, "req_capa"):
> + self.req_capabilities.update(test_object.req_capa)
>
> def create_config(self) -> TestSuiteConfig:
> """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
> @@ -76,6 +86,47 @@ def create_config(self) -> TestSuiteConfig:
> test_cases=[test_case.__name__ for test_case in self.test_cases],
> )
>
> + def mark_skip(self, supported_capabilities: MutableSet[NicCapability]) -> None:
> + """Mark the test suite and test cases to be skipped.
> +
> + The mark is applied is object to be skipped requires any capabilities and at least one of
"The mark is applied if the object to be skipped."
> + them is not among `capabilities`.
Maybe change 'capabilities' to 'supported_capabilities.' Unless I'm
just misunderstanding the comment.
> +
> + Args:
> + supported_capabilities: The supported capabilities.
> + """
> + for test_object in [self.test_suite_class] + self.test_cases:
> + if set(getattr(test_object, "req_capa", [])) - supported_capabilities:
> + test_object.skip = True
> +
> + @staticmethod
> + def should_not_be_skipped(test_object: Union[type[TestSuite] | MethodType]) -> bool:
> + """Figure out whether `test_object` should be skipped.
> +
> + If `test_object` is a :class:`TestSuite`, its test cases are not checked,
> + only the class itself.
> +
> + Args:
> + test_object: The test suite or test case to be inspected.
> +
> + Returns:
> + :data:`True` if the test suite or test case should be skipped, :data:`False` otherwise.
> + """
> + return not getattr(test_object, "skip", False)
> +
> + def __bool__(self) -> bool:
> + """The truth value is determined by whether the test suite should be run.
> +
> + Returns:
> + :data:`False` if the test suite should be skipped, :data:`True` otherwise.
> + """
> + found_test_case_to_run = False
> + for test_case in self.test_cases:
> + if self.should_not_be_skipped(test_case):
> + found_test_case_to_run = True
> + break
> + return found_test_case_to_run and self.should_not_be_skipped(self.test_suite_class)
> +
>
> class Result(Enum):
> """The possible states that a setup, a teardown or a test case may end up in."""
> @@ -170,12 +221,13 @@ def update_setup(self, result: Result, error: Exception | None = None) -> None:
> self.setup_result.result = result
> self.setup_result.error = error
>
> - if result in [Result.BLOCK, Result.ERROR, Result.FAIL]:
> - self.update_teardown(Result.BLOCK)
> - self._block_result()
> + if result != Result.PASS:
> + result_to_mark = Result.BLOCK if result != Result.SKIP else Result.SKIP
> + self.update_teardown(result_to_mark)
> + self._mark_results(result_to_mark)
>
> - def _block_result(self) -> None:
> - r"""Mark the result as :attr:`~Result.BLOCK`\ed.
> + def _mark_results(self, result) -> None:
> + """Mark the result as well as the child result as `result`.
>
> The blocking of child results should be done in overloaded methods.
> """
> @@ -390,11 +442,11 @@ def add_sut_info(self, sut_info: NodeInfo) -> None:
> self.sut_os_version = sut_info.os_version
> self.sut_kernel_version = sut_info.kernel_version
>
> - def _block_result(self) -> None:
> - r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
> + def _mark_results(self, result) -> None:
> + """Mark the result as well as the child result as `result`."""
> for build_target in self._config.build_targets:
> child_result = self.add_build_target(build_target)
> - child_result.update_setup(Result.BLOCK)
> + child_result.update_setup(result)
>
>
> class BuildTargetResult(BaseResult):
> @@ -464,11 +516,11 @@ def add_build_target_info(self, versions: BuildTargetInfo) -> None:
> self.compiler_version = versions.compiler_version
> self.dpdk_version = versions.dpdk_version
>
> - def _block_result(self) -> None:
> - r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
> + def _mark_results(self, result) -> None:
> + """Mark the result as well as the child result as `result`."""
> for test_suite_with_cases in self._test_suites_with_cases:
> child_result = self.add_test_suite(test_suite_with_cases)
> - child_result.update_setup(Result.BLOCK)
> + child_result.update_setup(result)
>
>
> class TestSuiteResult(BaseResult):
> @@ -508,11 +560,11 @@ def add_test_case(self, test_case_name: str) -> "TestCaseResult":
> self.child_results.append(result)
> return result
>
> - def _block_result(self) -> None:
> - r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
> + def _mark_results(self, result) -> None:
> + """Mark the result as well as the child result as `result`."""
> for test_case_method in self._test_suite_with_cases.test_cases:
> child_result = self.add_test_case(test_case_method.__name__)
> - child_result.update_setup(Result.BLOCK)
> + child_result.update_setup(result)
>
>
> class TestCaseResult(BaseResult, FixtureResult):
> @@ -566,9 +618,9 @@ def add_stats(self, statistics: "Statistics") -> None:
> """
> statistics += self.result
>
> - def _block_result(self) -> None:
> - r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
> - self.update(Result.BLOCK)
> + def _mark_results(self, result) -> None:
> + r"""Mark the result as `result`."""
> + self.update(result)
>
> def __bool__(self) -> bool:
> """The test case passed only if setup, teardown and the test case itself passed."""
> diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
> index 9c3b516002..07cdd294b9 100644
> --- a/dts/framework/test_suite.py
> +++ b/dts/framework/test_suite.py
> @@ -13,6 +13,7 @@
> * Test case verification.
> """
>
> +from collections.abc import Callable
> from ipaddress import IPv4Interface, IPv6Interface, ip_interface
> from typing import ClassVar, Union
>
> @@ -20,6 +21,8 @@
> from scapy.layers.l2 import Ether # type: ignore[import]
> from scapy.packet import Packet, Padding # type: ignore[import]
>
> +from framework.remote_session import NicCapability
> +
> from .exception import TestCaseVerifyError
> from .logger import DTSLogger, get_dts_logger
> from .testbed_model import Port, PortLink, SutNode, TGNode
> @@ -62,6 +65,7 @@ class TestSuite(object):
> #: Whether the test suite is blocking. A failure of a blocking test suite
> #: will block the execution of all subsequent test suites in the current build target.
> is_blocking: ClassVar[bool] = False
> + skip: bool
> _logger: DTSLogger
> _port_links: list[PortLink]
> _sut_port_ingress: Port
> @@ -89,6 +93,7 @@ def __init__(
> """
> self.sut_node = sut_node
> self.tg_node = tg_node
> + self.skip = False
> self._logger = get_dts_logger(self.__class__.__name__)
> self._port_links = []
> self._process_links()
> @@ -360,3 +365,23 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool:
> if received_packet.src != expected_packet.src or received_packet.dst != expected_packet.dst:
> return False
> return True
> +
> +
> +def requires(capability: NicCapability) -> Callable:
> + """A decorator that marks the decorated test case or test suite as one to be skipped.
I think there might be an error here. Are you trying to say "A
decorator that marks the test case/test suite as having additional
requirements" ?
> +
> + Args:
> + The capability that's required by the decorated test case or test suite.
> +
> + Returns:
> + The decorated function.
> + """
> +
> + def add_req_capa(func) -> Callable:
> + if hasattr(func, "req_capa"):
> + func.req_capa.append(capability)
> + else:
> + func.req_capa = [capability]
> + return func
> +
> + return add_req_capa
> diff --git a/dts/framework/testbed_model/sut_node.py b/dts/framework/testbed_model/sut_node.py
> index 97aa26d419..1fb536735d 100644
> --- a/dts/framework/testbed_model/sut_node.py
> +++ b/dts/framework/testbed_model/sut_node.py
> @@ -15,7 +15,7 @@
> import tarfile
> import time
> from pathlib import PurePath
> -from typing import Type
> +from typing import Iterable, Type
>
> from framework.config import (
> BuildTargetConfiguration,
> @@ -23,7 +23,7 @@
> NodeInfo,
> SutNodeConfiguration,
> )
> -from framework.remote_session import CommandResult
> +from framework.remote_session import CommandResult, NicCapability, TestPmdShell
> from framework.settings import SETTINGS
> from framework.utils import MesonArgs
>
> @@ -228,6 +228,27 @@ def get_build_target_info(self) -> BuildTargetInfo:
> def _guess_dpdk_remote_dir(self) -> PurePath:
> return self.main_session.guess_dpdk_remote_dir(self._remote_tmp_dir)
>
> + def get_supported_capabilities(
> + self, capabilities: Iterable[NicCapability]
> + ) -> set[NicCapability]:
> + """Get the supported capabilities of the current NIC from `capabilities`.
I wonder if it might be more readable to change the 'capabilities'
variable to 'required_capabilities' or something along those lines.
Although I do understand why you have selected 'capabilities' in the
first place if the capabilities being passed in may not necessarily be
required capabilities 100% of the time.
> +
> + Args:
> + capabilities: The capabilities to verify.
> +
> + Returns:
> + The set of supported capabilities of the current NIC.
> + """
> + supported_capas: set[NicCapability] = set()
> + unsupported_capas: set[NicCapability] = set()
> + self._logger.debug(f"Checking which capabilities from {capabilities} NIC are supported.")
> + testpmd_shell = self.create_interactive_shell(TestPmdShell, privileged=True)
> + for capability in capabilities:
> + if capability not in supported_capas or capability not in unsupported_capas:
> + capability.value(testpmd_shell, supported_capas, unsupported_capas)
> + del testpmd_shell
> + return supported_capas
> +
> def _set_up_build_target(self, build_target_config: BuildTargetConfiguration) -> None:
> """Setup DPDK on the SUT node.
>
> diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
> index fd7ff1534d..31b1564582 100644
> --- a/dts/tests/TestSuite_hello_world.py
> +++ b/dts/tests/TestSuite_hello_world.py
> @@ -7,7 +7,8 @@
> No other EAL parameters apart from cores are used.
> """
>
> -from framework.test_suite import TestSuite
> +from framework.remote_session import NicCapability
> +from framework.test_suite import TestSuite, requires
> from framework.testbed_model import (
> LogicalCoreCount,
> LogicalCoreCountFilter,
> @@ -26,6 +27,7 @@ def set_up_suite(self) -> None:
> """
> self.app_helloworld_path = self.sut_node.build_dpdk_app("helloworld")
>
> + @requires(NicCapability.scattered_rx)
> def test_hello_world_single_core(self) -> None:
> """Single core test case.
>
> --
> 2.34.1
>
The above comments are basically just nitpicks, but if nothing else, I
figured I'd bring it to your attention.
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [RFC PATCH v2] dts: skip test cases based on capabilities
2024-04-11 8:48 ` [RFC PATCH v2] " Juraj Linkeš
` (2 preceding siblings ...)
2024-05-24 20:51 ` Nicholas Pratte
@ 2024-05-31 16:44 ` Luca Vizzarro
2024-06-05 13:55 ` Patrick Robb
2024-06-03 14:40 ` Nicholas Pratte
4 siblings, 1 reply; 75+ messages in thread
From: Luca Vizzarro @ 2024-05-31 16:44 UTC (permalink / raw)
To: Juraj Linkeš,
thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
npratte
Cc: dev
Hi again Juraj,
sorry for yet another comment!
On 11/04/2024 09:48, Juraj Linkeš wrote:
> + def get_capas_rxq(
> + self, supported_capabilities: MutableSet, unsupported_capabilities: MutableSet
> + ) -> None:
> + """Get all rxq capabilities and divide them into supported and unsupported.
> +
> + Args:
> + supported_capabilities: A set where capabilities which are supported will be stored.
> + unsupported_capabilities: A set where capabilities which are
> + not supported will be stored.
> + """
> + self._logger.debug("Getting rxq capabilities.")
> + command = "show rxq info 0 0"
In my testing of Jeremy's patches which depend on this one ("Add second
scatter test case"), I've discovered that the Intel E810-C NIC I am
using to test does not automatically show "RX scattered packets: on".
But I've noticed it does if the MTU is set to something big like 9000.
I've tested a change of this by adding:
self.set_port_mtu(0, 9000)
> + rxq_info = self.send_command(command)
self.set_port_mtu(1, 9000)
And it seems to work alright. I've also tested this specific change with
Mellanox NICs and it didn't seem to affect them at all. No errors or
problems and they still showed "RX scattered packets: off" as expected.
The `set_port_mtu` method comes from Jeremy's patch...
> + for line in rxq_info.split("\n"):
> + bare_line = line.strip()
> + if bare_line.startswith("RX scattered packets:"):
> + if bare_line.endswith("on"):
> + supported_capabilities.add(NicCapability.scattered_rx)
> + else:
> + unsupported_capabilities.add(NicCapability.scattered_rx)
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [RFC PATCH v2] dts: skip test cases based on capabilities
2024-04-11 8:48 ` [RFC PATCH v2] " Juraj Linkeš
` (3 preceding siblings ...)
2024-05-31 16:44 ` Luca Vizzarro
@ 2024-06-03 14:40 ` Nicholas Pratte
2024-06-07 13:20 ` Juraj Linkeš
4 siblings, 1 reply; 75+ messages in thread
From: Nicholas Pratte @ 2024-06-03 14:40 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, dev
I was able to use this implementation on the in-development
jumboframes suite, setting restrictions on the required link speeds of
NICs and using this as a requirement to run all test cases. While the
framework you've developed is intuitive for true/false capabilities,
this may not be the case for other device capabilities such as link
speed, where perhaps someone might want to support a certain range of
speeds (I also acknowledge that this may be a needless feature). I
personally found implementing this to be a head-scratcher, and I
ultimately ended up implementing this using a lower bound link speed
instead of accepting a range of speeds. The reason for me implementing
this at all is because of some complications within old DTS's
jumboframes implementation. In old DTS, the test suite would check for
1GB NICs within certain test cases and modify the MTU lengths because
of some inconsistent logic. You can see what I am referring to in the
link below, take a look at test_jumboframes_bigger_jumbo, if you are
interested.
https://git.dpdk.org/tools/dts/tree/tests/TestSuite_jumboframes.py
A solution to this problem is to set a restriction on the speed of
NICs for the test suite, but whether or not this is a viable solution
may require further discussion. This issue is its own conversation,
but I'm bringing it up in this thread since we may run into
requirements issues like this in the future, but I'm not so sure what
the rest of you guys think, or if you guys think it is a viable
concern at all.
On Thu, Apr 11, 2024 at 4:48 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>
> The devices under test may not support the capabilities required by
> various test cases. Add support for:
> 1. Marking test suites and test cases with required capabilities,
> 2. Getting which required capabilities are supported by the device under
> test,
> 3. And then skipping test suites and test cases if their required
> capabilities are not supported by the device.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
> dts/framework/remote_session/__init__.py | 2 +-
> dts/framework/remote_session/testpmd_shell.py | 44 ++++++++-
> dts/framework/runner.py | 46 ++++++++--
> dts/framework/test_result.py | 90 +++++++++++++++----
> dts/framework/test_suite.py | 25 ++++++
> dts/framework/testbed_model/sut_node.py | 25 +++++-
> dts/tests/TestSuite_hello_world.py | 4 +-
> 7 files changed, 204 insertions(+), 32 deletions(-)
>
> diff --git a/dts/framework/remote_session/__init__.py b/dts/framework/remote_session/__init__.py
> index 1910c81c3c..f18a9f2259 100644
> --- a/dts/framework/remote_session/__init__.py
> +++ b/dts/framework/remote_session/__init__.py
> @@ -22,7 +22,7 @@
> from .python_shell import PythonShell
> from .remote_session import CommandResult, RemoteSession
> from .ssh_session import SSHSession
> -from .testpmd_shell import TestPmdShell
> +from .testpmd_shell import NicCapability, TestPmdShell
>
>
> def create_remote_session(
> diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
> index cb2ab6bd00..f6783af621 100644
> --- a/dts/framework/remote_session/testpmd_shell.py
> +++ b/dts/framework/remote_session/testpmd_shell.py
> @@ -16,7 +16,9 @@
> """
>
> import time
> -from enum import auto
> +from collections.abc import MutableSet
> +from enum import Enum, auto
> +from functools import partial
> from pathlib import PurePath
> from typing import Callable, ClassVar
>
> @@ -229,3 +231,43 @@ def close(self) -> None:
> """Overrides :meth:`~.interactive_shell.close`."""
> self.send_command("quit", "")
> return super().close()
> +
> + def get_capas_rxq(
> + self, supported_capabilities: MutableSet, unsupported_capabilities: MutableSet
> + ) -> None:
> + """Get all rxq capabilities and divide them into supported and unsupported.
> +
> + Args:
> + supported_capabilities: A set where capabilities which are supported will be stored.
> + unsupported_capabilities: A set where capabilities which are
> + not supported will be stored.
> + """
> + self._logger.debug("Getting rxq capabilities.")
> + command = "show rxq info 0 0"
> + rxq_info = self.send_command(command)
> + for line in rxq_info.split("\n"):
> + bare_line = line.strip()
> + if bare_line.startswith("RX scattered packets:"):
> + if bare_line.endswith("on"):
> + supported_capabilities.add(NicCapability.scattered_rx)
> + else:
> + unsupported_capabilities.add(NicCapability.scattered_rx)
> +
> +
> +class NicCapability(Enum):
> + """A mapping between capability names and the associated :class:`TestPmdShell` methods.
> +
> + The :class:`TestPmdShell` method executes the command that checks
> + whether the capability is supported.
> +
> + The signature of each :class:`TestPmdShell` method must be::
> +
> + fn(self, supported_capabilities: MutableSet, unsupported_capabilities: MutableSet) -> None
> +
> + The function must execute the testpmd command from which the capability support can be obtained.
> + If multiple capabilities can be obtained from the same testpmd command, each should be obtained
> + in one function. These capabilities should then be added to `supported_capabilities` or
> + `unsupported_capabilities` based on their support.
> + """
> +
> + scattered_rx = partial(TestPmdShell.get_capas_rxq)
> diff --git a/dts/framework/runner.py b/dts/framework/runner.py
> index db8e3ba96b..7407ea30b8 100644
> --- a/dts/framework/runner.py
> +++ b/dts/framework/runner.py
> @@ -501,6 +501,12 @@ def _run_test_suites(
> The method assumes the build target we're testing has already been built on the SUT node.
> The current build target thus corresponds to the current DPDK build present on the SUT node.
>
> + Before running any suites, the method determines whether they should be skipped
> + by inspecting any required capabilities the test suite needs and comparing those
> + to capabilities supported by the tested NIC. If all capabilities are supported,
> + the suite is run. If all test cases in a test suite would be skipped, the whole test suite
> + is skipped (the setup and teardown is not run).
> +
> If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites
> in the current build target won't be executed.
>
> @@ -512,10 +518,30 @@ def _run_test_suites(
> test_suites_with_cases: The test suites with test cases to run.
> """
> end_build_target = False
> + required_capabilities = set()
> + supported_capabilities = set()
> + for test_suite_with_cases in test_suites_with_cases:
> + required_capabilities.update(test_suite_with_cases.req_capabilities)
> + self._logger.debug(f"Found required capabilities: {required_capabilities}")
> + if required_capabilities:
> + supported_capabilities = sut_node.get_supported_capabilities(required_capabilities)
> for test_suite_with_cases in test_suites_with_cases:
> test_suite_result = build_target_result.add_test_suite(test_suite_with_cases)
> + test_suite_with_cases.mark_skip(supported_capabilities)
> try:
> - self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases)
> + if test_suite_with_cases:
> + self._run_test_suite(
> + sut_node,
> + tg_node,
> + test_suite_result,
> + test_suite_with_cases,
> + )
> + else:
> + self._logger.info(
> + f"Test suite execution SKIPPED: "
> + f"{test_suite_with_cases.test_suite_class.__name__}"
> + )
> + test_suite_result.update_setup(Result.SKIP)
> except BlockingTestSuiteError as e:
> self._logger.exception(
> f"An error occurred within {test_suite_with_cases.test_suite_class.__name__}. "
> @@ -614,14 +640,18 @@ def _execute_test_suite(
> test_case_result = test_suite_result.add_test_case(test_case_name)
> all_attempts = SETTINGS.re_run + 1
> attempt_nr = 1
> - self._run_test_case(test_suite, test_case_method, test_case_result)
> - while not test_case_result and attempt_nr < all_attempts:
> - attempt_nr += 1
> - self._logger.info(
> - f"Re-running FAILED test case '{test_case_name}'. "
> - f"Attempt number {attempt_nr} out of {all_attempts}."
> - )
> + if TestSuiteWithCases.should_not_be_skipped(test_case_method):
> self._run_test_case(test_suite, test_case_method, test_case_result)
> + while not test_case_result and attempt_nr < all_attempts:
> + attempt_nr += 1
> + self._logger.info(
> + f"Re-running FAILED test case '{test_case_name}'. "
> + f"Attempt number {attempt_nr} out of {all_attempts}."
> + )
> + self._run_test_case(test_suite, test_case_method, test_case_result)
> + else:
> + self._logger.info(f"Test case execution SKIPPED: {test_case_name}")
> + test_case_result.update_setup(Result.SKIP)
>
> def _run_test_case(
> self,
> diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
> index 28f84fd793..26c58a8606 100644
> --- a/dts/framework/test_result.py
> +++ b/dts/framework/test_result.py
> @@ -24,12 +24,14 @@
> """
>
> import os.path
> -from collections.abc import MutableSequence
> -from dataclasses import dataclass
> +from collections.abc import MutableSequence, MutableSet
> +from dataclasses import dataclass, field
> from enum import Enum, auto
> from types import MethodType
> from typing import Union
>
> +from framework.remote_session import NicCapability
> +
> from .config import (
> OS,
> Architecture,
> @@ -64,6 +66,14 @@ class is to hold a subset of test cases (which could be all test cases) because
>
> test_suite_class: type[TestSuite]
> test_cases: list[MethodType]
> + req_capabilities: set[NicCapability] = field(default_factory=set, init=False)
> +
> + def __post_init__(self):
> + """Gather the required capabilities of the test suite and all test cases."""
> + for test_object in [self.test_suite_class] + self.test_cases:
> + test_object.skip = False
> + if hasattr(test_object, "req_capa"):
> + self.req_capabilities.update(test_object.req_capa)
>
> def create_config(self) -> TestSuiteConfig:
> """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
> @@ -76,6 +86,47 @@ def create_config(self) -> TestSuiteConfig:
> test_cases=[test_case.__name__ for test_case in self.test_cases],
> )
>
> + def mark_skip(self, supported_capabilities: MutableSet[NicCapability]) -> None:
> + """Mark the test suite and test cases to be skipped.
> +
> + The mark is applied is object to be skipped requires any capabilities and at least one of
> + them is not among `capabilities`.
> +
> + Args:
> + supported_capabilities: The supported capabilities.
> + """
> + for test_object in [self.test_suite_class] + self.test_cases:
> + if set(getattr(test_object, "req_capa", [])) - supported_capabilities:
> + test_object.skip = True
> +
> + @staticmethod
> + def should_not_be_skipped(test_object: Union[type[TestSuite] | MethodType]) -> bool:
> + """Figure out whether `test_object` should be skipped.
> +
> + If `test_object` is a :class:`TestSuite`, its test cases are not checked,
> + only the class itself.
> +
> + Args:
> + test_object: The test suite or test case to be inspected.
> +
> + Returns:
> + :data:`True` if the test suite or test case should be skipped, :data:`False` otherwise.
> + """
> + return not getattr(test_object, "skip", False)
> +
> + def __bool__(self) -> bool:
> + """The truth value is determined by whether the test suite should be run.
> +
> + Returns:
> + :data:`False` if the test suite should be skipped, :data:`True` otherwise.
> + """
> + found_test_case_to_run = False
> + for test_case in self.test_cases:
> + if self.should_not_be_skipped(test_case):
> + found_test_case_to_run = True
> + break
> + return found_test_case_to_run and self.should_not_be_skipped(self.test_suite_class)
> +
>
> class Result(Enum):
> """The possible states that a setup, a teardown or a test case may end up in."""
> @@ -170,12 +221,13 @@ def update_setup(self, result: Result, error: Exception | None = None) -> None:
> self.setup_result.result = result
> self.setup_result.error = error
>
> - if result in [Result.BLOCK, Result.ERROR, Result.FAIL]:
> - self.update_teardown(Result.BLOCK)
> - self._block_result()
> + if result != Result.PASS:
> + result_to_mark = Result.BLOCK if result != Result.SKIP else Result.SKIP
> + self.update_teardown(result_to_mark)
> + self._mark_results(result_to_mark)
>
> - def _block_result(self) -> None:
> - r"""Mark the result as :attr:`~Result.BLOCK`\ed.
> + def _mark_results(self, result) -> None:
> + """Mark the result as well as the child result as `result`.
>
> The blocking of child results should be done in overloaded methods.
> """
> @@ -390,11 +442,11 @@ def add_sut_info(self, sut_info: NodeInfo) -> None:
> self.sut_os_version = sut_info.os_version
> self.sut_kernel_version = sut_info.kernel_version
>
> - def _block_result(self) -> None:
> - r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
> + def _mark_results(self, result) -> None:
> + """Mark the result as well as the child result as `result`."""
> for build_target in self._config.build_targets:
> child_result = self.add_build_target(build_target)
> - child_result.update_setup(Result.BLOCK)
> + child_result.update_setup(result)
>
>
> class BuildTargetResult(BaseResult):
> @@ -464,11 +516,11 @@ def add_build_target_info(self, versions: BuildTargetInfo) -> None:
> self.compiler_version = versions.compiler_version
> self.dpdk_version = versions.dpdk_version
>
> - def _block_result(self) -> None:
> - r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
> + def _mark_results(self, result) -> None:
> + """Mark the result as well as the child result as `result`."""
> for test_suite_with_cases in self._test_suites_with_cases:
> child_result = self.add_test_suite(test_suite_with_cases)
> - child_result.update_setup(Result.BLOCK)
> + child_result.update_setup(result)
>
>
> class TestSuiteResult(BaseResult):
> @@ -508,11 +560,11 @@ def add_test_case(self, test_case_name: str) -> "TestCaseResult":
> self.child_results.append(result)
> return result
>
> - def _block_result(self) -> None:
> - r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
> + def _mark_results(self, result) -> None:
> + """Mark the result as well as the child result as `result`."""
> for test_case_method in self._test_suite_with_cases.test_cases:
> child_result = self.add_test_case(test_case_method.__name__)
> - child_result.update_setup(Result.BLOCK)
> + child_result.update_setup(result)
>
>
> class TestCaseResult(BaseResult, FixtureResult):
> @@ -566,9 +618,9 @@ def add_stats(self, statistics: "Statistics") -> None:
> """
> statistics += self.result
>
> - def _block_result(self) -> None:
> - r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
> - self.update(Result.BLOCK)
> + def _mark_results(self, result) -> None:
> + r"""Mark the result as `result`."""
> + self.update(result)
>
> def __bool__(self) -> bool:
> """The test case passed only if setup, teardown and the test case itself passed."""
> diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
> index 9c3b516002..07cdd294b9 100644
> --- a/dts/framework/test_suite.py
> +++ b/dts/framework/test_suite.py
> @@ -13,6 +13,7 @@
> * Test case verification.
> """
>
> +from collections.abc import Callable
> from ipaddress import IPv4Interface, IPv6Interface, ip_interface
> from typing import ClassVar, Union
>
> @@ -20,6 +21,8 @@
> from scapy.layers.l2 import Ether # type: ignore[import]
> from scapy.packet import Packet, Padding # type: ignore[import]
>
> +from framework.remote_session import NicCapability
> +
> from .exception import TestCaseVerifyError
> from .logger import DTSLogger, get_dts_logger
> from .testbed_model import Port, PortLink, SutNode, TGNode
> @@ -62,6 +65,7 @@ class TestSuite(object):
> #: Whether the test suite is blocking. A failure of a blocking test suite
> #: will block the execution of all subsequent test suites in the current build target.
> is_blocking: ClassVar[bool] = False
> + skip: bool
> _logger: DTSLogger
> _port_links: list[PortLink]
> _sut_port_ingress: Port
> @@ -89,6 +93,7 @@ def __init__(
> """
> self.sut_node = sut_node
> self.tg_node = tg_node
> + self.skip = False
> self._logger = get_dts_logger(self.__class__.__name__)
> self._port_links = []
> self._process_links()
> @@ -360,3 +365,23 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool:
> if received_packet.src != expected_packet.src or received_packet.dst != expected_packet.dst:
> return False
> return True
> +
> +
> +def requires(capability: NicCapability) -> Callable:
> + """A decorator that marks the decorated test case or test suite as one to be skipped.
> +
> + Args:
> + The capability that's required by the decorated test case or test suite.
> +
> + Returns:
> + The decorated function.
> + """
> +
> + def add_req_capa(func) -> Callable:
> + if hasattr(func, "req_capa"):
> + func.req_capa.append(capability)
> + else:
> + func.req_capa = [capability]
> + return func
> +
> + return add_req_capa
> diff --git a/dts/framework/testbed_model/sut_node.py b/dts/framework/testbed_model/sut_node.py
> index 97aa26d419..1fb536735d 100644
> --- a/dts/framework/testbed_model/sut_node.py
> +++ b/dts/framework/testbed_model/sut_node.py
> @@ -15,7 +15,7 @@
> import tarfile
> import time
> from pathlib import PurePath
> -from typing import Type
> +from typing import Iterable, Type
>
> from framework.config import (
> BuildTargetConfiguration,
> @@ -23,7 +23,7 @@
> NodeInfo,
> SutNodeConfiguration,
> )
> -from framework.remote_session import CommandResult
> +from framework.remote_session import CommandResult, NicCapability, TestPmdShell
> from framework.settings import SETTINGS
> from framework.utils import MesonArgs
>
> @@ -228,6 +228,27 @@ def get_build_target_info(self) -> BuildTargetInfo:
> def _guess_dpdk_remote_dir(self) -> PurePath:
> return self.main_session.guess_dpdk_remote_dir(self._remote_tmp_dir)
>
> + def get_supported_capabilities(
> + self, capabilities: Iterable[NicCapability]
> + ) -> set[NicCapability]:
> + """Get the supported capabilities of the current NIC from `capabilities`.
> +
> + Args:
> + capabilities: The capabilities to verify.
> +
> + Returns:
> + The set of supported capabilities of the current NIC.
> + """
> + supported_capas: set[NicCapability] = set()
> + unsupported_capas: set[NicCapability] = set()
> + self._logger.debug(f"Checking which capabilities from {capabilities} NIC are supported.")
> + testpmd_shell = self.create_interactive_shell(TestPmdShell, privileged=True)
> + for capability in capabilities:
> + if capability not in supported_capas or capability not in unsupported_capas:
> + capability.value(testpmd_shell, supported_capas, unsupported_capas)
> + del testpmd_shell
> + return supported_capas
> +
> def _set_up_build_target(self, build_target_config: BuildTargetConfiguration) -> None:
> """Setup DPDK on the SUT node.
>
> diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
> index fd7ff1534d..31b1564582 100644
> --- a/dts/tests/TestSuite_hello_world.py
> +++ b/dts/tests/TestSuite_hello_world.py
> @@ -7,7 +7,8 @@
> No other EAL parameters apart from cores are used.
> """
>
> -from framework.test_suite import TestSuite
> +from framework.remote_session import NicCapability
> +from framework.test_suite import TestSuite, requires
> from framework.testbed_model import (
> LogicalCoreCount,
> LogicalCoreCountFilter,
> @@ -26,6 +27,7 @@ def set_up_suite(self) -> None:
> """
> self.app_helloworld_path = self.sut_node.build_dpdk_app("helloworld")
>
> + @requires(NicCapability.scattered_rx)
> def test_hello_world_single_core(self) -> None:
> """Single core test case.
>
> --
> 2.34.1
>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [RFC PATCH v2] dts: skip test cases based on capabilities
2024-05-31 16:44 ` Luca Vizzarro
@ 2024-06-05 13:55 ` Patrick Robb
2024-06-06 13:36 ` Jeremy Spewock
0 siblings, 1 reply; 75+ messages in thread
From: Patrick Robb @ 2024-06-05 13:55 UTC (permalink / raw)
To: jspewock
Cc: Juraj Linkeš,
thomas, Honnappa.Nagarahalli, paul.szczepanek, npratte, dev,
Luca Vizzarro
[-- Attachment #1: Type: text/plain, Size: 1697 bytes --]
On Fri, May 31, 2024 at 12:44 PM Luca Vizzarro <Luca.Vizzarro@arm.com>
wrote:
>
> In my testing of Jeremy's patches which depend on this one ("Add second
> scatter test case"), I've discovered that the Intel E810-C NIC I am
> using to test does not automatically show "RX scattered packets: on".
> But I've noticed it does if the MTU is set to something big like 9000.
>
> I've tested a change of this by adding:
>
> self.set_port_mtu(0, 9000)
> > + rxq_info = self.send_command(command)
> self.set_port_mtu(1, 9000)
>
> And it seems to work alright. I've also tested this specific change with
> Mellanox NICs and it didn't seem to affect them at all. No errors or
> problems and they still showed "RX scattered packets: off" as expected.
>
> The `set_port_mtu` method comes from Jeremy's patch...
>
>
>
Hi Jeremy,
Sounds like Luca's way ahead of me here, but I wanted to note that I did
run from the capabilities patch + Jeremy's new Scatter patch, across these
NICs:
Mellanox CX5
Broadcom 57414 25G
Broadcom P2100G
Intel XL710 40G
And in call cases scatter_mbuf_2048 skips,
and scatter_mbuf_2048_with_offload runs.
The 2nd case passed in all cases, excluding the XL710 where it errors with
"Test pmd failed to set fwd mode to mac." I can double check that to ensure
there was no setup error on my part, but I think the more interesting part
is the skip on the non-offload testcase, as I recall Jeremy saying that the
XL710 was expected to natively support scatter and run the first testcase.
I can do a rerun, adding in the MTU modifier, and see if the same
adjustment happens as with the E810 as Luca describes.
[-- Attachment #2: Type: text/html, Size: 2305 bytes --]
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [RFC PATCH v2] dts: skip test cases based on capabilities
2024-06-05 13:55 ` Patrick Robb
@ 2024-06-06 13:36 ` Jeremy Spewock
0 siblings, 0 replies; 75+ messages in thread
From: Jeremy Spewock @ 2024-06-06 13:36 UTC (permalink / raw)
To: Patrick Robb
Cc: Juraj Linkeš,
thomas, Honnappa.Nagarahalli, paul.szczepanek, npratte, dev,
Luca Vizzarro
On Wed, Jun 5, 2024 at 9:55 AM Patrick Robb <probb@iol.unh.edu> wrote:
>
>
>
> On Fri, May 31, 2024 at 12:44 PM Luca Vizzarro <Luca.Vizzarro@arm.com> wrote:
>>
>>
>> In my testing of Jeremy's patches which depend on this one ("Add second
>> scatter test case"), I've discovered that the Intel E810-C NIC I am
>> using to test does not automatically show "RX scattered packets: on".
>> But I've noticed it does if the MTU is set to something big like 9000.
>>
>> I've tested a change of this by adding:
>>
>> self.set_port_mtu(0, 9000)
>> > + rxq_info = self.send_command(command)
>> self.set_port_mtu(1, 9000)
>>
>> And it seems to work alright. I've also tested this specific change with
>> Mellanox NICs and it didn't seem to affect them at all. No errors or
>> problems and they still showed "RX scattered packets: off" as expected.
>>
>> The `set_port_mtu` method comes from Jeremy's patch...
>>
>>
>
> Hi Jeremy,
>
> Sounds like Luca's way ahead of me here, but I wanted to note that I did run from the capabilities patch + Jeremy's new Scatter patch, across these NICs:
>
> Mellanox CX5
> Broadcom 57414 25G
> Broadcom P2100G
> Intel XL710 40G
>
> And in call cases scatter_mbuf_2048 skips, and scatter_mbuf_2048_with_offload runs.
>
> The 2nd case passed in all cases, excluding the XL710 where it errors with "Test pmd failed to set fwd mode to mac." I can double check that to ensure there was no setup error on my part, but I think the more interesting part is the skip on the non-offload testcase, as I recall Jeremy saying that the XL710 was expected to natively support scatter and run the first testcase.
The "failing to set forwarding mode" is strange, I assume this is
likely because of another issue that Luca noted which was some Intel
NICs taking longer to start testpmd and the timeout not being long
enough to allow for proper startup. If this is the case it would have
essentially "poisoned" the testpmd output buffer and caused all of the
verification steps of subsequent commands to fail. This should be
fixed in the new series of my patch that increases this timeout to 5
seconds.
>
> I can do a rerun, adding in the MTU modifier, and see if the same adjustment happens as with the E810 as Luca describes.
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [RFC PATCH v2] dts: skip test cases based on capabilities
2024-05-22 14:58 ` Luca Vizzarro
@ 2024-06-07 13:13 ` Juraj Linkeš
2024-06-11 9:51 ` Luca Vizzarro
0 siblings, 1 reply; 75+ messages in thread
From: Juraj Linkeš @ 2024-06-07 13:13 UTC (permalink / raw)
To: Luca Vizzarro, thomas, Honnappa.Nagarahalli, jspewock, probb,
paul.szczepanek, npratte
Cc: dev
On 22. 5. 2024 16:58, Luca Vizzarro wrote:
> Hi Juraj,
>
> Here's my review. Excuse me for the unordinary format, but I thought
> it would have just been easier to convey my suggestions through code.
> Apart from the smaller suggestions, the most important one I think is
> that we should make sure to enforce type checking (and hinting).
> Overall I like your approach, but I think it'd be better to initialise
> all the required variables per test case, so we can access them
> directly without doing checks everytime. The easiest approach I can see
> to do this though, is to decorate all the test cases, for example
> through @test. It'd be a somewhat important change as it changes the
> test writing API, but it brings some improvements while making the
> system more resilient.
>
The format is a bit wonky, but I was able to figure it out. I answered
some of the questions I found.
I like the idea of decorating the test cases. I was thinking of
something similar in the past and now it makes sense to implement it.
I'll definitely use some (or all :-)) of the suggestions.
> The comments in the code are part of the review and may refer to
> either your code or mine. The diff is in working order, so you could
> test the functionality if you wished.
>
> Best regards,
> Luca
>
> ---
> diff --git a/dts/framework/remote_session/__init__.py
> b/dts/framework/remote_session/__init__.py
> index f18a9f2259..d4dfed3a58 100644
> --- a/dts/framework/remote_session/__init__.py
> +++ b/dts/framework/remote_session/__init__.py
> @@ -22,6 +22,9 @@
> from .python_shell import PythonShell
> from .remote_session import CommandResult, RemoteSession
> from .ssh_session import SSHSession
> +
> +# in my testpmd params series these imports are removed as they promote
> eager module loading,
> +# significantly increasing the chances of circular dependencies
> from .testpmd_shell import NicCapability, TestPmdShell
>
>
> diff --git a/dts/framework/remote_session/testpmd_shell.py
> b/dts/framework/remote_session/testpmd_shell.py
> index f6783af621..2b87e2e5c8 100644
> --- a/dts/framework/remote_session/testpmd_shell.py
> +++ b/dts/framework/remote_session/testpmd_shell.py
> @@ -16,7 +16,6 @@
> """
>
> import time
> -from collections.abc import MutableSet
> from enum import Enum, auto
> from functools import partial
> from pathlib import PurePath
> @@ -232,9 +231,8 @@ def close(self) -> None:
> self.send_command("quit", "")
> return super().close()
>
> - def get_capas_rxq(
> - self, supported_capabilities: MutableSet,
> unsupported_capabilities: MutableSet
> - ) -> None:
> + # the built-in `set` is a mutable set. Is there an advantage to
> using MutableSet?
From what I can tell, it's best practice to be as broad as possible
with input types. set is just one class, MutableSet could be any class
that's a mutable set.
> + def get_capas_rxq(self, supported_capabilities: set,
> unsupported_capabilities: set) -> None:
> """Get all rxq capabilities and divide them into supported and
> unsupported.
>
> Args:
> @@ -243,6 +241,7 @@ def get_capas_rxq(
> not supported will be stored.
> """
> self._logger.debug("Getting rxq capabilities.")
> + # this assumes that the used ports are all the same. Could this
> be of concern?
Our current assumption is one NIC per one execution, so this should be
safe, at least for now.
> command = "show rxq info 0 0"
> rxq_info = self.send_command(command)
> for line in rxq_info.split("\n"):
> @@ -270,4 +269,6 @@ class NicCapability(Enum):
> `unsupported_capabilities` based on their support.
> """
>
> + # partial is just a high-order function that pre-fills the
> arguments... but we have no arguments
> + # here? Was this intentional?
It's necessary because of the interaction between Enums and functions.
Without partial, accessing NicCapability.scattered_rx returns the
function instead of the enum.
> scattered_rx = partial(TestPmdShell.get_capas_rxq)
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [RFC PATCH v2] dts: skip test cases based on capabilities
2024-06-03 14:40 ` Nicholas Pratte
@ 2024-06-07 13:20 ` Juraj Linkeš
0 siblings, 0 replies; 75+ messages in thread
From: Juraj Linkeš @ 2024-06-07 13:20 UTC (permalink / raw)
To: Nicholas Pratte
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, dev
On 3. 6. 2024 16:40, Nicholas Pratte wrote:
> I was able to use this implementation on the in-development
> jumboframes suite, setting restrictions on the required link speeds of
> NICs and using this as a requirement to run all test cases. While the
> framework you've developed is intuitive for true/false capabilities,
> this may not be the case for other device capabilities such as link
> speed, where perhaps someone might want to support a certain range of
> speeds (I also acknowledge that this may be a needless feature).
Would using the same decorator multiple times work? In any case, I will
think about this.
And also, thanks for the comments in the other thread, I'll incorporate
those.
> I
> personally found implementing this to be a head-scratcher, and I
> ultimately ended up implementing this using a lower bound link speed
> instead of accepting a range of speeds. The reason for me implementing
> this at all is because of some complications within old DTS's
> jumboframes implementation. In old DTS, the test suite would check for
> 1GB NICs within certain test cases and modify the MTU lengths because
> of some inconsistent logic. You can see what I am referring to in the
> link below, take a look at test_jumboframes_bigger_jumbo, if you are
> interested.
>
> https://git.dpdk.org/tools/dts/tree/tests/TestSuite_jumboframes.py
>
> A solution to this problem is to set a restriction on the speed of
> NICs for the test suite, but whether or not this is a viable solution
> may require further discussion. This issue is its own conversation,
> but I'm bringing it up in this thread since we may run into
> requirements issues like this in the future, but I'm not so sure what
> the rest of you guys think, or if you guys think it is a viable
> concern at all.
>
>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [RFC PATCH v2] dts: skip test cases based on capabilities
2024-06-07 13:13 ` Juraj Linkeš
@ 2024-06-11 9:51 ` Luca Vizzarro
2024-06-12 9:15 ` Juraj Linkeš
0 siblings, 1 reply; 75+ messages in thread
From: Luca Vizzarro @ 2024-06-11 9:51 UTC (permalink / raw)
To: Juraj Linkeš,
thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
npratte
Cc: dev
While working on my blocklist patch, I've just realised I forgot to add
another comment. I think it would be ideal to make capabilities a
generic class, and NicCapability a child of this. When collecting
capabilities we could group these by the final class, and let this final
class create the environment to test the support. For example:
class Capability(ABC):
@staticmethod
@abstractmethod
def test_environment(node, capabilities):
"""Overridable"
class NicCapability(Capability):
def test_environment(node, capabilities):
shell = TestPmdShell(node)
# test capabilities against shell capabilities
Another thing that I don't remember if I pointed out, please let's use
complete names: required_capabilities instead of req_capas. I kept
forgetting what it meant. req commonly could mean "request". If you want
to use a widely used short version for capability, that's "cap",
although in a completely different context/meaning (hardware capabilities).
On 07/06/2024 14:13, Juraj Linkeš wrote:
>> - def get_capas_rxq(
>> - self, supported_capabilities: MutableSet,
>> unsupported_capabilities: MutableSet
>> - ) -> None:
>> + # the built-in `set` is a mutable set. Is there an advantage to
>> using MutableSet?
>
> From what I can tell, it's best practice to be as broad as possible
> with input types. set is just one class, MutableSet could be any class
> that's a mutable set.
Oh, yes! Great thinking. Didn't consider the usage of custom set
classes. Although, not sure if it'll ever be needed.
>> command = "show rxq info 0 0"
>> rxq_info = self.send_command(command)
>> for line in rxq_info.split("\n"):
>> @@ -270,4 +269,6 @@ class NicCapability(Enum):
>> `unsupported_capabilities` based on their support.
>> """
>>
>> + # partial is just a high-order function that pre-fills the
>> arguments... but we have no arguments
>> + # here? Was this intentional?
>
> It's necessary because of the interaction between Enums and functions.
> Without partial, accessing NicCapability.scattered_rx returns the
> function instead of the enum.
Oh interesting. Tested now and I see that it's not making it an enum
entry when done this way. I wonder why is this.
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [RFC PATCH v2] dts: skip test cases based on capabilities
2024-06-11 9:51 ` Luca Vizzarro
@ 2024-06-12 9:15 ` Juraj Linkeš
2024-06-17 15:07 ` Luca Vizzarro
0 siblings, 1 reply; 75+ messages in thread
From: Juraj Linkeš @ 2024-06-12 9:15 UTC (permalink / raw)
To: Luca Vizzarro, thomas, Honnappa.Nagarahalli, jspewock, probb,
paul.szczepanek, npratte
Cc: dev
On 11. 6. 2024 11:51, Luca Vizzarro wrote:
> While working on my blocklist patch, I've just realised I forgot to add
> another comment. I think it would be ideal to make capabilities a
> generic class, and NicCapability a child of this. When collecting
> capabilities we could group these by the final class, and let this final
> class create the environment to test the support. For example:
>
I'm trying to wrap my head around this:
1. What do you mean group these by the final class?
2. What is this addressing or improving?
> class Capability(ABC):
> @staticmethod
> @abstractmethod
> def test_environment(node, capabilities):
> """Overridable"
>
> class NicCapability(Capability):
> def test_environment(node, capabilities):
> shell = TestPmdShell(node)
> # test capabilities against shell capabilities
>
I'm looking at this and I don't even know what to replace with this.
> Another thing that I don't remember if I pointed out, please let's use
> complete names: required_capabilities instead of req_capas. I kept
> forgetting what it meant. req commonly could mean "request". If you want
> to use a widely used short version for capability, that's "cap",
> although in a completely different context/meaning (hardware capabilities).
>
No problem.
> On 07/06/2024 14:13, Juraj Linkeš wrote:
>
>>> - def get_capas_rxq(
>>> - self, supported_capabilities: MutableSet,
>>> unsupported_capabilities: MutableSet
>>> - ) -> None:
>>> + # the built-in `set` is a mutable set. Is there an advantage to
>>> using MutableSet?
>>
>> From what I can tell, it's best practice to be as broad as possible
>> with input types. set is just one class, MutableSet could be any class
>> that's a mutable set.
>
> Oh, yes! Great thinking. Didn't consider the usage of custom set
> classes. Although, not sure if it'll ever be needed.
>
>>> command = "show rxq info 0 0"
>>> rxq_info = self.send_command(command)
>>> for line in rxq_info.split("\n"):
>>> @@ -270,4 +269,6 @@ class NicCapability(Enum):
>>> `unsupported_capabilities` based on their support.
>>> """
>>>
>>> + # partial is just a high-order function that pre-fills the
>>> arguments... but we have no arguments
>>> + # here? Was this intentional?
>>
>> It's necessary because of the interaction between Enums and functions.
>> Without partial, accessing NicCapability.scattered_rx returns the
>> function instead of the enum.
>
> Oh interesting. Tested now and I see that it's not making it an enum
> entry when done this way. I wonder why is this.
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [RFC PATCH v2] dts: skip test cases based on capabilities
2024-06-12 9:15 ` Juraj Linkeš
@ 2024-06-17 15:07 ` Luca Vizzarro
0 siblings, 0 replies; 75+ messages in thread
From: Luca Vizzarro @ 2024-06-17 15:07 UTC (permalink / raw)
To: Juraj Linkeš,
thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
npratte
Cc: dev
On 12/06/2024 10:15, Juraj Linkeš wrote:
> On 11. 6. 2024 11:51, Luca Vizzarro wrote:
>> While working on my blocklist patch, I've just realised I forgot to
>> add another comment. I think it would be ideal to make capabilities a
>> generic class, and NicCapability a child of this. When collecting
>> capabilities we could group these by the final class, and let this
>> final class create the environment to test the support. For example:
>>
>
> I'm trying to wrap my head around this:
> 1. What do you mean group these by the final class?
The ultimate child of Capability, so NicCapability.
> 2. What is this addressing or improving?
It renders requirements more generic than just having to do with
testpmd. For example for the blocklist we could have a different kind of
"capability" and we could do:
@requires(TestbedCapability.min_port(2))
class TestSuite_blocklist(TestSuite):
Given the specific test requires a minimum of 2 ports. Albeit redundant
for DTS as we are always assuming minimum 2 ports right now, but it's an
example.
>> class Capability(ABC):
>> @staticmethod
>> @abstractmethod
>> def test_environment(node, capabilities):
>> """Overridable"
>>
>> class NicCapability(Capability):
>> def test_environment(node, capabilities):
>> shell = TestPmdShell(node)
>> # test capabilities against shell capabilities
>>
>
> I'm looking at this and I don't even know what to replace with this.
Replace with what? The intention of the original message was just not to
rely on a very concrete "NicCapability", but provide an abstract
Capability instead.
^ permalink raw reply [flat|nested] 75+ messages in thread
* [PATCH v3 00/12] dts: add test skipping based on capabilities
2024-03-01 15:54 [RFC PATCH v1] dts: skip test cases based on capabilities Juraj Linkeš
2024-04-11 8:48 ` [RFC PATCH v2] " Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-21 14:53 ` [PATCH v3 01/12] dts: fix default device error handling mode Juraj Linkeš
` (12 more replies)
1 sibling, 13 replies; 75+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
Add an automated way to gather available capabilities of the tested
hardware and skip test suites or cases which require capabilities that
are not available.
This is done through two decorators:
1. The first marks a test suite method as test case. This populates the
default attributes of each test case.
2. The seconds adds the required capabilities to a test suite or case,
using the attributes from 1).
Two types of capabilities are added:
1. NIC capabilities. These are gathered once DPDK has been built because
we use testpmd for this. It's possible to add a function that will
add configuration before assessing capabilities associated with the
function. This is because some capabilities return different status
with different configuration present.
2. The topology capability. Each test case is marked as requiring a
default topology. The required topology of a test case (or a whole
test suite) may be change with the second decorator.
This is how it all works:
1. The required capabilities are first all gathered from all test suites
and test cases.
2. The list of required capabilities is divided into supported and
unsupported capabilities. In this step, the probing of hardware
and/or anything else that needs to happen to gauge whether a
capability is supported is done.
3. Each test suite and test case is then marked to be skipped if any of
their required capabilities are not supported.
Depends-on: patch-142276 ("dts: add methods for modifying MTU to testpmd
shell")
Juraj Linkeš (12):
dts: fix default device error handling mode
dts: add the aenum dependency
dts: add test case decorators
dts: add mechanism to skip test cases or suites
dts: add support for simpler topologies
dst: add basic capability support
dts: add testpmd port information caching
dts: add NIC capability support
dts: add topology capability
doc: add DTS capability doc sources
dts: add Rx offload capabilities
dts: add NIC capabilities from show port info
.../framework.testbed_model.capability.rst | 6 +
doc/api/dts/framework.testbed_model.rst | 2 +
.../dts/framework.testbed_model.topology.rst | 6 +
dts/framework/remote_session/testpmd_shell.py | 461 +++++++++++++++-
dts/framework/runner.py | 155 +++---
dts/framework/test_result.py | 120 +++--
dts/framework/test_suite.py | 161 +++++-
dts/framework/testbed_model/capability.py | 491 ++++++++++++++++++
dts/framework/testbed_model/node.py | 2 +-
dts/framework/testbed_model/port.py | 4 +-
dts/framework/testbed_model/topology.py | 128 +++++
dts/poetry.lock | 14 +-
dts/pyproject.toml | 1 +
dts/tests/TestSuite_hello_world.py | 10 +-
dts/tests/TestSuite_os_udp.py | 3 +-
dts/tests/TestSuite_pmd_buffer_scatter.py | 14 +-
dts/tests/TestSuite_smoke_tests.py | 8 +-
17 files changed, 1429 insertions(+), 157 deletions(-)
create mode 100644 doc/api/dts/framework.testbed_model.capability.rst
create mode 100644 doc/api/dts/framework.testbed_model.topology.rst
create mode 100644 dts/framework/testbed_model/capability.py
create mode 100644 dts/framework/testbed_model/topology.py
--
2.34.1
^ permalink raw reply [flat|nested] 75+ messages in thread
* [PATCH v3 01/12] dts: fix default device error handling mode
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 16:42 ` Jeremy Spewock
` (2 more replies)
2024-08-21 14:53 ` [PATCH v3 02/12] dts: add the aenum dependency Juraj Linkeš
` (11 subsequent siblings)
12 siblings, 3 replies; 75+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
The device_error_handling_mode of testpmd port may not be present, e.g.
in VM ports.
Fixes: 61d5bc9bf974 ("dts: add port info command to testpmd shell")
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/remote_session/testpmd_shell.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index a5347b07dc..b4ad253020 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -465,8 +465,8 @@ class TestPmdPort(TextParser):
metadata=DeviceCapabilitiesFlag.make_parser(),
)
#:
- device_error_handling_mode: DeviceErrorHandlingMode = field(
- metadata=DeviceErrorHandlingMode.make_parser()
+ device_error_handling_mode: DeviceErrorHandlingMode | None = field(
+ default=None, metadata=DeviceErrorHandlingMode.make_parser()
)
#:
device_private_info: str | None = field(
--
2.34.1
^ permalink raw reply [flat|nested] 75+ messages in thread
* [PATCH v3 02/12] dts: add the aenum dependency
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
2024-08-21 14:53 ` [PATCH v3 01/12] dts: fix default device error handling mode Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 16:42 ` Jeremy Spewock
` (2 more replies)
2024-08-21 14:53 ` [PATCH v3 03/12] dts: add test case decorators Juraj Linkeš
` (10 subsequent siblings)
12 siblings, 3 replies; 75+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
Regular Python enumerations create only one instance for members with
the same value, such as:
class MyEnum(Enum):
foo = 1
bar = 1
MyEnum.foo and MyEnum.bar are aliases that return the same instance.
DTS needs to return different instances in the above scenario so that we
can map capabilities with different names to the same function that
retrieves the capabilities.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/poetry.lock | 14 +++++++++++++-
dts/pyproject.toml | 1 +
2 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/dts/poetry.lock b/dts/poetry.lock
index 2dd8bad498..cf5f6569c6 100644
--- a/dts/poetry.lock
+++ b/dts/poetry.lock
@@ -1,5 +1,17 @@
# This file is automatically @generated by Poetry 1.8.2 and should not be changed by hand.
+[[package]]
+name = "aenum"
+version = "3.1.15"
+description = "Advanced Enumerations (compatible with Python's stdlib Enum), NamedTuples, and NamedConstants"
+optional = false
+python-versions = "*"
+files = [
+ {file = "aenum-3.1.15-py2-none-any.whl", hash = "sha256:27b1710b9d084de6e2e695dab78fe9f269de924b51ae2850170ee7e1ca6288a5"},
+ {file = "aenum-3.1.15-py3-none-any.whl", hash = "sha256:e0dfaeea4c2bd362144b87377e2c61d91958c5ed0b4daf89cb6f45ae23af6288"},
+ {file = "aenum-3.1.15.tar.gz", hash = "sha256:8cbd76cd18c4f870ff39b24284d3ea028fbe8731a58df3aa581e434c575b9559"},
+]
+
[[package]]
name = "alabaster"
version = "0.7.13"
@@ -1350,4 +1362,4 @@ jsonschema = ">=4,<5"
[metadata]
lock-version = "2.0"
python-versions = "^3.10"
-content-hash = "6db17f96cb31fb463b0b0a31dff9c02aa72641e0bffd8a610033fe2324006c43"
+content-hash = "6f20ce05310df93fed1d392160d1653ae5de5c6f260a5865eb3c6111a7c2b394"
diff --git a/dts/pyproject.toml b/dts/pyproject.toml
index 38281f0e39..6e347852cc 100644
--- a/dts/pyproject.toml
+++ b/dts/pyproject.toml
@@ -26,6 +26,7 @@ fabric = "^2.7.1"
scapy = "^2.5.0"
pydocstyle = "6.1.1"
typing-extensions = "^4.11.0"
+aenum = "^3.1.15"
[tool.poetry.group.dev.dependencies]
mypy = "^1.10.0"
--
2.34.1
^ permalink raw reply [flat|nested] 75+ messages in thread
* [PATCH v3 03/12] dts: add test case decorators
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
2024-08-21 14:53 ` [PATCH v3 01/12] dts: fix default device error handling mode Juraj Linkeš
2024-08-21 14:53 ` [PATCH v3 02/12] dts: add the aenum dependency Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 16:50 ` Jeremy Spewock
` (2 more replies)
2024-08-21 14:53 ` [PATCH v3 04/12] dts: add mechanism to skip test cases or suites Juraj Linkeš
` (9 subsequent siblings)
12 siblings, 3 replies; 75+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
Add decorators for functional and performance test cases. These
decorators add attributes to the decorated test cases.
With the addition of decorators, we change the test case discovery
mechanism from looking at test case names according to a regex to simply
checking an attribute of the function added with one of the decorators.
The decorators allow us to add further variables to test cases.
Also move the test case filtering to TestSuite while changing the
mechanism to separate the logic in a more sensible manner.
Bugzilla ID: 1460
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/runner.py | 93 ++++------------
dts/framework/test_result.py | 5 +-
dts/framework/test_suite.py | 125 +++++++++++++++++++++-
dts/tests/TestSuite_hello_world.py | 8 +-
dts/tests/TestSuite_os_udp.py | 3 +-
dts/tests/TestSuite_pmd_buffer_scatter.py | 3 +-
dts/tests/TestSuite_smoke_tests.py | 6 +-
7 files changed, 160 insertions(+), 83 deletions(-)
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 6b6f6a05f5..525f119ab6 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -20,11 +20,10 @@
import importlib
import inspect
import os
-import re
import sys
from pathlib import Path
-from types import FunctionType
-from typing import Iterable, Sequence
+from types import MethodType
+from typing import Iterable
from framework.testbed_model.sut_node import SutNode
from framework.testbed_model.tg_node import TGNode
@@ -53,7 +52,7 @@
TestSuiteResult,
TestSuiteWithCases,
)
-from .test_suite import TestSuite
+from .test_suite import TestCase, TestSuite
class DTSRunner:
@@ -232,9 +231,9 @@ def _get_test_suites_with_cases(
for test_suite_config in test_suite_configs:
test_suite_class = self._get_test_suite_class(test_suite_config.test_suite)
- test_cases = []
- func_test_cases, perf_test_cases = self._filter_test_cases(
- test_suite_class, test_suite_config.test_cases
+ test_cases: list[type[TestCase]] = []
+ func_test_cases, perf_test_cases = test_suite_class.get_test_cases(
+ test_suite_config.test_cases
)
if func:
test_cases.extend(func_test_cases)
@@ -309,57 +308,6 @@ def is_test_suite(object) -> bool:
f"Couldn't find any valid test suites in {test_suite_module.__name__}."
)
- def _filter_test_cases(
- self, test_suite_class: type[TestSuite], test_cases_to_run: Sequence[str]
- ) -> tuple[list[FunctionType], list[FunctionType]]:
- """Filter `test_cases_to_run` from `test_suite_class`.
-
- There are two rounds of filtering if `test_cases_to_run` is not empty.
- The first filters `test_cases_to_run` from all methods of `test_suite_class`.
- Then the methods are separated into functional and performance test cases.
- If a method matches neither the functional nor performance name prefix, it's an error.
-
- Args:
- test_suite_class: The class of the test suite.
- test_cases_to_run: Test case names to filter from `test_suite_class`.
- If empty, return all matching test cases.
-
- Returns:
- A list of test case methods that should be executed.
-
- Raises:
- ConfigurationError: If a test case from `test_cases_to_run` is not found
- or it doesn't match either the functional nor performance name prefix.
- """
- func_test_cases = []
- perf_test_cases = []
- name_method_tuples = inspect.getmembers(test_suite_class, inspect.isfunction)
- if test_cases_to_run:
- name_method_tuples = [
- (name, method) for name, method in name_method_tuples if name in test_cases_to_run
- ]
- if len(name_method_tuples) < len(test_cases_to_run):
- missing_test_cases = set(test_cases_to_run) - {
- name for name, _ in name_method_tuples
- }
- raise ConfigurationError(
- f"Test cases {missing_test_cases} not found among methods "
- f"of {test_suite_class.__name__}."
- )
-
- for test_case_name, test_case_method in name_method_tuples:
- if re.match(self._func_test_case_regex, test_case_name):
- func_test_cases.append(test_case_method)
- elif re.match(self._perf_test_case_regex, test_case_name):
- perf_test_cases.append(test_case_method)
- elif test_cases_to_run:
- raise ConfigurationError(
- f"Method '{test_case_name}' matches neither "
- f"a functional nor a performance test case name."
- )
-
- return func_test_cases, perf_test_cases
-
def _connect_nodes_and_run_test_run(
self,
sut_nodes: dict[str, SutNode],
@@ -607,7 +555,7 @@ def _run_test_suite(
def _execute_test_suite(
self,
test_suite: TestSuite,
- test_cases: Iterable[FunctionType],
+ test_cases: Iterable[type[TestCase]],
test_suite_result: TestSuiteResult,
) -> None:
"""Execute all `test_cases` in `test_suite`.
@@ -618,29 +566,29 @@ def _execute_test_suite(
Args:
test_suite: The test suite object.
- test_cases: The list of test case methods.
+ test_cases: The list of test case functions.
test_suite_result: The test suite level result object associated
with the current test suite.
"""
self._logger.set_stage(DtsStage.test_suite)
- for test_case_method in test_cases:
- test_case_name = test_case_method.__name__
+ for test_case in test_cases:
+ test_case_name = test_case.__name__
test_case_result = test_suite_result.add_test_case(test_case_name)
all_attempts = SETTINGS.re_run + 1
attempt_nr = 1
- self._run_test_case(test_suite, test_case_method, test_case_result)
+ self._run_test_case(test_suite, test_case, test_case_result)
while not test_case_result and attempt_nr < all_attempts:
attempt_nr += 1
self._logger.info(
f"Re-running FAILED test case '{test_case_name}'. "
f"Attempt number {attempt_nr} out of {all_attempts}."
)
- self._run_test_case(test_suite, test_case_method, test_case_result)
+ self._run_test_case(test_suite, test_case, test_case_result)
def _run_test_case(
self,
test_suite: TestSuite,
- test_case_method: FunctionType,
+ test_case: type[TestCase],
test_case_result: TestCaseResult,
) -> None:
"""Setup, execute and teardown `test_case_method` from `test_suite`.
@@ -649,11 +597,11 @@ def _run_test_case(
Args:
test_suite: The test suite object.
- test_case_method: The test case method.
+ test_case: The test case function.
test_case_result: The test case level result object associated
with the current test case.
"""
- test_case_name = test_case_method.__name__
+ test_case_name = test_case.__name__
try:
# run set_up function for each case
@@ -668,7 +616,7 @@ def _run_test_case(
else:
# run test case if setup was successful
- self._execute_test_case(test_suite, test_case_method, test_case_result)
+ self._execute_test_case(test_suite, test_case, test_case_result)
finally:
try:
@@ -686,21 +634,22 @@ def _run_test_case(
def _execute_test_case(
self,
test_suite: TestSuite,
- test_case_method: FunctionType,
+ test_case: type[TestCase],
test_case_result: TestCaseResult,
) -> None:
"""Execute `test_case_method` from `test_suite`, record the result and handle failures.
Args:
test_suite: The test suite object.
- test_case_method: The test case method.
+ test_case: The test case function.
test_case_result: The test case level result object associated
with the current test case.
"""
- test_case_name = test_case_method.__name__
+ test_case_name = test_case.__name__
try:
self._logger.info(f"Starting test case execution: {test_case_name}")
- test_case_method(test_suite)
+ # Explicit method binding is required, otherwise mypy complains
+ MethodType(test_case, test_suite)()
test_case_result.update(Result.PASS)
self._logger.info(f"Test case execution PASSED: {test_case_name}")
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 5694a2482b..b1ca584523 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -27,7 +27,6 @@
from collections.abc import MutableSequence
from dataclasses import dataclass
from enum import Enum, auto
-from types import FunctionType
from typing import Union
from .config import (
@@ -44,7 +43,7 @@
from .exception import DTSError, ErrorSeverity
from .logger import DTSLogger
from .settings import SETTINGS
-from .test_suite import TestSuite
+from .test_suite import TestCase, TestSuite
@dataclass(slots=True, frozen=True)
@@ -63,7 +62,7 @@ class is to hold a subset of test cases (which could be all test cases) because
"""
test_suite_class: type[TestSuite]
- test_cases: list[FunctionType]
+ test_cases: list[type[TestCase]]
def create_config(self) -> TestSuiteConfig:
"""Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 694b2eba65..b4ee0f9039 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -13,8 +13,11 @@
* Test case verification.
"""
+import inspect
+from collections.abc import Callable, Sequence
+from enum import Enum, auto
from ipaddress import IPv4Interface, IPv6Interface, ip_interface
-from typing import ClassVar, Union
+from typing import ClassVar, Protocol, TypeVar, Union, cast
from scapy.layers.inet import IP # type: ignore[import-untyped]
from scapy.layers.l2 import Ether # type: ignore[import-untyped]
@@ -27,7 +30,7 @@
PacketFilteringConfig,
)
-from .exception import TestCaseVerifyError
+from .exception import ConfigurationError, TestCaseVerifyError
from .logger import DTSLogger, get_dts_logger
from .utils import get_packet_summaries
@@ -120,6 +123,68 @@ def _process_links(self) -> None:
):
self._port_links.append(PortLink(sut_port=sut_port, tg_port=tg_port))
+ @classmethod
+ def get_test_cases(
+ cls, test_case_sublist: Sequence[str] | None = None
+ ) -> tuple[set[type["TestCase"]], set[type["TestCase"]]]:
+ """Filter `test_case_subset` from this class.
+
+ Test cases are regular (or bound) methods decorated with :func:`func_test`
+ or :func:`perf_test`.
+
+ Args:
+ test_case_sublist: Test case names to filter from this class.
+ If empty or :data:`None`, return all test cases.
+
+ Returns:
+ The filtered test case functions. This method returns functions as opposed to methods,
+ as methods are bound to instances and this method only has access to the class.
+
+ Raises:
+ ConfigurationError: If a test case from `test_case_subset` is not found.
+ """
+
+ def is_test_case(function: Callable) -> bool:
+ if inspect.isfunction(function):
+ # TestCase is not used at runtime, so we can't use isinstance() with `function`.
+ # But function.test_type exists.
+ if hasattr(function, "test_type"):
+ return isinstance(function.test_type, TestCaseType)
+ return False
+
+ if test_case_sublist is None:
+ test_case_sublist = []
+
+ # the copy is needed so that the condition "elif test_case_sublist" doesn't
+ # change mid-cycle
+ test_case_sublist_copy = list(test_case_sublist)
+ func_test_cases = set()
+ perf_test_cases = set()
+
+ for test_case_name, test_case_function in inspect.getmembers(cls, is_test_case):
+ if test_case_name in test_case_sublist_copy:
+ # if test_case_sublist_copy is non-empty, remove the found test case
+ # so that we can look at the remainder at the end
+ test_case_sublist_copy.remove(test_case_name)
+ elif test_case_sublist:
+ # if the original list is not empty (meaning we're filtering test cases),
+ # we're dealing with a test case we would've
+ # removed in the other branch; since we didn't, we don't want to run it
+ continue
+
+ match test_case_function.test_type:
+ case TestCaseType.PERFORMANCE:
+ perf_test_cases.add(test_case_function)
+ case TestCaseType.FUNCTIONAL:
+ func_test_cases.add(test_case_function)
+
+ if test_case_sublist_copy:
+ raise ConfigurationError(
+ f"Test cases {test_case_sublist_copy} not found among functions of {cls.__name__}."
+ )
+
+ return func_test_cases, perf_test_cases
+
def set_up_suite(self) -> None:
"""Set up test fixtures common to all test cases.
@@ -365,3 +430,59 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool:
if received_packet.src != expected_packet.src or received_packet.dst != expected_packet.dst:
return False
return True
+
+
+#: The generic type for a method of an instance of TestSuite
+TestSuiteMethodType = TypeVar("TestSuiteMethodType", bound=Callable[[TestSuite], None])
+
+
+class TestCaseType(Enum):
+ """The types of test cases."""
+
+ #:
+ FUNCTIONAL = auto()
+ #:
+ PERFORMANCE = auto()
+
+
+class TestCase(Protocol[TestSuiteMethodType]):
+ """Definition of the test case type for static type checking purposes.
+
+ The type is applied to test case functions through a decorator, which casts the decorated
+ test case function to :class:`TestCase` and sets common variables.
+ """
+
+ #:
+ test_type: ClassVar[TestCaseType]
+ #: necessary for mypy so that it can treat this class as the function it's shadowing
+ __call__: TestSuiteMethodType
+
+ @classmethod
+ def make_decorator(
+ cls, test_case_type: TestCaseType
+ ) -> Callable[[TestSuiteMethodType], type["TestCase"]]:
+ """Create a decorator for test suites.
+
+ The decorator casts the decorated function as :class:`TestCase`,
+ sets it as `test_case_type`
+ and initializes common variables defined in :class:`RequiresCapabilities`.
+
+ Args:
+ test_case_type: Either a functional or performance test case.
+
+ Returns:
+ The decorator of a functional or performance test case.
+ """
+
+ def _decorator(func: TestSuiteMethodType) -> type[TestCase]:
+ test_case = cast(type[TestCase], func)
+ test_case.test_type = test_case_type
+ return test_case
+
+ return _decorator
+
+
+#: The decorator for functional test cases.
+func_test: Callable = TestCase.make_decorator(TestCaseType.FUNCTIONAL)
+#: The decorator for performance test cases.
+perf_test: Callable = TestCase.make_decorator(TestCaseType.PERFORMANCE)
diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
index d958f99030..16d064ffeb 100644
--- a/dts/tests/TestSuite_hello_world.py
+++ b/dts/tests/TestSuite_hello_world.py
@@ -8,7 +8,7 @@
"""
from framework.remote_session.dpdk_shell import compute_eal_params
-from framework.test_suite import TestSuite
+from framework.test_suite import TestSuite, func_test
from framework.testbed_model.cpu import (
LogicalCoreCount,
LogicalCoreCountFilter,
@@ -27,7 +27,8 @@ def set_up_suite(self) -> None:
"""
self.app_helloworld_path = self.sut_node.build_dpdk_app("helloworld")
- def test_hello_world_single_core(self) -> None:
+ @func_test
+ def hello_world_single_core(self) -> None:
"""Single core test case.
Steps:
@@ -46,7 +47,8 @@ def test_hello_world_single_core(self) -> None:
f"helloworld didn't start on lcore{lcores[0]}",
)
- def test_hello_world_all_cores(self) -> None:
+ @func_test
+ def hello_world_all_cores(self) -> None:
"""All cores test case.
Steps:
diff --git a/dts/tests/TestSuite_os_udp.py b/dts/tests/TestSuite_os_udp.py
index a78bd74139..beaa5f425d 100644
--- a/dts/tests/TestSuite_os_udp.py
+++ b/dts/tests/TestSuite_os_udp.py
@@ -10,7 +10,7 @@
from scapy.layers.inet import IP, UDP # type: ignore[import-untyped]
from scapy.layers.l2 import Ether # type: ignore[import-untyped]
-from framework.test_suite import TestSuite
+from framework.test_suite import TestSuite, func_test
class TestOsUdp(TestSuite):
@@ -26,6 +26,7 @@ def set_up_suite(self) -> None:
self.sut_node.bind_ports_to_driver(for_dpdk=False)
self.configure_testbed_ipv4()
+ @func_test
def test_os_udp(self) -> None:
"""Basic UDP IPv4 traffic test case.
diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
index 0d8e101e5c..020fb0ab62 100644
--- a/dts/tests/TestSuite_pmd_buffer_scatter.py
+++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
@@ -24,7 +24,7 @@
from framework.params.testpmd import SimpleForwardingModes
from framework.remote_session.testpmd_shell import TestPmdShell
-from framework.test_suite import TestSuite
+from framework.test_suite import TestSuite, func_test
class TestPmdBufferScatter(TestSuite):
@@ -123,6 +123,7 @@ def pmd_scatter(self, mbsize: int) -> None:
f"{offset}.",
)
+ @func_test
def test_scatter_mbuf_2048(self) -> None:
"""Run the :meth:`pmd_scatter` test with `mbsize` set to 2048."""
self.pmd_scatter(mbsize=2048)
diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
index c0b0e6bb00..94f90d9327 100644
--- a/dts/tests/TestSuite_smoke_tests.py
+++ b/dts/tests/TestSuite_smoke_tests.py
@@ -17,7 +17,7 @@
from framework.config import PortConfig
from framework.remote_session.testpmd_shell import TestPmdShell
from framework.settings import SETTINGS
-from framework.test_suite import TestSuite
+from framework.test_suite import TestSuite, func_test
from framework.utils import REGEX_FOR_PCI_ADDRESS
@@ -47,6 +47,7 @@ def set_up_suite(self) -> None:
self.dpdk_build_dir_path = self.sut_node.remote_dpdk_build_dir
self.nics_in_node = self.sut_node.config.ports
+ @func_test
def test_unit_tests(self) -> None:
"""DPDK meson ``fast-tests`` unit tests.
@@ -63,6 +64,7 @@ def test_unit_tests(self) -> None:
privileged=True,
)
+ @func_test
def test_driver_tests(self) -> None:
"""DPDK meson ``driver-tests`` unit tests.
@@ -91,6 +93,7 @@ def test_driver_tests(self) -> None:
privileged=True,
)
+ @func_test
def test_devices_listed_in_testpmd(self) -> None:
"""Testpmd device discovery.
@@ -108,6 +111,7 @@ def test_devices_listed_in_testpmd(self) -> None:
"please check your configuration",
)
+ @func_test
def test_device_bound_to_driver(self) -> None:
"""Device driver in OS.
--
2.34.1
^ permalink raw reply [flat|nested] 75+ messages in thread
* [PATCH v3 04/12] dts: add mechanism to skip test cases or suites
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
` (2 preceding siblings ...)
2024-08-21 14:53 ` [PATCH v3 03/12] dts: add test case decorators Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 16:52 ` Jeremy Spewock
2024-08-28 20:37 ` Dean Marx
2024-08-21 14:53 ` [PATCH v3 05/12] dts: add support for simpler topologies Juraj Linkeš
` (8 subsequent siblings)
12 siblings, 2 replies; 75+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
If a test case is not relevant to the testing environment (such as when
a NIC doesn't support a tested feature), the framework should skip it.
The mechanism is a skeleton without actual logic that would set a test
case or suite to be skipped.
The mechanism uses a protocol to extend test suites and test cases with
additional attributes that track whether the test case or suite should
be skipped the reason for skipping it.
Also update the results module with the new SKIP result.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/runner.py | 34 +++++++---
dts/framework/test_result.py | 77 ++++++++++++++---------
dts/framework/test_suite.py | 7 ++-
dts/framework/testbed_model/capability.py | 28 +++++++++
4 files changed, 109 insertions(+), 37 deletions(-)
create mode 100644 dts/framework/testbed_model/capability.py
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 525f119ab6..55357ea1fe 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -477,7 +477,20 @@ def _run_test_suites(
for test_suite_with_cases in test_suites_with_cases:
test_suite_result = build_target_result.add_test_suite(test_suite_with_cases)
try:
- self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases)
+ if not test_suite_with_cases.skip:
+ self._run_test_suite(
+ sut_node,
+ tg_node,
+ test_suite_result,
+ test_suite_with_cases,
+ )
+ else:
+ self._logger.info(
+ f"Test suite execution SKIPPED: "
+ f"'{test_suite_with_cases.test_suite_class.__name__}'. Reason: "
+ f"{test_suite_with_cases.test_suite_class.skip_reason}"
+ )
+ test_suite_result.update_setup(Result.SKIP)
except BlockingTestSuiteError as e:
self._logger.exception(
f"An error occurred within {test_suite_with_cases.test_suite_class.__name__}. "
@@ -576,14 +589,21 @@ def _execute_test_suite(
test_case_result = test_suite_result.add_test_case(test_case_name)
all_attempts = SETTINGS.re_run + 1
attempt_nr = 1
- self._run_test_case(test_suite, test_case, test_case_result)
- while not test_case_result and attempt_nr < all_attempts:
- attempt_nr += 1
+ if not test_case.skip:
+ self._run_test_case(test_suite, test_case, test_case_result)
+ while not test_case_result and attempt_nr < all_attempts:
+ attempt_nr += 1
+ self._logger.info(
+ f"Re-running FAILED test case '{test_case_name}'. "
+ f"Attempt number {attempt_nr} out of {all_attempts}."
+ )
+ self._run_test_case(test_suite, test_case, test_case_result)
+ else:
self._logger.info(
- f"Re-running FAILED test case '{test_case_name}'. "
- f"Attempt number {attempt_nr} out of {all_attempts}."
+ f"Test case execution SKIPPED: {test_case_name}. Reason: "
+ f"{test_case.skip_reason}"
)
- self._run_test_case(test_suite, test_case, test_case_result)
+ test_case_result.update_setup(Result.SKIP)
def _run_test_case(
self,
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index b1ca584523..306b100bc6 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -75,6 +75,20 @@ def create_config(self) -> TestSuiteConfig:
test_cases=[test_case.__name__ for test_case in self.test_cases],
)
+ @property
+ def skip(self) -> bool:
+ """Skip the test suite if all test cases or the suite itself are to be skipped.
+
+ Returns:
+ :data:`True` if the test suite should be skipped, :data:`False` otherwise.
+ """
+ all_test_cases_skipped = True
+ for test_case in self.test_cases:
+ if not test_case.skip:
+ all_test_cases_skipped = False
+ break
+ return all_test_cases_skipped or self.test_suite_class.skip
+
class Result(Enum):
"""The possible states that a setup, a teardown or a test case may end up in."""
@@ -86,12 +100,12 @@ class Result(Enum):
#:
ERROR = auto()
#:
- SKIP = auto()
- #:
BLOCK = auto()
+ #:
+ SKIP = auto()
def __bool__(self) -> bool:
- """Only PASS is True."""
+ """Only :attr:`PASS` is True."""
return self is self.PASS
@@ -169,12 +183,13 @@ def update_setup(self, result: Result, error: Exception | None = None) -> None:
self.setup_result.result = result
self.setup_result.error = error
- if result in [Result.BLOCK, Result.ERROR, Result.FAIL]:
- self.update_teardown(Result.BLOCK)
- self._block_result()
+ if result != Result.PASS:
+ result_to_mark = Result.BLOCK if result != Result.SKIP else Result.SKIP
+ self.update_teardown(result_to_mark)
+ self._mark_results(result_to_mark)
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed.
+ def _mark_results(self, result) -> None:
+ """Mark the result as well as the child result as `result`.
The blocking of child results should be done in overloaded methods.
"""
@@ -391,11 +406,11 @@ def add_sut_info(self, sut_info: NodeInfo) -> None:
self.sut_os_version = sut_info.os_version
self.sut_kernel_version = sut_info.kernel_version
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ def _mark_results(self, result) -> None:
+ """Mark the result as well as the child result as `result`."""
for build_target in self._config.build_targets:
child_result = self.add_build_target(build_target)
- child_result.update_setup(Result.BLOCK)
+ child_result.update_setup(result)
class BuildTargetResult(BaseResult):
@@ -465,11 +480,11 @@ def add_build_target_info(self, versions: BuildTargetInfo) -> None:
self.compiler_version = versions.compiler_version
self.dpdk_version = versions.dpdk_version
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ def _mark_results(self, result) -> None:
+ """Mark the result as well as the child result as `result`."""
for test_suite_with_cases in self._test_suites_with_cases:
child_result = self.add_test_suite(test_suite_with_cases)
- child_result.update_setup(Result.BLOCK)
+ child_result.update_setup(result)
class TestSuiteResult(BaseResult):
@@ -509,11 +524,11 @@ def add_test_case(self, test_case_name: str) -> "TestCaseResult":
self.child_results.append(result)
return result
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ def _mark_results(self, result) -> None:
+ """Mark the result as well as the child result as `result`."""
for test_case_method in self._test_suite_with_cases.test_cases:
child_result = self.add_test_case(test_case_method.__name__)
- child_result.update_setup(Result.BLOCK)
+ child_result.update_setup(result)
class TestCaseResult(BaseResult, FixtureResult):
@@ -567,9 +582,9 @@ def add_stats(self, statistics: "Statistics") -> None:
"""
statistics += self.result
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
- self.update(Result.BLOCK)
+ def _mark_results(self, result) -> None:
+ r"""Mark the result as `result`."""
+ self.update(result)
def __bool__(self) -> bool:
"""The test case passed only if setup, teardown and the test case itself passed."""
@@ -583,7 +598,8 @@ class Statistics(dict):
The data are stored in the following keys:
- * **PASS RATE** (:class:`int`) -- The FAIL/PASS ratio of all test cases.
+ * **PASS RATE** (:class:`int`) -- The :attr:`~Result.FAIL`/:attr:`~Result.PASS` ratio
+ of all test cases.
* **DPDK VERSION** (:class:`str`) -- The tested DPDK version.
"""
@@ -600,22 +616,27 @@ def __init__(self, dpdk_version: str | None):
self["DPDK VERSION"] = dpdk_version
def __iadd__(self, other: Result) -> "Statistics":
- """Add a Result to the final count.
+ """Add a :class:`Result` to the final count.
+
+ :attr:`~Result.SKIP` is not taken into account
Example:
- stats: Statistics = Statistics() # empty Statistics
- stats += Result.PASS # add a Result to `stats`
+ stats: Statistics = Statistics() # empty :class:`Statistics`
+ stats += Result.PASS # add a :class:`Result` to `stats`
Args:
- other: The Result to add to this statistics object.
+ other: The :class:`Result` to add to this statistics object.
Returns:
The modified statistics object.
"""
self[other.name] += 1
- self["PASS RATE"] = (
- float(self[Result.PASS.name]) * 100 / sum(self[result.name] for result in Result)
- )
+ if other != Result.SKIP:
+ self["PASS RATE"] = (
+ float(self[Result.PASS.name])
+ * 100
+ / sum([self[result.name] for result in Result if result != Result.SKIP])
+ )
return self
def __str__(self) -> str:
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index b4ee0f9039..c59fc9c6e6 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -23,6 +23,7 @@
from scapy.layers.l2 import Ether # type: ignore[import-untyped]
from scapy.packet import Packet, Padding # type: ignore[import-untyped]
+from framework.testbed_model.capability import TestProtocol
from framework.testbed_model.port import Port, PortLink
from framework.testbed_model.sut_node import SutNode
from framework.testbed_model.tg_node import TGNode
@@ -35,7 +36,7 @@
from .utils import get_packet_summaries
-class TestSuite:
+class TestSuite(TestProtocol):
"""The base class with building blocks needed by most test cases.
* Test suite setup/cleanup methods to override,
@@ -445,7 +446,7 @@ class TestCaseType(Enum):
PERFORMANCE = auto()
-class TestCase(Protocol[TestSuiteMethodType]):
+class TestCase(TestProtocol, Protocol[TestSuiteMethodType]):
"""Definition of the test case type for static type checking purposes.
The type is applied to test case functions through a decorator, which casts the decorated
@@ -476,6 +477,8 @@ def make_decorator(
def _decorator(func: TestSuiteMethodType) -> type[TestCase]:
test_case = cast(type[TestCase], func)
+ test_case.skip = cls.skip
+ test_case.skip_reason = cls.skip_reason
test_case.test_type = test_case_type
return test_case
diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py
new file mode 100644
index 0000000000..662f691a0e
--- /dev/null
+++ b/dts/framework/testbed_model/capability.py
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 PANTHEON.tech s.r.o.
+
+"""Testbed capabilities.
+
+This module provides a protocol that defines the common attributes of test cases and suites.
+"""
+
+from collections.abc import Sequence
+from typing import ClassVar, Protocol
+
+
+class TestProtocol(Protocol):
+ """Common test suite and test case attributes."""
+
+ #: Whether to skip the test case or suite.
+ skip: ClassVar[bool] = False
+ #: The reason for skipping the test case or suite.
+ skip_reason: ClassVar[str] = ""
+
+ @classmethod
+ def get_test_cases(cls, test_case_sublist: Sequence[str] | None = None) -> tuple[set, set]:
+ """Get test cases. Should be implemented by subclasses containing test cases.
+
+ Raises:
+ NotImplementedError: The subclass does not implement the method.
+ """
+ raise NotImplementedError()
--
2.34.1
^ permalink raw reply [flat|nested] 75+ messages in thread
* [PATCH v3 05/12] dts: add support for simpler topologies
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
` (3 preceding siblings ...)
2024-08-21 14:53 ` [PATCH v3 04/12] dts: add mechanism to skip test cases or suites Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 16:54 ` Jeremy Spewock
2024-08-28 20:56 ` Dean Marx
2024-08-21 14:53 ` [PATCH v3 06/12] dst: add basic capability support Juraj Linkeš
` (7 subsequent siblings)
12 siblings, 2 replies; 75+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
We currently assume there are two links between the SUT and TG nodes,
but that's too strict, even for some of the already existing test cases.
Add support for topologies with less than two links.
For topologies with no links, dummy ports are used. The expectation is
that test suites or cases that don't require any links won't be using
methods that use ports. Any test suites or cases requiring links will be
skipped in topologies with no links, but this feature is not implemented
in this commit.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/runner.py | 6 +-
dts/framework/test_suite.py | 32 +++----
dts/framework/testbed_model/node.py | 2 +-
dts/framework/testbed_model/port.py | 4 +-
dts/framework/testbed_model/topology.py | 101 ++++++++++++++++++++++
dts/tests/TestSuite_pmd_buffer_scatter.py | 2 +-
6 files changed, 120 insertions(+), 27 deletions(-)
create mode 100644 dts/framework/testbed_model/topology.py
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 55357ea1fe..48ae9cc215 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -53,6 +53,7 @@
TestSuiteWithCases,
)
from .test_suite import TestCase, TestSuite
+from .testbed_model.topology import Topology
class DTSRunner:
@@ -474,6 +475,7 @@ def _run_test_suites(
test_suites_with_cases: The test suites with test cases to run.
"""
end_build_target = False
+ topology = Topology(sut_node.ports, tg_node.ports)
for test_suite_with_cases in test_suites_with_cases:
test_suite_result = build_target_result.add_test_suite(test_suite_with_cases)
try:
@@ -481,6 +483,7 @@ def _run_test_suites(
self._run_test_suite(
sut_node,
tg_node,
+ topology,
test_suite_result,
test_suite_with_cases,
)
@@ -506,6 +509,7 @@ def _run_test_suite(
self,
sut_node: SutNode,
tg_node: TGNode,
+ topology: Topology,
test_suite_result: TestSuiteResult,
test_suite_with_cases: TestSuiteWithCases,
) -> None:
@@ -533,7 +537,7 @@ def _run_test_suite(
self._logger.set_stage(
DtsStage.test_suite_setup, Path(SETTINGS.output_dir, test_suite_name)
)
- test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node)
+ test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node, topology)
try:
self._logger.info(f"Starting test suite setup: {test_suite_name}")
test_suite.set_up_suite()
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index c59fc9c6e6..56f153bda6 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -24,9 +24,10 @@
from scapy.packet import Packet, Padding # type: ignore[import-untyped]
from framework.testbed_model.capability import TestProtocol
-from framework.testbed_model.port import Port, PortLink
+from framework.testbed_model.port import Port
from framework.testbed_model.sut_node import SutNode
from framework.testbed_model.tg_node import TGNode
+from framework.testbed_model.topology import Topology, TopologyType
from framework.testbed_model.traffic_generator.capturing_traffic_generator import (
PacketFilteringConfig,
)
@@ -72,7 +73,7 @@ class TestSuite(TestProtocol):
#: will block the execution of all subsequent test suites in the current build target.
is_blocking: ClassVar[bool] = False
_logger: DTSLogger
- _port_links: list[PortLink]
+ _topology_type: TopologyType
_sut_port_ingress: Port
_sut_port_egress: Port
_sut_ip_address_ingress: Union[IPv4Interface, IPv6Interface]
@@ -86,6 +87,7 @@ def __init__(
self,
sut_node: SutNode,
tg_node: TGNode,
+ topology: Topology,
):
"""Initialize the test suite testbed information and basic configuration.
@@ -95,35 +97,21 @@ def __init__(
Args:
sut_node: The SUT node where the test suite will run.
tg_node: The TG node where the test suite will run.
+ topology: The topology where the test suite will run.
"""
self.sut_node = sut_node
self.tg_node = tg_node
self._logger = get_dts_logger(self.__class__.__name__)
- self._port_links = []
- self._process_links()
- self._sut_port_ingress, self._tg_port_egress = (
- self._port_links[0].sut_port,
- self._port_links[0].tg_port,
- )
- self._sut_port_egress, self._tg_port_ingress = (
- self._port_links[1].sut_port,
- self._port_links[1].tg_port,
- )
+ self._topology_type = topology.type
+ self._tg_port_egress = topology.tg_port_egress
+ self._sut_port_ingress = topology.sut_port_ingress
+ self._sut_port_egress = topology.sut_port_egress
+ self._tg_port_ingress = topology.tg_port_ingress
self._sut_ip_address_ingress = ip_interface("192.168.100.2/24")
self._sut_ip_address_egress = ip_interface("192.168.101.2/24")
self._tg_ip_address_egress = ip_interface("192.168.100.3/24")
self._tg_ip_address_ingress = ip_interface("192.168.101.3/24")
- def _process_links(self) -> None:
- """Construct links between SUT and TG ports."""
- for sut_port in self.sut_node.ports:
- for tg_port in self.tg_node.ports:
- if (sut_port.identifier, sut_port.peer) == (
- tg_port.peer,
- tg_port.identifier,
- ):
- self._port_links.append(PortLink(sut_port=sut_port, tg_port=tg_port))
-
@classmethod
def get_test_cases(
cls, test_case_sublist: Sequence[str] | None = None
diff --git a/dts/framework/testbed_model/node.py b/dts/framework/testbed_model/node.py
index 12a40170ac..51443cd71f 100644
--- a/dts/framework/testbed_model/node.py
+++ b/dts/framework/testbed_model/node.py
@@ -90,7 +90,7 @@ def __init__(self, node_config: NodeConfiguration):
self._init_ports()
def _init_ports(self) -> None:
- self.ports = [Port(self.name, port_config) for port_config in self.config.ports]
+ self.ports = [Port(port_config) for port_config in self.config.ports]
self.main_session.update_ports(self.ports)
for port in self.ports:
self.configure_port_state(port)
diff --git a/dts/framework/testbed_model/port.py b/dts/framework/testbed_model/port.py
index 817405bea4..82c84cf4f8 100644
--- a/dts/framework/testbed_model/port.py
+++ b/dts/framework/testbed_model/port.py
@@ -54,7 +54,7 @@ class Port:
mac_address: str = ""
logical_name: str = ""
- def __init__(self, node_name: str, config: PortConfig):
+ def __init__(self, config: PortConfig):
"""Initialize the port from `node_name` and `config`.
Args:
@@ -62,7 +62,7 @@ def __init__(self, node_name: str, config: PortConfig):
config: The test run configuration of the port.
"""
self.identifier = PortIdentifier(
- node=node_name,
+ node=config.node,
pci=config.pci,
)
self.os_driver = config.os_driver
diff --git a/dts/framework/testbed_model/topology.py b/dts/framework/testbed_model/topology.py
new file mode 100644
index 0000000000..19632ee890
--- /dev/null
+++ b/dts/framework/testbed_model/topology.py
@@ -0,0 +1,101 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 PANTHEON.tech s.r.o.
+
+"""Testbed topology representation.
+
+A topology of a testbed captures what links are available between the testbed's nodes.
+The link information then implies what type of topology is available.
+"""
+
+from dataclasses import dataclass
+from enum import IntEnum
+from typing import Iterable
+
+from framework.config import PortConfig
+
+from .port import Port
+
+
+class TopologyType(IntEnum):
+ """Supported topology types."""
+
+ #: A topology with no Traffic Generator.
+ no_link = 0
+ #: A topology with one physical link between the SUT node and the TG node.
+ one_link = 1
+ #: A topology with two physical links between the Sut node and the TG node.
+ two_links = 2
+
+
+class Topology:
+ """Testbed topology.
+
+ The topology contains ports processed into ingress and egress ports.
+ It's assumed that port0 of the SUT node is connected to port0 of the TG node and so on.
+ If there are no ports on a node, dummy ports (ports with no actual values) are stored.
+ If there is only one link available, the ports of this link are stored
+ as both ingress and egress ports.
+
+ The dummy ports shouldn't be used. It's up to :class:`~framework.runner.DTSRunner`
+ to ensure no test case or suite requiring actual links is executed
+ when the topology prohibits it and up to the developers to make sure that test cases
+ not requiring any links don't use any ports. Otherwise, the underlying methods
+ using the ports will fail.
+
+ Attributes:
+ type: The type of the topology.
+ tg_port_egress: The egress port of the TG node.
+ sut_port_ingress: The ingress port of the SUT node.
+ sut_port_egress: The egress port of the SUT node.
+ tg_port_ingress: The ingress port of the TG node.
+ """
+
+ type: TopologyType
+ tg_port_egress: Port
+ sut_port_ingress: Port
+ sut_port_egress: Port
+ tg_port_ingress: Port
+
+ def __init__(self, sut_ports: Iterable[Port], tg_ports: Iterable[Port]):
+ """Create the topology from `sut_ports` and `tg_ports`.
+
+ Args:
+ sut_ports: The SUT node's ports.
+ tg_ports: The TG node's ports.
+ """
+ port_links = []
+ for sut_port in sut_ports:
+ for tg_port in tg_ports:
+ if (sut_port.identifier, sut_port.peer) == (
+ tg_port.peer,
+ tg_port.identifier,
+ ):
+ port_links.append(PortLink(sut_port=sut_port, tg_port=tg_port))
+
+ self.type = TopologyType(len(port_links))
+ dummy_port = Port(PortConfig("", "", "", "", "", ""))
+ self.tg_port_egress = dummy_port
+ self.sut_port_ingress = dummy_port
+ self.sut_port_egress = dummy_port
+ self.tg_port_ingress = dummy_port
+ if self.type > TopologyType.no_link:
+ self.tg_port_egress = port_links[0].tg_port
+ self.sut_port_ingress = port_links[0].sut_port
+ self.sut_port_egress = self.sut_port_ingress
+ self.tg_port_ingress = self.tg_port_egress
+ if self.type > TopologyType.one_link:
+ self.sut_port_egress = port_links[1].sut_port
+ self.tg_port_ingress = port_links[1].tg_port
+
+
+@dataclass(slots=True, frozen=True)
+class PortLink:
+ """The physical, cabled connection between the ports.
+
+ Attributes:
+ sut_port: The port on the SUT node connected to `tg_port`.
+ tg_port: The port on the TG node connected to `sut_port`.
+ """
+
+ sut_port: Port
+ tg_port: Port
diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
index 020fb0ab62..178a40385e 100644
--- a/dts/tests/TestSuite_pmd_buffer_scatter.py
+++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
@@ -58,7 +58,7 @@ def set_up_suite(self) -> None:
to support larger packet sizes.
"""
self.verify(
- len(self._port_links) > 1,
+ self._topology_type > 1,
"There must be at least two port links to run the scatter test suite",
)
--
2.34.1
^ permalink raw reply [flat|nested] 75+ messages in thread
* [PATCH v3 06/12] dst: add basic capability support
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
` (4 preceding siblings ...)
2024-08-21 14:53 ` [PATCH v3 05/12] dts: add support for simpler topologies Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 16:56 ` Jeremy Spewock
2024-09-03 16:03 ` Dean Marx
2024-08-21 14:53 ` [PATCH v3 07/12] dts: add testpmd port information caching Juraj Linkeš
` (6 subsequent siblings)
12 siblings, 2 replies; 75+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
A test case or suite may require certain capabilities to be present in
the tested environment. Add the basic infrastructure for checking the
support status of capabilities:
* The Capability ABC defining the common capability API
* Extension of the TestProtocol with required capabilities (each test
suite or case stores the capabilities it requires)
* Integration with the runner which calls the new APIs to get which
capabilities are supported.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/runner.py | 26 +++++
dts/framework/test_result.py | 38 ++++++-
dts/framework/test_suite.py | 1 +
dts/framework/testbed_model/capability.py | 117 +++++++++++++++++++++-
4 files changed, 179 insertions(+), 3 deletions(-)
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 48ae9cc215..43bb2bc830 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -25,6 +25,7 @@
from types import MethodType
from typing import Iterable
+from framework.testbed_model.capability import Capability, get_supported_capabilities
from framework.testbed_model.sut_node import SutNode
from framework.testbed_model.tg_node import TGNode
@@ -452,6 +453,21 @@ def _run_build_target(
self._logger.exception("Build target teardown failed.")
build_target_result.update_teardown(Result.FAIL, e)
+ def _get_supported_capabilities(
+ self,
+ sut_node: SutNode,
+ topology_config: Topology,
+ test_suites_with_cases: Iterable[TestSuiteWithCases],
+ ) -> set[Capability]:
+
+ capabilities_to_check = set()
+ for test_suite_with_cases in test_suites_with_cases:
+ capabilities_to_check.update(test_suite_with_cases.required_capabilities)
+
+ self._logger.debug(f"Found capabilities to check: {capabilities_to_check}")
+
+ return get_supported_capabilities(sut_node, topology_config, capabilities_to_check)
+
def _run_test_suites(
self,
sut_node: SutNode,
@@ -464,6 +480,12 @@ def _run_test_suites(
The method assumes the build target we're testing has already been built on the SUT node.
The current build target thus corresponds to the current DPDK build present on the SUT node.
+ Before running any suites, the method determines whether they should be skipped
+ by inspecting any required capabilities the test suite needs and comparing those
+ to capabilities supported by the tested environment. If all capabilities are supported,
+ the suite is run. If all test cases in a test suite would be skipped, the whole test suite
+ is skipped (the setup and teardown is not run).
+
If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites
in the current build target won't be executed.
@@ -476,7 +498,11 @@ def _run_test_suites(
"""
end_build_target = False
topology = Topology(sut_node.ports, tg_node.ports)
+ supported_capabilities = self._get_supported_capabilities(
+ sut_node, topology, test_suites_with_cases
+ )
for test_suite_with_cases in test_suites_with_cases:
+ test_suite_with_cases.mark_skip_unsupported(supported_capabilities)
test_suite_result = build_target_result.add_test_suite(test_suite_with_cases)
try:
if not test_suite_with_cases.skip:
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 306b100bc6..b4b58ef348 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -25,10 +25,12 @@
import os.path
from collections.abc import MutableSequence
-from dataclasses import dataclass
+from dataclasses import dataclass, field
from enum import Enum, auto
from typing import Union
+from framework.testbed_model.capability import Capability
+
from .config import (
OS,
Architecture,
@@ -63,6 +65,12 @@ class is to hold a subset of test cases (which could be all test cases) because
test_suite_class: type[TestSuite]
test_cases: list[type[TestCase]]
+ required_capabilities: set[Capability] = field(default_factory=set, init=False)
+
+ def __post_init__(self):
+ """Gather the required capabilities of the test suite and all test cases."""
+ for test_object in [self.test_suite_class] + self.test_cases:
+ self.required_capabilities.update(test_object.required_capabilities)
def create_config(self) -> TestSuiteConfig:
"""Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
@@ -75,6 +83,34 @@ def create_config(self) -> TestSuiteConfig:
test_cases=[test_case.__name__ for test_case in self.test_cases],
)
+ def mark_skip_unsupported(self, supported_capabilities: set[Capability]) -> None:
+ """Mark the test suite and test cases to be skipped.
+
+ The mark is applied if object to be skipped requires any capabilities and at least one of
+ them is not among `supported_capabilities`.
+
+ Args:
+ supported_capabilities: The supported capabilities.
+ """
+ for test_object in [self.test_suite_class, *self.test_cases]:
+ capabilities_not_supported = test_object.required_capabilities - supported_capabilities
+ if capabilities_not_supported:
+ test_object.skip = True
+ capability_str = (
+ "capability" if len(capabilities_not_supported) == 1 else "capabilities"
+ )
+ test_object.skip_reason = (
+ f"Required {capability_str} '{capabilities_not_supported}' not found."
+ )
+ if not self.test_suite_class.skip:
+ if all(test_case.skip for test_case in self.test_cases):
+ self.test_suite_class.skip = True
+
+ self.test_suite_class.skip_reason = (
+ "All test cases are marked to be skipped with reasons: "
+ f"{' '.join(test_case.skip_reason for test_case in self.test_cases)}"
+ )
+
@property
def skip(self) -> bool:
"""Skip the test suite if all test cases or the suite itself are to be skipped.
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 56f153bda6..5c393ce8bf 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -467,6 +467,7 @@ def _decorator(func: TestSuiteMethodType) -> type[TestCase]:
test_case = cast(type[TestCase], func)
test_case.skip = cls.skip
test_case.skip_reason = cls.skip_reason
+ test_case.required_capabilities = set()
test_case.test_type = test_case_type
return test_case
diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py
index 662f691a0e..8899f07f76 100644
--- a/dts/framework/testbed_model/capability.py
+++ b/dts/framework/testbed_model/capability.py
@@ -3,11 +3,97 @@
"""Testbed capabilities.
-This module provides a protocol that defines the common attributes of test cases and suites.
+This module provides a protocol that defines the common attributes of test cases and suites
+and support for test environment capabilities.
"""
+from abc import ABC, abstractmethod
from collections.abc import Sequence
-from typing import ClassVar, Protocol
+from typing import Callable, ClassVar, Protocol
+
+from typing_extensions import Self
+
+from .sut_node import SutNode
+from .topology import Topology
+
+
+class Capability(ABC):
+ """The base class for various capabilities.
+
+ The same capability should always be represented by the same object,
+ meaning the same capability required by different test cases or suites
+ should point to the same object.
+
+ Example:
+ ``test_case1`` and ``test_case2`` each require ``capability1``
+ and in both instances, ``capability1`` should point to the same capability object.
+
+ It is up to the subclasses how they implement this.
+
+ The instances are used in sets so they must be hashable.
+ """
+
+ #: A set storing the capabilities whose support should be checked.
+ capabilities_to_check: ClassVar[set[Self]] = set()
+
+ def register_to_check(self) -> Callable[[SutNode, "Topology"], set[Self]]:
+ """Register the capability to be checked for support.
+
+ Returns:
+ The callback function that checks the support of capabilities of the particular subclass
+ which should be called after all capabilities have been registered.
+ """
+ if not type(self).capabilities_to_check:
+ type(self).capabilities_to_check = set()
+ type(self).capabilities_to_check.add(self)
+ return type(self)._get_and_reset
+
+ def add_to_required(self, test_case_or_suite: type["TestProtocol"]) -> None:
+ """Add the capability instance to the required test case or suite's capabilities.
+
+ Args:
+ test_case_or_suite: The test case or suite among whose required capabilities
+ to add this instance.
+ """
+ if not test_case_or_suite.required_capabilities:
+ test_case_or_suite.required_capabilities = set()
+ self._preprocess_required(test_case_or_suite)
+ test_case_or_suite.required_capabilities.add(self)
+
+ def _preprocess_required(self, test_case_or_suite: type["TestProtocol"]) -> None:
+ """An optional method that modifies the required capabilities."""
+
+ @classmethod
+ def _get_and_reset(cls, sut_node: SutNode, topology: "Topology") -> set[Self]:
+ """The callback method to be called after all capabilities have been registered.
+
+ Not only does this method check the support of capabilities,
+ but it also reset the internal set of registered capabilities
+ so that the "register, then get support" workflow works in subsequent test runs.
+ """
+ supported_capabilities = cls.get_supported_capabilities(sut_node, topology)
+ cls.capabilities_to_check = set()
+ return supported_capabilities
+
+ @classmethod
+ @abstractmethod
+ def get_supported_capabilities(cls, sut_node: SutNode, topology: "Topology") -> set[Self]:
+ """Get the support status of each registered capability.
+
+ Each subclass must implement this method and return the subset of supported capabilities
+ of :attr:`capabilities_to_check`.
+
+ Args:
+ sut_node: The SUT node of the current test run.
+ topology: The topology of the current test run.
+
+ Returns:
+ The supported capabilities.
+ """
+
+ @abstractmethod
+ def __hash__(self) -> int:
+ """The subclasses must be hashable so that they can be stored in sets."""
class TestProtocol(Protocol):
@@ -17,6 +103,8 @@ class TestProtocol(Protocol):
skip: ClassVar[bool] = False
#: The reason for skipping the test case or suite.
skip_reason: ClassVar[str] = ""
+ #: The capabilities the test case or suite requires in order to be executed.
+ required_capabilities: ClassVar[set[Capability]] = set()
@classmethod
def get_test_cases(cls, test_case_sublist: Sequence[str] | None = None) -> tuple[set, set]:
@@ -26,3 +114,28 @@ def get_test_cases(cls, test_case_sublist: Sequence[str] | None = None) -> tuple
NotImplementedError: The subclass does not implement the method.
"""
raise NotImplementedError()
+
+
+def get_supported_capabilities(
+ sut_node: SutNode,
+ topology_config: Topology,
+ capabilities_to_check: set[Capability],
+) -> set[Capability]:
+ """Probe the environment for `capabilities_to_check` and return the supported ones.
+
+ Args:
+ sut_node: The SUT node to check for capabilities.
+ topology_config: The topology config to check for capabilities.
+ capabilities_to_check: The capabilities to check.
+
+ Returns:
+ The capabilities supported by the environment.
+ """
+ callbacks = set()
+ for capability_to_check in capabilities_to_check:
+ callbacks.add(capability_to_check.register_to_check())
+ supported_capabilities = set()
+ for callback in callbacks:
+ supported_capabilities.update(callback(sut_node, topology_config))
+
+ return supported_capabilities
--
2.34.1
^ permalink raw reply [flat|nested] 75+ messages in thread
* [PATCH v3 07/12] dts: add testpmd port information caching
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
` (5 preceding siblings ...)
2024-08-21 14:53 ` [PATCH v3 06/12] dst: add basic capability support Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 16:56 ` Jeremy Spewock
2024-09-03 16:12 ` Dean Marx
2024-08-21 14:53 ` [PATCH v3 08/12] dts: add NIC capability support Juraj Linkeš
` (5 subsequent siblings)
12 siblings, 2 replies; 75+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
When using port information multiple times in a testpmd shell instance
lifespan, it's desirable to not get the information each time, so
caching is added. In case the information changes, there's a way to
force the update.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/remote_session/testpmd_shell.py | 30 +++++++++++++++++--
1 file changed, 28 insertions(+), 2 deletions(-)
diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index b4ad253020..f0bcc918e5 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -654,6 +654,7 @@ class TestPmdShell(DPDKShell):
"""
_app_params: TestPmdParams
+ _ports: list[TestPmdPort] | None
#: The path to the testpmd executable.
path: ClassVar[PurePath] = PurePath("app", "dpdk-testpmd")
@@ -686,6 +687,21 @@ def __init__(
TestPmdParams(**app_params),
name,
)
+ self._ports = None
+
+ @property
+ def ports(self) -> list[TestPmdPort]:
+ """The ports of the instance.
+
+ This caches the ports returned by :meth:`show_port_info_all`.
+ To force an update of port information, execute :meth:`show_port_info_all` or
+ :meth:`show_port_info`.
+
+ Returns: The list of known testpmd ports.
+ """
+ if self._ports is None:
+ return self.show_port_info_all()
+ return self._ports
def start(self, verify: bool = True) -> None:
"""Start packet forwarding with the current configuration.
@@ -872,7 +888,8 @@ def show_port_info_all(self) -> list[TestPmdPort]:
# executed on a pseudo-terminal created by paramiko on the remote node, lines end with CRLF.
# Therefore we also need to take the carriage return into account.
iter = re.finditer(r"\*{21}.*?[\r\n]{4}", output + "\r\n", re.S)
- return [TestPmdPort.parse(block.group(0)) for block in iter]
+ self._ports = [TestPmdPort.parse(block.group(0)) for block in iter]
+ return self._ports
def show_port_info(self, port_id: int) -> TestPmdPort:
"""Returns the given port information.
@@ -890,7 +907,16 @@ def show_port_info(self, port_id: int) -> TestPmdPort:
if output.startswith("Invalid port"):
raise InteractiveCommandExecutionError("invalid port given")
- return TestPmdPort.parse(output)
+ port = TestPmdPort.parse(output)
+ self._update_port(port)
+ return port
+
+ def _update_port(self, port: TestPmdPort) -> None:
+ if self._ports:
+ self._ports = [
+ existing_port if port.id != existing_port.id else port
+ for existing_port in self._ports
+ ]
def show_port_stats_all(self) -> list[TestPmdPortStats]:
"""Returns the statistics of all the ports.
--
2.34.1
^ permalink raw reply [flat|nested] 75+ messages in thread
* [PATCH v3 08/12] dts: add NIC capability support
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
` (6 preceding siblings ...)
2024-08-21 14:53 ` [PATCH v3 07/12] dts: add testpmd port information caching Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 17:11 ` Jeremy Spewock
` (2 more replies)
2024-08-21 14:53 ` [PATCH v3 09/12] dts: add topology capability Juraj Linkeš
` (4 subsequent siblings)
12 siblings, 3 replies; 75+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
Some test cases or suites may be testing a NIC feature that is not
supported on all NICs, so add support for marking test cases or suites
as requiring NIC capabilities.
The marking is done with a decorator, which populates the internal
required_capabilities attribute of TestProtocol. The NIC capability
itself is a wrapper around the NicCapability defined in testpmd_shell.
The reason is twofold:
1. Enums cannot be extended and the class implements the methods of its
abstract base superclass,
2. The class also stores an optional decorator function which is used
before/after capability retrieval. This is needed because some
capabilities may be advertised differently under different
configuration.
The decorator API is designed to be simple to use. The arguments passed
to it are all from the testpmd shell. Everything else (even the actual
capability object creation) is done internally.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
Depends-on: patch-142276 ("dts: add methods for modifying MTU to testpmd
shell")
---
dts/framework/remote_session/testpmd_shell.py | 178 ++++++++++++++++-
dts/framework/testbed_model/capability.py | 180 +++++++++++++++++-
dts/tests/TestSuite_pmd_buffer_scatter.py | 2 +
3 files changed, 356 insertions(+), 4 deletions(-)
diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index f0bcc918e5..48c31124d1 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -16,11 +16,17 @@
import re
import time
-from collections.abc import Callable
+from collections.abc import Callable, MutableSet
from dataclasses import dataclass, field
from enum import Flag, auto
+from functools import partial
from pathlib import PurePath
-from typing import ClassVar
+from typing import TYPE_CHECKING, Any, ClassVar, TypeAlias
+
+if TYPE_CHECKING:
+ from enum import Enum as NoAliasEnum
+else:
+ from aenum import NoAliasEnum
from typing_extensions import Self, TypeVarTuple, Unpack
@@ -34,6 +40,16 @@
from framework.testbed_model.sut_node import SutNode
from framework.utils import StrEnum
+TestPmdShellCapabilityMethod: TypeAlias = Callable[
+ ["TestPmdShell", MutableSet["NicCapability"], MutableSet["NicCapability"]], None
+]
+
+TestPmdShellSimpleMethod: TypeAlias = Callable[["TestPmdShell"], Any]
+
+TestPmdShellDecoratedMethod: TypeAlias = Callable[["TestPmdShell"], None]
+
+TestPmdShellDecorator: TypeAlias = Callable[[TestPmdShellSimpleMethod], TestPmdShellDecoratedMethod]
+
class TestPmdDevice:
"""The data of a device that testpmd can recognize.
@@ -377,6 +393,71 @@ def _validate(info: str):
return TextParser.wrap(TextParser.find(r"Device private info:\s+([\s\S]+)"), _validate)
+class RxQueueState(StrEnum):
+ """RX queue states."""
+
+ #:
+ stopped = auto()
+ #:
+ started = auto()
+ #:
+ hairpin = auto()
+ #:
+ unknown = auto()
+
+ @classmethod
+ def make_parser(cls) -> ParserFn:
+ """Makes a parser function.
+
+ Returns:
+ ParserFn: A dictionary for the `dataclasses.field` metadata argument containing a
+ parser function that makes an instance of this enum from text.
+ """
+ return TextParser.wrap(TextParser.find(r"Rx queue state: ([^\r\n]+)"), cls)
+
+
+@dataclass
+class TestPmdRxqInfo(TextParser):
+ """Representation of testpmd's ``show rxq info <port_id> <queue_id>`` command."""
+
+ #:
+ port_id: int = field(metadata=TextParser.find_int(r"Infos for port (\d+)\b ?, RX queue \d+\b"))
+ #:
+ queue_id: int = field(metadata=TextParser.find_int(r"Infos for port \d+\b ?, RX queue (\d+)\b"))
+ #: Mempool used by that queue
+ mempool: str = field(metadata=TextParser.find(r"Mempool: ([^\r\n]+)"))
+ #: Ring prefetch threshold
+ rx_prefetch_threshold: int = field(
+ metadata=TextParser.find_int(r"RX prefetch threshold: (\d+)\b")
+ )
+ #: Ring host threshold
+ rx_host_threshold: int = field(metadata=TextParser.find_int(r"RX host threshold: (\d+)\b"))
+ #: Ring writeback threshold
+ rx_writeback_threshold: int = field(
+ metadata=TextParser.find_int(r"RX writeback threshold: (\d+)\b")
+ )
+ #: Drives the freeing of Rx descriptors
+ rx_free_threshold: int = field(metadata=TextParser.find_int(r"RX free threshold: (\d+)\b"))
+ #: Drop packets if no descriptors are available
+ rx_drop_packets: bool = field(metadata=TextParser.find(r"RX drop packets: on"))
+ #: Do not start queue with rte_eth_dev_start()
+ rx_deferred_start: bool = field(metadata=TextParser.find(r"RX deferred start: on"))
+ #: Scattered packets Rx enabled
+ rx_scattered_packets: bool = field(metadata=TextParser.find(r"RX scattered packets: on"))
+ #: The state of the queue
+ rx_queue_state: str = field(metadata=RxQueueState.make_parser())
+ #: Configured number of RXDs
+ number_of_rxds: int = field(metadata=TextParser.find_int(r"Number of RXDs: (\d+)\b"))
+ #: Hardware receive buffer size
+ rx_buffer_size: int | None = field(
+ default=None, metadata=TextParser.find_int(r"RX buffer size: (\d+)\b")
+ )
+ #: Burst mode information
+ burst_mode: str | None = field(
+ default=None, metadata=TextParser.find(r"Burst mode: ([^\r\n]+)")
+ )
+
+
@dataclass
class TestPmdPort(TextParser):
"""Dataclass representing the result of testpmd's ``show port info`` command."""
@@ -962,3 +1043,96 @@ def _close(self) -> None:
self.stop()
self.send_command("quit", "Bye...")
return super()._close()
+
+ """
+ ====== Capability retrieval methods ======
+ """
+
+ def get_capabilities_rxq_info(
+ self,
+ supported_capabilities: MutableSet["NicCapability"],
+ unsupported_capabilities: MutableSet["NicCapability"],
+ ) -> None:
+ """Get all rxq capabilities and divide them into supported and unsupported.
+
+ Args:
+ supported_capabilities: Supported capabilities will be added to this set.
+ unsupported_capabilities: Unsupported capabilities will be added to this set.
+ """
+ self._logger.debug("Getting rxq capabilities.")
+ command = f"show rxq info {self.ports[0].id} 0"
+ rxq_info = TestPmdRxqInfo.parse(self.send_command(command))
+ if rxq_info.rx_scattered_packets:
+ supported_capabilities.add(NicCapability.SCATTERED_RX_ENABLED)
+ else:
+ unsupported_capabilities.add(NicCapability.SCATTERED_RX_ENABLED)
+
+ """
+ ====== Decorator methods ======
+ """
+
+ @staticmethod
+ def config_mtu_9000(testpmd_method: TestPmdShellSimpleMethod) -> TestPmdShellDecoratedMethod:
+ """Configure MTU to 9000 on all ports, run `testpmd_method`, then revert.
+
+ Args:
+ testpmd_method: The method to decorate.
+
+ Returns:
+ The method decorated with setting and reverting MTU.
+ """
+
+ def wrapper(testpmd_shell: Self):
+ original_mtus = []
+ for port in testpmd_shell.ports:
+ original_mtus.append((port.id, port.mtu))
+ testpmd_shell.set_port_mtu(port_id=port.id, mtu=9000, verify=False)
+ testpmd_method(testpmd_shell)
+ for port_id, mtu in original_mtus:
+ testpmd_shell.set_port_mtu(port_id=port_id, mtu=mtu if mtu else 1500, verify=False)
+
+ return wrapper
+
+
+class NicCapability(NoAliasEnum):
+ """A mapping between capability names and the associated :class:`TestPmdShell` methods.
+
+ The :class:`TestPmdShell` method executes the command that checks
+ whether the capability is supported.
+
+ The signature of each :class:`TestPmdShell` capability checking method must be::
+
+ fn(self, supported_capabilities: MutableSet, unsupported_capabilities: MutableSet) -> None
+
+ The method must get whether a capability is supported or not from a testpmd command.
+ If multiple capabilities can be obtained from a testpmd command, each should be obtained
+ in the method. These capabilities should then be added to `supported_capabilities` or
+ `unsupported_capabilities` based on their support.
+
+ The two dictionaries are shared across all capability discovery function calls in a given
+ test run so that we don't call the same function multiple times,
+ e.g. when we find a :attr:`scattered_rx` in :meth:`TestPmdShell.get_capabilities_rxq`, we don't
+ go looking for it again if a different test case also needs it.
+ """
+
+ #: Scattered packets Rx enabled.
+ SCATTERED_RX_ENABLED: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rxq_info
+ )
+
+ def __call__(
+ self,
+ testpmd_shell: TestPmdShell,
+ supported_capabilities: MutableSet[Self],
+ unsupported_capabilities: MutableSet[Self],
+ ) -> None:
+ """Execute the associated capability retrieval function.
+
+ Args:
+ testpmd_shell: :class:`TestPmdShell` object to which the function will be bound to.
+ supported_capabilities: The dictionary storing the supported capabilities
+ of a given test run.
+ unsupported_capabilities: The dictionary storing the unsupported capabilities
+ of a given test run.
+ """
+ self.value(testpmd_shell, supported_capabilities, unsupported_capabilities)
diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py
index 8899f07f76..9a79e6ebb3 100644
--- a/dts/framework/testbed_model/capability.py
+++ b/dts/framework/testbed_model/capability.py
@@ -5,14 +5,40 @@
This module provides a protocol that defines the common attributes of test cases and suites
and support for test environment capabilities.
+
+Many test cases are testing features not available on all hardware.
+
+The module also allows developers to mark test cases or suites a requiring certain
+hardware capabilities with the :func:`requires` decorator.
+
+Example:
+ .. code:: python
+
+ from framework.test_suite import TestSuite, func_test
+ from framework.testbed_model.capability import NicCapability, requires
+ class TestPmdBufferScatter(TestSuite):
+ # only the test case requires the scattered_rx capability
+ # other test cases may not require it
+ @requires(NicCapability.scattered_rx)
+ @func_test
+ def test_scatter_mbuf_2048(self):
"""
from abc import ABC, abstractmethod
-from collections.abc import Sequence
-from typing import Callable, ClassVar, Protocol
+from collections.abc import MutableSet, Sequence
+from dataclasses import dataclass
+from typing import Callable, ClassVar, Protocol, Tuple
from typing_extensions import Self
+from framework.logger import get_dts_logger
+from framework.remote_session.testpmd_shell import (
+ NicCapability,
+ TestPmdShell,
+ TestPmdShellDecorator,
+ TestPmdShellSimpleMethod,
+)
+
from .sut_node import SutNode
from .topology import Topology
@@ -96,6 +122,128 @@ def __hash__(self) -> int:
"""The subclasses must be hashable so that they can be stored in sets."""
+@dataclass
+class DecoratedNicCapability(Capability):
+ """A wrapper around :class:`~framework.remote_session.testpmd_shell.NicCapability`.
+
+ Some NIC capabilities are only present or listed as supported only under certain conditions,
+ such as when a particular configuration is in place. This is achieved by allowing users to pass
+ a decorator function that decorates the function that gets the support status of the capability.
+
+ New instances should be created with the :meth:`create_unique` class method to ensure
+ there are no duplicate instances.
+
+ Attributes:
+ nic_capability: The NIC capability that partly defines each instance.
+ capability_decorator: The decorator function that will be passed the function associated
+ with `nic_capability` when discovering the support status of the capability.
+ Each instance is defined by `capability_decorator` along with `nic_capability`.
+ """
+
+ nic_capability: NicCapability
+ capability_decorator: TestPmdShellDecorator | None
+ _unique_capabilities: ClassVar[
+ dict[Tuple[NicCapability, TestPmdShellDecorator | None], Self]
+ ] = {}
+
+ @classmethod
+ def get_unique(
+ cls, nic_capability: NicCapability, decorator_fn: TestPmdShellDecorator | None
+ ) -> "DecoratedNicCapability":
+ """Get the capability uniquely identified by `nic_capability` and `decorator_fn`.
+
+ Args:
+ nic_capability: The NIC capability.
+ decorator_fn: The function that will be passed the function associated
+ with `nic_capability` when discovering the support status of the capability.
+
+ Returns:
+ The capability uniquely identified by `nic_capability` and `decorator_fn`.
+ """
+ if (nic_capability, decorator_fn) not in cls._unique_capabilities:
+ cls._unique_capabilities[(nic_capability, decorator_fn)] = cls(
+ nic_capability, decorator_fn
+ )
+ return cls._unique_capabilities[(nic_capability, decorator_fn)]
+
+ @classmethod
+ def get_supported_capabilities(
+ cls, sut_node: SutNode, topology: "Topology"
+ ) -> set["DecoratedNicCapability"]:
+ """Overrides :meth:`~Capability.get_supported_capabilities`.
+
+ The capabilities are first sorted by decorators, then reduced into a single function which
+ is then passed to the decorator. This way we only execute each decorator only once.
+ """
+ supported_conditional_capabilities: set["DecoratedNicCapability"] = set()
+ logger = get_dts_logger(f"{sut_node.name}.{cls.__name__}")
+ if topology.type is Topology.type.no_link:
+ logger.debug(
+ "No links available in the current topology, not getting NIC capabilities."
+ )
+ return supported_conditional_capabilities
+ logger.debug(
+ f"Checking which NIC capabilities from {cls.capabilities_to_check} are supported."
+ )
+ if cls.capabilities_to_check:
+ capabilities_to_check_map = cls._get_decorated_capabilities_map()
+ with TestPmdShell(sut_node, privileged=True) as testpmd_shell:
+ for conditional_capability_fn, capabilities in capabilities_to_check_map.items():
+ supported_capabilities: set[NicCapability] = set()
+ unsupported_capabilities: set[NicCapability] = set()
+ capability_fn = cls._reduce_capabilities(
+ capabilities, supported_capabilities, unsupported_capabilities
+ )
+ if conditional_capability_fn:
+ capability_fn = conditional_capability_fn(capability_fn)
+ capability_fn(testpmd_shell)
+ for supported_capability in supported_capabilities:
+ for capability in capabilities:
+ if supported_capability == capability.nic_capability:
+ supported_conditional_capabilities.add(capability)
+
+ logger.debug(f"Found supported capabilities {supported_conditional_capabilities}.")
+ return supported_conditional_capabilities
+
+ @classmethod
+ def _get_decorated_capabilities_map(
+ cls,
+ ) -> dict[TestPmdShellDecorator | None, set["DecoratedNicCapability"]]:
+ capabilities_map: dict[TestPmdShellDecorator | None, set["DecoratedNicCapability"]] = {}
+ for capability in cls.capabilities_to_check:
+ if capability.capability_decorator not in capabilities_map:
+ capabilities_map[capability.capability_decorator] = set()
+ capabilities_map[capability.capability_decorator].add(capability)
+
+ return capabilities_map
+
+ @classmethod
+ def _reduce_capabilities(
+ cls,
+ capabilities: set["DecoratedNicCapability"],
+ supported_capabilities: MutableSet,
+ unsupported_capabilities: MutableSet,
+ ) -> TestPmdShellSimpleMethod:
+ def reduced_fn(testpmd_shell: TestPmdShell) -> None:
+ for capability in capabilities:
+ capability.nic_capability(
+ testpmd_shell, supported_capabilities, unsupported_capabilities
+ )
+
+ return reduced_fn
+
+ def __hash__(self) -> int:
+ """Instances are identified by :attr:`nic_capability` and :attr:`capability_decorator`."""
+ return hash((self.nic_capability, self.capability_decorator))
+
+ def __repr__(self) -> str:
+ """Easy to read string of :attr:`nic_capability` and :attr:`capability_decorator`."""
+ condition_fn_name = ""
+ if self.capability_decorator:
+ condition_fn_name = f"{self.capability_decorator.__qualname__}|"
+ return f"{condition_fn_name}{self.nic_capability}"
+
+
class TestProtocol(Protocol):
"""Common test suite and test case attributes."""
@@ -116,6 +264,34 @@ def get_test_cases(cls, test_case_sublist: Sequence[str] | None = None) -> tuple
raise NotImplementedError()
+def requires(
+ *nic_capabilities: NicCapability,
+ decorator_fn: TestPmdShellDecorator | None = None,
+) -> Callable[[type[TestProtocol]], type[TestProtocol]]:
+ """A decorator that adds the required capabilities to a test case or test suite.
+
+ Args:
+ nic_capabilities: The NIC capabilities that are required by the test case or test suite.
+ decorator_fn: The decorator function that will be used when getting
+ NIC capability support status.
+ topology_type: The topology type the test suite or case requires.
+
+ Returns:
+ The decorated test case or test suite.
+ """
+
+ def add_required_capability(test_case_or_suite: type[TestProtocol]) -> type[TestProtocol]:
+ for nic_capability in nic_capabilities:
+ decorated_nic_capability = DecoratedNicCapability.get_unique(
+ nic_capability, decorator_fn
+ )
+ decorated_nic_capability.add_to_required(test_case_or_suite)
+
+ return test_case_or_suite
+
+ return add_required_capability
+
+
def get_supported_capabilities(
sut_node: SutNode,
topology_config: Topology,
diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
index 178a40385e..713549a5b2 100644
--- a/dts/tests/TestSuite_pmd_buffer_scatter.py
+++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
@@ -25,6 +25,7 @@
from framework.params.testpmd import SimpleForwardingModes
from framework.remote_session.testpmd_shell import TestPmdShell
from framework.test_suite import TestSuite, func_test
+from framework.testbed_model.capability import NicCapability, requires
class TestPmdBufferScatter(TestSuite):
@@ -123,6 +124,7 @@ def pmd_scatter(self, mbsize: int) -> None:
f"{offset}.",
)
+ @requires(NicCapability.SCATTERED_RX_ENABLED, decorator_fn=TestPmdShell.config_mtu_9000)
@func_test
def test_scatter_mbuf_2048(self) -> None:
"""Run the :meth:`pmd_scatter` test with `mbsize` set to 2048."""
--
2.34.1
^ permalink raw reply [flat|nested] 75+ messages in thread
* [PATCH v3 09/12] dts: add topology capability
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
` (7 preceding siblings ...)
2024-08-21 14:53 ` [PATCH v3 08/12] dts: add NIC capability support Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 17:13 ` Jeremy Spewock
2024-09-03 17:50 ` Dean Marx
2024-08-21 14:53 ` [PATCH v3 10/12] doc: add DTS capability doc sources Juraj Linkeš
` (3 subsequent siblings)
12 siblings, 2 replies; 75+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
Add support for marking test cases as requiring a certain topology. The
default topology is a two link topology and the other supported
topologies are one link and no link topologies.
The TestProtocol of test suites and cases is extended with the topology
type each test suite or case requires. Each test case starts out as
requiring a two link topology and can be marked as requiring as
topology directly (by decorating the test case) or through its test
suite. If a test suite is decorated as requiring a certain topology, all
its test cases are marked as such. If both test suite and a test case
are decorated as requiring a topology, the test case cannot require a
more complex topology than the whole suite (but it can require a less
complex one). If a test suite is not decorated, this has no effect on
required test case topology.
Since the default topology is defined as a reference to one of the
actual topologies, the NoAliasEnum from the aenum package is utilized,
which removes the aliasing of Enums so that TopologyType.two_links and
TopologyType.default are distinct. This is needed to distinguish between
a user passed value and the default value being used (which is used when
a test suite is or isn't decorated).
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/test_suite.py | 6 +-
dts/framework/testbed_model/capability.py | 182 +++++++++++++++++++++-
dts/framework/testbed_model/topology.py | 35 ++++-
dts/tests/TestSuite_hello_world.py | 2 +
dts/tests/TestSuite_pmd_buffer_scatter.py | 8 +-
dts/tests/TestSuite_smoke_tests.py | 2 +
6 files changed, 217 insertions(+), 18 deletions(-)
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 5c393ce8bf..51f49bd601 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -27,7 +27,7 @@
from framework.testbed_model.port import Port
from framework.testbed_model.sut_node import SutNode
from framework.testbed_model.tg_node import TGNode
-from framework.testbed_model.topology import Topology, TopologyType
+from framework.testbed_model.topology import Topology
from framework.testbed_model.traffic_generator.capturing_traffic_generator import (
PacketFilteringConfig,
)
@@ -73,7 +73,6 @@ class TestSuite(TestProtocol):
#: will block the execution of all subsequent test suites in the current build target.
is_blocking: ClassVar[bool] = False
_logger: DTSLogger
- _topology_type: TopologyType
_sut_port_ingress: Port
_sut_port_egress: Port
_sut_ip_address_ingress: Union[IPv4Interface, IPv6Interface]
@@ -102,7 +101,6 @@ def __init__(
self.sut_node = sut_node
self.tg_node = tg_node
self._logger = get_dts_logger(self.__class__.__name__)
- self._topology_type = topology.type
self._tg_port_egress = topology.tg_port_egress
self._sut_port_ingress = topology.sut_port_ingress
self._sut_port_egress = topology.sut_port_egress
@@ -468,6 +466,8 @@ def _decorator(func: TestSuiteMethodType) -> type[TestCase]:
test_case.skip = cls.skip
test_case.skip_reason = cls.skip_reason
test_case.required_capabilities = set()
+ test_case.topology_type = cls.topology_type
+ test_case.topology_type.add_to_required(test_case)
test_case.test_type = test_case_type
return test_case
diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py
index 9a79e6ebb3..998efa95d2 100644
--- a/dts/framework/testbed_model/capability.py
+++ b/dts/framework/testbed_model/capability.py
@@ -7,11 +7,29 @@
and support for test environment capabilities.
Many test cases are testing features not available on all hardware.
+On the other hand, some test cases or suites may not need the most complex topology available.
-The module also allows developers to mark test cases or suites a requiring certain
-hardware capabilities with the :func:`requires` decorator.
+The module allows developers to mark test cases or suites a requiring certain hardware capabilities
+or a particular topology with the :func:`requires` decorator.
+
+There are differences between hardware and topology capabilities:
+
+ * Hardware capabilities are assumed to not be required when not specified.
+ * However, some topology is always available, so each test case or suite is assigned
+ a default topology if no topology is specified in the decorator.
+
+Examples:
+ .. code:: python
+
+ from framework.test_suite import TestSuite, func_test
+ from framework.testbed_model.capability import TopologyType, requires
+ # The whole test suite (each test case within) doesn't require any links.
+ @requires(topology_type=TopologyType.no_link)
+ @func_test
+ class TestHelloWorld(TestSuite):
+ def hello_world_single_core(self):
+ ...
-Example:
.. code:: python
from framework.test_suite import TestSuite, func_test
@@ -24,6 +42,7 @@ class TestPmdBufferScatter(TestSuite):
def test_scatter_mbuf_2048(self):
"""
+import inspect
from abc import ABC, abstractmethod
from collections.abc import MutableSet, Sequence
from dataclasses import dataclass
@@ -31,6 +50,7 @@ def test_scatter_mbuf_2048(self):
from typing_extensions import Self
+from framework.exception import ConfigurationError
from framework.logger import get_dts_logger
from framework.remote_session.testpmd_shell import (
NicCapability,
@@ -40,7 +60,7 @@ def test_scatter_mbuf_2048(self):
)
from .sut_node import SutNode
-from .topology import Topology
+from .topology import Topology, TopologyType
class Capability(ABC):
@@ -244,6 +264,154 @@ def __repr__(self) -> str:
return f"{condition_fn_name}{self.nic_capability}"
+@dataclass
+class TopologyCapability(Capability):
+ """A wrapper around :class:`~.topology.TopologyType`.
+
+ Each test case must be assigned a topology. It could be done explicitly;
+ the implicit default is :attr:`~.topology.TopologyType.default`, which this class defines
+ as equal to :attr:`~.topology.TopologyType.two_links`.
+
+ Test case topology may be set by setting the topology for the whole suite.
+ The priority in which topology is set is as follows:
+
+ #. The topology set using the :func:`requires` decorator with a test case,
+ #. The topology set using the :func:`requires` decorator with a test suite,
+ #. The default topology if the decorator is not used.
+
+ The default topology of test suite (i.e. when not using the decorator
+ or not setting the topology with the decorator) does not affect the topology of test cases.
+
+ New instances should be created with the :meth:`create_unique` class method to ensure
+ there are no duplicate instances.
+
+ Attributes:
+ topology_type: The topology type that defines each instance.
+ """
+
+ topology_type: TopologyType
+
+ _unique_capabilities: ClassVar[dict[str, Self]] = {}
+
+ def _preprocess_required(self, test_case_or_suite: type["TestProtocol"]) -> None:
+ test_case_or_suite.required_capabilities.discard(test_case_or_suite.topology_type)
+ test_case_or_suite.topology_type = self
+
+ @classmethod
+ def get_unique(cls, topology_type: TopologyType) -> "TopologyCapability":
+ """Get the capability uniquely identified by `topology_type`.
+
+ Args:
+ topology_type: The topology type.
+
+ Returns:
+ The capability uniquely identified by `topology_type`.
+ """
+ if topology_type.name not in cls._unique_capabilities:
+ cls._unique_capabilities[topology_type.name] = cls(topology_type)
+ return cls._unique_capabilities[topology_type.name]
+
+ @classmethod
+ def get_supported_capabilities(
+ cls, sut_node: SutNode, topology: "Topology"
+ ) -> set["TopologyCapability"]:
+ """Overrides :meth:`~OSSession.get_supported_capabilities`."""
+ supported_capabilities = set()
+ topology_capability = cls.get_unique(topology.type)
+ for topology_type in TopologyType:
+ candidate_topology_type = cls.get_unique(topology_type)
+ if candidate_topology_type <= topology_capability:
+ supported_capabilities.add(candidate_topology_type)
+ return supported_capabilities
+
+ def set_required(self, test_case_or_suite: type["TestProtocol"]) -> None:
+ """The logic for setting the required topology of a test case or suite.
+
+ Decorators are applied on methods of a class first, then on the class.
+ This means we have to modify test case topologies when processing the test suite topologies.
+ At that point, the test case topologies have been set by the :func:`requires` decorator.
+ The test suite topology only affects the test case topologies
+ if not :attr:`~.topology.TopologyType.default`.
+ """
+ if inspect.isclass(test_case_or_suite):
+ if self.topology_type is not TopologyType.default:
+ self.add_to_required(test_case_or_suite)
+ func_test_cases, perf_test_cases = test_case_or_suite.get_test_cases()
+ for test_case in func_test_cases | perf_test_cases:
+ if test_case.topology_type.topology_type is TopologyType.default:
+ # test case topology has not been set, use the one set by the test suite
+ self.add_to_required(test_case)
+ elif test_case.topology_type > test_case_or_suite.topology_type:
+ raise ConfigurationError(
+ "The required topology type of a test case "
+ f"({test_case.__name__}|{test_case.topology_type}) "
+ "cannot be more complex than that of a suite "
+ f"({test_case_or_suite.__name__}|{test_case_or_suite.topology_type})."
+ )
+ else:
+ self.add_to_required(test_case_or_suite)
+
+ def __eq__(self, other) -> bool:
+ """Compare the :attr:`~TopologyCapability.topology_type`s.
+
+ Args:
+ other: The object to compare with.
+
+ Returns:
+ :data:`True` if the topology types are the same.
+ """
+ return self.topology_type == other.topology_type
+
+ def __lt__(self, other) -> bool:
+ """Compare the :attr:`~TopologyCapability.topology_type`s.
+
+ Args:
+ other: The object to compare with.
+
+ Returns:
+ :data:`True` if the instance's topology type is less complex than the compared object's.
+ """
+ return self.topology_type < other.topology_type
+
+ def __gt__(self, other) -> bool:
+ """Compare the :attr:`~TopologyCapability.topology_type`s.
+
+ Args:
+ other: The object to compare with.
+
+ Returns:
+ :data:`True` if the instance's topology type is more complex than the compared object's.
+ """
+ return other < self
+
+ def __le__(self, other) -> bool:
+ """Compare the :attr:`~TopologyCapability.topology_type`s.
+
+ Args:
+ other: The object to compare with.
+
+ Returns:
+ :data:`True` if the instance's topology type is less complex or equal than
+ the compared object's.
+ """
+ return not self > other
+
+ def __hash__(self):
+ """Each instance is identified by :attr:`topology_type`."""
+ return self.topology_type.__hash__()
+
+ def __str__(self):
+ """Easy to read string of class and name of :attr:`topology_type`."""
+ name = self.topology_type.name
+ if self.topology_type is TopologyType.default:
+ name = TopologyType.get_from_value(self.topology_type.value)
+ return f"{type(self.topology_type).__name__}.{name}"
+
+ def __repr__(self):
+ """Easy to read string of class and name of :attr:`topology_type`."""
+ return self.__str__()
+
+
class TestProtocol(Protocol):
"""Common test suite and test case attributes."""
@@ -251,6 +419,8 @@ class TestProtocol(Protocol):
skip: ClassVar[bool] = False
#: The reason for skipping the test case or suite.
skip_reason: ClassVar[str] = ""
+ #: The topology type of the test case or suite.
+ topology_type: ClassVar[TopologyCapability] = TopologyCapability(TopologyType.default)
#: The capabilities the test case or suite requires in order to be executed.
required_capabilities: ClassVar[set[Capability]] = set()
@@ -267,6 +437,7 @@ def get_test_cases(cls, test_case_sublist: Sequence[str] | None = None) -> tuple
def requires(
*nic_capabilities: NicCapability,
decorator_fn: TestPmdShellDecorator | None = None,
+ topology_type: TopologyType = TopologyType.default,
) -> Callable[[type[TestProtocol]], type[TestProtocol]]:
"""A decorator that adds the required capabilities to a test case or test suite.
@@ -287,6 +458,9 @@ def add_required_capability(test_case_or_suite: type[TestProtocol]) -> type[Test
)
decorated_nic_capability.add_to_required(test_case_or_suite)
+ topology_capability = TopologyCapability.get_unique(topology_type)
+ topology_capability.set_required(test_case_or_suite)
+
return test_case_or_suite
return add_required_capability
diff --git a/dts/framework/testbed_model/topology.py b/dts/framework/testbed_model/topology.py
index 19632ee890..712d252e44 100644
--- a/dts/framework/testbed_model/topology.py
+++ b/dts/framework/testbed_model/topology.py
@@ -8,15 +8,20 @@
"""
from dataclasses import dataclass
-from enum import IntEnum
-from typing import Iterable
+from typing import TYPE_CHECKING, Iterable
+
+if TYPE_CHECKING:
+ from enum import Enum as NoAliasEnum
+else:
+ from aenum import NoAliasEnum
from framework.config import PortConfig
+from framework.exception import ConfigurationError
from .port import Port
-class TopologyType(IntEnum):
+class TopologyType(int, NoAliasEnum):
"""Supported topology types."""
#: A topology with no Traffic Generator.
@@ -25,6 +30,28 @@ class TopologyType(IntEnum):
one_link = 1
#: A topology with two physical links between the Sut node and the TG node.
two_links = 2
+ #: The default topology required by test cases if not specified otherwise.
+ default = 2
+
+ @classmethod
+ def get_from_value(cls, value: int) -> "TopologyType":
+ """Get the corresponding instance from value.
+
+ :class:`Enum`s that don't allow aliases don't know which instance should be returned
+ as there could be multiple valid instances. Except for the :attr:`default` value,
+ :class:`TopologyType` is a regular Enum. When creating an instance from value, we're
+ not interested in the default, since we already know the value, allowing us to remove
+ the ambiguity.
+ """
+ match value:
+ case 0:
+ return TopologyType.no_link
+ case 1:
+ return TopologyType.one_link
+ case 2:
+ return TopologyType.two_links
+ case _:
+ raise ConfigurationError("More than two links in a topology are not supported.")
class Topology:
@@ -72,7 +99,7 @@ def __init__(self, sut_ports: Iterable[Port], tg_ports: Iterable[Port]):
):
port_links.append(PortLink(sut_port=sut_port, tg_port=tg_port))
- self.type = TopologyType(len(port_links))
+ self.type = TopologyType.get_from_value(len(port_links))
dummy_port = Port(PortConfig("", "", "", "", "", ""))
self.tg_port_egress = dummy_port
self.sut_port_ingress = dummy_port
diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
index 16d064ffeb..734f006026 100644
--- a/dts/tests/TestSuite_hello_world.py
+++ b/dts/tests/TestSuite_hello_world.py
@@ -9,6 +9,7 @@
from framework.remote_session.dpdk_shell import compute_eal_params
from framework.test_suite import TestSuite, func_test
+from framework.testbed_model.capability import TopologyType, requires
from framework.testbed_model.cpu import (
LogicalCoreCount,
LogicalCoreCountFilter,
@@ -16,6 +17,7 @@
)
+@requires(topology_type=TopologyType.no_link)
class TestHelloWorld(TestSuite):
"""DPDK hello world app test suite."""
diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
index 713549a5b2..89ece2ef56 100644
--- a/dts/tests/TestSuite_pmd_buffer_scatter.py
+++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
@@ -54,15 +54,9 @@ def set_up_suite(self) -> None:
"""Set up the test suite.
Setup:
- Verify that we have at least 2 port links in the current test run
- and increase the MTU of both ports on the traffic generator to 9000
+ Increase the MTU of both ports on the traffic generator to 9000
to support larger packet sizes.
"""
- self.verify(
- self._topology_type > 1,
- "There must be at least two port links to run the scatter test suite",
- )
-
self.tg_node.main_session.configure_port_mtu(9000, self._tg_port_egress)
self.tg_node.main_session.configure_port_mtu(9000, self._tg_port_ingress)
diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
index 94f90d9327..5f953a190f 100644
--- a/dts/tests/TestSuite_smoke_tests.py
+++ b/dts/tests/TestSuite_smoke_tests.py
@@ -18,9 +18,11 @@
from framework.remote_session.testpmd_shell import TestPmdShell
from framework.settings import SETTINGS
from framework.test_suite import TestSuite, func_test
+from framework.testbed_model.capability import TopologyType, requires
from framework.utils import REGEX_FOR_PCI_ADDRESS
+@requires(topology_type=TopologyType.no_link)
class TestSmokeTests(TestSuite):
"""DPDK and infrastructure smoke test suite.
--
2.34.1
^ permalink raw reply [flat|nested] 75+ messages in thread
* [PATCH v3 10/12] doc: add DTS capability doc sources
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
` (8 preceding siblings ...)
2024-08-21 14:53 ` [PATCH v3 09/12] dts: add topology capability Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 17:13 ` Jeremy Spewock
2024-09-03 17:52 ` Dean Marx
2024-08-21 14:53 ` [PATCH v3 11/12] dts: add Rx offload capabilities Juraj Linkeš
` (2 subsequent siblings)
12 siblings, 2 replies; 75+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
Add new files to generate DTS API documentation from.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
doc/api/dts/framework.testbed_model.capability.rst | 6 ++++++
doc/api/dts/framework.testbed_model.rst | 2 ++
doc/api/dts/framework.testbed_model.topology.rst | 6 ++++++
3 files changed, 14 insertions(+)
create mode 100644 doc/api/dts/framework.testbed_model.capability.rst
create mode 100644 doc/api/dts/framework.testbed_model.topology.rst
diff --git a/doc/api/dts/framework.testbed_model.capability.rst b/doc/api/dts/framework.testbed_model.capability.rst
new file mode 100644
index 0000000000..326fed9104
--- /dev/null
+++ b/doc/api/dts/framework.testbed_model.capability.rst
@@ -0,0 +1,6 @@
+capability - Testbed Capabilities
+=================================
+
+.. automodule:: framework.testbed_model.capability
+ :members:
+ :show-inheritance:
diff --git a/doc/api/dts/framework.testbed_model.rst b/doc/api/dts/framework.testbed_model.rst
index 4b024e47e6..e1f9543b28 100644
--- a/doc/api/dts/framework.testbed_model.rst
+++ b/doc/api/dts/framework.testbed_model.rst
@@ -21,6 +21,8 @@ testbed\_model - Testbed Modelling Package
framework.testbed_model.node
framework.testbed_model.sut_node
framework.testbed_model.tg_node
+ framework.testbed_model.capability
framework.testbed_model.cpu
framework.testbed_model.port
+ framework.testbed_model.topology
framework.testbed_model.virtual_device
diff --git a/doc/api/dts/framework.testbed_model.topology.rst b/doc/api/dts/framework.testbed_model.topology.rst
new file mode 100644
index 0000000000..f3d9d1f418
--- /dev/null
+++ b/doc/api/dts/framework.testbed_model.topology.rst
@@ -0,0 +1,6 @@
+topology - Testbed Topology
+===========================
+
+.. automodule:: framework.testbed_model.topology
+ :members:
+ :show-inheritance:
--
2.34.1
^ permalink raw reply [flat|nested] 75+ messages in thread
* [PATCH v3 11/12] dts: add Rx offload capabilities
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
` (9 preceding siblings ...)
2024-08-21 14:53 ` [PATCH v3 10/12] doc: add DTS capability doc sources Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 17:24 ` Jeremy Spewock
` (2 more replies)
2024-08-21 14:53 ` [PATCH v3 12/12] dts: add NIC capabilities from show port info Juraj Linkeš
2024-08-26 17:25 ` [PATCH v3 00/12] dts: add test skipping based on capabilities Jeremy Spewock
12 siblings, 3 replies; 75+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
The scatter Rx offload capability is needed for the pmd_buffer_scatter
test suite. The command that retrieves the capability is:
show port <port_id> rx_offload capabilities
The command also retrieves a lot of other capabilities (RX_OFFLOAD_*)
which are all added into a Flag. The Flag members correspond to NIC
capability names so a convenience function that looks for the supported
Flags in a testpmd output is also added.
The NIC capability names (mentioned above) are copy-pasted from the
Flag. Dynamic addition of Enum members runs into problems with typing
(mypy doesn't know about the members) and documentation generation
(Sphinx doesn't know about the members).
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/remote_session/testpmd_shell.py | 213 ++++++++++++++++++
dts/tests/TestSuite_pmd_buffer_scatter.py | 1 +
2 files changed, 214 insertions(+)
diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index 48c31124d1..f83569669e 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -659,6 +659,103 @@ class TestPmdPortStats(TextParser):
tx_bps: int = field(metadata=TextParser.find_int(r"Tx-bps:\s+(\d+)"))
+class RxOffloadCapability(Flag):
+ """Rx offload capabilities of a device."""
+
+ #:
+ RX_OFFLOAD_VLAN_STRIP = auto()
+ #: Device supports L3 checksum offload.
+ RX_OFFLOAD_IPV4_CKSUM = auto()
+ #: Device supports L4 checksum offload.
+ RX_OFFLOAD_UDP_CKSUM = auto()
+ #: Device supports L4 checksum offload.
+ RX_OFFLOAD_TCP_CKSUM = auto()
+ #: Device supports Large Receive Offload.
+ RX_OFFLOAD_TCP_LRO = auto()
+ #: Device supports QinQ (queue in queue) offload.
+ RX_OFFLOAD_QINQ_STRIP = auto()
+ #: Device supports inner packet L3 checksum.
+ RX_OFFLOAD_OUTER_IPV4_CKSUM = auto()
+ #: Device supports MACsec.
+ RX_OFFLOAD_MACSEC_STRIP = auto()
+ #: Device supports filtering of a VLAN Tag identifier.
+ RX_OFFLOAD_VLAN_FILTER = 1 << 9
+ #: Device supports VLAN offload.
+ RX_OFFLOAD_VLAN_EXTEND = auto()
+ #: Device supports receiving segmented mbufs.
+ RX_OFFLOAD_SCATTER = 1 << 13
+ #: Device supports Timestamp.
+ RX_OFFLOAD_TIMESTAMP = auto()
+ #: Device supports crypto processing while packet is received in NIC.
+ RX_OFFLOAD_SECURITY = auto()
+ #: Device supports CRC stripping.
+ RX_OFFLOAD_KEEP_CRC = auto()
+ #: Device supports L4 checksum offload.
+ RX_OFFLOAD_SCTP_CKSUM = auto()
+ #: Device supports inner packet L4 checksum.
+ RX_OFFLOAD_OUTER_UDP_CKSUM = auto()
+ #: Device supports RSS hashing.
+ RX_OFFLOAD_RSS_HASH = auto()
+ #: Device supports
+ RX_OFFLOAD_BUFFER_SPLIT = auto()
+ #: Device supports all checksum capabilities.
+ RX_OFFLOAD_CHECKSUM = RX_OFFLOAD_IPV4_CKSUM | RX_OFFLOAD_UDP_CKSUM | RX_OFFLOAD_TCP_CKSUM
+ #: Device supports all VLAN capabilities.
+ RX_OFFLOAD_VLAN = (
+ RX_OFFLOAD_VLAN_STRIP
+ | RX_OFFLOAD_VLAN_FILTER
+ | RX_OFFLOAD_VLAN_EXTEND
+ | RX_OFFLOAD_QINQ_STRIP
+ )
+
+ @classmethod
+ def from_string(cls, line: str) -> Self:
+ """Make an instance from a string containing the flag names separated with a space.
+
+ Args:
+ line: The line to parse.
+
+ Returns:
+ A new instance containing all found flags.
+ """
+ flag = cls(0)
+ for flag_name in line.split():
+ flag |= cls[f"RX_OFFLOAD_{flag_name}"]
+ return flag
+
+ @classmethod
+ def make_parser(cls, per_port: bool) -> ParserFn:
+ """Make a parser function.
+
+ Args:
+ per_port: If :data:`True`, will return capabilities per port. If :data:`False`,
+ will return capabilities per queue.
+
+ Returns:
+ ParserFn: A dictionary for the `dataclasses.field` metadata argument containing a
+ parser function that makes an instance of this flag from text.
+ """
+ granularity = "Port" if per_port else "Queue"
+ return TextParser.wrap(
+ TextParser.find(rf"Per {granularity}\s+:(.*)$", re.MULTILINE),
+ cls.from_string,
+ )
+
+
+@dataclass
+class RxOffloadCapabilities(TextParser):
+ """The result of testpmd's ``show port <port_id> rx_offload capabilities`` command."""
+
+ #:
+ port_id: int = field(
+ metadata=TextParser.find_int(r"Rx Offloading Capabilities of port (\d+) :")
+ )
+ #: Per-queue Rx offload capabilities.
+ per_queue: RxOffloadCapability = field(metadata=RxOffloadCapability.make_parser(False))
+ #: Capabilities other than per-queue Rx offload capabilities.
+ per_port: RxOffloadCapability = field(metadata=RxOffloadCapability.make_parser(True))
+
+
T = TypeVarTuple("T") # type: ignore[misc]
@@ -1048,6 +1145,42 @@ def _close(self) -> None:
====== Capability retrieval methods ======
"""
+ def get_capabilities_rx_offload(
+ self,
+ supported_capabilities: MutableSet["NicCapability"],
+ unsupported_capabilities: MutableSet["NicCapability"],
+ ) -> None:
+ """Get all rx offload capabilities and divide them into supported and unsupported.
+
+ Args:
+ supported_capabilities: Supported capabilities will be added to this set.
+ unsupported_capabilities: Unsupported capabilities will be added to this set.
+ """
+ self._logger.debug("Getting rx offload capabilities.")
+ command = f"show port {self.ports[0].id} rx_offload capabilities"
+ rx_offload_capabilities_out = self.send_command(command)
+ rx_offload_capabilities = RxOffloadCapabilities.parse(rx_offload_capabilities_out)
+ self._update_capabilities_from_flag(
+ supported_capabilities,
+ unsupported_capabilities,
+ RxOffloadCapability,
+ rx_offload_capabilities.per_port | rx_offload_capabilities.per_queue,
+ )
+
+ def _update_capabilities_from_flag(
+ self,
+ supported_capabilities: MutableSet["NicCapability"],
+ unsupported_capabilities: MutableSet["NicCapability"],
+ flag_class: type[Flag],
+ supported_flags: Flag,
+ ) -> None:
+ """Divide all flags from `flag_class` into supported and unsupported."""
+ for flag in flag_class:
+ if flag in supported_flags:
+ supported_capabilities.add(NicCapability[str(flag.name)])
+ else:
+ unsupported_capabilities.add(NicCapability[str(flag.name)])
+
def get_capabilities_rxq_info(
self,
supported_capabilities: MutableSet["NicCapability"],
@@ -1119,6 +1252,86 @@ class NicCapability(NoAliasEnum):
SCATTERED_RX_ENABLED: TestPmdShellCapabilityMethod = partial(
TestPmdShell.get_capabilities_rxq_info
)
+ #:
+ RX_OFFLOAD_VLAN_STRIP: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports L3 checksum offload.
+ RX_OFFLOAD_IPV4_CKSUM: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports L4 checksum offload.
+ RX_OFFLOAD_UDP_CKSUM: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports L4 checksum offload.
+ RX_OFFLOAD_TCP_CKSUM: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports Large Receive Offload.
+ RX_OFFLOAD_TCP_LRO: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports QinQ (queue in queue) offload.
+ RX_OFFLOAD_QINQ_STRIP: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports inner packet L3 checksum.
+ RX_OFFLOAD_OUTER_IPV4_CKSUM: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports MACsec.
+ RX_OFFLOAD_MACSEC_STRIP: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports filtering of a VLAN Tag identifier.
+ RX_OFFLOAD_VLAN_FILTER: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports VLAN offload.
+ RX_OFFLOAD_VLAN_EXTEND: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports receiving segmented mbufs.
+ RX_OFFLOAD_SCATTER: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports Timestamp.
+ RX_OFFLOAD_TIMESTAMP: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports crypto processing while packet is received in NIC.
+ RX_OFFLOAD_SECURITY: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports CRC stripping.
+ RX_OFFLOAD_KEEP_CRC: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports L4 checksum offload.
+ RX_OFFLOAD_SCTP_CKSUM: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports inner packet L4 checksum.
+ RX_OFFLOAD_OUTER_UDP_CKSUM: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports RSS hashing.
+ RX_OFFLOAD_RSS_HASH: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports scatter Rx packets to segmented mbufs.
+ RX_OFFLOAD_BUFFER_SPLIT: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports all checksum capabilities.
+ RX_OFFLOAD_CHECKSUM: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports all VLAN capabilities.
+ RX_OFFLOAD_VLAN: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
def __call__(
self,
diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
index 89ece2ef56..64c48b0793 100644
--- a/dts/tests/TestSuite_pmd_buffer_scatter.py
+++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
@@ -28,6 +28,7 @@
from framework.testbed_model.capability import NicCapability, requires
+@requires(NicCapability.RX_OFFLOAD_SCATTER)
class TestPmdBufferScatter(TestSuite):
"""DPDK PMD packet scattering test suite.
--
2.34.1
^ permalink raw reply [flat|nested] 75+ messages in thread
* [PATCH v3 12/12] dts: add NIC capabilities from show port info
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
` (10 preceding siblings ...)
2024-08-21 14:53 ` [PATCH v3 11/12] dts: add Rx offload capabilities Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 17:24 ` Jeremy Spewock
2024-09-03 18:02 ` Dean Marx
2024-08-26 17:25 ` [PATCH v3 00/12] dts: add test skipping based on capabilities Jeremy Spewock
12 siblings, 2 replies; 75+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
Add the capabilities advertised by the testpmd command "show port info"
so that test cases may be marked as requiring those capabilities:
RUNTIME_RX_QUEUE_SETUP
RUNTIME_TX_QUEUE_SETUP
RXQ_SHARE
FLOW_RULE_KEEP
FLOW_SHARED_OBJECT_KEEP
These names are copy pasted from the existing DeviceCapabilitiesFlag
class. Dynamic addition of Enum members runs into problems with typing
(mypy doesn't know about the members) and documentation generation
(Sphinx doesn't know about the members).
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/remote_session/testpmd_shell.py | 36 +++++++++++++++++++
1 file changed, 36 insertions(+)
diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index f83569669e..166ffc827e 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -1200,6 +1200,24 @@ def get_capabilities_rxq_info(
else:
unsupported_capabilities.add(NicCapability.SCATTERED_RX_ENABLED)
+ def get_capabilities_show_port_info(
+ self,
+ supported_capabilities: MutableSet["NicCapability"],
+ unsupported_capabilities: MutableSet["NicCapability"],
+ ) -> None:
+ """Get all capabilities from show port info and divide them into supported and unsupported.
+
+ Args:
+ supported_capabilities: Supported capabilities will be added to this set.
+ unsupported_capabilities: Unsupported capabilities will be added to this set.
+ """
+ self._update_capabilities_from_flag(
+ supported_capabilities,
+ unsupported_capabilities,
+ DeviceCapabilitiesFlag,
+ self.ports[0].device_capabilities,
+ )
+
"""
====== Decorator methods ======
"""
@@ -1332,6 +1350,24 @@ class NicCapability(NoAliasEnum):
RX_OFFLOAD_VLAN: TestPmdShellCapabilityMethod = partial(
TestPmdShell.get_capabilities_rx_offload
)
+ #: Device supports Rx queue setup after device started.
+ RUNTIME_RX_QUEUE_SETUP: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_show_port_info
+ )
+ #: Device supports Tx queue setup after device started.
+ RUNTIME_TX_QUEUE_SETUP: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_show_port_info
+ )
+ #: Device supports shared Rx queue among ports within Rx domain and switch domain.
+ RXQ_SHARE: TestPmdShellCapabilityMethod = partial(TestPmdShell.get_capabilities_show_port_info)
+ #: Device supports keeping flow rules across restart.
+ FLOW_RULE_KEEP: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_show_port_info
+ )
+ #: Device supports keeping shared flow objects across restart.
+ FLOW_SHARED_OBJECT_KEEP: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_show_port_info
+ )
def __call__(
self,
--
2.34.1
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 01/12] dts: fix default device error handling mode
2024-08-21 14:53 ` [PATCH v3 01/12] dts: fix default device error handling mode Juraj Linkeš
@ 2024-08-26 16:42 ` Jeremy Spewock
2024-08-27 16:15 ` Dean Marx
2024-08-27 20:09 ` Nicholas Pratte
2 siblings, 0 replies; 75+ messages in thread
From: Jeremy Spewock @ 2024-08-26 16:42 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
> The device_error_handling_mode of testpmd port may not be present, e.g.
> in VM ports.
>
> Fixes: 61d5bc9bf974 ("dts: add port info command to testpmd shell")
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 02/12] dts: add the aenum dependency
2024-08-21 14:53 ` [PATCH v3 02/12] dts: add the aenum dependency Juraj Linkeš
@ 2024-08-26 16:42 ` Jeremy Spewock
2024-08-27 16:28 ` Dean Marx
2024-08-27 20:21 ` Nicholas Pratte
2 siblings, 0 replies; 75+ messages in thread
From: Jeremy Spewock @ 2024-08-26 16:42 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
> Regular Python enumerations create only one instance for members with
> the same value, such as:
> class MyEnum(Enum):
> foo = 1
> bar = 1
>
> MyEnum.foo and MyEnum.bar are aliases that return the same instance.
I didn't know this was a thing in Python Enums. It was very strange to
me at first, but thinking about this more it makes some sense.
>
> DTS needs to return different instances in the above scenario so that we
> can map capabilities with different names to the same function that
> retrieves the capabilities.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 03/12] dts: add test case decorators
2024-08-21 14:53 ` [PATCH v3 03/12] dts: add test case decorators Juraj Linkeš
@ 2024-08-26 16:50 ` Jeremy Spewock
2024-09-05 8:07 ` Juraj Linkeš
2024-08-28 20:09 ` Dean Marx
2024-08-30 15:50 ` Nicholas Pratte
2 siblings, 1 reply; 75+ messages in thread
From: Jeremy Spewock @ 2024-08-26 16:50 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
<snip>
> class DTSRunner:
> @@ -232,9 +231,9 @@ def _get_test_suites_with_cases(
>
> for test_suite_config in test_suite_configs:
> test_suite_class = self._get_test_suite_class(test_suite_config.test_suite)
> - test_cases = []
> - func_test_cases, perf_test_cases = self._filter_test_cases(
> - test_suite_class, test_suite_config.test_cases
> + test_cases: list[type[TestCase]] = []
If TestCase is just a class, why is the `type[]` in the annotation
required? Are these not specific instances of the TestCase class? I
figured they would need to be in order for you to run the specific
test case methods. Maybe this has something to do with the class being
a Protocol?
> + func_test_cases, perf_test_cases = test_suite_class.get_test_cases(
> + test_suite_config.test_cases
> )
> if func:
> test_cases.extend(func_test_cases)
> @@ -309,57 +308,6 @@ def is_test_suite(object) -> bool:
> f"Couldn't find any valid test suites in {test_suite_module.__name__}."
> )
>
<snip>
> @@ -120,6 +123,68 @@ def _process_links(self) -> None:
> ):
> self._port_links.append(PortLink(sut_port=sut_port, tg_port=tg_port))
>
> + @classmethod
> + def get_test_cases(
> + cls, test_case_sublist: Sequence[str] | None = None
> + ) -> tuple[set[type["TestCase"]], set[type["TestCase"]]]:
> + """Filter `test_case_subset` from this class.
> +
> + Test cases are regular (or bound) methods decorated with :func:`func_test`
> + or :func:`perf_test`.
> +
> + Args:
> + test_case_sublist: Test case names to filter from this class.
> + If empty or :data:`None`, return all test cases.
> +
> + Returns:
> + The filtered test case functions. This method returns functions as opposed to methods,
> + as methods are bound to instances and this method only has access to the class.
> +
> + Raises:
> + ConfigurationError: If a test case from `test_case_subset` is not found.
> + """
> +
<snip>
> + for test_case_name, test_case_function in inspect.getmembers(cls, is_test_case):
> + if test_case_name in test_case_sublist_copy:
> + # if test_case_sublist_copy is non-empty, remove the found test case
> + # so that we can look at the remainder at the end
> + test_case_sublist_copy.remove(test_case_name)
> + elif test_case_sublist:
> + # if the original list is not empty (meaning we're filtering test cases),
> + # we're dealing with a test case we would've
I think this part of the comment about "we're dealing with a test case
we would've removed in the other branch" confused me a little bit. It
could just be a me thing, but I think this would have been more clear
for me if it was something more like "The original list is not empty
(meaning we're filtering test cases). Since we didn't remove this test
case in the other branch, it doesn't match the filter and we don't
want to run it."
> + # removed in the other branch; since we didn't, we don't want to run it
> + continue
> +
> + match test_case_function.test_type:
> + case TestCaseType.PERFORMANCE:
> + perf_test_cases.add(test_case_function)
> + case TestCaseType.FUNCTIONAL:
> + func_test_cases.add(test_case_function)
> +
> + if test_case_sublist_copy:
> + raise ConfigurationError(
> + f"Test cases {test_case_sublist_copy} not found among functions of {cls.__name__}."
> + )
> +
> + return func_test_cases, perf_test_cases
> +
<snip>
> 2.34.1
>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 04/12] dts: add mechanism to skip test cases or suites
2024-08-21 14:53 ` [PATCH v3 04/12] dts: add mechanism to skip test cases or suites Juraj Linkeš
@ 2024-08-26 16:52 ` Jeremy Spewock
2024-09-05 9:23 ` Juraj Linkeš
2024-08-28 20:37 ` Dean Marx
1 sibling, 1 reply; 75+ messages in thread
From: Jeremy Spewock @ 2024-08-26 16:52 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
<snip>
> --- a/dts/framework/test_result.py
> +++ b/dts/framework/test_result.py
> @@ -75,6 +75,20 @@ def create_config(self) -> TestSuiteConfig:
> test_cases=[test_case.__name__ for test_case in self.test_cases],
> )
>
> + @property
> + def skip(self) -> bool:
> + """Skip the test suite if all test cases or the suite itself are to be skipped.
> +
> + Returns:
> + :data:`True` if the test suite should be skipped, :data:`False` otherwise.
> + """
> + all_test_cases_skipped = True
> + for test_case in self.test_cases:
> + if not test_case.skip:
> + all_test_cases_skipped = False
> + break
You could also potentially implement this using the built-in `all()`
function. It would become a simple one-liner like
`all_test_cases_skipped = all(test_case.skip for test_case in
self.test_cases)`. That's probably short enough to even just put in
the return statement though if you wanted to.
> + return all_test_cases_skipped or self.test_suite_class.skip
> +
>
> class Result(Enum):
> """The possible states that a setup, a teardown or a test case may end up in."""
> @@ -86,12 +100,12 @@ class Result(Enum):
> #:
> ERROR = auto()
> #:
> - SKIP = auto()
> - #:
> BLOCK = auto()
> + #:
> + SKIP = auto()
>
> def __bool__(self) -> bool:
> - """Only PASS is True."""
> + """Only :attr:`PASS` is True."""
> return self is self.PASS
>
>
> @@ -169,12 +183,13 @@ def update_setup(self, result: Result, error: Exception | None = None) -> None:
> self.setup_result.result = result
> self.setup_result.error = error
>
> - if result in [Result.BLOCK, Result.ERROR, Result.FAIL]:
> - self.update_teardown(Result.BLOCK)
> - self._block_result()
> + if result != Result.PASS:
> + result_to_mark = Result.BLOCK if result != Result.SKIP else Result.SKIP
> + self.update_teardown(result_to_mark)
> + self._mark_results(result_to_mark)
>
> - def _block_result(self) -> None:
> - r"""Mark the result as :attr:`~Result.BLOCK`\ed.
> + def _mark_results(self, result) -> None:
Is it worth adding the type annotation for `result` here and to the
other places where this is implemented? I guess it doesn't matter that
much since it is a private method.
> + """Mark the result as well as the child result as `result`.
Are these methods even marking their own result or only their
children? It seems like it's only really updating the children
recursively and its result would have already been updated before this
was called.
>
> The blocking of child results should be done in overloaded methods.
> """
<snip>
>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 05/12] dts: add support for simpler topologies
2024-08-21 14:53 ` [PATCH v3 05/12] dts: add support for simpler topologies Juraj Linkeš
@ 2024-08-26 16:54 ` Jeremy Spewock
2024-09-05 9:42 ` Juraj Linkeš
2024-08-28 20:56 ` Dean Marx
1 sibling, 1 reply; 75+ messages in thread
From: Jeremy Spewock @ 2024-08-26 16:54 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
I just had one question below, otherwise:
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
<snip>
> diff --git a/dts/framework/testbed_model/topology.py b/dts/framework/testbed_model/topology.py
> new file mode 100644
> index 0000000000..19632ee890
> --- /dev/null
> +++ b/dts/framework/testbed_model/topology.py
<snip>
> +
> +
> +class TopologyType(IntEnum):
> + """Supported topology types."""
> +
> + #: A topology with no Traffic Generator.
> + no_link = 0
> + #: A topology with one physical link between the SUT node and the TG node.
> + one_link = 1
> + #: A topology with two physical links between the Sut node and the TG node.
> + two_links = 2
> +
> +
> +class Topology:
> + """Testbed topology.
> +
> + The topology contains ports processed into ingress and egress ports.
> + It's assumed that port0 of the SUT node is connected to port0 of the TG node and so on.
Do we need to make this assumption when you are comparing the port
directly to its peer and matching the addresses? I think you could
specify in conf.yaml that port 0 on the SUT is one of your ports and
its peer is port 1 on the TG and because you do the matching, this
would work fine.
> + If there are no ports on a node, dummy ports (ports with no actual values) are stored.
> + If there is only one link available, the ports of this link are stored
> + as both ingress and egress ports.
> +
> + The dummy ports shouldn't be used. It's up to :class:`~framework.runner.DTSRunner`
> + to ensure no test case or suite requiring actual links is executed
> + when the topology prohibits it and up to the developers to make sure that test cases
> + not requiring any links don't use any ports. Otherwise, the underlying methods
> + using the ports will fail.
> +
> + Attributes:
> + type: The type of the topology.
> + tg_port_egress: The egress port of the TG node.
> + sut_port_ingress: The ingress port of the SUT node.
> + sut_port_egress: The egress port of the SUT node.
> + tg_port_ingress: The ingress port of the TG node.
> + """
> +
> + type: TopologyType
> + tg_port_egress: Port
> + sut_port_ingress: Port
> + sut_port_egress: Port
> + tg_port_ingress: Port
> +
> + def __init__(self, sut_ports: Iterable[Port], tg_ports: Iterable[Port]):
> + """Create the topology from `sut_ports` and `tg_ports`.
> +
> + Args:
> + sut_ports: The SUT node's ports.
> + tg_ports: The TG node's ports.
> + """
> + port_links = []
> + for sut_port in sut_ports:
> + for tg_port in tg_ports:
> + if (sut_port.identifier, sut_port.peer) == (
> + tg_port.peer,
> + tg_port.identifier,
> + ):
> + port_links.append(PortLink(sut_port=sut_port, tg_port=tg_port))
> +
> + self.type = TopologyType(len(port_links))
<snip>
>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 06/12] dst: add basic capability support
2024-08-21 14:53 ` [PATCH v3 06/12] dst: add basic capability support Juraj Linkeš
@ 2024-08-26 16:56 ` Jeremy Spewock
2024-09-05 9:50 ` Juraj Linkeš
2024-09-03 16:03 ` Dean Marx
1 sibling, 1 reply; 75+ messages in thread
From: Jeremy Spewock @ 2024-08-26 16:56 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
Just one comment about adding something to a doc-string, otherwise
looks good to me:
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
<snip>
> diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
> index 306b100bc6..b4b58ef348 100644
> --- a/dts/framework/test_result.py
> +++ b/dts/framework/test_result.py
> @@ -25,10 +25,12 @@
>
> import os.path
> from collections.abc import MutableSequence
> -from dataclasses import dataclass
> +from dataclasses import dataclass, field
> from enum import Enum, auto
> from typing import Union
>
> +from framework.testbed_model.capability import Capability
> +
> from .config import (
> OS,
> Architecture,
> @@ -63,6 +65,12 @@ class is to hold a subset of test cases (which could be all test cases) because
>
> test_suite_class: type[TestSuite]
> test_cases: list[type[TestCase]]
> + required_capabilities: set[Capability] = field(default_factory=set, init=False)
This should probably be added to the Attributes section of the
doc-string for the class. When it's there, it might also be useful to
explain that this is used by the runner to determine what capabilities
need to be searched for to mark the suite for being skipped. The only
reason I think that would be useful is it helps differentiate this
list of capabilities from the list of required capabilities that every
test suite and test case has.
> +
> + def __post_init__(self):
> + """Gather the required capabilities of the test suite and all test cases."""
> + for test_object in [self.test_suite_class] + self.test_cases:
> + self.required_capabilities.update(test_object.required_capabilities)
<snip>
>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 07/12] dts: add testpmd port information caching
2024-08-21 14:53 ` [PATCH v3 07/12] dts: add testpmd port information caching Juraj Linkeš
@ 2024-08-26 16:56 ` Jeremy Spewock
2024-09-03 16:12 ` Dean Marx
1 sibling, 0 replies; 75+ messages in thread
From: Jeremy Spewock @ 2024-08-26 16:56 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
> When using port information multiple times in a testpmd shell instance
> lifespan, it's desirable to not get the information each time, so
> caching is added. In case the information changes, there's a way to
> force the update.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 08/12] dts: add NIC capability support
2024-08-21 14:53 ` [PATCH v3 08/12] dts: add NIC capability support Juraj Linkeš
@ 2024-08-26 17:11 ` Jeremy Spewock
2024-09-05 11:56 ` Juraj Linkeš
2024-08-27 16:36 ` Jeremy Spewock
2024-09-03 19:13 ` Dean Marx
2 siblings, 1 reply; 75+ messages in thread
From: Jeremy Spewock @ 2024-08-26 17:11 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
<snip>
> @dataclass
> class TestPmdPort(TextParser):
> """Dataclass representing the result of testpmd's ``show port info`` command."""
> @@ -962,3 +1043,96 @@ def _close(self) -> None:
> self.stop()
> self.send_command("quit", "Bye...")
> return super()._close()
> +
> + """
> + ====== Capability retrieval methods ======
> + """
> +
> + def get_capabilities_rxq_info(
> + self,
> + supported_capabilities: MutableSet["NicCapability"],
> + unsupported_capabilities: MutableSet["NicCapability"],
> + ) -> None:
> + """Get all rxq capabilities and divide them into supported and unsupported.
> +
> + Args:
> + supported_capabilities: Supported capabilities will be added to this set.
> + unsupported_capabilities: Unsupported capabilities will be added to this set.
> + """
> + self._logger.debug("Getting rxq capabilities.")
> + command = f"show rxq info {self.ports[0].id} 0"
> + rxq_info = TestPmdRxqInfo.parse(self.send_command(command))
> + if rxq_info.rx_scattered_packets:
> + supported_capabilities.add(NicCapability.SCATTERED_RX_ENABLED)
> + else:
> + unsupported_capabilities.add(NicCapability.SCATTERED_RX_ENABLED)
> +
> + """
> + ====== Decorator methods ======
> + """
> +
> + @staticmethod
> + def config_mtu_9000(testpmd_method: TestPmdShellSimpleMethod) -> TestPmdShellDecoratedMethod:
It might be more valuable for me to make a method for configuring the
MTU of all ports so that you don't have to do the loops yourself, I
can add this to the MTU patch once I update that and rebase it on
main.
> + """Configure MTU to 9000 on all ports, run `testpmd_method`, then revert.
> +
> + Args:
> + testpmd_method: The method to decorate.
> +
> + Returns:
> + The method decorated with setting and reverting MTU.
> + """
> +
> + def wrapper(testpmd_shell: Self):
> + original_mtus = []
> + for port in testpmd_shell.ports:
> + original_mtus.append((port.id, port.mtu))
> + testpmd_shell.set_port_mtu(port_id=port.id, mtu=9000, verify=False)
> + testpmd_method(testpmd_shell)
> + for port_id, mtu in original_mtus:
> + testpmd_shell.set_port_mtu(port_id=port_id, mtu=mtu if mtu else 1500, verify=False)
> +
> + return wrapper
<snip>
> diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py
> index 8899f07f76..9a79e6ebb3 100644
> --- a/dts/framework/testbed_model/capability.py
> +++ b/dts/framework/testbed_model/capability.py
> @@ -5,14 +5,40 @@
>
> This module provides a protocol that defines the common attributes of test cases and suites
> and support for test environment capabilities.
> +
> +Many test cases are testing features not available on all hardware.
> +
> +The module also allows developers to mark test cases or suites a requiring certain
small typo: I think you meant " mark test cases or suites *as*
requiring certain..."
> +hardware capabilities with the :func:`requires` decorator.
> +
> +Example:
> + .. code:: python
> +
> + from framework.test_suite import TestSuite, func_test
> + from framework.testbed_model.capability import NicCapability, requires
> + class TestPmdBufferScatter(TestSuite):
> + # only the test case requires the scattered_rx capability
> + # other test cases may not require it
> + @requires(NicCapability.scattered_rx)
Is it worth updating this to what the enum actually holds
(SCATTERED_RX_ENABLED) or not really since it is just an example in a
doc-string? I think it could do either way, but it might be better to
keep it consistent at least to start.
> + @func_test
> + def test_scatter_mbuf_2048(self):
<snip>
>
> @@ -96,6 +122,128 @@ def __hash__(self) -> int:
> """The subclasses must be hashable so that they can be stored in sets."""
>
>
> +@dataclass
> +class DecoratedNicCapability(Capability):
> + """A wrapper around :class:`~framework.remote_session.testpmd_shell.NicCapability`.
> +
> + Some NIC capabilities are only present or listed as supported only under certain conditions,
> + such as when a particular configuration is in place. This is achieved by allowing users to pass
> + a decorator function that decorates the function that gets the support status of the capability.
> +
> + New instances should be created with the :meth:`create_unique` class method to ensure
> + there are no duplicate instances.
> +
> + Attributes:
> + nic_capability: The NIC capability that partly defines each instance.
> + capability_decorator: The decorator function that will be passed the function associated
> + with `nic_capability` when discovering the support status of the capability.
> + Each instance is defined by `capability_decorator` along with `nic_capability`.
> + """
> +
> + nic_capability: NicCapability
> + capability_decorator: TestPmdShellDecorator | None
> + _unique_capabilities: ClassVar[
> + dict[Tuple[NicCapability, TestPmdShellDecorator | None], Self]
> + ] = {}
> +
> + @classmethod
> + def get_unique(
> + cls, nic_capability: NicCapability, decorator_fn: TestPmdShellDecorator | None
> + ) -> "DecoratedNicCapability":
This idea of get_unique really confused me at first. After reading
different parts of the code to learn how it is being used, I think I
understand now what it's for. My current understanding is basically
that you're using an uninstantiated class as essentially a factory
that stores a dictionary that you are using to hold singletons. It
might be confusing to me in general because I haven't really seen this
idea of dynamically modifying attributes of a class itself rather than
an instance of the class used this way. Understanding it now, it makes
sense what you are trying to do and how this is essentially a nice
cache/factory for singleton values for each capability, but It might
be helpful to document a little more somehow that _unique_capabilities
is really just a container for the singleton capabilities, and that
the top-level class is modified to keep a consistent state throughout
the framework.
Again, it could just be me having not really seen this idea used
before, but it was strange to wrap my head around at first since I'm
more used to class methods being used to read the state of attributes.
> + """Get the capability uniquely identified by `nic_capability` and `decorator_fn`.
> +
> + Args:
> + nic_capability: The NIC capability.
> + decorator_fn: The function that will be passed the function associated
> + with `nic_capability` when discovering the support status of the capability.
> +
> + Returns:
> + The capability uniquely identified by `nic_capability` and `decorator_fn`.
> + """
> + if (nic_capability, decorator_fn) not in cls._unique_capabilities:
> + cls._unique_capabilities[(nic_capability, decorator_fn)] = cls(
> + nic_capability, decorator_fn
> + )
> + return cls._unique_capabilities[(nic_capability, decorator_fn)]
> +
> + @classmethod
> + def get_supported_capabilities(
> + cls, sut_node: SutNode, topology: "Topology"
> + ) -> set["DecoratedNicCapability"]:
> + """Overrides :meth:`~Capability.get_supported_capabilities`.
> +
> + The capabilities are first sorted by decorators, then reduced into a single function which
> + is then passed to the decorator. This way we only execute each decorator only once.
This second sentence repeats the word "only" but I don't think it is
really necessary to and it might flow better with either one of them
instead of both.
> + """
> + supported_conditional_capabilities: set["DecoratedNicCapability"] = set()
> + logger = get_dts_logger(f"{sut_node.name}.{cls.__name__}")
> + if topology.type is Topology.type.no_link:
> + logger.debug(
> + "No links available in the current topology, not getting NIC capabilities."
> + )
> + return supported_conditional_capabilities
> + logger.debug(
> + f"Checking which NIC capabilities from {cls.capabilities_to_check} are supported."
> + )
> + if cls.capabilities_to_check:
> + capabilities_to_check_map = cls._get_decorated_capabilities_map()
> + with TestPmdShell(sut_node, privileged=True) as testpmd_shell:
> + for conditional_capability_fn, capabilities in capabilities_to_check_map.items():
> + supported_capabilities: set[NicCapability] = set()
> + unsupported_capabilities: set[NicCapability] = set()
> + capability_fn = cls._reduce_capabilities(
> + capabilities, supported_capabilities, unsupported_capabilities
> + )
This combines calling all of the capabilities into one function, but
if there are multiple capabilities that use the same underlying
testpmd function won't this call the same method multiple times? Or is
this handled by two Enum values in NicCapability that have the same
testpmd method as their value hashing to the same thing? For example,
if there are two capabilities that both require show rxq info and the
same decorator (scatter and some other capability X), won't this call
`show rxq info` twice even though you already know that the capability
is supported after the first call? It's not really harmful for this to
happen, but it would go against the idea of calling a method and
getting all of the capabilities that you can the first time. Maybe it
could be fixed with a conditional check which verifies if `capability`
is already in `supported_capabilities` or `unsupported_capabilities`
or not if it's a problem?
> + if conditional_capability_fn:
> + capability_fn = conditional_capability_fn(capability_fn)
> + capability_fn(testpmd_shell)
> + for supported_capability in supported_capabilities:
> + for capability in capabilities:
> + if supported_capability == capability.nic_capability:
> + supported_conditional_capabilities.add(capability)
I might be misunderstanding, but is this also achievable by just writing:
for capability in capabilities:
if capability.nic_capability in supported_capabilities:
supported_conditional_capabilities.add(capability)
I think that would be functionally the same, but I think it reads
easier than a nested loop.
> +
> + logger.debug(f"Found supported capabilities {supported_conditional_capabilities}.")
> + return supported_conditional_capabilities
> +
> + @classmethod
> + def _get_decorated_capabilities_map(
> + cls,
> + ) -> dict[TestPmdShellDecorator | None, set["DecoratedNicCapability"]]:
> + capabilities_map: dict[TestPmdShellDecorator | None, set["DecoratedNicCapability"]] = {}
> + for capability in cls.capabilities_to_check:
> + if capability.capability_decorator not in capabilities_map:
> + capabilities_map[capability.capability_decorator] = set()
> + capabilities_map[capability.capability_decorator].add(capability)
> +
> + return capabilities_map
> +
> + @classmethod
> + def _reduce_capabilities(
> + cls,
> + capabilities: set["DecoratedNicCapability"],
> + supported_capabilities: MutableSet,
> + unsupported_capabilities: MutableSet,
> + ) -> TestPmdShellSimpleMethod:
> + def reduced_fn(testpmd_shell: TestPmdShell) -> None:
> + for capability in capabilities:
> + capability.nic_capability(
> + testpmd_shell, supported_capabilities, unsupported_capabilities
> + )
> +
> + return reduced_fn
Would it make sense to put these two methods above
get_supported_capabilities since that is where they are used? I might
be in favor of it just because it would save you from having to look
further down in the diff to find what the method does and then go back
up, but I also understand that it looks like you might have been
sorting methods by private vs. public so if you think it makes more
sense to leave them here that is also viable.
> +
> + def __hash__(self) -> int:
> + """Instances are identified by :attr:`nic_capability` and :attr:`capability_decorator`."""
> + return hash((self.nic_capability, self.capability_decorator))
I guess my question above is asking if `hash(self.nic_capability) ==
hash(self.nic_capability.value())` because, if they aren't, then I
think the map will contain multiple capabilities that use the same
testpmd function since the capabilities themselves are unique, and
then because the get_supported_capabilities() method above just calls
whatever is in this map, it would call it twice. I think the whole
point of the NoAliasEnum is making sure that they don't hash to the
same thing. I could be missing something, but, if I am, maybe some
kind of comment showing where this is handled would be helpful.
> +
> + def __repr__(self) -> str:
> + """Easy to read string of :attr:`nic_capability` and :attr:`capability_decorator`."""
> + condition_fn_name = ""
> + if self.capability_decorator:
> + condition_fn_name = f"{self.capability_decorator.__qualname__}|"
> + return f"{condition_fn_name}{self.nic_capability}"
> +
> +
> class TestProtocol(Protocol):
> """Common test suite and test case attributes."""
>
> @@ -116,6 +264,34 @@ def get_test_cases(cls, test_case_sublist: Sequence[str] | None = None) -> tuple
> raise NotImplementedError()
>
>
> +def requires(
> + *nic_capabilities: NicCapability,
> + decorator_fn: TestPmdShellDecorator | None = None,
> +) -> Callable[[type[TestProtocol]], type[TestProtocol]]:
> + """A decorator that adds the required capabilities to a test case or test suite.
> +
> + Args:
> + nic_capabilities: The NIC capabilities that are required by the test case or test suite.
> + decorator_fn: The decorator function that will be used when getting
> + NIC capability support status.
> + topology_type: The topology type the test suite or case requires.
> +
> + Returns:
> + The decorated test case or test suite.
> + """
> +
> + def add_required_capability(test_case_or_suite: type[TestProtocol]) -> type[TestProtocol]:
> + for nic_capability in nic_capabilities:
> + decorated_nic_capability = DecoratedNicCapability.get_unique(
> + nic_capability, decorator_fn
> + )
> + decorated_nic_capability.add_to_required(test_case_or_suite)
> +
> + return test_case_or_suite
> +
> + return add_required_capability
> +
> +
> def get_supported_capabilities(
> sut_node: SutNode,
> topology_config: Topology,
> diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
> index 178a40385e..713549a5b2 100644
> --- a/dts/tests/TestSuite_pmd_buffer_scatter.py
> +++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
> @@ -25,6 +25,7 @@
> from framework.params.testpmd import SimpleForwardingModes
> from framework.remote_session.testpmd_shell import TestPmdShell
> from framework.test_suite import TestSuite, func_test
> +from framework.testbed_model.capability import NicCapability, requires
>
>
> class TestPmdBufferScatter(TestSuite):
> @@ -123,6 +124,7 @@ def pmd_scatter(self, mbsize: int) -> None:
> f"{offset}.",
> )
>
> + @requires(NicCapability.SCATTERED_RX_ENABLED, decorator_fn=TestPmdShell.config_mtu_9000)
Is it possible to instead associate the required decorator with the
scattered_rx capability itself? Since the configuration is required to
check the capability, I don't think there will ever be a case where
`decorator_fn` isn't required here, or a case where it is ever
anything other than modifying the MTU. Maybe it is more clear from the
reader's perspective this way that there are other things happening
under-the-hood, but it also saves developers from having to specify
something static when we already know beforehand what they need to
specify.
Doing so would probably mess up some of what you have written in the
way of DecoratedNicCapability and it might be more difficult to do it
in a way that only calls the decorator method once if there are
multiple capabilities that require the same decorator.
Maybe something that you could do is make the NicCapability class in
Testpmd have values that are tuples of (decorator_fn | None,
get_capabilities_fn), and then you can still have the
DecoratedNicCapabilitity class and the methods wouldn't really need to
change. I think the main thing that would change is just that the
decorator_fn is collected from the capability/enum instead of the
requires() method. You could potentially make get_unique easier as
well since you can just rely on the enum values since already know
what is required. Then you could take the pairs from that enum and
create a mapping like you have now of which ones require which
decorators and keep the same idea.
> @func_test
> def test_scatter_mbuf_2048(self) -> None:
> """Run the :meth:`pmd_scatter` test with `mbsize` set to 2048."""
> --
> 2.34.1
>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 09/12] dts: add topology capability
2024-08-21 14:53 ` [PATCH v3 09/12] dts: add topology capability Juraj Linkeš
@ 2024-08-26 17:13 ` Jeremy Spewock
2024-09-03 17:50 ` Dean Marx
1 sibling, 0 replies; 75+ messages in thread
From: Jeremy Spewock @ 2024-08-26 17:13 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
> Add support for marking test cases as requiring a certain topology. The
> default topology is a two link topology and the other supported
> topologies are one link and no link topologies.
>
> The TestProtocol of test suites and cases is extended with the topology
> type each test suite or case requires. Each test case starts out as
> requiring a two link topology and can be marked as requiring as
> topology directly (by decorating the test case) or through its test
> suite. If a test suite is decorated as requiring a certain topology, all
> its test cases are marked as such. If both test suite and a test case
> are decorated as requiring a topology, the test case cannot require a
> more complex topology than the whole suite (but it can require a less
> complex one). If a test suite is not decorated, this has no effect on
> required test case topology.
>
> Since the default topology is defined as a reference to one of the
> actual topologies, the NoAliasEnum from the aenum package is utilized,
> which removes the aliasing of Enums so that TopologyType.two_links and
> TopologyType.default are distinct. This is needed to distinguish between
> a user passed value and the default value being used (which is used when
> a test suite is or isn't decorated).
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
This patch looks good to me outside of some of the overlapping
comments from the DecoratedNicCapability class (mainly just
_get_unique).
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 10/12] doc: add DTS capability doc sources
2024-08-21 14:53 ` [PATCH v3 10/12] doc: add DTS capability doc sources Juraj Linkeš
@ 2024-08-26 17:13 ` Jeremy Spewock
2024-09-03 17:52 ` Dean Marx
1 sibling, 0 replies; 75+ messages in thread
From: Jeremy Spewock @ 2024-08-26 17:13 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
> Add new files to generate DTS API documentation from.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 11/12] dts: add Rx offload capabilities
2024-08-21 14:53 ` [PATCH v3 11/12] dts: add Rx offload capabilities Juraj Linkeš
@ 2024-08-26 17:24 ` Jeremy Spewock
2024-09-18 14:18 ` Juraj Linkeš
2024-08-28 17:44 ` Jeremy Spewock
2024-09-03 19:49 ` Dean Marx
2 siblings, 1 reply; 75+ messages in thread
From: Jeremy Spewock @ 2024-08-26 17:24 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
<snip>
> diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
> index 48c31124d1..f83569669e 100644
> --- a/dts/framework/remote_session/testpmd_shell.py
> +++ b/dts/framework/remote_session/testpmd_shell.py
> @@ -659,6 +659,103 @@ class TestPmdPortStats(TextParser):
> tx_bps: int = field(metadata=TextParser.find_int(r"Tx-bps:\s+(\d+)"))
>
>
> +class RxOffloadCapability(Flag):
> + """Rx offload capabilities of a device."""
> +
> + #:
> + RX_OFFLOAD_VLAN_STRIP = auto()
> + #: Device supports L3 checksum offload.
> + RX_OFFLOAD_IPV4_CKSUM = auto()
> + #: Device supports L4 checksum offload.
> + RX_OFFLOAD_UDP_CKSUM = auto()
> + #: Device supports L4 checksum offload.
> + RX_OFFLOAD_TCP_CKSUM = auto()
> + #: Device supports Large Receive Offload.
> + RX_OFFLOAD_TCP_LRO = auto()
> + #: Device supports QinQ (queue in queue) offload.
> + RX_OFFLOAD_QINQ_STRIP = auto()
> + #: Device supports inner packet L3 checksum.
> + RX_OFFLOAD_OUTER_IPV4_CKSUM = auto()
> + #: Device supports MACsec.
> + RX_OFFLOAD_MACSEC_STRIP = auto()
> + #: Device supports filtering of a VLAN Tag identifier.
> + RX_OFFLOAD_VLAN_FILTER = 1 << 9
> + #: Device supports VLAN offload.
> + RX_OFFLOAD_VLAN_EXTEND = auto()
> + #: Device supports receiving segmented mbufs.
> + RX_OFFLOAD_SCATTER = 1 << 13
I know you mentioned in the commit message that the auto() can cause
problems with mypy/sphinx, is that why this one is a specific value
instead? Regardless, I think we should probably make it consistent so
that either all of them are bit-shifts or none of them are unless
there is a specific reason that the scatter offload is different.
> + #: Device supports Timestamp.
> + RX_OFFLOAD_TIMESTAMP = auto()
> + #: Device supports crypto processing while packet is received in NIC.
> + RX_OFFLOAD_SECURITY = auto()
> + #: Device supports CRC stripping.
> + RX_OFFLOAD_KEEP_CRC = auto()
> + #: Device supports L4 checksum offload.
> + RX_OFFLOAD_SCTP_CKSUM = auto()
> + #: Device supports inner packet L4 checksum.
> + RX_OFFLOAD_OUTER_UDP_CKSUM = auto()
> + #: Device supports RSS hashing.
> + RX_OFFLOAD_RSS_HASH = auto()
> + #: Device supports
> + RX_OFFLOAD_BUFFER_SPLIT = auto()
> + #: Device supports all checksum capabilities.
> + RX_OFFLOAD_CHECKSUM = RX_OFFLOAD_IPV4_CKSUM | RX_OFFLOAD_UDP_CKSUM | RX_OFFLOAD_TCP_CKSUM
> + #: Device supports all VLAN capabilities.
> + RX_OFFLOAD_VLAN = (
> + RX_OFFLOAD_VLAN_STRIP
> + | RX_OFFLOAD_VLAN_FILTER
> + | RX_OFFLOAD_VLAN_EXTEND
> + | RX_OFFLOAD_QINQ_STRIP
> + )
<snip>
>
> @@ -1048,6 +1145,42 @@ def _close(self) -> None:
> ====== Capability retrieval methods ======
> """
>
> + def get_capabilities_rx_offload(
> + self,
> + supported_capabilities: MutableSet["NicCapability"],
> + unsupported_capabilities: MutableSet["NicCapability"],
> + ) -> None:
> + """Get all rx offload capabilities and divide them into supported and unsupported.
> +
> + Args:
> + supported_capabilities: Supported capabilities will be added to this set.
> + unsupported_capabilities: Unsupported capabilities will be added to this set.
> + """
> + self._logger.debug("Getting rx offload capabilities.")
> + command = f"show port {self.ports[0].id} rx_offload capabilities"
Is it desirable to only get the capabilities of the first port? In the
current framework I suppose it doesn't matter all that much since you
can only use the first few ports in the list of ports anyway, but will
there ever be a case where a test run has 2 different devices included
in the list of ports? Of course it's possible that it will happen, but
is it practical? Because, if so, then we would want this to aggregate
what all the devices are capable of and have capabilities basically
say "at least one of the ports in the list of ports is capable of
these things."
This consideration also applies to the rxq info capability gathering as well.
> + rx_offload_capabilities_out = self.send_command(command)
> + rx_offload_capabilities = RxOffloadCapabilities.parse(rx_offload_capabilities_out)
> + self._update_capabilities_from_flag(
> + supported_capabilities,
> + unsupported_capabilities,
> + RxOffloadCapability,
> + rx_offload_capabilities.per_port | rx_offload_capabilities.per_queue,
> + )
> +
<snip>
>
> def __call__(
> self,
> diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
> index 89ece2ef56..64c48b0793 100644
> --- a/dts/tests/TestSuite_pmd_buffer_scatter.py
> +++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
> @@ -28,6 +28,7 @@
> from framework.testbed_model.capability import NicCapability, requires
>
>
> +@requires(NicCapability.RX_OFFLOAD_SCATTER)
I know that we talked about this and how, in the environments we
looked at, it was true that the offload was supported in all cases
where the "native" or non-offloaded was supported, but thinking about
this more, I wonder if it is worth generalizing this assumption to all
NICs or if we can just decorate the second test case that I wrote
which uses the offloaded support. As long as the capabilities exposed
by testpmd are accurate, even if this assumption was true, the
capability for the non-offloaded one would show False when this
offload wasn't usable and it would skip the test case anyway, so I
don't think we lose anything by not including this test-suite-level
requirement and making it more narrow to the test cases that require
it.
Let me know your thoughts on that though and I would be interested to
hear if anyone else has any.
> class TestPmdBufferScatter(TestSuite):
> """DPDK PMD packet scattering test suite.
>
> --
> 2.34.1
>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 12/12] dts: add NIC capabilities from show port info
2024-08-21 14:53 ` [PATCH v3 12/12] dts: add NIC capabilities from show port info Juraj Linkeš
@ 2024-08-26 17:24 ` Jeremy Spewock
2024-09-03 18:02 ` Dean Marx
1 sibling, 0 replies; 75+ messages in thread
From: Jeremy Spewock @ 2024-08-26 17:24 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
> Add the capabilities advertised by the testpmd command "show port info"
> so that test cases may be marked as requiring those capabilities:
> RUNTIME_RX_QUEUE_SETUP
> RUNTIME_TX_QUEUE_SETUP
> RXQ_SHARE
> FLOW_RULE_KEEP
> FLOW_SHARED_OBJECT_KEEP
>
> These names are copy pasted from the existing DeviceCapabilitiesFlag
> class. Dynamic addition of Enum members runs into problems with typing
> (mypy doesn't know about the members) and documentation generation
> (Sphinx doesn't know about the members).
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 00/12] dts: add test skipping based on capabilities
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
` (11 preceding siblings ...)
2024-08-21 14:53 ` [PATCH v3 12/12] dts: add NIC capabilities from show port info Juraj Linkeš
@ 2024-08-26 17:25 ` Jeremy Spewock
12 siblings, 0 replies; 75+ messages in thread
From: Jeremy Spewock @ 2024-08-26 17:25 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
Hey Juraj,
Thanks for the series! This is definitely a large shift in how the
framework operates, but I think a lot of these changes are hugely
helpful and the code is very well written in general. I left some
comments mostly about places where I think some things could be a
little more clear, and one about a functional difference that I think
could be useful, but let me know what you think.
Also, I tried to apply this patch to help with the review process but
I couldn't get it to work. I think this is mainly due to the fact that
this uses the MTU updating commit on main, and my version of that
patch is far behind main right now, so we probably just resolved
conflicts differently somehow. I will work on updating that series now
and break the MTU patch into its own series to make it easier to use.
Thanks,
Jeremy
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 01/12] dts: fix default device error handling mode
2024-08-21 14:53 ` [PATCH v3 01/12] dts: fix default device error handling mode Juraj Linkeš
2024-08-26 16:42 ` Jeremy Spewock
@ 2024-08-27 16:15 ` Dean Marx
2024-08-27 20:09 ` Nicholas Pratte
2 siblings, 0 replies; 75+ messages in thread
From: Dean Marx @ 2024-08-27 16:15 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 367 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> The device_error_handling_mode of testpmd port may not be present, e.g.
> in VM ports.
>
> Fixes: 61d5bc9bf974 ("dts: add port info command to testpmd shell")
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 675 bytes --]
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 02/12] dts: add the aenum dependency
2024-08-21 14:53 ` [PATCH v3 02/12] dts: add the aenum dependency Juraj Linkeš
2024-08-26 16:42 ` Jeremy Spewock
@ 2024-08-27 16:28 ` Dean Marx
2024-08-27 20:21 ` Nicholas Pratte
2 siblings, 0 replies; 75+ messages in thread
From: Dean Marx @ 2024-08-27 16:28 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 612 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> Regular Python enumerations create only one instance for members with
> the same value, such as:
> class MyEnum(Enum):
> foo = 1
> bar = 1
>
> MyEnum.foo and MyEnum.bar are aliases that return the same instance.
>
> DTS needs to return different instances in the above scenario so that we
> can map capabilities with different names to the same function that
> retrieves the capabilities.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 929 bytes --]
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 08/12] dts: add NIC capability support
2024-08-21 14:53 ` [PATCH v3 08/12] dts: add NIC capability support Juraj Linkeš
2024-08-26 17:11 ` Jeremy Spewock
@ 2024-08-27 16:36 ` Jeremy Spewock
2024-09-18 12:58 ` Juraj Linkeš
2024-09-03 19:13 ` Dean Marx
2 siblings, 1 reply; 75+ messages in thread
From: Jeremy Spewock @ 2024-08-27 16:36 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
<snip>
> diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py
> index 8899f07f76..9a79e6ebb3 100644
> --- a/dts/framework/testbed_model/capability.py
> +++ b/dts/framework/testbed_model/capability.py
> @@ -5,14 +5,40 @@
<snip>
> + @classmethod
> + def get_supported_capabilities(
> + cls, sut_node: SutNode, topology: "Topology"
> + ) -> set["DecoratedNicCapability"]:
> + """Overrides :meth:`~Capability.get_supported_capabilities`.
> +
> + The capabilities are first sorted by decorators, then reduced into a single function which
> + is then passed to the decorator. This way we only execute each decorator only once.
> + """
> + supported_conditional_capabilities: set["DecoratedNicCapability"] = set()
> + logger = get_dts_logger(f"{sut_node.name}.{cls.__name__}")
> + if topology.type is Topology.type.no_link:
As a follow-up, I didn't notice this during my initial review, but in
testing this line was throwing attribute errors for me due to Topology
not having an attribute named `type`. I think this was because of
`Topology.type.no_link` since this attribute isn't initialized on the
class itself. I fixed this by just replacing it with
`TopologyType.no_link` locally.
> + logger.debug(
> + "No links available in the current topology, not getting NIC capabilities."
> + )
> + return supported_conditional_capabilities
> + logger.debug(
> + f"Checking which NIC capabilities from {cls.capabilities_to_check} are supported."
> + )
> + if cls.capabilities_to_check:
> + capabilities_to_check_map = cls._get_decorated_capabilities_map()
> + with TestPmdShell(sut_node, privileged=True) as testpmd_shell:
> + for conditional_capability_fn, capabilities in capabilities_to_check_map.items():
> + supported_capabilities: set[NicCapability] = set()
<snip>
>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 01/12] dts: fix default device error handling mode
2024-08-21 14:53 ` [PATCH v3 01/12] dts: fix default device error handling mode Juraj Linkeš
2024-08-26 16:42 ` Jeremy Spewock
2024-08-27 16:15 ` Dean Marx
@ 2024-08-27 20:09 ` Nicholas Pratte
2 siblings, 0 replies; 75+ messages in thread
From: Nicholas Pratte @ 2024-08-27 20:09 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
> The device_error_handling_mode of testpmd port may not be present, e.g.
> in VM ports.
>
> Fixes: 61d5bc9bf974 ("dts: add port info command to testpmd shell")
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
Reviewed-by: Nicholas Pratte <npratte@iol.unh.edu>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 02/12] dts: add the aenum dependency
2024-08-21 14:53 ` [PATCH v3 02/12] dts: add the aenum dependency Juraj Linkeš
2024-08-26 16:42 ` Jeremy Spewock
2024-08-27 16:28 ` Dean Marx
@ 2024-08-27 20:21 ` Nicholas Pratte
2 siblings, 0 replies; 75+ messages in thread
From: Nicholas Pratte @ 2024-08-27 20:21 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
> Regular Python enumerations create only one instance for members with
> the same value, such as:
> class MyEnum(Enum):
> foo = 1
> bar = 1
>
> MyEnum.foo and MyEnum.bar are aliases that return the same instance.
>
> DTS needs to return different instances in the above scenario so that we
> can map capabilities with different names to the same function that
> retrieves the capabilities.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Nicholas Pratte <npratte@iol.unh.edu>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 11/12] dts: add Rx offload capabilities
2024-08-21 14:53 ` [PATCH v3 11/12] dts: add Rx offload capabilities Juraj Linkeš
2024-08-26 17:24 ` Jeremy Spewock
@ 2024-08-28 17:44 ` Jeremy Spewock
2024-08-29 15:40 ` Jeremy Spewock
2024-09-03 19:49 ` Dean Marx
2 siblings, 1 reply; 75+ messages in thread
From: Jeremy Spewock @ 2024-08-28 17:44 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
<snip>
> diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
> index 48c31124d1..f83569669e 100644
> --- a/dts/framework/remote_session/testpmd_shell.py
> +++ b/dts/framework/remote_session/testpmd_shell.py
> @@ -659,6 +659,103 @@ class TestPmdPortStats(TextParser):
> tx_bps: int = field(metadata=TextParser.find_int(r"Tx-bps:\s+(\d+)"))
>
>
> +class RxOffloadCapability(Flag):
> + """Rx offload capabilities of a device."""
> +
> + #:
> + RX_OFFLOAD_VLAN_STRIP = auto()
One other thought that I had about this; was there a specific reason
that you decided to prefix all of these with `RX_OFFLOAD_`? I am
working on a test suite right now that uses both RX and TX offloads
and thought that it would be a great use of capabilities, so I am
working on adding a TxOffloadCapability flag as well and, since the
output is essentially the same, it made a lot of sense to make it a
sibling class of this one with similar parsing functionality. In what
I was writing, I found it much easier to remove this prefix so that
the parsing method can be the same for both RX and TX, and I didn't
have to restate some options that are shared between both (like
IPv4_CKSUM, UDP_CKSUM, etc.). Is there a reason you can think of why
removing this prefix is a bad idea? Hopefully I will have a patch out
soon that shows this extension that I've made so that you can see
in-code what I was thinking.
> + #: Device supports L3 checksum offload.
> + RX_OFFLOAD_IPV4_CKSUM = auto()
> + #: Device supports L4 checksum offload.
> + RX_OFFLOAD_UDP_CKSUM = auto()
> + #: Device supports L4 checksum offload.
> + RX_OFFLOAD_TCP_CKSUM = auto()
> + #: Device supports Large Receive Offload.
> + RX_OFFLOAD_TCP_LRO = auto()
> + #: Device supports QinQ (queue in queue) offload.
> + RX_OFFLOAD_QINQ_STRIP = auto()
> + #: Device supports inner packet L3 checksum.
> + RX_OFFLOAD_OUTER_IPV4_CKSUM = auto()
> + #: Device supports MACsec.
> + RX_OFFLOAD_MACSEC_STRIP = auto()
> + #: Device supports filtering of a VLAN Tag identifier.
> + RX_OFFLOAD_VLAN_FILTER = 1 << 9
> + #: Device supports VLAN offload.
> + RX_OFFLOAD_VLAN_EXTEND = auto()
> + #: Device supports receiving segmented mbufs.
> + RX_OFFLOAD_SCATTER = 1 << 13
> + #: Device supports Timestamp.
> + RX_OFFLOAD_TIMESTAMP = auto()
> + #: Device supports crypto processing while packet is received in NIC.
> + RX_OFFLOAD_SECURITY = auto()
> + #: Device supports CRC stripping.
> + RX_OFFLOAD_KEEP_CRC = auto()
> + #: Device supports L4 checksum offload.
> + RX_OFFLOAD_SCTP_CKSUM = auto()
> + #: Device supports inner packet L4 checksum.
> + RX_OFFLOAD_OUTER_UDP_CKSUM = auto()
> + #: Device supports RSS hashing.
> + RX_OFFLOAD_RSS_HASH = auto()
> + #: Device supports
> + RX_OFFLOAD_BUFFER_SPLIT = auto()
> + #: Device supports all checksum capabilities.
> + RX_OFFLOAD_CHECKSUM = RX_OFFLOAD_IPV4_CKSUM | RX_OFFLOAD_UDP_CKSUM | RX_OFFLOAD_TCP_CKSUM
> + #: Device supports all VLAN capabilities.
> + RX_OFFLOAD_VLAN = (
> + RX_OFFLOAD_VLAN_STRIP
> + | RX_OFFLOAD_VLAN_FILTER
> + | RX_OFFLOAD_VLAN_EXTEND
> + | RX_OFFLOAD_QINQ_STRIP
> + )
<snip>
>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 03/12] dts: add test case decorators
2024-08-21 14:53 ` [PATCH v3 03/12] dts: add test case decorators Juraj Linkeš
2024-08-26 16:50 ` Jeremy Spewock
@ 2024-08-28 20:09 ` Dean Marx
2024-08-30 15:50 ` Nicholas Pratte
2 siblings, 0 replies; 75+ messages in thread
From: Dean Marx @ 2024-08-28 20:09 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 779 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> Add decorators for functional and performance test cases. These
> decorators add attributes to the decorated test cases.
>
> With the addition of decorators, we change the test case discovery
> mechanism from looking at test case names according to a regex to simply
> checking an attribute of the function added with one of the decorators.
>
> The decorators allow us to add further variables to test cases.
>
> Also move the test case filtering to TestSuite while changing the
> mechanism to separate the logic in a more sensible manner.
>
> Bugzilla ID: 1460
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 1098 bytes --]
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 04/12] dts: add mechanism to skip test cases or suites
2024-08-21 14:53 ` [PATCH v3 04/12] dts: add mechanism to skip test cases or suites Juraj Linkeš
2024-08-26 16:52 ` Jeremy Spewock
@ 2024-08-28 20:37 ` Dean Marx
1 sibling, 0 replies; 75+ messages in thread
From: Dean Marx @ 2024-08-28 20:37 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 714 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> If a test case is not relevant to the testing environment (such as when
> a NIC doesn't support a tested feature), the framework should skip it.
> The mechanism is a skeleton without actual logic that would set a test
> case or suite to be skipped.
>
> The mechanism uses a protocol to extend test suites and test cases with
> additional attributes that track whether the test case or suite should
> be skipped the reason for skipping it.
>
> Also update the results module with the new SKIP result.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 1029 bytes --]
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 05/12] dts: add support for simpler topologies
2024-08-21 14:53 ` [PATCH v3 05/12] dts: add support for simpler topologies Juraj Linkeš
2024-08-26 16:54 ` Jeremy Spewock
@ 2024-08-28 20:56 ` Dean Marx
1 sibling, 0 replies; 75+ messages in thread
From: Dean Marx @ 2024-08-28 20:56 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 728 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> We currently assume there are two links between the SUT and TG nodes,
> but that's too strict, even for some of the already existing test cases.
> Add support for topologies with less than two links.
>
> For topologies with no links, dummy ports are used. The expectation is
> that test suites or cases that don't require any links won't be using
> methods that use ports. Any test suites or cases requiring links will be
> skipped in topologies with no links, but this feature is not implemented
> in this commit.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 1048 bytes --]
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 11/12] dts: add Rx offload capabilities
2024-08-28 17:44 ` Jeremy Spewock
@ 2024-08-29 15:40 ` Jeremy Spewock
2024-09-18 14:27 ` Juraj Linkeš
0 siblings, 1 reply; 75+ messages in thread
From: Jeremy Spewock @ 2024-08-29 15:40 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 28, 2024 at 1:44 PM Jeremy Spewock <jspewock@iol.unh.edu> wrote:
>
> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> <snip>
> > diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
> > index 48c31124d1..f83569669e 100644
> > --- a/dts/framework/remote_session/testpmd_shell.py
> > +++ b/dts/framework/remote_session/testpmd_shell.py
> > @@ -659,6 +659,103 @@ class TestPmdPortStats(TextParser):
> > tx_bps: int = field(metadata=TextParser.find_int(r"Tx-bps:\s+(\d+)"))
> >
> >
> > +class RxOffloadCapability(Flag):
> > + """Rx offload capabilities of a device."""
> > +
> > + #:
> > + RX_OFFLOAD_VLAN_STRIP = auto()
>
> One other thought that I had about this; was there a specific reason
> that you decided to prefix all of these with `RX_OFFLOAD_`? I am
> working on a test suite right now that uses both RX and TX offloads
> and thought that it would be a great use of capabilities, so I am
> working on adding a TxOffloadCapability flag as well and, since the
> output is essentially the same, it made a lot of sense to make it a
> sibling class of this one with similar parsing functionality. In what
> I was writing, I found it much easier to remove this prefix so that
> the parsing method can be the same for both RX and TX, and I didn't
> have to restate some options that are shared between both (like
> IPv4_CKSUM, UDP_CKSUM, etc.). Is there a reason you can think of why
> removing this prefix is a bad idea? Hopefully I will have a patch out
> soon that shows this extension that I've made so that you can see
> in-code what I was thinking.
I see now that you actually already answered this question, I was just
looking too much at that piece of code, and clearly not looking
further down at the helper-method mapping or the commit message that
you left :).
"The Flag members correspond to NIC
capability names so a convenience function that looks for the supported
Flags in a testpmd output is also added."
Having it prefixed with RX_OFFLOAD_ in NicCapability makes a lot of
sense since it is more explicit. Since there is a good reason to have
it like this, then the redundancy makes sense I think. There are some
ways to potentially avoid this like creating a StrFlag class that
overrides the __str__ method, or something like an additional type
that would contain a toString method, but it feels very situational
and specific to this one use-case so it probably isn't going to be
super valuable. Another thing I could think of to do would be allowing
the user to pass in a function or something to the helper-method that
mapped Flag names to their respective NicCapability name, or just
doing it in the method that gets the offloads instead of using a
helper at all, but this also just makes it more complicated and maybe
it isn't worth it.
I apologize for asking you about something that you already explained,
but maybe something we can get out of this is that, since these names
have to be consistent, it might be worth putting that in the
doc-strings of the flag for when people try to make further expansions
or changes in the future. Or it could also be generally clear that
flags used for capabilities should follow this idea, let me know what
you think.
>
> > + #: Device supports L3 checksum offload.
> > + RX_OFFLOAD_IPV4_CKSUM = auto()
> > + #: Device supports L4 checksum offload.
> > + RX_OFFLOAD_UDP_CKSUM = auto()
> > + #: Device supports L4 checksum offload.
> > + RX_OFFLOAD_TCP_CKSUM = auto()
> > + #: Device supports Large Receive Offload.
> > + RX_OFFLOAD_TCP_LRO = auto()
> > + #: Device supports QinQ (queue in queue) offload.
> > + RX_OFFLOAD_QINQ_STRIP = auto()
> > + #: Device supports inner packet L3 checksum.
> > + RX_OFFLOAD_OUTER_IPV4_CKSUM = auto()
> > + #: Device supports MACsec.
> > + RX_OFFLOAD_MACSEC_STRIP = auto()
> > + #: Device supports filtering of a VLAN Tag identifier.
> > + RX_OFFLOAD_VLAN_FILTER = 1 << 9
> > + #: Device supports VLAN offload.
> > + RX_OFFLOAD_VLAN_EXTEND = auto()
> > + #: Device supports receiving segmented mbufs.
> > + RX_OFFLOAD_SCATTER = 1 << 13
> > + #: Device supports Timestamp.
> > + RX_OFFLOAD_TIMESTAMP = auto()
> > + #: Device supports crypto processing while packet is received in NIC.
> > + RX_OFFLOAD_SECURITY = auto()
> > + #: Device supports CRC stripping.
> > + RX_OFFLOAD_KEEP_CRC = auto()
> > + #: Device supports L4 checksum offload.
> > + RX_OFFLOAD_SCTP_CKSUM = auto()
> > + #: Device supports inner packet L4 checksum.
> > + RX_OFFLOAD_OUTER_UDP_CKSUM = auto()
> > + #: Device supports RSS hashing.
> > + RX_OFFLOAD_RSS_HASH = auto()
> > + #: Device supports
> > + RX_OFFLOAD_BUFFER_SPLIT = auto()
> > + #: Device supports all checksum capabilities.
> > + RX_OFFLOAD_CHECKSUM = RX_OFFLOAD_IPV4_CKSUM | RX_OFFLOAD_UDP_CKSUM | RX_OFFLOAD_TCP_CKSUM
> > + #: Device supports all VLAN capabilities.
> > + RX_OFFLOAD_VLAN = (
> > + RX_OFFLOAD_VLAN_STRIP
> > + | RX_OFFLOAD_VLAN_FILTER
> > + | RX_OFFLOAD_VLAN_EXTEND
> > + | RX_OFFLOAD_QINQ_STRIP
> > + )
> <snip>
> >
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 03/12] dts: add test case decorators
2024-08-21 14:53 ` [PATCH v3 03/12] dts: add test case decorators Juraj Linkeš
2024-08-26 16:50 ` Jeremy Spewock
2024-08-28 20:09 ` Dean Marx
@ 2024-08-30 15:50 ` Nicholas Pratte
2 siblings, 0 replies; 75+ messages in thread
From: Nicholas Pratte @ 2024-08-30 15:50 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, dmarx, alex.chapman, dev
Reviewed-by: Nicholas Pratte <npratte@iol.unh.edu>
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
> Add decorators for functional and performance test cases. These
> decorators add attributes to the decorated test cases.
>
> With the addition of decorators, we change the test case discovery
> mechanism from looking at test case names according to a regex to simply
> checking an attribute of the function added with one of the decorators.
>
> The decorators allow us to add further variables to test cases.
>
> Also move the test case filtering to TestSuite while changing the
> mechanism to separate the logic in a more sensible manner.
>
> Bugzilla ID: 1460
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
> dts/framework/runner.py | 93 ++++------------
> dts/framework/test_result.py | 5 +-
> dts/framework/test_suite.py | 125 +++++++++++++++++++++-
> dts/tests/TestSuite_hello_world.py | 8 +-
> dts/tests/TestSuite_os_udp.py | 3 +-
> dts/tests/TestSuite_pmd_buffer_scatter.py | 3 +-
> dts/tests/TestSuite_smoke_tests.py | 6 +-
> 7 files changed, 160 insertions(+), 83 deletions(-)
>
> diff --git a/dts/framework/runner.py b/dts/framework/runner.py
> index 6b6f6a05f5..525f119ab6 100644
> --- a/dts/framework/runner.py
> +++ b/dts/framework/runner.py
> @@ -20,11 +20,10 @@
> import importlib
> import inspect
> import os
> -import re
> import sys
> from pathlib import Path
> -from types import FunctionType
> -from typing import Iterable, Sequence
> +from types import MethodType
> +from typing import Iterable
>
> from framework.testbed_model.sut_node import SutNode
> from framework.testbed_model.tg_node import TGNode
> @@ -53,7 +52,7 @@
> TestSuiteResult,
> TestSuiteWithCases,
> )
> -from .test_suite import TestSuite
> +from .test_suite import TestCase, TestSuite
>
>
> class DTSRunner:
> @@ -232,9 +231,9 @@ def _get_test_suites_with_cases(
>
> for test_suite_config in test_suite_configs:
> test_suite_class = self._get_test_suite_class(test_suite_config.test_suite)
> - test_cases = []
> - func_test_cases, perf_test_cases = self._filter_test_cases(
> - test_suite_class, test_suite_config.test_cases
> + test_cases: list[type[TestCase]] = []
> + func_test_cases, perf_test_cases = test_suite_class.get_test_cases(
> + test_suite_config.test_cases
> )
> if func:
> test_cases.extend(func_test_cases)
> @@ -309,57 +308,6 @@ def is_test_suite(object) -> bool:
> f"Couldn't find any valid test suites in {test_suite_module.__name__}."
> )
>
> - def _filter_test_cases(
> - self, test_suite_class: type[TestSuite], test_cases_to_run: Sequence[str]
> - ) -> tuple[list[FunctionType], list[FunctionType]]:
> - """Filter `test_cases_to_run` from `test_suite_class`.
> -
> - There are two rounds of filtering if `test_cases_to_run` is not empty.
> - The first filters `test_cases_to_run` from all methods of `test_suite_class`.
> - Then the methods are separated into functional and performance test cases.
> - If a method matches neither the functional nor performance name prefix, it's an error.
> -
> - Args:
> - test_suite_class: The class of the test suite.
> - test_cases_to_run: Test case names to filter from `test_suite_class`.
> - If empty, return all matching test cases.
> -
> - Returns:
> - A list of test case methods that should be executed.
> -
> - Raises:
> - ConfigurationError: If a test case from `test_cases_to_run` is not found
> - or it doesn't match either the functional nor performance name prefix.
> - """
> - func_test_cases = []
> - perf_test_cases = []
> - name_method_tuples = inspect.getmembers(test_suite_class, inspect.isfunction)
> - if test_cases_to_run:
> - name_method_tuples = [
> - (name, method) for name, method in name_method_tuples if name in test_cases_to_run
> - ]
> - if len(name_method_tuples) < len(test_cases_to_run):
> - missing_test_cases = set(test_cases_to_run) - {
> - name for name, _ in name_method_tuples
> - }
> - raise ConfigurationError(
> - f"Test cases {missing_test_cases} not found among methods "
> - f"of {test_suite_class.__name__}."
> - )
> -
> - for test_case_name, test_case_method in name_method_tuples:
> - if re.match(self._func_test_case_regex, test_case_name):
> - func_test_cases.append(test_case_method)
> - elif re.match(self._perf_test_case_regex, test_case_name):
> - perf_test_cases.append(test_case_method)
> - elif test_cases_to_run:
> - raise ConfigurationError(
> - f"Method '{test_case_name}' matches neither "
> - f"a functional nor a performance test case name."
> - )
> -
> - return func_test_cases, perf_test_cases
> -
> def _connect_nodes_and_run_test_run(
> self,
> sut_nodes: dict[str, SutNode],
> @@ -607,7 +555,7 @@ def _run_test_suite(
> def _execute_test_suite(
> self,
> test_suite: TestSuite,
> - test_cases: Iterable[FunctionType],
> + test_cases: Iterable[type[TestCase]],
> test_suite_result: TestSuiteResult,
> ) -> None:
> """Execute all `test_cases` in `test_suite`.
> @@ -618,29 +566,29 @@ def _execute_test_suite(
>
> Args:
> test_suite: The test suite object.
> - test_cases: The list of test case methods.
> + test_cases: The list of test case functions.
> test_suite_result: The test suite level result object associated
> with the current test suite.
> """
> self._logger.set_stage(DtsStage.test_suite)
> - for test_case_method in test_cases:
> - test_case_name = test_case_method.__name__
> + for test_case in test_cases:
> + test_case_name = test_case.__name__
> test_case_result = test_suite_result.add_test_case(test_case_name)
> all_attempts = SETTINGS.re_run + 1
> attempt_nr = 1
> - self._run_test_case(test_suite, test_case_method, test_case_result)
> + self._run_test_case(test_suite, test_case, test_case_result)
> while not test_case_result and attempt_nr < all_attempts:
> attempt_nr += 1
> self._logger.info(
> f"Re-running FAILED test case '{test_case_name}'. "
> f"Attempt number {attempt_nr} out of {all_attempts}."
> )
> - self._run_test_case(test_suite, test_case_method, test_case_result)
> + self._run_test_case(test_suite, test_case, test_case_result)
>
> def _run_test_case(
> self,
> test_suite: TestSuite,
> - test_case_method: FunctionType,
> + test_case: type[TestCase],
> test_case_result: TestCaseResult,
> ) -> None:
> """Setup, execute and teardown `test_case_method` from `test_suite`.
> @@ -649,11 +597,11 @@ def _run_test_case(
>
> Args:
> test_suite: The test suite object.
> - test_case_method: The test case method.
> + test_case: The test case function.
> test_case_result: The test case level result object associated
> with the current test case.
> """
> - test_case_name = test_case_method.__name__
> + test_case_name = test_case.__name__
>
> try:
> # run set_up function for each case
> @@ -668,7 +616,7 @@ def _run_test_case(
>
> else:
> # run test case if setup was successful
> - self._execute_test_case(test_suite, test_case_method, test_case_result)
> + self._execute_test_case(test_suite, test_case, test_case_result)
>
> finally:
> try:
> @@ -686,21 +634,22 @@ def _run_test_case(
> def _execute_test_case(
> self,
> test_suite: TestSuite,
> - test_case_method: FunctionType,
> + test_case: type[TestCase],
> test_case_result: TestCaseResult,
> ) -> None:
> """Execute `test_case_method` from `test_suite`, record the result and handle failures.
>
> Args:
> test_suite: The test suite object.
> - test_case_method: The test case method.
> + test_case: The test case function.
> test_case_result: The test case level result object associated
> with the current test case.
> """
> - test_case_name = test_case_method.__name__
> + test_case_name = test_case.__name__
> try:
> self._logger.info(f"Starting test case execution: {test_case_name}")
> - test_case_method(test_suite)
> + # Explicit method binding is required, otherwise mypy complains
> + MethodType(test_case, test_suite)()
> test_case_result.update(Result.PASS)
> self._logger.info(f"Test case execution PASSED: {test_case_name}")
>
> diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
> index 5694a2482b..b1ca584523 100644
> --- a/dts/framework/test_result.py
> +++ b/dts/framework/test_result.py
> @@ -27,7 +27,6 @@
> from collections.abc import MutableSequence
> from dataclasses import dataclass
> from enum import Enum, auto
> -from types import FunctionType
> from typing import Union
>
> from .config import (
> @@ -44,7 +43,7 @@
> from .exception import DTSError, ErrorSeverity
> from .logger import DTSLogger
> from .settings import SETTINGS
> -from .test_suite import TestSuite
> +from .test_suite import TestCase, TestSuite
>
>
> @dataclass(slots=True, frozen=True)
> @@ -63,7 +62,7 @@ class is to hold a subset of test cases (which could be all test cases) because
> """
>
> test_suite_class: type[TestSuite]
> - test_cases: list[FunctionType]
> + test_cases: list[type[TestCase]]
>
> def create_config(self) -> TestSuiteConfig:
> """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
> diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
> index 694b2eba65..b4ee0f9039 100644
> --- a/dts/framework/test_suite.py
> +++ b/dts/framework/test_suite.py
> @@ -13,8 +13,11 @@
> * Test case verification.
> """
>
> +import inspect
> +from collections.abc import Callable, Sequence
> +from enum import Enum, auto
> from ipaddress import IPv4Interface, IPv6Interface, ip_interface
> -from typing import ClassVar, Union
> +from typing import ClassVar, Protocol, TypeVar, Union, cast
>
> from scapy.layers.inet import IP # type: ignore[import-untyped]
> from scapy.layers.l2 import Ether # type: ignore[import-untyped]
> @@ -27,7 +30,7 @@
> PacketFilteringConfig,
> )
>
> -from .exception import TestCaseVerifyError
> +from .exception import ConfigurationError, TestCaseVerifyError
> from .logger import DTSLogger, get_dts_logger
> from .utils import get_packet_summaries
>
> @@ -120,6 +123,68 @@ def _process_links(self) -> None:
> ):
> self._port_links.append(PortLink(sut_port=sut_port, tg_port=tg_port))
>
> + @classmethod
> + def get_test_cases(
> + cls, test_case_sublist: Sequence[str] | None = None
> + ) -> tuple[set[type["TestCase"]], set[type["TestCase"]]]:
> + """Filter `test_case_subset` from this class.
> +
> + Test cases are regular (or bound) methods decorated with :func:`func_test`
> + or :func:`perf_test`.
> +
> + Args:
> + test_case_sublist: Test case names to filter from this class.
> + If empty or :data:`None`, return all test cases.
> +
> + Returns:
> + The filtered test case functions. This method returns functions as opposed to methods,
> + as methods are bound to instances and this method only has access to the class.
> +
> + Raises:
> + ConfigurationError: If a test case from `test_case_subset` is not found.
> + """
> +
> + def is_test_case(function: Callable) -> bool:
> + if inspect.isfunction(function):
> + # TestCase is not used at runtime, so we can't use isinstance() with `function`.
> + # But function.test_type exists.
> + if hasattr(function, "test_type"):
> + return isinstance(function.test_type, TestCaseType)
> + return False
> +
> + if test_case_sublist is None:
> + test_case_sublist = []
> +
> + # the copy is needed so that the condition "elif test_case_sublist" doesn't
> + # change mid-cycle
> + test_case_sublist_copy = list(test_case_sublist)
> + func_test_cases = set()
> + perf_test_cases = set()
> +
> + for test_case_name, test_case_function in inspect.getmembers(cls, is_test_case):
> + if test_case_name in test_case_sublist_copy:
> + # if test_case_sublist_copy is non-empty, remove the found test case
> + # so that we can look at the remainder at the end
> + test_case_sublist_copy.remove(test_case_name)
> + elif test_case_sublist:
> + # if the original list is not empty (meaning we're filtering test cases),
> + # we're dealing with a test case we would've
> + # removed in the other branch; since we didn't, we don't want to run it
> + continue
> +
> + match test_case_function.test_type:
> + case TestCaseType.PERFORMANCE:
> + perf_test_cases.add(test_case_function)
> + case TestCaseType.FUNCTIONAL:
> + func_test_cases.add(test_case_function)
> +
> + if test_case_sublist_copy:
> + raise ConfigurationError(
> + f"Test cases {test_case_sublist_copy} not found among functions of {cls.__name__}."
> + )
> +
> + return func_test_cases, perf_test_cases
> +
> def set_up_suite(self) -> None:
> """Set up test fixtures common to all test cases.
>
> @@ -365,3 +430,59 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool:
> if received_packet.src != expected_packet.src or received_packet.dst != expected_packet.dst:
> return False
> return True
> +
> +
> +#: The generic type for a method of an instance of TestSuite
> +TestSuiteMethodType = TypeVar("TestSuiteMethodType", bound=Callable[[TestSuite], None])
> +
> +
> +class TestCaseType(Enum):
> + """The types of test cases."""
> +
> + #:
> + FUNCTIONAL = auto()
> + #:
> + PERFORMANCE = auto()
> +
> +
> +class TestCase(Protocol[TestSuiteMethodType]):
> + """Definition of the test case type for static type checking purposes.
> +
> + The type is applied to test case functions through a decorator, which casts the decorated
> + test case function to :class:`TestCase` and sets common variables.
> + """
> +
> + #:
> + test_type: ClassVar[TestCaseType]
> + #: necessary for mypy so that it can treat this class as the function it's shadowing
> + __call__: TestSuiteMethodType
> +
> + @classmethod
> + def make_decorator(
> + cls, test_case_type: TestCaseType
> + ) -> Callable[[TestSuiteMethodType], type["TestCase"]]:
> + """Create a decorator for test suites.
> +
> + The decorator casts the decorated function as :class:`TestCase`,
> + sets it as `test_case_type`
> + and initializes common variables defined in :class:`RequiresCapabilities`.
> +
> + Args:
> + test_case_type: Either a functional or performance test case.
> +
> + Returns:
> + The decorator of a functional or performance test case.
> + """
> +
> + def _decorator(func: TestSuiteMethodType) -> type[TestCase]:
> + test_case = cast(type[TestCase], func)
> + test_case.test_type = test_case_type
> + return test_case
> +
> + return _decorator
> +
> +
> +#: The decorator for functional test cases.
> +func_test: Callable = TestCase.make_decorator(TestCaseType.FUNCTIONAL)
> +#: The decorator for performance test cases.
> +perf_test: Callable = TestCase.make_decorator(TestCaseType.PERFORMANCE)
> diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
> index d958f99030..16d064ffeb 100644
> --- a/dts/tests/TestSuite_hello_world.py
> +++ b/dts/tests/TestSuite_hello_world.py
> @@ -8,7 +8,7 @@
> """
>
> from framework.remote_session.dpdk_shell import compute_eal_params
> -from framework.test_suite import TestSuite
> +from framework.test_suite import TestSuite, func_test
> from framework.testbed_model.cpu import (
> LogicalCoreCount,
> LogicalCoreCountFilter,
> @@ -27,7 +27,8 @@ def set_up_suite(self) -> None:
> """
> self.app_helloworld_path = self.sut_node.build_dpdk_app("helloworld")
>
> - def test_hello_world_single_core(self) -> None:
> + @func_test
> + def hello_world_single_core(self) -> None:
> """Single core test case.
>
> Steps:
> @@ -46,7 +47,8 @@ def test_hello_world_single_core(self) -> None:
> f"helloworld didn't start on lcore{lcores[0]}",
> )
>
> - def test_hello_world_all_cores(self) -> None:
> + @func_test
> + def hello_world_all_cores(self) -> None:
> """All cores test case.
>
> Steps:
> diff --git a/dts/tests/TestSuite_os_udp.py b/dts/tests/TestSuite_os_udp.py
> index a78bd74139..beaa5f425d 100644
> --- a/dts/tests/TestSuite_os_udp.py
> +++ b/dts/tests/TestSuite_os_udp.py
> @@ -10,7 +10,7 @@
> from scapy.layers.inet import IP, UDP # type: ignore[import-untyped]
> from scapy.layers.l2 import Ether # type: ignore[import-untyped]
>
> -from framework.test_suite import TestSuite
> +from framework.test_suite import TestSuite, func_test
>
>
> class TestOsUdp(TestSuite):
> @@ -26,6 +26,7 @@ def set_up_suite(self) -> None:
> self.sut_node.bind_ports_to_driver(for_dpdk=False)
> self.configure_testbed_ipv4()
>
> + @func_test
> def test_os_udp(self) -> None:
> """Basic UDP IPv4 traffic test case.
>
> diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
> index 0d8e101e5c..020fb0ab62 100644
> --- a/dts/tests/TestSuite_pmd_buffer_scatter.py
> +++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
> @@ -24,7 +24,7 @@
>
> from framework.params.testpmd import SimpleForwardingModes
> from framework.remote_session.testpmd_shell import TestPmdShell
> -from framework.test_suite import TestSuite
> +from framework.test_suite import TestSuite, func_test
>
>
> class TestPmdBufferScatter(TestSuite):
> @@ -123,6 +123,7 @@ def pmd_scatter(self, mbsize: int) -> None:
> f"{offset}.",
> )
>
> + @func_test
> def test_scatter_mbuf_2048(self) -> None:
> """Run the :meth:`pmd_scatter` test with `mbsize` set to 2048."""
> self.pmd_scatter(mbsize=2048)
> diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
> index c0b0e6bb00..94f90d9327 100644
> --- a/dts/tests/TestSuite_smoke_tests.py
> +++ b/dts/tests/TestSuite_smoke_tests.py
> @@ -17,7 +17,7 @@
> from framework.config import PortConfig
> from framework.remote_session.testpmd_shell import TestPmdShell
> from framework.settings import SETTINGS
> -from framework.test_suite import TestSuite
> +from framework.test_suite import TestSuite, func_test
> from framework.utils import REGEX_FOR_PCI_ADDRESS
>
>
> @@ -47,6 +47,7 @@ def set_up_suite(self) -> None:
> self.dpdk_build_dir_path = self.sut_node.remote_dpdk_build_dir
> self.nics_in_node = self.sut_node.config.ports
>
> + @func_test
> def test_unit_tests(self) -> None:
> """DPDK meson ``fast-tests`` unit tests.
>
> @@ -63,6 +64,7 @@ def test_unit_tests(self) -> None:
> privileged=True,
> )
>
> + @func_test
> def test_driver_tests(self) -> None:
> """DPDK meson ``driver-tests`` unit tests.
>
> @@ -91,6 +93,7 @@ def test_driver_tests(self) -> None:
> privileged=True,
> )
>
> + @func_test
> def test_devices_listed_in_testpmd(self) -> None:
> """Testpmd device discovery.
>
> @@ -108,6 +111,7 @@ def test_devices_listed_in_testpmd(self) -> None:
> "please check your configuration",
> )
>
> + @func_test
> def test_device_bound_to_driver(self) -> None:
> """Device driver in OS.
>
> --
> 2.34.1
>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 06/12] dst: add basic capability support
2024-08-21 14:53 ` [PATCH v3 06/12] dst: add basic capability support Juraj Linkeš
2024-08-26 16:56 ` Jeremy Spewock
@ 2024-09-03 16:03 ` Dean Marx
2024-09-05 9:51 ` Juraj Linkeš
1 sibling, 1 reply; 75+ messages in thread
From: Dean Marx @ 2024-09-03 16:03 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 897 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> A test case or suite may require certain capabilities to be present in
> the tested environment. Add the basic infrastructure for checking the
> support status of capabilities:
> * The Capability ABC defining the common capability API
> * Extension of the TestProtocol with required capabilities (each test
> suite or case stores the capabilities it requires)
> * Integration with the runner which calls the new APIs to get which
> capabilities are supported.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Looks all good to me, it was interesting to see how you've used abstract
methods in the Capability class. The only thing I noticed was it seems like
you wrote "dst" instead of "dts" in the commit message, otherwise:
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 1246 bytes --]
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 07/12] dts: add testpmd port information caching
2024-08-21 14:53 ` [PATCH v3 07/12] dts: add testpmd port information caching Juraj Linkeš
2024-08-26 16:56 ` Jeremy Spewock
@ 2024-09-03 16:12 ` Dean Marx
1 sibling, 0 replies; 75+ messages in thread
From: Dean Marx @ 2024-09-03 16:12 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 437 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> When using port information multiple times in a testpmd shell instance
> lifespan, it's desirable to not get the information each time, so
> caching is added. In case the information changes, there's a way to
> force the update.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 742 bytes --]
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 09/12] dts: add topology capability
2024-08-21 14:53 ` [PATCH v3 09/12] dts: add topology capability Juraj Linkeš
2024-08-26 17:13 ` Jeremy Spewock
@ 2024-09-03 17:50 ` Dean Marx
1 sibling, 0 replies; 75+ messages in thread
From: Dean Marx @ 2024-09-03 17:50 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 1495 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> Add support for marking test cases as requiring a certain topology. The
> default topology is a two link topology and the other supported
> topologies are one link and no link topologies.
>
> The TestProtocol of test suites and cases is extended with the topology
> type each test suite or case requires. Each test case starts out as
> requiring a two link topology and can be marked as requiring as
> topology directly (by decorating the test case) or through its test
> suite. If a test suite is decorated as requiring a certain topology, all
> its test cases are marked as such. If both test suite and a test case
> are decorated as requiring a topology, the test case cannot require a
> more complex topology than the whole suite (but it can require a less
> complex one). If a test suite is not decorated, this has no effect on
> required test case topology.
>
> Since the default topology is defined as a reference to one of the
> actual topologies, the NoAliasEnum from the aenum package is utilized,
> which removes the aliasing of Enums so that TopologyType.two_links and
> TopologyType.default are distinct. This is needed to distinguish between
> a user passed value and the default value being used (which is used when
> a test suite is or isn't decorated).
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 1832 bytes --]
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 10/12] doc: add DTS capability doc sources
2024-08-21 14:53 ` [PATCH v3 10/12] doc: add DTS capability doc sources Juraj Linkeš
2024-08-26 17:13 ` Jeremy Spewock
@ 2024-09-03 17:52 ` Dean Marx
1 sibling, 0 replies; 75+ messages in thread
From: Dean Marx @ 2024-09-03 17:52 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 259 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> Add new files to generate DTS API documentation from.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 550 bytes --]
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 12/12] dts: add NIC capabilities from show port info
2024-08-21 14:53 ` [PATCH v3 12/12] dts: add NIC capabilities from show port info Juraj Linkeš
2024-08-26 17:24 ` Jeremy Spewock
@ 2024-09-03 18:02 ` Dean Marx
1 sibling, 0 replies; 75+ messages in thread
From: Dean Marx @ 2024-09-03 18:02 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 719 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> Add the capabilities advertised by the testpmd command "show port info"
> so that test cases may be marked as requiring those capabilities:
> RUNTIME_RX_QUEUE_SETUP
> RUNTIME_TX_QUEUE_SETUP
> RXQ_SHARE
> FLOW_RULE_KEEP
> FLOW_SHARED_OBJECT_KEEP
>
> These names are copy pasted from the existing DeviceCapabilitiesFlag
> class. Dynamic addition of Enum members runs into problems with typing
> (mypy doesn't know about the members) and documentation generation
> (Sphinx doesn't know about the members).
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 1051 bytes --]
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 08/12] dts: add NIC capability support
2024-08-21 14:53 ` [PATCH v3 08/12] dts: add NIC capability support Juraj Linkeš
2024-08-26 17:11 ` Jeremy Spewock
2024-08-27 16:36 ` Jeremy Spewock
@ 2024-09-03 19:13 ` Dean Marx
2 siblings, 0 replies; 75+ messages in thread
From: Dean Marx @ 2024-09-03 19:13 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 1162 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> Some test cases or suites may be testing a NIC feature that is not
> supported on all NICs, so add support for marking test cases or suites
> as requiring NIC capabilities.
>
> The marking is done with a decorator, which populates the internal
> required_capabilities attribute of TestProtocol. The NIC capability
> itself is a wrapper around the NicCapability defined in testpmd_shell.
> The reason is twofold:
> 1. Enums cannot be extended and the class implements the methods of its
> abstract base superclass,
> 2. The class also stores an optional decorator function which is used
> before/after capability retrieval. This is needed because some
> capabilities may be advertised differently under different
> configuration.
>
> The decorator API is designed to be simple to use. The arguments passed
> to it are all from the testpmd shell. Everything else (even the actual
> capability object creation) is done internally.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 1501 bytes --]
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 11/12] dts: add Rx offload capabilities
2024-08-21 14:53 ` [PATCH v3 11/12] dts: add Rx offload capabilities Juraj Linkeš
2024-08-26 17:24 ` Jeremy Spewock
2024-08-28 17:44 ` Jeremy Spewock
@ 2024-09-03 19:49 ` Dean Marx
2024-09-18 13:59 ` Juraj Linkeš
2 siblings, 1 reply; 75+ messages in thread
From: Dean Marx @ 2024-09-03 19:49 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 1391 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> The scatter Rx offload capability is needed for the pmd_buffer_scatter
> test suite. The command that retrieves the capability is:
> show port <port_id> rx_offload capabilities
>
> The command also retrieves a lot of other capabilities (RX_OFFLOAD_*)
> which are all added into a Flag. The Flag members correspond to NIC
> capability names so a convenience function that looks for the supported
> Flags in a testpmd output is also added.
>
> The NIC capability names (mentioned above) are copy-pasted from the
> Flag. Dynamic addition of Enum members runs into problems with typing
> (mypy doesn't know about the members) and documentation generation
> (Sphinx doesn't know about the members).
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
<snip>
> + RX_OFFLOAD_VLAN_FILTER = 1 << 9
> + #: Device supports VLAN offload.
> + RX_OFFLOAD_VLAN_EXTEND = auto()
> + #: Device supports receiving segmented mbufs.
> + RX_OFFLOAD_SCATTER = 1 << 13
>
This was an interesting section, I'm not super familiar with bitwise
shifting in python flags so I figured I'd ask while it's in mind if there's
any specific reason for shifting these two flags? Not a critique of the
code, just genuinely curious.
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 1930 bytes --]
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 03/12] dts: add test case decorators
2024-08-26 16:50 ` Jeremy Spewock
@ 2024-09-05 8:07 ` Juraj Linkeš
2024-09-05 15:24 ` Jeremy Spewock
0 siblings, 1 reply; 75+ messages in thread
From: Juraj Linkeš @ 2024-09-05 8:07 UTC (permalink / raw)
To: Jeremy Spewock
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On 26. 8. 2024 18:50, Jeremy Spewock wrote:
> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> <snip>
>> class DTSRunner:
>> @@ -232,9 +231,9 @@ def _get_test_suites_with_cases(
>>
>> for test_suite_config in test_suite_configs:
>> test_suite_class = self._get_test_suite_class(test_suite_config.test_suite)
>> - test_cases = []
>> - func_test_cases, perf_test_cases = self._filter_test_cases(
>> - test_suite_class, test_suite_config.test_cases
>> + test_cases: list[type[TestCase]] = []
>
> If TestCase is just a class, why is the `type[]` in the annotation
> required? Are these not specific instances of the TestCase class? I
> figured they would need to be in order for you to run the specific
> test case methods. Maybe this has something to do with the class being
> a Protocol?
>
The *_test decorators return type[TestCase]. The functions (test
methods) are cast to type[TestCase] (which kinda makes them subclasses
of TestCase).
This was a suggestion from Luca and I took it as as. Maybe the functions
could be cast as instances of TestCase, but I didn't try that.
>> + func_test_cases, perf_test_cases = test_suite_class.get_test_cases(
>> + test_suite_config.test_cases
>> )
>> if func:
>> test_cases.extend(func_test_cases)
>> @@ -309,57 +308,6 @@ def is_test_suite(object) -> bool:
>> f"Couldn't find any valid test suites in {test_suite_module.__name__}."
>> )
>>
> <snip>
>> @@ -120,6 +123,68 @@ def _process_links(self) -> None:
>> ):
>> self._port_links.append(PortLink(sut_port=sut_port, tg_port=tg_port))
>>
>> + @classmethod
>> + def get_test_cases(
>> + cls, test_case_sublist: Sequence[str] | None = None
>> + ) -> tuple[set[type["TestCase"]], set[type["TestCase"]]]:
>> + """Filter `test_case_subset` from this class.
>> +
>> + Test cases are regular (or bound) methods decorated with :func:`func_test`
>> + or :func:`perf_test`.
>> +
>> + Args:
>> + test_case_sublist: Test case names to filter from this class.
>> + If empty or :data:`None`, return all test cases.
>> +
>> + Returns:
>> + The filtered test case functions. This method returns functions as opposed to methods,
>> + as methods are bound to instances and this method only has access to the class.
>> +
>> + Raises:
>> + ConfigurationError: If a test case from `test_case_subset` is not found.
>> + """
>> +
> <snip>
>> + for test_case_name, test_case_function in inspect.getmembers(cls, is_test_case):
>> + if test_case_name in test_case_sublist_copy:
>> + # if test_case_sublist_copy is non-empty, remove the found test case
>> + # so that we can look at the remainder at the end
>> + test_case_sublist_copy.remove(test_case_name)
>> + elif test_case_sublist:
>> + # if the original list is not empty (meaning we're filtering test cases),
>> + # we're dealing with a test case we would've
>
> I think this part of the comment about "we're dealing with a test case
> we would've removed in the other branch" confused me a little bit. It
> could just be a me thing, but I think this would have been more clear
> for me if it was something more like "The original list is not empty
> (meaning we're filtering test cases). Since we didn't remove this test
> case in the other branch, it doesn't match the filter and we don't
> want to run it."
>
We should remove any confusion. I'll change it - your wording sound good.
>> + # removed in the other branch; since we didn't, we don't want to run it
>> + continue
>> +
>> + match test_case_function.test_type:
>> + case TestCaseType.PERFORMANCE:
>> + perf_test_cases.add(test_case_function)
>> + case TestCaseType.FUNCTIONAL:
>> + func_test_cases.add(test_case_function)
>> +
>> + if test_case_sublist_copy:
>> + raise ConfigurationError(
>> + f"Test cases {test_case_sublist_copy} not found among functions of {cls.__name__}."
>> + )
>> +
>> + return func_test_cases, perf_test_cases
>> +
> <snip>
>> 2.34.1
>>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 04/12] dts: add mechanism to skip test cases or suites
2024-08-26 16:52 ` Jeremy Spewock
@ 2024-09-05 9:23 ` Juraj Linkeš
2024-09-05 15:26 ` Jeremy Spewock
0 siblings, 1 reply; 75+ messages in thread
From: Juraj Linkeš @ 2024-09-05 9:23 UTC (permalink / raw)
To: Jeremy Spewock
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On 26. 8. 2024 18:52, Jeremy Spewock wrote:
> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> <snip>
>> --- a/dts/framework/test_result.py
>> +++ b/dts/framework/test_result.py
>> @@ -75,6 +75,20 @@ def create_config(self) -> TestSuiteConfig:
>> test_cases=[test_case.__name__ for test_case in self.test_cases],
>> )
>>
>> + @property
>> + def skip(self) -> bool:
>> + """Skip the test suite if all test cases or the suite itself are to be skipped.
>> +
>> + Returns:
>> + :data:`True` if the test suite should be skipped, :data:`False` otherwise.
>> + """
>> + all_test_cases_skipped = True
>> + for test_case in self.test_cases:
>> + if not test_case.skip:
>> + all_test_cases_skipped = False
>> + break
>
> You could also potentially implement this using the built-in `all()`
> function. It would become a simple one-liner like
> `all_test_cases_skipped = all(test_case.skip for test_case in
> self.test_cases)`. That's probably short enough to even just put in
> the return statement though if you wanted to.
>
Good catch, I'll use any() here.
>> + return all_test_cases_skipped or self.test_suite_class.skip
>> +
>>
>> class Result(Enum):
>> """The possible states that a setup, a teardown or a test case may end up in."""
>> @@ -86,12 +100,12 @@ class Result(Enum):
>> #:
>> ERROR = auto()
>> #:
>> - SKIP = auto()
>> - #:
>> BLOCK = auto()
>> + #:
>> + SKIP = auto()
>>
>> def __bool__(self) -> bool:
>> - """Only PASS is True."""
>> + """Only :attr:`PASS` is True."""
>> return self is self.PASS
>>
>>
>> @@ -169,12 +183,13 @@ def update_setup(self, result: Result, error: Exception | None = None) -> None:
>> self.setup_result.result = result
>> self.setup_result.error = error
>>
>> - if result in [Result.BLOCK, Result.ERROR, Result.FAIL]:
>> - self.update_teardown(Result.BLOCK)
>> - self._block_result()
>> + if result != Result.PASS:
>> + result_to_mark = Result.BLOCK if result != Result.SKIP else Result.SKIP
>> + self.update_teardown(result_to_mark)
>> + self._mark_results(result_to_mark)
>>
>> - def _block_result(self) -> None:
>> - r"""Mark the result as :attr:`~Result.BLOCK`\ed.
>> + def _mark_results(self, result) -> None:
>
> Is it worth adding the type annotation for `result` here and to the
> other places where this is implemented? I guess it doesn't matter that
> much since it is a private method.
>
I didn't add it precisely because it's a private method and it's pretty
self explanatory.
>> + """Mark the result as well as the child result as `result`.
>
> Are these methods even marking their own result or only their
> children? It seems like it's only really updating the children
> recursively and its result would have already been updated before this
> was called.
>
It's supposed to be just their result which is actually the result of
the children in all but the TestCaseResult classes. Conceptually, each
results level should contains these:
1. the result of setup
2. the result of teardown
3. the result of the level itself (outside of setup and teardown)
The result of the level itself is what's supposed to be set here. The
thing is we're making the child results for non-test cases and the
result of the test cases for test cases. Maybe I only need to update the
docstring.
>>
>> The blocking of child results should be done in overloaded methods.
>> """
> <snip>
>>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 05/12] dts: add support for simpler topologies
2024-08-26 16:54 ` Jeremy Spewock
@ 2024-09-05 9:42 ` Juraj Linkeš
0 siblings, 0 replies; 75+ messages in thread
From: Juraj Linkeš @ 2024-09-05 9:42 UTC (permalink / raw)
To: Jeremy Spewock
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On 26. 8. 2024 18:54, Jeremy Spewock wrote:
> I just had one question below, otherwise:
>
> Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
>
> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> <snip>
>> diff --git a/dts/framework/testbed_model/topology.py b/dts/framework/testbed_model/topology.py
>> new file mode 100644
>> index 0000000000..19632ee890
>> --- /dev/null
>> +++ b/dts/framework/testbed_model/topology.py
> <snip>
>> +
>> +
>> +class TopologyType(IntEnum):
>> + """Supported topology types."""
>> +
>> + #: A topology with no Traffic Generator.
>> + no_link = 0
>> + #: A topology with one physical link between the SUT node and the TG node.
>> + one_link = 1
>> + #: A topology with two physical links between the Sut node and the TG node.
>> + two_links = 2
>> +
>> +
>> +class Topology:
>> + """Testbed topology.
>> +
>> + The topology contains ports processed into ingress and egress ports.
>> + It's assumed that port0 of the SUT node is connected to port0 of the TG node and so on.
>
> Do we need to make this assumption when you are comparing the port
> directly to its peer and matching the addresses? I think you could
> specify in conf.yaml that port 0 on the SUT is one of your ports and
> its peer is port 1 on the TG and because you do the matching, this
> would work fine.
>
Yes, the assumption is not adhered to yet. I guess I put this here
because we've been discussing this in the calls, but the actual code
doesn't use this. I'll remove this line.
>> + If there are no ports on a node, dummy ports (ports with no actual values) are stored.
>> + If there is only one link available, the ports of this link are stored
>> + as both ingress and egress ports.
>> +
>> + The dummy ports shouldn't be used. It's up to :class:`~framework.runner.DTSRunner`
>> + to ensure no test case or suite requiring actual links is executed
>> + when the topology prohibits it and up to the developers to make sure that test cases
>> + not requiring any links don't use any ports. Otherwise, the underlying methods
>> + using the ports will fail.
>> +
>> + Attributes:
>> + type: The type of the topology.
>> + tg_port_egress: The egress port of the TG node.
>> + sut_port_ingress: The ingress port of the SUT node.
>> + sut_port_egress: The egress port of the SUT node.
>> + tg_port_ingress: The ingress port of the TG node.
>> + """
>> +
>> + type: TopologyType
>> + tg_port_egress: Port
>> + sut_port_ingress: Port
>> + sut_port_egress: Port
>> + tg_port_ingress: Port
>> +
>> + def __init__(self, sut_ports: Iterable[Port], tg_ports: Iterable[Port]):
>> + """Create the topology from `sut_ports` and `tg_ports`.
>> +
>> + Args:
>> + sut_ports: The SUT node's ports.
>> + tg_ports: The TG node's ports.
>> + """
>> + port_links = []
>> + for sut_port in sut_ports:
>> + for tg_port in tg_ports:
>> + if (sut_port.identifier, sut_port.peer) == (
>> + tg_port.peer,
>> + tg_port.identifier,
>> + ):
>> + port_links.append(PortLink(sut_port=sut_port, tg_port=tg_port))
>> +
>> + self.type = TopologyType(len(port_links))
> <snip>
>>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 06/12] dst: add basic capability support
2024-08-26 16:56 ` Jeremy Spewock
@ 2024-09-05 9:50 ` Juraj Linkeš
2024-09-05 15:27 ` Jeremy Spewock
0 siblings, 1 reply; 75+ messages in thread
From: Juraj Linkeš @ 2024-09-05 9:50 UTC (permalink / raw)
To: Jeremy Spewock
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On 26. 8. 2024 18:56, Jeremy Spewock wrote:
> Just one comment about adding something to a doc-string, otherwise
> looks good to me:
>
> Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
>
> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> <snip>
>> diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
>> index 306b100bc6..b4b58ef348 100644
>> --- a/dts/framework/test_result.py
>> +++ b/dts/framework/test_result.py
>> @@ -25,10 +25,12 @@
>>
>> import os.path
>> from collections.abc import MutableSequence
>> -from dataclasses import dataclass
>> +from dataclasses import dataclass, field
>> from enum import Enum, auto
>> from typing import Union
>>
>> +from framework.testbed_model.capability import Capability
>> +
>> from .config import (
>> OS,
>> Architecture,
>> @@ -63,6 +65,12 @@ class is to hold a subset of test cases (which could be all test cases) because
>>
>> test_suite_class: type[TestSuite]
>> test_cases: list[type[TestCase]]
>> + required_capabilities: set[Capability] = field(default_factory=set, init=False)
>
> This should probably be added to the Attributes section of the
> doc-string for the class.
Ah, I missed this, thanks.
> When it's there, it might also be useful to
> explain that this is used by the runner to determine what capabilities
> need to be searched for to mark the suite for being skipped.
And also test cases.
> The only
> reason I think that would be useful is it helps differentiate this
> list of capabilities from the list of required capabilities that every
> test suite and test case has.
>
I want to add this:
The combined required capabilities of both the test suite and the subset
of test cases.
I think this makes it clear that it's different from the individual
required capabilities of test suites and cases. Let me know what you think.
>> +
>> + def __post_init__(self):
>> + """Gather the required capabilities of the test suite and all test cases."""
>> + for test_object in [self.test_suite_class] + self.test_cases:
>> + self.required_capabilities.update(test_object.required_capabilities)
> <snip>
>>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 06/12] dst: add basic capability support
2024-09-03 16:03 ` Dean Marx
@ 2024-09-05 9:51 ` Juraj Linkeš
0 siblings, 0 replies; 75+ messages in thread
From: Juraj Linkeš @ 2024-09-05 9:51 UTC (permalink / raw)
To: Dean Marx
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
On 3. 9. 2024 18:03, Dean Marx wrote:
> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
>
> A test case or suite may require certain capabilities to be present in
> the tested environment. Add the basic infrastructure for checking the
> support status of capabilities:
> * The Capability ABC defining the common capability API
> * Extension of the TestProtocol with required capabilities (each test
> suite or case stores the capabilities it requires)
> * Integration with the runner which calls the new APIs to get which
> capabilities are supported.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
>
> Looks all good to me, it was interesting to see how you've used abstract
> methods in the Capability class. The only thing I noticed was it seems
> like you wrote "dst" instead of "dts" in the commit message, otherwise:
>
Oh, right, thanks for the catch.
> Reviewed-by: Dean Marx <dmarx@iol.unh.edu <mailto:dmarx@iol.unh.edu>>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 08/12] dts: add NIC capability support
2024-08-26 17:11 ` Jeremy Spewock
@ 2024-09-05 11:56 ` Juraj Linkeš
2024-09-05 15:30 ` Jeremy Spewock
0 siblings, 1 reply; 75+ messages in thread
From: Juraj Linkeš @ 2024-09-05 11:56 UTC (permalink / raw)
To: Jeremy Spewock
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On 26. 8. 2024 19:11, Jeremy Spewock wrote:
> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> <snip>
>> @dataclass
>> class TestPmdPort(TextParser):
>> """Dataclass representing the result of testpmd's ``show port info`` command."""
>> @@ -962,3 +1043,96 @@ def _close(self) -> None:
>> self.stop()
>> self.send_command("quit", "Bye...")
>> return super()._close()
>> +
>> + """
>> + ====== Capability retrieval methods ======
>> + """
>> +
>> + def get_capabilities_rxq_info(
>> + self,
>> + supported_capabilities: MutableSet["NicCapability"],
>> + unsupported_capabilities: MutableSet["NicCapability"],
>> + ) -> None:
>> + """Get all rxq capabilities and divide them into supported and unsupported.
>> +
>> + Args:
>> + supported_capabilities: Supported capabilities will be added to this set.
>> + unsupported_capabilities: Unsupported capabilities will be added to this set.
>> + """
>> + self._logger.debug("Getting rxq capabilities.")
>> + command = f"show rxq info {self.ports[0].id} 0"
>> + rxq_info = TestPmdRxqInfo.parse(self.send_command(command))
>> + if rxq_info.rx_scattered_packets:
>> + supported_capabilities.add(NicCapability.SCATTERED_RX_ENABLED)
>> + else:
>> + unsupported_capabilities.add(NicCapability.SCATTERED_RX_ENABLED)
>> +
>> + """
>> + ====== Decorator methods ======
>> + """
>> +
>> + @staticmethod
>> + def config_mtu_9000(testpmd_method: TestPmdShellSimpleMethod) -> TestPmdShellDecoratedMethod:
>
> It might be more valuable for me to make a method for configuring the
> MTU of all ports so that you don't have to do the loops yourself, I
> can add this to the MTU patch once I update that and rebase it on
> main.
>
Sure, if you add that, I'll use it here. :-)
What won't work with that is the per-port restoration of MTU. But if we
assume that MTU is always the same for all ports, then I don't think
that's going to be a problem. This assumption doesn't seem unreasonable,
I don't see a scenario where it would differ.
>> + """Configure MTU to 9000 on all ports, run `testpmd_method`, then revert.
>> +
>> + Args:
>> + testpmd_method: The method to decorate.
>> +
>> + Returns:
>> + The method decorated with setting and reverting MTU.
>> + """
>> +
>> + def wrapper(testpmd_shell: Self):
>> + original_mtus = []
>> + for port in testpmd_shell.ports:
>> + original_mtus.append((port.id, port.mtu))
>> + testpmd_shell.set_port_mtu(port_id=port.id, mtu=9000, verify=False)
>> + testpmd_method(testpmd_shell)
>> + for port_id, mtu in original_mtus:
>> + testpmd_shell.set_port_mtu(port_id=port_id, mtu=mtu if mtu else 1500, verify=False)
>> +
>> + return wrapper
> <snip>
>> diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py
>> index 8899f07f76..9a79e6ebb3 100644
>> --- a/dts/framework/testbed_model/capability.py
>> +++ b/dts/framework/testbed_model/capability.py
>> @@ -5,14 +5,40 @@
>>
>> This module provides a protocol that defines the common attributes of test cases and suites
>> and support for test environment capabilities.
>> +
>> +Many test cases are testing features not available on all hardware.
>> +
>> +The module also allows developers to mark test cases or suites a requiring certain
>
> small typo: I think you meant " mark test cases or suites *as*
> requiring certain..."
>
Ack.
>> +hardware capabilities with the :func:`requires` decorator.
>> +
>> +Example:
>> + .. code:: python
>> +
>> + from framework.test_suite import TestSuite, func_test
>> + from framework.testbed_model.capability import NicCapability, requires
>> + class TestPmdBufferScatter(TestSuite):
>> + # only the test case requires the scattered_rx capability
>> + # other test cases may not require it
>> + @requires(NicCapability.scattered_rx)
>
> Is it worth updating this to what the enum actually holds
> (SCATTERED_RX_ENABLED) or not really since it is just an example in a
> doc-string? I think it could do either way, but it might be better to
> keep it consistent at least to start.
>
Yes, I overlooked this.
>> + @func_test
>> + def test_scatter_mbuf_2048(self):
> <snip>
>>
>> @@ -96,6 +122,128 @@ def __hash__(self) -> int:
>> """The subclasses must be hashable so that they can be stored in sets."""
>>
>>
>> +@dataclass
>> +class DecoratedNicCapability(Capability):
>> + """A wrapper around :class:`~framework.remote_session.testpmd_shell.NicCapability`.
>> +
>> + Some NIC capabilities are only present or listed as supported only under certain conditions,
>> + such as when a particular configuration is in place. This is achieved by allowing users to pass
>> + a decorator function that decorates the function that gets the support status of the capability.
>> +
>> + New instances should be created with the :meth:`create_unique` class method to ensure
>> + there are no duplicate instances.
>> +
>> + Attributes:
>> + nic_capability: The NIC capability that partly defines each instance.
>> + capability_decorator: The decorator function that will be passed the function associated
>> + with `nic_capability` when discovering the support status of the capability.
>> + Each instance is defined by `capability_decorator` along with `nic_capability`.
>> + """
>> +
>> + nic_capability: NicCapability
>> + capability_decorator: TestPmdShellDecorator | None
>> + _unique_capabilities: ClassVar[
>> + dict[Tuple[NicCapability, TestPmdShellDecorator | None], Self]
>> + ] = {}
>> +
>> + @classmethod
>> + def get_unique(
>> + cls, nic_capability: NicCapability, decorator_fn: TestPmdShellDecorator | None
>> + ) -> "DecoratedNicCapability":
>
> This idea of get_unique really confused me at first. After reading
> different parts of the code to learn how it is being used, I think I
> understand now what it's for. My current understanding is basically
> that you're using an uninstantiated class as essentially a factory
> that stores a dictionary that you are using to hold singletons.
Just a note, these are not singletons, just similar to them. A singleton
is just one instance of class can exist. This class allows more
instances, but it does limit the instances. It closer to an Enum, which
work exactly the same way, but only attribute names are taken into
consideration (with Enums).
> It
> might be confusing to me in general because I haven't really seen this
> idea of dynamically modifying attributes of a class itself rather than
> an instance of the class used this way. Understanding it now, it makes
> sense what you are trying to do and how this is essentially a nice
> cache/factory for singleton values for each capability, but It might
> be helpful to document a little more somehow that _unique_capabilities
> is really just a container for the singleton capabilities, and that
> the top-level class is modified to keep a consistent state throughout
> the framework.
>
> Again, it could just be me having not really seen this idea used
> before, but it was strange to wrap my head around at first since I'm
> more used to class methods being used to read the state of attributes.
>
I'm thinking of adding this to get_unique's docstring:
This is a factory method that implements a quasi-enum pattern.
The instances of this class are stored in a class variable,
_unique_capabilities.
If an instance with `nic_capability` and `decorator_fn` as inputs
doesn't exist, it is created and added to _unique_capabilities.
If it exists, it is returned so that a new identical instance is not
created.
>> + """Get the capability uniquely identified by `nic_capability` and `decorator_fn`.
>> +
>> + Args:
>> + nic_capability: The NIC capability.
>> + decorator_fn: The function that will be passed the function associated
>> + with `nic_capability` when discovering the support status of the capability.
>> +
>> + Returns:
>> + The capability uniquely identified by `nic_capability` and `decorator_fn`.
>> + """
>> + if (nic_capability, decorator_fn) not in cls._unique_capabilities:
>> + cls._unique_capabilities[(nic_capability, decorator_fn)] = cls(
>> + nic_capability, decorator_fn
>> + )
>> + return cls._unique_capabilities[(nic_capability, decorator_fn)]
>> +
>> + @classmethod
>> + def get_supported_capabilities(
>> + cls, sut_node: SutNode, topology: "Topology"
>> + ) -> set["DecoratedNicCapability"]:
>> + """Overrides :meth:`~Capability.get_supported_capabilities`.
>> +
>> + The capabilities are first sorted by decorators, then reduced into a single function which
>> + is then passed to the decorator. This way we only execute each decorator only once.
>
> This second sentence repeats the word "only" but I don't think it is
> really necessary to and it might flow better with either one of them
> instead of both.
>
Ack.
>> + """
>> + supported_conditional_capabilities: set["DecoratedNicCapability"] = set()
>> + logger = get_dts_logger(f"{sut_node.name}.{cls.__name__}")
>> + if topology.type is Topology.type.no_link:
>> + logger.debug(
>> + "No links available in the current topology, not getting NIC capabilities."
>> + )
>> + return supported_conditional_capabilities
>> + logger.debug(
>> + f"Checking which NIC capabilities from {cls.capabilities_to_check} are supported."
>> + )
>> + if cls.capabilities_to_check:
>> + capabilities_to_check_map = cls._get_decorated_capabilities_map()
>> + with TestPmdShell(sut_node, privileged=True) as testpmd_shell:
>> + for conditional_capability_fn, capabilities in capabilities_to_check_map.items():
>> + supported_capabilities: set[NicCapability] = set()
>> + unsupported_capabilities: set[NicCapability] = set()
>> + capability_fn = cls._reduce_capabilities(
>> + capabilities, supported_capabilities, unsupported_capabilities
>> + )
>
> This combines calling all of the capabilities into one function, but
> if there are multiple capabilities that use the same underlying
> testpmd function won't this call the same method multiple times? Or is
> this handled by two Enum values in NicCapability that have the same
> testpmd method as their value hashing to the same thing? For example,
> if there are two capabilities that both require show rxq info and the
> same decorator (scatter and some other capability X), won't this call
> `show rxq info` twice even though you already know that the capability
> is supported after the first call? It's not really harmful for this to
> happen, but it would go against the idea of calling a method and
> getting all of the capabilities that you can the first time. Maybe it
> could be fixed with a conditional check which verifies if `capability`
> is already in `supported_capabilities` or `unsupported_capabilities`
> or not if it's a problem?
>
All you say is true. The whole reason for using all these sets is that
we don't call the functions multiple times. The check you mention is
exactly what's missing.
>> + if conditional_capability_fn:
>> + capability_fn = conditional_capability_fn(capability_fn)
>> + capability_fn(testpmd_shell)
>> + for supported_capability in supported_capabilities:
>> + for capability in capabilities:
>> + if supported_capability == capability.nic_capability:
>> + supported_conditional_capabilities.add(capability)
>
> I might be misunderstanding, but is this also achievable by just writing:
>
> for capability in capabilities:
> if capability.nic_capability in supported_capabilities:
> supported_conditional_capabilities.add(capability)
>
> I think that would be functionally the same, but I think it reads
> easier than a nested loop.
>
It is the same thing, I'll change it.
>> +
>> + logger.debug(f"Found supported capabilities {supported_conditional_capabilities}.")
>> + return supported_conditional_capabilities
>> +
>> + @classmethod
>> + def _get_decorated_capabilities_map(
>> + cls,
>> + ) -> dict[TestPmdShellDecorator | None, set["DecoratedNicCapability"]]:
>> + capabilities_map: dict[TestPmdShellDecorator | None, set["DecoratedNicCapability"]] = {}
>> + for capability in cls.capabilities_to_check:
>> + if capability.capability_decorator not in capabilities_map:
>> + capabilities_map[capability.capability_decorator] = set()
>> + capabilities_map[capability.capability_decorator].add(capability)
>> +
>> + return capabilities_map
>> +
>> + @classmethod
>> + def _reduce_capabilities(
>> + cls,
>> + capabilities: set["DecoratedNicCapability"],
>> + supported_capabilities: MutableSet,
>> + unsupported_capabilities: MutableSet,
>> + ) -> TestPmdShellSimpleMethod:
>> + def reduced_fn(testpmd_shell: TestPmdShell) -> None:
>> + for capability in capabilities:
This is where I'll add the fix:
if capability not in supported_capabilities | unsupported_capabilities:
>> + capability.nic_capability(
>> + testpmd_shell, supported_capabilities, unsupported_capabilities
>> + )
>> +
>> + return reduced_fn
>
> Would it make sense to put these two methods above
> get_supported_capabilities since that is where they are used? I might
> be in favor of it just because it would save you from having to look
> further down in the diff to find what the method does and then go back
> up, but I also understand that it looks like you might have been
> sorting methods by private vs. public so if you think it makes more
> sense to leave them here that is also viable.
>
I sorted it this what so that the code it's easier to read (in my
opinion). I read the method, what it does, then the method calls a
method I haven't seen so I go look beneath the method for the method
definition. To me, this is preferable that reading methods I haven't
seen before. Or, put in another way, the methods are sorted in the order
they're used in code (that's how the code is executed and that's why
this order feels natural to me).
>> +
>> + def __hash__(self) -> int:
>> + """Instances are identified by :attr:`nic_capability` and :attr:`capability_decorator`."""
>> + return hash((self.nic_capability, self.capability_decorator))
>
> I guess my question above is asking if `hash(self.nic_capability) ==
> hash(self.nic_capability.value())` because, if they aren't, then I
> think the map will contain multiple capabilities that use the same
> testpmd function since the capabilities themselves are unique, and
> then because the get_supported_capabilities() method above just calls
> whatever is in this map, it would call it twice. I think the whole
> point of the NoAliasEnum is making sure that they don't hash to the
> same thing. I could be missing something, but, if I am, maybe some
> kind of comment showing where this is handled would be helpful.
>
I think the simple fix in _reduce_capabilities() addresses this, right?
>> +
>> + def __repr__(self) -> str:
>> + """Easy to read string of :attr:`nic_capability` and :attr:`capability_decorator`."""
>> + condition_fn_name = ""
>> + if self.capability_decorator:
>> + condition_fn_name = f"{self.capability_decorator.__qualname__}|"
>> + return f"{condition_fn_name}{self.nic_capability}"
>> +
>> +
>> class TestProtocol(Protocol):
>> """Common test suite and test case attributes."""
>>
>> @@ -116,6 +264,34 @@ def get_test_cases(cls, test_case_sublist: Sequence[str] | None = None) -> tuple
>> raise NotImplementedError()
>>
>>
>> +def requires(
>> + *nic_capabilities: NicCapability,
>> + decorator_fn: TestPmdShellDecorator | None = None,
>> +) -> Callable[[type[TestProtocol]], type[TestProtocol]]:
>> + """A decorator that adds the required capabilities to a test case or test suite.
>> +
>> + Args:
>> + nic_capabilities: The NIC capabilities that are required by the test case or test suite.
>> + decorator_fn: The decorator function that will be used when getting
>> + NIC capability support status.
>> + topology_type: The topology type the test suite or case requires.
>> +
>> + Returns:
>> + The decorated test case or test suite.
>> + """
>> +
>> + def add_required_capability(test_case_or_suite: type[TestProtocol]) -> type[TestProtocol]:
>> + for nic_capability in nic_capabilities:
>> + decorated_nic_capability = DecoratedNicCapability.get_unique(
>> + nic_capability, decorator_fn
>> + )
>> + decorated_nic_capability.add_to_required(test_case_or_suite)
>> +
>> + return test_case_or_suite
>> +
>> + return add_required_capability
>> +
>> +
>> def get_supported_capabilities(
>> sut_node: SutNode,
>> topology_config: Topology,
>> diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
>> index 178a40385e..713549a5b2 100644
>> --- a/dts/tests/TestSuite_pmd_buffer_scatter.py
>> +++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
>> @@ -25,6 +25,7 @@
>> from framework.params.testpmd import SimpleForwardingModes
>> from framework.remote_session.testpmd_shell import TestPmdShell
>> from framework.test_suite import TestSuite, func_test
>> +from framework.testbed_model.capability import NicCapability, requires
>>
>>
>> class TestPmdBufferScatter(TestSuite):
>> @@ -123,6 +124,7 @@ def pmd_scatter(self, mbsize: int) -> None:
>> f"{offset}.",
>> )
>>
>> + @requires(NicCapability.SCATTERED_RX_ENABLED, decorator_fn=TestPmdShell.config_mtu_9000)
>
> Is it possible to instead associate the required decorator with the
> scattered_rx capability itself? Since the configuration is required to
> check the capability, I don't think there will ever be a case where
> `decorator_fn` isn't required here, or a case where it is ever
> anything other than modifying the MTU. Maybe it is more clear from the
> reader's perspective this way that there are other things happening
> under-the-hood, but it also saves developers from having to specify
> something static when we already know beforehand what they need to
> specify.
>
> Doing so would probably mess up some of what you have written in the
> way of DecoratedNicCapability and it might be more difficult to do it
> in a way that only calls the decorator method once if there are
> multiple capabilities that require the same decorator.
>
> Maybe something that you could do is make the NicCapability class in
> Testpmd have values that are tuples of (decorator_fn | None,
> get_capabilities_fn), and then you can still have the
> DecoratedNicCapabilitity class and the methods wouldn't really need to
> change. I think the main thing that would change is just that the
> decorator_fn is collected from the capability/enum instead of the
> requires() method. You could potentially make get_unique easier as
> well since you can just rely on the enum values since already know
> what is required. Then you could take the pairs from that enum and
> create a mapping like you have now of which ones require which
> decorators and keep the same idea.
>
All good points, this is a really good suggestion. Great for the writer
of the tests and basically no downsides, except maybe if there is a
capability which works under different conditions (and we'd want to use
all that), but even with that, we could have different capability names
for tuples with the same capability, but different decorator_fn.
I'll rework this, seems very much worth it.
>> @func_test
>> def test_scatter_mbuf_2048(self) -> None:
>> """Run the :meth:`pmd_scatter` test with `mbsize` set to 2048."""
>> --
>> 2.34.1
>>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 03/12] dts: add test case decorators
2024-09-05 8:07 ` Juraj Linkeš
@ 2024-09-05 15:24 ` Jeremy Spewock
0 siblings, 0 replies; 75+ messages in thread
From: Jeremy Spewock @ 2024-09-05 15:24 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Thu, Sep 5, 2024 at 4:07 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>
>
>
> On 26. 8. 2024 18:50, Jeremy Spewock wrote:
> > On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> > <juraj.linkes@pantheon.tech> wrote:
> > <snip>
> >> class DTSRunner:
> >> @@ -232,9 +231,9 @@ def _get_test_suites_with_cases(
> >>
> >> for test_suite_config in test_suite_configs:
> >> test_suite_class = self._get_test_suite_class(test_suite_config.test_suite)
> >> - test_cases = []
> >> - func_test_cases, perf_test_cases = self._filter_test_cases(
> >> - test_suite_class, test_suite_config.test_cases
> >> + test_cases: list[type[TestCase]] = []
> >
> > If TestCase is just a class, why is the `type[]` in the annotation
> > required? Are these not specific instances of the TestCase class? I
> > figured they would need to be in order for you to run the specific
> > test case methods. Maybe this has something to do with the class being
> > a Protocol?
> >
>
> The *_test decorators return type[TestCase]. The functions (test
> methods) are cast to type[TestCase] (which kinda makes them subclasses
> of TestCase).
Oh interesting, I didn't make the connection that casting them to
type[TestCase] was similar to having them be subclasses of the type,
but this actually makes a lot of sense. Thank you for the
clarification!
>
> This was a suggestion from Luca and I took it as as. Maybe the functions
> could be cast as instances of TestCase, but I didn't try that.
Right, I would think that they could be cast directly to it, but
there's no need obviously so that makes sense.
>
> >> + func_test_cases, perf_test_cases = test_suite_class.get_test_cases(
> >> + test_suite_config.test_cases
> >> )
> >> if func:
> >> test_cases.extend(func_test_cases)
<snip>
>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 04/12] dts: add mechanism to skip test cases or suites
2024-09-05 9:23 ` Juraj Linkeš
@ 2024-09-05 15:26 ` Jeremy Spewock
0 siblings, 0 replies; 75+ messages in thread
From: Jeremy Spewock @ 2024-09-05 15:26 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Thu, Sep 5, 2024 at 5:23 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
<snip>
> >> + def _mark_results(self, result) -> None:
> >
> > Is it worth adding the type annotation for `result` here and to the
> > other places where this is implemented? I guess it doesn't matter that
> > much since it is a private method.
> >
>
> I didn't add it precisely because it's a private method and it's pretty
> self explanatory.
Makes sense.
>
> >> + """Mark the result as well as the child result as `result`.
> >
> > Are these methods even marking their own result or only their
> > children? It seems like it's only really updating the children
> > recursively and its result would have already been updated before this
> > was called.
> >
>
> It's supposed to be just their result which is actually the result of
> the children in all but the TestCaseResult classes. Conceptually, each
Right, of course. Sorry, I was thinking too literally about this
isolated method, with the context that that's how the result of the
level itself is decided, this makes way more sense.
> results level should contains these:
> 1. the result of setup
> 2. the result of teardown
> 3. the result of the level itself (outside of setup and teardown)
>
> The result of the level itself is what's supposed to be set here. The
> thing is we're making the child results for non-test cases and the
> result of the test cases for test cases. Maybe I only need to update the
> docstring.
You probably don't even need to update the doc-string, this seems more
like a mistake on my part :).
>
> >>
> >> The blocking of child results should be done in overloaded methods.
> >> """
> > <snip>
> >>
>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 06/12] dst: add basic capability support
2024-09-05 9:50 ` Juraj Linkeš
@ 2024-09-05 15:27 ` Jeremy Spewock
0 siblings, 0 replies; 75+ messages in thread
From: Jeremy Spewock @ 2024-09-05 15:27 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Thu, Sep 5, 2024 at 5:50 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
<snip>
> >> @@ -63,6 +65,12 @@ class is to hold a subset of test cases (which could be all test cases) because
> >>
> >> test_suite_class: type[TestSuite]
> >> test_cases: list[type[TestCase]]
> >> + required_capabilities: set[Capability] = field(default_factory=set, init=False)
> >
> > This should probably be added to the Attributes section of the
> > doc-string for the class.
>
> Ah, I missed this, thanks.
>
> > When it's there, it might also be useful to
> > explain that this is used by the runner to determine what capabilities
> > need to be searched for to mark the suite for being skipped.
>
> And also test cases.
>
> > The only
> > reason I think that would be useful is it helps differentiate this
> > list of capabilities from the list of required capabilities that every
> > test suite and test case has.
> >
>
> I want to add this:
> The combined required capabilities of both the test suite and the subset
> of test cases.
>
> I think this makes it clear that it's different from the individual
> required capabilities of test suites and cases. Let me know what you think.
>
Yeah, I also think that makes the difference clear, sounds great to me, thanks!
> >> +
> >> + def __post_init__(self):
> >> + """Gather the required capabilities of the test suite and all test cases."""
> >> + for test_object in [self.test_suite_class] + self.test_cases:
> >> + self.required_capabilities.update(test_object.required_capabilities)
> > <snip>
> >>
>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 08/12] dts: add NIC capability support
2024-09-05 11:56 ` Juraj Linkeš
@ 2024-09-05 15:30 ` Jeremy Spewock
0 siblings, 0 replies; 75+ messages in thread
From: Jeremy Spewock @ 2024-09-05 15:30 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Thu, Sep 5, 2024 at 7:56 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>
>
>
> On 26. 8. 2024 19:11, Jeremy Spewock wrote:
> > On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> > <juraj.linkes@pantheon.tech> wrote:
> > <snip>
> >> @dataclass
> >> class TestPmdPort(TextParser):
> >> """Dataclass representing the result of testpmd's ``show port info`` command."""
> >> @@ -962,3 +1043,96 @@ def _close(self) -> None:
> >> self.stop()
> >> self.send_command("quit", "Bye...")
> >> return super()._close()
> >> +
> >> + """
> >> + ====== Capability retrieval methods ======
> >> + """
> >> +
> >> + def get_capabilities_rxq_info(
> >> + self,
> >> + supported_capabilities: MutableSet["NicCapability"],
> >> + unsupported_capabilities: MutableSet["NicCapability"],
> >> + ) -> None:
> >> + """Get all rxq capabilities and divide them into supported and unsupported.
> >> +
> >> + Args:
> >> + supported_capabilities: Supported capabilities will be added to this set.
> >> + unsupported_capabilities: Unsupported capabilities will be added to this set.
> >> + """
> >> + self._logger.debug("Getting rxq capabilities.")
> >> + command = f"show rxq info {self.ports[0].id} 0"
> >> + rxq_info = TestPmdRxqInfo.parse(self.send_command(command))
> >> + if rxq_info.rx_scattered_packets:
> >> + supported_capabilities.add(NicCapability.SCATTERED_RX_ENABLED)
> >> + else:
> >> + unsupported_capabilities.add(NicCapability.SCATTERED_RX_ENABLED)
> >> +
> >> + """
> >> + ====== Decorator methods ======
> >> + """
> >> +
> >> + @staticmethod
> >> + def config_mtu_9000(testpmd_method: TestPmdShellSimpleMethod) -> TestPmdShellDecoratedMethod:
> >
> > It might be more valuable for me to make a method for configuring the
> > MTU of all ports so that you don't have to do the loops yourself, I
> > can add this to the MTU patch once I update that and rebase it on
> > main.
> >
>
> Sure, if you add that, I'll use it here. :-)
> What won't work with that is the per-port restoration of MTU. But if we
> assume that MTU is always the same for all ports, then I don't think
> that's going to be a problem. This assumption doesn't seem unreasonable,
> I don't see a scenario where it would differ.
Good point, and something I didn't think about. I doubt they would be
different either though and I think it would generally be fine to
assume they are the same, but that could also be reason to do it on a
per-port basis. Whatever you think is best. Setting the MTU on all
ports isn't as efficient as I thought it would be when I first wrote
this comment anyway since testpmd doesn't offer something like a `port
config mtu all`, so I just do it one port at a time anyway.
>
> >> + """Configure MTU to 9000 on all ports, run `testpmd_method`, then revert.
> >> +
> >> + Args:
> >> + testpmd_method: The method to decorate.
> >> +
> >> + Returns:
> >> + The method decorated with setting and reverting MTU.
> >> + """
> >> +
<snip>
> >> + @classmethod
> >> + def get_unique(
> >> + cls, nic_capability: NicCapability, decorator_fn: TestPmdShellDecorator | None
> >> + ) -> "DecoratedNicCapability":
> >
> > This idea of get_unique really confused me at first. After reading
> > different parts of the code to learn how it is being used, I think I
> > understand now what it's for. My current understanding is basically
> > that you're using an uninstantiated class as essentially a factory
> > that stores a dictionary that you are using to hold singletons.
>
> Just a note, these are not singletons, just similar to them. A singleton
> is just one instance of class can exist. This class allows more
> instances, but it does limit the instances. It closer to an Enum, which
> work exactly the same way, but only attribute names are taken into
> consideration (with Enums).
That's a good distinction to make. Singleton was the closest thing
that I could make the connection too, but you're right that it isn't
the same and the comparison to Enums makes a lot of sense.
>
> > It
> > might be confusing to me in general because I haven't really seen this
> > idea of dynamically modifying attributes of a class itself rather than
> > an instance of the class used this way. Understanding it now, it makes
> > sense what you are trying to do and how this is essentially a nice
> > cache/factory for singleton values for each capability, but It might
> > be helpful to document a little more somehow that _unique_capabilities
> > is really just a container for the singleton capabilities, and that
> > the top-level class is modified to keep a consistent state throughout
> > the framework.
> >
> > Again, it could just be me having not really seen this idea used
> > before, but it was strange to wrap my head around at first since I'm
> > more used to class methods being used to read the state of attributes.
> >
>
> I'm thinking of adding this to get_unique's docstring:
>
> This is a factory method that implements a quasi-enum pattern.
> The instances of this class are stored in a class variable,
> _unique_capabilities.
>
> If an instance with `nic_capability` and `decorator_fn` as inputs
> doesn't exist, it is created and added to _unique_capabilities.
> If it exists, it is returned so that a new identical instance is not
> created.
Sure, I think this reads pretty well, and I like specifically calling
out the pattern so that if anyone was unfamiliar it gives them
something to research.
>
>
> >> + """Get the capability uniquely identified by `nic_capability` and `decorator_fn`.
> >> +
> >> + Args:
> >> + nic_capability: The NIC capability.
> >> + decorator_fn: The function that will be passed the function associated
> >> + with `nic_capability` when discovering the support status of the capability.
> >> +
> >> + Returns:
> >> + The capability uniquely identified by `nic_capability` and `decorator_fn`.
> >> + """
<snip>
> >> + @classmethod
> >> + def _reduce_capabilities(
> >> + cls,
> >> + capabilities: set["DecoratedNicCapability"],
> >> + supported_capabilities: MutableSet,
> >> + unsupported_capabilities: MutableSet,
> >> + ) -> TestPmdShellSimpleMethod:
> >> + def reduced_fn(testpmd_shell: TestPmdShell) -> None:
> >> + for capability in capabilities:
>
> This is where I'll add the fix:
> if capability not in supported_capabilities | unsupported_capabilities:
>
Perfect, I think that would solve it, yes.
> >> + capability.nic_capability(
> >> + testpmd_shell, supported_capabilities, unsupported_capabilities
> >> + )
> >> +
> >> + return reduced_fn
> >
> > Would it make sense to put these two methods above
> > get_supported_capabilities since that is where they are used? I might
> > be in favor of it just because it would save you from having to look
> > further down in the diff to find what the method does and then go back
> > up, but I also understand that it looks like you might have been
> > sorting methods by private vs. public so if you think it makes more
> > sense to leave them here that is also viable.
> >
>
> I sorted it this what so that the code it's easier to read (in my
> opinion). I read the method, what it does, then the method calls a
> method I haven't seen so I go look beneath the method for the method
> definition. To me, this is preferable that reading methods I haven't
> seen before. Or, put in another way, the methods are sorted in the order
> they're used in code (that's how the code is executed and that's why
> this order feels natural to me).
Right, that does also make sense and is more accurate to how the code
runs. I think it is fine to leave this way then.
>
> >> +
> >> + def __hash__(self) -> int:
> >> + """Instances are identified by :attr:`nic_capability` and :attr:`capability_decorator`."""
> >> + return hash((self.nic_capability, self.capability_decorator))
> >
> > I guess my question above is asking if `hash(self.nic_capability) ==
> > hash(self.nic_capability.value())` because, if they aren't, then I
> > think the map will contain multiple capabilities that use the same
> > testpmd function since the capabilities themselves are unique, and
> > then because the get_supported_capabilities() method above just calls
> > whatever is in this map, it would call it twice. I think the whole
> > point of the NoAliasEnum is making sure that they don't hash to the
> > same thing. I could be missing something, but, if I am, maybe some
> > kind of comment showing where this is handled would be helpful.
> >
>
> I think the simple fix in _reduce_capabilities() addresses this, right?
Yes it does, and it does so better than if the two hashes were equal anyway.
>
> >> +
<snip>
>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 08/12] dts: add NIC capability support
2024-08-27 16:36 ` Jeremy Spewock
@ 2024-09-18 12:58 ` Juraj Linkeš
2024-09-18 16:52 ` Jeremy Spewock
0 siblings, 1 reply; 75+ messages in thread
From: Juraj Linkeš @ 2024-09-18 12:58 UTC (permalink / raw)
To: Jeremy Spewock
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On 27. 8. 2024 18:36, Jeremy Spewock wrote:
> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> <snip>
>> diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py
>> index 8899f07f76..9a79e6ebb3 100644
>> --- a/dts/framework/testbed_model/capability.py
>> +++ b/dts/framework/testbed_model/capability.py
>> @@ -5,14 +5,40 @@
> <snip>
>> + @classmethod
>> + def get_supported_capabilities(
>> + cls, sut_node: SutNode, topology: "Topology"
>> + ) -> set["DecoratedNicCapability"]:
>> + """Overrides :meth:`~Capability.get_supported_capabilities`.
>> +
>> + The capabilities are first sorted by decorators, then reduced into a single function which
>> + is then passed to the decorator. This way we only execute each decorator only once.
>> + """
>> + supported_conditional_capabilities: set["DecoratedNicCapability"] = set()
>> + logger = get_dts_logger(f"{sut_node.name}.{cls.__name__}")
>> + if topology.type is Topology.type.no_link:
>
> As a follow-up, I didn't notice this during my initial review, but in
> testing this line was throwing attribute errors for me due to Topology
> not having an attribute named `type`. I think this was because of
> `Topology.type.no_link` since this attribute isn't initialized on the
> class itself. I fixed this by just replacing it with
> `TopologyType.no_link` locally.
>
I also ran into this, the type attribute is not a class variable. Your
solution works (and I also originally fixed it with exactly that), but I
then I realized topology.type.no_link also works (and was probably my
intention), which doesn't require the extra import of TopologyType.
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 11/12] dts: add Rx offload capabilities
2024-09-03 19:49 ` Dean Marx
@ 2024-09-18 13:59 ` Juraj Linkeš
0 siblings, 0 replies; 75+ messages in thread
From: Juraj Linkeš @ 2024-09-18 13:59 UTC (permalink / raw)
To: Dean Marx
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
On 3. 9. 2024 21:49, Dean Marx wrote:
> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
>
> The scatter Rx offload capability is needed for the pmd_buffer_scatter
> test suite. The command that retrieves the capability is:
> show port <port_id> rx_offload capabilities
>
> The command also retrieves a lot of other capabilities (RX_OFFLOAD_*)
> which are all added into a Flag. The Flag members correspond to NIC
> capability names so a convenience function that looks for the supported
> Flags in a testpmd output is also added.
>
> The NIC capability names (mentioned above) are copy-pasted from the
> Flag. Dynamic addition of Enum members runs into problems with typing
> (mypy doesn't know about the members) and documentation generation
> (Sphinx doesn't know about the members).
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
>
> <snip>
>
> + RX_OFFLOAD_VLAN_FILTER = 1 << 9
> + #: Device supports VLAN offload.
> + RX_OFFLOAD_VLAN_EXTEND = auto()
> + #: Device supports receiving segmented mbufs.
> + RX_OFFLOAD_SCATTER = 1 << 13
>
>
> This was an interesting section, I'm not super familiar with bitwise
> shifting in python flags so I figured I'd ask while it's in mind if
> there's any specific reason for shifting these two flags? Not a critique
> of the code, just genuinely curious.
>
It's there just to mirror the flags in DPDK code.
> Reviewed-by: Dean Marx <dmarx@iol.unh.edu <mailto:dmarx@iol.unh.edu>>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 11/12] dts: add Rx offload capabilities
2024-08-26 17:24 ` Jeremy Spewock
@ 2024-09-18 14:18 ` Juraj Linkeš
2024-09-18 16:53 ` Jeremy Spewock
0 siblings, 1 reply; 75+ messages in thread
From: Juraj Linkeš @ 2024-09-18 14:18 UTC (permalink / raw)
To: Jeremy Spewock
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On 26. 8. 2024 19:24, Jeremy Spewock wrote:
> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> <snip>
>> diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
>> index 48c31124d1..f83569669e 100644
>> --- a/dts/framework/remote_session/testpmd_shell.py
>> +++ b/dts/framework/remote_session/testpmd_shell.py
>> @@ -659,6 +659,103 @@ class TestPmdPortStats(TextParser):
>> tx_bps: int = field(metadata=TextParser.find_int(r"Tx-bps:\s+(\d+)"))
>>
>>
>> +class RxOffloadCapability(Flag):
>> + """Rx offload capabilities of a device."""
>> +
>> + #:
>> + RX_OFFLOAD_VLAN_STRIP = auto()
>> + #: Device supports L3 checksum offload.
>> + RX_OFFLOAD_IPV4_CKSUM = auto()
>> + #: Device supports L4 checksum offload.
>> + RX_OFFLOAD_UDP_CKSUM = auto()
>> + #: Device supports L4 checksum offload.
>> + RX_OFFLOAD_TCP_CKSUM = auto()
>> + #: Device supports Large Receive Offload.
>> + RX_OFFLOAD_TCP_LRO = auto()
>> + #: Device supports QinQ (queue in queue) offload.
>> + RX_OFFLOAD_QINQ_STRIP = auto()
>> + #: Device supports inner packet L3 checksum.
>> + RX_OFFLOAD_OUTER_IPV4_CKSUM = auto()
>> + #: Device supports MACsec.
>> + RX_OFFLOAD_MACSEC_STRIP = auto()
>> + #: Device supports filtering of a VLAN Tag identifier.
>> + RX_OFFLOAD_VLAN_FILTER = 1 << 9
>> + #: Device supports VLAN offload.
>> + RX_OFFLOAD_VLAN_EXTEND = auto()
>> + #: Device supports receiving segmented mbufs.
>> + RX_OFFLOAD_SCATTER = 1 << 13
>
> I know you mentioned in the commit message that the auto() can cause
> problems with mypy/sphinx, is that why this one is a specific value
> instead? Regardless, I think we should probably make it consistent so
> that either all of them are bit-shifts or none of them are unless
> there is a specific reason that the scatter offload is different.
>
Since both you and Dean asked, I'll add something to the docstring about
this.
There are actually two non-auto values (RX_OFFLOAD_VLAN_FILTER = 1 << 9
is the first one). I used the actual values to mirror the flags in DPDK
code.
>> + #: Device supports Timestamp.
>> + RX_OFFLOAD_TIMESTAMP = auto()
>> + #: Device supports crypto processing while packet is received in NIC.
>> + RX_OFFLOAD_SECURITY = auto()
>> + #: Device supports CRC stripping.
>> + RX_OFFLOAD_KEEP_CRC = auto()
>> + #: Device supports L4 checksum offload.
>> + RX_OFFLOAD_SCTP_CKSUM = auto()
>> + #: Device supports inner packet L4 checksum.
>> + RX_OFFLOAD_OUTER_UDP_CKSUM = auto()
>> + #: Device supports RSS hashing.
>> + RX_OFFLOAD_RSS_HASH = auto()
>> + #: Device supports
>> + RX_OFFLOAD_BUFFER_SPLIT = auto()
>> + #: Device supports all checksum capabilities.
>> + RX_OFFLOAD_CHECKSUM = RX_OFFLOAD_IPV4_CKSUM | RX_OFFLOAD_UDP_CKSUM | RX_OFFLOAD_TCP_CKSUM
>> + #: Device supports all VLAN capabilities.
>> + RX_OFFLOAD_VLAN = (
>> + RX_OFFLOAD_VLAN_STRIP
>> + | RX_OFFLOAD_VLAN_FILTER
>> + | RX_OFFLOAD_VLAN_EXTEND
>> + | RX_OFFLOAD_QINQ_STRIP
>> + )
> <snip>
>>
>> @@ -1048,6 +1145,42 @@ def _close(self) -> None:
>> ====== Capability retrieval methods ======
>> """
>>
>> + def get_capabilities_rx_offload(
>> + self,
>> + supported_capabilities: MutableSet["NicCapability"],
>> + unsupported_capabilities: MutableSet["NicCapability"],
>> + ) -> None:
>> + """Get all rx offload capabilities and divide them into supported and unsupported.
>> +
>> + Args:
>> + supported_capabilities: Supported capabilities will be added to this set.
>> + unsupported_capabilities: Unsupported capabilities will be added to this set.
>> + """
>> + self._logger.debug("Getting rx offload capabilities.")
>> + command = f"show port {self.ports[0].id} rx_offload capabilities"
>
> Is it desirable to only get the capabilities of the first port? In the
> current framework I suppose it doesn't matter all that much since you
> can only use the first few ports in the list of ports anyway, but will
> there ever be a case where a test run has 2 different devices included
> in the list of ports? Of course it's possible that it will happen, but
> is it practical? Because, if so, then we would want this to aggregate
> what all the devices are capable of and have capabilities basically
> say "at least one of the ports in the list of ports is capable of
> these things."
>
> This consideration also applies to the rxq info capability gathering as well.
>
No parts of the framework are adjusted to use multiple NIC in a single
test run (because we assume we're testing only one NIC at a time). If we
add this support, it's going to be a broader change.
I approached this with the above assumption in mind and in that case,
testing just one port of the NIC seemed just fine.
>> + rx_offload_capabilities_out = self.send_command(command)
>> + rx_offload_capabilities = RxOffloadCapabilities.parse(rx_offload_capabilities_out)
>> + self._update_capabilities_from_flag(
>> + supported_capabilities,
>> + unsupported_capabilities,
>> + RxOffloadCapability,
>> + rx_offload_capabilities.per_port | rx_offload_capabilities.per_queue,
>> + )
>> +
> <snip>
>>
>> def __call__(
>> self,
>> diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
>> index 89ece2ef56..64c48b0793 100644
>> --- a/dts/tests/TestSuite_pmd_buffer_scatter.py
>> +++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
>> @@ -28,6 +28,7 @@
>> from framework.testbed_model.capability import NicCapability, requires
>>
>>
>> +@requires(NicCapability.RX_OFFLOAD_SCATTER)
>
> I know that we talked about this and how, in the environments we
> looked at, it was true that the offload was supported in all cases
> where the "native" or non-offloaded was supported, but thinking about
> this more, I wonder if it is worth generalizing this assumption to all
> NICs or if we can just decorate the second test case that I wrote
> which uses the offloaded support. As long as the capabilities exposed
> by testpmd are accurate, even if this assumption was true, the
> capability for the non-offloaded one would show False when this
> offload wasn't usable and it would skip the test case anyway, so I
> don't think we lose anything by not including this test-suite-level
> requirement and making it more narrow to the test cases that require
> it.
>
> Let me know your thoughts on that though and I would be interested to
> hear if anyone else has any.
>
I'm not sure I understand what your point is. Let's talk about it in the
call.
>> class TestPmdBufferScatter(TestSuite):
>> """DPDK PMD packet scattering test suite.
>>
>> --
>> 2.34.1
>>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 11/12] dts: add Rx offload capabilities
2024-08-29 15:40 ` Jeremy Spewock
@ 2024-09-18 14:27 ` Juraj Linkeš
2024-09-18 16:57 ` Jeremy Spewock
0 siblings, 1 reply; 75+ messages in thread
From: Juraj Linkeš @ 2024-09-18 14:27 UTC (permalink / raw)
To: Jeremy Spewock
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On 29. 8. 2024 17:40, Jeremy Spewock wrote:
> On Wed, Aug 28, 2024 at 1:44 PM Jeremy Spewock <jspewock@iol.unh.edu> wrote:
>>
>> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
>> <juraj.linkes@pantheon.tech> wrote:
>> <snip>
>>> diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
>>> index 48c31124d1..f83569669e 100644
>>> --- a/dts/framework/remote_session/testpmd_shell.py
>>> +++ b/dts/framework/remote_session/testpmd_shell.py
>>> @@ -659,6 +659,103 @@ class TestPmdPortStats(TextParser):
>>> tx_bps: int = field(metadata=TextParser.find_int(r"Tx-bps:\s+(\d+)"))
>>>
>>>
>>> +class RxOffloadCapability(Flag):
>>> + """Rx offload capabilities of a device."""
>>> +
>>> + #:
>>> + RX_OFFLOAD_VLAN_STRIP = auto()
>>
>> One other thought that I had about this; was there a specific reason
>> that you decided to prefix all of these with `RX_OFFLOAD_`? I am
>> working on a test suite right now that uses both RX and TX offloads
>> and thought that it would be a great use of capabilities, so I am
>> working on adding a TxOffloadCapability flag as well and, since the
>> output is essentially the same, it made a lot of sense to make it a
>> sibling class of this one with similar parsing functionality. In what
>> I was writing, I found it much easier to remove this prefix so that
>> the parsing method can be the same for both RX and TX, and I didn't
>> have to restate some options that are shared between both (like
>> IPv4_CKSUM, UDP_CKSUM, etc.). Is there a reason you can think of why
>> removing this prefix is a bad idea? Hopefully I will have a patch out
>> soon that shows this extension that I've made so that you can see
>> in-code what I was thinking.
>
> I see now that you actually already answered this question, I was just
> looking too much at that piece of code, and clearly not looking
> further down at the helper-method mapping or the commit message that
> you left :).
>
> "The Flag members correspond to NIC
> capability names so a convenience function that looks for the supported
> Flags in a testpmd output is also added."
>
> Having it prefixed with RX_OFFLOAD_ in NicCapability makes a lot of
> sense since it is more explicit. Since there is a good reason to have
> it like this, then the redundancy makes sense I think. There are some
> ways to potentially avoid this like creating a StrFlag class that
> overrides the __str__ method, or something like an additional type
> that would contain a toString method, but it feels very situational
> and specific to this one use-case so it probably isn't going to be
> super valuable. Another thing I could think of to do would be allowing
> the user to pass in a function or something to the helper-method that
> mapped Flag names to their respective NicCapability name, or just
> doing it in the method that gets the offloads instead of using a
> helper at all, but this also just makes it more complicated and maybe
> it isn't worth it.
>
I also had it without the prefix, but then I also realized it's needed
in NicCapability so this is where I ended. I'm not sure complicating
things to remove the prefix is worth it, especially when these names are
basically only used internally. The prefix could actually confer some
benefit if the name appears in a log somewhere (although overriding
__str__ could be the way; maybe I'll think about that).
> I apologize for asking you about something that you already explained,
> but maybe something we can get out of this is that, since these names
> have to be consistent, it might be worth putting that in the
> doc-strings of the flag for when people try to make further expansions
> or changes in the future. Or it could also be generally clear that
> flags used for capabilities should follow this idea, let me know what
> you think.
>
Adding things to docstring is usually a good thing. What should I
document? I guess the correspondence between the flag and NicCapability,
anything else?
>>
>>> + #: Device supports L3 checksum offload.
>>> + RX_OFFLOAD_IPV4_CKSUM = auto()
>>> + #: Device supports L4 checksum offload.
>>> + RX_OFFLOAD_UDP_CKSUM = auto()
>>> + #: Device supports L4 checksum offload.
>>> + RX_OFFLOAD_TCP_CKSUM = auto()
>>> + #: Device supports Large Receive Offload.
>>> + RX_OFFLOAD_TCP_LRO = auto()
>>> + #: Device supports QinQ (queue in queue) offload.
>>> + RX_OFFLOAD_QINQ_STRIP = auto()
>>> + #: Device supports inner packet L3 checksum.
>>> + RX_OFFLOAD_OUTER_IPV4_CKSUM = auto()
>>> + #: Device supports MACsec.
>>> + RX_OFFLOAD_MACSEC_STRIP = auto()
>>> + #: Device supports filtering of a VLAN Tag identifier.
>>> + RX_OFFLOAD_VLAN_FILTER = 1 << 9
>>> + #: Device supports VLAN offload.
>>> + RX_OFFLOAD_VLAN_EXTEND = auto()
>>> + #: Device supports receiving segmented mbufs.
>>> + RX_OFFLOAD_SCATTER = 1 << 13
>>> + #: Device supports Timestamp.
>>> + RX_OFFLOAD_TIMESTAMP = auto()
>>> + #: Device supports crypto processing while packet is received in NIC.
>>> + RX_OFFLOAD_SECURITY = auto()
>>> + #: Device supports CRC stripping.
>>> + RX_OFFLOAD_KEEP_CRC = auto()
>>> + #: Device supports L4 checksum offload.
>>> + RX_OFFLOAD_SCTP_CKSUM = auto()
>>> + #: Device supports inner packet L4 checksum.
>>> + RX_OFFLOAD_OUTER_UDP_CKSUM = auto()
>>> + #: Device supports RSS hashing.
>>> + RX_OFFLOAD_RSS_HASH = auto()
>>> + #: Device supports
>>> + RX_OFFLOAD_BUFFER_SPLIT = auto()
>>> + #: Device supports all checksum capabilities.
>>> + RX_OFFLOAD_CHECKSUM = RX_OFFLOAD_IPV4_CKSUM | RX_OFFLOAD_UDP_CKSUM | RX_OFFLOAD_TCP_CKSUM
>>> + #: Device supports all VLAN capabilities.
>>> + RX_OFFLOAD_VLAN = (
>>> + RX_OFFLOAD_VLAN_STRIP
>>> + | RX_OFFLOAD_VLAN_FILTER
>>> + | RX_OFFLOAD_VLAN_EXTEND
>>> + | RX_OFFLOAD_QINQ_STRIP
>>> + )
>> <snip>
>>>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 08/12] dts: add NIC capability support
2024-09-18 12:58 ` Juraj Linkeš
@ 2024-09-18 16:52 ` Jeremy Spewock
0 siblings, 0 replies; 75+ messages in thread
From: Jeremy Spewock @ 2024-09-18 16:52 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Sep 18, 2024 at 8:58 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>
>
>
> On 27. 8. 2024 18:36, Jeremy Spewock wrote:
> > On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> > <juraj.linkes@pantheon.tech> wrote:
> > <snip>
> >> diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py
> >> index 8899f07f76..9a79e6ebb3 100644
> >> --- a/dts/framework/testbed_model/capability.py
> >> +++ b/dts/framework/testbed_model/capability.py
> >> @@ -5,14 +5,40 @@
> > <snip>
> >> + @classmethod
> >> + def get_supported_capabilities(
> >> + cls, sut_node: SutNode, topology: "Topology"
> >> + ) -> set["DecoratedNicCapability"]:
> >> + """Overrides :meth:`~Capability.get_supported_capabilities`.
> >> +
> >> + The capabilities are first sorted by decorators, then reduced into a single function which
> >> + is then passed to the decorator. This way we only execute each decorator only once.
> >> + """
> >> + supported_conditional_capabilities: set["DecoratedNicCapability"] = set()
> >> + logger = get_dts_logger(f"{sut_node.name}.{cls.__name__}")
> >> + if topology.type is Topology.type.no_link:
> >
> > As a follow-up, I didn't notice this during my initial review, but in
> > testing this line was throwing attribute errors for me due to Topology
> > not having an attribute named `type`. I think this was because of
> > `Topology.type.no_link` since this attribute isn't initialized on the
> > class itself. I fixed this by just replacing it with
> > `TopologyType.no_link` locally.
> >
>
> I also ran into this, the type attribute is not a class variable. Your
> solution works (and I also originally fixed it with exactly that), but I
> then I realized topology.type.no_link also works (and was probably my
> intention), which doesn't require the extra import of TopologyType.
Right, that's smart. I forget that you can do that with enums.
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 11/12] dts: add Rx offload capabilities
2024-09-18 14:18 ` Juraj Linkeš
@ 2024-09-18 16:53 ` Jeremy Spewock
0 siblings, 0 replies; 75+ messages in thread
From: Jeremy Spewock @ 2024-09-18 16:53 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Sep 18, 2024 at 10:18 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
>
>
> On 26. 8. 2024 19:24, Jeremy Spewock wrote:
> > On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> > <juraj.linkes@pantheon.tech> wrote:
> > <snip>
> >> diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
> >> index 48c31124d1..f83569669e 100644
> >> --- a/dts/framework/remote_session/testpmd_shell.py
> >> +++ b/dts/framework/remote_session/testpmd_shell.py
> >> @@ -659,6 +659,103 @@ class TestPmdPortStats(TextParser):
> >> tx_bps: int = field(metadata=TextParser.find_int(r"Tx-bps:\s+(\d+)"))
> >>
> >>
> >> +class RxOffloadCapability(Flag):
> >> + """Rx offload capabilities of a device."""
> >> +
> >> + #:
> >> + RX_OFFLOAD_VLAN_STRIP = auto()
> >> + #: Device supports L3 checksum offload.
> >> + RX_OFFLOAD_IPV4_CKSUM = auto()
> >> + #: Device supports L4 checksum offload.
> >> + RX_OFFLOAD_UDP_CKSUM = auto()
> >> + #: Device supports L4 checksum offload.
> >> + RX_OFFLOAD_TCP_CKSUM = auto()
> >> + #: Device supports Large Receive Offload.
> >> + RX_OFFLOAD_TCP_LRO = auto()
> >> + #: Device supports QinQ (queue in queue) offload.
> >> + RX_OFFLOAD_QINQ_STRIP = auto()
> >> + #: Device supports inner packet L3 checksum.
> >> + RX_OFFLOAD_OUTER_IPV4_CKSUM = auto()
> >> + #: Device supports MACsec.
> >> + RX_OFFLOAD_MACSEC_STRIP = auto()
> >> + #: Device supports filtering of a VLAN Tag identifier.
> >> + RX_OFFLOAD_VLAN_FILTER = 1 << 9
> >> + #: Device supports VLAN offload.
> >> + RX_OFFLOAD_VLAN_EXTEND = auto()
> >> + #: Device supports receiving segmented mbufs.
> >> + RX_OFFLOAD_SCATTER = 1 << 13
> >
> > I know you mentioned in the commit message that the auto() can cause
> > problems with mypy/sphinx, is that why this one is a specific value
> > instead? Regardless, I think we should probably make it consistent so
> > that either all of them are bit-shifts or none of them are unless
> > there is a specific reason that the scatter offload is different.
> >
>
> Since both you and Dean asked, I'll add something to the docstring about
> this.
>
> There are actually two non-auto values (RX_OFFLOAD_VLAN_FILTER = 1 << 9
> is the first one). I used the actual values to mirror the flags in DPDK
> code.
Gotcha, that makes sense.
>
> >> + #: Device supports Timestamp.
> >> + RX_OFFLOAD_TIMESTAMP = auto()
> >> + #: Device supports crypto processing while packet is received in NIC.
> >> + RX_OFFLOAD_SECURITY = auto()
> >> + #: Device supports CRC stripping.
> >> + RX_OFFLOAD_KEEP_CRC = auto()
> >> + #: Device supports L4 checksum offload.
> >> + RX_OFFLOAD_SCTP_CKSUM = auto()
> >> + #: Device supports inner packet L4 checksum.
> >> + RX_OFFLOAD_OUTER_UDP_CKSUM = auto()
> >> + #: Device supports RSS hashing.
> >> + RX_OFFLOAD_RSS_HASH = auto()
> >> + #: Device supports
> >> + RX_OFFLOAD_BUFFER_SPLIT = auto()
> >> + #: Device supports all checksum capabilities.
> >> + RX_OFFLOAD_CHECKSUM = RX_OFFLOAD_IPV4_CKSUM | RX_OFFLOAD_UDP_CKSUM | RX_OFFLOAD_TCP_CKSUM
> >> + #: Device supports all VLAN capabilities.
> >> + RX_OFFLOAD_VLAN = (
> >> + RX_OFFLOAD_VLAN_STRIP
> >> + | RX_OFFLOAD_VLAN_FILTER
> >> + | RX_OFFLOAD_VLAN_EXTEND
> >> + | RX_OFFLOAD_QINQ_STRIP
> >> + )
> > <snip>
> >>
> >> @@ -1048,6 +1145,42 @@ def _close(self) -> None:
> >> ====== Capability retrieval methods ======
> >> """
> >>
> >> + def get_capabilities_rx_offload(
> >> + self,
> >> + supported_capabilities: MutableSet["NicCapability"],
> >> + unsupported_capabilities: MutableSet["NicCapability"],
> >> + ) -> None:
> >> + """Get all rx offload capabilities and divide them into supported and unsupported.
> >> +
> >> + Args:
> >> + supported_capabilities: Supported capabilities will be added to this set.
> >> + unsupported_capabilities: Unsupported capabilities will be added to this set.
> >> + """
> >> + self._logger.debug("Getting rx offload capabilities.")
> >> + command = f"show port {self.ports[0].id} rx_offload capabilities"
> >
> > Is it desirable to only get the capabilities of the first port? In the
> > current framework I suppose it doesn't matter all that much since you
> > can only use the first few ports in the list of ports anyway, but will
> > there ever be a case where a test run has 2 different devices included
> > in the list of ports? Of course it's possible that it will happen, but
> > is it practical? Because, if so, then we would want this to aggregate
> > what all the devices are capable of and have capabilities basically
> > say "at least one of the ports in the list of ports is capable of
> > these things."
> >
> > This consideration also applies to the rxq info capability gathering as well.
> >
>
> No parts of the framework are adjusted to use multiple NIC in a single
> test run (because we assume we're testing only one NIC at a time). If we
> add this support, it's going to be a broader change.
>
> I approached this with the above assumption in mind and in that case,
> testing just one port of the NIC seemed just fine.
That's a good point that making the adjustment to allow for multiple
devices is a bigger change that is definitely out of scope for this
series. Makes sense to put it off and go with the current assumptions,
I only asked in case it was something simple so it would be one less
thing to do in the future :). This is fine as is then I think.
>
> >> + rx_offload_capabilities_out = self.send_command(command)
> >> + rx_offload_capabilities = RxOffloadCapabilities.parse(rx_offload_capabilities_out)
> >> + self._update_capabilities_from_flag(
> >> + supported_capabilities,
> >> + unsupported_capabilities,
> >> + RxOffloadCapability,
> >> + rx_offload_capabilities.per_port | rx_offload_capabilities.per_queue,
> >> + )
> >> +
> > <snip>
> >>
> >> def __call__(
> >> self,
> >> diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
> >> index 89ece2ef56..64c48b0793 100644
> >> --- a/dts/tests/TestSuite_pmd_buffer_scatter.py
> >> +++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
> >> @@ -28,6 +28,7 @@
> >> from framework.testbed_model.capability import NicCapability, requires
> >>
> >>
> >> +@requires(NicCapability.RX_OFFLOAD_SCATTER)
> >
> > I know that we talked about this and how, in the environments we
> > looked at, it was true that the offload was supported in all cases
> > where the "native" or non-offloaded was supported, but thinking about
> > this more, I wonder if it is worth generalizing this assumption to all
> > NICs or if we can just decorate the second test case that I wrote
> > which uses the offloaded support. As long as the capabilities exposed
> > by testpmd are accurate, even if this assumption was true, the
> > capability for the non-offloaded one would show False when this
> > offload wasn't usable and it would skip the test case anyway, so I
> > don't think we lose anything by not including this test-suite-level
> > requirement and making it more narrow to the test cases that require
> > it.
> >
> > Let me know your thoughts on that though and I would be interested to
> > hear if anyone else has any.
> >
>
> I'm not sure I understand what your point is. Let's talk about it in the
> call.
Sure, sounds good to me.
>
> >> class TestPmdBufferScatter(TestSuite):
> >> """DPDK PMD packet scattering test suite.
> >>
> >> --
> >> 2.34.1
> >>
>
^ permalink raw reply [flat|nested] 75+ messages in thread
* Re: [PATCH v3 11/12] dts: add Rx offload capabilities
2024-09-18 14:27 ` Juraj Linkeš
@ 2024-09-18 16:57 ` Jeremy Spewock
0 siblings, 0 replies; 75+ messages in thread
From: Jeremy Spewock @ 2024-09-18 16:57 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Sep 18, 2024 at 10:27 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
>
>
> On 29. 8. 2024 17:40, Jeremy Spewock wrote:
> > On Wed, Aug 28, 2024 at 1:44 PM Jeremy Spewock <jspewock@iol.unh.edu> wrote:
> >>
> >> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> >> <juraj.linkes@pantheon.tech> wrote:
> >> <snip>
> >>> diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
> >>> index 48c31124d1..f83569669e 100644
> >>> --- a/dts/framework/remote_session/testpmd_shell.py
> >>> +++ b/dts/framework/remote_session/testpmd_shell.py
> >>> @@ -659,6 +659,103 @@ class TestPmdPortStats(TextParser):
> >>> tx_bps: int = field(metadata=TextParser.find_int(r"Tx-bps:\s+(\d+)"))
> >>>
> >>>
> >>> +class RxOffloadCapability(Flag):
> >>> + """Rx offload capabilities of a device."""
> >>> +
> >>> + #:
> >>> + RX_OFFLOAD_VLAN_STRIP = auto()
> >>
> >> One other thought that I had about this; was there a specific reason
> >> that you decided to prefix all of these with `RX_OFFLOAD_`? I am
> >> working on a test suite right now that uses both RX and TX offloads
> >> and thought that it would be a great use of capabilities, so I am
> >> working on adding a TxOffloadCapability flag as well and, since the
> >> output is essentially the same, it made a lot of sense to make it a
> >> sibling class of this one with similar parsing functionality. In what
> >> I was writing, I found it much easier to remove this prefix so that
> >> the parsing method can be the same for both RX and TX, and I didn't
> >> have to restate some options that are shared between both (like
> >> IPv4_CKSUM, UDP_CKSUM, etc.). Is there a reason you can think of why
> >> removing this prefix is a bad idea? Hopefully I will have a patch out
> >> soon that shows this extension that I've made so that you can see
> >> in-code what I was thinking.
> >
> > I see now that you actually already answered this question, I was just
> > looking too much at that piece of code, and clearly not looking
> > further down at the helper-method mapping or the commit message that
> > you left :).
> >
> > "The Flag members correspond to NIC
> > capability names so a convenience function that looks for the supported
> > Flags in a testpmd output is also added."
> >
> > Having it prefixed with RX_OFFLOAD_ in NicCapability makes a lot of
> > sense since it is more explicit. Since there is a good reason to have
> > it like this, then the redundancy makes sense I think. There are some
> > ways to potentially avoid this like creating a StrFlag class that
> > overrides the __str__ method, or something like an additional type
> > that would contain a toString method, but it feels very situational
> > and specific to this one use-case so it probably isn't going to be
> > super valuable. Another thing I could think of to do would be allowing
> > the user to pass in a function or something to the helper-method that
> > mapped Flag names to their respective NicCapability name, or just
> > doing it in the method that gets the offloads instead of using a
> > helper at all, but this also just makes it more complicated and maybe
> > it isn't worth it.
> >
>
> I also had it without the prefix, but then I also realized it's needed
> in NicCapability so this is where I ended. I'm not sure complicating
> things to remove the prefix is worth it, especially when these names are
> basically only used internally. The prefix could actually confer some
> benefit if the name appears in a log somewhere (although overriding
> __str__ could be the way; maybe I'll think about that).
It could be done with modifying str, but I found that an approach that
was easier was just adding an optional prefix to the
_update_capabilities_from_flag() method since you will know whether
the capability is Rx or Tx at the point of calling this method. I feel
like either or could work, I'm not sure exactly which is better. The
change that adds the prefix is in the Rx/Tx offload suite in the first
commit [1] if you wanted to look at it. This commit and the one after
it are isolated to be only changes to the capabilities series.
[1] https://patchwork.dpdk.org/project/dpdk/patch/20240903194642.24458-2-jspewock@iol.unh.edu/
>
> > I apologize for asking you about something that you already explained,
> > but maybe something we can get out of this is that, since these names
> > have to be consistent, it might be worth putting that in the
> > doc-strings of the flag for when people try to make further expansions
> > or changes in the future. Or it could also be generally clear that
> > flags used for capabilities should follow this idea, let me know what
> > you think.
> >
>
> Adding things to docstring is usually a good thing. What should I
> document? I guess the correspondence between the flag and NicCapability,
> anything else?
The only thing I was thinking was that the flag values have to match
the values in NicCapability. I think explaining it this way is enough
just to make it clear that it is done that way for a purpose and
cannot be different (unless of course the no-prefix way is favorable).
>
> >>
> >>> + #: Device supports L3 checksum offload.
> >>> + RX_OFFLOAD_IPV4_CKSUM = auto()
> >>> + #: Device supports L4 checksum offload.
> >>> + RX_OFFLOAD_UDP_CKSUM = auto()
> >>> + #: Device supports L4 checksum offload.
> >>> + RX_OFFLOAD_TCP_CKSUM = auto()
> >>> + #: Device supports Large Receive Offload.
> >>> + RX_OFFLOAD_TCP_LRO = auto()
> >>> + #: Device supports QinQ (queue in queue) offload.
> >>> + RX_OFFLOAD_QINQ_STRIP = auto()
> >>> + #: Device supports inner packet L3 checksum.
> >>> + RX_OFFLOAD_OUTER_IPV4_CKSUM = auto()
> >>> + #: Device supports MACsec.
> >>> + RX_OFFLOAD_MACSEC_STRIP = auto()
> >>> + #: Device supports filtering of a VLAN Tag identifier.
> >>> + RX_OFFLOAD_VLAN_FILTER = 1 << 9
> >>> + #: Device supports VLAN offload.
> >>> + RX_OFFLOAD_VLAN_EXTEND = auto()
> >>> + #: Device supports receiving segmented mbufs.
> >>> + RX_OFFLOAD_SCATTER = 1 << 13
> >>> + #: Device supports Timestamp.
> >>> + RX_OFFLOAD_TIMESTAMP = auto()
> >>> + #: Device supports crypto processing while packet is received in NIC.
> >>> + RX_OFFLOAD_SECURITY = auto()
> >>> + #: Device supports CRC stripping.
> >>> + RX_OFFLOAD_KEEP_CRC = auto()
> >>> + #: Device supports L4 checksum offload.
> >>> + RX_OFFLOAD_SCTP_CKSUM = auto()
> >>> + #: Device supports inner packet L4 checksum.
> >>> + RX_OFFLOAD_OUTER_UDP_CKSUM = auto()
> >>> + #: Device supports RSS hashing.
> >>> + RX_OFFLOAD_RSS_HASH = auto()
> >>> + #: Device supports
> >>> + RX_OFFLOAD_BUFFER_SPLIT = auto()
> >>> + #: Device supports all checksum capabilities.
> >>> + RX_OFFLOAD_CHECKSUM = RX_OFFLOAD_IPV4_CKSUM | RX_OFFLOAD_UDP_CKSUM | RX_OFFLOAD_TCP_CKSUM
> >>> + #: Device supports all VLAN capabilities.
> >>> + RX_OFFLOAD_VLAN = (
> >>> + RX_OFFLOAD_VLAN_STRIP
> >>> + | RX_OFFLOAD_VLAN_FILTER
> >>> + | RX_OFFLOAD_VLAN_EXTEND
> >>> + | RX_OFFLOAD_QINQ_STRIP
> >>> + )
> >> <snip>
> >>>
>
^ permalink raw reply [flat|nested] 75+ messages in thread
end of thread, other threads:[~2024-09-18 16:57 UTC | newest]
Thread overview: 75+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-03-01 15:54 [RFC PATCH v1] dts: skip test cases based on capabilities Juraj Linkeš
2024-04-11 8:48 ` [RFC PATCH v2] " Juraj Linkeš
2024-05-21 15:47 ` Luca Vizzarro
2024-05-22 14:58 ` Luca Vizzarro
2024-06-07 13:13 ` Juraj Linkeš
2024-06-11 9:51 ` Luca Vizzarro
2024-06-12 9:15 ` Juraj Linkeš
2024-06-17 15:07 ` Luca Vizzarro
2024-05-24 20:51 ` Nicholas Pratte
2024-05-31 16:44 ` Luca Vizzarro
2024-06-05 13:55 ` Patrick Robb
2024-06-06 13:36 ` Jeremy Spewock
2024-06-03 14:40 ` Nicholas Pratte
2024-06-07 13:20 ` Juraj Linkeš
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
2024-08-21 14:53 ` [PATCH v3 01/12] dts: fix default device error handling mode Juraj Linkeš
2024-08-26 16:42 ` Jeremy Spewock
2024-08-27 16:15 ` Dean Marx
2024-08-27 20:09 ` Nicholas Pratte
2024-08-21 14:53 ` [PATCH v3 02/12] dts: add the aenum dependency Juraj Linkeš
2024-08-26 16:42 ` Jeremy Spewock
2024-08-27 16:28 ` Dean Marx
2024-08-27 20:21 ` Nicholas Pratte
2024-08-21 14:53 ` [PATCH v3 03/12] dts: add test case decorators Juraj Linkeš
2024-08-26 16:50 ` Jeremy Spewock
2024-09-05 8:07 ` Juraj Linkeš
2024-09-05 15:24 ` Jeremy Spewock
2024-08-28 20:09 ` Dean Marx
2024-08-30 15:50 ` Nicholas Pratte
2024-08-21 14:53 ` [PATCH v3 04/12] dts: add mechanism to skip test cases or suites Juraj Linkeš
2024-08-26 16:52 ` Jeremy Spewock
2024-09-05 9:23 ` Juraj Linkeš
2024-09-05 15:26 ` Jeremy Spewock
2024-08-28 20:37 ` Dean Marx
2024-08-21 14:53 ` [PATCH v3 05/12] dts: add support for simpler topologies Juraj Linkeš
2024-08-26 16:54 ` Jeremy Spewock
2024-09-05 9:42 ` Juraj Linkeš
2024-08-28 20:56 ` Dean Marx
2024-08-21 14:53 ` [PATCH v3 06/12] dst: add basic capability support Juraj Linkeš
2024-08-26 16:56 ` Jeremy Spewock
2024-09-05 9:50 ` Juraj Linkeš
2024-09-05 15:27 ` Jeremy Spewock
2024-09-03 16:03 ` Dean Marx
2024-09-05 9:51 ` Juraj Linkeš
2024-08-21 14:53 ` [PATCH v3 07/12] dts: add testpmd port information caching Juraj Linkeš
2024-08-26 16:56 ` Jeremy Spewock
2024-09-03 16:12 ` Dean Marx
2024-08-21 14:53 ` [PATCH v3 08/12] dts: add NIC capability support Juraj Linkeš
2024-08-26 17:11 ` Jeremy Spewock
2024-09-05 11:56 ` Juraj Linkeš
2024-09-05 15:30 ` Jeremy Spewock
2024-08-27 16:36 ` Jeremy Spewock
2024-09-18 12:58 ` Juraj Linkeš
2024-09-18 16:52 ` Jeremy Spewock
2024-09-03 19:13 ` Dean Marx
2024-08-21 14:53 ` [PATCH v3 09/12] dts: add topology capability Juraj Linkeš
2024-08-26 17:13 ` Jeremy Spewock
2024-09-03 17:50 ` Dean Marx
2024-08-21 14:53 ` [PATCH v3 10/12] doc: add DTS capability doc sources Juraj Linkeš
2024-08-26 17:13 ` Jeremy Spewock
2024-09-03 17:52 ` Dean Marx
2024-08-21 14:53 ` [PATCH v3 11/12] dts: add Rx offload capabilities Juraj Linkeš
2024-08-26 17:24 ` Jeremy Spewock
2024-09-18 14:18 ` Juraj Linkeš
2024-09-18 16:53 ` Jeremy Spewock
2024-08-28 17:44 ` Jeremy Spewock
2024-08-29 15:40 ` Jeremy Spewock
2024-09-18 14:27 ` Juraj Linkeš
2024-09-18 16:57 ` Jeremy Spewock
2024-09-03 19:49 ` Dean Marx
2024-09-18 13:59 ` Juraj Linkeš
2024-08-21 14:53 ` [PATCH v3 12/12] dts: add NIC capabilities from show port info Juraj Linkeš
2024-08-26 17:24 ` Jeremy Spewock
2024-09-03 18:02 ` Dean Marx
2024-08-26 17:25 ` [PATCH v3 00/12] dts: add test skipping based on capabilities Jeremy Spewock
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).