* [PATCH v3 01/12] dts: fix default device error handling mode
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 16:42 ` Jeremy Spewock
` (2 more replies)
2024-08-21 14:53 ` [PATCH v3 02/12] dts: add the aenum dependency Juraj Linkeš
` (11 subsequent siblings)
12 siblings, 3 replies; 107+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
The device_error_handling_mode of testpmd port may not be present, e.g.
in VM ports.
Fixes: 61d5bc9bf974 ("dts: add port info command to testpmd shell")
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/remote_session/testpmd_shell.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index a5347b07dc..b4ad253020 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -465,8 +465,8 @@ class TestPmdPort(TextParser):
metadata=DeviceCapabilitiesFlag.make_parser(),
)
#:
- device_error_handling_mode: DeviceErrorHandlingMode = field(
- metadata=DeviceErrorHandlingMode.make_parser()
+ device_error_handling_mode: DeviceErrorHandlingMode | None = field(
+ default=None, metadata=DeviceErrorHandlingMode.make_parser()
)
#:
device_private_info: str | None = field(
--
2.34.1
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 01/12] dts: fix default device error handling mode
2024-08-21 14:53 ` [PATCH v3 01/12] dts: fix default device error handling mode Juraj Linkeš
@ 2024-08-26 16:42 ` Jeremy Spewock
2024-08-27 16:15 ` Dean Marx
2024-08-27 20:09 ` Nicholas Pratte
2 siblings, 0 replies; 107+ messages in thread
From: Jeremy Spewock @ 2024-08-26 16:42 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
> The device_error_handling_mode of testpmd port may not be present, e.g.
> in VM ports.
>
> Fixes: 61d5bc9bf974 ("dts: add port info command to testpmd shell")
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 01/12] dts: fix default device error handling mode
2024-08-21 14:53 ` [PATCH v3 01/12] dts: fix default device error handling mode Juraj Linkeš
2024-08-26 16:42 ` Jeremy Spewock
@ 2024-08-27 16:15 ` Dean Marx
2024-08-27 20:09 ` Nicholas Pratte
2 siblings, 0 replies; 107+ messages in thread
From: Dean Marx @ 2024-08-27 16:15 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 367 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> The device_error_handling_mode of testpmd port may not be present, e.g.
> in VM ports.
>
> Fixes: 61d5bc9bf974 ("dts: add port info command to testpmd shell")
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 675 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 01/12] dts: fix default device error handling mode
2024-08-21 14:53 ` [PATCH v3 01/12] dts: fix default device error handling mode Juraj Linkeš
2024-08-26 16:42 ` Jeremy Spewock
2024-08-27 16:15 ` Dean Marx
@ 2024-08-27 20:09 ` Nicholas Pratte
2 siblings, 0 replies; 107+ messages in thread
From: Nicholas Pratte @ 2024-08-27 20:09 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
> The device_error_handling_mode of testpmd port may not be present, e.g.
> in VM ports.
>
> Fixes: 61d5bc9bf974 ("dts: add port info command to testpmd shell")
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
Reviewed-by: Nicholas Pratte <npratte@iol.unh.edu>
^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH v3 02/12] dts: add the aenum dependency
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
2024-08-21 14:53 ` [PATCH v3 01/12] dts: fix default device error handling mode Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 16:42 ` Jeremy Spewock
` (2 more replies)
2024-08-21 14:53 ` [PATCH v3 03/12] dts: add test case decorators Juraj Linkeš
` (10 subsequent siblings)
12 siblings, 3 replies; 107+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
Regular Python enumerations create only one instance for members with
the same value, such as:
class MyEnum(Enum):
foo = 1
bar = 1
MyEnum.foo and MyEnum.bar are aliases that return the same instance.
DTS needs to return different instances in the above scenario so that we
can map capabilities with different names to the same function that
retrieves the capabilities.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/poetry.lock | 14 +++++++++++++-
dts/pyproject.toml | 1 +
2 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/dts/poetry.lock b/dts/poetry.lock
index 2dd8bad498..cf5f6569c6 100644
--- a/dts/poetry.lock
+++ b/dts/poetry.lock
@@ -1,5 +1,17 @@
# This file is automatically @generated by Poetry 1.8.2 and should not be changed by hand.
+[[package]]
+name = "aenum"
+version = "3.1.15"
+description = "Advanced Enumerations (compatible with Python's stdlib Enum), NamedTuples, and NamedConstants"
+optional = false
+python-versions = "*"
+files = [
+ {file = "aenum-3.1.15-py2-none-any.whl", hash = "sha256:27b1710b9d084de6e2e695dab78fe9f269de924b51ae2850170ee7e1ca6288a5"},
+ {file = "aenum-3.1.15-py3-none-any.whl", hash = "sha256:e0dfaeea4c2bd362144b87377e2c61d91958c5ed0b4daf89cb6f45ae23af6288"},
+ {file = "aenum-3.1.15.tar.gz", hash = "sha256:8cbd76cd18c4f870ff39b24284d3ea028fbe8731a58df3aa581e434c575b9559"},
+]
+
[[package]]
name = "alabaster"
version = "0.7.13"
@@ -1350,4 +1362,4 @@ jsonschema = ">=4,<5"
[metadata]
lock-version = "2.0"
python-versions = "^3.10"
-content-hash = "6db17f96cb31fb463b0b0a31dff9c02aa72641e0bffd8a610033fe2324006c43"
+content-hash = "6f20ce05310df93fed1d392160d1653ae5de5c6f260a5865eb3c6111a7c2b394"
diff --git a/dts/pyproject.toml b/dts/pyproject.toml
index 38281f0e39..6e347852cc 100644
--- a/dts/pyproject.toml
+++ b/dts/pyproject.toml
@@ -26,6 +26,7 @@ fabric = "^2.7.1"
scapy = "^2.5.0"
pydocstyle = "6.1.1"
typing-extensions = "^4.11.0"
+aenum = "^3.1.15"
[tool.poetry.group.dev.dependencies]
mypy = "^1.10.0"
--
2.34.1
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 02/12] dts: add the aenum dependency
2024-08-21 14:53 ` [PATCH v3 02/12] dts: add the aenum dependency Juraj Linkeš
@ 2024-08-26 16:42 ` Jeremy Spewock
2024-08-27 16:28 ` Dean Marx
2024-08-27 20:21 ` Nicholas Pratte
2 siblings, 0 replies; 107+ messages in thread
From: Jeremy Spewock @ 2024-08-26 16:42 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
> Regular Python enumerations create only one instance for members with
> the same value, such as:
> class MyEnum(Enum):
> foo = 1
> bar = 1
>
> MyEnum.foo and MyEnum.bar are aliases that return the same instance.
I didn't know this was a thing in Python Enums. It was very strange to
me at first, but thinking about this more it makes some sense.
>
> DTS needs to return different instances in the above scenario so that we
> can map capabilities with different names to the same function that
> retrieves the capabilities.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 02/12] dts: add the aenum dependency
2024-08-21 14:53 ` [PATCH v3 02/12] dts: add the aenum dependency Juraj Linkeš
2024-08-26 16:42 ` Jeremy Spewock
@ 2024-08-27 16:28 ` Dean Marx
2024-08-27 20:21 ` Nicholas Pratte
2 siblings, 0 replies; 107+ messages in thread
From: Dean Marx @ 2024-08-27 16:28 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 612 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> Regular Python enumerations create only one instance for members with
> the same value, such as:
> class MyEnum(Enum):
> foo = 1
> bar = 1
>
> MyEnum.foo and MyEnum.bar are aliases that return the same instance.
>
> DTS needs to return different instances in the above scenario so that we
> can map capabilities with different names to the same function that
> retrieves the capabilities.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 929 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 02/12] dts: add the aenum dependency
2024-08-21 14:53 ` [PATCH v3 02/12] dts: add the aenum dependency Juraj Linkeš
2024-08-26 16:42 ` Jeremy Spewock
2024-08-27 16:28 ` Dean Marx
@ 2024-08-27 20:21 ` Nicholas Pratte
2 siblings, 0 replies; 107+ messages in thread
From: Nicholas Pratte @ 2024-08-27 20:21 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
> Regular Python enumerations create only one instance for members with
> the same value, such as:
> class MyEnum(Enum):
> foo = 1
> bar = 1
>
> MyEnum.foo and MyEnum.bar are aliases that return the same instance.
>
> DTS needs to return different instances in the above scenario so that we
> can map capabilities with different names to the same function that
> retrieves the capabilities.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Nicholas Pratte <npratte@iol.unh.edu>
^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH v3 03/12] dts: add test case decorators
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
2024-08-21 14:53 ` [PATCH v3 01/12] dts: fix default device error handling mode Juraj Linkeš
2024-08-21 14:53 ` [PATCH v3 02/12] dts: add the aenum dependency Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 16:50 ` Jeremy Spewock
` (2 more replies)
2024-08-21 14:53 ` [PATCH v3 04/12] dts: add mechanism to skip test cases or suites Juraj Linkeš
` (9 subsequent siblings)
12 siblings, 3 replies; 107+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
Add decorators for functional and performance test cases. These
decorators add attributes to the decorated test cases.
With the addition of decorators, we change the test case discovery
mechanism from looking at test case names according to a regex to simply
checking an attribute of the function added with one of the decorators.
The decorators allow us to add further variables to test cases.
Also move the test case filtering to TestSuite while changing the
mechanism to separate the logic in a more sensible manner.
Bugzilla ID: 1460
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/runner.py | 93 ++++------------
dts/framework/test_result.py | 5 +-
dts/framework/test_suite.py | 125 +++++++++++++++++++++-
dts/tests/TestSuite_hello_world.py | 8 +-
dts/tests/TestSuite_os_udp.py | 3 +-
dts/tests/TestSuite_pmd_buffer_scatter.py | 3 +-
dts/tests/TestSuite_smoke_tests.py | 6 +-
7 files changed, 160 insertions(+), 83 deletions(-)
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 6b6f6a05f5..525f119ab6 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -20,11 +20,10 @@
import importlib
import inspect
import os
-import re
import sys
from pathlib import Path
-from types import FunctionType
-from typing import Iterable, Sequence
+from types import MethodType
+from typing import Iterable
from framework.testbed_model.sut_node import SutNode
from framework.testbed_model.tg_node import TGNode
@@ -53,7 +52,7 @@
TestSuiteResult,
TestSuiteWithCases,
)
-from .test_suite import TestSuite
+from .test_suite import TestCase, TestSuite
class DTSRunner:
@@ -232,9 +231,9 @@ def _get_test_suites_with_cases(
for test_suite_config in test_suite_configs:
test_suite_class = self._get_test_suite_class(test_suite_config.test_suite)
- test_cases = []
- func_test_cases, perf_test_cases = self._filter_test_cases(
- test_suite_class, test_suite_config.test_cases
+ test_cases: list[type[TestCase]] = []
+ func_test_cases, perf_test_cases = test_suite_class.get_test_cases(
+ test_suite_config.test_cases
)
if func:
test_cases.extend(func_test_cases)
@@ -309,57 +308,6 @@ def is_test_suite(object) -> bool:
f"Couldn't find any valid test suites in {test_suite_module.__name__}."
)
- def _filter_test_cases(
- self, test_suite_class: type[TestSuite], test_cases_to_run: Sequence[str]
- ) -> tuple[list[FunctionType], list[FunctionType]]:
- """Filter `test_cases_to_run` from `test_suite_class`.
-
- There are two rounds of filtering if `test_cases_to_run` is not empty.
- The first filters `test_cases_to_run` from all methods of `test_suite_class`.
- Then the methods are separated into functional and performance test cases.
- If a method matches neither the functional nor performance name prefix, it's an error.
-
- Args:
- test_suite_class: The class of the test suite.
- test_cases_to_run: Test case names to filter from `test_suite_class`.
- If empty, return all matching test cases.
-
- Returns:
- A list of test case methods that should be executed.
-
- Raises:
- ConfigurationError: If a test case from `test_cases_to_run` is not found
- or it doesn't match either the functional nor performance name prefix.
- """
- func_test_cases = []
- perf_test_cases = []
- name_method_tuples = inspect.getmembers(test_suite_class, inspect.isfunction)
- if test_cases_to_run:
- name_method_tuples = [
- (name, method) for name, method in name_method_tuples if name in test_cases_to_run
- ]
- if len(name_method_tuples) < len(test_cases_to_run):
- missing_test_cases = set(test_cases_to_run) - {
- name for name, _ in name_method_tuples
- }
- raise ConfigurationError(
- f"Test cases {missing_test_cases} not found among methods "
- f"of {test_suite_class.__name__}."
- )
-
- for test_case_name, test_case_method in name_method_tuples:
- if re.match(self._func_test_case_regex, test_case_name):
- func_test_cases.append(test_case_method)
- elif re.match(self._perf_test_case_regex, test_case_name):
- perf_test_cases.append(test_case_method)
- elif test_cases_to_run:
- raise ConfigurationError(
- f"Method '{test_case_name}' matches neither "
- f"a functional nor a performance test case name."
- )
-
- return func_test_cases, perf_test_cases
-
def _connect_nodes_and_run_test_run(
self,
sut_nodes: dict[str, SutNode],
@@ -607,7 +555,7 @@ def _run_test_suite(
def _execute_test_suite(
self,
test_suite: TestSuite,
- test_cases: Iterable[FunctionType],
+ test_cases: Iterable[type[TestCase]],
test_suite_result: TestSuiteResult,
) -> None:
"""Execute all `test_cases` in `test_suite`.
@@ -618,29 +566,29 @@ def _execute_test_suite(
Args:
test_suite: The test suite object.
- test_cases: The list of test case methods.
+ test_cases: The list of test case functions.
test_suite_result: The test suite level result object associated
with the current test suite.
"""
self._logger.set_stage(DtsStage.test_suite)
- for test_case_method in test_cases:
- test_case_name = test_case_method.__name__
+ for test_case in test_cases:
+ test_case_name = test_case.__name__
test_case_result = test_suite_result.add_test_case(test_case_name)
all_attempts = SETTINGS.re_run + 1
attempt_nr = 1
- self._run_test_case(test_suite, test_case_method, test_case_result)
+ self._run_test_case(test_suite, test_case, test_case_result)
while not test_case_result and attempt_nr < all_attempts:
attempt_nr += 1
self._logger.info(
f"Re-running FAILED test case '{test_case_name}'. "
f"Attempt number {attempt_nr} out of {all_attempts}."
)
- self._run_test_case(test_suite, test_case_method, test_case_result)
+ self._run_test_case(test_suite, test_case, test_case_result)
def _run_test_case(
self,
test_suite: TestSuite,
- test_case_method: FunctionType,
+ test_case: type[TestCase],
test_case_result: TestCaseResult,
) -> None:
"""Setup, execute and teardown `test_case_method` from `test_suite`.
@@ -649,11 +597,11 @@ def _run_test_case(
Args:
test_suite: The test suite object.
- test_case_method: The test case method.
+ test_case: The test case function.
test_case_result: The test case level result object associated
with the current test case.
"""
- test_case_name = test_case_method.__name__
+ test_case_name = test_case.__name__
try:
# run set_up function for each case
@@ -668,7 +616,7 @@ def _run_test_case(
else:
# run test case if setup was successful
- self._execute_test_case(test_suite, test_case_method, test_case_result)
+ self._execute_test_case(test_suite, test_case, test_case_result)
finally:
try:
@@ -686,21 +634,22 @@ def _run_test_case(
def _execute_test_case(
self,
test_suite: TestSuite,
- test_case_method: FunctionType,
+ test_case: type[TestCase],
test_case_result: TestCaseResult,
) -> None:
"""Execute `test_case_method` from `test_suite`, record the result and handle failures.
Args:
test_suite: The test suite object.
- test_case_method: The test case method.
+ test_case: The test case function.
test_case_result: The test case level result object associated
with the current test case.
"""
- test_case_name = test_case_method.__name__
+ test_case_name = test_case.__name__
try:
self._logger.info(f"Starting test case execution: {test_case_name}")
- test_case_method(test_suite)
+ # Explicit method binding is required, otherwise mypy complains
+ MethodType(test_case, test_suite)()
test_case_result.update(Result.PASS)
self._logger.info(f"Test case execution PASSED: {test_case_name}")
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 5694a2482b..b1ca584523 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -27,7 +27,6 @@
from collections.abc import MutableSequence
from dataclasses import dataclass
from enum import Enum, auto
-from types import FunctionType
from typing import Union
from .config import (
@@ -44,7 +43,7 @@
from .exception import DTSError, ErrorSeverity
from .logger import DTSLogger
from .settings import SETTINGS
-from .test_suite import TestSuite
+from .test_suite import TestCase, TestSuite
@dataclass(slots=True, frozen=True)
@@ -63,7 +62,7 @@ class is to hold a subset of test cases (which could be all test cases) because
"""
test_suite_class: type[TestSuite]
- test_cases: list[FunctionType]
+ test_cases: list[type[TestCase]]
def create_config(self) -> TestSuiteConfig:
"""Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 694b2eba65..b4ee0f9039 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -13,8 +13,11 @@
* Test case verification.
"""
+import inspect
+from collections.abc import Callable, Sequence
+from enum import Enum, auto
from ipaddress import IPv4Interface, IPv6Interface, ip_interface
-from typing import ClassVar, Union
+from typing import ClassVar, Protocol, TypeVar, Union, cast
from scapy.layers.inet import IP # type: ignore[import-untyped]
from scapy.layers.l2 import Ether # type: ignore[import-untyped]
@@ -27,7 +30,7 @@
PacketFilteringConfig,
)
-from .exception import TestCaseVerifyError
+from .exception import ConfigurationError, TestCaseVerifyError
from .logger import DTSLogger, get_dts_logger
from .utils import get_packet_summaries
@@ -120,6 +123,68 @@ def _process_links(self) -> None:
):
self._port_links.append(PortLink(sut_port=sut_port, tg_port=tg_port))
+ @classmethod
+ def get_test_cases(
+ cls, test_case_sublist: Sequence[str] | None = None
+ ) -> tuple[set[type["TestCase"]], set[type["TestCase"]]]:
+ """Filter `test_case_subset` from this class.
+
+ Test cases are regular (or bound) methods decorated with :func:`func_test`
+ or :func:`perf_test`.
+
+ Args:
+ test_case_sublist: Test case names to filter from this class.
+ If empty or :data:`None`, return all test cases.
+
+ Returns:
+ The filtered test case functions. This method returns functions as opposed to methods,
+ as methods are bound to instances and this method only has access to the class.
+
+ Raises:
+ ConfigurationError: If a test case from `test_case_subset` is not found.
+ """
+
+ def is_test_case(function: Callable) -> bool:
+ if inspect.isfunction(function):
+ # TestCase is not used at runtime, so we can't use isinstance() with `function`.
+ # But function.test_type exists.
+ if hasattr(function, "test_type"):
+ return isinstance(function.test_type, TestCaseType)
+ return False
+
+ if test_case_sublist is None:
+ test_case_sublist = []
+
+ # the copy is needed so that the condition "elif test_case_sublist" doesn't
+ # change mid-cycle
+ test_case_sublist_copy = list(test_case_sublist)
+ func_test_cases = set()
+ perf_test_cases = set()
+
+ for test_case_name, test_case_function in inspect.getmembers(cls, is_test_case):
+ if test_case_name in test_case_sublist_copy:
+ # if test_case_sublist_copy is non-empty, remove the found test case
+ # so that we can look at the remainder at the end
+ test_case_sublist_copy.remove(test_case_name)
+ elif test_case_sublist:
+ # if the original list is not empty (meaning we're filtering test cases),
+ # we're dealing with a test case we would've
+ # removed in the other branch; since we didn't, we don't want to run it
+ continue
+
+ match test_case_function.test_type:
+ case TestCaseType.PERFORMANCE:
+ perf_test_cases.add(test_case_function)
+ case TestCaseType.FUNCTIONAL:
+ func_test_cases.add(test_case_function)
+
+ if test_case_sublist_copy:
+ raise ConfigurationError(
+ f"Test cases {test_case_sublist_copy} not found among functions of {cls.__name__}."
+ )
+
+ return func_test_cases, perf_test_cases
+
def set_up_suite(self) -> None:
"""Set up test fixtures common to all test cases.
@@ -365,3 +430,59 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool:
if received_packet.src != expected_packet.src or received_packet.dst != expected_packet.dst:
return False
return True
+
+
+#: The generic type for a method of an instance of TestSuite
+TestSuiteMethodType = TypeVar("TestSuiteMethodType", bound=Callable[[TestSuite], None])
+
+
+class TestCaseType(Enum):
+ """The types of test cases."""
+
+ #:
+ FUNCTIONAL = auto()
+ #:
+ PERFORMANCE = auto()
+
+
+class TestCase(Protocol[TestSuiteMethodType]):
+ """Definition of the test case type for static type checking purposes.
+
+ The type is applied to test case functions through a decorator, which casts the decorated
+ test case function to :class:`TestCase` and sets common variables.
+ """
+
+ #:
+ test_type: ClassVar[TestCaseType]
+ #: necessary for mypy so that it can treat this class as the function it's shadowing
+ __call__: TestSuiteMethodType
+
+ @classmethod
+ def make_decorator(
+ cls, test_case_type: TestCaseType
+ ) -> Callable[[TestSuiteMethodType], type["TestCase"]]:
+ """Create a decorator for test suites.
+
+ The decorator casts the decorated function as :class:`TestCase`,
+ sets it as `test_case_type`
+ and initializes common variables defined in :class:`RequiresCapabilities`.
+
+ Args:
+ test_case_type: Either a functional or performance test case.
+
+ Returns:
+ The decorator of a functional or performance test case.
+ """
+
+ def _decorator(func: TestSuiteMethodType) -> type[TestCase]:
+ test_case = cast(type[TestCase], func)
+ test_case.test_type = test_case_type
+ return test_case
+
+ return _decorator
+
+
+#: The decorator for functional test cases.
+func_test: Callable = TestCase.make_decorator(TestCaseType.FUNCTIONAL)
+#: The decorator for performance test cases.
+perf_test: Callable = TestCase.make_decorator(TestCaseType.PERFORMANCE)
diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
index d958f99030..16d064ffeb 100644
--- a/dts/tests/TestSuite_hello_world.py
+++ b/dts/tests/TestSuite_hello_world.py
@@ -8,7 +8,7 @@
"""
from framework.remote_session.dpdk_shell import compute_eal_params
-from framework.test_suite import TestSuite
+from framework.test_suite import TestSuite, func_test
from framework.testbed_model.cpu import (
LogicalCoreCount,
LogicalCoreCountFilter,
@@ -27,7 +27,8 @@ def set_up_suite(self) -> None:
"""
self.app_helloworld_path = self.sut_node.build_dpdk_app("helloworld")
- def test_hello_world_single_core(self) -> None:
+ @func_test
+ def hello_world_single_core(self) -> None:
"""Single core test case.
Steps:
@@ -46,7 +47,8 @@ def test_hello_world_single_core(self) -> None:
f"helloworld didn't start on lcore{lcores[0]}",
)
- def test_hello_world_all_cores(self) -> None:
+ @func_test
+ def hello_world_all_cores(self) -> None:
"""All cores test case.
Steps:
diff --git a/dts/tests/TestSuite_os_udp.py b/dts/tests/TestSuite_os_udp.py
index a78bd74139..beaa5f425d 100644
--- a/dts/tests/TestSuite_os_udp.py
+++ b/dts/tests/TestSuite_os_udp.py
@@ -10,7 +10,7 @@
from scapy.layers.inet import IP, UDP # type: ignore[import-untyped]
from scapy.layers.l2 import Ether # type: ignore[import-untyped]
-from framework.test_suite import TestSuite
+from framework.test_suite import TestSuite, func_test
class TestOsUdp(TestSuite):
@@ -26,6 +26,7 @@ def set_up_suite(self) -> None:
self.sut_node.bind_ports_to_driver(for_dpdk=False)
self.configure_testbed_ipv4()
+ @func_test
def test_os_udp(self) -> None:
"""Basic UDP IPv4 traffic test case.
diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
index 0d8e101e5c..020fb0ab62 100644
--- a/dts/tests/TestSuite_pmd_buffer_scatter.py
+++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
@@ -24,7 +24,7 @@
from framework.params.testpmd import SimpleForwardingModes
from framework.remote_session.testpmd_shell import TestPmdShell
-from framework.test_suite import TestSuite
+from framework.test_suite import TestSuite, func_test
class TestPmdBufferScatter(TestSuite):
@@ -123,6 +123,7 @@ def pmd_scatter(self, mbsize: int) -> None:
f"{offset}.",
)
+ @func_test
def test_scatter_mbuf_2048(self) -> None:
"""Run the :meth:`pmd_scatter` test with `mbsize` set to 2048."""
self.pmd_scatter(mbsize=2048)
diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
index c0b0e6bb00..94f90d9327 100644
--- a/dts/tests/TestSuite_smoke_tests.py
+++ b/dts/tests/TestSuite_smoke_tests.py
@@ -17,7 +17,7 @@
from framework.config import PortConfig
from framework.remote_session.testpmd_shell import TestPmdShell
from framework.settings import SETTINGS
-from framework.test_suite import TestSuite
+from framework.test_suite import TestSuite, func_test
from framework.utils import REGEX_FOR_PCI_ADDRESS
@@ -47,6 +47,7 @@ def set_up_suite(self) -> None:
self.dpdk_build_dir_path = self.sut_node.remote_dpdk_build_dir
self.nics_in_node = self.sut_node.config.ports
+ @func_test
def test_unit_tests(self) -> None:
"""DPDK meson ``fast-tests`` unit tests.
@@ -63,6 +64,7 @@ def test_unit_tests(self) -> None:
privileged=True,
)
+ @func_test
def test_driver_tests(self) -> None:
"""DPDK meson ``driver-tests`` unit tests.
@@ -91,6 +93,7 @@ def test_driver_tests(self) -> None:
privileged=True,
)
+ @func_test
def test_devices_listed_in_testpmd(self) -> None:
"""Testpmd device discovery.
@@ -108,6 +111,7 @@ def test_devices_listed_in_testpmd(self) -> None:
"please check your configuration",
)
+ @func_test
def test_device_bound_to_driver(self) -> None:
"""Device driver in OS.
--
2.34.1
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 03/12] dts: add test case decorators
2024-08-21 14:53 ` [PATCH v3 03/12] dts: add test case decorators Juraj Linkeš
@ 2024-08-26 16:50 ` Jeremy Spewock
2024-09-05 8:07 ` Juraj Linkeš
2024-08-28 20:09 ` Dean Marx
2024-08-30 15:50 ` Nicholas Pratte
2 siblings, 1 reply; 107+ messages in thread
From: Jeremy Spewock @ 2024-08-26 16:50 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
<snip>
> class DTSRunner:
> @@ -232,9 +231,9 @@ def _get_test_suites_with_cases(
>
> for test_suite_config in test_suite_configs:
> test_suite_class = self._get_test_suite_class(test_suite_config.test_suite)
> - test_cases = []
> - func_test_cases, perf_test_cases = self._filter_test_cases(
> - test_suite_class, test_suite_config.test_cases
> + test_cases: list[type[TestCase]] = []
If TestCase is just a class, why is the `type[]` in the annotation
required? Are these not specific instances of the TestCase class? I
figured they would need to be in order for you to run the specific
test case methods. Maybe this has something to do with the class being
a Protocol?
> + func_test_cases, perf_test_cases = test_suite_class.get_test_cases(
> + test_suite_config.test_cases
> )
> if func:
> test_cases.extend(func_test_cases)
> @@ -309,57 +308,6 @@ def is_test_suite(object) -> bool:
> f"Couldn't find any valid test suites in {test_suite_module.__name__}."
> )
>
<snip>
> @@ -120,6 +123,68 @@ def _process_links(self) -> None:
> ):
> self._port_links.append(PortLink(sut_port=sut_port, tg_port=tg_port))
>
> + @classmethod
> + def get_test_cases(
> + cls, test_case_sublist: Sequence[str] | None = None
> + ) -> tuple[set[type["TestCase"]], set[type["TestCase"]]]:
> + """Filter `test_case_subset` from this class.
> +
> + Test cases are regular (or bound) methods decorated with :func:`func_test`
> + or :func:`perf_test`.
> +
> + Args:
> + test_case_sublist: Test case names to filter from this class.
> + If empty or :data:`None`, return all test cases.
> +
> + Returns:
> + The filtered test case functions. This method returns functions as opposed to methods,
> + as methods are bound to instances and this method only has access to the class.
> +
> + Raises:
> + ConfigurationError: If a test case from `test_case_subset` is not found.
> + """
> +
<snip>
> + for test_case_name, test_case_function in inspect.getmembers(cls, is_test_case):
> + if test_case_name in test_case_sublist_copy:
> + # if test_case_sublist_copy is non-empty, remove the found test case
> + # so that we can look at the remainder at the end
> + test_case_sublist_copy.remove(test_case_name)
> + elif test_case_sublist:
> + # if the original list is not empty (meaning we're filtering test cases),
> + # we're dealing with a test case we would've
I think this part of the comment about "we're dealing with a test case
we would've removed in the other branch" confused me a little bit. It
could just be a me thing, but I think this would have been more clear
for me if it was something more like "The original list is not empty
(meaning we're filtering test cases). Since we didn't remove this test
case in the other branch, it doesn't match the filter and we don't
want to run it."
> + # removed in the other branch; since we didn't, we don't want to run it
> + continue
> +
> + match test_case_function.test_type:
> + case TestCaseType.PERFORMANCE:
> + perf_test_cases.add(test_case_function)
> + case TestCaseType.FUNCTIONAL:
> + func_test_cases.add(test_case_function)
> +
> + if test_case_sublist_copy:
> + raise ConfigurationError(
> + f"Test cases {test_case_sublist_copy} not found among functions of {cls.__name__}."
> + )
> +
> + return func_test_cases, perf_test_cases
> +
<snip>
> 2.34.1
>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 03/12] dts: add test case decorators
2024-08-26 16:50 ` Jeremy Spewock
@ 2024-09-05 8:07 ` Juraj Linkeš
2024-09-05 15:24 ` Jeremy Spewock
0 siblings, 1 reply; 107+ messages in thread
From: Juraj Linkeš @ 2024-09-05 8:07 UTC (permalink / raw)
To: Jeremy Spewock
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On 26. 8. 2024 18:50, Jeremy Spewock wrote:
> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> <snip>
>> class DTSRunner:
>> @@ -232,9 +231,9 @@ def _get_test_suites_with_cases(
>>
>> for test_suite_config in test_suite_configs:
>> test_suite_class = self._get_test_suite_class(test_suite_config.test_suite)
>> - test_cases = []
>> - func_test_cases, perf_test_cases = self._filter_test_cases(
>> - test_suite_class, test_suite_config.test_cases
>> + test_cases: list[type[TestCase]] = []
>
> If TestCase is just a class, why is the `type[]` in the annotation
> required? Are these not specific instances of the TestCase class? I
> figured they would need to be in order for you to run the specific
> test case methods. Maybe this has something to do with the class being
> a Protocol?
>
The *_test decorators return type[TestCase]. The functions (test
methods) are cast to type[TestCase] (which kinda makes them subclasses
of TestCase).
This was a suggestion from Luca and I took it as as. Maybe the functions
could be cast as instances of TestCase, but I didn't try that.
>> + func_test_cases, perf_test_cases = test_suite_class.get_test_cases(
>> + test_suite_config.test_cases
>> )
>> if func:
>> test_cases.extend(func_test_cases)
>> @@ -309,57 +308,6 @@ def is_test_suite(object) -> bool:
>> f"Couldn't find any valid test suites in {test_suite_module.__name__}."
>> )
>>
> <snip>
>> @@ -120,6 +123,68 @@ def _process_links(self) -> None:
>> ):
>> self._port_links.append(PortLink(sut_port=sut_port, tg_port=tg_port))
>>
>> + @classmethod
>> + def get_test_cases(
>> + cls, test_case_sublist: Sequence[str] | None = None
>> + ) -> tuple[set[type["TestCase"]], set[type["TestCase"]]]:
>> + """Filter `test_case_subset` from this class.
>> +
>> + Test cases are regular (or bound) methods decorated with :func:`func_test`
>> + or :func:`perf_test`.
>> +
>> + Args:
>> + test_case_sublist: Test case names to filter from this class.
>> + If empty or :data:`None`, return all test cases.
>> +
>> + Returns:
>> + The filtered test case functions. This method returns functions as opposed to methods,
>> + as methods are bound to instances and this method only has access to the class.
>> +
>> + Raises:
>> + ConfigurationError: If a test case from `test_case_subset` is not found.
>> + """
>> +
> <snip>
>> + for test_case_name, test_case_function in inspect.getmembers(cls, is_test_case):
>> + if test_case_name in test_case_sublist_copy:
>> + # if test_case_sublist_copy is non-empty, remove the found test case
>> + # so that we can look at the remainder at the end
>> + test_case_sublist_copy.remove(test_case_name)
>> + elif test_case_sublist:
>> + # if the original list is not empty (meaning we're filtering test cases),
>> + # we're dealing with a test case we would've
>
> I think this part of the comment about "we're dealing with a test case
> we would've removed in the other branch" confused me a little bit. It
> could just be a me thing, but I think this would have been more clear
> for me if it was something more like "The original list is not empty
> (meaning we're filtering test cases). Since we didn't remove this test
> case in the other branch, it doesn't match the filter and we don't
> want to run it."
>
We should remove any confusion. I'll change it - your wording sound good.
>> + # removed in the other branch; since we didn't, we don't want to run it
>> + continue
>> +
>> + match test_case_function.test_type:
>> + case TestCaseType.PERFORMANCE:
>> + perf_test_cases.add(test_case_function)
>> + case TestCaseType.FUNCTIONAL:
>> + func_test_cases.add(test_case_function)
>> +
>> + if test_case_sublist_copy:
>> + raise ConfigurationError(
>> + f"Test cases {test_case_sublist_copy} not found among functions of {cls.__name__}."
>> + )
>> +
>> + return func_test_cases, perf_test_cases
>> +
> <snip>
>> 2.34.1
>>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 03/12] dts: add test case decorators
2024-09-05 8:07 ` Juraj Linkeš
@ 2024-09-05 15:24 ` Jeremy Spewock
0 siblings, 0 replies; 107+ messages in thread
From: Jeremy Spewock @ 2024-09-05 15:24 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Thu, Sep 5, 2024 at 4:07 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>
>
>
> On 26. 8. 2024 18:50, Jeremy Spewock wrote:
> > On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> > <juraj.linkes@pantheon.tech> wrote:
> > <snip>
> >> class DTSRunner:
> >> @@ -232,9 +231,9 @@ def _get_test_suites_with_cases(
> >>
> >> for test_suite_config in test_suite_configs:
> >> test_suite_class = self._get_test_suite_class(test_suite_config.test_suite)
> >> - test_cases = []
> >> - func_test_cases, perf_test_cases = self._filter_test_cases(
> >> - test_suite_class, test_suite_config.test_cases
> >> + test_cases: list[type[TestCase]] = []
> >
> > If TestCase is just a class, why is the `type[]` in the annotation
> > required? Are these not specific instances of the TestCase class? I
> > figured they would need to be in order for you to run the specific
> > test case methods. Maybe this has something to do with the class being
> > a Protocol?
> >
>
> The *_test decorators return type[TestCase]. The functions (test
> methods) are cast to type[TestCase] (which kinda makes them subclasses
> of TestCase).
Oh interesting, I didn't make the connection that casting them to
type[TestCase] was similar to having them be subclasses of the type,
but this actually makes a lot of sense. Thank you for the
clarification!
>
> This was a suggestion from Luca and I took it as as. Maybe the functions
> could be cast as instances of TestCase, but I didn't try that.
Right, I would think that they could be cast directly to it, but
there's no need obviously so that makes sense.
>
> >> + func_test_cases, perf_test_cases = test_suite_class.get_test_cases(
> >> + test_suite_config.test_cases
> >> )
> >> if func:
> >> test_cases.extend(func_test_cases)
<snip>
>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 03/12] dts: add test case decorators
2024-08-21 14:53 ` [PATCH v3 03/12] dts: add test case decorators Juraj Linkeš
2024-08-26 16:50 ` Jeremy Spewock
@ 2024-08-28 20:09 ` Dean Marx
2024-08-30 15:50 ` Nicholas Pratte
2 siblings, 0 replies; 107+ messages in thread
From: Dean Marx @ 2024-08-28 20:09 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 779 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> Add decorators for functional and performance test cases. These
> decorators add attributes to the decorated test cases.
>
> With the addition of decorators, we change the test case discovery
> mechanism from looking at test case names according to a regex to simply
> checking an attribute of the function added with one of the decorators.
>
> The decorators allow us to add further variables to test cases.
>
> Also move the test case filtering to TestSuite while changing the
> mechanism to separate the logic in a more sensible manner.
>
> Bugzilla ID: 1460
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 1098 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 03/12] dts: add test case decorators
2024-08-21 14:53 ` [PATCH v3 03/12] dts: add test case decorators Juraj Linkeš
2024-08-26 16:50 ` Jeremy Spewock
2024-08-28 20:09 ` Dean Marx
@ 2024-08-30 15:50 ` Nicholas Pratte
2 siblings, 0 replies; 107+ messages in thread
From: Nicholas Pratte @ 2024-08-30 15:50 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, dmarx, alex.chapman, dev
Reviewed-by: Nicholas Pratte <npratte@iol.unh.edu>
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
> Add decorators for functional and performance test cases. These
> decorators add attributes to the decorated test cases.
>
> With the addition of decorators, we change the test case discovery
> mechanism from looking at test case names according to a regex to simply
> checking an attribute of the function added with one of the decorators.
>
> The decorators allow us to add further variables to test cases.
>
> Also move the test case filtering to TestSuite while changing the
> mechanism to separate the logic in a more sensible manner.
>
> Bugzilla ID: 1460
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
> dts/framework/runner.py | 93 ++++------------
> dts/framework/test_result.py | 5 +-
> dts/framework/test_suite.py | 125 +++++++++++++++++++++-
> dts/tests/TestSuite_hello_world.py | 8 +-
> dts/tests/TestSuite_os_udp.py | 3 +-
> dts/tests/TestSuite_pmd_buffer_scatter.py | 3 +-
> dts/tests/TestSuite_smoke_tests.py | 6 +-
> 7 files changed, 160 insertions(+), 83 deletions(-)
>
> diff --git a/dts/framework/runner.py b/dts/framework/runner.py
> index 6b6f6a05f5..525f119ab6 100644
> --- a/dts/framework/runner.py
> +++ b/dts/framework/runner.py
> @@ -20,11 +20,10 @@
> import importlib
> import inspect
> import os
> -import re
> import sys
> from pathlib import Path
> -from types import FunctionType
> -from typing import Iterable, Sequence
> +from types import MethodType
> +from typing import Iterable
>
> from framework.testbed_model.sut_node import SutNode
> from framework.testbed_model.tg_node import TGNode
> @@ -53,7 +52,7 @@
> TestSuiteResult,
> TestSuiteWithCases,
> )
> -from .test_suite import TestSuite
> +from .test_suite import TestCase, TestSuite
>
>
> class DTSRunner:
> @@ -232,9 +231,9 @@ def _get_test_suites_with_cases(
>
> for test_suite_config in test_suite_configs:
> test_suite_class = self._get_test_suite_class(test_suite_config.test_suite)
> - test_cases = []
> - func_test_cases, perf_test_cases = self._filter_test_cases(
> - test_suite_class, test_suite_config.test_cases
> + test_cases: list[type[TestCase]] = []
> + func_test_cases, perf_test_cases = test_suite_class.get_test_cases(
> + test_suite_config.test_cases
> )
> if func:
> test_cases.extend(func_test_cases)
> @@ -309,57 +308,6 @@ def is_test_suite(object) -> bool:
> f"Couldn't find any valid test suites in {test_suite_module.__name__}."
> )
>
> - def _filter_test_cases(
> - self, test_suite_class: type[TestSuite], test_cases_to_run: Sequence[str]
> - ) -> tuple[list[FunctionType], list[FunctionType]]:
> - """Filter `test_cases_to_run` from `test_suite_class`.
> -
> - There are two rounds of filtering if `test_cases_to_run` is not empty.
> - The first filters `test_cases_to_run` from all methods of `test_suite_class`.
> - Then the methods are separated into functional and performance test cases.
> - If a method matches neither the functional nor performance name prefix, it's an error.
> -
> - Args:
> - test_suite_class: The class of the test suite.
> - test_cases_to_run: Test case names to filter from `test_suite_class`.
> - If empty, return all matching test cases.
> -
> - Returns:
> - A list of test case methods that should be executed.
> -
> - Raises:
> - ConfigurationError: If a test case from `test_cases_to_run` is not found
> - or it doesn't match either the functional nor performance name prefix.
> - """
> - func_test_cases = []
> - perf_test_cases = []
> - name_method_tuples = inspect.getmembers(test_suite_class, inspect.isfunction)
> - if test_cases_to_run:
> - name_method_tuples = [
> - (name, method) for name, method in name_method_tuples if name in test_cases_to_run
> - ]
> - if len(name_method_tuples) < len(test_cases_to_run):
> - missing_test_cases = set(test_cases_to_run) - {
> - name for name, _ in name_method_tuples
> - }
> - raise ConfigurationError(
> - f"Test cases {missing_test_cases} not found among methods "
> - f"of {test_suite_class.__name__}."
> - )
> -
> - for test_case_name, test_case_method in name_method_tuples:
> - if re.match(self._func_test_case_regex, test_case_name):
> - func_test_cases.append(test_case_method)
> - elif re.match(self._perf_test_case_regex, test_case_name):
> - perf_test_cases.append(test_case_method)
> - elif test_cases_to_run:
> - raise ConfigurationError(
> - f"Method '{test_case_name}' matches neither "
> - f"a functional nor a performance test case name."
> - )
> -
> - return func_test_cases, perf_test_cases
> -
> def _connect_nodes_and_run_test_run(
> self,
> sut_nodes: dict[str, SutNode],
> @@ -607,7 +555,7 @@ def _run_test_suite(
> def _execute_test_suite(
> self,
> test_suite: TestSuite,
> - test_cases: Iterable[FunctionType],
> + test_cases: Iterable[type[TestCase]],
> test_suite_result: TestSuiteResult,
> ) -> None:
> """Execute all `test_cases` in `test_suite`.
> @@ -618,29 +566,29 @@ def _execute_test_suite(
>
> Args:
> test_suite: The test suite object.
> - test_cases: The list of test case methods.
> + test_cases: The list of test case functions.
> test_suite_result: The test suite level result object associated
> with the current test suite.
> """
> self._logger.set_stage(DtsStage.test_suite)
> - for test_case_method in test_cases:
> - test_case_name = test_case_method.__name__
> + for test_case in test_cases:
> + test_case_name = test_case.__name__
> test_case_result = test_suite_result.add_test_case(test_case_name)
> all_attempts = SETTINGS.re_run + 1
> attempt_nr = 1
> - self._run_test_case(test_suite, test_case_method, test_case_result)
> + self._run_test_case(test_suite, test_case, test_case_result)
> while not test_case_result and attempt_nr < all_attempts:
> attempt_nr += 1
> self._logger.info(
> f"Re-running FAILED test case '{test_case_name}'. "
> f"Attempt number {attempt_nr} out of {all_attempts}."
> )
> - self._run_test_case(test_suite, test_case_method, test_case_result)
> + self._run_test_case(test_suite, test_case, test_case_result)
>
> def _run_test_case(
> self,
> test_suite: TestSuite,
> - test_case_method: FunctionType,
> + test_case: type[TestCase],
> test_case_result: TestCaseResult,
> ) -> None:
> """Setup, execute and teardown `test_case_method` from `test_suite`.
> @@ -649,11 +597,11 @@ def _run_test_case(
>
> Args:
> test_suite: The test suite object.
> - test_case_method: The test case method.
> + test_case: The test case function.
> test_case_result: The test case level result object associated
> with the current test case.
> """
> - test_case_name = test_case_method.__name__
> + test_case_name = test_case.__name__
>
> try:
> # run set_up function for each case
> @@ -668,7 +616,7 @@ def _run_test_case(
>
> else:
> # run test case if setup was successful
> - self._execute_test_case(test_suite, test_case_method, test_case_result)
> + self._execute_test_case(test_suite, test_case, test_case_result)
>
> finally:
> try:
> @@ -686,21 +634,22 @@ def _run_test_case(
> def _execute_test_case(
> self,
> test_suite: TestSuite,
> - test_case_method: FunctionType,
> + test_case: type[TestCase],
> test_case_result: TestCaseResult,
> ) -> None:
> """Execute `test_case_method` from `test_suite`, record the result and handle failures.
>
> Args:
> test_suite: The test suite object.
> - test_case_method: The test case method.
> + test_case: The test case function.
> test_case_result: The test case level result object associated
> with the current test case.
> """
> - test_case_name = test_case_method.__name__
> + test_case_name = test_case.__name__
> try:
> self._logger.info(f"Starting test case execution: {test_case_name}")
> - test_case_method(test_suite)
> + # Explicit method binding is required, otherwise mypy complains
> + MethodType(test_case, test_suite)()
> test_case_result.update(Result.PASS)
> self._logger.info(f"Test case execution PASSED: {test_case_name}")
>
> diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
> index 5694a2482b..b1ca584523 100644
> --- a/dts/framework/test_result.py
> +++ b/dts/framework/test_result.py
> @@ -27,7 +27,6 @@
> from collections.abc import MutableSequence
> from dataclasses import dataclass
> from enum import Enum, auto
> -from types import FunctionType
> from typing import Union
>
> from .config import (
> @@ -44,7 +43,7 @@
> from .exception import DTSError, ErrorSeverity
> from .logger import DTSLogger
> from .settings import SETTINGS
> -from .test_suite import TestSuite
> +from .test_suite import TestCase, TestSuite
>
>
> @dataclass(slots=True, frozen=True)
> @@ -63,7 +62,7 @@ class is to hold a subset of test cases (which could be all test cases) because
> """
>
> test_suite_class: type[TestSuite]
> - test_cases: list[FunctionType]
> + test_cases: list[type[TestCase]]
>
> def create_config(self) -> TestSuiteConfig:
> """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
> diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
> index 694b2eba65..b4ee0f9039 100644
> --- a/dts/framework/test_suite.py
> +++ b/dts/framework/test_suite.py
> @@ -13,8 +13,11 @@
> * Test case verification.
> """
>
> +import inspect
> +from collections.abc import Callable, Sequence
> +from enum import Enum, auto
> from ipaddress import IPv4Interface, IPv6Interface, ip_interface
> -from typing import ClassVar, Union
> +from typing import ClassVar, Protocol, TypeVar, Union, cast
>
> from scapy.layers.inet import IP # type: ignore[import-untyped]
> from scapy.layers.l2 import Ether # type: ignore[import-untyped]
> @@ -27,7 +30,7 @@
> PacketFilteringConfig,
> )
>
> -from .exception import TestCaseVerifyError
> +from .exception import ConfigurationError, TestCaseVerifyError
> from .logger import DTSLogger, get_dts_logger
> from .utils import get_packet_summaries
>
> @@ -120,6 +123,68 @@ def _process_links(self) -> None:
> ):
> self._port_links.append(PortLink(sut_port=sut_port, tg_port=tg_port))
>
> + @classmethod
> + def get_test_cases(
> + cls, test_case_sublist: Sequence[str] | None = None
> + ) -> tuple[set[type["TestCase"]], set[type["TestCase"]]]:
> + """Filter `test_case_subset` from this class.
> +
> + Test cases are regular (or bound) methods decorated with :func:`func_test`
> + or :func:`perf_test`.
> +
> + Args:
> + test_case_sublist: Test case names to filter from this class.
> + If empty or :data:`None`, return all test cases.
> +
> + Returns:
> + The filtered test case functions. This method returns functions as opposed to methods,
> + as methods are bound to instances and this method only has access to the class.
> +
> + Raises:
> + ConfigurationError: If a test case from `test_case_subset` is not found.
> + """
> +
> + def is_test_case(function: Callable) -> bool:
> + if inspect.isfunction(function):
> + # TestCase is not used at runtime, so we can't use isinstance() with `function`.
> + # But function.test_type exists.
> + if hasattr(function, "test_type"):
> + return isinstance(function.test_type, TestCaseType)
> + return False
> +
> + if test_case_sublist is None:
> + test_case_sublist = []
> +
> + # the copy is needed so that the condition "elif test_case_sublist" doesn't
> + # change mid-cycle
> + test_case_sublist_copy = list(test_case_sublist)
> + func_test_cases = set()
> + perf_test_cases = set()
> +
> + for test_case_name, test_case_function in inspect.getmembers(cls, is_test_case):
> + if test_case_name in test_case_sublist_copy:
> + # if test_case_sublist_copy is non-empty, remove the found test case
> + # so that we can look at the remainder at the end
> + test_case_sublist_copy.remove(test_case_name)
> + elif test_case_sublist:
> + # if the original list is not empty (meaning we're filtering test cases),
> + # we're dealing with a test case we would've
> + # removed in the other branch; since we didn't, we don't want to run it
> + continue
> +
> + match test_case_function.test_type:
> + case TestCaseType.PERFORMANCE:
> + perf_test_cases.add(test_case_function)
> + case TestCaseType.FUNCTIONAL:
> + func_test_cases.add(test_case_function)
> +
> + if test_case_sublist_copy:
> + raise ConfigurationError(
> + f"Test cases {test_case_sublist_copy} not found among functions of {cls.__name__}."
> + )
> +
> + return func_test_cases, perf_test_cases
> +
> def set_up_suite(self) -> None:
> """Set up test fixtures common to all test cases.
>
> @@ -365,3 +430,59 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool:
> if received_packet.src != expected_packet.src or received_packet.dst != expected_packet.dst:
> return False
> return True
> +
> +
> +#: The generic type for a method of an instance of TestSuite
> +TestSuiteMethodType = TypeVar("TestSuiteMethodType", bound=Callable[[TestSuite], None])
> +
> +
> +class TestCaseType(Enum):
> + """The types of test cases."""
> +
> + #:
> + FUNCTIONAL = auto()
> + #:
> + PERFORMANCE = auto()
> +
> +
> +class TestCase(Protocol[TestSuiteMethodType]):
> + """Definition of the test case type for static type checking purposes.
> +
> + The type is applied to test case functions through a decorator, which casts the decorated
> + test case function to :class:`TestCase` and sets common variables.
> + """
> +
> + #:
> + test_type: ClassVar[TestCaseType]
> + #: necessary for mypy so that it can treat this class as the function it's shadowing
> + __call__: TestSuiteMethodType
> +
> + @classmethod
> + def make_decorator(
> + cls, test_case_type: TestCaseType
> + ) -> Callable[[TestSuiteMethodType], type["TestCase"]]:
> + """Create a decorator for test suites.
> +
> + The decorator casts the decorated function as :class:`TestCase`,
> + sets it as `test_case_type`
> + and initializes common variables defined in :class:`RequiresCapabilities`.
> +
> + Args:
> + test_case_type: Either a functional or performance test case.
> +
> + Returns:
> + The decorator of a functional or performance test case.
> + """
> +
> + def _decorator(func: TestSuiteMethodType) -> type[TestCase]:
> + test_case = cast(type[TestCase], func)
> + test_case.test_type = test_case_type
> + return test_case
> +
> + return _decorator
> +
> +
> +#: The decorator for functional test cases.
> +func_test: Callable = TestCase.make_decorator(TestCaseType.FUNCTIONAL)
> +#: The decorator for performance test cases.
> +perf_test: Callable = TestCase.make_decorator(TestCaseType.PERFORMANCE)
> diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
> index d958f99030..16d064ffeb 100644
> --- a/dts/tests/TestSuite_hello_world.py
> +++ b/dts/tests/TestSuite_hello_world.py
> @@ -8,7 +8,7 @@
> """
>
> from framework.remote_session.dpdk_shell import compute_eal_params
> -from framework.test_suite import TestSuite
> +from framework.test_suite import TestSuite, func_test
> from framework.testbed_model.cpu import (
> LogicalCoreCount,
> LogicalCoreCountFilter,
> @@ -27,7 +27,8 @@ def set_up_suite(self) -> None:
> """
> self.app_helloworld_path = self.sut_node.build_dpdk_app("helloworld")
>
> - def test_hello_world_single_core(self) -> None:
> + @func_test
> + def hello_world_single_core(self) -> None:
> """Single core test case.
>
> Steps:
> @@ -46,7 +47,8 @@ def test_hello_world_single_core(self) -> None:
> f"helloworld didn't start on lcore{lcores[0]}",
> )
>
> - def test_hello_world_all_cores(self) -> None:
> + @func_test
> + def hello_world_all_cores(self) -> None:
> """All cores test case.
>
> Steps:
> diff --git a/dts/tests/TestSuite_os_udp.py b/dts/tests/TestSuite_os_udp.py
> index a78bd74139..beaa5f425d 100644
> --- a/dts/tests/TestSuite_os_udp.py
> +++ b/dts/tests/TestSuite_os_udp.py
> @@ -10,7 +10,7 @@
> from scapy.layers.inet import IP, UDP # type: ignore[import-untyped]
> from scapy.layers.l2 import Ether # type: ignore[import-untyped]
>
> -from framework.test_suite import TestSuite
> +from framework.test_suite import TestSuite, func_test
>
>
> class TestOsUdp(TestSuite):
> @@ -26,6 +26,7 @@ def set_up_suite(self) -> None:
> self.sut_node.bind_ports_to_driver(for_dpdk=False)
> self.configure_testbed_ipv4()
>
> + @func_test
> def test_os_udp(self) -> None:
> """Basic UDP IPv4 traffic test case.
>
> diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
> index 0d8e101e5c..020fb0ab62 100644
> --- a/dts/tests/TestSuite_pmd_buffer_scatter.py
> +++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
> @@ -24,7 +24,7 @@
>
> from framework.params.testpmd import SimpleForwardingModes
> from framework.remote_session.testpmd_shell import TestPmdShell
> -from framework.test_suite import TestSuite
> +from framework.test_suite import TestSuite, func_test
>
>
> class TestPmdBufferScatter(TestSuite):
> @@ -123,6 +123,7 @@ def pmd_scatter(self, mbsize: int) -> None:
> f"{offset}.",
> )
>
> + @func_test
> def test_scatter_mbuf_2048(self) -> None:
> """Run the :meth:`pmd_scatter` test with `mbsize` set to 2048."""
> self.pmd_scatter(mbsize=2048)
> diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
> index c0b0e6bb00..94f90d9327 100644
> --- a/dts/tests/TestSuite_smoke_tests.py
> +++ b/dts/tests/TestSuite_smoke_tests.py
> @@ -17,7 +17,7 @@
> from framework.config import PortConfig
> from framework.remote_session.testpmd_shell import TestPmdShell
> from framework.settings import SETTINGS
> -from framework.test_suite import TestSuite
> +from framework.test_suite import TestSuite, func_test
> from framework.utils import REGEX_FOR_PCI_ADDRESS
>
>
> @@ -47,6 +47,7 @@ def set_up_suite(self) -> None:
> self.dpdk_build_dir_path = self.sut_node.remote_dpdk_build_dir
> self.nics_in_node = self.sut_node.config.ports
>
> + @func_test
> def test_unit_tests(self) -> None:
> """DPDK meson ``fast-tests`` unit tests.
>
> @@ -63,6 +64,7 @@ def test_unit_tests(self) -> None:
> privileged=True,
> )
>
> + @func_test
> def test_driver_tests(self) -> None:
> """DPDK meson ``driver-tests`` unit tests.
>
> @@ -91,6 +93,7 @@ def test_driver_tests(self) -> None:
> privileged=True,
> )
>
> + @func_test
> def test_devices_listed_in_testpmd(self) -> None:
> """Testpmd device discovery.
>
> @@ -108,6 +111,7 @@ def test_devices_listed_in_testpmd(self) -> None:
> "please check your configuration",
> )
>
> + @func_test
> def test_device_bound_to_driver(self) -> None:
> """Device driver in OS.
>
> --
> 2.34.1
>
^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH v3 04/12] dts: add mechanism to skip test cases or suites
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
` (2 preceding siblings ...)
2024-08-21 14:53 ` [PATCH v3 03/12] dts: add test case decorators Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 16:52 ` Jeremy Spewock
2024-08-28 20:37 ` Dean Marx
2024-08-21 14:53 ` [PATCH v3 05/12] dts: add support for simpler topologies Juraj Linkeš
` (8 subsequent siblings)
12 siblings, 2 replies; 107+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
If a test case is not relevant to the testing environment (such as when
a NIC doesn't support a tested feature), the framework should skip it.
The mechanism is a skeleton without actual logic that would set a test
case or suite to be skipped.
The mechanism uses a protocol to extend test suites and test cases with
additional attributes that track whether the test case or suite should
be skipped the reason for skipping it.
Also update the results module with the new SKIP result.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/runner.py | 34 +++++++---
dts/framework/test_result.py | 77 ++++++++++++++---------
dts/framework/test_suite.py | 7 ++-
dts/framework/testbed_model/capability.py | 28 +++++++++
4 files changed, 109 insertions(+), 37 deletions(-)
create mode 100644 dts/framework/testbed_model/capability.py
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 525f119ab6..55357ea1fe 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -477,7 +477,20 @@ def _run_test_suites(
for test_suite_with_cases in test_suites_with_cases:
test_suite_result = build_target_result.add_test_suite(test_suite_with_cases)
try:
- self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases)
+ if not test_suite_with_cases.skip:
+ self._run_test_suite(
+ sut_node,
+ tg_node,
+ test_suite_result,
+ test_suite_with_cases,
+ )
+ else:
+ self._logger.info(
+ f"Test suite execution SKIPPED: "
+ f"'{test_suite_with_cases.test_suite_class.__name__}'. Reason: "
+ f"{test_suite_with_cases.test_suite_class.skip_reason}"
+ )
+ test_suite_result.update_setup(Result.SKIP)
except BlockingTestSuiteError as e:
self._logger.exception(
f"An error occurred within {test_suite_with_cases.test_suite_class.__name__}. "
@@ -576,14 +589,21 @@ def _execute_test_suite(
test_case_result = test_suite_result.add_test_case(test_case_name)
all_attempts = SETTINGS.re_run + 1
attempt_nr = 1
- self._run_test_case(test_suite, test_case, test_case_result)
- while not test_case_result and attempt_nr < all_attempts:
- attempt_nr += 1
+ if not test_case.skip:
+ self._run_test_case(test_suite, test_case, test_case_result)
+ while not test_case_result and attempt_nr < all_attempts:
+ attempt_nr += 1
+ self._logger.info(
+ f"Re-running FAILED test case '{test_case_name}'. "
+ f"Attempt number {attempt_nr} out of {all_attempts}."
+ )
+ self._run_test_case(test_suite, test_case, test_case_result)
+ else:
self._logger.info(
- f"Re-running FAILED test case '{test_case_name}'. "
- f"Attempt number {attempt_nr} out of {all_attempts}."
+ f"Test case execution SKIPPED: {test_case_name}. Reason: "
+ f"{test_case.skip_reason}"
)
- self._run_test_case(test_suite, test_case, test_case_result)
+ test_case_result.update_setup(Result.SKIP)
def _run_test_case(
self,
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index b1ca584523..306b100bc6 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -75,6 +75,20 @@ def create_config(self) -> TestSuiteConfig:
test_cases=[test_case.__name__ for test_case in self.test_cases],
)
+ @property
+ def skip(self) -> bool:
+ """Skip the test suite if all test cases or the suite itself are to be skipped.
+
+ Returns:
+ :data:`True` if the test suite should be skipped, :data:`False` otherwise.
+ """
+ all_test_cases_skipped = True
+ for test_case in self.test_cases:
+ if not test_case.skip:
+ all_test_cases_skipped = False
+ break
+ return all_test_cases_skipped or self.test_suite_class.skip
+
class Result(Enum):
"""The possible states that a setup, a teardown or a test case may end up in."""
@@ -86,12 +100,12 @@ class Result(Enum):
#:
ERROR = auto()
#:
- SKIP = auto()
- #:
BLOCK = auto()
+ #:
+ SKIP = auto()
def __bool__(self) -> bool:
- """Only PASS is True."""
+ """Only :attr:`PASS` is True."""
return self is self.PASS
@@ -169,12 +183,13 @@ def update_setup(self, result: Result, error: Exception | None = None) -> None:
self.setup_result.result = result
self.setup_result.error = error
- if result in [Result.BLOCK, Result.ERROR, Result.FAIL]:
- self.update_teardown(Result.BLOCK)
- self._block_result()
+ if result != Result.PASS:
+ result_to_mark = Result.BLOCK if result != Result.SKIP else Result.SKIP
+ self.update_teardown(result_to_mark)
+ self._mark_results(result_to_mark)
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed.
+ def _mark_results(self, result) -> None:
+ """Mark the result as well as the child result as `result`.
The blocking of child results should be done in overloaded methods.
"""
@@ -391,11 +406,11 @@ def add_sut_info(self, sut_info: NodeInfo) -> None:
self.sut_os_version = sut_info.os_version
self.sut_kernel_version = sut_info.kernel_version
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ def _mark_results(self, result) -> None:
+ """Mark the result as well as the child result as `result`."""
for build_target in self._config.build_targets:
child_result = self.add_build_target(build_target)
- child_result.update_setup(Result.BLOCK)
+ child_result.update_setup(result)
class BuildTargetResult(BaseResult):
@@ -465,11 +480,11 @@ def add_build_target_info(self, versions: BuildTargetInfo) -> None:
self.compiler_version = versions.compiler_version
self.dpdk_version = versions.dpdk_version
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ def _mark_results(self, result) -> None:
+ """Mark the result as well as the child result as `result`."""
for test_suite_with_cases in self._test_suites_with_cases:
child_result = self.add_test_suite(test_suite_with_cases)
- child_result.update_setup(Result.BLOCK)
+ child_result.update_setup(result)
class TestSuiteResult(BaseResult):
@@ -509,11 +524,11 @@ def add_test_case(self, test_case_name: str) -> "TestCaseResult":
self.child_results.append(result)
return result
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ def _mark_results(self, result) -> None:
+ """Mark the result as well as the child result as `result`."""
for test_case_method in self._test_suite_with_cases.test_cases:
child_result = self.add_test_case(test_case_method.__name__)
- child_result.update_setup(Result.BLOCK)
+ child_result.update_setup(result)
class TestCaseResult(BaseResult, FixtureResult):
@@ -567,9 +582,9 @@ def add_stats(self, statistics: "Statistics") -> None:
"""
statistics += self.result
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
- self.update(Result.BLOCK)
+ def _mark_results(self, result) -> None:
+ r"""Mark the result as `result`."""
+ self.update(result)
def __bool__(self) -> bool:
"""The test case passed only if setup, teardown and the test case itself passed."""
@@ -583,7 +598,8 @@ class Statistics(dict):
The data are stored in the following keys:
- * **PASS RATE** (:class:`int`) -- The FAIL/PASS ratio of all test cases.
+ * **PASS RATE** (:class:`int`) -- The :attr:`~Result.FAIL`/:attr:`~Result.PASS` ratio
+ of all test cases.
* **DPDK VERSION** (:class:`str`) -- The tested DPDK version.
"""
@@ -600,22 +616,27 @@ def __init__(self, dpdk_version: str | None):
self["DPDK VERSION"] = dpdk_version
def __iadd__(self, other: Result) -> "Statistics":
- """Add a Result to the final count.
+ """Add a :class:`Result` to the final count.
+
+ :attr:`~Result.SKIP` is not taken into account
Example:
- stats: Statistics = Statistics() # empty Statistics
- stats += Result.PASS # add a Result to `stats`
+ stats: Statistics = Statistics() # empty :class:`Statistics`
+ stats += Result.PASS # add a :class:`Result` to `stats`
Args:
- other: The Result to add to this statistics object.
+ other: The :class:`Result` to add to this statistics object.
Returns:
The modified statistics object.
"""
self[other.name] += 1
- self["PASS RATE"] = (
- float(self[Result.PASS.name]) * 100 / sum(self[result.name] for result in Result)
- )
+ if other != Result.SKIP:
+ self["PASS RATE"] = (
+ float(self[Result.PASS.name])
+ * 100
+ / sum([self[result.name] for result in Result if result != Result.SKIP])
+ )
return self
def __str__(self) -> str:
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index b4ee0f9039..c59fc9c6e6 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -23,6 +23,7 @@
from scapy.layers.l2 import Ether # type: ignore[import-untyped]
from scapy.packet import Packet, Padding # type: ignore[import-untyped]
+from framework.testbed_model.capability import TestProtocol
from framework.testbed_model.port import Port, PortLink
from framework.testbed_model.sut_node import SutNode
from framework.testbed_model.tg_node import TGNode
@@ -35,7 +36,7 @@
from .utils import get_packet_summaries
-class TestSuite:
+class TestSuite(TestProtocol):
"""The base class with building blocks needed by most test cases.
* Test suite setup/cleanup methods to override,
@@ -445,7 +446,7 @@ class TestCaseType(Enum):
PERFORMANCE = auto()
-class TestCase(Protocol[TestSuiteMethodType]):
+class TestCase(TestProtocol, Protocol[TestSuiteMethodType]):
"""Definition of the test case type for static type checking purposes.
The type is applied to test case functions through a decorator, which casts the decorated
@@ -476,6 +477,8 @@ def make_decorator(
def _decorator(func: TestSuiteMethodType) -> type[TestCase]:
test_case = cast(type[TestCase], func)
+ test_case.skip = cls.skip
+ test_case.skip_reason = cls.skip_reason
test_case.test_type = test_case_type
return test_case
diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py
new file mode 100644
index 0000000000..662f691a0e
--- /dev/null
+++ b/dts/framework/testbed_model/capability.py
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 PANTHEON.tech s.r.o.
+
+"""Testbed capabilities.
+
+This module provides a protocol that defines the common attributes of test cases and suites.
+"""
+
+from collections.abc import Sequence
+from typing import ClassVar, Protocol
+
+
+class TestProtocol(Protocol):
+ """Common test suite and test case attributes."""
+
+ #: Whether to skip the test case or suite.
+ skip: ClassVar[bool] = False
+ #: The reason for skipping the test case or suite.
+ skip_reason: ClassVar[str] = ""
+
+ @classmethod
+ def get_test_cases(cls, test_case_sublist: Sequence[str] | None = None) -> tuple[set, set]:
+ """Get test cases. Should be implemented by subclasses containing test cases.
+
+ Raises:
+ NotImplementedError: The subclass does not implement the method.
+ """
+ raise NotImplementedError()
--
2.34.1
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 04/12] dts: add mechanism to skip test cases or suites
2024-08-21 14:53 ` [PATCH v3 04/12] dts: add mechanism to skip test cases or suites Juraj Linkeš
@ 2024-08-26 16:52 ` Jeremy Spewock
2024-09-05 9:23 ` Juraj Linkeš
2024-08-28 20:37 ` Dean Marx
1 sibling, 1 reply; 107+ messages in thread
From: Jeremy Spewock @ 2024-08-26 16:52 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
<snip>
> --- a/dts/framework/test_result.py
> +++ b/dts/framework/test_result.py
> @@ -75,6 +75,20 @@ def create_config(self) -> TestSuiteConfig:
> test_cases=[test_case.__name__ for test_case in self.test_cases],
> )
>
> + @property
> + def skip(self) -> bool:
> + """Skip the test suite if all test cases or the suite itself are to be skipped.
> +
> + Returns:
> + :data:`True` if the test suite should be skipped, :data:`False` otherwise.
> + """
> + all_test_cases_skipped = True
> + for test_case in self.test_cases:
> + if not test_case.skip:
> + all_test_cases_skipped = False
> + break
You could also potentially implement this using the built-in `all()`
function. It would become a simple one-liner like
`all_test_cases_skipped = all(test_case.skip for test_case in
self.test_cases)`. That's probably short enough to even just put in
the return statement though if you wanted to.
> + return all_test_cases_skipped or self.test_suite_class.skip
> +
>
> class Result(Enum):
> """The possible states that a setup, a teardown or a test case may end up in."""
> @@ -86,12 +100,12 @@ class Result(Enum):
> #:
> ERROR = auto()
> #:
> - SKIP = auto()
> - #:
> BLOCK = auto()
> + #:
> + SKIP = auto()
>
> def __bool__(self) -> bool:
> - """Only PASS is True."""
> + """Only :attr:`PASS` is True."""
> return self is self.PASS
>
>
> @@ -169,12 +183,13 @@ def update_setup(self, result: Result, error: Exception | None = None) -> None:
> self.setup_result.result = result
> self.setup_result.error = error
>
> - if result in [Result.BLOCK, Result.ERROR, Result.FAIL]:
> - self.update_teardown(Result.BLOCK)
> - self._block_result()
> + if result != Result.PASS:
> + result_to_mark = Result.BLOCK if result != Result.SKIP else Result.SKIP
> + self.update_teardown(result_to_mark)
> + self._mark_results(result_to_mark)
>
> - def _block_result(self) -> None:
> - r"""Mark the result as :attr:`~Result.BLOCK`\ed.
> + def _mark_results(self, result) -> None:
Is it worth adding the type annotation for `result` here and to the
other places where this is implemented? I guess it doesn't matter that
much since it is a private method.
> + """Mark the result as well as the child result as `result`.
Are these methods even marking their own result or only their
children? It seems like it's only really updating the children
recursively and its result would have already been updated before this
was called.
>
> The blocking of child results should be done in overloaded methods.
> """
<snip>
>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 04/12] dts: add mechanism to skip test cases or suites
2024-08-26 16:52 ` Jeremy Spewock
@ 2024-09-05 9:23 ` Juraj Linkeš
2024-09-05 15:26 ` Jeremy Spewock
0 siblings, 1 reply; 107+ messages in thread
From: Juraj Linkeš @ 2024-09-05 9:23 UTC (permalink / raw)
To: Jeremy Spewock
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On 26. 8. 2024 18:52, Jeremy Spewock wrote:
> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> <snip>
>> --- a/dts/framework/test_result.py
>> +++ b/dts/framework/test_result.py
>> @@ -75,6 +75,20 @@ def create_config(self) -> TestSuiteConfig:
>> test_cases=[test_case.__name__ for test_case in self.test_cases],
>> )
>>
>> + @property
>> + def skip(self) -> bool:
>> + """Skip the test suite if all test cases or the suite itself are to be skipped.
>> +
>> + Returns:
>> + :data:`True` if the test suite should be skipped, :data:`False` otherwise.
>> + """
>> + all_test_cases_skipped = True
>> + for test_case in self.test_cases:
>> + if not test_case.skip:
>> + all_test_cases_skipped = False
>> + break
>
> You could also potentially implement this using the built-in `all()`
> function. It would become a simple one-liner like
> `all_test_cases_skipped = all(test_case.skip for test_case in
> self.test_cases)`. That's probably short enough to even just put in
> the return statement though if you wanted to.
>
Good catch, I'll use any() here.
>> + return all_test_cases_skipped or self.test_suite_class.skip
>> +
>>
>> class Result(Enum):
>> """The possible states that a setup, a teardown or a test case may end up in."""
>> @@ -86,12 +100,12 @@ class Result(Enum):
>> #:
>> ERROR = auto()
>> #:
>> - SKIP = auto()
>> - #:
>> BLOCK = auto()
>> + #:
>> + SKIP = auto()
>>
>> def __bool__(self) -> bool:
>> - """Only PASS is True."""
>> + """Only :attr:`PASS` is True."""
>> return self is self.PASS
>>
>>
>> @@ -169,12 +183,13 @@ def update_setup(self, result: Result, error: Exception | None = None) -> None:
>> self.setup_result.result = result
>> self.setup_result.error = error
>>
>> - if result in [Result.BLOCK, Result.ERROR, Result.FAIL]:
>> - self.update_teardown(Result.BLOCK)
>> - self._block_result()
>> + if result != Result.PASS:
>> + result_to_mark = Result.BLOCK if result != Result.SKIP else Result.SKIP
>> + self.update_teardown(result_to_mark)
>> + self._mark_results(result_to_mark)
>>
>> - def _block_result(self) -> None:
>> - r"""Mark the result as :attr:`~Result.BLOCK`\ed.
>> + def _mark_results(self, result) -> None:
>
> Is it worth adding the type annotation for `result` here and to the
> other places where this is implemented? I guess it doesn't matter that
> much since it is a private method.
>
I didn't add it precisely because it's a private method and it's pretty
self explanatory.
>> + """Mark the result as well as the child result as `result`.
>
> Are these methods even marking their own result or only their
> children? It seems like it's only really updating the children
> recursively and its result would have already been updated before this
> was called.
>
It's supposed to be just their result which is actually the result of
the children in all but the TestCaseResult classes. Conceptually, each
results level should contains these:
1. the result of setup
2. the result of teardown
3. the result of the level itself (outside of setup and teardown)
The result of the level itself is what's supposed to be set here. The
thing is we're making the child results for non-test cases and the
result of the test cases for test cases. Maybe I only need to update the
docstring.
>>
>> The blocking of child results should be done in overloaded methods.
>> """
> <snip>
>>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 04/12] dts: add mechanism to skip test cases or suites
2024-09-05 9:23 ` Juraj Linkeš
@ 2024-09-05 15:26 ` Jeremy Spewock
0 siblings, 0 replies; 107+ messages in thread
From: Jeremy Spewock @ 2024-09-05 15:26 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Thu, Sep 5, 2024 at 5:23 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
<snip>
> >> + def _mark_results(self, result) -> None:
> >
> > Is it worth adding the type annotation for `result` here and to the
> > other places where this is implemented? I guess it doesn't matter that
> > much since it is a private method.
> >
>
> I didn't add it precisely because it's a private method and it's pretty
> self explanatory.
Makes sense.
>
> >> + """Mark the result as well as the child result as `result`.
> >
> > Are these methods even marking their own result or only their
> > children? It seems like it's only really updating the children
> > recursively and its result would have already been updated before this
> > was called.
> >
>
> It's supposed to be just their result which is actually the result of
> the children in all but the TestCaseResult classes. Conceptually, each
Right, of course. Sorry, I was thinking too literally about this
isolated method, with the context that that's how the result of the
level itself is decided, this makes way more sense.
> results level should contains these:
> 1. the result of setup
> 2. the result of teardown
> 3. the result of the level itself (outside of setup and teardown)
>
> The result of the level itself is what's supposed to be set here. The
> thing is we're making the child results for non-test cases and the
> result of the test cases for test cases. Maybe I only need to update the
> docstring.
You probably don't even need to update the doc-string, this seems more
like a mistake on my part :).
>
> >>
> >> The blocking of child results should be done in overloaded methods.
> >> """
> > <snip>
> >>
>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 04/12] dts: add mechanism to skip test cases or suites
2024-08-21 14:53 ` [PATCH v3 04/12] dts: add mechanism to skip test cases or suites Juraj Linkeš
2024-08-26 16:52 ` Jeremy Spewock
@ 2024-08-28 20:37 ` Dean Marx
1 sibling, 0 replies; 107+ messages in thread
From: Dean Marx @ 2024-08-28 20:37 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 714 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> If a test case is not relevant to the testing environment (such as when
> a NIC doesn't support a tested feature), the framework should skip it.
> The mechanism is a skeleton without actual logic that would set a test
> case or suite to be skipped.
>
> The mechanism uses a protocol to extend test suites and test cases with
> additional attributes that track whether the test case or suite should
> be skipped the reason for skipping it.
>
> Also update the results module with the new SKIP result.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 1029 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH v3 05/12] dts: add support for simpler topologies
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
` (3 preceding siblings ...)
2024-08-21 14:53 ` [PATCH v3 04/12] dts: add mechanism to skip test cases or suites Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 16:54 ` Jeremy Spewock
2024-08-28 20:56 ` Dean Marx
2024-08-21 14:53 ` [PATCH v3 06/12] dst: add basic capability support Juraj Linkeš
` (7 subsequent siblings)
12 siblings, 2 replies; 107+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
We currently assume there are two links between the SUT and TG nodes,
but that's too strict, even for some of the already existing test cases.
Add support for topologies with less than two links.
For topologies with no links, dummy ports are used. The expectation is
that test suites or cases that don't require any links won't be using
methods that use ports. Any test suites or cases requiring links will be
skipped in topologies with no links, but this feature is not implemented
in this commit.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/runner.py | 6 +-
dts/framework/test_suite.py | 32 +++----
dts/framework/testbed_model/node.py | 2 +-
dts/framework/testbed_model/port.py | 4 +-
dts/framework/testbed_model/topology.py | 101 ++++++++++++++++++++++
dts/tests/TestSuite_pmd_buffer_scatter.py | 2 +-
6 files changed, 120 insertions(+), 27 deletions(-)
create mode 100644 dts/framework/testbed_model/topology.py
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 55357ea1fe..48ae9cc215 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -53,6 +53,7 @@
TestSuiteWithCases,
)
from .test_suite import TestCase, TestSuite
+from .testbed_model.topology import Topology
class DTSRunner:
@@ -474,6 +475,7 @@ def _run_test_suites(
test_suites_with_cases: The test suites with test cases to run.
"""
end_build_target = False
+ topology = Topology(sut_node.ports, tg_node.ports)
for test_suite_with_cases in test_suites_with_cases:
test_suite_result = build_target_result.add_test_suite(test_suite_with_cases)
try:
@@ -481,6 +483,7 @@ def _run_test_suites(
self._run_test_suite(
sut_node,
tg_node,
+ topology,
test_suite_result,
test_suite_with_cases,
)
@@ -506,6 +509,7 @@ def _run_test_suite(
self,
sut_node: SutNode,
tg_node: TGNode,
+ topology: Topology,
test_suite_result: TestSuiteResult,
test_suite_with_cases: TestSuiteWithCases,
) -> None:
@@ -533,7 +537,7 @@ def _run_test_suite(
self._logger.set_stage(
DtsStage.test_suite_setup, Path(SETTINGS.output_dir, test_suite_name)
)
- test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node)
+ test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node, topology)
try:
self._logger.info(f"Starting test suite setup: {test_suite_name}")
test_suite.set_up_suite()
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index c59fc9c6e6..56f153bda6 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -24,9 +24,10 @@
from scapy.packet import Packet, Padding # type: ignore[import-untyped]
from framework.testbed_model.capability import TestProtocol
-from framework.testbed_model.port import Port, PortLink
+from framework.testbed_model.port import Port
from framework.testbed_model.sut_node import SutNode
from framework.testbed_model.tg_node import TGNode
+from framework.testbed_model.topology import Topology, TopologyType
from framework.testbed_model.traffic_generator.capturing_traffic_generator import (
PacketFilteringConfig,
)
@@ -72,7 +73,7 @@ class TestSuite(TestProtocol):
#: will block the execution of all subsequent test suites in the current build target.
is_blocking: ClassVar[bool] = False
_logger: DTSLogger
- _port_links: list[PortLink]
+ _topology_type: TopologyType
_sut_port_ingress: Port
_sut_port_egress: Port
_sut_ip_address_ingress: Union[IPv4Interface, IPv6Interface]
@@ -86,6 +87,7 @@ def __init__(
self,
sut_node: SutNode,
tg_node: TGNode,
+ topology: Topology,
):
"""Initialize the test suite testbed information and basic configuration.
@@ -95,35 +97,21 @@ def __init__(
Args:
sut_node: The SUT node where the test suite will run.
tg_node: The TG node where the test suite will run.
+ topology: The topology where the test suite will run.
"""
self.sut_node = sut_node
self.tg_node = tg_node
self._logger = get_dts_logger(self.__class__.__name__)
- self._port_links = []
- self._process_links()
- self._sut_port_ingress, self._tg_port_egress = (
- self._port_links[0].sut_port,
- self._port_links[0].tg_port,
- )
- self._sut_port_egress, self._tg_port_ingress = (
- self._port_links[1].sut_port,
- self._port_links[1].tg_port,
- )
+ self._topology_type = topology.type
+ self._tg_port_egress = topology.tg_port_egress
+ self._sut_port_ingress = topology.sut_port_ingress
+ self._sut_port_egress = topology.sut_port_egress
+ self._tg_port_ingress = topology.tg_port_ingress
self._sut_ip_address_ingress = ip_interface("192.168.100.2/24")
self._sut_ip_address_egress = ip_interface("192.168.101.2/24")
self._tg_ip_address_egress = ip_interface("192.168.100.3/24")
self._tg_ip_address_ingress = ip_interface("192.168.101.3/24")
- def _process_links(self) -> None:
- """Construct links between SUT and TG ports."""
- for sut_port in self.sut_node.ports:
- for tg_port in self.tg_node.ports:
- if (sut_port.identifier, sut_port.peer) == (
- tg_port.peer,
- tg_port.identifier,
- ):
- self._port_links.append(PortLink(sut_port=sut_port, tg_port=tg_port))
-
@classmethod
def get_test_cases(
cls, test_case_sublist: Sequence[str] | None = None
diff --git a/dts/framework/testbed_model/node.py b/dts/framework/testbed_model/node.py
index 12a40170ac..51443cd71f 100644
--- a/dts/framework/testbed_model/node.py
+++ b/dts/framework/testbed_model/node.py
@@ -90,7 +90,7 @@ def __init__(self, node_config: NodeConfiguration):
self._init_ports()
def _init_ports(self) -> None:
- self.ports = [Port(self.name, port_config) for port_config in self.config.ports]
+ self.ports = [Port(port_config) for port_config in self.config.ports]
self.main_session.update_ports(self.ports)
for port in self.ports:
self.configure_port_state(port)
diff --git a/dts/framework/testbed_model/port.py b/dts/framework/testbed_model/port.py
index 817405bea4..82c84cf4f8 100644
--- a/dts/framework/testbed_model/port.py
+++ b/dts/framework/testbed_model/port.py
@@ -54,7 +54,7 @@ class Port:
mac_address: str = ""
logical_name: str = ""
- def __init__(self, node_name: str, config: PortConfig):
+ def __init__(self, config: PortConfig):
"""Initialize the port from `node_name` and `config`.
Args:
@@ -62,7 +62,7 @@ def __init__(self, node_name: str, config: PortConfig):
config: The test run configuration of the port.
"""
self.identifier = PortIdentifier(
- node=node_name,
+ node=config.node,
pci=config.pci,
)
self.os_driver = config.os_driver
diff --git a/dts/framework/testbed_model/topology.py b/dts/framework/testbed_model/topology.py
new file mode 100644
index 0000000000..19632ee890
--- /dev/null
+++ b/dts/framework/testbed_model/topology.py
@@ -0,0 +1,101 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 PANTHEON.tech s.r.o.
+
+"""Testbed topology representation.
+
+A topology of a testbed captures what links are available between the testbed's nodes.
+The link information then implies what type of topology is available.
+"""
+
+from dataclasses import dataclass
+from enum import IntEnum
+from typing import Iterable
+
+from framework.config import PortConfig
+
+from .port import Port
+
+
+class TopologyType(IntEnum):
+ """Supported topology types."""
+
+ #: A topology with no Traffic Generator.
+ no_link = 0
+ #: A topology with one physical link between the SUT node and the TG node.
+ one_link = 1
+ #: A topology with two physical links between the Sut node and the TG node.
+ two_links = 2
+
+
+class Topology:
+ """Testbed topology.
+
+ The topology contains ports processed into ingress and egress ports.
+ It's assumed that port0 of the SUT node is connected to port0 of the TG node and so on.
+ If there are no ports on a node, dummy ports (ports with no actual values) are stored.
+ If there is only one link available, the ports of this link are stored
+ as both ingress and egress ports.
+
+ The dummy ports shouldn't be used. It's up to :class:`~framework.runner.DTSRunner`
+ to ensure no test case or suite requiring actual links is executed
+ when the topology prohibits it and up to the developers to make sure that test cases
+ not requiring any links don't use any ports. Otherwise, the underlying methods
+ using the ports will fail.
+
+ Attributes:
+ type: The type of the topology.
+ tg_port_egress: The egress port of the TG node.
+ sut_port_ingress: The ingress port of the SUT node.
+ sut_port_egress: The egress port of the SUT node.
+ tg_port_ingress: The ingress port of the TG node.
+ """
+
+ type: TopologyType
+ tg_port_egress: Port
+ sut_port_ingress: Port
+ sut_port_egress: Port
+ tg_port_ingress: Port
+
+ def __init__(self, sut_ports: Iterable[Port], tg_ports: Iterable[Port]):
+ """Create the topology from `sut_ports` and `tg_ports`.
+
+ Args:
+ sut_ports: The SUT node's ports.
+ tg_ports: The TG node's ports.
+ """
+ port_links = []
+ for sut_port in sut_ports:
+ for tg_port in tg_ports:
+ if (sut_port.identifier, sut_port.peer) == (
+ tg_port.peer,
+ tg_port.identifier,
+ ):
+ port_links.append(PortLink(sut_port=sut_port, tg_port=tg_port))
+
+ self.type = TopologyType(len(port_links))
+ dummy_port = Port(PortConfig("", "", "", "", "", ""))
+ self.tg_port_egress = dummy_port
+ self.sut_port_ingress = dummy_port
+ self.sut_port_egress = dummy_port
+ self.tg_port_ingress = dummy_port
+ if self.type > TopologyType.no_link:
+ self.tg_port_egress = port_links[0].tg_port
+ self.sut_port_ingress = port_links[0].sut_port
+ self.sut_port_egress = self.sut_port_ingress
+ self.tg_port_ingress = self.tg_port_egress
+ if self.type > TopologyType.one_link:
+ self.sut_port_egress = port_links[1].sut_port
+ self.tg_port_ingress = port_links[1].tg_port
+
+
+@dataclass(slots=True, frozen=True)
+class PortLink:
+ """The physical, cabled connection between the ports.
+
+ Attributes:
+ sut_port: The port on the SUT node connected to `tg_port`.
+ tg_port: The port on the TG node connected to `sut_port`.
+ """
+
+ sut_port: Port
+ tg_port: Port
diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
index 020fb0ab62..178a40385e 100644
--- a/dts/tests/TestSuite_pmd_buffer_scatter.py
+++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
@@ -58,7 +58,7 @@ def set_up_suite(self) -> None:
to support larger packet sizes.
"""
self.verify(
- len(self._port_links) > 1,
+ self._topology_type > 1,
"There must be at least two port links to run the scatter test suite",
)
--
2.34.1
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 05/12] dts: add support for simpler topologies
2024-08-21 14:53 ` [PATCH v3 05/12] dts: add support for simpler topologies Juraj Linkeš
@ 2024-08-26 16:54 ` Jeremy Spewock
2024-09-05 9:42 ` Juraj Linkeš
2024-08-28 20:56 ` Dean Marx
1 sibling, 1 reply; 107+ messages in thread
From: Jeremy Spewock @ 2024-08-26 16:54 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
I just had one question below, otherwise:
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
<snip>
> diff --git a/dts/framework/testbed_model/topology.py b/dts/framework/testbed_model/topology.py
> new file mode 100644
> index 0000000000..19632ee890
> --- /dev/null
> +++ b/dts/framework/testbed_model/topology.py
<snip>
> +
> +
> +class TopologyType(IntEnum):
> + """Supported topology types."""
> +
> + #: A topology with no Traffic Generator.
> + no_link = 0
> + #: A topology with one physical link between the SUT node and the TG node.
> + one_link = 1
> + #: A topology with two physical links between the Sut node and the TG node.
> + two_links = 2
> +
> +
> +class Topology:
> + """Testbed topology.
> +
> + The topology contains ports processed into ingress and egress ports.
> + It's assumed that port0 of the SUT node is connected to port0 of the TG node and so on.
Do we need to make this assumption when you are comparing the port
directly to its peer and matching the addresses? I think you could
specify in conf.yaml that port 0 on the SUT is one of your ports and
its peer is port 1 on the TG and because you do the matching, this
would work fine.
> + If there are no ports on a node, dummy ports (ports with no actual values) are stored.
> + If there is only one link available, the ports of this link are stored
> + as both ingress and egress ports.
> +
> + The dummy ports shouldn't be used. It's up to :class:`~framework.runner.DTSRunner`
> + to ensure no test case or suite requiring actual links is executed
> + when the topology prohibits it and up to the developers to make sure that test cases
> + not requiring any links don't use any ports. Otherwise, the underlying methods
> + using the ports will fail.
> +
> + Attributes:
> + type: The type of the topology.
> + tg_port_egress: The egress port of the TG node.
> + sut_port_ingress: The ingress port of the SUT node.
> + sut_port_egress: The egress port of the SUT node.
> + tg_port_ingress: The ingress port of the TG node.
> + """
> +
> + type: TopologyType
> + tg_port_egress: Port
> + sut_port_ingress: Port
> + sut_port_egress: Port
> + tg_port_ingress: Port
> +
> + def __init__(self, sut_ports: Iterable[Port], tg_ports: Iterable[Port]):
> + """Create the topology from `sut_ports` and `tg_ports`.
> +
> + Args:
> + sut_ports: The SUT node's ports.
> + tg_ports: The TG node's ports.
> + """
> + port_links = []
> + for sut_port in sut_ports:
> + for tg_port in tg_ports:
> + if (sut_port.identifier, sut_port.peer) == (
> + tg_port.peer,
> + tg_port.identifier,
> + ):
> + port_links.append(PortLink(sut_port=sut_port, tg_port=tg_port))
> +
> + self.type = TopologyType(len(port_links))
<snip>
>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 05/12] dts: add support for simpler topologies
2024-08-26 16:54 ` Jeremy Spewock
@ 2024-09-05 9:42 ` Juraj Linkeš
0 siblings, 0 replies; 107+ messages in thread
From: Juraj Linkeš @ 2024-09-05 9:42 UTC (permalink / raw)
To: Jeremy Spewock
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On 26. 8. 2024 18:54, Jeremy Spewock wrote:
> I just had one question below, otherwise:
>
> Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
>
> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> <snip>
>> diff --git a/dts/framework/testbed_model/topology.py b/dts/framework/testbed_model/topology.py
>> new file mode 100644
>> index 0000000000..19632ee890
>> --- /dev/null
>> +++ b/dts/framework/testbed_model/topology.py
> <snip>
>> +
>> +
>> +class TopologyType(IntEnum):
>> + """Supported topology types."""
>> +
>> + #: A topology with no Traffic Generator.
>> + no_link = 0
>> + #: A topology with one physical link between the SUT node and the TG node.
>> + one_link = 1
>> + #: A topology with two physical links between the Sut node and the TG node.
>> + two_links = 2
>> +
>> +
>> +class Topology:
>> + """Testbed topology.
>> +
>> + The topology contains ports processed into ingress and egress ports.
>> + It's assumed that port0 of the SUT node is connected to port0 of the TG node and so on.
>
> Do we need to make this assumption when you are comparing the port
> directly to its peer and matching the addresses? I think you could
> specify in conf.yaml that port 0 on the SUT is one of your ports and
> its peer is port 1 on the TG and because you do the matching, this
> would work fine.
>
Yes, the assumption is not adhered to yet. I guess I put this here
because we've been discussing this in the calls, but the actual code
doesn't use this. I'll remove this line.
>> + If there are no ports on a node, dummy ports (ports with no actual values) are stored.
>> + If there is only one link available, the ports of this link are stored
>> + as both ingress and egress ports.
>> +
>> + The dummy ports shouldn't be used. It's up to :class:`~framework.runner.DTSRunner`
>> + to ensure no test case or suite requiring actual links is executed
>> + when the topology prohibits it and up to the developers to make sure that test cases
>> + not requiring any links don't use any ports. Otherwise, the underlying methods
>> + using the ports will fail.
>> +
>> + Attributes:
>> + type: The type of the topology.
>> + tg_port_egress: The egress port of the TG node.
>> + sut_port_ingress: The ingress port of the SUT node.
>> + sut_port_egress: The egress port of the SUT node.
>> + tg_port_ingress: The ingress port of the TG node.
>> + """
>> +
>> + type: TopologyType
>> + tg_port_egress: Port
>> + sut_port_ingress: Port
>> + sut_port_egress: Port
>> + tg_port_ingress: Port
>> +
>> + def __init__(self, sut_ports: Iterable[Port], tg_ports: Iterable[Port]):
>> + """Create the topology from `sut_ports` and `tg_ports`.
>> +
>> + Args:
>> + sut_ports: The SUT node's ports.
>> + tg_ports: The TG node's ports.
>> + """
>> + port_links = []
>> + for sut_port in sut_ports:
>> + for tg_port in tg_ports:
>> + if (sut_port.identifier, sut_port.peer) == (
>> + tg_port.peer,
>> + tg_port.identifier,
>> + ):
>> + port_links.append(PortLink(sut_port=sut_port, tg_port=tg_port))
>> +
>> + self.type = TopologyType(len(port_links))
> <snip>
>>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 05/12] dts: add support for simpler topologies
2024-08-21 14:53 ` [PATCH v3 05/12] dts: add support for simpler topologies Juraj Linkeš
2024-08-26 16:54 ` Jeremy Spewock
@ 2024-08-28 20:56 ` Dean Marx
1 sibling, 0 replies; 107+ messages in thread
From: Dean Marx @ 2024-08-28 20:56 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 728 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> We currently assume there are two links between the SUT and TG nodes,
> but that's too strict, even for some of the already existing test cases.
> Add support for topologies with less than two links.
>
> For topologies with no links, dummy ports are used. The expectation is
> that test suites or cases that don't require any links won't be using
> methods that use ports. Any test suites or cases requiring links will be
> skipped in topologies with no links, but this feature is not implemented
> in this commit.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 1048 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH v3 06/12] dst: add basic capability support
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
` (4 preceding siblings ...)
2024-08-21 14:53 ` [PATCH v3 05/12] dts: add support for simpler topologies Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 16:56 ` Jeremy Spewock
2024-09-03 16:03 ` Dean Marx
2024-08-21 14:53 ` [PATCH v3 07/12] dts: add testpmd port information caching Juraj Linkeš
` (6 subsequent siblings)
12 siblings, 2 replies; 107+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
A test case or suite may require certain capabilities to be present in
the tested environment. Add the basic infrastructure for checking the
support status of capabilities:
* The Capability ABC defining the common capability API
* Extension of the TestProtocol with required capabilities (each test
suite or case stores the capabilities it requires)
* Integration with the runner which calls the new APIs to get which
capabilities are supported.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/runner.py | 26 +++++
dts/framework/test_result.py | 38 ++++++-
dts/framework/test_suite.py | 1 +
dts/framework/testbed_model/capability.py | 117 +++++++++++++++++++++-
4 files changed, 179 insertions(+), 3 deletions(-)
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 48ae9cc215..43bb2bc830 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -25,6 +25,7 @@
from types import MethodType
from typing import Iterable
+from framework.testbed_model.capability import Capability, get_supported_capabilities
from framework.testbed_model.sut_node import SutNode
from framework.testbed_model.tg_node import TGNode
@@ -452,6 +453,21 @@ def _run_build_target(
self._logger.exception("Build target teardown failed.")
build_target_result.update_teardown(Result.FAIL, e)
+ def _get_supported_capabilities(
+ self,
+ sut_node: SutNode,
+ topology_config: Topology,
+ test_suites_with_cases: Iterable[TestSuiteWithCases],
+ ) -> set[Capability]:
+
+ capabilities_to_check = set()
+ for test_suite_with_cases in test_suites_with_cases:
+ capabilities_to_check.update(test_suite_with_cases.required_capabilities)
+
+ self._logger.debug(f"Found capabilities to check: {capabilities_to_check}")
+
+ return get_supported_capabilities(sut_node, topology_config, capabilities_to_check)
+
def _run_test_suites(
self,
sut_node: SutNode,
@@ -464,6 +480,12 @@ def _run_test_suites(
The method assumes the build target we're testing has already been built on the SUT node.
The current build target thus corresponds to the current DPDK build present on the SUT node.
+ Before running any suites, the method determines whether they should be skipped
+ by inspecting any required capabilities the test suite needs and comparing those
+ to capabilities supported by the tested environment. If all capabilities are supported,
+ the suite is run. If all test cases in a test suite would be skipped, the whole test suite
+ is skipped (the setup and teardown is not run).
+
If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites
in the current build target won't be executed.
@@ -476,7 +498,11 @@ def _run_test_suites(
"""
end_build_target = False
topology = Topology(sut_node.ports, tg_node.ports)
+ supported_capabilities = self._get_supported_capabilities(
+ sut_node, topology, test_suites_with_cases
+ )
for test_suite_with_cases in test_suites_with_cases:
+ test_suite_with_cases.mark_skip_unsupported(supported_capabilities)
test_suite_result = build_target_result.add_test_suite(test_suite_with_cases)
try:
if not test_suite_with_cases.skip:
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 306b100bc6..b4b58ef348 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -25,10 +25,12 @@
import os.path
from collections.abc import MutableSequence
-from dataclasses import dataclass
+from dataclasses import dataclass, field
from enum import Enum, auto
from typing import Union
+from framework.testbed_model.capability import Capability
+
from .config import (
OS,
Architecture,
@@ -63,6 +65,12 @@ class is to hold a subset of test cases (which could be all test cases) because
test_suite_class: type[TestSuite]
test_cases: list[type[TestCase]]
+ required_capabilities: set[Capability] = field(default_factory=set, init=False)
+
+ def __post_init__(self):
+ """Gather the required capabilities of the test suite and all test cases."""
+ for test_object in [self.test_suite_class] + self.test_cases:
+ self.required_capabilities.update(test_object.required_capabilities)
def create_config(self) -> TestSuiteConfig:
"""Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
@@ -75,6 +83,34 @@ def create_config(self) -> TestSuiteConfig:
test_cases=[test_case.__name__ for test_case in self.test_cases],
)
+ def mark_skip_unsupported(self, supported_capabilities: set[Capability]) -> None:
+ """Mark the test suite and test cases to be skipped.
+
+ The mark is applied if object to be skipped requires any capabilities and at least one of
+ them is not among `supported_capabilities`.
+
+ Args:
+ supported_capabilities: The supported capabilities.
+ """
+ for test_object in [self.test_suite_class, *self.test_cases]:
+ capabilities_not_supported = test_object.required_capabilities - supported_capabilities
+ if capabilities_not_supported:
+ test_object.skip = True
+ capability_str = (
+ "capability" if len(capabilities_not_supported) == 1 else "capabilities"
+ )
+ test_object.skip_reason = (
+ f"Required {capability_str} '{capabilities_not_supported}' not found."
+ )
+ if not self.test_suite_class.skip:
+ if all(test_case.skip for test_case in self.test_cases):
+ self.test_suite_class.skip = True
+
+ self.test_suite_class.skip_reason = (
+ "All test cases are marked to be skipped with reasons: "
+ f"{' '.join(test_case.skip_reason for test_case in self.test_cases)}"
+ )
+
@property
def skip(self) -> bool:
"""Skip the test suite if all test cases or the suite itself are to be skipped.
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 56f153bda6..5c393ce8bf 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -467,6 +467,7 @@ def _decorator(func: TestSuiteMethodType) -> type[TestCase]:
test_case = cast(type[TestCase], func)
test_case.skip = cls.skip
test_case.skip_reason = cls.skip_reason
+ test_case.required_capabilities = set()
test_case.test_type = test_case_type
return test_case
diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py
index 662f691a0e..8899f07f76 100644
--- a/dts/framework/testbed_model/capability.py
+++ b/dts/framework/testbed_model/capability.py
@@ -3,11 +3,97 @@
"""Testbed capabilities.
-This module provides a protocol that defines the common attributes of test cases and suites.
+This module provides a protocol that defines the common attributes of test cases and suites
+and support for test environment capabilities.
"""
+from abc import ABC, abstractmethod
from collections.abc import Sequence
-from typing import ClassVar, Protocol
+from typing import Callable, ClassVar, Protocol
+
+from typing_extensions import Self
+
+from .sut_node import SutNode
+from .topology import Topology
+
+
+class Capability(ABC):
+ """The base class for various capabilities.
+
+ The same capability should always be represented by the same object,
+ meaning the same capability required by different test cases or suites
+ should point to the same object.
+
+ Example:
+ ``test_case1`` and ``test_case2`` each require ``capability1``
+ and in both instances, ``capability1`` should point to the same capability object.
+
+ It is up to the subclasses how they implement this.
+
+ The instances are used in sets so they must be hashable.
+ """
+
+ #: A set storing the capabilities whose support should be checked.
+ capabilities_to_check: ClassVar[set[Self]] = set()
+
+ def register_to_check(self) -> Callable[[SutNode, "Topology"], set[Self]]:
+ """Register the capability to be checked for support.
+
+ Returns:
+ The callback function that checks the support of capabilities of the particular subclass
+ which should be called after all capabilities have been registered.
+ """
+ if not type(self).capabilities_to_check:
+ type(self).capabilities_to_check = set()
+ type(self).capabilities_to_check.add(self)
+ return type(self)._get_and_reset
+
+ def add_to_required(self, test_case_or_suite: type["TestProtocol"]) -> None:
+ """Add the capability instance to the required test case or suite's capabilities.
+
+ Args:
+ test_case_or_suite: The test case or suite among whose required capabilities
+ to add this instance.
+ """
+ if not test_case_or_suite.required_capabilities:
+ test_case_or_suite.required_capabilities = set()
+ self._preprocess_required(test_case_or_suite)
+ test_case_or_suite.required_capabilities.add(self)
+
+ def _preprocess_required(self, test_case_or_suite: type["TestProtocol"]) -> None:
+ """An optional method that modifies the required capabilities."""
+
+ @classmethod
+ def _get_and_reset(cls, sut_node: SutNode, topology: "Topology") -> set[Self]:
+ """The callback method to be called after all capabilities have been registered.
+
+ Not only does this method check the support of capabilities,
+ but it also reset the internal set of registered capabilities
+ so that the "register, then get support" workflow works in subsequent test runs.
+ """
+ supported_capabilities = cls.get_supported_capabilities(sut_node, topology)
+ cls.capabilities_to_check = set()
+ return supported_capabilities
+
+ @classmethod
+ @abstractmethod
+ def get_supported_capabilities(cls, sut_node: SutNode, topology: "Topology") -> set[Self]:
+ """Get the support status of each registered capability.
+
+ Each subclass must implement this method and return the subset of supported capabilities
+ of :attr:`capabilities_to_check`.
+
+ Args:
+ sut_node: The SUT node of the current test run.
+ topology: The topology of the current test run.
+
+ Returns:
+ The supported capabilities.
+ """
+
+ @abstractmethod
+ def __hash__(self) -> int:
+ """The subclasses must be hashable so that they can be stored in sets."""
class TestProtocol(Protocol):
@@ -17,6 +103,8 @@ class TestProtocol(Protocol):
skip: ClassVar[bool] = False
#: The reason for skipping the test case or suite.
skip_reason: ClassVar[str] = ""
+ #: The capabilities the test case or suite requires in order to be executed.
+ required_capabilities: ClassVar[set[Capability]] = set()
@classmethod
def get_test_cases(cls, test_case_sublist: Sequence[str] | None = None) -> tuple[set, set]:
@@ -26,3 +114,28 @@ def get_test_cases(cls, test_case_sublist: Sequence[str] | None = None) -> tuple
NotImplementedError: The subclass does not implement the method.
"""
raise NotImplementedError()
+
+
+def get_supported_capabilities(
+ sut_node: SutNode,
+ topology_config: Topology,
+ capabilities_to_check: set[Capability],
+) -> set[Capability]:
+ """Probe the environment for `capabilities_to_check` and return the supported ones.
+
+ Args:
+ sut_node: The SUT node to check for capabilities.
+ topology_config: The topology config to check for capabilities.
+ capabilities_to_check: The capabilities to check.
+
+ Returns:
+ The capabilities supported by the environment.
+ """
+ callbacks = set()
+ for capability_to_check in capabilities_to_check:
+ callbacks.add(capability_to_check.register_to_check())
+ supported_capabilities = set()
+ for callback in callbacks:
+ supported_capabilities.update(callback(sut_node, topology_config))
+
+ return supported_capabilities
--
2.34.1
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 06/12] dst: add basic capability support
2024-08-21 14:53 ` [PATCH v3 06/12] dst: add basic capability support Juraj Linkeš
@ 2024-08-26 16:56 ` Jeremy Spewock
2024-09-05 9:50 ` Juraj Linkeš
2024-09-03 16:03 ` Dean Marx
1 sibling, 1 reply; 107+ messages in thread
From: Jeremy Spewock @ 2024-08-26 16:56 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
Just one comment about adding something to a doc-string, otherwise
looks good to me:
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
<snip>
> diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
> index 306b100bc6..b4b58ef348 100644
> --- a/dts/framework/test_result.py
> +++ b/dts/framework/test_result.py
> @@ -25,10 +25,12 @@
>
> import os.path
> from collections.abc import MutableSequence
> -from dataclasses import dataclass
> +from dataclasses import dataclass, field
> from enum import Enum, auto
> from typing import Union
>
> +from framework.testbed_model.capability import Capability
> +
> from .config import (
> OS,
> Architecture,
> @@ -63,6 +65,12 @@ class is to hold a subset of test cases (which could be all test cases) because
>
> test_suite_class: type[TestSuite]
> test_cases: list[type[TestCase]]
> + required_capabilities: set[Capability] = field(default_factory=set, init=False)
This should probably be added to the Attributes section of the
doc-string for the class. When it's there, it might also be useful to
explain that this is used by the runner to determine what capabilities
need to be searched for to mark the suite for being skipped. The only
reason I think that would be useful is it helps differentiate this
list of capabilities from the list of required capabilities that every
test suite and test case has.
> +
> + def __post_init__(self):
> + """Gather the required capabilities of the test suite and all test cases."""
> + for test_object in [self.test_suite_class] + self.test_cases:
> + self.required_capabilities.update(test_object.required_capabilities)
<snip>
>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 06/12] dst: add basic capability support
2024-08-26 16:56 ` Jeremy Spewock
@ 2024-09-05 9:50 ` Juraj Linkeš
2024-09-05 15:27 ` Jeremy Spewock
0 siblings, 1 reply; 107+ messages in thread
From: Juraj Linkeš @ 2024-09-05 9:50 UTC (permalink / raw)
To: Jeremy Spewock
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On 26. 8. 2024 18:56, Jeremy Spewock wrote:
> Just one comment about adding something to a doc-string, otherwise
> looks good to me:
>
> Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
>
> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> <snip>
>> diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
>> index 306b100bc6..b4b58ef348 100644
>> --- a/dts/framework/test_result.py
>> +++ b/dts/framework/test_result.py
>> @@ -25,10 +25,12 @@
>>
>> import os.path
>> from collections.abc import MutableSequence
>> -from dataclasses import dataclass
>> +from dataclasses import dataclass, field
>> from enum import Enum, auto
>> from typing import Union
>>
>> +from framework.testbed_model.capability import Capability
>> +
>> from .config import (
>> OS,
>> Architecture,
>> @@ -63,6 +65,12 @@ class is to hold a subset of test cases (which could be all test cases) because
>>
>> test_suite_class: type[TestSuite]
>> test_cases: list[type[TestCase]]
>> + required_capabilities: set[Capability] = field(default_factory=set, init=False)
>
> This should probably be added to the Attributes section of the
> doc-string for the class.
Ah, I missed this, thanks.
> When it's there, it might also be useful to
> explain that this is used by the runner to determine what capabilities
> need to be searched for to mark the suite for being skipped.
And also test cases.
> The only
> reason I think that would be useful is it helps differentiate this
> list of capabilities from the list of required capabilities that every
> test suite and test case has.
>
I want to add this:
The combined required capabilities of both the test suite and the subset
of test cases.
I think this makes it clear that it's different from the individual
required capabilities of test suites and cases. Let me know what you think.
>> +
>> + def __post_init__(self):
>> + """Gather the required capabilities of the test suite and all test cases."""
>> + for test_object in [self.test_suite_class] + self.test_cases:
>> + self.required_capabilities.update(test_object.required_capabilities)
> <snip>
>>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 06/12] dst: add basic capability support
2024-09-05 9:50 ` Juraj Linkeš
@ 2024-09-05 15:27 ` Jeremy Spewock
0 siblings, 0 replies; 107+ messages in thread
From: Jeremy Spewock @ 2024-09-05 15:27 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Thu, Sep 5, 2024 at 5:50 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
<snip>
> >> @@ -63,6 +65,12 @@ class is to hold a subset of test cases (which could be all test cases) because
> >>
> >> test_suite_class: type[TestSuite]
> >> test_cases: list[type[TestCase]]
> >> + required_capabilities: set[Capability] = field(default_factory=set, init=False)
> >
> > This should probably be added to the Attributes section of the
> > doc-string for the class.
>
> Ah, I missed this, thanks.
>
> > When it's there, it might also be useful to
> > explain that this is used by the runner to determine what capabilities
> > need to be searched for to mark the suite for being skipped.
>
> And also test cases.
>
> > The only
> > reason I think that would be useful is it helps differentiate this
> > list of capabilities from the list of required capabilities that every
> > test suite and test case has.
> >
>
> I want to add this:
> The combined required capabilities of both the test suite and the subset
> of test cases.
>
> I think this makes it clear that it's different from the individual
> required capabilities of test suites and cases. Let me know what you think.
>
Yeah, I also think that makes the difference clear, sounds great to me, thanks!
> >> +
> >> + def __post_init__(self):
> >> + """Gather the required capabilities of the test suite and all test cases."""
> >> + for test_object in [self.test_suite_class] + self.test_cases:
> >> + self.required_capabilities.update(test_object.required_capabilities)
> > <snip>
> >>
>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 06/12] dst: add basic capability support
2024-08-21 14:53 ` [PATCH v3 06/12] dst: add basic capability support Juraj Linkeš
2024-08-26 16:56 ` Jeremy Spewock
@ 2024-09-03 16:03 ` Dean Marx
2024-09-05 9:51 ` Juraj Linkeš
1 sibling, 1 reply; 107+ messages in thread
From: Dean Marx @ 2024-09-03 16:03 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 897 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> A test case or suite may require certain capabilities to be present in
> the tested environment. Add the basic infrastructure for checking the
> support status of capabilities:
> * The Capability ABC defining the common capability API
> * Extension of the TestProtocol with required capabilities (each test
> suite or case stores the capabilities it requires)
> * Integration with the runner which calls the new APIs to get which
> capabilities are supported.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Looks all good to me, it was interesting to see how you've used abstract
methods in the Capability class. The only thing I noticed was it seems like
you wrote "dst" instead of "dts" in the commit message, otherwise:
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 1246 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 06/12] dst: add basic capability support
2024-09-03 16:03 ` Dean Marx
@ 2024-09-05 9:51 ` Juraj Linkeš
0 siblings, 0 replies; 107+ messages in thread
From: Juraj Linkeš @ 2024-09-05 9:51 UTC (permalink / raw)
To: Dean Marx
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
On 3. 9. 2024 18:03, Dean Marx wrote:
> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
>
> A test case or suite may require certain capabilities to be present in
> the tested environment. Add the basic infrastructure for checking the
> support status of capabilities:
> * The Capability ABC defining the common capability API
> * Extension of the TestProtocol with required capabilities (each test
> suite or case stores the capabilities it requires)
> * Integration with the runner which calls the new APIs to get which
> capabilities are supported.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
>
> Looks all good to me, it was interesting to see how you've used abstract
> methods in the Capability class. The only thing I noticed was it seems
> like you wrote "dst" instead of "dts" in the commit message, otherwise:
>
Oh, right, thanks for the catch.
> Reviewed-by: Dean Marx <dmarx@iol.unh.edu <mailto:dmarx@iol.unh.edu>>
^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH v3 07/12] dts: add testpmd port information caching
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
` (5 preceding siblings ...)
2024-08-21 14:53 ` [PATCH v3 06/12] dst: add basic capability support Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 16:56 ` Jeremy Spewock
2024-09-03 16:12 ` Dean Marx
2024-08-21 14:53 ` [PATCH v3 08/12] dts: add NIC capability support Juraj Linkeš
` (5 subsequent siblings)
12 siblings, 2 replies; 107+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
When using port information multiple times in a testpmd shell instance
lifespan, it's desirable to not get the information each time, so
caching is added. In case the information changes, there's a way to
force the update.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/remote_session/testpmd_shell.py | 30 +++++++++++++++++--
1 file changed, 28 insertions(+), 2 deletions(-)
diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index b4ad253020..f0bcc918e5 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -654,6 +654,7 @@ class TestPmdShell(DPDKShell):
"""
_app_params: TestPmdParams
+ _ports: list[TestPmdPort] | None
#: The path to the testpmd executable.
path: ClassVar[PurePath] = PurePath("app", "dpdk-testpmd")
@@ -686,6 +687,21 @@ def __init__(
TestPmdParams(**app_params),
name,
)
+ self._ports = None
+
+ @property
+ def ports(self) -> list[TestPmdPort]:
+ """The ports of the instance.
+
+ This caches the ports returned by :meth:`show_port_info_all`.
+ To force an update of port information, execute :meth:`show_port_info_all` or
+ :meth:`show_port_info`.
+
+ Returns: The list of known testpmd ports.
+ """
+ if self._ports is None:
+ return self.show_port_info_all()
+ return self._ports
def start(self, verify: bool = True) -> None:
"""Start packet forwarding with the current configuration.
@@ -872,7 +888,8 @@ def show_port_info_all(self) -> list[TestPmdPort]:
# executed on a pseudo-terminal created by paramiko on the remote node, lines end with CRLF.
# Therefore we also need to take the carriage return into account.
iter = re.finditer(r"\*{21}.*?[\r\n]{4}", output + "\r\n", re.S)
- return [TestPmdPort.parse(block.group(0)) for block in iter]
+ self._ports = [TestPmdPort.parse(block.group(0)) for block in iter]
+ return self._ports
def show_port_info(self, port_id: int) -> TestPmdPort:
"""Returns the given port information.
@@ -890,7 +907,16 @@ def show_port_info(self, port_id: int) -> TestPmdPort:
if output.startswith("Invalid port"):
raise InteractiveCommandExecutionError("invalid port given")
- return TestPmdPort.parse(output)
+ port = TestPmdPort.parse(output)
+ self._update_port(port)
+ return port
+
+ def _update_port(self, port: TestPmdPort) -> None:
+ if self._ports:
+ self._ports = [
+ existing_port if port.id != existing_port.id else port
+ for existing_port in self._ports
+ ]
def show_port_stats_all(self) -> list[TestPmdPortStats]:
"""Returns the statistics of all the ports.
--
2.34.1
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 07/12] dts: add testpmd port information caching
2024-08-21 14:53 ` [PATCH v3 07/12] dts: add testpmd port information caching Juraj Linkeš
@ 2024-08-26 16:56 ` Jeremy Spewock
2024-09-03 16:12 ` Dean Marx
1 sibling, 0 replies; 107+ messages in thread
From: Jeremy Spewock @ 2024-08-26 16:56 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
> When using port information multiple times in a testpmd shell instance
> lifespan, it's desirable to not get the information each time, so
> caching is added. In case the information changes, there's a way to
> force the update.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 07/12] dts: add testpmd port information caching
2024-08-21 14:53 ` [PATCH v3 07/12] dts: add testpmd port information caching Juraj Linkeš
2024-08-26 16:56 ` Jeremy Spewock
@ 2024-09-03 16:12 ` Dean Marx
1 sibling, 0 replies; 107+ messages in thread
From: Dean Marx @ 2024-09-03 16:12 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 437 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> When using port information multiple times in a testpmd shell instance
> lifespan, it's desirable to not get the information each time, so
> caching is added. In case the information changes, there's a way to
> force the update.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 742 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH v3 08/12] dts: add NIC capability support
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
` (6 preceding siblings ...)
2024-08-21 14:53 ` [PATCH v3 07/12] dts: add testpmd port information caching Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 17:11 ` Jeremy Spewock
` (2 more replies)
2024-08-21 14:53 ` [PATCH v3 09/12] dts: add topology capability Juraj Linkeš
` (4 subsequent siblings)
12 siblings, 3 replies; 107+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
Some test cases or suites may be testing a NIC feature that is not
supported on all NICs, so add support for marking test cases or suites
as requiring NIC capabilities.
The marking is done with a decorator, which populates the internal
required_capabilities attribute of TestProtocol. The NIC capability
itself is a wrapper around the NicCapability defined in testpmd_shell.
The reason is twofold:
1. Enums cannot be extended and the class implements the methods of its
abstract base superclass,
2. The class also stores an optional decorator function which is used
before/after capability retrieval. This is needed because some
capabilities may be advertised differently under different
configuration.
The decorator API is designed to be simple to use. The arguments passed
to it are all from the testpmd shell. Everything else (even the actual
capability object creation) is done internally.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
Depends-on: patch-142276 ("dts: add methods for modifying MTU to testpmd
shell")
---
dts/framework/remote_session/testpmd_shell.py | 178 ++++++++++++++++-
dts/framework/testbed_model/capability.py | 180 +++++++++++++++++-
dts/tests/TestSuite_pmd_buffer_scatter.py | 2 +
3 files changed, 356 insertions(+), 4 deletions(-)
diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index f0bcc918e5..48c31124d1 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -16,11 +16,17 @@
import re
import time
-from collections.abc import Callable
+from collections.abc import Callable, MutableSet
from dataclasses import dataclass, field
from enum import Flag, auto
+from functools import partial
from pathlib import PurePath
-from typing import ClassVar
+from typing import TYPE_CHECKING, Any, ClassVar, TypeAlias
+
+if TYPE_CHECKING:
+ from enum import Enum as NoAliasEnum
+else:
+ from aenum import NoAliasEnum
from typing_extensions import Self, TypeVarTuple, Unpack
@@ -34,6 +40,16 @@
from framework.testbed_model.sut_node import SutNode
from framework.utils import StrEnum
+TestPmdShellCapabilityMethod: TypeAlias = Callable[
+ ["TestPmdShell", MutableSet["NicCapability"], MutableSet["NicCapability"]], None
+]
+
+TestPmdShellSimpleMethod: TypeAlias = Callable[["TestPmdShell"], Any]
+
+TestPmdShellDecoratedMethod: TypeAlias = Callable[["TestPmdShell"], None]
+
+TestPmdShellDecorator: TypeAlias = Callable[[TestPmdShellSimpleMethod], TestPmdShellDecoratedMethod]
+
class TestPmdDevice:
"""The data of a device that testpmd can recognize.
@@ -377,6 +393,71 @@ def _validate(info: str):
return TextParser.wrap(TextParser.find(r"Device private info:\s+([\s\S]+)"), _validate)
+class RxQueueState(StrEnum):
+ """RX queue states."""
+
+ #:
+ stopped = auto()
+ #:
+ started = auto()
+ #:
+ hairpin = auto()
+ #:
+ unknown = auto()
+
+ @classmethod
+ def make_parser(cls) -> ParserFn:
+ """Makes a parser function.
+
+ Returns:
+ ParserFn: A dictionary for the `dataclasses.field` metadata argument containing a
+ parser function that makes an instance of this enum from text.
+ """
+ return TextParser.wrap(TextParser.find(r"Rx queue state: ([^\r\n]+)"), cls)
+
+
+@dataclass
+class TestPmdRxqInfo(TextParser):
+ """Representation of testpmd's ``show rxq info <port_id> <queue_id>`` command."""
+
+ #:
+ port_id: int = field(metadata=TextParser.find_int(r"Infos for port (\d+)\b ?, RX queue \d+\b"))
+ #:
+ queue_id: int = field(metadata=TextParser.find_int(r"Infos for port \d+\b ?, RX queue (\d+)\b"))
+ #: Mempool used by that queue
+ mempool: str = field(metadata=TextParser.find(r"Mempool: ([^\r\n]+)"))
+ #: Ring prefetch threshold
+ rx_prefetch_threshold: int = field(
+ metadata=TextParser.find_int(r"RX prefetch threshold: (\d+)\b")
+ )
+ #: Ring host threshold
+ rx_host_threshold: int = field(metadata=TextParser.find_int(r"RX host threshold: (\d+)\b"))
+ #: Ring writeback threshold
+ rx_writeback_threshold: int = field(
+ metadata=TextParser.find_int(r"RX writeback threshold: (\d+)\b")
+ )
+ #: Drives the freeing of Rx descriptors
+ rx_free_threshold: int = field(metadata=TextParser.find_int(r"RX free threshold: (\d+)\b"))
+ #: Drop packets if no descriptors are available
+ rx_drop_packets: bool = field(metadata=TextParser.find(r"RX drop packets: on"))
+ #: Do not start queue with rte_eth_dev_start()
+ rx_deferred_start: bool = field(metadata=TextParser.find(r"RX deferred start: on"))
+ #: Scattered packets Rx enabled
+ rx_scattered_packets: bool = field(metadata=TextParser.find(r"RX scattered packets: on"))
+ #: The state of the queue
+ rx_queue_state: str = field(metadata=RxQueueState.make_parser())
+ #: Configured number of RXDs
+ number_of_rxds: int = field(metadata=TextParser.find_int(r"Number of RXDs: (\d+)\b"))
+ #: Hardware receive buffer size
+ rx_buffer_size: int | None = field(
+ default=None, metadata=TextParser.find_int(r"RX buffer size: (\d+)\b")
+ )
+ #: Burst mode information
+ burst_mode: str | None = field(
+ default=None, metadata=TextParser.find(r"Burst mode: ([^\r\n]+)")
+ )
+
+
@dataclass
class TestPmdPort(TextParser):
"""Dataclass representing the result of testpmd's ``show port info`` command."""
@@ -962,3 +1043,96 @@ def _close(self) -> None:
self.stop()
self.send_command("quit", "Bye...")
return super()._close()
+
+ """
+ ====== Capability retrieval methods ======
+ """
+
+ def get_capabilities_rxq_info(
+ self,
+ supported_capabilities: MutableSet["NicCapability"],
+ unsupported_capabilities: MutableSet["NicCapability"],
+ ) -> None:
+ """Get all rxq capabilities and divide them into supported and unsupported.
+
+ Args:
+ supported_capabilities: Supported capabilities will be added to this set.
+ unsupported_capabilities: Unsupported capabilities will be added to this set.
+ """
+ self._logger.debug("Getting rxq capabilities.")
+ command = f"show rxq info {self.ports[0].id} 0"
+ rxq_info = TestPmdRxqInfo.parse(self.send_command(command))
+ if rxq_info.rx_scattered_packets:
+ supported_capabilities.add(NicCapability.SCATTERED_RX_ENABLED)
+ else:
+ unsupported_capabilities.add(NicCapability.SCATTERED_RX_ENABLED)
+
+ """
+ ====== Decorator methods ======
+ """
+
+ @staticmethod
+ def config_mtu_9000(testpmd_method: TestPmdShellSimpleMethod) -> TestPmdShellDecoratedMethod:
+ """Configure MTU to 9000 on all ports, run `testpmd_method`, then revert.
+
+ Args:
+ testpmd_method: The method to decorate.
+
+ Returns:
+ The method decorated with setting and reverting MTU.
+ """
+
+ def wrapper(testpmd_shell: Self):
+ original_mtus = []
+ for port in testpmd_shell.ports:
+ original_mtus.append((port.id, port.mtu))
+ testpmd_shell.set_port_mtu(port_id=port.id, mtu=9000, verify=False)
+ testpmd_method(testpmd_shell)
+ for port_id, mtu in original_mtus:
+ testpmd_shell.set_port_mtu(port_id=port_id, mtu=mtu if mtu else 1500, verify=False)
+
+ return wrapper
+
+
+class NicCapability(NoAliasEnum):
+ """A mapping between capability names and the associated :class:`TestPmdShell` methods.
+
+ The :class:`TestPmdShell` method executes the command that checks
+ whether the capability is supported.
+
+ The signature of each :class:`TestPmdShell` capability checking method must be::
+
+ fn(self, supported_capabilities: MutableSet, unsupported_capabilities: MutableSet) -> None
+
+ The method must get whether a capability is supported or not from a testpmd command.
+ If multiple capabilities can be obtained from a testpmd command, each should be obtained
+ in the method. These capabilities should then be added to `supported_capabilities` or
+ `unsupported_capabilities` based on their support.
+
+ The two dictionaries are shared across all capability discovery function calls in a given
+ test run so that we don't call the same function multiple times,
+ e.g. when we find a :attr:`scattered_rx` in :meth:`TestPmdShell.get_capabilities_rxq`, we don't
+ go looking for it again if a different test case also needs it.
+ """
+
+ #: Scattered packets Rx enabled.
+ SCATTERED_RX_ENABLED: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rxq_info
+ )
+
+ def __call__(
+ self,
+ testpmd_shell: TestPmdShell,
+ supported_capabilities: MutableSet[Self],
+ unsupported_capabilities: MutableSet[Self],
+ ) -> None:
+ """Execute the associated capability retrieval function.
+
+ Args:
+ testpmd_shell: :class:`TestPmdShell` object to which the function will be bound to.
+ supported_capabilities: The dictionary storing the supported capabilities
+ of a given test run.
+ unsupported_capabilities: The dictionary storing the unsupported capabilities
+ of a given test run.
+ """
+ self.value(testpmd_shell, supported_capabilities, unsupported_capabilities)
diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py
index 8899f07f76..9a79e6ebb3 100644
--- a/dts/framework/testbed_model/capability.py
+++ b/dts/framework/testbed_model/capability.py
@@ -5,14 +5,40 @@
This module provides a protocol that defines the common attributes of test cases and suites
and support for test environment capabilities.
+
+Many test cases are testing features not available on all hardware.
+
+The module also allows developers to mark test cases or suites a requiring certain
+hardware capabilities with the :func:`requires` decorator.
+
+Example:
+ .. code:: python
+
+ from framework.test_suite import TestSuite, func_test
+ from framework.testbed_model.capability import NicCapability, requires
+ class TestPmdBufferScatter(TestSuite):
+ # only the test case requires the scattered_rx capability
+ # other test cases may not require it
+ @requires(NicCapability.scattered_rx)
+ @func_test
+ def test_scatter_mbuf_2048(self):
"""
from abc import ABC, abstractmethod
-from collections.abc import Sequence
-from typing import Callable, ClassVar, Protocol
+from collections.abc import MutableSet, Sequence
+from dataclasses import dataclass
+from typing import Callable, ClassVar, Protocol, Tuple
from typing_extensions import Self
+from framework.logger import get_dts_logger
+from framework.remote_session.testpmd_shell import (
+ NicCapability,
+ TestPmdShell,
+ TestPmdShellDecorator,
+ TestPmdShellSimpleMethod,
+)
+
from .sut_node import SutNode
from .topology import Topology
@@ -96,6 +122,128 @@ def __hash__(self) -> int:
"""The subclasses must be hashable so that they can be stored in sets."""
+@dataclass
+class DecoratedNicCapability(Capability):
+ """A wrapper around :class:`~framework.remote_session.testpmd_shell.NicCapability`.
+
+ Some NIC capabilities are only present or listed as supported only under certain conditions,
+ such as when a particular configuration is in place. This is achieved by allowing users to pass
+ a decorator function that decorates the function that gets the support status of the capability.
+
+ New instances should be created with the :meth:`create_unique` class method to ensure
+ there are no duplicate instances.
+
+ Attributes:
+ nic_capability: The NIC capability that partly defines each instance.
+ capability_decorator: The decorator function that will be passed the function associated
+ with `nic_capability` when discovering the support status of the capability.
+ Each instance is defined by `capability_decorator` along with `nic_capability`.
+ """
+
+ nic_capability: NicCapability
+ capability_decorator: TestPmdShellDecorator | None
+ _unique_capabilities: ClassVar[
+ dict[Tuple[NicCapability, TestPmdShellDecorator | None], Self]
+ ] = {}
+
+ @classmethod
+ def get_unique(
+ cls, nic_capability: NicCapability, decorator_fn: TestPmdShellDecorator | None
+ ) -> "DecoratedNicCapability":
+ """Get the capability uniquely identified by `nic_capability` and `decorator_fn`.
+
+ Args:
+ nic_capability: The NIC capability.
+ decorator_fn: The function that will be passed the function associated
+ with `nic_capability` when discovering the support status of the capability.
+
+ Returns:
+ The capability uniquely identified by `nic_capability` and `decorator_fn`.
+ """
+ if (nic_capability, decorator_fn) not in cls._unique_capabilities:
+ cls._unique_capabilities[(nic_capability, decorator_fn)] = cls(
+ nic_capability, decorator_fn
+ )
+ return cls._unique_capabilities[(nic_capability, decorator_fn)]
+
+ @classmethod
+ def get_supported_capabilities(
+ cls, sut_node: SutNode, topology: "Topology"
+ ) -> set["DecoratedNicCapability"]:
+ """Overrides :meth:`~Capability.get_supported_capabilities`.
+
+ The capabilities are first sorted by decorators, then reduced into a single function which
+ is then passed to the decorator. This way we only execute each decorator only once.
+ """
+ supported_conditional_capabilities: set["DecoratedNicCapability"] = set()
+ logger = get_dts_logger(f"{sut_node.name}.{cls.__name__}")
+ if topology.type is Topology.type.no_link:
+ logger.debug(
+ "No links available in the current topology, not getting NIC capabilities."
+ )
+ return supported_conditional_capabilities
+ logger.debug(
+ f"Checking which NIC capabilities from {cls.capabilities_to_check} are supported."
+ )
+ if cls.capabilities_to_check:
+ capabilities_to_check_map = cls._get_decorated_capabilities_map()
+ with TestPmdShell(sut_node, privileged=True) as testpmd_shell:
+ for conditional_capability_fn, capabilities in capabilities_to_check_map.items():
+ supported_capabilities: set[NicCapability] = set()
+ unsupported_capabilities: set[NicCapability] = set()
+ capability_fn = cls._reduce_capabilities(
+ capabilities, supported_capabilities, unsupported_capabilities
+ )
+ if conditional_capability_fn:
+ capability_fn = conditional_capability_fn(capability_fn)
+ capability_fn(testpmd_shell)
+ for supported_capability in supported_capabilities:
+ for capability in capabilities:
+ if supported_capability == capability.nic_capability:
+ supported_conditional_capabilities.add(capability)
+
+ logger.debug(f"Found supported capabilities {supported_conditional_capabilities}.")
+ return supported_conditional_capabilities
+
+ @classmethod
+ def _get_decorated_capabilities_map(
+ cls,
+ ) -> dict[TestPmdShellDecorator | None, set["DecoratedNicCapability"]]:
+ capabilities_map: dict[TestPmdShellDecorator | None, set["DecoratedNicCapability"]] = {}
+ for capability in cls.capabilities_to_check:
+ if capability.capability_decorator not in capabilities_map:
+ capabilities_map[capability.capability_decorator] = set()
+ capabilities_map[capability.capability_decorator].add(capability)
+
+ return capabilities_map
+
+ @classmethod
+ def _reduce_capabilities(
+ cls,
+ capabilities: set["DecoratedNicCapability"],
+ supported_capabilities: MutableSet,
+ unsupported_capabilities: MutableSet,
+ ) -> TestPmdShellSimpleMethod:
+ def reduced_fn(testpmd_shell: TestPmdShell) -> None:
+ for capability in capabilities:
+ capability.nic_capability(
+ testpmd_shell, supported_capabilities, unsupported_capabilities
+ )
+
+ return reduced_fn
+
+ def __hash__(self) -> int:
+ """Instances are identified by :attr:`nic_capability` and :attr:`capability_decorator`."""
+ return hash((self.nic_capability, self.capability_decorator))
+
+ def __repr__(self) -> str:
+ """Easy to read string of :attr:`nic_capability` and :attr:`capability_decorator`."""
+ condition_fn_name = ""
+ if self.capability_decorator:
+ condition_fn_name = f"{self.capability_decorator.__qualname__}|"
+ return f"{condition_fn_name}{self.nic_capability}"
+
+
class TestProtocol(Protocol):
"""Common test suite and test case attributes."""
@@ -116,6 +264,34 @@ def get_test_cases(cls, test_case_sublist: Sequence[str] | None = None) -> tuple
raise NotImplementedError()
+def requires(
+ *nic_capabilities: NicCapability,
+ decorator_fn: TestPmdShellDecorator | None = None,
+) -> Callable[[type[TestProtocol]], type[TestProtocol]]:
+ """A decorator that adds the required capabilities to a test case or test suite.
+
+ Args:
+ nic_capabilities: The NIC capabilities that are required by the test case or test suite.
+ decorator_fn: The decorator function that will be used when getting
+ NIC capability support status.
+ topology_type: The topology type the test suite or case requires.
+
+ Returns:
+ The decorated test case or test suite.
+ """
+
+ def add_required_capability(test_case_or_suite: type[TestProtocol]) -> type[TestProtocol]:
+ for nic_capability in nic_capabilities:
+ decorated_nic_capability = DecoratedNicCapability.get_unique(
+ nic_capability, decorator_fn
+ )
+ decorated_nic_capability.add_to_required(test_case_or_suite)
+
+ return test_case_or_suite
+
+ return add_required_capability
+
+
def get_supported_capabilities(
sut_node: SutNode,
topology_config: Topology,
diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
index 178a40385e..713549a5b2 100644
--- a/dts/tests/TestSuite_pmd_buffer_scatter.py
+++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
@@ -25,6 +25,7 @@
from framework.params.testpmd import SimpleForwardingModes
from framework.remote_session.testpmd_shell import TestPmdShell
from framework.test_suite import TestSuite, func_test
+from framework.testbed_model.capability import NicCapability, requires
class TestPmdBufferScatter(TestSuite):
@@ -123,6 +124,7 @@ def pmd_scatter(self, mbsize: int) -> None:
f"{offset}.",
)
+ @requires(NicCapability.SCATTERED_RX_ENABLED, decorator_fn=TestPmdShell.config_mtu_9000)
@func_test
def test_scatter_mbuf_2048(self) -> None:
"""Run the :meth:`pmd_scatter` test with `mbsize` set to 2048."""
--
2.34.1
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 08/12] dts: add NIC capability support
2024-08-21 14:53 ` [PATCH v3 08/12] dts: add NIC capability support Juraj Linkeš
@ 2024-08-26 17:11 ` Jeremy Spewock
2024-09-05 11:56 ` Juraj Linkeš
2024-08-27 16:36 ` Jeremy Spewock
2024-09-03 19:13 ` Dean Marx
2 siblings, 1 reply; 107+ messages in thread
From: Jeremy Spewock @ 2024-08-26 17:11 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
<snip>
> @dataclass
> class TestPmdPort(TextParser):
> """Dataclass representing the result of testpmd's ``show port info`` command."""
> @@ -962,3 +1043,96 @@ def _close(self) -> None:
> self.stop()
> self.send_command("quit", "Bye...")
> return super()._close()
> +
> + """
> + ====== Capability retrieval methods ======
> + """
> +
> + def get_capabilities_rxq_info(
> + self,
> + supported_capabilities: MutableSet["NicCapability"],
> + unsupported_capabilities: MutableSet["NicCapability"],
> + ) -> None:
> + """Get all rxq capabilities and divide them into supported and unsupported.
> +
> + Args:
> + supported_capabilities: Supported capabilities will be added to this set.
> + unsupported_capabilities: Unsupported capabilities will be added to this set.
> + """
> + self._logger.debug("Getting rxq capabilities.")
> + command = f"show rxq info {self.ports[0].id} 0"
> + rxq_info = TestPmdRxqInfo.parse(self.send_command(command))
> + if rxq_info.rx_scattered_packets:
> + supported_capabilities.add(NicCapability.SCATTERED_RX_ENABLED)
> + else:
> + unsupported_capabilities.add(NicCapability.SCATTERED_RX_ENABLED)
> +
> + """
> + ====== Decorator methods ======
> + """
> +
> + @staticmethod
> + def config_mtu_9000(testpmd_method: TestPmdShellSimpleMethod) -> TestPmdShellDecoratedMethod:
It might be more valuable for me to make a method for configuring the
MTU of all ports so that you don't have to do the loops yourself, I
can add this to the MTU patch once I update that and rebase it on
main.
> + """Configure MTU to 9000 on all ports, run `testpmd_method`, then revert.
> +
> + Args:
> + testpmd_method: The method to decorate.
> +
> + Returns:
> + The method decorated with setting and reverting MTU.
> + """
> +
> + def wrapper(testpmd_shell: Self):
> + original_mtus = []
> + for port in testpmd_shell.ports:
> + original_mtus.append((port.id, port.mtu))
> + testpmd_shell.set_port_mtu(port_id=port.id, mtu=9000, verify=False)
> + testpmd_method(testpmd_shell)
> + for port_id, mtu in original_mtus:
> + testpmd_shell.set_port_mtu(port_id=port_id, mtu=mtu if mtu else 1500, verify=False)
> +
> + return wrapper
<snip>
> diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py
> index 8899f07f76..9a79e6ebb3 100644
> --- a/dts/framework/testbed_model/capability.py
> +++ b/dts/framework/testbed_model/capability.py
> @@ -5,14 +5,40 @@
>
> This module provides a protocol that defines the common attributes of test cases and suites
> and support for test environment capabilities.
> +
> +Many test cases are testing features not available on all hardware.
> +
> +The module also allows developers to mark test cases or suites a requiring certain
small typo: I think you meant " mark test cases or suites *as*
requiring certain..."
> +hardware capabilities with the :func:`requires` decorator.
> +
> +Example:
> + .. code:: python
> +
> + from framework.test_suite import TestSuite, func_test
> + from framework.testbed_model.capability import NicCapability, requires
> + class TestPmdBufferScatter(TestSuite):
> + # only the test case requires the scattered_rx capability
> + # other test cases may not require it
> + @requires(NicCapability.scattered_rx)
Is it worth updating this to what the enum actually holds
(SCATTERED_RX_ENABLED) or not really since it is just an example in a
doc-string? I think it could do either way, but it might be better to
keep it consistent at least to start.
> + @func_test
> + def test_scatter_mbuf_2048(self):
<snip>
>
> @@ -96,6 +122,128 @@ def __hash__(self) -> int:
> """The subclasses must be hashable so that they can be stored in sets."""
>
>
> +@dataclass
> +class DecoratedNicCapability(Capability):
> + """A wrapper around :class:`~framework.remote_session.testpmd_shell.NicCapability`.
> +
> + Some NIC capabilities are only present or listed as supported only under certain conditions,
> + such as when a particular configuration is in place. This is achieved by allowing users to pass
> + a decorator function that decorates the function that gets the support status of the capability.
> +
> + New instances should be created with the :meth:`create_unique` class method to ensure
> + there are no duplicate instances.
> +
> + Attributes:
> + nic_capability: The NIC capability that partly defines each instance.
> + capability_decorator: The decorator function that will be passed the function associated
> + with `nic_capability` when discovering the support status of the capability.
> + Each instance is defined by `capability_decorator` along with `nic_capability`.
> + """
> +
> + nic_capability: NicCapability
> + capability_decorator: TestPmdShellDecorator | None
> + _unique_capabilities: ClassVar[
> + dict[Tuple[NicCapability, TestPmdShellDecorator | None], Self]
> + ] = {}
> +
> + @classmethod
> + def get_unique(
> + cls, nic_capability: NicCapability, decorator_fn: TestPmdShellDecorator | None
> + ) -> "DecoratedNicCapability":
This idea of get_unique really confused me at first. After reading
different parts of the code to learn how it is being used, I think I
understand now what it's for. My current understanding is basically
that you're using an uninstantiated class as essentially a factory
that stores a dictionary that you are using to hold singletons. It
might be confusing to me in general because I haven't really seen this
idea of dynamically modifying attributes of a class itself rather than
an instance of the class used this way. Understanding it now, it makes
sense what you are trying to do and how this is essentially a nice
cache/factory for singleton values for each capability, but It might
be helpful to document a little more somehow that _unique_capabilities
is really just a container for the singleton capabilities, and that
the top-level class is modified to keep a consistent state throughout
the framework.
Again, it could just be me having not really seen this idea used
before, but it was strange to wrap my head around at first since I'm
more used to class methods being used to read the state of attributes.
> + """Get the capability uniquely identified by `nic_capability` and `decorator_fn`.
> +
> + Args:
> + nic_capability: The NIC capability.
> + decorator_fn: The function that will be passed the function associated
> + with `nic_capability` when discovering the support status of the capability.
> +
> + Returns:
> + The capability uniquely identified by `nic_capability` and `decorator_fn`.
> + """
> + if (nic_capability, decorator_fn) not in cls._unique_capabilities:
> + cls._unique_capabilities[(nic_capability, decorator_fn)] = cls(
> + nic_capability, decorator_fn
> + )
> + return cls._unique_capabilities[(nic_capability, decorator_fn)]
> +
> + @classmethod
> + def get_supported_capabilities(
> + cls, sut_node: SutNode, topology: "Topology"
> + ) -> set["DecoratedNicCapability"]:
> + """Overrides :meth:`~Capability.get_supported_capabilities`.
> +
> + The capabilities are first sorted by decorators, then reduced into a single function which
> + is then passed to the decorator. This way we only execute each decorator only once.
This second sentence repeats the word "only" but I don't think it is
really necessary to and it might flow better with either one of them
instead of both.
> + """
> + supported_conditional_capabilities: set["DecoratedNicCapability"] = set()
> + logger = get_dts_logger(f"{sut_node.name}.{cls.__name__}")
> + if topology.type is Topology.type.no_link:
> + logger.debug(
> + "No links available in the current topology, not getting NIC capabilities."
> + )
> + return supported_conditional_capabilities
> + logger.debug(
> + f"Checking which NIC capabilities from {cls.capabilities_to_check} are supported."
> + )
> + if cls.capabilities_to_check:
> + capabilities_to_check_map = cls._get_decorated_capabilities_map()
> + with TestPmdShell(sut_node, privileged=True) as testpmd_shell:
> + for conditional_capability_fn, capabilities in capabilities_to_check_map.items():
> + supported_capabilities: set[NicCapability] = set()
> + unsupported_capabilities: set[NicCapability] = set()
> + capability_fn = cls._reduce_capabilities(
> + capabilities, supported_capabilities, unsupported_capabilities
> + )
This combines calling all of the capabilities into one function, but
if there are multiple capabilities that use the same underlying
testpmd function won't this call the same method multiple times? Or is
this handled by two Enum values in NicCapability that have the same
testpmd method as their value hashing to the same thing? For example,
if there are two capabilities that both require show rxq info and the
same decorator (scatter and some other capability X), won't this call
`show rxq info` twice even though you already know that the capability
is supported after the first call? It's not really harmful for this to
happen, but it would go against the idea of calling a method and
getting all of the capabilities that you can the first time. Maybe it
could be fixed with a conditional check which verifies if `capability`
is already in `supported_capabilities` or `unsupported_capabilities`
or not if it's a problem?
> + if conditional_capability_fn:
> + capability_fn = conditional_capability_fn(capability_fn)
> + capability_fn(testpmd_shell)
> + for supported_capability in supported_capabilities:
> + for capability in capabilities:
> + if supported_capability == capability.nic_capability:
> + supported_conditional_capabilities.add(capability)
I might be misunderstanding, but is this also achievable by just writing:
for capability in capabilities:
if capability.nic_capability in supported_capabilities:
supported_conditional_capabilities.add(capability)
I think that would be functionally the same, but I think it reads
easier than a nested loop.
> +
> + logger.debug(f"Found supported capabilities {supported_conditional_capabilities}.")
> + return supported_conditional_capabilities
> +
> + @classmethod
> + def _get_decorated_capabilities_map(
> + cls,
> + ) -> dict[TestPmdShellDecorator | None, set["DecoratedNicCapability"]]:
> + capabilities_map: dict[TestPmdShellDecorator | None, set["DecoratedNicCapability"]] = {}
> + for capability in cls.capabilities_to_check:
> + if capability.capability_decorator not in capabilities_map:
> + capabilities_map[capability.capability_decorator] = set()
> + capabilities_map[capability.capability_decorator].add(capability)
> +
> + return capabilities_map
> +
> + @classmethod
> + def _reduce_capabilities(
> + cls,
> + capabilities: set["DecoratedNicCapability"],
> + supported_capabilities: MutableSet,
> + unsupported_capabilities: MutableSet,
> + ) -> TestPmdShellSimpleMethod:
> + def reduced_fn(testpmd_shell: TestPmdShell) -> None:
> + for capability in capabilities:
> + capability.nic_capability(
> + testpmd_shell, supported_capabilities, unsupported_capabilities
> + )
> +
> + return reduced_fn
Would it make sense to put these two methods above
get_supported_capabilities since that is where they are used? I might
be in favor of it just because it would save you from having to look
further down in the diff to find what the method does and then go back
up, but I also understand that it looks like you might have been
sorting methods by private vs. public so if you think it makes more
sense to leave them here that is also viable.
> +
> + def __hash__(self) -> int:
> + """Instances are identified by :attr:`nic_capability` and :attr:`capability_decorator`."""
> + return hash((self.nic_capability, self.capability_decorator))
I guess my question above is asking if `hash(self.nic_capability) ==
hash(self.nic_capability.value())` because, if they aren't, then I
think the map will contain multiple capabilities that use the same
testpmd function since the capabilities themselves are unique, and
then because the get_supported_capabilities() method above just calls
whatever is in this map, it would call it twice. I think the whole
point of the NoAliasEnum is making sure that they don't hash to the
same thing. I could be missing something, but, if I am, maybe some
kind of comment showing where this is handled would be helpful.
> +
> + def __repr__(self) -> str:
> + """Easy to read string of :attr:`nic_capability` and :attr:`capability_decorator`."""
> + condition_fn_name = ""
> + if self.capability_decorator:
> + condition_fn_name = f"{self.capability_decorator.__qualname__}|"
> + return f"{condition_fn_name}{self.nic_capability}"
> +
> +
> class TestProtocol(Protocol):
> """Common test suite and test case attributes."""
>
> @@ -116,6 +264,34 @@ def get_test_cases(cls, test_case_sublist: Sequence[str] | None = None) -> tuple
> raise NotImplementedError()
>
>
> +def requires(
> + *nic_capabilities: NicCapability,
> + decorator_fn: TestPmdShellDecorator | None = None,
> +) -> Callable[[type[TestProtocol]], type[TestProtocol]]:
> + """A decorator that adds the required capabilities to a test case or test suite.
> +
> + Args:
> + nic_capabilities: The NIC capabilities that are required by the test case or test suite.
> + decorator_fn: The decorator function that will be used when getting
> + NIC capability support status.
> + topology_type: The topology type the test suite or case requires.
> +
> + Returns:
> + The decorated test case or test suite.
> + """
> +
> + def add_required_capability(test_case_or_suite: type[TestProtocol]) -> type[TestProtocol]:
> + for nic_capability in nic_capabilities:
> + decorated_nic_capability = DecoratedNicCapability.get_unique(
> + nic_capability, decorator_fn
> + )
> + decorated_nic_capability.add_to_required(test_case_or_suite)
> +
> + return test_case_or_suite
> +
> + return add_required_capability
> +
> +
> def get_supported_capabilities(
> sut_node: SutNode,
> topology_config: Topology,
> diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
> index 178a40385e..713549a5b2 100644
> --- a/dts/tests/TestSuite_pmd_buffer_scatter.py
> +++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
> @@ -25,6 +25,7 @@
> from framework.params.testpmd import SimpleForwardingModes
> from framework.remote_session.testpmd_shell import TestPmdShell
> from framework.test_suite import TestSuite, func_test
> +from framework.testbed_model.capability import NicCapability, requires
>
>
> class TestPmdBufferScatter(TestSuite):
> @@ -123,6 +124,7 @@ def pmd_scatter(self, mbsize: int) -> None:
> f"{offset}.",
> )
>
> + @requires(NicCapability.SCATTERED_RX_ENABLED, decorator_fn=TestPmdShell.config_mtu_9000)
Is it possible to instead associate the required decorator with the
scattered_rx capability itself? Since the configuration is required to
check the capability, I don't think there will ever be a case where
`decorator_fn` isn't required here, or a case where it is ever
anything other than modifying the MTU. Maybe it is more clear from the
reader's perspective this way that there are other things happening
under-the-hood, but it also saves developers from having to specify
something static when we already know beforehand what they need to
specify.
Doing so would probably mess up some of what you have written in the
way of DecoratedNicCapability and it might be more difficult to do it
in a way that only calls the decorator method once if there are
multiple capabilities that require the same decorator.
Maybe something that you could do is make the NicCapability class in
Testpmd have values that are tuples of (decorator_fn | None,
get_capabilities_fn), and then you can still have the
DecoratedNicCapabilitity class and the methods wouldn't really need to
change. I think the main thing that would change is just that the
decorator_fn is collected from the capability/enum instead of the
requires() method. You could potentially make get_unique easier as
well since you can just rely on the enum values since already know
what is required. Then you could take the pairs from that enum and
create a mapping like you have now of which ones require which
decorators and keep the same idea.
> @func_test
> def test_scatter_mbuf_2048(self) -> None:
> """Run the :meth:`pmd_scatter` test with `mbsize` set to 2048."""
> --
> 2.34.1
>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 08/12] dts: add NIC capability support
2024-08-26 17:11 ` Jeremy Spewock
@ 2024-09-05 11:56 ` Juraj Linkeš
2024-09-05 15:30 ` Jeremy Spewock
0 siblings, 1 reply; 107+ messages in thread
From: Juraj Linkeš @ 2024-09-05 11:56 UTC (permalink / raw)
To: Jeremy Spewock
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On 26. 8. 2024 19:11, Jeremy Spewock wrote:
> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> <snip>
>> @dataclass
>> class TestPmdPort(TextParser):
>> """Dataclass representing the result of testpmd's ``show port info`` command."""
>> @@ -962,3 +1043,96 @@ def _close(self) -> None:
>> self.stop()
>> self.send_command("quit", "Bye...")
>> return super()._close()
>> +
>> + """
>> + ====== Capability retrieval methods ======
>> + """
>> +
>> + def get_capabilities_rxq_info(
>> + self,
>> + supported_capabilities: MutableSet["NicCapability"],
>> + unsupported_capabilities: MutableSet["NicCapability"],
>> + ) -> None:
>> + """Get all rxq capabilities and divide them into supported and unsupported.
>> +
>> + Args:
>> + supported_capabilities: Supported capabilities will be added to this set.
>> + unsupported_capabilities: Unsupported capabilities will be added to this set.
>> + """
>> + self._logger.debug("Getting rxq capabilities.")
>> + command = f"show rxq info {self.ports[0].id} 0"
>> + rxq_info = TestPmdRxqInfo.parse(self.send_command(command))
>> + if rxq_info.rx_scattered_packets:
>> + supported_capabilities.add(NicCapability.SCATTERED_RX_ENABLED)
>> + else:
>> + unsupported_capabilities.add(NicCapability.SCATTERED_RX_ENABLED)
>> +
>> + """
>> + ====== Decorator methods ======
>> + """
>> +
>> + @staticmethod
>> + def config_mtu_9000(testpmd_method: TestPmdShellSimpleMethod) -> TestPmdShellDecoratedMethod:
>
> It might be more valuable for me to make a method for configuring the
> MTU of all ports so that you don't have to do the loops yourself, I
> can add this to the MTU patch once I update that and rebase it on
> main.
>
Sure, if you add that, I'll use it here. :-)
What won't work with that is the per-port restoration of MTU. But if we
assume that MTU is always the same for all ports, then I don't think
that's going to be a problem. This assumption doesn't seem unreasonable,
I don't see a scenario where it would differ.
>> + """Configure MTU to 9000 on all ports, run `testpmd_method`, then revert.
>> +
>> + Args:
>> + testpmd_method: The method to decorate.
>> +
>> + Returns:
>> + The method decorated with setting and reverting MTU.
>> + """
>> +
>> + def wrapper(testpmd_shell: Self):
>> + original_mtus = []
>> + for port in testpmd_shell.ports:
>> + original_mtus.append((port.id, port.mtu))
>> + testpmd_shell.set_port_mtu(port_id=port.id, mtu=9000, verify=False)
>> + testpmd_method(testpmd_shell)
>> + for port_id, mtu in original_mtus:
>> + testpmd_shell.set_port_mtu(port_id=port_id, mtu=mtu if mtu else 1500, verify=False)
>> +
>> + return wrapper
> <snip>
>> diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py
>> index 8899f07f76..9a79e6ebb3 100644
>> --- a/dts/framework/testbed_model/capability.py
>> +++ b/dts/framework/testbed_model/capability.py
>> @@ -5,14 +5,40 @@
>>
>> This module provides a protocol that defines the common attributes of test cases and suites
>> and support for test environment capabilities.
>> +
>> +Many test cases are testing features not available on all hardware.
>> +
>> +The module also allows developers to mark test cases or suites a requiring certain
>
> small typo: I think you meant " mark test cases or suites *as*
> requiring certain..."
>
Ack.
>> +hardware capabilities with the :func:`requires` decorator.
>> +
>> +Example:
>> + .. code:: python
>> +
>> + from framework.test_suite import TestSuite, func_test
>> + from framework.testbed_model.capability import NicCapability, requires
>> + class TestPmdBufferScatter(TestSuite):
>> + # only the test case requires the scattered_rx capability
>> + # other test cases may not require it
>> + @requires(NicCapability.scattered_rx)
>
> Is it worth updating this to what the enum actually holds
> (SCATTERED_RX_ENABLED) or not really since it is just an example in a
> doc-string? I think it could do either way, but it might be better to
> keep it consistent at least to start.
>
Yes, I overlooked this.
>> + @func_test
>> + def test_scatter_mbuf_2048(self):
> <snip>
>>
>> @@ -96,6 +122,128 @@ def __hash__(self) -> int:
>> """The subclasses must be hashable so that they can be stored in sets."""
>>
>>
>> +@dataclass
>> +class DecoratedNicCapability(Capability):
>> + """A wrapper around :class:`~framework.remote_session.testpmd_shell.NicCapability`.
>> +
>> + Some NIC capabilities are only present or listed as supported only under certain conditions,
>> + such as when a particular configuration is in place. This is achieved by allowing users to pass
>> + a decorator function that decorates the function that gets the support status of the capability.
>> +
>> + New instances should be created with the :meth:`create_unique` class method to ensure
>> + there are no duplicate instances.
>> +
>> + Attributes:
>> + nic_capability: The NIC capability that partly defines each instance.
>> + capability_decorator: The decorator function that will be passed the function associated
>> + with `nic_capability` when discovering the support status of the capability.
>> + Each instance is defined by `capability_decorator` along with `nic_capability`.
>> + """
>> +
>> + nic_capability: NicCapability
>> + capability_decorator: TestPmdShellDecorator | None
>> + _unique_capabilities: ClassVar[
>> + dict[Tuple[NicCapability, TestPmdShellDecorator | None], Self]
>> + ] = {}
>> +
>> + @classmethod
>> + def get_unique(
>> + cls, nic_capability: NicCapability, decorator_fn: TestPmdShellDecorator | None
>> + ) -> "DecoratedNicCapability":
>
> This idea of get_unique really confused me at first. After reading
> different parts of the code to learn how it is being used, I think I
> understand now what it's for. My current understanding is basically
> that you're using an uninstantiated class as essentially a factory
> that stores a dictionary that you are using to hold singletons.
Just a note, these are not singletons, just similar to them. A singleton
is just one instance of class can exist. This class allows more
instances, but it does limit the instances. It closer to an Enum, which
work exactly the same way, but only attribute names are taken into
consideration (with Enums).
> It
> might be confusing to me in general because I haven't really seen this
> idea of dynamically modifying attributes of a class itself rather than
> an instance of the class used this way. Understanding it now, it makes
> sense what you are trying to do and how this is essentially a nice
> cache/factory for singleton values for each capability, but It might
> be helpful to document a little more somehow that _unique_capabilities
> is really just a container for the singleton capabilities, and that
> the top-level class is modified to keep a consistent state throughout
> the framework.
>
> Again, it could just be me having not really seen this idea used
> before, but it was strange to wrap my head around at first since I'm
> more used to class methods being used to read the state of attributes.
>
I'm thinking of adding this to get_unique's docstring:
This is a factory method that implements a quasi-enum pattern.
The instances of this class are stored in a class variable,
_unique_capabilities.
If an instance with `nic_capability` and `decorator_fn` as inputs
doesn't exist, it is created and added to _unique_capabilities.
If it exists, it is returned so that a new identical instance is not
created.
>> + """Get the capability uniquely identified by `nic_capability` and `decorator_fn`.
>> +
>> + Args:
>> + nic_capability: The NIC capability.
>> + decorator_fn: The function that will be passed the function associated
>> + with `nic_capability` when discovering the support status of the capability.
>> +
>> + Returns:
>> + The capability uniquely identified by `nic_capability` and `decorator_fn`.
>> + """
>> + if (nic_capability, decorator_fn) not in cls._unique_capabilities:
>> + cls._unique_capabilities[(nic_capability, decorator_fn)] = cls(
>> + nic_capability, decorator_fn
>> + )
>> + return cls._unique_capabilities[(nic_capability, decorator_fn)]
>> +
>> + @classmethod
>> + def get_supported_capabilities(
>> + cls, sut_node: SutNode, topology: "Topology"
>> + ) -> set["DecoratedNicCapability"]:
>> + """Overrides :meth:`~Capability.get_supported_capabilities`.
>> +
>> + The capabilities are first sorted by decorators, then reduced into a single function which
>> + is then passed to the decorator. This way we only execute each decorator only once.
>
> This second sentence repeats the word "only" but I don't think it is
> really necessary to and it might flow better with either one of them
> instead of both.
>
Ack.
>> + """
>> + supported_conditional_capabilities: set["DecoratedNicCapability"] = set()
>> + logger = get_dts_logger(f"{sut_node.name}.{cls.__name__}")
>> + if topology.type is Topology.type.no_link:
>> + logger.debug(
>> + "No links available in the current topology, not getting NIC capabilities."
>> + )
>> + return supported_conditional_capabilities
>> + logger.debug(
>> + f"Checking which NIC capabilities from {cls.capabilities_to_check} are supported."
>> + )
>> + if cls.capabilities_to_check:
>> + capabilities_to_check_map = cls._get_decorated_capabilities_map()
>> + with TestPmdShell(sut_node, privileged=True) as testpmd_shell:
>> + for conditional_capability_fn, capabilities in capabilities_to_check_map.items():
>> + supported_capabilities: set[NicCapability] = set()
>> + unsupported_capabilities: set[NicCapability] = set()
>> + capability_fn = cls._reduce_capabilities(
>> + capabilities, supported_capabilities, unsupported_capabilities
>> + )
>
> This combines calling all of the capabilities into one function, but
> if there are multiple capabilities that use the same underlying
> testpmd function won't this call the same method multiple times? Or is
> this handled by two Enum values in NicCapability that have the same
> testpmd method as their value hashing to the same thing? For example,
> if there are two capabilities that both require show rxq info and the
> same decorator (scatter and some other capability X), won't this call
> `show rxq info` twice even though you already know that the capability
> is supported after the first call? It's not really harmful for this to
> happen, but it would go against the idea of calling a method and
> getting all of the capabilities that you can the first time. Maybe it
> could be fixed with a conditional check which verifies if `capability`
> is already in `supported_capabilities` or `unsupported_capabilities`
> or not if it's a problem?
>
All you say is true. The whole reason for using all these sets is that
we don't call the functions multiple times. The check you mention is
exactly what's missing.
>> + if conditional_capability_fn:
>> + capability_fn = conditional_capability_fn(capability_fn)
>> + capability_fn(testpmd_shell)
>> + for supported_capability in supported_capabilities:
>> + for capability in capabilities:
>> + if supported_capability == capability.nic_capability:
>> + supported_conditional_capabilities.add(capability)
>
> I might be misunderstanding, but is this also achievable by just writing:
>
> for capability in capabilities:
> if capability.nic_capability in supported_capabilities:
> supported_conditional_capabilities.add(capability)
>
> I think that would be functionally the same, but I think it reads
> easier than a nested loop.
>
It is the same thing, I'll change it.
>> +
>> + logger.debug(f"Found supported capabilities {supported_conditional_capabilities}.")
>> + return supported_conditional_capabilities
>> +
>> + @classmethod
>> + def _get_decorated_capabilities_map(
>> + cls,
>> + ) -> dict[TestPmdShellDecorator | None, set["DecoratedNicCapability"]]:
>> + capabilities_map: dict[TestPmdShellDecorator | None, set["DecoratedNicCapability"]] = {}
>> + for capability in cls.capabilities_to_check:
>> + if capability.capability_decorator not in capabilities_map:
>> + capabilities_map[capability.capability_decorator] = set()
>> + capabilities_map[capability.capability_decorator].add(capability)
>> +
>> + return capabilities_map
>> +
>> + @classmethod
>> + def _reduce_capabilities(
>> + cls,
>> + capabilities: set["DecoratedNicCapability"],
>> + supported_capabilities: MutableSet,
>> + unsupported_capabilities: MutableSet,
>> + ) -> TestPmdShellSimpleMethod:
>> + def reduced_fn(testpmd_shell: TestPmdShell) -> None:
>> + for capability in capabilities:
This is where I'll add the fix:
if capability not in supported_capabilities | unsupported_capabilities:
>> + capability.nic_capability(
>> + testpmd_shell, supported_capabilities, unsupported_capabilities
>> + )
>> +
>> + return reduced_fn
>
> Would it make sense to put these two methods above
> get_supported_capabilities since that is where they are used? I might
> be in favor of it just because it would save you from having to look
> further down in the diff to find what the method does and then go back
> up, but I also understand that it looks like you might have been
> sorting methods by private vs. public so if you think it makes more
> sense to leave them here that is also viable.
>
I sorted it this what so that the code it's easier to read (in my
opinion). I read the method, what it does, then the method calls a
method I haven't seen so I go look beneath the method for the method
definition. To me, this is preferable that reading methods I haven't
seen before. Or, put in another way, the methods are sorted in the order
they're used in code (that's how the code is executed and that's why
this order feels natural to me).
>> +
>> + def __hash__(self) -> int:
>> + """Instances are identified by :attr:`nic_capability` and :attr:`capability_decorator`."""
>> + return hash((self.nic_capability, self.capability_decorator))
>
> I guess my question above is asking if `hash(self.nic_capability) ==
> hash(self.nic_capability.value())` because, if they aren't, then I
> think the map will contain multiple capabilities that use the same
> testpmd function since the capabilities themselves are unique, and
> then because the get_supported_capabilities() method above just calls
> whatever is in this map, it would call it twice. I think the whole
> point of the NoAliasEnum is making sure that they don't hash to the
> same thing. I could be missing something, but, if I am, maybe some
> kind of comment showing where this is handled would be helpful.
>
I think the simple fix in _reduce_capabilities() addresses this, right?
>> +
>> + def __repr__(self) -> str:
>> + """Easy to read string of :attr:`nic_capability` and :attr:`capability_decorator`."""
>> + condition_fn_name = ""
>> + if self.capability_decorator:
>> + condition_fn_name = f"{self.capability_decorator.__qualname__}|"
>> + return f"{condition_fn_name}{self.nic_capability}"
>> +
>> +
>> class TestProtocol(Protocol):
>> """Common test suite and test case attributes."""
>>
>> @@ -116,6 +264,34 @@ def get_test_cases(cls, test_case_sublist: Sequence[str] | None = None) -> tuple
>> raise NotImplementedError()
>>
>>
>> +def requires(
>> + *nic_capabilities: NicCapability,
>> + decorator_fn: TestPmdShellDecorator | None = None,
>> +) -> Callable[[type[TestProtocol]], type[TestProtocol]]:
>> + """A decorator that adds the required capabilities to a test case or test suite.
>> +
>> + Args:
>> + nic_capabilities: The NIC capabilities that are required by the test case or test suite.
>> + decorator_fn: The decorator function that will be used when getting
>> + NIC capability support status.
>> + topology_type: The topology type the test suite or case requires.
>> +
>> + Returns:
>> + The decorated test case or test suite.
>> + """
>> +
>> + def add_required_capability(test_case_or_suite: type[TestProtocol]) -> type[TestProtocol]:
>> + for nic_capability in nic_capabilities:
>> + decorated_nic_capability = DecoratedNicCapability.get_unique(
>> + nic_capability, decorator_fn
>> + )
>> + decorated_nic_capability.add_to_required(test_case_or_suite)
>> +
>> + return test_case_or_suite
>> +
>> + return add_required_capability
>> +
>> +
>> def get_supported_capabilities(
>> sut_node: SutNode,
>> topology_config: Topology,
>> diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
>> index 178a40385e..713549a5b2 100644
>> --- a/dts/tests/TestSuite_pmd_buffer_scatter.py
>> +++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
>> @@ -25,6 +25,7 @@
>> from framework.params.testpmd import SimpleForwardingModes
>> from framework.remote_session.testpmd_shell import TestPmdShell
>> from framework.test_suite import TestSuite, func_test
>> +from framework.testbed_model.capability import NicCapability, requires
>>
>>
>> class TestPmdBufferScatter(TestSuite):
>> @@ -123,6 +124,7 @@ def pmd_scatter(self, mbsize: int) -> None:
>> f"{offset}.",
>> )
>>
>> + @requires(NicCapability.SCATTERED_RX_ENABLED, decorator_fn=TestPmdShell.config_mtu_9000)
>
> Is it possible to instead associate the required decorator with the
> scattered_rx capability itself? Since the configuration is required to
> check the capability, I don't think there will ever be a case where
> `decorator_fn` isn't required here, or a case where it is ever
> anything other than modifying the MTU. Maybe it is more clear from the
> reader's perspective this way that there are other things happening
> under-the-hood, but it also saves developers from having to specify
> something static when we already know beforehand what they need to
> specify.
>
> Doing so would probably mess up some of what you have written in the
> way of DecoratedNicCapability and it might be more difficult to do it
> in a way that only calls the decorator method once if there are
> multiple capabilities that require the same decorator.
>
> Maybe something that you could do is make the NicCapability class in
> Testpmd have values that are tuples of (decorator_fn | None,
> get_capabilities_fn), and then you can still have the
> DecoratedNicCapabilitity class and the methods wouldn't really need to
> change. I think the main thing that would change is just that the
> decorator_fn is collected from the capability/enum instead of the
> requires() method. You could potentially make get_unique easier as
> well since you can just rely on the enum values since already know
> what is required. Then you could take the pairs from that enum and
> create a mapping like you have now of which ones require which
> decorators and keep the same idea.
>
All good points, this is a really good suggestion. Great for the writer
of the tests and basically no downsides, except maybe if there is a
capability which works under different conditions (and we'd want to use
all that), but even with that, we could have different capability names
for tuples with the same capability, but different decorator_fn.
I'll rework this, seems very much worth it.
>> @func_test
>> def test_scatter_mbuf_2048(self) -> None:
>> """Run the :meth:`pmd_scatter` test with `mbsize` set to 2048."""
>> --
>> 2.34.1
>>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 08/12] dts: add NIC capability support
2024-09-05 11:56 ` Juraj Linkeš
@ 2024-09-05 15:30 ` Jeremy Spewock
0 siblings, 0 replies; 107+ messages in thread
From: Jeremy Spewock @ 2024-09-05 15:30 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Thu, Sep 5, 2024 at 7:56 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>
>
>
> On 26. 8. 2024 19:11, Jeremy Spewock wrote:
> > On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> > <juraj.linkes@pantheon.tech> wrote:
> > <snip>
> >> @dataclass
> >> class TestPmdPort(TextParser):
> >> """Dataclass representing the result of testpmd's ``show port info`` command."""
> >> @@ -962,3 +1043,96 @@ def _close(self) -> None:
> >> self.stop()
> >> self.send_command("quit", "Bye...")
> >> return super()._close()
> >> +
> >> + """
> >> + ====== Capability retrieval methods ======
> >> + """
> >> +
> >> + def get_capabilities_rxq_info(
> >> + self,
> >> + supported_capabilities: MutableSet["NicCapability"],
> >> + unsupported_capabilities: MutableSet["NicCapability"],
> >> + ) -> None:
> >> + """Get all rxq capabilities and divide them into supported and unsupported.
> >> +
> >> + Args:
> >> + supported_capabilities: Supported capabilities will be added to this set.
> >> + unsupported_capabilities: Unsupported capabilities will be added to this set.
> >> + """
> >> + self._logger.debug("Getting rxq capabilities.")
> >> + command = f"show rxq info {self.ports[0].id} 0"
> >> + rxq_info = TestPmdRxqInfo.parse(self.send_command(command))
> >> + if rxq_info.rx_scattered_packets:
> >> + supported_capabilities.add(NicCapability.SCATTERED_RX_ENABLED)
> >> + else:
> >> + unsupported_capabilities.add(NicCapability.SCATTERED_RX_ENABLED)
> >> +
> >> + """
> >> + ====== Decorator methods ======
> >> + """
> >> +
> >> + @staticmethod
> >> + def config_mtu_9000(testpmd_method: TestPmdShellSimpleMethod) -> TestPmdShellDecoratedMethod:
> >
> > It might be more valuable for me to make a method for configuring the
> > MTU of all ports so that you don't have to do the loops yourself, I
> > can add this to the MTU patch once I update that and rebase it on
> > main.
> >
>
> Sure, if you add that, I'll use it here. :-)
> What won't work with that is the per-port restoration of MTU. But if we
> assume that MTU is always the same for all ports, then I don't think
> that's going to be a problem. This assumption doesn't seem unreasonable,
> I don't see a scenario where it would differ.
Good point, and something I didn't think about. I doubt they would be
different either though and I think it would generally be fine to
assume they are the same, but that could also be reason to do it on a
per-port basis. Whatever you think is best. Setting the MTU on all
ports isn't as efficient as I thought it would be when I first wrote
this comment anyway since testpmd doesn't offer something like a `port
config mtu all`, so I just do it one port at a time anyway.
>
> >> + """Configure MTU to 9000 on all ports, run `testpmd_method`, then revert.
> >> +
> >> + Args:
> >> + testpmd_method: The method to decorate.
> >> +
> >> + Returns:
> >> + The method decorated with setting and reverting MTU.
> >> + """
> >> +
<snip>
> >> + @classmethod
> >> + def get_unique(
> >> + cls, nic_capability: NicCapability, decorator_fn: TestPmdShellDecorator | None
> >> + ) -> "DecoratedNicCapability":
> >
> > This idea of get_unique really confused me at first. After reading
> > different parts of the code to learn how it is being used, I think I
> > understand now what it's for. My current understanding is basically
> > that you're using an uninstantiated class as essentially a factory
> > that stores a dictionary that you are using to hold singletons.
>
> Just a note, these are not singletons, just similar to them. A singleton
> is just one instance of class can exist. This class allows more
> instances, but it does limit the instances. It closer to an Enum, which
> work exactly the same way, but only attribute names are taken into
> consideration (with Enums).
That's a good distinction to make. Singleton was the closest thing
that I could make the connection too, but you're right that it isn't
the same and the comparison to Enums makes a lot of sense.
>
> > It
> > might be confusing to me in general because I haven't really seen this
> > idea of dynamically modifying attributes of a class itself rather than
> > an instance of the class used this way. Understanding it now, it makes
> > sense what you are trying to do and how this is essentially a nice
> > cache/factory for singleton values for each capability, but It might
> > be helpful to document a little more somehow that _unique_capabilities
> > is really just a container for the singleton capabilities, and that
> > the top-level class is modified to keep a consistent state throughout
> > the framework.
> >
> > Again, it could just be me having not really seen this idea used
> > before, but it was strange to wrap my head around at first since I'm
> > more used to class methods being used to read the state of attributes.
> >
>
> I'm thinking of adding this to get_unique's docstring:
>
> This is a factory method that implements a quasi-enum pattern.
> The instances of this class are stored in a class variable,
> _unique_capabilities.
>
> If an instance with `nic_capability` and `decorator_fn` as inputs
> doesn't exist, it is created and added to _unique_capabilities.
> If it exists, it is returned so that a new identical instance is not
> created.
Sure, I think this reads pretty well, and I like specifically calling
out the pattern so that if anyone was unfamiliar it gives them
something to research.
>
>
> >> + """Get the capability uniquely identified by `nic_capability` and `decorator_fn`.
> >> +
> >> + Args:
> >> + nic_capability: The NIC capability.
> >> + decorator_fn: The function that will be passed the function associated
> >> + with `nic_capability` when discovering the support status of the capability.
> >> +
> >> + Returns:
> >> + The capability uniquely identified by `nic_capability` and `decorator_fn`.
> >> + """
<snip>
> >> + @classmethod
> >> + def _reduce_capabilities(
> >> + cls,
> >> + capabilities: set["DecoratedNicCapability"],
> >> + supported_capabilities: MutableSet,
> >> + unsupported_capabilities: MutableSet,
> >> + ) -> TestPmdShellSimpleMethod:
> >> + def reduced_fn(testpmd_shell: TestPmdShell) -> None:
> >> + for capability in capabilities:
>
> This is where I'll add the fix:
> if capability not in supported_capabilities | unsupported_capabilities:
>
Perfect, I think that would solve it, yes.
> >> + capability.nic_capability(
> >> + testpmd_shell, supported_capabilities, unsupported_capabilities
> >> + )
> >> +
> >> + return reduced_fn
> >
> > Would it make sense to put these two methods above
> > get_supported_capabilities since that is where they are used? I might
> > be in favor of it just because it would save you from having to look
> > further down in the diff to find what the method does and then go back
> > up, but I also understand that it looks like you might have been
> > sorting methods by private vs. public so if you think it makes more
> > sense to leave them here that is also viable.
> >
>
> I sorted it this what so that the code it's easier to read (in my
> opinion). I read the method, what it does, then the method calls a
> method I haven't seen so I go look beneath the method for the method
> definition. To me, this is preferable that reading methods I haven't
> seen before. Or, put in another way, the methods are sorted in the order
> they're used in code (that's how the code is executed and that's why
> this order feels natural to me).
Right, that does also make sense and is more accurate to how the code
runs. I think it is fine to leave this way then.
>
> >> +
> >> + def __hash__(self) -> int:
> >> + """Instances are identified by :attr:`nic_capability` and :attr:`capability_decorator`."""
> >> + return hash((self.nic_capability, self.capability_decorator))
> >
> > I guess my question above is asking if `hash(self.nic_capability) ==
> > hash(self.nic_capability.value())` because, if they aren't, then I
> > think the map will contain multiple capabilities that use the same
> > testpmd function since the capabilities themselves are unique, and
> > then because the get_supported_capabilities() method above just calls
> > whatever is in this map, it would call it twice. I think the whole
> > point of the NoAliasEnum is making sure that they don't hash to the
> > same thing. I could be missing something, but, if I am, maybe some
> > kind of comment showing where this is handled would be helpful.
> >
>
> I think the simple fix in _reduce_capabilities() addresses this, right?
Yes it does, and it does so better than if the two hashes were equal anyway.
>
> >> +
<snip>
>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 08/12] dts: add NIC capability support
2024-08-21 14:53 ` [PATCH v3 08/12] dts: add NIC capability support Juraj Linkeš
2024-08-26 17:11 ` Jeremy Spewock
@ 2024-08-27 16:36 ` Jeremy Spewock
2024-09-18 12:58 ` Juraj Linkeš
2024-09-03 19:13 ` Dean Marx
2 siblings, 1 reply; 107+ messages in thread
From: Jeremy Spewock @ 2024-08-27 16:36 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
<snip>
> diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py
> index 8899f07f76..9a79e6ebb3 100644
> --- a/dts/framework/testbed_model/capability.py
> +++ b/dts/framework/testbed_model/capability.py
> @@ -5,14 +5,40 @@
<snip>
> + @classmethod
> + def get_supported_capabilities(
> + cls, sut_node: SutNode, topology: "Topology"
> + ) -> set["DecoratedNicCapability"]:
> + """Overrides :meth:`~Capability.get_supported_capabilities`.
> +
> + The capabilities are first sorted by decorators, then reduced into a single function which
> + is then passed to the decorator. This way we only execute each decorator only once.
> + """
> + supported_conditional_capabilities: set["DecoratedNicCapability"] = set()
> + logger = get_dts_logger(f"{sut_node.name}.{cls.__name__}")
> + if topology.type is Topology.type.no_link:
As a follow-up, I didn't notice this during my initial review, but in
testing this line was throwing attribute errors for me due to Topology
not having an attribute named `type`. I think this was because of
`Topology.type.no_link` since this attribute isn't initialized on the
class itself. I fixed this by just replacing it with
`TopologyType.no_link` locally.
> + logger.debug(
> + "No links available in the current topology, not getting NIC capabilities."
> + )
> + return supported_conditional_capabilities
> + logger.debug(
> + f"Checking which NIC capabilities from {cls.capabilities_to_check} are supported."
> + )
> + if cls.capabilities_to_check:
> + capabilities_to_check_map = cls._get_decorated_capabilities_map()
> + with TestPmdShell(sut_node, privileged=True) as testpmd_shell:
> + for conditional_capability_fn, capabilities in capabilities_to_check_map.items():
> + supported_capabilities: set[NicCapability] = set()
<snip>
>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 08/12] dts: add NIC capability support
2024-08-27 16:36 ` Jeremy Spewock
@ 2024-09-18 12:58 ` Juraj Linkeš
2024-09-18 16:52 ` Jeremy Spewock
0 siblings, 1 reply; 107+ messages in thread
From: Juraj Linkeš @ 2024-09-18 12:58 UTC (permalink / raw)
To: Jeremy Spewock
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On 27. 8. 2024 18:36, Jeremy Spewock wrote:
> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> <snip>
>> diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py
>> index 8899f07f76..9a79e6ebb3 100644
>> --- a/dts/framework/testbed_model/capability.py
>> +++ b/dts/framework/testbed_model/capability.py
>> @@ -5,14 +5,40 @@
> <snip>
>> + @classmethod
>> + def get_supported_capabilities(
>> + cls, sut_node: SutNode, topology: "Topology"
>> + ) -> set["DecoratedNicCapability"]:
>> + """Overrides :meth:`~Capability.get_supported_capabilities`.
>> +
>> + The capabilities are first sorted by decorators, then reduced into a single function which
>> + is then passed to the decorator. This way we only execute each decorator only once.
>> + """
>> + supported_conditional_capabilities: set["DecoratedNicCapability"] = set()
>> + logger = get_dts_logger(f"{sut_node.name}.{cls.__name__}")
>> + if topology.type is Topology.type.no_link:
>
> As a follow-up, I didn't notice this during my initial review, but in
> testing this line was throwing attribute errors for me due to Topology
> not having an attribute named `type`. I think this was because of
> `Topology.type.no_link` since this attribute isn't initialized on the
> class itself. I fixed this by just replacing it with
> `TopologyType.no_link` locally.
>
I also ran into this, the type attribute is not a class variable. Your
solution works (and I also originally fixed it with exactly that), but I
then I realized topology.type.no_link also works (and was probably my
intention), which doesn't require the extra import of TopologyType.
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 08/12] dts: add NIC capability support
2024-09-18 12:58 ` Juraj Linkeš
@ 2024-09-18 16:52 ` Jeremy Spewock
0 siblings, 0 replies; 107+ messages in thread
From: Jeremy Spewock @ 2024-09-18 16:52 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Sep 18, 2024 at 8:58 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>
>
>
> On 27. 8. 2024 18:36, Jeremy Spewock wrote:
> > On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> > <juraj.linkes@pantheon.tech> wrote:
> > <snip>
> >> diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py
> >> index 8899f07f76..9a79e6ebb3 100644
> >> --- a/dts/framework/testbed_model/capability.py
> >> +++ b/dts/framework/testbed_model/capability.py
> >> @@ -5,14 +5,40 @@
> > <snip>
> >> + @classmethod
> >> + def get_supported_capabilities(
> >> + cls, sut_node: SutNode, topology: "Topology"
> >> + ) -> set["DecoratedNicCapability"]:
> >> + """Overrides :meth:`~Capability.get_supported_capabilities`.
> >> +
> >> + The capabilities are first sorted by decorators, then reduced into a single function which
> >> + is then passed to the decorator. This way we only execute each decorator only once.
> >> + """
> >> + supported_conditional_capabilities: set["DecoratedNicCapability"] = set()
> >> + logger = get_dts_logger(f"{sut_node.name}.{cls.__name__}")
> >> + if topology.type is Topology.type.no_link:
> >
> > As a follow-up, I didn't notice this during my initial review, but in
> > testing this line was throwing attribute errors for me due to Topology
> > not having an attribute named `type`. I think this was because of
> > `Topology.type.no_link` since this attribute isn't initialized on the
> > class itself. I fixed this by just replacing it with
> > `TopologyType.no_link` locally.
> >
>
> I also ran into this, the type attribute is not a class variable. Your
> solution works (and I also originally fixed it with exactly that), but I
> then I realized topology.type.no_link also works (and was probably my
> intention), which doesn't require the extra import of TopologyType.
Right, that's smart. I forget that you can do that with enums.
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 08/12] dts: add NIC capability support
2024-08-21 14:53 ` [PATCH v3 08/12] dts: add NIC capability support Juraj Linkeš
2024-08-26 17:11 ` Jeremy Spewock
2024-08-27 16:36 ` Jeremy Spewock
@ 2024-09-03 19:13 ` Dean Marx
2 siblings, 0 replies; 107+ messages in thread
From: Dean Marx @ 2024-09-03 19:13 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 1162 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> Some test cases or suites may be testing a NIC feature that is not
> supported on all NICs, so add support for marking test cases or suites
> as requiring NIC capabilities.
>
> The marking is done with a decorator, which populates the internal
> required_capabilities attribute of TestProtocol. The NIC capability
> itself is a wrapper around the NicCapability defined in testpmd_shell.
> The reason is twofold:
> 1. Enums cannot be extended and the class implements the methods of its
> abstract base superclass,
> 2. The class also stores an optional decorator function which is used
> before/after capability retrieval. This is needed because some
> capabilities may be advertised differently under different
> configuration.
>
> The decorator API is designed to be simple to use. The arguments passed
> to it are all from the testpmd shell. Everything else (even the actual
> capability object creation) is done internally.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 1501 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH v3 09/12] dts: add topology capability
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
` (7 preceding siblings ...)
2024-08-21 14:53 ` [PATCH v3 08/12] dts: add NIC capability support Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 17:13 ` Jeremy Spewock
2024-09-03 17:50 ` Dean Marx
2024-08-21 14:53 ` [PATCH v3 10/12] doc: add DTS capability doc sources Juraj Linkeš
` (3 subsequent siblings)
12 siblings, 2 replies; 107+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
Add support for marking test cases as requiring a certain topology. The
default topology is a two link topology and the other supported
topologies are one link and no link topologies.
The TestProtocol of test suites and cases is extended with the topology
type each test suite or case requires. Each test case starts out as
requiring a two link topology and can be marked as requiring as
topology directly (by decorating the test case) or through its test
suite. If a test suite is decorated as requiring a certain topology, all
its test cases are marked as such. If both test suite and a test case
are decorated as requiring a topology, the test case cannot require a
more complex topology than the whole suite (but it can require a less
complex one). If a test suite is not decorated, this has no effect on
required test case topology.
Since the default topology is defined as a reference to one of the
actual topologies, the NoAliasEnum from the aenum package is utilized,
which removes the aliasing of Enums so that TopologyType.two_links and
TopologyType.default are distinct. This is needed to distinguish between
a user passed value and the default value being used (which is used when
a test suite is or isn't decorated).
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/test_suite.py | 6 +-
dts/framework/testbed_model/capability.py | 182 +++++++++++++++++++++-
dts/framework/testbed_model/topology.py | 35 ++++-
dts/tests/TestSuite_hello_world.py | 2 +
dts/tests/TestSuite_pmd_buffer_scatter.py | 8 +-
dts/tests/TestSuite_smoke_tests.py | 2 +
6 files changed, 217 insertions(+), 18 deletions(-)
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 5c393ce8bf..51f49bd601 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -27,7 +27,7 @@
from framework.testbed_model.port import Port
from framework.testbed_model.sut_node import SutNode
from framework.testbed_model.tg_node import TGNode
-from framework.testbed_model.topology import Topology, TopologyType
+from framework.testbed_model.topology import Topology
from framework.testbed_model.traffic_generator.capturing_traffic_generator import (
PacketFilteringConfig,
)
@@ -73,7 +73,6 @@ class TestSuite(TestProtocol):
#: will block the execution of all subsequent test suites in the current build target.
is_blocking: ClassVar[bool] = False
_logger: DTSLogger
- _topology_type: TopologyType
_sut_port_ingress: Port
_sut_port_egress: Port
_sut_ip_address_ingress: Union[IPv4Interface, IPv6Interface]
@@ -102,7 +101,6 @@ def __init__(
self.sut_node = sut_node
self.tg_node = tg_node
self._logger = get_dts_logger(self.__class__.__name__)
- self._topology_type = topology.type
self._tg_port_egress = topology.tg_port_egress
self._sut_port_ingress = topology.sut_port_ingress
self._sut_port_egress = topology.sut_port_egress
@@ -468,6 +466,8 @@ def _decorator(func: TestSuiteMethodType) -> type[TestCase]:
test_case.skip = cls.skip
test_case.skip_reason = cls.skip_reason
test_case.required_capabilities = set()
+ test_case.topology_type = cls.topology_type
+ test_case.topology_type.add_to_required(test_case)
test_case.test_type = test_case_type
return test_case
diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py
index 9a79e6ebb3..998efa95d2 100644
--- a/dts/framework/testbed_model/capability.py
+++ b/dts/framework/testbed_model/capability.py
@@ -7,11 +7,29 @@
and support for test environment capabilities.
Many test cases are testing features not available on all hardware.
+On the other hand, some test cases or suites may not need the most complex topology available.
-The module also allows developers to mark test cases or suites a requiring certain
-hardware capabilities with the :func:`requires` decorator.
+The module allows developers to mark test cases or suites a requiring certain hardware capabilities
+or a particular topology with the :func:`requires` decorator.
+
+There are differences between hardware and topology capabilities:
+
+ * Hardware capabilities are assumed to not be required when not specified.
+ * However, some topology is always available, so each test case or suite is assigned
+ a default topology if no topology is specified in the decorator.
+
+Examples:
+ .. code:: python
+
+ from framework.test_suite import TestSuite, func_test
+ from framework.testbed_model.capability import TopologyType, requires
+ # The whole test suite (each test case within) doesn't require any links.
+ @requires(topology_type=TopologyType.no_link)
+ @func_test
+ class TestHelloWorld(TestSuite):
+ def hello_world_single_core(self):
+ ...
-Example:
.. code:: python
from framework.test_suite import TestSuite, func_test
@@ -24,6 +42,7 @@ class TestPmdBufferScatter(TestSuite):
def test_scatter_mbuf_2048(self):
"""
+import inspect
from abc import ABC, abstractmethod
from collections.abc import MutableSet, Sequence
from dataclasses import dataclass
@@ -31,6 +50,7 @@ def test_scatter_mbuf_2048(self):
from typing_extensions import Self
+from framework.exception import ConfigurationError
from framework.logger import get_dts_logger
from framework.remote_session.testpmd_shell import (
NicCapability,
@@ -40,7 +60,7 @@ def test_scatter_mbuf_2048(self):
)
from .sut_node import SutNode
-from .topology import Topology
+from .topology import Topology, TopologyType
class Capability(ABC):
@@ -244,6 +264,154 @@ def __repr__(self) -> str:
return f"{condition_fn_name}{self.nic_capability}"
+@dataclass
+class TopologyCapability(Capability):
+ """A wrapper around :class:`~.topology.TopologyType`.
+
+ Each test case must be assigned a topology. It could be done explicitly;
+ the implicit default is :attr:`~.topology.TopologyType.default`, which this class defines
+ as equal to :attr:`~.topology.TopologyType.two_links`.
+
+ Test case topology may be set by setting the topology for the whole suite.
+ The priority in which topology is set is as follows:
+
+ #. The topology set using the :func:`requires` decorator with a test case,
+ #. The topology set using the :func:`requires` decorator with a test suite,
+ #. The default topology if the decorator is not used.
+
+ The default topology of test suite (i.e. when not using the decorator
+ or not setting the topology with the decorator) does not affect the topology of test cases.
+
+ New instances should be created with the :meth:`create_unique` class method to ensure
+ there are no duplicate instances.
+
+ Attributes:
+ topology_type: The topology type that defines each instance.
+ """
+
+ topology_type: TopologyType
+
+ _unique_capabilities: ClassVar[dict[str, Self]] = {}
+
+ def _preprocess_required(self, test_case_or_suite: type["TestProtocol"]) -> None:
+ test_case_or_suite.required_capabilities.discard(test_case_or_suite.topology_type)
+ test_case_or_suite.topology_type = self
+
+ @classmethod
+ def get_unique(cls, topology_type: TopologyType) -> "TopologyCapability":
+ """Get the capability uniquely identified by `topology_type`.
+
+ Args:
+ topology_type: The topology type.
+
+ Returns:
+ The capability uniquely identified by `topology_type`.
+ """
+ if topology_type.name not in cls._unique_capabilities:
+ cls._unique_capabilities[topology_type.name] = cls(topology_type)
+ return cls._unique_capabilities[topology_type.name]
+
+ @classmethod
+ def get_supported_capabilities(
+ cls, sut_node: SutNode, topology: "Topology"
+ ) -> set["TopologyCapability"]:
+ """Overrides :meth:`~OSSession.get_supported_capabilities`."""
+ supported_capabilities = set()
+ topology_capability = cls.get_unique(topology.type)
+ for topology_type in TopologyType:
+ candidate_topology_type = cls.get_unique(topology_type)
+ if candidate_topology_type <= topology_capability:
+ supported_capabilities.add(candidate_topology_type)
+ return supported_capabilities
+
+ def set_required(self, test_case_or_suite: type["TestProtocol"]) -> None:
+ """The logic for setting the required topology of a test case or suite.
+
+ Decorators are applied on methods of a class first, then on the class.
+ This means we have to modify test case topologies when processing the test suite topologies.
+ At that point, the test case topologies have been set by the :func:`requires` decorator.
+ The test suite topology only affects the test case topologies
+ if not :attr:`~.topology.TopologyType.default`.
+ """
+ if inspect.isclass(test_case_or_suite):
+ if self.topology_type is not TopologyType.default:
+ self.add_to_required(test_case_or_suite)
+ func_test_cases, perf_test_cases = test_case_or_suite.get_test_cases()
+ for test_case in func_test_cases | perf_test_cases:
+ if test_case.topology_type.topology_type is TopologyType.default:
+ # test case topology has not been set, use the one set by the test suite
+ self.add_to_required(test_case)
+ elif test_case.topology_type > test_case_or_suite.topology_type:
+ raise ConfigurationError(
+ "The required topology type of a test case "
+ f"({test_case.__name__}|{test_case.topology_type}) "
+ "cannot be more complex than that of a suite "
+ f"({test_case_or_suite.__name__}|{test_case_or_suite.topology_type})."
+ )
+ else:
+ self.add_to_required(test_case_or_suite)
+
+ def __eq__(self, other) -> bool:
+ """Compare the :attr:`~TopologyCapability.topology_type`s.
+
+ Args:
+ other: The object to compare with.
+
+ Returns:
+ :data:`True` if the topology types are the same.
+ """
+ return self.topology_type == other.topology_type
+
+ def __lt__(self, other) -> bool:
+ """Compare the :attr:`~TopologyCapability.topology_type`s.
+
+ Args:
+ other: The object to compare with.
+
+ Returns:
+ :data:`True` if the instance's topology type is less complex than the compared object's.
+ """
+ return self.topology_type < other.topology_type
+
+ def __gt__(self, other) -> bool:
+ """Compare the :attr:`~TopologyCapability.topology_type`s.
+
+ Args:
+ other: The object to compare with.
+
+ Returns:
+ :data:`True` if the instance's topology type is more complex than the compared object's.
+ """
+ return other < self
+
+ def __le__(self, other) -> bool:
+ """Compare the :attr:`~TopologyCapability.topology_type`s.
+
+ Args:
+ other: The object to compare with.
+
+ Returns:
+ :data:`True` if the instance's topology type is less complex or equal than
+ the compared object's.
+ """
+ return not self > other
+
+ def __hash__(self):
+ """Each instance is identified by :attr:`topology_type`."""
+ return self.topology_type.__hash__()
+
+ def __str__(self):
+ """Easy to read string of class and name of :attr:`topology_type`."""
+ name = self.topology_type.name
+ if self.topology_type is TopologyType.default:
+ name = TopologyType.get_from_value(self.topology_type.value)
+ return f"{type(self.topology_type).__name__}.{name}"
+
+ def __repr__(self):
+ """Easy to read string of class and name of :attr:`topology_type`."""
+ return self.__str__()
+
+
class TestProtocol(Protocol):
"""Common test suite and test case attributes."""
@@ -251,6 +419,8 @@ class TestProtocol(Protocol):
skip: ClassVar[bool] = False
#: The reason for skipping the test case or suite.
skip_reason: ClassVar[str] = ""
+ #: The topology type of the test case or suite.
+ topology_type: ClassVar[TopologyCapability] = TopologyCapability(TopologyType.default)
#: The capabilities the test case or suite requires in order to be executed.
required_capabilities: ClassVar[set[Capability]] = set()
@@ -267,6 +437,7 @@ def get_test_cases(cls, test_case_sublist: Sequence[str] | None = None) -> tuple
def requires(
*nic_capabilities: NicCapability,
decorator_fn: TestPmdShellDecorator | None = None,
+ topology_type: TopologyType = TopologyType.default,
) -> Callable[[type[TestProtocol]], type[TestProtocol]]:
"""A decorator that adds the required capabilities to a test case or test suite.
@@ -287,6 +458,9 @@ def add_required_capability(test_case_or_suite: type[TestProtocol]) -> type[Test
)
decorated_nic_capability.add_to_required(test_case_or_suite)
+ topology_capability = TopologyCapability.get_unique(topology_type)
+ topology_capability.set_required(test_case_or_suite)
+
return test_case_or_suite
return add_required_capability
diff --git a/dts/framework/testbed_model/topology.py b/dts/framework/testbed_model/topology.py
index 19632ee890..712d252e44 100644
--- a/dts/framework/testbed_model/topology.py
+++ b/dts/framework/testbed_model/topology.py
@@ -8,15 +8,20 @@
"""
from dataclasses import dataclass
-from enum import IntEnum
-from typing import Iterable
+from typing import TYPE_CHECKING, Iterable
+
+if TYPE_CHECKING:
+ from enum import Enum as NoAliasEnum
+else:
+ from aenum import NoAliasEnum
from framework.config import PortConfig
+from framework.exception import ConfigurationError
from .port import Port
-class TopologyType(IntEnum):
+class TopologyType(int, NoAliasEnum):
"""Supported topology types."""
#: A topology with no Traffic Generator.
@@ -25,6 +30,28 @@ class TopologyType(IntEnum):
one_link = 1
#: A topology with two physical links between the Sut node and the TG node.
two_links = 2
+ #: The default topology required by test cases if not specified otherwise.
+ default = 2
+
+ @classmethod
+ def get_from_value(cls, value: int) -> "TopologyType":
+ """Get the corresponding instance from value.
+
+ :class:`Enum`s that don't allow aliases don't know which instance should be returned
+ as there could be multiple valid instances. Except for the :attr:`default` value,
+ :class:`TopologyType` is a regular Enum. When creating an instance from value, we're
+ not interested in the default, since we already know the value, allowing us to remove
+ the ambiguity.
+ """
+ match value:
+ case 0:
+ return TopologyType.no_link
+ case 1:
+ return TopologyType.one_link
+ case 2:
+ return TopologyType.two_links
+ case _:
+ raise ConfigurationError("More than two links in a topology are not supported.")
class Topology:
@@ -72,7 +99,7 @@ def __init__(self, sut_ports: Iterable[Port], tg_ports: Iterable[Port]):
):
port_links.append(PortLink(sut_port=sut_port, tg_port=tg_port))
- self.type = TopologyType(len(port_links))
+ self.type = TopologyType.get_from_value(len(port_links))
dummy_port = Port(PortConfig("", "", "", "", "", ""))
self.tg_port_egress = dummy_port
self.sut_port_ingress = dummy_port
diff --git a/dts/tests/TestSuite_hello_world.py b/dts/tests/TestSuite_hello_world.py
index 16d064ffeb..734f006026 100644
--- a/dts/tests/TestSuite_hello_world.py
+++ b/dts/tests/TestSuite_hello_world.py
@@ -9,6 +9,7 @@
from framework.remote_session.dpdk_shell import compute_eal_params
from framework.test_suite import TestSuite, func_test
+from framework.testbed_model.capability import TopologyType, requires
from framework.testbed_model.cpu import (
LogicalCoreCount,
LogicalCoreCountFilter,
@@ -16,6 +17,7 @@
)
+@requires(topology_type=TopologyType.no_link)
class TestHelloWorld(TestSuite):
"""DPDK hello world app test suite."""
diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
index 713549a5b2..89ece2ef56 100644
--- a/dts/tests/TestSuite_pmd_buffer_scatter.py
+++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
@@ -54,15 +54,9 @@ def set_up_suite(self) -> None:
"""Set up the test suite.
Setup:
- Verify that we have at least 2 port links in the current test run
- and increase the MTU of both ports on the traffic generator to 9000
+ Increase the MTU of both ports on the traffic generator to 9000
to support larger packet sizes.
"""
- self.verify(
- self._topology_type > 1,
- "There must be at least two port links to run the scatter test suite",
- )
-
self.tg_node.main_session.configure_port_mtu(9000, self._tg_port_egress)
self.tg_node.main_session.configure_port_mtu(9000, self._tg_port_ingress)
diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
index 94f90d9327..5f953a190f 100644
--- a/dts/tests/TestSuite_smoke_tests.py
+++ b/dts/tests/TestSuite_smoke_tests.py
@@ -18,9 +18,11 @@
from framework.remote_session.testpmd_shell import TestPmdShell
from framework.settings import SETTINGS
from framework.test_suite import TestSuite, func_test
+from framework.testbed_model.capability import TopologyType, requires
from framework.utils import REGEX_FOR_PCI_ADDRESS
+@requires(topology_type=TopologyType.no_link)
class TestSmokeTests(TestSuite):
"""DPDK and infrastructure smoke test suite.
--
2.34.1
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 09/12] dts: add topology capability
2024-08-21 14:53 ` [PATCH v3 09/12] dts: add topology capability Juraj Linkeš
@ 2024-08-26 17:13 ` Jeremy Spewock
2024-09-03 17:50 ` Dean Marx
1 sibling, 0 replies; 107+ messages in thread
From: Jeremy Spewock @ 2024-08-26 17:13 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
> Add support for marking test cases as requiring a certain topology. The
> default topology is a two link topology and the other supported
> topologies are one link and no link topologies.
>
> The TestProtocol of test suites and cases is extended with the topology
> type each test suite or case requires. Each test case starts out as
> requiring a two link topology and can be marked as requiring as
> topology directly (by decorating the test case) or through its test
> suite. If a test suite is decorated as requiring a certain topology, all
> its test cases are marked as such. If both test suite and a test case
> are decorated as requiring a topology, the test case cannot require a
> more complex topology than the whole suite (but it can require a less
> complex one). If a test suite is not decorated, this has no effect on
> required test case topology.
>
> Since the default topology is defined as a reference to one of the
> actual topologies, the NoAliasEnum from the aenum package is utilized,
> which removes the aliasing of Enums so that TopologyType.two_links and
> TopologyType.default are distinct. This is needed to distinguish between
> a user passed value and the default value being used (which is used when
> a test suite is or isn't decorated).
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
This patch looks good to me outside of some of the overlapping
comments from the DecoratedNicCapability class (mainly just
_get_unique).
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 09/12] dts: add topology capability
2024-08-21 14:53 ` [PATCH v3 09/12] dts: add topology capability Juraj Linkeš
2024-08-26 17:13 ` Jeremy Spewock
@ 2024-09-03 17:50 ` Dean Marx
1 sibling, 0 replies; 107+ messages in thread
From: Dean Marx @ 2024-09-03 17:50 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 1495 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> Add support for marking test cases as requiring a certain topology. The
> default topology is a two link topology and the other supported
> topologies are one link and no link topologies.
>
> The TestProtocol of test suites and cases is extended with the topology
> type each test suite or case requires. Each test case starts out as
> requiring a two link topology and can be marked as requiring as
> topology directly (by decorating the test case) or through its test
> suite. If a test suite is decorated as requiring a certain topology, all
> its test cases are marked as such. If both test suite and a test case
> are decorated as requiring a topology, the test case cannot require a
> more complex topology than the whole suite (but it can require a less
> complex one). If a test suite is not decorated, this has no effect on
> required test case topology.
>
> Since the default topology is defined as a reference to one of the
> actual topologies, the NoAliasEnum from the aenum package is utilized,
> which removes the aliasing of Enums so that TopologyType.two_links and
> TopologyType.default are distinct. This is needed to distinguish between
> a user passed value and the default value being used (which is used when
> a test suite is or isn't decorated).
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 1832 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH v3 10/12] doc: add DTS capability doc sources
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
` (8 preceding siblings ...)
2024-08-21 14:53 ` [PATCH v3 09/12] dts: add topology capability Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 17:13 ` Jeremy Spewock
2024-09-03 17:52 ` Dean Marx
2024-08-21 14:53 ` [PATCH v3 11/12] dts: add Rx offload capabilities Juraj Linkeš
` (2 subsequent siblings)
12 siblings, 2 replies; 107+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
Add new files to generate DTS API documentation from.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
doc/api/dts/framework.testbed_model.capability.rst | 6 ++++++
doc/api/dts/framework.testbed_model.rst | 2 ++
doc/api/dts/framework.testbed_model.topology.rst | 6 ++++++
3 files changed, 14 insertions(+)
create mode 100644 doc/api/dts/framework.testbed_model.capability.rst
create mode 100644 doc/api/dts/framework.testbed_model.topology.rst
diff --git a/doc/api/dts/framework.testbed_model.capability.rst b/doc/api/dts/framework.testbed_model.capability.rst
new file mode 100644
index 0000000000..326fed9104
--- /dev/null
+++ b/doc/api/dts/framework.testbed_model.capability.rst
@@ -0,0 +1,6 @@
+capability - Testbed Capabilities
+=================================
+
+.. automodule:: framework.testbed_model.capability
+ :members:
+ :show-inheritance:
diff --git a/doc/api/dts/framework.testbed_model.rst b/doc/api/dts/framework.testbed_model.rst
index 4b024e47e6..e1f9543b28 100644
--- a/doc/api/dts/framework.testbed_model.rst
+++ b/doc/api/dts/framework.testbed_model.rst
@@ -21,6 +21,8 @@ testbed\_model - Testbed Modelling Package
framework.testbed_model.node
framework.testbed_model.sut_node
framework.testbed_model.tg_node
+ framework.testbed_model.capability
framework.testbed_model.cpu
framework.testbed_model.port
+ framework.testbed_model.topology
framework.testbed_model.virtual_device
diff --git a/doc/api/dts/framework.testbed_model.topology.rst b/doc/api/dts/framework.testbed_model.topology.rst
new file mode 100644
index 0000000000..f3d9d1f418
--- /dev/null
+++ b/doc/api/dts/framework.testbed_model.topology.rst
@@ -0,0 +1,6 @@
+topology - Testbed Topology
+===========================
+
+.. automodule:: framework.testbed_model.topology
+ :members:
+ :show-inheritance:
--
2.34.1
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 10/12] doc: add DTS capability doc sources
2024-08-21 14:53 ` [PATCH v3 10/12] doc: add DTS capability doc sources Juraj Linkeš
@ 2024-08-26 17:13 ` Jeremy Spewock
2024-09-03 17:52 ` Dean Marx
1 sibling, 0 replies; 107+ messages in thread
From: Jeremy Spewock @ 2024-08-26 17:13 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
> Add new files to generate DTS API documentation from.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 10/12] doc: add DTS capability doc sources
2024-08-21 14:53 ` [PATCH v3 10/12] doc: add DTS capability doc sources Juraj Linkeš
2024-08-26 17:13 ` Jeremy Spewock
@ 2024-09-03 17:52 ` Dean Marx
1 sibling, 0 replies; 107+ messages in thread
From: Dean Marx @ 2024-09-03 17:52 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 259 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> Add new files to generate DTS API documentation from.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 550 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH v3 11/12] dts: add Rx offload capabilities
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
` (9 preceding siblings ...)
2024-08-21 14:53 ` [PATCH v3 10/12] doc: add DTS capability doc sources Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 17:24 ` Jeremy Spewock
` (2 more replies)
2024-08-21 14:53 ` [PATCH v3 12/12] dts: add NIC capabilities from show port info Juraj Linkeš
2024-08-26 17:25 ` [PATCH v3 00/12] dts: add test skipping based on capabilities Jeremy Spewock
12 siblings, 3 replies; 107+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
The scatter Rx offload capability is needed for the pmd_buffer_scatter
test suite. The command that retrieves the capability is:
show port <port_id> rx_offload capabilities
The command also retrieves a lot of other capabilities (RX_OFFLOAD_*)
which are all added into a Flag. The Flag members correspond to NIC
capability names so a convenience function that looks for the supported
Flags in a testpmd output is also added.
The NIC capability names (mentioned above) are copy-pasted from the
Flag. Dynamic addition of Enum members runs into problems with typing
(mypy doesn't know about the members) and documentation generation
(Sphinx doesn't know about the members).
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/remote_session/testpmd_shell.py | 213 ++++++++++++++++++
dts/tests/TestSuite_pmd_buffer_scatter.py | 1 +
2 files changed, 214 insertions(+)
diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index 48c31124d1..f83569669e 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -659,6 +659,103 @@ class TestPmdPortStats(TextParser):
tx_bps: int = field(metadata=TextParser.find_int(r"Tx-bps:\s+(\d+)"))
+class RxOffloadCapability(Flag):
+ """Rx offload capabilities of a device."""
+
+ #:
+ RX_OFFLOAD_VLAN_STRIP = auto()
+ #: Device supports L3 checksum offload.
+ RX_OFFLOAD_IPV4_CKSUM = auto()
+ #: Device supports L4 checksum offload.
+ RX_OFFLOAD_UDP_CKSUM = auto()
+ #: Device supports L4 checksum offload.
+ RX_OFFLOAD_TCP_CKSUM = auto()
+ #: Device supports Large Receive Offload.
+ RX_OFFLOAD_TCP_LRO = auto()
+ #: Device supports QinQ (queue in queue) offload.
+ RX_OFFLOAD_QINQ_STRIP = auto()
+ #: Device supports inner packet L3 checksum.
+ RX_OFFLOAD_OUTER_IPV4_CKSUM = auto()
+ #: Device supports MACsec.
+ RX_OFFLOAD_MACSEC_STRIP = auto()
+ #: Device supports filtering of a VLAN Tag identifier.
+ RX_OFFLOAD_VLAN_FILTER = 1 << 9
+ #: Device supports VLAN offload.
+ RX_OFFLOAD_VLAN_EXTEND = auto()
+ #: Device supports receiving segmented mbufs.
+ RX_OFFLOAD_SCATTER = 1 << 13
+ #: Device supports Timestamp.
+ RX_OFFLOAD_TIMESTAMP = auto()
+ #: Device supports crypto processing while packet is received in NIC.
+ RX_OFFLOAD_SECURITY = auto()
+ #: Device supports CRC stripping.
+ RX_OFFLOAD_KEEP_CRC = auto()
+ #: Device supports L4 checksum offload.
+ RX_OFFLOAD_SCTP_CKSUM = auto()
+ #: Device supports inner packet L4 checksum.
+ RX_OFFLOAD_OUTER_UDP_CKSUM = auto()
+ #: Device supports RSS hashing.
+ RX_OFFLOAD_RSS_HASH = auto()
+ #: Device supports
+ RX_OFFLOAD_BUFFER_SPLIT = auto()
+ #: Device supports all checksum capabilities.
+ RX_OFFLOAD_CHECKSUM = RX_OFFLOAD_IPV4_CKSUM | RX_OFFLOAD_UDP_CKSUM | RX_OFFLOAD_TCP_CKSUM
+ #: Device supports all VLAN capabilities.
+ RX_OFFLOAD_VLAN = (
+ RX_OFFLOAD_VLAN_STRIP
+ | RX_OFFLOAD_VLAN_FILTER
+ | RX_OFFLOAD_VLAN_EXTEND
+ | RX_OFFLOAD_QINQ_STRIP
+ )
+
+ @classmethod
+ def from_string(cls, line: str) -> Self:
+ """Make an instance from a string containing the flag names separated with a space.
+
+ Args:
+ line: The line to parse.
+
+ Returns:
+ A new instance containing all found flags.
+ """
+ flag = cls(0)
+ for flag_name in line.split():
+ flag |= cls[f"RX_OFFLOAD_{flag_name}"]
+ return flag
+
+ @classmethod
+ def make_parser(cls, per_port: bool) -> ParserFn:
+ """Make a parser function.
+
+ Args:
+ per_port: If :data:`True`, will return capabilities per port. If :data:`False`,
+ will return capabilities per queue.
+
+ Returns:
+ ParserFn: A dictionary for the `dataclasses.field` metadata argument containing a
+ parser function that makes an instance of this flag from text.
+ """
+ granularity = "Port" if per_port else "Queue"
+ return TextParser.wrap(
+ TextParser.find(rf"Per {granularity}\s+:(.*)$", re.MULTILINE),
+ cls.from_string,
+ )
+
+
+@dataclass
+class RxOffloadCapabilities(TextParser):
+ """The result of testpmd's ``show port <port_id> rx_offload capabilities`` command."""
+
+ #:
+ port_id: int = field(
+ metadata=TextParser.find_int(r"Rx Offloading Capabilities of port (\d+) :")
+ )
+ #: Per-queue Rx offload capabilities.
+ per_queue: RxOffloadCapability = field(metadata=RxOffloadCapability.make_parser(False))
+ #: Capabilities other than per-queue Rx offload capabilities.
+ per_port: RxOffloadCapability = field(metadata=RxOffloadCapability.make_parser(True))
+
+
T = TypeVarTuple("T") # type: ignore[misc]
@@ -1048,6 +1145,42 @@ def _close(self) -> None:
====== Capability retrieval methods ======
"""
+ def get_capabilities_rx_offload(
+ self,
+ supported_capabilities: MutableSet["NicCapability"],
+ unsupported_capabilities: MutableSet["NicCapability"],
+ ) -> None:
+ """Get all rx offload capabilities and divide them into supported and unsupported.
+
+ Args:
+ supported_capabilities: Supported capabilities will be added to this set.
+ unsupported_capabilities: Unsupported capabilities will be added to this set.
+ """
+ self._logger.debug("Getting rx offload capabilities.")
+ command = f"show port {self.ports[0].id} rx_offload capabilities"
+ rx_offload_capabilities_out = self.send_command(command)
+ rx_offload_capabilities = RxOffloadCapabilities.parse(rx_offload_capabilities_out)
+ self._update_capabilities_from_flag(
+ supported_capabilities,
+ unsupported_capabilities,
+ RxOffloadCapability,
+ rx_offload_capabilities.per_port | rx_offload_capabilities.per_queue,
+ )
+
+ def _update_capabilities_from_flag(
+ self,
+ supported_capabilities: MutableSet["NicCapability"],
+ unsupported_capabilities: MutableSet["NicCapability"],
+ flag_class: type[Flag],
+ supported_flags: Flag,
+ ) -> None:
+ """Divide all flags from `flag_class` into supported and unsupported."""
+ for flag in flag_class:
+ if flag in supported_flags:
+ supported_capabilities.add(NicCapability[str(flag.name)])
+ else:
+ unsupported_capabilities.add(NicCapability[str(flag.name)])
+
def get_capabilities_rxq_info(
self,
supported_capabilities: MutableSet["NicCapability"],
@@ -1119,6 +1252,86 @@ class NicCapability(NoAliasEnum):
SCATTERED_RX_ENABLED: TestPmdShellCapabilityMethod = partial(
TestPmdShell.get_capabilities_rxq_info
)
+ #:
+ RX_OFFLOAD_VLAN_STRIP: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports L3 checksum offload.
+ RX_OFFLOAD_IPV4_CKSUM: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports L4 checksum offload.
+ RX_OFFLOAD_UDP_CKSUM: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports L4 checksum offload.
+ RX_OFFLOAD_TCP_CKSUM: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports Large Receive Offload.
+ RX_OFFLOAD_TCP_LRO: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports QinQ (queue in queue) offload.
+ RX_OFFLOAD_QINQ_STRIP: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports inner packet L3 checksum.
+ RX_OFFLOAD_OUTER_IPV4_CKSUM: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports MACsec.
+ RX_OFFLOAD_MACSEC_STRIP: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports filtering of a VLAN Tag identifier.
+ RX_OFFLOAD_VLAN_FILTER: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports VLAN offload.
+ RX_OFFLOAD_VLAN_EXTEND: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports receiving segmented mbufs.
+ RX_OFFLOAD_SCATTER: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports Timestamp.
+ RX_OFFLOAD_TIMESTAMP: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports crypto processing while packet is received in NIC.
+ RX_OFFLOAD_SECURITY: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports CRC stripping.
+ RX_OFFLOAD_KEEP_CRC: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports L4 checksum offload.
+ RX_OFFLOAD_SCTP_CKSUM: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports inner packet L4 checksum.
+ RX_OFFLOAD_OUTER_UDP_CKSUM: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports RSS hashing.
+ RX_OFFLOAD_RSS_HASH: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports scatter Rx packets to segmented mbufs.
+ RX_OFFLOAD_BUFFER_SPLIT: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports all checksum capabilities.
+ RX_OFFLOAD_CHECKSUM: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
+ #: Device supports all VLAN capabilities.
+ RX_OFFLOAD_VLAN: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_rx_offload
+ )
def __call__(
self,
diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
index 89ece2ef56..64c48b0793 100644
--- a/dts/tests/TestSuite_pmd_buffer_scatter.py
+++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
@@ -28,6 +28,7 @@
from framework.testbed_model.capability import NicCapability, requires
+@requires(NicCapability.RX_OFFLOAD_SCATTER)
class TestPmdBufferScatter(TestSuite):
"""DPDK PMD packet scattering test suite.
--
2.34.1
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 11/12] dts: add Rx offload capabilities
2024-08-21 14:53 ` [PATCH v3 11/12] dts: add Rx offload capabilities Juraj Linkeš
@ 2024-08-26 17:24 ` Jeremy Spewock
2024-09-18 14:18 ` Juraj Linkeš
2024-08-28 17:44 ` Jeremy Spewock
2024-09-03 19:49 ` Dean Marx
2 siblings, 1 reply; 107+ messages in thread
From: Jeremy Spewock @ 2024-08-26 17:24 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
<snip>
> diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
> index 48c31124d1..f83569669e 100644
> --- a/dts/framework/remote_session/testpmd_shell.py
> +++ b/dts/framework/remote_session/testpmd_shell.py
> @@ -659,6 +659,103 @@ class TestPmdPortStats(TextParser):
> tx_bps: int = field(metadata=TextParser.find_int(r"Tx-bps:\s+(\d+)"))
>
>
> +class RxOffloadCapability(Flag):
> + """Rx offload capabilities of a device."""
> +
> + #:
> + RX_OFFLOAD_VLAN_STRIP = auto()
> + #: Device supports L3 checksum offload.
> + RX_OFFLOAD_IPV4_CKSUM = auto()
> + #: Device supports L4 checksum offload.
> + RX_OFFLOAD_UDP_CKSUM = auto()
> + #: Device supports L4 checksum offload.
> + RX_OFFLOAD_TCP_CKSUM = auto()
> + #: Device supports Large Receive Offload.
> + RX_OFFLOAD_TCP_LRO = auto()
> + #: Device supports QinQ (queue in queue) offload.
> + RX_OFFLOAD_QINQ_STRIP = auto()
> + #: Device supports inner packet L3 checksum.
> + RX_OFFLOAD_OUTER_IPV4_CKSUM = auto()
> + #: Device supports MACsec.
> + RX_OFFLOAD_MACSEC_STRIP = auto()
> + #: Device supports filtering of a VLAN Tag identifier.
> + RX_OFFLOAD_VLAN_FILTER = 1 << 9
> + #: Device supports VLAN offload.
> + RX_OFFLOAD_VLAN_EXTEND = auto()
> + #: Device supports receiving segmented mbufs.
> + RX_OFFLOAD_SCATTER = 1 << 13
I know you mentioned in the commit message that the auto() can cause
problems with mypy/sphinx, is that why this one is a specific value
instead? Regardless, I think we should probably make it consistent so
that either all of them are bit-shifts or none of them are unless
there is a specific reason that the scatter offload is different.
> + #: Device supports Timestamp.
> + RX_OFFLOAD_TIMESTAMP = auto()
> + #: Device supports crypto processing while packet is received in NIC.
> + RX_OFFLOAD_SECURITY = auto()
> + #: Device supports CRC stripping.
> + RX_OFFLOAD_KEEP_CRC = auto()
> + #: Device supports L4 checksum offload.
> + RX_OFFLOAD_SCTP_CKSUM = auto()
> + #: Device supports inner packet L4 checksum.
> + RX_OFFLOAD_OUTER_UDP_CKSUM = auto()
> + #: Device supports RSS hashing.
> + RX_OFFLOAD_RSS_HASH = auto()
> + #: Device supports
> + RX_OFFLOAD_BUFFER_SPLIT = auto()
> + #: Device supports all checksum capabilities.
> + RX_OFFLOAD_CHECKSUM = RX_OFFLOAD_IPV4_CKSUM | RX_OFFLOAD_UDP_CKSUM | RX_OFFLOAD_TCP_CKSUM
> + #: Device supports all VLAN capabilities.
> + RX_OFFLOAD_VLAN = (
> + RX_OFFLOAD_VLAN_STRIP
> + | RX_OFFLOAD_VLAN_FILTER
> + | RX_OFFLOAD_VLAN_EXTEND
> + | RX_OFFLOAD_QINQ_STRIP
> + )
<snip>
>
> @@ -1048,6 +1145,42 @@ def _close(self) -> None:
> ====== Capability retrieval methods ======
> """
>
> + def get_capabilities_rx_offload(
> + self,
> + supported_capabilities: MutableSet["NicCapability"],
> + unsupported_capabilities: MutableSet["NicCapability"],
> + ) -> None:
> + """Get all rx offload capabilities and divide them into supported and unsupported.
> +
> + Args:
> + supported_capabilities: Supported capabilities will be added to this set.
> + unsupported_capabilities: Unsupported capabilities will be added to this set.
> + """
> + self._logger.debug("Getting rx offload capabilities.")
> + command = f"show port {self.ports[0].id} rx_offload capabilities"
Is it desirable to only get the capabilities of the first port? In the
current framework I suppose it doesn't matter all that much since you
can only use the first few ports in the list of ports anyway, but will
there ever be a case where a test run has 2 different devices included
in the list of ports? Of course it's possible that it will happen, but
is it practical? Because, if so, then we would want this to aggregate
what all the devices are capable of and have capabilities basically
say "at least one of the ports in the list of ports is capable of
these things."
This consideration also applies to the rxq info capability gathering as well.
> + rx_offload_capabilities_out = self.send_command(command)
> + rx_offload_capabilities = RxOffloadCapabilities.parse(rx_offload_capabilities_out)
> + self._update_capabilities_from_flag(
> + supported_capabilities,
> + unsupported_capabilities,
> + RxOffloadCapability,
> + rx_offload_capabilities.per_port | rx_offload_capabilities.per_queue,
> + )
> +
<snip>
>
> def __call__(
> self,
> diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
> index 89ece2ef56..64c48b0793 100644
> --- a/dts/tests/TestSuite_pmd_buffer_scatter.py
> +++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
> @@ -28,6 +28,7 @@
> from framework.testbed_model.capability import NicCapability, requires
>
>
> +@requires(NicCapability.RX_OFFLOAD_SCATTER)
I know that we talked about this and how, in the environments we
looked at, it was true that the offload was supported in all cases
where the "native" or non-offloaded was supported, but thinking about
this more, I wonder if it is worth generalizing this assumption to all
NICs or if we can just decorate the second test case that I wrote
which uses the offloaded support. As long as the capabilities exposed
by testpmd are accurate, even if this assumption was true, the
capability for the non-offloaded one would show False when this
offload wasn't usable and it would skip the test case anyway, so I
don't think we lose anything by not including this test-suite-level
requirement and making it more narrow to the test cases that require
it.
Let me know your thoughts on that though and I would be interested to
hear if anyone else has any.
> class TestPmdBufferScatter(TestSuite):
> """DPDK PMD packet scattering test suite.
>
> --
> 2.34.1
>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 11/12] dts: add Rx offload capabilities
2024-08-26 17:24 ` Jeremy Spewock
@ 2024-09-18 14:18 ` Juraj Linkeš
2024-09-18 16:53 ` Jeremy Spewock
0 siblings, 1 reply; 107+ messages in thread
From: Juraj Linkeš @ 2024-09-18 14:18 UTC (permalink / raw)
To: Jeremy Spewock
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On 26. 8. 2024 19:24, Jeremy Spewock wrote:
> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> <snip>
>> diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
>> index 48c31124d1..f83569669e 100644
>> --- a/dts/framework/remote_session/testpmd_shell.py
>> +++ b/dts/framework/remote_session/testpmd_shell.py
>> @@ -659,6 +659,103 @@ class TestPmdPortStats(TextParser):
>> tx_bps: int = field(metadata=TextParser.find_int(r"Tx-bps:\s+(\d+)"))
>>
>>
>> +class RxOffloadCapability(Flag):
>> + """Rx offload capabilities of a device."""
>> +
>> + #:
>> + RX_OFFLOAD_VLAN_STRIP = auto()
>> + #: Device supports L3 checksum offload.
>> + RX_OFFLOAD_IPV4_CKSUM = auto()
>> + #: Device supports L4 checksum offload.
>> + RX_OFFLOAD_UDP_CKSUM = auto()
>> + #: Device supports L4 checksum offload.
>> + RX_OFFLOAD_TCP_CKSUM = auto()
>> + #: Device supports Large Receive Offload.
>> + RX_OFFLOAD_TCP_LRO = auto()
>> + #: Device supports QinQ (queue in queue) offload.
>> + RX_OFFLOAD_QINQ_STRIP = auto()
>> + #: Device supports inner packet L3 checksum.
>> + RX_OFFLOAD_OUTER_IPV4_CKSUM = auto()
>> + #: Device supports MACsec.
>> + RX_OFFLOAD_MACSEC_STRIP = auto()
>> + #: Device supports filtering of a VLAN Tag identifier.
>> + RX_OFFLOAD_VLAN_FILTER = 1 << 9
>> + #: Device supports VLAN offload.
>> + RX_OFFLOAD_VLAN_EXTEND = auto()
>> + #: Device supports receiving segmented mbufs.
>> + RX_OFFLOAD_SCATTER = 1 << 13
>
> I know you mentioned in the commit message that the auto() can cause
> problems with mypy/sphinx, is that why this one is a specific value
> instead? Regardless, I think we should probably make it consistent so
> that either all of them are bit-shifts or none of them are unless
> there is a specific reason that the scatter offload is different.
>
Since both you and Dean asked, I'll add something to the docstring about
this.
There are actually two non-auto values (RX_OFFLOAD_VLAN_FILTER = 1 << 9
is the first one). I used the actual values to mirror the flags in DPDK
code.
>> + #: Device supports Timestamp.
>> + RX_OFFLOAD_TIMESTAMP = auto()
>> + #: Device supports crypto processing while packet is received in NIC.
>> + RX_OFFLOAD_SECURITY = auto()
>> + #: Device supports CRC stripping.
>> + RX_OFFLOAD_KEEP_CRC = auto()
>> + #: Device supports L4 checksum offload.
>> + RX_OFFLOAD_SCTP_CKSUM = auto()
>> + #: Device supports inner packet L4 checksum.
>> + RX_OFFLOAD_OUTER_UDP_CKSUM = auto()
>> + #: Device supports RSS hashing.
>> + RX_OFFLOAD_RSS_HASH = auto()
>> + #: Device supports
>> + RX_OFFLOAD_BUFFER_SPLIT = auto()
>> + #: Device supports all checksum capabilities.
>> + RX_OFFLOAD_CHECKSUM = RX_OFFLOAD_IPV4_CKSUM | RX_OFFLOAD_UDP_CKSUM | RX_OFFLOAD_TCP_CKSUM
>> + #: Device supports all VLAN capabilities.
>> + RX_OFFLOAD_VLAN = (
>> + RX_OFFLOAD_VLAN_STRIP
>> + | RX_OFFLOAD_VLAN_FILTER
>> + | RX_OFFLOAD_VLAN_EXTEND
>> + | RX_OFFLOAD_QINQ_STRIP
>> + )
> <snip>
>>
>> @@ -1048,6 +1145,42 @@ def _close(self) -> None:
>> ====== Capability retrieval methods ======
>> """
>>
>> + def get_capabilities_rx_offload(
>> + self,
>> + supported_capabilities: MutableSet["NicCapability"],
>> + unsupported_capabilities: MutableSet["NicCapability"],
>> + ) -> None:
>> + """Get all rx offload capabilities and divide them into supported and unsupported.
>> +
>> + Args:
>> + supported_capabilities: Supported capabilities will be added to this set.
>> + unsupported_capabilities: Unsupported capabilities will be added to this set.
>> + """
>> + self._logger.debug("Getting rx offload capabilities.")
>> + command = f"show port {self.ports[0].id} rx_offload capabilities"
>
> Is it desirable to only get the capabilities of the first port? In the
> current framework I suppose it doesn't matter all that much since you
> can only use the first few ports in the list of ports anyway, but will
> there ever be a case where a test run has 2 different devices included
> in the list of ports? Of course it's possible that it will happen, but
> is it practical? Because, if so, then we would want this to aggregate
> what all the devices are capable of and have capabilities basically
> say "at least one of the ports in the list of ports is capable of
> these things."
>
> This consideration also applies to the rxq info capability gathering as well.
>
No parts of the framework are adjusted to use multiple NIC in a single
test run (because we assume we're testing only one NIC at a time). If we
add this support, it's going to be a broader change.
I approached this with the above assumption in mind and in that case,
testing just one port of the NIC seemed just fine.
>> + rx_offload_capabilities_out = self.send_command(command)
>> + rx_offload_capabilities = RxOffloadCapabilities.parse(rx_offload_capabilities_out)
>> + self._update_capabilities_from_flag(
>> + supported_capabilities,
>> + unsupported_capabilities,
>> + RxOffloadCapability,
>> + rx_offload_capabilities.per_port | rx_offload_capabilities.per_queue,
>> + )
>> +
> <snip>
>>
>> def __call__(
>> self,
>> diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
>> index 89ece2ef56..64c48b0793 100644
>> --- a/dts/tests/TestSuite_pmd_buffer_scatter.py
>> +++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
>> @@ -28,6 +28,7 @@
>> from framework.testbed_model.capability import NicCapability, requires
>>
>>
>> +@requires(NicCapability.RX_OFFLOAD_SCATTER)
>
> I know that we talked about this and how, in the environments we
> looked at, it was true that the offload was supported in all cases
> where the "native" or non-offloaded was supported, but thinking about
> this more, I wonder if it is worth generalizing this assumption to all
> NICs or if we can just decorate the second test case that I wrote
> which uses the offloaded support. As long as the capabilities exposed
> by testpmd are accurate, even if this assumption was true, the
> capability for the non-offloaded one would show False when this
> offload wasn't usable and it would skip the test case anyway, so I
> don't think we lose anything by not including this test-suite-level
> requirement and making it more narrow to the test cases that require
> it.
>
> Let me know your thoughts on that though and I would be interested to
> hear if anyone else has any.
>
I'm not sure I understand what your point is. Let's talk about it in the
call.
>> class TestPmdBufferScatter(TestSuite):
>> """DPDK PMD packet scattering test suite.
>>
>> --
>> 2.34.1
>>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 11/12] dts: add Rx offload capabilities
2024-09-18 14:18 ` Juraj Linkeš
@ 2024-09-18 16:53 ` Jeremy Spewock
0 siblings, 0 replies; 107+ messages in thread
From: Jeremy Spewock @ 2024-09-18 16:53 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Sep 18, 2024 at 10:18 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
>
>
> On 26. 8. 2024 19:24, Jeremy Spewock wrote:
> > On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> > <juraj.linkes@pantheon.tech> wrote:
> > <snip>
> >> diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
> >> index 48c31124d1..f83569669e 100644
> >> --- a/dts/framework/remote_session/testpmd_shell.py
> >> +++ b/dts/framework/remote_session/testpmd_shell.py
> >> @@ -659,6 +659,103 @@ class TestPmdPortStats(TextParser):
> >> tx_bps: int = field(metadata=TextParser.find_int(r"Tx-bps:\s+(\d+)"))
> >>
> >>
> >> +class RxOffloadCapability(Flag):
> >> + """Rx offload capabilities of a device."""
> >> +
> >> + #:
> >> + RX_OFFLOAD_VLAN_STRIP = auto()
> >> + #: Device supports L3 checksum offload.
> >> + RX_OFFLOAD_IPV4_CKSUM = auto()
> >> + #: Device supports L4 checksum offload.
> >> + RX_OFFLOAD_UDP_CKSUM = auto()
> >> + #: Device supports L4 checksum offload.
> >> + RX_OFFLOAD_TCP_CKSUM = auto()
> >> + #: Device supports Large Receive Offload.
> >> + RX_OFFLOAD_TCP_LRO = auto()
> >> + #: Device supports QinQ (queue in queue) offload.
> >> + RX_OFFLOAD_QINQ_STRIP = auto()
> >> + #: Device supports inner packet L3 checksum.
> >> + RX_OFFLOAD_OUTER_IPV4_CKSUM = auto()
> >> + #: Device supports MACsec.
> >> + RX_OFFLOAD_MACSEC_STRIP = auto()
> >> + #: Device supports filtering of a VLAN Tag identifier.
> >> + RX_OFFLOAD_VLAN_FILTER = 1 << 9
> >> + #: Device supports VLAN offload.
> >> + RX_OFFLOAD_VLAN_EXTEND = auto()
> >> + #: Device supports receiving segmented mbufs.
> >> + RX_OFFLOAD_SCATTER = 1 << 13
> >
> > I know you mentioned in the commit message that the auto() can cause
> > problems with mypy/sphinx, is that why this one is a specific value
> > instead? Regardless, I think we should probably make it consistent so
> > that either all of them are bit-shifts or none of them are unless
> > there is a specific reason that the scatter offload is different.
> >
>
> Since both you and Dean asked, I'll add something to the docstring about
> this.
>
> There are actually two non-auto values (RX_OFFLOAD_VLAN_FILTER = 1 << 9
> is the first one). I used the actual values to mirror the flags in DPDK
> code.
Gotcha, that makes sense.
>
> >> + #: Device supports Timestamp.
> >> + RX_OFFLOAD_TIMESTAMP = auto()
> >> + #: Device supports crypto processing while packet is received in NIC.
> >> + RX_OFFLOAD_SECURITY = auto()
> >> + #: Device supports CRC stripping.
> >> + RX_OFFLOAD_KEEP_CRC = auto()
> >> + #: Device supports L4 checksum offload.
> >> + RX_OFFLOAD_SCTP_CKSUM = auto()
> >> + #: Device supports inner packet L4 checksum.
> >> + RX_OFFLOAD_OUTER_UDP_CKSUM = auto()
> >> + #: Device supports RSS hashing.
> >> + RX_OFFLOAD_RSS_HASH = auto()
> >> + #: Device supports
> >> + RX_OFFLOAD_BUFFER_SPLIT = auto()
> >> + #: Device supports all checksum capabilities.
> >> + RX_OFFLOAD_CHECKSUM = RX_OFFLOAD_IPV4_CKSUM | RX_OFFLOAD_UDP_CKSUM | RX_OFFLOAD_TCP_CKSUM
> >> + #: Device supports all VLAN capabilities.
> >> + RX_OFFLOAD_VLAN = (
> >> + RX_OFFLOAD_VLAN_STRIP
> >> + | RX_OFFLOAD_VLAN_FILTER
> >> + | RX_OFFLOAD_VLAN_EXTEND
> >> + | RX_OFFLOAD_QINQ_STRIP
> >> + )
> > <snip>
> >>
> >> @@ -1048,6 +1145,42 @@ def _close(self) -> None:
> >> ====== Capability retrieval methods ======
> >> """
> >>
> >> + def get_capabilities_rx_offload(
> >> + self,
> >> + supported_capabilities: MutableSet["NicCapability"],
> >> + unsupported_capabilities: MutableSet["NicCapability"],
> >> + ) -> None:
> >> + """Get all rx offload capabilities and divide them into supported and unsupported.
> >> +
> >> + Args:
> >> + supported_capabilities: Supported capabilities will be added to this set.
> >> + unsupported_capabilities: Unsupported capabilities will be added to this set.
> >> + """
> >> + self._logger.debug("Getting rx offload capabilities.")
> >> + command = f"show port {self.ports[0].id} rx_offload capabilities"
> >
> > Is it desirable to only get the capabilities of the first port? In the
> > current framework I suppose it doesn't matter all that much since you
> > can only use the first few ports in the list of ports anyway, but will
> > there ever be a case where a test run has 2 different devices included
> > in the list of ports? Of course it's possible that it will happen, but
> > is it practical? Because, if so, then we would want this to aggregate
> > what all the devices are capable of and have capabilities basically
> > say "at least one of the ports in the list of ports is capable of
> > these things."
> >
> > This consideration also applies to the rxq info capability gathering as well.
> >
>
> No parts of the framework are adjusted to use multiple NIC in a single
> test run (because we assume we're testing only one NIC at a time). If we
> add this support, it's going to be a broader change.
>
> I approached this with the above assumption in mind and in that case,
> testing just one port of the NIC seemed just fine.
That's a good point that making the adjustment to allow for multiple
devices is a bigger change that is definitely out of scope for this
series. Makes sense to put it off and go with the current assumptions,
I only asked in case it was something simple so it would be one less
thing to do in the future :). This is fine as is then I think.
>
> >> + rx_offload_capabilities_out = self.send_command(command)
> >> + rx_offload_capabilities = RxOffloadCapabilities.parse(rx_offload_capabilities_out)
> >> + self._update_capabilities_from_flag(
> >> + supported_capabilities,
> >> + unsupported_capabilities,
> >> + RxOffloadCapability,
> >> + rx_offload_capabilities.per_port | rx_offload_capabilities.per_queue,
> >> + )
> >> +
> > <snip>
> >>
> >> def __call__(
> >> self,
> >> diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
> >> index 89ece2ef56..64c48b0793 100644
> >> --- a/dts/tests/TestSuite_pmd_buffer_scatter.py
> >> +++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
> >> @@ -28,6 +28,7 @@
> >> from framework.testbed_model.capability import NicCapability, requires
> >>
> >>
> >> +@requires(NicCapability.RX_OFFLOAD_SCATTER)
> >
> > I know that we talked about this and how, in the environments we
> > looked at, it was true that the offload was supported in all cases
> > where the "native" or non-offloaded was supported, but thinking about
> > this more, I wonder if it is worth generalizing this assumption to all
> > NICs or if we can just decorate the second test case that I wrote
> > which uses the offloaded support. As long as the capabilities exposed
> > by testpmd are accurate, even if this assumption was true, the
> > capability for the non-offloaded one would show False when this
> > offload wasn't usable and it would skip the test case anyway, so I
> > don't think we lose anything by not including this test-suite-level
> > requirement and making it more narrow to the test cases that require
> > it.
> >
> > Let me know your thoughts on that though and I would be interested to
> > hear if anyone else has any.
> >
>
> I'm not sure I understand what your point is. Let's talk about it in the
> call.
Sure, sounds good to me.
>
> >> class TestPmdBufferScatter(TestSuite):
> >> """DPDK PMD packet scattering test suite.
> >>
> >> --
> >> 2.34.1
> >>
>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 11/12] dts: add Rx offload capabilities
2024-08-21 14:53 ` [PATCH v3 11/12] dts: add Rx offload capabilities Juraj Linkeš
2024-08-26 17:24 ` Jeremy Spewock
@ 2024-08-28 17:44 ` Jeremy Spewock
2024-08-29 15:40 ` Jeremy Spewock
2024-09-03 19:49 ` Dean Marx
2 siblings, 1 reply; 107+ messages in thread
From: Jeremy Spewock @ 2024-08-28 17:44 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
<snip>
> diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
> index 48c31124d1..f83569669e 100644
> --- a/dts/framework/remote_session/testpmd_shell.py
> +++ b/dts/framework/remote_session/testpmd_shell.py
> @@ -659,6 +659,103 @@ class TestPmdPortStats(TextParser):
> tx_bps: int = field(metadata=TextParser.find_int(r"Tx-bps:\s+(\d+)"))
>
>
> +class RxOffloadCapability(Flag):
> + """Rx offload capabilities of a device."""
> +
> + #:
> + RX_OFFLOAD_VLAN_STRIP = auto()
One other thought that I had about this; was there a specific reason
that you decided to prefix all of these with `RX_OFFLOAD_`? I am
working on a test suite right now that uses both RX and TX offloads
and thought that it would be a great use of capabilities, so I am
working on adding a TxOffloadCapability flag as well and, since the
output is essentially the same, it made a lot of sense to make it a
sibling class of this one with similar parsing functionality. In what
I was writing, I found it much easier to remove this prefix so that
the parsing method can be the same for both RX and TX, and I didn't
have to restate some options that are shared between both (like
IPv4_CKSUM, UDP_CKSUM, etc.). Is there a reason you can think of why
removing this prefix is a bad idea? Hopefully I will have a patch out
soon that shows this extension that I've made so that you can see
in-code what I was thinking.
> + #: Device supports L3 checksum offload.
> + RX_OFFLOAD_IPV4_CKSUM = auto()
> + #: Device supports L4 checksum offload.
> + RX_OFFLOAD_UDP_CKSUM = auto()
> + #: Device supports L4 checksum offload.
> + RX_OFFLOAD_TCP_CKSUM = auto()
> + #: Device supports Large Receive Offload.
> + RX_OFFLOAD_TCP_LRO = auto()
> + #: Device supports QinQ (queue in queue) offload.
> + RX_OFFLOAD_QINQ_STRIP = auto()
> + #: Device supports inner packet L3 checksum.
> + RX_OFFLOAD_OUTER_IPV4_CKSUM = auto()
> + #: Device supports MACsec.
> + RX_OFFLOAD_MACSEC_STRIP = auto()
> + #: Device supports filtering of a VLAN Tag identifier.
> + RX_OFFLOAD_VLAN_FILTER = 1 << 9
> + #: Device supports VLAN offload.
> + RX_OFFLOAD_VLAN_EXTEND = auto()
> + #: Device supports receiving segmented mbufs.
> + RX_OFFLOAD_SCATTER = 1 << 13
> + #: Device supports Timestamp.
> + RX_OFFLOAD_TIMESTAMP = auto()
> + #: Device supports crypto processing while packet is received in NIC.
> + RX_OFFLOAD_SECURITY = auto()
> + #: Device supports CRC stripping.
> + RX_OFFLOAD_KEEP_CRC = auto()
> + #: Device supports L4 checksum offload.
> + RX_OFFLOAD_SCTP_CKSUM = auto()
> + #: Device supports inner packet L4 checksum.
> + RX_OFFLOAD_OUTER_UDP_CKSUM = auto()
> + #: Device supports RSS hashing.
> + RX_OFFLOAD_RSS_HASH = auto()
> + #: Device supports
> + RX_OFFLOAD_BUFFER_SPLIT = auto()
> + #: Device supports all checksum capabilities.
> + RX_OFFLOAD_CHECKSUM = RX_OFFLOAD_IPV4_CKSUM | RX_OFFLOAD_UDP_CKSUM | RX_OFFLOAD_TCP_CKSUM
> + #: Device supports all VLAN capabilities.
> + RX_OFFLOAD_VLAN = (
> + RX_OFFLOAD_VLAN_STRIP
> + | RX_OFFLOAD_VLAN_FILTER
> + | RX_OFFLOAD_VLAN_EXTEND
> + | RX_OFFLOAD_QINQ_STRIP
> + )
<snip>
>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 11/12] dts: add Rx offload capabilities
2024-08-28 17:44 ` Jeremy Spewock
@ 2024-08-29 15:40 ` Jeremy Spewock
2024-09-18 14:27 ` Juraj Linkeš
0 siblings, 1 reply; 107+ messages in thread
From: Jeremy Spewock @ 2024-08-29 15:40 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 28, 2024 at 1:44 PM Jeremy Spewock <jspewock@iol.unh.edu> wrote:
>
> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> <snip>
> > diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
> > index 48c31124d1..f83569669e 100644
> > --- a/dts/framework/remote_session/testpmd_shell.py
> > +++ b/dts/framework/remote_session/testpmd_shell.py
> > @@ -659,6 +659,103 @@ class TestPmdPortStats(TextParser):
> > tx_bps: int = field(metadata=TextParser.find_int(r"Tx-bps:\s+(\d+)"))
> >
> >
> > +class RxOffloadCapability(Flag):
> > + """Rx offload capabilities of a device."""
> > +
> > + #:
> > + RX_OFFLOAD_VLAN_STRIP = auto()
>
> One other thought that I had about this; was there a specific reason
> that you decided to prefix all of these with `RX_OFFLOAD_`? I am
> working on a test suite right now that uses both RX and TX offloads
> and thought that it would be a great use of capabilities, so I am
> working on adding a TxOffloadCapability flag as well and, since the
> output is essentially the same, it made a lot of sense to make it a
> sibling class of this one with similar parsing functionality. In what
> I was writing, I found it much easier to remove this prefix so that
> the parsing method can be the same for both RX and TX, and I didn't
> have to restate some options that are shared between both (like
> IPv4_CKSUM, UDP_CKSUM, etc.). Is there a reason you can think of why
> removing this prefix is a bad idea? Hopefully I will have a patch out
> soon that shows this extension that I've made so that you can see
> in-code what I was thinking.
I see now that you actually already answered this question, I was just
looking too much at that piece of code, and clearly not looking
further down at the helper-method mapping or the commit message that
you left :).
"The Flag members correspond to NIC
capability names so a convenience function that looks for the supported
Flags in a testpmd output is also added."
Having it prefixed with RX_OFFLOAD_ in NicCapability makes a lot of
sense since it is more explicit. Since there is a good reason to have
it like this, then the redundancy makes sense I think. There are some
ways to potentially avoid this like creating a StrFlag class that
overrides the __str__ method, or something like an additional type
that would contain a toString method, but it feels very situational
and specific to this one use-case so it probably isn't going to be
super valuable. Another thing I could think of to do would be allowing
the user to pass in a function or something to the helper-method that
mapped Flag names to their respective NicCapability name, or just
doing it in the method that gets the offloads instead of using a
helper at all, but this also just makes it more complicated and maybe
it isn't worth it.
I apologize for asking you about something that you already explained,
but maybe something we can get out of this is that, since these names
have to be consistent, it might be worth putting that in the
doc-strings of the flag for when people try to make further expansions
or changes in the future. Or it could also be generally clear that
flags used for capabilities should follow this idea, let me know what
you think.
>
> > + #: Device supports L3 checksum offload.
> > + RX_OFFLOAD_IPV4_CKSUM = auto()
> > + #: Device supports L4 checksum offload.
> > + RX_OFFLOAD_UDP_CKSUM = auto()
> > + #: Device supports L4 checksum offload.
> > + RX_OFFLOAD_TCP_CKSUM = auto()
> > + #: Device supports Large Receive Offload.
> > + RX_OFFLOAD_TCP_LRO = auto()
> > + #: Device supports QinQ (queue in queue) offload.
> > + RX_OFFLOAD_QINQ_STRIP = auto()
> > + #: Device supports inner packet L3 checksum.
> > + RX_OFFLOAD_OUTER_IPV4_CKSUM = auto()
> > + #: Device supports MACsec.
> > + RX_OFFLOAD_MACSEC_STRIP = auto()
> > + #: Device supports filtering of a VLAN Tag identifier.
> > + RX_OFFLOAD_VLAN_FILTER = 1 << 9
> > + #: Device supports VLAN offload.
> > + RX_OFFLOAD_VLAN_EXTEND = auto()
> > + #: Device supports receiving segmented mbufs.
> > + RX_OFFLOAD_SCATTER = 1 << 13
> > + #: Device supports Timestamp.
> > + RX_OFFLOAD_TIMESTAMP = auto()
> > + #: Device supports crypto processing while packet is received in NIC.
> > + RX_OFFLOAD_SECURITY = auto()
> > + #: Device supports CRC stripping.
> > + RX_OFFLOAD_KEEP_CRC = auto()
> > + #: Device supports L4 checksum offload.
> > + RX_OFFLOAD_SCTP_CKSUM = auto()
> > + #: Device supports inner packet L4 checksum.
> > + RX_OFFLOAD_OUTER_UDP_CKSUM = auto()
> > + #: Device supports RSS hashing.
> > + RX_OFFLOAD_RSS_HASH = auto()
> > + #: Device supports
> > + RX_OFFLOAD_BUFFER_SPLIT = auto()
> > + #: Device supports all checksum capabilities.
> > + RX_OFFLOAD_CHECKSUM = RX_OFFLOAD_IPV4_CKSUM | RX_OFFLOAD_UDP_CKSUM | RX_OFFLOAD_TCP_CKSUM
> > + #: Device supports all VLAN capabilities.
> > + RX_OFFLOAD_VLAN = (
> > + RX_OFFLOAD_VLAN_STRIP
> > + | RX_OFFLOAD_VLAN_FILTER
> > + | RX_OFFLOAD_VLAN_EXTEND
> > + | RX_OFFLOAD_QINQ_STRIP
> > + )
> <snip>
> >
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 11/12] dts: add Rx offload capabilities
2024-08-29 15:40 ` Jeremy Spewock
@ 2024-09-18 14:27 ` Juraj Linkeš
2024-09-18 16:57 ` Jeremy Spewock
0 siblings, 1 reply; 107+ messages in thread
From: Juraj Linkeš @ 2024-09-18 14:27 UTC (permalink / raw)
To: Jeremy Spewock
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On 29. 8. 2024 17:40, Jeremy Spewock wrote:
> On Wed, Aug 28, 2024 at 1:44 PM Jeremy Spewock <jspewock@iol.unh.edu> wrote:
>>
>> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
>> <juraj.linkes@pantheon.tech> wrote:
>> <snip>
>>> diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
>>> index 48c31124d1..f83569669e 100644
>>> --- a/dts/framework/remote_session/testpmd_shell.py
>>> +++ b/dts/framework/remote_session/testpmd_shell.py
>>> @@ -659,6 +659,103 @@ class TestPmdPortStats(TextParser):
>>> tx_bps: int = field(metadata=TextParser.find_int(r"Tx-bps:\s+(\d+)"))
>>>
>>>
>>> +class RxOffloadCapability(Flag):
>>> + """Rx offload capabilities of a device."""
>>> +
>>> + #:
>>> + RX_OFFLOAD_VLAN_STRIP = auto()
>>
>> One other thought that I had about this; was there a specific reason
>> that you decided to prefix all of these with `RX_OFFLOAD_`? I am
>> working on a test suite right now that uses both RX and TX offloads
>> and thought that it would be a great use of capabilities, so I am
>> working on adding a TxOffloadCapability flag as well and, since the
>> output is essentially the same, it made a lot of sense to make it a
>> sibling class of this one with similar parsing functionality. In what
>> I was writing, I found it much easier to remove this prefix so that
>> the parsing method can be the same for both RX and TX, and I didn't
>> have to restate some options that are shared between both (like
>> IPv4_CKSUM, UDP_CKSUM, etc.). Is there a reason you can think of why
>> removing this prefix is a bad idea? Hopefully I will have a patch out
>> soon that shows this extension that I've made so that you can see
>> in-code what I was thinking.
>
> I see now that you actually already answered this question, I was just
> looking too much at that piece of code, and clearly not looking
> further down at the helper-method mapping or the commit message that
> you left :).
>
> "The Flag members correspond to NIC
> capability names so a convenience function that looks for the supported
> Flags in a testpmd output is also added."
>
> Having it prefixed with RX_OFFLOAD_ in NicCapability makes a lot of
> sense since it is more explicit. Since there is a good reason to have
> it like this, then the redundancy makes sense I think. There are some
> ways to potentially avoid this like creating a StrFlag class that
> overrides the __str__ method, or something like an additional type
> that would contain a toString method, but it feels very situational
> and specific to this one use-case so it probably isn't going to be
> super valuable. Another thing I could think of to do would be allowing
> the user to pass in a function or something to the helper-method that
> mapped Flag names to their respective NicCapability name, or just
> doing it in the method that gets the offloads instead of using a
> helper at all, but this also just makes it more complicated and maybe
> it isn't worth it.
>
I also had it without the prefix, but then I also realized it's needed
in NicCapability so this is where I ended. I'm not sure complicating
things to remove the prefix is worth it, especially when these names are
basically only used internally. The prefix could actually confer some
benefit if the name appears in a log somewhere (although overriding
__str__ could be the way; maybe I'll think about that).
> I apologize for asking you about something that you already explained,
> but maybe something we can get out of this is that, since these names
> have to be consistent, it might be worth putting that in the
> doc-strings of the flag for when people try to make further expansions
> or changes in the future. Or it could also be generally clear that
> flags used for capabilities should follow this idea, let me know what
> you think.
>
Adding things to docstring is usually a good thing. What should I
document? I guess the correspondence between the flag and NicCapability,
anything else?
>>
>>> + #: Device supports L3 checksum offload.
>>> + RX_OFFLOAD_IPV4_CKSUM = auto()
>>> + #: Device supports L4 checksum offload.
>>> + RX_OFFLOAD_UDP_CKSUM = auto()
>>> + #: Device supports L4 checksum offload.
>>> + RX_OFFLOAD_TCP_CKSUM = auto()
>>> + #: Device supports Large Receive Offload.
>>> + RX_OFFLOAD_TCP_LRO = auto()
>>> + #: Device supports QinQ (queue in queue) offload.
>>> + RX_OFFLOAD_QINQ_STRIP = auto()
>>> + #: Device supports inner packet L3 checksum.
>>> + RX_OFFLOAD_OUTER_IPV4_CKSUM = auto()
>>> + #: Device supports MACsec.
>>> + RX_OFFLOAD_MACSEC_STRIP = auto()
>>> + #: Device supports filtering of a VLAN Tag identifier.
>>> + RX_OFFLOAD_VLAN_FILTER = 1 << 9
>>> + #: Device supports VLAN offload.
>>> + RX_OFFLOAD_VLAN_EXTEND = auto()
>>> + #: Device supports receiving segmented mbufs.
>>> + RX_OFFLOAD_SCATTER = 1 << 13
>>> + #: Device supports Timestamp.
>>> + RX_OFFLOAD_TIMESTAMP = auto()
>>> + #: Device supports crypto processing while packet is received in NIC.
>>> + RX_OFFLOAD_SECURITY = auto()
>>> + #: Device supports CRC stripping.
>>> + RX_OFFLOAD_KEEP_CRC = auto()
>>> + #: Device supports L4 checksum offload.
>>> + RX_OFFLOAD_SCTP_CKSUM = auto()
>>> + #: Device supports inner packet L4 checksum.
>>> + RX_OFFLOAD_OUTER_UDP_CKSUM = auto()
>>> + #: Device supports RSS hashing.
>>> + RX_OFFLOAD_RSS_HASH = auto()
>>> + #: Device supports
>>> + RX_OFFLOAD_BUFFER_SPLIT = auto()
>>> + #: Device supports all checksum capabilities.
>>> + RX_OFFLOAD_CHECKSUM = RX_OFFLOAD_IPV4_CKSUM | RX_OFFLOAD_UDP_CKSUM | RX_OFFLOAD_TCP_CKSUM
>>> + #: Device supports all VLAN capabilities.
>>> + RX_OFFLOAD_VLAN = (
>>> + RX_OFFLOAD_VLAN_STRIP
>>> + | RX_OFFLOAD_VLAN_FILTER
>>> + | RX_OFFLOAD_VLAN_EXTEND
>>> + | RX_OFFLOAD_QINQ_STRIP
>>> + )
>> <snip>
>>>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 11/12] dts: add Rx offload capabilities
2024-09-18 14:27 ` Juraj Linkeš
@ 2024-09-18 16:57 ` Jeremy Spewock
0 siblings, 0 replies; 107+ messages in thread
From: Jeremy Spewock @ 2024-09-18 16:57 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Sep 18, 2024 at 10:27 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
>
>
> On 29. 8. 2024 17:40, Jeremy Spewock wrote:
> > On Wed, Aug 28, 2024 at 1:44 PM Jeremy Spewock <jspewock@iol.unh.edu> wrote:
> >>
> >> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> >> <juraj.linkes@pantheon.tech> wrote:
> >> <snip>
> >>> diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
> >>> index 48c31124d1..f83569669e 100644
> >>> --- a/dts/framework/remote_session/testpmd_shell.py
> >>> +++ b/dts/framework/remote_session/testpmd_shell.py
> >>> @@ -659,6 +659,103 @@ class TestPmdPortStats(TextParser):
> >>> tx_bps: int = field(metadata=TextParser.find_int(r"Tx-bps:\s+(\d+)"))
> >>>
> >>>
> >>> +class RxOffloadCapability(Flag):
> >>> + """Rx offload capabilities of a device."""
> >>> +
> >>> + #:
> >>> + RX_OFFLOAD_VLAN_STRIP = auto()
> >>
> >> One other thought that I had about this; was there a specific reason
> >> that you decided to prefix all of these with `RX_OFFLOAD_`? I am
> >> working on a test suite right now that uses both RX and TX offloads
> >> and thought that it would be a great use of capabilities, so I am
> >> working on adding a TxOffloadCapability flag as well and, since the
> >> output is essentially the same, it made a lot of sense to make it a
> >> sibling class of this one with similar parsing functionality. In what
> >> I was writing, I found it much easier to remove this prefix so that
> >> the parsing method can be the same for both RX and TX, and I didn't
> >> have to restate some options that are shared between both (like
> >> IPv4_CKSUM, UDP_CKSUM, etc.). Is there a reason you can think of why
> >> removing this prefix is a bad idea? Hopefully I will have a patch out
> >> soon that shows this extension that I've made so that you can see
> >> in-code what I was thinking.
> >
> > I see now that you actually already answered this question, I was just
> > looking too much at that piece of code, and clearly not looking
> > further down at the helper-method mapping or the commit message that
> > you left :).
> >
> > "The Flag members correspond to NIC
> > capability names so a convenience function that looks for the supported
> > Flags in a testpmd output is also added."
> >
> > Having it prefixed with RX_OFFLOAD_ in NicCapability makes a lot of
> > sense since it is more explicit. Since there is a good reason to have
> > it like this, then the redundancy makes sense I think. There are some
> > ways to potentially avoid this like creating a StrFlag class that
> > overrides the __str__ method, or something like an additional type
> > that would contain a toString method, but it feels very situational
> > and specific to this one use-case so it probably isn't going to be
> > super valuable. Another thing I could think of to do would be allowing
> > the user to pass in a function or something to the helper-method that
> > mapped Flag names to their respective NicCapability name, or just
> > doing it in the method that gets the offloads instead of using a
> > helper at all, but this also just makes it more complicated and maybe
> > it isn't worth it.
> >
>
> I also had it without the prefix, but then I also realized it's needed
> in NicCapability so this is where I ended. I'm not sure complicating
> things to remove the prefix is worth it, especially when these names are
> basically only used internally. The prefix could actually confer some
> benefit if the name appears in a log somewhere (although overriding
> __str__ could be the way; maybe I'll think about that).
It could be done with modifying str, but I found that an approach that
was easier was just adding an optional prefix to the
_update_capabilities_from_flag() method since you will know whether
the capability is Rx or Tx at the point of calling this method. I feel
like either or could work, I'm not sure exactly which is better. The
change that adds the prefix is in the Rx/Tx offload suite in the first
commit [1] if you wanted to look at it. This commit and the one after
it are isolated to be only changes to the capabilities series.
[1] https://patchwork.dpdk.org/project/dpdk/patch/20240903194642.24458-2-jspewock@iol.unh.edu/
>
> > I apologize for asking you about something that you already explained,
> > but maybe something we can get out of this is that, since these names
> > have to be consistent, it might be worth putting that in the
> > doc-strings of the flag for when people try to make further expansions
> > or changes in the future. Or it could also be generally clear that
> > flags used for capabilities should follow this idea, let me know what
> > you think.
> >
>
> Adding things to docstring is usually a good thing. What should I
> document? I guess the correspondence between the flag and NicCapability,
> anything else?
The only thing I was thinking was that the flag values have to match
the values in NicCapability. I think explaining it this way is enough
just to make it clear that it is done that way for a purpose and
cannot be different (unless of course the no-prefix way is favorable).
>
> >>
> >>> + #: Device supports L3 checksum offload.
> >>> + RX_OFFLOAD_IPV4_CKSUM = auto()
> >>> + #: Device supports L4 checksum offload.
> >>> + RX_OFFLOAD_UDP_CKSUM = auto()
> >>> + #: Device supports L4 checksum offload.
> >>> + RX_OFFLOAD_TCP_CKSUM = auto()
> >>> + #: Device supports Large Receive Offload.
> >>> + RX_OFFLOAD_TCP_LRO = auto()
> >>> + #: Device supports QinQ (queue in queue) offload.
> >>> + RX_OFFLOAD_QINQ_STRIP = auto()
> >>> + #: Device supports inner packet L3 checksum.
> >>> + RX_OFFLOAD_OUTER_IPV4_CKSUM = auto()
> >>> + #: Device supports MACsec.
> >>> + RX_OFFLOAD_MACSEC_STRIP = auto()
> >>> + #: Device supports filtering of a VLAN Tag identifier.
> >>> + RX_OFFLOAD_VLAN_FILTER = 1 << 9
> >>> + #: Device supports VLAN offload.
> >>> + RX_OFFLOAD_VLAN_EXTEND = auto()
> >>> + #: Device supports receiving segmented mbufs.
> >>> + RX_OFFLOAD_SCATTER = 1 << 13
> >>> + #: Device supports Timestamp.
> >>> + RX_OFFLOAD_TIMESTAMP = auto()
> >>> + #: Device supports crypto processing while packet is received in NIC.
> >>> + RX_OFFLOAD_SECURITY = auto()
> >>> + #: Device supports CRC stripping.
> >>> + RX_OFFLOAD_KEEP_CRC = auto()
> >>> + #: Device supports L4 checksum offload.
> >>> + RX_OFFLOAD_SCTP_CKSUM = auto()
> >>> + #: Device supports inner packet L4 checksum.
> >>> + RX_OFFLOAD_OUTER_UDP_CKSUM = auto()
> >>> + #: Device supports RSS hashing.
> >>> + RX_OFFLOAD_RSS_HASH = auto()
> >>> + #: Device supports
> >>> + RX_OFFLOAD_BUFFER_SPLIT = auto()
> >>> + #: Device supports all checksum capabilities.
> >>> + RX_OFFLOAD_CHECKSUM = RX_OFFLOAD_IPV4_CKSUM | RX_OFFLOAD_UDP_CKSUM | RX_OFFLOAD_TCP_CKSUM
> >>> + #: Device supports all VLAN capabilities.
> >>> + RX_OFFLOAD_VLAN = (
> >>> + RX_OFFLOAD_VLAN_STRIP
> >>> + | RX_OFFLOAD_VLAN_FILTER
> >>> + | RX_OFFLOAD_VLAN_EXTEND
> >>> + | RX_OFFLOAD_QINQ_STRIP
> >>> + )
> >> <snip>
> >>>
>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 11/12] dts: add Rx offload capabilities
2024-08-21 14:53 ` [PATCH v3 11/12] dts: add Rx offload capabilities Juraj Linkeš
2024-08-26 17:24 ` Jeremy Spewock
2024-08-28 17:44 ` Jeremy Spewock
@ 2024-09-03 19:49 ` Dean Marx
2024-09-18 13:59 ` Juraj Linkeš
2 siblings, 1 reply; 107+ messages in thread
From: Dean Marx @ 2024-09-03 19:49 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 1391 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> The scatter Rx offload capability is needed for the pmd_buffer_scatter
> test suite. The command that retrieves the capability is:
> show port <port_id> rx_offload capabilities
>
> The command also retrieves a lot of other capabilities (RX_OFFLOAD_*)
> which are all added into a Flag. The Flag members correspond to NIC
> capability names so a convenience function that looks for the supported
> Flags in a testpmd output is also added.
>
> The NIC capability names (mentioned above) are copy-pasted from the
> Flag. Dynamic addition of Enum members runs into problems with typing
> (mypy doesn't know about the members) and documentation generation
> (Sphinx doesn't know about the members).
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
<snip>
> + RX_OFFLOAD_VLAN_FILTER = 1 << 9
> + #: Device supports VLAN offload.
> + RX_OFFLOAD_VLAN_EXTEND = auto()
> + #: Device supports receiving segmented mbufs.
> + RX_OFFLOAD_SCATTER = 1 << 13
>
This was an interesting section, I'm not super familiar with bitwise
shifting in python flags so I figured I'd ask while it's in mind if there's
any specific reason for shifting these two flags? Not a critique of the
code, just genuinely curious.
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 1930 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 11/12] dts: add Rx offload capabilities
2024-09-03 19:49 ` Dean Marx
@ 2024-09-18 13:59 ` Juraj Linkeš
0 siblings, 0 replies; 107+ messages in thread
From: Juraj Linkeš @ 2024-09-18 13:59 UTC (permalink / raw)
To: Dean Marx
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
On 3. 9. 2024 21:49, Dean Marx wrote:
> On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
>
> The scatter Rx offload capability is needed for the pmd_buffer_scatter
> test suite. The command that retrieves the capability is:
> show port <port_id> rx_offload capabilities
>
> The command also retrieves a lot of other capabilities (RX_OFFLOAD_*)
> which are all added into a Flag. The Flag members correspond to NIC
> capability names so a convenience function that looks for the supported
> Flags in a testpmd output is also added.
>
> The NIC capability names (mentioned above) are copy-pasted from the
> Flag. Dynamic addition of Enum members runs into problems with typing
> (mypy doesn't know about the members) and documentation generation
> (Sphinx doesn't know about the members).
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
>
> <snip>
>
> + RX_OFFLOAD_VLAN_FILTER = 1 << 9
> + #: Device supports VLAN offload.
> + RX_OFFLOAD_VLAN_EXTEND = auto()
> + #: Device supports receiving segmented mbufs.
> + RX_OFFLOAD_SCATTER = 1 << 13
>
>
> This was an interesting section, I'm not super familiar with bitwise
> shifting in python flags so I figured I'd ask while it's in mind if
> there's any specific reason for shifting these two flags? Not a critique
> of the code, just genuinely curious.
>
It's there just to mirror the flags in DPDK code.
> Reviewed-by: Dean Marx <dmarx@iol.unh.edu <mailto:dmarx@iol.unh.edu>>
^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH v3 12/12] dts: add NIC capabilities from show port info
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
` (10 preceding siblings ...)
2024-08-21 14:53 ` [PATCH v3 11/12] dts: add Rx offload capabilities Juraj Linkeš
@ 2024-08-21 14:53 ` Juraj Linkeš
2024-08-26 17:24 ` Jeremy Spewock
2024-09-03 18:02 ` Dean Marx
2024-08-26 17:25 ` [PATCH v3 00/12] dts: add test skipping based on capabilities Jeremy Spewock
12 siblings, 2 replies; 107+ messages in thread
From: Juraj Linkeš @ 2024-08-21 14:53 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman
Cc: dev, Juraj Linkeš
Add the capabilities advertised by the testpmd command "show port info"
so that test cases may be marked as requiring those capabilities:
RUNTIME_RX_QUEUE_SETUP
RUNTIME_TX_QUEUE_SETUP
RXQ_SHARE
FLOW_RULE_KEEP
FLOW_SHARED_OBJECT_KEEP
These names are copy pasted from the existing DeviceCapabilitiesFlag
class. Dynamic addition of Enum members runs into problems with typing
(mypy doesn't know about the members) and documentation generation
(Sphinx doesn't know about the members).
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/remote_session/testpmd_shell.py | 36 +++++++++++++++++++
1 file changed, 36 insertions(+)
diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index f83569669e..166ffc827e 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -1200,6 +1200,24 @@ def get_capabilities_rxq_info(
else:
unsupported_capabilities.add(NicCapability.SCATTERED_RX_ENABLED)
+ def get_capabilities_show_port_info(
+ self,
+ supported_capabilities: MutableSet["NicCapability"],
+ unsupported_capabilities: MutableSet["NicCapability"],
+ ) -> None:
+ """Get all capabilities from show port info and divide them into supported and unsupported.
+
+ Args:
+ supported_capabilities: Supported capabilities will be added to this set.
+ unsupported_capabilities: Unsupported capabilities will be added to this set.
+ """
+ self._update_capabilities_from_flag(
+ supported_capabilities,
+ unsupported_capabilities,
+ DeviceCapabilitiesFlag,
+ self.ports[0].device_capabilities,
+ )
+
"""
====== Decorator methods ======
"""
@@ -1332,6 +1350,24 @@ class NicCapability(NoAliasEnum):
RX_OFFLOAD_VLAN: TestPmdShellCapabilityMethod = partial(
TestPmdShell.get_capabilities_rx_offload
)
+ #: Device supports Rx queue setup after device started.
+ RUNTIME_RX_QUEUE_SETUP: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_show_port_info
+ )
+ #: Device supports Tx queue setup after device started.
+ RUNTIME_TX_QUEUE_SETUP: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_show_port_info
+ )
+ #: Device supports shared Rx queue among ports within Rx domain and switch domain.
+ RXQ_SHARE: TestPmdShellCapabilityMethod = partial(TestPmdShell.get_capabilities_show_port_info)
+ #: Device supports keeping flow rules across restart.
+ FLOW_RULE_KEEP: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_show_port_info
+ )
+ #: Device supports keeping shared flow objects across restart.
+ FLOW_SHARED_OBJECT_KEEP: TestPmdShellCapabilityMethod = partial(
+ TestPmdShell.get_capabilities_show_port_info
+ )
def __call__(
self,
--
2.34.1
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 12/12] dts: add NIC capabilities from show port info
2024-08-21 14:53 ` [PATCH v3 12/12] dts: add NIC capabilities from show port info Juraj Linkeš
@ 2024-08-26 17:24 ` Jeremy Spewock
2024-09-03 18:02 ` Dean Marx
1 sibling, 0 replies; 107+ messages in thread
From: Jeremy Spewock @ 2024-08-26 17:24 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
> Add the capabilities advertised by the testpmd command "show port info"
> so that test cases may be marked as requiring those capabilities:
> RUNTIME_RX_QUEUE_SETUP
> RUNTIME_TX_QUEUE_SETUP
> RXQ_SHARE
> FLOW_RULE_KEEP
> FLOW_SHARED_OBJECT_KEEP
>
> These names are copy pasted from the existing DeviceCapabilitiesFlag
> class. Dynamic addition of Enum members runs into problems with typing
> (mypy doesn't know about the members) and documentation generation
> (Sphinx doesn't know about the members).
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 12/12] dts: add NIC capabilities from show port info
2024-08-21 14:53 ` [PATCH v3 12/12] dts: add NIC capabilities from show port info Juraj Linkeš
2024-08-26 17:24 ` Jeremy Spewock
@ 2024-09-03 18:02 ` Dean Marx
1 sibling, 0 replies; 107+ messages in thread
From: Dean Marx @ 2024-09-03 18:02 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, alex.chapman, dev
[-- Attachment #1: Type: text/plain, Size: 719 bytes --]
On Wed, Aug 21, 2024 at 10:53 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> Add the capabilities advertised by the testpmd command "show port info"
> so that test cases may be marked as requiring those capabilities:
> RUNTIME_RX_QUEUE_SETUP
> RUNTIME_TX_QUEUE_SETUP
> RXQ_SHARE
> FLOW_RULE_KEEP
> FLOW_SHARED_OBJECT_KEEP
>
> These names are copy pasted from the existing DeviceCapabilitiesFlag
> class. Dynamic addition of Enum members runs into problems with typing
> (mypy doesn't know about the members) and documentation generation
> (Sphinx doesn't know about the members).
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 1051 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH v3 00/12] dts: add test skipping based on capabilities
2024-08-21 14:53 ` [PATCH v3 00/12] dts: add test skipping " Juraj Linkeš
` (11 preceding siblings ...)
2024-08-21 14:53 ` [PATCH v3 12/12] dts: add NIC capabilities from show port info Juraj Linkeš
@ 2024-08-26 17:25 ` Jeremy Spewock
12 siblings, 0 replies; 107+ messages in thread
From: Jeremy Spewock @ 2024-08-26 17:25 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dmarx, alex.chapman, dev
Hey Juraj,
Thanks for the series! This is definitely a large shift in how the
framework operates, but I think a lot of these changes are hugely
helpful and the code is very well written in general. I left some
comments mostly about places where I think some things could be a
little more clear, and one about a functional difference that I think
could be useful, but let me know what you think.
Also, I tried to apply this patch to help with the review process but
I couldn't get it to work. I think this is mainly due to the fact that
this uses the MTU updating commit on main, and my version of that
patch is far behind main right now, so we probably just resolved
conflicts differently somehow. I will work on updating that series now
and break the MTU patch into its own series to make it easier to use.
Thanks,
Jeremy
^ permalink raw reply [flat|nested] 107+ messages in thread